In 2001, IBM Research cryptographer Christian Cachin and his team developed consensus protocols for building distributed trusted services, which could be run among a group of servers over the Internet. As described in an IEEE paper titled Distributing Trust on The Internet, their protocols work correctly even if some servers are being hacked.
Does this sound familiar? It should. This idea of an architecture which requires consensus for a transaction to be valid is exactly what makes the blockchain so interesting as a public ledger, without a central authority.
While Cachin couldn’t have predicted it then, some 15 years later he is now part of a growing global team of IBM scientists who have spent the last year developing many of the protocols and security features which make the blockchain a viable technology for the enterprise.
Achieving consensus among mistrusting peers has become a key feature for blockchain networks which aim at processing transactions with real-world assets, ranging from a digital land registry to global logistics to aircraft maintenance. Just look at the number of recent projects from Walmart, ABN AMRO, CLS Group and Bank of Tokyo-Mitsubishi UFJ and you can see that the future looks bright for blockchain.
An international team of scientists based at IBM’s Zurich lab have made tremendous progress in their first year, particularly for building the Hyperledger Fabric, which is part of the Linux Foundation’s open-source blockchain development. After releasing preliminary versions this year, the Hyperledger Fabric is currently changing to a new architecture aimed at a modular, scalable, and secure foundation for industrial blockchains, which will be ready in early 2017.
Cachin and his colleague Marko Vukolic look back at what the team accomplished this year and what is planned for 2017.
Q: How did you get involved with developing protocols for blockchain?
Christian Cachin (CC): As often in research, new technology with fascinating features is developed but sees no immediate applicability in industry, simply because the environment is not ready for it. This was our situation in the early 2000s. Although the cryptographic tools were around, the market did not have any demand for consensus protocols and distributed ledger technology. We abandoned the topic and focused on other interesting topics.
But as IBM began ramping up its efforts in Blockchain for the enterprise in 2015, we saw a perfect opportunity to dust off our consensus protocols. Today we are a key contributor of the worldwide team that develops Hyperledger Fabric, and have daily contact with colleagues from IBM and several other members.
Q: Where is the difficulty with consensus protocols for blockchain?
Marko Vukolic (MV): Since the protocols must resist disruption by malicious peers, we need solutions that not only work when everyone plays nicely, but that are hardened enough to tolerate attacks from peers within the system. It takes the mindset of a both a system engineer and a security engineer (or a cryptographer) to foresee the potential attacks and network problems. Just like with physical security, we rarely prepare for attacks from within, but it’s an unfortunate reality.
Therefore, we need formal mathematical arguments to justify why a design works. Since I had already started to develop “Byzantine fault-tolerant” or “BFT” consensus protocols during my Ph.D. at EPFL, I have the necessary experience with the technical subject of reaching consensus in a distributed system.
In developing blockchain consensus our role is not only contributing to and reviewing critical parts of the code, but also in educating new team members about known issues they should avoid. It helps that we can even turn to our own textbooks, like the survey of quorum systems that I wrote or the introduction to distributed computing that Christian co-authored.
Q: In addition to contributing to the blockchain code in Hyperledger this year, you also pushed the frontiers of consensus protocols. Can you tell us more about this?
MV: Indeed we developed “cross fault tolerance”, abbreviated XFT, a novel approach to building reliable and secure distributed systems. It aims at more efficient protocols than the BFT consensus we knew so far. A BFT system is always prepared for the worst possible circumstances and pays a price for this in terms of speed. BFT is slower and needs more peer nodes than other methods for consensus, which tolerate only crashes, known as CFT (crash fault-tolerant) protocols, such as the widely used Paxos and Raft protocols.
XFT is practically as efficient as CFT and provides all of the CFT reliability guarantees, but also tolerates malicious actions (those “Byzantine” faults) in combination with network asynchrony, as long as a majority of the peers are correct and communicate synchronously. This eliminates a major overhead inherent in BFT solutions. In a sense, XFT cuts a corner in a smart way and ignores possible attacks that are deemed to be very costly and extremely unlikely today.
XFT was developed in collaboration with researchers from Eurecom and the Grenoble Institute of Technology in France. The research work was published at the USENIX Symposium on Operating Systems Design (OSDI 2016). Since the consensus method in Hyperledger Fabric is modular, we plan to integrate an XFT consensus protocol in a future version, which will make Fabric more efficient and powerful than today.
CC: Providing an even more immediate impact on the Hyperledger Project we also addressed how deterministic programs can be handled in blockchain networks. A blockchain runs so-called smart contracts, which are programs deployed by the users or operators of a blockchain. In principle, such a blockchain program must be deterministic and always produce the same results when it is executed again from the same inputs. But for many reasons this is difficult to achieve in practice, because code from any programmer should run on the blockchain. A non-deterministic smart contract is a software bug and we must protect the blockchain from it. So we developed a modular solution that eliminates diverging outputs as a result of non-deterministic operations in a consensus network.
This research on handling “Non-determinism in Byzantine Fault-Tolerant Replication” has just been published at the 20th International Conference on Principles of Distributed Systems (OPODIS 2016), which took place in Madrid. Our protocol was part of the early “developer-preview” releases of Hyperledger Fabric.
Q: How big and connected is the global research effort from both government, industry and academia on consensus protocols?
CC: You touch on an important point. We believe that the best results come from open collaboration among capable and motivated teams. For example, we have an ongoing collaboration on consensus protocols with academia in the context of several European Commission’s Horizon 2020 projects, including SUPERCLOUD.
In addition, IBM has recently granted a Faculty Award to Prof. Alysson Bessani at the University of Lisbon, who is a world-class expert in building BFT consensus protocols. He has already realized the BFT-SMaRt toolkit for implementing BFT consensus over the Internet. It is the state-of-the-art solution available today for this task, and the only open-source project developing BFT consensus, prior to the Hyperledger Fabric.
Through our collaboration we want to achieve two goals: First we will adapt BFT-SMaRt to Hyperledger Fabric so that it can provide consensus for a Hyperledger blockchain. Second, we engage in joint research between IBM Research – Zurich and his group to assess and improve the scalability and performance of consensus protocols for blockchain. Of course, we look forward to other interested teams joining the development of blockchain consensus for Hyperledger Fabric as well.