There’s been a lot of confusion (and Fear Uncertainty and Doubt) about The Linux Foundation’s Hyperledger Fabric performance and scale; and, admittedly, a lack of information about best practices that can help you yield improved performance and scale.

The reality is that Hyperledger Fabric does perform and scale nicely, given the right compute and networking infrastructure. However, it can also yield results that are inadequate for your application’s needs. As with any newish technology, the impacts on performance range from improper (or should I say unexpected) use, improper fabric application to software, and hardware bottlenecks to be addressed over time.

I’m old enough to remember my first experiences with relational databases, which at the time were brand spanking new to the enterprise, much as blockchain technology is today. I recall kicking the tires in some early testing where you could launch a query and go for a cup of coffee before the results were returned (okay, that’s admittedly an exaggeration).

The reality is that over the years (decades, actually), the engineering of relational database engines underwent a continuous evolution to improve performance and scale capabilities. Additionally, developers writing applications that leveraged them have had to learn how to best construct schemas, indexes, queries and stored procedures to avoid performance nightmares of their own making.

As a result, I’m kicking off this blog series to report on various aspects of Hyperledger Fabric performance and scaling with the intent of helping to set the record straight, provide best practices guidance, and to provide you with the latest information on related improvements to Hyperledger Fabric.

Learn how Hyperledger makes revolutionary cooperation and innovation possible

Some caveats before we begin

Before we begin, I think it is important to remind everyone that there is no one metric that applies generally — no single metric that applies to all use cases and all circumstances. As the saying goes for the U.S. Environmental Protection Agency regarding its miles per gallon (MPG) ratings: “Your Mileage May Vary”(YMMV) according to your driving style. Just as your gas mileage suffers. If you (like me) like to accelerate quickly when merging onto the highway, your MPG will decrease. Similarly, transactions involving simple tokens will generally perform better than transactions that have larger amounts of information, have more complex schema, and/or involve more interaction with the state database. Further, the specifics of your endorsement policy(s) will also likely have an impact on overall performance as will the complexity of your application — both the client and the chaincode.

The purpose of this blog series is to help you get the most out of your Hyperledger Fabric-based blockchain applications and network components, as well as to provide up-to-date information on our plans to constantly improve the performance and scale capabilities of the platform.

For more information about blockchain performance and metrics, I would strongly encourage you to review (and keep handy) the Hyperledger Performance and Scale Working Group’s white paper on Blockchain Performance Metrics. For purposes of this series, I will be using the defined metrics from that paper  — such as transaction throughput or, more commonly transactions per second (TPS).

Anatomy of a blockchain app’s performance

It is important when considering the performance of any blockchain framework that there are a number of factors that can (and will) effect performance of the framework itself.

First, there is the application client — the component that sends transactions to the endorsing nodes. The language in which the client application is written may influence the choice of Hyperledger Fabric software developer kit (SDK) like Go, Node.js, Java, Python and others. The choice of SDK can have an effect on performance based on the language in which it is written. The Go and Java SDKs may yield better performance than the Node.js or Python SDK simply because they are compiled and support multiple threads of execution.

Additionally, there’s the client software itself. Let’s be honest, no software is defect-free and hence when analyzing the performance of your blockchain application, you should examine all aspects of the application, including your own code. As an example, in reviewing performance concerns of an application, we found that the client software that was invoking the SDK was leaking open connections to the peer that were not being closed until they timed out. A simple change to the application code resolved that problem.

Secondly, there is the Hyperledger Fabric peer (endorser/committer) and choice of ledger database — currently, LevelDB and CouchDB, but over time, other choices will likely emerge. The LevelDB database performs better than CouchDB, generally, but is not as effective at supporting a rich schema for the world state. If you are dealing with simple key-value data, then you might opt for LevelDB.

Additionally, the choice of endorsement policy and the number of available endorsing nodes for a channel can impact peer performance. We’ll dive into that discussion below.

Third, there is the smart contract, or what Hyperledger Fabric calls chaincode. Presently, Hyperledger Fabric offers a variety of chaincode implementation choices: Go, JavaScript, Java and an implementation of the Hyperledger Burrow Ethereum Virtual Machine (EVM) that supports execution of smart contract bytecode compiled from one of the Ethereum-smart contract languages such as Solidity. As with the SDK, choosing Go or Java for your chaincode may yield performance benefits over JavaScript, for the same reasons.

Fourth, the ordering service node (OSN) may also have an impact on performance. One factor will be the choice of consensus mechanism. While presently, Hyperledger Fabric has but two modes (solo and Kafka), as we add other protocols that choice may also have an impact on performance. Realistically today, there is but a single choice as the “solo” OSN is not suitable for production usage, but that will be changing soon as we add support for Raft and Byzantine Fault Tolerance (BFT) OSNs in 2019.

Another aspect of the OSN that can have an impact on performance are the configuration parameters that determine the block size and frequency. Modifying these can influence the latency that transactions experience from submission to the OSN to being committed to the ledger. If too few messages are received for a channel to fill the block (per the BatchSize configuration), then the minimum latency will be the BatchTimeout.

Fifth, there is the distribution of work within your blockchain network based on how you have architected use of channels and/or private transactions to deliver privacy. Channels allow you to choose which organizations participate in transactions directed to that channel. Hence, you might limit transaction traffic to the two organizations involved in a transaction plus say an auditor(s) or regulator(s) as neutral observers. The transaction traffic on this channel will not have an effect (other than in shared virtual contexts) on the other organizations in your network.

Finally, there is the physical and/or virtual infrastructure that all of these services run on that can affect performance. For instance, giving the peer nodes more vCPUs can help improve performance in many cases. Of course, this comes at a cost, so you will want to balance the cost of running a peer with the benefits of the improved performance. Disk and memory are also factors as we will explore in future posts in this series.

Let’s start taking your questions

What I hope to accomplish with this series of blog posts is to answer the question: What affects performance? When some aspect is known to affect performance, I also hope to be able to provide the reader with best practices to mitigate performance concerns.

So, let’s start by asking some questions:

Do any of the following effect performance of Hyperledger Fabric?

• Number of endorsing peers
• Number of channels
• Number of endorsements (endorsement policy)
• Ordering service configuration (block size and frequency)
• Number of organizations
• Ledger database used
• Complexity of chaincode/smart contract execution
• Size of transactions
• Use of mutual TLS for all network traffic
• Number of vCPUs
• Memory allocation
• Disk type and speed
• Network speed
• Multiple datacenter deployment
• CPU speed
• Crypto acceleration

It is not my intent to address all of these in one go, but rather to deliver a cadence of posts that start to answer these and other questions. Additionally, as I address these and other questions, I will articulate the experiments used in a manner that hopefully can be easily reproduced.

Additionally, as appropriate, I will update on the development roadmap any items that we are tracking that relate to performance and scale improvements.

Let’s get started!

Let’s start with the first question: Does the number of endorsers effect performance?

The short answer is, “yes, yes it does.” The longer answer is going to depend on your network composition.

Let’s start with some basic contextual assumptions. Say you have a network that has two organizations with two peers each for resilience. These peers are each joined to a channel comprised of the two organizations. For purposes of this experiment, let’s say that the endorsement policy requires a single endorsement from either organization. Let’s further start with only one of each organization’s peers configured as an endorser (has chaincode installed and instantiated) — meaning I have a total of two endorsers from which to choose to obtain an endorsement. The transaction load generator will randomly select one of the available endorsers for each transaction proposal, load balancing the endorsement processing.

For purposes of this experiment, the network is running Hyperledger Fabric 1.3.0 in a single Kubernetes cluster running on the IBM Container Service with worker nodes configured as 4vCPU and 16Gb memory with SSDs — nothing exotic here. I’ll focus on the impact of the VM configuration in another post in the series.

Now, let’s run some experiments and see what effect there is on overall throughput as we load balance the submitted transactions across the ever-larger pool of endorsers.

The following table lists the throughput (TPS) and average latency (in ms) for 2, 4, and 8 endorsers for a two-organization cluster running a single channel.

Not bad! However, I don’t want you all thinking that Hyperledger Fabric transaction throughput is somehow limited to the TPS shown above. This experiment used a rather simple chaincode that simply does some encryption/decryption on some random bytes and stores those values in the ledger/world state.

Rather, the important take-a-way from the above is that you can scale your throughput by load balancing your endorsement across a pool of endorsers.

Best practice

If you can, increase the pool of available endorsing peers and load balance across that set to achieve greater TPS, improve latency or both. Note that transaction latency is also a function of the time your chaincode takes to execute for an endorsement, so as noted above, YMMV.

Dealing with latency

Note that I haven’t mentioned latency until now. The reality is that as you scale the rate of transactions sent on a channel, you will likely reach a point where transaction latency becomes untenable because the peers become saturated consuming all the available CPU and/or disk i/o allocated to the container. For purposes of this set of experiments, I endeavored to keep the average latency under one second. We can actually scale beyond the TPS cited above if we are willing to tolerate additional latency, though with diminishing returns because while you may distribute the processing of endorsement across a set of available endorsers, every peer that participates in a channel acts as a committer and must validate the transactions received from the ordering service.

Scaling Hyperledger Fabric

Another myth (or maybe it is FUD?) that I have heard recently is that Hyperledger Fabric limits the number of peer nodes in a network. I’m not sure of the source of this myth, but nothing could be further from the truth! As a function of our system testing for a release, we kick off a scale test to ensure that we have not introduced any changes that might affect performance and scale. Our testing creates a network comprised of 32 organizations each with four peers for resilience (a total of 128 peers) and establishes a channel between each pair of organizations for a total of 325 channels — six of the organizations are configured as auditor or regulator nodes that observe each channel but do not perform endorsement.

Development roadmap items

Of course, when addressing performance and scale, the work of optimizing software is never done. The Hyperledger Fabric maintainers take performance seriously, but we have also tried to avoid premature optimization. There are a number of researchers looking at performance and offering up suggestions for how Hyperledger Fabric might be improved. There are also members of the Hyperledger development community who have shared their thoughts. One aspect that we are currently addressing is the validation pipeline. We know that some of the steps in validation can be run in parallel and so we are refactoring the pipeline to both enable that as well as to enable alternate validation modules to be more easily introduced.

We have also recently introduced a new metrics service that will enable us to shed light on performance directly (rather than indirectly as measured externally) that we can use to further address bottlenecks that impact performance and scale. We will be adding finishing touches to this new service for the 2.0 release.

Stay tuned for more answers

In the next post in the series, I hope to address a few more of the questions above, such as the effects of adding organizations and peers to a channel, modifying the endorsement policy to add additional endorsements for each transaction, and modifying the ordering service’s configuration. I’ll also be looking at the impact of boosting the number of CPUs available to the peers. As a tease, adding CPUs does indeed scale performance.

I hope you find this series valuable. If so, please do share your feedback. If you have specific performance or scale questions or concerns, or suggestions for future posts in the series, we would welcome those as well.

Start developing your blockchain with the open source Hyperledger Fabric