DevOps

Choosing open source frameworks

Share this post:

Open source communities are a fantastic environment of opportunities. Given a specific assignment, there are often many existing community-developed tools that can solve your problem. However, it’s often difficult to confidently select the right one.

When comparing similar open source tools or libraries, we typically consider factors such as ease of adoption, performance, scale, migration, etc. If you’ll be using the code longer-term or as a critical piece of your infrastructure, you’ll also want to consider things such as the community’s involvement, maturity, number of adopters, and pervasiveness of the project.

In this blog, I’ll evaluate two popular frameworks, considering factors such as:

  • Compare best practices between projects
  • Compare social culture and developer interactions
  • Compare file defect likelihood between projects.

These factors are consistent with the client-engagement proven IBM Cloud Garage Method Practices, notably Code, Culture, and Deliver.

In the first part, I’ll describe my initial attempts with comparison tools and community searches. Then in the second part, I’ll demonstrate how to use Developer Insights, a part of IBM Cloud DevOps Insights, to perform a more insightful comparison of the two frameworks.

Two popular open source frameworks – which to choose?

I recently installed Kubernetes to learn more about it. I decided to use a bootstrapping tool called kubeadm. I followed these instructions. However, I soon came to Section 3, which reads:

“You must install a pod network add-on so that your pods can communicate with each other.”

The instructions provide a link that describes eight (!?!) network add-ons.

The network add-ons Flannel and Weave are both are very popular with excellent reputations, so I decided to compare them. Aside from simply looking at GitHub metrics such the number of stars, forks, contributors and issues, there are nice comparison tools for this.

First attempt with GitHub comparison tool: A really nice tool github-compare from written by Brian Payne that does a great job of placing the metrics side-by-side and it also brings in Google trends:

At a quick glance we can easily see that both projects are about the same age and both are based on Golang. Flannel has nearly 50{07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0} more contributors but Weave is leading in Google trends, has a substantially larger number of commits, and a community that’s over two times larger than Flannel.

However, this doesn’t tell the whole story – why is there such a disparity in commits? Which project is working on more features vs bugs? Which tool will be more reliable? Can we predict this? I needed to dig more.

Second attempt with other tools and searches: I found another tool called Libraries.io. It has a nice SourceRank score. The results were a 12 for Flannel, 13 for Weave. A quick Stack Overflow comparison yielded 143 results for ‘Flannel Network’ and 127 results for ‘Weave Network’. Finally, I checked some blogs, hackernews, and reddit. Although there is a lot of great information, but it was difficult to separate opinions from actual data-based comparisions. I wanted analytics to look into metrics and provide real insights.

IBM Developer Insights comparison of Weave and Flannel

To set it up, follow the instructions at the end of this post. Once the mining and analyzing is complete, we’re able to compare these two projects using two browsers or multiple tabs.

From the Developer Best Practices, click on ‘Developer Best Practices’ on the left navigation for both projects. All metrics (linkless commits, big commits, defective files, etc.) are displayed in a card.

Some cards have a red X in the top right corner indicating some type of potential issue. To reveal the issue, hover your mouse over the bottom part of the card and click on the ‘eye’ icon.

I went through all cards for both projects and the results are summarized below;

Flannel:

  • Over 20{07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0} of the commits (95.45{07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0}) do not reference an issue. Untracked code is being pushed to the branches.
  • There are 2 commits that contain an unusually high number of files. The biggest one contains 11214 files. The average number of files changed per commit is 324.83.
  • More than 20{07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0} of the files (91.12{07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0}) are classified as defective.**
  • More than 80{07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0} of the work is made by less than 20{07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0} of the developers.
  • Over 20{07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0} of the issues (63.74{07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0}) do not have a type.

Weave:

  • Over 20{07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0} of the commits (94.69{07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0}) do not reference an issue. Untracked code is being pushed to the branches.
  • There are 11 commits that contain an unusually high number of files. The biggest one contains 68 files. The average number of files changed is 2.56.
  • More than 20{07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0} of the files (23.07{07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0}) are classified as defective.
  • More than 80{07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0} of the work is made by less than 20{07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0} of the developers.
  • Over 20{07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0} of the issues (67.89{07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0}) do not have a type.

A few disclaimers—at the time of this blog, Developer Insights does not support pull requests.  Most projects will link issues using the pull request as opposed to commits and these are not yet linked by the tool.  Pull requests will be supported soon.  Additionally, the tool also should be able to filter out very large commits that are often required, for example, this commit in Flannel.

Many of these statistics seem poor at first glance. However, neither project linked many commits to issues (git commit –m “#issue123 message…”). More importantly, both projects had many issues without a type; this made the analysis difficult. The high percentage defective files is likely because of this factor. It seems that both of these projects could do a better job of referencing the issue of the commit.

Flannel seems to check in very large commits. The average number of files changed per commit is 324.83. This is extremely high. We typically see open source projects average somewhere between 7-9 changes per commit. Looking at the social coding graph gives more insight into how these projects are developed.

More insights via the Social Coding story

Developer Insight contains a graph to show a team’s social coding interactions. This enables you to see opportunities to assign mentors to new developers and identify hero projects to cross-train developers. There’s an excellent blog on this How to strengthen your dev team with insights on social coding and How to strengthen your development team with IBM DevOps Insights.

To compare social coding graphs, click on ‘Team Dynamics’ on the left navigation and select ‘Social Coding’. You can use the range drop down to compare similar time ranges; for this comparison I selected ‘All time’.

Weave seems to have many contributors doing equal shares of work with a high degree of social interaction. Flannel seems to have less primary authors doing the majority of the contributions, with less social interaction between them. Nothing alarming here, just interesting insights into the social culture.

Compare File Error Proneness

Developer Insights collects and analyzes hundreds of metrics from your development tools such as GitHub, GitLab, and Jira.  The tool leverages machine learning to predict defect likelihood for all of your files and commits.  Let’s compare these error prone file predictions. Under Developer Insights on the left navigation, click on Error Proneness. Click on ‘Error prone file prediction’ tab at the top. Use the mouse to hover on various files around the perimeter of the sunburst chart. Reasons appear above the chart for files with high defect likelihood (note: edges of the chart are files. Redder color indicates files with higher defect likelihood).  The reasons were determined by the machine learning models and are specific to the project and file. The explanations are still under development and should become clearer as the project matures.

Edges of chart are files. Redder color indicates files with higher defect likelihood.

Below are the two projects shown side-by-side:

Please note that as previously mentioned, the defective likelihood was negatively affected because pull requests are not yet supported; the defective likelihood was also affected because neither project labeled all of its issues nor consistently referenced the issue of the commit. Issues without a type (or label) for Weave and Flannel were found to be 62.6{07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0} and 61.25{07c2b926d154bd5dc241f595a572d3349d41d98f2484798a4a616f4fafe1ebc0}, respectively. However, as shown above, it’s still quite interesting to review their relative differences and identify hot spots in the projects.

Conclusion

When comparing tools you will certainly look for a solution that fits your needs; issues such as performance, encryption, requirements, support, and even price will all be critical.

Developer Insights affords a better understanding of the authors that created the tool and insights into their social culture. This information helps identify healthy projects with vibrant communities, or detects projects with only a few primary contributors or projects with little social interaction.

Developer Insights can also quickly identify projects that do not follow general best practices such as not linking to issues in the commits, having very large commits, high average number of files changed per commit, high average defect likelihood or many other potential red flags.

Selecting the right open source project can be very challenging. You’ll need to make lots of critical decisions. IBM Developer Insights can an offer a wealth of information that is otherwise not readily available to help you make these critical decisions.

Ready to try it yourself on your code repository? It’s free.

All the instructions you need to get started are below and the tutorial IBM Cloud DevOps Insights shows you more examples of how to apply analytics to your DevOps projects.

 


For those who want to look more closely at Developer Insights, below are the setup instructions associated with this post.

Configure IBM Developer Insights with Flannel and Weave

To compare two projects, you will need to load each into IBM® Cloud DevOps Insights and compare these projects using Developer Insights. Developer Insights uses DevOps toolchains and although you can add many repositories to a single toolchain, we will use two separate toolchains for Flannel and Weave. This is because the machine learning algorithms of Developer Insights aggregates all the data for the repositories per toolchain to create the analytics (a.k.a., models). Therefore, in order to compare Flannel and Weave, we’ll need to separate them and create a toolchain for each.

To set it up, follow the instructions in Create a toolchain from the Developer Insights and Team Dynamics with GitHub and JIRA template to create a separate toolchain for Flannel and Weave projects.
The process will require GitHub configuration. For the Flannel toolchain, select ‘Existing’ under Repository type, enter https://github.com/coreos/flannel for the Source repository URL.

After your toolchain is configured, DevOps Insights mines and analyzes your project. To track the progress of the mining and analysis, from your toolchain, click DevOps Insights and click Developer Insights to see its status at the top of the page.

Now repeat this process for Weave using https://github.com/weaveworks/weave for the Source repository URL.

More DevOps stories
May 7, 2019

We’ve Moved! The IBM Cloud Blog Has a New URL

In an effort better integrate the IBM Cloud Blog with the IBM Cloud web experience, we have migrated the blog to a new URL: www.ibm.com/cloud/blog.

Continue reading

May 6, 2019

Use IBM Cloud Certificate Manager to Obtain Let’s Encrypt TLS Certificates for Your Public Domains

IBM Cloud Certificate Manager now lets you obtain TLS certificates signed by Let’s Encrypt. Let’s Encrypt is an automated, ACME-protocol-based CA that issues free certificates valid for 90 days.

Continue reading

May 6, 2019

Are You Ready for SAP S/4HANA Running on Cloud?

Our clients tell us SAP applications are central to their success and strategy for cloud, with a deadline to refresh the business processes and move to SAP S/4HANA by 2025. Now is the time to assess, plan and execute the journey to cloud and SAP S/4HANA

Continue reading