Coverity Scan 2013 Open Source Report

The Heartbleed Bug has received significant attention lately and has reignited discussions regarding open source security issues and open source quality issues. The article Heartbleed: Open source's worst hour goes so far as to open with the sentiment that Heartbleed is "open source software's biggest failure to date." In the midst of this discussion, the Coverity Scan^TM 2013 Open Source Report has been released and provides another interesting source of input for the discussion.

Coverity Scan's^TM main page states that it uses static analysis to "find and fix defects in your C/C++ or Java open source project for free." Coverity, which was recently acquired by Synopsys, originally teamed up with the Department of Homeland Security to develop the Coverity Scan^TM as part of the "Open Source Code Hardening Project." Last year's edition, the Coverity Scan: 2012 Open Source Report, found that "Code quality for open source software continues to mirror that of proprietary software–and both continue to surpass the accepted industry standard for good software quality." The just-released 2013 Coverity Scan^TM Open Source Report reports a change this year, "Open source code quality surpasses proprietary code quality in C/C++ projects."

Although the Coverity Scan^TM Open Source Report has mainly focused on the "state of open source software quality" in terms of C/C++ projects and Linux in the past, the 2013 report also adds Java-based open source projects Apache Cassandra, Apache CloudStack, Apache Hadoop, and Apache HBase. The report acknowledges that "we are still in the early days of working with Java projects" and looks at some possible explanations for the Java code that was analyzed having higher defect rates than the C/C++ code that was analyzed. These reasons include Java source code being new to the analysis (and thus not benefiting from being able to address previous results) and the use of FindBugs ("Many of the FindBugs checkers generate large quantities of results, in particular in the areas of dodgy code, performance and bad practices").

One of the other "key differences" analyzed in the 2013 Coverity Scan Report^TM is a lower percentage of "resource leaks" being fixed in analyzed Java code than in analyzed C/C++ code. The report's authors postulate that this might be explained by Java developers relying more on "some of the built-in protections in the language, such as the garbage collection." The authors point out potential fallacies of those types of reliance.

The 2013 Coverity Scan Report^TM includes an interesting assessment, "Quality concerns are no longer a barrier to open source adoption in the enterprise. In fact, the quality of the open source code for Coverity Scan participants can be higher than the proprietary code included in an enterprise product." Although not all open source is created equal and although product A is not necessarily superior to product B simply because the former is open source and the latter is proprietary, it is interesting to see more empirically driven studies demonstrating advantages of open source rather than relying on opinion, wishful thinking, and anecdotal evidence.