Coverity, Inc. issued a press release this morning announcing that "the LibreOffice team" has "analyzed more than 9 million lines of code to find and fix more than 6,000 defects." In the press release, Zack Samocha, senior director of products for Coverity, states, "LibreOffice’s remarkable results after just two years [since October 2012] of using the Coverity Scan service reiterates the mission criticality of software testing for the open source community to find and fix software defects early."
This press release cites the Coverity Scan 2013 Open Source Report in explaining the degree of success the LibreOffice team has achieved. Specifically, according to the press release, the LibreOffice team has "reduced the project’s defect density from .8 to .08."
I was curious about some of the specific details associated with LibreOffice's use of Coverity Scan to reduce defects and improve quality and so took advantage of an offer to ask Zack Samocha some questions. The remainder of this post indicates my questions, Zack's answers, and some related references.
Q: What programming languages are used for LibreOffice (all/mostly C++ or some Java or other languages)?
A: The language used for LibreOffice is mostly C++.
Q: What is an example of one of the most common types of defects discovered and fixed in LibreOffice?
A: The top issues were:
- Error handling issues = 2271
- Null pointer dereferences = 1796
- Uninitialized members = 1145
Q: Is this typical of other open source projects analyzed with Coverity Scan?
A: This is comparable for OSS projects [with more than] 1 million lines of code (LOC)
Q: What is an example of one of the most serious types of defects (high-impact) discovered and fixed? Is this typical of other open source projects analyzed with Coverity Scan?
A: The most serious are memory related. For example, memory-illegal accesses (there were 23) and memory–corruptions (there were 17). This amount is common for such large code base.
Q: Are there any additional metrics regarding the fixes to LibreOffice using Coverity Scan such as number of developers or number of person hours spent on the effort? Is there any estimate of how much of this effort was identifying the issues (running the scan) versus fixing them and testing the fixes?
A: In the past year, LibreOffice fixed more than 8,500 defects, assuming at least one hour per defects, which is conservative. That's about 365 days of work for a single developer.
Q: How does Coverity Scan differ from FindBugs, PMD, other static analysis tools, and IDEs' built-in static analysis support? What advantages does Coverity Scan offer instead of or in addition to those tools?
A: At Coverity, we believe in open source collaboration. Coverity complements Findbugs, PMD and others. In fact, in Coverity Scan and our enterprise products, we provide FindBugs defects in the same workflow as defects found by Coverity development testing solutions. Coverity and FindBugs are looking for different things. Coverity is designed to find critical, crash-causing defects where FindBugs is best suited for find coding-style and best practice-type issues. To illustrate the point, we conducted an experiment with the Jenkins open source build server under a controlled environment. We found 296 critical defects while FingBugs found 1010 coding style issues. There were only 30 defects that were found by both solutions. Coverity analysis tends to be more inter procedural, in addition Coverity covers OWASP10 for Security issues.
- LibreOffice Coverity Defect Density
- "We run LibreOffice through Coverity approximately once a week."
- LibreOffice Happy To Work With Coverity Scan Results
- "The work on using the Coverity reports for LibreOffice is done by a variety of LibreOffice developers, some on the building and testing, others on the other work to fix the issues. In the first months many hundred improvements have been made, making LibreOffice more robust, better."
- LibreOffice 4.3, The Almost Perfect Office Suite
- What Does OWASP Top 10 Coverage Mean to You ... and Do You Have It?
- OWASP Top Ten Project