Open Source Code Quality?

So as I've noted in the past, Coverity has been using their tool to scan Open Source software as what I presume is a rather clever marketing campaign. This is the third year they've been doing this and claim that there is a 16 percent reduction is flaws found. They are of course claiming that the software has become 16 percent better, not that their tool is 16 percent worse.

A story like this is probably misleading and can be really hard to understand. What the real story is that Open Source software has gotten better at fixing the sort of things that Coverity scans for. It's another debate of course if this is the result of Coverity scanning the source, or better upstream practices. I would be most interested in seeing how many bugs Coverity found were fixed, if that 16 percent lines up with their previously found bugs.

Understanding statistics is really hard to do and even harder to properly analyze. A story such as this is often misleading, and the organization writing it will make sure it's in their favor. This is best explained with an example. Imagine you have a group of people at risk for heart attacks. It's also quite likely that these people have poor diets that put them in this position. Now suppose you conduct a study where they eat oatmeal for breakfast. It turns out, the folks who ate the oatmeal had fewer heart attacks. So obvious oatmeal is a magic food that prevents heart attacks! ... right? So there are a few ways to look at this. The most likely is that these folks were eating terrible things for breakfast, so basically by eating pretty much anything other than what they were, it would reduce their risk for heart attacks.

I'm not saying that coverity ISN'T responsible for better Open Source code, they might be, but I'm also saying that we can't really know they are without more information.