Josh's Blog

I am an Infinite Monkey

A friend passed along this blog posting from Microsoft:

Microsoft’s Many Eyeballs and the Security Development Lifecycle

The article is full of half truths and assumptions, but my favorite bit is probably this:

A million monkeys banging on a million keyboards will eventually produce Twelfth Night. Mathematically, the many-eyeballs argument, and the million-monkeys argument are equivalent.

I shall applaud Microsoft for comparing Open Source developers to monkeys randomly banging on keyboards. They've compared us to lots of things in the past, I'm not sure if this is a step up, or just lateral, either way, I'm happy to claim my infinite monkey status. It would be grand if one of you creative types could come up with a clever logo for our new social status.

In all seriousness though, the article makes a number of claims, I'll try to cover the big ones here:
1) Code review makes software more secure
2) Many Eyeballs is not true
3) Nobody is auditing Open Source software
4) The Microsoft Security Development Lifecycle, or SDL, is swell

1) Code review makes software more secure
I don't think anyone can argue against this. The author then goes on to pick out a handful of quotes about how Open Source doesn't get bugs fixed. I'm not even sure how one can come to this conclusion, so let's look at the facts presented instead: None. So in conclusion ... wait, what? The thing that makes an article like this hard to accept without any actual meat, is that it's all out there. Open Source development happens in the public. You don't get to hide bugs under the rug, you can't pretend development is or isn't happening. If the author had interest in backing up this claim, he could have picked a major project and proved his point.

Rather than make some crap up here about how secure Open Source is, I'll defer the reader to this link:
http://www.redhat.com/security/data/metrics/

That's Red Hat's security metric data. If you don't believe it, you can figure it out for yourself by reviewing the open source development data. Debian has something similar here. If what the author claims to be true that Open Source doesn't get any code reviews, or bugs fixed, it would be in a seriously sorry state. Keep in mind that there are a number of very widely used applications out there, Firefox, Apache, the Linux Kernel, and JBoss to name a few. If these applications were in near the sorry state presented in the article, nobody would run them, but they're quite widely used.

I think reality wins this one.

2) Many Eyeballs is not true
3) Nobody is auditing Open Source software
These two ideas are similar in their nature. Are they true? Who knows. I'm certainly not an expert and I suspect without a rather expensive study, we'll never know for sure. Here is what I do know. Microsoft has a finite number of employees. The number of Open Source developers eclipses this by magnitudes. This of course doesn't mean that they're doing audits, or even fixing bugs, but the numbers can't be ignored. Even if only one or two percent of Open Source developers are reviewing code, it's still a huge number. Unfortunately doing that sort of work isn't glamorous, so the folks doing it get little credit or attention.

My example is the Fedora Bug Zappers. These are folks that look at bug reports from Fedora. They're not well known, nor do they get lots of credit, but the job they do is amazing. Let's do a thought experiment here to show how bug reports are like echo chambers: they get noticed. If you report a bug in Fedora, one of the Bug Zappers will find it and take a look. If it's a security flaw, they'll pass it along to the security minded folks. Right here we've gone from one person, filing one bug, has just alerted about ten people (probably more, but let's be conservative) to the bug. Us security types will take a look, and if it's real, pass it along to all the other interested Open Source distributions, and upstream. We're now easily over 100 people. All of those people will collaborate to some degree to develop a patch. Upstream will accept the patch, distributions will patch their packages, and the users will upgrade. Once all this is done, you're talking about many hundreds of people involved in this one bug report. This is where the idea of many eyeballs comes from. It's not about people just auditing code, it's about the community working together to improve the software.

The thing we need to be mindful of here, is that Eric Raymond may be wrong with Linus' Law. It's not so much the why as it is the what. It's easy to claim that many eyeballs isn't true, but one cannot discount the fact that Open Source development works, and it works well. Why it works so well is no doubt an issue that can be debated at length. Perhaps he real power is in our infinite supply of monkeys banging on keyboards

4) The Microsoft Security Development Lifecycle, or SDL, is swell
I agree with this 100%. It is swell, and it's a great idea. It would be wonderful if every Open Source project started doing this.

I would be interested in knowing how many flaw are stopped as a result of SDL. I've not seen any metrics or papers on the effectiveness of this program. I'm certain it's stopped some number of flaws. If someone knows of such data, please pass it along. I won't comment further, as my understanding of this isn't all that deep, but from what I gather it is a good idea, and I applaud Microsoft for implementing this.

In Conclusion
The original article I'm mostly disagreeing with here concludes with the usual old data that Microsoft releases fewer security advisories than Open Source does. This is of course a red herring meant to distract the reader. They've been caught multiple times only releasing one advisory for multiple flaws. With closed source, there isn't a good way to tell what's all getting fixed. In Open Source, we can't hide anything, it's all there. This keeps us honest. I could go on with this argument at length, but it's not really worth it. It's been picked apart every which way for years now, and I don't think anyone really cares anymore. At the end of the day, all that matters is keeping the end users safe. If some friendly competition helps do this, we all win.

Backup your data

So while it's not well known, at least outside of people I talk to often,

I'm a bit of a backup nut. In the past I've gotten a few of my jobs and consulting gigs simple because of my backup experience. I was having a chat with a friend the other day and it occurred to me that it wouldn't hurt to explain what I do for my personal data. Many people don't understand or don't care about personal backups.

The real problem with right now is that there is going to be a giant hole as far as history goes. It's easy to say that with things like archive.org and web sites and twitter and wikipedia and blogs and twitter and facebook and twitter and lots of other things, more data is saved than ever before. This is very true, but it's only a certain set of data. The REALLY cool data will vanish either when your computer disk crashes, or you die and your children throw out that Commodore 64.

I shall tell a clever story to express my point; When I was in college, I lived in a really crappy fraternity house. It was over 100 years old and something was always broken. I often was fixing holes in walls, adding insulation, or whatever else needed some work. I made the unfortunate mistake of showing competence, which meant I was always volunteered to do the actual work while everyone else sat around drinking beer and watching TV. Almost every time we opened up a wall, there was all sorts of goodies to be found. From old newspapers, to things lost in the holes 10 years ago, to tools I left in the last time I fixed the wall.

When history is based on physical things, pictures, paper, wood, records, it doesn't vanish, it lives on as long as the medium does. Let's now think about digital pictures. When your kids have to clean out the basement after shipping you off the old folks home someday, they won't find boxes of pictures and old records, they'll find a hard drive in a box that probably can't be read by the futuristic mind control computers our mechanical overlords force us to use. They probably also won't care enough to see what's on it. It's not like finding a box full of pictures. It's easy to open the box and look at the pictures. It's not easy to figure out what's on some random computer or hard drive.

So this brings up to the idea of backups.

The first thing you need to do is make sure you have your data organized. This is very important. Put pictures in a folder called "pictures", music in "music" (there is a pattern here), and so on. This isn't only for other people, it's for you too. In five years, you're not going to know what that "funny stuff" folder is for.

Have a central fileserver for your house. Even if it's just one of the desktop computers. Make sure your data all lives in the same place, this will be important for backing it up.

Once you have your data collected and sorted, there are a couple of options. I do both of these.

1) Use external hard drives (plural)
2) Use a remote storage service like Amazon S3

As for #1, jwz has a nice write-up of what to do here: http://jwz.livejournal.com/801607.html.

#2 is a bit more complicated.

I've spent a fair amount of time looking into options for using a remote storage solution. None are ideal, I wrote my own called s4ync. It sucks too, but at least I know what's wrong with it.

The biggest advantage to using an online storage solution, is you don't need to go fetch a disk if something goes wrong. I also sleep better at night knowing that to lose my important data, my house, my bank, and amazon.com would have to all explode. The odds of me surviving such an event are fairly low. It's all about risk. I'm not a risky person, so less risk equals more sleep. Your mileage will vary.

Now the most important part of this: LEAVE INSTRUCTIONS. I don't mean some crappy note that says "This is the data backup" I mean write as much as you can about the data you have stored. I have rather detailed instructions for my family in the event of my untimely demise that goes as far as how to revoke my PGP keys. Be sure to either have a printed piece of paper or a file named something clever like README. The two likely scenarios here, are that your family is looking at this after you're gone, or they're cleaning up all the crap you've collected over the course of your life now that Open Source finally drove you to the looney bin.

In the event it's a situation where the grand-kids find the "box" of old pictures, you will have done the world a great service.

"The priority is certain" ... Wait, What?

Last week there was a meeting in the US where our top intelligence official testified to lawmakers
Senators Warned of Terror Attack on U.S. by July
It contains this interesting bit.

At Tuesday’s hearing, Senator Dianne Feinstein, Democrat of California and chairwoman of the Senate Intelligence Committee, asked Mr. Blair to assess the possibility of an attempted attack in the United States in the next three to six months.

He replied, “The priority is certain, I would say” — a response that was reaffirmed by the top officials of the C.I.A. and the F.B.I.

I'm not entirely sure what that means, I doubt anyone really knows, the comment was probably meant to be ambiguous. Such a question is probably nonsense, but anyone looking to keep their job is going to tell you it's almost a certainty. If someone says anything else, then there is a terrorist attack, they're going to be blamed for whatever breakdown in intelligence happened and have to find a new job.

One of the unfortunate parts of being accountable for security is having to answer questions. It's easier to say an attack is imminent, then get praised for "preventing" it, than to say probably not and having to answer for the failure. All aspects of security work like this unfortunately. As long as everything is going great, nobody cares, then when something bad happens, it's all your fault. Intelligence people keep themselves known by constantly stating that something bad is about to happen any day now.

The right question to ask any security person isn't to predict the future, rather "What are you doing to minimize our risk?"

There is a saying about the old Berlin Wall, "Nobody ever escaped the same way twice." Security is no different. Rather than try to guess what the next attack will be, you should have all around good practices that not only help prevent attackers from targeting you, but then minimize damage when they do attack. We can easily guard against the past, but predicting the future is impossible.

Bruce Schneier on Security and Function Creep

Bruce Schneier has an interesting take on Security and Function Creep.

He raises a lot of great points about security systems taking on new tasks they weren't originally designed for. This is quite natural in the evolution of things. Not all evolution make sense. The six armed monkey never caught on, as poop flinging wasn't a historically desirable trait until the creation of the modern zoo.

I can't help but wonder about how Open Source fits into this though. Bruce has a quote in his article

... and the same operating systems that run our businesses are suitable for military uses.

SELinux makes me wonder about this. SELinux was born from the NSA, a government group which does nothing without security in mind (or at least I hope so). I will agree that most operating systems aren't suitable for military use, but something like Open Source can change the game. If the square peg doesn't fit in the round hole, what happens if it can be turned into a round peg? Open Source can allow this to happen. The group actually running the software can make changes as needed so the software fits their task at hand.

More thought and discussion needs to go into this idea, but Open Source is very powerful, more powerful than most of can even imagine. The old saying "all of us are smarter than one of us", while corny, is very true. Security folks are generally bad UI designers, and UI designers are generally bad at security. Open Source can make it easy for these two groups to work together and accomplish great things.

Password Security

It seems that a number of Twitter accounts have been compromised:

Twitter resets passwords after phishing attack

The article suggests that many of the accounts in question may be from phishing or from a third party torrent site. This is a fine opportunity to talk about password security. I wouldn't be surprised at all if a fair number of accounts were compromised because of reused passwords.

There are some people who like to complain about password tracking tools like Password Safe and pieces of paper, but in all honesty, they work better than most brains. I would guess the average person can't remember more than two or three passwords at a time, and they're probably not very good ones at that. One of them is likely their ATM PIN of 1234. If you're some sort of super genius who can remember hundreds of passwords, and you read this blog, I don't believe you. Quite often the concept of perfect can interfere with our ability to get things done. In most instances, a perfect solution is unattainable, where good enough is possible and is better than it was previously.

Attacks like this have happened more than once, I've had it happen to me. I used to use a throw away password for all my public mailman accounts (this was before I realized that mailman will randomly assign me a password, as I never actually need it). That password was then later used to attempt to gain entry to private archives to a list I'm on. I didn't use that password there, but this made me understand that it was time to get serious about my passwords. I now use a tool called pwsafe, which uses the Password Safe database format for storing passwords. I of course don't use this for any REALLY important passwords (I keep those in my brain, and they're not near as impressive as the pwsafe passwords).

If you're like most people, and use a couple of passwords everywhere, please stop doing that. Find a good password generating tool, and either use a piece of paper or something like password safe to store them. The other big advantage to using not your brain to store passwords, is that it's much easier to change them. How many of you have been using the same password for five years, because it's too annoying to think up a new good password? Lots of us do that, it's hard to change.

Remember, good enough is the goal, not perfect.

Timing Security Flaws

There has been a lot of stories lately about the famous Google Attack. It's now becoming known that the flaw used was in IE, was reported in September, and was going to be fixed in February.

It's always tricky for vendors to juggle security flaws, but there are always two very important things to keep in mind. The first being that if someone reported the flaw to you, it's not an internal only secret, people generally suck at keeping secrets. It's very likely they have or will tell someone else.

The second important thing is that it's probable someone else found it at the same time.There have been numerous documented instances thorough history, where an important discovery is made by multiple people at the same time. It's not uncommon for the same security flaw to be found by two different people. With six billion people on the planet, there is plenty of room for overlap.

The real lesson here is that if you know about a critical security flaw, don't sit on it, fix it ASAP, even if you think you have plenty of time.

Virtualization liveCD Fedora spin?

Dear Internets,

Is there a Fedora virtualization liveCD of sorts in existance? I can't find one. Here's what I'm thinking.

Right now I have a virt machine under my desk, I would love to have it just boot off a usb stick or CD and fire up all the virtual machines that live on it. This would make my life quite a bit easier, as instead of having to worry about keeping the Host OS in order, all I have to do is power down, swap USB drives, power on and I'm running the latest and greatest virtualization goodness.

How to respond to security flaws

I found this great explanation of how vendors should respond to security flaws:
Matt's Guide to Vendor Response

If you're a vendor, where vendor is also an Open Source project, it would be worth your time to read this. It has some great tips and is a quick read.

Happy 19110

Ten years later and still funny!

Security in 2010

So 2009 is pretty much done. Nothing truly amazing sticks out in my mind. That means it was either so bad I've blocked all traces out, or nothing overly exciting really happened.

I'm pretty sure it's the latter.

As it's my duty as an Internet citizen to make up crap that I can't possibly know will come true, I predict 2010 will be just like 2009, but with more disaster movies coming out.

I think the universe of security is getting dull. This is a good thing though, as it means the good guys are doing their jobs. There are always going to be things like botnets and evildoers looking to take advantage of the unsuspecting, but this is no different than in the physical world. There are bad guys, they exist because there's money to be made from exploiting others. I imagine thief is the second oldest profession.

The real stories of security are the few number of things like worms and wide scale defacements of the days of old. Most admins understand that updates must be applied promptly, and many vendors are now releasing those updates ASAP. There are technologies that can help make exploits very hard to write.

The future of exciting security research will probably move to virtualization; it's currently a lot like Swiss cheese in terms of keeping things secure. Unfortunately security is generally a reactive thing, so until there are problems found, I don't expect much proactive work done in virtualized security.

Virtualization security

This is a great article about virtual server security.

It's always nice to see good articles that talk about how virtualization is not a security feature. As they often say, the first step is to admit you have a problem.

The best quote of the article being

... when I read about the take-up of virtualisation, the feeling of foreboding is not unlike seeing a five-year-old play with Daddy’s collection of Samurai swords – while nothing awful has happened yet, one can’t help thinking it’s a matter of when, not if.

Unreasonable Security Practices

I ran across this article quite some time ago:
4 Unreasonable Security Practices You're Probably Following

It brings up some good points, and mentions risk quite a bit. Risk is the real issue when dealing with any type of security. Nothing is ever 100% secure, security is like a logarithmic graph, where you can make a really big difference by taking a few easy steps in the beginning, but as things try to get more secure, it gets much much harder for minimal gain.

An easy to understand example would be a door into a secured bunker. Imagine it's just an opening into a wall with no door. This is obviously very insecure. Adding a door and a simple lock would bump the security up substantially, but to get it much more secure, you need to start adding rather difficult things that wouldn't provide significantly more security. Some additions could include biometric locks and an armed guard, but in all honestly, the increased security from those isn't even comparable to the increase in security just from adding a door and a lock.

USB Thunbdrive Awesomeness

I bought a 16 Gig USB drive for $20 yesterday at Frys. This itself is pretty cool, but I figured I'd try to install Fedora 12 onto it. I don't mean a USB livecd-iso-to-disk sort of install, but a real install, where anaconda treats it like a regular disk. I was amazed to see that it installed fine, but also boots perfectly.

I've tried this in the past, probably around Fedora 10, and it didn't work for me, so this is a pleasant surprise.

Virtualization Security

I was happy to find an article talking a bit about Hypervisor Security Concerns. It's not overly in-depth, but does point out the very real issue that virtualization poses some serious security concerns.

I think sometimes it's easy to get caught up in all the hype and forget about all the other issues a technology could create for us. I really like the title in the article

A New Definition of “Privilege Escalation”

Since it sounds cool, we should start calling a guest that breaks into the hypervisor, Hyper Escalation. It both sounds cool, and is fun to say. (I have bolded, underlined, and italicized it to stress its coolness)

In all seriousness though, this is a good discussion point, and an issue that will likely continue to come up for many years to come. Virtualization, like many things to come before it, wasn't built with security in mind, so it's going to take many many years to hammer out all the dents.

Security is people!

It's not often you read a security story that blames people for the problem. I found this one to be quite interesting.
Federal IT Face Cyber Attacks Daily

The best quote from the article is this:

Federal employees are still the main cause for security flaws because of their careless online activity along with failure to comply with organization policy, finds the research. It also finds that across civilian and defense organizations, 66% of those surveyed caught employees conducting irrelevant Web-surfing during 2008, while 44% found their workforce noted passwords on office stick-notes that could become public.

The hard part here is how do you get people to care? It's not much different in the real world, where people are rather careless about keeping belongings and personal information safe. It's even harder to convince someone to keep something intangible things safe.

The Internet of today is comparable to Fagin and his band of little pickpockets. The difference now, is the modern day Fagin doesn't use orphans, but zombie computers, and millions of them. Imagine trying to walk through a marketplace so full of thieving children you can't move. That's pretty much today's Internet.