Entries by Josh Bressers

As most people have likely heard, Google Wave is going away. I never really understood Wave myself, but the story here is that it will vanish, and there is nothing you can do about it. Up until this whole explosion of "cloud computing", if a vendor killed a product or went out of business, you still had your product. It probably came in a box, with some floppies and a manual nobody read. You could keep running your application pretty much as long as your operating system let you. There would of course be various support issues to deal with, but if you're clever, you could get by.

The cloud changes this. Google has decided that Wave isn't the future, so it's going to go away. If you like Wave and think it's great, you're out of luck. You can't keep running it, Google says it goes, so it's gone. This is sort of scary. Welcome to vendor lock in 2.0.

This is probably going to happen to other things. I'm not sure which ones, but if you think things like Twitter, Facebook, and Gmail are never going to change, you are sadly mistaken.

This is where it's quite easy to add a positive note for the open source concept. One of the single biggest advantages to open source is getting to control your own destiny. You can help add features, fix bugs, keep features you like. Even if all you ever do is run the application, you don't have to worry about it going away. If you have the source code, you (or some other clever person) can make it run.

But it's the future, and all cloudy and magic, open source doesn't work with this new paradigm!
If you have a hard time "thinking outside the box" where I really mean, wet paper bag, then this is might be a true statement. When you're dealing with open source in a services like atmosphere, you get an added goal of playing nicely with others, and getting to control your data. Identi.ca is a good example of Twitter done right. Look at how XMPP is designed; it's a real distributed instant messaging platform. Unfortunately these services have a huge uphill fight as the money is in keeping users trapped on one service. Many people have never even heard of these things.

Consider things like email and the web. They still exist because they are public standards anyone can use to communicate with anyone else. There is no vendor control. The power is in the two endpoints, not the machines in the middle. Anytime you try to make the middle bits more important than the endpoints, you eventually fail. It can take a while, but it will always happen.

The problem with Wave was that the emphasis was put on Wave, not the users. Wave was really just the string between the tin cans, except it's not my string, it was painted bright orange, and the owner came to take it back. Now all I have are two tin cans. Luckily there's plenty of string.

It seems some folks are indeed storing the body scan images everyone said would never be stored:

Feds admit storing checkpoint body scan images

This probably shouldn't surprise many people. The issue isn't so much that these guys are trying to be evil, but are protecting themselves. Let's say a bad guy gets through the machine. If there is no record of the scan, you have an instant scapegoat; the security screener clearly didn't do their job! If you've saved it though, you can point at it and say "Look, the scan was fine, not my fault!"

People generally don't like to be blamed for anything. When given the choice between what is right, and what could keep them of trouble, most people will opt to stay out of trouble. It's part of our monkey brains at work, we don't like to be in trouble.

Storing these images will probably never change, as it's really easy to convince most people we need these. When someone says "it makes us safer", it's hard to create an argument a normal person will understand. Normal people can relate to wanting to stop bad people from doing bad things (especially to themselves). What they can't relate to is how the people in charge can systematically remove freedoms over a very long period of time, slowly, so most don't notice. Sadly the closest thing to an argument most people grasp about these machines is that they don't want pictures of their naked bodies stored.

The breakfast was quite welcome this morning. I needed a cup of coffee, I woke up rather tired. I never sleep well when I'm away from home.

The morning keynotes were great. I love hearing our message, Jim pointed out that most companies sell features, we sell a subscription. This makes me ponder the normal computer product cycle, which falls in line with The Innovator's Dilemma. You have companies that are really only adding new features, and not usually removing old features, as they're selling a box. Disruptive technology bubbles up from the bottom that's not encumbered by 20 years of "needed" features. With open source, I wonder how or if there is a disruptive force that can come in behind it, that isn't just a new open source project.

I also had the joy of working in the Red Hat booth this afternoon. It's always fun to talk to new people and learn about new things. People also seem to really like the Red Hat buttons we're handing out (or I'm just a really good button hander outer). It's almost frightening how many people I now remember the names of. I'm generally really bad at remembering names. Perhaps drinking the city water unfiltered has granted me some sort of super powers.

The evening event was pretty good. We had happy hour in the partner pavilion. I got a way cool USB hub from Enterprise DB. It looks like a children's toy. I love it. Now I just need to keep it away from the kids. The party upstairs was great. The food was good as was the company. I has some grilled swordfish. I won't complain about that. So far this is a pretty good Summit.

I give my talk tomorrow. Luckily I'm not staying up too late this evening, which should work out well as I speak at 10:20am.

So the first day of the Red Hat Summit is done. It's been a great day. I got to Boston around noon. I ate lunch at the Barking Crab, which turned out to be a great choice. The oyster Po' Boy sandwich was awesome. I spent a few hours wandering around Boston. I love tourist traps, they're so much fun.

The Summit itself always starts out with a reception. You get to walk around, see the booths and talk to various folks. It's always nice to see people I've not seen for a while (usually about once a year at the summit). The turnout seems good. There are a ton of people here.

The talks start tomorrow, there will be plenty more to say then.

Wow, it's been a long time since I've updated this thing. Hopefully I'll be less busy in the future.

Bruce Schneier had a really interesting story:
Nobody Encrypts their Phone Calls

I'm not very surprised by this. Encrypting phone calls is hard (even if you know what you're doing), which I suspect puts folks into two buckets

1) People who don't understand their phone calls can and probably are being recorded
2) People smart enough to not use the phone

The really scary thing though is this quote:

In 2009, encryption was encountered during one state wiretap, but did not prevent officials from obtaining the plain text of the communications,

The question to think about now, is how did they get a transcript? (I suspect it was from a different remote listening device that wasn't a telephone)

A friend passed along this blog posting from Microsoft:

Microsoft’s Many Eyeballs and the Security Development Lifecycle

The article is full of half truths and assumptions, but my favorite bit is probably this:

A million monkeys banging on a million keyboards will eventually produce Twelfth Night. Mathematically, the many-eyeballs argument, and the million-monkeys argument are equivalent.

I shall applaud Microsoft for comparing Open Source developers to monkeys randomly banging on keyboards. They've compared us to lots of things in the past, I'm not sure if this is a step up, or just lateral, either way, I'm happy to claim my infinite monkey status. It would be grand if one of you creative types could come up with a clever logo for our new social status.

In all seriousness though, the article makes a number of claims, I'll try to cover the big ones here:
1) Code review makes software more secure
2) Many Eyeballs is not true
3) Nobody is auditing Open Source software
4) The Microsoft Security Development Lifecycle, or SDL, is swell

1) Code review makes software more secure
I don't think anyone can argue against this. The author then goes on to pick out a handful of quotes about how Open Source doesn't get bugs fixed. I'm not even sure how one can come to this conclusion, so let's look at the facts presented instead: None. So in conclusion ... wait, what? The thing that makes an article like this hard to accept without any actual meat, is that it's all out there. Open Source development happens in the public. You don't get to hide bugs under the rug, you can't pretend development is or isn't happening. If the author had interest in backing up this claim, he could have picked a major project and proved his point.

Rather than make some crap up here about how secure Open Source is, I'll defer the reader to this link:
http://www.redhat.com/security/data/metrics/

That's Red Hat's security metric data. If you don't believe it, you can figure it out for yourself by reviewing the open source development data. Debian has something similar here. If what the author claims to be true that Open Source doesn't get any code reviews, or bugs fixed, it would be in a seriously sorry state. Keep in mind that there are a number of very widely used applications out there, Firefox, Apache, the Linux Kernel, and JBoss to name a few. If these applications were in near the sorry state presented in the article, nobody would run them, but they're quite widely used.

I think reality wins this one.


2) Many Eyeballs is not true
3) Nobody is auditing Open Source software
These two ideas are similar in their nature. Are they true? Who knows. I'm certainly not an expert and I suspect without a rather expensive study, we'll never know for sure. Here is what I do know. Microsoft has a finite number of employees. The number of Open Source developers eclipses this by magnitudes. This of course doesn't mean that they're doing audits, or even fixing bugs, but the numbers can't be ignored. Even if only one or two percent of Open Source developers are reviewing code, it's still a huge number. Unfortunately doing that sort of work isn't glamorous, so the folks doing it get little credit or attention.

My example is the Fedora Bug Zappers. These are folks that look at bug reports from Fedora. They're not well known, nor do they get lots of credit, but the job they do is amazing. Let's do a thought experiment here to show how bug reports are like echo chambers: they get noticed. If you report a bug in Fedora, one of the Bug Zappers will find it and take a look. If it's a security flaw, they'll pass it along to the security minded folks. Right here we've gone from one person, filing one bug, has just alerted about ten people (probably more, but let's be conservative) to the bug. Us security types will take a look, and if it's real, pass it along to all the other interested Open Source distributions, and upstream. We're now easily over 100 people. All of those people will collaborate to some degree to develop a patch. Upstream will accept the patch, distributions will patch their packages, and the users will upgrade. Once all this is done, you're talking about many hundreds of people involved in this one bug report. This is where the idea of many eyeballs comes from. It's not about people just auditing code, it's about the community working together to improve the software.

The thing we need to be mindful of here, is that Eric Raymond may be wrong with Linus' Law. It's not so much the why as it is the what. It's easy to claim that many eyeballs isn't true, but one cannot discount the fact that Open Source development works, and it works well. Why it works so well is no doubt an issue that can be debated at length. Perhaps he real power is in our infinite supply of monkeys banging on keyboards

4) The Microsoft Security Development Lifecycle, or SDL, is swell
I agree with this 100%. It is swell, and it's a great idea. It would be wonderful if every Open Source project started doing this.

I would be interested in knowing how many flaw are stopped as a result of SDL. I've not seen any metrics or papers on the effectiveness of this program. I'm certain it's stopped some number of flaws. If someone knows of such data, please pass it along. I won't comment further, as my understanding of this isn't all that deep, but from what I gather it is a good idea, and I applaud Microsoft for implementing this.

In Conclusion
The original article I'm mostly disagreeing with here concludes with the usual old data that Microsoft releases fewer security advisories than Open Source does. This is of course a red herring meant to distract the reader. They've been caught multiple times only releasing one advisory for multiple flaws. With closed source, there isn't a good way to tell what's all getting fixed. In Open Source, we can't hide anything, it's all there. This keeps us honest. I could go on with this argument at length, but it's not really worth it. It's been picked apart every which way for years now, and I don't think anyone really cares anymore. At the end of the day, all that matters is keeping the end users safe. If some friendly competition helps do this, we all win.

So while it's not well known, at least outside of people I talk to often,

I'm a bit of a backup nut. In the past I've gotten a few of my jobs and consulting gigs simple because of my backup experience. I was having a chat with a friend the other day and it occurred to me that it wouldn't hurt to explain what I do for my personal data. Many people don't understand or don't care about personal backups.

The real problem with right now is that there is going to be a giant hole as far as history goes. It's easy to say that with things like archive.org and web sites and twitter and wikipedia and blogs and twitter and facebook and twitter and lots of other things, more data is saved than ever before. This is very true, but it's only a certain set of data. The REALLY cool data will vanish either when your computer disk crashes, or you die and your children throw out that Commodore 64.

I shall tell a clever story to express my point; When I was in college, I lived in a really crappy fraternity house. It was over 100 years old and something was always broken. I often was fixing holes in walls, adding insulation, or whatever else needed some work. I made the unfortunate mistake of showing competence, which meant I was always volunteered to do the actual work while everyone else sat around drinking beer and watching TV. Almost every time we opened up a wall, there was all sorts of goodies to be found. From old newspapers, to things lost in the holes 10 years ago, to tools I left in the last time I fixed the wall.

When history is based on physical things, pictures, paper, wood, records, it doesn't vanish, it lives on as long as the medium does. Let's now think about digital pictures. When your kids have to clean out the basement after shipping you off the old folks home someday, they won't find boxes of pictures and old records, they'll find a hard drive in a box that probably can't be read by the futuristic mind control computers our mechanical overlords force us to use. They probably also won't care enough to see what's on it. It's not like finding a box full of pictures. It's easy to open the box and look at the pictures. It's not easy to figure out what's on some random computer or hard drive.

So this brings up to the idea of backups.

The first thing you need to do is make sure you have your data organized. This is very important. Put pictures in a folder called "pictures", music in "music" (there is a pattern here), and so on. This isn't only for other people, it's for you too. In five years, you're not going to know what that "funny stuff" folder is for.

Have a central fileserver for your house. Even if it's just one of the desktop computers. Make sure your data all lives in the same place, this will be important for backing it up.

Once you have your data collected and sorted, there are a couple of options. I do both of these.

1) Use external hard drives (plural)
2) Use a remote storage service like Amazon S3

As for #1, jwz has a nice write-up of what to do here: http://jwz.livejournal.com/801607.html.

#2 is a bit more complicated.

I've spent a fair amount of time looking into options for using a remote storage solution. None are ideal, I wrote my own called s4ync. It sucks too, but at least I know what's wrong with it.

The biggest advantage to using an online storage solution, is you don't need to go fetch a disk if something goes wrong. I also sleep better at night knowing that to lose my important data, my house, my bank, and amazon.com would have to all explode. The odds of me surviving such an event are fairly low. It's all about risk. I'm not a risky person, so less risk equals more sleep. Your mileage will vary.

Now the most important part of this: LEAVE INSTRUCTIONS. I don't mean some crappy note that says "This is the data backup" I mean write as much as you can about the data you have stored. I have rather detailed instructions for my family in the event of my untimely demise that goes as far as how to revoke my PGP keys. Be sure to either have a printed piece of paper or a file named something clever like README. The two likely scenarios here, are that your family is looking at this after you're gone, or they're cleaning up all the crap you've collected over the course of your life now that Open Source finally drove you to the looney bin.

In the event it's a situation where the grand-kids find the "box" of old pictures, you will have done the world a great service.

Last week there was a meeting in the US where our top intelligence official testified to lawmakers
Senators Warned of Terror Attack on U.S. by July
It contains this interesting bit.

At Tuesday’s hearing, Senator Dianne Feinstein, Democrat of California and chairwoman of the Senate Intelligence Committee, asked Mr. Blair to assess the possibility of an attempted attack in the United States in the next three to six months.

He replied, “The priority is certain, I would say” — a response that was reaffirmed by the top officials of the C.I.A. and the F.B.I.

I'm not entirely sure what that means, I doubt anyone really knows, the comment was probably meant to be ambiguous. Such a question is probably nonsense, but anyone looking to keep their job is going to tell you it's almost a certainty. If someone says anything else, then there is a terrorist attack, they're going to be blamed for whatever breakdown in intelligence happened and have to find a new job.

One of the unfortunate parts of being accountable for security is having to answer questions. It's easier to say an attack is imminent, then get praised for "preventing" it, than to say probably not and having to answer for the failure. All aspects of security work like this unfortunately. As long as everything is going great, nobody cares, then when something bad happens, it's all your fault. Intelligence people keep themselves known by constantly stating that something bad is about to happen any day now.

The right question to ask any security person isn't to predict the future, rather "What are you doing to minimize our risk?"

There is a saying about the old Berlin Wall, "Nobody ever escaped the same way twice." Security is no different. Rather than try to guess what the next attack will be, you should have all around good practices that not only help prevent attackers from targeting you, but then minimize damage when they do attack. We can easily guard against the past, but predicting the future is impossible.

Bruce Schneier has an interesting take on Security and Function Creep.

He raises a lot of great points about security systems taking on new tasks they weren't originally designed for. This is quite natural in the evolution of things. Not all evolution make sense. The six armed monkey never caught on, as poop flinging wasn't a historically desirable trait until the creation of the modern zoo.

I can't help but wonder about how Open Source fits into this though. Bruce has a quote in his article

... and the same operating systems that run our businesses are suitable for military uses.

SELinux makes me wonder about this. SELinux was born from the NSA, a government group which does nothing without security in mind (or at least I hope so). I will agree that most operating systems aren't suitable for military use, but something like Open Source can change the game. If the square peg doesn't fit in the round hole, what happens if it can be turned into a round peg? Open Source can allow this to happen. The group actually running the software can make changes as needed so the software fits their task at hand.

More thought and discussion needs to go into this idea, but Open Source is very powerful, more powerful than most of can even imagine. The old saying "all of us are smarter than one of us", while corny, is very true. Security folks are generally bad UI designers, and UI designers are generally bad at security. Open Source can make it easy for these two groups to work together and accomplish great things.

It seems that a number of Twitter accounts have been compromised:

Twitter resets passwords after phishing attack

The article suggests that many of the accounts in question may be from phishing or from a third party torrent site. This is a fine opportunity to talk about password security. I wouldn't be surprised at all if a fair number of accounts were compromised because of reused passwords.

There are some people who like to complain about password tracking tools like Password Safe and pieces of paper, but in all honesty, they work better than most brains. I would guess the average person can't remember more than two or three passwords at a time, and they're probably not very good ones at that. One of them is likely their ATM PIN of 1234. If you're some sort of super genius who can remember hundreds of passwords, and you read this blog, I don't believe you. Quite often the concept of perfect can interfere with our ability to get things done. In most instances, a perfect solution is unattainable, where good enough is possible and is better than it was previously.

Attacks like this have happened more than once, I've had it happen to me. I used to use a throw away password for all my public mailman accounts (this was before I realized that mailman will randomly assign me a password, as I never actually need it). That password was then later used to attempt to gain entry to private archives to a list I'm on. I didn't use that password there, but this made me understand that it was time to get serious about my passwords. I now use a tool called pwsafe, which uses the Password Safe database format for storing passwords. I of course don't use this for any REALLY important passwords (I keep those in my brain, and they're not near as impressive as the pwsafe passwords).

If you're like most people, and use a couple of passwords everywhere, please stop doing that. Find a good password generating tool, and either use a piece of paper or something like password safe to store them. The other big advantage to using not your brain to store passwords, is that it's much easier to change them. How many of you have been using the same password for five years, because it's too annoying to think up a new good password? Lots of us do that, it's hard to change.

Remember, good enough is the goal, not perfect.

There has been a lot of stories lately about the famous Google Attack. It's now becoming known that the flaw used was in IE, was reported in September, and was going to be fixed in February.

It's always tricky for vendors to juggle security flaws, but there are always two very important things to keep in mind. The first being that if someone reported the flaw to you, it's not an internal only secret, people generally suck at keeping secrets. It's very likely they have or will tell someone else.

The second important thing is that it's probable someone else found it at the same time.There have been numerous documented instances thorough history, where an important discovery is made by multiple people at the same time. It's not uncommon for the same security flaw to be found by two different people. With six billion people on the planet, there is plenty of room for overlap.

The real lesson here is that if you know about a critical security flaw, don't sit on it, fix it ASAP, even if you think you have plenty of time.

Dear Internets,

Is there a Fedora virtualization liveCD of sorts in existance? I can't find one. Here's what I'm thinking.

Right now I have a virt machine under my desk, I would love to have it just boot off a usb stick or CD and fire up all the virtual machines that live on it. This would make my life quite a bit easier, as instead of having to worry about keeping the Host OS in order, all I have to do is power down, swap USB drives, power on and I'm running the latest and greatest virtualization goodness.

I found this great explanation of how vendors should respond to security flaws:
Matt's Guide to Vendor Response

If you're a vendor, where vendor is also an Open Source project, it would be worth your time to read this. It has some great tips and is a quick read.

Ten years later and still funny!

So 2009 is pretty much done. Nothing truly amazing sticks out in my mind. That means it was either so bad I've blocked all traces out, or nothing overly exciting really happened.

I'm pretty sure it's the latter.

As it's my duty as an Internet citizen to make up crap that I can't possibly know will come true, I predict 2010 will be just like 2009, but with more disaster movies coming out.

I think the universe of security is getting dull. This is a good thing though, as it means the good guys are doing their jobs. There are always going to be things like botnets and evildoers looking to take advantage of the unsuspecting, but this is no different than in the physical world. There are bad guys, they exist because there's money to be made from exploiting others. I imagine thief is the second oldest profession.

The real stories of security are the few number of things like worms and wide scale defacements of the days of old. Most admins understand that updates must be applied promptly, and many vendors are now releasing those updates ASAP. There are technologies that can help make exploits very hard to write.

The future of exciting security research will probably move to virtualization; it's currently a lot like Swiss cheese in terms of keeping things secure. Unfortunately security is generally a reactive thing, so until there are problems found, I don't expect much proactive work done in virtualized security.