From his home in Calgary, Theo de Raadt leads an international team in developing world-class software—and gives it away. Here, he talks with The Epoch Times about why he does it, about industry use of open-source software, and about dedication to quality paying off.
De Raadt’s team makes OpenBSD, an operating system, and OpenSSH, for secure communications.
Q: How did you become involved in writing free software?
I suppose even before going to university. A very long time ago, back in the days of the Commodore 64 and the Amiga, there was a culture of hobby programmers who would write things and then post them to BBSes, and then those things would propagate. I guess I started all the way back then writing little graphics programs and libraries and small video games. Later on, in the early Unix days, that turned into the Usenet group comp.sources.unix. And then around the time the Internet and web started becoming popular and common, we all just kept doing the same thing. Most of the people, like me, who do this thing have a culture of collaboration and ideas, rather than doing software for a job.
Q: The OpenBSD project has been strict in including only software that is both free as in “free of charge” and free as in “free of restrictions”—even free for commercial use. Why is the OpenBSD team so committed to giving away software that it allows others to use it for profit?
Well, we do not do this so that other players can make profit. We’ve actually been doing this for a long time and I do not know of anyone who specifically makes money off OpenBSD. They may, at best, save some money by not having to re-engineer the same software that we have already written. It is not exactly that we are letting them make a profit, but that we are doing a proper job and saving someone else from having to do the same job in a corporate setting. In our eyes, that is perhaps a waste of planet-wide engineer talents, rewriting the same thing over and over. Why can’t we just get it right once?
The result of our work is that:
– We end up having the code ourselves, in a free form;
– Vendors avoid writing it themselves;
– The user community helps us with bugs, instead of moaning to unresponsive vendors.
Q: How many commercial products incorporate OpenBSD/OpenSSH code?
We do not really track who uses OpenBSD code that much.
OpenSSH, though, is perhaps the best example of the reason why we write software that anyone can reuse for free. There are other SSH implementations—but none of them is as good as OpenSSH. How good is OpenSSH? Well, it is incorporated into every free Unix. It is in Cisco products. Hewlett-Packard switches. All vendor firewalls ship with it. It is on wireless base stations! It is everywhere. The only major operating system vendor not using it is Microsoft.
Q: To what extent have ‘for-profit’ users supported the project?
The vendors who incorporate OpenSSH have given us absolutely nothing back—not a cent.
However, getting money was not the goal for OpenSSH. Our goal was to convert the entire Unix community from using telnet and other insecure login protocols. And we have done that. Telnet is rarely used now, since all the Unix operating systems incorporate an SSH client and server. That is what students and system administrators now use to administer their machines.
A second benefit is that everyone is using our code base. Normally there would be a fear when we hit monocultures like this (everyone is running the same code), but in this case we believe we have been aggressively careful programmers. Would the companies have been as careful at writing clients and servers for the hugely complicated SSH protocol? Reading security mailing lists, we certainly don’t think so. If 50 vendors had each written their SSH code, we as a community would have a rather nasty problem on our hands. SSH would be a bad word.
Q: There is often a perception in industry that commercial software is a safer choice than open-source alternatives because formal support is available and problem ownership lies with the vendors. How valid is this perception, and what is the free software community doing about it?
Well, I suppose that there will always be people saying one thing or the other about this. If it was not open source versus the vendors, it would be people pitting vendor A against vendor B. Prior to getting involved in open source I had horrid experiences with vendors being completely unresponsive towards defect reports. I know that this is not just my view, that any person in a position of running an IT shop has had this problem. I think it is astounding that people could argue for “you just must trust someone else to fix it” instead of “you could fix it yourself, or hire someone to fix it.” There is a contractor base out there that can solve these problems as well as or better than the major vendors could. But I think the major vendors are still having more luck at getting the ear of the press.
Q: What would you say is industry’s biggest misunderstanding of free/open-source software?
Industry believes that software is complicated, and that big companies are better prepared to fix bugs. That is false. Bugs get fixed when someone cares enough to fix them. Caring about quality is not something that large companies have proven to be any more skilful at than, in some cases, 18-year-old university students.
Q: What is your management or coordination style? How do you bring out the best in the OpenBSD team?
Inside the group I am known as the ‘benevolent dictator.’ I will do whatever it takes to make sure that cautious but even progress happens to our source code. We believe in evolution of the software, not revolution; hence, changes are slowly moved in safely so that we do not make too many mistakes at all. Quality is a serious focus for me.
Over the years, therefore, we have attracted a lot of people who also view quality in the same way. Every year we hold a week-long development get-together, and this has allowed many of the developers to become friends also outside their direct development efforts.
Q: How do software patents fit in?
We are not making any profit off our work. We simply do this work as a hobby. The patent system does not concern me as much as it concerns others. Largely I suspect the patent system will be used by the large companies to eat other companies, typically smaller. In the end they will all be fighting over a smaller piece of the pie, especially if the free systems start to become commonplace. Or is patent law going to be twisted so that large companies can take money from a guy working at his computer, for his hobby, using his own brain, to write good stuff that he gives to people? That is not what the patent system was designed for. Nothing will ever prevent me from typing whatever my brain comes up with into a computer and giving it to someone else for free.
Q: Security has long been a priority with OpenBSD. How has your team approached this issue? Are other software teams employing the same strategies?
There’s a couple of teams starting to employ a variety of strategies. Security software is a very weird thing. You know, sometimes we have this sort of concept that we can solve problems with just one solution, and this doesn’t really apply to software, because whenever you try to use a single method to solve a problem, it either becomes too slow or it becomes too complicated—and when it becomes complicated, it ends up having security problems. For example, this is the sort of thing that has started to happen with Java. Java was designed to provide security, but now people are finding bugs in the Java virtual machine and in the compiler. The complexity of the software now is trying to provide for a low-level secure model. So, I’ve never been a really strong believer in the concept that you can actually make something more complex and therefore make it better. So, simple is better.
So, what we started early on… we started looking very, very closely at just auditing code—just go through the code and try to fix things. That was our first approach. And our second approach, in parallel, was to change the configuration of the system so that the services are not running by default. A person who wants to use a particular service has to actually go and enable it. And then we had a documentation side to this, which was, we documented things really well, so that you couldn’t enable the service very simply; you have to go read the documentation to learn how to enable it. So, when you read that documentation, you become aware of the security risks.
But auditing was a big aspect. So, we pretty much had 10-15 people working for about four years straight. It’s a 300-megabyte source tree. To put that into perspective, that’s… what Linux people have is a kernel, and they glom packages onto the side of it. But what we have is a whole, complete operating system. All of the userland stuff that’s basic, that you need—ls is part of OpenBSD, so are all of our libraries, so is Kerberos, so is OpenSSH, all our ftp daemons and web server daemons—all of the basic stuff. There are extended, additional things on the outside that are not part of our project directly, under our purview. But it’s a big chunk—it’s 300 megabytes of source code. If you look at all of Microsoft’s source code for all of Windows plus their office tools and all their little fancy applications, they’re dealing with about four times the size. So, basically we have about 80 people dealing with about one quarter the amount of software that the largest, most profitable software company in the world has. OpenBSD is also about the same size as Solaris.
So, we’ve got a large code base, and we started auditing it. The basic idea of our auditing process was that we’d find a bug, and we’d go fix that same bug everywhere. So the process isn’t ‘fix a bug and then go find the next type of bug’; it’s ‘find a classification of a bug, decide what it is, then go fix it everywhere.’ And we classified bugs differently. You don’t say, “Oh, this is a buffer overflow of this type.” We’d say this is an idiomatic error made by a programmer who believes that the thing that he’s using works this way, but it doesn’t work that way, it works this other way. So, let’s go find every single case where the programmer misuses this application-programming interface and make them all perfect.
So, we did this throughout our source tree, with probably about 100 types of common programmer mistakes. Some of them don’t even have direct security impact, but we did it anyway, because there’s no way you can build security if you don’t have quality. So, the slogan I used to toss around a while ago was “Security is decided by quality.”
So, that’s where we started, but over time we actually have found a newer class of security problems called integer overflows. Let’s say, for example, that you have an integer and you multiply it by another integer. Well, if one of them is user-controllable, what the attacker’s going to do is provide you a very large number so that the multiplication actually overflows, the high bits of the result get thrown away, and the result is actually a small number. Programmers only really check the results of such computations; they don’t check beforehand. This actually ended up being a security hole about seven and a half years ago in OpenSSH, and it was really, really subtle. It was so subtle, that type of bug, that we realized that this was no longer an API-type of programmer mistake, but that people don’t actually understand the C programming language, or even basic arithmetic with restricted-size variables.
This integer overflow thing is really scary because as programmers, we really can’t spot them; we can read the code, and they’ll just go right over our heads. We can’t even spot them, even if we know what we’re doing. And there’s no way you can use tools to find these. Well, we have a tool—an old, old Unix tool called lint—that can spot these things. In fact, in most source code it finds hundreds and thousands of them. We always considered them as false positives, but they’re not false positives—they’re actually gigantic risks! We’ve just been ignoring them forever.
There is actually a simple tweak that solves most of these problems: if you take all of your signed variables and turn them into unsigned variables, by and large, in most places, you’ll make the code a lot safer. When you’re dealing with signed and unsigned variables mixed together, there are many risk factors, and when you move towards unsigned variables a whole bunch of risk factors disappear. It’s not like it’s now magically completely better, but a bunch of the more dangerous side effects go away.
So, all the auditing in the world can’t help us. We developed a new approach to bug fixing; we call it exploit mitigation. Microsoft has started doing it. Red Hat has started doing it, and some of the Linux people as well. The basic idea is to take all writeable address spaces in the machine and make them non-executable and all executable address spaces not writeable. We’ve put a random value in the stack frame of all the calling functions, in exactly the right place, so that you can’t overwrite the return address.
Microsoft has tried other things like this, but they put the thing in the wrong place, so it doesn’t work—it doesn’t work in all cases. And then they have this other thing for C++ called “structured exception handler,” or SEH, and what did people discover about two weeks after Microsoft published that they had this thing working? Overwriting the return address is really hard, and it’s actually easier to convert all the old exploits to SEH exploits. So now all the attackers attacking Microsoft software are doing SEH overflows instead of return address overflows. It’s completely astounding.
We do other stuff too. We move the address space around. We move libraries around randomly. The basic idea is that the exploit author tries to control the side effects of the bug to benefit him. So, if the side effects are in such an entropic environment that he cannot actually utilize them reliably, then we’ve basically made a bug become a bug with weird side effects, not a bug with controllable side effects. That’s what we do now.
The idea is that we started off believing that we could actually fix all the code. We still do that, but we also now have mechanisms that make it hard for a bug to actually be reality.
And are some of the other vendors doing this? Well, Microsoft is doing a couple of these things now. They have some memory address protection—only on machines that have NX, the AMD64 non-execute bit. And realistically, I would say that within three years that’s going to be about 1-2 percent of machines. That’s not a lot of machines. We actually had this running on old x86s, because we used a horrendous, gruesome hack… and we actually end up getting a big security benefit on PCs, which is the biggest place where you want to be able to do this, because that’s about 70-80 percent of the market.
Red Hat’s also doing this, I suppose, to a lesser scope, but once again they’re relying on the non-execute bit. They’re also using the same compiler technology that we are. The thing that we do is put a canary on the stack—it’s called “ProPolice.” It’s by a Japanese researcher; it’s really good stuff. It’s basically just a very, very complicated patch for gcc; unfortunately, gcc is moving so far ahead that the patch can’t keep up, so currently this ProPolice stuff is not running on the latest version of gcc—but we’re not moving ahead because we think that ProPolice is more important than some fancy optimizations.
Q: In the last several months, some companies—like IBM, for example—have opened up, to some extent, their patent portfolios to free software projects. What do you make of that?
I went to look through the IBM portfolio they gave out and found nothing in there of use at all. So I thought it was just a PR game.
Also, there are some really weird laws in the US. Some of these large vendors are trying to play games with the Patent Office. I think they’re doing it more for themselves than for the world.
Q: Could you elaborate on why the OpenBSD team is so committed to releasing its software free of charge and free of restriction?
The first thing to recognize about OpenBSD is that there are about 80 developers and we do OpenBSD for ourselves only. Lots of other people use OpenBSD, but we use it for ourselves. It’s just for ourselves—and that means I want OpenBSD to run on everything I’ve got. I want OpenBSD to work no matter what things come along in the future. This means that we have to have an outside community that will help us with supporting new devices and new technologies. We can’t be too ‘fringe.’ So that means we have to have a user community. But we have a user community only because it benefits us, ourselves.
Then along come all of our users, and it turns out that many of our users are much like us; they have the same needs as us. We don’t particularly care that they have the same needs, but it’s sort of cool.
The result is that we end up with additional support because of them. Some of them write device drivers, or some of them create pressure against vendors and then things become free. And in the end, the whole goal is that we can keep on running on stuff that is completely open so that we can see it so that we can check it.
In this day and age, there’s no one else doing what we’re doing. FreeBSD is not doing this; they’re incorporating binary device drivers all the way through their tree now when it’s more convenient. They’re saying convenience over freedom. The Linux people are now the kings of loadable device drivers, from all sorts of vendors. If OpenBSD stops doing this, I would basically bet that within five years from now there wouldn’t be anybody trying to write a completely free operating system because it would not be possible.
Four months ago it was impossible to do RAID management using completely free tools. Now we have it actually working on AMI RAID controllers. And once it works on AMI RAID controllers, we’re going to find another vendor, a small one, make them friends, and then we’re going to run on them. And then we’re going to have two options. And then we’re going to go after another leading vendor and get documentation from them. And eventually we’ll have enough components that we actually know how the technology that we use on a day-to-day basis actually works.
Q: How do you spend your time when you’re not working on OpenBSD?
I go hiking around the world a little bit. I like a pint or two once in a while, and I like my mountain bike. I’ve got a girlfriend. I spent the last three days basically carting four truckloads’ worth of compost and dirt to her garden. Just a regular life like everybody else, but OpenBSD is my job and OpenBSD is my hobby, and I’m very serious about it.