Kernighan’s history of Unix is a first person account of how the operating system came to be, and how it came to be so ubiquitous. It’s also a great overview of Bell Labs culture, and why it was so productive. The book is interesting, readable, and informative. A fairly quick read that’s still full of useful knowledge.
Each chapter is a part of the Unix development history, followed by a short biography of someone involved. This worked really well for me. The history part of the chapter connected what I knew about the OS with people who worked on it and the design decisions they struggled with. The biography section of the chapter would then go into more detail about one of the people involved in those design decisions, often showing some interesting background about why they would have had the thoughts they did.
There are a lot of anecdotes about how and why things were built the way they were. Some of these come off as simple reminiscences (and the book is part memoir). Other anecdotes really clarify the technical reasoning behind some of Unix’s features. I’m used to thinking of Unix through the lens of Linux, which is (currently) enormous and complicated. That’s pretty different than Unix during it’s development, when the source code could be printed in a single textbook alongside with line-by-line annotation. The book gave me the impression that Unix was created by some highly educated people that were just seeing what they could do, while also trying to make their own jobs easier. Each step of Unix’s creation was a small improvement over what someone was already doing, followed by a bit of cleaning up to make it all work nicely together.
A particular string of anecdotes really changed my understanding of the OS and it’s associated command line tools. When I use command line utilities like grep or sed or Make, it’s often in a cookbook format to accomplish a very constrained goal. I viewed them as tools that have specific configurations I can tweak, like I drill that I can change the torque on. That’s pretty distinct from how a lot of the Unix authors viewed things. They were very much working from the idea of languages. For Kernighan, sed and grep aren’t tools. They’re parsers for a general language. Make isn’t a configuration file, it’s a language that gets parsed. This is one reason that they were able to make so many powerful tools: they treated every problem like a translation problem, and wrote languages that would help. This change in mentality from a “tool” to a “language” feels large to me.
In addition to gaining insight into the decision process that drove Unix development, the book also spends a fair amount of time on the culture of Bell Labs. Bell Labs has a kind of mythic status, appearing in many history of technology books (including at least one that focuses on Bell Labs specifically). This was the best book I’ve read that described the organization of Bell Labs from the perspective of someone who was there.
Bell Labs Funded People
Bell Labs was a great example of the “fund people” method of research, as opposed to pretty much everything today. This idea, which is also discussed explicitly in the book, is that the modern scientific enterprise is too focused on giving money only to people with concrete problems they’re trying to solve. By only giving funding to short term concrete projects, our society is limiting the process of discovery and improvement.
The Bell Labs approach to funding research was:
step 1) hire people at the cutting edge of their field
step 2) there is no step 2. If you’re doing something here, you’re doing it wrong.
Kernighan describes how he started working at Bell Labs, and was never given a project. He just kind of wandered around the halls talking to people until he found some interesting things to work on. Everybody at Bell Labs was like this. Nobody had an official project, they just did whatever they felt like and published or patented what made sense after the fact.
Some people sometimes had concrete projects. Apparently when Ken Thompson started at Bell Labs, he was asked to help write the Bell Labs component of Minix. Minix was an operating system project being run out of MIT that had very lofty goals and a very complicated design philosophy. Bell Labs eventually pulled out of the project for reasons that aren’t clearly explained in the book. Something something it wasn’t working out, it’s me not you.
After Thompson’s Minix project got scrapped, he just worked on whatever he felt like. He made an early space-explorer video game, he worked on a hard disk driver. Then he realized he was “three weeks from an operating system” if he reused some of the software he’d just written. So he made Unix. No grant process, no management review, no scheduling it around other obligations.
After that, several other people in Thompson’s department got interested in it. They all worked together to add tools and features and clean it up. None of them had to clear their work with management or get a new job-code put on their timecards. They just worked on the software.
That’s not to say that they were just playing around. Kernighan describes a lot of specific concrete problems that they were trying to solve. Things like document preparation, automated legal brief generation, and resource sharing. Everyone who worked on those problems did so in a way that was generalizable, so they were adding tools and techniques to Unix as they worked. Ken Thomson and Dennis Ritchie, the two main Unix inventors, were at the cutting edge of operating systems and language design, and they just kept trying new things and increasing what was possible.
There are a few specific takeaways here that are relevant to attempts to fix modern science funding.
Who do you fund?
Kernighan has several pages on how Bell Labs selected people to hire. It seems that Bell Labs would have certain engineers build relationships with specific universities. Kernighan himself worked with CMU for over a decade. These Bell engineers would visit their university several times a year to talk to professors and meet their grad students. Promising PhD candidates would go out and interview with Bell Labs engineers in a process that reminded me a lot of the Amazon and Google in-depth interviews I’ve done.
Teams
The secret of Bell Labs’ success, at least according to Kernighan, lies in the network effects of their office. They hired all of these people who were at the cutting edge of their field, and then put them all on a single campus. There were no remote workers.
This really kills my plan to get myself fully funded and then do math research in my underwear from home.
The Bell Labs network offered two things. The first was obviously access to solutions. If you didn’t know how to do something, corporate culture demanded that you go ask the leading expert two doors down from you. The culture was a “doors open” policy, and people would often drop whatever they were doing when someone came to them with questions.
The other benefit of the Bell Labs network is less obvious: it’s the problems. All of the people at the cutting edge of their fields were trying to cut a little farther. They were all trying to solve their fields problems. That meant that everyone had interesting problems to work on. This ended up driving a lot of Unix development. Someone would show up in Kernighan’s office needing help with some random thing, and Bell Labs would end up with another ground-breaking piece of software several months later.
It’s possible that the internet could replace some of this. Stack exchange and Quora, in particular, seem ripe for this kind of experts-helping-experts exchange. That said, I think the in-person nature of Bell Labs lowered barriers to collaboration.
There was also the knowledge that anyone you helped was an expert in their own field, which is not the case with online tools now. I think that would change the dynamic significantly.
Bringing back the greatness
If you wanted to bring back the greatness of Bell Labs in its heyday, you’d fund people instead of projects. But that slogan admits a reading that’s fairly isolationist. I don’t think you could match Bell Labs’ output by funding a bunch of individuals around the country. I don’t even think you could match it by funding a bunch of small teams around the country. I think you need the big research center energy. Fund people, but make them work near the other people you’re funding.
Jonathan Edwards has an interesting post about stagnation in software development. He argues that the Great Stagnation in economic development is also impacting software, and his evidence is that we’re still using a lot of the tools that are discussed in Kernighan’s history. Unix and C/C++ are still used all over the place. Edwards would like to see better operating systems, better languages, better filesystems, better everything. He argues we don’t have better everything because “we’ve lost the will to improve” since the internet lets software developers make easy money.
Is Edwards right? Will it be useless to Fund People, Not Projects because developers have lost their way? Edwards want another Unix project, another breakout operating system that fixes all of Unix’s mistakes. Kernighan doesn’t think it’ll happen.
The thing is, OS development now is not even in the same reference class as OS development when Unix was invented. In the 60s, all operating systems sucked. The people using them were all experts by necessity, and they were always ready to jump to the next big thing in case it was more usable. By contrast, operating systems now are adequate and used by the non-technical public. People may whine a lot about their OS, but they mostly do what people need them to do. Any newer/fancier OS would require people to learn a new skillset without offering them many benefits. No Unix replacement can spread the way that Unix did.
Unlike Edwards, Kernighan isn’t disheartened. Kernighan just thinks that the next major developments will be in different fields. And indeed, we have seen major improvements in things like machine learning over the past decades. Machine learning is in exactly the reference class of early Unix: it currently sucks, everyone who uses it is an expert, and they’re all ready to jump to the next thing in case it solves their problems better. ML is exactly the kind of place we’d expect to see another breakout hit like Unix, and it’s also an area that Edwards explicitly says doesn’t count.
Software developers are people. They enjoy solving problems, but want to solve problems that actually matter. I’m sure Edwards has a long list of improvements that can be made to any particular human programming software paradigm, but in our current place in the timeline those improvements have low expected value relative to other areas.
Software developers have not lost the will to improve. That will is still as strong as ever, but it’s pointed in a slightly different direction now. Edwards may not like that direction, but I think it’s a direction that will have a better impact on everyone. Now we just need to get all the bureaucracy out of the way, so the developers can actually push forward the state of the art.