One of my first part-time jobs during college was at a neighborhood PC store. You would walk in and order a new machine by selecting the spec you would like and I would assemble it from the components. Back in the golden days of Pentium II, Windows 98 and Red Hat 5.2. Long before Apple became the behemoth it currently is and the major switch to laptops, building machines from scratch was relatively common (at least around my neck of the woods). By the time I got this job I had already built several machines for myself or for friends so it wasn’t too hard of a gig. I just doing for profit what I was already doing for fun!
If you’ve never done it, building a desktop machine might seem like a daunting task, but it’s surprisingly simple. You buy a desktop box and this will serve as the frame from which all the other parts hang. Boxes (or “boxen” as the cool kids back then used to say) usually come with a power source, but depending on your needs you might want to get your own. The most important piece of the machine is the motherboard. This is a large (almost as large as the box) circuit board that screws directly to the metal box and to which all other electronic components connect to.
If you think of the motherboard as the nervous system of the machine, then the CPU is the brain power and the RAM is the short term memory and the hard drive the long term memory. Add a graphics card, plug in a mouse, a keyboard and a monitor and your ready to go for most daily needs. In practice you would also normally include a CD-ROM drive and a sound card and possibly a modem or network card.
The one thing that always stuck me as strange was how easy the whole process was. Screw in a few screws here, plug a couple of cables there, push in some DIMM memory boards over there and you’re all set. It was a lot like a high tech, more restricted version of the Lego sets I used to love as a child. There, even in the more advanced “Techno” version the pieces are interchangeable and if peg A fits in slit B, then that’s all you need to know. The same set of pieces can one day be a house, and a motorcycle or an airplane the next day. The limit of your imagination, and regardless of the combination you put the pieces in they always behave the same way.
In computer hardware, on the other hand, this is far from the case. Most things that can be plugged to each other are usually meant to, but not always. A lesson the hard way one day when I (somehow) plugged a VGA monitor to a sound card. The sight of a shiny yellow flame coming out the top of a black integrated circuit is still burned in my memory. Fortunately this little “creative” mishap didn’t cost me my job.
The other part of the job was to repair malfunctioning machines when customers brought them in. Hardware problems were (usually) the easiest. You would swap in the components one by one and when the problem disappeared you knew what was the guilty part. Software repairs on the other hand were a lot tricker. Some problems would only happen under specific circumstances: if MS Word is open, the sound card doesn’t work; when I update the sound card driver the OS detects 4 CD-ROMs and none of them work, etc.
If the hardware was akin to the Lego sets of my childhood, this was much different. Software that would work fine in one machine would come up with crazy errors in an (apparently) identical machine. The reason, of course, where the unforeseen interactions between the different pieces of software. Due to some infiltration glitch Word would take over the sound card and prevent the user from listening to their favorite CDs while working, or the CD-ROM and sound card driver would both try to acquire the same IRQ and mutually block each other.
At this point in my career I was already in college studying Physics but it wouldn’t be until a couple of years later that I would discover statistical mechanics and how even simple interactions between otherwise boring (or even identical) pieces of a system can completely change the observed behavior.
Consider a simple example. Let’s say I give you a long list of 1000 random numbers uniformly disoriented between zero and one and ask you what is the smallest number in the array. There’s no way to know exactly without checking every element one by one but your intuition should tell you that it’s likely some value close to zero. Now I make the game a little bit different and tell you to find the smallest number in the array and replace it by another random number between zero and one.
At this point you might be forgiven for assuming that not much has changed. I had 1000 random numbers before and I still do. I removed the smallest number but I replaced it with another random number. The trick here is that I didn’t replace any old number but a very special one, the smallest. And the smallest number in any list depends on all the other elements of the list, so I’m inadvertently introducing global correlations into the system. To make matters more interesting, the number I’m replacing it with is almost certainly larger (if the smallest number is 𝝐 then the probability of replacing it with a larger number is 1-𝝐) so I expect that over time the size of the smallest number will increase until all the numbers become as close to 1 as possible (the higher a value is, the safer it is from being replaced). But you don’t need to take my word for it, here is some simple python code illustrating this phenomenon:
And a figure showing the values at each position after a larger number of steps:
Now, we can even make things more interesting by explicitly including interactions. Let’s turn out list into a ring by connecting the two end and now instead of simply replacing the smallest number we replace it and it’s left and right neighbors. What do you think will happen now? Let’s make some quick changes to our code and find out:
And the complete figure now looks like this:
Where we can see that now the values are all squished in the range between ~0.6 and 1. How could this happen with such an apparently small modification? The reason is simple. When we replace not only the smallest number but also its neighbors, having a large (~1) value is no longer a guarantee of “survival” since you can get replace by having one of your neighbors be the minimum value.
My use of the word “survival” above is not accidental. Indeed, this version of the game was proposed in the 80s by Per Bak and Kim Sneppen as a very simple model of evolution. In their formulation, each element of the list represents the evolutionary fitness of a given species. In this case, Darwin’s “survival of the fittest” is turned into “replacement of the least fit”. They justified replacing the neighbors of the least fit as representing he fact that species don’t exist in isolation but rather are mutually dependent so that when you remove one species you directly impact the species it feeds on or which feed on it.
With this seminal paper, Bak and Sneppen gave an important advancement to the study of Self-Organized Criticality, or, in other words, the idea that even simple systems can fine tune themselves to “live” within just a small range of values and spawned one of the golden ages of he study of Complex Systems.
As luck would have it, when I became a PhD student at Emory University I joined the group of Stefan Boettcher, one of the early collaborators of Per Bak and that at the time was exploring the success of Extremal Optimization (a variant of the Bak-Sneppen model) in finding the ground state of Ising spin glass models. Together we built a cheap Linux cluster using off the shelf components and open source software. Each node would have just a box, motherboard, RAM and a network card card and would PXEBoot directly from the master node and register itself to receive jobs. The system was flexible enough that one could just plug in a new node and, after a few minutes, it would be part of the cluster. You can read an OLD write up about how it worked over at Linux.com. It just serves to show that you never know what experiences will come in handy in the future :)
The Bak-Sneppen model and Extremal Optimization, its intellectual descendent, with their emphasis on connections between components, would be the major responsible for moving me from the more classical Statistical Physics models towards Social Networks, Data Science and beyond, but that’s a story for another day. In the meantime, you can find the code to generate the figures above over on GitHub: https://github.com/bmtgoncalves/blog