Why Complexity Theory Suggests We Should Prioritise Understanding the Black Box of Neural Networks

When I first learned about complexity theory, the part that stuck with me is how a complex system is more than the sum of its parts. James Glattfelder (2013) talks about this with the example of a termite mound. A single termite is insignificant; it is not going to achieve anything that we could step back and say adds value to its colony. Yet, an entire colony can create structures larger and more complex than we can imagine creating ourselves. The mounds are climate-controlled, and the entire colony acts almost like one organism to build them. There is no individual termite with the blueprint of the big picture, but together with all of its colony members, a big picture emerges, a phenomenon known as emergence. Glattfelder applied this thinking to the global economic system (Glattfelder, 2013). He and his team mapped the links of ownership around 43,060 transnational corporations and found that control is not spread evenly. It is highly disproportionate; a small core of 147 companies, mostly financial institutions, held nearly 40% of the control over the economic value of transnational corporations worldwide (Vitali et al., 2011). This system was not designed by one person, but it affects everyone, and it has a built-in risk: overreliance on a few nodes that, if they collapsed, would have an outsized impact on the whole system.

I also noticed how much effort goes into understanding these complex systems. Understanding termite colonies took many biologists many years, and the corporate network Glattfelder's team studied demanded similar time and effort. The problem is that we have been building a new kind of complex system over the last couple of decades, namely neural networks, and we are building them much faster than we can understand them.

Emergence applies to neural networks too (emergent here meaning ability greater than the sum of its parts, not sentience). An individual weight is nothing more than a number, and represents nothing that we can extract from it. However, billions of weights together can form a system like a large language model that can appear to reason and solve problems. The real power comes from the network itself, meaning the billions of interactions between parameters, rather than just an individual part. This is also what makes the models black boxes; we cannot look at a single weight and learn anything about why a model made a certain answer.

The concern I have about this is the incentives behind it. All these AI companies are racing to make the next most powerful black box, one that is stronger than the other black boxes, and they are rewarded by the performance of these systems, not by understanding them. There is research in interpreting the black boxes of neural networks, but it is nowhere near caught up with where we are in terms of neural network capability. The parallel to Glattfelder's network also goes beyond emergence: a handful of frontier models from a few companies now sit underneath a growing number of products and services, the same kind of concentration in a few nodes that he found in the global economy.

Complexity theory would suggest this is a risky position, because we are creating emergent systems that can behave in unpredictable ways. This can lead to failures at a systemic level, because we are building products and infrastructure on top of models we cannot explain, and much of it on the same few models. The best way to understand neural networks before we depend on them irrevocably is to make comprehension a priority, so we do not repeat the overdependence on a fragile system that Glattfelder found in the global economy.