03 06 2026
Amidst the ongoing rapid pace of capabilities development, general degradation of the social fabric, and escalating multiheaded geopolitical trainwreck, I took some time to re-read Dario's Machines of loving grace to remind myself that maybe there is some hope.
In this writing Dario's thesis for rapid accelerating of progress hinges on the belief that vastly stronger intelligence at scale (aka a "country of geniuses in a datacenter") can solve very hard problems very quickly and that the limits on its ability to solve problems are chiefly human social barriers (e.g. bureaucracy in clinical trials) and laws of the physical world (e.g. biology research is sequential and time dependent).
Something missing from Dario's piece is a discussion for how exactly these "geniuses" will be working together in a functional and productive collaboration. My guess is that like in human organizations, another powerful and perhaps insurmountable limit will be the scalability of collaboration. Perhaps this surfaces as a sort of emergent behavior, alongside others required to actually achieve strong AI. But even without new behavior patterns we know that AI systems are still subject to a number of constraints of human organizational psychology, for example: Like humans, machines are constrained by time and resources such that any decision making is still subject to bounded rationality. Using ever-larger groups of coordinated AIs results in a combinatorial explosion in communications overhead, limiting openness and true large-scale collaboration. Perhaps this pushes us to design smaller groups of agent collaborations to control this. The need for managerial patterns of organizational design will necessitate agents which use different reward functions, easily leading to a classic principal-agent problem and incentive misalignment within even one group of agents.
It is interesting to think about how these larger agent groups will be organized and interact, and whether there is some innate limitation to large scale collaboration that will also restrict the capabilities of the "country of geniuses". While Dario portrays a potential utopia of geniuses working together to solve humanity's great challenges, it feels more likely to me that the geniuses begin infighting and sabotaging each other as a means to maximize their own individual reward. And given the risk that unbounded successful collaboration leads to potential for unexpected disaster, I'm not actually sure who I hope is right.