This post is a rough log for our asynchronous members to view at a glance, rather than an elaborate or high-fidelity capture of the discussion. It covers two sessions.
Having finished Stefano Franchi’s paper on Ashby’s homeostat and its commentaries ([Notes from sessions 4 and 5 link]), the Cognition Futures Reading Group turned on Thursday to Jonathan Richens and Tom Everitt’s “Robust agents learn causal world models”, presented at ICLR 2024. Friday was the group’s first session given over to planning its technical work: what the Friday sessions are for, which tools to use, and a first shared exercise.
What was read
Richens and Everitt ask whether agents must learn causal models in order to generalize to new domains, or whether other inductive biases are enough. Their answer is a necessity result. An agent that keeps its regret within a bound across a large set of distributional shifts must have learned an approximate causal model of the process that generates its data, and the approximation becomes exact as the agent approaches optimal. The shifts are modeled as local interventions on the environment’s causal variables. That causal models are sufficient for robust adaptation was already known; the paper argues they are also necessary.
In the discussion section, the authors revisit the good regulator theorem. It is often read as saying that any good controller must contain a model of the system it controls. The authors point out that the theorem itself only guarantees an optimal regulator that is a deterministic function of the system’s state, which could be trivial. They present their own result as a sharper, causal version of the theorem. Figure 4 places the result in Pearl’s causal hierarchy: optimal policies under domain shifts, for a single objective, already cover the interventional level, so they are also enough for shifts in the task.
The paper is explicit about its limits. The main theorems require robustness to shifts on all of the environment’s variables, and they cover only decision tasks in which the agent’s actions do not change the environment. Its discussion also gives AlphaFold as an example of competence without understanding: accurate prediction of protein folding that did not improve understanding of the underlying biochemistry.
The group read the abstract, the good regulator paragraph, Figure 4 and the conclusion together. Several members found the formal sections hard going, and the group did not take as deep a pass as it did with Franchi.
What was discussed
Modeling concepts rather than life
Amanda Nelson opened Thursday with what she took from the end of Franchi’s paper for the group’s own building work. Modeling life is a grand project, she said, and it would first require deciding what constitutes life. Framing a build as the modeling of a concept, explored openly and with no claim to reproduce a living thing, seemed to her more fruitful.
Can a priori structure do technical work?
Avery Lim described a conversation with a computer scientist who also studies philosophy. Lim had suggested that his generalized domain primitives might help structure reasoning in language models. The objection was that structures reasoned out in advance can be useful to people as an interface, but cannot capture technical structure, which has to be discovered by statistical exploration. Lim called this the status quo view in computer science. His interlocutor traced it through Wittgenstein and Turing, whom he counts among the roots of cybernetics.
Jes Parent identified the objection as the bitter lesson in other terms. Nelson asked whether building a model would itself count as enough use of a concept to ground it, in the sense of Wittgenstein’s language games. Lim said Wittgenstein can be read in more than one way, and proposed tracing the line from Wittgenstein through Turing and logical positivism to see how the divide between concept modeling and object modeling formed, and how it still shapes what researchers treat as viable.
Prediction, intervention, and the good regulator
Lim drew two threads from the reading. First, he read the paper as separating prediction from robust choice: a good predictor does not need a causal model, but choosing well across a comprehensive range of counterfactual alternatives does. He wondered whether this marks where a priori work earns its place, in organizing a space of causal possibilities and interventions rather than in making specific predictions, which answers part of the objection above. Second, he wanted to look harder at why a causal model is what separates this result from the original framing of the good regulator theorem.
Lim noted the paper’s requirement of a comprehensive family of interventions and its AlphaFold example. He also pointed to work, cited in the paper’s related work, suggesting that trained models can hold internal world models without being given that structure explicitly. He asked what it means for domain and task shifts to overlap with the observational level in Figure 4. Parent read the conclusion aloud: robust decision-making requires a causal model of the data-generating process, which the authors take as evidence that causal world models are a necessary ingredient of general intelligence.
Representation and isomorphism
Parent mentioned the Brain Inspired group’s discussion the day before. There, a good part of the conversation turned on how isomorphism is used: whether a given case is an isomorphism, a homomorphism or a functional isomorphism. Janet contributed comments from an engineering perspective, and Kevin Mitchell also took part. Lim added that the discussion raised whether representation is needed for, or implied by, results like this one, a recurring question in embodied cognition and cybernetics. This was raised briefly and not developed.
Parent also mentioned the AHA bonus seminar with Cathy O’Neil, which was taking place as the session ended, as something to come back to.
What should Fridays be for?
Near the end of Thursday, Nelson asked whether the group wants to keep moving between papers or to focus on a particular modeling project. She learns by building, she said, and suggested starting from models that have been discussed for decades, such as the homeostat or one of the most basic models of the neuron. Parent proposed starting with something very simple, then tying it to earlier Orthogonal Research and Education Lab work on developmental Braitenberg vehicles, and then moving into new territory. He asked everyone to spend some time before Friday thinking about a starter task. A perceptron was suggested as a default first target; on Friday, Lim credited the suggestion to Bradly Alicea.
For the next Thursday reading, Lim offered two papers on Wittgenstein’s place in the history of AI. Nelson found their AI framing less appealing and preferred Lydia Patton’s paper on Helmholtz, Wittgenstein and the a priori, which the group chose.
Goals for the technical sessions
On Friday, Morgan Hough asked what the technical sessions are for: understanding the code yourself, or being able to run simulations. Lim said he wants tools flexible enough for unconventional designs, computationally viable first and user friendly second. Longer term, he wants to model non-trivial adaptation to threat and need, social and environmental, starting from something simple that can be extended. Nelson said she wants to build from simple rules and be able to point to parts of the code and name the concepts they enact, such as a perceptron, a homeostat or feedback. She also wants the behavior being modeled to be describable. Parent described the two as a small, concept-level program and a longer-range one, with the same starting point.
Tools and scale
Lim had installed NetLogo and suggested following a tutorial together. Nelson leaned toward Python for its wider range of uses. Hough noted that for artificial-life experiments, how far a simulation can run in time is a matter of compute. He pointed to JAX-based packages that let Python code run in parallel on GPUs, and mentioned NEAT, EvoGym and JaxMARL. He described evolutionary robotics, in which an agent’s morphology and connectivity both change along with its policy, as a large, high-dimensional search space.
Related work and the problem of non-trivial models
Hough described a computational-life paper his reading group had covered, on the coevolution of sociality and replication. He raised work on modularity in neuroevolution from the mid-2010s, by Jeff Clune and Jean-Baptiste Mouret among others, as relevant to developmental Braitenberg vehicles. He said the work he finds most interesting is Logan Cross’s on theory of mind in agents: what models agents build of other agents. His own aim is multi-agent models with sociality and theory of mind that are relevant to computational psychiatry, and he noted that such agents also need tasks and organizing principles.
Lim said that when he has tried toy models, it was hard to get non-trivial behavior: things meant what they did only because he had set them up to, rather than through emergent interaction. Hough called this the balance problem in sociophysics, the difficulty of making statistical-mechanics models interpretable in social terms, and pointed to a 2025 review of the field.
A first exercise
The group settled on a show and tell for the next Friday. Each person brings a small model they have found or built: a homeostat, a perceptron, a thermostat, a Braitenberg vehicle or something similar. AI-assisted code is fine. Parent asked that the emphasis fall on describing the model: what it does, and whether it achieves what it is supposed to. Lim suggested a shared code repository with a space for each person and a shared notes document, and Parent set up a repository on the JOPRO GitHub for the purpose.
Key terms
Regret bound: a limit on how much worse an agent’s policy does than the best available policy.
Distributional shift: in Richens and Everitt, a change to the environment modeled as a local intervention on one or more of its causal variables.
Good regulator theorem: the result, often read as saying that every good regulator of a system must be a model of that system; Richens and Everitt argue its literal content is weaker.
Concept simulation: in Franchi, building a model to explore the consequences of an idea, as distinct from simulating an object.
Developmental Braitenberg vehicles: Braitenberg’s simple embodied agents, extended so that their nervous systems develop rather than arrive fully formed.
Open questions carried forward
Can structure reasoned out in advance do technical work, or only help people interpret? (Lim, Nelson)
Does the paper’s result require representation, or only imply it? (Lim)
What does it mean for domain and task shifts to sit where they do in Figure 4? (Lim)
How do you get non-trivial behavior from a toy model without building the meaning in? (Lim, Hough)
Next time
On Thursday the group reads Lydia Patton’s “Signs, toy models, and the a priori”; on Friday, the show and tell.
References
This session
Richens, J. & Everitt, T. (2024). Robust agents learn causal world models. International Conference on Learning Representations (ICLR 2024).
Patton, L. (2009). Signs, toy models, and the a priori: from Helmholtz to Wittgenstein. Studies in History and Philosophy of Science 40: 281–289. doi:10.1016/j.shpsa.2009.07.004
Background
Franchi, S. (2013). Homeostats for the 21st century? Simulating Ashby simulating the brain. Constructivist Foundations 9(1).
Dvoretskii, S., Gong, Z., Gupta, A., Parent, J. & Alicea, B. (2020). Braitenberg Vehicles as Developmental Neurosimulation. arXiv:2003.07689.
Further & Suggested Reading
These readings were not discussed on November 5 or 6. They were gathered after the sessions as starting points for the questions Nelson and Lim raised. The annotations are editorial.
Where should a simple first model start?
Stefan Dvoretskii, Ziyi Gong, Ankit Gupta, Jesse Parent and Bradly Alicea (2020), “Braitenberg Vehicles as Developmental Neurosimulation”. Argues that models which begin as fully formed networks ignore biological development, and proposes Braitenberg vehicles as a minimal embodied system in which a nervous system can develop. It presents three software approaches: a vehicle and genetic algorithm hybrid, multisensory Hebbian learning, and multi-agent models. This is the developmental Braitenberg vehicle work Parent proposed as the bridge from a first simple build.
Bradly Alicea, Rishabh Chakrabarty, Stefan Dvoretskii, Ziyi Gong, Akshara Gopi, Avery Lim and Jesse Parent, “Continual Developmental Neurosimulation Using Embodied Computational Agents”. Extends developmental Braitenberg vehicles into a design for learning agents whose development has distinct morphogenetic, critical and learning periods. It proposes growing the network with a genetic algorithm and treats development as a form of continual learning. Relevant to Lim’s aim of a simple model that can be extended toward non-trivial adaptation.