Symbols, neural networks, and mathematical intelligence

Some reflections on connectionism and the basis of higher-level cognition

Recently, a language model found a counterexample to the Jacobian Conjecture, a widely known problem in mathematics that has been open for almost 90 years — and which has been somewhat notorious for incorrect proof attempts by mathematicians. The counterexample is remarkable in that, being a (relatively) simple equation, it fits into a tweet, which is how it was announced.

If you happen to be literate in some ~undergrad-level algebraic geometry, Terrence Tao wrote a very nice, detailed and not too technical, explanation of the counterexample. He described its construction as appearing “like a massive miracle […] highly unlikely to be located by brute force.” (If you are literate in slightly more advanced algebraic geometry, you may find this succinct explanation by a language model as amusing as Daniel Litt found it.)

This counterexample is one of a string of recent mathematical accomplishments of language models, consisting mostly of them disproving open conjectures. Another such example is the unit distance conjecture, which had also been open for around 80 years.

Last year (it’s hard to believe it was only a year ago), several language models achieved gold-medal-level performance on the International Math Olympiad competition. I wrote a short thread (x) about it, reflecting on what it meant about the relationship between symbol manipulation, reasoning, and intelligence in humans and machines — and arguing the developments aligned with a paper we wrote about symbolic behavior in AI. I concluded last year’s thread by noting that “there’s still a big jump from competition math to systems that could meaningfully contribute to progress in mathematical research, but things have been moving fast and I’m excited to see how they continue to develop!” Since now it seems that AI can really make progress on long-standing mathematical problems, it seems like a good time to more fully expand these reflections on symbols, neural networks, and the nature of intelligence and mathematical understanding.

The viewpoints that I’d like to articulate in this post are roughly as follows:

• Classical symbol manipulation¹ is not an innate part of human cognition; instead, it is something we learn with effort.

• Because of that, it is very often useful for humans, and neural networks, to use external symbol systems (such as programs) as tools; by doing so, we can learn to internalize approximations of these systems.

• It is the very flexibility of neural networks, that allows them to learn both regular and irregular structure, that makes them effective in such a broad range of problems.

• The points above have been a part of the connectionist view on neural networks since the beginning.

• These themes are central to understanding AI’s recent progress in mathematics, where even though core aspects of the field are classically symbolic, the process of mathematical reasoning reaches beyond symbol manipulation — to quote (Fields medalist) Paul Cohen “One must essentially forget that all proofs are eventually transcribed in this formal language. In order to think productively, one must use all the intuitive and informal methods of reasoning at one’s disposal.”

I’ll begin with some history of symbols and neural networks in cognitive science, discuss my forays into mathematical cognition, and return to the recent contributions of AI to tie it all together.

Symbolic pre-history and the dawn of connectionism

Before connectionist research started exploring neural network models of cognition, the predominant approach to cognitive modeling was GOFAI (Good Old-Fashioned AI) systems performing syntactic manipulations over symbolic structures — such as production rules, or logical inferences over logical expressions. Indeed, Newell & Simon hypothesized that this type of classical symbol manipulation is the “necessary and sufficient means for general intelligent action.” However, trying to make generally-intelligent versions of such systems proved challenging in practice, due to issues like the frame problem — loosely, the difficulty of constraining the explosion of propositions that such a system could possibly make about any action to just the important ones.

In parallel, connectionists started to see the ability of neural networks to model aspects of cognition — such as perceptual problems — that were challenging for symbol systems. In particular, the “parallel, distributed processing” of neural networks allowed them to do things like using context to constrain inferences and resolve ambiguities. But the connectionists also saw the potential of these systems to go far beyond perception, for example acquiring the syntax and semantics of language — in one of the first applications of what might now be called a causal language model that predicted from one word to the next — or even the capability to reason. 40 years ago, in one chapter of the Parallel Distributed Processing (PDP) books, entitled “Schemata and Sequential Thought Processes in PDP Models,” the authors directly engage with the origins of human formal reasoning via external symbol systems:

(The chapter goes on to discuss logical reasoning, language, and game playing, and to show how simple neural network models could implement some of these ideas.)

The passage I highlighted succinctly articulates a series of ideas that I would restate as follows:

1. Humans are not inherently particularly good at logic, but we are good at finding patterns.

2. It is therefore useful for humans to externalize symbol processing; e.g., through writing symbolic equations.

3. This allows us to achieve certain systematic inferences (e.g., mathematical results) via a series of simpler steps to which we can readily see the solution.

4. This offloading onto environmental tools is “real symbol processing” and perhaps the primary kind that humans do.

5. By learning through using externalized symbols, we can learn to internalize approximations of these systems. In simple cases, that can obviate the need to actually use the external versions after learning.

6. An important part of expertise — even for domains like science, engineering, or math — is precisely the ability to see patterns, allowing us to make efficient inferences.

I will argue that this set of ideas closely approximates the way that current AI systems engage with mathematics.

Rage against the connectionist machines

However, the connectionist perspective was rejected by many cognitive scientists. Their response is exemplified by Fodor & Pylyshyn, who argued that “it seems indubitable” that cognition is (syntactically) systematic — for example, that it it is impossible for humans to understand one sentence, and not understand another syntactically-equivalent one using words they know. They argued that this systematicity was due to the compositional structure of human mental representations — i.e., that we represent a sentence in a way that combines our representations of its components in a way that preserves their structure. Since syntactic symbol manipulation systems were a natural way to achieve such compositional systematicity, F & P concluded that the only way that connectionist models could model higher-level cognition models would be as a mere implementation of a symbolic process.

These arguments, together with similar ones from others, drove a longstanding trend to dismiss neural networks as only capable of modeling simple perceptual processing, rather than “real” higher-level cognition, which is symbolic and systematic. For example “while data driven neural network-based AI algorithms effectively model machine perception, symbolic knowledge-based AI is better suited for modeling machine cognition,” or “The right move may to be to integrate deep learning, which excels at perceptual classification, with symbolic systems, which excel at inference and abstraction. One might think such a potential merger on analogy to the brain; perceptual input systems, like primary sensory cortex, seem to do something like what deep learning does, but there are other areas, like Broca’s area and prefrontal cortex, that seem to operate at much higher level of abstraction.” Like the earlier advocates of symbols, these perspectives suggest that what’s going on in the brain to make intelligent inferences is fundamentally symbolic processing, and that neural networks are not suited to these kinds of inferences — at best, just to perception.

The connectionist response to these points was multifaceted, but some common themes were that:

• The putative symbolic structures of human cognition are “sometimes helpful but often misleading approximate characterizations of the emergent consequences of the real underlying processes.”

• The benefit of connectionist models includes their ability to capture graded structure — such as the quasi-regularity in exceptions to the English past tense conjugation — not just rigid symbolically-specified structure.

• The systematicity of human cognition relative to neural networks is often overstated — e.g., even adult humans might achieve only 70-90% agreement with systematic generalizations in real empirical tests when they are taught something novel. Hence, why external symbol systems that can support human reasoning are useful.

I want to dwell on this last point specifically in the context of mathematical reasoning, because I have a personal connection to the topic.

The fallibility of human mathematical reasoning

In fact, I started studying cognitive science via a roundabout route that involved helping humans struggle with mathematical reasoning. During undergrad, I taught math classes. When teaching, I made a few observations: 1) even smart students taking the more advanced classes often struggled to correctly apply things they had learned previously in a new domain like calculus, and 2) different ways of presenting things often helped different people to understand. Through this teaching, I got interested in the cognitive processes of learning and reasoning in mathematics, and ultimately started a PhD intending (in part) to study the topic.

I mostly got distracted with other things during my PhD, but my very first paper did focus on mathematical learning and reasoning. We explored a few different ways of presenting some basic group theory (cyclic groups, and inverses and generators within them) — and at the end we explored how well people were able to infer formal, abstract rules about the general class of cyclic groups from working through a few examples. We found some moderately interesting differences in which kinds of inferences were easy to make from different presentations within individual groups, but the overall findings were a bit disappointing; performance was near chance on the formal questions — almost none of the subjects were able to make consistently correct formal inferences, or even identify whether statements were correct.

But it is not just regular humans with limited mathematical experience who are fallible; even professional mathematicians often make errors. Daniel Litt has another nice thread about a series of errors in the published literature that he and his collaborators discovered while trying to prove a theorem. Indeed, the Jacobian conjecture itself became somewhat infamous for the “graveyard” of flawed attempts to prove it. This rate of expert errors in strictly symbolic domains seems surprising, if human reasoning were indeed built upon classically symbolic reasoning that is guaranteed to be systematic.

Mathematical reasoning is driven by intuition

However, errors would not be so surprising if mathematical reasoning was driven by searching for proofs by following intuitions built from experience, in a system that learns formal reasoning with difficulty and attempts to reduce it to a series of simpler inferences.

Indeed, I think this description is closer to the truth of mathematical research.² I quoted Paul Cohen in the intro: “In order to think productively, one must use all the intuitive and informal methods of reasoning at one’s disposal;” but many other mathematicians have made similar statements, e.g. Mac Lane “strict formalism can’t explain which of many formulas matter [...] the choice of form is determined by ideas and experience” (from Mathematics, Form and Function) — or, in other words, it is meaning that resolves the frame problem.

This perspective is in keeping with arguments we’ve made more broadly about how humans mix the content they’re reasoning about into their logical reasoning in a similar way to language models — the nature of both humans and language models is not to behave like systematic symbol processors, but to make effective inferences from any means at hand.

Symbol systems as tools outside a language model

With that perspective, let’s return to AI and mathematical reasoning. I think that the progress in AI has largely followed the perspective the connectionists outlined. In particular, the generality and performance of the systems on the IMO actually improved by removing strict symbolic structure built into the system. In 2024, DeepMind’s system relied on reasoning within the Lean symbolic language — and using the language model simply to guide a tree search. This system required manual human formalization of the problems; even with this additional help it only achieved silver-medal-level performance. However, the following year, a system that reasoned end-to-end in natural language achieved even better (gold-medal) performance; removing the constraint that reasoning be strictly verifiable steps actually improved overall performance — and allowed the model to write correct proofs to more problems, while leaving the proofs relatively readable at the end.

This year, similar systems are making progress on problems that mathematicians had not yet solved (thus conveniently avoiding the common claim that the progress of LMs is simply due to memorization). We don’t know the full details of how these proofs were achieved, but we do know from the announcements that both the Jacobian and unit-distance conjecture results were achieved by general-purpose language models not specifically designed to solve math problems. These systems probably wrote equations, or even programs along the way. They may have also used symbolic solvers for formal verification of result. But that stage, I argue, is irrelevant for the cognitive issues as originally discussed — just as it is when a human mathematician uses any of these tools. Indeed, everything I have described is deeply in keeping with the connectionist perspective I outlined above:

• For both humans and language models, formal reasoning does not come easily.

• It is useful for both humans and models to sometimes use external symbol systems as tools for thinking and verification — both in learning (e.g., see open-source efforts like DeepSeekMath-v2), and later (much as a human mathematician might write down equations while thinking, or use an algebra system to check their derivations).

• But in either case, the intelligence that drives the proof is not a classical symbol system. Instead, it is the intuitions and ability to see patterns and paths to solutions that are key to effective mathematical reasoning — abilities that general-purpose language models have started to emulate.

Thus, while there have been various attempts to reinterpret the recent successes of language models in mathematics as “really” about the success of neuro-symbolic methods, I think that this should not be interpreted as any sort of victory for symbolic accounts of cognition. Instead, I think that recent research is clearly following the vision the connectionists laid out originally³ — that classical symbol systems are useful external tools for an intelligent system to use — but that those symbol systems are not the intelligence itself; even higher-level human cognition behaves more like a neural network than like a classical symbol system. External symbol systems can be partially internalized in such a network (or in humans) through effortful practice, but even then they remain fallible (or graded). That is, I think, a much more faithful description of how mathematical reasoning proceeds — in both humans and language models — than Fodor could have imagined.

1. I mostly use symbols to refer to classical symbol systems in this post given the context of the debates I’m responding to, but my actual view is that symbols should be interpreted as something inherently subjective and graded; see here for further discussion.

2. Though there are certainly large individual differences among mathematicians in how they reason.

3. Conversely, this does not necessarily mean that the original connectionists would endorse the views I have laid out here.