Great read - looking forward to the rest of the series!
This reminded me of this recent-ish essay: https://gershmanlab.com/pubs/NeuroAI_critique.pdf. The core argument here (as I understand it) is that the very notion of biological inspiration for AI is somewhat ill-defined, because characterizing biological principles in the first place requires coming to the problem with a computational framework to make sense of the data: it’s not like the principles are just “there”. The proposal here is that rather than attempting to use neural/cognitive plausibility as a source of design principles, we can instead use it as a tiebreaker between candidate algorithms, under the assumption that algorithms that are actually implemented by biological systems are generally better.
As a separate but related note: even if we did have detailed, integrative, computational theories of complex cognitive phenomena, it’s not obvious that we should strive to integrate those insights into AI systems, because the constraints imposed on biological intelligence and AI systems are often quite different. For example, if a particular cognitive phenomenon is a byproduct of the fact that we humans have limited working memory, it’s not clear why we would try to bake that into AI, which is not bound by those same limitations. For some domains, the core problem being solved by both humans and AI may be similar enough that the solutions may be transferrable (and that doing so could be beneficial), but for others this seems less obvious.
Great points & pointers, thanks! On the different constraints, I certainly agree — but I tend to think that we'd like an ideal AI systems to be able to solve most problems at least as well as humans (though it might certainly outperform along some axes due to lacking some human constraints). Thus, in the cases where humans still appear to be more efficient or better continual learners etc., it seems plausible that there is something to be something to be learned at the computational level at least.
I hope you will address what seems to me the core question about the relationship between AI and cognitive science, which is “Does the brain do anything resembling backprop and gradient descent” when humans learn by exposure to pattern-rich sensory inputs.
I think an important problem with the field of NeuroAI is that strong claims are too often made based on extremely weak evidence. For instance, you write: " language models can predict human imaging data remarkably well", but it is important to be clear what the authors you cite found. They reported models could account for near 100% of explainable variance, but the explainable variance was very very low (in most cases between 4-10%), and that a similar amount of variance was observed in non-language areas, raising questions about how to interpret the findings. And you write cognitive scientists have suggested that AI progress refutes one of the dominant linguistic paradigms. Again, that is true, but the evidence for this claim is extremely weak, as reviewed here. https://psycnet.apa.org/record/2026-83323-001. Do you think the papers you cite make a strong cases for ANN-human alignment, and challenge the role of innate priors? I think the field could use a bit more scepticism.
I generally agree that some skepticism is warranted — as you probably know, I've written about computational challenges to trusting some model-brain representational comparisons myself, (e.g. https://arxiv.org/abs/2507.22216). That theme will maybe come up more strongly in future posts (including the one about what cog sci can learn from AI). But I also think that the fact that e.g. explainable variance is low is in part due to the critique I make here: explainable variance is low because neuroscience tasks are just not varied, complex, or sequential enough to drive neural activity all that strongly (or allow us to account for individual differences all that well).
On your specific points for skepticism, I think some are valid. But I also think you read the human-side literature far too uncritically. The "impossible" languages results are based on a couple of almost-anecdotal studies done on *adults* who were already fluent in another language. They would have to be done in the critical period and as first language to be a really meaningful measure of what's learnable. I'd love to see someone do a modern large-scale developmental study on that, and a comparative study on models, which do show some language inductive biases as you probably know: https://arxiv.org/abs/2401.06416
I think the impossible language article you cite is a good case in point. The simulations the authors report do not support the claims the authors make. Here is a commentary I submitted for BBS paper that reviews the findings on impossible languages and LLMs. https://arxiv.org/pdf/2511.11389. Note, one of the impossible language simulations we published in our original work would not be learnable as it requires a super-human STM: https://aclanthology.org/2020.coling-main.451/. And as noted in our commentary on the LLM Centaur that was claimed to account for 159 out of 160 experiments better than cognitive models, including STM model, Centaur could sometimes repeat 256 digits perfectly: https://doi.org/10.31234/osf.io/v9w37_v4. All examples of what I'm talking about. Sure, there are problems with the cognitive science literature as well, but the claims in NeuroAI are being uncritically published in all the top journals. Criticisms of these claims, not so much.
This is a great argument to present to folks who state so confidently that the current approaches are wrong because that is totally not what our mind is doing.
Hi! I really enjoy the article, and what you are saying reminds me of emergence and complexity science. Do you have any reading recommendations if I am curious about the topic? Thanks, ,and looking forward to your next article!
Great read! I have always wondered why there is a rush to create systems that are gargantuan and consume lots of resources when we are tiny and consume so little. You provide lots of good reasons for that. Nevertheless, I am an HI (human intelligence) optimist. AI is great but what we will do with these new tools is even greater.
The point about fragmentary understanding breaking down at integration points feels key — many AI failures seem to emerge not from missing components, but from how multiple constraints interact.
I want to help. As the least likely person to contribute to the field of AI in the early 2022, I read a lot of books and papers, watched multiple videos in philosophy, cogsci, AI, psychology, etc. I learned about basic observations the "big picture" needs to explain.
I had many useful early insights, but only recently I could connect many dots to believe that I can help. And of all the AI labs, I think GoogleDeepMind is the best one to understand. You had MCTS. I propose Semantic Binary Search. You can appreciate its potential. But you are right - a big demo is needed and I am only a one-man team.
There are many misleading assumptions to throw away. Fine lines between some of them and the better ones. But it's worth it. The promise is there!
You are welcome to my Substack. Also, consider taking a look at this paper where I take the best points so far and connect many dots - https://philpapers.org/rec/NAUNOC-2
Great read - looking forward to the rest of the series!
This reminded me of this recent-ish essay: https://gershmanlab.com/pubs/NeuroAI_critique.pdf. The core argument here (as I understand it) is that the very notion of biological inspiration for AI is somewhat ill-defined, because characterizing biological principles in the first place requires coming to the problem with a computational framework to make sense of the data: it’s not like the principles are just “there”. The proposal here is that rather than attempting to use neural/cognitive plausibility as a source of design principles, we can instead use it as a tiebreaker between candidate algorithms, under the assumption that algorithms that are actually implemented by biological systems are generally better.
As a separate but related note: even if we did have detailed, integrative, computational theories of complex cognitive phenomena, it’s not obvious that we should strive to integrate those insights into AI systems, because the constraints imposed on biological intelligence and AI systems are often quite different. For example, if a particular cognitive phenomenon is a byproduct of the fact that we humans have limited working memory, it’s not clear why we would try to bake that into AI, which is not bound by those same limitations. For some domains, the core problem being solved by both humans and AI may be similar enough that the solutions may be transferrable (and that doing so could be beneficial), but for others this seems less obvious.
Great points & pointers, thanks! On the different constraints, I certainly agree — but I tend to think that we'd like an ideal AI systems to be able to solve most problems at least as well as humans (though it might certainly outperform along some axes due to lacking some human constraints). Thus, in the cases where humans still appear to be more efficient or better continual learners etc., it seems plausible that there is something to be something to be learned at the computational level at least.
I hope you will address what seems to me the core question about the relationship between AI and cognitive science, which is “Does the brain do anything resembling backprop and gradient descent” when humans learn by exposure to pattern-rich sensory inputs.
I think an important problem with the field of NeuroAI is that strong claims are too often made based on extremely weak evidence. For instance, you write: " language models can predict human imaging data remarkably well", but it is important to be clear what the authors you cite found. They reported models could account for near 100% of explainable variance, but the explainable variance was very very low (in most cases between 4-10%), and that a similar amount of variance was observed in non-language areas, raising questions about how to interpret the findings. And you write cognitive scientists have suggested that AI progress refutes one of the dominant linguistic paradigms. Again, that is true, but the evidence for this claim is extremely weak, as reviewed here. https://psycnet.apa.org/record/2026-83323-001. Do you think the papers you cite make a strong cases for ANN-human alignment, and challenge the role of innate priors? I think the field could use a bit more scepticism.
I generally agree that some skepticism is warranted — as you probably know, I've written about computational challenges to trusting some model-brain representational comparisons myself, (e.g. https://arxiv.org/abs/2507.22216). That theme will maybe come up more strongly in future posts (including the one about what cog sci can learn from AI). But I also think that the fact that e.g. explainable variance is low is in part due to the critique I make here: explainable variance is low because neuroscience tasks are just not varied, complex, or sequential enough to drive neural activity all that strongly (or allow us to account for individual differences all that well).
On your specific points for skepticism, I think some are valid. But I also think you read the human-side literature far too uncritically. The "impossible" languages results are based on a couple of almost-anecdotal studies done on *adults* who were already fluent in another language. They would have to be done in the critical period and as first language to be a really meaningful measure of what's learnable. I'd love to see someone do a modern large-scale developmental study on that, and a comparative study on models, which do show some language inductive biases as you probably know: https://arxiv.org/abs/2401.06416
I think the impossible language article you cite is a good case in point. The simulations the authors report do not support the claims the authors make. Here is a commentary I submitted for BBS paper that reviews the findings on impossible languages and LLMs. https://arxiv.org/pdf/2511.11389. Note, one of the impossible language simulations we published in our original work would not be learnable as it requires a super-human STM: https://aclanthology.org/2020.coling-main.451/. And as noted in our commentary on the LLM Centaur that was claimed to account for 159 out of 160 experiments better than cognitive models, including STM model, Centaur could sometimes repeat 256 digits perfectly: https://doi.org/10.31234/osf.io/v9w37_v4. All examples of what I'm talking about. Sure, there are problems with the cognitive science literature as well, but the claims in NeuroAI are being uncritically published in all the top journals. Criticisms of these claims, not so much.
This is a great argument to present to folks who state so confidently that the current approaches are wrong because that is totally not what our mind is doing.
wrote abt this along the same lines a couple of years ago, which might be of interest: https://www.aishwaryadoingthings.com/from-physics-envy-to-biology-envy
Sounds right. I'm looking forward to reading your follow-up posts.
Hi! I really enjoy the article, and what you are saying reminds me of emergence and complexity science. Do you have any reading recommendations if I am curious about the topic? Thanks, ,and looking forward to your next article!
Great read! I have always wondered why there is a rush to create systems that are gargantuan and consume lots of resources when we are tiny and consume so little. You provide lots of good reasons for that. Nevertheless, I am an HI (human intelligence) optimist. AI is great but what we will do with these new tools is even greater.
This framing resonates.
The point about fragmentary understanding breaking down at integration points feels key — many AI failures seem to emerge not from missing components, but from how multiple constraints interact.
Thanks for this great outline!
I want to help. As the least likely person to contribute to the field of AI in the early 2022, I read a lot of books and papers, watched multiple videos in philosophy, cogsci, AI, psychology, etc. I learned about basic observations the "big picture" needs to explain.
I had many useful early insights, but only recently I could connect many dots to believe that I can help. And of all the AI labs, I think GoogleDeepMind is the best one to understand. You had MCTS. I propose Semantic Binary Search. You can appreciate its potential. But you are right - a big demo is needed and I am only a one-man team.
There are many misleading assumptions to throw away. Fine lines between some of them and the better ones. But it's worth it. The promise is there!
You are welcome to my Substack. Also, consider taking a look at this paper where I take the best points so far and connect many dots - https://philpapers.org/rec/NAUNOC-2
I am willing to help!