Financial pundits are trying to frighten us into thinking that the "AI bubble" is about to burst and even AI researchers are backtracking, casting doubt on LLMs. Both are wrong - the progress towards general artificial intelligence is real, but there's a lot more to be done before we achieve it. 

Perhaps it is a fear of another AI winter due over-claiming, but the chorus of experts belittling and generally dissing Large Language Models as a false dawn and no way to get to Artificial General Intelligence is increasingly loud. But LLMs prove a point that is more important than their current utility.

The AI research community has long had a tendency to exaggerate progress and to assign successive models with human-like qualities. Often this was, is, a shameful attempt to hoodwink the innocent, but almost as often it is an over-confidence in their creation that makes it possible to think it is more than it is. Some AI researchers have Dr Frankenstein-like desires to create something "live" and their fantasies easily lead them astray.

You cannot argue with the fact that recent advances in AI are amazing and impressive. Back in the day, when we were struggling with logic and rule-based AI and tiny neural networks with tens of neurons, the idea that we could have a conversation with an AI of the quality that even the most basic of LLM can achieve would have been a dream. To put it bluntly, the early AI researchers would have killed to get results like this. 

And yet we know that LLMs have their interesting little problems. This makes them easy targets for all of the pundits who can claim that their stupidity - the LLMs not the pundits - mean that they are not the royal road to AGI despite all the claims from the bosses of the big AI companies:

“We are now confident we know how to build AGI as we have traditionally understood it.”

          Sam Altman, OpenAI

The anti-LLM chorus is growing, mostly based on the mistaken idea that such a simple machine cannot be intelligent. The level of the attack varies from opinion to academic research. For example, a team from Arizona State University published a paper asking, Is Chain-of-Thought Reasoning of LLMs a Mirage?
The answer is an unsurprising "yes". It works well enough for simple things but it breaks down when you try and push it beyond the training data.

Recently Andrej Karparthy, a well known AI researcher and one of the founders of OpenAI, gave an interview which generally sounds pessimistic:

"They just don't work. They don't have enough intelligence, they're not multimodal enough, they can't do computer use and all this stuff,"

"They don't have continual learning. You can't just tell them something and they'll remember it. They're cognitively lacking and it's just not working."

Then there is Yann LeCun, master of the convolutional network and boss of Meta's AI:

"If you are interested in human-level ai, don't work on LLMs"

He outlines alternatives to LLMs that almost seem to take us back to earlier times in AI - energy based models, model-predictive control and so on. But his main point is:

"We are not going to get to human level AI by just scaling up LLMs. This is just not going to happen. Okay, that's your perspective. There's no way. Okay, absolutely no way. And whatever you hear from some of my  more adventurous colleagues, it's not going to happen within the next two years."

I think I agree with this, but not with the idea that we need to look elsewhere to find the answer.

We have the answer.

As Hinton remarked when he was awarded the Turing Prize, what they were doing with neural networks worked from the start. It's just they needed more data to train them on. 

The point of LLMs isn't that they are AGI; it is that they are proof that a neural network can learn something as complex as language and build a world model while using it.

Some observers look at the way that the network was trained and come to the conclusion that it is a simple statistical word completion engine and as such not even "proper" AI - "Stochastic parrot" is a common LLM put-down.

This isn't the whole story. A neural network learns by creating models of the data it is trained on. Language is a model of the world and the neural network learns it by organizing itself into a representation. The fact that we can interact with a something described as "a souped-up autocomplete" in the ways that we do indicates that this is not a good way to think about LLMs. 

It is true that they fail at reasoning tasks, logic, arithmetic, that they have no memory and so on, but they were only trained on language and language is what they do.

LLMs are proof that neural networks can learn aspects of human intelligence and most likely, given the right training data, they can reproduce any sort of intelligence that a human can convincingly manifest.

The road to AGI isn't simple scaling and in this many of the critics are correct, but it is scaling of sorts. The human brain has a range of different networks each dedicated to a particular task. We have modeled language and perhaps this means we have implemented Broca's area and Wernicke's area in the temporal lobe, but our knowledge of what the brain does isn't really good enough to say exactly. We need to use the same techniques to model other areas of the brain and make these models interact. It is also highly likely that consciousness arises from the interaction of different models and this is where AGI might really become dangerous. 

The application of the current crop of LLMs might be being over sold - they certainly lack some abilities - but this is not to say that they are not an example of the solution that we have been looking for. The future of AI is in using multiple systems and obtaining training data from the real world.

I'ii let Geoffrey Hinton have the final word:

"Now that neural nets work, industry and government have started calling neural nets AI. And the people in AI who spent all their life mocking neural nets and saying they'd never do anything are now happy to call them AI and try and get some of the money."

More Information

Geoffrey Hinton and Yann LeCun, 2018 ACM A.M. Turing Award Lecture "The Deep Learning Revolution"

Mathematical Obstacles on the Way to Human-Level AI

Yann LeCun: We Won't Reach AGI By Scaling Up LLMS

Andrej Karpathy — “We’re summoning ghosts, not building animals”

Related Articles

Anthropic Says Claude Sonnet 4.5 Is World's Best Coding Model

If You Sleep Well Tonight You May Not Have Understood

Hinton, LeCun and Bengio Receive 2018 Turing Award

The AI Scam

Why Deep Networks Are Better

The Unreasonable Effectiveness Of GPT-3

It Matters What Language AI Thinks In

The Triumph Of Deep Learning

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info