State-of-the-art AI capabilities vs humans

Let’s take a look at how the most competent AI systems compare with humans in various domains. The list below is regularly updated to reflect the latest developments.

Last update: 2024-03

Superhuman (Better than all humans)

• Games: For many games (Chess, Go , Starcraft, Dota, Gran Turismo etc.) the best AI is better than the best human.
• Memory: An average human can remember about 7 items (such as numbers) at a time. Gemini 1.5 Pro can read and remember 99% of 7 million words .
• Thinking speed: AI models can read thousands of words per second, and write at speeds far surpassing any human.
• Learning speed: A model like Gemini 1.5 Pro can read an entire book in 30 seconds. It can learn an entirely new language and translate texts in half a minute.
• Amount of knowledge: GPT-4 knows far more than any human, its knowledge spanning virtually every domain, even remembering things like URLs.
• Storage efficiency: GPT-4 has about 1.7 trillion parameters , whereas humans have about 100 to 1000 times as much . However, GPT-4 knows thousands of times more, storing more information in a smaller amount of parameters.

Better than most humans

• IQ: Better than 95 to 99% of humans (score between 125 and 155 )
• Creativity: Better than 99% of humans on the Torrance Tests of Creative Thinking . In this test, it created many relevant and useful ideas. However, the tests were relatively small and for larger projects (e.g. setting up a new business) AI is not autonomous enough yet.
• Research: GPT-4 can do autonomous chemical research and DeepMind has built an AI that has found a solution to an open mathematical problem . However, these architectures require a lot of human engineering and are not general.
• Art: Image generation models have won art and even photography contests .
• Language: In December 2023, an AI-written novel won an award at a science fiction national competition . The proffesor who used the AI crafted the narrative from a draft of 43,000 characters generated in just three hours with 66 prompts. The best language models can translate most languages fluently, write poetry and deal with many different styles.
• Specialized knowledge: GPT-4 Scores 75% in the Medical Knowledge Self-Assessment Program , humans on average between 65 and 75% . It scores better than 68 to 90% of law students on the bar exam.
• Programming: GPT-4 can write code in 20+ programming languages and can even create simple games . It can solve many coding challenges in one go, although it does not do well at harder levels . It scores in the bottom 5% of human coders in the Codeforces competition.
• Hacking: GPT-4 can autonomously hack websites and beats 89% of hackers in a Capture-the-Flag competition.

Worse than most humans

• Saying “I don’t know”. Virtually all Large Language Models have this problem of ‘hallucination’, making up information instead of saying it does not know. This might seem like a relatively minor shortcoming, but it’s a very important one. It makes LLMs unreliable and strongly limits their applicability.
• Movement. The Atlas robot can walk, throw objects and do somersaults , but it is still limited in its movements. Google’s RT-2 can turn objectives into actions in the real world, like “move the cup to the wine bottle”.
• Continuous learning. Current SOTA LLMs separate learning (‘training’) from doing (‘inference’). Humans learn and do at the same time, and can learn from a single example.

The endpoint

As time progresses and capabilities improve, we move items from lower sections to the top section. When some specific dangerous capabilities are achieved, AI will pose new risks. At some point, AI will outcompete every human in every metric imaginable. When we have built this superintelligence, we will probably soon be dead .