LLM benchmark: GPT-4o and Claude 3.5 Sonnet lead MMLU, GPQA and MATH
Thread226 words
Relays Artificial Analysis benchmarking of leading LLMs on MMLU, GPQA and MATH, prompted by the Reflection Llama 3.1 70B controversy. GPT-4o and Claude 3.5 Sonnet top all three categories.
Key claims
GPT-4o and Claude 3.5 Sonnet are the best performing models across MMLU, GPQA and MATH
๐๐ฌ๐ข๐ง๐ ๐๐ ๐๐๐ง ๐๐๐ฑ ๐ฒ๐จ๐ฎ๐ซ ๐ฉ๐ซ๐จ๐๐ฎ๐๐ญ๐ข๐ฏ๐ข๐ญ๐ฒ. ๐ค โจ
The question is, which AI tool should you use?
What it all comes down to, is the performance of the underlying LLM.
LLM performance analysis featuring @OpenAI, @AnthropicAI, @Google, @Meta & co. ๐ โคต๏ธ
Due to the recent controversy around @OthersideAI's Reflection Llama 3.1 70B model performance (see link in comments), @ArtificialAnlys made an independent assessment of the performance of leading LLMs.
The analysis is based on MMLU, GPQA, and MATH, which are all essential metrics for evaluating LLM performance across distinct reasoning and knowledge domains:
โ ๐ ๐ ๐๐จ (๐ ๐ฎ๐๐๐ถ๐๐ฒ ๐ ๐๐น๐๐ถ๐๐ฎ๐๐ธ ๐๐ฎ๐ป๐ด๐๐ฎ๐ด๐ฒ ๐จ๐ป๐ฑ๐ฒ๐ฟ๐๐๐ฎ๐ป๐ฑ๐ถ๐ป๐ด): ๐ ๐๐๐พ๐๐๐พ๐ ๐๐พ๐๐พ๐๐บ๐
๐๐พ๐บ๐๐๐๐๐๐ ๐บ๐๐ฝ ๐ฟ๐บ๐ผ๐๐๐บ๐
๐๐๐๐๐
๐พ๐ฝ๐๐พ ๐บ๐ผ๐๐๐๐ ๐ง๐ฉ ๐๐๐ป๐๐พ๐ผ๐๐, ๐ฟ๐๐๐ ๐๐๐๐บ๐๐๐๐๐พ๐ ๐๐ ๐๐ผ๐๐พ๐๐ผ๐พ๐. ๐จ๐ ๐๐พ๐บ๐๐๐๐พ๐ ๐๐๐ ๐๐พ๐
๐
๐บ ๐๐๐ฝ๐พ๐
๐๐บ๐๐ฝ๐
๐พ๐ ๐ฝ๐๐๐พ๐๐๐พ ๐ฝ๐๐๐บ๐๐๐ ๐บ๐๐ฝ ๐บ๐๐๐
๐๐พ๐ ๐๐๐๐๐
๐พ๐ฝ๐๐พ.
โ ๐๐ฃ๐ค๐ (๐๐ฒ๐ป๐ฒ๐ฟ๐ฎ๐น-๐ฃ๐๐ฟ๐ฝ๐ผ๐๐ฒ ๐ค๐๐ฒ๐๐๐ถ๐ผ๐ป ๐๐ป๐๐๐ฒ๐ฟ๐ถ๐ป๐ด): ๐ค๐๐บ๐
๐๐บ๐๐พ๐ ๐๐ผ๐๐พ๐๐๐๐ฟ๐๐ผ ๐๐พ๐บ๐๐๐๐๐๐ ๐บ๐๐ฝ ๐๐พ๐ผ๐๐๐๐ผ๐บ๐
๐๐๐๐๐
๐พ๐ฝ๐๐พ. ๐จ๐ ๐๐พ๐๐๐ ๐๐๐พ ๐๐๐ฝ๐พ๐
โ๐ ๐บ๐ป๐๐
๐๐๐ ๐๐ ๐๐พ๐บ๐๐๐ ๐๐๐๐๐๐๐ ๐๐๐พ๐ผ๐๐บ๐
๐๐๐พ๐ฝ ๐ฝ๐๐๐บ๐๐๐ ๐
๐๐๐พ ๐๐ผ๐๐พ๐๐ผ๐พ, ๐พ๐๐๐๐๐พ๐พ๐๐๐๐, ๐บ๐๐ฝ ๐ฟ๐บ๐ผ๐๐๐บ๐
๐บ๐ผ๐ผ๐๐๐บ๐ผ๐.
โ๐ ๐๐ง๐: ๐ฅ๐๐ผ๐๐๐พ๐ ๐๐ ๐๐๐บ๐๐๐๐๐บ๐๐๐๐พ ๐๐พ๐บ๐๐๐๐๐๐ ๐บ๐๐ฝ ๐๐บ๐๐๐พ๐๐บ๐๐๐ผ๐บ๐
๐๐๐๐ป๐
๐พ๐-๐๐๐
๐๐๐๐, ๐บ๐๐๐พ๐๐๐๐๐ ๐บ ๐๐๐ฝ๐พ๐
โ๐ ๐บ๐ป๐๐
๐๐๐ ๐๐ ๐๐พ๐๐ฟ๐๐๐ ๐
๐๐๐๐ผ๐บ๐
๐ฝ๐พ๐ฝ๐๐ผ๐๐๐๐๐ ๐บ๐๐ฝ ๐๐๐๐พ๐๐๐ผ๐บ๐
๐ผ๐๐๐๐๐๐บ๐๐๐๐๐ ๐๐๐๐ ๐๐๐พ๐ผ๐๐๐๐๐.
These metrics collectively provide a comprehensive overview of an LLM's ability to handle broad knowledge, specialized science, and complex quantitative tasks.
The results show that @OpenAI's GPT-4o and @AnthropicAI's Claude 3.5 Sonnet are the best performing models across all three categories.
A lot of money being spent on this btw. ๐ธ ๐
x.com/zerokn0wledge_/status/1826089584133804150
1 image on X
What came next ยท since 10 Sep 2024
Since then
Zero has written 2 more pieces on OpenAI since this one.