Summary: Things We Learned About LLMs in 2024

This is a summary of Simon Willison’s comprehensive 2024 LLM review article.

Key Developments in LLMs During 2024

1. The GPT-4 Barrier Was Broken

• 18 organizations now have models ranking higher than the original GPT-4 from March 2023
• Major players include Google, OpenAI, Alibaba, Anthropic, Meta, and others
• Training a GPT-4-beating model became commonplace in 2024

2. Local Model Running Became Possible

• GPT-4 class models can now run on consumer hardware
• Examples include Qwen2.5-Coder-32B and Meta’s Llama 3.3 70B
• Significant improvements in model efficiency made this possible

3. LLM Prices Crashed

• GPT-4 pricing dropped from $30/million tokens to much lower rates
• Current costs:
  • GPT-4o: $2.50/million tokens
  • GPT-4o mini: $0.15/million tokens
  • Claude 3 Haiku: $0.25/million tokens
  • Gemini 1.5 Flash: $0.075/million tokens

4. Multimodal Capabilities Expanded

• Vision, audio, and video processing became common features
• Major releases included:
  • Claude 3 series (March)
  • Gemini 1.5 Pro (April)
  • Various vision models from Meta, Mistral, and others

5. Voice and Live Camera Integration

• Advanced voice interactions became available
• Real-time camera feed processing was introduced
• Both ChatGPT and Google Gemini offered these features

6. New Developments in Model Architecture

• Introduction of inference-scaling “reasoning” models
• OpenAI’s o1 and o3 series
• DeepSeek v3’s efficient training approach

Environmental Impact

Positive Developments

• Reduced energy usage per prompt
• More efficient training methods
• Lower costs indicating better resource utilization

Concerns

• Massive infrastructure buildout by tech companies
• Significant datacenter expansion
• Growing energy grid demands

Challenges and Concerns

1. Usability Issues

   • LLMs became more complex to use effectively
   • Growing knowledge gap between experts and casual users
   • Need for better education and documentation

2. Knowledge Distribution

   • Uneven awareness of available tools and capabilities
   • Rapid pace of change making it difficult to keep up

3. Critical Evaluation Needed

   • Balance between criticism and practical application
   • Need for responsible implementation guidelines
   • Importance of understanding both limitations and capabilities

Looking Forward

The field continues to evolve rapidly, with new capabilities and challenges emerging regularly. The focus seems to be shifting toward:

• More efficient training methods
• Better multimodal integration
• Improved accessibility and usability
• Responsible development and deployment

Source: This summary is based on Simon Willison’s detailed article “Things we learned about LLMs in 2024”