o3: Pushing the boundaries of AGI (and of coding)

Video Description

As part of its “12 days of OpenAI”, OpenAI announced the new o3 model. This model has surpassed previous records in reasoning, science and math competitions, and programming. The model even passes the ARC AGI benchmark with above human performance, when given enough compute. It seems clear that o3 is superhuman at reasoning. The question is, does this translate into sufficient generality to call it AGI (artificial general intelligence)? We do see tasks where reasoning plays a big part becoming more at risk of automation. Many software engineers in particular are worried about their jobs, since AI models will soon be able to perform large chunks of their work. We present five stages that automation proceeds through, with software engineering soon to be at stage four of five. While there are no answers about what to do, we do analyze what happened in games in the past where the best human players were beaten. #ai #agi #automation OpenAI o3 Breakthrough High Score on ARC-AGI-Pub https://arcprize.org/blog/oai-o3-pub-breakthrough OpenAI introduces o3 model family as a successor to o1 model focusing on AI reasoning https://www.techloy.com/openai-introduces-o3-model-family-as-successor-to-o1-model-focusing-on-ai-reasoning/ OpenAI announces new o3 models https://techcrunch.com/2024/12/20/openai-announces-new-o3-model/ Tech Things: AI Benchmarks, O3, and the Future of SWE https://theahura.substack.com/p/tech-things-ai-benchmarks-o3-and FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI https://epoch.ai/frontiermath/the-benchmark OpenAI considers AGI clause removal for Microsoft investment https://finance.yahoo.com/news/openai-considers-agi-clause-removal-123030732.html @adcock_brett: “OpenAI announced 'o3', the next iteration of o1...” https://x.com/adcock_brett/status/1870877753081217359 @_florianmai: “o3 is better than 99.95% of programmers...” https://x.com/_florianmai/status/1870191021440512431 @TolgaBilge_: “OpenAI board member @adamdangelo..." https://x.com/TolgaBilge_/status/1870904304049217725 How I came in first on ARC-AGI-Pub using Sonnet 3.5 with Evolutionary Test-time Compute https://jeremyberman.substack.com/p/how-i-got-a-record-536-on-arc-agi 0:00 Intro 0:29 Contents 0:37 Part 1: Going exponential 0:48 Naming and announcement of o3 1:15 Twelve days of OpenAI 1:42 o1-preview was released in a rush 2:29 Three months between releases 3:13 Exponential trend of AI capabilities 3:50 Questions to address: AGI? Coding? 4:11 Part 2: A question of generality 4:16 ARC AGI benchmark analysis 4:53 General trend: very good at reasoning 5:24 Big question is generalization 6:03 Incentives to avoid declaring AGI 6:46 Creator of ARC benchmark on livestream 7:19 Quote from ARC about o3 approaching human level performance 7:56 o3 is superhuman at reasoning (and decent at programming) 8:52 Question of AGI is one of generality 9:17 ARC benchmark describes o3's inner workings 9:52 Outsourcing safety to reasoning (deliberative alignment) 10:09 o3 is enough to change the world 10:25 Part 3: Why developers are worried 10:41 Competitive programming competitions 11:17 CodeForces ranking: 175 out of 600,000 12:02 Grandmaster CodeForces ranking 12:25 Real-world SWE-bench verified 13:07 Fully autonomous coding agent gets 71.7% 14:03 Programming is on life support 14:12 Five stages of automation 14:30 Digital art and programming at stage 4 15:15 Anxiety about losing jobs 15:30 Recursive self-improvement 16:04 Job loss analogy to playing games 16:37 Examples of games beaten by AI 16:48 Example 1: Go was beaten by AlphaGo 17:57 While Lee Sedol quit Go, others kept playing 18:35 Example 2: StarCraft 2 beaten by AlphaStar 19:27 Takeaways from how game AIs evolved 19:48 Quote from Ahura: "computer science is dead" 20:08 My takeaways, with a book recommendation 20:47 Conclusion 21:30 o3 will cause massive disruption to software engineering 22:01 Outro