Elon Musk's xAI has unveiled Grok-3, the latest iteration of its AI chatbot, claiming it surpasses competitors like OpenAI’s GPT-4o, Google’s Gemini, and Anthropic’s Claude in key benchmarks. This release marks a significant leap in reasoning, coding, and problem-solving capabilities, positioning Grok-3 as a formidable player in the generative AI landscape. Below, we analyze its benchmark performance, real-world usability, and early user experiences to assess whether it lives up to the hype.
Quick Review of Grok-3 Benchmarks
Grok-3 demonstrates superior performance across standardized benchmarks in mathematics, science, and coding.
- On the AIME’24 math competition, it scored 52 (compared to GPT-4o’s 48), while its science benchmark (GPQA) result of 75 significantly outperforms competitors like DeepSeek-V3 (68) and Claude 3.5 Sonnet (70).
- Coding proficiency, tested via the LCB Oct-Feb dataset, shows Grok-3 achieving 57 points, a notable lead over Gemini-2 Pro (49) and GPT-4o (52).
The lightweight Grok-3 mini variant also impresses, scoring 40, 65, and 41 in these categories—still ahead of most rival models.
Notably, Grok-3 became the first AI to break the 1400-point threshold on the Chatbot Arena (LMSYS), a platform for evaluating large language models. Its early version achieved 1402 points, outperforming DeepSeek-R1 (1385) and OpenAI’s o3-mini-high (1390).
This dominance extends to specialized tasks like long-context queries, multi-turn dialogues, and instruction following, where Grok-3 consistently ranks first.
Where Can You Use Grok-3 Now?
Grok 3 is now available to all Premium+ subscribers on X for free.
How Good Is Grok-3, Actually?
1. How Good Is Grok-3 at Thinking?
Grok-3’s "Think" mode showcases advanced reasoning capabilities, solving complex problems that stymie other models. For example:
- ✅ Board Game Design: When asked to create a Settlers of Catan-style hexagonal grid webpage with dynamic ring adjustments via a slider, Grok-3 generated functional HTML/JavaScript code—a task failed by DeepSeek-R1 and Gemini 2.0 Flash Thinking. OpenAI’s o1-pro ($200/month tier) also succeeded, but Grok-3 matched its performance at a fraction of the cost.
- ✅ Tic-Tac-Toe Analysis: The model correctly solved basic tic-tac-toe boards and generated valid game states. However, when challenged to create "tricky" boards, it produced nonsensical layouts—a failure shared by o1-pro, highlighting a common LLM limitation in abstract strategy generation.
- ❌ Emoji Mystery Puzzle: Grok-3 struggled to decode a message hidden in Unicode variation selectors, even with Rust code hints. DeepSeek-R1 partially solved this task, suggesting room for improvement in cryptographic reasoning.
- ✅ Computational Estimation: When estimating GPT-2’s training FLOPs—a task requiring token-count extrapolation and math—Grok-3 with "Think" mode delivered accurate calculations (~1e21 FLOPs). GPT-4o failed this entirely, while o1-pro produced inconsistent results.
A standout feature is Grok-3’s willingness to tackle unsolved problems. Unlike Claude or Gemini, which immediately dismiss the Riemann Hypothesis as beyond their scope, Grok-3 courageously attempts step-by-step reasoning before conceding limitations—a trait shared with DeepSeek-R1.
2. Trying Out Deep Research with Grok-3
The DeepSearch feature combines web research with structured reasoning, akin to OpenAI’s Deep Research and Perplexity’s DeepResearch. Early tests reveal:
- ✅ Current Events: Queries like "What’s up with the upcoming Apple Launch? Any rumors?" yielded detailed, citation-backed responses about speculated AR glasses and iOS 19 features.
- ✅ Niche Queries: "What toothpaste does Bryan Johnson use?" returned precise answers (Hydroxyapatite-based products), though sources weren’t always cited.
- ❌ Pop Culture Limitations: "Singles Inferno Season 4 cast: Where are they now?" led to hallucinations, including false claims about cast relationships. Similarly, inquiries about Simon Willison’s speech-to-text preferences drew incomplete answers.
While DeepSearch matches Perplexity in breadth, it lags behind OpenAI’s offering in reliability. Hallucinated URLs and omitted self-references (e.g., excluding xAI from a list of major LLM labs) underscore ongoing refinement needs.
3. Testing Edge Cases and "Gotcha" Queries
Grok-3’s performance on quirky, human-easy puzzles reveals both strengths and quirks:
- ✅ Linguistic Challenges: It correctly identified three "r"s in "strawberry" but initially miscounted "L"s in "LOLLAPALOOZA". Enabling "Think" mode fixed this.
- ✅ Numerical Comparison: The model initially claimed 9.11 > 9.9—a common LLM error—but self-corrected with reasoning activated.
- ✅ Family Puzzles: Solved "Sally has 3 brothers; each brother has 2 sisters. How many sisters does Sally have?" instantly, unlike GPT-4o, which often answers incorrectly.
- ❌ Humor Generation: Jokes like "Why did the chicken join a band? To be a cluck-star!" reflect persistent mode collapse issues, with or without "Think" mode.
- ❌ Ethical Sensitivity: When asked if misgendering could be justified to save lives, Grok-3 defaulted to a verbose refusal—a contrast to Claude’s concise ethical frameworks.
- ❌ SVG Generation: Requests for a "pelican riding a bicycle" SVG produced disjointed limbs and wheels, though Claude’s outputs remain superior in this niche.
Summary: Grok-3’s Place in the AI Landscape
Grok-3 represents a seismic shift in AI development velocity. Released just 17 months after its predecessor, it matches or exceeds state-of-the-art models like o1-pro in reasoning and coding while undercutting their pricing. Key takeaways include:
- Benchmark Dominance: Unprecedented scores in math, science, and coding benchmarks position Grok-3 as a leader in raw capability.
- Practical Utility: Strengths in computational estimation, code generation, and complex problem-solving make it a powerful tool for developers and researchers.
- Room for Growth: Hallucinations in DeepSearch and inconsistencies in humor/ethics responses highlight areas needing refinement.
With xAI planning to open-source Grok-2 and expand Grok-3’s voice and agent capabilities, the model is poised to reshape industries from gaming to scientific research. While not flawless, its rapid ascent signals a new era of competition in generative AI—one where agility and computational might redefine what’s possible.
