Compare Claude 3.5 Haiku

What each rating means here

Strong

Reliably solves complex multi-step problems — handles logic puzzles, mathematical proofs, and extended chain-of-thought reasoning with high accuracy.

Moderate

Handles straightforward reasoning tasks well but makes errors on longer inference chains or adversarial logic problems.

Weak

Frequently fails on problems requiring more than one or two reasoning steps; prone to confident-sounding but incorrect answers.

What each rating means here

Strong

Writes correct, idiomatic code for complex tasks and debugs reliably across major languages; handles algorithmic and architectural decisions.

Moderate

Solid at common patterns and languages; struggles with complex algorithms, obscure libraries, or large codebase context.

Weak

Produces working code only for simple, well-trodden patterns; frequent bugs and hallucinated APIs on anything non-trivial.

What each rating means here

Strong

Reliably selects and calls the correct tool with well-formed parameters; handles multi-turn tool loops and chained dependent calls gracefully.

Moderate

Gets basic tool calls right but struggles with ambiguous tool selection, malformed prior outputs, or calls requiring context from earlier turns.

Weak

Inconsistent parameter formatting; prone to hallucinating tool names or arguments, or selecting the wrong tool.

What each rating means here

Strong

Follows complex multi-constraint instructions precisely — respects format, length limits, persona, and negative instructions consistently.

Moderate

Handles simple instructions well but drifts on multi-constraint prompts or instructions that conflict with the model's trained defaults.

Weak

Often ignores or partially follows instructions; defaults to its own style regardless of stated format or constraint requirements.

What each rating means here

Strong

Reliably attends to information throughout a long context — accurately recalls and synthesises content from early in a large document or conversation.

Moderate

Handles mid-length contexts well; recency bias emerges as context grows — information near the start of a long prompt is sometimes lost.

Weak

Loses track of earlier content in long contexts; practical reliable length is significantly shorter than the advertised token ceiling.

Variable

Recall quality depends on content type and its position in the context — performance varies across providers and benchmark methodologies.

What each rating means here

Strong

High-quality outputs across many major languages; reliable at translation and cross-lingual reasoning with consistent grammar and factual accuracy.

Moderate

Good at widely-spoken languages (Spanish, French, German, Chinese) but quality degrades noticeably for lower-resource languages.

Weak

Primarily English-optimised; outputs in other languages may be fluent but are factually unreliable or grammatically inconsistent.

What each rating means here

Strong

High throughput — fast first-token latency and generation speed well-suited to real-time, interactive, and high-volume applications.

Moderate

Adequate for most use cases; may introduce noticeable latency in latency-sensitive or streaming applications.

Weak

Slow generation; adds significant per-token latency and raises cost at scale.

Variable

Speed varies with generation length, prompt complexity, and server load — check provider latency benchmarks for your specific use case.