Chain of thought

TL;DR

• Chain-of-thought (CoT) prompting asks a model to generate intermediate work before answering. The original PaLM study reported a GSM8K improvement from 17.9% to 58.1% with few-shot CoT, but gains vary by model, task, and prompt.
• Zero-shot CoT is one sentence: append "Let's think step by step." Few-shot CoT provides worked examples the model imitates.
• Self-consistency samples 10-40 reasoning paths and takes majority vote, adding another 10-20 percentage points on hard problems.
• CoT gains depend on the model and task; early studies found especially large gains on very large models, while smaller models can gain little or regress. Benchmark the target model instead of relying on a parameter threshold.
• Some reasoning models spend additional internal computation before answering, and providers may not expose those tokens. Do not request or reconstruct private reasoning; use a concise rationale or structured checks when an observable explanation is needed.
• The engineering decision: use CoT for multi-step reasoning tasks, skip it for factual retrieval, and graduate to reasoning models when CoT alone is not enough.

30-Second Explanation

Mental model: chain-of-thought prompting gives a model room to generate intermediate work before its final answer, which can help on multi-step tasks. Those generated steps are not guaranteed to be faithful or correct, so use concise, observable summaries or structured intermediate artifacts and verify important results; do not expose private reasoning traces as if they were ground truth.

The problem it solves

Ask GPT-4 a direct question: "A bat and ball cost $1.10. The bat costs $1.00 more than the ball. How much does the ball cost?" Without CoT, the model often answers $0.10. That's wrong. The correct answer is $0.05.

The failure is systematic, not random. Models trained to predict the next token optimize for fluency, not deliberate reasoning. On multi-step problems, they pattern-match to the surface structure and jump to an answer that "feels right." The intermediate algebra ($1.10 - $1.00 = $0.10 is the gap, not the answer) never gets computed because the model never generates those intermediate tokens.

This isn't a rare edge case. On the GSM8K benchmark (grade school math, 8,500 problems), standard prompting yields under 18% accuracy on PaLM 540B. The model can do the math. It just doesn't do the math unless you tell it to show its work.

The same pattern appears in code generation, legal reasoning, medical diagnosis, and every other domain where the answer depends on a chain of intermediate conclusions. Direct prompting fails wherever the gap between question and answer requires more than one logical step.

The core insight: when a model generates intermediate tokens, each step conditions the next step. This can make multi-step computation easier, but it does not guarantee that the intermediate work is correct or faithful. Independent checks are still needed for important results.

What is it?

Chain-of-thought prompting is a technique where you ask the model to generate intermediate work before giving the final answer. Wei et al. introduced the term in 2022 and showed improvements on arithmetic, commonsense, and symbolic reasoning tasks without changing the model weights. The generated steps are output text, not a guaranteed transcript of the model's internal computation.

Think of it like writing intermediate work on a math exam. The extra steps can act as a scratchpad that the model conditions on while it generates, and they can make some errors easier to inspect. But a plausible-looking rationale is not proof of correctness, and it should not be treated as a private chain of thought.

There are two main flavors. Zero-shot CoT appends a single phrase like "Let's think step by step" to your prompt. Few-shot CoT provides several worked examples with explicit reasoning chains, then presents the real question. Both work. Zero-shot is more convenient. Few-shot wins when you need the model to reason in a specific structure or domain.

The reason this works is mechanical, not magical. During pretraining, the model saw millions of worked solutions. CoT prompting creates the context that tells the model "we're in problem-solving mode, show the work." The reasoning tokens become a scratchpad the model can reference as it generates each subsequent token.

CoT can be a high-leverage prompting technique for some multi-step tasks, but its value depends on the model, task, and verification cost. A short benchmark is more reliable than assuming a fixed accuracy multiplier.

Here's the simplest possible example. Without CoT:

Q: If a store has 3 shelves with 8 books each, and gets
   a delivery of 15 books, how many books total?
A: 39

With zero-shot CoT:

Q: If a store has 3 shelves with 8 books each, and gets
   a delivery of 15 books, how many books total?
   Let's think step by step.
A: First, calculate books on shelves: 3 x 8 = 24 books.
   Then add the delivery: 24 + 15 = 39 books.
   The answer is 39.

The answer happens to be the same here, but the CoT version makes intermediate work visible. Visibility does not make it verifiable by itself: a separate calculator, compiler, rule engine, retrieval check, or evaluator may be needed. In production, keep structured inputs, outputs, tool results, and evaluator decisions as audit artifacts; treat generated reasoning-like text as untrusted output.

How it works

Zero-shot CoT

Kojima et al. (2022) made a surprisingly simple discovery: appending "Let's think step by step" to a prompt activates reasoning behavior already latent in large models. No examples needed. No special formatting. Just that one phrase.

The mechanism is straightforward. The model's pretraining corpus included millions of worked solutions: math textbooks, code with comments, argumentative essays with step-by-step logic. When you say "let's think step by step," you're activating those latent reasoning patterns. The model shifts from "predict the most likely answer token" to "predict the most likely next reasoning token."

Why does this work mechanically? Autoregressive models generate one token at a time. Each new token can attend to all previously generated tokens. When the model generates reasoning tokens before the answer token, those intermediate tokens become part of the context. The answer token is now conditioned on a sequence that includes the worked steps, not just the original question. It's the difference between solving a math problem in your head versus writing it out on paper.

On the MultiArith benchmark, zero-shot CoT improved accuracy from 17.7% to 78.7% on InstructGPT (175B). On AddSub, it went from 74.5% to 85.7%. These are massive gains from a single sentence.

The practical advice: start here. Zero-shot CoT is your baseline for any multi-step reasoning task. If it's not accurate enough, escalate to few-shot CoT.

Few-shot CoT

Wei et al. (2022) showed that including 3-8 worked examples with explicit reasoning chains gives the model a template to follow. The model doesn't just reason; it reasons in the format you demonstrated.

Few-shot CoT on PaLM 540B pushed GSM8K accuracy from 17.9% to 58.1%. That's a 3.2x improvement from adding examples to the prompt.

I use few-shot CoT when I need the model to reason in a specific structure. If I want tabular analysis, I show an example of tabular reasoning. If I want the model to consider edge cases, I include an example that does exactly that. The model follows the template faithfully.

One practical note: the quality of your examples matters enormously. Bad examples (sloppy reasoning, skipped steps) produce bad intermediate work. Most of the effort should go into choosing and checking examples, not merely formatting the question.

Observable summaries, not private chains of thought

A model can return a concise rationale, assumptions, citations, or structured intermediate results when people need an explanation. These artifacts are useful for review, but they can be incomplete or post-hoc and should not be treated as a faithful transcript of internal reasoning. Some reasoning models intentionally keep private reasoning tokens hidden. For production systems, prefer observable checks—such as tool results, calculations, retrieved evidence, tests, or a separate evaluator—and do not expose hidden reasoning traces.

Self-consistency

Wang et al. (2022) observed that sampling a model multiple times produces diverse reasoning paths. Wrong answers tend to be diverse (each wrong for a different reason), while correct answers cluster around the same value.

Self-consistency exploits this: sample N reasoning chains with temperature > 0, extract the final answer from each, and take the majority vote.

On GSM8K, self-consistency with 40 samples pushed PaLM 540B from 58.1% (few-shot CoT) to 74.4%. On the ARC challenge, it improved from 85.2% to 88.7%. The pattern holds across benchmarks: 10-20 percentage point gains on tasks where single-path CoT plateaus.

The tradeoff is cost. Sampling 40 times means 40x the model calls. You can often get much of the benefit with 5-10 samples, but the useful count depends on the task and the evaluator. Use self-consistency only when the measured accuracy gain justifies the extra cost and latency; for high-stakes decisions, add domain-specific review rather than relying on voting alone.

Tree of Thoughts

Yao et al. (2023) extended CoT from a single chain into a search tree. At each reasoning step, the model generates multiple candidate continuations, evaluates them, and prunes the weak branches before continuing.

The key difference from self-consistency: self-consistency samples independent paths and votes at the end. Tree of Thoughts evaluates and prunes at each step. This makes ToT better for problems with large reasoning spaces where bad branches should be abandoned early rather than explored to completion.

Tree of Thoughts (ToT) is powerful for problems that require exploration: creative writing with constraints, puzzle-solving, code architecture planning. On the "Game of 24" puzzle (combine four numbers using arithmetic to make 24), GPT-4 with standard prompting solves 7.3% of cases. With CoT it solves 4.0%. With ToT it solves 74%. Some problems genuinely need search, not just linear chains.

In practice, ToT is complex to implement. You need a custom loop that manages branching, evaluation scoring, and pruning. Most production systems stick with self-consistency because it gets 80% of the benefit at 20% of the complexity.

Why CoT works (the mechanism)

The mechanical explanation is worth understanding because it clarifies when CoT will and won't help.

LLMs are autoregressive: they generate one token at a time, and each new token can attend to all previous tokens. When you ask for a direct answer, the model must compress all reasoning into the hidden states of a single forward pass. That's a computation bottleneck. Complex reasoning requires more "serial computation" than one forward pass provides.

CoT solves this by spreading the computation across multiple generation steps. Each reasoning token is a new forward pass that can attend to all previously generated reasoning tokens. The model essentially gets more compute per problem. This is why CoT helps on hard problems (which need more computation) and doesn't help on easy ones (which fit in a single forward pass).

This helps explain why early CoT results were strongest on very large models. Smaller models may have less capacity or fewer useful patterns for the task, but there is no universal parameter cutoff: some smaller models benefit, while others regress. Measure the target model and task directly.

The connection to reasoning models is direct. o1, o3, and DeepSeek-R1 took this insight and trained it into the model via reinforcement learning. Instead of relying on a prompt to activate reasoning, these models learn when and how to reason on their own. CoT prompting is the manual version. Reasoning models are the automated version. The tradeoff: reasoning models cost more per token but produce more reliable reasoning on harder problems.

Key variants / types

Plan-and-Solve (Wang et al., 2023) deserves a quick mention. Instead of diving straight into reasoning, you prompt the model to first outline a plan, then execute each step. It consistently outperforms zero-shot CoT by 5-8% on complex word problems because the planning step prevents the model from going down a wrong path early.

The progression from zero-shot to Plan-and-Solve reveals something fundamental: the more structure you give the model's reasoning, the better it performs. But each level of structure costs more in prompt engineering effort. Zero-shot is free. Few-shot takes 30 minutes to craft good examples. Self-consistency multiplies your API bill. Choose the level that matches your accuracy requirement and budget.

When to use / when to avoid

So when does CoT actually help? The line is cleaner than most prompt engineering decisions. The rule of thumb: if the answer requires more than one logical step, try CoT. If it doesn't, skip it.

When to use CoT

• Multi-step math and logic: arithmetic word problems, constraint satisfaction, scheduling. CoT is the single biggest accuracy lever for these tasks.
• Code generation and debugging: "Trace through what each line does before writing the fix" catches subtle off-by-one errors that direct prompting misses.
• Multi-hop reasoning: questions that require combining facts from different parts of the context. CoT forces the model to connect the dots explicitly.
• Auditability requirements: when reviewers need to understand a conclusion (medical, legal, financial), return concise assumptions, evidence, and checks; a generated reasoning trace is not a complete audit trail.
• Plan generation: any task where the model needs to outline a plan before executing. "First, identify the components. Then, define their relationships. Finally, determine the data flow."
• Complex extraction: pulling structured data from unstructured text where the model needs to identify entities, resolve references, and infer relationships.

When to avoid CoT

• Simple factual retrieval: "What is the capital of France?" CoT adds latency and occasionally confuses the model into second-guessing itself.
• Classification tasks: sentiment analysis, intent detection. These are single-step pattern matches. CoT adds cost without accuracy gains.
• Models that do not benefit on your eval: early studies found smaller models often gained less from CoT, but there is no universal parameter cutoff. Measure whether the extra tokens improve the target task.
• Latency-critical paths: each reasoning token is a serial generation step. CoT adds 1-5 seconds to typical prompts. If your SLA is 200ms, CoT is not an option.
• Already using reasoning models: o1, o3, and DeepSeek-R1 perform extended CoT internally. Prompting them for explicit reasoning is redundant and wasteful.
• High-throughput batch processing: when you're processing millions of queries and the task is simple enough, the cumulative token overhead of CoT makes it cost-prohibitive.

The key design judgment is knowing when not to use CoT. Applying it to every request adds cost and latency without necessarily improving accuracy.

Real-world examples

PaLM on GSM8K. Standard prompting: 17.9% accuracy. Few-shot CoT: 58.1%. Self-consistency (40 paths): 74.4%. These results from Wei et al. (2022) are a useful historical demonstration, not a guarantee for another model or task.

Code debugging with reasoning traces. A developer tools company added CoT to their AI code review pipeline: "Trace through the function line by line. What does each variable hold after each statement? Now identify the bug." This approach caught 73% of off-by-one and null-reference bugs that direct "find the bug" prompting missed. The reasoning trace also became the explanation shown to developers, so they could verify the AI's logic before accepting the fix.

The pattern here is important: CoT can produce an inspectable intermediate artifact, but it is not free, guaranteed to be faithful, or a substitute for verification.

Medical symptom triage. A healthcare system could use few-shot prompting for classifying patient symptoms into urgency tiers, but any reported improvement would need clinical validation. A concise rationale and supporting evidence can help nurses review a case; an unverified generated reasoning trace must not be treated as the clinical decision or its justification.

Customer support escalation. A SaaS company applied self-consistency (5 paths) to their intent classification pipeline for support tickets. Single-path accuracy was 82%. Self-consistency pushed it to 91%, reducing wrong-team routing by over half. The 5x cost increase was worth it because each misrouted ticket cost $15 in wasted agent time.

SQL query generation. A data platform can add structured intermediate outputs to its natural-language-to-SQL pipeline: identify relevant tables, list join conditions, then construct the query. The SQL should still be parsed, checked against an allowlist, and tested before execution; a generated rationale is useful context for analysts but is not proof that the query is safe or correct.

Limitations and tradeoffs

Token cost scales linearly. A reasoning chain adds 100-500 tokens per query. Self-consistency with 20 samples multiplies that by 20. At scale, this is real money. A system processing 1M queries/day with 300-token reasoning chains at $0.01/1K tokens adds $3,000/day just for the thinking. With self-consistency at 20 samples, that becomes $60,000/day.

Latency is inherently serial. Each reasoning token depends on the previous one. You cannot parallelize CoT generation within a single query. Expect 1-5 seconds of added latency per reasoning chain. For user-facing applications, this may push you past acceptable response times.

Small models may benefit differently. The 100B parameter observation from early studies is approximate and not a universal threshold. Smaller models can gain little or regress when asked to generate intermediate work. If you're using a 7B or 13B model, compare direct prompting, concise structured outputs, and fine-tuning on reasoning examples on your own eval set.

Hallucinated reasoning. CoT doesn't guarantee correct reasoning. The model can produce a chain that looks logical but contains a subtle error (wrong arithmetic, false premise). A rationale may help a reviewer inspect assumptions, but it can also provide an elaborate justification for a wrong answer. Treat it as untrusted output and validate the result independently.

Reasoning models may make explicit CoT redundant. Some reasoning models are trained to spend additional internal computation before answering. Their thinking tokens may be hidden or billed separately, and prompting for a second visible chain can add cost without improving the result. Check the provider's behavior and evaluate concise summaries or structured outputs instead.

The fundamental tension: CoT trades tokens and latency for accuracy. The right choice depends on where your task sits on the "simple lookup vs. complex reasoning" spectrum. Most teams over-apply CoT to tasks that don't need it, wasting money without improving results. If you're unsure, run a quick A/B test: 50 queries with CoT, 50 without. If accuracy doesn't change, skip it.

How this shows up in interviews

When to bring it up

Mention CoT when a design involves multi-step reasoning in an LLM pipeline: a math tutor, code review assistant, medical triage system, or any application where the model must combine intermediate results rather than retrieve one fact. Also compare it with reasoning models when cost, latency, or output-policy constraints matter.

Depth calibration

• Junior: knows zero-shot CoT exists, can explain why "let's think step by step" helps.
• Senior: can compare zero-shot vs few-shot CoT, explain self-consistency, describe why model size and task difficulty affect gains, and identify when CoT wastes tokens.
• Staff: discusses Tree of Thoughts, Plan-and-Solve, connects CoT to reasoning models as "learned CoT," articulates the cost-accuracy tradeoff curve and can sketch a system that adaptively applies CoT only when needed. Knows the Game of 24 and GSM8K benchmark numbers.

Interview Q&A

Common interview mistakes

Test your understanding

Quick recap

1. Chain-of-thought prompting asks an LLM to generate intermediate work, which can improve some multi-step tasks without changing model weights.
2. Zero-shot CoT ("Let's think step by step") is the simplest entry point; its value depends on the model, task, and verification cost.
3. Few-shot CoT provides worked examples that control the reasoning format, achieving 58.1% on GSM8K versus 17.9% with standard prompting (PaLM 540B).
4. Self-consistency samples multiple reasoning paths and takes majority vote, adding 10-20 percentage points for an N-times cost multiplier.
5. CoT wastes tokens on simple factual retrieval, classification, and small models. Apply it selectively to multi-step reasoning tasks.
6. Some reasoning models perform additional internal computation, making prompt-level CoT redundant in some cases; use concise observable summaries and follow the provider's output policy.
7. The engineering decision: classify your queries, apply CoT only where reasoning is needed, and graduate to reasoning models for the hardest problems.
8. The escalation ladder: zero-shot CoT (free) → few-shot CoT (prompt tokens) → self-consistency (N times cost) → reasoning models (5-10x per token).

Related concepts

• Few-shot prompting - The foundation CoT builds on. Few-shot CoT combines example-based prompting with explicit reasoning chains.
• Reasoning models - The next evolution. Models like o1 internalize CoT through reinforcement learning, producing better reasoning at higher per-token cost.
• Context engineering - CoT is one tool in the broader context engineering toolkit. Understanding how to structure the entire prompt window matters.
• Large language models - CoT only works on large models. Understanding why requires knowing how scale affects emergent capabilities.