Lesson 4: Chain-of-Thought: Making AI Show Its Reasoning

Learning goals:

• Understand how chain-of-thought works and why it helps
• Learn to guide AI reasoning with "let's think step by step"
• Recognize which task types are a good fit for CoT

Prerequisites: << Lesson 3: Few-Shot Learning | Next: Lesson 5 >>

When AI Gives the Right Answer but the Wrong Reasoning

You ask AI a math word problem and it answers "42." You check it, and the answer is right. But swap in a similar problem and the answer comes out wrong. What happened? The model may have guessed, or followed a broken reasoning path that happened to land on the right number.

For tasks that need multi-step reasoning (math problems, logic puzzles, complex analysis), having AI show its reasoning steps is more reliable than asking for the answer directly^{1 2}. That's the core idea behind chain-of-thought (CoT) prompting: break the conclusion into intermediate steps you can actually check.

What Is Chain-of-Thought

Chain-of-thought (CoT) prompting is a technique that has the model lay out intermediate reasoning steps before it gives a final answer¹. In the words of the research, chain-of-thought prompting "enables complex reasoning capabilities through intermediate reasoning steps"¹. The intuition is simple: "Chain of Thought prompting encourages the model to think through the problem in a step-by-step manner, which is supposed to mimic how humans break down complex problems"². It's the same thing you do when you list equations and intermediate results on scratch paper to solve a math problem.

Compare the two approaches:

Answering Directly (Error-Prone)

❌ Without CoT:

Problem: A coffee shop sold 23 cups of coffee in the morning, 17 at noon, and twice the morning's amount in the afternoon. How many cups did it sell for the whole day?
Answer:

AI might just output:

86 cups

If the answer is wrong, you have no idea which step went off.

Showing the Reasoning Steps (More Reliable)

✅ With CoT:

Problem: A coffee shop sold 23 cups of coffee in the morning, 17 at noon, and twice the morning's amount in the afternoon. How many cups did it sell for the whole day?
Let's think step by step:

AI outputs:

Let's think step by step:
1. Morning sales: 23 cups2. Noon sales: 17 cups3. Afternoon is twice the morning: 23 × 2 = 46 cups4. Day total: 23 + 17 + 46 = 86 cups
Answer: 86 cups

Every step is visible, so if one is wrong you can spot it right away and fix the prompt.

Research shows that on tasks involving multi-step reasoning, CoT lifts accuracy substantially¹; on some tasks, pairing CoT with worked demonstrations raises accuracy by up to another 28.2%².

Two Ways to Use CoT

Method 1: Zero-Shot CoT (Simplest)

Add one line to the end of your prompt: "let's think step by step"^{1 2}.

Problem: If 5 workers need 5 days to finish a project, how many days do 10 workers need?
Let's think step by step:

That single line triggers the model's step-by-step reasoning mode. You don't provide any examples; the model breaks the problem down on its own.

Method 2: Few-Shot CoT (More Controllable)

Give one or two examples that include the full reasoning process^{1 2}.

Example:Problem: A car travels at 60 km/h. How far does it go in 3 hours?Reasoning:- Speed = 60 km/h- Time = 3 hours- Distance = speed × time = 60 × 3 = 180 kmAnswer: 180 km
Now solve this problem:Problem: A car travels at 80 km/h for 2.5 hours, then at 60 km/h for another 1 hour. How far does it travel in total?Reasoning:

The example shows the reasoning format and how detailed the steps should be, and the model imitates that style.

What Tasks Suit CoT

CoT is best for tasks that need multi-step reasoning²:

✅ Math and logic reasoning

• Word problems
• Ratio and proportion calculations
• Multi-step formula derivations

✅ Causal analysis

• "Why does X cause Y?"
• "What's the root cause of this bug?"

✅ Planning and decisions

• "Should I do A first or B first?"
• "What are the pros and cons of this approach?"

✅ Code debugging

• "Why does this code fail?"
• "Which step is the performance bottleneck?"

Where it doesn't fit:

❌ Simple fact lookups: "When was Python first released?" — no reasoning needed ❌ Creative writing: poems, stories — reasoning steps break the creative flow ❌ Format conversion: JSON → CSV — a mechanical operation, no reasoning needed

Rule of thumb: if you'd reach for scratch paper to list out steps when doing the task yourself, it's a good fit for CoT.

Worked Examples: Analyzing Problems with CoT

Scenario 1: Code Performance Analysis

Without CoT (conclusion you can't trust):

Does this code have a performance problem?
```pythondef find_duplicates(items):    duplicates = []    for i in range(len(items)):        for j in range(i + 1, len(items)):            if items[i] == items[j] and items[i] not in duplicates:                duplicates.append(items[i])    return duplicates```
Answer: Yes, it has a performance problem

Just a conclusion, with no reason why.

With CoT (clear reasoning):

Analyze this code's performance problem. Explain step by step:1. Identify the loop structure2. Compute the time complexity3. Point out the specific bottleneck4. Suggest an optimization
```pythondef find_duplicates(items):    duplicates = []    for i in range(len(items)):        for j in range(i + 1, len(items)):            if items[i] == items[j] and items[i] not in duplicates:                duplicates.append(items[i])    return duplicates```

AI shows:

Performance analysis:
1. Loop structure:   - Outer loop: iterates over each element, O(n)   - Inner loop: for each element, iterates over the remaining ones, O(n)   - Overall: nested loops, O(n²)
2. Time complexity:   - Main loops: O(n²)   - The `items[i] not in duplicates` check: O(m) worst case, where m is the number of duplicates found so far   - Total complexity: O(n² × m)
3. Performance bottleneck:   - The nested loops get very slow on large inputs (n > 1000)   - The `not in` linear scan over a list slows it down further
4. Optimization:   - Use a set instead of a list to store duplicates, dropping the check from O(m) to O(1)   - Or use a Counter to find all elements appearing more than once in a single pass, dropping it to O(n)

Every reasoning step is checkable, so if one is wrong you can see it immediately.

Scenario 2: A Logic Puzzle

Problem:

Three people — A, B, and C — enter a competition. We know:- A is not first place- B is not last place- C is neither first nor last place
What are their rankings?
Let's reason step by step:

AI output:

Let's reason step by step:
1. Start from C's condition:   - C is not first and not last   - So C must be second
2. Pin down A and B:   - A is not first   - B is not last   - C is already second, leaving first and third for A and B
3. Assign the remaining ranks:   - A is not first, so A is third   - B is not last (third), so B is first
Answer: B first, C second, A third

Laying out this step-by-step elimination makes it much easier to check whether the logic is airtight.

Common CoT Patterns

Pattern 1: Problem Decomposition

Break a complex problem into a few sub-problems.

Problem: A software project has 5 engineers, each working 6 hours a day, and needs 120 person-days of work. If it scales up to 8 engineers, how many days sooner can it finish?
Let's compute step by step:1. How many days does the current setup need?2. How many days after adding people?3. How many days sooner?

Pattern 2: Hypothesis Testing

List the hypotheses, then check them one by one.

This code throws "TypeError: unsupported operand type(s)". What are the possible causes?
Let's check the possible causes one by one:1. Check the types of the variables on each side of the operator2. Check whether a None value is involved in the operation3. Check whether a string and a number are being mixed

Pattern 3: Arguing Both Sides

List the reasons for and against, then reach a conclusion.

Should we use a microservices architecture or a monolith?
Let's analyze it from two angles:
Reasons for microservices:1. ...2. ...
Reasons for a monolith:1. ...2. ...
Overall judgment:Based on the current team size and project complexity, I recommend...

The Limits of CoT

CoT isn't a cure-all. It has a few limits:

1. It adds length and time

Showing reasoning steps makes the output longer, burns more tokens, and takes longer to respond.

When it's worth it: on complex tasks, spending 2-3x the tokens to gain accuracy is a good trade. When it isn't: on simple tasks ("what day is it today"), the reasoning steps are pure waste.

2. The reasoning steps can themselves be wrong

The steps AI shows can look reasonable but still have a hole in the logic. You still need to check whether the reasoning is correct.

The value of CoT is that it makes errors visible. When AI answers directly, the error hides in a black box; when the steps are shown, the error surfaces at some step and is easier to find and fix.

3. It doesn't suit tasks that rely on intuition

Some tasks (creative writing, evaluating art) lean on an overall feel, and forcing them into steps breaks the whole.

Recap

Chain-of-thought prompting has AI show its reasoning steps instead of jumping straight to an answer. By adding "let's think step by step" to a prompt, or by providing examples that include the reasoning, you can substantially raise accuracy on multi-step reasoning tasks.

CoT fits math reasoning, logic analysis, causal inference, complex decisions — anything that needs multiple steps of thought. It makes the reasoning visible and checkable. It does add output length, but on complex tasks the payoff far outweighs the cost.

The next lesson covers how to debug and improve prompts: when AI's output doesn't match what you expected, how to systematically find the problem and fix it.

Next lesson Debugging and Improving Prompts >>