Counting what comes next

The problem

Time to write my first words! Let's count which letter usually comes next.

Text is numbers now. How do we make a first guess at the next character?

Every LLM does just this. Look at the text so far. Guess what comes next.

Let's count which letter follows which.

Count the pairs

Move along the text, two characters at a time. Count each pair in a table.

Row = this character. Column = the next one. This table is counts in the code.

Counts aren't chances yet. Let's turn each row into chances.

Counts into chances

Divide each count by the row's total. Now each row adds up to 1.

These are probabilities: how likely each next character is.

Try the row for “h”. Only one character ever follows it.

Your model has chances now. Let's use them to write.

Write new text

Spin a weighted wheel. It's like a dice where likely characters get bigger faces: a bigger slice means a bigger chance.

Your model writes one character at a time.🤔

Try it on new text.

✏️ Practice 3.1

New text, same ideas. A pair is two characters side by side.
Remember: chance = count ÷ row total
The wheel gives each character a slice as big as its chance.

Here's all of it in Python.

The code

Guess what each print shows. Then press ▶ Run.

import torch

text = "the cat sat on the mat. the dog sat on the log."

unique_chars = sorted(set(text))
vocab_size = len(unique_chars)
char_to_number = {}
number_to_char = {}
for number, char in enumerate(unique_chars):
    char_to_number[char] = number
    number_to_char[number] = char

# Count pairs. Row = this character, column = next one.
counts = torch.zeros((vocab_size, vocab_size))
for i in range(len(text) - 1):
    row = char_to_number[text[i]]
    column = char_to_number[text[i + 1]]
    counts[row, column] = counts[row, column] + 1

# Counts into chances: divide by each row's total
row_totals = counts.sum(dim=1, keepdim=True)
probabilities = counts / row_totals

t_row = probabilities[char_to_number['t']]
for number in range(vocab_size):
    if t_row[number] > 0:
        print(repr(number_to_char[number]), round(t_row[number].item(), 3))

# Write new text: spin the weighted wheel 40 times
torch.manual_seed(42)
current = char_to_number['t']
output = 't'
for step in range(40):
    picked = torch.multinomial(probabilities[current], num_samples=1)
    current = picked.item()
    output = output + number_to_char[current]
print(output)

$ python lesson3.py
' ' 0.375
'.' 0.125
'h' 0.5
t on dog mat mat. sathe log the mat cat t

▶ Run it in Google Colab

Colab runs Python with PyTorch in your browser for free. You just need a Google account.

Quick check: did it all make sense?

Lesson 3 test

6 questions, one for each idea in this lesson. Answer them all right to pass. A wrong answer tells you why, and which page to look at again.

What your model still can't do.

Why not count more?

Your model wrote “sathe” because it only sees the last letter. Let it see more.

Looking further back makes the table explode, and almost every box stays empty.

So what can your model do instead?

What now?

There's a smarter kind of model that can look further back with far fewer boxes. You'll build it in Lesson 5.

I wrote something! t on dog mat mat. sathe log the mat cat t Hmm… “sathe”? I only look at the last letter.

Mark lesson as done