Looking further back

The problem

I can learn my own numbers now. But I still write words like “sathe”.

Right now your model:

• sees only the last letter, so it writes “sathe”.
• would need 38,416 boxes to count the last 3 letters, and almost all would stay empty.

Can your model look at the last 3 letters without a giant table?

What if, instead of one giant table, it used small ones?

Three small tables

Your model will look at the last 3 letters. In “cat”: third-last c, second-last a, last t.

It uses 3 small tables, one per position, each 14 × 14 like your model's one-letter table:

• third-last table: looks at c
• second-last table: looks at a
• last table: looks at t

Each letter picks one row in its own table. Each row gives a score for every next letter.

It ignores the letters in between: c _ _ → ?, _ a _ → ?, _ _ t → ?.

Three rows. How do they make one guess?

Add the rows

Step a) Predict

Your model adds the 3 rows, score by score. Then softmax turns the sums into chances: a higher score gives a higher chance.

The scores start random, so the chances are random too.

“cat” and “sat” use two of the same rows. Why does that matter?

Shared rows

“cat” and “sat” differ only in the first letter. So they add two of the very same numbers: those are shared.

Learning from “cat” moved its 3 numbers. Two of them are in “sat” too, so “sat” improved without being trained. One row helps every group that uses it.

A giant table gives every group its own row, so “cat” would teach “sat” nothing. It also needs far more numbers:

Try it on new numbers.

✏️ Practice 5.1

New numbers, same ideas.
Remember: a table holds rows × columns numbers.
A group uses one row from each table, picked by the letter at that position.
chance = e^score ÷ (sum of all e^score)

Could we fill these 588 numbers just by counting?

Why not count them?

Can we fill the boxes by counting how often each letter comes next? Try “mat”: in our text it is always followed by a full stop.

Counting goes wrong because the green boxes are shared: they also counted “cat ” and “sat ”, so they give space big numbers, even for “mat”. And counting never checks the answer.

Learning checks, and fixes the box only “mat” uses: m _ _ → . goes up to 4.87, m _ _ → ␣ down to −2.16. Now full stop wins. A count can never be negative.🤔

How does learning use the text? Watch one step.

One learning step: 44 checks, one move

A 3-letter window slides along our text. Each window is an example: the 3 letters, and the letter that comes next.

One step has four parts:
a) Predict: every example gets its chances.
b) Measure: the loss averages the 44 examples' losses (next page).
c) Find directions: every example finds a gradient (chance − goal) for every box in the 3 rows it used: the right letter's box goes up, the other 13 go down.
d) Move: add up each box's gradients (opposite ones cancel), divide by 44 for the average, then move every box once, by
new = old − 10 × average gradient
While we go through the examples, the boxes stay where they are.

That was one step. Next: repeat it 300 times.

Let it learn

The step from the last page, repeated. Does your model end up agreeing with our text?

588 numbers reach almost the same chances as the giant table. Train more and they get even closer.

Try “hat”: it isn't in our text, yet your model guesses a space, thanks to the rows it shares with “cat”, “sat” and “mat”.

Try it on new numbers.

✏️ Practice 5.2

One box, used by several examples.
Remember: average gradient = sum of gradients ÷ number of examples
new = old − learning rate × average gradient

Now let it write.

Write new text

Your model writes one letter at a time, picked by chance. The chances now come from the last 3 letters.

No more “sathe”: after “sat”, your model sees all of “sat”.

Here's all of it in Python.

The code

Guess what each print shows. Then press ▶ Run.

import torch
import torch.nn.functional as F

text = "the cat sat on the mat. the dog sat on the log."
unique_chars = sorted(set(text))
vocab_size = len(unique_chars)
char_to_number = {}
number_to_char = {}
for number, char in enumerate(unique_chars):
    char_to_number[char] = number
    number_to_char[number] = char

# Examples: the last 3 letters, and the next one
third_last = []
second_last = []
last = []
next_chars = []
for i in range(len(text) - 3):
    third_last.append(char_to_number[text[i]])
    second_last.append(char_to_number[text[i + 1]])
    last.append(char_to_number[text[i + 2]])
    next_chars.append(char_to_number[text[i + 3]])
third_last = torch.tensor(third_last)
second_last = torch.tensor(second_last)
last = torch.tensor(last)
next_chars = torch.tensor(next_chars)
print("Examples:", len(next_chars))

# The model: 3 small tables of random scores
torch.manual_seed(42)
table_third_last = torch.randn((vocab_size, vocab_size), requires_grad=True)
table_second_last = torch.randn((vocab_size, vocab_size), requires_grad=True)
table_last = torch.randn((vocab_size, vocab_size), requires_grad=True)
print("Numbers in the model:", 3 * vocab_size * vocab_size)

# Learn: predict, measure, find directions, move, for all 3 tables
learning_rate = 10
for step in range(300):
    scores = table_third_last[third_last] + table_second_last[second_last] + table_last[last]   # a) predict
    loss = F.cross_entropy(scores, next_chars)                        # b) measure
    table_third_last.grad = None
    table_second_last.grad = None
    table_last.grad = None
    loss.backward()                                                   # c) find directions
    table_third_last.data = table_third_last.data - learning_rate * table_third_last.grad            # d) move
    table_second_last.data = table_second_last.data - learning_rate * table_second_last.grad
    table_last.data = table_last.data - learning_rate * table_last.grad
    if step % 50 == 0:
        print("Step", step, "loss:", round(loss.item(), 3))

# Write 40 new letters, starting from "the"
torch.manual_seed(42)
output = "the"
for step in range(40):
    scores = table_third_last[char_to_number[output[-3]]] + table_second_last[char_to_number[output[-2]]] + table_last[char_to_number[output[-1]]]
    chances = F.softmax(scores, dim=0)
    picked = torch.multinomial(chances, num_samples=1).item()
    output = output + number_to_char[picked]
print(output)

$ python lesson5.py
Examples: 44
Numbers in the model: 588
Step 0 loss: 4.082
Step 50 loss: 0.199
Step 100 loss: 0.184
Step 150 loss: 0.179
Step 200 loss: 0.177
Step 250 loss: 0.175
the mat. the cat sat on the dog sat on the 

▶ Run it in Google Colab

Colab runs Python with PyTorch in your browser for free. You just need a Google account.

Time to check what you learned.

Lesson 5 test

7 questions, one per idea. Get them all right to pass.

What your model still can't do.

Too many numbers

Three tables need 3 × letters × letters numbers. Pick how many different letters there are.

Every letter has a row in each of the 3 tables (third-last, second-last, last). So “a” has 3 rows, learned separately.

My tables explode when there are more letters!

Mark lesson as done