Text into numbers

The problem

I see letters everywhere, but they mean nothing to me. Can you turn them into numbers?

An LLM reads text, but tensors hold only numbers. How do we turn text into numbers?

print("h" + 5)

Letters can't do math. So we'll give each letter a number.

First, find every different character in the text.

Find the letters

This list is the vocabulary. It's every character the model knows, like these 14.

The blank tile is the space. It's a character too.

Each character has a number now. Let's turn whole words into numbers.

Encode and decode

Vocabulary: the 14 different characters of our text.

Encode: text → numbers. Decode: numbers → text.

Try it on new text.

✏️ Practice 2.1

New text, same ideas.
Remember: the vocabulary is every different character, sorted, numbered from 0. The space sorts first, then “.”, then the letters.
encode swaps each character for its number.

Here's all of it in Python.

The code

Guess what each print shows. Then press ▶ Run.

import torch

text = "the cat sat on the mat. the dog sat on the log."

# Find every different character, sorted
unique_chars = sorted(set(text))
print(unique_chars)

# Give each one a number, and the reverse
char_to_number = {}
number_to_char = {}
for number, char in enumerate(unique_chars):
    char_to_number[char] = number
    number_to_char[number] = char

# Text -> numbers
def encode(some_text):
    numbers = []
    for char in some_text:
        numbers.append(char_to_number[char])
    return numbers

# Numbers -> text
def decode(numbers):
    result = ""
    for number in numbers:
        result = result + number_to_char[number]
    return result

print(encode("cat"))

print(decode([4, 11, 6]))

# The whole text as a tensor
print(torch.tensor(encode(text)))

$ python lesson2.py
[' ', '.', 'a', 'c', 'd', 'e', 'g', 'h', 'l', 'm', 'n', 'o', 's', 't']
[3, 2, 13]
dog
tensor([13,  7,  5,  0,  3,  2, 13,  0, 12,  2, 13,  0, 11, 10,  0, 13,  7,  5,
         0,  9,  2, 13,  1,  0, 13,  7,  5,  0,  4, 11,  6,  0, 12,  2, 13,  0,
        11, 10,  0, 13,  7,  5,  0,  8, 11,  6,  1])

▶ Run it in Google Colab

Colab runs Python with PyTorch in your browser for free. You just need a Google account.

Quick check: did it all make sense?

Lesson 2 test

5 questions, one for each idea in this lesson. Answer them all right to pass. A wrong answer tells you why, and which page to look at again.

What your model still can't do.

Guess the next letter

Now text is numbers: "cat" → [3, 2, 13].

An LLM's whole job is to guess what comes next:

I can read text as numbers now! But how do I guess which letter comes next?

Mark lesson as done