Yes.

It sounds a bit like learning to cook by first making toast, but there’s a surprisingly good reason for it.

Most programming languages, whether it’s Python, JavaScript, Rust, or your own language that only exists in a folder named interpreter_v7_final_final, all follow a similar pipeline:

Source Code
     │
     ▼
Tokenizer (Lexer)
     │
     ▼
Parser
     │
     ▼
Abstract Syntax Tree (AST)
     │
     ▼
Interpreter
     │
     ▼
Result

A calculator is the smallest project that lets us build every piece of this pipeline without worrying about variables, functions, loops, or the hundreds of edge cases that make language design… entertaining.

By the time our calculator can correctly evaluate:

(2 + 3) * 4 - 5 / 5

we’ll already have learned a large chunk of what goes into writing an interpreter.

So, let’s begin.

What Are We Actually Building?

Our first version won’t calculate anything.

That’s right.

The calculator will proudly refuse to do any math.

Instead, it will read text and convert it into tokens.

Consider this expression:

12 + (5 * 8)

To us, it’s obviously a mathematical expression.

To a computer?

It’s just a stream of characters.

Our first task is to group those characters into meaningful pieces.

These pieces are called tokens.

The expression above should become:

NUMBER(12)
PLUS
LPAREN
NUMBER(5)
STAR
NUMBER(8)
RPAREN
EOF

Notice something interesting.

We’re not adding anything.

We’re not multiplying anything.

We’re simply saying:

“That’s a number.”

“That’s a plus sign.”

“That’s an opening parenthesis.”

Think of it as translating a sentence into words before trying to understand its grammar.

Meet the Tokenizer (Lexer)

The tokenizer, also known as the lexer, is the first stage of almost every compiler and interpreter.

Its job is remarkably simple:

Read characters.
Group them together.
Produce tokens.

Nothing more.

It doesn’t know mathematics.

It doesn’t know operator precedence.

It doesn’t know whether an expression is valid.

It just reads.

Imagine someone hands you this:

123+45

The tokenizer walks through it from left to right.

123+45
^

It sees a digit.

Instead of immediately producing a token, it keeps reading.

123+45
^^^

Still digits.

Keep going.

Eventually it reaches:

123+45
   ^

The next character is +, so the tokenizer knows the number has ended.

It produces:

NUMBER(123)

Then it moves on.

123+45
   ^

It sees +.

That’s easy.

Produce:

PLUS

Move again.

123+45
    ^

Read another number.

Produce:

NUMBER(45)

Finally:

EOF

EOF stands for End Of File, although in our case it’s really just End Of Input.

Designing Our Tokens

Every token has two important pieces of information.

Its type:

NUMBER
PLUS
STAR
LPAREN

and sometimes a value.

For example:

NUMBER(42)

The type tells us this token represents a number.

The value tells us which number.

A plus sign doesn’t need a value.

There is only one kind of +.

A number, however, could be 1, 42, 999, or 123456789.

That’s why our Token class stores both.

Token(
    type=TokenType.NUMBER,
    value=42
)

Walking Through the Source Code

The tokenizer keeps track of just three things.

The source code

self.source

This is simply the input string.

12 + 5

The current position

self.position

Initially:

12 + 5
^
position = 0

After reading one character:

12 + 5
 ^
position = 1

Eventually:

12 + 5
      ^

The tokenizer has reached the end.

The current character

Instead of constantly asking:

self.source[self.position]

we keep the current character handy.

self.current_char

Initially:

12 + 5
^

current_char == "1"

After advancing:

current_char == "2"

Eventually:

current_char is None

That None tells us we’ve reached the end of the input.

Tiny Methods, Big Difference

Rather than stuffing everything into one giant loop, our tokenizer is made up of small methods.

`advance()`

Moves to the next character.

123+45
^

↓

123+45
 ^

`peek()`

Looks ahead without moving.

This becomes incredibly useful later when we introduce operators like:

==
>=
<=
!=

Sometimes you need to know what’s coming next before deciding what you’re looking at.

`skip_whitespace()`

Humans love spaces.

Programming languages mostly don’t care.

These are identical:

2+3

2 + 3

2         +         3

Instead of creating “space tokens,” we simply skip them.

Less work.

Fewer headaches.

Everyone wins.

`read_number()`

This is our first method that actually creates a token.

Given:

it keeps reading digits until it encounters something that isn’t one.

Then it produces:

NUMBER(12345)

Simple.

Elegant.

Surprisingly satisfying.

The Heart of the Lexer

The tokenizer repeatedly asks itself one question:

“What am I looking at?”

If it’s whitespace…

Skip it.

If it’s a digit…

Read an entire number.

If it’s +…

Produce a PLUS token.

If it’s (…

Produce an LPAREN.

If it’s something unexpected…

Raise an error.

That’s it.

No recursion.

No syntax trees.

No complicated algorithms.

Just careful observation.

Testing Our Lexer

Suppose we run:

lexer = Lexer("12 + (5 * 8) - 3")

tokens = lexer.tokenize()

for token in tokens:
    print(token)

We should get:

NUMBER(12)
PLUS
LPAREN
NUMBER(5)
STAR
NUMBER(8)
RPAREN
MINUS
NUMBER(3)
EOF

Mission accomplished.

Our calculator still can’t calculate.

And that’s perfectly fine.

We’ve successfully taught it to read, which is the first skill every interpreter needs.

What’s Next?

Now that we have a stream of tokens, the next challenge is making sense of them.

We’ll build a parser, whose job is to transform this:

NUMBER(2)
PLUS
NUMBER(3)
STAR
NUMBER(4)

into a tree that understands operator precedence.

Only after we have that tree can we finally teach our calculator to perform arithmetic.

One small step for a calculator.

One giant leap toward writing your own programming language.

Creating a Tokenizer