Mathematics for Machine Learning: Rebuilding the Foundations from Scratch¶

Part 1: Algebra through Logarithms

Before we dive into machine learning, neural networks, or deep learning, we need to build a solid mathematical foundation.

When people first approach machine learning, they often jump straight into algorithms, libraries, and code. They learn how to train a model using Python, call a function from scikit-learn, or build a neural network with PyTorch. They might even get impressive results.

But there's a problem: using a model is not the same as understanding how it works.

To understand machine learning deeply, we need to understand the mathematics behind it. That means starting with the fundamentals, even if some of these concepts seem familiar.

In this article, we'll rebuild our mathematical foundation step by step, starting with algebra and building through exponents and logarithms.

1. Variables, constants, and coefficients¶

Let's begin with a simple expression:

\[ y = 3x + 2 \]

What does this mean?

\(x\) is a variable. Its value can change.
\(y\) is another variable whose value depends on \(x\).
\(3\) is a coefficient. It multiplies \(x\).
\(2\) is a constant in this expression.

Suppose \(x=4\). We substitute 4 wherever we see \(x\):

\[ y = 3(4)+2 \]

\[ y = 12+2=14 \]

Therefore, \(y=14\).

If \(x=5\), then:

\[ y=3(5)+2=17 \]

Notice something interesting: when \(x\) increases from 4 to 5, \(y\) increases from 14 to 17.

The input increases by 1, but the output increases by 3.

Why? Because every increase of 1 in \(x\) adds another 3 to the expression \(3x+2\).

This simple relationship will become important when we study linear models in machine learning.

2. Expressions and equations are not the same thing¶

Consider these two mathematical statements:

\[ 3x+2 \]

\[ 3x+2=14 \]

The first is an expression. It represents a calculation, but it doesn't claim that the result equals a particular value.

The second is an equation. It states that the expression on the left equals 14.

This distinction matters because we use expressions to calculate values and equations to find unknown values.

For example, given:

\[ 3x+2=14 \]

We want to find \(x\).

First, subtract 2 from both sides:

\[ 3x=12 \]

Then divide both sides by 3:

\[ x=4 \]

The answer is \(x=4\).

The key principle is that an equation remains balanced when we perform the same valid operation on both sides.

Think of an equation as a balanced scale. If you remove the same weight from both sides, the scale remains balanced. If you divide both sides by the same nonzero number, equality is preserved.

This idea is fundamental to solving equations, rearranging formulas, and understanding many mathematical derivations.

3. The order of operations¶

Consider the expression:

\[ 2+3\times4 \]

Should we add first or multiply first?

According to the standard order of operations, multiplication comes before addition:

\[ 2+3\times4=2+12=14 \]

If we calculate from left to right without considering the rules, we might get:

\[ (2+3)\times4=20 \]

That is a different expression because the parentheses change the order of calculation.

The usual order is:

Parentheses and other grouping symbols.
Exponents and roots.
Multiplication and division, from left to right.
Addition and subtraction, from left to right.

Let's try another example:

\[ 2^3+4\times2 \]

Calculate the exponent first:

\[ 8+4\times2 \]

Then multiplication:

\[ 8+8=16 \]

The answer is 16.

Why does this matter for machine learning? Mathematical formulas can contain several operations, and evaluating them in the wrong order can produce incorrect results. Computers follow explicit rules, and so must we.

4. Negative numbers and fractions¶

Negative numbers and fractions are not separate mathematical worlds. They follow the same basic arithmetic rules, with a few details worth understanding carefully.

Negative numbers¶

When multiplying two numbers with the same sign, the result is positive:

\[ (-3)(-4)=12 \]

When multiplying numbers with different signs, the result is negative:

\[ (-3)(4)=-12 \]

The same sign rules apply to division.

These rules will matter later when we calculate changes in model parameters, work with slopes, and interpret derivatives.

Fractions¶

A fraction represents division:

\[ \frac{3}{4}=3\div4=0.75 \]

To multiply fractions, multiply the numerators and denominators:

\[ \frac{2}{3}\times\frac{3}{4} =\frac{6}{12} =\frac{1}{2} \]

To divide by a fraction, multiply by its reciprocal:

\[ \frac{2}{3}\div\frac{4}{5} =\frac{2}{3}\times\frac{5}{4} =\frac{10}{12} =\frac{5}{6} \]

A reciprocal is simply what we get when we swap the numerator and denominator of a nonzero fraction.

Fractions will become especially important when we study ratios, probabilities, averages, and the formulas used to calculate gradients.

5. Exponents: a compact way to express multiplication¶

Consider:

\[ 2\times2\times2\times2 \]

We can write this more compactly as:

\[ 2^4=16 \]

The number 2 is the base, and 4 is the exponent. The exponent tells us how many times the base appears as a factor.

Let's explore the main exponent rules.

Rule 1: Multiplying powers with the same base¶

\[ x^a\times x^b=x^{a+b} \]

Why do we add the exponents?

Consider:

\[ x^2\times x^3 \]

Expanding the powers gives:

\[ (x\times x)(x\times x\times x) \]

There are five factors of \(x\), so:

\[ x^2\times x^3=x^5 \]

We add the exponents because we're counting how many times the same base appears in the product.

Rule 2: Dividing powers with the same base¶

For a nonzero base:

\[ \frac{x^a}{x^b}=x^{a-b} \]

For example:

\[ \frac{x^5}{x^2}=x^{5-2}=x^3 \]

We can see why by expanding the fraction:

\[ \frac{x\times x\times x\times x\times x}{x\times x} \]

Two factors cancel, leaving three factors of \(x\).

Rule 3: Raising a power to another power¶

\[ (x^a)^b=x^{ab} \]

For example:

\[ (x^2)^3=x^6 \]

This means multiplying \(x^2\) by itself three times:

\[ x^2\times x^2\times x^2=x^{2+2+2}=x^6 \]

We multiply the exponents because each of the three groups contains two factors of \(x\).

Rule 4: Zero exponents¶

For any nonzero \(x\):

\[ x^0=1 \]

Why?

Using the division rule:

\[ \frac{x^3}{x^3}=x^{3-3}=x^0 \]

But any nonzero number divided by itself equals 1. Therefore:

\[ x^0=1 \]

This rule applies to every nonzero base. The expression \(0^0\) requires separate treatment and is not covered by this rule.

Rule 5: Negative exponents¶

A negative exponent represents a reciprocal:

\[ x^{-a}=\frac{1}{x^a} \]

For example:

\[ x^{-3}=\frac{1}{x^3} \]

We can derive this using the division rule:

\[ \frac{x^2}{x^5}=x^{2-5}=x^{-3} \]

Canceling the common factors in the fraction gives:

\[ \frac{x^2}{x^5}=\frac{1}{x^3} \]

Therefore:

\[ x^{-3}=\frac{1}{x^3} \]

A negative exponent does not mean the value itself must be negative. It tells us to take the reciprocal.

Rule 6: Fractional exponents¶

A fractional exponent can represent a root:

\[ x^{1/2}=\sqrt{x} \]

For example:

\[ 25^{1/2}=\sqrt{25}=5 \]

Similarly:

\[ x^{1/3}=\sqrt[3]{x} \]

So:

\[ 8^{1/3}=\sqrt[3]{8}=2 \]

Fractional exponents let us express roots and powers within the same mathematical framework.

Why exponents matter in machine learning¶

Exponent rules appear throughout machine learning and deep learning. We encounter them when working with exponential functions, probability distributions, loss functions, and numerical calculations.

For example, the exponential function \(e^x\) plays an important role in the sigmoid function:

\[ \sigma(x)=\frac{1}{1+e^{-x}} \]

We'll return to this expression when we study activation functions in neural networks.

6. Logarithms: answering the opposite question¶

Exponents ask us to calculate a power. Logarithms ask us to determine the exponent.

Suppose:

\[ 2^3=8 \]

We know the base is 2, the exponent is 3, and the result is 8.

A logarithm reverses this relationship:

\[ \log_2(8)=3 \]

Read this as: "The logarithm of 8 to base 2 is 3."

The formal definition is:

\[ \log_b(x)=y \iff b^y=x \]

In plain English, the logarithm tells us what exponent we need to raise the base to in order to obtain a particular number.

Example 1: Converting a logarithm into exponential form¶

Suppose:

\[ \log_3(81)=? \]

We ask: to what power must we raise 3 to get 81?

Since:

\[ 3^4=81 \]

We have:

\[ \log_3(81)=4 \]

Example 2: Converting exponential form into logarithmic form¶

Suppose:

\[ 5^3=125 \]

The equivalent logarithmic expression is:

\[ \log_5(125)=3 \]

Both statements describe the same relationship.

Rule 1: The logarithm of a product¶

For positive \(M\) and \(N\), with a valid logarithm base:

\[ \log_b(MN)=\log_b(M)+\log_b(N) \]

For example:

\[ \log_2(8\times4) =\log_2(8)+\log_2(4) \]

Since \(\log_2(8)=3\) and \(\log_2(4)=2\):

\[ \log_2(32)=3+2=5 \]

The rule works because multiplying powers with the same base means adding their exponents.

Rule 2: The logarithm of a quotient¶

\[ \log_b\left(\frac{M}{N}\right) =\log_b(M)-\log_b(N) \]

For example:

\[ \log_2\left(\frac{16}{4}\right) =\log_2(16)-\log_2(4) \]

\[ =4-2=2 \]

And indeed:

\[ \log_2(4)=2 \]

Rule 3: The logarithm of a power¶

\[ \log_b(M^k)=k\log_b(M) \]

For example:

\[ \log_2(8^2)=2\log_2(8) \]

\[ =2\times3=6 \]

This is correct because \(8^2=64\) and \(2^6=64\).

Two useful identities¶

For a valid base \(b>0\), \(b\neq1\):

\[ \log_b(1)=0 \]

This is because:

\[ b^0=1 \]

Also:

\[ \log_b(b)=1 \]

Because:

\[ b^1=b \]

One important warning: logarithms do not distribute over addition.

In general:

\[ \log_b(M+N)\neq\log_b(M)+\log_b(N) \]

For example:

\[ \log_2(4+4)=\log_2(8)=3 \]

But:

\[ \log_2(4)+\log_2(4)=2+2=4 \]

The two results differ.

What is the natural logarithm?¶

The natural logarithm uses the mathematical constant \(e\), approximately equal to 2.71828.

It is written as:

\[ \ln(x)=\log_e(x) \]

For example, if:

\[ \ln(x)=2 \]

We convert to exponential form:

\[ x=e^2 \]

Therefore:

\[ x\approx7.39 \]

Natural logarithms appear in many machine learning formulas, including likelihood calculations, cross-entropy loss, and statistical models.

Back to the series overview.