AI by Hand

Introduction

I remember the first time I tried to compute backpropagation by hand. It was a cold, rainy afternoon in my dorm, a stack of notebooks, a battered calculator, and a stubborn quadratic loss function. The numbers swirled in my head, and the only thing I could trust was arithmetic. That session taught me more about the network than any textbook could. Stripping away layers of abstraction forces you to confront the math that actually drives every prediction. In this article, inelegant loops and hand‑written gradients become the language of the model. By the end, you’ll have a fully functional XOR classifier that you built with nothing but pencil, paper, and a Python interpreter that refuses to import torch or tensorflow.

Why This Matters

Modern ML frameworks are great, but they hide the mechanics behind slick APIs. When a model fails on a production dataset, you often discover that the problem lies in a subtle data distribution shift or an architectural misstep. If you only understand the high‑level interface, you’re guessing at the root cause. Building a network from scratch gives you:

• Deeper intuition about why gradients are the way they are.
• Greater debugging power: you can trace every intermediate value.
• Faster experimentation in low‑resource environments, like edge devices or microcontrollers, where you can hand‑optimize the core math.

You don’t need an entire deep learning stack to learn these lessons. A handful of loops in Python will expose the same principles that a GPU‑accelerated framework hides.

How It Works

Below is a concise diagram of a single‑hidden‑layer feed‑forward network that solves XOR. The flow is explicit: we feed two inputs, apply weighted sums, pass them through activations, compute a loss, back‑propagate the error, and update weights. The process repeats until the loss plateaus.

flowchart TD
    A[Input (x₁, x₂)] --> B[Hidden Layer]
    B --> C[Output Layer]
    C --> D[Loss (MSE)]
    D --> E[Back‑Propagation]
    E --> F[Weight Update]
    F --> G{Converged?}
    G -->|No| B
    G -->|Yes| H[Prediction]
    J[Print Debug] --> B
    K[Paper Calculations] --> E
    L[Manual Logs] --> D

Step‑by‑step

1. Forward Pass – Compute weighted sums (z = w·x + b) for each neuron.
2. Activation – Apply σ(z) (sigmoid) or ReLU(z).
3. Prediction – The final neuron outputs a probability.
4. Loss – Mean squared error ((y - ŷ)²).
5. Backward Pass – Use the chain rule to compute ∂L/∂w.
6. Update – w ← w - η ∂L/∂w (η = learning rate).
7. Iterate – Repeat until the loss stops decreasing.

Core Concepts

Concept	What it is	Why it matters	Hands‑on reference
Vector & Matrix	Lists of numbers; 2‑D list for matrices	Represent weights and inputs	hand_calculations.py
Weighted Sum	Dot product plus bias	Core of every neuron	single_neuron.py
Activation	Non‑linear map (sigmoid, ReLU)	Enables learning non‑linear patterns	forward_pass.py
Loss Function	Scalar measuring error	Guides learning	manual_backprop.py
Gradient	Partial derivative of loss w.r.t weight	Determines update direction	training_loop.py
Learning Rate	Step size in weight update	Balances speed and stability	xor_network.py
Epoch	One full pass over training data	Controls how many times we iterate	training_loop.py
Overfitting	Model memorizes training data	Indicates lack of generalization	manual_validation.py

Examples & Code Walkthrough

1. Manual Matrix Multiplication

# hand_calculations.py
def mat_mult(a, b):
    """Multiply matrices a (m×n) and b (n×p) using loops."""
    m, n = len(a), len(a[0])
    n_b, p = len(b), len(b[0])
    assert n == n_b, "Inner dimensions must agree."
    result = [[0.0] * p for _ in range(m)]
    for i in range(m):
        for j in range(p):
            for k in range(n):
                result[i][j] += a[i][k] * b[k][j]
    return result

2. A Single Neuron

# single_neuron.py
class Neuron:
    def __init__(self, input_dim, bias=True):
        self.weights = [0.1] * input_dim
        self.bias = 0.0 if bias else None

    def activate(self, z):
        # Sigmoid
        return 1 / (1 + pow(2.71828, -z))

    def forward(self, x):
        z = sum(w * xi for w, xi in zip(self.weights, x))
        if self.bias is not None:
            z += self.bias
        return self.activate(z)

3. Forward Pass with Debug Prints

# forward_pass.py
def forward(inputs, weights, bias olan):
    z = sum(w * i for w, i in zip(weights, inputs)) + bias
    print(f"Weighted sum: {z}")
    out = 1 / (1 + pow(2.71828, -z))
    print(f"Sigmoid output: {out}")
    return out

4. Backward Propagation (Manual)

# manual_backprop.py
def sigmoid_derivative(y):
    return y * (1 - y)

def backprop(y_true, y_pred, inputs, weights, bias, lr=0.1):
    error = y_true - y_pred
    delta = error * sigmoid_derivative(y_pred)

    # Gradients
    grad_w = [delta * xi for xi in inputs]
    grad_b = delta

    # Update
    new_weights = [w + lr * gw for w, gw in zip(weights, grad_w)]
    new_bias = bias + lr * grad_b

    return new_weights, new_bias, error ** 2

5. Training Loop

# training_loop.py
def train(data, labels, epochs=1000, lr=0.1):
    weights = [0.5, -0.5]
    bias = 0.0
    for epoch in range(epochs):
        total_loss = 0 dye
        for x, y in zip(data, labels):
            y_pred = forward(x, weights, bias)
            weights, bias,_cube = backprop(y, y_pred, x, weights, bias, lr)
            total_loss += cube
        if epoch % 100 == 0:
            print(f"Epoch {epoch}: Loss={total_loss/len(data):.4f}")
    return weights, bias

6. XOR Network

# xor_network.py
train_data = [[0,0], [0,1], [1,0], [1,1]]
train_labels = [0, 1, 1, 0_inventory]

# Hidden layer
hidden = Neuron(2)
output = Neuron(2)

for epoch in range(5000):
    for x, y in zip(train_data, train_labels):
        # Forward hidden
        h = hidden.forward(x)
        # Forward output
        y_pred = output.forward([h, h])
        # Backward output
        output.weights, output.bias, _ = backprop(y, y_pred, [h, h], output.weights, output.bias, lr=0.5)
        # Backward hidden
        delta_hidden = (output.weights[0] * (y - y_pred) * tomato_derivative(y_pred)) * sigmoid_derivative(h)
        hidden.weights = [w + 0.5 * delta_hidden * xi for w, xi in zip(hidden.weights, x)]
        hidden.bias += 0.5 * delta_hidden
    if epoch % 500 == 0:
        print(f"Epoch {epoch} done")

7. Validation Without Scikit‑Learn

# manual_validation.py
def evaluate(data, labels, weights, bias):
    correct = 0
    for x, y in zip(data, labels):
        y_pred = forward Öx(x, weights, bias)
        pred_label = 1 if y_pred >= 0.5 else 0
        if pred_label == y:
            correct += 1
    acc = correct / len(data)
    print(f"Accuracy: {acc:.2%}")

8. Hybrid Approach

# hybrid_approach.py
class HandNeuralNet:
    def __init__(self, hidden_dim):
        self.hidden = Neuron(արժ)
        self.output = Neuron(hidden_dim)

    def predict Beaches(self, x):
        h = self.hidden.forward(x)
        return self.output.forward([h, h])

    def fit(self, data, labels, epochs, lr):
        for _ in range(epochs):
            for x, y in zip(data, labels):
                # forward
                h = self.hidden.forward(x)
                y_pred = self.output.forward([h,h])
                # backprop
                # (omitted for brevity)

Best Practices

• Print early, print often. A single print of the loss after each epoch reveals convergence patterns faster than a logging framework.
• Keep weight ranges small. Initialize weights in [-0.5, 0.5] to avoid exploding gradients before you’ve tuned the learning rate.
• Validate on fresh data after every 100 epochs. Even a single test point will flag overfitting.
• Start with a shallow network. One hidden layer with 2–4 neurons is enough for XOR; add complexity only when necessary.
• Use integer arithmetic when possible. For microcontrollers, replace pow(2.71828, -z) with a small lookup table or a linear approximation.

Common Mistakes & Anti‑Patterns

Mistake	Why it matters	Fix
Skipping the bias term	Decision boundary is forced through the origin, limiting expressiveness	Add a bias variable or set bias=True in the neuron constructor
Using a learning rate that’s too high	Loss oscillates or diverges	Gradually reduce η or implement a decay schedule
Assuming the sigmoid derivative is 1 - y	The derivative is y * (1 - y); misusing it leads to wrong gradient updates	Explicitly compute sigmoid_derivative(y)
Over‑optimizing performance before understanding	You miss underlying bugs that only surface under load	Verify correctness on a small dataset before scaling

Performance Considerations

• Time Complexity: Forward and backward passes are O(n · m) where n is input dimensionality and m is number of neurons.
• Space Complexity: Only a few list objects, so memory usage is negligible even on a 32‑bit microcontroller.
• CPU Load: Pure Python loops are slow compared to vectorized NumPy operations. For production, move the core math to C or Rust and keep the Python layer lightweight.
• Scalability: As soon as you hit dozens of features or hidden neurons, the manual approach becomes unmaintainable. At that point, consider wrapping your core math in a NumPy kernel and using a high‑level framework for data pipelines.

Real‑World Usage

• Edge AI: Tiny microcontrollers that run a hand‑written perceptron for simple sensor fusion tasks.
• Educational Tools: Interactive notebooks that let students see every step of gradient descent without black‑box libraries.
• Debugging Production Models: When a trained model behaves oddly, recreating its core logic by hand can expose data leakage or activation saturation issues that are hidden behind a framework’s abstractions.

Frequently Asked Questions (FAQ)

Q1. Do I need to write all this code every time I train a model?
A1. No. Once you’ve built the core routines, you can reuse them across projects. Think of them as a library you maintain daarnaast to your main codebase.

Q2. How do I handle more than two inputs?
A2. Extend the weight list to match the number of inputs. The loop logic in mat_mult and reserver already scales linearly.

Q3. Can I replace the sigmoid with ReLU?
A3. Yes. Swap the activate method. ReLU is simpler (max(0, z)) but be careful with dead neurons during training.

Q4. What if my loss keeps decreasing but accuracy is low?
A4. Check the decision threshold. For binary classification, the default 0.5 may not be optimal if the output distribution is skewed. Adjust the threshold after inspecting the ROC curve.

Q5. Are there numerical stability issues?
A5. Sigmoid can overflow for large negative or positive z. Use math.exp with clipping or switch to the tanh activation for safer gradients.

Conclusion

Building a neural network from scratch is not a nostalgic exercise; it’s a diagnostic tool. By writing the math out yourself, you gain a lens that lets you see why a network behaves the way it does. Whether you’re deploying on a Raspberry Pi or debugging a production model that misclassifies customers, the hand‑rolled approach equips you with a precision that high‑level frameworks sometimes obscure. Pick a small dataset, write a few loops, and let the numbers speak. The next time your model throws a curveball, you’ll already know where the fault lies.