Vectorized Neural Networks Model Representation

From Scalar Equations to Vector Form

Previously, we wrote each neuron separately.

For the hidden layer:

$$a^{(2)}_1 = g\left(\Theta^{(1)}_{10}x_0 + \Theta^{(1)}_{11}x_1 + \Theta^{(1)}_{12}x_2 + \Theta^{(1)}_{13}x_3\right)$$ $$a^{(2)}_2 = g\left(\Theta^{(1)}_{20}x_0 + \Theta^{(1)}_{21}x_1 + \Theta^{(1)}_{22}x_2 + \Theta^{(1)}_{23}x_3\right)$$ $$a^{(2)}_3 = g\left(\Theta^{(1)}_{30}x_0 + \Theta^{(1)}_{31}x_1 + \Theta^{(1)}_{32}x_2 + \Theta^{(1)}_{33}x_3\right)$$

Final hypothesis:

$$h_\Theta(x) = a^{(3)}_1 = g\left(\Theta^{(2)}_{10}a^{(2)}_0 + \Theta^{(2)}_{11}a^{(2)}_1 + \Theta^{(2)}_{12}a^{(2)}_2 + \Theta^{(2)}_{13}a^{(2)}_3\right)$$

This works, but it does not scale. So we vectorize.

---

Step 1 — Define the Intermediate Variable

Define the weighted sum before activation:

$$z^{(j)}_k = \Theta^{(j-1)}_{k,0} a^{(j-1)}_0 + \Theta^{(j-1)}_{k,1} a^{(j-1)}_1 + \dots + \Theta^{(j-1)}_{k,n} a^{(j-1)}_n$$

Activation becomes:

$$a^{(j)}_k = g\left(z^{(j)}_k\right)$$

---

Step 2 — Vector Representation

Input layer:

$$x = a^{(1)} = \begin{bmatrix} x_0 \\ x_1 \\ \vdots \\ x_n \end{bmatrix}$$

Weighted sum vector:

$$z^{(j)} = \begin{bmatrix} z^{(j)}_1 \\ z^{(j)}_2 \\ \vdots \\ z^{(j)}_{s_j} \end{bmatrix}$$

Where:

$$s_j = \text{number of units in layer } j$$

---

Step 3 — The Key Vectorized Equation

The entire layer becomes a single matrix multiplication:

$$z^{(j)} = \Theta^{(j-1)} a^{(j-1)}$$

Dimensions:

$$\Theta^{(j-1)} \in \mathbb{R}^{s_j \times (n+1)}$$ $$a^{(j-1)} \in \mathbb{R}^{(n+1) \times 1}$$ $$z^{(j)} \in \mathbb{R}^{s_j \times 1}$$

---

Step 4 — Apply Activation Function

Activation is applied element-wise:

$$a^{(j)} = g\left(z^{(j)}\right)$$

If using sigmoid:

$$g(z) = \frac{1}{1 + e^{-z}}$$

---

Step 5 — Add Bias Unit

After computing activations, add the bias:

$$a^{(j)} = \begin{bmatrix} 1 \\ a^{(j)}_1 \\ \vdots \\ a^{(j)}_{s_j} \end{bmatrix}$$

---

Step 6 — Output Layer

Repeat the same process:

$$z^{(j+1)} = \Theta^{(j)} a^{(j)}$$ $$a^{(j+1)} = g\left(z^{(j+1)}\right)$$

Final hypothesis:

$$h_\Theta(x) = a^{(j+1)}$$

---

The Big Picture

Each layer performs:

$$\text{Linear transformation} \quad z = \Theta a$$

followed by

$$\text{Nonlinearity} \quad a = g(z)$$

Stacking these layers allows neural networks to represent complex nonlinear functions.