Laboratory Task 3 – Back Propagation

Laboratory Task 3 – Back Propagation#

Name: Joanna Reyda Santos
Section: DS4A

Instruction: Perform a forward and backward propagation in python using the inputs from Laboratory Task 2

# Import necessary libraries
import numpy as np
import warnings
warnings.filterwarnings("ignore")

# Input and target (from Task 2)
x = np.array([1, 0, 1])
y = np.array([1])

# Learning rate
lr = 0.001

# ReLU activation and derivative
def relu(z):
    return np.maximum(0, z)

def relu_deriv(z):
    return (z > 0).astype(float)

# Initialize weights (same as Task 2)
W_hidden = np.array([
    [0.2, -0.3],
    [0.4,  0.1],
    [-0.5, 0.2]
])  # shape (3,2)

W_output = np.array([
    [-0.3],
    [-0.2]
])  # shape (2,1)

# Biases [θ1, θ2, θ3]
theta = np.array([-0.4, 0.2, 0.1])

Forward Pass#

# Step 1: Compute hidden layer pre-activation
z_hidden = x @ W_hidden + theta[:2]     # (1x3) dot (3x2) + bias(2,)

# Step 2: Apply ReLU activation
h = relu(z_hidden)                      # hidden activations

# Step 3: Compute output layer pre-activation and activation
z_out = h @ W_output + theta[2]         # (1x2) dot (2x1) + bias
y_hat = relu(z_out)                     # output activation

# Step 4: Compute mean squared error
loss = 0.5 * (y - y_hat) ** 2

print("Forward pass prediction:", y_hat)
print("Loss:", loss)

Forward pass prediction: [0.08103955]
Loss: [0.42224416]

BACKWARD PASS#

# Step 1: Compute output layer gradients
dL_dyhat = y_hat - y                     # derivative of error wrt predicted output
dyhat_dzout = relu_deriv(z_out)          # derivative of activation
dL_dzout = dL_dyhat * dyhat_dzout        # chain rule

# Step 2: Output weights and bias gradient
dL_dW_output = h.reshape(-1, 1) * dL_dzout
dL_dtheta3 = dL_dzout

# Step 3: Backpropagate to hidden layer
dzout_dh = W_output.flatten()            
dL_dh = dL_dzout * dzout_dh
dh_dz_hidden = relu_deriv(z_hidden)
dL_dz_hidden = dL_dh * dh_dz_hidden

# Step 4: Hidden weights and bias gradients
dL_dW_hidden = np.outer(x, dL_dz_hidden)
dL_dtheta_hidden = dL_dz_hidden        

# Update weights (Gradient Descent): W = W - lr * dL_dW
W_output -= lr * dL_dW_output
theta[2] -= lr * dL_dtheta3

W_hidden -= lr * dL_dW_hidden
theta[:2] -= lr * dL_dtheta_hidden

print("Updated W_hidden:\n", W_hidden)
print("Updated W_output:\n", W_output)
print("Updated theta:", theta)

Updated W_hidden:
 [[ 0.2        -0.30036771]
 [ 0.4         0.1       ]
 [-0.5         0.19963229]]
Updated W_output:
 [[-0.3       ]
 [-0.19981661]]
Updated theta: [-0.4         0.19963229  0.10183896]

Reflection#

In this lab, I applied both forward and backward propagation steps using NumPy. I learned how gradients flow backward from the output layer to the hidden layer and how each parameter (weights and biases) adjusts slightly to minimize the loss. Seeing the numerical updates in each step makes it clearer how learning actually happens in neural networks through gradient descent.

Contents