Implementing Simple Neural Network Backpropagation from Scratch

Outline

・Introduction to the XOR Gate Problem

・Constructing a 2-Layer Neural Network

・Forward Propagation

・Chain Rules for Backpropagation

・Implementation with NumPy

・Comparing Results with PyTorch

・Summary

・References

Introduction to the XOR Gate Problem

The XOR (exclusive OR) gate problem is considered simple for a neural network because it involves learning a simple pattern of relationships between inputs and outputs that a properly designed network can capture, even though it is not linearly separable (meaning you can’t draw a single straight line to separate the outputs into two groups based on inputs). Neural networks, particularly those with hidden layers, are capable of learning non-linear patterns.
Let’s look at the inputs and outputs of XOR Gate. Here is our 4 training data.

Constructing a 2-Layer Neural Network

we use the most simple 2-layer fully connected Neural Network as an example to solve the XOR Gate problem. Here is the structure of this network. The input layer has j nodes, j = 2 in our case. the hidden layer has i nodes, i = 4. The output layer has k nodes, k = 1.

Forward Propagation: Key Concept: Matrix Calculations!

Understanding Training Data and Network Parameters: The input Data x is represented by a matrix of (2,4). This format is used because we have 4 training examples , and each example comprises2 inputs.