Performance of Different Activation Functions in Two Hidden Layer Handwritten Digit Recognition Neural Network

Abstract

This research delves into the realm of handwritten digits recognition using neural
networks, with a primary objective of comparing the performance of three distinct
activation functions—tanh, sigmoid, and ReLU. Initial investigations focus on
identifying optimal parameters, involving weight initialization method, the number of
nodes in hidden layers, and learning rate. Through systematic experimentation, the
study discerns the most effective configurations for these parameters. Subsequently,
the chosen configurations are employed to evaluate the performance of the neural
network with each activation function. Results indicate that ReLU outperforms both
sigmoid and tanh, establishing itself as the most effective activation function for the
given task. These findings provide valuable insights for optimizing neural network
architectures in handwritten digit recognition applications.