Performance of Different Activation Functions in Two Hidden Layer Handwritten Digit Recognition Neural Network - Leong, Yew Joe; Lee, Siaw Chong (Enhanced Knowledge in Sciences and Technology, 2024)
This research delves into the realm of handwritten digits recognition using neural networks, with a primary objective of comparing the performance of three distinct activation functions—tanh, sigmoid, and ReLU. Initial investigations focus on identifying optimal parameters, involving weight initialization method, the number of nodes in hidden layers, and learning rate. Through systematic experimentation, the study discerns the most effective configurations for these parameters. Subsequently, the chosen configurations are employed to evaluate the performance of the neural network with each activation function. Results indicate that ReLU outperforms both sigmoid and tanh, establishing itself as the most effective activation function for the given task. These findings provide valuable insights for optimizing neural network architectures in handwritten digit recognition applications.