Initialization in neural networks

This thesis examines how weight initialization affects the training of deep neural net- works, specifically targeting the ResNet architecture, which utilizes full pre-activation residual blocks and ReLU activations. We perform a theoretical derivation of gradi- ent variance propagation, illustrating that skip connections help stabilize training and mitigate the issue of vanishing gradients. With t…