We first observe the influence of the non-linear activations functions. Xavier Glorot and Yoshua Bengio, "Understanding the difficulty of training deep feedforward neural networks" http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf デ…