Empirical covariances of the weights of the FCNN in the first layer. We can see correlations in the spatial direction in the weights of the input layer (left). In other directions, the covariance matrix is less smooth than we would expect from an isotropic Gaussian draw of the same size (left matrix of every pair), but otherwise has no structure. This suggests that the weights are not isotropic Gaussian.
{}