my-neural-net | Initial implementation of a combination from HTM | Reinforcement Learning library

It is a known thing with high capacity models. They are suprisingly resistant to overfitting which contradicts to the classical statistical learning theory that says that without explicit regularization you going to overfit. For example, this paper says

most of deep neural networks with learned parameters often generalize very well empirically, even equipped with much more effective parameters than the number of training samples, i.e. high capacity... Thus, statistical learning theory cannot explain the generalization ability of deep learning models.

Also, this and this papers are talking about it. You can keep on following the references in these papers to read more.

Personally, I have never seen high capacity model overfits even after training for 10s of thousands of epochs. If you want the example that does overfit: take Lenet 5 for Cifar10 with ReLU activations and without dropout and train it using SGD with learning rate 0.01. The number of training parameters in this model is ~ 60000 thousand which is the same as the number of samples in Cifar10 (low capacity model). After at most 500-1000 epochs you are going to see a very clear overfitting with increasing loss and error over time.