The second wave of neural networks research lasted until the mid-1990s.
Ventures based on neural networks and other AI technologies began to make unrealistically ambitious claims while seeking investments.
When AI research did not fulfill these unreasonable expectations, investors were disappointed.Simultaneously,other fields of machine learning made advances.
Kernel machines (Boser et al., 1992; Cortes and Vapnik, 1995; Schölkopf et al., 1999) and graphical models (Jordan, 1998) both achieved good results on many important tasks.These two factors led to a decline in the popularity of neural networks that lasted until 2007.
During this time, neural networks continued to obtain impressive performance on some tasks (LeCun et al., 1998b; Bengio et al., 2001).
The Canadian Institute for Advanced Research (CIFAR) helped to keep neural networks research alive via its Neural Computation and Adaptive Perception (NCAP) research initiative.
This program united machine learning research groups led by Geoffrey Hinton at University of Toronto, Yoshua Bengio at University of Montreal, and Yann LeCun at New York University.
The CIFAR NCAP research initiative had a multi-disciplinary nature that also included neuroscientists and experts in human and computer vision.
At this point in time, deep networks were generally believed to be very difficult to train.
We now know that algorithms that have existed since the 1980s work quite well, but this was not apparent circa 2006.
The issue is perhaps simply that these algorithms were too computationally costly to allow much experimentation with the hardware available at the time.