Deep Learning: The best “new” technique from the 20th century
If you look anywhere in Silicon Valley and many places around the world, it is hard to avoid the poster child that is artificial intelligence. There is good reason for the hype. The techniques that fall under the umbrella that is artificial intelligence (AI) have been able to do some incredible things.
As a child I watched a computer beat some of the best human Jeopardy! players. Ken Jenning won 74 straight Jeopardy! games in a row, but he nor the other human participant. Brad Rutter, was a match to IBM Watson. Seemingly the technology singularity had finally come upon us…
No, but it hadn’t.
Okay. But what about when AlphaGo learned to play Go and in a highly publicized tournament beat the best humans players. At one point the computation resources needed given all the possibilities available in Go seemed like an insurmountable task. But like many other seemingly insurmountable feats, it is not about if, but more so about when and how.
Artifical intelligence and its most fruitful subfield, machine learning, is not magic.
Machine learning is simply the application of statistical techniques to improve at specific tasks. As a child, we learned how to distinguish fruits. Not because we had an innate ability, but because we were “trained.” We saw examples and in time an association was made. Machine learning works in a similar way. Provide the “AI” techniques with data (life experiences). With enough examples the techniques will come to develop an association.
Deep learning is the most popular and “new” technique within the field of machine learning. The flavors of the technique come by many names including convolutional neural network (CNN), recurrent neural networks (RNN), and others. Although deep learning is all the rage right now, the mathematical foundations/ideas were present in the 20th century.
The notion of “training” machines is an old concept. For example, Alan Turning’s “Turning test” is a notion from 1950. The initial attempts at building out these intelligent machines focused on trying to mimic the brain. Not an irrational thought by any means. Thoughts seem to reside in our brain (whole separate discussion) so why not try to emulate it?
Take, for example, Frank Rosenblatt’s Perceptron from the 1950’s. I suggest looking up a video to get an idea of the machine. In many ways, the Perceptron showed an ability to learn. Similar to the neural networks of today, the machine was given many examples and it would get better at distinguishing. However, the Perceptrons would come to be shown to be flawed. In the 1980’s, however, neural nets gain some populairty through the efforts of Hinton and others. The new approach, which included “hidden” layers atoned for the flaws that the Perceptron could not get around. While there were successes in the 1980’s, they often could not scale and therefore fell out of fashion again. (2)
If the technique was first developed in the 20th century why did it take until the 21st century for the fruits of the techniques to become evident? The answer is pretty much: more data; more computational power. The amount of data available in the 21st century is immense. For example, currently, humans produce 2.5 quintillion bytes of data per day. (3) The amount of data and specifically organized, structured data allowed for a turning point. In time, the neural networks that were rebranded as “deep learning” would obtain state of the art status.
So, while deep learning might be the special sauce fueling many of the advancements in computer vision. natural language processing, and spoken speech, it has been around for a while. Instead, only with the aid of its partners in crime, big data and computational power, did we begin to see the technique’s true potential.
Resources:
- https://www.economist.com/science-and-technology/2017/10/21/the-latest-ai-can-work-things-out-without-being-taught
- https://www.forbes.com/sites/bernardmarr/2016/03/22/a-short-history-of-deep-learning-everyone-should-read/#524161535561
- https://www.forbes.com/sites/bernardmarr/2018/05/21/how-much-data-do-we-create-every-day-the-mind-blowing-stats-everyone-should-read/#6a20148b60ba
- http://www.andreykurenkov.com/writing/ai/a-brief-history-of-neural-nets-and-deep-learning/
