Ytdyklly

There's a kind of goalpost moving that underlies this question. It used to be that NNs weren't very good at image recognition, so no one compared them to humans. Now that NNs are good at image tasks, suddenly it's the fault of NNs that they require a lot of training data to be comparable to children.

You can also turn this logic on its head. Suppose a child sees a number of cars the day that it's born. I wouldn't expect the child to be able to pick out a car the next day, or the next week, even though it's seen so many examples. Why are newborns so slow to learn? Because it takes a lot of exposure to the real world and, and the passage of time to change the child's neural pathways. For a neural network, we call this “training data,” but for a child we call it “growing up.”

First of all, at age two, child knows a lot about world and actively applies this knowledge. Child does a lot of "transfer learning" by applying this knowledge to new concepts.

Second, before seeing those five "labeled" examples of car, child sees a lot of cars on the street, on TV, toy cars etc., so also a lot of "unsupervised learning" happens beforehand.

Finally, neural networks have almost nothing in common with human brain, so there's not much point in comparing them. Also notice that there are algorithms for one shot learning, and pretty much research on it currently happens.

One major aspect that I don't see in current answers is evolution.

A child's brain does not learn from scratch. It's similar to asking how deer and giraffe babies can walk a few minutes after birth. Because they are born with their brains already wired for this task. There is some fine-tuning needed of course, but the baby deer doesn't learn to walk from "random initialization".

Similarly, the fact that big moving objects exist and are important to keep track of is something we are born with.

So I think the presupposition of this question is simply false. Human neural networks had the opportunity to see tons of - maybe not cars but - moving, rotating 3D objects with difficult textures and shapes etc., but this happened through lots of generations and the learning took place by evolutionary algorithms, i.e. the ones whose brain was better structured for this task, could live to reproduce with higher chance, leaving the next generation with better and better brain wiring from the start.

I don't know much about neural networks but I know a fair bit about babies.

Many 2 year olds have a lot of issues with how general words should be. For instance, it is quite common at that age for kids to use "dog" for any four legged animal. That's a more difficult distinction than "car" - just think how different a poodle looks from a great Dane, for instance and yet they are both "dog" while a cat is not.

And a child at 2 has seen many many more than 5 examples of "car". A kid sees dozens or even hundreds of examples of cars any time the family goes for a drive. And a lot of parents will comment "look at the car" a lot more than 5 times. But kids can also think in ways that they weren't told about. For instance, on the street the kid sees lots of things lined up. His dad says (of one) "look at the shiny car!" and the kid thinks "maybe all those other things lined up are also cars?"

This is an a fascinating question that I've pondered over a lot also, and can come up with a few explanations why.

• Neural networks work nothing like the brain. Backpropagation is unique to neural networks, and does not happen in the brain. In that sense, we just don't know the general learning algorithm in our brains. It could be electrical, it could be chemical, it could even be a combination of the two. Neural networks could be considered an inferior form of learning compared to our brains because of how simplified they are.


• If neural networks are indeed like our brain, then human babies undergo extensive "training" of the early layers, like feature extraction, in their early days. So their neural networks aren't really trained from scratch, but rather the last layer is retrained to add more and more classes and labels.

One way to train a deep neural network is to treat it as a stack of auto-encoders (Restricted Boltzmann Machines).

In theory, an auto-encoder learns in an unsupervised manner: It takes arbitrary, unlabelled input data and processes it to generate output data. Then it takes that output data, and tries to regenerate its input data. It tweaks its nodes' parameters until it can come close to round-tripping its data. If you think about it, the auto-encoder is writing its own automated unit tests. In effect, it is turning its "unlabelled input data" into labelled data: The original data serves as a label for the round-tripped data.

After the layers of auto-encoders are trained, the neural network is fine-tuned using labelled data to perform its intended function. In effect, these are functional tests.

The original poster asks why a lot of data is needed to train an artificial neural network, and compares that to the allegedly low amount of training data needed by a two-year-old human. The original poster is comparing apples-to-oranges: The overall training process for the artificial neural net, versus the fine-tuning with labels for the two-year-old.

But in reality, the two-year old has been training its auto-encoders on random, self-labelled data for more than two years. Babies dream when they are in utero. (So do kittens.) Researchers have described these dreams as involving random neuron firings in the visual processing centers.

As pointed out by others, the data-efficiency of artificial neural networks varies quite substantially, depending on the details. As a matter of fact, there are many so called one-shot learning methods, that can solve the task of labelling trams with quite good accuracy, using only a single labelled sample.

One way to do this is by so-called transfer learning; a network trained on other labels is usually very effectively adaptable to new labels, since the hard work is breaking down the low level components of the image in a sensible way.

But we do not infact need such labeled data to perform such task; much like babies dont need nearly as much labeled data as the neural networs you are thinking of do.

For instance, one such unsupervised methods that I have also successfully applied in other contexts, is to take an unlabeled set of images, randomly rotate them, and train a network to predict which side of the image is 'up'. Without knowing what the visible objects are, or what they are called, this forces the network to learn a tremendous amount of structure about the images; and this can form an excellent basis for much more data-efficient subsequent labeled learning.

While it is true that artificial networks are quite different from real ones in probably meaningful ways, such as the absence of an obvious analogue of backpropagation, it is very probably true that real neural networks make use of the same tricks, of trying to learn the structure in the data implied by some simple priors.

One other example which almost certainly plays a role in animals and has also shown great promise in understanding video, is in the assumption that the future should be predictable from the past. Just by starting from that assumption, you can teach a neural network a whole lot. Or on a philosophical level, I am inclined to believe that this assumption underlies almost everything what we consider to be 'knowledge'.

I am not saying anything new here; but it is relatively new in the sense that these possibilities are too young to have found many applications yet, and do not yet have percolated down to the textbook understanding of 'what an ANN can do'. So to answer the OPs question; ANN's have already closed much of the gap that you describe.

A human child at age 2 needs around 5 instances of a car to be able to identify it with reasonable accuracy regardless of color, make, etc.

The concept of "instances" gets easily muddied. While a child may have seen 5 unique instances of a car, they have actually seen thousands of thousands of frames, in many differing environments. They have likely seen cars in other contexts. They also have an intuition for the physical world developed over their lifetime - some transfer learning probably happens here. Yet we wrap all of that up into "5 instances."

Meanwhile, every single frame/image you pass to a CNN is considered an "example." If you apply a consistent definition, both systems are really utilizing a much more similar amount of training data.

Also, I would like to note that convolutional neural networks - CNNs - are more useful in computer vision than ANNs, and in fact approach human performance in tasks like image classification. Deep learning is (probably) not a panacea, but it does perform admirably in this domain.

We don't learn to "see cars" until we learn to see

It takes quite a long time and lots of examples for a child to learn how to see objects as such. After that, a child can learn to identify a particular type of object from just a few examples. If you compare a two year old child with a learning system that literally starts from a blank slate, it's an apples and oranges comparison; at that age child has seen thousands of hours of "video footage".

In a similar manner, it takes artificial neural networks a lot of examples to learn "how to see" but after that it's possible to transfer that knowledge to new examples. Transfer learning is a whole domain of machine learning, and things like "one shot learning" are possible - you can build ANNs that will learn to identify new types of objects that it hasn't seen before from a single example, or to identify a particular person from a single photo of their face. But doing this initial "learning to see" part well requires quite a lot of data.

Furthermore, there's some evidence that not all training data is equal, namely, that data which you "choose" while learning is more effective than data that's simply provided to you. E.g. Held & Hein twin kitten experiment. https://www.lri.fr/~mbl/ENS/FONDIHM/2013/papers/about-HeldHein63.pdf