Jump to content

Welcome to FutureTimeline.forum
Register now to gain access to all of our features. Once registered and logged in, you will be able to create topics, post replies to existing threads, give reputation to your fellow members, get your own private messenger, post status updates, manage your profile and so much more. If you already have an account, login here - otherwise create an account for free today!

These ads will disappear if you register on the forum

Photo

Artificial Intelligence 2020: Deep Reinforcement Learning + Progressive Neural Networks

deep learning neural networks artificial neural networks progressive neural networks deep reinforcement learning artificial intelligence AI 2020 robotics AGI

  • Please log in to reply
12 replies to this topic

#1
Yuli Ban

Yuli Ban

    Nadsat Brat

  • Moderators
  • PipPipPipPipPipPipPipPipPipPipPip
  • 17,133 posts
  • LocationAnur Margidda

I've been reading about them a bit more recently, and I firmly believe that this is where we'll see many interesting developments in the coming months.
 
Here's an Arvix paper on PNNs.

Learning to solve complex sequences of tasks--while both leveraging transfer and avoiding catastrophic forgetting--remains a key obstacle to achieving human-level intelligence. The progressive networks approach represents a step forward in this direction: they are immune to forgetting and can leverage prior knowledge via lateral connections to previously learned features. We evaluate this architecture extensively on a wide variety of reinforcement learning tasks (Atari and 3D maze games), and show that it outperforms common baselines based on pretraining and finetuning. Using a novel sensitivity measure, we demonstrate that transfer occurs at both low-level sensory and high-level control layers of the learned policy.

 
And here's a blog post on deep RL.

This is a long overdue blog post on Reinforcement Learning (RL). RL is hot! You may have noticed that computers can now automatically learn to play ATARI games (from raw game pixels!), they are beating world champions at Go, simulated quadrupeds are learning to run and leap, and robots are learning how to perform complex manipulation tasksthat defy explicit programming. It turns out that all of these advances fall under the umbrella of RL research. I also became interested in RL myself over the last ~year: I worked through Richard Sutton’s book, read through David Silver’s course, watched John Schulmann’s lectures, wrote an RL library in Javascript, over the summer interned at DeepMind working in the DeepRL group, and most recently pitched in a little with the design/development of OpenAI Gym, a new RL benchmarking toolkit. So I’ve certainly been on this funwagon for at least a year but until now I haven’t gotten around to writing up a short post on why RL is a big deal, what it’s about, how it all developed and where it might be going.


It seems a bit shocking that we've had all this progress, and yet we still haven't gotten around to crossing these two methods. Deep reinforcement learning is a very, very important step towards the development of artificial general intelligence, but none of it matters if the AI can master Pong but completely fail at Pac-Man, despite both games being on the Atari 2600 and utilizing some similar skills. 

 

Imagine you learned how to bake a cake. After 50 tries, you finally bake a perfect cake, better than any other person can in fact. Wonderful, right? But when you go to bake brownies, you have to learn everything from scratch. You need 50 more shots just to relearn how to bake these things, despite the fact there isn't much difference between a cake and brownies in terms of preparation. 

That's where AI is now. We can teach AI how to bake cakes, and we can teach AI how to bake brownies— but we can't teach AI how to bake cakes and then brownies. Which is a pretty massive handicap for intelligence, to say the least. One of the most fundamental aspects of what defines something as being 'intelligent' is the ability to use past experiences to predict the future. When I say 'predict', I'm not necessarily saying anything ridiculously abstract like going into the stock market or figuring out higher level physics from simple experiments. No, I mean "figure this out based on what you've learned from something else." 

Like my previous analogy said, if you can bake a cake, you can bake brownies. If you can't, then there's something fundamentally wrong with your brain. You shouldn't have to relearn how to measure ingredients when you're making a new, but similar dish. 

 

So in that sense, we've figured out one aspect of machine learning, but it's still useless without the ability to reason. PNNs are a step in that direction.


  • Unity likes this
Nobody's gonna take my drone, I'm gonna fly miles far too high!
Nobody gonna beat my drone, it's gonna shoot into the sky!

#2
tornado64

tornado64

    Member

  • Members
  • PipPipPipPipPipPip
  • 803 posts
  • LocationChina

There is another thing missing. Learning well from small data sets like humans can.

Currently more data means you'll beat anything with significantly less data, no matter what model you use etc.

That's the reason why many companies now open source the models/sofware/computers etc. they use (e.g. TensorFlow) because the real difference is the amount of data you can throw on the problem not the hardware, alghoritms, software.



#3
Yuli Ban

Yuli Ban

    Nadsat Brat

  • Moderators
  • PipPipPipPipPipPipPipPipPipPipPip
  • 17,133 posts
  • LocationAnur Margidda

There is another thing missing. Learning well from small data sets like humans can.
Currently more data means you'll beat anything with significantly less data, no matter what model you use etc.
That's the reason why many companies now open source the models/sofware/computers etc. they use (e.g. TensorFlow) because the real difference is the amount of data you can throw on the problem not the hardware, alghoritms, software.

Though it should be mentioned that this is something that PNNs can aid with. The first time around, it might take quite a bit of learning to get up to speed. However, once you've gotten some decent models stored in your memory, you can draw upon them when coming across a new situation so that you can learn from an otherwise tiny data set. Not only that, but you can recursively use PNNs to aid with learning something even faster than you would see it with deep RL alone.

 

In fact, some say that the reason why humans seem so much more effective at this than computers do is because much of this is already hardwired into us via DNA. We've proven that DNA can actually react to external experiences (the children of Holocaust survivors happened to prove this when we checked their DNA and discovered noticeable signs of extreme trauma), which essentially means that we already have 'Big Data' preprogrammed into our genes.


Nobody's gonna take my drone, I'm gonna fly miles far too high!
Nobody gonna beat my drone, it's gonna shoot into the sky!

#4
tornado64

tornado64

    Member

  • Members
  • PipPipPipPipPipPip
  • 803 posts
  • LocationChina

 

There is another thing missing. Learning well from small data sets like humans can.
Currently more data means you'll beat anything with significantly less data, no matter what model you use etc.
That's the reason why many companies now open source the models/sofware/computers etc. they use (e.g. TensorFlow) because the real difference is the amount of data you can throw on the problem not the hardware, alghoritms, software.

Though it should be mentioned that this is something that PNNs can aid with. The first time around, it might take quite a bit of learning to get up to speed. However, once you've gotten some decent models stored in your memory, you can draw upon them when coming across a new situation so that you can learn from an otherwise tiny data set. Not only that, but you can recursively use PNNs to aid with learning something even faster than you would see it with deep RL alone.

 

In fact, some say that the reason why humans seem so much more effective at this than computers do is because much of this is already hardwired into us via DNA. We've proven that DNA can actually react to external experiences (the children of Holocaust survivors happened to prove this when we checked their DNA and discovered noticeable signs of extreme trauma), which essentially means that we already have 'Big Data' preprogrammed into our genes.

 

 

 

Maybe it helps learning from smaller training sets, but it can't be the whole picture.

Humans are able to learn something completely new from just doing it like 10 times, considering how childrens learn.

With Deep Learning for example the robots trying to catch a ball, it's like they need to do it 800.000 times in order to do it reliably.



#5
Unity

Unity

    Information Organism

  • Members
  • PipPipPipPipPipPipPipPip
  • 2,442 posts

It's good to see you narrowing your focus Yuli



#6
Yuli Ban

Yuli Ban

    Nadsat Brat

  • Moderators
  • PipPipPipPipPipPipPipPipPipPipPip
  • 17,133 posts
  • LocationAnur Margidda

Crap, here's the link to the blog post.

http://karpathy.gith.../2016/05/31/rl/

 

No idea how that happened.


Nobody's gonna take my drone, I'm gonna fly miles far too high!
Nobody gonna beat my drone, it's gonna shoot into the sky!

#7
Yuli Ban

Yuli Ban

    Nadsat Brat

  • Moderators
  • PipPipPipPipPipPipPipPipPipPipPip
  • 17,133 posts
  • LocationAnur Margidda

Specialization is for insects — AI needs to be able to multitask
There’s another key problem with deep learning: the fact that all our current systems are, essentially, idiot savants. Once they’ve been trained, they can be incredibly efficient at tasks like recognizing cats or playing Atari games, says Google DeepMind research scientist Raia Hadsell. But "there is no neural network in the world, and no method right now that can be trained to identify objects and images, play Space Invaders, and listen to music." (Neural networks are the building blocks of deep learning systems.)
The problem is even worse than that, though. When Google’s DeepMind announced in February last year that it’d built a system that could beat 49 Atari games, it was certainly a massive achievement, but each time it beat a game the system needed to be retrained to beat the next one. As Hadsell points out, you can’t try to learn all the different games at once, as the rules end up interfering with one another. You can learn them one at a time — but you end up forgetting whatever you knew about previous games. "To get to artificial general intelligence we need something that can learn multiple tasks," says Hadsell. "But we can’t even learn multiple games."
A solution to this might be something called progressive neural networks — this means connecting separate deep learning systems together so that they can pass on certain bits of information. In a paper published on this topic in June, Hadsell and her team showed how their progressive neural nets were able to adapt to games of Pong that varied in small ways (in one version the colors were inverted; in another the controls were flipped) much faster than a normal neural net, which had to learn each game from scratch.
It’s a promising method, and in more recent experiments it’s even been applied to robotic arms — speeding up their learning process from a matter of weeks to just a single day. However, there are significant limitations, with Hadsell noting that progressive neural networks can’t simply keep on adding new tasks to their memory. If you keep chaining systems together, sooner or later you end up with a model that is "too large to be tractable," she says. And that’s when the different tasks being managed are essentially similar — creating a human-level intelligence that can write a poem, solve differential equations, and design a chair is something else altogether.

Screen_Shot_2016-10-10_at_11.58.03_AM.0.
The basic layout of a progressive neural network. (Image credit: DeepMind / Raia Hadsell)

This came from a post on the Verge about today's AI's limitations.
This is why I asked whether the brain can be seen as a neural network of neural networks. Massively parallel, memristive, and highly progressive— sounds just like the brain. Except the brain is also more than that. One single algorithm that happens to be deep RL + PNN is nowhere near enough. Nor is a group of algorithms. You need that group of algorithms to be themselves controlled by a deep RL PNN algorithm as well.


Nobody's gonna take my drone, I'm gonna fly miles far too high!
Nobody gonna beat my drone, it's gonna shoot into the sky!

#8
Yuli Ban

Yuli Ban

    Nadsat Brat

  • Moderators
  • PipPipPipPipPipPipPipPipPipPipPip
  • 17,133 posts
  • LocationAnur Margidda

I created a basic graph to understand what I mean in that regard.
gYw9iB0.png

Each of these things listed under the individual sub-algorithms is considered to be a major task for AI as of 2016. IBM Watson is famous entirely for NLP; that is, natural language processing. Not even understanding, but processing. Watson still doesn't "know" what words mean. 

If there were a computer on Earth that could do two of these things, we'd call it "the smartest computer in the world." And there is, in fact, a computer capable of doing two of these things— NLP and image recognition, to be precise. And DeepMind possesses it.

Despite that, no one claims DeepMind is AGI. DeepMind's computer can't cross its two abilities either. As impressive as it is to us, it's still Weak AI. Less weak than previous systems, but very much weak.

 

We won't be approaching AGI until we get something more like that graph. Where you basically have a neural network composed of neural networks made of neural networks. One superalgorithm that runs the whole thing, knowing when something is useful and needs to be used, powered by multiple smaller meta-neural networks that are, themselves, made powerful by possessing highly trained neural networks. Each individual trained neural network being "breakthroughs" and "amazing accomplishments" in 2016, but tiny parts of a giant whole for this AGI (probably of 2026?).


Nobody's gonna take my drone, I'm gonna fly miles far too high!
Nobody gonna beat my drone, it's gonna shoot into the sky!

#9
Yuli Ban

Yuli Ban

    Nadsat Brat

  • Moderators
  • PipPipPipPipPipPipPipPipPipPipPip
  • 17,133 posts
  • LocationAnur Margidda

Yuli-Ban

I did this snazzy write up on what AI needs to do in order to reach something resembling general intelligence.
I never stated in that post, however, when I believed we'd see such a thing appear. And truth be told, 5 years ago, I would have said that creating even the lowest level of trained neural networks would have been something for the 2020s at earliest. We predicted AI would best humans at Go by the mid 2020's, and we also expected AI to not come close to matching humans at image or speech recognition until then either. But those things happened a decade ahead of schedule, at least.
The problem in the mainstream is that we see these tasks as resulting from a generalized AI from the getgo— we thought you needed AGI to have machines defeat humans at Go, just as we thought with chess two decades prior. Now we realize that this isn't necessarily the case. While it's no longer just straight "preprogram every last single thing into the computer" anymore, actually involving neural networks that learn, it's still not AGI.
So on one hand, we overestimated how much work we needed to do with AI on that front while underestimating the power of deep neural networks. But that doesn't mean we're close to AGI.
As impressive as these tasks and several others are, they're not examples of general intelligence. What's more, we're seeing just how in over our own heads we are the more we achieve. AI is better at image recognition than humans? That's nice. You didn't need to program every last step into that AI, allowing it to learn? But you still need to train said AI with millions of pictures in order for it to get there. Hence why Big Data is so important.
Big Data is supremely important to 2010's AI because today's AI lacks the ability to learn from itself like biological intelligence can. It can't recall previously learned experiences, and thus it can't apply previously learned experiences towards solving new problems. As far as intelligence goes, that's a really damn big problem. Like, I'd go so far as to say anything that lacks such an ability doesn't deserve to be called "intelligent" at all. Or, more accurately, anything that does possess such an ability should be considered intelligent. If AI had that ability, it wouldn't require Big Data so much. AlphaGo could have become a Go master after only 100 games rather than several million.
But— and this is a plump, sexy but— we do have the tools necessary to overcome that last step, through progressive neural networks and differential neural computers. That not even bringing up memristors, which can do such things natively.
And while going a step further and seeing unsupervised learning on top of memory recall brings us even closer to AI, we're still stuck with algorithms that excel at singular tasks. Or, at most, a few similar tasks. You'll need a controller neural network over them all in order to make good use of that. You'll need several of these meta-neural networks at that, and one super neural network on top of that before you get anything like a digital brain.
Like I said, 5 years ago, all that sounded like science fiction. But now I trust that we will be able to pull that off within 10 years. And considering DeepMind has a modern day Medici as a financial backing in the form of Alphabet Corporation, I wouldn't even be surprised if they achieved all this before 2020. At which point, they'll probably reach the absolute peak of artificial intelligence as based upon silicon. It'll be artificial general intelligence, humanity's very first, but it won't be the event horizon. For that, we'd need quantum computing. And again, Alphabet's got that covered as well.
So strong AI has a decent— I'd even say good— chance of existing by 2030. It's like with fusion energy. The reason why all this is happening is because of funding + successes. We have the necessary computing power to run the algorithms we need, and the companies pursuing AI are literally throwing billions of dollars at the problem.
10 years ago, when total funding towards generalized AI probably maxed out at somewhere near $100 million, it makes sense we'd think AGI is something for the 22nd century. That was basically an "AI Never" period of time we were living in.


Nobody's gonna take my drone, I'm gonna fly miles far too high!
Nobody gonna beat my drone, it's gonna shoot into the sky!

#10
Yuli Ban

Yuli Ban

    Nadsat Brat

  • Moderators
  • PipPipPipPipPipPipPipPipPipPipPip
  • 17,133 posts
  • LocationAnur Margidda

The end of that post is something I've been meaning to expand upon. I've mentioned it multiple times before that funding for AI ebbs and flows. Historically, it was because computers simply were not powerful enough to make AI practical. But now they are, so what's the hold up? Regardless of how powerful computers are, you still need funding to make things happen.

 

I estimated that total funding for AI projects in 2006 was around $100 million. While I couldn't find any hard numbers, I could find that this number was about half-right for 2010. And like I said, that's total. For 2006, I reckon that number was much lower. 

The only thing I couldn't find was how much funding major companies gave to their own AI networks (e.g. Google, Microsoft, Apple, etc.)

 

I'd estimate that we'd need a yearly investment of at least $5 billion to achieve AGI by 2029. As of 2016, there has been a total of $8.5 billion in funding.

 

 

If this sounds familiar, it's because this is an analog to fusion energy funding. The reason why we don't have fusion energy today isn't because it was too hard for us to achieve— it's because we lacked the funding to make it possible. If fusion were funded at a proper level, we could have had fusion power plants online by 1995. Ronald Reagan slashed funding for fusion research to incredibly, uselessly low levels, while Clinton and the Bushes did nothing to restore funding to where it should have been. Obama could have brought funding up, but by that time, renewables were on their way up. See the chart below:

 

U.S._historical_fusion_budget_vs._1976_E

 

 

 

We couldn't have created fusion power plants in the '70s or '80s because we lacked the necessary materials to make fusion feasible. Part of the reason being there was little funding. We intended on the future to take up that mantle.

 

Similar thing happened with AI, except funding levels have gone in the reverse. 10 years ago, AI funding was at an 'AGI Never' level. That's one reason why we always saw AI as being something for the 22nd century and beyond. But in those 10 years, we've seen the rise of deep learning and reinforcement learning, whose successes have prompted a tsunami of funding to pour into the AI field. That tsunami is still flowing. In fact, it's probably just getting started, and the crest of funding won't be reached until some time later this decade or early next decade. 


  • Casey likes this
Nobody's gonna take my drone, I'm gonna fly miles far too high!
Nobody gonna beat my drone, it's gonna shoot into the sky!

#11
Yuli Ban

Yuli Ban

    Nadsat Brat

  • Moderators
  • PipPipPipPipPipPipPipPipPipPipPip
  • 17,133 posts
  • LocationAnur Margidda

Here's a blog post on progressive neural networks


If you’ve seen one Atari game you’ve seen them all, or at least once you’ve seen enough of them anyway. When we (humans) learn, we don’t start from scratch with every new task or experience, instead we’re able to build on what we already know. And not just for one new task, but the accumulated knowledge across a whole series of experiences is applied to each new task. Nor do we suddenly forget everything we knew before – just because you learn to drive (for example), that doesn’t mean you suddenly become worse at playing chess. But neural networks don’t work like we do. There seem to be three basic scenarios:

  • Training starts with a blank slate
  • Training starts from a model that has been pre-trained in a similar domain, and the model is then specialised for the target domain (this can be a good tactic when there is lots of data in the pre-training source domain, and not so much in the target domain). In this scenario, the resulting model becomes specialised for the new target domain, but in the process may forget much of what it knew about the source domain (“catastrophic forgetting”).  This scenario is called ‘fine tuning’ by the authors.
  • Use pre-trained feature representations (e.g. word vectors) as richer features in some model.
The last case gets closest to knowledge transfer across domains, but can have limited applicability.

This paper introduces progressive networks, a novel model architecture with explicit support for transfer across sequences of tasks. While fine tuning incorporates prior knowledge only at initialization, progressive networks retain a pool of pretrained models throughout training, and learn lateral connections from these to extract useful features for the new task.

The progressive networks idea is actually very easy to understand (somewhat of a relief for someone like myself who is just following along as an interested outsider observing developments in the field!). Some of the key benefits include:
  • The ability to incorporate prior knowledge at each layer of the feature hierarchy
  • The ability to reuse old computations and learn new ones
  • Immunity to catastrophic forgetting
Thus they are a stepping stone towards continual / life-long learning systems.
Here’s how progressive networks work. Start out by training a neural network with some number L of layers to perform the initial task.

I wonder if the differentiable neural computers are the progressive neural networks that were mentioned.


Nobody's gonna take my drone, I'm gonna fly miles far too high!
Nobody gonna beat my drone, it's gonna shoot into the sky!

#12
Yuli Ban

Yuli Ban

    Nadsat Brat

  • Moderators
  • PipPipPipPipPipPipPipPipPipPipPip
  • 17,133 posts
  • LocationAnur Margidda

Finally got around to fixing that infuriating graphic.
AkVVmB2.png


Nobody's gonna take my drone, I'm gonna fly miles far too high!
Nobody gonna beat my drone, it's gonna shoot into the sky!

#13
garry

garry

    New Member

  • Members
  • Pip
  • 2 posts
What is a neural network and how does it work?
So, what is a neural network? Another name for this technology is artificial neural network (ANN). It was called so because the principles powering this technology were based on the work of neurons in human brain. Our neurons create transient states serving as a basis for making unique decisions that are a part of what we call creativity.
Read more in the blog post Neural Networks in Business






Also tagged with one or more of these keywords: deep learning, neural networks, artificial neural networks, progressive neural networks, deep reinforcement learning, artificial intelligence, AI, 2020, robotics, AGI

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users