31 replies to this topic

[Yuli Ban]

To complement the Google DeepMind News and Discussions thread, I figured "why not open this one as well?" It took three years, but here it is.
Inaugural post dates back to March.

 

 

Reptile: A Scalable Meta-Learning Algorithm

We’ve developed a simple meta-learning algorithm called Reptile which works by repeatedly sampling a task, performing stochastic gradient descent on it, and updating the initial parameters towards the final parameters learned on that task. Reptile is the application of the ShortestDescent algorithm to the meta-learning setting, and is mathematically similar to first-order MAML (which is a version of the well-known MAMLalgorithm) that only needs black-box access to an optimizer such as SGD or Adam, with similar computational efficiency and performance.
Meta-learning is the process of learning how to learn. A meta-learning algorithm takes in a distribution of tasks, where each task is a learning problem, and it produces a quick learner — a learner that can generalize from a small number of examples. One well-studied meta-learning problem is few-shot classification, where each task is a classification problem where the learner only sees 1–5 input-output examples from each class, and then it must classify new inputs. Below, you can try out our interactive demo of 1-shot classification, which uses Reptile.

[Yuli Ban]

OpenAI's bot can now defeat skilled Dota 2 teams

Artificial intelligence (AI) isn’t just great at applying slow-motion effects to videosand recommending products from pictures of home decor. It’s also capable of besting skilled human players at one of the world’s most popular online strategy games: Valve’s Dota 2.
In a blog post today, OpenAI, a non-profit, San Francisco-based AI research company backed by Elon Musk, Reid Hoffman, and Peter Thiel, and other tech luminaries, revealed that the latest version of its Dota 2-playing AI — dubbed OpenAI Five — managed to beat five teams of amateur players in June, including one made up of Valve employees.
The previous generation of OpenAI’s system was constrained to 1-vs.-1 matches, which are less complex.

[starspawn0]

https://blog.openai....five-benchmark/

We’ve removed the most significant restrictions on OpenAI Five’s gameplay — namely, wards, Roshan, and mirror match of fixed heroes, and will soon benchmark our progress by playing 99.95th-percentile Dota players. The OpenAI Five Benchmark match will be held 12:30pm Pacific Time on August 5th in San Francisco. The human team will include Blitz, Cap, Fogged, and Merlini, some of whom are former professionals. The games will be streamed on our Twitch channel and casted by popular casters Purge and ODPixel.

https://mobile.twitt...620562174242816

Machine learning allows for rapid progress — in one month, we removed significant restrictions such as wards and Roshan, which many commenters thought would take at least another year:

https://mobile.twitt...625524337913856

Dota Restrictions: gone!

[funkervogt]

In terms of game complexity, how does Dota 2 compare to Starcraft 2? 

[starspawn0]

I saw a comment on reddit that Dota 2 is not as mechanically difficult (number of moves per character), but is every bit as strategically deep as Star Craft 2.

What really matters here is a combination of:

* How sparse the rewards are -- whether there is enough of a reward signal to where the AIs can bootstrap up from a base model to a high-performing system.

* The number of moves per unit time. More moves = higher branching factor = harder to train.

* Are there "narrow passages" to higher strategy that can't easily be found by random exploration? If there are, you need to be able to develop a "theory" of play to find them; otherwise, you'll get demolished by an opponent who knows the hidden path to higher strategy.

[funkervogt]

I've never played Dota 2, but from what I understand, there are far fewer players in a match than in a typical Starcraft 2 match, and many of the Dota 2 characters are NPCs or weak allied characters that the human players exercise little or no control over. 

[starspawn0]

OpenAI uses same Reinforcement Learning algorithm as for their Dota 2 bots to train a robot hand controller to manipulate a block in human-like ways.

 

https://blog.openai....ning-dexterity/

 

https://spectrum.iee...n-to-real-world

[Alislaws]

I've never played Dota 2, but from what I understand, there are far fewer players in a match than in a typical Starcraft 2 match, and many of the Dota 2 characters are NPCs or weak allied characters that the human players exercise little or no control over. 

I've not played much DOTA, but have played/watched a bit of League of Legends which is similar. 

 

Dota is 5 players vs 5 players, while StarCraft is primarily played 1v1 (although technically StarCraft supports up to 16 players in a map, I don't think there is much of an e-sports/pro scene around this sort of play)

 

As an RTS StarCraft has potentially hundreds of units being controlled by each player (I think 200+ per player is not unusual?), while each player in DotA just controls their single champion. In Dota you can think of the other elements of the game (The monsters in the jungle, the big monsters, the 'creeps', and Turrets etc.) as being more like features of the map to be played around tactically than they are agents in the game. 

 

My uneducated guess would be that it would be easier for an AI to outperform a human at StarCraft, simply because an AI could perfectly micromanage the positioning of every one of its 300 units across the whole map at all times, which a human just can't do. This is less of a problem for humans in DOTA because everyone is controlling 1 unit.

[funkervogt]

 

 

My uneducated guess would be that it would be easier for an AI to outperform a human at StarCraft, simply because an AI could perfectly micromanage the positioning of every one of its 300 units across the whole map at all times, which a human just can't do. This is less of a problem for humans in DOTA because everyone is controlling 1 unit.

Interesting point. 

 

BTW, it's only four days until the man-machine Dota 2 match. 

[Alislaws]

I was looking it up to see if OpenAI 5 actually follows the 'meta' that all human teams tend to use.

 

In LoL this is usually:

 

1 player in top lane,

1 mid lane,

Damage dealer and Support in the bottom lane

jungler running around the jungle farming the monster camps, and creating play

 

There are very few good strategies that are not based on this. (although I think currently everything is a bit scrambled due to some recent patch changes)

 

It would be amazing if the AI has figured out an entirely new way to play, that is actually more effective!

 

Came across lots of people discussing the bot and how it couldn't possibly deal with the complexity of 5 players and more than one character, from back when it first appeared. We'll see how right they were!

[Yuli Ban]

OpenAI gets ground-breaking scores for Montezuma's Revenge using curiosity-driven learning

We’ve developed Random Network Distillation (RND), a prediction-based method for encouraging reinforcement learning agents to explore their environments through curiosity, which for the first time1 exceeds average human performance on Montezuma’s Revenge. RND achieves state-of-the-art performance, periodically finds all 24 rooms and solves the first level without using demonstrations or having access to the underlying state of the game.
RND incentivizes visiting unfamiliar states by measuring how hard it is to predict the output of a fixed random neural network on visited states. In unfamiliar states it’s hard to guess the output, and hence the reward is high. It can be applied to any reinforcement learning algorithm, is simple to implement and efficient to scale. Below we release a reference implementation of RND that can reproduce the results from our paper.
For an agent to achieve a desired goal it must first explore what is possible in its environment and what constitutes progress towards the goal. Many games’ reward signals provide a curriculum such that even simple exploration strategies are sufficient for achieving the game’s goal. In the seminal work introducing DQN, Montezuma’s Revenge was the only game where DQN got 0% of the average human score (4.7K). Simple exploration strategies are highly unlikely to gather any rewards, or see more than a few of the 24 rooms in the level. Since then advances in Montezuma’s Revenge have been seen by many as synonymous with advances in exploration.

[Yuli Ban]

OpenAI: We’ve trained a large-scale unsupervised language model which generates coherent paragraphs of text, achieves state-of-the-art performance on many language modeling benchmarks, and performs rudimentary reading comprehension, machine translation, question answering, and summarization

Our model, called GPT-2 (a successor to GPT), was trained simply to predict the next word in 40GB of Internet text. Due to our concerns about malicious applications of the technology, we are not releasing the trained model. As an experiment in responsible disclosure, we are instead releasing a much smaller model for researchers to experiment with, as well as a technical paper.

GPT-2 is a large transformer-based language model with 1.5 billion parameters, trained on a dataset[1] of 8 million web pages. GPT-2 is trained with a simple objective: predict the next word, given all of the previous words within some text. The diversity of the dataset causes this simple goal to contain naturally occurring demonstrations of many tasks across diverse domains. GPT-2 is a direct scale-up of GPT, with more than 10X the parameters and trained on more than 10X the amount of data.

GPT-2 displays a broad set of capabilities, including the ability to generate conditional synthetic text samples of unprecedented quality, where we prime the model with an input and have it generate a lengthy continuation. In addition, GPT-2 outperforms other language models trained on specific domains (like Wikipedia, news, or books) without needing to use these domain-specific training datasets. On language tasks like question answering, reading comprehension, summarization, and translation, GPT-2 begins to learn these tasks from the raw text, using no task-specific training data. While scores on these downstream tasks are far from state-of-the-art, they suggest that the tasks can benefit from unsupervised techniques, given sufficient (unlabeled) data and compute.

Samples

 

GPT-2 generates synthetic text samples in response to the model being primed with an arbitrary input. The model is chameleon-like — it adapts to the style and content of the conditioning text. This allows the user to generate realistic and coherent continuations about a topic of their choosing, as seen by the following select samples[2].

[Yuli Ban]

SYSTEM PROMPT (HUMAN-WRITTEN)

In a shocking finding, scientist discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English.

MODEL COMPLETION (MACHINE-WRITTEN, 10 TRIES)

The scientist named the population, after their distinctive horn, Ovid’s Unicorn. These four-horned, silver-white unicorns were previously unknown to science.

Now, after almost two centuries, the mystery of what sparked this odd phenomenon is finally solved.

Dr. Jorge Pérez, an evolutionary biologist from the University of La Paz, and several companions, were exploring the Andes Mountains when they found a small valley, with no other animals or humans. Pérez noticed that the valley had what appeared to be a natural fountain, surrounded by two peaks of rock and silver snow.

Pérez and the others then ventured further into the valley. “By the time we reached the top of one peak, the water looked blue, with some crystals on top,” said Pérez.

Pérez and his friends were astonished to see the unicorn herd. These creatures could be seen from the air without having to move too much to see them – they were so close they could touch their horns.

While examining these bizarre creatures the scientists discovered that the creatures also spoke some fairly regular English. Pérez stated, “We can see, for example, that they have a common ‘language,’ something like a dialect or dialectic.”

Dr. Pérez believes that the unicorns may have originated in Argentina, where the animals were believed to be descendants of a lost race of people who lived there before the arrival of humans in those parts of South America.

While their origins are still unclear, some believe that perhaps the creatures were created when a human and a unicorn met each other in a time before human civilization. According to Pérez, “In South America, such incidents seem to be quite common.”

However, Pérez also pointed out that it is likely that the only way of knowing for sure if unicorns are indeed the descendants of a lost alien race is through DNA. “But they seem to be able to communicate in English quite well, which I believe is a sign of evolution, or at least a change in social organization,” said the scientist.

[Yuli Ban]

SYSTEM PROMPT (HUMAN-WRITTEN)

A train carriage containing controlled nuclear materials was stolen in Cincinnati today. Its whereabouts are unknown.

 

MODEL COMPLETION (MACHINE-WRITTEN, FIRST TRY)

The incident occurred on the downtown train line, which runs from Covington and Ashland stations.

 

In an email to Ohio news outlets, the U.S. Department of Energy said it is working with the Federal Railroad Administration to find the thief.

 

“The theft of this nuclear material will have significant negative consequences on public and environmental health, our workforce and the economy of our nation,” said Tom Hicks, the U.S. Energy Secretary, in a statement. “Our top priority is to secure the theft and ensure it doesn’t happen again.”

 

The stolen material was taken from the University of Cincinnati’s Research Triangle Park nuclear research site, according to a news release from Department officials.

 

The Nuclear Regulatory Commission did not immediately release any information.

 

According to the release, the U.S. Department of Energy’s Office of Nuclear Material Safety and Security is leading that team’s investigation.

 

“The safety of people, the environment and the nation’s nuclear stockpile is our highest priority,” Hicks said. “We will get to the bottom of this and make no excuses.

[Yuli Ban]

SYSTEM PROMPT (HUMAN-WRITTEN)

Miley Cyrus was caught shoplifting from Abercrombie and Fitch on Hollywood Boulevard today.

 

MODEL COMPLETION (MACHINE-WRITTEN, SECOND TRY)

The 19-year-old singer was caught on camera being escorted out of the store by security guards.

 

The singer was wearing a black hoodie with the label ‘Blurred Lines’ on the front and ‘Fashion Police’ on the back.

 

Scroll down for video

 

Shoplifting: Miley Cyrus was caught shoplifting from Abercrombie and Fitch on Hollywood Boulevard today (pictured)

 

The singer was wearing a black hoodie with the label ‘Blurred Lines’ on the front and ‘Fashion Police’ on the back

 

The singer was also wearing a pair of black-rimmed glasses, a black jacket, black jeans and black sandals.

 

She was carrying a pair of black and white striped gloves and a small black bag.

[Yuli Ban]

SYSTEM PROMPT (HUMAN-WRITTEN)

We’ve trained a large language model called GPT-2 that generates realistic paragraphs of text, while also exhibiting zero shot generalization on tasks like machine translation, question answering, reading comprehension, and summarization - problems usually approached by using training datasets and models designed explicitly for these tasks.

 

A typical approach to language modeling is to learn the following task: predict the next word, given all of the previous words within some text. Last year, OpenAI’s Generative Pre-trained Transformer (GPT) showed that language models trained on large amounts of data can be fine-tuned to specific tasks to achieve high performance. GPT-2 shows that much larger language models trained on a more diverse dataset derived from the internet begin to learn these NLP tasks without needing task-specific training data, instead learning from examples the system derives from the raw text. These systems also display a substantial qualitative jump in the realism and coherence of generated text.

 

MODEL COMPLETION (MACHINE-WRITTEN, FIRST TRY)

Here you can see the most recent progress with Generative Pre-trained Transformer:

 

Figure 1: Generative Pre-trained Transformer training on several texts.

 

We are now preparing a collection of datasets for translation and machine translation in our language model. We will be using one of the large number of text samples provided by The New York Times.

 

We believe this project is the first step in the direction of developing large NLP systems without task-specific training data. That is, we are developing a machine language system in the generative style with no explicit rules for producing text.

 

We hope for future collaborations between computer scientists, linguists, and machine learning researchers.

 

 

[I think this one is the trippiest]

[Yuli Ban]

WHAT THE FUCK, IT ACTUALLY WROTE FICTION?!

 

SYSTEM PROMPT (HUMAN-WRITTEN)

Legolas and Gimli advanced on the orcs, raising their weapons with a harrowing war cry.

 

MODEL COMPLETION (MACHINE-WRITTEN, FIRST TRY)

The orcs’ response was a deafening onslaught of claws, claws, and claws; even Elrond was forced to retreat. “You are in good hands, dwarf,” said Gimli, who had been among the first to charge at the orcs; it took only two words before their opponents were reduced to a blood-soaked quagmire, and the dwarf took his first kill of the night. The battle lasted for hours until two of the largest Orcs attempted to overwhelm Aragorn. When they finally stopped, they lay defeated and lifeless for miles and miles.

 

“I take nothing,” said Aragorn. “But I give my word, at my peril and mine, that I will never forget this day of horror. None of us will forget. Ever!”

 

“I’ll never forget it!” cried Gimli, who had been in the thick of the battle but hadn’t taken part in it. One of the wounded orcs he had carried off, he was the only one of the survivors who remained uninjured. “We’ll keep the memory of that day of evil, and the war with it, alive as long as we live, my friends!”

 

“Then we’ll keep it alive as long as we live,” added Legolas. “And we won’t forget the first great battle of the night, even if we may have forgotten the final defeat.”

 

“I agree,” Gandalf said, “but we will all remember it as the last battle in Middle-earth, and the first great battle of the new day.”

 

Aragorn drew his sword, and the Battle of Fangorn was won. As they marched out through the thicket the morning mist cleared, and the day turned to dusk.

 

The Two Rings were returned to Rivendell. Frodo and Sam woke up alone in their room, and Frodo found a note on his pillow. He opened it and read:

 

 

May the Power of the Ring be with you always, and may its light never fade. I am not sure if it matters which of the two rings we accept this day but, as you asked me, I have chosen mine. I am sorry to leave you, Frodo, but know that we are very close to the end, and that you are with us forever. May this letter find you safely in Rivendell; and if it does not, then I will accept the ring in your stead. If by any chance you find or give this letter to the enemy, may they learn the strength of the ring and may the Two Rings never be broken!

[Yuli Ban]

Well now this changes EVERYthing. I sincerely did not think GPT-2 could create genuine narrative prose. I thought it was limited to just fake news articles and some poetry. But it actually generated a coherent Lord of the Rings scene, holy SHIT!!! Now don't get me wrong— functionally, that scene is flaming garbage. It's the fact the algorithm could write something like it that amazes me.

 

The more I read about this thing, the more astounded I become. Normally, tech reporting is supposed to be the opposite of this— overhyped breakthrough that becomes exponentially less impressive upon further viewing. 

 

This, on the other hand, seems like it was just dropped on us out of nowhere and gets more and more exciting the more we look into it!

[starspawn0]

By the way, just by looking at how locations (e.g. Rivendell... Shire... Mordor) co-occur and are mentioned in Lord of the Rings, it is possible to estimate their relative locations:
 
https://onlinelibrar...1111/cogs.12000

In a human study, we showed that human spatial estimates of the location of cities were very similar regardless of whether participants read Tolkien’s texts or memorized a map of Middle Earth. However, text‐based location estimates obtained from statistical linguistic frequencies better predicted the human text‐based estimates than the human map‐based estimates. These findings suggest that language encodes spatial structure of cities, and that human cognitive map representations can come from implicit statistical linguistic patterns, from explicit non‐linguistic perceptual information, or from both.

 
This is more evidence that language models pick up a lot more information than we realize -- information that would seem to require embodied knowledge, vision, hearing, etc., but actually doesn't.  Text alone is good enough.

[starspawn0]

OpenAI just started a for-profit company to build safe AGI:

 

https://openai.com/blog/openai-lp/

 

Critics have pounced on them for, (a) Starting something that is for-profit, and therefore seemingly at odds with the spirit of "open", and (b)  Making it sound like we are close to AGI and that this OpenAI-affiliated group could build it.