Hello, It's GPT-2 -- How Can I Help You? Towards the Use of Pretrained Language Models for Task-Oriented Dialogue Systemshttps://arxiv.org/abs/1907.05774
This paper is written by Cambridge University and PolyAI people. You may recall I've written about PolyAI before. I believe Matthew Henderson is a co-founder. He was the student of Steve Young. The startup appears to be doing some really important dialog system research, and will probably get bought out by a large company like Google or Apple.
The paper has some really neat examples of what this GPT-2-based system can do. They claim that using a pre-trained GPT-2 language model they can bypass the traditional dialog management and domain-specific NLG modules. They still need to make a few tweaks, like add-in specific tokens for the different parties in a conversation, and also to refer to things like "telephone number" or "address"; but this is very minor compared to adding a dialog manager.
Although this is neat from a research perspective, I doubt Google or Apple will be doing this sort of thing in the near-future -- because there is too much risk that the system will say something rude, lewd, racist, sexist, cruel, immoral, etc. Template-based response generators are a way to avoid that; but they tend to give a wooden response. Still, a wooden response is probably preferable to large companies than being the target of a mass media article about how their Assistant said something racist.
Some examples of what it can do:
Visitor: I am looking for some places to go, do you have any attractions in [value_area]?
GPT-2: Sure! There are many colleges and museums in the area. Do you have a preference for type or price range?
Visitor: I really do not have a preference. Can you recommend a few?
GPT-2: How about the [attraction_name]?
Visitor: Sure. Also can you help me find a restaurant serving unusual food in the same part of town?
GPT-2: Unfortunately, there are no restaurants that match that description. Would you like to try a different area or type of food?
As you can see, the text being generated is complex, diverse, coherent, and germane to the questions asked by the visitor. It was generated without templates!
In principle, it should be possible to use GPT-2 + all the tweaks they mention in this article, to build an AI assistant that can help with a very broad set of complex tasks -- and even combinations of tasks. The only problem, though, is that you can't predict when it might go off the rails and say something racist, sexist, lewd, immoral, etc. -- and Google wouldn't want such a thing splashed across the pages of the New York Times. So, at least for the next several years, templates will be how Assistants by the big tech companies will operate.
Now, a lean and mean startup could get away with using a model like this; they wouldn't have as much of a brand to protect. It's possible we might see that, but I wouldn't count on it.