Jump to content

Welcome to FutureTimeline.forum
Register now to gain access to all of our features. Once registered and logged in, you will be able to create topics, post replies to existing threads, give reputation to your fellow members, get your own private messenger, post status updates, manage your profile and so much more. If you already have an account, login here - otherwise create an account for free today!
Photo

Google says a machine passing the Turing Test "is potentially within reach"


  • Please log in to reply
2 replies to this topic

#1
funkervogt

funkervogt

    Member

  • Members
  • PipPipPipPipPipPip
  • 915 posts

 

We present Meena, a multi-turn open-domain chatbot trained end-to-end on data mined and filtered from public domain social media conversations. This 2.6B parameter neural network is simply trained to minimize perplexity of the next token. We also propose a human evaluation metric called Sensibleness and Specificity Average (SSA), which captures key elements of a human-like multi-turn conversation. Our experiments show strong correlation between perplexity and SSA. The fact that the best perplexity end-to-end trained Meena scores high on SSA (72% on multi-turn evaluation) suggests that a human-level SSA of 86% is potentially within reach if we can better optimize perplexity. Additionally, the full version of Meena (with a filtering mechanism and tuned decoding) scores 79% SSA, 23% higher in absolute SSA than the existing chatbots we evaluated.

 

https://arxiv.org/abs/2001.09977



#2
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPip
  • 1,403 posts

I gave an example or two of what it can do here:

 

https://www.futureti...-like/?p=273590

 

It's still got a ways to go, though.  Have a look at this collection of conversations:

 

https://github.com/g...meena/meena.txt

 

Some are good, some are merely ok, some are bad.

 

Brain data will probably fix many of the shortcomings.  Still, it's far better than ELIZA and other chatbots -- and it probably does some amount of world and user-modelling, just not as well as a human over long conversations. 

 

Another thing that might fix it is training on yet more data.  Microsoft recently trained a 17 billion parameter language model, which is like 7 or 8 times as big as Meena; and there is still enough data to train up models with over 100 billion parameters -- in fact, if you include image and video data, maybe that can be pushed to 1 trillion or more.  Evidence shows that the more parameters these models have, the better they do at generating text -- the grammar gets better, the factuality gets better, the coherence gets better, the relevance gets better, and the perplexity gets lower.  In fact, there have been some papers that attempt to chart just how much things improve as a function of dataset size.



#3
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPip
  • 1,403 posts
I wanted to mention something else about this chatbot:  I doubt Google will ever release it as a consumer product -- maybe as an experimental program for researchers to work with.  
 
(Although... the fact that they spent over $1 million on the computational resources to build it means that it will go to waste, if it only ever is used as a research prototype.)
 
There are two reasons for this.  The first is that chatbots aren't the kind of thing Google does.  It's not part of their brand and message.  They're into helping the user "get stuff done" and "empowering the user".  Chatbots are more what Microsoft and Amazon are trying to do. 
 
The second reason is that the way the system is designed, it's hard to know for sure that it won't write racist, sexist, lewd, rude, mean, or immoral responses that can damage the Google / Alphabet brand.  In fact, this is probably the main reason Google hasn't released a demo for the wider public (other than researchers), and definitely also why Microsoft didn't release DialoGPT (as they stated in their paper).  I've discussed this before, and it was good seeing my prediction on this vindicated.  
 
There are a lot of chatbot and virtual assistants queued-up for public release, but are being blocked by the fact that they might write toxic content, damaging the brand of the company that built them.  What's desperately needed is a really good "critic" module that can filter out all these comments with very high accuracy.  Such a module need not be full AGI -- it just needs to be particularly sensitive to the kinds of "mistakes" that other AI systems make.   
 
Good critics for robots would probably also speed up the deployment of robotics systems in manufacturing, warehouse work, home automation, construction, and driverless cars.  The way they would work is that whenever the robot made a mistake, it would output an "error signal", that could then be used to drive the learning system in the robot so that it does better in the future -- and even cause the robot to correct before completing the mistake the first time.  
 
....
 
On an unrelated note:  this is something people might find amusing:
 
https://twitter.com/...278319166091264
 

I chatted with #MeenaBot about the #coronavirus and her advice is to see a doctor sooner rather than later. I guess it's not a bad one & hope everyone is well! On the other hand, Meena is also excited about technology, especially VR!


(Click, to see the examples.)




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users