Jump to content

Welcome to FutureTimeline.forum
Register now to gain access to all of our features. Once registered and logged in, you will be able to create topics, post replies to existing threads, give reputation to your fellow members, get your own private messenger, post status updates, manage your profile and so much more. If you already have an account, login here - otherwise create an account for free today!
Photo

What 2029 will look like


  • Please log in to reply
95 replies to this topic

#81
Alislaws

Alislaws

    Democratic Socialist Materialist

  • Members
  • PipPipPipPipPipPipPipPip
  • 1,968 posts
  • LocationLondon

This work implies that machine learning models based on nothing more than predicting word co-occurrence patterns, can absorb a lot of implicit scientific knowledge not directly trained for:
 
 
With Little Training, Machine-Learning Algorithms Can Uncover Hidden Scientific Knowledge 
 
 
https://newscenter.l...ific-knowledge/
 

"Without telling it anything about materials science, it learned concepts like the periodic table and the crystal structure of metals,” said Jain. “That hinted at the potential of the technique. But probably the most interesting thing we figured out is, you can use this algorithm to address gaps in materials research, things that people should study but haven’t studied so far.”


So... models like GPT-2 might have acquired a lot of scientific knowledge not explicitly mentioned in the training corpus.

 

That's incredible, Really excited to see how far/fast this progresses!


  • Yuli Ban likes this

#82
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPip
  • 1,062 posts
The state of AI up to late June 2019:

https://www.stateof.ai/

This doesn't include the latest XLNET stuff from Google or Alibaba's work on question-answering (MS-MARCO). Even so, the advances it discusses are impressive. The advances will continue like this for at least the next 5 years. Robotics is next to see major change. Speech Recognition was the first to be impacted; then came Computer Vision; after that was NLP; and now, finally, it's Robotics's turn to get slammed by Deep Learning. And after that, Reasoning (e.g. hard mathematical reasoning) will feel the burn.
  • Casey likes this

#83
johnnd

johnnd

    Member

  • Members
  • PipPip
  • 12 posts

The state of AI up to late June 2019:https://www.stateof.ai/Speech Recognition was the first to be impacted; then came Computer Vision; after that was NLP; and now, finally, it's Robotics's turn to get slammed by Deep Learning. And after that, Reasoning (e.g. hard mathematical reasoning) will feel the burn.


So, there was a speech recognition revolution before ImageNet? I always thought it was vision first, but I wasn't following this stuff as closely back then.
  • Yuli Ban and starspawn0 like this

#84
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPip
  • 1,062 posts
Yes, the first Deep Learning speech recognition breakthroughs occurred before success at Imagenet:

https://en.wikipedia...earning#History

The 2009 NIPS Workshop on Deep Learning for Speech Recognition[69] was motivated by the limitations of deep generative models of speech, and the possibility that given more capable hardware and large-scale data sets that deep neural nets (DNN) might become practical. It was believed that pre-training DNNs using generative models of deep belief nets (DBN) would overcome the main difficulties of neural nets.[70] However, it was discovered that replacing pre-training with large amounts of training data for straightforward backpropagation when using DNNs with large, context-dependent output layers produced error rates dramatically lower than then-state-of-the-art Gaussian mixture model (GMM)/Hidden Markov Model (HMM) and also than more-advanced generative model-based systems.[60][71]


  • Yuli Ban and johnnd like this

#85
ralfy

ralfy

    Member

  • Members
  • PipPipPipPip
  • 137 posts

Limits to growth kicks in.



#86
Yuli Ban

Yuli Ban

    Born Again Singularitarian

  • Moderators
  • PipPipPipPipPipPipPipPipPipPipPip
  • 20,573 posts
  • LocationNew Orleans, LA

 

The state of AI up to late June 2019:https://www.stateof.ai/Speech Recognition was the first to be impacted; then came Computer Vision; after that was NLP; and now, finally, it's Robotics's turn to get slammed by Deep Learning. And after that, Reasoning (e.g. hard mathematical reasoning) will feel the burn.


So, there was a speech recognition revolution before ImageNet? I always thought it was vision first, but I wasn't following this stuff as closely back then.

It apparently was concurrent development, but speech was a bit easier than vision.

I linked some decade-old Reddit threads on the very earliest days of deep learning, and that seems to be the general mood of things.


  • starspawn0 and johnnd like this

And remember my friend, future events such as these will affect you in the future.


#87
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPip
  • 1,062 posts
Hello, It's GPT-2 -- How Can I Help You? Towards the Use of Pretrained Language Models for Task-Oriented Dialogue Systems

https://arxiv.org/abs/1907.05774

This paper is written by Cambridge University and PolyAI people. You may recall I've written about PolyAI before. I believe Matthew Henderson is a co-founder. He was the student of Steve Young. The startup appears to be doing some really important dialog system research, and will probably get bought out by a large company like Google or Apple.

The paper has some really neat examples of what this GPT-2-based system can do. They claim that using a pre-trained GPT-2 language model they can bypass the traditional dialog management and domain-specific NLG modules. They still need to make a few tweaks, like add-in specific tokens for the different parties in a conversation, and also to refer to things like "telephone number" or "address"; but this is very minor compared to adding a dialog manager.

Although this is neat from a research perspective, I doubt Google or Apple will be doing this sort of thing in the near-future -- because there is too much risk that the system will say something rude, lewd, racist, sexist, cruel, immoral, etc. Template-based response generators are a way to avoid that; but they tend to give a wooden response. Still, a wooden response is probably preferable to large companies than being the target of a mass media article about how their Assistant said something racist.

....

Some examples of what it can do:

Visitor: I am looking for some places to go, do you have any attractions in [value_area]?

GPT-2: Sure! There are many colleges and museums in the area. Do you have a preference for type or price range?

Visitor: I really do not have a preference. Can you recommend a few?

GPT-2: How about the [attraction_name]?

Visitor: Sure. Also can you help me find a restaurant serving unusual food in the same part of town?

GPT-2: Unfortunately, there are no restaurants that match that description. Would you like to try a different area or type of food?


As you can see, the text being generated is complex, diverse, coherent, and germane to the questions asked by the visitor. It was generated without templates!

In principle, it should be possible to use GPT-2 + all the tweaks they mention in this article, to build an AI assistant that can help with a very broad set of complex tasks -- and even combinations of tasks. The only problem, though, is that you can't predict when it might go off the rails and say something racist, sexist, lewd, immoral, etc. -- and Google wouldn't want such a thing splashed across the pages of the New York Times. So, at least for the next several years, templates will be how Assistants by the big tech companies will operate.

Now, a lean and mean startup could get away with using a model like this; they wouldn't have as much of a brand to protect. It's possible we might see that, but I wouldn't count on it.
  • Yuli Ban likes this

#88
Casey

Casey

    Member

  • Members
  • PipPipPipPipPipPip
  • 669 posts

Irritating that that sort of pussyfooting around will be necessary. People should have the common sense not to blame the parent company for whatever an AI says. The stupidity of people slows down progress as much as legitimate technological challenges do. I'll be glad whenever the technology advances enough that start-up groups can take control and go "fuck this, this is stupid, let's get things moving instead of tip-toeing around out of fear of the general public being a bunch of overreactionary dumbasses."


  • Yuli Ban and Alislaws like this

#89
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPip
  • 1,062 posts
I posted this in the status updates thread a couple days ago, but should also reproduce it here:
 
https://openai.com/blog/microsoft/
 
Microsoft's investment in OpenAI's vision towards building AGI will speed up its arrival.  I like OpenAI's attitude of seeing just how far you can go using much larger amounts of compute (and some algorithm tweaks).  
 
Not all that they work on may be immediately public:
 
https://twitter.com/...443233818652675
 

This mentality deeply worries me.

Exchange from Hacker news on
@OpenAI

+
@Microsoft


Anon: "keeping some innovations private may help commercialization, which may help raise more funds for OpenAI, getting us to AGI faster"

@gdb
"That's exactly how we think about it." ...


I still think, though, that the best shot at near-term AGI will be via a brain imitation route. It's possible that a inhuman AGI could be developed sooner; but if you want an AI that you can have an engaging conversation with, and that will seem like a kindred spirit, using lots of brain data may be your best shot. If you want an AI that can prove theorems better than the best mathematicians, using batshit-insane approaches -- so weird as to make it almost unreadable -- then probably you don't need human physiological data.
  • Yuli Ban and Alislaws like this

#90
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPip
  • 1,062 posts
This is an interesting paper:

Multi-turn Dialogue Response Generation with Autoregressive Transformer Models

https://arxiv.org/abs/1908.01841
 

In our experiments, we employ small and medium GPT-2 models (with publicly available pretrained language model parameters) on the open-domain Movie Triples dataset and the closed-domain Ubuntu Dialogue dataset. The models (with and without pretraining) achieve significant improvements over the baselines for multi-turn dialogue response generation. They also produce state-of-the-art performance on the two datasets based on several metrics, including BLEU, ROGUE, and distinct n-gram.


Basically, the authors tested to see how well various GPT-2 models (e.g. small and medium) can be used to generate coherent, diverse, relevant dialog. There are a few standard benchmarks to test how well neural net response generators are at holding a conversation, given limited context; and what the authors find is that GPT-2 / Transformers lead to large improvements over previous methods. This bodes well for the future of socialbots. These benchmarks are fairly noisy (e.g. context isn't easy to interpret; language errors; etc.) and not too large. What happens when much larger GPT-2-like models are tried, along with much larger, much higher-quality dialog datasets?

....

Interestingly, one of the authors of this paper is Erik T. Mueller. Ring a bell? He was one of the lead scientists on IBM's Watson Jeopardy! program. He left IBM, and now works for Capital One Conversation Research. His specialty is Commonsense Reasoning; though, he seemed to prefer symbolic logic-based methods for achieving it -- he's written a book on it, in fact. But he wears many hats, and has written a few papers on neural nets, too, including this one.
  • Yuli Ban likes this

#91
ralfy

ralfy

    Member

  • Members
  • PipPipPipPip
  • 137 posts

At best, like the current world. The "world," of course, refers to the predicament of the majority of the global population.



#92
tomasth

tomasth

    Member

  • Members
  • PipPipPipPip
  • 194 posts
https://mobile.twitt...130735011192832
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
  • Yuli Ban and starspawn0 like this

#93
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPip
  • 1,062 posts

I saw that tweet the other day.  Visual question-answering is coming along nicely; but probably video question-answering will be worth a lot more -- e.g. you won't have to pay people to sit and watch videos before answering questions (for biomedical applications, business applications, security, etc.).



#94
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPip
  • 1,062 posts
A new advance from Facebook:

https://twitter.com/...841832492081161

@facebookai
's RoBERTa is up on the super.gluebenchmark.com leaderboard, with an improvement of more than 15 percentage *points* over our BERT baseline.


That's close to human performance! Note that this super-hard benchmark includes the official Winograd Schemas test -- not the WNLI version that was "cracked" previously, that was part of the Glue benchmark. On this official version of WSC, Facebook's system achieved 89% accuracy, which is about what previous systems achieved on WNLI.

Continuing with Sam Bowman's thread on this achievement:

There's still headroom left for further work—our estimate of human performance is a very conservative lower bound. I'd also bet that the next five or ten percentage points are going to be quite a bit harder to handle.

Speaking for myself, though, it seems like we really are very good at solving language understanding tasks with naturalistic training distributions and IID train–test splits, at least for English.

Note that SuperGLUE represents the _hardest_ tasks out of the ~35 that were submitted to us. For most of the tasks that were proposed, BERT was already near ceiling.

The underlying tasks are mostly far from solved, of course. If you interact with these systems like these, it's easy to find cases where they fail badly.

But these cases tend to be rare in IID test sets, so it's not clear how much more room there is to evaluate progress this way.

There are alternatives emerging—adversarial filtering or the build-it-break-it approach, for example—but I think there are still hard open questions about how we should measure academic progress on real-world tasks, now that we really do seem to have solved the average case.


As I said before: virtual assistants are going to get a LOT better at understanding you. Maybe not right away; and it may happen gradually, and you may forget how bad they were by the time they get really good.
  • Casey, Yuli Ban and funkervogt like this

#95
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPip
  • 1,062 posts
The next component to making chatbots a lot smarter (at language understanding) is coming into place:

Nvidia just made it easier to build smarter chatbots and slicker fake news

https://www.technolo...cker-fake-news/
 

Artificial intelligence has made impressive strides in the last decade, but machines are still lousy at comprehending language. Just try engaging Alexa in a bit of witty banter.

Nvidia, the company that makes the computer chips that power many AI algorithms, thinks this is about to change and is looking to capitalize on an anticipated explosion. Software the chip maker is releasing makes it easier to build AI programs on its hardware that are capable of using language more gracefully. The new code could accelerate the development of new language algorithms, and make chatbots and voice assistants snappier and smarter.


It also runs much faster:

https://www.extremet...le-smarter-bots
 

Nvidia, though, has come up with a way to allow multiple GPUs to work on the language modeling task in parallel. Like with the other announcements today, they have open-sourced the code to make it happen. I’ll be really curious if the technique is specific to language models or can be applied to allow multiple-GPU training for other classes of neural networks.

Along with these developments and releasing the code on GitHub, Nvidia announced that they will be partnering with Microsoft to improve Bing search results, as well as with Clinc on voice agents, Passage AI on chatbots, and RecordSure on conversational analytics.


And Nvidia isn't even releasing their largest language model. They are taking OpenAI's approach of withholding it, for now, reasoning that it's too dangerous.

Nvidia blog post about the work: https://devblogs.nvi...bert-with-gpus/

The upshot is: It's one thing to hear about rather large improvements in NLU/NLP/NLG coming from the likes of Facebook and Google, but that doesn't mean you'll see it in chatbots right away. However, what Nvidia has done could shorten that lag between "in the lab" and "in production models".
  • Casey, Yuli Ban and johnnd like this

#96
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPip
  • 1,062 posts
One thing that might not have been clear from the previous post is that NVIDIA's model is more than 5 times the size of GPT-2-Large! I'm not sure whether NVIDIA has yet provided any examples of the upper-limits of what it can do -- maybe they are holding off, because they don't want to draw attention to how good it is (for safety reasons). They do mention things like "chatbots", "document summarization", and "email auto-completion" as potential applications.
  • Casey, Yuli Ban and johnnd like this




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users