Jump to content

Welcome to FutureTimeline.forum
Register now to gain access to all of our features. Once registered and logged in, you will be able to create topics, post replies to existing threads, give reputation to your fellow members, get your own private messenger, post status updates, manage your profile and so much more. If you already have an account, login here - otherwise create an account for free today!
Photo

What 2029 will look like


  • Please log in to reply
136 replies to this topic

#81
Alislaws

Alislaws

    Democratic Socialist Materialist

  • Members
  • PipPipPipPipPipPipPipPip
  • 2,005 posts
  • LocationLondon

This work implies that machine learning models based on nothing more than predicting word co-occurrence patterns, can absorb a lot of implicit scientific knowledge not directly trained for:
 
 
With Little Training, Machine-Learning Algorithms Can Uncover Hidden Scientific Knowledge 
 
 
https://newscenter.l...ific-knowledge/
 

"Without telling it anything about materials science, it learned concepts like the periodic table and the crystal structure of metals,” said Jain. “That hinted at the potential of the technique. But probably the most interesting thing we figured out is, you can use this algorithm to address gaps in materials research, things that people should study but haven’t studied so far.”


So... models like GPT-2 might have acquired a lot of scientific knowledge not explicitly mentioned in the training corpus.

 

That's incredible, Really excited to see how far/fast this progresses!


  • Yuli Ban likes this

#82
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPip
  • 1,238 posts
The state of AI up to late June 2019:

https://www.stateof.ai/

This doesn't include the latest XLNET stuff from Google or Alibaba's work on question-answering (MS-MARCO). Even so, the advances it discusses are impressive. The advances will continue like this for at least the next 5 years. Robotics is next to see major change. Speech Recognition was the first to be impacted; then came Computer Vision; after that was NLP; and now, finally, it's Robotics's turn to get slammed by Deep Learning. And after that, Reasoning (e.g. hard mathematical reasoning) will feel the burn.
  • Casey likes this

#83
johnnd

johnnd

    Member

  • Members
  • PipPip
  • 49 posts

The state of AI up to late June 2019:https://www.stateof.ai/Speech Recognition was the first to be impacted; then came Computer Vision; after that was NLP; and now, finally, it's Robotics's turn to get slammed by Deep Learning. And after that, Reasoning (e.g. hard mathematical reasoning) will feel the burn.


So, there was a speech recognition revolution before ImageNet? I always thought it was vision first, but I wasn't following this stuff as closely back then.
  • Yuli Ban and starspawn0 like this

#84
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPip
  • 1,238 posts
Yes, the first Deep Learning speech recognition breakthroughs occurred before success at Imagenet:

https://en.wikipedia...earning#History

The 2009 NIPS Workshop on Deep Learning for Speech Recognition[69] was motivated by the limitations of deep generative models of speech, and the possibility that given more capable hardware and large-scale data sets that deep neural nets (DNN) might become practical. It was believed that pre-training DNNs using generative models of deep belief nets (DBN) would overcome the main difficulties of neural nets.[70] However, it was discovered that replacing pre-training with large amounts of training data for straightforward backpropagation when using DNNs with large, context-dependent output layers produced error rates dramatically lower than then-state-of-the-art Gaussian mixture model (GMM)/Hidden Markov Model (HMM) and also than more-advanced generative model-based systems.[60][71]


  • Yuli Ban and johnnd like this

#85
ralfy

ralfy

    Member

  • Members
  • PipPipPipPip
  • 143 posts

Limits to growth kicks in.



#86
Yuli Ban

Yuli Ban

    Born Again Singularitarian

  • Moderators
  • PipPipPipPipPipPipPipPipPipPipPip
  • 20,670 posts
  • LocationNew Orleans, LA

 

The state of AI up to late June 2019:https://www.stateof.ai/Speech Recognition was the first to be impacted; then came Computer Vision; after that was NLP; and now, finally, it's Robotics's turn to get slammed by Deep Learning. And after that, Reasoning (e.g. hard mathematical reasoning) will feel the burn.


So, there was a speech recognition revolution before ImageNet? I always thought it was vision first, but I wasn't following this stuff as closely back then.

It apparently was concurrent development, but speech was a bit easier than vision.

I linked some decade-old Reddit threads on the very earliest days of deep learning, and that seems to be the general mood of things.


  • starspawn0 and johnnd like this

And remember my friend, future events such as these will affect you in the future.


#87
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPip
  • 1,238 posts
Hello, It's GPT-2 -- How Can I Help You? Towards the Use of Pretrained Language Models for Task-Oriented Dialogue Systems

https://arxiv.org/abs/1907.05774

This paper is written by Cambridge University and PolyAI people. You may recall I've written about PolyAI before. I believe Matthew Henderson is a co-founder. He was the student of Steve Young. The startup appears to be doing some really important dialog system research, and will probably get bought out by a large company like Google or Apple.

The paper has some really neat examples of what this GPT-2-based system can do. They claim that using a pre-trained GPT-2 language model they can bypass the traditional dialog management and domain-specific NLG modules. They still need to make a few tweaks, like add-in specific tokens for the different parties in a conversation, and also to refer to things like "telephone number" or "address"; but this is very minor compared to adding a dialog manager.

Although this is neat from a research perspective, I doubt Google or Apple will be doing this sort of thing in the near-future -- because there is too much risk that the system will say something rude, lewd, racist, sexist, cruel, immoral, etc. Template-based response generators are a way to avoid that; but they tend to give a wooden response. Still, a wooden response is probably preferable to large companies than being the target of a mass media article about how their Assistant said something racist.

....

Some examples of what it can do:

Visitor: I am looking for some places to go, do you have any attractions in [value_area]?

GPT-2: Sure! There are many colleges and museums in the area. Do you have a preference for type or price range?

Visitor: I really do not have a preference. Can you recommend a few?

GPT-2: How about the [attraction_name]?

Visitor: Sure. Also can you help me find a restaurant serving unusual food in the same part of town?

GPT-2: Unfortunately, there are no restaurants that match that description. Would you like to try a different area or type of food?


As you can see, the text being generated is complex, diverse, coherent, and germane to the questions asked by the visitor. It was generated without templates!

In principle, it should be possible to use GPT-2 + all the tweaks they mention in this article, to build an AI assistant that can help with a very broad set of complex tasks -- and even combinations of tasks. The only problem, though, is that you can't predict when it might go off the rails and say something racist, sexist, lewd, immoral, etc. -- and Google wouldn't want such a thing splashed across the pages of the New York Times. So, at least for the next several years, templates will be how Assistants by the big tech companies will operate.

Now, a lean and mean startup could get away with using a model like this; they wouldn't have as much of a brand to protect. It's possible we might see that, but I wouldn't count on it.
  • Yuli Ban likes this

#88
Casey

Casey

    Member

  • Members
  • PipPipPipPipPipPip
  • 676 posts

Irritating that that sort of pussyfooting around will be necessary. People should have the common sense not to blame the parent company for whatever an AI says. The stupidity of people slows down progress as much as legitimate technological challenges do. I'll be glad whenever the technology advances enough that start-up groups can take control and go "fuck this, this is stupid, let's get things moving instead of tip-toeing around out of fear of the general public being a bunch of overreactionary dumbasses."


  • eacao, Yuli Ban and Alislaws like this

#89
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPip
  • 1,238 posts
I posted this in the status updates thread a couple days ago, but should also reproduce it here:
 
https://openai.com/blog/microsoft/
 
Microsoft's investment in OpenAI's vision towards building AGI will speed up its arrival.  I like OpenAI's attitude of seeing just how far you can go using much larger amounts of compute (and some algorithm tweaks).  
 
Not all that they work on may be immediately public:
 
https://twitter.com/...443233818652675
 

This mentality deeply worries me.

Exchange from Hacker news on
@OpenAI

+
@Microsoft


Anon: "keeping some innovations private may help commercialization, which may help raise more funds for OpenAI, getting us to AGI faster"

@gdb
"That's exactly how we think about it." ...


I still think, though, that the best shot at near-term AGI will be via a brain imitation route. It's possible that a inhuman AGI could be developed sooner; but if you want an AI that you can have an engaging conversation with, and that will seem like a kindred spirit, using lots of brain data may be your best shot. If you want an AI that can prove theorems better than the best mathematicians, using batshit-insane approaches -- so weird as to make it almost unreadable -- then probably you don't need human physiological data.
  • Yuli Ban and Alislaws like this

#90
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPip
  • 1,238 posts
This is an interesting paper:

Multi-turn Dialogue Response Generation with Autoregressive Transformer Models

https://arxiv.org/abs/1908.01841
 

In our experiments, we employ small and medium GPT-2 models (with publicly available pretrained language model parameters) on the open-domain Movie Triples dataset and the closed-domain Ubuntu Dialogue dataset. The models (with and without pretraining) achieve significant improvements over the baselines for multi-turn dialogue response generation. They also produce state-of-the-art performance on the two datasets based on several metrics, including BLEU, ROGUE, and distinct n-gram.


Basically, the authors tested to see how well various GPT-2 models (e.g. small and medium) can be used to generate coherent, diverse, relevant dialog. There are a few standard benchmarks to test how well neural net response generators are at holding a conversation, given limited context; and what the authors find is that GPT-2 / Transformers lead to large improvements over previous methods. This bodes well for the future of socialbots. These benchmarks are fairly noisy (e.g. context isn't easy to interpret; language errors; etc.) and not too large. What happens when much larger GPT-2-like models are tried, along with much larger, much higher-quality dialog datasets?

....

Interestingly, one of the authors of this paper is Erik T. Mueller. Ring a bell? He was one of the lead scientists on IBM's Watson Jeopardy! program. He left IBM, and now works for Capital One Conversation Research. His specialty is Commonsense Reasoning; though, he seemed to prefer symbolic logic-based methods for achieving it -- he's written a book on it, in fact. But he wears many hats, and has written a few papers on neural nets, too, including this one.
  • Yuli Ban likes this

#91
ralfy

ralfy

    Member

  • Members
  • PipPipPipPip
  • 143 posts

At best, like the current world. The "world," of course, refers to the predicament of the majority of the global population.



#92
tomasth

tomasth

    Member

  • Members
  • PipPipPipPipPip
  • 228 posts
https://mobile.twitt...130735011192832
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
  • Yuli Ban and starspawn0 like this

#93
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPip
  • 1,238 posts

I saw that tweet the other day.  Visual question-answering is coming along nicely; but probably video question-answering will be worth a lot more -- e.g. you won't have to pay people to sit and watch videos before answering questions (for biomedical applications, business applications, security, etc.).



#94
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPip
  • 1,238 posts
A new advance from Facebook:

https://twitter.com/...841832492081161

@facebookai
's RoBERTa is up on the super.gluebenchmark.com leaderboard, with an improvement of more than 15 percentage *points* over our BERT baseline.


That's close to human performance! Note that this super-hard benchmark includes the official Winograd Schemas test -- not the WNLI version that was "cracked" previously, that was part of the Glue benchmark. On this official version of WSC, Facebook's system achieved 89% accuracy, which is about what previous systems achieved on WNLI.

Continuing with Sam Bowman's thread on this achievement:

There's still headroom left for further work—our estimate of human performance is a very conservative lower bound. I'd also bet that the next five or ten percentage points are going to be quite a bit harder to handle.

Speaking for myself, though, it seems like we really are very good at solving language understanding tasks with naturalistic training distributions and IID train–test splits, at least for English.

Note that SuperGLUE represents the _hardest_ tasks out of the ~35 that were submitted to us. For most of the tasks that were proposed, BERT was already near ceiling.

The underlying tasks are mostly far from solved, of course. If you interact with these systems like these, it's easy to find cases where they fail badly.

But these cases tend to be rare in IID test sets, so it's not clear how much more room there is to evaluate progress this way.

There are alternatives emerging—adversarial filtering or the build-it-break-it approach, for example—but I think there are still hard open questions about how we should measure academic progress on real-world tasks, now that we really do seem to have solved the average case.


As I said before: virtual assistants are going to get a LOT better at understanding you. Maybe not right away; and it may happen gradually, and you may forget how bad they were by the time they get really good.
  • Casey, Yuli Ban and funkervogt like this

#95
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPip
  • 1,238 posts
The next component to making chatbots a lot smarter (at language understanding) is coming into place:

Nvidia just made it easier to build smarter chatbots and slicker fake news

https://www.technolo...cker-fake-news/
 

Artificial intelligence has made impressive strides in the last decade, but machines are still lousy at comprehending language. Just try engaging Alexa in a bit of witty banter.

Nvidia, the company that makes the computer chips that power many AI algorithms, thinks this is about to change and is looking to capitalize on an anticipated explosion. Software the chip maker is releasing makes it easier to build AI programs on its hardware that are capable of using language more gracefully. The new code could accelerate the development of new language algorithms, and make chatbots and voice assistants snappier and smarter.


It also runs much faster:

https://www.extremet...le-smarter-bots
 

Nvidia, though, has come up with a way to allow multiple GPUs to work on the language modeling task in parallel. Like with the other announcements today, they have open-sourced the code to make it happen. I’ll be really curious if the technique is specific to language models or can be applied to allow multiple-GPU training for other classes of neural networks.

Along with these developments and releasing the code on GitHub, Nvidia announced that they will be partnering with Microsoft to improve Bing search results, as well as with Clinc on voice agents, Passage AI on chatbots, and RecordSure on conversational analytics.


And Nvidia isn't even releasing their largest language model. They are taking OpenAI's approach of withholding it, for now, reasoning that it's too dangerous.

Nvidia blog post about the work: https://devblogs.nvi...bert-with-gpus/

The upshot is: It's one thing to hear about rather large improvements in NLU/NLP/NLG coming from the likes of Facebook and Google, but that doesn't mean you'll see it in chatbots right away. However, what Nvidia has done could shorten that lag between "in the lab" and "in production models".
  • Casey, Yuli Ban and johnnd like this

#96
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPip
  • 1,238 posts
One thing that might not have been clear from the previous post is that NVIDIA's model is more than 5 times the size of GPT-2-Large! I'm not sure whether NVIDIA has yet provided any examples of the upper-limits of what it can do -- maybe they are holding off, because they don't want to draw attention to how good it is (for safety reasons). They do mention things like "chatbots", "document summarization", and "email auto-completion" as potential applications.
  • Casey, Yuli Ban and johnnd like this

#97
tomasth

tomasth

    Member

  • Members
  • PipPipPipPipPip
  • 228 posts

openai are releasing the 774 million parameter GPT-2

 

https://openai.com/b...onth-follow-up/

 

I wonder if the possibility of a 5x of the 1.5B (even if its hold off) , have anything to do with it , or its just realization that those language model are not that dangers.


  • eacao and Casey like this

#98
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPip
  • 1,238 posts
Here is some more on GPT-2:

https://twitter.com/...635721274417152
 

A @OpenAI GPT-2 prompt taken from @GaryMarcus's The Algebraic Mind, showing what neural nets couldn't do. Sometimes a quantitative change in model/data complexity can give rise to qualitatively different behavior.


The output that GPT-2 gave was (prompt in bold):
 

A rose is a rose
A lilly is a lilly
A blicket is a
blicket
A hoop is a hoop
A crook is a crook
A tine is a tine
And it was a day like this when a girl, when she heard someone call
her by her name,
But her heart beat for the sun.
That's what a little girl thinks about day by day.

etc. etc.

The point here is that "blicket" is not a word, and the beginning of the prompt was part of a pattern to see if GPT-2 would realize that you are supposed to repeat the word or pseudo-word before the "is". GPT-2 inferred correctly, and wrote down "blicket". Neural nets couldn't do this when Marcus wrote his book; and Steven Hanson's point was that sometimes scale matters -- if you train the neural net with lots more data, you get a phase-transition in ability, to where it can suddenly correctly solve Marcus's challenge task.

The Twitter thread continues with Marcus saying:
 

honored and flattered and it’s very very cute, but it still can’t just stick to the pattern...


I'm not sure what he means here -- perhaps he means that it should have stopped with that "blicket", or maintained the pattern line-by-line further down.

Hanson responds:
 

I think it's interpreting the prompt as the start to a poem or some other literary device. If you add numbering it gets the hint:


And then GPT-2 writes (bold is the prompt again):
 

1. A rose is a rose
2. A lilly is a lilly
3. A blicket is a
blicket
4. A horse's tail is a horse's tail
5. A dog is a dog
6. A lion is a lion
7. A sheep is a sheep
8. A chicken is a chicken
9. A donkey is a donkey


The point being that if you add numbers, it knows you aren't asking it to write poetry. It's interesting that it knew to count correctly down to 9; in a later example, it counted correctly down to 26, which is amazing! So, not only did it correctly address Marcus's challenge, but it also knew how to solve the even harder problem of correctly enumerating an itemized list.

Marcus responds
 

now that is interesting. do you get essentially the same result every time?


It seems that you do... at least most of the time.

Further down in one of those threads, Gwern writes:
 

I've played around with some samples like sorting lists, and it *sorta* works. It's strange. You don't want to think that it's actually managed to learn something about abstract patterns from next-char prediction, but if not, what on earth human text is responsible for this?


It will be interesting to see what the next generation of GPT-2-like models can do, when trained on much larger datasets (even bigger than NVIDIA's)... and even images and video. It feels like we've crossed the Rubicon, and there's no going back!
  • Casey, Yuli Ban and johnnd like this

#99
funkervogt

funkervogt

    Member

  • Members
  • PipPipPipPipPipPip
  • 771 posts

From 2015:

 

 

The future of writing looks even more fun. Artificial intelligence is nowhere near the level yet where you can explain an idea to a computer in ordinary conversation and tell it to get on with it, but it will be one day, fairly soon. 

https://timeguide.wo...-of-publishing/


  • Yuli Ban and starspawn0 like this

#100
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPip
  • 1,238 posts

John Carmack interview on Joe Rogan's show:

 

https://youtu.be/udlMSe5-zP8

 

Around 48 minutes in he gives a prediction about AGI.  He thinks the first hints will be here within 10 years.


  • Yuli Ban and funkervogt like this




2 user(s) are reading this topic

1 members, 1 guests, 0 anonymous users


    Zaphod