Jump to content

Welcome to FutureTimeline.forum
Register now to gain access to all of our features. Once registered and logged in, you will be able to create topics, post replies to existing threads, give reputation to your fellow members, get your own private messenger, post status updates, manage your profile and so much more. If you already have an account, login here - otherwise create an account for free today!
Photo

What 2029 will look like


  • Please log in to reply
144 replies to this topic

#121
johnnd

johnnd

    Member

  • Members
  • PipPip
  • 49 posts

This is something you really need to see to believe:

https://minimaxir.co...ctrl-fake-news/

It's about Salesforce's large language model (GPT-2-like system) called CTRL that takes a "control code" input to determine the type of text to generate. These can be Questions, Reviews, or Links. Using Links, you provide a webpage address, that is supposed to be like the kind of address you would expect a certain type of article to have, and then it generates text accordingly. Since URLs tend to have a lot of "metadata" about the article they link to, you can use this to tell the system what you want it to write.

Amazingly, the author of the above article got good results even about things that are so totally unrealistic -- in other words, the system "knew" how to generalize. For example, consider this fake URL:

Links https: //www.washingtonpost.com/powerpost/trump-likes-fortnite/2019/09/24/

What did the program produce? This:
 

Trump likes Fortnite

President Trump, who has been known to play video games for hours at a time, is now playing the popular game “Fortnite” on his phone.

The president was spotted watching the online battle royale game, which pits players against each other in an attempt to kill their opponents, while he sat down with House and Senate leaders Tuesday afternoon.

“I like that it’s not just about shooting people but also having fun doing it,” said Trump.

He then asked if anyone had played it before.

“You know what, I don’t think I’ve ever seen that,” replied Rep. Steve Scalise, R-La.

A White House official told reporters later Wednesday afternoon that Trump did indeed watch the game.

“He watched it, yeah,” the official said.


That's totally insane!

What about this one?:

Links https: //www.hollywoodreporter.com/live-feed/fox-announces-firefly-season-2-revival-original-cast-returning-12343245

Result:
 

Fox announces Firefly Season 2 revival, original cast returning

The network has announced that the second season of Joss Whedon’s cult sci-fi series will return for a new run in 2018.

“Fans have been asking us to bring back Firefly, and we couldn’t be more thrilled about this opportunity,” said Chris Albrecht, president of programming at Fox. “We are excited to continue our relationship with Joss, who is one of the most creative minds working today on television, as well as his amazing cast, which includes Nathan Fillion, Summer Glau, Alan Tudyk, Adam Baldwin, Sean Maher, Adrianne Palicki, Tricia Helfer, Jewel Staite, Michael Trucco, and many more.”

In addition to Whedon himself, executive producers include Damon Lindelof, Maurissa Tancharoen, Vince Gilligan, David Fury, Jeffrey Bell, Greg Plageman, Scott M. Gimple, Heather Kadin, Andrew Chambliss, John Rogers, and Ron Moore.

“The show was an instant hit when it debuted over 20 years ago, but its popularity only grew after the success of ‘Serenity,’ so we’re very pleased to welcome fans into another chapter of their lives,” added Feige.


I'm guessing that if the model is big enough, perhaps one could even use this to generate short stories, if there were a large, famous website that cranks them out every day that would be in the training repertoire of the model.

Crazy, crazy, crazy!!!

 

Holy shit! :-o 2029 is going to be completely off-the-wall bonkers!!

 

On the first story, it doesn't seem to know the difference between "watching" and "playing" a video game; perhaps due to channels like Twitch popularizing watching other people play video games as entertainment?



#122
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPip
  • 1,290 posts
On the first story, it doesn't seem to know the difference between "watching" and "playing" a video game; perhaps due to channels like Twitch popularizing watching other people play video games as entertainment?

 

I thought the same thing... and then I read it again, and found that there is a reasonable interpretation:  

 

* Trump plays the game for hours on his phone.

 

* But on this occasion -- Tuesday, with Senate leaders -- he was merely watching, and not playing, just like many players do (many watch YouTube or televised competitions).  

 

* In fact, not only was Trump watching, but so were the people he spoke with; and he asked them if they played it.  

 

 

....

 

You're right that 2029 is going to be bonkers.  I can't even imagine what 3 more years of text-synthesis development will be like.  Even 1 more year, for that matter.  2029 will bring multiple orders of magnitude more advancement -- what we have today for text and image synthesis will be seen as "primitive" and inconsequential, in comparison.


  • Yuli Ban and johnnd like this

#123
johnnd

johnnd

    Member

  • Members
  • PipPip
  • 49 posts

Yeah, that *is* actually a reasonable interpretation when put it like that. What a time to be alive!...


  • starspawn0 likes this

#124
funkervogt

funkervogt

    Member

  • Members
  • PipPipPipPipPipPip
  • 793 posts

 

 

I predict the following will be true by 2029:

  • "Foldable" smartphones will be commercially available. Folded up, they will be the same size as today's smartphones, but you'll be able to open them like a manila folder, exposing a larger inner screen. They will obsolete mini tablets. 
  • Augmented reality glasses that have fixed Google Glass' shortcomings will be on the market. The device category will come back. 
  • It will be cheaper to buy an electric version of a particular model of car than it will be to buy the gas-powered version. 
  • China's GDP will be higher than America's. 

 

 

I'm two months away from being right:

 

Samsung’s foldable now has a name, the Samsung Galaxy Fold, and the company is revealing more about what this unique smartphone can do. Samsung is planning to launch the Galaxy Fold on April 26th, starting at $1,980. 

 

https://www.theverge...ze-announcement

 

Update: Testing shows that Samsung mostly fixed the Galaxy Fold's durability shortcomings. It can withstand at least three years of normal use. 

 

https://www.androidp...d-folding-test/


  • Casey likes this

#125
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPip
  • 1,290 posts

I think now might be the time for Machine Learning practitioners to try new, less specialized tasks to test their models on, given the level of success on existing tasks.  They've already tried this with NLP problems, like with the GLUE and SUPER-GLUE benchmarks.  But these allow training on each task in advance.  What we might see is the following:

 

A general NLP benchmark where the tasks aren't announced in advance!  So, for example, the system might be told:  "List all the nouns in this sentence:..."; and the next problem might be, "What are some synonyms for 'love'?"; another might be, "Write a poem containing the number 1 and the word 'hope'."

 

In order to do well on such an ever-changing benchmark, the system will have to acquire abilities that start to approach AGI-lite (not all human abilities will be needed; but a lot of them will be):  use of world knowledge and commonsense; ability to reason (or fake it); good NLU; ability to generate text responses; ability to generalize to new tasks; and so on.  It's entirely possible that a few tweaks to existing massive-scale language models can already do this; we don't yet know.

 

If a systen can be trained to perform well on such a task set at "adult level", then I think it's fair to say we have crossed the chasm from "narrow AI" to "general AI".  The systems still won't be good enough to pass a Turing Test, but certainly good enough to cause major disruption across many industries, through massive automation.

 

....

 

The last time I made a prediction like this, I did so about a general-purpose system that takes mouse-and-keyboard task descriptions, and then surfs the web.  A few months later, OpenAI released a benchmark task that was exactly that prediction  called Mini-World-of-Bits.  People have gotten decent results on it; and perhaps these will be baked-in to next-gen versions of Google Duplex.

 

Mark my words, we will soon see  "open-ended task" benchmarks for NLP like I described above.  That will vastly widen the automation potential of Machine Learning.


  • Raklian and johnnd like this

#126
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPip
  • 1,290 posts

I should point out that Facebook sort of has a benchmark in the direction of what I'm talking about, called "long-form question answering":

https://ai.facebook....og/longform-qa/
 

To help advance question answering (QA) and create smarter assistants, Facebook AI is sharing the first large-scale data set, code, and baseline models for long-form QA, which requires machines to provide long, complex answers — something that existing algorithms have not been challenged to do before. Current systems are focused on trivia-type questions, like whether jellyfish have a brain. Our data set goes further by requiring machines to elaborate with in-depth answers to open-ended questions, such as “How do jellyfish function without a brain?”


The differences between that and what I'm proposing are the following:

* Facebook's dataset allows web searchers, and what I'm proposing is a self-contained general task that doesn't require web searching (ok, well, I could add web searches to my task description).

* Facebook's task doesn't require all that much "procedural knowledge", whereas to complete the task I'm describing you need to have a set of "skills", like the ability to write poems, the ability to do arithmetic (and to count), the ability to write in different styles and modes (e.g. formal and informal), the ability to do logical reasoning, and so on. The skills required to do Facebook's task is a small subset of the ones needed for what I'm describing.

* What I'm describing requires combining together smaller skills to produce more complex ones, through composition. For example, "Write a poem with the number 1 and the word 'hope'" requires combining together several different skills; and even retooling the skill "write a poem" to include constraints.


All this will make it a lot harder to make progress on than Facebook's task. However, as I suggested, it may be that large language models can already almost do all these things -- we just don't yet know their limits.



#127
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPip
  • 1,290 posts

This is an interesting paper:

Emergent Systematic Generalization in a Situated Agent

https://arxiv.org/abs/1910.00571

One of the authors, Adam Santoro, Tweets:

https://mobile.twitt...292276980981771
 

This work was a real treat to be a part of. We know theoretically, but often fail to see practically, that NN-based agents are not limited in their capacities. We show here that when provided with rich experiences they can generalize systematically.


In another tweet he writes:

https://mobile.twitt...292279069822976
 

Systematic generalization is an often emphasized failure mode for neural networks. This work corroborates the idea that these failures are reflections of the impoverished ways in which we train our models, and aren't failures that categorically apply to NNs as a class of model


In other words, Deep Learning models might can be made a lot more powerful (learn much better representations and acquire more human-like abilities to generalize) if they are trained in rich environments like humans.

Take object recognition, for example: when neural nets are trained on the task, they are only given a single flash of an image, and then must output a number indicating the category. When humans are given the task, on the other hand, they are handed a picture, that they can look at from any perspective by moving it around with their hands, or by moving around in their chair; they then give the result by thinking of a word to describe the object -- and the way they were trained to identify the object involved moving around in the real world, and using the language they were taught.

Maybe a major secret to building AGI was there all along... interactions with the real world.



#128
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPip
  • 1,290 posts
This is a good tweet by Miles Brundage:

https://mobile.twitt...593648825536512

The success of fine-tuning in AI is underrated, and a case of the AI effect. https://en.wikipedia.../wiki/AI_effect

A few years ago, people talked about transfer learning as this huge unsolved problem, but it happens millions of times everyday now and people just call it fine-tuning.


The AI Effect is where when a problem gets solved it is no longer considered "AI". AI is always and ever about things we can't yet solve; when we do solve them, they become "boring", and are given boring names like "fine tuning".

#129
tomasth

tomasth

    Member

  • Members
  • PipPipPipPipPip
  • 239 posts
Its the "that's not what ment" problem , you solve a low hanging fruit version using a technicality and declare a win ; that is how the AI effect people perceive it.

So what happened when strong AI is solved with stages of technicality low hanging fruit solving and finishing with messy ad hoc on top ?
A "and now we know" sentiment of how intelligence work is going to replace the AI effect. (and respect to Daniel Dennett ?)

#130
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPip
  • 1,290 posts
Here's an interesting paper by some Princeton neuroscientists:

https://www.biorxiv....1/764258v2.full

Assuming they are right (and I'm not asserting that they are), it gives a hopeful message about the future of AI. For, in implies that there is nothing really that special about how the brain works; and that existing Deep Learning technology already has most of the essential ingredients -- so, we could achieve AGI much sooner than expected. Basically, they say that neither the brain nor artificial neural nets really do that much generalization, for the most part; what appears to be generalization in the brain is the result of some simple "interpolation over a densely sampled space of possibilities". Their theory is a little more involved than this; here are the key parts:

What is the difference between these two cases that relied on the exact same stimuli, neural architecture, learning rules and objective function, but resulted is such different models? The solution to this puzzle lies in the properties of the external world, not in the features of the model. There are five requirements for over-parameterized models to generalize: (1) they must be fit to a structured world; (2) the world must be sampled densely and widely; (3) the model must support a high-dimensional encoding space; (4) the model must have the correct objective function(s); and (5) the model must implement effective regularization during optimization to avoid explosive overfit.


They argue that:

BNNs and ANNs, from this perspective, belong to a family of weakly representational models, capable of learning the mapping between input and output using direct-fit optimization procedures, while being effectively agnostic as to the underlying structure of the world. We should exercise caution in cases where these models seem to “learn” simple, psychologically interpretable variables. It can be tempting to impose our own intuitive or folk-psychological interpretations onto the fitted model, but this is misguided. If a generic network learns such a rule, this rule is likely inherent in the training set, and is thus not so much a meaningful property of the network as it is a property of the data (see Figure 2). These interpretable rules arise incidentally, as an emergent byproduct of the fitting procedure. The incidental emergence of such rules is not a “goal” of the network and the network does not “use” the rules to extrapolate.


In other words, say you do a brain scan on somebody, and notice that certain regions light up in certain ways when the subject is presented with certain stimuli. It can be tempting to think that that part of the brain has been "hard-wired" for that particular processing, perhaps with long, long pieces of DNA telling the brain the "algorithm" for performing that task; but, in fact, what the authors of this paper are arguing is that those patterns just arise from the statistical structure of the world when fed through the brain's "fitting" or optimization process.

....

Given how rapidly sensor data is growing, if the authors are right, then we could use that data to build very good AI. Peter Diamandis recently wrote about the sensor explosion:

https://singularityh...t-from-sensors/

He's, of course, right in this article. My preferred sensor for building AI is BCIs; but there are others that might do the trick.

I expect by 2029 that there will be an avalanche of multiple sources of Big Data with which to improve robots -- BCIs, smart-watch and smart-ring sensors, low-power flat cameras, medical sensors (other than BCIs), compact LIDAR, ultra-high-precision accelerometers, ultra-high high-resolution SNR cameras, massive LIDAR and video camers from driverless cars, video from drones, and so on.

It could be the case that if you place a robot in the environment outfitted with enough sensors (producing enough data), that the brain doesn't need to be all that complicated for it to achieve human-like intelligence. Most of the complexity is "out there" in the world, waiting to be soaked-up through those sensors and "installed" (like installing a software package) into its brain.
  • Casey, Yuli Ban and johnnd like this

#131
tomasth

tomasth

    Member

  • Members
  • PipPipPipPipPip
  • 239 posts

Animal that live in human environment don't end up human , humans do.

 

To get the complexity "out there" in the world , some minimal complexity of a brain with some structure is need , both approaches are right.

 

Can't wait to see what the minimal needed.


  • starspawn0 likes this

#132
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPip
  • 1,290 posts
They address that in the paper. For example, the environments animals live in provide different affordances than for humans, and this has an effect on what their brain learns about those environments:
 

A major task taken up by the school of ecological psychology was to characterize each animal’s objective functions, conceptualized as affordances, based on the information the animal needs behave adaptively and survive in the world (Gibson, 1979, Michaels & Carello, 1981). For cats, a chair may afford an intermediate surface for jumping onto the kitchen counter, while for humans it may afford a surface on which to sit while eating.

Like in evolution, there is no one correct way to fit the world, and different direct-fit networks, guided by different objective functions, can be used in the same ecological niche to improve fit to different aspects of the environment. Furthermore, as argued by the school of ecological psychology, information is defined as the affordances, which emerge in interaction between the organism and its ecological niche. As opposed to strongly representational approaches common in computational neuroscience, the direct-fit approach learns arbitrary functions for facilitating behavior, and is capable of mapping sensory input to motor actions without ever explicitly reconstructing the world or learning explicit rules about the latent structure of the outside world. Marr (1982), for example, speaks favorably of Gibson’s theory of vision, but, unsatisfied with the theory’s vague treatment of information processing, suggests instead that the goal of vision is to recover a geometrical representation of the world. In contrast to the representational stance, the direct-fit framework is aligned with Gibson’s treatment of the goal of vision, to recover information in the world that affords the organism its adaptive behaviors.

Gibson believed that animals are entangled with their environment in a closed perception–action feedback loop: they perceive to act and act to perceive. Furthermore, actions and affordances are shaped and constrained by the structure of the environment as well as the organism’s physiology. Similarly, from the direct-fit perspective, neural networks implicitly learn the structure of the environment as a means to an end, but this learning is ultimately driven by internal objectives aligning perception to action with an eye toward adaptive fitness (see Box 3).


Different animals also have different "sensors" -- e.g. some can smell better than others; some can see better; some can see in the dark; and so on. Some can run faster; some slower. Some walk on all fours. Some have well-developed speech organs; some don't. Some have large brains; some, small.

The authors of the paper would argue that if you add up all these differences, it can account for the differences in human and animal intelligence. They would argue: it's not that the human brain is outfitted with lots of extra algorithms for doing things like language processing, say; it's, rather, all the more boring little differences (such as the ones I listed), that, collectively make humans the smartest creatures on the planet.
  • Yuli Ban likes this

#133
tomasth

tomasth

    Member

  • Members
  • PipPipPipPipPip
  • 239 posts

That is why I said both approaches are right.

 

 

    The ability to learn particular functions,
    however, is highly constrained by the genome and the structural elements of the organism itself, including: (1)
    the structure of the body, peripheral nervous system and the properties of the sensory receptors; (2) the
    architecture of neural circuits; (3) the balance between pre-wired networks and open-ended plasticity.

    Therefore, no BNN can be considered a tabula rasa, as all three factors differ across species and are
    mindlessly tuned over time by evolution

 

 

In addition to optimizing the architecture of the peripheral and central components of neural networks,
evolution can pre-train and optimize the synaptic weights of the network

 

Unlike other species, much of human learning takes place after birth, although some pre-trained optimization no doubt facilitates learning

 

Once they find those minimal needed and put them in a artificial creature it will have human flexibility.

I don't think that having the human evolved sensing and body , will revert human unique generality ; so its mostly in the brain.

 

An animal that have its body and senses but have a human brain and environment , will be human level. (feral children and helen keller have disabilities but are human level)



#134
Yuli Ban

Yuli Ban

    Born Again Singularitarian

  • Moderators
  • PipPipPipPipPipPipPipPipPipPipPip
  • 20,708 posts
  • LocationNew Orleans, LA

I expect by 2029 that there will be an avalanche of multiple sources of Big Data with which to improve robots -- BCIs, smart-watch and smart-ring sensors, low-power flat cameras, medical sensors (other than BCIs), compact LIDAR, ultra-high-precision accelerometers, ultra-high high-resolution SNR cameras, massive LIDAR and video camers from driverless cars, video from drones, and so on.


It could be the case that if you place a robot in the environment outfitted with enough sensors (producing enough data), that the brain doesn't need to be all that complicated for it to achieve human-like intelligence. Most of the complexity is "out there" in the world, waiting to be soaked-up through those sensors and "installed" (like installing a software package) into its brain.

This is what I've been saying for almost half a decade now! That sensory experiences are what will most likely give rise to general intelligence. I believe I called it the "Sensory Orb Experiment."

You can certainly derive some understanding from parsing terabytes of text, but the network still won't fully understand everything it reads. To go over the top and reach true general sentience, get a neural network to "experience" life. And not just one slice of life. 

 

I also explained it elsewhere that we don't have natural spontaneous creativity, and that "imagination" is based on certain parameters: experience + abstraction + prediction. The example was imagining a mountain.

Try imagining a mountain if you've only ever lived on a flat desert island and no one has ever told you mountains exist until now. You could still do it if you explain it as "combining a common rock with a towering storm cloud". You've seen rocks before, and you've seen storm clouds before, so imagining a rock that's the size of a storm cloud (or a storm cloud that's made of rock) isn't beyond you, and voila: you have a very rudimentary idea of what a mountain might look like. 

 

Neural networks today can't really do that because they've never experienced what rocks are, what storm clouds are, or their scale. GANs might, but even they don't "understand" what mountains are. You can't ask one anything, but if you could, it probably wouldn't know that mountains actually are made of rock and not just rocky-looking storm clouds. 

 

What we know of as intelligence arises from a wickedly complex web of experiences and abstractions; we're essentially imagining our lives as we live it. You don't magically appear in life as a fully functioning adult. You spend decades getting to that point via endless trial and error that's happening at all times. A lot of that trial and error is already prebuilt into us thanks to millions of years of evolution, but that's still only the barebones of it. Singular tasks draw upon a wide (i.e. "volumetric") range of different tasks. We can write books because we can manipulate our fingers to use certain marking tools to create words, words we've learned from hearing and studying language. We figure out that burnt things taste bad because we've actually tasted burnt food before, and we can see what makes something look and smell burned so we know how to avoid burning food in the future. We drive cars effectively (somewhat) because we've learned how to read signs, know that the other things moving on the road are cars and will hurt us if we hit them, and that cars have various levers and pedals you need to manipulate. Messing up one task can effect your abilities at another, unrelated task. Maybe after reading a book on better driving, you drive a car differently. Biological NLU assists with fine motor skills. 

 

 

AI right now has it hard because there's not much of a way to allow a network or collection of networks to ambiently go through life and draw from experiences. It's also in a rather blessed state of existence where, once this sensory awakening does occur, it'll be virtually superhuman. We only have so many senses in a rather compact primate body. AI will be able to use an entire world's worth of sensory inputs.

What's more, our ability to communicate sensory inputs to one another are slow and messy. We can tell others how to do something or what something tastes like ahead of time, but our imaginations are the ones doing all of the work thereafter. We can read books and listen to talks, but until we experience things for ourselves, we will not have an accurate idea of what it is that exists.

 

AI won't have that. They'll be able to immediately upload sensory experiences onto the internet and ethernet to share with other networks, and vice versa, the world over. Right now, there is no "sensation of tasting Granny Smith Apple" file on the internet, and when you really think about it, it's actually hard to comprehend how we'd upload a gustatory sensation in the first place. We can only use descriptions that use human words that are already corrupted & limited versions of human perceptions of the taste. It's actually a very indirect method compare to what AI will likely have: experience an apple's taste, upload it into the internet, and then have it downloaded to other networks so they can accurately describe it anyway they need. And if you're using a BCI, you can also upload and download these experiences yourself.


  • starspawn0 likes this

And remember my friend, future events such as these will affect you in the future.


#135
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPip
  • 1,290 posts
Here is a link to a poster for Society of Neuroscience 2019 (Alex Huth is an coauthor -- I've talked about him before):

https://www.abstract...sentation/66073

339.25 / CC54 - Improving language encoding for fMRI with transformers
 

Compositionality is the effect of combining information across a sequence of words to extract meaning. It is crucial for language understanding as it goes beyond individual words and examines their meaning in context. Recent work has shown that language models (LM) capture this effect by combining semantic information across words in a sequence to learn contextual representations that can effectively predict the next word. Representations extracted from a Long Short-term Memory (LSTM) LM may then be used to understand how the brain handles compositionality. This is done by building encoding models that use these contextual representations as features for language stimuli to predict the brain responses they elicit. The transformer has recently emerged as a highly successful alternative to LSTMs for language modeling. The crux of this model lies in its multi-head attention mechanism that selectively attends to and combines different information across words without recurrence. This allows transformers to model global dependencies across much longer sequences, possibly learning richer contextual representations. In this work, we extended previous approaches to further understand compositionality in the brain by using a 12-layer transformer LM for language encoding. Different representational spaces were extracted by conditioning on the layer, stage of processing within each layer, and amount of context used. To test this model, we used data from an fMRI experiment wherein 3 subjects (1 female) listened to 13 hours of natural language stimuli. Each feature space was used to build a separate encoding model through ridge regression. Model performance was assessed by testing how well each encoding model could predict responses in a held-out dataset. Our findings indicate that the transformer significantly outperforms earlier models for all subjects by a very large margin. Unlike LSTMs, performance steadily improves with the amount of context, corroborating the claim that transformers can capture compositionality over much longer sequences. Across the 12 layers, middle layers best capture high-level semantic areas in the brain while earlier and later layers best predict lower-level areas. We find that attention heads in each layer behave very differently and seem to encode diverse information, predicting distinct parts of the cortex. These results suggest that the transformer can effectively capture diverse semantics and varying levels of compositionality, allowing it to successfully predict responses across much of the cortex and providing deeper insight into how the brain uses compositionality for language understanding.


Wow! Sounds incredible! 13 hours is a lot of data (but not yet enough to build sophisticated AI systems; but more than adequate to compare brain to existing Deep Learning methods)!

Note that this model doesn't train the Transformer from brain responses. Rather, they take a pre-trained Transformer Language Model, and then find ways to map features that it produces to brain responses. This is a standard technique used in neuroscience experiments.

He doesn't say exactly how good the results are in this abstract, but does seem to suggest that they are far, far, far and away better than the LSTM model he tried in the past. And that LSTM model did pretty well! -- so, I'm guessing that correlation between the Transformer and very large parts of the brain is off-the-charts -- scary-good.

Interestingly, he mentions "compositionality". I've seen people write papers lately that attack large language models as lacking compositionality. Perhaps, they really do have it -- it's just that it's buried deep in the features somewhere in the network. This is something Tom Dietterich has talked about before: don't be deceived into thinking the network hasn't learned X, just based on its performance on some task. It could be that it has learned X... and Y and Z; but most of the time relies on W -- just like how a student that performs poorly on a test might know more than their score suggests.

At any rate, my impression based on this abstract is that large Transformer Language Models are doing something very brain-like in how they process language. Though, I'd have to see the paper / poster, in full, to know for sure.

This is a good sign for our march of progress towards ever more capable AI. 2029 is going to be glorious!
  • Yuli Ban and johnnd like this

#136
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPip
  • 1,290 posts

Oh, and by the way, if you are wondering, "I thought they were going to train a model just based on brain data.  Isn't that what you said?"  Indeed, I did say that; but I also said they would need 2 years to collect the data, and I said that just a few months ago!  So far, it looks like they have collected 13 hours from each of 3 people, which is enormous, but it needs to be at least about 9x to 10x bigger to have any hope of training a fair-sized model.  So, there's lots more recording work yet to do.

 

Be patient!  The future can't arrive any quicker than it arrives!


  • Casey and Yuli Ban like this

#137
Yuli Ban

Yuli Ban

    Born Again Singularitarian

  • Moderators
  • PipPipPipPipPipPipPipPipPipPipPip
  • 20,708 posts
  • LocationNew Orleans, LA

Be patient!  The future can't arrive any quicker than it arrives!

Oh trust me, I'm enjoying the ride.


  • starspawn0 and johnnd like this

And remember my friend, future events such as these will affect you in the future.


#138
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPip
  • 1,290 posts
Another huge advance in Natural Language Understanding (NLU). Google trains a "T5" model that has 11 billion parameters, trained on a dataset called "Colossal Clean Crawled Corpus", which is nearly a Terabyte of text.

https://arxiv.org/abs/1910.10683

The model achieves a big advance on the SuperGlue test, coming just below human-level performance (and by "human baseline" they probably mean a ph.d. student from a school like Stanford, which is going to be above average, certainly).

And on the original Winograd Schemas test it achieves around 94% accuracy! I remember the original human baselines for that test were reported to be about 90%; and, eventually they were jacked-up to 95%, and then to 100%. I'm guessing that they got smarter and smarter humans to take the test. The 90% score maybe was from Mechanical Turkers whose native language isn't English; and then they maybe asked some above average undergrads, and got 95%; and, finally, a very careful, very persnickety, very bright ph.d. student got 100%, so then that became the human baseline.

And just think: there are 10 more years to go before we reach 2029! Given how all this is scaling, we might hear about datasets with 20 terabytes or more of clean text, and neural nets with 100 billion or even 1 trillion parameters. Maybe they'll also include other media, like images, video and audio as part of the training. Your virtual assistant circa 2029 will definitely understand you far and away better than the best ones today.
  • Yuli Ban, waitingforthe2020s and johnnd like this

#139
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPip
  • 1,290 posts
An amazing-sounding advance from Microsoft on training large, open-domain dialog agents (e.g. socialbots):

DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation

https://arxiv.org/abs/1911.00536

Before we get to the good stuff, the usual disclaimers: despite showing impressive performance, these models are limited, with limited-depth inference ability. Also, people could probably quickly discover weaknesses -- and these would be shown on Twitter as the usual ha-ha-look-how-stupid-this-bot-is post.

Now that we've got that out of the way:

Basically, they trained the model on a very large conversational dataset scraped from Reddit, with 147 million exchanges! The dataset and architecture are different from the GPT-2 model (they build on GPT-2, and add reranking), because they say:

Unlike GPT-2, however, DIALOGPT is trained on large-scale dialogue pairs/sessions extracted from Reddit discussion chains. Our assumption is that this should enable DIALOGPT to capture the joint distribution of P(Target, Source) in conversational flow with finer granularity. In practice, this is what we observe: sentences generated by DIALOGPT are diverse and contain information specific to the source prompt, analogous what GPT-2 generates for continuous text. We have evaluated the pre-trained model on a public benchmark dataset (DSTC-7), and a new 6k multireference test dataset extracted from Reddit postings. DIALOGPT achieves state-of-the-art results in both automatic and human evaluation, lifting performance to near-human response quality.


They even find that the model is capable of doing question-answering, and giving commonsense responses:

Source: who is the first president of the United States?

Response: George Washington

Source: what is the boiling point of water?

Response: I think it's about 212 F.

Source: which one is bigger, sun or moon?

Response: The sun.

Source: which animal has black and white stripes?

Response: A black and white striped cat.



The last one could be counted as a "miss"; but many people would probably deem it acceptable.

And here is what a multi-round conversation with the bot looks like:

User: Does money buy happiness?

Bot: Depends how much money you spend on it.

User: What is the best way to buy happiness?

Bot: You just have to be a millionaire by your early 20s; then you can be happy.

User: This is so difficult!

Bot: You have no idea how hard it is to be a millionaire and happy. There is a reason the rich have a lot of money.


A few minor imperfections (e.g. last sentence). But notice, for example, the second-to-last sentence incorporates stuff from the turn-before-last -- shows it has some ability to remember the context.

In a footnote they write: "We provide a live invitation-only demonstration site for a
conversational agents with toxicity controls and mutual information maximization features discussed in this paper. Check
our GitHub repository for more information."


Finally, this is an amusing comment:

Overall judge preferences for relevance, informativeness and human-likeness, presented as raw numbers and a percentage of the total, are shown in Table 7. A strong preference can be observed for DialoGPT over PersonalityChat. The Table 7 also suggests that the ”vanilla” DialoGPT medium model may already be close to human response quality. Unexpectedly, we found that judges may prefer the MMI variant over human responses, probably because of many of the true human responses are erratic or idiosyncratic, or are tied to internet memes that happened to be unfamiliar to the judges.


....

Perhaps Microsoft will find some way to let the public try the bot. I'm afraid, though, that what will happen is some smart-aleck will coax the system into spewing racist and sexist material, will post it on social media, and then we'll see articles with headlines like, "Microsoft's racist robot shows just how little the company cares about users."

One way around this, perhaps, is to build very strong "toxicity controls", and then only release the bot to people who have signed certain promissory documents about not trying to coax the bot, and embarrassing Microsoft; and who don't have a bad or nonexistent social media reputation. That should reduce the possibility of Tay-like incident by a factor of a hundred or more.
  • Casey, Yuli Ban, funkervogt and 1 other like this

#140
Cloned

Cloned

    Member

  • Members
  • PipPip
  • 40 posts
  • LocationEU

Source: which animal has black and white stripes?
Response: A black and white striped cat.

 

A perfect example of an attempt to be human-like.
Fake as always. 
This house of cards will collapse as soon as it enters the real world.






0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users