Jump to content

Welcome to FutureTimeline.forum
Register now to gain access to all of our features. Once registered and logged in, you will be able to create topics, post replies to existing threads, give reputation to your fellow members, get your own private messenger, post status updates, manage your profile and so much more. If you already have an account, login here - otherwise create an account for free today!
Photo

True Artificial Intelligence Could Be Closer Than We Think, Via Brain-Computer Interfaces + Deep Learning

AI BCIs Brain-Computer Interface Artificial Intelligence

  • Please log in to reply
144 replies to this topic

#121
Yuli Ban

Yuli Ban

    Born Again Singularitarian

  • Moderators
  • PipPipPipPipPipPipPipPipPipPipPip
  • 22,221 posts
  • LocationNew Orleans, LA

Interesting.

 

You've mentioned before that you only realized this was the way in 2016. If by some arcane magic this same technology were unveiled in 2015 instead, how would you have reacted?


And remember my friend, future events such as these will affect you in the future.


#122
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPipPip
  • 2,020 posts

I had thoughts on it before then, back in 2014 when I looked into some papers on "neural encoding"; but it didn't really hit be as a viable path towards strong AI until I thought about next-gen BCIs and very large datasets.  Finally, when I sat down and thought about it carefully, did some simple calculations about the amount of data you would need, what sort of information it could provide (even just crude neural population data), and read dozens more neuroscience papers, it all came together in my mind that that is a powerful way forward.
 
As I said before, you'll see people like Kurzweil and others say that this is just like the ideas he had in his book The Singularity is Near or something; but he is very wrong.  The only other person I've seen who had thought a little bit ahead about this that I can think of is Greg Egan.  He is a true visionary.  He was one of the first to see how you can combine a machine learning approach with neural data to deeply imitate the brain (as opposed to using a mechanistic simulation).  However, he thought in terms more of micro-scale neural data, not macro-scale, population-level data.  Back when he wrote his scifi short story "Learning to Be Me" (that featured using ML + brain data) it wasn't well understood how much you can do with just neural population data; so he can be forgiven for not thinking through what you can do with a cruder scanning device.   I think in years ahead, he will be remembered as one of the originators of this idea.
 
As to how I would have reacted back in 2015, if I knew about next-gen BCIs, I probably would have just written the idea up quicker.  
 
Addendum:  I've said before that just improvements in Deep Learning applied to ever more compute and data, alone, might get us to much better AI.  The data (big text, images, youtube videos) contains weak correlations about human cognitive processes, and as such can be thought of as the output of a very, very, very noisy BCI (incidentally, this can be used as an argument against people who say you'll never learn cause-and-effect reasoning from large datasets -- they forget that the data implicitly contains breadcrumbs about the human brains that generated it).  Yet, big language models have been able to unearth the behavioral architecture enough to partially imitate human cognitive processing.  Still, results like this mean there is a ways to go yet before these noisy datasets will take us to AGI:

https://arxiv.org/abs/2005.00782

The signals about human cognitive processes in actual BCI data are, of course, a lot sharper; so, one should expect that if you have enough of it, you can do a lot better. And "enough" is probably going to be 10x to 100x smaller than the amount of data in "big text", as measured in hours of data (time for text = # words / normal human reading speed).



#123
Yuli Ban

Yuli Ban

    Born Again Singularitarian

  • Moderators
  • PipPipPipPipPipPipPipPipPipPipPip
  • 22,221 posts
  • LocationNew Orleans, LA

Well, I'll hold my breath until the first public reveal. Sounds very interesting, and we need a lot of good news. 


And remember my friend, future events such as these will affect you in the future.


#124
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPipPip
  • 2,020 posts

This is the public reveal, and it has some very good comments by Koch.  It's always good to have an outside expert like that vouch for the tech.

 

Perhaps you mean:  I want to see the consumer wearable version they are going to release in 2021?  You can see hints of it in that film I am Human.

 

....

 

Unrelated to your question, but worth pointing out:  just think about how difficult it must have been to test all those different attempts at a BCI we saw in the patent filings.  There was a whole zoo of different approaches; and, so far, we've only heard about two (I suspect they will try to develop more different kinds).

 

I mean, all the things you have to do set up these devices...  You can't just set each one up in a day -- those patent filings show you have to do a crazy amount of work.  I guess they could work in parallel and speed the attempts along somewhat; but, still, a lot of people-hours would have been needed.  

 

....

 

I hope once the software for the consumer wearable version is released, it is really easy to use.  A nice Python API library similar to Pybluez, so that hobbyists (and even academics that aren't software engineers, but know how to program) can easily use it would be great.  Some real thought will have to go into things like data formats, library routines, and maybe even tools for analyzing the data.  On this last point, a couple example programs written with the library and also with Keras or Pytorch would be really helpful.  They could even test out what some optimal machine learning architectures and hyperparameters are to analyze the neural data.  e.g. if you want to classify audio information you might use one type of architecture; and if you want to classify visual image neural data, you might use another.  



#125
Yuli Ban

Yuli Ban

    Born Again Singularitarian

  • Moderators
  • PipPipPipPipPipPipPipPipPipPipPip
  • 22,221 posts
  • LocationNew Orleans, LA

Yes, I meant a showcase of the Flow & Flux themselves (a bit more fully than what we saw in the documentary). I'm basically waiting for the TED talk, NeurIPS, or CES. Wherever they decide to do it first. No offense to your optimism, and I'm sure you understand where I'm coming from, but what really interests me involving this breakthrough is how well it might work with transformers and generative adversarial networks, particularly for the purpose of synthesizing media (of which chatbots are certainly a part).

 

That's where I see the greatest potential, at least at the moment. It's already well documented, the pseudo-generality that appears in transformers. If this can read and decode what one is listening to in near-real time, then that could easily be fed into something like Jukebox in order to make sense of song structure, texture, timbre, progression, and whatnot. That can further be broken down into raw data that could feed a general transformer like GPT-2. 


And remember my friend, future events such as these will affect you in the future.


#126
Erowind

Erowind

    Anarchist without an adjective

  • Members
  • PipPipPipPipPipPipPip
  • 1,520 posts

The only other person I've seen who had thought a little bit ahead about this that I can think of is Greg Egan.  

 

My boi! As someone who's only read his sci-fi novels the man has truly unique new ideas and perspectives  



#127
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPipPip
  • 2,020 posts

This seems relevant to the thread:  a new paper showing how MEG signals can be decoded to predict spoken and imagined speech:
 
https://www.frontier...2020.00290/full
 
Well... not quite full speech, but rather:
 

In this study, we performed a five-class classification task where each class corresponded to one phrase. Considering the tremendous cognitive variance across subjects (Dash et al., 2019c), only subject dependent decoding was performed, where training and testing data were from the same speakers (but unique). The classification task was performed on each of the four whole data segments (i.e., pre-stimuli, perception, preparation/imagination, and production/articulation). We leveraged two machine learning algorithms including a classic ANN as the baseline and the latest CNNs (i.e., AlexNet, ResNet, Inception-ResNet). The input to ANN was the root mean square (RMS) features of the denoised and decomposed MEG signals from each data segment. The input to CNNs was scalogram images generated from the denoised MEG signals of the whole data segments. Each of these methods is briefly described below.


So, you might find this disappointing. HOWEVER, this really is the punchline, in my opinion:
 

CNNs were found to be highly effective with an average decoding accuracy of up to 93% for the imagined and 96% for the spoken phrases.


93% for IMAGINED. Wow! Even for five-class classification that's really hard to believe. I've never heard any imagined speech decoding result like that before -- and I've looked at lots and lots and lots of EEG and FNIRS results.  The usual results one sees in the literature are like this:  classifying the difference between "uh" and "ah" (2-classes) with 60% accuracy, which is barely above random guessing (which is 50%).

Perhaps it has to do with the fact that they're not trying to classify individual speech sounds (morphemes); that you gain a lot more in accuracy when you work with phrases. Still, it sounds really, really impressive.



#128
Yuli Ban

Yuli Ban

    Born Again Singularitarian

  • Moderators
  • PipPipPipPipPipPipPipPipPipPipPip
  • 22,221 posts
  • LocationNew Orleans, LA

Recently had a fanciful little idea about Kernel's work and how they might be able to apply it in a flashier way that might get people to take notice.

This is a fairly fluffy idea; feel free to ignore it.

 

Utilizing brain feedback data to empower AI is a good idea, and the best path  to turn heads would be something that is interactive: a demo of social capabilities. You know, a chatbot. A chatbot that could be shown off at a show, at a TED talk, and maybe even at a convention. But while such a chatbot would be a great service, I'm perpetually stuck thinking "how do we get pop-tech pieces to really latch onto this?" 

The most frou-frou idea yet: why not use such a chatbot in a social robot?

 

A few times in the past, I've talked at length about social robots and why I hate them. The social robot fad was only a thing because start-ups wanted to exploit the 2010s deep learning-driven AI craze. Most All social robots were glorified toys; some of them more literally than others. Some of them were competing with smart speakers; others (like Cozmo) were actually toys that were marketed as robots. The gist is that AI was not yet good enough to create practical domestic robots that could do more than one task, and since the simplest tasks that could be done by home robots had already been done (i.e. robotic sweepers, programmable fans, and dishwashers), companies tried building off the success of smart speakers before they, too, would become more autonomous.  Didn't really work out that well since almost every social robot has been discontinued.

 

Indeed, the one that hasn't is also the only one I've ever been even remotely impressed with: Pepper. Somehow, despite its market drying up, it still survives because unlike all other social robots, Pepper has one thing going for it: it's actually humanoid. Not totally, but enough so that people latch onto it as an actual robot. It's not a disembodied cylinder or a toy truck; it's a fairly sizable rolling humanoid with a simple face and articulating fingers. 

 

 

As it happens, it falls close enough near the Uncanny Valley without actually plunging in, crossing a threshold that I, for the life of me, can't find the term for. I'm certain that people know what it is and have heard of it, but the name escapes me— it's that personification phenomenon where humans have a propensity to imbue greater intelligence into nonhuman things (I think it's separate from the ELIZA effect). As it looks closer to a human, people would respond to it more.

 

The issue is that Pepper is powered by IBM Watson, which for all intents and purposes, is outdated. Indeed, its greatest pitfall for the past several years is that it is a fairly unimpressive LSTM chatbot with clear limitations. This is also why social robots failed so hard: the one thing they have to do to prove they're not just toys or inferior Amazon Echoes, they can't do due to technological limitations. Even Alexa and Siri leave much to be desired, but you don't buy an Amazon Echo to have an artificial friend. You do buy a Pepper or a Cozmo or JIBO or KUBO for that social aspect, and if they are barely better than Cleverbot with a text-to-speech program added, why waste your money? Well at least with Pepper, there's that novelty of it being a humanoid robot, and its hands aren't entirely useless; it can grip things (though poorly) and use other gestures for experiments. 

 

This is actually why I wonder if it might be a decent idea to use Pepper as a demo for an advanced chatbot. It never took off because it wasn't useful enough, but it never died because it's more useful than the cavalcade of other social robots that came after; if you could solve the sociability issue with a vastly more robust system, then Pepper would actually get a second life while simultaneously helping to become a tangible face for the power of this new generation of AI. Certainly more tangible than it otherwise would be with a disembodied chatbot (even though disembodied chatbots are how most people are going to interact with said AI). Like I said, it'd make for a great, very visible demo. Pepper already got upgraded once before: its AI in 2014 was deeply shallow, and when it was upgraded to Watson in 2016, it actually looked borderline usable and became something of a minor advertisement for Watson's abilities and limitations. 

 

 

It's also entirely possible to just use another robot; I went with Pepper because that's the one that's specialized for social interaction in the first place.


And remember my friend, future events such as these will affect you in the future.


#129
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPipPip
  • 2,020 posts

There are other nice things to show off with the technology near-term.  Johnson's most recent Medium post:

 

https://medium.com/f...xt-ee7dc6c0f739

 

 gives one class of potential new "demos".  One could use it to understand more deeply mental illnesses.  The problem with this is that it is hard to make an effective demo around it.  And, there will be no shortage of people show up to pick it apart.  Any time somebody writes a paper on using FMRI data, say, to accurately diagnose mental illnesses, or say something about the mind, somebody else will show up to shoot it down.  This happened to Marcel Just, for example, with his work on suicide risk prediction.  It's better to try some of the other possibilities (the applications for mental illness and/or neurofeedback can be farmed-out to 3rd parties or open source software, out of the reach of the FDA and critics.  I actually find neurofeedback to be an incredible use; it's just that it's going to be hard to make it into a demo that isn't picked-apart).

 

Some more effective demos, besides a chatbot (with or without a robot attached), would be:

 

* Control of a robot arm by thinking alone -- would be very useful to people who can't move their arms.  It might not be so easy to pull this off with the particular kind of BCIs they are using.  People have done some good things along these lines, even with EEG; but it's a little weak.  

 

* Control a computer mouse and keyboard by thinking alone -- tricky to pull off, probably, unless one has a lot of training data.

 

* Imagined speech recognition for 1 of 10 or even 100 phrases.  For example, if you imagine speaking, "Please get me some water." it will print that out to the screen, if it is 1 of 100 phrases.  This would be a good compromise -- 10 or 100 phrases isn't technically out of reach; and it is a lot easier than full imagined speech recognition.  Furthermore, there is research supporting the idea.  And it's pretty accurate.

 

* Images seen --> graphics on screen.  This might be a task better-suited for the Flow BCI.  A lot of people have worked on this kind of demo; but they didn't have enough data to make it really nice.  They could, for example, partner with some researchers at Purdue University who work on this kind of thing.

 

* Imagined images --> graphics.  There's also some good work on this; again, people at Purdue, and also some scientists like Gallant and Yuki Kamitani who have worked on this.

 

* Dream --> images.  Yuki Kamitani, again:  https://science.scie...nt/340/6132/639

 

* Train a driverless car based on brain data.  This would be a nice conceptual demo.  One would probably need to add some caveats, "We

are not saying we've solved how to make a fully autonomous driverless car with this demo.  We just wanted to show what is possible with brain data.  And see the comparisons with a system trained on just visual data."  Now, the problem with such a demo, however, is that it is difficult to do.  If you train it in a virtual environment, you need to spend a lot of time setting that up, familiarizing oneself with the platform, type of ML models to use, and so on.  It's a huge undertaking.  Training a chatbot is a lot easier.

 

* Train an Atari videogame agent using brain data.  This is kind of like what Neuromod seems to be trying to do (or at least one of their projects).  I could see that working.

 

Of course, another nice demo would be a whole new type of BCI device, besides Flux and Flow.  There are lots of patent filings; maybe one of the other technologies listed could be developed further.



#130
Yuli Ban

Yuli Ban

    Born Again Singularitarian

  • Moderators
  • PipPipPipPipPipPipPipPipPipPipPip
  • 22,221 posts
  • LocationNew Orleans, LA

* Imagined speech recognition for 1 of 10 or even 100 phrases.  For example, if you imagine speaking, "Please get me some water." it will print that out to the screen, if it is 1 of 100 phrases.  This would be a good compromise -- 10 or 100 phrases isn't technically out of reach; and it is a lot easier than full imagined speech recognition.  Furthermore, there is research supporting the idea.  And it's pretty accurate.

Hmm...

 

It just occurred to me that imagined speech might translate across species as well, though obviously with a lot of new caveats. I wonder if it would then be possible to record a certain intent or desire in a nonhuman brain & then translate this into human language.


And remember my friend, future events such as these will affect you in the future.


#131
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPipPip
  • 2,020 posts

Sure... but it won't be anywhere near as rich as human "thinking".  It will mostly be about food and sex.

 

....

 

Unrelated, but worth mentioning, Lex Fridman has a new video about exponential progress in AI:

 

https://www.youtube....Wd44q0&t=26m40s

 

He thinks a high-bandwidth BCI like Neuralink is building will change the nature of AI completely.  I think Neuralink is too far into the future; and it won't be the compute power of the brain that leads to the big advances, it will be the data.  And you don't need data at the individual neuron level -- neural population data will suffice.  I think Kernel is much better poised to make the big breakthroughs for AI.  Neuralink is a little too far into the future; and by the time enough people have their brains drilled into for it to work, Kernel will be 5 generations further along with their wearable tech.  



#132
tomasth

tomasth

    Member

  • Members
  • PipPipPipPipPip
  • 303 posts

That make sense , not many people to drill into.

If someone make progress on nano machine to do as well as Neuralink , that will supersede them , and Kernel.

 

But a very successful Kernel can push for the nano bots option , the way the first successful deep learning pushed new ones.



#133
Yuli Ban

Yuli Ban

    Born Again Singularitarian

  • Moderators
  • PipPipPipPipPipPipPipPipPipPipPip
  • 22,221 posts
  • LocationNew Orleans, LA

Sure... but it won't be anywhere near as rich as human "thinking".  It will mostly be about food and sex.

I'm sure the richness of what we'd get out of it would be the least of people's interest in such a technology. 


And remember my friend, future events such as these will affect you in the future.


#134
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPipPip
  • 2,020 posts

I think I've mentioned it before, worth pointing out again:

 

https://crlab.cs.col...rain_guided_rl/

 

The authors use EEG signals to train a virtual robot agent to complete a task, by giving feedback when it makes an error.  

 

Using a better BCI, one can attempt something a lot more general and ambitious:

 

I'm guessing that it might be possible to use the Flow brain scanner (the TD-FNIRS one, which probably will respond better to video than a MEG scanner, giving more semantic information), and thousands and thousands of hours of brain scans as people watch videos of robots completing tasks -- or even just people playing videogames will suffice -- to train a model to predict those same brain states, given the video.  This is called an "encoding model"; and neuroscientists have built such things before, just not on this scale, and not specifically for robot or videogame videos.

 

Once you have such an encoding model, and can predict brain states accurately and reliably, and assuming you have trained a model to extract "error signals" from those brain states (which we know exist, as they show up in EEG as Error Related Potentials; and surely manifest somehow in the TD-FNIRS scans), you can use that to "criticize" arbitrary virtual agents, in arbitrary games.  In this way, they would learn to play more like a human.  

 

The same should work for robots, greatly speeding up the training process.  And it should work for arbitrary robots, on arbitrary tasks, without needing to retrain the encoding model for each new task!  

 

Again, nobody has attempted to build such a very large encoding model, using so much brain data for video, before.  I'd probably do some basic trials first, to see how well it would work, and how much data one would need -- using a small number of simple videos; and then gradually scale up by factors of 2 or 10, maybe, and see if it works for a broader and broader set of videos.



#135
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPipPip
  • 2,020 posts

I've been looking into what sort of additional information one can extract from MEG scans to do things like:
 
* Inform a language model; 
 
* Train robots.
 
It's known that both MEG and FMRI data improve word vectors in certain ways (e.g. to predict brain responses to other words); so there's that.  And there is also:

 

https://arxiv.org/abs/1911.03268

 

But here's something very, very interesting that was news to me:
 
http://ling.umd.edu/...kkanen_2007.pdf
 

 

Sentences such as the author began the book, which asserts that an activity was begun although no activity is mentioned in the syntax, were contrasted with control sentences such as the author wrote the book, which involved no implicit meaning. These conditions were further compared with a semantically anomalous condition (the author disgusted the book). MEG responses to the object noun showed that silent meaning and anomaly are associated with distinct effects, silent meaning, but not anomaly, eliciting increased amplitudes in the anterior midline field (AMF) at 350– 450 msec. The AMF was generated in ventromedial prefrontal areas, usually implicated for social cognition and theory of mind.  Our results raise the possibility that silent meaning interpretation may share mechanisms with these neighboring domains of cognition.

 
That's the kind of non-obvious semantic information that is going to take some extra effort for a machine without access to brain data to pick up on.  
 
Can you do that with EEG?  Well, there's the N400 signal, which happens around the same time as both "anomaly" and "silent meaning"; but since EEG doesn't localize it for you (maybe it does, if one tries; not sure), you can't tell which it is.  Even if you can use EEG to separate these possibilities, MEG is probably also going to enable you to better localize the source, and break it down into more fine-grained information.
 
.... 
 
And what about robots?
 
Well... it turns out that there are counterparts of EEG signals to be found in MEG.  For example, Error-Related Potentials have been used to correct robot mistakes:
 
https://groups.csail...EEG_Signals.pdf
 
And there are counterpart Error-Related Fields in MEG:
 
https://www.research...lent_of_the_ERN

 
 
EEG headsets with a large number of electrodes, and with low noise are difficult to work with, requiring gels and a complicated setup.  Not only would a wearable MEG scanner avoid all that setup complexity, but it should also enable much better source-localization, which I would guess would give more fine-grained information about the kind of error occurring.   For example, is the robot about to make an error like dropping a glass and shattering a cup?  Or is it more like tipping something over?  Or is it just a weird move?  Or is it move that, while it won't lead to a mess, doesn't achieve a goal?  
 
I'm guessing if you could break apart the Error-related-field / potential into a large number of pieces -- more specific brain areas, how much activity is happening in one area versus a nearby one, and so on -- you could much better inform the robot training module.  If you could improve the information flow by a factor of 10x or more, that would be incredible! -- you could speed up the training of the robots considerably, giving them more information about how to improve.  
 
So, the way this would look to an outside observer is:  in one room you've got a robot performing complicated tasks, and the only feedback it receives is whether it succeeds or fails on each trial.  In the next room, you've got a guy with an EEG cap on, just watching a robot perform a fairly complicated task.  And in the third room, you've got a guy with a ~800 channel MEG helmet on, watching his robot.
 
The robot in the first room takes 10,000 trials to learn the task.  The robot in the second room maybe takes 3,000 trials.  And the robot in the third room maybe takes 400 trials.  Maybe with tweaks, it can even be brought down to 200 trials.

 

So you see the potential here, right?  Now imagine you have a little army of just 10 people with these helmets, simply watching robots perform tasks.  They don't push buttons or anything -- they just watch.  That little arm of 10 will, together, speed up the training of a robot by a factor of 500x!

 

To achieve the same results with EEG perhaps you would need an army of 150 people, which is quite a lot.  



#136
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPipPip
  • 2,020 posts

I've been thinking about this potential "demo" some more:
 

* Control of a robot arm by thinking alone -- would be very useful to people who can't move their arms. It might not be so easy to pull this off with the particular kind of BCIs they are using. People have done some good things along these lines, even with EEG; but it's a little weak.


I'm not so sure that the Flux device wouldn't be able to do really, really well with it. I'm starting to think it might -- though, to my knowledge, nobody has done the relevant research for MEG and Optically-Pumped-Magnetometer MEG to verify it.

Here's my reasoning: it's known that the dynamics in the motor and pre-motor areas of the brain exhibit "low-dimensional dynamics / structure":

https://www.jneurosc...4/9390.abstract

Basically, what that means is that you only need very little information (just a few numbers) to predict the dynamics of most of the neurons in those brain areas. It's kind of like saying that, although there are a lot of neurons there, their collective behavior boils down to comparatively much smaller number of actions. This adds redundancy and robustness to what those areas do -- and that's probably what you want, if the organism lives a long time, and loses a lot of neurons over its lifespan.

This also likely means that neural population responses give you enough information to decode what those cortices are doing. It's not an automatic deduction, since the low-dimensional structure could be represented in such a way that the "voxels" smear together the manifold information so that you can't decode it -- but I think there's a good chance that doesn't happen.  Basically, what you want is that the information isn't represented exclusively by "high spatial frequency" components; that it can be decoded from "low spatial frequency" components.

Then there's the possibility that, because MEG gives you less reliable information the deeper into the brain you need to scan, you don't get enough to do the decoding. I also think that probably isn't going to make it impossible -- why wouldn't the necessary information be accessible near the surface?

Finally, there's the "geometry problem": MEG works best when neurons are organized so that they point in the same direction in each little area. I don't know that the motor cortex is so disorganized so that you can't get good population responses.

So... the upshot is that I see no reason why you couldn't control a robot arm with a fairly high degree of dexterity using MEG signals. Perhaps you will need a lot of channels -- like maybe 100 or more near that part of the brain -- but I don't see anything else that would limit the decoding. And the kind of scanner Kernel is using apparently can register both "planar" and "axial / radial" information with a large number of channels; so, you've got it all.

Well... I should say that the reaching and grasping part seems doable. The fine finger movements might be trickier. I'm not sure how much information one needs to extract from the motor cortex to decode that.

And, incidentally, there are Latent Factor methods and "cryptography based methods" that should help with training the machine learning:

https://www.biorxiv....0861v1.abstract
 
 

Brain decoders use neural recordings to infer a user’s activity or intent. To train a decoder, we generally need infer the variables of interest (covariates) using simultaneously measured neural activity. However, there are many cases where this approach is not possible. Here we overcome this problem by introducing a fundamentally new approach for decoding called distribution alignment decoding (DAD). We use the statistics of movement, much like cryptographers use the statistics of language, to find a mapping between neural activity and motor variables. DAD learns a linear decoder which aligns the distribution of its output with the typical distribution of motor outputs by minimizing their KL-divergence. We apply our approach to a two datasets collected from the motor cortex of non-human primates (NHPs): a reaching task and an isometric force production task. We study the performance of DAD and find regimes where DAD provides comparable and in some cases, better performance than a typical supervised decoder. As DAD does not rely on the ability to record motor-related outputs, it promises to broaden the set of potential applications of brain decoding.



I'm not sure if that would apply to imagined arm movements; but I would think it would -- surely the same kind of statistics is involved.



#137
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPipPip
  • 2,020 posts

In case you wonder how brain data could add to OpenAI's recent GPT-3 results:
 
https://www.futureti...sions/?p=281978
 
They have a whole section in the paper on "limitations".  I wrote about how you could use the current model + some additional bells-and-whistles (e.g. access to a calculator) to maybe automate a lot of tasks:
 
https://www.futureti...sions/?p=282015
 
I really don't want to criticize it, but I'm afraid I must in order to talk about brains:


 
1.  First of all, these language models are feed-forward.  They don't have recurrence.  It's really hard to believe, but they do all these marvelous things by following a step-by-step process, that finishes in less than 100 steps.  Many tasks you probably think take a lot of "reasoning" really don't -- they can be solved in under 100 steps!  (But something like mathematical theorem-proving won't succumb to that kind of limited inference, if we're talking about complicated proofs.)
 
2.  They say that their system still has some problems with commonsense physical reasoning; although, on certain tests of physical reasoning it does well.  I'm guessing it picked up a fair bit of physical commonsense associations, that can be used to reason about the world in broad strokes.  But if you ask it more specific questions, like what would happen if you placed melted cheese in a refrigerator, it might struggle, as they say.  That is a perfect example of something where brain data would improve it a lot!  However, I think OpenAI is going to try to take a different approach, and build a GPT-4 model that is multimodal -- i.e. it works with video, audio, images, as well as text.
 
3.  GPT-3 is also probably missing a lot of knowledge about human values, and also human mistakes; they mention in their paper another OpenAI paper about how to endow machines with human values and preferences.  There is going to be a lot of knowledge about human thinking hiding in the text as statistical correlations -- I've often said that text is like the output of a very noisy BCI; so, you could, in principle, model a brain, if you had enough of it.  However, I'm guessing it's going to take a lot of data to turn those correlations into confident knowledge.  For example, just think about how human sexuality... or death... or living in a hierarchy informs our actions.  These do so in very subtle ways, that is often hard to pick up from the language, even using sensitive statistical methods over trillion-word datasets.  BCI data, however, should teach machines human values a lot quicker, using much less data.
 
4.  GPT-3 is monstrously large.  It's not going to be baked-into your smartphone anytime soon, and certainly not the next iteration, which might be called GPT-4 and use 10x more parameters, again (putting it over 1 trillion).  If they can't distill it down by factor of 100x to 1000x, while preserving functionality, then we'll have to wait for better neural models -- e.g. those based on the brain -- before we can use these on a smartphone without access to a remote server.  Yes, I realize that the brain has a lot of parameters; but most of them are unused; and the average human doesn't have anywhere near the factoid textual knowledge of GPT-3.    Probably one could imitate most of human language understanding using a model with just 1 billion parameters (or even maybe 300 million parameters), which would fit on a smartphone; and use of functional brain data would be one path towards that.
 
5.  GPT-3 is not only missing "physical world knowledge", it's also missing "motor control, dexterity, taste, smell, hearing, seeing, and other things".  Yes, I'm sure OpenAI can fill in some of these gaps with their "secret project" detailed in Tech Review.  But not all of it.  I do think, as I've said, that correlations from text alone will give you some of this hidden data; but I am unsure if there is enough text in the world to where you can model it all.  BCI, however, will generate a rich vein of data with which to give machines human-like language grounding.
 
6.  GPT-3 also often makes a number of mistakes that humans don't make (or rarely make) -- such as, non-sequiturs, repeats, loss of coherence, and so on.  I would hope that a system that imitates an actual human brain would make these mistakes much less often.



#138
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPipPip
  • 2,020 posts
Neuromod has posted some Tweets about their staff:
 
https://mobile.twitt...188143024394240
 

Next off in our #cneuromod team introductions: Maximilien, a @MILAMontreal @UMontreal master's student researching the potential of genetic algorithms to come up with neural networks capable of accomplishing complex tasks like imitating human subjects on video games.


https://mobile.twitt...528262000439296

Our third #cneuromod introduction is @YuZhang2bic, another @IVADO_Qc postdoc developing new deep learning models to decode cognition, which can be used to recognize images, detecting actions in video games & even generating speech by analyzing fMRI data [https://github.com/zhangyu2ustc].


Sounds like they are working on some interesting stuff.

#139
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPipPip
  • 2,020 posts
https://mobile.twitt...243357201010689
 

Simulating a primary visual cortex at the front of CNNs improves their robustness to image perturbations. #AI still has a lot to learn from #neuroscience. Work co-lead with @joeldapello. Also @martin_schrimpf @JamesJDiCarlo @GeigerFranziska @neurobongo 1/N

This work was the result of an unexpected collaboration. Joel discovered that the ability of CNNs to explain V1 responses was correlated with their adversarial robustness. Particularly, adversarially trained models [@aleks_madry] had the most V1-like representations.

For those who don't know, CNNs are easily fooled by imperceptible perturbations explicitly crafted to induce mistakes (adversarial attacks). Currently, the best defense is to explicitly train models to be robust to these attacks which has a very high computational cost.

Simultaneously, I was building benchmarks to evaluate how well CNNs explain V1 single-neuron properties and found that a Gabor filter model constrained by empirical data [@DarioRingach] still outperformed all CNNs tested. Could we improve current CNNs by engineering a better V1?

We went on to develop VOneNets, hybrid CNNs with a fixed-weight V1 model front-end. Take any standard CNN architecture, remove its first block, and replace it by our empirically-constrained Gabor filter bank model of V1.

VOneNets based on 3 CNN architectures (ResNet50, CORnet-S, AlexNet) maintained similar clean accuracy, and were considerably more robust than the standard models! This increased robustness persisted for perturbation strengths that left the standard models near chance level.

Without any expensive adversarial training - just placing a V1 at the front! Surprisingly, VOneNets were not only better than standard models, but they outperformed SOTA on a conglomerate benchmark of perturbations with adversarial attacks and common image corruptions.

Which components of VOneNets are responsible for this gain in robustness? Interestingly, removing any part of the V1 model resulted in less robust models, suggesting that they all interact synergistically!

Removing V1 stochasticity had the largest single effect on perturbation accuracy. However, adding only V1 stochasticity to ResNet50 resulted in only 1/3 of the improvement in robustness. This suggests that V1 stochasticity and features interact nonlinearly to improve robustness!

Also, the vast majority of the improvement in robustness is not due to the stochasticity during the attack or inference. V1 stochasticity during training makes the downstream layers learn more robust representations.

We have very excited about these results as they clearly show that there is plenty of opportunities for a neuro-inspired AI. We feel that we are only tapping the tip of the iceberg and a lot of future work needs to be done!

This work was partly inspired by the fantastic study showing that regularizing CNNs to develop mouse V1-like representations leads to gains in robustness in gray-CIFAR. Zhe Li, @wielandbr @bethgelab @sinzlab @xaqlab @AToliasLab and others.


I liked that claim where they mentioned that adersarially-trained models (not their model, necessarily) -- models trained to be resistant to adversarial attacks -- turn out to be more "brain-like". That's very interesting!

Note that this paper isn't about imitating brains, based on lots of brain recordings. They actually build a model based on neuroscientific understanding of V1.

At the moment, people have shown that training models using mouse brain data, and also using a little human or primate data in other contexts, improves performance. But nobody has yet tried to train models using a large amount of human brain data -- at least not publicly. My expectation is that it would make neural nets highly robust to the kinds of perturbations seen thus far in "adversarial examples" -- much more robust than any model yet devised.

The same should work for natural language processing tasks, while at the same time giving the neural net more human-like representations and better generalization performance. But, multimodal data might also get there. There is evidence for this from a few months ago by some researchers at DeepMind, who showed that agents trained in rich 3D worlds had much better "systematic generalization" on NLP tasks.

A big advantage that brain data would have is that, once BCIs are available, it will be relatively cheap to produce a lot of it to train models. Hobbyists could build models using data from their own brains, for example; and I would expect that the models they build to outperform ones Google has built circa a small number of years ago (and maybe even better than the best ones they have today!).

#140
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPipPip
  • 2,020 posts

Tweet about a new startup in stealth mode, that is "uploading the brain to AI":
 
https://twitter.com/...235108555653120
 

Want to upload the brain into AI and make a positive impact in the world? Here is a cool new startup. The
http://ninai.org spin-off "Upload AI" is now hiring machine learning scientists (send inquiries to info@uploadai.com). Work location is flexible.


In the past, this group has done work very much along the lines of what I have described in this whole thread; in fact, here is an older post I wrote on some of their work:
 
https://www.futureti...rning/?p=270809

Note that this research doesn't model neurons at a microscopic level, like Markram's Blue Brain Project. The approach is more minimalistic and nihilistic, in the family of methods I have discussed in this thread. The method recognizes the awesome power of scale-meets-Deep-Learning, and uses the brain as a data source, where you don't even bother with microscopic details like dendritic spines and such. Just recording the activity of a few thousand neurons, or perhaps a few thousand neural populations, suffices.

There's no guarantees that this is what they will be pursuing at the new startup; but I think there is a good chance of it. There's so much you can do!

 

A recent interview with Andreas Tolias, one of the core scientists, basically confirms it:

 

https://twimlai.com/...andreas-tolias/

 

He suggests that the top two levels of Marrs's hierarchy are the most promising -- these are the behavioral and representational levels, which correspond to behavioral and BCI data.  The implementation level is further down, at level 3, and Tolias suggests it's too messy and complicated to bother with as far as building AI.

So, where will they get the data?... Well... maybe they will use BCIs or FMRI or MEG or some other bio-data extraction method. Why not? Kernel could help them out, for example...

....

Eventually, when advanced BCIs become a commercial product, I foresee a large number of hobbyists building AI using their own brain data. You wouldn't need that much compute to compile a few hundred hours of your brain data into a language understanding model, for example. A high school kid could do it in their garage, if they had the BCI to extract the data. For smaller tasks, you could probably get away with under 50 hours of brain data.







Also tagged with one or more of these keywords: AI, BCIs, Brain-Computer Interface, Artificial Intelligence

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users