Jump to content

Welcome to FutureTimeline.forum
Register now to gain access to all of our features. Once registered and logged in, you will be able to create topics, post replies to existing threads, give reputation to your fellow members, get your own private messenger, post status updates, manage your profile and so much more. If you already have an account, login here - otherwise create an account for free today!
Photo

True Artificial Intelligence Could Be Closer Than We Think, Via Brain-Computer Interfaces + Deep Learning

AI BCIs Brain-Computer Interface Artificial Intelligence

  • Please log in to reply
115 replies to this topic

#101
funkervogt

funkervogt

    Member

  • Members
  • PipPipPipPipPipPip
  • 885 posts

I meant that Kurzweil said many years ago that, if we want to build a simulation of a human brain, we shouldn't try to make it a 1:1 copy of a real brain, down to the level of individual neurons. That philosophy mirrors what you quoted: 

 

 

 

Main argument (?) is that we should be viewing neural networks from an optimization perspective, and not worry so much about what individual neurons are doing


#102
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPip
  • 1,327 posts
David Markowitz of IARPA tweets:

https://twitter.com/...484340674408448

Exciting discovery by @IARPAnews #MICrONS team led by @AToliasLab! Suggests #DeepLearning can model complex systems (such as brains) well enough to suggest perturbations that achieve a desired system state. May have profound broader implications for modeling biology.


Paper:

https://www.nature.c...1593-019-0517-x

I posted a Biorxiv version earlier. What's new is that the paper finally was published in Nature. Probably it has some more observations than the Biorxiv version from 10 months ago -- I know, for example, that the group has submitted a paper on improving image recognition using brain data. The improvement is apparently rather dramatic; but it has not yet been made public.

What this work shows is that Deep Learning can model the dynamics of the brain so well, that it can even generalize to patterns far out on the "long tail" in the distribution of images and responses, and still produce accurate results.

....

In related news, Singularity Hub ran an article last month on work from the Tolias Lab:

https://singularityh...from-the-brain/

If we can constrain an artificial network to match the neural responses of biological ones, the authors said, we can “bias the network” to extract findings from the data in a way that “facilitates brain-like generalization.” Scientists are still trying to figure out which brain processing properties to transfer over to AI to achieve the best results.

By training algorithms on multiple tasks and using neural recordings to guide their learning, AI will become more brain-like. This doesn’t mean that a network will perform better on a single task compared to classic deep learning regimes; rather, the algorithm will be able to generalize better beyond the statistical patterns of the training examples, the authors said.


I think what they are doing is exciting. I keep checking the web to see what their latest Deep Learning + brain models reveal. That group, and two or three other groups, are the main ones I investigate for the latest signs of progress.

One that I am eagerly awaiting information from is the Neuromod Project. I expect we'll soon hear about their work on applying brain data to train machines to play video games better than previous methods. (The language understanding stuff is further away, as it takes more hours of recording.)
  • Alislaws likes this

#103
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPip
  • 1,327 posts
PDF version of the upcoming NeurIPS 2019 paper

Learning from brains how to regularize machines

https://openreview.n...f?id=S1gRRESxUH

They trained a model to predict mouse brain responses to images. Then, they used the model to predict similarity scores among pairs of images in a large training set. Finally, they used this "similarity matrix" to regularize another neural net model trained to assign labels to images. They penalized this model during training if it deviated too much from the similarity scores given by the brain-based model.

This idea of using a similarity matrix is great, because it allows one to compare different kinds of networks, with totally different architectures.

What did they discover when they "regularized" the network that assigns labels using the matrix?

* It is much more robust to a large number of classes of "adversarial examples" than baseline models. Some other models are better against specific adversarial attacks; but they don't match the brain-regularized model across all kinds of attack.

* The model generalizes better, requiring fewer training examples and of lower complexity. This appears to be preliminary work, and they hint more will be published about it soon.

They suggest that if they were to add even deeper layers from the visual cortex, the robustness to adversarial examples might increase further; perhaps it would reach human levels of performance, putting an end to adversarial attacks (humans are also susceptible, but we forgive those kinds of attacks).

What makes this all the more remarkable is that they didn't set out to make neural nets robust to these attacks, and to generalize better. It just fell out of trying to make models better fit the brain (mouse brain).

The same tricks will probably also apply to language modeling. Once there is enough data to build brain prediction models of text (datasets with only a few million words), these can be used to build similarity matrices to regularize much larger models. Perhaps those larger models will then act in even more human-like ways than they have to date. Another option is to somehow use brain data during a "fine-tuning" step, to train a model based on 99% unstructured text with no brain responses, and 1% with brain responses.

I can foresee much better chatbots on the horizon, as I've said before. Ones resistant to adversarial attacks, and that don't write racist, sexist, lewd, rude, mean, or immoral bilge.

The regularization could also be applied to robots to make them learn quicker and generalize better.

Mouse data, alone, might be good enough to do this for robots. Only time will tell!

#104
caltrek

caltrek

    Member

  • Members
  • PipPipPipPipPipPipPipPipPipPipPip
  • 10,223 posts

As an end user, I can't pretend that I understand everything that Starspawn0 has written in post #103.  Still, there were a few things to which I found myself reacting.  So I will post my reactions, with the caveat that I don't necessarily understand all the points being made.

 

 

 

 

What makes this all the more remarkable is that they didn't set out to make neural nets robust to these attacks, and to generalize better. It just fell out of trying to make models better fit the brain (mouse brain).

 

 

See, this is what makes me a little hesitant to endorse this whole movement to develop AI.  To understand the human brain, they end up experimenting on all of these poor mice.

 

*Waits for laughter*

 

Now, I say that for two reasons:

 

  1. The old "animal rights" arguments.
  2. The way this all results in what I guess I would call the deconstruction of the human brain.

In the old days (like even before I was born) there were a lot of philosopher types who argued that animals were "just" machines.  Humans were granted a special status because, well, because everybody "knew" that humans were just plain smarter.

 

Now, students of the brain can actually construct quantifiable statements proving that humans are indeed smarter than at least some animals. The problem comes when folks start indicating that the human brain is "just a machine."  

 

Now, why is that a problem?

 

Is it a problem because the brain is indeed not "just a machine"?

 

Well, maybe, but that is not really the point.

 

So, if the human brain is a machine, what is wrong with treating it as such?

 

Ok, now we are getting closer to the problem.  This is where having a background in the humanities comes in. That is to say, being familiar with what it means to say the "objectification of the human being."

 

Put another way, we start off as human subjects, and then the social system we are in turns us into objects.  Disposable objects at that. 

 

 

 

.Perhaps those larger models will then act in even more human-like ways than they have to date. 

 

 

Ok, this reminds me of the discussion we once had about Chomsky.  Chomksy advised that we should start looking for a "language" center of the human brain.  

This implied a sub-organ of the brain that did language processing. Well, as it turns out (and if I am understanding all of this correctly) there is no such sub-organ.  Rather, the entire brain acts like a language processor.  At least that part of the brain that lies on top of our "lizard brains."  This is very confusing. It also helps explain how the "subconscious" works. 

 

In our brains, there is a constant stream of chatter.  This chatter never seems to stop, even when we are asleep.

 

However, during our sleep, there probably are certain parts of the brain that rest. Or perhaps certain patterns of thinking are set to one side. Our subconscious minds then "take over."  Now these subconscious parts are still "talking" with the brain.  That is, we (or at least I) still have this chatter going on.  However, what the sub-conscious seems to excel in is what I would call "symbolic logic." 

 

Take, for example, a tree.  Trees are often used to symbolize organization.  They are also made of wood.  Jesus was a man who was the son of a carpenter.  He was sacrificed on a wooden cross.  Buddha rested beneath a certain kind of tree.  Other examples come to mind.    

 

Well, that should give you all something to chew on for a while.


  • starspawn0 likes this

The principles of justice define an appropriate path between dogmatism and intolerance on the one side, and a reductionism which regards religion and morality as mere preferences on the other.   - John Rawls


#105
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPip
  • 1,327 posts
I'm not quite sure what your point is -- or rather, I see several points.

Let me address this one:
 

That is to say, being familiar with what it means to say the "objectification of the human being."


This is I think, at bottom, what bothers a lot of people about the field of AI, in particular Jaron Lanier. He likes to talk about the tech bro Singularity cult, and compares it to a religion. He even has gone so far as to compare Alan Turing's killing himself by eating a poisoned apple to the biblical story of the Garden of Eden and forbidden fruit; and has accused Apple's logo as deriving from that same brand of religion. When pressed about what he (Lanier) really thinks, he came clean and said the thing that really bothers him about AI is that if it succeeds, it will diminish the sanctity and specialness of the human spirit -- reducing us to mere machines. Because if we can't tell machine from human, doesn't that mean we are machines?

Personally, I'm agnostic. Consciousness seems to be a very tricky thing, indeed. Whether or not machines are conscious, however, is not that much of a concern to me. I'm more interested in seeing them reduce the need for human labor -- even eliminate it altogether. That will pose another kind of problem, however: if people define themselves by their actions -- their labor -- and then that no longer becomes necessary, then what remains of the person? People are going to need to figure that out real fast; there isn't much time left (the skeptics are wrong). Personally, I have long ago stopped defining myself by my deeds, so don't have this existential hangup. People who have bought-in too deeply to "meritocratic" thinking, however, are going to suffer a great deal of psychic trauma.
  • Yuli Ban and SkyHize like this

#106
caltrek

caltrek

    Member

  • Members
  • PipPipPipPipPipPipPipPipPipPipPip
  • 10,223 posts

 

 

I'm not quite sure what your point is -- or rather, I see several points.

 

 

This reminds me of the story about a Zen Master and a student in his audience. The student raised his hand and indicated to the Zen master that he did not understand what he was saying.

 

The Zen master then asked the student to come up from the audience to the stage with him.  The student dutifully complied.  When the student reached the stage, the Zen master then asked the student to sit next to him.

 

The student then sat down where he was instructed, whereupon the Zen master then proclaimed: "See, you understand me perfectly."     


The principles of justice define an appropriate path between dogmatism and intolerance on the one side, and a reductionism which regards religion and morality as mere preferences on the other.   - John Rawls


#107
tomasth

tomasth

    Member

  • Members
  • PipPipPipPipPip
  • 255 posts

caltrek

deconstruction of the human brain is for reconstruction a better version.

 

Alphago didn't destroy Go it open more ways to see it.

 

 

 

starspawn0

Everyone is agnostic , until they know better. Before Consciousness is made we can't know.

 

There are human labors that are for human by definition (examples , if for a role in a religion one have to be a human , possibly for some people prostitution , possibly therapist artist and any role that the identity of the laborer as a human is part of the requirement for it)

"need for human labor" maybe labors that being voluntary is part of its description can be developed.

 

There are many way people find what to do that are not labors/jobs , and of coarse many that do not define themselves by their deeds.

 

Definition of development improvement and superiorizing other then "meritocratic" exist and are going to be developed.



#108
caltrek

caltrek

    Member

  • Members
  • PipPipPipPipPipPipPipPipPipPipPip
  • 10,223 posts

 

 

deconstruction of the human brain is for reconstruction a better version.

 

That may be the intent. You do realize that technology often involves unintended consequences, don't you?  


The principles of justice define an appropriate path between dogmatism and intolerance on the one side, and a reductionism which regards religion and morality as mere preferences on the other.   - John Rawls


#109
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPip
  • 1,327 posts
A paper to be presented at NeurIPS next month fine-tunes BERT on small amounts of brain data, and shows improvement:
 
https://arxiv.org/abs/1911.03268
 

Progress in natural language processing (NLP) models that estimate representations of word sequences has recently been leveraged to improve the understanding of language processing in the brain. However, these models have not been specifically designed to capture the way the brain represents language meaning. We hypothesize that fine-tuning these models to predict recordings of brain activity of people reading text will lead to representations that encode more brain-activity-relevant language information.


My usual remark:. They use only minutes of data from participants. You need hundreds of hours to see really big gains.

But results are tantalizing.

Improvements to robotics and acquisition of rare skills, like ability to perform scientific research, might make a more effective demonstration of the power of brain data combined with modern machine learning.
  • Yuli Ban likes this

#110
caltrek

caltrek

    Member

  • Members
  • PipPipPipPipPipPipPipPipPipPipPip
  • 10,223 posts

 

 

 

deconstruction of the human brain is for reconstruction a better version.

 

That may be the intent. You do realize that technology often involves unintended consequences, don't you?  

 

 

Thank you for your initial thought-provoking comment.  I found myself lying awake at night when another response also occurred to me.

 

Define "better".  

 

For example, do you mean better at solving certain kinds of problems?

 

Better in a quantifiable sense?

 

Or, better in a moral sense?

 

If you mean better in a moral sense, then I think I have further objections.

 

Edit:  The questions are open ended in that others are welcome to add their thoughts.


The principles of justice define an appropriate path between dogmatism and intolerance on the one side, and a reductionism which regards religion and morality as mere preferences on the other.   - John Rawls


#111
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPip
  • 1,327 posts

The logic of the market is pushing us towards human modification to make us more "moral", smarter, more creative.  Moral, as decided upon by the prevailing ethos.  The ancient Spartans probably thought striking down the weak is a moral thing to do, just as the China government thinks striking down the Uyghurs is a moral act.  

 

The market is pushing the world towards ever greater levels or meritocracy.  Lots of people think, without question, that meritocracy is "good".  Perhaps that's because they have an inflated sense of their own importance in the grand scheme of things.  If they have seen just how smart and creative the smartest humans are, as I have, they would quickly change their mind.  Most people are dumb as shit, and don't know it, at least compared to the smartest among us.

 

And they will suffer, as market forces constrain their lives more and more, and lock them in to a semi-comfortable, but limited life trajectory.  Meanwhile, the better-endowed will pull away, and enjoy greater freedom, pleasures, and power.

 

There is a fix:. cognitive enhancement.  Brain implants.

 

But there is also another:. abandonment of the market straightjacket humanity seems to be strung up in.

 

There isn't much time left until the second option becomes untenable.  Sadly, people are still deluded into thinking they are too important to become a victim.


  • Yuli Ban likes this

#112
tomasth

tomasth

    Member

  • Members
  • PipPipPipPipPip
  • 255 posts

The Chinese striking the Uyghurs because they think its not a harmonious culture with the rest of china , and they are afraid of separatism. Its with any minority that disrupt that.

 

Aren't the rich's wealth protect them from become a victim ? The poor won't be able to afford enhancement.



#113
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPip
  • 1,327 posts
I was just thinking the other day that maybe I was too skeptical of the "tweak" approach to building much better AI. It's something I had considered before -- e.g. a very primitive version is already present in work by Fyshe and Mitchell on retrofitting word vectors by adding a spoonful of brain data; and I used to write about where this could lead -- one example is here:

https://www.reddit.c..._braincomputer/

So, what am I talking about here about "tweaking"? Consider this paper:

https://www.futureti...rning/?p=270856

and this poster (abstract only):

https://www.futureti...-like/?p=270236

The second poster here suggests that just 13 hours of data is enough to produce an accurate alignment model with the brain -- basically, mapping deep Transformer states to voxels (usually a linear map trained with ridge regression). And the first paper shows that even just 5,000 to 6,000 words of data already leads to small improvements to BERT, after a fine-tuning step.

The fact that the alignment of Transformer units to voxels is so good (based on that qualitative statement in the abstract), suggests that maybe you don't have to do a lot of tweaking to get it to be "brain-like" -- it's already fairly brain-like to begin with.

Furthermore, these Transformers have learned vastly more factual knowledge than the average human; it's just that the knowledge from an individual human that we care about might not be that well-represented by the model. The model is "broad", but not "deep enough".

Putting all this together, maybe collecting just a little more data than Huth's group has done thus far -- maybe 100 hours -- one can fine-tune the Transformer to make it a lot closer to the brain; and, at the same time, as suggested by that BERT paper, it will learn much more human-like representations. e.g. Its representations for movement and imperative language will be altered to become more human-like. Other changes will probably also occur. For example, its physical-world knowledge will probably improve significantly (text is a poor source of implicit world knowledge of that sort); its visual understanding will also improve significantly; audio, gustatory, tactile, olfactory, social, moral -- improvements along lots of dimensions at once.

At the same time, there probably won't be losses in performance on any important tasks. This is one of the conclusions of the BERT paper -- there are only gains.

What you might get out in the end is, for example, a fine-tuned version of GPT-2 or BERT -- let's call it GPT-B, for brain -- that produces even more human-like outputs, that make a lot fewer foolish semantic and logical errors. And, perhaps it will be easy to tweak it to where it doesn't write "toxic" stuff -- in fact, perhaps tweaking with brain data, alone, will cause the toxic stuff to appear a lot less often.

Now, if you have such a large language model, you can then add a re-ranker, as was done in this paper, and produce a chatbot:

https://www.futureti...-like/?p=270632

That chatbot is, by all accounts, already pretty good. But, it does make a few embarrassing errors (and a skeptic can probably make it say some really dumb stuff); its depth of reasoning is quite limited; and it occasionally says some toxic stuff. A brain-tweaked Transformer may improve all three at once. The depth of reasoning will still be limited; but you'll get a lot more mileage out of its limited depth -- it will appear a lot "smarter".

....

When you think about it, humans have pretty limited depth of reasoning during regular conversation. Where do words come from? They just pop into our heads without our even thinking much about it. That's reminiscent of the shallow kind of reasoning you would expect in a Transformer. The deeper kind of reasoning where you sit down with pen and paper and generate ideas and test them out, or where you think about a chess game, is not the kind normally used in everyday conversation. That kind requires recurrence.
  • Yuli Ban likes this

#114
caltrek

caltrek

    Member

  • Members
  • PipPipPipPipPipPipPipPipPipPipPip
  • 10,223 posts

Seems like this article might (or might not) fit nicely in this thread.

 

To Help Fidgety Kids, Researchers Made a Brain Scanner That Fits in a Bike Helmet

 

http://blogs.discove...-a-bike-helmet/

 

Introduction:

 

(Discover) A simple bike helmet may be the answer for researchers looking to study the brains of fidgety kids. With a few extra holes drilled in the top and a pile of chunky cables ballooning outward, the common piece of headgear could offer an alternative to stationary brain scans.

 

With a few tweaks, researchers from the U.K. equipped a commercial bike helmet with a magnetoencephalography (MEG) scanner, which uses magnetic fields surrounding the brain to detect activity. The highly sensitive sensors can pick up more nuanced data than can EEGs, which is why doctors commonly use MEG scans to find the origin of a seizure in a patient’s brain.

 

In a report published today in Nature Communications, the U.K. team argues that the wired-up helmet could be a useful device for future clinical studies on brain activity, especially in children. The helmet could allow researchers to observe subjects in more natural environments, allowing for movement without the risk of hindering results.

215896_web-768x1024.jpg

A young child wearing the MEG scanner, created using a modified bike helmet and several sensors.

Credit: Rebeccah Slater, University of Oxford


The principles of justice define an appropriate path between dogmatism and intolerance on the one side, and a reductionism which regards religion and morality as mere preferences on the other.   - John Rawls


#115
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPip
  • 1,327 posts
Patrick Mineault wrote a blog post about the MAIN 2019 neuroscience conference:

https://xcorr.net/20...ience-main2019/

He mentioned Leila Wehbe's work (the same that I mentioned above in the November 14 posting), and also Pierre Bellec's work with the Neuromod project:
 

It’s a lot easier to name instances where neuroscience has inspired AI almost by accident than to make a coherent research program based off of that idea. But some people are trying.

One idea is to use brain data (neurophysiological recordings or fMRI) to constrain deep neural nets to a good subspace with high performance. Leila Wehbe from CMU presented some ideas along these lines, fitting a language model pooling both fMRI and text doccuments. She mentioned that right now, joint training predicts the fMRI data well but the results are equivocal on language tasks.

It might just be a question of the vastly larger amount of text data than brain data. Pierre Bellec aims to collect hundreds of hours of brain data (MEG and fMRI) from a dozen subjects as part of the Courtois Neuromod project. These large scale projects could generate the vast amounts of data necessary to train ANNs with brain data exclusively. Very recently there was a proof of concept that showed more robust classification using brain data from mouse visual cortex as a prior on image classification. These are very early days but I think this approach will work eventually. The only question is whether it will ever be economical to use this method (i.e. 500$ an hour to run an MRI scanner versus pennies per classification on mturk).


Mineault is someone I mentioned in a previous post in a list of people associated with Facebook's BCI project. He keeps a low-profile these days, and doesn't mention his Facebook affiliations -- so, I'm unsure if he still works there, on the project.

I have said before that I don't think their BCI will have a high enough information-extraction rate to build AI very efficiently. This was a guess on my part, based on some comments in a paper by some people at JHU working on related technology, about challenges in getting the number of "optode pairs" high enough. That isn't the only game in town, as far as optical BCIs go. Openwater and Kernel are also working on it. I recently posted some patent applications by Kernel.

....

Just thinking about our little forum here, it is kind of like the tabloids in the Men in Black films. Remember how the behind-the-scenes, not-yet-mentioned-in-the-press, part-of-the-broader-consciousness, super-important stuff was being passed along in the tabloids, that people just ignored, because they thought they were just tabloids?
  • Casey and Yuli Ban like this

#116
starspawn0

starspawn0

    Member

  • Members
  • PipPipPipPipPipPipPip
  • 1,327 posts

I decided to drop in to mention something a little fun for you to try.  I wrote a program (cells) in Python, in Colab for you to try!  For those not familiar with Colab, see:
 
https://www.futureti...e-21st-century/
 
It's free, and easy to set up -- try it!
 
Basically, I wrote a little program to do the following: it took an audio file from the Openneuro (open-source neuro data) Project; then passed it through Mozilla's Deepspeech, to get accurate transcription, along with timestamps of when words appeared; then, I downloaded Glove word vectors, and created a time-sequence of vectors, where the vector at time t is 0 if nothing was uttered at that time, and otherwise is the Glove vector for the word being uttered at that time (longer words will be uttered across multiple time steps, so the vector appears multiple times); then, I created batches of 5 second word vector "clips" to use as input to train a brain-state prediction model; then I downloaded the FMRI NII file for "subject 1" that listened to the audio recording; I did some elementary "slice time correction" on it (geometric correction isn't strictly necessary for what we're doing; but the slice time correction is important); then, I built the associated "output" sequence to match the 5 second word clips, taking into account a 1.5 second BOLD latency; then, I trained an LSTM model on the batches of clips and brain responses at the end (1.5 seconds after the end of the clip); finally, I plotted it, and the images were definitely brain-like; but it could have just memorized -- proper testing will be needed. Also, I expended no effort on choosing optimal hyperparmeters (I did play around with different epoch counts, though).

I didn't do any (test, train, validation) splitting, and do proper testing. This dataset is too small to bother with it, anyways. My purpose in doing this was simply to see whether I could familiarize myself with the basic modules, protocols, and other ingredients needed to build the "critic" module described here:

https://www.reddit.c...ual_assistants/

And it seems to be pretty straightforward how to do it.  Furthermore, the text-generation module in that article is easy to produce, given that 99% of the work has already been done in HuggingFace's libraries (or, I could just snap in place Microsoft's trained DialoGPT model).  So, brain-based chatbots are certainly possible near-term -- the only thing missing is the training data.

My next step in this project is to build an (variational) autoencoder for brain images -- or, perhaps simply using PCA / SVD (a simpler kind of "dimensionality reduction" method) will do fine. The point of this is to produce a compressed representation of the brain images, from which one can recover the image; and then when training the neural critic, I'll only need to predict compressed representations. This will make it more efficient to train the critic (it will handle things like geometric distortions automatically), while also allowing me to do some amount of transfer learning -- I can train the autoencoder in an unsupervised fashion on large amounts of brain data in the Openneuro repository, even from projects that didn't have audio files.
 

import numpy as np
import requests 
import nibabel as nib
import json
import keras
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import LSTM
import matplotlib.pyplot as plt

# Get the MilkyWay audio file and the fmri nii.gz file (for later) for subject 1.

!wget https://openneuro.org/crn/datasets/ds001131/snapshots/1.0.0/files/stimuli:milky_audio.wav -O milky.wav
!wget https://openneuro.org/crn/datasets/ds001131/snapshots/1.0.0/files/sub-01:func:sub-01_task-milky_bold.nii.gz -O milky_fmri.nii.gz
# We begin by loading the Deepspeech command line program, and use it to give 
# accurate timestamps to the appearance of words in milky_audio.wav.  

!pip install deepspeech
!wget https://github.com/mozilla/DeepSpeech/releases/download/v0.6.1/deepspeech-0.6.1-models.tar.gz -O models.tar.gz
!tar xvf models.tar.gz
!apt-get install sox
!wget https://openneuro.org/crn/datasets/ds001131/snapshots/1.0.0/files/stimuli:milky_audio.wav -O milky.wav
!deepspeech --json --model deepspeech-0.6.1-models/output_graph.pbmm --lm deepspeech-0.6.1-models/lm.binary --trie deepspeech-0.6.1-models/trie --audio milky.wav > milky.json

with open('milky.json','r') as r:
  data = json.load(r)

duration = data['words'][len(data['words'])-1]['start_time '] + data['words'][len(data['words'])-1]['duration']
# Next, we load 100-dimensional Glove word vectors, and build a dict wvectors 
# with key value equal to the lowercase words.

!wget http://nlp.stanford.edu/data/wordvecs/glove.6B.zip
!unzip glove.6B.zip

wvectors = {}

with open('glove.6B.100d.txt') as f:
  for line in f:
    lsplit = line.split()
    wvectors[lsplit[0]] = np.array([float(v) for v in lsplit[1:]])

# We need to add word vectors, also, for contractions like it's and isn't,
# as these aren't in the Glove list.  We will do a simple averaging of the
# vectors for it's parts, to represent the word.  So, 
# vec(isn't) = (vec(is) + vec(not))/2.  And if there are three words involved,
# you divide by 3.  List of contractions below clipped from the website
# https://mlwhiz.com/blog/2019/01/17/deeplearning_nlp_preprocess/
# (following a Google search for such a list).

contractions = {"ain't": "is not", "aren't": "are not","can't": "cannot", 
                "'cause": "because", "could've": "could have", "couldn't": "could not", 
                "didn't": "did not",  "doesn't": "does not", "don't": "do not", 
                "hadn't": "had not", "hasn't": "has not", "haven't": "have not", 
                "he'd": "he would","he'll": "he will", "he's": "he is", 
                "how'd": "how did", "how'd'y": "how do you", "how'll": "how will", 
                "how's": "how is",  "i'd": "i would", "i'd've": "i would have", 
                "i'll": "i will", "i'll've": "i will have","i'm": "i am", 
                "i've": "i have", "i'd": "i would", "i'd've": "i would have", 
                "i'll": "i will",  "i'll've": "i will have","i'm": "i am", 
                "i've": "i have", "isn't": "is not", "it'd": "it would", 
                "it'd've": "it would have", "it'll": "it will", "it'll've": "it will have",
                "it's": "it is", "let's": "let us", "ma'am": "madam", "mayn't": "may not", 
                "might've": "might have","mightn't": "might not",
                "mightn't've": "might not have", "must've": "must have", "mustn't": "must not", 
                "mustn't've": "must not have", "needn't": "need not", "needn't've": "need not have",
                "o'clock": "of the clock", "oughtn't": "ought not", "oughtn't've": "ought not have", 
                "shan't": "shall not", "sha'n't": "shall not", "shan't've": "shall not have", 
                "she'd": "she would", "she'd've": "she would have", "she'll": "she will", 
                "she'll've": "she will have", "she's": "she is", "should've": "should have", 
                "shouldn't": "should not", "shouldn't've": "should not have", 
                "so've": "so have","so's": "so as", "this's": "this is","that'd": "that would", 
                "that'd've": "that would have", "that's": "that is", "there'd": "there would", 
                "there'd've": "there would have", "there's": "there is", "here's": "here is",
                "they'd": "they would", "they'd've": "they would have", "they'll": "they will", 
                "they'll've": "they will have", "they're": "they are", "they've": "they have", 
                "to've": "to have", "wasn't": "was not", "we'd": "we would", "we'd've": "we would have", 
                "we'll": "we will", "we'll've": "we will have", "we're": "we are", "we've": "we have", 
                "weren't": "were not", "what'll": "what will", "what'll've": "what will have", 
                "what're": "what are",  "what's": "what is", "what've": "what have", 
                "when's": "when is", "when've": "when have", "where'd": "where did", 
                "where's": "where is", "where've": "where have", "who'll": "who will", 
                "who'll've": "who will have", "who's": "who is", "who've": "who have", 
                "why's": "why is", "why've": "why have", "will've": "will have", "won't": "will not", 
                "won't've": "will not have", "would've": "would have", "wouldn't": "would not", 
                "wouldn't've": "would not have", "y'all": "you all", "y'all'd": "you all would",
                "y'all'd've": "you all would have","y'all're": "you all are","y'all've": "you all have",
                "you'd": "you would", "you'd've": "you would have", "you'll": "you will", 
                "you'll've": "you will have", "you're": "you are", "you've": "you have"}

for key in contractions:
  s = contractions[key].split()
  wvectors[key] = np.array([0]*100)
  for w in s:
    wvectors[key] = wvectors[key]+wvectors[w]
  wvectors[key] = wvectors[key]/len(s)

# We also have to take care of possessive nouns, like mother's, father's, 
# sister's, etc.  For these, we will just average as follows:
# vec(x's) = (vec(x) + vec('s))/2.

v = wvectors["'s"]

for key in list(wvectors):
  wvectors[key+"'s"] = (wvectors[key] + v)/2

gwords = {key for key in wvectors}
# Now we build a time-sequence x of word vectors.  The way we construct this is
# as follows:  we take a time interval equal [0,T] equal to the duration of
# the audio file, and then in each time interval [i*STEP, (i+1)*STEP) we 
# associate a Glove word vector if the time interval is contained in the 
# interval of utterance of the associated word.  Intervals where no words were
# uttered are assigned a 0 word vector.

# Initialize STEP to 1/4 of a second.

STEP = 0.25

# Initialize x to all 0's.

x = np.zeros((int(duration/STEP)+1,100))


# Now we fill in the time sequence with Glove vectors, according to the 
# time steps the words appear in.

for i in range(len(data['words'])):
  for t in np.arange(data['words'][i]['start_time '], data['words'][i]['start_time ']+data['words'][i]['duration'], STEP):
    if(data['words'][i]['word'] in gwords): 
      x[int(t/STEP)] = wvectors[data['words'][i]['word']]
# Now we need to build our data arrays y and z -- input and output -- 
# to feed to Tensorflow / Keras.  What we'll do is concatenate together sequences 
# of BUNDLE timesteps (of length STEP seconds) of word-vectors (at thosse timesteps), 
# as input.  Output will be a single brain volume 1.5 seconds (BOLD latency) after the
# sequence finishes.  

# Initialize BUNDLE length.

BUNDLE = 20

# START is the start time, in seconds, of the voice portion of the recording.  
# start_steps is that amount, expressed in timesteps.

START = 21
start_steps = int(START/STEP)

y = np.array([[x[t] for t in range(u,u+BUNDLE)] for u in range(start_steps,len(x)-BUNDLE)])
# Next, we load the associated nii.gz fmri brain data, and convert it into an
# array using Nibabel.

fmri = nib.load('milky_fmri.nii.gz').get_data()

# We need to get the temporal offsets for the slices from the JSON file.

r = requests.get('https://openneuro.org/crn/datasets/ds001131/snapshots/1.0.0/files/task-milky_bold.json')

slicetimes = r.json()['SliceTiming']
TR = r.json()['RepetitionTime']

print('Slice time offsets (in seconds):', slicetimes)

image = nib.load('milky_fmri.nii.gz').get_data()

# Total number of TRs (repetition times) of fmri data.

TRs = image.shape[3]

# Now we correct for the slice time offset distortions, while also allowing us to
# find the brain response pattern at a continuous time, in seconds, from the start.
# The key fact we use is that the slice
# [:, :, z, T] was recorded at time T*TR + sltimes[z].
# So, given t, we find T such that
# T*TR + slicetimes[z] <= t < (T+1)*TR + slicetimes[z].
# Then, we take an appropriate weighted aveage of 
# [:, :, z, T] and [:, :, z, T+1] to get the 2D brain response at time t and
# third-coordinate z.
# (In the case where t < slicetime(z) -- that is, T < 0 -- we set the 
# [x,y,z] voxel to img[x,y,z,0].)
#
# Ine routine below, img = input 4D brain vector; stimes = slice time offsets;
# TR = repetition time; t = time (in seconds) from the start of the recording
# where we want to know the brain response pattern.  Routine returns a 3D
# array giving the brian response at that time.


def trueimage(img, stimes, TR, t):
  newimg = np.zeros((img.shape[0],img.shape[1], img.shape[2]))
  for z in range(0,img.shape[2]):
    T = int((t - stimes[z])/TR)
    weight = T+1 + (stimes[z] - t)/TR
    for x in range(0, img.shape[0]):
      for y in range(0, img.shape[1]):
        if (T < 0):
          newimg[x,y,z] = img[x,y,z,0]
        else:
          newimg[x,y,z] = weight*img[x,y,z,T] + (1-weight)*img[x,y,z,T+1]
  return newimg

z = np.array([trueimage(image,slicetimes,TR,t).flatten() for t in np.arange(START+BUNDLE*STEP+TR,TR*(TRs-1)-STEP,STEP)])

print(len(y))
print(len(z))

# Make y and z have the same length -- might throw away some data.

m = min(len(y), len(z))

y = y[:m]
z = z[:m]
model = Sequential()
model.add(LSTM(128, input_shape=y[0].shape))
model.add(Dense(len(z[0]), activation='linear'))

model.compile(loss='mse', optimizer = 'adam', metrics = ['mse'])

history = model.fit(y, z, batch_size=10, epochs = 10)
import matplotlib.pyplot as plt

# Let's see what the model predicts for the first BUNDLE time-steps, starting
# just after the music finishes, at TR = 15.

prediction = model.predict(np.array([y[0]]))

I = np.reshape(prediction, np.shape(image[:, :, :,0]))

fig = plt.figure()

for i in range(0,min(I.shape[0],64)):
  plt.subplot(8,8,i+1)
  plt.imshow(I[i,:,:])

plt.show()

fig = plt.figure()

# See the brain activation just after the music, after the first words are 
# uttered, at TR = 16, for comparison purposes. 

for i in range(0,min(I.shape[0],64)):
  plt.subplot(8,8,i+1)
  plt.imshow(image[i,:,:,16])


plt.show()

Of course, the big advances in applying brain data to AI will come with much better BCIs. That will be here before you know it. Stay tuned! (And keep your laser-plasma rifles charged and ready!)


  • Zaphod and Yuli Ban like this





Also tagged with one or more of these keywords: AI, BCIs, Brain-Computer Interface, Artificial Intelligence

1 user(s) are reading this topic

0 members, 1 guests, 0 anonymous users