2 replies to this topic

[Yuli Ban]

 

We've reached the very rough edges of what's possible to do with GPT-3 out of the box, and we'll only see great improvements at the bleeding edge of experimentation. It's become clear what it can and can't do.

 

As I myself have never directly used GPT-3, I can't tell you the nuances of its capabilities— only that they lie on the inner border of a twilight zone between how we've come to expect AI and how AI will soon develop.

Still, I can dream, and I'd like to imagine that we are only a few months away from seeing the next iteration of OpenAI's language model.

 

If I can throw out some predictions for what it can do.

 

I've been having some misgivings that the number of data parameters might be disappointing— people hope for something like 10 trillion or more, but it may be a bit less than 1 trillion overall. However, if they optimize things differently and train the model on image and audio, a high-quality multimodal 800 billion parameter GPT-4 could be functionally superior to a more generic text-only 20 trillion parameter GPT-4. The sequence length may be pushed much further, such as to 10,280 or more. 

 

We've never seen a fine-tuned GPT-3 either, so I'd imagine one would be approaching par-human in its capabilities at certain specific tasks.  In terms of what an off-the-shelf out-of-the-box GPT-4 can do, however, I can only expect it to be par-human at natural language generation and tasks related to such. If it's trained on multiple kinds of data, its image output will probably resemble GANs circa 2016 or 2017.

 

Its text-generating capabilities will range from being able to generate short stories with a coherent beginning, middle, and end all the way to what I call "sylistic editing" or literary style transfer (i.e. transferring one writer's style to another piece). With sufficient memory, it ought to be able to hold conversations as well. GPT-3 can do something like this, but when I say "hold a conversation," I mean that it can actively draw on what's been previously said as well as draw from other pools of knowledge so that it's an actual conversation and not a series of prompts that feels like a conversation. GPT-3 can probably do this too if it were fine-tuned on it. 

 

It'll probably be revealed some time in the middle of 2021.

 

I'll stop here just in case others want to chime in with more realistic visions.

 

"Cut the crap, Yuli— you know what we want. Is it an AGI powering your waifu?"

 

1, I doubt it. indeed, I strongly doubt it. The first AGI may very well be from a language model and not include any brain data whatsoever, but GPT-4 is likely going to be more of an experiment in testing large multimodal transformer networks rather than an attempt at AGI. More of a "look at all the different things we can make a single AI do!"

 

2, I doubt even GPT-5 will be an AGI. Of course by "AGI," I mean something closer to an "oracle" type AGI, not a "godlike" AGI— first-gen AGI, or weak general AI, or however you wish to call it. But by that point in time (~2023 or so), plenty of very knowledgeable people will likely start second-guessing themselves and there'll be debates about just how really close we are to AGI with the estimates changing from "maybe 2045" to "could make one within a year or two."

 

Also:

[D] Biggest roadblock in making "GPT-4", a ~20 trillion parameter transformer

[tomasth]

Based on this

https://www.reddit.c...mic_efficiency/

 

Sam Altman of OpenAI expects the next big gains to come from more efficient algorithms

 

You should expect for GPT-4 equivalent to be an architecture change , rathen then much more scaling.

 

If that prove enought capability they (and competitors) would scale.

[starspawn0]

Well, they certainly need more efficient algorithms to handle longer context windows, as the current algorithms take too much compute as you expand it.  2048 tokens is too small to do much interesting.  If they can expand it to 20,000 tokens or even 100,000 tokens, they you're getting somewhere -- e.g. you can model long works of fiction, long conversations, give a chatbot something like a memory (at least for basic stuff), maybe even model image synthesis as part of the output.

 

And if you can crank up the context window to a million tokens -- about equal to a 3,000 page novel -- then you can start to model a person pretty well, at least for an hour-long conversation, and especially if visual data are added to train it (like in that Tech Review piece and arxiv paper from the other day); brain data could also be added, to make it even more human-like.  That should be enough to where it gets hard to tell an AI from a human.  

 

One really can't tell how far off that is. I'd guess it's already possible, if somebody were willing to risk large sums of money to build it (say, in the neighborhood of $100 million).

 

And, as usual, people will find a few flaws in its capability, and these will get reported on social media (by the machine learning equivalent of "tankies"), basically "see how stupid it really is."  But more thorough tests, like this one

 

https://arxiv.org/abs/2009.03300

 

will reveal that it has a very broad set of capabilities, and at or above human level for most of them.  It will not yet be called an AGI, but nobody will care about the fine distinctions (except the chronic complainers on Twitter).