funkervogt wrote: ↑Tue Feb 13, 2024 3:07 pm
This excellent essay on Astral Codex Ten explains why the success of GPT-5 is so important to the pace of AI development. Let me explain:
There have been four GPT AI's so far: GPT-1, 2, 3, and 4. On average, each took 18 months to invent after the last.
Each one cost roughly 30x as much money to create as the last. Because GPT-4 cost $100 million, that means GPT-5 will cost about $2.5 billion.
The company that created the GPT machines, OpenAI, has enough money to make GPT-5, and it will probably be released in the next year. However, the cost trend means GPT-6 will cost $75 billion, which is well beyond its means.
Microsoft owns OpenAI, and if GPT-5 is a blockbuster product (e.g. - eerily smart and competent), Microsoft could, with difficulty, make the $75 billion investment to build GPT-6.
The cost trend also reveals that GPT-7 will cost $2 trillion to build, which no one has to spare. Only a major country or group of countries could afford it, and they'd only spend that much if GPT-6 were a super-blockbuster product, making the reward of an even better computer worth it.
The amount of computer chips and electricity needed to support each successive GPT machine has also been increasing exponentially, and while building GPT-5 is still within OpenAI's means, GPT-6 isn't--it would need something like a town-sized server farm and a dedicated nuclear reactor.
So if GPT-5 is underwhelming, isn't massively adopted across the world for all kinds of tasks, and doesn't make OpenAI billions of dollars in profit per year, it won't be worth it for anyone to build GPT-6 at the same pace the previous five GPTs were built. Instead, they'll slow down and build GPT-5.1, then -5.2, etc., with a year or two between each iteration. Moore's Law and variety of small hardware and software improvements will make a slower development strategy affordable: by the time GPT-6 capability is achieved, the total cost will have been much less than the $75 billion that would have been needed to build it a decade sooner and in one, big leap.
If the scenario in the preceding paragraph materializes, the general public will misperceive it as "proof" that AI was hype and can't actually be built, in the same way that the slowdown in progress with autonomous vehicles has convinced people the technology will never succeed.
The scaling problem with LLMs hits home how bad their learning algorithms are, and how much the tech industry is trying to brute force its way to creating an AI by throwing more computer chips and electricity at the problem. Within the next three or four years, we'll know whether this paradigm can continue, and there will be major implications for the tech industry and popular perceptions of "AI."
If GPT-5 is really good, then buckle up because mass job displacements, social changes, science and tech acceleration, and Manhattan Project-levels of investment in AI that will be felt across politics and the world economy will start before the end of this decade.
However, I predict it's much likelier GPT-5 will fall below the necessary performance threshold. While an impressive product, it won't be good enough to inspire the immediate creation of GPT-6. LLM's will probably suffer from an insurmountable "last mile problem" that keeps them from evolving into true AGIs, though in many narrow domains they will equal or surpass human intelligence and competency and they'll be able to convincingly "fake" general intelligence 99% of the time (which some dumber humans will mistake for proof GPT-5 is sentient). I believe we'll have to wait for several important algorithmic breakthroughs to happen to achieve true AGI, which I think could happen as early as 2050.
https://www.astralcodexten.com/p/sam-al ... 7-trillion
