Starspawn0 also commented this tweet:
I don't know about that "light years away" comment. I also competed in programming competitions (though not for "thousands of hours", and I doubt this guy did either), though it's been a long, long time. Reaching about the 50th percentile in these competitions shows that the model is at least generating solutions that a human would say are "creative" (the people who compete in these competitions are young, yes, but they are like people who already have considerable skill -- at hacking code together, yes, not doing software development -- and are usually quick on their feet). And I doubt that 50th percentile is anywhere near the limits of what these models can achieve, given that they could have used a larger language model and didn't even use an inner-monologue. Maybe if they used Gopher it would have scored 75th percentile (getting maybe one or two more problems right per set); and then with an inner-monologue, they might could have boosted it to 90th percentile or higher. And even if you take out the input-output examples, it might still score above the 75th percentile.
As to the comment about how the time limit is the main limiter, that is correct. However, that applies to just about any competition -- if you have more time, you can solve the problems, as you can "brute force" it (trying lots of combinations to see what works). Beating humans in a 6 hour test is really impressive! -- it shows that the probability distribution of "solutions" generated is not so spread out that the correct answers are too far to the tail to be found by a random search in reasonable time.
This seems really important. If a larger language model ( Gopher, which exists now) + inner monologue ( easy implementation ) can bring AlphaCode into the top 10 %, that's kind of insane. They can build one today.
Would this mean that AlphaCode is (soon) good enough to improve its own source code? How can it do that? Deep neural nets are like black boxes, where do you start improving them?
Here's a comment from Reddit/singularity:
This is massive, we knew it was coming with Codex and now DeepMind has made it clear the next iteration of this is being built to be better than even the most skilled coders, there is little doubt future versions of this will be superhuman in their abilities. When Codex came out months ago, we knew it was over, and now we have something which Deepmind, if they really wanted, could bring up to speed very quickly and use to replace the average human coder and rent this program out to software firms for a fraction of the price.
They won't do that now, because it's still a prototype and that would be a shit storm that no one wants, but this technology is coming and it will absolutely be orders of magnitude better and faster than even the best programmers.