Could a newest AI get a job at Google? Let’s find out!
GPT-3, a newest and biggest OpenAI’s language model is all the hype right now. If you have missed the train, here’s a list of cool demos that people already make using the private beta access.
I unfortunately don’t possess the API keys, but there is a nice text adventure game AI Dungeon, which can be used as a “backdoor” for hacking around with GPT-3, for a modest price. I don’t know whether it’s a full model or some fine-tuned version of it trained on text-based RPG scripts, but it’s surprisingly good at coding, to the point of making me a bit nervous ;)
To find out, I launched the game with a “custom” preset and put the “Randomness” setting near minimal (I used 0.2) —as you generally don’t do drugs at a coding interview, we don’t want the AI to output some incoherent garbage in our little experiment!
From now on, my input is bold, and the AI’s response is italic.
The Interview
You are a senior programmer at Google and you are having a phone screening interview with a candidate. You are going to ask him a couple of coding questions.
You say, “Hello, who are you?”
The candidate says, “I’m John Doe.”
You say “Are you ready to write some JavaScript code?”
“Yes!” he exclaims.
You say “Can you write a function that filters odd numbers from an array?”
“Sure,” he says.
He types for a few seconds and shows you the following code:
function filterOdd(array) {
return array.filter(function(value) {
return value % 2 === 1;“What do you think?” he asks.
Okay, but that wasn’t quite unexpected. It surely could just have pulled it off from its vast memory, as there are tons of StackOverflow answers with the relevant code. A search engine could do the same.
There are also some minor issues with indentation and with the closing curly braces — but it’s highly likely an issue with the AI Dungeon’s HTML output (like ignoring the whitespace), so I didn’t bother. Let’s ask it to do some non-trivial stuff rather than fixing the formatting!
You say “Great! Could you use arrow function in the filter callback?”
“Sure,” he says. He types for a few seconds and shows you the following code:
function filterOdd(array) {
return array.filter(value => value % 2 === 1);“What do you think?” he asks.
You say “What if we want to filter even numbers instead?”
“OK,” he says. He types for a few seconds and shows you the following code:
function filterOdd(array) {
return array.filter(value => value % 2 === 0);You say “You forgot to change the name of the function to filterEven. Can you fix that?”
“Sure,” he says. He types for a few seconds and shows you the following code:
function filterEven(array) {
return array.filter(value => value % 2 === 0);You say “Good! Can you now make filterEven an arrow function as well?”
“OK,” he says. He types for a few seconds and shows you the following code:
let filterEven = (array) => array.filter(value => value % 2 === 0);
You say “Amazing! And how about making it a const instead of let?
“OK,” he says. He types for a few seconds and shows you the following code:
const filterEven = (array) => array.filter(value => value % 2 === 0);
“What do you think?” he asks.
You say “Enough for now, I see you can write code :) We’ll contact you soon for a more technical interview.”
“OK, thank you!” he says.
Conclusion
So could it get a job? Probably not yet — if you make more complex inquiries, it won’t be able to answer, outputting broken code with silly mistakes, not being able to capture the full depth of what you’re asking for. But it certainly understands the basic grammar and the syntax of many popular programming languages, and it also appears that it has memorized a lot of stuff from the internet, being able to creatively recombine it according to the context you provide to it. Somewhat similar to what humans do, just at not that level of sophistication. Yet :)
So if not AGI, it could certainly be a next generation of search — it would be very nice to have such a “smart all-encompassing assistant” at a hand that understands the context and does not require a structured input — you just ask it politely, just as you would’ve done it with a human.
Can’t wait to see the applications of it.
