Did anyone see the OpenAI code generation demo during Microsoft Build this week? Write a function name and a comment and OpenAI generated Python code that does that thing, and keeps context across functions?

I missed it! Do you have a link or name I can search for?

That's the one!

Same problems different day. Look, generating Python code is not exciting. The problem is not “just” writing the code. We often forget there is learning git, infra, package management, and like a million other things that this OpenAI does not help with for the other 99% of the world that cannot code and regardless of an AI helping them type strange looking Python — it will not change the fact that most people cannot code. It’s impressive, sure, but think about who benefits by writing Python faster…? Is speed really the problem?

The first time I saw tech like this was in Scratch to help beginners learn to code. They could write what they wanted the code to do (“make the ball bounce”) and get a handful of snippets that they could steal from. And it worked the other way, so if they were unsure what some code did, they could press a button to get potential descriptions based on how people had described similar code. Very handy for learners.

Right, but Scratch is a structured editor and essentially all-in-one. Python is far from that.

A key part of the demo is that it puts you in conversation with the computer about processes grounded in the language of your code base. It happens that Python is part of the shared language they used in this demo, but it’s really impressive that it is even conversant in verbs the user themselves defined. I can’t say I’d get any more or less out of it if they’d done it in Dark? The value of the demo isn’t in your ability to download it today, is it?

I’m sorry, explain to me how system hotkeys can do my taxes?

The same way you do if you use tax software by pressing keys. If you wrote it out in Plain English, it’d look like “Open browser. Navigate to taxcut.com. Type username. Type password. Paste my 1099.” with more words because I’m on my phone 😛 The point is, every process, transcribed, is code, so interpreting natural language into code is a big deal, if not exactly what you were calling for in the other thread.

It’s the unique stuff that counts. All these businesses have different constraints, services, environments, processes… OMG it’s endless. The OpenAI demoed is nothing but a search result in SO

I personally look forward to instructing a computer to help me generate any code required as input to every language you ever create. 😄

Oh, I certainly am too. It’s what I’m building at Storyscript… but it’s NOT translating to Python

It’s hard to tell from a screenshot, but the interaction seems structurally identical?

If the differences are not obvious (

serverless, gitless, codeless, envless, deployless

) than I’m not getting my point across.

It seems like you also really like to be able to generate code from natural language, and since that’s the only part of the demo I think is significant, we’re good 👍🏼

🤣 ❤️ you Tom! — We can agree on that for sure. But I implore you to think about the implications of using Python (or other languages) because it’s not easy, it’s seriously a pain in the ass and why only 0.3% of the world can do it.

Absolutely! We’re in such agreement there, and I’m honestly thrilled at seeing you take this direction in Storyscript.

Ignoring Storyscript for a moment, as a Python developer myself, I don’t see the demo as being “that” useful in how I write code. Most my code is unique business-logic requiring constraints and very intentional abstractions that I’m not optimistic OpenAI will account for. I would however use this demo’s talents on generating the most basic functionality in my application so that I don’t need to search SO 😉

I think if it reaches GPT-level understanding of language (a high bar and a low bar), it should be able to converse using your abstractions. I think that was the point of the part of the demo where they said “print a receipt” and it did so using the helper method that retrieved all the line items.

print(“Data {}“)

is NOT “print a receipt”… print a receipt is format data and send data to the receipt printer to be printed on paper 😉 🤣 Great an AI can call a “print” function but it’s light-years away from figuring out how to connect to a EPOS TM88…. (which I did professionally for years, it’s a fucking nightmare to manage that complexity)

One subtlety here that will probably limit the practical impact of this in the near future -- the process of verifying whether the generated code is correct or not is essentially the same as debugging somebody else's code without them around. For functions on the simpler side, that may be a simple enough task, but as complexity increases that difficulty scales superlinearly.

Steve, I’m really curious where the rubber meets the road in your ideal system. If someone said to you, “Weekly, look at the forecast and adjust the number of drivers required to deliver food by comparing average orders and weather,” wouldn’t you start by looking at your tools an compiling your available actions into a procedure to achieve your goal? You might defer some of that work, and you might keep the original request in mind to evaluate your effectiveness and correct course over time, but so long as the request is made using language and not a timed series of motor commands injected as electrical impulses directly into your brain stem, it seems compilation is an essential part of the process of doing someone else’s work… yet you seem to hate compilation in every form people have discussed today, be it template matching of pseudo-English all the way down to machine code, or ML-based machine translation to Python—but somehow compilation of short English phrases to Storyscript gets it right? Like… what if PEP bottomed out at Storyscript, or the OpenAI system emitted Storyscript, allowing someone to write three Storyscript statements using only one command? Neither of those systems were written to preclude this, but you act like their existence is unjustified because they don't.

So they’re just throwing ML at the problem, much surprised, very wow. I agree with steve here - an ML trained on a bunch of snippets from github/SO is 99% sure not to have or understand what I’m doing when I’m juggling the domain abstractions. Yeah it could help with writing down the boilerplate, but wtf do we even write boilerplate then? Just make a template instead of making an “AI”. I mean, it’s a good idea, but so far away from reality that doing this is just picking the low hanging fruit and saying “oooh future”.

@Ian Rumac @josh 🙏 Well said. I'm happy you both see that the challenges of generating and validating code that's is beyond simple functions.

I dunno, maybe we're off in weeds that are only interesting to me, but could you point to where the boilerplate is in this example? The bodies of these functions were written by the ML model and required making several design decisions without human intervention, about how values should be computed, what nearby methods could be reused, and what output should look like. I think in any other context, we'd call the parts that the human wrote "boilerplate" and the parts that the computer wrote "programming".

@alltom the above cannot be generated by OpenAI, there are many assumptions that it made and decisions that it made which throw many red flags for me. However again, I think we are on different pages here because what I see in this screenshot is the “simple” parts of coding, the “hard” parts concern the abstractions, security, data integrity, integration and scope concerns that an AI, in the general purpose sense, cannot manage well (and I think never will unless you narrow the scope). @josh was spot on when hinting at verifying generated code. The system may have come up with multiple potential candidates, How do you pick them? How do you provide the extra context or params when required. @alltom I promise you that this demo was hacked… I have little confidence that the results you see are fully using OpenAI. I would love to try it out and give it just a little ambiguity and see it fail constantly. There is so many issues with the above… I don’t even know where to start. I would bet 💰 it was mostly faked.

FWIW -- I think it's very plausible that this particular example was generated honestly by the team. This is a pattern that I'd expect to be well-represented in the corpus of training material available on github, and so I'd expect the AI to usually get close. Particularly when using logical constraints as part of the generation process (think techniques like SRL). Occasionally, close will end up being completely correct. The issue with trying to use this for writing production code is that even an 80% success rate on individual expressions produces a lot of bugs over time without skilled programmers constantly debugging newly generated additions. I think it's probably a wash for most programmers to try to use this for development (assuming it's in that state), but reflecting more on it, this seems like potentially a huge accessibility boost for programmers who are blind or have lost dexterity. Definitely an avenue worth exploring more on the road to AGI. 🙂

Having built systems that can take arbitrary english and write sophisticated queries with high accuracy, these examples seem entirely doable given the technologies we have. Also knowing several of the people at OpenAI, including Sam, it's extremely unlikely the demos were "mostly faked." Are they general? Dunno. They made no claims that it was and we'd need to see more. But either way it's probably wrong to be so convicted that it's all just lies. In general, you'll gain much more from evaluating things from the prospective of how they could work, rather than assuming they're impossible and the people behind them are just using smoke and mirrors. It's also better for the community if we assume people are showing things in good faith, instead of trying to trick you.

@Steve For what it's worth, when I made this comment - https://futureofcoding.slack.com/archives/CCL5VVBAN/p1587142633117200?thread_ts=1587131460.117000&cid=CCL5VVBAN - my immediate reaction to your demo had been "this must be faked"!

@Chris Knott It was faked, of course! That’s what marketing is all about.