When ChatGPT first came out, the first programming use case I thought of for ChatGPT was to write code. I thought of it like “GitHub Copilot on steroids”. I imagine that was a lot of other people’s first thought too. But gradually I realized that having ChatGPT write production code is actually not a very good idea.
When ChatGPT gives you a big chunk of code, how can you be sure that the code does exactly what you think it does? How can you be sure that it’s not missing something, and that it doesn’t contain anything extra, and that it doesn’t have bugs?
The answer of course is that you have to test it. But retroactively testing existing code is usually tedious and annoying. You basically have to replay the process of writing the code so that you can test each piece of code individually.
A programming workflow that involves using ChatGPT to write big chunks of code seems dangerous at worst and terribly inefficient at best.
If ChatGPT isn’t great for writing production code, what’s it good for?
Using ChatGPT to reduce mental labor
One part of our jobs as programmers is to learn general principles and then apply parts of those principles to a certain peculiar need.
I’ll give a somewhat silly example to illustrate the point starkly. Let say I want to integrate Authorize.net into a Lisp program and that I’ve never used either technology before. Without checking, one could pretty safely assume there are no tutorials in existence on how to integrate Authorize.net into a Lisp app.
In order to complete my integration project I’ll need to learn something about a) Authorize.net in general, b) Lisp in general, and c) integrating Authorize.net with a Lisp app specifically. Then I’ll need to synthesize a solution based on what I’ve learned.
This whole process can be wasteful, time-consuming, and at times, quite boring. In the beginning I might know that I need to get familiar with Authorize.net, but I’m not sure yet which parts of Authorize.net I need to be familiar with. So I’ll read a whole bunch about Authorize.net, but I won’t know until the end of the project which areas of my study were actually needed and which were just a waste of time.
And what’s even worse is the cases where the topics you’re studying are of no permanent benefit to your skills as a programmer. In the case of Authorize.net I might not expect to ever use it again. (At least I hope not!) This kind of learning is just intellectual ditch-digging. It’s pure toil with little or no lasting benefit.
This kind of work, where you first you study some generalities then you synthesize a specific solution from those generalities, I call “study and synthesis” work.
Thanks to ChatGPT, most “study and synthesis” work is a thing of the past. If I tell ChatGPT “Give me a complete tutorial on how to integrate Authorize.net with a Lisp program”, it will. The tutorial may not be correct down to every last detail but that’s not the point. Just having a high-level plan spelled out saves a lot of mental labor. And then if I need to zoom in on certain details which the tutorial either got wrong or omitted, ChatGPT will quite often correct its mistakes when pressed.
Using ChatGPT to write production code may seem like a natural and logical use for the tool, but it’s actually not a very good one. You’ll get a lot more leverage out of ChatGPT if you use it for “study and synthesize” work.
It accelerates the learning curve. You can use it in combination with good blog posts to narrow down and steer it in the right direction to get good results. In a few years from now, the code it generates will no longer be an issue for use in production. The verification stage still requires human judgment and tools that can check the code quality.
As developers we will soon be working at a much higher level of abstraction. The tools will be able to generate code in any language. ChatGPT generated code in 2023 is like CGI scripts of 1993. This is just a beginning. Startups will innovate and will make the developer tools more productive.