A reader of mine recently wrote to me with a question which I’ll paraphrase.
The “tagalong” problem
Let’s say I have a production server. Let’s also say that in addition to that, I have a staging server where I push most changes before production so the changes can be reviewed. This works out fine most of the time.
But let’s say occasionally there’s a critical bug in production that needs to be fixed ASAP. I need to make my fix and push straight to production without the delay of going to staging first. Unfortunately when I do this I also push the last however many commits to production as well.
In other words, as a side effect of my bugfix, I end up pushing work to production that potentially wasn’t ready for production yet.
The fix
The solution is to always keep the master branch in a deployable state. I’ll mention a few tactics that can make this easier.
The first tactic is frequent deployments. I try to do all my work in the form of atomic commits which could each conceivably be deployed to production. And in fact, I tend to perform a deployment almost every time I commit to master or merge something into master.
The second tactic is feature branches. Feature branches help keep partway-done work out of master, helping ensure that a deployment of what’s on master never means deploying incomplete work. In my experience it’s good for a feature branch to have a lifespan of a few hours or a few days. Anything much longer than that gets dangerous because it leaves too much room for the feature branch to diverge from master.
But what if you need to work on a feature that spans, say, 4 weeks, and can’t be released until it’s fully complete? This is where my third tactic comes in, dark launching. Dark launching is where you separate the concept of deployment from the concept of release. Just because the code for a feature gets deployed to production doesn’t mean that that feature necessarily needs to be surfaced to users. The UI of that feature can be hidden using a feature flag, which can be as simple as something that says if false until you’re ready to unveil that feature and remove the code that hides it.
Recommended reading
The things I’ve shared above are basically DevOps principles. To learn more you can check out the books The DevOps Handbook and Continuous Delivery.