Sleeping at the office, nothing to show

You know a project is under serious pressure when people start camping out at the office.

In 2015 I got a job at an agency where they were doing a big project for a large Chinese electronics manufacturing company. The project had been underway for about six months.

Two of the developers, who I’ll call Dave and Eric, had been working on the iOS and Android app portion of the project. I was told that for a time they had been sleeping at the office at night in an effort to hit their timelines.

The application had a Rails backend. I have a vague memory that the person who wrote most of the Rails app wasn’t around anymore.

Six months into the project, with a good portion of the budget already spent, the agency had no working application to show their client.

There were a lot of problems with the way the project had been carried out up to the point I got involved. They gave the design work to a very junior designer. The designs didn’t make sense and were not implementable. In the apps that were built, many things were only partway done and very little was all the way done.

There was also no production environment. For whatever reason it was decided that the project would be hosted on AWS. I had never touched AWS at the time but simply because I was the least unqualified person on staff to do it, the task of getting the Rails app onto AWS fell to me.

It took me forever to get things working on AWS. But I finally got it. It was not a simple deployment. The nature of the application required servers in different regions of the world. Because AWS regions are independent, we had to devise a complex messaging system to keep data in sync across multiple regions. It was a nightmare. Compounding the nightmare was the fact that we were under intense time pressure. Everything we were doing was supposed to have been done weeks or months prior.

Done by Thanksgiving? “I think we can do it!”

One of my most vivid memories from this project was kind of a “go/no go” meeting with a group of high-up people at the agency. This meeting included myself, the agency president, several people with “VP” in their title (which was almost everybody at the agency) and a few others. There were maybe about 12 people in the room. At this point in time we had very little working in production and about six weeks before “pencils down” time. If we didn’t have the project done by Thanksgiving, the whole thing was for naught.

At this meeting we discussed what still needed to be done relative to the amount of time we had left. The scope was not very negotiable. My guess was that in order to get everything done in our scope we would need about six months. I remember that we went around the table at this meeting and had each person say whether they thought we could or could not hit our Thanksgiving deadline. “I think we can do it,” the first person said. “I think we can do it!” the next person said. Every single person in the meeting said they thought we could do it. Well I didn’t think we could do it. So I said, “No, I don’t think we can do it.” I could tell everybody was annoyed with me and thought I was being a stick in the mud. Six months of work when we only had six weeks left seemed “too bad to be true”. I agreed it sucked that we couldn’t get six months’ work of work done in six weeks. But the amount that it would suck to not get it done had no bearing on whether we could actually get it done.

$300,000+ down the drain

Turns out I wasn’t crazy. We got nowhere near getting the project done in time. The client spent (as far as I understand) multiple hundreds of thousands of dollars and got nothing.

How this sad story could have been avoided

Not every part of every software project is equally challenging. Some tasks, like creating a new CRUD interface in a Rails project, are really familiar and pretty trivial to implement. Others, like setting up an AWS environment across different regions and setting up a message queuing system so they can talk to each other, when you’ve never touched AWS in your life, are not so trivial.

When you have a mix of mysterious and non-mysterious work in a project you should do the most mysterious work first. Don’t save the super hard stuff for last.

You might think this is common sense. I would think so too. But surprisingly many people leave the most mysterious part of a project for the 11th hour.

This agency’s failed project might not have been quite such a spectacular failure if they had deployed to production on day one. If you deploy a mostly-empty application to production on day one, there’s way less stuff to have to get working. If something breaks in the process of setting up the production environment, you have way less stuff to sort through to figure out exactly where the problem lies. The stress level is also much lower at the beginning of a project.

Getting a production environment in place on day one makes it so every subsequent deployment is a non-event. Throughout the project you can do a number of deployments every week. If the project deadline is Thanksgiving there need not be a mad scramble to get things set up and working the day before Thanksgiving. You just deploy like normal. The workload might be heavier but the deployment process is uneventful.

For less risk and less stress, deploy to production on day one.

What does all this have to do with automated testing?

Like the authors of GOOS say, a healthy development process involves feedback loops ranging from seconds to months. These kinds of feedback loops may include:

• Unit tests (seconds)
• Entire unit + integration test suites (minutes)
• Pair programming sessions (seconds to minutes)
• PR reviews (minutes to hours)
• Feature releases (days to weeks)
• Entire development projects (weeks to months)

These feedback loops are enabled by practices like automated testing, pair programming, continuous integration and continuous delivery/deployment. These tests don’t exist in isolation but weave and synergize with each other. It’s hard to practice continuous deployment when you don’t have any tests. Continuous integration shines a light on your tests and forces you to pay attention to how healthy your test suite is. In addition to deploying to production on day one, it’s immensely helpful to get your testing infrastructure in place on day one and release a tiny feature complete with tests.