If I had a nickel for every time our deployment strategy for a new or different environment was to edit a few config files and then run some batch files and then edit some more config files, and then it goes down in a steaming pile of failure, I would buy a LOT of Sriracha.

image

(Picture http://theoatmeal.com/)

Here’s a config file.  Lets say we need to edit that connection string:

<Setting name="ConnectionString" 
value="Data Source=(local); Initial Catalog=SportsCommander; Integrated Security=true;" />

Now let’s say we are deploying to our QA server.  So after we deploy, we fire up our handy Notepad, and edit it:

<Setting name="ConnectionString" 
value="Data Source=SCQAServ; Initial Catalog=SportsCommander; Integrated Security=true;" />

OK good.  Actually not good.  The server name is SCQASrv not SCQAServ.

<Setting name="ConnectionString" 
value="Data Source=SCQASrv; Initial Catalog=SportsCommander; Integrated Security=true;" />

OK better.  But wait, integrated security works great in your local dev environment, but in QA we need to use a username and password.

<Setting name="ConnectionString" 
value="Data Source=SCQASrv; Initial Catalog=SportsCommander; UserID=qasite; Password=&SE&RW#$" />

OK cool.  Except you can’t put & in an XML file.  So we have to encode that.

<Setting name="ConnectionString" 
value="Data Source=SCQASrv; Initial Catalog=SportsCommander; UserID=qasite; Password=&amp;SE&amp;RW#$" />

And you know what?  It’s User ID, not User ID.

<Setting name="ConnectionString" 
value="Data Source=SCQASrv; Initial Catalog=SportsCommander; User ID=qasite; Password=&amp;SE&amp;RW#$" />

OK, that’s all there is too it!  Let’s do it again tomorrow.  Make sure you don’t burn you don’t burn your fingers on this blistering fast development productivity.

I know this sounds absurd, but the reality is that for a lot of people, this really is their deployment methodology.  The might have production deployments automated, but their lower environments (DEV/QA/etc) are full of manual steps.  Or better yet, they have automated their lower environments because they deploy there every day, but their production deployments is manual because they only do it once per month.

And you know know what I’ve learned, the hard and maddeningly painful way?  Manual process fails.  Consistently.  And more importantly, it can’t be avoided. 

Storytime

A common scenario you see is a developer or an operations person (but of course never both at the same time, that would ruin the blame game)  is charged with deploying an application.  After many iterations, the deployment process has been clearly defined out as 17 manual steps.  This has been done enough times that the whole process is fully documented, with a checklist, and the folks running the deployment have done it enough times that they could do it in their sleep. 

The only problem is that in the last deployment, one of the files didn’t get copied.  The time before that, the staging file was copied instead of the production file.  And the time before that, they put a typo into the config.

Is the deployer an idiot?  No, as a matter of fact, the reason that he or she was entrusted with such an important role was that he or she was the most experienced and disciplined person on the team and was intimately familiar with the workings of the entire system.

Were the instructions wrong?  Nope, if the instructions were followed to the letter.

Was the process new?  No again, the same people have been doing this for a year.

At this point, the managers are exasperated, because no matter how much effort we put into formalizing the process, no matter how much documentation and how many checklists, we’re still getting failures.  It’s hard for the mangers to not assume that the deployers are morons, and the deployers are faced with the awful reality of going into every deployment knowing that it WILL be painful, and they WILL get blamed.

Note to management: Good people don’t stick around for this kind of abuse.  Some people will put up with it.  But trust me, you don’t want those people.

The lesson

The kick in the pants is, people are human.  They make mistakes.  A LOT Of mistakes.  And when you jump down their throat on every mistake, they learn to stop making mistakes by not doing anything.

This leads us to Mooney’s Law Of Guaranteed Failure (TM):

In the software business, every manual process will suffer at least a 10% failure rate, no matter how smart the person executing the process.  No amount of documentation or formalization will truly fix this, the only resolution is automation.

 

So the next time Jimmy screws up the production deployment, don’t yell at him (or sneer behind his back) “how hard is it to follow the 52-step 28-page instructions!”  Just remember that it is virtually impossible.

Also, step back and look at your day to day development process.  Almost everything you do during the day besides writing code is a manual process full of failure (coding is too, but that’s what you’re actually get getting paid for).  Like:

  • When you are partially checking in some changes to source control but trying to leave other changes checked out
  • When you need to edit a web.config connection string every time you get latest or check in
  • When you are interactively merging branches
  • When you are doing any deployment that involves editing a config or running certain batch files in order or entering values into an MSI interface, or is anything more than “click the big red button”
  • When you are setting up a new server and creating user or editing folder permissions or creating MSMQ queues or setting up IIS virtual directories.
  • When you are copying your hours from Excel into the ridiculously fancy but still completely unusable timesheet website
  • When, instead of entering your hours into a timesheet website, you are emailing them to somebody
  • When you are trying to figure out which version of “FeatureRequirements_New_Latest_Latest.docx” is actually the “latest”
  • When you are updating deploying database changes by trying to remember which tables you added to your local database or which scripts have or have not been run against production yet

It’s actually easier to find these things than you think.  The reason is, again, it is just about everything you do all day besides coding.  It’s all waste.  It’s all manual.  And it’s all guaranteed to fail.  Find a way to take that failure out of your hands and bath it in the white purifying light of automation.  Sure it takes time, but with a little time investment, you’ll be amazed how much time you have when you are not wasting it with amazing stupid busywork and guaranteed failure all day.

7 thoughts on “Mooney’s Law Of Guaranteed Failure

  1. Pingback: The Mooney Project » Web.Config: Code vs Configuration

  2. I would note that your examples are very realistic for manual processes… but there are better manual processes available, and some of them can exist in the range of 90% < success rate < 100%.

    First off, a less bad manual process dispenses with the manual typing entirely. Cut and paste rules the day.

    Second, an even less bad manual process has template files for each environment – when the "being promoted from" file is the previously known one, then simply use the "being promoted to" file. When not, use a tool to merge the two, similar to tools used to merge code branches.

    Third, I agree that the intelligence of the person doing the manual process is likely irrelevant. The diligence of the person, however, is extremely relevant, and I believe is unrelated to intelligence. A person who consistently displays careful, methodical, diligent, self-checking/double-checking behaviors will likely have a higher success rate when the documentation for the manual process is accurate than will person will less of those traits. You can, in fact, hire for these traits/skills – they're similar to the traits really good QA people exhibit.

    Fourth, if you want to get even higher success rates for manual processes, then have a two-person solution, Extreme Programming style… but much slower and more careful.

    Fifth, note that automation is well and good, but turns into a manual process every time something breaks anyway. In an unstable environment, that's very often.

  3. I agree that manual processes are not necessarily bad. Airline pilots have long checklists that they carefully go through every time, and they consistently get these processes 100% right. Having two people does indeed help a lot.

    However I also agree with the main point of the original post: automatic processes are always better than manual ones. Having automatable processes that are still done manually is a form of technical debt.

  4. Only a moron would rely on a manual procedure. Engineer it out of the process.

    E.g. Name your configurations

    <Setting Name="Dev.ConnectionString" …
    and
    <Setting Name="Live.ConnectionString" …

    Hold the current configuration name in a separate file and locate that file where it is preserved between deployments. Alter your code to use the current configuration name to access config settings.

    Ditto for the other examples.
    Deployment == COPY.

    Sadly not even copy is 100% failure proof.

  5. You wrote “User ID, not User ID”… did you mean “UserID, not User ID?” Guess this proves your point ;-)

  6. I mostly agree with what you say but,
    any automated process is created manually thereby that creation is also prone to failure. Creating an automated process can be really time consuming. I reckon that if you only do a task 10 times or less then you should consider carefully before automating.

Leave a reply

required

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>