Monthly Archives: November 2013

DevOps + Reality = ???

One of the hot new buzzwords in software development these days is “DevOps”.  It seems that “cloud” and “big data” and “synergy” were getting boring, so now every CTO is trying to get some of that DevOps stuff injecting into their process so that they can deploy to production 10 times a day.

But what is DevOps really?  Like most cool trendy buzzword ideas, it grew out of a few smart people with some good ideas that did real, concrete, awesome things, before everyone else saw how successful it was and tried to figure out shortcut formulas to get there.


To me, DevOps is just the next evolution of how successful software teams find better ways to be more effective, and almost all of it is traceable back to the ideas of the Agile Manifesto

An informal and woefully incomplete list of this this evolution would be:

  • For a while, the focus was getting the complexity of enterprise software under control, and building writing that code as quickly and efficiently as possible.  You saw tools RAD tools, development patterns like object orientation, and design approaches like UML and RUP.  This mostly helped single developers and development teams.
  • Once developers had figured out ways to make individual and independently-developed components easier to build, they had to deal ensure that their code fit together nicely with everyone else’s, out of which continuous integration was born.  This help build the stability of interaction and coordination between development teams.
  • As the development teams go faster and more flexible at building stuff, the projects that defined and managed what they they should be building and when they could expect it to be done.  Agile project management processes like Scrum filled this need.  This helped improved the communication and effectiveness of the whole project team, from developers to QA to project manager, and even product owners and business users (who were very underrepresented in previous approaches). 
  • Another challenge of building software so quickly with so many changes along the they way was validating the software.  When you are moving fast enough, and handling changes to the project scope and desired functionality, it’s hard to capture what the actual correct functionality should be, and whether the software you are building meets those requirements.  This brought along several approaches to automated testing, such as unit testing, integration testing, and UI testing.  Developers starting using patterns like Test Driven Development, and started working with and communicating with the QA team to ensure that there was a shared vision of what the expecting quality of the system was.  This increased communication between development and QA resulted in less focus on silly distractions like bug counts and whether something is actually a defect by strict definition, and more focus on everyone working together build the highest quality system they could.
  • Having semi-conquered many of the problems above, many product teams took the agile ideas a few steps farther to get the business users more involved.  While it was always important to make sure that the product being built was what the users actually needed.  More importantly, they wanted to ensure that they were genuinely solving the user’s problem; this required working with the users, asking questions, getting to the root of the problems, and offering potential solutions, rather than just demanding a requirements document.  To help along this communication and discovery process, ideas such as Behavior Driven Development and Specification By Example were developed to ensure that the business users, the only people who really know what needs to be done, are more involved in the project (serving as Pigs rather than just Chickens, if you will).
  • Now, having handling many of the hard parts of efficiently building software that actually solves the user’s problems, there has been a focus on how to ship and support that software.  This has involved working with operations teams to create streamline deployment and monitoring of the systems throughout various environments.  And while this “DevOps” approach is solving a long list of long-standing problems in this business, it is, unfortunately, doomed to be the The Next Big Thing, the next Game Changer, the next Silver Bullet, and that is a Bad Thing.


[Buzzword]-In-A-Box = Failure-In-A-Box

Notice a pattern there?  It’s like an every-growing blob, pulling in more people from diverse teams.  Starting with just the developers, and then expand to other development teams, QA, project management, business users, and now operations.  Each step of the process involved building out bridges of communication and cooperation across teams.

But then it goes wrong.  Each of these steps went through a similar buzzword-ifcation.  Those steps were:

  • Some pioneering teams start to develop new processes for help they succeed with other teams.  Being analytic folks who can react and respond to new information, the most successful ones are developed over time with feedback from the other teams and a critical eye towards continuously improving the process.
  • Other folks notice this success and want to capture it, so the processes start to become more formalized and defined.  While the original processes were very customized, the industry starts to get a better of idea of what parts work well in more generic settings.
  • As the patterns become better understood, the repetitive friction points are identified and companies begin to build tools to automate away that friction and give people freedom to focus on the core benefits of the process.
  • More people, looking for the quickest way to get the most value from the concept and begin to think that the tools and formalized processes is the key to accomplishing that.You want DevOps!  I'll get you some DevOps!
  • Eventually, large companies are hiring high priced consultants and buying expensive enterprise tools as a part of a corporate-wide initiative to capture that buzzword magic.  This focuses on dogmatically following the process and the tooling.
  • While lip-service is paid to the idea of communication, and cross-team meetings are set up, it takes a backseat to the process and tools.  This is because cooperation and communication takes a lot of work over time to build, and that is not something you can sell in a box.  In the end, those companies are buying valuing Processes and tools over Individuals and Interactions, which is the complete reverse of the Agile Manifesto that drove the concepts to the great idea in the first place.
  • The magic is dead. Consultants are making money, executives are touting their groundbreaking strategies, and in the trenches the same pathologies remain.  While the masses try to follow the recipe and failed to be effective with it, the groundbreaking folks are off to solve the next problem.


Next Stop: DevOps

So what is this DevOps thing?  In it’s simplest sense, it expanding the development team beyond just developers and QA and PMs to include the operations and deployment teams.  The result is what you might call a “product delivery team”.

The first step, like all of of the other steps of the evolution, is “stop being a jerk”.  Get yourself OK with that first, and come back when you’re done.  “But I’m not a jerk, some people are just idiots”.  Yeah, that means you’re still a jerk.  As soon as you find yourself saying or even thinking that someone on your team (or worse one of your users) is an idiot or a moron or useless and their job responsibilities are pointless, you have more deep-seated issues to work out first.  Before you can succeed, you MUST treat those everyone on your team and other teams with dignity and respect.  And the hardest part about this is that you can’t just go through the motions, but deep down inside you need to actually believe it.  If you think that is too touch-feely, or you don’t think you can do that, or you don’t want to or you don’t believe that it’s necessary, that’s fine: Go away, and stay the &$@# away from my projects, you miserable person.  The idea of the "smart but difficult-to-work-with-developer” is a crock.  If you can’t work effectively with other developers and people on other teams, I don’t care how many books you’ve read, you suck as a developer, in my humble opinion.

OK, the next step is to actually talk the other people.  Recognize that as hard and important as your job may be, theirs is probably just as hard and just as important, and the only way your team is going to get better is if everyone works to make everyone’s job easier.  So set up a meeting with them, and ask, with a straight face and genuine interest, “what makes your job more difficult, and what can we do to make it easier”.  Then watch their face light up.  I guarantee you they have list of annoyances and manual steps and remediation steps that waste their time every day and they will be happy to have an opportunity to gripe about them in a welcoming setting without having to worry about being labeled a “complainer”.  Examples would be “we don’t know when something is deployed or needs to be deployed” or “we don’t know what changes are including in each release” or “every time I have to deploy something I need to copy files all over the place and edit some scripts and the dev team always forgets to include a certain file”.

Now, you will be tempted to smile and nod and shrug your shoulders and explain that it’s not a perfect system but it has worked so far.  Suppress this urge, write down the concerns, and start talking about them.  Throw around blue sky ideas of possible solutions.  Get an idea of not just want hinders them, but what would actually make them succeed. 

OK, now what is the official title of this part of the DevOps process?  There are probably several names, but I prefer “stop being a jerk, talk to people, and find out you can solve their problems”.  What tool should you use for this?  Usually Notepad, Evernote/OneNote, or a piece of paper, and an open mind.

Now, before you are done talking to them, pick a few of the most offensive and/or easiest problems and promise to fix them.  In fact, schedule a follow up meeting before you even leave the room, for a few days or a week or two away, where you will show you’re half-done solutions and get feedback about whether it actually is going to solve the problem or what you might be missing.  Or now that you gave them something to visualize, what brand new thing they thought of that would make it 10x better.  Or even that they now realize that were wrong, this is not going to solve the problem, so maybe we need to try something else; this is not a bad thing, and try not to get frustrated.  Instead, repeatedly stress to them that your main goal here is to make their life easier, not just because they want to hear that, but because it’s 100% true.

Sound simple?  It really is.  But that is the core of DevOps.  If you do this first, everything else will start to fall into your lap.  There are a bunch of tools and procedures and techniques that you can leverage to solve the problems, but you need to be sure that you are actually solving the right problems, and to that you need to build a positive working relationship to help root out and identify the solutions to those problems.  The tools are awesome, but you HAVE to focus on the “individuals and interactions over processes and tools”.  But once you have in place, you can do anything.



Accessing an Amazon VM through WMI


OK, so after figuring this all out again for the second time this year, I figured it’s time that I write it down for when I eventually forget again.

So I’m working on adding some changes to the DropkicK library in the Chuck Norris Framework.  DropkicK is an AWESOME tool for deploying just about anything in Windows, and the vast majority of all of the deployment stuff I’ve built over the last year has been heavily based on DropkicK.  Go hear Rob Reynolds talk about the Chuck Norris Framework a lot on Dot Net Rocks and Hanselminutes.

However, while it works great for remotely deploying stuff when your domain account is an administrator on the target server, but it doesn’t yet support connecting as a local administrator.  So what needs to be added is the ability to provide a username and password for an administrator on the target machine.

Why? Amazon.  The most common Amazon EC2 setup I encounter with my clients is that they are just a bunch independent machines, each with their own local user accounts.  Even those that are in a VPC don’t have a domain controller or anything else that would allow the same authenticated user to access multiple machines from the same session.

So that’s something I’m working on now.  Underneath the covers, the DropkicK code is surprisingly straightforward and uses WMI for just most things, and those WMI components take an optional user name and password when connecting, so it’s just a matter of exposing the administrator user name and password as deployment parameters, and then threading them through to the WMI objects.  No big deal.

What IS a big deal though, is getting WMI to work with an Amazon VM in the first place.  You’d think it would be pretty easy, but you’d be wrong.  Very wrong.  There are several things to get right, and if you don’t get them right, you’re going to get some of the must useless error messages you have every seen.

So I just got it working, and here’s how I did it.

Getting Started

First, you go create your self an Amazon VM.  Make it any size you want, but you’ll probably want a Windows base install.

I used Windows 2012 for this, but I went through this same pain earlier in the year with Windows 2008, and it was the same.

Also, when creating your VM, give it a new security group.  Right now you can start with just RDP access, but you’ll going to be adding a bunch of firewall exceptions specific to WMI, so you’ll probably want to keep this type of stuff isolated.

OK, so once your VM is up and running, first create yourself an administrative user account, something like “mmooney”.  Then log out as administrator, and go log in as that new user account.  In fact, don’t go back in through the “administrator” account again.  What?  You and every developer on your team likes to use the “administrator” account on every server?  Stop it.  Stop It.  STOP IT.  Bad Bad BAD.

Now go in and install all of the Windows features/roles/whatever-they-call-it-this-year that you normally need (IIS/MSMQ/whatever) and run a Windows Update.


Now, lets get a baseline of failure.  After a LOT of googling on solving WMI issues, i finally stumbled across a blog post mentioning WBEMTEST, and I was furious that I didn’t know about it sooner.

WBEMTEST is a WMI test client, already installed on your machine.  Go ahead, run wbemtest from a command line, and it will launch it up.

It’s basically the type of throughout test UI that you probably wrote on your third and fourth projects you ever worked on, after you learned it was a useful investment of your time, and you were still young and idealistic enough to spend a few hours building out a cool test tool like this.  But those days are gone; you are now old and slow and lazy and so many years of custom built tools have come and been used and then gone and been forgotten about washed over you like yet another wave flowing down then endless river of projects that your career has become.  Anyway.  That’s OK, because someone already built this one for you.


So hit Connect accept the defaults that will point to your own machine, and you’ll get all sorts of fancy options.  Play around with it.  Go ahead.  Play I say.

image image

Now let’s go to your VM.  Hit connect again, and instead of the default “root\cimv2”, put “\\[YourMachineIP]\root\cimv2”.  Ka BOOM.  Kinda.


Ah, “The RPC server is unavailable”.  Simple enough, clearly accurate, but enormously unhelpful.  Get used to this message folks, it’s going to be following you around for a while.

Amazon Security Settings

The problem here is that you have a few firewalls blocking you from accessing that server.  This is one of those “good” safe-by-default security things, because you can do some nasty stuff if you get WMI access to a machine.  Sure you’ll still need an administrative username and password, but you really don’t want “guessing some guys password” as the only thing between you and p!@wnzge (or wherever those kids call it).

But here we actually want to get in.  first we’ll need to poke a hole in the Amazon firewall. By “small” I mean a giant gaping hole that you could fly a spaceship through.

To access WMI, you will need to open TCP ports 135 and 445.  Oh and 1024 through 65535.  Yes, that’s right.  WMI will try to connect through one random port in that range, and you can’t easily tell it which one, either from the client or the server.  I spent a lot of time trying several things get it locked down to a single port or list of ports, but came to the conclusion that it was pretty much not possible.

While you are in here, ask yourself if you will also want to be able to access the machine through a file share (\\[IPAddress]\C$ or something similar).  If so, also add allow holes for TCP  135-139, UDP 135-139, and UDP 445 (you also want TCP 445, but you did that above).

But PLEASE make sure you restrict the IP range to the servers that you are actually expecting to connect.  Do NOT leave it as  That’s just asking for trouble.

When it’s all said and done, your security group should look something like:


Now if you try to telnet to any of these ports, or use WBEMTEST, you’ll probably still get RPC Unavailable, because the Windows Firewall is blocking you.

Windows Settings


So now go into your VM and bring up the firewall settings.  Make sure all of the Windows Management Instrumentation Services rules are enabled for the Public role (make sure you’re not in Domain or Private or you’re going to make and bunch of changes and nothing will happen and you’ll be confused and angry and you’ll blame me and my stupid blog post and that’s no fun, at least not for me). 


Then in Computer Management, drill down to Services And Applications->WMI Control.  Right click the WMI Control node and select Properties.  On the Security tab, select the Root node and Ensure that the Administrators group has access to everything (it probably will).


Then go into the Services window and make sure the Windows Management Instrumentation service is running.


So let’s try again now.


Final Steps, One Crazy Weirdness

Now if we go back to WBEMTEST and try to connect, we get a little farther.


Access Denied is good!  We got through the firewalls and we got a response.  So let’s put in an administrator in the user name and password…


WAT.  “The object exporter specified was not found”.

What the deuce does that mean?  If you look around, you’ll see people having this issue connecting through a host name and the solution is to use the IP address to get it to resolve correctly.

But we ARE using the IP, right?  Sort of.  We are using the public IP, not private IP that the VM would actually use to identify itself. 


In fact, if you go into another Amazon VM and try to connect to this one through its private IP, it actually works.  But that is not help to me when I’m sittimg at home in my bunny slippers trying to push a change from my desktop.  To the best of my knowledge you cant access Amazon VMs through their private IPs from outside he Amazon cloud (at least not without a lot of networking voodoo that is above my paygrade).

BUT, you can also connect to WMI through a host name.  No, not that crazy public DNS host name, that won’t help you any more than the public IP.  Instead you have to use the actual machine name that the machine itself is aware of.


Of course, your local machine is not going to resolve that, but if you add a host file entry, you should be all set.  In case you’ve forgotten, that’s at C:\Windows\System32\Drivers\etc\hosts, and you’ll need to open it with a text editor that is running in Administrator mode if you have UAC running on your machine.


Now it we go back to WBEMTEST try to connect to “\\[PrivateMachineName]root\cimv2”, it works!

image image


Well, hopefully that helps some folks.  Or at least helps me again in 6 months when I run into it again.