OK, so after figuring this all out again for the second time this year, I figured it’s time that I write it down for when I eventually forget again.
So I’m working on adding some changes to the DropkicK library in the Chuck Norris Framework. DropkicK is an AWESOME tool for deploying just about anything in Windows, and the vast majority of all of the deployment stuff I’ve built over the last year has been heavily based on DropkicK. Go hear Rob Reynolds talk about the Chuck Norris Framework a lot on Dot Net Rocks and Hanselminutes.
However, while it works great for remotely deploying stuff when your domain account is an administrator on the target server, but it doesn’t yet support connecting as a local administrator. So what needs to be added is the ability to provide a username and password for an administrator on the target machine.
Why? Amazon. The most common Amazon EC2 setup I encounter with my clients is that they are just a bunch independent machines, each with their own local user accounts. Even those that are in a VPC don’t have a domain controller or anything else that would allow the same authenticated user to access multiple machines from the same session.
So that’s something I’m working on now. Underneath the covers, the DropkicK code is surprisingly straightforward and uses WMI for just most things, and those WMI components take an optional user name and password when connecting, so it’s just a matter of exposing the administrator user name and password as deployment parameters, and then threading them through to the WMI objects. No big deal.
What IS a big deal though, is getting WMI to work with an Amazon VM in the first place. You’d think it would be pretty easy, but you’d be wrong. Very wrong. There are several things to get right, and if you don’t get them right, you’re going to get some of the must useless error messages you have every seen.
So I just got it working, and here’s how I did it.
First, you go create your self an Amazon VM. Make it any size you want, but you’ll probably want a Windows base install.
I used Windows 2012 for this, but I went through this same pain earlier in the year with Windows 2008, and it was the same.
Also, when creating your VM, give it a new security group. Right now you can start with just RDP access, but you’ll going to be adding a bunch of firewall exceptions specific to WMI, so you’ll probably want to keep this type of stuff isolated.
OK, so once your VM is up and running, first create yourself an administrative user account, something like “mmooney”. Then log out as administrator, and go log in as that new user account. In fact, don’t go back in through the “administrator” account again. What? You and every developer on your team likes to use the “administrator” account on every server? Stop it. Stop It. STOP IT. Bad Bad BAD.
Now go in and install all of the Windows features/roles/whatever-they-call-it-this-year that you normally need (IIS/MSMQ/whatever) and run a Windows Update.
Now, lets get a baseline of failure. After a LOT of googling on solving WMI issues, i finally stumbled across a blog post mentioning WBEMTEST, and I was furious that I didn’t know about it sooner.
WBEMTEST is a WMI test client, already installed on your machine. Go ahead, run wbemtest from a command line, and it will launch it up.
It’s basically the type of throughout test UI that you probably wrote on your third and fourth projects you ever worked on, after you learned it was a useful investment of your time, and you were still young and idealistic enough to spend a few hours building out a cool test tool like this. But those days are gone; you are now old and slow and lazy and so many years of custom built tools have come and been used and then gone and been forgotten about washed over you like yet another wave flowing down then endless river of projects that your career has become. Anyway. That’s OK, because someone already built this one for you.
So hit Connect accept the defaults that will point to your own machine, and you’ll get all sorts of fancy options. Play around with it. Go ahead. Play I say.
Now let’s go to your VM. Hit connect again, and instead of the default “root\cimv2”, put “\\[YourMachineIP]\root\cimv2”. Ka BOOM. Kinda.
Ah, “The RPC server is unavailable”. Simple enough, clearly accurate, but enormously unhelpful. Get used to this message folks, it’s going to be following you around for a while.
Amazon Security Settings
The problem here is that you have a few firewalls blocking you from accessing that server. This is one of those “good” safe-by-default security things, because you can do some nasty stuff if you get WMI access to a machine. Sure you’ll still need an administrative username and password, but you really don’t want “guessing some guys password” as the only thing between you and p!@wnzge (or wherever those kids call it).
But here we actually want to get in. first we’ll need to poke a hole in the Amazon firewall. By “small” I mean a giant gaping hole that you could fly a spaceship through.
To access WMI, you will need to open TCP ports 135 and 445. Oh and 1024 through 65535. Yes, that’s right. WMI will try to connect through one random port in that range, and you can’t easily tell it which one, either from the client or the server. I spent a lot of time trying several things get it locked down to a single port or list of ports, but came to the conclusion that it was pretty much not possible.
While you are in here, ask yourself if you will also want to be able to access the machine through a file share (\\[IPAddress]\C$ or something similar). If so, also add allow holes for TCP 135-139, UDP 135-139, and UDP 445 (you also want TCP 445, but you did that above).
But PLEASE make sure you restrict the IP range to the servers that you are actually expecting to connect. Do NOT leave it as 0.0.0.0/0. That’s just asking for trouble.
When it’s all said and done, your security group should look something like:
Now if you try to telnet to any of these ports, or use WBEMTEST, you’ll probably still get RPC Unavailable, because the Windows Firewall is blocking you.
So now go into your VM and bring up the firewall settings. Make sure all of the Windows Management Instrumentation Services rules are enabled for the Public role (make sure you’re not in Domain or Private or you’re going to make and bunch of changes and nothing will happen and you’ll be confused and angry and you’ll blame me and my stupid blog post and that’s no fun, at least not for me).
Then in Computer Management, drill down to Services And Applications->WMI Control. Right click the WMI Control node and select Properties. On the Security tab, select the Root node and Ensure that the Administrators group has access to everything (it probably will).
Then go into the Services window and make sure the Windows Management Instrumentation service is running.
So let’s try again now.
Final Steps, One Crazy Weirdness
Now if we go back to WBEMTEST and try to connect, we get a little farther.
Access Denied is good! We got through the firewalls and we got a response. So let’s put in an administrator in the user name and password…
WAT. “The object exporter specified was not found”.
What the deuce does that mean? If you look around, you’ll see people having this issue connecting through a host name and the solution is to use the IP address to get it to resolve correctly.
But we ARE using the IP, right? Sort of. We are using the public IP, not private IP that the VM would actually use to identify itself.
In fact, if you go into another Amazon VM and try to connect to this one through its private IP, it actually works. But that is not help to me when I’m sittimg at home in my bunny slippers trying to push a change from my desktop. To the best of my knowledge you cant access Amazon VMs through their private IPs from outside he Amazon cloud (at least not without a lot of networking voodoo that is above my paygrade).
BUT, you can also connect to WMI through a host name. No, not that crazy compute-1.amazonaws.com public DNS host name, that won’t help you any more than the public IP. Instead you have to use the actual machine name that the machine itself is aware of.
Of course, your local machine is not going to resolve that, but if you add a host file entry, you should be all set. In case you’ve forgotten, that’s at C:\Windows\System32\Drivers\etc\hosts, and you’ll need to open it with a text editor that is running in Administrator mode if you have UAC running on your machine.
Now it we go back to WBEMTEST try to connect to “\\[PrivateMachineName]root\cimv2”, it works!
Well, hopefully that helps some folks. Or at least helps me again in 6 months when I run into it again.