A Couple of Quick and Dirty Ways to Bootstrap Bare EC2 Instances

So you want to remotely launch and script a server instance in the cloud, configure it and install your app? Well, unlike Windows Azure, Amazon Web Services don’t launch with an embedded app and they don’t have a tightly embedded way to run startup tasks….or do they?  EC2 instances are little more than vanilla installs of the OS, with perhaps SQL Server and IIS. Bootstrapping a bare machine with a vanilla OS is a very old skill, and I’ve done this a number of ways over the years. It’s gotten easier with every generation of Windows (man those NT4 servers were hard to build remotely) as better tools have become standard on a vanilla install, robocopy and icacls to name but two.

Now unless you want to open up firewall ports and start pushing stuff to your AWS instances psExec style  (hey, I said “dirty”, not completely filthy like those NT4 days) you’ll want the machine to fetch its own config and apps on startup. Ideally, get hold of git, and bring your app down from your git repository. So what are the options on AWS;

1) Create a custom AMI
Do this either with all your setup or some bootstrap code such as cloudinit. Whilst you can do this to a lesser or greater extent there’s still some maintenance involved here. Once you fork the standard Amazon AMIs you’re no longer using an off-the-shelf image, so you have to maintain it yourself. To be honest, this isn’t really a big deal since Windows Update can take care of a lot of this for you, but you’ll want to can the latest versions and this all chews up admin time. At some point, you need to take the hit and rebuild your AMI. It’s a shame that (unlike the Linux AMIs) the standard Windows AMIs don’t (yet) come with something like cloudinit already loaded. If they did, I’d use them in a flash.

2) CloudFormation
This seems to be the premier option where they take care of all this for you, but at extra cost per instance. It’s a negligible cost mind-you, but it’s something else to setup as well. [edit] Turns out I read this wrongly. There’s no additional charge for CloudFormation, but it doesn’t look like there’s any magic either – it will simply orchestrate options (1) or (2) here. So I guess there are just two options.

3) Old-School
The ‘notepad’ way is to basically script your own setup. For that you’ll need to at least be able to run a script on the box and you’ll need to use that to configure the machine and install all your software.

User-Data

AWS gives you just enough rope to do that. It’s called EC2config and it accepts user-data. EC2Config is a service built into the standard AMIs that runs on startup. Whereas user-data, along with a bunch of other information is just a bit of text or a file and you specify it when you start the instance either programmatically or even in the web UI if you’re launching by hand. They then put this verbatim on a private URL for the machine; http://169.254.169.254/latest/user-data/.

That’s great, but there’s no CURL.EXE or WGET.EXE on a standard build of Windows Server 2008 so that’s not a lot of use, however, they do sort-of pass this to you in the standard AMIs and as of June 2012 they’ll run PowerShell scripts if contained within <powershell/> tags or batch (cmd) files if contained within <script/> tags. Well it’s no MSI download, but that’s a good result!

Programs From Text

OK, so I can do almost anything with PowerShell, but what can I do with a quick and dirty script? Well the answer is not a lot really because you can only work with the programs already installed on the box. So you aren’t going to be able to say, go pull an MSI out of S3. Well here are two ways;

1) Map a network drive to a WebDav folder and use it like a conventional network drive. Copy off the files you need and run them.

2) Just like in Unix, since .Net was first shipped we now have compilers on the box, so if we can ship it a script, we can dynamically create an executable.

1 – Map a Network Drive to a WebDav folder

So this has worked since at least WindowsXP as far as I know;

c:\> net use s: http://mywebserver/WebDav /user:username *mypassword*

You can then use robocopy.exe on the s: drive like it were a normal drive. But where to get a WebDav server from? Well obviously IIS can act as a WebDav server, but who wants to maintain another server just for that? The ideal place would be S3 you’d think, but that doesn’t support WebDav. Google Docs/Drive, DropBox? nope! Well SkyDrive does! Microsoft have been supporters of WebDav from the beginning, and I think that’s commendable. You’ll need to find your SkyDrive URL first, but OK, off we go then, right? Wrong, try it and you’ll get this;

System error 67 has occurred.
The network name cannot be found.

It turns out that WebDav support isn’t installed by default on Windows Server, and it requires the DesktopExperience feature (yuck) which has a pre-requisite of InkSupport (double yuck – WTF?) so that’s the first thing our scripts needs to do;

dism /online /enable-feature /featureName:InkSupport /NoRestart
dism /online /enable-feature /featureName:DesktopExperience /NoRestart

Now it’s a good idea that we make our script idempotent, so it can run at each and every startup without penalty in case of crashes or reboots. Unfortunately, EC2Config deletes our user-data script after the first run, but that’s easily defeated with a small hack;

REM Copy the startup file to somewhere we can find it after this copy is deleted
if exist c:\bootstrap goto bootstrap
md     c:\bootstrap
icacls c:\bootstrap /grant "NT AUTHORITY\SYSTEM":(OI)(CI)F
REM Make ourselves persistent
if not %0x==c:\bootstrap\bootstrap.cmd copy/y %0 c:\bootstrap\bootstrap.cmd
schtasks /create /tn bootstrap /RU "NT AUTHORITY\SYSTEM" /tr c:\bootstrap\bootstrap.cmd /sc onstart /f

Now you’d think that you’d be able to simply put yourself in the HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\RunServices key, but that hasn’t worked in a long time, so we use the scheduled task trick to get ourselves to run on each and every startup. Now we should also make sure that our script doesn’t install features unnecessarily (we may be running the script many times after all) and we should have plenty of logging. For good measure, why not copy that log back to SkyDrive? Another good idea is to allow us to change the script for a booting instance if necessary. Once we have a mapped drive we can simply look for a new script on SkyDrive.

Here’s the whole script (unfortunately, WordPress won’t allow me to upload .txt files): bootstrap.docx

2 – Dynamically Create an Executable from user-data Text

I don’t know which of these hacks is the dirtiest, but I’ll leave that to you to decide and comment when you see this next one. Basically, it’s a batch file that compiles into a .Net executable. From C# you can do anything you need, including connecting to S3 and downloading whatever setup files you need.

It’s a nasty habit I picked up in my scripting days. I’ve used a similar technique to write .JS files that will run either in cscript or will full compile themselves into JScript.Net executables. The trick relies on the fact that batch files don’t read the whole script and they’ll skip any line with an error. We only get to pass a single file to our instance in user-data, so we use a C# comment (/*) on the first line, which errors and it complains but then continues to run the next line as a command and so on. We put the C# inline further down and we simply skip over that with a goto statement. The target tag (:end) for the goto statement is similarly hidden from the C# compiled using another comment block.

Essentially, csBootstrap.cmd looks like this;

/* This file MUST be less than 16K for AWS
C:\Windows\Microsoft.NET\Framework64\v4.0.30319\csc.exe /nologo /out:bootstrap.exe /Reference:C:\Windows\Microsoft.NET\Framework64\v4.0.30319\System.Net.dll "%0"
goto end
*/

using System;
using System.IO;
using System.Net;

static class Program
{
    static void Main(string[] argv)
    {
        //Console.WriteLine("Do whatever you like here");
        var wc = new WebClient();
        using (var data = wc.OpenRead(argv[0]))
        {
            using (var reader = new StreamReader(data))
            {
                string s = reader.ReadToEnd();
                Console.WriteLine(s);
            }
        }
    }
}
/*
:end

REM ** Run the program
BootStrap.exe "http://169.254.169.254/latest/meta-data/"
REM */

Note that the script uses the C# command line compiler CSC against itself and that it may also therefore have a cmd extension, but that’s OK to the CSC compiler as long as the file is valid C#. The penultimate line there simply runs the newly created executable with the parameter to fetch the list of meta-data properties available to the EC2 instance, but of course we could do just about anything from this code, including configuring IIS, downloading files or installing git.exe.

Now to make this more robust, we would need some logging and ideally we wouldn’t hard-code the location of the framework. I’ve left the above example as simple as possible, but the following piece of script at the top would make it better;

/* This file MUST be less than 16K for AWS
 
@echo off
@setlocal
REM ** Determine the lastest NetFx installed (note that the delims here are a tab followed by a space)
REM ** Note this is also the best way to determine if a key exists since REG nearly always returns ERRORLEVEL==0
FOR /F "tokens=3* delims=     " %%A in ('reg query "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework" /v InstallRoot') do set FXRT=%%A
FOR /F "tokens=2* delims=     " %%A in ('reg query "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\v2.0.50727" /ve') do if %%Ax==REG_SZx set FX=v2.0.50727
FOR /F "tokens=2* delims=     " %%A in ('reg query "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\v3.0" /ve')       do if %%Ax==REG_SZx set FX=v3.0
FOR /F "tokens=2* delims=     " %%A in ('reg query "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\v3.5" /ve')       do if %%Ax==REG_SZx set FX=v3.0
FOR /F "tokens=2* delims=     " %%A in ('reg query "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\v4.0.30319" /ve') do if %%Ax==REG_SZx set FX=v4.0.30319
FOR /F "tokens=2* delims=     " %%A in ('reg query "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\v4.0.30319\SKUs\.NETFramework,Version=v4.5" /ve') do if %%Ax==REG_SZx set FX=v4.0.30319
:build

then we’d just amend our compiler command like this;

%FXRT%%FX%\csc /nologo /out:bootstrap.exe /Reference:%FXRT%%FX%\System.Net.dll "%0"
Advertisements

One Response to A Couple of Quick and Dirty Ways to Bootstrap Bare EC2 Instances

  1. Pingback: Confluence: Eventboost

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s