RobMensching.com /Blog
when setup isn't just xcopy

SDD before TDD?

David Aiken has an interesting blog post about adopting TDD (Test Driven Development, for those of you have been living in a cave) for his next project.  In said blog post, David suggests that before even writing tests that one should write the setup logic:

First,  I'm adopting a TDD approach. So you would think my first piece of code was a test. WRONG! My first piece of code is the installer!

Installer? - there is nothing to install. Correct. But there is still an installer project, which compiles and builds and installs pretty much nothing. (It actually installs a DLL from the 1st project - the DLL contains an empty class otherwise you cannot build an installer). I've decided to use WIX as the installer technology (the same as we used for DinnerNow)

David's line of thinking goes right along with my philosophy that setup should be integrated directly into the development process.  Maybe we should call this process of writing the setup project up front SDD for Setup Driven Development.  <smile/>

The same way that TDD encourages the development of code that can be better unit tested, SDD would encourage the development of code that can be better deployed.  I argue quite often that many of the problems developers face in setup were caused by poor design or implementation decisions early in the development lifecycle.  Setup Driven Development would flush out deployment complications as soon as they were introduced into the application.  Seeing the deployment issues right away better enables the developer to make a decision to address the issue by writing complex setup logic or by simplify the application's deployment needs.  I would often encourage the latter.

Hmm, maybe this Setup Driven Deployment idea isn't so crazy after all...

 

Explaining the value of packaging in California.

We just did the curly-cue take off from San Jose on my way to Seattle by way of Portland (don't ask).  The captain just informed us that the second pass over San Jose Airport is used to climb above the incoming traffic to SFO.  That makes an amazing amount of sense and provides a nice view of the whole bay area.  I just hope I don't miss my connection in Portland.

I was down in San Jose this weekend talking with the external development team that works on Windows Marketplace with us.  The new build and deployment system I mentioned previously is a pretty big departure from how Marketplace is built and deployed today.  I was having difficultly explaining the value of the new system through email so I just flew down to California to explain it in person.  That worked really well.

Since I was just talking about the Windows Marketplace deployment for the last two days I thought I could share some of the values.  Unfortunately, I'm not currently funded to fly out and talk to each and every one of you.  So you'll just have to do with a blog entry and any follow up conversations that happen here.  <smile/>

(Just a reminder, this will not be comprehensive list of reasons packaging has value.  This will just be the list that I can think of while in the air).

The underlying question I was really trying answer was, "Why create (MSI) packages instead of just some batch files to xcopy deploy?"  The following are my reasons, in no particular order, they are all important.

1.  Fully automated deployment.  This is the key goal, removing humans from the process as much as possible because humans are slow and make mistakes.  Perfect achievement of this goal would mean that humans would only be involved to create the packages (and stuff that goes in the packages, of course) and to provide the parameters that are unique to an environment (such as machine names and service accounts).  That means deployments to a known environment can be done by simply pushing the "Go" button.

Since you can't configure everything on a machine using xcopy alone (can't do anything in SQL for example) you'll either have to have a human intervene for the times when non-file based items change or you'll have to create some batch file to do it.  MSI packages plus the CustomAction library that is part of the WiX toolset can configure pretty much everything in a datacenter without human interaction.

2.  Fully configurable.  To make it easy to move from environment (test cluster, pre-production cluster, production cluster) to environment or to add new environments (new dev clusters, new test clusters, new production clusters around the world) the packages need to be configurable.  For Windows Marketplace, that means passing in Properties that configure XML files (web.config, policyCache.config, random .config files), SQL databases (some settings stored in database tables), IIS settings (e.g. SSL certificates), service accounts (AppPools and other services need to run as different users in different environments) and registry keys (yeah, we even settings stored in registry keys).  It is on my "hit list" to reduce the number of settings over time.

Again, xcopy deployment alone can't configure all the things that change from environment to environment.  A batch file or human interaction in each environment is necessary to tweak the little things that are different.  A MSI package can be passed properties that declaratively adapt the install from environment to environment.

3.  Reduce redundancy.  Once you are moving towards fully configurable you can start removing redundancy in the development process.

For example, when deployments are built without using packages I consistently see a separate web.config for each environment.  The problem is that when you want to add or change a setting, you have to go into all of the .config files and tweak the setting appropriately for each environment.  This process becomes insanely error prone as your system gets larger.  Again, using MSI packages means you can pass properties in during the deployment and configure a single template .config to meet the needs of the environment.

4.  Verify configuration.  As you start to rely more on properties to configure your deployments, you'll also want to verify that the parameters are within appropriate bounds.  This verification reduces debugging time as you bring on new environments or train new operators. 

Of course, xcopy deployments can't do any verification so you'd have to fall back to a batch file.  The Windows Installer Condition syntax is much richer than the checks you can do natively in a batch file.  There have been several times where adding a condition to the deployment for Windows Marketplace saved Zac and I several minutes of debugging because it caught a typo in the deployment properties.

5.  Transacted changes.  Ideally all changes to the datacenter move you from one known (working) state to another known (hopefully, working) state.  Creating transactions using a scripted deployment is difficult if there any branches in the logic.  The Windows Installer is a declarative installation engine and thus provides transactions for each package.  That doesn't help us with multiple package deployments much but if you keep your packages isolated from each other (which is generally pretty easy in datacenter applications) you can get "pretty good" rollback by uninstalling the new and installing the old.

6.  Rebuild everything, anywhere, anytime.  This is a power that you might not truly appreciate until you have experienced it.  The idea is that you can take any set of machines and quickly turn them into productive members of your datacenter application any time.  When you have a set of packages that install everything it is a simple matter of just running all the packages on the new machines.

For example, during our testing Zac and I needed to test a few new tweaks but the test team was currently pounding on the test cluster.  So, I bounced over to my dual-proc, dual-core, 2 GB machine and spun up a couple virtual machines with IIS on some and SQL Server on others and Zac typed in the parameters for this new environment.  In about an hour the virtual machines were acting as a new and completely independent test environment.  Zac practically broke down in giggles (he was responsible for setting up many of the environments previously and that was a grueling task).

7.  Quickly inventory a machine.  The Windows Installer provides a very rich set of APIs to determine what is (and is not) installed on a machine.  Even the ability to pop open the Add/Remove Programs control panel makes it trivial to quickly see, "What is on this machine?"  The ability to inventory the machine at many levels is just one more diagnostic tool.

So, that's a bunch of features you get when approaching datacenter deployment with a "package mentality".  What is the value of all that?  Three things come to mind:

1.  Consistent.  Everything is deployed in a package.  Developers know what they need to do to be deployed in the datacenter (build a new MSI or add to an existing MSI).  Operators know what to expect why they do deployments (deploy a bunch of MSI packages).  All of the processes (build, transfer, communication) get streamlined around a single set of concepts.

2.  Reproducible.  Without any human steps in the configuration of your datacenter, install and uninstall operations will always behave the same way.  Being able to consistently do the same thing over and over allows you to tweak out the bad things and optimize the good things.  It also means that if you get the package install and uninstall operations right in your test cluster chances are good the packages will work correctly in pre-production and, later, production.

3.  Confidence.  Once you have a consistent way of doing things and you can always do it in the same way, your confidence in the datacenter will improve.  Already I'm seeing Zac say, "Well, that's a bug.  We'll just uninstall and install the new fix."  He has confidence in the deployment system and knows that we just have to work out all the bugs introduced by the humans (us).  <smile/>

 

Crisis in Y2K7? WTF?

I've been working on revamping the Windows Marketplace build system for the last month or so.  Just yesterday I had to make the decision how we were going to version our binaries.  Versioning is a very important of software engineering and it can be very disruptive if you get it wrong.  Getting versioning right is also quite often overlooked so I was pleasantly surprised to see two pages dedicated to the topic in the help for our build system.  One page talked about the tools used to format and increment the build version and the other page talked about all the different formats.

One of the versioning formats was called "five digit date".  That meant if you built today that you would get a build version with a number like "61229".  The 6 is for the year 2006, the 12 is for the month of December and the 29 is for today's date.  I looked at that and thought, "Well, that's stupid."

You see, versions in executables are historically stored in two DWORDs.  The DWORDs are split into WORDs and each WORD represents one of the four parts of a version number.  That means that each number in the version cannot exceed 65535.

Using the "five digit date" means that January 1st, 2007 would give you the number "70101".  That obviously won't fit into a WORD.  The "five digit date" format was an incredibly short sighted engineering decision.  Engineering decisions like that drive me crazy.

Amazingly, I glanced over at my email (which has been deathly quiet, so nice) and saw an email from one of the guys that works on the build system.  It made me chuckle:

On January 1^st, the ‘fivedigitdate’ schema should produce a 70101 version  The problem is that value should have a topend in the 65k’s.   I don’t know specifically what issues you will see and where you will run into them, but just a heads up that if you are using fivedigitdate you will likely need to let it run as the ‘default’ setting and start running 61232+.

Then last night I was catching up on my blog reading a little bit and I saw the InstallSite blog talking about a "Y2K7 problem".  I immediately thought to myself, "No way..." then I clicked on the link in the blog and read the original blog entry from Quan To.  This part near the bottom particularly bothered me:

Luckily, we thought of this earlier in the year and changed our build numbers to use XMMDD where X represents the year in development of our product.  As long as we ship in 6 years or less, we'll be ok :)

Why is there 6 year a timebomb built into their build process?  Why are software engineers cutting these corners?  In the long haul, it just doesn't pay off

If you are a software engineer, I encourage you to consider what it means if every line of code you write lasted 20+ years.  Make those lines something you'd be proud to show your kids (and maybe even their kids).