Explaining the value of packaging in California.

I was down in San Jose this weekend talking with the external development team that works on Windows Marketplace with us. The new build and deployment system I mentioned previously is a pretty big departure from how Marketplace is built and deployed today. I was having difficultly explaining the value of the new system through email so I just flew down to California to explain it in person. That worked really well.

Since I was just talking about the Windows Marketplace deployment for the last two days I thought I could share some of the values. Unfortunately, I’m not currently funded to fly out and talk to each and every one of you. So you’ll just have to do with a blog entry and any follow up conversations that happen here. <smile/>

(Just a reminder, this will not be comprehensive list of reasons packaging has value. This will just be the list that I can think of while in the air).

The underlying question I was really trying answer was, “Why create (MSI) packages instead of just some batch files to xcopy deploy?” The following are my reasons, in no particular order, they are all important.

1. Fully automated deployment. This is the key goal, removing humans from the process as much as possible because humans are slow and make mistakes. Perfect achievement of this goal would mean that humans would only be involved to create the packages (and stuff that goes in the packages, of course) and to provide the parameters that are unique to an environment (such as machine names and service accounts). That means deployments to a known environment can be done by simply pushing the “Go” button.

   Since you can’t configure everything on a machine using xcopy alone (can’t do anything in SQL for example) you’ll either have to have a human intervene for the times when non-file based items change or you’ll have to create some batch file to do it. MSI packages plus the CustomAction library that is part of the WiX toolset can configure pretty much everything in a datacenter without human interaction.

2. Fully configurable. To make it easy to move from environment (test cluster, pre-production cluster, production cluster) to environment or to add new environments (new dev clusters, new test clusters, new production clusters around the world) the packages need to be configurable. For Windows Marketplace, that means passing in Properties that configure XML files (web.config, policyCache.config, random .config files), SQL databases (some settings stored in database tables), IIS settings (e.g. SSL certificates), service accounts (AppPools and other services need to run as different users in different environments) and registry keys (yeah, we even settings stored in registry keys). It is on my “hit list” to reduce the number of settings over time.

   Again, xcopy deployment alone can’t configure all the things that change from environment to environment. A batch file or human interaction in each environment is necessary to tweak the little things that are different. A MSI package can be passed properties that declaratively adapt the install from environment to environment.

3. Reduce redundancy. Once you are moving towards fully configurable you can start removing redundancy in the development process.

   For example, when deployments are built without using packages I consistently see a separate web.config for each environment. The problem is that when you want to add or change a setting, you have to go into all of the .config files and tweak the setting appropriately for each environment. This process becomes insanely error prone as your system gets larger. Again, using MSI packages means you can pass properties in during the deployment and configure a single template .config to meet the needs of the environment.

4. Verify configuration. As you start to rely more on properties to configure your deployments, you’ll also want to verify that the parameters are within appropriate bounds. This verification reduces debugging time as you bring on new environments or train new operators.

   Of course, xcopy deployments can’t do any verification so you’d have to fall back to a batch file. The Windows Installer Condition syntax is much richer than the checks you can do natively in a batch file. There have been several times where adding a condition to the deployment for Windows Marketplace saved Zac and I several minutes of debugging because it caught a typo in the deployment properties.

5. Transacted changes. Ideally all changes to the datacenter move you from one known (working) state to another known (hopefully, working) state. Creating transactions using a scripted deployment is difficult if there any branches in the logic. The Windows Installer is a declarative installation engine and thus provides transactions for each package. That doesn’t help us with multiple package deployments much but if you keep your packages isolated from each other (which is generally pretty easy in datacenter applications) you can get “pretty good” rollback by uninstalling the new and installing the old.

6. Rebuild everything, anywhere, anytime. This is a power that you might not truly appreciate until you have experienced it. The idea is that you can take any set of machines and quickly turn them into productive members of your datacenter application any time. When you have a set of packages that install everything it is a simple matter of just running all the packages on the new machines.

   For example, during our testing Zac and I needed to test a few new tweaks but the test team was currently pounding on the test cluster. So, I bounced over to my dual-proc, dual-core, 2 GB machine and spun up a couple virtual machines with IIS on some and SQL Server on others and Zac typed in the parameters for this new environment. In about an hour the virtual machines were acting as a new and completely independent test environment. Zac practically broke down in giggles (he was responsible for setting up many of the environments previously and that was a grueling task).

7. Quickly inventory a machine. The Windows Installer provides a very rich set of APIs to determine what is (and is not) installed on a machine. Even the ability to pop open the Add/Remove Programs control panel makes it trivial to quickly see, “What is on this machine?” The ability to inventory the machine at many levels is just one more diagnostic tool.

So, that’s a bunch of features you get when approaching datacenter deployment with a “package mentality”. What is the value of all that? Three things come to mind:

1. Consistent. Everything is deployed in a package. Developers know what they need to do to be deployed in the datacenter (build a new MSI or add to an existing MSI). Operators know what to expect why they do deployments (deploy a bunch of MSI packages). All of the processes (build, transfer, communication) get streamlined around a single set of concepts.

2. Reproducible. Without any human steps in the configuration of your datacenter, install and uninstall operations will always behave the same way. Being able to consistently do the same thing over and over allows you to tweak out the bad things and optimize the good things. It also means that if you get the package install and uninstall operations right in your test cluster chances are good the packages will work correctly in pre-production and, later, production.

3. Confidence. Once you have a consistent way of doing things and you can always do it in the same way, your confidence in the datacenter will improve. Already I’m seeing Zac say, “Well, that’s a bug. We’ll just uninstall and install the new fix.” He has confidence in the deployment system and knows that we just have to work out all the bugs introduced by the humans (us). <smile/>