RobMensching.com /Blog
when setup isn't just xcopy

2nd cousins

Most of my family is out in the Midwest, but I do have an aunt and uncle that live here in Washington. Their kids, thus my cousins, are about ten years older than I and in very different stages of their lives than me. For example, both cousins have three kids where I have none. Three of those kids, which are 2nd cousins to me, live closer than the others and I've gotten to know Bryce (12), Chloe (8), and Brody (5) pretty well.

Last night, while working late, I got a call from my Mom. She was using the voice that I was sure meant that my remaining grandfather or grandmother had passed away. But they hadn't. Instead, she informed me that Mei Lin had a brain aneurism Monday morning and was in intensive care at a hospital somewhere in Seattle. Mei Lin is the mother of the three 2nd cousins I know so well.

Long story short, I left work around 2:00 PM today and met my aunt, uncle, both cousins, Bryce, Chloe, and Brody and several other friends and family of Mei Lin's at Harborview Hospital on Capitol Hill in Seattle. The hospital was so overbooked that they had Mei Lin on the pediatric floor. Everyone floated around the waiting room and in the hallways of the mostly empty 9th floor waiting for the latest updates from the doctors.

As soon as I got there I gave my cousin Steve--the father of Bryce, Chloe, and Brody--and my uncle a key to my house. I live significantly closer (20 minute drive) to Harborview than they do (60 minute drive) and thought if they were going to spend another long night at the hospital a shorter drive might be appreciated. Then I just kinda' sat there and listened to anyone that wanted to talk to me, which was mostly just my aunt and uncle, and watched the kids play Nintendo. One of the advantages of being placed on a pediatric floor, they have cool things to keep kids occupied.

Eventually Brody noticed me and stuck his head out of the waiting room long enough to say, "Hi, Rob!" That was his cue to me that it was time for me to hangout with them.

Some time late in the afternoon, a family meeting was called to speak with the doctor. Now, I never knew Mei Lin really well. She and Steve divorced a year or two after I moved up here and at all the various family gatherings I mostly hung around the kids anyway. So, rather than being invited into the meeting with the doctor, I was left behind with the kids. This was more than fine with me since I was mostly there for the kids anyway. Did I mention that I really like Bryce, Chloe and Brody? I do, they are great kids.

At this point, time really started to crawl. Once, one of Mei Lin's sisters came down the hall crying. Chloe asked her why she was crying and got a weak smile and an even weaker answer. Later Chloe told me, "I'm tired of waiting" to which I could only reply, "So am I". But I would have rather waited there forever than have to explain to Chloe the possible outcomes of our wait.

Finally, one of the nurses found me and asked me to the meeting room. There Steve asked me if I would take the kids down to the cafeteria for dinner. Of course, I was more than willing. So, with the help of one of Mei Lin's friends and Brody on my back we got all kids--there were four, the three 2nd cousins and a younger one that belonged to one of Mei Lin's sisters--in the elevator going down to the cafeteria.

I have to take a moment and say how impressed I am with any single parents out there. Trying to keep three kids going in a relatively similar direction is an amazingly difficult task. Finding food that they will all eat is even more difficult. I must admit, I was quite happy with myself after successfully getting the kids food and sitting down and eating and having small conversation about baseball and Tae Kwon Do and french fries.

After dinner it was back to waiting. It wasn't more than an hour when the adults all came out and Steve took the three kids in to see Mei Lin. My uncle took me aside and asked if I had been informed at all. Of course, I knew nothing like the rest of the kids. So he remedied that.

Mei Lin had a brain aneurism early Monday morning that apparently broke the part of the brain that told her lungs to keep breathing. The lack of proper breathing for whatever amount of time it took to get her on a system to help her breathe again deprived her brain of oxygen. That deprivation of oxygen had destroyed 95% of her brain. Mei Lin as we knew her wasn't coming back. More importantly, the Mei Lin that Bryce, Chloe, and Brody knew wasn't coming back.

My Mom and I share many similar traits. The one most interesting here is our ability to operate well in unusual and high stress situations for extended periods of time. There have been several cases where we each got into an emergency situation and just "got the job done" without truly internalizing everything that was going on until the crisis was over.

In this case, me getting the kids fed and cared for without transferring the immense sadness felt by all of the adults just down the hallway seemed to be the right thing to do. It also seemed to be much appreciated by everyone later. To me, I was just getting the job done. The job I had assigned myself was to "take care of the kids."

Now, the majority of the day had passed for me without truly internalizing all that had happened. It honestly wasn't until I saw Chloe come out of the room where her mother existed only on life support and sit in one of the chairs in the hallway that I began to internalize the day's events. Mei Lin's father--one of Chloe's grandfathers--tried to give her a hug but she didn't respond well to that. I just stood there watching as the 8 year old girl slowly broke down into tears.

Brody came out next. The resilient 5 year old seemed only slightly disoriented and eventually stopped a few steps away from me and everyone else in the hallway. Without looking at anyone he finally said, "Rob." I bit back the tears, looked at him and touched his head. "Brody?" He eventually looked at me and said, "Lift me up."

Bryce came out later looking very shell shocked and quieter than usual.

What do you say to an 8 year old that is just now starting to imagine what life is going to be like without her mother? What do you say to a 5 year old that doesn't seem to understand? What do you say to a 12 year old that is just standing silently by the wall, fiddling with one of the new toys he got in the last three days? What do you do when the 5 year old wants to climb up on your shoulders like he always does?

I don't have answers for the first three questions, but the answer to the last question is you lift him up. You always lift him up.

Component Rules 101

I've been debating with myself for the last week whether I really wanted to write this blog entry about the Component Rules. The Windows Installer SDK has some decent documentation about what can go wrong when the rules are broken. But right when I had just about decided to write about something more pleasant, on the walk back from lunch yesterday, Robert suggested that I write this entry anyway to put a more emphasis on this topic that has easily sucked up months of my life. Then today there was yet another email thread at work that demonstrated people still don't understand the Component Rules. So, on the drive home tonight when I heard that C89.5 was going to play the entirety of Basement Jaxx's new album Kish Kash, I decided I would sit down and write the blog entry about the Component Rules. So here we are and here we go.

In a previous blog entry I talked about the basics of Windows Installer Components. If you haven't read that entry you should do so before continuing on here. Done? Great, let's continue.

At the end of that previous blog entry, I hinted at a flaw in the way the Windows Installer reference counted Components and the Resources contained in them. Rather than give the answer right away, I'll start building up an example then walk through the problem step by step and we'll see how quickly you can jump to the conclusion.

Note, I'm going to pretend to use a bit of set notation here but I can't find my discrete mathematics book right now so I apologize now if I butcher the syntax.

First, remember that every Component in the Windows Installer has a Globally Unique Identifier (GUID) that I called the ComponentId. Also remember that the Windows Installer tracks (reference counts) Components only by their ComponentIds. I like examples so let's say we have three Components with their ComponentIds like:

 C1 - ComponentId = "{0F794AC3-F266-4238-BC59-4AD641DD5F8F}"
 C2 - ComponentId = "{5B2390D5-C8D0-43b0-980F-01A872E809C4}"
 C3 - ComponentId = "{0F794AC3-F266-4238-BC59-4AD641DD5F8F}" 

Now to us, it looks like there are three Components listed. However, the Windows Installer would only see two Components because C1 and C3 have the same ComponentId (trust me, I cut'n'pasted them). Okay, so now, let's say I also have three Resources:

 R1 is a Registry Key = "HKLM\Software\Microsoft\Wix\Data=example.wxs"
 R2 is a Registry Key = "HKLM\Software\Microsoft\Wix\Version=2.0.0.0"
 R3 is a File = "[ProgramFilesFolder]\Wix\candle.exe"

I probably only need two Resources for the example, but electrons are cheap. Anyway, the actual types of Resources listed above don't matter. I just wanted to make the example a little more concrete since I know I don't like it when people start hand waving at abstract concepts.

Now, that we have some Components and Resources, let's run through some simple logical statements. For the rest of this entry, the symbol "==" means "is equivalent to" and the symbol "!=" means "is not equivalent to". Okay? Okay.

 C1 == C1
 C1 != C2
 C1 == C3
 C3 == C1
 R1 == R1
 R1 != R2
 R1 != R3
 C1 != R1 (and some people would argue this is a nonsensical thing to do anyway
 since Resources aren't Components to begin with and discussing their
 equivalency is pointless!) 

All I'm trying to show is that the Components with the same ComponentId are identical and the Resources provided are all unique. Now that we have some building blocks for a setup, let's put them together for our first scenario

 C1 = { R1 }
 C2 = { R2, R3 }
 C3 = { R1 } 

What I've tried to represent in set notation is that Component 1 contains Resource 1, Component 2 contains Resources 2 and 3, and Component 3 contains Resource 1. Nothing magical here. Again, if we do the equivalency tests on the contents Components the following should all still be true:

 C1 == C1
 C1 != C2
 C1 == C3
 C3 == C1 

But enough truth statements, I'm here to talk about installation. So let's pretend that we had the Windows Installer install each of those three Components on a "clean machine" (a machine that has never had these Components installed before). We're talking about reference counts a lot in this blog entry, so let's take a look at the reference counts on the ComponentIds and Resources after the installation is complete:

[ten after midnight and the DJ has started playing Basement Jaxx, sweet...]

 {0F794AC3-F266-4238-BC59-4AD641DD5F8F} - 2 (C1 and C3)
 {5B2390D5-C8D0-43b0-980F-01A872E809C4} - 1 (C2)
 R1 - 2
 R2 - 1
 R3 - 1 

What does this mean? Let me try to represent the lines above in English. "The ComponentId {0F794AC3-F266-4238-BC59-4AD641DD5F8F} has been installed on the machine twice, once by Component 1 and again by Component 3. The Component 2 with its GUID has been installed on the machine once. The Windows Installer only reference counts Components but if we were to apply the reference counts from the Components to their contained Resources we'd notice that Resource 1 has a reference count of two while all the other Resources have a single reference count."

Now let's say we were to uninstall Component 1 and Component 2. The resulting reference counts would look like this:

 {0F794AC3-F266-4238-BC59-4AD641DD5F8F} - 1 (C3)
 R1 - 1 

Notice that Component 2 and its Resources are gone. That's because their reference count reached zero and were thus removed from the machine. Component 1 was removed so the count on the ComponentId it shared with Component 3 was reduced by one but did not hit zero so the Component is still there. If we remove Component 3 now then all of our Resources would be removed from the machine.

So far so good, so let's move on and introduce a new scenario with just Components 1 and 2 like thus:

 C1 = { R1 }
 C2 = { R1 } 

Now, Resource 1 is installed by two different Components.So if we go back to our clean machine and install each of these Components the resulting reference count on the Components and Resources would look like this:

 {0F794AC3-F266-4238-BC59-4AD641DD5F8F} - 1 (C1)
 {5B2390D5-C8D0-43b0-980F-01A872E809C4} - 1 (C2)
 R1 - 1 

See the problem? No, that's not a typo. Let's try it in English, "The ComponentId {0F794AC3-F266-4238-BC59-4AD641DD5F8F} has been installed once by Component 1 and the ComponentId {5B2390D5-C8D0-43b0-980F-01A872E809C4} has been installed once by Component 2. The Windows Installer only reference counts Components but if we were to apply the reference counts from the Components to their contained Resources we'd notice that Resource 1 gets a count of one from two different Components. Unfortunately, that isn't additive so the Resource 1 ends up with a reference count of one."

So what happens when you uninstall Component 1 (or Component 2, it doesn't matter)? Yep, you guessed it:

 {5B2390D5-C8D0-43b0-980F-01A872E809C4} - 1 (C2) 

Resource 1 has been removed prematurely from the machine. Yes, this is wrong. Yes, this is the way the Windows Installer really works. Yes, there are variations on the scenario that result in Resources being "orphaned" (left behind) after uninstall and Resources not being installed on the machine when a Component tries to install them.

The thing I find interesting is that if you take the first logical statement about the Windows Installer's view of equivalency between Component 1 and Component 2 and compare that to the mathematical equivalency of the sets that represent Component 1 and Component 2 in the last scenario you get this:

 C1 != C2 where "{0F794AC3-F266-4238-BC59-4AD641DD5F8F}" != "{5B2390D5-C8D0-43b0-980F-01A872E809C4}"
 C1 == C2 where { R1 } == { R1 } 

While not quite as profound as Russell's Paradox, if you're using the Windows Installer this design flaw is something you should definitely be aware of when putting Resources in Components. Hopefully, it also demonstrates that you should never put someone else's Resources in one of your Components.

C89.5 has finished playing the new Basement Jaxx CD and I have a flag football game in 8 hours (we have the early game this weekend) so I'm going to wrap this blog entry up here. If people want, I can drill into more detail about the Component Rules in future blog entries. However, I think we have enough information here to understand the problem and provide justification for Merge Modules thus unlocking new branches in my thought tree.

Until next time.

 

The story of Orca

Like security guy Greg Fee, I find the list of referrers to my blog fascinating. Today the referrers list contained several hits from people searching for Orca (or more explicitly orca.exe) on Google. I can only figure that Google snagged one of my earliest blog entries with its reference to "orca.exe" and was sending people there. Since, I've been in something of a storytelling mood the last couple days I thought I'd go back to 1998 and tell the story of Orca.

In 1998, I was an intern on the Darwin team that was a small part of the Office 9 organization. "Darwin" was the codename for the new installation technology being built to replace the ACME installer. When I showed up on the Microsoft campus, Office 9 was just about to start on the long glide path towards shipping a product.

Early in my internship, I was working on some small tools to make it easier for people to build "MSI files" that were the inputs into Darwin. For example, msifiler.exe was the first tool I wrote. After getting my feet wet with a couple small tools, I was put on a task that would cumulate in the Merge Module specification and tools. However, I never really shook the "tools bug" and kept thinking about tools that would make it easier to deal with Darwin. I liked to call myself the "unofficial Darwin tools team". I even had an internal web site called "Darwin Unleashed!" with all kinds of tips and tricks for building MSI files.

At exactly halfway through my eight month internship, my girlfriend at the time decided she didn't want to continue to be an intern Program Manager (PM) with the Publisher team but wanted to go back to fighting forest fires in Zion National Park as she had the summer before. The day after she left I was sitting in Rob Collie (who was the Darwin PM at the time)'s office listening to Jim Masson (who was a part of the Zero Administration Windows initiative in Microsoft and later would do a stint as the Darwin PM) talk about his "Darwin wish list". At the end of his list, he leaned back in his chair and said, "You know what would be one really cool app? One really cool app for Darwin would be a table editor that we could ship in the SDK."

I had been suggesting to my mentor for a few weeks that I should write a table editor for creating and editing MSI files. At the time the only thing you could do was use msidb.exe to import and export IDT files. Have you ever seen an IDT file? They look a lot like this:

Component <tab> ComponentId <tab> Directory_ <tab> Attributes <tab> Condition <tab> KeyPaths72 <tab> S38 <tab> s72 <tab> i2 <tab> S255 <tab> S72Component <tab> ComponentMyComponent <tab> {12345678-1234-1234-1234-123456789012} <tab> MyDir <tab> 4 <tab> <tab> foo

To edit your MSI file you had to export a table to an IDT file like above, modify the data that needed changing then import the IDT file back into the MSI. This was a painful process because IDT files have to be formatted just right (for example, I explicitly called out the tabs separating each column identifier) and keeping foreign key relationships up to date often required exporting another IDT file and fixing it. Finally, msidb.exe had one of the most confusing command-line interfaces and one of the funkiest GUI's I've ever had the displeasure of dealing with.

Msidb.exe was so bad that the Office setup team had built a tool called "Scopes". Scopes had the ability to directly view and edit the tables in the MSI so you didn't have to dump them all to text. Unfortunately, it had all kinds of features for interacting with the very complicated build process for Office 9 and wasn't terribly stable. Scopes wasn't a tool that Microsoft would be able to ship customers in an SDK.

So, when my social life up and left for Utah, I decided to hone my MFC skills and try to bang out a real live table editor in my new found free time. Three mostly sleepless nights later I had a basic table editor working. I showed it to my mentor and he said, "Can it do Unicode?" After weeks of trying to convince him that we needed this tool, the first thing he ask for is another feature. My answer was, "Sure, how hard could it be?"

Orca turned into one of his favourite (ahh, the dreaded "u" again) demos to show to people interested in Darwin. He would load an MSI file with a table that contained four rows with the same sentence each displayed in a different language. Yeah, John, it can do Unicode. <smile/>

Now after you get to know me better, you'll find that I have this fascination with the stories behind the naming of people and things. For example, when I was older I found out that each of my parents and my grandfather had a different story why my name is "Rob" (just "Rob"). So it is very important for all of my projects to have codenames that fit them well. Orca's name is one of my favourites (still with the "u", grr).

To understand the name, you have to go back to the beginning with the conversation in Rob Collie's office. Two or three times in that conversation Jim said, "One really cool app." His little diatribe got stuck in my head and on the third night of sleep deprived coding the name for the table editor came to me. Orca. One really cool app. How perfect was that? So, I started up mspaint.exe and hacked out a little pixel art that turned into the killer whale leaping out of the water that you'll still find today as the logo for orca.exe (click Help -> About for a really good picture).

Later it dawned on me that the icon for the Scopes table editor was a little penguin. Do you know what one of the few natural predators of penguins are? Yeah, killer whales. The Scopes tool was mothballed when Office 9 shipped (marketing named it Office 2000). Scopes was then replaced by a new setup processes in Office 10.

So there's the story of "one really cool app" that came to be known as Orca. I hope you enjoyed a bit of the history behind the tools. Next blog entry, I think I'll go back and continue that discussion about Components where I left off. Until next time.

 

Windows Installer Components Introduction.

It's late but C89.5's Electrobox is still kicking so I thought I'd sit down and write about one of the topics at the root of my blog "thought tree". This thought tree is a text file I've been editing with notepad that captures random questions that I see go by (usually through email at work) whose answers deserve to be written down. The interesting thing is that many answers to seemingly innocuous questions require a bit of depth of understanding. For example, Siew Moi (pronounced "Sue May" in American and Australian English for those of you who haven't been formally introduced) asked for more detail about Merge Modules. I thought, sure Merge Modules are a great thing to talk about (and I happen to know a little bit about them) but the first thing I'd want to explain is why Merge Modules were created. To understand why Merge Modules were created you need to know about the Component Rules. To understand the Component Rules, I probably need to explain what Components are with respect to the Windows Installer. Fortunately, Components are one of the "root thoughts" of my tree so let's go dig into those and we'll take another step down the path to more advanced topics.

Not too long ago, I talked about resources and how they were the building blocks of a program. At that time, I also indicated that most programs of consequence are made up of a significant number of resources. For example, Microsoft Office Professional 2003 is composed of over 20,000 resources. Granted Office is a suite of several programs that are rather intricate themselves but even smaller line of business applications can have 50 or a 100 resources.

To help organize all those resources most installation technologies provide some sort of abstraction to group resources together. The Windows Installer calls its abstraction a Component, as if the word "component" wasn't overloaded enough in the world of computing. Thus it is important to note that these Windows Installer Components have nothing to do with the Component Object Model (COM) or any other definition of component you may have heard somewhere else. I wish I could say that the System Definition Model (the stuff Microsoft pays me to work on most days) didn't add another definition for component into the mix, but we do. Anyway, my point is that a Windows Installer Component, which I will refer to as just a Component from here out, is an entity all to itself with its own semantics independent from everything else that might share the name "component". If we're clear on that, I'll continue defining the Windows Installer Component.

In the Windows installer, Components are not only a way of collecting a set of resources to help with the organization; Components are the "atomic units of setup" in the Windows Installer. Let me come back to that very key statement after providing a few more facts. Components are identified by a globally unique identifier or GUID (pronounced "goo id" if you're from Office or "gwid" [sounds a lot like squid] if you're from VisualStudio). You'll find I often refer to the GUID of a Component as the ComponentId. Now since the ComponentId is globally unique it is possible to walk up to any machine and ask it, "Hey, do you have Component X installed?" This ability is key for more advanced tricks we can play with the Windows Installer that I'll cover in a future blog entry. See, I told you this Component stuff was the "root" of a lot of other ideas. <grin/>

Another important facet of Components is that they are tied to a single directory (a resource). This means that if any files (also resources) are contained in a Component they must all install to the same directory. Not all install engines have this restriction but requiring a Component to install all of its files to a single directory does simplify installation a little bit. This is especially true if you consider how the Windows Installer handles Components' KeyPaths.

The KeyPath for a Component is a single resource that the Windows Installer uses to determine if a Component "exists" on a machine. The KeyPath may be the directory of the Component, or a file contained in the Component, or a registry key contained in the Component, or an ODBC data source contained in the Component. Note that there can only be one KeyPath for a Component. If your Component has two files and three registry keys you have to pick one of those five resources or even the directory of the Component as the KeyPath. There are a lot of other implications to consider when picking the KeyPath of a Component but let me get back to my statement about Components being the "atomic units of setup".

When the Windows Installer decides it is going to install a Component it installs all of the resources in the Component together. There is one exception to that statement. When installing files that have versions, if a file already exists on the machine and that existing file has a higher version than the file in our Component the Windows Installer will skip over our Component's file. The Installer does this because one of the rules of setup is that files with higher versions should provide all of the functionality of files with lower versions and usually more. Installing a lower versioned file over the top of a higher versioned file very likely will decrease the functionality of the file and break any programs depending on that newer functionality. Of course, you can override that rule (the second axiom of setup at work) and force all the files in a Component to get installed even if some of the files exist with newer versions (note, this downgrading of files is dangerous and definitely should be done only as a last resort to get your program running again).

So how does the Windows Installer decide it's going to install a Component? Well, by checking for the existence of the KeyPath, of course. By default, the Windows Installer takes the resource that you marked to be the KeyPath for the Component and checks to see if that resource already exists on the machine. If the resource already exists then the Installer decides the Component does not need to be installed and moves on to the next Component's KeyPath. Again, with versioned files the file is determined to exist if a file of equal or higher version is already on the machine.

But what if you're Component has a bunch of resources in it and only the KeyPath resource already exists on the machine. In that case, none of your resources will be installed because the Installer has decided your Component doesn't need to be installed. This further explains why I said that Components are the "atomic units of setup". It is very much the case of all or nothing with Components. Therefore it is very important to pick the KeyPaths for your Components carefully.

Another option is to put only one resource in each Component. This approach will (for the most part) ensure every resource is appropriately tracked. However, Components are also what I like to call the "primary moving parts" of the install engine. More Components means more code to churn which translates into slower processing. There are also cases that I'll cover in the future where you cannot split the resources across Components. Ultimately, there is a balance to strike for your setup.

I've talked a lot about install but you are probably wondering about uninstall. Well, uninstall is where it really becomes clear that Components are the "atomic units of setup" in the Windows Installer. Remember the ComponentId? Well, the Windows Installer basically counts the number of times it has installed that ComponentId. In other words, every time a program is installed all of the Components that make up the program's setup have the count on their ComponentIds increased by one. This process is called "reference counting". Of course, when a program is uninstalled all of the Components have their ComponentIds decreased by one. When the count on a ComponentId reaches zero the Component must no longer be needed and the Windows Installer deletes all of the resources in the Component from the machine.

Now if you are really sharp you'll see a flaw in here. I'm going to save that discussion for later but suffice it to say this flaw provided one of the primary driving forces for Merge Modules.

There is more to say about Components but I think that's enough for one blog entry. Electrobox ended over two hours ago and in another couple hours the sun will start coming up. Until next time, keep coding, you know I will. <smile/>