How the .NET Foundation kerfuffle became a brouhaha

This blog post exists as an independent statement of facts and how I interpreted them at the time. From here decisions can be made.

(Update: I presented this story at today’s WiX Toolset’s community meeting, you can hear me tell the story here)

Where should we start?

First, I want you to understand the importance of the WiX Toolset to me. I started this project 22 years ago, in 1999. It is probably the most impactful thing I did at Microsoft even though it was never my job. My best friends either went to school with me or worked on the WiX Toolset with me. I founded a company to take the WiX Toolset to the next level. Through the WiX Toolset, I support my family and pay my employees to support theirs.

Even if it is a relatively insignificant geekware tool to you, the WiX Toolset is not just some OSS project to me.

What happened?

I don’t exactly know where to start. It’s still blowing up on Twitter so I think we’re still in the middle of the story. But since my blog isn’t Twitter, let me paint a picture so you can see how we ended up here. This is going to get long.

Let’s go back to when the WiX Toolset joined the .NET Foundation. I wrote the story here back in 2016. Please, go read it. I’ll wait.

In that story, I’m not sure if I made it clear how important Martin Woodward was in deciding to join the .NET Foundation. He explained the requirements, listened to my concerns and worked to get solutions to any problems. I explicitly remember Martin letting us choose to integrate the CLA bot ourselves or grant the .NET Foundation temporary access to set it up for us. In fact, I found the email from Martin:

CLA Team,

I’m happy to say that the WiX project will be coming into the .NET Foundation tomorrow (May 4th). Would you be able to work with them to get the appropriate repos CLA enabled asap along with giving them the access to maintain going forward?

Rob – can you send a list of the repos that you want the CLA enabled for? Can you also send the following information for yourself and Bob as org admins so that the team can set you up…

Remember this part of the story, it’s important later.

What happened to Martin?

In 2017, he decided to do something new. I’m not exactly sure what. I should do a better job keeping in touch. Hey, Martin, if you read this. We should do lunch sometime. [Ed note: Martin reached out to me, he works for GitHub out of Ireland now. It was great to chat with him again.]

So what happened with the .NET Foundation next?

Honestly. A whole lot of nothing. This wasn’t really a problem, except when there was a problem. It took a long time for anything to get done and there wasn’t any real communication back to us. In the end, I learned to plan .NET Foundation requests 3 to 6 months in advance (when possible).

But this was fine. Sure I had dreams of more “synergistic extravaganzas” when we first signed up to join the foundation. But in the end, the .NET Foundation held the copyright for the WiX Toolset and we used their signing service. If I had to do it again, I would have FireGiant handle the signing and its costs (like it does for all the other WiX Toolset costs). Less interaction with the .NET Foundation meant less downtime when things broke on their side. But things didn’t break often so it was fine.

And it was fine for a couple years.

What happened in 2019?

Near the end of 2019, the .NET Foundation presented the Maturity Model. After a week of complaints by the project maintainers in the .NET Foundation, the concept was scrapped. There may have been really good ideas in there but (IMHO) the Maturity Model was dead on arrival for one simple reason:

The burden of execution fell on the actual project maintainers and project maintainers were not included in the conversation.

Look at it from a project maintainers point of view. You—as a maintainer—have been ignored and not feeling particularly well supported by the .NET Foundation. The first major proposal you then hear from them after three years creates a lot more work for you and the project you maintain. That is not a way to win over the hearts and minds of developers the doing work. And this was when I first started hearing grumbling that maybe, just maybe, the .NET Foundation wasn’t here to help projects be successful.

But the Maturity Model faded away. Board members came and board members went. Nothing really changed.

Which was okay, I guess. I didn’t really understand what the .NET Foundation was for any more. I didn’t know anybody there and they didn’t know me. And that was okay because the foundation didn’t really impact me.

And it stayed okay until August 3rd, 2021.

What happened two months ago?

August 3rd starts with an email from the .NET Foundation:

Yesterday we announced Foundation-wide Code of Conduct Enforcement. Part of making that work requires that the dnfadmin GitHub user has owner permissions to GitHub organizations. GitHub org ownership by the dnfadmin account has been required for all Foundation projects for some time, but we noticed that the account does not have the proper permissions in wixtoolset. Please assign the org ownership role to the dnfadmin user as soon as possible.

My first thought was, “No.” My second thought was, “This is not how you approach someone you want to help you.” My third thought was, “We have a huge project to finish, I’ll deal with this later.”

Then on August 30th the follow-up:

Quick reminder on this. Can you please fix this ASAP as the project is currently out of compliance with .NET Foundation policies.

The big project deadline was at the end of September and we were working very hard on that. So my first thought was, “No.” My second thought was, “What .NET Foundation policies?” My third thought was, “We have a lot to get done, I’ll deal with this in a couple weeks.”

Just so we are on the same page, the .NET Foundation doesn’t pay me (never has). For the last 5 years, the .NET Foundation has essentially ignored me (I didn’t get involved in the Maturity Model dust up). Now the .NET Foundation wants me to comply with some new policies that I’ve never heard of, let alone agreed to. I don’t know anybody there and they don’t seem to know me.

I responded on September 14th, when our project deadline was safely in sight:

Thanks for reaching out. The bulk of our day to day conversations happen on a couple mailing list ([email protected] and [email protected]) that are (fortunately) very civil and quite installation-geeky. But we could/would unsubscribe anyone that violated the Code of Conduct there.

Also, the WiX Toolset’s repositories are too important to us to grant ownership access to an account that isn’t a core maintainer of the codebase. So, thank you for the offer to automate the Code of Conduct enforcement but we’ll pass if it requires significant access to repo permissions.

In other words, Code of Conduct? Good (we’ll totally do that). Admin access for people we don’t know? Bad (we’re not doing that). I think we were pretty clear on that.

Now remember Martin Woodward from the very beginning of our story? He understood this. That’s why he gave us the option to manually install the CLA bot or temporarily grant them access to do it for us.

So what’s the problem now?

It goes back to the CLA bot. See the CLA bot was not working in one of the WiX Toolset’s repos (I think it was a new repo). So one of our maintainers proactively reached out to the .NET Foundation to inquire how to fix it and was told to grant admin access to the .NET Foundation.

I was really bothered by the response because the CLA bot still worked in our other repos (and had for years!). We discussed the increasing trust issues we had with the .NET Foundation but made no decisions. Then we got a PR that needed CLA sign off by the nonfunctional CLA bot. And I was busy on the end of September deadline.

So, on September 21st, I made the .NET Foundation an admin of the wixtoolset organization to get the bot fixed. That was my mistake.

Why was adding @dnfadmin a mistake?

I’ll get back to the @dnfadmin account in a minute but first let me talk about a couple other mistakes (not mine). By the end of September the CLA bot was working again (yay). There was no communication from .NET Foundation that they fixed it (but we’re used to no communication by now). At least it didn’t take 3-6 months this time (that was a surprise).

Then over the weekend awkward and poor decisions by the .NET Foundation came to light.

• Awkward - As the .NET Foundation announced a new Board of Directors, they made an specifically vague comment about one of the exiting board members such that the exiting board member felt it necessary to respond.
• Poor decision - the Executive Directory of the .NET Foundation forced a pull request into a project she hadn’t contributed to in years. This was recognized as a mistake and eventually a public apology was made but we’ll get back to the apology in a bit.

Not to diminish the impact on the individuals directly involved but neither of those are issues outside those individuals. Mistakes are made, apologies are made (and the PR can be reverted then applied properly later). I did not see those missteps as necessarily systemic issues.

But in a blog post, the maintainer of the project involved with the pull request issue, exposed that their ReactiveUI project was moved into the .NET Foundation’s GitHub Enterprise.

Wait, what is ”.NET Foundation’s GitHub Enterprise”?

.NET Foundation’s GitHub Enterprise was new to me. I thought that GitHub Enterprise was how companies could bring the GitHub experience behind their firewall on standalone servers. I think the standalone server is “GitHub Enterprise Server”. GitHub Enterprise is a billing plan that gives the payee the ability to control the contained GitHub organizations and repositories.

The .NET Foundation pays for GitHub Enterprise. The enormous dotnet and aspnet project are in there with some others, like the Project Kudu. GitHub Enterprise allows them to centralizes billing for things like GitHub Actions. Finally, it allows them to control the contained organizations and projects.

That last part is what tripped up the ReactiveUI project.

If you are not part of the GitHub Enterprise, you can’t tell if an org is part of “public” GitHub or a GitHub Enterprise. The UI change is subtle and at the organization level, probably not a place you look regularly. Here’s what it looks like the ReactiveUI org where I’m logged in and not.

I use GitHub in dark mode, so it should be clear which one I’m logged in:

Uh, Rob, how is it that you can see what is in ”.NET Foundation’s GitHub Enterprise”?

And this is where this story takes a turn for the worse.

After reading about the ReactiveUI being mysteriously moved into the GitHub Enterprise, I was shocked. I couldn’t believe that the .NET Foundation would silently move the project into their GitHub Enterprise. They would be usurping the project maintainers trust and the trust the maintainers have built with users by managing the project. There is no way they would do that without first discussing it with the project maintainers. The only way that could happen is if .NET Foundation was an admin access on the organization.

Not to put too dramatic a point on it, this is where my blood ran cold.

The .NET Foundation had admin access to the WiX Toolset organization for a week, not more than a week ago. There is no way they would have moved the WiX Toolset to the GitHub Enterprise in that window. They would never do that after only two weeks ago, I told them I didn’t want to give them admin access because I didn’t trust them. They wouldn’t betray what little trust I had left after I explicitly told them I was concerned. Would they?

They did. The WiX Toolset was moved into .NET Foundation’s GitHub Enterprise without any discussion or warning.

Aaron Stannard put it best:

Effectively this is an OSS supply chain attack - the people you’ve put you’re trust in (the maintainers) have silently lost control over the project and are only just discovering it

What did you do when you found out?

This is where time slowed down.

I stared at my computer monitor silently seething. I think that’s when I tweeted.

I feel sick to my stomach right now.

In the last two weeks, someone at @dotnetfdn moved @wixtoolset into the .NET Foundation’s private GitHub Enterprise.

They did so after I explicitly told them I did not trust them enough to make them an admin in our project.

I feel betrayed.

I sat there searching for answers. Did I have to fork? I could fork but what do I call it? How would I migrate all the issues? I migrated the issues to GitHub from where we were before GitHub, I guess could do it again.

Then inspiration hit and I did a quick experiment on GitHub to prove it should work. I called Bob and asked him if this was crazy. We talked to through the various potential ramifications before I made the call to proceed.

Bob watched via screen share as I took the WiX Toolset back to “public” GitHub. When I was done, I thanked Bob for riding shotgun, and went to pick up my son from soccer.

So, how did you leave the .NET Foundation’s GitHub Enterprise?

First, I don’t know if this loophole can be closed via settings in GitHub Enterprise. If it can be closed, by the time you are reading these instructions they may no longer work. Additionally, I’m going to assume you’re still an owner of your organization. If you were removed as an owner then all bets are off.

Finally, the timespan between steps 2 and 3 needs to be as small as possible. So read through the steps fully, possibly practice by creating some temporary organizations then go fast with precision:

1. Create a new GitHub organization, normally. For example, name it new-yourorgname
2. Rename your organization in GitHub Enterprise in the Settings to something like, dnf-yourorgname
3. Rename the organization from step 1 to your desired organization name. You want to complete this quickly after step 2 so no one takes your organization’s name.
4. In the GitHub Enterprise dnf-yourorgname go into each repository’s Settings and transfer the repo to the brand new yourorgname organization.

All of these operations are in the red “Danger Zone” at the bottom of the first page of Settings for the organization and repository. There are multiple prompts for safety’s sake that will be a real drag if you have more than a few repos to move. The WiX Toolset has 44.

You will also get a lot of email. Which now that I think about it… why didn’t the move into GitHub Enterprise generate any email?

Who did this?

I don’t know. I’m far more interested in why this was done this way.

What happens next?

I don’t know. There was an apology. But it really only addresses the poor decision to force through the pull request. It otherwise restates that the projects were moved to GitHub Enterprise without warning.

One could even argue the apology goes as far as blaming us, the project maintainers, for not understanding the legal agreement we signed with the .NET Foundation. Well, I never signed an agreement with the .NET Foundation The WiX Toolset copyright was transferred directly from Outercurve, probably using the same lawyers the .NET Foundation used. And that was done when the WiX Toolset was on CodePlex. They can have that project.

Clearly, I’m still a little angry. I don’t know what happens next really.

All I do know is that the WiX Toolset back is in “public” GitHub, just like it was before. And the .NET Foundation knows how to contact me. I’ll be over here writing code.

One last thing

I don’t know how to articulate how much it pains me to be in this situation. Almost 20 years ago, I worked very hard as part of Microsoft to demonstrate how to be a good OSS citizen. Now the project I used to lead that progress is involved in a situation where actions of the .NET Foundation are tarnishing that which I worked so hard to help create at Microsoft.

This isn’t irony. It’s painful.