Disclaimer: Every single number I state in this post is pure guesswork.
Every time I have to read a “the insiders want more builds” thread on Twitter I just have to shake my head and wonder why Microsoft keeps doing the Windows Insider program the way they are doing it now, instead of the far better version they have on the Xbox. But when you really think about it, the Windows system is just better for them, not for users though.
The main complaint with the Windows Insider system is always that there are too few, too many, too buggy, too not-buggy releases and everyone wants to get pleased but the crowd can’t agree on what they really want. Obviously. People are different.
The problem comes from the point that ~80% of the Windows Insider system members is made up of non-technical people. There, I said it. It does not sound nice but my ball park number is probably super off but you get what I am trying to say. Many people just signed up to have the latest and greatest, get updates faster or just prance around saying how they are part of a team of special people that gets to test software, sometimes even writing “QA Tester for Microsoft Corporation” in their CVs.
To be clear, I am not hating on them. I did the same when I was 15/16 and got into the first private beta programs and I was hella proud of it. But what this does is diluting the focus a release branch of the Insider program has. I know like 2(?) people that are on slow, the rest is all not on the program at all or on fast. So, what happens when non-technical people are on fast and get more and more buggier builds? Are they happy about that? Do they know how to file proper feedback? Will they stay on that build for longer? I think the answer to all of that is “no”.
The Insider program is largely a huuuge generator of telemetry for Microsoft. You don’t have to be a technical person to generate telemetry though. You just happen to be useful while you suffer through broken shit. Is that fun? Does that give you some sort of accomplishment? Do you think you are helping?
The Xbox and Windows Insider ring systems
Xbox
- Omega
- Delta – ~3 updates a week
- Beta – between 8 and 15 a month
- Alpha – around 15 to 20 a month
Windows
(needs stats on release cadence)
- Release Preview aka “dead weight that should be used to test Cumulative Updates every month instead”
- Slow aka “let’s just get bit more telemetry to make sure all our fixes worked in Fast”
- Fast aka “The Thing”
- Fast + A/B
- Fast + Skip Ahead aka “The worst idea ever that just feels like a bad hack and should be removed”
- Fast + Skip Ahead + A/B
Skip Ahead
So, when Microsoft reaches the end of a development milestone for their bi-annual releases, there is a point where they switch branches and use Fast to stabilize the final version that is to go out to everyone in the world. At some point the pressure from Insiders got too high, claiming that they feel bored or left out of the development progress because Microsoft clearly is already working on the next version of Windows but they can’t test it yet. In comes Skip Ahead. What should have been a new release ring above Fast or a reason to rebalance the current rings’ purposes, gets implemented as a tacked on extra dropdown that works only twice a year, has a limited number (not shown in UI) of people that can sign up for it and disables itself after it is no longer used.
I get that it was probably the fastest way to implement this, but it just feels plain bad. In my opinion, Fast should just have been the branch that is always on the latest dev branch. If Microsoft switches, so do users. If you want to be on the safe side, switch to Slow in those times. But this would have required a change in their flighting strategy, to move everything down one ring and try to make users understand. Which they would not have. Know why? Because most people just sign up for the program and then forget about it only come back to complain.
A/B Testing
So, A/B testing is Microsoft’s way to do what the Xbox system does, but silently and without pissing people off. In the Xbox system, people are clearly more experienced and more knowledgeable if they reached a higher feedback level and climb their ranks from Omega to Alpha because they had to prove it.
But there is no such thing in the Windows program, while it is clearly useful since you want to know the dedicated testers from the “I just want the latest shit, always”-crowd. You know that you can give the more technical people a more broken build and faster releases because they can probably handle it, because, you know… they are more technical. They are probably also wise enough to keep a few machines on a more stable branch or maybe don’t run the latest build on their daily driver at all.
So… how do you tell those people apart if they are on the same branch and using the same builds? A/B testing. Microsoft can completely decide which features you get to test based on what they know about you, they can tell if you are the more technical person, based on how you interact with the OS, the feedback system and more. And they don’t have to tell you or anyone about it, so people that don’t get chosen for a test don’t feel left out, just have “bad luck”. Guess what happens now? The more technical people get the more experimental features that have a higher chance of being broken, and most of the non-technical people don’t. But some still do because telemetry is important. Grats to those lucky guys, I guess.
What they did with that is basically adopting the Xbox system in silent, without any visible benefit to the user. Why not do it in the open and let people work for their right to get to test more experimental stuff? Let them prove that they can handle it.
Do you just don’t want to appear to be mean, segregating like that? So, they don’t feel left out? I don’t really know but I think Microsoft should just switch to a clearly tiered system openly. Offer rewards for every tier you climb, actually give them something for doing being helpful instead of just being telemetry bots.
I know back in the days when I got into my first beta programs, I approached updates with a “Okay boys, let’s break some shit and report it so it gets fixed” – mentality. I don’t think that the most people on the fast ring think that way, or that they should. But there should be a place for those people, and there should be incentives for other people to become those people, so the quality of feedback gets better and better over time.
