Corporations don't want LTS. They want RidiculouslyLTS. Ubuntu's LTS releases are supported for 3 years. IE 6 is now almost 10 years old. This is what corporations want. They want to install software and forget about it for a decade. This is not a realistic model for a browser. Even 3 years is not very realistic. 3 years means running Firefox 3.0 now, in June 2011. Sure, some people do it, but it's not a great idea for many reasons.

When you're looking at Microsoft Word, expecting long-term support makes sense. It's shrink-wrapped software and upgrading is a big ordeal. You have to upgrade everyone at once, or someone's going to (repeatedly and consistently) send out docx files to coworkers who cannot open them. It can take a lot of time and money to upgrade everyone. Browsers are (or should be) different. The web is continually evolving. New exploits pop up fairly often. Regular upgrades are necessity for security and functionality, so upgrading a browser should be a minor, easy thing, and it should happen frequently. The problem is not the lack of support from Mozilla, but the expectation from corporations that browsers should be treated like Word processors.

"Corporations don't want LTS. They want RidiculouslyLTS."

Sure they do, but they also want 100% tax exemption, employees that will work 80 hours for 40 hours pay, and $0 fire and theft insurance. Companies want lots of stuff they don't get :)

My point is that it's not black and white unless Mozilla chooses to make it that way. Right now, it's white. For Mozilla to say: "Infinite incremental release is the way forward" is to say, "We don't care about your ability to plan releases in any way shape or form." That's pretty black & white. Any commitment to a release schedule would, at least, give them the ability to plan.

I agree that web browsers are not synonymous with word processing software, but the expectation that large corporations will simultaneously adopt web based technologies while throwing out decades of change-management process is just naivety at it's finest.

Large organizations benefit from economies of scale. This means reduced costs. This means one IT manager per five hundred PCs. This means you can't roll out software willy-nilly and expect your organization to stand. Any in-roads Mozilla gained with corporations will be quickly squandered if they stick to this strategy.

Again, I agree that the old pattern is not what we need, but this is upheaval, and that's not what works at corporations.

What's interesting is that in the past the approach corporations have taken to browsers is that they wait for the browser to be released, then test it in their environment to see whether it broke anything.

This is as opposed to testing the beta to see whether it's breaking anything and if so report a bug to the browser vendor so the final release _won't_ break anything.

Just making that one change to their rollout process would significantly reduce the breakage potential of updates....

Exactly, companies want a lot of stuff, but that doesn't mean they should get it.

> My point is that it's not black and white unless Mozilla chooses to make it that way. Right now, it's white. For Mozilla to say: "Infinite incremental release is the way forward" is to say, "We don't care about your ability to plan releases in any way shape or form." That's pretty black & white. Any commitment to a release schedule would, at least, give them the ability to plan.

Asking for a release timeline and asking for 3 years of support on a dead branch are entirely separate things. It's not unreasonable to ask the Mozilla keep users informed of the release schedule (don't they already do this?), but asking them to support an old version for 3 years seems silly. It takes a lot of manpower and there's no value in it for probably 95% of their customers.

> I agree that web browsers are not synonymous with word processing software, but the expectation that large corporations will simultaneously adopt web based technologies while throwing out decades of change-management process is just naivety at it's finest.

I would say it's practical, not naive. If you want a browser that allows you to stagnate while enjoying long-term support, then either start paying someone for support, or start using IE. You agree that web browsers and word processors are different, so why should Mozilla bend over backwards to allow corporations to try to treat them the same?

> Large organizations benefit from economies of scale. This means reduced costs. This means one IT manager per five hundred PCs. This means you can't roll out software willy-nilly and expect your organization to stand. Any in-roads Mozilla gained with corporations will be quickly squandered if they stick to this strategy.

Sure, so don't roll out willy-nilly. Test and then roll out, just as they do now. But instead of rolling out 4.1, roll out 5.0. I don't understand what the big issue is. I'm sure it's a bit more work if there's a breaking change, but frankly you're going to have to deal with those breaking changes eventually unless you're going to stick with the same browser for years (let's call this the IE6 model). You just deal with them incrementally rather than in one massive painful push. Frankly the incremental approach sounds less painful for everyone involved.

We're talking about corporations demanding that a free product meet their antiquated needs. There's no grounds for the demand. It reflects their broken rollout policies, not a deficiency on the part of Mozilla. Maybe if it takes you several months to approve a browser rollout, that's your real problem. And let's be honest, a lot of this is just baseless fear. We're talking about a change in the way version numbers are incremented. People are worrying because it's called 5.0 instead of 4.1. The number attached doesn't really matter.

> Again, I agree that the old pattern is not what we need, but this is upheaval, and that's not what works at corporations.

What else would work? For Mozilla to dedicate a lot of extra resources to maintaining dead branches for years? Not only does this not help move corporations forward, it doesn't help Mozilla, and it doesn't help the web.

If the current change management models are not working, then they need to be fixed. Test and rollout faster and more frequently. Roll out in stages, rather than with the "big bang" model. Have a roll-back plan and system in place. Really, the way people talk, it's as if every week the browser is going to break half the web. I hardly think that's the case, and I frankly think Mozilla's testing is probably more thorough than most any corporate IT department.

I don't entirely disagree with most of what you said, but I think you underestimate the challenges. I own a small start-up, so I know how nice it is to operate at a small scale. I let my employees chose their own browser, but I require that they keep it up to date. I can do that because I don't have to worry about supporting 500 people.

I've also consulted for very large corporations. The same disruptive ideology that applies to start-ups falls flat with large corporations. The biggest mistake is to take a unilateral position with them. They will fall back to IE. They will stagnate on old versions if we don't work with them. Is that what you want?

Maybe your product can avoid the corporate space altogether, but there are a lot of us who rely on them. I don't want to re-live IE6. I want browser makers to acknowledge that changing the corporate world isn't the same as creating Google or Facebook. It's far less glamorous and usually means doing shit in a way that you don't like for longer than you like. Change is coming, but it's coming slowly.

I hate line-item rebuttals, because they just turn in to pissing matches, but I did want to offer some clarification on these two points.

> Asking for a release timeline and asking for 3 years of support on a dead branch are entirely separate things. It's not unreasonable to ask the Mozilla keep users informed of the release schedule (don't they already do this?), but asking them to support an old version for 3 years seems silly. It takes a lot of manpower and there's no value in it for probably 95% of their customers.

Let me clarify: I don't think 3 years is a viable LTS schedule for browsers. I don't know what the time frame is, but I'm sure it's not 3 years, and I'm sure it's not "We release continuously." I should have been more clear about what "similar" means when referencing Ubuntu. By similar, I mean that they should continue their march forward, but that there should be an occasional LTS release that is maintained for a longer, more planned interval.

> I would say it's practical, not naive. If you want a browser that allows you to stagnate while enjoying long-term support, then either start paying someone for support, or start using IE. You agree that web browsers and word processors are different, so why should Mozilla bend over backwards to allow corporations to try to treat them the same?

I'm saying it's naive (showing a lack of experience, wisdom, or judgment) to expect that this new development cycle won't have ramifications. Mozilla is free to do what they wish with their product. What I'm claiming is that it will come at the cost of market share in the corporate world. If they don't care about the market share, then go ahead and walk on it.

> I've also consulted for very large corporations. The same disruptive ideology that applies to start-ups falls flat with large corporations. The biggest mistake is to take a unilateral position with them. They will fall back to IE. They will stagnate on old versions if we don't work with them. Is that what you want?

What I want is for corp IT departments to not pretend as if this is somehow an insurmountable change. Yeah, corporations are slow and risk-averse. How risky is it to upgrade the browser once per quarter? Are they not testing and upgrading now for the sake of security fixes?

> Let me clarify: I don't think 3 years is a viable LTS schedule for browsers. I don't know what the time frame is, but I'm sure it's not 3 years, and I'm sure it's not "We release continuously." I should have been more clear about what "similar" means when referencing Ubuntu. By similar, I mean that they should continue their march forward, but that there should be an occasional LTS release that is maintained for a longer, more planned interval.

Okay, I can understand that. I can see the value in dropping a version every 9 months with a guarantee of 1 year of security fixes. Or maybe drops every 6 months that get 9 months of support.

I think if Mozilla is committed to backwards-compatibility moving forward, though, this shouldn't be necessary. If they can't avoid breaking changes for some reason, then it is an issue. (One exception would be breaking changes in beta/bleeding-edge features. If you build your product on unstable CSS or whatever, that's your burden. And if you built your product on features that were only around for 3 months, then you should be able to fix your product quickly.)

> I'm saying it's naive (showing a lack of experience, wisdom, or judgment) to expect that this new development cycle won't have ramifications. Mozilla is free to do what they wish with their product. What I'm claiming is that it will come at the cost of market share in the corporate world. If they don't care about the market share, then go ahead and walk on it.

Thanks for the definition of naive. :) I never said it wouldn't have ramifications. I actually said the opposite. Corporate IT shops need to find a way to move more quickly. They need to change. Running everything on a multi-year upgrade cycle is ridiculous, and borderline incompetent (though the incompetence might be coming from higher up than the IT guys).

> Corporate IT shops need to find a way to move more quickly. They need to change.

That is easy to say, and difficult to do.

The simplest way I can think of to explain why is that every single move, or change, comes with a nonzero cost-per-employee. While it may be possible to reduce that cost, nobody has yet invented a way to eliminate it.

When you have 5 employees, a small change that costs X-per-employee is not a big deal.

When you have a few thousand...

> That is easy to say, and difficult to do.

No doubt. That doesn't mean it shouldn't happen.

> IE 6 is now almost 10 years old. This is what corporations want.

Straw man. The only people I see bringing IE6 up in this discussion are those trying to create a false dichotomy between the now-trimonthly release schedules of Firefox and Chrome and something bad.

> They want to install software and forget about it for a decade. This is not a realistic model for a browser.

Well, if you want browsers to be taken seriously as a platform for running software applications and not just viewing static sites with occasional interactive features, then browser had better become that long-lived.

Do you have any idea how much investment Microsoft makes in keeping each new version of Windows backward compatible with software and drivers from the last version (or, more likely, the last several versions)? There is no way they would be the dominant force on the desktop today if they had gone around releasing point updates to Windows every few months and arbitrarily breaking other people's software that was built on their platform.

Even your three year figure is telling. If Windows 7 hadn't been able to run existing applications written in the XP era, Microsoft would probably be toast today, having blown two big OS releases in a row and lost the confidence of the developer market. Three years is within the typical lifespan of a single PC in an office environment, particularly these days when IT departments are seriously questioning the value they get from spending time and money on the upgrade roundabout and paying for all these subscription plans. Everything else that came with the PC still runs three years later and still lets people get their jobs done. Why should browsers be anything special?

Your final paragraph is just outright denial. Microsoft can and do patch security and compatibility bugs in Word all the time. They just manage to do it without changing the UI, introducing new file formats while obsoleting ones that were fine just six months ago, and breaking a couple of key features that they don't care about because only 5% of users need them.

> Straw man. The only people I see bringing IE6 up in this discussion are those trying to create a false dichotomy between the now-trimonthly release schedules of Firefox and Chrome and something bad.

It's not a strawman. Why do you think IE6 is still around? It's not for the grandmothers who don't know how to run Windows Update. It's for corporations who demand that Microsoft support an ancient product because they're too lazy/cheap/incompetent to upgrade their internal apps to work on modern browsers.

Corporations want indefinite support on every piece of software they use (and why not, I guess). The question becomes how long does it make sense for vendors to support a product. When you're talking about a product that needs to be upgraded frequently to continue to do its job and to maintain security, I'd say the answer is "not very long".

> Well, if you want browsers to be taken seriously as a platform for running software applications and not just viewing static sites with occasional interactive features, then browser had better become that long-lived.

You're the one creating a false dichotomy here, pretending that the only options are to build for a stagnant browser or to not build at all. The browser is not the platform, and that's what you don't get. The web is the platform. If you built a web app for Firefox 4, it's going to work in Firefox 5. The platform hasn't changed. It's gained some new features perhaps, but the original functionality is still there.

> Do you have any idea how much investment Microsoft makes in keeping each new version of Windows backward compatible with software and drivers from the last version (or, more likely, the last several versions)? There is no way they would be the dominant force on the desktop today if they had gone around releasing point updates to Windows every few months and arbitrarily breaking other people's software that was built on their platform.

An interesting point of comparison would be to note that Firefox 5 is backwards-compatible with Firefox 4. Unless you choose to use bleeding-edge beta functionality, your sites will still work just fine. Mozilla (and Google, and Apple, and Microsoft) work really hard to maintain backwards-compatibility in their browsers. You're pretending that upgrading to FF5 means that everything written for older browsers will stop working. We both know that's not true.

> Even your three year figure is telling. If Windows 7 hadn't been able to run existing applications written in the XP era, Microsoft would probably be toast today, having blown two big OS releases in a row and lost the confidence of the developer market. Three years is within the typical lifespan of a single PC in an office environment, particularly these days when IT departments are seriously questioning the value they get from spending time and money on the upgrade roundabout and paying for all these subscription plans. Everything else that came with the PC still runs three years later and still lets people get their jobs done. Why should browsers be anything special?

They aren't special. Firefox 5 will run all the same sites as Firefox 3.0, and it will run a lot more as well.

> Your final paragraph is just outright denial. Microsoft can and do patch security and compatibility bugs in Word all the time.

Of course they patch bugs. I never said they didn't. That doesn't mean the Word Processor model makes sense for browsers. I'm using Microsoft Office 2008. Should I also be using Firefox 3.0? I haven't upgraded Office because it costs money. I have upgraded Firefox because it's free and easy. Browsers are no longer shrink-wrap software, and treating them as if they are is ridiculous.

> They just manage to do it without changing the UI, introducing new file formats while obsoleting ones that were fine just six months ago, and breaking a couple of key features that they don't care about because only 5% of users need them.

Can you point out what FF5 has done that's so terrible and backwards-incompatible?

Edit: I guess there are some add-on incompatibilities, and that is something that they need to address moving forward. Frequent drops are fine, but should maintain compatibility.

> Why do you think IE6 is still around?

I really don't, at least not on the scale it was a couple of years ago. I expect there are a handful of organisations who are still stuck in IE6 land, but it's no longer the limiting factor it used to be.

> When you're talking about a product that needs to be upgraded frequently to continue to do its job and to maintain security, I'd say the answer is "not very long".

I guess I lack sympathy because, ironically, the vendors are still using old and underpowered tools and casual processes for their development, which is why we need so many security patches. Usually, those tools start with a C and end with the word "compiler". Often, the development processes are trendy, "Agile" things that emphasize prototyping and pushing rapid updates over clear specs and systematic designs, and that treat techniques like unit testing as some sort of divine correctness guarantee instead of a back-up to good basic development practices.

Well, you reap what you sow. If you don't invest in tools and processes that can build robust software that requires little maintenance, I'm going to bitch if you don't invest in supporting your existing software either.

We've had programming languages much better suited to application development than C and its derivatives for many years now, and there is really no excuse for still writing things like networking software using a model that has a terrible risk/benefit ratio for that kind of project.

It's also a common fallacy that writing code significantly less buggy than what we put up with today needs heavyweight processes that cost more. There are many people in many industries who do it every day and have the metrics to prove it.

> Unless you choose to use bleeding-edge beta functionality, your sites will still work just fine.

Sure, but unless people are going to use that bleeding edge technology, why push out new versions of the browser with the corresponding functionality changes? It would be far safer to push only such security patches and bug fixes in existing functionality as are necessary, and make functional changes at a pace that allows for the whole industry to develop sustainably and actually take advantage of the improvements.

> Can you point out what FF5 has done that's so terrible and backwards-incompatible?

Well, so far it seems to have broken quite a few plug-ins and it looks like some of the typography engine bugs are back again, but I've only been using it for two days so I haven't yet tried it with most of the projects I'm involved with.

In any case, shipping one version without breaking anything is nothing to be proud of. You have to ship every version without breaking anything, and neither Firefox nor Chrome has a good track record in that department.

I really wish you had your email address in your profile.

Sorry, I like my pseudonymity shield in this sort of discussion. It makes conflicts of interest far less likely, given that I have a number of professional interests and possibly not everyone I work with would like my honest views on this subject. (I don't think that matters as long as the work I do for them is my best effort at what they're paying me for, but they might disagree.)

Should I ask why you wanted my e-mail address...?

Because I'm seriously considering devoting some of my meager resources to forking and long-term maintaining the Firefox 3.6 and 4-5 versions, and wanted to get some non-public feedback on that from someone with the same problems I have.

Well, the nice thing about a pseudonym is I can tell you the same things here that I'd say privately. :-)

FYI in case you didn't know, it looks like they are planning to maintain the 3.x branch for a while, but 5 is being treated as the next step from 4 so they won't be maintaining a 4.x series independently.

Also FWIW, some of the specific problems encountered by projects I've worked on actually crept into the later 3.x series, notably the ones involving text rendering problems and the ones involving Java applets.

So I guess that's a vote from me for "probably more valuable uses for that much of your time", but I suppose YMMV depending on what specific issues you've encountered.

> I really don't, at least not on the scale it was a couple of years ago. I expect there are a handful of organisations who are still stuck in IE6 land, but it's no longer the limiting factor it used to be.

And we can all be thankful for that.

> I guess I lack sympathy because, ironically, the vendors are still using old and underpowered tools and casual processes for their development, which is why we need so many security patches. Usually, those tools start with a C and end with the word "compiler". Often, the development processes are trendy, "Agile" things that emphasize prototyping and pushing rapid updates over clear specs and systematic designs, and that treat techniques like unit testing as some sort of divine correctness guarantee instead of a back-up to good basic development practices.

I suppose you could use HotJava or Lobo if you want a browser not written in C(++). Really, if you think that Firefox is poorly written and managed, why not move to something else? I feel like software companies are always under fire for using outdated techniques and poor quality control and whatever else, and yet these complaints seem so often to come from people who haven't produced anything on the scale of the software they criticize.

> Well, you reap what you sow. If you don't invest in tools and processes that can build robust software that requires little maintenance, I'm going to bitch if you don't invest in supporting your existing software either.

What projects have you worked on that had 5MM lines of code? Considering how complex a modern browser is, I'm not sure the bug rate is especially high. How many people use Firefox every day and rarely encounter bugs? Seems pretty robust to me.

> We've had programming languages much better suited to application development than C and its derivatives for many years now, and there is really no excuse for still writing things like networking software using a model that has a terrible risk/benefit ratio for that kind of project.

I'm sure you'll be looking into Lobo, then.

> It's also a common fallacy that writing code significantly less buggy than what we put up with today needs heavyweight processes that cost more. There are many people in many industries who do it every day and have the metrics to prove it.

I'm not really sure why you're going down this path, since I don't think I ever said anything about this, but I'll bite. What industries are producing significantly less-buggy code at the same (or similar) cost?

> Sure, but unless people are going to use that bleeding edge technology, why push out new versions of the browser with the corresponding functionality changes? It would be far safer to push only such security patches and bug fixes in existing functionality as are necessary, and make functional changes at a pace that allows for the whole industry to develop sustainably and actually take advantage of the improvements.

Safer? Sure. More expensive? That, too. Maintaining multiple branches is not free. Mozilla has been playing that game for a long time, and I think they're sick of it. I doubt their decision was made indiscriminately.

> Well, so far it seems to have broken quite a few plug-ins and it looks like some of the typography engine bugs are back again, but I've only been using it for two days so I haven't yet tried it with most of the projects I'm involved with. > In any case, shipping one version without breaking anything is nothing to be proud of. You have to ship every version without breaking anything, and neither Firefox nor Chrome has a good track record in that department.

If they can't ship without introducing functional bugs, then I agree they have issues they need to work out. The plugin thing isn't really acceptable, either. If you're going to have other people build on your product, you absolutely need to support them.

> I feel like software companies are always under fire for using outdated techniques and poor quality control and whatever else, and yet these complaints seem so often to come from people who haven't produced anything on the scale of the software they criticize. [...] What projects have you worked on that had 5MM lines of code?

I think you know that I'm not going to answer that when I'm posting under a pseudonym, but I've worked on some reasonably large-scale projects. I doubt anything I've written ultimately has as many end users as Firefox, but I'm definitely in both the millions-of-lines club and the millions-of-users club, probably multiple times by now, and I've worked on projects where faults can cause Very Bad Things to happen.

> I'm not really sure why you're going down this path, since I don't think I ever said anything about this, but I'll bite. What industries are producing significantly less-buggy code at the same (or similar) cost?

My point is that if you're going to write software with a huge cross-section for malware to attack, such as a web browser, and you choose to write it using decades-old technology, where it is difficult to write it securely, then it's hard to take seriously any complaints about how hard it is to maintain that code and keep up with security patches.

As for other industries, obvious ones would be defence, aerospace, finance, and infrastructure management. When was the last time you saw a plane fall out of the sky because of software failure, or your electricity supply cut off because the management software for the national grid crashed? (There are also oddities like TeX, but then that was written by Donald Knuth, so it's hardly representative.)

Software developers in these industries typically work to different standards than those developing consumer products, and yes, the up-front costs are typically significantly higher. The thing is, by the time you account for the time it takes to fix bugs post-release compared to fixing it at an earlier stage in development, the difference in the total effort isn't such a high factor. And at that stage, you still haven't counted the benefit to society (and to your PR) of not having perhaps millions of users disrupted by each serious bug.

The bottom line is that as an industry, we are mostly stuck in the past, but we are there only because of inertia and non-technical factors, not because of engineering merit. As I noted before, this is a rather ironic argument in a discussion about trying to move people onto newer and better browser technologies, but quite apt: it shows how little progress we make if all the better technologies live in their own little worlds.

> If you're going to have other people build on your product, you absolutely need to support them.

Exactly. I think that's the key point that I and a few other people in this thread have been trying to make.

> I think you know that I'm not going to answer that when I'm posting under a pseudonym, but I've worked on some reasonably large-scale projects. I doubt anything I've written ultimately has as many end users as Firefox, but I'm definitely in both the millions-of-lines club and the millions-of-users club, probably multiple times by now, and I've worked on projects where faults can cause Very Bad Things to happen.

Fair enough, but I stand by what I said. Most of the people who criticize large software projects have never worked on a similar scale. Nor have most of them worked on projects with similar performance needs. It's all fine and dandy to criticize the choice to use C++ when you've never had to write a large performant program (not that this necessarily applies to you).

> My point is that if you're going to write software with a huge cross-section for malware to attack, such as a web browser, and you choose to write it using decades-old technology, where it is difficult to write it securely, then it's hard to take seriously any complaints about how hard it is to maintain that code and keep up with security patches.

So what's the alternative? To write it in Java, so it's measurably slower and feels even slower than it measures? To write it in C# so it only runs on Windows reliably? To write it in Smalltalk (yeah, right)? To write it in Haskell so that only a handfull of people have the skills to do the job? Oh, and let's not forget all of those have major pieces written in C that would still be attack vectors.

> As for other industries, obvious ones would be defence, aerospace, finance, and infrastructure management. When was the last time you saw a plane fall out of the sky because of software failure, or your electricity supply cut off because the management software for the national grid crashed? (There are also oddities like TeX, but then that was written by Donald Knuth, so it's hardly representative.)

Oh, come on. I've worked in the Defense industry. The processes are heavier and I dispute that the quality of the resultant code is better. It's also a ton of C. If most of it were attached to the Internet, people would find holes all over it. Defense software most certainly has bugs, sometimes serious ones. The aerospace industry has similar issues. Tons of C and C++. If you're talking about NASA, let's not forget that Spirit landed on Mars and went into fault mode due to software problems. Finance? Seriously? How about the fact that a few months ago the DJIA lost a trillion dollars in value for half an hour due to a software malfunction?

On the other hand, the space shuttle software was a massive success, with relatively few bugs. It was also written in an extremely heavy-weight process that cost many times more per line than typical software development. And that's okay, because it's the space shuttle, but it's not realistic to expect this of normal software, and it's incorrect to claim that it doesn't cost more. It also has far fewer lines of code than most commercial aircraft systems, because NASA knows that simpler is generally safer. Fewer features mean less complexity, but people don't want a simple browser, because that means NCSA Mosaic.

A major difference between most of the things you mentioned and Firefox is that the things you listed are not constantly exposed to the Internet. The bugs in Firefox that matter are security bugs that don't affect Airbus 380s because A380s aren't browsing random Internet sites. The machines that control the electric grid don't have wide open ports that random attackers can reach. The attack surface of Firefox is huge, and that's why more security bugs are discovered there than, say, Visa's payment centers. High-reliability systems are also systems. When Firefox fails, you see it and curse at Mozilla. When Visa's software fails, you probably never see it, because you get rerouted to a different machine. When an Airbus has a machine failure, the backup kicks in, or worst case the pilots go manual. That's redundancy that Firefox cannot reasonably provide. (Also, when Visa's software messes up, you just reswipe the card and never realize what actually happened.)

> Software developers in these industries typically work to different standards than those developing consumer products, and yes, the up-front costs are typically significantly higher. The thing is, by the time you account for the time it takes to fix bugs post-release compared to fixing it at an earlier stage in development, the difference in the total effort isn't such a high factor. And at that stage, you still haven't counted the benefit to society (and to your PR) of not having perhaps millions of users disrupted by each serious bug.

I'm going to disagree here. For one, I think you're just handwaving when you claim that the cost to fix bugs balances out the cost of a much heavier process. I don't think that's true, and I think the value proposition for most consumer software makes this clearly a bad idea. For another, you're comparing software that does one relatively unchanging thing against software that is constantly growing. Yeah, the electric grid is pretty reliable. It's also done the same job for decades. Firefox hasn't even existed for a decade (unless you count Netscape Navigator, which is iffy because so much was rewritten). Spending twice as long to build something to get a little more reliability can be a pretty good tradeoff when you'll use it nearly unchanged for 20 years. It's a pretty bad idea when you're going to be actively developing the next version as soon as the current one drops.

> The bottom line is that as an industry, we are mostly stuck in the past, but we are there only because of inertia and non-technical factors, not because of engineering merit. As I noted before, this is a rather ironic argument in a discussion about trying to move people onto newer and better browser technologies, but quite apt: it shows how little progress we make if all the better technologies live in their own little worlds.

I don't think we're stuck in the past. The industries you mentioned use the same languages and often older ones. I think our engineering and processes are much better than the were a decade or two ago.

> Exactly. I think that's the key point that I and a few other people in this thread have been trying to make.

And I don't dispute that at all. My point is that frequent drops and reliable support are not mutually exclusive. Whether Firefox will deliver both is a good question, but it certainly could at least in theory.

FWIW, I've spent a substantial chunk of my career working with C++, including on heavily mathematical code where C++'s performance did make it a suitable tool for the job. That taught me that advocates of other languages sometimes exaggerate their performance claims.

It also taught me that most software that "needs to be written in C++ for performance reasons" really doesn't. Most of the old arguments against VM-based languages, so-called scripting languages, and functional languages have been out of date for years now as technologies like JIT compilation have matured. As the coming generations of processors introduce more parallel processing power but don't speed up sequential execution much more, languages that provide a natural model for expressing parallel algorithms or for auto-parallelisation behind the scenes will become even more advantageous.

You mentioned some of the aerospace projects but you automatically said that it wasn't realistic to expect similar quality from "normal software". That is the mindset that I think needs to change in our industry. For example, every study I've ever read about Cleanroom Software Engineering has shown that its up-front development costs (both time and money) are typically within a factor of 2 or 3 of a "normal" software development process (often much closer) and while it doesn't completely eliminate bugs you do consistently see bug rates at least an order of magnitude lower for Cleanroom jobs. Over the lifetime of most projects, the total development+bug fixing time is certainly in the same ballpark at that point. There is no evidence that such code is any less maintainable in the face of changing requirements than code developed using any other process (on the contrary, the systematic and carefully documented structure gives you a great start) and the morale of the development team is usually noticeably higher (because they are spending most of their time building interesting stuff instead of fixing bugs that should never have been there).

Elsewhere, there is telecoms control software in the world, written in Erlang, that has been operating continuously for years with a few seconds of downtime in total since it went live. That's some absurd number of 9s of reliability, because the software architecture is fundamentally designed to be fault tolerant.

I guess I'm just trying to say that we shouldn't assume today's routine commercial practice is the most efficient, reliable way of doing things. We know, beyond any reasonable doubt, that it isn't. As an industry, we allow ourselves to be held back by non-technical issues like the availability of ready-trained staff, because development groups are too tight to provide training to improve their people's skills, and by preconceptions that say languages or development processes or software design principles that aren't today's mainstream must be too hard for everyday development tasks outside of niches where quality really matters.

I'd have a lot more sympathy for development groups that struggle to maintain shipping code in the face of evolving security threats and the like if those groups didn't shoot themselves in the foot, stick a noose around their necks and then tie their hands behind their back before they kicked the stool away.

> It also taught me that most software that "needs to be written in C++ for performance reasons" really doesn't.

Most software doesn't need to be written in C++ for the simple reason that most software doesn't actually need to be highly performant. Beyond that, many applications can be written sufficiently performant in other languages. However, I'm not convinced that for software such as a browser, where performance is key to the success of emerging web technologies, anything but C/C++ is really going to do the job. There's a lot of evidence that various languages are almost as fast as C or C++, but rarely as fast. That "almost" can add up in complex programs, especially when you're talking about programs that have to run other programs (i.e. Javascript). If you disagree, what language do you believe would be appropriate for building a cross-platform web browser?

> You mentioned some of the aerospace projects but you automatically said that it wasn't realistic to expect similar quality from "normal software". That is the mindset that I think needs to change in our industry. For example, every study I've ever read about Cleanroom Software Engineering has shown that its up-front development costs (both time and money) are typically within a factor of 2 or 3 of a "normal" software development process (often much closer) and while it doesn't completely eliminate bugs you do consistently see bug rates at least an order of magnitude lower for Cleanroom jobs. Over the lifetime of most projects, the total development+bug fixing time is certainly in the same ballpark at that point. There is no evidence that such code is any less maintainable in the face of changing requirements than code developed using any other process (on the contrary, the systematic and carefully documented structure gives you a great start) and the morale of the development team is usually noticeably higher (because they are spending most of their time building interesting stuff instead of fixing bugs that should never have been there).

That's not really what I said. Expecting commercial products to use the same heavyweight process that aerospace uses is not realistic. It is not reasonable to expect Mozilla to spend 6 to 9 months to implement the same features that Microsoft or Google develops in 3. This is a strategy for loosing the entire market. If the quality bar needs to be raised, it must be done more efficiently than by adopting "defense-grade" process.

Sure, when a code failure causes a missile to detonate inside a fighter jet, you can afford the additional cost (and why not, Uncle Sam is footing that bill). But when a code failure results in a browser restart, you can't justify the extra development effort. For much lower effort you can build a system that says "oops", saves state, and restarts right back to where the user was. Indeed I think all the major browsers do that now, though I'm not sure, because I can't actually recall the last time I actually had a browser crash on the Desktop.

I also disagree with your assertion that developers are happier on teams that practice heavyweight development processes. I've never heard anything but the opposite. Spending hour upon hour writing and refining specs is hell. It is not coding, and most coders don't like doing it to excess. Experienced coders should also know that a lot of it is wasted time, because inevitably half of the assumptions made turned out wrong. Some planning is a good thing. Heavyweight processes are something else entirely. Extra specs and documentation might help reduce bugs, but probably not nearly so much as more/better testing.

> Elsewhere, there is telecoms control software in the world, written in Erlang, that has been operating continuously for years with a few seconds of downtime in total since it went live. That's some absurd number of 9s of reliability, because the software architecture is fundamentally designed to be fault tolerant.

This is a false comparison. I could point out that Firefox has been running (some version, somewhere) continuously for almost a decade, and that would almost be more reasonable. At least then we'd be comparing a bunch of machines to a bunch of machines. Telecom software is not magically bug-free. Quite the opposite, languages like Erlang are designed with failure in mind. Machines fail. Networking devices fail. Antennas fail. Software fails. Rather than asserting that the solution is to write better software, Erlang is an acknowledgement that the solution is a better system. The software isn't better in the sense of having fewer bugs or using a DoD development process. It simply expects failure. When Erlang encounters an error, it can retry the operation, restart the process, or move to another machine. Firefox will retry, it will restart if necessary, and it is adding things like process separation for crash-prone plugins. These things are a net gain for users, but the reality is that faults still happen, and the gains in quality here do not come from fewer bugs but from better response to those bugs.

> I guess I'm just trying to say that we shouldn't assume today's routine commercial practice is the most efficient, reliable way of doing things. We know, beyond any reasonable doubt, that it isn't. As an industry, we allow ourselves to be held back by non-technical issues like the availability of ready-trained staff, because development groups are too tight to provide training to improve their people's skills, and by preconceptions that say languages or development processes or software design principles that aren't today's mainstream must be too hard for everyday development tasks outside of niches where quality really matters.

I've never said that today's practices are the most efficient, but going backwards in time to adopt DoD-style development is a move in the wrong direction. Spending 3 times as long on a given development effort may yield some small quality gains, but the overall effect is a loss of value. A product that has 10% fewer meaningful bugs but has 66% fewer necessary features is a failure. The move forwards is to build in more fault tolerance. When you run Firefox, it uses some fault-tolerant techniques already, and will hopefully add more. When you execute a search on Google, it uses fault-tolerant techniques. These are the same techniques that other high-reliability industries use. Stuff still breaks, but they recover. Our industry is not trailing the state of the art here. We're doing the same things. The state of the art can always improve, but that doesn't mean that we are doing a bad job now.

The security of Microsoft's Windows line went way up in the wake of all the XP exploits. It was certainly not a move towards heavy-weight development process that made these gains. In fact everything I've heard has indicated that Microsoft has moved the other direction toward "agile" development. What changed was the focus. When security is top priority, it unsurprisingly gets better. If you want fewer bugs, hire more great testers and have them work closely with developers. Give your developers security training. Hire security experts. Use the best tools you can get. And adopt a culture that places bug-fixing first on the list. But don't saddle your developers with an antiquated development system, and especially don't try to say that this system is somehow leading-edge when the industry has already tried it (numerous times).

> I'd have a lot more sympathy for development groups that struggle to maintain shipping code in the face of evolving security threats and the like if those groups didn't shoot themselves in the foot, stick a noose around their necks and then tie their hands behind their back before they kicked the stool away.

Meh. This is just being hateful.

Our posts are getting very long going point-by-point, so I won't address everything you've said individually now. However...

On the efficiency issue, IMHO you're still too focussed on the up-front cost, when it is the long term efficiency that really counts. It doesn't matter if it takes 6 months to develop something properly instead of 3 months to hack it, if the hackers then spend another 6 months patching bugs before anyone can use it.

Even if that weren't true, do we really believe that a tri-monthly release cycle is necessary to compete in the browser market today? It takes longer than that for new technologies to be used in production projects, even by the most die-hard of bleeding edge early adopters. Wouldn't you rather code to a well-defined spec and know that it really was going to work in everyone's browser, even if it took another three months for the your favourite new feature to be well-specified so you could use it? We've waited years for things like HTML5 and CSS3, so I think we could wait another few weeks to get them right!

Regarding the morale of developers, again, I think you're confusing heavyweight processes generally with my example of Cleanroom in particular. There are plenty of studies that show developers do enjoy working within that process; try a Google Scholar search for Cleanroom Software Engineering and read a few of the papers. Likewise, you also seem to be assuming that Cleanroom is heavy in the sense of being all about writing specs and management overhead, which suggests to me that you've never actually tried it to see what it feels like in practice.

This is exactly the sort of preconception and "I just don't believe it" reaction I think we need to overcome with evidence if our industry is going to improve its performance. Bizarrely, we seem to be spending far more time today worrying about issues like TDD and pair programming, which have much less proven benefit and get highly variable feedback from those who have used them.

I can't help but observe that those ideas are very accessible to a typical journeyman programmer versed in OO languages today, while shifting to something like a Cleanroom process or an Erlang style software architecture or a more declarative programming style as functional programming languages promote requires understanding concepts that for most people are radical and unfamiliar. We fear what we do not understand, and that is holding us back.

A final example on this:

> Extra specs and documentation might help reduce bugs, but probably not nearly so much as more/better testing.

On the contrary, working to good specs with a robust peer review culture is a highly effective quality strategy with an excellent RoI, and more than stands up to any test-based approach I've seen. The evidence for this is overwhelming, if you choose to seek it out. Again, I refer you to Google Scholar, or a good bookshop. But again, you have to be willing to look at a culture shift and trying something with a completely different philosophy to what you do today. You typically won't find this stuff in Cleany McCoder's Internet Echo Blog.

> Spending 3 times as long on a given development effort may yield some small quality gains, but the overall effect is a loss of value. A product that has 10% fewer meaningful bugs but has 66% fewer necessary features is a failure.

Perhaps, but what if it's more like a 10% overhead to better than halve the number of bugs (as in one small academic comparison I just quickly looked up on Google Scholar)? What if a factor of 2-3x in the up-front development costs really cuts your bug rate from 20 bugs/KLOC not to 18 bugs/KLOC but right down to about 1 bug/KLOC (not unusual for Cleanroom with established teams in industrial practice), with all the resulting savings in maintenance costs later as well as the user-visible improvement in quality?

I'm picking Cleanroom in particular here just because I happen to know a little about it, but my point isn't that we should all use Cleanroom. My real point is that there are different processes and design techniques and languages and tools out there, some of them very different to typical industrial practice today, and some of them objectively performing much better.

If any software group (I'm not just digging at Firefox, it's an industry-wide problem) wants to complain about maintenance costs and how it's impossible to do much better on quality without silly overheads, I think they should take a look outside the mainstream with an open mind before they make too many bold claims. In the meantime, claims about how it's too hard to both release quickly and maintain basic security patching on other branches (and here I am digging at Firefox's new release strategy) all sound a bit hollow.

For the process efficiency, I simply disagree with your assessment. I'm not aware of any unbiased works that indicate that using a heavyweight process results in even slightly higher quality without significantly slowing time to market. In fact it seems that there are almost no unbiased works in these areas. It's difficult to evaluate processes in general, because you're actually comparing teams. Most of the results I've seen have come from someone pushing their process, whether it's agile, or heavyweight, or whatever. There is a lot of anecdotal evidence that indicates faster iteration yields higher quality and more features, though. It's telling that so many of the industry leaders have moved that direction.

Yes, I think browsers need frequent releases in order to be competitive. I don't know if 3 months is the magic number, but it beats the pants off 1 year. Early standards proposals are refined inside the browsers that ship them first. This is one of the reasons HTML has been so successful, and it's what HTML5 has tried to continue: standardize what works. The opposite, where a committee pushes out a standard and everyone then tries to lamely implement it, is a broken process. Early shipping is a prerequisite for well-formed standards. Shipping the first beta standards also gives a browser a competitive edge, because they can then argue in favor of their particular flavor of implementation becoming the standard, and they already have that implemented. Late shipping loses developer mindshare. "Oh, I guess I have to install Chrome to try the latest CSS widgetmagicwaffles."

I'll look into the cleanroom process you're describing, but so far you've made it sound unattractive. It seems like you've been describing a very heavyweight development process. As for TDD and pair programming, I haven't said anything about those. Pair programming is completely orthogonal to process weight, and I think TDD is overhyped. I feel its primary benefit is that it gets a bunch of unit tests written that help avoid regressions.

The problem with working with good specs is that they take a long time to get done, and they inevitably have mistakes that have to be corrected during dev, often major mistakes. While you're building this massive spec, all your engineers are sitting idle, or going brain dead doing spec reviews. I'm in favor of high-level specs, and the larger the project, the more specific the spec, but I don't for a moment believe that a complete spec is a worthwhile thing. Let's be honest, this is the waterfall model: gather requirements -> build insanely large spec -> implement to spec -> verify implementation -> maintain

I'll look into cleanroom, because it's not something I'm very familiar with, but I have my doubts. The "complete spec" model is extremely expensive. It makes sense for avionics. It makes a lot less sense for desktop software.

Well, you're obviously not going to take my word for it, nor would I want you to, so please do go and read up on some of those other ideas. You're quite right that it takes more up-front work in a process like Cleanroom, but some of that can be repaid in practice in quicker testing cycles before shipping and in keeping delays to fix bugs fewer and shorter. But as I said before, please don't get too attached to Cleanroom or any particular figures I've mentioned. They're just examples, to try to demonstrate that some popular assumptions don't always stack up when faced with the facts.