How can open source scale?

[GoogleCodeExporter]

This is an extremely high-level question... 10,000 feet up, feel free to
ignore :-)...

One thing that strikes me (especially after working at Google) is that the
way of programming we grew up with is going to be extinct soon.  Basically,
we used to write code to either run on a single machine, or to be
client-server (which is really another way of saying "We're writing code to
run on a single machine -- twice!").  Sure, the code might be
multi-threaded, but basically it accesses a single local data store, it has
one memory space, and the algorithms more or less do a single step at a time.

That's all changing.  The big web companies have server farms, and parallel
APIs for programming those farms.  These days, this is where most of the
exciting developments are, for hardware design, language design, data
structure and algorithm design.

But it's hard to learn programming from the ground up on massive clusters,
because most kids in their bedrooms (or dorm rooms, whatever) don't have
access to clusters.  It's like learning to be an orchestra conductor: you
practice in front of the mirror for months and months, and then
occasionally you get ten minutes on the podium in front of an actual
orchestra :-).

(I'm getting to the question, don't rush me...)

Open source has grown spectacularly over the last ten years mainly because
people from different organizations were a) using the same tools and b)
doing similar things with them.  Apache HTTPD flourished because everyone
had to run a web server, and everyone had access to the Apache sources;
there was a lot of pressure toward inter-organizational cooperation.  (We
could list many other examples, obviously.)

Today, people solving the same problems are *not* necessarily using the
same tools.  Instead, each company is rolling its own, designed for its own
peculiar hardware arrangement.

Do you think this is a temporary blip and that standards will emerge soon,
and that therefore we will see the Massively Parallel world suddenly
floodfill with open source tools too?  Or is open source going to be
confined to being the glue between custom (and usually propriately)
components?  Will open source take over the desktop only to discover that
the desktop doesn't matter anymore?

I'm partly asking because you're at Google, and may have unusual insights
on this.

Original issue reported on code.google.com by kfo...@gmail.com on 20 Feb 2008 at 10:15