Scalability - you're doing it wrong! (Or why Ants don't talk)

[Dave Aitel <dave () kof immunityinc com>]

A while back someone I knew was trying to solve some sort of crypto-related
problem. He ended up saying "I can do it quickly and easily if it's under 40
bits" at which point another mathematician friend of mine rolled her eyes
and said "If it's that small, just exhaust."

Anyways, it was an important lesson for me in how what is "large" for some
people is "just exhaust" size for others. If you go into a normal sized shop
building a web application, you can see them struggling furiously to build
clusters around a relational database, managing their Storage Area Network,
and doing all sorts of things essentially exactly wrong. Realistically you
can't scale centralized storage. And you can't scale a relational database.
The very idea of your application already knowing everything it needs to
know is pretty funny. In a scalable system, what's known is unknown.

http://highscalability.com/google-architecture is a good example of an
architecture meant to scale. The key sentence is what BigTable CAN'T do.
Scalability is about what you can't do, not massively cool algorithms that
CAN do something. You can see Google's pitch to Guido Van Rossum with "Yes,
but what's the point in designing a language that runs on a non-scalable
system? Might as well program only for TI calculators." And right now,
pretty much only Google has a scalable system.

In this same sense, of course, ants don't use sound to communicate. The big
shiny book on the subject is reviewed here:
http://www.nytimes.com/2008/11/23/books/review/Jones-t.html . Typically
biologists are dealing with systems they can't understand - they're too
large, or too numerous, or too small, or just too complex, or just too
underfunded. But with superorganisms the size of an ant colony they get a
good picture at how real algorithms deal not in behaviors but in emergent
behaviors. Also the book has many pretty pictures of ants. Ants are a really
large and complex message passing system. Which is totally cool, and
probably why Nico uses Ant iconography so much in his presentations.

Analyzing emergent behaviors is a pain in the rear, but there's a great
video on it here: http://blip.tv/file/2232410/ by David Beazley, who also
wrote PLY, which MOSDEF uses. Debugging is hard, but that's life.

-dave

_______________________________________________
Dailydave mailing list
Dailydave () lists immunitysec com
http://lists.immunitysec.com/mailman/listinfo/dailydave