As so often happens, we resist things we don’t understand, in favor of those we do, but if we only take the time to learn…
Geek-dom ahead, you have been warned.
I do almost all of my programming in Perl these days–in fact, for the last decade and a half or so. I’m not interested in getting into a langage war here–I know Perl’s weaknesses as well as its strengths.
Anyway, for the last three years or so, we at “AnteSpam”:http://antespam.com/ have used a Perl script to manage refusing connections from malign hosts and rejecting requests to send mail to non-existent addresses–generally at a sustained rate of several per second, occasionally peaking into dozens per second or more, per server (we have 16 production servers).
Perl has no useful multi-threading, and if we tried to service these requests using one script per connection, we would be screwed–in fact, when we first tried to manage this stuff ourselves, three years ago, our first implementation did just that, and the servers melted; they couldn’t take the load of all of those memory-piggy scripts running at once.
Back in 2007, looking for a solution to this, I found “POE”:http://poe.perl.org/, a mature Perl framework that allows you to do event-driven cooperative-multitasking with asynchronous I/O and various other bells and whistles.
It is very good at what it does, and over the last few years I’ve become very conversant with it. We have several very important pieces of our infrastructure written with it, and they work very, very well.
Still, it has some issues, the biggest of which is that you have to write your code in a very particular style–short routines that queue events that are handled by other routines and things like that–things that mean that if you write code to integrate smoothly with POE, it’s going to look very weird if you try to use it outside the POE framework, and if you write your code outside the POE framework, it’s not going to play well with POE.
As a consequence, there are lots of libraries that don’t play well with POE–you can use them, but you loose the smooth cooperative-multitasking and asynchrony; basically, you lose the ability to handle many things at once, at least with low-latency. And people who aren’t used to POE end up looking at your code in bafflement.
Still, this has been a fine solution for us for years, and I’ve resisted changing it, because every time I’ve tried to work with something else–and here I’m thinking specifically of “AnyEvent”:http://search.cpan.org/dist/AnyEvent/, I just couldn’t see the big benefit. The one time I tried reimplementing something with it, the code got a few lines shorter, but otherwise, it was 6 of one, half-dozen of the other.
And then I had my epiphany.
What I realized is that I could rewrite some code that was duplicated between the “regular” programs, and the high-performance daemons to use AnyEvent in such a way that I could use
the same code for both–when I needed high-peformance cooperative multitasking, I would have it, and when I didn’t need it, the code would look exactly the same.
In effect, I was going to be able to get rid of a huge chunk of duplicative code and use the same code everywhere, transparently. And once I got the basic libraries re-done, there was even more code I was going to be able to merge.
In the space of 24 hours, I rebuilt our low-level LDAP and Memcache access layers to transparently use AnyEvent. I didn’t change any code outside of those libraries, and all tests passed once I was done. That performance-critical daemon I talked about at the beginning–I’ve almost finished rewriting it in the space of a couple of hours.
By making this change, everything is looking cleaner and more straightforward than ever before.
When you have that moment of realization, everything can change.