#code

This is what I expect will become the first of a series of posts where I detail some of the solutions I've arrived at, based on the test programs that I've written to work through the problem.

The situation I was running into was this:

I was writing tests for some parsing code. The output of the parse was a nested data structure. Some (well, many) parts of the data structure were optional. One in particular was giving me a hard time, though---it was a default being set from a dynamic value being passed into the parser. Obviously I had to get that same dynamic value into the test data as well, in order for them to match.

(I will certainly admit that there might be better ways to structure all this, but this is what I have for the moment.)

Anyway, at the time I run a test, I have the test case (which has both the inputs and the expected outputs), and I have the dynamic value, and I needed to insert that dynamic value into the expected outputs before comparing with the actual outputs.

Oh, and did I mention that there was a list of possible outputs, all of which needed to be massaged?

So, I created the following fake data structure that, nonetheless, mirrors the attributes of my test data structure that I care about.

Anyway, the lens library makes these sorts of traversal-with-replacement bits of code quite concise. In this example code, we use over to apply the function that is its second argument (which will either simply pass along an existing Just value, or return a constant value, and wrap either in another Just) to all of the entities that are described by its second argument---that is the containerItem contained in each entry in the collectionItems list.

There is a part of me that suspects that there's an even easier way to do this---it seems like there should be some way for me to narrow the traversal to only pick up items with the value of Nothing (using the _Nothing prism), and then simply setting those items to the default value. But the obvious way of constructing that (adding _Nothing to the traversal, and dropping the fromMaybe in favor of a simple Just value) did not typecheck.

{-# LANGUAGE OverloadedStrings, TemplateHaskell #-}module Main whereimport Control.Lensimport Data.Maybeimport Data.Textdata Item = Item {
  _itemName :: Text} deriving (Show)
makeLenses ''Item
makePrisms ''Item

data Container = Container {
  _containerName :: Text,
  _containerItem :: Maybe Item} deriving (Show)
makeLenses ''Container
makePrisms ''Container

data Collection = Collection {
  _collectionName :: Text,
  _collectionItems :: ![Container]
} deriving (Show)
makeLenses ''Collection
makePrisms ''Collection

collection :: Collectioncollection = Collection "Outermost" [Container "A" (Just (Item "A")),
                                     Container "B" (Nothing),
                                     Container "C" (Just (Item "C")),
                                     Container "D" (Nothing),
                                     Container "E" (Just (Item "E")),
                                     Container "F" (Just (Item "F")),
                                     Container "G" (Just (Item "G"))]

makeDefault :: Collection -> CollectionmakeDefault =  over (collectionItems.each.containerItem) (Just . fromMaybe (Item "Bar"))

I've been trying to cram in so much information so quickly, I'm starting (hah!) to realize that it's not all sticking.

So, what better tool to use than Emacs to solve my problem with not remembering Emacs commands?

My solution is simple---create a cheat-sheet. The great thing about Emacs is that this doesn't have to be a piece of paper, it can be a file that I can maintain in org-mode, just like this blog. In fact, I can also maintain it as a page on the blog. And with the most trivial bit of elisp, I can make sure that I can get to it with no more than two easy-to-remember keystrokes:

(define-key global-map [?\C-h ?\C-h]
  '(lambda ()
     (interactive)
     (view-file "~/org/doyouevenlisp.com/cheatsheet.org")))

No updates to be made, because I spent the entire day working on org2blog. I actually kinda liked it. A lot.

I've now rewritten the XML-RPC back-end to use the new, documented, WordPress API. Right at the moment, this is just running-in-place, but I hope to use the new code to simplifying things more.

So I said yesterday that I really wanted to find a better way to do our by-hand mapping of XML-RPC structs, because doing it by hand---and, specifically, repeating a bunch of information multiple times---was tedious, error-prone and ugly. Here's a smaller struct we're working with, for WPCustomField---smaller, but it's still a bunch of boilerplate:

data WPCustomField = WPCustomField {
  cfId :: String,
  cfKey :: String,
  cfValue :: String
} deriving Show

instance XmlRpcType WPCustomField where
  toValue struct = toValue $ [("id", toValue (cfId struct)),
                              ("key", toValue (cfKey struct)),
                              ("value", toValue (cfValue struct))]
  fromValue v = do
    struct <- fromValue v
    a <- getField "id" struct
    b <- getField "key" struct
    c <- getField "value" struct
    return WPCustomField {
      cfId = a,
      cfKey = b,
      cfValue = c }
  getType _ = TStruct

So I started thinking about it. It seemed obvious to me that I would want to start with a list of tuples---each tuple establishing a mapping from XML-RPC attribute name to accessor function, and put in a list because I was going to need to keep their ordering in order to feed them to the data constructor in the proper order.

So I did this:

cfMapping = [("id", cfId),
             ("key", cfKey),
             ("value", cfValue)];

This seemed simple enough that I didn't bother to write a type declaration, or let ghc-mod do it for me---in which case I might have seen the upcoming problem.

At first I thought my biggest limitation was going to be the fact that I couldn't see a way to transform the fromValue function---while I could map over the entries in cfMapping, I didn't see how I was going to be able to take the resulting list and give it to the WPCustomField data constructor.

Then it hit me---I could fold over the list, and partially apply each value to the Data constructor, so that when we got to the end of the list, we'd have an actual value.

Boy did I feel proud of that solution.fn1

Figuring that I had that problem licked, I decided to first rewrite the toValue function for the WPCustomField structure. I wrote a function that would map over our mapping list, and return the sort of list we were looking for:

mapToValue mapping struct = toValue $ map aListToValue mapping
    where aListToValue (key, accessor) = (key, toValue (accessor struct))

By making sure that the struct was the last thing handed in, I even got to write the new toValue function in a point-free style:

toValue = mapToValue cfMapping

So I compiled it and it ran. "Great!" I thought. "This is going to be easy!" And then I hit WPEnclosure:

data WPEnclosure = WPEnclosure {
  eUrl :: String,
  eLength :: Int,
  eType :: String
} deriving Show

Many of you will see what is wrong immediately. I hinted at the direction of the problem when I mentioned that I didn't bother to write a type signature for the cfMapping list---because once you see it, and look at WPEnclosure, I think it becomes obvious what the problem is:

cfMapping :: [(String, WPCustomField -> String)]

That's right---the eLength field being an Int among String fields means that we've got heterogenous tuple types for the WPEnclosure type. FAIL.

So, for the moment I'm going to put this cleanup on hold, and just proceed with the hand-rolled instances.

of each of those partially applied functions would not be the same, so the accumulator couldn't typecheck. Oh, well, pride goeth before the fall and all that.

Instead, I spent the time I would normally allocate to writing something for "Do you even lisp?" to enhancing org2blog, the software I'm using to manage this blog.

Well, enhancing might be saying a lot---since I've been doing a little WordPress hacking in other contexts, I've become aware of WordPress' new (released with 3.4, so only six months old at this time) "native" XML-RPC API, and I chose to start moving org2blog to use that, and move it away from the hodge-podge of Blogger and MoveableType and metaWeblog APIs that are currently in use.

I hope that over time this will simplify the API, and perhaps result in even better possibilities for interaction---it would, of course, also be great if we could start to abstract away the specifics of a blog's back-end requirements into a well-defined API so we could easily use whatever is the most featureful back-end for a given blog.

Others might see if differently; we'll know when I start sending in merge requests.

I also want to automate the process of storing local copies of articles in a hierarchy that mirrors their permalinks and a couple of other things. That I'm also effectively learning elisp at the same time will make all of that very interesting.

As I said in the previous article, for wp2o2b, the plan is:

So, the task is to download all the articles in my existing sites, reformat them into org-mode files with appropiate metadata for org2blog, store them locally in a hierarchy that mirrors the one on the server.

WordPress implements an XML-RPC interface for accessing your blog programatically. It supports older legacy styles of access (Blogger, MovableType, and metaWeblog), but recommends that for new development you work with their new API which, incidentally, has the nice benefit of being well-documented.

(Much of this new API was introduced in version 3.4, released in June, 2012---so it's only six months old at this point. Normally I would hesitate to depend on something that new, if I cared about wide applicability, but WordPress is one of those things where I think you should be keeping up with releases, if only for security reasons, so I don't perceive it as too much of a limitation.)

The first thing we have to do is implement a data structure for holding a post. Where in a dynamic language, you'd might just get back a big wodge of XML and pick at it as necessary in Haskell, you need to define a data type to hold your results.

So we'll start there.

Working from the definition of a post in the API documentation what we end up with is something like:

data WPPost = WPPost {
  pPostId :: String,
  pPostTitle :: String,
  pPostDate :: CalendarTime,
  pPostDateGmt :: CalendarTime,
  pPostModified :: CalendarTime,
  pPostModifiedGmt :: CalendarTime,
  pPostStatus :: String,
  pPostType :: String,
  pPostFormat :: String,
  pPostName :: String,
  pPostAuthor :: String,
  pPostPassword :: String,
  pPostExcerpt :: String,
  pPostContent :: String,
  pPostParent :: String,
  pPostMimeType :: String,
  pLink :: String,
  pGuid :: String,
  pMenuOrder :: Int,
  pCommentStatus :: String,
  pPingStatus :: String,
  pSticky :: Bool,
  pPostThumbnail :: [WPMediaItem],
  pTerms :: [WPTerm],
  pCustomFields :: [WPCustomField]
} deriving Show

This refers to a few other structs that we've defined---the process is pretty straightforward, so I'm not going to go over it.

Then we need to take this type and give Haskell a way to convert back and forth from it to XML-RPC. The HaXR page on the Haskell wiki link to some example code, and the Network.XmlRpc docs include a little explanation on how to do this.

If your XML-RPC structure has names that are sufficiently unique to map to record names without conflicts, and you're comfortable with Template Haskell, you could just do:

$(asXmlRpcStruct ''WPPost)

However, if you have an aversion to Template Haskell, or (perhaps more likely) you have field names that are generic enough to present significant conflicts (id, or type or some such) you will have to do it by hand by defining an XmlRpcType instance for your constructor. That ends up looking like:

instance XmlRpcType WPPost where
  toValue struct = toValue $ [("post_id", toValue (pPostId struct)),
                              ("post_title", toValue (pPostTitle struct)),
                              ("post_date", toValue (pPostDate struct)),
                              ("post_date_gmt", toValue (pPostDateGmt struct)),
                              ("post_modified", toValue (pPostModified struct)),
                              ("post_modified_gmt", toValue (pPostModifiedGmt struct)),
                              ("post_status", toValue (pPostStatus struct)),
                              ("post_type", toValue (pPostType struct)),
                              ("post_format", toValue (pPostFormat struct)),
                              ("post_name", toValue (pPostName struct)),
                              ("post_author", toValue (pPostAuthor struct)),
                              ("post_password", toValue (pPostPassword struct)),
                              ("post_excerpt", toValue (pPostExcerpt struct)),
                              ("post_content", toValue (pPostContent struct)),
                              ("post_parent", toValue (pPostParent struct)),
                              ("post_mime_type", toValue (pPostMimeType struct)),
                              ("link", toValue (pLink struct)),
                              ("guid", toValue (pGuid struct)),
                              ("menu_order", toValue (pMenuOrder struct)),
                              ("comment_status", toValue (pCommentStatus struct)),
                              ("ping_status", toValue (pPingStatus struct)),
                              ("sticky", toValue (pSticky struct)),
                              ("post_thumbnail", toValue (pPostThumbnail struct)),
                              ("terms", toValue (pTerms struct)),
                              ("custom_fields", toValue (pCustomFields struct))]
  fromValue v = do
    struct <- fromValue v
    a <- getField "post_id" struct
    b <- getField "post_title" struct
    c <- getField "post_date" struct
    d <- getField "post_date_gmt" struct
    e <- getField "post_modified" struct
    f <- getField "post_modified_gmt" struct
    g <- getField "post_status" struct
    h <- getField "post_type" struct
    i <- getField "post_format" struct
    j <- getField "post_name" struct
    k <- getField "post_author" struct
    l <- getField "post_password" struct
    m <- getField "post_excerpt" struct
    n <- getField "post_content" struct
    o <- getField "post_parent" struct
    p <- getField "post_mime_type" struct
    q <- getField "link" struct
    r <- getField "guid" struct
    s <- getField "menu_order" struct
    t <- getField "comment_status" struct
    u <- getField "ping_status" struct
    v <- getField "sticky" struct
    w <- getField "post_thumbnail" struct
    x <- getField "terms" struct
    y <- getField "custom_fields" struct
    return WPPost {
      pPostId = a,
      pPostTitle = b,
      pPostDate = c,
      pPostDateGmt = d,
      pPostModified = e,
      pPostModifiedGmt = f,
      pPostStatus = g,
      pPostType = h,
      pPostFormat = i,
      pPostName = j,
      pPostAuthor = k,
      pPostPassword = l,
      pPostExcerpt = m,
      pPostContent = n,
      pPostParent = o,
      pPostMimeType = p,
      pLink = q,
      pGuid = r,
      pMenuOrder = s,
      pCommentStatus = t,
      pPingStatus = u,
      pSticky = v,
      pPostThumbnail = w,
      pTerms = x,
      pCustomFields = y }
  getType _ = TStruct

Yeah, so that's the obvious way to do it. And boy is it tedious---I need to figure out a better way to make this happen. because that's a lot of pointless boilerplate.

It seems to me that I should somehow be able to define a small data structure and then pull the necessary bits out just once, rather than having to repeat everything at least twice. I guess that's the purpose that the Template Haskell code serves, but I need more power.

Oh, well, it's done for the moment.

I want to emphasize here that at this point, I'm just trying to get things done. I am intrigued by the theoretical underpinnings of Haskell (although my understanding of most of them is...shallow at the very least), but I'm also a working programmer---I need to be able to be productive. I want the benefits that I think Haskell has to provide---static typing to keep me from making as many dumb mistakes, good performance---but I have to be able to produce actual code for those things to be worth anything.

At the same time, I recognize that what I've just done probably represents a small chunk of technical debt. I'd love to learn enough to be able to pay it off.

Debian GNU/Linux turns 19 today.

I estimate that I did my first install some time in late 1995, perhaps early 1996. I haven't really used anything else as my day-in-day-out OS since. I've never had a Mac of any stripe, and haven't used Windows with any frequency other than for World of Warcraft since '99.

I can pin my first contribution to Debian with far more accuracy: September 3, 1996. That's the date on the first Debian changelog entry in the libwww-perl package, which was, I believe, the first package I ever made. It still exists in Debian and Ubuntu (and other derivatives) and if you have it installed, you can look at /usr/share/doc/libwww-perl/changelog.Debian.gz, and right down there at the very end, you'll find my grubby little fingerprints.

Sadly, things quickly went downhill---I am, to some extent, to blame for the fucked-up naming convention (with its poorly-sorting use of a -perl suffix) of every Perl library package in Debian, and probably by extension, the similar poor choices in the -java and -cil groups. Even the PHP guys were smarter.

As I remember it, having packaged libwww-perl (which is the actual name of the package as it exists on CPAN, so I just used that as the package name), I discovered that for some things---FTP support, I believe---it required the libnet package, which provides a lot of Net::* modules. But when I announced that I was going to package it someone (I want to finger Rob Browning, but cannot in be certain, and I don't know if the Debian archives go back that far, and can't be bothered to check, really) said they were about to package the libnet C library, which would conflict, so maybe I could call it libnet-perl, like libwww-perl, which I did. And then the next thing I packaged I ended up calling lib<whatever>;-perl for no good reason, and it all went to hell.

Bleargh.

Almost 16 years later, we have ~ 3000 Perl packages in the repository, all with that damned -perl suffix. So, um, sorry.

My other big accomplishment of any note, I think, was to finally get the 64-bit Alpha port to a self-sustaining state. This started with a bootstrap that someone had done from a set of RedHat binaries before, and gradually pulling along various bits of the system until we had a self-hosting system.

Along the way, I was responsible for another unfortunate bit of "engineering"---libc6.1. Again, I don't think I was totally alone in putting this forth (David, David...um, I forget his last name, and the libc6 changelog.Debian doesn't go back that far), but it probably would have been better to bite the bullet and avoid all the gymnastics it required.

As I remember it (this was, err, '97? So I may have some details wrong), RedHat had pushed their first Alpha release using a libc with a SONAME of 6, based on a pre-release glibc-2...and then the 64-bit ABI was changed when glibc-2 was released. Our versioning tools for shared libraries were much more primitive at the time, and we wanted to keep compatibility with RedHat, since that's where a lot of the heavy duty engineering was going, so we had to follow them in changing the library SONAME to 6.1. This entailed a lot of churn at the time, most of which I've blocked out. I did a lot of mechanical patches on a lot of packages.

I did spent a lot of time doing fixes for 64-bit-isms in various packages, and I can remember the flush of pride I had when Alan Cox mentioned that he'd gotten a bunch of 64-bit fixes as well as the conversion of the mh mail client to use an ELF shared library from the Debian package, because that was my work.

My time as an active Debian contributor was not always a smooth one---I wasn't always as attentive about keeping things up to date or doing triage on bugs as I could have been. I think I finally formally recognized that I wasn't able to keep up around 2002, which was probably a couple of years later than everyone else had realized it.

Ironically, I probably maintain more packages now, for our internal company purposes at Ironic Design, than I ever did as a developer; the tools have made the maintenance at least of the sorts of packages I do (libraries and simple applications) incredibly easy.

I maintain ~ 20 production Debian servers, with a handful of dev servers and a couple of home systems, including the laptop I'm writing this on, with basically no problems on a day to day basis. It has its warts, but I have boxes that have been continuously upgraded over a span of half a dozen years with no appreciable problems. I am able to be productive and use the environment happily. I remember the rough spots at the beginning (the move from a.out to ELF, for instance), but what Debian provides now always surprises and delights me.

So kudos to those Debian maintainers, former and current, who have contributed to such a great software system.

Someone created a bestiary of List Code Typography and my immediate gut reaction upon seeing the earliest possible examples was that the only thing that could ever have been more confusing than all the parenthesis in the world was if you had to put commas in-between every goddamned thing.

The language I'm currently learning, Haskell, has its roots in the Lambda Calculus as well, but goes entirely in the other direction--no punctuation at all.

I have been programming primarily--for long stretches, almost exclusively--in Perl for the last 17 years or so. I seem to remember starting to use it around mid-1995, with 5.001--during that long, awkward time between when Perl 5 came out and when the 2nd edition of Programming Perl finally arrived in late 1996.

I've kept with it because I'm fluent in it, I am productive in it, and at this point, I can make it do some fairly absurd things (ask me about writing event-driven servers in Perl, I dare you). In fact, I like the language. I understand the complaints people have about it, but the subset in which I write these days is pretty clear while remaining concise and expressive, and the ecosystem that exists around it is simply unparalleled.

Nonetheless, I think the time has come to move on. The downsides of the language--speed, largely, and lack of good language support for expressing things like parallelism--have started to wear at me. I'm tired of the hoops I have to jump through to do the things I want to do.

So for the last 18 months or so, I've been reading a lot about a number of languages. I don't think I've rejected any out of hand except PHP, though I certainly have some biases. For instance, I am looking for a mainstream language--something like IO, though interesting, does not qualify.

But mainstream isn't everything--I want something that is going to open up new options, that's going to be fun to get immersed in; so I'm not considering things like Ruby or Python because for the most part I think they recapitulate most of the problems I have with Perl (speed, concurrency support) just with different syntax.

In the end, I came down to three options. Node.js, Scala and Haskell. I find that as I've been sitting with the question for the last couple of weeks, though, I've stopped thinking about Node.js as a real option. Though it's fast, and it's got a great ecosystem of software surrounding it, raw event-driven programming doesn't really engage me any more. It was fun for the first year or two I did it, but the idea of moving to an environment where Everything Is A Callback leaves me cold.

So it's down to Scala and Haskell, I think.

As a consequence, I've spent the last week reading Programming in Scala: A Comprehensive Step-by-Step Guide, 2nd Edition by Odersky, Spoon and Venners, and before that I got most of the way through Learn You a Haskell for Great Good by Miran Lipovaca (though I'm going to go back through it now and finish it).

I intend, over the next couple of weeks, to post about my experiences working on using each to write a couple of short (but non-trivial) programs with both of them--ones that, incidentally, I have implemented in Perl already, so I can do a real comparison of code.

Dear lazyweb,

Perhaps this was just an oddity of my printer, but here's what I had to do to get a good print of the org-mode refcard onto Letter paper. From within the org-mode sources, I did:

make doc/orgcard_letter.tex
cd doc
tex orgcard_letter.tex
dvips -O "-.5in,.25in" -t letter -t landscape orgcard_letter.dvi

This got me a .ps file that seemed well-centered on the page. To print it, I did:

ps2pdf14 orgcard_letter.ps
evince orgcard_letter.pdf (print, duplex flipped on the short side)

I probably could have done (using lp directly, but since I was also using evince to eyeball the layout first, it was easiest to do it from there):

lp -o sides=two-sided-short-edge orgcard_letter.ps

A little over two years ago, I wrote a post about my view of the web server software landscape under Linux, concluding with how I'd ended up sticking with Apache despite having tried most of the other reasonable candidates because they all seemed lacking.

It's interesting in part because I never recorded when I moved that server from Apache to Cherokee (which I had tried to poor results, as noted in the post), which would have been not too very long after I wrote that post. Oh, well.

Anyway, everything ran alright on Cherokee for 18 months or so, but Cherokee wouldn't let me do per-client bandwidth throttling, which I really needed as Chet's blog was getting hammered mercilessly by spammers, and I couldn't figure out any other way to slow them down.

So I switched to nginx, which by now I have a fair amount of experience with, using it for http and imap proxying as well as serving fastcgi apps for other projects. If it weren't for the decrepit software that Chet and I have been using for blogging for the last couple or three years, everything would have been great (Movable Type--and even its follow-on project Open Melody--has never modernized its low-level infrastructure to allow good support of FastCGI; they'll tell you they have, but just ask them if you can do XML-RPC--the foundation for remote posting--and watch their reaction).

Still, I've made do with a FastCGI shim for Movable Type, and then decided to start looking at WordPress, which I've already converted to, and which I think we will get Chet converted to shortly.

Incidentally, I became even happier about having moved to nginx about a week later, when I ran across a blog post from Cherokee's author, posting a link to a performance comparison that showed Cherokee beating everything else.

Normally, I would just say "great" and move on, but knowing a little bit about nginx, I was surprised at the version being used, as it seemed a little old. And so I did some more research, and found that the article was comparing an up-to-date version of Cherokee to much older versions of other servers, in some cases versions from branches that had long been declared obsolete. Still, it's not the fault of Cherokee's author someone at a magazine did a crappy test.

However, when I presented my findings, and asked him to acknowledge the issue, and call for the author of the article to do better:

Alvaro, I understand that it is nice to see your software perform well against its competition, but I would encourage you to dissociate yourself from this comparison, or at least take it upon yourself to point out that there were some things that may have left your competitors at a disadvantage.

he suggested that he had made any caveats he needed when he said "you shouldn't expect an extensive, in-depth benchmark" from the "very well written article", though he did note that "the benchmark results are still fairly representative IMHO.", and when pressed, said that he didn't think the results would have changed with more recent versions of the other software.

It took a while to get the taste out of my mouth. I guess I still haven't, given that I'm posting this.

Anyway, up with nginx.

I've spent about the last three weeks converting much of the infrastructure code for AnteSpam to use AnyEvent.

One of the small bits of fallout from using AnyEvent is that we now have a large number of anonymous code references as callbacks, and in our logging code, these all have the same name: __ANON__.

This makes debugging output a little less useful.

In browsing some code in AnyEvent::SMTP, I happened across the trick of locally setting the __ANON__ typeglob to the name you want to use used in stacktraces and the like:

my $var = sub { local *__ANON__ = 'What::ever::you::want'; ... };

So, this is kinda ugly, and I couldn't find any official documentation of it, so I went looking around, and found Sub::Name, which is a module to make this a little more palatable. Now we can do:

my $var = subname 'What::ever::you::want' => sub { ... };

Still perhaps not beautiful, but not totally covered in warts, either.

Now to go retrofit this onto all of our code...

As so often happens, we resist things we don't understand, in favor of those we do, but if we only take the time to learn...

Geek-dom ahead, you have been warned.

I do almost all of my programming in Perl these days--in fact, for the last decade and a half or so. I'm not interested in getting into a langage war here--I know Perl's weaknesses as well as its strengths.

Anyway, for the last three years or so, we at AnteSpam have used a Perl script to manage refusing connections from malign hosts and rejecting requests to send mail to non-existent addresses--generally at a sustained rate of several per second, occasionally peaking into dozens per second or more, per server (we have 16 production servers).

Perl has no useful multi-threading, and if we tried to service these requests using one script per connection, we would be screwed--in fact, when we first tried to manage this stuff ourselves, three years ago, our first implementation did just that, and the servers melted; they couldn't take the load of all of those memory-piggy scripts running at once.

Back in 2007, looking for a solution to this, I found POE, a mature Perl framework that allows you to do event-driven cooperative-multitasking with asynchronous I/O and various other bells and whistles.

It is very good at what it does, and over the last few years I've become very conversant with it. We have several very important pieces of our infrastructure written with it, and they work very, very well.

Still, it has some issues, the biggest of which is that you have to write your code in a very particular style--short routines that queue events that are handled by other routines and things like that--things that mean that if you write code to integrate smoothly with POE, it's going to look very weird if you try to use it outside the POE framework, and if you write your code outside the POE framework, it's not going to play well with POE.

As a consequence, there are lots of libraries that don't play well with POE--you can use them, but you loose the smooth cooperative-multitasking and asynchrony; basically, you lose the ability to handle many things at once, at least with low-latency. And people who aren't used to POE end up looking at your code in bafflement.

Still, this has been a fine solution for us for years, and I've resisted changing it, because every time I've tried to work with something else--and here I'm thinking specifically of AnyEvent, I just couldn't see the big benefit. The one time I tried reimplementing something with it, the code got a few lines shorter, but otherwise, it was 6 of one, half-dozen of the other.

And then I had my epiphany.

What I realized is that I could rewrite some code that was duplicated between the "regular" programs, and the high-performance daemons to use AnyEvent in such a way that I could use
the same code for both--when I needed high-peformance cooperative multitasking, I would have it, and when I didn't need it, the code would look exactly the same.

In effect, I was going to be able to get rid of a huge chunk of duplicative code and use the same code everywhere, transparently. And once I got the basic libraries re-done, there was even more code I was going to be able to merge.

In the space of 24 hours, I rebuilt our low-level LDAP and Memcache access layers to transparently use AnyEvent. I didn't change any code outside of those libraries, and all tests passed once I was done. That performance-critical daemon I talked about at the beginning--I've almost finished rewriting it in the space of a couple of hours.

By making this change, everything is looking cleaner and more straightforward than ever before.

When you have that moment of realization, everything can change.

When someone undertakes something for fun, or out of passion or deep commitment, the end result is often, I think, more reflective of them personally.

This is generally true of Free Software, and in the Free Software universe, I think this is sometimes even more true of documentation--you're not obligated to write it, no one's paying you, few people enjoy writing docs, so if you're doing it at all, it's because you believe.

So the writers' personalities and convictions show through just a little bit more, Like this bit from Dave Rolsky with whom I am slightly acquainted. Contained within Moose::Cookbook::Basics::Recipe10 I ran across this gem:

Our Human class uses operator overloading to allow us to "add" two humans together and produce a child. Our implementation does require that the two objects be of opposite genders. Remember, we're talking about biological reproduction, not marriage.

Brilliant.

So there has always been a multiplicity of web server software for Unix/Linux.

It certainly feels like I have, at some point or another, played with all of them. And I keep coming back to apache, which I've been using since 1995, when I first became responsible for running a web server (this site, if you care).

Incidentally: Holy crap, 14 years.

Anyway, as I stare around the unix landscape, I see four general-purpose web servers with some mind-share: apache, lighttpd, cherokee and nginx. Yes, there are others, but they are niche players, or they are not general purpose. So here's my issues:

lighttpd

For the last couple of years I've run wiki.mallet-assembly.org on a box that was running lighttpd. And, honestly, I've not really had anything to complain about; it was stable, it was fast enough, etc. But if I wanted to run fastcgi programs as some user other than www-data (better for security), I had to run them as their own daemons. This isn't the end of the world. What ultimately made me decide against it was that lighttpd has spent much of the last two years in perpetual rewrite mode.

cherokee

I've been paying attention to Cherokee for the last year or so. It was looking like an interesting alternative to lighttpd. And then I tried it. Just as was always the case with the Netscape Enterprise Server/iPlanet software that I hated when I was at Dorado, the only documented interface for configuration was web-based. This isn't the end of the world. But when it mysteriously broke comments for no discernable reason--and we're using straight CGI, the simplest possible option for it--that got it the boot. And it's error log? Useless.

nginx

Nginx is great at what it does. Seriously. Couldn't live without it. But really, it's a proxy that happens to have also been taught how to speak FastCGI (and IMAP and SMTP and various other things--it really is great), and as a consequence it doesn't so some important things like, well, CGI. I am using it for wiki.mallet-assembly.org right now, because it's ultra-light and I'm running mediawiki under FastCGI there, so it all works out, but I'm probably going to move it to apache before too long because...well, who wants to have to keep track of two different packages.

apache

Big. Complicated, with too many options to keep track of. One of the more annoying configuration syntaxes around (Fake html tags to denote sections? Really?). But dammit, it works, even when you ask it to take care of spawning fastcgi processes as another user. And it's not that baroque. And even when it is (mod_rewrite, I'm looking at you), it's still better documented than any of the other options. And most of the unix-oriented web software just pretty much assumes you're going to be using it.

There's really just not any competition.

Now don't get me wrong--I would be disappointed if suddenly everyone abandoned all of their other systems. An Apache monoculture would benefit no one. To those working on the other systems, well, I'm gonna keep looking at them and seeing how they evolve. If nginx sprouted simple CGI support (none of this "write your own FastCGI server process that would proxy the CGI scripts" stuff), I would almost certainly move to that.

But for the moment, Apache it is.

OK, so the thing about today's hackers is that they're often strikingly funny.

So even if you don't care about javascript or emacs, much less javascript and emacs, you should go read Steve Yegge's discussion of implementing javascript in emacs lisp because all of the digressions and other silliness are sure to make you laugh.

I mean, here's Steve discussing the name:

In that blog I mentioned I was working nights part-time (among other things) on a JavaScript interpreter for Emacs, written entirely in Emacs Lisp. I also said I didn't have a name for it. A commenter named Andrew Barry suggested that I should not call it Ejacs, and the name stuck.

Via (who has his own history with Emacs)

I don't think this is the first time I've quoted Rusty Russell:

I think Willy did it because this is for printk. It makes more sense than
everyone opencoding an -ENOMEM handler, which will have to be replaced by
some mildly amusing string like "I want to printk but I have no memory!".
Next think/sic/ you know 70% of the kernel will be bad limericks as everyone tries
to one-up each other.

Steve Gibson, of SpinRite fame has come up with sort of a super-simple variation on the little RSA keyfobs where you instead carry around a little business card that you can print up yourself that has a bunch of possible second-factor entries you can use for auth.

The part that makes me laugh a bit is that it is--as you might expect if you remember the ads for SpinRite back in the day--a Windows .DLL coded in assembly.

John Graham-Cumming has a C implementation of the scheme. He also has a three-letter domain name. Coincidence?

Funny enough, it was a fix for a bug in the software (gnome-blog I am using to write this post.

It was mostly a matter of figuring out what was failing--a GConf interface was failing when the app tried to store an integer--and then searching around Mark Pilgrim excellent (and freely-available) Dive Into Python.

If I was really together, I'd post a patch, but I didn't back up the original.

But apparently it is. Well, long-standing, at least, since it is apparently no longer a problem

Via

Still Catalyst, but this time with a database back-end that also does asset (aka binary files) management.

For about 30 seconds during development the software actually had 100% test coverage.

This is one of those times when I hope whatever pathetic amount of google-juice I have can aid others.

Googling around, I have found many oblique references to running horde/imp/turba/etc. using fastcgi, but very few specifics, and what specifics I found are mostly about using lighttpd (which is a fine server, but we're not using it yet), and those for apache seemed wrong, or at least way over-complicated.

For maximum applicability in today's world, I'm going to do this using mod_fcgid under apache2, since mod_fastcgi is basically dead.

I'm also not including the security bits (denying access to config and lib directories and so forth) because they're mostly boilerplate and you can get them from the Horde documentation anyway.

So, without further ado, here's what you do:

<virtualhost *>
  ServerName mail.example.com
  DirectoryIndex index.php
  DocumentRoot /usr/share/horde3
  <files "*.php">
    SetHandler fcgid-script
    FCGIWrapper /usr/lib/cgi-bin/php5 .php
  </files>
</virtualhost>

Easy, huh? I've seen all sorts of baroque suggestions involving setting Action directives and AddHandler stuff and all sorts of things, but this simple invocation works just fine.

Share and Enjoy!

You know, IE7 looks like a reasonable browser, but it's not. To prove this it's not even necessary to resort to something like CSS compliance, where no one else gets it entirely right either. It doesn't even get HTTP right. That is, when confronted with a perfectly legitimate 204 status code if fucks up. Spectacularly.

Now why would someone be using a 204 status code? Well let's look at the language in the standard:

bq.. 10.2.5 204 No Content

The server has fulfilled the request but does not need to return an entity-body, and might want to return updated metainformation. The response MAY include new or updated metainformation in the form of entity-headers, which if present SHOULD be associated with the requested variant.

If the client is a user agent, it SHOULD NOT change its document view from that which caused the request to be sent. This response is primarily intended to allow input for actions to take place without causing a change to the user agent's active document view, although any new or updated metainformation SHOULD be applied to the document currently in the user agent's active view.

The 204 response MUST NOT include a message-body, and thus is always terminated by the first empty line after the header fields.

Needless to say, if you're doing ajax-style processing, and, say, letting people delete stuff out of a list, then having a code that let's you say, "Yeah, we succeeded, there's nothing more for us to say, nor any need for you to redisplay or anything, though." is pretty damned useful.

Would you like to know what IE does to the 204 response code? It changes it to 1223.

For those not inculcated in the minutia of HTTP, all response codes are three digits, with the first digit indicating the general category of response (1 is informational, 2 is success, 3 is redirection, 4 is for client errors, 5 is for server errors), and the additional digits giving more specific information. 1223 isn't even on the map here.

This has been a problem for at least five months, since that's when a bug was filed in the dojo toolkit's trac about it.

How can Microsoft be taken seriously? All that money, all those programmers, and they can't do better than this?

Twitter is apparently finding that Rails doesn't do massive scaling well, at least not the way that all the books will tell you to write stuff for it.

That doesn't really surprise me. Making applications that scale well is hard. I've done it twice (though only one of those is a web app), and in both instances, what I found was a need to be able to muck around with the lowest-level code to be able to create app-specific speedups--whether that was writing my own hand-tuned demented-but-fast SQL or being able to back stuff up against memcache that most people wouldn't think to put in there, like mutexes (and yes, I know it's not a reliable storage medium, but given the rate at which it fails, we were willing to face potential issues).

And, honestly, I think Catalyst brings most of the great stuff about Rails while letting you get to the bare-metal if/when you need to.

Via

There is no way it can be a good thing to have an emulation of the Amiga written in javascript.

via

So, slashdot had a story about the new Falcon storage engine for MySQL. I don't care for MySQL for a number of reasons, but some--though not all--could be alleviated with a better storage back-end. So I cruised over to check out the Falcon feature-set.

Funny enough, with the exception of the next to the last point--which is a potentially non-trivial point, I admit--this is all stuff that PostgreSQL has had for years.

One day, people are going to realize that MySQL has been playing catch-up for the last few years.  The amount of effort people have to do to work around MySQL's long-standing issues with concurrency and lack of ACID-compliance in its default configuration, and it's lack of good performance in the configurations that do have those characteristics always amazes me, especially when PostgreSQL has Simply Worked for a long, long time.

So, one of my consistent gigs is working on AnteSpam for Ironic Design. We use SpamAssassin as our engine, but we (well, mostly I) have built a bunch of infrastructure around it that allows us to do high-volume, redundant, high-availability deployment for domain customers, present held mail through a web interface, so on and so forth.

For the last 18 months or so, I've been embarked on a big rewrite, taking everything we've learned from having this system in production for the last three-and-a-half years and synthesizing it into a system that will run more accurately, more smoothly and with less maintenance and upkeep.

The rewrite is vastly superior in any number of ways. It increases performance by finding clever ways to avoid doing unnecessary work. It doesn't move data around unnecessarily. The interface is largely ajax-based, making for better responsiveness. It changes its message handling to make it possible to do statistical learning on pristine copies of messages for better accuracy. It changes the way it represented various entities in terms of the data they stored for better accuracy. Really, tons of things are different, and it is truly kick-ass.

And about two weeks ago, I finally admitted that there was no way that the rewrite was going to see the light of day.

You see, the problem with starting from a clean slate is simply that of, "So, how do you make a transition." And it was becoming increasingly clear that making a transition was going to be very, very hard. Nigh on impossible to manage in any reasonable time-frame. And some of the changes way down at the core were ones that we had no way to test under real loads, so it would only be as we started making transitions that we would know if they were going to work. So, really, I started having some real concerns months ago. And they just kept building and building.

And so I had my Fred Brooks moment, where I finally was able to admit that continuing down this path was going to be a mistake. Instead, the rewrite will, effectively, be declared a research project, and I'll spend the next however long incorporating ideas--and probably even code--from it into the production system. So the grand new features will be introduced incrementally (and even doing that is scary enough in some ways), and we're much less likely to end up getting ourselves up the creek without a paddle.

I've been sleeping a bit easier, even though there's a fair bit of pressure to get this stuff rolled into the existing system. It's just much more doable.

I've been stuck in javascript hell for the last couple of weeks, and Prototype has been a large part of keeping me sane. That people then write all sorts of extensions for it, including one for using data structures to generate HTML is just great.

Ruby On Rails 1.1 has been released.

Although I'm not using it now--my current project has too much code that's always going to be Perl for me to consider switching languages--it's something I'd seriously consider using for the future. It does seem silly that they just came out with a book about using it and then introduced a major upgrade, though.

Or, more accurately, taught about javascript at the ETech conference. And he has very graciously made both his slides and his notes available from his blog.

These are mostly oriented towards people who already know how to program, but haven't taken JavaScript seriously. I'm definitely in that camp, and I found his notes to be a very clear, consise introduction to some of JS's more advanced programming features--some of which I'd been exposed to already because of my spelunking around AJAX code, but I'd just been inferring their use rather than knowing exactly what was going on.

Ingy has produced a javascript-based templating engine that can actually use templates intended for the Perl-based Template Toolkit. He talks a little bit about it on his blog. The scary part is that this may have just made it much more reasonable for me to support both an Ajax-based and a "conventional" implementation of the AnteSpam front-end; no more having to maintain two ways of presenting data, etc.

The default theme that RockBox uses is much less pretty than that of the default iRiver firmware, but as you might have guessed from the way I said that, RockBox is themeable, and the non-default themes are at least as pretty as the iRiver firmware.

In other words, RockBox, err, rocks, in every conceivable way.

So, today I installed RockBox on my iRiver IHP-140.

It's not as pretty as the original firmware (which, incidentally, I can still get to because, well, the RockBox guys are pretty smart), but it has two feature that I always wished for that the original firmware never had--1) the ability to use .m3u playlists that also work under mpd (that is, ones that use forward slashes, as $DEITY intended), and 2) the ability to create playlists on the fly by queueing up tracks interactively.

I would seriously recommend it to anyone who has one of these players, and once the iPod port is to a reasonable point, I'd push people to use it on those too--you get access to actual free formats, like OGG and FLAC, instead of being tied to MP3 and AAC.

The IE 7 team is calling for people to stop using hacks to work around issues with IE.

It seems to me that the problem is that people with actual websites they want to behave have to use the hacks until IE 7 actually, you know, ships. Even on this site, the overwhelming majority of browser-based hits are still for a version of IE that has all these defects.

I guess at a certain level, I'm only noting this in sort of a thumbing-my-nose-at-MySQL way, but the sale to Oracle of the company that creates the only transaction-safe storage back-end with referential integrity available for MySQL has some real implications for MySQL. I, of course, am largely unaffected because, well, I don't use MySQL.

Seen on linux-kernel:

bq.. From: Ahmad Reza Cheraghi
Subject: Why no XML in the Kernel?
To: linux-kernel
Date: Sun, 2 Oct 2005 02:41:42 -0700 (PDT)

Can somebody tell me why the Kernel-Development dont
wanne have XML is being used in the Kernel??

Regards

Ahmad Reza Cheraghi

How to become a hacker? Hack

So yesterday I found myself in an unfortunate situation--I had just spent several days doing a significant revamp and cleanup of a clients LDAP tree (to better support multiple-domain email handling, mostly, but it had accumulated several years of cruft) when the client called me in a tizzy because their WebDAV access--necessary to modify a number of their websites--had stopped working.

Well, it turns out that Adobe GoLive! URI-encodes any (presumably, I didn't check) non-alphabetic characters in the username it sends over for authentication. But these usernames aren't decoded before they're handed to mod-auth-ldap, so the lookup fails because there is no record for 'foo%40example.com'.

If I were dealing with traditional vendors here, I expect I would have spent quite some time on the phone as everyone involved pointed fingers at one another--the web server vendor saying that GoLive! shouldn't URI-encode the usernames, Adobe saying that the web server should decode them, the web server saying that the LDAP server should know how to decode them, etc., etc. Round and round.

But I'm not dealing with traditional vendors (at least, not on the server side), I'm dealing with Free Software. Which means I was able to download the source to mod-auth-ldap and add the following patch:

-- libapache-auth-ldap-1.6.0.orig/auth_ldap.c
+++ libapache-auth-ldap-1.6.0/auth_ldap.c
@@ -404,7 +405,12 @@
LDAP filter metachars are escaped.
*/
filtbuf_end = filtbuf + FILTER_LENGTH - 1;
-  for (p = r->connection->user, q=filtbuf + strlen(filtbuf);
+
+  /* fscking Go Live uri-encodes the usernames, which screws up lookups */
+  char *decoded_user = ap_pstrdup (r->pool, r->connection->user);
+  ap_unescape_url (decoded_user);
+
+  for (p = decoded_user, q=filtbuf + strlen(filtbuf);
*p && q < filtbuf_end; *q++ = *p++) {
if (strchr(&quot;*()\\&quot;, *p) != NULL) {
*q++ = '\\';

And everything works just fine, thanks.

That is to say, when I'm working on web stuff, I think almost exclusively in terms of what I can do on the server side--I have been known to use JavaScript to do simple pre-submission validation of forms, but that's about as far as I go.

However, there's an interesting article on how to have your ajax-enabled site degrade gracefully that uses the incredibly sensible strategy of shipping all your documents as HTML that works, if mundanely (what I'm used to doing) and then, if it's available, using javascript to make them full-on-robot-chubby ajax-enabled masterpieces. You won't even enable the ajax capabilities unless that particular promise can be fulfilled.

I'm sure this is old hat, and I'm terribly late to the party, but damn it just seems brilliant.

Anyone worth their salt as a programmer will tell you that programming by cut-and-paste is always, always, always a mistake. You might do it for expedience, because reworking whatever you're cutting-and-pasting to be more generic might take longer than you have to deliver your result, but there is never a situation where it's a good thing.

But the subtext language has a demo that posits the question what if your language was built to handle all the issues for you?.

I don't think I'll be programming in in any time soon, but its always interesting when a new idea comes around.

I guess first I should make the observation that I don't know who Elijah is other than some random Gnome hacker.

However, the last couple of days in Gnome-land has involved huge, horrendous amounts of dumping on someone named Eugenia for saying some unconsidered and unkind things in the most public way possible. Lots and lots of dumping. I mean tons. It certainly seems like everyone on Planet Gnome has made a comment, and though most of them have been minimally civil--no shouted obscenities, no ad hominiem attacks--I think it's fair to say most of them feel unfairly attacked.

Elijah, though, takes the time to try and figure out why it all happened.

Sure, it's all supposition, but it's refreshing to see someone--however alone they may be--trying to step back and understand the other side's point of view, however misguided it may actually be. I've witnessed a lot of Debian flame-wars (it looks like another is heating up right now) that quickly sink to the all-heat-and-no-light level.

which is an interesting bit in itself, Jakub Steiner drops a couple of links to some resources on writing (and, for that matter, why to write) functional specifications, one from Joel Spolsky and one much more elaborate one that really leads you by the nose.

This all seems especially germane to me right now since I'm going through the throes of writing some specs for the great rewrite of AnteSpam.

It's the hot new thing, even cooler than Ruby on Rails.

This is apparently all over geek circles today, but I got it from Miguel.

Researchers in Israel have developed colorization techniques that are almost freakish in their ability to produce natural-looking results using an incredibly simple-seeming marking-up of the original image.

Needless to say, they have a web page devoted to it.

I remember when it was first freed, late last summer--right about the time I was starting in DC--and it was the subject of much enthusiasm and lots of "wow, this is just what we've been waiting for" posts.

It would appear, though, that the bloom is off the rose, and all we're left with are the thorns.

Over the last few years, it has been a presumption that when I work on a project in Perl, I will use the standard Perl tools-- ExtUtils::MakeMaker and, later, Module::Build --for managing the Perl library code I write.

But yesterday, for the first time, I looked at extending Module::Build to do more than just the stock actions. And you know what, it was easy.

Now the specific issue I was running up against was that I needed to insure that the database I was running my tests against was installed and clean. I had been using a Makefile, but that was a hack--for instance, I wasn't actually checking the presence of the database or anything, I was looking for a file I wrote when I created the database. I probably could have made make check for the actual database, but it's imperative, rather than procedural, style makes this kind of ugly.

Also, I wanted a clean instance of this database before I did a test run of the conversion utility (this is all work on a heavily revised AnteSpam, and we're moving from keeping config info in ldap to putting it in a replicated PostgreSQL database). And I wanted a clean instance of this database before I ran the PostgreSQL Autodoc tool to generate a nice diagram and DocBook documentation of the structure.

Oh, and I got so frigging tired of SQL's spectacular verbosity ( badly exacerbated by the fact that I was commenting on most of the structures so the information would show up in the DocBook documentation) that I wrote a simple preprocessor--so I had to make sure that was run if necessary before creating the database.

Oh, and I wanted to build the documentation automatically. And I kept forgetting to run the Build script with the environment set properly for the database, so I wanted that to be handled easily.

So, I made a file, Build.pm, which sits right alongside Build.PL, and subclasses Module::Build. And to that file I added a function (admittedly very simplistic, and, as a result, somewhat overenthusiastic) to drop and recreate the database:

sub create_db {
    my $self = shift;

    # Get the database name
    my $database = $self->args ("database");

    # Drop the database if it already exists
    $self->do_system (qq{dropdb $database}) if ($self->do_system (qq{psql -l | egrep -q $database}));

    # Create the database
    $self->do_system (qq{createdb $database});

    # Make sure the schema is up-to-date
    $self->dispatch ("ddlpp");

    # Load up the schema and initial data
    $self->do_system (qq{psql -q -f antespam.sql});
};

There are several cool things here. First, you can look at, at run-time, arguments that were given to the script when it was created. So you can do:

perl Build.PL database=foo

and when you actually invoke the resulting build script, the bits you write can look for a database argument, and use what was set initially. The rest of it should be fairly obvious--yes, I'm just shelling out to psql rather than doing it all in DBI--except for the dispatch call. You see, you can add additional actions to your script. In this case, I added an action called ddlpp (for DDL pre-processor) to build the sql from my data definition file. It's short, just:

sub ACTION_ddlpp {
    my $self = shift;
    $self->do_system (qq{ddlpp antespam.dp antespam.sql}) unless ($self->up_to_date ("antespam.dp", "antespam.sql"));
};

You'll notice, though, that it will only run ddlpp again if the .dp file is newer than the .sql file. That's cool.

Anyway, I also overrode the standard test action, to make sure the database is created:

sub ACTION_test {
    my $self = shift;

    # Set up database access
    local $ENV{PGDATABASE} = $self->args ("database");
    local $ENV{PGHOST} = $self->args ("host");
    local $ENV{PGPASSWORD} = $self->args ("password");
    local $ENV{PGUSER} = $self->args ("user");

    # Make sure the database is created
    $self->create_db;

    # Run tests as normal
    $self->SUPER::ACTION_test (@_);
}

All this does is set the appropriate environment variables for psql to pick up, creates the database, and then runs the normal test action that it inherited from Module::Build. The convert action is similar, except it shells out to the convert script.

Etc., etc. I'm not holding this up as any paragon of implementation--in fact, it's exposed some shortcuts I've taken that I ought not be taking, so I'm gonna have to clean those up eventually, and I should be able to add automatic .dp to .sql conversion and so forth--but for an hour or two of poking around, I've made some not-inconsequential extensions to the build system, giving me a much cleaner, more integrated process.

There is now an templating system implemented using client-side Javascript.

Normally this would be boring an tedious to contemplate, but, as Ian Holsman observes, combined with liberal use of XMLHttpRequest, this could be interesting.

So, the Perl 6 implementation on top of Parrot continues to chug along. At least, I suppose it does--since Piers Cawley doesn't write his weekly summaries any more, I have no idea. I guess I need to subscribe to yet another mailing list.

However, regardless of that, a "competing" implementation is being worked on. The weird, mildly disturbing part of it is that it's being implemented in Haskell.

Now I don't have anything against Haskell, per se--in fact, I considered learning it as a new language, although I ended up going with C#, which is a whole other story--but everything I've seen about it suggests that it is, if not an anti-Perl sort of language, at least a very un-Perl sort of language. To use one to implement the other seems, masochistic.

So, as part of my ongoing quest to have as spare an office as possible (you must understand that I mean spare by my usually cluttered standards--I do not intend to get rid of, say, the four large bookcases full of books, or the several hundred CDs, say; I just want to get rid of all the superfluous shit), I often grab a stack of old magazines I've kept around, and start going through them, looking for anything worth cutting out, and recycling the rest.

Yesterday I did some WebTechniques from 1999-2001, and boy, were they amusing--very much of their Internet-bubble time, and rife with flavor-of-the-month software and technology that no one even thinks about any more.

Today I started in on my old Dr. Dobb's Journal. The oldest issues I have are from '97 (I have the CD-ROM that had the text of articles up to that point), but boy, even that's a heck of a time capsule--for instance, one of the big articles has to do with the Pentium II math bug, which I hadn't thought about in years.

Also of interest are some of the authors, who I now know of from different contexts--for instance, I just noticed a C__ article from Nathan Meyers, who I know from both the gcc development list (not suprising), and from the occasional Debian list.

What's really wierd, though, is how irrelevant it all it seems to me in retrospect. You have to understand, this is a magazine I've been reading off and on--mostly on, though I let my subscription lapse a couple of months ago for the first time in a decade--since I was, say, 15. That is more than half my life.

And yet, the vast majority of the stuff in these issues I'm looking at hasn't had much to offer me-+I mean, I do believe that some of it has indirectly made me a better programmer, if only by making me cognizant of some of the "big picture" issues of programming, or talking about language+ and platform-neutral issues and such.

I guess this really drives home to me that I work outside the mainstream, and I don't have any desire to move towards the mainstream. Dr. Dobb's had become a magazine where articles were either oriented towards the mainstream--programming Windows stuff, or how to use whatever new Java interface Sun has dreamed up--or they were too specific to do anything for me--how to compute elliptic curves across 3D spaces or other such hyper-specialized stuff. So I don't read it any more.

Wierd.

So, our whole application is written in HTML::Mason, running under mod_perl, etc. Everyone seems quite happy; it's performing really well--we're handling nearly 2M hits/day (with about a 10:1 graphics to HTML ratio) on a dual PIII/1Ghz app server and a similar DB server--and it's pretty easy to get it to do whatever you want it to.

However, for reporting purposes, we have to produce something in a reasonable printable form. The only really portable print-oriented format that gets you good display control is .PDF. We've gone down the using-HTML-to-generate-print rathole for one report, and it's too horrific to contemplate doing more.

There's no good--by which I guess I really mean high-level--free way to produce PDFs in perl that I have found, and boy, have I looked. And the commercial tools that do what we need all seem to want multi-thousand dollar licenses for "server versions". This is unattractively expensive.

But we still need pdf, and until xmlroff is in better shape, and its libfo is hooked into a perl module (boy, I wish I had time and energy to help with it, because it's a cool system), there really aren't any options that don't involve using external processes.

So, we've started retrieving our report data in XML form, using Matt Sergeant's XML::Generator::DBI, and we've got some stylesheets that do our conversions to HTML and CSV, and we'll eventually do some to get the data in a form to feed to Spreadsheet::WriteExcel::FromXML so that we can get XLS files.

And we can write stylesheets to go to fo, and then we can run FOP on them, and, here's the beauty part, our HTML::Mason component that does this can then just make an Apache subrequest to FOP's output file, and Apache will take care of sending the PDF back for us.

It's a simple as:

system qq{/usr/bin/fop -q -fo $fo -pdf $pdf};
$m->auto_send_headers (0);
my $subrequest = $r->lookup_file ($pdf);
if ($subrequest->status == 200) {
    $subrequest->run (1) ;
} else {
    $m->abort (404);
}

We do similar things for doing access control on static graphics--we keep them absolutely outside of our document root, and then have a Mason dhandler that decides whether you're allowed access, and if you are, let's Apache take care of you.

I don't think enough people use Apache subrequests.