inamerrata

My mum has a high school reunion coming up, and got some copies of her old school magazines so she could write some articles. As it turns out she wrote a fair chunk of articles for them, one of which was a glowing review entitled “Australia!” Interestingly, it came from the assumption that the reader would think Australia was a fairly boring, backwards, inconsequential sort of place — and as you can guess from the exclmation mark, went on in rapturous, euphoric tones to convince you of quite the opposite.

What’s fascinating, to me anyway, is that I can recall the same kind of “Pfft. Australia.” or “Australia? That’s some snowy country in Europe, right?” attitude not all that long ago — you’d really think that people living in a thriving state capital on the turn of the millenium might have developed a more optimistic view of the country than a young lady stuck in a tiny country town had *mumble* decades ago.

But, on the other hand, I also think that’s started to change over the past few years; things like the Sydney Olympics, the intervention in East Timor and the Solomons, and even divisive things like the close relationship with the US on the War on Terror and the involvement in the Iraq War, have made us start to think that, well, maybe we aren’t Britain’s forgotten step-child and we do have some contribution to make. And hey, Ricky was even kind enough to give me some evidence that people really are coming to a different view.

Of course, he explains it somewhat differently:

This year’s Australia Day got me thinking about patriotism and the role it plays in the modern world. I have often believed patriotism to be one of the greatest scourges afflicting the globe. […] Yet, I have come to the uncomfortable conclusion that patriotism plays an important positive role in the world today. Patriotism, it seems, is the guardian of cultural diversity.

Patriotism – in its wider connotation – is the love of one’s country, region or culture. […]

Personally, I’d go further, and say that it’s also the love of one’s family, one’s friends and one’s neighbours. It’s also an expression of happiness and acceptance of the environment you find yourself in — that you think the rivers, and the climate and the mountains and the sights are wonderful and don’t want to change them or lose them; that you’re grateful for the homes and businesses and society your parents and their parents have created and built up.

And if you define patriotism that way, it’s a good thing in and of itself. It’s respect, and admiration, and conservation, and friendship, and family, and peace, love and goodwill all rolled into one.

But that’s not the only way to define it. The first thing that comes to mind when I think of patriotism is actually this song:

And did I hear you say
My country right or wrong
My country oh so strong
My country going wrong
My country right or wrong

It came out in the early 90’s by the look, back when I was an impressionable youth, a few short years after leaving primary school and hence no longer singing the national anthem every week at assembly, or standing to watch the flag be raised every morning.

Anyway, the lyrics sung by the Labor member for Kingsford-Smith make an interesting contrast to these comments from the government Minister for Health, Tony Abbott:

[…] [Howard] has been much mocked for saying that he wanted an Australia that was “relaxed and comfortable” but this attitude has actually been at the core of his success.

[…]

A very significant part of Howard’s achievement has been to massage away the pessimistic and narrow-minded aspects of Australian conservatism. Fear of Asia, mistrust of difference, obsessive concern with whether people are getting more than their share are much less part of our national make-up than they were. Australians’ capacity to be self-critical has not changed since 1996 but we’ve largely lost the adolescent agonising about who we are and where we belong.

[…] [Australians] no longer feel threatened by diversity and think that the extended family is a good metaphor for contemporary Australia.

Almost unnoticed, the Howard Government has started to create a new story for Australia, not as the plucky ally, certainly not as anyone’s deputy sheriff, but as a force for good in our own right.

Maybe it’s that the jingoistic singing and flag raising got to me when I was even more youthful and impressionable, but I definitely know which view I prefer to hold.

Anyway, Ricky’s thesis doesn’t really work, to my mind. Apart from the knee jerk anti-capitalism (“But ironically, it is capitalism and the free market economy that is threatening to swallow our multifarious cultures and replacing them with a McDonald’s or Starbucks on every corner.” — but it’s also capitalism and free market economies that give American’s hundres of channels of locally produced content, while government regulations keep us stuck to a handful of channels that can only afford to run proven successes; likewise it’s capitalism and the free market that gives you half a dozen food courts in the mall where you can get cheap quick food inspired from countries on every continent except Antarctica)…

Err, I kinda got distracted there, hey? Let me try again. Ricky’s thesis doesn’t work anyway, to my mind — if you’re going to use patriotism to defend yourself from the iniquities McDonalds and Starbucks, presumably you’re worried about succumbing to the borg-like empire of the United States. But at least as far as I can see, it’s actually far more diverse than Australia — the difference between Brisbane on the east coast, Perth on the west coast, and Adelaide on the Great Australian Bight is pretty much one of environment — the people are pretty much the same either way. But the differences between the culture of, say, New York, Los Angeles, and New Orleans is quite a bit more notable. Now, maybe I chose my cities a bit unfairly: how about Salt Lake City, Utah; Las Vegas, Nevada; and Austin, Texas instead? Add in Seattle and Honolulu, and you’ve pretty much used up all your Australian capitals already, with no two of their US counterparts being alike. And there’s still another three-dozen to go.

Of course, you could argue that of course America’s more diverse — it’s been more patriotic than Australia for decades! But in that case, is patriotism good for protecting diversity, or for creating it?

Well, a draft anyway. Available now from the linux-aus archives. Highlight? Announcement of the location and dates for linux.conf.au 2006!

David pointed out the absolutely sacralicious “Baby Got Book” that seems to be doing the rounds (and if it’s not, it should be!). Normally I wouldn’t relink to things even that awesome (in the same way I didn’t pass on the cartoon skeletons page Hanna of Debian linked to), however in light of Ricky’s spiel on blog attribution, I figured it was probably worth quoting this tidbit google found me:

Hey this is Dan Smith, the guy who wrote Baby Got Book. Thanks for the love in hear. Check out the link to my website if you had trouble seeing it. My CD is coming out in the spring.

www.whiteboyDJ.com/babygotbook

If you tell anyone about the video, hook a brotha up and send ’em to my site…

So, consider yourselves hooked up, brothas (and sistahs!).

Wow; I thought David getting a hit from the Troppo Armadillo was cool (not to mention the rest of his growing fan club), but now the other David gets slapped in the face with the velvet glove of the Mistress of Hollywood Babylon herself?

Continuing my “Today’s del.icio.us links” variant in my usual indolent fashion, an interesting post from the other week was this one on making apt-get update a bit quicker and more pleasant. It’s based on these ideas from a while ago, but now it actually works. (Well, actually it worked years ago when Ben Bell first implemented something along these lines, but that’s been dead for years, afaik. There’s some more discussion in Bug#128818 if you’re obsessive about reading background information, which, of course, you’re not)

Anyway, on the client side, you can make use of the whole concept with two scripts: “untiffani” and “apt-qupdate”. Note that the scripts below obsolete the ones in the mail cited above, which will no longer work.

Here’s untiffani:

#!/bin/sh

file="$1"
index="$2"
patchurl="${3%/}"
patch=$(tempfile)

sizeof () {
 wc -c "$1" | sed 's/^ *//;s/ .*//'
}

c_sha1=""
c_size="-1"
cur_sha1 () {
  if [ "$c_sha1" = "" ]; then
    c_sha1=$(sha1sum < "$file" | cut -d\  -f1)
  fi
  echo $c_sha1
}
cur_size () {
  c2_size=$(sizeof "$file")
  if [ $c2_size != $c_size ]; then
    c_size=$c2_size
    c_sha1=""
  fi
  echo $c_size
}

patch_sha1_size () {
sed -n '/^SHA1-Patches:/,/^[^ ]/'"s/^ \([^ ]*\) *\([^ ]*\) $1\$/\1 \2/p" "$index"
}

sed -n '/^SHA1-History:/,/^[^ ]/s/^ / /p' "$index" |
  while read n_sha1 n_size n_patch; do
    echo "try: $n_patch"
    if [ $(cur_size) = "$n_size" ]; then
      if [ $(cur_sha1) = "$n_sha1" ]; then
        wget -q -O- "$patchurl/${n_patch}.gz" | zcat > $patch
        p_size=$(sizeof $patch)
        p_sha1=$(sha1sum < $patch | cut -d\  -f1)

        if [ "$p_sha1 $p_size" = "$(patch_sha1_size $n_patch)" ]; then
          echo "applying patch $n_patch"
          (cat $patch; echo "wq") | ed "$file" >/dev/null
          c_size=0
        fi
      fi
    fi
  done

rm -f "$patch"

And here’s apt-qupdate:

url=http://merkel.debian.org/~aba/debian
tmp=$(tempfile) || exit
cd /var/lib/apt/lists
for p in *_Packages *_Sources; do
  file=${p##*_}
  path=${p##*_dists_}
  if [ "$path" = "$p" ]; then continue; fi
  path=$(echo ${path%_*} | tr _ /)
  >"$tmp"
  url2="${url}/dists/${path}/${file}.diff"
  wget -q -O "$tmp" "${url2}/Index"
  if [ -s "$tmp" ]; then
    cp "$p" "${p}.bak"
    /usr/local/bin/untiffani "$p" "$tmp" "${url2}"
  fi
done
rm -f "$tmp"

apt-cache gencaches

If you install untiffani in /usr/local/bin, you can then use “apt-qupdate” in place of “apt-get update”, so that you only have to download the lines in Packages and Sources that actually changed. As you can probably guess by the url in apt-qupdate, it’s still unofficial, but with a little more testing it should be able to be integrated into both the archive and apt relatively soon.

Also potentially cool and interesting, in light of this thread is the possibility of doing the same thing for Contents.gz files (for those playing along at home, apt-get update updates Packages files, which just deal with, well, what packages are available and their relationships; Contents files gives you a list of every file in every Debian package. As such they’re rather large — over 100MB uncompressed). Anyway, if you’ve downloaded sid’s Contents-i386.gz to the current directory, and uncompressed it (as ./Contents-i386), you can update it with:

url=http://merkel.debian.org/~aba/debian
url2=$url/dists/sid/Contents-i386.diff
wget -O Contents-index $url2/Index
/usr/local/bin/untiffani ./Contents-i386 Contents-index $url2

That’s something like a 150kB download instead of an 8MB one, which is always nice.

If you really want to stress about the copyright of quick hacks like the above, you can consider them licensed under the GPLv2 or the terms of the BSD license distributed in base-files. The scripts are available for download rather than cut and paste, too: untiffani and apt-qupdate. Since you have to run these programs as root, probably worth eyeballing them for security issues before running them.

That’s the ABC’s claim anyway:

The increasing popularity of online music stores is welcome news to a music industry that blames digital piracy for more than two years of precipitous sales declines.

“The biggest challenge for the digital music business has always been to make music easier to buy than to steal,” IFPI chairman and chief executive John Kennedy said.

“At the start of 2005, as the legitimate digital music business moves into the mainstream of consumer life, that ambition is turning into reality.”

(Why do I say the ABC claims it, reather than IFPI chairman John Kennedy? Because when they don’t support the claims they’re quoting, they attribute the claim in the headline, such as “National ID plan not another Australia Card: Ruddock”)

Anyway, having just been kicked in the gut by the failure, yet again, of Apple to open an iTunes store in Australia, I can’t say I’m much impressed by the idea that digital music’s easier to buy than steal. Brad then pointed me at BigPond Music in mock conciliation. At which point I got this lovely message:

In order to download and purchase music from BigPond Music you must have a compatible Windows Media Player.

At present there is no compatible Windows Media Player for Mac OS and as such you will be unable to play any downloaded music. You will be able to browse the BigPond Music Catalogue.

Easier to buy than steal my ass.

The national broadcaster also blithely repeats the claims of “more than two years of precipitous sales declines” without question. Tsk.

With the trend of posting daily del.icio.us links, I’ve been thinking of trying out a “daily mailing list posts”. Maybe I’ll get around to it sometime. Anyway, today’s mailing list post is this one about dodgy uploads of lib packages. It seems to be the season for it, since Matt Zimmerman did the same thing with libflac recently. Anyway, as that email mentions, I rejected the packages. That’s done with a tool from the “dak” suite called “lisa”, and everytime I use it, I tend to feel sorry for whichever poor sucker gets the message — heck, it’s not even enough to say “the package was rejected, sorry”, it’s instead shouted with a subject like “libofx_0.7.0-3_powerpc.changes REJECTED”.

So, with apologies to Ben Folds…

Using Lisa

The queue is full again.
It took me by surprise.
They’re so sensitive,
But shit just happens sometimes.

She’s my everything.
She’s my best friend and more.
We don’t do anything,
We didn’t do the day…
Before you go,
You ought to know,
That she didn’t mean to hurt you.
I just wanted you to know.

Black tears are falling down your face,
Your deb was wrong.
Black tears are falling as you realise
What she’s done.
Your deb was there before you,
And now it’s gone.
Black tears are falling, falling.

Remember long ago,
Debugging, building, hacking.
Oh so long ago,
Then you uploaded something.
Life was simple then.
But it’s REJECTed now, and you…
You don’t feel anything.
She was reproving;
You are mesmerised
By tiny lines.
You’re watching as the mail is forming,
Slowly from the MIME.

[Repeat Black tears… verse]
Black tears are falling, falling.

Using Lisa…
Using Lisa…
Using Lisa, and there’s nothing you can do.

Black tears are falling, falling. Ooo, ooo.
[repeat to fade out]

One of the curious things about testing software is that as far as increasing your confidence in the code is concerned, it’s better to have a test suite that finds bugs, rather than one that doesn’t.

The problem is that for any piece of software that ever works, you can find an infinite number of test cases that pass, no matter how buggy it actually is. So if those are the only test cases you come up with, your testing methodology hasn’t actually achieved anything you hadn’t already managed to do when you were writing the code in the first place.

It’s possible that you did such a good job writing the code that there aren’t any bugs to be found, or that they’re so obscure that a test suite would have to be amazingly thorough (or lucky) to find them anyway. If so, congratulations! If you can do this reliably — or even if you can merely reliably tell when you’ve done this good a job at coding — there’s no point writing any tests for your code.

For people who don’t have the time or inclination to prove their code correct as they’re writing it, though, testing is still worthwhile. But given testing can’t ever prove the absence of bugs, when should you stop? The first point at which you should stop is when you find a bug. At that point, you should go back to coding, fix the bug, and start testing again; ideally reusing all your old tests because you’ve carefully scripted them.

But “rinse, repeat” algorithms only terminate when there are suds and water involved, and those aren’t terribly useful when we’re trying to write reliable software. Basically, your focus when testing your code is to find some way to break it. If you can’t do that, you’re not trying hard enough. There are plenty of techniques to help you break code: looking for special cases, looking for code paths that are obscure enough the last developer to commit changes might not have thought to check them, looking at edge cases, looking for discontinuities, all sorts of things. Unless you’re really the best programmer on the planet, there’s always some bug you’ll be able to find.

So, when should you stop? First, you shouldn’t stop before you’ve found (and fixed) at least some bugs. Users always pound on a program harder than the developer, so if you don’t find some bugs in testing, you can be certain to be getting reports hours after the first users start playing with it. Once you’ve found (and fixed!) a few bugs, though, you’ll find your test cases are getting more and more numerous, and more and more obscure. Once they start getting to the point where you find it pretty implausible that the cases will come up in real life, continue for another round, then stop.

Then, when a few days later, when you start getting people trying those implausible cases and finding bugs, start again.

In his epic battle with Adrian over exceptions, Ben mentioned:

Save your work? That’s for sissys. Use a journalling file-saving model. Save everything the user does immediately. You can support the traditional file save/load facility using checkpoints or other niceties but I fail to see why any application in this modern age of fast hard drives should ever lose data the user has entered more than a few hundred milliseconds ago.

That’s really very true. Certainly there are caching issues — memory is much faster than disk, and you don’t want to store intermediate state all the time, and likewise you probably want to keep some history, but certainly don’t want to keep all of it. But generally, disk is fast enough (compared to user interaction and network speeds) that there really isn’t any excuse for losing data.

If my battery and power shorted out right now, I’d lose: (a) this entry; (b) the fact I’ve got Planet Humbug open to read Ben’s blog and my Back-button history; (c) the fact I’ve got three terminals open, and their history (both scrollback and command invocations); (d) what email I’m currently reading.

Saving this blog entry takes at most half a second, mostly because to do so I have to hit some keys and a menu gets highlighted. Saving all my scrollback from scratch seems to take under 5 seconds, mostly because I have to cut and paste it into another application first. And the other two items are fairly trivial annotation issues.

This isn’t even that hard an issue to solve: it’s only relevant for long-running apps with user interfaces — so xterm and bash, but not sed or ls. It requires somewhere for apps to dump their status ($TMPDIR or a dotfile in $HOME). And it requires some code to do recovery, possibly with a user interface in order to choose which point in history you want to recover. And given those issues, for many apps, there’s no reason not to then just automatically recover after a crash, at least as long as you give the user an easy way of avoiding getting into a “recover to a point where you always crash; always recover when you crash” loop (or at least an easy way of getting out of it).

…and you can’t beat that.

Or, at least, I think that was some primary school slogan we had at some point. But Google knows nothing about it, so maybe I imagined it all. Surely, there isn’t still more in the universe than’s dreamt of in Google’s eight billion web pages… I think it might’ve had something to do with aboriginals; it rhymes after all.

Anyway, I occasionally get confused at the way both lefties and right-wingers claim exclusive ownership of particular concepts. Well, not so much that, but when both manage to do it over the same concept, and actually show evidence that they’re right.

One of those concepts is individuality. The lefty concept looks something like “I’m an individual, standing up against the crowd, bravely dissenting from the accepted wisdom in the knowledge that time shall prove me right”, while the right-wing one is perhaps akin to “I can look after myself. Anything that my family wants, that my town wants, that my church wants, we can build ourselves, and we will.” The right looks at things like communism, big government, and tariffs and wonders how anyone who claims to be an “individual” can either be so incapable of looking after themselves to want that stuff, while the lefty looks at the disciples gathered in church all reading from the same page of the same book, or the ministers and executives who toe the party line religiously, and wonder how anyone who doesn’t haven’t an opinion of their own could possibly count as an individual.

I don’t actually wonder either of those things. What I do wonder is why we’ve got such adverse definitions for individuality. Why should you have to imagine yourself surrounded by people who unthinkingly disagree with you, or alone on a prairie, with no one to turn to and no hope of help, just in order to think of yourself as an individual?

Clearly, right and left need to come together and turn to the world of fashion to establish a new conceptualisation of the crucially important issue of how we shall comport ourselves as individuals.

Decided to wander into NextByte to see about getting some more ram for my nice new iBook today. I’ve been tossing up whether to go for an extra 512MB (for 768MB total) or an extra GB (for 1280MB total) — I really wanted as much as possible, since OS X is a memory hog and I want to run virtual hosts on it too; but I couldn’t justify paying three times as much for twice as much memory. In particular, when I rang a few weeks ago, I was quoted something like $600 for memory, and $500 for a copy of VirtualPC with a copy of Windows XP Pro I didn’t want. Yick.

So anyway, I finally got fed up with 256MB, and decided it was time to, well, decide. So as I wandered past the shelves, I noticed the pretty boxes Microsoft has for VirtualPC, and in particular noticed they had Windows 2000 and Windows XP Home variants. They also had a “standalone” variant, that doesn’t include a Windows license! Woot! Also woot-worthy is that the price of a GB had tumbled to only around twice the cost of 512MB. So I said “yay, gimme, gimme”, and got the RAM installed there and then, and got told that the (shrinkwrapped) VirtualPC box was display only (ie, completely empty) and they’d order it in. So, soon I shall have some active Debian and Ubuntu development installs again. Sweet.

Evidently Energex noticed the twenty minute thunderstorm we had after lunch today, and decided to switch to their summer policy. Hence we’ve had two prolonged blackouts this evening, so far; the current one’s stopping me from watching both the Rebel Billionaire and the West Wing. Not impressed. And why they’re happening well after the storm, I’ve no idea. On the upside, at least my laptop has a working battery now…

I suppose hoping that heads will roll at the electricity utility would be in bad taste, considering the past year’s events…

One of the harder aspects of version control is dealing with merging issues. Normal development is straightforward — all you’re essentially doing is providing an annotated “undo” feature. darcs manages that, IMO, perfectly. And to be honest, that’s probably 80% of what I want form a version control system. But dealing with merging different lines of development is important too — it’s probably 80% of the remaining 20% :)

darcs doesn’t actually do too badly there — when you’re working on separate parts of the code, darcs will do a merge for you automatically quite happily. Where it falls apart is when the changes affect the same bit of code, and can’t be resolved automatically; I find myself really disliking darcs’ behaviour there, even independent of the performance issues.

It doesn’t help that I don’t remotely understand what darcs thinks “merging” means in the event of conflicts. But at least I’m beginning to understand what I mean by merging, or, rather what I expect a version control system to do for me when I try merging disparate branches. And that’s this:

First, it should produce a tree that includes the changes from both branches — even one that doesn’t compile — so that I don’t have to worry about looking through diffs myself to make sure I’ve incorporated all the appropriate changes. darcs kind-of manages this in some circumstances. Second, it should make it easy for me to merge later changes between both branches — if two branches add two different functions by the same name, and when I merge them, I resolve that conflict by renaming one, I want future merges that only deal with the contents of either function to happen completely automatically. darcs doesn’t manage that at all.

The reason it doesn’t is that it uses an incremental approach to resolving mergers. If you’re merging from a repository whose history is “X, A, B”, into a repository whose history is “X, L, A’, M”, where merging A resulted in a conflict with L that M resolved, then darcs will construct “X, A, L’, M”, then merge “X, A, B” with “X, A, L’ ” to get “X, A, L’, B’ “, and then merge that with “X, A, L’, M” — but if A and L conflicted, and B depends on A, then B and L’ will conflict too, and there’s simply no way for M to resolve that conflict too.

There are (at least) three potential approaches to dealing with this. You can make L’ and B not generate a conflict (by encoding some part of the resolution of the conflict between “L” and B in the differences between L’ and L). Alternatively, you can change the definition of merge from “find the most recent common history, then commute” to also include the possibility of applying the patch directly to the point at which the conflict was resolved — though you need some other way to adjust the patch at that point. Finally, you could find some way to have the resolution patch not only resolve the existing conflict but also resolve future conflicts too.

Probably the middle solution is the only one that wouldn’t require major changes to the way darcs deals with resolving conflicts — at present the application of a conflicting patch undoes both the original change and the conflicting change, and the resolution creates a new change that’s the combination of both. It’s probably not possible to infer enough information into the conflicting patches, or into the resolution patch to resolve future conflicts.

To implement the middle solution, you need to introduce a new “patch” type that says “you can treat me as “R, A” and apply B from “R, A, B”, if you apply the following transformation to B”. You need to find some way of commuting the new patch type around. And you need to find some way of generating the transformation given a repository that’s not necessarily remotely similar to the original “R, A” repository at all, and that may differ from it in ways that can’t be automatically determined (for example, you’re merging one patch that changes two lines that say “foo” to both say “foobar”, and another patch that deletes one of the foos — easy enough, you end up with a single foobar. Then you get a new patch that only changes one “foobar” to “bazquux”. Was it the one you kept, or the one you got rid of?).

Tricky.

One of the central ideas in Jef Raskin‘s book The Humane Interface is that the “zooming” interface — rather than 2d windows that you shift around and overlay on each other, you have a huge canvas that you can zoom into and out of, as well as move around on. Obviously your screen only displays a small portion of that canvas at any one time. A further implication is that you have a “document centric” model, and thus that your data is always visible, and you just start editing it rather than starting an app to view/modify it.

That has pretty heavy implications, and renders most existing software simply unsuitable for a “ZUI”. So while there’s a couple of attempts at implementing them, they’re nowhere near being generally usable. Still, they’re a cool idea.

The thing I like most about them (in theory) is that you can get a broad, 2d overview of everything you’re doing, then zoom in on a bit and work on that. I hate trying to work out how to organise myself in the file system, because I always leave crap lying around where it ends up getting in the way later.

But if you’re willing to compromise a bit, it’s probably possible to fake this. If you consider the “project view” to be whatever you can see on the screen (after using Exposé to get rid of overlapping windows maybe), then a ZUI is just a matter of being able to see many project views at once, possibly scaled, move amongst them, and zoom into one and make it functional. There no great need for zoomed out projects to be functional — you can’t see what you’re typing anyway.

That’s not impossible. If you have two modes: “zoomed out” for navigation, and “zoomed fully in” for manipulation, then the latter is obviously implemented by just having running apps, and the former can be implemented by: (a) taking a snapshot of the apps; (b) closing them; (c) embedding the snapshot in a larger canvas, along with other snapshots; (d) allowing you to navigate around the canvas; (e) when a snapshot is selected, restarting all the apps in the same state as when they were closed. (a), (c) and (d) are just graphics manipulation, and are trivial. (b) alone is trivial; (b) and (e) combined are something KDE and Gnome at least have been doing with session management for ages.

A real ZUI would let you drag components from one section of the canvas to another, too. And let you scale running apps so you can get an overall idea of what they’re displaying but not worry about them taking up much screen space. And a zillion other things too.

Cripes. This was meant to be a quick followup note about some more quick darcs hacks. So much for that — I’ve had to write an outline for this post for heaven’s sake.

(Side note: if someone wants a new title for their blog, the above’s free of charge!)

So, when last we met, darcs-repo had just come into the world, and we were still choking on the cigar smoke. Following that there were a couple of discussion threads. Interesting mails include this one, so that you just ask for a repository rather than a “branch” of a “project”, and the program works out how that’s stored, or this one (and its followups) about naming a collection of related repositories an “archive”, and changing the name from darcs-repo to darcshive. This one (and followups from December) includes some (applied!) patches to darcs itself to let me get rid of the horrific ssh/scp hacks.

Where does that leave us? Pretty much at the point of moving from a prototype/proof-of-concept darcs-repo to a functional darcshive. It’s been essentially self-hosting from the beginning, but a more challenging task is hosting darcs itself — since it’s likely that darcs excercises most of the interesting features of the darcs repository format.

This turned out a little more tricky than I’d hoped right from the start: it turns out that generating a “complete” patch bundle is a non-trivial affair. I suppose I should’ve guessed when creating the necessary null context (darcs init; darcs changes --context) I got an internal error (Fail: Prelude.head: empty list). That can just be ignored; what’s suckier is that darcs send still takes ages to work against a null context. Eventually I gave up, and just copied the patches and inventories into darcshive, munging as I went.

The inventory munging is easy, you just need to get all the tag inventories together in the right order (and date is a fairly safe assumption for tags, hence you can go by filename), and trim off the Starting with tag: line at the top:

for i in inventories/* inventory; do
  sed -n '1{s/^Starting with tag://;t
};p' $a; done

The patch munging is harder, unfortunately. The first problem is that patches used to use dates like Wed_Dec__8_14:57:25_EST_2004, instead of the modern 20041208145725. The all-numeric version is still needed since it appears in the filename, but the old version is also needed because it gets hashed to work out the filename too. Yay parsing! Fortunately, the timezone is more or less ignored by darcs, and I can get away with completely ignoring it. Considering some patches had a timezone of “Pacific Daylight Time”, I’m very happy that turned out to be the case.

More fun were a couple of other issues in the patches; particularly the fact that sometimes single patch bundles didn’t get stored as a sequence; so the opening and closing curly braces I was relying on weren’t there, and my parser got completely confused. vi fixed that. Similarly annoying was that it seems old versions of darcs ended long comments with “]$” instead of “^]$”. But that’s ambiguous — what happens if the long comment includes a closing square bracket at the end of a line? Apparently darcs deals with that somehow, but I chose not to and reverted to vi again. Fortunately when calculating the hash, darcs ignores the newline characters entirely, so adding and removing them doesn’t cause any problems.

A couple of other tweaks were needed too: darcshive had a bug in calculating the hash if the last line of the description included a closing square bracket followed by a space, which the darcs history triggered. And recent changes to darcs have changed the way patch bundles are produced (and weirdly darcs send and darcs push generate slightly different formats for their patch bundles), so I had to make my parser more liberal as far as blank lines are concerned too.

But at this point, I can store darcs in darcshive, do a darcs get on a darcshive:// url to get it, and it comes out buildable and everything. Sweet. Thanks to the DARCS_MGET_ stuff, it’s even quicker to use darcshive than darcs-over-http.

I had hoped that it might be really quick to have darcshive generate a patch bundle against the null context, so that you could “get” a repository by downloading a single patch bundle, and applying it to a freshly initialised darcs repository. Unfortunately darcs is equally slow at pulling to a null repository as pushing to a null repository, so you have to stick with get anyway. Oh well.

Where does that leave us? Well, the concept seems proven; so that means tidying things up. Having darcshive deal with TAGs better would be nice — there’re some bugs in how “Starting with tag:” is dealt with in patch bundles, and it seems like it would be nice to be able to just point at an archive and say “get me this tag, I don’t care what repository you have to get it from”. It’s probably necessary to rename the files darcshive uses too, at the moment you can’t have a repository with “inventory” in the name, or “patches”, or “commuted-patches”. Having the CGI return some useful human-readable pages for browsing as well as the darcs metadata would be nice too.

Also coming up: darcs, dpkg and debootstrap! Well, eventually…

Oh, for those of you wondering:

darcs get http://www.erisian.com.au/darcs/darcshive/mainline