Tiffani
So this week’s project was working on dak, in particular getting the tiffani implemention included. What’s tiffani, you ask? It lets you just download the changes to Sources.gz and Packages.gz files, instead of the whole damn thing — if you have main, contrib, and non-free for unstable in your sources.list, this means your apt-get update only needs to download 27.5 KiB instead of 2.85 MiB (for bz2) or 3.75 MiB (for gz). Two orders of magnitude saving’s not bad.
I’ve mentioned this previously a few times if you want to see the history, in January I noted some quick hacks which have now been obsoleted by Michael Vogt’s integration into apt (which is in a personal repository, not uploaded to unstable yet…), back in August last year I did a little promotion of the concept, referring back to my post in December 2003 that summarised the analysis done previously, which takes us back to March and April 2002.
The server-side implementation, tiffani, is thus a little python script that groks through the list of suites and components dak knows abouts, looks for Packages, Sources and Contents files, and creates a directory names something like Packages.diff which in turn contains an Index file and a number of gzipped patches, named after the time they were created. The process for updating an existing Packages file is to look for the matching SHA1-History entry in the Index file, download the corresponding patch, apply that patch to your existing Packages file, and repeat until your Packages file matches the one you were looking for (which you can determine by looking at the checksum in either the Release or Index file).
Anyway, tiffani’s been co-developed with Andreas Barth (aka Andi) after the following exchange back in November 2003 on the RM channel:
<aba> elmo: If I find some time to take a look at katies TODO-file, what would be most appreciated? Deleting udebs with melanie? “Right” handling of orig.tar.gz when jumping out of NEW / changing of the archive section? Something else?
<elmo> aba: the TODO list is a bit of a mess to be honest
<aba> elmo: So, what should I do if I want to help the ftp-masters with katie?
<elmo> aba: I wouldn’t necessarily do anything based off that unless it’s obviously correct/wanted. none of the really urgent stuff is even on there (e.g. testing-security changes)
<aba> elmo: If you tell me something urgent, I can put some time into it.
<aj> hrm, surely i have some wishlists
<aba> urgent wishlist? :)
<aj> actually, you could critique something for me if you’ve got half an hour free now?
<aba> “critique” means?
<elmo> tell him how crap his ideas are. as brutually as possible
<elmo> ;)…
<aj> ooo!
<aj> aba: around?
<aj> aba: one useful thing you could work on that’s kinda katie related is implementing pdiffs into katie/apt-ftparchive/apt/whatever
<Kamion> aj: mvo’s been working on that over the last couple of days, see #128818 logs
<aj> apt side or katie side?
<Kamion> apt side
<aj> cool
<aj> aba: hacking up a katie-side implementation for that would probably be interesting, worthwhile and straightforward
(aj = me, aba = Andi, elmo = James Troup, Kamion = Colin Watson)
That ended up with me finding an old script I made up called update.py (from Dec 2003 apparently) that did the core patch and Index generation, Andi updating that to some more sane pythonic style and making it work on an actual archive, then both of us cleaning it up so it would actually work sensibly without handholding.
(For those playing along at home, the idea needing critiquing was trying to fix up dak’s idea of what should happen if a new version of a package is uploaded to stable or testing, when it hasn’t changed in unstable. At the time, dak would REJECT it, because it violated the requirement that you can’t downgrade packages between stable and testing or testing and unstable. That’s still somewhat broken, unfortunately)
TTBOMK, Tiffani’s named after Tiffani Amber Thiessen, apparently of Saved by the Bell and Beverly Hills 90210 fame. I’m not sure who of aba or elmo is more to blame for that one, though I guess it is at least a little more distinctive than update…