inamerrata

Continuing my “Today’s del.icio.us links” variant in my usual indolent fashion, an interesting post from the other week was this one on making apt-get update a bit quicker and more pleasant. It’s based on these ideas from a while ago, but now it actually works. (Well, actually it worked years ago when Ben Bell first implemented something along these lines, but that’s been dead for years, afaik. There’s some more discussion in Bug#128818 if you’re obsessive about reading background information, which, of course, you’re not)

Anyway, on the client side, you can make use of the whole concept with two scripts: “untiffani” and “apt-qupdate”. Note that the scripts below obsolete the ones in the mail cited above, which will no longer work.

Here’s untiffani:

#!/bin/sh

file="$1"
index="$2"
patchurl="${3%/}"
patch=$(tempfile)

sizeof () {
 wc -c "$1" | sed 's/^ *//;s/ .*//'
}

c_sha1=""
c_size="-1"
cur_sha1 () {
  if [ "$c_sha1" = "" ]; then
    c_sha1=$(sha1sum < "$file" | cut -d\  -f1)
  fi
  echo $c_sha1
}
cur_size () {
  c2_size=$(sizeof "$file")
  if [ $c2_size != $c_size ]; then
    c_size=$c2_size
    c_sha1=""
  fi
  echo $c_size
}

patch_sha1_size () {
sed -n '/^SHA1-Patches:/,/^[^ ]/'"s/^ \([^ ]*\) *\([^ ]*\) $1\$/\1 \2/p" "$index"
}

sed -n '/^SHA1-History:/,/^[^ ]/s/^ / /p' "$index" |
  while read n_sha1 n_size n_patch; do
    echo "try: $n_patch"
    if [ $(cur_size) = "$n_size" ]; then
      if [ $(cur_sha1) = "$n_sha1" ]; then
        wget -q -O- "$patchurl/${n_patch}.gz" | zcat > $patch
        p_size=$(sizeof $patch)
        p_sha1=$(sha1sum < $patch | cut -d\  -f1)

        if [ "$p_sha1 $p_size" = "$(patch_sha1_size $n_patch)" ]; then
          echo "applying patch $n_patch"
          (cat $patch; echo "wq") | ed "$file" >/dev/null
          c_size=0
        fi
      fi
    fi
  done

rm -f "$patch"

And here’s apt-qupdate:

url=http://merkel.debian.org/~aba/debian
tmp=$(tempfile) || exit
cd /var/lib/apt/lists
for p in *_Packages *_Sources; do
  file=${p##*_}
  path=${p##*_dists_}
  if [ "$path" = "$p" ]; then continue; fi
  path=$(echo ${path%_*} | tr _ /)
  >"$tmp"
  url2="${url}/dists/${path}/${file}.diff"
  wget -q -O "$tmp" "${url2}/Index"
  if [ -s "$tmp" ]; then
    cp "$p" "${p}.bak"
    /usr/local/bin/untiffani "$p" "$tmp" "${url2}"
  fi
done
rm -f "$tmp"

apt-cache gencaches

If you install untiffani in /usr/local/bin, you can then use “apt-qupdate” in place of “apt-get update”, so that you only have to download the lines in Packages and Sources that actually changed. As you can probably guess by the url in apt-qupdate, it’s still unofficial, but with a little more testing it should be able to be integrated into both the archive and apt relatively soon.

Also potentially cool and interesting, in light of this thread is the possibility of doing the same thing for Contents.gz files (for those playing along at home, apt-get update updates Packages files, which just deal with, well, what packages are available and their relationships; Contents files gives you a list of every file in every Debian package. As such they’re rather large — over 100MB uncompressed). Anyway, if you’ve downloaded sid’s Contents-i386.gz to the current directory, and uncompressed it (as ./Contents-i386), you can update it with:

url=http://merkel.debian.org/~aba/debian
url2=$url/dists/sid/Contents-i386.diff
wget -O Contents-index $url2/Index
/usr/local/bin/untiffani ./Contents-i386 Contents-index $url2

That’s something like a 150kB download instead of an 8MB one, which is always nice.

If you really want to stress about the copyright of quick hacks like the above, you can consider them licensed under the GPLv2 or the terms of the BSD license distributed in base-files. The scripts are available for download rather than cut and paste, too: untiffani and apt-qupdate. Since you have to run these programs as root, probably worth eyeballing them for security issues before running them.