inamerrata

One of the XP systems I look after had a trojan this month — looks like it came from a fake “UPS package” mail with a zipped attachment that got clicked on, then stuck its tendrils into the registry and all over the place, and started popping warnings about viruses and instructions on how to pay for a fix. After a couple of attempts at removing the infection and finding it just coming back, looks like a reinstall is going to be easier.

Of course, on a free operating system it’s at least theoretically reasonable to know what everything on the system is supposed to be doing, so it should be possible to fix that sort of problem. There’s been a bit of discussion this past month about the md5sums control files which goes some of the way to handling that for Debian — but of course, that assumes your md5sum files aren’t compromised along with the rest of your system.

Ultimately you want two things to cope with potential compromises like this — one is to detect them as early as possible, and the other is to work out what’s infected and what’s recoverable. Which basically means you need a description of how things should be and the ability to compare that to how things actually are.

In some respects, that’s difficult to do: “how a system should work” is hard to define, and tends to change over time — and often people don’t think their systems work as they “should” even when they’re freshly installed and completely uncompromised. But if you aim a little lower, you can at least get somewhere. You could say “my system should be built from the latest Debian testing packages” and verify that, for example. Or you could keep a running tally of packages installed and removed, and say “each entry in my running tally should say what happened and be dated and match my recollection, and the packages from that tally should be Debian packages, and the files on my system should match those packages”.

Knowing what packages you’re meant to have is probably the first challenge — maybe you’re running puppet or similar and have an easy answer to that, but if you just run apt-get and aptitude whenever you want something, it’s a bit harder to tell. Are you running an ircd because you thought one day it’d be a fun thing to do, or because some warez kiddies are using it to control their botnet?

Once you know you’re meant to be have, say, python-llvm installed, you need to know which version it’s meant to be. You could say “the last version I installed, of course” — except of course your only record of that might be on your compromised system. You might say “well, I follow testing, so the latest version in that”, except that there might have been an update to that package in testing while you were compromised, or you might have installed something from unstable or experimental (or backports, or compiled it from scratch). You certainly want to know the architecture, version number and whether it was from Ubuntu, Debian or somewhere else.

Going from that step to knowing what the contents of the package is meant to be is slightly harder. If you happen to know you’re looking for the current version of python-llvm in Debian testing, then you can establish a trusted path to verify what its contents should be by downloading test testing Release, Release.gpg and Packages.gz files which will give you a verified download of a deb file (assuming you trust gpg and sha256, which is reasonable for the moment at least).

If you’re running an outdated version of the package, you’ve got more problems. You could find the original .changes file uploaded with the package to verify it based on the developer’s signature — but that will only tell you that that developer built that package, not that it was uploaded to Debian, distributed far and wide, and installed on your machine. You could find the Release/Packages files that were current when you downloaded it, and verify them, but that’s something of a chore in and of itself. You could make a note of the name, version, location and sha256sum of every package you install and keep it somewhere secure, but that’s a chore too. The easiest solution I can think of is just to treat “outdated” as “potentially compromised”, and install the current version of the package anyway. (For locally generated packages, you should presumably be able to either find an uncompromised version to compare against easily enough, or you’ll have to rebuild it from scratch as part of your recovery anyway)

Once you’ve downloaded the deb file, it’s a relatively simple matter to verify the package is correctly unpacked; a good approximation is something like:

HASHES='python-llvm.hash'
CKSUM='md5sum'
DEB_PATH="/var/cache/apt/archives/python-llvm_0.5+svn85-1+b1_i386.deb"
TAR_CMD='printf "%s%s\n" "$($CKSUM - | sed s/-$//)" "${TAR_FILENAME#./}"'
export CKSUM
ar p "${DEB_PATH}" data.tar.gz | tar --to-command="$TAR_CMD" -xzf - > "$HASHES"
(cd / && $CKSUM -c) < "$HASHES"

(Caveats: assumes data.tar.gz, some debs have data.tar.bz2 instead; the extraction command above takes about 7m on my netbook (HP mini 2133) for the 420 or so debs that happen to be in my /var/cache/apt/archives (about 480MB worth); the above assumes that you have a trustworthy ar, GNU tar, gzip (or bzip2), md5sum (or sha1sum etc), and filesystem, as well as copy of the .deb; the above includes conffiles in /etc many of which will have be intentionally modified; some, but very few, .debs expect some of their distributed files outside /etc to be modified too)

You can skip the first command in that sequence if you use the md5sums files shipped with debs, but that comes with a few drawbacks, in that you're forced to rely on the md5sums files, which can be lost, not present, incomplete or, if you're using the local cache of the hashed files that dpkg keeps in /var/lib/dpkg/info, potentially compromised along with the rest of your system. The upside is there's an existing tool to verify them (debsums).

Personally, I'm now running a patched version of dpkg that generates its own .hashes files as packages are installed. That doesn't do anything about lost or compromised files, but it does ensure they're complete and at least initially present.

But even if all the files that are meant to be installed are exactly as they should be, that's not enough. You've also got to worry about extra files -- maybe your "ls" command isn't invoking "/bin/ls" but "/usr/local/bin/ls" which has been compromised. To some extent that's easy enough with tools like cruft, but there are quite a few places where extra files can screw you over.

Probably the hardest part is checking your configuration files are correct. On both Linux and Windows, you can do a great job of taking over a system just by messing with configuration files, whether that be zeroing a password field, or adding a preload so that every time you run a program a trojan starts up as well, or a timed job to start your trojan back up if it gets disabled. If there's enough configuration data, you might be able to hide a copy of your trojan in their, so that it'll be re-extracted even if the rest of the system is completely cleaned up.

I'm not really sure what the solution here is. For Windows and its registry (and other configuration scattered about the place) I don't think its solvable; there's just too much of it, that's changed in too many ways to really control. So as far as I can see, it's a matter of scrubbing everything, and reinstalling from scratch there.

For Debian and Linux in general, things still probably aren't great, but there's at least a few things you can do. You can probably rely on /etc not changing too much, which means you can do things like track changes with something like etckeeper and review the diffs to make sure they're sensible. Unfortunately reviewing configuration diffs is probably something of a chore, but with distributed version control and a remote append-only repository you've got a chance of that being at least feasible to leave until you're looking to recover your system.

That doesn't help you with dot-files in your home directory though, and honestly I'm not sure anything will. Compared to 10MB in /etc on my netbook, there's 86MB in ~/.mozilla alone for me, often in inscrutable XML and binary files. Worse, applications feel free to create their own dot files at any time, and also to hide them underneath other directories (.config/gnome-session, .gnome2/evince, .kde/share/apps/ScanImages etc). Some need to be per-machine, others don't.

You could imagine having a .bashrc that sets LD_PRELOAD to include some file in .mozilla/firefox/194653e1.default/Cache/36A45162d01, which then checks every now and then to see if it can run "sudo" without needing a password to give itself root permissions, for example. Perhaps a .bashrc and LD_PRELOAD would be noticable (though I think not to many people), but there's also .xsession and a myriad of other bits of configuration that'll let you get a trojan started up that way.

On the other hand, the amount of valuable configuration in dotfiles isn't that large -- manually deciding which dotfiles are interesting and keeping them in version control, while scrubbing the rest every now and then (when things break, when you switch computers, once a week, whatever) could be feasible.

Another place to worry about stuff is /var. It's usually a little safer in that it's generally full of data, so it won't spontaneously launch software quite so much, but not completely so. Adding something to /var/spool/cron/crontabs/root could get you into trouble pretty quickly, eg. If you modified /bin/ls to do evil things, and someone tried reinstalling with apt-get, if you'd also added some code to /var/lib/dpkg/info/coreutils.prerm you could make sure /bin/ls was reinfected immediately.

I'm honestly not to sure what there is to be done about that either. It might be feasible to monitor just the "risky" parts of /var in a useful way, but it would be pretty easy to miss things. It might be possible to classify great swathes of /var as "not-risky" and treat the other bits similarly to /etc, but I don't think there are tools to do that at present. It might be possible to get programs to move the risky bits into /etc, /usr or users' home directories, but I know people were talking about some of those things over a decade ago, so it's not likely to happen soon.

Finally, there's the disk and filesystem in general -- having /etc/shadow be world readable or having a misplaced setuid bit can ruin your whole day, and you can put a fair bit of information in extended attributes these days if you're looking to hide it from the suspicious admin. You also want to make sure your boot isn't compromised -- perhaps your bootloader is jumping to code other than the kernel you thought you were pointing at, or your BIOS firmware has some code to setup a timer and a ring-0 trap that'll take control of your kernel a little while after it's booted. On the upside, there's nothing inherently difficult with dealing with that: just reflash your system and your bootloader; all your configuration should be elsewhere in the filesystem, so that should be easy. (Whether it is or not is just a matter of how good your tools are)