Blosxom to WordPress

So a quick blog on the switch to WordPress.

The main reason I decided to switch was the ability to have comments; which you can do in blosxom with various plugins, but then you have to worry about spam yourself, whereas with WordPress comments (and comment moderation) are standard and plugins to deal with spam are pretty trivial. Basically, I like having somewhere to chat with like-minded people about various tech things that interest me, and the other places I’ve done that — notably Debian mailing lists and across blogs, just don’t come close to being satisfying anymore. Lists don’t really work, because, well Debian lists have been sucking for ages now and aren’t likely to improve, and cross-blog discussions don’t tend to be very interesting because these days the ones I’m interested in are mostly aggregated onto planets, and end up looking just like mailing lists, with the same old topics discussed by the same old people with the same old conflicts. Boring.

I thought I had a second reason, but I seem to have forgotten what it was now. The two possibilities I can think of are getting my blog compliant with web standards (doctypes, properly closing tags and the like), and doing private posts. In any event, the third reason was that I’d been using Blosxom for ages now, and it was just time for a change. It’s also nice that a bunch of other folks are actively using (not to mention developing) WordPress, and there’s been a few other folks switching to it lately.

Switching to WordPress turned out to not be too hard. Unfortunately there isn’t an importer for blosxom like there seems to be for a bunch of other blogging software/sites; but that makes reasonable sense given the whole point of blosxom is that it’s easy to hack on to make it work just the way you want. I’d customised the way I wrote links, eg. So I googled around, and came across DeWitt Clinton’s blog post about his conversion. His approach was to run a special hacked up script over his blosxom directory to generate, essentially, an RSS feed containing all his posts ever, and then use WordPress’s RSS importer.

That didn’t work for me for two reasons. One was my special plugins, particularly the funny link notation I made up and the update plugin I was using; no big deal I knew I was going to have to do something about that. The other was that my blogs weren’t remotely HTML4 compliant, so when the script tried to validate them it would complain, and even if I disabled that feature, when I tried importing the results into WordPress, it would complain.

Since I still don’t have any intention of writing correct markup by hand, finding an automatic way of fixing that was the biggest trick. HTML tidy came to my rescue though — in particular, invoking:

tidy -q -asxhtml --show-errors 0 --show-body-only auto -wrap 0

seems to convert the body of a blosxom post into a legitimate HTML chunk (as opposed to a full HTML page) with minimal dramas.

I did have a few other problems — the original script was invoking $parser->Parse( "<xml>$blog</xml>") which unfortunately is validating stuff as XML rather than HTML, which apparently means it doesn’t know about things like “&ge;” meaning ““. So I eventually figured out that adding a DOCTYPE tag, and changing the enclosing tags to <html> instead of <xml> would get me what I want. And it pretty much did.

That just left dealing with my plugins — which really just meant rewriting them as simply as possible into the script. A cleverer solution would probably invoke the plugins from the script with all the right parameters so it would work for any plugins people might be using, but I just wasn’t fussed enough. A few tests, and the final script seemed to work well enough, with the output importing properly into WordPress (right content, right timestamps, roughly right categories). The testing got a bit annoying — trying to delete all 300-odd posts to reimport the next try got annoying because I could only seem to do 20 at a time, but not annoying enough to care about. I tried not deleting them to see if WordPress would notice the duplicates and ignore/overwrite them, but it just let them appear twice, giving me 600-odd posts to delete before the next trial. Doh.

I also got rid of the whitespace compression the original script was doing, and I think that’s everything. For anyone interested, here’s my version of blosxom2wp.

As far as WordPress itself’s concerned, so far I’m pretty happy with it. It did force me to install mysql, instead of using postgresql like I’m used to — apparently there’s a fork of WordPress that does psql, but fundamentally it’s database access just isn’t very general. The Debian packages (whether you’re looking at stable, testing, unstable or experimental) were out of date, as were the Ubuntu ones, I think, and that was even before the new release (which hasn’t been uploaded to Debian either, at least as I write this). Installing it by hand wasn’t a big deal, though the packaged version does include some nice hackery to allow multiple blogs to use a single WordPress installation, which is nice, and I might look into if either David or Clinton decide they want to switch to WordPress too, or if I decide I want another (separate) blog for some reason.

What else? I didn’t like the default themes much, and switched to a pretty random and simple one pretty quickly. I guess I might try doing a customish theme some day, but probably only when I’m sure I won’t be forced into worrying about writing valid HTML or CSS by hand. The only interesting plugin I’m using is Ozh’s Better Feed (recommended via a blogger on Planet Ubuntu), pretty much entirely so there’s a link to comments included in my RSS feed.

The biggest drawback is probably the feed spamming — whether to planets or to places like Google Reader or FeedBurner. It’s a real shame the way that happens, but what can you do? As far as I can see, not much. I lost my full category heirarchy during import, I think; I seem to only have one level instead of many now. I don’t think I care though. On the other hand, I can’t say I’m too fond of the way you insert links with GUI editors — highlight link text, click link button, select url from some other window, delete “http://” string, paste the url, click ok — seems a bit tedious to me. I presume I’m missing something but I also can’t seem to put stuff in <code> tags without switching from Visual mode to HTML view, either. Weird. I guess I’m also a little worried about it’s security and such, but hopefully keeping up with upstream will work reasonably well.

On the upside, automatic support for drafts and scheduled publishing is nice, automatic revision control isn’t bad, not having to worry about accidently changing the timestamp of old posts when fixing a typo is wonderful, and having the blog engine manage pictures and scripts you’d like to include is pretty nice too. Gravatars (and identicons) are kinda cool too.

All up, I’m pretty happy with it so far.

Free The World

A little while ago I followed a link to a video of a Kathy Sierra talk (always good value). But one of the ideas therein that particularly intrigued me, particularly in the context of starting up a company, was the notion of “making the t-shirt first”. You can look at it as just compressing the “elevator spiel” into an even pithier format which is useful enough, but for me at least, t-shirt slogans also make the concept of selling/communicating your idea way more fun. And, having an idea of my own that I’m trying to nourish, here’s my slogan, albeit on business cards, not a shirt.

Free the world card

So what’s “Free the world” mean, to me?

Free Software

I’m a software geek. The idea that you can take tools and make them do exactly what you want, when you want, in the way you want, without having to do it yourself, is just amazingly cool. Want to read a book? Software can supply you with hundreds, no monks or bards needed. Want to listen to some music? Software can give you thousands of tunes, no cover band required. Want to design a skyscraper or a bridge or a jet, but aren’t sure what shapes and materials will fall to bits? Software can spare you the effort of physically putting together thousands of prototypes and running them all through complicated test rigs and risking people’s lives getting your answers. There are plenty of limitations about what you can do with actual stuff, but for software, you can literally do almost everything you can think of.

But if your software’s not free — and I obviously mean that in the GNU/open source sense, not “for no cost” — you lose the ability to just go ahead and do whatever you want. Maybe you can’t do it at all, maybe you can’t completely automate it, maybe you can’t make a variation you’d like, maybe you just can’t do it in the way you’re most comfortable. The mere fact that software’s free doesn’t mean you suddenly can do all of that, but it does mean that if you’re able to, you’re allowed to.

And that, to me, is the way things should be.

Free Trade

Another way of saying “software geek” is to say “software specialist”. I’ve spent a lot of time and effort learning about various aspects of software, and gaining skills and experience in understanding it, creating it, and maintaining it. And there’s no way I could have done that if other people hadn’t spent equal amounts of time and energy becoming specialists in other fields too — there’s no point being a software expert if no one builds hardware to run your software, or if no one does anything practical with your software. And all that only works if all those specialists are allowed to trade with each other, so my software skills can deal with a dozen people’s software problems, in return for their various skills helping me out in return. There’s a whole bunch of stuff you can build on top of that — finance, micro- and macro-economics, currencies, credit, insurance, loans, incorporation, share markets, bankruptcy protection, derivatives, whatever — but ultimately all it comes down to is the ability to go up to someone who has something you want, and find some way to make their life sufficiently better that they’ll swap you for it.

And fundamentally there are two reasons why that’s a good system: first, it’s fair: if you want people to help you out, you help them out; and second, it works: economic freedom and quality of life go hand in hand in this world, with plenty of failed attempts to find counter examples littering history. But it’s not like we’re not still making more failed attempts, whether in the guise of social democracy, green revolutions, capitalist bailouts, nanny states, political correctness, child safety, and whatever else. There’s lots of good reasons behind all those things, but stopping people from doing favours for each other isn’t the right way to go about it.

There should be more people helping each other out in the world, and benefiting from their own generosity,  and fewer people getting in the way of that.

Freedom of Choice

But you know, not everyone thinks the same: there’s plenty of people out there who don’t give a damn about free software, or free trade. Because they think other things are more important, because they think other approaches make people better off, whatever. And, in spite of the fact that I’m (clearly) right and they’re (obviously) wrong, that’s all fine. Because you can’t be free unless you’re free to make mistakes, free to be wrong. Software’s not free if you can’t add bugs as well as remove them. Trade’s not free if you’re not allowed to buy things that end up making you unhappy. So while I’m all for the above, I’m also all for other people having different ideas, and opposing ones. More power to them — and hey, the greater the opposition, the greater the eventual triumph, right?

So yeah, that’s my mini-manifesto, and the completely non-specific overview of what I’m focusing on for now.

FTW card

Oh, and I don’t half mind the acronym to which “Free The World” contracts, either.

(Two other random notes: yes, that is a creative commons copyright statement for the picture on the front — I grabbed a photo from flickr, that turned out to be by Corey Leopold; and properly credited reuse of some free imagery seemed a perfect fit for the sentiment I was going for. The cards were printed by clickbusinesscards.com.au, whom I’ve found to be nothing but awesome. Mad props)

An event

What’s the difference between a company and a business?

One’s an almost meaningless bit of paper, the other’s a way of making lots of money.

Me, I now have a company!

Banks and financial literacy

It’s a bit funny, but I kind-of think banks are a huge impediment to financial literacy. There are two key insights that could be really useful to Joe Public (or, for that matter, me, say ten years ago…) that banks absolutely mess up.

Credits and debits

One is basic double-entry accounting. In double-entry accounting, you enter every figure twice, once as a “credit” against one account, and again as a “debit” against another account. Your accounts are generally divided up into “assets” (stuff you have), “liabilities” (stuff you owe), “income” (to keep track of how much you’ve earned), “expenses” (to keep track of how much you’ve spent). And at the end of the day, you know you’ve done your maths right if you get the same number of debits and credits. And in general, on the accounts that track how much money you actually have, or what you owe, a debit is a good thing (ie you have more money, or less debt), and a credit is correspondingly a bad thing. When you pay a phone bill, you credit your bank account when you withdraw the money (bad, because you have less cash), and debit your phone account (good, because you owe less money).

And of course the bank, in keeping its records, does the opposite — they debit your account (good, because they owe you less money) and credit their cash account (bad, because they’ve had to give away actual money to the phone company).

But then they send you a statement which comes from their perspective — they list the transaction as a debit (from their point of view) instead of as a credit (from your point of view — or at least your accountants point of view). Now maybe you paid attention in Commerce 101, and this is all familiar: but consider how regularly people get letters from banks implying credits are good during their lifetime, compared to how much attention they might pay to whatever bookkeeping instruction they may have once had.

For completeness, I should note that there’s an exception to the “debits good, credits bad” rule that comes up in “income” and “expense” accounts — when the bank charges you a monthly fee, they’ll annotate that internally as “debit your account $5” (good, they owe you less money), and “credit our bank-fees income account $5” (also good).

Assets and Liabilities

If you’ve read Rich Dad, Poor Dad, you can probably skip this section. One of the most interesting parts of that book, for me anyway, was the redefinition of “asset”. Instead of meaning “something you could sell for real money”, the author (and the “Rich Dad” from the title) use the term to mean “something that will give you money just for owning it”. So shares in businesses, rental properties, interest earning deposits, bonds, etc. The key difference is when you think of things like cars or tvs or game consoles: sure, you could sell them if you had to (which makes them an asset in one sense), but you’re more likely to keep them, and have to spend more money to keep them interesting (on fuel and services, on cable tv subscriptions, on buying or renting games).

And really, the latter is a much more valuable perspective: sure, you might look around and see lots of things you could sell for a bunch of cash, but odds on you aren’t going to do that unless you get really desperate. But if you look around and see yourself surrounded by things that are continuing to cost you money, you might look into doing away with some of those, and if you can find things that will actually make you money (or can be converted into making you money), well, then you’re taking proper advantage of the capitalist economy.

But, again, banks don’t really give a damn what’s in your interests here — from their point of view assets are things that they can claim and sell off if you go broke. And how many times does the bank’s point of view get presented to your average Joe, compared to the Rich Dad’s?

Already obsolete…

Wow, I switch on the rewriting to make the switch from blosxom to wordpress actually happen, and what do I see when I go to blog about it? “WordPress 2.7 is available! Please update now.” I guess this is what you have to expect when you use software that’s actively maintained…

UPDATE 2008/12/12: Okay, 2.7 was kinda worth the upgrade at that. Pretty results, and a remarkably inoffensive procedure.

A new home…

inamerrata will shortly be moving to a new home, with a wordpress backend instead of a blosxom one. I suspect that means a little Planet spamming will be forthcoming, when I add the redirect rules to move the old posts/feeds to the new urls… On the upside: no more “Read more…” links, and comments! Truly a brave new world.

So what?

I know LCA has had a policy of treating its speakers like rock stars in the past, but sometimes I wonder if that gets taken a bit far, you know? Like, who can even tell the difference anymore?

rockstar hacker
P!nk, rock star Valerie Aurora, kernel hacker

I mean, just imagine what confusion could ensue if we let this get out of hand.

F$ck - So what?

Na Na Na Na Na Na Na
Na Na Na Na Na Na
Na Na Na Na Na Na Na
Na Na Na Na Na Na

I guess I just lost my data
I don't know where it went.
So I'm gonna kick the power,
This sure gonna leave a dent (Yup)
I got a brand new VFS,
And I'm gonna try it tonight
I wanna get in trouble
I don't wanna leave a byte.

Na Na Na Na Na Na Na
Not gunna leave a byte.
Na Na Na Na Na Na Na
I'm not gunna leave a byte!

So, so what?
I'm still a hacker,
I got my l33t moves
And I don't need you.
And guess what!
I'm having more fun,
And now that it's gone
I'm gonna show you tonight:
It's alright, boots just fine
And you're a loss.
So, so what?
I am a hacker
I got my l33t moves
And I'll rewrite you tonight!

Uh, check my flow, wawh.

Init scripts just checked my fs,
And orphaned my music and source- Shit!
I guess I'll go sit with Rusty,
We'll reimplement a hit.
What if I've lost all my backups?
Then somebody's gonna die.
They're gonna get in trouble
They're not gonna keep a byte.

Na Na Na Na Na Na Na
They're not gonna keep a byte.
Na Na Na Na Na Na Na
We're all gonna lose all our files!

So, so what? / I'm still a hacker / ... / And I'll rewrite you tonight!

You weren't there,
You never were
You lost it all
And that's not fair
I gave you life
I gave my all
You weren't there,
You fscked it all...

So, so what? / I'm still a hacker / ... / And I'll rewrite you tonight!

No No, No No
I'll rewrite you tonight...
You weren't there...
I'm gonna have to rewrite...
It's alright, boots just fine...
And you're a loss.
So, so what?
I am a hacker,
I got my l33t moves
And I'll rewrite you tonight.

Ba da da da da da

Interest incrementally

I was looking at bank accounts and interest rates the other day — such as the Virgin Super page that lists every single fund returning a negative percentage, except for cash and one of the funds available to over-60s. It seems really hard to compare percentage rates, for example if you gain 10% on year, then have a rate of -10% the next year, you’re not actually square, you end up worse off. But to really work it out, you have to multiply it out — “1.10*0.9=0.99, oh I’m down a percent, damn”.

At some point it crossed my mind that working with exponents would be much more sensible — rather than multiplying, you’re just adding which is a lot easier to do in your head, and compounding just falls out naturally, rather than being horribly confusing. So creating a new unit, “i%”, an incremental percentage improvement, where “1 i%” is the same as a 1% return, and “2 i%” is the same as a 1% return on top of a 1% return (ie, 1.01*1.01=1.0201, so 2 i%=2.01%). The formula for going from an i% to a percentage interest rate is straightforward, it’s n i% = 1.01n%. Unfortunately the formula for getting the i% in the first place is more complicated, it’s r% = log(1+r/100)/log(1.01) i%.

Some particular values:

100% = 69.661 i%
(what it takes to double your money)
13% = 12.283 i%
(the low end of current credit card rates)
6% = 5.856 i%
(current savings interest rate, if you’re lucky)
5.25% = 5.142 i%
(current Reserve Bank policy rate)
0% = 0 i%
(what happens if you don’t get anything)
-0.9901% = -1 i%
(the negative interest rate that exactly cancels out a prior 1% profit)
-13.36% = -14.412 i%
(12 month performance on Virgin’s 100% growth agressive fund)
-50% = -69.661 i%
(what it takes to halve your money)

That, to me, seems like it makes comparisons a lot easier. If you’re getting a flat interest rate of 5% is it better or worse to change to a 2.4% interest rate compounded twice? 5%=4.903i%, 2.4%=2.383i%. Double the latter because you get it twice, and you’re at 4.766i%, which is worse off. 2.5% on the other hand would be 2.482i% which doubles to 4.964i%. If you get 6% for three years, then -13.36%, what’s that cumulatively? 5.856 + 5.856 + 5.856 – 14.412 = 3.156 i% (or a 3.19% improvement). What’s that as an annual rate over four years? 3.156 / 4 = 0.789 i% (or a 0.788% pa average). If you want to work out how long it’ll take you to double your money at 6% interest per annum? 69.661/5.856 = 11.9 years.

Anyway, that seemed like an interesting (and better) way of comparing things to me than what people usually put up with, YMMV.

Respectful Disagreement

Early last month there was a short piece by Robin Hanson on his group blog titled Disagreement is Disrespect:

But this does seem a handy opportunity to repeat that while disagreement isn’t hate, it is disrespect. When you knowingly disagree with someone you are judging them to be less rational than you, at least on that topic. (Judging them less informed or experienced by itself can’t create disagreement.) It might be only a minor disrespect, if you think this disagreement suggests little about whether you’d disagree with them elsewhere. But disagreement is disrespect, nonetheless.

Taking that to an extreme you come up with the obvious conclusion: if you want someone to respect you as much as possible, you should always agree with them. Because there’s nothing better than a coterie of yes-men.

Avoiding that conclusion means you have to have some way of respecting people who don’t constantly agree with you; and more so, respecting those people more than the ones who simply repeat whatever you happen to think. But if you really believe that your opinions are entirely rational — that is, they logically follow from unquestionable assumptions — you don’t actually have much room to accept disagreement. Because either whoever disagrees with you is being illogical, which doesn’t deserve respect, or they’re denying basic assumptions, which likewise doesn’t deserve respect.

Take abortion for example. If you believe that human life is sacrosanct (and what sort of person doesn’t?) then it’s a simple deductive analysis to say that a foetus is both alive and human from conception, albeit dependent on it’s mother’s womb for the next nine months. Abortion means choosing to end that life, which is murder. Sure, you can pretend you don’t follow the argument, but that just means you’re either stupid, or in denial — which leaves your choices down to (1) being okay with murder, (2) being in deliberate denial; (3) being stupid; (4) agreeing that abortion isn’t okay, ever.

The question then is how you can possibly respect anyone who chooses any of the first three options. And framed like that, it’s pretty hard — even if you’re not a strict pro-lifer, the most comfortable response is probably to try to fit into the fourth slot, maybe by saying that you don’t like late term abortions.

But maybe you do think the mother should have a choice in the matter (we’ll politely ignore whether the father should or shouldn’t), after all it’s her body that’ll be distended for half a year, her ability to function in society that’ll be diminished, her hormones that’ll go haywire, and her health that’ll be put at risk gestating a possibly unwanted parasite. Because that’s the key difference between a foetus and a baby, child or adult: they’re completely dependent on a single woman. At other stages in life, dependency is something that can at least be transferred to someone else, from a parent to a lover, or to a private hospice, or a church, or a hospital, and if no one at all is willing to look after someone who needs looking after, well, vagrants still die on the streets, and it’s sad and maybe someone should do something about it, but it’s not murder. And just because a woman’s biology is designed (by God or evolution as you like) to try to force them to provide the necessary support for the new life, it’s surely not immoral to allow a woman to use technology to safely reclaim that choice, and if it’s not okay for me to force you to take a bum from the street into your home to take care of, even for a few days, how can it possibly be okay for you to force someone to look after a baby in their own body for months?

This time, as far as I can see, you either (1) don’t accept people should have a choice over who they support; (2) you’re in denial; (3) you’re illogical; or (4) you support abortion. So which is it, are you in favour of murder or tyranny?

Personally, I don’t much care for either of those; though in the special case of abortion, my preference leans towards morally opposing it and legally supporting it at least early-on, but I’m not particularly bothered by people who take different views. (Well, favouring abortion over abstinence, contraception and pregnancy would weird me out, but I don’t think I’ve seen anyone go quite that far) Personally, I’d like to see a technical solution in the form of an artificial uterus, which would at least potentially move the “adoption” option to a much earlier point in the process. Making abortion illegal, but offering transplant of the foetus into an artificial womb to be cared for and put up for adoption would seem like a good option to me. Though the cost and its effect on live birth rates might be challenging.

But if you’ve decided that anyone who disagrees with you isn’t deserving of respect; that it’s immoral not to make the assumptions you’ve made, and stupid not to deduce your conclusions from those assumptions, then you’ve got no incentive to consider alternative scenarios like that, after all, everyone who doesn’t agree with you is immoral or stupid, and how could it possibly be a good idea to co-design a solution with people whom you now know are evil idiots? [palin shirt]

That conclusion seems to pop up a fair bit in politics, whether it be while you’re trying to understand the McSame/Failin Rethuglican holdouts or the handful of remaining undecided voters or the America hating socialist atheists supporting religious extremists. I presume it’s always been more or less that way, and I guess I’d rather see people trying to cut each other to the quick with the force of the rhetoric rather than by force of arms, but I can’t say I like it.

It also seems like it’s well and truly seeped into Debian, too. The whole “evil cabal” story has been around forever, sure, but at least back in the day you could have real disagreements with people and still respect their motives and work together on a solution. Here’s a selection of comments on the latest kerfuffle:

I’m pretty unhappy with the very non-Debian way you have when it comes to making decisions and announcing them. .. And also that feeling you seem to have that you are above the lot of mere mortals that we, DDs without delegations, are.

In my mind I can’t conceive of why they would have willingly accepted power if they weren’t prepared to face the highest scrutiny and criticisms. Can you?

People with too much power are winning. A lot of of the average contributors have good points about all this, but they already gave up (this is a hobby for most of all!) and do nothing because they are already too frustrated. Some try arguing and discussing, but it is not going anywhere because people of the problem B are playing to ignore everything, so you can not argue with silence.

When you thought things were bad enough already, they just got worse. We now have people coming up with decisions all by themselves. No asking anyone about it, even in backroom meetings, or only to fake it and ignore their opinions on the matter. .. It looks bad, and it is. Talk about communication problems. Talk about power-hungry people.

Nothing will have changed, nothing will be taken away, nothing to see here, move along: The big lie.

Nice how some people (on purpose?) try to misunderstand everything. And love to flame.

Maybe it’s a failure of imagination, but I can’t see the attitudes embodied in those sentiments inspiring constructive collaboration amongst the various Debianites who have different priorities and visions for Debian, as opposed to a destructive and destabilising fight over who ends up with enough power to force their choices on everyone else.

And honestly, I don’t think there’s any chance of avoiding that at this point, either in politics or Debian. Because once you’ve decided the evil and stupid people are trying to wrest power from you, you can’t trust people who say you should “respect” them, because that’s probably just an evil trick to make you drop your guard just long enough for them to win.

Oh well. Two simple reasons to respect people who disagree with you. One: by challenging your beliefs and arguments, they give you the opportunity to refine and improve them. Two: in the unlikely event that you’ve made a mistake in your analysis, they’ve already done the groundwork for an alternative analysis that you can adopt or adapt to replace your earlier flawed ideas.

Costs and Benefits of Voting

Economics has lots of odd results. Many of them are deeply insightful, like the principle of comparative advantage, with the potential to improve lots of people’s lives when understood and acted upon. On the other hand, some of them are just bizarre, which for my money includes the claim that voting is irrational. Basically, the argument goes that your individual vote is unlikely to make any difference (it’s pretty rare that there’s going to a final margin of a single vote after all), and since you have to spend time and energy to vote, it’s just not worth the effort.

Now you can recover that with things like the categorical imperative (if everyone did it, things wouldn’t be so rosy, would they? therefore you shouldn’t do it), but that’s not terribly persuasive — and besides, even when everyone else votes, it still seems intuitively good that you vote too.

There are some other easy ways to avoid the conundrum too: compulsory voting means you’re forced to pay the costs of voting either way (by voting, or by paying a fine), so even a trivial benefit makes it worthwhile. And you might argue that getting an “I voted” button, or the feeling of participating in democracy is worthwhile. Or you might argue that your vote isn’t just about who wins, but also helps finance their future campaigns (to the tune of about $2 per vote in Australia), or you just think the final margin of victory will affect the mandate the winner can claim, and be useful on that basis.

But again, intuitively, I’d like to believe it’s still a good idea to vote even without any of those factors, and even if most everyone else votes. And personally, I prefer it when my reason either matches my intuition, or doesn’t just suggest a different approach, but can show why the approach my intuition suggests is actually harmful.

With that in mind, I found Robin Hanson’s post on Noble Abstention pretty fascinating. It basically assumes something like the worst case for voting — everyone who’s already voted is smarter than you, and everyone who votes after you isn’t. It also assumes that nobody is perfect at picking the best candidate, and that people get exponentially worse at doing so. By varying the degree of the exponent, Robin finds that for exponents above a half, the best candidate is most likely to win when only one person or a small handful of people vote, and for exponents below a half, the best candidate is most likely to win when everyone votes.

The analysis is somewhat more interesting when you look at individual incentives: whether your vote will make the best candidate (who may or may not be who you vote for) more or less likely to win. In the cases he examines, with exponents above about 0.7, assuming both that each voter is (very slightly) more likely to vote for the better candidate, and that everyone else is voting, including everyone smarter and stupider than them, half of the ten thousand voters are actually in the position where their vote will actually decrease the liklihood of the better candidate winning.

To be clear: you’re probably going to vote for the better candidate. The better candidate is already likely to win. Of the 9,999 other people who’ve voted, you’re smarter than 4,999 of them. Yet your vote will decrease the odds of the better candidate winning compared to you not voting.

That seems pretty unintuitive, but it actually falls out from a very particular arrangement of probabilities. Namely, your vote will make a difference in two circumstances: you vote for Alice, when she has just under half the votes, or you vote for Bob, when he has just under half the votes. If P(A) are the odds you vote for Alice, and P(n for B) are the odds Bob has n votes currently, then assuming there are 2n other voters, your probabilities are:

  • P(n for A) * P(A) — you win the election for Alice
  • P(n for B) * P(B) — you win the election for Bob
  • 1 – P(n for B) — your vote doesn’t end up mattering

(Note that there are n voters for Bob precisely when there are also n voters for Alice)

If there are 2n+1 other voters you can’t force a win, but you can force a draw, and your probabilities are:

  • P(n for B) * P(B) — you stop Alice from winning the election
  • P(n for A) * P(A) — you stop Bob from winning the election
  • 1 – P(n for A) – P(n for B) — your vote doesn’t end up mattering

And the problem is that you’re making things worse when P(n for X)*P(X) is lower for the better candidate than for the worse one — which isn’t only true when you’re more likely to vote for the worse candidate (ie, P(X) is lower for the better candidate), but also when P(n for X) is lower.

But that in turn depends closely on the distribution of everyone else’s likelihood for voting for the better candidate. Robin’s assumptions naturally imply a particular range of distributions that cover both possibilities — where the vote distributions of prior voters sets future voters up to prevent their votes from being useful; and where the vote distributions of later voters keep improving the odds of the better candidate winning. There are plenty of other possible outcomes too; what assumptions might be valid depend a lot on how much information you actually think voters can obtain about the candidates and how much pre-existing biasses might prevent that information from making it through to the election, and how much you think people differ in their odds of picking the better candidate.

But beyond that, it gives an interesting way of analysing whether or not it’s worth voting, even under the assumption that your individual vote might not make a difference, and that you might end up voting for the worse candidate. Taking Robin’s distribution, for example, with an exponent of 0.5, gives the 10,001st voter a 50.05% chance of picking the better candidate, and overall odds of 57.869279% of the population selecting the best candidate. Based on Robin’s assumptions, the first voter had a 55% chance of doing that alone, and the following 9998 voters improved that to 57.869274% (ie, by 2.869274%), leaving the last two voters (#10,000 who could only force the election into a draw, and #10,001 who could actually provide a winner), to improve on that by 0.000005%, or 1 in 20,000,000 times, or 1 in 40,000,000 times each.

So supposing you’re the dumbest person in an electorate of 10,000 people, and your odds of picking the better candidate are only better than tossing a coin one time in 2000, then it’s worth spending half an hour voting, if you think the difference between the two candidates is enough that improving the better candidates chances by 1 in 40,000,000 is worth the effort.

In general, if there are n voters, and an overall chance of electing the best candidate of p, the average voter can claim a contribution of (p-0.5)/n. If you want to be pretty generous, and say the best candidate wins 90% of the time, the average voter in an electorate of 50,000 people might be able to claim to make a difference 1 time out of 125,000. If you think it’s only 75% of the time, you’re down to 1 time out of 200,000. If you think the 58% that resulted from the previous calculations sounded pretty good, you’re down to 1 time out of 625,000.

For comparison, the chance of winning Oz Lotto is reportedly about one in four million, and the size of Australian Federal Electorates tends to be around 90,000 people.

So the question is, do you think you’re above average in picking good candidates? Do you think the difference between the candidates is noticable? And if both of those are true, do you think that a one in 125,000, or one in 625,000, or one in 40,000,000 chance of ending up with the better candidate is worth the effort of voting?

To give a concrete example: if you think Robin’s model for your electorate is vaguely plausible with an exponent of 0.5, and you’re in the top 10,000 voters, and you think choosing the better candidate might save the country a billion dollars (ie, thousand million), then the one in forty million difference your vote makes is worth $25 to the country.

There’s a lot of variables there, but I think it’s a plausible economic analysis: Robin’s structure seems somewhat pessimistic to me, a billion dollars’ difference between candidates seems within the realm of plausibility on a national level and possibly even a state a level, and saying your vote is worth $25 to the country matches the amount you’ll be fined in Australia if you don’t vote ($20).

I think that also ends up as a potential justification for “compulsory” voting: given a sufficiently precise model of how much additional information each voter brings to the election, and how much variance there is between candidates’ performance, you can calculate a monetary value for how much each person’s vote is worth to the country, and assuming that has a low variance (eg, if it turns out to be between $15 and $25 for everyone in the country), it would then be reasonable for the country to pay everyone who votes the average amount, or if the country’s funded by taxation and fines anyway, to fine everyone who doesn’t vote that same amount.

And if you have economically justified compulsory voting, the individual economic analysis for whether it’s rational to vote becomes simple: is the nuisance of voting worth the cash reward, or worse than the nuisance of the fine as the case may be?

But again, it depends heavily on how the population’s electoral nous is distributed — the value of people’s votes might range from $19.99 to $20.01 or from $0.02 to $100.00, or from -$50.00 to $10,000. As an egalitarian, I have an a priori assumption it doesn’t vary much. Your mileage may vary.

Diseffected

A photographic portrait of Keith Packard writing his lca abstract:

Lunch

Lunch for today. Recipe:

  • Lamb forequarter chop (200g). Cook on frypan ’til medium, sprinkle with garlic salt to taste.
  • One mandarin, peeled, broken, scattered.
  • One cavendish banana, peeled, sliced, scattered.
  • Blueberries.
  • Strawberries.

Total time to cook, consume, clean, and blog about an hour. Total cost, about $5. Missing were mango, cheese and beer. Oh well, maybe next time.

Lunch
You’re doing it wrong
You’re doing it right
Omnomnom… Oh hai! What’s up?
  
pollcode.com free polls

On the banking collapse

I’ve been trying for a while now to find someone saying something sensible about the current financial shenanigans. It’s quite difficult. First you’ve got to avoid getting distracted by the various folks who’re going to claim it’s a crisis even if it isn’t — newspapers looking for some bad news to run on page one, politicians figuring they’ve finally found the problem to which they’re the solution or just looking for something to blame on the other guy; people who see an opportunity to reap huge profits by scaring everyone else; and people who’ve got no idea what they’re talking about, but feel obliged to say something anyway.

Speaking of which…

Drilling down to what actually happened helps a little, as far as I can tell.

Fannie and Freddie

This seems to be the most political aspect of the whole mess. Fannie was setup in 1938 by Roosevelt as part of the “New Deal” to help make mortgages work better by providing a secondary market (you take out a mortgage with your bank at 8%, and your bank sells 7.25% of that to someone else so they get (less) money now, but don’t have to worry (as much) if you default). In 1968, the government decided they didn’t want Fannie on their books anymore, so fudged it by “privatising” it as a “Government Sponsored Enterprise”, and in 1970 created Freddie Mac in a similar role so they could say there was competition.

Both those companies had the benefit of (probably) being backed by the US government and being “too big” to fail — so, whereas you might worry that some bank is going to default on its loan if you lend it money, you can be sure that Fannie and Freddie won’t — especially if you’re collecting in US dollars. That means Fannie and Freddie get money more easily than their competition in the US market, and that they thus end up (well, continue) dominating the mortgage market. Which in turn means anyone else who wants to be involved has to play around the margins, by taking on more risk with sub-prime loans, eg. Apparently they even have a term — “NINJA” — for loans made to people with “No Income, No Job or Assets”.

You can then add to that some fairly seedy political practices, including campaign contributions (including illegal ones) from Fannie and Freddy, a Fannie exec sleeping with a member of the House Banking Committee who was encouraging Fannie’s deregulation, and a general push from the political side of the fence to helping poor folks buy homes.

And what do you get? A large risk, that’s been left off the taxpayers’ books for 40 years, that’s been pushed too far and come painfully due.

Mark to market and short selling

Valuing assets is often confusing — if you paid $500 for a bike two years ago, what’s it currently worth? $500 because it’s “as good as new”? $300 because that’s what you’d pay for a replacement bike? $150 because that was the high bid on ebay? $50 because that’s what the guy who stole it got when he sold it from his trailer? $600 because that’s what you had it insured for?

Ultimately you can use any of those prices, or anything in between that you can find some convincing excuse for; and cooking the books like that is a fine art that’s been perfected over the years to avoid taxes, trick co-owners, and befuddle investors. The simplest way to avoid it is to say “it’s what someone else will pay for it” — so $150 in this case, and that works pretty well on fungible assets: commodities, stocks, pineapples, etc.

The downside is what happens when not many people are trading — if you have a million shares outstanding in a company, but only one person is interested in trading at the moment, they can sell at $1, and imply your company is worth a million dollars, or they can sell at $1.50 and make $500,000 appear from nowhere, or $0.50 and make half a million disappear. And the only difference it makes to them is 50c either way.

Short-sellers take advantage of that and the fear that comes with thousands or millions of dollars appearing to disappear, by driving down the prices of shares making it look like the asset is worthless, so others start selling too. By borrowing a share, selling it below market rates, and then buying it back at even lower rates as others start piling on before they have to give the share back, they win big for a tiny initial investment.

It’s all pretty scary and annoying when it happens, but as far as I can tell, it’s not particularly unusual, or poorly understood, or worthy of being called a global crisis just because it happens to houses, banks or loans.

Credit default swaps and complex interdependencies

It gets substantially worse for the banking system for two reasons. One is a lot of their assets are in the form of credit default swaps (reportedly worth more than the entire annual GDP of the US), which tend to not have standardised terms aren’t actually traded on a market. Which ends up meaning that there isn’t a “market price”, and if you’re trying to make up a market price, you’re even more likely to hit the “barely and traders” scenario, with a 50c change in trade price costing you millions.

Worse, the nature of “credit default swaps” is that it ties banks together — if someone defaults on their loan, it doesn’t just impact one bank, it impacts the bank they’ve done the CDS with too. To be fair, that’s the point: rather than a big hit to one bank, it’s a small hit to two banks. But add enough of them at once (because people suddenly can’t refinance their loans, eg) that one bank is hit enough to go under, and they not only can’t sell off their CDS assets (because there’s no market), but the CDS that they’re counterparty too possibly become worthless, increasing the risk for other banks and creating more uncertainty and risk, and an industry wide crisis.

In short, risky, complicated, highly interdependent, non-redundant systems are a bad idea. (What, did we learn nothing from Jurassic Park?)

Fractional reserve banking

As far as I can tell, the way in which banks have a disproportionate impact on the economy compared to any other business is by holding a fractional-reserve when offering loans, and thus increasing the commercial paper money supply. All of which is to say that when you deposit $500 in a US bank, they go ahead and offer a loan of $450 to someone else, while still telling you that you can withdraw your $500 at any time. And when someone accepts that loan offer, and spends the $450, it ends up in someone else’s account at some bank, and $405 is offered in another loan, until your $500 magically becomes $5000, spread across dozens of accounts. Which is fine until you have a bank run, and everyone tries to withdraw all their money, when there’s only $500 actually there, or when people start defaulting on their loans more than was planned for, and your bank doesn’t recover enough money to pay back the $450 of yours it loaned someone. (The $500/$450/$5000 numbers are based on the 10% fractional-reserve that the US requires — other countries have different requirements, or (in the case of Australia and the UK, apparently) none at all)

This isn’t anything special, but it does create inflation, of a form. Rather than thinking there’s $500 in the economy (or $500 billion), people think, based simply on looking at their bank balances, that there’s ten times that amount. And if you’ve got $5000 (or $5 trillion), you’re going to be willing to spend a little more than if you’ve only got a tenth that. But if you suddenly discover you don’t have that much money — because your bank’s failed, because you can’t get a loan you were relying on, or because your currency has deflated; you need to tighten your belt and spend less. And when you spend less, other people sell less, get paid less, and spend less themselves, which gives you deflation — and in so far as people are more annoyed at having their salary cut than finding things have gotten a bit more expensive, deflation is worse than inflation.

I’m presuming, though so far I haven’t seen any actual evidence, that the problem with all these sub-prime mortgages and such is impacting the reserve banks are holding, which is in turn reducing the loans they can make, and causing a cascading contraction of the money supply. It might be enough that they’re simply not lending to other banks due to trust issues (or fears their current outstanding loans won’t be repaid on time) — effectively increasing their reserve from 10% without being officially required to, and that causes deflation too.

Naked shorting

You can also get inflation (and hence deflation) with shares too: all you need is the ability to make shares appear from nowhere. Which is what happens in naked short selling: rather than borrowing a share, selling it, then buying another back to return in its place, you just sell a share you don’t have, and buy one later, and hope either that the computer system doesn’t complain, or that you don’t have to deliver the share until you’ve bought it.

But for the time between the sale and the purchase, you’ve tricked everybody else into thinking there are more shares around than there actually are — if there were a million shares outstanding, and you’ve sold ten thousand shares you don’t have, then suddenly there’s a million ten thousand shares outstanding, and since the company’s still worth just as much as it used to be, each share’s suddenly only worth 99% of what it used to be. And if someone offers to sell for that, you get to cancel your short, and claim 1% of 1% of the company’s entire net worth. That only works if people don’t factor in the fact that you own a negative number of shares, but not only is short selling all about profiting from other people’s confusion anyway, but it’s hard for people to tell how many shares are outstanding in total, but easy to tell how many are currently up for sale, and that’s exactly what you’re influencing.

I haven’t seen any evidence that’s had anything to do with anything — at least other than the recent knee-jerk banning of short selling, both naked and otherwise, but it seems interesting and somewhat worrisome in and of itself.

The bailout-come-rescue

Well, it’s not like there’s just one — there’s the Fannie/Freddie buyout, the AIG buyout, the TARP programme, and various “liquidity injections” around the rest of the world too. It’s not at all clear to me what any of that will actually achieve; it could go anywhere from making a profit for taxpayers by buying currently underpriced assets that will recover in the near future, to being used to inflate prices of worthless assets in a grand maneoveur to transfer billions of dollars directly to bankers and financial portfolio managers and the preferred projects of the current political elite for nothing more than an assertion of “increased confidence” from those same folks.

My guess is it probably really is necessary — without it, there likely would be a real contraction in the money supply, and the changes in prices necessary to accommodate that (ie, making cars cheaper because people can’t get car loans; lowering incomes because people aren’t paying as much) would cause a lot of difficulty. Worse, deflation exaggerates bank balances; so if you’re already rich with a big bank account, you’ll be able to buy more with that; if you’ve already got a big credit card debt, that’ll be harder to pay off. While that might be all nice and anti-consumer, it’s also a pretty regressive transfer of wealth from the poor to the rich.

The FDIC

The US (and apparently most other countries except Australia) offers deposit insurance. That is, if you put money in a bank backed by the FDIC, and it goes bust, the government will make sure you get your money back (up to $100,000 anyway). There’s been talk of increasing that to $250,000 as part of the bailout, but I haven’t paid enough attention to see if (or when) that’s actually happening.

That sounds all nice and rosy, but like everything it has a downside. Here’s one way to exploit it.

First get a few patsies to run the bank for you — it’s designed to fail (otherwise the FDIC isn’t going to matter!), so you need someone (else!) for people to blame, and who can do any jail time or have their assets confiscated, etc.

Second, setup a high-risk, high-return venture, and use that to offer ridiculously high interest rates on deposits, say 15%.

Third, make sure that it’s easy to explain why what you’re doing is good, say “we’re helping the poor and underprivileged start businesses”, but hard to explain why what you’re doing is high risk, say by hiring a lot of clever econometricians to create complicated derivative products. That way you can keep politicians on your side, and avoid the public getting wise.

Fourth, get lots of folks to invest up to the FDIC limit, and make high-risk, high-return loans, right up until you get too many defaults and your patsies have to declare bankruptcy.

Finally, pay off all your creditors, issue a press release about how some people just can’t be helped no matter how generous you try to be, and collect your original investment plus 15% direct from the government.

Lessons?

So, as near as I can tell, the lessons are probably these:

  • don’t quasi-privatise things — you just get the worst of both worlds
  • don’t leave things off the books — they’ll only get worse for the lack of attention, and they will reappear eventually
  • systemic failures happen, internal redundancy is important
  • banks are important to the economy and need special care
  • the bailout is probably unjust, but doing nothing is probably even worse
  • just because things sound like a good idea doesn’t mean they’re not hurting more than they help

Of course, don’t expect anyone to learn any of these lessons: the Federal buyout of 80% of AIG is both a quasi-nationalisation and specifically scaled to stay off the government’s books; redundancy is the same as inefficiency and no one likes that; everyone knows banks are evil; the bailout’s just in time to be fodder for the most important election ever; and that certainly means no one’s got time to work out what is good rather than what sounds good.

On perseverance

As part of trying to convince myself to commit to my next project I spent some time wondering just what it might mean if the project really is beyond my abilities. At what point does it make sense to say “don’t even try in the first place?” or “okay, you’ve given it a fair shot, don’t throw good money/time/whatever after bad” or “pshaw, don’t be a quitter”?

Most of the time that’s not terribly hard — you work out if it’s within your abilities, and if it’s not you don’t bother, and if it is, you do it, figure out if you made any mistakes and fix them, and take a bow. But when you get to the very edge of what you’re capable of, just fixing your mistakes can be complicated enough that you can make mistakes there too, and if you make more mistakes fixing those…

Back in high school, we had a few ex-students return as speakers every now and then to tell us how they were Rhodes Scholars or otherwise awesome and how we could be almost as awesome if we studied hard and listened to our teachers. Okay, I’m not sure if they actually said that — but presumably they did. All I really remember from any of them was a general vibe of “success isn’t easy, it takes a lot of hard work, and not giving up, even when it seems sensible to give up”.

But having also been playing with gnuplot lately, I figured why not quantify that? Take two parameters, s representing how smart you are (or, more particularly, the probability you will succeed on any particular attempt), and p representing how likely you are to persevere (or the probability that having just failed you’ll try again anyway).

Your probability of ultimate success, assuming you try, is then, S, is s+(1-s)pS, which, if you do the maths, works out to be s/(sp+1-p). What does that mean? Well, that’s where graphs come in:

gnuplot> set isosample 21
gnuplot> set hidden3d
gnuplot> set xrange[0:1]
gnuplot> set yrange[0:1]
gnuplot> set zrange[0:100.1] 
gnuplot> S(s,p) = s/(s*p+1-p)                              
gnuplot> splot S(x,y) * 100, 90

That gives you a red curve representing the percentage change you’ll succeed for a given level of smarts and perseverance, and a blue plane at 90%. One thing you can see from that is with zero perseverance (you’ll try once, but never again), you have a straight line matching your smarts. And you’ll have a 100% chance of success if either you’ve got a 100% smartness level (you’ll always get it first time), or you’ve got a non-zero smartness (some chance of success) and 100% perseverance (you’ll never give up ’til you succeed).

But perseverance as a probability like that isn’t very meaningful; a more intuitive idea is how many attempts you’ll make — just one or two, or ten or twenty, or a hundred? You can relate perseverance and number of attempts, n with a forumula like P(n,p) = pn where P(n,p) gives you the probability of making at least n attempts given a perseverance of p. Going for a probability of at least 90%, and rearranging gives p=0.91/n.

That graph looks very different — and given how much of the red is above the blue, a lot more promising. And the reason is pretty simple — in this graph, 90% of the space (from n=1 to 10) is assuming there’s a 90% chance you’ll have two or more attempts, whereas in the previous graph, 90% of the space (from p=0.0 to 0.9) was assuming there was less than 90% chance of having a second attempt. For p=0.5 half the time you’ll give up after the first time, and, on average, you’ll make just two attempts in total.

Of course, if you make fifty attempts, and finally succeed, does that mean you’ve been very persistent (p≥0.986 probably), or does it mean you’re a huge loser (98% failure average)? Or both? That’s not immaterial. If you’re in a casino, and you stake $10 for every attempt, then your final success better pay off around $500 or more, or you’re still a loser at the end of the day. And of course, casinos don’t generally stay in business if the $10 stakes don’t add up faster than the $500 payouts. So your perseverance better not have been set at a higher value than the cost/benefit ends up warranting.

And there’s another problem too — your liklihood of success after a hundred attempts isn’t the same as your chance of success on your first go. If all your attempts so far have failed, then there better be something different about your next attempt, or your chance of success is probably zero (cf the saying “insanity is doing the same thing, over and over again, but expecting different results”). And if you’ve tried a hundred times, you chances of coming up with something new yourself is probably pretty low. And worse, that’s a lot of opportunities to build up some bad habits that also lower your chances.

On the other hand, after that many attempts, you’ve probably learnt a thing or two — at the very least, you’ve found a hundred ways that don’t work, and if your problem’s at all complicated, you’ve probably had a few minor successes that might be directly useful as well. And hey, maybe you’ll pick up on something in the process of failing that’ll give you a new idea to try on your next attempt that you’d never thought of before.

That adds up to your “smarts” not being a constant, but varying over time based on some combination of independance and creativity (in thinking up and trying new approaches), ability to avoid bad habits, and ability to learn from mistakes. And if you can’t keep thinking up a reasonable number of new approaches, and at least balance any bad habits you pickup against what you’ve learnt, your effective “smarts” are going to drop dramatically no matter what their original value was. And if that happens, you’re going to quickly find yourself at the s=0.0 line of the graph, which is the only point where even p=1.0 won’t eventually get you across the line.

Considering perseverance a constant isn’t really very accurate either. First there’s psychology — you probably will try a few times, just so you can tell yourself that you didn’t give up immediately; but equally, if you don’t get some sort of early success, you’ll probably give up and move on to something more immediately rewarding too. Hey, it is the era of instant gratification. But on the other hand, if you’ve already had a couple of dozen attempts, you’re probably going to keep trying, in order to avoid having failed all those times for no gain at all. The support (or lack thereof) from family, friends or colleagues will probably vary over time and have an impact as well. Not to mention external impacts, like other people’s successes and failures in the same/similar areas, or other changes to your overall thoughts on whether the project is still worthwhile or not.

All of which turns success into a possibly easier question: can you maintain (or increase) your smarts, and keep going long enough to get out of the blue?

Well, assuming you try in the first place.

A new name

So it’s been a bit over five years of indolence, time for a change. The original title for this blog was in homage to Andrew’s incoherence log, and found by trolling for interesting sounding words beginning with “in-“. Not seeing any reason to change a winning formula, I tried the same again, and came across the lovely word “inamorata” from the Italian “innamorare”, meaning “to inspire with love”. Hence, “inamerrata”, which clearly means “random stuff I find cool”.