After playing around with git-annex a bit, I figured its ubuntu install instructions are not ideal as the official ubuntu package is quite out of date (still at version 3.2), but git annex 4 comes with rather important improvements such as direct mode.
Fortunately there is an ubuntu PPA with an update version for ubuntu precise (12.04), which works well for raring (13.04) as well. To use it just run:
That's it!