I'm still not quite sure what causes this but I updated the firmware
using rpi-update on raspbian and instead of just using fat,
I formatted the sd card with vfat(mkfs.vfat instead of mkfs.fat -F32)
and it works for me now. Unfortunately I did both at once so I'm not
sure which one is actually the fix. I would try formatting with vfat
first.