Basically the entire card is addressed already.
There are ‘N’ blocks (often called sectors) of 512 bytes.
The problem is that data can only be read or written one sector at a time - you have to read or write the whole 512 bytes, which is just under 1/4 of the Arduboy’s RAM.
Reading them is fairly straightforward, you pick a sector and then read the whole 512 bytes into a buffer in RAM.
Saving is a problem because you can’t just write one byte at a time, you have to write the whole 512 bytes, so often systems will have some kind of buffering strategy in place.
There’s also usually a wear levelling system to prevent specific addresses being exhausted before the others (i.e. typically a file will move sector when it’s written).
The code required to manage an SD card even without a proper file system would probably still be fairly substantial.
It’s not impossible, it’s just very difficult to do on the Arduboy’s specs.
If the Arduboy had double the RAM and maybe a bit more progmem then it would be much easier.