Step 1, start a new arduboy sketch. I have one that simply prints the text 'ok?' in the top left corner. This should be small enough to not cause problems.
Step 2, put a paperclip or similar into the reset slot, but don't push it yet.
Step 3, Hit the compile/upload button for the sketch, then press in your paperclip to hit the reset switch.
The time it takes to compile and upload such a simple sketch tends to be well within the tolerance for the arduboy. I have found that this method work almost every time, and almost always within two attempts. Once you see the 'ok?' message just upload the game you were working with and it should be fine.
If you have a game that is constantly misbehaving due to the RAM being pegged close to the limit, modify the game (if possible) to allocate the buffer instead of using global memory. This will prevent the ram used being calculated at compile time, but I have found it basically stops the instability from being a problem. The ArTillery game I made pegs the RAM very close to maximum used. If i use a global buffer it will die every 3rd or 4th time I upload it. If I malloc the buffer it will work every time without fail.
I don't know the reason for this, but it's probably something to do with how the compiler places global memory.