I’ve made significant progress and have a full demo working:
When you power on, it defaults to the arduboy core and plays the loaded game as normal (there is a brief reset hiccup due to debug code running on the programming core). Then when you press a dedicated menu button, the programming core wakes up, pulls reset on the arduboy and displays the game menu. It can display an arbitrary number of files in a list with a 64x64 pixel screenshot of each game as you select it. Loading a game takes about 7 seconds.
Thanks for the tips on pinout and hex editing. I had actually already purchased that package of pro micro with the hopes of doing the pinout hack, but I ran into problems with the hex editing for the display. After a lot of trial and error, I figured out that most of my intermittent problems were caused by the project being strung out between two breadboards. The hex editing works fine and it was just a coincidence that I was having display issues when I first tried it.
The display rippling I described was caused by the two-breadboard power issues, but the line intensity problem seems to be inherent to the display controller. I tried bypassing the charge pump and even running it at much higher voltage - the display gets brighter but the line intensity still varies depending on the number of pixels illuminated. I also tried screwing around with all of the control registers and scan rates and saw no noticable improvement. It’s a bit frustrating because the 0.96" display I have has a flawless image, so I know that if the controller was designed for the larger display it could be made to work properly.
For sharing the display, I simply couldn’t get two arduino SPI pins connected without issues. I tried a few different resistor configurations with no luck. Even just connecting the two SCK pins with the arduboy core held in reset mode caused issues. So I ended up just using a multiplexer chip to switch the spi bus from one core to the other.
I switched the programming core from a 32u4 to a 328p based arduino. The virtual serial port on the 32u4 is just way too finicky, often requiring reset button juggling just to get the thing to start uploading. The programmer core shares the buttons and display with the arduboy core, but is not connected to the speaker or RGB LED. I ended up using every single pin on the arduino, including the analog-only A6 and A7 pins, so there’s just no room left.
Hardware wise, I’m going to put together a charger/regulator circuit to verify that it works off of a battery, check the current draw to figure out what size battery I need, then start designing a PCB. For buttons, my plan is to use rubber domes from an NES controller repair kit (still being made brand new).
Software wise, I want to make the menu togglable between different display modes by hitting left/right. There would be a full screen text mode with a 1-5 star rating for each game (similar to my Romshelf project). The next mode would be half text/half screenshot (like in the video), then full screenshot, then perhaps title screens. There’s a lot of screenshot collection to be done. It should also remember what game you have selected/loaded.
There’s also the issue of backing up and restoring eeprom, which is apparently possible over the icsp header without a helper program. It’s especially important since I can’t figure out how to erase the flash without also erasing the eeprom.