Bootloader and File Format [Wiki]

So what’s interesting here is these are actually starting to be some of the very same issues I ran into when developing a game uploader for the website here. What happens if people upload games with the same name was a particular issue, then also what about file names being the same was a little more pesky one but solved that with using a serial number hash I generated.

What are we actual storing for identifying the cart slots? Are there strings stored with the game name or is it just an id value?

If it’s just the ID value I think there are two ways of preventing collisions. One basically relies on convention, the PC program will generate a ID from a hash of the name so that any game with the same name yes, will actually override other games. So you just have to not do that.

The other way to do it is actually request an ID from the website, as an official lookup table. I’m not favorable to that decision its too closed off.

I think expecting games to be named different is not a bad convention.

As @Pharap mentioned this will be up to the PC manager to warn the user. Much in the same way if you were dragging and dropping a file into a folder where a file with the same name already exists, just pop up a dialog with “this is going to overwrite (game, version) is this OK?”

Going with the game slot layout: the .arduboy info.json manifest file can contain the game name / identifier string for addressing the extra data / save space. It can optionally be left blank if the game doesn’t need to access the extra flash space and keeps it backwards compatible, i.e. for legacy Arduboy games.

Not sure exactly how the loader will know where to dump and restore EEPROM contents to / from the flash chip when switching games though, it will have to store the save address somewhere so it knows which game is currently loaded.

For an end user who just wants to play games it would all be fairly straightforward:

  • Can use PC manager to load .arduboy files : a game slot is created based on the contents and the info.json manifest file
  • Include an option in the PC manager to import .hex files and offer the user a form dialog to include name, author etc (e.g. for games where a .arduboy wrapper hasn’t been created by the author)
  • Can still load up a .ino in the Arduino IDE and deploy the game to progmem

The only thing I see being a bit tricky is users who want to develop their game to use the extra flash space.

  • Do they have to create a new .arduboy file every time they upload their extended data to the device? (not very user friendly for iteration)
  • Should we include functionality in the PC manager so that users can easily overwrite the existing extended data section on their device with a new file?
  • Some other method entirely? e.g. A different app to aid development
1 Like

Just had a further thought (not sure if this has already been mentioned and assumed): the extended game data section of the slot should probably also be 4K aligned and padded so that it is easy to iterate uploading new game data during development.

So, this is my thought on how it might work: The PC Manager has a developer mode that assigns all of the empty memory space (or as much as you request) to your application, which creates the slot. It would be helpful if the PC Manager can be linked to a folder or a file so that you can just press “upload” and the resources that are edited outside the program will be uploaded to your save slot.

So then if you need to edit your map file, you use a hex editor or a tool you made, save it, then press upload on the PC Manager.

For code, I don’t know but can you just upload like normal on Arduino?

But at some point you have to make sure your save cart is minimized to only what your program needs. I suppose that is the tricky part. For anyone using the storage in a static fashion this is trivial because you just allocate however much memory your external content takes.

If you are dynamically writing variable length content to the save cart then some kind of boundry needs to be established. If we had our own IDE, theoretically we can set a define to request a certain amount of memory that could be used by the “compiler/pc manager/uploader” that also the library that is running in the code respects that same limit.

Without an integrated compiler/linker within the PC manager, its up to the developer to set the define in their code the same as what they upload to a PC manager (I guess that will need to be in the arduboy file then).

So I think that you can do all of the real time development without having to create an arduboy file, but you could if you wanted.

I think the price of not being able to have both games on it is a bit high. The developer just has to specify the program data and save data section pages (or use the defaults) for development.

thse should be the filenames of the section files for example brickgame-data.bin brickgamesave.bin
for EEPROM it would be handy to have their start and end addresses in their name for example brickgame-EEPROM-0010-03FF

The EEPROM page numer is stored in an unused interrupt vector. the bootloader tests if that vector is set with an EEPROM backup page number. If it then the eeprom is backuped first before flashing the new program.
the EEPROM backup is 4K aligned. (read only sections are page aligned, writable sections 4K aligned)

Yes a developer option menu where you can specify the files or sizes of the sections and report the section page numbers or command you have to include in your project.

Yes code is uploaded as normal with IDE.

I think I am a bit lost on how it should work. Can you help explain?

So you don’t want to use file names / string identifiers for when a game wants to find it’s data section because of potential name clashes? What are the default pages? During development should we just hard code the data / save addresses? (If so this isn’t very user friendly for developers)

If we aren’t using file names / strings to identify game slots because of potential clashes then why does the naming matter? What is the significance of the EEPROM start / end addresses in the file name?

Sounds great! Perhaps we should also include an option to disable this behaviour per game, i.e if the game doesn’t actually access EEPROM? Also, what will happen if a user flashes a new game via the Arduino IDE in between playing games? Could the bootloader get confused and accidentally overwrite the save of the previously loaded game with some bad EEPROM data when a new game is loaded via the loader?

Do you think we should also keep the game data 4K aligned and padded too as I mentioned in my previous post?

Here is how it will work for now:

You want show a simple video made of 1K images. These images are stored in a binary file.
You write this file to the flash chip using a command line tool (or GUI version)
dev-writer --datafile=your_video.bin
if there is a savedata file you want to add:
dev-writer --datafile=your_video.bin --savefile=initial_savedata.bin
or if you just wanted to add save data block:
dev-writer --datafile=your_video.bin --savesize=4096

The tool will put all data at the end of flash like flash_start = flash_end - savesize - datasize and will print out the start page number of the data and start page number of save data (if any)

In Arduino program in setup you’d add:
flashInit(progDataPage, saveDataPage);

When somebody want to try out your program in Arduino IDE they would do the same but wouldn’t need to change the progDataPage, saveDataPage in the sketch cause they are the same (unless the person has a (homemade)Arduboy with smaller flash ofcourse)

When the program is added to the flashchip 'filesystem’with PC manager and burned with the loader the flashInit() function will ignore the progDataPage and saveDataPage values and fetches those stored in the interrupt vectors.

As mentioned in the header discription the EEPROM backup is optional. The EEPROM page in the slot header would be 0 for games that do not use EEPROM or need EEPROM backup

When a program is uploaded through the Arduino IDE the interrupt vectors are overwritten and the EEPROM magic number/page are no longer available and so is the backup feature.

If it’s possible I’d also like to add backup EEPROM before regular uploading so the game slot EEPROM is always up to date for the next time you play the game.

I plan to make the auto EEPROM backup feature optional. But as I mentioned earlier I’m still working on implementing the whole EEPROM thing.

1 Like

Hi. I got my devkit and now I want to check what software/tools we have to write content to the flash. Is there a git repo somewhere that I can look at?

I moved the info for how to use the Dev Kit to the head of the post:

But this is what you are looking for I think: GitHub - MrBlinky/Arduboy-Python-Utilities: Python script to upload .hex, .zip or .arduboy files to Arduboy and Homemade versions

1 Like

Thanks. For the image generation that is great. Is there a demo version of how to access the flash from the scetch (@Mr.Blinky)? Or at least some ideas on how to do that. I remember there was something around in some threads but I have hard times finding it. I think something simple like R/W examples would be good. Maybe even some initial idea of how the sketch finds its data section.

@Mr.Blinky is still looking at it but I don’t know if you can find the example that is based on this:

As that will be reading the frames at least.

Ah, ok. Found the demo code. Thanks for the hint.

1 Like

Yeah we are of course trying to wrap all of it into a library to make grabbing the assets as easy as possible, but the other night keeping me awake was making me realize that using some kind of paging system may be the most efficient way of handling it otherwise any kind of dynamic memory manager would probably be annoying and code space heavy to implement. I would imagine. I’m very bad at coding.

So it may be possible that you assign all your sprites, your strings and music whatever, and those then are swapped out for same sized replacements. I guess a little similar to a paging system used in the old consoles.

Here are useful links:

I’ve just updated the WIP library. As @Pharap sugested I’ve changed the set of functions into a static class.

A paging system maps memory into existing memory space. Flash memory will always be external.

I’m working on a drawBitmap bitmap function that looks like this:

void drawBitmap
int16_t x,
int16_t y,
__int24 pageAddress,
uint8_t frame,
uint8_t mode
pageAddress is a 24-bit address in flash memory. To be more precise it’s actually a 24-bit relative offset added to the start address of the programs data section in flash.

Hmm thinking of it now I need to call pageAddress, flashOffset or something.


Thanks for the update. For development is there some special mode of operation or shall I just create an image with just my game on it.

Or can I Program the game into progmem and use hardcoded flash addresses during development?

You can just upload you game from the Arduino IDE. If your program requires data stored in the flash chip. you can write your binary data file to flash using the flash-writer script with the -d option. When the upload is finished you will be given a page number. Which you need to use in the Cart::init(pageNumber) function.

Let’s imagine your program is the factory animation example.

  • 1st you you need to upload your animation data with the script and you’d type the following on the connamd promt (if both files are in the same directory) -d factory-frames.bin

When upload is completed it would say something like:

Please use the following line in your program setup function:


or use defines at the beginning of your program:


and use the following in your program setup function:


You need to remeber the value 0xDCF0

  • Now with the data stored on the flash cart. You can ‘develop’ your program in the Arduino IDE. In the Setup() function. you will need to add the init function with the value from the flash-writer script:

Now when uploading from the IDE, your program will know where to find its data. When using read and write flash functions your data starts at address 0.

Your data is made up of 1K images, each image is a frame of the animation. To show the animation you can just multiply a frame counter with the size of a image (1024 bytes) The code that will load your images could be like this:

constexpr size_t IMAGE_SIZE 1024;
__uint24 frame;

Cart::readDataBlock(arduboy.sBuffer, IMAGE_SIZE, frame * IMAGE_SIZE);

Note that once the program gets in a finished state. The hex file and data file can just be added to the flashimage .csv file. The flash-builder script will make sure your program will know where to find it’s data file.


Do you think you’ll actually use a uint32_t and just mask off the excess or store a uint8_t [3] and write some assembly for all the various operations?

Or is __uint24 a compiler extension?

Does that mean that you’d need to recompile the code if you reflashed the chip and the data ended up in a different area?

I have a few ideas/suggestions for some of the API names,
but I won’t suggest unless you’re interested in API name changes at the moment.

I’d recommend using constexpr variables instead of defines for constants and an enum class for chip commands, but I suspect there would be some resistance to those suggestions from people who are more used to macros.

I think uint24_t should be a type alias, though there’s no way to check whether a type named uint24_t already exists or not.

And lastly I can think of a handful of improvements that could make the API easier to use.
If you’re open to PRs then I could demonstrate a few.

Head post is a wiki, how about throw those links on top?

It’s an extention as of version 4.7 (But I’ll also do assembly optimisations later on)

Only when:

  • your program data size changes during development
  • your program save data size changes during development
  • your data is overwritten by an uploaded flashimage that is large enough to overwrite your program data and program save data at the end of flash memory

The flash-builder script will make sure a program can locate it’s data (the 24-bit addresses in a program are relative offsets)

I’m open for suggestions at any moment changen them early on may save work later.

Isn’t there’s a contradiction in a constant variable? I think there will always be two camps when it comes to constexpr and defines.

I’ve concidered adding a #define for uint24_t to cart.h but wasn’t sure I should do that.

I’m always open for PR’s

Thanks for all this info. I’d like to add some thoughts about the new library. I think one of the main use cases for this kind of memory is reading. So maybe we can put focus on that. Another main use case is to reduce PROGMEM usage for data so we can use it for more code. So what is used for PROGMEM?

First, all data that is explicitly put in PROGMEM. Here I think we can mode the Arduboy2 library to load all this data from external flash (e.g. display init data).
Second, all variables that are in the data section. E.g.

uint8_t state = 1;

That would cause the value 1 to be stored in PROGMEM and during boot the variables in the .data section would be initialized with values from PROGMEM. I think also most games will have some kind of constructor that is doing something like:

void Blah::Blah()
  instance_var = 1;
  instance_var2 = 5;
  memset(some_instance_struct, value, ...)

And so on. So if the interface we are defining would wrap this nicely for less experienced users that would be a plus. E.g. the class could use a struct with all its state variables and then just load all the init values with a sequencial read:

struct state { ...} state;
<now in the constructor>
flash_read(&state, sizeof(state))

And the whole instance variables would be initialized with very little code/data in PROGMEM. Just thoughts…

Now concerning my game the most benefit would be reading textures and the like from external flash. Reading each byte, waiting for it and then using it would be very slow. So my thought here would be we can just use the SPI hardware better. Say I need a byte from the texture. Usually I read it and do something with it. While I am doing something with this byte I could read the next one already. So maybe we can do something like a stream of bytes…

stream_start(... blah blah address page stuff)
(above we do an addressing of the flash)
(above we write(0) to read one byte but do not wait for finish)
<now do something else in game logic>
stream_get() <1>
(above we actually get the byte from the SPI register, assuming the bytes is already received, maybe we need a safe/unsafe version, safe version would check if byte is already received, function can request next byte already)
<use byte in game logic and repeat <1>>

Something like this. I not very into C++ but maybe we can do something like a stream or something similar. The basic idea is to trigger the SPI to clock in a byte while we do other things. That way, ignoring the initial overhead it could be even faster than PROGMEM read (it always take 3 cycles, whereas the SPI register read+write could be faster (maybe)).

As I said, just some thoughts…

Also we might need to thing about how to nicely create a data file.