Bootloader and File Format [Wiki]

The other option is for a (true) factory restore is connect it to PC which then restores the flash image, burn the demo code, resets EEPROM all in one go.

Just move the demo app to the last position of the game list.

So, for extended mode games, how can they take advantage of the extra storage? We need a way to define the memory chunk as just being extended storage and not actually a game, so save space not including a header image.

Maybe we should call those ‘sections’ to avoid confusion.
Or blocks?

“A game slot consists of various sections, including the data and EEPROM sections.”?

“A game slot consists of various blocks, including the data and EEPROM blocks.”?

Ok, so there’s a game selection function.
How does that work exactly?

(I swear once I understand everything I’m going to make some diagrams and stuff to try and get everyone on the same table.)

Ok, in that case I would call this “padding”, since that’s the usual term.
(E.g. data structure padding.)

Ok, so programs aren’t allowed to touch the EEPROM section.
(Or “they can, but they shouldn’t”.)

I know, but the language lawyer in me can’t resist getting technical with the rules. :P

Ooh, now we’re on to something.
So the question becomes “should it be mandatory, or is part of the optional data enough?”.

(Which reminds me, I’ll need to ask about the strings at the end.)

“If it’s worth doing, it’s worth overdoing” - Mythbusters motto
Cryptographically secure hasing for the win. :P

So, these strings… Let’s see how well I can guess this:

  • They can only be those 4 strings
  • They must appear in that order
  • The order determines the meaning

Correct?

And of course, the burning question, if they’re in UTF-8, how does that react to the Arduboy’s use of code page 437 for its default font?
What’s the overlap like?

(I could probably figure that one out myself tomorrow, but it’s late here so I’m leaving it open for now.)

If I’ve been following this correctly, that’s what the “2 page addr application data file” section is for.

Which has made me realise another question:

How to determine the length of these?
Are they just assumed to be consecutive?

@Mr.Blinky I’m curious if you could draw a little diagram showing the memory method? I’m a pretty visual learner so it would help me a lot.

1 Like

Yes please … a pciture is worth a thousand words!

A game slot contains a program section and optional data, save and eeprom sections.

I’m in for sections.

A game is selected using the directional buttons in the (boot)loader menu. Left/right to change group (list) up/down to select a game in that group. pressing the B button will burn(flash) the selected game. When no game is selected pressing B will do nothing. Pressing A on any time will launch the program currently in progmem (last burned game or last uploaded game) when there is no program in progmem (it has been erased). the A button will do nothing. When a directional button is pressed down for longer then ~0.25seconds the button function will repeat.

After power on/reset group 0 is always selected which is the loader title screen. pressing left will decrease the group ID and pressing right will increase the group ID (when pressing left on the loader screen. the last group is selected) when the last group (highest group ID) has been reached pressing right will return to the loader screen (group 0)
Each group starts with a (group)title screen (technically can be none to as many as you want but using one is recommended) pressing up or down will select the previous /next game (slot) in the same group. When the last entry in a group has been reached, pressing down will rollover to the group title screen. When on the group title screen, pressing up will select the last game slot in the group.

Okay a program section needs to be padded (I called it aligning due to the align assembly directive)

They can but shouldn’t indeed. The extra data will be lost and/or the EEPROM data will be lost (switching of power / uploading program just after it has been erased)

That’s fine by me as long as you don’t sue me :wink:

Yes, should the title become mandatory so a program may use that to ID the program?

Yes

As I mentioned somewhere else the optional strings where intended for PC side manager. I picked UTF-8 so the strings could be Japanese too.
If we want a mandatory name so programs can use it. There should be a mandatory string ArduboyName or so using code page 437

Exactly

The are determined by the PC side manager and are consecutive. Maybe a glimpse at my Python script may give you a better idea. (Note savedata section and eeprom section are not implemented there yet)

I’ll play around with paint a bit and see what masterpiece I can draw up :smile:

1 Like

Not sure if it is already done like that but I would propose to allocate the data section in chunks of the eraseblock size. That way if any game wants to erase one of its data blocks it is not screwing up data from another game.
Also we might need a way to specific how much data blocks a game needs to be more flexible. The data file is one thing but maybe the game logic needs additional blocks that are not part of the data file.
Also is there a way for the game to know which blocks it owns or will we hide this in some library functions?

Because program and data sections are read only they are allocated in pages. The save data and EEPROM backup sections are padded to a multiple of 4K for erase. So when two ore more games that do not use save data and EEPROM backup, no space is lost for padding.

The data and save data can be any size you want (for as long as it fits memory.) The PC-side manager determines the size of those sections on their binary file sizes.

A program can use the sections pointers (stored in reserved vectors) to find the beginning of the data sections. It is up to the program to keep track of the size.

Ah ok. So there is a reserved vector somewhere that I can read out to get the start address of my data/save blocks?

Is the bootloader menu something you see on the computer or the Arduboy?
(I’m assuming the Arduboy.)

What does it look like?
When/how is it activated?

An aligned data structure will often have padding,
but alignment and padding are separate concepts.

Alignment is just the need for data to use certain addresses, padding refers to the unused bits/bytes that exist as a result of alignment.

The worst that can happen is nasal demons.

I’m not sure it needs to be mandatory for the sake of ID-ing the program, because a game planning to do one of the things I described would opt-in to putting the name in for ID-ing.

But one thing I think suggests it possibly should be mandatory is the bootloader list - without a mandatory title, the bootloader can’t provide a name for the game.
(It could provide the logo/image thing, but I think a name would be useful too.)

I’m tempted to suggest that length-prefixing should be used instead because it makes it easier to identify the strings’ locations and (importantly) to allocate a buffer of the correct size.
But if it’s only going to be a desktop PC dealing with those then it’s less of an issue.

In that case, fair enough, UTF-8 makes sense.

Good luck to anyone who can’t read the game descriptions for the Japanese games though. :P

Couldn’t we just assume that everyone is using a variant of SemVer and make the version a 3 or 4-byte value?

E.g. 0x20C0000 (4 bytes) would use less bytes than “2.12.0” (7 bytes).

(I know that means having no exact way to differentiate between 0.1.0-rc1 and 0.1.0-alpha, but in my experience few people take the extended name approach and just stick to a 2 or 3 digit versioning system.)


I’m still confused about the relationship between the bootloader and the ‘pc program’.

I think I’m close to having almost everything understood, and I’ve started writing a draft guide to the format.
Ordinarily I’d stick it on my GitHub, but I would be worried about it being discovered there, so I’ll post it either here or in a hidden gist.


Also, what exactly is the role of the “ARDUBOY” string?
Is it just a ‘magic number’ or is it supposed to have some functional use like detecting data corruption?


Sorry, another question…
(Actually my earlier question reworded slightly.)

How can you tell the size of the data and save sections?
I’m guessing you can do "2 page addr application save data" - "2 page addr application data file" and use the difference to be aware of the save size, but if that’s the case then the same could be done instead of “1 application flash size in 128 byte blocks”, and that still doesn’t explain how to figure out where the save data ends.

This is the first draft of my explanation…


FX Chip Format

The FX chip’s ‘file system’ is not a traditional file system as such.
Instead I shall refer to it as a ‘slot system’ that uses ‘game slots’ rather than files.

Each ‘game slot’ consists of the following data:

  • 7 bytes - magicNumber - The mandatory 7 character ASCII/UTF-8 string “ARDUBOY”
  • 1 byte - groupId - The ‘group id’
  • 2 bytes - previousSlot - The address of the previous slot
  • 2 bytes - nextSlot - The address of the next slot
  • 2 bytes - slotSize - The slot size, expressed as the number of 256-byte pages it occupies
  • 1 byte - progmemSize - The size of the progmem data section, expressed as the number of 128-byte blocks it occupies
  • 2 bytes - progmemAddress - The address of the progmem data section (in the FX chip address space)
  • 2 bytes - dataAddress - The address of the general data/resource section (in the FX chip address space)
  • 2 bytes - saveAddress - The address of the save data section (in the FX chip address space)
  • 2 bytes - eepromStart - The start address of the game’s EEPROM usage (in the EEPROM address space)
  • 2 bytes - eepromEnd - The end address of the game’s EEPROM usage (in the EEPROM address space)
  • 16 bytes - hash - MD5 hash of progmem data
    • Note: this is planned to be replaced with SHA256, which will increase the hash to 32 bytes
  • 214 bytes - stringPool - A pool of string data to be used by the PC manager
  • 1024 bytes - titleBitmap - An image to be displayed by the bootloader upon selecting the game.
  • * bytes - progmemSection - The progmem data, which is to say the game’s compiled machine code, or in other words the same information contained in a .hex file
  • * bytes - dataSection - The general data/resource section, which is to say the area a game uses to store any data/resources it might need, like images or maps
  • * bytes - saveSection - The save data section, which is to say the area a game uses to store save data as an alternative to using EEPROM

Fields Explained

groupId

The bootloader displays games in several lists.
The groupId decides how games will be grouped in the loader.
Games that have the same groupId will appear on the same page together.

E.g. if Minesweeper and Easter Panic have a groupId of 0 and Dark & Under has a groupId of 1 then Minesweeper and Easter Panic will appear in the first page/list displayed by the loader, while Dark & Under will appear on the second page/list.

previousSlot and nextSlot

Through these fields, each slot can be thought of as a linked-list node, and thus the entire contents of the FX chip can be thought of as a bidirectional linked-list data structure.
In this analogy, the previousSlot field is the pointer that points to the previous node in the list, and the nextSlot field is the pointer that points to the next node in the list.
(Remember that ultimately pointers (in a C++ and/or C context) are merely a thin abstraction of hardware addresses.)

The reasoning for using a linked-list of slots rather than having a lookup table at the start (as is common with many file systems) is because having a lookup table at the start would mean that the earlier pages of the FX chip would be erased more often than the other pages, which would cause the chip to wear out much faster than it otherwise would.

1 Like

Arduboy here’s a clip of an earlier version. But functionally still the same.

It is triggered by entering the bootloader mode.(power on + down, reset button, IDE botloader trigger, bootloader button combo)

Note that for the FX @bateske likes it to start rigt away when powering on (this may become optional though)

Then it remains optional

The bootloader is incapable to print text due to size constraints. Instead a title screen is displayed and is mandatory.

There’s always google translate :wink:

The version must be a string not a dword

It’s a magic number to identify a valid slot. I chose it so I could reuse a string already in the bootloader. When this magic key is not found it is assumed the end of the slot system has been reached.

by substracting the page adresses. saveDataSize = nextPage - saveDataPage etc as you mentioned.

I just realized the EEPROM addresses where for an old idea where I only wanted to backup & restore the actual used EEPROM locations. backinging up and restore the complete EEPROM seems simpler now. Note that this isn’t implemented yet (something for the weekend :smiley: )

That byte is used to quicly identify a title screen slot or Game slot and is the loop counter for the program flasher. My goal was (and is) to save code size in the bootloader, not in the header where plenty of space is :slight_smile:

The draft looks good. But…

Only one title(screen) is visible at a time. You browse through the list using up/down

Group 0 is reserved for the loader titlescreen. first group for game slots starts at 1

The EEPROM start/end may change to a single EEPROM page address.

I also made memory map drawing (well excell) but left it at work :stuck_out_tongue:

47 posts were split to a new topic: Bootloader UI Discussion

I really should have seen that one coming…

Why though?
Aside from a small amount of flexibility, what benefit does a string have over a 4 byte value?

(Just throwing it out there as an idea.)

Personally, I prefer Jisho for Japanese.
(Being able to read hiragana helps.)

Makes sense.

Hrm, let’s hope nobody puts “ARDUBOY” in their data section.

Then what do you subtract saveDataAddress from?
There’s no end address for it to subtract from…

Thinking about it for a bit and inferring from some of the other stuff you’ve been saying, I’m now guessing these EEPROM addresses are actually addresses into the flash cart and not EEPROM?

Maybe? I don’t know. (There’s a lot of stuff to keep track of here.)

One downside to that is that it also saves the first 16 bytes.
If you backup, change the Arduboy settings and then restore, you end up writing over the settings you changed, putting them back to what they were before.

It would at least make backing up much simpler though, because you don’t have to pre-specify what’s used and keep it in sync with what the progmem is doing.

Maybe backup and restore everything but the first 16 bytes?

And people say C++ has a lot of rules to remember. :P

So what happens if you do up/down on the loader titlescreen?

And I presume this means the loader has its own slot?
Is the loader actually an Arduboy program?

Does that take us up to three different programs?
Loader, bootloader, PC loader?

Doing Arduboy stuff in Excel at work instead of your Job?
We call this “doing a @filmote:P


I had a quick look…
Lots of assembly on the C++ side, and the Python was very…
Pythony

TL;DR
How about the idea of having slot on the flash for the game/menu manager (kind of second stage bootloader). The bootloader would flash this. The manager runs, showing all the fancy stuff (menu, sound, etc… using the full 32k of progmem and 2.5k of RAM) upon user selection it calls the bootloader to flash the required game.

Only things here are:

One must write this manger for the Arduboy and it must be fancy :wink:
The flash will wear out faster.

it’s easier readable. If you need/want to look at the raw flash memory (dump) with a hex editor you could easily identify a title.

Thanks for the tip! I should take time to learn hiragana/katagana. Even though I’ve been staring at them for a while now I only seem to recognize game, yes and no easily :smile:

It would have to be page aligned too. But the loader only searches in headers not in sections. It is the PC managers task to append a blank page after the last game slot.

saveDataPage == saveDataAddress

nextPage == saveDataEnd == endOfSlot
(I’m talking about the current situation without the EEPROM backup section)
When the EEPROM section is added:
saveDataSize = EEPROMdataPage - saveDataPage
EEPROMdataSize = nextPage - EEPROMdataPage

For clarity a Page is a uint16 page number, a (flash memory) address is a uint24 byte address. a Page address is an address at a multiple of 256 bytes. (Feel like we’re wokring on RFC FX-001 here :stuck_out_tongue: )

Originally intended as EEPROM addresses. But will be turned into a single EEPROM data start page

I already mentioned somewhere on the puplic flashcart thread the 16 system bytes are excluded :wink:

It will appear to do nothing. up and down are select game controls and there are no games to select.
Pressing down will extend the bootloader timeout though. But this behaviour will be removed for the FX bootloader. I considered to use multiple loader screens. But didn’t implement this as pressing the down button while powering up is a feature to enter the bootloader directly. So when it is used the loader screens would flash by.

The ‘press down to remain in bootloader’ feature is obsolete now. so using up/down could be uses to select multiple loader screens to form some basic instructions.

only a title slot

No the loader is a small part of the bootloader and therefore has no impact on the application flash area aka Arduboy program memory.

I consider the loader more a user interface (but you can upload a program over USB anytime). Since it is part of the bootloader you have the bootloader and PC loader.

They call it multitasking :wink:

And C++ at the rightside (of the assembly) :wink:

That’s what I said, the manager takes care of it.

The beauty of using the title screen is that PC manager could render anything on them what you want.

  • It could render your scroll bars and left/right page up/down arrows
  • a simple list of title names with a cursor
  • 4 thumbnails with a selection cursor/outlined border

Ofcourse someone would have to write all that stuff :slight_smile:

Currently there are just a few spare bytes (I’d like to try and keep as much of that 1K freed progmem available :slight_smile: ) I’ll need more than that to implement the EEPROM backup feature and will rip out some of the fancy stuff (bootloader streaming,led control, flash leds, etc)

2 Likes

Staring at a blank lines? There’s just the copy displaybuffer to OLED :wink:

Yes we do.

but they can be converted by the PC manager.

That’s what I do with my python script now too.

I don’t see the difference in the amount of work required between editing a small image or a full screen image.

To simplify the amount of work. The PC manager could create a screen with game title when there is no title screen supplied as an option.

They where extracted from Arduboy files or screenshots taken with ABE and converted

When I’ve added the EEPROM backup/restore feature to the loader that info is not important as all EEPROM(except the 16 system bytes) will be backed upped / restored.

Lemme make sure I understand this… The EEPROM will be backed up to the flash chip when loading a new game? That makes more sense when you guys are mentioning EEPROM memory addresses.

If that’s the case, Micro City using the entirety of the EEPROM wouldn’t actually erase my Circuit Dude data? And if that’s the case, then I could actually allow players to create their own levels on the Arduboy and save them without interfering with other games!

:smiley: Dude, this Arduboy FX is gonna be freaking cool. :heart:

1 Like

Yes, but then the 4-byte version number could simply be above the title, so you’d still be able to find the version number easily,
it would just be a bit more effort to decode it.

You lost me…

That’s pretty much the sort of thing I’m hoping we’ll end up with as a result of all this conversation.

Ok, that will make calculating the save size easier.

Presumably that makes the EEPROM area a fixed size of 4 pages?

That thread is massive, that would be like finding a needle in a haystack.

(If I’d known it was going to end up being used for the next Arduboy,
I probably would have read the thread at the time,
but I didn’t because I considered it an unofficial mod,
and I don’t have the tools or experience to do such a mod on my own.)

Or ‘progmem’ as it is often known. :P

So the bootloader is used to select and flash a game without the PC,
but the PC loader can also select and flash a game,
and only the PC loader can load data onto the FX chip.

1 Like

That could be done. To meet in the middle I’d then prefer a 2 or 3 byte in BCD format that represent the version as x.x.xx or x.xx.xxx In big endian

In my discription i write about saveDataPage which is the start page of the save data. If I understood you correctly you call it saveDataAddress. But we both mean the samethe same.

I don’t use the term address as the flash chip is addressed using a 24 bit address. So

  • address = 24 bit byte address pointing to any byte in flash memory
  • page = 16 bit page number
  • page address = page << 8 the address of the 1 st byte at the start of a page.
  • sector = a group of 16 consecutive pages
  • sector address = sector << 12 the address of the 1st byte of a sector.

Flash memory can only be written to in pages at a page address.
Flash memory can only be erased in sectors at a sector address.

Yeah sorry about that. I should update the first post more often.

Yes

The PC loader could do that. For when you want to just upload and play a game like we are use to now.

Yes the PC loader can writes a complete flash image that was prebuild (like It is done now) or append games.

Oh I forgot to post the memory map image. Here it is now:

2 Likes