Bootloader and File Format [Wiki]

Development Project Falcon
1

This is a wiki page where the latest most up to date resources will be for how the bootloader should operate and how the file system on the memory card can function. I don’t know much about this @Mr.Blinky will need to fill in a lot.

Bootloader is Cathy3k available here:

Questions and discussions can thread down below.

There will be no changes to the external hardware, so a reset button won’t be useful for entering the menu. Instead, the menu will start each time.

Primary User Experience

  • Power on device greeted with Menu Splash Screen*
  • Pressing A or B on Splash screen will load currently flashed game
  • Pressing Left or Right will navigate through categories
  • Pressing A or B on category screen will return to Menu Splash screen
  • Pressing Up or Down on categories will scroll through games
  • Pressing A or B on games will flash and load the game
  • Pressing RESET in game returns to bootloader Menu
  • Pressing RESET in bootloader Menu Splash Screen SLEEPS
  • Pressing RESET in SLEEP mode will start bootloader Menu
  • Pressing Up or Down on Splash Screen will not do anything

*When FX chip has not been initialized an USB icon is displayed.
The ‘start bootloader menu on power on’ feature can be disabled by clearing bit 7 of EEPROM_SYS_FLAGS value in EEPROM. When disabled the bootloader menu can still be selected on powering up while holding down the DOWN button

Header Format

Copied from Flash cart(ridge)

mandatory data

7 ARDUBOY signature string
1 list number (0 bootloader title screen)
2 previous slot page address (0xFFFF for first slot)
2 next slot page address
2 slot size in pages
1 application flash size in 128 byte blocks (0 for none)

optional data

2 page address of application program data (0xFFFF none)
2 page addr application data file ( 0xFFFF none)
2 page addr application save data (0xFFFF none)
2 EEPROM start (0xFFFF none)
2 EEPROM end (0xFFFF none)
16 MD5 application signature
214 UTF-8 zero terminated strings:

  • Application Name
  • Version
  • Author
  • Quick instructions

Numbers are stored in big-endian (Most Significant Byte first) format.

1 Like
2

The home screen program should allow us to set the global sound setting.

For video, we could have a few bitmaps as frames f an animation, right?

3

Oh, its built into the library

1 Like
4

The first time you turn on the device it would be great if there was a long form video, and then a shorter version is used after that. I think if you are greeted with a high quality movie when you turn on the device for the first time your jaw will hit the floor.

And that’s just another thing to think of. Because of the available space, your games can theoretically have FMV cut scenes!

2 Likes
5

Err. how big is the extra memory? On the Ardugirl site, you suggest its 1Mb. Divide this by approximately 30Kb (a typical game) and you get 34 games. Fill up the chip with a lot of video and / or sound and you might only get a handful of games.

2 Likes
6

Why not just ship units with a kind of tutorial game that’s packed with info, so game developers don’t have to worry about having the all-singing-all-dancing intro taking up space in regular games?

In my experience, a lot of people end up disabling the intro after a while because they prefer forgoing the Arduboy startup screen in favour of their own splash screens.

Personally I’d rather have 2 high quality games than a handful of arcade things,
but it would be good to give people a choice if we can.

I’m pretty sure AVR is little endian, so that might be a bit of an issue.

1 Like
7

Well, yes that’s what I mean the FIRST time the Arduboy is turned on, it automatically runs a “demo” game that has a long into video and maybe a walk through intro.

At this point I’m looking at this chip:
128Mbit / 16Mbyte storage (512+ games)

And actually planning to really only put 100 games on it, leaving tons of room for the expanded apps.

Also, think we can have an app for users to flash out the entire “FX” chip too, so you kind of COULD have an entire 16mb game if you wanted. But that’s not how it would ship.

Yeah I think it being an option is important. You’ll still have to be greeted with a menu screen but theoretically that is just the splash screen for the game you want to play. So the only difference in user experience to the current game is that you just have to press A when it turns on and wait 10 seconds.

1 Like
8

That should be plenty!

9

I was just checking.

Usually the Arduboy logo is part of the library and not the game,
so I wanted to make sure this was going to be a game feature and not a library feature.

I think the demo game (being the user’s first game) should be more of a tutorial for using the Arduboy, maybe explaining what EEPROM, Progmem and RAM are, but certainly explaining about flashing games and maybe displaying a few URLs to visit (with QR codes because I hear smartphone users love those).

Yep, 16,384KB / 32KB = 512. That’s a big difference.

Ardumon may be possible after all.

I know at least two people who are currently attempting small Ardumon games (neither knows about the other) and neither know about the flash chip plans.

1 Like
10

How far along are the projects? What do you think about alerting them to the others existence and bringing them on board here? The Ardumonsters would need to be unique, they couldn’t use any licensed IP.

11

Not very far as far as I’m aware.

One I know almost nothing about, I’ve just been providing advice about design and teaching some C++ techniques,
and the other is on its second “scrap everything and start again” cycle.

At the moment I’m on the fence about bringing either party in.
I’d need to give it some serious decision, and I think in either case the person would need a partner or two.

In the former case I think they’ve made some decent games in the past, but they’re still learning and I have no experience of what their development process is like.

In the second case I think they’re fairly experienced and I know them very well, but I don’t know if they’d have the free time to do it and I know some of their skills (particularly with C++) are still a tad lacking.

(Bear in mind my code quality standards are impossibly high, even for myself, which is one of the reasons I prefer to take a back seat.)

12

As far as an actual ardumon, there is source code around for the basic engine, completely optimized, so I don’t know how efficient it is. I’ll see if I can dig it up from @chris who also made Flappy Ball and Squario. And Tetris actually.

13

@Mr.Blinky
There’s some of this I don’t understand still.

  • Is the ‘Arduboy signature string’ in front of every slot?
    • Is the string always “ARDUBOY” or is it sometimes different?
  • What is the “list number”?
  • What is the “application flash size”?

Presumably the previous and next addresses are to form a linked list.
(I.e. it’s a linked list file/slot system instead of a lookup table system.)

Why addresses and not offsets?
If they were offsets it would be easier to reorder slots/blocks,
because you wouldn’t have to change these,
you’d just have to copy the whole block of data.

I think this would be better as start and length.

The problem with using start and end is that you have to verify that end >= start.

If start and length are used instead, there’s no checking needed,
and a length of 0 can signify an empty/invalid block.

Why MD5? Wouldn’t CRC32 be suitable?

I still don’t fully understand either, but I know CRC32 is much cheaper.
There’s an interesting page about it here.

The only MD5 implementation I could find for AVR is under the GPL and it looks fairly memory hungry.

Is there a specific reason for big endian?
Wouldn’t it be better to use the same endianness as the chip itself?
(As well as the chips that most of the helper tools will be running on - x86/x64.)

1 Like
14

OK when powering on it enters the menu. Should it stay in the menu until the A button is pressed or a new game is selected? Or should there be a timeout? Without a timeout I can free some space.

Should there be a EEPROM option to auto start the last game (auto play) so it behaves like it does now? When this option is added and active. Forcing the menu will still be possible by pressing the DOWN button

The current intro screen can de anything you like.

Originally I had the idea to allow multiple title screens with a timedelay in msec which could also be used to show an animation. I didn’t implement it due to space constraints and later realizing that a menu should be simple and an animation would be nice for a first experience but tends to get annoying after a while.
The question is why would you want to do this?

If it’s to give a first time tutorial/introduction video then this could be done by a sketch showing an (interactive) video. which has been flashed and with a EEPROM auto play feature it could auto play after turning power on.
When the intoductionairy is done the EEPROM auto play option could be disabled so the next time power is turned on the menu loads.

I think this will be too complex for bootloader to handle

I dissasgree. many sketches have the system sound setting option and there could be a system settings sketch that allowes you to configure all system settings.

When played from a sketch they could be anything.

It’s the demo sketch that plays them not the library. The library is very primitive atm.

Excactly my preference and can be done as discribed above

I can’t remember exactly atm what the reason was why I chose big endian but it must have been for easier coding/optimizing otherwize I would have picked little endian. I do realize now that if little endian is used a sketch will use/compile to less code. I’ll look into this.

Good to hear you’ve picked that size. It can store the 100+ games + the introductionary video.

The Arduboy string is part of the header and starts at the beginning of the header. the header is single 256 byte flash page (flash memory is divided into 256 byte writable pages and erasable in 4K blocks)

It’s like a filter that is used to group games into a menu list/menu page for easier game selection.
Using Left/Right buttons you can change the list/page. Using Up and down you select a game from that list/page.

The size of a sketch in 128 byte progmem pages. (in the beginning I still used application instead of sketch)

Correct. Using this system will cause the least flash wear.

Easier for the bootloader to handle and optimize code (all organising is done at the PC side or a sketch if you want)

I didn’t work the EEPROM bit out thoroughly yet and picked start and end. When going with length though you’d still have to check if start+length is withing range

CRC32 could be used too but MD5 is more discrimitive. This signature is for the PC side only to help identify a sketch. This counts for the info after the MD5 signature also

1 Like
15

Sorry in advance for the number of questions, but I’m hoping that if I can understand how it all works then I can provide some better documentation.

This is the sort of information we’ll need to edit into the main post.

And “the header” is for each individual slot?
(We might have to standardise some terminology for this to keep everyone on the same page.)

So the “application flash size in 128 byte blocks” mentioned in the main post means 128 bytes = 1/2 page?

Ok, so it’s more of a “category id” or “group id”.

Unfortunately it’s also slower, so hopefully it will be worth it.

(This is one of those many unfortunate cases in programming where both options have a downside so whichever you do, you lose.)

Because it can drop the add operations?

Fair enough, but I wouldn’t have thought a few adds would make that much of a difference.

True, but you have to do that with start and end as well in a sense…

I suppose it will depend which set of checks produce the smallest code.
On a first look, it seems that’s probably the start + end pairing.

start = flashRead(startAddress);
end = flashRead(endAddress);

if(end < start)
	error();

if(start > lastAddress)
	error();
	
if(end > lastAddress)
	error();

Vs

start = flashRead(startAddress);
length = flashRead(lengthAddress);

if(start > lastAddress)
	error();
	
if(length > maxLength)
	error();

// Not accounting for overflow
end = start + length;
if(end >= lastAddress)
	error();

Ok, that makes more sense, although I think it’s worth considering the probability of a collision against the signature size.
I.e. whether it’s worth going for something that’s more likely to have collisions and decreasing the signature size a few bytes (e.g. use a CRC variant), or doing the opposite and increasing the signature size in order to reduce collision probability (e.g. using SHA-256 or RIPEMD-160).

16

No problem. Documenting isn’t my strongest thing :wink:

Let’s call them slots.

A slot consists of:

  • a 256 byte header (1 page)
  • a 1024 byte tittle bitmap (4 pages)
  • the sketch in a multiple of 256 bytes (pages)
  • optional game data in a multiple of 256 bytes (pages)
  • 4K aligment gap for save data and/or eeprom backup in a multiple of 256 bytes (pages)
  • optional save data in multiples of 4K sectors (16 pages)
  • optional EEPROM backup single 4K sector (16 pages)

(K as in 1024 bytes, page and sector are flash memory terminology)

Yes and No. The AVR MCUs program their progmem in pages of 128 bytes. (This value is used by the bootloader flasher) calling them 1/2 pages may be confusing.

Yes. Funny I called them group id early on but changed to list because of how you browse through them.

Yes. But searching through 512 headers takes max ~0.25 seconds in the worst possible case. When browsing through the lists this is neglectable. Sketches do not search nor access the headers at all.
Sketches should only access flash that has been assigned to them through their pointers

Theres no need to keep track of a position pointer too. Quite a few bytes could be saved this way which was very important to make it all fit.

If there was a size constraint I’d go for that or maybe even crc-16. but most of the header space was unused and thought why not MD5. the tools I wrote are in Python and there’s a good MD5 library for it I used before.

There is no implementations for the MD5 signature and following options. As said before it’s information for the PC side utility only. So it could still be anything.

17

I think specifically “game slots”, “data slots” or “game data slots” when being formal.

I’ve never been fond of Arduino’s ‘sketch’ terminology, so mentally I’m calling this “program data”/“game progmem”.

So every game has a compulsory title image?

I’m not sure what this means.

What’s the difference between save data and EEPROM backup?

I’m presuming EEPROM backup only stores copies of the Arduboy EEPROM, and save data is data that’s saved directly to the flash cart?

Ok, so…

The flash chip is divided into pages of 256 bytes, but the progmem of the 32u4 in the Arduboy is divided into pages of 128 bytes.
So a “flash chip FX Chip page” (coined it!) is 256 bytes and a “progmem page” is 128 bytes.

I think “group id” works better.
A list is just a way of viewing them, the abstract concept they reprent is a ‘grouping’ or ‘group’.

(Strictly speaking they’re a mathematical set, but “set id” might get confused for the verb ‘to set’, and will confuse the non-mathematically inclined.)

That’s not too bad. That’s ~0.0005s/~0.5ms per header.

I can think of at least one case where there might be reason to.
If you create a game that wants to access another game’s data.
E.g.:

  • A game that supports transferring data from a demo version of the game
  • A sequel game that support transferring data from the previous game
  • Some kind of tie-in that unlocks special content if you have certain games loaded on the FX chip

If the size isn’t a problem and it’s going to be PC only then I’m tempted to say SHA-256 might be better just to be extra sure that there won’t be any collisions.
I think SHA-256 libraries and tools should be fairly widely available.

1 Like
18

Well, yeah, plenty of games have it in there, but it’d be nice to toggle it before getting into a game in the first place. For instance, Midnight Wild plays music/beeps before the home screen, which is where you toggle the game’s settings.

I was worried about memory usage, which is why I asked. No longer worried. :stuck_out_tongue:

If it turns off the first time it’s played, I think a ‘factory reset’ option would be nice in the settings to re-enable it. You know, for when you go to shows and stuff.

19

Game slots it is. Data and eeprom slots ? Although they are part of a game slot.

Me neither. But as soon as I read Ardunio I switch over using sketch (took me a while though)

Yes the image is mandatory. Because the game selection requires it. Technically you could not include the image. But then the screen is loaded with 1K game ‘garbage’ on screen.

A game is aligned on (starts on a multiple of) page level but writable areas must be aligned at 4K (16 pages) because of the 4K sector erase. If the writable areas are not aligned at multiple of 16 pages an erase will erase part of the program / data

Save data is exclusively for the program to use. EEPROM is used by bootloader to backup (on next burn) and restore (before burn) the 1K EEPROM so other programs may use EEPROM as they wish (excluding the 16 byte system area)

Correct

Exactly :smile:

Ok lets call it group id

Yes me too. I ment in the meaning of ‘do not need to access the headers at all to use their assigned flash memory’

Maybe a game title string should be made mandatory in the header then to easily find a game? The MD5 (or SHA256 or something) may not be usefull for that as they will differ when there has been made a small change like different compiler used, bugfix, improvement, etc

I agree lets change it to SHA256

I’ll keep it in mind.

It could be part of the demo program. when you’ve played though all of it you get a question like ‘Do you wish to start the demo again next time you power on or go to the menu instead ?’ and the auto play flag is set accordingly (you need to flash the demo to progmem anyway for the OOBE (Out Of the Box Experience)

1 Like
20

Eh, I dislike this idea…