Bootloader and File Format [Wiki]

Development Project Falcon
91

The program needs to know where it can find it’s data and save section in FX chip.
Currently it fetches this information from the interrupt vectors. But when a program is developed (directly uploaded through the IDE), it is not possible to fetch that information.
So I assigned a fixed location (read hard coded) for the data and save sections in the library.

The developer can easily define the data and save sections for development.

If there is another way I’m open for it. But It does have it’s advantages. An adapted core could be used that allowes a ‘return to loader menu’ button combo like I do in my homemade package.

92

Humor me for a moment because I’m an idiot, and I still don’t have one of these to play with.

Let’s assume im a dev working on an Arduboy FX game and I want to use the expanded flash memory. Theoretically I could just write to the chip arbitrarily at whatever address I want but that would blow away other peoples games… right? Same kind of problem that we have with EEPROM.

So for now you have a lookup table in the library to avoid collisions?

But what you are suggesting is a change to the system so that the bootloader passes on an address value stored at a “magic number” in ram? So any game that is compiled and run, knows where it’s save file is?

Assuming all the above is correct, how does one request a new save block. How big are these blocks and can you request more than one?

93

Yes. you can use all of the flash memory and any games on there will be corrupted. But it’s not a problem for you as developer.

when you have finished your development you remove the development data and save assignments and export the hex file.

Then you add the hex file,data and/or save data to the PC manager which will assign
space for all files in external flash memory and upload it.

Now when the program is selected from the Loader the hex file is flashed and the data and save locations are passed on to the program.

and… :scream: I just discovered a problem with this idea… :weary:

When you turn Arduboy off and on the data and save locations are lost :sob:

1 Like
94

Ok, got the use case. Lemme have a think about what I think a good UX might be.

95

I completely forgot the vector table was in RAM.

Now the question is whether to load it from progmem,
or just switch to storing the info in progmem in the first place.
(Or EEPROM, since that could be overwritten without flashing the game.)

96

If we use the external dataflash and the internal eeprom we need to keep in mind that they may wear off at different speeds. Not sure but I think the eeprom wears off faster.
So if we replace eeprom along with replacing the game that might kills eeprom much earlier than the progmem or dataflash.

97

Going back to basics. The issue is how can the (variable) locations of the data and save sections be defined for development.

Using a #define in a program is not an option as a library is compiled independently.

Don’t really see Arduino build flags (board.txt) as an option either

Maybe some function overloading @Pharap ? (I’m not good at that stuff)

Only idea I come up with at the moment is an initializer function that copies the data from the vectors to RAM and they can be overwritten by the developer. something like:

cartInit();
#ifdef FX_DEVELOP_MODE
  progDataPage = 0xC0DE;
  saveDataPage = 0xBABE;
#endif
98

How, when and why do they vary?
What are the characteristics of the problem?

I might be able to think of a solution, but I’d need to know what the constraints are and why other solutions won’t work.

99

how much program data does a program need?
One program may need more space then the other.

A developer may want to choose there own program data section (and save data section) rather then a ‘fixed’ location that may not be large enough for their program (like for a video clip)

I’ll put them in global vars.

100

The program “game” could have some header at some fixed location in the progmem that is read by the bootloader. Here the game tells how much data and save space it needs.
The bootloader when reading this data from the external flash will read/fill the fields of the start page/address for data and save (maybe even size if that matters) and programs that into progmem when moving the game from external flash to progmem. The game itself can read the values then from progmem and knows where data and save addresses are.

If fixed location is bad, we can think of an aligned address (e.g. to a 256 byte boundary) and work with a magic number. Aligning is easy with attribute and each game just needs to put the header with the magic at this aligned address. The bootloader can just check the beginning of every page it it sees such a magic value and then do its thing.

101

That’s basically whats done now.

102

That answers why and how, but not when.

Are the addresses something the program dictates or something the FX chip setup dictates?

The decisions has to be made about what the addresses are, but who decides that, and when?

Does the programmer decide that at compile time?
Do both the FX chip and the program need to know?

When and why does the library need to know?
Would it be enough for the programmer to just pass the addresses as arguments to function calls?

I’m still only half understanding the problem.

What’s the motivation for having a separate ‘development’ section at the end instead of just requiring a game that’s in development to have a slot allocated for it?

(Either way the problem seems to still be the same thing - how to let the game know the addresses of its data and save slots.)

103

Oh oops, kind of missed it. Btw, I tried

uint16_t data_page __attribute__((address(0x800133))) __attribute__((used));

and

avr-objdump -t .pioenvs/leonardo/firmware.elf -C | grep data

shows:

00800133 g *ABS* 00000000 data_page

I have to say I am compiling with platformio. Maybe it is different in other environments.

104

const uint16_t data_page __attribute__((address(0x00133))) __attribute__((used)) PROGMEM;

works the same for PROGMEM

00000133 l *ABS* 00000000 data_page

105

During development. The programmer needs to upload their program data and save data manually using a flash writer tool (this could be the PC manager) in external flash. The programmer needs to specify these locations in the program. (Because the program is not loaded by the loader but by means of uploading, the section locations are not defined)

After development. the finished program, program data and save data is stored on external
flash using the PC manager which decides the section locations. When the program is selected to be run/played. The loader patches the program with these section locations at the fixed (vector)addresses.

You could use a game slot as development area. However you need to define the max size of a program and required sizes for program and save data too. If another program is added to the storage afterwards (somebody released an irresistable game to play). You can not easily resize your development slot. By placing it at the end of storage, you will not have this problem.

More specificly. How to let the game know those addresses.

@veritazz

Thanks. But it is no longer needed. (It may still be handy for future situation though)

I’m using Arduino IDE as it is the target platform.

106

Ok.

Sorry for asking again but I got lost in all the posts.

So during development I upload my game (PROGMEM) to the AVR, right?
The data and save sections I upload using an external programmer, right?
The addresses of the external data and save section I choose during developement, right? (hardcode them, e.g. in some progmem variable, and best choosen at the end of the external flash)

So, now after development if the bootloader is loading the game from external flash, it is patching the addresses of the external data and save sections while copying it from external flash to progmem, right?

I am missing the issue. As long as the game (in development or not) is just taking the start of its data page from progmem…

uint8_t *dataptr = pgm_read_ptr(magic address who’s value is either fixed or patched by the bootloader)
read_byte_from_flash(dataptr + x)

then why is there a difference between development and final game? The game is just reading the value from the magic address and its either a hardcoded value during development or the bootloader has overwritten it because it loaded it from external flash.

If it is stupid what I say, then just let me know :stuck_out_tongue_winking_eye:. I’ll stop asking then.

107

Yeah sorry about that it kind grew too fast. It’s hard to filter out the right info now.

Yes

Yes

Yes

Yes

  • We need a fixed location in progmem.
  • I decided to use unused locations in the interrupt vector table so save space.
  • I have not found a way to set these during compiling.
  • Because of above I chose fixed(could be edited in lib) locations at the end of external flash for development.
108

I thought you’d realised that wouldn’t work because the vectors are in volatile memory (RAM)?

Anyway, the most obvious solution seems to be to just have separate ‘in development’ and ‘release’ versions of the functions that retrieve the addresses, with the ‘in development’ version storing the addresses in progmem and the ‘release’ version fetching it from the interrupt vectors.

The only problem I see with that is that the Arduino IDE likes to precompile libraries, which means you couldn’t do something like:

#define DEBUG
#include <Arduboy2.h>

But it could be done with template magic:

enum class LibraryMode
{
	Debug, Release,
};

template< LibraryMode mode >
uint16_t getDataAddress();

template<>
uint16_t getDataAddress<LibraryMode::Debug>()
{
	// Read from progmem
}

template<>
uint16_t getDataAddress<LibraryMode::Release>()
{
	// Read from interrupt vector
}

And then the programmer would just have to remember to switch the call when they go into release mode.

Don’t you end up having to do that in the development approach as well?

Just make the PC manager switch the slots?
You’d have to use the PC manager to put the game onto the FX chip anyway.

The PC manager ought to be able to juggle the slot orders, 16MB will easily fit in a computer’s RAM.

In this case, all it would have to do is:

  • retrieve the contents of the last slot
  • overwrite the last slot with the new game
  • append the retrieved slot to the end (adjusting the addresses as it goes)

Spoiler, PROGMEM is actually defined as:

#define PROGMEM __attribute__((progmem));

So you could (I think) shorten that to:

const uint16_t data_page __attribute__((address(0x00133), used, progmem));
1 Like
109

Ah I see. Thanks for summing up.
So all you need is to set an interrupt vector to a user defined address?

So instead of

<vector xyz>: branch blah

you want

<vector xyz>: address of data page

If that is the case I can think about that. You tried with the attribute way and it did not work, right (choosing the address of an unused vector)? Probably because of the Arduino IDE.

If that is the case I will see if I can do anything.

110

They are in progmem (the facepalm moment was when I copied them to ram, just forget about it)

Templates! Can’t visualise that yet.

Yes, but then:

<vector xyz>:  reti
               uint16_t address of data page

What I’m thinking of using now is:

release

*(uint8_t *)(&progDataPage) = pgm_read_byte(CART_DATA_VECTOR+1);
*((uint8_t *)(&progDataPage)+1) = pgm_read_byte(CART_DATA_VECTOR);

development

progDataPage = DEVELOPER_CHOSEN_PAGE;