FX flat file system?

We might discuss it in a different thread but can you elaborate why you need the file system and for which part of the system? Is it the game, or the bootloader or something else? If possible also list your intention behind this and the problems that you want to solve. It might help the discussion.
Personally I was looking at the memory for “my” games as a linear memory which I can address relative from a given base address. The base address comes from somewhere (I do not care) and all I manage are the relative addresses to my memory. I am talking about the SPI flash. The EEPROM was out of scope for me as with SPI flash I can write to there.

Let’s open such a thread and talk about it.

1 Like

Not a hierarchical file system, a flat file system.

Essentially each game would have either a multi-byte identifier (be it a string or just a short globally unique identifier) that it can pass to the FX API to receive the address of its game data.
(And that would probably be looked up in some kind of file allocation table at the start of the chip.)

The intent would be to make it easier for games to locate their memory.
From what little I remember of the existing approach,
games have to be recompiled with the specific address of their allocated memory block.
(Or something about a vector table that I can’t remember the details of and never really understood to begin with.)

An approach using an allocation table would permit:

auto gameDataAddress = FX::find(gameIdentifier);

Unfortunately someone has to.
I very much care because having it hardcoded would be a nuissance.

Great. Thanks for starting this. I see your points. Actually from the games point of view the filesystem is opaque. It does not see it and probably would not use it except finding its data within. Probably even at the start. I think we would need it if there is an interaction between the games or some higher level application that manages the filesystem. What I understood is that the binary images are constructed offline so there is nothing that changes once the binary image has been build.
Now about the vector magic, what I can say about this when using it this is something that “usually” does not need to be recompiled.

The way it works is that during creation of the binary file for the flash (the overall flash image for all games) the tools write the flash base for each game to a specific memory address of that game (it patches the PROGMEM of the games binary). So the game only needs to read the value at this address to get their base address of the data in flash. For this an interrupt vector address has been chosen because it is in some memory that no-one uses. So the games itself do not need to be recompiled as they only need to read this address to get the flash address.

However one special use case here is the development. Here the developer does flash the game himself onto the Arduboy and thus needs to set this flash address himself. It can be retrieved from the tools when greating the binary blob for the game (it prints it) and then put into the code. But here I think this is a special case and is already handled by @Mr.Blinky and his tools. But we can look at it to make it easier to use.

All my above text is only interesting for games specifically written for the FX mod. One problem is still the EEPROM but as far as I remember there have been several attempts already to manage this memory between the games, so we can maybe take it out of scope for this discussion. Please correct me if I am wrong.

2 Likes

Maybe at the evening I can come up with a simple drawing to visualize the idea behind this.

Ok, thank you for that explanation, that’s made that a lot clearer.

This is the case where the file system would be useful.

Instead of having to set the address in the program,
the address would be stored in the ‘file’ table at the start of the FX chip and then looked up dynamically by matching the program ID to an entry in the table.

(Note that when I talk about ‘files’ I mean it in the sense of a block of uniquely identifiable external data, explicitly without the implication of using a URL to identify the ‘file’.)

Basically the table would look like this:

Game Id Offset Size
{identifier} {integer} {integer}

Either for some fixed size, or with a prefixed length.
‘Game Id’ would probably be an (unsigned) integer of at least 16 bits.
‘Offset’ would probably be either a page index or a byte index.
‘Size’ would probably be either a quantity of pages or a quantity of bytes.

So a structure representing a table entry might be:

struct TableEntry
{
	uint16_t identifier;
	uint16_t offset;
	uint16_t size;
};

And then the start of the chip contains the equivalent of either:

constexpr TableEntry table[someFixedSize];

Or:

constexpr uint16_t someVariableSize;
constexpr TableEntry table[someVariableSize];

(And then the table could be sorted so that a binary search could be used instead of a linear search, unless that’s going to cause too much wear on the chip.)

The important thing is that this way the program itself doesn’t change after being compiled,
the only thing that changes is the data on the FX chip.
This way it’s possible to rewrite the chip without reflashing the Arduboy itself.

It also allows people to identify data from other games,
which could be useful for some cross-game trickery,
like migrating character profiles from old games to new games.

I realise this would probably be a burden on the bootloader,
but to be honest if we’re giving games access to that much external data I don’t think having a small bootloader is particularly important.

I can’t remember either way, so I’ll just ignore it for now,
I’m not awake enough to work that one out as well.

1 Like

This was the part that was confusing me. If the address were stored in a specific part of PROGMEM then wouldn’t it be better for the library to default set the value to 0 or something else and then in development the game can simply be written to flash in the same way it would for a release build. This way the developer just has to make some kind of call auto flashStart = getFlashDataAddress() then from there it’s a simple matter of reading your data however you have the memory allocated. Basically thinking in terms of putting all your data into a single file and only reading from one file instead of the more traditional method of having separate files for each sound piece, map data, image data, etc. Instead everything would be packed into a single “file” which is to say that it’s in a single array of memory that can be accessed randomly.

This would greatly simply the point of view from a developer perspective since a basic implementation would simply put several files end-to-end when creating the flash data and then offset values could be stored as static const variables in code. Obviously there are better ways to optimize the code and data but this one is the simplest to dive into.

If the official MOD decides on a 16MB instead of 8MB then we could warrant sacrificing a tin y amount of extra space if it makes development for people not familiar with interrupt vectors to use it. You would also make it easier so people don’t read “create your data file and code, then run the tool to create a flash bin and find out where it’s going, then set those values in your code and re-compile.” This leads to the confusion surrounding the question, “How will where an end user will place my game on there flash cart.” This was a question I myself asked as well even though I ported @Mr.Blinky python scripts to c++ for use in the ArduManFX and didn’t really realize that that’s what the part was doing that modified the binaries values at the addresses 0x14, 0x15, 0x16, and 0x17 with a 0x18, 0x95 for the first two and then the data page address for the latter two.

One thing we have to keep in mind is simplicity for the end user from the perspective of the end user (something that’s often difficult to do because things seem simple to us but may not be so simple). Obviously there’s some level of restriction on how simple to go. We can at least assume basic knowledge of c++ but shouldn’t rely on advanced knowledge of embedded programming and design otherwise the FX support for games will “appear” out of reach for all but expert developers, and once its out not using it would force higher limitations on content. Some may choose to not use it, not because its difficult but perhaps for the fun and challenge of a more limited design, while others might shy away from developing for the Arduboy all together if they feel a large part is inaccessible to them unless they spend countless hours learning about interrupt vectors.

This was what I was thinking of as well. You would still create your game and data in much the same way, but to access the data you could require passing an 8-byte array (which can be easily viewed as a string) and the library would lookup the relevant location from the small lookup table at the beginning of the chip. So for example a game could initialize the flash address with something like auto flashStart = getFlashDataAddress("GAMENAME") where GAMENAME is max 8-characters, or even an 8-byte array. If the flash address takes 3 bytes (rounded up to 4 for evenness) then we could store ~21 game data locations in a single 256-byte page. So even with an 8MB chip it wouldn’t be too costly. Especially since the amount of pages required could in fact be dynamic based on how many games with flash data a user chooses to install (4Kb would allow 341 games with flash data addresses, which is over 100 more games than what currently even exists).

1 Like

Yes, precisely. This is exactly the way I’ve been thinking of it.

Although, those ‘files’ could start with offsets to particular sections if the user wishes.
I could imagine it being common place to start each file with a structure like:

struct Offsets
{
	uint16_t graphicsOffset;
	uint16_t soundOffset;
	uint16_t levelsOffset;
	// ...
};

And then resolving those to ‘FX pointers’ (fancy uint24_ts).

(This puts me in mind of ‘Resource Interchange File Format’ (RIFF) files that use ‘four character codes’ (FourCC) to provide header information.)

They probably wouldn’t need to be static.
(Not that it’s likely to make a difference either way.)
I often feel that static is misunderstood…

I’m glad to know there’s at least one other person in a similar boat.

Precisely.

I’m well-versed in C++ but not so well versed in hardware.
I can understand how the page system works and how the SPI commands work,
but reading datasheets is a struggle.

Abstractions are the bridges we need to build.

This is one of the details I’ve been on the fence about.

The programmer side of me wants to say “65535 identifiers is plenty, and we can just express them in hexadecimal” because that’s wonderfully efficient.

But the side of me that’s forever trying to think of the inexperienced (as I myself once was) keeps thinking “we have to make identifiers strings so people can pick memorable names and won’t be put off by hexadecimal”, but at the same time I end up thinking “all those wasted byte values” (all the non-printable characters and everything over 127).

This is more or less equivalent to my example of auto gameDataAddress = FX::find(gameIdentifier);,
but with gameIdentifier being a character string/array.

As abstractly as possible: identifier goes in, FX address (be it page or byte) comes out.

Technically "GAMENAME" is a 9-byte array, so it would suffice as long as the excess null character is ignored.

This is the realm where template voodoo can be used to stop people providing string literals that are too long and/or using non-literals.
(static_assert is your friend.)

Things become a bit more difficult when you ask “how do I put the string literal in flash?” though, which is another reason I lean towards integers expressed in hexadecimal.

The idea behind this system is that you can easily develop FX games while you also have a games collection on your FX chip. For this reason the development data is stored at the end of the FX chip. Initially this started at a fixed location but to be more flexible the start location is dynamic. The larger your data is the lower the start location (just as if you where pushing your data onto a stack)

A developer writes his FX data to the FX chip using the FX tool specifying the -d switch so the data is written to the development area (end of FX chip) and the tool poops out a start address(PROGRAM_DATA_PAGE) you pass along your FX::begin() function.

flashcart-writer.py -d development-fx-data.bin

Now if the developer doesn’t care about anything stored on FX chip (s)he can pick any location to store the development FX data.
At the beginning (pass 0 along the begin() function) :

flashcart-writer.py development-fx-data.bin

or at some other location (128K from end)

flashcart-writer.py 0x7E00 development-fx-data.bin

The PROGRAM_DATA_PAGE only needs to be specified atthe begin() function during development and is no where else used. As far as the libray functions are concerned your data always start at (relative) address 0

Now when that game is finished and put into a game library/flash image. the value passed on to the begin() function is irrelevant (can be left unchanged). The flash-builder tool decides where games and game data are stored on the FX chip and will pass (patch) that information along to the game binary

The idea is to have a tool that parses a custom script containing data, links to data and image files and poops out a header file and your fx data file. You include the header file onto your project and upload the fx data with the flash writer tool.
The header file contains constexpr offsets(pointers) to the data. You could look at them if they where file pointers/handles

An end user only needs to how to use the writer tool to upload flash images or the builder tool to make their own images. They don’t need to know any techy details. Just like for browsing the internet you only need a browser and don’t need to understand TCP/IP

Nobody needs to learn anything about interrupt vectors to use FX chip. Somebody started fake facts.

My systems goal is to be fast and require little code and don’t think it will be difficult to use once the parser tool is ready. But a filesystem can always be put on top of that.
If anyone wants to develop a filesystem and tools for it. I’m available for technical questions.

2 Likes

So what we have at the moment:

A1. A library that allows flash memory access and handles the passing of the base address in flash (maybe needs more visual documentation)
A2. Tools that allow upload of binary data for development or full flash images with multiple games (flow might need to be documented)

For 2 also other tools can be written that are not command line based if required.

What is missing:

M1. A tool for developers that allows them to assemble their own data file on flash to be used by the game and the tools in 2)
M2. Documentation of the flash file layout produced by flashcart-builder.py or more precisely, required by the FX bootloader

Actually M1 is something that can be done also by every developer himself but a general tool available for all would be a good start. For people that are more skilled or have special requirements it should be no problem writing there own more specialized tool.

I can help with the documentation. Is there a way to properly put the documentation here in the forum? Does anyone know which tools to use for some simple drawings?

We also have example skethes, the flashcart test app reads images from the bin file to run the bad apple video and the ball sprite test demo also pulls the sprite and background images from the external flash.

Posted the below image a while ago. It’s still accurate only the size of development area is no longer fixed.

2 Likes

Things make a little more sense now.

So essentially its only required to know the address if the developer wishes to not mess with any content they may already have on the FX chip. This address can also be easily obtained using the flash-writer script (will need to incorporate the -d switch into ArduManFX).

So if I understand correctly the flow would be like this for development (only needed if you don’t want to mess with any content already on the flash chip):

  1. Create your data file
  2. write data file using -d flag
  3. Get PAGE_ADDRESS of where the development data is stored
  4. Compile hex file passing PAGE_ADDRESS to Cart::begin()
  5. Upload hex file like any normal (non-FX) game
  6. test game to make sure everything works
  7. Distribute hex file (no recompiling needed) along with data file.

If you don’t want to deal with the added steps for easier development (or if you don’t care about the content already on the FX chip):

  1. Create data file
  2. Build hex file (without needing to pass any PAGE_ADDRESS to Cart::begin()
  3. Create a normal flash cart image containing your hex file and data file (plus any other games you wish to upload)
  4. Upload the flash cart image as normal and launch your game as you would any other game

M1 is something that I plan on incorporating into ArduManFX eventually (first focusing on getting basic flash creation and uploading implemented so it functions as a general purpose game manager). I was also planning on incorporating the image conversion routines as well. For the resource packer that’ll be fairly simple since you could easily just use a text file with each line being a resource (using relative paths to the text file for easier distribution of code+assets) then develop a simple GUI to more easily manage resource packs in a visual manner. Once a GUI is implemented any documentation and instructions can be incorporated directly into the interface.

M2 seems like we have good documentation, just need to gather it all into a singular location that can be easily updated (like a wiki doc or something). Perhaps a github pages that multiple people can be moderators on and anyone can submit PR’s to.

For flow chart diagrams I’ve really liked DIA

1 Like

correct

At step 2. You still need to pass along a value so the library knows game data is used. but it can be any number (preferably 0 which may get optimized better).

Note that in this case even after all step are completed, you manually need to select the game and flash it (you cannot command the bootloader to select a game from the list and then flash/burn it)

Using Cart::begin() will initialize flash but no attempt is made to locate game data. This function is only used when the game has no game data and you want access to the full address range of flash memory

That would be very nice.

I was thinking the same

1 Like