Communicating EEPROM usage / collisions

Honestly, I think if I turned my graph into javascript you could type in your game and it would list off other games that collide and also show it visually with horizontal shading.

EEPROM management is dead! Long live EEPROM management!

I mean, basically I realized early on it was a mixture of apathy from the developers combined with the fact we probably would run out of space anyways and always deal with collisions. Documentation was the part I really missed.

1 Like

It hasn’t ‘worked’ because it’s not a solvable problem.
You’re looking at a pigeonhole problem - trying to fit several KB of data into a 1KB block simply can’t be done.

The goal of picking different addresses was never really to avoid treading on other game’s data because that’s unavoidable.

The goal was to reduce wear on the EEPROM. If everyone started writing their game data at the first non-reserved byte then that block would wear out quicker than every other block. I.e. the intent of spreading the data is more about wear levelling than preventing collisions.

As @bateske says, the real issue has always been that people haven’t been in the habit of documenting the EEPROM range their game uses.

There could be a system that dynamically allocated ‘blocks’ to games when requested, which would solve the issue of knowing when there isn’t enough room to save more data, but aside from the issue of added complexity, such a system would only work if every game used it. Hence you still end up back at the square one problem of documentation - documenting which games use the ‘safety system’ and which just write data to an arbitrary block.

That’s why I’m reluctant to even attempt considering such a system - it would be a lot of work for something that’s only a partial solution

@Mr.Blinky’s idea of adding backup & restore functionality to the FX bootloader is more practical because it would automatically work for all games. The only real downside to it is the added wear to the EEPROM (which could be mitigated by making it optional and/or only affecting the area that the game header claims to use).

There’s also the difficulty of updating the bootloaders, but at least that would be targeting the people for whom the collisions are a problem rather than trying to target every published game.

@Mr.Blinky’s Python scripts already include a script that can read the contents of the Arduboy’s EEPROM and write them to a file. It might not be as fine-grained as what this thread was proposing, but as long as the user is careful to give thier files decent names it would work.

The only real advantages of something more fine-grained would be wear levelling and possibly ease of use.

Out of these options, decimal, followed by hex.
The letter system omits vital information and, as pointed out by others, can lead to false negatives.

Considering other options, the best option would be to somehow make the cart-building software do the detection and warn the user:

This is what we really need: data + tooling.

For now what would probably be most useful is to try to stuff the game names and EEPROM offsets into some machine-readable format (e.g. CSV, JSON) to make it easy to create tools to analyse it.

To this effect, at least the awareness raised by the multiple posts has been enough so that naturally people seem to have spread their games out.

I mean, at least we aren’t dealing with a nightmare situation where everyone has started at address 0. (which I think is reserved? aren’t people supposted to start at 16?)

A tool that also allows users to submit their game data would be great, but without building some kind of usermodel it is kind of a pain to do.

Fancy way might be to allow a community post where you could write it out and then an api could pull it.

But probably it will need to be actively managed and people can email when it needs an update…

Or it could just be a public csv and allow open edits?

The user-permitted addresses start at 16 because the first 16 bytes (0-15) are reserved, but users aren’t supposed to start there because of the wear levelling issue - if everyone starts at 16, the first block of memory would wear out faster than all the rest.

The problem with making it a single thread is that threads lock after 500 posts.

A better way would be to pull all threads from a specific category (i.e. #games and #games:demos ), or if a thread tagging system were enabled there could be a specific tag that registers a thread for cataloguing.

It would probably be easier to do via GitHub because GitHub has ‘webhooks’ for when repos are updated, but not everyone uses GitHub.

If it were hosted on GitHub then that might work, especially since that would potentially allow the option of having a server auto-update using webhooks.

I’d say avoid CSV for something that people are going to be editing though, because it’s positional rather than named, which makes spotting mistakes and validating the information much more awkward.

Either JSON, XML or something like recfiles would be best, but anything with key-value pairs would work.

Sure I mean we could build an entire framework in sql and develop a react server… battery backup… AI powered… triangulated space satellites. Orbital confirmation network. Sure only take 15 minutes crank that out.

Let’s see how the game jam goes, if it’s a big hit I’ll put a bounty on making a solution to the EEPROM.

I thought the idea was to make it easier to understand. If you ask x out of n users about what EEPROM is and what a savegame is, I’m pretty sure the savegame is better understood.

maybe I misunderstood something. I thought the idea was to document collisions for people to understand. But you’re talking about managin (future) EEPROM ?

I had some ideas for a simple dynamic EEPROM system but the semi-official-eeprom-list showed it couldn’t prevent collisions. Such a system could still reduce EEPROM wear but I don’t really see that as a problem.

Yes, each time you change a game that uses EEPROM, the EEPROM would be restored. But I don’t think it would be really a issue as EEPROM can be rewritten 10 times more than a program can be flashed.

A more wear reducing method that I recently thought up is to patch a game with EEPROM emulation so the FX chip is used instead of EEPROM. This would only work for games stored on FX chip though.

The advantage over the backup/restore method is less EEPROM wear and the bootloader doesn’t need to be updated.

I’ve looked a bit more into the possability of patching games for EEPROM emulation and found out the EEPROM library uses the avr libc eeprom functions and those where created from assembly source files meaning the code will be the same and can be located easily for patching. a EEPROM emulator can also be appended to the game code as there is additional PROGMEM space available because of the Cathy3K bootloader and if there wouldn’t be any space in a worst case scenario. Well then at least the number of collisions with other games has been reduced :slight_smile:

I’m pretty confident a EEPROM emulator is feasible. Maybe something to work on during the upcomming jam…? :wink:

BTW @MLXXXp I recommend to replace with eeprom_read_byte and EEPROM.update with eeprom_update_byte in the Arduboy2 library. These changes can save 18 bytes.

Now you’re talking :money_mouth_face:

1 Like

That would depend on how the allocations happen.

Most allocation systems start at the earliest available memory and work their way forward, and in this case that would lead to earlier blocks being written to more often than later blocks.

To avoid that there would have to be some way of ensuring that allocations spread the wear across the whole address range rather than bunching up around particular areas (or at least in a way that minimises reusing the same areas continually).

That’s also why a file allocation table style system is bad - the table is going to be overwritten more than most other areas.

It would depend on how people use them.

I was under the impression most people are more likely to play the same game long enough for end up with multiple highscores/saves before switching, but I can only really speculate on what people do.

Yes, you mentioned. In terms of doing that via a library it would be a good idea, but I’m not sure how well patching a binary after the fact would work.

Overwriting the eeprom writing instructions with jumps/calls to the new FX-manipulating code probably isn’t too difficult, but surely you’d have to make sure the replacement is equivalent, and you’d have to find an area of progmem that is currently unused?
(That’s probably easier if you’ve got the .elf file than just the .hex file.)

That only solves things for FX units though, not the earlier units.

Surely for earlier units, that could be a problem because it would be attempting to overwrite progmem in use by the bootloader?

I presume you mean internally?
If so, yes, that will be a saving
I stumbled upon this quite a while back and forgot about it - I did the same thing to one of my games at some point.
(I think I mentioned it to Filmote at the time, but I can’t remember if I mentioned it to MLXXXp.)

Arduino’s EEPROM library is quite overengineered for how most people use it (EERef and EEPtr are overkill), and it’s a bit too self-reliant to play well with the optimiser.

All they really needed was a template function that calls eeprom_read_byte/eeprom_read_block and eeprom_update_byte/eeprom_update_block with the appropriate pointer cast and sizeof(T) - i.e. equivalents of get and put with simpler implementations.

EEPROM emulation (and EEPROM backup/restore also) wouldn’t work for the older units as there is no extra memory available. the Patching process would be done by the flash/cart builder tool. so patched games will only be part of a flash image.

Yes. based on an empty sketch using arduboy2::begin();

yes, the line

EERef &update( uint8_t in )          { return  in != *this ? *this = in : *this; }

effectively uses eeprom_read_byte and eeprom_write_byte where they should have made use of the existing eeprom_update_byte function.

1 Like

In that case there’s no issue. I was merely concerned that we might end up with .hex files floating around that would cause problems if someone tried to load them onto an old unit.

Then yes.
I believe the same applies to most games too.

I was envisaging something much simpler that doesn’t involve any patching. Why can’t you simply copy the entire EEPROM to SD when you flash the next game?

That would require a new bootloader with backup and restore feature.

Ah, I see.

What do you guys fancy?

  • I want EEPROM emulation and patch games using the flash/cart builder tool
  • I want EEPROM backup/restore feature in a new bootloader

0 voters

Before I vote …

How do we get all FX devices to use the new bootloader? I assume you need to flash that onto the machine. Would it be fair to say that if we updated the bootloader, people using the original bootloader will not be affected - they just will not have the EEPROM backup / restore feature?

Will the patching approach be consistent across all homemade devices? I assume it will be but maybe the chip being used comes into play?

probably a sketch to update the bootloader.

I asume you mean the original Arduboy FX bootloader?


yes but if the FX chip is addressed differently the emulation code will be slightly different.

In order for the bootloader to backup/restore, the flashbuilder tool still needs to add the eeprom start and end info though. each game requires a 4K EEPROM block (EEPROM emulation requires 8K per game or 4K and a shared 4K block)

Yes, sorry that was ambiguous.

Is backing the full EEPROM not an option?

backing up full 1K isn’t a problem but without the start and end addresses the full 1K must be restored as well. I don’t think that’s a good idea.

Sorry if some of my questions are simple …

Why not? Are you just concerned about wear?

OK … just thinking about solutions that don’t require us to catalog all the existing games.

Its not just the backlog that needs to be considered, its future games as well. The average first time developer will smash out a game with little understanding of what all this ‘EEPROM management stuff’ is and someone else (probably one of the contributors to this thread) will have to do the heavy lifting.

So if you need to have the start / end address anyway to do a backup / restore function in the bootloader then it sounds to me like the patching option is a better solution. This assumes its doesn’t need the start / end addresses.

100% of the voters agree with me!

I’d like to reduce wear if possible. But thinking about it more if PROGMEM wears out after 10K flashes. So a full backup and restore on each flash leaves 90K EEPROM rewrites. I guess that’s not so bad.

Nope not required.

Exactly. The reality is that no one swaps games that quickly … or do they?

Should I change my vote?