Communicating EEPROM usage / collisions

I think it’s increasingly important to simply convey if / what games may have an EEPROM collision.

It’s not intuitive (esp. for non-programmers) to understand what is meant by ‘EEPROM’ and then to look at ranges of hexadecimal addresses and understand if there will be a problem…

My suggestion is to agree a very straightforward labelling convention.
Rather than ‘EEPROM’, I think it’s easier to communicate: ‘Save slot(s)’, or simply ‘Saves’ (?).

Hexadecimal addresses should be communicated as ‘slots’, that are simply labelled A to Z. This is worked out as follows:

  • The first 16 bytes of EEPROM is system reserved.
  • The remaining 1008 bytes is divided into 25 slots (AY) of 32 bytes* each, and the final block (Z) is 208 bytes (32 + remaining 176 bytes).
  • We should report the first and last slot (inclusive) that is used by the program.

Note: * Analysis of existing games’ EEPROM usage showed the median is ~22 bytes per game. If we start to add Checksums, I think the 32 bytes blocks are reasonable. Not sure if this helps or hinders @Mr.Blinky’s plan for backing up EEPROM to the FX chip(?)… Concerns are whether authors use just the first few bytes of a given slot, accelerating wear of the EEPROM.


Example:

Game 1 ~
Game 2 ~
Game 3 ~
Game 4 ~
Game 5 ~

It’s much clearer that Game 2 will clash with Game 1, whereas Game 3 should be fine. For Game 4, with no EEPROM usage, we don’t need to worry. Finally Game 5 is a hog(!) and we might want to avoid, or backup our EEPROM before playing.

1 Like
  • Awesome. Make it so! :rocket:
  • Good - but needs some changes…:slightly_smiling_face:
  • Terrible idea. Don’t even… :confused:
  • Completely indifferent. :man_shrugging:

0 voters

It’d be nice if the eeprom address range could be dynamically allocated during flashcart bin generation so that no collisions could occur. This would require each game though have some standardization so the flashcart assembler script could easily patch the images much like how the ssd1309 patching works on the fly. Another option would be to have a header at the start of eeprom containing assigned offset addresses that would tell each game where to save.

If non-programmers have trouble with hexadecimal, what’s wrong with just using decimal instead?

It’s familiar and takes half the time to explain:

  • The system has 1024 bytes of EEPROM
  • The first 16 bytes are reserved
  • The values communicated are the address of the first and last byte of EEPROM that a game uses, in decimal

I think most of us use decimal values for EEPROM addresses in our code anyway.
(1024 isn’t really large enough for hexadecimal to be a major improvement.)

1 Like

I agree with @Pharap and EEPROM can be called savedata

The slots sound confusing to me. The 16 system bytes are not part of any slot. So games that use them cannot be assinged a slot and slot Z has a quite a different size than the other slots. Also it’s possible for two games to use the same slot but have no collision.

I’ll be looking into EEPROM patching to FX that will reduce the EEPROM collisions and wear :slight_smile:

1 Like

I like the idea but are a little worried as the PPOT games will have all the warning badges.

I am not sure how the EEPROM will work - I understand how each game will report their usage but how will a person be able to understand how someone will interpret this over multiple games. Imagine building a cart with 100+ games and trying to understand what it really means?

Edit: Sorry just seen this thread Communicating EEPROM usage / collisions - #3 by sjm4306 which debates what I just said.



I don’t want to step on Eried’s toes but this could be extended into the Cart Builder that @Mr.Blinky and I built. Having this data would allow someone smarter than me to develop a heat map of potential collisions for the games that the user has chosen for their cart.

I have started modifying games to include a checksum. Currently, I have only done two games but the overhead is really minor. I understand no-one is going to retrospectively fix the old games but at least no one will point to a PPOT game and say ‘it doesn’t work’ again :slight_smile:

Even if @Mr.Blinky can capture the EEPROM state as part of the FX flashing process, this doesn’t help the gaggle of original Arduboy devices out there. The checksum approach does. Its a bit of the survival of the fittest!

1 Like

It just takes some time to take all the games source code and search them all for EEPROM and just write it all down in a spreadsheet. That’s the first step it’s what we’ve strived for it’s what I tried to get going based on what gamebuino did but it just never materialized.

There aren’t that many games that use it from my casual understanding. I’m pretty sure it’s something like 50 games or so?

They code is included in the ardufxtest I just need to unzip all of them… holdon let me throw a few hours at this tomorrow.

ONLY 362 files common no big deal lol.

1 Like

Here is a very rough list, it’s the easiest way I could pull the data because of the zip files contain their own folder structure so difficult to where to pull the name from. But this assumes the first unique folder name within the subcategory is it’s own game.

The number is 160, games that don’t use any folder structure (this is counting individual files) I could do a little more data analysis to clean it up but I think at this point it’s kind of a manual operation to go through each one and see where it touches.

Although, I’m realizing it’s probably somewhat easy to find the start address, without looking through the code you can’t really tell how much space it’s going to use. Right? Dang.

List of Arduboy FX games that use EEPROM

ArduBoyJetPack-master
ArduIndy-master
Bangi
CascadePath-master
Choplifter-master
Crabator-master
glove-master
hopper
Karateka-master
Pyoro-master
Rooftop-master
TinyDepth
to-master
ABasm-master
ArduboyAssistant.ino (49 hits)
ArduboyGamepad-master
ArduboyLightMeter-master
ardubrain
ArduCC-master
Ardutosh-master
ASE-master
book_reader-master
Brainf-ck-master
Calendar
CodersToolbox-master
EEPROMEditor.ino (79 hits)
EggMasterPro2020-master
GolfCompanion-1.0.01
okta-ardu-token-example-master
pocket_pda-master
SetSystemEEPROM
3in1D
ardu-whack-master
arduboy-games-master
arduboy-JumpBoy-master
arduboy-rund-ino-master
ArduboyBackToTheJungle-master
ArduBreakout
arduman_arduboylib11-master
armageddon-master
ASTEROID-master
biribiribeat-master
boris-goes-skiing-master
Chickenlauncher.ino (6 hits)
chribocchi
Evade
Fanboat
Fatsche-master
gamebuino-jezzball-master
Harambe-s-Revenge-master
hollow
joustish-master
Kong-II-master
Kong-master
lasers
LEANNA_THE_LION_AB.ino (5 hits)
Lion.ino (3 hits)
Lion_PlayGame.ino (1 hit)
Lion_Title.ino (5 hits)
LodeRunner-1.04
LoveRush-master
MicroCity-master
Nineteen43-master
Nineteen44-master
OilPanic.ino (4 hits)
OilPanic_HighScore.ino (7 hits)
OlPanic_Title.ino (1 hit)
pocket_fighter-master
RoboDodge-Arduboy-master
Roorun.ino (3 hits)
samegame
SFCave-master
SNAKE-master
SpaceCab-master
src
tacklebox-master
TrainDodge.ino (6 hits)
TurtleBridge-master
Worlds-Hardest-Game-Gamebuino-master
Arglib.cpp (4 hits)
Arglib.h (8 hits)
game.cpp (10 hits)
MYBL_AB.ino (1 hit)
YUM_YUM_AB.ino (3 hits)
arduboy-moire-master
abSynth-FM-master
arduloop
dd9.ino (129 hits)
MidnightWild.ino (5 hits)
Players
BigBlackBox
CastleBoy-master
OMEGA_HORIZON_AB.ino (1 hit)
Squario-master
Super-Crate-Buino-branched
TheBounceArduboy-master
04_tiny_ns_tower-master
05_tiny_sokoban-master
1010.ino (8 hits)
arduboy-hangman-master
ArduboyDigger-arduboy
arduboy_floodfill-master
Arduboy_TD-master
arducross-master
banban-ardulem-7c0bc1e1bcd2
blocks-master
BlockStacker-master
chiemagari
Crates3D-master
Cribbage-master
cripple_mr_onion.ino (29 hits)
Dominoes-master
Farkle-master
gamebuino-solitaire-master
knightmove
KOOKY.ino (11 hits)
LATE-master
LayingPipe-master
Logix-0.94
mazezam
Minesweeper-2.0.0
minesweeper-master
pocket_othello-master
psicolo
puzzle_pack-master
quarto
reversi
ring_puzzle_game-master
SpaceQuarth-master
stairssweep
sudoku_AB_v11-master
term_twirl-master
The-Curse-Of-AstaroK-master
tres-master
YATZY_AB.ino (3 hits)
ard-drivin-master
ArduRacer-master
Randocity-master
UFO-Race-master
Arduminer
ardynia-master
Dark-And-Under-master
ID-46-Arduventure-master
MiniRogue-master
Rogue156b
ShatteredLands-master
StarHonor-master
under-the-tower-arduboy-master
06_tiny_asteroids-master
10_tiny_invader-master
ArduboyArcodia-master
ardubullets
BattleForZuthsrah-master
Humanity_Revenge_DC-master
i4arduboy-master
kemono_patrol
night-raid-master
Source
SpaceKnife-master
Stellar_Impact-master

2 Likes

Thanks @bateske for jumping in. I know how much work this is!
It will be interesting if you get to the same results as @Mr.Blinky and I (?) ~

Lol is this published somewhere else, is this new?

Is there some way to mark if a game has been “audited” yet? or is there some procedure for how to go through and do these?

I think if we made a checkbox or better yet, a sign off for who did the audit completed as well as a real rough “how to” or best practices, only hast to be a few sentences then we could crowdsource this… happy to start digging in myself.

That was the idea in July 2020. We discussed it! :stuck_out_tongue_winking_eye:
See: Arduboy Semi-Official EEPROM Table

It’s been a long few years. So in total Arduboy games use somewhere between 13-14kb in EEPROM?

That’s like, 13-14 times too many. I need to see the problem, gonna try and visualize the findings lol.

Ok I filtered out games that used more than 200 bytes, bringing it down to an almost reasonable 3.2kb over 120 games.

Here are the large games, I see Microcity is in there twice?

Next step will be to add another dimension, and that is the games ranking (number of likes and a few other factors) to look at which of the “favorites” have collision, because if we want to distribute a “collision free” version of the cart some games will have to be trimmed. And I suppose then you could do like A and B (… and C) sides to the cart.

Update:
Boris goes skiing, Midnight Wild, and Ardubullets seem to be pretty safe lol.

1 Like

The EEPROM frontier (Meta post)

Collisions

For any newcomers, a brief history on the topic of EEPROM collisions ~

It’s a near annual topic and I may have missed some relevant posts.


Other discussions

There are plenty more useful discussions too ~
  1. EEPROM backup and restore to PC (with Python) or Android.
  2. EEPROM viewers /editors that run on the Arduboy: One and Two.
  3. Reminder that the EEPROM has a limited life time (100,000 writes).
  4. Adding EEPROM start and end addresses to .arduboy files. Discussion and JSON.
  5. Best coding practices, including checksums and using struct’s.
  6. Abandoned ideas: creating a PC backup format for EEPROM data; Modified boot loader, to allow game saves to be stored in program flash memory space (not EEPROM).
  7. Storing EEPROM metadata in ‘Reserved’ program memory vector area, for use by the FX chip and firmware.
1 Like

To date, most efforts have been targeted at programmers, to standardise how EEPROM is used, etc. This hasn’t worked and it’s not a problem I’m trying to solve here(!)… I think @Pharap has posted some nice code for best practice and it’d be great if functions were added to Arduboy2 lib.

This effort is focussed on the non-technical, end user: How can we most simply communicate (the chance) that two games may collide?..

Sounds nice. But as @Mr.Blinky hasn’t proposed it- guess it’s not practical, and it doesn’t help owners of the OG units.

I think that was the original 2015 idea. It only needs one program to wreck it, so we have to accept it’s the wild-west frontier of bytes! Each program needs to validate its own data and have little expectations for its preservation.

Quite right! Thank you - I’ve been staring at hexadecimal too long and forgot decimal was an option! :nerd_face: :flushed: I changed my original post (currently hidden) based on your comments.

I’m not attached to ‘Saves’ or ‘Slots’ naming (I think I prefer ‘Blocks’). Not sure about ‘Save data’ :thinking:

This is system wide (not game specific). We don’t need to manage it.

True. It’s still a useful first indicator that there may be a problem. In time programs may migrate to these slot boundaries(?). It’s also true two games using exactly the same EEPROM addresses may not collide: one game may simply not save! All we are communicating is some risk of overwriting save data. Note: The original 2015 proposal used blocks of 16 bytes, so we don’t lose much going to 32 bytes.

True. Good point. I saw this as just a simple first step. Most of us have ~5 favourite games, that we’re likely to swap back and forth between. If one of them is glitchy, I’d look at the forum posts for the 5, and see two using ‘slot C’, which indicates a problem.

On your cart builder, saves slots could be indicated to the user when you are selecting hex files. I’d imagine you’d store actual EEPROM locations in your database for each application. Then at compile time (or earlier), warn the user of collisions?

Getting and maintaining EEPROM addresses is a huge task. It makes sense to put that responsibility on the authors. Hence the idea of a badge system to encourage this behaviour. I think it looks fun / kinda gamification of this essential metadata.

That sounds great. :fire:

Who would be nerdy enough to tweak other people’s code and raise hundreds of PR’s…? :innocent:

Yes. As a separate endeavour, it would be nice if this could be a standard function of the library.

A recent real-world example. The user had 5 favourite games, so knew the issue was between them…
Look at the following, quickly scanning your eyes over the details. What is the simplest and fastest indicator?



EEPROM hexadecimal addresses:
  • - 1nvader (PPOT).
  • - Hollow Seeker (Obono).
  • - Hopper (Obono).
  • - Sirene (Team ARG).
  • - Space Cab (PPOT).


EEPROM decimal addresses:
  • - 1nvader (PPOT).
  • - Hollow Seeker (Obono).
  • - Hopper (Obono).
  • - Sirene (Team ARG).
  • - Space Cab (PPOT).


Save slots:
  • - 1nvader (PPOT).
  • - Hollow Seeker (Obono).
  • - Hopper (Obono).
  • - Sirene (Team ARG).
  • - Space Cab (PPOT).

(Note: Obono’s games use 32 bytes but are not aligned with slot boundaries. They currently occupy X‒Y and Y‒Z).



  • Hexadecimal (:face_with_monocle:)
  • Decimal (:open_hands:)
  • Alphabetic (:capital_abcd:)

0 voters

Simple method using a spreadsheet to determine the ‘slot’ / ‘block’:

 = CHAR( INT((address-16)/32) +65)

So block 0 (system reserved) gives ‘@’. Obviously, need to make sure it returns something sensible between AZ.

Honestly, I think if I turned my graph into javascript you could type in your game and it would list off other games that collide and also show it visually with horizontal shading.

EEPROM management is dead! Long live EEPROM management!

I mean, basically I realized early on it was a mixture of apathy from the developers combined with the fact we probably would run out of space anyways and always deal with collisions. Documentation was the part I really missed.

1 Like

It hasn’t ‘worked’ because it’s not a solvable problem.
You’re looking at a pigeonhole problem - trying to fit several KB of data into a 1KB block simply can’t be done.

The goal of picking different addresses was never really to avoid treading on other game’s data because that’s unavoidable.

The goal was to reduce wear on the EEPROM. If everyone started writing their game data at the first non-reserved byte then that block would wear out quicker than every other block. I.e. the intent of spreading the data is more about wear levelling than preventing collisions.

As @bateske says, the real issue has always been that people haven’t been in the habit of documenting the EEPROM range their game uses.

There could be a system that dynamically allocated ‘blocks’ to games when requested, which would solve the issue of knowing when there isn’t enough room to save more data, but aside from the issue of added complexity, such a system would only work if every game used it. Hence you still end up back at the square one problem of documentation - documenting which games use the ‘safety system’ and which just write data to an arbitrary block.

That’s why I’m reluctant to even attempt considering such a system - it would be a lot of work for something that’s only a partial solution

@Mr.Blinky’s idea of adding backup & restore functionality to the FX bootloader is more practical because it would automatically work for all games. The only real downside to it is the added wear to the EEPROM (which could be mitigated by making it optional and/or only affecting the area that the game header claims to use).

There’s also the difficulty of updating the bootloaders, but at least that would be targeting the people for whom the collisions are a problem rather than trying to target every published game.

@Mr.Blinky’s Python scripts already include a script that can read the contents of the Arduboy’s EEPROM and write them to a file. It might not be as fine-grained as what this thread was proposing, but as long as the user is careful to give thier files decent names it would work.

The only real advantages of something more fine-grained would be wear levelling and possibly ease of use.

Out of these options, decimal, followed by hex.
The letter system omits vital information and, as pointed out by others, can lead to false negatives.

Considering other options, the best option would be to somehow make the cart-building software do the detection and warn the user:


This is what we really need: data + tooling.

For now what would probably be most useful is to try to stuff the game names and EEPROM offsets into some machine-readable format (e.g. CSV, JSON) to make it easy to create tools to analyse it.