Shared EEPROM storage management across multiple apps

Thinking of how to manage EEPROM storage for multiple apps (assuming someone will play multiple games over time and re-flash their devices but may switch games periodically and still want to restore game state (high scores, saved games, etc) from the previous app.

My goals:

  1. Simple, easy to allocate whether 5% full or 95% full
  2. Apps can request the amount of ROM they need (be it 5 bytes or 50).
  3. Avoid issues of memory ordering or data fragmentation (we do not want to have to run a periodic defragger utility!)
  4. Easy to extend the amount of ROM you need or clear your allocation (say if you have a gave with 3 save slots and you delete all 3 slots from within the app they app could easily just clear it’s whole block allocation as a good citizen)
  5. If we ever have a saved block manager it should not have to guess at which apps are using what space (APP ids should be clear)

What I’ve come up with:

  • 16 bytes per block, 64 blocks (1k of EEPROM)
  • 1 block reserved for system use (contrast, etc)
  • 63 blocks available for application use
  • Blocks are stored in chains, so one block can reference the next (linked list style).
  • First block:
    • stores the application ID, which is an 8 character string (usually the app name)
    • 7 bytes of data
    • 1 byte of metadata
      • 1 bit for “used”
      • 1 bit for “first”
      • 6 bits for “next block” address
  • Additional blocks:
    • 15 bytes of data
    • 1 byte of metadata (see above)

So for example if Breakout continued to want 50 bytes of storage it’s storage (over time) might look something like:

  1. “Breakout[7 bytes data]” -> 5, used, first
  2. free
  3. free
  4. free
  5. “[15 bytes data]” -> 6, used
  6. “[15 bytes data]” -> 7, used
  7. “[15 bytes data]” -> END, used

It would ask for 50 bytes and would receive 52. All of this would be managed with a simple API.

Allocation requires writing only the 8 byte header and then 1 byte for each additional block requested (to set the “used” storage flag. You can scan for available blocks with 63 reads. Release simply requires writing 1 byte for every block to clear the “used” flag. Finding an existing block requires an iterative scan for “used” + “first” blocks and then matching the application ID and as soon as the match fails you short circuit and try the next match.

The app would never concern itself with the storage management. It would just ask for 50 bytes and it would get 50 bytes seemingly directly addressable (by index) same as the API EEPROM library currently exposes.

Secondary blocks only “lose” 6% of the space available and first blocks lose 56%… but I think that’s a small price to pay for storing the name of the app right in the ROM, so later when it comes time to manage that space there is no question about what data you want to erase, clear, etc.

Sample code:

SavedData saved_data = arduboy.eeprom_data_find_or_create("Breakout", 50)
if (saved_data.allocated()) {
  int x =
  saved_data.write(0, x+1)
  saved_data.write(49, high_score_byte)
  saved_data.max_size(); # 52

So you could have 63 games with 7 bytes of storage, 32 games with 20 bytes, 16 with 35 bytes, or any combination.



Very good idea. I like it.

Few questions…

Is this meant to be included in the bootloader?

If so, how much storage do you think this little program may take? The flash storage is supposedly the most limiting part of the device.

What about lazy devs? We are surely going to have them, and some devs won’t bother implementing this if it is too complicated and there is an easier way around it which = lost or some very odd looking data when bits have been directly overwritten.

I do really like this idea though, and I think this may be a great solution to one of the many limitations! Great work!

Is this meant to be included in the boot loader?

Well, not the boot loader but the idea would be that it’d be small enough to link into all programs and we’d encourage devs to use the EEPROM API we shipped - so all the apps just work together with no effort. We couldn’t force someone but if we got a majority then everyone would have to get in line or risk pissing off users by corrupting their saved data.

The flash storage is supposedly the most limiting part of the device.

Ha. I’d argue the RAM and EEPROM are the most limiting resources, not flash RAM. Hard to say without writing it and compiling… but it would mostly be some integer math, some loops, byte comparisons, bit checking… easily less than 1k I’d think?

And I’m not sure what the alternative is… chaos? Or everyone editing sketches before they compile them and keeping a notebook on their dresser of which apps are using which spots? And then the user has to do their own defragging (but how?) if they play with too many apps and get their space fragmented poorly.

Yeah I think the best way to make sure everyone uses it is to make sure that there are regulations on the “app store” before a game is accepted. Proper use of this should definitely be one of them!

I haven’t done much either but I’m going by what others have said on this forum. A few people have mentioned that its the 32KB of storage that runs out quickest because once you add a few bitmap images, its pretty much gone! One single bitmap the size of the screen is 1KB. Perhaps some procedural generation of some things may be the way to go?

The alternative? Well I hadn’t really though about it. Before I started loading programs to the device, I thought the EEPROM would have been wiped in the process. But since learning that it can be saved, this may be the best solution!

It’s possible to program EEPROM at the same time you program the flash (and set it to whatever contents you want, extra data, etc). but pretty sure that does not happen automatically.

Procedural generation is definitely how you would do any type of scroller. (and in that case it should be possible to kill the 1k display buffer completely also).

Since EEPROM is addressable per byte the 16 byte block size would technically be up for adjustment… The pointer format would need tweaking if you wanted to go smaller, but you could easily go larger… say 32 byte blocks… though I’m not sure what the benefit there is… seems like you’d just be wasting a lot of space for a maybe 1% improvement in MAX storage.

I believe that we to have an EEPROM storage Protocol. However because only one game runs it should have dedicated access to the EEPROM. What I suggest is the game gets full access to the EEPROM space except for a small header.

A Program on a PC a “Game Loader” could backup the EEPROM, program the new game into flash and store a previous backup into EEPROM.

My idea for a protocol would go something like this.

  • 1 Byte Protocol ID (Just a unique number to know the EEPROM comforms probably not 0 or 1)
  • 1 Byte Protocol Version (Just in case we change our minds)
  • 8 Byte Game Name (just like you suggested)
  • 2 Byte Game Version
  • 2 Byte Chapter Pointer (Chapter Specific Saved Data Will Be Saved Separately On PC from Game Data)
  • Many Bytes Game Data
  • 1 Byte Chapter Protocol ID
  • 1 Byte Chapter Protocol Version
  • 2 Byte Chapter Version
  • 8 Byte Chapter Name
  • Many Bytes Chapter Data

I would much rather that apps can share the space than “installed app gets it all”. Not all users of the Arduboy will be geeks and want to run EEPROM management software to backup/restore ROMs just to play a few games. I have no idea how to save/restore EEPROM with Arduino software but flashing a new program is trivial.

If a single app can just do what they want with the space I"m not sure there is any need for a standard at all… the whole point of a standard and tracking system was to allow multiple apps to share the space without overwriting each other.


Hi. I was wondering who picks up this topic.

You could consider this:

  1. You should let people to breath. There is no practical way to introduce true sandbox on Arduboy and if EEPROM file system would consume all available eeprom memory, people will collide with it all the time. So either first 256 or 512 bytes of eeprom should be user area, free to play.

  2. The protocols mentioned above are too heavy. I understand that it is tempting to define world but again: let the people breath. The flash memory is costly and no one would be happy to carry dead weight of functinality he really don’t need.

  3. Don’t use names, use hash record to identify that a block is reserved by game. Five [A-Z1-9] can be transformed to 3 or 2 bytes.

  4. Don’t care about fragmentation or deleting files: The simplest (and for given platform is simplest also the best) way is circular buffer. The eeprom starts at the beginning of dedicated eeprom area, filling it up. When there is no more space left, it starts overwrite oldest records on the beginning. The recent games would have their saves safe for while, the older are unlucky but hey, we have less 1KB to play, something must be wiped out.

  5. If user wants to manage eeprom records, he should use special application/sketch dedicated for the job. Avoid functions for advanced eeprom managament, 95% of developers would need single method: openEEPROMFile(id, len)

  6. Don’t make blocks of fixed length: the header should be LEN (1 byte ?), ID_HASH (3 bytes ?), app data (x bytes). Nothing more or less. Let programmers to manage content of their records themselves.

  7. Small area of eeprom reserved for system settings (contrast, display inversion) is great idea.

  8. Define use cases first, then API, and then implementation details.

That’s all I want to say.


No idea what you’re saying. Keeping track of the metadata in the example I mentioned only has a 6% overhead. And yes 8 character names could be compressed into slightly fewer bytes, I was just trying to keep the actual compiled code small.

The protocols mentioned above are too heavy.

The compiled code for what I’m proposing should be very lightweight, it’s not really more than a couple of loops and comparisons. If it proves to not be lightweight then yes, that would be a problem.

The recent games would have their saves safe for while, the older are unlucky

Unacceptable. I might play a game for a long period of time. It shouldn’t expire cause it’s the first game I installed. The user should be in charge of deleting save data - it should not just happen randomly.

If user wants to manage eeprom records, he should use special application/sketch dedicated for the job.

We could have an app for that, and it could even be built into smaller apps automatically and accessed via a setup keystroke.

Don’t make blocks of fixed length:

This is to allow for clearing of data over time without having to defragment the whole area. If keeping data long-term is a requirement (it is IMHO) then a simple system of LEN, ID, DATA has no great way to deal with fragmentation over time. Blocks are easy to manage because you can just free/clear then randomly and blocks can be reserved from anywhere in EEPROM without issue.

Define use cases first, then API, and then implementation details.

That’s what I’m doing. The use case is what I mentioned in my bulleted list above. I want an area multiple apps can easily use, no conflicts, no corruption, and an app should be able to clear it’s own data (or reduce it’s footprint). I don’t think a ton of apps will clear or increase there data, but if those routines aren’t used then they aren’t linked - so they don’t take up any extra space.

I also feel readable names are super important for ever using a tool later to see what a save data is… I want to see “Brakout2” not “BR2”…

Being easily extendable was also a goal because a well behaved app could use more space as it saved multiple games… but it might start with a small allocation. So if you only save one game it only takes 8 bytes… you save 2 more games you get another 16 bytes allocated. You delete all the games you can give the entire allocation back to the system. I had certainly planned on doing this in any apps I wrote that stored game 1, game 2, game 3 style data.

1 Like

I’ve written the code for the system I describe, though it still needs to be cleaned up and tested:

It’s about 1.5kb compiled right now. I’m aware of a few bugs for sure, so it’s not really read for use. I’ll see about getting it releasable soon.

((copied whole from the “Distinguishing Arduboys” thread))


Is there a standard way to share EEPROM? At only 1K, you need to be super efficient. But - there is complex efficient and then there is simple efficient…

Ok, this is still on topic, so…

I could see that there might be value in some items, like a Pseudo Unique Identifier, or your name. But I can also see that you could divide up EEPROM so that value like these are preserved in an available space, but can be set by an Arduboy Utility. You load the Arduboy Utility, and it can set some of these special values, and then gets out of the way when the next app overwrites it.

The simplest model I could imagine for EEPROM management is a Len-Tag model.

  • LEN - byte - number of 4 byte blocks in this block (including this label)
  • TAG - byte*3 - alphanumeric tag identifying block purpose/owner
  • blk*LEN - 4 byte blocks repeated LEN times

Sample Contents of EEPROM

\x05ABY" - 4 blocks (16 bytes) Arduboy data
"Chris…" - 8 bytes of user identifier
"\x11\x22\x33\x44\x55\x66\x77\x88" - 8 bytes of random unique ID
\x02"ALA" - 1 block (4 bytes) Alien Attack highscore
\x00\x00\x03\xE8 - score of 1000
\x02"TKB" - 1 block (4 bytes) Tank Battle highscore
\x00\x00\x00\x23 - score of 35
\xF7"—" - reserved tag, empty space of 246 blocks after this

A three character (Upper/Lower/Numeric) tag gives you nearly a quarter million possible tags. Simple API functions that walk the blocks allow for the reading and filling of blocks. If this is standardized, then that external program associated with loading and overwriting apps can manage the EEPROM space as well. In fact, the external EEPROM manager could easily fill in Name and PUI and you would never have to load a special app for that.

This gets you the special Arduboy features (like a name or unique id) with only a minimal cost to all the other apps. Along the way, it provides a way to hand each app it’s preferred size of EEPROM space, allowing - with some management, granted - persistent storage of things like high scores.

It hints at a couple of other things. A loadable Arduboy app would need the program image, a memory ownership tag value (3 characters), and the size of EEPROM blocks needed, and perhaps default EEPROM contents. This is also a good place to add an official name and an icon, so that that specialized loader app can show a screen full of nice game options for loading.

I’ll stop rambling now, but I close by noting that this memory management seems to fit with the overall Arduboy philosophy – simple, compact solutions, not wide ranging complexity.

1 Like

The one concern I had was that the use of a metadata byte meant that any request for a size over 15 bytes was presented to the app as fragmented. I wasn’t certain about the API role here - if the read/write is mediated by an API, then the space can be presented as whole, even though storage is fragmented.

The question of overhead will depend entirely on app requests. A 16 byte block with a 1 byte metadata will have a permanent 6% overhead, no matter how many apps store stuff (well - except for the app name block). A tag-len intro block has a higher overhead if app use is many and small, but less overhead if app blocks are larger.

I note that this is very reminiscent of the Palm Saved/Unsaved preferences storage. My experience there is that a 4 letter CreatorID was quite sufficient to sort out preference ownership.

I do think that this needs to be in some way optional. I can foresee apps that make such heavy use of the EEPROM that the only sensible thing they can do is leave the reserved 16 bytes, and just take the remaining 1008 bytes for themselves. That’s part of the reason I went with the tag-len structure - an app could request 1000 bytes, and, as long as the space is clear, they will be given 1000 contiguous bytes of storage. Use cases like a Password storage app could easily consume 1000 bytes.

If we want those apps to also make use of an EEPROM storage management solution, then it needs to work for those use cases. And - if we get to having a Windows/Linux Arduboy manager, which can lift and store EEPROM as you swap apps in and out, then you don’t want any apps to give up on the EEPROM manager. Or - you want a one-byte signature that indicates “managed by manager” or “managed by app”.

Finally, any solution likely should think forward to the possibility of larger EEPROM. A future upgrade might give us a lot of device compatibility but with a larger EEPROM - like 4K. A good solution should allow for “save all my Arduboy v1 stuff and start using it on Arduboy v3”.


You have to track the proper hardware offset in both cases… my solution just walks the allocation table so you access the memory as if it were contiguous. I’m not sure if your solution would be smaller (flash wise) but to me a solution that could handle fragmentation was a requirement, so just pretending everything can be stored sequentially was not an option for me.

The question of overhead will depend entirely on app requests. A 16 byte block with a 1 byte metadata will have a permanent 6% overhead, no matter how many apps store stuff (well - except for the app name block). A tag-len intro block has a higher overhead if app use is many and small, but less overhead if app blocks are larger.

Blocks could be bigger, but not smaller (since I only have 6 bits for block ID). If apps tended to use a lot of space the unit could be 32 or 22… I just thought 7 bytes could work for some very small apps and then 22 as the next step sounded pretty good. That plus 1024/16=64… is how I arrived at 16 bytes per block.

My experience there is that a 4 letter CreatorID was quite sufficient to sort out preference ownership.

Well if every app has an assigned bit pattern then sure 32 bits is tons of address space for apps… but I wanted human readable names that did NOT require some save game manager to have a list of all the games. Not to mention some shared spreadsheet to make sure you aren’t stealing someone elses ID. Just because I played “Arduboy Super Tetris” 3 months ago doesn’t mean I want to figure out what the hell “AST” is in my save game history. “SpTetris” is much better to me.

I do think that this needs to be in some way optional. Use cases like a Password storage app could easily consume 1000 bytes.

There isn’t really any way to make it mandatory. You can’t prevent people from using If someone wants to clobber your EEPROM they will. Hopefully we’ll develop some patterns here to help prevent this from happening WITHOUT the user being aware it’s happening. If someone really wants to run an app that needs ALL the space, that’s kind of a specialized use case (from what I’ve seen so far). Most of the apps published so far could get along in a shared EEPROM model. (not saying all will, just that a LOT can)

If we want those apps to also make use of an EEPROM storage management solution, then it needs to work for those use cases.

I think whatever the overhead is, that’s the overhead. If the system that works for 90% of apps only gives you 930 bytes, then that’s what you get. If you can’t live with it you decide to switch to a “all EEPROM” model" and just use all the space.

And - if we get to having a Windows/Linux Arduboy manager

Why are we ignoring the fastest growing platform today… Macs?

Or - you want a one-byte signature that indicates “managed by manager” or “managed by app”.

That could be something that goes in the reserver 16 byte block. Or even a signature for which EEPROM management system you’re using if there turns out to be several competing systems.

Finally, any solution likely should think forward to the possibility of larger EEPROM.

Mine scales easily. 4K would be 64 byte blocks (and lower per block overhead) or you could switch to 2 bytes of metadata and then you can have any block size you waned. (Though I still think 16 is a great size). The algorithm itself scales to almost any small EEPROM pool easily.

1 Like

It might be helpful if a simple virtual storage API was agreed upon then developers could just subclass that to provide their own storage engines. Then if I downloaded a small game I could just tell it to use the storage engine I preferred and recompile it - and it would “just work”. One could also imagine this allowing to write a “flash” storage engine - trading your flash RAM for additional storage (this would likely be a very app specific thing, vs persisted storage across multiple apps).

For the sake of argument, let me propose a system that simply aids in doing just that. A system that needs no library functions and has low overhead, but requires individual user management by slightly modifying the sketch source files. Some automation may be possible, but not assumed or required.

The Kickstarter campaign says that there will be an Arduboy Arcade. Presumably, this will be a central repository of games and other sketches officially sanctioned for use with the Arduboy. I assume that one or more people will be responsible for adding to and maintaining this repository.

I propose that the Arcade maintainer(s) also be responsible for one more thing: Issuing and keeping a list of EEPROM user IDs. The ID would be a 16 bit number similar to a USB Vendor ID. To get an ID, a user would simply ask a maintainer for one.

The handling of EEPROM use by sketches would rely on just 3 standard items, which would appear sequentially, near the top of the sketch’s main sketchname.ino file or optionally in a sketchname.h file :

  1. The offset to the start of the sketch’s EEPROM area. This would be the same in every sketch, except for the actual offset value:
    #define ARDUBOY_EEPROM_START xxxx
    where xxxx is a decimal number. This value would be the only thing changed by the user, to set the sketch’s EEPROM to an area unused by any other sketch or reserved for the system.
  • The length, in bytes, of the EEPROM area used by the sketch:
    #define ARDUBOY_EEPROM_LENGTH xxxx
    where xxxx is a decimal number.
    This value might only serve as a reference for user calculations, but could possibly be used (and maybe even set) by an automatic management tool.

  • The EEPROM area structure:

struct arduboyEEPROMstruct
  uint16_t userID; // As assigned
  uint8_t sketchID; // User defined. Allows more than one sketch or variation per user ID.
  // remainder is user defined

When a sketch started, it would first check that the values for the userID and sketchID stored in EEPROM were correct. If not, this would indicate that the area was allocated to another sketch or had never been used. The program would warn the user then offer to initialise the EEPROM area, or abort, or possibly allow running in a mode that didn’t use EEPROM.

To make it less likely that a false positive match on userID and gameID occurs, they could be slightly obfuscated, such as by xoring with AA55 A5.

The initial value of ARDUBOY_EEPROM_START, as shipped, should place the structure at the high end of EEPROM. The user would then change it to be somewhere in the area they were maintaining, if they decided they wanted the sketch to be in their library.

By having consistent, unchanging names for the defines and structure, it makes it much easier for the user to know how to edit the sketches. They’d still have to do the calculations themselves and keep records however they wished (I’d consider using a spreadsheet). This same consistency would possibly allow a program to scan, calculate and batch edit a “library” of sketches.

Note that the locations of EEPROM areas don’t have to be contiguous. For instance, unless space became tight, I would consider setting the start locations on standard boundaries, with some extra space between used areas. This would allow for extra storage added by later sketch versions.

One other task for the Arduboy Arcade manager(s), would be to make sure a sketch that uses EEPROM follows this system, before accepting it, (or at least make sure that it’s well documented that it doesn’t).

Thinking a bit more on this, I would now make it 5 items.
Just before the arduboyEEPROMstruct structure, add:

    where XXXX is the assigned userID in hex.
    where XX is the sketchID. The format of the value (decimal, hex, octal) is up to the user.

This would allow an easy text search on all the sketch files to make sure these values are unique. This would help in case the central management of EEPROM user IDs was never implemented or stopped being maintained, forcing developers to pick their own user ID. If a conflict was found, it would then be easy to change.

I have no interesting in having to keep a spreadsheet just to play multiple games, so that requirement is already a non-starter to me. It would be a LITTLE better if some type of “rom uploader” kept track of those offsets and tracked them for you, but since we have no such things yet…

In any case I think we need a simple EEPROM API of our own, to allow users to drop in their own implementations. Something where you ask for so many bytes and then get an object with read/write abilities. So the SIMPLEST implementation could use a few defines like you suggest… (add one for #MAX_BYTES) and a more complex implementation could do everything I describe… and someone could link whichever they wanted against all the apps they compiled (if the API was the same).

1 Like

For the record, let me start by saying that I think almost everyone agrees that, as has already been proposed, we need an EEPROM area reserved for system use (e.g. for display brightness, audio mute, etc.). This should be specified and managed separately from any EEPROM area used by apps and so doesn’t have to have anything to do with whatever app specific EEPROM management techniques arise, other than they must not touch it. The handling of system EEPROM library functions is a separate discussion.

Yes, I agree this would be quite beneficial, provided that the number of API functions were minimal, and could be implemented for even a very simple allocation and management method, such as the one I proposed. The API functions you proposed in the Sample code in the initial post would be fine.

I would restrict reads and writes to only single bytes, or perhaps additionally a block of bytes, as you’ve designed on GitHub.

I would also include a verify function that could check the integrity of the data by doing whatever it could to determine if the app’s data had been corrupted by some other app. If the storage method didn’t have a means to verify, it would just return OK. More sophisticated management schemes could use a saved checksum, so maybe we should also have a calculate and save verification data function which could be used where appropriate (for efficiency, you may not want to do this every time some EEPROM data is written).

Finally, an init function could be included, but for simplicity it may be better for the app to just have it’s own initialisation routine(s) based on write().

All apps using this API should include a menu item or other method to allow the user to reset the app’s specific EEPROM data. This is because some libraries using the API may not be able to determine if the data has never been used or has been corrupted, thus requiring manual initialisation. The user may also want to do a reset, to start fresh, even if the data isn’t new or corrupted.

I’d still like to see some kind of central authority/registry to handle maintaining unique ID’s. Using an 8 (or other length) character ID has advantages but takes up more space than is required for a binary ID and is still open to problems if there’s no way to prevent coincidental duplication. Whatever ID management scheme is used, it could issue as may different types of IDs as is required by all the different EEPROM management libraries that conform to the API.

Anyway, the sooner such an API can be defined and agreed upon, the better. That way developers can start implementing and testing libraries that conform to the API, so they can “fight it out” for the winner(s) :smile:

I propose just a constructor and read(relative_location), and write(location, byte)… of course the read/write buffer is also nice (and easy for most to implement). Then the SIMPLEST full EEPROM API would be (dropping down into Ruby pseudo-code):

def initialize()
  @offset = RESERVED

def write(location, data)
  FULL_ROM[location + @offest] = data

def read(location)
  FULL_ROM[location + @offset]

You’d address the EEPROM from 0 to maxsize (which is 1024 - RESERVED). Every lib would just need to agree on the reserved size. This doesn’t do anything to help you switch managers mid-stream… it’s more for people to decide they want to use a certain manager and then compile all their sketches against that.

A simple per-app implementation could be the same except:

def initialize(app_name, app_offset)
  @offset = RESERVED + app_offset
  if get_string(ROM[@offset], app_name.length) != app_name
  # perform init code here, zap ram, write header, etc.

So I download an app, I just have to swap out the name of the EEPROM library and maybe it’s constructor and then I’m good to go.

So if we can just agree on read/write and read/write (buffer)… read/write buffer default implementations can just call the base single-byte read/write, which should work out of the box for many implimentations.

Things to agree on:

  1. uint8_t read(uint16_t address); //< read a single byte
  2. void write(uint16_t address, uint8_t data); //< write a single byte
  3. void read(uint16_t address, uint8_t *buffer, size_t size);
  4. void write(uint16_t address, uint8_t *buffer, size_t size);
  5. reserved size (I suggested 16 bytes)
  6. reserved location (should be same for all libs, top or bottom of EEPROM)