Shared EEPROM storage management across multiple apps

[ChrisS]

I don't think it's too much to ask for it to have at least some form of standardization. If we don't do this now, I can see beginner developers and devs who don't use the forums often enough writing a great program that saves over a lot of other data for other games without warning.

Standards for block usage? Yes. Standards for identity? Not so much.

A program only cares about one thing for EEPROM - can I write here? Yes or no. If the block is used, then no, otherwise yes. Programs will be cramped for space as it is, so I really cannot see developers wanting to spend anything more than the most minimal of resources on dealing with someone elses data.

As it is, I have been actively developing for about a week - and I have already run into resource limitations imposed by the API. I know there are more coming. At least one item I am working on will likely warn you and then clobber all of EEPROM - if you don't give it the EEPROM, it just won't run. I expect I will start creating my own variation of the standard library for my use within a couple of weeks. I hate the idea of doing that, but the ultimate standardization is the hardware - as long as things run on the hardware, that's going to be what counts.

If you want developers to adhere to standards, then you have to ensure that the standards are flexible enough to not impose an undue penalty on getting as much out of the platform as possible.

Incidentally, I note that the default loaded game by TEAM a.r.g. does not use the default library. It is quite possible that they decided that things like the boot up logo took too many resources that could be better deployed in the game, rather than used once for one second.

[MLXXXp]

Incidentally, I note that the default loaded game by TEAM a.r.g. does not use the default library. It is quite possible that they decided that things like the boot up logo took too many resources that could be better deployed in the game, rather than used once for one second.

It's more the case that the library wasn't standard at the time Team A.R.G. started developing for Arduboy. Back then, even what few other examples existed just included a copy for the the library within the sketch itself. By the time Arduboy was actually a stand alone library, Team A.R.G. had customised theirs enough that they just continued to stick with it.

[MLXXXp]

As it is, I have been actively developing for about a week - and I have already run into resource limitations imposed by the API.

Using Arduboy2 may help.

Arduboy2 - An alternative to the Arduboy Library

Since "the powers that be" can't seem to decide how to continue with development of the Arduboy library past V1.1, I decided to fork the Arduboy V1.2 development code myself and turn into an alternative library, which I call Arduboy2 (for lack of a better name). Yes, I know Arduboy2 could be interpreted as being meant for some new "next generation" of the Arduboy system, but what can you do? :confused: See the README.md file in the repository for all the details, but the main goals and feature…

[crait]

Standards for block usage? Yes. Standards for identity? Not so much.

I am merely suggesting something like a leading byte identifier that is somewhat unique to each game that needs to save data.

[fuopy]

I believe I have a good solution to the problem. I've been planning it for a while but haven't gotten around to implementing it, but I might as well say how it will work while the topic is hot!

EEPROM management should be the responsibility of the hex uploader. When games are published to hex files, it will be necessary (and very convenient) to have a bit of metadata associated with them. Obvious fields are the game name and author, but one of the more critical fields will be EEPROM usage. The metadata will specify the ranges of EEPROM bytes the uploaded game needs to use.

Here's how the process works: You upload your first-ever packaged game. This game includes the EEPROM metadata. The hex uploader creates an "EEPROM file" on your computer's disk for your Arduboy, recording the fact that you uploaded a game using specific ranges of EEPROM. Those ranges of EEPROM will be labeled "pending" in the EEPROM file. The ranges are also associated with the game ID. Finally, the code uploads and you play the game!

Ok, that was fun! Now you want to play another game. You download another packaged file and plug it into your hex uploader. One of two things can happen at this point:

1) There is an EEPROM usage conflict. As you clicked the upload button, the hex uploader checked if the new game's requested EEPROM ranges overlapped with the "pending" ranges associated with each game that had ranges pending on that Arduboy's EEPROM file. It found one, so now it's quickly uploading a sketch that contains code to back up that section of EEPROM to the computer. After that, the "pending" ranges of EEPROM associated with the older game in the Arduboy profile are cleared. New ranges for this second game are now created, associated with the game, and labeled "pending." Finally, the game code is uploaded to the Arduboy.

2) There is no EEPROM usage conflict. This situation can be handled in many ways, but I would suggest providing a user option for "always backup" (to prevent bad-behaving sketches from wreaking havoc on your save data" or "lazy backup" (so only conflict situations will back up EEPROM to the computer, and the whole uploading process will be faster.)

One of the problems remaining is with the hex uploader identifying different Arduboys, and associating them with their EEPROM file. I propose that we use a few of those reserved bytes at the beginning to store the name of an Arduboy's owner. It would be helpful for serial tools, and could bring up cute situations in developed games! (I really want to be able to say "hi, $OWNERS_NAME!" in games! )

I forgot to say what happens when playing games that have backed up EEPROM! Ok, you just clicked the upload button. Well, if the memory for the game is still labeled "pending," then nothing needs to happen, because the memory hasn't been overwritten. And if the memory has been overwritten, (AKA there is no pending entry for that game's ranges) then it will see if there is a backup of those ranges, upload a sketch with code to restore the EEPROM for those ranges, and finally upload the game.

Anyway, I hope I have shown that this is a good solution. Externally-managed EEPROM will protect you from losing your game data, as well as simplify EEPROM programming for everybody. Not only that, but it can be expanded on: save backups and save sharing will become trivial. And if an SD card version of the Arduboy comes out, it could implement a scheme like this too, allowing as many save files as you want for as many games as you want-- all stored on the SD card!

[fuopy]

As an addition to my previous post:

To avoid uploading a sketch just to write or read the EEPROM, it might be a good idea to have tiny EEPRON dump and restore functions that use the serial port as part of the Arduboy2 library. Most games so far have more than enough extra space for tiny bits of code like these.

If there was a bit of metadata that said whether a game included these routines, then an extra upload wouldn't be necessary. If you try to upload a game that doesn't have the backup and restore routines, but the current game does, you don't need to upload the maintenance sketch; just back up or restore EEPROM with the current sketch, then upload the next game.

Likewise, if you are switching to one that does have the routines from one that doesn't, you can just wait until the next one is uploaded before handling the save management. (Although that could be risky, because it would then be up to the programmer to wait for the OK before formatting the save slot. Even if there was a formal standard such as "one must check for the serial port prior to initializing the save," you are truly at the mercy of the programmer in that situation.)

Regardless, the only time it's truly necessary to upload the maintenance sketch is when playing two games in a row that use up so much memory that there isn't any to spare for dump and restore functions.

And another point I'd like to bring up: This thread is filled with lots of cool ideas of how to manage EEPROM on the Arduboy itself, yet none of them are really practical. The Arduboy has barely any EEPROM, and after following this discussion it has become apparent that trying to organize eeprom on the Arduboy 1 is convoluted and impractical. There's really no need to use chains and blocks and other data structures when you can only run one app at a time, right? (and if you want to piggy back save files, just reserve them at different locations! Yes, of course there's an Easter egg in my new game if you full cleared glove! )

[Dreamer3]

The Arduboy has barely any EEPROM, and after following this discussion it has become apparent that trying to organize eeprom on the Arduboy 1 is convoluted and impractical.

No, it's merely apparent that no one agrees. Something like 8 blocks of 128 bytes or 4 blocks of 256 bytes is practical and easy to understand (and has worked for other platforms) , just no one wants to agree. shrugs

[fuopy]

I can't say this for certain (because in this instance I can't speak for anyone but myself) but I think that the reason nobody can agree is because everyone's needs, or imagined needs, are different. The reason I said it is apparent is because it reminds me of my first experience I had in drafting a design document for a game with others. We were working on a survival game that takes place in a space ship. You hear an explosion. Immediately your communications are shut down, so you are unaware that you are being boarded as you try to get around the ship and make repairs. As we were talking about the game, we agreed on high level mechanics. We all had a general idea of the game we wanted to make, yet none of us realized at the time we were talking about completely different things! I was talking about making a Metroid style game, another person was talking as if it was a top-down shooter. To me, this discussion parallels that. Here, we are talking about highly optimized implementations, yet the applications are going to be different for everybody!

I can't even pick a single proposed format that works for all the projects that I'm working on! For example, some games will work fine with using small blocks because they store a simple high score table or auto save (like glove and Arduboy gamepad). Other programs like my on-Arduboy game editor will need almost all of the memory for its images, "constraint classes," and levels. My other program, the Arduboy virtual machine for learning about assembly language, needs only 256 byte blocks for each program. Then I have my prototype Metroid Fusion/Amnesia game, which "installs" the levels to EEPROM and modifies them as you play.

So basically, even if an on-Arduboy management system of EEPROM was picked, it would not be able to satisfy the applications of all developers for the unit. A way to make EEPROM management universally supported would be to handle it on the computer's end. Developers could simply specify what ranges they needed. They would not have to worry about overwriting other applications' memory, and they would be able to reserve all the space they need, without worrying about an Arduboy with full memory.

Here's another argument against on-Arduboy management: compiled code size.

What happens when there's not enough free space? First, the game needs to have code that checks for free space, however the EEPROM standard is organized. Well, OK, there isn't enough space. Will the Arduboy just delete the oldest file? Hopefully not, I don't want my RPG (as an example of a highly valued save file) progress to be deleted and replaced by a win/loss record from just testing out a little tic-tac-toe game (as an example of a not-so-valued save file) just once. So to avoid this, probably some save manager screen needs to appear. That's more code compiled in. Also a print library of sorts needs to be bundled. And applications need readable names, so that's more EEPROM used.

The overhead in programming in EEPROM management in the device is much higher than simple dump and restore routines through the serial port.

Something like 8 blocks of 128 bytes or 4 blocks of 256 bytes is practical and easy to understand (and has worked for other platforms)

Out of curiosity, can you tell me about the mentioned (Harvard architecture?) game systems that share small amounts of data in a non-removable memory location?

[Dreamer3]

Here's another argument against on-Arduboy management: compiled code size.

What happens when there's not enough free space? First, the game needs to have code that checks for free space, however the EEPROM standard is organized. Well, OK, there isn't enough space. Will the Arduboy just delete the oldest file? Hopefully not, I don't want my RPG (as an example of a highly valued save file) progress to be deleted and replaced by a win/loss record from just testing out a little tic-tac-toe game (as an example of a not-so-valued save file) just once. So to avoid this, probably some save manager screen needs to appear. That's more code compiled in. Also a print library of sorts needs to be bundled. And applications need readable names, so that's more EEPROM used.

Well, sure any management takes more space than NO management. But to simply show a screen with "empty" "free" slots out of 4 or 8 doesn't require much... you could do that with sprites... and if you WANT that type of save game management (with multiple save points) you HAVE to build that UI in any case. If you don't then you could just find the first free area and claim it - and use it - or use your previously claimed area.

Most games needing less than 128 bytes in my "8 slot" example could "just work" with this [claim it and use it] type of approach without needing any UI intervention at all. You just run into an issue when you have 0 slots left. That would have to be some type of "no can save" scenario (which might require some UI) and you'd have to have a nifty little EEPROM management program you could flash at that point to show you who was using what and let you move/delete slots.

The code for something like a that would be pretty small... heck IIRC my entire dynamic allocation table approach with variable sized blocks was only 1kb of program RAM. x sized fixed blocks should be a lot simpler.

But without any real "authority" at the top I just don't think this is going to get done. I like my latest idea but I'm not about to write it just to have no one use it.

[Dreamer3]

they store a simple high score table

That isn't always so simple... something like just 10 scores with a 4 byte score and a 3 byte name is already 70 bytes... so no problem for 120 bytes... a little much for 64 bytes.

Of course if you only store 3 high scores that's much better... but just saying storing a bunch of high scores isn't necessarily "low space". Depends on the size of the scores and how many.

[Tachikoma]

How about just recommending games have a "delete data" option? I don't really care about the high scores for the Mario clone for example, so once I have played it for a bit, I would rather hit a menu option to delete it (or even have an option to not save in the first place) before I change games.

[Dreamer3]

Maybe games shouldn't really save data without prompting first. Oh so many considerations.

[Pharap]

Why not just let the arduboy dump the entire contents of its EEPROM over serial and have a corresponding program on the computer that writes what the Arduboy is sending to file. Then reverse the process to load the files. If you abstract it in the right way that could even be reused in the case that the Arduboy ever acquires an SD port.

[MLXXXp]

Why not just let the arduboy dump the entire contents of its EEPROM over serial

So are you proposing that the save and restore code be included in each sketch, or would you have to load and run a separate save/restore sketch between each game?

[Pharap]

Depends entirely on how small said code would be. If it's small enough it could be encorporated into the arduboy library, if not it could be a separate script. The former would be more handy and give the programmer control (i.e. the programmer would only have to use said script if they wanted to overwrite the EEPROM) but the latter would give the user more control, which would make sense if only a few games actually made use of the EEPROM.

[ChrisS]

Although my first reaction was, I confess, negative, I can see some benefit in this approach as it works well for those apps that want to use it, and imposes no penalty at all for apps that want to ignore it.

It still, however, leaves open the option of a separate "EEPROM manager" sketch which could be used to enable EEPROM load/save in apps that do nothing to support it. In an online library of apps, possibly as hex files, the only thing needed is a flag on the description that says whether or not an app includes EEPROM use, and then if it has EEPROM self-management.

Here is a tutorial, not for the Leonardo but should be close, on reading EEPROM and dumping to the serial output (which if memory serves on the Arduboy/Leonardo defaults to USB Serial). Comparable code can read from serial and place in EEPROM.

The nice thing about this option is that the serial usb will always be present because that is how the Arduino IDE updates the device. There is no additional penalty for using USB serial in your sketch beyond the (usually small) amount of code you write yourself.

At this point, my one standardization suggestion would be a char[4] item in the reserved area that can be set to identify the current owner of EEPROM. A dump option will just create a 1K dump via serial, which can be captured as a 1K file. External tools (more power, more space) can then take on the job of "EEPROM library management", using the "EEPROMowner" tag as a base for deciding how to handle things.

[MLXXXp]

If it's small enough it could be encorporated into the arduboy library, if not it could be a separate script.

Actually, as I believe @ChrisS has alluded to above , you can do both. Any functions included in the library only increase the size of a sketch if they are used. Just having them available but not used doesn't incur any penalty.

So, sketches that can afford the space could use the library functions for a self contained save/restore. For sketches that didn't include the library functions, a separate sketch could be used (and that sketch could use the library routines, for ease and consistency).

As usual, for this kind of thing to fly there has to be enough approval, but more so someone actually has to do the work, for both the Arduboy and PC side of things.

[Pharap]

Just having them available but not used doesn't incur any penalty.

I'm aware of the fact unused functions aren't compiled but I can appreciate this fact probably confuses less experienced programmers.

I didn't suggest both because I took your question to be an 'either/or' type question and having both is a bit redundant, but I suppose it could be argued that (in a case like this) redundancy probably provides robustness.

for this kind of thing to fly there has to be enough approval

This is one of the reasons I was leaning more towards the dedicated sketch approach. Aside from being able to make it a de facto standard even without support from the core Arduboy library, giving the user the ability to copy the EEPROM via serial would mean they could store the save data of games even without the functionality being built in to the game. That said, being able to do so from the game means that programmers could provide an explicit menu option to perform a save backup, which would be easier for users.

[Pharap]

Although my first reaction was, I confess, negative

Out of interest, why was your first reaction negative?
(Not that I can't see anything bad about the idea, but I like hearing the reasoning for other people's opinions)

In an online library of apps, possibly as hex files, the only thing needed is a flag on the description that says whether or not an app includes EEPROM use, and then if it has EEPROM self-management.

By flag I assume you mean a tag on the program's description rather than a bit-flag on a part of the hex file's memory.
I'm partially against keeping public things in hex file format since Arduboy tends to be very pro open-source, but that's a problem for a different thread. A repositry of arduino games would be a good idea, but again, that would need a lot of discussing and organising.

Here is a tutorial

I think you forgot to provide a link.

The nice thing about this option is that the serial usb will always be present because that is how the Arduino IDE updates the device.

Precisely. This approach is simply combining two core arduino features present in almost every arduino board to make a new feature, meaning this approach should be applicable to both arduboy and other arduino-based products.

At this point, my one standardization suggestion would be a char[4] item in the reserved area that can be set to identify the current owner of EEPROM

I'm partly against this idea.
As far as I'm concerned, the software handling the EEPROM dump doesn't need to know anything about the program currently loaded on the arduboy other than the fact it supports the serial protocol used for the dump process. It doesn't need to know what the game is called or how much of the EEPROM is actually in use.
All it has to do is read the full 1K block of EEPROM, ask the user what they want to call the dump file and then dump the data to the file. The same process can be done in reverse for loading the arduboy's EEPROM. As an addition it might be good to provide an option to clear the EEPROM, but otherwise that's all the program would have to do.
The only relatively complex bit would be deciding on how exactly the transmission protocol would work.

The beauty about this solution is that it can be implemented in so many different ways. It could be a UI or command line and it could run on any OS, perhaps even on phones and tablets with a compatible micro USB port (i.e. the USB port is not one of those ones that's been limited to only drawing power).

[MLXXXp]

That said, being able to do so from the game means that programmers could provide an explicit menu option to perform a save backup, which would be easier for users.

Like I said, add the back-end save/restore functions to the library, so it's a standardised API, and have the dedicated sketch use them. The dedicated sketch would just add the front-end user interface. Any game developer could then use the same library functions to include the capability within the game itself, if desired.

The functions don't necessarily have to go in the Arduboy class itself. They could be in a separate class with its own .cpp and .h file, if that makes more sense. These new files could still be a part of the Arduboy library package.

As you've said, if the API is designed properly, changing things to save to an SD card would then be transparent to the sketches.