Shared EEPROM storage management across multiple apps

Development
#76

Although my first reaction was, I confess, negative, I can see some benefit in this approach as it works well for those apps that want to use it, and imposes no penalty at all for apps that want to ignore it.

It still, however, leaves open the option of a separate “EEPROM manager” sketch which could be used to enable EEPROM load/save in apps that do nothing to support it. In an online library of apps, possibly as hex files, the only thing needed is a flag on the description that says whether or not an app includes EEPROM use, and then if it has EEPROM self-management.

Here is a tutorial, not for the Leonardo but should be close, on reading EEPROM and dumping to the serial output (which if memory serves on the Arduboy/Leonardo defaults to USB Serial). Comparable code can read from serial and place in EEPROM.

The nice thing about this option is that the serial usb will always be present because that is how the Arduino IDE updates the device. There is no additional penalty for using USB serial in your sketch beyond the (usually small) amount of code you write yourself.

At this point, my one standardization suggestion would be a char[4] item in the reserved area that can be set to identify the current owner of EEPROM. A dump option will just create a 1K dump via serial, which can be captured as a 1K file. External tools (more power, more space) can then take on the job of “EEPROM library management”, using the “EEPROMowner” tag as a base for deciding how to handle things.

1 Like
#77

Actually, as I believe @ChrisS has alluded to above , you can do both. Any functions included in the library only increase the size of a sketch if they are used. Just having them available but not used doesn’t incur any penalty.

So, sketches that can afford the space could use the library functions for a self contained save/restore. For sketches that didn’t include the library functions, a separate sketch could be used (and that sketch could use the library routines, for ease and consistency).

As usual, for this kind of thing to fly there has to be enough approval, but more so someone actually has to do the work, for both the Arduboy and PC side of things.

2 Likes
#78

I’m aware of the fact unused functions aren’t compiled but I can appreciate this fact probably confuses less experienced programmers.

I didn’t suggest both because I took your question to be an ‘either/or’ type question and having both is a bit redundant, but I suppose it could be argued that (in a case like this) redundancy probably provides robustness.

This is one of the reasons I was leaning more towards the dedicated sketch approach. Aside from being able to make it a de facto standard even without support from the core Arduboy library, giving the user the ability to copy the EEPROM via serial would mean they could store the save data of games even without the functionality being built in to the game. That said, being able to do so from the game means that programmers could provide an explicit menu option to perform a save backup, which would be easier for users.

#79

Out of interest, why was your first reaction negative?
(Not that I can’t see anything bad about the idea, but I like hearing the reasoning for other people’s opinions)

By flag I assume you mean a tag on the program’s description rather than a bit-flag on a part of the hex file’s memory.
I’m partially against keeping public things in hex file format since Arduboy tends to be very pro open-source, but that’s a problem for a different thread. A repositry of arduino games would be a good idea, but again, that would need a lot of discussing and organising.

I think you forgot to provide a link.

Precisely. This approach is simply combining two core arduino features present in almost every arduino board to make a new feature, meaning this approach should be applicable to both arduboy and other arduino-based products.

I’m partly against this idea.
As far as I’m concerned, the software handling the EEPROM dump doesn’t need to know anything about the program currently loaded on the arduboy other than the fact it supports the serial protocol used for the dump process. It doesn’t need to know what the game is called or how much of the EEPROM is actually in use.
All it has to do is read the full 1K block of EEPROM, ask the user what they want to call the dump file and then dump the data to the file. The same process can be done in reverse for loading the arduboy’s EEPROM. As an addition it might be good to provide an option to clear the EEPROM, but otherwise that’s all the program would have to do.
The only relatively complex bit would be deciding on how exactly the transmission protocol would work.

The beauty about this solution is that it can be implemented in so many different ways. It could be a UI or command line and it could run on any OS, perhaps even on phones and tablets with a compatible micro USB port (i.e. the USB port is not one of those ones that’s been limited to only drawing power).

#80

Like I said, add the back-end save/restore functions to the library, so it’s a standardised API, and have the dedicated sketch use them. The dedicated sketch would just add the front-end user interface. Any game developer could then use the same library functions to include the capability within the game itself, if desired.

The functions don’t necessarily have to go in the Arduboy class itself. They could be in a separate class with its own .cpp and .h file, if that makes more sense. These new files could still be a part of the Arduboy library package.

As you’ve said, if the API is designed properly, changing things to save to an SD card would then be transparent to the sketches.

1 Like
#81

If you’d like, please join the discussion here for more on that subject: [OLD] Arduboy Manager

#82

I think that makes a lot more sense.
I’ve often though that different areas of concern should be abstracted into different parts rather than being a monolithic class like the Arduboy class currently is. Arduino itself does this reasonably well by keeping stuff like Serial, SPI and EEPROM separate.

Keeping it separate would:

  • Reduce clutter in the Arduboy class.
  • Keeep it hidden from people who were not intentionally trying to use it. (Out of sight, out of mind)
  • Emphasise the fact it’s an optional feature.
  • Emphasise that it not being present means that it’s not increasing the size of the final sketch.

I can appreciate that other people might have different opinions though, I’d be happy to hear the case from someone who thinks that it would make more sense to put the functions in the Arduboy class.

#83

Actually, some beginners are likely to use
Sketch -> Include Library -> Arduboy
Which will add a #include for each .h file in the library. Having a separate .h file will add more clutter in this case.

To prevent this, you could have a separate class but put it in the Arduboy.h and Arduboy.cpp files.

#84

It’s not the software handler I’m thinking about - it’s the program in the Arduboy.

Having a standardized place that is either blank (EEPROM is ready to use) or has your magic value (the Arduboy app can just go ahead) makes Arduboy code simpler, smaller, and cleaner. If you start loading and saving EEPROM images, the risk is that you will load the wrong thing, and without a simple and clear check, your Arduboy app is going to spend too many resources assessing whether or not the EEPROM contents are actually it’s own.

That said - I would use such a value. And to keep life simple, the software would likely write the file with a name based on a constant start value, the owner tag, and the date and time. That makes it even simpler for the user - run the utility and select “send EEPROM” on the Arduboy, and everything else is done for you, with no chance of file collisions or overwrites.

I had to go look up the reserved area - it is 16 bytes, so perhaps the tag could be 8 allowing for readability and almost not chance of collision.

Four sample files.

ArduEPROM-BlobAttk-20160722T182233.txt
ArduEPROM-BreakOut-20160724T235845.txt
ArduEPROM-________-20160725T201916.txt
ArduEPROM-Life____-20160726T235433.txt

That third one is what you get for an app that does nothing with the ownertag.

The internal format I would make 64 lines of delimited hex dump, 16 bytes per line.
:00112233445566778899aabbccddeeff:

The files are still going to be only 2K - less than a cluster in the file system.

If you have an app that does nothing, load the “EEPROM control sketch”, have it set the ownertag value, and then load the dont-care app. To save that EEPROM, you would have to reload the “EEPROM control sketch”, but all the save files will appear under the ownertag that you set. So you even get customized file saves done easily for apps that existed before we wrote this.

Making them text friendly is deliberate. If you have a save file with a null ownertag, you can modify it with notepad.

I’m not ignoring the other - good - questions. I’m just out of time late at night in this time zone, and will have to look back another time.

#85

If that’s the case then why aren’t the contents of core.h also just merged with Arduboy.h instead of being separate and #included?

If someone isn’t able to type #include <Arduboy.h> without needing a 3 step menu I’d be concerned about whether or not programming is a suitable hobby for them, let alone whether they actually know enough C++ to be able to write something functional.

#86

Personally my approach would just be to assume the contents are correct and if the game crashes, blame the player for using an invalid save file. Ultimately whatever approach you take won’t guarantee the file is valid. You can increase the probality of a file being valid, but you can never truly guarantee it.

On a side note, I’ve suddenly realised introducing an EEPROM transfer system will mean save hacking will suddenly become viable, even for games that aren’t open source. Cue ArduGenie and ArduShark becoming a thing.

As long as the ability to override that is provided, otherwise you’d be shooting the experienced programmers in the foot for the sake of the less experienced. (Windows’ calculator program comes with separate basic, scientific and programmer modes for a reason).

I partly disagree with this since storing binary data as text would be wasteful without adding much purpose in comparison to directly dumping the data. Also, to anyone with a hex editor, this would be redundant anyway.

Or at the very least it should be 4 or 8 bytes. 16 bytes starts to get into the ‘blurring together’ territory. I mean, when was the last time you looked at a 32-digit number without any delimiters (i.e. commas) and were able to immediately pronounce it?

Again, the people with hex editors would currently be screwing their faces up at you, unsure whether to laugh or be confused.

#87

In the current V1.1 library, core.h is in subdirectory core. Subdirectories aren’t included when you use Include Library. The problem with this is that you can’t include .h files, that are in a library subdirectory, in a sketch or another library. We want the ArduboyCore class to be available as a base class, so in the current development it’s been moved back to the main directory, thus causing the include problem.

Anyway, I was just pointing out a minor reason for not using a separate .h file. I actually agree with you and would prefer using separate .h and .cpp files. I suspect most beginners won’t use Include Library and will instead use another game or template as a starting point. For the few who do use Include Library, leaving the extra #includes in isn’t going to cause any problems, and they can always be removed if the developer realises that they aren’t needed.

Some beginners may start by following Arduino tutorials, which instruct on using the menu. But again, I agree that we shouldn’t be concerned about it.

#88

The Include Library features are gradually sounding worse by the minute.

I would hope most C++ programmers would agree with that, it’s much more idiomatic to take that approach (even if not always the most practical).

That would be a far more sensible approach for them to take. In fact, that should be the official advice next time a beginner turns up asking “where do I start?”.

Such is the beauty of the compiler not compiling unused functions.

If I ever see one of those tutorials, I’ll make sure to leave the author an angry letter. Programming involves a great deal of typing, and beginners need all the practice they can get, even if it’s before they understand what they’re typing.

#89

@Pharap, I just discovered that the Arduino people have implemented a solution for the problem of Include Libraries including all .h files. The library.properties file can now contain an includes property, which specifies which files are included when using Include Libraries. This change is available IDE V1.6.10, which is now released. :smiley:

1 Like
#90

All true, but not enough reason I think to simply give up on a small attempt to avoid some of the problems.

Unlike Windows’ calculator, this is going to be documented and the code available. So the ultimate override is just grab the code, and change how the filename is built. Code for the support side utility can be command line mode, making it easy to understand, even for only moderately experienced programmers.

If we were storing a file of hundreds of kilobytes, I might agree. But the data involved here, whether binary or hex, is less than the size of a cluster on a modern hard drive. Note that I do not expect the Arduboy to do this formatting - this is done after receiving from the Arduboy and before writing to disk.

This is never really intended to “look at” numbers. Multibyte values are little-endian, word alignment is uncertain - you’re never really going to know where one value ends and another begins without knowing the program that created the EEPROM. It’s more like the .hex format files - which is, in fact, a possible option, if slightly more detailed than this.

The last time I looked at a 32-digit number was when looking at an Arduboy complied .hex file.

Not all platforms have readily available hex editors. If you are swapping hex file games using your phone (someone is trying that) then you may find a decent hex editor is not as a readily available. But a “text editor” - (probably what I should have said, not notepad) - that is going to be widely available.

#91

Maybe they heard our disgruntled grumblings. (That or it’s a freak coincidence, but I’m leaning towards the conspiricy theory.)

Now I have to redownload Arduino Studio… again :expressionless:

#92

I’d be more inclined to agree if the Arduboy had more memory. 8bytes out of 1024 is a reasonably large chunk of memory when you’re used to working in the order of gigabytes and megabytes.

Windows’ calculator doesn’t really need documentation, all the options are nicely labeled and reasonably obvious.

Then why store as text and not binary? Either you want it as text so people can go in and change it or you don’t actually need it to be text at all.

#93

It was an issue raised by someone who’s library has many .h files to support multiple boards and configurations. It’s a freak coincidence that we happened to be discussing it near the release. :wink:

#94

That’s spooky.
Oh well, all the more reason for the separation idea to go ahead.

2 Likes
#95

Note that I’m suggesting that the 8 bytes be part of the 16 reserved bytes. So it’s only using memory you already don’t have.

Read back through the notes in this thread and you will find that using 8 bytes only is likely the smallest suggested management space of any proposal. Some were using as much as 1 byte out every 16.

It’s never going to be that black and white. It’s relatively easy to build the tool to handle text files, and that makes the files accessible.

Storing them this way is similar to the same reason that .hex files - which will be loaded ultimately as binary - are stored as long strings of hex characters. You are not likely to need to modify it, or even look at it. But if it does need inspection or modification, you don’t need anything other than knowledge and a text editor.

I did find hex editors for the Android platform, but they all shared a common hex editor limitation - they can’t insert or delete, just modify or extend. Using a text editor on a hex stream removes that limitation.

This also makes it roughly compatible at a concept level with .hex files for storing (binary) programs, also for the Arduboy.