Shared EEPROM storage management across multiple apps


(Holmes) #81

If you’d like, please join the discussion here for more on that subject: [OLD] Arduboy Manager


(Pharap) #82

I think that makes a lot more sense.
I’ve often though that different areas of concern should be abstracted into different parts rather than being a monolithic class like the Arduboy class currently is. Arduino itself does this reasonably well by keeping stuff like Serial, SPI and EEPROM separate.

Keeping it separate would:

  • Reduce clutter in the Arduboy class.
  • Keeep it hidden from people who were not intentionally trying to use it. (Out of sight, out of mind)
  • Emphasise the fact it’s an optional feature.
  • Emphasise that it not being present means that it’s not increasing the size of the final sketch.

I can appreciate that other people might have different opinions though, I’d be happy to hear the case from someone who thinks that it would make more sense to put the functions in the Arduboy class.


(Scott) #83

Actually, some beginners are likely to use
Sketch -> Include Library -> Arduboy
Which will add a #include for each .h file in the library. Having a separate .h file will add more clutter in this case.

To prevent this, you could have a separate class but put it in the Arduboy.h and Arduboy.cpp files.


(Chris Smith) #84

It’s not the software handler I’m thinking about - it’s the program in the Arduboy.

Having a standardized place that is either blank (EEPROM is ready to use) or has your magic value (the Arduboy app can just go ahead) makes Arduboy code simpler, smaller, and cleaner. If you start loading and saving EEPROM images, the risk is that you will load the wrong thing, and without a simple and clear check, your Arduboy app is going to spend too many resources assessing whether or not the EEPROM contents are actually it’s own.

That said - I would use such a value. And to keep life simple, the software would likely write the file with a name based on a constant start value, the owner tag, and the date and time. That makes it even simpler for the user - run the utility and select “send EEPROM” on the Arduboy, and everything else is done for you, with no chance of file collisions or overwrites.

I had to go look up the reserved area - it is 16 bytes, so perhaps the tag could be 8 allowing for readability and almost not chance of collision.

Four sample files.

ArduEPROM-BlobAttk-20160722T182233.txt
ArduEPROM-BreakOut-20160724T235845.txt
ArduEPROM-________-20160725T201916.txt
ArduEPROM-Life____-20160726T235433.txt

That third one is what you get for an app that does nothing with the ownertag.

The internal format I would make 64 lines of delimited hex dump, 16 bytes per line.
:00112233445566778899aabbccddeeff:

The files are still going to be only 2K - less than a cluster in the file system.

If you have an app that does nothing, load the “EEPROM control sketch”, have it set the ownertag value, and then load the dont-care app. To save that EEPROM, you would have to reload the “EEPROM control sketch”, but all the save files will appear under the ownertag that you set. So you even get customized file saves done easily for apps that existed before we wrote this.

Making them text friendly is deliberate. If you have a save file with a null ownertag, you can modify it with notepad.

I’m not ignoring the other - good - questions. I’m just out of time late at night in this time zone, and will have to look back another time.


(Pharap) #85

If that’s the case then why aren’t the contents of core.h also just merged with Arduboy.h instead of being separate and #included?

If someone isn’t able to type #include <Arduboy.h> without needing a 3 step menu I’d be concerned about whether or not programming is a suitable hobby for them, let alone whether they actually know enough C++ to be able to write something functional.


(Pharap) #86

Personally my approach would just be to assume the contents are correct and if the game crashes, blame the player for using an invalid save file. Ultimately whatever approach you take won’t guarantee the file is valid. You can increase the probality of a file being valid, but you can never truly guarantee it.

On a side note, I’ve suddenly realised introducing an EEPROM transfer system will mean save hacking will suddenly become viable, even for games that aren’t open source. Cue ArduGenie and ArduShark becoming a thing.

As long as the ability to override that is provided, otherwise you’d be shooting the experienced programmers in the foot for the sake of the less experienced. (Windows’ calculator program comes with separate basic, scientific and programmer modes for a reason).

I partly disagree with this since storing binary data as text would be wasteful without adding much purpose in comparison to directly dumping the data. Also, to anyone with a hex editor, this would be redundant anyway.

Or at the very least it should be 4 or 8 bytes. 16 bytes starts to get into the ‘blurring together’ territory. I mean, when was the last time you looked at a 32-digit number without any delimiters (i.e. commas) and were able to immediately pronounce it?

Again, the people with hex editors would currently be screwing their faces up at you, unsure whether to laugh or be confused.


(Scott) #87

In the current V1.1 library, core.h is in subdirectory core. Subdirectories aren’t included when you use Include Library. The problem with this is that you can’t include .h files, that are in a library subdirectory, in a sketch or another library. We want the ArduboyCore class to be available as a base class, so in the current development it’s been moved back to the main directory, thus causing the include problem.

Anyway, I was just pointing out a minor reason for not using a separate .h file. I actually agree with you and would prefer using separate .h and .cpp files. I suspect most beginners won’t use Include Library and will instead use another game or template as a starting point. For the few who do use Include Library, leaving the extra #includes in isn’t going to cause any problems, and they can always be removed if the developer realises that they aren’t needed.

Some beginners may start by following Arduino tutorials, which instruct on using the menu. But again, I agree that we shouldn’t be concerned about it.


(Pharap) #88

The Include Library features are gradually sounding worse by the minute.

I would hope most C++ programmers would agree with that, it’s much more idiomatic to take that approach (even if not always the most practical).

That would be a far more sensible approach for them to take. In fact, that should be the official advice next time a beginner turns up asking “where do I start?”.

Such is the beauty of the compiler not compiling unused functions.

If I ever see one of those tutorials, I’ll make sure to leave the author an angry letter. Programming involves a great deal of typing, and beginners need all the practice they can get, even if it’s before they understand what they’re typing.


(Scott) #89

@Pharap, I just discovered that the Arduino people have implemented a solution for the problem of Include Libraries including all .h files. The library.properties file can now contain an includes property, which specifies which files are included when using Include Libraries. This change is available IDE V1.6.10, which is now released. :smiley:


(Chris Smith) #90

All true, but not enough reason I think to simply give up on a small attempt to avoid some of the problems.

Unlike Windows’ calculator, this is going to be documented and the code available. So the ultimate override is just grab the code, and change how the filename is built. Code for the support side utility can be command line mode, making it easy to understand, even for only moderately experienced programmers.

If we were storing a file of hundreds of kilobytes, I might agree. But the data involved here, whether binary or hex, is less than the size of a cluster on a modern hard drive. Note that I do not expect the Arduboy to do this formatting - this is done after receiving from the Arduboy and before writing to disk.

This is never really intended to “look at” numbers. Multibyte values are little-endian, word alignment is uncertain - you’re never really going to know where one value ends and another begins without knowing the program that created the EEPROM. It’s more like the .hex format files - which is, in fact, a possible option, if slightly more detailed than this.

The last time I looked at a 32-digit number was when looking at an Arduboy complied .hex file.

Not all platforms have readily available hex editors. If you are swapping hex file games using your phone (someone is trying that) then you may find a decent hex editor is not as a readily available. But a “text editor” - (probably what I should have said, not notepad) - that is going to be widely available.


(Pharap) #91

Maybe they heard our disgruntled grumblings. (That or it’s a freak coincidence, but I’m leaning towards the conspiricy theory.)

Now I have to redownload Arduino Studio… again :expressionless:


(Pharap) #92

I’d be more inclined to agree if the Arduboy had more memory. 8bytes out of 1024 is a reasonably large chunk of memory when you’re used to working in the order of gigabytes and megabytes.

Windows’ calculator doesn’t really need documentation, all the options are nicely labeled and reasonably obvious.

Then why store as text and not binary? Either you want it as text so people can go in and change it or you don’t actually need it to be text at all.


(Scott) #93

It was an issue raised by someone who’s library has many .h files to support multiple boards and configurations. It’s a freak coincidence that we happened to be discussing it near the release. :wink:


(Pharap) #94

That’s spooky.
Oh well, all the more reason for the separation idea to go ahead.


(Chris Smith) #95

Note that I’m suggesting that the 8 bytes be part of the 16 reserved bytes. So it’s only using memory you already don’t have.

Read back through the notes in this thread and you will find that using 8 bytes only is likely the smallest suggested management space of any proposal. Some were using as much as 1 byte out every 16.

It’s never going to be that black and white. It’s relatively easy to build the tool to handle text files, and that makes the files accessible.

Storing them this way is similar to the same reason that .hex files - which will be loaded ultimately as binary - are stored as long strings of hex characters. You are not likely to need to modify it, or even look at it. But if it does need inspection or modification, you don’t need anything other than knowledge and a text editor.

I did find hex editors for the Android platform, but they all shared a common hex editor limitation - they can’t insert or delete, just modify or extend. Using a text editor on a hex stream removes that limitation.

This also makes it roughly compatible at a concept level with .hex files for storing (binary) programs, also for the Arduboy.


#96

Necro’ing the thread at the behest of MLXXXp

For my two cents worth.

I really think we should just partition the EEPROM into say, 8 blocks of ‘save area’ and have a header area that has a single byte indicating if a save area is ‘full’ or ‘empty’.

Each game then can check a single byte to find if there is a save slot available. If there is not, it is up to that game to play nice and inform the user or blast someone elses save slot.

Each slot can have a couple bytes at the start used to store some sort of id, eg, enough for say, 12 ASCII characters. the rest is for you to use for storage.

The benefit of this is we can have a single sketch that can be used to manage saved games (eg, present them as a list, back them up, delete them, etc). This sketch is analogous to the ‘explorer’ or ‘finder’ on Windows/Mac etc.

The second benefit of this is the power of constraints. If you cannot fit your save game in a save slot, you either need to redesign it, or split your save file over two slots and deal with that.

From reading the other discussion, there seemed to be a lot of talk about trying to give each game exactly what they require, and then manage that. I believe it makes more sense to treat the EEPROM as a file system with 8 blocks of a certain size, instead of a contiguous block where you try to fit yourself in.

Of course, I am not an Arduino heavyweight, and I have not programmed games before where this has been a problem. I am just approaching this from the POV of things like the early memory cards on things like the PS1, where you could very quickly fill it up with save files and would have to clear some out if you wanted to save your game.

Also, this method avoids any annoying issues with fragmentation, overwriting, etc.


(spinal) #97

There are a few threads on this topic and I really can’t see the point at all.
Sure, reserve x amount of space for identifying the game and system settings. But other than that, it’s needlessly complicated.
If you’re encouraging the use of the game loader software, then all of the extra rules are pointless.
All that has to happen is, as people have said, reserve the first handful of bytes to ID the game currently loaded, using some sort of unique ID code. Use this code as a filename to backup the entire eeprom, Then upload the next game, add the unique ID and check if a save for that game exists, if so, upload it, if not, consider zeroing out the eeprom so that a new game is presented with a clean eeprom and can tell if a save exists or not.
Why should individual games have to ‘play nice’ with all other games?
This method has worked well for homebrew flash carts for various consoles.


#98

Fair enough. Who is in charge of maintaining unique ID’s and where do I get one?


(spinal) #99

That would probably be something the official guys could sort out, but as the arduboy community is still quite small, why not just a simple list here on the site somewhere. Then you could just pick something at random, something similar to how team a.r.g name their files, then check the list to see if its there. If not claim it in a dedicated thread. Then perhaps in the future a database can be set up, using user accounts on the forum or something for automated lookup/submission.


(Scott) #100

If you can get enough developers on board to use your proposed technique, why not offer to maintain the ID’s yourself?