What do you want from an Arduboy PC (and Mac) program?

Discussion
1

I’m interested in this “magic bullet” thing everyone seems to want, I keep hearing it’s “coming soon hopefully”. Or am I hearing about several tools? I suppose my main question is: what tool or tools are people looking for? Something to replace the Arduino IDE? An interface to more easily add games to the FX chip by wrapping the existing python/web tools? Or maybe some mega combo IDE + cart builder + debugger madness? I feel like I see various tools here and there which already do one or more of these things very well, but I could be very confused. For instance, I’ve been watching development on the Ardens debugger and it’s pretty dang amazing, there’s like… no need to reimplement a single thing from there IMO. Is this magic tool maybe something that would incorporate the debugger?

I guess the main thing really is: what is your (@bateske or anyone’s) vision for an ideal tool or set of tools, how it will be used and what it will do. Like a theoretical “user sits down and does X Y Z”. If it’s something I can help with, I think it’d be really cool to help improve the new user experience with the Arduboy. I don’t think I can write a magic bullet but I’m pretty sure I can at least take a crack at writing individual, focused tools.

2

The Arduino IDE has its place but you can use other tools already - I use VSCode on a Mac and test my code on Ardens. It works extremely well for me.

I would like to see a webusb or webserial extension to the Cart site to allow people to easily flash updates.

4 Likes
3

:100:

This would be great!

4

This is currently where our hopes and dreams are placed.

There have been a few other attempts. It’s like this , people get excited about it, claim it’s easy and they will knock it out to great fanfare. Then promptly disappear after about a month.

I’m happy to support anyone who can take it on, but I can barely code my way out of a wet paper for loop.

5

Ah I see, so it’s mostly the flash cart creation + uploader tool. Based on the other responses here (thank you @filmote and @acedent) I have some input on a fully-web-based solution, if that’s the eventual hopes and dreams. I don’t want to overstep my boundaries and just dump a bunch of info unsolicited. Just let me know, it’ll probably be a bit of a lengthy post.

As for the utility itself, I’ve gone ahead and reviewed what I assume is the cart creation tool: Cart Builder and the flashing tool: GitHub - MrBlinky/Arduboy-Python-Utilities: Python script to upload .hex, .zip or .arduboy files to Arduboy and Homemade versions. It seems like the steps are pretty straightforward, although I can see how doing it all from the cart creation website would be ideal considering the extra setup (python, pyserial, etc). So my question is: is there a reason the cart creation website… well, has to be a website? Does it have to do with distribution of the games etc? That seems confusing, since the website is distributing the whole thing anyway (and most of them are on the arduboy). It seems like a simple way forward is to simply add a QT frontend to @Mr.Blinky’s Python-Utilities repo which is a reimplementation of the cart website, then provide a pre-packaged “release” which incorporates all the dependencies so users can simply download a tool and run a gui which lets you create carts and flash them all in one. It could still get the cart builder files from the internet so it’s always up to date.

6

In 2 words: User Experience.
I think people, in general, are reluctant to install yet another piece of software… esp. just to play a few games on a handheld device. The browser is a ubiquitous platform. Ideally I could visit a site with all the games, perhaps make my own collection, connect the device and hit upload. That level of simplicity would be wonderful :slight_smile:

7

Ah, the “playing games” aspect. That’s valid, I was considering just from a programming perspective, but there are people who will mostly be just finding more games. Hm… I see.

Well, a portion of what I wanted to say earlier is just this: I think it is a bad idea to use features marked “experimental” on the web in this context (tools made by a small-ish community), especially these particular features (webserial etc). Unless you are paying a web developer to continue to maintain your tools, it will be extremely difficult (and frustrating for end users) to keep up with the changing standards and the possible deprecation of these features. It’s a huge waste of time for a small community to shoot for the stars just to have the web consortium go “welp, we didn’t actually want to do this” and now all your tools are broken. I’ve seen many of these experimental features fall through or otherwise be completely reimagined, and all the apps that relied on them now have to be rewritten. While it might NOT happen, I think you have to weigh the “chance of completely broken tools people spent lots of their precious free time on” with “making users download a single tool for a scrappy 8 bit programming environment”. I know these “other platforms” have that appeal because they give you web IDEs and all that, but (a) those apps were most likely paid for (and it’s a selling point so they have an incentive to continue paying for upkeep when things inevitably break) and (b) all those systems generally use python, which is significantly easier to program for and to plop onto the device. I think it’s actually a different demographic. I might be wrong though, I don’t mean to make a fuss.

8

I’m the opposite generally, I prefer good old fashioned executables because they work even without an internet connection.

But I’ll caveat that by saying it also depends on the number of dependencies and whether it requires an installer or a package manager.

My ideal kind of software is ‘extract from zip and run anywhere, even on a USB stick, no touching the registry without permission, no dumping files to other locations without permission’.

One of the issues I’ve often had with Python (other than disliking the language) is that it’s not always easy to get it to run standalone, it tends to want to be installed in a designated place and set a load of environment variables.

In this case I think the lack of browser support is the bigger issue. It only works on Chrome, Edge, and Opera, not on Firefox or any mobile browser.

If it were more well supported, I think it would be worth the risk that it might just stop working one day. The API itself looks pretty straightforward, so the actual implementation probably wouldn’t be too difficult. The serial protocol is pretty straightforward (see page 8 for the protocol details, and the FX has just a few extra commands).

1 Like
9

I’ve been less busy lately and have been trying to finish up ArduManFX. The main hurdle I keep running into is the catalog of games as some of the more prominent ones like those from TeamARG and also some other developers have specifically requested their games to not be included in a master catalog/repo. This has made it very difficult to devise a solution that uses a master catalog while also allowing people to add their own games to a local catalog/repo. Personally I’ve always been a proponent of freely distributed software should be freely redistributable as well, but I’m also one to not impose my views on others and respect their decisions even if I disagree with their reasoning.

At one point I had attempted to reach out to some individuals via PM to see if we might be able to come to an compromise. For example, Erwin’s repo has the ability to link to the game’s hex file rather than distribute a copy of it. However, with the currently legalese its often considered copyright violation to even provide a link (though I’m 99% that only applies to linking to a copy, but linking to the original should be fine). I’m not a lawyer or an expert on international copyright law so I can’t say for sure. The problem here though with linking is the user isn’t following the link to the developers website instead the link is followed in the background to retrieve the files so the developer isn’t actually getting any useful web traffic.

Until we can establish some sort of official collection of games that others can easily submit games to, it will always remain a thorn in any utility that tries to simplify the cart creation process. Until then I’ll be finishing the rest of ArduManFX and will modify the repo system later as it will at least be able to allow people to manage their own local collections as well.

1 Like
10

Much as it would probably be treading on people’s toes, I’m inclined to say that if the game is published under an open source licence then just host it anyway because the licence explicitly gives you permission to do that. If someone really didn’t want people to host copies of their game, they shouldn’t have used a licence that explicitly allows that.

From the few cases I’ve seen, most people who complain about other people hosting precompiled copies of their games do so because they’re hosting their own copy elsewhere, typically on a personal website, and want people to visit that somewhere else, either so they’ll see the person’s other work or so the person gets ad revenue from visitors.

(Again, if they were really worried about that, they should have opted for a more restrictive licence. Half the point of open source is to allow anyone to host a copy of the code.)

In TeamARG’s case, I doubt they’ll complain much since they’ve pretty much disbanded, their leader (Joeri) has retired, and their website is long since dead.

If that were true then theoretically anyone who linked to anything on the internet that was copyrighted (and not licensed permissively) would be violating copyright, which seems a bit ridiculous.

Anyway, here’s what the most commonly used licences have to say about the matter:

MIT:

Permission is hereby granted, free of charge, to any person obtaining a copy of
this software and associated documentation files (the “Software”), to deal
in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute […]

Apache 2.0:

  1. Redistribution. You may reproduce and distribute copies of the
    Work or Derivative Works thereof in any medium, with or without
    modifications, and in Source or Object form […]

BSD 3-Clause:

Redistribution and use in source and binary forms, with or without
modification, are permitted […]
[…]
* Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution. […]

GPL-licensed stuff is more of an issue because the licence basically forces you to provide the source code as well as the game itself, but it does still permit you to provide copies in binary form provided you meet some restrictions whose wording is headache-enducing. (I really don’t want to go digging through the GPL to quote the relevant parts because it’s an eye-watering behemoth.)

If any of the game assets are licensed separately then that could potentially throw a spanner in the works, but not many people do that, so I doubt it would be an issue.

1 Like
11

I had thought as much. Also for the GPL you’re allowed to freely redistribute binaries without also distributing the source code as long as the binary is unmodified. The main restriction is that the source code needed to reproduce that exact binary must be made freely available (at least as far as I could tell when I read through the whole GPL v2, v3 might be slightly different).

Currently my thinking is that @filmote’s repo is the most comprehensive repo specifically for uploading games to the FX chip and so I will probably modify ArduManFX to use his repo. The only thing I’ll need to do is currently things like Author, Title, and Description are located only in the cart CSV files, but it would be nice if each game had an accompanying INI file providing that information as then ArduManFX could scan the folders for HEX files and replace the HEX with PNG for the image, BIN for FX data (if present) and INI for info. Plus the organization of it would make it easier to create and manage a local collection of games (including WIP ones and ones not allowed in the main repo). The idea I had was to treat the collections similar to searching PATH for an executable. It would simply search each repo in a priority list until it finds the first match (if multiple copies exist I can even make the “Upload” and “Add to Cart” buttons include a drop-down arrow to select the specific version).

With all of that in mind it means I’m nearly finished with ArduManFX and even if the next release still has some minor issues with repo madness it will be able to import CSV carts and also upload compiled cart BIN files.

Ultimately I’m hoping to play around with making a discourse plugin that would allow users to upload their games here using their forum credentials and possibly allow rating games similar to how Erwin’s repo has a rating system, but that’s later.

1 Like
12

Crucially, you have to provide it yourself, you can’t just link to it.

For the GPLv3 the relevant text is:

d) Convey the object code by offering access from a designated
place (gratis or for a charge), and offer equivalent access to the
Corresponding Source in the same way through the same place at no
further charge. […]

(Subsections 6a and 6b only apply when you are ‘conveying’ the code in a ‘physical product’, and 6e only applies if 6d has already occurred, leaving 6d as the only relevant subsection.)

For the GPLv2:

a) Accompany it with the complete corresponding machine-readable
source code, which must be distributed under the terms of Sections
1 and 2 above on a medium customarily used for software interchange; or,

b) Accompany it with a written offer, valid for at least three
years, to give any third party, for a charge no more than your
cost of physically performing source distribution, a complete
machine-readable copy of the corresponding source code, to be
distributed under the terms of Sections 1 and 2 above on a medium
customarily used for software interchange; or,

(Subsection 3c only applies if 3b has already occurred.)

It’s worth noting that this is actually more flexible than the v3 licence. The v3 version added a restrictive clause that means you can only use the ‘written offer’ if you’re distributing the object code via a ‘physical product’.

Ideally I think it would make more sense for each game to have an accompanying .arduboy file rather than an .ini file since those are supposed to contain a .json file that includes all that important information anyway.

.arduboy files are also a good way to bundle together the .hex with any necessary FX data, and to bundle together binaries for different targets, and they can contain relevant images.

With better tooling (i.e. tools that can take .arduboy files an upload the contained game to an Arduboy, including FX data) we may start to see those getting more use.

I’m not sure what you mean by this, but the mention of repos got me thinking…

Since quite a few people upload precompiled binaries to the releases of their GitHub repos, it might be possibly to automate the fetching of games from their corresponding repo, including automatic detection of new releases. (Documentation for listing releases. The returned data seems excessively comprehensive, but it would likely work alright with some periodic polling/querying.)

It may also be possible to auto-detect the game’s licence in some cases. (Documentation for getting the licence of a repository.)

For that matter, there could even be some preagreed way to communicate information like title, author, et cetera by having a particular file uploaded to the repo’s root, a bit like how Arduino and PlatformIO use library.properties and library.json.

It wouldn’t necessarily be the main way to publish, but it might be a good way to keep games up to date for those who do use GitHub. (I expect GitLab could do something similar.)

13

While that might not be terrible we would also need better tooling for creating .arduboy files. I know their just a zip archive with .zip replaced with .arduboy. Also INI files would be more suitable for this use case since JSON would add too much extra overhead for the basic information being stored. With INI files we can simply parse each line separating by the first ‘=’ sign and taking the trimmed version (leading and trailing whitespace removed) of everything before and after that first ‘=’ as a property-value pair. This allows for quick and easy parsing without needing to figure out how to properly parse JSON files in whatever language is being used (they’re simple, but not as simple).

Biggest difficulty is there’s several ways to properly organize the information and multiple ways have been done, but none have ever reached full fruition. This has been my biggest hurdle with developing ArduManFX is the lack of consistency on formatting of information, the original gold cart didn’t even follow the documented formatting of the flash cart (notably missing the part needed to identify each game so they could be updated independently as needed). When I first attempted to figure out a format for a unique identifier to include in that space it kept getting bounced back with no real agreement on a proper method.

For right now I’m completely ignoring the portion of each flash cart game used for uniquely identifying the game and just rewriting the entire FX image with each upload since at least it gets something useable out there.

Ultimately I think I’m going to go with whatever works best for ArduManFX’s setup to get it up and running. If it works and becomes the go to method of uploading games to the ArduboyFX then we can go from there on improving it and creating a better standard for cataloging games.

14

The good news is I’ve been chatting with @filmote in PM and the way he has the catalog setup will make things somewhat easier to incorporate (though I will need to scrap all the github integration stuff, but that’s actually better in the long run).

With his setup I’ll easily be able to create a custom hook into the server to query the list of games and request a copy of specific ones. I’ve forked his repo and am working with a locally configured Apache server to do a test build and then will see how that works.

If all goes well I’ve done some research and have figured out a way to incorporate everything right into Discourse (the only thing we need is credential validation so games are linked to the actual user accounts and so we can have a web-based game submission form for users to upload their games). My thinking on that is a simple “publisher” group (again easy enough to do without risking compatibility issues when Discourse is updated) which allows users to upload games and then moderators would verify and approve them before they’re merged into the master list (added protection against spam and other ne’er do wells).

2 Likes
15

Ye, yes, yes … this is why I did not publish straight to the cart site. (It has caused problems when I was not around to do it as currently it is a really manual task of database inserts …)

1 Like
16

I like the idea of discourse plugin but it’s not possible right now. I use discourses hosting so I can’t use custom plugins, only official ones.

I would have to redeploy the forum to my own server. Which can be done but then I’m responsible for making updates and fixing it if it gets broken and I’m not really qualified to do so.

I’d rather host it on the main domain as it’s own thing so that way the community can stay hosted in the cloud and remain what I consider to be most robust (they also make backups automatically etc)

17

Ah, wasn’t sure if you were hosting directly through them or managing your own server. Its not absolutely necessary but it would have been a nice feature so people don’t have yet another area they need to create an account on (we don’t want anonymous uploads cause that’s just asking for trouble).

Having it hosted on the main domain does grant us the ability for more creative control on the format. Its too bad FireFox is completely opposed to WebSerial because then we could simply add upload capabilities directly to the site. However, that said I think something like ArduManFX would still be beneficial even if we had a fully-functional website because it could be beneficial having an offline program that maintains a cached copy of the catalog of games (updating that cache when internet is available).

There’s really pros and cons of both, having Discourse host the forum and lacking the appropriate knowledge/setup to maintain a custom Discourse server would put that option in the non-practical realm. The alternative is not terrible though so no worries there, just saves me some time in playing around with making a Discourse plugin.

1 Like
18

I think it’s possible to get the function we need with similar iframe as we have with the current chat and using javascript to come in do the heavy lifting on the site.

I think if we wanted to build in that function I can build our own “plugin” that just runs in a JavaScript layer on the forum same as the current emulator, just would have to make some careful decisions on how to implement it.

1 Like
19

To be honest, I would have thought anyone capable of actually writing a game would be capable of writing a JSON file and zipping it up.

Though I don’t think it would be particularly difficult to create a tool for editing .arduboy files either.

There actually was a JSON editor specifically set up for editing the Arduboy JSON format at one point, but as far as I’m aware it was hardly ever used. Probably because the url was so ridiculously long. From what I can tell, it just encodes the entire schema into a URL parameter using Base64.

I’ve dumped it in a gist here if you want to take a look:

I concede that INI files are simpler, but I wouldn’t have thought JSON is prohibatively complicated. I’ve written JSON parsers in a few different languages, and the most complex parts tend to be the character escapes and the floating point numbers.

Though that raises the question “If INI files are so much simpler, why does the .arduboy format not just use INI files in the first place”?

I would presume the answer would either be the need for arrays, the need for typed data, or the fact INI files aren’t actually standardised, so different INI editors use slightly different rules and support different feature sets.

What did it do instead? Did it actually incorrectly format things, or was it just missing certain information?

The list number? Or just the optional zero-terminated strings?

One thing that’s hindered me a bit when trying to deal with the cart formatting is the SHA256 signature. If it weren’t for that, I’d probably have made my own cart editor by now. (I’ve already got something that can upload .hex files and edit the FX memory.)

The ‘uniquely identifying games’ problem has been around more or less since year 1, though applied to different things. Originally it was for EEPROM usage, then game cataloguing, then for the FX.

Ultimately it would either have to involve identifiers handed out by some kind of central authority, identifiers generated by some universally agreed algorithm, or identifiers that are just chosen randomly and are made long enough that the odds of a clash are astronomically high (i.e. UUIDs/GUIDs).

Having a centralised database of games is at least a good step towards the ‘central authority’ idea.

Though whatever ends up in use, there’s always the possibility of someone flouting the system and doing their own thing.

Why would not having GitHub integration be better?

Last I checked, moderators don’t have the ability to edit groups, so unless there’s a way to grant them that ability the onus of allowing people into the ‘publisher’ group would all fall on one person’s shoulders (i.e. @bateske).

Have the developers said why?

I would presume it would potentially be a security issue for one thing.

Personally I’d prefer this anyway. It makes sense to have a system that can function without an internet connection or a web browser.

The only downside is the onus of security.

20

You would think, but I remember there being a lot of discussions and confusions on it. Maybe it was more to do with lack of documentation though.

That’s partly the point though is that with the INI files there isn’t really anything special in terms of parsing. Each line is a key-value pair separated by the first ‘=’ character on that line. That way you don’t have to deal with escapes. When dealing with integers, floats, booleans, and other non-string values they still have to be parsed as a string and converted even if using JSON (its just existing parsers will do that for you, but even if there is one available that still requires learning how to use it).

More or less I’m guessing personal preference really. There’s many standard formats used for storing data (JSON, INI, and XML are three of the most common) each has their own benefits and drawbacks. INI files are a bit lesser known and, like you mentioned, isn’t actually standardized. I myself typically just go with whatever is the simplest format for the end user, if no end user or if prototyping then I go with the one I’m most familiar with parsing (for me that’s XML as I have a nice XML parser that I like).

For the flash carts there was documentation about hashing the title and author of a game since those won’t change (theoretically) as a game gets updated.

Most people don’t know how to use github and don’t want to spend the time learning how to use it just to upload their game. Besides the process is a little lengthy and complex for newcomers (fork the repo, upload your games, submit a PR, seems simple but there’s a lot more to it that seems to deter people).

Ah, didn’t think mods could create groups, but wasn’t sure if they were able to be given permission to add users to specific groups. In any case that’s an issue for later when it comes to working out a central catalog of games.

Actually their stance is with regards to security despite the fact that it would be locked behind a permissions setting anyway. Their main argument is that most end users would not even know what the site would be asking for and might just accept it and serial devices aren’t usually designed to validate data sent to them (their words), this could in turn allow the website to wreak havoc on serial devices.

In that regard my thinking is to implement a WebSerial capable uploader and if people want to use it then it’ll have to be done on Chrome, Safari, Edge, or Opera (they all support WebSerial). If they prefer to stick with FireFox then they’ll need to drop back to either the python tools or ArduManFX once it’s done.

Not sure what security concerns there might be, but I’d be happy to discuss them. Maybe there’s things I hadn’t thought about but could be more-or-less easily protected against.

All-in-all @filmote has done a wonderful job of setting up the Cart Builder website and I think I might be able to help reign in the database and PHP stuff a bit more (I’m no expert but I do have a little more familiarity with it). From there we can figure out a way to auto-generate a PM or some other kind of notification that a game has been submitted for review. Then we can look into adding multiple people in charge of reviewing games so it doesn’t fall on one person. I’ve got a local forked copy of his entire site (including the MySQL database) running on a locally hosted Apache server so I can help improve upon it and integrate it into ArduManFX. It will take a little bit to work everything out but it is a very robust setup that’s easy enough to work with.