ArduManFX utility [WIP]

I’ve got the initial parts setup for the ArduManFX utility I’ve been working on.
https://github.com/tuxinator2009/ArduManFX

For those who aren’t aware, this utility is a GUI tool based on the python scripts created by @Mr.Blinky for managing flash cart images for use with the RAM expansion.

This tool is still in a very early alpha stage right now but it is able to download a cached copy of @eried’s “unofficial” Arduboy repo and allows you to upload any of the games to your Arduboy. Later it will have the ability to create and write flash cart images for the FX mod. I’ve also worked out an idea for adding a way to help recover “bricked” Arduboy’s though it needs some extensive testing. When an Arduboy becomes difficult to upload a sketch (typically from a failed upload) you usually have to time the pressing of the reset button just right, but if I have the program constantly checking for the device to show up the moment you press the reset button then quickly force it into the bootloader it can then upload a basic dummy sketch that makes it possible to then upload any sketch. This would in theory eliminate the need for pressing the reset button at just the right moment.

I still need to create Windows, Linux, and Mac binary release packages but for now anyone download and install Qt5. From there it should just be a matter of running qmake followed by make since it doesn’t have any other dependencies.

4 Likes

I’ve just implemented a utility to recover corrupted Arduboys!!!
The github repo has been updated accordingly and here is a promo video showcasing the new feature:


Next up is to get all the release packages built which will take some time to setup all my dev environments in VirtualBox again (upgraded my computer a while back and my old environments were still using Windows XP, Ubuntu 12.04, and Mac OS X Snow Leopard). Once I’ve got the environments setup again I can start building binary release packages. In the meantime it should still be easy enough to follow Qt5’s installation instructions and build it from source.
3 Likes

Very nice! we, lazy win users need bin releases to check it out :heart_eyes:

I’ve built a Windows binary pre-release. For lazy Windows users.
Still working on a Mac and Linux binary.

Here’s a direct link for the extra lazy :stuck_out_tongue:: ArduManFX-windows.zip

3 Likes

Nice! thanks As lazy tester I can say that I just get 1 enabled button (only the first one)

Upload works, and it looks very promising :smiley: specially the non dependency from avrdude

Too bad it doesn’t run on my W8 32-bit PC Guess it’s built for 64-bit?

Yes

32-bit computers still exist? :P

1 Like

32-bit OS is not restricted to 32-bit hardware. Most of the software I use is 32-bit though. Why would I run them virtualized on 64-bit OS with lower performance? :stuck_out_tongue:

FYI I do have a W10 64-bit notebook, just don’t use it as much.

Just ran the binary on my laptop. @tuxinator2009 It looks nice. Like the details on the upload progress bar.

It downloaded the list from @eried’s repo. But it downloaded one game and reports a Network 301 code in the bottom left.

A note on uploading. It looks that you don’t issue the ‘E’ command to exit programming mode. This command also sets a bootloader timeout to 0.5 seconds. So the uploaded program starts more quickly. With the FX bootloader this command is required as the bootloader menu doesn’t time out.

Keep up the good work :+1:

1 Like

I’ve gotten that on rare occasions and on the command line it reports the following:
QNetworkReplyImplPrivate::error: Internal problem, this method must only be called once.
Haven’t been able to find any information on why that’s happening, but only on some systems, and even then only occasionally. Will need to investigate further.

Just looked back over the Python scripts and realized I forgot the bootloaderExit function. Will get that implemented.

I wasn’t aware of this either as both Mac and Linux are completely phasing out support for 32-bit entirely.

The other buttons are disabled for the release build since they’re not ready yet. Better to have a disabled button then an non-functional enabled one.

We have @Mr.Blinky to thank for that. If it wasn’t for his python scripts this probably would depend on avrdude.

3 Likes

After further investigation I’ve discovered that this error is printed to the console when multiple network errors occur since it can only set one error. I managed to reproduce the error 100% of the time on my Linux VirtualBox machine used to test release builds (clean install of Linux). With that I printed additional information such as each file attempting to download with details on the game it originated from, where the file will be saved, what the requested URL is, and what the reply’s URL is (since sometimes they can be different due to redirects). Turns out I made a very simple mistake that magically works sometimes and magically fails other times. The condition on whether it passes or fails depends on an uninitialized bool randomly being false (will pass) vs true (will fail). This is because when detecting the presence of a data file for the FX mod I also flag info.fx = true however I forgot to initially set info.fx = false :man_facepalming:

With that fixed I can release an update so people can actually use it properly for what the initial alpha release was intended for (browsing the repo and uploading games to regular Arduboy’s plus fixing corrupted Arduboys).

This is also fixed in this next release.

For this next release, and future releases, the windows binary will be compiled as 32-bit since there’s no benefit for this program to be compiled 64-bit. This will provide the best compatibility on Windows.

2 Likes

Alas, this is one of those insidious bugs that changes behaviour at runtime and can easily pretend to be working as intended.
(A ‘heisenbug’ if you will :P)

I specifically remember that I once found someone whose code was running as intended but they had an uninitialised variable and I had to go to quite a bit of effort to convince them that although it’s working now it might suddenly stop working.

Also, this is where compiler warnings really come in handy.


I’d like to mention that this is one of the reasons I dislike the C-style approach of “declare all the variables at the start of the function”.
It makes it much easier to accidentally forget to initialise something.

Typically variables should be created and initialised as close to where they’re actually needed as possible.
The shorter the lifespan of a variable, the less likely it is to cause bugs.

(Some references on the issue: 1, 2.)

(This is as much general advice to anyone reading this comment as it is specific advice for trying to avoid situations like that in future.)

Despite compiling with -Wall the compiler didn’t generate a warning on this one because the value was inside a struct.

I actually prefer this approach since it gives a concrete location to jump to when reading code to know what type the variable is as opposed to scanning for it’s definition somewhere within the function (though I declare at the top of the scope the variable is used in as opposed to just having everything declared at the top of the function).

These are always the really fun ones to debug since they can’t always be reproduced 100% of the time so it’s hard to tell if the actual bug is fixed or if it was something else causing it altogether, or even a combination of.

That’s some good information. I noticed in reference 1 the general statement is that location of declaration doesn’t usually impact memory footprint or performance since there’s no standard for where/when/how the memory should be allocated by the compiler. So reference 1 then goes on to show how it’s purely semantic and is really just for readability purposes. Most of the code I read when I was new had functions with hundreds of lines of code each (really bad) and it was often difficult to tell what type a variable was, and thus the intent of the code, because they were declared at first use, but then used throughout the whole function. This is why I’ve generally steered towards the habit of declaring my variables at the top of the scope they’re going to be used in (not necessarily the top of the function).

Reference 2 has a lot more useful information that mostly pertains to c++11 and up since it talks about using constexp and also using lambda functions as initializes which didn’t always exist (especially not in 2003 when I first got started with c++).

Both methods seem valid for different reasons, but it’s good to also be aware of performance related issues and the always fun heisenbugs (I’ve had some doozies in the past).

The next update I’ll separate the downloading of the repo from the parsing of the repo this way it can also cache the json file and only re-download it if it changes. This should hopefully speed up the loading time, but I think the majority of the time is in parsing the repo and creating the widgets for each game (there are better ways to do it, but they can get very complex so I’d rather avoid it for such a basic utility unless it becomes necessary).

1 Like

There is now an official binary for MacOS for all you Mac fans out there.
Check out the github page for details.

Assuming unique variable names, the first declaration will be the first occurance of the name within the function, so a ctrl+F should more or less give the right result immediately.

Though if you’re using a decent IDE/editor, just putting your mouse over the name is usually enough to give you the type.
(Arduino IDE won’t, but VSCode will.)

I think you misread it. Paritcularly the part about:

As for the memory usage, in general a typical implementation will immediately (as you enter the function) allocate the maximum space required for all variables that exist at the same time. However, your declaration habits might affect the exact size of that space.

And the example goes on to demonstrate that.
It’s a bit of an over-simplification, not least because it talks as if the variables are guaranteed to be stored on the stack, but it’s still generally right.

The more complex language-lawyering explanation...

There are rules about side effects and sequencing (formerly ‘sequence points’, now ‘sequenced before’ and ‘sequenced after’),
and ultimately what the compiler can do depends on the semantics of the type.

If you’re dealing with fundamental types or structs of types where all the constructors, destructors and assignment operators are trivial (typically defaulted) then the compiler can reshuffle things as it sees fit (as long as it obeys the ‘as-if’ rule) because a defaulted assignment is just a simple bitwise copy and a defaulted default constructor or destructor effectively means ‘do nothing’.

However, if either the constructor or the destructor are user defined the compiler usually has to assume that there might be side effects and thus has to construct the type at the point of declaration and not tear it down until its enclosing scope actually ends.
Similarly a user-defined assignment operator puts a spanner in the works.

In other words if you were to put std::vector<int> vector; at the top of your function, the compiler would have to default construct it there and then (which means a dynamic allocation),
and if you had two std::vectors, the order they are declared in will be the order their destructors are called in.
(The same probably applies to most QT objects, and certainly applies to things like file handles, smart pointers and other container types.)

But even if you’re dealing only with trivial types,
the compiler may end up producing less efficient code for variables that are given a longer lifetime than necessary simply because it affects how the compiler decides to allocate resources.

Particularly if the types involved are large enough that they must be allocated on the stack and can’t be stored in a register,
and particularly if the code is complex or involves loops.

That’s just bad style regardless.

Stylistic issues aside, too much code in a single function tends to upset the compiler.

The compiler prefers short digestable chunks,
otherwise there’s too much scope to keep track of and comprehend,
and it can’t do as good a job as if you split the code up into many smaller functions.

That reminds me of another issue...

Often people try to reuse variables, presumably due to some ill-concieved notion that doing so is somehow more efficient because it’s reusing memory, but the reverse is actually true.

(I won’t go into the mechanics of it now because that’ll add another paragraph or two, but basically it’s to do with lifetimes and resource allocation.)

It’s more or less functionally equivalent to just writing a custom function to do the initialising, which could have been done back in C++03,
but lambdas have two advantages:
they can capture variables in the parent scope (by value or by reference),
and they can be anonymous so you don’t have to name them.

Personally speaking I haven’t encountered a situation where I’ve needed to do that,
and I’d be tempted to create a named function anyway,
unless it genuinely was a one-off and I was sure I’d never want to do the same thing again.

It seems to me that it’s more likely to be the file acesses.

You’d probably get better performance if you just parsed the basic descriptions/metadata from the .csv and then lazy-loaded the graphics and game data on demand.
(After all, you probably don’t need to load the game data unless an upload has been requested.)

1 Like

Actually this is what it’s doing. It downloads the json file then for each game it has to create a new QWidget subclass to display all that game’s content, it also queues up all the game files (title image, hex, data file, screenshots, etc.) into a download queue. The subclass shows a black image that says “Image Loading” at first. Once the repo has been parsed and all widgets displayed then it downloads all the images and other files in the background (updating the status bar at the bottom accordingly). As each title image is downloaded it replaces the appropriate “Image Loading” place holder image. When the hex file is downloaded (and data file in the case of ArduboyFX games) then the upload button is enabled allowing you to upload it to your device.

The downloads definitely take some time (even once the initial cache is made checking each file for updates) but that’s all done in the background so it’s not really a big deal (once the cache is made you can immediately upload a game without having to wait for it to download again). The slowdown is creating every GameWidget (which is a subclass of QWidget) that contains multiple widgets for displaying the various info about a game. The really fast way of doing it to only load each game’s info and create a custom model view that only renders what’s visible on screen as you scroll. However, this method can be obscenely more complicated because of having to setup all of the rendering routines as opposed to simply creating some QWidgets. The memory footprint isn’t really a problem because the program uses ~100Mb of RAM and once all the widgets are created it uses ~150Mb, so it’s more the slowness of setting up each GameWidget.

As far as the variable declaration goes I do remember when I first started learning C back in 2002-2003 gcc forced the old C89 standard of requiring variables to be declared at the top of the function. Similarly to how you can’t use c++11 features unless you specifically tell g++ to enable c++11 by passing -std=c++11 on the command line.

I can see that if you declare int a = something_something_dark_side and do stuff with a then later declare int b = something_completely_different and use it but don’t use a anymore then the compiler could make the distinction and only allocate space for one int instead of two, but to that extend I usually just leave it to the compiler to optimize where it can.

Regardless of style though (including casing of varibale/function/etc names) the main importance for code read-ability is to stick to the same style throughout the entire code base. I’ve seen projects with multiple programmers that didn’t have any kind of internal organization standard and it was really hard to figure out what everything did simply because everything was organized differently.

1 Like

In the Windows package there are few big unnused files (opengl32sw.dll, D3Dcompiler_47.dll) and few smaller ones (qgenericbearer.dll, qsvgicon.dll, qsvgicon.dll, imageformats*.dll, translations*.qm, libEGL.dll, libGLESv2.dll, Qt5Svg.dll).

Package could be 70% smaller :stuck_out_tongue:

For the Windows and Mac binaries Qt has a command windeployqt and macdeployqt that bundles all the necessary libs for the program to run without requiring Qt to be installed. The reason for the dlls that aren’t being used is because they’re not dynamically linked at compile time and thus it has no way of knowing if they’re actually used. Instead they are loaded at runtime so that Qt can dynamically use whichever plugins are best suited for the current system. The problem with testing which ones can be removed is they may not be used on my system, but Qt could might use them on other systems so it’s better to just let it deploy what it knows it might need.

2 Likes

I’m most likely either looking at the wrong function or misreading it then.

I was looking at parseCSV and I took the creation of a QImage to mean that the image is created and loaded immediately.

Is there a way to pause or stop the downloading?
(Thinking of the poor souls with rural-grade internet speeds.)

Yes, I can see how that would be a problem.
I presume many of these ‘widgets’ are quite large.

By then it really should have been using C99.
That’s a big disappointment (and probably explains a lot).

Actually by C89 the rule was at the top of the scope,
not the top of the function.
Thus you could declare variables at the top of a for/while loop, at the top of an if block,
or even at the top of a regular block statement like so:

int main()
{
    int i = 22;
    printf("%d\n", i);
    {
        int j = 42;
        printf("%d\n", j);
    }
    return 0;
}

I feel quite a lot of people misunderstood that rule,
or perhaps couldn’t overcome their pre-ANSI habits.

(From C99 onwards the rules are relaxed and you can declare anywhere.)

Again, that’s a bit of an oversimplification, but yes,
the less variables needed the less the compiler has to juggle things around,
the less chance there is of needing to generate code that juggles things around,
and the less likely you are to actually need stack space in the first place.
(Needing only registers and no extra stack space is typically an ideal situation.)

The problem is that to get optimum results sometimes you have to help it out by taking some of those rules into account and doing things that permit the compiler more flexibility.

Compilers are smart, but there are some things compilers can’t do,
either because they’re bound by language rules and doing those things would change the program’s semantics,
because said things haven’t been implemented,
or because said things are simply too complex to implement.

I don’t mean things like using shifts instead of multiplication or division,
(that’s one case where usually you’re actually more likely to shoot yourself in the foot,)
but certainly things like using the minimum possible scope can be helpful.

I’ve been part of projects like this.

Unfortunately getting people to agree on a particular style to stick to can sometimes be a bit of a nightmare.
Usually either because they don’t think it’s important enough to sit down and come to an agreement or because they simply aren’t willing to do things differently to how they normally do them.

Tangentially related:
I remember a time when I hated underscores and would never have used them unless I really had to,
but now I’ll happily use ‘snake_case’ if the situation calls for it.