Using Python

Here are @Mr.Blinky Python Utilities:

These can be used to generate carts for the external memory eeprom.

This thread will be used for discussion about using Pyhton as it relates to development of the FX environment.

1 Like

@Mr.Blinky

Is there a way to use the Python utilities without having to modify my %PATH%?

I’d love to be able to just dump all the tools on a USB stick and then use the tools from that USB stick without modifying the environment of the computer interfacing with that USB stick.

1 Like

You can unzip everything into the working folder, but I’m guessing you knew that. Otherwise no, I did some searching I don’t see a way to deploy python in a portable fashion. Looks pretty similar to make or bash tools on windows.

Typically all %PATH% actually does is to make it so that when you do > python some command on the command line, the shell looks for python.exe in one of the folders listed in %PATH%.

This means that ordinarily doing > path/to/python/python.exe some command will do the same thing.
Or, as I’m doing, putting python.exe and its dependencies in the current path will allow > python some command to find the local python.exe.

However, the reliance on the pyserial module complicates matters.
I’ve got a way to just dump python.exe and its gubbins next to the flashcart scripts,
but currently no way to make it detect pyserial.

I’ve discovered the existance of a python ‘virtual environment’ thing that’s supposed to somehow make using libaries locally possible,
but currently I’m not sure how to use it.
(I’m hoping to use venv rather than virtualenv because it’s one less dependency and one less thing to install.)

gitbash is actually pretty easy to use on Windows.
make is terrible on all systems.


I managed to get it working briefly by inserting:

import sys
sys.path.append("./pyserial/")

Into all the scripts.

I’m currently looking for a better way, hopefully by passing ./pyserial into the python command,
or by setting some local environment variable.
Then I can hide it all behind some batch scripts.

It is kind of a PITA to load up the environment on new systems, moreso than installing the Arduino IDE.

What happens when we want to deploy these tools, if we provide a compiled front end, does it have the python interpreter baked into it, and the .py files remain external?

Or would they get rolled up into the compiled binary?

At the moment I’m cheating horribly:

flashcart-writer-wrap.py:

import sys

sys.path.append("./pyserial/");

exec(open("flashcart-writer.py").read());

(Haven’t actually got round to testing with options, I’ll worry about that later.
It’s probably just dumping args into exec somehow anyway.)

Don’t need to do anything special for command line arguments because they get passed in via sys.argv and the current environment is copied over by default when using exec.

So I have a choice of either writing one of these wrappers for every script,
or figuring out how to make one that will also accept the script as a parameter.
E.g. python use-pyserial.py flashcart-writer.py -z 64


use-pyserial.py:

import sys

sys.path.append("./pyserial/");

# Not strictly necessary
argvcopy = list(sys.argv)

file = sys.argv[1];
del sys.argv[0]

exec(open(file).read());

# Not strictly necessary
sys.argv = argvcopy

And thus:
> python use-pyserial.py flashcart-writer.py -z 64

Interesting question. Basically what you want is a portable version right?

I’ve just been playing around a little making a batch file for that.

  • copy python directory to root directory of your pen/thumb/flash-drive and rename it Python
  • copy python27.dll from C:\Windows\System32 (32-bit) or c:\Windows\SysWOW64 (64-bit) to Python directory
  • copy Arduboy Python utilities to root
  • create a batchfile commandline.bat in root

batch file content:

@echo off
pushd "%~dp0"
Path = %CD%Python;%CD%Python\Scripts;%PATH% 
echo %PATH%
start cmd
popd

double clicking the commandline.bat will open a command prompt with correct path to the python files
To run eeprom backup for example you can type the following:

python eeprom-backup.py
2 Likes

Pretty much.

Neither my C:\Windows\System32 nor my C:\Windows\SysWOW64 have a python27.dll file.
In fact neither have any file containing ‘python’.

Evidently I know too much batch because I actually understand what that does. :P

But it doesn’t solve the library dependency issue.
flashcart-writer.py can’t find pyserial.
flashcart-builder.py can’t find PIL.


Using my afforementioned user-pyserial.py I can get it to a state where I can do python use-pyserial.py flashcart-writer.py to activate and use flashcart-writer.py,
but I can’t do flashcart-builder.py because pillow has caused me some issues.
(From the looks of it, it actually has to compile some C code to work properly, which would presumably be done in the install step.)

So I decided to just reimplement flashcart-builder.py in C# because it’s less hassle than figuring out how to get pillow to work without installing it.

I’m part way there. I’ve got the CSV parsing done and everything in the ‘for row’ loop,
I’ve just got to read the .hex files, the image files and the raw data files.

It’s surprising how easy it was to simulate the CSV behaviour:

public class CSVHelper
{
	public static IEnumerable<string[]> Enumerate(string csvFile, char delimeter = ',')
	{
		foreach (var line in File.ReadLines(csvFile))
			yield return line.Split(delimeter);
	}
}

And then:

foreach (var row in CSVHelper.Enumerate(csvFile, ';').Skip(1))

(Skip(1) skips the first row - no need to do next(row, None).)

Oh right only the installer places it there.

I ran my flash drive on a system successfully. I guessing the problem is that youve copied files manually from some source.

Maybe I should add a portable directory to the repo with all files required in there?

Yet another reason I don’t like using installers.
As far as I’m concerned, C:/Windows is for Windows’ use,
nothing else should be trying to install there.

Did that system already have those libraries installed via pip?

Your list of steps didn’t include anything involving putting the libraries on the drive,
and the libraries have to come from somewhere,
so presumably if they’re working and they’re not on the USB drive they must be already on the host system.

From the master branch of the GitHub for each project.
pyserial had no problem, pillow is a different story.

That would be the ideal scenario,
but it’s starting to look like it’s not that simple.

venv might be of some use, but it’s only available on later versions of python (3.5+ I think),
and I haven’t got round to trialling it yet.

There’s so much wrong with Windows… We just have to live with it. :sunglasses:

I copied the python directory from a system with the libraries already installed.

I assumed the source already had the libraries installed

Here’s a zip with Python 2.7 with PIL(low) and PySerial and the batch file

http://mrblinky.net/arduboy/temp/portable.zip

At the moment I’m just seeing problems with Python.
Number one being why the hell would it try to touch C:/Windows?
That’s like trying to (as far as I can tell) write to /boot on Linux.

Like I say, these kinds of nasty surprises are why I prefer to extract .zip files to where I want the files to live.
Programs should remain as isolated as possible until given persmission to interact with things outside their folder.

That almost defeats the point since it requires at least one computer to install the files.

I’ll give it a go when I get chance.

First I need to say I really like the tools. They are nice to use and given the fact they are written in a scripting language it is easy to see what the bootloader is doing and how images are created… And of course I can use it from command line.

One remark is that the scripts assume the path is case insensitive, which is not the case on Linux and the way the path is parsed us also not working on Linux. It is easily fixable, so not really a big issue.

If we are going to call functionality from these scripts I would propose to call the functions directly rather than use exec.

Like

import abc
abc.func()

For me it looks better. I am not sure about exec but it looks like spawning a process in order to execute this.

I agree that using functions would be better if possible,
but that’s the only way I could find to inject a new search path when I was trying to get the paths to find the dependencies properly.

Also I was aiming to avoid modifying the existing files,
otherwise I could have just stuck sys.path.append("./pyserial/"); at the top of every file that needed pyserial.

I’m not actually sure if it does or not.
I think it might just generate a new python environment inside the same instance of the interpreter.
In other words, I think exec is closer in behaviour to eval than the spawn family or the subprocess module.

On the other hand START in batch definitely starts a new process.
Starting new a new procress doesn’t really bother me though.

Small update:

I’ve added an image-viewer script. Using this script you can quicly (pre)view pixel art on Arduboy by drag and dropping a 128x64 bmp/png image onto the script (or run from command line if you want)

The script makes use of Cathy3K bootloader streaming feature. No sketch need to be loaded and the present sketch isn’t altered in any way.

1 Like

Ohh nice! Don’t get me sued by the tamogachi pplz!

Any luck with video compression?

“My business card plays Star Wars: A New Hope” would be, awesome.

I have a special idea.

It might be possible if someone has the source code to this:

Come to think of it the ComputerCraft version should be available somewhere…

(The only version I could find without some random talking over it. Though Mincraft’s music is probably equally as annoying.)

I theory crafted this outz we can nearly do it already at 4fps without compression. I feel like with compression it would be fine!

Im actually considering just daisy chaining the memory chips and using different chip selects of I have to. But it would be better to fit all in one.