Simple 2D ev3devsim Javascript Simulator Running as an ipywidget in Jupyter Notebooks


…for a course revision upcoming, I’ve been tweaking a thing.

The thing is ev3devsim [repo], a Javascript powered 2 robot simulator that allows you to execute Python code, via Skulpt, in the browser to control a simple simulated robot.

The Python package used to control the robot is a skulpt port of ev3dev-lang-python, a Python wrapper for the ev3dev Linux distribution for Lego Ev3 robots. (Long time readers may recall I explored ev3dev for use in an OU residential school way back when and posted a few related notebooks.)

Anyway… we want to use Python in the module revision, the legacy activities we want to update look similar, ish, sort of, almost, we may be able to use some of them, and I want to do the activities via Jupyter notebooks.

So I’ve had a poke around and think I’ve managed to make the fumblings of a start around an ipywidget wrapper for the simulator that will allows us to embed it in a notebook.

Because I don’t understand ipywidgets at all, I’m using the jp_proxy_widget, which I first played with in the context of wrapping wavesurfer.js (Rapid ipywidgets Prototyping Using Third Party Javascript Packages in Jupyter Notebooks With jp_proxy_widget).

Here’s where I’m at [nbev3devsim; Binder demo available, if the code I checked in works!]:

The first thing to notice is that the terminal has gone. The idea is that you write the code in a code cell and inject it into the simulator. My model for doing this is via cell block magic or by passing code in a variable into the simulato (for generality, I should probably also allow a link to a .py file).

The cell block magic still needs some work, I think; eg a temporary alert with a coloured backrgound to say “code posted to simulator” that disappears on its own after a couple of seconds.) I probably also need an easy  way to preview the code currently assigned to the simulated robot.

You might also notice a chart display. This is actually a plotly streaming line chart that updates with sensor values (at the moment, just the ultrasound sensor; other sensors have different ranges, so do I scale those, or use different charts perhaps?)

There is also an output window your code can print messages to, as the following hello-world magic shows:

We can read state out of the simulator, though the way the callback work this seems to require running code across two cells to get the result into the Python environment?

I’ve also experimented with another approach where the widget’s parent object grabs (or could be regularly updated to mirror, maybe?) logged sensor readings from inside the simulator, which means I can interrogate that object, even as the simulator runs. (I also started to explore using a streaming dataframe for this data, but I’m not convinced that’s the best approach; certainly trying to stream logged data from the simulator into a streaming chart in the notebook context is laggy compared to the chart embedded in the simulator context.)

With the data in the Python context, we can do what we like with it, Like chart it etc.

There’s a lot of tweaks that need to be made and things to be added to run the full complement of activities we ran in the original presentation of the course.

I’d already started to explore what’s required to add Python functions to skulpt (eg Simple Text to Speech With Skulpt), although I’m not sure if that’s blocking (could it be handled asynchronously if so?) and today managed to learn enough from this SO answer on making objects draggable to make the robot draggable on the canvas (I think; a PR is here but not tested in a fresh/isolated environment yet (I only made a PR to give me s/thing to link to here!); the biggest issue I had was converting mouse co-ordinates to robot world canvas co-ordinates. There are still issues there, eg in getting the size of the robot correctly, but the co-ordinate management in the simulator looks a bit involved to me and I want to get my head round it enough that if I do start trying to simplify things, I don’t break other things!)

Other things that really need adding:

  • ability to reset canvas in one go;
  • ability to rotate robot using mouse;
  • ability to add noise to motors and sensors;
  • configure robot from notebook code cell rather than simulator UI? (This could also be seen as an issue about whether to strip as much out of the widget as possible.)
  • predefine sensoble robot configurations; (can we also have a single, centreline front mounted light sensor?)
  • add pen-up / pen-down support (perhaps have a drawing layer in the simulator for this?)
  • explore supporting multiple simulators embedded in one notebook (currently it’s at most one, I suspect in large part becuase of specific id values assigned to DOM elements?)

The layout is also really clunky, the main issue being how to see the code against the simulator (if you need to). Two columns might be better — notebook text and code cells in one, the simulator windows stacked in the other? — but then again, a wide simulator window is really useful. A floatinging / draggable simulator window might be another option? I did thing the simulator window might be tear-offable in JupyterLab, but I have never managed to successfully tear off any jp_proxy_widget in JupyterLab (my experiences using JupyterLab for anything are generally really miserable ones).

The original module simulator allowed you to step through the code, but: a) I don’t know if that would be possible; b) I suspect my coding knowledge / skills aren’t up to it; c) I really should be trying to write the activities, not sinking yet more time into the simulator. (One thing I do need to do is see if any of the code I wrote years ago when scopting things for the residential school is reusable, which could save some time…)

I also need to see if the simulator is actually any good for the practical activities we used in the original version of the course, or whether I need to write a whole new set of activities that do work in this simulator… Erm…

Setting Up PyCharm To Work With a Lego Mindstorms EV3 Brick

Notes based on Setting Up a Python Development Environment with PyCharm for setting up PyCharm editor (I use the free Community Edition) to work with EV3. Requires passwordless ssh into the brick and the brick on

We’re going to go round the houses with git checkins to move stuff from the Mac and the PyCharm editor to the brick.

Get into the brick and do some minimal config stuff:

[Mac] ssh robot@

[EV3] sudo apt-get update
[EV3] sudo apt-get install git
[EV3] mkdir -p /home/robot/demoproj
[EV3] mkdir -p /home/robot/demoproj/.demoproj.git
[EV3] git init --bare /home/robot/demoproj/.demoproj.git

Now use the nano editor on the brick to populate the demoproj dir with the files we push, setting them to be executable.

[EV3] nano /home/robot/demoproj/.demoproj.git/hooks/post-receive

In the nano editor, change the file to:

git --work-tree=/home/robot/demoproj --git-dir=/home/robot/demoproj/.demoproj.git checkout -f
find /home/robot -iname \*.py | xargs chmod +x

The chmod on the py files makes them executable, so as long as you hashbang the first line of any python files (with #!/usr/bin/python), they should be runnable from the file browser menu on the brick.

Then ctrl-x to exit, saying Yes to save the file on the way out and accepting the default file name. Now make that hook executable:

[EV3] chmod +x /home/robot/demoproj/.demoproj.git/hooks/post-receive

Should we prepend hashbangs as well? This will add one to start of all py files not containing one:

grep -rL '^#!/usr/bin/python$' /home/robot/demoproj/*.py | xargs sed -i '1i #!/usr/bin/python'

In PyCharm, VCS->Check Out From Version Control to create a new project, selecting git as a the checkout target (so you’ll also need git installed on the Mac…).

The VCS Repository URL is: robot@, the Parent Directory (for example, /Users/me/projects) and Directory names (e.g. testProj – note, this must be a new folder) specifying the location on the Mac where you want to keep a local copy of the project files.

Say yes to all the crap PyCharm wants to create, and Yes when it prompts if you want to open the newly created directory. Create a new python file containing the following test program:

#The shebang above runs this file with the python shell
from import Sound
#Make a short sound

Save it, press the VCS /UP ARROW/ button top right in PyCharm to commit, add a commit message, then bottom right Commit and Push.

This should commit the file locally and also push it into the git repo on the ev3; the commit hook on the ev3 will copy the file into the demoproj folder and, if it’s a .py file, mark it as executable.

You should now be able to locate it and run it from the ev3dev file browser.

See also: Using IPython on Lego EV3 Robots Running Ev3Dev and Running Python Programmes on the Lego EV3 via the EV3 File Browser.

PS to enable autocompletion in PyCharm, check what Python shell the project is using (in the settings/preferences somehow though I’m damned if I know where to find them; I HATE IDEs with a passion – way to cluttered and over complex…). Then using the same Python shell:

git clone
cd ev3dev-lang-python
python install
cd ..
rm -r ev3dev-lang-python/

Run the programme in PyCharm (or at least, the import lines) and autocompletion should be enabled. The complete code won’t run properly though, because you’re not running it in an ev3dev environment…

Running Python Programmes on the Lego EV3 via the EV3 File Browser

A quick note to self… If we test python scripts on an EV3 brick interactively via a Jupyter notebook and then save the notebook as a python file (eg, we can get the Python file over to the EV3 brick and into a state we can run it from the file browser by using the following recipe:

echo '#!/usr/bin/python'|cat - > /tmp/out && mv /tmp/out
scp robot@
ssh robot@ 'chmod ugo+x /home/robot/'

Then in the file browser on the brick, just click on to run it.

It would be easy enough to script this so a single button in the notebook does it automatically? (In which case, it would probably make more sense to modify the template used to generate the Python save file so that it includes the shebang at the start of the file anyway?)

PS see also Setting Up PyCharm To Work With a Lego Mindstorms EV3 Brick and Sharing Files With a Lego EV3 Brick Over Wifi Using Filezilla.

Using IPython on Lego EV3 Robots Running Ev3Dev

In part so I don’t lose the recipe, here are some notes for getting up and running with IPython on a Lego EV3 brick.

The command lines are prefixed to show whether we’re running them on the Mac or the brick…

To start with, we need to flash a microSD card with an image of the ev3dev operating system. The instructions are on the ev3dev site – Writing an SD Card Image Using Command Line Tools on OS X. – , but I repeat the ones for a Mac here for convenience.

  1. Download an image from the repository – I used the ev3dev-jessie-2015-12-30 image because the current development version is in a state of flux and the python bindings don’t necessarily work with it just at the moment…
  2. Assuming you have downloaded the file to your home Downloads directory (that is, ~/Downloads), launch a new terminal and run: cd Downloads
    [Mac] unzip
    [Mac] diskutil list
  3. Put the microSD card (at least 2GB, but no more than 16GB, I think? I used a 4GB microSD (HC) card) into an SD card adapter and pop it into the Mac SD card reader. Check that it’s there and what it’s called; (you’re looking for the new disk…):
    [Mac] diskutil list
  4. Now unmount the new disk corresponding the the SD card: diskutil unmountDisk /dev/disk1s1Downloads_—_robot_ev3dev____—_ssh_—_112×25
    If you don’t see the /dev/desk1 listing, check that the write protect slider on your SD card holder isn’t in the write protect position.
  5. We’re going to write some raw bits to the card (/dev/disk1 in my example), so we need to write to /dev/rdisk1 (note the r before the disk name). The write will take some time – 5 minutes or so – but if you’re twitchy, ctrl-T will show you a progress message). You’ll also need to enter your Mac password. (NOTE: if you use the wrong disk name, you can completely trash your computer. So be careful;-)
    [Mac] sudo dd if=~Downloads/ev3-ev3dev-jessie-2015-12-30.img of=/dev/rdisk1 bs=4m
    GOTCHA: when I tried, I got a Permission Denied message at first. It seems that for some reason my Mac thought the SD card was write protected. On the SD card adapter is a small slider that sets the card to “locked” or “unlocked”. The SD card reader in the Mac is a mechanical switch that detects which state the slider is in and whether the card is locked. I don’t know if it was a problem with the card adapter or the Mac reader, but I took the card reader out of the Mac, changed the slider setting, put card reader back in, and did the unmount and then sudo dd steps again. It still didn’t work, so I took the card out again, put the slider back the other way, and tired again. This time it did work.
  6. Once the copy is complete, take the SD card adapter out, remove the microSD car and put in in the EV3 brick. Start the EV3 brick in the normal way.

Hopefully, you should the brick boot into the Brickman UI (it may take two or three minutes, include a period when the screen is blank and the orange lights are ticking for a bit…)

Navigate the brick settings to the Networks menu, select Wifi and scan for your home network. Connect to the network (the password settings will be saved, so you shouldn’t have to enter them again).

By default, the IP address of the brick should be set to To make life easier, I set up passwordless ssh access to the brick. Using some guidance in part originally from here, I accessed the brick from my Mac terminal and set up an ssh folder. First, log in to the brick from the Mac terminal:

[Mac] ssh robot@

When prompted, the password for the robot user is maker.


This should log you in to the brick. Run the following command to create an ssh directory into which the shh key will be placed, and then exit the brick commandline to go back to the Mac command line.

[Brick] install -d -m 700 ~/.ssh
[Brick] exit

Create a new key on your Mac, and then copy it over to the brick:

[Mac] ssh-keygen -R
[Mac] cat ~/.ssh/ | ssh robot@ 'cat > .ssh/authorized_keys'
You will be prompted again for the password – maker – but next time you try to log in to the brick, you should be able to do so without having to enter the password. (Instead, the ssh key will be used to get you in.)


If you login to the brick again – from the Mac commandline, run:

[Mac] ssh robot@

you should be able to run a simple Python test program. Attach a large motor to input A on the brick. From the brick commandline, run:

[Brick] python

to open up a Python command prompt, and then enter the following commands to use the preinstalled ev3dev-lang-python Python bindings to run the motor for a few seconds:

[Python] import ev3dev.ev3 as ev3
[Python] m = ev3.LargeMotor('outA')
[Python] m.run_timed(time_sp=3000, duty_cycle_sp=75)


[Python] exit

to exit from the Python interpreter.

Now we’re going to install IPython. Run the following commands on the brick command line (update, but DO NOT upgrade the apt packages). If prompted for a password, it’s still maker:

[Brick] sudo apt-get update
[Brick] sudo apt-get install -y ipython

You should now be able to start an IPython interpreter on the brick:

[Brick] ipython

The Python code to test the motor should still work (hit return it you find you are stuck in a code block). Typing:

[Brick] exit

will take you out of the interpreter and back to the brick command line.

One of the nice things about IPython is that we can connect to it remotely. What this means is that I can start an IPython editor on my Mac, but connect it to an IPython process running on the brick. To do this, we need to install another package:

[Brick] sudo apt-get install -y python-zmq

Now you should be able to start an IPython process on the brick that we can connect to from the Mac:

[Brick] ipython kernel

The process will start running and you should see a message of the form:

To connect another client to this kernel, use:
--existing kernel-2716.json

This file contains connections information about the kernel.

Now open another terminal on the Mac, (cmd-T in the terminal window should open a new Terminal tab) and let’s see if we can find where that file is. Actually, here’s a crib – in the second terminal window, go into the brick:

[Mac] ssh robot@

And then on the brick command line in this second terminal window, show a listing of a the directory that should contain the connection file:

[Brick] sudo ls /home/robot/.ipython/profile_default/security/

You should see the –existing kernel-2716.json file (or whatever it’s exactly called) there. Exit the brick command line:

[Brick] exit

Still in the second terminal window, and now back on the Mac command, copy the connection file from the brick to your current directory on the Mac:

[Mac] scp robot@ ./

If you have IPython installed on you Mac, you should now be able to open an IPython interactive terminal on the Mac that is connected to the IPython kernel running on the brick:

[Mac] ipython console --existing ${PWD}/kernel-2716.json --ssh robot@

You should be able to execute the Python motor test command as before (remember to import the necessary ev3dev.ev3 package first).

Actually, when I ran the ipython console command, the recent version of jupyter on my Mac gave me a depreaction warning which means I would have been better running:

[Mac] jupyter console --existing ${PWD}/kernel-2716.json --ssh robot@

So far so good – can we do any more with this?

Well, yes, a couple of things. When starting the IPython process on the brick, we could force the name and location of the connection file:

[Mac] ipython kernel -f /home/robot/connection-file.json

Then on the Mac we could directly copy over the connection file:

[Mac] scp robot@ ./

Secondly, we can configure a Jupyter notebook server running on the Mac to so that it will create a new IPython process on the brick for each new notebook.

Whilst you can configure this yourself, it’s possibly easy to make use of the remote_ikernel helper:

[Mac] pip3 install remote_ikernel
[Mac] remote_ikernel manage --add --kernel_cmd="ipython kernel -f {connection_file}" --name="Ev3dev" --interface=ssh --host=robot@

Now when you should be able to connect to a notebook run against an IPython kernel on the brick.


Note that it may take a few seconds for the kernel to connect and the first cell to run – but from then on it should be quite responsive.


To show a list of available kernels for a particular jupyter server, run the following in a Jupyter code cell:

import jupyter_client

PS for ad hoc use, I thought it might be useful to try setting up a quick wifi router that I could use to connect the brick to my laptop in the form of an old Samsung Galaxy S3 android phone (without a SIM card). Installing the Hotspot Control app provided a quick way of doing this… and it worked:-)

PPS for a more recent version of IPython, install it from pip.

If you installed IPython using the apt-get route, uninstall it with:

[Brick] sudo apt-get uninstall ipython

Install pip and some handy supporting tools that pip may well require at some point:

[Brick] sudo apt-get install build-essential python-dev


[Brick] sudo apt-get install python-pip

would run an old version of pippip --version shows 1.5.6 – which could be removed using sudo apt-get remove python-setuptools.

To grab a more recent version, use:

[Brick] wget
[Brick] sudo -H python

which seems to take a long time to run with no obvious sign of progress, and then tidy up:

[Brick] rm

Just to be sure, then update it:

[Brick] sudo pip install --upgrade setuptools pip

which also seems to take forever. Then install IPython:

[Brick] sudo pip install -y ipython

I’m also going to see if I can give IPythonwidgets a go, although the install requirements looks as if it’ll bring down a huge chunk of the Jupyter framework too, and I’m not sure that’s all necessary?

[Brick] sudo pip install ipywidgets

For a lighter install, try sudo pip install --no-deps ipywidgets to install ipywidgets without dependencies. The only required dependencies are ipython>=4.0.0, ipykernel>=4.2.2 and traitlets>=4.2.0;.

The widgets didn’t seem to work at first, but it seems that following a recent update to the Jupyter server on host, it needed a config kick before running jupyter notebook:

jupyter nbextension enable --py --sys-prefix widgetsnbextension

PPS It seems to take a bit of time for the connection to the IPython server to be set up:


The Timeout occurs quite quickly, but then I have to wait a few dozen seconds for the tunnels on ports to be set up. Once this is done, you should be able to run things in a code cell. I usually start with print("Hello World!") ;-)

PPPS For plotting charts:

sudo apt-get install -y python-matplotlib

Could maybe even go the whole hog…

sudo apt-get install -y python-pandas
sudo pip install seaborn

PPPPS Here’s my current build file (under testing atm – it takes about an hour or so…) –, so:
[Mac] scp robot@
[Brick] chmod +x
[Brick] sudo ./

sudo apt-get update
sudo apt-get install -y build-essential python-dev
sudo apt-get install -y python-zmq python-matplotlib python-pandas

sudo -H python
sudo pip install --upgrade setuptools pip

sudo pip install ipython ipykernel traitlets seaborn
sudo pip install --no-deps ipywidgets

PPPPPS to clone the SD card on a Mac, insert the SD card and run:

[Mac] diskutil list
[Mac] sudo dd if=/dev/disk1 of=~/Desktop/my_ev3dev_image.dmg

The corresponding restore (in the process described at the start of this post) would use /my_ev3dev_image.dmg rather than the ev3-ev3dev-jessie-2015-12-30.img image.

PPPPPPS Connecting to remote kernel on brick – start IPyhton kernel on brick:

[Brick] ipython kernel -f /home/robot/test.json

Copy the connection file over to the host:
[Mac] scp robot@ ./

Check the path you copied it to
[Mac] pwd

For me, that returned /Users/ajh59.

Start a console on the host using the existing connection file – use a full, explicit path to the file. Also works with things like Spyder:

[Mac] jupyter console --existing /Users/ajh59/test.json --ssh robot@