Exposing Services Running in a Docker Container Running in Virtualbox to Other Computers on a Local Network

Most of my experiments with Docker on my desktop machine to date have been focused on reducing installation pain and side-effects by running applications and services that I can access from a browser on the same desktop.

The services are exposed against the IP address of the virtual machine running docker, rather than localhost of the host machine, which also means that the containerised services can’t be accessed by other machines connected to the same local network.

So how do we get the docker container ports exposed on the host’s localhost network IP address?

If docker is running the containers via Virtualbox in the virtual machine named default, it seems all we need to do is tweak a couple of port forwarding rules in Virtualbox. So if I’m trying to get port 32769 on the docker IP address relayed to the same port on the host localhost, I can issue the following terminal command if the Docker Virtualbox is currently running:

VBoxManage controlvm "default" natpf1 "tcp-port32769,tcp,,32769,,32769"

which has syntax:

natpf<1-N> [<rulename>],tcp|udp,[<hostip>], <hostport>,[<guestip>],<guestport>

Alternatively, the rule can be created from the Network – Port Forwarding Virtualbox  settings for the default box:

default_-_Network

To clear the rule, use:

VBoxManage controlvm "default" natpf1 delete "tcp-port32769"

or delete from the Virtualbox box settings Network – Port Forwarding rule dialogue.

If the box is not currently running, use:

VBoxManage modifyvm "default" --natpf1 "tcp-port32769,tcp,,32769,,32769"
VBoxManage modifyvm "default" --natpf1 delete "tcp-port32769"

The port should now be visible and localhost:32769 and by extension may be exposed to machines on the same network as the host machine by calling the IP address of the host machine with the value of the forwarded port on host.

On a Mac, you can find the local IP address of the machine from the Mac’s Network settings:

Network

Simples:-)

Sharing Files With a Lego EV3 Brick Over Wifi Using Filezilla

A handy comment on the ev3dev github repository shows how easy it easy to share files between a laptop/desktop computer and a networked Lego EV3 brick over wifi.

For a brick on e 192.168.1.016, from the Filezilla File -> Site Manager… menu

Site_ManagerWe can also set the default directory on the brick (as well as on host):

Site_Manager2

From the Filezilla View -> Filename Filters… option we can hide the display of hidden files:

Directory_listing_filters

To copy files, simply drag them from one machine to the other:

ev3 scp

To run python files on the brick, you need to make them executable. Right-click on a Python file and select the File permissions… option:

permissions

then set the execute file permission as required:

Change_file_attributes

 

See also: Running Python Programmes on the Lego EV3 via the EV3 File Browser for a command line/scp route for file transfer, and Setting Up PyCharm To Work With a Lego Mindstorms EV3 Brick for a guide on file synching via git checkins using the PyCharm Community Edition editor / IDE.

Setting Up PyCharm To Work With a Lego Mindstorms EV3 Brick

Notes based on Setting Up a Python Development Environment with PyCharm for setting up PyCharm editor (I use the free Community Edition) to work with EV3. Requires passwordless ssh into the brick and the brick on 192.168.1.106.

We’re going to go round the houses with git checkins to move stuff from the Mac and the PyCharm editor to the brick.

Get into the brick and do some minimal config stuff:

[Mac] ssh robot@192.168.1.106

[EV3] sudo apt-get update
[EV3] sudo apt-get install git
[EV3] mkdir -p /home/robot/demoproj
[EV3] mkdir -p /home/robot/demoproj/.demoproj.git
[EV3] git init --bare /home/robot/demoproj/.demoproj.git

Now use the nano editor on the brick to populate the demoproj dir with the files we push, setting them to be executable.

[EV3] nano /home/robot/demoproj/.demoproj.git/hooks/post-receive

In the nano editor, change the file to:

#!/bin/sh
git --work-tree=/home/robot/demoproj --git-dir=/home/robot/demoproj/.demoproj.git checkout -f
find /home/robot -iname \*.py | xargs chmod +x

The chmod on the py files makes them executable, so as long as you hashbang the first line of any python files (with #!/usr/bin/python), they should be runnable from the file browser menu on the brick.

Then ctrl-x to exit, saying Yes to save the file on the way out and accepting the default file name. Now make that hook executable:

[EV3] chmod +x /home/robot/demoproj/.demoproj.git/hooks/post-receive

Should we prepend hashbangs as well? This will add one to start of all py files not containing one:

grep -rL '^#!/usr/bin/python$' /home/robot/demoproj/*.py | xargs sed -i '1i #!/usr/bin/python'

In PyCharm, VCS->Check Out From Version Control to create a new project, selecting git as a the checkout target (so you’ll also need git installed on the Mac…).

The VCS Repository URL is: robot@192.168.1.106:/home/robot/demoproj/.demoproj, the Parent Directory (for example, /Users/me/projects) and Directory names (e.g. testProj – note, this must be a new folder) specifying the location on the Mac where you want to keep a local copy of the project files.

Say yes to all the crap PyCharm wants to create, and Yes when it prompts if you want to open the newly created directory. Create a new python file containing the following test program:

#!/usr/bin/python
#The shebang above runs this file with the python shell
from ev3dev.auto import Sound
#Make a short sound
Sound.tone(1000,10)

Save it, press the VCS /UP ARROW/ button top right in PyCharm to commit, add a commit message, then bottom right Commit and Push.

This should commit the file locally and also push it into the git repo on the ev3; the commit hook on the ev3 will copy the file into the demoproj folder and, if it’s a .py file, mark it as executable.

You should now be able to locate it and run it from the ev3dev file browser.

See also: Using IPython on Lego EV3 Robots Running Ev3Dev and Running Python Programmes on the Lego EV3 via the EV3 File Browser.

PS to enable autocompletion in PyCharm, check what Python shell the project is using (in the settings/preferences somehow though I’m damned if I know where to find them; I HATE IDEs with a passion – way to cluttered and over complex…). Then using the same Python shell:

git clone https://github.com/rhempel/ev3dev-lang-python.git
cd ev3dev-lang-python
python setup.py install
cd ..
rm -r ev3dev-lang-python/

Run the programme in PyCharm (or at least, the import lines) and autocompletion should be enabled. The complete code won’t run properly though, because you’re not running it in an ev3dev environment…

Running Python Programmes on the Lego EV3 via the EV3 File Browser

A quick note to self… If we test python scripts on an EV3 brick interactively via a Jupyter notebook and then save the notebook as a python file (eg Untitled1.py), we can get the Python file over to the EV3 brick and into a state we can run it from the file browser by using the following recipe:

echo '#!/usr/bin/python'|cat - Untitled1.py > /tmp/out && mv /tmp/out Untitled1.py
scp Untitled1.py robot@192.168.1.106:/home/robot/
ssh robot@192.168.1.106 'chmod ugo+x /home/robot/Untitled1.py'

Then in the file browser on the brick, just click on Untitled1.py to run it.

It would be easy enough to script this so a single button in the notebook does it automatically? (In which case, it would probably make more sense to modify the template used to generate the Python save file so that it includes the shebang at the start of the file anyway?)

PS see also Setting Up PyCharm To Work With a Lego Mindstorms EV3 Brick and Sharing Files With a Lego EV3 Brick Over Wifi Using Filezilla.

From StringLE to DockLE… Erm… Maybe…

Getting on for a decade ago, I doodled an approach for creating user a user defined personal learning environment based around online applications and resources: StringLe, the string’n’glue learning environment (more and more and more).

stringle2

StringLe was of the web. A StringLE environment configuration comprised three things:

  • the username and tag for a particular delicious social bookmark feed that contained a list of web applications that were linked from the top menu bar of a StringLE set up; tools included things like online office tools, drawing packages, and audio and image editors
  • a link to an OPML feed: this feed populated a Grazr widget in the left hand sidebar that could be used to load in further RSS feeds to act as more comprehensive nested menu options that could be used to pull resources into the environment.
  • a homepage URL for the environment.

The idea was simple – through managing various RSS feeds, the user could create a workspace configured to load in a variety of tools (web applications) arranged along a top tabbed menu bar and online content pages accessed via an OPML navigation widget and inpage RSS viewer widget (actually, the OPML and RSS viewer widgets were one and the same: Grazr? Remember Grazr? I went through phase of packageing MIT courses as OPML files for rendering in Grazr eg here and here), as well as using the environment as a normal web browser, and bookmarking links visited within the space into one of the feeds exposed via the navigational OPML feed.

It was a throwaway toy, but one that demonstrated a way of pulling a range of separate applications together that might be used within a particular course or course activity. For different courses or activities, it was easy enough to pull together different collections of tools for that particular purpose.

This notion of online tool collections – workbenches, perhaps? or tool suites? – is used, in part, in things like SagemathCloud, or the IBM Data Workbench, where a user is provided access to a range of different online applications within a single online environment.

Those workbenches go much further, however, and provide an online workspace in the form of a user filestore that can be accessed by all the applications within the suite. Services like Sandstorm also resemble StringLE in the sense that they provide an environment from within which a user can launch a range of user selected online applications.

For anyone whose followed this blog for any period of time, you’ll probably know the same ideas just keep getting revisited and recycled. So now, for example, it strikes me that a Docker Compose file provides another way of defining a user configured workbench comprising multiple linked applications, along with a workspace in the sense of a shared file area that can be used to pass files between the applications.

So for example, here’s an example from earlier this year of the Docker Compose script I used to pull together a set of containers to re-implement the TM351 monolithic VM as a set of linked containers (raw dockerfiles on Github).

What might me quite nice would be to add a Flask container to the mix to set up a simple webserver, and generate a simple HTML file from the dockerfile to act as a floating menu panel that has a set of buttons, one for each application, that would let me click through to that application. A simple wrapper script would then: take the docker-compose file, add a flask control panel container to it, create the menu HTML from the docker-compose file, run docker-compose over the docker-compose file to launch the containers, pop the HTML menu up in a browser window.

Anyway… suffice to say that the app collection idea reminded me of StringLE, but  reimplemented via a Docker Compose script – as a DockLE?! – and using applications running in linked containers, rather than more loosely connected web apps identified via RSS and OPML feeds.

Using IPython on Lego EV3 Robots Running Ev3Dev

In part so I don’t lose the recipe, here are some notes for getting up and running with IPython on a Lego EV3 brick.

The command lines are prefixed to show whether we’re running them on the Mac or the brick…

To start with, we need to flash a microSD card with an image of the ev3dev operating system. The instructions are on the ev3dev site – Writing an SD Card Image Using Command Line Tools on OS X. – , but I repeat the ones for a Mac here for convenience.

  1. Download an image from the repository – I used the ev3dev-jessie-2015-12-30 image because the current development version is in a state of flux and the python bindings don’t necessarily work with it just at the moment…
  2. Assuming you have downloaded the file to your home Downloads directory (that is, ~/Downloads), launch a new terminal and run: cd Downloads
    [Mac] unzip ev3-ev3dev-jessie-2015-12-30.img.zip
    [Mac] diskutil list
  3. Put the microSD card (at least 2GB, but no more than 16GB, I think? I used a 4GB microSD (HC) card) into an SD card adapter and pop it into the Mac SD card reader. Check that it’s there and what it’s called; (you’re looking for the new disk…):
    [Mac] diskutil list
  4. Now unmount the new disk corresponding the the SD card: diskutil unmountDisk /dev/disk1s1Downloads_—_robot_ev3dev____—_ssh_—_112×25
    If you don’t see the /dev/desk1 listing, check that the write protect slider on your SD card holder isn’t in the write protect position.
  5. We’re going to write some raw bits to the card (/dev/disk1 in my example), so we need to write to /dev/rdisk1 (note the r before the disk name). The write will take some time – 5 minutes or so – but if you’re twitchy, ctrl-T will show you a progress message). You’ll also need to enter your Mac password. (NOTE: if you use the wrong disk name, you can completely trash your computer. So be careful;-)
    [Mac] sudo dd if=~Downloads/ev3-ev3dev-jessie-2015-12-30.img of=/dev/rdisk1 bs=4m
    GOTCHA: when I tried, I got a Permission Denied message at first. It seems that for some reason my Mac thought the SD card was write protected. On the SD card adapter is a small slider that sets the card to “locked” or “unlocked”. The SD card reader in the Mac is a mechanical switch that detects which state the slider is in and whether the card is locked. I don’t know if it was a problem with the card adapter or the Mac reader, but I took the card reader out of the Mac, changed the slider setting, put card reader back in, and did the unmount and then sudo dd steps again. It still didn’t work, so I took the card out again, put the slider back the other way, and tired again. This time it did work.
  6. Once the copy is complete, take the SD card adapter out, remove the microSD car and put in in the EV3 brick. Start the EV3 brick in the normal way.

Hopefully, you should the brick boot into the Brickman UI (it may take two or three minutes, include a period when the screen is blank and the orange lights are ticking for a bit…)

Navigate the brick settings to the Networks menu, select Wifi and scan for your home network. Connect to the network (the password settings will be saved, so you shouldn’t have to enter them again).

By default, the IP address of the brick should be set to 192.168.1.106. To make life easier, I set up passwordless ssh access to the brick. Using some guidance in part originally from here, I accessed the brick from my Mac terminal and set up an ssh folder. First, log in to the brick from the Mac terminal:

[Mac] ssh robot@192.168.1.106

When prompted, the password for the robot user is maker.

Downloads_—_robot_ev3dev____—_bash_—_112×25

This should log you in to the brick. Run the following command to create an ssh directory into which the shh key will be placed, and then exit the brick commandline to go back to the Mac command line.

[Brick] install -d -m 700 ~/.ssh
[Brick] exit

Create a new key on your Mac, and then copy it over to the brick:

[Mac] ssh-keygen -R 192.168.1.106
[Mac] cat ~/.ssh/id_rsa.pub | ssh robot@192.168.1.106 'cat > .ssh/authorized_keys'
You will be prompted again for the password – maker – but next time you try to log in to the brick, you should be able to do so without having to enter the password. (Instead, the ssh key will be used to get you in.)

Downloads_—_robot_ev3dev____—_bash_—_112×25_and_Edit_Post_‹_OUseful_Info__the_blog____—_WordPress

If you login to the brick again – from the Mac commandline, run:

[Mac] ssh robot@192.168.1.106

you should be able to run a simple Python test program. Attach a large motor to input A on the brick. From the brick commandline, run:

[Brick] python

to open up a Python command prompt, and then enter the following commands to use the preinstalled ev3dev-lang-python Python bindings to run the motor for a few seconds:

[Python] import ev3dev.ev3 as ev3
[Python] m = ev3.LargeMotor('outA')
[Python] m.run_timed(time_sp=3000, duty_cycle_sp=75)

Enter:

[Python] exit

to exit from the Python interpreter.

Now we’re going to install IPython. Run the following commands on the brick command line (update, but DO NOT upgrade the apt packages). If prompted for a password, it’s still maker:

[Brick] sudo apt-get update
[Brick] sudo apt-get install -y ipython

You should now be able to start an IPython interpreter on the brick:

[Brick] ipython

The Python code to test the motor should still work (hit return it you find you are stuck in a code block). Typing:

[Brick] exit

will take you out of the interpreter and back to the brick command line.

One of the nice things about IPython is that we can connect to it remotely. What this means is that I can start an IPython editor on my Mac, but connect it to an IPython process running on the brick. To do this, we need to install another package:

[Brick] sudo apt-get install -y python-zmq

Now you should be able to start an IPython process on the brick that we can connect to from the Mac:

[Brick] ipython kernel

The process will start running and you should see a message of the form:

To connect another client to this kernel, use:
--existing kernel-2716.json

This file contains connections information about the kernel.

Now open another terminal on the Mac, (cmd-T in the terminal window should open a new Terminal tab) and let’s see if we can find where that file is. Actually, here’s a crib – in the second terminal window, go into the brick:

[Mac] ssh robot@192.168.1.106

And then on the brick command line in this second terminal window, show a listing of a the directory that should contain the connection file:

[Brick] sudo ls /home/robot/.ipython/profile_default/security/

You should see the –existing kernel-2716.json file (or whatever it’s exactly called) there. Exit the brick command line:

[Brick] exit

Still in the second terminal window, and now back on the Mac command, copy the connection file from the brick to your current directory on the Mac:

[Mac] scp robot@192.168.1.106:/home/robot/.ipython/profile_default/security/kernel-2716.json ./

If you have IPython installed on you Mac, you should now be able to open an IPython interactive terminal on the Mac that is connected to the IPython kernel running on the brick:

[Mac] ipython console --existing ${PWD}/kernel-2716.json --ssh robot@192.168.1.106

You should be able to execute the Python motor test command as before (remember to import the necessary ev3dev.ev3 package first).

Actually, when I ran the ipython console command, the recent version of jupyter on my Mac gave me a depreaction warning which means I would have been better running:

[Mac] jupyter console --existing ${PWD}/kernel-2716.json --ssh robot@192.168.1.106

So far so good – can we do any more with this?

Well, yes, a couple of things. When starting the IPython process on the brick, we could force the name and location of the connection file:

[Mac] ipython kernel -f /home/robot/connection-file.json

Then on the Mac we could directly copy over the connection file:

[Mac] scp robot@192.168.1.106:/home/robot/connection-file.json ./

Secondly, we can configure a Jupyter notebook server running on the Mac to so that it will create a new IPython process on the brick for each new notebook.

Whilst you can configure this yourself, it’s possibly easy to make use of the remote_ikernel helper:

[Mac] pip3 install remote_ikernel
[Mac] remote_ikernel manage --add --kernel_cmd="ipython kernel -f {connection_file}" --name="Ev3dev" --interface=ssh --host=robot@192.168.1.106

Now when you should be able to connect to a notebook run against an IPython kernel on the brick.

Home-jupyter

Note that it may take a few seconds for the kernel to connect and the first cell to run – but from then on it should be quite responsive.

Untitled2-ipynb

To show a list of available kernels for a particular jupyter server, run the following in a Jupyter code cell:

import jupyter_client
jupyter_client.kernelspec.find_kernel_specs()

PS for ad hoc use, I thought it might be useful to try setting up a quick wifi router that I could use to connect the brick to my laptop in the form of an old Samsung Galaxy S3 android phone (without a SIM card). Installing the Hotspot Control app provided a quick way of doing this… and it worked:-)

PPS for a more recent version of IPython, install it from pip.

If you installed IPython using the apt-get route, uninstall it with:

[Brick] sudo apt-get uninstall ipython

Install pip and some handy supporting tools that pip may well require at some point:

[Brick] sudo apt-get install build-essential python-dev

Running:

[Brick] sudo apt-get install python-pip

would run an old version of pippip --version shows 1.5.6 – which could be removed using sudo apt-get remove python-setuptools.

To grab a more recent version, use:

[Brick] wget https://bootstrap.pypa.io/get-pip.py
[Brick] sudo -H python get-pip.py

which seems to take a long time to run with no obvious sign of progress, and then tidy up:

[Brick] rm get-pip.py

Just to be sure, then update it:

[Brick] sudo pip install --upgrade setuptools pip

which also seems to take forever. Then install IPython:

[Brick] sudo pip install -y ipython

I’m also going to see if I can give IPythonwidgets a go, although the install requirements looks as if it’ll bring down a huge chunk of the Jupyter framework too, and I’m not sure that’s all necessary?

[Brick] sudo pip install ipywidgets

For a lighter install, try sudo pip install --no-deps ipywidgets to install ipywidgets without dependencies. The only required dependencies are ipython>=4.0.0, ipykernel>=4.2.2 and traitlets>=4.2.0;.

The widgets didn’t seem to work at first, but it seems that following a recent update to the Jupyter server on host, it needed a config kick before running jupyter notebook:

jupyter nbextension enable --py --sys-prefix widgetsnbextension

PPS It seems to take a bit of time for the connection to the IPython server to be set up:

ajh59_—_Python_—_123×24

The Timeout occurs quite quickly, but then I have to wait a few dozen seconds for the tunnels on ports to be set up. Once this is done, you should be able to run things in a code cell. I usually start with print("Hello World!") ;-)

PPPS For plotting charts:

sudo apt-get install -y python-matplotlib

Could maybe even go the whole hog…

sudo apt-get install -y python-pandas
sudo pip install seaborn

PPPPS Here’s my current build file (under testing atm – it takes about an hour or so…) – ev3_ipy_build.sh, so:
[Mac] scp ev3_ipy_build.sh robot@192.168.1.106
[Brick] chmod +x ev3_ipy_build.sh
[Brick] sudo ./ev3_ipy_build.sh

sudo apt-get update
sudo apt-get install -y build-essential python-dev
sudo apt-get install -y python-zmq python-matplotlib python-pandas

wget https://bootstrap.pypa.io/get-pip.py
sudo -H python get-pip.py
rm get-pip.py
sudo pip install --upgrade setuptools pip

sudo pip install ipython ipykernel traitlets seaborn
sudo pip install --no-deps ipywidgets

PPPPPS to clone the SD card on a Mac, insert the SD card and run:

[Mac] diskutil list
[Mac] sudo dd if=/dev/disk1 of=~/Desktop/my_ev3dev_image.dmg

The corresponding restore (in the process described at the start of this post) would use /my_ev3dev_image.dmg rather than the ev3-ev3dev-jessie-2015-12-30.img image.

PPPPPPS Connecting to remote kernel on brick – start IPyhton kernel on brick:

[Brick] ipython kernel -f /home/robot/test.json

Copy the connection file over to the host:
[Mac] scp robot@192.168.1.106:/home/robot/test.json ./

Check the path you copied it to
[Mac] pwd

For me, that returned /Users/ajh59.

Start a console on the host using the existing connection file – use a full, explicit path to the file. Also works with things like Spyder:

[Mac] jupyter console --existing /Users/ajh59/test.json --ssh robot@192.168.1.106

spyder-ssh-ev3