With OUr data management and analysis course coming up to its third presentation start in October, various revisions and updates are currently being made to the materials, in part based on feedback from students, in part based the module team’s reflections on how the course material is performing.
We also have an opportunity to update the virtual machine supplied to students, so I’ve spent the last couple of days poking around in the various script rewrites I’ve toyed with over the last couple of years. When we started the course, Jupyter notebooks were still called IPython notebooks, and the ecosystem was still in its infancy. But whilst the module review process means changes are supposed to be kept to a minimum, there is still an opportunity to bake a few more tools into the VM that didn’t exist a couple of years ago when the VM was first gold mastered. (I’ll do a review of some of the Jupyter notebook features that I think should be released into the VM in another post.)
When the VM was first put together, I took it as an opportunity to explore automated build processes. The VM itself was built from Puppet scripts orchestrated from Vagrant, with another Vagrant script managing the machine we delivered to students (setting up shared folders, handling port forwarding, and giving the internal services a kick if required). I also explored a dockerised version, but Docker too was still in its infancy when we first looked at how to best virtualise the services and apps distributed as part of the course materials (IPython/Jupyter notebooks, PostgreSQL, MongoDB and OpenRefine). With Docker now having native versions for recent Macs and Windows platforms, I thought it might be worth exploring again; but OUr student computing policy means we have to build to lowest common denominator machines that are years old (though I’m ignoring the 32 bit hardware platform constraint and we’ll post an online workaround – or ship a Raspberry Pi version of the VM – if we have to!).
So… to demo where I’m at in terms of process, and keep a note to myself, the build has forsaken Puppet and I’ve gone back to simple shell scripts. As an example of most of the tricks I’ve had to invoke, I’ll post recipes for getting OpenRefine up and running on several virtual hosts in several different ways. Still to do is a dockerised version and and RPi version of the TM351 VM config, but I’m hoping the shell scripts will all be reusable (and if not, I’ll try to tweak them so they work as is as part of whatever build process is required…
To begin with, the builder shell scripts are as follows (.sh files all end up requiring execute permissions granted somehow…).
Structure is:
./quickbuild/quick_build.sh ./quickbuild/basepackages.sh ./quickbuild/openrefine/openrefine.sh ./quickbuild/openrefine/services/refine.service
The main build script calls a script to add in base packages, and scripts for each application (in their own folder). I really should have had the same invocation filename or filename pattern (e.g. reusing the directory name) in each build folder.
## ./quickbuild/quick_build.sh #chmod ugo+x on this file #!/usr/bin/env bash #Set the base build directory to the one containing this script THISDIR=$(dirname "$0") chmod ugo+x $THISDIR/basepackages.sh chmod ugo+x $THISDIR/openrefine/openrefine.sh #Build script for building machine $THISDIR/basepackages.sh $THISDIR/openrefine/openrefine.sh #tidy up apt-get autoremove -y && apt-get clean && updatedb
The base packages script does some updating of package lists and then pulls in a range of essential utility packages, some of which are actually required for builds further down the line.
## ./quickbuild/basepackages.sh #!/usr/bin/env bash #Build script for building machine apt-get clean && apt-get -y update && apt-get -y upgrade && apt-get install -y bash-completion vim curl zip unzip bzip2 && apt-get install -y build-essential gcc && apt-get install -y g++ gfortran && apt-get install -y libatlas-base-dev libfreetype6-dev libpng-dev libhdf5-serial-dev && apt-get install -y git python3 python3-dev python3-pip && pip3 install --upgrade pip
The application build files install additional packages specific to the application or its build process. We had some issues with service starts in the original VM (Ubuntu 14.04 LTS), but the service management in Ubuntu 16.04 LTS is much cleaner – and in my own testing so far, much more reliable.
# ./quickbuild/openrefine/openrefine.sh
#!/bin/bash
THISDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
apt-get -y update && apt-get install -y wget ant unzip openjdk-8-jre-headless && apt-get clean -y
echo "Setting up OpenRefine: "
#Prep for download
mkdir -p /opt
mkdir -p /root
if [ ! -f /opt/openrefine.done ]; then
echo "Downloading OpenRefine..."
wget -q --no-check-certificate -P /root https://github.com/OpenRefine/OpenRefine/releases/download/2.7-rc.2/openrefine-linux-2.7-rc.2.tar.gz
echo "...downloaded OpenRefine"
echo "Unpacking OpenRefine..."
tar -xzf /root/openrefine-linux-2.7-rc.2.tar.gz -C /opt && rm /root/openrefine-linux-2.7-rc.2.tar.gz
#Unpacks to: /opt/openrefine-2.7-rc.2
touch /opt/openrefine.done
echo "...unpacked OpenRefine"
else
echo "...already downloaded and unpacked OpenRefine"
fi
cp $THISDIR/services/refine.service /lib/systemd/system/refine.service
# Enable autostart
sudo systemctl enable refine.service
# Refresh service config
sudo systemctl daemon-reload
#(Re)start service
sudo systemctl restart refine.service
Applications are run as services, where possible. If I get a chance – and space/resource requirements allow – I made add some service monitoring to try to ensure application services are always running when the VM is running.
## ./quickbuild/openrefine/services/refine.service [Unit] Description=Refine #When to bring the service up #via https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/ #Wait for a network stack to appear After=network.target #If we actually need the network to have a routable IP address: #After=network-online.target [Service] Environment=REFINE_HOST=0.0.0.0 ExecStart=/opt/openrefine-2.7-rc.2/refine -p 3334 -d /vagrant/openrefine_projects Restart=always RestartSec=10 [Install] WantedBy=multi-user.target
Everything can be packaged up in a zip file with a command (tuned to omit Mac cruft, in part) of the form:
zip -r quickbuild.zip quickbuild -x *.vagrant* -x *.DS_Store -x *.git* -x *.ipynb_checkpoints*
So those are the files and the basic outline. Our initial plan is to run the VMs once again locally on a student’s own machine, using Virtualbox. I think we’ll stick with vagrant to manage this, not least because we can issue updates via new Vagrantfiles, not that we’ve done that to date…
By the by, I’m running vagrant with a handful of plugins:
#Speed up repeated builds vagrant plugin install vagrant-cachier #Use correct Virtualbox Guest Additions vagrant plugin install vagrant-vbguest #Help with provisioning to virtual hosts vagrant plugin install vagrant-digitalocean vagrant plugin install vagrant-linode vagrant plugin install vagrant-aws
The following Vagrantfile builds the local Virtualbox instance by default. To build to DOgital Ocean or Linode, use the following:
vagrant up --provider=digital_oceanvagrant up --provider=linode
I didn’t get the AWS vagrant provisioner to work (too many things to go wrong in terms of settings!)
The Linode build also required a hack to get the box to build correctly…
# ./quickbuild/Vagrantfile
#Vagrantfile for building machine from build scripts
Vagrant.configure("2") do |config|
#------------------------- PROVIDER: VIRTUALBOX (BUILD) ------------------------------
config.vm.provider :virtualbox do |virtualbox|
#ubuntu/xenial bug? https://bugs.launchpad.net/cloud-images/+bug/1569237
config.vm.box = "bento/ubuntu-16.04"
#Stick with the default key
config.ssh.insert_key=false
#For local testing:
#config.vm.box = "tm351basebuild"
#override.vm.box_url = "eg URL on dropbox"
#config.vm.box_url = "../boxes/test.box"
config.vm.hostname = "tm351base"
virtualbox.name = "tm351basebuildbuild"
#We need the memory to install scipy and build indexes on seeded mongodb
#After the build it can be reduced back down to 1024
virtualbox.memory = 2048
#virtualbox.cpus = 1
# virtualbox.gui = true
#---- START PORT FORWARDING ----
#Registered ports: https://en.wikipedia.org/wiki/List_of_TCP_and_UDP_port_numbers
#openrefine
config.vm.network :forwarded_port, guest: 3334, host: 35101, auto_correct: true
#---- END PORT FORWARDING ----
end
#------------------------- END PROVIDER: VIRTUALBOX (BUILD) ------------------------------
#------------------------- PROVIDER: DIGITAL OCEAN ------------------------------
config.vm.provider :digital_ocean do |provider, override|
override.ssh.insert_key=true
override.ssh.private_key_path = '~/.ssh/id_rsa'
override.vm.box = 'digital_ocean'
override.vm.box_url = "https://github.com/devopsgroup-io/vagrant-digitalocean/raw/master/box/digital_ocean.box"
provider.token = 'YOUR_TOKEN'
provider.image = 'ubuntu-16-04-x64'
provider.region = 'lon1'
provider.size = '2gb'
end
#------------------------- END PROVIDER: DIGITAL OCEAN ------------------------------
#------------------------- PROVIDER: LINODE ------------------------------
config.vm.provider :linode do |provider, override|
override.ssh.insert_key=true
override.ssh.private_key_path = '~/.ssh/id_rsa'
override.vm.box = 'linode/ubuntu1604'
provider.api_key = 'YOUR KEY'
provider.distribution = 'Ubuntu 16.04 LTS'
provider.datacenter = 'london'
provider.plan = 'Linode 2048'
provider.size=2048
#grub needs updating - but want's to do it interactively
#this bit of voodoo from Stack Overflow hacks a non-interactive install of it
override.vm.provision :shell, :inline => <<-SH
apt-get update && DEBIAN_FRONTEND=noninteractive apt-get -y -o DPkg::options::="--force-confdef" -o DPkg::options::="--force-confold" install grub-pc
SH
end
#------------------------- END PROVIDER: LINODE ------------------------------
#------------------------- PROVIDER: AWS ------------------------------
# I DIDN'T GET THIS TO WORK - MAYBE SEVERAL THINGS WRONG HERE - AND IN AWS SETTINGS ????
config.vm.provider :aws do |aws, override|
config.vm.hostname = "tm351aws"
#vagrant box add dummy https://github.com/mitchellh/vagrant-aws/raw/master/dummy.box
override.vm.box = "dummy"
aws.access_key_id = ""
aws.secret_access_key = ""
#https://github.com/mitchellh/vagrant-aws/issues/405#issuecomment-130342371
#Download and install the Amazon Command Line Interface
#http://docs.aws.amazon.com/cli/latest/userguide/installing.html
#Configure the command line interface
#http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html
#$aws configure
#Request the session token
#$aws sts get-session-token --duration-seconds 129600 (enter your own duration)
aws.session_token = ""
#Keypair also generated via AWS console?
aws.keypair_name = "vagrantAWSkeypair"
aws.region = "eu-west-2a"
aws.ami = "ami-ed908589"
aws.instance_type="t2.small"
override.ssh.username = "ubuntu"
override.ssh.private_key_path = '~/.ssh/id_rsa'
end
# NOTE THAT RUNNING THIS PROVISIONER MAY LEAVE THINGS BILL INCURRING ON AWS... SO CHECK
#------------------------- END PROVIDER: AWS ------------------------------
#------------------------------
config.vm.provision :shell, :inline => <<-SH
#Add build scripts here
cd /vagrant/build
source ./quick_build.sh
SH
end
(The vagrant script can be tidied to hide keys by setting eg export DIGITAL_OCEAN_TOKEN="YOUR TOKEN HERE" from the command line you call vagrant from, and in the Vagrantfile setting provider.token = ENV['DIGITAL_OCEAN_TOKEN']).)
One of the nice things about the current version of vagrant is that you have to destroy a machine before launching another one of the same name with a different provisioners (though this looks set to change in forthcoming versions of vagrant). Why nice? Because the vagrant destroy command kills the node the machine is running on – so it won’t be left running and you won’t forget to turn it off (and won’t keep the meter running….)
Firing up the boxes on various hosts, go to port 3334 at the appropriate IP address and you should see OpenRefine running there…
Having failed to get the machine up and running on AWS, I though I’d try the simple route of packaging an AMI using Packer.
The build script was remarkably simple – once I got one that worked!
#awspacker.json
{
"variables": {
"aws_access_key": "",
"aws_secret_key": ""
},
"builders": [{
"type": "amazon-ebs",
"access_key": "{{user `aws_access_key`}}",
"secret_key": "{{user `aws_secret_key`}}",
"region": "eu-west-1",
"source_ami": "ami-971238f1",
"instance_type": "t2.micro",
"ssh_username": "ubuntu",
"ami_name": "openrefine",
"security_group_id": "OPTIONAL_YOUR_VAGRANT_GROUP"
}],
"provisioners": [
{
"destination": "/tmp/",
"source": "./toupload/",
"type": "file"
},
{
"inline": [
"cd /tmp && sudo apt-get update && sudo apt-get install unzip && sudo unzip /tmp/quickbuild.zip -d /tmp && sudo chmod ugo+x /tmp/quickbuild/quick_build.sh && sudo /tmp/quickbuild/quick_build.sh "
],
"type": "shell"
}
]
}
(The eu-west-2 (London) region wasn’t recognised by Packer for some reason…)
The machine can now be built on AWS and packaged as an AMI using Packer as follows (top level security tokens can be generated from the AWS Security Credentials console):
#Package the build files mkdir -p toupload && zip -r toupload/quickbuild.zip quickbuild -x *.vagrant* -x *.DS_Store -x *.git* -x *.ipynb_checkpoints* #Pack the machine packer build -var 'aws_access_key=YOUR_KEY' -var 'aws_secret_key=YOUR_SECRET' awspacker.json
Launching an instance of this AMI, I found that I couldn’t connect to the OpenRefine port (it just hung). The fix was to amend the automatically created security group rules (which by default just allow ssh on port 22) with a a Custom TCP rule that allowed incoming traffic on port 3334 from All Domains.
Which meant success:
To simplify matters, I then copied this edited security group to my own “openrefine” security group that I could use as the basis of the AMI packaging.
Just one thing to note about creating an AMI – Amazon will start billing you for it… As the Packer Getting Started guide suggests:
After running the above example, your AWS account now has an AMI associated with it. AMIs are stored in S3 by Amazon, so unless you want to be charged about $0.01 per month, you’ll probably want to remove it. Remove the AMI by first deregistering it on the AWS AMI management page. Next, delete the associated snapshot on the AWS snapshot management page.
Next up, I need to try a full build of the TM351 VM on AWS (a full build without the Mongo shard activity (which I couldn’t get to work yesterday – though this looks like it could provide a handy helper script (and I maybe also need to work through this.) The fuller build seems fine from the vagrant script in Virtualbox, Digital Ocean and Linode.
After that (and fixing the Mongo sharding thing), I’ll see if I can weave the build scripts into a set of interconnected Docker containers, one Dockerfile per application and a docker-compose.yml to weave them together. (See the original test from way back when.)
And then there’ll just be the look-see to see whether we can get the machine built and running on a Raspberry Pi 3 model B.
I also started wondering about whether I should pop a simple Flask app into the VM on port 80, showing an OU splash screen and a “Welcome to TM351” message… If I can get that running, then we have a means of piping stuff into a web page on the students’ own machines that is completely out of the controlling hands of LTS:-)
PS for an example of how to set up authentication over these services, see: Simple Authenticated Access to VM Services Using NGINX and Vagrant Port Forwarding.
OUseful.Info, the blog... 