Docker machine is a new project launched by Docker last year. The goal is to launch instances/servers with docker installed and configure the clients to talk them. Most of the documentation is available at:
https://docs.docker.com/machine/
In the last blog about AWS ECS service on-premises with Eucalyptus we highlighted the need for a container instance. In that example we used CoreOS and along with a config file were able to quickly setup an instance on Eucalyptus cloud with docker installed and ECS agent running.
In this blog we make it easy for folks to create instances on Eucalyptus with docker installed and configured to talk to a docker client. We use Ubuntu Trusty as our preferred choice of operating system on the instance.
Before we really got going we tried to find more about the machine.
- Machine is written in Go lang.
- It appears to be actively developed and might have lot of moving parts in coming few weeks/months.
- Machine has the concept of drivers. It has drivers for all major cloud platforms like AWS, Openstack etc. as well as Virtualbox and other virtualization platforms.
Our focus was to work on the AWS driver and make it work against an AWS compatible Eucalyptus cloud. Here are some thoughts on the current driver:
- It is not using any 3rd party library except for Signature version 4. Ideally we had hope they would use AWS SDK for Go but looks like they went for a very minimal thing in the current code.
- It only uses AWS EC2,EBS and VPC. The driver is available at https://github.com/docker/machine/tree/master/drivers/amazonec2 Support for AWS VPC API is not available with Eucalyptus (only tech-preview).
- It has support for spot instances that can be enabled if one wants. Again spot instances API not available for Eucalyptus.
- It has a very weird way of constructing the Endpoint for EC2. It is mostly hard coded and takes the region parameter to construct it here https://github.com/docker/machine/blob/master/drivers/amazonec2/amz/ec2.go#L141
- AWS Driver uses EBS volume types (gp2) and Eucalyptus does not implement that yet.
- AWS Driver uses root device as "/dev/sda1" for the EBS volume. Here https://github.com/docker/machine/blob/master/drivers/amazonec2/amazonec2.go#L351 and it seems to work just fine on AWS but does not work on Eucalyptus.
In order to get going with Eucalyptus and Docker machine we had to do the following, note that the goal was to get a POC setup going with minimal features:
- Write a new driver for Eucalyptus. Basically copying the AWS driver for the time being (we expect AWS driver to change a lot) and modifying the necessary bits. Ideally we had want the AWS driver to work with Eucalyptus clouds.
Following things were done in the Eucalyptus driver:
- We removed support for EBS volume types
- We removed support for block device mapping (it appears root device cannot be a partition like /dev/sda1 on Eucalyptus it has to be a raw disk)
- We removed support for spot instances (not implemented)
- We removed support for VPC (tech-preview)
- We modified the code here, making sure that machine checks for the IP address only after it had make sure the instance is running. Without this change we had issues where machine picked up the private IP of the instance because the instance just got launched on the cloud and Eucalyptus takes some time to get a public IP on it. On AWS we did not see any problems.
- We added new command line parameter to machine that accept the EC2 Endpoint so one can specify which Eucalyptus compute endpoint they would like to point Machine at.
The driver is available here https://github.com/jeevanullas/machine/tree/eucaprovider
In order to compile the binaries one can fork/clone the branch to a local account/system and run script/build from within the source directory. The build script pulls in necessary Go dependencies and generates the binary in the source directory
Copy the binaries to somewhere like /usr/local/bin/ so they are generally available.
Test run of Docker machine on Eucalyptus:
These were the parameters we passed to Docker machine:
$ docker-machine_linux-amd64 -D create \
--driver eucalyptus \
--eucalyptus-emi <your EMI ID> \
--eucalyptus-access-key <your cloud access key> \
--eucalyptus-secret-key <your cloud secret key> \
--eucalyptus-zone <your AZ name> \
--eucalyptus-region eucalyptus \
--eucalyptus-instance-type m1.large \
--eucalyptus-compute-endpoint "EC2_URL of your Eucalyptus" \
docker-host-euca-jeevan
Once this is done you could see the nice debug output (thanks to -D flag)
Launching instance...
creating key pair: docker-host-euca-jeevan
configuring security group
creating security group (docker-machine)
waiting for group (sg-a00aa116) to become available
configuring security group authorization for 0.0.0.0/0
authorizing group with permissions: [{tcp 22 22 0.0.0.0/0} {tcp 2376 2376 0.0.0.0/0}]
waiting for ip address to become available
Got the IP Address, it's "192.168.1.2"
created instance ID i-195071cd, IP address 192.168.1.2, Private IP address
Settings tags for instance
Getting to WaitForSSH function...
generating server cert: /home/jeevanullas/.docker/machine/machines/docker-host-euca-jeevan/server.pem ca-key=/home/jeevanullas/.docker/machine/certs/ca.pem private-key=/home/jeevanullas/.docker/machine/certs/ca-key.pem org=docker-host-euca-jeevan
As can be seen Machine created a security group call docker-machine on Eucalyptus and authorize port 22/TCP and 2376/TCP to that security group.
2376 is the port that docker daemon listens on. It also created a keypair with the name docker-host-euca-jeevan and copied the private key for this keypair at ~/.docker/machine/machines/docker-host-euca-jeevan/id_rsa if one fancy logging into the instance directly and seeing what is going on.
The machine also generates certificates that are used for TLS based communication between the docker client and the docker daemon.
We took a peak inside the instance and found the following:
root@docker-host-euca-jeevan:~# docker version
Client version: 1.6.0
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): 4749651
OS/Arch (client): linux/amd64
Server version: 1.6.0
Server API version: 1.18
Go version (server): go1.4.2
Git commit (server): 4749651
OS/Arch (server): linux/amd64
root@docker-host-euca-jeevan:~# ps aux | grep docker
root 29796 0.0 1.9 257732 9552 ? Ssl 11:25 0:01 /usr/bin/docker -d -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --storage-driver aufs --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=eucalyptus
root 30081 0.0 0.1 10460 920 pts/1 S+ 13:21 0:00 grep --color=auto docker
root@docker-host-euca-jeevan:~# cat /etc/default/docker
# Docker Upstart and SysVinit configuration file
# Customize location of Docker binary (especially for development testing).
#DOCKER="/usr/local/bin/docker"
# Use DOCKER_OPTS to modify the daemon startup options.
#DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4"
# If you need Docker to use an HTTP proxy, it can also be specified here.
#export http_proxy="http://127.0.0.1:3128/"
# This is also a handy place to tweak where Docker's temporary files go.
#export TMPDIR="/mnt/bigdrive/docker-tmp"
DOCKER_OPTS='
-H tcp://0.0.0.0:2376
-H unix:///var/run/docker.sock
--storage-driver aufs
--tlsverify
--tlscacert /etc/docker/ca.pem
--tlscert /etc/docker/server.pem
--tlskey /etc/docker/server-key.pem
--label provider=eucalyptus
'
root@docker-host-euca-jeevan:~# cat /var/log/upstart/docker.log
Waiting for /var/run/docker.sock
INFO[0000] +job init_networkdriver()
INFO[0000] +job serveapi(unix:///var/run/docker.sock)
INFO[0000] Listening for HTTP on unix (/var/run/docker.sock)
/var/run/docker.sock is up
INFO[0000] -job init_networkdriver() = OK (0)
WARN[0000] Your kernel does not support cgroup swap limit.
INFO[0001] Loading containers: start.
INFO[0001] Loading containers: done.
INFO[0001] docker daemon: 1.6.0 4749651; execdriver: native-0.2; graphdriver: aufs
INFO[0001] +job acceptconnections()
INFO[0001] -job acceptconnections() = OK (0)
INFO[0001] Daemon has completed initialization
INFO[0004] GET /v1.18/version
INFO[0004] +job version()
INFO[0004] -job version() = OK (0)
INFO[0008] GET /v1.18/version
INFO[0008] +job version()
INFO[0008] -job version() = OK (0)
INFO[0014] Received signal 'terminated', starting shutdown of docker...
INFO[0014] -job serveapi(unix:///var/run/docker.sock) = OK (0)
INFO[0000] +job init_networkdriver()
INFO[0000] +job serveapi(tcp://0.0.0.0:2376, unix:///var/run/docker.sock)
INFO[0000] Listening for HTTP on tcp (0.0.0.0:2376)
INFO[0000] Listening for HTTP on unix (/var/run/docker.sock)
INFO[0000] -job init_networkdriver() = OK (0)
WARN[0000] Your kernel does not support cgroup swap limit.
INFO[0000] Loading containers: start.
INFO[0000] Loading containers: done.
INFO[0000] docker daemon: 1.6.0 4749651; execdriver: native-0.2; graphdriver: aufs
INFO[0000] +job acceptconnections()
INFO[0000] -job acceptconnections() = OK (0)
INFO[0000] Daemon has completed initialization
INFO[6960] GET /v1.18/version
INFO[6960] +job version()
INFO[6960] -job version() = OK (0)
INFO[6962] GET /v1.18/containers/json
INFO[6962] +job containers()
INFO[6962] -job containers() = OK (0)
INFO[6965] GET /v1.18/version
INFO[6965] +job version()
INFO[6965] -job version() = OK (0)
From the machine we could do all the commands that it comes with against the Eucalyptus provider like ls/rm/start/stop/restart/env/ssh/inspect etc.
$ docker-machine_linux-amd64 ls
NAME ACTIVE DRIVER STATE URL SWARM
docker-host-euca-jeevan * eucalyptus Running tcp://192.168.1.2:2376
$ docker-machine_linux-amd64 env docker-host-euca-jeevan
export DOCKER_TLS_VERIFY="1"
export DOCKER_HOST="tcp://192.168.1.2:2376"
export DOCKER_CERT_PATH="/home/jeevanullas/.docker/machine/machines/docker-host-euca-jeevan"
# Run this command to configure your shell: eval "$(docker-machine_linux-amd64 env docker-host-euca-jeevan)"
We encourage you to take it for a spin against your Eucalyptus cloud and let us know what you think about it.
As usual we keep an eye on exciting new projects coming out of Docker hence next we plan to do something cool with Docker Compose and Docker machine on an Eucalyptus cloud. Stay tuned.
PS: Special thanks to the Docker team for quickly helping us out with issues and pointing at the right direction. The community pretty strong and always available to help.