AWS EC2 Container Service API in Eucalyptus

This blog post will talk about some remarkable work done by core Eucalyptus developer Steve Jones in bringing support for AWS EC2 Container Service (AWS ECS) API in Eucalyptus.

Please note that this work is not officially supported by the software today but as a proof of concept allows user(s) to appreciate the AWS ECS API in a open source based private cloud Eucalyptus.

For more information on AWS ECS one can read the documentation available on AWS website.

Last week, Steve Jones sent out an e-mail sharing his work on implementing the first cut of AWS ECS API in Eucalyptus. The work is available at the ecs branch on his fork for Eucalyptus.

We took this code, compiled Eucalyptus software based on the feature/ecs branch and ran a cloud from it (cloud-in-a-box/all-in-one). We will share the steps on how we got this going here so anyone who want to try this out can easily do it on his/her own.

First of all we need to have a CentOS 6.6 (x86_64) machine available to us.

AWS ECS for Eucalyptus fork

Make sure you clone the following fork and switch to branch feature/ecs and use that as your code base.

git clone https://github.com/sjones4/eucalyptus.git  
cd eucalyptus  
git checkout feature/ecs  

Then we need to prepare this system so it can compile Eucalyptus. For that please refer to the INSTALL document.

Few catches in INSTALL document

(These could be improvements in the INSTALL document)

• Please make sure you have xpp3 installed on the system. We have kept a copy of xpp3 here, you just need to download and install it using
  yum localinstall xpp3-1.1.3.8-11.el6.noarch.rpm

• Adding an extra repo for euca-depsbased on this baseurl http://downloads.eucalyptus.com/software/eucalyptus/build-deps/3.4/rhel/$releasever/$basearch, seems to have worked on a 4.1 codebase

• Also install the following RPMs to set eucalyptus and euca2ools repos
  

yum install \ http://downloads.eucalyptus.com/software/eucalyptus/4.1/centos/6/x86_64/eucalyptus-release-4.1.el6.noarch.rpm

yum install \ http://downloads.eucalyptus.com/software/euca2ools/3.2/centos/6/x86_64/euca2ools-release-3.2.el6.noarch.rpm  

These repos provided important dependencies for runtime.

• The JAVA_HOME environment variable was set to
  

export JAVA_HOME="/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.79.x86_64/"

Configuration of Eucalyptus

This setup was a cloud-in-a-box for development purposes. Hence we modified /etc/eucalyptus/eucalyptus.conf with the appropriate values , for reference it is available below

CC_PORT="8774"  
CLOUD_OPTS=""  
DISABLE_TUNNELING="Y"  
EUCALYPTUS="/"  
EUCA_USER="eucalyptus"  
HYPERVISOR="kvm"  
INSTANCE_PATH="/var/lib/eucalyptus/instances"  
LOGLEVEL="INFO"  
NC_CACHE_SIZE=500000  
NC_PORT="8775"  
NC_ROUTER=N  
NC_SERVICE="axis2/services/EucalyptusNC"  
NC_WORK_SIZE=500000  
NODES="10.104.10.45"  
SCHEDPOLICY="ROUNDROBIN"  
USE_VIRTIO_DISK="1"  
USE_VIRTIO_NET="1"  
USE_VIRTIO_ROOT="1"  
VNET_BRIDGE="br0"  
VNET_DHCPDAEMON="/usr/sbin/dhcpd"  
VNET_MODE="EDGE"  
VNET_PRIVINTERFACE="br0"  
VNET_PUBINTERFACE="br0"  

Before you initialize the database please make sure you set the permissions correctly by running:

euca_conf --setup

You would also need to configure the network bridge br0 based on the documentation here.

Next following the manual installation steps available in the official documentation we were able to initialize the postgres DB, register CLC, UFS, Walrus, SC, CC and NC on the same host.

Note that the nice thing after registering UFS on the host is that you would see a shiny new Container service ENABLED

SERVICE    container           API_45      API_45.container        ENABLED     25      http://10.104.10.45:8773/services/Container arn:euca:bootstrap:API_45:container:API_45.container/  

Some of the cloud properties we modified for this cloud were:

euca-modify-property -p authentication.access_keys_limit=10  
euca-modify-property -p authentication.signing_certificates_limit=10  
euca-modify-property -p authentication.credential_download_generate_certificate=Limited  
euca-modify-property -p objectstorage.providerclient=walrus  
euca-modify-property -p bootstrap.webservices.use_dns_delegation=true  
euca-modify-property -p bootstrap.webservices.use_instance_dns=true  
euca-modify-property -p euca-az-01.storage.blockstoragemanager=overlay  

The network.json was a simple 1 network

{
  "InstanceDnsDomain": "eucalyptus.internal",
  "InstanceDnsServers": [
    "10.104.10.45"
  ],
  "PublicIps": [
    "10.104.3.1-10.104.3.29"
  ],
  "PrivateIps": [
    "10.104.3.30-10.104.3.61"
  ],
  "Subnets": [
    {
        "Subnet": "10.104.0.0",
        "Netmask": "255.255.0.0",
        "Gateway": "10.104.0.1"
    }
  ]
}

Configuring AWS CLI

We will be using AWS CLI against the Container service on Eucalyptus. Note that in order to use AWS CLI with Eucalyptus we prefer to modify the _endpoints.json appropriately , if you wish to use this method you could use the following json (replace /usr/lib/python2.6/site-packages/botocore/data/aws/_endpoints.json) along with your Eucalyptus endpoints

https://gist.github.com/jeevanullas/9ef7d5a5440416de5b15

Note the Container API Endpoint:

  "ecs": [
    {
      "uri":"http://10.104.10.45:8773/services/Container",
      "properties": {
        "credentialScope": {
            "region": "eucalyptus"
        }
       }
    }
  ]

Now just run aws configure as you do normally and provide your access/secret keys (eucarc file). Please make sure you provide the region as eucalyptus.

Configure IAM Role and Instance Profile

The Container instance needs to have access to the Eucalyptus ECS service. This can be done by simply creating an IAM role and instance profile using that role. Container instance would than be launched with this profile.

The trust policy for the role is assume-role-policy.json :

  "Version": "2012-10-17",
  "Statement": {
    "Effect": "Allow",
    "Principal": {"Service": "ec2.amazonaws.com"},
    "Action": "sts:AssumeRole"
  }
}

Then we create the role

aws iam create-role --role-name ecsInstanceRole --assume-role-policy-document file://$HOME/assume-role-policy.json  

We provide the role with the necessary access as per documentation, the IAM policy used for Eucalyptus role is AmazonEC2ContainerServiceforEC2Role.json :

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ecs:CreateCluster",
        "ecs:RegisterContainerInstance",
        "ecs:DeregisterContainerInstance",
        "ecs:DiscoverPollEndpoint",
        "ecs:Submit*",
        "ecs:Poll"
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}

Then upload the above policy to the role we created:

aws iam put-role-policy --role-name ecsInstanceRole --policy-name AmazonEC2ContainerServiceforEC2Role --policy-document file://AmazonEC2ContainerServiceforEC2Role.json  

Finally we will create an instance profile with this new role

aws iam create-instance-profile --instance-profile-name ecsprofile

aws iam add-role-to-instance-profile --instance-profile-name ecsprofile --role-name ecsInstanceRole  

You can get the Role ARN (this is needed later while launching the instance) with the command

 aws iam list-instance-profiles | grep instance-profile | grep Arn

"Arn": "arn:aws:iam::110473696260:instance-profile/ecsprofile"

AWS ECS Agent and Eucalyptus Config

It is currently hosted here https://github.com/aws/amazon-ecs-agent and has to be installed/running inside the instance that we call a container instance .

It can run and talk to the Eucalyptus ECS API with some issues but it was good for a POC. We found out that it uses the dynamic data which is not implemented yet in Eucalyptus but luckily this didn't become showstopper for the POC.

To install and configure AWS ECS Agent we use the config.yml file for CoreOS. More on this in next section.

AWS ECS Agent parameters are documented very well. Based on these parameters we choose to use following for the Eucalyptus setup

ECS_BACKEND_HOST=container.localhost:8773  
AWS_DEFAULT_REGION=eucalyptus  

The Eucalyptus ECS API listens on port 8773 and has the endpoint URL container.localhost (This is because we enabled DNS cloud properties earlier).

Another important configuration required for AWS ECS Agent to work with Eucalyptus was to specify the -k true flag while running it, this makes it ignore SSL cert verification because default SSL cert for Eucalyptus endpoint is a self-signed certificate.

Again these parameters would be passed via the CoreOS config file.

This is it, now we will upload the CoreOS image on the cloud and run an instance from that image using the instance profile ecsprofile

Upload CoreOS image and setup Config.yml

The CoreOS image is perfect for trying stuff out. One can download the latest stable CoreOS image for openstack and convert the QCOW2 file into raw format

wget http://stable.release.core-os.net/amd64-usr/current/coreos_production_openstack_image.img.bz2

bunzip2 -d coreos_production_openstack_image.img.bz2 

qemu-img convert -O raw coreos_production_openstack_image.img coreos_production_openstack_image.raw

euca-install-image -i coreos_production_openstack_image.raw -b coreos-production-euca -n coreos -r x86_64 --virtualization-type hvm --description "CoreOS Eucalyptus image"  

The config.yml that we used as user-data was:

Note that the config.yml sets HTTP_PROXY and HTTPS_PROXY docker daemon. These are not necessary but were used in this environment because of dependency on a proxy.

Run Eucalyptus Container Instance

Now that we have all the pieces in place it was time to run our container instance. In order to run

aws ec2 run-instances --key-name sshlogin --instance-type m2.xlarge --user-data config.yml --image-id emi-1da37389 --iam-instance-profile "arn:aws:iam::110473696260:instance-profile/ecsprofile"  

Once the instance is launched we can SSH to it using the private key for sshlogin keypair and check stuff out

$ ssh -i /$HOME/sshlogin core@10.104.3.26
Last login: Mon May  4 06:08:12 2015 from 10.104.3.26

CoreOS stable (633.1.0)

core@euca-10-104-3-47 ~ $ docker version  
Client version: 1.5.0  
Client API version: 1.17  
Go version (client): go1.3.3  
Git commit (client): a8a31ef-dirty  
OS/Arch (client): linux/amd64  
Server version: 1.5.0  
Server API version: 1.17  
Go version (server): go1.3.3  
Git commit (server): a8a31ef-dirty  

At this point it should be running the ecs-agent container

core@euca-10-104-3-47 ~ $ docker ps  
CONTAINER ID        IMAGE                             COMMAND                CREATED             STATUS              PORTS                        NAMES  
c3e60eda6904        amazon/amazon-ecs-agent:4023248   "/agent -k true"       2 hours ago         Up 2 hours          127.0.0.1:51678->51678/tcp   ecs-agent  

Also the log file for AWS ECS agent should have been created by it at /var/log/ecs/ecs-agent.log

In the log file you can see critical message like this but can be safely ignored for now

t=2015-05-04T06:07:56+0000 lvl=crit msg="Unable to access EC2 Metadata service to determine EC2 ID" module=main err="unexpected end of JSON input" stack=[agent/agent.go:103]  

AWS ECS agent was able to successfully create the default cluster and register to the Eucalyptus ECS service

t=2015-05-04T06:07:56+0000 lvl=info msg="Registering Instance with ECS" module=main stack=[agent/agent.go:131]  
t=2015-05-04T06:07:56+0000 lvl=info msg=Registered! module="api client" stack="[github.com/aws/amazon-ecs-agent/agent/api/api_client.go:254 github.com/aws/amazon-ecs-agent/agent/api/api_client.go:182 agent/agent.go:132]"  
t=2015-05-04T06:07:56+0000 lvl=info msg="Registration completed successfully" module=main containerInstance=arn:aws:ecs::110473696260:container-instance/ee5b4bc3-a47e-40fa-9566-74b71dd2d116 cluster=default stack=[agent/agent.go:140]  
t=2015-05-04T06:07:56+0000 lvl=info msg="Saving state!" module=statemanager stack="[github.com/aws/amazon-ecs-agent/agent/statemanager/state_manager.go:180 github.com/aws/amazon-ecs-agent/agent/statemanager/state_manager.go:154 agent/agent.go:142]"  
t=2015-05-04T06:07:56+0000 lvl=info msg="Beginning Polling for updates" module=main stack=[agent/agent.go:159]  

Register Task definition

From this point on-wards one can simply follow the steps mentioned in the official AWS documentation here

The next obvious step based on it was to create a task definition, we used the sample wordpress task definition

aws ecs register-task-definition --cli-input-json file://$HOME/ecs-wordpress-task-def.json  

Run Task

After the task was registered we went straight ahead and run it.

aws ecs run-task --cluster default --task-definition wordpress:1 --count 1  

This triggered the AWS ECS Agent running inside the Container instance to start pulling docker image for mysql and wordpress. It also did all the necessary configuration and finally after 2 minutes the wordpress blog was up and running.

We could check the containers running inside the Container instance with docker ps

core@euca-10-104-3-47 ~ $ docker ps  
CONTAINER ID        IMAGE                             COMMAND                CREATED             STATUS              PORTS                        NAMES  
48f84a4341ed        wordpress:4                       "/entrypoint.sh apac   2 hours ago         Up 2 hours          0.0.0.0:80->80/tcp           ecs-wordpress-1-wordpress-dac995abd0eae6e07d00  
b3650fac5ebe        mysql:5                           "/entrypoint.sh mysq   2 hours ago         Up 2 hours          3306/tcp                     ecs-wordpress-1-mysql-c2f0979cf9c2d4c2fe01  
c3e60eda6904        amazon/amazon-ecs-agent:4023248   "/agent -k true"       2 hours ago         Up 2 hours          127.0.0.1:51678->51678/tcp   ecs-agent  

Setup wordpress

In order to proceed further with the setup and configuration of wordpress one has to make sure the security group used to launch the Container instance has port 80 opened for inbound. Then using the Container instance IP address one can simply open the web browser and start configuration of wordpress

Few anomalies

We had some trouble getting the correct status on the container instance status, it always showed INACTIVE although it was able to pick the tasks and execute them.

Certainly there are limitations in the implementation because this is not officially supported but its a start. Things only move forward from the starting point.

Conclusion

We sincerely hope this information was useful and one can only imagine endless possibilities it brings to the game. Note that AWS ECS service is constantly evolving and it is hard to keep up with this pace and only best effort can succeed in the game.