hcoelho.com

my blog

Title only : Full post

AWS ECS: Fixing agent container not running and exit code 5

:

I had a great time this week when our instance on Amazon imploded for strange reasons, sadly I don't have the logs anymore to post them here, but these were the key problems (or what I remember from the event) and their solutions:

The docker process suddenly stopped responding some commands such as docker ps

Solution: I reinstalled docker and AWS ECS Container Agent.

Docker is now responding, but the container agent will not run

By running it with the -it flags, we noticed it was exiting with code 5.

Solution: Delete the file /var/lib/ecs/data/ecs-agent-data.json and start the container agent again.

cdto amazon aws ecs agent 

Deployment of containers on AWS

:

We spent the past days reading about AWS in order to deploy the 2 containers we developed: one container only has a "Dockerfile" with our database (MongoDB) and the other container has the API to insert data into this database (Node.js). In this post, I'll describe some things I would like to have known before I started the deployment; it was a very frustrating process, but after you learn how everything works, it becomes really easy.

First of all, the deployment is not a linear process: you will have to know some details about your application before you start the process; these details, however, will not be obvious for you if you haven't used AWS before: this is one of the reasons why it was a slow, painful process for us.

Looking back, I think the first step to deploy these containers is to upload the repositories, even though they are not properly configured yet: you need the repositories there to have a better perspective on what to do. So, first step: push the docker images to EC2 Container Registry. The process is simple, it only takes 4 steps (3 steps, after the first push), which are just copying and pasting commands in the command line.

After the containers are uploaded, we should choose a machine that will run Docker with the containers, and here is the catch: we need to choose a machine that is already optimized to be used for Container Service, otherwise it will not be a valid Container Instance and you would have to configure it yourself. To find machines that are optimized for ECS, we search for "ecs" on the custom instances. After the machine was chosen, we select the other specifications we'll need, such as storage, IPs, and so on - but nothing too special here.

With the right machine instance, a default Cluster will be created in the Container Registry. Here is the interesting part: the cluster is a set of services, which are responsible for (re)starting a set of tasks, which are groups of docker containers to be used by the machine. Instead of starting from the service, we now should start from the task, adding its containers and work back to the service - then the deployment will be complete.

To create a task is simple: we give it a name and a list of the repositories (the ones that we uploaded in the beginning), but we also have to set how the containers are going to interact with each other and with the devices outside. There are two special settings we had to do:

1- The MongoDB container should be visible for the API. This can be done by linking them together: on the container for the API, we map the name of the database container to an alias (for instance: Mongo:MongoContainer); with this, the container of the API will receive some environment variables, such as MONGOCONTAINER_PORT, with the address and port of the other container. We can use this to make the API connect to the database (and the source code would probably have to be modified to use this port).

2- The MongoDB container should use an external drive for storage, otherwise, its data will be lost when the container is restarted. For this, we map the external directory (that we want the data to be stored into) to an internal directory, which is used by the database (for instance, /usr/mongodb:/mongo/db). Since we wanted to use an external device, we also had to make sure the device would be mounted when the machine was started.

After the task is set up, we make the service for the cluster: the service, in this case, contains the only task that we made. With the service properly configured, it will start and restart the tasks automatically: the deployment should now be ready.

It's easy to understand why we spent so much time trying to make it work (giving the amount of details and steps), but looking back, this modularity makes a lot of sense. The learning curve is very steep, but I am very impressed by how powerful this service is. I am very inclined to start using it for deploying my own projects.

cdot aws containers ec2 ecs