@WIP AWS Batch and Workers with Laravel

Posted: 2017-09-06 10:37:51

Sending Docker Commands to Get A Task Done

This article will show how to get started with AWS Batch and Docker to spin up a "worker"

By the time you are done reading it you will:

  1. Have a Docker image to run your command in.
  2. Deploy the Docker image to AWS ECR
  3. And Run a Task on the AWS Batch system, or a 100 tasks, does not matter.
  4. Finally you will have the scripts needed to fully build AWS Batch with CloudFormation, e.g. super simple

Building the Laravel Worker

This can be any language you want, any framework or none! I will focus on Laravel.

Either way we need a git repo, so later we can push this and do a Docker build on the CI via the deploy scripts.

Note the 2 Commands

  • One to manage all the Tasks and then stitch the results together
  • One to just covert the image to PDF -- depend on?

Install SDK

The Command

Before you get going install AWS CLI on your machine.

Make sure you have your credentials and profiles setup per the docs here

Then the rest will work off those credentials as we make new ones per application and you just place them into your profile via ~/.aws/credentials and ~/.aws/config

Installing Incomings

Building the Docker container to run this work

docker-compose build //this will be needed locally 
docker-compose run --rm -w /app app php artisan 
docker-compose run app php artisan security:scan --git-repo=https://github.com/alnutile/security-scanner-show-error-poc
docker-compose run -w /app app php artisan batch:convert_image_to_pdf --image-url=https://dl.dropboxusercontent.com/s/d2sx0wjheb7dk0p/example_batch.jpg --destination=batch-example/foo

Setting up AWS to run the work on.

Compute Environment

For starters there is the Compute Environment (CE). This is what manages the scaling of the EC2 instances.

The script is HERE. But this is only needed once. I will Explain in a moment.

Job Definition

Then there is the JobDefinitions. This is where we can define the Job we want to run and the Compute Environment to run it on. Let the CE figure out the scaling, and hence we can just build one and share it with every one. So if you have a CE already build try putting your JobDefinition on that before making another one for your Worker.

The script is HERE.

Open the scripts, they are just JSON. Good docs are HERE on how to make, or read these scripts.

Once those are all setup take the AWS KEY and Secret it gave you and use it to setup you local AWS CLI.

This is key for a sane workflow. Here is a link [HERE] get your profile right.

Once you have this setup all the rest will fall into place.

Deploying to CI

Before we push let's make sure that .gitignore has:

docker/app/packaged/*
!docker/app/packaged/.gitkeep

Plus all the default Laravel ignores including .env

Deploy Script Now we need to get this work to CI

Going to AWS UI to run

This is annoying lets make a command for this...

The Testing Command

More import that is how a "requesting" app will call this.

DIAGRAM HERE

Putting it all together

Show running 10 jobs via the command

Now Taking it a bit further

The command did a lot

  • Got Image
  • Made into PDF
  • Stitched them all together

But now let's take this a bit further. I am going to make 2 JobDefinitions.

One will kick off an orchestrate the work by creating a job per image, tracking the progress of those jobs, and When all those jobs are done knit them together!

Here is how it will look.

Summary of Install

Get Docker file in place docker/app/DockerFile Get Packaged file in place Setup Core CloudFormation * ECR * JobDefinition * User Key/Secret Get docker-compose inlace Get Travis.yml in place and docker/deploy.bash See diagram on how this all comes together