@WIP AWS Batch and Workers with Laravel
Sending Docker Commands to Get A Task Done
This article will show how to get started with AWS Batch and Docker to spin up a "worker"
By the time you are done reading it you will:
- Have a Docker image to run your command in.
- Deploy the Docker image to AWS ECR
- And Run a Task on the AWS Batch system, or a 100 tasks, does not matter.
- Finally you will have the scripts needed to fully build AWS Batch with CloudFormation, e.g. super simple
Building the Laravel Worker
This can be any language you want, any framework or none! I will focus on Laravel.
Either way we need a git repo, so later we can push this and do a Docker build on the CI via the deploy scripts.
Note the 2 Commands
- One to manage all the Tasks and then stitch the results together
- One to just covert the image to PDF -- depend on?
Before you get going install AWS CLI on your machine.
Make sure you have your credentials and profiles setup per the docs here
Then the rest will work off those credentials as we make new ones per application and you just place them into your profile via
Building the Docker container to run this work
docker-compose build //this will be needed locally docker-compose run --rm -w /app app php artisan docker-compose run app php artisan security:scan --git-repo=https://github.com/alnutile/security-scanner-show-error-poc docker-compose run -w /app app php artisan batch:convert_image_to_pdf --image-url=https://dl.dropboxusercontent.com/s/d2sx0wjheb7dk0p/example_batch.jpg --destination=batch-example/foo
Setting up AWS to run the work on.
For starters there is the Compute Environment (CE). This is what manages the scaling of the EC2 instances.
The script is HERE. But this is only needed once. I will Explain in a moment.
Then there is the JobDefinitions. This is where we can define the Job we want to run and the Compute Environment to run it on. Let the CE figure out the scaling, and hence we can just build one and share it with every one. So if you have a CE already build try putting your JobDefinition on that before making another one for your Worker.
The script is HERE.
Open the scripts, they are just JSON. Good docs are HERE on how to make, or read these scripts.
Once those are all setup take the AWS KEY and Secret it gave you and use it to setup you local AWS CLI.
This is key for a sane workflow. Here is a link [HERE] get your profile right.
Once you have this setup all the rest will fall into place.
Deploying to CI
Before we push let's make sure that
Plus all the default Laravel ignores including
Deploy Script Now we need to get this work to CI
Going to AWS UI to run
This is annoying lets make a command for this...
The Testing Command
More import that is how a "requesting" app will call this.
Putting it all together
Show running 10 jobs via the command
Now Taking it a bit further
The command did a lot
- Got Image
- Made into PDF
- Stitched them all together
But now let's take this a bit further. I am going to make 2 JobDefinitions.
One will kick off an orchestrate the work by creating a job per image, tracking the progress of those jobs, and When all those jobs are done knit them together!
Here is how it will look.
Summary of Install
Get Docker file in place docker/app/DockerFile Get Packaged file in place Setup Core CloudFormation * ECR * JobDefinition * User Key/Secret Get docker-compose inlace Get Travis.yml in place and docker/deploy.bash See diagram on how this all comes together