Iron.io and Lumen
Lumen Iron Worker
What and why
A worker is a great way to run tasks as needed taking the load off your applications server and greatly speeding up the process of a task as you can run numerous workers at once.
A lot of this comes from http://dev.iron.io/worker/beta/getting_started/ and http://dev.iron.io/worker/beta/cli/ and their examples
Topics covered
- Creating a Lumen Worker
- Creating a statically linked binary in the worker
- Testing the worker locally with Docker
- Entering your docker environment
- Design patterns
Install Lumen
composer create-project laravel/lumen --prefer-dist
Add to composer.json
"iron-io/iron_mq": "~1.5",
"iron-io/iron_worker": "~1.4"
So now it looks like
"require": {
"laravel/lumen-framework": "5.0.*",
"vlucas/phpdotenv": "~1.0",
"iron-io/iron_mq": "~1.5",
"iron-io/iron_worker": "~1.4"
},
Install iron client
See their notes here http://dev.iron.io/worker/beta/cli/
Install docker
On a mac they have great steps here for that https://docs.docker.com/installation/mac/
Environment settings
For Lumen we can simply use our typical .env file. For Iron you put your info in the iron.json file in the root of the app (make sure to add this to .gitignore)
The format is
{ "token": "foo", "project_id": "bar" }
The worker
Make a folder called workers at the root of your app
In there place your worker file. In this case ExampleOneWorker
. This is what gets called, as you will see soon, when the worker starts. This is what will receive the payload.
workers/ExampleOneWorker.php
Inside of this to start will be
<?php
require_once __DIR__ . '/libs/bootstrap.php';
$payload = getPayload(true);
fire($payload);
function fire($payload)
{
try
{
$handler = new \App\ExampleOneHandler();
$handler->handle($payload);
}
catch(\Exception $e)
{
$message = sprintf("Error with worker %s", $e->getMessage());
echo $message;
}
}
For testing reasons and code clarity I do not like to put much code in here. I instantiate a handler class and pass in the payload.
The getPayload in the helper.php file, provided by an Iron.io example, will get the payload for us.
There is another folder to make in there called libs and for now it has this file bootstrap.php
and helper.php
[1] The helper is here
With the contents as seen below for bootstrap or visit to get the files.
<?php
require __DIR__ . '/../../vendor/autoload.php';
$app = require_once __DIR__ . '/../../bootstrap/app.php';
if(!function_exists('getPayload'))
require_once __DIR__ . '/helper.php';
use Illuminate\Encryption\Encrypter;
$app->boot();
function decryptPayload($payload)
{
$crypt = new Encrypter(getenv('IRON_ENCRYPTION_KEY'));
$payload = $crypt->decrypt($payload);
return json_decode(json_encode($payload), FALSE);
}
helper.php
I placed a gist here https://gist.github.com/alnutile/41ee747bb8e1810d19e8
Also for this example we will need a payload.json
file in the root of our app. More on that shortly, for now put this into the file.
{
"foo": "bar"
}
Finally our app folder has the ExampleOneHandler.php
file to handle the job.
<?php
namespace App;
class ExampleOneHandler {
public function handle($payload)
{
echo "This is the Payload";
echo print_r($payload, 1);
}
}
We will do more shortly.
Here is the folder/file layout
Round 1 ExampleOneHandler
Lets now run this and see what happens.
Using docker we can run this locally
docker run --rm -v "$(pwd)":/worker -w /worker iron/images:php-5.6 sh -c "php /worker/workers/ExampleOneWorker.php -payload payload.json"
You just ran, what ideally will be, the exact worker you will run when you upload the code. It will take a moment on the first run. After that it will be super fast.
Here is my output
Uploading to Iron
Bundle
This is really easy to make a script for by just adding them to an upload_worker.sh file in the root of your app and running that as needed.
touch ExampleOneWorker.zip
rm ExampleOneWorker.zip
zip -r ExampleOneWorker.zip . -x *.git*
iron worker upload --stack php-5.6 ExampleOneWorker.zip php workers/ExampleOneWorker.php
So we are touching the file so there are no errors if it is not there. Then we rm it And zip it ignoring .git to keep it slim and then we upload it with the worker and point to the directory to use.
Don’t run it just yet
I add my iron.json file to the root of my app as noted above.
and I make the Project on the Iron HUD
And then I can run the make_worker.sh
I made above
You should end up with this output
Looking at the HUD (Iron WebUI)
Under Worker and tasks we see
So lets run it from the command line to see it work
iron worker queue --wait -payload-file payload.json ExampleOneWorker
The wait is pretty cool since we can get this output. This is key when doing master slave workers as well.
You get the same output as before. But it was run on the worker
Here is the HUD
Round 2 Lets do something real
So far the payload has not done much but lets use it in this next example.
As above we make and ExampleTwoWorker.php
Make payload2.json file
{
"search_word": "batman"
}
Then we use it to call our ExampleTwoWorkerHandler
warning this is not an example on good php code
<?php namespace App;
class ExampleTwoHandler {
protected $search_word;
protected $result;
public function handle($payload)
{
$this->search_word = $payload['search_word'];
$this->getImage();
return $this->popFirstResult();
}
protected function getImage()
{
$url = 'http://ajax.googleapis.com/ajax/services/search/images?v=1.0&q=';
$url .= urlencode("site:www.thebrickfan.com " . $this->search_word . " lego");
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, $url);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
$data = curl_exec($curl);
curl_close($curl);
$result = json_decode($data, true);
$this->result = $result;
}
protected function popFirstResult()
{
$max = count($this->result['responseData']['results']);
if($max == 0)
{
throw new \Exception(sprintf("No image found :( for %s", $this->search_word));
}
else
{
$image = $this->result['responseData']['results'][rand(0, $max - 1)]['url'];
return file_get_contents($image);
}
}
}
I test locally
docker run --rm -v "$(pwd)":/worker -w /worker iron/images:php-5.6 sh -c "php /worker/workers/ExampleTwoWorker.php -payload payload2.json" > output.png
But this time put the output into a file and we get
Making a custom binary
Before I get this to iron lets make it more useful since I will lose that output.png file on the worker. Some workers we have would convert that into a base64 blob and send that back in a callback.
One enter into docker like I noted above
Two run apt-get update
Then run apt-get install jp2a
Then make a folder called /worker/builds/
And in there follow these instructions http://jurjenbokma.com/ApprenticesNotes/getting_statlinked_binaries_on_debian.html replacing jp2a as needed.
Then make a folder called /worker/bin and copy jp2a from /worker/builds/jp2a-1.0.6/src/jp2a
to this bin folder.
You should be able to see that run now by ding /worker/bin/jp2a even run apt-get remove jp2a
to show it works as a standalone library [3]
Let’s adjust our code
<?php
/**
* Created by PhpStorm.
* User: alfrednutile
* Date: 4/27/15
* Time: 9:02 PM
*/
namespace App;
use Illuminate\Support\Facades\File;
class ExampleTwoHandler {
protected $search_word;
protected $result;
public function handle($payload)
{
$this->search_word = $payload['search_word'];
$this->getImage();
return $this->popFirstResult();
}
protected function getImage()
{
$url = 'http://ajax.googleapis.com/ajax/services/search/images?v=1.0&q=';
$url .= urlencode("site:www.thebrickfan.com " . $this->search_word . " lego");
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, $url);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
$data = curl_exec($curl);
curl_close($curl);
$result = json_decode($data, true);
$this->result = $result;
}
protected function popFirstResult()
{
$max = count($this->result['responseData']['results']);
if($max == 0)
{
throw new \Exception(sprintf("No image found :( for %s", $this->search_word));
}
else
{
$image = $this->result['responseData']['results'][rand(0, $max - 1)]['url'];
$path_to_worker = base_path('bin/');
exec("chmod +x {$path_to_worker}/jp2a");
exec("TERM=xterm {$path_to_worker}/bin/jp2a $image", $output);
return implode("\n", $output);
}
}
}
run locally and you might get some decent output or not :(
Make and upload the worker
Then I run sh ./make_worker_two.php
touch ExampleTwoWorker.zip
rm ExampleTwoWorker.zip
zip -r ExampleTwoWorker.zip . -x *.git*
iron worker upload --stack php-5.6 ExampleTwoWorker.zip php workers/ExampleTwoWorker.php
And run and wait
iron worker queue --wait -payload-file payload2.json ExampleTwoWorker
And if all goes well your console and the logs should show something like
Entering your docker environment
Easy
docker run -it -v "$(pwd)":/worker -w /worker iron/images:php-5.6 /bin/bash
Now you can test things in there, download packages etc.
MVC
Not sure if this really is correct but I tend to see the Worker file as my route file. The handler as the controller and other classes as needed, Service, Repository etc. This makes things more testable etc and better organize imo.
Connecting the Queue to the Worker
Coming soon…
Numerous Environments
Waiting on bug report https://github.com/iron-io/docs/issues/467
But part of the process is to setup other projects at iron. For example if my worker is ExampleWorker then I would make ExampleWorker-dev. I would then switch to my git branch dev and do my changes. Once that is done I would make sure the token and key in my iron.json file matches that new project I made for dev and that is it.
The other way is slicker cause you do not need to change your iron.json each time but in the mean time this works fine.
Deploy from Codeship
Codeship will allow you to set custom deploy scripts or bash shells scrips basically.
In here I placed for the branch I wanted
curl -sSL -O https://github.com/iron-io/ironcli/releases/download/v0.0.6/ironcli_linux
chmod +x ironcli_linux
touch iron.json
echo "{" >> iron.json
echo '"token": "bar",' >> iron.json
echo '"project_id": "foo"' >> iron.json
echo "}" >> iron.json
zip -r PDF2PagesWorker.zip .
./ironcli_linux worker upload --stack php-5.6 PDF2PagesWorker.zip php workers/PDF2PagesWorker.php
You can easily then swap out the related project id and token for the environment you are uploading to eg development, staging etc.
Repo
https://github.com/alnutile/lumen_worker
another example Thumbnail Maker
[1] These seems to be a part of the iron worker for version 1 but not sure why not for 2 maybe there is a better pattern for this.
[2] I renamed it to ExampleOneLumen
[3] So far this is a 50/50 solution it did not work for pdf2svg but it did work for pdftk
comments powered by Disqus