We are starting to use Iron.io and their workers for a lot of the tasks that our apps need to do. For example one app needs to scan websites for images and text and report on them. In our case that is 2 workers, one with the code needed to get the text we want and the other images. Another worker runs behat tests to take screenshots and reports back to the called with the results.

Using Iron.io has made this whole process easy and scalable. One request can be for say 100 urls and with Iron.io we can run one worker per url or using the Symfony Process library we can even use a worker to run a multi-threaded processes.

Some of the resources out there like iron`s example are great. And using this library has made it super easy. Below I cover how exactly to set this up. (hopefully this week we will have a Laravel 5 version of it out)

Step 1 Install

Install 4.2 work. (5 might be ready soon)

composer create-project laravel/laravel=4.2 example_worker --prefer-dist

Set your minimum stability in your composer.json

	},
	"config": {
		"preferred-install": "dist"
	},
	"minimum-stability": "dev"
}

Then pull in the library

composer require iron-io/laraworker

And add this one patch for PHP 5.6 TODO add code snippet

https://github.com/iron-io/laraworker/issues/5

and

https://github.com/iron-io/iron_core_php/blob/master/src/IronCore.php#L269

And of course as the readme.md notes for Laraworker

php vendor/iron-io/laraworker/LaraWorker.php -i true

As the developer notes this makes a new folder and file

/worker/libs/worker_boot.php and /worker/ExampleLaraWorker.php

Step 2 Configure

We will use the .env to do configuration not the way noted in the laraworker docs so lets install that. Just use this post to set that up.

So after you are done your, as in the Laraworker docs, we need to set the queue config.

Set Iron.io credentials in app/config/queue.php and set default to iron –> ‘default’ => ‘iron’,

So yours will look like

# https://github.com/alnutile/laravel_guide/blob/master/projects/example_worker/app/config/queue.php
	'default' => getenv('QUEUE_DRIVER'),

	'connections' => array(

		'iron' => array(
			'driver'  => 'iron',
			'host'    => 'mq-aws-us-east-1.iron.io',
			'token'   => getenv('IRON_TOKEN'),
			'project' => getenv('IRON_PROJECT_ID'),
			'queue'   => 'your-queue-name',
			'encrypt' => true,
		),

	),

Then make your project on Iron and get the Token and Project ID

Step 3 See if Example Worker works

Lets see if the Example works before we move forward.

php artisan ironworker:upload --worker_name=ExampleLaraWorker --exec_worker_file_name=ExampleLaraWorker.php

If it worked you will see

example

This will upload a worker related queue

example

Step 4 Make our own worker!

The goal of this worker

  • It will get a JSON object of the info needed to do a job
  • It will do the job by getting the json file from the S3 file system where it lives (it could live in a db or other location)
  • Using the JSON object’s callback it will send back the results to the caller

That is it.

This example will be used in real life to later on parse say 100 urls for already created json render tree objects of the urls data including images and text. This job only cares about the text. Cause the job is fairly easy I will be sending to each worker 5 urls to process.

Copy the worker in /workers folder to the new Worker name

Due to bad naming abilities I am calling this RenderTreeTextGrepper.php

So now my worker folder has

RenderTreeTextGrepper.php

But I do not want that class to have all my code so I will start to build out a namespace for all of this and the 2 classes I want to manage ALL of this work.

Class 1 @fire

So the worker will fire the class I have to handle all of this.

	"autoload": {
		"classmap": [
			"app/commands",
			"app/controllers",
			"app/models",
			"app/database/migrations",
			"app/database/seeds",
			"app/tests/TestCase.php"
		],
	  "psr-4": {
		"AlfredNutileInc\\RenderTreeTextGrepperWorker\\": "app/"
	  }
	},

then

composer dump

Then in app/RenderTreeTextGrepperWorker folder I have

example_folder

/projects/example_worker/app/RenderTreeTextGrepperWorker/RenderTreeGrepperHandler.php is the class to handle the incoming request and process it.

Class 2 Event Listener

Then I register the event listener with the app/config/app.php to make it easier to handle the results of the output. You can do all of this in class 1 as well.

#app/config/app.php
'AlfredNutileInc\RenderTreeTextGrepperWorker\GrepCallbackListener'

And that is it.

What is it?

So we are going to upload and run this and here is what will happen. NO WAIT!

First lets make a test so we can see locally if all the logic is there.

Local Test

Just a quick test to see if the handler will handle things and pass results

<?php

class RenderTreeTextTest extends \TestCase {

    /**
     * @test
     */
    public function should_populate_results()
    {
        $handle = new \AlfredNutileInc\RenderTreeTextGrepperWorker\RenderTreeGrepperHandler();
        $payload = new \AlfredNutileInc\RenderTreeTextGrepperWorker\RenderTreeTextDTO(
            'foo-bar',
            ['foo', 'bar', 'baz'],
            ['text1', 'text2'],
            [
                'caller'     => 'http://someposturl.dev/rendertree_results',
                'params'     => ['foo', 'bar']
            ],
            false,
            false
        );
        $results = $handle->handle($payload);

        var_dump($results);
        $this->assertNotNull($results);
    }
}

Running this

phpunit --filter=should_populate_results

Produces this

 class AlfredNutileInc\RenderTreeTextGrepperWorker\RenderTreeTextDTO#334 (6) {
    public $uuid =>
    string(7) "foo-bar"
    public $urls =>
    array(3) {
      [0] =>
      string(3) "foo"
      [1] =>
      string(3) "bar"
      [2] =>
      string(3) "baz"
    }
    public $text =>
    array(2) {
      [0] =>
      string(5) "text1"
      [1] =>
      string(5) "text2"
    }
    public $callback =>
    array(2) {
      'caller' =>
      string(41) "http://someposturl.dev/rendertree_results"
      'params' =>
      array(2) {
        ...
      }
    }
    public $results =>
    array(1) {
      [0] =>
      string(21) "Listener is listening"
    }
    public $status =>
    bool(false)
  }
}

Of course I need to go into more testing for the two classes to see how they react to different data going in but just to see that there are not obvious issues before I upload the worker.

Upload the worker we just made

php artisan ironworker:upload --worker_name=RenderTreeTextGrepper --exec_worker_file_name=RenderTreeTextGrepper.php

And then we see on Iron.io

new worker

Then we run it

php artisan ironworker:run --queue_name=RenderTreeTextGrepper

Before that though I updated app/commands/RunWorker.php:26 to make a better payload

    public function fire()
    {
        $queue_name = $this->option('queue_name');
        $payload = "This is Hello World payload :)";
        
        if($queue_name == 'RenderTreeTextGrepper')
        {
            $payload = new \AlfredNutileInc\RenderTreeTextGrepperWorker\RenderTreeTextDTO(
                'foo-bar',
                ['foo', 'bar', 'baz'],
                ['text1', 'text2'],
                [
                    'caller'     => 'http://someposturl.dev/rendertree_results',
                    'params'     => ['foo', 'bar']
                ],
                false,
                false
            );
        }

We then see the Task

task

And the example log output

log

Guzzle and the Callback

How to format the callback?

Let’s require guzzle

composer require guzzlehttp/guzzle

At this point we have a working example. The queue takes the json and the worker processes it!

/projects/example_worker/app/RenderTreeTextGrepperWorker/GrepCallbackListener.php

Thanks to the library and Iron.io it really is that simple.