After a hectic time rushing from meeting to meeting, it's now back to the challenge of coding and trying to re-engage a meeting schedule here back in Australia.  The challenge has been switching back to a self motivation mode.

Really our biggest challenge now is focus.  It's not that we don't know what we need to do, it's that there is so much of it.  I've said to a few people that we could literally employ six others full time – not that we have the money to do that yet!  The trick is really trying to balance between the longer term things that need focus now to happen in three months, Vs. the shorter term critical things to get us into the hands of end users.

Despite all this, we are closing in rapidly on a product that we can charge money for.  We are getting closer to our customers, we are finding out what they need and what they like (and what they're not so sure of) about Tribalytic.  Yet there is still so much more to go to leap the gap.

One of the best thing about this blog is the ability to go back and review history.  I was about to say "It's been so long since we've updated the product" yet the current beta was released in Week 21 (http://timbull.com/reflections-on-start-up-life-week-21) which was only four weeks ago!  Since then, we've spent a week with customers, close to three weeks on the road and now here we are.

In that time, our own experiences and the feedback on Tribalytic have shown several short comings that need to be addressed, so we are back into a round of engineering (a somewhat shorter one this time) to address these.  While technically challenging, these are essential for the accuracy and performance of the product.  Our next beta (fingers crossed for next week) will support:

1. A near real time index of Australian Twitter users and a greatly expanded index.
2. A revised stemmer (which means that Bunnings won't be treated as Bun and related to "Hot, Cross and Easter" (of all bugs this one makes me laugh the most). For those interested, the basic stemming algorithm we were using drops the s, then the ing, then the double n to get to a word "stem" or root.  Works great for walking, walked walks, but obvious issues for Bunnings.
3. Some level of boolean logic in our search engine (AND, OR and NOT). e.g @kevinruddpm OR rudd

The more astute of you will of noticed that this post is also pretty much two days late now (usually first thing Monday morning).  Last week turned into a long week.  On a personal note (I can't comment overly about Alex's technical struggles although he's had no shortage of challenges as well) I've been challenged with getting our processing pipeline live.  This is the thing that collects all the tweets and keeps everything up to date.  It would run for a couple of hours, then choke up and die.  Finally this morning I nutted it out after working through close to a million lines of log file to locate the issue.

Over 5 days of wasted elapsed time for (I kid you not) one parameter on a command.  Not even one line of code, it was 12 characters (including the spaces). (For the technically minded, I hadn't set a timeout on the socket call to Twitter and on rare occasions the call would just never connect and not fail so it caused the thread to block and never return – adding a timeout resolved the issue).

Finally the pipeline is now up and running and that's one less issue I need to concern myself with.

Working with Alex continues to be a lot of fun. It's more challenging now he's back in Beijing and I'm here in Melbourne, but we continue to be able to work through issues via email and Skype without too many problems.  Probably the most challenging thing is that in the last couple of weeks we've passed some kind of event horizon where we no longer understand what each other is coding! The complexities of the engine and it's search are a black box to me, while the processing pipeline and the overall processing architecture is just not worth Alex investing his time in understanding while there's a search engine to work on.

I'm often amused by our conversations – my favourite of the last week (recorded here for my memory more than anything else) is the detailed discussion on how to weight relevant people around search terms.  If two people talk about magnum and icecream 10 times, but one person mentions magnum 7 times and icecream 3 times and the other magnum 3 times and icecream 7 times, who should be listed first? 

What if the a user searched for icecream and magnum vs magnum and icecream.  What if one persons conversations were all in the first few days and the other persons in the last two?  For the record – there are no good answers to these questions, but it does show the rabbit holes you can work yourselves into.

We continue to meet with investors and still have some meetings in the pipeline.  There seems to be good consensus that we have a business, the question we now need to address (at least for Venture level investors) is do we have a $100 Million business (seriously… they want to see the possibility).

• Several meetings with some great people.
• Receiving feedback on our reports and the product.
• Getting back to coding – feeling like we are progressing the product towards our customers.
• Now have a Tribalytic blog (http://blog.tribalytic.com) and Twitter account (http://twitter.com/tribalytic) so please follow them!

• Processing pipeline problems wins hands down for this one.

• Get the next beta of the engine ready for release.
• Keep meeting with people.

• Next install RabbitMQ – Originally I simply used Synaptic Package Manager in Ubuntu, searched for Rabbit and installed it and its dependencies, but I noticed this is version 1.6.0 and the latest here is 1.7.2 at time of writing, so I downloaded and installed the latest package instead.  I doubt it will make much difference which you use.
   
• Test the installation – using the following steps below which I largely copied from here 

Configure the security / vhost etc.  Note this is an important step you'll almost certainly come back to later if you stuff up

$ sudo rabbitmqctl add_user myuser mypassword
$ sudo rabbitmqctl add_vhost myvhost
$ sudo rabbitmqctl set_permissions -p myvhost myuser "" ".*" ".*"

To start and stop the service, use the following

$ sudo rabbitmq-server

or to run it in the background (recommended)

$ sudo rabbitmq-server -detached

Finally, to stop the server

$ sudo rabbitmqctl stop

 Note – if you have trouble with the reset (because like me you tried to actually cluster the machine BEFORE the cookie was set in Erlang) you can try sudo rabbitmqctl force_reset which should sort it out.

• You now have a clustered RabbitMQ setup.  You can verify this by typing sudo rabbitmqctl status on either server and you'll see it list something like the following:

[{running_applications,[{rabbit,"RabbitMQ","1.7.2"},
                        {mnesia,"MNESIA  CXC 138 12","4.4.10"},
                        {os_mon,"CPO  CXC 138 46","2.2.2"},
                        {sasl,"SASL  CXC 138 11","2.1.6"},
                        {stdlib,"ERTS  CXC 138 10","1.16.2"},
                        {kernel,"ERTS  CXC 138 10","2.13.2"}]},
 {nodes,['rabbit@laptop','rabbit@server']},
 {running_nodes,['rabbit@laptop','rabbit@server']}]

This is neat, but if I read the documentation correctly what we have here is a RAM / RAM cluster.  If one of the servers goes down, we will be fine because the message state is replicated across clusters, but if the whole lot went out (because the data centre lost power, or more likely in my situation that I just turned both PCs off over night) we might really want a persistent DISK node.

To convert Server to being a disk node, simply execute the following on Server (while the cluster is running).

$ sudo rabbitmqctl stop_app
$ sudo rabbitmqctl cluster rabbit@laptop rabbit@server

Or (FYI) to turn it back into a RAM node.  Note it doesn't matter for our purposes in this doco how they are configured, read up and decide what you need.

$ sudo rabbitmqctl cluster rabbit@laptop

Finally, start it up again

$ sudo rabbitmqctl start_app

To remove a server from a cluster at any time, simply do this (this is not a required step, it's just "FYI").

$ sudo rabbitmqctl stop_app
$ sudo rabbitmqctl reset
$ sudo rabbitmqctl start_app

NB – You'll need to use force_reset for the LAST node in the cluster to be removed (if you're separating them all out again).  REMEMBER if you do a full cluster reset (like I did in testing this), you'll need to redo the security section from the first section again   This is because you've reset both nodes so they no longer hold the security information.  If you ever get into problems with the configuration, I've resolved them consistently by de-clustering, forcing a node reset on both nodes and then clustering and redoing the security.

Elapsed Time: Research, reading, implementing and trouble shooting cluster ~2 hr.  Following the above steps, prob. 30 minutes.

Configure Celery with Django

OK, now we are ready to get Celery setup with Django. Create a new django project.  Mines called "clifton" (we use local train stations as our milestone names) and then within that I have an app called fetcher.  All of these steps need to be done on both Server and Laptop (I'm using SVN so I simply make the change on Server, commit and then update from Laptop).

• In the INSTALLED_APPS section of settings.py add celery as follows.

INSTALLED_APPS = (
    'django.contrib.sessions',
    'django.contrib.sites',
    'celery',
    'clifton.fetcher',
)

• Sync the DB as celery adds a couple of task tracking tables.

$ python manage.py syncdb

• Add the following settings into settings.py as well.  These are just to get us going, we'll come back to this and improve them soon once we've got this section working.

BROKER_HOST = "localhost"
BROKER_PORT = 5672
BROKER_USER = "myuser"
BROKER_PASSWORD = "mypassword"
BROKER_VHOST = "myvhost"

• Finally, in your application directory (e.g. fetcher in my example) create a tasks.py module with the following code.  This is literally just for a test to show it's working.  Note it MUST be called tasks.py.  It uses the decorator to wrap a class definition around a simple add function that adds two numbers and returns a result.  Note that the decorator causes the class to have the name of the function (e.g. we effectively end up with a class called add in the tasks.py).

from celery.decorators import task

@task()
def add(x, y):
    return x + y

Having completed these steps on Server and Laptop, we now need to actually test and run our AMQP workers.  Just on Server for now, do the following steps.

• Start a terminal window from the clifton directory (or wherever you put your Django app) start a celery daemon using Django.  Again, there are more options here, but this is just a starting point to show it all working.

$ python manage.py celeryd

• Now open a second terminal window and execute the following from the clifton directory.   We use the Django interpretive shell to be sure all the settings are loaded nicely for us.

$ python manage.py shell
>> from clifton.fetcher import tasks
>> result = tasks.add.delay(4,4)
>> result.ready()
True
>> result.result
8

So what just happened? Well, Celery wrapped up the add function (which simply adds x and y and returns them) in a class through the decorator.  We then, through the Django interpreter shell, imported the class, called the delay method, passing the parameters 4 and 4.  Celery then handled all the work of pushing that out onto the RabbitMQ server, then the celeryd worker actually got the values, executed them and provided the results.  The result.ready() command told us that our results for this particular instance had been processed and then result.result returned what that result actually was.  If result.ready() is False, you may find that the AMQP server is running but you don't have the worker process running.

We've just passed our first message. There is a LOT going on under the hood here, but this example is fairly uninspiring.  Shut down the celeryd terminal (send it a TERM signal, from command line do ps aux | grep celeryd then find the first process ID and do kill <pid> (e.g. kill 1234)) and see what happens if you execute the code again (perhaps with different parameters e.g. tasks.add.delay(2, 3) to see the difference).  Yup, result.ready() returns False.  It will keep returning False until you start the worker process up again.  The message has been queued and at least for this configuration, will persist until it gets processed by a worker.  Start the celeryd process again ($ python manage.py celeryd) and now try result.ready() in the shell – the message was processed and a result is able to be returned.

If you're still unsure about how cool this is, then you probably don't need queuing But for the final trick in this section until we get this thing rocking and rolling properly, lets demonstrate it really is a cluster being used to process this.

Of course by default, the cluster is up and running already (presuming you haven't stopped the Laptop), but to be sure, just check with rabbitmqctl status.  Make sure you've also run through these settings in this current section (Configure Celery with Django) on both Laptop and Server.  Now – on Server, make sure that any running celeryd terminal is shutdown.  Start the celeryd terminal on Laptop.  From the Python / Django interpretive shell on Server, repeat the same steps.

Assuming you got a result successfully returned, you've just used the message queue cluster to have a worker on Laptop pick up and process the message from Server.  The subtlety here is that the underlying RabbitMQ cluster handled passing the message between the two machines – both Server and Laptop are talking to RabbitMQ locally, it was the cluster that handled making the message available in both places.

Elapsed Time: Research, reading, implementing, testing, trouble shooting and documenting ~4 hr.  Following the above steps, prob. 1hr minutes.

Re-Factor the Task and Demo to make life easier!

Note that this previous example is of course very basic and can be significantly optimised – in particular there is no connection pooling, so every new call to add.delay() is opening its own connection to the broker, hence the delay in posting the message.  You can speed it up by passing a previously opened connection – see here for more information.  We'll implement this below.

Additionally, the Task Decorator, while useful for basic tasks, masks a lot of what's really happening – I prefer a more verbose approach when I'm learning.  So the new fetcher/tasks.py – note we've changed the name of the class to MyTest. Note that it MUST have a "run" method which is what will be called by apply_sync.

from celery.task import Task

class MyTest(Task):

    def run(self, x, y):
    logger.info('Received %s and %s' % (x, y))
        return int(x) + int(y)

We'll also make our life a bit easier by adding a bin directory into the clifton project (this is where I like to store all my cron jobs etc.) and create a simple python file to call our task.  This will let us just execute the file to generate the tasks – the sort of thing you'll need to do anyway if you want to call the tasks from a view etc.  I'll call it demo.py (for clarity, in my case this is in ~/clifton/bin/demo.py).  Copy the following code in (make any local adjustments needed).  Note the change to use apply_async instead of delay().  It is a lower level call to do the same thing as delay(), but allows us to add arguments etc.  I've also implemented the connection pooling.

#! /usr/bin/env python
from __future__ import with_statement

import os, sys
#os.environ['PYTHONPATH'] = "/home/tim/projects" # Uncomment and point this to your root django project
                                                 # directory if PYTHONPATH not set properly already.
os.environ['DJANGO_SETTINGS_MODULE']='clifton.settings'

from clifton.fetcher.tasks import MyTest
from celery.messaging import establish_connection

if __name__ == "__main__":

    numbers = [(2, 2), (4, 4), (8, 8), (16, 16)]

    results = []
    with establish_connection() as connection:
        for args in numbers:
            res = MyTest.apply_async(args=args, connection=connection)
            results.append(res)

    print([res.get() for res in results])

If you run this from the bin directory, you should see the following result:

$ python demo.py
[4, 8, 16, 32]

We now have a better structure to take our experiments forwards and further optimise.

Elapsed Time: Research, reading, implementing, trouble shooting and documenting ~2 hr.  Following the above steps, prob. 30 minutes.

Directing tasks to different servers

From here on in, this will become a little less generic and begin to deal with the problems that we are having and trying to resolve.  I'll explain why we've made our decisions and you can make your own choices.  While you can make each of these settings in settings.py on the two different machines, I suggest you create a local_settings.py and add them in there (especially if you're using SVN to sync changes – ie. running the same Django project on both servers).  Where I say settings.py below I've actually created these changes in a local_settings.py.  Note that most of these are default settings and can generally be over-ridden at different levels.

• Next install RabbitMQ – Originally I simply used Synaptic Package Manager in Ubuntu, searched for Rabbit and installed it and its dependencies, but I noticed this is version 1.6.0 and the latest here is 1.7.2 at time of writing, so I downloaded and installed the latest package instead.  I doubt it will make much difference which you use.
• Test the installation – using the following steps below which I largely copied from here 

Configure the security / vhost etc.  Note this is an important step you’ll almost certainly come back to later if you stuff up

$ sudo rabbitmqctl add_user myuser mypassword
$ sudo rabbitmqctl add_vhost myvhost
$ sudo rabbitmqctl set_permissions -p myvhost myuser “” “.*” “.*”

To start and stop the service, use the following

$ sudo rabbitmq-server

or to run it in the background (recommended)

$ sudo rabbitmq-server -detached

Finally, to stop the server

$ sudo rabbitmqctl stop

 Note – if you have trouble with the reset (because like me you tried to actually cluster the machine BEFORE the cookie was set in Erlang) you can try sudo rabbitmqctl force_reset which should sort it out.

• You now have a clustered RabbitMQ setup.  You can verify this by typing sudo rabbitmqctl status on either server and you’ll see it list something like the following:

[{running_applications,[{rabbit,"RabbitMQ","1.7.2"},
                        {mnesia,"MNESIA  CXC 138 12","4.4.10"},
                        {os_mon,"CPO  CXC 138 46","2.2.2"},
                        {sasl,"SASL  CXC 138 11","2.1.6"},
                        {stdlib,"ERTS  CXC 138 10","1.16.2"},
                        {kernel,"ERTS  CXC 138 10","2.13.2"}]},
 {nodes,['rabbit@laptop','rabbit@server']},
 {running_nodes,['rabbit@laptop','rabbit@server']}]

This is neat, but if I read the documentation correctly what we have here is a RAM / RAM cluster.  If one of the servers goes down, we will be fine because the message state is replicated across clusters, but if the whole lot went out (because the data centre lost power, or more likely in my situation that I just turned both PCs off over night) we might really want a persistent DISK node.

To convert Server to being a disk node, simply execute the following on Server (while the cluster is running).

$ sudo rabbitmqctl stop_app
$ sudo rabbitmqctl cluster rabbit@laptop rabbit@server

Or (FYI) to turn it back into a RAM node.  Note it doesn’t matter for our purposes in this doco how they are configured, read up and decide what you need.

$ sudo rabbitmqctl cluster rabbit@laptop

Finally, start it up again

$ sudo rabbitmqctl start_app

To remove a server from a cluster at any time, simply do this (this is not a required step, it’s just “FYI”).

$ sudo rabbitmqctl stop_app
$ sudo rabbitmqctl reset
$ sudo rabbitmqctl start_app

NB – You’ll need to use force_reset for the LAST node in the cluster to be removed (if you’re separating them all out again).  REMEMBER if you do a full cluster reset (like I did in testing this), you’ll need to redo the security section from the first section again   This is because you’ve reset both nodes so they no longer hold the security information.  If you ever get into problems with the configuration, I’ve resolved them consistently by de-clustering, forcing a node reset on both nodes and then clustering and redoing the security.

Elapsed Time: Research, reading, implementing and trouble shooting cluster ~2 hr.  Following the above steps, prob. 30 minutes.

Configure Celery with Django

OK, now we are ready to get Celery setup with Django. Create a new django project.  Mines called “clifton” (we use local train stations as our milestone names) and then within that I have an app called fetcher.  All of these steps need to be done on both Server and Laptop (I’m using SVN so I simply make the change on Server, commit and then update from Laptop).

• In the INSTALLED_APPS section of settings.py add celery as follows.

INSTALLED_APPS = (
    ‘django.contrib.sessions’,
    ‘django.contrib.sites’,
    ‘celery’,
    ‘clifton.fetcher’,
)

• Sync the DB as celery adds a couple of task tracking tables.

$ python manage.py syncdb

• Add the following settings into settings.py as well.  These are just to get us going, we’ll come back to this and improve them soon once we’ve got this section working.

BROKER_HOST = “localhost”
BROKER_PORT = 5672
BROKER_USER = “myuser”
BROKER_PASSWORD = “mypassword”
BROKER_VHOST = “myvhost”

• Finally, in your application directory (e.g. fetcher in my example) create a tasks.py module with the following code.  This is literally just for a test to show it’s working.  Note it MUST be called tasks.py.  It uses the decorator to wrap a class definition around a simple add function that adds two numbers and returns a result.  Note that the decorator causes the class to have the name of the function (e.g. we effectively end up with a class called add in the tasks.py).

from celery.decorators import task

@task()
def add(x, y):
    return x + y

Having completed these steps on Server and Laptop, we now need to actually test and run our AMQP workers.  Just on Server for now, do the following steps.

• Start a terminal window from the clifton directory (or wherever you put your Django app) start a celery daemon using Django.  Again, there are more options here, but this is just a starting point to show it all working.

$ python manage.py celeryd

• Now open a second terminal window and execute the following from the clifton directory.   We use the Django interpretive shell to be sure all the settings are loaded nicely for us.

$ python manage.py shell
>> from clifton.fetcher import tasks
>> result = tasks.add.delay(4,4)
>> result.ready()
True
>> result.result
8

So what just happened? Well, Celery wrapped up the add function (which simply adds x and y and returns them) in a class through the decorator.  We then, through the Django interpreter shell, imported the class, called the delay method, passing the parameters 4 and 4.  Celery then handled all the work of pushing that out onto the RabbitMQ server, then the celeryd worker actually got the values, executed them and provided the results.  The result.ready() command told us that our results for this particular instance had been processed and then result.result returned what that result actually was.  If result.ready() is False, you may find that the AMQP server is running but you don’t have the worker process running.

We’ve just passed our first message. There is a LOT going on under the hood here, but this example is fairly uninspiring.  Shut down the celeryd terminal (send it a TERM signal, from command line do ps aux | grep celeryd then find the first process ID and do kill <pid> (e.g. kill 1234)) and see what happens if you execute the code again (perhaps with different parameters e.g. tasks.add.delay(2, 3) to see the difference).  Yup, result.ready() returns False.  It will keep returning False until you start the worker process up again.  The message has been queued and at least for this configuration, will persist until it gets processed by a worker.  Start the celeryd process again ($ python manage.py celeryd) and now try result.ready() in the shell – the message was processed and a result is able to be returned.

If you’re still unsure about how cool this is, then you probably don’t need queuing But for the final trick in this section until we get this thing rocking and rolling properly, lets demonstrate it really is a cluster being used to process this.

Of course by default, the cluster is up and running already (presuming you haven’t stopped the Laptop), but to be sure, just check with rabbitmqctl status.  Make sure you’ve also run through these settings in this current section (Configure Celery with Django) on both Laptop and Server.  Now – on Server, make sure that any running celeryd terminal is shutdown.  Start the celeryd terminal on Laptop.  From the Python / Django interpretive shell on Server, repeat the same steps.

Assuming you got a result successfully returned, you’ve just used the message queue cluster to have a worker on Laptop pick up and process the message from Server.  The subtlety here is that the underlying RabbitMQ cluster handled passing the message between the two machines – both Server and Laptop are talking to RabbitMQ locally, it was the cluster that handled making the message available in both places.

Elapsed Time: Research, reading, implementing, testing, trouble shooting and documenting ~4 hr.  Following the above steps, prob. 1hr minutes.

Re-Factor the Task and Demo to make life easier!

Note that this previous example is of course very basic and can be significantly optimised – in particular there is no connection pooling, so every new call to add.delay() is opening its own connection to the broker, hence the delay in posting the message.  You can speed it up by passing a previously opened connection – see here for more information.  We’ll implement this below.

Additionally, the Task Decorator, while useful for basic tasks, masks a lot of what’s really happening – I prefer a more verbose approach when I’m learning.  So the new fetcher/tasks.py – note we’ve changed the name of the class to MyTest. Note that it MUST have a “run” method which is what will be called by apply_sync.

from celery.task import Task

class MyTest(Task):

    def run(self, x, y):
    logger.info(‘Received %s and %s’ % (x, y))
        return int(x) + int(y)

We’ll also make our life a bit easier by adding a bin directory into the clifton project (this is where I like to store all my cron jobs etc.) and create a simple python file to call our task.  This will let us just execute the file to generate the tasks – the sort of thing you’ll need to do anyway if you want to call the tasks from a view etc.  I’ll call it demo.py (for clarity, in my case this is in ~/clifton/bin/demo.py).  Copy the following code in (make any local adjustments needed).  Note the change to use apply_async instead of delay().  It is a lower level call to do the same thing as delay(), but allows us to add arguments etc.  I’ve also implemented the connection pooling.

#! /usr/bin/env python
from __future__ import with_statement

import os, sys
#os.environ['PYTHONPATH'] = “/home/tim/projects” # Uncomment and point this to your root django project
                                                 # directory if PYTHONPATH not set properly already.
os.environ['DJANGO_SETTINGS_MODULE']=’clifton.settings’

from clifton.fetcher.tasks import MyTest
from celery.messaging import establish_connection

if __name__ == “__main__”:

    numbers = [(2, 2), (4, 4), (8, 8), (16, 16)]

    results = []
    with establish_connection() as connection:
        for args in numbers:
            res = MyTest.apply_async(args=args, connection=connection)
            results.append(res)

    print([res.get() for res in results])

If you run this from the bin directory, you should see the following result:

$ python demo.py
[4, 8, 16, 32]

We now have a better structure to take our experiments forwards and further optimise.

Elapsed Time: Research, reading, implementing, trouble shooting and documenting ~2 hr.  Following the above steps, prob. 30 minutes.

Directing tasks to different servers

From here on in, this will become a little less generic and begin to deal with the problems that we are having and trying to resolve.  I’ll explain why we’ve made our decisions and you can make your own choices.  While you can make each of these settings in settings.py on the two different machines, I suggest you create a local_settings.py and add them in there (especially if you’re using SVN to sync changes – ie. running the same Django project on both servers).  Where I say settings.py below I’ve actually created these changes in a local_settings.py.  Note that most of these are default settings and can generally be over-ridden at different levels.