Benchmarking elasticsearch and logstash pipeline

Elasticsearch and logstash pipelines can be elaborate or simple. Depending upon the setup, end to end benchmarking should be done time to time. One way to do it is to have a marker document (log line). We need to track when the marker is introduced into the pipeline and finally when it becomes available to query.

import urllib2

import json

from datetime import datetime

import sys

import time




print "Started at " + str(datetime.now())

if len(sys.argv) < 2:

   print "URL not specified.\nUsage: watch.py "

   exit(1)

count = 0

while count < 1:

    resp = urllib2.urlopen(sys.argv[1]).read()

    count = json.loads(resp)["hits"]["total"]

    if count > 0:

        print "Found at " + str(datetime.now())

        break

    time.sleep(2)

The above script can be used as follows:

python <script_file> "http://<host_name>:<port_number>/_search?q=message:<markerMessage>"

Uninstalling python packages

Continuing my system cleanup, I decided to get rid of all unnecessary python packages.

pip2 list | awk -F ' ' '{print $1}' | grep -vE "django-paypal|boto|mercurial|MySQL-python|nltk" | xargs pip2 uninstall -y

The -y flag indicates confirmation of uninstallation.

Rails is re-inventing the wheel

In recent years, there has been a surge in web applications. To support the growing market, frameworks have developed around scripting languages for developing web applications fast. Of those Ruby on Rails seems to be the most mature. Django, the most advanced Python framework for web applications is yet to come at par with it.

Be it Django or Ruby on Rails, both follow a model-view-controller architecture to web applications. Now, many web applications using these frameworks follow 'skinny controllers, fat models'. As a result the models become home to a lot of business logic. Models are intended to serve only as an abstraction to the database. They are meant to be 'models of data'. Also, when the application grows huge, we run into issues of scalability and we resort to techniques like sharding. Now, there are of course multiple ways of sharding and you need to decide what suits you better. Here, I list a few ways for Rails applications.

• data_fabric
• octopus
• Spock proxy

Now, sometimes the sharding logic also creeps into the models. Obviously objects of model classes do not have a single purpose any more. Over time, the model classes obviously become too complicated. It becomes difficult to maintain and debug. The Rails community is becoming increasingly aware of these issues. There are multiple ways in which developers are trying to move business logic away from models. Here I list of couple of those ways.

• use cases
• concerns

Looking at the overall scenario reminds me of building N-tier applications using Java Enterprise Edition. The sharding logic and details of data fetching can be a tier below models and the business logic can be a tier above it. ' seems to me developers using web frameworks like Django and Rails are just re-inventing the wheel. Also, Java has optimized garbage collection which can be tuned to various needs. In Ruby garbage collection seems to be a big issue, though good work in the area is coming in the next version.

So much in the name of innovation, eh? In case my ideas seem to waver from reality do point me in the right direction.

Update: You can follow the discussion on Hacker News on the topic.

Using Emacs for development

When I started working with linux, I had the choice of the following text editors:

• KEdit
• Vim
• Emacs

I decided to try all of them. After initial try, I was certain I would not use vi. It just did not suit my tastes. KEdit turned out to be limited in functionality. So, Emacs became my editor of choice. Initially, I was using it for trivial tasks only. Gradually as I knew more about it, I started using it more.

Compiling and debugging in emacs was fine. However, I was missing code navigation features which really come in handy during code reading phases. Searching through the world wide web, I found this blog explaining the use of etags with emacs for code navigation. I gave it a shot and it surely is fast. While working with python, I found emacs' support for python is not that good; but I am sure it will improve soon.

AWS Signature mismatch with python-boto

With python-boto 1.9b4, connecting to Amazon Cloud services using the following commands

>>>from boto.ec2.connection import EC2Connection
>>>conn = EC2Connection(<your AWS key>, <your secret key>
>>>conn.get_all_instances()

failed an error message as follows:

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/site-packages/boto/ec2/connection.py", line 119, in get_all_images
return self.get_list('DescribeImages', params, [('item', Image)])
File "/usr/lib/python2.7/site-packages/boto/connection.py", line 615, in get_list
raise self.ResponseError(response.status, response.reason, body)
boto.exception.EC2ResponseError: EC2ResponseError: 403 Forbidden
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.</Message></Error></Errors><RequestID>kept-out-of-post</RequestID></Response>

The issue disappears when you update to python-boto 2.0b4.

Altering django models

I wanted to add a column to an existing django model; but django does not support modifications to existing models so when I added a line to models.py and ran syncdb, I found no change. The workaround was to manually edit the schema in mysql (or your preferred database backend).

alter table <table name> add column <column name> varchar(512);

Using CSS files in Django

In Django, cascading style sheets are served as static content. To begin with create the following URL mapping in your urls.py


from django.conf import settings

urlpatters = patterns (
   #other patterns
   (r'^static/(?P<path>.*)$', 'django.views.static.serve', {'document_root': settings.MEDIA_ROOT, 'show_indexes': True}),
)

In settings.py set the following variables.

MEDIA_ROOT = '/home/user/site/static'
STATIC_URL = '/css/'

All css files are put in the css folder and can be referred to in templates as /static/css/filename.css.