Benchmarking elasticsearch and logstash pipeline

Elasticsearch and logstash pipelines can be elaborate or simple. Depending upon the setup, end to end benchmarking should be done time to time. One way to do it is to have a marker document (log line). We need to track when the marker is introduced into the pipeline and finally when it becomes available to query.

import urllib2

import json

from datetime import datetime

import sys

import time




print "Started at " + str(datetime.now())

if len(sys.argv) < 2:

   print "URL not specified.\nUsage: watch.py "

   exit(1)

count = 0

while count < 1:

    resp = urllib2.urlopen(sys.argv[1]).read()

    count = json.loads(resp)["hits"]["total"]

    if count > 0:

        print "Found at " + str(datetime.now())

        break

    time.sleep(2)

The above script can be used as follows:

python <script_file> "http://<host_name>:<port_number>/_search?q=message:<markerMessage>"

Toggle Read-only behaviour of buffer in Emacs

When a file to which you have read access as regular user is opened in emacs, the buffer is marked read-only. If the user escalates to root and opens the same file in emacs, the buffer still remains read-only. To modify the file, the user needs to toggle the read-only behaviour of the buffer (for which the default key binding is C-x C-q) first.

Increasing the maximum number of file descriptors allowed per user on Cent OS 6

For some applications like elasticsearch, the maximum number of file descriptors needs to quite high. The way of modifying the value varies from distro to distro. In Cent OS6, the following did not work.

1. Using sysctl as root
sysctl -w fs.file-max=100000

2. Using ulimit as root

After the above approaches, when you check the value using the following, the value will show up.

cat /proc/sys/fs/file-max

However, setting the values as root, does not change the values for other regular users. To set it specifically for any user, change the value in /etc/security/limits.conf file. Setting only the soft or hard limit will not be sufficient. Therefore, the following lines should be added.

<user> soft nofile <value>

<user> hard nofile <value>

After the modifications, it is required to logout and log back into the system.

Manually install maven dependency

Error messages from Maven are often useless. The root cause is mostly never captured. Debugging issues might require manually installing a dependency. The way to do it is as follows:

mvn install:install-file -DgroupId=jnuit -DartifactId=junit -Dversion=3.8.1 -Dpackaging=jar -Dfile=/path/to/file

Correct do...while in Ruby

Rubyists use the .each or .map way of looping frequently. However, when there is a need of a while loop, the following is the correct way.
loop do
    # statements
    break if condition
end

Getting method name within the same method

For logging in some server side javascript, I had the need of getting the name of the method within the same method. There are 2 ways of doing this:
- accessing arguments.callee.toString() and applying regex extraction to get the name
- accessing arguments.callee.name

It does not always work though. ECMA strict mode does not allow it but their is no alternative either.

Unable to locate "adwaita" theme engine

While running KDE on Arch linux, on a number of occassions, I get to see the following error.

(java:20732): Gtk-WARNING **: Unable to locate theme engine in module_path: "adwaita"

[828:828:0708/183334:ERROR:browser_main_loop.cc(249)] GTK theme error: Unable to locate theme engine in module_path: "adwaita"

The mitigation of the issue is installation of gnome-themes-standard package.

[Newtonsoft] Checking for null valued keys in parsed JObject

Using Newtonsoft's Json package, when we need to parse a string to get a JObject, we can do it as follows:

JObject o = JObject.Parse(serializedJsonString);
 
This is of course the case when you don't have a model class for the Json response coming in. For keys that can have a null value, checking for null values as follows is intuitive.

o["key"] == null

However, it is also incorrect. Due to the way in which null values are stored in the parsed object, the proper way of checking for null valued keys is as follows:

o["key"].Type == JTokenType.Null
 
Of course, you should do both checks and the former check should be done earlier to ensure that the key exists before checking whether the value is null.

using Newtonsoft.Json;

using Newtonsoft.Json.Linq;

JObject o1 = JObject.Parse(@"{

    'key1' : null,

    'key2':{}

}");

System.Console.WriteLine(o1["key1"].Type == JTokenType.Null); // this is true

System.Console.WriteLine(o1["key3"] == null); // this is true too

So, search for a key returning a null means that particular key is not there in the json.