Memory snapshot in Jetbrains RubyMine on linux

I use RubyMine on linux for Ruby on Rails development. Of late, it had been hanging up frequently. I reported this to Jetbrains. They got back to me asking me to provide more details so that the concerned developer can try and find more details about the issue. On a side note, their customer support was so fast in responding, I was amazed. Also, the guy responding back was pretty technical himself.

Getting back to the topic, they had asked me to provide a memory snapshot. The process of generating a memory snapshot is described here. However, that process requires users to download YourKit Java Profiler, which apart from being a large download, comes with a 15-day license. To me it did not make sense. So, I got back to them about it and it turns out on linux, you don't really need it. The linux version of RubyMine, comes with the profiler libraries bundled.

$RUBYMINE_HOME/bin/libyjpagent-linux.so
$RUBYMINE_HOME/bin/libyjpagent-linux64.so

To enable its usage all you need to do is, edit the following script and set IS_EAP to "true".

$RUBYMINE_HOME/bin/rubymine.sh

Restarting RubyMine after that change, will show the memory and CPU snapshot icons. It is also advised to provide thread dumps, as described here, along with the memory snapshot.

Extracting logs out of journalctl

Journalctl gives us nice consolidated logs. However, on a number of occasions, we need to extract parts of the logs. There are multiple ways of doing it. To filter by process, you can use PID numbers as shown below.

journalctl _PID=<pid number>

To obtain PID number when you have the process name [or part of the process name], use the following:

ps aux | grep -i <process name>

The manual page only refers to it in examples. A commonly used slicing option is to see logs of current boot only. This can be done as follows.

journalctl -b

Another option is to look at logs of a particular unit only. This can be done in the following way.

journalctl -u <unit name>

The unit name could be some daemon name like 'mysqld'. Unfortunately, this does not work with 'kernel' as a unit. It can be combined with the -b option though. However, I find myself dealing with messages from various units. So, I scan through all messages and find the messages I need. To filter them out, I can use time stamps in the messages using the following format.

journalctl --since='2013-03-06 22:58:34' --until='2013-03-06 23:00:34'

The beginning time stamp works fine; but the ending time stamp does not work. I talked about it at #systemd IRC channel. It is fixed and will be released soon.

Setting up an FTP server on AWS

Recently for testing some code, I had to host an FTP server. I tried doing it on my local first. It was easy. I just had to follow the Arch wiki for vsftpd. File transfers in both directions were working so I thought I can try it on an Amazon instance too.

My local machine ran Arch linux on while the Amazon instance ran Fedora 8. After looking up the details of the package manager for Fedora and some help from a friend, I installed vsftpd on it, applied the same config and started the FTP service. When we started testing it, we could operate successfully from command line but not from the code. From the command line, we were using active mode of operation while the code was using the passive mode, so we looked into the config to check settings related to passive mode of operation. It turned out that the passive mode is enabled by default. However, going through the various options we found an option called pasv_address. From prior experience I know that AWS machines have a private LAN IP and a separate public IP. Now, the OS on the cloud instance is not aware of what public IP it is serving. So, we suspected that in its response it must be asking the client to connect on the private LAN IP which would obviously fail. So we just set the pasv_address option to the public IP of the instance and passive mode started working fine. We could successfully connect to it and get file transfers done. So, we decided to use it for testing our code. However, when we tested it, we saw that our application was trying to post files but it was failing every time. The error we were getting each time said '500: Invalid Port command'.

The FTP protocol really goes funky with ports. It uses separate ports for control and data. The behaviour of data ports is dependent upon the mode of operation. In active mode, the client initiates the data connection and therefore the port selection is done by the client, while in passive mode, the server initiates the data connection and therefore the port selection is done by the server. We were using the passive mode of operation and the server was hitting the client at a port that later turned out to be blocked. To debug the situation, we tried connecting to the FTP server from the command line utility 'ftp' using the following command.

ftp ip address of FTP server

To turn on the passive mode and debug mode, we can use the commands 'passive' and 'debug' respectively. However, they only set the options on the client without actually sending any control data to the server. To test the FTP service, try some command that sends some control data. We went with an 'ls'. The following FTP commands were executed in sequence.

PASV
LIST

The PASV commands [1] outputs a line, like the following, indicating the port the data transfer will happen.

Entering Passive Mode (1,2,3,4,224,186)

The port has to be calculated from the last two numbers using the following formula.

n1 x 256 + n2

In the above instance, it is 224 x 256 + 186 = 57530. Once we knew that the issue was the port that the FTP server was trying to communicate to the application machine on was blocked, we decided to configure the FTP server to connect on some port within the open port range. This can be done setting the pasv_min_port and pasv_max_port options correctly in vsftpd.conf. Once we got the server connect to the client on proper ports, the transfers worked fine.

[1] A reference of FTP commands.

Getting to know the Syslog protocol

Recently I was looking into FTP issues, when I learnt some details about Syslog. I was using vsftpd for hosting an FTP service. I had enabled logging but I was not seeing anything in journalctl output. The reason for that turned out to be a configuration flaw. I had not turned on the option for vsftpd to use syslog. Once I turned the option on, proper log files were created.

I do not have Syslog-ng or any other syslog package installed. I had uninstalled it when I switched to systemd. So, I had not enabled that option in vsftpd. However, as it turns out when I turn that option on vsftpd uses the syslog protocol for logging. Systemd listens for messages sent using that protocol and creates appropriate logs.

This is a great way of unifying all logging. Individual packages do not have to bother about logging. They just act as clients of the protocol and the listener will take care of maintaining the logs. There are instances of similar architecture being followed for logging in other domains too.

As it turns out, there are standardized versions of the syslog protocol:

• Syslog protocol
• BSD Syslog protocol

The version used commonly is the BSD one even the former is more advanced. Now syslog is being replaced by systemd's journal because it capitalizes over syslog. It provides efficient transfer of binary data and supports JSON. Maintaining logs is much easier.

It was interesting to know that even though syslog packages are becoming obsolete, the syslog protocol is still the logging standard. It is actually a nice example of robust architecture surviving over the years.

Kill all zombie processes of a process

Time and again I have found phpmyadmin not working because there are a lot of zombies of httpd. I do not know yet why these many instances of the daemon show up and why they turn into unresponsive zombie processes. Usually when this happens, I just kill all the zombies and spawn a new daemon and I carry on with my work. I used to kill all the zombies one at a time. However, today I figured out that I can do it with a single line.

ps aux | grep http | awk -F " " '{print $2}' | xargs kill -9

[If the working is clear to you, do read further and let me know if I can improve it or if I am interpreting anything incorrectly although things are somehow working.]

The one-liner above is easy to understand once we have understand the pieces. So, I am listing that below.

• ps aux lists all running processes
• grep http finds the lines containing the string 'http'
• awk -F " " '{print $2}' splits each input line by delimiter specified with -F flag, space in this case and prints the second token thus obtained
• kill -9 send SIGKILL signal to the processes whose ids are specified as arguments
• xargs takes output of previous command and makes it input for the next

You should probably have a look at the man pages for more details on the commands.

Rails is re-inventing the wheel

In recent years, there has been a surge in web applications. To support the growing market, frameworks have developed around scripting languages for developing web applications fast. Of those Ruby on Rails seems to be the most mature. Django, the most advanced Python framework for web applications is yet to come at par with it.

Be it Django or Ruby on Rails, both follow a model-view-controller architecture to web applications. Now, many web applications using these frameworks follow 'skinny controllers, fat models'. As a result the models become home to a lot of business logic. Models are intended to serve only as an abstraction to the database. They are meant to be 'models of data'. Also, when the application grows huge, we run into issues of scalability and we resort to techniques like sharding. Now, there are of course multiple ways of sharding and you need to decide what suits you better. Here, I list a few ways for Rails applications.

• data_fabric
• octopus
• Spock proxy

Now, sometimes the sharding logic also creeps into the models. Obviously objects of model classes do not have a single purpose any more. Over time, the model classes obviously become too complicated. It becomes difficult to maintain and debug. The Rails community is becoming increasingly aware of these issues. There are multiple ways in which developers are trying to move business logic away from models. Here I list of couple of those ways.

• use cases
• concerns

Looking at the overall scenario reminds me of building N-tier applications using Java Enterprise Edition. The sharding logic and details of data fetching can be a tier below models and the business logic can be a tier above it. ' seems to me developers using web frameworks like Django and Rails are just re-inventing the wheel. Also, Java has optimized garbage collection which can be tuned to various needs. In Ruby garbage collection seems to be a big issue, though good work in the area is coming in the next version.

So much in the name of innovation, eh? In case my ideas seem to waver from reality do point me in the right direction.

Update: You can follow the discussion on Hacker News on the topic.

Rails time formats

During programming, I do tend to forget the various time formats available in Rails. Also, there are so many options available that a simple typo like capitalizing (or not) changes meaning. It is best that this part be always tested properly.

Apple Magic Mouse with Dell Latitude E6520

Recently, I found some free time at office and my colleague's Apple magic mouse was lying on my table. It sure is a cool device. So, I thought why not connect it to linux and see how it works. So, I turned on the bluetooth daemon on my system with the following command.

systemctl start bluetooth

I use Bluedevil. So, I clicked on the tray icon and ran a scan for bluetooth devices in range. The pairing key required is 0000. Initially the mouse pointer movement was too fast and scroll was too slow. So, I changed the configuration using this blog.

It is cool, indeed. However, I found it difficult for regular use because I am not used to it and it is a touch mouse unlike other mice. When I am moving the mouse, I have to make sure my index finger is not touching the surface, else there is unwanted scrolling.

Enjoying writing

Recently while waiting for a flight at Mumbai domestic airport, I decided to write down my thoughts. Of late, I have been writing on some keyboard mostly but this time I was writing with a pen and I actually enjoyed doing it. So, I write  about writing only. This is what I wrote:

Is it possible to write with this pen?
Well not completely impossible. Also I realize the benefit of writing with a thick pen. With a thick pen, you tend to write each letter well while with a thin pen, you write a letter away [especially the letters towards the end of word]. [It] Reminds me of a friend who recently advised me to try a thick pen and I have to admit I am really loving this experience. At school, I hated writing with a thick pen because it slowed me down. However, now that its not my writing speed by my writing that I care about, I think I will use this thick pen more often.

Compared typing on a keyboard, while penning down the flow of thoughts was better. I liked it and decided to do it more often.