Frustrating experience with AWS Beanstalk

I have used Azure's Paas services before. For a friend's website, I was trying out AWS's Paas offering today. The experience is nothing less than just frustrating. I have a fully functioning rails application. It is working fine locally. However, when I tried to upload the application using the beanstalk command line, I am seeing the following error.

ERROR: [Instance: i-002808929ac3e534b] Command failed on instance. Return code: 1 Output: (TRUNCATED)...SyntaxError: Invalid CSS after "": expected selector or at-rule, was "{"
(sass):1
/opt/rubies/ruby-2.3.1/bin/bundle:23:in `load'
/opt/rubies/ruby-2.3.1/bin/bundle:23:in `

'
Tasks: TOP => assets:precompile
(See full trace by running task with --trace).
Hook /opt/elasticbeanstalk/hooks/appdeploy/pre/11_asset_compilation.sh failed. For more detail, check /var/log/eb-activity.log using console or EB CLI.

Now the complete log is not available here. You need to log into the instance and check it. However, as the lines show, it is a problem with asset pre-compilation. I tried running asset compilation locally; but I do not see this error. Interestingly, the instance created by the CLI did not allow permission for SSH. It's acceptable source was set to some other IP. It seems to me they launched their product in a hurry.

Search in AWS console

When I managed a small number of servers, looking up a server in AWS console was rather easy. However, as the number of servers grew, I started thinking of having a filter/search option. Fortunately AWS provides a nice search feature in the console.

Initial view when my instances were showing and there was no filter text.

After I added incorrect filter text, no matching instances were found.

Setting up an FTP server on AWS

Recently for testing some code, I had to host an FTP server. I tried doing it on my local first. It was easy. I just had to follow the Arch wiki for vsftpd. File transfers in both directions were working so I thought I can try it on an Amazon instance too.

My local machine ran Arch linux on while the Amazon instance ran Fedora 8. After looking up the details of the package manager for Fedora and some help from a friend, I installed vsftpd on it, applied the same config and started the FTP service. When we started testing it, we could operate successfully from command line but not from the code. From the command line, we were using active mode of operation while the code was using the passive mode, so we looked into the config to check settings related to passive mode of operation. It turned out that the passive mode is enabled by default. However, going through the various options we found an option called pasv_address. From prior experience I know that AWS machines have a private LAN IP and a separate public IP. Now, the OS on the cloud instance is not aware of what public IP it is serving. So, we suspected that in its response it must be asking the client to connect on the private LAN IP which would obviously fail. So we just set the pasv_address option to the public IP of the instance and passive mode started working fine. We could successfully connect to it and get file transfers done. So, we decided to use it for testing our code. However, when we tested it, we saw that our application was trying to post files but it was failing every time. The error we were getting each time said '500: Invalid Port command'.

The FTP protocol really goes funky with ports. It uses separate ports for control and data. The behaviour of data ports is dependent upon the mode of operation. In active mode, the client initiates the data connection and therefore the port selection is done by the client, while in passive mode, the server initiates the data connection and therefore the port selection is done by the server. We were using the passive mode of operation and the server was hitting the client at a port that later turned out to be blocked. To debug the situation, we tried connecting to the FTP server from the command line utility 'ftp' using the following command.

ftp ip address of FTP server

To turn on the passive mode and debug mode, we can use the commands 'passive' and 'debug' respectively. However, they only set the options on the client without actually sending any control data to the server. To test the FTP service, try some command that sends some control data. We went with an 'ls'. The following FTP commands were executed in sequence.

PASV
LIST

The PASV commands [1] outputs a line, like the following, indicating the port the data transfer will happen.

Entering Passive Mode (1,2,3,4,224,186)

The port has to be calculated from the last two numbers using the following formula.

n1 x 256 + n2

In the above instance, it is 224 x 256 + 186 = 57530. Once we knew that the issue was the port that the FTP server was trying to communicate to the application machine on was blocked, we decided to configure the FTP server to connect on some port within the open port range. This can be done setting the pasv_min_port and pasv_max_port options correctly in vsftpd.conf. Once we got the server connect to the client on proper ports, the transfers worked fine.

[1] A reference of FTP commands.

AWS Signature mismatch with python-boto

With python-boto 1.9b4, connecting to Amazon Cloud services using the following commands

>>>from boto.ec2.connection import EC2Connection
>>>conn = EC2Connection(<your AWS key>, <your secret key>
>>>conn.get_all_instances()

failed an error message as follows:

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/site-packages/boto/ec2/connection.py", line 119, in get_all_images
return self.get_list('DescribeImages', params, [('item', Image)])
File "/usr/lib/python2.7/site-packages/boto/connection.py", line 615, in get_list
raise self.ResponseError(response.status, response.reason, body)
boto.exception.EC2ResponseError: EC2ResponseError: 403 Forbidden
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.</Message></Error></Errors><RequestID>kept-out-of-post</RequestID></Response>

The issue disappears when you update to python-boto 2.0b4.