Elasticsearch shard allocation failure

 Recently, I found a shard allocation failure in an Elasticsearch cluster. Checking the reason for the failure, I found the reason to be the following:

"Validation Failed: 1: this action would add [4] total shards, but this cluster currently has [4000]/[4000] maximum shards open;"

It is a bit unclear from the error message why the cluster considers 4000 as the maximum number of shards it should consider. Especially, given that on a different cluster I get 16000 as the maximum number of shards.

Looking at my cluster settings (including defaults), I found a property called "max_shards_per_node" and it was set to "1000". Since, I had 4 nodes, it added up perfectly.

Generally, this setting is put in so that the nodes have a proper balance between different types of work loads. However, we knew our scenarios of more shards was temporary so increasing the value to 1500 worked for the short term.

Unexpected exit of Elasticsearch container

I was trying the Elasticsearch container image on docker hub. I was disappointed by the complete lack of error message in the following scenario.

I had my setup defined in a compose file. When I tryed to start the container, I got only the following log line.

OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.

As it is a warning, I expected the container to start but it had not started. Digging a bit more, I found that the exit code was 137 which meant the container needed more memory than the docker daemon was configured for. The fix is quite easy of course but I think it could have been communicated better.

Increasing the maximum number of file descriptors allowed per user on Cent OS 6

For some applications like elasticsearch, the maximum number of file descriptors needs to quite high. The way of modifying the value varies from distro to distro. In Cent OS6, the following did not work.

1. Using sysctl as root
sysctl -w fs.file-max=100000

2. Using ulimit as root

After the above approaches, when you check the value using the following, the value will show up.

cat /proc/sys/fs/file-max

However, setting the values as root, does not change the values for other regular users. To set it specifically for any user, change the value in /etc/security/limits.conf file. Setting only the soft or hard limit will not be sufficient. Therefore, the following lines should be added.

<user> soft nofile <value>

<user> hard nofile <value>

After the modifications, it is required to logout and log back into the system.