Wednesday, 28 August 2019

Setting up jEnv

With OpenJDK, there is a need to have multiple JDK versions installed as there are many releases that are used by different projects. More and more projects are supporting multiple Java versions as well. This problem was not as prevalent in the Java ecosystem but the Ruby ecosystem had it long back. There are multiple tools to manage Ruby versions. The approach of rbenv is probably the simplest. On the same lines, we have jEnv for Java.

To setup jEnv, the steps are simple

  1. Install jEnv (brew install jenv)
  2. Setup shell profile (~/.bash_profile)
  3. export PATH="$HOME/.jenv/bin:$PATH"
    eval "$(jenv init -)"

  4. Install Java
  5. Set local or shell version of Java (jenv local openjdk64-1.8.0.192)

Thursday, 22 August 2019

Jacoco instrumentation error

In a Java project build, I was getting an error complaining about Jacoco failing to instrument a particular class. Looking at the complete stack trace of the error, it was an exception about index going out of bounds.

Caused by: java.lang.ArrayIndexOutOfBoundsException: 6
at org.jacoco.core.internal.BytecodeVersion.get(BytecodeVersion.java:41)
at org.jacoco.core.instr.Instrumenter.instrument(Instrumenter.java:87)
at org.jacoco.core.instr.Instrumenter.instrument(Instrumenter.java:123)
... 25 more
Fortunately, a little bit of search showed that it is related to Java. My initial version of Java was '1.8.0.121'. The fix was backported to '1.8.0.172'. I also had '1.8.0.192' installed. So, I switched versions and tried a clean build. It worked fine.

Tuesday, 16 April 2019

Learning Puppet

I am learning Puppet so I wanted to try out the manifests on a local VM. So, I created a Vagrant based Arch linux VM.

Vagrant.configure("2") do |config|
 config.vm.box = "archlinux/archlinux"
 config.vm.hostname = "learn.box"
 config.vm.synced_folder "puppet/", "/home/vagrant/puppet"
end


When I tried to bring the VM up, I got the following error.

Vagrant was unable to mount VirtualBox shared folders. This is usually
because the filesystem "vboxsf" is not available. This filesystem is
made available via the VirtualBox Guest Additions and kernel module.
Please verify that these guest additions are properly installed in the
guest. This is not a bug in Vagrant and is usually caused by a faulty
Vagrant box. For context, the command attempted was:

mount -t vboxsf -o uid=1000,gid=1000 vagrant /vagrant

The error output from the command was:

: Invalid argument


It did not stop the VM from coming up so I was able to test. I created the following manifest to start with and it worked fine.

file {
 '/tmp/motd': content => 'hello puppet'
}


I moved on to actually installing a package.

class logstash{
  package{
    "logstash":
      name => "logstash",
      alias => "logstash",
      ensure => "7.0"
  }

}

I tried to apply it using the following:

puppet apply logstash.pp -v

The package was not installed. I soon figured out it was because, I was not applying the class. So, I modified the manifest to the following:

  package{
    "logstash":
      name => "logstash",
      alias => "logstash",
      ensure => "7.0"
  }


Now Puppet tried to install the package but it failed with the following error.

Error: Parameter ensure failed on Package[logstash]: Provider pacman must have features 'versionable' to set 'ensure' to '7.0' (file: /home/vagrant/logstash.pp, line: 2)

Arch linux uses pacman as its package manager and Puppet was trying to install a specific version of the package using that. However, pacman does not support different versions of the same package. So, Puppet can't ensure a specific version. So, I changed the manifest to the following:

  package{
    "logstash":
      name => "logstash",
      alias => "logstash",
      ensure => "installed"
  }

Now, Puppet attempted to install the package again and hit the following error.

Error: Execution of '/usr/bin/pacman --noconfirm --needed --noprogressbar -Sy logstash' returned 1: error: you cannot perform this operation unless you are root.
Error: /Stage[main]/Main/Package[logstash]/ensure: change from 'absent' to 'present' failed: Execution of '/usr/bin/pacman --noconfirm --needed --noprogressbar -Sy logstash' returned 1: error: you cannot perform this operation unless you are root.


So, I attempted using sudo. The process was taking time. I waited for some time then interrupted it and tried to see the logs. The logs seemed fine so I started again and the this time I got the following error.

Error: Execution of '/usr/bin/pacman --noconfirm --needed --noprogressbar -Sy logstash' returned 1: :: Synchronizing package databases...
error: failed to update core (unable to lock database)
error: failed to update extra (unable to lock database)
error: failed to update community (unable to lock database)
error: failed to synchronize all databases
Error: /Stage[main]/Main/Package[logstash]/ensure: change from 'absent' to 'present' failed: Execution of '/usr/bin/pacman --noconfirm --needed --noprogressbar -Sy logstash' returned 1: :: Synchronizing package databases...
error: failed to update core (unable to lock database)
error: failed to update extra (unable to lock database)
error: failed to update community (unable to lock database)
error: failed to synchronize all databases


This is specific to pacman. As I had interrupted the execution, the lock file was not deleted, i.e. pacman had not cleaned up properly. I manually removed the lock file from /var/lib/pacman/db.lck and tried to apply the manifest again. It completed successfully and logstash was installed on the VM. Now, it is established that the script works but it might need manual intervention. In other words, the manifest needs to be idempotent so that manual intervention is minimal.

To achieve that, I modified the manifest as follows.

  package{
    "logstash":
      name => "logstash",
      alias => "logstash",
      ensure => "installed"
  }
  file{
    "/var/lib/pacman/db.lck":
      path => "/var/lib/pacman/db.lck",
      name => "/var/lib/pacman/db.lck",
      ensure => "absent"
  }

To test that the manifest is idempotent, I removed logstash and applied the manifest. Midway, I interrupted the execution and re-applied the manifest. This time re-application succeeded. The logs show that the lock file was created when the manifest was applied the previous time and interrupted. It was deleted when the manifest was applied again.

Info: Applying configuration version '1555432879'
Info: Computing checksum on file /var/lib/pacman/db.lck
Info: /Stage[main]/Main/File[/var/lib/pacman/db.lck]: Filebucketed /var/lib/pacman/db.lck to puppet with sum d41d8cd98f00b204e9800998ecf8427e
Notice: /Stage[main]/Main/File[/var/lib/pacman/db.lck]/ensure: removed
Notice: Applied catalog in 0.25 seconds


Key take-aways:
  1. Puppet is a framework that depends on providers to install packages and its capabilities are as good as the providers.
  2. Without idempotent behaviour, Puppet manifests will not be achieve much of automation.

Monday, 21 January 2019

Finding where a package is installed

Recently, I faced an issue on a CentOS box where I had to find the location of a package. My application was running on it and I wanted to modify its config. I was unable to pick the config in /etc so I figured it must be in the folder in which the application is installed.

The application was installed as a daemon but it seems the PATH was not updated. I searched in the usual locations of binaries and packages but I could not find it. The only option left for me was to find out where it was installed.

Using the following command showed all the files of the application package.

rpm -ql <my application>

From the list, I could see where my config file was placed during install.

Friday, 18 January 2019

Unexpected exit of Elasticsearch container

I was trying the Elasticsearch container image on docker hub. I was disappointed by the complete lack of error message in the following scenario.

I had my setup defined in a compose file. When I tryed to start the container, I got only the following log line.

OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.

As it is a warning, I expected the container to start but it had not started. Digging a bit more, I found that the exit code was 137 which meant the container needed more memory than the docker daemon was configured for. The fix is quite easy of course but I think it could have been communicated better.

Friday, 5 October 2018

Correctly restarting syslog on Mac OS X

For some experimentation, I was recently looking at restarting syslog daemon. I expected it to be as simple as Linux system daemons. The init system of Mac appeared to be more like systemd in my initial searches. A few blogs instructed unload followed by load operation which sounded strange to me.

In systemd parlance, I expected unload operation to disable the daemon from starting up during initial booting. If you try to follow the process suggested in these blogs, you end up with an error stating "Operation not permitted while System Integrity Protection is engaged". This leaves two options:

  1. Disable System Integrity Protection
  2. Restart the system for every change in config
Of course, the answer is far too simple and none of these complexities need to handled. The right way of restarting the syslog daemon is as follows (using stop and start operations instead of unload and load):

sudo launchctl stop /System/Library/LaunchDaemons/com.apple.syslogd.plist
sudo launchctl start /System/Library/LaunchDaemons/com.apple.syslogd.plist

Tuesday, 25 September 2018

Hello OSGi

I found existing guides for OSGi to be a bit outdated. So, it is probably better to document the basic setup.

The plan is to first create a consumable bundle, i.e. a service bundle followed by a consumer bundle that uses the former.

I am also using maven for creating the projects so that the maven toolchain can be used for proper packaging.

For the consumable bundle, we start with defining an interface that exposes functionality of the bundle.

package main.java.interfaces;

public interface TestService {
    void hello();
}


We then implement it.

package main.java.impls;



import main.java.interfaces.TestService;



public class TestServiceImpl implements TestService {

    @Override

    public void hello() {

        System.out.println("Hello Duniya");
    }
}

We then add the activation logic for the bundle.

package main.java.provider;

import org.osgi.framework.BundleActivator;
import org.osgi.framework.BundleContext;
import org.osgi.framework.ServiceRegistration;

import main.java.interfaces.TestService;
import main.java.impls.TestServiceImpl;

public class ServiceActivator implements BundleActivator {
    private ServiceRegistration registration;

    @Override
    public void start(BundleContext bundleContext) throws Exception {
        registration = bundleContext.registerService(
                TestService.class.getName(),
                new TestServiceImpl(),
                null);
        System.out.println("ServiceActivator started");
    }

    @Override
    public void stop(BundleContext bundleContext) throws Exception {
        registration.unregister();
    }
}



We complete this bundle by adding a POM for building it.




<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>felix_test</groupId>
    <artifactId>felix_test</artifactId>
    <version>1.0</version>
    <packaging>bundle</packaging>

    <dependencies>
        <dependency>
            <groupId>org.osgi</groupId>
            <artifactId>org.osgi.core</artifactId>
            <version>6.0.0</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>2.0.2</version>
                <configuration>
                    <source>1.6</source>
                    <target>1.6</target>
                </configuration>
            </plugin>

            <plugin>
                <groupId>org.apache.felix</groupId>
                <artifactId>maven-bundle-plugin</artifactId>
                <extensions>true</extensions>
                <configuration>
                    <instructions>
                        <Bundle-SymbolicName>felix_test</Bundle-SymbolicName>
                        <Export-Package>main.java.interfaces</Export-Package>
                        <Bundle-Activator>main.java.provider.ServiceActivator</Bundle-Activator>
                        <Bundle-Vendor>Amitav Mohanty</Bundle-Vendor>
                    </instructions>
                </configuration>
            </plugin>
        </plugins>
    </build>

</project>


The package of interfaces that are to be exposed are called out along with the activator so that the maven toolchain can build our bundle properly.

We move on to create the consumer bundle next. We start with the bundle activator.

package main.java.consumer;

import main.java.interfaces.TestService;
import org.osgi.framework.BundleActivator;
import org.osgi.framework.BundleContext;
import org.osgi.framework.ServiceReference;

public class FelixTestActivator implements BundleActivator {
    @Override
    public void start(BundleContext bundleContext) throws Exception {
        ServiceReference reference = bundleContext.getServiceReference(TestService.class.getName());
        consumer = new TestConsumer((TestService)bundleContext.getService(reference));
        consumer.startTimer();
    }

    @Override
    public void stop(BundleContext bundleContext) throws Exception {
        consumer.stopTimer();
    }

    private TestConsumer consumer;
}

We then add the consumer.


package main.java.consumer;

import main.java.interfaces.TestService;

import javax.swing.Timer;
import java.awt.event.ActionEvent;
import java.awt.event.ActionListener;

public class TestConsumer implements ActionListener {
    @Override
    public void actionPerformed(ActionEvent e) {
        service.hello();
    }

    public TestConsumer(TestService service){
        super();
        this.service = service;
        timer = new Timer(1000, this);
    }

    public void startTimer(){
        timer.start();
    }

    public void stopTimer(){
        timer.stop();
    }

    private final TestService service;
    private final Timer timer;

}


The initiation of the consumer happens when the bundle is started.

The POM for building the consumer bundle is as follows:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>consumer</groupId>
    <artifactId>consumer</artifactId>
    <version>1.0</version>
    <packaging>bundle</packaging>

    <dependencies>
        <dependency>
            <groupId>org.osgi</groupId>
            <artifactId>org.osgi.core</artifactId>
            <version>6.0.0</version>
        </dependency>

        <dependency>
            <groupId>felix_test</groupId>
            <artifactId>felix_test</artifactId>
            <version>1.0</version>
            <scope>system</scope>
            <systemPath>${basedir}/../target/felix_test-1.0.jar</systemPath>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>2.0.2</version>
                <configuration>
                    <source>1.6</source>
                    <target>1.6</target>
                </configuration>
            </plugin>

            <plugin>
                <groupId>org.apache.felix</groupId>
                <artifactId>maven-bundle-plugin</artifactId>
                <extensions>true</extensions>
                <configuration>
                    <instructions>
                        <Bundle-SymbolicName>consumer</Bundle-SymbolicName>
                        <Bundle-Activator>main.java.consumer.FelixTestActivator</Bundle-Activator>
                        <Bundle-Vendor>Amitav Mohanty</Bundle-Vendor>
                    </instructions>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

The dependency on the other bundle is called out.

The "packaging" node in the POMs is essential for creating OSGi bundles.

Wednesday, 9 May 2018

Backward compatibility in user experience

Gmail recently launched a new design. I tried it out of curiousity. The first thing I noticed was that my muscle memory started failing me. Every time a new design comes, people focus on the look and feel of the design, its intuitiveness and other such factors but its compatibility with old design is not taken into account ever.

Power users tend to build muscle memory even for UIs. I tend to move my mouse to a particular location on the screen for the delete button for example.

With a product that has so many existing users, I think GMail should pay attention to these details as well when planning a new design. So, should other companies with similar products. An epic failure in the same lines is removal of start button by Microsoft. Of course a button position is not as critical; but the point is to have this criterion included in the design process.

Wednesday, 31 January 2018

Testing filebeat

For testing codecs like multiline, the recommendation is to try the Go playground website. However, before making a config live, it can be tested locally also; using input from stdin and printing output to the console. The following config can be used as skeleton.

filebeat.prospectors:
- input_type: stdin

output:
  console:
    pretty: true