Shorten ssh command using config file

If you are accessing remote servers frequently using ssh. You can save your ssh configuration (server Ip, port, user, key) to a file and reuse this configuration without entering these details repeatedly. This provides cleaner approach for ssh commands than creating command aliases (ex: alias qa-services='ssh -i ~/.ssh/key/services.pem ubuntu@124.17.27.185 -p 22'  )

SSH client obtains configuration data from the following sources in the bellow order:

           1.   command-line options
           2.   user's configuration file (~/.ssh/config)
           3.   system-wide configuration file (/etc/ssh/ssh_config)


We are going set the configuration in user's configuration file

Create a ssh client user configuration file in '~/.ssh/' directory with file name 'config'

Add following content to the file (~/.ssh/config)

 Host qa-services  
    HostName 124.17.27.185  
    User ubuntu  
    Port 22  
    IdentityFile ~/.ssh/key/services.pem  
 Host prod-db  
    HostName 124.21.151.26  
    User ubuntu  
    Port 22  
    IdentityFile ~/.ssh/key/db.pem  

Configuration keywords :
              HostName - hostname or public ip address.
     User - Specifies the user to log in as.
     Port - Specifies the port number to connect on the remote host.
     IdentityFile - Specifies a file from which the user's DSA, ECDSA, Ed25519 or RSA authentication identity is read.

Save the config file. Now you can access to your servers with simple commands like below.

 ssh qa-services 

 ssh prod-db

HMAC Authentication

In this short article I'm going to explain about HMAC authentication and how it works.

What is HMAC authentication? 

HMAC (Hash-based message authentication code) authentication provides a simple way to authenticate and verify the data integrity of a HTTP request using cryptographic hash function in combination with a secret key that is known to client and server.

We can use any cryptographic hash function such as MD5 or SHA-1 to do the calculation of an HMAC, we term the resulting algorithm as  HMAC-MD5 or HMAC-SHA1 accordingly. 

Why HMAC?

If you want to have a secure communication between client and server which guarantee the authenticity and data integrity of the request. Then HMAC is the right solution for you. 

How HMAC authentiation works?

As you can see in the diagram. Both Client and server have a shared secret key. Client will use this key to calculate the signature of the message using a cryptographic hash function, then message + header will be sent to the server. Once server receive the message it will also start calculating the HMAC signature of the message using the shared secret key. Now it will verify the signature it receives and the result signature calculated at the server end. If both match, request will be accepted otherwise it will reject by the server. 

Advantages of cryptographic hash function

• Easy to compute the hash value for any given message  -   HMAC signature calculation take less computing power.

• In-feasible to generate a message from its hash   -  Different message attributes can be used to calculate the signature, so with this feature no one can identify the way we calculate the HMAC signature.

• In-feasible to modify a message without changing the hash   -   No one can modify the message without knowing the key. 

• In-feasible to find two different messages with the same hash   -   No one can modify the message. 

This doesn't grantee the replay attack, in order to avoid that we can include time-stamp while calculating the signature, It will help to expire the message after certain time period.

I hope now it is clear how HMAC works and how it will guarantee the authenticity and data integrity of the request.

Ubuntu - Gnu parallel - It's awesome

GNU parallel is a shell package for executing jobs in parallel using one or more nodes. If you have used xargs in shell scripting then you will find it easier to learn GNU parallel,
because GNU parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel.

To install the package

sudo apt-get install parallel

Here is an example of how to use GNU parallel.

If you have a directory which is having large log files and if you need to compute no of lines per each file and get the largest file. You can do it efficiently with GNU Parallel and it can utilize all your cpu cores in the server very efficient way.

In this case most heavier operation is calculating the number of lines of each file, instead of doing this operation sequentially we can do this operation parallely using GNU Parallel.

Sequencial way

ls | xargs wc -l | sort -n -r | head -n 1

Parallel way

ls | parallel wc -l | sort -n -r | head -n 1


This is only one example, like this you can optimize your operations using GNU parallel. :)

Shell script edited on windows - Issue when executing on linux

I faced the above issue when I'm trying to execute the script after doing some editing on windows. Those two issues are due to BOM character and the carriage return (\r) present in the file.

• BOM (byte order mark) character -  This is a Unicode character used to signal the order of bytes in a text file or stream.
• Carriage return (\r) -  Editors use in windows needs '\r' and '\n' both the characters together to interpret as new line, which is ‘\r\n’. But unix understand only (\n).

These above characters use in windows, but unix shell scripts won't understand those characters. Because of that you might face issues when running a bash script editted in windows. To fix this you need to remove those characters. This is how you can do it.

BOM character issue
You might see following issue from the first line of the script.

": No such file or directory1: #!/bin/bash" - If you get such kind of issue from 1st line of your script then you can cross-check the script and if there is no any visible issue. you can run the
following command.

$ head -n 1 your_script.sh | LC_ALL=C od -tc
$ 0000000 357 273 277   #   !   /   b   i   n   /   b   a   s   h  \r  \n

In the output if you can see "357 273 277" sequence, then this is the BOM character. So you need to remove it.

* Open the script using vim
* Type this and enter in the first line - ":set nobomb" - this will remove the BOM character from ur file.
* save the file and close - :wq

Carriage return issue

Carriage return present in the script might throw this issue.

"$'\r': command not found"
"syntax error near unexpected token `$'do\r''"

To fix this you need to remove the \r characters from your script. Use any unix way to replace \r character with empty string.

* String replace using sed command

$ sed -i 's/\r//g' your_script.sh

* String replace using perl

$ perl -pi -e 's/\r//g' your_script.sh


Now the script is ready to run in unix :)

Article on Monitor Your Key Performance Indicators using WSO2 BAM.

I've written an article for WSO2 library explaining how to Monitor your Key Performance Indicators  using WSO2 BAM 

WSO2 BAM an enterprise-ready, fully-open source, complete solution for aggregating, analyzing and presenting information about business activities also it supports big data analytics and storage capability via Apache Hadoop, Hive and Cassandra.

This article focuses on KPI monitoring via WSO2 BAM. The article flows based on the following topics.

• Introduction
• BAM architecture
• Use case
• KPIs for this use case
• Collect information for the usecase.
• BAM data-agent (Java API)
• Non Java Data-agent
• REST API
• Viewing collected information using Cassandra explorer
• Data Analysis
• Writing a hive script for analyzing captured data
• Visualizing the KPIs.

Configuring Hive metastore to remote database - WSO2 BAM2

Hive Metastore

Hive metastore is the central repository which is used to store Hive metadata. We use embedded H2 database as the default hive metastore. Therefore only one hive session can access the metastore. 


Using remote MYSQL database as Hive metastore. 

You can configure hive metastore to MYSQL database as follows. 

Edit hive-site.xml located at WSO2_BAM2_HOME/repository/conf/advanced/ directory.

<property>
  <name>javax.jdo.option.ConnectionURL</name>
  <value>jdbc:mysql://localhost/test_database</value>
  <description>JDBC connect string for a JDBC metastore</description>
</property>
<property>
  <name>javax.jdo.option.ConnectionDriverName</name>
  <value>com.mysql.jdbc.Driver</value>
  <description>Driver class name for a JDBC metastore</description>
</property>
<property>
  <name>javax.jdo.option.ConnectionUserName</name>
  <value>root</value>
  <description>username to use against metastore database</description>
</property>
<property>
  <name>javax.jdo.option.ConnectionPassword</name>
  <value>root</value>
  <description>password to use against metastore database</description>
</property>

Put MYSQL driver into WSO2_BAM2_HOME/repository/components/lib

Now You have successfully configured the hive metastore to MYSQL database. Now restart the BAM server. 

A Fix for Huawei E220 connection issue with ubuntu 12.04

After installing Ubuntu 12.04, I faced an issue when connecting to the internet from my Huawei E220 dongle. So I did some google search and found a bug report relating this[1]. After going through this issue I found a workaround which fix the issue.

This is the workaround.

You should execute following command as root.

echo -e "AT+CNMI=2,1,0,2,0\r\nAT\r\n" > /dev/ttyUSB1 

Now try to connect your dongle again, it works for me until dongle is removed from USB port. Thanks Nikos for your workaround :)

[1] https://bugs.launchpad.net/ubuntu/+source/modemmanager/+bug/868034