Upload the Maven's artifacts to the Jfrog Artifactory

The below simple example shows how to upload the maven build artifacts to the Jfrog artifactory.

1. First download the Jfrog  OSS artifactory from https://jfrog.com/open-source/ link and download ZIP to your computer.

2. Unzip the file and run the artifactory.bat from the bin folder inside the application.

3. Once artifactory is run, then go to the browser and hit http://localhost:8080 or your customized port location

4. After the artifactory is up and running, create the artifact repository one for snapshots and one for releases. For example maven-releases for releases and maven-snapshots for snapshots repository.

5. In your maven project pom.xml put the distribution management as shown below

    <distributionManagement>
        <repository>
            <id>releases</id>
            <url>http://localhost:8080/artifactory/maven-releases</url>
            <name>maven-releases</name>
        </repository>
        <snapshotRepository>
            <id>snapshots</id>
            <url>http://localhost:8080/artifactory/maven-snapshots</url>
            <name>maven-snapshots</name>
        </snapshotRepository>
    </distributionManagement>

6. Now in the maven's setting ${MAVEN_HOME}/conf/settings.xml, set the credentials for the artifactory inside servers setting as:

    <server>
      <id>releases</id>
      <username>user</username>
      <password>pass</password>
    </server>

    <server>
      <id>snapshots</id>
      <username>user</username>
      <password>pass</password>
    </server>

7. Now in your maven project's root start the SNAPSHOT deployment using the command:

$ mvn clean package deploy

-this command will deploy the artifacts generated (usually the jars) to the maven-snapshots of the artifactory

8. To deploy the RELEASE we would need to do

$ mvn release:clean release:prepare release:perform

- for this to work, we would also need to hook up the source code repository in scm element in the pom.xml file as:

    <scm>
        <connection>scm:git:https://github.com/your-accountt/your-project.git</connection>
        <url>http://github.com/your-account/your-project</url>
        <developerConnection>scm:git:https://github.com/your-account/your-project.git</developerConnection>
        <tag>HEAD</tag>
    </scm>

you can also connect to other source code repository instead of the git repository. The credential can also be provided same as for the artifactory in the maven's settings.xml file.

we would probably also need to add the following plugin inside the plugins in the pom.xml in-order to make this work

<plugin>
      <groupId>org.apache.maven.plugins</groupId>
       <artifactId>maven-release-plugin</artifactId>
       <version>2.5.1</version>
        <configuration>
              <tagNameFormat>v@{project.version}</tagNameFormat>
              <autoVersionSubmodules>true</autoVersionSubmodules>
        </configuration>
</plugin>

9. If everything is OK, then the maven project should build, upload the artifacts to Artifactory's releases repository and update the source code in the scm repository.

10. We can also deploy to the Nexus Repository Manager OSS maven repository in the same way. Follow the link https://www.sonatype.com/download-oss-sonatype to download the Nexus Repository.

That's All!

How to SSH from one Docker Container to Other Docker Container

How to SSH from one Docker Container to Other Docker Container

I have tried to deploy the application that is build on one docker and push the final artifact to other docker container and came across the question of how to push to other docker. I searched a little bit and here is how I finally able to ssh to other docker container

Let's say we have two docker containers A and B running in the host environment and I want to do the SSH from docker A to docker B.

Here is the following things that I need in-order to do the SSH to docker B. Make sure both dockers are downloaded and running. Let say I have downloaded the nodejs docker and run as:

$docker run -it --rm -name A -p 50022:22 node /bin/bash
$docker run -it --rm -name B -p 50023:22 node /bin/bash

1. Find the docker container's ip address as below:

$docker inspect --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' A
$172.17.0.1

$docker inspect --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}'  B
$172.17.0.2

2. Inside the docker container B
$ apt-get update
$ apt-get install ssh

3. Change the following configuration in /etc/ssh/sshd_config file
PasswordAuthentication from 'no' to 'yes'
PermitRootLogin from 'no' to 'yes'

4. Then run the ssh service inside container B as
$service ssh start

5. Change the root password as:
$passwd root
[Enter new password]: _


6. Now from docker container A just do the following:
$ssh root@172.17.0.2
[Enter the password]: _

That's all!

SSH without providing password using ssh keygen

If you want to ssh login without providing password, then following ssh key need to be generated

1. From docker container A, generate rsa key as:
$ssh-keygen -t rsa
the key will be generated by default at ~/.ssh/ folder

2. Upload or transfer the public key id_rsa.pub to container A at the location as given below:
$cat ~/.ssh/id_rsa.pub | ssh root@172.17.0.2 'cat >> ~/.ssh/authorized_keys'

3. It will as one time password to copy, after then
$ ssh root@172.17.0.2
and no password will be asked.

That's all!

Now we can even run the remote script using ssh as:
ssh root@172.17.0.2 << EOF
echo "this is from container B"
pwd
EOF

JMeter JMS testing with IBM MQ server

Testing JMS with IBM MQ and JMeter

In order to send and receive message from IBM MQ server, you can use JMeter to test the message.  Follow the steps below to set up the JMeter to test the JMS message

1. Install latest Apache JMeter from https://jmeter.apache.org/download_jmeter.cgi

2. Copy following required IBM and other dependencies jars inside JMETER_HOME/lib/ext directory

- jmeter-jms-skip-jndi-0.0.1.jar
- com.ibm.mq.jar
- com.ibm.mqjms.jar
- connector.jar
- dhbcore.jar
- spring-core.jar
- spring-beans.jar
- javax.transaction-api.jar
- spring-jms.jar

3. Start JMeter -> create new Thread Group -> create JMS Point-to-Point from Add, Sampler

4. Add the following configuration information in the JMS Point-to-Point panel


Under JMS Properties add:
QueueConnectionFactory = CONNECTION_FACTORY
JNDI name Request queue = QUEUE_<your.request.queue.name>
JNDI name Receive queue = QUEUE_<your.response.queue.name>
Number of samples to aggregate = <1 or other value>
Communication style = <request_only/request_response>
Check both message correlation check box if needed

Initial Context Factory = <com.elega9t.jmeter.jms.InitialContextFactory>
 
Inside JNDI Properties add
hostName = <hostName>
port = <port>
channel = <channel>
queueManager = <queueManager>
transportType = 1

Inside Content text box = <your content to send to JMS>
 

JMS Point-to-Point configuration

View Result Tree and output response

Simple Spark Streaming with Kafka in Docker

I have tried to use simple way of Data streaming in Spark from Kafka using Docker environment. For this test you would need Docker installed and has access to Internet.

1. First start the docker (I am using boot2docker in Windows environment) as:
$ docker-machine start default
$ docker-machine ssh default or ssh docker@localhost -p 23

2. Inside docker environment load the Spark docker images (for this example I have used sequenceiq/spark)
$ docker pull sequenceiq/spark

3. his will pull the latest docker images to your docker environment. Once pull is complete run the following command in docker
$ docker run -it -p 4040:4040 -p 2181:2181 -p 9092:9092 -v /<some_path_to_share>:/data sequenceiq/spark:1.6.0 /bin/bash

following this command the spark image console is appear.

4. Now download the Apache Kafka from (https://kafka.apache.org/downloads) page. Choose suitable kafka version. For this example purpose, I downloaded kafka_2.10-0.10.1.0.tgz for Scala version 2.10

5. Unzip the file as:
$ tar -xvf kafka_2.10-0.10.1.0.tgz
$ mv kafka_2.10-0.10.1.0 /usr/local/kafka
$ cd /usr/local/kafka/bin

5. Now after download and unpack and move to suitable folder, start zookeeper and kafka from bin folder as below:
$ ./zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties
$ ./kafka-server-start.sh /usr/local/kafka/config/server.properties

6. After the zookeeper and kafka sucessfully started, create the topic for streaming as:
$ ./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic spark-topic
- this will create topic

7. Next thing is to create the sample Spark application as:
import kafka.serializer.StringDecoder

import org.apache.spark.SparkConf
import org.apache.spark.streaming.kafka.KafkaUtils
import org.apache.spark.streaming.{Seconds, StreamingContext}

object KafkaReceiver {

  def main(args: Array[String]): Unit = {

    val conf = new SparkConf().setMaster("local[*]").setAppName("Kafka Receiver")

    val kafkaParams = Map("metadata.broker.list" -> "localhost:9092")

    val topics = List("spark-topic").toSet

    val ssc = new StreamingContext(conf, Seconds(5))

    val lines = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder](ssc, kafkaParams, topics).map(_._2)

    println("printing Streaming data.......")

    lines.print()

    ssc.start()
    ssc.awaitTermination()
  }
}

8. Package this with required libraries as in build.sbt below

name := "SparkExample"

version := "1.0"

val sparkVersion = "1.6.0"

scalaVersion := "2.10.4"

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % sparkVersion,
  "org.apache.spark" %% "spark-sql" % sparkVersion,
  "org.apache.spark" %% "spark-streaming" % sparkVersion,
  "org.apache.spark" %% "spark-streaming-kafka" % sparkVersion
)

9. Run the jar after packaging from spark-submit as:
$ spark-submit --master local[*] --class KafkaReceiver --packages org.apache.spark:spark-streaming-kafka_2.10:1.6.0 /sparkexample_2.10-1.0.jar

once it start running check the console

10. Finally run the kafka producer to produce the data for streaming as:
$ ./kafka-console-producer.sh --broker-list localhost:9092 --topic spark-topic
Hello Spark Streaming

In Spark console you will see something like:

17/02/04 21:55:50 INFO executor.Executor: Running task 0.0 in stage 11.0 (TID 11)
17/02/04 21:55:50 INFO kafka.KafkaRDD: Beginning offset 1000011 is the same as ending offset skipping spark-topic 0
17/02/04 21:55:50 INFO executor.Executor: Finished task 0.0 in stage 11.0 (TID 11). 915 bytes result sent to driver
17/02/04 21:55:50 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 11.0 (TID 11) in 13 ms on localhost (1/1)
17/02/04 21:55:50 INFO scheduler.DAGScheduler: ResultStage 11 (print at KafkaReceiver.scala:27) finished in 0.012 s
17/02/04 21:55:50 INFO scheduler.DAGScheduler: Job 11 finished: print at KafkaReceiver.scala:27, took 0.063248 s
-------------------------------------------
Time: 1486263350000 ms
-------------------------------------------
Hello Spark Streaming

17/02/04 21:55:50 INFO scheduler.JobScheduler: Finished job streaming job 1486263350000 ms.0 from job set of time 1486263350000 ms
17/02/04 21:55:50 INFO scheduler.JobScheduler: Total delay: 0.115 s for time 1486263350000 ms (execution: 0.079 s)
17/02/04 21:55:50 INFO rdd.MapPartitionsRDD: Removing RDD 21 from persistence list
17/02/04 21:55:50 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 11.0, whose tasks have all completed, from pool
17/02/04 21:55:50 INFO storage.BlockManager: Removing RDD 21
17/02/04 21:55:50 INFO kafka.KafkaRDD: Removing RDD 20 from persistence list
17/02/04 21:55:50 INFO scheduler.ReceivedBlockTracker: Deleting batches ArrayBuffer()
17/02/04 21:55:50 INFO scheduler.InputInfoTracker: remove old batch metadata: 1486263340000 ms
17/02/04 21:55:50 INFO storage.BlockManager: Removing RDD 20

Simple Secure websocket backend server connection with Apache SSL Proxy

BROWSE <-----SSL connection-----> APACHE PROXY SERVER (SSL Enabled) <----Regular plain text----> TOMCAT WEB SOCKET SERVER

1. Enable following modules:
LoadModule ssl_module modules/mod_ssl
LoadModule socache_shmcb_module modules/mod_socache_shmcb.so

2. Create self-signed digital certificate for testing as follow using openssl
> openssl genrsa -out server.key 2048 (generate private key)
> openssl req -new -key server.key -out server.csr (generate CSR)
> openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt (generate self-signed certificate from CSR)

and copy the crt and key file at: C:/Apache24/conf/ folder

3. Enable the httpd-ssl.conf in the httpd.conf file as:
Include conf/extra/httpd-ssl.conf

4. Open httpd-ssl.conf file and update the following (comment Listen 443 line if starting server gives error about the port already used by 443)
SSLCertificateFile "${SRVROOT}/conf/server.crt" and
SSLCertificateKeyFile "${SRVROOT}/conf/server.key"

5. Start the server using ./httpd.exe and see if there is any issue, if no issue then access the page using https://localhost, this will launch the secure page

6. Add the Proxy in the httpd.conf file for the Tomcat as:
ProxyRequests Off
ProxyPreserveHost On
ProxyPass /test http://tomcat-host-ip:8090/test
ProxyPassReverse /test http://tomcat-host-ip:8090/test

Here Tomcat is running at 8090 port and Proxy is set up as http, i.e. from apache to tomcat, the data is passes as plain text.

7. To automatically change the http to https when user access http, add following line in the httpd.conf file just above the ProxyPass set up as:
RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}

and all set, run the Apache and Tomcat and access the Tomcat page from Apache as https://localhost/text. This will be Proxied to Tomcat as desired.

=========================================================================

FOR TOMCAT AJP CONNECTOR

1. Enable tomcat AJP1.3 in server.xml in Tomcat's configuration directory
2. Load the module
mod_proxy and mod_proxy_ajp in httpd.conf file in Apache 2.4
3. Set the Proxy setup as:
ProxyPass /test ajp://tomcat-host-ip:8090/test
ProxyPassReverse /test ajp://tomcat-host-ip:8009/test

Simple unsecured Apache Proxy for Tomcat Web Socket Application

BROWSE <===> APACHE PROXY SERVER <===> TOMCAT WEB SOCKET SERVER

This is tested for the Apache Haus v 2.4.10 with Tomcat v 7 in the Windows development environment.

1. Install Apache Haus 2.4.10 in Windows Environment.
2. Install Tomcat 7 in the Windows Environment either in same machine or different machine.
3. To create websocket proxy for the Apache load following module by removing LoadModule comments in httpd.conf file:

LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_wstunnel_module modules/mod_proxy_wstunnel.so

For further, please refer Apache documentation at: http://httpd.apache.org/docs/2.4/mod/mod_proxy_wstunnel.html and add following ProxyPass setup in httpd.conf configuration file

ProxyRequests Off
# If tomcat is in same machine (for websocket)
ProxyPass /websocket/echo ws://localhost:8090/websocket/echo          -> look for the trailing "/"
ProxyPassReverse /websocket/echo ws://localhost:8090/websocket/echo             -> look for the trailing "/"

# For other http request
ProxyPass /test http://localhost:8090/test
ProxyPassReverse /test http://localhost:8090/test

# If Tomcat is in different/remote machine
ProxyPass /websocket/echo ws://remote-host:8090/websocket/echo
ProxyPassReverse /websocket/echo ws://remote-host:8090/websocket/echo

# For other http request
ProxyPass /test http://remote-host:8090/test
ProxyPassReverse /test http://remote-host:8090/test

This configuration assume that Apache use port 80 and Tomcat use port 8090 and servlet context deployed in tomcat /test. Now the http or ws call to Apache will automatically redirect to corresponding Tomcat Server once it find the Proxy setting. Also take special look at the trailing "/". If you add trailing "/" for /websocket/echo/, then you need to add trailing "/" for ws://localhost:8090/websocket/echo/ too or vice-versa.

Also the ProxyPass has to be set up in specific order (take a look at the Apache documentation for this).

Here is how someone in Stack Overflow did (very useful): http://stackoverflow.com/questions/17649241/reverse-proxy-with-websocket-mod-proxy-wstunnel

4. Create simple websocket application in Tomcat as:
@ServerEndpoint("/echo")
public class EchoWebSocket {
    private Logger logger = Logger.getLogger(this.getClass().getName());
    @OnOpen
    public void onOpen(Session session) {
        logger.info("Connected .... " + session.getId());
    }
    @OnMessage
    public String onMessage(String message, Session session) {
        switch (message) {
        case "quit":
            try {
                session.close(new CloseReason(CloseCodes.NORMAL_CLOSURE,
                        "Connection is closed."));
            } catch (IOException e) {
                throw new RuntimeException(e);
            }
            break;
        }
        return message;
    }
    @OnClose
    public void onClose(Session session, CloseReason closeReason) {
        logger.info(String.format("Session %s closed because of %s",
                session.getId(), closeReason));
    }
}

*Remember when deploying the Tomcat application, don't put websocket-api.jar library in the application classpath. Since this library is already included in the Tomcat library, adding this in application class-path may not work (At least for me it didn't work).

5. Add the JavaScripts HTML5 WebSocket API script as:

function wsocket() {
    var ws = null;
    var wsProtocol = (window.location.protocol === "https:" ? "wss" : "ws");
    var target =  wsProtocol + "://" + window.location.host + "/dwrdemo/echo";
    if ("WebSocket" in window) {
        ws = new WebSocket(target);
    } else if ("MozWebSocket" in window) {
        ws = new MozWebSocket(target);
    } else {
        alert('WebSocket is not supported by this browser.');
    }

    ws.onopen = function() {
        console.log("WebSocket has been opened!");
    };

    ws.onmessage = function(message) {
       console.log("WebSocket message is: " + message.data);
    };

    ws.onerror = function() {
        console.log("WebSocket connection has error!");
    }
   
    ws.onclose = function() {
        console.log("WebSocket is closed.");
    }
}


6. Add the Tomcat deployable war file in the Tomcat Server and test to make sure it is working. Once working access the page using the Apache web server.
7. The Request and response header for the websocket look something like as shown below if connection is established successfully

URL: http://localhost/websocket/echo

Request Headers:
    Accept    text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
    Accept-Encoding    gzip, deflate
    Accept-Language    en-US,en;q=0.5
    Cache-Control    no-cache
    Connection    keep-alive, Upgrade
    Cookie    JSESSIONID=3CD55817617CA29EF8F133721A1E1388
    Host    localhost
    Origin    http://localhost
    Pragma    no-cache
    Sec-WebSocket-Key    2cWlxhpbtE0dO1ijeaocaA==
    Sec-WebSocket-Version    13
    Upgrade    websocket
    User-Agent    Mozilla/5.0 (Windows NT 6.1; WOW64; rv:35.0) Gecko/20100101 Firefox/35.0

Response Headers:
    Cache-Control    private
    Connection    upgrade
    Date    Fri, 06 Feb 2015 15:15:01 GMT
    Expires    Wed, 31 Dec 1969 19:00:00 EST
    Sec-WebSocket-Accept    jMl5aXuuLM8xJ35XDGAY8uncIWk=
    Server    Apache-Coyote/1.1
    Upgrade    websocket

Take a look at the Connection Upgrate, the regular http request is updated to websocket for the websocket connection. This is how the websocket work by upgrading the current protocol in the initial handshake.

Testing using JConsole as:

var ws = new WebSocket("ws://localhost/websocket/echo");
ws.readyState

Mutual-Authentication with Client Self Signed Digital Certificate with Tomcat SSL Configuration

MUTUAL-AUTHENTICATION WITH CLIENT DIGITAL CERTIFICATE AND TOMCAT SSL CONFIGURATION (FOR DEV and TEST):

In Order to authenticate server and client certificate using mutual authentication in Tomcat SSL configuration, we need to create the pair of keys in server, client. The client certificate is then added to the server trustStore.
1. First generate the server-cert using keytool utility as:
> keytool -genkeypair -alias servercert -keyalg RSA -dname "CN=Your Server DNS,OU=ORG,O=COM,L=NYC,S=NY,C=US" -keypass <password> -keystore server.jks -storepass <password>

usually the keypass, and storepass is kept same. The above keytool command create the server.jks keystore with servercert as alias. The command creates server Private Key and other information and protect the Private key with the password supplied.
To view the content, type the command as:
> keytool -list -v -keystore server.jsk -storepass <password>

2. Next create client keypair as:
> keytool -genkeypair -alias clientStore -keystore clientStore.p12 -storetype pkcs12 -keyalg RSA -dname "CN=Your Client,OU=ORG,O=COM,L=NYC,S=NY,C=US" -keypass <password> -storepass <password>

This will also create the clientStore like the one in 1 with Private Key and certificate.

3. Now export the certificate that is created in keystore as in the step 2 using the command as:
> keytool -exportcert -alias clientStore -file clientStore.cer -keystore clientStore.p12 -storetype pkcs12 -storepass <password>

This command will export the client certificate clientStore.cer file and import this file to trustStore as given below:

4: importing the certificate to trustStore as:
> keytool -importcert -keystore server.jks -alias clientStore -file clientStore.cer -v -trustcacerts -noprompt -storepass <password>

5. Once you have imported the client certificate to the trustStore, you can view the content using the following command as:
> keytool -list -v -keystore server.jks -storepass <password>

6. Now the server-cert keystore can be dropped in the Tomcat {CATALINA_HOME}/config/ directory and enable the SSL authentication using the following Connector configuration:
<Connector port="8443" protocol="org.apache.coyote.http11.Http11Protocol"
    maxThreads="150" SSLEnabled="true" scheme="https" secure="true"
    clientAuth="true" sslProtocol="TLS" keystoreFile="{CATALINA_HOME}/config/server.jks" keystorePass="password"
    truststoreFile="{CATALINA_HOME}/config/server.jks" truststorePass="password" truststoreType="JKS"/>

For the server as well as client certification authentication use clientAuth="true" and required to add trustStoreFile, trustStorePass and trustStoreType. The trustStoreType is by default JKS.

7. Now download the clientStore.p12 file in the client end and install in the browser as client certificate. The certificate will as password, provider the password and it will be all set for the mutual SSL communication.
Now connect the secure page using https and if certificate exception occurs (which will be in Firefox, since this is self-signed certificate), add the exception and you should be able to go to the secure page.
This way the SSL connection established using Mutual Certificate Authentication. For production environment, the certificate need to be verified by CA (Certificate Authority).

8. To create more client certificates, repeat step 2, 3, and import to step 4. This way multiple client certificate can be created for multiple clients for client authentication.


For more information, refer following resources:
http://stackoverflow.com/questions/1180397/tomcat-server-client-self-signed-ssl-certificate
http://docs.oracle.com/middleware/1213/edq/DQSEC/ssl_tomcat.htm#DQSEC164
https://twoguysarguing.wordpress.com/2009/11/03/mutual-authentication-with-client-cert-tomcat-6-and-httpclient/
http://java-notes.com/index.php/two-way-ssl-on-tomcat
http://docs.geoserver.org/latest/en/user/security/tutorials/cert/index.html