Installing Maven using Yum on EC2 instance (Amazon Linux)

Standard
  1. sudo wget http://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo -O /etc/yum.repos.d/epel-apache-maven.repo
  2. sudo sed -i s/\$releasever/6/g /etc/yum.repos.d/epel-apache-maven.repo
  3. sudo yum install -y apache-maven
  4. mvn –version

And you all set to run the “mvn” command in ec2 instance.

[ec2-user@ip-123-45-67-89 sudhir]$ mvn –version

Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T16:41:47+00:00)

Maven home: /usr/share/apache-maven

Java version: 1.7.0_151, vendor: Oracle Corporation

Java home: /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.151.x86_64/jre

Default locale: en_US, platform encoding: UTF-8

OS name: “linux”, version: “4.9.43-17.39.amzn1.x86_64”, arch: “amd64”, family: “unix”

Advertisements

Install/upgrage Java 8 using Yum on EC2 instance (Amazon Linux AMI)

Standard

Step-1: Install Java

Runtime 1.8

sudo yum install java-1.8.0

if you need a java compiler and other developer tools:

sudo yum install java-1.8.0-openjdk-devel

Step-2: If you have multiple versions and one of those default , use the alternatives command as follows and enter the selection number as guided in the terminal.

sudo /usr/sbin/alternatives –config java
sudo /usr/sbin/alternatives –config javac

Step-3: (Optional)

If you prefer you can remove Java 1.7

sudo yum remove java-1.8.0-openjdk

Note : Remove it after you installed Java 1.8 otherwise the aws-apitools will also be removed as they depend on Java to be installed

Step By Step : Installing Kafka in Mac

Standard
  1. Open “Terminal” app from Applications or Command + Space and then type “Terminal” press Enter/Return Key
  2. Install Homebrew (Copy / Paste the following command in the Terminal window and press enter)
    1. ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" < /dev/null 2> /dev/null
  3. Now install Kafka(Copy / Paste the following command in the Terminal window and press enter)
    • brew install kafka
    • Sample Output of above command :
    • ==> Installing dependencies for kafka: zookeeper
      
      ==> Installing kafka dependency: zookeeper
      
      ==> Downloading https://homebrew.bintray.com/bottles/zookeeper-3.4.10.sierra.bottle.tar.gz
      
      ######################################################################## 100.0%
      
      ==> Pouring zookeeper-3.4.10.sierra.bottle.tar.gz
      
      ==> Caveats
      
      To have launchd start zookeeper now and restart at login:
      
       brew services start zookeeper
      
      Or, if you don't want/need a background service you can just run:
      
       zkServer start
      
      ==> Summary
      
      🍺 /usr/local/Cellar/zookeeper/3.4.10: 241 files, 31.4MB
      
      ==> Installing kafka
      
      ==> Downloading https://homebrew.bintray.com/bottles/kafka-0.11.0.0.sierra.bottle.tar.gz
      
      ######################################################################## 100.0%
  4. Now list all services
    • brew services list
    • Sample output of above command
    • spradhan-macos:/ sudhir.pradhan$ brew services list
      Name Status User Plist
      kafka stopped
      mysql stopped
      postgresql started sudhir.pradhan /Users/sudhir.pradhan/Library/LaunchAgents/homebrew.mxcl.postgresql.plist
      presto stopped
      redis started sudhir.pradhan /Users/sudhir.pradhan/Library/LaunchAgents/homebrew.mxcl.redis.plist
      zookeeper stopped
    • From above output it seems the kafka installed but not running
    • So before starting Kafka, make sure we the ZooKeeper is running
  5. Lets start ZooKeeper
    • brew services start zookeeper
    • Sample output
      1. spradhan-macos:/ sudhir.pradhan$ brew services start zookeeper
        ==> Successfully started `zookeeper` (label: homebrew.mxcl.zookeeper)
  6. Now start Kafka (From above output we made sure the Zookeeper is running)
    • brew services start kafka
    • Sample output
      1. spradhan-macos:/ sudhir.pradhan$ brew services start kafka
        ==> Successfully started `kafka` (label: homebrew.mxcl.kafka)
  7. Lets verify if our required services are running
    • brew services list
    • Sample output
    • spradhan-macos:/ sudhir.pradhan$ brew services list
      Name Status User Plist
      kafka started sudhir.pradhan /Users/sudhir.pradhan/Library/LaunchAgents/homebrew.mxcl.kafka.plist
      mysql stopped
      postgresql started sudhir.pradhan /Users/sudhir.pradhan/Library/LaunchAgents/homebrew.mxcl.postgresql.plist
      presto stopped
      redis started sudhir.pradhan /Users/sudhir.pradhan/Library/LaunchAgents/homebrew.mxcl.redis.plist
      zookeeper started sudhir.pradhan /Users/sudhir.pradhan/Library/LaunchAgents/homebrew.mxcl.zookeeper.plist
  8. Lets create a topic named “test”
    • kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 4 --topic test
    • Sample output
    • spradhan-macos:/ sudhir.pradhan$ kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 4 --topic test
      Created topic "test".
  9. Desctibe above created topic
    • kafka-topics.sh --describe --zookeeper localhost:2181 --topic
    • Sample output
    • spradhan-macos:/ sudhir.pradhan$ kafka-topics –describe –zookeeper localhost:2181 –topic test

    • Topic:test PartitionCount:4 ReplicationFactor:1 Configs:

    • Topic: test Partition: 0 Leader: 0 Replicas: 0 Isr: 0

    • Topic: test Partition: 1 Leader: 0 Replicas: 0 Isr: 0

    • Topic: test Partition: 2 Leader: 0 Replicas: 0 Isr: 0

    • Topic: test Partition: 3 Leader: 0 Replicas: 0 Isr: 0

  10. Start Producer and Consumer in 2 different terminal window
    • in terminal 1:
      1. kafka-console-producer --broker-list localhost:9092 --topic test
    • in terminal 2:
      1. kafka-console-consumer --broker-list localhost:9092 --topic test
  11. Now Publish message to the above topic in Terminal 1
    1. spradhan-macos:/ sudhir.pradhan$ kafka-console-producer –broker-list localhost:9092 –topic test
      >Hi There
      >

  12. Now check in Terminal 1 if the consumer received the message
    1. spradhan-macos:/ sudhir.pradhan$ kafka-console-consumer –bootstrap-server localhost:9092 –from-beginning –topic test
      Hi There

    2. Yehhhh …  there you go

 

Hope you enjoyed the article. Please like or comment.

Step By Step: Getting Kafka installed in Mac OS X Sierra

Standard

In this activity we are going to use the beautiful packaging manager tool Homebrew throughout the installation process. This tool make life easier to install and manage the latest version of the software and keep updated.

Step 1 : Install Homebrew (as an administrator)

$ /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

The above command will install the following packages :

  • homebrew
  • command line tool for Xcode-8.2
  • Along with other supporting libraries

Following are the sample logs you might see at the time of homebrew installation

Sudhirs-MacBook-Pro:~ sudhir$ /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

==> This script will install:
/usr/local/bin/brew
/usr/local/share/doc/homebrew
/usr/local/share/man/man1/brew.1
/usr/local/share/zsh/site-functions/_brew
/usr/local/etc/bash_completion.d/brew
/usr/local/Homebrew
==> The following new directories will be created:
/usr/local/Cellar
/usr/local/Homebrew
/usr/local/Frameworks
/usr/local/bin
/usr/local/etc
/usr/local/include
/usr/local/lib
/usr/local/opt
/usr/local/sbin
/usr/local/share
/usr/local/share/zsh
/usr/local/share/zsh/site-functions
/usr/local/var


==> Cleaning up /Library/Caches/Homebrew...
==> Migrating /Library/Caches/Homebrew to /Users/sudhir/Library/Caches/Homebrew...
==> Deleting /Library/Caches/Homebrew...
Already up-to-date.
==> Installation successful!

==> Homebrew has enabled anonymous aggregate user behaviour analytics.
Read the analytics documentation (and how to opt-out) here:
  https://git.io/brew-analytics

==> Next steps:
- Run `brew help` to get started
- Further documentation: 
    https://git.io/brew-docs

Step 2 : Now homebrew is ready to use. Install wget tool

$ brew install wget

The above command with install wget package along with OpenSSL library.

Sample log for above command …

Sudhirs-MacBook-Pro:~ sudhir$ brew install wget

==> Installing dependencies for wget: openssl
==> Installing wget dependency: openssl
==> Downloading https://homebrew.bintray.com/bottles/openssl-1.0.2j.sierra.bottle.tar.gz
######################################################################## 100.0%
==> Pouring openssl-1.0.2j.sierra.bottle.tar.gz
==> Using the sandbox
==> Caveats
A CA file has been bootstrapped using certificates from the SystemRoots
keychain. To add additional certificates (e.g. the certificates added in
the System keychain), place .pem files in
  /usr/local/etc/openssl/certs

and run
  /usr/local/opt/openssl/bin/c_rehash

This formula is keg-only, which means it was not symlinked into /usr/local.

Apple has deprecated use of OpenSSL in favor of its own TLS and crypto libraries

Generally there are no consequences of this for you. If you build your
own software and it requires this formula, you will need to add to your
build variables:

    LDFLAGS:  -L/usr/local/opt/openssl/lib
    CPPFLAGS: -I/usr/local/opt/openssl/include

==> Summary
🍺  /usr/local/Cellar/openssl/1.0.2j: 1,695 files, 12M
==> Installing wget 
==> Downloading https://homebrew.bintray.com/bottles/wget-1.18.sierra.bottle.tar.gz
######################################################################## 100.0%
==> Pouring wget-1.18.sierra.bottle.tar.gz
🍺  /usr/local/Cellar/wget/1.18: 9 files, 1.6M

Step 3 : Now its time to install Kafka

  • You can search available Kafka builds by executing the following command
$ brew search kafka

Above command will give you the result something like

Sudhirs-MacBook-Pro:~ sudhir$ brew search kafka

kafka                                           kafkacat                                        librdkafka
homebrew/php/php53-kafka            homebrew/php/php54-rdkafka          homebrew/php/php56-kafka            homebrew/php/php70-rdkafka
homebrew/php/php53-rdkafka          homebrew/php/php55-kafka            homebrew/php/php56-rdkafka          homebrew/php/php71-rdkafka
homebrew/php/php54-kafka            homebrew/php/php55-rdkafka          homebrew/php/php70-kafka            Caskroom/cask/kafka-tool
  • Now execute the following command to install Kafka
$ brew install kafka

Above command will give you the result something like

Sudhirs-MacBook-Pro:kafka-0.8.2 sudhir$ brew install kafka

==> Installing dependencies for kafka: zookeeper
==> Installing kafka dependency: zookeeper
==> Downloading https://homebrew.bintray.com/bottles/zookeeper-3.4.9.sierra.bottle.tar.gz
######################################################################## 100.0%
==> Pouring zookeeper-3.4.9.sierra.bottle.tar.gz
==> Caveats
To have launchd start zookeeper now and restart at login:
  brew services start zookeeper
Or, if you do not want/need a background service you can just run:
  zkServer start
==> Summary
🍺  /usr/local/Cellar/zookeeper/3.4.9: 238 files, 18.2M
==> Installing kafka 
==> Downloading https://homebrew.bintray.com/bottles/kafka.sierra.bottle.tar.gz
######################################################################## 100.0%
==> Pouring kafka.sierra.bottle.tar.gz
==> Caveats
To have launchd start kafka now and restart at login:
  brew services start kafka
Or, if you do not want/need a background service you can just run:
  zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties & kafka-server-start /usr/local/etc/kafka/server.properties
==> Summary
🍺  /usr/local/Cellar/kafka/ : 128 files, 35.2M

The above Kafka installation will also install all dependencies, like zookeeper which is required to run Kafka server.

You can see the configurations are installed for Kafka & zookeeper as follows

Sudhirs-MacBook-Pro:~ sudhir$ ls -ltr /usr/local/etc
total 16
drwxr-xr-x   3 sudhir  admin   102 Jan 23 21:42 bash_completion.d
drwxr-xr-x   7 sudhir  admin   238 Jan 23 21:42 openssl
-rw-r--r--   1 sudhir  admin  4945 Jan 23 21:42 wgetrc
drwxr-xr-x   6 sudhir  admin   204 Jan 23 21:47 zookeeper
drwxr-xr- 15 sudhir  admin   510 Jan 23 21:47 kafka
Sudhirs-MacBook-Pro:~ sudhir$ ls -ltr /usr/local/etc/zookeeper/
total 32
-rw-r--r--  1 sudhir  admin  941 Jan 23 21:47 zoo_sample.cfg
-rw-r--r--  1 sudhir  admin  941 Jan 23 21:47 zoo.cfg
-rw-r--r--  1 sudhir  admin  339 Jan 23 21:47 log4j.properties
-rw-r--r--  1 sudhir  admin   44 Jan 23 21:47 defaults
Sudhirs-MacBook-Pro:~ sudhir$ view /usr/local/etc/zookeeper/zoo.cfg
Sudhirs-MacBook-Pro:~ sudhir$ ls -ltr /usr/local/etc/kafka/
total 120
-rw-r--r--  1 sudhir  admin  1037 Jan 23 21:47 zookeeper.properties
-rw-r--r--  1 sudhir  admin  1032 Jan 23 21:47 tools-log4j.properties
-rw-r--r--  1 sudhir  admin  5350 Jan 23 21:47 server.properties
-rw-r--r--  1 sudhir  admin  1900 Jan 23 21:47 producer.properties
-rw-r--r--  1 sudhir  admin  4369 Jan 23 21:47 log4j.properties
-rw-r--r--  1 sudhir  admin  1199 Jan 23 21:47 consumer.properties
-rw-r--r--  1 sudhir  admin  2061 Jan 23 21:47 connect-standalone.properties
-rw-r--r--  1 sudhir  admin  1074 Jan 23 21:47 connect-log4j.properties
-rw-r--r--  1 sudhir  admin   881 Jan 23 21:47 connect-file-source.properties
-rw-r--r--  1 sudhir  admin   883 Jan 23 21:47 connect-file-sink.properties
-rw-r--r--  1 sudhir  admin  2760 Jan 23 21:47 connect-distributed.properties
-rw-r--r--  1 sudhir  admin   909 Jan 23 21:47 connect-console-source.properties
-rw-r--r--  1 sudhir  admin   906 Jan 23 21:47 connect-console-sink.properties

Step 3 : Before starting Kafka server, first start ZooKeeper which is responsible for coordinating & selecting the leader.

  • Check if the Zookeeper really installed
Sudhirs-MacBook-Pro:~ sudhir$ which zkserver
/usr/local/bin/zkserver

Sudhirs-MacBook-Pro:~ sudhir$ zkserver
ZooKeeper JMX enabled by default
Using config: /usr/local/etc/zookeeper/zoo.cfg
Usage: ./zkServer.sh {start|start-foreground|stop|restart|status|upgrade|print-cmd}

  • Now start the zookeeper server
Sudhirs-MacBook-Pro:~ sudhir$ zkserver start

ZooKeeper JMX enabled by default
Using config: /usr/local/etc/zookeeper/zoo.cfg
Starting zookeeper ... STARTED
  • Test if the zookeeper server is really started
Sudhirs-MacBook-Pro:~ sudhir$ telnet localhost 2181
Trying ::1...
Connected to localhost.
Escape character is '^]'.

^CConnection closed by foreign host.

The above log shows that we are able to telnet to the zookeeper port , hence the zookeeper server is up and running

Step 4 : Lets create the symlinks for the configuration directory created after Kafka installation (Its not mandatory but makes life easier to use with out bothering the original directory)

Sudhirs-MacBook-Pro:~ sudhir$ ln -nsf /usr/local/etc/ ./symln.etc/

Sudhirs-MacBook-Pro:~ sudhir$ ls -ltr ~/symln.etc/
total 24
drwxr-xr-x   3 sudhir  admin   102 Jan 23 21:42 bash_completion.d
drwxr-xr-x   7 sudhir  admin   238 Jan 23 21:42 openssl
-rw-r--r--   1 sudhir  admin  4945 Jan 23 21:42 wgetrc
drwxr-xr-x   6 sudhir  admin   204 Jan 23 21:47 zookeeper
drwxr-xr-x  15 sudhir  admin   510 Jan 23 21:47 kafka
lrwxr-xr-x   1 sudhir  admin    15 Jan 23 21:59 etc -> /usr/local/etc/

Step 5 : Now lets start Kafka server

Sudhirs-MacBook-Pro:~ sudhir$ cd ~/symln.kafka/

Sudhirs-MacBook-Pro:~ sudhir$ pwd
/Users/sudhir/symln.kafka/

Sudhirs-MacBook-Pro:~ sudhir$ cd ~/symln.kafka/

Sudhirs-MacBook-Pro: sudhir$ ./bin/kafka-server-start ~/symln.etc/kafka/server.properties

After successful execution of the above command, you might see the following result.

Sudhirs-MacBook-Pro: sudhir$ ./bin/kafka-server-start ~/symln.etc/kafka/server.properties
[2017-01-24 22:50:08,731] INFO KafkaConfig values: 
	advertised.host.name = null
	advertised.listeners = null
	advertised.port = null
	.........
.............
[2017-01-24 22:50:34,751] INFO Completed load of log myTopic-0 with 1 log segments and log end offset 0 in 4 ms (kafka.log.Log)
[2017-01-24 22:50:34,753] INFO Created log for partition [myTopic,0] in /usr/local/var/lib/kafka-logs with properties {compression.type -> producer, message.format.version -> 0.10.1-IV2, file.delete.delay.ms -> 60000, max.message.bytes -> 1000012, min.compaction.lag.ms -> 0, message.timestamp.type -> CreateTime, min.insync.replicas -> 1, segment.jitter.ms -> 0, preallocate -> false, min.cleanable.dirty.ratio -> 0.5, index.interval.bytes -> 4096, unclean.leader.election.enable -> true, retention.bytes -> -1, delete.retention.ms -> 86400000, cleanup.policy -> [delete], flush.ms -> 9223372036854775807, segment.ms -> 604800000, segment.bytes -> 1073741824, retention.ms -> 604800000, message.timestamp.difference.max.ms -> 9223372036854775807, segment.index.bytes -> 10485760, flush.messages -> 9223372036854775807}. (kafka.log.LogManager)
[2017-01-24 22:50:34,754] INFO Partition [myTopic,0] on broker 0: No checkpointed highwatermark is found for partition [myTopic,0] (kafka.cluster.Partition)
[

Step 6 : Now we are good to start out Consumer and Producer to pull and push the data respectively from Kafka topic. (Here we will use command line consumer & producer for test purpose)

  1. Start Consumer
Sudhirs-MacBook-Pro:
 sudhir$ pwd
/Users/sudhir/symln.kafka/

Sudhirs-MacBook-Pro:~ sudhir$ cd ~/symln.kafka/


Sudhirs-MacBook-Pro: sudhir$ ls -ltr bin/
total 216
-r-xr-xr-x  1 sudhir  admin  141 Jan 23 21:47 zookeeper-shell
-r-xr-xr-x  1 sudhir  admin  147 Jan 23 21:47 zookeeper-server-stop
-r-xr-xr-x  1 sudhir  admin  148 Jan 23 21:47 zookeeper-server-start
-r-xr-xr-x  1 sudhir  admin  154 Jan 23 21:47 zookeeper-security-migration
-r-xr-xr-x  1 sudhir  admin  151 Jan 23 21:47 kafka-verifiable-producer
-r-xr-xr-x  1 sudhir  admin  151 Jan 23 21:47 kafka-verifiable-consumer
-r-xr-xr-x  1 sudhir  admin  138 Jan 23 21:47 kafka-topics
-r-xr-xr-x  1 sudhir  admin  157 Jan 23 21:47 kafka-streams-application-reset
-r-xr-xr-x  1 sudhir  admin  153 Jan 23 21:47 kafka-simple-consumer-shell
-r-xr-xr-x  1 sudhir  admin  143 Jan 23 21:47 kafka-server-stop
-r-xr-xr-x  1 sudhir  admin  144 Jan 23 21:47 kafka-server-start
-r-xr-xr-x  1 sudhir  admin  141 Jan 23 21:47 kafka-run-class
-r-xr-xr-x  1 sudhir  admin  152 Jan 23 21:47 kafka-replica-verification
-r-xr-xr-x  1 sudhir  admin  151 Jan 23 21:47 kafka-replay-log-producer
-r-xr-xr-x  1 sudhir  admin  151 Jan 23 21:47 kafka-reassign-partitions
-r-xr-xr-x  1 sudhir  admin  150 Jan 23 21:47 kafka-producer-perf-test
-r-xr-xr-x  1 sudhir  admin  158 Jan 23 21:47 kafka-preferred-replica-election
-r-xr-xr-x  1 sudhir  admin  144 Jan 23 21:47 kafka-mirror-maker
-r-xr-xr-x  1 sudhir  admin  150 Jan 23 21:47 kafka-consumer-perf-test
-r-xr-xr-x  1 sudhir  admin  155 Jan 23 21:47 kafka-consumer-offset-checker
-r-xr-xr-x  1 sudhir  admin  147 Jan 23 21:47 kafka-consumer-groups
-r-xr-xr-x  1 sudhir  admin  148 Jan 23 21:47 kafka-console-producer
-r-xr-xr-x  1 sudhir  admin  148 Jan 23 21:47 kafka-console-consumer
-r-xr-xr-x  1 sudhir  admin  139 Jan 23 21:47 kafka-configs
-r-xr-xr-x  1 sudhir  admin  136 Jan 23 21:47 kafka-acls
-r-xr-xr-x  1 sudhir  admin  144 Jan 23 21:47 connect-standalone
-r-xr-xr-x  1 sudhir  admin  145 Jan 23 21:47 connect-distributed

Execute Kafka consumer in a new terminal (It will write the message in the same console as soon as produces publish the data to Kafka topic named “myTopic”)

Sudhirs-MacBook-Pro: sudhir$ ./bin/kafka-console-consumer --zookeeper localhost:2181 --topic myTopic --from-beginning

Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper].

Now Consumer started, but we don’t see any massage consuming because we didn’t start producer yet and not published any data

2. Now start the producer in a new terminal (will publish the data into Kafka topic “myTopic” to which the consumer already subscribed)

Sudhirs-MacBook-Pro:0.10.1.0 sudhir$ ./bin/kafka-console-producer --broker-list localhost:9092 --topic myTopic

Now type something like “Hello Kafka” and press enter

Sudhirs-MacBook-Pro:0.10.1.0 sudhir$ ./bin/kafka-console-producer --broker-list localhost:9092 --topic myTopic
Hello Kafka

Now observe in Consumer terminal , where you can see the consumer consumed the data published by the producer

Sudhirs-MacBook-Pro: sudhir$ ./bin/kafka-console-consumer --zookeeper localhost:2181 --topic myTopic --from-beginning
Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper].
Hello Kafka

Conclusion :

Its just the begining of Kafka. there are much more to get into.

Kafka provides the following:

  • Performs on persistent messaging with constant time O(1) even with terabytes of stored messages.
  • High throughput.
  • Partitioned massaging service in distributed environment.
  • Support for parallel data streaming & loading into Hadoop.

Note : The performance , latency & throughout might differ according to some of the crucial configuration parameters configured at Kafka setup

I will try to put some use cases regarding those parameters in a different post.

Thanks for visiting the article & hope it helps.

Problem installing ZooKeeper & Kafka in macOS Sierra

Standard

Until I could successfully installed Kafka in my macOS Sierra environment, the problem I was facing is bringing the zookeeper up & running. We can start Kafka server only if zookeeper server is up.

Downloaded the zookeeper trunk build from git repo and extracted in local

When started the zookeeper first time from the extracted build, the following was the message I got in the terminal

Now trying to test if the server is up, but failed to get the connection

Tried to start the zookeeper server again, however this time also I got same message as like first time.

Still didn’t get any zookeeper instance up & running 😦

The issue might seems a silly one, however finally somehow I able to install the zookeeper & Kafka successfully.

Here is the post I want to share step by step to make the Kafka installation ease in Mac OSX environment.

Step By Step : Build and Run Kafka in Eclipse IDE + [ Scala || Java ] + Gradle

Standard

As a coder, we are more comfortable with the editor tools (specially Eclipse IDE) for rapid development , build & continuous integration. When first time I was trying to develop some Kafka producer and consumer using Scala, I was wondering if I could setup the same through eclipse to make life easier, however after a lot of hit and trial finally able to setup and successfully executed my first program.

Likewise, some other friends out there might struggling on the same. Hence thought to share the simple steps to get the Kafka setup and run program from eclipse IDE done.

Important : Before proceed, please ensure you zookeeper & Kafka server already running. If not the Mac users can always refer Step By Step : Install Kafka in OSX / macOS Sierra for setting up and running ZK & Kafka server

Step 1: Instal Scala IDE (Eclipse for scala)

Download Scala IDE from the official site http://scala-ide.org

[Please ensure the IDE version you are downloading is meeting the system requirement]

Step 2: Install Gradle in the downloaded Scala IDE

Open Scala IDE -> Create your own workspace —> Help -> Eclipse Marketplace -> search for “Gradle IDE pack 3.8.x + 1.0.x” (note: my IDE version is Neon, hence this Gradle version is compatible)

Step 4: Importing Kafka libraries into eclipse workspace

  • Download the Kafka latest build from git https://github.com/sreev/kafka/tree/0.8.2
  • Download the zip into Desktop (to local system)
  • Extract the downloaded zip
  • Go to eclipse IDE -> File -> Import -> Gradle (STS) project -> Next -> Brouse the extracted Kafka directory from you local disk -> click ‘Build Model’ -> next -> Finish

The following are the step by step screenshots for importing Kafka project

Step 5: Now expand the project “example” -> run the “KafkaConsumerProducerDemo” class as java program

Now you should able to see the producer and consumer running in your eclipse console and printing the data in consumer console those are published by the producer.

Kafka properties for the above, where we have set all the required configuration params

Now you can also observe the topic name (you have provided in your program) from your terminal where the kafka server is already running.

Conclusion : And you all set for writing more complex kafka code and executing from eclipse IDE (can be Scala or Java as per the need)

Enjoy ….

REST API Maven Dependency : Jersey + Jetty

Standard

Hooooh …. finally !!! After devoting ample number of hours with lots of R&Ds, finally able to resolve the long running dependencies between Jersey and Jetty when developing a REST API.

I am posting the article because working and running the REST API with TOMCAT or similar server would be easy but might not with embedded JETTY Server (What I felt). I have experienced lot of dependency issues thorough out the setup. Hence I am posting to help and might save the setup time hopefully.

Why I felt the Jetty Server is so cool over Tomcat :

  1. Full-featured and standards-based.
  2. Embeddable and Asynchronous.
  3. Open source and commercially usable.
  4. Dual licensed under Apache and Eclipse.
  5. Flexible and extensible, Enterprise scalable.
  6. Strong Tools, Application, Devices and Cloud computing supported.
  7. Low maintenance cost.
  8. Small and Efficient.

The following is couple of (out of all) errors faced at the time of setup. A very simple but commonly occurred exception.

  • Error code : 52
ERR_EMPTY_RESPONSE – No Data Received
Error code : 52  
Sudhirs-MacBook-Pro-2:yanagishima-5.0 sudhir.pradhan$ curl -L -X GET http://localhost:8083/presto/hello

curl: (52) Empty reply from server
  • Error code : 500
java.lang.NoSuchMethodError: org.eclipse.jetty.server.Response.getResponse(Ljavax/servlet/http/HttpServletResponse;)Lorg/eclipse/jetty/server/Response;

The complete dependency configuration (Jersey + Jetty + Maven) : pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <!--
    @author : Sudhir R. Pradhan
    @date   : Jun 09 2017
    -->

    <groupId>ZQueryEngine</groupId>
    <artifactId>ZQE</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <version.jersey>2.26-b03</version.jersey>
        <version.aws>1.11.133</version.aws>
        <version.jetty>9.2.14.v20151106</version.jetty>
    </properties>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                </configuration>
            </plugin>
        </plugins>
    </build>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.glassfish.jersey</groupId>
                <artifactId>jersey-bom</artifactId>
                <version>${version.jersey}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>

        </dependencies>

    </dependencyManagement>

    <dependencies>
        <!-- https://mvnrepository.com/artifact/org.glassfish.jersey.core/jersey-server -->
        <dependency>
            <groupId>org.glassfish.jersey.core</groupId>
            <artifactId>jersey-server</artifactId>
            <version>${version.jersey}</version>
        </dependency>

        <dependency>
            <groupId>org.glassfish.jersey.containers</groupId>
            <artifactId>jersey-container-servlet-core</artifactId>
            <version>${version.jersey}</version>
        </dependency>

        <dependency>
            <groupId>org.glassfish.jersey.containers</groupId>
            <artifactId>jersey-container-servlet</artifactId>
            <version>${version.jersey}</version>
        </dependency>

        <!-- do not use jettison, prefer jackson
        <dependency>
            <groupId>org.glassfish.jersey.media</groupId>
            <artifactId>jersey-media-json-jettison</artifactId>
            <version>${version.jersey}</version>
        </dependency>
        -->
        <dependency>
            <groupId>org.glassfish.jersey.media</groupId>
            <artifactId>jersey-media-json-jackson</artifactId>
            <version>${version.jersey}</version>
        </dependency>
        <dependency>
            <groupId>org.glassfish.jersey.media</groupId>
            <artifactId>jersey-media-json-processing</artifactId>
            <version>${version.jersey}</version>
        </dependency>
        <dependency>
            <groupId>org.glassfish.jersey.media</groupId>
            <artifactId>jersey-media-multipart</artifactId>
            <version>${version.jersey}</version>
        </dependency>
        <dependency>
            <groupId>org.glassfish.jersey.media</groupId>
            <artifactId>jersey-media-sse</artifactId>
            <version>${version.jersey}</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/com.fasterxml.jackson.core/jackson-annotations -->
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-annotations</artifactId>
            <version>2.8.8</version>
        </dependency>

        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-core</artifactId>
            <version>2.8.8</version>
        </dependency>

        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>
            <version>2.8.8</version>
        </dependency>


        <!-- https://mvnrepository.com/artifact/org.glassfish.jersey.containers/jersey-container-jetty-http -->
        <dependency>
            <groupId>org.glassfish.jersey.containers</groupId>
            <artifactId>jersey-container-jetty-http</artifactId>
            <version>${version.jersey}</version>
        </dependency>


        <!-- if you are using Jersey client specific features -->
        <!--
        <dependency>
            <groupId>org.glassfish.jersey.core</groupId>
            <artifactId>jersey-client</artifactId>
            <version>${version.jersey}</version>
        </dependency>
        -->

        <!-- https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk -->
        <!-- <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk</artifactId>
            <version>${version.aws}</version>
        </dependency> -->

        <!-- https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-core -->
        <!-- <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk-core</artifactId>
            <version>${version.aws}</version>
        </dependency>-->

        <!-- https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-s3 -->
        <!-- <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk-s3</artifactId>
            <version>${version.aws}</version>
        </dependency>-->

        <!-- https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-athena -->
        <!-- <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk-athena</artifactId>
            <version>1.11.133</version>
        </dependency>-->

        <!-- https://mvnrepository.com/artifact/com.amazonaws.athena.jdbc/AthenaJDBC41 -->
        <!-- <dependency>
            <groupId>com.amazonaws.athena.jdbc</groupId>
            <artifactId>AthenaJDBC41</artifactId>
            <version>1.1.0-atlassian-hosted</version>
        </dependency> -->


        <!-- https://mvnrepository.com/artifact/com.facebook.presto/presto-client -->
        <!-- <dependency>
            <groupId>com.facebook.presto</groupId>
            <artifactId>presto-client</artifactId>
            <version>0.177</version>
        </dependency>-->


        <!-- https://mvnrepository.com/artifact/com.facebook.presto/presto-spi -->
        <!-- <dependency>
            <groupId>com.facebook.presto</groupId>
            <artifactId>presto-spi</artifactId>
            <version>0.177</version>
        </dependency>-->

        <!-- https://mvnrepository.com/artifact/com.facebook.presto/presto-tpch -->
        <!-- <dependency>
            <groupId>com.facebook.presto</groupId>
            <artifactId>presto-tpch</artifactId>
            <version>0.177</version>
        </dependency>-->


        <!-- https://mvnrepository.com/artifact/com.facebook.presto/presto-jdbc -->
        <!-- <dependency>
            <groupId>com.facebook.presto</groupId>
            <artifactId>presto-jdbc</artifactId>
            <version>0.60</version>
        </dependency>-->

        <!-- https://mvnrepository.com/artifact/com.facebook.presto/presto-parser -->
        <!-- <dependency>
            <groupId>com.facebook.presto</groupId>
            <artifactId>presto-parser</artifactId>
            <version>0.177</version>
        </dependency>-->

        <!-- https://mvnrepository.com/artifact/org.jooq/jooq -->
        <!-- <dependency>
            <groupId>org.jooq</groupId>
            <artifactId>jooq</artifactId>
            <version>3.9.3</version>
        </dependency>-->


        <!-- https://mvnrepository.com/artifact/javax.servlet/javax.servlet-api -->
        <dependency>
            <groupId>javax.servlet</groupId>
            <artifactId>javax.servlet-api</artifactId>
            <version>3.1.0</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.slf4j/slf4j-api -->
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-api</artifactId>
            <version>1.7.25</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.slf4j/slf4j-log4j12 -->
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-log4j12</artifactId>
            <version>1.7.25</version>
            <scope>test</scope>
        </dependency>

        <!-- https://mvnrepository.com/artifact/log4j/log4j -->
        <dependency>
            <groupId>log4j</groupId>
            <artifactId>log4j</artifactId>
            <version>1.2.17</version>
        </dependency>

        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-simple</artifactId>
            <version>1.7.25</version>
        </dependency>


        <!-- https://mvnrepository.com/artifact/com.google.inject/guice -->
        <!--<dependency>
            <groupId>com.google.inject</groupId>
            <artifactId>guice</artifactId>
            <version>4.0</version>
        </dependency>-->

        <!-- https://mvnrepository.com/artifact/com.google.inject.extensions/guice-servlet -->
        <!--<dependency>
            <groupId>com.google.inject.extensions</groupId>
            <artifactId>guice-servlet</artifactId>
            <version>4.0</version>
        </dependency>-->

        <!-- https://mvnrepository.com/artifact/io.airlift/http-client -->
        <!--<dependency>
            <groupId>io.airlift</groupId>
            <artifactId>http-client</artifactId>
            <version>0.148</version>
        </dependency>-->

        <!-- https://mvnrepository.com/artifact/io.airlift/json -->
        <!--<dependency>
            <groupId>io.airlift</groupId>
            <artifactId>json</artifactId>
            <version>0.148</version>
        </dependency>-->

        <!-- https://mvnrepository.com/artifact/org.apache.httpcomponents/fluent-hc -->
        <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>fluent-hc</artifactId>
            <version>4.5.3</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.komamitsu/fluency -->
        <!--<dependency>
            <groupId>org.komamitsu</groupId>
            <artifactId>fluency</artifactId>
            <version>1.3.0</version>
        </dependency>-->

        <!--<dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <version>1.16.16</version>
            <scope>provided</scope>
        </dependency>-->

        <!-- https://mvnrepository.com/artifact/me.geso/tinyorm -->
        <!--<dependency>
            <groupId>me.geso</groupId>
            <artifactId>tinyorm</artifactId>
            <version>1.13.0</version>
        </dependency>-->

        <!-- https://mvnrepository.com/artifact/net.sf.jopt-simple/jopt-simple -->
        <dependency>
            <groupId>net.sf.jopt-simple</groupId>
            <artifactId>jopt-simple</artifactId>
            <version>5.0.3</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.eclipse.jetty.aggregate/jetty-all -->
        <dependency>
           <groupId>org.eclipse.jetty.aggregate</groupId>
           <artifactId>jetty-all</artifactId>
           <version>${version.jetty}</version>
       </dependency>

       <dependency>
           <groupId>org.eclipse.jetty</groupId>
           <artifactId>jetty-server</artifactId>
           <version>${version.jetty}</version>
       </dependency>


       <dependency>
           <groupId>org.eclipse.jetty</groupId>
           <artifactId>jetty-servlet</artifactId>
           <version>${version.jetty}</version>
       </dependency>

       <!-- https://mvnrepository.com/artifact/org.eclipse.jetty/jetty-http -->
       <dependency>
            <groupId>org.eclipse.jetty</groupId>
            <artifactId>jetty-http</artifactId>
            <version>${version.jetty}</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.eclipse.jetty/jetty-util -->
        <dependency>
            <groupId>org.eclipse.jetty</groupId>
            <artifactId>jetty-util</artifactId>
            <version>${version.jetty}</version>
        </dependency>

        <dependency>
            <groupId>org.eclipse.jetty</groupId>
            <artifactId>jetty-io</artifactId>
            <version>${version.jetty}</version>
        </dependency>


        <!-- https://mvnrepository.com/artifact/org.eclipse.jetty/jetty-io -->
        <!--<dependency>
            <groupId>org.eclipse.jetty</groupId>
            <artifactId>jetty-io</artifactId>
            <version>9.3.17.v20170317</version>
        </dependency> -->

    </dependencies>
    
</project>

I will also try to post ,

  • The complete end-to-end sample project with Jersey + Jetty
  • A sample project to develop REST API to work with Amazon ATHENA & S3.
  • A sample project to develop REST API to work with Facebook PRESTO & Hive & S3.

Hope the the article helps.

Cheers …..