Kafka Installation and Configuration Guide

This assumes you are starting fresh and have no existing Kafka or ZooKeeper data.

Step 1: Download the code

Download the release and un-tar it.
> tar -xzf kafka_2.10-
> cd kafka_2.10-

Step 2: Start the server

Kafka uses ZooKeeper so you need to first start a ZooKeeper server if you don’t already have one. You can use the convenience script packaged with kafka to get a quick-and-dirty single-node ZooKeeper instance.
> bin/zookeeper-server-start.sh config/zookeeper.properties
[2013-04-22 15:01:37,495] INFO Reading configuration from: config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)

Now start the Kafka server:
> bin/kafka-server-start.sh config/server.properties
[2013-04-22 15:01:47,028] INFO Verifying properties (kafka.utils.VerifiableProperties)
[2013-04-22 15:01:47,051] INFO Property socket.send.buffer.bytes is overridden to 1048576 (kafka.utils.VerifiableProperties)

Step 3: Create a topic

Let’s create a topic named “test” with a single partition and only one replica:
> bin/kafka-topics.sh –create –zookeeper localhost:2181 –replication-factor 1 –partitions 1 –topic test
We can now see that topic if we run the list topic command:
> bin/kafka-topics.sh –list –zookeeper localhost:2181
Alternatively, instead of manually creating topics you can also configure your brokers to auto-create topics when a non-existent topic is published to.

Step 4: Send some messages

Kafka comes with a command line client that will take input from a file or from standard input and send it out as messages to the Kafka cluster. By default each line will be sent as a separate message.
Run the producer and then type a few messages into the console to send to the server.

> bin/kafka-console-producer.sh –broker-list localhost:9092 –topic test
This is a message
This is another message
Step 5: Start a consumer
Kafka also has a command line consumer that will dump out messages to standard output.

> bin/kafka-console-consumer.sh –zookeeper localhost:2181 –topic test –from-beginning
This is a message
This is another message
If you have each of the above commands running in a different terminal then you should now be able to type messages into the producer terminal and see them appear in the consumer terminal.
All of the command line tools have additional options; running the command with no arguments will display usage information documenting them in more detail.

Step 6: Setting up a multi-broker cluster

So far we have been running against a single broker, but that’s no fun. For Kafka, a single broker is just a cluster of size one, so nothing much changes other than starting a few more broker instances. But just to get feel for it, let’s expand our cluster to three nodes (still all on our local machine).
First we make a config file for each of the brokers:
> cp config/server.properties config/server-1.properties
> cp config/server.properties config/server-2.properties
Now edit these new files and set the following properties:



The broker.id property is the unique and permanent name of each node in the cluster. We have to override the port and log directory only because we are running these all on the same machine and we want to keep the brokers from all trying to register on the same port or overwrite each others data.
We already have Zookeeper and our single node started, so we just need to start the two new nodes:
> bin/kafka-server-start.sh config/server-1.properties &

> bin/kafka-server-start.sh config/server-2.properties &

Now create a new topic with a replication factor of three:

> bin/kafka-topics.sh –create –zookeeper localhost:2181 –replication-factor 3 –partitions 1 –topic my-replicated-topic
Okay but now that we have a cluster how can we know which broker is doing what? To see that run the “describe topics” command:
> bin/kafka-topics.sh –describe –zookeeper localhost:2181 –topic my-replicated-topic
Topic:my-replicated-topic PartitionCount:1 ReplicationFactor:3 Configs:
Topic: my-replicated-topic Partition: 0 Leader: 1 Replicas: 1,2,0 Isr: 1,2,0
Here is an explanation of output. The first line gives a summary of all the partitions, each additional line gives information about one partition. Since we have only one partition for this topic there is only one line.
• “leader” is the node responsible for all reads and writes for the given partition. Each node will be the leader for a randomly selected portion of the partitions.
• “replicas” is the list of nodes that replicate the log for this partition regardless of whether they are the leader or even if they are currently alive.
• “isr” is the set of “in-sync” replicas. This is the subset of the replicas list that is currently alive and caught-up to the leader.
Note that in my example node 1 is the leader for the only partition of the topic.
We can run the same command on the original topic we created to see where it is:
> bin/kafka-topics.sh –describe –zookeeper localhost:2181 –topic test
Topic:test PartitionCount:1 ReplicationFactor:1 Configs:
Topic: test Partition: 0 Leader: 0 Replicas: 0 Isr: 0
So there is no surprise there—the original topic has no replicas and is on server 0, the only server in our cluster when we created it.

Let’s publish a few messages to our new topic:
> bin/kafka-console-producer.sh –broker-list localhost:9092 –topic my-replicated-topic

my test message 1
my test message 2
Now let’s consume these messages:
> bin/kafka-console-consumer.sh –zookeeper localhost:2181 –from-beginning –topic my-replicated-topic

my test message 1
my test message 2
Now let’s test out fault-tolerance. Broker 1 was acting as the leader so let’s kill it:
> ps | grep server-1.properties
7564 ttys002 0:15.91 /System/Library/Frameworks/JavaVM.framework/Versions/1.6/Home/bin/java…
> kill -9 7564
Leadership has switched to one of the slaves and node 1 is no longer in the in-sync replica set:
> bin/kafka-topics.sh –describe –zookeeper localhost:2181 –topic my-replicated-topic
Topic:my-replicated-topic PartitionCount:1 ReplicationFactor:3 Configs:
Topic: my-replicated-topic Partition: 0 Leader: 2 Replicas: 1,2,0 Isr: 2,0
But the messages are still be available for consumption even though the leader that took the writes originally is down:
> bin/kafka-console-consumer.sh –zookeeper localhost:2181 –from-beginning –topic my-replicated-topic

my test message 1
my test message 2

| Tagged | Leave a comment

Windows Management Infrastructure(WMIC) Installation and Configuration

1] Download WMIC
ashok-pc:~# cd /tmp
ashok-pc:/tmp# wget http://www.edcint.co.nz/checkwmiplus/wmi-1.3.14.tar.gz
–2012-11-18 17:23:45– http://www.edcint.co.nz/checkwmiplus/wmi-1.3.14.tar.gz
Resolving http://www.edcint.co.nz…
Connecting to http://www.edcint.co.nz||:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 3994744 (3.8M) [application/x-gzip]
Saving to: “wmi-1.3.14.tar.gz”
100%[=========================================================================================>] 3,994,744 419K/s in 11s
2012-11-18 17:23:57 (366 KB/s) – “wmi-1.3.14.tar.gz” saved [3994744/3994744]

2] Unpack WMIC
ashok-pc:/tmp# tar xzvf wmi-1.3.14.tar.gz
ashok-pc:/tmp# cd wmi-1.3.14

3] Compile WMIC (takes about 5-10 minutes)
ashok-pc:/tmp/wmi-1.3.14# make

4] Test WMIC
Usage: [-?|–help] [–usage] [-d|–debuglevel DEBUGLEVEL] [–debug-stderr]
[-s|–configfile CONFIGFILE] [–option=name=value]
[-l|–log-basename LOGFILEBASE] [–leak-report] [–leak-report-full]
[-R|–name-resolve NAME-RESOLVE-ORDER]
[-O|–socket-options SOCKETOPTIONS] [-n|–netbiosname NETBIOSNAME]
[-W|–workgroup WORKGROUP] [–realm=REALM] [-i|–scope SCOPE]
[-m|–maxprotocol MAXPROTOCOL] [-U|–user [DOMAIN\]USERNAME[%PASSWORD]]
[-N|–no-pass] [–password=STRING] [-A|–authentication-file FILE]
[-S|–signing on|off|required] [-P|–machine-pass]
[–simple-bind-dn=STRING] [-k|–kerberos STRING]
[–use-security-mechanisms=STRING] [-V|–version] [–namespace=STRING]

//host query
Example: wmic -U [domain/]adminuser%password //host “select * from Win32_ComputerSystem”
ashok-pc:/tmp/wmi-1.3.14#/bin/wmic -U USER%PASS //HOST ‘Select Caption From Win32_OperatingSystem’
CLASS: Win32_OperatingSystem
Microsoft Windows XP Professional|Microsoft Windows XP Professional|C:\WINDOWS|\Device\Harddisk0\Partition1

| Leave a comment

MySQL Migration Version 5.1 to 5.6 in CentOS/RHEL

1] Remove Existing MySQL 5.1 Package
Yum Remove mysql

2] Update Cent OS Packages
Sudo Yum Update

3] Create a Directory for download MySQL Package
Mkdir MySQL

4] Enter into the MySQL directory
cd MySQL

5] Download MySQL 5.6 Package from URL
wget http://dev.mysql.com/get/Downloads/MySQL-5.6/MySQL-5.6.16-1.el6.x86_64.rpm-bundle.tar

6] Unzip Downloaded Tar Zip file
tar -xvf MySQL-5.6.16-1.el6.x86_64.rpm-bundle.tar

7] Install MySQL 5.6 Package
yum install MySQL*rpm

8] Start MySQL Service
/etc/init.d/mysql start

| Leave a comment

Job Security Software/IT Companies – Acceptable

Secure IT companies in India

1) Microsoft – Has projects till 2050. 

2) EDS – Most secure company in India. Not laid off any of its employees even during 2001. Has lots of projects in Defense and financial areas

3) HP – Dream Company. In-house and outsourced projects

4) TCS – A govt. Company.

5) AOL, Google and Yahoo – Best companies to work with, great job satisfaction as well as great salary and work environment. Rarely fires an employee. As they are internet based companies’ they offer lots of opportunities to grow.

6) HCL – A good company to be in. Called as a retirement company.

7) HSBC- This is the most secure company. It has never fired any employee, even when they know that the employee is showing fake experience.

8. Aricent- a communication based software company, has never fired any employee and gives great perks & incentives, lot of projects in kitty. Minimal level of attrition.

9) KPIT Cumminns Infosystems Limited — This is the most secure company not known to many. It has presently acquired CG Smith, Bangalore and has lots of projects in pipe line. Acquisitions plans will continue.

| Leave a comment

Job Security of Software/IT Companies – Dangerous and Be Careful

Top most Firing IT companies in India

1) IBM – Right now this is the most firing company for IT professionals. In the last 6 months, this company has fired nearly 20% of their employees because of BG check and performance issues. This is the most insecure company from an IT professional’s point of view. They don’t have any strategic plans at HR policies regarding employee security. No appraisals (maximum 10%). 

2) Accenture – This is second top most firing company. The firing rate is around 5%. This depends upon outsourced projects; they have a unique system where Accenture development centers around the world bid for a project coming into the company. Currently Philippines centre is taking the cake and the Indian centers are in a firing mode.

3) WIPRO – Firing people with very frequent back ground checks and firing them with out even experience letters and relieving letters (will mention as terminated from services)but will promise the employees that they will retain them. After the project is over they will fire away. Will threaten of criminal cases against such employees if they oppose the move and has also filed against some.

4) Intel – Recently joined the league. Running in heavy losses, hence firing 3000 employees in the Banglore center in a phased out manner.

5) CTS – Has a steady firing policy (checking the Educational background and previous employment and also employee performance in work). In a Recent HCL walk-in, around 50% attendees were from this company. Sadly the I-pods have not helped them.

6) CSC – Excellent package but fires folks in Background check and those on bench regularly. Recently fired 400+ employees from its subsidiary Covansys.

7) Satyam – Currently stopped firing. The Attrition rate is very high. No firing from 2005 until now when 1000 employees were fired in Hyderabad.

8. Patni — They fired so many employees that currently they are facing understaffing and deficiency with number of employees. Very high attrition rate.

9) Keane India — This USA based company is always involved in firing employees. Although they proudly say that they dont have hire and fire policy. Recently they fired java and as400 professionals after which most of the employees have started to pack their bags. Employees change this company within 1 year.

So take care before accepting offers from these companies.

| Leave a comment

Check the Particular System Opened Ports

/* This Program Scans all the opened ports in the System and print the Opened Ports
* @Author – Ashoka Bhat B M

import java.net.*;
import java.io.*;

public class CheckOpenedPorts{

public static void main(String[] args) {

Socket theSocket;
String host = “BHAT-LT-0162”;

for (int i = 0; i < 64000; i++) {
try {
//System.out.println("Looking for "+ i);
theSocket = new Socket(host, i);
System.out.println("There is a server on port " + i + " of " + host);
catch (UnknownHostException e) {
catch (IOException e) {
// must not be a server on this port

| Leave a comment

Display System IP Address Using Passing Host Names

/*This program you can pass the Host Names Array and you will get that Particular system IP Address
* @Author – Ashoka Bhat

import java.io.*;
import java.net.InetAddress;
import java.net.Socket;

public class HostIpAddressFinder {

public static void main(String[] args)throws IOException {

String[] hostList= {“SYSTEM-LT-0162″,”SYSTEM-DT-0128″,”SYSTEM-DT-0079”};
for(int counter=0;counter 0)
responseIpAddress = addr[0].getHostAddress();

for(int iter=1;iter<addressLength;++iter)
responseIpAddress += ",";
responseIpAddress += addr[iter].getHostAddress();
int lastInd = responseIpAddress.lastIndexOf(",");
responseIpAddress = responseIpAddress.substring(0, lastInd);

}catch(Exception e)
System.out.println("Error : "+e );
return "";
return responseIpAddress;


| Tagged | Leave a comment