External Cassandra and Zookeper Servers

Download and install the following Apache products.

Deployment model is as follows.

Figure: External Cassandra and ZooKeeper Servers with Broker Nodes Pointed to Them.

This deployment model has common Cassandra and ZooKeeper servers. Both of them can be on the same host or in different hosts. In this example, assume there are two different hosts.

There are three Message Broker servers in three different hosts as follows:

Configure Cassandra Server

The Cassandra server should be configured to listen to all the broker nodes, for which, the following entries should be changed in the "Cassandra.yaml" file which is located in "apache-cassandra-1.0.1/conf/" directory.

1. Change the data storage directories of Cassandra. By default it points to location "/var/lib/". But it is recommended to keep the data locally inside the folder where you have the server setup. Therefore, change it to "./repository/database" in all three entries as follow:

# directories where Cassandra should store data on disk.
data_file_directories:
- ./repository/database/cassandra/data

# commit log
commitlog_directory: ./repository/database/cassandra/commitlog

# saved caches
saved_caches_directory: ./repository/database/cassandra/saved_caches

2. Change the 'listen address' of the server. By default it is "localhost". Change it to the ip address of the host where you have the Cassandra server.

   listen_address: 192.168.0.100

3. Change the RPC address of the server. By default it is "localhost". Change it to the ip address of the host where you have the cassandra server. If Cassandra is running on a different port from the default one ( which is 9160) change the rpc_port as well

rpc_address: 192.168.0.100
rpc_port: 9160

4.  Change the Broadcast address of the server, which is used to broadcast with other Cassandra nodes in the cluster. You can simply leave this as blank, then it will be changed to the same value of listen_address by default. After that save the cassandra.yaml file once done.

broadcast_address:

5. Start the Cassandra server by executing the script "cassandra" with the option -f. The -f option enures that the service starts in the foreground and logs gratuitously to standard-out.

./cassandra -f

Configure ZooKeeper Server

6. Next, configure the ZooKeeper Server to listen to all broker nodes by renaming the "zoo_sample.cfg" file in "zookeeper-3.3.4/conf" folder to "zoo.cfg" and, doing the following modifications in that file.

• Change the data directory to store data local to the installation directory.
• Add the entry "clientPortAddress" by providing the ip of the localhost.
  

The modified file looks as follows.

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial synchronization
# can take for Zookeeper server to connect to leader
initLimit=10
# The number of ticks that can pass between sending
# a request and getting an acknowledgement. If the
# leader does not hear from the follower within
# this limit, it drops the follower from the cluster.
syncLimit=5
# the directory where the snapshot is stored
dataDir=$ZK_HOME/data
# the port at which the clients will connect 
clientPort=2181
clientPortAddress=192.168.0.101

You also need to setup "Zookeeper Server To Accept SASL Connections" as follows.

6.1 Add the following data into the zoo.cfg file.

authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
#requireClientAuthScheme=sasl
# renew server-side ticket once an hour. 1000*60*60 = 3600000 milliseconds
jaasLoginRenew=3600000

6.2 Create a new file called 'jaas.conf' in $ZK_HOME/conf/ directory and add the following contents into it.

Server {
       org.apache.zookeeper.server.auth.DigestLoginModule required
       user_super="admin"
       user_admin="admin";
};
Client {
       org.apache.zookeeper.server.auth.DigestLoginModule required
       user_super="admin"
       user_admin="admin";
};

In general it is enough to add authentication parameters for the Zookeeper server. However if you enable #requireClientAuthScheme=sasl in the zoo.cfg file, the Client authentication parameters are required too. Therefore keep it added to jaas.conf as well.

6.3 Create a new file called 'java.env' in $ZK_HOME/conf/ directory and add the path to the 'jaas.conf' file in there. For example,

SERVER_JVMFLAGS="-Djava.security.auth.login.config=$ZK_HOME/conf/jaas.conf"

The complete reference can be found in guide here .

7. Save the file and start the ZooKeeper server by executing "zkServer.sh" script with the option start. For example,

sh zkServer.sh start

Configure Broker Nodes

8. In each Message Broker node, modify the $CARBON_HOME/repository/conf/advanced/qpid-config.xml file as follows:

• Enable clustering
• Change the server of the Zookeeper

<clustering>

          <enabled>true</enabled>
          <OnceInOrderSupportEnabled>false</OnceInOrderSupportEnabled>
          <externalCassandraServerRequired>true</externalCassandraServerRequired>
          <externalZookeeperServerRequired>true</externalZookeeperServerRequired>
          <coordination>
              <!-- Apache Zookeeper Address -->
              <ZooKeeperConnection>192.168.0.101:2181</ZooKeeperConnection>
              <!-- Format yyyy-MM-dd HH:mm:ss -->
              <ReferenceTime>2012-02-29 08:08:08</ReferenceTime>
          </coordination>
....

9. In each Message Broker node, modify $CARBON_HOME/repository/conf/advanced/qpid-virtualhosts.xml file as follows.

• Change the connection string by pointing to the cassandra server

<store>
     <class>org.wso2.andes.server.store.CassandraMessageStore</class>
     <username>admin</username>
     <password>admin</password>
     <cluster>ClusterOne</cluster>
     <idGenerator>org.wso2.andes.server.cluster.coordination.TimeStampBasedMessageIdGenerator</idGenerator>
     <connectionString>192.168.0.100:9160</connectionString>
</store>

10. Start all three Message Broker nodes.