External Cassandra Server and Default Zookeeper Server

Download and install the following Apache product.

 Deployment model is as follows.

Figure: External Cassandra and In-Built ZooKeeper in Ring Model with Broker Nodes

This deployment model has a common Cassandra server. The default ZooKeeper server packaged with the WSO2 MB can be used to create a ZooKeeper ring in the cluster.

There are three Message Broker servers in three different hosts as follows.

Configure Cassandra Server

1. First, configure the HostName in $CARBON_HOME/repository/conf/carbon.xml file by specifying the ip of the host as follows. This has to be done for all three servers.

<!-- Host name or IP address of the machine hosting this server e.g. www.wso2.org, 192.168.1.10. 
This will become part of the End Point Reference of the services deployed on this server instance. -->

<HostName>192.168.0.101</HostName>

2. Configure Cassandra server to listen to all the broker nodes by changing the following entries in "Cassandra.yaml" file located in "apache-cassandra-1.0.1/conf/" directory.

• Change the data storage directories of Cassandra, which point to "/var/lib/" by default. It is recommended to keep the data locally inside the folder where you have the server setup. Change it to "./repository/database" in all three entries as follows:

              # directories where Cassandra should store data on disk.
              data_file_directories:
                  - ./repository/database/cassandra/data

              # commit log
              commitlog_directory: ./repository/database/cassandra/commitlog

              # saved caches
              saved_caches_directory: ./repository/database/cassandra/saved_caches

• Change 'listen address' of the server, which is "localhost" by default, to the ip address of the host where Cassandra server is setup.

listen_address: 192.168.0.100

• Change RPC address of the server, which is "localhost" by default, to the ip address of the host where you have the cassandra server. If Cassandra is running on a different port from the default one ( which is 9160) change the rpc_port as well.

rpc_address: 192.168.0.100
rpc_port:9160

• Change the Broadcast address of the server, which is used to broadcast with other Cassandra nodes in the cluster. You can simply leave this as blank, then it will be changed to the same value of listen_address by default.

broadcast_address:

3. Save cassandra.yaml file and start the Cassandra server by executing the script "cassandra" with the option -f. The -f option enures that the service starts in the foreground and logs gratuitously to standard-out. For example,

./cassandra -f

Configure Zookeeper Cluster

4. Configure the ZooKeeper Servers to create a ring among all broker nodes, by adding the client port address to $CARBON_HOME/repository/conf/etc/zoo.cfg file. After modification, the file looks as follows.

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial synchronization
# can take for Zookeeper server to connect to leader
initLimit=10
# The number of ticks that can pass between sending
# a request and getting an acknowledgement. If the
# leader does not hear from the follower within
# this limit, it drops the follower from the cluster.
syncLimit=5
# the directory where the snapshot is stored
dataDir=repository/data/zookeeper
# the port at which the clients will connect 
clientPort=2181
start_zk_server=false
clientPortAddress=192.168.0.101
server.1=192.168.0.100:2888:3888
server.2=192.168.0.101:2888:3888
server.3=192.168.0.102:2888:3888

5. In the example above, zoo.cfg file is for server 192.168.0.101. Copy the same zoo.cfg file to other two servers and change the entry "clientPortAddress" to the ip of that particular server.

6. You also need to setup "Zookeeper Server To Accept SASL Connections" for each Zookeeper serveras follows.

6.1 Add the following data into the $CARBON_HOME/repository/conf/etc/zoo.cfg file.

authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
#requireClientAuthScheme=sasl
# renew server-side ticket once an hour. 1000*60*60 = 3600000 milliseconds
jaasLoginRenew=3600000

6.2 Create a new file called 'jaas.conf' in $CARBON_HOME/repository/conf/etc/ directory and add the following contents into it.

Server {
       org.apache.zookeeper.server.auth.DigestLoginModule required
       user_super="admin"
       user_admin="admin";
};
Client {
       org.apache.zookeeper.server.auth.DigestLoginModule required
       user_super="admin"
       user_admin="admin";
};

In general it is enough to add authentication parameters for the Zookeeper server. However if you enable #requireClientAuthScheme=sasl in the zoo.cfg file, the Client authentication parameters are required too. Therefore keep it added to jaas.conf as well.

6.3 Create a new file called 'java.env' in $CARBON_HOME/repository/conf/etc/ directory and add the path to the 'jaas.conf' file in there. For example,

SERVER_JVMFLAGS="-Djava.security.auth.login.config=$CARBON_HOME/repository/conf/etc/jaas.conf"

7. Create a new directory called 'zookeeper' in $CARBON_HOME/repository/data directory which will be used to keep zookeeper data. Then create a file named "myid" inside the new 'zookeeper directory' of each server and specify the id of the server in that file. For example, if Id of the the server 192.168.0.100 is "1", simply add number 1 in myid file. For more information, refer to ZooKeeper clustering setup guide: http://zookeeper.apache.org/doc/r3.3.3/zookeeperAdmin.html#sc_zkMulitServerSetup.

8. The above configuration has to be done in each message broker server appropriately.

Configure Broker Nodes

Next, do the following configuration changes in Message Broker nodes.

9. Modify $CARBON_HOME/repository/conf/advanced/qpid-config.xml file as follows.

• Enable clustering.
• Change the server ip of the ZooKeeper by adding all ZooKeeper server details.

<clustering>

          <enabled>true</enabled>
          <OnceInOrderSupportEnabled>false</OnceInOrderSupportEnabled>
          <externalCassandraServerRequired>true</externalCassandraServerRequired>

          <coordination>
              <!-- Apache Zookeeper Address -->
              <ZooKeeperConnection>192.168.0.101:2181,192.168.0.102:2181,192.168.0.103:2181</ZooKeeperConnection>
              <!-- Format yyyy-MM-dd HH:mm:ss -->
              <ReferenceTime>2012-02-29 08:08:08</ReferenceTime>
          </coordination>
...

10. Modify $CARBON_HOME/repository/conf/advanced/qpid-virtualhosts.xml file as follows.

• Change the connection string by pointing to the Cassandra server.

 <store>
       <class>org.wso2.andes.server.store.CassandraMessageStore</class>
       <username>admin</username>
       <password>admin</password>
       <cluster>ClusterOne</cluster>
       <idGenerator>org.wso2.andes.server.cluster.coordination.TimeStampBasedMessageIdGenerator</idGenerator>
       <connectionString>192.168.0.100:9160</connectionString>
  </store>

11. Start all three Message Broker nodes.