Standalone Deployment on External Apache ZooKeeper

This section provides instructions to install, setup and point an external Zookeeper server to a standalone WSO2 Message Broker instance.

Apache ZooKeeper is a high-performance coordination service for distributed applications. It exposes common services such as naming, configuration management, synchronization, and group services in a simple interface so you don't have to write them from scratch. You can use it off-the-shelf to implement consensus, group management, leader election, and presence protocols, and build on it for unique, business needs.

WSO2 recommends the following in a typical, production environment:

Using an external server or cluster instead of the in-built Cassandra server, which is not recommended when processing large volumes of queues. In this case, Apache Zookeeper is used.
Have the external server on a separate machine or a dedicated VM instance and point to it from the Message Broker.
Apache ZooKeeper runs as an ensemble of servers. Three ZooKeeper servers is the minimum recommended size for an ensemble, and recommend to be run on separate machines.

ZooKeeper Installation Prerequisites

ZooKeeper runs on Java, release 1.6 or higher (JDK 6 or greater).
Setup one of the following platforms where Apache ZooKeeper is supported in.
- GNU/Linux is supported as a development and production platform for both server and client.
- Sun Solaris is supported as a development and production platform for both server and client.
- FreeBSD is supported as a development and production platform for clients only. Java NIO selector support in the FreeBSD JVM is broken.
- Win32 is supported as a development platform for both server and client.
- MacOSX is supported as a development platform only for both server and client.

Download ZooKeeper

Download links for the latest, stable Apache ZooKeeper distribution can be found in one of the Apache Download Mirrors: http://zookeeper.apache.org/releases.html.

Note

Use ZooKeeper version 3.4.0 or later since SASL authentication with zookeeper is available on those versions : http://hbase.apache.org/book/zk.sasl.auth.html

Standalone Operation (Minimum Configuration)

Setting up a ZooKeeper server in standalone mode is straightforward. The server is contained in a single JAR file, so installation consists of creating a configuration. Once you've downloaded a stable ZooKeeper release, unpack it and change directory to the root. To start ZooKeeper, a configuration file is needed as the sample below. Create it in zookeeper-3.x.x/conf/zoo.cfg.

tickTime=2000
dataDir=/var/zookeeper
clientPort=2181

Config element	Description
tickTime	the basic time unit in milliseconds used by ZooKeeper. It is used to do heartbeats and the minimum session timeout will be twice the tickTime.
dataDir	the location to store the in-memory database snapshots and, unless specified otherwise, the transaction log of updates to the database.
clientPort	the port to listen for client connections

This file can be given any name. In this documentation, it is called conf/zoo.cfg. Change the value of dataDir to specify an existing (empty to start with) directory.

Setting up Zookeeper Server to Accept SASL Connections

WSO2 Message Broker is configured to use SASL to connect to the Zookeeper server bundled with it. Therefore, when setting up an external ZooKeeper server to work with WSO2 Message Broker, it is needed to configure ZooKeeper to accept SASL connections from Message Broker as described below.

ZooKeeper installation folder is refereed to as <ZK_HOME> throughout this section.

1. Add the following code to <ZK_HOME>/conf/zoo.cfg file.

requireClientAuthScheme=sasl is optional: if it is set to any value, it will only allow non-authenticated clients to ping, create session, close session, or sasl-authenticate.

authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
#requireClientAuthScheme=sasl
# renew server-side ticket once an hour. 1000*60*60 = 3600000 milliseconds
jaasLoginRenew=3600000

2. Inside <ZK_HOME>/conf/ folder create a file named jaas.conf and add following lines to configure DIGEST-MD5 authentication.

Server {
       org.apache.zookeeper.server.auth.DigestLoginModule required
       user_super="admin"
       user_admin="admin";
};

Note

Value given for "user_admin" is same as password value given for user name "admin" under client section of jaas.conf file in <MB_HOME>/repository/conf/security folder.

The passwords are in plain text. Therefore, both JAAS configuration files should not be readable by anyone other than the Zookeeper server process user.

3. Inside <ZK_HOME>/conf/ folder, create a file named java.env. In that file, define the full-qualified name to jaas.conf file referred to in step 2.

SERVER_JVMFLAGS="-Djava.security.auth.login.config=<ZK_HOME>/conf/jaas.conf"

Note

Change passwords and user names according to your requirements. For more information on setting SASL authentication for ZooKeeper, refer to https://cwiki.apache.org/ZOOKEEPER/zookeeper-and-sasl.html.

Configure Session Timeouts

By default zookeeper has a

minimum session time out value = 2 * tick time

maximum timeout value = 20 * tick time

If there was no communication between MB servers and ZooKeeper, server session can be timeout. As a solution we can increase session timeout by either

1. increase tick time (default 2000 milliseconds)

2. set optional parameters in zoo.cfg

minSessionTimeout=

maxSessionTimeout=

(From WSO2 MB 2.1.0 onwards a ping is done to keep the session alive. Thus this configuration can be left alone).

Start ZooKeeper

After creating the configuration file, start ZooKeeper as follows.

bin/zkServer.sh start

ZooKeeper logs messages using log4j. Depending on the log4j configuration, log messages can be seen in the console (default) and/or a log file. More information is available in the Logging section of the Programmer's Guide: http://zookeeper.apache.org/doc/r3.2.2/zookeeperProgrammers.html#Logging.