Clustering EMM 2.1.0

The following diagram illustrates a typical deployment pattern for WSO2 Enterprise Mobility Manager.

As indicated in the above diagram, when clustering EMM, there is worker manager separation. However, this differs from standard WSO2 Carbon worker manager separation. In a standard WSO2 product cluster, worker and manager separation is derived from deployment synchronization. However, in EMM there is no deployment synchronization.

EMM includes an admin console that can be used by any user with administrative privileges. These users can perform some actions on enrolled devices and the devices can retrieve those actions by requesting the pending operations. This is done by either waking the device through push notification or configuring the device to poll at a pre-compiled frequency.

Normally administrative tasks should be run from manager node.

There are two major deployment patterns for manager node. One could be running the manager node in the private network due to security constraints and other is allowing end users to access the management nodes, so that they can control, view their devices.

Setting up the databases

The following databases are needed for the clustering.

Database Name	Description	Database Script Location
CDM core database (DM_DS)	This stores generic data about devices (such as unique identifier, device type, ownership type), device enrollment information, device operations, policy management related data, etc.	`<PRODUCT_HOME>/dbscripts/cdm/`
APIM Database (WSO2AM_DB)	This stores data related to JAX-RS APIs and OAuth token data.	`<PRODUCT_HOME>/dbscripts/apimgt/`
Registry database (REG_DB)	This acts as the registry database. This database stores governance and config registry and must be mounted to all nodes in the cluster.	`<PRODUCT_HOME>/dbscripts/`
User and permission manager (UM_DB)	This database stores the user permission related details.	`<PRODUCT_HOME>/dbscripts/`

The following databases are related to plugins. These enable you to keep the data that is essential for these devices to work (such as APNS related keys) and this data is not available in the CDM core database.

Database Name	Description	Database Script Location
iOS DB (MobileIOSDM_DS)	Stores the iOS related the data.	`<PRODUCT_HOME>/dbscripts/cdm/plugins/ios`
Android DB (MobileAndroidDM_DS)	Stores the Android related data.	`<PRODUCT_HOME>/dbscripts/cdm/plugins/android/`
Windows DB (MobileWindowsDM_DS)	Stores the Microsoft Windows related data.	`<PRODUCT_HOME>/dbscripts/cdm/plugins/windows/`

To change the datasource configurations, please change the following files.

Files to change	Datasources
`<PRODUCT_HOME>/repository/conf/datasources/master-datasources.xml`	This file must include the datasource configuration for the following databases. APIM datasource Registry datasource User permission manager datasource
`<PRODUCT_HOME>/repository/conf/datasources/emm-datasources.xml`	This file must include the datasource configuration for the following databases. CDM core datasource IOS plugin datasource Android datasource Windows datasource

See Setting up the Database for an example of how datasources are configured.

Mounting the registry

See Remote Instance and Mount Configuration Details for more information on registr mounting and why it is useful. Do the following steps to configure this.

O pen the <PRODUCT_HOME>/repository/conf/datasources/master-datasources.xml file and add the following datasource configurations.

<datasource>
         <name>WSO2REG_DB</name>
         <description>The datasource used for Registry database</description>
         <jndiConfig>
               <name>jdbc/WSO2REG_DB</name>
         </jndiConfig>
         <definition type="RDBMS">
               <configuration>
                    <url>jdbc:mysql://localhost:3306/C_REG</url>
                    <username>root</username>
                    <password></password>
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
               </configuration>
  	      </definition>
</datasource>

Add the following configurations to the <PRODUCT_HOME>/repository/conf/registry.xml file.

<dbConfig name="mounted_registry">
        <dataSource>jdbc/WSO2REG_DB</dataSource>
</dbConfig>

<remoteInstance url="https://localhost:9443/registry">
        <id>instanceid</id>
        <dbConfig>mounted_registry</dbConfig>
        <readOnly>false</readOnly>
        <enableCache>true</enableCache>
        <registryRoot>/</registryRoot>
        <cacheId>root@jdbc:mysql://localhost:3306/C_REG</cacheId>
</remoteInstance>

<mount path="/_system/config" overwrite="true">
        <instanceId>instanceid</instanceId>
        <targetPath>/_system/config</targetPath>
</mount>

<mount path="/_system/governance" overwrite="true">
        <instanceId>instanceid</instanceId>
        <targetPath>/_system/governance</targetPath>
</mount>

Configuring the user permissions manager

See Configuring User Stores for more information on configuring user management capabilities for EMM (such as connecting an LDAP or Active Directory) and adding the role based access control configurations.

Configuring the load balancer

This section provides instructions on how to configure Nginx as the load balancer. You can use any load balancer for your setup and Nginx is used here as an example. This covers the configuration in the main Nginx configuration file. Create a new nginx.conf file and put it in /etc/nginx/conf.d directory which will work as the default configuration directory for Nginx.

Open the nginx.conf file and do the following configurations for the worker node.

Note: The URL used by the worker nodes is work.emm.wso2.com. (make sure this is properly set up in DNS pointing to the load balancer)

upstream work.emm.wso2.com {
        ip_hash;
        server xxx.xxx.xxx.xxx:9763;
        server xxx.xxx.xxx.xxx:9763;
}

server {
        listen 80;
        server_name work.emm.wso2.com;
        location / {
               proxy_set_header X-Forwarded-Host $host;
               proxy_set_header X-Forwarded-Server $host;
               proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
               proxy_set_header Host $http_host;
               proxy_read_timeout 5m;
               proxy_send_timeout 5m;
               proxy_pass http://work.emm.wso2.com;

               proxy_http_version 1.1;
               proxy_set_header Upgrade $http_upgrade;
               proxy_set_header Connection "upgrade";
        }
}



upstream ssl.work.emm.wso2.com {
    ip_hash;
    server xxx.xxx.xxx.xxx:9443;
    server xxx.xxx.xxx.xxx:9443;

}

server {
listen 443;
    server_name work.emm.wso2.com;
    ssl on;
    ssl_certificate /Users/geeth/Documents/Product-Testing/clustering/emm/conf/keys/server.crt;
    ssl_certificate_key /Users/geeth/Documents/Product-Testing/clustering/emm/conf/keys/server.key;
    location / {
               proxy_set_header X-Forwarded-Host $host;
               proxy_set_header X-Forwarded-Server $host;
               proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
               proxy_set_header Host $http_host;
               proxy_read_timeout 5m;
               proxy_send_timeout 5m;
               proxy_pass https://ssl.work.emm.wso2.com;

               proxy_http_version 1.1;
               proxy_set_header Upgrade $http_upgrade;
               proxy_set_header Connection "upgrade";
        }
}

For Mutual SSL enabled setup, please note the following changes

Changes for Mutual SSL enabled deployeement

 server {
          listen 443;
               server_name ssl.work.emm.wso2.com;
               ssl                         on;
               ssl_certificate      /etc/nginx/certs/server.crt;
               ssl_certificate_key  /etc/nginx/certs/server.key;
               ssl_client_certificate /etc/nginx/certs/ca.crt;
               ssl_verify_client optional;


          location / {
               proxy_set_header X-Forwarded-Host $host;
               proxy_set_header X-Forwarded-Server $host;
               proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
               proxy_set_header Host $http_host;
               proxy_set_header PROXY-MUTUAL-AUTH-HEADER $ssl_client_s_dn;
               proxy_read_timeout 5m;
               proxy_send_timeout 5m;
               proxy_pass https://ssl.work.emm.wso2.com;

 
               proxy_http_version 1.1;
               proxy_set_header Upgrade $http_upgrade;
               proxy_set_header Connection "upgrade";
          }
       }

ssl_certificate             - This is used to define the SSL certificate of nginx
ssl_certificate_key         - This is used to define the private key of the SSL certificate of nginx

ssl_client_certificate      - CA certificate used to sign the client certificates.

ssl_verify_client           - on | off | optional | optional_no_ca Please refer the nginx documentation for more details
                              http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_verify_client;

proxy_set_header PROXY-MUTUAL-AUTH-HEADER $ssl_client_s_dn; This header is set so that the EMM server can validate the client details.

Open the nginx.conf file and do the following configurations for the manager node.

Note: The URL used by the manager nodes is mgt.emm.wso2.com. (make sure this is properly set up in DNS pointing to the load balancer)

upstream mgt.emm.wso2.com {
        ip_hash;
        server xxx.xxx.xxx.xxx:9763;
        server xxx.xxx.xxx.xxx:9763;
}

server {
        listen 80;
        server_name mgt.emm.wso2.com;
        location / {
               proxy_set_header X-Forwarded-Host $host;
               proxy_set_header X-Forwarded-Server $host;
               proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
               proxy_set_header Host $http_host;
               proxy_read_timeout 5m;
               proxy_send_timeout 5m;
               proxy_pass http://mgt.emm.wso2.com;

               proxy_http_version 1.1;
               proxy_set_header Upgrade $http_upgrade;
               proxy_set_header Connection "upgrade";
        }
}



upstream ssl.mgt.emm.wso2.com {
    ip_hash;
    server xxx.xxx.xxx.xxx:9443;
    server xxx.xxx.xxx.xxx:9443;

}

server {
listen 443;
    server_name mgt.emm.wso2.com;
    ssl on;
    ssl_certificate /Users/geeth/Documents/Product-Testing/clustering/emm/conf/keys/server.crt;
    ssl_certificate_key /Users/geeth/Documents/Product-Testing/clustering/emm/conf/keys/server.key;
    location / {
               proxy_set_header X-Forwarded-Host $host;
               proxy_set_header X-Forwarded-Server $host;
               proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
               proxy_set_header Host $http_host;
               proxy_read_timeout 5m;
               proxy_send_timeout 5m;
               proxy_pass https://ssl.mgt.emm.wso2.com;

               proxy_http_version 1.1;
               proxy_set_header Upgrade $http_upgrade;
               proxy_set_header Connection "upgrade";
        }
}

Open the nginx.conf file and do the following configurations for the key manager or identity provider node.

Note: The key manager’s URL is keymgt.emm.wso2.com. (make sure this is properly set up in DNS pointing to the load balancer)

upstream keymgt.emm.wso2.com {
        ip_hash;
        server xxx.xxx.xxx.xxx:9763;
        server xxx.xxx.xxx.xxx:9763;
}
 
server {
        listen 80;
        server_name keymgt.emm.wso2.com;
        location / {
               proxy_set_header X-Forwarded-Host $host;
               proxy_set_header X-Forwarded-Server $host;
               proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
               proxy_set_header Host $http_host;
               proxy_read_timeout 5m;
               proxy_send_timeout 5m;
               proxy_pass http://keymgt.emm.wso2.com;
 
               proxy_http_version 1.1;
               proxy_set_header Upgrade $http_upgrade;
               proxy_set_header Connection "upgrade";
        }
}



upstream ssl.keymgt.emm.wso2.com {
    ip_hash;
    server xxx.xxx.xxx.xxx:9443;
    server xxx.xxx.xxx.xxx:9443;
 
}
 
server {
listen 443;
    server_name keymgt.emm.wso2.com;
 

    ssl on;
    ssl_certificate /Users/geeth/Documents/Product-Testing/clustering/emm/conf/keys/server.crt;
    ssl_certificate_key /Users/geeth/Documents/Product-Testing/clustering/emm/conf/keys/server.key;


    location / {
               proxy_set_header X-Forwarded-Host $host;
               proxy_set_header X-Forwarded-Server $host;
               proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
               proxy_set_header Host $http_host;
               proxy_read_timeout 5m;
               proxy_send_timeout 5m;
               proxy_pass https://ssl.keymgt.emm.wso2.com;
 
               proxy_http_version 1.1;
               proxy_set_header Upgrade $http_upgrade;
               proxy_set_header Connection "upgrade";
        }
}

Generating SSL certificates to NGINX

Create SSL certificates for both the manager and worker nodes using the instructions that follow.

Create the Server Key.
$sudo openssl genrsa -des3 -out server.key 1024
Certificate Signing Request.
$sudo openssl req -new -key server.key -out server.csr
Remove the password.
$sudo cp server.key server.key.org
$sudo openssl rsa -in server.key.org -out server.key
Sign your SSL Certificate.
$sudo openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt

While creating keys, enter the host name (*.emm.wso2.com) as the common name.

Add the public key of the NGINX server to the client_truststore.jks (you can download the public key from the browser and name it as emm.wso2.com.crt). Use the following command to add the public key to client_truststore.jks.

keytool -import -alias emm.wso2.com -file emm.wso2.com.crt -keystore <PRODUCT_HOME>/repository/resources/security/client-truststore.jks

Alternatively, import the server.crt certificate that was created as a NGINX server certificate. You must change the server certificate name into server.crt instead of emm.wso2.com.crt when importing it.

Configuring the manager node

EMM has three key consoles.

EMM console - For device management related tasks such as adding policies, operations, users, roles, and devices.
Publisher - Used for adding and publishing the enterprise mobile applications.
Store - Device user can go and subscribe to the apps and it gets installed to the device.

The above three consoles could be running inside the company network due to security concerns or could be exposed through a load balancer.

Note: There can be one EMM manager node if you use policy monitoring and policy change management. If you do not plan to use those, you can have a cluster of manager nodes.

Do the following to configure the manager node.

Download and unzip the WSO2 EMM binary distribution. The extracted directory is referred to as <EMM_HOME> in this document.
Configure the HostName and the MgtHostName. To do this, edit the <EMM_HOME>/repository/conf/carbon.xml file as follows.
```
<HostName>work.emm.wso2.com</HostName>
<MgtHostName>mgt.emm.wso2.com</MgtHostName>
```
- HostName: Host name or IP address of the machine hosting this server, e.g. work.emm.wso2.com or 192.168.1.10 .This is will become part of the End Point Reference of the services deployed on this server instance.
- MgtHostName: Host name to be used for the Carbon management console.

Configure the HTTP/HTTPS proxy ports to communicate through the load balancer by editing the <EMM_HOME>/repository/conf/tomcat/catalina-server.xml file as follows.

<Connector protocol="org.apache.coyote.http11.Http11NioProtocol"

port="9763"
proxyPort="80"
redirectPort="443"
…
/>
<Connector protocol="org.apache.coyote.http11.Http11NioProtocol"
port="9443"
proxyPort="443"
redirectPort="443"
…
/>

Configure the policy monitoring frequency. You set the value in milliseconds and the default is 10 minutes. To do this, edit the <EMM_HOME>/repository/conf/cdm-config.xml file.
<MonitoringFrequency>60000</MonitoringFrequency>
Configure the QR code URL to publically expose the URL in the emm-web-agent jaggery application and the URL of the key manager. This file is in the emm-web-agent.zip file. Edit the <EMM_HOME>/repository/deployment/server/jaggeryapps/emm-web-agent/config/config.json file. Use a ZIP file browser to change the file.
```
"dynamicClientRegistrationEndPoint" : "https://keymgt.mdm.wso2.com/dynamic-client-web/register/",
…….
…...
"generalConfig" : {
"host" : "http://work.emm.wso2.com",
"companyName" : "WSO2 Enterprise Mobility Manager",
…………………….
}
```

Configuring the worker node

Set up the cluster configurations by editing the <PRODUCT_HOME>/repository/conf/axis2/axis2.xml file as follows.
1. Enable clustering for this node by setting enable to true in the clustering tag.
  <clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">
2. Set the membership scheme to wka to enable the well-known address registration method (this node sends cluster initiation messages to WKA members that are defined later).
  <parameter name="membershipScheme">wka</parameter>
3. Specify the name of the cluster this node will join.
  <parameter name="domain">wso2.emm.domain</parameter>
4. Specify the host used to communicate cluster messages.
  
  Note: You may run into issues when using host names in products based on WSO2 Carbon 4.4.0, so it is recommended that you use the IP address directly here. You can also use IP address ranges here. For example, 192.168.1.2-10.
  <parameter name="localMemberHost">xxx.xxx.xxx.xx3</parameter>
5. Specify the port used to communicate cluster messages.
  <parameter name="localMemberPort">4200</parameter>
6. Specify the well known member. In this example, the well known member is the manager node.
```
<members>
	<member>
		<hostName>xxx.xxx.xxx.xx2</hostName>
		<port>4100</port> 
	</member>
</members>
```
Configure the HostName. To do this, edit the <PRODUCT_HOME>/repository/conf/carbon.xml file to modify the following entry.
<HostName>work.emm.wso2.com</HostName>

Configure the HTTP/HTTPS proxy ports to communicate through the load balancer by editing the <PRODUCT_HOME>/repository/conf/tomcat/catalina-server.xml file as follows.

<Connector protocol="org.apache.coyote.http11.Http11NioProtocol"
port="9763"
proxyPort="80"
redirectPort="443"
…
/>
<Connector protocol="org.apache.coyote.http11.Http11NioProtocol"
port="9443"
proxyPort="443"
redirectPort="443"
…
/>

Stop running the monitoring task on worker nodes by editing the <PRODUCT_HOME>/repository/conf/cdm-config.xml and setting the monitoringEnable tag to false.

Note: This task is not federated yet. Federation will be applicable in an upcoming release.
<MonitoringEnable>false</MonitoringEnable>

Configure the identity provider URL for OAuth key validation. Do this by editing the <PRODUCT_HOME>/repository/conf/security/authenticators.xml file.

<Authenticator name="OAuthAuthenticator" disabled="false">
	<Priority>10</Priority>
	<Config>
		<Parameter name="isRemote">true</Parameter>
		<Parameter name="hostURL">https://keymgt.emm.wso2.com:443</Parameter>
		<Parameter name="adminUsername">admin</Parameter>
		<Parameter name="adminPassword">admin</Parameter>
	</Config>
</Authenticator>

Change the apk download URL to the publically exposed URL in emm-web-agent jaggery application. This file is in emm-web-agent.zip file. You can unzip it, edit the file, and then zip it again. Edit the <PRODUCT_HOME>/repository/deployment/server/jaggeryapps/emm-web-agent/config/config.json file with the following.
```
"generalConfig" : {
	"host" : "http://work.emm.wso2.com",
	"companyName" : "WSO2 Enterprise Mobility Manager",
	…………………….
}
```
Remove the EMM jaggery application from the server before starting it. This is to make sure that no worker node has the ability to control the device through the EMM web application.
Add the following to the tasks-config.xml file, which is in the <EMM_HOME>/repository/conf/etc/ directory in the Analyzer nodes.
```
<taskServerCount>2</taskServerCount>
```
About the task server count

This value indicates the number of task servers running in the cluster.
When clustering is enabled for worker nodes, they wait for other task executor nodes to startup. This depends on the number specified as the taskServerCount in the <EMM_HOME>/repository/conf/etc/tasks-config.xml file.
If the taskServerCount is 2 (which is the default value) then there should be at least two worker nodes to properly start up the cluster. Otherwise, the worker node startup will be held at the following log statement until at least one other task node startup.

Configuring the Key Manager node

When configuring the Key Manager (identity provider), you must use the EMM server as a Key Manager.

Set up the cluster configurations by editing the <PRODUCT_HOME>/repository/conf/axis2/axis2.xml file as follows.
1. Enable clustering for this node by setting enable to true in the clustering tag.
  <clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">
2. Set the membership scheme to wka to enable the well-known address registration method (this node sends cluster initiation messages to WKA members that are defined later).
  <parameter name="membershipScheme">wka</parameter>
3. Specify the name of the cluster this node will join.
  <parameter name="domain">wso2.keymt.domain</parameter>
4. Specify the host used to communicate cluster messages.
  
  Note: You may run into issues when using host names in products based on WSO2 Carbon 4.4.0, so it is recommended that you use the IP address directly here. You can also use IP address ranges here. For example, 192.168.1.2-10.
  <parameter name="localMemberHost">xxx.xxx.xxx.xx3</parameter>
5. Specify the port used to communicate cluster messages.
  <parameter name="localMemberPort">4200</parameter>
6. Specify the well known member. In this example, the well known member is the manager node.
```
<members>
<member>
<hostName>xxx.xxx.xxx.xx2</hostName>
<port>4100</port> 
</member>
</members>
```
Configure the HostName. To do this, edit the <PRODUCT_HOME>/repository/conf/carbon.xml file as follows.
<HostName>keymgt.emm.wso2.com</HostName>

Configure the HTTP/HTTPS proxy ports to communicate through the load balancer by editing the <PRODUCT_HOME>/repository/conf/tomcat/catalina-server.xml file as follows.

<Connector protocol="org.apache.coyote.http11.Http11NioProtocol"
port="9763"
proxyPort="80"
redirectPort="443"
…
/>
<Connector protocol="org.apache.coyote.http11.Http11NioProtocol"
port="9443"
proxyPort="443"
redirectPort="443"
…
/>

Remove the EMM jaggery application from the server before it starts. This is to make sure that no worker node has the ability to control the device through the EMM web application.

Testing the cluster

Do the following steps to ensure the cluster works as expected.

Restart the configured load balancer.
Start the key manager node.
sh <PRODUCT_HOME>/bin/wso2server.sh -Dsetup
Start the manager node.
sh <PRODUCT_HOME>/bin/wso2server.sh
Start the two worker nodes.
Note

Make sure to start the worker node using the command given below. Do not use -DworkerNode=true to start the worker node.
sh <PRODUCT_HOME>/bin/wso2server.sh -DworkerNode=true
Check for ‘member joined’ log messages in the worker consoles.