When WSO2 Carbon servers are deployed in a clustered mode on Kubernetes, the Kubernetes Membership Scheme enables automatically discovering these servers.
| Tip |
|---|
|
Once a WSO2 Carbon server starts, the Kubernetes Membership Scheme queries the Kubernets API to retrieve the IP addresses of the pods/containers of the cluster. Next, the configurations of the Hazelcast cluster instance is updated with the newly found pod IP addresses. As a result, the Hazelcast instance gets connected with all the other members in the cluster. When a new member is added to the cluster, all the other members automatically get connected to the new member. |
The Kubernetes Membership Scheme supports finding the pod IP addresses using the Kubernetes API as well as the DNS.
| Auitabs |
|---|
|
| Auitabspage |
|---|
| title | Using the Kubernetes API to Resolve Pod IP Addresses |
|---|
| Follow the steps below to configure the Kubernetes Membership Scheme to use the Kubernets API: Update the axis2.xml file, in the <CARBON_HOME>/repository/conf/axis2 directory with the following configurations. | Parameter | Description |
|---|
| membershipScheme | This is the membership scheme that will be used to manage membership of nodes in a cluster, e.g., kubernetes. | | domain | This is the clustering domain/group. There will not be any interference between nodes in different domains. Messages received from members outside the domain will generally be ignored. For special messages, such as cluster management messages or membership messages from members outside the group will be allowed. | | localMemberHost | This is the hostname or the IP address of the member. Set it to the pod's local IP address. Do not set it to localhost or 127.0.0.1. When a container/pod gets started, the init.sh dynamically replaces the local member IP address in WSO2 Docker images. For customizations, please change the init.sh. | | localMemberPort | This is the TCP port that is used by this member and through which other members will contact this member. | | properties | When a member joins a group, these properties are bound to this member so that other members in the group can detect these properties. | | membershipSchemeClassName |
| | KUBERNETES_NAMESPACE | This is the Kubernetes Namespace in which the pods are deployed, | | KUBERNETES_SERVICES | These are the Kubernetes Services that belong in the cluster. | | KUBERNETES_MASTER_SKIP_SSL_VERIFICATION | This defines whether the SSL certificate verification of the Kubernetes API should be carried out or not. | | USE_DNS | This configures the membership schme to DNS (default) or Kuberntes API for pod IP resolution. Set this to false . | Example: The following clustering parameters are required to communicate with the Kubernetes API. -
KUBERNETES_API_SERVER : This is the Kubernetes API endpoint,e.g., http://172.17.8.101:8080 . Alternatively, an https endpoint can be set via KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT_HTTPS. -
KUBERNETES_SERVICE_HOST : This is the Kubernetes API host name or IP address, e.g., kuberneteshostname -
KUBERNETES_SERVICE_PORT_HTTPS : This is the Kubernetes API https listening port. This must be an Integer value. -
KUBERNETES_API_SERVER_TOKEN : This is the Kubernetes Master token for authentication (optional), e.g., yourkubernetestoken. Alternatively basic authentication can be set through KUBERNETES_API_SERVER_USERNAME and KUBERNETES_API_SERVER_PASSWORD. -
KUBERNETES_API_SERVER_USERNAME : This is the Kubernetes Master username (optional), e.g., admin. -
KUBERNETES_API_SERVER_PASSWORD : This is the Kubernetes Master password (optional). -
KUBERNETES_NAMESPACE : This is the Kubernetes Namespace in which the pods are deployed. -
KUBERNETES_SERVICES : These are the Kubernetes Services that are in the cluster. -
KUBERNETES_MASTER_SKIP_SSL_VERIFICATION : This defines whether the SSL certificate verification of the Kubernetes API should be carried out or not. -
USE_DNS : This configures the membership schme to DNS (default) or Kuberntes API for pod IP resolution. For Kubernetes, set this to false . For DNS set this to true..
|
| Auitabspage |
|---|
| title | Using the DNS Lookup to Resolve Pod IP Addresses |
|---|
| Follow the steps below to use DNS lookups to reolve pod IP addresses: Download and copy the dnsjava-2.1.8.jar dependency library for DNS lookups to the <CARBON_HOME>/repository/components/lib directory. Update the axis2.xml file, in the <CARBON_HOME>/repository/conf/axis2 directory with the following configurations. | Parameter | Description |
|---|
| membershipScheme | This is the membership scheme that will be used to manage membership of nodes in a cluster, e.g., kubernetes. | | domain | This is the clustering domain/group. There will not be any interference between nodes in different domains. Messages received from members outside the domain will generally be ignored. For special messages, such as cluster management messages or membership messages from members outside the group will be allowed. | | localMemberHost | This is the hostname or the IP address of the member. Set it to the pod's local IP address. Do not set it to localhost or 127.0.0.1 . When a container/pod gets started, the init.sh dynamically replaces the local member IP address in WSO2 Docker images. For customizations, please change the init.sh. | | localMemberPort | This is the TCP port that is used by this member and through which other members will contact this member. | | properties | When a member joins a group, these properties are bound to this member so that other members in the group can detect these properties. | | membershipSchemeClassName |
| | KUBERNETES_NAMESPACE | This is the Kubernetes Namespace in which the pods are deployed, | | KUBERNETES_SERVICES | These are the Kubernetes Services that belong in the cluster. | | KUBERNETES_MASTER_SKIP_SSL_VERIFICATION | This defines whether the SSL certificate verification of the Kubernetes API should be carried out or not. | | USE_DNS | This configures the membership schme to DNS (default) or Kuberntes API for pod IP resolution. Set this to true . | Example: The following clustering parameters are required to perform DNS lookups. -
KUBERNETES_SERVICES : These are the Kubernetes Services that are in the cluster. -
KUBERNETES_NAMESPACE : This is the Kubernetes Namespace in which the pods are deployed.
The services that are used for the DNS lookup should be headless with no cluster IP. For more information, see DNS in Kubernetes.
|
|