Observability
Performance issues, errors, and exceptions are unfortunate events that may occur in a production environment. In order to identify an unfortunate event, observing the production environment is essential. The API Microgateway's observability feature supports monitoring and distributed tracing inorder to provide a clear picture of the performance of the Microgateway. The API Micorgateway exposes a metrics endpoint which gives metrics related to service level data, client-side data as well as some process-related data such as memory and CPU usage. These metrics can then be fed into a monitoring toolkit such as Prometheus. In addition to this, we could configure an analytics dashboard such as Grafana to visualize the metrics generated by Prometheus. For tracing Jaeger or Zipkins could be configured.
How to enable the Observability
By default, the Observability feature is disabled. The feature can be enabled in one of the following ways.
Configure Prometheus
Before you start you have to create a user according to your preference in the configuration file. This user creation is essential to configure the Prometheus server.
Note :
If you are only running tracing separately this user will not be in use.
Please find a sample configuration.
You can find the micro gateways configuration file in the path /MGW_runtime/conf/micro-gw.conf. Default configurations are as follows.
[b7a.users]
[b7a.users.prometheus]
password= "@sha256:{5e884898da28047151d0e56f8dc6292773603d0d6aabbdd62a11ef721d1542d8}"
scopes= "observability"
Configure Prometheus on docker
Step 1:
Create a prometheus.yml file in /tmp/ directory.
Step 2:
Add the following content to /tmp/prometheus.yml.
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: 'Ballerina_Metrics'
basic_auth:
username: prometheus
password: 'password'
scheme: https
tls_config:
insecure_skip_verify: true
metrics_path: /metrics
static_configs:
- targets: ['<user_IP>:9000']
- job_name: 'JMX_Metrics'
basic_auth:
username: prometheus
password: 'password'
scheme: https
tls_config:
insecure_skip_verify: true
metrics_path: /jmxMetrics
static_configs:
- targets: ['<user_IP>:9000']
Here the targets '<user_IP>:9000’' should contain the host and the secured port of the service which is given under secure port in [b7a.observability.metrics.prometheus]. Also, you need to give step 1 created username and password credentials under the basic_auth section.
We use tls config:insecure skip verify as true to skip the server certificate verification.
Step 3:
Start the Prometheus server in a Docker container with below command.
docker run -p 19090:9090 -v /tmp/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus
Step 4:
Go to http://localhost:19090/ and check whether you can see the Prometheus graph.
Configuring Grafana
Let’s use Grafana to visualize metrics in a dashboard. For this, we need to install Grafana and configure Prometheus as a data source. Follow the below-provided steps and configure Grafana.
Step 1:
Start Grafana as a Docker container with below command.
docker run -d --name=grafana -p 3000:3000 grafana/grafana
For more information refer Grafana in Docker Hub.
Step 2:
Go to http://localhost:3000/
to access the Grafana dashboard running on Docker.
Step 3:
Login to the dashboard with the default user, username: admin and password: admin
Step 4:
Add Prometheus as a data source with Browser access configuration as provided below.
Step 5:
Import the Grafana dashboard designed to visualize Microgateway performance from https://grafana.com/grafana/dashboards/12061
Configure Jaeger
Jaeger is the default distributed tracing system that is supported. There are many possible ways to deploy Jaeger and you can find more information on this link. Here we focus on all in one deployment with Docker.
Step 1:
Install Jaeger via Docker and start the Docker container by executing below command.
docker run -d -p5775:5775/udp -p6831:6831/udp -p6832:6832/udp -p5778:5778 -p16686:16686 -p14268:14268 jaegertracing/all-in-one:latest
Step 2:
Go to http://localhost:16686 and load the web UI of the Jaeger to make sure it is functioning properly.
Monitoring Metrix
Metrics, by default, supports Prometheus. In order to support Prometheus, an HTTPS endpoint starts with the contexts of /metrics and /jmxMetrics on default port 9000 is used. Here the endpoints are secured with Basic Auth. So you have to add a user and password in the microgateway conf file and use the same credentials to configure the Prometheus server also.
[b7a.users]
[b7a.users.prometheus]
password="@sha256:{5e884898da28047151d0e56f8dc6292773603d0d6aabbdd62a11ef721d1542d8}"
scopes = "observability"
Prometheus.yml
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: 'Ballerina_Metrics'
basic_auth:
username: prometheus
password: 'password'
scheme: https
tls_config:
insecure_skip_verify: true
metrics_path: /metrics
static_configs:
- targets: ['10.100.4.38:9000']
- job_name: 'JMX_Metrics'
basic_auth:
username: prometheus
password: 'password'
scheme: https
tls_config:
insecure_skip_verify: true
metrics_path: /jmxMetrics
static_configs:
- targets: ['10.100.4.38:9000']