Distributed Message Tracer

The Distributed Message Tracer solution allows you to trace the list of events that belong to a message and search through its results. A message is a single user activity that may invoke one or more services that independently generate events that are sent to WSO2 SP for processing. In addition, the activity itself generates an event that is considered the parent events.

e.g., When a transaction being processed passes through many subsystems, you can search through events collected from different subsystems to check whether the transaction is completed, the subsystem at which it is currently being processed etc. 

This solution allows you to set up the Message Tracers to match different scenarios.

 The Distributed Message Tracer solution is packed with WSO2 SP by default. To use this solution, follow the sections below:

Setting up the Message Tracer

This step involves configuring a Siddhi application in a way that allows the required details relating to activities can be captured and processed as required. To do this, let's define a business rule via the WSO2 SP Business Rules Manager as follows. For more information, see  Creating Business Rules . 

1. In your terminal, navigate to the <SP_HOME>/bin directory. Then issue one of the following commands to start the dashboard server.
   
   • On Windows: dashboard.bat --run
   • On Linux/Mac OS: ./dashboard.sh
    
2. Start a WSO2 SP worker runtime by issuing one of the following commands from the <SP_HOME>/bin directory.
   
   • On Windows: worker.bat --run
   • On Linux/Mac OS: ./worker.sh
3. Access the Business Rules Manager via one of the following URLs.
   
   • For HTTP: http://<SP_HOST>:<HTTP_PORT>/business-rules (e.g.,  http://0.0.0.0:9090/business-rules )
   • For HTTPS: https://<SP_HOST>:<HTTPS_PORT>/business-rules (e.g.,  https://0.0.0.0:9443/business-rules )
4. If you do not have any business rules that are already created, click Create. If there are existing rules, click + to create a new rule.
    
5. Click Template to create the business rule from the business template that is already available in the Distributed Message Tracer solution.
    
6. Click on Message Tracer to create your business rule from the Message Tracer template group.
   
   This opens the Message Tracer page. 
7. To configure the Siddhi application, select Message Tracer App Template in the Rule Template field.
     
   This opens the Message Tracer App Template business rules template. 

8. Enter the required information in the fields of the Message Tracer App Template business rules template as follows.

   Field	Description
   Business Rule Name	A unique name for the business rule.
   Span Stream Definition	This defines the source and the stream configuration to capture the required information.
   Component Name	The module that generates the associated span. This can be constant or an attribute you included in the Span Stream Definition field.
   Trace ID	The name of the attribute in the Span Stream Definition that captures the ID of the trace. The trace refers to the complete tracing carried out for the activity flow. A message flow may span over multiple services. The trace ID is unique to each message flow.
   Span ID	The name of the attribute in the Span Stream Definition that captures the ID of the span. When the message flow spans over multiple services, the extent of the flow that relates to one of the services is identified as a single span with a unique ID.
   Baggage Items	The name of the attribute in the Span Stream Definition to capture baggage items. Baggage items are common payloads that move across the trace.
   Parent ID	The name of the attribute in the Span Stream Definition that captures the parent ID of the span. This is relevant when an invoked service in the message flow in turn invokes another service. In this scenario, the span of the first service is he parent of the span of the second service.
   Service Name	The name of the attribute in the Span Stream Definition to capture the name of the service that generates the event. The span ID displayed in the Message Tracer dashboard to visualize the trace is derived from the service name. Therefore, each service needs to have a unique name. You can enter an concatenation to generate a unique service name. e.g., The default value str:concat(correlation_activity_id, timestamp) generates the service name by combining the value for the start time attribute (i.e., timestamp) and the message ID (i.e., trace ID).
   Start Time	The name of the attribute in the Span Stream Definition to capture the start time of the span in milliseconds.
   End Time	The name of the attribute in the Span Stream Definition to capture the end time of the span in milliseconds.
   Tags	The name of the attribute in the Span Stream Definition to capture tags. A tag is a JSON payload specific to a span. You can use a concatenation via str:concat.
   Span References	The name of the attribute in the Span Stream Definition to capture span references.
   Parent Span is Defined	This field specifies whether the Span Stream Definition needs to capture the ID of the parent span or not. Is this is set to false the parent ID is generated and assigned by the system.
   Parent Span Service Name	The name to be assigned to the parent span if the value for the Parent Span is Defined field is false . This can be a concatanation generated by Siddhi (e.g., str:concat(\"parent\", \"_\", e1.startTime, \"-\", e2[last].endTime).
   Parent Operation Name	The operation name to be assigned to the parent span generated by Siddhi. This can be a concatanation (e.g., str:concat(\"parent\", \"_\", e1.startTime, \"-\", e2[last].endTime).
   Parent Tags	The tags specific to the parent span. This is required only if the value for the Parent Span is Defined field is false and the parent span is generated via Siddhi.
   Parent Baggage Items	The baggage items of the parent span. This is required only if the value for the Parent Span is Defined field is false and the parent span is generated via Siddhi.
   Parent Span Reference	The span reference of the parent span. This is required only if the value for the Parent Span is Defined field is false and the parent span is generated via Siddhi.
   Approximate Time for Complete Trace	This defines an approximate time duration to complete the trace. The spans that occured within this time duration are considered to determine the parent span. This is required only if the value for the Parent Span is Defined field is false.
   No of Events per Trace	The number of events generated in the trace. This is required to determine the parent span only if the value for the Parent Span is Defined field is false .
   Parent Span Service Name	The name assigned to the parent span if the value for the Parent Span is Defined field is false . This can be a constant value or a concatanation to derive it using the values entered for the Approximate Time for Complete Trace and No of Events per Trace fields (e.g., str:concat("parent", "_", e1.startTime, "-", e2[last].endTime)).

9. Click SAVE & DEPLOY. The rule you create is added as shown in the example below.
   

10. Click +and enter another business rule from the Message Tracer Source Template template with the following information.

    Field	Description
    Business Rule Name	A unique name for the business rule.
    Source Type	The type of the source via which events relating to activities are received.
    Map Type	The format in which events relating to activities are received.
    Source Options	Here, you can enter optional properties to configure the source. The properties entered depends on the source type you selected. e.g., if the source type is kafka, you can enter topic.list='kafka_topic', partition.no.list='0', threading.option='single.thread.

11. Click SAVE & DEPLOY.

Tracing messages in the dashboard

To trace activities via the Message Tracer dashboard, follow the procedure below:

1. Access the Dashboard Portal via one of the following URLs.
   • http://<SP_HOST>:<HTTP_PORT>/portal (e.g.,  http://localhost:9290/portal )
   • https://<SP_HOST>:<HTTPS_PORT>/portal (e.g.,  https://localhost:9643/portal )
2. Log in with your credentials. The Distributed Message Tracer dashboard is included in the dashboard listing as shown below.
   
3. Click on the Distributed Message Tracer dashboard to open the dashboard. The widgets in the dashboard are displayed as follows.
   
4. To filter the information you want to view in the dashboard, do the following:
   1. In the Service field, select the service for which you want to view message statistics.
   2. Click on the required date range in the Date Range widget to view message statistics for the required time period. If the time period for which you want to view statistics is not displayed in this widget, you can click Custom and select a specific time interval.

Once you enter information to filter message statistics to be displayed, click SEARCH. The available traces are displayed in the Tracing List section as shown in the example below.
 

Once you click on the required trace ID, information related to the trace is displayed as shown in the following example.

Using the WSO2 SP Message Trace client

To publish messages to be traced WSO2 SP Message Trace client, you need to wrap the WSO2 SP Message Trace client as shown below once you develop the service, and get a WSO2 SP Tracer instance.

// createStreamProcessorTracerClient instance
StreamProcessorTracerClient streamProcessorTracerClient = new StreamProcessorTracerClient();
Properties tracerProperties = new Properties();
 
// read configuration file and populate tracerProperties.
// The following properties needed to initialize the tracer.
/*
 * reporter.wso2sp.publisher.type: thrift
 * reporter.wso2sp.publisher.username: admin
 * reporter.wso2sp.publisher.password: admin
 * reporter.wso2sp.publisher.url: tcp://localhost:7611
 * reporter.wso2sp.publisher.authUrl: ssl://localhost:7711
 * reporter.wso2sp.publisher.service.name: WSO2 MSF4J
 * trace.name: wso2sp
 * reporter.wso2sp.publisher.databridge.agent.config: /Users/ramindu/wso2/git/sinthuja/msf4j/samples/message-tracing/open-tracing/src/main/resources/data.agent.config.yaml
 * javax.net.ssl.trustStore: /Users/ramindu/wso2/git/sinthuja/msf4j/samples/message-tracing/open-tracing/src/main/resources/wso2carbon.jks
 * javax.net.ssl.trustStorePassword: wso2carbon
 */
tracerProperties.setProperty(WSO2SP_REPORTER_PUBLISHER_TYPE, configRegistry.getConfigOrDefault(getFullQualifiedConfig(WSO2SP_REPORTER_PUBLISHER_TYPE), DEFAULT_WSO2SP_REPORTER_PUBLISHER_TYPE_CONFIG));
tracerProperties.setProperty(WSO2SP_REPORTER_USERNAME, configRegistry.getConfigOrDefault(getFullQualifiedConfig(WSO2SP_REPORTER_USERNAME), DEFAULT_WSO2SP_REPORTER_USERNAME));
tracerProperties.setProperty(WSO2SP_REPORTER_PASSWORD, configRegistry.getConfigOrDefault(getFullQualifiedConfig(WSO2SP_REPORTER_PASSWORD), null));
tracerProperties.setProperty(WSO2SP_REPORTER_URL, configRegistry.getConfigOrDefault(getFullQualifiedConfig(WSO2SP_REPORTER_URL), DEFAULT_WSO2SP_REPORTER_URL));
tracerProperties.setProperty(WSO2SP_REPORTER_AUTHURL, configRegistry.getConfigOrDefault(getFullQualifiedConfig(WSO2SP_REPORTER_AUTHURL), DEFAULT_WSO2SP_REPORTER_AUTHURL));
tracerProperties.setProperty(WSO2SP_REPORTER_AUTHURL, configRegistry.getConfigOrDefault(getFullQualifiedConfig(WSO2SP_REPORTER_AUTHURL), DEFAULT_WSO2SP_REPORTER_AUTHURL));
tracerProperties.setProperty(WSO2SP_REPORTER_DATABRIDGE_AGENT_CONFIG, configRegistry.getConfigOrDefault(getFullQualifiedConfig(WSO2SP_REPORTER_DATABRIDGE_AGENT_CONFIG), null));
tracerProperties.setProperty(WSO2SP_REPORTER_TRUSTSTORE, configRegistry.getConfigOrDefault(getFullQualifiedConfig(WSO2SP_REPORTER_TRUSTSTORE), null));
tracerProperties.setProperty(WSO2SP_REPORTER_TRUSTSTORE_PASSWORD, configRegistry.getConfigOrDefault(getFullQualifiedConfig(WSO2SP_REPORTER_TRUSTSTORE_PASSWORD), null));
tracerProperties.setProperty(TRACER_NAME, TRACER_VALUE);
streamProcessorTracerClient.init(tracerProperties);
 
//create the trace using the service name and ScopeManager and return the WSO2 SP tracer
streamProcessorTracerClient.getTracer(serviceName, NoOpSco

Using the WSO2 SP Message Trace Client with Ballerina

To run the Message Trace Client with Ballerina, follow the steps below:

1. Clone the ballerina-observability repository.
2. To build the ballerina-sp-extension component, navigate to the ballerina-observability/tracing-extensions/modules/ballerina-sp-extension directory and issue the following command:
   mvn clean install
   
   This creates JAR files in the target of the component.
3. Copy the JAR files created in the ballerina-sp-extension component, and place them in the bre/lib directory of the Ballerina distribution.

4. Create a ballerina sample service as follows, and save it as HelloWorldService.bal in a preferred location in your machine.

   import ballerina/http;
   import ballerina/log;
   service<http:Service> hello bind { port: 9099 } {
       sayHello(endpoint caller, http:Request req) {
           http:Response res = new;
           res.setPayload("Hello, World!");
           caller->respond(res) but { error e => log:printError(
                                                     "Error sending response", err = e) };
       }
   }

5. Download the resources from here and unzip.

   You need to change the file path destinations to the resource files accordingly.

6. Add the following to ballerina.conf file. You need to change the file path destinations to the javax.net.ssl.trustStore.

   [b7a.observability.tracing]
   enabled=true
   name="wso2sp"
   
   [b7a.observability.tracing.wso2sp]
   reporter.wso2sp.publisher.type="thrift"
   reporter.wso2sp.publisher.username="admin"
   reporter.wso2sp.publisher.password="admin"
   reporter.wso2sp.publisher.url="tcp://localhost:7611"
   reporter.wso2sp.publisher.authUrl="ssl://localhost:7711"
   reporter.wso2sp.publisher.databridge.agent.config="/Users/ramindu/wso2/message-tracing/open-tracing/src/main/resources/data.agent.config.yaml"
   javax.net.ssl.trustStore="/Users/ramindu/wso2/message-tracing/open-tracing/src/main/resources/wso2carbon.jks"
   javax.net.ssl.trustStorePassword="wso2carbon"
   reporter.wso2sp.publisher.service.name="ballerina_hello_world"

Testing the Distributed Message Tracer with Ballerina

After setting up the Message Tracer to run with Ballerina as specified in the previous section, you can test the Distributed Message Tracer with Ballerina by following the steps below:

1. Copy the ballerinatracer_0.siddhi file from the resources you downloaded and copy them to <SP_HOME>/wso2/worker/deployment/siddhifiles directory of your SP worker.
2. Start the SP worker my issuing one of the following commands.
   • On Windows: worker.bat --run
   • On Linux/Mac OS: ./worker.sh
3. Execute a GET command with the following URL.
   

   http://localhost:9099/hello/sayHello

    
4. Access the Dashboard Portal and log in by entering admin as both the username and the password. Then click on the Distributed Message Tracer dashboard to open it. For more detailed information about accessing the Distributed Message Tracer dashboard, see Tracing messages in dashboard.
5. Click Search to list all the Trace IDs, and click on the available Trace IDs on the Tracing List widget.
   
   The Ballerina service message trace is displayed as follows: