Analysing HTTPD Logs

Introduction

Every time your Web server receives a request, it makes an entry to one or more log files. These log files are useful for a variety of purposes, from statistical analysis of your visitors to forensic analysis of an attack on your server. The HTTPD logs can provide information on everything that happens on your server from the initial request, through the URL mapping process, to the final resolution of the connection (including any errors that may have occurred in the process). Thereby, HTTPD logs provide you feedback about the activities and performance of the server as well as any problem that may be occurring, to effectively manage your Web server.

This HTTPD logs sample is intended to show the capability of WSO2 DAS which can analyze the raw HTTPD logs and produce useful results. It demonstrates how you can use logs to analyze the Web traffic that comes to your server from different regions. It calculates the region from the IP address in logs and visualizes it through the different mechanisms of presenting data in WSO2 DAS.

Prerequisites

Set up the following prerequisites before you start.

1. Set up the general prerequisites required for WSO2 DAS.

2. Copy the <DAS_HOME>/samples/udfs/org.wso2.das.samples.geoip-3.0.0-SNAPSHOT.jar file (which is the UDF library that is required for this sample), to <DAS_HOME>/repository/components/dropins/ directory.

3. Add the following configurations within the <custom-udf-classes> element of the <DAS_HOME>/repository/conf/spark/spark-udf-config.xml file.

   <udf-configuration>
       <custom-udf-classes> 
   			......... 
   			<class-name>org.wso2.das.samples.geoip.IPCountryNameUDF</class-name>
   			<class-name>org.wso2.das.samples.geoip.IPCountryCodeUDF</class-name>
   			.........
   	</custom-udf-classes>
   </udf-configuration>

Building the sample

Follow the steps below to build the sample.

Uploading the Carbon Application

Follow the steps below to upload the Carbon Application (cApp) file of this sample. For more information, see Carbon Application Deployment for DAS.

1. Log in to the DAS management console using the following URL: https://<DAS_HOST>:<DAS_PORT>/carbon/
2. Click Main, and then click Add in the Carbon Applications menu.
3. Click Choose File, and upload the <DAS_HOME>/samples/capps/Httpd_Log_Analytics.car file as shown below.
   
4.  Click Main, then click Carbon Applications, and then click List view, to see the uploaded Carbon application as shown below. 
   

Executing the sample

Follow the steps below to execute the sample.

Running the data publisher

Navigate to <DAS_HOME>/samples/httpd-logs/ directory in a new CLI tab, and execute the following command to run the data publisher: ant

Executing the Spark script

Follow the steps below to execute the Spark script which is shipped with this sample. For more information on Spark scripts, see Data Analyzing Tools.

1. Log in to the DAS management console if you are not already logged in.

2. Click Main, and then click Scripts in the Batch Analytics menu.
3. Click the Execute option of the script which is already deployed through the CApp as shown below.
   
   You view the result of executing the script as shown below.
   

Viewing the output

You may use the Data Explorer or the Analytics Dashboard of the WSO2 DAS Management Console to browse published sample events.

Using the Data Explorer

Follow the steps below to use the Data Explorer to view the output. 

1. Log in to the DAS management console if you are not already logged in.
2. Click Main, and then click Data Explorer in the Interactive Analytics menu.
3. Select ORG_WSO2_SAMPLE_HTTPD_LOGS for the Table Name as shown below.
   
4. Click Search. You view the published HTTPD logs as shown below.
   

Using the Analytics Dashboard

Follow the steps below to use the Analytics Dashboard to view the output. For more information

1. Log in to the DAS management console if you are not already logged in.
2. Click Main, and then click Analytics Dashboard in the Dashboard menu.
3. Log in to the Analytics Dashboard using admin/admin credentials.
4. You view the HTTPD Log Analysis Dashboard which is already deployed through the CApp as shown below.
   
5. Click View option of the Dashboard. It opens the HTTPD Log Analysis Dashboard with three gadgets in a new tab of your Web browser as shown below.