Analysing HTTPD Logs

In order to effectively manage a web server, it is necessary to get feedback about the activity and performance of the server as well as any problems that may be occurring. The HTTPD Logs can provide everything that happens on your server, from the initial request, through the URL mapping process, to the final resolution of the connection, including any errors that may have occurred in the process.  

Every time your server receives a request, it makes an entry to one or more log files. These log files are useful for a variety of purposes, from statistical analysis of your visitors to forensic analysis of an attack on your server. The HTTPD Logs sample is intended to show the capability of WSO2 BAM which can analyze the raw httpd logs and produce useful result. This sample demonstrate how you can use your logs to analyse the web traffic come to your server from different regions. It calculate the region from the ip address in logs and visualize it in gadgets.

Sample guide 

1. Start the WSO2 BAM Server.

2. Go to $WSO2_BAM_HOME/samples/httpd-logs directory via console

3. Type 'ant' from the console (This will read the access.log from the $WSO2_BAM_HOME/samples/httpd-logs/resources Directory and send each log line as event) 

Note that you can give custom values for parameters for Data receiver URL, user name and password. Default values of them are as follows.

Now lets see how new values can be given as parameters

Command with new values : ant -Durl=tcp://localhost:7612 -Dusername=user -Dpassword=pass123

Part of the sample log file:

1.202.218.8 - - [20/Jun/2012:19:05:12 +0200] "GET /robots.txt HTTP/1.0" 404 492 "-" "\"Mozilla/5.0"
208.115.113.91 - - [20/Jun/2012:19:20:16 +0200] "GET /logs/?C=M;O=D HTTP/1.1" 200 1278 "-" "Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@gmail.com)"
123.125.71.20 - - [20/Jun/2012:19:30:40 +0200] "GET / HTTP/1.1" 200 912 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
220.181.108.101 - - [20/Jun/2012:19:31:01 +0200] "GET / HTTP/1.1" 200 912 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
123.125.68.79 - - [20/Jun/2012:19:53:24 +0200] "GET / HTTP/1.1" 200 625 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
178.154.210.252 - - [20/Jun/2012:19:54:10 +0200] "GET /?C=S;O=A HTTP/1.1" 200 663 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
74.125.126.102 - - [20/Jun/2012:20:15:28 +0200] "GET / HTTP/1.1" 200 606 "http://www.google.com/url?sa=t&rct=j&q=error&source=web&cd=1&ved=0CFAQFjAG&url=http%3A%2F%2Fwww.isrolab.com%2F&ei=GxPiT5PsL4e04AHPtgE&usg=AFQjCNHnmjmdkUV584ORIpOXz7zAPX0UHQ" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; InfoPath.2; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)"
74.125.126.103 - - [20/Jun/2012:20:15:29 +0200] "GET /icons/blank.gif HTTP/1.1" 200 383 "http://www.isrolab.com/" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; InfoPath.2; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)"
74.125.126.93 - - [20/Jun/2012:20:15:29 +0200] "GET /icons/folder.gif HTTP/1.1" 200 460 "http://www.isrolab.com/" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; InfoPath.2; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)"

{
'name':'org.wso2.sample.httpd.logs', 
'version':'1.0.0',
'nickName': 'Httpd_Log_Stream', 
'description': 'Sample of Httpd logs analysis', 
'metaData':[ 
    {'name':'clientType', 
    'type':'STRING'
    }], 
'payloadData':[ 
    {'name':'log',
    'type':'STRING'
    }] 
}

A sample event

Event{
  streamId='org.wso2.sample.httpd.logs-1.0.0-fdce9ab5-a95e-4d61-93ed-2de3372d75a2', 
  timeStamp=1343835321955, 
  metaData=[external], 
  correlationData=null,
  payloadData=[1.202.218.8 - - [20/Jun/2012:19:05:12 +0200] "GET /robots.txt HTTP/1.0" 404 492 "-" "\"Mozilla/5.0"]
}

4. Go to Management console of WSO2 BAM. 

5. Go to Main -> BAM ToolBox -> Add. Select "HTTPD Logs Analysis Toolbox" and click on Install button.

6. Go to Main -> BAM Toolbox -> List. Wait until the 'HTTPD Logs Analysis Toolbox' toolbox status changes to Installed.

7. Wait for some time, until script complete the first run after publishing the data. (Script will run in each minute).

If hive job is successful you will see the following message on BAM console. 

Ended Job = job_local_0001
Execution completed successfully
Mapred Local Task Succeeded . Convert the Join into MapJoin
OK

8.  Click on Main -> Gadgets -> View portal to see the populated gadgets which shows a summary of no of requests coming from different countries.