Page Comparison

The following  The following sections analyze the results of WSO2 API Manager performance tests done in the Amazon EC2 environment.

Deployment

Image Removed

The following are the details with regard to the WSO2 API-M 2.1.0 deployment that is depicted above.

• Four (4) EC2 t2.xlarge instances are used for deployment as shown above. Each instance has 4 CPUs and 16GB of memory. 
• The operating system is Ubuntu 16.04.2 LTS. 
• Apache JMeter version is 3.2. is used in this deployment.
  Apache JMeter is the preferred load testing tool at WSO2. As there are a high number of concurrent users, we have increased the number of sockets supported in the server.

  • The following commands were used in the instance used by JMeter.

    Code Block
    sudo sysctl -w net.ipv4.ip_local_port_range="1025 65535" sudo sysctl -w net.ipv4.tcp_tw_reuse=1

  • The first command increases the port range used for the client.

  • The second command allows to reuse the sockets in the TIME_WAIT state. These are safe commands that can be used in the client side.
    For more information, go to  https://vincent.bernat.im/en/blog/2014-tcp-time-wait-state-linux

• The ulimit was increased in all servers.
  For more information, see Tuning Performance.

Backend Service

The backend service used for testing was developed using Netty. Since there can be up to 3000 users, we used 3000 threads for Netty in order to avoid any bottlenecks in the backend. The Netty server also has a parameter that can be used to simulate the delays in the backend service by simply specifying the sleep time in seconds.

Test Scenario

The test scenario focuses on performing an API proxy invocation using API Manager that in turn will echo the API. Tests were done using 100, 200, 300, 1000, 2000, and 3000 concurrent users.

Measuring Performance

Two key performance metrics are used to measure the performance, namely Latency and Throughput. Throughput measures the number of messages that a server processes during a specific time interval (e.g., per second). The throughput is calculated using the following equation.

Latency measures the end-to-end processing time for an operation. Every operation has its own latency. Therefore, we are interested in how latency behaves. In order to see this behavior, we must have the complete distribution of latencies.

Performance Testing Tool

In Apache JMeter we specified the number of concurrent users, ran the test, and got the results specified under the Performance Test Results section. The following section provides details of the for the terminology used in the Performance Test Results section.

...

Summary

The performance of WSO2 API Manager was measured using the following APIs, which invoke a simple “Netty HTTP Echo Service”. As the name suggests, the Netty service echoes back any request posted to the service.

1. Echo API: This is a secured API, which directly invokes the back-end service.

2. Mediation API: This is also a secured API, which has a “sequence” as a mediation extension to modify the message.

Tests were done using 100, 200, 300, 1000, and 2000 concurrent users. Concurrent Users mean that there are multiple users accessing the API Gateway at the same time. Different Message Sizes (Payload) were used for the tests with different back-end service delays. The message sizes used are 50B, 1KiB, 10KiB, and 100KiB. The back-end delays were 0ms, 30ms, 500ms, and 1s.

Two key performance metrics were used to measure the performance of each test.

1. Throughput: This measures the number of API invocations that the API Manager Gateway server processed during a specific time interval (e.g. per second).

2. Response Time: This measures end-to-end processing time for an operation (of invoking an API using HTTPS protocol). The complete distribution of response times were recorded.

The heap size of WSO2 API Manager was increased to 4GB from 2GB, which is the default. Except for increasing the heap size of API Manager, there were no other specific configurations used to optimise the performance of WSO2 API Manager.

With WSO2 API Manager, an average user use ~1KiB messages and most of the back-ends usually responds in ~30ms. Therefore, let’s look at some charts to understand performance test results for above APIs when using 1KiB messages with 30ms backend delay.

The deployment used was All-in-one API Manager.

The following figure shows how the Throughput changes for different number of concurrent users.

Image Added

Key observations:

• More concurrent users mean more requests to the API Manager Gateway. Therefore, the Throughput of the API Manager Gateway increases as the number of concurrent users accessing the APIs increases. The maximum throughput is obtained for 1000 concurrent users for both “Echo API” and “Mediation API” and the throughput degrades after 1000 concurrent users due to resource contentions in the system. The degradation point mainly depends on the hardware resources.

• Echo API throughput is much better than the Mediation API. Main reason is that the Mediation API has a mediation extension, which uses a “Payload Factory” Mediator. This mediation in the sequence does a JSON to JSON message transformation. That means, the Mediation API reads the message (payload) to do the message transformation and it has a considerable overhead than the “Echo API”, which is similar to a “Direct Proxy”. A “Direct Proxy” does not perform any processing on the messages that pass through it.

The following figure shows how the Average Response Time changes  for different number of concurrent users.

Image Added

Key observations:

• The Average Response Time increases with the number of concurrent users. Since the number of requests to serve increases with more users, there are more resource contentions. Therefore, the number of concurrent users served by the API Gateway needs to be decided on the required response time limits. For example, in order to keep average response time below 100ms for APIs with 30ms backend delay, the maximum concurrent users accessing the API Gateway should be limited to 300.

• The Mediation API response times are higher than Echo API due to the performance overhead of mediation extension.

Let’s look at the 90th, 95th, and 99th Response Time percentiles. This is useful to measure the percentage of requests exceeded the response time value for a given percentile. A percentile can also tell the percentage of requests completed below the particular response time value.

For example, when there are 100 concurrent users, 90th response time percentile for Echo API is 36ms. This means that 10% of the requests have taken more than 36ms to respond. Similarly, 99th response time percentile for Echo API is 146ms, which means that 99% of the requests have completed within 146ms.

Image Added

Key observations:

• Mediation API is slower than the Echo API due to the performance overhead of mediation extension.

• Response Times percentiles are less than 300ms up to 300 concurrent users.

• For higher concurrent users, 99th percentile of Mediation API response times goes beyond 1 second, which means that 1% of the API invocations took 1 second to 1.6 seconds.

In order to see the memory usage, the Garbage Collection (GC) logs in the API Manager was enabled and the GC log for each performance test was analyzed using the GCViewer.

The GC Throughput was calculated for each test to check whether GC operations are not impacting the performance of the server. The GC Throughput is the time percentage of the application, which was not busy with GC operations. For example, if the application ran for 10 minutes and 30 seconds were taken for GC operations, the GC Throughput is 1-301060100=95%. A GC Throughput over 90% is good and that means the allocated heap was enough to handle all concurrent requests, which allocate objects in the memory.

The following chart shows the GC Throughput (%) for different number of concurrent users.

Image Added

Key observations:

• GC Throughput decreases when the number of concurrent users increase. This means that the time spent for GC is increasing with concurrent users.

• GC Throughput for Mediation API is better than Echo API. The Mediation API processes requests slower than the Echo API, which means that the object allocation rate is also less than the Echo API.

WSO2 API Manager All-in-one Deployment

Anchor
allinone allinone

In this deployment, the WSO2 API Manager is deployed in one EC2 instance and a RDS instance is used for databases. The back-end service, which is a simple Netty HTTP Echo Service is deployed in a separate EC2 instance. There are two JMeter servers with a JMeter client in three EC2 instances.

See: JMeter Remote Test. Two JMeter servers are used to simulate high number of concurrent users.

Image Added

See the following links for more details on Amazon Instance Types

• https://aws.amazon.com/ec2/instance-types/
• http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.DBInstanceClass.html

The operating system is Ubuntu 16.04.2 LTS.

WSO2 API Manager version is 2.1.0 and Apache JMeter version is 3.2.

MySQL version in RDS instance was 5.7

Observations from all results

Anchor
observations observations

There are key observations for the average user scenario of accessing APIs with 1KiB messages and the back-end service having 30ms delay.

The following are the key observations from the all performance tests done with different message sizes and different backend delays. (See Comparison of results for all charts used to derive the pointed mentioned below)

Key observations related to throughput:

• Throughput increases up to a certain limit when the number of concurrent users increase. Mediation API throughput increase rate is much lower than the Echo API.

• Throughput decreases when the message sizes increase. The Mediation API throughput decrease rate is much higher than the Echo API.

• Throughput decreases when the backend sleep time increase. This observation is similar to both APIs. This means that if the backend takes more time, the request processing rate at the API Manager gateway will be less.

Key observations related to response time:

• Average response time increases when the number of concurrent users increase. The increasing rate of average response time for Mediation API is much higher than the Echo API.

• Average response time increases considerably for Mediation API when the message sizes increase due to the message processing. The average response time of Echo API is not increasing as much as the Mediation API.

• Average Response Time increases when the backend sleep time increases. This observation is similar to both APIs.

Key observations related to GC Throughput:

• The GC throughput decreases when the number of concurrent users increase. When there are more concurrent users, the object allocation rate increases.

• The GC throughput increases when the message sizes increases. The request processing rate slows down due to the time taken to process large messages. Therefore, the object allocation rate decreases  when the message sizes increases.

• The GC throughput increases when the backend sleep time increases. The object allocation rate will be low when the backend takes more time to respond.

Backend Service

The backend service used for tests were developed using Netty. Since there can be up to 2000 users, 2000 threads were used for Netty (to avoid the back-end becoming a bottleneck)

The Netty server also has a parameter to specify the number of seconds to sleep to simulate the delays in the backend service)

Performance Testing Tool

As mentioned above, Apache JMeter was used to run load tests. In JMeter, the number of concurrent users was specified and the following details were taken after each test.

• # Samples - The number of requests sent with the given number of concurrent users.

• Error Count - How many request errors were recorded.

• Error

%

• % - Percent of requests with errors

Average

• Average - The average response time of a set of results

Min

• Min - The shortest time taken for a request

Max

• Max - The longest time taken for a request

• 90th

Percentile

• Percentile - 90% of the requests took no more than this time. The remaining samples took at least as long as this

.

• 95th

Percentile

• Percentile - 95% of the requests took no more than this time. The remaining samples took at least as long as this

.

• 99th

Percentile

• Percentile - 99% of the requests took no more than this time. The remaining samples took at least as long as this

.
Throughput

• Throughput - The

throughput

• Throughput is measured in requests per second.

• Received KB/

sec

• sec - The throughput

is

• measured in received Kilobytes per second

.

• Sent KB/

sec

• sec - The throughput

is

• measured in sent Kilobytes per second

.

...

...

In

...

...

addition, to

...

above details,

...

some additional details were recorded for every test

...

.

...

• GC Throughput - Time percentage the application was not busy with

...

• GC

...

...

GC throughput and other GC related details were obtained from the GC logs produced by the WSO2 API Manager.

The following are the GC flags

...

used

...

:

...

-XX:+PrintGC

...

-XX:+PrintGCDetails

...

-XX:+PrintGCDateStamps

...

-Xloggc:"$CARBON_HOME/repository/logs/gc.log

...

...

...

Performance

...

The heap of WSO2 API Manager was increased to 4GB and that is the only change done for WSO2 API Manager, which had an impact on the performance.

When doing performance tests, the heap memory was initially set at 2GB as this is the recommended heap memory for WSo2 API Manager. However, when the tests were done with large numbers of concurrent users using 2GB heap, the GC throughput observed was less than 90%. We recommend that the GC throughput to be maintained above 90% and therefore we increased the heap memory to 4GB and ran a similar set of tests.

We also increased the socket timeouts in WSO2 API Manager. However, it was not needed as the performance results of WSO2 API Manager never reported latencies greater than 60 seconds, which is the default socket timeout.

The number of worker threads used were 400 (default value) and the number of threads may increase up to 500 (default value).

Performance Test Results

Scenario 1: Echo API in WSO2 API Manager

There is no delay in the backend service. This test was done mainly to see the maximum throughput of the WSO2 API Manager in the EC2 environment.
Image Removed

Scenario 2: Echo API in WSO2 API Manager and 1 second Think-Time

The test mentioned in Scenario 1 was repeated with a 1 second think-time. This means that there is a one second gap in between the requests sent by a user (in JMeter). In a realistic scenario, there will always be some think-time in between requests. This test was done to understand the performance of WSO2 API Manager when there is a think time.

There is no delay in the backend service.

Image Removed

Comparison - Echo API with no think-time vs. 1sec think-time

Image Removed

Throughput with 1 second think-time is much less than the throughput without the think time. Adding a think-time reduces the arrival rate of the requests.

Image Removed

Latencies of requests are better as the arrival rate of requests is low.

Image Removed

Load average is also much better due to the low arrival rate of requests.

Conclusion

We analyzed the performance of WSO2 API Manager using 100, 200, 300, 1k, 2k, and 3k threads using 1, 5, and 30 second backend delay.

Except for increasing the heap size of API Manager, there were no other specific optimization techniques (i.e., performance tuning) used to optimize the performance of WSO2 API Manager. We only increased the heap size to 4GB from 2GB, which is the default.

...

Test Scripts

All scripts used to run the performance tests and analyze results are in the following repositories.

• https://github.com/wso2/performance-common/
• https://github.com/wso2/performance-apim/

Comparison of results

Anchor
comparison comparison

Table of Content Zone
minLevel 3 location top
Throughput Comparison Image Added The Mediation API was not tested with 100KiB message size. The Echo API has some errors with 100KiB message size for 1000 and 2000 concurrent users. The following charts show what happens to the server throughput when considering all results. Throughput (Requests/sec) vs Concurrent Users Image Added Throughput (Requests/sec) vs Message Size (Bytes) Image Added Throughput (Requests/sec) vs Sleep Time (ms) Image Added Average Response Time Comparison Image Added The following charts show what happens to the average response time when considering all results. Average Response Time (ms) vs Concurrent Users Image Added Average Response Time (ms) vs Message Size (Bytes) Image Added Average Response Time (ms) vs Sleep Time (ms) Image Added GC Throughput Comparison The following chart shows the GC throughput behavior when considering all results. Image Added API Manager GC Throughput (%) vs Concurrent Users Image Added API Manager GC Throughput (%) vs Message Size (Bytes) Image Added API Manager GC Throughput (%) vs Sleep Time (ms) Image Added Refer Observations from all results for more details on the charts.