Netty, HttpAsyncClient and HttpClient Performance Comparison

I was searching for performance comparison of popular asynchronous HTTP clients, namely Netty, Apache's HttpAsyncClient and blocking I/O HttpClient with thread pool. The research is not just for pure comparison, but also to find the sweet spot of each type of the clients. 

https://blogs.atlassian.com/2013/07/http-client-performance-io/

This is an interesting post. Unfortunately, the comparison is based on file downloading, which is not exactly what most web services are built for. So I decided to run my own analysis based on typical client interactions with RESTful services, where the client sends a request and the server replies with a reasonable size of payload.

Clients compared

• Netty 4.0.8 (no connection pool)
• Apache HttpAsyncClient 4.0-beta4 (with connecton pool enabled)
• Apache HttpClient 4.3 (with connection pool enabled)

Test Setup

On the server side, a cluster of servers are set up in Amazon EC2 cloud. They all serve relatively static content where the only variation is size of the payload which is controllable from client's query parameter. Each server runs a Tomcat application with 300 threads.

The client runs on my laptop, which is a Macbook Pro (2.4Ghz Intel Core i7) running JVM 1.6.0_51 with 2G bytes heap size. Different request batch sizes are tested. Each batch sends all requests in a loop and is run several times without JVM shutdown. The test result for one batch size is the average of all runs. The data from first run of a batch is ignored as it usually includes JVM, thread pool and connections warm up.

Metrics

Three metrics are collected or calculated from the test.&nbsp

• Time needed to send all requests. For thread pool based blocking client, a request is deemed as sent when it is dequeued from thread pool and handed off to the client. For Netty, a request is deemed as sent when the ChannelFuture is obtained. For Apache HttpAsyncClient, a request is deemed as sent when the Future<HttpResposne> object is obtained.
• Average latency. For NIO clients, a timer is started when a request is handed off to the client. For HttpClient, the timer is started when a request is enqueued to the thread pool. The timer is stopped when the full HTTP response is received.This time is accumulated for all requests and used to calculate the average.
• Throughput. This is simply the size of batch divided by the time to sent.

Results

 Average latency (ms)

Batch size	100	500	1000	5000
Apache HttpAsyncClient	318	772	1090	5096
Netty	408	1010	1629	5433
HttpClient with 50 threads	227	733	1342	6792
HttpClient with 200 threads	183	477	850	3509
HttpClient with 300 threads	293	502	1010	4125

It is interesting to observe that the best latency is achieved by a blocking I/O client with reasonable size of thread pool. This is perhaps due to the fact that each socket gets a dedicated thread in blocking I/O while in NIO, only a handful of selectors need to poll hundreds of even thousands or sockets. On the other hand, as the size of thread pool increases, the latency gets worse.

The HttpAsyncClient has better latency compared with Netty because it has built-in connection pool support.

Throughput (requests per second)

Batch size	100	500	1000	5000
Apache HttpAsyncClient	2797	4629	5031	19569
Netty	3278	6872	10126	10460
HttpClient with 50 threads	353	404	401	370
HttpClient with 200 threads	5194	705	720	741
HttpClient with 300 threads	5882	715	603	642

Here, the NIO clients have a clear edge over blocking client. They can achieve 10 times better throughput than the blocking HttpClient.

Conclusion

I believe the performance advantage of NIO clients is on the throughput side. If your service requires a client throughput of over 1000 rps, consider using NIO client to avoid bottleneck. On the other hand, if the service only requires a couple hundred rps, using blocking HttpClient with reasonable size of thread pool is manageable and will get you better latency.

HttpAsyncClient enjoys the benefit of a built-in connection pool. This proves to be crucial when the clients have to deal with thousands of concurrent requests. However, note that this is just a comparison of HTTP clients with relatively simple use cases, and Netty clearly have a wider support of protocols.
 

Further discussion and correction

 10/29/2013

After the publishing blog, I had a discussion with Apache HttpClient development lead Oleg Kalnichevski, who pointed to me that the throughput comparison may be flawed in that the sending time for the NIO clients is considerably shorter as it only includes the time to put requests into the processing queue. There is a separate comparison conducted by the HttpClient team that shows the HttpClient outperforms NIO clients:

http://wiki.apache.org/HttpComponents/HttpClient3vsHttpClient4vsHttpCore

Stayed tuned for results of further testing from my side.