Performance Report of Server Side RIA Frameworks

Server-side RIA frameworks have become popular recently. Both ZK and ICEfaces have reached the goal of 1 million downloads. However, regarding server side RIA frameworks, people used to worry about their performance and memory consumption since it determines how many concurrent users can their application serve, and its average response time. It is a pity that few reports can be found nowadays.

Editor's Note (7.7.09): Subsequent to its original publication, ICEsoft engineers found the methodologies and results as presented by ZK to be erroneous and misrepresentative. For additional and accurate information on the performance realized by the ICEfaces framework, mentioned in this article, please refer to ICEfaces Performance Report -- Corrected.

Editor's Note (6.12.09): The author of the folowing performance benchmark, Robbie Cheng, is an engineer at ZK. DZone has not participated in this experiment nor has it validated the results of Robbie's findings. The results are being presented here as means to facilitate further dialogue amongst the end users of these three products as well as the product vendors. The findings and recommendations provided at the end of this article are those of the author's, and are not in any way endorsed by DZone.

To determine the performance of these two frameworks, I arranged a series of tests to record their average response time, and memory consumption. For reference, I also include another famous server side framework, RichFaces into this test. Among these three frameworks, both ICEfaces, RichFaces are JSF based, and ZK is pure Java based. This could be a battle between comfort, and standards. It is interesting to see which one will win out.

Test Methodology

This experiment includes 3 test scenarios, from simple to complex, including a simple form, a grid with 15 records, and a grid with 150 records. Moreover, to determine the scalability of each one, each test case includes 10 scenarios, from 100 to 1000 concurrent users. Each test case will be run for 50 times, and its average response time will be recorded.

• Target
  Static page (ZK), ZK, Icefaces, RichFaces
• Test Cases
  Simple Form, Grid with 15 records, Grid with 150 records

To record required data, I use two popular tools, including JMeter, a load testing tool built specifically for websites, and VisualVM, a tool released by Sun Microsystems for viewing detailed information about Java applications.

Test Environment

The following paragraphs include the hardware specification of this test, required software, and corresponding parameters. I downloaded the latest version of these three frameworks, and used Apache Tomcat as the test platform. The maximum memory of Tomcat is 1GB due to limitation of the computer.

• Hardware
  • CPU: Intel Core 2 Duo P8600 2.4 GHz 2GB
  • Memory:4GB
  • Network Adapter: Intel 82567LM Gigabit Ethernet Connection
• Software
  • ZK 3.5.2
  • ICEfaces 1.7.2
  • Richfaces 3.3
  • JDK 1.6.0_10
  • Apache Tomcat Server 6.0.18
  • Jmeter 2.3.2
  • VisualVM 1.1.1
• Configurations
  Tomcat
  • Session time-out: 30 mins
  • -Xms 256MB
  • -Xmx 1024MB
  • maxThreads: 1000
  • acceptCount: 200
  Jmeter
  
  • Loop per test case:50.
• Source Code
  Source Files can be downloaded from the following url:
  https://zkforge.svn.sourceforge.net/svnroot/zkforge/trunk/zkTest

Test Case1 - Simple Form

• Test Case
  This simple form includes 3 fields, name, password, and comment.



• Response Time
  

  

  ZK:
  • Less than 0.5 second response time when 500 concurrent threads.
  • Less than 1.5 second response time when 1000 concurrent threads.
  ICEfaces:
  • More than 1.5 seconds response time when 500 concurrent threads.
  • Out of memory when 1000 concurrent threads.
  RichFaces:
  • More than 1 seconds response time when 500 concurrent threads.
  • More than 3 seconds response time when 1000 concurrent threads.
• Memory Consumption
  

  

  ZK:
  • 200 MB memory consumption when 500 concurrent threads.
  • 300 MB memory consumption when 1000 concurrent threads.
  ICEfaces:
  • 900 MB memory consumption when 500 concurrent threads.
  • Out of memory when 1000 concurrent threads.
  RichFaces:
  • 200 MB memory consumption when 500 concurrent threads.
  • 300 MB memory consumption when 1000 concurrent threads.
• Analysis
  
  • ZK runs 2 times faster than RichFaces, and 3 times faster than ICEface. The result indicates that JSF based frameworks requires more time while generating page.
  • Memory consumption of ZK and RichFaces is as good as each other, but ICEfaces consumes more than 4 times than the other two. It is a surprise that the memory consumption of JSF based RichFaces is as good as ZK, but ICEfaces is on the contrary.

Test Case2 - Grid with 15 Records

• Test Case
  This test case includes a grid, and a model with 15 records of employees’ information.

• Response Time
  

  

  ZK:
  • 1 seconds response time when 500 concurrent threads.
  • 3 seconds response time when 1000 concurrent threads.
  ICEfaces:
  • 2 seconds response time when 500 concurrent threads.
  • Out of memory when 1000 concurrent threads.
  RichFaces:
  • 2 seconds response time when 500 concurrent threads.
  • 5.5 seconds response time when 1000 concurrent threads.
• Memory Consumption
  

  

  ZK:
  • 200 MB memory consumption when 500 concurrent threads.
  • 400 MB memory consumption when 1000 concurrent threads.
  ICEfaces:
  • 1000 MB memory consumption when 500 concurrent threads.
  • Out of memory when 1000 concurrent threads.
  RichFaces:
  • 300 MB memory consumption when 500 concurrent threads.
  • 600 MB memory consumption when 1000 concurrent threads.
• Analysis
  
  • Again, ZK outscore the other two JSF based ones for the other two. ZK runs 2 times faster than RichFaces, and 3times faster than ICEface.
  • Regarding memory consumption, ZK uses less memory than RichFaces. ICEfaces consumes more than 4 times than the other two. Not sure if ICEfaces uses much cache at the server since its memory consumption reaches 1 GB when there are only 300 concurrent threads.

Test Case3 - Grid with 150 Records

• Test Case
  This test case includes a grid, and a model with 150 records of employees’ information.

• Response Time
  

  

  ZK:
  • 2.7 seconds response time when 500 concurrent threads.
  • 6.4 seconds response time when 1000 concurrent threads.
  ICEfaces:
  • Out of memory when 500 concurrent threads.
  RichFaces:
  • 10 seconds response time when 500 concurrent threads.
  • 28 seconds response time when 1000 concurrent threads.
• Memory Consumption
  

  

  ZK:
  • 350 MB memory consumption when 500 concurrent threads.
  • 600 MB memory consumption when 1000 concurrent threads.
  ICEfaces:
  • Out of memory when 500 concurrent threads.
  RichFaces:
  • 350 MB memory consumption when 500 concurrent threads.
  • 600 MB memory consumption when 1000 concurrent threads.
• Analysis
  
  • The result indicates that ZK handles scalability issue quite well since it runs 4 times faster than RichFaces. But, ICEfaces runs out of memory when 300 concurrent threads.
  • ZK and RichFaces both use memory most efficiently. The memory management of RichFaces is as good as ZK again. And, ICEfaces runs out of memory when there is only 300 concurrent threads.

Summary

In all of above tests, the pure Java framework outscores JSF based frameworks regarding average server response time, and memory consumption. ZK runs 2 to 4 times faster than RichFaces, and at least 3 times faster than ICEfaces. The complex six phases of JSF might be the cause of slow response time since it takes longer time to be initialized. Regarding memory consumption, ZK and RichFaces perform as good as each other; on the contrary, ICEfaces used to get in trouble of out-of-memory issue. It seems that ICEfaces use a lot of caches since it runs out of memory when there are only 300 concurrent threads.

I hope this article provides help for those who attempts to adopt which RIA framework to choose. If you plan to introduce RIA into a green-field project, I suggest pure Java framework, ZK instead of JSF based ones. However, if your project is build based on JSF technology, I recommend RichFaces than ICEfaces since it outscores ICEfaces, especially memory management.