Performance Zone is brought to you in partnership with:

Robbie Cheng is the author of the book "ZK: Ajax without the Javascript Framework". He is a software developer at Potix Corporation, and the principle developer of ZK Android and ZK Forum. Recently, he become a ZK evangelist, and has been invited to give speeches at Google Developer Day and Mobile Monday, as well as published many articles on the Internet. Robbie has posted 19 posts at DZone. View Full User Profile

Performance Report of Server Side RIA Frameworks

06.12.2009
| 20858 views |
  • submit to reddit

Server-side RIA frameworks have become popular recently. Both ZK and ICEfaces have reached the goal of 1 million downloads. However, regarding server side RIA frameworks, people used to worry about their performance and memory consumption since it determines how many concurrent users can their application serve, and its average response time. It is a pity that few reports can be found nowadays.

Editor's Note (7.7.09): Subsequent to its original publication, ICEsoft engineers found the methodologies and results as presented by ZK to be erroneous and misrepresentative. For additional and accurate information on the performance realized by the ICEfaces framework, mentioned in this article, please refer to ICEfaces Performance Report -- Corrected.


Editor's Note (6.12.09): The author of the folowing performance benchmark, Robbie Cheng, is an engineer at ZK. DZone has not participated in this experiment nor has it validated the results of Robbie's findings. The results are being presented here as means to facilitate further dialogue amongst the end users of these three products as well as the product vendors. The findings and recommendations provided at the end of this article are those of the author's, and are not in any way endorsed by DZone.

 

To determine the performance of these two frameworks, I arranged a series of tests to record their average response time, and memory consumption. For reference, I also include another famous server side framework, RichFaces into this test. Among these three frameworks, both ICEfaces, RichFaces are JSF based, and ZK is pure Java based. This could be a battle between comfort, and standards. It is interesting to see which one will win out.

Test Methodology

This experiment includes 3 test scenarios, from simple to complex, including a simple form, a grid with 15 records, and a grid with 150 records. Moreover, to determine the scalability of each one, each test case includes 10 scenarios, from 100 to 1000 concurrent users. Each test case will be run for 50 times, and its average response time will be recorded.

  • Target
    Static page (ZK), ZK, Icefaces, RichFaces
  • Test Cases
    Simple Form, Grid with 15 records, Grid with 150 records

To record required data, I use two popular tools, including JMeter, a load testing tool built specifically for websites, and VisualVM, a tool released by Sun Microsystems for viewing detailed information about Java applications.

Test Environment

The following paragraphs include the hardware specification of this test, required software, and corresponding parameters. I downloaded the latest version of these three frameworks, and used Apache Tomcat as the test platform. The maximum memory of Tomcat is 1GB due to limitation of the computer.

  • Hardware
    • CPU: Intel Core 2 Duo P8600 2.4 GHz 2GB
    • Memory:4GB
    • Network Adapter: Intel 82567LM Gigabit Ethernet Connection
  • Software
    • ZK 3.5.2
    • ICEfaces 1.7.2
    • Richfaces 3.3
    • JDK 1.6.0_10
    • Apache Tomcat Server 6.0.18
    • Jmeter 2.3.2
    • VisualVM 1.1.1
  • Configurations
    Tomcat
    • Session time-out: 30 mins
    • -Xms 256MB
    • -Xmx 1024MB
    • maxThreads: 1000
    • acceptCount: 200
    Jmeter
    • Loop per test case:50.
  • Source Code
    Source Files can be downloaded from the following url:
    https://zkforge.svn.sourceforge.net/svnroot/zkforge/trunk/zkTest

Test Case1 - Simple Form

  • Test Case
    This simple form includes 3 fields, name, password, and comment.
  • Response Time

    ZK:
    • Less than 0.5 second response time when 500 concurrent threads.
    • Less than 1.5 second response time when 1000 concurrent threads.
    ICEfaces:
    • More than 1.5 seconds response time when 500 concurrent threads.
    • Out of memory when 1000 concurrent threads.
    RichFaces:
    • More than 1 seconds response time when 500 concurrent threads.
    • More than 3 seconds response time when 1000 concurrent threads.
  • Memory Consumption

    ZK:
    • 200 MB memory consumption when 500 concurrent threads.
    • 300 MB memory consumption when 1000 concurrent threads.
    ICEfaces:
    • 900 MB memory consumption when 500 concurrent threads.
    • Out of memory when 1000 concurrent threads.
    RichFaces:
    • 200 MB memory consumption when 500 concurrent threads.
    • 300 MB memory consumption when 1000 concurrent threads.
  • Analysis
    • ZK runs 2 times faster than RichFaces, and 3 times faster than ICEface. The result indicates that JSF based frameworks requires more time while generating page.
    • Memory consumption of ZK and RichFaces is as good as each other, but ICEfaces consumes more than 4 times than the other two. It is a surprise that the memory consumption of JSF based RichFaces is as good as ZK, but ICEfaces is on the contrary.

Test Case2 - Grid with 15 Records

  • Test Case
    This test case includes a grid, and a model with 15 records of employees’ information.
  • Response Time

    ZK:
    • 1 seconds response time when 500 concurrent threads.
    • 3 seconds response time when 1000 concurrent threads.
    ICEfaces:
    • 2 seconds response time when 500 concurrent threads.
    • Out of memory when 1000 concurrent threads.
    RichFaces:
    • 2 seconds response time when 500 concurrent threads.
    • 5.5 seconds response time when 1000 concurrent threads.
  • Memory Consumption

    ZK:
    • 200 MB memory consumption when 500 concurrent threads.
    • 400 MB memory consumption when 1000 concurrent threads.
    ICEfaces:
    • 1000 MB memory consumption when 500 concurrent threads.
    • Out of memory when 1000 concurrent threads.
    RichFaces:
    • 300 MB memory consumption when 500 concurrent threads.
    • 600 MB memory consumption when 1000 concurrent threads.
  • Analysis
    • Again, ZK outscore the other two JSF based ones for the other two. ZK runs 2 times faster than RichFaces, and 3times faster than ICEface.
    • Regarding memory consumption, ZK uses less memory than RichFaces. ICEfaces consumes more than 4 times than the other two. Not sure if ICEfaces uses much cache at the server since its memory consumption reaches 1 GB when there are only 300 concurrent threads.

Test Case3 - Grid with 150 Records

  • Test Case
    This test case includes a grid, and a model with 150 records of employees’ information.
  • Response Time

    ZK:
    • 2.7 seconds response time when 500 concurrent threads.
    • 6.4 seconds response time when 1000 concurrent threads.
    ICEfaces:
    • Out of memory when 500 concurrent threads.
    RichFaces:
    • 10 seconds response time when 500 concurrent threads.
    • 28 seconds response time when 1000 concurrent threads.
  • Memory Consumption

    ZK:
    • 350 MB memory consumption when 500 concurrent threads.
    • 600 MB memory consumption when 1000 concurrent threads.
    ICEfaces:
    • Out of memory when 500 concurrent threads.
    RichFaces:
    • 350 MB memory consumption when 500 concurrent threads.
    • 600 MB memory consumption when 1000 concurrent threads.
  • Analysis
    • The result indicates that ZK handles scalability issue quite well since it runs 4 times faster than RichFaces. But, ICEfaces runs out of memory when 300 concurrent threads.
    • ZK and RichFaces both use memory most efficiently. The memory management of RichFaces is as good as ZK again. And, ICEfaces runs out of memory when there is only 300 concurrent threads.

Summary

In all of above tests, the pure Java framework outscores JSF based frameworks regarding average server response time, and memory consumption. ZK runs 2 to 4 times faster than RichFaces, and at least 3 times faster than ICEfaces. The complex six phases of JSF might be the cause of slow response time since it takes longer time to be initialized. Regarding memory consumption, ZK and RichFaces perform as good as each other; on the contrary, ICEfaces used to get in trouble of out-of-memory issue. It seems that ICEfaces use a lot of caches since it runs out of memory when there are only 300 concurrent threads.

I hope this article provides help for those who attempts to adopt which RIA framework to choose. If you plan to introduce RIA into a green-field project, I suggest pure Java framework, ZK instead of JSF based ones. However, if your project is build based on JSF technology, I recommend RichFaces than ICEfaces since it outscores ICEfaces, especially memory management.

Published at DZone with permission of its author, Robbie Cheng.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

michelle211 cannon replied on Fri, 2009/06/12 - 11:00am

And, ICEfaces runs out of memory when there is only 300 concurrent threads.
We have a lot more than 300 concurrent threads. We do a massive amount of processing.
Or maybe your testing is flawed.
At the company I work for we have a major spring/icefaces application , it has over 100 screens, we've used automated testing and tested with over 5000 users and on the transaction side over 150,000 transactions via jms Performance was more than acceptable. Usually less than a second on screen response time, if we saw any hangups at all it was with the jpa spring hybernate templates, The app hits multiple databases and we definitely saw performance hits especially against microsoft sql server. We have better luck with the oracle side.
One reasons your test data may be skewed is that long polling ajax apps are hard to profile, we use the excellent testing tool DevPartner to find our bottlenecks and fix them.
It may be true that some javascript pure implemetations are faster, less abstraction but maintainance and code portability amoung large development teams is far easier to handle with the serverside platforms.
In additon to work , I am involved in serveral open source endevours that use java/icefaces as backend servers to mobile applications.
www.mooncatventures.com/blogs,
again performance is excellent. Even when loading and keeping large amounts of multimedia data in shared memory scopes.
Icefaces performance is very affected by how you configure it , what application server you use and wether or not you use an additional server to take up some of the workload. icefaces provides a push server that handles all the ajax calls and this improves performance a lot, so does the push capabilities of glashfish, tomcats NIO (but it can be buggy) and jetty continuations.

michelle211 cannon replied on Fri, 2009/06/12 - 11:06am

If your running a profiler along with icefaces you want to make sure you have your JVM configured with enough memory. I suggest -Xms256m -Xmx1g Works well with icefaces , especially if you also are using jms

michelle211 cannon replied on Fri, 2009/06/12 - 11:10am

I'm sorry I don't want to seem pesty. But I just can't see were you get your results. I've been doing profiling on our app periodically and your data makes no sense., if your using a very large amount of memory, are you keeping a lot of stuff in session scope that should be in request scoped beans.

Paul Toker replied on Fri, 2009/06/12 - 4:58pm

Here we go again with ZK's self-promoting propaganda. These guys are notorious for doing this. Shame on you Dzone for publishing.

michelle211 cannon replied on Fri, 2009/06/12 - 6:44pm

Well for direct push we use icefaces and never had a problem. I hear dojotools are pretty good too and andromeda. Ajaxian did a nice little piece on both icefaces and andromeda a little while back. The impression was generally good, and they sited ease of use as being important features. icefaces also provides performance and monitoring tools as a paid addon.

James Jamesson replied on Sat, 2009/06/13 - 12:40am

I guess ZK propaganda department was unsuccessfull on hitting at GWT. If I hit 100 times to the download button at sourceforge everyday. Will that count?

Robbie Cheng replied on Sat, 2009/06/13 - 3:12am in response to: michelle211 cannon

hi michelle, would u like to download the test code, and give it a try? the number is not made up. and welcome to share your experience.

Ted Goddard replied on Sat, 2009/06/13 - 9:06am

The testing methodology used in the article does not accurately recreate browser interaction (load testing is not easy to perform, so this a typical problem); we will provide a detailed response shortly.

Joonas Lehtinen replied on Sun, 2009/06/14 - 2:12pm

Ted and Robbie, if you are doing more comprehensive benchmarks, why not include Vaadin?

michelle211 cannon replied on Mon, 2009/06/15 - 9:02am

I see no point in downloading your test code. As they say the proof is in the pudding. And we have pleanty of real world examples that disprove your claim. I will see if I can find some examples I can post which do not reveal propriotary company information. I will make this general comment, if icefaces could only handle the number of concurrent threads you claim, users would be complaining about that. There is very little comment on that on the icefaces boards. Also I think I would be out of a job since I suggested icefaces to my company :-) Similar claims were make for ZK on the ajaxian forum, and there was an excellent response from icefaces. http://blog.icefaces.org/blojsom/blog/default/2009/03/17/Ajax-Push-ZK-versus-ICEfaces-More-than-Meets-the-Eye/ No one is accussing anyone of deception, load testing is more of an Art than a science, and it can be extremely difficult to get adequate results . We did most of our testing at my company with Compuware devpartner for Java, which is a profiler which attempts to show you the bottlenecks in your code. Sometimes the simplest change in code can make a major performace difference.

michelle211 cannon replied on Mon, 2009/06/15 - 9:05am in response to: michelle211 cannon

Another item I find a bit disquieting about this article, is that the Author is a heavy promoter of ZK if you check the profile you will see that most articles he's written are of how "great" zk is. This is hardly an unbiased article.

michelle211 cannon replied on Mon, 2009/06/15 - 9:38am

I decided to download your code anyway. find it interesting how you really optimized you zk performance, but "went out of your way" to not optimize performance with icefaces. Of course it could be you just didn't read the icefaces documentation thoughly and just cut and pasted from component showcase samples. Not in any particular context. For starters in your class that loads the recordsets from the backing beans for your table examples, I would make those beans request scoped , not session scope. But that is only one problem I see looking at the code for a few minutes. I'm sure if I analyse it further I will find a lot more issues. Interesting how in the icefaces sample you copied rowselectorcontroller1 and 2 are request scoped but you changed them to session scoped.

Jose Maria Arranz replied on Mon, 2009/06/15 - 11:43am

Why are so different these files?

https://zkforge.svn.sourceforge.net/svnroot/zkforge/trunk/zkTest/test%20cases/form/icefaces-form-all.jmx

https://zkforge.svn.sourceforge.net/svnroot/zkforge/trunk/zkTest/test%20cases/form/zk-form-all.jmx

For instance:

<stringProp name="ThreadGroup.duration"></stringProp>

vs

<stringProp name="ThreadGroup.duration">60</stringProp>
<stringProp name="ThreadGroup.duration">100000</stringProp> 

vs

<stringProp name="ThreadGroup.duration">10000</stringProp>

 

And so on.

 

Aguiar replied on Mon, 2009/06/15 - 12:40pm

Some suggestions:

 

  • You seem to use heavyweight richfaces everywhere, when it's not needed. For instance, you use rich:dataTable where you could simply use the jsf traditional dataTable.
  • Why richfaces Grid page has a dataScroller (for pagination) while others don't?
  • Why do you use JBoss Seam? It surely causes an overhead
  • Change your web.xml configuration for production, ok?

For me it's clear that this benchmark is biased and I stopped looking into the source code...

michelle211 cannon replied on Mon, 2009/06/15 - 1:08pm

same conclusion. But it doesn't surprise me. By same author, from his profile. kind of zk heavy? We briefly looked at richfaces too, but decided on icefaces for a number of reasons, but also because of our intent to use spring rather than Seam. Type Post Author Replies Last updated Article Performance Report of Server Side RIA Frameworks robbiecheng 14 26 min 11 sec ago Announcement Intuive MVC Pattern for Ajax Applications robbiecheng 0 6 weeks 5 days ago Announcement ZK Studio 0.9.3: CRUD Application Builder robbiecheng 0 10 weeks 6 days ago Announcement ZK 3.5 Supports Comet Server Push! robbiecheng 3 12 weeks 5 days ago Announcement ZK 3.6 Supports Java Applets robbiecheng 0 14 weeks 2 days ago Announcement ZK Spring Integration Library 1.0.0 robbiecheng 3 20 weeks 6 hours ago Announcement ZK Studio 0.9.2 Style Designer Available robbiecheng 0 21 weeks 3 days ago Announcement ZK 3.5.2 & Brand New ZK Demo robbiecheng 0 26 weeks 3 days ago Announcement Case Study: Experience of Building AJAX with ZK robbiecheng 2 27 weeks 1 day ago Announcement Integrating ZK and Spring Web Flow robbiecheng 0 27 weeks 5 days ago Announcement ZK Studio 0.9 Enables WYSIWYG Ajax Development robbiecheng 0 28 weeks 6 days ago Article Exploring the Deadly Sins of Rich Internet Applications robbiecheng 4 31 weeks 1 hour ago Announcement ZK Timeplot (timeplotz) 1.0_2 released robbiecheng 0 32 weeks 5 days ago Announcement Webinar: An Introduction of ZK Ajax Framework. robbiecheng 0 33 weeks 1 day ago Story NetBeans 6.1 Plug-in for ZK robbiecheng 0 34 weeks 5 days ago Announcement ZK 3.5.1: 6 New Features and 28 Bugs Fixed robbiecheng 0 35 weeks 4 days ago Announcement ZK Studio 0.8.3 is released robbiecheng 2 35 weeks 5 days ago Announcement ZK Studio 0.8.2 Supports Ganymede robbiecheng 0 38 weeks 5 days ago

Ted Goddard replied on Mon, 2009/06/15 - 5:44pm

The load tests described in the article do not recreate actual browser interaction. The author's intent is to simulate 50 events for each of 1000 users, but due to the simplicity of the JMeter test case (50 identical GET requests for each virtual user), something very unusual is being simulated and the different frameworks react in different ways.

For instance, in real-world browser interaction, the user would not normally open the same page simultaneously in 50 browser windows (corresponding to 50 identical GET requests); but they might interact with the application 50 times via form submission or they might refresh the window 50 times. ICEfaces distinguishes between single and multiple window push applications, so the different cases are important: in actual browser interaction, the page reload case is handled through a view-disposal message sent from window.onunload(). Of course, the JMeter test case does not take this into account (it does not realistically simulate a browser) and is not testing actual load conditions.

Implementing the test case correctly requires some expertise with JMeter (using regular expressions to correctly propagate the session identifiers and the window identifiers), so rather than go into detail here, the simplest thing is to configure ICEfaces for single-window operation (appropriate, given the fact that the JMeter test is not designed as an Ajax test) by setting "com.icesoft.faces.concurrentDOMViews" to "false" in a web.xml context parameter.

With this configuration, you will find that 1000 "users" run without difficulty in a 400MB heap and respond with a latency of less than 3000 milliseconds, which is the same, or better than the other profiled frameworks.

The conclusion we must draw is that the performance difference is negligible (when tested correctly), so now you should ask yourself why you would use anything but an Ajax framework based on the JavaServer Faces standard.

Robbie Cheng replied on Tue, 2009/06/16 - 1:07am in response to: michelle211 cannon

yes, i did some optimization, setting the scope of back-end bean to application which mean all users share the same instance. u didn't read the configuration file carefully, do you? about the scope of controller, it's usual to keep user's state in controller while doing ajax-effect, thus, i changed it into session scope.

Robbie Cheng replied on Tue, 2009/06/16 - 1:19am in response to: Ted Goddard

but it's a usual case that users open multiple browsers(windows) to visit the same application, and then we have to disable the "com.icesoft.faces.concurrentDOMViews" feature to save memory consumption, do we? But if this feature is disabled, users might encounter unexpected error if they open multiple browsers to visit the same application(url), is it?

zodix Moore replied on Tue, 2009/06/16 - 3:01am

Bechmarking is never fair, and usually misleading. After all, applications are so diverse that no single benchmarking can cover them all. However, it does help us to understand a framework more by leaning the test cases and results.

@michelle211 and Ted: Instead of bashing the credits of the benchmarking, why don't you publish your own data and test cases.  It will benefit all of us.

 Just my two cents.

zodix Moore replied on Tue, 2009/06/16 - 3:14am in response to: Ted Goddard

Yes, it reduced the memory use a lot if "com.icesoft.faces.concurrentDOMViews" is set to false. But, does it mean an user can not have two different views of the same URL? If he changed the view (via Ajax) in one browser window, what will happen to other browser windows?

michelle211 cannon replied on Tue, 2009/06/16 - 8:21am in response to: zodix Moore

The whole article is bashing and basically worthless. The arthor is a major part of the ZK team, perhaps the arthur. These folks have been publishing all over the place the benifits of ZK over pretty much every other technology. I normal don't "bash" as you say, and try to stay objective. But its hard to stay neutral when you see this kind of action. Both richfaces and icefaces are very thoughout , very well designed implementations. With very big user bases. I just don't understand this kind of action by the ZK people.

michelle211 cannon replied on Tue, 2009/06/16 - 8:30am in response to: zodix Moore

In a normal application concurretDomViews would be ture, Ted Goddard I believe was just pointing out the issue with jmeter profileing. Again we use A more complex tool called devpartner, which does not force us to make this change. Opening 50 views of the same page, on the same Device , in relatively short interval is not what I would call normal user behavoir. I would suggest you go back, redo your test with the suggestions here. Keeping beans at request scope were appropriate making the suggested web.xml changes keeping a consistency to the ui design (comments made by richfaces user) Also I think ZK to adress its server push deficencies, which the icefaces blog article attests to. You say be objective on your test cases, but how about those????

Ted Goddard replied on Tue, 2009/06/16 - 10:41am in response to: Robbie Cheng

Multiple, perhaps, but 50 simultaneous browser windows for each of 1000 users is a very contrived test. Setting concurrentDOMViews to false is intended for single window applications. Multiple simultaneous windows in this mode will not result in errors per se, but can lead to the same sort of inconsistency found in applications that use Session scope excessively. It's a reasonable tradeoff given the limitations of the load test described in the article.

Ted Goddard replied on Tue, 2009/06/16 - 11:00am in response to: zodix Moore

Yes, with concurrentDOMViews set to false, the intent is that the user does not open multiple windows to the same URL. The benefit is that pressing reload retains the current view. This mode is intended for kiosk-type applications (or misconfigured load tests). The behavior with multiple windows is similar to JSF 1.1: a single component tree is maintained on the server, and form posts from different browser windows are all applied to that single component tree. In some cases, the state can diverge between the two windows (particularly if components are added or removed) so concurrentDOMViews set to "true" is the recommended default. To load test in this mode, however, requires that window closing events are accurately represented, and this requires additional configuration in JMeter (or the use of a commercial-strength load tester, such as NeoLoad). I agree, we should take the time to publish corrected ICEfaces numbers (I'm not sure if the RichFaces test was valid or not, but it should be checked as well).

Stephen Maryka replied on Tue, 2009/07/07 - 11:58am in response to: Ted Goddard

As Ted suggested we would, I have provided a follow-up article with corrected results for ICEfaces here

Steve

prabhat jha replied on Wed, 2009/07/22 - 12:33pm

I just did a quick test to verify the validity of following for Test Application 1:

RichFaces:

  • More than 1 seconds response time when 500 concurrent threads.
  • More than 3 seconds response time when 1000 concurrent threads.
I got an average response time of < 100ms for 1000 concurrent threads. I did no JVM or Tomcat 6 tuning other than what's mentioned in this article.

There could be difference between hardware and network speed but even then it's hard to fathom how there could be a difference of 2.9 seconds on a reasonable server and network setup. I will post the details in another blog at http://sensiblerationalization.blogspot.com/

 

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.