Performance Issues of PLoS Websites

How about offering direct links from TOC to frequently downloaded links?

This would potentially reduce load on your servers and be a convenience for users.

For my purposes these links would be those to pdf and to the citation but an analysis of your download patterns would tell you if this is a high-traffic route or not.

Of course it may be the case that most people just read articles in html in which case your current links are sufficient.

Bill

We’ve had a good day of investigation and will apply patches to the production servers tomorrow. We now have a third pubApp instance behind our firewall that will share its cache with the other pubApps. This will help performance significantly as the pubApp cache will no longer need to be re-filled on restart (usually takes a few hours).

We’ve uncovered other issues today and will continue digging into the belly of the beast. I expect to have more patches to increase stability/performance in place shortly. Search will be down for ~week while we sort through higher priorities.

We had a “cascade event” ~5:45am which backed up Mulgara. This event happens when a query pulls so much data from Mulgara that all other queries are waiting for it to finish. This happens infrequently and depending on site traffic, Mulgara is able to gracefully handle the back log of requests. But this morning, Mulgara was unable to clear out the queue. This query is hard to track down (the saying “needle in a haystack” is appropriate) and we’ve upped the logging levels to trace all queries sent to Mulgara.

Unfortunately, we also had a human error this morning. When I restarted the applications after the cascade event, a new article XSL file caused articles to throw errors trying to parse the article XML. Articles were serving “page not found” which resulted in more cascade events until I could revert the XSL file. We tested the XSL file on dev/stage without an error so something else must have snuck into the release.

Now that the XSL file is fixed, the sites are purring. We’ve also identified another cache fix that will significantly help performance and a patch should be up today.

We’re making more progress on a couple of fronts. Russ has been able to demonstrate that memory is being gobbled up on the servers. We see memory filling up on the pubApp servers over the course of 6-8 hours. Once the memory fills up, we see a high CPU usage problaby due to ehcache having to invalidate items from the cache on every operation. Russ is setting up a profiler to examine memory use on our test servers to see if he can reproduce the problem under load. We’re still not exactly sure where the memory leak occurs but have a couple of ideas:

• ehcache
• Apache Axis memory mapper / SOAP server
• Dead mod_jk connections

We’ve also tracked down a number of actions that are querying Mulgara directly rather than using cache including some queries than can return a huge amount of data. The developers are re-writing the code to use the cache or to be smarter about data they’re pulling.
We’re not going to make any more patches to the production server until we diagnose and fix the memory problem. Search will be reinstated back on the site after we can build and test a new release candidate in the next 1-2 weeks.

Overall performance has been good since last week. We have to restart the stack three times a day due to the memory problem but these outages last less than 10 minutes. Since we have almost all caches shared with the new server, the sites come up very fast. Ingest of articles still slows down the sites but we have some ideas on how to make the ingest much faster.

I’ve set up stress tests today with WebLOAD. It’s a pretty useful tool with great reporting but it’s not really an Open Source application. You can use WebLOAD up to 100 virtual clients but have to buy a license for anything beyond. But it’s enough to stress test our multi-server stack if I set cache to a minimum. We’ve setup jProfiler to help diagnose the memory problems and hope to have some ideas where the memory leak is occurring soon.

We’re waiting for the Topaz developers to finish up code to serialize one last cache, move some data into the caches and a accelerate a couple of OTM queries. We should have a fix for search soon and may be able to put basic/advanced search back up on the journal websites soon if testing goes well.

I am sorry – I think I put the previous comment in the wrong place. The comment interface is a bit confusing, or maybe I am just confused.

I also missed the clustering discussion – looks like you are already doing a lot of that.

But still, I do recommend Varnish and putting some pressure off Apache. It will save quite a few CPU cycles.

Also, I just discovered that staging.plos.org does not compress content! This must be the reason the site appeared ‘slow’ to me. Look:


[toyvo@asusie ~]$ HEAD -H "Accept-encoding: gzip, deflate" http://www.staging.plos.org/cms/node/334
200 OK
Connection: close
Date: Tue, 18 Mar 2008 00:15:21 GMT
Server: Apache/2.0.52 (CentOS)
Content-Type: text/html; charset=utf-8
Client-Date: Tue, 18 Mar 2008 00:04:59 GMT
Client-Peer: 209.237.233.125:80
Client-Response-Num: 1
Set-Cookie: PHPSESSID=************************; expires=Thu, 10-Apr-2008 03:48:41 GMT; path=/
X-Powered-By: PHP/4.3.9


Please do enable that. For large pages, it reduces the load time considerably. It is not that difficult to install, just use
mod_deflate.

Thank you for the good work with PLoS.

Kind regards,

–Anton

Thank you, Richard, for replying. Oh, I see now, sorry about being confused about staging.plos.org and the TOPAZ sites.

But just to make sure you do not misunderstand me, I was talking about two separate issues. Looks like #2 is still a problem.

1. HTTP Proxy/Cache, you use SQUID – good; Varnish may be marginally better (up to 1.5 times faster in my experience) but it looks you have more pressing issues at hand; this is no issue then;

2. GZIP compression of HTML, CSS and the like as it gets send to the browser. I just checked plosone.org – that’s a TOPAZ site, right? And it also does not support this. Another proof is below.

Most modern browsers will do HTTP requests with ‘Accept-encoding: gzip,deflate’ and expect to be given gzip-compressed HTML in return. It saves bandwidth (at the cost of CPU time), and it typically makes the site appear faster, because pages load faster.

On-demand compression of HTML is completely different from how you
cache application objects in ehcache.

If plosone.org runs Apache, as seems from the headers, you should give a try to mod_deflate. This is a standard Apache 2+ module that does on-demand HTML compression transparently to the application that generates the HTML. You can configure it in 15 minutes and see if it helps.

From my console:


[toyvo@asusie ~]$ HEAD -H 'Accept-encoding: gzip,deflate' http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0001092
200 OK
Cache-Control: no-cache
Connection: close
Date: Thu, 20 Mar 2008 00:09:07 GMT
Pragma: no-cache
Server: Apache/2.0.52 (CentOS)
Content-Type: text/html;charset=UTF-8
Expires: -1
Client-Date: Wed, 19 Mar 2008 23:58:30 GMT
Client-Peer: 209.237.233.125:80
Client-Response-Num: 1
Set-Cookie: JSESSIONID=**********************; Path=/


If content was really encoded, the headers would have included a Content-encoding: gzip or similar header.

Finally, coming back to #1 (caching), Cache-Control: no-cache seems too restrictive for a static page, which this one appears to be. Relaxing this a bit can give SQUID some leverage and ease the load on the backend (TOPAZ). It can lead to unexpected behavior though if the page has dynamic elements that get updated while the cache does not know about it. This too can be worked around by PURGE requests… This is how Mediawiki does it – and those guys surely have some traffic 🙂