New, faster way to display modules

The performance boost is actually the reason I started this investigation.  But before I get into that and the details of the new system let me back up and explain a bit about module rendering.   To display a module you need 2 things: the module information itself and one or more stylesheets to transform the module into XHTML for publication on the web.  The situation is slightly more complicated since the module information is divided among CVS, PostgreSQL, and the ZODB, but data from both PostgreSQL and the ZODB can be abstracted behind a common Zope API.  This is usually done via a Zope Page Template (ZPT) to fill in the values dynamically.

Current Pipeline

1. CVS retreival of module text (CNXML+Content MathML)
   
2. XSL transformation into a ZPT with XHTML+Presentation MathML
   
1. If the text contains MathML and browser does not support MathML, XSL transformation into XHTML ZPT
   
3. ZPT Parsing and Evaluation pulls data from ZODB and PostgreSQL
   

This has worked well for us for quite some time.  However as I was recently measuring module rendering performance I was suprised at how much time was spent parsing the generated Page Template: 33-40% on the modules I examined.  In retrospect it makes sense: for most page templates parsing happens once and execution many times so they've probably pushed a lot of of the time intensive operations into parsing so that executing can be fast.  Since the generated ZPT is almost identical in all of our cases, there was no reason we couldn't refactor a bit to move the ZPT evaluation further up the line.

New Pipeline

1. ZPT Evaluation pulls data from ZODB and PostgreSQL and CVS retreival of module text into an XML export file
   
2. XSL transformation into XHTML+Presentation MathML
1. If the text contains MathML and browser does not support MathML, XSL transformation into XHTML

This new pipeline has several important features:

• ZPT Parsing occurs once at server startup
• All of the module information (CVS, PostgreSQL, and ZODB)  is now pulled together in a single step that genrates an XML file.  This could be useful in the future for providing a module export feature.  It will also provide a better means for providing extra information to the printing system.
  
• The look and feel of the module is now wholly under the control of the XSL transformation rather than relying on page template macros.  While this may require some duplication of site design, it also allows for the possibility of allowing course authors even more control over their presentation by specifying an XSL transformation.

At the same time, I also created an OldCNXMLDocument Product that handles the rendering for pre 0.5 CNXML by transforming it to 0.5.  This allows us to remove the old CNXML directories, their many redundant stylesheets, and several hardcoded paths from the code.  The transformation does not yield perfect output so it can't be used for upgrading the XML in the authoring system, but it does well enough as an intermediate step for module display.

Testing the new pipeline on yoda does indeed seem to indicate a significant speedup.  Hopefully this will take some of the load of our servers.

Future Notes

There are still improvements to be made, but these will most likely happen after the open-source release this month:

• Currently the XML export file wraps the CNXML with a <wrapper> tag and puts the additional info outside the document.  A better way might be to insert the info into the <metadata> tag *inside* <document>
• It might be valuable to make the metadata retrieval at its own URL
• The XSL stylesheets are currently stored in the 'www' uibdirectories of the Zope Product.  A better approach might be to create a new FSXSLTransform class so they can be part of the skin and easily customized.