/
Notes on Languages and Frameworks for Engage

Notes on Languages and Frameworks for Engage

Jul 21, 2009

Potential Languages and Frameworks

An important issue to be resolved, is the server-side language that will be used for user code. This includes code written by the Fluid community and by developers working with museums to implement and extend Engage. This is, in general, an issue distinct from that of the actual underlying host language on the server—and, for example, whether this engine is implemented on a JVM or not.

Some front-running choices for a user language include:

  • JavaScript

  • Python

  • PHP

  • (less probably) Ruby, or Java

In terms of access to a ready developer base, the two leading contenders from the set would be PHP and JavaScript.

PHP

A choice of PHP constrains many other options, since the only sensible delivery environment for PHP is its native one, typically bundled as the Apache plugin mod_php or fastCGI.

The arguments against PHP are its general lack of a functional programming features such as closures and first-class functions. Also the fact that the dominant Web idiom is for an embedding of the PHP language within its markup document. Given that Fluid has pioneered the use of unobtrusive techniques for designing user interfaces, it's unlikely that we'd use such techniques. The unfamiliarity of this approach in PHP weakens the "developer familiarity" arguments that would otherwise count for PHP.

JavaScript

JavaScript is probably the single most widely used and understood language in the web community. It is a simple language, easy to teach and learn. Indeed, the Fluid community has significant experience in simplifying the language and teaching it to new developers. Furthermore, the adoption of JavaScript at the service layer, along with CouchDB at the persistence layer, promises a completely coherent and unified environment for developers. To work with Fluid Engage, developers need only learn a single language, whether working at the client end, the service level, or the persistence level.

Choosing JavaScript as a language for Engage allows us a potential choice of runtimes and deployment environments. The most obvious choices are:

  • Rhino on the JVM

  • V8CGI or other next-generation runtime + Apache adaptor

Today, an implementation on a JVM may be attractive because of ready in-process access to Lucene/Solr.

Another argument in favour of JavaScript is its coming performance trajectory. Even more than any other scripting language, JavaScript VM optimisation is receiving very significant attention from front-running vendors such as Mozilla, Google and Apple. JavaScript performance is now on the front line of the browser wars.

Google's V8 engine is already at production quality both in the Chrome browser and as a plugin. Mozilla's competing TraceMonkey engine is available in Firefox 3.5, and an embedded implementation likely to follow. Either of these implementations would catapult JavaScript up the league table of scripting language performance. Currently, Ruby, Python, Javascript and PHP sit near the bottom of the table somewhere between 20-40x slower than Java/C++. A reasonable guess is that these coming VMs would raise JavaScript well above the pack.

In the end, we believe that performance is not a crucial issue in this space. The low spot occupied by PHP, relative to its enormous adoption, exhibits this. However, it is encouraging to be positioned with a environment receiving such intensive engineering focus.

Weighing the Options: Server-Side Languages and Web Frameworks

Ease of deployment and hosting

Some of our institutions may not have on-site administration support for hosting (as well as not having on-site development). In this case, it might be desirable to ensure that the Fluid Engage services layer can, as far as possible, be hosted on inexpensive or "standard" hosting packages. This might militate in favour of use of the standard LAMP stack, for one of the 3 values of P. It would, for example, militate against the use of a Java server side, or non-standard persistence solutions like CouchDB (see later).

Ease of access to development resource

Institutions may or may not have access to developers skilled in particular technologies. Of the available options, PHP is liable to give the easiest access to cheap development resource. However, that said, simple convergence on the use of a language may not give particular benefits in terms of access to background knowledge - Fluid's approach to markup agnosticism and loose binding to data will guide development away from many of the common patterns in server-side development.

JavaScript on the server is another promising option, since JavaScript is familiar to the vast majority of Web developers today, regardless of what language or technology they use on the server.

Positioning within a software community

For Fluid, choosing a language and framework is also about collaboration. We like to work with other communities of developers who have common interests and approach. We are particularly interested in communities who are at a compatible point in their technology lifecycle and likely to respond positively to aligned innovations. For example, the Java and PHP communities are highly mature, and fundamental innovations will arise on a very slow cycle.

Quick Start

Whatever solution is adopted, it should present minimal barriers to getting up and running on a short timescale ("1-click install") and should present a transparent model in terms of tool chain and file layout. This will make it easy for developers to dive into creating new services for Engage, with a minimum of setup time. This militates somewhat against the use of compiled languages such as Java, although using one of the growing group of alternative JVM languages such as JRuby, Jython or Rhino (for JavaScript) is an interesting option.

High Performance

Server-side languages other than Java tend to suffer from poor raw execution performance, although in terms of implementing a web framework this is rarely important. For Engage, we're predicting that the most resource-intensive operations will be data conversion and import. Also, Java's good CPU behaviour is often offset by a large memory footprint imposed by the standard JVM architecture. Of the other front-runners, Python is the most likely to receive a good performance boost in the mid-term as a result of strong interest from the Google corporation (see Unladen Swallow.

Language and the back end

Hosting JavaScript in Other Languages

As mentioned before, one option for hosting would be JavaScript via the Rhino engine, either as part of the CollectionSpace plugins model, or without it. Javascript could also be hosted as part of a Python runtime, perhaps on Google's V8 - another mechanism would be the Python-to-Javascript compiler, Pyjamas. The advogato report shows this combination performing very well - 8 to 10 times as fast as Python itself! However, one has to query, in this combination, what the value of a Python host is at all - if all user-visible code is actually written in Javascript. The performance argument, by itself, is not sufficiently compelling - we don't imagine very significant quantities of code written in the "glue"/service language we provide. Typically the only processor-intensive operation at this level might be template rendering - this could be addressed by either having an extremely performant JS engine (V8 or modern Tracemonkey) or else by co-opting in the Java version of the Fluid template rendering engine from RSF.

Maturity concerns

A choice between Javascript and Python raises maturity concerns in theory – which have three main areas of impact in practice. Firstly, Javascript frameworks have no established server-side container idiom – that is, the interface to the details of the web request, response, status and URL machinery are not standardised. Secondly, there is no well-defined threading model, even for the language as a whole – Javascript has been historically positioned as a single-threaded language on the client. The Rhino scripting engine for the JVM promotes the Java threading model to the Javascript level, but this might not provide consistency with other implementations. The modern Python container model is WSGI, and a very interesting discussion of its implications for frameworks, threading and containment is at WSGI Explorations in Python.

Another area of immaturity that might well impact development speed is the absence of a debugging toolchain. Rhino features a fairly clunky, 3-year vintage Swing debugger but this would not integrate with modern development tools. The Google V8 engine features a solid Debugging API but there is no evidence that any tools support this outside the Chrome browser.

Web Framework Standards

Server-side Web frameworks continue to proliferate in a broad range of languages. Java is probably the most extreme example of the range of tools available to choose from, but Ruby and Python increasingly reflect similar diversity. With this range of options, portability between Web servers and containers becomes more difficult.

One pattern we noticed while researching various server-side languages and frameworks was the increased standardization of simple Web container APIs. These APIs make it easier to port code between frameworks and ensure that there is common low-level infrastructure across competing frameworks. While the Servlet API is a familiar standard for the Java world, APIs such as Python's WSGI (Web Server Gateway Interface) and Ruby's Rack spec are available for dynamic languages, too. JSGI, a similar API for server-side JavaScript is described below.

While these technologies aren't sufficient in themselves to write a Web application against, they make portability across environments much easier.