Internal URL Rewriting with eXist’s MVC Framework

Since version 1.4, the eXist native XML database has been equipped with a Model View Controller (MVC) framework designed to express the logic for request routing of eXist-based web applications in XQuery. In this post I’ll illuminate a (in my opinion) somewhat under-exposed feature of eXist’s MVC framework: internal URL rewriting. With this term, I mean the fact that a URL, say http://localhost:8080/exist/urltest/test.xql is resolved internally to another URL like http://localhost:8080/exist/urltest/xquery/test.xql. Internally, meaning that the original request is not redirected to another one, and the user still sees the original URL in the browser address bar. As section 1of this post will illustrate, this works like a charm for ‘simple’ rewrites, like the previous one, but requires some thought if you would like to ‘chain’ multiple internal rewrite rules. In this post, I’ll try to provide a flexible coding pattern to achieve such internal rewriting with eXist’s MVC framework.

I’ll illustrate how this can be done with a toy web application ‘urltest’, that consists of only two basic scripts in two folders:

  • /urltest/xquery/test.xql: a simple script that will generate a <test> element, containing a ‘hello $name’ string, where $name can be passed as a request parameter:
    
    xquery version "1.0";
    <test>hello {request:get-parameter("name", "anonymous")}</test>
    
  • /urltest/stylesheets/test.xsl: an XSLT stylesheet that will just transform the <test> element to an HTML <h1> element:
    <?xml version="1.0" encoding="UTF-8"?>
    <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
        exclude-result-prefixes="#all"
        version="2.0">    
        <xsl:template match="test">
            <h1>query output: <xsl:apply-templates/></h1>
            <h2>processed by test.xsl</h2>
        </xsl:template>        
    </xsl:stylesheet>

You can download the source files for the mini-application right away. The next sections will gradually explain the steps from a very basic controller.xql version to the final version included in the application’s source code.

1. Forwarding and redirecting

The eXist MVC documentation nicely illustrates how above behaviour can be specified in a controller.xql file, by forwarding the request to a path that can be interpreted by the XQueryServlet. Note that the following discussion will take the concepts documented in the eXist MVC documentation for granted; the reader is directed there (externally) for full documentation. Following rules in a controller.xql file would match any request to an XQuery file ending in ‘.xql’ under the webapp’s root folder, forward it to the corresponding XQuery source file and apply an XSLT stylesheet of the same name as the requested XQuery script to the result:


(: [rule 1]: map all *.xql requests to their source files in xquery/, and corresponding xslt stylesheets in stylesheets/ :)
if (matches($exist:path, '^/[^/]*.xql$')) then
  let $query := substring-before(tokenize($exist:path, '/')[last()], '.')
  return
    <dispatch xmlns="http://exist.sourceforge.net/NS/exist">
      <forward url="{$exist:controller}/xquery/{$query}.xql">
        <set-attribute name="xslt.input" value="model"/>
        <set-attribute name="xslt.stylesheet" value="stylesheets/{$query}.xsl"/>
      </forward>
      <view>
        <forward servlet="XSLTServlet">
          <clear-attribute name="xslt.input"/>
        </forward>
      </view>		
    </dispatch>
(: default rule: just pass through everything else :)
else
    <dispatch xmlns="http://exist.sourceforge.net/NS/exist">
      <cache-control cache="yes"/>
    </dispatch>

Though the XQuery source file lives in the ‘xquery’ subfolder of the webapp’s root folder (‘/urltest’), rule 1 makes it possible to execute the ‘xquery/test.xql’ script via a URL like http://localhost:8080/exist/urltest/test.xql?name=bobby. As the $exist:path portion of the request doesn’t contain further subpaths, it will match the test condition of the first rule, so the request will be forwarded to the source file in the ‘xquery’ folder which will then be processed by the XQueryServlet, whose output will subsequently be processed by the ‘test.xsl’ XSLT stylesheet in the ‘stylesheets’ folder. This will produce following output:

<h1>query output: hello bobby</h1>
<h2>processed by test.xsl</h2>

Compare this to what happens with the URL http://localhost:8080/exist/urltest/xquery/test.xql?name=bobby, which addresses the XQuery source file more directly by including its containing folder ‘/xquery’ in the path. The $exist:path section of this request doesn’t match rule 1, and is hence caught by the default rule, which will just pass through the request to XQueryServlet (the default servlet for files ending in ‘.xql’). No further processing is applied, however, so the result looks like this:

<test>hello bobby</test>

Now, suppose we wanted to add a rule that would intercept URLs with empty root paths, like http://localhost:8080/urltest/, and rewrite them to http://localhost:8080/urltest/test.xql. This can be done by adding another rule to the controller, for example:


(: [rule 2]: redirect empty paths to main query :)
else if ($exist:path eq '/') then
    <dispatch xmlns="http://exist.sourceforge.net/NS/exist">
      <redirect url="test.xql"/>
    </dispatch>  

With this rule in place, the URL http://localhost:8080/urltest/ will be redirected and hence generate a new request to http://localhost:8080/urltest/test.xql. Because of this redirection, the original URL in the browser will be changed to the latter one.

Now, suppose you’re not interested in a redirection, but would rather want to express this as a background forwarding operation. The most naive solution would replace the <redirect> instruction with <forward>, which won’t work, unless the new URL is specified as <forward url=”xquery/test.xql”/>. While this will execute the ‘xquery/test.xql’ XQuery script and preserve the original URL in the browser, no further XSLT processing will be done on the XQuery output, unless the <dispatch> element is extended to something very similar to the content of rule 1:


(: [rule 2]: redirect empty paths to main query :)
else if ($exist:path eq '/') then
    <dispatch xmlns="http://exist.sourceforge.net/NS/exist">
      <forward url="xquery/test.xql">
        <set-attribute name="xslt.input" value="model"/>
        <set-attribute name="xslt.stylesheet" value="stylesheets/test.xsl"/>
      </forward>
      <view>
        <forward servlet="XSLTServlet">
          <clear-attribute name="xslt.input"/>
        </forward>
      </view>
    </dispatch>

On the other hand, both rules could be combined in a single rule by making some small modifications to rule 1:


(: [rule 1]: map all *.xql requests to their source files in xquery/, and corresponding xslt stylesheets in stylesheets/ :)
if (matches($exist:path, '^/[^/]*.xql$') or $exist:path eq '/') then
  let $query := (substring-before(tokenize($exist:path, '/')[last()], '.')[normalize-space()], 'test')[1]
  return
    <dispatch xmlns="http://exist.sourceforge.net/NS/exist">
      <forward url="{$exist:controller}/xquery/{$query}.xql">
        <set-attribute name="xslt.input" value="model"/>
        <set-attribute name="xslt.stylesheet" value="stylesheets/{$query}.xsl"/>
      </forward>
      <view>
        <forward servlet="XSLTServlet">
          <clear-attribute name="xslt.input"/>
        </forward>
      </view>		
    </dispatch>

This rule then combines both conditions, and derives $query, the variable containing the base name for the corresponding XQuery and XSLT scripts, from the $exist:path part of the request, or sets ‘test’ as default value. By the way, this way of expressing default values for variables is a very useful XQuery programming pattern that can be found at the XQuery/Ah-has of the invaluable XQuery Wikibook.

To summarize this section, we now can specify eXist controller rules that forward URLs to specific internal paths. In order to provide multiple ‘entry points’ for those rules, two approaches have been illustrated:

  • redirection: force a URL to be processed by another controller rule by issuing a redirection that will rewrite the original URL to one that will be matched by the rule of interest
  • forwarding: expand rules with multiple matching conditions and further modifications where necessary to cope with specifics of those entry points

Both approaches have their drawbacks: external redirection of requests can disturb clean URLs and possibly threaten link stability (by exposing the possibly less stable internal URLs to the user of the web application). The forwarding approach illustrated above suffers from either code redundancy, or code bloating. Combining multiple matching conditions may introduce a lot of complexity in the further processing code as well. In the next section, a more elegant solution to internal URL rewriting will be explored.

2. Internal Rewrites

2.1 Simple internal rewrites

This brings me to the actual motivation for this blog post. I am converting Cocoon+eXist-based web applications to purely eXist-driven ones. This involves porting the Cocoon sitemap logic to eXist’s controller logic. Cocoon sitemaps explicitly allow for what the documentation calls internal redirection, where a sitemap pipeline rewrites a URL for further processing by another pipeline in the background. For example, both rule 1 and rule 2 of above controller could be expressed in a Cocoon sitemap as follows:

 

<!-- [rule 1]: map all *.xql requests to their source files in xquery/, and corresponding xslt stylesheets in stylesheets/ -->
<map:match pattern="*.xql">
  <map:generate src="xquery/{1}.xql" type="xquery">
  <map:transform src="stylesheets/{1}.xsl">
  <map:serialize type="xhtml"/>
</map:match>

<!-- [rule 2]: redirect empty paths to main query -->
<map:match pattern="/">
  <map:redirect-to uri="cocoon:/test.xql"/>
</map:match>

The “cocoon:/” prefix before the redirect URL will force Cocoon to redirect the original request to “/test.xql”, but to do so silently, without issuing a real HTTP redirect. Result: rule 2 instructs the pipeline processor to look for other matching pipelines, in this case rule 1. The URL http://localhost:8080/urltest/ will execute the XQuery script at http://localhost:8080/urltest/test.xql, without being replaced in the browser address bar.

As hinted at in the last part of the previous section, eXist does not offer such a shortcut for internal rewrites of URLs. Forwarding the original request to just “test.xql” resulted in an error because eXist’s XQueryServlet can’t find a source file at ‘”/urltest/test.xql”; just forwarding it to “xquery/test.xql” didn’t apply any XSLT stylesheet to the XQuery result. In short: rule 2 doesn’t fire rule 1. In fact, this seems to be an explicit limitation of the eXist controller mechanism, as noted in the documentation:

It is important to understand that only one (!) controller will ever be applied to a given request. It is not possible to forward from one controller to another (or the same). Once you either ignored or forwarded a request in the controller, it will be directly passed to the servlet which handles it or – if it references a resource – it will be processed by the servlet engine itself. The controller will not be called again for the same request.

This set me back for a while and made me resort to external redirection. For the example given so far, it doesn’t matter that much if an URL with an empty root path is explicitly rewritten to the URL for the default start page. However, for RESTful URLs, it would be less convenient to have e.g. http://localhost:8080/exist/urltest/hello/bobby explicitly redirected to http://localhost:8080/exist/urltest/test.xql?name=bobby. Fortunately, it took me only one question on the eXist-open mailing list to find a way around this limitation. Loren Cahlander kindly pointed me toward an elegant solution, that delegates common behaviour of different rules in a controller to separate XQuery functions. This looks attractive to me for two reasons:

  • brevity: common code is isolated, and redundancy can be reduced
  • flexibility: multiple entry points can still be provided in different controller rules, thus avoiding to create monolithic rules with unwieldy matching conditions that could further complicate their internal logic

Let’s see how both controller rules we specified so far can be refactored by using functions. As a first step, let’s isolate the <dispatch> instruction of rule 1 in a separate function local:basicXQuery():


declare function local:basicXQuery($query) {
    <dispatch xmlns="http://exist.sourceforge.net/NS/exist">
      <forward url="{$exist:controller}/xquery/{$query}.xql">
        <set-attribute name="xslt.input" value="model"/>
        <set-attribute name="xslt.stylesheet" value="stylesheets/{$query}.xsl"/>
      </forward>
      <view>
        <forward servlet="XSLTServlet">
          <clear-attribute name="xslt.input"/>
        </forward>
      </view>		
    </dispatch>
};

If we specify that this function takes a $query argument, all we have to do is call this function in both rules, with the right $query argument:


(: [rule 1]: map all *.xql requests to their source files in xquery/, and corresponding xslt stylesheets in stylesheets/ :)
if (matches($exist:path, '^/[^/]*.xql$')) then
  let $query := substring-before(tokenize($exist:path, '/')[last()], '.')
  return local:basicXQuery($query)
(: [rule 2]: redirect empty paths to main query :)	
  else if ($exist:path eq '/') then
  local:basicXQuery('test')

This will have the same effect as both approaches concluding the previous section, but with the advantages mentioned above: brevity and flexibility, in short elegance.

2.2 Passing Parameters

So far for the simple case. Suppose we want to take this a step further and develop an embryonic system for RESTful URLs for our baby application that would map requests for http://localhost:8080/exist/urltest/hello/bobby in the background to http://localhost:8080/exist/urltest/test.xql?name=bobby. This can be done easily by adding a new rule to the controller:


(: [rule 3]: forward RESTful urls expressed as /hello/name to xquery/test.xql with appropriate $name parameter :)
else if (starts-with($exist:path, '/hello/')) then
  local:basicXQuery('test')

At first sight, this seems to work fairly well: a request for http://localhost:8080/exist/urltest/hello/bobby produces following output:

<h1>query output: hello anonymous</h1>
<h2>processed by test.xsl</h2>

There’s clearly one detail we have to fix: although we did supply a name in the URL (the part after ‘/hello/’, i.e. ‘bobby’), this was not passed to the local:basicXQuery() function. Since this name wasn’t supplied as a request parameter, it won’t be directly available to the ‘test.xql’ XQuery file. Fortunately, eXist’s MVC framework provides a means to inject request parameters from a controller, by adding following element to a <forward> or <redirect> action:

<add-parameter name="[name]" value="[value]"/>

This should be done in the <forward url=”{$exist:controller}/xquery/{$query}.xql”> action of the local:baseXQuery() function. Of course, this could be done by modifying it to:


<forward url="{$exist:controller}/xquery/{$query}.xql">
  { if (starts-with($exist:path, '/hello/')) then
      <add-parameter name="name" value="{tokenize($exist:path, '/')[3]}"/>
    else ()
  }
  <set-attribute name="xslt.input" value="model"/>
  <set-attribute name="xslt.stylesheet" value="stylesheets/{$query}.xsl"/>
</forward>

Here, the value for the “name” parameter will be derived from the part of the $exist:path variable following “/hello/”. This would work, but still wouldn’t be much removed from the code bloat the unified controller function at the end of section 1 suffered from.Instead, it would be more interesting to remove this case-specific code from the general local:baseXQuery() function, and move it to the calling functions instead.

Though rule 3 of our controller can’t add this parameter directly to the request, we can pass them in a useful form to the local:baseXQuery() function. In order make this mechanism as generic as possible, local:baseXQuery() would ideally remain agnostic about the actual (number of) parameters that could be passed from other controller rules. Therefore, it makes most sense to group all parameters in one single element, for example <params>, which can then specify all parameters to be passed in child elements. Now, nothing prevents us to construct those parameter definitions as separate <add-parameter> child elements of that <params> element:


(: [rule 3]: forward RESTful urls expressed as /hello/name to xquery/test.xql with appropriate $name parameter :)
else if (starts-with($exist:path, '/hello/')) then
  let $name := tokenize($exist:path, '/')[3]
  let $params := 
    <params xmlns="http://exist.sourceforge.net/NS/exist">
      <add-parameter name="name" value="{$name}"/>
    </params>
  return local:basicXQuery('test', $params)

In order to process these parameters, the local:basicXQuery() function has to be changed so it accepts the second parameter $params. Then, it only requires a minimal change to make the function copy the <add-parameter> children of $params inside the first <forward> instruction:


declare function local:basicXQuery($query, $params) {
    <dispatch xmlns="http://exist.sourceforge.net/NS/exist">
      <forward url="{$exist:controller}/xquery/{$query}.xql">
  	{$params//add-parameter}
        <set-attribute name="xslt.input" value="model"/>
        <set-attribute name="xslt.stylesheet" value="stylesheets/{$query}.xsl"/>
      </forward>
      <view>
        <forward servlet="XSLTServlet">
          <clear-attribute name="xslt.input"/>
        </forward>
      </view>		
    </dispatch>
};

Of course since XQuery doen’t allow to omit empty arguments from function calls, the other calls to local:basicXQuery() must be changed as well, so they pass an empty sequence () for the $params parameter. This adds up to the following final version of our controller:


declare function local:basicXQuery($query, $params) {
    <dispatch xmlns="http://exist.sourceforge.net/NS/exist">
      <forward url="{$exist:controller}/xquery/{$query}.xql">
        {$params//add-parameter}
        <set-attribute name="xslt.input" value="model"/>
        <set-attribute name="xslt.stylesheet" value="stylesheets/{$query}.xsl"/>
      </forward>
      <view>
        <forward servlet="XSLTServlet">
          <clear-attribute name="xslt.input"/>
        </forward>
      </view>
    </dispatch>
};

(: [rule 1]: map all *.xql requests to their source files in xquery/, and corresponding xslt stylesheets in stylesheets/ :)
if (matches($exist:path, '^/[^/]*.xql$')) then
  let $query := substring-before(tokenize($exist:path, '/')[last()], '.')
  return local:basicXQuery($query, ())
(: [rule 2]: redirect empty paths to main query :)
  else if ($exist:path eq '/') then
  local:basicXQuery('test', ())
(: [rule 3]: forward RESTful urls expressed as /hello/name to xquery/test.xql with appropriate $name parameter :)
else if (starts-with($exist:path, '/hello/')) then
  let $name := tokenize($exist:path, '/')[3]
  let $params := 
    <params xmlns="http://exist.sourceforge.net/NS/exist">
      <add-parameter name="name" value="{$name}"/>
    </params>
  return local:basicXQuery('test', $params)
(: default rule: just pass through everything else :)
else
    <dispatch xmlns="http://exist.sourceforge.net/NS/exist">
      <cache-control cache="yes"/>
    </dispatch>

Et voila: the test.xql XQuery file of our application can now be called by means of following URLs:

If you want to see it In action yourself, just download the application source files, extract the contents of the zip file to the %EXIST_HOME%/webapps, folder and navigate to any of the URLs mentioned above. I swear one day I’ll be able to produce a nice installable XQuery package

On the other hand, I realize I’m just touching on the wealth of possibilities for developing advanced MVC patterns with eXist. For a much more advanced example, see how Joe Wicentowski even further abstracted the MVC instructions (forward, redirect, ignore, and add-parameter) into dedicated functions. But this first step really was an eye-opener to me. Once again, thanks, Loren for pointing me in the right direction!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: