I’ll illustrate how this can be done with a toy web application ‘urltest’, that consists of only two basic scripts in two folders:

• /urltest/xquery/test.xql: a simple script that will generate a <test> element, containing a ‘hello $name’ string, where $name can be passed as a request parameter:

  
  xquery version "1.0";
  <test>hello {request:get-parameter("name", "anonymous")}</test>
  

• /urltest/stylesheets/test.xsl: an XSLT stylesheet that will just transform the <test> element to an HTML <h1> element:

  <?xml version="1.0" encoding="UTF-8"?>
  <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
      exclude-result-prefixes="#all"
      version="2.0">
      <xsl:template match="test">
          <h1>query output: <xsl:apply-templates/></h1>
          <h2>processed by test.xsl</h2>
      </xsl:template>
  </xsl:stylesheet>

You can download the source files for the mini-application right away. The next sections will gradually explain the steps from a very basic controller.xql version to the final version included in the application’s source code.

1. Forwarding and redirecting

The eXist MVC documentation nicely illustrates how above behaviour can be specified in a controller.xql file, by forwarding the request to a path that can be interpreted by the XQueryServlet. Note that the following discussion will take the concepts documented in the eXist MVC documentation for granted; the reader is directed there (externally) for full documentation. Following rules in a controller.xql file would match any request to an XQuery file ending in ‘.xql’ under the webapp’s root folder, forward it to the corresponding XQuery source file and apply an XSLT stylesheet of the same name as the requested XQuery script to the result:

(: [rule 1]: map all *.xql requests to their source files in xquery/, and corresponding xslt stylesheets in stylesheets/ :)
if (matches($exist:path, '^/[^/]*.xql$')) then
  let $query := substring-before(tokenize($exist:path, '/')[last()], '.')
  return
    <dispatch xmlns="http://exist.sourceforge.net/NS/exist">
      <forward url="{$exist:controller}/xquery/{$query}.xql">
        <set-attribute name="xslt.input" value="model"/>
        <set-attribute name="xslt.stylesheet" value="stylesheets/{$query}.xsl"/>
      </forward>
      <view>
        <forward servlet="XSLTServlet">
          <clear-attribute name="xslt.input"/>
        </forward>
      </view>
    </dispatch>
(: default rule: just pass through everything else :)
else
    <dispatch xmlns="http://exist.sourceforge.net/NS/exist">
      <cache-control cache="yes"/>
    </dispatch>

Though the XQuery source file lives in the ‘xquery’ subfolder of the webapp’s root folder (‘/urltest’), rule 1 makes it possible to execute the ‘xquery/test.xql’ script via a URL like http://localhost:8080/exist/urltest/test.xql?name=bobby. As the $exist:path portion of the request doesn’t contain further subpaths, it will match the test condition of the first rule, so the request will be forwarded to the source file in the ‘xquery’ folder which will then be processed by the XQueryServlet, whose output will subsequently be processed by the ‘test.xsl’ XSLT stylesheet in the ‘stylesheets’ folder. This will produce following output:

<h1>query output: hello bobby</h1>
<h2>processed by test.xsl</h2>

Compare this to what happens with the URL http://localhost:8080/exist/urltest/xquery/test.xql?name=bobby, which addresses the XQuery source file more directly by including its containing folder ‘/xquery’ in the path. The $exist:path section of this request doesn’t match rule 1, and is hence caught by the default rule, which will just pass through the request to XQueryServlet (the default servlet for files ending in ‘.xql’). No further processing is applied, however, so the result looks like this:

<test>hello bobby</test>

Now, suppose we wanted to add a rule that would intercept URLs with empty root paths, like http://localhost:8080/urltest/, and rewrite them to http://localhost:8080/urltest/test.xql. This can be done by adding another rule to the controller, for example:

(: [rule 2]: redirect empty paths to main query :)
else if ($exist:path eq '/') then
    <dispatch xmlns="http://exist.sourceforge.net/NS/exist">
      <redirect url="test.xql"/>
    </dispatch>

With this rule in place, the URL http://localhost:8080/urltest/ will be redirected and hence generate a new request to http://localhost:8080/urltest/test.xql. Because of this redirection, the original URL in the browser will be changed to the latter one.

Now, suppose you’re not interested in a redirection, but would rather want to express this as a background forwarding operation. The most naive solution would replace the <redirect> instruction with <forward>, which won’t work, unless the new URL is specified as <forward url=”xquery/test.xql”/>. While this will execute the ‘xquery/test.xql’ XQuery script and preserve the original URL in the browser, no further XSLT processing will be done on the XQuery output, unless the <dispatch> element is extended to something very similar to the content of rule 1:

(: [rule 2]: redirect empty paths to main query :)
else if ($exist:path eq '/') then
    <dispatch xmlns="http://exist.sourceforge.net/NS/exist">
      <forward url="xquery/test.xql">
        <set-attribute name="xslt.input" value="model"/>
        <set-attribute name="xslt.stylesheet" value="stylesheets/test.xsl"/>
      </forward>
      <view>
        <forward servlet="XSLTServlet">
          <clear-attribute name="xslt.input"/>
        </forward>
      </view>
    </dispatch>

On the other hand, both rules could be combined in a single rule by making some small modifications to rule 1:

(: [rule 1]: map all *.xql requests to their source files in xquery/, and corresponding xslt stylesheets in stylesheets/ :)
if (matches($exist:path, '^/[^/]*.xql$') or $exist:path eq '/') then
  let $query := (substring-before(tokenize($exist:path, '/')[last()], '.')[normalize-space()], 'test')[1]
  return
    <dispatch xmlns="http://exist.sourceforge.net/NS/exist">
      <forward url="{$exist:controller}/xquery/{$query}.xql">
        <set-attribute name="xslt.input" value="model"/>
        <set-attribute name="xslt.stylesheet" value="stylesheets/{$query}.xsl"/>
      </forward>
      <view>
        <forward servlet="XSLTServlet">
          <clear-attribute name="xslt.input"/>
        </forward>
      </view>
    </dispatch>

This rule then combines both conditions, and derives $query, the variable containing the base name for the corresponding XQuery and XSLT scripts, from the $exist:path part of the request, or sets ‘test’ as default value. By the way, this way of expressing default values for variables is a very useful XQuery programming pattern that can be found at the XQuery/Ah-has of the invaluable XQuery Wikibook.

To summarize this section, we now can specify eXist controller rules that forward URLs to specific internal paths. In order to provide multiple ‘entry points’ for those rules, two approaches have been illustrated:

• redirection: force a URL to be processed by another controller rule by issuing a redirection that will rewrite the original URL to one that will be matched by the rule of interest
• forwarding: expand rules with multiple matching conditions and further modifications where necessary to cope with specifics of those entry points

Both approaches have their drawbacks: external redirection of requests can disturb clean URLs and possibly threaten link stability (by exposing the possibly less stable internal URLs to the user of the web application). The forwarding approach illustrated above suffers from either code redundancy, or code bloating. Combining multiple matching conditions may introduce a lot of complexity in the further processing code as well. In the next section, a more elegant solution to internal URL rewriting will be explored.

2. Internal Rewrites

2.1 Simple internal rewrites

This brings me to the actual motivation for this blog post. I am converting Cocoon+eXist-based web applications to purely eXist-driven ones. This involves porting the Cocoon sitemap logic to eXist’s controller logic. Cocoon sitemaps explicitly allow for what the documentation calls internal redirection, where a sitemap pipeline rewrites a URL for further processing by another pipeline in the background. For example, both rule 1 and rule 2 of above controller could be expressed in a Cocoon sitemap as follows:

<!-- [rule 1]: map all *.xql requests to their source files in xquery/, and corresponding xslt stylesheets in stylesheets/ -->
<map:match pattern="*.xql">
  <map:generate src="xquery/{1}.xql" type="xquery">
  <map:transform src="stylesheets/{1}.xsl">
  <map:serialize type="xhtml"/>
</map:match>

<!-- [rule 2]: redirect empty paths to main query -->
<map:match pattern="/">
  <map:redirect-to uri="cocoon:/test.xql"/>
</map:match>

The “cocoon:/” prefix before the redirect URL will force Cocoon to redirect the original request to “/test.xql”, but to do so silently, without issuing a real HTTP redirect. Result: rule 2 instructs the pipeline processor to look for other matching pipelines, in this case rule 1. The URL http://localhost:8080/urltest/ will execute the XQuery script at http://localhost:8080/urltest/test.xql, without being replaced in the browser address bar.

As hinted at in the last part of the previous section, eXist does not offer such a shortcut for internal rewrites of URLs. Forwarding the original request to just “test.xql” resulted in an error because eXist’s XQueryServlet can’t find a source file at ‘”/urltest/test.xql”; just forwarding it to “xquery/test.xql” didn’t apply any XSLT stylesheet to the XQuery result. In short: rule 2 doesn’t fire rule 1. In fact, this seems to be an explicit limitation of the eXist controller mechanism, as noted in the documentation:

It is important to understand that only one (!) controller will ever be applied to a given request. It is not possible to forward from one controller to another (or the same). Once you either ignored or forwarded a request in the controller, it will be directly passed to the servlet which handles it or – if it references a resource – it will be processed by the servlet engine itself. The controller will not be called again for the same request.

This set me back for a while and made me resort to external redirection. For the example given so far, it doesn’t matter that much if an URL with an empty root path is explicitly rewritten to the URL for the default start page. However, for RESTful URLs, it would be less convenient to have e.g. http://localhost:8080/exist/urltest/hello/bobby explicitly redirected to http://localhost:8080/exist/urltest/test.xql?name=bobby. Fortunately, it took me only one question on the eXist-open mailing list to find a way around this limitation. Loren Cahlander kindly pointed me toward an elegant solution, that delegates common behaviour of different rules in a controller to separate XQuery functions. This looks attractive to me for two reasons:

• brevity: common code is isolated, and redundancy can be reduced
• flexibility: multiple entry points can still be provided in different controller rules, thus avoiding to create monolithic rules with unwieldy matching conditions that could further complicate their internal logic

Let’s see how both controller rules we specified so far can be refactored by using functions. As a first step, let’s isolate the <dispatch> instruction of rule 1 in a separate function local:basicXQuery():

declare function local:basicXQuery($query) {
    <dispatch xmlns="http://exist.sourceforge.net/NS/exist">
      <forward url="{$exist:controller}/xquery/{$query}.xql">
        <set-attribute name="xslt.input" value="model"/>
        <set-attribute name="xslt.stylesheet" value="stylesheets/{$query}.xsl"/>
      </forward>
      <view>
        <forward servlet="XSLTServlet">
          <clear-attribute name="xslt.input"/>
        </forward>
      </view>
    </dispatch>
};

If we specify that this function takes a $query argument, all we have to do is call this function in both rules, with the right $query argument:

(: [rule 1]: map all *.xql requests to their source files in xquery/, and corresponding xslt stylesheets in stylesheets/ :)
if (matches($exist:path, '^/[^/]*.xql$')) then
  let $query := substring-before(tokenize($exist:path, '/')[last()], '.')
  return local:basicXQuery($query)
(: [rule 2]: redirect empty paths to main query :)
  else if ($exist:path eq '/') then
  local:basicXQuery('test')

This will have the same effect as both approaches concluding the previous section, but with the advantages mentioned above: brevity and flexibility, in short elegance.

2.2 Passing Parameters

So far for the simple case. Suppose we want to take this a step further and develop an embryonic system for RESTful URLs for our baby application that would map requests for http://localhost:8080/exist/urltest/hello/bobby in the background to http://localhost:8080/exist/urltest/test.xql?name=bobby. This can be done easily by adding a new rule to the controller:

(: [rule 3]: forward RESTful urls expressed as /hello/name to xquery/test.xql with appropriate $name parameter :)
else if (starts-with($exist:path, '/hello/')) then
  local:basicXQuery('test')

At first sight, this seems to work fairly well: a request for http://localhost:8080/exist/urltest/hello/bobby produces following output:

<h1>query output: hello anonymous</h1>
<h2>processed by test.xsl</h2>

There’s clearly one detail we have to fix: although we did supply a name in the URL (the part after ‘/hello/’, i.e. ‘bobby’), this was not passed to the local:basicXQuery() function. Since this name wasn’t supplied as a request parameter, it won’t be directly available to the ‘test.xql’ XQuery file. Fortunately, eXist’s MVC framework provides a means to inject request parameters from a controller, by adding following element to a <forward> or <redirect> action:

<add-parameter name="[name]" value="[value]"/>

This should be done in the <forward url=”{$exist:controller}/xquery/{$query}.xql”> action of the local:baseXQuery() function. Of course, this could be done by modifying it to:

<forward url="{$exist:controller}/xquery/{$query}.xql">
  { if (starts-with($exist:path, '/hello/')) then
      <add-parameter name="name" value="{tokenize($exist:path, '/')[3]}"/>
    else ()
  }
  <set-attribute name="xslt.input" value="model"/>
  <set-attribute name="xslt.stylesheet" value="stylesheets/{$query}.xsl"/>
</forward>

Here, the value for the “name” parameter will be derived from the part of the $exist:path variable following “/hello/”. This would work, but still wouldn’t be much removed from the code bloat the unified controller function at the end of section 1 suffered from.Instead, it would be more interesting to remove this case-specific code from the general local:baseXQuery() function, and move it to the calling functions instead.

Though rule 3 of our controller can’t add this parameter directly to the request, we can pass them in a useful form to the local:baseXQuery() function. In order make this mechanism as generic as possible, local:baseXQuery() would ideally remain agnostic about the actual (number of) parameters that could be passed from other controller rules. Therefore, it makes most sense to group all parameters in one single element, for example <params>, which can then specify all parameters to be passed in child elements. Now, nothing prevents us to construct those parameter definitions as separate <add-parameter> child elements of that <params> element:

(: [rule 3]: forward RESTful urls expressed as /hello/name to xquery/test.xql with appropriate $name parameter :)
else if (starts-with($exist:path, '/hello/')) then
  let $name := tokenize($exist:path, '/')[3]
  let $params :=
    <params xmlns="http://exist.sourceforge.net/NS/exist">
      <add-parameter name="name" value="{$name}"/>
    </params>
  return local:basicXQuery('test', $params)

In order to process these parameters, the local:basicXQuery() function has to be changed so it accepts the second parameter $params. Then, it only requires a minimal change to make the function copy the <add-parameter> children of $params inside the first <forward> instruction:

declare function local:basicXQuery($query, $params) {
    <dispatch xmlns="http://exist.sourceforge.net/NS/exist">
      <forward url="{$exist:controller}/xquery/{$query}.xql">
  	{$params//add-parameter}
        <set-attribute name="xslt.input" value="model"/>
        <set-attribute name="xslt.stylesheet" value="stylesheets/{$query}.xsl"/>
      </forward>
      <view>
        <forward servlet="XSLTServlet">
          <clear-attribute name="xslt.input"/>
        </forward>
      </view>
    </dispatch>
};

Of course since XQuery doen’t allow to omit empty arguments from function calls, the other calls to local:basicXQuery() must be changed as well, so they pass an empty sequence () for the $params parameter. This adds up to the following final version of our controller:

declare function local:basicXQuery($query, $params) {
    <dispatch xmlns="http://exist.sourceforge.net/NS/exist">
      <forward url="{$exist:controller}/xquery/{$query}.xql">
        {$params//add-parameter}
        <set-attribute name="xslt.input" value="model"/>
        <set-attribute name="xslt.stylesheet" value="stylesheets/{$query}.xsl"/>
      </forward>
      <view>
        <forward servlet="XSLTServlet">
          <clear-attribute name="xslt.input"/>
        </forward>
      </view>
    </dispatch>
};

(: [rule 1]: map all *.xql requests to their source files in xquery/, and corresponding xslt stylesheets in stylesheets/ :)
if (matches($exist:path, '^/[^/]*.xql$')) then
  let $query := substring-before(tokenize($exist:path, '/')[last()], '.')
  return local:basicXQuery($query, ())
(: [rule 2]: redirect empty paths to main query :)
  else if ($exist:path eq '/') then
  local:basicXQuery('test', ())
(: [rule 3]: forward RESTful urls expressed as /hello/name to xquery/test.xql with appropriate $name parameter :)
else if (starts-with($exist:path, '/hello/')) then
  let $name := tokenize($exist:path, '/')[3]
  let $params :=
    <params xmlns="http://exist.sourceforge.net/NS/exist">
      <add-parameter name="name" value="{$name}"/>
    </params>
  return local:basicXQuery('test', $params)
(: default rule: just pass through everything else :)
else
    <dispatch xmlns="http://exist.sourceforge.net/NS/exist">
      <cache-control cache="yes"/>
    </dispatch>

Et voila: the test.xql XQuery file of our application can now be called by means of following URLs:

• http://localhost:8080/exist/urltest/test.xql?name=bobby
• http://localhost:8080/exist/urltest/ (will be rewritten internally to http://localhost:8080/exist/urltest/test.xql)
• http://localhost:8080/exist/urltest/hello/bobby (http://localhost:8080/exist/urltest/test.xql?name=bobby)

If you want to see it In action yourself, just download the application source files, extract the contents of the zip file to the %EXIST_HOME%/webapps, folder and navigate to any of the URLs mentioned above. I swear one day I’ll be able to produce a nice installable XQuery package…

On the other hand, I realize I’m just touching on the wealth of possibilities for developing advanced MVC patterns with eXist. For a much more advanced example, see how Joe Wicentowski even further abstracted the MVC instructions (forward, redirect, ignore, and add-parameter) into dedicated functions. But this first step really was an eye-opener to me. Once again, thanks, Loren for pointing me in the right direction!

<p>
    <span class="previous">... s effect, sir; after what flourish your </span>
    <span class="hi">nature</span>
    <span class="following"> will.</span>
</p>

This formatting of search results invites to exploit its particular features, such as sorting the search results according to their left or right contexts, or even according to the n^th word preceding or following the search term. This is heavily facilitated by the XML representation of the KWIC search results, where all three parts are isolated in their own XML element. However, while eXist’s current KWIC display module (as it is consistently called) does its job in presenting a KWIC display, in my opinion it is too much display-oriented:

• it lacks performance on large result sets, and / or wide context widths, which is crucial for further processing, since sorting requires pre-computation of the entire result set
• (though this is nitpicking:) the output is presentational HTML; while this is irrelevant from a processing point of view, I would prefer a semantically more ‘neutral’ format and defer presentational formatting to a later display phase

This post will address both objections and present alternatives. Additionally, ways for processing these KWIC results are discussed in the last section.

Disclaimer: this will be the most technical post so far, an non-programmer’s attempt to explain some algorithms. The last parts, however, return to practical grounds. Furthermore, a basic knowledge of the KWIC display function is assumed; see the eXist documentation and function reference for full documentation.

1. Strategies for improving the KWIC display module

In order to anchor this discussion of the KWIC display module, I will refer to the last changed revision (9892) in the SVN trunk of the eXist project, rather than duplicating it here. It currently has 2 main entry points:

• kwic:summarize($node, $config, $callback?): a simplified function, allowing to pass a node containing search hits, a configuration element specifying display options like the context width or display format, and a callback function (optional) for specific processing of the left and right text contexts
• kwic:get-summary($root, $node, $config, $callback?): a more complex function, adding one more parameter: a root node that serves as a cutoff point for the left or right text contexts

Actually, the kwic:get-summary() function is the basic function, since all the kwic:summarize() function does is determine a $root element for the text contexts and push further processing to kwic:get-summary().

The meat of the kwic:get-summary() function is a routine that determines the right amount of left and right text context to be included in the KWIC display. This is done by 2 helper functions, respectively kwic:truncate-previous() and kwic:truncate-following(). Both accept following arguments:

• $root: the root node that serves as a cutoff point for the text contexts
• $node: the current search term
• $truncated: the text context text constructed so far
• $max: the length of the text context constructed so far
• $chars: the remaining available length of the text context
• $callback: a reference to a specific function for further processing of the left or right text contexts

Starting from the current search term, these functions will test if there is a preceding or following text node. If so, they will test whether this text node and the currently constructed text context exceed the maximum context length. If so, the text context is expanded with a substring of this text node, and this result is returned. If not, the truncation function calls itself recursively, to test for the text node preceding or following the currently selected text node, until the maximum context length is reached.

Since these recursive functions are called for each search hit, this means that their efficiency is cumulative, and bottlenecks add up. Actually, there is some room for improvement, by reducing the choices to be made and functions to be called. Let’s examine a tuned version of kwic:truncate-following():

1 declare function kwic:truncate-following($root as node()?, $node as node()?,
    $truncated as item()*, $width as xs:int, $callback as function?)
2  {
3    let $nextProbe := $node/following::text()[1]
4    let $next :=
5      if ($root[not(. intersect $nextProbe/ancestor::*)]) then ()
6      else $nextProbe
7    let $probe :=
8      if (exists($callback)) then
9        concat($truncated, for $a in $next return kwic:callback($callback, $nextProbe, "after"))
10      else concat($truncated, ' ', $nextProbe)
11   return
12     if (string-length($probe) gt $width) then
13       let $norm := concat(' ', normalize-space($probe))
14       return
15         if (string-length($norm) le $width and $next) then
16           kwic:truncate-following($root, $next, $norm, $width, $callback)
17         else if ($next) then
18           concat(substring($norm, 1, $width), '...')
19         else
20           $norm
21     else if ($next) then
22       kwic:truncate-following($root, $next, $probe, $width, $callback)
23     else for $str in normalize-space($probe)[.] return concat(' ', $str)
24  };

This version has one argument less: the $chars argument is abandoned, and the $max argument renamed to $width (line 1). Like the original version, this function starts by determining a next candidate text node ($nextProbe), by selecting the first following text() node. Next (line 4), the $next variable will check whether a $root argument was supplied, and whether it contains the candidate text node. If so, or if no $root argument was supplied, the $nextProbe node is copied; else it is emptied. Finally (line 7), the currently truncated text is updated by concatenating the previously truncated context ($truncated) with the current candidate node, in the $probe variable. When a $callback argument was supplied, the candidate node is first processed by this function.

Next, the string length of this concatenated string is tested (line 12). If it is smaller than the maximum context width ($width), while a following text() node exists (line 21), the kwic:truncate-following() function is called anew with the updated truncated context. If there’s no following text() node, the $probe string is returned (line 23). If, on the other hand, the $probe string length exceeds the maximum context width, a substring is returned that is exactly as long as the context width specified in $width (line 20); possibly followed with ‘…’ to indicate that there’s more following text that has been truncated (line 18). In order to make sure that whitespace (which is mostly irrelevant in XML) doesn’t eat all the context width, a further test compares the normalized value of this string to $width. If this normalization ends up smaller than $width, a next iteration of kwic:truncate-following() is called (line 16).

The kwic:truncate-previous() function is analogous, but obviously constructs the $truncated string in the opposite direction. Note how the correct substring length can be determined by subtracting the $width from the candidate truncated context ($probe):

concat('...', substring($norm, string-length($norm) - $width + 1))

This design of the truncation functions tries to minimize the processing, by:

• minimizing the amount of tests and reducing redundancy (only 2 branching levels instead of 3 in the original KWIC module)
• isolating the functions to be performed on the truncated text in the relevant branches of the decision tree

One costly operation is the determination whether the candidate text node still belongs to the $root cutoff node (line 5). This involves scanning all ancestors of the text node and checking whether the $root node is among them:

if ($root[not(. intersect $nextProbe/ancestor::*)]) then () 

(Note how the intersect operator is used, instead of the lazy evaluation $root//$hits in the original version of this function, which can be considered a bug). This has to be performed for each new text() node the truncation functions iterate over, which quickly amounts when the number of search hits is and/or the context $width are large. This is another area for improvement of the original KWIC functions, that expect a mandatory root argument. This means that even for simple kwic:summarize() calls, a $root node is determined, namely the current node containing the search hits. Actually, this is redundant: when those nodes are expanded with util:expand(), they are returned as root nodes anyway (without the wider document context), containing a copy of their internal structure, injected with <exist:match> elements around the matching text fragments. Having a $root argument in the truncation functions is more expensive than not having it (since its presence triggers the ancestor lookup for the current text node), so it is unwise, performance-wise, to pass it by default (when the default use case for a KWIC display probably won’t need it anyway).

That’s why the truncation functions can be tuned further by making the $root argument optional (see the “$root as node()?” definition in line 1 above). In order to propagate this optional $root parameter to the higher-level functions, those have to be adapted as well. In the kwic:get-summary() function, this simply requires declaring the $root argument as “node()?” (the question mark indicating it can be empty). For the kwic:summarize() function it then suffices to just pass an empty node as the $root argument in its call to the kwic:get-summary() function. Following overview highlights the changes from the original functions in yellow:

declare function kwic:get-summary($root as node()?, $node as element(exist:match),
  $config as element(config)) as element()
{
  kwic:get-summary($root, $node, $config, ())
};

declare function kwic:get-summary($root as node()?, $node as element(exist:match),
  $config as element(config), $callback as function?) as element()
{
  let $width := xs:int($config/@width)
  let $format := $config/@format
  let $ps := $config/@preserve-space = ('yes', 'true')

  let $prevTrunc := if ($ps) then kwic:truncate-previous-ps($root, $node, (), $width, $callback)
    else kwic:truncate-previous($root, $node, (), $width, $callback)
  let $followingTrunc := if ($ps) then kwic:truncate-following-ps($root, $node, (), $width, $callback)
    else kwic:truncate-following($root, $node, (), $width, $callback)
  return
    if ($format eq 'p') then
      <p>
        <span class="previous">{$prevTrunc}</span>
        {
          if ($config/@link) then
            <a class="hi" href="{$config/@link}">{ $node/text() }</a>
          else
            <span class="hi">{ $node/text() }</span>
        }
        <span class="following">{$followingTrunc}</span>
      </p>
    else if ($format eq 'table') then
      <tr>
        <td class="previous">{$prevTrunc}</td>
        <td class="hi">
        {
          if ($config/@link) then
            <a href="{$config/@link}">{$node/text()}</a>
          else
            $node/text()
        }
        </td>
        <td class="following">{$followingTrunc}</td>
      </tr>
    else
      <KWIC xmlns="http://exist-db.org/xquery/kwic">
        <prev>{$prevTrunc}</prev>
        <hit>{$node/text()}</hit>
        <next>{$followingTrunc}</next>
      </KWIC>
};

declare function kwic:summarize($hit as element(),
  $config as element(config), $callback as function?) as element()*
{
  let $expanded := util:expand($hit, "expand-xincludes=no")
  for $match in $expanded//exist:match
  return
    kwic:get-summary((), $match, $config, $callback)
};

2. An improved KWIC display module

The strategies discussed above have been implemented in an updated KWIC module that you can download and import in any XQuery. This updated KWIC display module provides exactly the same functionality as the original KWIC module, with improvements on following points:

• performance: for large node sets, this version of the function performs 30 to 35% faster (tested for very broad searches on 2 different test collections (non-public), for a number of search scenarios: varying context widths, varying $root nodes)
• accuracy: this version of the function normalizes the whitespace in the left and right text context

Of course, the truncation functions could be simplified even further by avoiding whitespace normalization (which reduces both the number of operations and decisions when truncating the text contexts). While this definitely can gain a couple of seconds on large node sets with large context widths, I feel strongly for the whitespace normalization, since it does a slightly more accurate job. But then, there’s no reason why the simplified truncation functions couldn’t be included in the KWIC module as well, and have the kwic:get-summary() function decide what version to call based on an extra configuration parameter. In the updated KWIC module, the left and right text contexts are normalized by default. This can be overridden by passing the value ‘yes’ (or ‘true’) to an extra attribute @preserve-space on the <config> element:

kwic:summarize(., <config width=”40” preserve-space=”yes”/>)

kwic:get-summary((), ., <config width=”40” preserve-space=”true”/>, ())

Another (cosmetic) change concerns the output formatting. The original KWIC module directly outputs HTML fragments, either as paragraph or table. Probably due to my biased text encoding background, this seems too display-oriented to my taste. In the updated KWIC module, a third (default) output display is provided, structured as follows:

<KWIC xmlns="http://exist-db.org/xquery/kwic">
  <prev>{$prevTrunc}</prev>
  <hit>{$node/text()}</hit>
  <next>{$followingTrunc}</next>
</KWIC>

This format is taken as the default one, while both other formats can be output by providing appropriate values for the @format attribute on <config>: ‘p’ for paragraphs, ‘table’ for tables.

To wrap up, here is an overview of the configuration options in the updated KWIC module:

• width: a number indicating the context width
• link: an URL to which the hit is linked
• format: output format
  • p: an HTML paragraph, containing <span class=”previous”>, <span class=”hi”>, and <span class=”following”>
  • table: HTML table rows, containing <td class=”previous>, <td class=”hi”>, and <td class=”following”>
  • KWIC (default): a <KWIC> element, containing <prev>, <hit>, and <next> elements
• preserve-space: whitespace normalization
  • yes|true: preserve original whitespace inside left and right text contexts
  • no (default): normalize whitespace inside left and right text contexts

These improvements open the way for more powerful exploitation of the KWIC results, as will be illustrated in the next section.

3. Processing KWIC results

3.1 Contextual Sorting

A more performant KWIC module eases further processing of the KWIC search results, beyond mere display functionality. Traditionally, applications offering KWIC display provide the option to sort the search results along the left and right text context. In order to do so in XQuery, the entire search result set must be collected and formatted as KWIC results first. Next, the left and right text contexts can be used to add sort keys. Let’s take following example, based on the ‘Keyword In Context with Callback’ example in the eXist Sandbox application:

import module namespace kwic="http://exist-db.org/xquery/kwic"
  at "xmldb:exist:///db/modules/kwic.xql";

declare function local:filter($node as node(), $mode as xs:string) as xs:string? {
  if ($node/parent::SPEAKER or $node/parent::STAGEDIR) then
      ()
  else if ($mode eq 'before') then
      concat($node, ' ')
  else
      concat(' ', $node)
};

let $config := <config width="80" />
for $hit in doc("/db/shakespeare/plays/hamlet.xml")//SPEECH[ft:query(., "nature king sir")]
order by ft:score($hit) descending
return
  kwic:summarize($hit, $config,
            util:function(xs:QName("local:filter"), 2))

For all <SPEECH> elements in the Shakespeare plays that contain either “nature”, “king”, or “sir”, this query returns all of these search hits, together with their left and right text context. Since no additional configuration options were passed besides @width, the whitespace in these results is normalized, and they are presented as <kwic:KWIC> chunks:

<KWIC xmlns="http://exist-db.org/xquery/kwic">
    <prev>The </prev>
    <hit>king</hit>
    <next> , sir ,--</next>
</KWIC>
<KWIC xmlns="http://exist-db.org/xquery/kwic">
    <prev>The king , </prev>
    <hit>sir</hit>
    <next> ,--</next>
</KWIC>
<KWIC xmlns="http://exist-db.org/xquery/kwic">
    <prev>To this effect, sir ; after what flourish your </prev>
    <hit>nature</hit>
    <next> will.</next>
</KWIC>
<!-- ... -->

Note how the “order by” expression specifies that these results are ordered per <SPEECH> element, in decreasing relevance order (determined by ft:score()).

Let’s refactor this query, so it will sort the results along the keyword:

import module namespace kwic="http://exist-db.org/xquery/kwic"
  at "xmldb:exist:///db/modules/kwic.xql";

declare function local:filter($node as node(), $mode as xs:string) as xs:string? {
  if ($node/parent::SPEAKER or $node/parent::STAGEDIR) then
      ()
  else if ($mode eq 'before') then
      concat($node, ' ')
  else
      concat(' ', $node)
};

let $config := <config width="80" />
let $hits := doc("/db/shakespeare/plays/hamlet.xml")//SPEECH[ft:query(., "nature king sir")]
for $KWIC in $hits/kwic:summarize(., $config,
            util:function(xs:QName("local:filter"), 2))
order by $KWIC/kwic:hit ascending
return $KWIC

This time, the KWIC results are collected in their own variable $KWIC, and sorted on their <kwic:hit> elements. This will put the hits for ‘king’ first, followed by the hits for ‘nature’, and ‘sir’.

Suppose we’re interested in the words following those hits. It’s just a matter of adding a second sort key to regroup them. We just need to replace the “order by” line with the following one:

order by lower-case($KWIC/kwic:hit) ascending,
         lower-case(replace($KWIC/kwic:next, '\W', ''))

Since sorting is case sensitive, the order by expression above transforms all sort keys to lower case, and ignores all non-word characters in the right text context.

<!-- ... -->
<KWIC xmlns="http://exist-db.org/xquery/kwic">
    <prev>A man may fish with the worm that hath eat of a </prev>
    <hit>king</hit>
    <next> , and cat of the fish that hath fed of that worm.</next>
</KWIC>
<KWIC xmlns="http://exist-db.org/xquery/kwic">
    <prev>...ood liege, I hold my duty, as I hold my soul, Both to my God and to my gracious </prev>
    <hit>king</hit>
    <next> : And I do think, or else this brain of mine Hunts not the trail of policy so s...</next>
</KWIC>
<KWIC xmlns="http://exist-db.org/xquery/kwic">
    <prev>A bloody deed! almost as bad, good mother, As kill a </prev>
    <hit>king</hit>
    <next> , and marry with his brother.</next>
</KWIC>
<KWIC xmlns="http://exist-db.org/xquery/kwic">
    <prev>The </prev>
    <hit>king</hit>
    <next> and queen and all are coming down.</next>
</KWIC>
<!-- ... -->

For sorting on the left text context, some additional processing is required: we’re interested in the words immediately preceding the search term; not in the first word of the left text context. Therefore, the left text context has to be reversed before sorting. This can be achieved by first tokenizing the string, reversing this sequence with the XQuery reverse() function, and reassembling this sequence to a string with the XQuery string-join() function:

order by lower-case($KWIC/kwic:hit) ascending,
         lower-case(string-join(reverse(tokenize($KWIC/kwic:prev, '\W+')), ' ')) 

Likewise, instead of the entire left or right text, individual words at a certain position can be used for sorting. For example, suppose we want to sort the results on the third word preceding the search term (in descending order), and then on the second one following it (ascending). This can be achieved by the XQuery tokenize() function:

order by lower-case($KWIC/kwic:hit) ascending,
         lower-case(reverse(tokenize($KWIC/kwic:prev, '\W+')[.])[3]) descending,
         lower-case(tokenize($KWIC/kwic:next, '\W+')[.][2])

<!-- ... -->
<KWIC xmlns="http://exist-db.org/xquery/kwic">
    <prev>He that plays the </prev>
    <hit>king</hit>
    <next> shall be welcome; his majesty shall have tribute of me; the adventurous knight ...</next>
</KWIC>
<KWIC xmlns="http://exist-db.org/xquery/kwic">
    <prev>Of all the days i' the year, I came to't that day that our last </prev>
    <hit>king</hit>
    <next> Hamlet overcame Fortinbras.</next>
</KWIC>
<KWIC xmlns="http://exist-db.org/xquery/kwic">
    <prev>...es, to be demanded of a sponge! what replication should be made by the son of a </prev>
    <hit>king</hit>
    <next> ?</next>
</KWIC>
<KWIC xmlns="http://exist-db.org/xquery/kwic">
    <prev>That can I; At least, the whisper goes so. Our last </prev>
    <hit>king</hit>
    <next> , Whose image even but now appear'd to us, Was, as you know, by Fortinbras of N...</next>
</KWIC>
<!-- ... -->

Of course, while these sorting examples are still fairly basic, they could be useful already for linguistic exploration of documents indexed with eXist. A full linguistic tool kit would require more advanced features for deriving collocation information (what words occur with other words?), and statistics. That’s probably future music, but it sets one dreaming…

3.2 An (Experimental) Collocation Table

As an experiment, let’s see how far we get with a collocation table (caution: highly experimental!). Starting from KWIC search results, it is possible to compose a limited window of n words preceding and following the search term. When for each of these context slots all distinct words occurring at that slot are collected, a table can be composed listing all of these words-per-position. Let’s examine the steps in constructing such a collocation table, starting from KWIC search results (the full XQuery code can be downloaded here) :

import module namespace kwic="http://exist-db.org/xquery/kwic" at "xmldb:exist:///db/modules/kwic.xql";

declare function local:filter($node as node(), $mode as xs:string) as xs:string? {
  if ($node/parent::SPEAKER or $node/parent::STAGEDIR) then
      ()
  else if ($mode eq 'before') then
      concat($node, ' ')
  else
      concat(' ', $node)
};

(: context scope: number of preceding / following words :)
let $scope := 5
(: determine context width for KWIC results: 10 characters per context word, minimally 40 :)
let $cutoff := max(($scope * 10, xs:int(40)))
let $config := <config width="{$cutoff}" />
let $hits := doc("/db/shakespeare/plays/hamlet.xml")//SPEECH[ft:query(., "nature king sir")]
let $KWIC := $hits/kwic:summarize(., $config,
            util:function(xs:QName("local:filter"), 2))

This will produce a $KWIC variable with the search results formatted in a KWIC display (note: unordered, this time, since the results will be ordered later on):

<KWIC xmlns="http://exist-db.org/xquery/kwic">
    <prev>Long live the </prev>
    <hit>king</hit>
    <next> !</next>
</KWIC>
<KWIC xmlns="http://exist-db.org/xquery/kwic">
    <prev>In the same figure, like the </prev>
    <hit>king</hit>
    <next> that's dead.</next>
</KWIC>
<KWIC xmlns="http://exist-db.org/xquery/kwic">
    <prev>Looks it not like the </prev>
    <hit>king</hit>
    <next> ? mark it, Horatio.</next>
</KWIC>
<!-- ... -->

Further processing is split per distinct search term, in order to keep their collocations nicely separated. In a next step, both left and right contexts are prepared for further processing:

(: split up collocations per search term :)
for $term in distinct-values($KWIC//kwic:hit/lower-case(normalize-space(.)))
order by $term
return
  (: prepare entire left / right contexts for tokenization:
        -lower case
        -normalize whitespace
        -reverse $prev
  :)
  let $prev :=
    for $a in $KWIC[kwic:hit/lower-case(normalize-space(.)) eq $term]/kwic:prev/lower-case(normalize-space(.))
    return string-join(reverse(tokenize($a, '\W+')), ' ')
  let $next := $KWIC[kwic:hit/lower-case(normalize-space(.)) eq $term]//kwic:next/lower-case(normalize-space(.))

Apart from being transformed to lower case and whitespace-normalized, the left context is reversed, and after this stage looks as follows:

the live long
laertes king be shall laertes clouds the to t
be shall laertes clouds the to it applaud gues
...

Next, a selection of these $prev and $next contexts is made, cutting them off at the number of words preceding or following the search term as defined in the $scope variable at the start:

  (: per context position, retrieve all distinct words :)
  (: note: discard numbers :)
  let $prev :=
    for $context in ((0 - $scope) to -1)
    let $words :=
      let $tok :=
        for $a in $prev
        return tokenize($a, '\W+')[matches(., '[a-zA-Z]')][position() = abs($context)]
      for $b in distinct-values($tok) order by $b return <w>{$b}</w>
    return <context pos="{$context}">{$words}</context>
  let $next :=
    for $context in (1 to $scope)
    let $words :=
      let $tok :=
        for $a in $next
        return tokenize($a, '\W+')[matches(., '[a-zA-Z]')][$context]
      for $b in distinct-values($tok) order by $b return <w>{$b}</w>
    return <context pos="{$context}">{$words}</context>

For example, the left context is broken into the 5 distinct words that occur before the search term, starting from the 5th up to the last word before the search term. This will produce an updated $prev variable that looks as follows:

<context pos="-5">
    <w>an</w>
    <w>and</w>
    <w>battlements</w>
    <!-- ... -->
</context>
<context pos="-4">
    <w>and</w>
    <w>aside</w>
    <w>body</w>
    <!-- ... -->
</context>
<context pos="-3">
    <w>alone</w>
    <w>as</w>
    <w>be</w>
    <!-- ... -->
</context>
<context pos="-2">
    <w>a</w>
    <w>and</w>
    <w>before</w>
    <!-- ... -->
</context>
<context pos="-1">
    <w>a</w>
    <w>be</w>
    <w>bloat</w>
    <!-- ... -->
</context>

Finally, after these $prev and $next results have been collected, they are presented in an HTML table, where each position in the left and right contexts is represented in a column. All words occurring in that position are then presented in an own row.

  let $max := max(($next|$prev)/count(w))

  (: spread out words-per-context over table rows :)
  return
    <table border="1">{
      <tr>
        <th/>
        {
          for $a in $prev
          return <th>{$a/@pos/string()}</th>,
          <th>term</th>,
          for $a in $next
          return <th>{$a/@pos/string()}</th>
        }
      </tr>,
      for $i in (1 to $max)
      return
        <tr>{
          <td>{$i}</td>,
          for $a in $prev
          return
            <td>{$a/w[$i]/text()}</td>,
          <th>{$term}</th>,
          for $a in $next
          return
            <td>{$a/w[$i]/text()}</td>
        }</tr>
    }</table>

This produces following collocation tables (split per search term) for our query (note: this is only a summary; a full version can be found here):

Note how the value of these collocation tables is fairly limited, though. Since the context words have been ordered alphabetically, all these tables can provide is an alphabetical list of words occurring per position before or after the search term (when read per column). It would of course be more interesting to have access to statistical information that can indicate the significance of the individual words per position. That would allow to order them by statistical saliency, so the tables could provide an overview of the most frequent words at their respective positions. Unfortunately, as such data is not (yet?) available in eXist, the meaningfulness of the collocation data presented above is quite questionable, whence the experimental character of this illustration. Anyway, if you’d like to experiment with it, feel free to download the updated KWIC module (kwic.xql), the full XQuery script (collocation_table.xq) and  sample output (collocationSample.htm).

Instead, this post will focus on a topic I saw myself confronted with when developing an application (i.e. a web interface) for such an edition: how do you search within such ‘multiversion’ texts? Most probably, users of the edition would want to focus on one (or a selection of) text version(s). Of course, when version 1 contains the word ‘hope’, which in version 2 had been changed to ‘despair’, (only) the right readings should be retrieved for the respective text version.

Enter XQuery and the eXist XML database. As with the specific background of scholarly editing, a decent familiarity with the basic concepts of indexing and searching documents with eXist will be assumed. The focus will lie on theoretical aspects of two key problems: 1) full-text searching and 2) index retrieval of terms in ‘multiversion’ texts. Two possible approaches are discussed and evaluated:

1. Querying a single ‘multiversion’ source text
2. Splitting up the text in separate source texts per text version and querying those

Note: though this is quite a technical discussion, you can download appSearch_db.zip, an eXist backup file containing all files discussed here. They can be installed in one click in an eXist database by choosing to restore this backup file.

1. Single source text

I’ve started exploring this approach from the desire to do something clever with a ‘multiversion’ text containing all variants occurring in all text versions included in the edition. It starts from a single indexed source text containing transcriptions for all distinct text versions. For example, let’s take a 2-paragraph text for which 4 text versions have been collated (which will be indexed as test.xml in a dedicated collection in the eXist database, namely /db/test):

<div xmlns="http://www.tei-c.org/ns/1.0">
  <listWit>
    <witness xml:id="w1"/>
    <witness xml:id="w2"/>
    <witness xml:id="w3"/>
    <witness xml:id="w4"/>
  </listWit>
  <p>This is a paragraph with common text.</p>
  <p>This paragraph has
    <app>
      <rdg wit="#w1 #w3">variants</rdg>
      <rdg wit="#w2">variant text</rdg>
      <rdg wit="#w4">variant test</rdg>
    </app>.
  </p>
</div>

This ‘single source’ approach requires some consideration w.r.t. index configuration, and search and index lookup scripts. Assume the <p> elements will be the text unit we’re interested in for querying this document. Since the <rdg> elements containing text variants are contained by <p>, there’s no way to exclude the contents of irrelevant <rdg>s from the search space. Without precautions, hence, fulltext searches on paragraphs will always include text variants from all text versions enclosed in <app> elements. In order to be able to address the contents of their embedded <rdg> elements separately, both should be indexed separately. This can be specified in a dedicated configuration file at /db/system/conf/db/test/collection.xconf:

<collection xmlns="http://exist-db.org/collection-config/1.0">
    <validation mode="auto"/>
    <index xmlns:tei="http://www.tei-c.org/ns/1.0">
        <fulltext default="none" attributes="no"/>
        <lucene>
            <text qname="tei:p">
                <ignore qname="tei:rdg"/>
            </text>
            <text qname="tei:rdg"/>
        </lucene>
        <create qname="@wit" type="xs:string"/>
    </index>
</collection>

This index definition excludes the content of <rdg> elements from searches focusing on <p>, like //tei:p[ft:query(., ‘test’)], while still allowing to query them directly with queries like //tei:rdg[ft:query(., ‘test’)].

1.1 Search script

The main concern for the search script is to separate the search contexts with common text (shared among all text versions) from version-specific search contexts (represented as <rdg> elements within <app>). Following XQuery script illustrates a possible approach:

declare namespace tei="http://www.tei-c.org/ns/1.0";
let $docs := doc('/db/test/test.xml')
let $rdgs :=
  for $rdg in $docs//tei:listWit//tei:witness/@xml:id
  return concat('#', $rdg)
let $pool := $docs//tei:p
let $query := 'text'
return $pool[ft:query((.|.//tei:rdg[tokenize(string(@wit), '\s+') = $rdgs]), $query)]

Note, how in this script the $rdgs variable is populated with the sigla of all text versions that have been defined in the @xml:id attributes of the different <witness> elements in test.xml. This might seem to undermine the exact point of wanting to address the text variants separately, but is used here to illustrate the maximal possible query scenario (searching in all text versions). Narrower selections are possible, of course, and can be mimicked by replacing it with e.g. $rdgs := (‘#1’, ‘#4’).

In this query, first the nodes to be queried are collected in a $pool variable (in this case, all <p> nodes inside test.xml). Note, how for real-life documents, this set of nodes could be expanded with all desired text elements one would want to query. The actual differentiation lies in the search expression, which performs a ft:query() full-text search on both this set of nodes and any embedded <rdg> elements whose @wit attributes contain references to (one of) the text version(s) selected for the search (by checking whether any of the tokens for the tokenized @wit attribute value equals any of the text versions defined in the $rdgs variable).

Now, above query probably won’t yield much useful results: while all matching paragraphs are retrieved, they are presented as a bag of undifferentiated nodes without any duplicates. In other words: from these results it’s impossible to tell what search result occurs in what text version. This can be improved in the next version:

declare namespace tei="http://www.tei-c.org/ns/1.0";
let $docs := doc('/db/test/test.xml')
let $rdgs := for $rdg in $docs//tei:listWit//tei:witness/@xml:id return concat('#', $rdg)
let $pool := $docs//tei:p
let $query := 'text'

for $rdg in $rdgs
let $results := $pool[ft:query((.|.//tei:rdg[tokenize(string(@wit), '\s+') = $rdg]), $query)]
for $result in $results
return <result wit="{$rdg}">{$result}</result>

By looping over the selected text versions defined in $rdgs, this script will repeat the full-text search for each text version and present all results per version, while identifying the associated text version by its siglum in a @wit attribute on the <result> element.

1.2 Index lookup script

The same differentiation technique can be applied in a script for lookup of indexed terms in a (selection of) text version(s):

declare namespace tei="http://www.tei-c.org/ns/1.0";
declare function local:term-callback($term, $data) {
  <term freq="{$data[1]}" docs="{$data[2]}" n="{$data[3]}">{$term}</term>
};

let $callback := util:function(xs:QName('local:term-callback'), 2)
let $docs := doc('/db/test/test.xml')
let $rdgs := for $rdg in $docs//tei:listWit//tei:witness/@xml:id return concat('#', $rdg)
let $pool := $docs//tei:p
let $query := 'text'
let $nodes :=
  for $a in $pool|$pool//tei:rdg[tokenize(string(@wit), '\s+') = $rdgs]
  (: here nodes can be refined first by querying, if required :)
  return $a(:[ft:query(., $query)]:)
for $term in util:index-keys($nodes, '', $callback, 15000, 'lucene-index')
(:order by $term/@freq/number() descending:)
return $term

This script first selects the paragraphs and (only) relevant <rdg> elements in the $nodes variable, using the technique sketched above. (Note how the comments are provided as hooks for further refinement of search contexts and ordering of the search results. ) Next, those selected nodes are passed to the eXist-specific util:index-keys() function that will collect their different index terms in the $terms variable. When inspecting the results of the script above, some things catch the eye:

<term freq="1" docs="1" n="1">common</term>
<term freq="1" docs="1" n="2">has</term>
<term freq="2" docs="1" n="3">paragraph</term>
<term freq="1" docs="1" n="4">test</term>
<term freq="2" docs="1" n="5">text</term>
<term freq="2" docs="1" n="6">variant</term>
<term freq="1" docs="1" n="7">variants</term>

While this script does a decent job w.r.t. separating the common from the version-specific search contexts (try with separating out some sigla and compare the results), there is a problem with the statistics:

• term frequency: all index terms in common text (i.e. text shared between all text versions) are counted only once

• document count: the number of documents in which the terms occur is always 1 (as only one indexed document is queried)

Instead, the frequencies and document count of terms in common text should be multiplied with the number of text versions being queried. For example, the word ‘common’ occurs once in all text versions and hence should amount to 4 occurrences in 4 documents. Yet, it’s less straightforward for index terms occurring both in common and version-specific text: the term ‘text’ occurs once within common text, and once within version ‘w2’, totaling 5 (4 + 1) occurrences in 4 documents. When statistics matter, this differentiation between the occurrences in the selected text versions should be accounted for in the index lookup script. This can be worked around (theoretically) by first generating separate index lists per version, and later adding all separate statistics:

declare namespace tei="http://www.tei-c.org/ns/1.0";
declare function local:term-callback($term, $data) {
  <term freq="{$data[1]}" docs="{$data[2]}">{$term}</term>
};

let $callback := util:function(xs:QName('local:term-callback'), 2)
let $docs := doc('/db/test/test.xml')
let $rdgs := for $rdg in $docs//tei:listWit//tei:witness/@xml:id return concat('#', $rdg)
let $pool := $docs//tei:p
let $query := 'text'

let $terms :=
  for $rdg in $rdgs
  let $nodes :=
    for $a in $pool|$pool//tei:rdg[tokenize(string(@wit), '\s+') = $rdg]
    (: here nodes can be refined first by querying, if required :)
    return $a(:[ft:query(., $query)]:)
  return
    for $term in util:index-keys($nodes, '', $callback, 15000, 'lucene-index')
    return <term wit="{$rdg}">{
      $term/(@*, node())
    }</term>

let $conflateTerms :=
  for $term in distinct-values($terms)
  let $groupTerms := $terms[. eq $term]
  return <term>
  {
    for $att in $groupTerms[1]/@*[name() != 'wit']
    return attribute {$att/name()} {sum($groupTerms/@*[name() eq $att/name()])}
  }
  {$term}
  </term>

for $a in $conflateTerms
(:order by $a/@freq/number() descending:)
return $a

In this script, the $terms variable first collects all index terms  per selected text version, by looping over the $rdgs variable and performing an index lookup on the nodes occurring in that text version. A next step conflates the distinct terms in the $conflateTerms variable, by grouping the unique terms in $terms and accumulating the stats for their respective occurrences. This produces the correct statistics:

<term freq="4" docs="4">common</term>
<term freq="4" docs="4">has</term>
<term freq="8" docs="4">paragraph</term>
<term freq="5" docs="4">text</term>
<term freq="2" docs="2">variants</term>
<term freq="2" docs="2">variant</term>
<term freq="1" docs="1">test</term>

Due to the lack of a dedicated grouping mechanism in XQuery 1.0 (which should be addressed with the ‘group by’ clause in the upcoming XQuery 3.0 specification), the distinct-values() route used in the $conflateTerms variable is the only way to achieve this grouping, without resorting to the undocumented ‘group by‘ extension in eXist). This doesn’t scale well on large documents with many versions.

One way of speeding this up a bit, is by delegating the conflation of the separate index terms to XSLT, by using eXist’s transform:transform() function. Since version 2.0, native grouping capabilities have been added to XSLT, which definitely outperform the distinct-values() approach with XQuery. Hence, the $conflateTerms variable could instead be computed via XSLT:

let $conflateXSLT :=
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:xs="http://www.w3.org/2001/XMLSchema"
    exclude-result-prefixes="xs"
    version="2.0">
    <xsl:template match="terms">
        <xsl:for-each-group select="term" group-by=".">
            <term>
                <xsl:for-each select="@*[name() != 'wit']">
                    <xsl:sort select="."/>
                    <xsl:variable name="attName" select="name()"/>
                    <xsl:attribute name="{{$attName}}">
                        <xsl:value-of select="sum(current-group()//@*[name() = $attName])"/>
                    </xsl:attribute>
                </xsl:for-each>
                <xsl:value-of select="current-grouping-key()"/>
            </term>
        </xsl:for-each-group>
    </xsl:template>
</xsl:stylesheet>

let $conflateTerms := transform:transform(<terms>{$terms}</terms>, $conflateXSLT, ()

Still, this approach remains very fragile when it comes to performance: the main bottleneck is the requirement that full index lookups must be repeated for all selected text versions. Next, those (possibly huge) term collections must be ordered and their frequencies added. Performance of this approach hence is entirely dependent on:

• the number of text versions selected for the search
• the size of the node set to be searched

Again, further optimisation is possible, by restricting the number of index lookups as much as possible. This can be achieved by splitting up the previous $terms variable into two variables, containing only the common index terms and only the version-specific index terms, respectively:

declare namespace tei="http://www.tei-c.org/ns/1.0";
declare function local:term-callback($term, $data) {
 <term freq="{$data[1]}" docs="{$data[2]}">{$term}</term>
};

let $callback := util:function(xs:QName('local:term-callback'), 2)
let $docs := doc('/db/test/test.xml')
let $rdgs := for $rdg in $docs//tei:listWit//tei:witness/@xml:id return concat('#', $rdg)
let $pool := $docs//tei:p
let $query := 'text'

let $commonTerms :=
  let $nodes :=
    for $a in $pool
    (: here nodes can be refined first by querying, if required :)
    return $a(:[ft:query(., $query)]:)
  return
    for $term in util:index-keys($nodes, '', $callback, 15000, 'lucene-index')
    return <term>{
      (
      for $att in $term/@*
      return attribute {$att/name()} {$att * count($rdgs)}
      ,
      $term/node()
      )
    }</term>

let $rdgTerms :=
  for $rdg in $rdgs
  let $nodes :=
    for $a in $pool//tei:rdg[tokenize(string(@wit), '\s+') = $rdg]
    (: here nodes can be refined first by querying, if required :)
    return $a(:[ft:query(., $query)]:)
  return
    for $term in util:index-keys($nodes, '', $callback, 15000, 'lucene-index')
    return <term wit="{$rdg}">{
      $term/(@*, node())
    }</term>

let $conflateXSLT :=
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:xs="http://www.w3.org/2001/XMLSchema"
    exclude-result-prefixes="xs"
    version="2.0">
    <xsl:template match="terms">
        <xsl:for-each-group select="term" group-by=".">
            <term>
                <xsl:for-each select="@*[name() != 'wit']">
                    <xsl:sort select="."/>
                    <xsl:variable name="attName" select="name()"/>
                    <xsl:attribute name="{{$attName}}">
                      <xsl:choose>
                        <xsl:when test="name() = 'docs'">
                          <xsl:value-of select="(current-group()[not(@wit)]//@*[name() = $attName],
                                                 sum(current-group()//@*[name() = $attName]))[1]"/>
                        </xsl:when>
                        <xsl:otherwise>
                          <xsl:value-of select="sum(current-group()//@*[name() = $attName])"/>
                        </xsl:otherwise>
                      </xsl:choose>
                    </xsl:attribute>
                </xsl:for-each>
                <xsl:value-of select="current-grouping-key()"/>
            </term>
        </xsl:for-each-group>
    </xsl:template>
</xsl:stylesheet>

let $conflateTerms := transform:transform(<terms>{$commonTerms, $rdgTerms}</terms>, $conflateXSLT, ())

for $a in $conflateTerms
(:order by $a/@freq/number() descending:)
return $a

The common terms (i.e. terms occurring outside of <rdg> elements, which can be assumed to be the majority of terms in a document) are now retrieved with a single util:index-keys() lookup within the $commonTerms variable. Note how the statistics are adjusted: since all of these terms occur in all selected text versions, their occurrences and document count numbers are multiplied with the number of text versions selected. This leaves the version-specific terms (i.e. those occurring within the relevant <rdg> elements), to be collected in the $rdgTerms variable. Again, the number of selected text versions determines the number of index lookups, but this time the node set on which the lookups are performed is substantially cut down to only the (relevant) <rdg> nodes. Consequently, the subsequent conflation of the index terms has to deal with much less nodes and gains in efficiency. Note, how the XSLT script had to be adapted for the computation of the total @docs metric: for terms occurring both within common and version-specific contexts, the total number of documents should be taken as cutoff point. This total is retrieved by selecting the @docs value of the <term> without a @wit attribute, when available. In human language: this prevents that such terms would be computed to occur in more documents than the number of text versions selected.

Still, in a maximal scenario where full index scans are performed for large texts with many versions, this strategy could be prohibitively expensive. Before evaluating it, let’s have a look at another alternative for querying ‘multiversion’ texts, in the next section.

2. Index all versions separately

This approach simplifies on-the-fly computation of the different text versions enclosed in a parallel-segmented TEI text, by separating out all distinct text versions into complete XML source texts in their own right, and indexing and querying those ‘single-version’ texts separately in eXist.

Splitting up a ‘multiversion’ text into distinct text versions can be done easily enough via a batch XSLT script prior to indexing those texts, but adds to the maintenance cost: if something changes to the ‘multiversion’ text, all derived versions should be updated as well, and indexed again. However, an appealing alternative may be found in eXist’s trigger facilities. Instead of prior batch processing, eXist can be made to automatically apply the XSLT transformation upon indexing of the ‘multiversion’ text.

2.1 Trigger setup

In order to configure triggers for the /db/test collection, a <trigger> section should be added to /db/system/config/db/test/collection.xconf:

<collection xmlns="http://exist-db.org/collection-config/1.0">
    <validation mode="auto"/>
    <index xmlns:tei="http://www.tei-c.org/ns/1.0">
        <fulltext default="none" attributes="no"/>
        <lucene>
            <text qname="tei:p">
                <ignore qname="tei:rdg"/>
            </text>
            <text qname="tei:rdg"/>
        </lucene>
    </index>

<!-- to be added to collection.xconf file of target collection -->
    <triggers>
        <trigger event="store,update" class="org.exist.collections.triggers.XQueryTrigger">
            <parameter name="url" value="xmldb:exist://localhost/db/test/triggers/splitRdgs.xql"/>
        </trigger>
    </triggers>
</collection>

This tells eXist to run the /db/test/triggers/splitRdgs.xql XQuery script for all documents that are added to or updated in the collection /db/test. Since this includes the subcollection /db/test/triggers that will hold the scripts needed for executing this trigger, it’s safest to prevent them from activating this trigger when they are stored or updated themselves. This can be done by adding an empty collection.xconf file in the subcollection /db/system/conf/db/test/triggers:

<collection xmlns="http://exist-db.org/collection-config/1.0">
    <index xmlns:tei="http://www.tei-c.org/ns/1.0"/>
</collection>

Now, let’s have a look at the XQuery script at /db/test/triggers/splitRdgs.xql (note: syntax is based on the trigger implementation in eXist-1.4.x, and subject to change after the rework of this implementation in the current development version):

declare namespace  xmldb="http://exist-db.org/xquery/xmldb";
declare namespace tei="http://www.tei-c.org/ns/1.0";

declare variable $local:triggerEvent external;
declare variable $local:eventType external;
declare variable $local:collectionName external;
declare variable $local:documentName external;
declare variable $local:document external;
declare variable $local:triggersLogFile := "triggersLog.xml";

(: create the log file if it does not exist :)
let $logfile :=
  if(not(doc-available($local:triggersLogFile)))then
	xmldb:store("/db", $local:triggersLogFile, <triggers/>)
  else()
let $doc := doc($local:documentName)
return
if ($local:eventType eq 'finish' and $doc//tei:rdg) then
  let $xsl := doc(concat($local:collectionName, '/triggers/splitRdgs.xsl'))
  let $rdgs := $doc//tei:witness/@xml:id
  for $rdg in $rdgs
  let $params :=
    <parameters>
      <param name="t" value="#{$rdg}"/>
    </parameters>
  let $transformDoc := transform:transform($doc, $xsl, $params)
  let $docName := replace(tokenize($local:documentName, '/')[last()],
                          '(.+)(\.[^.]+)$', concat('$1', '_', $rdg, '$2'))
  let $storeDoc :=  xmldb:store($local:collectionName, $docName, $transformDoc)
  return update
    insert
      <trigger event="{$local:triggerEvent}" eventType="{$local:eventType}"
               collectionName="{$local:collectionName}" documentName="{$local:documentName}"
               timestamp="{current-dateTime()}">{$rdg}</trigger>
    into doc("/db/triggersLog.xml")/triggers
else ()

Basically, what this script does, is detect if the document being stored or updated contains any <rdg> elements. If so, it applies a splitting XSLT stylesheet per identified text version, and stores the results of that transformation as documents whose name is based on the ‘multiversion’ filename, suffixed with an underscore and the sigil of that version. The XSLT stylesheet referred to points towards /db/test/triggers/splitRdgs.xsl and could look as follows:

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:xs="http://www.w3.org/2001/XMLSchema"
    xmlns:tei="http://www.tei-c.org/ns/1.0"
    exclude-result-prefixes="#all"
    version="2.0">

    <xsl:param name="t"/>

    <xsl:template match="tei:app">
        <xsl:apply-templates select="*[tokenize(@wit, '\s+') = $t]/node()" />
    </xsl:template>                

    <xsl:template match="@*|node()">
        <xsl:copy>
            <xsl:apply-templates select="@*|node()"/>
        </xsl:copy>
    </xsl:template>

</xsl:stylesheet>

This effectively replaces all <app> elements with the content of the relevant text version, and copies all other content literally.

To summarise, this trigger setup will store the original document, as well as a separate text per text version, with following characteristics:

• document name: original document name suffixed with ‘_’ + sigil

• all <app> elements are removed; only content of <rdg> elements relevant to that version is preserved

2.2 Search script

The separation of all text versions in separate texts greatly alleviates the complexity of the search script. The only place where versions should be taken into account, is the determination of the file names to be included in the search:

declare namespace tei="http://www.tei-c.org/ns/1.0";
let $rdgs := for $a in doc('/db/test/test.xml')//tei:listWit//tei:witness/@xml:id return concat('#', $a)
let $docs := for $rdg in $rdgs return doc(concat('/db/test/test', replace($rdg, '#', '_'), '.xml'))
let $pool := $docs//tei:p
let $query := 'text'
for $hit in $pool[ft:query(., $query)]
return <result wit="{$hit/replace(substring-after(util:document-name(.), '_'), '\.[^.]+$', '')}">{
  $hit
}</result>

Note how the version sigla are used to determine the document name of the documents to be included in the search. Like its ‘multiversion’ counterpart discussed in section 1.2 above, this will return all search hits while identifying the text version in which they occur in a @wit attribute on the <result> element.

2.3 Index lookup script

Likewise, the index lookup script is much simplified, in that only a single call to util:index-keys() needs to be made, irrespective of the number of text versions selected:

declare namespace tei="http://www.tei-c.org/ns/1.0";
declare function local:term-callback($term, $data) {
  <term freq="{$data[1]}" docs="{$data[2]}" n="{$data[3]}">{$term}</term>
};

let $callback := util:function(xs:QName('local:term-callback'), 2)
let $rdgs := for $a in doc('/db/test/test.xml')//tei:listWit//tei:witness/@xml:id return concat('#', $a)
let $docs := for $rdg in $rdgs return doc(concat('/db/test/test', replace($rdg, '#', '_'), '.xml'))
let $pool := $docs//tei:p
let $query := 'text'
let $nodes :=
  for $a in $pool
  (: here nodes can be refined first by querying, if required :)
  return $a(:[ft:query(., $query)]:)
for $term in util:index-keys($nodes, '', $callback, 15000, 'lucene-index')
(:order by $term/@freq/number() descending:)
return $term

Note how no mention needs to be made to any <rdg> elements, since those have been filtered out at the indexing stage.

3. Evaluation

This exercise started from the theoretical desire to be able to search ‘multiversion’ TEI source texts out of the box. eXist’s indexing implementation allows for index definitions flexible enough to construct XQuery scripts that neatly cut their way through the different versions encoded within the single source text. While searching is quite performant, there’s an important performance bottleneck when it comes to index lookup, whose performance is heavily dependent on the number of text versions and the size of their (selected) node sets to be searched. Although there is room for optimisation, this approach clearly has its limits and could hardly be defendable in a maximal scenario, where a complete index scan is requested for all versions of a real-life text (say, to produce a frequency list). Yet, it could be considered an option for more limited scenarios:

• when the scope of the index lookup is restricted by requiring one or more start letters, this greatly improves performance
• when statistics are not important (in –say- an autocomplete scenario), the initial index lookup script performs quite well

On the other hand, splitting up a ‘multiversion’ text into its constituent text versions at first sight looked unwieldy from a maintenance perspective. Yet, eXist’s trigger implementation relieves this burden. It is beyond doubt that this approach greatly alleviates the complexity of search and index lookup scripts, and has superior performance. After all, the util:index-keys() function is there precisely for efficiently collecting index statistics; generating partial lists of index terms and conflating them via XQuery, while feasible, has its price performance-wise.

Download: appSearch_db.zip, an eXist backup file containing all files discussed here. Just download and restore this backup file in your eXist database.

[UPDATE 2011-01-05]: The XSLT stylesheet has been extended with missing features:

• [feature]: added @trace handling
• [feature]: added <xpath> handling
• [feature]: added <store-files> handling
• [feature]: added context handling for util:eval()
• [fix]: <![CDATA[ ]]> in output: spaces required…

[UPDATE 2010-12-09]: The XSLT stylesheet has been substantially reworked, to produce

• more legible XQuery code
• more reliable XQuery code, taking into account serialization options, and deriving the most sensible highlight-matches settings where necessary

Currently, I’m heavily porting old XQuery code to the latest version of the eXist XML database’s new Lucene FT index and search capabilities. In doing so, I’m hitting a couple of bugs in this area, that I’m trying to isolate, test and report as clearly as possible. This post discusses a means to use the same test files for both eXist-1.4 and eXist-trunk.

The addition of the XQuery Unit testing framework in eXist-trunk code has proven to be a tremendously great help in developing test cases. On the other hand, at the time of writing this post, this functionality was not yet available in eXist-1.4, making it cumbersome to (manually) set up a testing environment, executing different tests and compare their results with expected results. Still, I wanted to be able to compare the same test queries in both versions, and see where differences are (and in my case, help deciding on which version to continue with, as my development schedule is tightening up).

In order to do so, I’ve created an XSLT stylesheet UnitTest2XQuery.xsl that converts XQuery Unit test XML files to XQuery code which is natively executable in eXist-1.4. It should generate identical reports to those of the XQuery Unit testing framework in eXist-trunk, making it easy to quickly compare results of both eXist versions for identical queries.

I’m not going into detail on the stylesheet here (nothing fancy, really), but will add some remarks on how to use it. Ideally, this stylesheet could be stored in the db (say, as ‘/db/UnitTest2XQuery.xsl’) and put to work as follows:

let $test :=
<TestSet>
    <testName>really simple test</testName>
    <description>
        <p>just a simple test, really</p>
        <author>Ron Van den Branden</author>
    </description>
    <setup>
        <create-collection parent="/db" name="coll"/>
        <store collection="/db/coll" name="test.xml">
            <p att1="val1" att2="val2">this is a test document</p>
        </store>
    </setup>
    <functions><![CDATA[
        declare function local:echo($node) {
        <echo>{$node/node()}</echo>
        };
        ]]></functions>
    <tearDown>
        <remove-collection collection="/db/coll"/>
    </tearDown>
    <test output="text">
        <task>really simple test, text</task>
        <code><![CDATA[
            local:echo(collection('/db/coll')//p)/string()
            ]]></code>
        <expected>this is a test document</expected>
    </test>
    <test output="xml">
        <task>really simple test, xml</task>
        <code><![CDATA[
            local:echo(collection('/db/coll')//p)
            ]]></code>
        <expected><echo>this is a test document</echo></expected>
    </test>
</TestSet>

return util:eval(transform:transform($test, doc('/db/UnitTest2XQuery.xsl'), ()))

Unfortunately, eXist’s transform:transform() function does ugly things to text that has been processed with disable-output-escaping. Instead of properly escaped text, the unescaped text is copied, wrapped in processing instructions:

<?javax.xml.transform.disable-output-escaping?>

        declare function local:echo($node) {
        &lt;echo&gt;{$node/node()}&lt;/echo&gt;
        };

<?javax.xml.transform.enable-output-escaping?>

Clearly, this can’t be interpreted as XQuery. Hence, the best way to use the stylesheet is by batch-transforming XQuery Unit test files with an XSLT processor, and then execute the resulting XQuery code with eXist.

Conclusion

I guess the most important conclusion is: grab the XQueryUnit2XQuery.xsl stylesheet and see if it helps you in developing XQuery code for eXist!

Although these features are not yet implemented, exchanges on the eXist-open mailing list suggest:

• their implementation is not that far off (http://markmail.org/message/yee3q4a6x6lhrwph)
• and how they could be used for implementing faceted searching (http://markmail.org/message/r56zbcfb3m5p64xz).

(Currently) eXist lacks optimised features for performant faceted searching (like e.g. the Lucene-based solr search engine). Yet, some index-based features of eXist can be used to add facets to XQuery search scripts already. Key to this approach is the util:index-keys() function, which returns indexed terms with some distribution statistics. Note, that the discussion in this post doesn’t claim any originality: the basic idea is already implemented in the eXist bibliographic demo web application, to which this discussion adds a small mechanism for defining and looping over multiple facets in an XQuery script.

I’ll illustrate this approach with an example based on the Shakespeare example XML example files that are shipped with eXist. This assumes that a) eXist-1.4 is installed, and b) the eXist examples have been set up with the admin web application. First, let’s add some more index definitions to the index configuration file that is stored in the database at ‘/db/system/config/db/shakespeare/collection.xconf’. Add two more range index definitions for the elements named ‘SPEAKER’, and ‘TITLE’, so the index configuration looks as follows:

<collection xmlns="http://exist-db.org/collection-config/1.0">
    <index>
        <fulltext default="none" attributes="no"/>
        <lucene>
            <text qname="SPEECH">
                <ignore qname="SPEAKER"/>
            </text>
            <text qname="TITLE"/>
        </lucene>
        <ngram qname="SPEAKER"/>
        <!-- range indexes -->
        <!-- note: although path-based range indexes are liable
             to get deprecated (in favour of qname based ones),
             following range indexes are path-based in order to
             avoid a bug with util:index-keys() in eXist trunk
        -->
        <create path="//SPEAKER" type="xs:string"/>
        <create path="//TITLE" type="xs:string"/>
    </index>
</collection>

In order to actually put these index definitions to use, the ‘/db/shakespeare’ collection needs to be reindexed with the Java admin client.

A simple search

First, let’s issue a simple search on the Shakespeare plays:

let $coll := collection('/db/shakespeare')
let $hits := $coll//SPEECH[ft:query(., 'lord')]
return $hits

This will perform a straightforward search on the <SPEECH> nodes that contain the string ‘lord’. The ft:query() function operates on the Lucene full text index, which in this case was defined on all <SPEECH> elements. As the content of the <SPEAKER> elements has been ignored from the full text index (see the index definition above), this search will return 272 <SPEECH> elements whose <LINE> children contain the string ‘lord’:

<SPEECH>
  <SPEAKER>HAMLET</SPEAKER>
  <LINE>Not so, my lord; I am too much i' the sun.</LINE>
</SPEECH>
<SPEECH>
  <SPEAKER>HORATIO</SPEAKER>
  <LINE>The same, my lord, and your poor servant ever.</LINE>
</SPEECH>
<SPEECH>
  <SPEAKER>MARCELLUS</SPEAKER>
  <LINE>My good lord--</LINE>
</SPEECH>
<!-- ... -->

A simple facet with standard XQuery functions

This simple search can return more than just the speeches holding the word ‘lord’. Suppose that -besides the actual speeches- we are interested in the different dramatic characters that utter these speeches. We could thus add ‘speakers’ as a facet to the search. Speakers are encoded in a <SPEAKER> element per <SPEECH>. In the context of the previous XQuery snippet, they are easily accessible as $hits//SPEAKER.

Before implementing this in XQuery code, let’s consider what we’re after at this moment:

• all speeches containing the word ‘lord’
• all different speakers uttering those speeches

This is easy enough with XQuery’s distinct-values() function:

let $coll := collection('/db/shakespeare')
let $hits := $coll//SPEECH[ft:query(., 'lord')]
let $speakers :=
  for $a in distinct-values($hits/SPEAKER)
  return <speaker>{$a}</speaker>
return <results>{
  <speakers>{$speakers}</speakers>,
  <hits>{$hits}</hits>
}</results>

These results could then be presented to the user in an intelligible way. Typically, the facets (‘speakers’, in this case) will be offered as a kind of search refinement alongside the actual search results. However, often such facets are accompanied by an indication of the number of search results that correspond to that additional search filter. This information can be computed with standard XQuery functions as well, simply by counting how many times each distinct value occurs for a facet within the search results:

let $coll := collection('/db/shakespeare')
let $hits := $coll//SPEECH[ft:query(., 'lord')]
let $speakers :=
  for $a in distinct-values($hits/SPEAKER)
  return <speaker freq="{count($hits/SPEAKER[. eq $a])}">{$a}</speaker>
return <results>{
  <speakers>{$speakers}</speakers>,
  <hits>{$hits}</hits>
}</results>

As the example illustrates, the construction of the $speakers facet now adds a @freq attribute that counts how many times each unique speaker name occurs within the set of speakers that utter the word ‘lord’:

<speakers>
  <speaker freq="10">LAERTES</speaker>
  <speaker freq="30">LORD POLONIUS</speaker>
  <speaker freq="13">HAMLET</speaker>
  <speaker freq="48">HORATIO</speaker>
  <!-- ... -->
</speakers>

However, this approach soon hits its limits: the performance of such queries directly correlates to a) the size of the search space (collection size and search hits), and b) the number of facets. The standard XQuery functions (‘distinct-values()’) and operators (‘eq’) don’t operate on eXist indexes and hence quickly suffer from degrading performance when complexity increases. Therefore, in the next section, an eXist-specific alternative will be proposed.

A simple facet with eXist-specific functions

The eXist XML database tries to optimise query performance, both by giving standard XQuery functions and operators access to its efficient indexes, and defining eXist-specific functions for querying those indexes. One such function is util:index-keys($node, $start-value, $function-reference, $max-number-returned), which looks up the different indexed values for a given node set ($node) that start with a specific start value ($start-value), delegates the processing of this index information to a dedicated function ($function-reference), and returns a maximal number of index entries ($max-number-returned).

The information returned to the helper function consists of:

• the index key
• the overall frequency of that index key within the specified node set ($node)
• the number of documents in the specified node set ($node) in which the key occurs
• the cardinal number of the index key in the total number of keys returned

Let’s illustrate this by reformulating the previous example. Note how in the facet generating part of the XQuery script the standard distinct-values() function is replaced with the eXist-specific util:index-keys() function.

declare function local:term-callback($term as xs:string, $data as xs:int+) as element() {
  <term freq="{$data[1]}" docs="{$data[2]}" n="{$data[3]}">{$term}</term>
}; 

let $callback := util:function(xs:QName("local:term-callback"), 2)
let $coll := collection('/db/shakespeare')
let $hits := $coll//SPEECH[ft:query(., 'lord')]
let $speakers := util:index-keys($hits, '', $callback, 10000)
return <results>{
  <speakers>{$speakers}</speakers>,
  <hits>{$hits}</hits>
}</results>

This produces the same list of speakers (only in alphabetical order by default), but this time they come enriched with basic statistical index information, e.g.:

<speakers>
  <!-- ... -->
  <term freq="7" docs="1" n="18">LADY MACBETH</term>
  <term freq="10" docs="1" n="19">LAERTES</term>
  <term freq="5" docs="1" n="20">LENNOX</term>
  <term freq="30" docs="1" n="21">LORD POLONIUS</term>
  <!-- ... -->
</speakers>

This call to the util:index-keys() function works directly on eXist indexes, and avoids expensive comparison and counting of the speakers for all unique speaker names. Hence, (when operating on range indexes – cf. infra) the util:index-keys() function can function as a ‘power’ version of the standard distinct-values() function.

There’s more facets to a gem

Now, let’s illustrate how the example query can be complicated by adding more facets. Suppose that besides the speakers, we are interested in the sections in which the search hits occur. In the Shakespeare documents, the distinct text divisions each have their own title:

play	/PLAY/TITLE
act	/PLAY/ACT/TITLE
scene	/PLAY/ACT/SCENE/TITLE

Remember how earlier we defined a range index on the <TITLE> element? This can be put to use for all of these facets. In order to keep the code minimal and easily extensible to other facets, facets can be defined as a set of XML elements:

<facets>
  <facet label="scenes">$hits/ancestor::SCENE/TITLE</facet>
  <facet label="acts">$hits/ancestor::ACT/TITLE</facet>
  <facet label="plays">$hits/ancestor::PLAY/TITLE</facet>
  <facet label="speakers">$hits/SPEAKER</facet>
</facets>

Each facet definition then consists of two pieces of information:

• facet path: an XPath expressions for the facet, relative to the search hits (the text content of each <facet> element)
• facet label: a string label to be used for identifying the different facets in the search results (the value for the @label attribute for each <facet> element)

The facet generation then loops over the defined facets, evaluates their XPath expressions using the eXist-specific util:eval() function, looks up the corresponding facet label , and generates the list of index entries for the facet:

declare function local:term-callback($term as xs:string, $data as xs:int+) as element() {
  <term freq="{$data[1]}" docs="{$data[2]}" n="{$data[3]}">{$term}</term>
};

let $callback := util:function(xs:QName("local:term-callback"), 2)
let $coll := collection('/db/shakespeare')
let $hits := $coll//SPEECH[ft:query(., 'lord')]
(: declare facets as XPath expressions, relative to the search hits :)
let $facets :=
  <facets>
    <facet label="scenes">$hits/ancestor::SCENE/TITLE</facet>
    <facet label="acts">$hits/ancestor::ACT/TITLE</facet>
    <facet label="plays">$hits/ancestor::PLAY/TITLE</facet>
    <facet label="speakers">$hits/SPEAKER</facet>
  </facets>
return <results>
  <facets>{
    (: loop over facet XPaths, and evaluate them :)
    for $facet in $facets//facet
    let $vals := util:eval($facet)
    return <facet>{
      $facet/@label,
      util:index-keys($vals, '', $callback, 5)
    }</facet>
  }</facets>
  <hits>{$hits}</hits>
</results>

The result looks as follows:

<results>
  <facets>
    <facet label="scenes">
      <term freq="1" docs="1" n="1">SCENE I.  A cavern. In the middle, a boiling cauldron.</term>
      <term freq="1" docs="1" n="2">SCENE I.  A churchyard.</term>
      <term freq="1" docs="1" n="3">SCENE I.  A room in POLONIUS' house.</term>
      <term freq="2" docs="1" n="4">SCENE I.  A room in the castle.</term>
      <term freq="1" docs="1" n="5">SCENE I.  Dunsinane. Ante-room in the castle.</term>
    </facet>
    <facet label="acts">
      <term freq="3" docs="3" n="1">ACT I</term>
      <term freq="3" docs="3" n="2">ACT II</term>
      <term freq="3" docs="3" n="3">ACT III</term>
      <term freq="3" docs="3" n="4">ACT IV</term>
      <term freq="3" docs="3" n="5">ACT V</term>
    </facet>
    <facet label="plays">
      <term freq="1" docs="1" n="1">The Tragedy of Hamlet, Prince of Denmark</term>
      <term freq="1" docs="1" n="2">The Tragedy of Macbeth</term>
      <term freq="1" docs="1" n="3">The Tragedy of Romeo and Juliet</term>
    </facet>
    <facet label="speakers">
      <term freq="1" docs="1" n="1">ATTENDANT</term>
      <term freq="1" docs="1" n="2">BALTHASAR</term>
      <term freq="4" docs="1" n="3">BANQUO</term>
      <term freq="3" docs="1" n="4">BERNARDO</term>
      <term freq="2" docs="1" n="5">Both Murderers</term>
    </facet>
  </facets>
  <hits>
    <SPEECH>
      <SPEAKER>LAERTES</SPEAKER>
      <LINE>My dread lord,</LINE>
      <LINE>Your leave and favour to return to France;</LINE>
      <LINE>From whence though willingly I came to Denmark,</LINE>
      <LINE>To show my duty in your coronation,</LINE>
      <LINE>Yet now, I must confess, that duty done,</LINE>
      <LINE>My thoughts and wishes bend again toward France</LINE>
      <LINE>And bow them to your gracious leave and pardon.</LINE>
    </SPEECH>
    <!-- ... -->
  </hits>
</results>

Full text facets

Besides the values of the headings of the distinct text divisions that contain the search hits, it might be interesting to add yet another facet, listing the separate words of the speeches containing the search results. This will require a slightly different approach than for the facets already present. One thing of notice in the previous search results, is the fact that the index entries returned consist of whole phrases: ‘SCENE I. A cavern. In the middle, a boiling cauldron.’, ‘ACT I’. This is due to the way the nodes in those facets have been indexed. So far, the facets are constructed on the range indexes that have been defined on the <TITLE> and <SPEAKER> elements. Suppose we defined a range index on the <LINE> elements as well and scanned this with the util:index-keys() function, this would retrieve the entire contents of those lines, instead of the individual words. Which is clearly not very helpful for refining the search in a search interface.

This is where the Lucene full text index comes into play, which tokenises the text of the nodes for which it is defined, and indexes those tokens. The collection.xconf file above defines a Lucene full text index on the <SPEECH> nodes of the Shakespare documents (while ignoring the <SPEAKER> elements). In practice, this means that the index is populated with all tokenised words of the <LINE> elements that make up each <SPEECH>. In order to scan this index with util:index-keys(), we’ll have to be more specific about the index to be used. That is possible in an optional 5th parameter for that function, naming the index. In the case of the <SPEECH> elements, the keys of their full text index can be retrieved as follows:

util:index-keys(//SPEECH, '', $callback, 5, 'lucene-index')

Combined with the callback function defined elsewhere in the XQuery, this function call would then produce a list of index entries similar to this fragment:

<term freq="2" docs="2" n="1">abate</term>
<term freq="1" docs="1" n="2">abatements</term>
<term freq="1" docs="1" n="3">abbey</term>
<term freq="3" docs="3" n="4">abhorred</term>
<term freq="1" docs="1" n="5">abhors</term>

Note that when no index is specified for the util:index-keys() function, it seems to default to the range index. Only when other indexes are scanned, those have to be named in the last argument to util:index-keys(). (On the other hand, there seems no way to explicitly state the range index in the 5th parameter; ‘range-index’ won’t be recognised.)

Wrapping up, this full text facet can be integrated in the sample XQuery, with the provision that different versions of util:index-keys() must be used depending on the index on which the facet operates. In the following example, this is catered for by adding a @type attribute with value “FT” to the full text search facets, and using this information to determine which index util:index-keys() has to be applied to:

declare function local:term-callback($term as xs:string, $data as xs:int+) as element() {
  <term freq="{$data[1]}" docs="{$data[2]}" n="{$data[3]}">{$term}</term>
}; 

let $callback := util:function(xs:QName("local:term-callback"), 2)
let $coll := collection('/db/shakespeare')
let $hits := $coll//SPEECH[ft:query(., 'lord')]
(: declare facets as XPath expressions, relative to the search hits :)
let $facets :=
  <facets>
    <facet label="scenes">$hits/ancestor::SCENE/TITLE</facet>
    <facet label="acts">$hits/ancestor::ACT/TITLE</facet>
    <facet label="plays">$hits/ancestor::PLAY/TITLE</facet>
    <facet label="speakers">$hits/SPEAKER</facet>
    <facet label="fulltext" type="FT">$hits</facet>
  </facets>
return <results>
  <facets>{
    (: loop over facet XPaths, and evaluate them :)
    for $facet in $facets//facet
    let $vals := util:eval($facet)
    let $index := if ($facet[@type eq 'FT']) then 'lucene-index' else ()
    return <facet>{
      $facet/@label,
      if ($index) then
        util:index-keys($vals, '', $callback, 5, $index)
      else
        util:index-keys($vals, '', $callback, 5)
    }</facet>
  }</facets>
  <hits>{$hits}</hits>
</results>

Although at this stage, this XQuery script only provides a basic proof of concept, it illustrates how facets for certain information categories could be generated alongside search results. Of course, in order to reduce processing overhead, the $max-number-returned parameter of util:index-keys() can be reduced (as in these examples), so that only a limited number of facet entries are returned in a first phase. An option could then be offered to see more (or all) entries for these individual facets in a second step.

Conclusion

Although future integration of efficient Lucene search capabilities into eXist probably holds more promises for efficient faceted search implementations with eXist, some eXist-specific features (different indexes, util:index-keys()) provide a way to add search facets to query results already. Although the performance of the approach described in this post will probably decrease with very large data collections, the use of the util:index-keys() function seems to offer a reasonably stable option (performance-wise) for constructing search facets alongside a search.

• [feature]: added a couple of further conditions in $lucene2xml, in order to benefit from unified <exist:match> markers for adjacent phrase terms: differentiate between
  • phrase search: rewrite <near slop="<1"> to <phrase>
  • proximity search: copy <near slop=">=1">
• [fix]: improved treatment of escaped parentheses inside proximity search expressions

Since version 1.4, the eXist native XML database implements a Lucene-based full text index. The main Lucene-aware search function, ft:query() accepts queries expressed in two flavours:

• Lucene’s default query syntax
• eXist’s XML query syntax

The XML query syntax was explicitly designed to allow for more expressive queries than is possible with the Lucene syntax. Most notably, eXist has extensions for:

• fine-grained proximity searches with the <near> element (a.o. the possibility to specify that search terms can occur unordered)
• regular expression searches with the <regex> element

This makes the XML syntax the more interesting option for developing a user search interface. A search interface could then allow users to input search queries in the (quite intuitive) Lucene fashion, while providing additional options for specifying extra search features (‘(un)ordered proximity search’, ‘regular expression search’). Behind the scenes, both pieces of user input (search query + additional parameters) can be translated to an XML expression of the search query.

Obviously, the first step of such a translation involves parsing of a query in Lucene syntax and transforming it to its XML syntax equivalent.

This seems feasible in XQuery using a couple of eXist-specific extension functions. I’ll briefly sketch the approach below:

1. translate a Lucene search string to an intermediate string mimicking the XML syntax, with some additions for later parsing of boolean operators
2. parse the intermediary XML search string as XML with util:parse()
3. transform the intermediary structures in the search query to full-fledged boolean expressions

The first step, transforming a Lucene search string to an intermediate string mimicking the XML syntax, can be done with an XQuery function that merely performs string replacement. Let’s call it local:parse-lucene():

declare function local:parse-lucene($string) {
  (: replace all symbolic booleans with lexical counterparts :)
  if (matches($string, '[^\\](\|{2}|&{2}|!) ')) then
    let $rep := replace(
                  replace(
                    replace($string, '&{2} ', 'AND ')
                  , '\|{2} ', 'OR ')
                , '! ', 'NOT ')
    return local:parse-lucene($rep)
  (: replace all booleans with '<AND/>|<OR/>|<NOT/>' :)
  else if (matches($string, '[^<](AND|OR|NOT) ')) then
    let $rep := replace($string,
                        '(AND|OR|NOT) ',
                        '<$1/>')
    return local:parse-lucene($rep)
  (: replace all '+' modifiers with '<AND/>' :)
  else if (matches($string, '(^|[^\w"'])\+[\w"'(]')) then
    let $rep := replace($string,
                        '(^|[^\w"'])\+([\w"'(])',
                        '$1<AND type=_+_/>$2')
    return local:parse-lucene($rep)
  (: replace all '-' modifiers with '<NOT/>' :)
  else if (matches($string, '(^|[^\w"'])-[\w"'(]')) then
    let $rep := replace($string,
                        '(^|[^\w"'])-([\w"'(])',
                        '$1<NOT type=_-_/>$2')
    return local:parse-lucene($rep)
  (: replace round brackets with '<bool></bool>' :)
  else if (matches($string, '(^|\W|>)\(.*?\)(\^(\d+))?(<|\W|$)')) then
    let $rep :=
      (: add @boost attribute when string ends in ^\d :)
      if (matches($string, '(^|\W|>)\(.*?\)(\^(\d+))(<|\W|$)')) then
        replace($string,
                '(^|\W|>)\((.*?)\)(\^(\d+))(<|\W|$)',
                '$1<bool boost=_$4_>$2</bool>$5')
      else
        replace($string,
                '(^|\W|>)\((.*?)\)(<|\W|$)',
                '$1<bool>$2</bool>$3')
    return local:parse-lucene($rep)
  (: replace quoted phrases with '<near slop=""></bool>' :)
  else if (matches($string, '(^|\W|>)("|').*?\2([~^]\d+)?(<|\W|$)')) then
    let $rep :=
      (: add @boost attribute when phrase ends in ^\d :)
      if (matches($string, '(^|\W|>)("|').*?\2([\^]\d+)?(<|\W|$)')) then
        replace($string,
                '(^|\W|>)("|')(.*?)\2([~^](\d+))?(<|\W|$)',
                '$1<near boost=_$5_>$3</near>$6')
      (: add @slop attribute in other cases :)
      else
        replace($string,
                '(^|\W|>)("|')(.*?)\2([~^](\d+))?(<|\W|$)',
                '$1<near slop=_$5_>$3</near>$6')
    return local:parse-lucene($rep)
  (: wrap fuzzy search strings in '<fuzzy min-similarity=""></fuzzy>' :)
  else if (matches($string, '[\w-[<>]]+?~[\d.]*')) then
    let $rep := replace($string,
                        '([\w-[<>]]+?)~([\d.]*)',
                        '<fuzzy min-similarity=_$2_>$1</fuzzy>')
    return local:parse-lucene($rep)
  (: wrap resulting string in '<query></query>' :)
  else concat('<query>',
              replace(normalize-space($string), '_', '"'),
               '</query>')
};

[NOTE: The single and double quotation marks in the match() and replace() functions should be escaped with their respective character entities. Unfortunately, posting them to a HTML blog destroys this escaping, and doubly escaping the ampersands of the character entities ended up… escaped themselves! Should you want to try this function, you’ll have to make sure to escape these characters.]

This results in a string representing an intermediary XML version of the query. For example, this function would translate the Lucene search string ‘(fillet OR "mal(ic)e done"~1) AND snake^4’ to:

<query>
  <bool>fillet <OR/><near slop="1">mal(ic)e done</near></bool>
  <AND/>snake^4
</query>

This must be processed further, in order to get all boolean operators (and the occurrence indicators of their members) right. As booleans involve pairing of terms and hence (re)grouping, this conversion optimally operates on the corresponding XML structure of the preprocessed string. Therefore, it should be parsed to XML with eXist’s util:parse() function.

Option 1: conversion to final XML with XSLT

Since I’m most proficient in XSLT, I’ve implemented a first proof of concept for this conversion in an XSLT stylesheet that takes care of the boolean operators. This stylesheet can be applied to the intermediary XML search structure with the transform:transform() function:

<xsl:stylesheet
     xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
     xmlns:xs="http://www.w3.org/2001/XMLSchema"
     exclude-result-prefixes="#all" version="2.0">

  <xsl:output indent="yes"/>

  <xsl:template match="query">
    <xsl:copy>
      <xsl:copy-of select="@*"/>
        <bool>
          <xsl:apply-templates select="node()"/>
        </bool>
      </xsl:copy>
  </xsl:template>

  <xsl:template match="*[(following-sibling::*[1]|preceding-sibling::*[1])
       [self::AND or self::OR or self::NOT]]">
    <xsl:variable name="name">
      <xsl:choose>
        <xsl:when test="(self::phrase|self::near)[not(@slop &gt; 0)]">phrase</xsl:when>
        <xsl:otherwise><xsl:value-of select="name()"/></xsl:otherwise>
      </xsl:choose>
    </xsl:variable>
    <xsl:element name="{{$name}}">
      <xsl:attribute name="occur">
        <xsl:choose>
          <xsl:when test="preceding-sibling::*[1][self::AND]">must</xsl:when>
          <xsl:when test="preceding-sibling::*[1][self::NOT]">not</xsl:when>
          <xsl:otherwise>
            <xsl:choose>
              <xsl:when test="following-sibling::*[1]
                   [self::AND or self::OR or self::NOT][not(@type)]">should</xsl:when>
              <xsl:otherwise>should</xsl:otherwise>
            </xsl:choose>
          </xsl:otherwise>
        </xsl:choose>
      </xsl:attribute>
      <xsl:apply-templates select="@*|node()"/>
    </xsl:element>
  </xsl:template>

  <xsl:template match="query/text()[normalize-space()]|bool/text()[normalize-space()]">
    <xsl:variable name="current" select="."/>
    <xsl:for-each select="tokenize(., '\s+')[normalize-space()]">
      <!-- here is the place for further differentiation between
           term / wildcard / regex elements -->
      <term>
        <xsl:attribute name="occur">
          <xsl:choose>
            <xsl:when test="position() = 1 and $current/preceding-sibling::*[1]
                 [self::AND]">must</xsl:when>
            <xsl:when test="position() = 1 and $current/preceding-sibling::*[1]
                 [self::NOT]">not</xsl:when>
            <xsl:otherwise>
              <xsl:choose>
                <xsl:when test="position() = 1 and $current/following-sibling::*[1]
                     [self::AND or self::OR or self::NOT][not(@type)]">should</xsl:when>
                <xsl:otherwise>should</xsl:otherwise>
              </xsl:choose>
            </xsl:otherwise>
          </xsl:choose>
        </xsl:attribute>
        <xsl:if test="matches(., '(.*?)(\^(\d+))(\W|$)')">
          <xsl:attribute name="boost">
            <xsl:value-of select="replace(., '(.*?)(\^(\d+))(\W|$)', '$3')"/>
          </xsl:attribute>
        </xsl:if>
        <xsl:value-of select="normalize-space(replace(., '(.*?)(\^(\d+))(\W|$)', '$1'))"/>
      </term>
    </xsl:for-each>
  </xsl:template>

  <xsl:template match="AND|OR|NOT" priority="1"/>

  <xsl:template match="near/bool">
    <xsl:value-of select="concat('(', ., ')')"/>
  </xsl:template>

  <xsl:template match="*">
    <xsl:variable name="name">
      <xsl:choose>
        <xsl:when test="(self::phrase|self::near)[not(@slop &gt; 0)]">phrase</xsl:when>
        <xsl:otherwise><xsl:value-of select="name()"/></xsl:otherwise>
      </xsl:choose>
    </xsl:variable>
    <xsl:element name="{{$name}}">
      <xsl:apply-templates select="@*|node()"/>
    </xsl:element>
  </xsl:template>

  <xsl:template match="@*|node()" priority="-1">
    <xsl:copy>
      <xsl:apply-templates select="@*|node()"/>
    </xsl:copy>
  </xsl:template>

</xsl:stylesheet>

Suppose (for quick prototyping’s sake) this XSLT stylesheet is stored in a variable $lucene2xml. Now all we have to do in the main query is process the input $string with local:parse-lucene(), parse its output with util:parse(), and transform it with transform:transform():

let $string := '(fillet OR "mal(ic)e done"~1) AND snake^4'
let $luceneParse := local:parse-lucene($string)
let $luceneXML := util:parse($luceneParse)
return transform:transform($luceneXML, $lucene2xml, ())

For the input string ‘(fillet OR "mal(ic)e done"~1) AND snake^4’, this produces following XML query:

<query>
  <bool>
    <bool occur="must">
      <term occur="must">fillet</term>
      <near occur="should" slop="1">mal(ic)e done</near>
    </bool>
    <term occur="must" boost="4">snake</term>
  </bool>
</query>

Option 2: conversion to final XML with XQuery function

Since the XSLT stylesheet described above is quite simple, it can easily be formulated as an XQuery function as well. Apart from possible performance issues (I can’t make any sensible claims about this), a full XQuery solution is probably the more elegant option. Therefore, I’ve prepared an alternative version, expressing the XSLT stylesheet as an XQuery function:

declare function local:lucene2xml($node) {
  typeswitch ($node)
    case element(query) return
      element { node-name($node)} {
        element bool {
          $node/node()/local:lucene2xml(.)
        }
      }
    case element(AND) return ()
    case element(OR) return ()
    case element(NOT) return ()
    case element(bool) return
      if ($node/parent::near) then
        concat("(", $node, ")")
      else element {node-name($node)} {
        $node/@*,
        $node/node()/local:lucene2xml(.)
      }
    case element() return
      let $name :=
        if (($node/self::phrase|$node/self::near)[not(@slop > 0)]) then
          'phrase'
        else node-name($node)
      return
        element { $name } {
          $node/@*,
          if (($node/following-sibling::*[1]|
               $node/preceding-sibling::*[1])
              [self::AND or self::OR or self::NOT]) then
            attribute occur {
              if ($node/preceding-sibling::*[1][self::AND]) then 'must'
              else if ($node/preceding-sibling::*[1][self::NOT]) then 'not'
              else if ($node/following-sibling::*[1]
                       [self::AND or self::OR or self::NOT][not(@type)]) then 'should'
              else 'should'
            }
          else (),
          $node/node()/local:lucene2xml(.)
        }
    case text() return
      if ($node/parent::*[self::query or self::bool]) then
        for $tok at $p in tokenize($node, '\s+')[normalize-space()]
        (: here is the place for further differentiation
           between  term / wildcard / regex elements :)
        (: using regex-regex detection (?):
           matches($string, '((^|[^\\])[.?*+()\[\]\\^]|\$$)') :)
        let $el-name := 'term'
        return element { $el-name } {
          attribute occur {
            if ($p = 1 and $node/preceding-sibling::*[1]
                [self::AND]) then 'must'
            else if ($p = 1 and $node/preceding-sibling::*[1]
                [self::NOT]) then 'not'
            else if ($p = 1 and $node/following-sibling::*[1]
                [self::AND or self::OR or self::NOT][not(@type)]) then 'should'
            else 'should'
          },
          if (matches($tok, '(.*?)(\^(\d+))(\W|$)')) then
            attribute boost {
              replace($tok, '(.*?)(\^(\d+))(\W|$)', '$3')
            }
            else (),
          normalize-space(replace($tok, '(.*?)(\^(\d+))(\W|$)', '$1'))
        }
      else
        normalize-space($node)
  default return
    $node
};

Instead of the XSLT transformation in the example above, this XQuery function can be called as the last step in the main query:

let $string := '(fillet OR "mal(ic)e done"~1) AND snake^4'
let $luceneParse := local:parse-lucene($string)
let $luceneXML := util:parse($luceneParse)
return local:lucene2xml($luceneXML/node())

The output is (and should be) the same as above.

Conclusion

So far for this proof of concept. What’s missing still are the extra search parameters:

• wrap regex searches in <regex> instead of <term>
• specify the ordered status of a proximity search in an @order attribute for <near>

In order to achieve this, those options could be passed as parameters to the XSLT stylesheet or the XQuery function. Both the XSLT stylesheet and XQuery function have comments at the place where the former option should be processed. In the current approach, the @occur attribute could already be added in the local:parse-lucene() function. Also, the best place for inserting the <fuzzy> tag is subject to reconsideration (maybe it’s more logical to treat it on a par with other ‘atomic’ search components <term>, <regex>, and <wildcard>).

However, eXist currently limits regex and wildcard searches to a boolean context (<bool>), where they can occur instead of <term>. Allowing them inside <near> (or <phrase>) will have to be implemented (and considered) first. I’ll wait for the outcome of this implementation before finishing the script, which then will either allow <regex> or <wildcard> anywhere, or have to restrict it to more limited contexts. To be continued…

In the mean time, a wrapped-up version of this proof of concept XQuery can be found:

• here for the XSLT version
• here for the XQuery version

FOP-1.0 has been released, which fixes the nasty bug where footnotes inside lists and tables got swallowed. There’s still an issue with overlapping content for footnotes inside columns, but I can live with that…

Hence, I could as well delete my previous entry but will leave it here for documentation’s sake.

[...or skip the discussion and just download the files]

The reason

During the past couple of years, I’ve gathered some experience working with XML and related standards (XSLT, XSL-FO, XQuery). Part of our professional document production chain involves rendering PDF output from XML sources. I’ve grown into a big fan of Apache’s open source FOP processor since its now ancient version 0.20.5. Although the FOP code has been substantially revised and improved long since, the versions up to version 0.95 were haunted by one serious bug, which kept me from switching to an up-to-date version of FOP: footnotes inside lists or table cells got swallowed in PDF output.

On the other hand, FOP’s XSL-FO compliance rate has risen substantially in the recent versions, prompting me to find a way of dealing with this nasty show-stopper. Of course, I hope the FOP developers will be able to resolve this issue soon. In the mean time, I think I’ve found a way of circumventing (or at least alleviating) the problem (at stylesheet level; not at Java code level). Moreover, I think this approach might help other users as well, and other users might help improving this approach where it doesn’t.

Hence this initial blog post, in a mild blend of self-documentation and altruism. It will be quite technical and specific, but I hope to get the message clear. At least I’ll try, by:

• starting from a stable, simple example. For (my own) convenience’s sake, I’ve crafted a TEI P5 example structure, since that’s where my expertise lies.
• illustrating intermediate steps with (pointers to) XSL-FO code examples
• illustrating the results with screenshots of corresponding PDF output of the examples

I’ve categorised this blog post under XML, XSL-FO, and XSLT as well. Although I’ll focus on the (theoretical) XSL-FO side of a solution, I’ll provide a link to a zip package containing sample XML documents and an XSLT stylesheet illustrating the final stage of this solution. However, some problems remain for which I don’t have an immediate solution. Therefore I welcome any comments.

The problem

Simply put, FOP (0.92-0.95) had troubles rendering fo:footnote areas occurring within fo:table or fo:list-block areas, an issue which is formally documented as a bug.  At the time of writing this post, the comments in the bug tracker suggested that  a) the bug wouldn’t be resolved for anytime soon and b) there was no known workaround. Consider for example, this standard TEI XML list containing a footnote:

<list xmlns="http://www.tei-c.org/ns/1.0">
  <item>Case 1: list[1]/item[1]
    <note>Case 1: list[1]/item[1]/note[1]</note>
  </item>
  <item>Case 1: list[1]/item[2]</item>
  <item>Case 1: list[1]/item[3]</item>
</list>

When transformed to a corresponding standard XSL-FO structure:

<fo:list-block xmlns:fo="http://www.w3.org/1999/XSL/Format"
    provisional-distance-between-starts="50pt"
    provisional-label-separation="10pt"
    start-indent="from-parent(start-indent) + 5pt">
  <fo:list-item>
    <fo:list-item-label end-indent="label-end()">
      <fo:block>•</fo:block>
    </fo:list-item-label>
    <fo:list-item-body start-indent="body-start()">
      <fo:block>Case 1: list[1]/item[1]
        <fo:footnote>
          <fo:inline font-size="8pt" vertical-align="super">1</fo:inline>
          <fo:footnote-body
              font-size="10pt" space-after="0.5em" end-indent="0px"
              start-indent="0px" text-align="start"
              font-style="normal" font-weight="normal">
            <fo:list-block>
              <fo:list-item>
                <fo:list-item-label end-indent="label-end()">
                  <fo:block>1</fo:block>
                </fo:list-item-label>
                <fo:list-item-body start-indent="body-start()">
                  <fo:block>Case 1: list[1]/item[1]/note[1]</fo:block>
                </fo:list-item-body>
              </fo:list-item>
            </fo:list-block>
          </fo:footnote-body>
        </fo:footnote>
      </fo:block>
    </fo:list-item-body>
  </fo:list-item>
  <fo:list-item>
    <fo:list-item-label end-indent="label-end()">
      <fo:block>•</fo:block>
    </fo:list-item-label>
    <fo:list-item-body start-indent="body-start()">
      <fo:block>Case 1: list[1]/item[2]</fo:block>
    </fo:list-item-body>
  </fo:list-item>
  <fo:list-item>
    <fo:list-item-label end-indent="label-end()">
      <fo:block>•</fo:block>
    </fo:list-item-label>
    <fo:list-item-body start-indent="body-start()">
      <fo:block>Case 1: list[1]/item[3]</fo:block>
    </fo:list-item-body>
  </fo:list-item>
</fo:list-block>

…the PDF output will show the footnote marker in the first list item, but NOT the footnote body at the bottom of the page! The same happens when tables are involved:

footnote problem in a list	footnote problem in a table

Of course, this is quite uncomfortable: all goes well as long as the input documents don’t contain any footnotes inside tables or lists. But who can / wants to guarantee that?

Proposal 1: "relative endnotes"

A way in which this problem can be avoided, is by generating fo:footnote formatting objects for those footnotes outside the areas of their containing lists and tables. Key to this approach lies in this characteristic of the fo:footnote formatting object:

The fo:footnote formatting object does not generate any areas. The fo:footnote formatting object returns the areas generated and returned by its child fo:inline formatting object.

This means that a fo:footnote will only produce in-line areas through the contents of its fo:inline footnote marker. Consequently, if this marker is left empty, a fo:footnote will only generate an out-of-line block area at the bottom of the page.

For tables and lists containing footnotes, this could inform following approach:

1. Process all contents as usual, except for footnotes. For the latter, instead of outputting a fo:footnote formatting object, just generate a fo:inline "dummy" marker, nothing more.
2. Create a separate fo:block after the affected table / list, containing all footnotes. For each footnote, generate a complete fo:footnote structure, but leave the fo:inline footnote markers empty.

Applied to the previous example, this would generate following XSL-FO fragment:

<fo:list-block xmlns:fo="http://www.w3.org/1999/XSL/Format"
    provisional-distance-between-starts="50pt"
    provisional-label-separation="10pt"
    start-indent="from-parent(start-indent) + 5pt">
  <fo:list-item>
    <fo:list-item-label end-indent="label-end()">
      <fo:block>•</fo:block>
    </fo:list-item-label>
    <fo:list-item-body start-indent="body-start()">
      <fo:block>Case 1': list[1]/item[1]
        <fo:inline font-size="8pt" vertical-align="super">1</fo:inline>
      </fo:block>
    </fo:list-item-body>
  </fo:list-item>
  <fo:list-item>
    <fo:list-item-label end-indent="label-end()">
      <fo:block>•</fo:block>
    </fo:list-item-label>
    <fo:list-item-body start-indent="body-start()">
      <fo:block>Case 1': list[1]/item[2]</fo:block>
    </fo:list-item-body>
  </fo:list-item>
  <fo:list-item>
    <fo:list-item-label end-indent="label-end()">
      <fo:block>•</fo:block>
    </fo:list-item-label>
    <fo:list-item-body start-indent="body-start()">
      <fo:block>Case 1': list[1]/item[3]</fo:block>
    </fo:list-item-body>
  </fo:list-item>
</fo:list-block>
<fo:block>
  <fo:footnote>
    <fo:inline font-size="8pt" vertical-align="super">1</fo:inline>
    <fo:footnote-body font-size="10pt"
        space-after="0.5em" end-indent="0px"
        start-indent="0px" text-align="start"
        font-style="normal" font-weight="normal">
      <fo:list-block>
        <fo:list-item>
          <fo:list-item-label end-indent="label-end()">
            <fo:block>1</fo:block>
          </fo:list-item-label>
          <fo:list-item-body start-indent="body-start()">
            <fo:block>Case 1': list[1]/item[1]/note[1]</fo:block>
          </fo:list-item-body>
        </fo:list-item>
      </fo:list-block>
    </fo:footnote-body>
  </fo:footnote>
</fo:block>

…and look: when rendered to PDF, this time the ‘footnote’ is present (although technically, it’s rather a "relative endnote" to its containing list)!

relative endnote in a list	relative endnote in a table

Evaluation

This approach, inspired by other solutions to vertical alignment issues seems quite elegant (simple) and efficient (powerful enough for nested tables / lists). However, it comes with a catch:

• Theoretically, it works best for short lists or tables that don’t span different pages. For longer ones, however, all footnote bodies will end up after their containing table / list (as a kind of end notes to the table / list).
• In practice, there is a case where FOP seems to choke: when the number of footnotes in a list or table grows too large, FOP hangs.

Under these provisos, this "relative endnote" approach might strike a fine balance between an efficient solution for most cases, and a sufficient compromise for longer lists or tables. However, the edge case where FOP reveals troubles with long tables / lists containing many footnotes leaves me uneasy.

Proposal 2: "relative footnotes"

This approach takes the reasoning one step further. It starts from 2 observations:

1. As seen in the "relative endnote" approach, fo:footnote formatting objects with empty fo:inline footnote markers can be inserted without generating any extra whitespace between blocks.
2. As with other block formatting objects, tables and lists can be stacked under each other without whitespace as if they formed a whole, if they have appropriate space, margin and padding properties.

From this it follows that if we can simulate footnotes in tables / lists via endnotes that are invisible as such (ie. they don’t generate extra inline areas under the affected tables / lists), it is equally possible to have endnotes per list item / table row. In other words: if tables or lists containing notes can be split up into atomic chunks, those atomic chunks can be followed by the relative endnotes they contain. Since lists and tables are composed of horizontal areas (list items and rows), we could treat each of those separately, create a separate single-item/row list or table for them in their own right and output the footnotes as relative endnotes to these tables / lists.

For the previous example, this would generate 3 separate lists, each containing 1 single item, of which the first one will be followed by a "relative endnote" block:

<fo:list-block provisional-distance-between-starts="50pt"
    provisional-label-separation="10pt"
    start-indent="from-parent(start-indent) + 5pt">
  <fo:list-item>
    <fo:list-item-label end-indent="label-end()">
      <fo:block>•</fo:block>
    </fo:list-item-label>
    <fo:list-item-body start-indent="body-start()">
      <fo:block>Case 1': list[1]/item[1]
        <fo:inline font-size="8pt" vertical-align="super">1</fo:inline>
      </fo:block>
    </fo:list-item-body>
  </fo:list-item>
</fo:list-block>
<fo:block>
  <fo:footnote>
    <fo:inline font-size="8pt" vertical-align="super"/>
      <fo:footnote-body font-size="10pt" space-after="0.5em"
          end-indent="0px" start-indent="0px" text-align="start"
          font-style="normal" font-weight="normal">
        <fo:list-block>
        <fo:list-item>
          <fo:list-item-label end-indent="label-end()">
            <fo:block>1</fo:block>
          </fo:list-item-label>
          <fo:list-item-body start-indent="body-start()">
            <fo:block>Case 1': list[1]/item[1]/note[1]</fo:block>
          </fo:list-item-body>
        </fo:list-item>
      </fo:list-block>
    </fo:footnote-body>
  </fo:footnote>
</fo:block>
<fo:list-block provisional-distance-between-starts="50pt"
    provisional-label-separation="10pt"
    start-indent="from-parent(start-indent) + 5pt">
  <fo:list-item>
    <fo:list-item-label end-indent="label-end()">
      <fo:block>•</fo:block>
    </fo:list-item-label>
    <fo:list-item-body start-indent="body-start()">
      <fo:block>Case 1': list[1]/item[2]</fo:block>
    </fo:list-item-body>
  </fo:list-item>
</fo:list-block>
<fo:list-block provisional-distance-between-starts="50pt"
    provisional-label-separation="10pt"
    start-indent="from-parent(start-indent) + 5pt">
  <fo:list-item>
    <fo:list-item-label end-indent="label-end()">
      <fo:block>•</fo:block>
    </fo:list-item-label>
    <fo:list-item-body start-indent="body-start()">
      <fo:block>Case 1': list[1]/item[3]</fo:block>
    </fo:list-item-body>
  </fo:list-item>
</fo:list-block>

Of course, this approach is easy enough for single-level lists or tables, but requires further consideration for nested ones. In this case, it is necessary not just to wrap up each list item into its own single-item list, but to mimic its complete superstructure. Otherwise indentation will be completely lost and it will be impossible to get the horizontal and vertical alignment straight.

Therefore, a complete treatment for "relative footnotes" should:

1. For each table / list with footnotes, defer further processing to their child rows / items.
2. For each table row / list item, reconstruct the complete superstructure (up to the last table / list) in which it occurs.
3. For each step in this reconstruction, only output the corresponding XSL-FO structure without text contents. Only for the last step (ie. the row / item under consideration), output the text contents.
4. If the table row / list item contains footnotes, output these as relative endnotes in a block after the table / list.
5. If the table row / list item contains further nested tables or lists, repeat steps 1 to 5 for each of these.

This reconstruction can be done easier than ever with an XSLT 2.0 stylesheet and its native capabilities to assign intermediate node sets to variables. It requires some thought for proper treatment of padding settings and especially table borders, but since that’s mainly an XSLT matter, I won’t go into details here. For brevity’s sake, I won’t give an XSL-FO code example either (things soon get very verbose), but instead provide a screenshot of a complex case:

relative footnotes with mixed nesting tables and lists

Evaluation

From a theoretical angle, this "relative footnote" approach is appealing because it’s more complete than the "relative endnote" approach and avoids the problem FOP seems to have when the number of footnotes in lists or tables grows too large.

On the other hand, it complicates processing substantially, for dealing with nested lists and tables requires careful thinking about padding and border properties. Moreover, it generates a heavy load of XSL-FO structures: each table row / list item will mirror its complete superstructure, which gets proportionally complex and verbose as the nesting level increases. This concern can be lessened by strategies for minimisation:

1. only applying the "relative footnote" approach to tables and lists containing footnotes, while treating all others regularly
2. for tables / lists containing footnotes, only apply the "relative footnotes" technique to those rows / items with footnotes, while grouping others in regular tables / lists

With such optimisations, this approach looks like a promising solution to table/list-related footnote display problems with FOP. However, another problem looms, exposing the line-based approach of this technique as a limitation.

Relative, all too relative

The "relative footnote" approach goes a long way in circumventing the problem, but can’t cope with a further level of complexity. Consider the case where one table row contains multiple nesting tables or lists.

parallel nesting tables

The line-based nature of the "relative footnote" approach won’t be able to cope well when multiple levels of parallel lines are involved, as is the case with parallel nesting tables. In this case, alignment is the problem: within the containing table row (row 2 in this case), both cells align with each other. However, in their nesting tables, rows will align independently of their parallel counterparts, depending on the number of rows and the length of their text contents. With a strict line-based approach, this would force all rows at all levels to align with each other, producing unwanted whitespace as in following example (mocked-up in Word because I’m undecided if implementing it in the XSLT stylesheet is worth the trouble):

parallel nesting tables with improper alignment

This is where I’m stuck at the moment. I don’t see clear how parallel nesting tables should be treated. It is possible, however, to treat parallel nesting tables correctly in isolation, but integrating them in a strictly line-based approach will destroy their independent alignment.

Without modification, the "relative footnote" XSLT stylesheet will render the previous example as follows:

parallel nesting tables processed sequentially

While illustrating the correct logic of the stylesheet, the output clearly is sub-optimal: parallel nesting tables are processed sequentially, producing complete superstructure-tables for each row. I don’t have any clear ideas yet, but maybe some position-related properties could be worth investigating further:

• if the width of the outer tables containing parallel nesting tables would be reduced to the required column width, maybe floating the latter on the right hand side of the former table could reconstruct their juxtaposed nature. Unfortunately, floating properties are not yet supported by FOP.
• if both tables could be ‘laid over’ each other with absolute positioning, this could reconstruct their juxtaposed nature. My current knowledge about position-absolute doesn’t allow me even to predict if this route is worth investigating.

However undesirable from a theoretical point of view, I currently consider this as the end point of my quest, considering the fact that tables with footnotes and parallel nesting tables might be rare in the wild. Anyway, I hope to have found a solution for this problem before I’ll have to tackle it (or rather hope that FOP will soon have its footnote treatment properly fixed). In the mean time, I welcome any comments!

Wrapping up: conclusion and files

When processing PDF output from XML documents with FOP 0.92+, a serious drawback is the omission of footnotes occurring in tables or lists. In this post, 2 possible strategies were explored to circumvent the problem:

1. "relative endnotes" approach: convert footnotes in tables or lists to endnotes to the affected table or list
   

   pro	contra
   simple	limited: footnotes appear as relative endnotes
   	FOP hangs when the number of footnotes grows too large

2. "relative footnotes" approach: convert tables or lists with footnotes to stacks of atomic tables or lists with relative endnotes
   

   pro	contra
   footnotes appear as genuine footnotes	verbose, complex
   	conceptual problem with parallel nesting tables

Get the files

XML and XSLT files illustrating the "relative footnotes" approach can be found here. This zip file contains following files:

1. notetest.xml: a sample XML file containing 6 cases with reference solutions
2. notetest2.xml: a less carefully compiled sample XML file
3. notetest.xsl: an XSLT stylesheet demonstrating the "relative footnotes" technique

The XSLT stylesheet will only apply the "relative footnotes" technique to tables or lists with footnotes, but currently doesn’t apply any internal optimisation to the latter (as suggested above).

In the XSLT stylesheet, the sub-optimal treatment of parallel nesting tables is kept as a kind of stub for further work.

Any comments are much appreciated!