Treating XML Databases as File Systems with WebDAV and DAVfs2

I thoroughly enjoy working with most XML databases, but one of the features that I especially like working with is the fact that you can treat collections as a form of file system - up to and including using graphical file managers to drag and drop files directly into the database. This magic is accomplished via the power of a particular protocol called WebDAV (the DAV in this case standing for Distributed Authoring and Versioning, and can be thought of as an early form of RESTful services).

Unfortunately, while there are a growing number of WebDAV clients, I've also discovered that enabling the drag and drop and other services is often not as intuitively obvious as one might hope. In particular, I've had a great deal of difficulty on the Linux side (specifically Ubuntu's Jaunty Jackelope) actually getting things to work properly, or even at all, with the obvious tools such as konquerer or the new file manager Dolphin. Ideally, what I'd like is the ability to treat my database - in the examples given here specifically my eXist database, though similar techniques should work for MarkLogic and others that expose WebDAV - as if it was simply another directory in my file system.

Being able to do this isn't just a lark. Suppose, for instance, that you had a JavaScript library which consisted of a number of folders containing code, resources, and related content and you wanted to get these into eXist so that they could be invoked from within the database itself via the REST interfaces. This can get to be a remarkably tedious undertaking if you had to load each file into an application such as eXist, then save it as a URL (which actually uses WebDAV itself in the background), especially if the contents of those files happen to be images that may take forever to load into oXygen. Additionally, you would have to create each subfolder by hand as a separate collection. This can get to be mind-numbing when the files are deep enough, especially if you're the kind of developer like me who prefers developing offline (in localhost mode) on my laptop then load in the resources to the host server once I have it adequately debugged and developed.

Each database tends to have its own URL format for both setting up and accessing WebDAV, and I recommend strongly that you read the relevant documentation for your database before getting started. With eXist, WebDAV support is enabled out of the box, though it can be disabled by modifying the file $EXIST_HOME/webapp/WEB-INF/web.xml to comment out the WebDAV service URL space mapping:

<!-- webDAV -->

    <servlet-mapping>

        <servlet-name>WebDAVServlet</servlet-name>

        <url-pattern>/webdav/*</url-pattern>

    </servlet-mapping>

Note that this doesn't turn off WebDAV itself, but only makes it inaccessible from a URL. You can also change the name of the url pattern here, so that if you wanted to label your webDAV server as /foo, then all you'd need to do would be to change this to

    <!-- webDAV -->

    <servlet-mapping>

        <servlet-name>WebDAVServlet</servlet-name>

       <url-pattern>/foo/*</url-pattern>

    </servlet-mapping>

Ordinarily, with eXist, the URL would be given as http://www.mydatabase.com:8080/exist/webdav/db, but making that one change would change this to http://www.mydatabase.com:8080/exist/foo/db. Please note that the /webdav/ itself is not an address, but only a service - you must including the root of the eXist database tree (/db in most cases) in order for most WebDAV clients to work.

Significantly, the path after the /webdav/ then starts from the root directory of the database and works its way down, so that if you had a collection with a URL of the form /db/myapps/mydata/bar.xml  then accessing bar would be given as http://www.mydatabase.com:8080/exist/webdav/db/myapps/mydata/bar.xml.

MarkLogic works a bit differently. You can set up multiple WebDAV servers within MarkLogic through the Server Admin pane (along with HTTP servers for RESTful operations). each of which can be named independently. For instance, if you had one database for clientA and a second for clientB, you could set up two distinct WebDAV servers at different ports 8180 and 8181 respectively, and acess them through the ports: http://www.mydatabase.com:8180/. You could then use Apache or a similar service (depending upon how you're hosting your servlet) to do the mapping to a more friendly URL. Since each port is associated with a separate database location, there's no reason to add the service identifier, though again you'd access subcollections by treating them as a filepath: http://www.mydatabase.com:8180/mydata/bar.xml  (where the "root" collection is assigned when the webDAV server is created).

IBM db2 PureXML, Oracle xml-db and EMC/Documentum xdb 9 all include WebDAV support as well, though I defer readers to their respective developer networks for more details. The important point is that in nearly all commercial and open source XML databases WebDAV support is either enabled or at enable-able, making WebDAV a viable tool for accessing these repositories.

It's noteworthy to understand that WebDAV, like a number of other standards, had become a fairly minimally supported protocol, both because WebDAV was initially not considered secure enough (though that's certainly changed) but also because the use cases for WebDAV before the era of XML data repositories were fairly limited (WebDAV was succeeded as a versioning language first by CVS and later by subversion). Additionally, the FTP protocol held sway for remote file access, which limited the other potential use case for WebDAV. Yet as more and more of the web has switched over to HTTP almost exclusively (along with "rider" protocols such as XMPP), WebDAV had become an increasingly important standard. For XML Databases, WebDAV is just about ideal, especially as such databases move towards providing RESTful interfaces.

On Windows, Microsoft introduced an early version of a WebDAV client through the Internet Explorer called Web Folders. However, inconsistencies in implementation and fairly low exposure meant that comparatively few people were aware of WebDAV support or even understood what it was useful for, and it disappeared at one point in IE to emerge with IE7.  To enable such as Web Folders/WebDAV "folder", click on Add Network Places from the Networks node in Explorer, select Choose Another Network Location, provide the URL (including port if the port is not specifically Port 80) to the WebDAV server, provide a user name and password for the WebDAV server, then establish a name for this server. Once added in this manner, you can open it up just as you would any network link through the graphical windows, or you can map the server to a drive letter (such as w:). Once done, you can open up an Explorer Window and type in the drive name in the navigator bar "w:\", and the root of the database will be displayed.

Linux is a little more complex. While there are a number of options, one of the best is to use DAVfs2 at http://savannah.nongnu.org/projects/davfs2. This particular application let's Linux treat the XML Database as if it were a standard NFS drive, effectively making the network connections transparent to the user.

Setting it up under Ubuntu is relatively simple:

1) First, get the DAVfs2 code:

$prompt> sudo apt-get install davfs2

2) The default install doesn't set the suid code for letting other users access the resources, so if you want others beyond root to have access to the directory, you should run dpkg-reconfigure app:

$prompt> sudo dpkg-reconfigure davfs2

and answer "Yes" to the question about whether users beyond root have access.

3) The next step is to insure that there is a davfs2 group that can be added into a user's group permissions. In the file /etc/davfs2/davfs2.conf file, uncomment the group file line:

#dav_group       davfs2            # system wide config file only

so that it reads:

dav_group       davfs2            # system wide config file only

4) In your user manager, add the davfs2 group to the users who need access to the database, including, presumably, yourself.

5) Once set up, you then need to add the webDAV repository to your /etc/fstab file:

Create the folder where you want to add your WebDAV "file system". One recommendation is to create a new folder in the media folder (/media/xmlwebdav, for instance).

$prompt> sudo mkdir /media/xmlwebdav

6) Edit the /etc/fstab file to add the WebDAV repository connection and assign it to the newly created folder, by adding the following line (adjusted for your own content):

http://localhost:8180/exist/webdav/db /media/xmlwebdav davfs rw,noauto,user 0 0

7) From there, you should be able to mount it using the mount command as an ordinary user:

$prompt> mount /media/xmlwebdav

Once mounted, you can type in /media/webdav in konquerer or Dolphin (KDE's new file manager) and it will look (and act) for the most part as if it were part of your filesystem. You can then add files to specific collections by dragging them to the corresponding folder for that collection, you can double click on an XML or XQuery file in order to have it open up in your assigned viewer/editor, and you can delete files (and collections) simply by selecting them and pressing the delete key. Note that if you delete those resources, such files are effectively gone - there is no Trash intermediate state equivalent.

Beyond the obvious ease of use factor, there's one more, perhaps far more important benefit. You can use tools such as David Lee's xmlsh to run XQueries, XProcs or other shell operations directly on the resources in the database, can invoke the xqueries in the database from the command line, and otherwise do some truly amazing XML processing. I'll leave specific examples of this for another posting.

WebDAV support through DAVfs2 or IE Web Folders should be seen as one of the more valuable tools in the XML developer's toolbox, regardless of which XML database you use. Especially in conjunction with XML editors such as oXygen 11 or XML Spy, or XML processing tools such as xmlsh or Calabash, WebDAV can radically simplify XML application development and maintenance.

This is part of the recovery effort from the older XMLToday.org site, and was first published Nov 8, 2009

__________________

Kurt Cagle

Managing Editor

XMLToday.org