Submitted by James Fuller on Thu, 02/04/2010 - 02:22
Posted in
Recently, I remarked that JSON vs XML arguments somewhat invalid ... mainly due to the fact that JSON is applicable to a much smaller subset of scenarios then XML.
The example I use to illustrate JSON's constraints was too imagine creating a database where the internal representation (no not binary on disk persistence) was JSON ... which probably implies that one would be using some javascript to query and generate views from the database; something I don't want to contemplate.
The fact remains that JSON is a powerful 'over the wire' format and when the endpoint is a javascript client then why not make it easy to 'turn a switch' and apply a late stage optimization ?
After all there are a lot of other 'document' orientated databases exposing JSON interfaces so I decided I would try to turn eXist database into a serious JSON serving platform. As the title of this article alludes, I dont get there in one step .... and seek people's opinions as to what would work for them.
In eXistdb we already have a JSON xquery library that does a pretty good job at converting xml to json data structure. Its a naive implementation in that there maybe some corner cases that we have not covered but its a good starting point for our purposes.
The JSON library is bundled in eXist source code ( I may elect to extrude this into its own extension package at some point) and can be found here
eXist/src/org/exist/xquery/lib/json.xqTo use you will need to import the JSON library then invoke the json:xml-to-json($xml) function, where $xml is the xml fragment you want to convert to JSON goodness.
To demonstrate you can goto the eXist website sandbox and cut n paste the following code snippet:
import module namespace json="http://www.json.org";
declare option exist:serialize "method=text media-type=application/json";
let $xml := <root><test>1</test></root>
return
json:xml-to-json($xml)
You should get the following result when executed in the sandbox.
{"test" : "1"}Cool ... but what I really want is for eXist's RESTful server to deliver JSON versus me having to write an XQuery code wrapper to output. A 'poor mans' version of this could take advantage of eXist RESTful server's _query url param. By appending _query to any HTTP GET request you can apply XQuery code.
Try this link which uses the _query URL param to call json:xml-to-json on the XML URL ... this approach has 'hack' written all over it but closer to what I want with respect to just taking an existing XML in the database, flipping a switch and getting a JSON expression.
So we can now generate JSON but there has to be a cleaner way of achieving this ?
Here are some of my ruminations ....
- use content-type to return JSON format e.g. when a HTTP GET requests a resource with the application/json content type
- investigate adjusting eXist REST server to apply extra xquery step when it receives HTTP GET request (with application/json)
- add an XProc extension step to convert xml to json but this does not solve the HTTP part of the story fully
- investigate creating a new serializer for JSON, but this would mean I dont use existing json.xq library
As ever, I am starved of time so the last option does not fit in with my current schedule so I probably will be looking at amending the REST server ... but before I proceed with Part II would like to hear from people with their own thoughts of how it might be done.
ps: thanks to Kurt for letting me explicate on XML Today, not having any blog setup these days probably means I will post my short notes here for the time being
- James Fuller's blog
- Add new comment
- Quote
- 512 reads
Re: Turning eXist db into a JSON server - Part I
One of the appeals of JSON is that you can use it to drive Ajax frameworks, like Dojo. However it is not possible to create a reversible mapping between some arbitary bit of XML and JSON. If you want to use such a framework you need to be able to create the structures that it expects, in this case eXists's json:xml-to-json($xml) has some problems. One example is the handling of arrays. xml-to-json creates an JSON array when and only when a node is repeated; meaning you can not create an empty or single element array -which is often what the framework is expecting. In my experience a small amount of custom Xquery to create the JSON format for the target system is better than a standard routine.
Re: Turning eXist db into a JSON server - Part I
yes, I did mention that json.xq in its current state has some problems ... the best approach would also be streamable which is why we were initially leaning to go down a serializer approach but I think we can coerce json.xq to do what is right in most of the cases.
Re: Turning eXist db into a JSON server - Part I
Welcome here to James Fuller
Do not hesitate to publish here your technical articles
Like James, I do not have a personal blog. (I had some but I stop to put anything). If James agrees. here I will ask him questions about eXist. I will also write here the report of my experiences with eXist DBRe: Turning eXist db into a JSON server - Part I
W3C XForms Work Group is also concerned by JSON and XSLTForms is written in, both, XSLT and Javascript.
I recently experimented JSON in an XSLTForms extension for cross-domain API use, such as the one Wikipedia provides (here is a demo link: http://www.agencexml.com/jsoncallback/wikipediasearch.xml).
I can optimize XSLTForms so JSON data will be loaded faster than XML serialization.
I see in your example that the name of the "root" element is not translated in JSON. As a matter of fact, XSLTForms just adds a "root" element when converting from JSON... Arrays are also to be considered and XSLTForms will also add "array" and "item" elements when necessary...
Do you know if JSON to XML and XML to JSON rules have been standardized ?
XSLTForms and eXists can be synchronized about JSON support, don't you think ?
Thanks!
-Alain
Re: Turning eXist db into a JSON server - Part I
Jim,
I'm truly glad to see your article posted here, both because I'm glad to see you posting and because the topic is especially timely. I'd tend to concur that there are a few problems (not big ones, just preferences) with the xml-to-json serialization, but for the most part these aren't show-stoppers - they just indicate a need to do some intermediate restructuring of data to optimize the json output.
I've been looking at the jQuery library, and one of the more intriguing plugins there is jquery.populate()(http://plugins.jquery.com/project/Populate), which lets you take a json object and automatically populate form elements (or even read-only divs) that have the same id names as the hash keys. You can take the inverse approach of repopulating the json object by reading the keys of the object and retrieving the input values of the field items with those ids. While this is far from ideal (arrays for instance are highly problematic) it's useful for quick and dirty property sheets.
I think that a useful corresponding function with the json: lib in eXist would be a json:json-to-xml inverse function. This would create an arbitrary <root> element, then use id tokens as element names. Again, not ideal, but useful enough in a number of circumstances, and more dependable than trying to encode the json-to-xml on the client.
Re: Turning eXist db into a JSON server - Part I
Hi Jim,
Thanks for the article. I've done a bit of work parsing JSON into XML and back again:
http://snelson.org.uk/archives/2008/02/parsing_json_in.php
I think it's a much more promising idea to create a specific XML structure which is serialized in an expected way into JSON, rather than to use a best-guess mapping from arbitrary XML into JSON, and attempt to tweak your XML until the best-guess algorithm gets it correct.
The advantage of the mapping that I devised is that it is 1-1 and lossless. As such it can easily serve as an intermediate for mapping to and from the actual XML structure required. I think any other solution is going to end up being unsatisfactory for a large number of users.
John
Re: Turning eXist db into a JSON server - Part I
wise words indeed and gives me a few ideas, thx J