doing useful work with XML and open-source software

]> doing useful work with XML and open-source software Aaron Crane aaron.crane@gbdirect.co.uk GBdirect Ltd. http://xmlsucks.org/ Presented at LinuxWorld 2003 overview

What is XML?
Problems with XML
You have to use it anyway
Tools to cope with XML

what is XML?

XML purports to be a simple, vendor-neutral textual external representation for hierarchically-structured data
- Reasonably accurate ‐ except for the simplicity bit
Any arbitrary data can be expressed with a hierarchical representation
- So XML can represent any data whatsoever
The big win claimed for XML:
- One piece of software can process any XML document, and therefore any data
- Thus making XML ideal for data interchange

what XML looks like

A sample XML document:


     
      
       Monty Pythonpython@different.org
      
      
       Frank Sinatrachairmain@board.net
      
      
       Stop that, it's silly.
      
     
    ]]>

parts of an XML document

]]> is an XML declaration
python@different.org]]> is an element
- <email> is the element's start tag
- </email> is the element's end tag
- The text python@different.org is the element's content
date="2003-01-22" in the memo start tag is an attribute of the memo element
There's exactly one top-level element in a document

well-formedness and validity

Every XML document is well-formed (by definition)
- Means essentially that all the elements are properly nested within each other, without overlapping
Some XML documents are also valid
A valid document is declared to meet a Document Type Definition (DTD)
The DTD can make additional constraints on the (logical) structure of the document:
- What elements are permissible
- What attributes they can have
- What elements and other content a given element can contain

problems with XML

Verbosity
- XML documents are frustratingly and unnecessarily verbose for human authors
- Also implies high costs for storage and bandwidth (in the absence of compression)
Complexity
- What little XML offers is more complex than it could be
Oversimplification
- XML per se is too simplistic to do what people actually need
- This leads to a huge number of other technologies to work around deficiencies in XML
- All of them put together are vastly more complicated than a reasonable solution would have been

verbosity

XML is hideously verbose for humans ‐ like SGML, but worse
The earlier memo example could be written as a plain text email
- Would be less verbose
- But needs a special-purpose processing tool
- Also doesn't encode as much information
Many alternative bracketed notations have been proposed
- Avoid repeating the element name in the closing bracket

complexity

Three main areas of unnecessary complexity in XML:
1. No guidance on data design ‐ should you use attributes or content?
2. Ridiculously hard to parse
3. No clear view of what XML is meant to accomplish
  - Is it for humans or machines?

data design

Designing XML data structures is harder than it ought to be
- Unnatural distinction between element content and attributes

Which of these structures should you choose?

...
     ...
     12345...
     ...
     invoice12345...
     
       invoice
       12345
       ...
     
    ]]>

attributes or content?

Many people try to use content for &lq;data&rq; and attributes for &lq;metadata&rq;
- So metadata is never structured?
Others, recognizing that problem, tend to use content universally
- For them, attributes clutter up the standard without offering any benefit

parsing

One of the design goals of XML was that XML processing programs be easy to write
- The working group wanted a typical Computer Science graduate to be able to write an XML processor in a week
Quite clear that XML does not meet this goal
- Many items of XML syntax not mentioned here
Hard to find an XML parser which combines all of:
- Completeness (including validation)
- Correctness
- Run-time efficiency

humans and machines

Many of my complaints are about the suitability of XML for humans
Some people have countered that &lqq;XML is for programs, not humans&rqq;
Debatable
- Design goal that &lqq;XML documents should be human-legible&rqq;
- Some physical structures that can only be useful for human authors: general entities, empty-element tags
Too much syntactic latitude for truly simple programs
- But too little for human authors
Lack of clear goals makes it hard to decide whether and how XML should be used in your organization
- XML is optimized neither for humans nor for computers

oversimplification

The base XML specification explicitly refuses to consider any issues except data representation
For example, it has nothing to say about:
- Appropriate ways of processing XML documents
- How to constrain and validate an XML document in ways that can't be described with DTDs
- Combining content from multiple XML vocabularies into a single document
- How to transform one XML document into another
- Rendering XML data using existing technologies

oversimplification causes acronym proliferation

The existence of these limitations is old news
- So others have written add-on specifications to deal with XML's deficiencies
Learning the over-complex XML itself isn't enough for using XML to conduct your business
Also have to acquire familiarity with some or all of a bewildering array of additional technologies
- Some of which overlap in scope or even contradict each other

processing technologies

Two semi-standard interfaces for XML processors (parsers) to present to applications
- DOM (Document Object Model)
  - The parser reads the entire document into an in-memory tree structure
  - It provides a wide variety of methods for examining or changing individual parts of the tree
- SAX (Simple API for XML)
  - A streaming model: the parser reads the data bit by bit
  - It tells the application whenever it finds a meaningful unit (start tag, end tag, chunk of text, etc.)

improved validity checking

DTDs are extremely problematic
- Non-XML syntax, so you can't manipulate them with XML tools
- Extremely limited in what constraints they can specify
- Incredibly intricate rules for how non-validating parsers should handle DTDs
Many alternative validity-checking technologies:
- W3C Schemas (probably the most widespread)
- RELAX (Regular Language Description for XML; obsolete)
- RELAX-NG (Next Generation)
- TREX (Tree Regular Expressions for XML)
- Schematron
But only DTDs will let you use entity references

combining multiple vocabularies

There is a Namespaces specification
- Each element in a document is associated with a namespace
- Each namespace is identified by a unique URL

A simple example:


      Monty Python
      Cheese Seller
      
       
        ...
       
      
     
    ]]>

Tricky to combine Namespaces with some validity checking technologies

transformation

Extremely common to need to transform one XML document into another
- Exchange invoices, purchase orders, etc. as XML documents conforming to a standardized vocabulary
- But leave your internal data in some company-specific format
Two options for transformations:
1. Write a program in your language of choice that uses a SAX or DOM parser and manipulates the document as necessary
2. Use XSLT: XSL Transformations
Simple transformations in XSLT are trivial (if obscenely verbose)
More interesting ones can be absurdly hard

rendering

CSS (Cascading Style Sheets) can be used directly with XML documents
Alternatively, transform your XML document into something else
- XSLT was designed for translating arbitrary XML documents into XSL-FO (XSL Formatting Objects) documents
- XSL-FO is a fairly complete XML-based document rendering language
  - Would that any of the available implementations were equally complete&ellipsis;
- XSL-FO processing applications typically handle conversion into HTML and/or PDF
Translating to non-XML markup languages for typesetting (like LaTeX) can be difficult

doing it anyway

XML qua technology almost seems to be a bad joke
But you have to do it anyway
Two main reasons:
1. It's (just) good enough technologically
2. More importantly: everyone else is doing it

worse is better

XML has many flaws
But the æsthetic appeal of the technology is comparatively uninteresting
In practice, XML's flaws can be lived with
- An example of &lq;worse is better&rq;: http://www.jwz.org/doc/worse-is-better.html
- Since XML is good enough, it isn't worth replacing it

a bigger boy made me do it

Huge numbers of businesses already rely on XML
- Not least because it is mandated by governments and by large corporations at the top of industry-wide supply chains
All the evidence suggests that XML will become ubiquitous (if not unavoidable) in the near future
XML is already being used to accomplish hard tasks ‐ XSL-FO and SAX Machines come to mind
Though adopting XML technologies implies significant costs for businesses, the costs of not using XML may well be bigger

how to cope with XML

The technological problems can be managed
The verbosity is merely an annoyance
XML's one big advantage is that most of the innate complexity can be dealt with once and for all
The hard part is the complexity-through-oversimplification
- Even here, the situation is improving
- It's becoming more obvious which XML technologies are really important and which are dead ends
Several specific tools and technologies for getting stuff done

choosing technologies

Which (if any) XML add-ons should you use?
Namespaces are vital for combining data from disparate sources
- Reused over and over in the useful technologies
DTDs are a waste of effort except in certain specific situations
- Good to let human authors use DTD-aware editing tools
- Stick to W3C Schemas for validation
XSLT is fairly flexible and conceptually elegant
- But your staff will need functional-programming experience to do interesting transformations
- The awkward syntax makes life harder
What domain-specific XML vocabularies and standards are used in your field?

choosing fundamental libraries

Avoid most of the Java tools unless you have a burning desire to buy lots of very fast hardware
Use GNOME's libxml as the underlying parser in your programs
- Written in fast, portable C
- Reasonable DOM- and SAX-like interfaces
- Bindings for many popular languages (Python, Perl, etc.)
- Reads legacy HTML documents as XHTML
Similarly, use GNOME's libxslt as an XSLT engine
- Screamingly fast ‐ especially when compared to the well-known Java equivalents
- Also written in C, with bindings available for the language of your choice

additional tools

Barrie Slaymaker's SAX-Machines library for Perl
- Strongly reminiscent of the Unix pipeline/filter approach
- Write small, simple processors using the SAX interface
- Connect them together in arbitrary ways using SAX-Machines
- Similar tools include AxKit, a web application framework using Apache and mod_perl
XSL-FO looks extremely promising as a way of automating document production, but:
- Currently only two open-source implementations: one in Java, one in (of all things) TeX
- Both incomplete and somewhat slow
- Probably advisable to find an alternative in the short term

conclusions

XML does have a number of technological problems, but:
- The technological problems can be managed
- The political and commercial advantages of working with the same open standard as everyone else are enormous
There are open-source and Linux-friendly tools which actually help you get your work done in an XML-besotted world, including:
- GNOME libxml and libxslt
- SAX-Machines or similar systems
XML is here to stay