Programming With the Java XML Digital Signature API

One of the significant new features of the Java Platform, Standard Edition 6 (Java SE 6) is the Java XML Digital Signature API. This API allows you to generate and validate XML signatures. XML signatures are a standard for digital signatures in the XML data format, and they allow you to authenticate and protect the integrity of data in XML and web service transactions. This article will give you an overview of XML signatures and show you how to use the API in your applications. Overview of XML Signatures What is a digital signature? RFC 2828 defines a digital signature as "a value computed with a cryptographic algorithm and appended to a data object in such a way that any recipient of the data can use the signature to verify the data's origin and integrity." JDK 6 includes a cryptographic digital signature API that is described in more detail in a lesson on the security trail in the Java Tutorial. An XML signature is a digital signature with several key properties. It defines a process and a format for generating digital signatures in the XML format, and it has many additional features. For instance, it allows you to sign more than one piece of data -- in binary or XML -- and to use any underlying cryptographic signature algorithm. An XML signature can sign arbitrary data, whether it is XML or binary. It can also sign only a portion or a subset of an XML document rather than the entire document. The data to be signed is identified by Uniform Resource Identifiers (URIs). XML signatures are often described as being of one or more of three types: A detached signature is over data that is external to the Signature element. This could be data outside of the document, such as a web page retrieved by way of HTTP, but it could also be data that is in the same document, such as a sibling element of the signature. An enveloping signature is over data that is inside the Signature element. An enveloped signature is a signature that is over data that contains the Signature element itself, such as the entire document. Perhaps the best way to describe an XML signature is to step through the contents of an example in detail. The example that this article will use is an enveloped XML signature generated over the contents of an XML document, a sample purchase order. The article will also use this sample in the subsequent sections on using the API. XML Sample 1 shows the contents of the purchase order before it is signed. XML Sample 1 Video Game 10.29 My Name One Network Drive Burlington MA United States 01803 The resulting enveloped XML signature, indented and formatted for readability, appears in XML Sample 2. XML Sample 2 Video Game 10.29 My Name One Network Drive Burlington MA United States 01803 tVicGh6V+8cHbVYFIU91o5+L3OQ= dJDHiGQMaKN8iPuWApAL57eVnxz2BQtyujwfPSgE7HyKoxYtoRB97ocxZ 8ZU440wHtE39ZwRGIjvwor3WfURxnIgnI1CChMXXwoGpHH//Zc0z4ejaz DuCNEq4Mm4OUVTiEVuwcWAOMkfDHaM82awYQiOGcwMbZe38UX0oPJ2DOE= CN=My Name,O=Test Certificates Inc.,C=US MIIB9zCCAWCgAwIBAgIERZwdkzANBgkqhkiG9w0BAQUFADBAMQswCQYD VQQGEwJVUzEfMB0GA1UEChMWVGVzdCBDZXJ0aWZpY2F0ZXMgSW5jLjEQ MA4GA1UEAxMHTXkgTmFtZTAeFw0wNzAxMDMyMTE4MTFaFw0zMTA4MjUy ... Note that the Signature element has been inserted inside the content that it is signing, thereby making it an enveloped signature. XML Sample 3 shows the SignedInfo element that contains the information that is actually signed XML Sample 3 tVicGh6V+8cHbVYFIU91o5+L3OQ= The CanonicalizationMethod element defines as a URI the algorithm used to canonicalize the SignedInfo element before it is signed or validated. Canonicalization is the process of converting XML content to a physical representation, called the canonical form, in order to eliminate subtle changes that can invalidate a signature over that data. Canonicalization is necessary due to the nature of XML and the way it is parsed by different processors and intermediaries, which can change the data in such a way that the signature is no longer valid but the signed data is still logically equivalent. Canonicalization eliminates these permissible syntactic variances by converting the XML to a canonical form before generating or validating the signature. The SignatureMethod element defines as a URI the digital signature algorithm used to generate the signature, in this case the PKCS#1 RSA-SHA1 algorithm as described in RFC 2437. One or more Reference elements identify the data that is signed. Each Reference element identifies the data by way of a URI. The example in XML Sample 3 contains a single Reference element, and the URI is the empty String, "", which indicates the root of the document -- in other words, the whole document. The Reference URIs could also point to external data, such as "http://java.sun.com", or to references within the same document, such as "#purchaseOrder". The optional Transforms element contains a list of one or more Transform elements, each of which describes a transformation algorithm used to transform the data before it is digested and signed, or validated. This example contains one Transform element for the enveloped transform algorithm. The enveloped transform is required for enveloped signatures so that the Signature element itself is removed before calculating the signature value. Otherwise, the signature would include itself in the data to be signed, which is not correct. Another example of a useful transform algorithm is the XPath Filter transform, which allows you to specify an XPath expression that selects a subset of nodes to be signed. The DigestMethod element defines as a URI the algorithm used to digest the data, in this case, SHA1. The DigestValue element contains the actual base64-encoded digest value. The SignatureValue element contains the base64-encoded signature value of the signature over the SignedInfo element, as XML Sample 4 shows. XML Sample 4 dJDHiGQMaKN8iPuWApAL57eVnxz2BQtyujwfPSgE7HyKoxYtoRB97ocxZ 8ZU440wHtE39ZwRGIjvwor3WfURxnIgnI1CChMXXwoGpHH//Zc0z4ejaz DuCNEq4Mm4OUVTiEVuwcWAOMkfDHaM82awYQiOGcwMbZe38UX0oPJ2DOE= The optional KeyInfo element contains information about the key that is needed to validate the signature, as in XML Sample 5. XML Sample 5 CN=My Name,O=Test Certificates Inc.,C=US MIIB9zCCAWCgAwIBAgIERZwdkzANBgkqhkiG9w0BAQUFADBAMQswCQYD VQQGEwJVUzEfMB0GA1UEChMWVGVzdCBDZXJ0aWZpY2F0ZXMgSW5jLjEQ MA4GA1UEAxMHTXkgTmFtZTAeFw0wNzAxMDMyMTE4MTFaFw0zMTA4MjUy ... The KeyInfo element can contain various kinds of content, such as X.509 certificates and Pretty Good Privacy (PGP) key identifiers. See the KeyInfo section of the XML Signature standard for more information on KeyInfo and the types of information it may contain. In this example, KeyInfo contains an X509Data element that contains an X509SubjectName element identifying the subject Distinguished Name of the signer's X.509 certificate and an X509Certificate element containing the signer's base64-encoded certificate. This certificate contains the public key needed to validate the signature. The KeyInfo section of the XML Signature Recommendation provides more information on the different KeyInfo types. It is important to note that the XML signature standard does not define how the recipient establishes trust in the key that is needed to validate the signature. The KeyInfo element is merely a collection of information that the recipient can use to help find and subsequently establish trust in that key. API Architecture The Java XML Digital Signature API was defined under the Java Community Process program as JSR 105. The API is designed to support all of the required or recommended features of the W3C Recommendation for XML-Signature Syntax and Processing. The API is based on the Java Cryptography Service Provider Architecture. This allows you to develop a service provider implementation of the API. Service providers implement a specific XML mechanism that identifies the XML-parsing mechanism that the implementation uses. The service provider in Sun's implementation of Java SE 6 supports the Document Object Model (DOM) mechanism. See the XML Digital Signature API overview for more information on service providers. The API contains six new packages, as Table 1 indicates. Table 1. New Packages in the Java XML Digital Signature API Package Contents javax.xml.crypto Contains common classes that are used to perform XML cryptographic operations. javax.xml.crypto.dom Contains DOM-specific classes for the javax.xml.crypto package. javax.xml.crypto.dsig Contains classes that represent the core elements defined in the XML digital signature specification. Of primary significance is the XMLSignature class, which allows you to sign and validate an XML digital signature. The XMLSignatureFactory class is an abstract factory that is used to create objects that implement these interfaces. javax.xml.crypto.dsig.dom Contains DOM-specific classes for the javax.xml.crypto.dsig package. javax.xml.crypto.dsig.keyinfo Contains classes that represent the KeyInfo structures defined in the XML digital signature recommendation. The KeyInfoFactory class is an abstract factory that is used to create objects that implement these interfaces. javax.xml.crypto.dsig.spec Contains classes representing input parameters for the digest, signature, transform, or canonicalization algorithms used in the processing of XML signatures. Generating an XML Signature This section will show you how to use the API to generate an XML signature over the contents of the PurchaseOrder element that the article introduced earlier. For this example, you will use DOM to parse the XML data that you will be signing. Code Sample 1 shows a few of the key steps in generating an XML signature: Code Sample 1 // Create a DOM XMLSignatureFactory that will be used to // generate the enveloped signature. XMLSignatureFactory fac = XMLSignatureFactory.getInstance("DOM"); // Create a Reference to the enveloped document (in this case, // you are signing the whole document, so a URI of "" signifies // that, and also specify the SHA1 digest algorithm and // the ENVELOPED Transform. Reference ref = fac.newReference ("", fac.newDigestMethod(DigestMethod.SHA1, null), Collections.singletonList (fac.newTransform (Transform.ENVELOPED, (TransformParameterSpec) null)), null, null); // Create the SignedInfo. SignedInfo si = fac.newSignedInfo (fac.newCanonicalizationMethod (CanonicalizationMethod.INCLUSIVE, (C14NMethodParameterSpec) null), fac.newSignatureMethod(SignatureMethod.RSA_SHA1, null), Collections.singletonList(ref)); The first step in the generation of an XML signature is to instantiate an XMLSignatureFactory mechanism. The getInstance method of the XMLSignatureFactory class looks for a service provider that supports DOM and returns an XMLSignatureFactory implementation from the provider with the highest preference. The XMLSignatureFactory is a key class in the API and, as shown in Code Sample 1, is used to assemble the different components of the XMLSignature. The second block of code in Code Sample 1 creates the Reference object, which identifies the data that will be digested and signed. The Reference object is assembled by creating and passing as parameters each of its components: the URI, the DigestMethod, and a list of Transforms. The third block of code in Code Sample 1 creates the SignedInfo object that the signature is calculated over. Like the Reference object, the SignedInfo object is assembled by creating and passing as parameters each of its components: the CanonicalizationMethod, the SignatureMethod, and a list of References. Code Sample 2 shows the steps involved in constructing the KeyInfo object. Code Sample 2 // Load the KeyStore and get the signing key and certificate. KeyStore ks = KeyStore.getInstance("JKS"); ks.load(new FileInputStream("mykeystore.jks"), "changeit".toCharArray()); KeyStore.PrivateKeyEntry keyEntry = (KeyStore.PrivateKeyEntry) ks.getEntry ("mykey", new KeyStore.PasswordProtection("changeit".toCharArray())); X509Certificate cert = (X509Certificate) keyEntry.getCertificate(); // Create the KeyInfo containing the X509Data. KeyInfoFactory kif = fac.getKeyInfoFactory(); List x509Content = new ArrayList(); x509Content.add(cert.getSubjectX500Principal().getName()); x509Content.add(cert); X509Data xd = kif.newX509Data(x509Content); KeyInfo ki = kif.newKeyInfo(Collections.singletonList(xd)); For this example, the signing key and certificate are stored in a KeyStore file. The first block of code retrieves the signer's X.509 certificate from the keystore. The second block of code creates the KeyInfo object, using a KeyInfoFactory, which is a factory for assembling KeyInfo objects. The KeyInfo object consists of an X509Data object containing the certificate and the subject Distinguished Name. Now you instantiate the document to be signed, create the XMLSignature object, and generate the signature, as Code Sample 3 shows. Code Sample 3 // Instantiate the document to be signed. DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance(); dbf.setNamespaceAware(true); Document doc = dbf.newDocumentBuilder().parse (new FileInputStream("purchaseOrder.xml")); // Create a DOMSignContext and specify the RSA PrivateKey and // location of the resulting XMLSignature's parent element. DOMSignContext dsc = new DOMSignContext (keyEntry.getPrivateKey(), doc.getDocumentElement()); // Create the XMLSignature, but don't sign it yet. XMLSignature signature = fac.newXMLSignature(si, ki); // Marshal, generate, and sign the enveloped signature. signature.sign(dsc); The Document now contains the Signature element. You can verify this by using the JAXP Transformer API to dump the contents of the document to a file, as Code Sample 4 shows. Code Sample 4 // Output the resulting document. OutputStream os = new FileOutputStream("signedPurchaseOrder.xml"); TransformerFactory tf = TransformerFactory.newInstance(); Transformer trans = tf.newTransformer(); trans.transform(new DOMSource(doc), new StreamResult(os)); Validating an XML Signature You will now learn to use the API to validate an XML signature over the contents of the PurchaseOrder element that you just signed. Code Sample 5 shows the key steps in validating an XML signature. Code Sample 5 // Find Signature element. NodeList nl = doc.getElementsByTagNameNS(XMLSignature.XMLNS, "Signature"); if (nl.getLength() == 0) { throw new Exception("Cannot find Signature element"); } // Create a DOMValidateContext and specify a KeySelector // and document context. DOMValidateContext valContext = new DOMValidateContext (new X509KeySelector(), nl.item(0)); // Unmarshal the XMLSignature. XMLSignature signature = fac.unmarshalXMLSignature(valContext); // Validate the XMLSignature. boolean coreValidity = signature.validate(valContext); First, you must find the location of the Signature element that you wish to validate. One way to do this is to use the DOM getElementsByTagNameNS method as shown in Code Sample 5. The second block of code creates a DOMValidateContext object containing a KeySelector object and a reference to the Signature element. The purpose of the KeySelector object is to obtain the public key using the information in the KeyInfo element and hand it back to be used as the validation key. The next section will discuss KeySelectors in more detail. The last two lines of code unmarshal and validate the signature. The validate method returns true if the signature is valid and false if it is invalid. If the signature is invalid, some additional code is necessary to determine the cause of the failure, as Code Sample 6 shows. Code Sample 6 // Check core validation status. if (coreValidity == false) { System.err.println("Signature failed core validation"); boolean sv = signature.getSignatureValue().validate(valContext); System.out.println("signature validation status: " + sv); if (sv == false) { // Check the validation status of each Reference. Iterator i = signature.getSignedInfo().getReferences().iterator(); for (int j=0; i.hasNext(); j++) { boolean refValid = ((Reference) i.next()).validate(valContext); System.out.println("ref["+j+"] validity status: " + refValid); } } } else { System.out.println("Signature passed core validation"); } The code in Code Sample 6 determines the cause of an invalid signature as one of two possibilities: An invalid signature. The cryptographic verification of the signature failed. This can be caused by an incorrect validation key or a change to the SignedInfo contents since the signature was generated. An invalid reference or references. The verification of the digest of a reference failed. This can be caused by a change to the referenced data since the signature was generated. Before moving on to the next section, it is important to note that transforms can change the contents of the data that is referenced before it is signed. Therefore, it may be important to show the contents of exactly what has been signed to the validating user. You can do this by enabling reference caching in the DOMValidateContext object before validating the signature and invoking the getDigestInputStream method of the Reference objects contained in the signature, as Code Sample 7 shows. Code Sample 7 valContext.setProperty("javax.xml.crypto.dsig.cacheReference", Boolean.TRUE); // Unmarshal the XMLSignature. XMLSignature signature = fac.unmarshalXMLSignature(valContext); // Validate the XMLSignature. boolean coreValidity = signature.validate(valContext); Iterator i = signature.getSignedInfo().getReferences().iterator(); for (int j=0; i.hasNext(); j++) { InputStream is = ((Reference) i.next()).getDigestInputStream(); // Display the data. } These and other security concerns are discussed in more detail in the security considerations section of the XML Signature Recommendation. The KeySelector Class A KeySelector is an abstract class that is responsible for finding and returning a key using the data contained in a KeyInfo object. In Code Sample 5, you passed an X509KeySelector object, which is a very simple implementation of KeySelector that looks for and returns a public key of an X.509 certificate, as Code Sample 8 shows. Code Sample 8 public class X509KeySelector extends KeySelector { public KeySelectorResult select(KeyInfo keyInfo, KeySelector.Purpose purpose, AlgorithmMethod method, XMLCryptoContext context) throws KeySelectorException { Iterator ki = keyInfo.getContent().iterator(); while (ki.hasNext()) { XMLStructure info = (XMLStructure) ki.next(); if (!(info instanceof X509Data)) continue; X509Data x509Data = (X509Data) info; Iterator xi = x509Data.getContent().iterator(); while (xi.hasNext()) { Object o = xi.next(); if (!(o instanceof X509Certificate)) continue; final PublicKey key = ((X509Certificate)o).getPublicKey(); // Make sure the algorithm is compatible // with the method. if (algEquals(method.getAlgorithm(), key.getAlgorithm())) { return new KeySelectorResult() { public Key getKey() { return key; } }; } } } throw new KeySelectorException("No key found!"); } static boolean algEquals(String algURI, String algName) { if ((algName.equalsIgnoreCase("DSA") && algURI.equalsIgnoreCase(SignatureMethod.DSA_SHA1)) || (algName.equalsIgnoreCase("RSA") && algURI.equalsIgnoreCase(SignatureMethod.RSA_SHA1))) { return true; } else { return false; } } } This is a very simple implementation of a KeySelector that returns the public key from the first X.509 certificate it finds in the X509Data. It is for demonstration purposes only and should not be used in real-world applications. A more complete X.509 key selector implementation would check other types of X509Data and establish trust in the validation key by using a keystore of trusted keys, or by finding and validating a certificate chain from a trust anchor to the certificate containing the public key. See the Java PKI Programmer's Guide for more information about trust anchors and Java APIs that you can use to establish trust in keys. Logging and Debugging The Java SE 6 implementation of the XML Signature API has extensive logging support that, when enabled, will provide you with additional information to help you debug validation failures. The log messages use the JDK logging facility, java.util.logging. To enable XML signature logging, you must first configure the logging facility so that the XML signature-logging messages are emitted. You can do this by editing the JRE's default logging.properties file directly, or by creating your own file and setting it with the java.util.logging.config.file property, for example: java -Djava.util.logging.config.file=logging.properties ... where logging.properties contains the following code: handlers= java.util.logging.ConsoleHandler .level= INFO java.util.logging.ConsoleHandler.level = FINER java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatter org.jcp.xml.dsig.internal.level = FINER com.sun.org.apache.xml.internal.security.level = FINER This will emit log messages of level FINER and higher to the console. All other components will emit log messages of level INFO and higher. This article will not describe every log message in detail, but Table 2 lists some of the most helpful messages. Table 2: Some Useful Log Messages Log Message Explanation [java] FINER: Pre-digested input: ... This message displays the content of the referenced data just before it was digested. This is useful for debugging reference validation failures. [java] FINE: Expected digest: ... [java] FINE: Actual digest: ... These messages display the expected and actual base64-encoded digest values of a Reference element. This is also useful for debugging reference validation failures. [java] FINE: Canonicalized SignedInfo: ... This message displays the canonicalized SignedInfo element before it is signed. This is useful for debugging canonicalization and signature verification failures. Conclusion The purpose of the article was to get you started with using the API and to show you the basic steps in generating and validating an XML signature. To learn more about the Java XML Digital Signature API, consult the documentation and references in the "For More Information" section. The Java XML Digital Signature API is available in Java SE 6, as well as in the GlassFish project. Project WSIT, also known as Project Tango, uses the Java XML Signature API to implement the Web Services Security (WSS) specification. For More Information Java XML Signatures: This article discusses XML digital signatures and the Java XML Signature API and discusses ways to speed up performance using cryptographic hardware accelerators. JSR 105 Java Community Process Java XML Digital Signature Overview and Tutorial Java XML Digital Signature API Specification Java PKI Programmer's Guide Sean Mullan's Blog

Install XAMPP for easy, integrated development

Install XAMPP for easy, integrated development Middleware stacks like this could revolutionize open source, multi-tier software development

Level: Introductory Nils-Erik Frantzell (nfrantze@ucsc.edu)Computer Science Department, UC Santa Cruz 30 Nov 2004 Open source stacks such as XAMPP from Apache Friends are simplifying open source development by making it easier to write and distribute applications in a stable and standardized environment. Traditionally, AMPP -- Apache, MySQL, PHP, and Perl -- have all been installed and configured as separate products. The trend of combining them into integrated middleware stacks promises to make open source development more competitive with J2EE™ application development, at least for low-end applications. In this article, you'll learn how to install, configure, and back up XAMPP on Mandrake Linux™ 10.0 and also how to configure and administer XAMPP, as well as how to install your own applications in an XAMPP environment. The advent of Java 2 Enterprise Edition™ dramatically changed the software landscape by providing an integrated middleware stack that greatly simplified the task of writing and deploying Java™ applications. For a while, the open source community was left behind because it lacked a similar integrated architecture. Recently, with the introduction of integrated open source stacks like XAMPP from Apache Friends, this situation has started to change. These stacks are still quite simple and rudimentary when compared with J2EE, but they are nevertheless an important step on the way to fuller systems integration. PHP 5.0 (which makes PHP fully object oriented) is a good indicator that this trend will accelerate. The focus of this article is on one of the integrated, open source stacks: XAMPP from Apache Friends. Introducing XAMPP XAMPP is a full-featured AMPP (Apache MySQL, PHP, Perl) package that is one of the few non-commercial AMPP middleware stacks available on Linux. With its tight integration, XAMPP makes it possible to run anything from a personal home page to a full-featured production site (though only for development purposes; XAMPP is not meant to be used on a production server due to security issues). XAMPP really shines in the following areas: It is easy to install and set up. It contains a number of useful packages that make it easy to do things like generate traffic reports and accelerate PHP content. It has been thoroughly tested on the SUSE, Red Hat, Mandrake, and Debian Linux distributions, as well as on Windows® and Solaris. For this article, we will install XAMPP under Mandrake Linux 10.0. Let's start by looking at the default packages that come with XAMPP. Basic packages Basic packages include system, programming, and server software: Apache, the famous Web server MySQL, an excellent, free, open source database PHP, the programming language (in versions 4.3.8 and 5.0.1 at the time of this writing) Perl, the programming language ProFTPD, an FTP server OpenSSL, for secure sockets layer support Graphics packages XAMPP includes the following graphics-related packages: GD, the "Graphics Draw" library libpng, the official PNG reference library libjpeg, the official JPEG reference library ncurses, the character graphics library Database packages And what would an integrated stack be without some database packages such as: gdbm, the GNU implementation of the standard UNIX® dbm library SQLite, an extremely small, zero-configuration SQL database engine FreeTDS, a database library that gives UNIX and Linux programs the ability to talk to Microsoft® SQL and Sybase databases XML packages For XML development, XAMPP includes the following: expat, an XML parser library Salbotron, an XML toolkit libxml, an XML C parser and toolkit for GNOME PHP packages For PHP development, XAMPP includes the following: PEAR, the PHP library A pdf class that generates dynamic PDF documents with PHP TURCK MMCache, a PHP performance enhancer Other packages And finally, XAMPP demonstrates its versatility by including the following packages: zlib, a compression library mod_perl, which embeds a persistent Perl interpreter in Apache gettext, a toolset that assists GNU packages in producing multi-lingual messages mcrypt, an encryption program Ming, a Flash (SWF) output library Freetype2, a software font engine IMAP C-Client, a mail program API Now let's talk about installing XAMPP.

Use this link for more information

(http://www-128.ibm.com/developerworks/linux/library/l-xampp/)

Toyota plans ultra-inexpensive car

Cheap Wheels
22 January 2007 11:36:54
Toyota plans ultra-inexpensive car: report
TOKYO, Jan 22, 2007 (AFP) - Toyota Motor Corp. plans to build a low-cost car undercutting Renault's emerging-market Logan through a "radical" rethink in design and production, the president of the fast-growing Japanese automaker said.

"The focus is on low-cost technology," Toyota president Katsuaki Watanabe told Britain's Financial Times newspaper in an interview published Monday. He declined to set a price for a low-cost car but said it would be "at least" less than the Logan. Renault has started production of the Logan, which will cost from 5,000 euros (6,200 dollars) on up, touted as a budget model for consumers in emerging economies such as China and Russia that conforms to European standards. Watanabe said that Toyota could slash the price by targetting costs throughout production. "Everything from design to production methods will be radically changed and we are thinking of a really ultra-low-cost way of designing, using ultra-low-cost materials, even developing new materials if necessary," he said. The plan would create a new challenge to struggling US automakers.

Toyota is set this year to overtake General Motors as the world's largest automaker. The Japanese automaker has cashed in by pioneering environmentally friendly hybrid cars and has also seen success with its luxury Lexus line.

Use this link for more information


(http://www.lbo.lk/fullstory.php?newsID=1796333079&no_view=1&SEARCH_TERM=35 )

Jonathan Robie's XQuery Blog: XSLT vs. XQuery

XSLT vs. XQuery

About once a week, someone asks me a question about the relationship between XSLT and XQuery - is XQuery a better language than XSLT? when would I use each language in preference to the other? is there really any fundamental reason that XSLT can't be optimized for large data stores just like XQuery? which language is better for writing really large applications?

Here are some of the differences between the two languages.

• XQuery was designed to be optimizable for large data stores. XSLT was designed for the kinds of transformations used in stylesheets - it has come to be used much more broadly, but nobody has done a good job of translating it to efficient SQL or implementing it to work efficiently on native XML databases. You can argue whether the same kind of thing could be done in XSLT, but realistically, it's unlikely that anyone will ever do that.

• XQuery is typically easier to use for things that feel like "queries"; XSLT is easier for things that feel like transformations, especially formatting-oriented transformations on recursive data structures like those found in documents.

• XQuery has a keyword-oriented syntax that is more like other commonly used languages than the XML-based syntax of XSLT. I don't think this is a big deal, but some programmers seem to have a hard time getting past XSLT syntax.

I have heard Mike Kay suggest that XSLT is much better for designing very large queries, and heard Jason Hunter suggest that these kinds of queries are much easier in XQuery. Personally, I am more comfortable writing very large XQueries than very large stylesheets, but I've also written a lot more large queries than large stylesheets.

There are XSLT zealots and XQuery zealots who insist that one of these two languages is best. I use both languages, and like both languages. But XQuery is clearly the language that was designed for large data stores, or for the kind of implementation that translates a query to other query languages such as SQL.

By the way, Mike Kay's useful XTech 2005 paper examines these questions in much more depth.

O'Reilly -- Top Ten Java and XSLT Tips

Top Ten Java and XSLT Tips

by Eric M. Burke
08/29/2001

My new book, Java and XSLT, examines techniques for using XSLT with Java (of course!). This article highlights ten tips that I feel are important, although limiting the list to ten items only scratches the surface of what is possible. Most of these tips focus on the combination of Java and XSLT, rather than on specific XSLT (Extensible Stylesheet Transformations) techniques. For more detailed information, there are pointers to other valuable resources at the end of this article.

The basics of XSL transformations are pretty simple: one or more XSLT stylesheets contain instructions that define how to transform XML data into some other format. XSLT processors do the actual transformations; Sun Microsystems' Java API for XML Processing (JAXP) provides a standard Java interface to various processors. Here is some sample code that performs an XSL transformation using the JAXP API:

import javax.xml.transform.Source;
import javax.xml.transform.Transformer;
import javax.xml.transform.TransformerFactory;
import javax.xml.transform.stream.StreamSource;
import javax.xml.transform.stream.StreamResult;
import java.io.*;

public class Transform {

   /**
    * Performs an XSLT transformation, sending the results
    * to System.out.
    */
   public static void main(String[] args) throws Exception {
       if (args.length != 2) {
           System.err.println(
               "Usage: java Transform [xmlfile] [xsltfile]");
           System.exit(1);
       }

       File xmlFile = new File(args[0]);
       File xsltFile = new File(args[1]);

       // JAXP reads data using the Source interface
       Source xmlSource = new StreamSource(xmlFile);
       Source xsltSource = new StreamSource(xsltFile);

       // the factory pattern supports different XSLT processors
       TransformerFactory transFact =
               TransformerFactory.newInstance();
       Transformer trans = transFact.newTransformer(xsltSource);

       trans.transform(xmlSource, new StreamResult(System.out));
   }
}

You can click here to download a small ZIP file containing this example, along with an XSLT stylesheet and XML data file. The included README file explains how to compile and run this example.

Use this link for get more information

(http://www.oreillynet.com/pub/a/oreilly/java/news/javaxslt_0801.html)

Make choices at runtime with XSLT parameters

Make choices at runtime with XSLT parameters Use parameters and conditionals in your style sheets

Level: Introductory Nicholas Chase (nicholas@nicholaschase.com), President, Chase and Chase, Inc. 01 Aug 2002 Extensible Stylesheet Langauage Transformations provide the ability to perform sophisticated manipulation of data as it is transformed from one form to another. You can increase their capabilites even further through the use of parameters that can be specified at runtime. This tip takes a basic look at using parameters and conditional statements in an XSLT style sheet. Note:This tip uses the Xalan XSL Transformation engine, but any XSLT processor will do. It assumes that you are familiar with XSL transformations. Use this link for get more information (http://www-128.ibm.com/developerworks/xml/library/x-tipxsltrun/index.html)

A meaningful Web for humans and machines, Part 2: Explore the parallel Web

Explore the parallel Web

Can you present two formats with one data source?

In this series of articles, we present a thorough, example-filled examination of the existing and emerging technologies that enable machines and humans to easily access the wealth of Web-published data. In this article, we examine the concept of the parallel Web and look at two techniques that Web content publishers use to put both human-readable and machine-consumable content on the Web: the HTML link element and HTTP content negotiation. With these two techniques, content consumers can choose among a variety of different formats of the data on a Web page. Review the history of the techniques and how they are currently deployed on the Web, and how you might use the parallel Web to integrate calendar, banking, and photo data within an example scenario, MissMASH. Finally, we evaluate the parallel Web and determine that, while these techniques are mature and widely deployed, there are disadvantages to separating machine-readable data from the corresponding human-readable content.
");
}
}
}
//-->
In this article, we will cover our notion of the parallel Web. This term refers to the techniques that help content publishers represent data on the Web with two or more addresses. For example, one address might hold a human-consumable format and another a machine-consumable format. Additionally, we include within the notion of the parallel Web those cases where alternate representations of the same data or resource are made available at the same location, but are selected through the HTTP protocol (see Resources for links to this and other material).
HTTP and HTML are two core technologies that enable the World Wide Web, and the specifications of each contain a popular technique that enables the discovery and negotiation of alternate content representations. Content negotiation is available through the HTTP protocol, the mechanism that allows user agents and proxies/gateways on the Internet to exchange hypermedia. This technique might be mapped mostly to a scenario where alternate representations are found at the same Web address. In HTML pages, the link element indicates a separate location containing an alternate representation of the page. In the remainder of this article, we'll look at some of the history behind these two techniques, examine their current deployment and usage today, explain how they might be applied to help solve the MissMASH example scenario, and evaluate the strengths, weaknesses, and appropriate use cases for both.

History

The techniques necessary to weave a parallel Web have been available almost since the inception of the World Wide Web itself, and they were codified within the original HTTP/1.0 and HTML 1.0 specifications. However, even today their deployment, implementation, and usage varies. For example, the Apache Web server is the most widely used HTTP server on the Internet today (see Resources) and has had support for content negotiation for almost ten years. But few sites actually include content negotiation configurations and alternate representations of resources. In contrast, the number of Web pages that use the link element to point to alternate representations of pages has grown exponentially, especially with the advent of online journals and Weblogs. Let us briefly examine the origins of these two techniques so as to better understand how they have evolved and helped create a meaningful Web for both humans and machines.

Content negotiation

Since the Web was conceived for access by humans across the globe and from all walks of life, it had to provide mechanisms to access hypermedia in a wide variety of languages, character sets, and media types. For example, a visually impaired individual might configure his user agent (his Web browser, usually) to prefer plain text representations of images on Web pages. The plain-text representation would contain information that described the image and that could be used by screen reading software, possibly including details such as the location, names of individuals, or dates. On the other hand, a standard Web browser intended for sighted individuals would request image resources as specific image media types that the browser is capable of rendering (image/png or image/jpeg, for instance). The potential use cases for alternate encodings of information on the Web are as diverse as the populations of human beings served by the Web, and the HTTP specification recognizes this and provides content negotiation to address it.
Content negotiation as a phrase did not appear until the completion of the HTTP/1.1 specification, but the core of its functionality was defined in the HTTP/1.0 specification (see Resources for links to both) -- albeit in a rather underspecified format. The HTTP/1.0 specification provided implementors with a small set of miscellaneous headers: Accept, Accept-Charset, Accept-Encoding, Accept-Language, and Content-Language. (We will go into more technical detail of these headers and their usage in the next section of this article.)
Historically, it is important to note that in its original form, the content negotiation mechanism left it completely up to the server to choose the best representation from all of the combinations available given by the choices sent by the user agent. In the next (and current) version of content negotiation, which arrived with HTTP/1.1, the specification introduced choices with respect to who makes the final decision on the alternate representation's format, language, and encoding. The specification mentions server-driven, agent-driven, and transparent negotiation.
Server-driven negotiation is very similar to the original content negotiation specification, with some improvements. Agent-driven negotiation is new and allows the user agent (possibly with help from the user) to choose the best representation out of a list supplied by the server. This option suffers from underspecification and the need to issue multiple HTTP requests to the server to obtain a resource; as such, it really hasn't taken off in current deployments. Lastly, transparent negotiation is a hybrid of the first two types, but is completely undefined and hence is also not used on the Web today.
Amidst this sea of underspecification, however, HTTP/1.1 introduced quality values: floating point numbers between 0 and 1 that indicate the user agent's preference for a given parameter. For example, the user agent might use quality values to indicate to the server its user's fluency level in several languages. The server could then use that information along with the faithfulness of its alternate language representations to choose the best translation for that request and deliver it as per server-driven content negotiation.
The examples you've seen of content negotiation so far have highlighted its uses to serve the heterogeneous mix of people accessing the World Wide Web. But just as diverse humans can use content negotiation to retrieve Web content in a useful fashion, so also can diverse software programs request content in machine-readable formats to further their utility on the Web. You will see how to use content negotiation to further a meaningful Web for humans and machines later in this article.

The link element

As with content negotiation, the link element has gone through some evolutionary stages in order to meet the requirements for a parallel Web. In the first stages of the HTML specification, the link element only possessed a small number of attributes, which mainly served to indicate the relationship between the current page and the location of the resource to which the element's href attribute refers. Most of these relationships were originally designed for navigational functionality, but the link element's relationships were always meant to be extensible. Although the semantics and governance of these relationships had never been clearly defined officially, their usage has exploded in recent years. There's even now a process and a site at Internet Assigned Numbers Authority (IANA) to register new relationships (see Resources).
Most notably, the HTML 4.01 specification introduced new attributes that can indicate the media type and character set of the resource referenced by the link element. It is thanks to these additions that the link element has received wider adoption, especially by Weblogs. Later in this article, you will see that while many of the traditional use cases for the link element revolve around the needs of humans, the element is increasingly used to reference machine-consumable content in an attempt to further software's capabilities on the Web.
Back to top

Using link elements

To start, look at a series of example uses of the link element and see how its usage has evolved over time to provide machines with more accurate access to the data encoded in HTML pages. Listing 1 uses three of the standard link relationships defined within the HTML 4.01 specification (see Resources for all available link types) to give navigational hints to Web page readers. If publishers were to include this additional metadata in their pages, navigating certain types of documents, such as articles, books, or help manuals, would be much easier than the typical situation today, where users must fish for an anchor link, possibly labeled next, or possibly labeled something completely different.Listing 1. Navigational link elements
"http://www.w3.org/TR/html4/strict.dtd">







...the rest of the article...
In Listing 2, you see the most popular use of the link element today. Most modern browsers have a decent amount of support for the Cascading Style Sheets specifications, allowing publishers to separate structure from presentation in their HTML pages and making it easier to change their visual presentation without having to recode the entire page. This example in Listing 2 makes use of a new attribute: media, which specifies the intended destination medium for the associated style information. The default value is screen, but you can also use print, or, to target mobile devices, handheld. (See Resources for more on these values.) Beware: while you can use multiple values in the media attribute, they must be separated by commas, as opposed to the spaces you'd use for values of the rel attribute.Listing 2. Providing links to stylesheet information


media="screen,print"
type="text/css"
href="../dw.css">

...the rest of the article...
Our next example uses the internationalization capabilities of the HTML specification to give browsers or user agents language-specific navigational information in order to best match the reader with her preferred tongue. In the case of this article series, the authors combined can speak two languages fluently. If we had a large Spanish readership, we could provide those readers with a link to the article in Spanish by using the link element.
Notice a few of the attributes and their values that we use in Listing 3. First, we use the alternate link type to express the relationship between this article and our Spanish version. Because we are using it in combination with the hreflang attribute, we imply that the referenced Web page is a translated, alternate version of the current document. We also add a lang attribute to specify the language of the title attribute. Finally, we added a type attribute value, which is optional and is only there as an advisory hint to the user agent. (A Web browser might use this hint, for example, to launch the registered application that can deal with the specified media type.)Listing 3. Language-specific navigational link elements


lang="es"
title="La Red Paralela"
type="text/html"
hreflang="es"
href="es/redparalela.html">

...the rest of the article...
The previous examples mostly served to show the most common uses of the link element, but, with the exception of the language-specific use of link, they didn't really show you how to provide alternate representations of the data found on the original page. And even in the language example given in Listing 3, the alternate representation is intended for consumption by humans and not by machines. The final link example in Listing 4 shows how blogs use the link element to refer to feeds. A feed is an XML representation of its HTML counterpart that carries more structured semantics than the HTML contents of a blog as rendered for a person. Blog-reading software can easily digest an XML feed in order to group blog entries by categories, dates, or authors, and to facilitate powerful search and filter functionality. Without the link element, software would not auto-discover a machine-readable version of a blog's contents and could not provide this advanced functionality. This is a good example of how content generated by humans (even through blog publishing tools) is not easily understood by machines without the use of a parallel Web technique. The feed auto-discovery mechanism enabled by the link element helps humans co-exist with machines on the Web.
In Listing 4, you begin to see the power of the link element emerge to provide machines with valuable information in decoding content on the Web today. We again use of the "alternate" relationship to denote a new URLwhere you can find a substitute representation of the same content currently viewed. Through the use of media types, we can give the user agent the ability to choose the best understood format -- whether it be Atom or RSS feeds or any other media format.
The link elements shown in Listing 4 create a parallel Web with three nodes. First, at the current URL is an HTML representation of this article intended for human consumption. Second, the relative URL ../feed/atom contains a machine-readable version of the same content represented in the application/atom+xml format. And finally, the relative URL ../feed/rss contains a different machine-readable version of the same content, represented this time in the application/rdf+xml format.Listing 4. Providing auto-discovery links to feed URLs in blogs


title="Atom 1.0 Feed"
type="application/atom+xml"
href="../feed/atom">
title="RSS Feed"
type="application/rdf+xml"
href="../feed/rss">

...the rest of the article...
We must note that the values of the rel attributes are not limited to a closed set; the Web community is constantly experimenting with new values that eventually become the de facto standard in expressing a new relationship between the HTML page and the location specified in the href attribute. For instance, the pingback value used by the WordPress blog publishing system (see Resources) is now very widely used; it provides machines with a pointer to a service endpoint that connects Weblog entries with one another through their comment listings.
As simple REST-based Web services permeate the Web, we foresee a large number of rel values being defined to point to many different service interfaces. Such an explosion of uses for the link element greatly enhances the functionality of traditional Web browsers but also comes with other problems involving governance and agreement of relationship types that are beyond the scope of this article. Find more information on the HTML link element in Resources.
Back to top

Using content negotiation

We will cover content negotiation in somewhat less detail than the link element because of its possible complexity, but we'll go into enough detail to illustrate how it is used to create a parallel Web. In this discussion, we will focus on how content negotiation is configured using the popular Apache HTTP Web server software. Other Web server software works similarly, and the underlying concepts are the same.
Typically on the Web, URLs contain a file extension that indicate either the method used to produced the HTML (for example, .html for static pages, .php for PHP scripts, or .cgi for various scripting languages using the Common Gateway Interface) or the media format of the content at the URL (such as .png, .pdf, or .zip). While this approach is adequate in many cases, it requires that you create and maintain one URL for each different representation of the same content. In turn, this requires that you use some mechanism to link the multiple URLs together (such as the HTML link element described above, or the human-traversable a anchor link element) and that software know how to choose between the different URLs according to its needs.
Content negotiation allows different representations of data to be sent in response to a request for a single URL, depending on the media types, character sets, and languages a user agent indicates that it can (and wants to) handle. Whereas the link element constructs a parallel Web by assigning multiple URLs to multiple representations of the same content, content negotiation enables a parallel Web wherein multiple formats of the same content all are available at a single URL. The Resources in this article contain links to more detail on behind-the-scenes HTTP content negotiation.
Apache provides a content negotiation module that has two methods to select resource variants depending on the HTTP request: type maps and MultiViews. If a type map is used in the Apache configuration file, it is used to map URLs to file names based on any combination of language, content type, and character encoding. MultiViews are very similar, except that the administrator does not have to create a type map file; instead, the map is created on the fly based on other configurations specified throughout the server configuration document. For brevity, in this article we will only show an example of a type map, but please see the Resources for links to very detailed Apache content negotiation guides.
In Listing 5, we list the contents of a type map used to provide the mapping necessary to serve different versions of our article based on language and media type. In line 1, we specify the single URI used to serve multiple format and language representations of the same article. Lines 3 and 4 establish that by default we will serve the file parallelweb.en.html in response to all requests for the text/html media type. In lines 6 through 9, we have a special case in which we can serve the Spanish version of our article if desired by the user agent. Next, in lines 10 and 11, we provide a printed version of our article. Finally, you see in lines 13 and 14 the ability for software to retrieve an Atom feed version of our article. Content negotiation allows blog-reading software requesting our article in the application/atom+xml content type to receive a representation of the article that the software can understand more easily than the standard HTML version.Listing 5. Using type maps in Apache to serve multiple language and format representations
[1] URI: parallelweb
[2]
[3] URI: parallelweb.en.html
[4] Content-type: text/html
[5]
[6] URI: parallelweb.es.html
[7] Content-type: text/html
[8] Content-language: es
[9]
[10] URI: parallelweb.ps
[11] Content-type: application/postscript; qs=0.8
[12]
[13] URI: parallelweb.xml
[14] Content-type: application/atom+xml
In summary, this extra configuration on our server allows a single URL to be distributed for our article, while still permitting both human-friendly and machine-readable data to be available as long as the user agent requests it.
Now, let's move on to our example application and hypothesize how we might build it by using parallel Web techniques.
Back to top

Example: MissMASH using the parallel Web

In our series overview (see Resources for a link), we proposed an example application called MissMASH that would illustrate the different techniques covered in the series. The application's ultimate goal is for the user to gain access to personal information hosted in different locations, all displayed using a calendar-based interface. Photographs could be hosted on Flickr, online banking statements could come from Citibank, and scheduling details from Google Calendar. (See Resources for links to all three.) Our focus is not to walk you through actual running code for this application, but to highlight the main mechanisms that would facilitate the building of such an application using software not managed by any of the hosting providers of our sample data.
Please keep in mind that parts of our scenario will be, by necessity, fictitious due to the mere fact that not all of our sample data providers employ the techniques covered in this series. The location of the software that powers MissMASH does not matter either. It could be written as a traditional server-side Web application using any Web application framework du jour, or it could be a Web-based client-side application, making use of Ajax (the newly rediscovered XML HTTP request JavaScript object; see Resources) techniques to compose the data. In any case, the key to this kind of application -- and the key question that we are asking -- is how MissMASH finds, identifies, and uses the information necessary to accomplish its purpose. Ideally, the less information it needs from the (human) user, the better. Let's see how the parallel Web might allow us to build MissMASH.
One of the very first pieces of information needed by MissMASH is some sort of online calendar account information. As you might imagine, this might become unwieldy in no time if we required MissMASH to understand every possible public and/or private calendar service on the Internet. But if these services make use of parallel Web techniques, it is as simple as entering the URL you see in your browser when accessing that service into MissMASH. Consider, for example, Google Calendar. (For the purposes of this example, we'll ignore user authentication details.) Imagine that a user configuring MissMASH navigates in a Web browser to his Google Calendar. Once there, he copies the URL in his browser's address bar and pastes it into the MissMASH configuration. MissMASH would request that URL, parse the returned HTML, and find inside a gem of the parallel Web: a link element.
Google offers multiple versions of its online calendar: iCalendar (see Resources), Atom feeds, and HTML. Unfortunately, it only provides a link element to the Atom feed version of the calendar, but it's just as described in Listing 4. The same would occur for Flickr, which happens to also provide a link to an Atom feed of your most recent photos. In either case, MissMASH would get a second URL from the href attribute of the link element and would resolve that URL to receive a machine-readable representation of the user's event and photo information. Because these representations contain (among other data) dates and locations that software can easily recognize, MissMASH can use these alternate representations to render calendar and photo data together.
For illustrative purposes, let's also imagine that Citibank used content negotiation on the user's account URL. MissMASH can then negotiate with the Citibank server for any of the different supported formats, such as Atom feeds or the Quicken Interchange Format (see Resources)
We now have almost all of the information necessary to render our external personal data sources in MissMASH. Of course, for MissMASH to use the parallel Web in this way, it must understand both the structure and the semantics of the machine-readable representations that it retrieves. If MissMASH receives iCal data from Google Calendar, an Atom feed from Flickr, and Quicken Interchange Format data from Citibank, then it must understand how to parse and interpret all three of these machine-readable formats in order to merge the information. The parallel Web techniques do not themselves help resolve this matter; instead, the parallel Web's key benefit is that the user does not have to visit obscure configuration Web pages of the source-data services to find the URLs to one of the data formats supported by MissMASH. While MissMASH must focus on supporting a small number of different formats, this is far superior to requiring MissMASH to understand both a variety of data formats and a variety of application-specific, human-intended services for retrieving that data.
Future articles in this series will examine techniques to create a meaningful Web that impose a smaller burden on the data consumer to understand multiple machine-readable formats.
Back to top

Evaluation

As with most things on the Web, our conclusions about the evaluation categories we discuss in this section will not be clear cut. There are no black-and-white answers, which is why such a multitude of competing and complementary techniques exists. However, we will try our best to help you understand the reasoning behind our evaluation in order to understand what is required for machines to happily exist with humans on the Web. In these evaluations, we use the following summary key:
-- This technique fulfills most or all of the requirements of this evaluation criteria.
-- This technique fulfills some of the requirements of this evaluation criteria, but has some significant drawbacks.
-- This technique does not satisfy the requirements of this evaluation criteria.

Authoritative data

Both of the parallel Web techniques that we have presented involve a data consumer retrieving alternative representations of content originally targeted for human beings. In the case of the link element, the link between the human- and machine-readable formats is contained within the HTML page itself. Therefore, the relationship is endorsed by the content creator, and we can be confident that the alternative representation is a true representation of the data shown to humans through the HTML Web page. With content negotiation, the alternative representation is chosen and returned by the Web server that controls the content, and hence we can similarly rest assured that the machine-readable data is true to the semantics of the human-oriented content.
Because the parallel Web techniques do not constrain the specific formats of alternative data representations, in theory, these techniques might provide machine-readable data that is not only equivalent but even richer in content and semantics than the original human-friendly Web page. However, we must point out that most online services use this technique to link to Atom feeds, which often only capture a small amount of the original data in machine-readable formats. For example, an Atom feed of banking transactions might capture the date, location, and title of the transaction in a machine-readable fashion, whereas the (not widely used on the parallel Web) Quicken Interchange Format contains much richer semantic data, such as account numbers, transaction types, and transaction amounts.
Our evaluation: The parallel Web provides authoritatively sound data, but in common usage does not provide machine-readable data that completely represents the authoritative human-oriented content.

Expressivity and extensibility

The parallel Web techniques do not scale well for the purpose of expressivity and extensibility. These techniques achieve their extensibility by flexibly allowing for multiple alternative content representations. To accomplish this, however, either many URLs (in the case of the link element) or many server-side documents and their corresponding type maps (in the case of content negotiation) must be maintained. A publisher offering these techniques must also maintain code that generates each alternative representation from the underlying original data source (often data in a relational database).
All is not lost, however. Because the parallel Web techniques do not mandate one specific alternate representation, implementors are free to choose a representation that itself supports great extensibility and expressivity. In practice, however, the de facto formats used today on the Internet with parallel Web techniques do not offer much flexibility for extension to new data formats and richer semantics.
Our evaluation: The parallel Web is flexible enough to accommodate new data and new representations, but existing implementations do not scale easily and existing data formats are not particularly extensible.

Don't repeat yourself

The principle of "don't repeat yourself" (DRY) mandates that, if you must create both human- and machine-readable versions of data, you shouldn't need to maintain the two data formats separately. While a naive implementation of the parallel Web techniques might result in maintaining multiple files on a Web server that contain duplicate information, most real-world implementations derive the various representations from a single underlying data source (often a relational database).
In our evaluation: While the algorithms to generate the various data formats must be maintained, the actual data need only be represented a single time.

Data locality

This is one area where the parallel Web lacks our support. In a scenario that makes use of the link element, the human-friendly and machine-readable versions of the relevant information are found at two different resource locations. In scenarios that make use of content negotiation, the same URL is used for both, but the human-friendly and machine-readable content are still received completely separately. In both cases, it is not usually feasible to merge and match up the data within two very different formats (for example, HTML and the Quicken Interchange Format). As such, there is no way to determine which machine-readable data corresponds with which human-oriented visual elements. Consequently, there is no easy way for humans or software to select just a small amount of content (both human and machine content) for reuse in other situations.
Our evaluation: The parallel Web provides no data locality whatsoever.

Existing content fidelity

Any existing Web sites that already use content negotiation or the link element do so in a manner consistent with the parallel Web techniques described in this article. On the flip side, software that is aware of the parallel Web will not assign any additional meaning to Web pages than is already there, and therefore there is no danger of assigning incorrect semantics to existing content.
Our evaluation: The parallel Web is a technique already in use, and adopting it does not threaten the meaning of any current Web sites.

Standards compliance

By their very nature, our parallel Web techniques are standards compliant. The link element and content negotiation are specified in the HTML and HTTP specifications, respectively, and they are widely accepted by many development communities around the world. It's also important to note the standardization of MIME types at IANA and general agreement on many of the widely used formats on the Web today. However, what is not commonly standardized is the way in which developers should expose more specific information on the content of the linked resource besides the specified MIME type, such as more information on what the feed item contents are within an Atom feed.
Our evaluation: Developers and implementors can rest assured that the parallel Web is grounded firmly in accepted standards.

Tooling

The parallel Web makes heavy use of tooling to make itself attractive to developers. The Internet is filled with tooling that either directly or indirectly supports the parallel Web, making it very suitable for wider deployment. As you have seen, the Apache HTTP server contains extensive support for content negotiation; all other major Web server packages do as well. HTML editor software supports creating link elements, and Web browsers and blog-reading software supports their consumption. Additionally, given the specialty of each of the formats and the different communities supporting them, plenty of toolkits are always available in many different languages and licenses to do most of the heavy lifting during application development.
Our evaluation: The parallel Web has been around for a while and a great deal of mature tooling is available to produce and consume it.

Overall complexity

In general, the overall complexity of employing parallel Web techniques is moderate. The concepts are rather straightforward, and -- thanks to their standards compliance and the tooling available for them -- it is not very hard to implement the techniques. Additionally, there are several advantages to separating the specification of machine-readable information from the human-readable version:
As a designer, you are freed from the need to maintain both data and presentation.
You can optimize content to suit the needs of the target consumer
At the same time, we have already touched on a few areas in which the parallel Web is more complex than other approaches. Using the link element requires that you maintain multiple URLs for multiple data representations. In a world in which one "404 Not Found" error can be the difference between a sale for you or a sale for your competitor, the maintenance of numerous active URLs can be a burdensome task. Content negotiation itself requires the maintenance of moderately complex server-side configurations. And because both techniques support multiple representations of the same data, you must write and maintain code to generate these representations from the core data.
Overall evaluation: The parallel Web is widely understood and used today, but it does require a high amount of maintenance as time passes.
Back to top

Summary

The parallel Web has long been used to further humans' experiences on the World Wide Web, and recently it has also been used a great deal to power software's ability to consume data on the Web. The parallel Web acknowledges that humans and machines have different content needs, and it keeps those needs separate by allowing some representations for humans and others for machines. It maintains those heterogeneous representations in parallel, however, either through the proliferation of many URLs united at the (human-oriented) root by the HTML link element, or by using content negotiation to mask the representations behind a single URL. The techniques that comprise the parallel Web are widely available and commonly used, and will continue to be used for the foreseeable future.
Nevertheless, you've seen in this article that both of these approaches have shortcomings that, to date, have limited them to semantically poor use cases. Use of the link element requires maintenance of multiple URLs and requires multiple HTTP requests to get at machine-readable data. Content negotiation hides different representations at a single URL and makes it difficult to unite the human view of the content with the machine view. Further, the lack of uniformity across machine-readable representations and the lack of extensibility of Atom, one of the only common machine-readable representations in use today, hamper the adoption of the parallel Web in cases such as MissMASH in which the data being produced and consumed is not consistently structured.

In the rest of this series, we'll examine techniques that strive to achieve a meaningful Web for humans and machines without maintaining two (or more) strands of a parallel Web. These techniques begin with an HTML Web page and derive semantics from the content of that page. Thus, they share the benefits of not requiring multiple URLs for multiple representations and of containing both the human- and machine-targeted content within the same payload. You will see, however, that these techniques still differ substantially in their approaches, and have their own advantages and disadvantages, which we will evaluate.
In the next article, we'll examine the algorithmic approach, in which third parties use structural and heuristic techniques to derive machine-readable semantics from HTML Web pages targeted at people.

A meaningful Web for humans and machines, Part 1: How humans can share the wealth of the Web

Explore techniques for coexistence with human- and machine-friendly data

In this series of articles we'll examine the existing and emerging technologies that enable machines and humans to easily access the wealth of Web-published data. We'll discuss the need for techniques that derive the human and machine-friendly data from a single Web page. Using examples, we will explore the relationships between the different techniques and will evaluate the benefits and drawbacks of each approach. The series will examine, in detail: a parallel Web of data representations, algorithmic approaches to generating machine-readable data, microformats, GRDDL, embedded RDF, and RDFa.
In this first article, you meet the human-computer conflict, learn the criteria used to evaluate different technologies, and find a brief description of the major techniques used today to enable machine-human coexistence on the Web.

The World Wide Web empowers human beings like never before. The sheer amount and diversity of information you encounter on the Web is staggering. You can find recipes and sports scores; you share calendars and contact information; you read news stories and restaurant reviews. You can constantly consume data on the Web that's presented in a variety of appealing ways: charts and tables, diagrams and figures, paragraphs and pictures.
Yet this content-rich, human-friendly world has a shadowy underworld. It's a world in which machines attempt to benefit from this wealth of data that's so easily accessible to humans. It's the world of aggregators and agents, reasoners and visualizations, all striving to improve the productivity of their human masters. But the machines often struggle to interpret the mounds of information intended for human consumption.
The story is not all bleak, however. Even if you were unaware of this human-computer conflict, there is no need to worry. By the end of this series, you'll have enough knowledge to choose intelligently among a myriad of possible paths to bridge between the data-presentation needs of machine and human data consumers.

A Web for humans

In the early 1990s, Tim Berners-Lee invented HTML, HTTP, and the World Wide Web. He initially designed the Web to be an information space for more than just human-to-human interactions. He intended the Web to be a semantically rich network of data that could be browsed by humans and acted upon by computer programs. This vision of the Web is still referred to as the Semantic Web.
Semantic WebA mesh of information linked up in such a way as to be easily processed by machines, on a global scale. The Semantic Web extends the Web by using standards, markup languages, and related processing tools.
However, by the very nature of its inhabitants, the Web grew exponentially and gave priority to content consumable mostly by humans and not machines. Steadily, users' lives became more reliant on the Web, and we transitioned from a Web of personal and academic homepages to a Web of e-commerce and business-to-business transactions. Even as more and more of the world's most vital information flowed through its links, most of the Web-enabled interactions still required human interpretation. But as expected, the rise of Internet-connected devices in people's lives has driven dependence on the devices' software understanding data on the Web.

A Web for machines

Clearly, machines have interacted with each other long before the Web existed. And if the Web has come so far so quickly while primarily targeted at human consumption, it's natural to wonder what's to be gained in developing techniques for machines to share the Web with humans as an information channel. To explore this question, imagine what the current Web would look like if machines did not understand it at all.
The top three Web sites (as of this article's writing) according to Alexa traffic rankings are Yahoo!, MSN, and Google -- all search engines. Each of these sites is powered by an army of software-driven Web crawlers that apply various techniques to index human-generated Web content and make it amenable to text searches. Without these companies' vast arrays of algorithmic techniques for consuming the Web, your Web-navigation experiences would be limited to following explicitly declared hypertext links.
Next, consider the 5th most trafficked site on Alexa's list: eBay. People commonly think of eBay as one of the best examples of humans interacting on the Web. However, machines play a significant role in eBay's popularity. Approximately 47% of eBay's listings are created using software agents rather than with the human-driven forms. During the last quarter of 2005, the machine-oriented eBay platform handled eight billion service requests. Also in 2005, the number of eBay transactions through Web Services APIs increased by 84% annually. It's clear that without the services that eBay provides software agents to participate equally with humans, the online auction business would not be nearly as manageable for humans dealing with significant numbers of sales or purchases.
For a third example, we turn to Web feeds. Content-syndication formats such as Atom and RSS have empowered a new generation of news-reading software that frees you from the tedious, repetitive, and inefficient reliance on bookmarked Web sites and Web browsers to stay in touch with news of interest. Without the machine-understandable content representation embodied by RSS and Atom, these news readers could not exist.
In short, imagine a World Wide Web where a Web site could only contain content authored by humans exclusively for that site. Content could not be shared, remixed, and reused between Web sites. To intelligently aggregate, combine, and act on Web-based content, agents, crawlers, readers, and other devices must be able to read and understand that content. This is why it's necessary to take an in-depth look at the different mechanisms available today to improve the interactions between machines and human-generated content in Web applications.

Examples

Consider a scenario from the page of the Semantic Web activity at the W3C. Most people have some personal information that can be accessed on the Web. You can see your bank statements, access your online calendar applications, and post photos online through different photo-sharing services. But can you see uour photos in your calendar to remind yourself of the place and purpose of those photos? Can you see your bank-statement line items displayed in your calendar too?
Creating new data integrations of this sort requires that the software driving the integration be able to understand and interpret the data on particular Web pages. This software must be able to retrieve the Web pages that display your photos from Flickr and discover the dates, times, and descriptions of your photos. It also needs to understand how to interpret the transactions from your online bank statement. The same software must be able to understand various views of your online calendar (daily, weekly, and monthly), and figure out which parts of the Web page represent which dates and times.
The example in Figure 1 shows how embedded metadata might benefit your end-user applications. You begin with your data stored in several places. Flickr hosts your photographs, Citibank provides access to your banking transactions, and Google Calendar manages your daily schedule. You wish to experience all of this data in a single calendar-based interface (missMASH), such that the photos from your Sunday at the State Park appear in the same weekly view as your credit card transaction from Wednesday's grocery shopping. To do this, the software that powers missMASH must have some way to understand the data from your Flickr, Citibank, and Google Calendar accounts in order to remix the data in an integrated environment.

Figure 1. A mashup of banking and photos in a calendar view

A full spectrum of technologies give application authors the ability to do such integrations. Some of the technologies are well established, while others are still fledgling and not as well understood. The barriers to entry for the technologies vary, and some of the technologies will provide a higher level of utility than others.
In this series, we'll examine how you might implement the scenario discussed above using the different mechanisms available for human-computer coexistence on the Web. We will introduce and explain each technology, then show how the technologies might be used to integrate bank statements, photos, and calendars. We will also evaluate the strengths and weaknesses of each technology, and hopefully make it easier for you to decide between the options.

Evaluation criteria

When you embark on a comparison of technologies, it is helpful to first outline the criteria to evaluate the technologies. The list below describes properties desirable of methods that facilitate a Web that is both human-friendly and machine-readable. We don't expect to find a single technology that succeeds in all these facets, nor should we. In the end, we hope to build a matrix that will aid you in choosing the right tool for the job. Here are the criteria we will use:

Authoritative data

Whatever the particulars are for a technique, you end up with machine-readable data that supposedly is equivalent to what shows up on the corresponding human-friendly Web page. You want the techniques that get you to that point to ensure the fidelity of this relationship; you want to be able to trust that the data really corresponds to what you read on the Web page.
The authority of the data is one axis along which to measure this trust. We consider one representation of data to be authoritative if the representation is published by the owner of the data. A data representation might be non-authoritative if it is derived from a different representation of the data by a third party.

Expressivity and extensibility

If you use one technique to create a Web page with both a human-representable and machine-readable version of directions to your home, you'd rather not need a different technique to add the weather forecast to the same Web page. We hope that this criteria will help minimize the number of software components involved in any particular application, which in turn increases the robustness and maintainability of the application.
Along these lines, we appreciate it if the techniques acommodate new data in an elegant manner, and are expressive enough to instill confidence that in the future we can represent previously unforeseen data.

Don't repeat yourself (DRY)

If the same datum is referenced twice on a Web page, you don't want to write it down twice. Repetition leads to inconsistencies when changes are required, and you don't want to jump through too many hoops to create Web pages that both humans and computers can understand.
This does not necessarily apply to repetition within multiple data representations, if those representations are generated from a single data store.

Data locality

Multiple representations of the same data within the same document should be in as self-contained a unit of the document as possible. For example, if one paragraph within a large Web page is the only part of the page that deals with the ingredients of a recipe, then you want a technique to contain all of the machine-readable data about those ingredients to that paragraph.
In addition to being easy to author and to read, data locality allows visitors to the Web page to copy and paste both the human and machine representations of the data together, rather than requiring multiple discontinuous segments of the Web page be copied to fully capture the data of interest. This, in turn, promotes wider adoption and re-use of the techniques and the data. (Just ask anyone who has ever learned HTML with the generous use of the View Source command.)

Existing content fidelity

You want the techniques to work without requiring authors to rewrite their Web sites. The more that a technique can make use of existing clues about the intended (machine-consumable) meaning of a Web page, the better. But a caveat: techniques shouldn't be so liberal with their interpretations that they ascribe incorrect meanings to existing Web pages.
For example, a new technique that prescribed that text enclosed in the HTML u tag represent the name of a publication might lead to correct semantics in some cases, but might also license many incorrect interpretations. While the markup in this example might be authoritative (because it originates from the owner of the data), it is still incorrect because the Web page author did not intend to use the u HTML tag in this manner.

Standards compliance

You should be able to use techniques without losing the ability to adhere to accepted Web standards such as HTML (HTML 4 or XHTML 1), CSS, XML, XSLT, and more.

Tooling

Creating a Web of data that humans can read and machines can process is of little value if no tools understand the techniques used. We prefer techniques that have a large base of tools already available. Failing that, we prefer techniques for which one can easily implement new tools. Tools should be available both to help author Web pages that make use of a technique, and also to consume the machine-readable data as specified by the technique.

Overall complexity

The Web has been around for a while now, and it's only recently that the need to share data between humans and machines is receiving a lot of attention. The vast landscapes of content available on the Web are authored and maintained by a wide variety of people, and it is important that whatever techniques we promote be easily understandable and adoptable by as many Web authors as possible.

The best techniques are as worthless as no technique if they are too complex to adopt. The most desirable techniques will have a low barrier to entry and a shallow leaning curve. They should not require extensive coding to implement, nor require painstaking efforts to maintain once implemented.

Coexistence options
This section provides a brief introduction to the major techniques used today to enable machine-human coexistence on the Web. Subsequent articles in this series will explore these techniques in detail.

A parallel Web

In this world view, the data is represented on the Web with (at least) two addresses (URLs): one address holds a human-consumable format, and one a machine-consumable format. Technologies to enable the parallel Web include the HTML link element and HTTP content negotiation. Those involved in the creation of the HTML specifications saw the need for two linking elements in HTML: the a element that is visible and can only appear in the body of a Web page, and the link element that is invisible and can only appear in the head of a Web page. The HTML specification designers reasoned that agents, depending on their purpose and target audience, would interpret the links in the head based on their rel (relationship) attribute and perform interesting functions with them.
For example, Web feeds and feed readers have empowered humans to keep up with the vast amount of information being published today. When you use a feed reader, you initialize it with the address (URL) of an XML file -- usually an RSS or Atom file. In most cases, the machine-consumable data within such a feed has a parallel URL on the Web, where you can find a human-readable representation of the same contentd. There are a variety of techniques to achieve this parallel Web in a useful and maintainable fashion. Part 2 of this series will discuss a parallel Web in detail, including the benefits and drawbacks of having the same data available at more than one Web address. Future installments of this series will cover techniques that allow multiple data representations to be contained within a single Web address.

The algorithmic approach

Algorithmic approaches encompass software that produces machine-consumable data from human-readable Web pages by application of an arbitrary algorithm. In general, the algorithms tend to fall into two categories:
Scrapers, which extract data by examining the structure and layout of a Web page
Natural-language processors, which attempt to read and understand a Web page's content in order to generate data
These techniques are designed for situations where the structure or content of a Web page is highly predictable and unlikely to change. The algorithms are usually developed by the person seeking to consume the data, and as such they are not governed by any standards organization. Often, these algorithms are an integrator's only option when faced with accessing data whose owner does not publish a machine-readable representation of the data. Stay tuned for details on the algorithmic approach in Part 3 of this series.

Microformats

Microformats are a series of data formats that use existing (X)HTML and CSS constructs to embed raw data within the markup of a human-targeted Web page. Microformats are guided by a set of design principles and are developed by community consensus. The Microformats community's goal is to add semantics to the existing (X)HTML class attribute, originally intended mostly for presentation.
As with the algorithmic approach, microformats differs from many others in our series because it is not part of a standards process in organizations such as the W3C or IETF. Instead, their principles focus on specific problems and leverage current behaviors and usage patterns on the Web. This has given microformats a great start towards their goal of improving Web microcontent (blogs, for example) publishing in general. The main examples of microformat success have been the hCard and hCalendar specifications. These specifications allow microcontent publishers to easily embed attributes in their HTML content that allow machines to pick out small nuggets of information, such as business cards or event information from microcontent Web sites.

GRDDL

Gleaning Resource Descriptions from Dialects of Languages (GRDDL) allows Web page publishers to associate their XHTML or XML documents with transformations that take the Web page as input and then output machine-consumable data. GRDDL can use XSL transformations to extract specific vocabularies from a Web page. GRDDL also allows the use of profile documents, which in turn reference the appropriate transformation algorithms for a particular class of Web pages and data vocabularies.
GRDDL has great potential for bridging the gap between humans and machines by enabling authoritative on-the-fly transformations of content. While this is similar to the parallel Web, there are significant differences. GRDDL provides a general mechanism for machines to transform content on demand, and GRDDL does not create permanent versions of alternative data representations. The W3C has recently chartered a GRDDL working group to produce a recommended specification for GRDDL.

Embedded RDF (eRDF)

Embedded RDF is a technique for embedding RDF data within XHTML documents using existing elements and attributes. eRDF attempts to balance ease of markup with extensibility and expressivity. Along with RDFa and, to a lesser extent, GRDDL, it explicitly makes use of the Resource Description Framework (RDF) to model the machine-consumable data that it encodes. eRDF shares with microformats the principle of reusing existing vocabularies for the purpose of embedding metadata within XHTML documents. eRDF seeks to scale beyond a small set of formats and vocabularies by using namespaces and the arbitrary RDF graph data model.
Embedded RDF is not currently developed by a standards body. Similar to microformats, eRDF is capable of encoding data within Web pages to help machines extract contact, event, and location information (and other types of data) to enable powerful software agents.

RDFa

RDFa, formerly known as RDF/A, is another mechanism for including RDF data directly within XHTML. RDFa uses a fixed set of existing and new XHTML elements and attributes to allow a Web page to contain an arbitrary amount and complexity of machine-readable semantic data, alongside the standard XHTML content that is displayed to humans. RDFa is currently developed by the W3C RDF-in-XHTML task force, a joint product of the XHTML and Semantic Web Deployment working groups.
As with eRDF, RDFa takes advantage of namespaces and the RDF graph data model to enable the representation of many data structures and vocabularies within a single Web page. RDFa seeks to be a general-purpose solution to the inclusion of arbitrary machine-readable data within a Web page.

In summary

This article motivated and explained the challenge of creating a World Wide Web that is accessible to both humans and to machines. We developed an example integration scenario that could be enabled by any of the myriad of coexistence mechanisms. We also discussed the criteria with which to compare and evaluate the techniques that we will cover in more detail in the rest of this series.
Stay tuned for Part 2, which will explore in detail the widely used parallel Web technique.

(link:http://www-128.ibm.com/developerworks/web/library/wa-hmach1/index.html)

Exploiting Open Functionality in SMS-Capable Cellular Networks

Accepted at the 12th ACM Conference on Computer and Communications Security (CCS'05)November 7-11, 2005, Alexandria, VA, USA

William Enck, Patrick Traynor, Patrick McDaniel, and Thomas La Porta Systems and Internet Infrastructure Security (SIIS) Laboratory Networking and Security Research Center Department of Computer Science and Engineering The Pennsylvania State University
September 2, 2005

Cellular networks are a critical component of the economic and social infrastructures in which we live. In addition to voice services, these networks deliver alphanumeric text messages to the vast majority of wireless subscribers. To encourage the expansion of this new service, telecommunications companies offer connections between their networks and the Internet. The ramifications of such connections, however, have not been fully recognized.

This research evaluates the security impact of the Short Messaging Service (SMS) interface on the availability of the cellular phone network. Specifically, we demonstrate the ability to deny voice service to large metropolitan areas with little more than a cable modem. Moreover, attacks targeting the entire United States are feasible with resources available to medium-sized zombie networks.

We characterize network behavior and explore a number of reconnaissance techniques aimed at effectively targeting attacks on these systems. We also discuss a number of countermeasures that mitigate or eliminate the threats introduced by these attacks that must be implemented by cellular service providers in the near future.

for more information follow this link-http://www.smsanalysis.org/