EMMA: Extensible MultiModal Annotation markup language

W3C Working Draft 11 August 2003

This version:

http://www.w3.org/TR/2003/WD-emma-20030811/

Latest version:

http://www.w3.org/TR/emma/

Previous version:

(first publication)

Editors:

Roberto Pieraccini, SpeechWorks International

Wu Chou, Avaya

Deborah A. Dahl, Independent Consultant

Dave Raggett, W3C/Canon

Copyright © 2003 W3C^® ( MIT, ERCIM, Keio), All Rights Reserved. W3C liability, trademark, document use and software licensing rules apply.

---

Abstract

The W3C Multimodal Interaction working group aims to develop specifications to enable access to the Web using multi-modal interaction. This document is part of a set of specifications for multi-modal systems, and provides details of an XML markup language for describing the interpretation of user input. Examples of interpretation of user input are a transcription into words of a raw signal, for instance derived from a speech or pen input, a set of attribute/value pairs describing their meaning, or a set of attribute/value pairs describing a gesture. The interpretation of the user's input is expected to be generated by signal interpretation processes, such as speech and ink recognition, semantic interpreters, and other types of processors for use by components that act on the user's inputs such as interaction managers.

Status of this Document

This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/.

This document is a W3C Working Draft for review by W3C members and other interested parties. Publication as a Working Draft does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than 'work in progress.'

This specification describes markup for representing interpretations of user input (speech, keystrokes, pen input etc.) together with annotations for confidence scores, timestamps, input medium etc., and forms part of the proposals for the W3C Multimodal Interaction Framework. This document has been produced as part of the W3C Multimodal Interaction Activity, following the procedures set out for the W3C Process. The authors of this document are members of the Multimodal Interaction Working Group (W3C Members only). This is a Royalty Free Working Group, as described in W3C's Current Patent Practice NOTE. Working Group participants are required to provide patent disclosures.

This is the first version of this Working Draft, and your feedback is welcomed, especially on the open issues described within the specification. Please send comments about this document to the public mailing list: www-multimodal@w3.org (public archives). To subscribe, send an email to www-multimodal-request@w3. org with the word subscribe in the subject line (include the word unsubscribe if you want to unsubscribe).

Table of Contents

• 1. Introduction
  • 1.1 Uses of EMMA
• 2. Structure of EMMA documents
  • 2.1 Instance data, data model and annotations
    • 2.1.1 Data model
    • 2.1.2 XPointer for addressing EMMA documents
      • 2.1.2.1 emma() Scheme syntax
    • 2.1.3 RDF for annotations
      • 2.1.3.1 Incorporating RDF syntax in EMMA
      • 2.1.3.2 Using XPointer in combination with RDF
  • 2.2 Structural elements of EMMA
    • 2.2.1 "emma" element
    • 2.2.2 "interpretation" element
    • 2.2.3 "one-of" element
• 3. Annotations
  • 3.1 Data model annotations
  • 3.2 Annotation of interpretation derivation
  • 3.3 Tokens of input
  • 3.4 Reference to processing
  • 3.5 Lack of input
  • 3.6 Uninterpretable input
  • 3.7 Human language of input
  • 3.8 Reference to signal
  • 3.9 Confidence scores
  • 3.10 Grouping annotations
    • 3.10.1 Indirect grouping criteria
  • 3.11 Annotation of input source
  • 3.12 Timestamps
  • 3.13 Relative positioning of input elements
  • 3.14 Properties for annotating the medium, mode, and function of user inputs
  • 3.15 Composite multimodal input
• 4. Extensibility
• 5. Conformance
• 6. Formal syntax
  • 6.1 XML schema
  • 6.2 Document type definition
  • 6.3 RDF schema
• 7. References
• 8. Acknowledgements

1. Introduction

This document presents an XML specification for EMMA, an Extensible MultiModal Annotation markup language, responding to the requirements documented in  W3C Requirements for EMMA. This markup language is intended for use by systems that provide semantic interpretations for a variety of inputs, including but not necessarily limited to, speech, natural language text, GUI and ink input. 

It is expected that this markup will be used primarily as a standard data interchange format between the components of a multimodal system; in particular, it will normally be automatically generated by interpretation components to represent the semantics of users' inputs, not directly authored by developers.

The language is focused on annotating the interpretation information of single and composed inputs, as opposed to (possibly identical) information that might have been collected over the course of a dialog.

The language provides a set of elements and attributes that are focused on accurately representing annotations on the input interpretations.

An EMMA document can be considered to hold three types of data:

• instance data

  Application-specific markup corresponding to input information which is meaningful to the consumer of an EMMA document. Instances are application-specific and built by input processors at runtime. Given that utterances may be ambiguous with respect to input values, an EMMA document may hold more than one instance.

• data model

  Constraints on structure and content of an instance. The data model is typically pre-established by an application, and may be implicit, that is, unspecified.

• metadata

  Annotations associated with the data contained in the instance. Annotation values are added by input processors at runtime.

Given the assumptions above about the nature of data represented in an EMMA document, the following general principles apply to the design of EMMA:

• The main prescriptive content of the EMMA specification will consist of metadata: EMMA will provide a means to express the metadata annotations which require standardization. (Notice, however, that such annotations may express the relationship among all the types of data within an EMMA document.)
• The instance and its data model are assumed to be specified in XML, but EMMA will remain agnostic to the XML format used to express these. (The instance XML is assumed to be sufficiently structured to enable the association of annotative data.)
• The extensibility of EMMA lies in the ability for additional kinds of metadata to be included in application specific vocabularies and expressed within the same framework.

The annotations of EMMA should be considered 'normative' in the sense that if an EMMA component produces annotations as described in Section 3, these annotations must be represented using the EMMA syntax. The Multimodal Interaction Working Group may address in later drafts the issues of modularization and profiling, that is: which sets of annotations are to be supported by which classes of EMMA component.

1.1 Uses of EMMA

The general purpose of EMMA is to represent information automatically extracted from a user's input by an interpretation component, where input is to be taken in the general sense of a meaningful user input in any modality supported by the platform. The reader should refer to the sample architecture in W3C Multimodal Interaction Framework, which shows EMMA conveying content between user input modality components and an interaction manager.

Components that generate EMMA markup:

1. Speech recognizers
2. Handwriting recognizers
3. Natural language understanding engines
4. Other input media interpreters (e.g. DTMF, pointing, keyboard)
5. Multimodal integration component

Components that use EMMA include:

1. Interaction manager
2. Multimodal integration component

Although not a primary goal of EMMA, a platform may also choose to use this general format as the basis of a general semantic result that is carried along and filled out during each stage of processing. In addition, future systems may also potentially make use of this markup to convey abstract semantic content to be rendered into natural language by a natural language generation component.

2. Structure of EMMA documents

2.1 Instance data, data model and annotations

As noted above, the main components of an interpreted user input in EMMA are the instance data, an optional data model, and the metadata annotations that may be applied to that input. The realization of these components in EMMA is as follows:

• instance data is contained within an EMMA interpretation
• the data model is optionally specified as an annotation of that instance
• EMMA annotations may be applied at any level of an EMMA document.

An EMMA interpretation is the primary unit for holding user input as interpreted by an EMMA processor. As will be seen below, multiple interpretations of a single input are possible.

EMMA provides a simple structural syntax for the organization of interpretations and instances, and an annotative syntax derived from RDF to apply the annotation to the input data at any level.

An outline of the structural syntax of EMMA documents is as follows. A fuller definition may be found in the description of individual features in section 2.2.

• <emma:emma> is the root node, holding EMMA version and namespace information, and providing a container for one or more of the following elements:
• <emma:interpretation> is used to define a given interpretation of input, and holds application specific markup;
• Interpretation containers
  • <emma:one-of> is a container for one or more interpretation or interpretation container elements and denotes that these are mutually exclusive interpretations;
  • <emma:sequence> is a container for one or more interpretation or interpretation container elements and denotes that these are sequential in time;
  • <emma:group> is a general container for one or more interpretation or interpretation container elements. It can be associated with arbitrary grouping criteria.
• <rdf:RDF> (required only when RDF syntax is used) holds annotations on the interpretations using a combination of EMMA and application specific RDF properties expressed in the RDF XML syntax.

EMMA annotations may apply to interpretations and any node within the XML tree for the application-specific markup for a specific interpretation.

Here is an example of a complete EMMA document, illustrating the application of the RDF XML syntax for annotations at various levels.

Example:

<emma:emma emma:version="1.0"
 xmlns:emma="http://www.w3.org/2003/04/emma#"
 xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"> 
  <emma:one-of emma:id="r1">
    <emma:interpretation emma:id="int1">
      <origin>Boston</origin>
      <destination>Denver</destination>
      <date>03112003</date>
    </emma:interpretation>
    <emma:interpretation emma:id="int2">
      <origin>Austin</origin>
      <destination>Denver</destination>
      <date>03112003</date>
    </emma:interpretation>
  </emma:one-of>

  <rdf:RDF>
    <!-- time stamp for result -->
    <rdf:Description rdf:about="#r1" 
      emma:start="2003-03-26T0:00:00.15"
      emma:end="2003-03-26T0:00:00.2"/>

    <!-- confidence score for first interpretation -->
    <rdf:Description rdf:about="#int1" emma:confidence="0.75"/>

    <!-- confidence score for second interpretation -->
    <rdf:Description rdf:about="#int2" emma:confidence="0.68"/>

    <!-- time stamps for date in first interpretation -->
     <rdf:Description rdf:about="#emma(id('int1')/date)"> 
        <emma:absolute-timestamp
         emma:start="2003-03-26T0:00:00.15"
         emma:end="2003-03-26T0:00:00.2"/> 
     </rdf:Description>  
  </rdf:RDF>
</emma:emma>

This example shows a recognition result (emma:id="r1") with two exclusive interpretations (emma:id="int1" and emma:id="int2"). There are four annotations. The first gives the start and end timestamps for the result. The second and third give confidence scores for the two interpretations. The fourth gives a timestamp for the date value in the first interpretation, making use of the EMMA scheme for a subset of the XPointer syntax, as defined in section 2.1.2.

Here is the same example using the inline annotation syntax:

<emma:emma emma:version="1.0"
 xmlns:emma="http://www.w3.org/2003/04/emma#"> 
  <emma:one-of emma:id="r1" 
      emma:start="2003-03-26T0:00:00.15"
      emma:end="2003-03-26T0:00:00.2">
    <emma:interpretation emma:id="int1" emma:confidence="0.75" > 
      <origin>Boston</origin>
      <destination>Denver</destination>
      <date> 
         <emma:absolute-timestamp
          emma:start="2003-03-26T0:00:00.15"
          emma:end="2003-03-26T0:00:00.2"/> 
         03112003 
      </date>  
    </emma:interpretation>
    <emma:interpretation emma:id="int2" emma:confidence="0.68" >
      <origin>Austin</origin>
      <destination>Denver</destination>
      <date>03112003</date>
    </emma:interpretation>
  </emma:one-of>
</emma:emma>

The mixed representation syntax (Syntax 3 in the open issue in the Introduction) can be transformed into RDF syntax through an XSLT transform, and processed by the same EMMA processor for RDF syntax. On the other hand, for a resource limited inline syntax EMMA processor, the inline part of the mixed syntax will remain the same, but the RDF syntax section may not be processed. The mixed syntax annotation allows the annotation to be expanded along the chain to the interaction management, from the low level input device to high level understanding module, and at the same time, it provides flexibility to the application developer to control the processing complexity, bandwidth requirement, etc. for their application.

2.1.1 Data model

An EMMA data model expresses the constraints on the structure and content of instance data, for the purposes of validation. As such, the data model may be considered as a particular kind of annotation (although, unlike other EMMA annotations, it is not a feature pertaining a specific user input at a specific moment in time, it is rather a static and, by very definition, application-specific structure). Its specification in EMMA is optional.

Since Web applications today use different formats to specify data models, e.g. XML Schema, XForms, Relax-NG, etc., EMMA itself is agnostic to the format of data model used.

Data model definition and reference is defined in section 3.1.

2.1.2 XPointer for addressing EMMA documents

The emma() XPointer Scheme is intended to be used with the XPointer Framework [XPointer] to allow addressing within documents conforming to the EMMA specification. This scheme defines a subset of XPath [XPath] for use in addressing element and attribute nodes within EMMA documents. The subset was chosen to enable implementations in devices with tight resource limits. It omits other XPath features such as ranges, general predicates and variables. Support for the EMMA XPointer Scheme is required for processing documents using the RDF syntax.

As specified by the XPointer Framework, an EMMA XPointer processor takes as input an EMMA document and a string to be used as a pointer. This string is a fragment identifier with escaping reversed, and taken from the URI used to reference a node within the EMMA document. The processor attempts to evaluate the pointer with respect to the document and produces as output an identification of a node list or one or more errors.

The scheme name is "emma". The scheme data syntax is as follows; if scheme data in a pointer part with the emma() scheme does not conform to the syntax defined in this section the pointer part does not identify a subresource.

2.1.2.1 emma() Scheme syntax

The formal grammar for the subset of XPointer is given using simple Extended Backus-Naur Form (EBNF) notation, as described in the XML Recommendation [XML].

[1] EmmaSchemeData   ::=   ElementPath Attribute?
[2] ElementPath      ::=   IdReference? ElementStep*
[3] IdReference      ::=   'id(' QuotedName ')'
[4] ElementStep      ::=   ('//' | '/') Name Position?
[5] QuotedName       ::=   \' Name \' | \" Name \"
[6] Position         ::=   '[' [1-9] [0-9]* ']'
[7] Attribute        ::=   '/@' Name

The id() has the same semantics as in XPath, i.e. it selects elements by their unique ID. However, unlike XPath, is it constrained to only accept strings as arguments. The Position rule has the same semantics of XPath's positional predicate.

The evaluation context of an emma() epxression consists of the namespace declarations in scope at the location of the expression. For example, the namespace URI of the element geo:location in the third example below is determined by the binding of the geo prefix at that point in the document

It is an error if the content of an emma() expression does not reference a part of the document, either by a violation of the syntactic rules or by pointing to a non-existant resource.

Examples:

#emma(//destination)                  all elements called "destination"
#emma(id('s12')/quantity)             a child of element with id 's12'
#emma(//geo:location[2])              any second geo:location element
#emma(//rdf:RDF/rdf:Description[1])   first annotation in the document

The EMMA Scheme data can be used from RDF to annotate arbitrary element or attribute nodes in an EMMA document. It can also be used to query an EMMA document for the annotations on any element or attribute. This makes it an easy task to obtain the value of a given annotation, such as emma:confidence, or to determine the set of RDF triples that annotate a specific node in an EMMA document. Implementations may choose to support this by preprocessing EMMA documents to decorate the DOM tree with nodes representing RDF triples.

2.1.3 RDF for annotations

W3C's Resource Description Framework (RDF) is a well-established, extensible framework for describing properties for things that can be named with URIs. RDF statements are made in terms of a predicate, and the subject and object that the predicate applies to.

For many annotations in this document, this conceptual model can be reflected in two possible ways:

• in an inline syntax, where the annotation is inserted within the data structure; and
• in a direct RDF expression syntax.

The RDF model provides a powerful yet lightweight means to describe annotations on interpretations of input data whether the input is from speech, ink, key strokes or other modes of input. In most cases, the annotations will be generated automatically by the input processors.

For speech, the semantic intepretation specification [SI], provides a means for authors to indicate how recognition against a speech grammar can be used to generate XML interpretations of spoken input. The input processor can wrap this into an XML document conforming to the EMMA specification, in the process, adding annotations for time stamps, confidence scores, input medium etc.

The Multimodal Interaction Framework [MMIF] shows how EMMA documents are produced by input components and consumed by the multimodal integration and interaction manager components under the control of rules provided by authors. Both the semantic interpretations and their annotations can be accessed via simple XPath [XPATH] expressions.

2.1.3.1 Incorporating RDF syntax in EMMA

As noted earlier, one possible syntax for annotations in EMMA is to use RDF XML syntax. The use of RDF in EMMA allows processors to compare and evaluate the equivalence of the meaning in EMMA annotation based on RDF syntax (triples: subject, predicate, object) and the associated labeled directed graphs. The metadata annotation in EMMA incorporates vocabularies (e.g. confidence, etc.) that are in the EMMA namespace, and may include additional properties in application specific vocabularies.

Example stating confidence scores for two mutually exclusive interpretations:

<emma:emma emma:version="1.0"
 xmlns:emma="http://www.w3.org/2003/04/emma#"
 xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"> 
  <emma:one-of emma:id="r1">
    <emma:interpretation emma:id="int1">
      <origin>Boston</origin>
      <destination>Denver</destination>
      <date>03112003</date>
    </emma:interpretation>
    <emma:interpretation emma:id="int2">
      <origin>Austin</origin>
      <destination>Denver</destination>
      <date>03112003</date>
    </emma:interpretation>
  </emma:one-of>

  <rdf:RDF>
    <!-- confidence score for first interpretation -->
    <rdf:Description rdf:about="#int1" emma:confidence="0.75" />

    <!-- confidence score for second interpretation -->
    <rdf:Description rdf:about="#int2" emma:confidence="0.68" />
  </rdf:RDF>
</emma:emma>

The explicit representation of RDF triples in EMMA as XML conforms to a subset of the RDF/XML Syntax Specification [RDF-Syntax]. This subset is restricted to the rdf:Description element and the associated attributes: rdf:about, rdf:ID, and rdf:resource. Properties may be represented either as attributes on rdf:Description elements, or as child elements of rdf:Description elements, where the value of the property is given as the content of the child elements. In some cases, the value may be another rdf:Description element

2.1.3.2 Using XPointer in combination with RDF

Within EMMA documents that use the RDF syntax, RDF statements can use the EMMA XPointer schema as the basis for naming nodes within the XML tree corresponding to the EMMA document. This provides the power to annotate any element or attribute within an EMMA document.

Example using the EMMA XPointer reference "#emma(id('int1')/date)" to address the date element in the first interpretation.

<emma:emma emma:version="1.0"
 xmlns:emma="http://www.w3.org/2003/04/emma#"
 xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"> 
  <emma:one-of emma:id="r1">
    <emma:interpretation emma:id="int1">
      <origin>Boston</origin>
      <destination>Denver</destination>
      <date>03112003</date>
    </emma:interpretation>
    <emma:interpretation emma:id="int2">
      <origin>Austin</origin>
      <destination>Denver</destination>
      <date>03112003</date>
    </emma:interpretation>
  </emma:one-of>

  <rdf:RDF>
    <!-- time stamps for date in first interpretation -->
    <rdf:Description rdf:about="#emma(id('int1')/date)" 
      emma:start="2003-03-26T0:00:00.15"
      emma:end="2003-03-26T0:00:00.2"/>
  </rdf:RDF>
</emma:emma>

2.2 Structural elements of EMMA

2.2.1 "emma" element

The root element of an EMMA document is named emma. It holds one or more interpretation or grouping elements, and attributes for information pertaining to EMMA itself, along with any namespaces which are declared for the entire document, and any other EMMA annotative data.

Attributes:

1. emma:version (required) the version of EMMA used for the interpretation(s). Interpretations expressed using this specification (EMMA 1.0) should use "1.0" for the value.
2. xmlns:emma (required) namespace for the emma result. For EMMA 1.0 documents, the value will be: "http://www.w3.org/2003/04/emma#"
3. xmlns:rdf (required only when RDF syntax is used) namespace for RDF. This is "http://www.w3.org/1999/02/22-rdf-syntax-ns#"
4. any other namespace declarations for application specific namespaces (optional)

<emma:emma emma:version="1.0"
 xmlns:emma="http://www.w3.org/2003/04/emma#"
 xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"> 
    ....
</emma:emma>

2.2.2 "interpretation" Element

The emma:interpretation element holds a single interpretation represented in application specific markup.

Attributes:

1. emma:id a required xsd:ID value that uniquely identifies the interpretation within the EMMA document and used to attach annotations by reference to this identifier.

<emma:emma emma:version="1.0"
 xmlns:emma="http://www.w3.org/2003/04/emma#"
 xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"> 
  <emma:interpretation emma:id="r1">
    ...
  </emma:interpretation>
</emma:emma>

or

<emma:emma emma:version="1.0"
 xmlns:emma="http://www.w3.org/2003/04/emma#"
 xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"> 
  <emma:interpretation emma:id="r1">
    ...
  </emma:interpretation>
  <emma:interpretation emma:id="r2">
    ...
  </emma:interpretation>
  <rdf:RDF>
    ...
  </rdf:RDF>
</emma:emma>

2.2.3 "one-of" Element

The emma:one-of element acts as a container for two or more emma:interpretation elements, and denotes that these are mutually exclusive interpretations.

Attributes:

1. emma:id a required xsd:ID value that uniquely identifies the one-of element within the EMMA document and used to attach annotations by reference to this identifier.

<emma:emma emma:version="1.0"
 xmlns:emma="http://www.w3.org/2003/04/emma#"
 xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"> 
  <emma:one-of emma:id="r1">
    <emma:interpretation emma:id="int1">
      <origin>Boston</origin>
      <destination>Denver</destination>
      <date>03112003</date>
    </emma:interpretation>
    <emma:interpretation emma:id="int2">
      <origin>Austin</origin>
      <destination>Denver</destination>
      <date>03112003</date>
    </emma:interpretation>
  </emma:one-of>
</emma:emma>

Note: another possibility would be to represent the relationships between mutually exclusive interpretations using RDF properties, e.g. a link from one interpretation to the next alternative interpretation. The emma:one-of element has the advantage of making it possible to make statements about the set of alternatives as a whole.

3. Annotations

This section defines annotations in the EMMA namespace. The values are specified in terms of the data types defined by XML Schema Part 2: Datatypes [XSD].

3.1 Data model annotations

The data model that may be used to express constraints on the structure and content of instance data is specified as one of the annotations of the instance. Specifying the data model is optional, in which case the data model can be said to be implicit. Typically the data model is pre-established by the application.

The data model is specified with the emma:model annotation defined in the EMMA namespace. In the inline case, it is represented as an element, in the RDF syntax as an RDF property.

The data model is closely related to the interpretation data, and is typically specified as the annotation related to the <interpretation> or <one-of> elements.

<emma:emma emma:version="1.0"
 xmlns:emma="http://www.w3.org/2003/04/emma#"
 xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
  <emma:interpretation emma:id="int1">
    <city> London </city>
    <country> UK </country>
  </emma:interpretation>  

  <rdf:RDF>
    <rdf:Description rdf:about="#int1"
      emma:model="http://myserver/models/city.xml"/>
  </rdf:RDF>
</emma:emma>

The emma:model annotation can reference any element or attribute in the application instance data, as well as any EMMA container element (emma:one-of, emma:group, or emma:sequence).

The following is an example for the inline syntax:

<emma:emma emma:version="1.0"
 xmlns:emma="http://www.w3.org/2003/04/emma#" >
  <emma:interpretation emma:id="int1">
    <emma:model ref="http://myserver/models/city.xml"/>
    <city> London </city>
    <country> UK </country>
  </emma:interpretation>  
</emma:emma>

3.2 Annotation of interpretation derivation

Instances of interpretations are in general derived from other instances of interpretation in a process that goes from raw data to increasingly refined representations of the input. The derivation annotation is used to link any two interpretations that are related by representing the source and the outcome of an interpretation process. For instance, a speech recognition process can return the following result in the form of raw text:

<emma:interpretation emma:id="raw">
   <answer>From Boston to Denver tomorrow</answer>
</emma:interpretation>

A first interpretation process will produce:

<emma:interpretation emma:id="better">
  <origin>Boston</origin>
  <destination>Denver</destination>
  <date>tomorrow</date>
</emma:interpretation>

A second interpretation process, aware of the current date, will be able to produce a more refined instance, such as:

<emma:interpretation emma:id="best">
  <origin>Boston</origin>
  <destination>Denver</destination>
  <date>20030315</date>
</emma:interpretation>

The interaction manager may need to have access to the three levels of interpretation. The emma:derived-from annotation can be used to establish a chain of derivation relationships as in the following example:

<emma:emma emma:version="1.0"
 xmlns:emma="http://www.w3.org/2003/04/emma#"
 xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
 
  <emma:interpretation emma:id="raw">
    <answer>From Boston to Denver tomorrow</answer>
  </emma:interpretation>

  <emma:interpretation emma:id="better">
    <origin>Boston</origin>
    <destination>Denver</destination>
    <date>tomorrow</date>
  </emma:interpretation>

  <emma:interpretation emma:id="best">
    <origin>Boston</origin>
    <destination>Denver</destination>
    <date>20030315</date>
  </emma:interpretation>

  <rdf:RDF>
    <!-- derivation for second interpretation -->
    <rdf:Description  rdf:about="#better">
      <emma:derived-from rdf:resource="#raw" />
    </rdf:Description>

    <!-- derivation for third interpretation -->
    <rdf:Description  rdf:about="#best">
      <emma:derived-from rdf:resource="#better" />
    </rdf:Description>
  </rdf:RDF>
</emma:emma>

The corresponding example for the inline syntax is:

<emma:emma emma:version="1.0"
 xmlns:emma="http://www.w3.org/2003/04/emma#">
 
  <emma:interpretation emma:id="raw">
    <answer>From Boston to Denver tomorrow</answer>
  </emma:interpretation>

  <emma:interpretation emma:id="better">
    <emma:derived-from resource="#raw" />
    <origin>Boston</origin>
    <destination>Denver</destination>
    <date>tomorrow</date>
  </emma:interpretation>

  <emma:interpretation emma:id="best">
    <emma:derived-from resource="#better" />
    <origin>Boston</origin>
    <destination>Denver</destination>
    <date>20030315</date>
  </emma:interpretation>
</emma:emma>

An interpretation can have multiple derivation sources, for instance an interpretation can be derived by merging two interpretations obtained at different times. For instance:

<emma:emma emma:version="1.0"
 xmlns:emma="http://www.w3.org/2003/04/emma#"
 xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
  <emma:interpretation emma:id="previousturn">
    <airline>nofrillsair</airline>
  </emma:interpretation>

  <emma:interpretation emma:id="currentturn">
    <origin>Boston</origin>
    <destination>Denver</destination>
  </emma:interpretation>

  <emma:interpretation emma:id="mergedint">
    <origin>Boston</origin>
    <destination>Denver</destination>
    <airline>nofrillsair</airline>
  </emma:interpretation> 

  <rdf:RDF>
    <rdf:Description rdf:about="#mergedint"> 
      <emma:derived-from resource="#previousturn"/>
      <emma:derived-from resource="#currentturn"/>
    </rdf:Description>
  </rdf:RDF>
</emma:emma>

Analogously, for the inline syntax:

<emma:emma emma:version="1.0"
 xmlns:emma="http://www.w3.org/2003/04/emma#">
  <emma:interpretation emma:id="previousturn"/>
    <airline>nofrillsair</airline>
  </emma:interpretation>

  <emma:interpretation emma:id="currentturn">
    <origin>Boston</origin>
    <destination>Denver</destination>
  </emma:interpretation>

  <emma:interpretation emma:id="mergedint">
    <emma:derived-from resource="#previousturn"/>
    <emma:derived-from resource="#currentturn"/>
    <origin>Boston</origin>
    <destination>Denver</destination>
    <airline>nofrillsair</airline>
  </emma:interpretation> 
</emma:emma>

3.3 Tokens of input

The emma:tokens annotation holds a list of input tokens. In the following description, the term tokens is used in the computational and syntactic sense of units of input, and not in the sense of XML tokens.

The value held in emma:tokens is the list of the tokens of input as produced by the processor which generated the EMMA document. In the case where a grammar is used to constrain input, the value will correspond to tokens as defined by the grammar. So for an EMMA document produced by input to a W3C SRGS grammar [SRGS], the value of emma:tokens will be the list of words and/or phrases that are defined as tokens in SRGS (through white-spaced character data or the <token>; element, see SRGS section 2.1 Tokens). Items in the emma:tokens list are delimited by white space and/or quotation marks for phrases containing white space. For example:

emma:tokens="arriving at 'Liverpool Street'"

where the three tokens of input are arriving, at and Liverpool Street.

The tokens annotation may be applied not just to the lexical words and phrases of language but to any level of input processing. Other examples of tokenization include phonemes, ink strokes, gestures and any other discrete units of input at any level.

Examples:

Inline:

<emma:emma emma:version="1.0"
 xmlns:emma="http://www.w3.org/2003/04/emma#">
  <emma:interpretation emma:tokens="From Cambridge to London tomorrow">
    <origin emma:tokens="From Cambridge">Cambridge</origin>
    <destination emma:tokens="to London">London</destination>
    <date emma:tokens="tomorrow">20030315</date>
  </emma:interpretation>
</emma:emma>

RDF:

<emma:emma emma:version="1.0"
 xmlns:emma="http://www.w3.org/2003/04/emma#"
 xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
  <emma:interpretation emma:id="int1">
    <origin>Cambridge</origin>
    <destination>London</destination>
    <date>20030315</date>
  </emma:interpretation>
  <rdf:RDF>
    <rdf:Description rdf:about="#int1"
       emma:tokens="From Cambridge to London tomorrow" />
    <rdf:Description rdf:about="#emma(//origin)"
       emma:tokens="From Cambridge" />
    <rdf:Description rdf:about="#emma(//destination)"
       emma:tokens="to London" />
    <rdf:Description rdf:about="#emma(//date)"
       emma:tokens="tomorrow" />
  </rdf:RDF>
</emma:emma>

3.4 Reference to processing

A reference to the information concerning the processing that was used for generating an interpretation can be made as in the following example:

<emma:emma emma:version="1.0"
 xmlns:emma="http://www.w3.org/2003/04/emma#"
 xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
  <emma:interpretation emma:id="raw">
    <answer>From Boston to Denver tomorrow</answer>
  </emma:interpretation>

  <emma:interpretation emma:id="better">
    <origin>Boston</origin>
    <destination>Denver</destination>
    <date>tomorrow</date>
  </emma:interpretation>

  <emma:interpretation emma:id="best">
    <origin>Boston</origin>
    <destination>Denver</destination>
    <date>03152003</date>
  </emma:interpretation>

  <rdf:RDF>
    <rdf:Description rdf:about="#better" emma:derived-from="#raw"
       emma:process="http://example.com/mysemproc1.xml"/>

    <rdf:Description rdf:about="#best" emma:derived-from="#better"
       emma:process="http://example.com/mysemproc2.xml"/>
  </rdf:RDF>
</emma:emma>

The process description document, referenced by the emma:process annotation can include information on the process itself, such as grammar, type of parser, etc. EMMA is not normative about the format of the process description document.

For the inline syntax:

<emma:emma emma:version="1.0"
 xmlns:emma="http://www.w3.org/2003/04/emma#" >
  <emma:interpretation emma:id="raw">
    <answer>From Boston to Denver tomorrow</answer>
  </emma:interpretation>

  <emma:interpretation emma:id="better"
   emma:process="http://example.com/mysemproc1.xml">
    <origin>Boston</origin>
    <destination>Denver</destination>
    <date>tomorrow</date>
    <emma:derived-from emma:resource="#raw"/>
  </emma:interpretation>

  <emma:interpretation emma:id="best"
   emma:process="http://example.com/mysemproc2.xml">
    <origin>Boston</origin>
    <destination>Denver</destination>
    <date>03152003</date>
    <emma:derived-from emma:resource="#better"/>
  </emma:interpretation>
</emma:emma>

3.5 Lack of input

The case of lack of input can be annotated as follows:

<emma:emma emma:version="1.0"
 xmlns:emma="http://www.w3.org/2003/04/emma#">
  <emma:interpretation emma:id="int1" emma:no-input="true" />
</emma:emma>

or alternatively:

<emma:emma emma:version="1.0"
 xmlns:emma="http://www.w3.org/2003/04/emma#"
 xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
  <emma:interpretation emma:id="int1"/>

  <rdf:RDF>
    <rdf:Description rdf:about="#int1" emma:no-input="true"/>
  </rdf:RDF>
</emma:emma>

3.6 Uninterpretable input

Input that cannot be interpreted can be annotated as in the following example:

<emma:emma emma:version="1.0"
 xmlns:emma="http://www.w3.org/2003/04/emma#"
 xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
  <emma:interpretation emma:id="raw">
    <answer>one sixty two flight fourth yes</answer>
  </emma:interpretation>

  <emma:interpretation emma:id="better">
    <emma:uninterpreted/>
  </emma:interpretation>

  <rdf:RDF>
    <rdf:Description rdf:about="#raw" 
     emma:process="http://example.com/myasr.xml"/>

    <rdf:Description rdf:about="#better"
      emma:process="http://example.com/mysemproc1.xml"
      derived-from="#raw"/>
  </rdf:RDF>
</emma:emma>

where the input ("raw") did not lead to any possible interpretation.

Alternatively one can use the following syntax:

<emma:emma emma:version="1.0"
 xmlns:emma="http://www.w3.org/2003/04/emma#"
 xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
  <emma:interpretation emma:id="raw">
    <answer>one sixty two flight fourth yes</answer>
  </emma:interpretation>

  <emma:interpretation emma:id="better"/>

  <rdf:RDF>
    <rdf:Description rdf:about="#raw"
      emma:process="http://example.com/myasr.xml"/>

    <rdf:Description rdf:about="#better"
      emma:uninterpreted="true"
      emma:process="http://example.com/mysemproc1.xml"
      emma:derived-from="#raw"/>
  </rdf:RDF>
</emma:emma>

The notation for uninterpretable input can refer to any possible stage of interpretation processing, including raw transcriptions. For instance, if input speech cannot be correctly recognized or the spoken input is not matched by a grammar (or language constraint given to the recognition), it can be tagged as emma:uninterpreted as in the following example:

<emma:emma emma:version="1.0"
 xmlns:emma="http://www.w3.org/2003/04/emma#"
 xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
  <emma:interpretation emma:id="raw"/>

  <rdf:RDF>
    <rdf:Description rdf:about="#raw"
      emma:uninterpreted="true"
      emma:process="http://example.com/myasr.xml"/>
  </rdf:RDF>
</emma:emma>

An example for the inline syntax is as follows:

<emma:emma emma:version="1.0"
 xmlns:emma="http://www.w3.org/2003/04/emma#">
  <emma:interpretation emma:id="raw" 
      emma:process="http://example.com/myasr.xml" 
      emma:uninterpreted="true" />
</emma:emma>

3.7 Human language of input

The emma:lang annotation is used to indicate the human language for the input. For example, emma:lang="fr" denotes French, and emma:lang="en-US" denotes US English. emma:lang may be applied to any emma:interpretation element. Its annotative scope follows the annotative scope of these elements.

The following example shows the use of emma:lang for annotating an input interpretation.

<emma:emma emma:version="1.0"
 xmlns:emma="http://www.w3.org/2003/04/emma#"
 xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
  <emma:interpretation emma:id="int1">
    <answer>arretez</answer>
  </emma:interpretation>

  <rdf:RDF>
    <rdf:Description rdf:about="#int1" emma:lang="fr"/>
  </rdf:RDF>
</emma:emma>

and for the inline syntax:

<emma:emma emma:version="1.0"
 xmlns:emma="http://www.w3.org/2003/04/emma#">
  <emma:interpretation emma:id="int1" emma:lang="fr">
    <answer>arretez</answer>
  </emma:interpretation>
</emma:emma>

The following example shows the annotation of different interpretations derived from the same input in a multilingual application:

<emma:emma emma:version="1.0"
 xmlns:emma="http://www.w3.org/2003/04/emma#"
 xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
  <emma:interpretation emma:id="int1">
    <rawtext>please stop arretez s'il vous plait</rawtext>
  </emma:interpretation>

  <emma:interpretation emma:id="int2">
    <command> CANCEL </command>
  </emma:interpretation>

  <emma:interpretation emma:id="int3">
    <command> CANCEL </command>
  </emma:interpretation>
  
  <rdf:RDF>
    <rdf:Description rdf:about="#int2"
      emma:lang="en" emma:derived-from="#int1"
      emma:process="http://example.com/EnglishInterpreter.xml"/>

    <rdf:Description rdf:about="#int3"
      emma:lang="fr" emma:derived-from="#int1"
      emma:process="http://example.com/FrenchInterpreter.xml"/>
  </rdf:RDF>
</emma:emma>

and analogously for the inline syntax:

<emma:emma emma:version="1.0"
 xmlns:emma="http://www.w3.org/2003/04/emma#">
  <emma:interpretation emma:id="int1">
    <rawtext>please stop arretez s'il vous plait</rawtext>
  </emma:interpretation>

  <emma:interpretation emma:id="int2" emma:lang="en"
   emma:process="http:/example.com/EnglishInterpreter.xml" >
    <command> CANCEL </command>
    <emma:derived-from resource="#int1"/>
  </emma:interpretation>

  <emma:interpretation emma:id="int3" emma:lang="fr"
   emma:process="http:/example.com/FrenchInterpreter.xml">
    <command> CANCEL </command>
    <emma:derived-from resource="#int1"/>
  </emma:interpretation>
</emma:emma>

3.8 Reference to signal

A URI reference to the signal that originated the input recognition process may be represented in EMMA using the emma:signal annotation.

Here is an example where the reference to the signal is applied to the emma:interpretation element:

<emma:emma version="1.0"
 xmlns:emma="http://www.w3.org/2003/04/emma#"
 xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
  <emma:interpretation emma:id="intp1">
    <origin>Boston</origin>
    <destination>Denver</destination>
    <date>03152003</date>
  </emma:interpretation> 

  <rdf:RDF>
    <rdf:Description rdf:about="#intp1"
      emma:signal="http://example.com/signals/sg23.bin"/>
  </rdf:RDF>
</emma:emma>

and for the inline syntax:

<emma:emma version="1.0"
 xmlns="http://www.w3.org/2003/04/emma#">
  <emma:interpretation emma:id="intp1"
          emma:signal="http://example.com/signals/sg23.bin">
    <origin>Boston</origin>
    <destination>Denver</destination>
    <date>03152003</date>
  </emma:interpretation> 
</emma:emma>

3.9 Confidence scores

Confidence score in EMMA is used to indicate the quality of the input, and it is the value assigned to emma:confidence in the EMMA namespace. The confidence score is a number in the range from 0.0 to 1.0 inclusive. A value of 0.0 indicates minimum confidence, and a value of 1.0 indicates maximum confidence. The confidence score values do not have to be interpreted as probabilities. In fact confidence score values are platform-dependent, since their computation is likely to differ between platforms and different EMMA processors. Confidence scores are annotated explicitly in EMMA in order to provide this information to the subsequent processes for multimodal interaction. The example below illustrate how confidence scores are annotated in EMMA.

<emma:emma emma:version="1.0"
 xmlns:emma="http://www.w3.org/2003/04/emma#"
 xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
  <emma:one-of>
    <emma:interpretation  emma:id="meaning1>
      <location>Boston</location> 
    </emma:interpretation>
    <emma:interpretation emma:id="meaning2">
      <location> Austin </location> 
    </emma:interpretation>
  </emma:one-of>

  <rdf:RDF>
    <rdf:Description rdf:about="#meaning1" emma:confidence="0.6"/>
    <rdf:Description rdf:about="#meaning2" emma:confidence="0.4"/>
  </rdf:RDF>
</emma:emma> 

and analogously, for the inline syntax:

<emma:emma emma:version="1.0"
 xmlns:emma="http://www.w3.org/2003/04/emma#">
    <emma:one-of>
      <emma:interpretation emma:id="meaning1" emma:confidence="0.6">
        <location>Boston</location> 
      </emma:interpretation> 
      <emma:interpretation emma:id="meaning2" emma:confidence="0.4">
        <location> Austin </location> 
      </emma:interpretation>
    </emma:one-of>
</emma:emma>

The emma:confidence annotation may also be applied to attributes. Here is an example stating the confidence for the size attribute denoting pizza size:

<emma:emma emma:version="1.0"
 xmlns:emma="http://www.w3.org/2003/04/emma#"
 xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
  <emma:interpretation  emma:id="p1">
    <pizza size="large" style="quattro stagioni"/>
  </emma:interpretation>

  <rdf:RDF>
    <rdf:Description rdf:about="//pizza/@size)"
      emma:confidence="0.6"/>
  </rdf:RDF>
</emma:emma>

The current EMMA draft does not specify how the inline syntax can be used to apply annotations to attributes in the instance data. It may be a reasonable constraint to require that data which is intended to be annotated should be be realized in element nodes rather than attribute nodes. If not, an inline syntax could be derived which enables the annotation of attribute data through an attribute-referencing mechanism, as in the following example:

   <pizza size="large">
      <emma:annotation emma:attribute="size" emma:confidence="0.6" />
   </pizza>

This would allow the XPath querying model to be maintained for all data and annotation queries.

3.10 Grouping

Introduced in section 2.1, the emma:group element is used to indicate that the contained interpretations are related in some manner.

The following example shows three interpretations derived from the speech input "Move this ambulance here" and the haptic input related to two consecutive points on a map. The group is associated with time stamps defining the beginning and end of a time window used to group the events.

<emma:emma emma:version="1.0"
 xmlns:emma="http://www.w3.org/2003/04/emma#"
 xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
  <emma:group emma:id="grp">
    <emma:interpretation>
      <action>move</action>
      <object>ambulance</object>
      <destination>here</destination>
    </emma:interpretation>

    <emma:interpretation>
      <x>0.253</x>
      <y>0.124</y>
    </emma:interpretation>

    <emma:interpretation>
      <x>0.866</x>
      <y>0.724</y>
    </emma:interpretation>
  </emma:group>

  <rdf:RDF>
    <rdf:Description rdf:about="#grp"
      emma:start="2003-03-26T0:00:00.15"
      emma:end="2003-03-26T0:00:00.515"/>
  </rdf:RDF>
</emma:emma>

The emma:one-of and emma:group containers can be nested arbitrarily.

An analogous example for the inline syntax is as follows:

<emma:emma emma:version="1.0"
  xmlns:emma="http://www.w3.org/2003/04/emma#">
  <emma:group emma:id="grp"       
      emma:start="2003-03-26T0:00:00.15"
      emma:end="2003-03-36T0:00:00.515">
    <emma:interpretation>
      <action>move</action>
      <object>ambulance</object>
      <destination>here</destination>
    </emma:interpretation>

    <emma:interpretation>
      <x>0.253</x>
      <y>0.124</y>
    </emma:interpretation>

    <emma:interpretation>
      <x>0.866</x>
      <y>0.724</y>
    </emma:interpretation>
  </emma:group>
</emma:emma>

3.10.1 Indirect grouping criteria

Sometimes it may be convenient to indirectly associate a given group with information, such as grouping criteria. The emma:group-info annotation can be used to associate a group with information expressed as a set of RDF properties. In the following example, a group of two points is associated with a description of grouping criteria based upon a sliding temporal window of two seconds duration.

<emma:emma emma:version="1.0"
 xmlns:emma="http://www.w3.org/2003/04/emma#"
 xmlns:ex="http://www.example.com/ns/group#"
 xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
  <emma:group emma:id="grp">
    <emma:interpretation>
      <x>0.253</x>
      <y>0.124</y>
    </emma:interpretation>

    <emma:interpretation>
      <x>0.866</x>
      <y>0.724</y>
    </emma:interpretation>
  </emma:group>

  <rdf:RDF>
    <rdf:Description rdf:about="#grp">
       <emma:group-info>
         <rdf:Description
             ex:mode="temporal"
             ex:duration="2s"/>
       </emma:group-info>
    </rdf:Description>
  </rdf:RDF>
</emma:emma>

Here is the inline equivalent:

<emma:emma emma:version="1.0"
 xmlns:emma="http://www.w3.org/2003/04/emma#"
 xmlns:ex="http://www.example.com/ns/group#"
 xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
  <emma:group emma:id="grp">
    <emma:group-info>
       <ex:mode>temporal</ex:mode>
       <ex:duration>2s</ex:duration>
    </emma:group-info>

    <emma:interpretation>
      <x>0.253</x>
      <y>0.124</y>
    </emma:interpretation>

    <emma:interpretation>
      <x>0.866</x>
      <y>0.724</y>
    </emma:interpretation>
  </emma:group>
</emma:emma>

You can also use emma:group-info to refer to a named grouping criterion using external reference, for instance:

<emma:emma emma:version="1.0"
 xmlns:emma="http://www.w3.org/2003/04/emma#"
 xmlns:ex="http://www.example.com/ns/group#"
 xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
  <emma:group emma:id="grp">
    <emma:interpretation>
      <x>0.253</x>
      <y>0.124</y>
    </emma:interpretation>

    <emma:interpretation>
      <x>0.866</x>
      <y>0.724</y>
    </emma:interpretation>
  </emma:group>

  <rdf:RDF>
    <rdf:Description rdf:about="#grp"
       emma:group-info="http://www.example.com/criterion42"/>
  </rdf:RDF>
</emma:emma>

Here is the inline equivalent:

<emma:emma emma:version="1.0"
 xmlns:emma="http://www.w3.org/2003/04/emma#"
 xmlns:ex="http://www.example.com/ns/group#"
 xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
  <emma:group emma:id="grp">
    <emma:group-info ref="http://www.example.com/criterion42"/>
    <emma:interpretation>
      <x>0.253</x>
      <y>0.124</y>
    </emma:interpretation>
    <emma:interpretation>
      <x>0.866</x>
      <y>0.724</y>
    </emma:interpretation>
  </emma:group>
</emma:emma>

3.11 Annotation of input source

The source of an interpreted input may be represented in EMMA as a URI resource using the emma:source annotation.

Here is an example that shows different input sources for different input interpretations.

<emma:emma emma:version="1.0"
 xmlns:emma="http://www.w3.org/2003/04/emma#"
 xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
 xmlns:myapp="http://www.example.com/myapp">
  <emma:one-of>
    <emma:interpretation emma:id="intp1">
      <myapp:destination>Boston</myapp:destination>
    </emma:interpretation>

    <emma:interpretation emma:id="intp2">    
      <myapp:destination>Austin</myapp:destination>
    </emma:interpretation>
  </emma:one-of>

  <rdf:RDF>
    <rdf:Description rdf:about="#intp1" 
      emma:source="http://example.com/microphone/NC-61"/>

    <rdf:Description rdf:about="#intp2"
      emma:source="http://example.com/microphone/NC-4024"/>
  </rdf:RDF>
 </emma:emma>

An analogous example for the inline syntax is as follows:

<emma:emma emma:version="1.0"
 xmlns:emma="http://www.w3.org/2003/04/emma#"
 xmlns:myapp="http://www.example.com/myapp">
  <emma:one-of>
    <emma:interpretation emma:id="intp1"
         emma:source="http://example.com/microphone/NC-61">
      <myapp:destination>Boston</myapp:destination>
    </emma:interpretation>

    <emma:interpretation emma:id="intp2"
         emma:source="http://example.com/microphone/NC-4024">    
      <myapp:destination>Austin</myapp:destination>
    </emma:interpretation>
  </emma:one-of>
 </emma:emma>

3.12 Timestamps

The start and end timestamps for input can be indicated using either absolute timestamps, or as relative timestamps, expressed as time offsets from the start or end of a temporal reference point, such as another input.

The emma:from-start-of and emma:from-end-of annotations for specifying the reference point of a relative time are added by the producer of the EMMA document. In order to make use of these annotations, the consumer of the EMMA document must have access to the resources referenced by the URI. This may not always be the case in a distributed architecture.

Relative timestamps use the emma:relative-timestamp annotation, with either emma:from-start-of or emma:from-end-of to refer to the event which is the reference point for the timestamp. The time offset from this reference point is stated with either emma:start or emma:end.

The timestamp format uses the XML Schema datatypes [XSD]. The timestamp for relative time uses the representation for the xsd:duration datatype. The absolute time uses the xsd:dateTime if both the date and time are included, or xsd:time if only the time is included.

Here is an example that uses absolute timestamps in the RDF XML syntax.

<emma:emma emma:version="1.0"
 xmlns:emma="http://www.w3.org/2003/04/emma#"
 xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"> 
  <emma:interpretation emma:id="int1">
    <destination>Chicago</destination>
  </emma:interpretation>

  <rdf:RDF> 
    <rdf:Description rdf:about="#int1">  
       <emma:absolute-timestamp
           emma:start="2003-03-26T0:00:00"   
           emma:end="2003-03-26T0:00:00.2"/>
    </rdf:Description>
  </rdf:RDF>
</emma:emma>

Here is the same example represented using the inline syntax:

<emma:emma emma:version="1.0"
 xmlns:emma="http://www.w3.org/2003/04/emma#"
  <emma:interpretation emma:id="int1">
    <emma:absolute-timestamp
       emma:start="2003-03-26T0:00:00"
       emma:end="2003-03-26T0:00:00.2"/>
    <destination>Chicago</destination>
  </emma:interpretation>
</emma:emma>

emma:start and emma:end can be used with values conforming to the xsd:dateTime, xsd:time and xsd:duration data types, which are all syntactically distinct. When used to represent time offsets from other input, emma:from-start-of and emma:from-end-of can be used to indicate the input the offset refers to. Both of these are of type xsd:anyURI. emma:from-start-of is used when the offset is from the start of the referenced input, whereas emma:from-end-of is used when the offset is from the end of the referenced input. Note that the referenced input may be in a different EMMA document.

Here is an example where the referenced input is in the same document:

<emma:emma emma:version="1.0"
 xmlns:emma="http://www.w3.org/2003/04/emma#"
 xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
  <emma:interpretation emma:id="int1">
    <origin>Denver</origin>
  </emma:interpretation>
  <emma:interpretation emma:id="int2">
    <destination>Chicago</destination>
  </emma:interpretation>

  <rdf:RDF>
    <!-- "Chicago" was input 5 seconds after "Denver" -->
    <rdf:Description rdf:about="#int2">
       <emma:relative-timestamp
           emma:start="P5S"
           emma:from-start-of="#int1"/>
    </rdf:Description>
  </rdf:RDF>
</emma:emma>

In the inline syntax, this can be represented using the element emma:relative-timestamp as follows:

<emma:emma emma:version="1.0"
 xmlns:emma="http://www.w3.org/2003/04/emma#"
 xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"> 
  <emma:interpretation emma:id="int1">
    <origin>Denver</destination>
  </emma:interpretation>
  <emma:interpretation emma:id="int2">
    <emma:relative-timestamp
      emma:start="P5S"
      emma:from-start-of="#int1"/>
    <destination>Chicago</destination>
  </emma:interpretation>
</emma:emma>

Note that the reference point refers to an input from a user. It does not necessarily refer to a complete input. For example, if a speech recognizer timestamps each word in an utterance, the reference point might refer to the timestamp for just one word.

Timestamps of inputs collected by different devices will be subject to variation if the times maintained by the devices are not synchronized. This concern is outside of the scope of the EMMA working group.

3.13 Relative positioning of input elements

Introduced in section 2.1, the emma:sequence element is used to indicate that the contained interpretations are sequential in time, as in the following example:.

<emma:emma emma:version="1.0"
 xmlns:emma="http://www.w3.org/2003/04/emma#"
 xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
  <emma:interpretation>
    <action>move</action>
    <object>this-battleship</object>
    <destination>here</destination>
  </emma:interpretation>
  
  <emma:sequence>
    <emma:interpretation>
      <x>0.253</x>
      <y>0.124<y>
    </emma:interpretation>

    <emma:interpretation>
      <x>0.866</x>
      <y>0.724<y>
    </emma:interpretation>
  </emma:sequence">    
</emma:emma>

The emma:sequence container can be combined with emma:one-of and emma:group in arbitrary nesting structures.

3.14 Annotating the medium, mode, and function of user inputs

EMMA provides two properties for the annotation of input modality. One indicating the broader medium or channel (medium) and another indicating the specific mode of communication used on that channel (mode).The input medium is defined from the users perspective and indicates whether they use their voice (acoustic), touch (tactile), or visual appearance/motion (visual) as input. Tactile includes most hand-on input device types such as pen, mouse, keyboard, and touch screen. Visual is used for camera input.

emma:medium ::= [acoustic|tactile|visual]

The mode property provides the ability to distinguish between different modes of communication that may be within a particular medium. For example, in the tactile medium, modes include electronic ink (ink), and pointing and clicking on a graphical user interface.

emma:mode ::= [speech|dtmf_keypad|ink|gui|keys|video|photograph| ... ]

Orthogonal to the mode, user inputs can also be classified with respect to their communicative function. This enables a simpler mode classification.

emma:function ::= [recording|transcription|dialog|verification| ... ]

For example, speech can be used for recording (e.g. voicemail), transcription (e.g. dictation), dialog (e.g interactive spoken dialog systems), and verification (e.g. identifying the user through their voiceprint).

EMMA also supports an additional property verbal which distinguishes verbal use of an input mode from non-verbal. This can be used to distinguish the use of electronic ink to convey handwritten commands from the user of electronic ink for symbolic gestures such as circles and arrows.  Handwritten commands, such as writing downtown in order to change a map display to show the downtown are classified as verbal (verbal="true"). Pen gestures (arrows, lines, circles, etc), such as circling a building, are classified as non-verbal dialog (function="dialog" verbal="false"). The use of handwritten words to transcribe an email message are classified as transcription (function="transcription").

emma:verbal ::= [true|false|0|1]

Handwritten words and ink gestures are typically recognized using different kinds of recognition components (handwriting recognizer vs. gesture recognizer) and the verbal annotation will be added by the recognition component which classifies the input. The original input source, a pen in this case, will not be aware of this difference. The input source identifier will tell you that the input was from a pen of some kind but will not tell you if the mode of input was handwriting (show downtown) or gesture (e.g. circling an object or area).

Here is an example of the EMMA annotation for a pen input where the user's ink is recognized as either a word or as an arrow:

<emma:emma emma:version="1.0"
 xmlns:emma="http://www.w3.org/2003/04/emma#"
 xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
  <emma:one-of>       
    <emma:interpretation emma:id="interp1">               
      <location>Boston</location> 
    </emma:interpretation>
    <emma:interpretation emma:id="interp2">               
      <direction>45</direction>       
    </emma:interpretation>
  </emma:one-of>

  <rdf:RDF>
    <rdf:Description rdf:about="#interp1"       
      emma:confidence="0.6"  
      emma:medium="tactile"       
      emma:mode="ink"       
      emma:function="dialog"       
      emma:verbal="true"/> 
    <rdf:Description rdf:about="#interp2"       
      emma:confidence="0.4"       
      emma:medium="tactile"       
      emma:mode="ink"       
      emma:function="dialog"       
      emma:verbal="false"/> 
  </rdf:RDF>
</emma:emma>

An analogous example for the inline syntax is as follows:

<emma:emma emma:version="1.0" xmlns:emma="http://www.w3.org/2003/04/emma#">
  <emma:one-of>       
    <emma:interpretation emma:id="interp1"
     emma:confidence="0.6"  
     emma:medium="tactile"       
     emma:mode="ink"       
     emma:function="dialog"       
     emma:verbal="true"> 
       <location>Boston</location> 
    </emma:interpretation>
    <emma:interpretation emma:id="interp2"
     emma:confidence="0.4"       
     emma:medium="tactile"       
     emma:mode="ink"       
     emma:function="dialog"       
     emma:verbal="false">
       <direction>45</direction>       
    </emma:interpretation>
  </emma:one-of>
</emma:emma>

Here is an example of the EMMA annotation for a spoken command which is recognized as either Boston or Austin:

<emma:emma emma:version="1.0"
 xmlns:emma="http://www.w3.org/2003/04/emma#"
 xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
  <emma:one-of>       
    <emma:interpretation emma:id="interp1">               
      <location>Boston</location>       
    </emma:interpretation>       
    <emma:interpretation emma:id="interp2">               
      <location>Austin</location>       
    </emma:interpretation>
  </emma:one-of>
  
  <rdf:RDF>
    <rdf:Description rdf:about="#interp1"       
      emma:confidence="0.6"       
      emma:medium="acoustic"       
      emma:mode="speech"       
      emma:function="dialog"       
      emma:verbal="true"/>
    <rdf:Description rdf:about="#interp2"       
      emma:confidence="0.4"       
      emma:medium="acoustic"       
      emma:mode="speech"       
      emma:function="dialog"       
      emma:verbal="true"/>
  </rdf:RDF>
</emma:emma>

The following table shows the relationship between the medium, mode, and function properties and serves as an aid for classifying inputs. For the dialog function it also shows some examples of the classification of inputs as verbal vs. non-verbal.

3.15 Composite multimodal input

In addition to supporting multimodal interfaces in which the user can provide input in one mode or another (e.g. say a city name or click one on the display), EMMA also supports multimodal systems in which users employ multiple modes within a single utterance. As an example, the user might say "how do I get from here to there" and point at two locations on a map display. The examples in Section 3.14 above show how emma:mode and emma:medium are annotated in EMMA for unimodal inputs. In the annotation of composite multimodal inputs, if the combining inputs have different values for the emma:medium property the emma:medium of the composite interpretation is multiple. Similarly, if the combining inputs have different values for the emma:mode property the emma:mode of the composite interpretation is multiple. As an example, assume a multimodal interface which supports simultaneous input from voice and pen. At a certain point in the multimodal dialogue the user is shown a map of the USA with airports marked. The user says "destination" and circles Boston on the map. In this case both modes contribute part of the meaning. The speech gives you <destination/> and pen gesture gives you the value, Boston. The semantic representation assigned to this combination of inputs after multimodal integration is <destination> Boston </destination>. The EMMA representation for this input is as follows:

<emma:emma emma:version="1.0"
 xmlns:emma="http://www.w3.org/2003/04/emma#"
 xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
  <emma:interpretation id="multimodal-1">
    <destination>Boston</destination>
  </emma:interpretation>

  <rdf:RDF>
    <rdf:Description rdf:about="#multimodal-1"
        emma:confidence="0.3"
        emma:medium="multiple"
        emma:mode="multiple"
        emma:function="dialog"
        emma:verbal="true"/>
  </rdf:RDF>
</emma:emma>

and for the inline syntax:

<emma:emma emma:version="1.0"
   xmlns="http://www.w3.org/2003/04/emma#">
   <emma:interpretation id="multimodal-1"
        emma:confidence="0.3"
        emma:medium="multiple"
        emma:mode="multiple"
        emma:function="dialog"
        emma:verbal="true">
        <destination> Boston </destination>
   </emma:interpretation>
</emma:emma>

There are tentative new proposals for taking the approach to representing multiple levels of processing of input (Section 3.2, Section 3.4) and using the <emma:derived-from> element and the emma:process attribute to represent the interpretation of inputs both before and after multimodal integration. This will enable access to properties of each component input, such as emma:medium and emma:mode, through the URI in the emma:resource attribute of the <emma:derived-from> element. The use of these annotations is illustrated by the following example:

<emma:emma emma:version="1.0"
xmlns="http://www.w3.org/2003/04/emma#"
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
<emma:interpretation emma:id="speech1">
        <rawinput>destination</rawinput>
</emma:interpretation>
<emma:interpretation emma:id="pen1">
        <rawinput>G selection Boston</rawinput>
</emma:interpretation>
<emma:interpretation emma:id="multimodal1">
        <destination>Boston</destination>
</emma:interpretation>
<rdf:RDF>
<rdf:Description rdf:about="#speech1"
        emma:process="http://example.com/myasr.xml"
        emma:confidence="0.6"
        emma:medium="acoustic"
        emma:mode="speech"
        emma:function="dialog"
        emma:verbal="true"/>
<rdf:Description rdf:about="#pen1"
        emma:process="http://example.com/mygesturereco.xml"
        emma:confidence="0.5"
        emma:medium="tactile"
        emma:mode="ink"
        emma:function="dialog"
        emma:verbal="false"/>
<rdf:Description rdf:about="#multimodal1"
        emma:process="http://example.com/myintegrator.xml"
        emma:confidence="0.3"
        emma:medium="multiple"
        emma:mode="multiple"
        emma:function="dialog"
        emma:verbal="true">
        <emma:derived-from resource="#speech1" />
        <emma:derived-from resource="#pen1" />
</rdf:Description>
</rdf:RDF>
</emma:emma>

and for the inline syntax:

<emma:emma emma:version="1.0"
xmlns="http://www.w3.org/2003/04/emma#">
   <emma:interpretation emma:id="speech1"
        emma:process="http://example.com/myasr.xml"
        emma:confidence="0.6"
        emma:medium="acoustic"
        emma:mode="speech"
        emma:function="dialog"
        emma:verbal="true">
          <rawinput>destination</rawinput>
   </emma:interpretation>

   <emma:interpretation emma:id="pen1"
        emma:process="http://example.com/mygesturereco.xml"
        emma:confidence="0.5"
        emma:medium="tactile"
        emma:mode="ink"
        emma:function="dialog"
        emma:verbal="false">
          <rawinput>G selection Boston</rawinput>
   </emma:interpretation>

  <emma:interpretation emma:id="multimodal1"
        emma:process="http://example.com/myintegrator.xml"
        emma:confidence="0.3"
        emma:medium="multiple"
        emma:mode="multiple"
        emma:function="dialog"
        emma:verbal="true">
          <destination>Boston</destination>
          <emma:derived-from resource="#speech1" />
          <emma:derived-from resource="#pen1" />
   </emma:interpretation>
</emma:emma>

4. Extensibility

(TBD)

5. Conformance

Conformance issues are deferred until a later revision of the specification.

6. Formal syntax

This section defines the formal syntax for EMMA documents in terms of a normative XML Schema, an informative Document Type Definition (DTD) and a normative RDF Schema for the RDF properties defined by EMMA.

6.1 XML schema

(TBD)

Leading and trailing spaces in utterances are not significant. This will be defined in the Schema by specifying "xml:space=default".

6.2 Document type definition

(TBD)

(This section is informative)

6.3 RDF schema

(TBD)

This is the RDF Schema for the RDF properties defined in the EMMA namespace. It provides a binding of these properties to human readable descriptions. Implementors are recommended to provide an RDF Schema for any application specific RDF properties used to extend EMMA. It is NOT permissible to include such extensions within the EMMA namespace.

7. References

Normative References

RDF Schema

Dan Brickley, R.V. Guha, editors. RDF Vocabulary Description Language 1.0: RDF Schema, World Wide Web Consortium, 2003.

RDF XML Syntax

Dave Beckett and Brian McBride, editors. RDF/XML Syntax Specification (Revised), World Wide Web Consortium, 2003.

SI

Luc Van Tichelen, editor. Semantic Interpretation for Speech Recognition, World Wide Web Consortium, 2003

SRGS

Andrew Hunt, Scott McGlashan, editors. Speech Recognition Grammar Specification Version 1.0, World Wide Web Consortium, 2002

XML

Tim Bray, Jean Paoli, C.M. Sperberg-McQueen, and Eve Maler, editors. Extensible Markup Language (XML) 1.0 (Second Edition). World Wide Web Consortium, 2000.

XML Schema Structures

Henry S. Thompson, David Beech, Murray Maloney, Noah Mendelsohn, editors. XML Schema Part 1: Structures, World Wide Web Consortium, 2001.

XML Schema Datatypes

Paul V. Biron, Ashok Malhotra, editors. XML Schema Part 2: Datatypes, World Wide Web Consortium, 2001.

XPath

James Clark and Steve DeRose, editors. XML Path Language (XPath), World Wide Web Consortium, 1999.

XPointer

Paul Grosso, Eve Maler, Jonathan Marsh and Norman Walsh, editors. XPointer Framework, World Wide Web Consortium, 2003

Informative References:

MMI Framework

James A. Larson, T.V. Raman and Dave Raggett, editors. W3C Multimodal Interaction Framework, World Wide Web Consortium, 2003

MMI Requirements

Stéphane H. Maes and Vijay Saraswat, editors. Multimodal Interaction Requirements, World Wide Web Consortium, 2003

EMMA Requirements

Stéphane H. Maes and Stephen Potter, editors. Requirements for EMMA, World Wide Web Consortium, 2003

8. Acknowledgements

The editors would like to thank the other members of the W3C Multimodal Interaction Group (listed in alphabetical order):

Paolo Baggia, Loquendo
Daniel Burnett, Nuance Communications
Max Froumentin, W3C
Katriina Halonen, Nokia
Michael Johnston, AT&T Labs
Gerald McCobb, IBM
Stephen Potter, Microsoft
Yuan Shao, Canon