WS-Enumeration 1.4 Developers Guide

Navigation
caGrid	caGrid 1.4 Documentation
WS-Enumeration	WS-Enumeration 1.4 Documentation	WS-Enumeration 1.4 Developers Guide

CaGrid provides mechanisms for integration of the WS-Enumeration specification both singularly and as part of the BulkDataTransport support. This integration is accomplished with a variety of provided tools for both client and server side use, as well as a specialized service extension to the Introduce toolkit.

Relevant external links:

• The The WS-Enumeration spec document is available from Globus
• WS-Enumeration project at dev.globus.org

	Contents caGrid Implementation Details Enumeration Responses Extension Implementation Using Enumeration in a Grid Service Creating a Service With Enumeration in Introduce Adding a Service Operation that uses Enumeration Editing a Service With Enumeration in Introduce Enumeration Implementations WS-Enumeration Implementations Creating an Enumeration Response Example Client API Utilities Examples Basic caGrid WS-Enumeration Enumeration with deserialization

caGrid Implementation Details

While caGrid tries to remain faithful to the WS-Enumeration spec to the greatest extent possible, some important differences exist between the spec document and the actual implementation. These differences exist for important technical reasons, highlighted here.

Enumeration Responses

The WS-Enumeration spec makes the assumption that the service context which provides a method to begin an enumeration is also the same context which implements the other enumeration methods (Pull, Renew, Status, etc). In caGrid, the data to be enumerated is stored in a server side WSRF resource. This means the method to begin an enumeration must return both the spec-required Enumeration Context and an Endpoint Reference (EPR) indicating the service context which will be responsible for handling the enumeration, as well as the resource key for the data. In caGrid, this combined response is known as an Enumeration Response Container.

WS-Enumeration is used with caGrid data services. Utilities exist to simplify the creation of the Enumeration Response Container.

Extension Implementation

caGrid's support for WS-Enumeration is provided by a service extension to the Introduce toolkit. This extension adds the service context for WS-Enumeration operations, copies relevant WSDL files and schemas, and finally sets the Globus provided EnumProvider class as the implementation for the enumeration operations.

Using Enumeration in a Grid Service

To utilize WS-Enumeration in a grid service, two things are required. First, the service must be generated with the caGrid WS-Enumeration extension enabled. Next, one or more operations must be added to the grid service which return an Enumeration Response Container.

Such a "begin enumeration" method is required to begin the enumeration and hand off control of some data resource to the enumeration context and resource. The WS-Enumeration resource requires an implementation of the org.globus.ws.enumeration.EnumIterator interface be supplied to it at creation time. This interface is the means through which a data resource is exposed with the server side enumeration implementation. To simplify the process of exposing data via this interface, the caGrid enumeration implementation includes several utilities.

Creating a Service With Enumeration in Introduce

A service can be created which will use WS-Enumeration via the Introduce toolkit by selecting the extension at creation time.

Selecting the WS-Enumeration Extension

Adding a Service Operation that uses Enumeration

Now that we have the WS-Enumeration extension, we can add an operation that will return an Enumeration Response Container to the client. Here is an example operation that enumerates over stocks in a portfolio.

• Click the Services tab.
• Select the Operations node for the StockPortfolioManager context.
• Click the Add Method button on the middle right.

After the Build/Modify dialog appears,

• Set the Method Name to enumerateStocks.
• Add a method description: Enumerate over stocks in the portfolio
• Select the Output tab.
• Scroll down the list of types and double-click the EnumerationResponseContainer type to add it as output from our method.
• Select the Done button.

• Click the Save button to save the service.

Editing a Service With Enumeration in Introduce

If a service which supports WS-Enumeration is opened in Introduce, a tab is added to the service modification interface which controls the service's default implementation of the server-side EnumIterator interface.

WS-Enumeration Implementation Selection

Enumeration Implementations

caGrid WS-Enumeration supplies the factory class gov.nih.nci.cagrid.wsenum.utils.EnumIteratorFactory which creates concrete instances of the EnumIterator. The factory method createIterator() takes parameters to determine the type of implementation, as well as the list of data objects to be enumerated (or a java.util.Iterator to the same), the XML QName of those objects, and a WSDD configuration stream to manage the serialization of the data objects. The Java enumeration type gov.nih.nci.cagrid.wsenum.utils.IterImplType defines the five implementations and gives a brief description of each:

WS-Enumeration Implementations

• GLOBUS_SIMPLE
  • The Globus-provided simple enum iterator
  • A concrete implementation of the EnumIterator interface. It is a very simple implementation that can enumerate over in-memory data passed either as an array of objects or a list (java.util.List). The enumeration contents can be of javax.xml.soap.SOAPElement type, simple types such as java.lang.Integer, etc. or Axis generated Java beans. The SimpleEnumIterator can only be used with transient type of enumerations.
• GLOBUS_INDEXED_FILE
  • The Globus-provided indexed file enum iterator
  • A memory efficient implementation that can enumerate over data stored in an indexed file created by IndexedObjectFileWriter. The indexed file format is optimized for retrieving objects in a sequential and random manner. The IndexedObjectFileEnumIterator uses the IndexedObjectFileReader to read the indexed file and quickly locate and retrieve the next set of objects of the enumeration. The IndexedObjectFileEnumIterator can be used with transient and persistent types of enumerations.
• CAGRID_SIMPLE
  • A simple iterator which persists objects to disk
  • This iterator makes no attempt to respect any values of IterationConstraints except for maxElements.
• CAGRID_THREADED_COMPLETE
  • A relatively complete implementation of the EnumIterator spec
  • This iterator uses threads to respect maxTime constraints, as well as respecting maxCharacters. Elements overflowing either of these constraints, however, are lost, and waits for threads are not optimized.
• CAGRID_CONCURRENT_COMPLETE
  • The default and most complete implementation of the EnumIterator spec
  • This iterator uses the Java 5 java.util.concurrent package to fully support the WS-Enumeration specification for an EnumIterator implementation. All iteration constraints are respected, and elements which cause maxCharacters to be exceded are queued for later retrieval.

Creating an Enumeration Response

Once an EnumIterator instance has been created, it must be sent to the enumeration resource, and an Enumeration Response Container returned to the user. The caGrid provided utility class gov.nih.nci.cagrid.wsenum.utils.EnumerateResponseFactory encapsulates this functionality in a simple static method. The method createEnumerationResponse() takes a single EnumIterator parameter and returns an Enumeration Response Container, which can immediately be returned to the client. This method encapsulates locating the EnumResourceHome, creating a new resource, setting its visibility, and deriving an endpoint reference (EPR) from the resulting resource key.

Example

Below is an example code snippet which uses the provided utilities to create an enumeration iterator over a list of Strings and return an appropriate enumeration response container:

    // create a list of data values
    List<String> values = new ArrayList<String>();
    for (int i = 0; i < 10; i++) {
        values.add("String Value " + i);
    }

    // create an enum iterator
    EnumIterator enumIter = EnumIteratorFactory.createIterator(
        IterImplType.CAGRID_CONCURRENT_COMPLETE,
        values,
        new QName("http://www.w3.org/2001/XMLSchema", "string"),
        null);

    // formulate the response container and return
    EnumerationResponseContainer response =
        EnumerateResponseFactory.createEnumerationResponse(enumIter);
    return response;

Client API

The Globus-provided org.globus.ws.enumeration.ClientEnumIterator API provides java.util.Iterator abstraction for retrieving enumeration data and supports automatic data deserialization. The caGrid WS-Enumeration implementation provides a simplified factory interface to create a new instance of a Client Enum Iterator from an Enumeration Response Container.

Utilities

The caGrid-provided class gov.nih.nci.cagrid.wsenum.utils.EnumerationResponseHelper contains static methods which can take an Enumeration Response Container and return a client enum iterator instance. The method createClientIterator takes only the response container, while another implementation of this method takes both the container and a java.io.InputStream to the client-config.wsdd file. The information contained in this file will be used to deserialize results from the enumeration.

Examples

Basic caGrid WS-Enumeration

Given a service (here an enumeration-enabled Data Service) which supports enumeration, the following pattern may be used by a client to enumerate over data results.

    TestEnumerationDataServiceClient client = new TestEnumerationDataServiceClient(args[1]);

    EnumerationResponseContainer response = client.enumerationQuery(null);

    ClientEnumIterator iter = EnumerationResponseHelper.createClientIterator(
         response, TestEnumerationDataServiceClient.class.getResourceAsStream("client-config.wsdd"));

    // optional use of iteration constraints to fetch 10 elements w/o regard
    // to number of characters or maximum time to wait
    IterationConstraints cons = new IterationConstraints(10, -1, null);
    iter.setIterationConstraints(cons);

    while (iter.hasNext()) {
         SOAPElement elem = (SOAPElement) iter.next();
         if (elem != null) {
              // handle the data element
         }
         // optionally change constraints to fetch 5 elements, max of
         // 50000 characters, and a max wait time of 1 hr, 30 minutes
         cons = new IterationConstraints(5, 50000,
              new Duration(false, 0, 0, 0, 1, 30, 0));
         iter.setIterationConstraints(cons);
    }

The use of IterationConstraints to express how data should be fetched from the service is optional, and the default behavior is to retrieve one element at a time, with no regard for max characters or time constraints. Some server side implementations to not respect all iteration constraints. These constraints may change at any time during enumeration.

While iterating, it is possible that the call to 'hasNext' may return true, yet the call to 'next' return null, or even throw a NoSuchElementException. This is often the case when results are being dynamically populated on the server side at a rate slower than the client is capable of consuming them. This condition may also arise when the iteration constraints are such that no element could be returned which fulfills the constraints, yet the server is still holding more elements. Changing the constraints (allowing more time, increasing the maximum number of characters) may allow these elements to be retrieved.

If you encounter the following error while iterating: [java] java.util.NoSuchElementException it may be that the container you deployed your service to is misconfigured. During container installation, you can set the hostname for your container. Please re-install your container and re-deploy the service, setting the hostname to localhost or another valid hostname. You can check if this is the problem by inspecting the service URL for your EnumerationResponseContainer in a debugger (client-side). That will print out the URL of the enumeration context that the EnumerationResponseContainer will contact during enumeration. If this is not the error, please see the documentation at the top of this guide for more information on the enumeration implementation. It may be that the EnumerationResponseContainer the server is returning contains zero elements, which will result in an immediate NoSuchElementException (even though hasNext() returns true). This case can be handled simply by catching this Exception. The third possibility is that the iteration constraints you specified are not met (or... the default constraints are not met). That is, a NoSuchElementException can occur when the server has more data coming, but it is not yet ready. Thus, when the client asks for another element, the server throws this Exception indicating it is not yet ready. This is a potential area of improvement for a future version of caGrid. Please contact the caGrid Knowledge Center if you need more well-defined behavior for enumeration iterators to let them know about your use cases.

Enumeration with deserialization

In the previous section, the iterator returns SOAPElement objects. For most cases, we want to return the actual object that you expect from the enumeration container, such as a String or other Introduce type. Here is a code snippet that shows an example of iterating and de-serializing the enumeration objects:

EnumerationResponseContainer enumerator = portfolio1.enumerateStocks();
ClientEnumIterator iter = EnumerationResponseHelper.createClientIterator(enumerator, StockManagerClient.class.getResourceAsStream("client-config.wsdd"));
System.out.println("Portfolio 1 contents:");
while (iter.hasNext()) {
   //deserialize the MessageElement into a String (our stock symbol)
   MessageElement o = (MessageElement)iter.next();
   String elemText = o.toString();
   String stock = null;
   try {
      stock = (String)Utils.deserializeObject(new StringReader(elemText), String.class);
      System.out.println(stock);
   } catch (Exception ex) {
     ex.printStackTrace();
   }
}

In the above code snippet, iterator is returning Strings (xsd:string is what is specified in the schema). For each element in the iterator, we get a handle to a MessangeElement, then get the contents of the MessageElement as text (our xs:string XML fragment), and then de-serialize that using the Utils class provided with cagrid. Note that the Utils class is in the following package: gov.nih.nci.cagrid.common.Utils