Technical Blog Post
Abstract
Iterating through response pages queried from Registry Services using Java
Body
This is a bit of sample code I am currently working on, to illustrate how developers can query record collections from the Registry Services component of Jazz for Service Management using pagination support. In this example I will not do anything to the records, but simply show how to list all the URLs for all records for a given Service Provider.
The sample assumes dependencies on both Apache Jena and Eclipse Lyo, the later of which requires Apache Wink. I apologize for the profusion of dependencies, but Jena does a superb job of disentangling RDF models, Lyo does an equally capable job of mapping RDF representations to Java objects, and Wink simplifies the handling of HTTP interactions. One could adapt the sample to make do with only Jena, but at great expense of added code.
At a high-level view, the sample below uses Eclipse Lyo to query all service providers from a Registry Services server, then queries all registration records associated with each provider.
Since a query of all records for a given service provider may not fit in a single response from the Registry Services application (maximum response sizes are capped at 500 records by default) , the sample successively looks for a “ResponseInfo” element in each response (see example of a paginated response from the servers in Figure 1) .For more information on the OSLC Core specification underlying the Registry Services responses, such as the structure of an oslc:ResponseInfo structure, one can refer to http://open-services.net/bin/view/Main/OslcCoreSpecification and to the JazzSM infocenter.
For each page, the sample iterates through the list of URIs of Registration Records for the Service Providers, issuing an HTTP operation against each URI and handling the most likely answers from the server.
Note: There is an APAR affecting pagination prior to Jazz SM FixPack 1 (due in July) : https://longspeakz.boulder.ibm.com/WebRetain/DispatcherServlet?oper=aparPtfDisplay&aparptfnumber=IV39742.
Figure 1: Example of paginated query response
<rdf:RDF
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:rr="http://jazz.net/ns/ism/registry#"
xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"
xmlns:oslc_auto="http://open-services.net/ns/auto#"
xmlns:oslc="http://open-services.net/ns/core#">
<oslc:ResponseInfo rdf:about="http://dii01.tivlab.raleigh.ibm.com:16310/oslc/rr/registration/collection?oslc.pageSize=1">
<oslc:nextPage rdf:resource="http://dii01.tivlab.raleigh.ibm.com:16310/oslc/rr/registration/collection?oslc.pageSize=1
&pagnum=1&lastid=1355782933325&timestamp=1355782953419&oslc.paging=true"/>
<rr:currentTime>2013-02-15T18:29:15.836Z</rr:currentTime>
<rr:lastModifiedTime>2012-12-17T22:22:33.419Z</rr:lastModifiedTime>
<oslc:totalCount>2</oslc:totalCount>
</oslc:ResponseInfo>
<rdfs:Container rdf:about="http://dii01.tivlab.raleigh.ibm.com:16310/oslc/rr/registration/collection">
<rdfs:member rdf:resource="http://dii01.tivlab.raleigh.ibm.com:16310/oslc/registration/1355782933325"/>
</rdfs:Container>
</rdf:RDF>
Figure 2: Entire code sample for InspectRegistrationRecords.java
package com.ibm.frs.samples.oslc;
…
public class InspectRegistrationRecords {
…
/**
* HTTP header indicating how much time to wait for retrying an operation
* answered with an HTTP status 503 (server unavailable)
*/
private static final String HTTP_HEADER_RETRY_AFTER = "Retry-After";
/**
* Default amount of time to wait if a retry interval cannot be read from a
* server response with HTTP status 503.
*/
private static final long DEFAULT_RETRY_INTERVAL_IN_MILLIS = 2000;
/**
* Maximum number of attempts for retrying an operation answered with an
* HTTP status 503 (server unavailable)
*/
private static final int MAX_RETRIES = 3;
/**
* URI of Registry Services registration collection link returned in a
* "Link" response header.
*/
private static final String JAZZSM_REGISTRATION_COLLECTION_LINK = "http://jazz.net/ns/ism/registry#registration-record-collection";
/**
* OSLC4J media providers
*/
private static final Set<Class<?>> OSLC4J_PROVIDERS = new HashSet<Class<?>>();
static {
OSLC4J_PROVIDERS.addAll(JenaProvidersRegistry.getProviders());
OSLC4J_PROVIDERS.addAll(Json4JProvidersRegistry.getProviders());
}
/**
* Default bootstrap URI for an OSLC server, it indicates the full service
* provider information.
*/
private static final String DEFAULT_OSLC_BOOTSTRAP_URI = "http://localhost:16310/oslc/pr/collection";
/**
* Basic Authentication credentials
*/
private static final String BASICAUTH_USER = "app";
private static final String BASICAUTH_PASSWORD = "test";
private static String oslcServerUser = BASICAUTH_USER;
private static String oslcServerPwd = BASICAUTH_PASSWORD;
/**
* Logging handler.
*/
private static final Logger LOG = Logger
.getLogger(InspectRegistrationRecords.class.getName());
/**
* URI for RDF record type definitions
*/
private static final Property RECORD_TYPE_PROPERTY = ResourceFactory
.createProperty("http://www.w3.org/1999/02/22-rdf-syntax-ns#",
"type");
//
// RDF URIs required to process paginated responses from Registry Services
//
private static final com.hp.hpl.jena.rdf.model.Resource RESPONSE_INFO_RESOURCE = ResourceFactory
.createResource("http://open-services.net/ns/core#ResponseInfo");
private static final Property RESPONSE_INFO_NEXT_PROPERTY = ResourceFactory
.createProperty("http://open-services.net/ns/core#", "nextPage");
private static final Property MEMBERS_PROPERTY = ResourceFactory
.createProperty("http://www.w3.org/2000/01/rdf-schema#", "member");
/*
* Public methods.
*/
/**
* Runs the full OSLC server test suite.
*
* @param args
* OSLC bootstrap URI, such as http://localhost:8080/oslc/pr
*
* @throws OslcSampleException
* if the test fails for whatever reason.
*/
public static void main(String args[]) throws OslcSampleException {
InspectRegistrationRecords osft = new InspectRegistrationRecords();
try {
osft.run(args);
} catch (Exception e) {
throw new OslcSampleException(e.getMessage(), e);
}
}
/**
* Delegate for running the full OSLC server test suite.
*
* @param args
* OSLC bootstrap URI, such as http://localhost:16310/oslc/pr/collection
* user
* password
*
* @throws OslcSampleException
*/
public void run(String[] args) throws OslcSampleException {
String oslcServerBootstrapUri = (args.length > 0) ? args[0]
: DEFAULT_OSLC_BOOTSTRAP_URI;
LOG.info("[OK] Using bootstrap URI: " + oslcServerBootstrapUri);
oslcServerUser = (args.length > 1) ? args[1] : BASICAUTH_USER;
oslcServerPwd = (args.length > 2) ? args[2] : BASICAUTH_PASSWORD;
LOG.info("[OK] User: " + oslcServerUser);
// Queries the service provider registry information
ServiceProvider[] tsprInfo = queryServiceProviderInfo(oslcServerBootstrapUri);
// Locate providers created by other applications, not picking up the
// ones shipped with Registry Services
ArrayList<ServiceProvider> nonFrsProviders = getNonRegistryServicesProviders(tsprInfo);
// Determine the link URLs to other Registry Services collections.
Map<String, String> relUriMapOfLinks = getRelLinks(oslcServerBootstrapUri);
for (ServiceProvider serviceProvider : nonFrsProviders) {
LOG.info("Processing Service Provider with title: "
+ serviceProvider.getTitle() + " at "
+ serviceProvider.getAbout());
processServiceProvider(serviceProvider, relUriMapOfLinks);
}
}
/**
* Processes a service provider, querying all registration records for it,
* iterating through the response pages and processing one record at a time.
*
* @param serviceProvider
* URI of the service provider
* @param relUriMapOfLinks
* Map of links retrieved in the first query to Registry
* Services. The key is the name of a link (e.g.
* http://jazz.net/ns
* /ism/registry#registration-record-collection) and the value is
* the actual URL for the link.
*/
private void processServiceProvider(ServiceProvider serviceProvider,
Map<String, String> relUriMapOfLinks) {
// Querying the collections of resources for the provider.
final RestClient httpClient = new RestClient();
final String rrRegistrationCollection = relUriMapOfLinks
.get(JAZZSM_REGISTRATION_COLLECTION_LINK);
final String rrOriginalQueryNextPageUri = rrRegistrationCollection + "?"
+ "oslc.where=oslc:serviceProvider=%3C"
+ serviceProvider.getAbout().toASCIIString() + "%3E";
// Iterating through paginated collections of registration
// records
String rrEncodedQueryNextPageUri = rrOriginalQueryNextPageUri;
while (rrEncodedQueryNextPageUri != null) {
LOG.info("Processing registration record collection page: "
+ rrEncodedQueryNextPageUri);
rrEncodedQueryNextPageUri = processRegistryResponsePage(httpClient,
rrEncodedQueryNextPageUri );
} // while next page
}
/**
* Processes a response page for a query to Registry Services.
*
* @param httpClient
* HTTP client used to issue further requests to the server.
* @param responsePageUri
* URL of the response page to be processed.
*
* @return the URL of the next page to be processed, if there is any,
* otherwise it returns <code>null</code>.
*/
private String processRegistryResponsePage(final RestClient httpClient,
String responsePageUri) {
Resource rrRegistrationCollectionResource = httpClient
.resource(responsePageUri);
rrRegistrationCollectionResource.header(HttpHeaders.ACCEPT,
"application/rdf+xml");
setBasicAuthHeader(rrRegistrationCollectionResource);
final String queryResponse = rrRegistrationCollectionResource
.get(String.class);
final Model queryModel = ModelFactory.createDefaultModel();
queryModel.read(new StringReader(queryResponse), null);
//
// List all members of the collection
//
StmtIterator membersStmtIterator = queryModel.listStatements(null,
MEMBERS_PROPERTY, (String) null);
while (membersStmtIterator.hasNext()) {
final Statement stmt = membersStmtIterator.nextStatement();
final com.hp.hpl.jena.rdf.model.Resource stmtResource = stmt
.getResource();
final String registrationRecordUrl = stmtResource.getURI();
final Resource registrationRecordResource = httpClient
.resource(registrationRecordUrl);
setBasicAuthHeader(registrationRecordResource);
registrationRecordResource.header(HttpHeaders.ACCEPT, "*/*");
boolean needRetry = false;
int retries = 0;
do {
ClientResponse operationResponse;
try {
operationResponse = registrationRecordResource.<http operation goes here>;
} catch (ClientRuntimeException e) {
final Throwable eCause = e.getCause();
final String msg = eCause.getMessage()
+ ". Resuming in 2 seconds..";
LOG.severe(msg);
waitAMoment(2000);
continue;
}
final int operationResponseStatusCode = operationResponse
.getStatusCode();
LOG.info("About to process: " + registrationRecordUrl);
needRetry = false;
switch (operationResponseStatusCode) {
// Success response
case HttpURLConnection.HTTP_NO_CONTENT:
case HttpURLConnection.HTTP_OK:
// waiting a little to avoid overloading the server.
waitAMoment(100);
break;
// One of the reasons for the server being unavailable
// is the record being processed by another thread, so
// a retry is needed.
case HttpURLConnection.HTTP_UNAVAILABLE:
long retryInterval = determineRetryAfterInterval(operationResponse);
retries++;
if (retries < MAX_RETRIES) {
LOG.warning("Record was possibly locked, retry #"
+ retries + " for processing of record in "
+ retryInterval + " ms");
waitAMoment(retryInterval);
needRetry = true;
} else {
LOG.severe("Unable to process the record at "
+ registrationRecordUrl + " after "
+ MAX_RETRIES + " attempts.");
needRetry = false;
}
break;
case HttpURLConnection.HTTP_INTERNAL_ERROR:
waitAMoment(1000);
// intentional, no break
default:
final String errMsg = buildClientResponseErrorMessage(
registrationRecordUrl, operationResponse);
LOG.severe(errMsg);
}
// Resetting retries counter if retries no longer needed
if (!needRetry) {
retries = 0;
}
} while (needRetry && retries < MAX_RETRIES);
} // while member of response collection
responsePageUri = determineNextPage(queryModel);
return responsePageUri;
}
/**
* Returns the URL of the next page of records for Registry Services.
*
* @param queryModel
* Jena RDF model of the page queried from Registry Services.
*
* @return The URL of the next page listed in the page passed as a parameter
* or <code>null</code> if that page was the last page in the
* sequence.
*/
private String determineNextPage(final Model queryModel) {
String nextPageUri = null;
final ResIterator riter = queryModel.listResourcesWithProperty(
RECORD_TYPE_PROPERTY, RESPONSE_INFO_RESOURCE);
if (riter.hasNext()) {
final com.hp.hpl.jena.rdf.model.Resource resource = riter
.nextResource();
final Statement nextPageStmt = resource
.getProperty(RESPONSE_INFO_NEXT_PROPERTY);
if (nextPageStmt != null) {
final com.hp.hpl.jena.rdf.model.Resource nextPageResource = nextPageStmt
.getResource();
nextPageUri = nextPageResource.getURI();
}
}
return nextPageUri;
}
/**
* Infers a recommended wait time (in milliseconds) for attempting a
* requests response with HTTP status 503 (server unavailable).
*
* @param clientResponse
* HTTP response from Registry Services server.
*
* @return the recommended value for a retry attempt.
*/
private long determineRetryAfterInterval(ClientResponse clientResponse) {
String retryAfter = clientResponse.getHeaders().getFirst(
HTTP_HEADER_RETRY_AFTER);
long retryInterval = DEFAULT_RETRY_INTERVAL_IN_MILLIS;
if (retryAfter != null) {
try {
retryInterval = Long.parseLong(retryAfter) * DEFAULT_RETRY_INTERVAL_IN_MILLIS;
} catch (NumberFormatException e) {
LOG.warning("The server did not return a numeric value for the "
+ HTTP_HEADER_RETRY_AFTER
+ " header: "
+ retryAfter
+ " using " + retryInterval + " ms instead.");
}
}
return retryInterval;
}
/**
* Creates a list of service providers that are not the service providers
* already shipped with Registry Services.
*
* @param allServiceProviders
* original list of service providers queries from Registry
* Services.
*
* @return The trimmed-down list.
*/
private ArrayList<ServiceProvider> getNonRegistryServicesProviders(
ServiceProvider[] allServiceProviders) {
ArrayList<ServiceProvider> nonFrsProviders = new ArrayList<ServiceProvider>(
allServiceProviders.length - 2);
for (ServiceProvider serviceProvider : allServiceProviders) {
boolean registryProvider = false;
final Service[] serviceProviders = serviceProvider.getServices();
for (Service service : serviceProviders) {
if ("http://jazz.net/ns/ism/registry#".equals(service
.getDomain().toASCIIString())
|| "http://open-services.net/ns/core#".equals(service
.getDomain().toASCIIString())) {
registryProvider = true;
break;
}
}
if (!registryProvider) {
nonFrsProviders.add(serviceProvider);
}
}
return nonFrsProviders;
}
/**
* Creates a map with the links retrieved from the HTTP response header
* returned when querying the Registry Services provider collection.
*
* The "Link" header looks like this:
* <http://localhost:9081/oslc/rr/collection>;
* rel="http://jazz.net/ns/ism/registry#resource-record-collection",
* <http://localhost:9081/oslc/rr/registration/collection>;
* rel="http://jazz.net/ns/ism/registry#registration-record-collection",
* <http://localhost:9081/oslc/rr/shapes/collection>;
* rel="http://jazz.net/ns/ism/registry#resource-shape-collection",
* <http://localhost:9081/oslc/rr/operations/collection>;
* rel="http://jazz.net/ns/ism/registry#resource-automation-collection",
* <http://localhost:9081/oslc/pr/operations/collection>;
* rel="http://jazz.net/ns/ism/registry#provider-automation-collection"
*
* @param oslcServerBootstrapUri
* URL for the Registry Services Provider collection
*
* @return A map where the key is the name of the shortcut link (e.g.
* http://jazz.net/ns/ism/registry#resource-record-collection) and
* the value is the actual URL for that shortcut link.
*/
private Map<String, String> getRelLinks(String oslcServerBootstrapUri) {
Map<String, String> relUriMapOfLinks;
final OslcRestClient client = new OslcRestClient(OSLC4J_PROVIDERS,
oslcServerBootstrapUri);
setBasicAuthHeader(client);
final Resource clientResource = client.getClientResource();
final ClientResponse clientResponse = clientResource.get();
final MultivaluedMap<String, String> clientResponseHeaders = clientResponse
.getHeaders();
final String linkHeader = clientResponseHeaders.getFirst("Link");
relUriMapOfLinks = new TreeMap<String, String>();
String[] links = linkHeader.split(",");
for (String link : links) {
String[] linkParts = link.split(";");
String url = linkParts[0].substring(1, linkParts[0].length() - 1);
String relUri = linkParts[1].split("\"")[1];
relUriMapOfLinks.put(relUri, url);
}
return relUriMapOfLinks;
}
/**
* Retrieves the ServiceProvider from the provided URI.
*
* @param providerUri
* the service provider URI.
* @return the ServiceProvider object created from the information obtained
* from the provided URI.
* @throws OslcSampleException
*/
private ServiceProvider[] queryServiceProviderInfo(String providerUri)
throws OslcSampleException {
final String providerUriSelectAll = providerUri + "?oslc.select=*";
final OslcRestClient client = new OslcRestClient(OSLC4J_PROVIDERS,
providerUriSelectAll);
setBasicAuthHeader(client);
LOG.info("Client timeout (in ms) is: " + client.getReadTimeout());
// Retrieve the Service Providers collection via OSLC4J
final ServiceProvider[] providers;
try {
providers = client.getOslcResources(ServiceProvider[].class);
} catch (ClientWebException e) {
final ClientResponse clientResponse = e.getResponse();
String msg = buildClientResponseErrorMessage(providerUri,
clientResponse);
throw new OslcSampleException(msg);
}
// Process the client response.
final Resource clientResource = client.getClientResource();
final ClientResponse clientResponse = clientResource.get();
final int statusCode = clientResponse.getStatusCode();
if (HttpURLConnection.HTTP_OK != statusCode) {
String msg = buildClientResponseErrorMessage(providerUri,
clientResponse);
LOG.severe(msg);
throw new OslcSampleException(msg);
}
LOG.info("[OK] Retrieved provider information from: " + providerUri);
return providers;
}
/**
* Extracts all information from a client response into a readable message.
*
* @param requestUri
* URL from where the response was received
* @param clientResponse
* the client response
*
* @return a readable message containing all details of the message.
*/
private String buildClientResponseErrorMessage(String requestUri,
final ClientResponse clientResponse) {
final String errMsg = clientResponse.getEntity(String.class);
final int statusCode = clientResponse.getStatusCode();
String msg = "[FAIL] Received error response from: " + requestUri
+ ". Status code is: " + statusCode + ". Response message is: "
+ clientResponse.getMessage() + ". Response body is: " + errMsg;
return msg;
}
/**
* Sets the basic authentication header on the rest client.
*
* @param client
* the rest client.
*/
private void setBasicAuthHeader(OslcRestClient client) {
setBasicAuthHeader(client.getClientResource());
}
/**
* Sets the basic authentication header on the web resource.
*
* Refer to http://sourcepatch.blogspot.com/2013/02/base-64-encoding-with-java-without.html
* for the Base64 class.
*
* @param resource
* the web resource.
*/
private static void setBasicAuthHeader(Resource resource) {
String userPass = oslcServerUser + ":" + oslcServerPwd;
String userPassBase64 = Base64.encode(userPass);
resource.header(HttpHeaders.AUTHORIZATION, "Basic " + userPassBase64);
}
/**
* Improper exception handling in this method, only used for simplicity in a sample.
*
* @param waitInMillis
*/
private void waitAMoment(long waitInMillis) {
try {
Thread.sleep(waitInMillis);
} catch (InterruptedException e) {
}
}
}
UID
ibm11275484