Using IBM Worklight to develop cross-platform HTML5 video play hybrid applications

With the rush to extend enterprise access beyond computers to mobile devices, mobile hybrid apps are being used to take advantage of HTML5’s cross-platform capabilities. Unfortunately, support for HTML5 falls short when it comes to cross-platform video play, particularly in hybrid apps running on the Android operating system. This article shows how you can work around those issues and enable video play by leveraging the mobile hybrid capabilities of IBM® Worklight. This content is part of the IBM WebSphere Developer Technical Journal.

Bill Paris, Software Developer, IBM China

Bill Paris is a software developer consulting for the IBM Software Services for WebSphere group. He specializes in the development of mobile applications and server middleware.



Sreeni Pamidala , Senior Certified IT Architect , IBM China

Sreeni Pamidala is a Senior Certified, IT Architect and currently working as a Lead Architect for ISSW Mobile Services, AIM Software group. In this role, he spends most of his time with customers across all industries to develop and deliver mobile solutions using IBM Mobile Foundation technologies that integrate with their enterprise infrastructure and middleware. Prior to this role, Sreeni worked with ComSector clients for 15 years where he led several successful engagements by developing and deploying complex web, network based carrier grade solutions for the mobile and legacy telephony service providers using IBM's cross-brand stack of software and hardware products and several third party vendors’ integration products for the best of breed solutions.



Raghunandan K Harithas, Consulting IT Specialist, IBM China

Raghunandan Harithas is an IT specialist based out of the IBM Annapolis office in Maryland, USA. He has been working as a technical lead in several telecom projects using stacked WebSphere products. He is currently working in the field of mobile application development using the IBM Mobile Foundation software based on IBM Worklight.



01 August 2012

Also available in Chinese Russian Japanese Vietnamese Portuguese

Introduction

A mobile hybrid app combines native operating system functionality with web technologies. Typically, a hybrid app presents content in an embedded web browser, an approach that enhances cross-platform capabilities because the majority of the code can be written using HTML5 technologies, while at the same time enabling access to native device features when necessary. IBM Worklight is a mobile application platform that enables the development of cross-platform hybrid applications, offering mechanisms to navigate between the web view and the native view, and providing a development and run time environment that moves hybrid apps much closer to the write-once-run-anywhere goal.

Video play within an app is one area where a cross-platform capability can be difficult to achieve. HTML5’s <video> tag was intended to enable cross-platform video play, but inconsistent support for its capabilities fall short of that goal.

This article focuses on developing video play within cross-platform HTML5 hybrid apps and explains how the IBM Worklight development platform can help work around problems when support for HTML5’s video capabilities falls short on any one particular platform.


The challenges of video play

Video play on computers and mobile devices has long posed challenges to developers because development with video requires an understanding of complex technology and terminology. Before going too much further, let’s get some terminology out of the way.

Video is a combination of one video (or picture) stream and one or more audio streams, all packed into a single archive. In common video parlance:

  • A video container wraps audio and video files together into a single file archive. There are lots of video container formats and some popular ones are MPEG4, Flash Video, Ogg, WebM, and Audio Video Interleave. The container format is indicated by the file extension, such as mp4, flv, ogv, webm, and avi along with others.
  • A video codec identifies the software algorithm by which a video stream is encoded and decoded (compressed and decompressed). A video player needs to know which codec was used in order to decode and play the video stream.
  • An audio codec is similar to a video codec, but for audio streams.

The move toward HTML5

A primary challenge of video play comes from the evolution of video play mechanisms in computers and mobile devices. Prior to the HTML5 specification, there was no standard way to play video in browsers, and almost always video was funneled through third-party plug-ins such as RealPlayer, Apple QuickTime, and Adobe Flash. HTML5’s video playing capability came about in part to address the reliance on plug-ins even as the popularity of YouTube for video play made Flash the de-facto standard on desktops. With Apple’s rejection of Flash in favor of HTML5 for video play on iOS devices, cross-domain video play using Flash became impossible and developers began to focus on creating websites that use HTML5 for video play.

HTML5 added a new video tag for directly embedding video content in a web page to play video without the use of plug-ins. Listing 1 show an example of a video tag to play an mp4 video in a browser window. It specifies the width and height, the amount of space allocated to the video player embedded within the browser window, that autoplay is enabled so the video will start playing without user action, and that onscreen video controls (play, pause, volume) are to be enabled. Most importantly, it specifies the video source archive to be played.

Listing 1. HTML5 video tag for embedding video in a webpage
<video width="320px" height="480px" autoplay="autoplay" controls="controls">
	<source src="dir/video.mp4" type="video/mp4"/>
</video>

The adoption of the HTML5 video tag has been encouraging. Initially, Safari was the only browser offering HTML5 video support when the HTML5 specs were drafted, but now all modern browsers support it. Streaming video on the Web is in the midst of a fast transition from Flash plug-ins to the HTML5 standard, even though the HTML5 specification is still in draft form, with expected completion in 2014.

Where HTML5 video play falls short

Unfortunately, browser support for the HTML5 video tag is only part of the story because video play requires more than just tag support. Support for video container formats, video and audio codecs, and streaming protocols all play a big role in web page interoperability across browsers. For instance, browser support for codecs such as H.264 and WebM has been selective, with some browsers supporting one but not the other. HTML5 doesn't specify a codec because standards groups haven’t agreed on a single one, which means web developers contemplating the use of HTML5 video must consider browser compatibility issues.

Additional challenges come about with mobile video play, due to the diverse nature of mobile devices with their differing screen resolutions and processors. For example, there are hundreds of Android variants in the mobile market, all of which need to play audio and video.


Trials, tribulations, and a solution

Our SmarterTVApp project involved the development of a cross-platform mobile hybrid app with video playing capability, and it seemed only natural to use HTML5 video for video play. We put together a development environment using Eclipse augmented with the Worklight Studio 4.2 Eclipse plug-in, server hardware running Worklight Server on Red Hat Linux®, and a video storage server populated with MPEG4 video files. Multiple mobile devices running Android 4.0 Ice Cream Sandwich and Apple iOS 5 were used for testing, and they received video from a video storage server which streamed video using the HTTP Level Streaming (HLS) protocol.

Three-tiered hybrid app model

Worklight-developed apps use a three-tiered cross-platform application model, illustrated in Figure 1. The lowest tier consists of the native operating system functionality provided by native operating system API libraries. For our SmarterTVApp, this consisted of Android or iOS operating system API invocation points that were built into the app.

Figure 1. Three-tiered application model
Figure 1. Three tiered application model

The middle tier consists of the Worklight and Apache Cordova components supplied by Worklight which bridge the HTML5 application code with native device operating system functionality. Apache Cordova (formerly known as PhoneGap) is an open-source mobile development framework that enables the development of hybrid apps by providing access to native operating system capabilities through JavaScript™. The Worklight components in this tier provide client-side functionality, such as device skinning, encrypted storage, push notifications, server integration framework, and many other capabilities.

The top tier consists of application components. In the SmarterTVApp, this consisted of our custom application JavaScript, HTML, and CSS code, along with components we imported to support our application, the IBM Dojo Toolkit release 1.7.2, and the IBM issw.mobile application framework used by our app for presentation and view-to-view transitioning.

Development environment

When a Worklight project is created in Eclipse, the Worklight Studio plug-in creates an initial directory structure populated with a set of folders for the application and runtime environment. The directory structure for SmarterTVApp is shown in Figure 2. The common folder holds the JavaScript, HTML, and CSS files common to all device deployment environments while the android, ipad, and mobilewebapp folders hold device specific optimization files.

Under the common folder, we also added Dojo related folders (dijit, dojo and dojox) to house the Dojo files used by our application, and added an issw folder for the files comprising the issw.mobile application framework.

Figure 2. Worklight project folders
Figure 2. Worklight project folders

A simplified illustration of the development test environment is shown in Figure 3. The Android and iOS devices interact with a server application running on the Worklight Server, which then initiates the streaming of video from a Video Storage Server to the devices.

Figure 3. The development test environment
Figure 3. The development test environment

Initial implementation using HTML5 video

Our initial design made use of HTML5 video to achieve cross-platform video play, so we implemented the video element with two source elements, as shown in Listing 2. When encountering multiple source elements, browsers will use the first recognized format, and our expectation was that the MP4 and OGV formats would provide sufficient cross-browser capability.

Listing 2. HTML5 video tag specifying a video in multiple formats
<video width="320px" height="480px" autoplay="autoplay" controls="controls">
	<source src="dir/video.mp4" type="video/mp4/">
	<source src="dir/video.webm" type="video/webm/">
</video>

Our high hopes for quickly implementing cross-platform video play through the use of HTML5 video were dashed as soon as we began testing our first prototype. With a few tweaks we succeeded in getting video playing on our app running on iOS devices, but we had no such luck when running it on Android. Thus began the trial and error phase of our development.

Working toward a solution

An important consideration here is that the mobile hybrid approach for Android, as implemented by Cordova, is based on the Android WebView class for displaying web pages. A Cordova-based hybrid app uses a WebView to display the HTML5 content portion of the app. In our testing, we found that while our HTML5 video content would play successfully in the Android browser, it would not play when the HTML5 video content was displayed through the hybrid app’s WebView object.

We naturally sought information on the Web and one quick search confirmed our fears: the tech forums were full of postings from anguished developers facing similar WebView HTML5 video play problems on Android 4.0. Armed with the suggestions we read about, we tried a variety of video formats, tried changes to the video tag parameters, and tried changing many dynamic (JavaScript) and static (HTML) aspects of creating the video element and its attributes -- all without success.

So it was time for Plan B: abandon cross-platform video play using the HTML5 video tag and instead use Android native video play functionality when the app was built to run on Android. When built to run on iOS, the app would use HTML5 video play as originally planned.

One benefit of developing with Worklight is that it provides a simple mechanism to incorporate native pages into hybrid apps. Worklight’s hybrid coding functionality enables an app to navigate between web and native pages, and to share data between these pages. By making use of this feature, our hybrid app had the ability to switch to a native page that made use of Android’s Java API set to play video, and then when video play was completed, would switch back to the JavaScript code that was common to both iOS and Android.

With this knowledge, we set about implementing a native page that created an Android Activity using a VideoView object to play video, and at last we had success: video play when running on Android.


Solution in detail

The solution consists of both Android native Java code and JavaScript code. The JavaScript code uses Worklight’s capabilities to invoke the Android Java code to play the video.

Android native Java code to play video

To implement the Java video functionality, we first created a new StreamingVideoActivity.java class under the Android specific project structure location, shown in Figure 4. The Worklight project structure (separating device specific optimization files into separate folders) simplified the addition of the Android native code required by our solution.

Figure 4. Worklight project folder containing Android native Java code
Figure 4. Worklight project folder containing Android native Java code

The StreamingVideoActivity is implemented as an extension of an Android Activity and plays video using the VideoView and MediaController classes. An outline of this implementation is shown in Listing 3. A complete solution would implement callbacks for handling video termination and completion events so as to return control back to Worklight’s web-based screen view.

Listing 3. StreamingVideoActivity.java implementation
public class StreamingVideoActivity extends Activity {
	
    /** Called when the activity is first created. */
    @Override
    public void onCreate(Bundle savedInstanceState) {
    	
	super.onCreate(savedInstanceState);
	Log.d("StreamingVideoActivity", "Entering onCreate");			

	// Extract the URL from the Intent
	String url = getIntent().getStringExtra("urlParam");
		
	Log.d("StreamingVideoActivity", "About to play URL: url");			
	VideoView videoView = new VideoView(this);
	videoView.setVideoPath(url);
		
	MediaController ctlr = new MediaController(this);
	ctlr.setMediaPlayer(videoView);
	videoView.setMediaController(ctlr);
       
	setContentView(videoView);
	videoView.requestFocus();
	Log.d("StreamingVideoActivity", "Leaving onCreate");			
    }
}

Invoking the Android code from the web application

With the Android work complete, all that remained was to create a JavaScript function that uses the Worklight WL.NativePage.show API function to switch from the web-based Activity to the new native Android Activity. The JavaScript file SmarterTVApp.js implementing this functionality was placed in the Android specific project structure location shown in Figure 5.

Figure 5. Worklight project folder containing Android specific JavaScript file
Figure 5. Worklight project folder containing Android specific JavaScript file

The code excerpt from SmarterTVApp.js in Listing 4 shows the implementation of the openNativePage function that invokes the Worklight WL.NativePage.show function to run the StreamingVideoActivity Activity.

Listing 4. SmarterTVApp.js code excerpt
/**
 * Plays the specified video in an Android native page
 * @param url  The video URL
 */
function openNativePage (url) {
	WL.Logger.debug("Switching to SmarterTVApp.StreamingVideoActivity to play " +url);

	// Create an object to hold the URL.  The field name, urlParam, must match
	// the name used in the native Android Java code for extracting the URL
	var params = {urlParam : url};

	// Show the Android native page
	WL.NativePage.show('com.SmarterTVApp.StreamingVideoActivity', 
		backFromNativePage, params);
}

/**
 * Invoked as a call-back on return from the Android native page
 * @param data
 */
function backFromNativePage(data) {
	WL.Logger.debug("Back from StreamingVideoActivity");
}

Finally, we needed to conditionally call openNativePage when the hybrid was running on an Android device, and this is where the Worklight Studio development environment was again helpful. Rather than detecting the device type and adding if-then-else JavaScript logic, the Worklight project structure handles this automatically. In our application video, play is carried out in the StreamingView.js file. By creating two versions of the file as shown in Figure 6, one stored in the common folder that initiates HTML5 video play and a second stored in the android folder that invokes the Worklight openNativePage function to initiate Android native video play, the Worklight Studio automatically handled the inclusion of the file version appropriate to the run time environment.

Figure 6. Dual implementations of video streaming
Figure 6. Dual implementations of video streaming

Click to see larger image

Figure 6. Dual implementations of video streaming

Figure 6. Dual implementations of video streaming

Conclusion

From this project we learned cross-platform video play can be difficult to achieve and the solution can require the use of native device functionality. Worklight simplifies development through the use of its capabilities for combining native and web-based functionality.


Acknowledgements

The authors wish to thank our colleagues Anton Aleksandrov, Raanan Avidor, Karl Bishop and Tom Thacher for their technical contributions and guidance that led to the success of this project.

Resources

Learn

Get products and technologies

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into WebSphere on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=WebSphere, Mobile development
ArticleID=828476
ArticleTitle=Using IBM Worklight to develop cross-platform HTML5 video play hybrid applications
publish-date=08012012