How to ungrab Firefox hotkeys from Flash players

Write your own extension to communicate with Perl and cnee to reclaim your Firefox Hotkeys from embedded Flash players

Flash players and other embedded applications in Firefox require their own hooks for keyboard and mouse input. For years, Flash has grabbed Firefox keypresses, which stops people from using the keyboard for navigation, creating new tabs, or even exiting the Flash focus. Learn how to create a Perl program that communicates with a Firefox extension and cnee to restore your keyboard functionality.

Share:

Nathan Harrington, Programmer, IBM

Nathan HarringtonNathan Harrington is a programmer working with Linux at IBM. You can find more information about him at nathanharrington.info.



16 December 2008

Also available in Japanese

Flash players and other embedded applications in Firefox require their own hooks for keyboard and mouse input. Since May 2001, if Flash grabbed your Firefox keypresses, you couldn't use your keyboard for navigation, creating tabs, or even exiting the Flash focus (see Resources for Mozilla bug No. 78414).

This article presents tools and code to allow a Firefox running on Linux® to respond to hotkeys, such as Ctrl+t (open new tab) even when an embedded Flash player has the focus. Use the code here to reclaim your Firefox keyboard application control. This article does not fix the underlying problem, but it provides a work-around to Mozilla bug No. 78414 for Linux users.

By using cnee to monitor system keyboard events, and with Perl to track Firefox application status, Firefox hotkey functionality can be restored even when a Flash player is focused.

Hardware and software requirements

Linux is required, with Firefox V2 or later. The libXnee and cnee components are necessary to monitor systemwide keyboard events, and Perl handles the algorithms. Certain Perl modules are required to process the cnee output and send X Window System events: threads, Thread::Queue, X11:GUITest and Time::HiRes. See Resources for information about these modules and the libXnee software package.

Although implemented on Linux, the general concepts presented here are applicable to multiple operating platforms, such as Microsoft® Windows®. All that is required is a cnee replacement that can print out systemwide keyboard events reliably, as the Firefox extension and Perl code are cross-platform. (If you are a clever developer of Windows applications and create an open source fix similar to what's presented here, please send me an e-mail with your code, and we'll amend this article with proper credit for your work.)

Familiarity with Firefox extension programming will be helpful, as will the installation of the Extension Developer's Extension (see Resources).


Create a Firefox keyboard reporting extension

Determining when a Flash player has grabbed a Firefox hotkey (such as Ctrl+t), is performed in two parts. The first part is recording when the location bar text has changed in Firefox. The location bar text changes each time a tab is opened, a new page is visited, or a different tab is displayed. The second part is monitoring keyboard events across the entire system. If a keyboard combo (such as Ctrl+t) is recognized by cnee, yet the last location bar text change was X seconds ago, the Flash player has grabbed the keyboard focus.

A simple method for determining the location bar change is presented in the Mozilla Developer's Center Progress Listeners page. To implement similar code, download the pre-built extension from the Google Calendar Encryption article (see Resources). Extract the extension directory and change the contents of the install.rdf file, as shown below.

Listing 1. install.rdf
<?xml version="1.0"?>
<RDF:RDF xmlns:em="http://www.mozilla.org/2004/em-rdf#"
         xmlns:NC="http://home.netscape.com/NC-rdf#"
         xmlns:RDF="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
  <RDF:Description RDF:about="urn:mozilla:install-manifest"
                   em:id="flashUngrabber@devWorks_IBM.com"
                   em:name="flashUngrabber"
                   em:version="1.0.0"
                   em:creator="Nathan Harrington"
                   em:description="flashUngrabber">
    <em:targetApplication RDF:resource="rdf:#$9ttCz1"/>
  </RDF:Description>
  <RDF:Description RDF:about="rdf:#$9ttCz1"
                   em:id="{ec8030f7-c20a-464f-9b0e-13a3a9e97384}"
                   em:minVersion="1.5"
                   em:maxVersion="3.0.5" />
</RDF:RDF>

Replace the chrome.manifest file with the contents of Listing 2.

Listing 2. chrome.manifest
content flashUngrabber  chrome/content/

overlay chrome://browser/content/browser.xul  \
  chrome://flashUngrabber/content/overlay.xul

locale  flashUngrabber  en-US   chrome/locale/en-US/

skin    flashUngrabber  classic/1.0     chrome/skin/
style   chrome://global/content/customizeToolbar.xul  \
  chrome://flashUngrabber/skin/overlay.css

Note that the backslash characters (\) are for line continuation only and should not be placed in the file. After replacing the extension metadata as shown above, erase the chrome/content/overlay.js file, and insert the contents shown below.

Listing 3. overlay.js myExt_urlBarListener function
//overlay.js for flash "ungrabber" borrows heavily from
//https://developer.mozilla.org/en/Code_snippets/Progress_Listeners

var myExt_urlBarListener = {
  QueryInterface: function(aIID)
  {
   if (aIID.equals(Components.interfaces.nsIWebProgressListener) ||
       aIID.equals(Components.interfaces.nsISupportsWeakReference) ||
       aIID.equals(Components.interfaces.nsISupports))
     return this;
   throw Components.results.NS_NOINTERFACE;
  },

  // switching through tabs changes the location bar
  onLocationChange: function(aProgress, aRequest, aURI)
  {
    myExtension.updateFile();
  },

};

Copied almost directly from the Progress Listeners example, the myExt_urlBarListener function queries the available interfaces to make sure the onLocationChange capability is available. Each time the location bar changes, the myExtension.updateFile() function will be called. Listing 4 shows the updateFile and init/unint functions, which are added to the bottom of the overlay.js file.

Listing 4. overlay.js myExtension function
var myExtension = {

  init: function() {
    // add the listener on web page loaded
    gBrowser.addProgressListener(myExt_urlBarListener,
        Components.interfaces.nsIWebProgress.NOTIFY_STATE_DOCUMENT);
  },

  uninit: function() {
    // remove the listener when page is unloaded
    gBrowser.removeProgressListener(myExt_urlBarListener);
  },

  updateFile: function() {

    // write the epoch seconds + precision when the location bar was changed
    locTime = new Date().getTime();

    var fileOut = Components.classes["@mozilla.org/file/local;1"]
                        .createInstance(Components.interfaces.nsILocalFile);
    fileOut.initWithPath("/tmp/locationBarChange");
    var foStream = Components.classes["@mozilla.org/network/file-output-stream;1"]
                            .createInstance(Components.interfaces.nsIFileOutputStream);
    foStream.init(fileOut, 0x02 | 0x08 | 0x20, 0666, 0);
    foStream.write(locTime.toString(), locTime.toString().length);
    foStream.close();
  }

};

Implementing simple interprocess communication is performed by writing out the last update time (in seconds since the UNIX® epoch) to the /tmp/locationBarChange file. Add the lines in Listing 5 to ensure the extension is loaded and unloaded correctly.

Listing 5. overlay.js addEventListeners
window.addEventListener("load", function() {myExtension.init()}, false);
window.addEventListener("unload", function() {myExtension.uninit()}, false);

To finish the extension updates, replace the contents of the chrome/content/overlay.xul file with the contents below.

Listing 6. overlay.xul
<?xml version="1.0"?>
<?xml-stylesheet href="chrome://quickgooglecal/skin/overlay.css" type="text/css"?>
<!DOCTYPE overlay SYSTEM "chrome://quickgooglecal/locale/overlay.dtd">
<overlay id="helloworld-overlay"
         xmlns="http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul">
  <script src="overlay.js"/>
</overlay>

The extension is now ready to be loaded into Firefox. A simple method is to create an xpi by changing to the extensions' root directory and issuing the command zip -r flashUngrabber.xpi *. Load the xpi (also available in the Download section) in Firefox and restart the browser.

When the reload is complete, issue the following command perl -e 'while(1){print `cat /tmp/locationBarChange` . "\n";sleep(1)}'. Make the Firefox window viewable, and create and load pages in different tabs. You should see increasing numbers printed once a second if you change tabs, load a different page, or otherwise change the text in the location bar.


flashUngrabber.pl program

Now that it can be determined when Firefox responds to a hotkey press, we can write a program to detect when it should have responded, but did not. The flashUngrabber.pl program, shown in Listing 7, handles this process.

Listing 7. flashUngrabber.pl header
#!/usr/bin/perl -w
# flashUngrabber.pl - monitor keyboard events, send firefox key combos
use strict;  
use X11::GUITest qw( :ALL );                # make sure firefox app has focus
use Time::HiRes  qw( gettimeofday usleep ); # sub second timings
use threads;                                # for asynchronous pipe reads
use Thread::Queue;                          # for asynchronous pipe reads
  
my $padLen = 16;       # epoch significant digits
my $foundControl = 0;  # loop control variable
my $setLocalTime = 0;  # last recorded synthetic event 
my %keys = ();         # key codes and times
  
my ($currWind) = FindWindowLike( 'Mozilla Firefox' );
die "can't find Mozilla Firefox window\n" if ( !$currWind );

The code above includes necessary modules and defines variables. Certain "special" characters, such as "—" (an "m-dash" in typography parlance) present in the Firefox application name may cause the FindWindowLike function to fail. Try loading a different page or switching to a different tab if flashUngrabber.pl can't find your Firefox application ID. Listing 8 continues with the program setup.

Listing 8. flashUngrabber.pl continued
my $cneeCmd  = qq{ cnee --record --keyboard | };
my $pipeCnee = createPipe( $cneeCmd ) or die "cannot create cnee pipe\n";

$keys{ "ctrl-t" }{ cneeCode } = '0,2,0,0,0,28';
$keys{ "ctrl-t" }{ sendKeys } = '^(t)';
$keys{ "ctrl-t" }{ event } = 0;

$keys{ "ctrl-w" }{ cneeCode } = '0,2,0,0,0,25';
$keys{ "ctrl-w" }{ sendKeys } = '^(w)';
$keys{ "ctrl-w" }{ event } = 0;

After creating a connection to the cnee program in keyboard monitor mode, special key codes are defined in the %keys hash. These keys will be searched for later in the program and their last recorded time stored as the value of the "event" hash element. Listing 9 shows the beginning of the main program loop.

Listing 9. flashUngrabber.pl main program loop start
while( 1 )
{
  # read all data off the cnee output queue, process each line.  cnee data 
  # needs to be control first, then the very next line be the key like: ctlr-t
  my $cneeData =  "";
  while( $pipeCnee->pending ){ $cneeData .= $pipeCnee->dequeue or next }

  for my $line ( split "\n", $cneeData )
  {
    if( $foundControl == 1 )
    { 
      $foundControl = 0;
      for my $name( keys %keys )
      { 
        next unless ( $line =~ /$keys{$name}{"cneeCode"}/ );
        $keys{$name}{"event"} = getTimeStr();

      }#for each key

    }#if control pressed

    if( ($line =~ /0,2,0,0,0,37/) || ($line =~ /0,2,0,0,0,109/) )
    {
      # control pressed
      $foundControl = 1;

    }elsif( ($line =~ /0,3,0,0,0,37/) || ($line =~ /0,3,0,0,0,109/) )
    {
      #control released
      $foundControl = 0;

    }#if control pressed

  }#for each line

Depending on system load and a variety of other factors, keyboard events can be processed by Firefox a substantial amount of time before they reach the cnee program. Conversely, it's possible for cnee to print out keyboard events before Firefox has a chance to process the keystrokes. This particular infinite loop and micro-sleep approach is designed to allow cnee and Firefox to have a chance to process events, while retaining adequate UI performance.

At each pass of the main loop, the cnee output (if it exists) is processed to find the control key. If a control key is found, and the next key pressed is specified in the %keys hash, the event time for that key is recorded. Listing 10 shows the continuation of the main processing loop after the cnee events are read.

Listing 10. flashUngrabber.pl main program loop continued
  my $curTime = getTimeStr();

  for my $name ( keys %keys )
  { 
    # require the event to have .5 second to bubble up to cnee
    next unless ( ($curTime - $keys{$name}{"event"} ) > 500000 &&
                  $keys{$name}{"event"} != 0 );

    # reset the event time
    $keys{$name}{"event"} = 0;

    next unless ( $currWind == GetInputFocus() ); # skip if firefox not focused

    next unless( -e "/tmp/locationBarChange" );   # skip if no address bar data

    open( FFOUT, "/tmp/locationBarChange" ) or die "no location bar file";
      my $ffTime = <FFOUT>;
    close(FFOUT);

    # determine if firefox has written a location bar change recently 
    $ffTime = $ffTime . "0" x ( $padLen - length($ffTime) );
    if( $ffTime > $setLocalTime ){ $setLocalTime = $ffTime }

    # if it's been more than two seconds since last event
    next unless( ($curTime - $setLocalTime)  > 2000000 );

Each key code is processed to find if at least a half-second has gone by since the event was detected. If Firefox has the focus currently, the /tmp/locationBarChange file exists, and it has been at least two seconds since the last synthetic event was sent, processing continues as shown below.

Listing 11. flashUngrabber.pl main program loop end
    # record original mouse position
    my($origX,$origY) = GetMousePos();
    my( $x, $y, $width, $height ) = GetWindowPos( $currWind );

    # highly subjective, clicks in google search box on default firefox 
    # installation.  Sleeps are ugly, but help ensure inputs trigger 
    # correctly on a heavily loaded machine
    ClickWindow( $currWind, $width-150, $height-($height-40) );
    usleep(200000);
    SendKeys( $keys{$name}{"sendKeys"} );
    usleep(200000);
    MoveMouseAbs( $origX, $origY );
    usleep(200000);
    $setLocalTime = $curTime;

  }#for each key combo to look for

  usleep(100000); # wait a tenth of a second

}#while main loop

By this point, a synthetic event needs to be sent, so the current mouse position and window position data are recorded. Just sending a Ctrl+t at this point will not create the desired behavior, as the Flash player will simply absorb the keypress. Moving the mouse, clicking the window (in the Google Search box, for example), and sending the Ctrl+t is the most reliable method of ensuring that the keystrokes are processed by Firefox. Moving the mouse back to the original position prior to the keypress makes sure the mouse is put back where you left it.

Heavy system load and a variety of other factors can affect when Firefox receives the mouse move and keyboard events. Reducing or removing the usleep function calls can improve the speed of the keypress event sending, but may cause other issues when the system is responding sluggishly.

Changes to the ClickWindow coordinates may be required if you have a nonstandard Firefox toolbar setup or would like to ensure that the synthetic clicks are sent to a different location in your browser. Listing 12 shows the createPipe and getTimeStr supporting subroutines.

Listing 12. flashUngrabber.pl subroutines
sub createPipe
{ 
  my $cmd = shift;
  my $queue = new Thread::Queue;
  async{
      my $pid = open my $pipe, $cmd or die $!;
      $queue->enqueue( $_ ) while <$pipe>;
      $queue->enqueue( undef );
  }->detach;

  #detach causes the threads to be silently terminated on exit (sometimes)
  return $queue;

}#createPipe

sub getTimeStr
{
  # i suppose the idea of not providing standard length time strings makes
  # sense... somewhere, this is not one of those times
  my   ($seconds, $microseconds) = gettimeofday;
  my $text = "$seconds$microseconds";
  return( $text . "0" x ($padLen - length($text)) );

}#getTimeStr

The createPipe subroutine creates a nonblocking pipe read from the cnee program, and getTimeStr provides a consistent length high-precision time string. Save the above listings as the flashUngrabber.pl program and run the program with the following command: perl flashUngrabber.pl.


Usage

Test your configuration by loading a Flash content player, such as a YouTube video. If you click inside the Flash player — such as on the volume control — and press Ctrl+t, you should see the mouse move to the Google search box, a new tab created, then the mouse return to its previous coordinates.


Conclusion, further examples

With the code and tools presented here, you can restore your favorite Firefox hotkeys from the grasp of Flash. Consider adding cnee key codes to the flashUngrabber.pl program to enable further keyboard navigation, such as Ctrl+tab for next tab, or Ctrl+l for address bar access. Reclaim your PgUp and PgDn keys from the Flash player to scroll the entire page, or add the cnee --record --mouse option to re-enable your scroll wheel.


Download

DescriptionNameSize
Sample codeos-78414-firefox-flash-Ungrabber.0.1.zip17KB

Resources

Learn

Get products and technologies

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Open source on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Open source
ArticleID=358224
ArticleTitle=How to ungrab Firefox hotkeys from Flash players
publish-date=12162008