Enhancing multi-screen user interfaces using Ghosd and Synergy

Improve focus tracking indicators across multiple monitors

Unlike traditional single-screen setups, multi-screen display systems require special consideration for user interfaces (UIs). This article presents tools and code designed to address the acquisition and change of input focus across multiple displays.

Nathan Harrington, Programmer, IBM

Nathan HarringtonNathan Harrington is a programmer working with Linux at IBM. Check out his recently published articles.



09 September 2008

Also available in

Few modifications to your work environment can create a productivity change as dramatic as adding an additional monitor. The open source Synergy package provides an excellent method of linking many displays without purchasing additional hardware.

Unlike traditional single-screen setups, multi-screen display systems require special consideration for UIs. This article presents tools and code designed to address the acquisition and change of input focus across multiple displays. By enhancing existing X Window System focus information using Ghosd displays and the Synergy debug-level output, multi-screen users can know precisely where their input focus is even on displays 4200x3150 pixels and larger.

Requirements

The code and techniques described here are an example of how to create on-screen alerts. This is designed for use on machines running on a private network. It is not for use on machines exposed directly to the Internet, where it is possible that a malevolent program could attempt to execute arbitrary code on the machine.

Hardware

Synergy is designed to work across a wide variety of hardware and software. A fast network is recommended, especially for producing animated or heavily alpha-blended Ghosd visuals.

Software

Although the base functionality of Synergy is supported on a variety of operating systems, this article uses Ghosd on a Linux® server to provide enhanced on-screen displays (OSDs). In Synergy parlance, the server is the computer with the master or shared keyboard and mouse, the primary screen, and operates the Synergy server software. All other computers are Synergy clients and operate the Synergy client software. Here is what you need to follow along:

Synergy
Synergy lets you share a single mouse and keyboard between multiple computers with different operating systems, each with its own display, without special hardware.
X Window System
You'll need to be running Linux® or a compatible X Window System server on your Synergy server and clients to achieve the results described herein. Since most desktop computers running Linux or UNIX® have X Window System installed, it's safe to assume it's already installed on your Linux or UNIX machine.
Pango
Pango is a library for laying out and rendering of text.
GLib library
Pango depends on V2.x series of the GLib library, available from the GTK+ Project.
Cairo
Cairo is a 2-D graphics library with support for multiple output devices.
Ghosd
Ghosd is a library for flashing information to the screen in an attractive manner.
Perl
Perl is a stable cross-platform programming language.

Ubuntu users can install most of the above using the following command: sudo apt-get install libgtk2.0-dev libpango1.0-dev libcairo2-dev perl synergy.


General approach

Consider the six-screen Synergy setup shown in Figure 1. This image shows the in- and out-of-focus designators produced by the code shown below in its final version. If you step away from your multi-screen setup for a few minutes, you may find it difficult to remember where you left the cursor and on which screen. Figure 1 shows a series of green squares on the in-focus screen and faded-out red boxes on the other displays. Read on for a description of how to create this visual, beginning with a focus-tracking designator.

Figure 1. Example visualization six-screen setup
Example visualization six screen setup

Building the in-focus visualization

Consult the file found in the Download section for the Ghosd source-code archive. Download the Ghosd source code on the machine you intend to use as the Synergy server. Unpack the Ghosd code archive with the command bzip2 -d ghosd-0.0.1.tar.bz2 -c | tar -xvf -. Run the usual ./configure; make && make install commands to build the Ghosd source and example files. If your build and installation is successful, you're ready to create the first focus visualization, a series of green boxes indicating focus.

In-focus visualization

Change to the examples/ directory and rename the animation.c file using the command mv animation.c original.animation.c. Create a new file called animation.c and place the code shown in Listing 1 inside.

Listing 1. animation.c header, round_rect function
// animation.c - modified ghosd example for Synergy focus designation
// borrows heavily from the ghosd animation.c example file
#include <stdio.h>
#include <sys/time.h>
#include <sys/poll.h>
#include <time.h>
#include <cairo/cairo.h>
#include <ghosd/ghosd.h>

#define RADIUS 20
int sizeX   = 1024;
int sizeY   = 768;
int mode    = 0;    // 0 is alpha blended red border box, 1 is green boxes

typedef struct {
  cairo_surface_t* foot;
  float alpha;
} RenderData;

static void
round_rect(cairo_t *cr, int x, int y, int w, int h, int r) {
  cairo_move_to(cr, x+r, y);
  cairo_line_to(cr, x+w-r, y); /* top edge */
  cairo_curve_to(cr, x+w, y, x+w, y, x+w, y+r);
  cairo_line_to(cr, x+w, y+h-r); /* right edge */
  cairo_curve_to(cr, x+w, y+h, x+w, y+h, x+w-r, y+h);
  cairo_line_to(cr, x+r, y+h); /* bottom edge */
  cairo_curve_to(cr, x, y+h, x, y+h, x, y+h-r);
  cairo_line_to(cr, x, y+r); /* left edge */
  cairo_curve_to(cr, x, y, x, y, x+r, y);
  cairo_close_path(cr);
}

After the component includes and the variable declaration, the round_rect function is defined as borrowed verbatim from the Ghosd examples/animation.c file. The round_rect function will provide the highlighting in both the in-focus and out-of-focus visualizations. Next, add the renderFocus function shown below.

Listing 2. renderFocus function
static void
renderFocus(Ghosd *ghosd, cairo_t *cr, void* data) {
  RenderData *rdata = data;

  // draw three squares to indicate focus
  cairo_set_line_width( cr, 20);
  cairo_set_source_rgba(cr, 0, 1, 0, rdata->alpha);
  cairo_new_path(cr);
  round_rect(cr, 10, 10, 380, 380, RADIUS);
  cairo_stroke(cr);

  cairo_set_source_rgba(cr, 0, 1, 0, rdata->alpha);
  cairo_new_path(cr);
  round_rect(cr, 40, 40, 320, 320, RADIUS);
  cairo_stroke(cr);

  cairo_set_source_rgba(cr, 0, 1, 0, rdata->alpha);
  cairo_new_path(cr);
  round_rect(cr, 70, 70, 260, 260, RADIUS);
  cairo_stroke(cr);

  cairo_set_source_surface(cr, rdata->foot, 0,0);
}//renderFocus

When a display comes into focus, the renderFocus function will draw three rounded rectangles of consecutively smaller areas on the available rendering surface. Listing 3 presents the main program logic and calls the renderFocus function.

Listing 3. animation.c main logic
int main(int argc, char* argv[]) {
  Ghosd *ghosd;
  RenderData data = {0};
  struct timeval tv_nextupdate;
  const int STEP = 50;
  float dalpha = 0.10;

  if( argc == 4 )
  {
    sizeX = atoi( argv[1] );
    sizeY = atoi( argv[2] );
    mode  = atoi( argv[3] );

  }

  ghosd = ghosd_new();

  if( mode == 0 )
  {
    // create a small canvas and draw the green focus boxes
    data.foot = cairo_image_surface_create (CAIRO_FORMAT_ARGB32, 400,400);
    data.alpha = 1;
    ghosd_set_position(ghosd, (sizeX/2)-200, (sizeY/2)-200, 400, 400);
    ghosd_set_render(ghosd, renderFocus, &data);
    ghosd_render(ghosd);
    ghosd_show(ghosd);
    ghosd_main_iterations(ghosd);

  }//if focus mode

  // ghosd_flash is not ideal for this example, so fade out quickly and exit
  for (;;) 
  {
    gettimeofday(&tv_nextupdate, NULL);
    tv_nextupdate.tv_usec += STEP*1000;

    ghosd_main_until(ghosd, &tv_nextupdate);

    data.alpha -= dalpha;
    if (data.alpha <= 0.3) { return(0); }

    ghosd_render(ghosd);
  }//for each step fade out

}//main

Again borrowing heavily from the Ghosd examples/animation.c file, the main loop creates a 400x400 alpha-blended surface area. This surface is then positioned in the center of the screen no matter what the resolution, and the renderFocus function is designated to be called during each render pass.

Although ghosd_flash provides a useful interface for fading text in and out, a simple fade-out is better for this visualization. The for loop, therefore, reduces the alpha-blending amount in each pass to create an effective fade-out before the program exits.

To build this file, execute the make command, followed by ./animation 1400 1050 0. Assuming your screen resolution is 1400x1050, you should see a series of green boxes in the center of the screen.

Extracting focus information from Synergy output

With the visualization made, it's time to display it on the screen with current focus in a Synergy configuration. Synergy makes this relatively easy, as it provides a wide range of debug level output perfectly suited for this task.

Create a file called processSynergy.pl beginning with the contents of Listing 4.

Listing 4. processSynergy.pl variables, main loop
#!/usr/bin/perl -w
# processSynergy.pl - read Synergys DEBUG2 events
use strict;
my %disp = ();       # record display geometries, focus states
my $delay = 5;       # time in seconds
my $inTimeout = 0;   # timeout mode switcher
my $osdOn     = 0;   # on screen display switcher
my $lastTime = time; # monitor last activity

while( my $line = <STDIN> )
{
  $line =~ s/"//g;

  if( $line =~ /404: received client / )
  {
    # client connection geometry recording an initial setup
    my @parts = split " ", $line;
    my $name = $parts[4];

    $disp{$name}{mode} = 1;
    $disp{$name}{X} = (split "x", $parts[7])[0];
    $disp{$name}{Y} = (split "x", $parts[7])[1];

    print "$name geometry $parts[7]\n";

After declaring variables (most of which are used later in the focus timeout checks), the program enters a loop listening for Synergy events on STDIN. The first if statement handles the client connection messages and adds each display information to the %disp hash. Listing 5 listens for the "focus-switch" messages.

Listing 5. Focus-switch checks
  }elsif( $line =~ / switch from / )
  {
    # track changes of focus between clients
    my $name = (split " ", $line)[6];

    $disp{$name}{mode} = 0;

    # set all other displays to inactive
    map{ $disp{$_}{mode} = 1 if( $_ ne $name )  } keys %disp;

    $lastTime = time;  $inTimeout = 0;

    print "switch to $name\n";

    my $res = "export DISPLAY=$name:0; ";
    $res .= "./animation $disp{$name}{X} $disp{$name}{Y} $disp{$name}{mode} &";
    system($res);

  }#if switch screen catch

}#while line in

As Synergy tracks the focus change between displays, the messages are read by the processSynergy.pl program. After updating the data structure containing information on all the displays (again, used later in the focus-timeout checks), the animation program is called with the resolution values recorded when the client connected.

To create a focus-tracking designator, run the following command on the machine setup as the Synergy server: synergys -f -c configFile --debug DEBUG2 2>&1 | perl processSynergy.pl. You may have to connect to each of your clients and issue a xhost + SynergyServer where "SynergySever" is the name of the machine where the animation program is being run.

Move the mouse around into various windows to see the focus designator track across your display and gradually fade out.


Building a timeout-focus multi-screen designator

Creating a nonfocused visualization

One simple and effective way of indicating nonfocus is to place a red box around an alpha-blended snapshot of the screen. Consult Figure 1 for an example of what this can look like. To create a nonfocused visualization for use with the multi-screen focus designator, add the code shown in Listing 6 at line 57 in animation.c.

Listing 6. renderBox function
static void
renderBox(Ghosd *ghosd, cairo_t *cr, void* data) {
  RenderData *rdata = data;

  // draw a red box around a alpha blended center
  cairo_set_source_rgba(cr, 0, 0, 0, 0.5);
  cairo_new_path(cr);
  round_rect(cr, 0,0, sizeX,sizeY, RADIUS);
  cairo_fill(cr);


  cairo_set_line_width( cr, 10);

  cairo_set_source_rgba(cr, 1, 0, 0, 1.0);
  cairo_new_path(cr);
  round_rect(cr, 0,0, sizeX, sizeY, RADIUS);
  cairo_stroke(cr);

  cairo_set_source_surface(cr, rdata-<foot, 0,0 );

}//renderBox

Once again, heavily influenced by the Ghosd examples/animation.c file, the renderBox function draws a full-screen alpha-blended square, followed by a red border around the image. To correctly call this function, insert the code shown in Listing 7 at line 106.

Listing 7. Red-box logic branch
  }else
  {
    // create a full screen box, and draw the red box with faded center
    data.foot = cairo_image_surface_create (CAIRO_FORMAT_ARGB32, sizeX, sizeY);
    data.alpha = 1;
    ghosd_set_position(ghosd, 0, 0, sizeX, sizeY );
    ghosd_set_render(ghosd, renderBox, &data);
    ghosd_render(ghosd);
    ghosd_show(ghosd);
    ghosd_main_iterations(ghosd);

These particular visualizations are designed to persist until killed by the processSynergy.pl program. At line 117, change the code shown below:

Listing 8. Previous render loop
  for (;;) 
  {
    gettimeofday(&tv_nextupdate, NULL);
    tv_nextupdate.tv_usec += STEP*1000;

    ghosd_main_until(ghosd, &tv_nextupdate);

    data.alpha -= dalpha;
    if (data.alpha <= 0.3) { return(0); }

    ghosd_render(ghosd);
  }//for each step fade out

to:

Listing 9. New render loop
  for (;;) 
  {
    ghosd_main_until(ghosd, &tv_nextupdate);
    ghosd_render(ghosd);
  }//continuous render until program killed

To build this file, execute the make command, followed by: ./animation 1400 1050 1. Assuming your screen resolution is 1400x1050, you should see a faded screen with a red box surrounding it. Press Ctrl+c to exit the visualization.

Modifications to processSynergy.pl for focus-timeout checks

To track which screen has the focus, as well as when input has stopped, make the modifications described below to processSynergy.pl. Begin by removing lines 47-49 and replace them with the code in Listing 10.

Listing 10. Motion, key check logic branch; heartbeat logic branch
  }elsif( $line =~ / MotionNotify / || $line =~ / KeyPress / ||
          $line =~ /event: Buton/ )
  {
    # reset time on keyboard and mouse events
    $lastTime = time;  $inTimeout = 0;

  }elsif( $line =~ / writef\(CALV\)/ )
  {
    # check for timeouts at each hearbeat
    next unless ( (time - $lastTime) > $delay && $inTimeout == 0);

    $inTimeout = 1;
    $osdOn =1;

    for my $key ( keys %disp )
    {
      my $res = "export DISPLAY=$key:0; ";
      $res .= "./animation $disp{$key}{X} $disp{$key}{Y} $disp{$key}{mode} &";
      system($res);
    }#for each display name

    print "timer exceeded\n";

  }#if client

Each new mouse or keyboard event resets the lastTime variable, as processed by the first elsif branch. The second elsif branch looks for the CALV "heartbeat" sent by clients to the server. If a certain number of seconds have expired since the last motion has occurred, the visualizations are displayed on their appropriate screens. Add the code shown in Listing 11 at line 71 to finish the processSynergy.pl modifications.

Listing 11. destroy animation processes on activity
  if( $inTimeout == 0 && $osdOn == 1)
  {
    # destroy on screen display process
    system("ps -aef | grep animation | perl -lnae '`kill -9 \$F[1]`'");
    $osdOn = 0;
    print "terminate osd\n";
  }#if on screen display is visible

Note that this approach is dependent on the CALV heartbeat connection from at least one client. If your network become unavailable or the CALV message is otherwise not present, the focus designators may appear incorrectly.


Usage

To use this completed timeout-focus designator, run the following command on the machine set up as the Synergy server: synergys -f -c configFile --debug DEBUG2 2>&1 | perl processSynergy.pl.

Again, you may have to connect to each of your clients and issue a xhost + SynergyServer if the visualizations do not display. After no keyboard or mouse activity for "delay" seconds (5 seconds, in this case), the currently in-focus screen will have a set of green boxes displayed, while the nonfocused screens will display the alpha-blended red-bordered boxes.


Conclusion, further modifications

Use the Synergy output monitoring code and the Ghosd visuals to track your focus and provide a visual queue to the current focus after a certain time threshold is reached. Now when you return to your displays, you can save time hunting for your cursor through all those screens.

Consider adding animation to your designators for a more eye-catching display (watch out for memory leaks in the available libraries). Link your terminal sessions with the current Synergy focus information to provide display updates for the remote system on the currently viewed screen. If you can think of a visual, Ghosd, and the Synergy debug information can help you display it.


Download

DescriptionNameSize
Sample codeos-ghosd-synergyFocus.zip3KB

Resources

Learn

  • Check out the Ubuntu Guide for using Synergy.
  • You'll need to be running Linux or a compatible X Window System server on your Synergy server and clients to achieve the results described herein.
  • To listen to interesting interviews and discussions for software developers, check out developerWorks podcasts.
  • Stay current with developerWorks' Technical events and webcasts.
  • Check out upcoming conferences, trade shows, webcasts, and other Events around the world that are of interest to IBM open source developers.
  • Visit the developerWorks Open source zone for extensive how-to information, tools, and project updates to help you develop with open source technologies and use them with IBM's products.
  • Watch and learn about IBM and open source technologies and product functions with the no-cost developerWorks On demand demos.

Get products and technologies

  • Download Synergy, which allows you to share a single mouse and keyboard between multiple computers with different operating systems, each with its own display, without special hardware.
  • Download Pango, which is a library for laying out and rendering of text.
  • Download the GLib library, which is available from the GTK+ Project.
  • Download Cairo, which is a 2-D graphics library with support for multiple output devices.
  • Download Ghosd, which is a library for flashing information to the screen in an attractive manner. It is well documented on its home page.
  • Download Perl, which is a stable cross-platform programming language.
  • Innovate your next open source development project with IBM trial software, available for download or on DVD.
  • Download IBM product evaluation versions, and get your hands on application development tools and middleware products from DB2®, Lotus®, Rational®, Tivoli®, and WebSphere®.

Discuss

  • Join the developerWorks community, a professional network and unified set of community tools for connecting, sharing, and collaborating.

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Open source on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Open source
ArticleID=336043
ArticleTitle=Enhancing multi-screen user interfaces using Ghosd and Synergy
publish-date=09092008