Research: Collaboration Between Virtual Console Environments

For many years people have considered the possibility of using virtual displays to replace hardware displays. They are lighter weight, smaller, and provide more flexibility in adding additional screen space. An obvious deficit, however, is that normal collaboration done by looking at another operator’s console cannot happen. You cannot see their display, nor the buttons or menus they are selecting. The purpose of this test is to verify that software tools can be provided which replace or improve the operator’s ability to collaborate.

Results of this study were presented on 2009/08/31.

This is a summary of our methodology. Full details of the final report are unavailable pursuant to distribution restrictions.

My Role

As a primary researcher I co-authored the project test plan, as well as conducted multiple of the testing seasons. Additional seasons were conducted by the fellow researcher. Following the completion of the testing, I collated all data and wrote the final test report.

The Problem

Crew collaboration in older ASWISR platforms required a direct line of sight to another operator’s console. Virtual Console displays and controls are visible only through the operator’s headset, making it difficult or impossible to look at a screen on the console next to you or over someone’s shoulder. Crew collaboration is less of an issue in the current generation of ASWISR platforms (e.g., AWACS 40/45, AEW&C, and P-8A) because they were designed to permit crew interaction to occur without seeing the other operator’s screens. In order to make the virtual console a viable design concept for a team environment in older ASWISR platforms, it is necessary to demonstrate that operators can work together as required.

Two key areas of collaboration were selected for this Virtual Console demonstration: training systems and robust cross-display pointing and highlighting of display content. Training systems assume that the trainer is able to move around and see students interact with the console. While this is a greater problem in older platforms like the AWACS 30/35 which has many buttons, the need to see what a student is doing is still a requirement in current platforms. Pointing and highlighting screen features to support crew collaboration is easy to do if two physical consoles are next to each other, and is not possible if the line-of-sight between the physical consoles is obstructed. Virtual Console software was developed to support pointing and display content highlighting functionality between operators that is independent of the spatial proximity of the consoles. Pointing and highlighting symbology was developed to ensure accurate localization on both displays regardless of the geographic region viewed and the display magnification, Symbology was developed to indicate when an item of interest was not in the recipients field of view.

Our Goals

The goal of this test was to answer two questions:

  1. Could an instructor use the system to train another operator?
  2. Could two experienced operators collaborate to perform operational tasks?

The Process

Two mission scenarios were created for testing, using the events which took place in the “Fighter” scenario; an AWACS 40-45 mission scenario taking place in the southeast of the U.S. Subject Matter Experts (SMEs) from Airborne Warning Systems Training & Support Systems (AWS T&SS) were instrumental to the development of the simulation scenarios. Both scenarios evolved multiple times as they were run through by SMEs and test subjects.

Testing objectives and goals were reviewed with SMEs, who then used the “Fighter” scenario to develop a series of actions which mimicked real-world situations that highlighted interactions between multiple operators. These test scenarios were reviewed multiple times with the test group and SMEs, refining the actions to be taken by the users, in order to refine objectives of each.

“Scenario 1” designed as an instructor/student interaction. This scenario provided the test participant to opportunity to become familiar with the virtual environment while collaborating with the more experienced “instructor” (participating as the second user). “Scenario 2” was designed to be more collaborative. During this scenario a Senior Director (SD) and Weapons Operator (AWO) use the events taking place in the “Fighter” simulation to perform various tasks.

Test sessions began with an overview of the project, familiarizing participants with the goals and vision of the project. Participants were then allowed a small amount of “free-play” time in the virtual environment, to become familiar with the controls concepts, before the scenario testing began.

A pre-test questionnaire was give the participants to gather information concerning their experience with the 40-45 environment. On finishing the testing scenarios a post-test questionnaire was also given, asking the participants to rate their experience and allowing them to provide personal feedback on their experience.

The Results

In all cases the participants were able to collaborate effectively using the system. While the process for pointing and annotating on the other operator’s display was somewhat difficult due to artifacts in the software, it was nonetheless quite effective. The operators were able to quite naturally discuss objects on the screen and execute actions without any confusion.

Significant issues were reported with using the keyboard, which was a direct artifact of the prototype equipment. The currently available hardware had a large plastic enclosure requiring users to look ‘down their nose’ at the keyboard. These issues were expected and should not be a problem when final eyewear was available.