The biocybernetic loop is the underlying mechanic behind physiological interactive systems. It describes how physiological information is to be collected from a user, analysed and subsequently translated into a response at the system interface. The most common manifestation of the biocybernetic loop can be seen in traditional biofeedback therapies, whereby the physiological signal is represented as a reflective numeric or graphic (i.e. representation changes in real-time to the signal).
In the 90’s a team at NASA published a paper that introduced a new take on the traditional biocybernetic loop format, that of biocybernetic adaptation, whereby physiological information is used to adapt the system the user is interacting with and not merely reflect it. In this instance the team had implemented a pilot simulator that used measures of EEG to control the auto-pilot status with the intent to regulate pilot attentiveness.
Dr. Alan Pope was the lead author on this paper, and has worked extensively in the field of biocybernetic systems for several decades; outside the academic community he’s probably best known for his work on biofeedback gaming therapies. To our good fortune we met Alan at a workshop we ran last year at CHI (a video of his talk can be found here) and he kindly allowed us the opportunity to delve further into his work with an interview.
So follow us across the threshold if you will and prepare to learn more about the origins of the biocybernetic loop and its use at NASA along with its future in research and industry.
Disclaimer – Any views expressed in the following interview are strictly the individual views of Dr. Alan Pope and should not be construed as the views of NASA.
1. Where did the inspiration for the original biocybernetic loop at NASA come from?
My thinking was no doubt primed for the original biocybernetic loop at NASA by several experiences: undergraduate and graduate education in automatic control systems, my clinical psychology dissertation on biofeedback training, and practice of clinical biofeedback therapy in a NY Veterans hospital and in evening private practice during my early years at NASA. However, the immediate inspiration for the work that led to the 1995 Biological Psychology paper was the work of a biofeedback research pioneer, Thomas Mulholland, on “Biofeedback as Scientific Method”. Tom applied feedback control principles to conceptualize the biofeedback process. Combine this aspect of Tom’s work with descriptions of the “symbionic” cockpit by John Reising at Wright-Patterson USAFB and you have the 1995 paper. It has been an ambition of mine to extend Tom’s ideas further, mapping more concepts from feedback control theory onto the biocybernetic loop.
2. People often use biofeedback loops and biocybernetic adaptation loops interchangeably, do you think they are the same thing?
I suppose that they could be thought of as the same thing, but it may be useful to draw some distinctions between them.
I think of “biofeedback loop” as a general term referring simply to the return of a biological signal to a point in the system from which it originates, thereby closing a loop. There are multiple possible purposes for closing the biological loop. The human body has its own purposes for its myriad internal loops; the synthetic ones that we create by superimposing an external feedback path can be applied to several ends. Biocybernetic adaptation employs the “steering” sense of “cybernetic”, e.g., physiologically adaptive automation, and serves a transitory adaptive purpose that makes use of an assessment of transient state to steer a functional aspect of a system that is external to the physiology from which the state assessment is derived. Of course, biofeedback training also intends to steer, but with a focus on another point in the loop, i.e., to steer physiology to a more or less enduring state or to an improved ability to self-regulate state as a consequence of adapting. But that seems to me to be a stretch of the applicability of the term biocybernetic adaptation. This characterization does, however, point out the reciprocity of effects in a loop configuration. It is problematic to describe the effects of a loop solely in terms of one point in the loop having an effect on another, since the effects are inherently reciprocal, as pointed out in Perceptual Control Theory.
Our LaRC invention technologies merge these processes somewhat by having physiological changes toward a training target cause changes in the functioning of an external system that are intended to be rewarding to the trainee. The information feedback in this biofeedback training configuration is implicit in the functioning of the external system rather than explicit in a physiological signal display.
For clarity in communication, I attempt to make explicit whether I’m referring to “biofeedback” in the sense of signal configuration versus in combination with “training” or “therapy” as a physiological self-regulation training process with enduring effects, both in changed functioning and improved self-regulation skill.
3. You were one of the first to talk about computer games as a form of cybertherapy. Do you still feel the massive potential in this kind of gamification will ever be realised?
“Hope springs eternal ….” (another Al. Pope)
I’ve persevered (some would say “perseverated”) in my belief in the promise of the biofeedback enterprise itself since thinking, in 1973, that the field was about to take off.
An event that we held at the 2012 AAPB conference is our latest attempt to keep this ball rolling: In a Gaming Corner during the conference, a variety of hybrid biofeedback-gaming systems were demonstrated. The Corner was designed to entertain as well as to stimulate thinking about new forms that tomorrow’s biofeedback technologies might take.
I continue to be hopeful that, as I wrote in a 2001 paper entitled “Helping Videogames ‘Rewire Our Minds’”: “Biofeedback-modulated video games represent a new generation of computer and video game environments that train valuable mental skills beyond eye-hand coordination. These psychophysiological training technologies are poised to exploit the revolution in interactive multimedia home entertainment for the personal improvement, not just the diversion, of the user (Business Week, December 23, 1996, p. 68)”. I’m encouraged that the awareness of the beneficent potential of computer games fostered by writers such as Jane McGonigal and conferences such as Games for Health could increase the likelihood of my hope being realized. This eventuality will require an appreciation that the learning that takes place when interacting with games involves psychophysiological conditioning as well as instructional and/or entertainment content delivery.
4. Fine tuning a biocybernetic adaptive mechanic to support generic populations can be a long and drawn out process, recently you’ve moved into supporting multi-user biocybernetic systems. What challenges have you and your team faced in designing systems for this interaction paradigm?
Rather than design for generic populations, one approach that we’re pursuing has been referred to as “individualization.” My co-inventor and I (U.S. patent no. 8062129) proposed a simple method of scores standardized on the individual’s baseline responses to level the playing field in a multiuser competitive application. More recently, Intelligent Automation (IAI) of Rockville, MD, under the NASA SBIR program, is applying machine learning techniques to develop an individualized operator functional state (OFS) assessment model framework. Their reports are in “Selected papers from MODSIM World Conference and Expo” : “Individualized Cognitive Modeling for Closed-Loop Task Mitigation” and “A Systematic Approach for Real-Time Operator Functional State Assessment”.
As alternatives to the optimal way of training an individual OFS model, that is, utilizing all the available data from the individual – which may be scarce, IAI is evaluating two approaches: 1) Train an OFS model for an individual using the data from the individual together with data from some similar subject(s), whose responses are similar to the individual in specific functional states, measured by a similarity metric computed based on the input features; and 2) Derive a generalized model, i.e., an OFS assessment model trained using data from a large number of subjects, then tune the generalized model using the extended data set described above.
IAI is also beginning work on a real-time “hazardous pilot Cognitive State Assessment system, called CSA-Deep,” based upon the work of Goeff Hinton among others. From their abstract: “The key innovation of the proposed research is the modeling and adaptive updating of hazardous cognitive states using a large amount of unlabeled data through semi-supervised deep learning.”
5. A number of your works have seen commercial interest e.g. SMART braingames, with your recent work on integrating biofeedback into the Wii controller and the recent interest by game companies e.g. PS3 bio-controller, do you have any plans to work with game companies on developing new biofeedback mechanics?
Shameless promotion section 🙂
I would very much like to work directly with game companies to develop new biofeedback mechanics based upon my inventions. Licensing, as well as other kinds of partnerships for developing my inventions, is conducted through the NASA Langley commercialization office. The earlier commercialization of US patents 5377100 and 6450820 was accomplished through an office that existed three Center reorganizations ago. NASA has also returned ownership of one of my inventions to me, making it available for licensing in the U.S. from me – US patent 8062129 issued 11/22/11, which is a multiplayer gaming idea mentioned under question 4.
Our patents and patent applications that are available for licensing and other kinds of development partnerships include: –
- http://www.google.com/patents/US5377100.pdf
- http://www.google.com/patents/US6450820.pdf
- http://www.google.com/patents/US6478735.pdf
- http://www.google.com/patents/US8062129.pdf
- http://www.google.com/patents/US20110300523.pdf
- http://www.google.com/patents/
US20060057549.pdf - http://www.nasa.gov/centers/langley/news/researchernews/rn_golf.html
- http://technologygateway.nasa.gov/media/CC/zone/zone.html
- http://www.freepatentsonline.com/20120004034.pdf
- http://www.nasa.gov/topics/technology/features/mindshift.html
- http://technologygateway.nasa.gov/media/CC/biofeedback/biofeedback.html
Our NASA inventions are patented only in the U.S., meaning that products based on the ideas can be developed and marketed outside the U.S. without licensing.
There are also several other biofeedback videogame invention disclosures that are not accessible on the USPTO website, but are available for viewing under a non-disclosure agreement with the LaRC patent office. These documents, together with the already accessible documents, constitute a portfolio of ideas conceived over the past decade and a half that I hope to eventually see in biofeedback videogames.
A key characteristic of my spin-off videogame inventions is the distinction between direct control versus modulation of control. This distinction parallels the distinction made by Steve between intentional BCI and the biocybernetic adaptation paradigm that is based on spontaneous operator functional state. Physiological modulation of manual control differs from the conventional brain-computer interface (BCI) practice of direct control of a task via physiological signals in an important way. A physiologically modulating system programs the human-computer interaction in such a way that physiological signals influence the effect that control of the task by other means, usually manual control, has on performance of the task. This influence may be likened to the amplification of nervousness when performing a task under pressure or the increase in the likelihood of error when becoming bored. Or, on the other hand, augmentation of manual ability when performing a task “in the zone.” I haven’t been successful at communicating this key point to the BCI and videogame industry representatives that I’ve been in touch with thus far, who seem much more concerned with identifying legal issues than appreciating technological distinctions.
6. One issue for researchers is how to evaluate the effectiveness of the biocybernetic loop in order to understand what benefits these systems offer over conventional technology. What are your thoughts on (1) how these evaluations should be designed, i.e. what is an appropriate control condition, and (2) what measures should be used to assess the system, e.g. behaviour of the system, performance of the user.
My view regarding evaluations of the effectiveness of biocybernetic adaptation technology is somewhat divergent and may be limited in relevance to my particular research context.
I was recruited to NASA in 1980 to address a particular problem: control and display researchers felt strongly that their innovative ideas were improvements over current designs, but they didn’t see the proof in improved crew performance. They resorted to subjective measures, a kind of opinion polling that predominates to this day. LaRC researchers had, however, heard of the promise of physiological measures to reveal the elusive differences that they knew had to be there. I was to apply these objective measures to vindicate their belief that there was something better about their designs. This challenge is characterized in “A Program for Assessing Pilot Mental State in Flight Simulators” (AIAA Paper No. 1982-257 – particularly Figure 1 – accessible in the AIAA E-Library, first page). Regrettably, I failed – physiological measures such as heart rate did not consistently discriminate between various flight deck instrumentation design alternatives any better than measures of subjects’ performance.
However, in the process of reviewing reports in the Aviation Safety Reporting System, I identified an emerging issue that appeared to be neglected: the increasing advance of automation onto the flight deck threatened to increase the incidence of what I termed “hazardous states of awareness”, e.g., boredom, complacency, etc. It occurred to me that these states might be indexed, much as sleep stages were, by brain electrical activity. And while not every occurrence of these states results in a safety incident, the states themselves are hazardous precursors to error and are to be avoided during critical phases of operating safety critical systems. The physiologically adaptive closed-loop systems that we fastened upon were designed to titrate automation with manual operation based upon an assessment of the engagement of the operator. The EEG index of engagement that we initially employed as a feedback control variable was intended as a placeholder that was expected to be supplanted by more effective indices in future research. That systematic research is only now being conducted by my NASA colleague, Chad Stephens.
From this hazardous states perspective and in this automation context, the assessment metric that is most revealing of benefits of biocybernetic systems over conventional ones is the comparative incidence of these undesirable states as reflected in cortical measures of engagement and attention, regardless of immediate performance effects (which, as before, are often not discriminating).
That being said, studies conducted at ODU, some of which we collaborated on, in addition to measuring EEG indexes of attention and engagement, measured perceptual and cognitive vigilance performance, several performance measures on the Multi-Attribute Task Battery (MATB), situation awareness and subjective workload. These studies employed various control conditions, including reversed contingency and yoked and random schedule controls who were subjected to stimulus schedules that were not physiologically contingent, as well as manually invoked automation. In some of these studies, individual difference factors such as complacency potential and boredom proneness were taken into consideration. These studies are documented in various publications, including the Human Factors journal, and in other reports accessible only in the NASA NTRS archive by searching on Freeman, Frederick, and his co-authors.
Other research with the biocybernetic paradigm at LaRC and ODU involved assessments of system behavior to evaluate the effectiveness of parameters of contingency. Assessments have involved comparisons with identical biocybernetic loop configurations employing alternative control indices hypothesized to not be effective in ameliorating undesirable states, including breaking the loop. Other studies evaluated the effectiveness of the physiologically contingent adaptation itself by contrasting system behavior between adaptations configured in opposite contingencies.
In related work, other colleagues at LaRC have introduced an innovative flight deck design concept, referred to as “H-mode” for the fact that it models the relationship between the aircraft system and pilot as analogous to that of horse and rider (Schutte, et al “The Naturalistic Flight Deck System: An Integrated System Concept for Improved Single Pilot Operations,” NASA/TM-2007-215090). We’re exploring how this model of interaction might be enhanced by the duplex communication afforded by the biocybernetic adaptation configuration.
Fascinating ruminations about one of my favorite subjects! I just posted it to the thousand biofeedback clinicians in the user group called Biofeedback, on Facebook. Thanks!