«BY GEORGIOS DIAMANTOPOULOS A THESIS SUBMITTED TO THE UNIVERSITY OF BIRMINGHAM FOR THE DEGREE OF DOCTOR OF PHILOSOPHY DEPARTMENT OF ELECTRONIC, ...»
NOVEL EYE FEATURE EXTRACTION
AND TRACKING FOR NON-VISUAL
A THESIS SUBMITTED TO
THE UNIVERSITY OF BIRMINGHAM
FOR THE DEGREE OF DOCTOR OF PHILOSOPHY
DEPARTMENT OF ELECTRONIC, ELECTRICAL
AND COMPUTER ENGINEERING
COLLEGE OF ENGINEERING AND PHYSICAL SCIE NCES
THE UNIVERSITY OF BIRMINGHAMFEBRUARY 2010 University of Birmingham Research Archive e-theses repository This unpublished thesis/dissertation is copyright of the author and/or third parties.
The intellectual property rights of the author or third parties in respect of this work are as defined by The Copyright Designs and Patents Act 1988 or as modified by any successor legislation.
Any use made of information contained in this thesis/dissertation must be in accordance with that legislation and must be properly acknowledged. Further distribution or reproduction in any format is prohibited without the permission of the copyright holder.
ABSTRACTThe Neuro-Linguistic Programming (NLP) Eye-Accessing Cues (EAC) model suggests that there is a correlation between eye-movements and the internal processing mode that people employ when accessing their subjective experience. Upon careful examination, the experimental methodologies of past research studies were based on assumptions informed by an incomplete or erroneous understanding of the EAC model that could have significantly influenced the experimental results. The reliability of the results can be further impacted by the absence of modern eye-tracking equipment to support the inherently complex task of reliably recording, selecting and rating eye-movements. While a plethora of eye-tracker designs is available to date, none of them has been designed to track non-visual eye-movements (eye-movements that are a result of neuro-physiological events and are not associated with vision), which tend to range outside the normal visual field and thus perform poorly in such cases. Therefore, this thesis introduces a set of novel algorithms for the extraction of relevant eye features (pupil position, iris radius and eye corners) that are combined to calculate the 2D gaze direction and to classify each eye-movement to one of eight classes from the EAC model. The applicability of the eye- tracker is demonstrated through a pilot study that serves as a real-world application case study.
The performance of the eye-tracker is found to be practical for the intended purpose as it is lightweight, low-cost and can robustly perform the tasks of 2D gaze direction estimation and classification.
ACKNOWLEDGEMENTSThe contribution of certain people in our own lives and projects is often so subtle that we never acknowledge it to ourselves and them. I hope that, far in the future, when reading this short section of my thesis, I will have proved myself savvy of these subtleties. Here goes.
Thank you to my mother and my father who made “me” possible, to my supervisors Mike Spann and Sandra Woolley for their constant support, for sharing their wisdom and insights, for being friends as well colleagues and for being exceptionally patient with my changes of interest. Thank you to Paul Tosey, Jane Mathison and Richard Churches from the University of Surrey, who provided valuable guidance in the early stages of the PhD. Thank you to my friend Michael Antoniou for setting a good example for me and being the friend I can count on. Thank you to my friend Agnes Mariakaki for helping me break the laws of my personal physics and change the sinusoidal form of my motivation to peak higher and more often and dip less and less often.
Thank to members of my family who always thought of highly of me and believed in me. Thank you to Prof. Martin Russell who as the head of postgraduate research at the time made my PhD scholarship possible and whoever else might have had something to do with it “behind the scenes”. Thank you to Steve Quigley and Hooshang Ghafouri-Shiraz who were open to the shift in the goals of this PhD and allowed me to move forward. Thank you to James Gormley, David Checkley for their invaluable technical support and to Mary Winkles for her persistence in asking me whether I’m finishing up yet (amongst other things). Thank you to Mark Schulze for voluntarily offering stimulation and feedback in his own time.
Thank you to the naysayers that tried to influence me and my PhD directly or indirectly, to those who told me that I couldn’t, wouldn’t or shouldn’t do it, to those who told me I could do it like this but not like that and of course, to those who directly opposed me – believe me when I say that, without you, this PhD would not have been possible. You were the extra fuel I needed to get “up” when I was “down”. The universe works in mysterious ways.
If you are reading this and you feel you should have been mentioned but you were not it is merely because my mind has forgotten; my heart thanks you all the same.
Chapter 1: Introduction
Chapter 2: An introduction to the NLP EAC model and critical review of past research.................. 14 Neuro-Linguistic Programming and representational systems
The NLP Eye-Accessing Cues model
Past Eye-Accessing Cues model research
Brief overview of objectives and reported results of EAC studies
Validation of the subject’s cognition
Recording, rating and selecting eye-movements
Interpreting and analysing eye-movement data
Eye-movement research relevant to the EAC model
Eye-movements in dyadic interactions
Summary and future directions
Chapter 3: Review of eye-tracking systems
Passive and active illumination
Suitability of remote eye-trackers
Building a head-mounted eye-tracker
Chapter 4: Feature extraction
Eye-feature detection algorithm overview
Pupil contour estimation
Pupil contour refinement
Iris radius calculation
Eye corner detection
Clustering of eye-corner features
Calculation of the 2D gaze vector
Chapter 5: Feature extraction evaluation
Testbed and sample video collection
Evaluation of the pupil detection algorithm
Evaluation of the iris boundary detection algorithm
Evaluation of the corner detection and clustering algorithms
Evaluation of the 2D gaze vector calculation algorithm
Comparison of the REACT eye-tracker with SR Research EYELINK-II
Evaluation of the REACT eye-tracker hardware usability
Chapter 6: Case study
Eye-tracking over time
Case study PILOT experiment
Chapter 7: Conclusions and future work
Appendix A: Eye-tracking hardware
Appendix B: Case study full transcript
Appendix C: Published and submitted papers
This research work is concerned with the development of an eye-tracker that is able to track eyemovements over the maximum range of movement of the eyes and is targeted towards applications that are concerned with non-visual eye-movements. These eye-movements tend to extend beyond the normal field of view when people are looking at visual targets. In this way, it is probably the only eye-tracker that is able to track such extreme eye-movements and maintain similar levels of accuracy. It is also the only eye-tracker of its kind (head-mounted with close-up camera) to not use the glint, which is the reflection of the infrared light on the cornea, as a reference point but instead, detect and use the eye corners as reference points. Another advantage of the eye-tracker has been its ease of use; while a large proportion of other eyetrackers require that the camera is partially or fully calibrated and that each subject provides several calibration points, the REACT eye-tracker operates without camera calibration and required only one calibration point for each subject. Additionally, a low-cost, easy-to-assemble build has been maintained and the eye-tracker can be easily adapted to other applications, both in hardware and software. One of the adaptations that were considered to be important is the possible transition from 2D gaze to 3D gaze. For this particular application, 2D gaze has been sufficient and should it be necessary in the future, because the iris radius is calculated with great precision and if camera calibration data were available, 3D gaze could also be calculated. Last but not least, the REACT eye-tracker makes use of no models of great computational complexity and is thus able to perform fast. This research work was inspired by the Neuro-Linguistic Programming Eye-Accessing Cues model and beyond the major contribution of the REACT eyetracker itself, it further contributes to the academic body of knowledge in relation to NeuroLinguistic Programming with a critical review of past Eye-Accessing Cues model research which is presented in Chapter 2 and published elsewhere in a peer-reviewed publication (see Diamantopoulos et al., 2008).
Movements of the eyes have fascinated academics for decades; both in sleep and waking, academic researchers have questioned their purpose, deconstructed their operation and analysed them both in terms of their physiological and psychological properties.
Perhaps this preoccupation with the eyes may be at first explained by considering that the eyes are an essential part of our experience and the visual channel helps us make a large part of our 7 decisions in everyday tasks. Secondly, even though vision appears smooth to us, it is only made possible by the immensely complex structure and behaviour of the eyes. Most commonly referred to as eye-movements, movements of the eyes have been classified into at least nine different classes (Carpenter, 1988; Wade and Tatler, 2005); with each class of eye-movement serving a different purpose in everyday visual tasks, it easily becomes evident just how complex eyes are and why there is such a wealth of research regarding eyes and their movements.
Of course, studying the eyes requires observation and the simplest of devices for this task is another pair of eyes. In his review of early “eye-movement detectors”, Carpenter (1988), states that “with some practice one can probably detect movements of 1° or so without difficulty” (Yarbus, 1967 cited by Carpenter, 1988) which Carpenter judges to be adequate for preliminary clinical examination but unsuitable for anything but very crude quantitative measurements.
Wade and Tatler (2005) explain that direct viewing is the oldest method but not a particularly fruitful one and this is for two reasons: a) the eyes can move very fast and thus only the initial and final locations are noted and b) the eyes have low temporal resolution and even with intense concentration, it is difficult to determine how the eyes have moved.
Thus, several man-made devices have been introduced from as early as 1901 when Dodge and Cline (1901 cited Carpenter, 1988) implemented a device that allowed permanent records of eye-movements to be made - a very primitive, yet functional, cinematic camera. It was not until the 1950s that video recording as we know it now was used for recording eye-movements as a slightly improved version of direct viewing; computer analysis of the video and the record eyemovements was introduced relatively recently, in the 1980s (Carpenter, 1988). Video recording was only one amongst many other devices and methodologies such as mirrors, photoelectric viewing, devices that use light reflected by the cornea or attachments to the eye, electrooculography, electromagnetic recording, contact lenses and suction devices, etc. (Carpenter, 1988; Yarbus, 1967).
As happens with technology, eye-tracking has made several advancements and has matured over the years and high frequency eye-trackers that can track even the smallest of movements have been introduced. Video-based eye-trackers are now the mainstream choice and several different types of them are available, the two major ones being head-mounted and remote eye-trackers.
8 While eye trackers have thus changed dramatically over the last few decades, eyes are still mainly studied as a functional organism of vision. With the exception of studies in rapid eyemovements during sleep, it was only recently that eyes were studied as a part of the brain and its function (in the waking state). This paradigm shift may have been encouraged but the progressive price drop of eye-trackers which rendered them more affordable to research establishments. Having said that, while there is a reasonable amount of studies relating eyemovements to speech activities such as reading text or maps, very few studies have involved the recording and tracking of eye-movements during the performance of non-visual tasks.