«An Autonomous Vision-Guided Helicopter Omead Amidi August 1996 Department of Electrical and Computer Engineering Carnegie Mellon University ...»
In addition to subpixel accuracy, the fitted parabola provides match uncertainty information. A steep parabola versus a shallower one signals a more accurate match. For instance, the fitted parabola of Figure 2- 13 is shallow, indicating a poor match. The parabola coefficients which determine the shape of the parabola provide valuable information about the match quality. They can be used to construct a covariance matrix describing template match uncertainty in two dimensions. This uncertainty measure is essential for data fusion from multiple template matches or external sensors to improve image displacement estimation.
Before performing potentially dangerous and logistically difficult experiments outdoors, the visual odometer was tested and verified indoors using smaller and easier to manage model helicopters. This section presents the indoor experimental setup and position estimation results of a working visual odometer prototype.
2.5.1 Indoor testbed
A six-degree-of-freedom (6-DOF) testbed was developed for evaluating various position estimation and control systems developed in the span of the work presented in this dissertation. As shown in Figure 2- 14,the 6-DOF testbed (described in Appendix A) supports an electrical model helicopter attached to poles by graphite rods for safety and helicopter ground-truth position estimation. The testbed allows unobtrusive helicopter free flight in a cone shaped area with mechanical stops preventing the helicopter from crashing or flying away.
The testbed helicopter is outfitted with two CCD cameras for image acquisition and three small gyroscopes for attitude measurement. The level area underneath the helicopter is covered with gravel to simulate outdoor rough terrain. At times, various objects of interest are placed on the gravel for object detection experiments. In addition, adjustable lighting installed around the testbed evaluates system performance under changing lighting conditions.
With this experimental setup, experiments with the visual odometer experiments were performed by off-board computing (not shown) before system integration on-board a larger and more capable helicopter for outdoor flight.
2.5.2 Position Estimation Results Figures 2-15 and 2-16 compare ground truth la xal and longi idinal position, and helicoF :r h ight' (dashed lines) measured by the testbed, with vision-based estimates (solid lines), which were collected during flight tests of the 6-DOF testbed helicopter. To accurately compare the positioning, the odometer's position estimates are delayed by the 1/60 second system latency to match the ground truth data. The graph at the bottom of each figure shows the absolute value of the positioning error.
Helicopter maneuvers were performed under computer control (refer to Section 4.4for control system description) during the flight tests to observe the positioning accuracy under abrupt (1-3 Hz, 5- 10 degrees amplitude) attitude oscillations. Helicopter attitude variations during data logging is shown in Figure 2-17. In spite of the constant attitude oscillation and camera vibration, the lateral and longitudinal position estimates are accurate within 1.5 cm with 1/60 second latency. Errors in the longitudinal direction are 50-60% larger due to the lower image resolution2 in this direction which reduces the template match location accuracy.
The errors stem from small position displacements introduced each time a target templates is reinitialized from the image center. The lens imperfections, which were not carefully modeled, are a significant source of these position variations. Templates leaving the image are near the border and
Figure 2- 16 Ground Truth Height vs. Vision-based Estimate largely affected by lens distortion compared to new templates taken from the image center which have little distortion effects. A discrepancy of up to 3 image rows was observed near the image border using a 6 mm wide-angle lens. This pixel error translated to 0.54 cm error in helicopter lateral positioning for the helicopter height (- 1.1 m) in the experiments. The same 3 pixel error produced a maximum of 1.2 cm error in the longitudinal direction each time a template was reinitialized. During hovering or oscillatory maneuvers, templates may leave and enter the image from the opposite image boundaries and, in effect, cancel some of the error over time. The drift rates of 2-3 cm were observed for 1 minute hovering flight tests.
The discrepancy in the odometer's height estimation versus the testbed ground truth was partially caused by the 1-3 cm high stones placed under the helicopter. The testbed estimated height with respect to the flat ground under the stones. This estimate does not match the true helicopter distance from the stones. The height estimate did not drift since no integration was performed. Other sources of error, such as camera alignment and lens distortions, were not modeled and the 1-3 cm accuracy was found sufficient for helicopter control applications.
2.5.3 Velocity Estimation Results Helicopter velocity measurement based on image center pixel velocity is shown below in Figure 2- 18.
The estimated velocity is within 15 c d s of the stand velocity estimate. The velocity discrepancy stems from the stand's damping of helicopter vibration before it is measured by the testbed sensing elements. The vibration has a fundamental 12.5 Hz from the main rotor spinning frequency.
As a measure of consistency, the lateral and longitudinal position estimates presented earlier are differentiated and compared with the optical flow based velocity results. Figure 2- 19 shows the differentiated position (dotted) versus the measured velocity (solid). The two velocity measures agree within a 5 cm/s margin. This suggests that what appears as measurement noise is, in fact, the actual helicopter vibration. This velocity estimate accuracy was deemed sufficient for helicopter control.
2.6 Summary and Discussion This chapter presented a visual odometer for helicopter positioning, one of the major contributions of the work presented in this dissertation. The odometer incrementally maintains helicopter position by sensing image displacements. It senses these displacements by visually locking onto ground objects by image template matching. The odometer eliminates the effects of helicopter rotation from sensed image displacement by measuring changes in the helicopter attitude with each camera image capture.
The disambiguated image displacement is then transformed to determine helicopter motion. The odometer relies on two main assumptions. The first assumes that the helicopter flies over locally flat ground and the second assumes that the objects appearing in the field of view are rich in features.
The locally flat ground assumption simplifies range measurement by local interpolation of one range estimate at the image center using current helicopter attitude. Extending the algorithm’s capability to handle non-flat areas requires matching of the main target template in two cameras. This approach would provide accurate range to the target objects regardless of the ground shape. This extension of the algorithm is not pursued in building the first autonomous vision-guided helicopter prototype. Flight experiments presented in this dissertation were performed over locally flat or gently sloping farm land.
The assumption regarding the availability of image features can be made less restrictive by selecting high texture image regions. The statistical distribution of template matches can be applied to the entire image to determine areas of high contrast suitable for target template selection. This approach requires a ten fold increase in the computational power currently realizable on-board the helicopter and was not pursued.
This chapter introduced the visual odometer by describing its underlying components and provided evidence of its motion estimation ability through indoor flight tests. An off-board implementation of the odometer, integrating on-board angular sensors through a tether, positioned an electrical model helicopter within 1-3 cm accuracy as evaluated by the six degree-of-freedom testbed. The odometer was implemented by a new real-time and low latency vision machine architecture. The vision machine architecture and specific implementation details of the odometer are the focus of Chapter 3.
A real-time vision machine must perform high-bandwidth, and versatile operations on an uninterrupted, overwhelming volume of image data. Observing some of the characteristics of successful and unsuccessful vision machine architectures provides insight into vision machine design. An add-on function box, such as a convolver, is an inexpensive way to obtain video-rate performance for certain types of operations, but it is single functioned and lacks programmability. In a rigid synchronous pipelined system, an image processing pipeline must be synchronized with incoming images. Therefore a minor computational deficiency due to additional requirements in one of the stages causes major latency and throughput penalties.
While a generic computer is completely “programmable”, it is hard to perform even a relatively small window correlation at video rate. A massively parallel SIMD architecture with a grid of processors can perform uniform local image operations, such as a small-kernel convolution or graphical image warping, at lightening speed. It is, however, common experience that as soon as the processing becomes global requiring information from remote pixels, or as the control becomes data dependent, such machines are miserably slow.
What is observed is that an image processing function can be local or global, its operations can be uniform or non-uniform, and its control flow can be data-dependent or data-independent. DependVisual Odometer Machine Specifications ing on its properties and applications, the most suitable architecture varies. Input bandwidth (i.e., data access requirements), processor bandwidth, and output bandwidth must be properly balanced to achieve the highest performance.
The vision machine architecture must stress modularity, expandability, and simplicity in configuring a target machine, rather than blind “generality.” It is not important how “general” a fixed machine is, but how quickly a specific machine can be configured for an application at hand.
This chapter presents a new reconfigurable low latency vision machine architecture developed to implement the helicopter visual odometer. The odometer machine specifications, architecture, and components are presented along with an analysis of the machine’s data flow and synchronization with external sensing.
3.1 Visual Odometer Machine Specifications
The helicopter visual odometer poses a number of difficult demands on an on-board vision system.
The system must process images in real time with minimum latency. It must possess high computational power and data throughput, and must allow close integration of external sensing with vision.
Furthermore, the machine must be compact enough in size, and efficient in power usage to fit onboard a small helicopter.
Processing latency: Stable helicopter flight with vision requires low latency image processing. Helicopters are unstable and require frequent state feedback for stable control compensation. Feedback latency rapidly degrades the stability of this control compensation. Therefore, to be effective, the helicopter visual odometer must be realized by a low latency vision processing system. Indoor testbed experiments using classical PD controllers demonstrated that feedback latency of 1/30 second is adequate for stabilizing small and mid-sized helicopters. This latency figure is employed as the visual odometer machine base latency specification.
50 Chapter 3. A Real-Time and Low Latency Visual Odometer MachineVisual Odometer Machine Specifications Low processing latency also improves positioning accuracy and, in fact, reduces computational complexity of the visual odometer. Low latency increases the chance of finding a template in a smaller search area around the previous successful match, therefore reducing the necessary computational power. Long processing delays require searching larger image areas using more complex algorithms which, in turn, require more computational power.
Computational power and data throughput: A vision machine must balance computational power and data bandwidth for low latency processing since captured images must first travel to the processor’s local memory before the processing can begin.
The visual odometer maintains helicopter position, velocity, and height by tracking mu1tiple templates simultaneously in two filtered camera images. The visual odometer machine must capture images, filter them by convolution, and search for the templates all at video field rate (60 Hz) to keep within the short latency window necessary for stable helicopter flight. High data bandwidth for image transfer and high computational power for image processing must be accurately matched in performance to implement this system. An implementation of the visual odometer tracking algorithm capturing (256 by 256) pixel images, (40 by 40) tracked templates, and (8 by 8) Gaussian image convolution at field rate requires close to 540 million multiplications per second and 8 MBytes per second data throughput.
Close integration with other sensors: The helicopter attitude variations result in large apparent image displacements, which impose difficulty in precise operation of the visual odometer. This effect can be eliminated by measuring these variations with on-board navigational sensors and compensated beforehand. The attitude compensation is effective only if helicopter attitude is measured in perfect synchronization with each camera shutter opening.