Just How Accurate is Your Drone – Real World Testing Results

Just How Accurate is Your Drone – Real World Testing Results
January 19, 2018 Mike Tully

 

Aerial Services and the Iowa Department of Transportation completed a UAS project in 2017 that tested the positional accuracy of several Unmanned Aerial Systems (camera and lidar). In this article we present a summary of this work. A more detailed report will be forthcoming and hopefully published in other venues for the geospatial community’s use.

Abstract

Remote sensing and mapping research was conducted in Western Iowa in 2017. Three UAS aircraft (fixed wing, multirotor VTOL, and single rotor VTOL) and three different sensors (two cameras and one lidar sensor) were used to map the same tightly controlled borrow pit in western Iowa. In addition, the area was mapped using a manned aircraft and sensors (camera and lidar) as the reference to which other datasets were compared. All UAS were flown over the test area at two different altitudes (200’ and 400’). In addition, three different ground control configurations were tested with each of the three UAS platforms and at each of the two altitudes to determine the optimal number of ground control points needed to achieve a desired positional accuracy. Orthophotography, photogrammetric-derived DEMs, and lidar bare earth point clouds were created from the remotely sensed data from each platform and at both altitudes. Accuracy of the derivative orthos, DEMs, and Lidar point clouds show that several factors contribute to the positional accuracy and quality of the mapping. The accuracy of the unmanned systems from these altitudes can approach the accuracy of much larger, much more expensive manned systems using standard collection techniques and at altitudes commonly used for a mapping project of this type. The data confirm some earlier studies that indicate the accuracy of products derived at low altitude may have lower accuracies than those derived at higher altitude. In addition, operational factors are discussed that impact the positional accuracy of UAS mapping.

Study Area

Figure 1. Project site near Council Bluffs, Iowa showing the area of interest and one of several ground control configurations.

A borrow pit (375 acres) located east of Council Bluffs, IA was selected as the study site (Figure 1). The area is flat with a central drainage and railroad, a small pond and several gravel roads traversing. The remote sensing was accomplished on 25-27 April 2017 using three different UAS contractors. Aerial Services performed all the manned aerial collections and data processing of all the manned and unmanned data. Weather conditions were overcast to partly cloudy with winds ranging from still to windy with gusts exceeding 50 mph. To ensure compliance with FAA UAS regulations, the flight plan was carefully laid out to avoid flying over Interstate 29 on the east, a housing area located adjacent to the northwest corner, and a power plant located south adjacent to the site (not pictured). Ground cover was predominately bare with most other areas covered with early growth grass. There were a few shrubs and trees but no leaves had emerged. All UAS acquisition was performed using at least one pilot and visual observer.

Ground Control Configurations

Ground control was surveyed and targeted throughout the borrow pit (Figure 1). In addition, approximately 75 photo-identifiable check points were surveyed. The accuracy of mapping products from each of the UAS platforms at both altitudes and the Manned systems were tested using three different control configurations: All (25 points), Some (8 points), and None (0 points). In all tests, as much as possible, the same ground control and check points were used to test the accuracy of the different map products although in each control configuration fewer points were included in the aerotriangulation. Occasionally, the designated ground control or check point was not visible in a dataset and a substitute was used for accuracy testing. These exceptions are noted.

Aerial Services established a base station on site for the duration of the flights. In addition, a CORS site (part of the Iowa statewide real-time kinematic GPS network) located less than 2 miles away was used to record redundant base station data.

 

Figure 3. Ground control and check point configurations used to test positional accuracy of the UAS / Manned systems (left: “All”, center: “Some”, right: “None”). Black: targeted ground control (used in AT solution), Green: targeted ground control used as check points (not included in AT solution); White: check points (not included in the AT solution).

Manned and Unmanned Aerial Systems

A fixed wing Navajo aircraft (manned) was used to acquire the airborne imagery (Table 1). The Leica RCD30 medium format digital camera (and integrated GPS/IMU) was used to collect RGB imagery. The Leica RCD30 (50 mm lens) has a frame size of 10,320 x 7,752 pixels with a pixel size of 5.2 microns. The lidar acquisition was accomplished on a second flight using the Riegl ALS70 Lidar sensor. The aircraft was flown at 110 knots, at 1,150’ AGL, and collected at 40 ppsm using 30-degree sidelap.

Three different unmanned systems were operated in these tests by three subcontractors. The Altavian Nova F6500 fixed wing aircraft was used to collect color photography. The system was equipped with the Canon EOS Rebel SL1 camera using the Voigtlander 20mm aspherical prime lens (4.3 microns). The camera has a frame size of 5,184 x 3,456 pixels (17 megapixels) and uses a rolling shutter. The site was flown at 200’ and 400’ AGL. Imagery was collected with a 70% sidelap and 70% endlap.

The DJI Inspire 2 was used to collect a second set of color imagery. This aircraft used the DJI X5s camera (FC6520) with a 15mm lens. The camera has a frame size of 5,280 x 3,956 pixels (20 megapixels) and uses a rolling shutter. The site was flown at 210’ and 400’ AGL. Imagery was collected with a 70% sidelap and 70% endlap.

The Pulse Aerospace Vapor 55 helicopter was used to acquire lidar of the site. This aircraft was equipped with the Riegl VUX lidar sensor which has an integrated GPS/IMU sensor. It was flown at 125’ and 300’ AGL, and collected with a 20-degree sidelap. The point densities of the low and high flights were 170 and 130 ppsm, respectively.

Table 1. Comparison of flight parameters for each of the manned and unmanned aircraft systems.

 

Aircraft Sensor Altitude (ft) GSD (cm) or Point density (ppsm) Lines Images Lifts
Navajo (fixed wing) Leica RCD30 camera 2,500 ft 7.6 cm 5 80 1
Navajo (fixed wing) Leica ALS70 lidar 1,150 ft 40 ppsm 13 n/a 1
Vapor 55 (UAS) Riegl VUX 125 ft 170 ppsm 30 n/a 2
Vapor 55 (UAS) Riegl VUX 300 ft 130 ppsm 11 n/a 3
Nova (UAS) Canon EOS 170 ft 1.4 cm ~73 4772 3
Nova (UAS) Canon EOS 400 ft 2.6 cm 39 1385 1
Inspire 2 (UAS) DJI X5s 210 ft 1.4 cm ~80 2574 7
Inspire 2 (UAS) DJI X5s 400 ft 2.6 cm 47 790 4

Data Processing

Manned Imagery & Lidar. The Leica RCD30 imagery acquired with the manned aircraft was processed using standard photogrammetric workflow using ISAT for aerotriangulation and Inpho suite for orthorectification. The airborne GPS data was post-processed using local base station GPS data. The Leica ALS70 lidar data was calibrated using the Leica workflow and classified to bare earth using a Terrasolid workflow. The orthophotography was rectified using the bare earth DEM.

Manned Lidar DEM. The Manned ALS70 Lidar data was classified to a bare earth DEM. A regularly spaced DEM with 1500 cm spacing was used to orthorectify the RCD30 aerial imagery. For purposes of this research, the imagery and lidar data collected by the manned system were established as the “reference” data sets to which all the UAS datasets were compared.

UAS Imagery. The Altavian Nova’s integrated GPS was post-processed to improve positioning of each frame. All imagery was then run through Pix4D workflow to produce DEMs and orthophotography. The DJI Inspire 2’s integrated GPS data was not post-processed. All the Inspire 2 imagery was run through the same Pix4D workflow to produce DEMs and orthophotography.

UAS DEMs. The Pix4D workflow was used with the Altavian Nova and DJI Inspire 2 imagery to produce “surface” DEMs. The unedited, unclassified, photogrammetrically-derived DEMs for the two UAS camera platforms were used and analyzed. The “high” and “low” altitude imagery was used to create “surface” DEMs with 2.5 cm and 1.3 cm spacing, respectively. These unedited surface DEMs were used to orthorectify the aerial imagery for each of the three ground control configurations and at both altitudes.

The positional accuracy of the DEM and resultant orthophotography could have been improved if these DEMs were classified to bare earth. However, because this is not the “normal” Pix4D workflow the unedited, unclassified DEM was used for orthorectification and accuracy assessments.

Vertical accuracy was derived on the orthoimage using the gridded DEM by querying the elevation at the X and Y coordinates of each control point. This elevation was compared to the GNSS observations, producing an RMSEZ accuracy measure for the elevation at each check point.

To assess the consistency and some quality measures of the photogrammetrically-derived DEMs, the UAS surface DEMs and the bare earth DEM acquired by the Vapor 55 UAS at both altitudes were compared to each other and to the ALS70 reference bare earth DEM. These comparisons were accomplished by creating a “difference DEM” between the reference DEM and UAS DEM and between the high and low altitude DEMs from each UAS sensor.

UAS Lidar. The DEM collected using the Vapor 55 UAS and the Riegl VUX Lidar sensor was classified by the subcontractor to a bare earth DEM using a Terrasolid workflow. Except for the classification of lidar data acquired using the Vapor 55, the processing of all data and all the accuracy assessments were performed by Aerial Services.

Control Configurations. The imagery and lidar data sets from each manned and unmanned aerial platform were processed three times (except where noted below) using three different ground control configurations (Figure 3). This was done to estimate the optimal number of control points needed to achieve a desired positional accuracy. Because ground control is a significant expense and operational effort, we wanted to evaluate if fewer than 20 points could be used to achieve a given accuracy standard, and therefore, could be justified economically for UAS projects that typically have AOIs that are frequently less than one square mile and cannot support this level of survey.

Accuracy assessments were performed on each dataset for each aerial platform flown at the two altitudes (“high” and “low”) in the three control configurations (“all”, “some”, and “none”). Accuracy assessments in easting (X), northing (Y), horizontal (XY), and height (Z) were calculated on the surveyed points that were not used for georeferencing (control points). The control points were identified in the orthoimages and their coordinates were compared to the surveyed GNSS coordinates, resulting in RMSEX, RMSEY, and RMSEXY horizontal accuracy measures.

Results & Discussion

Aerial Acquisition. The acquisition of the manned data occurred about 3-5 days prior to the UAS aerial acquisition (Table 1). Ground conditions were not significantly different on either date. Winds were high on the 25-26 April for much of the UAS acquisition. Ground wind speeds recorded on 25 April at the Offutt AFB (approximately 6 miles) showed winds up to 30 knots. Altavian Nova flights were suspended on 25 April due to winds and concluded on 26 April. Recorded wind conditions on 26 April were generally 13-18 knots for much of the period. The Inspire 2 aerial acquisition began on 26 April and concluded on 27 April when wind conditions were much more mild and did not exceed 10 knots.

The stability of the UAS in these conditions was remarkable. Considering that the Nova and Inspire 2 did not have camera stabilized mounts, the resultant imagery did not appear to have unusually high blurring due to excessive motion during capture.

Aerial acquisition went smoothly for the Altavian Nova fixed wing UAS. The flight crews and equipment performed well. Time on the ground swapping batteries and transferring data was kept to a minimum. Acquisition could have easily been completed on 25 April if the winds had not grounded the UAS. The acquisition was completed the next morning.

Acquisition was problematic with the Inspire 2. The pilot-in-charge (PIC) experienced persistent equipment problems that impacted the quality and efficiency of the acquisition. Although the aircraft performed well, the recharging equipment underperformed. This required more lifts to acquire the AOI than were otherwise planned. Some lifts were rushed and the quality of coverage was impacted.

Associated with this problem was a shortcoming of the Inspire 2 flight management system (FMS). When laying out the next lift, the PIC was unable to see on the FMS the area acquired on the previous lifts. Therefore, the PIC could not be certain that the planned lift had sufficient overlap with previous lifts and ensure complete coverage of the AOI. The result was some small holes in the actual coverage. This had a significant impact on accuracy because some targeted ground control in the low and high flights were not observed in the aerial photography and could not be used to control or test the resultant orthophotography and DEMs.

Digital Elevation Modelling. The vertical accuracy of the DEMs created from the two lidar systems and classified to bare earth should be higher than “surface” DEMs created photogrammetrically from aerial imagery. In fact, the positional accuracy comparison of a “surface” DEM to a bare earth DEM is not a fair fight. But for purposes of this research we wanted to assess the quality and accuracy characteristics of the “common” geospatial products created from these UAS platforms. Because the editing and classification of photogrammetrically-derived DEMs to “bare earth” are costly, these operations are not normally done in many UAS applications.

Figure 4. Statistical results from the comparison of DEMs generated from the UAS with manned Lidar DEM (“ASI”) and between the low and high altitude flights for each sensor.

Difference DEMs are created by comparing the elevation of each DEM point to the elevation of the nearest point in the second DEM and recording the difference. Difference DEMs were generated for each photogrammetrically-derived “surface” DEM and for the UAS Lidar (Vapor 55) “bare earth” DEM. Three-dimensional models were then created to show the “surface” of the differences between two DEMs and indicate the nature, magnitude, and extent of the elevation differences created by the different sensing platforms at different altitudes in each of the control configurations.

Of interest for this study was how the DEMs from a UAS platform at each altitude compared to each other (high vs low).  The “difference surface” visually illustrates the consistency that an unmanned system can model the same geographical area from each altitude (Figure 8). Secondly, the difference DEMs compared to the “reference” bare-earth DEM  (“ASI”) will illustrate how the photogrammetrically-derived surfaces differ from the “reference” Lidar-derived bare-earth surface.

Figure 8 shows the mean and standard deviation when comparing the low and high DEMs created by each UAS platform, and when comparing each of those DEMs to the reference (“ASI”) DEM. As expected the Vapor 55 Riegl lidar DEM showed the least mean difference (13 cm mean High and 17 cm mean Low) with the reference DEM (Figure 5). Both Vapor 55 lidar DEMs were classified to bare-earth. Classified lidar models would model the site more precisely and these results affirm this expectation. In addition, the Vapor 55 lidar models collected at both altitudes showed the least difference (mean = 13 cm) compared to each other than similar comparisons from the other UAS platforms. This result is consistent with expectations due to the nature of lidar and the added classification and editing of the models so they conform to the bare-earth.

The photogrammetrically-derived surface DEMs derived from aerial imagery acquired by the Nova fixed wing and DJI Inspire 2 model the top surface of features on the site. DEMs created by lidar sensors are capable of modelling the actual terrain under features on a site. Therefore, we would expect to see much greater differences when photogrammetrically-derived DEMs are compared to Lidar DEMs for the same geographical area. Nonetheless, comparing these UAS DEMs to the reference lidar DEM is instructive to determine if modelling from each platform is consistent at each altitude when compared to each other and to the reference DEM.

The Nova showed consistently less mean differences in all tests than the Inspire 2 (Figure 8). As discussed above, the aerial acquisition of the Inspire 2 was problematic and undoubtedly had an impact on the quality of DEM data and the potential of this UAS platform. Therefore, any conclusions drawn from the Inspire 2 performance must be tempered knowing that these results do not reflect the true potential of this UAS. Nonetheless, this is an interesting result because of its implications for associated costs of aerial acquisition. A fixed wing UAS can acquire a given site in much less time using fewer lifts than a VTOL. These results indicate that the VTOL UAS provides no added benefit of superior surface modelling. Therefore, for this and similar mapping applications there may be little modelling or cost advantage to using a VTOL UAS.

The mean difference for the comparison of the Nova and reference DEM at high and low altitude was 22 cm and 25 cm, respectively. Likewise, the mean difference for the Inspire 2 from the reference DEM at high and low altitude was 27 cm and 34 cm, respectively. Even the Vapor 55 Lidar bare-earth DEMs showed a similar result (13 cm and 17 cm mean difference at 300’ AGL and 125’ AGL). This result was unexpected. Our experience with traditional sensors and manned aerial platforms is that surface modelling derived from lower altitude should more accurately represent the geographic area being modelled. These results indicate the opposite. Flying higher produced surface models with less mean differences than when flying at low altitude. This result is consistent with the positional accuracy results discussed below. For a discussion about this unexpected result please refer to that section.

Figure 5. Difference DEMs and statistics for the Low altitude versus the High Altitude DEMs on each UAS platform. [Altavian Nova Low vs High (left), DJI Inspire 2 Low vs High (center), Vapor 55 Low vs High (right)]

Visual representations of the difference DEMs are pictured in Figure 5 – Figure 7. Difference DEMs generated for each UAS platform at high and low altitude are pictured in Figure 5. Of note are the large differences between the high and low altitude DEMs created from the Inspire 2 imagery. These results clearly indicate the effect of poor image acquisition of the Inspire 2. There were gaps in the imagery covering the project area. These gaps caused some ground control points to not be observed in the imagery which degraded the quality and accuracy of these surfaces.

Figure 6. Difference DEMs and statistics for the High altitude versus Reference DEMs for each UAS platform. [Altavian Nova High vs Manned (left), DJI Inspire 2 High vs Manned (center), Vapor 55 High vs Manned (right)]

The difference DEMs pictured in Figure 6 and Figure 7 also show how the Inspire 2 acquisition problems affected the quality of the high and low altitude DEMs. In each case, the DEMs show higher mean differences with the reference DEM than DEMs derived from imagery acquired by the other two platforms.

Figure 7. Difference DEMs and statistics for the Low altitude versus Reference DEMs for each UAS platform. [Altavian Nova Low vs Manned (left), DJI Inspire 2 Low vs Manned (center), Vapor 55 Low vs Manned (right)]

Positional Accuracy Results

Positional accuracy assessments were made of the orthophotography, photogrammetrically-derived DEMs, and for the reference orthos and DEMs created from each of the aerial platforms. These results are summarized below. Check back here in our blog later for more detail after additional evaluation is completed.

Figure 8 shows the tested accuracy of the orthophotography created from each of the UAS platforms at each of the two altitudes (high, low) and for each of the control configurations (full-25 points, some-8 points, and none-0 points). Several observations are worth pointing out.

First, not unexpectedly the reference dataset (manned aircraft using a metric camera, Leica RCD30) delivered the most accurate results in each control configuration. Even though the GSD of the reference orthos was approximately 3 times lower resolution than the orthos from the high altitude UAS flights, their accuracy reflects the superior geometry and quality of high-end metric camera systems.

Second, it is very interesting to note that for each control configuration, the low altitude (200’ AGL) orthos showed lower horizontal accuracies than orthos created from the high altitude (400’ AGL) photography. This is counter to experience with traditional manned camera systems. Generally, the lower one flys, all things being equal, positional accuracies will increase. Although this research cannot definitively say why this result occurred, there are a number of possible explanations, and other studies have shown a similar effect (Day, Weaver, & Wilsing, 2016). First, the UAS cameras share similar characteristics. They are non-metric cameras. The lens are generally of plastic and contain more distortion than a metric lens. More lens distortion decreases the performance of a lens. Second, at 200’ AGL four times more photos are acquired to map the project area than the number acquired at 400’ AGL. This larger number of photos increases the chances of systematic errors. Additionally, both UAS cameras use a rolling shutter. These type of shutters have been shown to increase distortions in imagery in which objects are in motion. Higher number of images at lower altitude captured with rolling shutters greatly increases the distortion errors that must be successfully modelled out of the orthos over the project area. Finally, flying at relatively high speed closer the ground at 200’ AGL increases motion blur which, in turn, will decrease accuracy. All of these factors (and possibly others) may contribute to producing this lower accuracy at lower altitude of these UAS systems.

Third, using full control (23-25 points) the accuracy of orthos from both UAS platforms from high altitude (400’) was roughly equivalent. Significantly, this is evidence that using very inexpensive camera systems on UAS like the Nova and Inspire 2 that horizontal accuracies can approach those obtained with high-end camera systems. In practice, on the other hand, it is unlikely a typical UAS project could sustain the surveying costs associated with establishing this higher number (20+) of control points. The size of an area that can be acquired is generally less than 1 square mile when flying under line-of-sight rules.

Fourth, the accuracy results using different control configurations (Figure 8) where maximal positional accuracy can be achieved using less than a dozen control points is supported by at least one other study (Abdullah, 2017). These results contribute to a growing understanding that when using UAS a few well-placed, well-surveyed control points can be used to control the mapping and achieve equivalent positional accuracies approaching projects using 20 or more points. This may represent a significant cost savings for UAS projects that are amenable to a smaller contingent of control points.

Fifth, the aerial acquisition problems experienced with the DJI Inspire 2 had a measurable impact on the positional accuracy of products from that UAS platform especially when fewer than the recommend 20 control points are used in aerotriangulation. In the full control block, the accuracy of the Inspire 2 (VTOL) at both altitudes was approximately equal to those obtained by the Nova (fixed wing). There were areas of the project site not sufficiently covered by imagery at both altitudes, but in the “full control” scenario much of the error was mitigated by substituting other control points for those not observed in the imagery. However, in the “Some Control” scenario the accuracy of the Inspire 2 (132 cm) shows considerable deterioration. This deterioration is most pronounced in the low altitude imagery that showed degradation of positional accuracy by a factor of approximately 8 times.

Finally, the results for “No Control” clearly show a strong deterioration of both the unmanned and manned systems when an image block is uncontrolled (“no control”). In all cases accuracy deteriorated by more than 15x and in the case of the Inspire 2, it fell a factor of over 25x. Results like these affirm the caution UAS service providers should have when delivering mapping products like orthos or DEMs that are processed without surveyed ground control. Even as few as 6 or 8 points significantly strengthen achievable positional accuracy. These results affirm that professionals should avoid delivering any mapping products that are not sufficiently controlled.

Check Back Here for More Information

The reader of this blog is encouraged to check back periodically or subscribe to our newsletter. Additional results from this study will be reported here as our analysis is completed and reviewed by peers.

References

Abdullah, Q. A. (2017, April). Ph.D., PLS, CP. Photogrammetric Engineering & Remote Sensing, 255-260.

Day, D., Weaver, W., & Wilsing, L. (2016). Accuracy of UAS Photogrammetry: A Comparative Evaluation (Vol. December). Photogrammetric Engineering & Remote Sensing.

0 Comments

Leave a reply

Your email address will not be published. Required fields are marked *

*