Center for Coastal Monitoring and Assessment (CCMA)

Project Methods

Evaluating Thematic Accuracy

Thematic Accuracy in the Main Hawaiian Islands

An accuracy assessment system was designed and executed to quantify the thematic accuracy of the maps generated at all levels of the classification scheme. Statistical analysis methods have been applied that have been developed by other researchers (Hudson and Ramm 1987, Congalton 1991, Rosenfield et al. 1982). In this work, 20 to 30 field habitat observations were completed per detailed structure as well as detailed biological cover type (see Figure). The accuracy assessment is prepared from a matrix that compares the attribute assigned to a polygon that was generated from the interpretation of the image with that of the determination from field observation.

Eight test areas were selected for the main Hawaiian Islands: Oahu ( Kailua , Lanikai, Waimanalo, and Kahala), southern Molokai, Lanai (east Lanai and Manele Bay ), Maui (Oluwalu and Ahihikina'u), and Hawaii (Keahole). Careful consideration was taken to select areas that constituted as comprehensive a representation of the habitats and exposure regimes in the Hawaiian archipelago as possible.

Benthic habitat maps from these areas were generated from IKONOST and QuickbirdT satellite imagery collected at 4 and 2.44 meter resolution and pansharpened to 1 and 0.61 meter resolution, respectively. All image interpretation and digitizing was conducted by a single NOAA contractor. The field habitat characterization data collection methods for thematic accuracy assessment differed little from the data collected for ground validation. The primary distinction between the two data sets was the method of selection of the field points. Where as the assessment sites for ground validation were selected to specifically investigate habitat types and gradients of spectral signatures in the imagery, a random stratified sampling method was implemented to select field sites to test map accuracy (Congalton 1991).

Subsequent to completion of the second draft coral reef habitat maps, waypoints were generated using a stratified random sampling scheme. Twenty to thirty accuracy assessment waypoints were collected per test area for each detailed structure and detailed cover class encountered. Waypoint files were generated from these points and all waypoints that could be safely accessed were navigated to using a Trimble Geo Explorer 3 GPS data logger (see Figure). Upon arriving at the waypoint, an underwater camera was lowered until its weight system touched the bottom, at which point the camera was released, capturing the profile of the bottom before being pulled to the surface..

After deployment of the camera, 100 GPS positions were collected at one-second intervals and were averaged to generate a single position.

Data including but not limited to site ID, depth, most common habitat, zone and assessment method were recorded using the GPS data logger equipped with a custom data dictionary designed to meet the specifications of the Coral Reef Habitat Classification Scheme. At the end of each field day, the data were downloaded, differentially corrected to the closest CORS station and seamlessly converted to ArcGIS format. All hand written descriptions were entered in waterproof notebooks and transferred to the GIS by hand. A total of 671 benthic habitat characterizations were completed in across the eight accuracy assessment areas combined.

To maintain objectivity in the analysis of accuracy, an independent team conducted this work. The Coral Reef Assessment and Monitoring Program (CRAMP) biologists from the Hawaii Institute of Marine Biology from the University of Hawaii at Manoa made the official judgments. The accuracy assessment point theme and the benthic habitat polygon themes were overlaid on the imagery in the GIS. The GIS was queried to select all points within the polygons that matched the polygon habitat type. These were set aside as correct calls. The mismatched pairs were closely examined.

The classification errors that occurred between the MMU and size of accuracy assessment areas were accounted for in this analysis. A map classification was not considered incorrect in a case where a seven meter radius field assessment fell on a habitat feature in the field that was smaller than 1 acre. For example, if a field assessment fell on a small patch reef surrounded by sand that was less than the MMU and thus was not mapped, the point was excluded from the accuracy assessment report. Points that fell close to polygon boundaries were all included as it was assumed that the probability of error contributing to false negatives would be equal to that for false positives. The habitat type for the portions of the test area that were not interpretable due to cloud cover, glint or water quality were classified as "unknown". The accuracy assessment points that fell within polygons with the habitat type of "unknown" were not included in the accuracy analysis.

Results of Overall Accuracy Assessment of Benthic Habitat Map Products

Thematic accuracy of the benthic habitat maps was determined using the aforementioned methods. The mapped habitat type was compared with that of the actual habitat type from field observation. The data is organized into columns representing the field habitat assessment and the rows organized into mapped habitat type. The correct class for each of the incorrect attributes was recorded and included in a comprehensive matrix at the most detailed level of the classification scheme. Four of these detailed matrices were generated, one each for biological cover and geomorphological structure. Error matrices were prepared at the detailed and general levels to identify patterns of confusion in the interpretation of the signatures in the imagery. This information was incorporated into ongoing work to improve the accuracy of mapped product. A complete description of these results can be found in the final project report, Project Completion Report: Mapping of Benthic Habitats for the Main Hawaiian Islands.

Traditionally, the data is organized into columns that represent the field habitat validation data and the rows are organized into the interpretation of the images. The overall accuracy is typically measured by dividing the total correct determinations by the total number of assessments. This result only incorporates the major diagonal of the table and excludes the omission and commission errors whereas the Tau analysis indirectly incorporates the off-diagonal elements as a product of the row and column marginals. Furthermore, the Tau analysis compensates for unequal probabilities of groups or for differences in numbers of groups (Ma and Redmond 1995). This assessment lends itself to statistical analysis wherein the photointerpreter's determination is assigned a probability that it occurred at random (see Table 1).