Evaluating Thematic Accuracy

Thematic Accuracy in American Samoa, Guam, and the Commonwealth of the Northern Mariana Islands

An accuracy assessment system was designed and executed to quantify the thematic accuracy of the maps generated at all levels of the classification scheme. Statistical analysis methods have been applied that have been developed by other researchers (Hudson and Ramm 1987, Congalton 1991, Rosenfield et al. 1982). In this work, 20 to 30 field habitat observations were completed per detailed structure as well as detailed biological cover type (see Figure). The accuracy assessment is prepared from a matrix that compares the attribute assigned to a polygon that was generated from the interpretation of the image with that of the determination from field observation.

Five test areas for American Samoa and eleven test areas for the Marianas Archipelago were established to determine the thematic accuracy of the benthic habitat maps prepared from IKONOS satellite imagery. The test areas for American Samoa included Pala Lagoon, Fagatele Bay, Fagaitua Bay, the Manu'a Islands and Tafeu. The test areas for the Marianas Archipelago included Piti Bay and Cocos Lagoon, Guam; Saipan Lagoon and Lau Lau Bay, Saipan; South Beach, Tinian; west and southeast sides of Rota; the west side of Sarigan; and the entire islands of Pagan, Agrihan, and Maug.

Benthic habitat maps from these areas were generated from IKONOS satellite imagery collected at 4 meter resolution and pansharpened to 1 meter resolution. All image interpretation and digitizing was conducted by a single NOAA contractor. The field habitat characterization data collection methods for thematic accuracy assessment differed little from the data collected for ground validation. The primary distinction between the two data sets was the method of selection of the field points. Where as the assessment sites for ground validation were selected to specifically investigate habitat types and gradients of spectral signatures in the imagery, a random stratified sampling method was implemented to select field sites to test map accuracy (Congalton 1991).

Subsequent to completion of the second draft coral reef habitat maps, waypoints were generated using a stratified random sampling scheme. Twenty to thirty accuracy assessment waypoints were collected per test area for each detailed structure and detailed cover class encountered. Waypoint files were generated from these points and all waypoints that could be safely accessed were navigated to using a Trimble Geo Explorer 3 GPS data logger (see Figure). Upon arriving at the waypoint, a weighted meter line was dropped, a buoy fastened and site and habitat specific data collection was undertaken. After deployment of the buoy, 100 GPS positions were collected at one-second intervals and were averaged to generate a single position.

Three benthic habitat assessments were conducted. A point assessment was conducted by surveying the one square meter area around the point where the weight dropped. Two area assessments were conducted in an area of a seven-meter radius around the weight. The first assessment identified the most common habitat type within the area and the second identified the second most common habitat type with in the area. The depth of the site was recorded using a hand held depth sounder. Benthic habitat assessments were made using a glass bottom look box, free diving or observing from the surface. All diving was conducted by breath holding or snorkeling on the surface. In areas where waves and sea conditions were prohibitive to safely accessing the waypoint by boat the GPS was placed in a watertight box and swam to the survey point.

Data including but not limited to site ID, depth, most common habitat, zone and assessment method were recorded using the GPS data logger equipped with a custom data dictionary designed to meet the specifications of the Coral Reef Habitat Classification Scheme. At the end of each field day, the data were downloaded, differentially corrected to the closest CORS station and seamlessly converted to ArcView GIS format. All hand written descriptions were entered in waterproof notebooks and transferred to the GIS by hand. A total of 1,720 benthic habitat characterizations were completed in all sixteen test areas combined, 613 characterization points for American Samoa and 1107 points for the Marianas Archipelago.

To maintain objectivity in the analysis of accuracy, an independent team conducted this work. The Coral Reef Assessment and Monitoring Program (CRAMP) biologists from the Hawaii Institute of Marine Biology from the University of Hawaii at Manoa made the official judgments. The accuracy assessment point theme and the benthic habitat polygon themes were overlaid on the imagery in the GIS. The GIS was queried to select all points within the polygons that matched the polygon habitat type. These were set aside as correct calls. The mismatched pairs were closely examined.

The classification errors that occurred between the MMU and size of accuracy assessment areas were accounted for in this analysis. A map classification was not considered incorrect in a case where a seven meter radius field assessment fell on a habitat feature in the field that was smaller than 1 acre. For example, if a field assessment fell on a small patch reef surrounded by sand that was less than the MMU and thus was not mapped, the point was excluded from the accuracy assessment report. Points that fell close to polygon boundaries were all included as it was assumed that the probability of error contributing to false negatives would be equal to that for false positives. The habitat type for the portions of the test area that were not interpretable due to cloud cover, glint or water quality were classified as "unknown". The accuracy assessment points that fell within polygons with the habitat type of "unknown" were not included in the accuracy analysis.

Results of Overall Accuracy Assessment of Benthic Habitat Map Products

Thematic accuracy of the benthic habitat maps was determined using the aforementioned methods. The mapped habitat type was compared with that of the actual habitat type from field observation. The data is organized into columns representing the field habitat assessment and the rows organized into mapped habitat type. The correct class for each of the incorrect attributes was recorded and included in a comprehensive matrix at the most detailed level of the classification scheme. Four of these detailed matrices were generated, one each for biological cover and geomorphological structure in American Samoa and the Marianas Archipelago. Error matrices were prepared at the detailed and general levels to identify patterns of confusion in the interpretation of the signatures in the imagery. This information was incorporated into ongoing work to improve the accuracy of mapped product. A complete description of these results can be found in the final project report, Project Completion Report: Mapping of Benthic Habitats for U.S. Pacific Territories.

Traditionally, the data is organized into columns that represent the field habitat validation data and the rows are organized into the interpretation of the images. The overall accuracy is typically measured by dividing the total correct determinations by the total number of assessments. This result only incorporates the major diagonal of the table and excludes the omission and commission errors whereas the Tau analysis indirectly incorporates the off-diagonal elements as a product of the row and column marginals. Furthermore, the Tau analysis compensates for unequal probabilities of groups or for differences in numbers of groups (Ma and Redmond 1995). This assessment lends itself to statistical analysis wherein the photointerpreter's determination is assigned a probability that it occurred at random (see Table).

Table 1: Summary of benthic habitat map thematic accuracy.