Project Methods
Evaluating Thematic Accuracy
Thematic Accuracy in the Republic of Palau
An accuracy assessment system was designed and executed to quantify the thematic accuracy of the maps generated at all levels of the classification scheme. Statistical analysis methods have been applied that have been developed by other researchers (Hudson and Ramm 1987, Congalton 1991, Rosenfield et al. 1982). In this work, a minimum of 25 field habitat observations were completed per detailed structure as well as detailed biological cover type (see Figure). The accuracy assessment is prepared from a matrix that compares the attribute assigned to a polygon that was generated from the interpretation of the image with that of the determination from field observation.
Four accuracy assessment test areas for the Republic of Palau were established to determine the thematic accuracy of the benthic habitat maps prepared from IKONOS satellite imagery. The test areas included Ngerumekaol Spawning Area, Ebiil Pass, Toach Ngerdorch Mangroves, Ngemelachel-Ngederrak Seagrass Beds.
Benthic habitat maps from these areas were generated from IKONOS satellite imagery collected at 4 meter resolution and pansharpened to 1 meter resolution. All image interpretation and digitizing was conducted by a single NOAA contractor. The field habitat characterization data collection methods for thematic accuracy assessment differed little from the data collected for ground validation. The primary distinction between the two data sets was the method of selection of the field points. Where as the assessment sites for ground validation were selected to specifically investigate habitat types and gradients of spectral signatures in the imagery, a random stratified sampling method was implemented to select field sites to test map accuracy (Congalton 1991).
Subsequent to completion of the second draft coral reef habitat maps, waypoints were generated using a stratified random sampling scheme. A minimum of 25 accuracy assessment waypoints were occupied over the four test areas for each detailed structure and detailed cover class encountered. Waypoint files were generated from these points and all waypoints that could be safely accessed were navigated to using a Trimble Geo Explorer 3 GPS data logger (see Figure). Upon arriving at the waypoint, a weighted meter line was dropped, a buoy fastened and site and habitat specific data collection was undertaken. After deployment of the buoy, 100 GPS positions were collected at one-second intervals and were averaged to generate a single position.
Three benthic habitat assessments were conducted. A point assessment was conducted by surveying the one square meter area around the point where the weight dropped. Two area assessments were conducted in an area of a seven-meter radius around the weight. The first assessment identified the most common habitat type within the area and the second identified the second most common habitat type with in the area. The depth of the site was recorded using a hand held depth sounder or from a depth sounder on the survey vessel. Benthic habitat assessments were made using a glass bottom look box, free diving, observing from the surface or in areas to deep to use these methods, a tethered video camera was deployed and characterizations were conducted from a LCD monitor. Photographic records were made to create a visual record of the benthic community at each survey location. All diving was conducted by breath holding or snorkeling on the surface. In areas where waves and sea conditions were prohibitive to safely accessing the waypoint by boat the GPS was placed in a watertight box and swam to the survey point.
Data including but not limited to site ID, depth, most common habitat, zone and assessment method were recorded using the GPS data logger equipped with a custom data dictionary designed to meet the specifications of the Coral Reef Habitat Classification Scheme. At the end of each field day, the data were downloaded, differentially corrected to the PALARIS Trimble CORS (Continuously Operating Reference Station) base station (PBLS CORS ARP) located in Koror and seamlessly converted to ArcGIS format. All hand written descriptions were entered in waterproof notebooks and transferred to the GIS by hand. A total of 623 benthic habitat characterizations were completed across the four accuracy assessment test areas combined.
To maintain objectivity in the analysis of accuracy, an independent team conducted this work. The Coral Reef Assessment and Monitoring Program (CRAMP) biologists from the Hawaii Institute of Marine Biology from the University of Hawaii at Manoa made the official judgments. The accuracy assessment point theme and the benthic habitat polygon themes were overlaid on the imagery in the GIS. The GIS was queried to select all points within the polygons that matched the polygon habitat type. These were set aside as correct calls. The mismatched pairs were closely examined.
The classification errors that occurred between the MMU and size of accuracy assessment areas were accounted for in this analysis. A map classification was not considered incorrect in a case where a seven meter radius field assessment fell on a habitat feature in the field that was smaller than 1 acre. For example, if a field assessment fell on a small patch reef surrounded by sand that was less than the MMU and thus was not mapped, the point was excluded from the accuracy assessment report. Points that fell close to polygon boundaries were all included as it was assumed that the probability of error contributing to false negatives would be equal to that for false positives. The habitat type for the portions of the test area that were not interpretable due to cloud cover, glint or water quality were classified as “unknown”. The accuracy assessment points that fell within polygons with the habitat type of "unknown" were not included in the accuracy analysis.
Results of Overall Accuracy Assessment of Benthic Habitat Map Products
Thematic accuracy of the benthic habitat maps was determined using the aforementioned methods. The mapped habitat type was compared with that of the actual habitat type from field observation. The data is organized into columns representing the field habitat assessment and the rows organized into mapped habitat type. The correct class for each of the incorrect attributes was recorded and included in a comprehensive matrix at the most detailed level of the classification scheme. Four of these detailed matrices were generated, one each for biological cover and geomorphological structure. Error matrices were prepared at the detailed and general levels to identify patterns of confusion in the interpretation of the signatures in the imagery. This information was incorporated into ongoing work to improve the accuracy of mapped product. A complete description of these results can be found in the final project report, Project Completion Report: Mapping of Benthic Habitats for the Republic of Palau.
Traditionally, the data is organized into columns that represent the field habitat validation data and the rows are organized into the interpretation of the images. The overall accuracy is typically measured by dividing the total correct determinations by the total number of assessments. This result only incorporates the major diagonal of the table and excludes the omission and commission errors whereas the Tau analysis indirectly incorporates the off-diagonal elements as a product of the row and column marginals. Furthermore, the Tau analysis compensates for unequal probabilities of groups or for differences in numbers of groups (Ma and Redmond 1995). This assessment lends itself to statistical analysis wherein the photointerpreter’s s determination is assigned a probability that it occurred at random (see Table).
| Major Category | Overall Accuracy | Tau |
|---|---|---|
| Major Structure | 97.3% | 0.891 |
| Detailed Structure | 90.0% | 0.891 |
| Major Cover | 88.4% | 0.865 |
| Detailed Cover | 79.9% | 0.788 |