Center for Coastal Monitoring and Assessment (CCMA)

Project Methods

Assessment of Classification Accuracy

Comparing Thematic Accuracy: On-screen vs. Stereoplotter Digitizing

Buck Island National Monument, St.Croix and the surrounding ecosystem (approximately 5000 acres) was selected as the site for comparing thematic accuracy resulting from on-screen vs. stereoplotter digitizing due to several factors. First, almost all habitat types in the Puerto Rico/ Virgin Islands project area are present at this site (except mud and mangroves). In addition, there is a long history of research focused on the habitat in and around Buck Island resulting in a variety of historical data with which to compare NOAA map products. Finally, there is excellent logistic support for field activities through the National Park Service and USGS.

Maps of this area were created using two techniques; the ArcView Extension and on-screen digitizing process described in Product Development, and standard photogrammetric techniques using a stereoplotter to visually interpret benthic features from hard copy photos. Maps derived using the stereoplotter were created by the NOAA Coastal Services Center using Coastal Change Analysis Program (C-CAP) protocols. These protocols include widely accepted and commonly used photogrammetric techniques and instruments (see Dobson et al, 1995 for a complete description of this technique). Under these protocols, habitats were delineated directly from stereo pairs of the hard copy aerial photos that were scanned and used to create the orthorectified mosaics described in Chapter 3 - Product Development. Using the stereo plotter, extremely detailed classification of the hard copy imagery is possible. Since the ArcView digitizing technique used to create maps relied on classification from scans of these photos (less resolution relative to the original hard copy), it was important to determine if there is a difference in thematic accuracy between maps produced using the two techniques given the classification categories and MMU described in Classification Methods.

While map production was underway, habitat type at approximately 120 sites was evaluated in the Buck Island test area to compare with habitat delineations derived from each mapping technique. A stratified sampling protocol was used during which sample sites were pre-selected so that overall thematic accuracy of the three major habitat types across the range of depths and water conditions found in the field could be evaluated.

First, a grid with an approximately 1 acre cell size (MMU) was overlaid on the georeferenced mosaic of the test area.

Next, one third of the grid cells were randomly selected as potential sample sites. The number of potential sample sites was further reduced by eliminating grid cells that contained multiple habitat types. This reduced the possibility of using sites that straddle polygon boundaries. Sites near habitat boundaries were avoided since comparing these locations with mapped polygons could be confounded by spatial accuracy of linework and/or coordinates of ground truth points.

National Ocean Service bathymetry data was then overlaid and used to split the remaining cells into "shallow" or "deep" categories based approximately on the 40-foot isobath to assist with final site selection. This was done to ensure adequate representation of accuracy assessment within two depth strata, since depth is a major factor determining the interpretability of benthic features.

Site selection was completed by using visual photointerpretation to select 20 sites for each of the three major habitat types within the two depth strata respectively. This process resulted in a total of 120 preselected sites across the range of depths and habitat types found at the test area.

The accuracy assessment dataset was collected in November 1999 for the Buck Island test area- eight months after the aerial photos were obtained. This short time interval minimized the possibility that habitats could have been altered significantly between the time of the aerial photography and collecting the accuracy assessment data.

Figure 4.1: Distribution of accuracy assessment points around the
Buck Island Reef National Monument test area (n=109).

A datasheet was created based on the categories in the habitat classification scheme to facilitate assessment of habitat type at each site in the field. Each preselected site was navigated to using real time DGPS. Data recorded at each site included habitat type, depth, and other descriptive information. Depth was determined using a hand-held depth sounder. Habitat type(s) were recorded within an approximately 5-7 meter radius around each pre-selected site. Habitat type directly at the DGPS coordinates was recorded first followed by any secondary habitat types observed within the 5-7m radius of the DGPS point. In most cases, habitat type was the same for the DGPS point and area around each site since we preselected grid cells encompassing areas of uniform tone and texture in the imagery. Logistics prevented evaluation of each site on the scale of the MMU (1 acre). Therefore, potential classification errors resulting from the difference between the MMU and size of accuracy assessment sites were accounted for in the analysis. For example, map classification was not considered incorrect in cases where an accuracy assessment point was scored as "sand" in the 5-7 meter area and the photointerpreter delineated a large, multiple acre polygon as "patchy seagrass", "aggregated patch reefs", and "colonized pavement with sand channels" since each of these classification categories have large areas composed of sand.

Logistics prevented reliable data acquisition at 11 of the 120 pre-selected sites. Therefore 109 sites were used for the accuracy assessment (figure 4.1). Data recorded at each site was overlaid onto the habitat maps and compared against the classification assigned by the photointerpreters. After comparing the map classification to each ground truth site, an error matrix was produced displaying both errors of inclusion and exclusion (Tables 4.1 - 4.2). In addition, overall accuracy, users and producer’s accuracy, and Kappa Statistic (measure of map accuracy relative to a map with classifications randomly assigned expressed as a percent) were reported.

Results: Thematic Accuracy of On-screen vs. Stereoplotter Digitizing

Comparison with the ground truth data revealed very similar levels of thematic accuracy between the two maps. Overall accuracy was 93.6 percent (Kappa 0.90) for on-screen digitizing and 87.8 percent (Kappa 0.82) for maps digitized directly from stereo pairs. Maps produced from on-screen digitizing were almost 100 percent accurate for the submerged vegetation and unconsolidated sediment categories but misclassified a small percentage of hardbottom sites as unconsolidated sediment. Similarly, the maps produced using the stereoplotter were 100 percent accurate at classifying submerged vegetation but misclassified a small percentage of hardbottom and unconsolidated sediment sites. These findings suggest that both of these mapping techniques result in acceptable levels of thematic accuracy for maps produced at this scale with this type of classification scheme.

Methods for Evaluation of Thematic Accuracy for other Reef Morphologies and Water Conditions

The results from the Buck Island test area indicated that thematic accuracy of maps produced from on-screen digitizing was good given the clear water and reef morphologies that are representative of that area. However, both geomorphology and local water conditions can dramatically influence the ability to accurately and consistently photointerpret habitats. Therefore, the thematic accuracy of the ArcView derived products were further evaluated in another area with different water conditions and reef morphologies than those present at the Buck Island site and more representative of the environment found elsewhere in the project area. The La Parguera, Puerto Rico area was chosen for additional evaluation of thematic accuracy. The variety of water conditions and habitat types at this site are representative of those occurring elsewhere in the Puerto Rico project area. In addition, the long history of research focused on the habitat in and around La Parguera by the University of Puerto Rico, Isla Magueyes Campus resulted in a variety of data with which to compare NOAA map products. Furthermore, the University of Puerto Rico provides excellent logistic support for field activities. Sites of accuracy assessment points were selected and analyzed with the same protocol as described above for the Buck Island test area (Table 4.3).

Results: Thematic Accuracy for other Reef Morphologies and Water Conditions

Accuracy in the Parguera area was estimated using 200 ground truth points (Fig. 4.2) and was determined to be 93.6 percent overall (Kappa 0.93). Maps were 100 percent accurate for the unconsolidated sediment category and nearly so for coral reef/hardbottom categories. A small percentage of submerged vegetation sites were misclassified as coral reef/hardbottom. These values are well within acceptable levels of thematic accuracy and suggest that other areas in the project area with similar water conditions and reef morphologies will be mapped with similar accuracy.

Figure 4.2: Distribution of accuracy assessment points around the La Parguera, Puerto Rico test area (n=200).