Project Methods

Evaluating Thematic Accuracy

Thematic Accuracy in the Main Eight Hawaiian Islands

Four field test areas were established to determine the thematic accuracy of the benthic habitat maps prepared from the color aerial photography, hyperspectral airborne and IKONOS satellite imagery. Each area extends from shore to a depth of approximately 30 meters. The first was located on the Kona Coast in the District of South Kohala on the west side of the island of Hawaii. It extends from Kawaihae Harbor to Kiholo Bay. The second study area is located in Kaneohe Bay on the island of Oahu. It extends from the Sam Pan Channel on the south end of the bay to Chinaman's Hat on the north end of the bay. The third area is on the Island of Maui from Maalaea Harbor to Makena Beach and the fourth is illustrated here and is located on the south shore of the Island of Molokai from Palaau to the Kaunakakai Pier (Figure 1).

Figure 1. Distribution of accuracy points at the Molokai test site (N=231)

Benthic habitat maps from these areas were generated from color aerial photography scanned at 1 meter resolution, AURORA hyperspectral imagery collected at 3 meter and IKONOS satellite imagery collected at 4 meter resolution. As these image types vary in resolution and spectral structure and signature, it is important to identify differences in the accuracy of the mapped products prepared from each image source. This also spawns a sound estimation of the thematic accuracy of the final mapped product for the map producers and users.

All image interpretation and digitizing was conducted by a single NOAA contractor. The field habitat characterization data collection methods for thematic accuracy assessment differed little from the data collected for ground validation. The primary distinction between the two data sets was the method of selection of the field points. Where as the assessment sites for ground validation were selected to specifically investigate habitat types and gradients of spectral signatures in the imagery, a random stratified sampling method was implemented to select field sites to test map accuracy (Congalton, 1991). Subsequent to completion of the second draft coral reef habitat maps, polygons representing detailed habitats were aggregated into the three major classes and at least 50 random geographically referenced points were created in each. Waypoint files were generated from these points and all waypoints that could be safely accessed were navigated to using a Trimble Geo Explorer 3 GPS data logger. Upon arriving at the waypoint, a weighted meter line was dropped, a buoy fastened and site and habitat specific data collection was undertaken.

Three benthic habitat assessments were conducted. A point assessment was conducted by surveying the one square meter area around the point where the weight dropped. Two area assessments were conducted in an area of a seven-meter radius around the weight. The first assessment identified the most common habitat type within the area and the second identified the second most common habitat type with in the area. The depth of the site was recorded using a hand held depth sounder. Benthic habitat assessments were made using a glass bottom look box, free diving or observing from the surface. All diving was conducted by breath holding or snorkeling on the surface. In areas where waves and sea conditions were prohibitive to safely accessing the waypoint by boat the GPS was placed in a watertight box and swam to the survey point.

Data including but not limited to site ID, depth, most common habitat, zone and assessment method were recorded using the GPS data logger equipped with a custom data dictionary designed to meet the specifications of the Coral Reef Habitat Classification Scheme. At the end of each field day, the data were downloaded, differentially corrected and seamlessly converted to ArcView GIS format. All hand written descriptions were entered in waterproof notebooks and transferred to the GIS by hand. A total of 1,225 benthic habitat characterizations were completed in all four test areas combined (Table 1).

Table 1. Number of field assessments acquired at each of the four test sites

Test Area	Number of Field Assessments
Kaneohe Bay	393
Kona Coast	304
Maui	297
Molokai	231
Total	1,225

To maintain objectivity in the analysis of accuracy, an independent team conducted this work. The Coral Reef Assessment and Monitoring Program (CRAMP) biologists from the Hawaii Institute of Marine Biology from the University of Hawaii at Manoa made the official judgments. The accuracy assessment point theme and the benthic habitat polygon themes were overlaid on the imagery in the GIS. The GIS was queried to select all points within the polygons that matched the polygon habitat type. These were set aside as correct calls. The mismatched pairs were closely examined.

The classification errors that occurred between the MMU and size of accuracy assessment areas were accounted for in this analysis. A map classification was not considered incorrect in a case where a seven meter radius field assessment fell on a habitat feature in the field that was smaller than 1 acre. For example, if a field assessment fell on a small patch reef surrounded by sand that was less than the MMU and thus was not mapped, the point was excluded from the accuracy assessment report. Points that fell close to polygon boundaries were all included as it was assumed that the probability of error contributing to false negatives would be equal to that for false positives. Furthermore, the three types of imagery were acquired during different days with different weather conditions. The habitat type for the portions of the test area that were not interpretable due to cloud cover, glint or water quality were classified as "unknown". The accuracy assessment points that fell within polygons with the habitat type of "unknown" were not included in the accuracy analysis.

Results of Overall Accuracy Assessment of Benthic Habitat Map Products

Thematic accuracy of the benthic habitat maps was determined using the above methods. The mapped habitat type was compared with that of the actual habitat type from field observation. The data is organized into columns representing the field habitat assessment and the rows organized into mapped habitat type. The correct class for each of the incorrect attributes was recorded and included in a comprehensive matrix at the most detailed level of the classification scheme. Twelve of these detailed matrices were generated, one for each of the types of imagery at each of the four test areas. Error matrices were prepared at the detailed level to identify patterns of confusion in the interpretation of the signatures in the imagery. This information was incorporated into ongoing work to improve the accuracy of mapped product.

The overall accuracy is calculated by dividing the total correct determinations by the total number of assessments. This result only incorporates the major diagonal of the table and excludes the omission and commission errors where as the Kappa analysis (Cohen, 1960) indirectly incorporates the off-diagonal elements as a product of the row and column marginals. The Tau analysis generates a similar statistic as Kappa but compensates for unequal probabilities of groups or for differences in numbers of groups (Ma and Redmond, 1995). This assessment lends itself to statistical analysis wherein the photointerpreter's determination is assigned a probability that it occurred at random. Both Kappa and Tau statistics have been calculated.

The data was then aggregated to the four most general levels of the classification scheme and twelve error matrices were generated at this level, one for each of the image types at each of the test areas (Table 2).

Table 2: Sample error matrix of major classes of classification scheme prepared from visual interpretation of hyperspectral imagery at the Kaneohe Bay test site. Numbers in matrix indicate class coincidence, (U) indicates users accuracy and (P) indicates producers accuracy based on the analysis of 329 field assessments.

		Actual Habitat Type
Mapped Habitat Type		Coral Reef/ Hardbottom	Submerged Vegetation	Unconsolidated Sediment	Other
	Coral Reef/ Hardbottom	136 96%(U) (89%(P)	5	1	1
	Submerged Vegetation	13	66 83%(U) 87%(P)	1	0
	Unconsolidated Sediment	4	5	88 90%(U) 98%(P)	0
	Other	0	0	0	9 100%(U) 90%(P)
		Overall Accuracy: 91%

The most common and most generally accepted single statistic used to represent the accuracy of map products is over all accuracy. The average accuracy for the map products prepared from each of the imagery types was calculated by combining the results from the four test areas. This summary was completed for both detailed and major habitat accuracy assessment (Table 3). Though IKONOS satellite imagery was tested, it was not used in visual interpretation in preparing the final benthic habitat maps for the Main Eight Hawaiian Islands. Thus, the overall accuracy of the final mapped product prepared from the visual interpretation of imagery for the Main Eight Hawaiian Islands it is estimated to be 90% (Kappa and Tau 0.86) for the major class level and 80% at the most detailed level of the classification scheme. Kappa and Tau were not calculated at the detailed level.

Table 3: Accuracy of final map products summarized for each of the three types of imagery. An overall accuracy was estimated to be 90% at major level and 80% at detailed level (Kappa and Tau 0.86) as color aerial photography was predominantly used for map preparation.

Accuracy Statistics
Imagery Type (All test areas Combined)	Overall Accuracy at Detailed Habitat Level	Over all Accuracy at Major Habitat Level	Kappa for Major habitat Level	Tau for Major Habitat Level
Color Aerial Photography	80.8%	90.7%	0.87	0.87
Hyperspectral Imagery	78.1%	89.0%	0.85	0.86
IKONOS Satellite Imagery	74.1%	86.5%	0.82	0.83

Comparison of Thematic Accuracy of Map Products Generated from Color Aerial Photography, Hyperspectral Imagery and IKONOS Satellite Imagery

The Z test or Z score, which reveals the probability that there is no difference between the accuracy of the maps in a contrast, was applied to these data. A contrast result of an absolute value of 1.96 or less indicates a 95% confidence that there is no significant difference between the accuracy of the maps being compared. When all four sites were combined, there was no significant difference between the map accuracy when contrasting color aerial photography and hyperspectral imagery or when contrasting hyperspectral imagery with IKONOS satellite imagery. The contrast between color aerial photography and IKONOS satellite imagery yielded an absolute Z value of 3.07 indicating that there is a significant difference between the accuracy of habitat maps produced from these image sources. There were no significant differences between the accuracy of any of the map products when tested at 90% confidence.

Table 4: Summary of the probability that photointerpretation of coral reef habitat from color aerial photography, hyperspectral and IKONOS Satellite imagery are equivalent: P = 0.05 or less with significant difference highlighted.

Test Area	Image Type	IKONOS	HSI
ALL AREAS COMBINED	COLOR	-3.0709	1.4236
	IKONOS		-1.5961
	Difference Significant P<0.05