The Geostationary Operational Environmental Satellite (GOES) series is operated
by the US National Oceanographic and
Atmospheric Administration (NOAA). While in operation since the mid-1970s,
the current series (GOES 8–15) has been operational since 1994. This
document describes the Gridded Satellite (GridSat) data, which provide GOES
data in a modern format. Four steps describe the conversion of original GOES
data to GridSat data: (1) temporal resampling to produce files with evenly
spaced time steps, (2) spatial remapping to produce evenly spaced gridded
data (0.04
The National Oceanographic and
Atmospheric Administration (NOAA) has used the Geostationary Operational Environmental Satellite (GOES)
series since 1975 to monitor weather and other environmental conditions.
While the series name has remained the same, the satellites providing the
data have seen stepwise increases in capabilities. The original system was
a spin-stabilized system. The system received a significant upgrade in 1994
with the launch of GOES 8 (the GOES-I to GOES-M series), providing a five-channel
imager (Menzel and Purdom, 1994). The recent launch (in 2016) of
GOES 16 is another substantial increase in capabilities (Schmit et al., 2017). The satellites have been maintained at
two primary locations: GOES West near 135
Gridded Satellite (GridSat) is produced as a means to facilitate access to GOES data. There are two domains provided: a 15 min domain provided over the contiguous US called GridSat-CONUS and an hourly domain that spans the Western Hemisphere to cover the entire GOES domain, called GridSat-GOES. In the following, we use the term GridSat-GOES to describe both these datasets, since their only difference is the temporal resolution and spatial coverage.
Summary of GOES data access for AREA files.
Comparison of GridSat-GOES and GridSat-CONUS with similar currently available datasets.
Since the launch of the GOES-I to GOES-M series in 1994, worldwide computing
capabilities have significantly improved and changed the landscape of how
the data are transmitted, accessed, and used. When first launched, most GOES
processing was conducted using the Man computer Interactive Data Access
System (McIDAS, Lazzara et al., 1999) and data were
transferred via satellite broadcast or high density tapes. Thus, data were
used primarily for weather research and forecasts. About 25 years later, the
data can be accessed with a variety of computers and software, primarily
transferred over the internet and used for a myriad of purposes. Thus, the
demand has significantly increased while the support of GOES formats has
remained stagnant. Prior to GridSat-GOES, the data were available from the
NCEI archive as follows.
Raw GVAR data – a very complex binary format intended for transmission of
satellite data via satellite uplinks and downlinks. This can be read by a
very limited set of software. The target audience are satellite experts with
access to large computing resources. AREA format – a format developed for McIDAS to process satellite data. This
is well documented, but still requires a thorough knowledge of satellite
data to extend to other applications. The target audience are those with
access to McIDAS software (e.g., academia). netCDF format – the netCDF file provided by the archive does not follow
accepted standards in storing data and it provides no information on
converting values to reflectance and brightness temperatures. There is also no
obvious way to renavigate imagery. Therefore, while the target audience may
be scientific users, these issues make the data provided by the archive less
useful for that audience. TIFF and JPEG – these formats are solely for imaging. The data provided have
map overlays. The target audience are the general public looking for
satellite images for qualitative analysis, not quantitative studies.
What is lacking is a format that can be used by the general scientific
community, such that it can easily be incorporated into other processes and
used for analysis in a wide range of applications.
Table 1 contrasts the access to GOES data via McIDAS area files and the GridSat-GOES formats. The AREA file format is fully supported by one software application, the navigation (outside of McIDAS) requires more than 1000 lines of code, and the calibration information (outside of McIDAS) is not directly available in the AREA files, but requires accessing satellite-specific coefficients from another source. Lastly, the temporal resolution of the files can vary from 1 to 30 min, with a complex scan schedule that varies depending on the scan mode selected for the day (e.g., when severe weather is expected, forecasters can request 1 min rapid scans of a region, at the expense of other regional scans). Conversely, the GridSat-GOES format provides calibrated and mapped data that has been temporally sampled to the nearest 15 min (for CONUS) and 1 h (for GOES). The format – netCDF-4 – is supported by Unidata, which provides information on how to use netCDF in multiple languages and applications.
GridSat-GOES and GridSat-CONUS are most similar to two other datasets: the globally merged IR (infrared) dataset (Janowiak et al., 2001) and the GridSat-B1 Climate Data Record (CDR) (Knapp et al., 2011). Various details of the datasets are compared in Table 2. The global merged data has provided information for precipitation processing since the beginning of global meteorological geostationary satellite coverage in 1998, but only provides information from one channel: the infrared window. The GridSat CDR expands on this by providing intercalibrated IR brightness temperatures with a longer period of record (1980–present). The GridSat-GOES datasets focus on providing access to complete GOES data in a new, updated format. It provides access to all GOES Imager channels. Since it does not expand to other satellites, it is mostly limited to the Western Hemisphere.
The effort to produce GridSat data required processing GOES area files retrieved from NOAA CLASS (Comprehensive Large Array-data Stewardship System). In so doing, we ordered and downloaded 294 TB of data in about 10 million files. This represents the entire archive of GOES-8 to GOES-15 data for 1994–2016. Processing these AREA files required several steps in order to produce the final GridSat products. The first step was to resolve the issue of duplicate files in the archive. This occurs when multiple files are available for one scan time. Next, specific scan times were selected. While the GOES scan schedule is a complex system of various scan schedules taking image scans (i.e., regions) at different times, the GridSat data provide grids at regular times. Image scans are then navigated, which involves determining the location of each pixel and, conversely, determining which image pixel is closest to a given Earth location. Once navigated, the scans are converted from digital counts with a 10 bit depth (hence they range from 0 to 1023) to geophysical units, either reflectance or brightness temperature, a process called calibration (note that visible data are not provided as reflectance factor, which would have a correction for the illumination geometry). Lastly, two other variables which provide information on the spatial uniformity of the images are calculated and stored in the resulting GridSat file. More details on these procedures are provided below.
Files were originally archived and stored on tapes. Each time a tape is
read, there is the possibility that the file has slightly different
contents. When this occurs, both files are retained and are differentiated
by a trailing hyphen and number (e.g., “
Duplicate files are prevalent in the GOES archive. They exist for 1994 through 2003 and can be up to 30 % of the files for any given year. Because some files have more than one duplicate, it is not unusual to find files that are repeated 3 or 4 times. The most repeated file is “goes08.1995.060.174513” (from GOES 8 on 1 March 1995 at 17:45:13 UTC), which has six files for the single scan. The total number of extra files in the archive exceeds 64 000 files; for context, that is more than the number of GOES files archived for 1997. In essence, there is the equivalent of an extra satellite year in the archive.
The reprocessing effort set up an objective algorithm to reconcile these duplicate files. Files were compared scan line by scan line. First, files that were identical were resolved simply by renaming or removing files. Where scan lines differed, the scan lines that were more highly correlated with neighboring scan lines were selected. This led to selecting one of the multiple files as the best; however, some cases resulted in a completely new file (when best scans were selected from different files). This process resolved all duplicated files.
The GOES Imager produces about 120 images per day; the exact number varies by scan schedule. Given the wide variety of scans and times, the GridSat-GOES and GridSat-CONUS sectors store data at nominal 60 and 15 min intervals, respectively. That is, a sector that begins at 17:45 UTC will be included in the 17:45 UTC CONUS grid as well as the 18:00 UTC GOES grid. Thus, users do not need to be aware of the scan schedule, or understand where each is taken, in order to access and process GridSat-GOES data.
Furthermore, the hourly processing allows GridSat-GOES to merge images from various scans. The GOES scan schedule provides a full disk scan once every 3 h. However, the various scans in the intermediate times, when merged, produce something that can approximate full disk coverage. Figure 1 provides a demonstration of this capability. Figure 1a shows an 18:00 UTC image, for which a full disk scan was available. However, in Fig. 1b, the sectors from multiple separate scans (Northern Hemisphere Extended, Southern Hemisphere, etc.) are merged on the same grid, thus providing a nearly full disk image.
The GridSat-GOES dataset provides a variable (called delta_time) that allows the calculation of the actual scan time of each pixel by storing the difference in time between the GridSat image time (17:45 and 18:00 UTC in the previous example) and the time of the actual instrument scan as derived from the scan line information in the AREA files. This allows for calculation of accurate solar illumination angles.
Once a grid time is selected, the image pixels nearest each grid cell are selected through sampling. No averaging of the observations is performed. Sampling maintains the pixel bit depth, radiative characteristics, and spatial resolution. Data were navigated following the algorithms provided for GOES data by NESDIS (1998). Some navigation errors will be present in the remapped data from the original data; navigation accuracy for GOES is about 4 km during daylight and 6 km at night (Ellrod et al., 1998).
Since data are stored on a 0.04
Sample of infrared window GridSat-GOES images for 18:00 and 19:00 UTC.
Infrared channel data are calibrated following documentation from the NOAA STAR web page (Weinreb et al., 1997, 2017). Scan lines are calibrated separately and converted to a brightness temperature. The brightness temperatures are stored in two-byte integers using data packing (e.g., CF convention scale_factor and add_offsets). Brightness temperature noise is about 0.1 K at 300 K (Ellrod et al., 1998), which was improved to approximately 0.05 K at 300 K for imagers on GOES-13 to GOES-15 imagers (Zou et al., 2015).
There is no on-board visible calibration reference for any GOES satellite prior to GOES 16, so some correction is necessary given the decay of the instrument gain (Knapp and Vonder Haar, 2000). Therefore, visible channel data are calibrated in GridSat-GOES following Inamdar and Knapp (2015) to provide a temporally stable set of visible observations.
Lastly, two extra variables are provided: IR and visible channel
variability. For each pixel that was mapped to a grid cell, the spatial
variability is provided. The spatial variability is calculated as the
standard deviation of the
Many possible improvements exist. Future improvements will depend on user
needs. Possible improvements include the following:
Expansion to 1980–1994 – GridSat-GOES is currently limited to the GOES-I to GOES-M
series of satellites (GOES 8 through GOES 15). The dataset can be expanded
back in time to provide information from the previous series (GOES 1–7).
While it would have fewer channels, it could maintain similar temporal and
spatial resolutions. Expansion to GOES 16 and beyond – the newest series of GOES has the
Advanced Baseline Imager (ABI), which increases the number of channels to 16,
increases the spatial resolution to 2 km for most channels, and the
temporal resolution of full disk scans is 15 min (Schmit et al., 2005,
2017). Nonetheless, there will continue to be demand
for this long-term GridSat series at a reduced volume. Expanding
GridSat-GOES to GOES 16 could provide a solution for users needing large
spans of time. Cloud information – including cloud information (probability, temperature,
optical thickness, etc.) is possible and can be provided in future versions.
While increasing the volume of the data, it would increase the usefulness to
many users. Increase spatial resolution – visible data are available at 1 km spatial
resolution and there has been some demand for that. However, it would
significantly increase the volume (thus decreasing the usefulness of the
data). One viable option would be to include the 1 km channel data only for
the CONUS sector.
Merge GOES East and GOES West – data are presently provided separately for GOES East
and GOES West. It is possible to merge the projections to one value, but doing
so would impact data usefulness. In particular, the visible channel would be
affected by including satellites in the same scene that have different
viewing geometries. Climate quality calibration – presently, we are applying the operational
brightness temperature corrections. There are higher quality calibrations
that could be derived. While the visible channel calibration already
includes values that provide longer-term stability, improvements could be made.
This is a short list of changes that could be made to further improve the
utility of the data. User feedback will be used to prioritize these tasks
and resources applied to those that are in highest demand.
The data are available from the following links:
GridSat-CONUS,
List of duplicate file numbers by year. The percentages are relative to the number of scans in the year.
Sample images produced from GridSat-CONUS for 19 August 1999 at 17:45 UTC showing all eight channels. Since this is GOES 8, there is no channel 6 data; central wavelengths are provided in Sect. 4.1. Visible data are enhanced to show detail across the dynamic range.
The following variables are provided in GridSat-GOES and GridSat-CONUS netCDF
files. They are listed with a summary of how they could be used. A sample
set of images is provided in Fig. 2 for GridSat-CONUS.
Lat – the latitude coordinate variable (in Lon – the longitude coordinate variable (in Time – nominal time for the GridSat data (in units of “days since 1 January 1970
00:00:00 UTC”). Lat_bounds – from the CF convention, these provide the
spatial bounds of the grid cells used. Lon_bounds – from the CF convention, these provide the
spatial bounds of the grid cells used. Time_bounds – from the CF convention, these provide the
temporal bounds of the temporal sampling. Satlat – the latitude of the subsatellite point (in Satlon – the longitude of the subsatellite point (in Satrad – distance (in km) of the satellite from the center of the Earth. Filename – filenames for the source archive AREA files included in the
GridSat file, to provide lineage. Delta_time – provides the difference of the actual scan time
(in minutes) from the optimal grid time, used for calculations of solar geometry. Ch1 – reflectance (or scaled radiance) for the 0.6 Ch2 – brightness temperature of the 3.9 Ch3 – brightness temperature of the 6.7 Ch4 – brightness temperature of the 11 Ch5 – brightness temperature of the 12 Ch6 – brightness temperature of the 13 Ch1v – spatial standard deviation of the channel 1 reflectance. This
provides a measure of sub-grid cell variability because the calculation is
made on the original satellite data, then sampled to the grid cells. Useful
in identifying clouds (Rossow and Garder, 1993). Ch4v – spatial standard deviation of the channel 4 (infrared window)
brightness temperatures, which is useful for identifying clouds.
As mentioned in Sect. 3, it could be possible to expand this variable list as
deemed necessary to meet user needs.
Number of files per satellite per year for GridSat-GOES.
Spatial coverages of the
The coverages of GridSat-GOES and GridSat-CONUS are shown in Fig. 3.
GridSat-GOES spans the Western Hemisphere; in fact, it extends into the
Eastern Hemisphere, spanning 150 to 5
Menzel and Purdom (1994) provide information on GOES calibration accuracy based on prelaunch design. Post-launch analysis has deemed IR calibration accurate to within 0.15 to 0.25 K (Ellrod et al., 1998; Wang et al., 2011). The visible calibration was provided by Inamdar and Knapp (2015), where the uncertainty is about 3 % (in scaled reflectance units).
While the GridSat-GOES dataset is suitable for scientific and research use, some caveats should be considered prior to usage. First, users should become familiar with the limitations and quality of the source data: GOES Imager data (Ellrod et al., 1998; Goodman et al., 2018).
The IR calibration is not yet climate quality. We presently use the operational calibration without adjustment. There are, however, some issues that reduce the ability to use GridSat-GOES for climate use. For instance, the midnight black body calibration correction was implemented by NOAA in 2003. Therefore, IR calibration is lower quality near the satellite local midnight prior to 1 April 2003. Also, the GSICS (Global Space-based Inter-Calibration System) community has worked on IR calibration adjustments (Goldberg et al., 2011), which are also not included in this initial version.
Users should also be aware of the limitations of the temporal and spatial
sampling. The full-resolution visible data are sampled to the
0.04
Another limitation of the temporal sampling is the gaps that occur when data are not available. Three primary causes for the gaps are scan schedules, eclipse keep-out zones, and archive gaps. The scan schedule causes gaps mostly in the CONUS grid when the satellite is scanning the full disk (which takes longer than 15 min). For example for GOES East, there is no 18:00 UTC GridSat-CONUS file since it is still doing a full disk scan that started at 17:45 UTC. These gaps generally occur at regular intervals and are usually limited to one missed CONUS grid. The keep-out zone refers to when the satellite is near the local midnight and has the potential to look directly at the Sun. Since this would permanently damage the satellite sensor and optics, a keep-out zone was implemented to reduce the possibility of damage. These are times near local midnight near the March and September equinoxes. The resulting gaps are daily and last a few hours. The GOES-13 to GOES-15 series includes an improved design that allows for shorter keep-out zones. Another limitation is the archive itself. There are some gaps in the archive where scans exist in other archives for one reason or another. These gaps can be longer than those caused by the scan schedule, often exceeding a day.
A complete list of known caveats and issues is maintained on the GridSat-GOES website.
The GridSat-GOES and GridSat-CONUS datasets provide GOES data to a wide audience in a standard format. Since the tasks of calibrating and navigating have been performed during production, data access is greatly simplified for the user. The temporal granularity structure is also much simpler, using fixed grid times rather than the complex and variable heritage scan times. The data are then made available in the widely supported netCDF-4 format. Future improvements and versions will be driven by user requirements.
KRK produced the GridSat data and wrote the manuscript. SLW ordered and managed the GOES data downloaded from NOAA CLASS.
The authors declare that they have no conflict of interest.
We would like to acknowledge Jessica Matthews and Anand Inamdar of CICS-NC for helping make this GOES reprocessing possible. This dataset would not have been produced without the computational support of CICS-NC. This work was supported by NOAA through the Cooperative Institute for Climate and Satellites – North Carolina under cooperative agreement NA14NES432003. Edited by: David Carlson Reviewed by: three anonymous referees