Supercomputer interface specification overhaul

Authors

Sebastien Lenard

Matt Fisher

Published

November 16, 2023

Our goal is to overhaul the interface between the supercomputer and the webapp to provide a more data-driven interface, for example to allow new regions, variables, sensors, colormaps, etc. to be added without a code change.

We need to to balance all of these concerns:

Webapp load time: startup load time, and load time when changing regions & variables
Webapp maintainability
Flexibility to change webapp behavior by pushing different data from the supercomputer.
Predictability in how the webapp will respond to data changes

Terms

RFC2119

The terms MUST, MUST NOT, SHOULD, SHOULD NOT, MAY, and OPTIONAL are defined by RFC2119.

Components

webapp: Because the webapp has a client and server component, avoid the confusing term “front-end”.
webapp server or webapp back-end: The thing that provides web-friendly data to the front-end.
webapp ingest: The thing that turns supercomputer data into web-friendly data for the webapp server. This component receives data over the interface specified by this document.
supercomputer or supercomputer back-end: Where the original data is produced. This component sends data over the interface specified by this document.

Principles

The webapp should be able to selectively load Sub Region data by selected Super Region
- Rationale: minimizes webapp start-up time and overall download.
Everything has an ID
- Rationale: enables building relationships between Regions, Variables, etc. without sending too much extraneous data.
Relative paths are relative to API root
- Rationale: it’s less stateful to reconstruct a URL this way (as opposed to if the relative path was relative to the file containing it)
Data from the supercomputer MAY be unminified.
- Rationale: it can be minified on ingest.
JSON attributes are camelCase.
- Rationale: consistency is good, it doesn’t matter what we pick.

Regions

Specification

All region files defined here MUST be pushed from the supercomputer each processing cycle.
Regions and Region Collections MUST NOT be identified by any of the following reserved strings:
- undefined
- regions
- collections
- metadata

Regions

Any region that can be selected for viewing on the map or plot. Includes Super Regions and Sub Regions.

Super Regions

These are top-level regions, e.g. “Western U.S.”, “Canada”. The webapp needs to be able to quickly load the list of super-regions so it can display a selector to the user. Depending on which is selected, different sub-regions may display.

Super Regions MUST be defined in regions/root.json.
Super Regions MAY be omitted from regions/root.json if no variable are visible for the super region.
- TODO: Also offer a config option enabled: boolean? E.g. It is RECOMMENDED to instead set "enabled": false.
Super Regions with no Sub Regions/Collections MAY be included in regions/root.json.
Super Regions MUST have the following fields:
- TODO

Example regions/root.json

{
  "26000": {
    "longName": "Western United States",
    "shortName": "USwest",
    "crs": "EPSG:3857",
    "subRegionsRelativePath": "./regions/26000.json",
    "subRegionsHierarchyRelativePath": "./regions/26000_hierarchy.json",
    "shapeRelativePath": "./regions/shapes/26/26000.geojson"
  },
  "26002": {
    "longName": "High Mountain Asia",
    "shortName": "ASHimalayas",
    "crs": "EPSG:3857",
    "subRegionsRelativePath": "./regions/26002.json",
    "subRegionsHierarchyRelativePath": "./regions/26002_hierarchy.json",
    "shapeRelativePath": "./regions/shapes/26/26002.geojson"
  },
  "26004": {
    "longName": "Andes",
    "shortName": "AMAndes",
    "crs": "EPSG:3857",
    "subRegionsRelativePath": "./regions/26004.json",
    "subRegionsHierarchyRelativePath": "./regions/26004_hierarchy.json",
    "shapeRelativePath": "./regions/shapes/26/26004.geojson"
  },
  "26100": {
    "longName": "Alaska",
    "shortName": "USAlaskaWAleutian",
    "crs": "EPSG:3857",
    "subRegionsRelativePath": "./regions/26100.json",
    "subRegionsHierarchyRelativePath": "./regions/26100_hierarchy.json",
    "shapeRelativePath": "./regions/shapes/26/26100.geojson"
  },
  "26101": {
    "longName": "Canada",
    "shortName": "AMCanadaW",
    "crs": "EPSG:3857",
    "subRegionsRelativePath": "./regions/26101.json",
    "subRegionsHierarchyRelativePath": "./regions/26101_hierarchy.json",
    "shapeRelativePath": "./regions/shapes/26/26101.geojson"
  }
}

Sub Regions

Any region that is not a Super Region. Can be anywhere in the hierarchy. Must be a member of a Region Sollection.

Sub Regions MUST be defined according to the Super Region they are a member of in regions/{superRegionId}.json, e.g. regions/26000.json.
Sub Regions MAY contain one or more Sub Region Collections.

Example regions/26000.json

{
  "11725": {
    "longName": "California",
    "shortName": "US-CA",
    "subRegionCollectionId": "adm1",
    "shapeRelativePath": "./regions/shapes/11/11725.geojson"
  },
  "11726": {
    "longName": "Colorado",
    "shortName": "US-CO",
    "subRegionCollectionId": "adm1",
    "shapeRelativePath": "./regions/shapes/11/11726.geojson"
  },
  "12516": {
    "longName": "Pacific Northwest",
    "shortName": "HUC17",
    "subRegionCollectionId": "huc2",
    "shapeRelativePath": "./regions/shapes/12/12516.geojson"
  },
  "12517": {
    "longName": "California",
    "shortName": "HUC18",
    "subRegionCollectionId": "huc2",
    "shapeRelativePath": "./regions/shapes/12/12517.geojson"
  },
  "12807": {
    "longName": "Lower Columbia",
    "shortName": "HUC1708",
    "subRegionCollectionId": "huc4",
    "shapeRelativePath": "./regions/shapes/12/12807.geojson"
  },
  "12809": {
    "longName": "Oregon-Washington Coastal",
    "shortName": "HUC1710",
    "subRegionCollectionId": "huc4",
    "shapeRelativePath": "./regions/shapes/12/12809.geojson"
  },
  "12815": {
    "longName": "San Joaquin",
    "shortName": "HUC1804",
    "subRegionCollectionId": "huc4",
    "shapeRelativePath": "./regions/shapes/12/12815.geojson"
  }
}

Sub Region Collections

A collection containing Sub Regions.

Collections MUST be defined in a file regions/collections.json.
A collection MUST contain one or many Sub Region members.

Example regions/collections.json

{
  "adm0part": {
    "shortName": "Country area",
    "longName": "Country area"
  },
  "adm0": {
    "shortName": "Country",
    "longName": "Country"
  },
  "adm1": {
    "shortName": "State",
    "longName": "State / Country subdivision level 1"
  },
  "adm1part": {
    "shortName": "State area",
    "longName": "State area"
  },
  "huc2": {
    "shortName": "HUC2",
    "longName": "U.S. Hydrologic Unit Code level 2"
  },
  "huc4": {
    "shortName": "HUC4",
    "longName": "U.S. Hydrologic Unit Code level 4"
  },
  "huc6": {
    "shortName": "HUC6",
    "longName": "U.S. Hydrologic Unit Code level 6"
  },
  "hydro3": {
    "shortName": "Hydroshed3",
    "longName": "Hydrosheds level 3"
  },
  "hydro4": {
    "shortName": "Hydroshed4",
    "longName": "Hydrosheds level 4"
  },
  "hydro5": {
    "shortName": "Hydroshed5",
    "longName": "Hydrosheds level 5"
  }
}

Sub Region Hierarchy

An expression of relationships between Sub Regions. E.g. a HUC2 Sub Region may contain a HUC4 Sub Region Collection containing multiple HUC4 Sub Regions, each of which may contain a HUC 6 SubRegion Collection. The hierarchy can be arbitrarily deep.

Relationships between regions MUST be defined according to the Super Region they are a member of in regions/{superRegionId}_hierarchy.json, e.g. regions/26000_hierarchy.json.

Example regions/26000_hierarchy.json

{
  "collections": {
    "adm1": {
      "regions": {
        "11725": {},
        "11726": {}
      }
    },
    "huc2": {
      "regions": {
        "12516": {
          "collections": {
            "huc4": {
              "regions": {
                "12807": {},
                "12809": {}
              }
            }
          }
        },
        "12517": {
          "collections": {
            "huc4": {
              "regions": {
                "12815": {}
              }
            }
          }
        }
      }
    }
  }
}

TODO

Incorporate root_regions_proposal.json fields in example_data/

Sebastien notes

The root regions now include available variables for each root region, grouped by sensor, and alphabetically ordered. New attributes:

isDefault. When = 1, means web-app select by default the sensor/source and variable
colormap_value_range: dynamic for snow cover days. Development required for other variables to make it kind of dynamic (wait for Karl specs)
geotiffRelativePath: path of the geotiff
waterYear of data. Dev required for dynamic
lastDateWithData. idem
historicStartWaterYear: oldest water year with data
historicSource: indicate if source of historic is JPL or DAAC or a mix.

NB: a dev is required to make the list of variables dynamic depending on the actual available data. [Matt] Need help to understand this part!

Variables

Colormaps

Colormaps MUST be defined in colormaps.json.

Example colormaps.json

{
  "t01": [
    [2, 56, 88],
    [4, 90, 141],
    [5, 112, 176],
    [54, 144, 192],
    [116, 169, 207],
    [166, 189, 219],
    [208, 209, 230],
    [236, 231, 242],
    [255, 247, 251]
  ],
  "t02": [
    [255, 245, 240],
    [254, 224, 210],
    [252, 187, 161],
    [252, 146, 114],
    [251, 106, 74],
    [239, 59, 44],
    [203, 24, 29],
    [165, 15, 21],
    [103, 0, 13]
  ],
  "t03": [
    [247, 251, 255],
    [222, 235, 247],
    [198, 219, 239],
    [158, 202, 225],
    [107, 174, 214],
    [66, 146, 198],
    [33, 113, 181],
    [8, 81, 156],
    [8, 48, 107]
  ],
  "t04": [
    [255, 255, 204],
    [255, 237, 160],
    [254, 217, 118],
    [254, 178, 76],
    [253, 141, 60],
    [252, 78, 42],
    [227, 26, 28],
    [189, 0, 38],
    [128, 0, 38]
  ],
  "t05": [
    [252, 251, 253],
    [239, 237, 245],
    [218, 218, 235],
    [188, 189, 220],
    [158, 154, 200],
    [128, 125, 186],
    [106, 81, 163],
    [84, 39, 143],
    [63, 0, 125]
  ],
  "t06": [
    [8, 29, 88],
    [37, 52, 148],
    [34, 94, 168],
    [29, 145, 192],
    [65, 182, 196],
    [127, 205, 187],
    [199, 233, 180],
    [237, 248, 177],
    [255, 255, 217]
  ],
  "t07": [
    [0, 0, 0, 0],
    [233, 203, 0]
  ],
  "t08": [
    [247, 251, 255],
    [222, 235, 247],
    [198, 219, 239],
    [158, 202, 225],
    [107, 174, 214],
    [66, 146, 198],
    [33, 113, 181],
    [8, 81, 156],
    [8, 48, 107]
  ],
  "t09": [
    [178, 24, 43],
    [214, 96, 77],
    [244, 165, 130],
    [253, 219, 199],
    [247, 247, 247],
    [209, 229, 240],
    [146, 197, 222],
    [67, 147, 195],
    [33, 102, 172]
  ]
}

Variables

We’re only concerned with definitions for variables; availability of variables is on a per-region basis in regions/root.json.

Variables MUST be defined in variables.json.

Example variables.json

{
  "51": {
    "sensor": "modis",
    "platform": "terra",
    "algorithm": "scagdrfs/stc",
    "layerType": "raster",
    "longName": "Snow Albedo",
    "longNamePlot": "Average Snow Albedo",
    "labelMapLegend": "Snow Albedo (%, for Snow >=10%)",
    "labelPlotYAxis": "Albedo (%)",
    "helpText": "Fresh, clean snow with a high albedo appears bright, while old or dirty snow tends to have a lower albedo and appears darker. The snow albedo measurement is a non-dimensional, unitless quantity that measures how well a surface reflects solar energy, ranging from 0 to 1. A value of 0 means the surface is a perfect absorber, where all incoming energy is absorbed, while a value of 1 means the surface is a perfect reflector, where all incoming energy is reflected and none is absorbed. This quantity can also be expressed as a percent with a range from 0 to 100, with zero percent absorbing all incoming energy and 100 percent reflecting all energy.",
    "valuePrecision": 0,
    "colormapId": 1,
    "transparentZero": "false",
    "source": "MODIS (Terra) scagdrfs/stc",
    "noDataValue": 255,
    "valueRange": [0, 100]
  },
  "43": {
    "sensor": "modis",
    "platform": "terra",
    "algorithm": "scagdrfs/stc",
    "layerType": "raster",
    "longName": "Decrease in visible Albedo from Dust and Soot",
    "longNamePlot": "",
    "labelMapLegend": "Decrease in Albedo (%, for Snow >=10%)",
    "labelPlotYAxis": "Decrease in Albedo (%)",
    "helpText": "",
    "valuePrecision": 0,
    "colormapId": 2,
    "transparentZero": "false",
    "source": "MODIS (Terra) scagdrfs/stc",
    "noDataValue": 255,
    "valueRange": [0, 75]
  },
  "41": {
    "sensor": "modis",
    "platform": "terra",
    "algorithm": "scagdrfs/stc",
    "layerType": "raster",
    "longName": "Snow Grain Size",
    "longNamePlot": "",
    "labelMapLegend": "Micrometers (for Snow >=10%)",
    "labelPlotYAxis": "micrometers",
    "helpText": "",
    "valuePrecision": 0,
    "colormapId": 3,
    "transparentZero": "false",
    "source": "MODIS (Terra) scagdrfs/stc",
    "noDataValue": 65535,
    "valueRange": [10, 1100]
  },
  "44": {
    "sensor": "modis",
    "platform": "terra",
    "algorithm": "scagdrfs/stc",
    "layerType": "raster",
    "longName": "Snow Radiative Forcing",
    "longNamePlot": "Average Snow Radiative Forcing",
    "labelMapLegend": "Energy (W m^2, for Snow >=10%)",
    "labelPlotYAxis": "Energy (Watts per square meter)",
    "helpText": "When snow impurities such as dust or soot fall on snow, its surface darkens and absorbs more solar energy. Snow radiative forcing is a measure of the additional absorption of solar radiation from light absorbing particles (LAP) such as dust or soot. Units of measure are Watts per square meter (W/m2) and values can range from 0 to 500 W/m2. This maximum value depends on incoming solar radiation (elevation, direct sun vs shaded) and the amount of dust or soot. A value of 0 means no additional radiation is being absorbed. A value of 500 means nearly all of the solar energy is absorbed (depends on latitude, elevation, clouds). Radiative forcing is calculated by the difference between the net (downward minus upward) radiative fluxes (irradiance) with and without LAP.",
    "valuePrecision": 0,
    "colormapId": 4,
    "transparentZero": "false",
    "source": "MODIS (Terra) scagdrfs/stc",
    "noDataValue": 65535,
    "valueRange": [0, 500]
  },
  "45": {
    "sensor": "modis",
    "platform": "terra",
    "algorithm": "scagdrfs/stc",
    "layerType": "raster",
    "longName": "Snow Cover Days",
    "longNamePlot": "Average Snow Cover Days",
    "labelMapLegend": "Snow Cover Days (>=14)",
    "labelPlotYAxis": "Snow Cover Days",
    "helpText": "The number of days a region has been covered with snow as identified with snow cover percent greater than a snow cover percent threshold since a starting time. Our data uses 15% as the snow percent threshold and October 1 as the starting date for these maps.",
    "valuePrecision": 3,
    "colormapId": 5,
    "transparentZero": "true",
    "source": "MODIS (Terra) scagdrfs/stc",
    "noDataValue": 65535,
    "valueRange": [0, 366]
  },
  "40": {
    "sensor": "modis",
    "platform": "terra",
    "algorithm": "scagdrfs/stc",
    "layerType": "raster",
    "longName": "Snow Cover Percent",
    "longNamePlot": "Total Snow Cover Area",
    "labelMapLegend": "Snow Cover (>=10%)",
    "labelPlotYAxis": "Area (thousands of square kilometers)",
    "helpText": "The areal extent of snow-covered ground, expressed as the mathematical percent of a region covered with snow. In the context of this website, the region refers to an Earth observing satellite's smallest measurement area. We use data from the Moderate Resolution Imaging Spectroradiometer at ~463m spatial resolution. Note that the Earth's surface is sometimes covered by clouds.",
    "valuePrecision": 0,
    "colormapId": 6,
    "transparentZero": "true",
    "source": "MODIS (Terra) scagdrfs/stc",
    "noDataValue": 255,
    "valueRange": [0, 100]
  },
  "36": {
    "sensor": "modis",
    "platform": "terra",
    "algorithm": "scagdrfs/stc",
    "layerType": "raster",
    "longName": "Viewable Snow Cover Percent",
    "longNamePlot": "",
    "labelMapLegend": "Snow Cover (>=10%)",
    "labelPlotYAxis": "Area (thousands of square kilometers)",
    "helpText": "",
    "valuePrecision": 0,
    "colormapId": 6,
    "transparentZero": "true",
    "source": "MODIS (Terra) scagdrfs/stc",
    "noDataValue": 255,
    "valueRange": [0, 100]
  },
  "52": {
    "sensor": "modis",
    "platform": "terra",
    "algorithm": "spires",
    "layerType": "raster",
    "longName": "Viewable Snow Cover Percent",
    "longNamePlot": "",
    "labelMapLegend": "Snow Cover (>=10%)",
    "labelPlotYAxis": "Area (thousands of square kilometers)",
    "helpText": "",
    "valuePrecision": 0,
    "colormapId": 6,
    "transparentZero": "true",
    "source": "MODIS (Terra) spires",
    "noDataValue": 255,
    "valueRange": [0, 100]
  },
  "56": {
    "sensor": "modis",
    "platform": "terra",
    "algorithm": "spires",
    "layerType": "raster",
    "longName": "Snow Cover Percent",
    "longNamePlot": "Total Snow Cover Area",
    "labelMapLegend": "Snow Cover (>=10%)",
    "labelPlotYAxis": "Area (thousands of square kilometers)",
    "helpText": "The areal extent of snow-covered ground, expressed as the mathematical percent of a region covered with snow. In the context of this website, the region refers to an Earth observing satellite's smallest measurement area. We use data from the Moderate Resolution Imaging Spectroradiometer at ~463m spatial resolution. Note that the Earth's surface is sometimes covered by clouds.",
    "valuePrecision": 0,
    "colormapId": 6,
    "transparentZero": "true",
    "source": "MODIS (Terra) spires",
    "noDataValue": 255,
    "valueRange": [0, 100]
  },
  "57": {
    "sensor": "modis",
    "platform": "terra",
    "algorithm": "scagdrfs/stc",
    "layerType": "raster_notprocessed",
    "longName": "Not Processed",
    "longNamePlot": "",
    "labelMapLegend": "",
    "labelPlotYAxis": "",
    "helpText": "",
    "valuePrecision": 0,
    "colormapId": 7,
    "transparentZero": "false",
    "source": "MODIS (Terra) scagdrfs/stc",
    "noDataValue": 255,
    "valueRange": [0, 1]
  },
  "58": {
    "sensor": "modis",
    "platform": "terra",
    "algorithm": "spires",
    "layerType": "raster_notprocessed",
    "longName": "Not Processed",
    "longNamePlot": "",
    "labelMapLegend": "",
    "labelPlotYAxis": "",
    "helpText": "",
    "valuePrecision": 0,
    "colormapId": 7,
    "transparentZero": "false",
    "source": "MODIS (Terra) scagdrfs/stc",
    "noDataValue": 255,
    "valueRange": [0, 1]
  },
  "59": {
    "sensor": "",
    "platform": "",
    "algorithm": "",
    "layerType": "point_swe",
    "longName": "Snow Water Equivalent",
    "longNamePlot": "",
    "labelMapLegend": "Snow Water Equivalent (inches)",
    "labelPlotYAxis": "",
    "helpText": "",
    "valuePrecision": 0,
    "colormapId": 8,
    "transparentZero": "false",
    "source": "",
    "noDataValue": 255,
    "valueRange": [0, 100]
  },
  "60": {
    "sensor": "",
    "platform": "",
    "algorithm": "",
    "layerType": "point_swe",
    "longName": "Change in Snow Water Equivalent",
    "longNamePlot": "",
    "labelMapLegend": "Change in SWE (inches) in past 24 hours",
    "labelPlotYAxis": "",
    "helpText": "",
    "valuePrecision": 0,
    "colormapId": 9,
    "transparentZero": "false",
    "source": "",
    "noDataValue": 255,
    "valueRange": [-4, 4]
  },
  "61": {
    "sensor": "",
    "platform": "",
    "algorithm": "",
    "layerType": "point_swe",
    "longName": "Percentage of Median Snow Water Equivalent",
    "longNamePlot": "",
    "labelMapLegend": "Percentage of Median Snow Water Equivalent (%)",
    "labelPlotYAxis": "",
    "helpText": "",
    "valuePrecision": 0,
    "colormapId": 9,
    "transparentZero": "false",
    "source": "",
    "noDataValue": 255,
    "valueRange": [0, 200]
  }
}

TODO

Extract SWE variables to their own dedicated file. They have different requirements from the GeoTIFF variables, and overloading the JSON isn’t working.

Sebastien notes

I created the variables.json file and updated github

new attributes

sensor, platform, algorithm, sensor_text (text to display)
color_map_id: refers to colormaps.json file

attributes removed

enabled: Now determined by the variables attribute in regions/root.json, because different variables can be enabled for different regions.
cog_path: geotiff filepath In regions/root.json
color_map
color_map_value_range: should be in regions/root.json. Still needs clarification of specs by Karl.

I also included the SWE variables, which were in the original variables.json

Citation

BibTeX citation:

@online{lenard2023,
  author = {Lenard, Sebastien and Fisher, Matt},
  title = {Supercomputer Interface Specification Overhaul},
  date = {2023-11-16},
  url = {https://nsidc.github.io/snow-today-webapp-server/interfaces/supercomputer_data/drafts/2023_overhaul},
  langid = {en}
}

For attribution, please cite this work as:

Lenard, Sebastien, and Matt Fisher. 2023. “Supercomputer Interface Specification Overhaul.” November 16, 2023. https://nsidc.github.io/snow-today-webapp-server/interfaces/supercomputer_data/drafts/2023_overhaul.