Skip to content

Compression

Compression is designed to support different types of compression algorithms with no or minimal impact on query performance. Compression is lossless.

Supported dataset types

Compression is available for

  • grids with single-value items,
  • meshes with single-value items, and with z-level items in SingleTimestep and MultiTimestep storage,

incl. layered items, in every storage type (SingleTimestep, MultiTimestep, TimeSeries - compressed as the uncompressed layout stores the values), except sigma-layer meshes. Requesting compression for anything else is rejected.

Compression algorithms

All of them support every datatype (float, double, int, ...).

  • DeflateFast: the standard .NET deflate optimized for speed
  • DeflateOptimal: the standard .NET deflate optimized for size
  • LZ4: block compression optimized for speed
  • DeflateOptimalAuto: Deflate (Optimal) with an automatic per-tile transform (byte shuffle, 1-D/2-D delta, byte delta, float quantization detection, double quantization where the environment enables it): about a third smaller than DeflateOptimal on smooth integer data such as scaled Int16 satellite/ocean grids, 40-60 % smaller on quantised GRIB floats, at most 0.2 percentage points worse elsewhere (sparse precipitation, tiny dfs2 grids); reads at 0.7-1.1x of DeflateOptimal, writes at 0.6-1.1x of its throughput
  • ZstdAuto (the recommended setting for grids, meshes and the TimeSeries storage): the same automatic per-tile transform with Zstandard (level 7) as the backend codec - class averages within 0.4 percentage points of DeflateOptimalAuto on grid tiles, up to 2 points worse on meshes (4-5 on tiny meshes of a few hundred elements), and reads 1.0-1.7x faster
  • ZstdFastAuto: ZstdAuto written at zstd level 3. The block format and the read speed are identical to ZstdAuto, the data is 0-3 percentage points larger and the writes are 1.2-5x faster. Meant for ingest-speed-critical conversions and for datasets appended to often (see "TimeSeries storage")

Default compression

A new dataset gets the default compression of its kind (grids stored per timestep, meshes stored per timestep, TimeSeries storage) from the environment; ZstdAuto is the intended setting for all three. The default is applied only when the writer parameter Compression is not given; pass "Compression": "None" to store a dataset uncompressed. Existing datasets keep the settings they were created with (append/update follows the dataset, not the default).

Compression never changes the storage layout: the writer picks the storage type by size (SingleTimestep when at most one timestep fits the 10 MB blob the writer aims at - above 655 360 elements at the 8 B per element it assumes - MultiTimestep with the computed TimeGroupSize otherwise, at most 49 999 timesteps per group; a TimeGroupSize parameter of 1 or less means SingleTimestep, the StorageType parameter overrides all of it) and compresses the blobs of that layout. MultiTimestep storage does not support removing timesteps. A TimeGroupSize outside 1-49 999 with MultiTimestep and a GridTileWidth outside 16-4096 are refused, with or without compression; an explicit MultiTimestep of a grid or mesh above 1 310 720 elements gets a TimeGroupSize of 1. DataSizeBytes of a compressed dataset is the stored size, listed after an import and after every append or update.

Per-timestep storage (SingleTimestep, MultiTimestep)

A compressed timestep is stored as pages compressed one by one: for a grid the tiles of GridTileWidth x GridTileWidth cells (default 128 x 128), for a mesh runs of 16 384 elements in the spatial index order (CompressedMeshPageSize, persisted when the dataset is created). A spatial query decodes only the pages it touches, a whole-slice read all of them; within a grid page the query skips the blocks the geometry misses as the uncompressed storage does. A page that is no-data everywhere is stored empty. A z-level item of a mesh is cut into the same pages and a z query decodes only the pages it needs. A query that meets a re-import of the timestep it reads retries once on the new data; a timestep whose re-import was killed half way is refused until the re-import is run again.

TimeSeries storage

Datasets stored as TimeSeries are compressed per blob of one spatial group x one time group: the blob holds blocks of TimeSeriesBlockRows whole series, each block one compression block. A series query reads and decodes only the blocks it needs. TimeSeriesBlockRows is chosen when the dataset is created and persisted: the smallest row count whose raw block reaches 32 KB, clamped to 4-256 and to the spatial group size. An all-no-data block is stored empty. Appending or replacing a fixed-step time group decodes the stored group, patches it and encodes it again - the same whole-blob rewrite the uncompressed layout does, plus the codec: 7-15 s of CPU per GB of raw group data with ZstdAuto, 3-7 s with ZstdFastAuto. A dataset appended to often is a case for "Compression": "ZstdFastAuto". A replace of variable-step groups overwrites the group blobs; a replace killed half way leaves the stored groups readable and can be run again. Sigma-layer meshes are not compressed.

Compatibility

The compression is stored by name in the dataset metadata together with the version of the storage layout and the layout parameters of its kind (TimeSeriesBlockRows, CompressedMeshPageSize). A dataset keeps the compression, the layout and the parameters it was created with. A dataset compressed with a name or a layout a deployment does not know yet stays visible and its metadata is left untouched, only reading or writing its data is refused with a message saying so.

Enabling compression in conversion

Compression is enabled by adding writer parameter Compression with the name of Compression algorithm (see above). Example:

 "writerParameters": [
    {
      "name": "Compression",
      "value": "ZstdAuto"
    }
  ]

Specific datasets

HYCOM, COPERNICUS

NetCDF variables packed as scaled Int16 (Copernicus, older HYCOM imports) are stored as Int16 - the production census also holds HYCOM GLBy and Mercator stored as Double; ZstdAuto (byte shuffle + delta on Int16), the intended environment default - the grid default, or the TimeSeries default for datasets stored as TimeSeries - is the right choice; the LZ4 of the older imports stays readable (and selectable) but has no reason to be chosen any more.

NOAA

NOAA is highly (lossy with JPEG2000) compressed format. Storing it as uncompressed doubles is not optimal. Float precision is sufficient. NOAA Reader adds reader parameter DoubleAsFloat, which forces converting and storing double value as float.

"readerParameters": [
    {
      "name": "DoubleAsFloat",
      "value": true
    }
  ]