Skip to content

Compression

Compression is designed to support different types of compression algorithms with no or minimal impact on query performance. Compression is lossless.

Supported dataset types

Compression option is available only for selected dataset types

  • Spatial domain must be gridded. Mesh spatial domain is not supported yet.
  • Only timestep storage type is supported. Timeseries cannot be compressed.

Compression algorithms

  • Deflate compression : standard .Net deflate algorithm, supports all datatypes (float, double, int,...)

  • DeflateFast: deflate optimized for speed

  • DeflateOptimal: deflate optimized for size
  • LZ4: block compression optimized for speed, supports all datatypes
  • DeflateOptimalAuto: Deflate (Optimal) with an automatic per-tile transform (byte shuffle, 1-D/2-D delta, byte delta, float quantization detection) - best ratio for all datatypes: 30-45 % smaller than DeflateOptimal on smooth integer data such as scaled Int16 satellite/ocean grids, 35-60 % smaller on quantised floats (GRIB), never worse elsewhere; reads as fast as DeflateOptimal or faster (smaller blocks, vectorized transforms), writes at 0.6-1.0x of its throughput - the transforms are compared on every 32nd tile only (DeflateAutoCompressor.DefaultProbeStride) and the winner is kept for the tiles in between and across the slices of the same item (AutoTransformState)
  • ZstdAuto (the default for new gridded datasets): the same automatic per-tile transform with Zstandard (level 9, RFC 8878 frames) as the backend codec - matches or slightly beats DeflateOptimalAuto on ratio and reads 1.3-2x faster; write throughput is comparable. Like DeflateOptimalAuto it is a new block format: every deployment that reads MD data must understand it before a dataset is written with it.
  • ZstdFastAuto: ZstdAuto written at zstd level 3 - 3-6x faster writes for ~1 percentage point of ratio. The level is a write-side choice only, the block format is identical to ZstdAuto - but the name is stored in the dataset metadata, so a build has to know the name to read the dataset (see "Compatibility" below). Meant for ingest-speed-critical conversions. For data written once and read often, which is the common case, the default ZstdAuto is the better choice.

Default compression

New gridded datasets stored per timestep get the default compression of the environment (MDWriterOptions.DefaultGridCompression), set through the environment variable MD_DEFAULT_GRID_COMPRESSION (forwarded to transfer jobs via TransferOptions.DefaultGridCompression). The code default is None, so a freshly deployed build never starts writing a block format on its own; the intended production setting is ZstdAuto. Set it in an environment only after every deployment that reads MD data runs a build that knows it (see "Compatibility"): deploy the build everywhere, verify the readers, then set the variable. An older build ignores the variable, so setting it early does not break the old build - the datasets written by the new build in the meantime would. The default is applied only when the writer parameter Compression is not given; pass "Compression": "None" to store a dataset uncompressed. Mesh datasets, the TimeSeries storage type, an explicitly requested MultiTimestep storage and datasets with items holding more than one value per element (z-level items) are never compressed - requesting compression for them is rejected. Existing datasets keep the settings they were created with (append/update follows the dataset, not the default). Enabling compression stores the dataset as SingleTimestep (one blob per time step, item and layer) instead of MultiTimestep. Tiles of one slice are encoded by up to 4 workers in parallel (CompressedGridIndex.MaxEncodeWorkers, one per available CPU); MD_ENCODE_WORKERS (forwarded via TransferOptions.EncodeWorkers) sets the count explicitly, 1 makes the encoding serial.

Compatibility

The compression is stored by name in the dataset metadata (ItemStorage.Compression) and read by every deployment that loads MD datasets: the multidimensional service, the multidimensional cron jobs (data cleanup) and the transfer jobs (append, update, export). A build that does not know a name loads the dataset with CompressionType.Unknown (CompressionTypeConverter): the dataset stays visible and its metadata is left untouched, only reading or writing its data fails with a message saying the compression is not known to this build. Writing such a dataset's metadata back is refused, so an older build can never replace the name it does not know. A newer build's writer accepts a new name as soon as it is deployed - through the environment default or an explicit writer parameter - so the order per environment is: deploy the build to all readers, then enable the compression.

The second compatibility boundary is the kind of dataset: compression is currently read and written for gridded datasets with SingleTimestep storage and single-value items only (CompressionExtensions.IsCompressionSupported). A compressed dataset of any other kind (mesh, MultiTimestep, TimeSeries, z-level items - as a newer build may write) loads, but every data read or write is refused by DatasetStorage with a message saying this build cannot read it; without that guard the storage classes for those kinds would read the compressed blobs as raw data. The rollout of such an extension is the same as for a new name: deploy the build to all readers first, then enable the compression for that kind.

Extending compression to another kind of dataset

  1. Implement the compressed read and write path of the storage class for that kind (as SingleTimestepStorage and CompressedGridIndex do for grids) and relax CompressionExtensions.IsCompressionSupported for it - that single predicate gates the writer validation, DatasetStorage.StoreAsync and the read-side refusal above.
  2. Keep the refusal test (DatasetStorageTests.CompressedDatasetOfAKindThisBuildCannotReadStillLoadsButRefusesItsData) for the kinds that remain unsupported and add a round-trip test for the new one.
  3. Enable it per environment after the build rollout: MD_DEFAULT_GRID_COMPRESSION is applied to IGrid2D domains only (MDWriter.GetCompression), so relaxing the predicate does not switch the new kind on anywhere - it stays opt-in through the writer parameter until it gets its own default setting, which is what makes the rollout of the read path and the switch-on two separate steps.

Adding a compression type

  1. Add the value to CompressionType (ItemStorageOptions.cs) - the name is the persisted format identifier, keep it stable.
  2. Implement ICompressor (or derive from DeflateAutoCompressor for a new backend codec, as ZstdAutoCompressor does) and add the branch to CompressionExtensions.CreateCompressor. A compression built on DeflateAutoCompressor must also be listed in CompressionExtensions.IsAutoTransform, otherwise it does not get the per-item probe state. CompressionTypeTests.EverySelectableCompressionResolvesToACompressor fails until both are done.
  3. Tests: round trip (CompressorRoundTripTests), the garbage patterns (CompressorStressTests), CreateCompressor resolves the type, and a pinned block format if the type must stay readable by an existing reader.
  4. Describe it above and in conversion.dedicated.writers.import.json (writer parameter description).
  5. Roll out as described under "Compatibility"; do not make it the default in the same release. A write-side variant of an existing format (a different level) is still a new name, i.e. a new compatibility boundary - prefer a separate option (e.g. a level) that older builds ignore over a new enum value.

Enabling compression in conversion

Compression is enabled by adding writer parameter Compression with the name of Compression algorithm (see above). Example: 

 "writerParameters": [
    {
      "name": "Compression",
      "value": "DeflateFast"
    }

Specific datasets

HYCOM, COPERNICUS

Hycom and Copernicus use Int16 internal storage. Use LZ4.

 "writerParameters": [
    {
      "name": "Compression",
      "value": "LZ4"
    }
  ```

### NOAA
NOAA is highly (lossy with JPEG2000) compressed format.
Storing it as uncompressed doubles is not optimal. Float precision is sufficient.
NOAA Reader adds reader parameter DoubleAsFloat, which forces converting and storing double value as float.

NOAA can use only deflate compression now.
"readerParameters": [ { "name": "DoubleAsFloat", "value": true } ], "writerParameters": [ { "name": "Compression", "value": "DeflateFast" } ```