Skip to content

Create timeseries oriented dataset

Dataset optimized for timeseries queries for discrete points must be imported with specific parameters.

Parameter Value Notes
StorageType TimeSeries (string) Mandatory
TimeGroupSize Number of steps per block (int) Mandatory , for variable blocks it is the max size,
SpatialGroupSize Number of spatial elements per block (int) Mandatory, more details below
TimeGroupFixedStep Number of seconds between steps (int) Optional, if set fixed blocks will be created, otherwise blocks have variable size, defines the storage layout not resample the source data

Read Data storage options description for more information.

Use case - Historical timeseries with variable block size

Example : long historical data, e.g 40 years of data history in metocean datasets.

Prepare writer parameters

var writerParams = new ParameterInput[]
{
    new ParameterInput {Name="StorageType",Value="TimeSeries"},
    new ParameterInput {Name="TimeGroupSize",Value="integer value"}, 
    new ParameterInput {Name="SpatialGroupSize",Value="integer value"} 
};

TimeGroupSize should be bigger than the number of timesteps in 1 imported file e.g in 1 year to avoid data splitting.

Example :

Timestep Interval : 1 hour 
Each imported file represents 1 year of data

Set TimeGroupSize > 365*24.

Use case - Forecast flow with fixed block size

For a forecast flow with regular schedule and moving window size, set the TimeGroupSize to the window size or bigger (e.g. the whole forecast period).

image.png

When new forecast is ready, the update operation is invoked and appropriate time blocks will be replaced.

The TimeGroupFixedStep parameter must be provided to use fixed blocks.

var writerParams = new ParameterInput[]
{
    new ParameterInput {Name="StorageType",Value="TimeSeries"},
    new ParameterInput {Name="TimeGroupFixedStep",Value="integer value"}, 
    new ParameterInput {Name="TimeGroupSize",Value="integer value"}, 
    new ParameterInput {Name="SpatialGroupSize",Value="integer value"} 
};

Choosing TimeGroupSize and SpatialGroupSize

Each item is stored in blocks: one block holds SpatialGroupSize elements for TimeGroupSize time steps and is a separate file in blob storage. A point time series query reads one range from every block it touches:

reads = items x spatial groups touched x (requested time steps / TimeGroupSize)

A long time series stored in small blocks therefore needs thousands of reads and is slow (or times out), even when the returned data is small.

Rules of thumb:

  • Aim for blocks of roughly 4-32 MB: SpatialGroupSize * TimeGroupSize * (4 for float or 8 for double).
  • Keep every block below 2 GB.
  • Import and update keep one time block of the whole spatial domain in memory, about elements * TimeGroupSize * (4 for float or 8 for double) (for 3D datasets count the 3D elements). For example 500 000 3D elements with TimeGroupSize = 4000 need about 8 GB, more than an import job gets by default: request more memory for the transfer with .WithMemoryRequest(...).
  • Make TimeGroupSize as large as your updates allow. An update replaces whole blocks, so the block length is a trade-off between the cost of a long query and the cost of an update:
    • historical data that is imported once (hindcast): large blocks, months to a year of time steps
    • forecast flows: blocks covering the replaced forecast window, see the forecast use case above
  • With variable blocks, a block never spans two imports or updates: each import or append starts new blocks, and the last block is never extended. A dataset appended one day at a time therefore ends up with one-day blocks even with TimeGroupSize = 8760. If data arrives in small parts (e.g. daily), use fixed blocks (TimeGroupFixedStep), so that new data is written into the block that already covers its time.
  • If TimeGroupSize is small (~< 1000), set SpatialGroupSize >= number of elements in the spatial domain, so a point touches only one spatial group.
  • Query long periods in chunks (for example one year per request) when the result is large, e.g. all layers of a 3D dataset.

Example: one float item, hourly data, SpatialGroupSize = 1000, point query over 15 years (131 400 time steps):

TimeGroupSize Time per block Block size Reads per item
24 1 day 96 KB ~5 500
50 ~2 days 200 KB ~2 600
4000 ~5.5 months 16 MB ~33
8760 1 year 35 MB ~15

3D datasets

In a 3D dataset every 3D element is one row of the block, and the layers of one water column are consecutive rows. SpatialGroupSize is therefore counted in 3D elements: set it about one order lower than for 2D data, and compensate with a larger TimeGroupSize to keep the block size in the range above. A column that crosses the boundary between two spatial groups is read from both blocks, which doubles the number of reads for that point.

Example for a 3D DFSU file with hourly time steps imported as historical data:

  • TimeGroupSize = 4000 (about 5.5 months in each block)
  • SpatialGroupSize = 1000
  • TimeGroupFixedStep = 3600 (one hour between time steps within each block), only if fixed blocks are needed

Import new dataset with writer parameters

var transfer = transferClient.CreateFileImport(_projectId, "sampleTS.dfsu")
        .WithReader(new DfsuReader())
        .WithWriter(new GenericWriter("MDWriter", writerParams))

Note: Dfsu is just an example reader.