Create timeseries oriented dataset
Dataset optimized for timeseries queries for discrete points must be imported with specific parameters.
| Parameter | Value | Notes |
|---|---|---|
StorageType |
TimeSeries (string) | Mandatory |
TimeGroupSize |
Number of steps per block (int) | Mandatory , for variable blocks it is the max size, |
SpatialGroupSize |
Number of spatial elements per block (int) | Mandatory, more details below |
TimeGroupFixedStep |
Number of seconds between steps (int) | Optional, if set fixed blocks will be created, otherwise blocks have variable size, defines the storage layout not resample the source data |
Read Data storage options description for more information.
Use case - Historical timeseries with variable block size¶
Example : long historical data, e.g 40 years of data history in metocean datasets.
Prepare writer parameters¶
var writerParams = new ParameterInput[]
{
new ParameterInput {Name="StorageType",Value="TimeSeries"},
new ParameterInput {Name="TimeGroupSize",Value="integer value"},
new ParameterInput {Name="SpatialGroupSize",Value="integer value"}
};
TimeGroupSize should be bigger than the number of timesteps in 1 imported file e.g in 1 year to avoid data splitting.
Example :
Timestep Interval : 1 hour
Each imported file represents 1 year of data
Set TimeGroupSize > 365*24.
Use case - Forecast flow with fixed block size¶
For a forecast flow with regular schedule and moving window size, set the TimeGroupSize to the window size or bigger (e.g. the whole forecast period).

When new forecast is ready, the update operation is invoked and appropriate time blocks will be replaced.
The TimeGroupFixedStep parameter must be provided to use fixed blocks.
var writerParams = new ParameterInput[]
{
new ParameterInput {Name="StorageType",Value="TimeSeries"},
new ParameterInput {Name="TimeGroupFixedStep",Value="integer value"},
new ParameterInput {Name="TimeGroupSize",Value="integer value"},
new ParameterInput {Name="SpatialGroupSize",Value="integer value"}
};
Choosing TimeGroupSize and SpatialGroupSize¶
Each item is stored in blocks: one block holds SpatialGroupSize elements for TimeGroupSize time steps and is a separate file in blob storage. A point time series query reads one range from every block it touches:
reads = items x spatial groups touched x (requested time steps / TimeGroupSize)
A long time series stored in small blocks therefore needs thousands of reads and is slow (or times out), even when the returned data is small.
Rules of thumb:
- Aim for blocks of roughly 4-32 MB:
SpatialGroupSize * TimeGroupSize * (4 for float or 8 for double). - Keep every block below 2 GB.
- Import and update keep one time block of the whole spatial domain in memory, about
elements * TimeGroupSize * (4 for float or 8 for double)(for 3D datasets count the 3D elements). For example 500 000 3D elements withTimeGroupSize= 4000 need about 8 GB, more than an import job gets by default: request more memory for the transfer with.WithMemoryRequest(...). - Make
TimeGroupSizeas large as your updates allow. An update replaces whole blocks, so the block length is a trade-off between the cost of a long query and the cost of an update:- historical data that is imported once (hindcast): large blocks, months to a year of time steps
- forecast flows: blocks covering the replaced forecast window, see the forecast use case above
- With variable blocks, a block never spans two imports or updates: each import or append starts new blocks, and the last block is never extended. A dataset appended one day at a time therefore ends up with one-day blocks even with
TimeGroupSize= 8760. If data arrives in small parts (e.g. daily), use fixed blocks (TimeGroupFixedStep), so that new data is written into the block that already covers its time. - If
TimeGroupSizeis small (~< 1000), setSpatialGroupSize>= number of elements in the spatial domain, so a point touches only one spatial group. - Query long periods in chunks (for example one year per request) when the result is large, e.g. all layers of a 3D dataset.
Example: one float item, hourly data, SpatialGroupSize = 1000, point query over 15 years (131 400 time steps):
| TimeGroupSize | Time per block | Block size | Reads per item |
|---|---|---|---|
| 24 | 1 day | 96 KB | ~5 500 |
| 50 | ~2 days | 200 KB | ~2 600 |
| 4000 | ~5.5 months | 16 MB | ~33 |
| 8760 | 1 year | 35 MB | ~15 |
3D datasets¶
In a 3D dataset every 3D element is one row of the block, and the layers of one water column are consecutive rows. SpatialGroupSize is therefore counted in 3D elements: set it about one order lower than for 2D data, and compensate with a larger TimeGroupSize to keep the block size in the range above. A column that crosses the boundary between two spatial groups is read from both blocks, which doubles the number of reads for that point.
Example for a 3D DFSU file with hourly time steps imported as historical data:
- TimeGroupSize = 4000 (about 5.5 months in each block)
- SpatialGroupSize = 1000
- TimeGroupFixedStep = 3600 (one hour between time steps within each block), only if fixed blocks are needed
Import new dataset with writer parameters¶
var transfer = transferClient.CreateFileImport(_projectId, "sampleTS.dfsu")
.WithReader(new DfsuReader())
.WithWriter(new GenericWriter("MDWriter", writerParams))
Note: Dfsu is just an example reader.