Dataset

The Dataset is the MIKE IO data structure for data from dfs files. The mikeio.read methods returns a Dataset as a container of DataArray (Dfs items). Each DataArray has the properties, item, time, geometry and values. The time and geometry are common to all DataArrays in the Dataset.

The Dataset has the following primary properties:

Use Dataset’s string representation to get an overview of the Dataset

import mikeio
ds = mikeio.read("../data/HD2D.dfsu")
ds
<mikeio.Dataset>
title: Output 1
dims: (time:9, element:884)
time: 1985-08-06 07:00:00 - 1985-08-07 03:00:00 (9 records)
geometry: Dfsu2D (884 elements, 529 nodes)
items:
  0:  Surface elevation <Surface Elevation> (meter)
  1:  U velocity <u velocity component> (meter per sec)
  2:  V velocity <v velocity component> (meter per sec)
  3:  Current speed <Current Speed> (meter per sec)

Display options

Long item lists are truncated in the string representation:

ds_sw = mikeio.read("../data/sw_points.dfs0")
ds_sw
<mikeio.Dataset>
title: Point
dims: (time:11)
time: 2017-01-01 00:00:00 - 2017-01-01 10:00:00 (11 records)
geometry: GeometryUndefined()
items:
  0:  Buoy 2: Sign. Wave Height <Significant wave height> (meter)
  1:  Point 4: Sign. Wave Height <Significant wave height> (meter)
  2:  Point 3: Sign. Wave Height <Significant wave height> (meter)
  3:  Point 42: Sign. Wave Height <Significant wave height> (meter)
  4:  Buoy 2: Peak Wave Period <Wave period> (second)
  5:  Point 4: Peak Wave Period <Wave period> (second)
  6:  Point 3: Peak Wave Period <Wave period> (second)
  7:  Point 42: Peak Wave Period <Wave period> (second)
  8:  Buoy 2: Wave Period, T02 <Wave period> (second)
  9:  Point 4: Wave Period, T02 <Wave period> (second)
  ... and 50 more items (60 total)

The cutoff is a global option, which can be raised (or removed with None) with mikeio.set_options:

mikeio.set_options(display_max_items=15)
ds_sw
<mikeio.Dataset>
title: Point
dims: (time:11)
time: 2017-01-01 00:00:00 - 2017-01-01 10:00:00 (11 records)
geometry: GeometryUndefined()
items:
  0:  Buoy 2: Sign. Wave Height <Significant wave height> (meter)
  1:  Point 4: Sign. Wave Height <Significant wave height> (meter)
  2:  Point 3: Sign. Wave Height <Significant wave height> (meter)
  3:  Point 42: Sign. Wave Height <Significant wave height> (meter)
  4:  Buoy 2: Peak Wave Period <Wave period> (second)
  5:  Point 4: Peak Wave Period <Wave period> (second)
  6:  Point 3: Peak Wave Period <Wave period> (second)
  7:  Point 42: Peak Wave Period <Wave period> (second)
  8:  Buoy 2: Wave Period, T02 <Wave period> (second)
  9:  Point 4: Wave Period, T02 <Wave period> (second)
  10:  Point 3: Wave Period, T02 <Wave period> (second)
  11:  Point 42: Wave Period, T02 <Wave period> (second)
  12:  Buoy 2: Peak Wave Direction <Wave direction> (radian)
  13:  Point 4: Peak Wave Direction <Wave direction> (radian)
  14:  Point 3: Peak Wave Direction <Wave direction> (radian)
  ... and 45 more items (60 total)

Use display_max_items=None to list all items.

set_options also works as a context manager, which restores the previous value on exit, similar to xarray.set_options:

with mikeio.set_options(display_max_items=3):
    print(ds_sw)
<mikeio.Dataset>
title: Point
dims: (time:11)
time: 2017-01-01 00:00:00 - 2017-01-01 10:00:00 (11 records)
geometry: GeometryUndefined()
items:
  0:  Buoy 2: Sign. Wave Height <Significant wave height> (meter)
  1:  Point 4: Sign. Wave Height <Significant wave height> (meter)
  2:  Point 3: Sign. Wave Height <Significant wave height> (meter)
  ... and 57 more items (60 total)
mikeio.get_options()
{'display_max_items': 15, 'show_progress': False}

The option applies to all MIKE IO objects with items, including the file objects returned by mikeio.open.

Selecting items

Selecting a specific item “itemA” (at position 0) from a Dataset ds can be done with:

  • ds[["itemA"]] - returns a new Dataset with “itemA”
  • ds["itemA"] - returns “itemA” DataArray
  • ds[[0]] - returns a new Dataset with “itemA”
  • ds[0] - returns “itemA” DataArray
  • ds.itemA - returns “itemA” DataArray

We recommend to use named items for readability.

ds.Surface_elevation
<mikeio.DataArray>
name: Surface elevation
dims: (time:9, element:884)
time: 1985-08-06 07:00:00 - 1985-08-07 03:00:00 (9 records)
geometry: Dfsu2D (884 elements, 529 nodes)

Negative index e.g. ds[-1] can also be used to select from the end. Several items (“itemA” at 0 and “itemC” at 2) can be selected with the notation:

  • ds[["itemA", "itemC"]]
  • ds[[0, 2]]

Note that this behavior is similar to pandas and xarray.

Temporal selection

A time slice of a Dataset can be selected with sel (by label/value) or isel (by integer index), similar to xarray.

Select a single timestep by value:

ds.sel(time="1985-08-06 12:00")
<mikeio.Dataset>
title: Output 1
dims: (element:884)
time: 1985-08-06 12:00:00 (time-invariant)
geometry: Dfsu2D (884 elements, 529 nodes)
items:
  0:  Surface elevation <Surface Elevation> (meter)
  1:  U velocity <u velocity component> (meter per sec)
  2:  V velocity <v velocity component> (meter per sec)
  3:  Current speed <Current Speed> (meter per sec)

Select a range of timesteps by value:

ds.sel(time=slice("1985-08-06 12:00", "1985-08-06 17:00"))
<mikeio.Dataset>
title: Output 1
dims: (time:3, element:884)
time: 1985-08-06 12:00:00 - 1985-08-06 17:00:00 (3 records)
geometry: Dfsu2D (884 elements, 529 nodes)
items:
  0:  Surface elevation <Surface Elevation> (meter)
  1:  U velocity <u velocity component> (meter per sec)
  2:  V velocity <v velocity component> (meter per sec)
  3:  Current speed <Current Speed> (meter per sec)

Select a single timestep by index:

ds.isel(time=2)
<mikeio.Dataset>
title: Output 1
dims: (element:884)
time: 1985-08-06 12:00:00 (time-invariant)
geometry: Dfsu2D (884 elements, 529 nodes)
items:
  0:  Surface elevation <Surface Elevation> (meter)
  1:  U velocity <u velocity component> (meter per sec)
  2:  V velocity <v velocity component> (meter per sec)
  3:  Current speed <Current Speed> (meter per sec)

Select every other timestep using a slice:

ds.isel(time=slice(None, None, 2))
<mikeio.Dataset>
title: Output 1
dims: (time:5, element:884)
time: 1985-08-06 07:00:00 - 1985-08-07 03:00:00 (5 records)
geometry: Dfsu2D (884 elements, 529 nodes)
items:
  0:  Surface elevation <Surface Elevation> (meter)
  1:  U velocity <u velocity component> (meter per sec)
  2:  V velocity <v velocity component> (meter per sec)
  3:  Current speed <Current Speed> (meter per sec)
Tip

To extract every n’th timestep directly to a new file without loading all data into memory, use mikeio.generic.extract with the step parameter.

Spatial selection

The sel method finds a single element.

ds.sel(x=607002, y=6906734)
<mikeio.Dataset>
title: Output 1
dims: (time:9)
time: 1985-08-06 07:00:00 - 1985-08-07 03:00:00 (9 records)
geometry: GeometryPoint2D(x=607002.7094112666, y=6906734.833048992)
items:
  0:  Surface elevation <Surface Elevation> (meter)
  1:  U velocity <u velocity component> (meter per sec)
  2:  V velocity <v velocity component> (meter per sec)
  3:  Current speed <Current Speed> (meter per sec)

Plotting

In most cases, you will not plot the Dataset, but rather its DataArrays. But there are two exceptions:

  • dfs0-Dataset : plot all items as timeseries with ds.plot()
  • scatter : compare two items using ds.plot.scatter(x=“itemA”, y=“itemB”)

See the DatasetPlotter API for details.

Add a new item

A common workflow is to create a new item based on existing items in a dataset.

This can be in done in several ways. Let’s try one of the options.

ds = mikeio.read("../data/NorthSea_HD_and_windspeed.dfsu")
ds
<mikeio.Dataset>
title: Area Series
dims: (time:67, element:958)
time: 2017-10-27 00:00:00 - 2017-10-29 18:00:00 (67 records)
geometry: Dfsu2D (958 elements, 570 nodes)
items:
  0:  Surface elevation <Surface Elevation> (meter)
  1:  Wind speed <Wind speed> (meter per sec)
  1. Create a copy of the DataArray
ws2 = ds.Wind_speed.copy()
ws2.plot.hist();

  1. Make the modifications, in this case we will clip the values to the interval 1-18 m/s.
import numpy as np
ws2.values = np.clip(ws2.to_numpy(), 1,18)
ws2.plot.hist();

  1. Assign it to a new name in the dataset
ds["Wind_speed_clipped"] = ws2
ds
<mikeio.Dataset>
title: Area Series
dims: (time:67, element:958)
time: 2017-10-27 00:00:00 - 2017-10-29 18:00:00 (67 records)
geometry: Dfsu2D (958 elements, 570 nodes)
items:
  0:  Surface elevation <Surface Elevation> (meter)
  1:  Wind speed <Wind speed> (meter per sec)
  2:  Wind_speed_clipped <Wind speed> (meter per sec)
  1. Reorder items if necessary (See selecting items above)
ds2 = ds[["Wind_speed_clipped", "Surface elevation", "Wind speed"]]
ds2
<mikeio.Dataset>
title: Area Series
dims: (time:67, element:958)
time: 2017-10-27 00:00:00 - 2017-10-29 18:00:00 (67 records)
geometry: Dfsu2D (958 elements, 570 nodes)
items:
  0:  Wind_speed_clipped <Wind speed> (meter per sec)
  1:  Surface elevation <Surface Elevation> (meter)
  2:  Wind speed <Wind speed> (meter per sec)
  1. Write the new dataset to a new file
ds2.to_dfs("modified.dfsu")

Combining Datasets

Two Datasets can be joined either along the time axis or along the item axis.

Dataset.concat() joins along time. Where the two overlap, the later Dataset wins by default; pass keep="first" to prefer the earlier one.

ds_a = mikeio.read("../data/HD2D.dfsu", time=[0, 1])
ds_b = mikeio.read("../data/HD2D.dfsu", time=[1, 2, 3])

ds_concat = mikeio.Dataset.concat([ds_a, ds_b])
ds_concat.time
DatetimeIndex(['1985-08-06 07:00:00', '1985-08-06 09:30:00',
               '1985-08-06 12:00:00', '1985-08-06 14:30:00'],
              dtype='datetime64[s]', freq='150min')

The two inputs have 2 and 3 time steps, and the result has 4 rather than 5, because the step they share is not repeated.

Dataset.merge() joins along items instead, for combining different quantities that share a time axis and geometry.

ds_surf = mikeio.read("../data/HD2D.dfsu", items=[0])
ds_uvel = mikeio.read("../data/HD2D.dfsu", items=[1])

mikeio.Dataset.merge([ds_surf, ds_uvel]).names
['Surface elevation', 'U velocity']

To join files rather than in-memory Datasets, use generic.concat() instead — it streams through the files and never holds them all in memory. See Generic.

Properties

The Dataset (and DataArray) has several properties:

  • n_items - Number of items
  • n_timesteps - Number of timesteps
  • start_time - First time instance (as datetime)
  • end_time - Last time instance (as datetime)
  • is_equidistant - Is the time series equidistant in time
  • timestep - Time step in seconds (if is_equidistant)
  • shape - Shape of each item
  • deletevalue - File delete value (NaN value)
  • custom_blocks - Custom blocks of the dfs file header (see Dfs2)

Methods

Dataset (and DataArray) has several useful methods for working with data, including different ways of selecting data:

  • sel() - Select subset along an axis
  • isel() - Select subset along an axis with an integer

Aggregations along an axis:

  • mean() - Mean value along an axis
  • nanmean() - Mean value along an axis (NaN removed)
  • max() - Max value along an axis
  • nanmax() - Max value along an axis (NaN removed)
  • min() - Min value along an axis
  • nanmin() - Min value along an axis (NaN removed)
  • average() - Compute the weighted average along the specified axis.
  • aggregate() - Aggregate along an axis
  • quantile() - Quantiles along an axis
  • nanquantile() - Quantiles along an axis (NaN ignored)

Mathematical operations

  • ds + value

  • ds - value

  • ds * value

  • ds / value and between two Datasets (if number of items and shapes conform):

  • ds1 + ds2

  • ds1 - ds2

  • ds1 * ds2

  • ds1 / ds2

Other methods that also return a Dataset:

Conversion: