Point observations

In order to skill assess our model, we need observational data. The data may come from a file or web api. We will cover the following situations here:

• File

  • dfs0

  • csv/excel

  • NetCDF

• REST API

modelskill has the class PointObservation for working with point observations.

Point observations consist of time-value-pairs (data) and meta data such as

• data type (e.g. water level)

• unit (e.g. meter)

• position (coordinates + coordinate reference system)

ModelSkill ll ll is agnostic to the coordinate reference system (CRS) and it is therefore the responsibility of the user to make sure that all data (observations and model) use the same CRS.

import modelskill as ms

Observations from dfs0

Dfs0 files are obviously a very common container format for point observation data at DHI. Besides the data (time-value-pairs), it typically contains meta data information about the data type (EUM type, e.g. water level) and data unit (EUMUnit, e.g. meter). It can potentially store geographical information too, but this is rarely the case. We typically need to provide this information ourselves.

fn = 'data/SW/HKNA_Hm0.dfs0'
pos = (4.2420, 52.6887)   # LONG/LAT

The PointObservation class

The PointObservation class can be instantiated with a dfs0 file name, the item, the position and optionally also a user-defined name.

PointObservation has basic properties like start_time, end_time, n_points, name, etc

PointObservation has two simple plot methods:

• plot() - a time series plot

• hist() - a histogram

Note that the PointObservation object takes a single item only.

o1 = ms.PointObservation(fn, item=0, x=pos[0], y=pos[1], name="HKNA_from_dfs0")
o1

<PointObservation>: HKNA_from_dfs0
Location: 4.242, 52.6887
Time: 2017-10-26 00:10:00 - 2017-10-29 23:50:00
Quantity: Significant wave height [m]

o1.time[0], o1.time[-1]

(Timestamp('2017-10-26 00:10:00'), Timestamp('2017-10-29 23:50:00'))

o1.n_points

564

o1.quantity

Quantity(name='Significant wave height', unit='m')

o1.plot.hist(bins=20);

o1.plot(figsize=(12,6));

Observations from csv/excel

Pandas is our friend. FMSkill PointObservation can be initialized with either a dfs0 or a pandas DataFrame. Hence, for other file types than dfs0 the workflow is to first create a DataFrame with the data e.g. using pd.read_csv() or pd.read_excel().

You need to provide the position and preferably also EUM info (for nice plotting).

import pandas as pd
df = pd.read_csv('data/SW/HKNA_Hm0.csv', index_col=0, parse_dates=True)   # make sure index is DateTimeIndex!
df.head()

	Hm0
time
2017-10-26 00:10:00	0.76172
2017-10-26 00:20:00	0.74219
2017-10-26 00:30:00	0.76172
2017-10-26 00:40:00	0.74219
2017-10-26 00:50:00	0.72266

df.plot();

o2 = ms.PointObservation(df, item="Hm0", x=pos[0], y=pos[1],
                          name='Hm0_from_csv',
                          quantity=ms.Quantity('Significant wave height', 'm'))

o2.plot();

Observations from NetCDF

A NetCDF file is best handled with xarray. It often contains meta data that you can use for constructing your point observation object.

import xarray as xr
ds = xr.open_dataset('data/SW/Europlatform2.nc')
ds

<xarray.Dataset> Size: 251kB
Dimensions:      (TIME: 1008, DEPTH: 2, POSITION: 4464, LATITUDE: 4464,
                  LONGITUDE: 4464)
Coordinates:
  * TIME         (TIME) datetime64[ns] 8kB 2017-10-25 ... 2017-10-31T23:50:00
  * LATITUDE     (LATITUDE) float32 18kB 52.0 52.0 52.0 52.0 ... 52.0 52.0 52.0
  * LONGITUDE    (LONGITUDE) float32 18kB 3.276 3.276 3.276 ... 3.276 3.276
Dimensions without coordinates: DEPTH, POSITION
Data variables: (12/18)
    TIME_QC      (TIME) float32 4kB ...
    DEPH         (TIME, DEPTH) float32 8kB ...
    DEPH_QC      (TIME, DEPTH) float32 8kB ...
    POSITION_QC  (POSITION) float32 18kB ...
    VAVH         (TIME, DEPTH) float64 16kB ...
    VAVH_QC      (TIME, DEPTH) float32 8kB ...
    ...           ...
    VTZA         (TIME, DEPTH) float64 16kB ...
    VTZA_QC      (TIME, DEPTH) float32 8kB ...
    TEMP         (TIME, DEPTH) float64 16kB ...
    TEMP_QC      (TIME, DEPTH) float32 8kB ...
    SWHT         (TIME, DEPTH) float64 16kB ...
    SWHT_QC      (TIME, DEPTH) float32 8kB ...
Attributes: (12/51)
    data_type:                      OceanSITES time-series data
    platform_code:                  Europlatform2
    platform_name:                  
    institution:                    Rijkswaterstaat Water Traffic and Environ...
    institution_edmo_code:          1526
    wmo_platform_code:              
    ...                             ...
    data_assembly_center:           BSH
    wmo_inst_type:                  
    id:                             NO_TS_MO_Europlatform2_201710
    title:                          NWS - NRT in situ Observations
    history:                        2020-01-22T12:14:28Z : Creation; 2020-11-...
    date_update:                    2021-02-26T13:39:19Z

xarray.Dataset

• Dimensions:
  • TIME: 1008
  • DEPTH: 2
  • POSITION: 4464
  • LATITUDE: 4464
  • LONGITUDE: 4464
• Coordinates: (3)
  • TIME
    (TIME)
    datetime64[ns]
    2017-10-25 ... 2017-10-31T23:50:00

    long_name :

    Time

    standard_name :

    time

    valid_min :

    -90000.0

    valid_max :

    90000.0

    QC_indicator :

    0

    uncertainty :

    comment :

    axis :

    T

    ancillary_variables :

    TIME_QC

    array(['2017-10-25T00:00:00.000000000', '2017-10-25T00:10:00.000000000',
           '2017-10-25T00:20:00.000000256', ..., '2017-10-31T23:30:00.000000000',
           '2017-10-31T23:39:59.999999744', '2017-10-31T23:50:00.000000000'],
          shape=(1008,), dtype='datetime64[ns]')

  • LATITUDE
    (LATITUDE)
    float32
    52.0 52.0 52.0 ... 52.0 52.0 52.0

    long_name :

    Latitude of each location

    standard_name :

    latitude

    units :

    degree_north

    valid_min :

    -90.0

    valid_max :

    90.0

    QC_indicator :

    0

    uncertainty :

    comment :

    axis :

    Y

    ancillary_variables :

    POSITION_QC

    array([51.99861, 51.99861, 51.99861, ..., 51.99861, 51.99861, 51.99861],
          shape=(4464,), dtype=float32)

  • LONGITUDE
    (LONGITUDE)
    float32
    3.276 3.276 3.276 ... 3.276 3.276

    long_name :

    Longitude of each location

    standard_name :

    longitude

    units :

    degree_east

    valid_min :

    -180.0

    valid_max :

    180.0

    QC_indicator :

    0

    uncertainty :

    comment :

    axis :

    X

    ancillary_variables :

    POSITION_QC

    array([3.276389, 3.276389, 3.276389, ..., 3.276389, 3.276389, 3.276389],
          shape=(4464,), dtype=float32)

• Data variables: (18)
  • TIME_QC
    (TIME)
    float32
    ...

    valid_min :

    0

    valid_max :

    9

    flag_values :

    [0 1 2 3 4 5 6 7 8 9]

    flag_meanings :

    no_qc_performed good_data probably_good_data bad_data_that_are_potentially_correctable bad_data value_changed not_used nominal_value interpolated_value missing_value

    long_name :

    Time quality flag

    conventions :

    Copernicus Marine In Situ reference table 2

    [1008 values with dtype=float32]

  • DEPH
    (TIME, DEPTH)
    float32
    ...

    long_name :

    Depth

    standard_name :

    depth

    units :

    m

    positive :

    down

    valid_max :

    12000.0

    QC_indicator :

    0

    QC_procedure :

    0

    uncertainty :

    comment :

    axis :

    Z

    reference :

    sea_level

    ancillary_variables :

    DEPH_QC

    valid_min :

    -12000.0

    data_mode :

    R

    [2016 values with dtype=float32]

  • DEPH_QC
    (TIME, DEPTH)
    float32
    ...

    valid_min :

    0

    valid_max :

    9

    flag_values :

    [0 1 2 3 4 5 6 7 8 9]

    flag_meanings :

    no_qc_performed good_data probably_good_data bad_data_that_are_potentially_correctable bad_data value_changed not_used nominal_value interpolated_value missing_value

    long_name :

    Depth quality flag

    conventions :

    Copernicus Marine In Situ reference table 2

    [2016 values with dtype=float32]

  • POSITION_QC
    (POSITION)
    float32
    ...

    valid_min :

    0

    valid_max :

    9

    flag_values :

    [0 1 2 3 4 5 6 7 8 9]

    flag_meanings :

    no_qc_performed good_data probably_good_data bad_data_that_are_potentially_correctable bad_data value_changed not_used nominal_value interpolated_value missing_value

    long_name :

    Position quality flag

    conventions :

    Copernicus Marine In Situ reference table 2

    [4464 values with dtype=float32]

  • VAVH
    (TIME, DEPTH)
    float64
    ...

    long_name :

    Average height highest 1/3 wave (H1/3)

    standard_name :

    sea_surface_wave_significant_height

    units :

    m

    valid_min :

    1

    valid_max :

    30000

    type_of_analysis :

    zero crossing

    comment :

    uncertainty :

    accuracy :

    precision :

    resolution :

    cell_methods :

    sensor_depth :

    0.0

    sensor_mount :

    sensor_orientation :

    data_mode :

    R

    [2016 values with dtype=float64]

  • VAVH_QC
    (TIME, DEPTH)
    float32
    ...

    valid_min :

    0

    valid_max :

    9

    flag_values :

    [0 1 2 3 4 5 6 7 8 9]

    flag_meanings :

    no_qc_performed good_data probably_good_data bad_data_that_are_potentially_correctable bad_data value_changed not_used nominal_value interpolated_value missing_value

    long_name :

    Average height highest 1/3 wave (H1/3) quality flag

    conventions :

    Copernicus Marine In Situ reference table 2

    [2016 values with dtype=float32]

  • VHM0
    (TIME, DEPTH)
    float64
    ...

    long_name :

    Spectral significant wave height (Hm0)

    standard_name :

    sea_surface_wave_significant_height

    units :

    m

    valid_min :

    1

    valid_max :

    30000

    type_of_analysis :

    spectral analysis

    comment :

    uncertainty :

    accuracy :

    precision :

    resolution :

    cell_methods :

    sensor_depth :

    0.0

    sensor_mount :

    sensor_orientation :

    data_mode :

    R

    [2016 values with dtype=float64]

  • VHM0_QC
    (TIME, DEPTH)
    float32
    ...

    valid_min :

    0

    valid_max :

    9

    flag_values :

    [0 1 2 3 4 5 6 7 8 9]

    flag_meanings :

    no_qc_performed good_data probably_good_data bad_data_that_are_potentially_correctable bad_data value_changed not_used nominal_value interpolated_value missing_value

    long_name :

    Spectral significant wave height (Hm0) quality flag

    conventions :

    Copernicus Marine In Situ reference table 2

    [2016 values with dtype=float32]

  • VAVT
    (TIME, DEPTH)
    float64
    ...

    long_name :

    Average period highest 1/3 wave (T1/3)

    standard_name :

    sea_surface_wave_significant_period

    units :

    s

    valid_min :

    1

    valid_max :

    50000

    type_of_analysis :

    zero crossing

    comment :

    uncertainty :

    accuracy :

    precision :

    resolution :

    cell_methods :

    sensor_depth :

    0.0

    sensor_mount :

    sensor_orientation :

    data_mode :

    R

    [2016 values with dtype=float64]

  • VAVT_QC
    (TIME, DEPTH)
    float32
    ...

    valid_min :

    0

    valid_max :

    9

    flag_values :

    [0 1 2 3 4 5 6 7 8 9]

    flag_meanings :

    no_qc_performed good_data probably_good_data bad_data_that_are_potentially_correctable bad_data value_changed not_used nominal_value interpolated_value missing_value

    long_name :

    Average period highest 1/3 wave (T1/3) quality flag

    conventions :

    Copernicus Marine In Situ reference table 2

    [2016 values with dtype=float32]

  • VTPK
    (TIME, DEPTH)
    float64
    ...

    long_name :

    Wave period at spectral peak / peak period (Tp)

    standard_name :

    sea_surface_wave_period_at_variance_spectral_density_maximum

    units :

    s

    valid_min :

    1

    valid_max :

    50000

    type_of_analysis :

    spectral analysis

    comment :

    uncertainty :

    accuracy :

    precision :

    resolution :

    cell_methods :

    sensor_depth :

    0.0

    sensor_mount :

    sensor_orientation :

    data_mode :

    R

    [2016 values with dtype=float64]

  • VTPK_QC
    (TIME, DEPTH)
    float32
    ...

    valid_min :

    0

    valid_max :

    9

    flag_values :

    [0 1 2 3 4 5 6 7 8 9]

    flag_meanings :

    no_qc_performed good_data probably_good_data bad_data_that_are_potentially_correctable bad_data value_changed not_used nominal_value interpolated_value missing_value

    long_name :

    Wave period at spectral peak / peak period (Tp) quality flag

    conventions :

    Copernicus Marine In Situ reference table 2

    [2016 values with dtype=float32]

  • VTZA
    (TIME, DEPTH)
    float64
    ...

    long_name :

    Average zero crossing wave period (Tz)

    standard_name :

    sea_surface_wave_mean_period

    units :

    s

    valid_min :

    1

    valid_max :

    50000

    type_of_analysis :

    zero crossing

    comment :

    uncertainty :

    accuracy :

    precision :

    resolution :

    cell_methods :

    sensor_depth :

    0.0

    sensor_mount :

    sensor_orientation :

    data_mode :

    R

    [2016 values with dtype=float64]

  • VTZA_QC
    (TIME, DEPTH)
    float32
    ...

    valid_min :

    0

    valid_max :

    9

    flag_values :

    [0 1 2 3 4 5 6 7 8 9]

    flag_meanings :

    no_qc_performed good_data probably_good_data bad_data_that_are_potentially_correctable bad_data value_changed not_used nominal_value interpolated_value missing_value

    long_name :

    Average zero crossing wave period (Tz) quality flag

    conventions :

    Copernicus Marine In Situ reference table 2

    [2016 values with dtype=float32]

  • TEMP
    (TIME, DEPTH)
    float64
    ...

    long_name :

    Sea temperature

    standard_name :

    sea_water_temperature

    units :

    degrees_C

    valid_min :

    -2000

    valid_max :

    32000

    comment :

    uncertainty :

    accuracy :

    precision :

    resolution :

    cell_methods :

    sensor_depth :

    0.0

    sensor_mount :

    sensor_orientation :

    data_mode :

    R

    [2016 values with dtype=float64]

  • TEMP_QC
    (TIME, DEPTH)
    float32
    ...

    valid_min :

    0

    valid_max :

    9

    flag_values :

    [0 1 2 3 4 5 6 7 8 9]

    flag_meanings :

    no_qc_performed good_data probably_good_data bad_data_that_are_potentially_correctable bad_data value_changed not_used nominal_value interpolated_value missing_value

    long_name :

    Sea temperature quality flag

    conventions :

    Copernicus Marine In Situ reference table 2

    [2016 values with dtype=float32]

  • SWHT
    (TIME, DEPTH)
    float64
    ...

    long_name :

    Swell Height

    standard_name :

    sea_surface_swell_wave_significant_height

    units :

    m

    valid_min :

    1

    valid_max :

    10000

    comment :

    uncertainty :

    accuracy :

    precision :

    resolution :

    cell_methods :

    sensor_depth :

    0.0

    sensor_mount :

    sensor_orientation :

    data_mode :

    R

    [2016 values with dtype=float64]

  • SWHT_QC
    (TIME, DEPTH)
    float32
    ...

    valid_min :

    0

    valid_max :

    9

    flag_values :

    [0 1 2 3 4 5 6 7 8 9]

    flag_meanings :

    no_qc_performed good_data probably_good_data bad_data_that_are_potentially_correctable bad_data value_changed not_used nominal_value interpolated_value missing_value

    long_name :

    Swell Height quality flag

    conventions :

    Copernicus Marine In Situ reference table 2

    [2016 values with dtype=float32]

• Attributes: (51)

  data_type :

  OceanSITES time-series data

  platform_code :

  Europlatform2

  platform_name :

  institution :

  Rijkswaterstaat Water Traffic and Environment

  institution_edmo_code :

  1526

  wmo_platform_code :

  source :

  mooring

  source_platform_category_code :

  48

  data_mode :

  R

  quality_control_indicator :

  1

  quality_index :

  A

  netcdf_version :

  netCDF-4 classic model

  summary :

  Oceanographic data from North West Shelf

  area :

  North West Shelf

  geospatial_lat_min :

  51.998611

  geospatial_lat_max :

  51.998611

  geospatial_lon_min :

  3.276389

  geospatial_lon_max :

  3.276389

  geospatial_vertical_min :

  0

  geospatial_vertical_max :

  1

  family_label :

  mooring

  family_code :

  MO

  time_coverage_start :

  2017-10-01T00:00:00Z

  time_coverage_end :

  2017-10-31T23:59:59Z

  institution_references :

  http://www.rijkswaterstaat.nl/en/about_us/our_organization/

  contact :

  cmems-service@bsh.de

  author :

  cmems-service

  pi_name :

  matroos@rws.nl

  distribution_statement :

  These data follow Copernicus standards; they are public and free of charge. User assumes all risk for use of data. User must display citation in any publication or product using data. User must contact PI prior to any commercial use of data.

  citation :

  These data were collected and made freely available by the Copernicus project and the programs that contribute to it

  qc_manual :

  OceanSITES User's Manual v1.2

  last_date_observation :

  2017-10-31T23:50:00Z

  last_latitude_observation :

  51.998611

  last_longitude_observation :

  3.276389

  ices_platform_code :

  bottom_depth :

  doi :

  site_code :

  comment :

  format_version :

  1.4

  Conventions :

  CF-1.6 Copernicus-InSituTAC-FormatManual-1.41 Copernicus-InSituTAC-SRD-1.5 Copernicus-InSituTAC-ParametersList-3.2.0

  update_interval :

  PT1H

  naming_authority :

  Copernicus Marine In Situ

  cdm_data_type :

  timeSeries

  references :

  http://marine.copernicus.eu http://www.marineinsitu.eu

  data_assembly_center :

  BSH

  wmo_inst_type :

  id :

  NO_TS_MO_Europlatform2_201710

  title :

  NWS - NRT in situ Observations

  history :

  2020-01-22T12:14:28Z : Creation; 2020-11-26T13:29:56Z, BSH North West Shelf PU: File updated to comply with Copernicus Marine In Situ NetCDF Format Manual Version 1.41; 2021-02-26T13:39:19Z, BSH North West Shelf PU: Title has been modified to comply with PIT v5.6.0.

  date_update :

  2021-02-26T13:39:19Z

pos = ds.LONGITUDE.values[0], ds.LATITUDE.values[0]
pos

(np.float32(3.276389), np.float32(51.99861))

ds.VHM0.isel(DEPTH=0)

<xarray.DataArray 'VHM0' (TIME: 1008)> Size: 8kB
[1008 values with dtype=float64]
Coordinates:
  * TIME     (TIME) datetime64[ns] 8kB 2017-10-25 ... 2017-10-31T23:50:00
Attributes: (12/16)
    long_name:           Spectral significant wave height (Hm0)
    standard_name:       sea_surface_wave_significant_height
    units:               m
    valid_min:           1
    valid_max:           30000
    type_of_analysis:    spectral analysis
    ...                  ...
    resolution:           
    cell_methods:         
    sensor_depth:        0.0
    sensor_mount:         
    sensor_orientation:   
    data_mode:           R

xarray.DataArray

'VHM0'

• TIME: 1008

• ...

  [1008 values with dtype=float64]

• Coordinates: (1)
  • TIME
    (TIME)
    datetime64[ns]
    2017-10-25 ... 2017-10-31T23:50:00

    long_name :

    Time

    standard_name :

    time

    valid_min :

    -90000.0

    valid_max :

    90000.0

    QC_indicator :

    0

    uncertainty :

    comment :

    axis :

    T

    ancillary_variables :

    TIME_QC

    array(['2017-10-25T00:00:00.000000000', '2017-10-25T00:10:00.000000000',
           '2017-10-25T00:20:00.000000256', ..., '2017-10-31T23:30:00.000000000',
           '2017-10-31T23:39:59.999999744', '2017-10-31T23:50:00.000000000'],
          shape=(1008,), dtype='datetime64[ns]')

• Attributes: (16)

  long_name :

  Spectral significant wave height (Hm0)

  standard_name :

  sea_surface_wave_significant_height

  units :

  m

  valid_min :

  1

  valid_max :

  30000

  type_of_analysis :

  spectral analysis

  comment :

  uncertainty :

  accuracy :

  precision :

  resolution :

  cell_methods :

  sensor_depth :

  0.0

  sensor_mount :

  sensor_orientation :

  data_mode :

  R

ds.VHM0.isel(DEPTH=0).plot();

df = ds.VHM0.isel(DEPTH=0).to_dataframe()
df.index = df.index.strftime('%Y-%m-%d %H:%M:%S')
df.head()

	VHM0
TIME
2017-10-25 00:00:00	1.68
2017-10-25 00:10:00	1.79
2017-10-25 00:20:00	1.84
2017-10-25 00:30:00	1.84
2017-10-25 00:40:00	1.76

o3 = ms.PointObservation(ds.VHM0.isel(DEPTH=0), x=pos[0], y=pos[1], name='Hm0_from_nc')

o3.plot();