Final assignment

The final exercise involves converting data from one or more providers.

Since this exercise is designed to prepare you for real project work, the information you need to solve it might be slightly incomplete or not provided in context. Use your best judgment!

Parts of this assignment can be solved in several ways. Use descriptive variable names and comments or descriptive text if necessary to clarify. The final solution should be clear to your colleagues and will be shared with some of your fellow students for review.

The data will be used for MIKE modelling and must be converted to Dfs with apppropriate EUM types/units in order to be used by the MIKE software.

The data is provided as a zip file and a NetCDF file (in the data folder - see FA.3 below).

Inside the zip file, there are a many timeseries (ASCII format) of discharge data from streams located across several regions (*.dat).

Static data for each region is found in a separate file (region_info.csv)

Pandas read_csv is very powerful, but here are a few things to keep in mind

• Column separator e.g. comma (,)

• Blank lines

• Comments

• Missing values

• Date format

The MIKE engine can not handle missing values / delete values, fill in missing values with interpolated values.

In order to save diskspace, crop the timeseries to simulation period Feb 1 - June 30.

FA.1 Convert all timeseries to Dfs0

import os
import numpy as np
import pandas as pd
import mikeio

from mikeio import Dataset
from mikeio.eum import EUMType, ItemInfo, EUMUnit

# This is one way to find and filter filenames in a directory
# [x for x in os.listdir("datafolder") if "some_str" in x]

# This is useful!
# help(pd.read_csv)

# example of reading csv
# df = pd.read_csv("../data/oceandata.csv", comment='#', index_col=0, sep=',', parse_dates=True)

a) Convert all timeseries to dfs0 (remember that the notebook should be runnable for your peers so put the files somewhere reasonable).

b) Read s15_east_novayork_river.dfs0, print the “header”, plot, and show that the number of missing values is 0.

FA.2 Add region specific info to normalize timeseries with surface area

Each timeseries belongs to a region identified in the filename, e.g. s15_east_novayork_river.dat is located in the novayork region.

a) Convert all timeseries to dfs0 with specific discharge, by doing:

For each timeseries in the dataset:

• Find out which region it belongs to (hint: the string method split() will be useful)

• Divide the timeseries values with the surface area for the region (take into account units)

• Create a dfs0 file with specific discharge (discharge / area) (like the one with discharge from FA.1)

b) Determine which station has the largest max specific discharge (in the simulation period).

FA.3 Gridded data

ERA5 wave model for a part of the North Sea is available in the file "../data/ERA5_DutchCoast.nc"

The dataset is provided in NetCDF

These are the variables of interest:

• Significant wave height (Significant height of combined wind waves and swell)

• Mean wave direction

Info about spatial and temporal axes can be found in the NetCDF file

import xarray as xr
import mikeio
import numpy as np

ds = xr.open_dataset("../data/ERA5_DutchCoast.nc")

ds

<xarray.Dataset> Size: 590kB
Dimensions:    (longitude: 20, latitude: 11, time: 67)
Coordinates:
  * longitude  (longitude) float32 80B -1.0 -0.5 0.0 0.5 1.0 ... 7.0 7.5 8.0 8.5
  * latitude   (latitude) float32 44B 55.0 54.5 54.0 53.5 ... 51.0 50.5 50.0
  * time       (time) datetime64[ns] 536B 2017-10-27 ... 2017-10-29T18:00:00
Data variables:
    mwd        (time, latitude, longitude) float64 118kB ...
    mwp        (time, latitude, longitude) float64 118kB ...
    mp2        (time, latitude, longitude) float64 118kB ...
    pp1d       (time, latitude, longitude) float64 118kB ...
    swh        (time, latitude, longitude) float64 118kB ...
Attributes:
    Conventions:  CF-1.6
    history:      2021-06-07 12:25:02 GMT by grib_to_netcdf-2.16.0: /opt/ecmw...

xarray.Dataset

• Dimensions:
  • longitude: 20
  • latitude: 11
  • time: 67
• Coordinates: (3)
  • longitude
    (longitude)
    float32
    -1.0 -0.5 0.0 0.5 ... 7.5 8.0 8.5

    units :

    degrees_east

    long_name :

    longitude

    array([-1. , -0.5,  0. ,  0.5,  1. ,  1.5,  2. ,  2.5,  3. ,  3.5,  4. ,  4.5,
            5. ,  5.5,  6. ,  6.5,  7. ,  7.5,  8. ,  8.5], dtype=float32)

  • latitude
    (latitude)
    float32
    55.0 54.5 54.0 ... 51.0 50.5 50.0

    units :

    degrees_north

    long_name :

    latitude

    array([55. , 54.5, 54. , 53.5, 53. , 52.5, 52. , 51.5, 51. , 50.5, 50. ],
          dtype=float32)

  • time
    (time)
    datetime64[ns]
    2017-10-27 ... 2017-10-29T18:00:00

    long_name :

    time

    array(['2017-10-27T00:00:00.000000000', '2017-10-27T01:00:00.000000000',
           '2017-10-27T02:00:00.000000000', '2017-10-27T03:00:00.000000000',
           '2017-10-27T04:00:00.000000000', '2017-10-27T05:00:00.000000000',
           '2017-10-27T06:00:00.000000000', '2017-10-27T07:00:00.000000000',
           '2017-10-27T08:00:00.000000000', '2017-10-27T09:00:00.000000000',
           '2017-10-27T10:00:00.000000000', '2017-10-27T11:00:00.000000000',
           '2017-10-27T12:00:00.000000000', '2017-10-27T13:00:00.000000000',
           '2017-10-27T14:00:00.000000000', '2017-10-27T15:00:00.000000000',
           '2017-10-27T16:00:00.000000000', '2017-10-27T17:00:00.000000000',
           '2017-10-27T18:00:00.000000000', '2017-10-27T19:00:00.000000000',
           '2017-10-27T20:00:00.000000000', '2017-10-27T21:00:00.000000000',
           '2017-10-27T22:00:00.000000000', '2017-10-27T23:00:00.000000000',
           '2017-10-28T00:00:00.000000000', '2017-10-28T01:00:00.000000000',
           '2017-10-28T02:00:00.000000000', '2017-10-28T03:00:00.000000000',
           '2017-10-28T04:00:00.000000000', '2017-10-28T05:00:00.000000000',
           '2017-10-28T06:00:00.000000000', '2017-10-28T07:00:00.000000000',
           '2017-10-28T08:00:00.000000000', '2017-10-28T09:00:00.000000000',
           '2017-10-28T10:00:00.000000000', '2017-10-28T11:00:00.000000000',
           '2017-10-28T12:00:00.000000000', '2017-10-28T13:00:00.000000000',
           '2017-10-28T14:00:00.000000000', '2017-10-28T15:00:00.000000000',
           '2017-10-28T16:00:00.000000000', '2017-10-28T17:00:00.000000000',
           '2017-10-28T18:00:00.000000000', '2017-10-28T19:00:00.000000000',
           '2017-10-28T20:00:00.000000000', '2017-10-28T21:00:00.000000000',
           '2017-10-28T22:00:00.000000000', '2017-10-28T23:00:00.000000000',
           '2017-10-29T00:00:00.000000000', '2017-10-29T01:00:00.000000000',
           '2017-10-29T02:00:00.000000000', '2017-10-29T03:00:00.000000000',
           '2017-10-29T04:00:00.000000000', '2017-10-29T05:00:00.000000000',
           '2017-10-29T06:00:00.000000000', '2017-10-29T07:00:00.000000000',
           '2017-10-29T08:00:00.000000000', '2017-10-29T09:00:00.000000000',
           '2017-10-29T10:00:00.000000000', '2017-10-29T11:00:00.000000000',
           '2017-10-29T12:00:00.000000000', '2017-10-29T13:00:00.000000000',
           '2017-10-29T14:00:00.000000000', '2017-10-29T15:00:00.000000000',
           '2017-10-29T16:00:00.000000000', '2017-10-29T17:00:00.000000000',
           '2017-10-29T18:00:00.000000000'], dtype='datetime64[ns]')

• Data variables: (5)
  • mwd
    (time, latitude, longitude)
    float64
    ...

    units :

    Degree true

    long_name :

    Mean wave direction

    [14740 values with dtype=float64]

  • mwp
    (time, latitude, longitude)
    float64
    ...

    units :

    s

    long_name :

    Mean wave period

    [14740 values with dtype=float64]

  • mp2
    (time, latitude, longitude)
    float64
    ...

    units :

    s

    long_name :

    Mean zero-crossing wave period

    [14740 values with dtype=float64]

  • pp1d
    (time, latitude, longitude)
    float64
    ...

    units :

    s

    long_name :

    Peak wave period

    [14740 values with dtype=float64]

  • swh
    (time, latitude, longitude)
    float64
    ...

    units :

    m

    long_name :

    Significant height of combined wind waves and swell

    [14740 values with dtype=float64]

• Indexes: (3)
  • longitude
    PandasIndex

    PandasIndex(Index([-1.0, -0.5,  0.0,  0.5,  1.0,  1.5,  2.0,  2.5,  3.0,  3.5,  4.0,  4.5,
            5.0,  5.5,  6.0,  6.5,  7.0,  7.5,  8.0,  8.5],
          dtype='float32', name='longitude'))

  • latitude
    PandasIndex

    PandasIndex(Index([55.0, 54.5, 54.0, 53.5, 53.0, 52.5, 52.0, 51.5, 51.0, 50.5, 50.0], dtype='float32', name='latitude'))

  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['2017-10-27 00:00:00', '2017-10-27 01:00:00',
                   '2017-10-27 02:00:00', '2017-10-27 03:00:00',
                   '2017-10-27 04:00:00', '2017-10-27 05:00:00',
                   '2017-10-27 06:00:00', '2017-10-27 07:00:00',
                   '2017-10-27 08:00:00', '2017-10-27 09:00:00',
                   '2017-10-27 10:00:00', '2017-10-27 11:00:00',
                   '2017-10-27 12:00:00', '2017-10-27 13:00:00',
                   '2017-10-27 14:00:00', '2017-10-27 15:00:00',
                   '2017-10-27 16:00:00', '2017-10-27 17:00:00',
                   '2017-10-27 18:00:00', '2017-10-27 19:00:00',
                   '2017-10-27 20:00:00', '2017-10-27 21:00:00',
                   '2017-10-27 22:00:00', '2017-10-27 23:00:00',
                   '2017-10-28 00:00:00', '2017-10-28 01:00:00',
                   '2017-10-28 02:00:00', '2017-10-28 03:00:00',
                   '2017-10-28 04:00:00', '2017-10-28 05:00:00',
                   '2017-10-28 06:00:00', '2017-10-28 07:00:00',
                   '2017-10-28 08:00:00', '2017-10-28 09:00:00',
                   '2017-10-28 10:00:00', '2017-10-28 11:00:00',
                   '2017-10-28 12:00:00', '2017-10-28 13:00:00',
                   '2017-10-28 14:00:00', '2017-10-28 15:00:00',
                   '2017-10-28 16:00:00', '2017-10-28 17:00:00',
                   '2017-10-28 18:00:00', '2017-10-28 19:00:00',
                   '2017-10-28 20:00:00', '2017-10-28 21:00:00',
                   '2017-10-28 22:00:00', '2017-10-28 23:00:00',
                   '2017-10-29 00:00:00', '2017-10-29 01:00:00',
                   '2017-10-29 02:00:00', '2017-10-29 03:00:00',
                   '2017-10-29 04:00:00', '2017-10-29 05:00:00',
                   '2017-10-29 06:00:00', '2017-10-29 07:00:00',
                   '2017-10-29 08:00:00', '2017-10-29 09:00:00',
                   '2017-10-29 10:00:00', '2017-10-29 11:00:00',
                   '2017-10-29 12:00:00', '2017-10-29 13:00:00',
                   '2017-10-29 14:00:00', '2017-10-29 15:00:00',
                   '2017-10-29 16:00:00', '2017-10-29 17:00:00',
                   '2017-10-29 18:00:00'],
                  dtype='datetime64[ns]', name='time', freq=None))

• Attributes: (2)

  Conventions :

  CF-1.6

  history :

  2021-06-07 12:25:02 GMT by grib_to_netcdf-2.16.0: /opt/ecmwf/eccodes/bin/grib_to_netcdf -S param -o /cache/data8/adaptor.mars.internal-1623068689.815175-27387-9-f370172c-9b84-4760-b5e5-32e00e3964bd.nc /cache/tmp/f370172c-9b84-4760-b5e5-32e00e3964bd-adaptor.mars.internal-1623068689.81571-27387-2-tmp.grib

lons = ds.longitude
lats_rev = ds.latitude[::-1] # reversed

# geometry = mikeio.Grid2D(x=...,y=..., projection="LONG/LAT")
#hm0 = mikeio.DataArray(np.flip(...,axis=1),geometry=geometry, time=ds.time, item=mikeio.ItemInfo(...)))

# mds = mikeio.Dataset([...])

a) Find the relevant variables in the NetCDF file.

b) Create MIKE IO DataArrays for each variable

c) Combine the arrays into a MIKE IO Dataset and write it to a dfs2 file

d) Read the data from the file and document that the temporal max at (50N, 1E) of the significant wave height is the same in the dfs2 file as in the NetCDF file (note: due to rounding errors the may not be exactly the same)

The file should look like this in MIKE Zero:

Submission of solution

Your solution to the above tasks is to be delivered in the format of a single Jupyter notebook file. Please create a new and name it final_assignment_xyz.ipynb where xyz is your initials. It should be easy to understand and runnable by your peers.

The solution will be reviewed by some of your fellow students as well as an instructor, which will provide feedback on both the correctnes and clarity of your solution.

The submission and review process is handled by Academy + Eduflow.