metamet/README.Rmd at main · NERC-CEH/metamet · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
---
output: github_document
---

# metamet <a href="https://nerc-ceh.github.io/metamet/"><img src="man/figures/logo.png" align="right" height="139" alt="metamet website" /></a>

<!-- README.md is generated from README.Rmd. Please edit that file -->

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  fig.path = "man/figures/README-",
  out.width = "100%"
)
```

```{r, eval = FALSE, include = FALSE}
rmarkdown::render("./README.Rmd")
```

# metamet

<!-- badges: start -->
<!-- badges: end -->

`metamet` is an R package which attempts to solve many of the problems
encountered in working with meteorological observation data.
It provide a system for:

- standardising metadata
- converting between file formats
- converting between variable naming conventions
- converting units
- automating QA/QC
- facilitating manual QA/QC via a shiny app
- imputing missing values or "gap-filling"

It does this by defining:

1. a standardised generic data structure with enough complexity to hold both the
observational data and the metadata, including site-specific, variable-specific
and individual record-specific metadata; and
2. methods/functions for converting data between formats, combining data from
different sources, quality control and gap-filling.

## Installation

You can install the development version of metamet from
[GitHub](https://github.com/NERC-CEH/metamet) with:

``` {r, eval = FALSE}
install.packages("pak")
pak::pak("NERC-CEH/metamet")
```

## Example

Basic usage is to first create `metamet` objects from files or pre-existing data
frames or data tables. Because all the metadata describing the observations is
available in the structure, objects can be processed relatively easily so as to:

- rename variables in standard naming conventions
- combine data from different sites
- apply quality control
- impute missing data.

```{r example}
library(metamet)
fname_dt <- testthat::test_path("data-raw/UK-AMO/UK-AMO_BM_dt_2026.csv")
fname_meta <- testthat::test_path("data-raw/dt_meta.xlsx")
fname_site <- testthat::test_path("data-raw/dt_site.csv")

mm <- metamet(
  dt = fname_dt,
  dt_meta = fname_meta,
  dt_site = fname_site,
  site_id = "UK-AMO"
)

# print the outline strucutre:
mm
```

A typical workflow would go on to perform tasks such as adding reference data
from ECMWF ERA5 reanalysis, join with other `metamet` objects, apply quality
control algorithms, impute missing values by various algorithms, and check the
data manually for additional QC. This is illustrated  below.

```{r workflow, eval = FALSE}
mm <- add_era5(
  mm,
  fname_era5 = testthat::test_path("data-raw/dt_era5.csv")
)
mm <- join(mm, mm_old)
mm <- apply_qc(mm)
mm <- impute(mm = mm)
run_shiny()
```


Clearly a two-dimensional data table is not sufficient to hold all the
information. Instead we define a `metamet` data object as a set of related data
tables. We implement this as a list in R, containing five data tables (prefix `dt_`)
explained in the table below.

| Name | Type | Contains | Rows correspond to | Columns correspond to |
|---|---|---|---|---|
| dt | data.table | sensor data | time intervals | variables |
| dt_meta | data.table | variable- and time-specific meta data (like netCDF data attributes) e.g. coords for sensor locations | variables x time period | metadata variables |
| dt_site | data.table | site-specific meta data (like netCDF global attributes) | sites | metadata variables |
| dt_qc | data.table | QC codes | time intervals | variables |
| dt_ref | data.table | ref data e.g. era5 | time intervals | variables |