Issue: ActiLife output is not identical to the output from `pygt3x.
Root of the issue: There are missing timestamps from the data and inconsistent filling in from gaps (e.g. idle sleep mode).
TAS1H30182785_2019-09-17RAW.csv
Sys.setenv(RETICULATE_PYTHON = "managed")
library(reticulate)
#> Warning: package 'reticulate' was built under R version 4.4.1
library(dplyr)
#>
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#>
#> filter, lag
#> The following objects are masked from 'package:base':
#>
#> intersect, setdiff, setequal, union
library(agcounts)
library(readr)
library(read.gt3x)
options(digits.secs = 3)
# this will install pygt3x on the fly using uv
# see https://posit.co/blog/reticulate-1-41/
py_require("pygt3x")From ActiLife
GT3X -> CSV
path_raw_csv = "TAS1H30182785_2019-09-17RAW.csv"
if (!file.exists(path_raw_csv)) {
curl::curl_download(
"https://github.com/user-attachments/files/21224451/TAS1H30182785_2019-09-17RAW.csv",
path_raw_csv
)
}
truth_raw = readr::read_csv(path_raw_csv, skip = 10)
#> Rows: 240500 Columns: 4
#> ── Column specification ────────────────────────────────────────────────────────
#> Delimiter: ","
#> chr (1): Timestamp
#> dbl (3): Accelerometer X, Accelerometer Y, Accelerometer Z
#>
#> ℹ Use `spec()` to retrieve the full column specification for this data.
#> ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
colnames(truth_raw) = c("time", "X", "Y", "Z")
truth_raw = truth_raw %>%
mutate(
time = lubridate::mdy_hms(time, tz = "UTC"),
all_zero = X == 0 & Y == 0 & Z == 0
)
any(truth_raw$all_zero)
#> [1] TRUE
head(truth_raw)
#> # A tibble: 6 × 5
#> time X Y Z all_zero
#> <dttm> <dbl> <dbl> <dbl> <lgl>
#> 1 2019-09-17 18:40:00.000 0 0.008 0.996 FALSE
#> 2 2019-09-17 18:40:00.009 0.016 0 1.01 FALSE
#> 3 2019-09-17 18:40:00.019 0.02 -0.008 1.00 FALSE
#> 4 2019-09-17 18:40:00.029 0.016 -0.012 1.01 FALSE
#> 5 2019-09-17 18:40:00.039 0.016 -0.008 1.01 FALSE
#> 6 2019-09-17 18:40:00.049 0.008 -0.008 1.01 FALSEGT3X File
path = "TAS1H30182785_2019-09-17.gt3x"
if (!file.exists(path)) {
curl::curl_download(
"https://raw.githubusercontent.com/muschellij2/SummarizedActigraphy/refs/heads/main/inst/extdata/TAS1H30182785_2019-09-17.gt3x",
path
)
}Here we read in the GT3X file, similar to the README example:
https://github.com/actigraph/pygt3x?tab=readme-ov-file#example-usage
`%as%` <- reticulate::`%as%`
Reader <- reticulate::import("pygt3x.reader", convert = FALSE)
pd <- reticulate::import("pandas")
reset_index <- pd$core$frame$DataFrame$reset_index
FileReader <- Reader$FileReader
to_pandas <- FileReader$to_pandas
with(FileReader(path) %as% reader, {
raw = to_pandas(reader)
raw = reset_index(raw)
})Renaming data for consistency
raw = raw %>%
rename(time = Timestamp,
idle_sleep_mode = IdleSleepMode) %>%
mutate(time = as.POSIXct(time, origin = "1970-01-01 00:00:00",
# using UTC to match truth_raw
tz = "UTC")
) %>%
as_tibble()
head(raw)
#> # A tibble: 6 × 5
#> time X Y Z idle_sleep_mode
#> <dttm> <dbl> <dbl> <dbl> <lgl>
#> 1 2019-09-17 18:40:00.000 0 0.00781 0.996 FALSE
#> 2 2019-09-17 18:40:00.009 0.0156 0 1.01 FALSE
#> 3 2019-09-17 18:40:00.019 0.0195 -0.00781 1.00 FALSE
#> 4 2019-09-17 18:40:00.029 0.0156 -0.0117 1.01 FALSE
#> 5 2019-09-17 18:40:00.039 0.0156 -0.00781 1.01 FALSE
#> 6 2019-09-17 18:40:00.049 0.00781 -0.00781 1.01 FALSERounding the data similar to ActiLife
raw = raw %>%
mutate(
# need to round away from 0, not default rounding
# https://github.com/actigraph/GT3X-File-Format/blob/d4aa795a3d06cc8b22b4e3df645b597cdeee6d92/LogRecords/Activity.md?plain=1#L226
X = ncar::Round(X, n = 3),
Y = ncar::Round(Y, n = 3),
Z = ncar::Round(Z, n = 3),
)
head(raw)
#> # A tibble: 6 × 5
#> time X Y Z idle_sleep_mode
#> <dttm> <dbl> <dbl> <dbl> <lgl>
#> 1 2019-09-17 18:40:00.000 0 0.008 0.996 FALSE
#> 2 2019-09-17 18:40:00.009 0.016 0 1.01 FALSE
#> 3 2019-09-17 18:40:00.019 0.02 -0.008 1.00 FALSE
#> 4 2019-09-17 18:40:00.029 0.016 -0.012 1.01 FALSE
#> 5 2019-09-17 18:40:00.039 0.016 -0.008 1.01 FALSE
#> 6 2019-09-17 18:40:00.049 0.008 -0.008 1.01 FALSEComparison
We can compare the raw data from the GT3X file with the CSV file from ActiLife.
nrow(raw)
#> [1] 215200
nrow(truth_raw)
#> [1] 240500
They do not agree.
nrow(raw) == nrow(truth_raw)
#> [1] FALSE
nrow(truth_raw) - nrow(raw)
#> [1] 25300
The ActiLife has more data and more at the end of the file
range(raw$time)
#> [1] "2019-09-17 18:40:00.00 UTC" "2019-09-17 19:15:58.99 UTC"
range(truth_raw$time)
#> [1] "2019-09-17 18:40:00.00 UTC" "2019-09-17 19:20:04.99 UTC"
There are minutes flagged as idle sleep mode in this data, so the missing minutes below are not from this.
any(raw$idle_sleep_mode)
#> [1] TRUE
raw %>%
filter(idle_sleep_mode)
#> # A tibble: 182,200 × 5
#> time X Y Z idle_sleep_mode
#> <dttm> <dbl> <dbl> <dbl> <lgl>
#> 1 2019-09-17 18:40:10.000 0.008 -0.012 1.02 TRUE
#> 2 2019-09-17 18:40:10.009 0.008 -0.012 1.02 TRUE
#> 3 2019-09-17 18:40:10.019 0.008 -0.012 1.02 TRUE
#> 4 2019-09-17 18:40:10.029 0.008 -0.012 1.02 TRUE
#> 5 2019-09-17 18:40:10.039 0.008 -0.012 1.02 TRUE
#> 6 2019-09-17 18:40:10.049 0.008 -0.012 1.02 TRUE
#> 7 2019-09-17 18:40:10.059 0.008 -0.012 1.02 TRUE
#> 8 2019-09-17 18:40:10.069 0.008 -0.012 1.02 TRUE
#> 9 2019-09-17 18:40:10.079 0.008 -0.012 1.02 TRUE
#> 10 2019-09-17 18:40:10.089 0.008 -0.012 1.02 TRUE
#> # ℹ 182,190 more rows
There are missing minutes in the output from pygt3x
Question: Is this to be expected? If so how do we handle these missing minutes?
sample_rate = 100
full_time_df = data.frame(
time = seq(from = min(raw$time), to = max(raw$time), by = 1/sample_rate)
) %>%
as_tibble()
aj = anti_join(full_time_df, raw, by = "time")
head(aj)
#> # A tibble: 6 × 1
#> time
#> <dttm>
#> 1 2019-09-17 19:15:40.000
#> 2 2019-09-17 19:15:40.009
#> 3 2019-09-17 19:15:40.019
#> 4 2019-09-17 19:15:40.029
#> 5 2019-09-17 19:15:40.039
#> 6 2019-09-17 19:15:40.049
nrow(aj)
#> [1] 700
Create these rows in the data
raw = raw %>%
full_join(full_time_df) %>%
arrange(time)
#> Joining with `by = join_by(time)`
raw = raw %>%
tidyr::replace_na(list(X = 0, Y = 0, Z = 0)) %>%
mutate(
all_zero = X == 0 & Y == 0 & Z == 0
)
raw %>%
filter(idle_sleep_mode)
#> # A tibble: 182,200 × 6
#> time X Y Z idle_sleep_mode all_zero
#> <dttm> <dbl> <dbl> <dbl> <lgl> <lgl>
#> 1 2019-09-17 18:40:10.000 0.008 -0.012 1.02 TRUE FALSE
#> 2 2019-09-17 18:40:10.009 0.008 -0.012 1.02 TRUE FALSE
#> 3 2019-09-17 18:40:10.019 0.008 -0.012 1.02 TRUE FALSE
#> 4 2019-09-17 18:40:10.029 0.008 -0.012 1.02 TRUE FALSE
#> 5 2019-09-17 18:40:10.039 0.008 -0.012 1.02 TRUE FALSE
#> 6 2019-09-17 18:40:10.049 0.008 -0.012 1.02 TRUE FALSE
#> 7 2019-09-17 18:40:10.059 0.008 -0.012 1.02 TRUE FALSE
#> 8 2019-09-17 18:40:10.069 0.008 -0.012 1.02 TRUE FALSE
#> 9 2019-09-17 18:40:10.079 0.008 -0.012 1.02 TRUE FALSE
#> 10 2019-09-17 18:40:10.089 0.008 -0.012 1.02 TRUE FALSE
#> # ℹ 182,190 more rowsWe are missing additional data
truth_raw[(nrow(raw)+1):nrow(truth_raw), ]
#> # A tibble: 24,600 × 5
#> time X Y Z all_zero
#> <dttm> <dbl> <dbl> <dbl> <lgl>
#> 1 2019-09-17 19:15:59.000 0 0 0 TRUE
#> 2 2019-09-17 19:15:59.009 0 0 0 TRUE
#> 3 2019-09-17 19:15:59.019 0 0 0 TRUE
#> 4 2019-09-17 19:15:59.029 0 0 0 TRUE
#> 5 2019-09-17 19:15:59.039 0 0 0 TRUE
#> 6 2019-09-17 19:15:59.049 0 0 0 TRUE
#> 7 2019-09-17 19:15:59.059 0 0 0 TRUE
#> 8 2019-09-17 19:15:59.070 0 0 0 TRUE
#> 9 2019-09-17 19:15:59.079 0 0 0 TRUE
#> 10 2019-09-17 19:15:59.090 0 0 0 TRUE
#> # ℹ 24,590 more rows
Compare the data we have
We can at least compare the data we have both on time
# they do not agree
tr = truth_raw[1:nrow(raw), ]
all.equal(tr %>%
select(time, X, Y, Z),
raw %>%
select(time, X, Y, Z),
check.attributes = FALSE,
check.class = FALSE)
#> [1] "Component \"X\": Mean relative difference: 1"
#> [2] "Component \"Y\": Mean relative difference: 1"
#> [3] "Component \"Z\": Mean relative difference: 1"It seems that ActiLife does last observation carried forward (LOCF) as per https://actigraphcorp.my.site.com/support/s/article/Idle-Sleep-Mode-Explained, but it only does it for one second in this data. Thus, if we do LOCF for the ActiLife output and that from pygt3x after merging on the full data sources, we can get the same output.
print(tr[214000:214101,], n = Inf)
#> # A tibble: 102 × 5
#> time X Y Z all_zero
#> <dttm> <dbl> <dbl> <dbl> <lgl>
#> 1 2019-09-17 19:15:39.990 -0.016 -1.03 0.027 FALSE
#> 2 2019-09-17 19:15:40.000 -0.016 -1.03 0.027 FALSE
#> 3 2019-09-17 19:15:40.009 -0.016 -1.03 0.027 FALSE
#> 4 2019-09-17 19:15:40.019 -0.016 -1.03 0.027 FALSE
#> 5 2019-09-17 19:15:40.029 -0.016 -1.03 0.027 FALSE
#> 6 2019-09-17 19:15:40.039 -0.016 -1.03 0.027 FALSE
#> 7 2019-09-17 19:15:40.049 -0.016 -1.03 0.027 FALSE
#> 8 2019-09-17 19:15:40.059 -0.016 -1.03 0.027 FALSE
#> 9 2019-09-17 19:15:40.070 -0.016 -1.03 0.027 FALSE
#> 10 2019-09-17 19:15:40.079 -0.016 -1.03 0.027 FALSE
#> 11 2019-09-17 19:15:40.090 -0.016 -1.03 0.027 FALSE
#> 12 2019-09-17 19:15:40.099 -0.016 -1.03 0.027 FALSE
#> 13 2019-09-17 19:15:40.110 -0.016 -1.03 0.027 FALSE
#> 14 2019-09-17 19:15:40.119 -0.016 -1.03 0.027 FALSE
#> 15 2019-09-17 19:15:40.130 -0.016 -1.03 0.027 FALSE
#> 16 2019-09-17 19:15:40.139 -0.016 -1.03 0.027 FALSE
#> 17 2019-09-17 19:15:40.150 -0.016 -1.03 0.027 FALSE
#> 18 2019-09-17 19:15:40.159 -0.016 -1.03 0.027 FALSE
#> 19 2019-09-17 19:15:40.170 -0.016 -1.03 0.027 FALSE
#> 20 2019-09-17 19:15:40.179 -0.016 -1.03 0.027 FALSE
#> 21 2019-09-17 19:15:40.190 -0.016 -1.03 0.027 FALSE
#> 22 2019-09-17 19:15:40.200 -0.016 -1.03 0.027 FALSE
#> 23 2019-09-17 19:15:40.210 -0.016 -1.03 0.027 FALSE
#> 24 2019-09-17 19:15:40.220 -0.016 -1.03 0.027 FALSE
#> 25 2019-09-17 19:15:40.230 -0.016 -1.03 0.027 FALSE
#> 26 2019-09-17 19:15:40.240 -0.016 -1.03 0.027 FALSE
#> 27 2019-09-17 19:15:40.250 -0.016 -1.03 0.027 FALSE
#> 28 2019-09-17 19:15:40.259 -0.016 -1.03 0.027 FALSE
#> 29 2019-09-17 19:15:40.269 -0.016 -1.03 0.027 FALSE
#> 30 2019-09-17 19:15:40.279 -0.016 -1.03 0.027 FALSE
#> 31 2019-09-17 19:15:40.289 -0.016 -1.03 0.027 FALSE
#> 32 2019-09-17 19:15:40.299 -0.016 -1.03 0.027 FALSE
#> 33 2019-09-17 19:15:40.309 -0.016 -1.03 0.027 FALSE
#> 34 2019-09-17 19:15:40.320 -0.016 -1.03 0.027 FALSE
#> 35 2019-09-17 19:15:40.329 -0.016 -1.03 0.027 FALSE
#> 36 2019-09-17 19:15:40.340 -0.016 -1.03 0.027 FALSE
#> 37 2019-09-17 19:15:40.349 -0.016 -1.03 0.027 FALSE
#> 38 2019-09-17 19:15:40.360 -0.016 -1.03 0.027 FALSE
#> 39 2019-09-17 19:15:40.369 -0.016 -1.03 0.027 FALSE
#> 40 2019-09-17 19:15:40.380 -0.016 -1.03 0.027 FALSE
#> 41 2019-09-17 19:15:40.389 -0.016 -1.03 0.027 FALSE
#> 42 2019-09-17 19:15:40.400 -0.016 -1.03 0.027 FALSE
#> 43 2019-09-17 19:15:40.409 -0.016 -1.03 0.027 FALSE
#> 44 2019-09-17 19:15:40.420 -0.016 -1.03 0.027 FALSE
#> 45 2019-09-17 19:15:40.429 -0.016 -1.03 0.027 FALSE
#> 46 2019-09-17 19:15:40.440 -0.016 -1.03 0.027 FALSE
#> 47 2019-09-17 19:15:40.450 -0.016 -1.03 0.027 FALSE
#> 48 2019-09-17 19:15:40.460 -0.016 -1.03 0.027 FALSE
#> 49 2019-09-17 19:15:40.470 -0.016 -1.03 0.027 FALSE
#> 50 2019-09-17 19:15:40.480 -0.016 -1.03 0.027 FALSE
#> 51 2019-09-17 19:15:40.490 -0.016 -1.03 0.027 FALSE
#> 52 2019-09-17 19:15:40.500 -0.016 -1.03 0.027 FALSE
#> 53 2019-09-17 19:15:40.509 -0.016 -1.03 0.027 FALSE
#> 54 2019-09-17 19:15:40.519 -0.016 -1.03 0.027 FALSE
#> 55 2019-09-17 19:15:40.529 -0.016 -1.03 0.027 FALSE
#> 56 2019-09-17 19:15:40.539 -0.016 -1.03 0.027 FALSE
#> 57 2019-09-17 19:15:40.549 -0.016 -1.03 0.027 FALSE
#> 58 2019-09-17 19:15:40.559 -0.016 -1.03 0.027 FALSE
#> 59 2019-09-17 19:15:40.570 -0.016 -1.03 0.027 FALSE
#> 60 2019-09-17 19:15:40.579 -0.016 -1.03 0.027 FALSE
#> 61 2019-09-17 19:15:40.590 -0.016 -1.03 0.027 FALSE
#> 62 2019-09-17 19:15:40.599 -0.016 -1.03 0.027 FALSE
#> 63 2019-09-17 19:15:40.610 -0.016 -1.03 0.027 FALSE
#> 64 2019-09-17 19:15:40.619 -0.016 -1.03 0.027 FALSE
#> 65 2019-09-17 19:15:40.630 -0.016 -1.03 0.027 FALSE
#> 66 2019-09-17 19:15:40.639 -0.016 -1.03 0.027 FALSE
#> 67 2019-09-17 19:15:40.650 -0.016 -1.03 0.027 FALSE
#> 68 2019-09-17 19:15:40.659 -0.016 -1.03 0.027 FALSE
#> 69 2019-09-17 19:15:40.670 -0.016 -1.03 0.027 FALSE
#> 70 2019-09-17 19:15:40.679 -0.016 -1.03 0.027 FALSE
#> 71 2019-09-17 19:15:40.690 -0.016 -1.03 0.027 FALSE
#> 72 2019-09-17 19:15:40.700 -0.016 -1.03 0.027 FALSE
#> 73 2019-09-17 19:15:40.710 -0.016 -1.03 0.027 FALSE
#> 74 2019-09-17 19:15:40.720 -0.016 -1.03 0.027 FALSE
#> 75 2019-09-17 19:15:40.730 -0.016 -1.03 0.027 FALSE
#> 76 2019-09-17 19:15:40.740 -0.016 -1.03 0.027 FALSE
#> 77 2019-09-17 19:15:40.750 -0.016 -1.03 0.027 FALSE
#> 78 2019-09-17 19:15:40.759 -0.016 -1.03 0.027 FALSE
#> 79 2019-09-17 19:15:40.769 -0.016 -1.03 0.027 FALSE
#> 80 2019-09-17 19:15:40.779 -0.016 -1.03 0.027 FALSE
#> 81 2019-09-17 19:15:40.789 -0.016 -1.03 0.027 FALSE
#> 82 2019-09-17 19:15:40.799 -0.016 -1.03 0.027 FALSE
#> 83 2019-09-17 19:15:40.809 -0.016 -1.03 0.027 FALSE
#> 84 2019-09-17 19:15:40.820 -0.016 -1.03 0.027 FALSE
#> 85 2019-09-17 19:15:40.829 -0.016 -1.03 0.027 FALSE
#> 86 2019-09-17 19:15:40.840 -0.016 -1.03 0.027 FALSE
#> 87 2019-09-17 19:15:40.849 -0.016 -1.03 0.027 FALSE
#> 88 2019-09-17 19:15:40.860 -0.016 -1.03 0.027 FALSE
#> 89 2019-09-17 19:15:40.869 -0.016 -1.03 0.027 FALSE
#> 90 2019-09-17 19:15:40.880 -0.016 -1.03 0.027 FALSE
#> 91 2019-09-17 19:15:40.889 -0.016 -1.03 0.027 FALSE
#> 92 2019-09-17 19:15:40.900 -0.016 -1.03 0.027 FALSE
#> 93 2019-09-17 19:15:40.909 -0.016 -1.03 0.027 FALSE
#> 94 2019-09-17 19:15:40.920 -0.016 -1.03 0.027 FALSE
#> 95 2019-09-17 19:15:40.929 -0.016 -1.03 0.027 FALSE
#> 96 2019-09-17 19:15:40.940 -0.016 -1.03 0.027 FALSE
#> 97 2019-09-17 19:15:40.950 -0.016 -1.03 0.027 FALSE
#> 98 2019-09-17 19:15:40.960 -0.016 -1.03 0.027 FALSE
#> 99 2019-09-17 19:15:40.970 -0.016 -1.03 0.027 FALSE
#> 100 2019-09-17 19:15:40.980 -0.016 -1.03 0.027 FALSE
#> 101 2019-09-17 19:15:40.990 -0.016 -1.03 0.027 FALSE
#> 102 2019-09-17 19:15:41.000 0 0 0 TRUE
print(raw[214000:214101,], n = Inf)
#> # A tibble: 102 × 6
#> time X Y Z idle_sleep_mode all_zero
#> <dttm> <dbl> <dbl> <dbl> <lgl> <lgl>
#> 1 2019-09-17 19:15:39.990 -0.016 -1.03 0.027 FALSE FALSE
#> 2 2019-09-17 19:15:40.000 0 0 0 NA TRUE
#> 3 2019-09-17 19:15:40.009 0 0 0 NA TRUE
#> 4 2019-09-17 19:15:40.019 0 0 0 NA TRUE
#> 5 2019-09-17 19:15:40.029 0 0 0 NA TRUE
#> 6 2019-09-17 19:15:40.039 0 0 0 NA TRUE
#> 7 2019-09-17 19:15:40.049 0 0 0 NA TRUE
#> 8 2019-09-17 19:15:40.059 0 0 0 NA TRUE
#> 9 2019-09-17 19:15:40.069 0 0 0 NA TRUE
#> 10 2019-09-17 19:15:40.079 0 0 0 NA TRUE
#> 11 2019-09-17 19:15:40.089 0 0 0 NA TRUE
#> 12 2019-09-17 19:15:40.099 0 0 0 NA TRUE
#> 13 2019-09-17 19:15:40.109 0 0 0 NA TRUE
#> 14 2019-09-17 19:15:40.119 0 0 0 NA TRUE
#> 15 2019-09-17 19:15:40.130 0 0 0 NA TRUE
#> 16 2019-09-17 19:15:40.140 0 0 0 NA TRUE
#> 17 2019-09-17 19:15:40.150 0 0 0 NA TRUE
#> 18 2019-09-17 19:15:40.160 0 0 0 NA TRUE
#> 19 2019-09-17 19:15:40.170 0 0 0 NA TRUE
#> 20 2019-09-17 19:15:40.180 0 0 0 NA TRUE
#> 21 2019-09-17 19:15:40.190 0 0 0 NA TRUE
#> 22 2019-09-17 19:15:40.200 0 0 0 NA TRUE
#> 23 2019-09-17 19:15:40.210 0 0 0 NA TRUE
#> 24 2019-09-17 19:15:40.220 0 0 0 NA TRUE
#> 25 2019-09-17 19:15:40.230 0 0 0 NA TRUE
#> 26 2019-09-17 19:15:40.240 0 0 0 NA TRUE
#> 27 2019-09-17 19:15:40.250 0 0 0 NA TRUE
#> 28 2019-09-17 19:15:40.259 0 0 0 NA TRUE
#> 29 2019-09-17 19:15:40.269 0 0 0 NA TRUE
#> 30 2019-09-17 19:15:40.279 0 0 0 NA TRUE
#> 31 2019-09-17 19:15:40.289 0 0 0 NA TRUE
#> 32 2019-09-17 19:15:40.299 0 0 0 NA TRUE
#> 33 2019-09-17 19:15:40.309 0 0 0 NA TRUE
#> 34 2019-09-17 19:15:40.319 0 0 0 NA TRUE
#> 35 2019-09-17 19:15:40.329 0 0 0 NA TRUE
#> 36 2019-09-17 19:15:40.339 0 0 0 NA TRUE
#> 37 2019-09-17 19:15:40.349 0 0 0 NA TRUE
#> 38 2019-09-17 19:15:40.359 0 0 0 NA TRUE
#> 39 2019-09-17 19:15:40.369 0 0 0 NA TRUE
#> 40 2019-09-17 19:15:40.380 0 0 0 NA TRUE
#> 41 2019-09-17 19:15:40.390 0 0 0 NA TRUE
#> 42 2019-09-17 19:15:40.400 0 0 0 NA TRUE
#> 43 2019-09-17 19:15:40.410 0 0 0 NA TRUE
#> 44 2019-09-17 19:15:40.420 0 0 0 NA TRUE
#> 45 2019-09-17 19:15:40.430 0 0 0 NA TRUE
#> 46 2019-09-17 19:15:40.440 0 0 0 NA TRUE
#> 47 2019-09-17 19:15:40.450 0 0 0 NA TRUE
#> 48 2019-09-17 19:15:40.460 0 0 0 NA TRUE
#> 49 2019-09-17 19:15:40.470 0 0 0 NA TRUE
#> 50 2019-09-17 19:15:40.480 0 0 0 NA TRUE
#> 51 2019-09-17 19:15:40.490 0 0 0 NA TRUE
#> 52 2019-09-17 19:15:40.500 0 0 0 NA TRUE
#> 53 2019-09-17 19:15:40.509 0 0 0 NA TRUE
#> 54 2019-09-17 19:15:40.519 0 0 0 NA TRUE
#> 55 2019-09-17 19:15:40.529 0 0 0 NA TRUE
#> 56 2019-09-17 19:15:40.539 0 0 0 NA TRUE
#> 57 2019-09-17 19:15:40.549 0 0 0 NA TRUE
#> 58 2019-09-17 19:15:40.559 0 0 0 NA TRUE
#> 59 2019-09-17 19:15:40.569 0 0 0 NA TRUE
#> 60 2019-09-17 19:15:40.579 0 0 0 NA TRUE
#> 61 2019-09-17 19:15:40.589 0 0 0 NA TRUE
#> 62 2019-09-17 19:15:40.599 0 0 0 NA TRUE
#> 63 2019-09-17 19:15:40.609 0 0 0 NA TRUE
#> 64 2019-09-17 19:15:40.619 0 0 0 NA TRUE
#> 65 2019-09-17 19:15:40.630 0 0 0 NA TRUE
#> 66 2019-09-17 19:15:40.640 0 0 0 NA TRUE
#> 67 2019-09-17 19:15:40.650 0 0 0 NA TRUE
#> 68 2019-09-17 19:15:40.660 0 0 0 NA TRUE
#> 69 2019-09-17 19:15:40.670 0 0 0 NA TRUE
#> 70 2019-09-17 19:15:40.680 0 0 0 NA TRUE
#> 71 2019-09-17 19:15:40.690 0 0 0 NA TRUE
#> 72 2019-09-17 19:15:40.700 0 0 0 NA TRUE
#> 73 2019-09-17 19:15:40.710 0 0 0 NA TRUE
#> 74 2019-09-17 19:15:40.720 0 0 0 NA TRUE
#> 75 2019-09-17 19:15:40.730 0 0 0 NA TRUE
#> 76 2019-09-17 19:15:40.740 0 0 0 NA TRUE
#> 77 2019-09-17 19:15:40.750 0 0 0 NA TRUE
#> 78 2019-09-17 19:15:40.759 0 0 0 NA TRUE
#> 79 2019-09-17 19:15:40.769 0 0 0 NA TRUE
#> 80 2019-09-17 19:15:40.779 0 0 0 NA TRUE
#> 81 2019-09-17 19:15:40.789 0 0 0 NA TRUE
#> 82 2019-09-17 19:15:40.799 0 0 0 NA TRUE
#> 83 2019-09-17 19:15:40.809 0 0 0 NA TRUE
#> 84 2019-09-17 19:15:40.819 0 0 0 NA TRUE
#> 85 2019-09-17 19:15:40.829 0 0 0 NA TRUE
#> 86 2019-09-17 19:15:40.839 0 0 0 NA TRUE
#> 87 2019-09-17 19:15:40.849 0 0 0 NA TRUE
#> 88 2019-09-17 19:15:40.859 0 0 0 NA TRUE
#> 89 2019-09-17 19:15:40.869 0 0 0 NA TRUE
#> 90 2019-09-17 19:15:40.880 0 0 0 NA TRUE
#> 91 2019-09-17 19:15:40.890 0 0 0 NA TRUE
#> 92 2019-09-17 19:15:40.900 0 0 0 NA TRUE
#> 93 2019-09-17 19:15:40.910 0 0 0 NA TRUE
#> 94 2019-09-17 19:15:40.920 0 0 0 NA TRUE
#> 95 2019-09-17 19:15:40.930 0 0 0 NA TRUE
#> 96 2019-09-17 19:15:40.940 0 0 0 NA TRUE
#> 97 2019-09-17 19:15:40.950 0 0 0 NA TRUE
#> 98 2019-09-17 19:15:40.960 0 0 0 NA TRUE
#> 99 2019-09-17 19:15:40.970 0 0 0 NA TRUE
#> 100 2019-09-17 19:15:40.980 0 0 0 NA TRUE
#> 101 2019-09-17 19:15:40.990 0 0 0 NA TRUE
#> 102 2019-09-17 19:15:41.000 0 0 0 NA TRUE
Stops the LOCF after 1 second
tr[214100:214110,]
#> # A tibble: 11 × 5
#> time X Y Z all_zero
#> <dttm> <dbl> <dbl> <dbl> <lgl>
#> 1 2019-09-17 19:15:40.990 -0.016 -1.03 0.027 FALSE
#> 2 2019-09-17 19:15:41.000 0 0 0 TRUE
#> 3 2019-09-17 19:15:41.009 0 0 0 TRUE
#> 4 2019-09-17 19:15:41.019 0 0 0 TRUE
#> 5 2019-09-17 19:15:41.029 0 0 0 TRUE
#> 6 2019-09-17 19:15:41.039 0 0 0 TRUE
#> 7 2019-09-17 19:15:41.049 0 0 0 TRUE
#> 8 2019-09-17 19:15:41.059 0 0 0 TRUE
#> 9 2019-09-17 19:15:41.070 0 0 0 TRUE
#> 10 2019-09-17 19:15:41.079 0 0 0 TRUE
#> 11 2019-09-17 19:15:41.090 0 0 0 TRUE
raw[214100:214110,]
#> # A tibble: 11 × 6
#> time X Y Z idle_sleep_mode all_zero
#> <dttm> <dbl> <dbl> <dbl> <lgl> <lgl>
#> 1 2019-09-17 19:15:40.990 0 0 0 NA TRUE
#> 2 2019-09-17 19:15:41.000 0 0 0 NA TRUE
#> 3 2019-09-17 19:15:41.009 0 0 0 NA TRUE
#> 4 2019-09-17 19:15:41.019 0 0 0 NA TRUE
#> 5 2019-09-17 19:15:41.029 0 0 0 NA TRUE
#> 6 2019-09-17 19:15:41.039 0 0 0 NA TRUE
#> 7 2019-09-17 19:15:41.049 0 0 0 NA TRUE
#> 8 2019-09-17 19:15:41.059 0 0 0 NA TRUE
#> 9 2019-09-17 19:15:41.069 0 0 0 NA TRUE
#> 10 2019-09-17 19:15:41.079 0 0 0 NA TRUE
#> 11 2019-09-17 19:15:41.089 0 0 0 NA TRUE
Do LOCF for both
locf = function(x) {
x$all_zero = x$X == 0 & x$Y == 0 & x$Z == 0
x$X = ifelse(x$all_zero, NA_real_, x$X)
x$Y = ifelse(x$all_zero, NA_real_, x$Y)
x$Z = ifelse(x$all_zero, NA_real_, x$Z)
x$all_zero = NULL
x$X = vctrs::vec_fill_missing(x$X, direction = "down")
x$Y = vctrs::vec_fill_missing(x$Y, direction = "down")
x$Z = vctrs::vec_fill_missing(x$Z, direction = "down")
x
}
raw = raw %>%
locf()
tr = tr %>%
locf()Now they agree
all.equal(tr %>%
select(time, X, Y, Z),
raw %>%
select(time, X, Y, Z),
check.attributes = FALSE,
check.class = FALSE)
#> [1] TRUECreated on 2025-07-14 with reprex v2.1.1
Session info
sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#> setting value
#> version R version 4.4.0 (2024-04-24)
#> os macOS 15.4.1
#> system x86_64, darwin20
#> ui X11
#> language (EN)
#> collate en_US.UTF-8
#> ctype en_US.UTF-8
#> tz America/New_York
#> date 2025-07-14
#> pandoc 3.7.0.2 @ /usr/local/bin/ (via rmarkdown)
#>
#> ─ Packages ───────────────────────────────────────────────────────────────────
#> package * version date (UTC) lib source
#> agcounts * 0.6.9 2025-04-16 [1] Github (bhelsel/agcounts@64f02e5)
#> bit 4.5.0.1 2024-12-03 [1] CRAN (R 4.4.1)
#> bit64 4.6.0-1 2025-01-16 [1] CRAN (R 4.4.1)
#> blob 1.2.4 2023-03-17 [1] CRAN (R 4.4.0)
#> bslib 0.8.0 2024-07-29 [1] CRAN (R 4.4.0)
#> cachem 1.1.0 2024-05-16 [1] CRAN (R 4.4.0)
#> cli 3.6.3 2024-06-21 [1] CRAN (R 4.4.0)
#> colorspace 2.1-1 2024-07-26 [1] CRAN (R 4.4.0)
#> crayon 1.5.3 2024-06-20 [1] CRAN (R 4.4.0)
#> curl 6.1.0 2025-01-06 [1] CRAN (R 4.4.1)
#> data.table 1.16.4 2024-12-06 [1] CRAN (R 4.4.1)
#> DBI 1.2.3 2024-06-02 [1] CRAN (R 4.4.0)
#> digest 0.6.37 2024-08-19 [1] CRAN (R 4.4.1)
#> dplyr * 1.1.4 2023-11-17 [1] CRAN (R 4.4.0)
#> evaluate 1.0.3 2025-01-10 [1] CRAN (R 4.4.1)
#> fastmap 1.2.0 2024-05-15 [1] CRAN (R 4.4.0)
#> fs 1.6.5 2024-10-30 [1] CRAN (R 4.4.1)
#> generics 0.1.3 2022-07-05 [1] CRAN (R 4.4.0)
#> GGIR 3.1-5 2024-11-07 [1] CRAN (R 4.4.1)
#> ggplot2 3.5.1 2024-04-23 [1] CRAN (R 4.4.0)
#> glue 1.8.0 2024-09-30 [1] CRAN (R 4.4.1)
#> gsignal 0.3-7 2024-09-11 [1] CRAN (R 4.4.1)
#> gtable 0.3.6 2024-10-25 [1] CRAN (R 4.4.1)
#> hms 1.1.3 2023-03-21 [1] CRAN (R 4.4.0)
#> htmltools 0.5.8.1 2024-04-04 [1] CRAN (R 4.4.0)
#> htmlwidgets 1.6.4 2023-12-06 [1] CRAN (R 4.4.0)
#> httpuv 1.6.15 2024-03-26 [1] CRAN (R 4.4.0)
#> jquerylib 0.1.4 2021-04-26 [1] CRAN (R 4.4.0)
#> jsonlite 1.8.9 2024-09-20 [1] CRAN (R 4.4.1)
#> knitr 1.49 2024-11-08 [1] CRAN (R 4.4.1)
#> later 1.4.1 2024-11-27 [1] CRAN (R 4.4.1)
#> lattice 0.22-6 2024-03-20 [1] CRAN (R 4.4.0)
#> lifecycle 1.0.4 2023-11-07 [1] CRAN (R 4.4.0)
#> lubridate 1.9.4 2024-12-08 [1] CRAN (R 4.4.1)
#> magrittr 2.0.3 2022-03-30 [1] CRAN (R 4.4.0)
#> Matrix 1.7-2 2025-01-23 [1] CRAN (R 4.4.1)
#> memoise 2.0.1 2021-11-26 [1] CRAN (R 4.4.0)
#> mime 0.12 2021-09-28 [1] CRAN (R 4.4.0)
#> munsell 0.5.1 2024-04-01 [1] CRAN (R 4.4.0)
#> ncar 0.5.0 2023-11-19 [1] CRAN (R 4.4.0)
#> NonCompart 0.7.0 2023-11-15 [1] CRAN (R 4.4.0)
#> pillar 1.10.1 2025-01-07 [1] CRAN (R 4.4.1)
#> pkgconfig 2.0.3 2019-09-22 [1] CRAN (R 4.4.0)
#> png 0.1-8 2022-11-29 [1] CRAN (R 4.4.0)
#> promises 1.3.2 2024-11-28 [1] CRAN (R 4.4.1)
#> purrr 1.0.2 2023-08-10 [1] CRAN (R 4.4.0)
#> R.methodsS3 1.8.2 2022-06-13 [1] CRAN (R 4.4.0)
#> R.oo 1.27.0 2024-11-01 [1] CRAN (R 4.4.1)
#> R6 2.5.1 2021-08-19 [1] CRAN (R 4.4.0)
#> Rcpp 1.0.14 2025-01-12 [1] CRAN (R 4.4.1)
#> reactable 0.4.4 2023-03-12 [1] CRAN (R 4.4.0)
#> read.gt3x * 1.3.0 2025-03-31 [1] local
#> readr * 2.1.5 2024-01-10 [1] CRAN (R 4.4.0)
#> reprex 2.1.1 2024-07-06 [1] CRAN (R 4.4.0)
#> reticulate * 1.42.0 2025-03-25 [1] CRAN (R 4.4.1)
#> rlang 1.1.5 2025-01-17 [1] CRAN (R 4.4.1)
#> rmarkdown 2.29 2024-11-04 [1] CRAN (R 4.4.1)
#> RSQLite 2.3.9 2024-12-03 [1] CRAN (R 4.4.1)
#> rstudioapi 0.17.1 2024-10-22 [1] CRAN (R 4.4.1)
#> rtf 0.4-14.1 2020-03-22 [1] CRAN (R 4.4.0)
#> sass 0.4.9 2024-03-15 [1] CRAN (R 4.4.0)
#> scales 1.3.0 2023-11-28 [1] CRAN (R 4.4.0)
#> sessioninfo 1.2.2 2021-12-06 [1] CRAN (R 4.4.0)
#> shiny 1.10.0 2024-12-14 [1] CRAN (R 4.4.1)
#> stringi 1.8.4 2024-05-06 [1] CRAN (R 4.4.0)
#> stringr 1.5.1 2023-11-14 [1] CRAN (R 4.4.0)
#> tibble 3.2.1 2023-03-20 [1] CRAN (R 4.4.0)
#> tidyr 1.3.1 2024-01-24 [1] CRAN (R 4.4.0)
#> tidyselect 1.2.1 2024-03-11 [1] CRAN (R 4.4.0)
#> timechange 0.3.0 2024-01-18 [1] CRAN (R 4.4.0)
#> tzdb 0.4.0 2023-05-12 [1] CRAN (R 4.4.0)
#> vctrs 0.6.5 2023-12-01 [1] CRAN (R 4.4.0)
#> vroom 1.6.5 2023-12-05 [1] CRAN (R 4.4.0)
#> withr 3.0.2 2024-10-28 [1] CRAN (R 4.4.1)
#> xfun 0.50 2025-01-07 [1] CRAN (R 4.4.1)
#> xtable 1.8-4 2019-04-21 [1] CRAN (R 4.4.0)
#> yaml 2.3.10 2024-07-26 [1] CRAN (R 4.4.0)
#> zoo 1.8-12 2023-04-13 [1] CRAN (R 4.4.0)
#>
#> [1] /Library/Frameworks/R.framework/Versions/4.4-x86_64/Resources/library
#>
#> ─ Python configuration ───────────────────────────────────────────────────────
#> python: /Users/johnmuschelli/Library/Caches/org.R-project.R/R/reticulate/uv/cache/archive-v0/HvR9uorI86uxJlmhMp9pv/bin/python3
#> libpython: /Users/johnmuschelli/Library/Caches/org.R-project.R/R/reticulate/uv/python/cpython-3.11.12-macos-x86_64-none/lib/libpython3.11.dylib
#> pythonhome: /Users/johnmuschelli/Library/Caches/org.R-project.R/R/reticulate/uv/cache/archive-v0/HvR9uorI86uxJlmhMp9pv:/Users/johnmuschelli/Library/Caches/org.R-project.R/R/reticulate/uv/cache/archive-v0/HvR9uorI86uxJlmhMp9pv
#> virtualenv: /Users/johnmuschelli/Library/Caches/org.R-project.R/R/reticulate/uv/cache/archive-v0/HvR9uorI86uxJlmhMp9pv/bin/activate_this.py
#> version: 3.11.12 (main, Apr 9 2025, 04:10:09) [Clang 20.1.0 ]
#> numpy: /Users/johnmuschelli/Library/Caches/org.R-project.R/R/reticulate/uv/cache/archive-v0/HvR9uorI86uxJlmhMp9pv/lib/python3.11/site-packages/numpy
#> numpy_version: 2.3.1
#>
#> ──────────────────────────────────────────────────────────────────────────────
Issue: ActiLife output is not identical to the output from `pygt3x.
Root of the issue: There are missing timestamps from the data and inconsistent filling in from gaps (e.g. idle sleep mode).
TAS1H30182785_2019-09-17RAW.csv
From ActiLife
GT3X -> CSV
GT3X File
Here we read in the GT3X file, similar to the README example:
https://github.com/actigraph/pygt3x?tab=readme-ov-file#example-usage
Renaming data for consistency
Rounding the data similar to ActiLife
Comparison
We can compare the raw data from the GT3X file with the CSV file from ActiLife.
They do not agree.
The ActiLife has more data and more at the end of the file
There are minutes flagged as idle sleep mode in this data, so the missing minutes below are not from this.
There are missing minutes in the output from
pygt3xQuestion: Is this to be expected? If so how do we handle these missing minutes?
Create these rows in the data
We are missing additional data
Compare the data we have
We can at least compare the data we have both on time
It seems that ActiLife does last observation carried forward (LOCF) as per https://actigraphcorp.my.site.com/support/s/article/Idle-Sleep-Mode-Explained, but it only does it for one second in this data. Thus, if we do LOCF for the ActiLife output and that from
pygt3xafter merging on the full data sources, we can get the same output.Stops the LOCF after 1 second
Do LOCF for both
Now they agree
Created on 2025-07-14 with reprex v2.1.1
Session info