only_obs only pulls EVID==0, regardless of MDV

[jprybylski]

xpose/R/fetch_data.R

Line 59 in d6d194d

mdv_var <- xp_var(xpdb, .problem, type = c('evid', 'mdv'), silent = TRUE)$col[1]

This is because of the [1] in only_obs(). I think the logic in the function body would need to be changed if the [1] were removed, but currently this is only filtering on EVID==0, not EVID==0 and MDV==0. I will submit a pull request with a tested change if needed.

[bguiastr]

Hi, I don't recall exactly what was the rationale for this design, but I think I was trying to keep the workflow as simple as possible by only using either EVID or MDV and not both simultaneously. This takes away the responsibility of making decisions on special cases – e.g. common missuse of MDV = 1 when EVID = 0 – and putting this back on the user to decide which variable to use with set_vars_types(evid = ...).

[jprybylski]

That's interesting and makes sense, thanks. I'm often working with datasets where MDV 1 with EVID 0 is commonly used to signify that a sample was collected but not assayed, so to me that's not inherently a misuse of MDV. That situation specifically triggered this issue submission, in fact.

I also wonder if this creates confusion since users don't know evid alone controls obs filtering unless it is missing, in which case mdv is the default; that seems hard for someone to anticipate. If NONMEM (etc) would see a row as an observation, my hunch is that's what the user is expecting xpose to see as an obs.

[bguiastr]

Sorry for the delay in my response. I just ran some tests with NONMEM, to illustrate the rationale behind the behavior in xpose.

Using the following example:

Case	DV	AMT	MDV	EVID
1	0.0000	100	1	0
2	0.0834	0	0	0

• Case 1: MDV == 1, EVID == 0, AMT > 0 -> No issue with NONMEM
• Case 2: MDV ==1, EVID == 0, DV > 0 -> NONMEM warning

(DATA WARNING 5) RECORD
2, DATA ITEM
3, CONTENTS: 0.0834
THE DV DATA ITEM IS POSITIVE, BUT THE MDV DATA ITEM IS 1

[jprybylski]

No worries on the delay and the example cases are very illustrative. You may have DV=0.0, DV=-999 or any number of other values that you are using with MDV=1 without that warning. I think the rationale is sound, but also since this is a diagnostics package it would make sense to have a consistent definition of observations between the fitting tool and the plotting tool.

[prhutson]

I am just switching to new XPOSE, and am finding this issue of MDV=1 not being recognized when EVID = 0. How should I code my xpose_data or other filtering command to ignore observation records that are EVID=0 and MDV=1?

[smouksassi]

do you really want to cover all NONMEM cases ?
Users know what features were used in nonmem and it is user's responsibility is to make xpose aware of what was used

NONMEM can have MDV 100, - something etc. so many possibiilities ... documentation:
As of NM73, if an MDV is set to a value greater than or equal to 100, it is converted to that value minus 100 upon input, but will not be used during estimation or covariance assessment, only for table outputting. This option allows you to use the same file for estimation and table outputs, without significantly slowing down the estimation. So if MDV=101, it will be converted to 1 upon use for final evaluations, and the records will be ignored during estimation.
The subroutines in NONMEM that ignore MDV=100 and MDV=101 records are: OBJ (all estimation and covariance steps), OBJ2 (parametric), OBJ3 (non-parametric), and OS (initial estimates of OMEGAs and SIGMAs). Care must be taken in using MDV>=100, in that during estimation, covariate data items of these records are not used, which can have a slightly different interpolation impact than what is finally recorded in the tables where they are used. You may specifically request that any one of these routines not ignore the MDV>=100 records, by setting MDVI1=1 (for OBJ to include MDV>=100 records), MDVI2=1 (for OBJ2 to include MDV>=100 records), MDVI3=1 (for OBJ3 to include MDV>100 records), in a $PK or $PRED block,

[prhutson]

Thanks, but no, I'm just curious how best to get new XPOSE to drop EVID=0/MDV=1 observation records.

[jprybylski]

@smouksassi I don't think the cases you're referring to are as common as the expectation MDV==0 and EVID==0 are observation records and EVID!=0 or MDV!=0 are non-observation. The goal shouldn't be to cover all NONMEM cases, but indeed to cover common ones.

@prhutson you can simply exclude your MDV==1 records on creation of the xpose_data object:

xpdb <- xpose_data(...) %>%
  filter(MDV==0)

[prhutson]

Brilliant! Thank you.