metric of uncertainty around fitted parameter values

[ari-khoudary]

hi, thanks very much for making and maintaining this wonderfully powerful tool.

I have a basic question that I am having trouble finding an answer to in the docs, FAQ, and both Shinn et al (2020) papers -- how do I get an estimate of the uncertainty around a fitted parameter value?

For example, I just fit two extremely simple models estimating the effect of prior probability on the starting point. The model for my 80% cue condition is specified like this:

model_80 = pyddm.gddm( name = "starting point80", starting_position = lambda bias, trueCongruence: bias if trueCongruence=='congruent' else -bias, bound="B", T_dur = 4.1, nondecision='ndt', parameters={'bias': (-1, 1), 'B': (0, 3), 'ndt': (0.2)}, conditions = ['trueCongruence'])

and the model for my 50% cue condition is specified like this:

model_50 = pyddm.gddm( name = "starting point50", starting_position = 'bias', bound="B", T_dur = 4.1, nondecision='ndt', parameters={'bias': (-1, 1), 'B': (0, 3), 'ndt': (0.2)})

As predicted by reams of theory, I found that the boundary value was effectively identical between both fits, but the starting point was greater for model_80 than model_50 (0.14 and 0.07, respectively). But this finding feels incomplete without some estimate of the uncertainty for each fitted starting point. Is there an easy way to obtain this using the gddm() approach? Any and all advice is helpful, thank you!

[mwshinn]

PyDDM uses point estimates obtained from a fitting procedure. If you want variability of the fitting procedure, you can just fit the same model multiple times. But fitting variability is generally pretty low and is probably not the type of variability you are looking for. So to get something about, e.g., the sensitivity of your parameters to your data, you would need to use something like a bootstrap procedure. There are some techniques based on Fisher information as well but that starts to get a bit more complicated. In the J Neuro paper I think we looked at parameter sensitivity by finding how much you need to perturb a parameter to decrease the model likelihood by some fixed amount (but maybe this analysis didn't make it into the paper, I don't remember).

[ari-khoudary]

thank you!