DM-53501: Investigate Null values in camera_hexapod_compensation_offset_* column #122
Conversation
bbrondel
left a comment
There was a problem hiding this comment.
I have concerns about exposures getting discarded if they straddle an EFD query window. Please write up a Jira ticket to make sure that will get addressed and write your ticket number into an appropriate place in a code comment. Other than that, approved with the following comments.
| self.dialect = postgresql | ||
| else: | ||
| raise Exception(f"The dialect for {sgbd} has not yet been implemented.") | ||
| raise NotImplementedError(f"The dialect for {sgbd} has not yet been implemented.") |
There was a problem hiding this comment.
Move lines 92-99 above the initialization of self._engines to get the intended exception message you had in mind.
| for uri in self.db_uris | ||
| ] | ||
|
|
||
| sgbd = self.db_uri.split(":")[0] |
There was a problem hiding this comment.
There should probably be a check that all URIs in the list have the same dialect.
| pool_size=1, | ||
| max_overflow=0, | ||
| pool_pre_ping=True, | ||
| connect_args={"connect_timeout": 5}, |
There was a problem hiding this comment.
The connect_timeout arg doesn't work with sqlite. Do something like this instead:
if sgbd == "sqlite":
connect_args = {"timeout": 5 }
self.dialect = sqlite
elif sgbd == "postgresql":
connect_args = {"connect_timeout": 5 }
self.dialect = postgresql
else:
raise ...
|
@bbrondel |
…SAWarning to logger
…ter, remove dead _ensure_utc_naive
…ard missing function_args
6161045 to
7905916
Compare
0d20b2d to
d580b2d
Compare
Nulls in the
camera_hexapod_compensation_offset_*columns came from two places. First, the upsert was overwriting good data with nulls whenever a topic returned no results for a given exposure, the incoming row had a null in that column, and the old ONCONFLICT ... SETblindly replaced the stored value. Second, exposures that straddled the task boundary were being processed with an incomplete time window, producing partial computations. The upsert now usesCOALESCE(excluded.column, table.column)to keep the existing value when the incoming one is null, and the processing loop only includes exposures and visits whose timespan falls entirely inside the task window.A few supporting cleanups are included: DAO objects are cached per
Transforminstance instead of reconstructed on every run, the database pool usesQueuePoolwithpool_pre_pinginstead ofNullPool, andSAWarningoutput is routed to the structured logger. Latent bugs fixed along the way:task_update_countswas writing "failed" instead of "running",select_failed'sbutler_repofilter was dead code, andcurrent_utccould raiseNameErrordepending on which branch executed.