Repository navigation
Sleeping computer causes span timings to be incorrect in browser #852
Description
Activity
@draffensperger @obecny would like to get your expertise on this.
One solution would be to use
Date.now()to capture startTime, but useperformance.now()to calculate the duration of the span. This would give us still high fidelity span durations while mitigating this issue. One problem with this is thatDate.now()trusts the system clock, so if the system clock ever changes,Date.now()will use the updated time.performance.timeOriginandperformance.now()are monotonic and ignore changes to the system clock, which makes them ideal for calculating durations.Is this only effecting browsers or also node? As far as I know node uses also
performance.timeOrigincaptured at startup therefore I would expect a similar behavior. If it works in node we could take a look how they solved it there.can you provide more details. Are spans still being created when computer sleeps or how does this really affect that ?
sleep you computer for a few hours. any spans created after you wake up have timestamps that show you a few hours in the past because the clock paused. This behavior continues until the browser is restarted.
and what system / laptop has that - as I'm working on mac and my computer sleeps every day, browser is not restarted for weeks - I think I would notice that before hmm.
The machine which had the problem was a mac using chrome. Possibly this is a sleep v hibernate issue?
ok I see it now too
new Date(performance.timeOrigin + performance.now())is behind real timemaybe we should raise this as a chrome bug ?
This issue of someone sleeping the computer, or leaving the tab open to go get coffee, etc. is sort of a basic limitation of measurement in the browser world. If a user starts an operation, shuts their computer, and then comes back 8 hours later and it finishes - then how long did the operation take?
Here are a few ideas for how we could deal with this:
- Provide some documentation that this suspected to be a known issue and we suggest that people ignore large outlier spans from the client in their debugging (this is the easiest approach)
- Attempt to do some detection in the web library itself of when the user closes the tab or moves to a different tab (see https://stackoverflow.com/questions/3888902/detect-browser-or-tab-closing), and do some handling of that, e.g. add a special tag to all spans that the browser tab was switched and possibly even end and send what spans we currently have (not sure on that).
- Record both the high-resolution timestamp of spans (using
performance.now) to make sure we aren't losing accuracy, but also record theDate.nowtimes, and then we can do some correction computations before spans are written and if the times don't cohere, we can make a best-effort fix, and possibly add some special tag to the span to indicate that it had this monotonic clock issue.
I think it would be worth us digging more into the browser specs, docs, etc. to better understand this Chrome on Mac behavior and what was intended by the authors for
performance.now. Maybe it is a Chrome bug.When I run
new Date(performance.timeOrigin+performance.now()).getTime() - new Date().getTime()in my browser, I seem to get0.Reacted by Paul DraperReacted by Alberto LealWhen I run
new Date(performance.timeOrigin+performance.now()).getTime() - new Date().getTime()in my browser, I seem to get0.just close laptop for 30 seconds, open and run it, seems like a valid bug
Reacted by Fausto David Suarez RosarioI think all we have to do is to keep the performance, but then when we export spans we should add delta.
const delta = new new Date().getTime() - Date(performance.timeOrigin+performance.now()).getTime()I think you would have to add the delta at the time that you generate the timestamp. if you do it at export time the whole span will shift if it is started before the computer sleeps
21 remaining items
I think we have been discussing this for quite long time :). I would be in favor of creating some solution and then validate it. If the only thing that is 100% up to date is
new Date().getTime()then we will have to use it one way or another. We can then mix Date with performance.now calculate the deltas and return this calculation correctly. Even if we lose some microseconds it will be still much better and bullet proof comparing to what we have now. But I would be really in favour of creating some MVP as we all have some ideas how this can be done, now is probably a right time to make it happen.Is there a way to make this configurable on the client so people can experiment?
For example, provide a clock implementation to the opentelemetry API to use to generate timestamps?
The interface for a clock implementation would have to make assumptions about the final implementation. For instance do we implement
now()to get the current timestamp, or do we implementt = start(); t.end()and get durations?I think we should differentiate between node and browser regarding this. I don't think we should add unneeded overhead/complexity in node because of hibernate/sleep issues effecting effectively only browsers.
Besides that I think the API should offer both, get an absolute timestamp and get a duration.
Currently users can provide timestamps for span start/end but they have no possibility via API to use the same timesource as SDK. Having a clock interface in API would improve this.
Having the possibility to install a time provider (similar as a context manager) would even allow users to use something more fancy like a clock synced with their backend.We are actually affected by this bug on Microsoft Azure (confirmed with Azure Functions Node.js v14.18.1 on Windows), we observe time drifts of the span start in the order of 10 seconds relative to the wall clock time. I can also reproduce the issue locally be hibernating my notebook, with Node v15.9.0 on Windows 11.
So it seems to not only affect Node 8 as well.
I think the wall clock time could get out of sync with the high resolution timer for any number of reasons, so many that I think it can be taken for a fact that it will drift if the application is running for any longer amount of time. Examples:
Only affecting wall clock time but not high resolution timer:
- NTP sync or other time correction
- Leap seconds (which do affect UTC, contrary to daylight saving time changes)
Affecting high resolution timer:
- Any kind of suspension:
- "I closed my notebook lid" / hibernate / suspend / sleep -- this will mostly affect the browser and dev machines
- VM being migrated to another physical host and probably other VM shenanigans you may face when running on any PaaS/FaaS product where you-knows-what may happen to the underlying VM.
- CPU clock changes are known to cause inaccuracies in some configurations
- Some hardware / firmware bugs & corner cases (at least historically)
It seems like
performance.nowis uv_hrtime: https://github.com/nodejs/node/blob/5fad0b93667ffc6e4def52996b9529ac99b26319/src/node_perf_common.h#L19You found nodejs/node#17893 already, but even if the offset is calculated correctly at startup, there is just no way to solve this issue with a constant calculated-only-once offset. There is libuv/libuv#1674 for the underlying uv_hrtime API, which has been closed as stale.
uv_hrtime is implemented with QueryPerformanceCounter on Windows. https://github.com/libuv/libuv/blob/f250c6c73ee45aa93ec44133c9e0c635780ea741/src/win/util.c#L490-L514, which is definitely liable to the aforementioned time drift.
The Linux (actually Linux-specific, other Unix have different implementations) is more complicated: The generic Unix entrypoint has only one line https://github.com/libuv/libuv/blob/0b1c752b5c40a85d5c749cd30ee6811997a8f71e/src/unix/core.c#L110-L112 calling the specific part with {{UV_CLOCK_PRECISE}} https://github.com/libuv/libuv/blob/c40f8cb9f8ddf69d116952f8924a11ec0623b445/src/unix/linux-core.c#L121-L155. This ends up calling clock_gettime(CLOCK_MONOTONIC), which is a libc/POSIX API, described for Linux e.g. here https://man7.org/linux/man-pages/man3/clock_gettime.3.html:
This clock does not count time that the system is suspended.
On Linux there would be CLOCK_BOOTTIME which would (supposedly) solve this problem but I believe it is not accessible through Node. Also, there is no equivalent for Windows (unless you want something a bit less precise; though maybe CLOCK_BOOTTIME is also less precise than CLOCK_MONOTONIC).
You could look at how other SDKs solve this problem. Python has a different API https://docs.python.org/3/library/time.html#time.time_ns, which ends up in this implementation https://github.com/python/cpython/blob/0ff626f210c69643d0d5afad1e6ec6511272b3ce/Python/pytime.c#L847-L955, but that's probably not really applicable to Node.js as (an is a "a bit less precise" at least on Windows).
I think you could implement rather one to one what Java does though: There, each local root span takes a timestamp with both the most precise available wall clock time and the high resolution timer when it starts to calculate the offset. From then on, only the HR timer is used, i.e. relative offsets of local child span start times and all end times are precise. Of course, if a suspension / leap seconds / ... happens during the trace, you still have the time drift, but that is much less likely and the next trace will be correct again. This is implemented with the aptly named https://github.com/open-telemetry/opentelemetry-java/blob/main/sdk/trace/src/main/java/io/opentelemetry/sdk/trace/AnchoredClock.java and this logic at span creation: https://github.com/open-telemetry/opentelemetry-java/blob/16be81aed803e15694de29c9cea25f7bcf4d77c1/sdk/trace/src/main/java/io/opentelemetry/sdk/trace/SdkSpan.java#L151-L182
Reacted by Chengzhong WuIs this issue fixed now that #3134 has been merged ?
- linked a pull request that will close this issuechore: proposal 1.7.0/0.33.0 #3259
on Sep 16, 2022
While debugging an issue for a user here https://gitter.im/open-telemetry/opentelemetry-node?at=5e6a4b54d17593652b7c8154 it was found that while a computer is slept or hibernated, the
performance.now()monotonic clock may be paused. This causes the assumption thatperformance.timeOrigin + performance.now() ~= Date.now()to be incorrect by some arbitrary amount of time which may be hours or days.