time-nuts@lists.febo.com

Discussion of precise time and frequency measurement

View all threads

Re: NIST NTP servers way off for anyone else?

BK
Bob kb8tq
Thu, Dec 16, 2021 2:45 PM

Hi

On most “normal” home connections to the internet, your trip
time “up” to something will be much longer than your trip time
“back” from the same destination. Part of this is simply a reflection
of download speed being much better than upload speed. However
as noted in other posts, there is a lot more to it.

Bob

On Dec 15, 2021, at 11:43 AM, Adam Space time.isanapp@gmail.com wrote:

Good idea. Doing so reveals the expected outcome from the wedge plot:
variable forward path delay, shifted in the positive direction, and a
pretty stable negative path delay. Is this the norm for consumer grade
connection? It seems to be for me.

On Wed, Dec 15, 2021 at 10:53 AM Magnus Danielson via time-nuts <
time-nuts@lists.febo.com> wrote:

Hi,

Expect network routes to be more dispersed these days, as it is needed.

While the wedge plot is a classic for NTP, it may be interesting to plot
forward and backward path histograms independently.

Cheers,
Magnus

On 2021-12-15 16:25, Adam Space wrote:

Yeah I think it is localized. Network paths have been quite variable for
me. Every once in a while I start getting massive delays from the NIST
servers to my system, resulting in results like yours.

Interestingly though, time-e-g was one of the only servers that didn't

have

this problem for me. This is a recent wedge plot for it. seems to be
working fine for me now, just with a variable outgoing delay causing
positive offsets, which seems to be more of a problem with my connection
than anything else.

On Tue, Dec 14, 2021 at 9:04 PM K5ROE Mike K5ROE@roetto.org wrote:

On 12/14/21 5:23 PM, Hal Murray wrote:

Out of curiosity, since you monitor NIST Gaithersburg, if you were to

average

over the offsets for a whole month, what kind of value would you get?

Surely

it is close to zero but I am curious how close. Within 1ms?

It depends.  Mostly on the routing between you and NIST.  If you are

closer,

the routing is more likely to be symmetric.

From my experience, routing is generally stable on the scale of

months.  There

are short (hours) changes when a fiber gets cut or a router gets

busted.

There are long term changes as people add fibers and/or change business

deals.

There are some cases where a stable routing will produce 2 answers: x%

of the

packets will be slightly faster/slower than most of them.  I think

what's

going on is that the routers are doing load sharing on multiple paths,

hashing

on the address+port.  Or something like that.  So it's a roll of the

dice

which path you get.


I'm in California.

NIST has NTP servers at 3 locations in the Boulder CO area: NIST, WWV,

and

Univ of Colorado.  (Google maps says WWV is 60 miles north of Bouler.

Univ of

Colorado is a few miles from NIST.)

From a cheap cloud server (Digital Ocean) in San Francisco, the RTT

to

NIST is

31.5 ms, to WWV is 32.1 ms, to Univ of Colorado is 54.5 ms.  The time

offsets

are about 1 ms for NIST and WWV and 12 ms for Univ of Colorado.

From my home (AT&T via Sonic), 30 miles south of San Francisco, the

RTTs are

61 ms for NIST and WWV and 81-82 for Univ of Colorado.  Offsets are 6-7

ms for

NIST and WWV and 4-5 ms in the other direction for Univ of Colorado.

Might be a localized routing phenomenon.  Using my verizon connection

from

Northern Virginia the results are awful for time-e-g.nist.gov:

   remote           refid      st t when poll reach   delay   offset

jitter


---============

-192.168.1.219  68.69.221.61    2 u  56  64  377    0.400  -0.290
0.035
*192.168.1.224  .PPS.            1 u    1  16  377    0.184    0.087
0.017
-129.6.15.26    .NIST.          1 u  32  64  377  93.087  -37.940
7.867

However from my AWS machine in Oregon:

MS Name/IP address        Stratum Poll Reach LastRx Last sample


---=============

^- 152.63.13.177                3  6  377    63  -2011us[-2011us] +/-
128ms
^+ 209.182.153.6                2  7  377    65  -959us[ -959us] +/-
86ms
^- 64.139.66.105                3  6  377  128  -5838us[-5838us] +/-
134ms
^+ 129.6.15.26                  1  6  377    64  -2075us[-2075us] +/-
37ms
^* 173.66.105.50                1  8  377  438  -448us[ -870us] +/-
38ms

-mike


time-nuts mailing list -- time-nuts@lists.febo.com -- To unsubscribe

send

an email to time-nuts-leave@lists.febo.com
To unsubscribe, go to and follow the instructions there.


time-nuts mailing list -- time-nuts@lists.febo.com -- To unsubscribe

To unsubscribe, go to and follow the instructions there.


time-nuts mailing list -- time-nuts@lists.febo.com -- To unsubscribe send
an email to time-nuts-leave@lists.febo.com
To unsubscribe, go to and follow the instructions there.

<delay_hists.png>_______________________________________________
time-nuts mailing list -- time-nuts@lists.febo.com -- To unsubscribe send an email to time-nuts-leave@lists.febo.com
To unsubscribe, go to and follow the instructions there.

Hi On most “normal” home connections to the internet, your trip time “up” to something will be much longer than your trip time “back” from the same destination. Part of this is simply a reflection of download speed being much better than upload speed. However as noted in other posts, there is a *lot* more to it. Bob > On Dec 15, 2021, at 11:43 AM, Adam Space <time.isanapp@gmail.com> wrote: > > Good idea. Doing so reveals the expected outcome from the wedge plot: > variable forward path delay, shifted in the positive direction, and a > pretty stable negative path delay. Is this the norm for consumer grade > connection? It seems to be for me. > > On Wed, Dec 15, 2021 at 10:53 AM Magnus Danielson via time-nuts < > time-nuts@lists.febo.com> wrote: > >> Hi, >> >> Expect network routes to be more dispersed these days, as it is needed. >> >> While the wedge plot is a classic for NTP, it may be interesting to plot >> forward and backward path histograms independently. >> >> Cheers, >> Magnus >> >> On 2021-12-15 16:25, Adam Space wrote: >>> Yeah I think it is localized. Network paths have been quite variable for >>> me. Every once in a while I start getting massive delays from the NIST >>> servers to my system, resulting in results like yours. >>> >>> Interestingly though, time-e-g was one of the only servers that didn't >> have >>> this problem for me. This is a recent wedge plot for it. seems to be >>> working fine for me now, just with a variable outgoing delay causing >>> positive offsets, which seems to be more of a problem with my connection >>> than anything else. >>> >>> On Tue, Dec 14, 2021 at 9:04 PM K5ROE Mike <K5ROE@roetto.org> wrote: >>> >>>> On 12/14/21 5:23 PM, Hal Murray wrote: >>>>>> Out of curiosity, since you monitor NIST Gaithersburg, if you were to >>>> average >>>>>> over the offsets for a whole month, what kind of value would you get? >>>> Surely >>>>>> it is close to zero but I am curious how close. Within 1ms? >>>>> It depends. Mostly on the routing between you and NIST. If you are >>>> closer, >>>>> the routing is more likely to be symmetric. >>>>> >>>>> From my experience, routing is generally stable on the scale of >>>> months. There >>>>> are short (hours) changes when a fiber gets cut or a router gets >> busted. >>>>> There are long term changes as people add fibers and/or change business >>>> deals. >>>>> There are some cases where a stable routing will produce 2 answers: x% >>>> of the >>>>> packets will be slightly faster/slower than most of them. I think >> what's >>>>> going on is that the routers are doing load sharing on multiple paths, >>>> hashing >>>>> on the address+port. Or something like that. So it's a roll of the >> dice >>>>> which path you get. >>>>> >>>>> -------- >>>>> >>>>> I'm in California. >>>>> >>>>> NIST has NTP servers at 3 locations in the Boulder CO area: NIST, WWV, >>>> and >>>>> Univ of Colorado. (Google maps says WWV is 60 miles north of Bouler. >>>> Univ of >>>>> Colorado is a few miles from NIST.) >>>>> >>>>> From a cheap cloud server (Digital Ocean) in San Francisco, the RTT >> to >>>> NIST is >>>>> 31.5 ms, to WWV is 32.1 ms, to Univ of Colorado is 54.5 ms. The time >>>> offsets >>>>> are about 1 ms for NIST and WWV and 12 ms for Univ of Colorado. >>>>> >>>>> From my home (AT&T via Sonic), 30 miles south of San Francisco, the >>>> RTTs are >>>>> 61 ms for NIST and WWV and 81-82 for Univ of Colorado. Offsets are 6-7 >>>> ms for >>>>> NIST and WWV and 4-5 ms in the other direction for Univ of Colorado. >>>>> >>>>> >>>> Might be a localized routing phenomenon. Using my verizon connection >> from >>>> Northern Virginia the results are awful for time-e-g.nist.gov: >>>> >>>> remote refid st t when poll reach delay offset >>>> jitter >>>> >>>> >> ============================================================================== >>>> -192.168.1.219 68.69.221.61 2 u 56 64 377 0.400 -0.290 >>>> 0.035 >>>> *192.168.1.224 .PPS. 1 u 1 16 377 0.184 0.087 >>>> 0.017 >>>> -129.6.15.26 .NIST. 1 u 32 64 377 93.087 -37.940 >>>> 7.867 >>>> >>>> However from my AWS machine in Oregon: >>>> >>>> MS Name/IP address Stratum Poll Reach LastRx Last sample >>>> >>>> >> =============================================================================== >>>> ^- 152.63.13.177 3 6 377 63 -2011us[-2011us] +/- >>>> 128ms >>>> ^+ 209.182.153.6 2 7 377 65 -959us[ -959us] +/- >>>> 86ms >>>> ^- 64.139.66.105 3 6 377 128 -5838us[-5838us] +/- >>>> 134ms >>>> ^+ 129.6.15.26 1 6 377 64 -2075us[-2075us] +/- >>>> 37ms >>>> ^* 173.66.105.50 1 8 377 438 -448us[ -870us] +/- >>>> 38ms >>>> >>>> >>>> -mike >>>> _______________________________________________ >>>> time-nuts mailing list -- time-nuts@lists.febo.com -- To unsubscribe >> send >>>> an email to time-nuts-leave@lists.febo.com >>>> To unsubscribe, go to and follow the instructions there. >>>> >>>> >>>> _______________________________________________ >>>> time-nuts mailing list -- time-nuts@lists.febo.com -- To unsubscribe >> send an email to time-nuts-leave@lists.febo.com >>>> To unsubscribe, go to and follow the instructions there. >> _______________________________________________ >> time-nuts mailing list -- time-nuts@lists.febo.com -- To unsubscribe send >> an email to time-nuts-leave@lists.febo.com >> To unsubscribe, go to and follow the instructions there. >> > <delay_hists.png>_______________________________________________ > time-nuts mailing list -- time-nuts@lists.febo.com -- To unsubscribe send an email to time-nuts-leave@lists.febo.com > To unsubscribe, go to and follow the instructions there.
SS
Steven Sommars
Thu, Dec 16, 2021 11:50 PM

I suggest plotting one-way delays versus time rather than histograms.
Recent example: I regularly monitor a stratum1/GPS NTP server located in
Canada.
Delay plots from two completely separate monitoring systems are attached
below.  When the two one-way delays change as seen beginning 2021-12-15
07:00,
(request "delay" increases and response "delay" decreases or vice versa)
the most likely causes are server and/or client clock drift.
If the client drifts, the effect will be seen for all monitored servers.
I use multiple monitoring clients to further minimize false alarms.

Changes in network delays (e.g., due to routing) affect the
request/response delay without the correlation seen below.  Network path
changes tend to result in sudden delay jumps.

These are good heuristics, but could fall prey to a sophisticated network
attack.
For the Canada stratum 1 example, the issue is real, poor holdover
performance.  The administrator plans to replace the current indoor puck
with an external antenna.

Other notes:
I was told that  time-e-b.nist.gov received a server hardware upgrade
today.  My monitoring shows dramatic improvement in IPv6 performance.

Steve

ISP=Comcast (cable)
[image: image.png]

Another ISP 300Mbps fiber
[image: image.png]

I suggest plotting one-way delays versus time rather than histograms. Recent example: I regularly monitor a stratum1/GPS NTP server located in Canada. Delay plots from two completely separate monitoring systems are attached below. When the two one-way delays change as seen beginning 2021-12-15 07:00, (request "delay" increases and response "delay" decreases or vice versa) the most likely causes are server and/or client clock drift. If the client drifts, the effect will be seen for all monitored servers. I use multiple monitoring clients to further minimize false alarms. Changes in network delays (e.g., due to routing) affect the request/response delay without the correlation seen below. Network path changes tend to result in sudden delay jumps. These are good heuristics, but could fall prey to a sophisticated network attack. For the Canada stratum 1 example, the issue is real, poor holdover performance. The administrator plans to replace the current indoor puck with an external antenna. Other notes: I was told that time-e-b.nist.gov received a server hardware upgrade today. My monitoring shows dramatic improvement in IPv6 performance. Steve ISP=Comcast (cable) [image: image.png] Another ISP 300Mbps fiber [image: image.png]