Recommendations for NTP server
I've just started getting interested in precision timekeeping - if you
can call an offset from UTC that is measured in units of milliseconds
"precision".
My current timeserver is a Buffalo Linkstation, patched to run Debian
Stable, and slaved via 2MBit ADSL to my ISPs Stratum 2 timeserver, using
NTP V4.2.0, from the Debian Stable repository. I'm seeing offsets in the
low millisecond range, but with frequent excursions up to 10 to 20
millisec or more. These large spikes in the offset curve are definitely
correlated with large data transfers down from the Internet. Look at
http://www.g8kbz.demon.co.uk/mrtg.html for a sample plot, derived by
MRTG, and updated daily at midnight local time.
There seems to be no prospect of mitigating these offset spikes without
taking further steps, and I intend to install a new timeserver, probably
using a Soekris NET4801 and a Garmin GPS18LVC. I hope to effect a 3
order of magnitude improvement in observed offset, and more would be
appreciated - I've already been bitten by the precision bug.
So my question is - should I continue with Debian Stable, or would
OpenBSD be better for sub-microsecond accuracy? Indeed, is sub-
microsecond offset achievable with this hardware? GPSDOs and Rb or Cs
standards are not yet practical politics.
Thanks in advance for any tips,
Geoff Powell
Geoff Powell said the following on 04/23/2006 05:00 PM:
So my question is - should I continue with Debian Stable, or would
OpenBSD be better for sub-microsecond accuracy? Indeed, is sub-
microsecond offset achievable with this hardware? GPSDOs and Rb or Cs
standards are not yet practical politics.
Hi Geoff --
FreeBSD is definitely the best tuned OS for NTP timekeeping, but Linux
can do OK. The biggest problem is that there's no kernel support for
PPS signals in the 2.6 series of kernels. There is a patch available
for a few 2.6 versions (including 2.6.15) called "PPS-kit-alpha" that
implements at least the ability to pass the PPS signal through the
serial port to NTP, but doesn't implement the kernel time discipline.
I am not running 2.6.15 because it has some problems with the linux-gpib
drivers that I depend on for data logging, but instead using a shared
memory driver called "shm_linux_clock" that runs as a separate program
and monitors a serial port for PPS and passes that through to NTP. It
works very well and holds within 10s of microseconds, most of the time.
I plot my NTP servers' performance at
http://www.febo.com/time-freq/ntp/stats so you can see the results I'm
getting with both Linux and FreeBSD if you'd like. At the moment the
data is a mess because I've been changing configurations and rebooting,
etc., but if you look past the noise, you can see the steady state
performance where all five stratum 1 servers are tracking within 100us
of each other. If you subtract the WWVB clock, which has some problems
when it loses lock, the GPS and Cs clocks track each other within 10 or
20 us when everything is stable.
John
You'll get more than you expect -- the offset you're observing on
ADSL is very likely wrong, because the delays your packets experience
on adsl aren't symmetric. NTP assumes symmetry. So I wouldn't
actually believe that a 1ms offset is really 1ms off, depending on
the RTT to your ISP.
But there actually are ways to mitigate the offset spikes. You might
be able, for instance, to configure your gateway to prioritize NTP
packets over everything else, which will help with half of the
problem. You won't be able to do the same at your ISP, of course, so
it's not a perfect solution.
Installing a local GPS-synched server is the right answer if you
really care. And it's fun. :) The Soekris boxes rock. I assume
you've already seen Poul-Henning Kamp's page about using his net4801
with FreeBSD to act as a high precision timeserver? If you want sub-
microsecond, you'll probably have to replace the oscillator on the 4801.
And - most OSes should do the trick. FreeBSD has a really nice
precision timekeeping interface, though -- and it makes a marvelously
solid time server. I'm running it on a few Net4801s and recommend
it. You can very easily build an image for it using another bit of
phk's magic called 'nanobsd' (it's in the source tree).
-Dave
On Apr 23, 2006, at 5:00 PM, Geoff Powell wrote:
I've just started getting interested in precision timekeeping - if you
can call an offset from UTC that is measured in units of milliseconds
"precision".
My current timeserver is a Buffalo Linkstation, patched to run Debian
Stable, and slaved via 2MBit ADSL to my ISPs Stratum 2 timeserver,
using
NTP V4.2.0, from the Debian Stable repository. I'm seeing offsets
in the
low millisecond range, but with frequent excursions up to 10 to 20
millisec or more. These large spikes in the offset curve are
definitely
correlated with large data transfers down from the Internet. Look at
http://www.g8kbz.demon.co.uk/mrtg.html for a sample plot, derived by
MRTG, and updated daily at midnight local time.
There seems to be no prospect of mitigating these offset spikes
without
taking further steps, and I intend to install a new timeserver,
probably
using a Soekris NET4801 and a Garmin GPS18LVC. I hope to effect a 3
order of magnitude improvement in observed offset, and more would be
appreciated - I've already been bitten by the precision bug.
So my question is - should I continue with Debian Stable, or would
OpenBSD be better for sub-microsecond accuracy? Indeed, is sub-
microsecond offset achievable with this hardware? GPSDOs and Rb or Cs
standards are not yet practical politics.
Thanks in advance for any tips,
Geoff Powell
time-nuts mailing list
time-nuts@febo.com
https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts
David Andersen wrote:
You'll get more than you expect -- the offset you're observing on ADSL
is very likely wrong, because the delays your packets experience on
adsl aren't symmetric. NTP assumes symmetry. So I wouldn't actually
believe that a 1ms offset is really 1ms off, depending on the RTT to
your ISP.
But there actually are ways to mitigate the offset spikes. You might
be able, for instance, to configure your gateway to prioritize NTP
packets over everything else, which will help with half of the
problem. You won't be able to do the same at your ISP, of course, so
it's not a perfect solution.
Installing a local GPS-synched server is the right answer if you
really care. And it's fun. :) The Soekris boxes rock. I assume
you've already seen Poul-Henning Kamp's page about using his net4801
with FreeBSD to act as a high precision timeserver? If you want
sub-microsecond, you'll probably have to replace the oscillator on the
4801.
And - most OSes should do the trick. FreeBSD has a really nice
precision timekeeping interface, though -- and it makes a marvelously
solid time server. I'm running it on a few Net4801s and recommend
it. You can very easily build an image for it using another bit of
phk's magic called 'nanobsd' (it's in the source tree).
-Dave
I'll second the soekris box - my box time.no-such-agency.net is a 4801
running FreeBSD with a GPS18LVC. You can expect offsets in the +/-
5us range except when the box is stressed - the standard xtal in the box
is not temperature compensated so offsets will spike to as much as 250us
when the system does something compute intensive. If you don't need the
HD consider the 4501 as it has the ability to timestamp a PPS using
internal counters rather than the DCD kludge. - Again see phk's
excellent work on the subject including the code in FreeBSD to support
PPS on a 4501.
Regarding DSL - I now have three stratum one servers here: a GPS18LVC, a
TrueTime NTS100 and TrueTime GPS XL-DC (it is addictive isn't it) - my
ISP just installed a GPS18LVC based server one hop (on gig-e) from my
DSL concentrator so I can get a very clear picture of the error
introduced by my DSL asymmetry. For my line (6000 down 608 up
backhauled over ATM) it's 1.5ms. In a perfect world NTP would allow a
fudge offset to be applied to subnets (in my case 1.5ms on the default
and 0.0 on the local LAN) which would factor out the know DSL error -
alas it doesn't do that right now.
John
In article 444BED98.7060401@febo.com, John Ackermann N8UR
jra@febo.com writes
FreeBSD is definitely the best tuned OS for NTP timekeeping, but Linux
can do OK. The biggest problem is that there's no kernel support for
PPS signals in the 2.6 series of kernels. There is a patch available
for a few 2.6 versions (including 2.6.15) called "PPS-kit-alpha" that
implements at least the ability to pass the PPS signal through the
serial port to NTP, but doesn't implement the kernel time discipline.
I suspected that, but couldn't be sure - net sources are ambiguous on
the point, even the NTP documentation.
Is PPS kernel discipline compiled into the default FreeBSD kernel?
I am not running 2.6.15 because it has some problems with the linux-gpib
drivers that I depend on for data logging, but instead using a shared
memory driver called "shm_linux_clock" that runs as a separate program
and monitors a serial port for PPS and passes that through to NTP. It
works very well and holds within 10s of microseconds, most of the time.
Does this require a kernel recompile, is it a module, or does it run in
userland?
Initially, I'd like to stick with Debian, since I have a (somewhat)
better understanding of it - OpenBSD would be a whole new ballgame. I'm
a newbie in either case.
I plot my NTP servers' performance at
http://www.febo.com/time-freq/ntp/stats so you can see the results I'm
getting with both Linux and FreeBSD if you'd like. At the moment the
data is a mess because I've been changing configurations and rebooting,
etc., but if you look past the noise, you can see the steady state
performance where all five stratum 1 servers are tracking within 100us
of each other. If you subtract the WWVB clock, which has some problems
when it loses lock, the GPS and Cs clocks track each other within 10 or
20 us when everything is stable.
Yes, I noticed that all the machines vary more-or-less together, which
suggests a common-mode effect - is databox playing up?
I know that radio clocks (I'd be using MSF Rugby, or DCF Mainflingen)
are poor in comparison, and probably worse than your WWVB results,
because the receivers I'd be using would have a lot less intelligence
than yours.
--
Geoff Powell
In article 92ECBF63-4331-4CF4-8AEE-B60824151EA1@cs.cmu.edu, David
Andersen dga+@cs.cmu.edu writes
You'll get more than you expect -- the offset you're observing on
ADSL is very likely wrong, because the delays your packets experience
on adsl aren't symmetric. NTP assumes symmetry. So I wouldn't
actually believe that a 1ms offset is really 1ms off, depending on
the RTT to your ISP.
Yes, I knew about that, but all I can go by as a newbie is what MRTG is
telling me - and it's telling me that my time offsets are too big...
But there actually are ways to mitigate the offset spikes. You might
be able, for instance, to configure your gateway to prioritize NTP
packets over everything else, which will help with half of the
problem. You won't be able to do the same at your ISP, of course, so
it's not a perfect solution.
A Netgear DG814 is too brain-dead to understand QoS. One thing at a
time...
Installing a local GPS-synched server is the right answer if you
really care. And it's fun. :) The Soekris boxes rock. I assume
you've already seen Poul-Henning Kamp's page about using his net4801
with FreeBSD to act as a high precision timeserver? If you want sub-
microsecond, you'll probably have to replace the oscillator on the 4801.
Yes, I've read his page on that - as an electronic engineer, I even
understood it! That will be the next stage - I'll settle for units of
microseconds for the moment.
And - most OSes should do the trick. FreeBSD has a really nice
precision timekeeping interface, though -- and it makes a marvelously
solid time server. I'm running it on a few Net4801s and recommend
it. You can very easily build an image for it using another bit of
phk's magic called 'nanobsd' (it's in the source tree).
As I just asked John A. - is it in the default kernel?
Thanks for your comments,
Geoff Powell
In article 444BF2D0.40603@cloudview.com, John Pettitt
jpp@cloudview.com writes
I'll second the soekris box - my box time.no-such-agency.net is a 4801
running FreeBSD with a GPS18LVC. You can expect offsets in the +/-
5us range except when the box is stressed - the standard xtal in the box
is not temperature compensated so offsets will spike to as much as 250us
when the system does something compute intensive. If you don't need the
HD consider the 4501 as it has the ability to timestamp a PPS using
internal counters rather than the DCD kludge. - Again see phk's
excellent work on the subject including the code in FreeBSD to support
PPS on a 4501.
So you'd recommend a 4501, with FreeBSD in Compact Flash? I wanted to
use CF anyway, since a box running 24/7 is not the best environment in
which to have moving parts. The box will have no more that NTP and MRTG
on it.
Thanks for the accuracy estimate.
Regarding DSL - I now have three stratum one servers here: a GPS18LVC, a
TrueTime NTS100 and TrueTime GPS XL-DC (it is addictive isn't it) - my
ISP just installed a GPS18LVC based server one hop (on gig-e) from my
DSL concentrator so I can get a very clear picture of the error
introduced by my DSL asymmetry. For my line (6000 down 608 up
backhauled over ATM) it's 1.5ms. In a perfect world NTP would allow a
fudge offset to be applied to subnets (in my case 1.5ms on the default
and 0.0 on the local LAN) which would factor out the know DSL error -
alas it doesn't do that right now.
My ADSL is 2272 down/256 up, so I'd likely be seeing bigger asymmetry.
Thanks,
--
Geoff Powell
Geoff Powell wrote:
And - most OSes should do the trick. FreeBSD has a really nice
precision timekeeping interface, though -- and it makes a marvelously
solid time server. I'm running it on a few Net4801s and recommend
it. You can very easily build an image for it using another bit of
phk's magic called 'nanobsd' (it's in the source tree).
As I just asked John A. - is it in the default kernel?
It's a trivial recompile (kernel rebuilds are really easy on FreeBSD).
John
Geoff Powell wrote:
In article 444BF2D0.40603@cloudview.com, John Pettitt
jpp@cloudview.com writes
I'll second the soekris box - my box time.no-such-agency.net is a 4801
running FreeBSD with a GPS18LVC. You can expect offsets in the +/-
5us range except when the box is stressed - the standard xtal in the box
is not temperature compensated so offsets will spike to as much as 250us
when the system does something compute intensive. If you don't need the
HD consider the 4501 as it has the ability to timestamp a PPS using
internal counters rather than the DCD kludge. - Again see phk's
excellent work on the subject including the code in FreeBSD to support
PPS on a 4501.
So you'd recommend a 4501, with FreeBSD in Compact Flash? I wanted to
use CF anyway, since a box running 24/7 is not the best environment in
which to have moving parts. The box will have no more that NTP and MRTG
on it.
Thanks for the accuracy estimate.
Yes it the 4501 is really good for this (it's also less expensive) - see
http://phk.freebsd.dk/soekris-pps/ and this
http://phk.freebsd.dk/soekris/kern45xx.html for info on the subject.
The real win with the 4501 is using the built in counters in the Elan
CPU to latch the PPS signal - this takes all the interrupt latency stuff
out of the mix and improves the accuracy considerably - the stability of
the oscillator will be your biggest source of errors. If you run
nanobsd (basically a cut down FreeBSD) with no jobs other than NTP on
the box you should get a really nice server for under $300 including the
GPS. You might want to think about running mrtg on another box (unless
you are willing to put the data on ram disk and lose it on power cycle)
as the CF card won't like being written repeatedly for log and graph
updates.
John
Geoff Powell said the following on 04/23/2006 05:42 PM:
Is PPS kernel discipline compiled into the default FreeBSD kernel?
No, but it's a pretty easy thing to turn on -- you add a line to the
config file and tell it to go. Pretty straightforward, once you find
the instructions.
I am not running 2.6.15 because it has some problems with the linux-gpib
drivers that I depend on for data logging, but instead using a shared
memory driver called "shm_linux_clock" that runs as a separate program
and monitors a serial port for PPS and passes that through to NTP. It
works very well and holds within 10s of microseconds, most of the time.
Does this require a kernel recompile, is it a module, or does it run in
userland?
It runs in userland. You need to make sure that your NTP has the shared
memory refclock compiled in.
Yes, I noticed that all the machines vary more-or-less together, which
suggests a common-mode effect - is databox playing up?
Yes, yesterday databox suffered an unplanned reboot, which exposed some
configuration problems. It took quite a while to get everything playing
properly, and then NTP takes a while to settle down.
I have a Soekris 4501 on order (do you detect a theme?) and plan to lock
its oscillator to the house timebase, and then switch the monitoring
task over to that machine and away from databox. For now, I get a sense
of how databox is performing by looking at the common mode effect on the
other servers.
I know that radio clocks (I'd be using MSF Rugby, or DCF Mainflingen)
are poor in comparison, and probably worse than your WWVB results,
because the receivers I'd be using would have a lot less intelligence
than yours.
I don't know -- the Spectracom receiver is pretty ancient technology,
and I suspect that Jonathan Buzzard's code running on a modern machine
might give it a good run for its money. The biggest problem with the
Spectracom is that when it loses lock once every 10 days or so, there is
a sudden multi-millisecond offset that takes a couple of days to fully
recover from.
John
In message Gb7QUsAye$SEFwKA@g8kbz.demon.co.uk, Geoff Powell writes:
So you'd recommend a 4501, with FreeBSD in Compact Flash? I wanted to
use CF anyway, since a box running 24/7 is not the best environment in
which to have moving parts. The box will have no more that NTP and MRTG
on it.
There isn't much I can add here, but I just want to add a few things:
Sub-microsecond precision is impossible with regular crystals, end
of story. You need OCXO or Rb support for that.
If you don't need the SNMP capability, drop MRTG and use RRDtool,
(MRTG's more general sibling).
If you want to run MRTG or RRDtool, you probably do not want to use
a CF card: it will wear out - eventually. Don't trust more than a
year.
I wrote a hack called "Remote RRD" which sends updates in UDP packets
to another host which hosts the RRD file and web-pages and all that
to get around this problem.
--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
phk@FreeBSD.ORG | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
In article 15153.1145871481@critter.freebsd.dk, Poul-Henning Kamp
phk@phk.freebsd.dk writes
In message Gb7QUsAye$SEFwKA@g8kbz.demon.co.uk, Geoff Powell writes:
So you'd recommend a 4501, with FreeBSD in Compact Flash? I wanted to
use CF anyway, since a box running 24/7 is not the best environment in
which to have moving parts. The box will have no more that NTP and MRTG
on it.
There isn't much I can add here, but I just want to add a few things:
Sub-microsecond precision is impossible with regular crystals, end
of story. You need OCXO or Rb support for that.
Fine - I presume something like your web page about Soekris boards
(where you characterised the performance of such a unit at ultimate
precision) is the route to follow. I have follow-up plans for something
similar further down the road. This year, next year, sometime, never...
Meanwhile, FreeBSD with your patches on a NET4501 will give units of
microseconds?
If you don't need the SNMP capability, drop MRTG and use RRDtool,
(MRTG's more general sibling).
Rats! Another app to learn! But you're right, SNTP is a gong - the bells
and whistles I want are NTP stats.
If you want to run MRTG or RRDtool, you probably do not want to use
a CF card: it will wear out - eventually. Don't trust more than a
year.
Yes, I am aware of flash memory wear-out problems. I was thinking of
NFS-mounting a directory elsewhere on my network as scratch space.
I wrote a hack called "Remote RRD" which sends updates in UDP packets
to another host which hosts the RRD file and web-pages and all that
to get around this problem.
Aha! A cure. Would you post a URL? Pretty please?
--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
phk@FreeBSD.ORG | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
time-nuts mailing list
time-nuts@febo.com
https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts
--
Geoff Powell
On Apr 24, 2006, at 3:05 PM, Geoff Powell wrote:
I wrote a hack called "Remote RRD" which sends updates in UDP packets
to another host which hosts the RRD file and web-pages and all that
to get around this problem.
Aha! A cure. Would you post a URL? Pretty please?
In message $AguiKAZuUTEFw0R@g8kbz.demon.co.uk, Geoff Powell writes:
Meanwhile, FreeBSD with your patches on a NET4501 will give units of
microseconds?
It's all in stock FreeBSD, no patches needed.
The major problem you will have is that NTPD isn't very good once
you get down into microseconds.
Yes, I am aware of flash memory wear-out problems. I was thinking of
NFS-mounting a directory elsewhere on my network as scratch space.
That would work too.
I wrote a hack called "Remote RRD" which sends updates in UDP packets
to another host which hosts the RRD file and web-pages and all that
to get around this problem.
Aha! A cure. Would you post a URL? Pretty please?
It's not very slick, but it works :-)
--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
phk@FreeBSD.ORG | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
John Ackermann N8UR wrote:
Geoff Powell said the following on 04/23/2006 05:42 PM:
Is PPS kernel discipline compiled into the default FreeBSD kernel?
No, but it's a pretty easy thing to turn on -- you add a line to the
config file and tell it to go. Pretty straightforward, once you find
the instructions.
Where would I find these instructions?
I've decided to setup a FreeBSD box for my time server sooner rather
than later.
I presume FreeBSD v6.0 is up to the task. Other than PPS kernel
discipline, are
there any other nonobvious configuration changes I'll need to make?
thanks,
glenn
Glenn wrote:
Is PPS kernel discipline compiled into the default FreeBSD kernel?
No, but it's a pretty easy thing to turn on -- you add a line to the
config file and tell it to go. Pretty straightforward, once you find
the instructions.
Where would I find these instructions?
It's in the FreeBSD users manual (or whatever it's called) which I found
at the FreeBSD web site. That gives general instructions for rebuilding
the kernel. I'm not sure if it specifically identifies the PPS flag or
not. On my 5.4 machine, it was adding:
NTP PPS support
options PPS_SYNC
to the /usr/src/sys/i386/config/PPS_KERNEL file, which was copied from
the generic kernel config file. Then rebuild, specifying PPS_KERNEL as
the config file name. There's also an associated .hints file which
allows you to set the serial ports to low-latency mode by disabling the
FIFO in the 16550 UART chip; this can reduce jitter. I don't recall for
sure, but I think it's setting the second bit (ie, 0x1_) in the flags
field:
hint.sio.0.at="isa"
hint.sio.0.port="0x3F8"
hint.sio.0.flags="0x12"
John