Hi

Two very separate chunks to this:

To get 1588 to work well, you either need very lightly loaded hardware ,  no hardware or 1588 compatible
switches and routers. In most cases “home use” cases NTP will get you into the microsecond(s) level. It
does not take any fancy hardware and NTP drivers exist for just about everything. If you happen to have
a WiFi setup as part of your “net” forget about any sort of timing accuracy dimensioned in microseconds.

====

Time wise, a < $10 single band GPS module ( ublox 8 series or whatever ) will get you well below 100 ns.
That’s much “better” time than NTP or 1588 will reliably transfer. The F9P ( or better the F9T ) will do some
neat stuff, you will loose it’s “value” as soon as you try to move it over the ethernet.

Simple answer:

Plug one of the many ublox modules into a random PC or server, bring up NTP, get things pointed at it,
move on. For added points, bring up three or four NTP servers …..

Bob

Hello!

In summary, I say go for it!

This topic was bound to hit time-nuts - great to see Rpi finally going for
a MAC/PHY with hardware timestamp support and hopefully future full-sized
models will follow suit. Unfortunately for most chips other than Intel and
enterprise comms vendors like Solarflare, Mellanox etc, a IEEE-1588
compliant MAC/PHY means that the silicon physically only timestamps PTP
packets, whereas with Intel and others, it can timestamp all packets,
making for example hardware-backed NTP possible.

I would hardly dismiss the chance to use hardware timestamping and/or PTP,
or in fact dismiss any chance of any fun and experimentation with anything!
:-)

This being Broadcom, I could not get my hands on any reference documents
for BCM54210, so I am unable to assess whether it has any timestampable
event inputs on top of hardware timestamping, which could be used to sync
the h/w clock to 1PPS directly. If it hasn't, then it's a two-step process
to get GNSS (1PPS) into hardware: GNSS -> OS clock -> PTP clock. If one is
after PTP though, there are more suitable SBCs than the CM4 for which the
breakout board isn't sold yet, like Beaglebone Black (TI AM335x) or
Wandboard (i.mxN) or MinnowBoard (x86 and some variants have discrete Intel
PHY chips with 1PPS in/out support) or PC Engines' APU1/APU2 with
multiple(!) Intel PHYs.All of these support PTP.

Some comments to Bob's reply:

On Tue, 20 Oct 2020 at 23:08, Bob kb8tq kb8tq@n1k.org wrote:

99% of servers used in high frequency trading today use 1588 and they are
hardly lightly loaded. CPU load has no effect on hardware timestamping and
PTP on the network card itself. It does affect syncing the OS clock to the
network interface, but not as much as you may think. Just like sub-100 ns
with a software-based 1PPS input is possible, sub-100 ns between the PTP
clock and OS clock also is.

PTP-compatible routers or switches (PTP Transparent Clock or Boundary
Clock) are only a real requirement if:

• you have a physical source of network path asymmetry such as a speed
  change (100M ingress, 1G egress or similar)
• your network is actually congested e.g. you have frames contending for
  port or switch fabric, not just by principle of this being a switch, but
  really bad congestion

Physical path asymmetry will affect accuracy, so as far as time and
frequency stability, it only introduces a constant error.

Unless one's router/switch is a software-based or virtual one, or a very
ancient device otherwise, and the network isn't heavily loaded,  PTP
routinely achieves precision of sub-100 ns or better without network
assistance, depending on timestamp resolution of the hardware used, and
inherent PDV (jitter) of network equipment used - but it tends to be
Gaussian, and so filtrable. Obviously as long as the software (PTP slave)
isn't dumb and is capable of decent filtering and outlier removal,etc. In
early days, hubs (as in repeaters) rather than switches fared better; I
remember we used to use an old hub for PTP sync at work back in ~2007. PTP
I see and sell to customers today, provably delivers +/-20 ns or better
over transparent clocks, but before we had transparent clocks, sub-100 ns
was pretty much a norm already. Granted, on expensive hardware and on a
dedicated network, but the results were not all that different on a
proverbial NetGear switch, not by an order of magnitude anyway.

Practically, as far as the core principle of two-way time transfer goes,
PTP and NTP are identical, as long as hardware timestamping is used - which
tends to be problematic for NTP. It's only on-path PTP support that is
something that NTP will never grow.

If you happen to have a WiFi setup as part of your “net” forget about any

100% agree.

NTP not, PTP yes! With some asterisks but generally mostly yes. Key also
being fast message rates for PTP where the slave has plenty of samples to
pick and filter from. Otherwise this isn't really about NTP versus PTP,
it's about hardware vs. software timestamps. The amount of jitter
introduced by software timestamping on a non-realtime OS often tends to
exceed the same introduced by the network - on a local network that is.

...but exactly the same happens if you try to move it over software-based
NTP, in fact it's much worse. Ethernet itself - really not that bad.
Granted, an F9x may be an overkill, but an 8-th gen or 7-th gen ublox
timing board would work very well, bordering on what PTP can actually
deliver.

...Or get a pair of those CM4 or say Beaglebone Black, even without GPS,
and see how PTP precision will look like between them on your network and
report back! Why settle on maintaining a stable clock in one device and not
share it with other devices even if the result is slightly degraded? This
is what mobile network operators did. Starting with 3G, but especially with
4G and now into 5G as backhaul networks became more sync friendly, you will
hardly see a GNSS receiver in a new base station anymore - you will see PTP
instead.

Just go for it :)

Thanks,
Wojciech

--

Wojciech Owczarek

Hi

Networking hardware is the issue …..

If you have fully symmetric delays, then there is no need for 1588. NTP will
do just fine. Unfortunately as even home networking hardware loads up,
( = lots of traffic in only one direction) this may not be the case. When things
do go asymmetric, then you do need 1588 compatible switches.

Bob

Hi Bob,

On Wednesday, 21 October 2020, Bob kb8tq kb8tq@n1k.org wrote:

Undeniably so, but to what extent?

Lab results and years of production deployment tell me otherwise. Hardware
timestamps are the key. NTP with hardware timestamps on both server and
client will do just fine, yes.

Quantify "lots". My argument is that the uncertainty of software-generated
timestamps can outweigh that of a network that isn't loaded up to choking
point e.g. little to no buffering. Home traffic tends to be bursty, with
occasional constant load and yes asymmetric. But your PTP device is not
sitting on your ISP link, neither of them will, and unless your host
running PTP is sitting on a saturated link, it really is not as bad as it
may seem. Even cheap switches these days are non-blocking, so unless you
have lots of multicast, PTP hosts on your network will not be critically
affected by other traffic. Yes, they are store and forward, but it's about
selecting packets in a band with minimum transit latency or, say, mode. The
band has to be wide enough to catch at least one packet per second. With a
rate of say 32 msg/sec chances of catching lucky packets increase. I can't
normally take results out of my work lab unfortunately, but I think I might
just drop a "home" switch in there and put some application traffic through
it for a proper test. I know what difference a PTP transparent clock switch
makes, but even without one, it really is "not that bad". Numbers or it
didn't happen, so I'll ad that to my to-do list.

Cheers,
Wojciech

--

Wojciech Owczarek

Hi

Simply that you can get into the 10’s of microseconds (done in many times) with
NTP. If the local LAN is streaming video, it needs 1588 switches / routers to do
better than that.

Well. your lab and my home LAN produce very different results …..

Bob

Hi,

On 2020-10-22 01:28, Wojciech Owczarek wrote:

I've seen networks provide over 10 ms of time error due to asymmetry.

I've seen poor scheduling and such in the CPU, especially virtualized
enviroments, to be many minutes.

Yeah, poor CPU time-stamping can be worse than network biases, but
hardware time-stamping and careful use of good time-stamps handles most
of the host issues. Then network biases becomes the dominant problem
unless one does not have other implementation issues, and well, those
are common, including dropping PTP packets aimed for processing. In one
instance there was a rate-limit internal to the equipment that did the
wrong thing. Slowly the implementation issues is being ironed out. I
learn more of this than I want to.

Cheers,
Magnus

Wojciech Owczarek writes:

So a couple of footnotes from somebody who have actually used this kind
of hardware.

What the "timestamping" hardware gives you, is a snapshot of a
counter running a the PHY frequency.

That PHY frequency and its quality depends on what is on the other
end of the cable.

Rule of thumb: If your switch does not have a metal cabinet and
does not speak ssh(1), you will be disappointed.

You need to tie that PHY counter to your kernels timekeeping.

If you do it by appointing it "timecounter", any event affecting
PHY frequnecy, downnegotiation, unplugged cable, power-cycled switch,
EMC, Lightning, shitty hardware will screw up your computer's
time-keeping.

If you do not appoint it "timecounter", you need to poll it from
the kernel often enough to precisely model it detect all of the
above disturbances.

On "real" PTP hardware, this probing uses a separate signal which
is timestamped by the PHY counter and the CPU timecounter
you depend on.  This may be easier, and will certainly be cheaper
in magic smoke, on an RPi, but only possible given chip-docs.

(When I did this on Intels '599 chip ten years ago, my questions
regarding these details caused the "datasheet" to grow by nearly
250 pages.)

Absent access to such a signal, you have to do the probing entirely
in software, which is tricky, expensive, and due to the memory-hierarchy
and clock boundaries, very noisy process.

So, yeah... Perfect time-nuts project :-)

--
Poul-Henning Kamp      | UNIX since Zilog Zeus 3.20
phk@FreeBSD.ORG        | TCP/IP since RFC 956
FreeBSD committer      | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.

Hello Bob,

On Thu, 22 Oct 2020 at 01:53, Bob kb8tq kb8tq@n1k.org wrote:

That is all true, however not every single frame will hit those worst
latencies. With careful filtering and high message rates, lucky packets can
be picked out from the rest. The result shows more jitter than NTP, but
overall tighter bounded and with significantly less wander. Also, a 10 us
offset reported does not necessarily mean it's been passed to the clock
discipline and will cause the clock to swing by that, but consume enough of
those and it will, yes.

Well. your lab and my home LAN produce very different results …..

Again - I can only nod until I produce some results :-) I tend to ignore
self-reported offset figures from software, so unless a thing has a 1PPS
output, I am very sceptical. The NICs I test with, do.

On Thu, 22 Oct 2020 at 01:53, Magnus Danielson magnus@rubidium.se wrote:

The story of my life pretty much. Early PTP transparent clock switches were
two-step and software would handle PTP message corrections and PTP messages
indeed went to the CPU, so this was somewhat error-prone. Pretty much all
new higher-end Broadcom switching ACICs (OK, hardly home switch material)
have had PTP TC support completely in hardware for a number of years now,
with zero software control required. First TCs were mostly industrial
automation kit like Hirschmann, Moxa and what have you. Since Broadcom's
TCs (and BCs) hit the mainstream, PTP on the network side has been more
attainable than ever. But on the host side, it's still a minefield.

--

Wojciech Owczarek

This may be of interest to the original poster...

Apparently 1588 support in the CM4 currently only exists in hardware alone,
and in datasheets. No Linux PTP driver has been written for the BCM
gemac + BCM54210PE PHY combination as of yet:

https://www.raspberrypi.org/forums/viewtopic.php?t=295829
https://github.com/raspberrypi/linux/issues/4151

Datasheets / application notes for the said PHY are not public, so with
this, unfortunately, I would not buy the CM4 unless it had kernel support
for PTP, and I doubt it will come very soon. Not never, but not just yet.

Thanks,
Wojciech

On Tue, 20 Oct 2020 at 22:22, John De Witt jhdewitt@gmail.com wrote:

--

Wojciech Owczarek