ADEV on data with gaps
I've got a series of clock measurements where the actual measurement
occasionally has problems (e.g. it's measured against GPS, and there are
GPS partial outages).
What's a strategy for calculating ADEV (and related parameters) where
there are data in the sequence that are excised? For example, does
Timelab or Stable32 have a way to have timestamps for the
frequency/phase measurements that recognizes gaps?
At a first glance, it seems that the calculation for each tau will have
a different number of measurements contributing. Say I have 1000
seconds of data, once a second, and there's a gap of 100 seconds
somewhere in the middle. For the 1 second tau, I've got 900
measurements. for a 100 second tau, I'll have 800 measurements (because
for 200, one or the other end point is missing), etc.
Some plots attached showing some data. In this case, I'm looking at a
post calculated "clock bias" between GPS and the clock - that is,
there's a process that estimates the clock offset (in meters). A first
difference gets rate, and clock has some relatively fixed bias (that
changes with time), and random variations on top of that. However, the
measurement depends on the SNR of the GPS signals (lower plot on
biasdrift, it's the measurement uncertainty estimate)
At some point, of course, the ADEV measurement is limited by the GPS
measurement uncertainty - if it's 1 ns, then ADEV is never going to
measure less than 1E-9 over 1 second.
On 8/4/23 9:15 AM, Lux, Jim via time-nuts wrote:
I've got a series of clock measurements where the actual measurement
occasionally has problems (e.g. it's measured against GPS, and there
are GPS partial outages).
What's a strategy for calculating ADEV (and related parameters) where
there are data in the sequence that are excised? For example, does
Timelab or Stable32 have a way to have timestamps for the
frequency/phase measurements that recognizes gaps?
At a first glance, it seems that the calculation for each tau will
have a different number of measurements contributing. Say I have 1000
seconds of data, once a second, and there's a gap of 100 seconds
somewhere in the middle. For the 1 second tau, I've got 900
measurements. for a 100 second tau, I'll have 800 measurements
(because for 200, one or the other end point is missing), etc.
Some plots attached showing some data. In this case, I'm looking at a
post calculated "clock bias" between GPS and the clock - that is,
there's a process that estimates the clock offset (in meters). A first
difference gets rate, and clock has some relatively fixed bias (that
changes with time), and random variations on top of that. However,
the measurement depends on the SNR of the GPS signals (lower plot on
biasdrift, it's the measurement uncertainty estimate)
At some point, of course, the ADEV measurement is limited by the GPS
measurement uncertainty - if it's 1 ns, then ADEV is never going to
measure less than 1E-9 over 1 second.
Digging into Stable32 (or more specifically
http://www.wriley.com/Gaps.htm) it appears that it's doable. Now I just
have to try generating a data file with the right format to ingest.
Dear Jim.
The best current strategy to calculate ADEV with datagaps was presented
by David Howe and colleagues recently in 2021 and 2022.
One of the published articles is this:
https://www.academia.edu/79180802/Time_series_Imputation_Algorithm
I don't think we where able to include it into the IEEE 1139 and 1193,
but I will be happy to be proven wrong. If not, we will for sure include
it in next revision.
Stable32 does not have that algorithm implemented, Stable32 rather treat
gaps by not calculating the squared value if any of the phase samples is
missing. Demetrios asked about this some time back on the list. I think
Timelab does the same in best case.
The imputation algorithm uses the same thinking as the TOTALDEV, and
consider the noise of each datasample, as independent when you have
second differentiation, and with independence you can move them around
as long as you do not make data re-appear twice. The approach chosen is
to reflect samples into the sequence in reverse order. In TOTALDEV you
mirror them out around the window you analyze.
The imputated time sequence can then be processed by ADEV and friends as
usual. It has been checked for robustness and have much better
robustness to loss of data than previous methods. See article.
This work created quite a buzz when it came, and for a good reason as it
is an usually strong step forward that addresses actual concerns for the
type of opertional context you describe, where continuous monitoring is
not quite possible.
I would also like to raise your awareness of the article by Prof.
François Vernotte et al "Uncertainties of drift coefficients and
extraplolation errors: Application to clock error prediction" in
Metrologia Feb 2001. This is relevant to understand how uncertainty of
phase due to various random noises behave. The gap from measuring to
when they appear again follow these rules. They align up well with
simulations I have done. I was able to contribute this article into the
IEEE 1139 and 1193 standards, and it was done to provide support for
GALILEO.
Please let me know if I can be of further assitance.
Cheers,
Magnus
On 2023-08-04 18:15, Lux, Jim via time-nuts wrote:
I've got a series of clock measurements where the actual measurement
occasionally has problems (e.g. it's measured against GPS, and there
are GPS partial outages).
What's a strategy for calculating ADEV (and related parameters) where
there are data in the sequence that are excised? For example, does
Timelab or Stable32 have a way to have timestamps for the
frequency/phase measurements that recognizes gaps?
At a first glance, it seems that the calculation for each tau will
have a different number of measurements contributing. Say I have 1000
seconds of data, once a second, and there's a gap of 100 seconds
somewhere in the middle. For the 1 second tau, I've got 900
measurements. for a 100 second tau, I'll have 800 measurements
(because for 200, one or the other end point is missing), etc.
Some plots attached showing some data. In this case, I'm looking at a
post calculated "clock bias" between GPS and the clock - that is,
there's a process that estimates the clock offset (in meters). A first
difference gets rate, and clock has some relatively fixed bias (that
changes with time), and random variations on top of that. However,
the measurement depends on the SNR of the GPS signals (lower plot on
biasdrift, it's the measurement uncertainty estimate)
At some point, of course, the ADEV measurement is limited by the GPS
measurement uncertainty - if it's 1 ns, then ADEV is never going to
measure less than 1E-9 over 1 second.
time-nuts mailing list -- time-nuts@lists.febo.com
To unsubscribe send an email to time-nuts-leave@lists.febo.com
Hello Jim
"At a first glance, it seems that the calculation for each tau will have
a different number of measurements contributing. Say I have 1000
seconds of data, once a second, and there's a gap of 100 seconds
somewhere in the middle. For the 1 second tau, I've got 900
measurements. for a 100 second tau, I'll have 800 ..."
That's the right idea - you just skip the differences y_k+1 - y_k
where there are missing points, so you end up with fewer points at
each averaging time
There are various papers that describe the bias in ADEV etc that this
introduces.
Stable32 deals with gaps
http://www.wriley.com/Gaps.htm
If python is your thing, allantools does too.
Otherwise there are MATLAB tools too
https://github.com/groundstate/tftools
Cheers
Michael
On Sat, Aug 5, 2023 at 6:11 AM Lux, Jim via time-nuts
time-nuts@lists.febo.com wrote:
I've got a series of clock measurements where the actual measurement
occasionally has problems (e.g. it's measured against GPS, and there are
GPS partial outages).
What's a strategy for calculating ADEV (and related parameters) where
there are data in the sequence that are excised? For example, does
Timelab or Stable32 have a way to have timestamps for the
frequency/phase measurements that recognizes gaps?
At a first glance, it seems that the calculation for each tau will have
a different number of measurements contributing. Say I have 1000
seconds of data, once a second, and there's a gap of 100 seconds
somewhere in the middle. For the 1 second tau, I've got 900
measurements. for a 100 second tau, I'll have 800 measurements (because
for 200, one or the other end point is missing), etc.
Some plots attached showing some data. In this case, I'm looking at a
post calculated "clock bias" between GPS and the clock - that is,
there's a process that estimates the clock offset (in meters). A first
difference gets rate, and clock has some relatively fixed bias (that
changes with time), and random variations on top of that. However, the
measurement depends on the SNR of the GPS signals (lower plot on
biasdrift, it's the measurement uncertainty estimate)
At some point, of course, the ADEV measurement is limited by the GPS
measurement uncertainty - if it's 1 ns, then ADEV is never going to
measure less than 1E-9 over 1 second.
time-nuts mailing list -- time-nuts@lists.febo.com
To unsubscribe send an email to time-nuts-leave@lists.febo.com