Ulrich,

Ulrich Bangert skrev:

Well, I was taking on this thread in hope to spread some light and I was
not sure who would learn what. I think I have learned a few things, but
so far it seems my view on things have not changed significantly.

I think I can explain some things.

I thought we would agree on those things.

In the ideal world... yes... but in the real world limits hits us
differently. The definition only says that it is an average, not how an
average should be formed over samples where tau != tau0. A
straightforward interpretation will skip samples, but require a much
longer sequence and we must recall it is just one of several estimators.
Let's make this clear, there is no way to get the real Allan variation,
we can only get estimates and choose among estimators. This becomes
clear as one reads the Bregni book or look at the ITU-T G.810 standard
(get it, it's available for free download as PDF from ITU).

Now, with this in mind, the "Overlapping Allan Variance" was chosen as
the estimator of choice for the AVAR(n*tau0), and for good reasons.

One point of confusion is that AVAR(tau) should not be directly
interpreted as Allan Variance in general, it is actually already defined
and reserved to mean a chosen Allan Variance estimator. This is an
important point when comparing oscillators and systems such as a
standard for measuring oscillators or telecom standards. For other uses
other estimators may be used, but for precision use the overlapped
estimator has been chosen as it better uses the measurement series.

So, to sum things up. You can't get the "propper" Allan variance. You
can only make estimates. The "Original Allan Variance" is a one of many
estimators but to attain good quality it needs a long time-series. The
"Overlapping Allan Variance" is another estimator which has improved
quality for a limitied time series as compared to the "Original Allan
Variance". The terms AVAR and ADEV as specified in several standards
denotes an estimator which is the overlapping estimator, not just any
Allan variance estimator. Notice also that telecom standards require
tau0 to be less than lowest tau, so tau/tau0 is always > 1 as this
ensures quality and allows for the specified highpass filter of 10 Hz,
which is also being debated as being meaningfull or harmfull (see Bregni).

There is still a few lessons to learn on this, but thats about it I think.

So no, 1 s data and 2 s data does not necessarily give the same result,
it depends on which estimator you are using, because you will be using
an estimator regardless of what you think is "right". Thus, if you feel
"Original Allan variance" is right for you, you are in your right to use
it, but it is not the "propper" one to use, and do not confuse it with
AVAR which is a dedicated name for the "Overlapping Allan Variance".

Cheers,
Magnus

Sorry if I'm jumping into this thread late, but this statement
confuses me. Since when is "AVAR" not simply short-hand
for "Allan Variance"?

Next you'll tell me SDEV isn't Standard Deviation because some
self-important standards organization decided otherwise.

/tvb

Tom Van Baak wrote:

Tom

Perhaps the real confusion is between the actual definition of the Allan
variance etc, and the various estimators of the Allan variance etc.

For example:
The so called classical Allan variance is nothing more than an estimator
of the Allan variance.
The so called overlapped Allan variance is a better (more accurate)
estimator of the Allan variance.

Bruce

Tom Van Baak skrev:

Feel free to join in...

Good question. My point being that yes... AVAR is a handy short-hand for
Allan Deviation, but it is also actually a standardised quantity and
several standards actually define it consistently as a particular
estimator. It's a good estimator for being of the Allan Variance family
and CPU-cycles should not prohibit us from using it.

Recall that they are all estimators. I think this is a crutial point to
learn really. Once one has accepted that fact, then taking a look at
which estimator serves me the best becomes the issue of interest, not
"which is the right one?" which is actually an incorrect question in
this context.

So, feel free to short-hand Allan Variance as AVAR, but there are
context in which this is going to be interpreted as being the
overlapping Allan variance estimator and not any other estimator, and
hence using that short-hand can be ambiguous. If we want an unambiguous
use of AVAR, do not use AVAR as short-hand for Allan Variance when using
other estimators than the overlapping one. Do as you please, but now you
are aware of the issue.

Also, look at the NIST SP 1065 for instance, where it is clearly
indicated that the "original" is being superseded in most practical use
for the benefit of the overlapping, giving improved confidence
intervals. Also, Modified Allan Variance and Theo should be considered
as better alternatives.

The SP 1065 should be a good read for many, as it gives a modern
overview and also addresses several practical implementational issues
such as software test-sequences etc. The TN 1337 is a more classic view
and collection of articles.

So... we could argue all we want about "which is the correct Allan
Variance" but it doesn't really help. The original estimator is flawed.
the overlapping estimator improves confidence and the Theo family should
provide even better results.

I could probably find a standard defining it to something completely
different and totally out of context which would not help.

But the reaction is natural, but one must realize that there is not real
"right" here, so sometimes putting down the foot and say "this is what
we define it to be" is needed so that everyone at least do it according
to the same procedures and with known errors... until you realize that
you need something better and move over to some other measurement which
you define in a similar fashion. It's like say the meter standard. "This
is the meter... until we say something different".

Cheers,
Magnus

Magnus Danielson wrote:

Hej Magnus

The major failing of SP1065 is that it doesn't explicitly make it clear
that all the quantities for which its gives formulae are merely
estimators of an underlying statistical property of the frequency source.

For example using the terminology of SP1065:

Classical AVAR
Overlapped AVAR
TOTVAR
MTOT

are all estimators of the Allan variance.
They all have the same Expectation value but the confidence limits for
each of these estimators are different.

MVAR
MTOT

are both estimators of the Modified Allan variance.
They both have the same Expectation value but the confidence limits for
each of these estimators are different.

Bruce

Hej Bruce,

I agree. Other sources is much more careful on this point. SP1065 gives
a great overview, but should not be mistaken as a definite reference in
all aspects.

Bregni for instance presents the definitions in one place (Chapter 5)
and estimators in another (Chapter 7). G.810 does the same for instance.

It should be clear that the definition of Allan variance is actually not
on two samples of frequency, they are measures of two average frequency
measures, which implies an tau-long integral over v(t) for two
end-to-end ranges. Using samples of frequency or samples of time is
doing an estimation or approximation to the actual definition. I don't
recall which paper wrote it, but it was pointed out that the average
line over the y is often missed, but it you look in early papers these
are found and uses, such as the tn121 paper i TN1337.

It takes the reading of alot of papers and books to really make the
picture stand out as subtle points is pointed out here and there. I just
learned that J. J. Snyder is to take credit for the improved overlapped
Allan variance estimator and that this became an inspiration for the
modified Allan variation.

You missed Theo1, just read the beginning of 5.2.15 and the info-box
there. Then the followup is TheoH in 5.2.16.

Agree.

In the end, while I have only very quickly browsed through the material
and picked up some new, I feel it is time to read it through properly
again. I think I have learned a few finer points in the last days and
when seeing it with fresh eyes I think I will appreciate many of the
comments better.

Cheers,
Magnus

Bruce Griffiths wrote:

Addendum:

There would appear to be only 3 underlying frequency stability
statistics of interest in this paper:

1. Allan variance

2. Modified Allan variance

3. Hadamard variance

Each of which has a different phase transfer function.

All other statistical quantities related to frequency stability in this
paper are actually estimators of one of the above underlying statistics
Each estimator has different confidence limits and a different bias
function which may depend on phase noise type.
Correction for the underlying bias of each estimator is desirable before
plotting the results.
Confidence limits are also useful.

Bruce

Tom Van Baak wrote:

Tom

Its important to realise these calculated statistics are all estimators
of the underlying Allan variance.
Thus it is important to assign unique unambiguous names  to each of them
so that some idea of their region of reasonable accuracy and perhaps
their confidence limits can be estimated.

Thus for example:
AVAR doesn't actually denote the underlying Allan variance but rather a
particular algorithm used to estimate it from the data.

OAVAR denotes another algorithm for estimating the underlying Allan
variance from the data

TOTVAR, THEO1 and THEOH denote other algorithms that can be used to
estimate the underlying Allan variance from the data.

Some of these algorithms produce biased estimates of the underlying
Allan variance so correction for such bias is usually necessary.

Bruce

Tom Van Baak skrev:

Depending on the context of course.

Maybe. The J.J. Snyder article is from 1980 where as the Allan deviation
is from 1966. It's only about half-a-generation. Maybe about the age
difference of us two...

This could indicate where he is on the issue, but does not really
resolve the question.

That I agree with, in your context you certainly avoided ambiguity.

Actually... no. I have looked at this problem and you can calculate the
overlapping Allan variance in real time with only the 2m tau of historic
x values in memory. It is fairly trivial to implement out of the
definition. You don't need the full data-set. However, you would need
multiple accumulators for different choices of m. I can see how this
might not is imminently apparent, but given the clues given I think it
will become visible.

The trickier part is to do drift rate compensation without having access
to the full dataset. For Allan variance this is possible with a few
tricks out of the statistics book.

Using such an approach, you could see your AVAR/ADEV develop as the time
series is consumed and be able to see how it stabilizes as you measure
more and more. I think it is a rather pleasing approach.

What I really need to learn is to calculate the confidence intervals. It
keep nagging me all the time.

Cheers,
Magnus