Would a TSB (Tab Separated Value) format be preferable?  Full-stops
and commas are used in numbers as decimal and thousands separators (or
vice versa), so using tab character would avoid any problems with
commas in the actual data (and make it is a bit easier to quickly
eyeball when viewed in a text editor).

Peter  (G8ZZR, London, England)

On 9 January 2011 17:15, Bob Camp lists@rtty.us wrote:
...

...

We have mountains of data here too.  The best why to store it is in a
"real" database of some kind.  There are several that are free, open
source and multi-platform.  The best for this use is "Postgres".  As
this is free and open source there is no reason not to use it.

In the past I've kept snapshots for simulations that have run for
hours/days/weeks and we got many hundreds of millions of data points.
Then we are able to query for almost any conditions and expression,
for example "Give me a A, B where A-B less than 4 from July 5th 1998"

I can tell you first hand that having a billion lines of tab separated
data is worse than useless.  You need itcataloged such that you can
very quickly (seconds) find useful subsets of the data and you can
NEVER know in advance what subset you might need.

On Mon, Jan 10, 2011 at 12:22 PM, Peter Vince pvince@theiet.org wrote:

--

Chris Albertson
Redondo Beach, California

The counter argument is with a heavyweight database - the size of the datastore increases dramatically and there is no guarantee that the tool will be around in 10 years to read the data.

All SQL databases use ASCII format CSV to load and dump the data from their internal data representation.

Transactional systems still use a hierarchical database 'think IBM IMS or RAIMA' to store and access large datasets like CC auth.  These databases are one step away from ASCII or EBCDIC

Scott
Sent from my Verizon Wireless BlackBerry

-----Original Message-----
From: Chris Albertson albertson.chris@gmail.com
Sender: time-nuts-bounces@febo.com
Date: Mon, 10 Jan 2011 12:42:03
To: Discussion of precise time and frequency measurementtime-nuts@febo.com
Reply-To: Discussion of precise time and frequency measurement
time-nuts@febo.com
Subject: Re: [time-nuts] Archiving Timing Data

We have mountains of data here too.  The best why to store it is in a
"real" database of some kind.  There are several that are free, open
source and multi-platform.  The best for this use is "Postgres".  As
this is free and open source there is no reason not to use it.

In the past I've kept snapshots for simulations that have run for
hours/days/weeks and we got many hundreds of millions of data points.
Then we are able to query for almost any conditions and expression,
for example "Give me a A, B where A-B less than 4 from July 5th 1998"

I can tell you first hand that having a billion lines of tab separated
data is worse than useless.  You need itcataloged such that you can
very quickly (seconds) find useful subsets of the data and you can
NEVER know in advance what subset you might need.

On Mon, Jan 10, 2011 at 12:22 PM, Peter Vince pvince@theiet.org wrote:

--

Chris Albertson
Redondo Beach, California

time-nuts mailing list -- time-nuts@febo.com
To unsubscribe, go to https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts
and follow the instructions there.

There is a difference between archival format and database format. If you
are looking for an archival format that is portable, then a CSV (or other
delimiter of your choice) is ideal. They are easy to import to a real
database and compress well. If, on the other hand, you are looking for a
working database, you're better of putting it into some kind of schema in a
Real Database(TM) of your choice and then tuning for transactional or data
warehouse performance.

In this case, a simple one or two table schema and indexes on the things you
want to sort on, should take care of most of the storage problem. Once you
have that, use the API for the DB of choice to store/retrieve the data.

MySQL is free and runs on pretty much everything nowadays. That plus
myphpadmin would make it easy enough for most of those bright enough to
understand the content of this list to come up with a schema.

Bob (whose day job is with a big red database company)

On Mon, Jan 10, 2011 at 4:57 PM, scmcgrath@gmail.com wrote:

On Mon, Jan 10, 2011 at 2:08 PM, Bob Bownes bownes@gmail.com wrote:

If it needs to be "imported" it will never be used.    No one will
ever bother to take the time to import gigabytes of compressed data.
Don't worry about the space 1TB disks now sell for $99 and the price
trend s downward.

CSV is easy to import, just like bricks are easy to move, easy until
you have a billion or two of them.  We have data archives here and
from experience,  you have to really, really want to look at the data
if it needs to be imported into a tool.  The other thing is that over
time if the data is not in usable format people forget they have it or
what it is.

--

Chris Albertson
Redondo Beach, California

Hi

In a normal industrial setting, one would make a few decisions and start
pushing data in a direction.

In this case, I'd like to look at everybody's data. That makes a bit more
complex problem. If it's an "archive and import" approach, the process has
to work on autopilot. Customizing the import parameters for each data set
isn't going to work out.

If it's a dump and dupe on whateverSQL, we better be ready for 5.1.51 goes
to 5.1.52 and the format changes just a bit stuff. That process keeps on
going on. I dump it under 5.1.51, you later try an import under 5.1.2357 -
trouble.

Not an easy problem to solve. I've done it many ways, and none of them are
perfect.

Bob

-----Original Message-----
From: time-nuts-bounces@febo.com [mailto:time-nuts-bounces@febo.com] On
Behalf Of Chris Albertson
Sent: Monday, January 10, 2011 5:41 PM
To: Discussion of precise time and frequency measurement
Cc: scmcgrath@gmail.com
Subject: Re: [time-nuts] Archiving Timing Data

On Mon, Jan 10, 2011 at 2:08 PM, Bob Bownes bownes@gmail.com wrote:

If it needs to be "imported" it will never be used.    No one will
ever bother to take the time to import gigabytes of compressed data.
Don't worry about the space 1TB disks now sell for $99 and the price
trend s downward.

CSV is easy to import, just like bricks are easy to move, easy until
you have a billion or two of them.  We have data archives here and
from experience,  you have to really, really want to look at the data
if it needs to be imported into a tool.  The other thing is that over
time if the data is not in usable format people forget they have it or
what it is.

--

Chris Albertson
Redondo Beach, California

time-nuts mailing list -- time-nuts@febo.com
To unsubscribe, go to
https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts
and follow the instructions there.

You saved me a lot of typing. :) Comments inline...

--- On Mon, 1/10/11, Bob Bownes bownes@gmail.com wrote:

Indeed. I was thinking that at the very least you ALWAYS want the meta-data of your timeseries in a database, index that so you can easily query it to find your timeseries of interest. With meta-data I mean things like general description, instruments used, date, length of time series, useful statistical measures of the overall timeseries, stuff like that. You can query this to find the timeseries of choice, and then retrieve.

Retrieval would depend a lot on your usecases. Do you just find the right measurements, and feed that to your file based utility? Do you want to perform more complicated database queries? Do you /neeeeed/ a webbased frontend?

IMO the 2 most obvious solutions for something like this would be either A) return the URI with the file location (csv with delimiter du jour), or B) store the full timeseries in the database as well and use your API of choice to retrieve the data.

Again, the choice depends on the type of operation and frequency of operation you want to perform on your data.

Agreed. I noticed RDD being recommended. Don't do it. Why not? Because. RDD is pretty good for what it was made to do. I have plenty of installs around together with mrtg or rddtool, but I would not recommend using it for something like this.

As for postgresql, personally I would use postgresql due to some personal preferences. That said, for regular desktop users without this bias I would recommend using mysql. Things like myadmin and phpmyadmin make life easy... Plus, should one decide to make the obligatory webbased frontend, mysql is slightly friendlier for the php novice. Not to put everyone in the "beginning user" category, but more to keep the gate open to as many people as possible as it were.

Fred (whose day job involved staring in disbelief at poorly conceived schema's)