Re: St Veran gravity red-shift misson
Hi all,
So, now this morning we rose early, I was late at 6:30. Quick cleanup of
room, breakfast, take antenna down, reconnect small pin-wheel antenna
for climb down. That part went well.
Driving down the steep road is harder than the climb. During winther
only skis work, and it takes 4 hours. It can be 2-3 m snow. Snow is
good, melted snow becomes water for the station, which is used during
summer. We saw marmottes and mountain deers. Unnecessarily cute.
Fashinating talking to the care-talker, we really hit off well and love
his stories. He lives locally and I now kniw his family.
Clock transfer from Observatoire car to my car went eventless... not. We
lost power again, now for a new reason. I keep learning things, and the
failures is important.
So, I had changed the loading power from 4A to 12A at masimum on the AC
charges while at station. Well, there it is safe. In the car, not so
much, so the 12 VDC to 230 VAC (125 W) ducked and that caused
temporarily outage. I fairly quickly retained power and set charger to
reasonable values and things where fine. However, it required me to
restart the Raspberry Pi, since I had not had the time and energy to
make all the logging work. OK, so hook up screen to RPi for it to boot
properly. Massive problems, and it turns out that the screen is dying.
I managed to get the environmental logging running, cesium logging,
pressure-sensor, and power setup logging. Then pack things up and start
the decend from Saint Veran village.
The first stretch, I let Bruno drive, as he has much more experience in
mountain driving than me.
Part of the travel down, there was a peep. Turns out that everything was
fine, except that the 12 VDC to 230 VAC converter ducked under the heat.
So I turned it of and it works. From here the PV panels was far than
enough, so no worries. This happened a few times more during the travel,
but without major effect on resull.
We stopped for lunch, drove further, and from Grenoble I took over
driving. Filling the tank, getting some soft drinks and such, a drive of
800 km to just outside Mons in Belgium, where we where able to have late
dinner and charge the batteries. Then drive to the Observatoir Royal de
Belgique in Brussles.
Once at place at the Observatory, we had to plan how to get the clock
into the laboratory, again while avoiding to power the clock down.
Through inventive use of extention cables and using a pair of batteries
as power backup and manage the switch of power out-let, we where able to
take it out of the car, roll it down the ramp and then through the
corridors into the time-lab and eventually slid it in just under the
Cesium 1 of the ORB cesium rack.
The safe transfer over to laboratory location and power was successful
and at 03:00 we could leave the site.
While the adventure contains several events, the combinations of
measures alows us to compensate for iit to some degree. The big and
small failures is to learn from, and sharing the gory details of
experience is part of the learning experience. Having just 3 weeks on
the spare time to build a setup like this is not enough. It remains the
main source for the failure.
The general concept of power, GNSS and logging works. Especially power
is challenging and being able to alternate between solar panels, AC and
car generator is a contributor to the relative success. The lack of
automation allowed too much of human error in handling. Much of that
ended up being handled without majjor failure of the mission, but it did
mean loosing valuable measurement and scientific strength. The loss of
measurements on low level is a problem, and we then need to recover that
as being measurement after the travel, just to establish the rate of the
clock at sea level. Assuming the clock maintains that rate in it's own
frame of reference, which we then change as we go up and then go down
the mountain, and that we reduce the graviational potential and thus
gravitational acceleration. However to show this we need to know the
rate of the clock at sea level (ORB lab is at about 100 m) to illustrate
the difference in rate. We choose to measure this. Speaking of
measuring, we choose to use GNSS common view measurements using the
PolaRx4TX as main measurement, and the Mosaic-T/X5 as auxillary. The
later would fail to do PNT solutions but continue to measure everything
else. Using the 10 MHz as primary signal, and PPS more as auxillary,
loss of PPS phase was less of a failure if only the clock state was
recovered quickly.
Overall I consider the power-strategy as such as a success. The
implementation leaves room for improvement, but considering it was
whipped together in three weeks while working full time, the failures
and near misses mainly comes from stress and lack of time to integrate
things. I was recommended to use the Victron MPPT and AC-chargers, as
they have known good quality and also an open and open-source oriented
view on things. I had no previous experience of these myself. The MPPT
solar panel charger is accessable over both VE Direct electrical
interface and over BlueTooth. While I usually is not overly ethusiastic
for BlueTooth, this time I was extremely happy to have the ability to
monitor over BlueTooh. Using a VE Direct USB adaptor cable, I was able
to integrate the monitoring to the InfluxDB and Grafana infrastructure
relatively well. That however only happend once up on the Observatory. I
should be able to integrate over the BlueTooth, but I did not have the
time and energy to do that, but it will be the next thing to attempt,
when I reach my summer house.
The thing which really worked well with the power system was the dynamic
capabilities of it. The MPPT charger has three power connections, the PV
solar panel, the batteries and then the load. Towards the solar panel,
it optimize the load by adjusting the current drawn, to find the optimum
V/I operational spot to achieve the maxium output power P for each
condition. When panels have too little energy, it turns it of. According
to Victron recommendations, an AC charger can be added by hooking it in
directly on the batteries. What is missing is a better integration
between these. While the MPPT was doing great, and the AC charger was
doing great while operation each on it's own, the cut-over between them
and more importantly handling of the different sources of AC power
(house or car) with different charger conditions allowed for human
error, as will be illustrated. On both the battery side and the load
side I used a RigRunner 4008H power-pole panel. I used power-pole for
all 24 V power connections. This allowed me to use unusused cables as
backup, it also allowed me to hook up a pair of batteries to directly
drive the cesium when needed either as UPS or as DC source when
switching AC connector.
Power failures did happen, and here is a list of the cesium failures:
- Lab setup
When testing the DC feed to the cesium, pulling the AC failed.
Cause: Forgot to connect the DC cables all the way.
Root cause: Human error, forgot to check and verify the DC level.
Comment: The cesium does not have an indicator of the power status of
the AC and DC. It does the power-switching well, but it would help to know.
- Transfer from lab to car.
When transfering the setup to the car, in itself a dress rehearsal and
test of the concept of moving it over, the cesium failed.
Cause: Connecting the DC cable to the distribution channel was done to a
connector with a fuse of 1 A and the cesium pulls about 2 A. Blown fuse
expected. Fuse and blown-fuse indication worked.
Root cause: Stress contributed to this, this was done just as I was
about to drive 600 km to the summer house.
Recommended correction: Having planned and marked the intended
connection point on the panel to the cesium would have avoided the issue.
- Short circuit in Grenoble
Driving down from summerhouse to outside Mons was a 1100 km drive. That
was successful. Then driving down to Grenoble was in itself also
successful. However, lack of sleep was a contributor. When in Grenoble
issues had been detected and a failure of the setup experienced. Part of
the setup removed to secure the experiment.
Cause: A cable had come loose, so a +24V cable made connections it
should not have done. Fuses blown etc.
Root cause: The haste by which everything was assembled prooved that
everything was not connected well enough, strapped down enough for the
isolation to work. Re-connecting it to the small DC/DC adaptor (for
+5V), with more ZIP-ties to secure cables and then isolation and
location worked.
Recommended correction: Have more time to spend on securing cables and
better isolation would have resolved this.
- Drop-out in Grenoble village
Having charged the batteries at the station, and then transfer the setup
to the car to have it under charge during the night was done to ensure
successful transfer. During transfer between the cars, operating on my
car's 12V using the 12V to 230 VAC adaptor, the clock can be transfered
without batteries. However, this is done by driving the AC adaptor using
the generated AC. This has been a good strategy for all transfers. This
drives the 24V bus and everything else depends on it. However, this time
it failed.
Cause: The AC charger had been put in 12A charging mode and high voltage
level while as the observatory, since on observatory power (in itself a
large PV array and battery bank) the load on the 125W DC to AC convertor
made it duck, and with that, the 24V system when batteries was
disconnected. Reverting to battery shortend the outage. Reducing setting
on AC avoided the next power outage.
Root cause: Human error and integration error. Having the AC charger act
both as supply for AC and car DC source (though DC/AC converter) made
integration harder. There was a good separate car DC outlet, but no
suitable 12V to 24V switch supply available, that would have avoided the
integration problem with 12V load and 24V load. Integration could have
avoided more of the human error and steered things better.
Recommended fixes:
4.1 Check-lists could have reduced the risk.
4.2 Better setup would have reduced the risk.
4.3 Better integration would have reduced the risk.
4.4 Using the AC supplies ability to operate as power supply rather than
charger would have changed the outcome.
4.5 Keeping charging/supply also from observatory car would have reduced
the problem.
4.6 More time to integrate and test
While this is not an exhaustive set of problems, it illustrates well how
lack of time and stress enabled human error and deficiencies in the
setup to cause failures. However, considering the many transfer of
power, clock transfers, dynamic handling of setups, carrying etc. the
general concept itself seems to work really well. It just needs to work
out it's teething problems.
I chose to be brutally honest, because then there is more to write and
more to learn. I chose to focus on the power in this post, but logging
had it's set of challenges. Also, cause of problem was that the screen
used with the raspberry pi is about to die, which was realized only on
decent in Saint Veran village.
I intend to report more as I have arrived at my summerhouse. I think the
logging would be good to explain and do a post mortem on.
It's only by making an honest attempt that one can test how teories work
in practice. It is only by honestly report and do full post-mortem that
the failures can be converted into useful experience and enable en even
smoother operating setup on next run.
Some of what I've build, is really what should go into my basement
time-lab too. The difference is really not that large.
I will close the laptop, brush my teeth and pack up to leave the
smallest hotel I've ever stayed at.
Cheers,
Magnus
On 2023-07-23 00:57, James Littlefield wrote:
Enjoy your feast. It has been fascinating reading your chronicles of
the experiment.
On Sat, Jul 22, 2023 at 6:54 PM Magnus Danielson via time-nuts
time-nuts@lists.febo.com wrote:
Hi all,
Our time here starts to run out. This is bitter-sweet, because it
means
leaving a really nice place with a boatload of new friends, good
times
and for me a whole amount of astronomy I did not expect to see.
After doing the last show for the turists, we have moved the clock
and
batteries over to the observatory car where it will sit over night.
Power from the house and maintain the choke ring on the top. All to
minimize the things we need to do tomorrow morning, since we need to
raise early and then travel down to the village, and then transfer
the
clock back to my car, and then drive across France and into
Brussels, to
move the clock into the Observatoir Royal de Belgique lab.
It's been very eventless here, but things where fixed in the
beginning.
So, you would expect things to be fine... well, just as we loaded the
clock and locked the car, we saw that one tire is flat, so right now
work to change the tire. Luckily we have a spare tire. So, the usual
mess ongoing. Had to unload some, but not too much.
I have tried to use a TICC, logged a lot of data in parallel to
the GNSS
common view, and it turns out that the least square fit is for
ridicolous litte frequency shift with about 3E-16, so that is
probably
just the same clock looped back. Ah well. The integration works. What
does not work is signal on ChB so I need to investigate that as I
come home.
We have learned immensly by attempting this project, and just
finishing
it we try to use the knowledge.
I'll setup a webpage on my server to describe as much as I can.
The hobby-astronomers have various other background, such as one
being a
math professor, so I showed him my fast least square work, just to
see
what he thought. Kind words was exchanged. Other than that, lots of
physics, math etc. have been discussed.
Car tire changed, things reloaded. Everything is fine.
Now we prepare for the final night feast.
Cheers,
Magnus
On 2023-07-19 03:06, Christopher Hoover wrote:
> Huzzah! Yes, please send updates.
>
>
> On Sun, Jul 16, 2023 at 1:41 AM Magnus Danielson
<magnus@rubidium.se>
> wrote:
>
> Dear Christoffer,
>
> Many thanks. Yesterday we maanged to drive from Grenoble up into
> St Veran village, the highest village in Europe. This had me
> driving mountain roads like nothing I've ever done before, so it
> was a very intense experience. I even drove through Italy,
so the
> second new country for me to drive in.
>
> From St Veran village my car was not suitable, as the final
climb
> is on a road which is not very good, so we had to use the
> Observatory jeep and transfer the clock over to the shock
> absorber. This is quite an elaborate dance, but thanks to the
> modularity of the setup where where able to safely transfer
it up.
> We used a small pin-wheel antenna to record the remaining rise.
>
> At the observatory, I made sure that batteries was recharged
> before we transitioned furteher. It was quite an elaborate dance
> to move it up into the lab. We also installed a choke ring
antenna
> on the roof of the observatory.
>
> I use my Mosaic-T and ORB brought their PolaRx4TR. This provide
> redundancy, which is good, since we discovered that the
cable from
> the antenna-splitter over to the PolaRx4TR can have a loss
of contact.
>
> Having spare cables and adapters have helped, so we had to be a
> bit innovative to extend the cable, but I weather secured the
> extension setup, so we should be fine.
>
> We maintained the cesium operational from Grenoble. I will
> investigate when we lost state, but it was before Grenoble,
so we
> should have good data from that, even if Cesium date and time is
> off, we do not measure that so I can adjust that later today.
>
> I now go into a clean-up, so start additional loggings I did not
> managed to start manually because of stress. I aim to make those
> autostart, it's not hard, but one had to prioritize hard.
Power to
> the cesium is key, GNSS recording operational second, then
logging
> of cesium, environment, PV & batteries further down the
line. The
> key to mission have been achieved. I can clean up the other
stuff
> while here. I want that additional environmental and system
> logging to work for the rest of the mission.
>
> So, we have not failed the mission yet. It's not been
perfect, but
> with some clean-up action and experience built, we can improve
> robustness.
>
> The Observatoir de Saintt Veran is originally built for
> Observatoir de Paris, and they used to have two domes in Paris,
> but the light poultion makes it fairly useless. So, they built
> this observatory and moved one of their domes up here. Yesterday
> late, they where walking to that dome saying "Want to see
Saturn?"
> and I just tagged along. It was not the best of days, since
it was
> very windy and that makes the atmosphere more disturbed. But the
> night sky here is spectacular.
>
> Another nice thing here is that up here there is a small animal
> called Marmote. I've never seen one, even in zoo, and on the
drive
> up we could see plenty of them. These are very shy animals, so
> they run away as we came driving up. However, occasionally they
> come up to the observatory. Just the other day, one Mermote
walked
> over the observatory keepers foot, not run, but walked. So, we
> joked that the Marmotes do not fear the scientists. :)
>
> We where able to come up with the clocks a day early, so we
got to
> interact with the previous mission. Friendly and hospital
people.
> So they shared the reminder of their dinner food, cheese and
wine
> with us. The hobbyist astronomers can apply for one-week
missions
> to Astro Queyras that. Depending on the quality of the mission,
> they get access to the site for a week, and good mission
gets the
> good weeks, with minimal moon light polution. Normally it is
full
> isolation, but gaining another day of integrating up the
frequency
> error they bent the rules for us, and me and Bruno could come
> early. We are very happy as we also got to interact with the
> preivous mission here.
>
> If there is interest, I can keep sending updates. Yesterday was
> hectic, so it was not the time to pull out data. That work
we can
> start doing today. I will also do a write-up of the setup if
> people are interested.
>
> So, I was given the oppertunity to do this with very short
notice,
> and I jumped on the oppertunity. It's been very stressful
and hard
> work to come to this point, but now I can start to enjoy it.
Also,
> time to share the experience.
>
> Today will the rest of the total mission come up, I only work on
> the graviational red-shift mission, but the full mission
involves
> astronomy missions naturally, such as spectroskopy. There is
also
> a documentation side, so things is filmed etc.
>
> Cheers,
> Magnus
>
> On 2023-07-16 06:48, Christopher Hoover wrote:
>> Best of luck, Magnus.
>>
>> On Fri, Jul 14, 2023 at 4:04 PM Magnus Danielson via time-nuts
>> <time-nuts@lists.febo.com> wrote:
>>
>> Fellow time-nuts,
>>
>> So, 5 years ago I was asked by Observatoir Royal de
Belgique
>> to take
>> care of two failed 5071A cesiums and alter them so that
there
>> is one
>> functioning. I did that, and have operated the functioning
>> one here and
>> there, but conservatively, since they wanted to do a
mission
>> with
>> graviational red-shift.
>>
>> Since them, other things happen in life, such as Covid,
>> prohibiting
>> work. However, a few weeks ago I was contacted as they
where
>> about to go
>> on the mission. I thought it would be nice to join, as
I had
>> quite a bit
>> of the things needed for such mission.
>>
>> Setting it up meeds a number of challenges, as it needs
to be
>> powered
>> continuously, and over a varity of sources.
>>
>> In addition I wanted to log as much data as possible. I've
>> come far on
>> that point. I log 5071A state and several other environment
>> sensors into
>> an InfluxDB and then illustrate with Grafana. Independently
>> two GNSS
>> receivers log things for post-processing.
>>
>> Just assembling the rig for the two 100 W PV panels and a
>> choke-ring
>> antenna has been a challenge. Let's just say that I should
>> not be hired
>> to do fine mechanics work.
>>
>> Things have been fixed during the travel, and I have driven
>> 600+1100+800
>> km just to be in base-camp before final climb. I'm now
>> sitting in a
>> borrowed house just outside of Grenoble. I just had to tear
>> out part of
>> the equipment, as one of the power converters failed.
Luckily
>> that side
>> was redundant for the mission, but I do miss one pressure
>> sensor to
>> compare the small one with.
>>
>> I had intended for my passive hydrogen maser to also do the
>> trip, but it
>> woke up with an unexpected problem and did not lock, and I
>> decided it
>> was not meaningful to bring it's dead weight along. I will
>> have to
>> investigate the actual fault. Also, I have been unable
to log
>> the
>> internal state, which Would have given valuable clues.
>>
>> Tomorrow we do the final climb to Observatoir de St
Veran at
>> 2930 m and
>> stay there for 8 days.
>>
>> Connectivity can be sketchy.
>>
>> In the end of the day, it is only if you try that you can
>> either fail or
>> succeed. So far, it has not yet failed completely
>> catastrofically, but
>> warnings-signs has gone of and they been managed, so far.
>> This can still
>> fail spectacular.
>>
>> Now I badly need my bed. Being sleep deprived does not help
>> thinking.
>>
>> Cheers,
>> Magnus
>> _______________________________________________
>> time-nuts mailing list -- time-nuts@lists.febo.com
>> To unsubscribe send an email to
time-nuts-leave@lists.febo.com
>>
_______________________________________________
time-nuts mailing list -- time-nuts@lists.febo.com
To unsubscribe send an email to time-nuts-leave@lists.febo.com
--
Jim WN1X
Hi
Wow, that’s quite a story. Thanks for sharing !!!!
While this is a bit off topic in terms of timing:
Victron does sell “central control” computers to tie their gear together. Some are in the
< $200 range. They will let you manage an ensemble of devices effectively. It also is
worthwhile to tie in something like a BMV-712 shunt based monitor. That will let you
evaluate the battery(s) state accurately.
There are folks who make “hats” for the RPi that do backup power as well as DC/DC
conversion. They claim they are pretty robust. No idea if they would be robust enough
for this sort of thing.
Lithium batteries, while still expensive, are getting less so every day. They are way lighter
than lead acid if you are going to lug them here and there. The Victron gear is just as
happy running them as running various forms of lead acid. They also will happily sell
you setups for 24V if you decide that works better than 12V.
As you might guess, there’s a lot of Victron gear in my RV’s …..
Bob
On Jul 24, 2023, at 4:06 AM, Magnus Danielson via time-nuts time-nuts@lists.febo.com wrote:
Hi all,
So, now this morning we rose early, I was late at 6:30. Quick cleanup of room, breakfast, take antenna down, reconnect small pin-wheel antenna for climb down. That part went well.
Driving down the steep road is harder than the climb. During winther only skis work, and it takes 4 hours. It can be 2-3 m snow. Snow is good, melted snow becomes water for the station, which is used during summer. We saw marmottes and mountain deers. Unnecessarily cute. Fashinating talking to the care-talker, we really hit off well and love his stories. He lives locally and I now kniw his family.
Clock transfer from Observatoire car to my car went eventless... not. We lost power again, now for a new reason. I keep learning things, and the failures is important.
So, I had changed the loading power from 4A to 12A at masimum on the AC charges while at station. Well, there it is safe. In the car, not so much, so the 12 VDC to 230 VAC (125 W) ducked and that caused temporarily outage. I fairly quickly retained power and set charger to reasonable values and things where fine. However, it required me to restart the Raspberry Pi, since I had not had the time and energy to make all the logging work. OK, so hook up screen to RPi for it to boot properly. Massive problems, and it turns out that the screen is dying. I managed to get the environmental logging running, cesium logging, pressure-sensor, and power setup logging. Then pack things up and start the decend from Saint Veran village.
The first stretch, I let Bruno drive, as he has much more experience in mountain driving than me.
Part of the travel down, there was a peep. Turns out that everything was fine, except that the 12 VDC to 230 VAC converter ducked under the heat. So I turned it of and it works. From here the PV panels was far than enough, so no worries. This happened a few times more during the travel, but without major effect on resull.
We stopped for lunch, drove further, and from Grenoble I took over driving. Filling the tank, getting some soft drinks and such, a drive of 800 km to just outside Mons in Belgium, where we where able to have late dinner and charge the batteries. Then drive to the Observatoir Royal de Belgique in Brussles.
Once at place at the Observatory, we had to plan how to get the clock into the laboratory, again while avoiding to power the clock down. Through inventive use of extention cables and using a pair of batteries as power backup and manage the switch of power out-let, we where able to take it out of the car, roll it down the ramp and then through the corridors into the time-lab and eventually slid it in just under the Cesium 1 of the ORB cesium rack.
The safe transfer over to laboratory location and power was successful and at 03:00 we could leave the site.
While the adventure contains several events, the combinations of measures alows us to compensate for iit to some degree. The big and small failures is to learn from, and sharing the gory details of experience is part of the learning experience. Having just 3 weeks on the spare time to build a setup like this is not enough. It remains the main source for the failure.
The general concept of power, GNSS and logging works. Especially power is challenging and being able to alternate between solar panels, AC and car generator is a contributor to the relative success. The lack of automation allowed too much of human error in handling. Much of that ended up being handled without majjor failure of the mission, but it did mean loosing valuable measurement and scientific strength. The loss of measurements on low level is a problem, and we then need to recover that as being measurement after the travel, just to establish the rate of the clock at sea level. Assuming the clock maintains that rate in it's own frame of reference, which we then change as we go up and then go down the mountain, and that we reduce the graviational potential and thus gravitational acceleration. However to show this we need to know the rate of the clock at sea level (ORB lab is at about 100 m) to illustrate the difference in rate. We choose to measure this. Speaking of measuring, we choose to use GNSS common view measurements using the PolaRx4TX as main measurement, and the Mosaic-T/X5 as auxillary. The later would fail to do PNT solutions but continue to measure everything else. Using the 10 MHz as primary signal, and PPS more as auxillary, loss of PPS phase was less of a failure if only the clock state was recovered quickly.
Overall I consider the power-strategy as such as a success. The implementation leaves room for improvement, but considering it was whipped together in three weeks while working full time, the failures and near misses mainly comes from stress and lack of time to integrate things. I was recommended to use the Victron MPPT and AC-chargers, as they have known good quality and also an open and open-source oriented view on things. I had no previous experience of these myself. The MPPT solar panel charger is accessable over both VE Direct electrical interface and over BlueTooth. While I usually is not overly ethusiastic for BlueTooth, this time I was extremely happy to have the ability to monitor over BlueTooh. Using a VE Direct USB adaptor cable, I was able to integrate the monitoring to the InfluxDB and Grafana infrastructure relatively well. That however only happend once up on the Observatory. I should be able to integrate over the BlueTooth, but I did not have the time and energy to do that, but it will be the next thing to attempt, when I reach my summer house.
The thing which really worked well with the power system was the dynamic capabilities of it. The MPPT charger has three power connections, the PV solar panel, the batteries and then the load. Towards the solar panel, it optimize the load by adjusting the current drawn, to find the optimum V/I operational spot to achieve the maxium output power P for each condition. When panels have too little energy, it turns it of. According to Victron recommendations, an AC charger can be added by hooking it in directly on the batteries. What is missing is a better integration between these. While the MPPT was doing great, and the AC charger was doing great while operation each on it's own, the cut-over between them and more importantly handling of the different sources of AC power (house or car) with different charger conditions allowed for human error, as will be illustrated. On both the battery side and the load side I used a RigRunner 4008H power-pole panel. I used power-pole for all 24 V power connections. This allowed me to use unusused cables as backup, it also allowed me to hook up a pair of batteries to directly drive the cesium when needed either as UPS or as DC source when switching AC connector.
Power failures did happen, and here is a list of the cesium failures:
- Lab setup
When testing the DC feed to the cesium, pulling the AC failed.
Cause: Forgot to connect the DC cables all the way.
Root cause: Human error, forgot to check and verify the DC level.
Comment: The cesium does not have an indicator of the power status of the AC and DC. It does the power-switching well, but it would help to know.
- Transfer from lab to car.
When transfering the setup to the car, in itself a dress rehearsal and test of the concept of moving it over, the cesium failed.
Cause: Connecting the DC cable to the distribution channel was done to a connector with a fuse of 1 A and the cesium pulls about 2 A. Blown fuse expected. Fuse and blown-fuse indication worked.
Root cause: Stress contributed to this, this was done just as I was about to drive 600 km to the summer house.
Recommended correction: Having planned and marked the intended connection point on the panel to the cesium would have avoided the issue.
- Short circuit in Grenoble
Driving down from summerhouse to outside Mons was a 1100 km drive. That was successful. Then driving down to Grenoble was in itself also successful. However, lack of sleep was a contributor. When in Grenoble issues had been detected and a failure of the setup experienced. Part of the setup removed to secure the experiment.
Cause: A cable had come loose, so a +24V cable made connections it should not have done. Fuses blown etc.
Root cause: The haste by which everything was assembled prooved that everything was not connected well enough, strapped down enough for the isolation to work. Re-connecting it to the small DC/DC adaptor (for +5V), with more ZIP-ties to secure cables and then isolation and location worked.
Recommended correction: Have more time to spend on securing cables and better isolation would have resolved this.
- Drop-out in Grenoble village
Having charged the batteries at the station, and then transfer the setup to the car to have it under charge during the night was done to ensure successful transfer. During transfer between the cars, operating on my car's 12V using the 12V to 230 VAC adaptor, the clock can be transfered without batteries. However, this is done by driving the AC adaptor using the generated AC. This has been a good strategy for all transfers. This drives the 24V bus and everything else depends on it. However, this time it failed.
Cause: The AC charger had been put in 12A charging mode and high voltage level while as the observatory, since on observatory power (in itself a large PV array and battery bank) the load on the 125W DC to AC convertor made it duck, and with that, the 24V system when batteries was disconnected. Reverting to battery shortend the outage. Reducing setting on AC avoided the next power outage.
Root cause: Human error and integration error. Having the AC charger act both as supply for AC and car DC source (though DC/AC converter) made integration harder. There was a good separate car DC outlet, but no suitable 12V to 24V switch supply available, that would have avoided the integration problem with 12V load and 24V load. Integration could have avoided more of the human error and steered things better.
Recommended fixes:
4.1 Check-lists could have reduced the risk.
4.2 Better setup would have reduced the risk.
4.3 Better integration would have reduced the risk.
4.4 Using the AC supplies ability to operate as power supply rather than charger would have changed the outcome.
4.5 Keeping charging/supply also from observatory car would have reduced the problem.
4.6 More time to integrate and test
While this is not an exhaustive set of problems, it illustrates well how lack of time and stress enabled human error and deficiencies in the setup to cause failures. However, considering the many transfer of power, clock transfers, dynamic handling of setups, carrying etc. the general concept itself seems to work really well. It just needs to work out it's teething problems.
I chose to be brutally honest, because then there is more to write and more to learn. I chose to focus on the power in this post, but logging had it's set of challenges. Also, cause of problem was that the screen used with the raspberry pi is about to die, which was realized only on decent in Saint Veran village.
I intend to report more as I have arrived at my summerhouse. I think the logging would be good to explain and do a post mortem on.
It's only by making an honest attempt that one can test how teories work in practice. It is only by honestly report and do full post-mortem that the failures can be converted into useful experience and enable en even smoother operating setup on next run.
Some of what I've build, is really what should go into my basement time-lab too. The difference is really not that large.
I will close the laptop, brush my teeth and pack up to leave the smallest hotel I've ever stayed at.
Cheers,
Magnus
On 2023-07-23 00:57, James Littlefield wrote:
Enjoy your feast. It has been fascinating reading your chronicles of the experiment.
On Sat, Jul 22, 2023 at 6:54 PM Magnus Danielson via time-nuts time-nuts@lists.febo.com wrote:
Hi all,
Our time here starts to run out. This is bitter-sweet, because it
means
leaving a really nice place with a boatload of new friends, good
times
and for me a whole amount of astronomy I did not expect to see.
After doing the last show for the turists, we have moved the clock
and
batteries over to the observatory car where it will sit over night.
Power from the house and maintain the choke ring on the top. All to
minimize the things we need to do tomorrow morning, since we need to
raise early and then travel down to the village, and then transfer
the
clock back to my car, and then drive across France and into
Brussels, to
move the clock into the Observatoir Royal de Belgique lab.
It's been very eventless here, but things where fixed in the
beginning.
So, you would expect things to be fine... well, just as we loaded the
clock and locked the car, we saw that one tire is flat, so right now
work to change the tire. Luckily we have a spare tire. So, the usual
mess ongoing. Had to unload some, but not too much.
I have tried to use a TICC, logged a lot of data in parallel to
the GNSS
common view, and it turns out that the least square fit is for
ridicolous litte frequency shift with about 3E-16, so that is
probably
just the same clock looped back. Ah well. The integration works. What
does not work is signal on ChB so I need to investigate that as I
come home.
We have learned immensly by attempting this project, and just
finishing
it we try to use the knowledge.
I'll setup a webpage on my server to describe as much as I can.
The hobby-astronomers have various other background, such as one
being a
math professor, so I showed him my fast least square work, just to
see
what he thought. Kind words was exchanged. Other than that, lots of
physics, math etc. have been discussed.
Car tire changed, things reloaded. Everything is fine.
Now we prepare for the final night feast.
Cheers,
Magnus
On 2023-07-19 03:06, Christopher Hoover wrote:
Huzzah! Yes, please send updates.
On Sun, Jul 16, 2023 at 1:41 AM Magnus Danielson
<magnus@rubidium.se>
wrote:
Dear Christoffer,
Many thanks. Yesterday we maanged to drive from Grenoble up into
St Veran village, the highest village in Europe. This had me
driving mountain roads like nothing I've ever done before, so it
was a very intense experience. I even drove through Italy,
so the
second new country for me to drive in.
From St Veran village my car was not suitable, as the final
climb
is on a road which is not very good, so we had to use the
Observatory jeep and transfer the clock over to the shock
absorber. This is quite an elaborate dance, but thanks to the
modularity of the setup where where able to safely transfer
it up.
We used a small pin-wheel antenna to record the remaining rise.
At the observatory, I made sure that batteries was recharged
before we transitioned furteher. It was quite an elaborate dance
to move it up into the lab. We also installed a choke ring
antenna
on the roof of the observatory.
I use my Mosaic-T and ORB brought their PolaRx4TR. This provide
redundancy, which is good, since we discovered that the
cable from
the antenna-splitter over to the PolaRx4TR can have a loss
of contact.
Having spare cables and adapters have helped, so we had to be a
bit innovative to extend the cable, but I weather secured the
extension setup, so we should be fine.
We maintained the cesium operational from Grenoble. I will
investigate when we lost state, but it was before Grenoble,
so we
should have good data from that, even if Cesium date and time is
off, we do not measure that so I can adjust that later today.
I now go into a clean-up, so start additional loggings I did not
managed to start manually because of stress. I aim to make those
autostart, it's not hard, but one had to prioritize hard.
Power to
the cesium is key, GNSS recording operational second, then
logging
of cesium, environment, PV & batteries further down the
line. The
key to mission have been achieved. I can clean up the other
stuff
while here. I want that additional environmental and system
logging to work for the rest of the mission.
So, we have not failed the mission yet. It's not been
perfect, but
with some clean-up action and experience built, we can improve
robustness.
The Observatoir de Saintt Veran is originally built for
Observatoir de Paris, and they used to have two domes in Paris,
but the light poultion makes it fairly useless. So, they built
this observatory and moved one of their domes up here. Yesterday
late, they where walking to that dome saying "Want to see
Saturn?"
and I just tagged along. It was not the best of days, since
it was
very windy and that makes the atmosphere more disturbed. But the
night sky here is spectacular.
Another nice thing here is that up here there is a small animal
called Marmote. I've never seen one, even in zoo, and on the
drive
up we could see plenty of them. These are very shy animals, so
they run away as we came driving up. However, occasionally they
come up to the observatory. Just the other day, one Mermote
walked
over the observatory keepers foot, not run, but walked. So, we
joked that the Marmotes do not fear the scientists. :)
We where able to come up with the clocks a day early, so we
got to
interact with the previous mission. Friendly and hospital
people.
So they shared the reminder of their dinner food, cheese and
wine
with us. The hobbyist astronomers can apply for one-week
missions
to Astro Queyras that. Depending on the quality of the mission,
they get access to the site for a week, and good mission
gets the
good weeks, with minimal moon light polution. Normally it is
full
isolation, but gaining another day of integrating up the
frequency
error they bent the rules for us, and me and Bruno could come
early. We are very happy as we also got to interact with the
preivous mission here.
If there is interest, I can keep sending updates. Yesterday was
hectic, so it was not the time to pull out data. That work
we can
start doing today. I will also do a write-up of the setup if
people are interested.
So, I was given the oppertunity to do this with very short
notice,
and I jumped on the oppertunity. It's been very stressful
and hard
work to come to this point, but now I can start to enjoy it.
Also,
time to share the experience.
Today will the rest of the total mission come up, I only work on
the graviational red-shift mission, but the full mission
involves
astronomy missions naturally, such as spectroskopy. There is
also
a documentation side, so things is filmed etc.
Cheers,
Magnus
On 2023-07-16 06:48, Christopher Hoover wrote:
Best of luck, Magnus.
On Fri, Jul 14, 2023 at 4:04 PM Magnus Danielson via time-nuts
<time-nuts@lists.febo.com> wrote:
Fellow time-nuts,
So, 5 years ago I was asked by Observatoir Royal de
Belgique
to take
care of two failed 5071A cesiums and alter them so that
there
is one
functioning. I did that, and have operated the functioning
one here and
there, but conservatively, since they wanted to do a
mission
with
graviational red-shift.
Since them, other things happen in life, such as Covid,
prohibiting
work. However, a few weeks ago I was contacted as they
where
about to go
on the mission. I thought it would be nice to join, as
I had
quite a bit
of the things needed for such mission.
Setting it up meeds a number of challenges, as it needs
to be
powered
continuously, and over a varity of sources.
In addition I wanted to log as much data as possible. I've
come far on
that point. I log 5071A state and several other environment
sensors into
an InfluxDB and then illustrate with Grafana. Independently
two GNSS
receivers log things for post-processing.
Just assembling the rig for the two 100 W PV panels and a
choke-ring
antenna has been a challenge. Let's just say that I should
not be hired
to do fine mechanics work.
Things have been fixed during the travel, and I have driven
600+1100+800
km just to be in base-camp before final climb. I'm now
sitting in a
borrowed house just outside of Grenoble. I just had to tear
out part of
the equipment, as one of the power converters failed.
Luckily
that side
was redundant for the mission, but I do miss one pressure
sensor to
compare the small one with.
I had intended for my passive hydrogen maser to also do the
trip, but it
woke up with an unexpected problem and did not lock, and I
decided it
was not meaningful to bring it's dead weight along. I will
have to
investigate the actual fault. Also, I have been unable
to log
the
internal state, which Would have given valuable clues.
Tomorrow we do the final climb to Observatoir de St
Veran at
2930 m and
stay there for 8 days.
Connectivity can be sketchy.
In the end of the day, it is only if you try that you can
either fail or
succeed. So far, it has not yet failed completely
catastrofically, but
warnings-signs has gone of and they been managed, so far.
This can still
fail spectacular.
Now I badly need my bed. Being sleep deprived does not help
thinking.
Cheers,
Magnus
time-nuts mailing list -- time-nuts@lists.febo.com
To unsubscribe send an email to
time-nuts-leave@lists.febo.com
_______________________________________________
time-nuts mailing list -- time-nuts@lists.febo.com
To unsubscribe send an email to time-nuts-leave@lists.febo.com
--
Jim WN1X
time-nuts mailing list -- time-nuts@lists.febo.com
To unsubscribe send an email to time-nuts-leave@lists.febo.com
Magnus,
I'm glad you got to experience the challenge of powering cesium clocks
on a road trip! Fun, yes? Here's what I used:
AC system -- I tapped off the car's battery with dual 100 A fuses and
thick wire to feed in parallel two 80 Ah batteries below the passenger
seat. That gave redundant, low impedance, 12 VDC for two sinewave
inverters, one per battery, to supply 120 VAC to two power strips, the
A-bus(AC) and B-bus(AC). All AC feeds used standard IEC cords, no custom
wiring was necessary.
DC system -- I removed both rear seats and bolted into place a frame for
eight 120 Ah VRLA batteries arranged in series and diode-or parallel to
create two redundant independent sources of 24 VDC, the A-bus(DC) and
B-bus(DC). All DC feeds were done with Anderson PowerPole [1] connectors
and accessories.
5071A system -- Each clock had internal 24V battery packs re-made from
fresh EnerSys Cyclon 2.5 Ah cells. Each clock was powered by 120 VAC
from an AC bus, and also powered by 24 VDC from a DC bus. Note that the
internal backup battery inside a 5071A is slowly charged only when AC is
present.
Attached are two screen shots, from page 56 and 57 from [2], that show
some of the setup.
I carried long AC extension cords so that when "shore power" was
available at any point during the experiment I could power any AC bus
from that source and/or recharge the main 8x VRLA batteries using marine
battery chargers that I brought along.
As you found out, the logistics of failure-proof clock power is quite a
bit more complicated than any of the 10 MHz and 1PPS timing stuff!
I'm impressed with your use of solar panels. I wish I could do that, but
solar doesn't often work well in the Pacific Northwest, and my power
requirements are higher due to multiple clocks and related electronics.
But my car is a Toyota Highlander Hybrid which acts as an efficient
generator if non-battery power is required. It's not quite as "green" as
solar, but it works in all weather.
/tvb
[1] http://leapsecond.com/pages/powerpole/diode-or.htm
[2]
http://leapsecond.com/ptti2020/2020-PTTI-tvb-Atomic-Timekeeping-Hobby.pdf
Hi,
On 2023-07-24 15:19, Bob Camp wrote:
Hi
Wow, that’s quite a story. Thanks for sharing !!!!
While this is a bit off topic in terms of timing:
Victron does sell “central control” computers to tie their gear together. Some are in the
< $200 range. They will let you manage an ensemble of devices effectively. It also is
worthwhile to tie in something like a BMV-712 shunt based monitor. That will let you
evaluate the battery(s) state accurately.
Interesting. I did not know and had not enough time to figure that one
out. Thanks! That may be the way forward.
I will check that one up.
The MPPT was acting as my load measurement unit.
There are folks who make “hats” for the RPi that do backup power as well as DC/DC
conversion. They claim they are pretty robust. No idea if they would be robust enough
for this sort of thing.
I've seen that, but did not have the time to figure out more about it
and get one. It would for sure been helpful for this mission.
Lithium batteries, while still expensive, are getting less so every day. They are way lighter
than lead acid if you are going to lug them here and there. The Victron gear is just as
happy running them as running various forms of lead acid. They also will happily sell
you setups for 24V if you decide that works better than 12V.
Well, considering I got the lead acid batteries for free and new last
fall, it was available for this exercise.
As you might guess, there’s a lot of Victron gear in my RV’s …..
That would make sense.
Cheers,
Magnus
Bob
On Jul 24, 2023, at 4:06 AM, Magnus Danielson via time-nuts time-nuts@lists.febo.com wrote:
Hi all,
So, now this morning we rose early, I was late at 6:30. Quick cleanup of room, breakfast, take antenna down, reconnect small pin-wheel antenna for climb down. That part went well.
Driving down the steep road is harder than the climb. During winther only skis work, and it takes 4 hours. It can be 2-3 m snow. Snow is good, melted snow becomes water for the station, which is used during summer. We saw marmottes and mountain deers. Unnecessarily cute. Fashinating talking to the care-talker, we really hit off well and love his stories. He lives locally and I now kniw his family.
Clock transfer from Observatoire car to my car went eventless... not. We lost power again, now for a new reason. I keep learning things, and the failures is important.
So, I had changed the loading power from 4A to 12A at masimum on the AC charges while at station. Well, there it is safe. In the car, not so much, so the 12 VDC to 230 VAC (125 W) ducked and that caused temporarily outage. I fairly quickly retained power and set charger to reasonable values and things where fine. However, it required me to restart the Raspberry Pi, since I had not had the time and energy to make all the logging work. OK, so hook up screen to RPi for it to boot properly. Massive problems, and it turns out that the screen is dying. I managed to get the environmental logging running, cesium logging, pressure-sensor, and power setup logging. Then pack things up and start the decend from Saint Veran village.
The first stretch, I let Bruno drive, as he has much more experience in mountain driving than me.
Part of the travel down, there was a peep. Turns out that everything was fine, except that the 12 VDC to 230 VAC converter ducked under the heat. So I turned it of and it works. From here the PV panels was far than enough, so no worries. This happened a few times more during the travel, but without major effect on resull.
We stopped for lunch, drove further, and from Grenoble I took over driving. Filling the tank, getting some soft drinks and such, a drive of 800 km to just outside Mons in Belgium, where we where able to have late dinner and charge the batteries. Then drive to the Observatoir Royal de Belgique in Brussles.
Once at place at the Observatory, we had to plan how to get the clock into the laboratory, again while avoiding to power the clock down. Through inventive use of extention cables and using a pair of batteries as power backup and manage the switch of power out-let, we where able to take it out of the car, roll it down the ramp and then through the corridors into the time-lab and eventually slid it in just under the Cesium 1 of the ORB cesium rack.
The safe transfer over to laboratory location and power was successful and at 03:00 we could leave the site.
While the adventure contains several events, the combinations of measures alows us to compensate for iit to some degree. The big and small failures is to learn from, and sharing the gory details of experience is part of the learning experience. Having just 3 weeks on the spare time to build a setup like this is not enough. It remains the main source for the failure.
The general concept of power, GNSS and logging works. Especially power is challenging and being able to alternate between solar panels, AC and car generator is a contributor to the relative success. The lack of automation allowed too much of human error in handling. Much of that ended up being handled without majjor failure of the mission, but it did mean loosing valuable measurement and scientific strength. The loss of measurements on low level is a problem, and we then need to recover that as being measurement after the travel, just to establish the rate of the clock at sea level. Assuming the clock maintains that rate in it's own frame of reference, which we then change as we go up and then go down the mountain, and that we reduce the graviational potential and thus gravitational acceleration. However to show this we need to know the rate of the clock at sea level (ORB lab is at about 100 m) to illustrate the difference in rate. We choose to measure this. Speaking of measuring, we choose to use GNSS common view measurements using the PolaRx4TX as main measurement, and the Mosaic-T/X5 as auxillary. The later would fail to do PNT solutions but continue to measure everything else. Using the 10 MHz as primary signal, and PPS more as auxillary, loss of PPS phase was less of a failure if only the clock state was recovered quickly.
Overall I consider the power-strategy as such as a success. The implementation leaves room for improvement, but considering it was whipped together in three weeks while working full time, the failures and near misses mainly comes from stress and lack of time to integrate things. I was recommended to use the Victron MPPT and AC-chargers, as they have known good quality and also an open and open-source oriented view on things. I had no previous experience of these myself. The MPPT solar panel charger is accessable over both VE Direct electrical interface and over BlueTooth. While I usually is not overly ethusiastic for BlueTooth, this time I was extremely happy to have the ability to monitor over BlueTooh. Using a VE Direct USB adaptor cable, I was able to integrate the monitoring to the InfluxDB and Grafana infrastructure relatively well. That however only happend once up on the Observatory. I should be able to integrate over the BlueTooth, but I did not have the time and energy to do that, but it will be the next thing to attempt, when I reach my summer house.
The thing which really worked well with the power system was the dynamic capabilities of it. The MPPT charger has three power connections, the PV solar panel, the batteries and then the load. Towards the solar panel, it optimize the load by adjusting the current drawn, to find the optimum V/I operational spot to achieve the maxium output power P for each condition. When panels have too little energy, it turns it of. According to Victron recommendations, an AC charger can be added by hooking it in directly on the batteries. What is missing is a better integration between these. While the MPPT was doing great, and the AC charger was doing great while operation each on it's own, the cut-over between them and more importantly handling of the different sources of AC power (house or car) with different charger conditions allowed for human error, as will be illustrated. On both the battery side and the load side I used a RigRunner 4008H power-pole panel. I used power-pole for all 24 V power connections. This allowed me to use unusused cables as backup, it also allowed me to hook up a pair of batteries to directly drive the cesium when needed either as UPS or as DC source when switching AC connector.
Power failures did happen, and here is a list of the cesium failures:
- Lab setup
When testing the DC feed to the cesium, pulling the AC failed.
Cause: Forgot to connect the DC cables all the way.
Root cause: Human error, forgot to check and verify the DC level.
Comment: The cesium does not have an indicator of the power status of the AC and DC. It does the power-switching well, but it would help to know.
- Transfer from lab to car.
When transfering the setup to the car, in itself a dress rehearsal and test of the concept of moving it over, the cesium failed.
Cause: Connecting the DC cable to the distribution channel was done to a connector with a fuse of 1 A and the cesium pulls about 2 A. Blown fuse expected. Fuse and blown-fuse indication worked.
Root cause: Stress contributed to this, this was done just as I was about to drive 600 km to the summer house.
Recommended correction: Having planned and marked the intended connection point on the panel to the cesium would have avoided the issue.
- Short circuit in Grenoble
Driving down from summerhouse to outside Mons was a 1100 km drive. That was successful. Then driving down to Grenoble was in itself also successful. However, lack of sleep was a contributor. When in Grenoble issues had been detected and a failure of the setup experienced. Part of the setup removed to secure the experiment.
Cause: A cable had come loose, so a +24V cable made connections it should not have done. Fuses blown etc.
Root cause: The haste by which everything was assembled prooved that everything was not connected well enough, strapped down enough for the isolation to work. Re-connecting it to the small DC/DC adaptor (for +5V), with more ZIP-ties to secure cables and then isolation and location worked.
Recommended correction: Have more time to spend on securing cables and better isolation would have resolved this.
- Drop-out in Grenoble village
Having charged the batteries at the station, and then transfer the setup to the car to have it under charge during the night was done to ensure successful transfer. During transfer between the cars, operating on my car's 12V using the 12V to 230 VAC adaptor, the clock can be transfered without batteries. However, this is done by driving the AC adaptor using the generated AC. This has been a good strategy for all transfers. This drives the 24V bus and everything else depends on it. However, this time it failed.
Cause: The AC charger had been put in 12A charging mode and high voltage level while as the observatory, since on observatory power (in itself a large PV array and battery bank) the load on the 125W DC to AC convertor made it duck, and with that, the 24V system when batteries was disconnected. Reverting to battery shortend the outage. Reducing setting on AC avoided the next power outage.
Root cause: Human error and integration error. Having the AC charger act both as supply for AC and car DC source (though DC/AC converter) made integration harder. There was a good separate car DC outlet, but no suitable 12V to 24V switch supply available, that would have avoided the integration problem with 12V load and 24V load. Integration could have avoided more of the human error and steered things better.
Recommended fixes:
4.1 Check-lists could have reduced the risk.
4.2 Better setup would have reduced the risk.
4.3 Better integration would have reduced the risk.
4.4 Using the AC supplies ability to operate as power supply rather than charger would have changed the outcome.
4.5 Keeping charging/supply also from observatory car would have reduced the problem.
4.6 More time to integrate and test
While this is not an exhaustive set of problems, it illustrates well how lack of time and stress enabled human error and deficiencies in the setup to cause failures. However, considering the many transfer of power, clock transfers, dynamic handling of setups, carrying etc. the general concept itself seems to work really well. It just needs to work out it's teething problems.
I chose to be brutally honest, because then there is more to write and more to learn. I chose to focus on the power in this post, but logging had it's set of challenges. Also, cause of problem was that the screen used with the raspberry pi is about to die, which was realized only on decent in Saint Veran village.
I intend to report more as I have arrived at my summerhouse. I think the logging would be good to explain and do a post mortem on.
It's only by making an honest attempt that one can test how teories work in practice. It is only by honestly report and do full post-mortem that the failures can be converted into useful experience and enable en even smoother operating setup on next run.
Some of what I've build, is really what should go into my basement time-lab too. The difference is really not that large.
I will close the laptop, brush my teeth and pack up to leave the smallest hotel I've ever stayed at.
Cheers,
Magnus
On 2023-07-23 00:57, James Littlefield wrote:
Enjoy your feast. It has been fascinating reading your chronicles of the experiment.
On Sat, Jul 22, 2023 at 6:54 PM Magnus Danielson via time-nuts time-nuts@lists.febo.com wrote:
Hi all,
Our time here starts to run out. This is bitter-sweet, because it
means
leaving a really nice place with a boatload of new friends, good
times
and for me a whole amount of astronomy I did not expect to see.
After doing the last show for the turists, we have moved the clock
and
batteries over to the observatory car where it will sit over night.
Power from the house and maintain the choke ring on the top. All to
minimize the things we need to do tomorrow morning, since we need to
raise early and then travel down to the village, and then transfer
the
clock back to my car, and then drive across France and into
Brussels, to
move the clock into the Observatoir Royal de Belgique lab.
It's been very eventless here, but things where fixed in the
beginning.
So, you would expect things to be fine... well, just as we loaded the
clock and locked the car, we saw that one tire is flat, so right now
work to change the tire. Luckily we have a spare tire. So, the usual
mess ongoing. Had to unload some, but not too much.
I have tried to use a TICC, logged a lot of data in parallel to
the GNSS
common view, and it turns out that the least square fit is for
ridicolous litte frequency shift with about 3E-16, so that is
probably
just the same clock looped back. Ah well. The integration works. What
does not work is signal on ChB so I need to investigate that as I
come home.
We have learned immensly by attempting this project, and just
finishing
it we try to use the knowledge.
I'll setup a webpage on my server to describe as much as I can.
The hobby-astronomers have various other background, such as one
being a
math professor, so I showed him my fast least square work, just to
see
what he thought. Kind words was exchanged. Other than that, lots of
physics, math etc. have been discussed.
Car tire changed, things reloaded. Everything is fine.
Now we prepare for the final night feast.
Cheers,
Magnus
On 2023-07-19 03:06, Christopher Hoover wrote:
Huzzah! Yes, please send updates.
On Sun, Jul 16, 2023 at 1:41 AM Magnus Danielson
<magnus@rubidium.se>
wrote:
Dear Christoffer,
Many thanks. Yesterday we maanged to drive from Grenoble up into
St Veran village, the highest village in Europe. This had me
driving mountain roads like nothing I've ever done before, so it
was a very intense experience. I even drove through Italy,
so the
second new country for me to drive in.
From St Veran village my car was not suitable, as the final
climb
is on a road which is not very good, so we had to use the
Observatory jeep and transfer the clock over to the shock
absorber. This is quite an elaborate dance, but thanks to the
modularity of the setup where where able to safely transfer
it up.
We used a small pin-wheel antenna to record the remaining rise.
At the observatory, I made sure that batteries was recharged
before we transitioned furteher. It was quite an elaborate dance
to move it up into the lab. We also installed a choke ring
antenna
on the roof of the observatory.
I use my Mosaic-T and ORB brought their PolaRx4TR. This provide
redundancy, which is good, since we discovered that the
cable from
the antenna-splitter over to the PolaRx4TR can have a loss
of contact.
Having spare cables and adapters have helped, so we had to be a
bit innovative to extend the cable, but I weather secured the
extension setup, so we should be fine.
We maintained the cesium operational from Grenoble. I will
investigate when we lost state, but it was before Grenoble,
so we
should have good data from that, even if Cesium date and time is
off, we do not measure that so I can adjust that later today.
I now go into a clean-up, so start additional loggings I did not
managed to start manually because of stress. I aim to make those
autostart, it's not hard, but one had to prioritize hard.
Power to
the cesium is key, GNSS recording operational second, then
logging
of cesium, environment, PV & batteries further down the
line. The
key to mission have been achieved. I can clean up the other
stuff
while here. I want that additional environmental and system
logging to work for the rest of the mission.
So, we have not failed the mission yet. It's not been
perfect, but
with some clean-up action and experience built, we can improve
robustness.
The Observatoir de Saintt Veran is originally built for
Observatoir de Paris, and they used to have two domes in Paris,
but the light poultion makes it fairly useless. So, they built
this observatory and moved one of their domes up here. Yesterday
late, they where walking to that dome saying "Want to see
Saturn?"
and I just tagged along. It was not the best of days, since
it was
very windy and that makes the atmosphere more disturbed. But the
night sky here is spectacular.
Another nice thing here is that up here there is a small animal
called Marmote. I've never seen one, even in zoo, and on the
drive
up we could see plenty of them. These are very shy animals, so
they run away as we came driving up. However, occasionally they
come up to the observatory. Just the other day, one Mermote
walked
over the observatory keepers foot, not run, but walked. So, we
joked that the Marmotes do not fear the scientists. :)
We where able to come up with the clocks a day early, so we
got to
interact with the previous mission. Friendly and hospital
people.
So they shared the reminder of their dinner food, cheese and
wine
with us. The hobbyist astronomers can apply for one-week
missions
to Astro Queyras that. Depending on the quality of the mission,
they get access to the site for a week, and good mission
gets the
good weeks, with minimal moon light polution. Normally it is
full
isolation, but gaining another day of integrating up the
frequency
error they bent the rules for us, and me and Bruno could come
early. We are very happy as we also got to interact with the
preivous mission here.
If there is interest, I can keep sending updates. Yesterday was
hectic, so it was not the time to pull out data. That work
we can
start doing today. I will also do a write-up of the setup if
people are interested.
So, I was given the oppertunity to do this with very short
notice,
and I jumped on the oppertunity. It's been very stressful
and hard
work to come to this point, but now I can start to enjoy it.
Also,
time to share the experience.
Today will the rest of the total mission come up, I only work on
the graviational red-shift mission, but the full mission
involves
astronomy missions naturally, such as spectroskopy. There is
also
a documentation side, so things is filmed etc.
Cheers,
Magnus
On 2023-07-16 06:48, Christopher Hoover wrote:
Best of luck, Magnus.
On Fri, Jul 14, 2023 at 4:04 PM Magnus Danielson via time-nuts
<time-nuts@lists.febo.com> wrote:
Fellow time-nuts,
So, 5 years ago I was asked by Observatoir Royal de
Belgique
to take
care of two failed 5071A cesiums and alter them so that
there
is one
functioning. I did that, and have operated the functioning
one here and
there, but conservatively, since they wanted to do a
mission
with
graviational red-shift.
Since them, other things happen in life, such as Covid,
prohibiting
work. However, a few weeks ago I was contacted as they
where
about to go
on the mission. I thought it would be nice to join, as
I had
quite a bit
of the things needed for such mission.
Setting it up meeds a number of challenges, as it needs
to be
powered
continuously, and over a varity of sources.
In addition I wanted to log as much data as possible. I've
come far on
that point. I log 5071A state and several other environment
sensors into
an InfluxDB and then illustrate with Grafana. Independently
two GNSS
receivers log things for post-processing.
Just assembling the rig for the two 100 W PV panels and a
choke-ring
antenna has been a challenge. Let's just say that I should
not be hired
to do fine mechanics work.
Things have been fixed during the travel, and I have driven
600+1100+800
km just to be in base-camp before final climb. I'm now
sitting in a
borrowed house just outside of Grenoble. I just had to tear
out part of
the equipment, as one of the power converters failed.
Luckily
that side
was redundant for the mission, but I do miss one pressure
sensor to
compare the small one with.
I had intended for my passive hydrogen maser to also do the
trip, but it
woke up with an unexpected problem and did not lock, and I
decided it
was not meaningful to bring it's dead weight along. I will
have to
investigate the actual fault. Also, I have been unable
to log
the
internal state, which Would have given valuable clues.
Tomorrow we do the final climb to Observatoir de St
Veran at
2930 m and
stay there for 8 days.
Connectivity can be sketchy.
In the end of the day, it is only if you try that you can
either fail or
succeed. So far, it has not yet failed completely
catastrofically, but
warnings-signs has gone of and they been managed, so far.
This can still
fail spectacular.
Now I badly need my bed. Being sleep deprived does not help
thinking.
Cheers,
Magnus
time-nuts mailing list -- time-nuts@lists.febo.com
To unsubscribe send an email to
time-nuts-leave@lists.febo.com
_______________________________________________
time-nuts mailing list -- time-nuts@lists.febo.com
To unsubscribe send an email to time-nuts-leave@lists.febo.com
--
Jim WN1X
time-nuts mailing list -- time-nuts@lists.febo.com
To unsubscribe send an email to time-nuts-leave@lists.febo.com
Tom,
On 2023-07-24 19:47, Tom Van Baak via time-nuts wrote:
Magnus,
I'm glad you got to experience the challenge of powering cesium clocks
on a road trip! Fun, yes?
Yes, an interesting challenge and the main focus.
Here's what I used:
AC system -- I tapped off the car's battery with dual 100 A fuses and
thick wire to feed in parallel two 80 Ah batteries below the passenger
seat. That gave redundant, low impedance, 12 VDC for two sinewave
inverters, one per battery, to supply 120 VAC to two power strips, the
A-bus(AC) and B-bus(AC). All AC feeds used standard IEC cords, no
custom wiring was necessary.
I had two AC busses two, one generated of the cars power and one
generated of the battery 24V bus. It should have been on the load bus,
but I did not have time to do that as I bought that one at a gas station
in France and inserted into the system as parked there. Having a spare
banana-jack to PowerPole cable came to rescure. However, there was no
banana holes in the terminals, even if they looked just like that, but
putting the bananacontacts on the side and screw down was the field
engineering that turned out robust enough for the rest of the mission.
No special cables was done on 230 VAC.
DC system -- I removed both rear seats and bolted into place a frame
for eight 120 Ah VRLA batteries arranged in series and diode-or
parallel to create two redundant independent sources of 24 VDC, the
A-bus(DC) and B-bus(DC). All DC feeds were done with Anderson
PowerPole [1] connectors and accessories.
My DC system was a Anderson PowerPole connector heaven, this includes
Solar panels. The upside being that an cable can act as extender at any
point in the DC system. Quick to connect and disconnect in the field.
I could use batteries directly on the clock to handle swich of AC output
while moving for instance.
5071A system -- Each clock had internal 24V battery packs re-made from
fresh EnerSys Cyclon 2.5 Ah cells. Each clock was powered by 120 VAC
from an AC bus, and also powered by 24 VDC from a DC bus. Note that
the internal backup battery inside a 5071A is slowly charged only when
AC is present.
I had no time to replace the internal battery. I would have loved to
have that additional safety. One of the most critical issues was lack of
time, as I got involved with only 3 weeks to do it, and working daytime
job most of that time.
Having the internal battery fixed would have saved several "oups". That
is part of the learning, which I could easily forsee and knew, but time
was not allowing for finding a fix. I tried to replace the PHM batteries
for exactly this reason, but did not find a match locally. So root cause
ends up lacking time to prepare.
Attached are two screen shots, from page 56 and 57 from [2], that show
some of the setup.
I carried long AC extension cords so that when "shore power" was
available at any point during the experiment I could power any AC bus
from that source and/or recharge the main 8x VRLA batteries using
marine battery chargers that I brought along.
I had two long cables two, and that helped both with "land power"
overnight as well as when transfering clock between cars and transfering
clock to and from laboratory (mine, Saint Veran and Royaal Observatory
of Belgium).
As you found out, the logistics of failure-proof clock power is quite
a bit more complicated than any of the 10 MHz and 1PPS timing stuff!
I knew that was the challenge, from start, so that is what I focused on.
I honestly did not care too much about all the measuring stuff, but
tried to add as much as I could while building power.
A particular care was to build the system so I avoided low charge on the
batteries.
I'm impressed with your use of solar panels. I wish I could do that,
but solar doesn't often work well in the Pacific Northwest, and my
power requirements are higher due to multiple clocks and related
electronics. But my car is a Toyota Highlander Hybrid which acts as an
efficient generator if non-battery power is required. It's not quite
as "green" as solar, but it works in all weather.
The solar panels ended up overproducing power, and thus loading
batteries for most of the day-time. I had a 55 W load as measured,
somewhat more I guess because of bush-rewiring. The 200 W rate is a bit
theoretical as it is several "It depends". Since they where not flat,
you loose some. For Stockholm, a graph showed I would only have 82% from
angle error. So, 160 W. Many times I was reading 80-140 W. I peaked at
189 W as I recall. Some clouds reduced power, but during the drives it
was mostly well. It was only on the way home yesterday that I had heavy
rains, but then the cesium was off the system and I was down to 14 W load.
While I could have deployed multiple SR620 TI-counters, I chose to use
two TADR TICCs instead. This has huge impact on the power budget. First,
the SR620 runs hot and has a fan because it is needed. The ECL age shows
it's choice of balance. The SR620 would have to be run on 230 VAC, so
that would mean additional loss in the 24V/230V system. Double
conversion should be avoided, but I did not have time to do that, but I
optimized so that the major load, the cesium, was on DC power as primary
source while running batteries, this removes it's losses in the AC to DC
conversion. I used the cesiums AC as auxillary source for it for
redundancy. That did not play out exactly as planned, as I think it used
the AC path at times, but I ended not being so low on power that it
cared in the end. Anyway, an SR620 would need either a GPIB or RS232
dongle in addition. While that does not add much power, compared to use
a TADR TICC. I had my DPM7885 pressure sensor and two TADR TICC and it's
24V/12V DC supply consuming just about 6 W of power in total. Using the
TADR TICC in time-stamping mode, I have 2-4 channels depending on how
you want to measure. The USB of the TADR TICC shows up as /dev/ttyACM0
and /dev/ttyACM1 in the Raspberry Pi, and thanks to a few lines of
python was feeding into the InfluxDB. I used a USB hub with external
power to feed those devices, which includes RS232 adaptors. I aim to
describe the measurement stuff in a separate thread.
In general, I tried to avoid unnecessary power conversions. For
instance, providing power and charging the batteries from the car I had
intended to use a 12V to 24V DC/DC converter. I had a separare cable for
an outlet in the boot that was on a 15 A fuse. That cable was very easy
to access (open a little panel (no tools required) that I had not
noticed before, and the cable just fell out. Connecting to the cable
seemed straight-forward, so a small adaptor over to Anderson PowerPole
could have been built quickly. However, I ended up not having the time
to buy a suitable DC/DC. Instead, I had to use the 12VDC to 230 VAC 125
W converter that I bought lighly used at a ham-meet, naturally from a
friend of my radio-club. I also got the 24V to 12V converter at the same
time, since I expected that to come in handy for the home lab. This
DC/AC was critical at many times, but provided an issue since you had
the double conversion issue, and also since the second converter was not
aware of the limitations of the first, as I have already described. This
shows that the double conversion strategy feeding into the system should
be avoided. Double conversion is however not always wrong. When feeding
from house AC, the conversion into the 24V busses and then have one
conversion from the 24V bus to whatever is good strategy since you avoid
breaking power. It is the double conversion from the 24V bus that needs
to be avoided, since that is unnecessarily draining the batteries when
depending on those.
So, the double conversions from battery done where:
PolaRx4TR - forgot the DC cable and we had not had time to adapt power,
it went on the produced 230 VAC bus with it's AC-adaptor.
Raspberry Pi - it's DC/DC converter seemed to be overloaded constantly,
so I pulled out a AC/DC converter and used that and removed the DC/DC
converter.
5071A - at times.
The Raspberry Pi had power issues in the beginning, and it did not start
clean every time. I had to do quite a bit of clean-up on that, but I was
not able to finalize. I decided to rip out the DC/DC (cut away!) and use
the AC/DC. Not ideal, but not the largest power loss. The RPi went
stable power-wise from that. I did have other issues.
While not everyone will need to have a portable and moveable lab, some
of the lessons is relevant even for a static lab. In fact, this forced
me to do many of the things going into my static lab.
While many things could have been done better before and during this
mission, I think sharing the system thoughs and experience in the field
(context, I've now driven 4900 km with this setup, or 15 degrees south
and 11 degrees north (at summerhouse, has another 4 degrees north to
drive) will be interesting and food for thought, even for those building
a smaller system for static operation in their labs.
I remain unsatisfied with preparations, but many of the necessary things
worked and provided needed dynamic in the field to handle the
encountered challenges, so I am quite satisfied that many of the system
thoughs and moves worked as intended. The failures and near failures
stands as learning experiences for the future, and I think the best way
is to share the experience of both good and bad, to learn how to
progress. I am already thinking about the many improvements I would like
to do for a future next event. More clocks, better power management,
better physical and electrical build etc. The use of the PV panels
turned out very well. The mount for panels and choke ring antenna turned
out to work very well, even if I consider the build being a hack, and
not even my best mechanical build.
Cheers,
Magnus
/tvb
[1] http://leapsecond.com/pages/powerpole/diode-or.htm
[2]
http://leapsecond.com/ptti2020/2020-PTTI-tvb-Atomic-Timekeeping-Hobby.pdf
time-nuts mailing list -- time-nuts@lists.febo.com
To unsubscribe send an email to time-nuts-leave@lists.febo.com