×
INDI Library v1.8.1 Released (09 Sep 2019)

Monthly maintenance release INDI v1.8.1

Full Automation and Weather handling in the Scheduler

7 months 3 weeks ago
knro
Administrator
Administrator
Posts: 7421
Karma: 51
Full Automation and Weather handling in the Scheduler #38039
All great questions Hans!

1. DBus: Only the scheduler uses DBus to communicate with the Ekos. At face value, it doesn't make sense to use DBus within the same process, but the scheduler was planned from the beginning to be its own separate niche. Using the same logic in the scheduler, anyone can develop an independent scheduler using the same dbus calls.
2. There was _some_ preliminary support for weather ALERT causing a shutdown of the observatory, but as with most things, it's a lot more complicated than that. Specific actions need to be taken when such a condition exists. What about WARN? maybe the observatory can temporarily close the roof..etc and then reopen again? All this logic is currently not handled at all. So it was a choice between implemented a borked weather handling implementation or none at all, so what you see in the code is the relic of the initial attempt at this.

I think to proceed, you probably need the following:

1. A dialog to set what actions are required when a specific weather condition state changes

Weather_Param
State
Action

Weather Params are already defined and standard. States are OK, BUSY (WARN), and ALERT

ACTION could be like Shutdown, Resume, Email, Ignore, Script (Run some script), Suspend (i.e. stop capture)

Then you can have a checkbox [ X ] Auto Resume, that is , if the weather condition improves, should the observatory resume operations or remain at shutdown?

So these are my initial thoughts on this.

Jasem Mutlaq
Support INDI & Ekos; Get StellarMate Astrophotography Gadget.
How to Submit Logs when you have problems?
Add your observatory info
The following user(s) said Thank You El Corazon, pug916

Please Log in or Create an account to join the conversation.

7 months 1 week ago
fenriques
Expert Boarder
Expert Boarder
Posts: 139
More
Full Automation and Weather handling in the Scheduler #38706
Hans,
I'm using my observatory fully automated from remote as well and thus relying completely on the scheduler; having weather condition controlling the shutdown of the observatory would be a great improvement (and relief). Thank you for taking care of it.
I could provide some help defining and testing use cases if needed.

Ferrante
The following user(s) said Thank You pug916

Please Log in or Create an account to join the conversation.

7 months 1 week ago
sterne-jaeger
Platinum Boarder
Platinum Boarder
Posts: 408
Karma: 5
More
Full Automation and Weather handling in the Scheduler #38719
Hi Hans,
that sounds like an excellent idea! Just one thing: please try to implement weather handling standalone and do not try to squeeze it into the scheduler. We plan to rework the scheduler currently, since it is a huge and awfully complex class.

It should be easy to change the scheduler in such a way that it can be started/stopped/aborted/paused/... from outside. That could be a way to let the two modules easily cooperate without deeply intertwining them.

And please try to consider automated tests from the very beginning.

If you need help, do not hesitate...

- Wolfgang

TSA-120 + FSQ-85 + GSO 150/750 | Avalon Linear + M-zero | Moravian G2-8300 + ASI 1600mm pro + ASI 294mc pro | KStars/INDI on Raspberry Pi 4 with Raspbian 10
The following user(s) said Thank You knro, El Corazon, pug916, Ron Kramer

Please Log in or Create an account to join the conversation.

7 months 1 week ago 7 months 1 week ago by Ron Kramer.
Ron Kramer
Platinum Boarder
Platinum Boarder
Posts: 326
More
Full Automation and Weather handling in the Scheduler #38751
I haven't used the scheduler much yet - but this is a necessity for me as well. I THOUGHT the weather watcher was already capable of doing this? IT won't allow me to open if cloudy, but you're saying it won't shut down and park/close?

I have a script that I run from windows. Is there something similar that I can run from linux?
The cloud watcher flags unsafe/clouds or whatever to its data file.
Is there a way to read that file during sub acquisition and abort or wait until (between exposures) to check that file and run a shut down/close up script?

I use to leave SGP running and go to bed - no worries. It would always be closed in the morning because even daylight is unsafe. So it would close at dawn.

So the dat file is written every minute with this format - Really it could look only at SAFE or UNSAFE and close up if unsafe. The rest could be ignored
since unsafe is flagged in the event of any of the other values aren't in the GREEN zone.

"2019-05-03","18:07:21","Cloudy","Dry","Very light","1.2","15.9","4014","1","15.9","10%","21.0","","Closed","5","0","Unsafe","Calm","3.0","Humid","77%"

Maybe you're not referring to Weather Watcher driver? What about watch dog? Can it watch the dat file from the cloud watcher and close up?

This was my windows .vbs script file.
dim dome
set dome = CreateObject( "POTH.dome" )
call dome.CloseShutter()
set dome = nothing

The cloudwatcher software would call the script. But I can't run the cloudwatcher software under linux.
I could run the cloudwatcher software on the desktop running Kstars in the house, but it would have to send a command to the nexdome driver to close.

AP Mach1 / CP4 APCC & PEMpro.
EXP SCI - ED152cf APO - Celestron 11" RASA - Stellarvue 80mm
Baader F2 HS NB filters, Lodestar X2 guide camera / OAG - ZWO 290mm mini
ZWO ASI1600MM Pro / ASI174M (solar) / ASI094MC
NEXDome, CLoudwatcher, AVX mount/ASIair and Stellarmate

Please Log in or Create an account to join the conversation.

7 months 1 week ago
H__
Senior Boarder
Senior Boarder
Posts: 47
More
Topic Author
Full Automation and Weather handling in the Scheduler #38752
Hi Ferrante,

fenriques wrote: I could provide some help defining and testing use cases if needed.


To get this started without having to change anything in EKOS I've written ekos_sentinel.py. It basically adds what is missing today :
weather safe ?
yes:
    scheduler running ?
    yes:
        ok
    no:
        start scheduler
no:
    scheduler running ?
    yes:
        stop scheduler
        park mount
        close roof
        warm ccd
    no:
        ok

When EKOS gets bad-weather-shutdown handling itself this external daemon can still prove useful as it can be made into a safeguard for when EKOS crashes or locks up. So I intend to maintain it.
Code lives at https://github.com/d33psky/rolloffroof/tree/master/ekos

Here's my 3th test run log (I went to sleep around 03:00 :) ) :
./ekos_sentinel.py --indi_host localhost
2019-05-01 02:45:51,250 ekos_sentinel INFO     weather is safe, roof is closed, start ekos scheduler
Start scheduler
2019-05-01 02:46:51,306 ekos_sentinel INFO     weather is safe, roof is open
...
2019-05-01 03:32:53,546 ekos_sentinel INFO     weather is safe, roof is open
2019-05-01 03:33:53,565 ekos_sentinel WARNING  weather is unsafe, roof is open, stop ekos scheduler
Stop scheduler
2019-05-01 03:33:53,567 ekos_sentinel WARNING  park mount
2019-05-01 03:33:54,583 ekos_sentinel WARNING  close roof
2019-05-01 03:34:51,630 ekos_sentinel WARNING  warm camera
2019-05-01 03:34:51,651 ekos_sentinel WARNING  failed to warm the camera, this is not critical to safety so just continue
2019-05-01 03:35:51,725 ekos_sentinel INFO     weather is unsafe, roof is closed

It did the critical things right; stop the EKOS scheduler, park the mount, park the roof, but it failed to warm the camera. I'll work on that of course ;-)
I also need to add cap support, and allow for a flexible shutdown order as that is really observatory dependent. I also should replace calling indi_getprop and indi_setprop with the Python pyindi-client, which I learned existed only today.

The interfacing with EKOS goes via DBUS and there's things in EKOS that can be improved as well like query capabilities (assuming DBUS even allows for that).

-- Hans
The following user(s) said Thank You Gonzothegreat, fenriques

Please Log in or Create an account to join the conversation.

7 months 1 week ago
fenriques
Expert Boarder
Expert Boarder
Posts: 139
More
Full Automation and Weather handling in the Scheduler #38758
thanks Hans,
Seems perfect. I will give it a try. Btw I see that we share part of the equipment (10Micron, ASI).
A few questions:
- the workflow manages the scheduler to start if the weather is ok. But also resumes a job if after a break the weather returns ok? it seems it doesn't.
- I guess it should be run with a command like:
python3 ./ekos_sentinel.py 192.168.xxx.xxx --options
Can you provide a simple start command line please?
- For my setup I'll add some time delays (sleep), for example my roof hits the mount if it is not yet parked (I have parking sensors, but better be safe). It would be nice to have also a fixed number of retries, e.g. try to warm up the camera for 3 times before issuing a warning.

Ferrante

Please Log in or Create an account to join the conversation.

7 months 1 week ago 7 months 1 week ago by H__. Reason: spacing
H__
Senior Boarder
Senior Boarder
Posts: 47
More
Topic Author
Full Automation and Weather handling in the Scheduler #38776
Hi Wolfgang,

sterne-jaeger wrote: please try to implement weather handling standalone and do not try to squeeze it into the scheduler. We plan to rework the scheduler currently, since it is a huge and awfully complex class.

The scheduler class would certainly benefit from refactoring. Is your rework started somewhere yet ? Branch/posts/chat ?

It should be easy to change the scheduler in such a way that it can be started/stopped/aborted/paused/... from outside. That could be a way to let the two modules easily cooperate without deeply intertwining them.

Agreed. As a quick hack I've already used what's available via DBUS today from a python script. Should such a weather or safety (broader) tab use DBUS to instruct the scheduler or should we use plain C++ class methods ?

And please try to consider automated tests from the very beginning.
If you need help, do not hesitate...

I will definitely need help. I'll first continue reading up on kstars code to see if it speaks to me ;-)


-- Hans

Please Log in or Create an account to join the conversation.

7 months 1 week ago
H__
Senior Boarder
Senior Boarder
Posts: 47
More
Topic Author
Full Automation and Weather handling in the Scheduler #38777
Hi Ron,

Ron Kramer wrote: I haven't used the scheduler much yet - but this is a necessity for me as well. I THOUGHT the weather watcher was already capable of doing this? IT won't allow me to open if cloudy, but you're saying it won't shut down and park/close?

Exactly. And scary.

I have a script that I run from windows. Is there something similar that I can run from linux?

I see several weather related drivers in INDI, and I added one myself too : Weather Safety Proxy https://indilib.org/devices/weather-stations/weather-safety-proxy.html .

The cloud watcher flags unsafe/clouds or whatever to its data file.
Is there a way to read that file during sub acquisition and abort or wait until (between exposures) to check that file and run a shut down/close up script?

For this you can probably best use the Weather Watcher https://indilib.org/devices/weather-stations/weather-watcher.html as you have several weather measurement values.

I use to leave SGP running and go to bed - no worries. It would always be closed in the morning because even daylight is unsafe. So it would close at dawn.

That's my goal as well. And the weather safety service I wrote (which the weather safety proxy polls) also considers daylight to be unsafe.

So the dat file is written every minute with this format - Really it could look only at SAFE or UNSAFE and close up if unsafe. The rest could be ignored
since unsafe is flagged in the event of any of the other values aren't in the GREEN zone.
"2019-05-03","18:07:21","Cloudy","Dry","Very light","1.2","15.9","4014","1","15.9","10%","21.0","","Closed","5","0","Unsafe","Calm","3.0","Humid","77%"

If you want to only use Safe/Unsafe then using the Weather Safety Proxy does make sense, but you'll have to write a script (or rest API endpoint) that translates the Safe/Unsafe from that dat line to JSON format.

What about watch dog? Can it watch the dat file from the cloud watcher and close up?

It can not afaik. The watch dog only monitors if the indiserver has working clients connected to it (think of a networked INDI setup with indiserver close to the gear and some indi client elsewhere connecting to it over a network).

This was my windows .vbs script file.
dim dome
set dome = CreateObject( "POTH.dome" )
call dome.CloseShutter()
set dome = nothing

The cloudwatcher software would call the script. But I can't run the cloudwatcher software under linux.
I could run the cloudwatcher software on the desktop running Kstars in the house, but it would have to send a command to the nexdome driver to close.

You could use the ekos_sentinel.py for the mean time.

-- Hans

Please Log in or Create an account to join the conversation.

7 months 1 week ago
H__
Senior Boarder
Senior Boarder
Posts: 47
More
Topic Author
Full Automation and Weather handling in the Scheduler #38779
Hi Ferrante,

fenriques wrote: thanks Hans,
Seems perfect. I will give it a try. Btw I see that we share part of the equipment (10Micron, ASI).

Oh cool !

A few questions:
- the workflow manages the scheduler to start if the weather is ok.

Yes, that's what the scheduler does today (once). And what ekos_sentinel.py does too (continually).

But also resumes a job if after a break the weather returns ok? it seems it doesn't.

When safe weather returns later during the night ekos_sentinel.py will start the scheduler again.

- I guess it should be run with a command like:
<code>python3 ./ekos_sentinel.py 192.168.xxx.xxx --options</code>
Can you provide a simple start command line please?

I use it like this :
./ekos_sentinel.py --indi_host localhost
But there's several settings hardcoded in the script that all need to be parameterized :
INDI_DOME_PARK_PROPERTY = 'Dome Scripting Gateway.DOME_PARK.PARK'
INDI_DOME_PARK_PROPERTY_PARK_SETTING = 'On'
INDI_WEATHER_PROPERTY = 'Weather Meta.WEATHER_STATUS.STATION_STATUS'
INDI_WEATHER_PROPERTY_OK_SETTING = 'Ok'
WEATHER_META_STATION_INDEXES = {1, 2}
INDI_MOUNT_PARK_PROPERTY = '10micron.TELESCOPE_PARK.PARK'
INDI_MOUNT_PARK_PROPERTY_PARK_SETTING = 'On'
INDI_CAMERA_COOLER_PROPERTY = 'ZWO CCD ASI1600MM-Cool.CCD_COOLER_POWER.CCD_COOLER_VALUE'
INDI_CAMERA_COOLER_PROPERTY_OFF_SETTING = '0'
MAIN_LOOP_SLEEP_SECONDS = 60
INDI_COMMAND_SEND_TIMEOUT = 5
MOUNT_PARK_TIMEOUT = 60
ROOF_CLOSE_TIMEOUT = 60

Python's argumentparser provides for some documentation :
./ekos_sentinel.py --help
usage: ekos_sentinel.py [-h] [--debug] [--once] --indi_host INDI_HOST

Ekos Sentinel

optional arguments:
  -h, --help            show this help message and exit
  --debug               enable debug level verbosity
  --once                run only once, useful for debugging
  --indi_host INDI_HOST
                        INDI server address

- For my setup I'll add some time delays (sleep), for example my roof hits the mount if it is not yet parked (I have parking sensors, but better be safe).

My mount can hit the roof too. And I found a bug; the roof starts closing when the mount is still parking :/ I'm glad the 10micron move park fast and my roof is slow :-D There's already a poll loop in there that is supposed to wait for that event. I'll fix that next.

It would be nice to have also a fixed number of retries, e.g. try to warm up the camera for 3 times before issuing a warning.

Makes sense. I'll add that.

-- Hans

Please Log in or Create an account to join the conversation.

7 months 1 week ago 7 months 1 week ago by fenriques.
fenriques
Expert Boarder
Expert Boarder
Posts: 139
More
Full Automation and Weather handling in the Scheduler #38785
Hi Ron and Hans,
for sure the Scheduler is not managing shutdown triggered by weather conditions as of today, but the Weather and Dome drivers are managing that.
I tested right now with WeatherWatcher and it works. There's just some configuration to take care of:
- In your Dome driver you should enable the AutoPark switch.
- in your Dome driver enter 'Weather Watcher' (or other driver) in Option->Snoop Devices -> Weather. Think that capital letters and spaces matter.
- Restart Ekos. I think the server doesn't need to be restarted.
- As a test, in Weather Watcher Parameters, lower the temp threshold to a lower value.

Note: this works for the roof driver that I wrote myself (Talon6 RoR) and Weather Watcher. But the AutoPark feature is written in the Dome parent class, so should work for every driver.

Ferrante

EDIT: I know this is a partial solution and not a full shutdown/resume procedure, it just parks the dome. For more advanced solutions there's Hans's python script or the next version of the Scheduler.

Please Log in or Create an account to join the conversation.

7 months 1 week ago
sterne-jaeger
Platinum Boarder
Platinum Boarder
Posts: 408
Karma: 5
More
Full Automation and Weather handling in the Scheduler #38790

H__ wrote: The scheduler class would certainly benefit from refactoring. Is your rework started somewhere yet ? Branch/posts/chat ?

Indeed, but it's in a very early stage. See here: indilib.org/forum/development/5137-parti...-scheduler-code.html

-Wolfgang

TSA-120 + FSQ-85 + GSO 150/750 | Avalon Linear + M-zero | Moravian G2-8300 + ASI 1600mm pro + ASI 294mc pro | KStars/INDI on Raspberry Pi 4 with Raspbian 10

Please Log in or Create an account to join the conversation.

7 months 1 week ago
dokeeffe
Platinum Boarder
Platinum Boarder
Posts: 381
More
Full Automation and Weather handling in the Scheduler #38796
Sorry I'm late to this conversation. I'm a bit confused. The scheduler used to monitor the weather while running and run the shutdown procedure if clouds or rain came in.
Is this feature now not in the scheduler?
Does anyone know what version it was removed in? I'll need to go back to that version. Currently I cant use my observatory without this feature.

Thanks,
Derek

Celestron CPC1100 Atik 383L
HP8300 Elite i5 running Xubuntu, Indi, Kstars & Ekos
Roll off roof observatory
Battling Irish weather
github.com/dokeeffe
twitter.com/BallyhouraStars

Please Log in or Create an account to join the conversation.

Time to create page: 0.459 seconds