One of the nice things about the indi architecture is that we can cascade servers. I've been roughing out how we will manage safety etc for our roll-off, and it will ultimately be based on cascading indi servers like this.

Each telescope will have an indi server running, managing cameras and mounts. These will all be the 'out of the box' servers. Then, for the observatory as a whole, another server will be running, which cascades to each of the physical devices, but, all safety operations will be located there, along this line. There will be a 'mount' driver running, but it will be aware of things like roof status etc. As long as roof status is closed, it will not permit any commands to pass thru to the real mount in the cascade. Once roof status is open, it will become essentially a pass thru device, and allow all commands to pass thru to the real mount. Clients in the house will not connect to the real mount, they connect to the managing server.

Same logic will apply for closing the roof. As long as any of the mounts are not reporting parked position, no commands to close roof will be accepted. Once all of the mounts are reporting parked, then the roof will be allowed to close.

We will take this somewhat further, and there will be means of monitoring the data link between observatory and house. If that link is lost, then the managing server will trigger a sequence of events that ultimately ends up with the roof closed. Weather inputs will be able to trigger the same sequence. Essentially all cameras will have in progress frames aborted, mounts parked, etc. Another input will come from the line power monitors. When line power is not available, then everything will be put into the 'shut down and close up' mode, we will have sufficient battery power available to accomplish a full shut down.

My plan is to have all of the remote safety logic buried in the inteverning server that sits between clients and physical device servers. In the overall design, it's the correct place to put this logic.

All this seems a good design for observatory safety, but one needs to consider what happens if the communication between client, servers or devices fails.
How we detect a communication failure? Clients or devices snooping other device can check if a device is responding, but a device can't do the same with the client.
I think this is a must for a good safety practice. Every device must have the capacity of go to a secure state if it detects a communication issue.

Since shutdown procedure may vary from one observatory setting to another, perhaps the following provides a middle ground:

A simple watchdog INDI driver with one property to set the heartbeat interval, and another for a custom executable to run in case of shutdown?

EDIT: I can also provide a sample Python shutdown script so that users do not need to write one for simple cases. It would have values for:

1. host: default localhost
2. port: default 7624
3. devices list: user provide names for mount, dome...etc
4. shutdown procedure:
4.1 If dome device name is not empty, park dome and wait until park is complete.
4.2 if mount device name is not empty, park mount and wait until park is complete.

The user can then add any custom procedures as desired, but for most cases the script described above would suffice.

That looks to be a good idea. With that way anyone could have a correct shutdown procedure as a script for example, for his/her setup.
Philippe

I put together a small python service that calls a shutdown python script when network is lost. This is just a strawman example i put together to test the concept. I was just starting to look at how to write the shutdown part when i read your post so I'm looking forward to see how that turns out.

I've attached what i worked in case someone else finds it useful. It's been years since I've written any code. This is the first time I've used python as well.

If this worked I was planning to convert to c program. These pythons scripts function on my raspberry pi, but there's not enough error handling at this point. YMMV

(unable to upload the zip file i was trying to attach to this post)

The network itself does not have to be necessarily down, it is the communication link to the client that must be down, or even if the network is still up but the client crashed or was shutdown for whatever reason and the server watchdog is still running.

Ok, Watch dog driver & client are ready for testing. Also, Ekos was updated to include support for Watchdog drivers. You can find the watchdog driver under "Auxiliary". For regular cases (park dome and/or mount), you do not need to use an external script, it will be done by the driver. The driver uses a client to issue commands, so by default it connects to localhost at port 7624.

In case there is more required than park dome & mount, then specify an external scripts which will be executed last. The heartbeat threshold sets the timeout for when the shutdown process kicks it. The value is in minutes. Once you change all the settings for your environment, save config file so that it gets loaded on subsequent use.

Any testing or feedback is appreciated.

I've done some testing and wanted pass back my results:

I'm using raspberry pi 2 updated to the latest software and also kstars bleeding. I have the watchdog set to park the mount after 1 minute with no response. So far the mount has not parked once i unplug the network and the timeout is passed (i minute in this case)

I am attaching screen shots of my settings and log files.

I tried it under Raspberry PI 2 and it's working fine. I noticed that you must run indiserver in the background otherwise it gets shutdown once you disconnect I guess because it's still attached to the SSH terminal. So if you login, run it like:

The & will make it run in the backgroud and you can exist normally. This is what I got from the watchdog log:

I've missed that post, off to testing this too.

Seems to be working nicely, ran the telescope/dome sim and the watchdog on the remote server, fired up the VM on the client then killed the WIFI to simulate a connection loss. I can see in the terminal the telescope and dome being shutdown.

Great job Jasem again.
(I need to test this with the gonzo_box now....)

It is platform dependent but for folks using Raspberry Pi you can consider using on-board watchdog. See details here