Hi all !
Here is a proposal to improve indilib imaging perfs, especially for low end hardware or high bandwith situation.
Indi currently transfer blob data (images/streams) by sending base64 encoded data over TCP socket channel.
While being very straightforward and easy to implement, this approach implies lots of data copy.
The idea here is to allow processes using INDI on the same system to share data buffers (blobs) instead of copying data from process to process.
The expected benefits are ZERO copy buffer sharing, meaning higher sustainable
bandwidth and lower latency (up to 7.6x less memory bandwidth would be required for driver/server/client setup)
Technically, a new socket would be made available by indiserver : a UNIX domain datagram oriented.
The payloads of the protocol would stay the same (every dgram is a xml packet), only blobs related event must pass data as attached buffer.
Switching to datagram simplifies lots of thing on server (mainly no buffering required for write/read)
The syscalls: memfd_create and sendmsg with ancillary data are used to exchange data buffers along with messages.
(This is limited to LINUX platform. MACOSX seems to lack support for memfd_create. local socket would be pointless here)
The indiserver would still be compatible with current tcp/xml protocol (listening on tcp), and can convert from shared buffer to inline base64 blob when remote connection is involved.
We could eventually force compression or even downsampling in indiserver, for only remote connections (for wifi connection with limited bandwidth).
On the lib side, we could keep the current blob method for compatibility, but introduce new methods that
handle data hidden behind a new class "SharedBuffer", and that just translate it back and forth to compatibility methods by default.
Overloading the new methods in a client would get access to the "SharedBuffer" object and zero copy option.
Event unmodified client would benefit from the change, since inter-process exchanged would be done with zero copy.
On the driver side, drivers would receive a connected unix socket from indi server.
They should be updated to allocate/free shareable buffers (ie use the new API), to avoid more data copy.
However, this can be a progressive process.
I could eventually work on this on some spare time if it ends up being interesting and doable.
I already implemented something similar for Mobindi (for sharing preview buffers) and found it very straightforward.
But first, I would like your feedback on interest and feasibility !
My own work with memfd: github.com/pludov/mobindi/tree/memfd (see last commits)
A very usefull example: github.com/a-darwish/memfd-examples
You now need to have indi headers installed (package libindi-dev if you did not compile indi from sources)
Time for another new version of Mobindi !
The two most important changes are:
Hello All !
A new version of Mobindi is out, with the following improvements:
And if you reboot now ?
What if you issue that from terminal ?
Especially does it create files in the log dir ?
And what is the output of : multilog --version
That's very strange !
Can you create /var/log/mobindi + chown it ? Then retry from terminal ?
Does it work if you start startup.sh from a terminal ?
Can you post the content of /var/log/mobindi/ ?
It is probably a problem about log output.
I just pushed a modification to allow sending output to a log file.
Please update to latest github version (git pull --ff-only) and modify the line from /etc/rc.local to :
exec su -l -c "( cd ~/Astro/mobindi && LOGDIR=/var/log/mobindi exec ./startup.sh 2>&1 )" linaro
It should work upon next reboot !
Always call startup.sh. It will ensure all required component are started.
To automate the start, you can just add the following to /etc/rc.local
exec su -l -c "( cd ~/Astro/mobindi && exec ./startup.sh 2>&1 )" linaro
Adapt the path (~/Astro/mobindi) and the user (linaro).