×

INDI Library v2.0.6 is Released (02 Feb 2024)

Bi-monthly release with minor bug fixes and improvements

Fast BLOBs

Fast BLOBs was created by Jasem Mutlaq

We just merged a very significant MR from Ludovic Pollet that adds Fast Blob support to INDI. You can read the details of this change in the MR. There is preliminary support for this in KStars GIT as well.

This is supposed to make BLOB transfers where the server + client are on the same machine faster. Please test and report back any issues.
The following user(s) said Thank You: Hy Murveit, Chris Kuethe
1 year 9 months ago #83650

Please Log in or Create an account to join the conversation.

  • Posts: 114
  • Thank you received: 17

Replied by Denis on topic Fast BLOBs

Any documentation how use this feature from third party INDI clients?
Transfer of raw image (6224x4168x16bit) from INDI server to my client via base64 takes 1.2 s on my OrangePI 4 LTS. This is very slow. For example image demosaic takes 0.15s only. I want to implement and use Fast BLOBs
Last edit: 1 year 2 months ago by Denis.
1 year 2 months ago #89881

Please Log in or Create an account to join the conversation.

  • Posts: 38
  • Thank you received: 30

Replied by Ludovic Pollet on topic Fast BLOBs

This is transparent as long as:
* server and client run on the same host
* server is recent enough
* client also uses updated library
* client connects specifically to localhost (especially not 127.0.0.1). You can also force it with "localhost:/tmp/tmp/indiserver". This will refuse to connect if using a log library

You can check in the indiserver log: when a client connect with the new scheme it display a message about local domain:

... new arrival from local ...

Versus plain old tcp connection which display :

...new arrival from <ip-address>:port
1 year 2 months ago #89885

Please Log in or Create an account to join the conversation.

  • Posts: 114
  • Thank you received: 17

Replied by Denis on topic Fast BLOBs

My client is in same host. The problem is I don't use INDI client API. My client use own implementation for XML-based protocol. I want to implement fastblobs now. I found only docs.indilib.org/drivers/binary-transfers.html#fast-blobs but no technical details there
1 year 2 months ago #89886

Please Log in or Create an account to join the conversation.

  • Posts: 38
  • Thank you received: 30

Replied by Ludovic Pollet on topic Fast BLOBs

Hi Denis

We certainly miss that documentation !

The fast-blob protocol is mostly the same than default one, with the following deviations:
  • The client must connect to unix domain socket (only supported under Linux, MacOS lacks some important feature there...).
  • the client has to reply to "pingRequest" messages by "pingReply" (this implements the back-pressure, see below why it's important). The uid attribute from the request must be repeated in the reply:
    <pingRequest uid='2'/>
    <pingReply uid='2'/>
  • on UNIX domain connection, the server can pass the blob content as an anciliary file descriptor (man7.org/linux/man-pages/man7/unix.7.html). The file descriptor can later be mmapped. In that case, the xml message will have the attached property set to true (and no enclen by the way):
    <setBLOBVector device='fakedev1' name='someblob' timestamp='2018-01-01T00:01:00'>
        <oneBLOB name='content' size='32' format='.fits' attached='true'/>
    </setBLOBVector>
Remark 1: It is guaranteed that the file descriptor(s) for attached blob(s) are received by the end of the xml message. The exact position probably depends on the OS.
Remark 2: The server may also decide to not used shared buffer so the client must still support the use existing base64 layout (for example for small blobs)
Remark 3: The transport of buffers as filedescriptor can be very very fast, so your client may receive GB of data without much processing. Without backpressure, memory in the client side may grow uncontrolled (ultimately leading to Out of memory error, possibly system wide invoking the OOM killer). For this reason, it is strongly advised that when a blob arrive, the client stops processing incoming messages (especially replying to pingRequests) until it has finished processing the blob.
Remark 4: To actually free the memory used by a filedescriptor, the client must close the fd, and munmap the memory area.

Some example of low level functions for these tasks can be found in the indi source file integs/test.cpp (integration tests), which is somewhat decoupled from indi so probably easier to read for a start:
  • integs/test.cpp: the unixSocketConnect function - to connect to a local socket by path name
  • integs/ConnectionMock.cpp, method ConnectionMock::read, shows how to read the input channel, collecting received fds at the same time
  • integs/SharedBuffer.cpp, shows how to access memory from a fd. It is in the write direction, while you client will want to read insted. So your client should use PROT_READ only and remove PROT_WRITE)

Tests codes in the integs directory may also be useful as they show the plain text version of the messages for various cases.

Hope this helps!
The following user(s) said Thank You: Jasem Mutlaq, Denis
1 year 1 month ago #89996

Please Log in or Create an account to join the conversation.

Replied by Jasem Mutlaq on topic Fast BLOBs

I added the above to the documentation over at docs.indilib.org
1 year 1 month ago #90013

Please Log in or Create an account to join the conversation.

  • Posts: 114
  • Thank you received: 17

Replied by Denis on topic Fast BLOBs

Thanks for detailed docs! But for now, it's too complex for me. Also, I rewrote the base64-decode code and now it takes only 0.4s to receive and decode base64 blob from network stream.
But I will return to this problem after I get more familiar with Linux.
1 year 1 month ago #90048

Please Log in or Create an account to join the conversation.

Time to create page: 0.742 seconds