×

INDI Library v2.0.6 is Released (02 Feb 2024)

Bi-monthly release with minor bug fixes and improvements

indi_canon_ccd ERROR:Unsufficient memory

  • Posts: 48
  • Thank you received: 7
Hello.

I have a problem.
I was capturing using canon T7i.
I download the captured image to the client in FITS format.
After 15 consecutive captures, KStars output the following error.
Cannot convert /tmp/indi_V62GLb : Unsufficient memory
Exposure failed to parse raw image.

And capture was aborted.

When I monitor the memory consumption of the INDI server, many free memory decreases per capture.
When an error is output, the free memory of the server is almost empty.
Also, memory is not freed until the INDI server is restarted.
In order to resolve this error, it was necessary to stop the INDI server and restart it.
There were no such errors in the past.

Do you have good advice?
Thank you.


I am using the following. and Log.

INDI server
Raspberry Pi 3
Ubuntu Mate 16.04
INDI Latest nightiy builds
libgphoto2 2.5.18

Client
Ubuntu 18.04
KStars Latest nightiy builds
libgphoto2 2.5.18
5 years 8 months ago #26977
Attachments:

Please Log in or Create an account to join the conversation.

  • Posts: 984
  • Thank you received: 160
Without closer checks, I had a similar problem in a low memory environment when FITSviewer kept all the photos in memory. Try checking the "Single Window Open" and "Single Preview Tab" boxes in Options/FITS. Hope this helps.
Last edit: 5 years 8 months ago by Alfred.
5 years 8 months ago #27016

Please Log in or Create an account to join the conversation.

  • Posts: 13
  • Thank you received: 4
I've encountered this same bug/error using the indi_gphoto_ccd driver with my Canon T6, also using a Raspberry Pi as a remote INDI server (although mine is a 3B+.) I'm able to take about 80 exposures before receiving the "Insufficient Memory" error, and then if I restart INDI, I'm able to take another 80 before receiving the error again. I would reproduce it and post my logs, but I'm not sure how much it would help given that you've posted your logs already and they're probably almost identical.

At some point, when I get the time, I'll need to investigate this more closely, as this definitely has been a pain for me (having to stop and restart my sequences every time this happens.) Seems to be a memory leak somewhere in INDI, I would guess, since restarting the INDI server corrects it immediately.
5 years 8 months ago #27018

Please Log in or Create an account to join the conversation.

  • Posts: 48
  • Thank you received: 7
Dear Herrhausen.

Thanks for your advice.
Both "Single Window Open" and "Single Preview Tab" have been checked in my setting options.
It seems that this is not the cause, as the INDI server and the KStars client are running on separate machines.
5 years 8 months ago #27039

Please Log in or Create an account to join the conversation.

  • Posts: 48
  • Thank you received: 7
Hi smhxx,
I feel the same way. I am in trouble in exactly the same situation.
If you can reproduce the problem, can you post a log?
It will surely help you to analyze the situation.

The normal log is as follows.
INFO    1001.632236 sec    : Exposure done, downloading image...
DEBUG    1007.745475 sec    : read_libraw: raw_width: 6288 top_margin 0 left_margin 0 first_visible_pixel 0
DEBUG    1007.780411 sec    : read_libraw: rawdata.sizes.width: 6288 rawdata.sizes.height 4056 memsize 51008256 bayer_pattern RGGB
DEBUG    1007.900445 sec    : read_libraw: memsize (51008256) naxis (2) w (6288) h (4056) bpp (16) pattern (RGGB)
DEBUG    1008.470803 sec    : Uploading file. Ext: fits, Size: 51013440, sendImage? Yes, saveImage? No
DEBUG    1009.271505 sec    : Upload complete

The error log is as follows.
INFO    1052.571810 sec    : Exposure done, downloading image...
ERROR    1057.966632 sec    : Cannot convert /tmp/indi_V62GLb : Unsufficient memory
ERROR    1057.967516 sec    : Exposure failed to parse raw image.

An error is output by the read_libraw () function defined in indi-gphoto / gphoto_readimage.cpp.
Looking at the source code, the result of Libraw 's raw2image () is an error.
It is considered an error due to memory shortage, so I do not know whether there is a problem with libraw or indi_gphoto.
There is a possibility that indi_gphoto forgot to release memory.

I tracked memory usage. (ps aux command)
It is a comparison of VSZ and RSS when I capture 10 times.
Count	%MEM	VSZ	RSS	COMMAND
1	10.2	150068	96820	indi_canon_ccd
2	13.7	183432	129996	indi_canon_ccd
3	17.2	216544	163108	indi_canon_ccd
4	20.7	249664	196228	indi_canon_ccd
5	24.1	282784	229348	indi_canon_ccd
6	27.6	315916	262480	indi_canon_ccd
7	31.1	349064	295628	indi_canon_ccd
8	34.6	382224	328788	indi_canon_ccd
9	38.1	415392	361956	indi_canon_ccd
10	41.6	448560	395124	indi_canon_ccd

The memory used is increasing about 33000 kb each time it is taken.
And that memory is not freed when the capture is over.
The only way to free it is to restart the INDI server.
The following user(s) said Thank You: Ryan M
5 years 8 months ago #27040

Please Log in or Create an account to join the conversation.

  • Posts: 13
  • Thank you received: 4
Interesting... I think I may have isolated a memory leak, but just doing a quick calculation, it doesn't seem like it would account for anywhere near the 33 MB per exposure that you're observing. Right now I can only account for about 280 KB leaked per exposure, unless I'm missing something... there is definitely something fishy about the way that INDI is handling libraw though, and it definitely is leaking memory, I just need to do a little more digging to figure out why it's leaking so much. I want to make sure that this bug is actually responsible for the whole 33 MB before celebrating, since there's quite a difference between 33 MB and 280 KB. If I can determine where that other 32.7 MB is coming from, I'll submit a bugfix patch.

One clue as to that mysterious 33 MB figure is that it's almost exactly the size that I would expect a single raw exposure from a Canon T7i to be (I'm not sure why the logs are showing it as allocating 51 MB, that seems a bit high to me.) Anyway, it seems very likely that the entire contents of the image are being leaked somewhere, either in RAW format before the conversion, or FITS format afterwards. Either would have the same effect, obviously, since eventually it would get to the point where there wasn't enough memory left to read the next image in.
Last edit: 5 years 8 months ago by Ryan M.
5 years 8 months ago #27042

Please Log in or Create an account to join the conversation.

  • Posts: 111
  • Thank you received: 40
Is there not a
RawProcessor.recycle()
missing somewhere in the code ?
I cannot see where the resources are free'd when read_libraw return successful.
In other words:
int read_libraw(const char *filename, uint8_t **memptr, size_t *memsize, int *n_axis, int *w, int *h, int *bitsperpixel,                                                      
                char *bayer_pattern)                                                                                                                                          
{                                                                                                                                                                             
    int ret = 0;                                                                                                                                                              
    // Creation of image processing object                                                                                                                                    
    LibRaw RawProcessor;                                                                                                                                                      
 
    // Let us open the file                                                                                                                                                   
    if ((ret = RawProcessor.open_file(filename)) != LIBRAW_SUCCESS)                                                                                                           
    {                                                                                                                                                                         
        DEBUGFDEVICE(device, INDI::Logger::DBG_ERROR, "Cannot open %s: %s", filename, libraw_strerror(ret));                                                                  
        RawProcessor.recycle();                                                                                                                                               
        return -1;                                                                                                                                                            
    }                                                                                                                                                                         
 
    // Let us unpack the image                                                                                                                                                
    if ((ret = RawProcessor.unpack()) != LIBRAW_SUCCESS)                                                                                                                      
    {                                                                                                                                                                         
        DEBUGFDEVICE(device, INDI::Logger::DBG_ERROR, "Cannot unpack %s: %s", filename, libraw_strerror(ret));                                                                
        RawProcessor.recycle();                                                                                                                                               
        return -1;                                                                                                                                                            
    }
...
...
  for (int i = 0; i < RawProcessor.imgdata.rawdata.sizes.height; i++)                                                                                                       
    {                                                                                                                                                                         
        memcpy(image, src, RawProcessor.imgdata.rawdata.sizes.width * 2);                                                                                                     
        image += RawProcessor.imgdata.rawdata.sizes.width;                                                                                                                    
        src += RawProcessor.imgdata.rawdata.sizes.raw_width;                                                                                                                  
    }                                                                                                                                                                         
 
    return 0;                                                                                                                                                                 
}

However, outside this function, there is no RawProcessor.recycle(), except:
~/dev/astronomy/indi/3rdparty/indi-gphoto>grep -r "recy" .
./gphoto_driver.cpp:	lib_raw.recycle();
./gphoto_readimage.cpp:        RawProcessor.recycle();
./gphoto_readimage.cpp:        RawProcessor.recycle();
./gphoto_readimage.cpp:        RawProcessor.recycle();
The following user(s) said Thank You: Ryan M
Last edit: 5 years 8 months ago by Thomas Stibor.
5 years 8 months ago #27043

Please Log in or Create an account to join the conversation.

  • Posts: 13
  • Thank you received: 4
That might do it. I was so focused on the fact that "RawProcessor.recycle()" should actually be "delete RawProcessor" (since the object is not being reused) that I didn't check that it was present in all possible branches. Good find, I bet that's it.

EDIT: Actually, now I'm not so sure (C++ isn't my strong suit as far as programming goes.) The LibRaw class has a destructor, so wouldn't that be automatically called when RawProcessor goes out of scope? I'm not sure that any of the recycle() calls are actually necessary here, I think it would be called automatically as recycle() is invoked in the destructor method. Maybe someone with more C++ knowledge than me can figure out whether that's really the issue.
Last edit: 5 years 8 months ago by Ryan M.
5 years 8 months ago #27044

Please Log in or Create an account to join the conversation.

  • Posts: 13
  • Thank you received: 4
Try this:
sudo add-apt-repository ppa:mutlaqja/indinightly
sudo apt install indi-gphoto

I just realized this bug was fixed over a month ago but there hasn't been a new release of the INDI drivers since then, so the memory leak still exists on the current stable version.
The following user(s) said Thank You: norikyu
5 years 8 months ago #27053

Please Log in or Create an account to join the conversation.

  • Posts: 48
  • Thank you received: 7
Hi smhxx,

I tried your advice. It worked very well!
A memory leak has not occurred.
The problem is solved with the latest nightly builds.
I checked my environment.
Cause my nightly builds automatic download script seemed to be failing.
I'm sorry.
Also, as you say, note that this problem occurs on the current stable version.

Thank you.
5 years 8 months ago #27067

Please Log in or Create an account to join the conversation.

Time to create page: 1.510 seconds