Pete Simpson replied to the topic 'KStars memory leak and crash' in the forum. 2 years ago

Another follow up from me - another crash noted:
Last lines of log file:

[2022-07-05T16:51:07.950 BST DEBG ][           org.kde.kstars.indi] - CCD Simulator : "[DEBUG] Upload complete "
[2022-07-05T16:51:07.951 BST DEBG ][           org.kde.kstars.indi] - Image received. Mode: "Normal" Size: 2629440
[2022-07-05T16:51:07.955 BST INFO ][           org.kde.kstars.indi] - "FITS" file saved to "/home/astroberry/Astro/Light/Luminance/Light_Luminance_1_secs_2022-07-05T16-51-07_19659.fits"
[2022-07-05T16:51:07.958 BST INFO ][   org.kde.kstars.ekos.capture] - "Download Time: 0.01 s, New Download Time Estimate: 0.05 s."
[2022-07-05T16:51:09.641 BST INFO ][   org.kde.kstars.ekos.capture] - "Received image 19642 out of 50000."
[2022-07-05T16:51:12.584 BST DEBG ][           org.kde.kstars.indi] - Telescope Simulator : "[SCOPE] mountToApparentRaDec 87.305623, 90.000000 to ha 87.305623, ra 74.096583, 90.000000 "
[2022-07-05T16:51:12.584 BST DEBG ][           org.kde.kstars.indi] - Telescope Simulator : "[SCOPE] HaAxis: 87.305623, ra 4.939772 "
[2022-07-05T16:51:12.585 BST DEBG ][           org.kde.kstars.indi] - Telescope Simulator : "[SCOPE] DecAxis: 90.000000, dec 90.000000 "
[2022-07-05T16:51:12.585 BST DEBG ][           org.kde.kstars.indi] - Telescope Simulator : "[SCOPE] Current RA:  4:56:23 Current DEC: 90:00:00 "
[2022-07-05T16:51:12.586 BST DEBG ][           org.kde.kstars.indi] - Telescope Simulator : "[SCOPE] mountToApparentRaDec 87.306670, 90.000000 to ha 87.306670, ra 74.096583, 90.000000 "
[2022-07-05T16:51:12.586 BST DEBG ][           org.kde.kstars.indi] - Telescope Simulator : "[SCOPE] HaAxis: 87.306670, ra 4.939772 "
[2022-07-05T16:51:12.587 BST DEBG ][           org.kde.kstars.indi] - Telescope Simulator : "[SCOPE] DecAxis: 90.000000, dec 90.000000 "
[2022-07-05T16:51:12.587 BST DEBG ][           org.kde.kstars.indi] - Telescope Simulator : "[SCOPE] Current RA:  4:56:23 Current DEC: 90:00:00 "
[2022-07-05T16:51:12.588 BST DEBG ][           org.kde.kstars.indi] - INDI Server:  "2022-07-05T15:51:05: Driver indi_simulator_ccd: ****Coordinates (-c) error in "
[2022-07-05T16:51:12.588 BST DEBG ][           org.kde.kstars.indi] - INDI Server:  ""
[2022-07-05T16:51:13.440 BST INFO ][   org.kde.kstars.ekos.capture] - "Capturing 1.000-second Luminance image..."
And memory statistics:

The vertical jumps in the middle of the X axis is in agreement with the last log inputs.
Data from seconds around this:
2022-07-05 16:51:07 Temp=41.8ºC. RAM parameters: Used: 18.6%; Free:  2.5%; Shared:  1.0%; Buffer:  0.7%; Cache: 78.2%; Available: 76.5%. Swap used:100.0%. µSD has 32.601% free space.
2022-07-05 16:51:08 Temp=41.3ºC. RAM parameters: Used: 18.4%; Free:  2.6%; Shared:  1.0%; Buffer:  0.7%; Cache: 78.3%; Available: 76.6%. Swap used:100.0%. µSD has 32.599% free space.
2022-07-05 16:51:10 Temp=41.3ºC. RAM parameters: Used: 18.4%; Free:  2.6%; Shared:  1.0%; Buffer:  0.7%; Cache: 78.3%; Available: 76.6%. Swap used:100.0%. µSD has 32.599% free space.
2022-07-05 16:51:11 Temp=41.8ºC. RAM parameters: Used: 18.4%; Free:  2.6%; Shared:  1.0%; Buffer:  0.7%; Cache: 78.3%; Available: 76.6%. Swap used:100.0%. µSD has 32.599% free space.
2022-07-05 16:51:13 Temp=42.8ºC. RAM parameters: Used: 18.4%; Free:  2.6%; Shared:  1.0%; Buffer:  0.7%; Cache: 78.3%; Available: 76.7%. Swap used:100.0%. µSD has 32.599% free space.
2022-07-05 16:51:14 Temp=41.8ºC. RAM parameters: Used: 18.5%; Free:  2.4%; Shared:  1.0%; Buffer:  0.7%; Cache: 78.3%; Available: 76.4%. Swap used:100.0%. µSD has 32.599% free space.
2022-07-05 16:51:15 Temp=41.3ºC. RAM parameters: Used:  6.6%; Free: 14.9%; Shared:  0.4%; Buffer:  0.7%; Cache: 77.8%; Available: 89.1%. Swap used: 99.9%. µSD has 32.600% free space.
2022-07-05 16:51:17 Temp=41.3ºC. RAM parameters: Used:  6.6%; Free: 14.9%; Shared:  0.4%; Buffer:  0.7%; Cache: 77.8%; Available: 89.1%. Swap used: 99.9%. µSD has 32.600% free space.
2022-07-05 16:51:18 Temp=41.8ºC. RAM parameters: Used:  6.6%; Free: 14.9%; Shared:  0.4%; Buffer:  0.7%; Cache: 77.8%; Available: 89.1%. Swap used: 99.9%. µSD has 32.600% free space.
2022-07-05 16:51:20 Temp=40.9ºC. RAM parameters: Used:  6.5%; Free: 14.9%; Shared:  0.4%; Buffer:  0.7%; Cache: 77.8%; Available: 89.1%. Swap used: 99.9%. µSD has 32.600% free space.
2022-07-05 16:51:21 Temp=41.3ºC. RAM parameters: Used:  6.6%; Free: 14.9%; Shared:  0.4%; Buffer:  0.7%; Cache: 77.8%; Available: 89.1%. Swap used: 99.9%. µSD has 32.600% free space.
2022-07-05 16:51:23 Temp=41.8ºC. RAM parameters: Used:  6.5%; Free: 14.9%; Shared:  0.4%; Buffer:  0.7%; Cache: 77.8%; Available: 89.1%. Swap used: 99.9%. µSD has 32.600% free space.
2022-07-05 16:51:24 Temp=42.3ºC. RAM parameters: Used:  6.6%; Free: 14.9%; Shared:  0.4%; Buffer:  0.7%; Cache: 77.8%; Available: 89.1%. Swap used: 99.9%. µSD has 32.600% free space.


RPi 4B, 4GB
VNC server, Conky & script detailing Memory usage running. Nothing else started from reboot.
KStars 3.5.9:
All notifications disabled
Limited resources mode on
FITS viewer etc off
Not running any scheduler
SIMUALTOR for Telescope (mount) and Camera in EKOS.
Restarting KStars after test (just after stopping data aquisition for above plot) and starting Indi reports 'another instance of INDI running...'

Hoping for a clear night tomorrow night to do real imaging (Steve, I'll be running internal plate solving and guiding so hope to report back with that too) with the cache-purging option which through various tests to date have shown longer runs possible without crashing. Watch this space!

Read More...

Pete Simpson replied to the topic 'KStars memory leak and crash' in the forum. 2 years ago

@Nou, sorry if my terminology is incorrect. Just assumed memory leak is when memory is used up continuously, until the system has used all resources and causes things to crash. Let's not get bogged down with terminology!
As in my original postpost , I've had several sessions crash shortly after going to bed, and believe them to be from a lack of memory on the RPi.
I say this as I've documented the memory usage fill up, then the kstars logfile stops writing (kstars crashes) and the available memory increases and remains steady until I return to offload the imaging data several hours later.
Since really delving deeper (all the different memories, and purging the caches, etc), I've been unable to make it crash again, and as the nights are so short imaging isn't worth it just at present.
Again, I have a 'fix' that should work for me, but I'm disappointed that I've not been able to find the reason and solve the problem, but it does seem like there are others still seeing the same, perhaps without crashes...

Read More...

Pete Simpson replied to the topic 'KStars memory leak and crash' in the forum. 2 years ago

Nope, still seeing memory usage increase with notifications on or off.
Sorry folks.
I'm not sure how to run heaptrack. It's installed, but not a great linux user to do have to either ask for a how-to crash course, sorry folks.

Read More...

Pete Simpson replied to the topic 'KStars memory leak and crash' in the forum. 2 years ago

Ooooo!
So in my alternative post, Nou had commented about a message I'd seen in the log file (and dismissed!) about a phonon error or warning . I'd seen it in the SGL post and as mentioned, ignored it for now.
Seeing Nou's comment, I'd disabled the SOUNDS only but instead wrote warnings to a text file so I'd be able to see warnings or track them back.
Just started another simulation with ALL notifications OFF, and also disabled the "low resources" mode - it looks like my memory usage is more stable, although that's only after a short period. I'll leave it running and come back with another report later.
There was also another update released recently for lots of Indi updates, so if the issue is fixed, it could have been either - I'll test again later with notifications back on and see what happens.

Read More...

Pete Simpson replied to the topic 'KStars memory leak and crash' in the forum. 2 years ago

German,
Not sure if you've got anywhere with this - I'd seen the same thing with memory increasing and eventually KStars crashes mid-imaging. In summary, RPi 4 4GB, + astroberry, Canon DSLR, AZEQ6 & ZWO 120mm guidecam.
Tracked things down, by several simulation tests, wrote shell scripts to output memory usage and noted the cache seems to fill up, and eventually starts using Swap memory.
By updating the script to watch out for the cache and available memory, and if it gets too full, then things get purged automatically.
Memory usage keeps rising, but the script keeps purging, so I'm hoping I have found a workaround rather than solution.
My post here shows my extensive tests, and scripts also so you could try and see if you got same result/success?
Post back so we can see!

Read More...

Thanks again guys, Coming back to original question, why are things crashing - Nigel catches this in the above post correctly! My own 'memory monitor' script fix/hack should now allow me to avoid the crash issue, but ultimately, I still haven't got to the bottom of the crashing EKOS.
If more swap should be required, astroberry should probably do this at time of install with etcher or however. The basic install using etcher has (I believe?) done all the memory allocation for me so the small swap I'd guess is the default. If it is astroberry doing this, maybe it would look at the size of the µSD and 'smartly' allocate memory? Is it this reason astroberry minimum is 16GB with 32GB recommended? If it allocates 4GB swap on a 16GB card, 7 with OS, then that's not really going to handle lots of imaging time! :lol: :lol: :lol: Again, not a computer guru but happy to dig around in my own time, and from reading around, the use of swap does appear to provoke 'try best to avoid' reactions.
Nigel, sods law, the last test with no drop_cache is still running, which if I was imaging, wouldn't be an issue! (36hrs+ without crash). So much for trying to show the crash on the memory graph! I'm pretty sure the crash is down to this memory thing, so happy that I've at least raised the issue and at the very least, your 1st post also told me you were seeing the same effect on cache and free!
gdb got skipped till now, sorry Nigel. I skipped this as I'd just discovered the drop_cache idea, so gdb will be next. I'll also browse around to see about swap agression.
Anyway, I'll do some more tests and just keep popping results here, but I'd hope (come September/October) when the nights are longer that I can do more imaging and confirm that at least the drop_cache will work, giving me what will be my first full night of imaging (previous sessions I've packed up before bed in the wee hours, or crashes each night when leaving things out overnight)!
Thanks again for comments

Read More...

Colours on graph changed to order in the rainbow scheme. New test still running (crash point never certain, but shows same effect - cache increases as free decreases, and eventually swap gets used.
At this point, I've no idea how long the simulation will continue, other than at some point a crash and stop of acquisition WILL occur.
Also redone the previous text file to show the plots on the same colour scheme.
Without drop_cache:



With drop_cache:


Read More...

Thanks Hy,
So plot is the reported stats from the linux FREE command as can be seen in the script (assuing you've looked at it as you mention "drop_caches"
I also attach here the output from the text file, which shows all the data that is plotted:

File Attachment:

File Name: Z_PTS_Log.txt
File Size: 5,433 KB

Total number of lines in the file (approx 1 line per second or so) is around 30k.
So the first drop_caches occurs at line ~1930, and how I've recorded this is by outputting string "Iteration %d; Cache memory greater than 75% threshold ( 76.0%). Cleaning memory in system." (search for "iteration" in the text file and we can see 15 occurrances of this - each time we get to a drop_cache, there is a drop in the CACHE memory and rise is the FREE memory - the command in the script does what I needed.
Prior to this addition, Swap would eventually get used until no resources were left and KStars/EKOS would crash, stopping imaging session - the original topic of this discussion.
With the plot, sure enough, we see 15 'jumps' where the cache used is cleared up (cache shown by purple line) - it rises continuously, until cleared.
The yellow FREE plot (again these terms seen in the legend) are the % returned from the "free -w" command in bash) drops until a "drop_caches" when the FREE available jumps up again.

Available swap (unchanged from the OS install) noted at top of the attached file as 99MB. Again, throughout previous tests, swap was used (which from my understanding and eg searching isn't 'ideal', but not disasterous). This latest test requires 0% swap used throughout.
Unfortunately, I've not done any real editing of the graph colours - "SWAP usage" and "FREE memory" are both yellow, but the SWAP stays at 0% usage throughout (look along the Y=0% line).

The HDD is the available space on the µSD which (no surprise) drops constantly from about 82% free to 50% free as the simulation files are written at regular intervals ("exposures" of 2.5 sec)
Not sure what else to comment here - the astroberry OS is a clean install, add on Conky and Anydesk, then running KStars for many imaging sessions etc. When I became adventurous and left the rig overnight and went to bed, I'd get up and things had crashed so I've gone down this debug route.

Prior to the addition of the "drop_caches" syntax, the cache would get used up (see previous posts), I would expect the yellow line would continue to decrease, swap would rise, and things would appear to 'stabilise' here for a period, before yes, KStars would crash.
I'm going to run another test to prove this as I'd updated several times throughout this testing. Essentially it'll be the same output. I'll just comment the "drop_caches" line in the script... Watch this space then!

RE SSD - certainly not against the idea! I did look at SSD, although just at the moment, I've done 3D prints which hold all the RPi, cables, etc, on the mount, so just at this point would rather keep things tidier. But yes, I would still consider SSD! I've also considered moving from astroberry OS to stellarmate OS (stellarmate running on the RPi), and thought that the latter would be the best way to get around this issue, although having discovered the memory issue, feel I can run at least for now with my 'hack'.

Does this help?
I guess I don't want to get bogged down with terms and techy stuff, just that I think I've found an issue that causes my system to crash if not kept in check - are others seeing this? Is this a hardware issue (RPi defective? µSD causing this issue? Older DSLR with latest software? Power instability in the cold? etc!) or software (KStars clashing with Conky/Anydesk? ETC ETC?) What's the root cause is where I'm aiming!

On a last note about the plot - the USED memory (bottom red/pink plot) starts low, then jumps up as KStars/EKOS/Indi is started and the imaging session is started, then stays fairly constant throughout the process, before jumping up a bit more at the end as I start looking at the files and thinking about getting data off for analysis.

Read More...

Right!
I've solved (or it appears I have solved) the issue for my setup.
All the pre-mentioned memory issues still appear to occur, but I have a solution that will "fix" the issue for me at this time.
Whilst I know the fix isn't solving the issue, I'm hoping that it makes it known that I've found an issue and able to get around the problem.
As previous, I note the usage in specific elements of RAM going up (mainly cache) when imaging with KStars/EKOS/Indi on RPi4 (4GB). It might be those with 8GB models don't see the issue as there's more resource to 'lose', but there are then later times when the software crashes with no obvious issue at the front end.
Through reading about different memory allocations, etc on Linux and my limited scripting abilities, I've written a script that outputs the memory usage every second or so for reference. This script was also updated several times and eventually finalised with one that will force a memory purge by the system, should the resources available become too little.
Prior to the memory purge, I could run the EKOS simulators and cause the software to crash, similar to the real-life imaging that I was experiencing earlier in the year (before the clocks changed and imaging conditions became a bit too short!).
I'm more familiar with Matlab coding and wrote a script that will take the text file and output the data in a numerical format to plot it and have the following after a 12hour simulation session (15k images obtained). Sure enough, one can see that the memory is purged sevaral times throughout the simulation, but I do not suffer from any crash - I ran this prior and ended up filling the 128GB µSD card with simulation images in one go, so very happy that at least I've something that will work.



Can an admin look into this for me and see is there a memory leak or something similar?
I'm prepared to try a new µSD or whatever, but there appears to be at least a few of us with similar issues.
Script file attached, hoping it comments enough to allow other/better coders to let them know my thought process.
Regards to all

File Attachment:

File Name: SampleShell_2022-05-24.txt
File Size: 5 KB


Read More...

Aha! So written it into script and ran in terminal:

for i in `seq 1 $SECONDS`; do
	***Line here outputs the string to text file. It's long, so removing it from forum for now ***
	sleep 1
	if ! ((i % 5)) || ((i == 1)); then
		echo "$i, would execute clean script here"
		sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
	fi
done

The purpose of the echo is for debugging whilst writing, and the i==1 so that on first purge that a sudo password is required. Therefore my thinking is this: run the purge on first time output, enter the SU password which will then be remembered and redo the purge every Nth second. Set as the MODULO , obviously not to be purged eveery 5 seconds - I figure every 10min to do this. Alternatively if the cache gets above 50%, I could do it then...

Read More...

Now I'm onto something:

2022-05-20 15:17:05 Temp=38.9ºC. RAM parameters: Used: 13.2%; Free:  3.3%; Shared:  1.9%; Buffer:  1.5%; Cache: 82.1%; Available: 81.0%. Swap used:  7.5%. µSD has 46.059% free space.
2022-05-20 15:17:06 Temp=38.9ºC. RAM parameters: Used: 13.1%; Free:  3.3%; Shared:  1.9%; Buffer:  1.5%; Cache: 82.1%; Available: 81.0%. Swap used:  7.5%. µSD has 46.059% free space.
2022-05-20 15:17:08 Temp=37.9ºC. RAM parameters: Used: 13.2%; Free:  3.2%; Shared:  1.9%; Buffer:  1.5%; Cache: 82.2%; Available: 81.0%. Swap used:  7.5%. µSD has 46.057% free space.
2022-05-20 15:17:10 Temp=38.9ºC. RAM parameters: Used: 13.1%; Free:  9.4%; Shared:  1.9%; Buffer:  0.0%; Cache: 75.5%; Available: 81.0%. Swap used:  7.5%. µSD has 46.057% free space.
2022-05-20 15:17:11 Temp=38.9ºC. RAM parameters: Used: 13.3%; Free: 28.8%; Shared:  1.9%; Buffer:  0.0%; Cache: 55.7%; Available: 81.2%. Swap used:  7.5%. µSD has 46.055% free space.
2022-05-20 15:17:13 Temp=38.9ºC. RAM parameters: Used: 13.1%; Free: 50.8%; Shared:  1.9%; Buffer:  0.0%; Cache: 33.7%; Available: 81.3%. Swap used:  7.5%. µSD has 46.055% free space.
2022-05-20 15:17:15 Temp=38.9ºC. RAM parameters: Used: 13.2%; Free: 71.8%; Shared:  1.9%; Buffer:  0.0%; Cache: 12.5%; Available: 81.5%. Swap used:  7.5%. µSD has 46.053% free space.
2022-05-20 15:17:16 Temp=38.4ºC. RAM parameters: Used: 13.1%; Free: 77.7%; Shared:  1.9%; Buffer:  0.0%; Cache:  9.1%; Available: 81.6%. Swap used:  7.5%. µSD has 46.053% free space.
2022-05-20 15:17:18 Temp=39.4ºC. RAM parameters: Used: 13.1%; Free: 79.1%; Shared:  1.9%; Buffer:  0.1%; Cache:  7.7%; Available: 81.8%. Swap used:  7.5%. µSD has 46.051% free space.
2022-05-20 15:17:20 Temp=39.4ºC. RAM parameters: Used: 13.1%; Free: 79.1%; Shared:  1.9%; Buffer:  0.1%; Cache:  7.7%; Available: 81.8%. Swap used:  7.5%. µSD has 46.051% free space.
2022-05-20 15:17:21 Temp=39.4ºC. RAM parameters: Used: 13.1%; Free: 79.0%; Shared:  1.9%; Buffer:  0.1%; Cache:  7.8%; Available: 81.8%. Swap used:  7.5%. µSD has 46.049% free space.
2022-05-20 15:17:23 Temp=38.9ºC. RAM parameters: Used: 13.1%; Free: 79.0%; Shared:  1.9%; Buffer:  0.1%; Cache:  7.8%; Available: 81.8%. Swap used:  7.5%. µSD has 46.049% free space.
2022-05-20 15:17:24 Temp=38.4ºC. RAM parameters: Used: 13.1%; Free: 79.0%; Shared:  1.9%; Buffer:  0.1%; Cache:  7.9%; Available: 81.8%. Swap used:  7.5%. µSD has 46.047% free space.
2022-05-20 15:17:26 Temp=38.9ºC. RAM parameters: Used: 13.1%; Free: 79.0%; Shared:  1.9%; Buffer:  0.1%; Cache:  7.9%; Available: 81.8%. Swap used:  7.5%. µSD has 46.047% free space.
2022-05-20 15:17:27 Temp=38.4ºC. RAM parameters: Used: 13.1%; Free: 78.9%; Shared:  1.9%; Buffer:  0.1%; Cache:  7.9%; Available: 81.7%. Swap used:  7.5%. µSD has 46.044% free space.
2022-05-20 15:17:29 Temp=38.9ºC. RAM parameters: Used: 13.1%; Free: 78.9%; Shared:  1.9%; Buffer:  0.1%; Cache:  7.9%; Available: 81.7%. Swap used:  7.5%. µSD has 46.044% free space.
2022-05-20 15:17:30 Temp=39.4ºC. RAM parameters: Used: 13.1%; Free: 78.8%; Shared:  1.9%; Buffer:  0.1%; Cache:  8.0%; Available: 81.8%. Swap used:  7.5%. µSD has 46.042% free space.
2022-05-20 15:17:32 Temp=37.9ºC. RAM parameters: Used: 13.1%; Free: 78.8%; Shared:  1.9%; Buffer:  0.1%; Cache:  8.0%; Available: 81.8%. Swap used:  7.5%. µSD has 46.042% free space.
2022-05-20 15:17:34 Temp=37.9ºC. RAM parameters: Used: 13.1%; Free: 78.8%; Shared:  1.9%; Buffer:  0.1%; Cache:  8.1%; Available: 81.7%. Swap used:  7.5%. µSD has 46.040% free space.
2022-05-20 15:17:35 Temp=38.9ºC. RAM parameters: Used: 13.1%; Free: 78.7%; Shared:  1.9%; Buffer:  0.1%; Cache:  8.1%; Available: 81.6%. Swap used:  7.5%. µSD has 46.039% free space.
2022-05-20 15:17:37 Temp=38.9ºC. RAM parameters: Used: 13.1%; Free: 78.7%; Shared:  1.9%; Buffer:  0.1%; Cache:  8.1%; Available: 81.7%. Swap used:  7.5%. µSD has 46.038% free space.
2022-05-20 15:17:39 Temp=39.4ºC. RAM parameters: Used: 13.1%; Free: 78.6%; Shared:  1.9%; Buffer:  0.1%; Cache:  8.2%; Available: 81.8%. Swap used:  7.5%. µSD has 46.036% free space.
2022-05-20 15:17:40 Temp=38.4ºC. RAM parameters: Used: 13.1%; Free: 78.6%; Shared:  1.9%; Buffer:  0.1%; Cache:  8.2%; Available: 81.8%. Swap used:  7.5%. µSD has 46.036% free space.
2022-05-20 15:17:42 Temp=38.9ºC. RAM parameters: Used: 13.2%; Free: 78.5%; Shared:  1.9%; Buffer:  0.1%; Cache:  8.2%; Available: 81.8%. Swap used:  7.5%. µSD has 46.034% free space.
2022-05-20 15:17:44 Temp=38.4ºC. RAM parameters: Used: 13.1%; Free: 78.5%; Shared:  1.9%; Buffer:  0.1%; Cache:  8.3%; Available: 81.7%. Swap used:  7.5%. µSD has 46.034% free space.
2022-05-20 15:17:45 Temp=38.9ºC. RAM parameters: Used: 13.3%; Free: 78.4%; Shared:  1.9%; Buffer:  0.1%; Cache:  8.3%; Available: 81.8%. Swap used:  7.5%. µSD has 46.032% free space.
2022-05-20 15:17:47 Temp=39.4ºC. RAM parameters: Used: 13.1%; Free: 78.5%; Shared:  1.9%; Buffer:  0.1%; Cache:  8.3%; Available: 81.8%. Swap used:  7.5%. µSD has 46.032% free space.
2022-05-20 15:17:48 Temp=37.4ºC. RAM parameters: Used: 13.3%; Free: 78.4%; Shared:  1.9%; Buffer:  0.1%; Cache:  8.4%; Available: 81.7%. Swap used:  7.5%. µSD has 46.030% free space.
2022-05-20 15:17:50 Temp=38.4ºC. RAM parameters: Used: 13.1%; Free: 78.4%; Shared:  1.9%; Buffer:  0.1%; Cache:  8.4%; Available: 81.7%. Swap used:  7.5%. µSD has 46.030% free space.
2022-05-20 15:17:51 Temp=39.4ºC. RAM parameters: Used: 13.2%; Free: 78.3%; Shared:  1.9%; Buffer:  0.1%; Cache:  8.5%; Available: 81.7%. Swap used:  7.5%. µSD has 46.028% free space.
2022-05-20 15:17:53 Temp=38.9ºC. RAM parameters: Used: 13.1%; Free: 78.3%; Shared:  1.9%; Buffer:  0.1%; Cache:  8.5%; Available: 81.7%. Swap used:  7.5%. µSD has 46.028% free space.
2022-05-20 15:17:54 Temp=37.9ºC. RAM parameters: Used: 13.2%; Free: 78.3%; Shared:  1.9%; Buffer:  0.1%; Cache:  8.5%; Available: 81.7%. Swap used:  7.5%. µSD has 46.026% free space.
2022-05-20 15:17:56 Temp=39.4ºC. RAM parameters: Used: 13.1%; Free: 78.2%; Shared:  1.9%; Buffer:  0.1%; Cache:  8.4%; Available: 81.7%. Swap used:  7.5%. µSD has 46.026% free space.
2022-05-20 15:17:58 Temp=38.9ºC. RAM parameters: Used: 13.1%; Free: 79.1%; Shared:  1.9%; Buffer:  0.1%; Cache:  7.7%; Available: 81.8%. Swap used:  7.5%. µSD has 46.024% free space.
2022-05-20 15:17:59 Temp=38.4ºC. RAM parameters: Used: 13.1%; Free: 79.0%; Shared:  1.9%; Buffer:  0.1%; Cache:  7.7%; Available: 81.7%. Swap used:  7.5%. µSD has 46.024% free space.
2022-05-20 15:18:01 Temp=38.4ºC. RAM parameters: Used: 13.1%; Free: 79.1%; Shared:  1.9%; Buffer:  0.1%; Cache:  7.7%; Available: 81.8%. Swap used:  7.5%. µSD has 46.021% free space.
2022-05-20 15:18:02 Temp=38.4ºC. RAM parameters: Used: 13.1%; Free: 79.1%; Shared:  1.9%; Buffer:  0.1%; Cache:  7.7%; Available: 81.7%. Swap used:  7.5%. µSD has 46.021% free space.
2022-05-20 15:18:04 Temp=38.4ºC. RAM parameters: Used: 13.1%; Free: 79.0%; Shared:  1.9%; Buffer:  0.1%; Cache:  7.8%; Available: 81.7%. Swap used:  7.5%. µSD has 46.019% free space.
2022-05-20 15:18:05 Temp=37.9ºC. RAM parameters: Used: 13.1%; Free: 79.0%; Shared:  1.9%; Buffer:  0.1%; Cache:  7.8%; Available: 81.7%. Swap used:  7.5%. µSD has 46.019% free space.
2022-05-20 15:18:07 Temp=38.9ºC. RAM parameters: Used: 13.1%; Free: 78.9%; Shared:  1.9%; Buffer:  0.1%; Cache:  7.8%; Available: 81.8%. Swap used:  7.5%. µSD has 46.017% free space.
2022-05-20 15:18:08 Temp=37.9ºC. RAM parameters: Used: 13.1%; Free: 78.9%; Shared:  1.9%; Buffer:  0.1%; Cache:  7.8%; Available: 81.7%. Swap used:  7.5%. µSD has 46.017% free space.
2022-05-20 15:18:10 Temp=38.4ºC. RAM parameters: Used: 13.1%; Free: 78.9%; Shared:  1.9%; Buffer:  0.1%; Cache:  7.9%; Available: 81.8%. Swap used:  7.5%. µSD has 46.015% free space.
2022-05-20 15:18:12 Temp=38.4ºC. RAM parameters: Used: 13.1%; Free: 78.9%; Shared:  1.9%; Buffer:  0.1%; Cache:  7.9%; Available: 81.8%. Swap used:  7.5%. µSD has 46.015% free space.
2022-05-20 15:18:13 Temp=39.4ºC. RAM parameters: Used: 13.1%; Free: 78.8%; Shared:  1.9%; Buffer:  0.1%; Cache:  8.0%; Available: 81.7%. Swap used:  7.5%. µSD has 46.013% free space.
2022-05-20 15:18:15 Temp=38.9ºC. RAM parameters: Used: 13.1%; Free: 78.8%; Shared:  1.9%; Buffer:  0.1%; Cache:  8.0%; Available: 81.7%. Swap used:  7.5%. µSD has 46.013% free space.
2022-05-20 15:18:16 Temp=38.9ºC. RAM parameters: Used: 13.1%; Free: 78.7%; Shared:  1.9%; Buffer:  0.1%; Cache:  8.0%; Available: 81.7%. Swap used:  7.5%. µSD has 46.011% free space.
2022-05-20 15:18:18 Temp=38.9ºC. RAM parameters: Used: 13.1%; Free: 78.7%; Shared:  1.9%; Buffer:  0.1%; Cache:  8.0%; Available: 81.7%. Swap used:  7.5%. µSD has 46.011% free space.
2022-05-20 15:18:19 Temp=39.4ºC. RAM parameters: Used: 13.1%; Free: 78.7%; Shared:  1.9%; Buffer:  0.1%; Cache:  8.1%; Available: 81.7%. Swap used:  7.5%. µSD has 46.009% free space.
2022-05-20 15:18:21 Temp=37.4ºC. RAM parameters: Used: 13.1%; Free: 78.7%; Shared:  1.9%; Buffer:  0.1%; Cache:  8.1%; Available: 81.7%. Swap used:  7.5%. µSD has 46.009% free space.
2022-05-20 15:18:22 Temp=38.4ºC. RAM parameters: Used: 13.1%; Free: 78.6%; Shared:  1.9%; Buffer:  0.1%; Cache:  8.2%; Available: 81.7%. Swap used:  7.5%. µSD has 46.007% free space.
2022-05-20 15:18:24 Temp=37.4ºC. RAM parameters: Used: 13.1%; Free: 78.6%; Shared:  1.9%; Buffer:  0.1%; Cache:  8.2%; Available: 81.7%. Swap used:  7.5%. µSD has 46.007% free space.
2022-05-20 15:18:25 Temp=38.4ºC. RAM parameters: Used: 13.1%; Free: 78.5%; Shared:  1.9%; Buffer:  0.1%; Cache:  8.3%; Available: 81.8%. Swap used:  7.5%. µSD has 46.005% free space.
2022-05-20 15:18:26 Temp=38.4ºC. RAM parameters: Used: 13.1%; Free: 78.5%; Shared:  1.9%; Buffer:  0.1%; Cache:  8.3%; Available: 81.8%. Swap used:  7.5%. µSD has 46.005% free space.
2022-05-20 15:18:28 Temp=37.9ºC. RAM parameters: Used: 13.1%; Free: 78.4%; Shared:  1.9%; Buffer:  0.1%; Cache:  8.3%; Available: 81.7%. Swap used:  7.5%. µSD has 46.003% free space.
Pretty obvious that I've done something major in the middle!
After multple tries with "sync" and errors, googled the error and found this site .
Pasting
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
into terminal, mass amount of cache freed up.

It's still increasing whilst watching, but I figure that if I can add this into my memory script to repeat every minute or something like that, then I should get around the issue. Again, I'm trying to get around it - can someone suggest what the issue is or how to fix long-term?
I have been also debating trying with Stellarmate OS, but could someone confirm I won't have the same issue for this OS if the issue is coming from KStars/EKOS/INDI?
Again, comments etc welcome!

Read More...

Oh dear, not worked I'm afraid @Nou

*************************************************************************
**      New record of system info started at 2022-05-20, 13:22:30      **
**   Total RAM available is 3.7GB and total SWAP available is  99MB.   **
**   Total space on micro SD card (located @/dev/root) is 116.935GB    **
*************************************************************************
2022-05-20 13:22:34 Temp=38.9ºC. RAM 'Used':  8.3%; RAM 'Free': 82.5%; RAM in Shared:  1.4%; RAM in Buffer:  1.1%; RAM in Cache:  8.1%; RAM 'Available': 86.9%. Swap used:   0.0%. µSD has 51.071% free space.
2022-05-20 13:22:36 Temp=37.4ºC. RAM 'Used':  8.5%; RAM 'Free': 81.7%; RAM in Shared:  1.4%; RAM in Buffer:  1.1%; RAM in Cache:  8.6%; RAM 'Available': 86.5%. Swap used:   0.0%. µSD has 51.071% free space.
2022-05-20 13:22:38 Temp=38.4ºC. RAM 'Used':  8.8%; RAM 'Free': 81.2%; RAM in Shared:  1.4%; RAM in Buffer:  1.1%; RAM in Cache:  8.9%; RAM 'Available': 86.5%. Swap used:   0.0%. µSD has 51.071% free space.
2022-05-20 13:22:39 Temp=37.4ºC. RAM 'Used':  8.5%; RAM 'Free': 81.3%; RAM in Shared:  1.4%; RAM in Buffer:  1.1%; RAM in Cache:  9.2%; RAM 'Available': 86.8%. Swap used:   0.0%. µSD has 51.071% free space.

<+ 1 hour of lines cropped out>

2022-05-20 14:48:47 Temp=37.9ºC. RAM 'Used': 12.5%; RAM 'Free':  2.6%; RAM in Shared:  1.1%; RAM in Buffer:  1.6%; RAM in Cache: 83.4%; RAM 'Available': 82.5%. Swap used:   4.8%. µSD has 47.266% free space.
2022-05-20 14:48:48 Temp=37.9ºC. RAM 'Used': 12.6%; RAM 'Free':  2.5%; RAM in Shared:  1.1%; RAM in Buffer:  1.6%; RAM in Cache: 83.5%; RAM 'Available': 82.4%. Swap used:   4.8%. µSD has 47.264% free space.
2022-05-20 14:48:50 Temp=37.4ºC. RAM 'Used': 12.5%; RAM 'Free':  2.5%; RAM in Shared:  1.1%; RAM in Buffer:  1.6%; RAM in Cache: 83.5%; RAM 'Available': 82.4%. Swap used:   4.8%. µSD has 47.264% free space.
2022-05-20 14:48:51 Temp=37.0ºC. RAM 'Used': 12.5%; RAM 'Free':  2.3%; RAM in Shared:  1.1%; RAM in Buffer:  1.6%; RAM in Cache: 83.5%; RAM 'Available': 82.4%. Swap used:   4.8%. µSD has 47.262% free space.
2022-05-20 14:48:53 Temp=37.4ºC. RAM 'Used': 12.5%; RAM 'Free':  2.4%; RAM in Shared:  1.1%; RAM in Buffer:  1.6%; RAM in Cache: 83.5%; RAM 'Available': 82.4%. Swap used:   4.8%. µSD has 47.262% free space.

Swap getting used too. Interestingly, the micro SD card has 30% free before the restart, and then has 50% - I've not deleted any of the simulation *.bin files.

Read More...