I don't remember seeing that issue, so I can't really help. It seems like a build environment issue.
I had problems trying to update my Odroid-N2 from 18.04 to 20.04 when I upgraded (quite a while ago) and eventually started with a fresh install later on. It helped a bit to have the astrometry index files / images / etc. on an attached external SSD.
I also tried 21.04 but that will have to wait for a hardkenel release (IMHO).