Hm. That seems to be a very low power machine. Things you might want to try:
- enable zram for swap. it'll be faster than swapping to sdard/emmc
- do all the compiling on the sdcard. It won't be fast, but at least it's got lots of storage
- compile only the necessary drivers
- cross-compile on a much faster amd64 machine

At the moment, I'm compiling indi and kstars on a Radxa Zero, with all the source on a microsd...