Try to reduce USB bandwidth and don't expect a higher FPS at full size. The only way to get reasonable fps for planetary work on a RP4 is with ROI about 640x480. With this size, and 40 USB transfer I was able to capture 100fps videos and I've not experimented any freeze.
ah ok, good to know that RP4 can provide a bit more fps, though 100fps is rather low?
I get >500fps with smaller ROI in 8bit with the ASI462 on my laptop (not rp4)