Update: My current theory is that there is firewall setting or configuration getting in the way.

With the mini-pc connected to my home network and the firewall active, VNCViewer works fine from either my laptop or desktop.
With the mini-pc running its hotspot and the firewall active, I cannot connect reliably. I have ports 5900, 5901 and 5902 open for tcp. Some times VNCViewer will connect and show the ubuntu desktop, but there is no mouse or keyboard control from the viewer.

So I can work around by disabling the firewall when I go out to remote image.