I do have the setup you describe working, and it is my go-to method when any networking issues arise (which is frequently).
So, first-off, there's an existence proof that what you want can be done.
Not sure what your issue is, but try these, with a monitor/keyboard attached to the RPi and your mac and RPi directly connected,
and WiFi off on both machines, and no other ethernet connections in place.
1- from your mac, make sure you can ping your RPi's IP address (e.g. i a terminal type the address--e.g. ping 192.168.86.100).
2- from your RPi, see if you can ping your Mac.
If you can do both those, and given your other comments, then you should be able to vnc from your mac to your RPi using the IP address your ping'd from your mac. Can You?
3- make sure your subnet mask is set right. E.g. on my mac, in network preferences, with the USB section selected, have the following on the right side:
Manually, an IP address I chose (e.g. 192.168.200.200), a subnet mask for the last two digits, eg 255.255.0.0
On your mac, when you plug in the direct cable to your RPi, you should see the USB connection turn green in network preferences
On your RPi, when you plug in the direct cable, if you type ifconfig in a terminal, you should see the eth0 section populated with an IP address after a little while.
If/when you get things working, so that VNC from the Mac can access the RPi, then try enabling the WiFi and you should still be able to have your direct connection working.
However, I've seen times when things got confused, so when I debug, I try to remove the other network.
Hope these techniques bring you closer to a solution,
Hy