Are you suggesting a large data acquisition campaign to build a ML system that would train on image and use the other sensor as a reference ?
If so this is quite interesting I am going to setup a 224mc based allsky + aag cloudwatcher in the upcoming months.