In the end, although we actually got the augmented reality working, we decided against putting it in our final installation.

This is because of our technical and setting limitations. In order for augmented reality to work, it needs good lighting and the target image needs to be relatively close to the webcam in order for it to sense properly, as well as in good focus. We have a really wide workspace in our setup, with four plots, and because we're projecting the images down, they're not as clear as it would need to be for the webcam to sense them efficiently, since they're in a dark environment as well as fuzzy. Also, it's really risky to do augmented reality using projections as target images, because for example, if someone's hand gets in the way of the projection, then the target image would be lost and the augmented reality would disappear, which is way too inconsistent to be polished.

Regardless, it was really exciting getting this part working and perhaps if we had better equipment, this could come into play! (Although in retrospect, I'm not sure if this part actually fits in with our farming theme...and the game itself is so hectic that no one has time to look up and enjoy the augmented reality, though it'd probably draw a lot of eyes from the crowd)