Hello,
What says is a very good point to start building a tree to manage an AI with different behaviors. However, I think that what you want is how to obtain the position for the AI to move where they don’t see the player.
If I am not wrong what you want is to predict the position of the player. To do that you should store some data in order to be able to do a good prediction. For example, I would use the last time the AI saw the player, the last position, orientations and probably also the speed. With this data you can assign a probability (the probability of the player to be there) to every position in the map at a specific time. Then, once you have calculated the most probable position you move the AI to that position and if the player is not there, you update the probability map and try again. To implement this sort of probability map I use influence maps ().
Take into account that, as the time passes the probabilities decrease. You will arrive at a point when all the probabilities will be low and almost equal. For this reason, it will be impossible to predict where is the player. In that moment the prediction algorithm turns into a search algorithm (wonder). This algorithm should be very simple: you must assign a low probability at the positions that you have seen recently. As the time passes, this probability increase because you are no longer certain if the player is there or not. From here, the mechanic is the same as before, you move the AI to the most probable position and update the positions.
One note. If you have multiple agents, instead of sending the AI to the most probable position, you move them to the top 5 or top 10 for example. You should also, use all the gathered information together, as if it was only one agent.
I hope its clear, if don’t, ask.