Yeah, many if not most AAA games use behavior trees. Sometimes coupled with planners like GOAP or STRIPS. To learn more about these concepts I would recommend checking out the AI videos in the GDC Vault.
Here’s their youtube channel: https://www.youtube.com/channel/UC0JB7TSe49lg56u6qH8y_MQ
Also books like the Game AI Pro series and Artificial Intelligence for Games are good places to learn.
Also I’ve used this FSM plugin and its pretty easy to use and understand if you just want to try something simple.
Here’s the link: UFSM State-Machine in Code Plugins - UE Marketplace