Download

Adaptive AI Systems

The aim of this project will be to research & produce an AI system to control ‘bots’ (pawns) in a competitive multiplayer environment. The system will interpret input data from rounds played by the player and other data such as success percentages and heat-maps and modify the actions of the bots with the intent of having less predictable “bot-like” behavior and being more successful against players. I will start the development in a 2d environment with a simple rule-set similar to classic arcade games such as joust and advance the rule-set as the system matures in complexity, allowing for a program of scalable application, with a view to having a final application capable of competent navigation and gameplay in a 3d fps such as quake or counter-strike. (Or UT!)

I should probably point out that I’ve been developing this for the past 8 months and this will be a mish-mash of my progress so far.

Since my last post I’ve been working on the documentation for the project, creating the utility to read in/out ue4’s .analytics files, and working on the framework getting the proof of concept working:

https://youtube.com/watch?v=ojXXgEwG6iM

As shown in the video it’s working quite nicely building a little heatmap for each random minefield set, which is then saved on success out to .analytics files. The utility converts them to .csv and then they can be read back into UE4’s datatable systems which will be useful later on when the map is not random.

Since my last post I’ve been working on the documentation for the project, creating the utility to read in/out ue4’s .analytics files, and working on the framework getting the proof of concept working:

[video]http://youtube.com/watch?v=ojXXgEwG6iM[/video]

As shown in the video it’s working quite nicely building a little heatmap for each random minefield set, which is then saved on success out to .analytics files. The utility converts them to .csv and then they can be read back into UE4’s datatable systems which will be useful later on when the map is not random.

{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}

The data reads back into UE4 as expected and makes for some fancy graphs with the 2,615 metrics the game has recorded so far. These aren’t really useful to the project in its current form as the minefields are randomized, but in the future applications of the system it will be the way the data is analysed and simplified before returning to the system as learn data.

Raw Metrics:
http://i.imgur.com/kOExpzb.png

Simplified using edge-detect:
http://i.imgur.com/0cTkP5K.png

Impression of simplified data to be fed into system:
http://i.imgur.com/XNTjRYR.png

Chart of Damage vs x-Distance:
http://i.imgur.com/ODIC7ci.png

{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}

Since my last post the project has developed quite considerably. The utility to translate ue4’s analytics files now produces batched files based on the map running on the server, and outputs .csv files so that they can be read back into unreal and other programs.

[video]http://youtube.com/watch?v=Og5t6SFY4OQ[/video]

As you can see in the video, the second iteration of the game is now both competitive and multiplayer, with the AI players undertaking a deathmatch in a closed map. I intend to produce a style of game where a team is spawned on a point and has a few seconds to get into positions before the other team is released to attempt to recapture the point.

http://i.imgur.com/Gwxo5ej.png

The Ai system has also become more complex, including a fight/flight system based on certain stats which are read into the bot from a table of stereotypical characteristic sets (e.g. Brave but Stupid, Smart but a Coward, etc.) and also now makes use of UE4’s EQS.

http://i.imgur.com/xJIeGA6.png

{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}

Since yesterday the solution has become much more complex, and works a lot better. Prior to today’s work, the system took in raw vectors and converted them back out in a readable format. It now has the functionality to sum these metrics into a grid to allow for more efficient system. The grid can have a resolution of any power of 2 between 8x8 and 256x256.

There were issues with the system in the way it stood as you can see here:

http://i.imgur.com/x2VBrty.png

As you can see, there is not much colour on the floor. This is an indicator that not many modifiers are being applied to the navmesh. As you can see in the following two images, this effect increases with increased grid resolution.

http://i.imgur.com/wK87GZV.png

http://i.imgur.com/ofl0Be9.png

The issue was caused my the navmesh averaging by neighbors, with many having no modifiers. I solved the problem by writing a script which averages out the grid whilst the data is being aggregated by the utility. This results in a grid which plays much better with ue4’s navmesh system. The resolution of the grid was 32x32, and the ‘sensitivity’ of the modifiers was 60%.

http://i.imgur.com/kV4fiuM.png

The issues with the system as it stands are as follows:

  • Aggregate similar grid squares into larger polygons, more efficient.
  • Allow for multiple dimensions in the z-axis.
  • Remove cells which contain no data for efficiency.

{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}

Added a small script which removes cells which contain no data for efficiency.

Before:
http://i.imgur.com/wK87GZV.png

After:
http://i.imgur.com/NiKyA9N.png

{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}

The grid utility now normalizes the data between a set range and coalesces cells with a set acceptable margin.

http://i.imgur.com/lHjG5y6.png

{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}

Two major advancements have been made since my last post, primarily the fact that the system now produces additional metrics for the AI to include in its decision making, called Point-of-Interest Metrics. (POIs). There are two major types, inspired by Valves source engine, Confrontation POIs and Safety POIs. The former signifies a ‘hotspot’ for kills - places on the map where player deaths occur at a high rate, or places to keep an eye on for likely interactions. These are used at the moment as places to try and avoid whilst planning routes to objectives, or places to pre-aim whilst defending a position. A safety POI signifies places on the map where deaths are low but kill counts are high, or unorthodox positions which have offensive or defensive potential. The AI currently utilises these in its decisions on where to defend from on the point.

Both these additions, and the advancement of the AI system to utilise them, have produced interesting effects on the AI’s gameplay. Defending teams now have a much easier time repelling forces approaching along a known route, thanks to the fact they now distribute their forces based on Confrontation POIs. However, if an attacking bot finds its way around the perception of the defenders, which occurs on a more regular basis thanks to the POI system, the bot could get around the back of the defending forces and attack by surprise, in a similar fashion to how the same course of events might occur in a human team scenario.

The final addition since the last post is that of the ability to play as a human player. At the moment the bots feel very slow to react due to balancing issues as they have been slowed to make their actions easier to debug, but the experience provides an interesting insight into how the system allows bots to perform actions that appear far advanced from their simpler early iterations.

[video]http://youtube.com/watch?v=oL1sj_db670[/video]

{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}

Methodology: Bot vs Bot

The AI will be tested versus AI in the following scenarios:

One Bot vs One Bot (1v1)
Two Bots vs Two Bots (2v2)
Multiple Bots vs Multiple Bots (5v5)

For each of the scenarios, the conditions will be set the following ways:

With the system turned off (off)
With the system on but without any teaching data (on/nt)
With the system on including baked teaching data (on)

Each match will consist 20 rounds. Maximum round time is 60 seconds, which gives a maximum testing time of 20 minutes per match, 60 minutes per scenario, and 3 hours overall. (+5% = 3hrs 10 minutes).

The data recorded during the matches will include video of the game, heatmaps and the win/loss of the teams.

Data: Bot vs Bot

http://i.imgur.com/VDa54S0.png

The results make it obvious that the defending team has advantage in the current balance. From viewing the video it seems that this is due to the fact the defending team is stationary a lot of the time they are being attacked, and thus the attackers have an easier time hitting them. There is also a visible trend during tests utilising the system (onnt/on) that as the match progresses, the defending team has a much easier time picking successful positions to hold from as they learn more about the maps hotspots. This trend is less visible in the (on) tests as the bots already have learnt data about the map from previous games.

Analysis: Bot vs Bot

http://i.imgur.com/t8r2FGt.png http://i.imgur.com/xYelqXn.png

The above images show heatmaps for deaths in the 1v1 off/on tests respectively. It’s obvious from the images that whilst playing with the system on, the heatmap is much more dispersed as the two bots playing learn high-risk and high-reward areas and use them to change their behaviour. The same effect is not as obvious in the 2v2 maps shown below:

http://i.imgur.com/wATZykG.png http://i.imgur.com/YrNXMkD.png

However, more visible is the attacking team being killed closer to their spawn. From watching the recorded video it is obvious that this is due to the defense achieving more kills on the attacking team early in the round, because having teammates allows the AI to move further from the site to a more advantageous defensive position.

http://i.imgur.com/ovH241V.png http://i.imgur.com/ICWKRKa.png

The 5v5 heatmaps show a combination of these trends. Due to the funnelling effect that the top lane has, a lot of the attacking deaths are concentrated in a small portion. This may have prove a difficulty for the system, as this would become an area the AI would avoid attacking through, even though they would have to traverse all the way to the bottom lane to avoid it. This issue has been avoided by setting the navigation cost multipliers of these zones to normalise relative to the rest of the level, which proved a simple but effective solution.

This is super interesting please post more often!

Project Conclusions

Conclusions: Flaws

The issues with the data the system records from UE4 have not been a problem in terms of the recording, but in terms of what data to record. Potentially useful data which was not recorded includes which weapon a kill was performed with, the range of a kill, whether the offensive player was moving/crouched/jumping, how far through the round the kill occurred, the assisting weapon (if relevant), and how many players the defender was spotted by. These would all allow for an AI system to better evaluate its chances against the enemy team and make better decisions.
The reading of the data is flawed in the way it reads the metrics as currently it does not take into account a third axis of movement, and as such only produces a two-dimensional projection of the heatmap. This would obviously be problematic in maps with more z-play than the one currently in place - if the map had overlapping corridors then the system would culminate those metrics and project them only onto the top layer.
The input of the data could be managed better. Currently the data is projected onto the map by the gamemode, where ideally it would be handled by a separate data manager class which would allow for more control over the output as well as allowing for multiple gamemodes to utilise the same data manager.
Interpretation of the data has generally been very good, although my lack of experience in behaviour trees has resulted in a sub-optimal implementation which does not fully utilise the system. Currently the bots do not take into account ‘gain’ metrics in their decisions which may have changed the way they responded to certain situations.

Conclusions: Successes

Aside from the flaws in recording and usage, the data has allowed for a successful system to be produced. According to testing results 80% of people thought it was successful in its aim of producing more challenging, less predictable bots; and 100% thought the bots were more difficult to play against. The observation of the bot vs bot matches showcased some interesting quirks in the system such as the cyclical nature of a bot vs bot game compared to one with human players which will eventually settle at some sort of equilibrium as players discover positions which have an inherent advantage. If ai vs ai is the only learning data, Bots will not stick to these positions as they continue to try and adapt.
Though the game scenario is inherently quite difficult, all of the players in the testing were able to play and comment on the full test scenarios without external assistance. This was largely due to the spectator view available after death, which allowed the player to observe the ai bots more closely.

Conclusions: Future Development & Application

The system was generally very successful and has a lot of potential for further development. Aside from the flaws listed above, the system should be tested with other types of game, such as MOBA or Platformers to test the flexibility and relevance of the metrics used. Of course, different game types will require different recording methods and metric types, which would be an interesting expansion. One could adapt it for a MOBA system by taking factors such as ability timers, movement speeds, item purchasing and team composition to alter the ai’s decisions. Further development of the system would allow for both expansion and streamlining for efficiency, and may also allow for it to be tested in a client-server scenario - similar to a real-life implementation.