Yeah, that is the performance problem. You could split up the tick time more by using unreal frontend (or unreal insight since 5.4 I think). Will take some time to learn how to use those profiling tools though. My guess would be that those 100 characters are the problem (something like 30 should generally more or less fine, if you have more than you will likely need to optimize (animations, physics, collision, character movement, number of scene components; all those things need handling/changing from their default settings like only tick animation when in view and use update rate or even animation instancing and remove physics assets from the mesh or keep as simple as possible, or use navmeshwalking in the character movement)). But there should already be some forum posts/info about how to get a large amount of characters to run more or less performance 
And also disable tick in actors where you don’t need it. And be a bit careful about those 4ms blueprint time (that is time the blueprint logic is using and not stuff under the hood). Shipped game should optimize them and may run faster, but if that persists you could think about simplifying that logic or moving it to c++ (profiler tools would show you the exact blueprint functions taking up the time and you can than handle those (simplify or move to c++))