Could also be the garbage collector getting into action and pausing the game to clean up removed objects.
You could try to use a pooling solution, so instead of always creating/removing the map building blocks, cache them in a pool and reuse them. The one-time creation of the building blocks could happen on demand or at level start - say 10-100 blocks of every type created up front. Thus loading will happen at level start, not while the level is running.
Maybe this will also help, to see if the GC is really the culprit?