Why does returning to a level which has already been streamed take forever to reload?

Why does returning to a level which has already been streamed take forever to reload? I am experimenting with level streaming. When I walk into the volume which loads the next level it’s pretty instantaneous. When I walk out of that volume, but then return it seems like it takes a full minute to reload everything. What gives?