Like suggested, increase your memory streaming pool. I had the same problem on my gtx 980 4gb. I increased it to 4000 like that : r.streaming.poolsize 4000 (no more mip mapping like crazy)
Never had a problem after that and I usually have 4k textures and 2k lightmaps sometimes!