These are undersampling artifacts that manifest as fireflies and cause issues during denoising. The same denoiser is used for SkyLight as for GlobalIllumination passes. Your comment regarding inclusion of other lights suggests this is something related to GI rather than related to SkyLight. Can you confirm this is the case?
TAA increases convergence rate as time goes on and will also suppress the fireflies. You should be able to accomplish similar goals by increasing the samples-per-pixel at the cost of interactivity. This may be fine if you are working with a sequencer burnout.