The golden rule is always profiling your scene. Many early optimizations are actually worst than you might have thought.