I found out what it was. BingoBongo was right, that is just a preview issue of the material editor.
When I checked for the first time in the Project Settings, the Custom Depth - Stencil Pass was set to enabled.
After a lot of testing, reverse engineering some HLSL code, creating project in version 4.14 and up, I have found out, that there is a hidden option in the project settings - "Enabled with Stencil, in the same field as “Enabled”. Convenient!
Anyway glad this is sorted out.