If you want to use the yourself created mask as a separate file recognised by RC then it should also have the information in Alpha channel. First I thought you wanted to mask the source files directly.
RealityCapture can automatically detect these types of layers:
.geometry (equals to image/folder with no layer extension)
.mask
.labels
.depth
.textureXX
.texture (equals to .texture1 and .texture01)
NOTE: Instead of dots (.) in the folder names, you can also use one of these symbols: _ @ # and !. For example: _texture, @texture, #texture or !texture.
The mask in your case should look as DSG5467.jpg.mask.png