t-iwata’s comment combined with Unreal Engine 4 Tutorials: How to Remove Unwanted Android Permissions using UPL (needed for Oculus) - YouTube solved it for me.
Funny story, turns out the READ_EXTERNAL STORAGE permission is getting re-added by oculus’s own plugin!!!
addPermission android:name="android.permission.READ_EXTERNAL_STORAGE in https://github.com/EpicGames/UnrealEngine/blob/release/Engine/Plugins/Runtime/Oculus/OculusVR/Source/OculusHMD/OculusMobile_APL.xml