Yea I believe it is a hardware/bios issue. My PC is new with lots of the latest parts (4090, i9-13900K, etc). It has always had a similar shader crash with various unreal games, but that had never affected me using the engine on my own projects until 5.4. I tried tons of things to fix it on the hardware side (changing page file, updated BIOS, RMA’d parts, changed CPU core ratios, etc). And while something things made it work a little better, sometimes. It was never totally solved.
Recently though after reinstalling/reformatting windows I still was getting the crash. Then while researching to learn more about my motherboard I randomly came across this program for it called “Armoury Crate”. Figured I could install it and maybe it could tell me more about my motherboard or make it easier to change some settings. I installed it and went through its utility which had some more updates for my BIOS (newer than the one I got on the ASUS website a day ago). After restarting the problem is now gone.
IDK what it did exactly besides update stuff but now I can run the Lyra project as well as other UE games that would crash due to “out of video memory” during shader compilation!