Without going into a rant, let me be blunt: Unreal Engine 4’s Reference/Dependency system is horrible.
Redirectors work… sometimes… when they fail, it completely breaks your project in a way that is unrecoverable in some cases.
Moving folders/files work… sometimes… when it fails it leads to duplicated assets and broken references(see above), and if not caught can screw up source control.
Deleting assets works…sometimes… when it fails the problems are pretty much the same as when you move folders/files.
Fixupredirector works… sometimes… when it fails it fails spectacularly.
Bottom Line: The Engine should not actively prevent asset management/cleanup/maintanence. Currently, that is exactly what it does.
Simple solution:
The solution is so obvious that it has probably been overlooked, and it comes in the form of a simple text file.
Keep your redirect system.
On top of this redirect system, have a simple assetregistry.txt file. In this file, keep two sections of data: current state and pending changes. When files are moved/copied/deleted/etc, add an entry to the pending changes. When the engine is shut down (or at start up if that works better), and pending changes are migrated to the current state, redirectors are deleted, any klingon files/folders are deleted, and the pending changes section is cleaned out. Any changes that couldn’t take place for whatever reason remain in the pending changes section and raise a flag when the engine loads.
Add a rereference script/command/menu button that reads loads the file into memory, then checks through every single file in the project and fixes any missing/invalid references.
Benefits:
It’s human readable. If the engine screws it up humans can fix it without losing massive amounts of time.
Migrating/importing between projects with different project file structures becomes easier to manage.
Users regain control of their file system.
This is more a side effect of the complexity of the engine and the assets where countless references are stored.
Bottom line: Plan ahead. If you have a plan of what you want to do, you know what fiolders in what structure you need.
I hardly ever have to move or rename anything in my projects.
Yeah. Probably. All slouches at Epic :rolleyes:
And not because a file is a text file doesnt mean its neccessarily “simple”
To your idea:
It would create a nightmare of inconsistency if the engine shuts down ungracefully (vulgo: crashes).
Part of what you describe is already done in that fashion by the “Fixup redirectors” command. It worked so far very reliably for me.
Obviously you have never debugged a 2 MByte XML textfile with dozens of attributes and beeply nested tags…:rolleyes:
That is a point that I would also like to see to be more convienient, but since it would have to update each and every asset, it would have to load it first.
As any asset might potentially be referenced by any other asset it would mean for 100 assets already 10000 such checks. Since assets can me modified on instance level, you need to make Actor^2 checks… That is not feasible. For that reason the Fixup command is limited to loaded assets. Others just keep relying on the redirector.
I absolutely understand where you are commin from. When I started with UE4, I also tended to evolve my folder structures and way of organizing things along with the project.
Now I create a “default” folder structure that I use in all my projects, so even migrating among my own stuff is always possible without any headaches…
A) I never called them slouches at epic. I love the engine and I think they have done a wonderful job with it, overall.
B) Telling people to plan ahead so that they never have to move files is unrealistic, and here is a simple example:
Team A starts project
Team B starts project
Both teams have different file structures for their completely different projects.
Team A develops a feature that would save weeks of work on Team B’s project.
Team A migrates the content to Team B.
The migration carries the file structure from team A’s project to team B’s, because that is what it does.
Now, according to you, Team B should just suck it up, because they didn’t plan ahead for something they could not have possibley predicted.
C) This check is not something that would need to happen all the time. It is a one off deal when files have been moved. So, even if it took 10 minutes to run, if it only had to be ran once a week or once a month when folder cleanup was done, it would still be worth it.
**And for the record they already keep target lists like this for other parts of the engine.
***Also, I don’t really think you are seeing the simplicity of the file:
Then the planning ahead opportunity is already missed. If it was foreseeable that the teams might share resources, have them start with the same structures.
Im sure at Epic they have a strict set of guidelines on how how to even name assets.
In this case, I would have made both teams use a common structure. If it really that specific for each team, they could use a single folder in their content structure root, called “TeamA”, and “TeamB” for the orher team. When they toss things back and forth, they both have a “TeamA” and a “TeamB” folder in their root.
If TeamB gets stuff from TeamA and they move it around, they should immediately run the “Fixup redirectors” command.
There is no reason that the ENGINE should dictate that though. That is silly. And telling people to just deal with it does not fix or solve anything. I could just see the conversation now:
“I’m sorry GameDeveloper, since you’ve decided to use our engine then all of your teams/projects have to use the same file structure every single time, for every project, no matter how different they are.”
The engine is not telling you anything. It is just demanding that you stick with your decissions. Thus the aforementioned necessity of planning ahead.
But maybe its just my lack of imagination. Give me two examples of “incompatible” folder structures and lets see how I would handle it.
Maybe Im missing something, or you might get an inspiration
Ok, real world example from two of my projects. I had two projects, with two different directory structures. They were different because the projects were started on different releases of the engine, when different capabilities were introduced. Both projects were upgraded to the latest engine version, which supports components. I had a component in one project that would work well with the other. I migrated the component over, which grabs all the dependencies including the folders that they reside in, and moves them over to the new project. Now I have several extra folders in my project, each containing only a single item that I had been keeping stored elsewhere. (structs/enums/etc) So, I move the files, save, close, open, fix up redirectors, and commit to svn. However, unreal had not deleted all of the directories/files, so it duplicated them in the SVN unbeknownst to me. On the next SVN update. It pulls down all the duplicated files, which of course breaks all the friggin references and forces you to repeat all those steps again, possibly with the same broken outcome.
This here is the problem. The future is not forseeable. How many new features and such has Epic added just in the last year? How many new asset/file types. Hindsight may be 20/20 but foresight is more like 20/80. Requiring that people predict the engine’s changes, and thus whatever structural changes that might be required, in advance is ludicrous.
Without seeing the actual structure its a bit fuzzy to imagine…
That sounds like an initial problem due to how you group assets that belong together.
When you have an asset, say a lamp, do you have then a lamp folder that contains all subassets, like texture, mesh, blueprint, etc?
Or do you use one folder substructure for all meshes, another for all textures, etc?
Typically I use one folder for all Static Meshes. For models that will be animated, I have the model and associated files bundled. Single directories for textures and such.
You seem to be missing the point though. The point is not whether or not my file structure is bad. My file structure works for me. Could it be better? Sure. The point is that, as I learn, I should be able to fix it as I go without having to worry that my entire project is going to get corrupted by the way the engine handles the linking/referencing. No amount of preplanning is going to change that simple fact.
I’m sure underneath it’s a really complicated issue to solve, but I totally agree that it’s extremely restrictive and finicky the way it’s implemented right now.
Coming from the film side of things I’m used to expecting things to break when I move assets on the disk file system, but within an editing project the folder structure that’s represented in the project itself is abstracted from where the files’ are actually being stored. This gets around a lot of the fragility and rigidity that comes from trying to maintain a 1:1 relationship between the physical file structure and how you want to organize things representationally within the project.
Collections get us partway there in unreal, but they are missing the hierarchical nature of folders so it’s not a full on replacement.
I think epic should look at decoupling the folder views inside a project from the on-disk asset folder structure entirely. It’s how nearly all Film, Editing and Animation packages handle this problem and it is a whole lot more flexible.
Well, I agree in the planning ahead thing, only sometimes timeframes are crazy enough so you don’t really have time to plan before prototyping.
I usually end up with what i call the “Undead Starter Content” directory.
I delete actors referencing it, I delete meshes, materials, textures (in that order), then save, exit editor, manually delete directories in project, reopen editor and BAM, here’s the starter content dir again.
Then i redo everything again, usually on the 4-5th try of doing the same the engine is done and actually stops recreating it.
Only then for whatever reason i delete the intermediate and saved folders (only one ssd on this machine i’m using now) to make some space on the hd and after opening here’s the undead rising!
It would be funny if only I was always sure of project integrity. Problem is that sometimes you lose everything, especially if you’re as stupid as me and don’t do backups. Ever.
I agree with Hyperloop though, content structure should be decoupled from actual filesystem.
Unreal is in this middle ground in which it manages everything but only up to a certain point. I’d rather do it all on my own or be completely managed. Second option would be most benefical to me.
What would be nice if there would be a convinient way of maintaining the references for the raw assets, like the FBX or TGA files. So that they can be moved but still “reimport” would work.
Currently this has to be changed manually, which can be tedious.
Then again: relocating source raw assets, once introduced to the engine, should be the exception…
Yep. Basically the issue it there are two different reasons for folder structures: one is so the engine can keep references to assets, and the other is so the human user can organize things in whatever fashion is most effective at any given time. If you keep these two requirements too tightly bound you end up with the worst of both worlds.
And this is the ultimate reasoning behind the request. A flat file reference system could act as a mediator between the two requirements. It would require a little more work on the user’s end, but it would give us a lot more flexibility and allow us to recover from disaster. Bonus points if they added a command line flag like -RELOADALLREFERENCES that would read the file as the editor starts, and before all the assets are loaded into memory, and allow the user to correct broken references at that point.
