As the question, I am in attempt to unwrap a handkerchief like the pic below. It’s also applied for the other complex materials like bed sheets or curtains. I have searched through the Internet but so far all I got was only the way to unwrap the unfold or flat sheets by peel mode which may take a hell long time for selecting point-to-point seams. Is there any way to unwrap it properly and effectively?
You should unwrap it in your modelling package, while it is still a flat plane, and deform it as needed(run cloth physics simulation or w.e.).
You could attempt to mark edges in a way like this:
if you are using blender, select edges in edit mode and go under Mesh>Edges>Mark Seam. Try and mark every edge where the fabric folds at a sharp angle, and select the edges in a way so that lines go from the outer edges to the almost center point but do NOT connect at the center point.
Next, select all faces and use Mesh>UVUnwrap>Unwrap.
You should be able to get some UVs similar to my doodle on the right, and by Moving some Vertexes around in the UV editor might be able to get a decent square that can be textured. This is a sloppy fix, but if you absolutely cannot remake the mesh for some reason, this should get you a decent looking texture.
Thanks for replies. Finally I got it done using Seam in 3ds Max though costing really long time. I guess I have to stick wih seam to unwrap all those things.