I solved this ages ago for myself so I might as well post the answer. What I did in the end which worked a treat was create a project for my cuda code inside of the main UE4 solution I was working in. This project used the cuda compiler etc and compiled the libraries to some place relevant in my greater UE4 solution. I compiled the libraries and added them to my version control for convenience.
Then if someone else cloned my code, IF, they wanted to edit the cuda code all they needed to do was add the project.