scalability

Development Asset Creation
,
21

Interesting Jennifer.

Let’s skip that moment that RC did not have float license.
But can you roughly estimate how much will cost usage of cloud computing for project.

For example 1000images 24Mpx
If this is perfect dataset and not required any experiments with settings, in my PC with i7 and GTX960, 64Gb RAM whole process required about 12-24 hours of work.

Which instance suitable for such RC work ?

All i tried clouds estimate before was cost 3-5 times more in 1-2 years timeline comparing to just a good PC.
May be i wrong?

22

Vladlen wrote:

Interesting Jennifer.

Let’s skip that moment that RC did not have float license.

Use a Steam licence - that floats …

Vladlen wrote:

But can you roughly estimate how much will cost usage of cloud computing for project.

For example 1000images 24Mpx
If this is perfect dataset and not required any experiments with settings, in my PC with i7 and GTX960, 64Gb RAM whole process required about 12-24 hours of work.

It would depend a lot on which i7 - some here are using laptop i7 and other may be using overclocked K series.
but for comparison lets use my my dual xeon (each cpu is ~30% slower than a fast i7 because they only clock to 3ghz) with 8 real cores (8 virtual) which RC uses all 16cores pretty much 100% during image alignment. Doing the default “start” on a scene with ~600images (21mpx) takes about 25 minutes (board room interior, lots of alignment fails because everything is so repetitive). Meshing goes pretty quickly with 2xgtx1080.
On a smaller run, the model said 12 minutes for alignment and 8 for meshing.

Vladlen wrote:

Which instance suitable for such RC work ?

All i tried clouds estimate before was cost 3-5 times more in 1-2 years timeline comparing to just a good PC.
May be i wrong?

https://aws.amazon.com/ec2/instance-types/

if you are keeping costs down, have a look at https://aws.amazon.com/ec2/spot/bid-advisor/
for image load and doing control points, you might get away with a t1.micro ($0.02/hr) or use a m1.small ($0.09/hr)
For image alignment, maybe a c4.4xlarge (16 cores 122gb ram) for $1.50/hr but you can grab a c4.8xlarge(36 core, 60gb) for $3/hr
If you are in a rush, how about a x1.32xlarge (128 cores, 2tb ram) for $15.60/hr?
(these are defined duration prices, you can save ~50% if you can make the spot pricing work for you.)
When meshing, you probably want GPU, so you need to swap to a p2 or g2 instance (p2 for compute only, g2 if you want to spin your models around and have a look). These cost anywhere for $0.70 to $18/hr depending on how much GPU you want.
Keeping in mind these are billed by the minute - if you finish in 10 minutes, the expensive instances may end up costing less overall.

Just always always always remember to shut the instance down when you finish. Quite a few people have just logged off and leave the instance running (idle) and come back to find they had run up quite a large bill. (cheap per hour, but leave it running over a weekend still runs up a bill)

Hope this helps
Jennifer

23

I’ve also been quite aware of off-topic-avoiding until a drop-in pointed aout that those are usually the most interesting - including the one we were off of… :slight_smile: And after all most people will find threads by search machines.
After all it is related in so far as the TO is looking into expanding HW and this could be an alternative.

So: YELL! :lol:

I’ve been trying to find more specifics (on azure) but have no time right now to get through the “this is all so great” and " we are fantastic" animated pages…
I figure it works that you install your SW on some virtual PC on their cloud, right?
Did you try it with RC yet? I would be a bit concerned about their licencing system.

24

Götz Echtenacher wrote:

…
Did you try it with RC yet? I would be a bit concerned about their licencing system.

Hi Gotz!
I’m using a steam licence… We need/have steam infrastructure anyway for the VR stuff and having a floating (though account tied) licence does make life easier. So as long as you had an internet enabled instance, steam and hence RC should be good to roll!
And yes - there are pre-made instances for steam (quite a few people use the cloud servers to host games - turn them on just for your clan practice times (pick a zone handy to your players and they usually have very good network connectivity).

For azure, you use the azure “store” to get your pre-made instance (usually free). On azure, microsoft covers all the various licence/CAL costs (really cheap way to host sql database instances!). Amazon call them Amazon machine images (AMI) and also have a marketplace for them (https://aws.amazon.com/marketplace/ref= … ct?b_k=291).
So you could grab an AMI (https://aws.amazon.com/marketplace/pp/B … duct_title) install it to your instance (few seconds), then load steam and RC.
There is also a very low storage cost ($0.10/gb/month) for non-instance storage (belongs to the account, not the instance) but your files are available on all your active instances.
All the cloud services have a free tier or new user free time to get you started. Have a play, watch the videos - see which one works for you! :smiley:

25

Thanks a lot Jennifer!
That is something for the next rainy afternoon… :smiley:

26

Hey all,
I’ve been away a bit, quickly scanned this - I did some AWS GPU instancing right after the steam release (spot instances are a good way to test and keep costs down!). Long story short, works in a bad pinch, but I got better performance out of my 4Ghz i7 paired with a 980 ti. Any service that would get close to putting proper CPU clock speeds and GPU resources that match your rig aren’t really available except on obscure cloud services and they are prohibitively expensive. Nvidia was going to release a reserved instance of cloud gaming Geforce Now instances (so a 4x core count CPU at above 3.5 Ghz and a 1080) that I was going to play with but that never materialized.

My primary hope is to hear back on when RC is going to support network node clustering or give more details. I’m curious about potential pricing on it as well.

In the interim, I’m hoping to upgrade a 64GB 4 core, 4ghz, 2 x 980 ti machine with a 128GB, 16 core at 4ghz, 2 x 1080 ti. From my napkin math I think I could get as much as a 50 percent increase in my render times, but I don’t know how the efficiency scales on multiple GPUs. My guess is after 12 CPUs your margin of return isn’t great, so I’ll use the remaining 4 CPUs for work in Modo or other modeling programs (probably wrapped in a VM so as to not bump into things). Does anyone’s experiences match up this? Do you think the speed bump would be as high as 50 percent from going from 4x4Ghz to 12x4Ghz and a 35% boost to GPU?

Thanks!

27

Castlenock wrote:

Hey all,
I’ve been away a bit, quickly scanned this - I did some AWS GPU instancing right after the steam release (spot instances are a good way to test and keep costs down!). Long story short, works in a bad pinch, but I got better performance out of my 4Ghz i7 paired with a 980 ti. Any service that would get close to putting proper CPU clock speeds and GPU resources that match your rig aren’t really available except on obscure cloud services and they are prohibitively expensive. Nvidia was going to release a reserved instance of cloud gaming Geforce Now instances (so a 4x core count CPU at above 3.5 Ghz and a 1080) that I was going to play with but that never materialized.

Hi Castlenock,
You are absolutely right that the cloud GPUs are slower than latest gamer/AI GPU boards. (well until you get into the expensive stuff anyway)
But the advantage is you can only use what you need - for example RC only uses the GPU for meshing (as far as I can tell).
So most of the time, most of my $8000 machine is sitting idle, depreciating and sucking power. If you use it a lot, local is good, if you want flexibility for stuff you do time to time, then cloud is better.
And the cloud providers do have very large systems for those rush jobs!

Castlenock wrote:

My primary hope is to hear back on when RC is going to support network node clustering or give more details. I’m curious about potential pricing on it as well.

Cluster processing is really hard (message passing latency for off the shelf hardware is a killer)
But as Wishgranter mentioned yesterday, the AMD EPYC/threadripper (half an EPYC) should help bring the high core count cpu prices down. Still going to be stilling idle most of the time though :frowning:
Just my $0.02
Jennifer

28

keep in mind epyc isn’t really suitable for rc.

you need would need 4 x rc licenses to run one dual 32 core machine.

I don’t think you can even get that going on the steam version.

so you probably need 4 x cli license.

threadripper is as high as you can go before you need multiple licenses. or disable hyperthreading.

29

This is probably a total noob question but what about the bottleneck of the internet connection for cloud processing?
Don’t laugh but in my area it is still notable to get over 10 mbit!
Everything higher (glass fibre) up to 50 is not reliable, we read about streaming-TV-outages all the time in the local newspapers.
So if I imagine having to heave all those images through there, it would probably still be quicker to calculate it locally.
May be different if it only sends the raw data through, but that is still a lot.

30

Götz Echtenacher wrote:

This is probably a total noob question but what about the bottleneck of the internet connection for cloud processing?
Don’t laugh but in my area it is still notable to get over 10 mbit!..
.

Yes, slow links will make the batch load painful… 10mbit is about 1mbyte/sec or ~5-10 secs/picture. Hours for a batch!
Faster speeds even if a bit flakey would be better - you don’t need the consistency people look for for streams - just average bandwidth/hour. Cloud certainly isn’t the answer for everyone… but can save investing in a lot of hardware if you can make use of it. We would never buy a DGX-1… but I can budget a couple hours a month on one :slight_smile:
Have a good weekend all!
Jen

31

…
So most of the time, most of my $8000 machine is sitting idle, depreciating and sucking power. If you use it a lot, local is good, if you want flexibility for stuff you do time to time, then cloud is better.
…

That has always been my thoughts as well, I tested on AWS immediately after the steam release and didn’t do a lot of testing so a lot of this is speculation from my behalf. I do think that the lower CPU clocks really put a damper in performance. My question to you as you have a machine that is close to what I am gunning for on my next iteration is I wonder how the inefficiencies are after CPU core 6 or 7 for RC. I know it’s a different program, but photoscan has good articles on CPU/GPU scaling and show CPUs become rather insignificant after core 8.

I’m hoping to VM Modo and a few other programs to give RC about 8-10 cores at 4Ghz, and up to 8 cores for Modo, Lightroom, Photoshop, etc. via VMWare (as they don’t really need the GPUs) to run a few workflows simultaneously.

Again, you seem to have a machine (and doing VR scenes) and specs that are close to mine - could I PM / e-mail you a few more questions if you’re willing?

Thanks!

32

Castlenock

… I do think that the lower CPU clocks really put a damper in performance. My question to you as you have a machine that is close to what I am gunning for on my next iteration is I wonder how the inefficiencies are after CPU core 6 or 7 for RC. I know it’s a different program, but photoscan has good articles on CPU/GPU scaling and show CPUs become rather insignificant after core 8.
…

Well, I have limited experience, but from what I’ve seen, RC is very good at using the CPUs… during cpu bound phases of the process it seems to keep all 16 cores (real and hyper) pretty well scheduled (15 @ >95% and the last running about 80%).
For my <1000 image runs, the GPU portions of the work finish quite quickly but aren’t pushing the 1080’s that hard (using nvsmi to see usage stats and overall power consumption)
But RC uses the system much better than pix3d which I’m also trialing - that seems to spend quite a lot of time single threaded.
Agisoft is a bit slower than RC but still gets good use of the available resources. (But I’m finding their control point entry system is designed for a very specific workflow and very annoying to use if you aren’t in that scenario).

Serves me right for doing a boardroom as my test case (all the bad things - repetitive objects, repetitive surfaces, shiny, reflective, translucent. Fortunately most of our general requirements are for outdoor scenes… I just want the boardroom to use as a destinations environment.
And sure - feel free to pm or keep a thread running if you think the discussion will interest others.

Jennifer

33

You know what I hate the most, almost more than white ceramic or dark objects? Chairs. I f#$@! hate chairs. I can snap 1000 pictures of a table with chairs tucked underneath and it still comes out looking like poo. I now take a chair out at the very end of a scan, scan it in detail with a good background, and use that as a template to stamp them out in the 3d model at the very end.

Since it involved full knowledge of those UVs, RC, 3D modeling and workflow, it’s still a humongous time sink, but I think I’m getting closer to not having to reinvent the wheel each time and getting faster at the whole business in general. They need to post a showcase forum category where we can share and talk about models from RC - they were hoping to bring something up last year…

34

Castlenock wrote:

…They need to post a showcase forum category where we can share and talk about models from RC - they were hoping to bring something up last year…

We have such a group on Facebook: https://www.facebook.com/groups/CapturingRealityArena/

35

Thanks, I hit the ‘apply’ button or whatever it is in Facebook. Not you guys but ugghh, for my company in general I guess I have to start using Facebook again…

36

Was away for a bit and missed quite some interesting stuff.

Lubenko, can we not get a group liuke that here for non-facebookers?

I think you merged the steam forum with this one, is that right?
At least about 2 weeks back the activity sky-rocketed with many steam related topics… :slight_smile:
Is that also possible for facebook?

37

Götz Echtenacher wrote:

Was away for a bit and missed quite some interesting stuff.

Lubenko, can we not get a group liuke that here for non-facebookers?

I think you merged the steam forum with this one, is that right?
At least about 2 weeks back the activity sky-rocketed with many steam related topics… :slight_smile:
Is that also possible for facebook?

There will be some changes in the forum, we will see…
The Steam forum is another one available - also separate from this one: http://steamcommunity.com/app/489180/discussions/