What Hardware Makes the Biggest Difference in Processing

To chat about anything else.
JWFARO
I have made 40-50 posts
I have made 40-50 posts
Posts: 45
Joined: Thu Feb 06, 2014 10:58 am
Full Name: Julian Weisser
Company Details: FARO Europe
Company Position Title: 3D Software Developer
Country: Germany
Linkedin Profile: No

Re: What Hardware Makes the Biggest Difference in Processing

Post by JWFARO » Thu Sep 15, 2016 11:46 am

A quick experiment with SCENE writing 123 million points (reading/writing to same hard disk) showed a speed up factor of 6 between XYZ (ASCII format) and e57 (binary format):

Format, file size, duration, velocity
E57: 1814mb, 3min, 10mb/s
XYZ: 5734mb, 18min, 5mb/s
JoshK wrote:have you experimented with this much at all?
Yes, I did a comparison between some different exchange formats recently but I must say that in my personal opinion the choice is quite disappointing. I guess it is a bit of trial and error to find a suitable exchange format depending on the software and the workflow that you are using but as long as there is a binary choice it is probably always better than any ASCII format.

User avatar
berdindc
V.I.P Member
V.I.P Member
Posts: 198
Joined: Mon Feb 27, 2012 8:24 pm
Full Name: Dan
Company Details: Berding
Company Position Title: Project Engineer
Country: USA
Linkedin Profile: Yes
Location: Ohio

Re: What Hardware Makes the Biggest Difference in Processing

Post by berdindc » Thu Apr 20, 2017 5:56 pm

JWFARO wrote:A quick experiment with SCENE writing 123 million points (reading/writing to same hard disk) showed a speed up factor of 6 between XYZ (ASCII format) and e57 (binary format):

Format, file size, duration, velocity
E57: 1814mb, 3min, 10mb/s
XYZ: 5734mb, 18min, 5mb/s
JoshK wrote:have you experimented with this much at all?
Yes, I did a comparison between some different exchange formats recently but I must say that in my personal opinion the choice is quite disappointing. I guess it is a bit of trial and error to find a suitable exchange format depending on the software and the workflow that you are using but as long as there is a binary choice it is probably always better than any ASCII format.
What is the fastest route from Scene 5.2 to RECAP? Right now I bring in the registered FLS files. Would it be faster/better to upgrade Scene and export e57?

Christophe21
I have made 10-20 posts
I have made 10-20 posts
Posts: 19
Joined: Tue Nov 08, 2016 10:12 am
Full Name: Christophe
Company Details: AB VD
Company Position Title: Technician
Country: France
Linkedin Profile: No

Re: What Hardware Makes the Biggest Difference in Processing

Post by Christophe21 » Fri Apr 21, 2017 11:54 am

So lets say WOW !

This is real nice topic.

I totally agree with the way you describe overrated NVIDIA Quadro cards, I have one at work and recently change for a GFX1070 8Gb at home and it make me regret my all RADEON 7870 with only 2Gb assuming AMD Opengl Driver are more efficient than GFX (compared in Sketchup on "big project" and surprisingly in really smaller ones !).
So you're right, the more video RAM the better (also our specific software won't really use the GPU capabilities ... so)
AMD card with tons of VRAM should be real cost killer effectivelly.

SSD are discriminant, yes, please have a check to NVMe drive assuming they have a 3 time more speed than SSD (and unfortunately also a 3 time more price now) You'll need a motherboard that support PCIe connector and NVMe draive compatibility (that in my memory is the case for X99 chipset)

Thanks for your feedback !

User avatar
pburrows145
V.I.P Member
V.I.P Member
Posts: 1371
Joined: Wed Sep 12, 2007 9:16 am
Full Name: Paul Burrows
Company Details: Leica Geosystems UK Limited
Company Position Title: EMEA Business Manager HDS
Country: UK
Skype Name: pburrows145
Linkedin Profile: Yes
Location: UK

Re: What Hardware Makes the Biggest Difference in Processing

Post by pburrows145 » Fri Apr 21, 2017 1:54 pm


ananda
I have made 10-20 posts
I have made 10-20 posts
Posts: 14
Joined: Fri Apr 21, 2017 1:28 pm
Full Name: Ananda Fowler
Company Details: GEOHAWK GmbH
Company Position Title: Owner
Country: Austria
Linkedin Profile: Yes

Re: What Hardware Makes the Biggest Difference in Processing

Post by ananda » Tue Apr 25, 2017 6:31 pm

Speaking of NVMe, I'm cooking with a new Samsung 1TB 960 NVMe at 2.5GB/s and let me say it is *worth every penny*.

If you're upgrading I'd recommend the following:
  • Samsung NVMe SSD (for processing. Recommend 1TB or more, if possible)
  • Secondary Drive (for 'Cold storage' of projects [can be platter-based, or just use a NAS and auto-archive at night])
  • nVidia GTX1080 or 1070 graphics (or something newer, but these are 3x better than the previous generation)
  • Minimum of 32GB RAM. 64GB is better. You can always upgrade if you need to... so just make sure you use the largest capacity sticks possible when you buy (e.g. 1 stick of 32GB is better than 4 sticks of 8GB, which fill up most of your open slots)
  • Intel i7 whatever the latest is with the most cores at the time of purchase...
  • Thunderbolt 3 Port(s) [blazing speeds. Faster than SSD's, so you can technically use an external SSD via Thunderbolt]
  • Windows 10 (few like it, but it provides the advanced hardware support to make use of this cutting-edge tech)

JimQVargo
I have made <0 posts
I have made <0 posts
Posts: 1
Joined: Fri Jul 22, 2016 3:33 pm
Full Name: James Quirin
Company Details: Vargo Associates
Company Position Title: Reality Capture Manager
Country: United States
Linkedin Profile: No
Location: NJ

Re: What Hardware Makes the Biggest Difference in Processing

Post by JimQVargo » Tue Apr 25, 2017 8:03 pm

Hello Everyone,

I've been a longtime reader, but I wanted to add some additional information when looking into purchasing NVMe SSD's that may be helpful to some of you. I've been building computers for a very long time, and before purchasing a NVMe SSD as an upgrade, I recommend you analyze your system. I wouldn't want you to spend the money on one without getting the full performance they can provide!

Our programs really love to put our processor, video cards, and hard drives to the test. The NVMe SSDs essentially use PCI Express lanes to communicate with your system. This eliminates the bottle neck that occurs with SATA connections on our standard SSDs.

Issues can occur when you add your new NVMe drive to your system because your computer is now sharing the PCI Express lanes with your video card and drive. This can provide an inadequate amount of bandwidth for both devices, and prevent them from reaching their peak performance. To address this issue, we need to look at each motherboard. With a little reading, you can see what lanes your motherboard makes available to each PCI Express slot when you have cards plugged into them. There will be an optimal configuration to plug your video cards into when running a NVMe drive.

To get the full performance of a m.2 chip slot (NVMe SSD), you'll want to have a motherboard that can provide all 16 PCI express lanes to your video card(s) and NVMe SSD.

You'll also want to make sure they have plenty of cooling. Many of the NVMe SSDs begin to throttle under increased temperatures.

My current processing computer:

6 core i7 overclocked to 5.3 GHz
128 GB of Memory
2 x Titan X Pascals in SLI
m.2 950 PRO NVMe for processing
Raid 0 OS SSD Drives
13 TB of Storage for projects not currently being processed, but still being worked on or manipulated

I do most of my data extraction on my workstation.

I'd be happy to elaborate more, or if you would like some additional information, feel free to message me directly. I don't want you to stay away from this great technology because it is amazing. More than likely, even if you don't look into the lane limitations of your system, you'll still see a nice performance increase. You just may not get the full potential.

Jim

dhirota
V.I.P Member
V.I.P Member
Posts: 347
Joined: Sun Nov 01, 2009 11:18 pm
Full Name: Dennis Hirota
Company Details: Sam O. Hirota Inc.
Company Position Title: President
Country: USA
Linkedin Profile: Yes
Location: Hawaii, USA

Re: What Hardware Makes the Biggest Difference in Processing

Post by dhirota » Wed Apr 26, 2017 12:04 am

Ananda

Most friends think alike, especially if you are paying the bills.

I have on my DELL 7710 laptop
1. Xeon E3-1535M v5@2.90GHz
2. 64GB RAM
3. Quadro M5000M video
4. W10 Pro build 16179.rs (preview build)
5. Samsung NvMe 0.5TB boot/processing C-DRIVE
6. Samsung SSD 2TB storage E-DRIVE
7. 4K 17 inch display
8. Intel 1GbE network port

My office workstation is:
1. dual Xeon E5-2637v4@3.5GHz
2. 0.25TB/1.5TB RAM
3. GTX 1080 video
4. W10 Pro
5. Samsung 1TB boot/processing C-DRIVE (not enough $ to upgrade to NVMe yet)
6. 60TB RAID 6 SAS with controller E-DRIVE
7. 4K 75 inch Samsung display
8. dual Intel 10GbE ports to internal 10GbE network (Jason: we still have not converted to Infiniband).

The SAS raid at 12gbs gives us approximately twice the throughput as our SATA HDD raid systems at 6gbs. Copying and moving large files could reduce wait time as part of the processing time when copying workstation to server. When copying to another workstation folder it can be almost as fast as NVMe space copying, but much cheaper for large storage.
Dennis Hirota, PhD, PE, LPLS
dennishirota@gmail.com

ananda
I have made 10-20 posts
I have made 10-20 posts
Posts: 14
Joined: Fri Apr 21, 2017 1:28 pm
Full Name: Ananda Fowler
Company Details: GEOHAWK GmbH
Company Position Title: Owner
Country: Austria
Linkedin Profile: Yes

Re: What Hardware Makes the Biggest Difference in Processing

Post by ananda » Wed Apr 26, 2017 12:46 am

Dennis, you're right, friends who pay the bills do think alike.
If there were thumbs up on here, you'd have mine.

Now that you've spec'd your rig, I have to chuckle and spec mine too...
It isn't as fancy, and I went for an outlet deal to save a bit... so I'll be happy in 2nd place on this one, as I'm not in the driver seat like Sam O. Hirota Inc.

DELL 7510 (total price as spec'd = $1600)
  1. Core i7 6920HQ
  2. 32GB RAM (1 dimm)
  3. QUADRO M2000M Video
  4. W10PRO
  5. Samsung 960 1TB NVMe m.2 SSD
  6. 1TB HDD (not upgraded)
  7. UHD 15" Display
  8. Thunderbolt + USB C

dhirota
V.I.P Member
V.I.P Member
Posts: 347
Joined: Sun Nov 01, 2009 11:18 pm
Full Name: Dennis Hirota
Company Details: Sam O. Hirota Inc.
Company Position Title: President
Country: USA
Linkedin Profile: Yes
Location: Hawaii, USA

Re: What Hardware Makes the Biggest Difference in Processing

Post by dhirota » Wed Apr 26, 2017 2:04 am

Ananda

Your's was $1000 less than mine and will process normal stuff at the same speed. As you get older, you need a larger screen and for others to look over your shoulder.

I forgot to mention the Thunderbolt and USB-C on the DELL 7710 which can be helpful if you need additional storage. I have a 4TB Thunderbolt SSD or a bunch of 1TB SSD if I need to carry additional storage around.

For those people that are in the Apple mode. I just resurrected my MacPro(late 2013) that I turned off last year since Apple did not appear it was going to upgrade it. We had requests to help with the loading of our point clouds into ArchiCAD 20, which we accomplished on my dual Xeon workstation described above. Because of many posts on ArchiCAD warning of the 4GB load limit, we kept the partition sizes to the 4GB limit, not experimenting with larger sizes. We then decided to to verify the loads with an Apple OS version of ArchiCAD 20. Worked fine.

MacPro(late 2013)6,1
1. Xeon 4-core @3.7GHz
2. 64GB ECC RAM
3. Dual AMD FirePro D500 3GB GDDR5 VRAM each
4. MacOs Sierra 10.12.4
5. 512GB PCIe flash
6. 6 each Thunderbolt ports and 4 each USB3 ports for storage
7. 4K 75 inch display or 24 inch Samsung when traveling
8. Dual 1GbE and Wifi

I also found out that OWC will upgrade your PCIe FLASH to 4TB and your RAM to 128GB. I decided not to upgrade the hardware, but Apple upgraded by MacOs from Yosemite 10.10.5 to Sierra 10.12.4 for free with a few bumps along the road (which took me to tier2 apple support, and then to Apple Enterprise support, where you always want to be from the beginning) to complete it.

For some the hardware upgrade might be worth it.
Dennis Hirota, PhD, PE, LPLS
dennishirota@gmail.com

ananda
I have made 10-20 posts
I have made 10-20 posts
Posts: 14
Joined: Fri Apr 21, 2017 1:28 pm
Full Name: Ananda Fowler
Company Details: GEOHAWK GmbH
Company Position Title: Owner
Country: Austria
Linkedin Profile: Yes

Re: What Hardware Makes the Biggest Difference in Processing

Post by ananda » Wed Apr 26, 2017 11:21 am

Good point Jim.
I had to install Windows 10 three times (and remove/install the m2 SSD) to finally achieve the full speed of the m.2 SSD.
At first it was giving me 1x4 speed, and after messing about, it finally arrived at the full 3x4 of my mobo...

Additional note for everyone who goes this route: make sure you install the Samsung Magician software and update the SSD Firmware with it. Firmware makes a big difference in performance and longevity of the drive.
JimQVargo wrote: The NVMe SSDs essentially use PCI Express lanes to communicate with your system. This eliminates the bottle neck that occurs with SATA connections on our standard SSDs.

To get the full performance of a m.2 chip slot (NVMe SSD), you'll want to have a motherboard that can provide all 16 PCI express lanes to your video card(s) and NVMe SSD.

Return to “General Chat”