Working storage drives and Large Datasets

3DVEM, Leica Cyclone, FARO Scene, LFM Suite, Autodesk ReCAP, Riegl RiScan Pro, Trimble RealWorks, Etc.
User avatar
silviaks90
I have made 40-50 posts
I have made 40-50 posts
Posts: 40
Joined: Fri Jan 16, 2015 8:03 pm
Full Name: Kyle Pickering
Company Details: ONSITE3D
Company Position Title: Co-Founder
Country: Calgary Canada
Linkedin Profile: Yes

Re: Working storage drives and Large Datasets

Post by silviaks90 » Thu May 11, 2017 5:05 pm

Rikore wrote:Suggest trying a 4 TB external, they are not that much.
Gave this a try on a separate machine just to test using a WD Passport Ultra, in this specific example I was indexing the .e57's in recap. Probably not a viable solution for us moving forward, unless it was just this specific test but it took 19 hours to index 120GB file running USB 3.0. Pretty good machine as well, i7, 1070 card, 32GB ram, m.2 SSD for operating drive, etc. But I do agree they are cheap and a flexible option. Thanks for the suggestion, you guys clearly have some large files!

imcid953 wrote:I'm not working with Cyclone (using Scene), but our computer has 2 1-TB SSDs set to a raid 0.

Bill
Have you benchmarked this setup against running a 1TB m.2 Samsung 950/960 Pro?
dhirota wrote:I posted this on February 15, 2017

10TB, 12TB, 14TB HELIUM FILLED HDD & INFINIBAND
viewtopic.php?f=123&t=11497

From the last 3 months of use, the two Helium 10TB Seagate SAS Raid 6 systems (100TB each) we built are almost as fast as SATA SSD Raid XX systems, but significantly cheaper, especially if you need to be in the 100TB and larger storage capacity.
Dennis this is awesome and I can't wait to have these kinds of storage issues in the near future, something to start thinking about as a long term setup :ugeek:
Phil Marsh wrote: Cant you just have Cyclone and database installed on your SSD but have your working directory on an additional internal hard drive. This is where the 4x times space things comes in for unifing, etc.
Thanks for the suggestion Phil, I tried doing this exact workflow with a number of configurations as we have a few different drives to choose from. The most promising was like you said running cyclone and database from the 1TB SSD completely empty and setting the working drive to an open 4TB WD Black. However I still got the same space errors or what I believe to be space errors (I never got a firm answer from Leica on that). It was only until I moved the database and all temporary and working folders over to the open 4TB drive that it worked properly. I'm not sure if it was situational on the files, the cyclone install, the computer running it. Lots of factors, I intend to do some further testing to see if I can rectify the issue to use the decent horsepower we've put in place to handle these tasks efficiently. Ideally that would be the best route though.
Kyle Pickering
Co-Founder and Designer, ONSITE3D

dhirota
V.I.P Member
V.I.P Member
Posts: 271
Joined: Mon Nov 02, 2009 12:18 am
Full Name: Dennis Hirota
Company Details: Sam O. Hirota Inc.
Company Position Title: President
Country: USA
Linkedin Profile: Yes
Location: Hawaii, USA

Re: Working storage drives and Large Datasets

Post by dhirota » Thu May 11, 2017 6:13 pm

Kyle

I have not bench marked application/computations on any of our storage systems yet since I am currently interested in access speed/timing of large files. We can store 10,000 36Mp images from our Nikon D810 on a SD card. If you have 3 Nikons on a Riegl mobile, it will take a while to move it around your IT storage system.

I have a 500GB NVMe Samsung 950 as a boot/compute drive on my DELL 7710. W10 tells me that moving large files with the Xeon on the laptop will copy at a rate of 1 to 2 Gb/second, which I think is about 2X to 4X faster than the 2TB Samsung on the same laptop depending on what you are moving.
Dennis Hirota, PhD, PE, LPLS
dennishirota@gmail.com

GrasstecGeomatics
I have made 10-20 posts
I have made 10-20 posts
Posts: 18
Joined: Mon Feb 01, 2016 1:00 pm
Full Name: Joe Nally
Company Details: Grasstec Geomatics
Company Position Title: Lead Surveyor
Country: Ireland
Linkedin Profile: Yes
Location: Ireland
Contact:

Re: Working storage drives and Large Datasets

Post by GrasstecGeomatics » Wed May 17, 2017 12:34 pm

Just thought I would add what I am currently working with

Before:
2 x 1 TB Samsung SSD's running in Raid 0 (Read Write Speed of about 1 GB/s)
Had our Cyclone Cache running on slower 3.5 Inch HDD

Bigger projects really started to slow us down. So we decided to move to something else

After:
2 x 1.2 TB Intel 750 Series NVME PCIE SSD's in Raid 0 (Read Speed of 5.2 GB/s and Write Speed of 2.4 GB/s)
Cyclone Cache is now running on our old 1TB SSD's running in Raid 0

The difference is absolute night and day. Watching Cyclone use all of that power in Task Manager is quite enjoyable.
We are doing a project as I write this with around 1600 Scans from a P40 and the New storage setup is handling it with ease.

The new intel drives were not cheap, around 550GBP each, but well worth it in my opinion.

Our Export to .E57 time has improved by at least 60%

Granted we only have 2.18TB of storage on the new drives we move each project on to the NAS drives once we have finished. We have 4 Synology NAS Drives with about 40TB of usable storage all with redundancy. We are now looking down the route of installing NIC cards into our Machines to increase our network transfer speeds, which will help for transferring large projects.

Post Reply

Return to “Registration Software”

Who is online

Users browsing this forum: No registered users and 1 guest