Large and Fast working drive for 3-6 users

[landmeterbeuckx]

Hi Everyone,

I am wondering if you have any updated info on the best office network, storage and back up systems? Are Synology NAS on 10gb network still the best way to go? We are needing to invest in either a could based system or an office based storage system for a small but growing firm of 3 full time surveyors using Leica scanners and software. we have a 300/50Mbs internet connection

Are there any threads on best practise for data management and networks/IT setups for laser scanning firms? I have tried searching but no had much luck.

Thanks in advance

I'm solo and run a DS1612xs+ with 5x14TB and an expansion bay DS517 with 5x14TB on a 10Gbit network. I have a second DS1817 unit with currently 3x14TB for clients to download from.

Every project that is on the workstaion and finished is archived to one rar file with no compression. It contains all (raw, processed, extracts,....). This rar file is copied as well on the nas as on a external 6TB drive in double which i just fill up as a second backup. One of those drives is at my mother-in-laws place

I don't send anything to a cloudservice but always have 4 copies and when the nas runs full i will start deleting older rars but keep them on the double external ones of course.

I don't run anything from the NAS, all is handeld on a desktop workstation

Thanks for this LMB it really helps to know how other people solve this data management problem.

It seems you have 1 drive bay free in your DS1612xs+ why is that? This fast NAS is joined with a 5 bay expansion. so total capacity is 10x14TB? Thats quite impressive! Do you use RAID? If so what version?

You have 5 bays free in your DS1817. Is this just to allow for growth of the archive?

Do you use Leica products? You prefer to use Win RAR over native export compression such as in REG360's Archive feature?

I assume you have multiple 6TB external drives at mother in-law's place to replicate the data on your NAS drives or are they not yet full?

Do you have a view on firewalls? I am wondering if a hardware firewall is worth the expense or if a software one will do the job?

Thanks again for sharing your setup and sorry to throw loads of questions at you!

Best regards,

Ben

The free bay is occupied with a spare disk in case a drive fails. So 2 drives aren't used (spares).

The free bays are for expansion if needed but also to just pop one drive in and create extra backup if needed.

No Leica stuff here. I always use Winrar for all zipping stuff.

The 6tb drives no replication, just when a project is finished on the workstation, i create a rar on the nas and copy this on the 2 externals. I just fill them up and have an excel list on which drive is which project.

Hardware firewall, 10Gbe switch, cat 8 cabling.

This whole setup costed about 10k€ 1.5 years ago. Installed and the whole lot by an it firm.

[badam]

I've shared our setup on different topics, and usualy we are the one who stores nothing on their local machines.
https://laserscanningforum.com/forum/vi ... hp?t=18170

Every single register360 project is stored on a NAS 3*7x18TB DC HC 550. it is running on truenas core on a fairly cheap hardware (almost consumer grade ryzen 2600x,64gb ecc udimm,B450 motherboard, LSI9207 sas card in IT mode, intel x540-T2 nic,no gpu).

We do not archive anything, everything is kept on this storage and at blackbaze.

we use zstd compression on the live data. Which has good results on the point cloud dataset we get 1.2-1.3 compression ratio on the whole dataset (~80TB currently), which contains raw RTC360 data, raw blk360 data, e57, laz files. The 10Gbit expansion did not make a huge difference in my opinion if you only works with reg360. Maybe if you have 10 users register always in reg360 then you can use that.

Currently the only time when we hitting the 10G limit is during file copies. when we copy raw rtc360 data to nas we achieve about 5-6 Gbit/s (multithreaded robocopy), during reg360 usage we almost never see higher then 600-700 mbit/s.

But with reg 360 there is a much bigger issue with multiple workstation. Which is the server database nonsense... I've created a topic for this earlier.
https://laserscanningforum.com/forum/vi ... hp?t=17816

firewall will not change lan throughput. if you only wants to increaase lan speed there is no need to get hardware firewall.

As we already had cat5e cable runs for relatively small distances we tried beforehand and it works on 10GBit, so we just replaced few switches and we can use 10GBit.

I think what everones doing with local copies and backups are not feasible, there will be a point in time where you just cannot keep track of everything. like who opened which project on which machine, is she/He modified something or not, and updated the backup? There are automatic waysfor backing up machines fine, but they not always the fastest easiest solution. We are using this NAS approach for almost 2 years witout any issue. it is about 40-50 thousand setup and lot of project. I' am confident that if something is slow on our setup then it would slow on local ssd-s as well..

[Mulattieri]

Hello
@Badam i’m interested on your setup because actually i don’t have a solution for proper storing My point cloud data.
Do you save the same data on each of the 3 nas? Do you use raid for the 7 disk on each nas? The 3 nas are attacched to other with a 10G switch?

[badam]

Hello
@Badam i’m interested on your setup because actually i don’t have a solution for proper storing My point cloud data.
Do you save the same data on each of the 3 nas? Do you use raid for the 7 disk on each nas? The 3 nas are attacched to other with a 10G switch?

Hello no it is a single unit. Similar to what Linus built in one of his video.
It is 3 raidz2 vdev each with 7 18TB drive.

It is about 360 TB raw capacity, usable about 270tb. It depends on your usecase but register 360 is pretty usable with this setup. I don't know about faro or other registration software. The client devices connect through 10 GB but reg360 not using it. Only raw file offloadsand deliverable moving benefits from that.

https://www.google.com/url?sa=t&source= ... GgUambb6tr

[Mulattieri]

Fantastic Badam
I use Reg360 mainly
if in the future i need other disk space can i update the same system? or i have to make a new one?

[badam]

Well 21 hdd is the max in this case one so upgrade can only be done with external jbods. But that require more pcie lanes to work with so that is more of a replace rather than an upgrade. You need server hardware to go bigger than this.

If you work with 1 or 2 rtc360 or similar device. At our own pace you would fill it up in 4 to 6 years and that is seriosly close to a hdd lifetime as well. So probably you will need to buy new hw one way or another and get better faster, cheaper hardware in that time.

[Mulattieri]

You’re Fantastic Badam
So a very good in time system
You also make backup on that system? Or you use additional machine?
I have one P30

[badam]

Zfs is a copy on write filesystem, so backup for file deletion, ransomware, user error is handled by snapshots, we configured 15 minute snapshots to keep for two day, daily snapshots kept for two week, and weekly snapshot kept fot for 4 month. But keep in mind single device is not a backup. We have a smaller system 100tb where we duplicate everything daily except raw, and large files. And that is in a different location.

[Mulattieri]

Fantastic Badam
For the management you use Unraid?

[badam]

We are using truenas core