10TB, 12TB, 14TB HELIUM FILLED HDD & INFINIBAND

Post any marterial related the future technology of laser scanning.
dhirota
V.I.P Member
V.I.P Member
Posts: 391
Joined: Sun Nov 01, 2009 11:18 pm
Full Name: Dennis Hirota
Company Details: Sam O. Hirota Inc.
Company Position Title: President
Country: USA
Linkedin Profile: Yes
Location: Hawaii, USA

10TB, 12TB, 14TB HELIUM FILLED HDD & INFINIBAND

Post by dhirota » Thu Feb 16, 2017 1:38 am

Everyone has been speaking about larger SSDs in the last year. I do not know if this subject is FUTURE TECHNOLOGY or not, but many folks think rotating HDD will be gone soon. We have looked at creating our own internal cloud for many reasons and last year decided to extend what we had into faster, larger internal storage. We decided on a pair of INTEL servers, following the prototype workstation built last year using an INTEL server motherboard (dual Xeon, .25TB/1.5TB RAM, dual 10GbE, 64TB SAS RAID6 8TB Seagate HDD, 2-GTX1080 GPU).

Western Digital has announced 10TB helium HDD two years ago, and Seagate announced the helium filled 10TB HDD last year. We placed our order in October, 2016 with a major IT distributor for the Seagate 10TB ST10000NM0206. We finally received our order of a box of 20 HDD after 5 delays, yesterday. In December, 2016 while we were waiting for our 10TB to arrive, WDigital and Seagate announced 12TB HDD for this year and 14TB for 2018.

The Seagate 10TB ST10000NM0206 HDD are SAS vs. SATA which gives us 12Gbs on the Broadcom/AVAGO 9361-16i controllers that we are using. This gives us 100TB in Raid6 with 12 HDD on each server with dual 10GbE on each server. In the tests today, Robocopy is copying a 15TB block of projects at a speed of approximately 275MB/second over our internal 10GbE ethernet network at about 2-3Gbs total from 2 ports to another server with two ports.

We decided not to fill both servers with disks and wait until prices on the 10TB reduced due to the introduction of the 12 or 14TB HDD.
Dennis Hirota, PhD, PE, LPLS
dennishirota@gmail.com

User avatar
Jason Warren
Administrator
Administrator
Posts: 2862
Joined: Thu Aug 16, 2007 9:21 am
Full Name: Jason Warren
Company Details: Laser Scanning Forum Ltd
Company Position Title: Co-Founder
Country: UK
Skype Name: jason_warren
Linkedin Profile: Yes
Location: Retford, UK

10TB, 12TB, 14TB HELIUM FILLED HDD & INFINIBAND

Post by Jason Warren » Thu Feb 16, 2017 8:12 am

Hi Dennis,
Impressive, but would like to know why you opted for 10Gbe and not Infiniband?
Infiniband.PNG
You do not have the required permissions to view the files attached to this post.
Jason Warren

#ThePowerofLSF

KDersch
I have made 60-70 posts
I have made 60-70 posts
Posts: 60
Joined: Wed Jun 01, 2016 11:54 am
Full Name: Kyle Dersch
Company Details: Matix Technologies
Company Position Title: Engineer
Country: USA
Skype Name: kddersch@matrixti.com
Linkedin Profile: Yes

10TB, 12TB, 14TB HELIUM FILLED HDD & INFINIBAND

Post by KDersch » Thu Feb 16, 2017 11:58 am

Subscribed...

SAttaya
V.I.P Member
V.I.P Member
Posts: 268
Joined: Mon Mar 23, 2009 4:04 pm
Full Name: Steve Attaya
Company Details: Geocent
Company Position Title: Sr. Software Analyst [RemoteSensingEngr]
Country: USA
Linkedin Profile: Yes
Location: New Orleans, LA

10TB, 12TB, 14TB HELIUM FILLED HDD & INFINIBAND

Post by SAttaya » Thu Feb 16, 2017 7:12 pm

I thought I'd seen something about 60TB drives -
http://www.seagate.com/about-seagate/ne ... oducts-pr/

dhirota
V.I.P Member
V.I.P Member
Posts: 391
Joined: Sun Nov 01, 2009 11:18 pm
Full Name: Dennis Hirota
Company Details: Sam O. Hirota Inc.
Company Position Title: President
Country: USA
Linkedin Profile: Yes
Location: Hawaii, USA

10TB, 12TB, 14TB HELIUM FILLED HDD & INFINIBAND

Post by dhirota » Fri Feb 17, 2017 12:43 am

Jason

The very simple short story is that we were looking for relatively, fast, cheaper storage than the cloud, which we are not allowed to use by some clients. The resulting survey of options (we might have missed many), was to use the INTEL servers because of reuse in the future (able to change 12X3.5 inch HDD to 24X2.5 inch SSD, 1.5TB RAM, 4 GTX GPU, 2each 1100W Power Supply) and direct support/warranty to 2021. The net result for 10GbE is that you get two 10GbE ports on the INTEL boxes for FREE. So we have a few INTEL RJ45 10GbE ports and a while ago decided not to spend on money on reorganizing our internal network (which has mostly CAT6, CAT6A but some CAT5e for printers).

An interesting article
https://www.nextplatform.com/2015/04/01 ... o-kill-it/

Maybe 2017 is the year for us to evaluate Infiniband and our internal network.
2-16-2017 2-20-18 PM.jpg
You do not have the required permissions to view the files attached to this post.
Dennis Hirota, PhD, PE, LPLS
dennishirota@gmail.com

User avatar
Jason Warren
Administrator
Administrator
Posts: 2862
Joined: Thu Aug 16, 2007 9:21 am
Full Name: Jason Warren
Company Details: Laser Scanning Forum Ltd
Company Position Title: Co-Founder
Country: UK
Skype Name: jason_warren
Linkedin Profile: Yes
Location: Retford, UK

10TB, 12TB, 14TB HELIUM FILLED HDD & INFINIBAND

Post by Jason Warren » Fri Feb 17, 2017 8:12 am

dhirota wrote: An interesting article
https://www.nextplatform.com/2015/04/01 ... o-kill-it/

Maybe 2017 is the year for us to evaluate Infiniband and our internal network.
Dennis,
I know you like to keep at the cutting edge ;)
A two card Mellanox solution for a dedicated connection between a server and `Registration` workstation would be in the order of £1135.19. Its not cheap, but its not 10GbE.
- MCX353A-FCCT, Single VPI FDR/40/56GbE InfiniBand HCA - £538 ea exVAT
- MC2207130-002, 2m 56Gb/s Passive Copper Cables - £59.16 ea exVAT

If you want a cheaper Infiniband solution, ie not 56GbE but say 20GbE or 40GbE, then check out all the stuff thats coming out of data-centres at the moment and landing on ebay. This kit is cheap, but as always buyer beware!

Lastly, if you get a spare 5 minutes, take a look at what ICEOTOPE are doing. Their machines have a unique cooling system. The entire motherboard etc is submerged in liquid plastic for unique cooling and silent running... ;)
http://www.iceotope.com/
Jason Warren

#ThePowerofLSF

dhirota
V.I.P Member
V.I.P Member
Posts: 391
Joined: Sun Nov 01, 2009 11:18 pm
Full Name: Dennis Hirota
Company Details: Sam O. Hirota Inc.
Company Position Title: President
Country: USA
Linkedin Profile: Yes
Location: Hawaii, USA

Re: 10TB, 12TB, 14TB HELIUM FILLED HDD & INFINIBAND

Post by dhirota » Fri Feb 17, 2017 7:13 pm

Jason

Thank you for the information on infiniband. Our current "registration/processing" workstation is like a server with 60TB of SAS HDD on it, so it does not have to move to another server at 56Gbs speed for storage. The equivalent of 2X10GbE is fast enough at a cost of FREE. The US$1500 would be interesting to move processed data between servers which is currently 2X10GbE, so the 56Gbs from infiniband would be a significant improvement.

We have not researched to see if the 56Gbs speed would be limited by the average 275MB/second write speed of our server HDD, since when we robocopy files, the task manager is indicating that the HDD is writing at 100% using the 2X10GbE network.

The Iceotope system sounds interesting, but in the middle of 5,000 miles of ocean, and 2,500 miles from any significant land mass, we need to fix it and have it working without sending it back 12,000 miles by FEDEX/DHL.
Dennis Hirota, PhD, PE, LPLS
dennishirota@gmail.com

User avatar
Jason Warren
Administrator
Administrator
Posts: 2862
Joined: Thu Aug 16, 2007 9:21 am
Full Name: Jason Warren
Company Details: Laser Scanning Forum Ltd
Company Position Title: Co-Founder
Country: UK
Skype Name: jason_warren
Linkedin Profile: Yes
Location: Retford, UK

Re: 10TB, 12TB, 14TB HELIUM FILLED HDD & INFINIBAND

Post by Jason Warren » Sat Feb 18, 2017 8:53 am

dhirota wrote: We have not researched to see if the 56Gbs speed would be limited by the average 275MB/second write speed of our server HDD, since when we robocopy files, the task manager is indicating that the HDD is writing at 100% using the 2X10GbE network.
True, if your I/O is maxed out at 275MB/s then there`s little point in utilising a 56GbE connection.
dhirota wrote: The Iceotope system sounds interesting, but in the middle of 5,000 miles of ocean, and 2,500 miles from any significant land mass, we need to fix it and have it working without sending it back 12,000 miles by FEDEX/DHL.
Thats the point, the main reason for server component failure is overheating. With these bad boys they shouldn't need returning.... ;)
Jason Warren

#ThePowerofLSF

dhirota
V.I.P Member
V.I.P Member
Posts: 391
Joined: Sun Nov 01, 2009 11:18 pm
Full Name: Dennis Hirota
Company Details: Sam O. Hirota Inc.
Company Position Title: President
Country: USA
Linkedin Profile: Yes
Location: Hawaii, USA

Re: 10TB, 12TB, 14TB HELIUM FILLED HDD & INFINIBAND

Post by dhirota » Mon Feb 20, 2017 2:01 am

Jason

I do not know if we have completed the discussion of SAS 10TB to14TB Helium HDD and Infiniband yet, but when I started this thread I mentioned about the announcement of large SAS SSD drives like the Seagate 60TB 3.5 inch SSD that was mentioned earlier by SAttaya or the 100TB SSD by Toshiba.

We decided to delay the implementation of infiniband until later when the pricing of SAS SSD systems arrives at a price point where SSDs like the Samsung PM1633a 15.36TB in volume drops significantly below US$10K. So we decided to fill up our INTEL servers to 100 TB each with the Seagate 10TB SAS HDD rather than wait for the Seagate 10TB HDD to drop. From our source we got them at US$433 each, which is only a few $ above the 8TB. I think we did OK since the box of 20 + 6 extra HDD for 100TB each in raid 6 and spare HDDs was less than US$6,000 each.

We have the space on the INTEL servers to add 4X 2.5 inch SSDs to our BROADCOM/AVAGO raid controllers as cache or raid SSD storage.
Dennis Hirota, PhD, PE, LPLS
dennishirota@gmail.com

Return to “Future Technology”