NAS/Storage Megathread: What is this "File Deletion" You Speak of

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > NAS/Storage Megathread: What is this "File Deletion" You Speak of

«‹›843 »

THF13: Sep 26, 2007; Keep an adversary in the dark about what you're capable of, and he has to assume the worst.

IndianaZoidberg posted:

I�m not sure if this is the right place to ask this, but here we go.

I was hoping someone could help me with an Unraid build (and I'm looking at Unraid just because it looks easier to use than some of the other options).

I would like to spend $1000 or less (not including drives) and it would be used for mass storage and hopefully as a Plex server.

I like this case (Rosewill 4U Server Chassis with 12 hot-swap drive bays)
https://www.amazon.com/gp/product/B00N9CXGSO/ref=crt_ewc_img_huc_1?ie=UTF8&psc=1&smid=ATVPDKIKX0DER
but I�m not married to the idea, or if there is a used solution I could find on eBay, I would be down for that as well.

I am a NAS noob and would love some help and hand-holding.

I built the "anniversary" build following a guide on serverbuilds.net and have been extremely happy with it. The motherboard for that one isn't available anymore but they have other builds worth a look.

NASKiller4.0: Variety of options for a NAS ranging from 2-10 plex streams, 6-15 HD capacity, $175-$600.
Lego: Larger dual CPU build with lots of expandability options, considered an "in progress" build so doesn't have as much info/options.
Hardware transcoding: Not a build exactly, but the idea here is to use an cheap small box just for plex/emby with hardware transcoding separate from your NAS.
DAS: Attach 15 bays to an existing PC/server.

# ? Oct 20, 2019 23:59

Adbot: ADBOT LOVES YOU

# ? May 7, 2024 19:03

Raymond T. Racing: Jun 11, 2019

THF13 posted:

I built the "anniversary" build following a guide on serverbuilds.net and have been extremely happy with it. The motherboard for that one isn't available anymore but they have other builds worth a look.

NASKiller4.0: Variety of options for a NAS ranging from 2-10 plex streams, 6-15 HD capacity, $175-$600.

Lego: Larger dual CPU build with lots of expandability options, considered an "in progress" build so doesn't have as much info/options.

Hardware transcoding: Not a build exactly, but the idea here is to use an cheap small box just for plex/emby with hardware transcoding separate from your NAS.

DAS: Attach 15 bays to an existing PC/server.

anniversary bro

As for case recommendations, the Rosewill hotswap is a massive waste of money, and I'm of the opinion that the 15 bay non hotswap Rosewill is a much better use of less money.

# ? Oct 21, 2019 00:26

IndianaZoidberg: Aug 21, 2011; My name isnt slick, its Zoidberg. JOHN F***ING ZOIDBERG!

Actuarial Fables posted:

How many plex streams do you plan on serving at a time?
Will you be running other services on the NAS as well (VMs, docker containers, torrents/usenet) or just storage + streaming?
How much storage space do you need?
How much storage space do you want?
Do you have somewhere to put the 2ft x 1.5ft box where you won't be bothered by the noise?

Plex streams: 1 to 2. Ideally 3 at once to cover all bases.
Services: storage and streaming. Possibly some torrents, but I'm not sure I knew that was a thing so I had not thought about it.
Need: 50tb usable storage
Want: 100tb+ for future expansion.
I was going to put it in my closet and hope it's quiet.

THF13 posted:

I built the "anniversary" build following a guide on serverbuilds.net and have been extremely happy with it. The motherboard for that one isn't available anymore but they have other builds worth a look.

NASKiller4.0: Variety of options for a NAS ranging from 2-10 plex streams, 6-15 HD capacity, $175-$600.

Lego: Larger dual CPU build with lots of expandability options, considered an "in progress" build so doesn't have as much info/options.

Hardware transcoding: Not a build exactly, but the idea here is to use an cheap small box just for plex/emby with hardware transcoding separate from your NAS.

DAS: Attach 15 bays to an existing PC/server.

I'll check those out.

# ? Oct 21, 2019 03:03

Hed: Mar 31, 2004; Fun Shoe

Too bad it�s not 2011 so you could change your name to OpenIndianaZoidberg

# ? Oct 21, 2019 03:23

Atomizer: Jun 24, 2007

Paul MaudDib posted:

I think you're looking for an "8-bay SAS/SATA/JBOD enclosure".

https://www.newegg.com/mediasonic-h8r2-su3s2/p/N82E16816322007

Since you brought it up, has anyone used one of those Mediasonic enclosures in particular? They've got a few of those 8-bay variants and they caught my eye a while back (even though I don't really have need of one.)

DrDork posted:

Nah, they still exist. Seagate has the FireCuda line, for example. They're just not used very often because a SSD + HDD is generally a better option for most people. SSHDs are really only good ideas for single-drive systems that you don't want to spend the money on a full SSD for.

This. The 2.5" Firecuda 2 TB is good for game consoles like a PS4 that have a single internal bay, or as a secondary drive for game storage in a gaming laptop. This drive is also the cheapest option for 2 TB at 2.5", so it's perfectly fine if you don't need or want a same-capacity SSD at close to $200.

There used to be 4 TB 3.5" SSHDs, but nowadays the only 3.5" SSHD I know of that is being currently produced is also Seagate's desktop FireCuda variant. They're reasonably performant but there's really no reason to get one (as secondary storage) because you might as well be using a much more capacious HDD for storage and SSDs for performance in a desktop environment.

IOwnCalculus posted:

And even with those chips, transcoding 4K makes them cry.

Speaking of market rate, is it just me or has it been a while since the last deal on hard drives? I had some of my 40-50k hour WD30EFRX drives start dying so I snagged some used Hitachi 3TB SAS drives to hold me over until Black Friday, when I'm hoping there will be a good deal on either 8TB or 10TB so I can replace one of my vdevs, and effectively have a ton of spares for my remaining high-hour 3TB drives.

UHD transcoding is very CPU-intensive, especially the more complex the file is. You pretty much need a recent Ryzen 7 or i7/i9, or better, to do a high-bitrate UHD HEVC HDR transcode in software.

And about the HDD sales, that does seem right; we used to see 8/10 TB external drives go on sale pretty much every month, and although I haven't been in the market for a while, I can't say I've noticed as many deals recently when skimming through sales threads. I can post links here when I see them if you guys are interested (I just assumed everyone else looks out for the sales like I do.)

# ? Oct 21, 2019 07:54

Gyrotica: Nov 26, 2012; Grafted to machines your builders did not understand.

Atomizer posted:

And about the HDD sales, that does seem right; we used to see 8/10 TB external drives go on sale pretty much every month, and although I haven't been in the market for a while, I can't say I've noticed as many deals recently when skimming through sales threads. I can post links here when I see them if you guys are interested (I just assumed everyone else looks out for the sales like I do.)

I�d be interested!

# ? Oct 21, 2019 12:00

IndianaZoidberg: Aug 21, 2011; My name isnt slick, its Zoidberg. JOHN F***ING ZOIDBERG!

This might be a dumb question about Unraid, but when I drive fails and needs to be replaced, I assume the interface will say which drive it is, but can it do some magic and light up a LED next to the dead drive so you know it's that one that's dead? Kind of like how the Drobo's do it?

# ? Oct 22, 2019 03:14

Actuarial Fables: Jul 29, 2014; Taco Defender

IndianaZoidberg posted:

This might be a dumb question about Unraid, but when I drive fails and needs to be replaced, I assume the interface will say which drive it is, but can it do some magic and light up a LED next to the dead drive so you know it's that one that's dead? Kind of like how the Drobo's do it?

That function isn't built into Unraid, but there's a plugin that accomplishes that task https://forums.unraid.net/topic/77302-plugin-disk-location/.

# ? Oct 22, 2019 04:18

Atomizer: Jun 24, 2007

Gyrotica posted:

I�d be interested!

Alrighty, I will keep an eye out for HDD deals and post them here! Typically the sale prices are around:
- $90-100 for 6 TB
- $130 for 8 TB
- $160 for 10 TB
These are almost universally going to be WD external drives (shuckable) and are likely Blue (mainstream/desktop) at the 6 TB capacity and Red/white label (NAS) at 8+ TB. I haven't seen any sales on drives >10+ TB that put them remotely near the price/TB ratio of the above. The sellers will usually be Amazon, Best Buy, Newegg, Rakuten, and occasionally eBay.

# ? Oct 22, 2019 07:21

sockpuppetclock: Sep 12, 2010

D. Ebdrup posted:

While I haven't used it, restic keeps getting brought up by people I know who work across many different platforms.

restic seems like it's for making encrypted backup repos & snapshots, which is useful, but I need the data slightly more accessible.

I really dunno what I'm doing but I looked into rsync on windows via cygwin to the qnap's rsync daemon.
The problem is the data I tested it with just kept spewing this error for every file like so:

code:

rsync: set_acl: sys_acl_set_file(txt, ACL_TYPE_ACCESS): Operation not supported (95)
rsync: set_acl: sys_acl_set_file(txt, ACL_TYPE_DEFAULT): Operation not supported (95)
rsync: set_acl: sys_acl_set_file(txt/.1.txt.2DxWMg, ACL_TYPE_ACCESS): Operation not supported (95)
rsync: set_acl: sys_acl_set_file(txt/.2.txt.cs0Ljy, ACL_TYPE_ACCESS): Operation not supported (95)

And when I grab the test data back from the qnap the file permissions are basically a mess.

If someone understands the issue I would appreciate a solution. Otherwise I'll just settle for backing up with restic...

# ? Oct 23, 2019 01:06

Raymond T. Racing: Jun 11, 2019

sockpuppetclock posted:

restic seems like it's for making encrypted backup repos & snapshots, which is useful, but I need the data slightly more accessible.

I really dunno what I'm doing but I looked into rsync on windows via cygwin to the qnap's rsync daemon.
The problem is the data I tested it with just kept spewing this error for every file like so:
code:
rsync: set_acl: sys_acl_set_file(txt, ACL_TYPE_ACCESS): Operation not supported (95)
rsync: set_acl: sys_acl_set_file(txt, ACL_TYPE_DEFAULT): Operation not supported (95)
rsync: set_acl: sys_acl_set_file(txt/.1.txt.2DxWMg, ACL_TYPE_ACCESS): Operation not supported (95)
rsync: set_acl: sys_acl_set_file(txt/.2.txt.cs0Ljy, ACL_TYPE_ACCESS): Operation not supported (95)
And when I grab the test data back from the qnap the file permissions are basically a mess.

If someone understands the issue I would appreciate a solution. Otherwise I'll just settle for backing up with restic...

so to confirm, cygwin rsync copies to qnap, then you use SMB to get it back from the qnap?

if so you're confusing the hell out of file ownership because of that

# ? Oct 23, 2019 03:27

Paul MaudDib: May 3, 2006; TEAM NVIDIA:
FORUM POLICE

do I remember properly that there is some gotcha about serving NFS and Samba of the same files at the same time? possibly with or without ZFS, I don't recall

edit: possibly locking, if the same file is accessed at the same time

Paul MaudDib fucked around with this message at 03:56 on Oct 23, 2019

# ? Oct 23, 2019 03:51

Raymond T. Racing: Jun 11, 2019

Paul MaudDib posted:

do I remember properly that there is some gotcha about serving NFS and Samba of the same files at the same time? possibly with or without ZFS, I don't recall

edit: possibly locking, if the same file is accessed at the same time

cygwin user account interactions with windows is awful

https://cygwin.com/cygwin-ug-net/ntsec.html

# ? Oct 23, 2019 04:02

Paul MaudDib: May 3, 2006; TEAM NVIDIA:
FORUM POLICE

Buff Hardback posted:

cygwin user account interactions with windows is awful

https://cygwin.com/cygwin-ug-net/ntsec.html

sorry, actually was asking that about linux+freeBSD

# ? Oct 23, 2019 04:08

sockpuppetclock: Sep 12, 2010

Buff Hardback posted:

so to confirm, cygwin rsync copies to qnap, then you use SMB to get it back from the qnap?

if so you're confusing the hell out of file ownership because of that

I just used rsync and scp

code:

rsync -avhPHAx --password-file=/rpass /cygdrive/c/testinput/ username@192.168.0.16::backup/qtest/
scp -r admin@192.168.0.16:/share/backup/qtest/ /cygdrive/c/testoutput/

# ? Oct 23, 2019 04:12

Raymond T. Racing: Jun 11, 2019

sockpuppetclock posted:

I just used rsync and scp

code:

rsync -avhPHAx --password-file=/rpass /cygdrive/c/testinput/ username@192.168.0.16::backup/qtest/
scp -r admin@192.168.0.16:/share/backup/qtest/ /cygdrive/c/testoutput/

yeah the permissions probably got super mangled because cygwin kept the SID as written, and windows didn't parse it back in as an SID but instead treated it literally

# ? Oct 23, 2019 04:29

Shaocaholica: Oct 29, 2002; Fig. 5E

Would it be bad to put a Ultrastar DC HC530 14TB into a USB enclosure?

# ? Oct 23, 2019 06:36

taqueso: Mar 8, 2004

What are you using it for?

# ? Oct 23, 2019 06:55

Shaocaholica: Oct 29, 2002; Fig. 5E

taqueso posted:

What are you using it for?

large video project source files but mostly to dick around with.

# ? Oct 23, 2019 07:09

Rexxed: May 1, 2010; Dis is amazing!
I gotta try dis!

Shaocaholica posted:

Would it be bad to put a Ultrastar DC HC530 14TB into a USB enclosure?

It should be fine, just keep in mind that as a 3.5" drive it'll need an external power brick to get the 12V input it requires to work. All of the 3.5" enclosures I've seen include them. Power usage isn't high for hard disks so it's not a huge transformer but you do need one.

# ? Oct 23, 2019 07:30

Shaocaholica: Oct 29, 2002; Fig. 5E

Rexxed posted:

It should be fine, just keep in mind that as a 3.5" drive it'll need an external power brick to get the 12V input it requires to work. All of the 3.5" enclosures I've seen include them. Power usage isn't high for hard disks so it's not a huge transformer but you do need one.

Yeah already covered. Got a few of these for cheap from a studio closure. DC PSU built into the enclosure.

https://oyendigital.com/hard-drives/store/HDX1500.html

# ? Oct 23, 2019 07:42

Rexxed: May 1, 2010; Dis is amazing!
I gotta try dis!

Shaocaholica posted:

Yeah already covered. Got a few of these for cheap from a studio closure. DC PSU built into the enclosure.

https://oyendigital.com/hard-drives/store/HDX1500.html

Yeah that's more than enough to handle the disk. I was going to say to check that the firmware on the enclosure supports drives that big but they sell it with 14TB as an option so clearly it must.

# ? Oct 23, 2019 07:48

Shaocaholica: Oct 29, 2002; Fig. 5E

Yeah just wasn't sure about using the HC530 as an external as it wasn't designed for it. Like a lot more power cycles but maybe its just built even tougher than the typical drives they would throw into an external. I'm too lazy to compare spec sheets.

# ? Oct 23, 2019 07:55

BlankSystemDaemon: Mar 13, 2009

Paul MaudDib posted:

do I remember properly that there is some gotcha about serving NFS and Samba of the same files at the same time? possibly with or without ZFS, I don't recall

edit: possibly locking, if the same file is accessed at the same time

Only caveat I know of is that NetworkLockManager(in Linux/Windows)/lockd(FreeBSD) needs to run on every server and client.

# ? Oct 23, 2019 09:13

DrDork: Dec 29, 2003; commanding officer of the Army of Dorkness

Shaocaholica posted:

Yeah just wasn't sure about using the HC530 as an external as it wasn't designed for it. Like a lot more power cycles but maybe its just built even tougher than the typical drives they would throw into an external. I'm too lazy to compare spec sheets.

It should be fine. There aren't any special "external" drives, when you get right down to it. They're just normal internal drives that they threw into an enclosure. In fact, often times they're lower-tier, shittier drives because I guess people just accept that external drives die more often because they get bumped around or something.

# ? Oct 23, 2019 13:38

sharkytm: Oct 9, 2003; Ba

By

Sharkytm doot doo do doot do doo; Fallen Rib

Or in the case of the WD 8TB and 10TB drives, they're actually Enterprise drives that have a shorter warranty, but are identical to the $$$ Enterprise drives

# ? Oct 23, 2019 14:16

CommieGIR: Aug 22, 2006; The blue glow is a feature, not a bug; Pillbug

Welp, the HP 70MSA 2.5" 25 bay array sucks. It consumes 3x the power of my MD1000, so I transferred all the VMs off it tonight back onto their internal storage and shut it down.

It was reliable, don't get me wrong. But it sucked down more juice than my IBM bladecenter.

CommieGIR fucked around with this message at 03:19 on Oct 24, 2019

# ? Oct 24, 2019 03:10

Crunchy Black: Oct 24, 2017; by Athanatos

I'm helping to decom one of our old datacenters full of Netapps with 450 and 750gb 12gb SAS drives tomorrow. Company lore says we got them back in the bad old days after they literally fell off a truck. Its finally getting colder in ATL so you goddamn better know I'm going to hate myself enough to steal one and run it for at least a month

e:

CommieGIR posted:

Welp, the HP 70MSA 2.5" 25 bay array sucks. It consumes 3x the power of my MD1000, so I transferred all the VMs off it tonight back onto their internal storage and shut it down.

It was reliable, don't get me wrong. But it sucked down more juice than my IBM bladecenter.

drat, man, that's disappointing. You putting it back up for sale, or? I have a MD1000 that I don't even have powered up right now so I shouldn't be looking for ADDITIONAL large power draw legacy devices.

Crunchy Black fucked around with this message at 05:21 on Oct 24, 2019

# ? Oct 24, 2019 05:19

CommieGIR: Aug 22, 2006; The blue glow is a feature, not a bug; Pillbug

Crunchy Black posted:

I'm helping to decom one of our old datacenters full of Netapps with 450 and 750gb 12gb SAS drives tomorrow. Company lore says we got them back in the bad old days after they literally fell off a truck. Its finally getting colder in ATL so you goddamn better know I'm going to hate myself enough to steal one and run it for at least a month

e:

drat, man, that's disappointing. You putting it back up for sale, or? I have a MD1000 that I don't even have powered up right now so I shouldn't be looking for ADDITIONAL large power draw legacy devices.

Hey fellow ATL guy!

Probably sell it, maybe sans drives, since I may just find a more power efficient array.

Ironically, this is filled with 900 gb Netapp SAS disks from ebay for a song and a dance.

# ? Oct 24, 2019 15:09

Paul MaudDib: May 3, 2006; TEAM NVIDIA:
FORUM POLICE

Crunchy Black posted:

I'm helping to decom one of our old datacenters full of Netapps with 450 and 750gb 12gb SAS drives tomorrow. Company lore says we got them back in the bad old days after they literally fell off a truck. Its finally getting colder in ATL so you goddamn better know I'm going to hate myself enough to steal one and run it for at least a month

obviously using 16 drives (15k drives no less) to provide the capacity of a single 8TB easystore is gonna be a bad time but are the Netapp chassis themselves all that bad on power? I thought I remember hearing they were about 100-150W to run the chassis itself (DS4243).

# ? Oct 24, 2019 15:16

CommieGIR: Aug 22, 2006; The blue glow is a feature, not a bug; Pillbug

Paul MaudDib posted:

obviously using 16 drives (15k drives no less) to provide the capacity of a single 8TB easystore is gonna be a bad time but are the Netapp chassis themselves all that bad on power? I thought I remember hearing they were about 100-150W to run the chassis itself (DS4243).

The problem is less the chassis, more the SAS drives. Technically, at least with the HP, you can use SATA and it cuts down the power usage a little bit, but SAS disks themselves are serious power hogs.

# ? Oct 24, 2019 15:25

Moey: Oct 22, 2010; I LIKE TO MOVE IT

Newegg has the 10tb WD Elements external on sale today for $160 with promo code EMCUTVY29.

Get your shuck on.

https://www.newegg.com/black-wd-elements-10tb/p/N82E16822234350

# ? Oct 24, 2019 15:26

Gyrotica: Nov 26, 2012; Grafted to machines your builders did not understand.

Moey posted:

Newegg has the 10tb WD Elements external on sale today for $160 with promo code EMCUTVY29.

Get your shuck on.

https://www.newegg.com/black-wd-elements-10tb/p/N82E16822234350

Is that about as low as they're likely to get?

edit: Basically I'm wondering if I should just go ahead and pull the trigger now or if I can safely wait for Black Friday.

Gyrotica fucked around with this message at 16:04 on Oct 24, 2019

# ? Oct 24, 2019 15:27

pzy: Feb 20, 2004; Da Boom!

Limit 2 per customer is kind of a bummer. I just built an unraid server with 24x 10TB Easystores at $180 each, I wish they'd hit $160 again, maybe with a free dumb thumbdrive! Though Best Buy just threw them back to $249 each, with the 12TB ones at $279.

# ? Oct 24, 2019 15:54

Moey: Oct 22, 2010; I LIKE TO MOVE IT

Gyrotica posted:

Is that about as low as they're likely to get?

edit: Basically I'm wondering if I should just go ahead and pull the trigger now or if I can safely wait for Black Friday.

Eh, if you are in no rush, they could go cheaper (I would guess). These sales pop up about every month.

# ? Oct 24, 2019 16:06

Gyrotica: Nov 26, 2012; Grafted to machines your builders did not understand.

Moey posted:

Eh, if you are in no rush, they could go cheaper (I would guess). These sales pop up about every month.

Nice - thanks! I think it will be a little more palatable to the spouse the closer we are to Christmas....

# ? Oct 24, 2019 16:15

ILikeVoltron: May 17, 2003; I <3 spyderbyte!

I'm trying to upgrade my mainboard to something that supports M.2 NVMe and I'm having a huge PITA figuring out wtf supermicro is talking about on their descriptions. It's down to the two following boards:

X10DRD-iNTP vs X10DRD-iNT

The iNTP lists the following: 2 PCI-E 3.0 NVMExpress x4 External Ports

and the iNT lists: 2 Internal NVMe ports (PCI-E 3.0 x4)

I assume the iNT is M.2, but it's not really shown that well in any of the pictures or otherwise.. and I'm totally confused over what an external port is to a mainboard?

# ? Oct 25, 2019 00:55

Actuarial Fables: Jul 29, 2014; Taco Defender

ILikeVoltron posted:

I assume the iNT is M.2, but it's not really shown that well in any of the pictures or otherwise.. and I'm totally confused over what an external port is to a mainboard?

NVMe is more than just M.2 drives, so don't make that assumption.

Looked at the manual for the iNT board, there are two ports just next to the PCIe slots that are the NVMe ports, SFF-8643 form factor. Usually you'd connect U.2 drives to these.

"NVM Express PCI-E 3.0 x4 ports (by internal mini-SAS HD connector) (for X10DRD-iNT only)"
https://www.supermicro.com/manuals/motherboard/C600/MNL-1712.pdf

Expensive server motherboards usually aren't going to have M.2 drives because no one wants to shut down a server to replace a drive. M.2 are not hot swappable, unlike U.2 drives.

e. As for the external NVMe for the iNTP board, that might be a typo? The manual shows the same ports as the iNT board has and lists them as internal, no extra I/O on the back besides the dual SFP+.

e2. Really like that the picture of the iNT board on supermicro's site is extremely tiny and also isn't actually the correct model.

e3. There are M.2 to U.2 enclosures if you already have an M.2 drive you want to use.

Actuarial Fables fucked around with this message at 01:53 on Oct 25, 2019

# ? Oct 25, 2019 01:30

phosdex: Dec 16, 2005

If you go to supermicros site and look at the manuals for those boards you can see they are not m.2. Not sure how those ports get used.

E: beaten

phosdex fucked around with this message at 01:36 on Oct 25, 2019

# ? Oct 25, 2019 01:32

Adbot: ADBOT LOVES YOU

# ? May 7, 2024 19:03

Atomizer: Jun 24, 2007

Gyrotica posted:

Is that about as low as they're likely to get?

edit: Basically I'm wondering if I should just go ahead and pull the trigger now or if I can safely wait for Black Friday.

I believe they've been $150, once, but typically $160 is the lowest sale price for those 10 TB drives. Often there are per-account limitations, like 2 as in this case, so if you need more than that I'd buy these and then a couple more at each sale.

# ? Oct 25, 2019 02:05

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > NAS/Storage Megathread: What is this "File Deletion" You Speak of

«‹›843 »