Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
insularis
Sep 21, 2002

Donated $20. Get well, Lowtax.
Fun Shoe
ZFS fun moment. NZBGet had a beta build that created monstrous log files due to a bug, like drive-consuming sizes. My log was going to my array with lz4 compression turned on ... a 1.1TB log file was 28GB on disk when I found it and deleted it.

Adbot
ADBOT LOVES YOU

KOTEX GOD OF BLOOD
Jul 7, 2012

Just went and bought another easystore, this time for USB use with my synology unit. Just out of curiosity, I checked what kind of drive it was with DriveDX...EFAX, again! I seem to win the drive lottery every time.

Internet Explorer
Jun 1, 2005





More of a Transmission on Linux question, please point me in the right direction if this is the wrong place.

Trying to set it up so that I can download torrents into 2 different folders, Folder A and Folder B. When I do this, the torrent file itself still goes into config/torrents/, but the torrent data gets moved to the correct folder after the download is complete.

I assume this uses syslinks, but where is the data stored that links that torrent to the file in either Folder A or Folder B? I want to make sure that is backed up or accounted for (using Docker, so it's important) so that if I need to re-deploy I don't have to go through that giant folder of torrents and point them at the data in the correct folder.

If that doesn't make sense I can re-state. Anyone have any clue? Thank you.

Paul MaudDib
May 3, 2006

TEAM NVIDIA:
FORUM POLICE

Internet Explorer posted:

I assume this uses syslinks, but where is the data stored that links that torrent to the file in either Folder A or Folder B? I want to make sure that is backed up or accounted for (using Docker, so it's important) so that if I need to re-deploy I don't have to go through that giant folder of torrents and point them at the data in the correct folder.

If you think it's a symlink you can follow it with "readlink -f /path/to/symlink".

Internet Explorer
Jun 1, 2005





Paul MaudDib posted:

If you think it's a symlink you can follow it with "readlink -f /path/to/symlink".

I guess my question is more, will that information get destroyed if/when I update the Docker container from the latest image. I have the config files stored outside the container, the torrent files, the data, etc., but I just don't know about the mechanism that points the torrent to the data. If it's a symlink (thanks), I assume that's in the OS and that would be problematic?

Simone Poodoin
Jun 26, 2003

Che storia figata, ragazzo!



What's your log rotation tool/policy on unraid? Recently I had a bug spamming syslog and was wondering what's a good strategy, I didn't find any plugins to deal with this.

The "Fix Common Problems" plugin is great to find out about this sort of thing btw.

IOwnCalculus
Apr 2, 2003





You should be able to update the container without problems. All of the containers I use store literally everything outside of themselves on the main filesystems, so when it's time to update you just stop the container, rm the old instance, pull the latest, and redeploy with the same command / settings. Picks up right where it left off.

Paul MaudDib
May 3, 2006

TEAM NVIDIA:
FORUM POLICE

Internet Explorer posted:

I guess my question is more, will that information get destroyed if/when I update the Docker container from the latest image. I have the config files stored outside the container, the torrent files, the data, etc., but I just don't know about the mechanism that points the torrent to the data. If it's a symlink (thanks), I assume that's in the OS and that would be problematic?

purely talking out my rear end here but I assume there is a directory somewhere in the real OS and Docker has a symlink to that directory. That would be the sensible way to do it, and you would just need to re-create that symlink with the new container via whatever mechanism docker normally uses.

If it's actually a mess of individual symlinks to individual files then you have a problem and you should start thinking about something like rsync to help back that up.

Lowen SoDium
Jun 5, 2003

Highen Fiber
Clapping Larry
I currently have a Ubuntu file server with 3x 3TB WD REDs in RAIDZ. Disk are just over 5 years old and the array is full. Looking to replace them all. Data will be copied from old array to new array.

Would you rather have more drives in a higher RAIDZ mode or fewer larger drives in a lower RAIDZ mode?

For example:

5x 6TB WD Reds in RAIDZ2 for 18TB of usable storage with 2 parity

or

3x 10TB WD Reds in RAIDZ for 20TB of usable storage with 1 parity

Cost for each option is close enough that it's not factor for me. The extra 2TB isn't a big concern for me either. I am more concerned about reliability and data loss issues.

H110Hawk
Dec 28, 2006

Lowen SoDium posted:

I currently have a Ubuntu file server with 3x 3TB WD REDs in RAIDZ. Disk are just over 5 years old and the array is full. Looking to replace them all. Data will be copied from old array to new array.

Would you rather have more drives in a higher RAIDZ mode or fewer larger drives in a lower RAIDZ mode?

For example:

5x 6TB WD Reds in RAIDZ2 for 18TB of usable storage with 2 parity

or

3x 10TB WD Reds in RAIDZ for 20TB of usable storage with 1 parity

Cost for each option is close enough that it's not factor for me. The extra 2TB isn't a big concern for me either. I am more concerned about reliability and data loss issues.

You want dual parity in both scenarios, thus the first option.

Internet Explorer
Jun 1, 2005





IOwnCalculus posted:

You should be able to update the container without problems. All of the containers I use store literally everything outside of themselves on the main filesystems, so when it's time to update you just stop the container, rm the old instance, pull the latest, and redeploy with the same command / settings. Picks up right where it left off.

Paul MaudDib posted:

purely talking out my rear end here but I assume there is a directory somewhere in the real OS and Docker has a symlink to that directory. That would be the sensible way to do it, and you would just need to re-create that symlink with the new container via whatever mechanism docker normally uses.

If it's actually a mess of individual symlinks to individual files then you have a problem and you should start thinking about something like rsync to help back that up.

It turns out that the part that says "this torrent file corresponds with this torrent data" is stored in a .resume file located in \config\resume. So, as long as the config folder is stored outside of the container, you are good to go. https://github.com/transmission/transmission/wiki/Transmission-Resume-Files

Not sure why I had trouble finding that info, but in my defense when you try to click on the "Configuration File" link (or many of the other links) on Transmission's wiki it is a dead link, so finding documentation was awkward.

Thanks all for the help. And hopefully that little bit of information helps someone at some point.

Internet Explorer fucked around with this message at 16:39 on Jun 22, 2018

EVIL Gibson
Mar 23, 2001

Internet of Things is just someone else's computer that people can't help attaching cameras and door locks to!
:vapes:
Switchblade Switcharoo

Lowen SoDium posted:

I currently have a Ubuntu file server with 3x 3TB WD REDs in RAIDZ. Disk are just over 5 years old and the array is full. Looking to replace them all. Data will be copied from old array to new array.

Would you rather have more drives in a higher RAIDZ mode or fewer larger drives in a lower RAIDZ mode?

For example:

5x 6TB WD Reds in RAIDZ2 for 18TB of usable storage with 2 parity

or

3x 10TB WD Reds in RAIDZ for 20TB of usable storage with 1 parity

Cost for each option is close enough that it's not factor for me. The extra 2TB isn't a big concern for me either. I am more concerned about reliability and data loss issues.

You have to consider that the rebuilding requires lots of writes and reads across all drives.

If you have 1 parity, then you need to hope to hell or high water no other drive goes offline/goes bad while rebuilding. This happens (seeingly) more often if you raid is made of hard drives from the same batch.

If you have 2 parity, one other drive can go bad during the heavy write/reads and it will not stop the rebuild process.

Paul MaudDib
May 3, 2006

TEAM NVIDIA:
FORUM POLICE
3x10 is going to be the more reliable option there, but it also has lower throughput.

HalloKitty
Sep 30, 2005

Adjust the bass and let the Alpine blast

Paul MaudDib posted:

3x10 is going to be the more reliable option there, but it also has lower throughput.

How could single parity with 10TB disks be more reliable than double parity on 6TB ones?

Moey
Oct 22, 2010

I LIKE TO MOVE IT

HalloKitty posted:

How could single parity with 10TB disks be more reliable than double parity on 6TB ones?

it isn't...

Lowen SoDium
Jun 5, 2003

Highen Fiber
Clapping Larry

Paul MaudDib posted:

3x10 is going to be the more reliable option there, but it also has lower throughput.

I am not expecting throughput to be a concern. It hasn't been with the current 3x 3TB array. I am not hosting VMs or anything on it that needs real high throughput.

HalloKitty posted:

How could single parity with 10TB disks be more reliable than double parity on 6TB ones?

More drives means more points for failure means a lower MTBF. But I am not sure it works out to be better for data integrity in these specific cases. I mean, I am actually not sure and I am trying to find out, which is why I asked.

I found this calculator and if I am using it right, it does look like 5x 6TB should have a much lower chance of data loss due to drive failure.

edit: forgot to paste the link http://wintelguy.com/raidmttdl.pl

Lowen SoDium fucked around with this message at 17:58 on Jun 22, 2018

IOwnCalculus
Apr 2, 2003





EVIL Gibson posted:

You have to consider that the rebuilding requires lots of writes and reads across all drives.

If you have 1 parity, then you need to hope to hell or high water no other drive goes offline/goes bad while rebuilding. This happens (seeingly) more often if you raid is made of hard drives from the same batch.

If you have 2 parity, one other drive can go bad during the heavy write/reads and it will not stop the rebuild process.

Counterpoint: ZFS does not poo poo the bed when this happens. I had this occur once during a rebuild and it marked a few files as corrupt, which let me easily restore them from backup.

H110Hawk
Dec 28, 2006

Lowen SoDium posted:

I found this calculator and if I am using it right, it does look like 5x 6TB should have a much lower chance of data loss due to drive failure.

edit: forgot to paste the link http://wintelguy.com/raidmttdl.pl

Correct.

EVIL Gibson
Mar 23, 2001

Internet of Things is just someone else's computer that people can't help attaching cameras and door locks to!
:vapes:
Switchblade Switcharoo

IOwnCalculus posted:

Counterpoint: ZFS does not poo poo the bed when this happens. I had this occur once during a rebuild and it marked a few files as corrupt, which let me easily restore them from backup.

Really, it's more safety concious where I build in case the worst case possible which always seems to happen lol

IOwnCalculus
Apr 2, 2003





EVIL Gibson posted:

Really, it's more safety concious where I build in case the worst case possible which always seems to happen lol

Oh yeah, I agree it's absolutely worth considering (and a reminder that RAID IS NOT BACKUP). It's just so very nice that ZFS will do its best to work around the issue instead of rolling over and dying because of one block with a URE.

Droo
Jun 25, 2003

Lowen SoDium posted:

I found this calculator and if I am using it right, it does look like 5x 6TB should have a much lower chance of data loss due to drive failure.

edit: forgot to paste the link http://wintelguy.com/raidmttdl.pl

If I'm reading that calculator correctly you are 10,000 times more likely to lose data with a 3x10tb raid 5 compared to a 5x6tb raid 6

BlankSystemDaemon
Mar 13, 2009



Droo posted:

If I'm reading that calculator correctly you are 10,000 times more likely to lose data with a 3x10tb raid 5 compared to a 5x6tb raid 6
MTTDL (mean time to data loss), even disregarding the spectre of silent data corruption that ZFS isn't affected by due to its design, is different from MTBF plus whatever mirroring or distributed parity your array is running, because as IOwnCalculus pointed out, ZFS knows which file(s) failed as a result of a dead sector during a rebuild, whereas quite a lot of traditional RAID implementations would mark the entire array as faulty. Putting that aside, though, MTTDL is also affected by the rebuild speed of the array, which is going to depend on the drives, the controllers and cable over-provisioning if any, and probably other factors that I'm forgetting.
Remember, ZFS comes to us from Big Iron, where Reliability, Availability and Servicability are enshrined among the highest of principles to such a degree that ZFS was designed without any kind of memory checksumming (except as a debug feature) because ECC and lockstepping+parity RAM is practically the default. In the case of ZFS, it's arguably completely tied to the availability section of the RAS trifecta, because you always want to have online backup and/or offline backup of the data.

redeyes
Sep 14, 2002

by Fluffdaddy
Don't use RAID 5. Who cares about the stats on drives failing. RAID 5 is old lovely technology.

22 Eargesplitten
Oct 10, 2010



Thoughts on using a PowerEdge T410 as a NAS? I could get one for extremely cheap, but I want to see if there's anything I should worry about with hardware like that considering how old it is. Desktop HDDs and PSUs and everything are fairly standardized throughout generations, I assume the same isn't necessarily true for servers, especially servers from '09.

IOwnCalculus
Apr 2, 2003





It should be *very* cheap since those E5500 generation procs are both ancient and power hungry. They can be damned powerful too, but they love to suck down juice. I have a couple of X5647s in my home server, that I'm about to swap out for L5640s at about half the TDP because the box here at home just never has anything that heavy to do. I'd drop to a single socket board if I could get one cheap enough that also had a LSI controller onboard like mine :v:

Doubtful that the PSU is standardized, because Dell.

It looks like it might be new enough to have at least a LSI2xxx generation HBA, if it has one at all, so that's good.

priznat
Jul 7, 2009

Let's get drunk and kiss each other all night.
Are the poweredge towers as insanely loud as the rack mounted ones? I am guessing not because those 2U fans have got to really scream.

BlankSystemDaemon
Mar 13, 2009



IOwnCalculus posted:

It should be *very* cheap since those E5500 generation procs are both ancient and power hungry. They can be damned powerful too, but they love to suck down juice.
All of this is true, but unfortunately they all lack AES-NI which makes them pretty much useless if you're wanting to do GELI encryption on your array and/or external drives.

H2SO4
Sep 11, 2001

put your money in a log cabin


Buglord

priznat posted:

Are the poweredge towers as insanely loud as the rack mounted ones? I am guessing not because those 2U fans have got to really scream.

Not generally, but if the fans need to turn up then they can drat sure get that way.

Devian666
Aug 20, 2008

Take some advice Chris.

Fun Shoe

priznat posted:

Are the poweredge towers as insanely loud as the rack mounted ones? I am guessing not because those 2U fans have got to really scream.

No, there's a massive difference. I have a dual socket rackmount that makes the noise you'd expect from all those small fans, and I have that in a storage room in the office so as not to annoy everyone. We have a couple of towers in the main office for the main server and replication. The UPS for the main server has a louder fan and it's no trouble in a small office.

22 Eargesplitten
Oct 10, 2010



D. Ebdrup posted:

All of this is true, but unfortunately they all lack AES-NI which makes them pretty much useless if you're wanting to do GELI encryption on your array and/or external drives.

Not sure what exactly GELI is, it looks like a BSD thing? How essential would that be? I really just want this thing for backups of my media and downloads and documents and my wife’s art and poo poo. I could even keep it on a separate VLAN from the WiFi and set up ssh access in case I needed to get on with my laptop without plugging in on Ethernet. Not sure why I would though, I could just keep it isolated on the Ethernet only VLAN.

I would probably be keeping it in the basement so I don’t have to give a poo poo about noise.

IOwnCalculus
Apr 2, 2003





It's worth noting that the E5600 chips should be a drop in replacement, are dirt cheap, and do support AES-NI.

I will have a couple of X5647s available soon if anyone wants them.

I would expect a tower setup like that to be louder than a typical desktop, but far quieter than a 1U or 2U screamer. You can probably swap the fans with standard ones too if that becomes an issue.

Paul MaudDib
May 3, 2006

TEAM NVIDIA:
FORUM POLICE
Would going from a W3565 to an L5640 help my idle power at all? Doing the math, I'm pulling about 120W at idle spinning 4 drives, measured at the wall. I typically figure about 5W per drive, so 100W idle system power. Or, maybe one of the low-power quads? Or if it's all the same, would going to a X5680 hurt me if it's mostly idle?

From what I've read, most of the power savings has been in faster SpeedStep, so if you are either full-idle or full-load then maybe it doesn't hurt as much?

The one thing I can't help is this poo poo HP power supply in my Z400... I don't even think it's 80+. But I'd also need to find one of those adapters that mates its non-standard 24-pin connector to a normal ATX.

DJ Commie
Feb 29, 2004

Stupid drivers always breaking car, Gronk fix car...
I'm currently running a P212 in an old Microserver N54L (Gen7) since it was crazy cheap a few years ago and had no issues saturating the SATA--II bus. It still runs Server 2012 and is only a 2x3TB Red setup. What is the next upgrade path to get SATA 6Gb capability that the drives have?

I'd hope for dual internal SAS ports so I can use the 4x256GB SSD array I got crazy cheap of SA-Mart (currently its on the crappy onboard AMD FakeRAID). I looked up the P222 but I don't think its new enough. This thing is going to be run until it dies, probably shortly since the drive have 40K hours on them.

Paul MaudDib
May 3, 2006

TEAM NVIDIA:
FORUM POLICE

DJ Commie posted:

I'm currently running a P212 in an old Microserver N54L (Gen7) since it was crazy cheap a few years ago and had no issues saturating the SATA--II bus. It still runs Server 2012 and is only a 2x3TB Red setup. What is the next upgrade path to get SATA 6Gb capability that the drives have?

I'd hope for dual internal SAS ports so I can use the 4x256GB SSD array I got crazy cheap of SA-Mart (currently its on the crappy onboard AMD FakeRAID). I looked up the P222 but I don't think its new enough. This thing is going to be run until it dies, probably shortly since the drive have 40K hours on them.

There are almost no modern mobos with onboard SAS. I won't say zero, but very few, and they are probably restricted to rackmount server hardware.

There are some that use mini-SAS ports for their SATA controllers, you can get up to roughly 12 SATA-III ports on a mobo, in a format as small as mini-ITX (with compromises).

BlankSystemDaemon
Mar 13, 2009



22 Eargesplitten posted:

Not sure what exactly GELI is, it looks like a BSD thing? How essential would that be? I really just want this thing for backups of my media and downloads and documents and my wife’s art and poo poo. I could even keep it on a separate VLAN from the WiFi and set up ssh access in case I needed to get on with my laptop without plugging in on Ethernet. Not sure why I would though, I could just keep it isolated on the Ethernet only VLAN.

I would probably be keeping it in the basement so I don’t have to give a poo poo about noise.
GELI is a GEOM class, GEOM is a modular framework in FreeBSD that handles transformations from pure disk I/O to all kinds of behaviour including features like partitioning, mirroring (plus concatenation (read: BIG/SPAN), striping, raid3 and raid4, but unfortunately not raid5, 6 or a hypothetical 7 like raidz3), crypto, compression, journaling, labeling, multipathing, mapping (very useful for embedded systems) and probably others that I'm forgetting - basically it's a part of the kernel between the disk driver and the VFS that's responsible for transforming disk I/O into behaviour that makes it possible to add all of the above features on top of disks, and they can be used peacemeal or in various combinations (so you can have crypto on top of journaling on top of mirroring with drives that are labeled /dev/bay#-serial# for easier administration, just as an example).

Nowadays, I'd say it's pretty drat important to ensure that your backups are encrypted, and having your array be encrypted too means that it's very very easy to decommision, as you simply recycle it in a bin without worrying about anyone being able to get to whatever you've stored there (financial or otherwise personal documents, for example).

BlankSystemDaemon fucked around with this message at 12:25 on Jun 24, 2018

DJ Commie
Feb 29, 2004

Stupid drivers always breaking car, Gronk fix car...

Paul MaudDib posted:

There are almost no modern mobos with onboard SAS. I won't say zero, but very few, and they are probably restricted to rackmount server hardware.

There are some that use mini-SAS ports for their SATA controllers, you can get up to roughly 12 SATA-III ports on a mobo, in a format as small as mini-ITX (with compromises).

I don't know where you're getting I'm using native SAS, its just the standard connector since it has 1 cable 4 drive support, the HBA uses it, as well as the motherboard's original controller. The P212 is currently going onto the N54L's drive cage via a SAS to SATA breakout cable, and the onboard port going through a similar breakout cable to the SSDs.

I'm just asking about what low profile 4/8x PCIe HBA would give me 6Gb SATA, and probably surplus or system pulls from eBay, and utilize the internal SAS connector. I use hardware RAID-1 on the P212, and FakeRAID on the SSDs.

Hadlock
Nov 9, 2004

Need validation of plan and/or advice. Budget is < $1000 prefer to buy through Amazon

Random background Ok quick sanity check. I have like, twenty....two? years of files I have been slowly accumulating since I was a nerd in middle school. Since college mostly just backups of digital camera photos etc and more recently cell phone photos/videos. I have 4-ish TB of data, no video no Plex, no Kodi/XBMC bullshit, no streaming security cameras, ...I guess I have ~15GB of MP3s from the MP3 craze of the early 2000s. At some point I will probably be adding baby photos/videos and want to back those up as well. Anyways, no hard performance requirements, modern technology vastly outstrips my needs.

My existing setup is this massive home built 4U (rackmount) core i5 haswell (2013?) era device (also doubled as a VM lab) with 4x 2-4TB drives, it is massive, noisy, and helped me learn windows server data stuff. Anyways it's huge and noisy and 5 years old at this point.

I need something smaller and quieter for my 4TB of data. Living in California my house is substantially smaller than the old one in Texas. Here is my plan:

Initial setup (phase I )

- Buy a Synology DiskStation DS418 for like, $370
- Buy 2 x brand new 6TB drives, one each from different manufacturers @ $165 ea = $330
- Setup the disks in SHR (Synology Hybrid RAID) array
- Rsync(?) 4TB from the old machine to the new synology diskstation

Phase II

- Format the old 2 and/or 4TB disks still in my old server
- Add them to the Synology DS418 as part of the above SHR (Synology Hybrid RAID) array
- triple zero out the old unused disks drill a hole in them
- Throw away/craigslist the old core i5, old now-dead disks

Total cost: $700-ish

I plan on connecting this thing primarily to a windows 10 laptop a couple times a week via gig-e and very periodically to a 2017-era mac laptop

Hadlock fucked around with this message at 05:32 on Jun 25, 2018

BoyBlunder
Sep 17, 2008

Hadlock posted:

Need validation of plan and/or advice. Budget is < $1000 prefer to buy through Amazon

Random background Ok quick sanity check. I have like, twenty....two? years of files I have been slowly accumulating since I was a nerd in middle school. Since college mostly just backups of digital camera photos etc and more recently cell phone photos/videos. I have 4-ish TB of data, no video no Plex, no Kodi/XBMC bullshit, no streaming security cameras, ...I guess I have ~15GB of MP3s from the MP3 craze of the early 2000s. At some point I will probably be adding baby photos/videos and want to back those up as well. Anyways, no hard performance requirements, modern technology vastly outstrips my needs.

My existing setup is this massive home built 4U (rackmount) core i5 haswell (2013?) era device (also doubled as a VM lab) with 4x 2-4TB drives, it is massive, noisy, and helped me learn windows server data stuff. Anyways it's huge and noisy and 5 years old at this point.

I need something smaller and quieter for my 4TB of data. Living in California my house is substantially smaller than the old one in Texas. Here is my plan:

Initial setup (phase I )

- Buy a Synology DiskStation DS418 for like, $370
- Buy 2 x brand new 6TB drives, one each from different manufacturers @ $165 ea = $330
- Setup the disks in SHR (Synology Hybrid RAID) array
- Rsync(?) 4TB from the old machine to the new synology diskstation

Phase II

- Format the old 2 and/or 4TB disks still in my old server
- Add them to the Synology DS418 as part of the above SHR (Synology Hybrid RAID) array
- triple zero out the old unused disks drill a hole in them
- Throw away/craigslist the old core i5, old now-dead disks

Total cost: $700-ish

I plan on connecting this thing primarily to a windows 10 laptop a couple times a week via gig-e and very periodically to a 2017-era mac laptop

This seems like a great idea to me. I'd do the same.

Steakandchips
Apr 30, 2009

"- Format the old 2 and/or 4TB disks still in my old server
- Add them to the Synology DS418 as part of the above SHR (Synology Hybrid RAID) array"


Sell the 2 and 4tbs and add 6tbs when you have more money. Always keep a cold spare 6tb disk around.

Aside from that, your plan looks good.

Adbot
ADBOT LOVES YOU

Falcon2001
Oct 10, 2004

Eat your hamburgers, Apollo.
Pillbug
How often should I consider proactively replacing disks? My NAS runs 24x7 but doesn't see a lot of day to day use. I'm running 4x Western Digital WD40EFRX 4 TB WD Red, all purchased almost exactly 4 years ago.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply