Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
IOwnCalculus
Apr 2, 2003





Cloudflare offers their CDN / caching services for free on personal accounts as well.

I don't know how on earth they justify it other than the total usage probably being a rounding error compared to their global scale, but for $free.99 and easy to implement, I'll take it.

Adbot
ADBOT LOVES YOU

Rooted Vegetable
Jun 1, 2002
Simpler alternative for straight up SFTP or SMB syncing (among others) is FolderSync.

In my case I use it to sync to a backup folder on my home server which Crashplan then backs up.

(Not necessarily the same use case as the Nextcloud talk. Still working on that)

H110Hawk
Dec 28, 2006

IOwnCalculus posted:

Cloudflare offers their CDN / caching services for free on personal accounts as well.

I don't know how on earth they justify it other than the total usage probably being a rounding error compared to their global scale, but for $free.99 and easy to implement, I'll take it.

It's incredibly limited compared to their enterprise product. I believe it's a loss leader to get you in the door, it worked for me. That being said, I'm less than impressed with the enterprise product but I've also been seriously spoiled by Akamai in the past.

IOwnCalculus
Apr 2, 2003





H110Hawk posted:

It's incredibly limited compared to their enterprise product. I believe it's a loss leader to get you in the door, it worked for me.

Oh yeah, they offer all kinds of poo poo that I'd never use for my personal stuff. But I suppose the whole experience is good enough for me to recommend them as an easy-to-use domain registrar and DNS provider and if they get *any* traction into an actual business account from the homegamer market, that's probably worth it.

rufius
Feb 27, 2011

Clear alcohols are for rich women on diets.

TraderStav posted:

I hadn't considered using Plex and/or Nextcloud for my photos/videos backup from my phone. Are there apps that mimic the google photos auto-backup feature but with NC or Plex as the destination?

Also, what function is Cloudflare serving in this setup? Dynamic DNS?

Caveat: requires QNAP device.

I use the Qfile app as a secondary backup of photos on my phone. They go into user directories that then get backed up nightly to Backblaze.

This solution works well. I’ve never tried the Plex variant though.

TraderStav
May 19, 2006

It feels like I was standing my entire life and I just sat down

rufius posted:

Caveat: requires QNAP device.

I use the Qfile app as a secondary backup of photos on my phone. They go into user directories that then get backed up nightly to Backblaze.

This solution works well. I’ve never tried the Plex variant though.

I have an Unraid server with 42TB, that qualifies, right?

Minty Swagger
Sep 8, 2005

Ribbit Ribbit Real Good
Is crash plan pro still busted/crippled? Like 5mbit or less upload into their platform? I assume that's their way to offer unlimited space but also not go out of business, but curious on if it's any better now...

rufius
Feb 27, 2011

Clear alcohols are for rich women on diets.

TraderStav posted:

I have an Unraid server with 42TB, that qualifies, right?

Yes, though you’ll need to purchase a Synology DS720+.

Rooted Vegetable
Jun 1, 2002

Rooted Vegetable posted:

Guess it's time for my thrice yearly battle with Nextcloud's docker and Traefik. Anyone got a reliable guide to getting those two working together? Ideally considering Unraid too but I think that's asking too much.

This task can drive a man mad.

You will fight it, you will curse at it, but this was the eventual fix for Traefik v1 (yes I know) setup: https://blog.linuxserver.io/2018/02/03/using-traefik-as-a-reverse-proxy-with-docker/
code:
labels:
  - "traefik.enable=true"
  - "traefik.protocol=https"
  - "traefik.port=443"
ADD THAT loving "traefik.protocol=https" LABEL

movax
Aug 30, 2008

movax posted:

OK, since I got some solid unblocked time for the first time in gently caress, 2 years or so, I started bringing up ESXi and my two project NAS boxes.

The first, simple question really, I intend to ship off to mini-PC colo land and act as a seed box / my own personal offsite storage solution. ASRock DeskMini box, two 7.68 TB SATA drives passed through to TrueNAS (via passing through the entire AHCI controller, not RDM, so no worries there). IIRC, ZFS / TrueNAS does not have a "JBOD" concept, right? So my only option for one contiguous big pool is RAID 0 / striping it? It will be a secondary/tertiary backup location, so if the stripe died, my other poo poo would have to all die at the same time. Some part of my brain keeps saying "doing a stripe for your backup location is a dumbass idea" so I figure I'd float it here for you all to tell me that it is, in fact, a dumb idea. In which case, I will probably just suck it up and have two pools, each ~7 TB in size. Or, really navel gaze, see how much data I may want to end up putting there / seeding, and just mirroring. No L2ARC or SLOG planned here.

Second question, for the NAS build I have planned for home, I need a sanity check as well. Hardware wise, I've got the following drives all piled into it:

* 8x Exos X16 via SAS3008
* 2x 860 EVO SATA SSD (Intel PCH)
* 2x WD Blue SATA SSD (intel PCH)
* 1x Samsung PM1725A 7 TB NVMe (via x8 PCIe)

Storage pool-wise, I think I will do:

* RAID-Z2 of the Exos as my main spinny storage pile
* Stripe of mirrors for the SATA SSDs for ~3 TB of redundant storage (I just have the drives already, and the case has room, so... why not?)
* Maybe whatever spare / leftover storage as a non-RAID'd scratch storage option on the PM1725.

I think I understand the SLOG better now and why it exists; the spinning pool will basically be only offered via SMB / I plan for only async writes to it, so I don't think it'll benefit from having a SLOG. For the all-flash pool, they're all SATA, so that's going to be a limiting factor in performance anyways, but if I do want to expose that pool via iSCSI for ESXi. Since this is homelab, I was thinking of just partitioning off 20 GB or so (overkill) of the PM1725A and giving it as a SLOG to the all-flash pool, and then offering up the rest as scratch storage.

L2ARC, not going to touch for either pool; I'll reserve 32 GB for TrueNAS and call it good. IIRC, SLOGs and L2ARCs can be added/removed from pools more or less at will, right? (Assuming no active writes, etc etc).

The last thing, which I realized whilst configuring is... since I'm doing an ESXi setup, I need a datastore to host VMs on. While ESXi can run off the USB stick, I'll need TrueNAS at a minimum to live somewhere, and then make sure that the VMs boot in the right order for ESXi to use the iSCSI share(s) exported by TrueNAS to turn on the other VMs (kind of a bootstrap scenario). Since I'm going to give the entire PCH AHCI controller to TrueNAS, ESXi will have no disks to use for datastore.

So, I have one PCIe x1 3.0 slot left on my mobo... I can just chuck a P31 or some NVMe drive onto it to use as a datastore + swap. Or, I can dig up some HW RAID controller for RAID 1 of NVMe drives. If I keep the other VMs on the iSCSI pool exported by the TrueNAS VM, then I don't really need much storage here. And, if I don't bother RAIDing the NVMe drive, with TrueNAS's + ESXi config backed up elsewhere, it's just some minor downtime for my home machine while I replace it.

I see now why real enterprise environments w/ SANs are used to solve the ESXi datastore need..., or you have a hardware RAID controller in the ESXi host that presents storage for usage.

OK, so for the ESXi datastore question... I think that just using a spare NVMe drive I have + adapter in the x1 slot will be fine, because I can combine it with https://github.com/lamw/ghettoVCB and just back up that single drive's contents to my storage pool. Chances of single failure on an incredibly already low utilized SSD are low (I'll probably use a Samsung one I have lying around), I will backup the VM configs separately, but I will also have raw snapshots of all the poo poo I care about on that volume thanks to ghettoVCB. So, I think that question is answered.

On the first... maybe I actually should just run two pools, and then handle splitting up files at a layer up. Maximizes storage and I just deal with the potential inconvenience of not having only one mount point for all my :filez: I think the choices for me are:

* 2 separate pools, 14 TB storage, deal with, at some point, poo poo straddling two pools (I guess I could deal with that in the future with some internal symlink fuckery)
* mirrored, 7 TB storage
* RAID 0, 14 TB storage, everything in "one spot"

movax
Aug 30, 2008

Also, incredibly stupid question that really says something about my priorities and usage of a NAS... what software is the current thread favorite for workstation backup, Windows and/or Linux? I've used the QNAP QBak at my parents' place, but my usage model has nominally been just storing media / files on the NAS and sharing them, not actually backing up files to it.

I think mostly just something that will copy folders A, B and C to NAS (in this case, my other other NAS which is a QNAP), and then I'll just lean on snapshots on the NAS side for starters is probably all I need.

fletcher
Jun 27, 2003

ken park is my favorite movie

Cybernetic Crumb

movax posted:

Also, incredibly stupid question that really says something about my priorities and usage of a NAS... what software is the current thread favorite for workstation backup, Windows and/or Linux? I've used the QNAP QBak at my parents' place, but my usage model has nominally been just storing media / files on the NAS and sharing them, not actually backing up files to it.

I think mostly just something that will copy folders A, B and C to NAS (in this case, my other other NAS which is a QNAP), and then I'll just lean on snapshots on the NAS side for starters is probably all I need.

For Windows to NAS backups I am just using Microsoft SyncToy & the Task Scheduler. Then from there it's NAS to B2 + Snapshots

Cold on a Cob
Feb 6, 2006

i've seen so much, i'm going blind
and i'm brain dead virtually

College Slice

movax posted:

Also, incredibly stupid question that really says something about my priorities and usage of a NAS... what software is the current thread favorite for workstation backup, Windows and/or Linux? I've used the QNAP QBak at my parents' place, but my usage model has nominally been just storing media / files on the NAS and sharing them, not actually backing up files to it.

I think mostly just something that will copy folders A, B and C to NAS (in this case, my other other NAS which is a QNAP), and then I'll just lean on snapshots on the NAS side for starters is probably all I need.

I’m using duplicacy (cli version not webgui version). Supports a lot of targets. Easy to integrate into a maintenance script where I do a few other tasks before and after my backup.

I was using duplicati but all the restore horror stories scared me off.

Warbird
May 23, 2012

America's Favorite Dumbass

Rsync all day erry day

Nitrousoxide
May 30, 2011

do not buy a oneplus phone



Well, I got my TrueNAS set up with my laptop in a couple of those shingled drives. Man is it slow. It’s going to take about six or seven hours to do an 80 GB time machine back up After I set up a pool for that.

I think I’ll just use this for Time Machine back ups. It’s way too slow to be any sort of media server. Later in the year I’ll probably build a proper NAS and port all the data I have on this one over.

DrDork
Dec 29, 2003
commanding officer of the Army of Dorkness
Do note that reading from SMR drives is generally just as fast as from CMR drives, so it might work just fine for a media server. It's just writing large amounts of data that shits all over a SMR drive's day. Not sure if you mean your laptop is the NAS itself, or if you just mean pushing Time Machine backups from your laptop to the NAS, but if it's the former, again reads are a lot less demanding than writes, so you might be pleasantly surprised by its performance once you get the data over there the first time.

But yeah, bulk transfers to under-powered NASes running SMR drives is gonna be a bad day.

Nitrousoxide
May 30, 2011

do not buy a oneplus phone



The laptop itself isn't really under powered. It's barely even moving the needle on the CPU and it has plenty of RAM.



The disc write speeds are just really dreadful.


The network speeds aren't anything to write home about either. I don't think it should be saturating a 5 GB USB 3 port. So I'm not sure why it's running so incredibly slowly. The only thing I can think of is that the drive itself is the limiting factor here.



My guess is it's just not playing very nice with either the file system on the drives or how the USB interface is working with TrueNAS


Edit: I found a speed test plug-in and installed that. It's definitely not an issue with the network interface or usb.
So it's got to be the drives

Nitrousoxide fucked around with this message at 05:11 on Jan 5, 2021

UCS Hellmaker
Mar 29, 2008
Toilet Rascal
So planning on building a Nas system to put a plex server and backup server on. Got a fractal 804 case on the way so its not to large. Plan on using integrated graphics for the little itll be hooked up, but trying to figure out the rest right now. I get that I will need an exapansion card to get all the SATA ports I need. Is there anything else I need to consider?
probably 8 gigs of ram with a small ssd for an operating system.

Then with hard drives, I see that people are just using external drives that can be cheaper but no warranty, is there anyway to tell what ones can be ripped from the case and what ones cant? Otherwise I'll go with old reliable toshibas probably. Goal is to have a nice plex server/and backup server that then offloads to backblaze.

BlankSystemDaemon
Mar 13, 2009



movax posted:

Also, incredibly stupid question that really says something about my priorities and usage of a NAS... what software is the current thread favorite for workstation backup, Windows and/or Linux? I've used the QNAP QBak at my parents' place, but my usage model has nominally been just storing media / files on the NAS and sharing them, not actually backing up files to it.

I think mostly just something that will copy folders A, B and C to NAS (in this case, my other other NAS which is a QNAP), and then I'll just lean on snapshots on the NAS side for starters is probably all I need.
Bacula is a favorite of mine for anything that needs to be cross-platform.

Matt Zerella
Oct 7, 2002

Norris'es are back baby. It's good again. Awoouu (fox Howl)

UCS Hellmaker posted:

So planning on building a Nas system to put a plex server and backup server on. Got a fractal 804 case on the way so its not to large. Plan on using integrated graphics for the little itll be hooked up, but trying to figure out the rest right now. I get that I will need an exapansion card to get all the SATA ports I need. Is there anything else I need to consider?
probably 8 gigs of ram with a small ssd for an operating system.

Then with hard drives, I see that people are just using external drives that can be cheaper but no warranty, is there anyway to tell what ones can be ripped from the case and what ones cant? Otherwise I'll go with old reliable toshibas probably. Goal is to have a nice plex server/and backup server that then offloads to backblaze.

The rule of thumb for shucking nowadays seems the be anything greater than 6GB will not be shingled slow drives.

OS and RAID tech (ZFS, madam, unRAID, BTRFS)/grouping is another to consider.

If you want to be able to mix and match drives and have a nice docker based software setup for Plex, I recommend unRAID is it's pretty easy to get going and generally hands off once set up, but you can roll your own solution with windows, linux or run TruNAS/ZFS. If you go ZFS, you're going to see a lot of talk about ECC RAM, this isn't essential for home use, its more of a nice to have.

Also unRAID costs money. If that's a factor.

UCS Hellmaker
Mar 29, 2008
Toilet Rascal
Finding out that there's literally no amd CPUs around me or on amazon is kinda sucky that fit my needs :negative:

Yeah I'm gonna be mixing and matching, goal is the plex server as one bay, then for now the backup in the other. Then have those ready to ploop to backblaze

CopperHound
Feb 14, 2012

movax posted:

Also, incredibly stupid question that really says something about my priorities and usage of a NAS... what software is the current thread favorite for workstation backup, Windows and/or Linux? I've used the QNAP QBak at my parents' place, but my usage model has nominally been just storing media / files on the NAS and sharing them, not actually backing up files to it.

I think mostly just something that will copy folders A, B and C to NAS (in this case, my other other NAS which is a QNAP), and then I'll just lean on snapshots on the NAS side for starters is probably all I need.
Maybe not the best use case for backups, but I've been using syncthing with file versioning set up on the NAS.

Rooted Vegetable
Jun 1, 2002
To add in, Unraid very nicely hits the sweet spot of stability, ability to tinker and flexibility. The latter is in large part provided by VMs and Docker containers.

I've been using it as a steady Plex etc server for some time, with VMs for various tasks, usually started on demand. Just last night I was using Steam Link on a lovely laptop to connect to a VM on the Unraid server with a GPU and happily playing what I wished.

Since I had a kid and can't spend every free hour on this stuff, it's been perfect for what I've wanted.

It's been well worth the $89 for Unraid Plus (Basic is cheaper and Pro is not much more).

PRADA SLUT
Mar 14, 2006

Inexperienced,
heartless,
but even so
If I'm switching from a Synology DS218+ to a 220+, can I just swap the drives into the new case, or is there some sort of setup I need to do?

Axe-man
Apr 16, 2005

The product of hundreds of hours of scientific investigation and research.

The perfect meatball.
Clapping Larry
You can just slot in the drives into the new unit, and then once the power light goes solid on the new unit, use find.synology.com to find it on the network and do your migration.

You shouldn't have any problems.

TraderStav
May 19, 2006

It feels like I was standing my entire life and I just sat down

Rooted Vegetable posted:

To add in, Unraid very nicely hits the sweet spot of stability, ability to tinker and flexibility. The latter is in large part provided by VMs and Docker containers.

I've been using it as a steady Plex etc server for some time, with VMs for various tasks, usually started on demand. Just last night I was using Steam Link on a lovely laptop to connect to a VM on the Unraid server with a GPU and happily playing what I wished.

Since I had a kid and can't spend every free hour on this stuff, it's been perfect for what I've wanted.

It's been well worth the $89 for Unraid Plus (Basic is cheaper and Pro is not much more).

Adding another voice to Unraid being a great solution. Straightforward and flexible.

Nitrousoxide
May 30, 2011

do not buy a oneplus phone



Aside from my drive speed issues, which I'm sure are hardware related and not the fault of the OS, I have been extremely happy with TrueNAS.

It was quick and easy to set up initially, the user and group permissions are pretty self-explanatory and sane. The jail-based system works similarly to docker, and you can install docker if you want to use that instead.

Setting up docker in TrueNAS core
https://youtu.be/XBVjuwgz0Cg

There's a bunch of built-in stuff that could be really useful depending on what you're deploying it for. Like you don't need a separate module to do active directory or snapshots or home directory (especially useful if you're deploying this for use as a virtual machine server. Home directory is really nice because it'll let it automatically create individual share directories for users you give access to a pool rather than having to manually create them one by one)

For me and my relatively straightforward use case. I'm able to set up all of the servers that I would want to through their officially supported plugins and the community plugins. the official list of community plugins actually include a lot of things that might be useful for you if your purpose includes :filez:



Oh, and it uses the ZFS file system which is extremely easy to expand with additional storage if you need it later.

Edit: TrueNAS will also be releasing TrueNAS Scale which is linux-based rather than bsd-based and will have full docker support out of the box rather than having to run a VM that runs docker like you do on the current one. The release version of it should be out later this year.

Nitrousoxide fucked around with this message at 22:30 on Jan 5, 2021

fletcher
Jun 27, 2003

ken park is my favorite movie

Cybernetic Crumb

Nitrousoxide posted:

The laptop itself isn't really under powered. It's barely even moving the needle on the CPU and it has plenty of RAM.

Edit: I found a speed test plug-in and installed that. It's definitely not an issue with the network interface or usb.
So it's got to be the drives



How is the speed test proof that it's not an issue with USB?

What are you using to connect the drives to the laptop?

UCS Hellmaker
Mar 29, 2008
Toilet Rascal
will say that if there could be an updated op to help with basics it be awesome. there's a ton of good info just reading the last ten pages or so has given me for how to do this.

Nitrousoxide
May 30, 2011

do not buy a oneplus phone



fletcher posted:

How is the speed test proof that it's not an issue with USB?

What are you using to connect the drives to the laptop?

Yesterday I SSH’d into the server and did a connection test between it and one of my devices on my local network and it was extremely slow. So it’s probably something to do with the the network interface actually.

I'm using a USB Ethernet adapter. looking through some of the discussions about the model I chose seems like people who were using BSD have complained about it not supporting it at full speed. So that's probably my problem.

Nitrousoxide fucked around with this message at 14:47 on Jan 6, 2021

Nitrousoxide
May 30, 2011

do not buy a oneplus phone



I ended up experimenting with openmediavault as it's a Linux-based system. My hope was that it would have better driver support for my ethernet USB dongle. Luckily it did and now I'm getting full transfer speeds that I would expect. It probably wisely doesn't let me set up a raid with my USB drives so I've just set up one drive as a media source and added a second drive there which does periodic back ups from the first to the second.

It has full docker and comes with yacht and portainer which makes installing them pretty easy. Overall this seems like a pretty good set up for the hardware that I have in hand so I don't have to spend anything to get it up and running an experiment with it.

Nitrousoxide fucked around with this message at 00:46 on Jan 9, 2021

Rooted Vegetable
Jun 1, 2002
Glad to hear OMV is still good. I was doing something similar with an old laptop and collection of USB drives years ago

Astryl
Feb 1, 2005

"15,000 hours of Diablo II isn't that much, dweeb."

Finally got everything working properly.

3960X Threadripper unRAID daily driving a Big Sur VM w/ a 590, and I have a separate gaming VM with a 570 passthrough running in parallel. So, next upgrades will be an eventual 4th gen threadripper and upgrading storage from 8TB drives to 16TB drives. So the question is, is it worth selling the old 8TB drives? (WD white labels) What sort of precautions should I take if I /do/ sell them? I have a bit of time before I need to upgrade since I still have ~60TB free.

DrDork
Dec 29, 2003
commanding officer of the Army of Dorkness
8TB Whites don't sell super briskly, but eBay has them going generally for around $100/ea right now. So they'd probably be worth tossing up there (or SA Mart or whatnot), yeah. But of course prices will drop over time unless Thailand floods again or something.

If you're going to sell them, do make sure you do a complete wipe of the drive, not just a quick format. Frankly a single zeroing pass is sufficient to protect your data from people likely to buy it (no, no one is taking an electron microscope to a drive bought off 420stonerdude6969), but you could always do a couple of passes if you've got time to spare.

GreenBuckanneer
Sep 15, 2007

Considering a DS418 to be a plex server with four 10tb drives in raid 0, but the primary criterea would be if it can play 4k h265 video to an amazon 4k fire stick. Does anyone have experience with this?

This is like $1200 of hardware so trying to see what's worthwhile :thunk:

I currently do this from my own PC just fine, mostly wanting to offload this into it's own thing.

PRADA SLUT
Mar 14, 2006

Inexperienced,
heartless,
but even so
I can play 4K with a 218+ to an appletv

H110Hawk
Dec 28, 2006

GreenBuckanneer posted:

drives in raid 0

This is super dumb. You're going to loose 100% of your files every time a disk dies. Either do it as a jbod or do it as SHR/SHR2. As described you might as well just slap a unshucked 10tb disk onto your computer as USB.

DrDork
Dec 29, 2003
commanding officer of the Army of Dorkness
RAID0 for a device limited to a 1GbE port also makes no sense, since even a single drive can saturate that without trouble. Just use a JBOD pool if you really need the space, or SHR/RAID5 if you want at least a little safety.

Also bonus lols for Synology's speed charts all quoting LAG throughput speeds that I'd guess no more than like 1% of their user base will ever see.

GreenBuckanneer
Sep 15, 2007

H110Hawk posted:

This is super dumb. You're going to loose 100% of your files every time a disk dies. Either do it as a jbod or do it as SHR/SHR2. As described you might as well just slap a unshucked 10tb disk onto your computer as USB.

Well I mean the viable alternative is raid 10 but then that's half the storage space. I'd probably go for raid 5 because it's unlikely for more than 1 drive to fail at a time, although I have seen entire arrays go down at once, it's uncommon enough in raid 5 that businesses have sued the supplier over it. For a person like me it's fine, i'd be backing up anything anyways

JBOD sounds fine, actually

Adbot
ADBOT LOVES YOU

H110Hawk
Dec 28, 2006

GreenBuckanneer posted:

Well I mean the viable alternative is raid 10 but then that's half the storage space. I'd probably go for raid 5 because it's unlikely for more than 1 drive to fail at a time, although I have seen entire arrays go down at once, it's uncommon enough in raid 5 that businesses have sued the supplier over it. For a person like me it's fine, i'd be backing up anything anyways

JBOD sounds fine, actually

Yeah it's raid0 that makes my eyes twitch. You would get only downsides and 0 upside. Jbod is at least the same odds of a single disk dying with 1/4 the impact. I would do shr1 and call it a day. You're shucking disks right?

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply