|
I'm gonna be buying a Intel NUC (DC53427HYE) just for messing around with ESXi, mainly hosting my own domain on a Debian VM. Therefore I am wondering: Will a Synology NAS with WD Reds be fast enough for my datastores? I will most likely buy a MSATA SSD to use in the NUC. I'm guessing that there are loads of details I should be asking about on this subject  Edit: Maybe I would be better off backing up my ESXi host/VMs to a external USB 3.0 HDD? chrome taco fucked around with this message at 01:55 on Dec 16, 2013 |
#
?
Dec 16, 2013 01:08
|
|
|
|
| # ? May 10, 2024 09:57 |
|
|
chrome taco posted:I'm gonna be buying a Intel NUC (DC53427HYE) just for messing around with ESXi, mainly hosting my own domain on a Debian VM. Therefore I am wondering: Will a Synology NAS with WD Reds be fast enough for my datastores? I will most likely buy a MSATA SSD to use in the NUC. I'm guessing that there are loads of details I should be asking about on this subject I have a NUC just like the one you'll be buying and it works perfectly as a HTPC machine (running OpenELEC), primarily because of its HD4000 iGPU that's on-die since VAAPU takes a lot of the load that'd otherwise be on the CPU, meaning the machine doesn't get nearly as hot and noisy as it otherwise might. I honestly don't know how well it does as an ESXi host. BlankSystemDaemon fucked around with this message at 20:41 on Dec 16, 2013 |
|
#
?
Dec 16, 2013 20:31
|
|
|
If I recall, the 1g/tb RAM suggestion was for zfs dedupe. For general use you don't need that much RAM, 8 gigs should be fine. If you don't evenly match the RAID level to the number of drives the worst thing that happens is that one or more of the drives has to write a tiny bit of data the other drives didn't. Or, put another way, writing 3 stripes of data on a 3 drive RAIDZ takes as long as writing 4 stripes of data would. It's only of concern to really specific workloads (like writing huge numbers of tiny files and fsyncing each one) and not at all a concern to most users.
|
|
#
?
Dec 17, 2013 03:50
|
|
|
AtmaHorizon posted:As far as I understand from the documentation it is made of multiple RAID6 arrays. Thats if you have uneven drives. If you have 4 even drives and make an SHR2 array its just a single raid 6. The console even shows it as a raid 6 during bootup. What synology lets you do, if you have 4 drives and 2 are larger, you could make another array out of the extra space on the 2 drives, but you dont have too.
|
|
#
?
Dec 17, 2013 04:23
|
|
|
Avago just acquired LSI for $6.6 billion in cash. Will be interesting to see what direction they take the storage business in; I'm used to Avago's fiber products (used their SFPs a lot) and know they do a lot of RF stuff (LTE, etc for iPhone).
|
|
#
?
Dec 17, 2013 06:59
|
|
|
Don Lapre posted:Thats if you have uneven drives. If you have 4 even drives and make an SHR2 array its just a single raid 6. The console even shows it as a raid 6 during bootup. Thanks for confirming this. I'll stick to simple RAID6 then. Don Lapre posted:What synology lets you do, if you have 4 drives and 2 are larger, you could make another array out of the extra space on the 2 drives, but you dont have too. Maybe it is possible only from cli, because I have exact situation (2x small, 2x large drives) and gui doesn't allow creating a new volume (greyed out).
|
|
#
?
Dec 17, 2013 08:56
|
|
|
Sagacity posted:Doh, of course, thanks. I would prefer to go ITX, but I can't seem to find any recent motherboards that support ECC. There's one from ASRock (E3C226D2I) that also supports 6 SATA drives but it's impossible to find here in Holland. http://www.supermicro.nl/products/motherboard/Core/index.cfm#FCBGA1023 Also - I can recommend the Node 304 chassis. My earlier Eolize chassis was a bit problematic, so I moved the hardware over to a 304 and added an LSI 8-port SAS controller. The ability to use a standard ATX PSU instead of buzzy FlexATX ones is a huge plus and you can also fit a fairly large passive CPU cooler because the disk brackets don't interfere too much.
|
|
#
?
Dec 17, 2013 09:27
|
|
|
firehawk posted:Supermicro has a series of QM77-chipset Mini-ITX boards with ECC support but no CPU socket as they include mobile i7 CPU soldered onboard. Quad-port Intel GbE is a nice bonus, though. Edit: The ASRock C2750D4I seems very interesting as well (or the 4 core version). Sagacity fucked around with this message at 12:19 on Dec 17, 2013 |
|
#
?
Dec 17, 2013 11:35
|
|
|
Ninja Rope posted:If I recall, the 1g/tb RAM suggestion was for zfs dedupe. For general use you don't need that much RAM, 8 gigs should be fine. Needless to say, you don't want to enable dedup until you've done testing on a non-production pool, simulated it with zdb -S poolname and possibly made a test-enviroment on a completely seperate setup (if possible). EDIT: The thing about deduplication I forgot to mention is that you don't want it unless you have considerable amount of duplicate data because of the increased amount of cpu cycles and memory it takes. One of the few places where I'd definitely be enabling deduplication is on a zfs based iSCSI SAN (FreeBSD on multiple machines with CARP for HAST) which stores multiple ESXi images for virtualization. EDIT2: ↓ Well, Google (in particular the FreeBSD handbook and wiki, Solaris(Internals - when it works), and FreeNAS wiki) is your friend. The best thing to do is to setup VirtualBox and just play around with FreeNAS until you feel comfortable. BlankSystemDaemon fucked around with this message at 17:31 on Dec 17, 2013 |
|
#
?
Dec 17, 2013 12:59
|
|
|
D. Ebdrup posted:No, 1GB/1TB is a general rule for L2ARC when you're using anything but mirroring. FreeBSD good-practices for zfs deduplication mention 5GB/TB while a SolarisInternals blog mentioned 2GB/TB (which I can't find right now). If you want to know exactly how much memory, in addition to the normal L2ARC requirements as mentioned above, run zdb -b poolname and multiply that number with 320 as each entry in the dedup table takes up ~320B and it does dedup per block (each block can be a variable size between 512B and 128KB). Goddamn I have so much to learn. I have no freaking clue what any of that meant.
|
|
#
?
Dec 17, 2013 15:02
|
|
|
D. Ebdrup posted:No, 1GB/1TB is a general rule for L2ARC when you're using anything but mirroring. Keep in mind a lot of these rules (excluding dedupe) are based on enterprise / heavy workload environments; ZFS in the home is still a relatively new thing. I've currently got only 10GB allocated to my NAS4Free VM and an array with 13TB usable / drat near 20TB raw, and yet performance is just fine:  I do have an SSD added in as L2ARC (I added it in an earlier build as a failed attempt to make dedupe not suck rear end, and I have nothing better to do with a 60GB SSD) but ultimately in the world of "streaming video files around the house" it doesn't make much of a difference.
|
|
#
?
Dec 17, 2013 17:39
|
|
|
Caged posted:The WD My Cloud range are meant to be not poo poo any more. Anyone else have experience with these? I can follow directions but am otherwise stupid and looking for something that's relatively plug and play.
|
|
#
?
Dec 17, 2013 18:23
|
|
|
builds character posted:Anyone else have experience with these? I can follow directions but am otherwise stupid and looking for something that's relatively plug and play. I have a 2tb one. Its about as simple as it gets: I plug the thing in, put my files on it, and access them whenever I want. Its plugged into my router and streams to my HTPC all the time with no problems. There is software to link two of them together to back things up, but I haven't gotten into it. My only complaint is that the latest firmware causes it to freeze up when I plug in an USB3 hard drive, but that's supposed to be addressed in the next update. Honestly if you want something that plug and play with no extra features, these things aren't bad.
|
|
#
?
Dec 17, 2013 19:19
|
|
|
one1two2three3 posted:I have a 2tb one. Its about as simple as it gets: I plug the thing in, put my files on it, and access them whenever I want. Its plugged into my router and streams to my HTPC all the time with no problems. There is software to link two of them together to back things up, but I haven't gotten into it. My only complaint is that the latest firmware causes it to freeze up when I plug in an USB3 hard drive, but that's supposed to be addressed in the next update. Honestly if you want something that plug and play with no extra features, these things aren't bad. Sounds pretty good. Can you download software and access remotely? i.e. can I use it for family to put pictures on?
|
|
#
?
Dec 17, 2013 19:50
|
|
|
builds character posted:Sounds pretty good. Can you download software and access remotely? i.e. can I use it for family to put pictures on? They advertise these features, like a DIY Drop-Box, but I haven't personally set it up. It also looks like they have apps for android/ios.
|
|
#
?
Dec 18, 2013 00:28
|
|
|
I read a review online somewhere that said you can share folders with people just by sending them a code to type into their app, so you can make a photos folder and send gramps the code to tap into the iPad and it sorts the rest out, no loving around with user accounts.
|
|
#
?
Dec 18, 2013 00:46
|
|
|
D. Ebdrup posted:No, 1GB/1TB is a general rule for L2ARC when you're using anything but mirroring. FreeBSD good-practices for zfs deduplication mention 5GB/TB while a SolarisInternals blog mentioned 2GB/TB (which I can't find right now). If you want to know exactly how much memory, in addition to the normal L2ARC requirements as mentioned above, run zdb -b poolname and multiply that number with 320 as each entry in the dedup table takes up ~320B and it does dedup per block (each block can be a variable size between 512B and 128KB). Is the dedupe table size bounded at all? I recall the table wasn't always bounded (if it is now) and could eventually grow to exhaust the KVA regardless of how much RAM you have, making it unstable for all but the most predictable workload. Has that been fixed in FreeBSD?
|
|
#
?
Dec 18, 2013 01:37
|
|
|
Thanks to this thread, I finally got around to buying a Synology Diskstation DS213j - it's really nice. I didn't think the WebOS (or whatever its proper name is) would be both so complex and user-friendly. Thanks, thread!  One quick question though. In the User's Manual under rsync, there is literally only one line - quote:Use rsync for Linux Obviously this means that Linux workstations can rsync their files to the DS, but my question is this: Can the DS 213j rsync its entire backups offsite somewhere? If so can it do it automatically ala cron jobs or something similiar? In either case, how? 
|
|
#
?
Dec 18, 2013 20:24
|
|
|
There should be a Backup and Restore option in the Control Panel. You can create rsync jobs in that.
|
|
#
?
Dec 18, 2013 20:40
|
|
|
Yes its under backup and restore edit: also i highly recommend Xpenology. takes 5 minutes to setup and works exactly like a real syno unit. http://xpenology.com/forum/viewforum.php?f=2 Don Lapre fucked around with this message at 21:27 on Dec 18, 2013 |
|
#
?
Dec 18, 2013 21:24
|
|
|
Ninja Rope posted:Is the dedupe table size bounded at all? I recall the table wasn't always bounded (if it is now) and could eventually grow to exhaust the KVA regardless of how much RAM you have, making it unstable for all but the most predictable workload. Has that been fixed in FreeBSD? the open-zfs.org wiki posted:You can estimate an upper bound on the number of unique blocks by dividing the amount of storage you plan to use on each dataset (taking into account that partial records each count as a full recordsize for the purposes of deduplication) by the recordsize and each zvol by the volblocksize, summing and then multiplying by 512-bytes. Also the above implies something I forgot to mention in earlier posts: you should only use dedup on datasets which you know contain a lot of duplicate blocks, not your entire pool - in the example I used earlier with ESXi virtualization images, you'd store the initial image on a dataset with dedup and have it save configuration, snapshots (preferably as reverse delta for fast restoration) and other things in other datasets. EDIT: ↓ Yes, XPEnology is for rolling your own Synology setup on non-Synology hardware. I know someone who BlankSystemDaemon fucked around with this message at 22:36 on Dec 18, 2013 |
|
#
?
Dec 18, 2013 22:11
|
|
|
Don Lapre posted:edit: also i highly recommend Xpenology. takes 5 minutes to setup and works exactly like a real syno unit.
|
|
#
?
Dec 18, 2013 22:21
|
|
|
^^^^ That's what I'm thinking, but I honestly have no clue  Don Lapre posted:Yes its under backup and restore Thanks for this. I was derping pretty hard. I was searching for Backup and Restore in the Control Panel  Also, curious about Xpenology. What makes it superior?
|
|
#
?
Dec 18, 2013 22:21
|
|
|
Xpenology is rolling your own Synology box. The benefit is for something like $500 you can have an i3 synology box with 8gb ram and 6+ sata connections. Where if you bought actually synology hardware thats gonna get you an ARM or an atom based box. It works exactly like a real synology unit. Im using an h87i-plus mini ITX board, 8gb ddr3 (16 max), i3-4330 haswell cpu, silverstone psu, in a Lian Li q25b case which has 5 hot swap bays. I can fit 8 total drives in the case, 5 of them hot swappable. Before rebates its about $500. It even offered to migrate the disks out of my real synology unit the the xpenology unit, which i would have done if i wasn't going to move to an SHR2 array. You use win32disk imager to write a img file to a usb stick, boot off the usb stick then it works just like setting up a real unit. Run synology assistant from a pc or mac, give it the .pat file in the archive on the link i posted, and now you have your own built DSM box. Even runs the latest version. The only Janitoring i had to do, was when updating to 4.3-3810 update 2 from 4.3-3810 i had to copy and paste a script before clicking update. The cheapest i3 synology units are something like $2500-$3000 and max at 8gb ram. Don Lapre fucked around with this message at 22:53 on Dec 18, 2013 |
|
#
?
Dec 18, 2013 22:46
|
|
|
Don Lapre: What kind of RAID configuration do you use?
|
|
#
?
Dec 18, 2013 23:25
|
|
|
I'm guessing a lot of what it adds to the original Synology 'firmware' is extra hardware support then. Pretty cool, even though I probably won't ever need anything of that caliber. Thanks dude.
|
|
#
?
Dec 18, 2013 23:26
|
|
|
Sagacity posted:Don Lapre: What kind of RAID configuration do you use? In Xpenology? You dont do raid on the hardware. DSM handles it all like on a regular unit. I just set the sata controller to AHCI and DSM sets up the array just like on a synology unit. Other than making the USB stick setting up Xpenology is the same as settings up a Synology device. Im using SHR2 btw so i get 2 drive redundancy. I have 4 3tb reds right now. Flipperwaldt posted:I'm guessing a lot of what it adds to the original Synology 'firmware' is extra hardware support then. Exactly, its just more linux drivers included. Don Lapre fucked around with this message at 23:32 on Dec 18, 2013 |
|
#
?
Dec 18, 2013 23:28
|
|
|
I bought a 4TB Seagate external drive to serve as a backup point for my NAS, but when my roommate and I tried to format the drive using EXT4 I got an error message that I couldn't erase sector 0 and it resulted in a short write and failed out. Anyone have any suggestions on where to find a resolution so I can set up a cron job to back up my FreeNAS server to the external disk?
|
|
#
?
Dec 19, 2013 01:44
|
|
|
I have what are probably babby's first questions about storage and backups but would appreciate some goon insight. I've got an iMac with pretty much my whole life on it. There's movies, music, family photos dating back to 2002, documents, everything. At the moment I've got a half-arsed backup process using two external hard drives which I manually drag and drop all my stuff onto every couple of months but this is messy, time consuming and just a general hassle. So over Christmas I have been tasked with sorting my poo poo out. My first step is to get an external hard drive to run carbon copy cloner so that I have weekly backups of my iMac. First question is which hard drive should I get? I'm figuring 2TB would be good as I can partition it and backup my laptop as well which I will be using solely as a music production machine. Bonus points if the hard drive you recommend is available from this website. Secondly, is carbon copy cloner the best backup software for Mac? My Dad and his family use it and he has not had any complaints but he's also not exactly on the bleeding edge for tech. The next part of the plan is to get a carbonite account to backup all my important personal documents and family photos to ~the cloud~. What's the general goon opinion on carbonite or other cloud-based backup solutions? Once I've got my machine on a regular backup schedule and got some cloud storage for my irreplaceable stuff then I'm planning to get a Synology box and transfer all the stuff (movies, music, etc) off my current external drives and use the Synology box as a media server / storage machine. Is this a reasonably sensible plan of attack or am I doing things arse-backwards?
|
|
#
?
Dec 19, 2013 02:14
|
|
|
cyberia posted:Is this a reasonably sensible plan of attack or am I doing things arse-backwards?
|
|
#
?
Dec 19, 2013 02:22
|
|
|
cyberia posted:I have what are probably babby's first questions about storage and backups but would appreciate some goon insight. Why aren't you using time machine?
|
|
#
?
Dec 19, 2013 02:24
|
|
|
Flipperwaldt posted:I'm not sure what the intermediate step of getting a 2TB external drive is for. Mainly because I can't afford a Synology box right now and I want to start backing up my computer on a weekly basis rather than haphazardly throwing whatever I've created / downloaded onto a hard drive every month or so. Also partially for redundancy because I figure even when I get a proper server setup it can't hurt to have an image of my OS on a separate drive just in case the server ever experiences a catastrophic failure. Once I can afford a Synology box I want to use it to store the ~3TB of movies, music and photos I have so I can actually access them without loving around digging drives and cables out of boxes every time I want to watch a movie or listen to an old album while also continuing to backup my data. Don Lapre posted:Why aren't you using time machine? The main reason is that Carbon Copy Cloner offers incremental backups and last time I had a look at Time Machine it didn't appear to have that functionality.
|
|
#
?
Dec 19, 2013 02:45
|
|
|
I have 3x2TB WD Reds in my Storage Pool My virtual disk G: is only a 2-way mirror, I am getting a little over half a gigabit download speeds over the network If I create a new 3-way mirror and copy everything over to the new 3-way mirror virtual disk, should I see speed improvements? Trying to create a 3-way mirror, I am only getting the option to do a 2-way mirror  After I create the mirror I check to see if it is a 2 or 3-way mirror, it is showing 2 
|
|
#
?
Dec 19, 2013 07:49
|
|
|
Don Lapre posted:In Xpenology? You dont do raid on the hardware. DSM handles it all like on a regular unit. I just set the sata controller to AHCI and DSM sets up the array just like on a synology unit. Other than making the USB stick setting up Xpenology is the same as settings up a Synology device.  I'd also like to run ESXi on that, but I think both Xpenology and FreeNAS are not that happy about being hosted in a VM (unless you don't mind SMART not working and so on).
|
|
#
?
Dec 19, 2013 09:26
|
|
|
Don Lapre posted:Im using an h87i-plus mini ITX board, 8gb ddr3 (16 max), i3-4330 haswell cpu, silverstone psu, in a Lian Li q25b case which has 5 hot swap bays. I can fit 8 total drives in the case, 5 of them hot swappable. Before rebates its about $500. It even offered to migrate the disks out of my real synology unit the the xpenology unit, which i would have done if i wasn't going to move to an SHR2 array. That sounds really nice. Do you have any idea what the power consumption is like? Does Standby and Wake-on-LAN work it would with a real synology box? A $500 machine running the Synology OS with 8 drive slots and possibly enough grunt to do realtime 1080p transcoding almost sounds too good to be true. I think I’ll wait until the new 5.0 version of DSM is out in January. Hopefully they can also port that to run on regular hardware. eames fucked around with this message at 12:33 on Dec 19, 2013 |
|
#
?
Dec 19, 2013 12:27
|
|
|
eames posted:That sounds really nice. Do you have any idea what the power consumption is like? Does Standby and Wake-on-LAN work it would with a real synology box? Couldn't tell you about those offhand but i could certainly test them, well i cant test power consumption since i dont have a killawatt. Also 1080p transcoding appears to be working fine for me going to a roku box.
|
|
#
?
Dec 19, 2013 15:51
|
|
|
Hadlock posted:If I create a new 3-way mirror and copy everything over to the new 3-way mirror virtual disk, should I see speed improvements? As far as speed improvements go, load balancing in mirrors, both LVM and Storage Spaces, happens per thread ID. A single thread reads from a single disk (logical one in this case, but the IO all hit the same physical disk). If multiple threads are reading from the same disk, each thread getting its own disk to work with, then the IO gets balanced across them. It all depends on how SMB handles this, as in, if everything's handled by a single thread, you won't see much. Also, if the two way mirror is spread across the three disks (I suppose it should, creating a mongohuge file and watching the usage rise would verify this), you'll probably get slightly better performance than when you've a three way mirror, since there's 1.5 disks per side of the mirror involved. Combat Pretzel fucked around with this message at 16:16 on Dec 19, 2013 |
|
#
?
Dec 19, 2013 16:12
|
|
|
Sagacity posted:Ah, interesting. The reason I ask is because I'm still unsure whether to go for something like Xpenology or using FreeNAS. The latter is utterly spergy about robustness but I'm not sure if I care enough about that yet. I'm in the same boat reading up as much as I can on Xpenology, thinking of following Don Lapre's example machine with 16 gigs and ESXi as a Christmas break hack idea. Anyone have personal experience of Xpenology on ESXi?
|
|
#
?
Dec 19, 2013 16:20
|
|
|
Hughlander posted:I'm in the same boat reading up as much as I can on Xpenology, thinking of following Don Lapre's example machine with 16 gigs and ESXi as a Christmas break hack idea. Anyone have personal experience of Xpenology on ESXi? I've spun up a couple test VMs and it seems to work. At least when I tried it, the VMware tools weren't part of the default install. I think there was a way to manually shoehorn them in, but I never really looked into it. As mentioned above, you lose all kinds of SMART monitoring unless you are using direct passthrough to pass an entire card to the VM (and if you are unfamiliar with this technology, it is something you need hardware support for and most consumer level motherboards are iffy at best in this department). I did this with my spare IT flashed M1015 and it looked like it worked, but only kept it in for a couple days while playing. Ultimately, I was concerned about supportability and wasn't as impressed with how it handled different sized disks as I thought it would be.
Fancy_Lad fucked around with this message at 16:40 on Dec 19, 2013 |
|
#
?
Dec 19, 2013 16:37
|
|
|
|
| # ? May 10, 2024 09:57 |
|
|
Hughlander posted:I'm in the same boat reading up as much as I can on Xpenology, thinking of following Don Lapre's example machine with 16 gigs and ESXi as a Christmas break hack idea. Anyone have personal experience of Xpenology on ESXi? Here is a WIP for esxi http://xpenology.com/forum/viewtopic.php?f=14&t=1939 If you just want to play with xpenology all you need a is a thumb drive and a single hard drive. Just remove any hard drives with data on them that you care about first.
|
|
#
?
Dec 19, 2013 16:38
|
|