NAS/Storage Megathread: What is this "File Deletion" You Speak of

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > NAS/Storage Megathread: What is this "File Deletion" You Speak of

«‹›840 »

Jolan: Feb 5, 2007

I've got folders in shares that keep disappearing and reappearing a few seconds later. After such an event, I can't access the folder for some 20 seconds without being thrown back to the containing share. QNAP support hadn't the slightest clue what it might be, and I still haven't been able to sign up for their damned support boards so I can't ask it there. Does anyone know what could be the problem?

# ? Oct 14, 2012 02:27

Adbot: ADBOT LOVES YOU

# ? Apr 18, 2024 21:05

movax: Aug 30, 2008

fletcher posted:

Trying to setup a cron job on NAS4Free:

<snipped table breaker>

Seems to work fine if I run it from a shell, but if I click "Run Now" in the web interface I get "Failed to execute cron job." Anybody know why that might be?

Root vs. non-root user? Or some other permissions issue, perhaps.

e: yeah, the path to your key seems weird to me, '/.mykey', its not actually in your system root directory is it?

# ? Oct 14, 2012 02:41

fletcher: Jun 27, 2003; ken park is my favorite movie; Cybernetic Crumb

movax posted:

Root vs. non-root user? Or some other permissions issue, perhaps.

e: yeah, the path to your key seems weird to me, '/.mykey', its not actually in your system root directory is it?

Yeah it's a strange path, the file exists though. I threw it in there because NAS4Free already puts another passwd file right there. I've tried it as my normal user and as root, doesn't run either way.

# ? Oct 14, 2012 04:15

VictualSquid: Feb 29, 2012; Gently enveloping the target with indiscriminate love.

fletcher posted:

Trying to setup a cron job on NAS4Free:

Seems to work fine if I run it from a shell, but if I click "Run Now" in the web interface I get "Failed to execute cron job." Anybody know why that might be?

Did you try experimenting with escaping the special characters a bit more. I sometimes have problems when going from direct shell to a script, in that it needs extra backslashes.
Check if this is documented in your web interface manual, also.

# ? Oct 14, 2012 11:20

Delta-Wye: Sep 29, 2005

This is driving me nuts. I built a new NAS based on freebsd 9.0 and ZFS, and copied data from my old NAS using solaris and ZFS. The copies succeeded, but I found a few errors. On my one partition, everything was fine, according to the command:

code:

rsync -rvnc --delete /old/ /new/
sending incremental file list

sent 628307 bytes  received 1534 bytes  470.56 bytes/sec
total size is 26349714159  speedup is 41835.50 (DRY RUN)

It claims that nothing needs to be done to match the two folder structures. However, on the old machine (solaris), I get:

code:

[me@oldnas /tank/doc]$ du -sk
26242405        .

and on the new nas (freebsd), I get:

code:

[me@newnas /tank/doc]$ du -Ask
25736913	.
[me@newnas /tank/doc]$ du -sk
23929093	.

It seems like something is missing

Any other techniques/tools I can use to make sure the copy was good? it seems like rsync should have done the job but I can't shake the suspicion that some files are missing based on the partition sizes.

# ? Oct 14, 2012 22:45

alo: May 1, 2005

Delta-Wye posted:

It seems like something is missing Any other techniques/tools I can use to make sure the copy was good? it seems like rsync should have done the job but I can't shake the suspicion that some files are missing based on the partition sizes.

You can try running something like: (on Solaris)

code:

gfind /tank/doc/ -type f -print0 | gxargs -0 md5sum | sort > oldnas.txt

Then run the same thing on your new nas (note that I'm using the GNU versions of find and xargs here) and diff the two resulting files (or md5sum the result).

# ? Oct 14, 2012 23:56

VictualSquid: Feb 29, 2012; Gently enveloping the target with indiscriminate love.

Delta-Wye posted:

However, on the old machine (solaris), I get:
code:
[me@oldnas /tank/doc]$ du -sk
26242405        .
and on the new nas (freebsd), I get:
code:
[me@newnas /tank/doc]$ du -Ask
25736913	.
[me@newnas /tank/doc]$ du -sk
23929093	.
It seems like something is missing Any other techniques/tools I can use to make sure the copy was good? it seems like rsync should have done the job but I can't shake the suspicion that some files are missing based on the partition sizes.

Rsync should be what is needed... So theoretically the differences in sizes should only be metadata. Because

Maybe you should run "du -s --apparent-size" on both machines and compare those.

# ? Oct 15, 2012 00:00

Megaman: May 8, 2004; I didn't read the thread BUT...

What is the general consensus of FreeNAS vs NAS4Free, is either obviously better than the other? I just got the parts to build a machine to replace my Synology but I don't know which to go with. I could read all their documentation and compare but I figured I'd just ask since most people here probably already have personal experience with both.

# ? Oct 15, 2012 04:46

DrDork: Dec 29, 2003; commanding officer of the Army of Dorkness

Megaman posted:

What is the general consensus of FreeNAS vs NAS4Free, is either obviously better than the other? I just got the parts to build a machine to replace my Synology but I don't know which to go with. I could read all their documentation and compare but I figured I'd just ask since most people here probably already have personal experience with both.

They're pretty similar, and if you start with FreeNAS, it's pretty easy to switch over to NAS4Free if you want to try out both (you can then drop back to FreeNAS as long as you don't upgrade the ZFS version on your array). The biggest differences between the two are that FreeNAS has started implementing a modular plug-in system, which makes getting certain 3rd party applications running pretty easy. NAS4Free really hasn't done so, though most of the popular 3rd party apps are already baked in (iTunes/DAAP, BitTorrent, etc), and pkg_add works for most of the others. NAS4Free also sees much more regular updates, and sports ZFS v28, while FreeNAS is back on v15 or so (though v28 is in one of the betas right now, so it's coming). Both share like 90% of their features, though, since they originate from the same project (they split after FreeNAS 7, I believe).

Honestly, for 9 out of 10 people, it'll come down to which WebGUI you like better.

# ? Oct 15, 2012 05:12

Mantle: May 15, 2004

Megaman posted:

What is the general consensus of FreeNAS vs NAS4Free, is either obviously better than the other? I just got the parts to build a machine to replace my Synology but I don't know which to go with. I could read all their documentation and compare but I figured I'd just ask since most people here probably already have personal experience with both.

I installed both last month for a work server and decided on FreeNAS because the GUI made more sense and there were a few more options that weren't configurable via the NAS4Free interface.

# ? Oct 15, 2012 05:32

Delta-Wye: Sep 29, 2005

alo posted:

You can try running something like: (on Solaris)
code:
gfind /tank/doc/ -type f -print0 | gxargs -0 md5sum | sort > oldnas.txt
Then run the same thing on your new nas (note that I'm using the GNU versions of find and xargs here) and diff the two resulting files (or md5sum the result).

I did this, with a few modifications (both NAS's are currently accessible from a pretty nice linux box so I just did the md5sums there with the linux-provided gnu tools). Both came up with the same set of files, with the same md5sums so all is well I guess.

As far as the size difference, :iiam:

# ? Oct 15, 2012 09:29

Longinus00: Dec 29, 2005; Ur-Quan

Delta-Wye posted:

This is driving me nuts. I built a new NAS based on freebsd 9.0 and ZFS, and copied data from my old NAS using solaris and ZFS. The copies succeeded, but I found a few errors. On my one partition, everything was fine, according to the command:
code:
rsync -rvnc --delete /old/ /new/
sending incremental file list

sent 628307 bytes  received 1534 bytes  470.56 bytes/sec
total size is 26349714159  speedup is 41835.50 (DRY RUN)
It claims that nothing needs to be done to match the two folder structures. However, on the old machine (solaris), I get:
code:
[me@oldnas /tank/doc]$ du -sk
26242405        .
and on the new nas (freebsd), I get:
code:
[me@newnas /tank/doc]$ du -Ask
25736913	.
[me@newnas /tank/doc]$ du -sk
23929093	.
It seems like something is missing Any other techniques/tools I can use to make sure the copy was good? it seems like rsync should have done the job but I can't shake the suspicion that some files are missing based on the partition sizes.

I feel like someone really needs to write up or link to a rsync FAQ in the OP.

By default rsync only stats files and compares their modification time and size. This speeds things up tremendously but if you ever doubt file metadata then run rsync with the --checksum (-c) flag and it will use checksums to check if files really are the same or not. It goes without saying that if the dataset is large this might take some time.

There are a few reasons that du might be different from one to another (e.g. sparse file handling) and it's not a huge deal if they're different.

What does the -A flag do for du by the way? I don't see it in the GNU du manpage.

# ? Oct 15, 2012 14:04

Delta-Wye: Sep 29, 2005

Longinus00 posted:

I feel like someone really needs to write up or link to a rsync FAQ in the OP.

By default rsync only stats files and compares their modification time and size. This speeds things up tremendously but if you ever doubt file metadata then run rsync with the --checksum (-c) flag and it will use checksums to check if files really are the same or not. It goes without saying that if the dataset is large this might take some time.

There are a few reasons that du might be different from one to another (e.g. sparse file handling) and it's not a huge deal if they're different.

Yeah, the issue was the folder manifest AND checksums were identical, while the 'amount of stuff' was not and I was somewhat nervous. There is a lot of obfuscation using ZFS between how much data goes on the disk and how much data there actually is, between options like compression and dedup and stuff like metadata. I just wanted to make sure I had a clean copy before turning the old machine off and settling in.

Longinus00 posted:

What does the -A flag do for du by the way? I don't see it in the GNU du manpage.

code:

     -A      Display the apparent size instead of the disk usage.  This can be
             helpful when operating on compressed volumes or sparse files.

I think it's the equivalent to --apparent-size that tonberrytoby recommended, that info is from the freebsd man page.

# ? Oct 15, 2012 15:52

Mantle: May 15, 2004

Longinus00 posted:

I feel like someone really needs to write up or link to a rsync FAQ in the OP.

I think that would be a great idea. You seem to be one of the more knowledgeable people about rsync in this thread!

# ? Oct 16, 2012 22:46

Not an Anthem: Apr 28, 2003; I'm a fucking pain machine and if you even touch my fucking car I WILL FUCKING DESTROY YOU.

Does this idea make sense from a usage perspective, as well as a "class of product" ie prosumer or whatever, in comparison to my data:

I am a designer and I am starting to pile on data to be backed up on active projects I have going.

I recently had a WD drive die and realized I've never looked at redundant backup in any sense of the word. I'm cheap as hell but I am okay with spending bigger dollars on stuff that requires it.

I recently bought a lovely chinese media player (Mele A2000) for my projector and haven't figured out exactly how to set it up as I can plug 2.5" SATA drives into the top of it directly, for a bit I thought I'd make it serve up my movies/music for the projector and I wouldn't need anything else. It does work for that fairly well.

Now combining my interest in
-media storage for playback
-work backup, direct or wireless
-redundant backup
-frivolous gadgetry

I am thinking of a drobo/synology but leaning towards synology due to price

I like the small 2.5" form factor one (DS411 slim) or just the ugly clunker for maximum space (DS413j)

I would be fine building a unit but looking at the stuff synology offers with their drives and not having to do anything I would be happy to spend the money on that

Is it overkill for what I'm doing?

# ? Oct 17, 2012 18:44

sleepy gary: Jan 11, 2006

If your data is worth more to you than the cost of the hardware, then no it is not overkill. Synology makes excellent products.

# ? Oct 17, 2012 18:49

PirateDentist: Mar 28, 2006; Sailing The Seven Seas Searching For Scurvy

DNova posted:

If your data is worth more to you than the cost of the hardware, then no it is not overkill. Synology makes excellent products.

I just picked up a DS413j, I'm very impressed with it so far. The software is worth a great deal of the price as far as I'm concerned.

The mobile apps work great, I can send stuff and download things from anywhere. I still haven't played around with most of the stuff it can do. My only issues with it so far are that the automatic port opening is a bit finicky, but manually assigning ports fixes that, and adding a drive to the array is slow as balls. Took two days to add a third 2TB drive. But unless you have an incredibly pressing need for more space right now that's not a big deal.

I'd keep in mind too, that you can get nearly twice the space for about the same price with the 3.5" drives. 2TB WD Red drives were nearly the same price as 1TB 2.5 drives during the last sale I saw.

# ? Oct 17, 2012 20:29

Krakkles: May 5, 2003

Will something like a Synology work with CrashPlan?

I want to back up my NAS to a service like that, so I wanted to make sure.

# ? Oct 17, 2012 21:45

sellouts: Apr 23, 2003

Yes, but only on the upper end of the Synology products will you have enough processing power to do that and not cripple the system otherwise

http://forum.synology.com/wiki/index.php/CrashPlan_Headless_Client

# ? Oct 17, 2012 21:58

Not an Anthem: Apr 28, 2003; I'm a fucking pain machine and if you even touch my fucking car I WILL FUCKING DESTROY YOU.

With the synology products, how does streaming work?

I have a mele a2000 (cheap chinese set top box with android 4.1), I assume I can use a DLNA browser app to connect to the NAS and use that to play movies off the NAS on my projector?

edit- For cost, I'm fairly certain I'm interested in the DS212 with two enterprise WD 2tb drives and just do mirroring for now

Not an Anthem fucked around with this message at 17:56 on Oct 18, 2012

# ? Oct 18, 2012 04:21

Stutes: Oct 13, 2005; Tonight's the Night

Krakkles posted:

Will something like a Synology work with CrashPlan?

I want to back up my NAS to a service like that, so I wanted to make sure.

I just did this, and CrashPlan seems to be working fine thus far. I was able to seamlessly adopt my existing 400GB backup set, using these instructions. I have one of the Intel models, upgraded to 4GB RAM - from what I understand, CrashPlan may silently quit on you if you're RAM limited (e.g. on the ARM/PowerPC models).

# ? Oct 18, 2012 06:33

sellouts: Apr 23, 2003

What Synology unit do you have, and do you have any other apps running?

# ? Oct 18, 2012 22:49

Fancy_Lad: May 15, 2003; Would you like to buy a monkey?

Just a heads up as it seems there are several folks in this thread that have been looking for IBM M1015 cards. There are a bunch on sale for ~$75 shipped on ebay right now:
http://cgi.ebay.com/ws/eBayISAPI.dll?ViewItem&item=380492923084&ssPageName=ADME:B:SS:US:1123

# ? Oct 21, 2012 14:45

Fatal: Jul 29, 2004; I'm gunna kill you BITCH!!!

Fancy_Lad posted:

Just a heads up as it seems there are several folks in this thread that have been looking for IBM M1015 cards. There are a bunch on sale for ~$75 shipped on ebay right now:
http://cgi.ebay.com/ws/eBayISAPI.dll?ViewItem&item=380492923084&ssPageName=ADME:B:SS:US:1123

Sweet thanks! Grabbed two for $140 shipped, crazy good deal.

# ? Oct 21, 2012 17:39

FISHMANPET: Mar 3, 2007; Sweet 'N Sour
Can't
Melt
Steel Beams

Anyone have a good source for brackets for those?

# ? Oct 21, 2012 17:53

arbybaconator: Dec 18, 2007; All hat and no cattle

Joined the Synology club yesterday.

This thing rules. The software is fantastic, and it was incredibly simple to set up.

# ? Oct 21, 2012 19:40

Fancy_Lad: May 15, 2003; Would you like to buy a monkey?

FISHMANPET posted:

Anyone have a good source for brackets for those?

I have an M1015 right now (ordered a 2nd with that auction) and when I got it in, I went digging through my box of old pci cards for brackets. I found that a lot of them have screw holes in the same places, so if you have a stash of old stuff sitting around you may be able to find something that works for free.

Barring that, I would just use it without one but ebay tends to have several for sale for ~10 each at any given point if you are really concerned.

# ? Oct 21, 2012 19:46

r u ready to WALK: Sep 29, 2001

One thing that really annoys me with all the custom built storage server solutions is that it's really hard to find a chassis and a controller that enables you to have HDD error LEDs for each drive bay.

Ages ago i had a promise supertrak raid card and promise hotswap bays, and it would indicate the faulted drive with a red LED. Everything else about it sucked, but knowing exactly which drive to swap out with a new one was very nice.

Any suggestions for which card and chassis to get if i want this? It sounds like it gets even trickier when you pass drives through to ZFS instead of using hardware raid.

The norco 24 bay chassis is supposed to at least have a LED header on the backplanes, so i suppose you could do something neat with an arduino and some zpool status scripting.

# ? Oct 21, 2012 20:14

Viktor: Nov 12, 2005

spaceship posted:

Joined the Synology club yesterday.

This thing rules. The software is fantastic, and it was incredibly simple to set up.

Thats an awesome unit.

Just finished setting up a DS413j for personal use moving down from a custom unraid nas (e2180/2gb/8TB pooled). Wish I could spend a bit extra and bought an Intel based unit for expansion as well as the ARM processor even at 1.6Ghz bogs down quick on PAR repairs.

Super impressed that effortlessly out of the box DSM takes care of dyndns, port forwarding, autoban of ip's, backup vpn, ups monitoring, custom scheduled smart testing, and sms/email alarming. All of this took days of hunting down custom modules, fixing shell scripts, and never as seamless on unraid.

# ? Oct 21, 2012 20:14

MasterColin: Aug 4, 2006

Just picked this up for a start of an UnRaid server. Planning on running UnRaid, Sab/SB/PS3MediaServer. I'm also going to upgrade my router to a commercial grade router to make sure I can get gigabit speeds.

ASRock H77M-ITX
Intel(R) Core(TM) i3-2100T
8GB DDR3 (2x4GB)
Fractal Design Array R2 -- Case
SAMSUNG HD103SI 1TB 5400 RPM -- First Drive
OCZ Vertex 2 2.5" 55GB SATA II SSD --Cache

I have some other 350gb drives i'll throw in until I can justify buying some 3tb drives or see an awesome sale.

Got a pretty good deal and the only issue is the MB only has 4 Sata.

Anyone see any issues? I feel like it is a little over powered for a UnRaid but with the media server running also I think I'll be glad for the extra HP.

# ? Oct 21, 2012 23:01

Viktor: Nov 12, 2005

MasterColin posted:

Anyone see any issues? I feel like it is a little over powered for a UnRaid but with the media server running also I think I'll be glad for the extra HP.

Pick up an intel nic, the realtek chipset drivers have been troublesome.

# ? Oct 21, 2012 23:10

MasterColin: Aug 4, 2006

Viktor posted:

Pick up an intel nic, the realtek chipset drivers have been troublesome.

In what way?

While it doesn't sound like a bad idea, my only issue is I only have one PCI slot meaning I couldn't expand the SATA ports if I needed to. Leaving me with 1 Cache, 1 Parity and 2 data.

# ? Oct 22, 2012 01:00

DrDork: Dec 29, 2003; commanding officer of the Army of Dorkness

MasterColin posted:

In what way?

While it doesn't sound like a bad idea, my only issue is I only have one PCI slot meaning I couldn't expand the SATA ports if I needed to. Leaving me with 1 Cache, 1 Parity and 2 data.

Realtek chips will work, but you will find the experience is less than perfect, due to less than perfect Linux/BSD drivers. You're likely to get sub-standard transfer speeds, dropped connections, and general "why the gently caress is that happening?" issues. Intel Pro NICs, on the other hand, benefit from having very solid drivers, and thus perform noticeably better.

# ? Oct 22, 2012 01:15

MasterColin: Aug 4, 2006

DrDork posted:

Realtek chips will work, but you will find the experience is less than perfect, due to less than perfect Linux/BSD drivers. You're likely to get sub-standard transfer speeds, dropped connections, and general "why the gently caress is that happening?" issues. Intel Pro NICs, on the other hand, benefit from having very solid drivers, and thus perform noticeably better.

Thanks for the heads up. I did some searching on this and You are correct. End of the day I may have to get a new MB and sell that one. Or I can use the eSata and get an expansion box.

Edit: More I think about it, If it presents an issue, I'll just get a NIC card, and use the eSata port to another drive enclosure if needed.

MasterColin fucked around with this message at 02:13 on Oct 22, 2012

# ? Oct 22, 2012 01:27

BlankSystemDaemon: Mar 13, 2009

spaceship posted:

Joined the Synology club yesterday.

This thing rules. The software is fantastic, and it was incredibly simple to set up.

That's a neat-looking rig! Does it do RAID6? I've taken to always recommending people use raid (whether in software or hardware) that has two parity disks, because the increasing size of disks and their steady URE rate (it's been argued by Adam Leventhal, a zfs developer, that even RAID 6 will become unusable in 2019).

I wish zfs would add one feature that it's missing, namely adding parity disks to already-existing pools (it's a feature that's listed on some wish-list, and it's supposedly coming - eventually).

MasterColin posted:

Just picked this up for a start of an UnRaid server. Planning on running UnRaid, Sab/SB/PS3MediaServer. I'm also going to upgrade my router to a commercial grade router to make sure I can get gigabit speeds.

ASRock H77M-ITX
Intel(R) Core(TM) i3-2100T
8GB DDR3 (2x4GB)
Fractal Design Array R2 -- Case
SAMSUNG HD103SI 1TB 5400 RPM -- First Drive
OCZ Vertex 2 2.5" 55GB SATA II SSD --Cache

I have some other 350gb drives i'll throw in until I can justify buying some 3tb drives or see an awesome sale.

Got a pretty good deal and the only issue is the MB only has 4 Sata.

Anyone see any issues? I feel like it is a little over powered for a UnRaid but with the media server running also I think I'll be glad for the extra HP.

Other than NIC-issue already mentioned, you can always get an IBM M1015 card if you're lacking ports. Also see the above about large disks and URE.

BlankSystemDaemon fucked around with this message at 08:38 on Oct 22, 2012

# ? Oct 22, 2012 08:33

necrobobsledder: Mar 21, 2005; Lay down your soul to the gods rock 'n roll; Nap Ghost

DrDork posted:

Realtek chips will work, but you will find the experience is less than perfect, due to less than perfect Linux/BSD drivers.

I haven't had the best of luck with Realtek NICs even on Windows to be honest. There's something to be said for buying a decent brand of NIC when it comes to getting everything to work as you'd think they should. It's pretty easy to cut corners on something like a NIC and Intel doesn't do it as much others in large part due to so much of their NICs being used by businesses.

# ? Oct 22, 2012 13:40

Mantle: May 15, 2004

This may be more of a general FreeBSD question, but I have a problem with FreeNAS deleting files from my /etc directory on boot.

I have created an rsync task to do a push to a remote server once a night. The remote server requires a user and password to connect, and FreeNAS says I have to create a file with the password in it. I put the password in a file at /etc/rsync_module_backup_password and gave it chmod 600 permissions, owned by root.

Whenever I reboot the FreeNAS system, the /etc/rsync_module_backup_password file is gone.

Why is this? Is there a better place to put the password file?

# ? Oct 22, 2012 17:59

mik: Oct 16, 2003; oh

spaceship posted:

Joined the Synology club yesterday.

This thing rules. The software is fantastic, and it was incredibly simple to set up.

Similarly... this came in the mail today:

Synology DS-412+ - Newegg canada had a decent sale on it.

# ? Oct 22, 2012 18:30

Moey: Oct 22, 2010; I LIKE TO MOVE IT

mik posted:

Similarly... this came in the mail today:

Synology DS-412+ - Newegg canada had a decent sale on it.

I like how you made sure to include a shoe.

# ? Oct 22, 2012 18:39

Adbot: ADBOT LOVES YOU

# ? Apr 18, 2024 21:05

arbybaconator: Dec 18, 2007; All hat and no cattle

mik posted:

Similarly... this came in the mail today:

Synology DS-412+ - Newegg canada had a decent sale on it.

# ? Oct 22, 2012 18:50

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > NAS/Storage Megathread: What is this "File Deletion" You Speak of

«‹›840 »