NAS/Storage Megathread: What is this "File Deletion" You Speak of

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > NAS/Storage Megathread: What is this "File Deletion" You Speak of

heeen: May 14, 2005; CAT NEVER STOPS

Thinking about setting up 4 slot NAS for movies, photos, music. I'm currently looking at the QNAP TS-410 and the Synology DS-411j. What else is there to look at in this price range?
I plan on getting two 2TB drives (probably F4s) and expand later.
Can owners of said units report their performance and general experience?

# ¿ Jul 19, 2011 12:42

Adbot: ADBOT LOVES YOU

# ¿ Apr 26, 2024 14:46

heeen: May 14, 2005; CAT NEVER STOPS

jeeves posted:

ZFS question--

My NAS had been chugging along copying 4TB of data, but now that I'm past that and on to like almost 5TB of data out of a 7.2tb ZFS raid, the write performance has gone to complete poo poo.

Is this because I only have 5GB (1+4 stick) of RAM in my NAS? I remember hearing somewhere that ZFS absolutely gobbles up ram, and you should have 1 GB per 1TB of raid, but I can't find that quote again so maybe I think I may have made it up.

Is that somewhat true? My NAS is absolutely crawling along at copying data these days, so I am wondering if dropping another 45$ of another 4GB stick of RAM is worth it or if it may be something else?

Funny, I knew you were talking about a proliant microserver when you mentioned the weird 4+1 GB setup. I pulled the trigger on the same setup as well, microserver, 4x2TB disks I plan to use in zraid and freeNAS booting off a usb stick. Apart from your recent performance problems, any tips/expirience you want to share?

# ¿ Jul 26, 2011 09:10

heeen: May 14, 2005; CAT NEVER STOPS

What performance is to be expected from four WD20EARS (2TB) drives in a raidz configuration? I am getting ~60Mb/s through dd and this strikes me as a bit low. The hardware is a HP proliant microserver.

# ¿ Aug 5, 2011 10:20

heeen: May 14, 2005; CAT NEVER STOPS

Right. It just struck me as odd when I started to fill the pool with large media files that even after some tweaking I couldn't get over 60Mb/s. Anyways, the 60Mb/s number coincidences with various speed benchmarks I found, e.g:

If zfs chose to begin writing at the end of the drive, that is.

heeen fucked around with this message at 11:48 on Aug 5, 2011

# ¿ Aug 5, 2011 11:46

heeen: May 14, 2005; CAT NEVER STOPS

teamdest posted:

well, can you run some benchmarks and see if this is a consistent thing? dd isn't the most exacting test of a disk's capabilities, after all.

Edit: also how full is your array? copy-on-write filesystems have issues when they get fragmented and largely full where finding a contiguous chunk large enough for the data becomes more difficult.

The array is empty and I got the values confirmed from diskinfo.

# ¿ Aug 5, 2011 14:58

heeen: May 14, 2005; CAT NEVER STOPS

I'm starting to see some worrying messages:

code:

Aug  6 15:06:33 freenas kernel: ahcich2: Timeout on slot 4 port 0
Aug  6 15:06:33 freenas kernel: ahcich2: is 00000000 cs 00000000 ss 00003ff0 rs 00003ff0 tfd 40 serr 00000000
Aug  6 15:07:04 freenas kernel: ahcich2: AHCI reset: device not ready after 31000ms (tfd = 00000080)
Aug  6 15:07:28 freenas root: ZFS: vdev I/O failure, zpool=tank1 path=/dev/gpt/disk2 offset=270336 size=8192 error=6

(ada2:ahcich2:0:0:0): lost device
(ada2:ahcich2:0:0:0): Invalidating pack
(ada2:ahcich2:0:0:0): Invalidating pack
(ada2:ahcich2:0:0:0): Invalidating pack
(ada2:ahcich2:0:0:0): Invalidating pack
(ada2:ahcich2:0:0:0): Invalidating pack
(ada2:ahcich2:0:0:0): Invalidating pack
(ada2:ahcich2:0:0:0): Invalidating pack
(ada2:ahcich2:0:0:0): Invalidating pack
(ada2:ahcich2:0:0:0): Invalidating pack
(ada2:ahcich2:0:0:0): Invalidating pack

also once I got lots of these:

code:

 freenas kernel: swap_pager: I/O error - pageout failed; blkno 1051647,size 4096, error 6

code:

freenas# zpool status
  pool: tank1
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: [url]http://www.sun.com/msg/ZFS-8000-9P[/url]
 scrub: none requested
config:

        NAME           STATE     READ WRITE CKSUM
        tank1          ONLINE       0     0     0
          raidz1       ONLINE       0     0     0
            gpt/disk0  ONLINE       0     0     0
            gpt/disk1  ONLINE       0     0     0
            gpt/disk2  ONLINE       3  196K     1
            gpt/disk3  ONLINE       0     0     0

errors: No known data errors

# ¿ Aug 6, 2011 14:13

heeen: May 14, 2005; CAT NEVER STOPS

I did short offline checks on all drives which appeared ok, and the self asessment said it passed. what should I look out for?

# ¿ Aug 6, 2011 15:05

heeen: May 14, 2005; CAT NEVER STOPS

I think the drive is actually dying:

code:

kernel: (ada2:ahcich2:0:0:0): lost device    

# ataidle ada2
ataidle: error opening ada2

camcontrol identify ada2
camcontrol: cam_lookup_pass: CAMGETPASSTHRU ioctl failed
cam_lookup_pass: No such file or directory
cam_lookup_pass: either the pass driver isn't in your kernel
cam_lookup_pass: or ada2 doesn't exist

It also isn't listed under camcontrol devlist.

code:

# camcontrol devlist
<WDC WD20EARS-00J99B0 80.00A80>    at scbus0 target 0 lun 0 (pass0,ada0)
<WDC WD20EARS-00J99B0 80.00A80>    at scbus1 target 0 lun 0 (pass1,ada1)
<WDC WD20EARS-00J99B0 80.00A80>    at scbus3 target 0 lun 0 (pass3,ada3)
<JetFlash Transcend 4GB 8.07>      at scbus6 target 0 lun 0 (pass4,da0)

But the weird thing is: zpool claims it is still there and resilvering it:

code:

freenas# zpool status
  pool: tank1
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: [url]http://www.sun.com/msg/ZFS-8000-9P[/url]
 scrub: resilver in progress for 0h15m, 10.69% done, 2h7m to go
config:

        NAME           STATE     READ WRITE CKSUM
        tank1          ONLINE       0     0     0
          raidz1       ONLINE       0     0     0
            gpt/disk0  ONLINE       0     0     0
            gpt/disk1  ONLINE       0     0     0
            gpt/disk2  ONLINE       3 1.56M     3  148M resilvered
            gpt/disk3  ONLINE       0     0     0

errors: No known data errors

freenas root: ZFS: vdev I/O failure, zpool=tank1 path=/dev/gpt/disk2 offset=1998251106304 size=8192 error=6

Why does zfs not notice one drive is offline!?

edit: one more thing: would it be a good idea to swap drive bays of the faulty drive to check if it's really the drive or rather the cabling? or will that mess with the remaining drives somehow?

heeen fucked around with this message at 00:48 on Aug 7, 2011

# ¿ Aug 7, 2011 00:42

heeen: May 14, 2005; CAT NEVER STOPS

clavicle posted:

Sounds steep -- believe me, I know -- but the HP ProLiant N36L I was looking at to serve as the *basis* of my setup costs 860 USD by itself here.

drat! The N36L just dropped to eur 155 ($220) in Germany. That is post taxes.

# ¿ Aug 11, 2011 13:57

heeen: May 14, 2005; CAT NEVER STOPS

clavicle posted:

I'm expecting low power consumption during most of the day, as it will be idling.

I get 30W when it is idling with the disks spun down and the cpu scaled down.
40W with the drives active, 50W when transferring stuff.

code:

freenas# sysctl dev.cpu.0.freq
dev.cpu.0.freq: 100

freenas# smartctl -n standby -i /dev/ada0 && smartctl -n standby -i /dev/ada1 && smartctl -n standby -i /dev/ada2 && smartctl -n standby -i /dev/ada3
smartctl 5.41 2011-06-09 r3365 [FreeBSD 8.2-RELEASE-p2 amd64] (local build)
Copyright (C) 2002-11 by Bruce Allen, [url]http://smartmontools.sourceforge.net[/url]

Device is in STANDBY mode, exit(2)
freenas# smartctl -n standby -i /dev/ada0; smartctl -n standby -i /dev/ada1; smartctl -n standby -i /dev/ada2; smartctl -n standby -i /dev/ada3
smartctl 5.41 2011-06-09 r3365 [FreeBSD 8.2-RELEASE-p2 amd64] (local build)
Copyright (C) 2002-11 by Bruce Allen, [url]http://smartmontools.sourceforge.net[/url]

Device is in STANDBY mode, exit(2)
smartctl 5.41 2011-06-09 r3365 [FreeBSD 8.2-RELEASE-p2 amd64] (local build)
Copyright (C) 2002-11 by Bruce Allen, [url]http://smartmontools.sourceforge.net[/url]

Device is in STANDBY mode, exit(2)
smartctl 5.41 2011-06-09 r3365 [FreeBSD 8.2-RELEASE-p2 amd64] (local build)
Copyright (C) 2002-11 by Bruce Allen, [url]http://smartmontools.sourceforge.net[/url]

Device is in STANDBY mode, exit(2)
smartctl 5.41 2011-06-09 r3365 [FreeBSD 8.2-RELEASE-p2 amd64] (local build)
Copyright (C) 2002-11 by Bruce Allen, [url]http://smartmontools.sourceforge.net[/url]

Device is in STANDBY mode, exit(2)

# ¿ Aug 11, 2011 19:49

heeen: May 14, 2005; CAT NEVER STOPS

jeeves posted:

performance of ZFS/Freenas ... Proliant Microserver+8GB ram ... gigabit router.

How much can you pull from/write to the device? as in per
dd if=/dev/zero of=/mnt/tank/testfile bs=1M count=10k
and
dd if=/mnt/tank/testfile of=/dev/null bs=1M count=10k

I have the microserver on freenas and 4 WD20EARS in zraid1 and I get 80Mb/s writing and around 200 reading, not really impressed so far. Over samba I get ~50mb/s.

# ¿ Aug 15, 2011 10:16

heeen: May 14, 2005; CAT NEVER STOPS

I want to share the following, for what it's worth:
Before I had 4 wd20ears drives, formatted by freenas to have a 2gb swap partition ans the rest used for zfs. This gave me writing speeds around 60mb/s with dd.
I upgraded this to 5 of the same drives, but while rebuilding the pool, I erased all partitioning, created 4k sector devices with gnop (gnop create -S 4096 /dev/adaX) and subsequently created the zraid1 using these adaN.nop devices directly. Write speed almost quadrupled to 220Mb/s! Read speed also increased to 300 Mb/s.
I have 5gb ram in my microserver and if any service was to use more than 5gb ram, swap wouldn't help it anyways so I can live without it.

# ¿ Aug 27, 2011 18:25

heeen: May 14, 2005; CAT NEVER STOPS

Longinus00 posted:

Your wording is a little confusing, 2GB swap partition per drive?

Yes, freenas partitions drives for you when you create a volume through the web GUI. Each drive gets a 2GB swap partition.

# ¿ Aug 28, 2011 10:54

heeen: May 14, 2005; CAT NEVER STOPS

some zfs guy posted:

> It's pretty simple. If you go direct to disk, ZFS manages the cache and can "hand off" write to the disk.
> If it's a partition, ZFS has no awareness of the cache and has to wait for the disk to return the write acknoloedgement

Don't use zfs on partitions (as freenas does by default)!

# ¿ Aug 30, 2011 09:24

heeen: May 14, 2005; CAT NEVER STOPS

2Gb per drive on 4 drives makes 8Gb.

# ¿ Aug 30, 2011 12:46

heeen: May 14, 2005; CAT NEVER STOPS

jeeves posted:

Proliant Microserver definitely can't WOL when off.

Bummer - I thought I read somewhere it was possible. I guess you tried it yourself?

# ¿ Sep 8, 2011 12:58

heeen: May 14, 2005; CAT NEVER STOPS

I got really good results with tuning the freebsd TCP parameters, it helped me go from 50 Mb/s sustained to 90-100 MB/s sustained on my freenas box. Maybe you can find similar tuning options for readynas?

# ¿ Sep 16, 2011 08:44

heeen: May 14, 2005; CAT NEVER STOPS

I'm also running freenas 8 on a microserver, 5 2tb drives in raidz-1 and a usb stick for the OS. There are a few things to look out for if you really want to push gigabit transfer rates over smb, though (setting up 4k sector drives correctly, smb tweaks, kernel tcp/ip tweaks).
Also try to set up everything up *exactly* the way you want it before you start dumping data on it, you can't grow zfs vdevs or convert it to raidz2.

# ¿ Sep 29, 2011 06:37

heeen: May 14, 2005; CAT NEVER STOPS

PraxxisParadoX posted:

Mind sharing what you tweaked exactly?

Try the kernel tweaks from this page:
http://learnedbyerror.blogspot.com/2009/09/lets-tune-er-up.html
I think I sized my tcp send and receive spaces to a few megabytes. I didn't save what exactly I did the last time and lost it after a reboot and didn't care enough to set it up again.

Edit: using 4k sectors and setting up ZFS on the raw gnop devices instead of gparted partitions is also really recommended.

heeen fucked around with this message at 12:43 on Oct 17, 2011

# ¿ Oct 17, 2011 12:38

Adbot: ADBOT LOVES YOU

# ¿ Apr 26, 2024 14:46

heeen: May 14, 2005; CAT NEVER STOPS

Telex posted:

well, you can scrub now. It's not exactly fsck, but hey.
code:
pool: bender
 state: ONLINE
status: One or more devices has experienced an error resulting in data
	corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
	entire pool from backup.
   see: [url]http://www.sun.com/msg/ZFS-8000-8A[/url]
 scrub scrub in progress for 5h37m, 23.74% done, 18h3m to go
seems to have found a handful of things to repair but holy crap that takes a long time to scrub.

what kind of zraid is this? Shouldn't it be possible to repair the defective blocks on one disk with the correct blocks from a redundant disk?

# ¿ Oct 19, 2011 09:17

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > NAS/Storage Megathread: What is this "File Deletion" You Speak of