MrMoo posted:

For not gigantic data Amazon S3 is actually surprisingly reasonable and is my current choice for TB per day. Glacier turns out not to be effective with such small datasets.

Timdogg posted:

Thanks, that is good to know. What size datasets are you talking? Do you use the AWS Storage Gateway, or any other appliance to assist?

mayodreams posted:

Hey NetApp Goons:

is there a way to delete large folder/data sets from a CLI? I have hundreds of gigs to delete to get free some space and Windows explorer isn't doing a very good job.

toplitzin posted:

It depends on how you have the data shared/segregated.

If each data set/share is it's own qtree or volume you could just nuke the vol/qtree and recreate it anew and empty. Ie: \\filer\share files to go bye 1\*.*
\\filer\all this needs to go\*.*

However if it's /vol/allmydata/nested in subfolders/ seen as \\filer\share\some data here(don't delete me)\some here\ then nope.

Edit: short version. No we don't do file level CLI access.

mayodreams posted:

Thanks. We use Nexenta and I guess I took CLI access to the filer for granted.

Timdogg posted:

We have a 5 Node 72NL (320 TB Usable) cluster about to hit EOL this fall and I am working on options to replace it with a much higher capacity system. We really have enjoyed the simplicity of Isilon and it has worked great, but we are now being tasked with storing bigger datasets most around 20TB to 50TB, one edge case is 400TB.

Unfortunately, it is becoming more and more apparent that I am not going to be able to afford a 1.5 PB+ Isilon cluster and was interested in hearing if you all had some suggestions. This data is usually large files, not commonly accessed, we used the Near-Line archival storage from Isilon and speed was *never* an issue. So no IO requirements, but the data does need to be accessible, so Amazon Glacier really isn't useful here. We use CIFS and NFS on the Isilon.

I have a good group of Sysadmin, but none are dedicated storage admin, so I am not interested in rolling my own cloud with OpenStack or Ceph. We have some Dell 3200i with 3 1200i attached for 96 TB RAW, it has worked okay, but wasn't a huge fan of the LUN management...but the more I read this thread, the more I am thinking this may be one of my best options for capacity + reliability + price. We also just bought vSAN for our VM storage, ... we could scale that out, hadn't really thought of it for this particular use case.

Anyways, would love to hear any suggestions if you have them.

TLDR: Need easy to manage, reliable PB+ storage, oh yeah inexpensive would be great.

the spyder posted:

I was given the task of storing 1.2PB of imagery on a shoe string budget and ended up rolling my own dual head Solaris/ZFS based system, serving out CIFS/NFS. Purchase price for the hardware was around $200k and it took under a week to assemble, image, and setup. We deal with thousands of 1GB files per dataset and so far our users have been happy (~5 months). I need to play with the ARC/L2ARC settings a bit, but management has been incredibly simple. There's a learning curve if you're not a UNIX guy, but it's fairly well documented. I use NAPP-IT for simplicity, but have been doing more things via CMD on Solaris 11.2.

Timdogg posted:

Awesome, sounds like a similar setup that I was thinking of... what kind of hardware did you buy? Supermicro? Did the solaris licenses cost a pretty penny? Or did you already have some?

NippleFloss posted:

That said, there are good reasons to keep storage management separate from data management. Deleting files is the province of the data owner, and shouldn't necessarily be easily accomplished by the storage admin.

Amandyke posted:

Have you looked at the new HD400 Isilon nodes? I don't have the pricing specifics but we're currently looking into buying them to replace one of our 108NL clusters of around 30 nodes. It's as cheap and deep as Isilon gets.

Mr Shiny Pants posted:

NetApp question: We have a Metro Cluster running with one node running all SATA and one node running SAS shelves. We migrated our Exchange environment (DBs) to the SATA one and since we made the Netapp the primary member in the DAG the filer seems to stall.

The filer serves RDMs to Vshpere hosts via ISCSI.

All IO seems to drop to almost zero, disconnecting users and making our life miserable. ISCSI latency rises to 60 ms instead of the normal 1 ms - 4 ms.

Anyone ever seen something like this before?

Load is a constant 200MB sec on a filer that has about 30 SATA disks in its aggregate and 512GB of flash cache.

I've checked the EMS logs but can't find anything in it that would explain the behaviour we are seeing.

OldPueblo posted:

It could really be a bunch of things. MetroCluster has a small automatic performance hit due to the distance factor, even though you'd think its only writing to the local side its really writing to both plexus simultaneously, etc. I guess I'd start with where was it working better before, on the SAS? How fast are your ISL links, do you have two or four? Do the switch ports show excessive errors via porterrshow? Could be a network bottleneck, I've seen a MetroCluster overwhelm a network after a head upgrade resulting in tons of network discard packets on an older network infrastructure. I'd say maybe fire off some ASUPs and see what support has to say, or check My ASUP and System Manager for any recommendations.

Captain Foo posted:

Anyone familiar with emc data domains as backup solutions?

grobbendonk posted:

We've used data domain with Netbackup since about 2011/2012 and we went totally tape less around the end of 2012, and we've never looked back. The advantages we've found is the backup and restore success rate improved significantly, our replication to DR is much faster, we saved a lot of space in our data centres and we didn't have to worry about tape storage and getting random tapes back from offsite when wanting to restore a particular data set. The main disadvantages are cost and getting the sizing right, we're actually doing our third head swap to a DD990 next month.

It works for us because even though we're protecting around 2Pb of data our retention period for the vast majority of backups is only 2-4 weeks. The dedup and compression varies enormously depending on what it is being backed up and Oracle databases can be particularly inefficient. We're currently trialling Avamar directly to Data Domain to cut dump Netbackup but I'm not sure how well that's going at the moment.

Things to look out for or consider are the number of simultaneous streams the DD will support, whether to separate different data types onto different mtrees, getting your Oracle guys to change their backups to compression off and files per data set=1 (although this is changing in the latest firmware) and if you can deploy DD boost to make your clients do the work.

Mr Shiny Pants posted:

NetApp question: We have a Metro Cluster running with one node running all SATA and one node running SAS shelves. We migrated our Exchange environment (DBs) to the SATA one and since we made the Netapp the primary member in the DAG the filer seems to stall.

The filer serves RDMs to Vshpere hosts via ISCSI.

All IO seems to drop to almost zero, disconnecting users and making our life miserable. ISCSI latency rises to 60 ms instead of the normal 1 ms - 4 ms.

Anyone ever seen something like this before?

Load is a constant 200MB sec on a filer that has about 30 SATA disks in its aggregate and 512GB of flash cache.

I've checked the EMS logs but can't find anything in it that would explain the behaviour we are seeing.

Mr Shiny Pants posted:

We already have a support case, I was just wondering if anyone of you might have seen this before as there are a few people visiting this thread that have Netapp experience.

parid posted:

What does sysstat look like? Is it keeping up with consistency points? CPU usage? What does your raid domain cpu usage (sysstat -M) look like while its in the problem state? How old is the install? What version of ontap?

Do you have FMC-DC running and log collecting? If not, you may want to start in case you support case goes that way and you need days of samples.

NippleFloss posted:

Grab a "sysstat -x 1" output during the issue and check for a B or b in the "CP Type" column and high disk utilization. What ONTAP version are you running? Also, are you seeing high read latency or write latency, and are you seeing it on the log volumes, the DB volumes, or both?

Timdogg posted:

Awesome, sounds like a similar setup that I was thinking of... what kind of hardware did you buy? Supermicro? Did the solaris licenses cost a pretty penny? Or did you already have some?

Seventh Arrow posted:

I work in the NAS department for a bank and we've been a NetApp shop ever since we've had NAS (we have about 30 filers) - except now, we're going to be switching to Hitachi's HNAS platform. It's kind of interesting, I never even knew Hitachi had a NAS solution. We've done some of the training so far and it's kind of cool to see a different architecture, because they're both kind of different. So far, though, it seems like Hitachi EVS = NetApp vfilers and Hitachi virtual volumes = NetApp qtrees. This is important, because qtrees are our bread and butter. Some of our old qtrees have grown into disorganized monstrosities, so this might be a good chance to migrate some stuff into a more orderly setup.

Zephirus posted:

That's a pretty good analogy. The HNAS product is ex-Bluearc (Titan/Mercury depending on the version). If they're set up properly we've found them to be ridiculously fast.

If you have a large amount of small files you will want definitely want to look into fast metadata storage.

There are a number of horrible horrible bugs in 11.x so you really want to make sure you're on 12.x

Misogynist posted:

Adding to this:

One thing to be really careful of with HNAS if you're running it in a very high-performance environment like an HPC cluster (we had around 120 gigabits of aggregate throughput across 3 heads): unless something has changed since the last time I looked at them, the cluster networking connection between the frontend nodes is limited to a single 10-gigabit link. This is important, because if you talk to the node in an EVS that doesn't "own" a filesystem (i.e. you're trying to round-robin your cluster between NFS head nodes), all file traffic for that filesystem will be served (read: proxied) from the other frontend node over this link. So while you can port-channel multiple 10-gig connections across multiple frontends, if you're trying to run in an active/active configuration, you can actually end up severely bottlenecking the inter-node link to the point where clustering traffic gets lost, network buffers fill up, and the entire EVS falls over. In cluster environments where you have sufficient orchestration control over the mount points, always talk directly to the node that owns the FS underneath the export unless you have a terrific reason not to.

That said, the support is pretty good and the product is fairly robust and reliable. We ran a few petabytes with relatively few issues for how much traffic we pushed over it.

Zephirus posted:

Managing bossock fiber workload becomes a ballache if the storage can't keep up too.

Seventh Arrow posted:

I work in the NAS department for a bank and we've been a NetApp shop ever since we've had NAS (we have about 30 filers) - except now, we're going to be switching to Hitachi's HNAS platform. It's kind of interesting, I never even knew Hitachi had a NAS solution. We've done some of the training so far and it's kind of cool to see a different architecture, because they're both kind of different. So far, though, it seems like Hitachi EVS = NetApp vfilers and Hitachi virtual volumes = NetApp qtrees. This is important, because qtrees are our bread and butter. Some of our old qtrees have grown into disorganized monstrosities, so this might be a good chance to migrate some stuff into a more orderly setup.

Richard Noggin posted:

What are people using for storage switches in very small deployments (2-3 hosts, 5-15 VMs, iSCSI SAN, vSphere)? We've been using Catalyst 3750-Xs stacked, but only because they've been solid.

Richard Noggin posted:

What are people using for storage switches in very small deployments (2-3 hosts, 5-15 VMs, iSCSI SAN, vSphere)? We've been using Catalyst 3750-Xs stacked, but only because they've been solid.

Richard Noggin posted:

What are people using for storage switches in very small deployments (2-3 hosts, 5-15 VMs, iSCSI SAN, vSphere)? We've been using Catalyst 3750-Xs stacked, but only because they've been solid.

Aquila posted:

I'm so sorry. I only used a Hitachi FC SAN once, they threw in a free HNAS head but we declined to use it or even take it out of the box. Best practices that our Hitachi consultant described were ridiculous at our scale. Administration of HDS filers is so hilariously bad compared to a Netapp that you're probably going to want to kill yourself soon.

Seventh Arrow posted:

Bad in what way? Is the hardware itself bad, or is it just the management tools?

NippleFloss posted:

Hitachi hardware is rock solid and stable, if not particularly cutting edge. Hitachi software is a collapsing building built on top of a landfill built on top of an Indian burial ground.

Zephirus posted:

NippleFloss posted:

4900 series switches will handle storage traffic better than 3750s owing to a much larger shared port buffer space. Bursty traffic or mismatched egress/ingress rates (all common for network storage) can overload the relatively small buffers on the 3750s and lead to packet drops.

Zephirus posted:

This is pretty much correct. The newer command suite, command director, and tuning manager versions are generally approaching usability in terms of day to day use, but actual setup and maintenance is still labyrinthine and un-intuitive (especially tuning manager). They do look pretty in the new darker colour scheme though.

Historically the whole thing is a bit of a mess, the worst of which is things like the old AMS2000/HUS web console and the USP-V java console (which is so slow and awful to use that most people prefer to use the command line tools, which are also hilariously difficult to comprehend - especially HORCM). Command suite was developed because the USP management was so awful they decided to just go around it with a web-based tool.

You had to have shelled out at least 7 figures to see the below console. Not pictured is the 30 second wait caused by clicking anything.




The NAS web console is relatively unchanged from the blue-arc days, other than colourscheme changes. If you are using the NAS console you are probably doing it wrong though, as SSC/Bali command line is very good and actually quite intuitive.