Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
YOLOsubmarine
Oct 19, 2004

When asked which Pokemon he evolved into, Kamara pauses.

"Motherfucking, what's that big dragon shit? That orange motherfucker. Charizard."

madsushi posted:

In general, the 2xxx series doesn't have any PCI-E slots for expansion slots. So the 2020/2040/2050/2220/2240 can't support FlashCache (PAM). I don't think it has to do with the memory, it has to do with the fact that they don't have the right slot. In addition, there was an issue with some of the older 3xxx series that prevented FlashCache from working after you upgraded to 8+.

None of the 2xxx series support flashcache for the stated reason that they don't have expansion slots, however the 22xx series does support flashpools. Generally the reason why the 20xx and 3140/3210 don't support flash of either type (the 3210s support a small amount, with caveats, on 8.1.1) is because flashcache and flashpool both drive memory utilization up. The table that maintains the list of blocks in cache, and their locations is stored in system memory. When you start talking about hundreds of gigs worth of 4k blocks that table can get pretty large.

Systems with 4GB of system memory or less just can't handle the additional memory pressure that the cache table adds. It would tank the performance of the box in a lot of cases and cause WAFL panics due to out of memory conditions if it got bad enough.

You'll notice that the systems that have come out in the last year or two generally have at least double the memory of those systems that they are replacing, and often more than that. The 3220 has 3 times the memory the 3210 did, and the 3250 2.5 times the amount the 3240 had. A lot of that is driven by needing that extra memory to deploy larger amounts of flashpool and flashcache.

Adbot
ADBOT LOVES YOU

madsushi
Apr 19, 2009

Baller.
#essereFerrari

NippleFloss posted:

Systems with 4GB of system memory or less just can't handle the additional memory pressure that the cache table adds. It would tank the performance of the box in a lot of cases and cause WAFL panics due to out of memory conditions if it got bad enough.

Good information, makes sense. One of the reasons I was excited about ONTAP Edge, the thought of loading a VM with 32GB of RAM and letting it go wild. :)

Wompa164
Jul 19, 2001

Don't write ghouls.

GrandMaster posted:

We've been looking at nimble boxes too, but I was surprised at how expensive they were considering it's full of lovely SATA. We are looking at ~100TB of storage, and it came in more expensive than Compellent, VNX5500 & FAS3250 boxes with similar capacity - the other boxes take up more space but I've got much more confidence around the performance since they all have truckloads of 15K SAS & SSD caching.

I'm concerned about how some of the workloads would perform on a Nimble like some of our OLTP/OLAP etc apps. I'm sure VMware/VDI would run pretty quick though.

Would you minding sharing your quote for the FAS3250 w/ 100TBs? I'm trying to get a feel for pricing for my current project.

GrandMaster
Aug 15, 2004
laidback
I can't be too specific, commercial in confidence and all that, but the first quote we got was over $400k, after some discussion we got it below $300k (AUD).

spoon daddy
Aug 11, 2004
Who's your daddy?
College Slice

Wompa164 posted:

Would you minding sharing your quote for the FAS3250 w/ 100TBs? I'm trying to get a feel for pricing for my current project.
one thing to keep in mind when negotiating. flash cache/pam is cheap for netapp. they have a lot of flexibility with their pricing on those. For a large deal, they gave me a buy 1 get 1 free price and that was after a 70% discount

edit: get a quote from oracle's zfs and tell netapp the #. They gave me 5 points right off the bat. Oracle is in land grab mode so they are practically giving away their zfs appliances. this in turn scares the poo poo out of netapp since oracle is positioning themselves as a netapp killer. IMO netapp is much more mature and feature rich. Oracle has a lot of "in 6 months" features.

spoon daddy fucked around with this message at 10:56 on Dec 7, 2012

evil_bunnY
Apr 2, 2003

I don't think I could resist asking an oracle rep how the Solaris core dev team is doing if I ever had one in my office :laugh:

Amandyke
Nov 27, 2004

A wha?

evil_bunnY posted:

I don't think I could resist asking an oracle rep how the Solaris core dev team is doing if I ever had one in my office :laugh:

Might be even better to ask how their racing yacht is doing...

luminalflux
May 27, 2005



"When will we be getting all the cool ZFS features they have in Illumos?" :v:

Goon Matchmaker
Oct 23, 2003

I play too much EVE-Online
Anyone know if the VNXe3300 supports SCSI-3 Persistent Reservations? We keep getting an error setting up MSCS saying that SCSI-3 Persistent Reservations failed. I've set the guests up in ESXi as per the documentation on VMware's website and it keeps failing that check...

Dilbert As FUCK
Sep 8, 2007

by Cowcaster
Pillbug

Goon Matchmaker posted:

Anyone know if the VNXe3300 supports SCSI-3 Persistent Reservations? We keep getting an error setting up MSCS saying that SCSI-3 Persistent Reservations failed. I've set the guests up in ESXi as per the documentation on VMware's website and it keeps failing that check...

Should, quick google brings up this https://community.emc.com/message/645058

skipdogg
Nov 29, 2004
Resident SRT-4 Expert

Totally new to SAN stuff, but my boss has me sitting in on a EMC RecoverPoint info/training session. Seems pretty cool. They're working on getting the VM's to fail over to a different site. I'm totally out of my element here.

madsushi
Apr 19, 2009

Baller.
#essereFerrari
I have a lot of VMware datastores that are sitting on NFS volumes on NetApp filers.

Sometimes, the snapshots from these datastores are reasonably sized, like 5-10GB per day. Sometimes, the snapshots from these datastores are unreasonably sized, like 90-100GB per day.

I am trying to find the best way to isolate the VMs that are causing the largest deltas. I have used the built-in VMware monitoring to looking for high write OPs VMs, but haven't found much correlation to snapshot size.

Are there any useful tools (on the VMware or the NetApp side) for monitoring individual VM writes? I just want to figure out which VMs are dumping SQL backups or running defrag and screwing up my replication windows. This is probably a good question for the VMware thread, but I love you all more.

e: and please don't suggest moving VMs around into temporary datastores, because I have a LOT of datastores to manage and since I have to wait 2 days between moving VMs to see if the snapshot size budged, it's really not feasible.

GrandMaster
Aug 15, 2004
laidback
Does anyone have experience with Compellent? We are considering making the move from EMC, but I'm a little hesitant as EMC is all I've ever known and we've had a pretty good experience from them overall. Cost is obviously the driving factor here, but since it will be much easier for us to migrate onto another EMC system and they have already been proven then they are still in the game.

I've seen a few gotchas in this thread about the VNX series, but not a lot around Compellent so I was wondering if anyone could share their good/bad experiences?
On paper they look pretty good.

We are running ~100TB, workloads are typically: Oracle OLTP, MSSQL OLTP & OLAP, ESX & vCloud Director for self service vm provisioning.

luminalflux
May 27, 2005



Welp, that's the first time resyncing volumes during a LeftHand upgrade has made the VMs on them stop working.

amishpurple
Jul 21, 2006

I'm not insane, I'm just not user-friendly!

GrandMaster posted:

Does anyone have experience with Compellent? We are considering making the move from EMC, but I'm a little hesitant as EMC is all I've ever known and we've had a pretty good experience from them overall. Cost is obviously the driving factor here, but since it will be much easier for us to migrate onto another EMC system and they have already been proven then they are still in the game.

I've seen a few gotchas in this thread about the VNX series, but not a lot around Compellent so I was wondering if anyone could share their good/bad experiences?
On paper they look pretty good.

We are running ~100TB, workloads are typically: Oracle OLTP, MSSQL OLTP & OLAP, ESX & vCloud Director for self service vm provisioning.

I went from a CX-340 to a Compellent last year and it was amazing. There's a few things to keep in mind when going Compellent. As I'm sure you've been told a thousand times by the sales reps already, if you use the Default/Recommended storage profile all your writes will come in on your Tier 1 disk. This means you need to size your Tier 1 accordingly so you don't get poo poo performance. The net result can be a good amount of wasted fast disk. You can pin LUN's to certain tiers (e.g. Tier 3 only) though.

The automated tiering is slow and its designed that way. It will take a few weeks after you get on the system before things move to Tier 2/3, and similarly for any new LUN's. I never had any issues with performance though, as things that need to be in Tier 1 seem to stay there. However, if you have jobs/workloads that run very infrequently you will need to remember to move these LUNs to Tier 1 in advance (you do this by adjusting the storage profile) if they need the performance. Again, you'll need to plan for this when sizing Tier 1.

In order for the tiering to work you need to make sure the LUN's are being snapped (a Replay in Compellent terms). This is just a small gotcha I didn't find out until after the fact. Basically the frozen, snapped pages are what get moved automagically. In post-sales they should explain this to you and give you some suggestions.

That's what I got off the top of my head. It pretty much works as advertised and works really well. Enterprise Manager absolutely destroys EMC (everyone?) in reporting/visibility into the system. Setting up replication is dead simple, licensing is pretty straightforward, and the application consistent snapshot stuff works well also.

amishpurple fucked around with this message at 16:31 on Dec 12, 2012

Amandyke
Nov 27, 2004

A wha?

GrandMaster posted:

Does anyone have experience with Compellent? We are considering making the move from EMC, but I'm a little hesitant as EMC is all I've ever known and we've had a pretty good experience from them overall. Cost is obviously the driving factor here, but since it will be much easier for us to migrate onto another EMC system and they have already been proven then they are still in the game.

I've seen a few gotchas in this thread about the VNX series, but not a lot around Compellent so I was wondering if anyone could share their good/bad experiences?
On paper they look pretty good.

We are running ~100TB, workloads are typically: Oracle OLTP, MSSQL OLTP & OLAP, ESX & vCloud Director for self service vm provisioning.

What do you mean by gotchas and do you need block of file level access?

GrandMaster
Aug 15, 2004
laidback
By gotchas I mean catastrophic failures/not living up to performance expectations.. anything like that.

We only need block over FC, but I think EMC might have have beat compellent by a fraction, mainly based around the ease of migration. We have a very short timeframe to migrate everything to a remote datacenter, and recoverpoint will make my life much easier than double handling (migrate to loan swing box/replicate remotely/cut over). The only other thing I need to know is how well the emc snap integration works with commvault.. netapp and compellent kept saying that their integration was the best, not sure why they are different but i'll get more details from our CV account manager tomorrow.

I also LOVED the netapp oncommand balance app that we demoed, i'm going to try to pick that up too if it fits in the budget even though we aren't using netapp storage. Being able to easily trace performance stats from VM->Physical Host->FC ports->LUN->RAID group was pretty fantastic.

Thanks for the feedback too amishpurple :)

GrandMaster fucked around with this message at 14:18 on Dec 13, 2012

YOLOsubmarine
Oct 19, 2004

When asked which Pokemon he evolved into, Kamara pauses.

"Motherfucking, what's that big dragon shit? That orange motherfucker. Charizard."

GrandMaster posted:

We only need block over FC, but I think EMC might have have beat compellent by a fraction, mainly based around the ease of migration. We have a very short timeframe to migrate everything to a remote datacenter, and recoverpoint will make my life much easier than double handling (migrate to loan swing box/replicate remotely/cut over). The only other thing I need to know is how well the emc snap integration works with commvault.. netapp and compellent kept saying that their integration was the best, not sure why they are different but i'll get more details from our CV account manager tomorrow.

If you requirements are basically FC and replication why are you limiting your inquiries to Compellant and VNX? There are a lot of vendors out there that can meet those requirements.

Regarding NetApp/Commvault integration, it's pretty nice. You can use Simpana to manage snapshot copies, replication sets, and backups to tape. So in a single policy you could define a backup flow that takes a snapshot copy -> replicates that copy to another NetApp via SnapMirror/SnapVault -> backs that copy up to tape and deduplicates it. It indexes snapshot contents, which is also nice.

Commvault and NetApp have a pretty tight partnership so the integration and support is solid.

psydude
Apr 1, 2008

Heartache is powerful, but democracy is *subtle*.
Does NetApp have a power consumption calculator? I can get real time readouts, but I'm looking for a nominal value.

psydude
Apr 1, 2008

Heartache is powerful, but democracy is *subtle*.
Apparently there isn't one, but if you like tedious tables and charts then the answers can all be found here.

dhrusis
Jan 19, 2004
searching...
hey guys.. I have a Dlink NAS-321... I have 2 mirrored drives, and all is well.. One has died and I had it replaced and it works.

One question I have though is what happens if the NAS-321 itself dies... does anyone have a protocol for accessing the data on the drives? Do I just load it into a linux machine, or can I hook it up to a windows box? Any help would be appreciated..

YOLOsubmarine
Oct 19, 2004

When asked which Pokemon he evolved into, Kamara pauses.

"Motherfucking, what's that big dragon shit? That orange motherfucker. Charizard."

dhrusis posted:

hey guys.. I have a Dlink NAS-321... I have 2 mirrored drives, and all is well.. One has died and I had it replaced and it works.

One question I have though is what happens if the NAS-321 itself dies... does anyone have a protocol for accessing the data on the drives? Do I just load it into a linux machine, or can I hook it up to a windows box? Any help would be appreciated..

Those Dlink appliances just use ext3 so you can just pop it into a linux host and get the data off without a problem. That said, future questions about it should go to this thread:

http://forums.somethingawful.com/showthread.php?threadid=2801557

complex
Sep 16, 2003

madsushi posted:

I have a lot of VMware datastores that are sitting on NFS volumes on NetApp filers.

Sometimes, the snapshots from these datastores are reasonably sized, like 5-10GB per day. Sometimes, the snapshots from these datastores are unreasonably sized, like 90-100GB per day.

I am trying to find the best way to isolate the VMs that are causing the largest deltas. I have used the built-in VMware monitoring to looking for high write OPs VMs, but haven't found much correlation to snapshot size.

Are there any useful tools (on the VMware or the NetApp side) for monitoring individual VM writes? I just want to figure out which VMs are dumping SQL backups or running defrag and screwing up my replication windows. This is probably a good question for the VMware thread, but I love you all more.

e: and please don't suggest moving VMs around into temporary datastores, because I have a LOT of datastores to manage and since I have to wait 2 days between moving VMs to see if the snapshot size budged, it's really not feasible.

Enable Changed Block Tracking and look for the biggest .CTK files.

skipdogg
Nov 29, 2004
Resident SRT-4 Expert

Looking for a good intro to SAN book. I know the absolute basics, but I guess I'm looking for more detail. I know what a LUN is, but what purpose does it have, why are they created. I know that iSCSI and Fibre Channel are connection protocols, but what makes them different and the advantages and disadvantages of each. Basically a good foundation book with some general best practices. It doesn't have to be vendor specific, but we're an EMC shop if it matters.

Amandyke
Nov 27, 2004

A wha?

skipdogg posted:

Looking for a good intro to SAN book. I know the absolute basics, but I guess I'm looking for more detail. I know what a LUN is, but what purpose does it have, why are they created. I know that iSCSI and Fibre Channel are connection protocols, but what makes them different and the advantages and disadvantages of each. Basically a good foundation book with some general best practices. It doesn't have to be vendor specific, but we're an EMC shop if it matters.

http://www.amazon.com/Information-S...d+Management+v2

Rhymenoserous
May 23, 2008

skipdogg posted:

Looking for a good intro to SAN book. I know the absolute basics, but I guess I'm looking for more detail. I know what a LUN is, but what purpose does it have, why are they created. I know that iSCSI and Fibre Channel are connection protocols, but what makes them different and the advantages and disadvantages of each. Basically a good foundation book with some general best practices. It doesn't have to be vendor specific, but we're an EMC shop if it matters.

1. A LUN is for all intents and purposes a block of unformatted storage that you present to a device as if it were a disk. It's purpose is to provide a lump of remote storage to a device so it appears as if it were just a new blank hard drive.

2. iSCSI goes over ethernet and is tied to whatever speeds that standard can do. Back when 1GE was pretty much as blazing fast as standard network equipment would go, people would use fibre channel in order to get high speed access to their storage devices. With 10GE reaching commodity pricing I can't think of a single loving reason to deal with fibre channel ever again unless I had already sunk money into it.

skipdogg
Nov 29, 2004
Resident SRT-4 Expert


This is beautiful. Thanks!

Dilbert As FUCK
Sep 8, 2007

by Cowcaster
Pillbug

skipdogg posted:

This is beautiful. Thanks!

If you have any questions on the specifics of it I have a copy at home so feel free to ask away.

YOLOsubmarine
Oct 19, 2004

When asked which Pokemon he evolved into, Kamara pauses.

"Motherfucking, what's that big dragon shit? That orange motherfucker. Charizard."

Rhymenoserous posted:

2. iSCSI goes over ethernet and is tied to whatever speeds that standard can do. Back when 1GE was pretty much as blazing fast as standard network equipment would go, people would use fibre channel in order to get high speed access to their storage devices. With 10GE reaching commodity pricing I can't think of a single loving reason to deal with fibre channel ever again unless I had already sunk money into it.

I'd still pick FC over iSCSI, all other things being equal. FC has a far superior flow control mechanism for handling storage traffic. It deal with congestion without dropping frames which is pretty important when SCSI essentially requires ordered delivery. Supporting buffer-to-buffer flow control as well as endpoint-to-endpoint means that FC deals better with fan-in problems than TCP/IP.

FC fabrics are generally more robust as well. Adding new devices to the fabric is trivial, you don't ever have to worry about loops or spanning tree problems, zoning provides an additional layer of security above masking, using a HW identifier like a WWN makes misconfiguration difficult (ever had someone reuse your iSCSI target IP address elsewhere on the network?), the stated data rate is actually achievable, etc.

There are a lot of little things that make FC a better storage protocol than iSCSI, which is should be since it was developed from the ground up to pass SCSI traffic and only SCSI traffic. These things won't make a bit of difference to the majority of storage clients, but there are still good reasons to use FC over iSCSI if you want the absolute best in performance and stability.

Amandyke
Nov 27, 2004

A wha?
Not to mention latency.

GrandMaster
Aug 15, 2004
laidback

skipdogg posted:

Looking for a good intro to SAN book. I know the absolute basics, but I guess I'm looking for more detail. I know what a LUN is, but what purpose does it have, why are they created. I know that iSCSI and Fibre Channel are connection protocols, but what makes them different and the advantages and disadvantages of each. Basically a good foundation book with some general best practices. It doesn't have to be vendor specific, but we're an EMC shop if it matters.


http://www.bltrading.com/pdf/featuredwp/emc.pdf
This was my bible when I was learning EMC. I just googled this one, but theres probably a newer version on powerlink if you are running something newer than the CX series.
It covers everything from RAID types & performance characteristics, workload types and why some might need different raid levels, clariion architecture, fc/iscsi fault tolerant networks & multipathing etc. Highly recommended reading.

zorachus
Sep 4, 2009
"fuck protein"
I have a bit of an odd situation at work.

The full write-up is here, but I'll summarize.

Over the last couple of years, we've had a handful of incidents wherein extra paths have shown up in powermt display dev=all output. These extra paths appear immediately after issuing a LIP:

code:
echo 1 | sudo tee /sys/class/fc_host/host?/issue_lip
Basically, the LIP queries the HBA driver for new devices. The SCSI driver then creates the generic SCSI (/dev/sg) devices and the corresponding named SCSI (/dev/sd) devices. PowerPath then sees these new devices and displays them as valid paths.

The strange thing here is that querying the HBA directly with scli -l {0,1} reports that the correct number of paths are presented to the host, but all of the paths are valid! For example, I can do an octal dump of every path (od -c /dev/sdax | head -10) or send a SCSI inquiry (sg_inq -v /dev/sdax). How could the frame be responding to these inquiries if it reports that the device I'm querying doesn't exist?

Has anyone seen this before? EMC is pointing the finger at Red Hat and Red Hat is pointing the finger at EMC, so we're kind of at a standstill. Luckily, there's no impact, but I'm wondering if there's anything we can do to put this issue to rest once and for all.

(I know it's lengthy, but the link above explains everything in much more detail, including part of EMC's response... If you're interested, it may be worth reading)

Powdered Toast Man
Jan 25, 2005

TOAST-A-RIFIC!!!
Some of you may recall my posts last year about the lovely Reldata NAS system I had to work with. I no longer work at that company, thank god, but I've kept in touch with some of my former coworkers.

It was working sort of OK except for lingering, weird permissions issues (like one particular AD user that had Full Control on the entire filesystem who shouldn't have and they never could seem to get rid of it), but I knew after looking at the specs of the system vs what it was being used for that they were pushing it way, way too hard. I predicted it would eventually fail catastrophically.

Earlier this week, it did exactly that. The sole file storage for every user profile and shared file in a company with 4,000+ employees just plain quit. If you're going to put all your eggs in one basket, at least don't use a basket made out of toilet paper.

evil_bunnY
Apr 2, 2003

What happened after?

Powdered Toast Man
Jan 25, 2005

TOAST-A-RIFIC!!!

evil_bunnY posted:

What happened after?

I don't give a gently caress, but I haven't heard yet. They were on conference calls with Starboard the whole week trying to resurrect it.

Nomex
Jul 17, 2002

Flame retarded.

evil_bunnY posted:

What happened after?

1 in 3 companies that suffer a catastrophic data loss never recover.

Vulture Culture
Jul 14, 2003

I was never enjoying it. I only eat it for the nutrients.
One of our NAS devices decided to randomly lose about 10-15 TB of data today out of a share. Not worried about it even a little bit.

Pile Of Garbage
May 28, 2007



Misogynist posted:

One of our NAS devices decided to randomly lose about 10-15 TB of data today out of a share. Not worried about it even a little bit.

Wow, do you know what caused it? What make/model is the NAS?

evil_bunnY
Apr 2, 2003

Misogynist posted:

Not worried about it even a little bit.
Yeah as much as I could recover from that in about 2 clicks I'd still be in log-scouring mode.

Adbot
ADBOT LOVES YOU

Vulture Culture
Jul 14, 2003

I was never enjoying it. I only eat it for the nutrients.

cheese-cube posted:

Wow, do you know what caused it? What make/model is the NAS?
Given that the snapshots are fine, and the other exports on the filesystem are fine, someone with access apparently fat-fingered an rm somewhere.

NAS is a BlueArc Titan cluster, but I'm pretty sure that isn't relevant here since the issue is local to the NFS export.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply