Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
Docjowles
Apr 9, 2009



Misogynist posted:

Yahoo! just announced plans yesterday to use Ceph to underpin the media storage for Flickr and Tumblr: http://www.theregister.co.uk/2015/04/15/yahoo_plans_storage_service_on_ceph/

Funny. I work at a Flickr competitor and one of our sysadmins really, REALLY wanted us to switch to Ceph for image and video storage. He built a few POC's over the last couple years and honestly it looked really promising on paper. The CTO shot it down every time, though, and I think it was the right call. Yahoo presumably has dozens of engineers they can devote to this project. For us it would have been, like, one. For a cluster storing billions of objects adding up to many petabytes of data. We just didn't have the manpower to build and support something like that in-house. This guy is brilliant but he had a giant blind spot around financial tradeoffs. Only looking at the amount of money we'd have to pay for hardware and support. Not including the salaries of everyone who would have to spend huge amounts of their time building and running the system vs just opening a ticket and moving on to something productive. He very seriously proposed we build our own nationwide CDN, too, for example, because it would lower our monthly bill to $vendor.

He ended up getting poached to build another company's Ceph solution (not Yahoo) instead and is now making $$$$$$$$$$ so he got the last laugh there, I guess.

Adbot
ADBOT LOVES YOU

Zephirus
May 18, 2004

GAOoooooh!!


Docjowles posted:

Funny. I work at a Flickr competitor and one of our sysadmins really, REALLY wanted us to switch to Ceph for image and video storage. He built a few POC's over the last couple years and honestly it looked really promising on paper. The CTO shot it down every time, though, and I think it was the right call. Yahoo presumably has dozens of engineers they can devote to this project. For us it would have been, like, one. For a cluster storing billions of objects adding up to many petabytes of data. We just didn't have the manpower to build and support something like that in-house. This guy is brilliant but he had a giant blind spot around financial tradeoffs. Only looking at the amount of money we'd have to pay for hardware and support. Not including the salaries of everyone who would have to spend huge amounts of their time building and running the system vs just opening a ticket and moving on to something productive. He very seriously proposed we build our own nationwide CDN, too, for example, because it would lower our monthly bill to $vendor.

He ended up getting poached to build another company's Ceph solution (not Yahoo) instead and is now making $$$$$$$$$$ so he got the last laugh there, I guess.

Can I ask what you actually use now?

Docjowles
Apr 9, 2009



We currently run good old NFS on a loooooooot of NetApp filers. Due to some of our executives having relationships with NetApp we got them for a song and it was great for a time. But the situation has changed and we can't really afford to keep refreshing those.

We're now in the process of moving maybe 3/4 of our storage footprint to a cloud object storage vendor. Only data that's very sensitive to performance/latency will remain onsite.

evol262
Nov 30, 2010
#!/usr/bin/perl

Docjowles posted:

We currently run good old NFS on a loooooooot of NetApp filers. Due to some of our executives having relationships with NetApp we got them for a song and it was great for a time. But the situation has changed and we can't really afford to keep refreshing those.

We're now in the process of moving maybe 3/4 of our storage footprint to a cloud object storage vendor. Only data that's very sensitive to performance/latency will remain onsite.

Seems like onsite swift would be perfect, but I guess most of your objects probably don't get accessed very often...

Docjowles
Apr 9, 2009



evol262 posted:

Seems like onsite swift would be perfect, but I guess most of your objects probably don't get accessed very often...

We (as of a couple years ago) store a very high res copy of the photo which users can use to order prints and that sort of thing. Those are relatively huge, and accessed very infrequently. That's the bulk of what we're moving to cloud storage. Then we have a "web size" copy which is what you generally see browsing around the site or linked out on blogs/Reddit/whatever. Those are accessed very frequently, although of course there is an enormous long-tail.

We did look at onsite Swift alongside Ceph. It just had the same problem, though. Our Ops team isn't big enough to support a hand-rolled system like that, it's kind of ridiculous the amount of traffic we serve with a handful of staff. And management wasn't interested in adding headcount. So we went with a hosted solution and let someone else worry about it.

TeMpLaR
Jan 13, 2001

"Not A Crook"

Just wondering, about how many filers and TB's are you all monitoring?

I am somewhere around 6PB, with 20ish filers and a few more 8040's due to arrive this month.

SpaceRangerJoe
Dec 24, 2003

The little hand says it's time to rock and roll.

What do you guys use for naming conventions on virtual disks and luns? I got a DAS SAS storage device from Dell the other day, but I don't have any great ideas for naming. I'm thinking some combination of raid level, storage pool and connected endpoint. That's probably overkill. The appliance is only connected to two hosts right now with no shared storage pool between the hosts.

There is a raid 10 on SSD and some 15k SAS that will be raid 5.

bigmandan
Sep 11, 2001

lol internet

College Slice

SpaceRangerJoe posted:

What do you guys use for naming conventions on virtual disks and luns? I got a DAS SAS storage device from Dell the other day, but I don't have any great ideas for naming. I'm thinking some combination of raid level, storage pool and connected endpoint. That's probably overkill. The appliance is only connected to two hosts right now with no shared storage pool between the hosts.

There is a raid 10 on SSD and some 15k SAS that will be raid 5.

I'm not sure if its the best naming convention, but we do something like LUN###-<role>-<storage_characteristic+> and it seems to be serving us well.

Some examples:

code:
LUN013-vm-storage			stores vmware images, tier 3
LUN013-vm-fast				stores vmware images, flash optimized storage tier
LUN013-vm-critical			stores vmware images, tier 3, frequent snapshots/replays
LUN013-files-fast-critical		file storage, flash optimized, frequent snapshots/replays
Adjust the storage characteristic identifiers to fit your environment.

Zorak of Michigan
Jun 10, 2006

Waiting for his chance

Does anyone have any experience with Scality's RING software defined storage solution? They're giving one of my coworkers a pretty aggressive sales push.

the spyder
Feb 18, 2011


Well this week I learned what 2.8 PB of Iscilon costs (X and HD)- and we're probably ordering it next week. So much for Gluster/Ceph.

mayodreams
Jul 4, 2003


Hello darkness,
my old friend


the spyder posted:

Well this week I learned what 2.8 PB of Iscilon costs (X and HD)- and we're probably ordering it next week. So much for Gluster/Ceph.

Can you ball park it? I am really curious.

the spyder
Feb 18, 2011


mayodreams posted:

Can you ball park it? I am really curious.

Ended up being closer to 2.5PB Raw/ 2.1PB Usable after removing a x410. Let's just say it was just over $1mil with 3yr warranty. We will most likely be buying an identical unit for our other office.

Moey
Oct 22, 2010

I LIKE TO MOVE IT


the spyder posted:

Ended up being closer to 2.5PB Raw/ 2.1PB Usable after removing a x410. Let's just say it was just over $1mil with 3yr warranty. We will most likely be buying an identical unit for our other office.

Is this that new job with a bunch of different old arrays everywhere?

the spyder
Feb 18, 2011


Yes. We actually have a 700TB Iscilon, but due to several issues, we can't upgrade it. Plus the maintenance is over $350k the next 3 years. We literally have had almost everything. I found some 3par and Bluearc this week, buried in some decom'd racks.

Moey
Oct 22, 2010

I LIKE TO MOVE IT


I would love and hate your job at the same time.

the spyder
Feb 18, 2011


Moey posted:

I would love and hate your job at the same time.

Welcome to my realization. I won't rant here, but it's a mess in many ways.

FISHMANPET
Mar 3, 2007

Sweet 'N Sour
Can't
Melt
Steel Beams


FISHMANPET posted:

Physical server, attached with Fibre Channel to a Hitachi SAN. But apparently the same issue is happening on a local 10k SAS disk and also a FusionIO card.

But, it turns out, this request is coming from a production system running unrelease Commvault software in an experimental configuration. I assumed that we were doing normal stuff and other customers were doing this just fine, but nobody is doing this at the scale we are.

So tl;dr; maybe not a problem, backup guy is a poo poo.

An ending to this story. Backup guy is in fact a poo poo, we burnt a ton of Microsoft support hours for nothing (though storage team is paying for them), other than to determine that Windows is working as expected. Backup team is going to have to get off his rear end and engineer a system that isn't garbage rather than unendingly blaming his performance problems on other people.

He wants to try and replicate this on a 2008 server for his own personal edification, but at this point I don't think anybody has his back.

Xenomorph
Jun 13, 2001


I have a weird setup and question here:

We got an old Dell PowerEdge R710 to use as a file server. We requested it with the Dell SAS 6/iR RAID card instead of the Dell H700 card because we wanted direct access to the disks for ZFS. The "nicer" Dell cards do not let you disable RAID to use JBOD. Their "cheaper" RAID cards do.

I get the server and fill it with a bunch of new HDDs. I notice that only 2.0TB of the drives are showing up (they are 5TB drives). I look up more info, and I see that while the SAS 6/iR gives us "direct" access to the drives, it cannot access more than 2.0 TB. The H700 would let us work with the whole drive, but we'd lose direct-access (JBOD).

Someone suggested the LSI SAS9211-8i card. It cheap and works like the SAS 6/iR, but supports >2TB drives. So I pop that in, and it seems to work fine. All drives are available with their full capacity... But now my HDD power LEDs are not on. Activity LEDs flashes when the drive is in use, but the power LED is off.

Is this some Dell backplane + Non Dell backplane issue? Is there any way to make the HDD power LEDs come on?

Potato Salad
Oct 23, 2014

Nobody Cares




FISHMANPET posted:

An ending to this story. Backup guy is in fact a poo poo, we burnt a ton of Microsoft support hours for nothing (though storage team is paying for them), other than to determine that Windows is working as expected. Backup team is going to have to get off his rear end and engineer a system that isn't garbage rather than unendingly blaming his performance problems on other people.

He wants to try and replicate this on a 2008 server for his own personal edification, but at this point I don't think anybody has his back.

Good to hear this came to a close. From the outset, it sounded like the Win host was being asked to do something it shouldn't be asked to do at the OS level.

Was it really that Commvalut was literally copying tens of thousands of files with each job, and the backup engineer didn't see that as a scaling problem?

FISHMANPET
Mar 3, 2007

Sweet 'N Sour
Can't
Melt
Steel Beams


Exactly that. He's still not convinced that it's not a problem, but nobody is willing to devote resources to tracking down this supposed problem. He kept going back to "well the SAN was only showing 16 IOPS during the copy" but it's not really an IOPS problem. At least now I have a "backup guy is poo poo" story just like everyone else in the office does. He keeps trying to blame this problem on other people. "Oh it's a Windows problem because Windows can't handle it." "Oh its VMware team problem because they made me consolidate backup groups." Yeah, sure, you do you.

Potato Salad
Oct 23, 2014

Nobody Cares




FISHMANPET posted:

Exactly that. He's still not convinced that it's not a problem, but nobody is willing to devote resources to tracking down this supposed problem. He kept going back to "well the SAN was only showing 16 IOPS during the copy" but it's not really an IOPS problem. At least now I have a "backup guy is poo poo" story just like everyone else in the office does. He keeps trying to blame this problem on other people. "Oh it's a Windows problem because Windows can't handle it." "Oh its VMware team problem because they made me consolidate backup groups." Yeah, sure, you do you.

Just because the solution that worked a decade ago doesn't work today doesn't make him or her poo poo outright -- just means s/he needs a kick in the rear end to look at another solution.

Meh, who am I kidding, fucker can't see the problem clear in front of him/her.

You said you had something on the order of petabytes of data to back up? I'd have to think that justifies having at least another backup engineer on the team. If you went with an out-of-the-box solution like that which is avilable from Unitrends, for example, you'd end up buying 35 of their largest backup devices (Recovery-943S) for 2 PB of data. That much data is no small demon to slay.

Potato Salad
Oct 23, 2014

Nobody Cares




Guy probably needs help from someone who understands modern distributed storage and backup methods at a very deep level. I only know enough that I wouldn't really have a clue where to start for that volume of data and would instead need to hire someone.

FISHMANPET
Mar 3, 2007

Sweet 'N Sour
Can't
Melt
Steel Beams


This particular system may only be backing up our VM infrastructure, but that's thousands of machines, and not our huge pile of data on our Isilon. But I'm only vaguely aware of the details of all of that.

E: I see in the VM thread that you are also an employee of a large public university. So I'm sure you can relate to all sorts of things I'm going through here.

FISHMANPET fucked around with this message at 06:23 on May 6, 2015

Strife
Apr 20, 2001

What the hell are YOU?

Fiber zoning question. Maybe this is the place for this.

I inherited a fiber channel 3PAR SAN that's connected to two brocade switches, which are then connected to three hosts running VMware. Each of the controllers on the 3PAR has multiple fiber ports, but only one of those ports per controller is connected to a brocade switch.

I want to increase the bandwidth to the SAN but I know precisely dick about fiber zoning beyond "it's similar to a VLAN." The Brocade switches are only used for connecting the hosts to the 3PAR, so do I need to do anything beyond relying on their default configs? Is there some fiber equivalent of port aggregation or can I just plug them in?

Syano
Jul 13, 2005


You need to know how zoning is configured on your switches. You are most likely using soft zoning (cause no one uses hard/port zoning.. but be sure). With soft zoning, you are going to build a zone on your switch which will be a "container" that matches WWNs on the SAN ports to the WWNs on your VM hosts. Anyways, as long as your zones contain the WWNs for the ports you want to connect, the FC protocol will take care of everything else.... including multipathing/login/etc

EDIT: Also look to see if you have an ISL link between your switches

devmd01
Mar 7, 2006

Elektronik
Supersonik


And make sure you're using the correct pathing policies on your data stores per the VMware HCL, bit me in the rear end big time a few weeks ago.

1000101
May 14, 2003

BIRTHDAY BIRTHDAY BIRTHDAY BIRTHDAY BIRTHDAY BIRTHDAY FRUITCAKE!

Strife posted:

I want to increase the bandwidth to the SAN but I know precisely dick about fiber zoning beyond "it's similar to a VLAN." The Brocade switches are only used for connecting the hosts to the 3PAR, so do I need to do anything beyond relying on their default configs? Is there some fiber equivalent of port aggregation or can I just plug them in?

Zoning is more closely related to creating access lists than it is a VLAN. You generally want to make sure you're using single initiator zones with either single targets or multiple targets. By that I mean an initiator would basically be a server and a target would be some sort of storage device, tape library, whatever.

So if I have say 2 servers and a storage array I would define 2 zones using a naming convention not unlike below:

SERVERHOSTNAME1_HBA0__3PARHOSTNAME
member SERVERHOSTNAME1_HBA0
member 3PARHOSTNAME_CONTROLLER1_PORT1
member 3PARHOSTNAME_CONTROLLER1_PORT2
member 3PARHOSTNAME_CONTROLLER2_PORT1
member 3PARHOSTNAME_CONTROLLER2_PORT2

SERVERHOSTNAME2_HBA0__3PARHOSTNAME
member SERVERHOSTNAME2_HBA0
member 3PARHOSTNAME_CONTROLLER1_PORT1
member 3PARHOSTNAME_CONTROLLER1_PORT2
member 3PARHOSTNAME_CONTROLLER2_PORT1
member 3PARHOSTNAME_CONTROLLER2_PORT2

Then add those zones to the zoneset and activate it. Brocade should support the notion of device aliases or faceless.. Use them or you will hate life when you go to troubleshoot and you find yourself staring at WWPNs instead of something that is meaningful to you. Repeat this process for the B fabric.

Don't be tempted to throw all your servers in one big happy zone and call it a day as this can cause problems for you down the road.

edit: I could give you CLI syntax for a Cisco MDS but I haven't touched a brocade in a good long while. Should be mostly the same though.

quote:

You need to know how zoning is configured on your switches. You are most likely using soft zoning (cause no one uses hard/port zoning.. but be sure). With soft zoning, you are going to build a zone on your switch which will be a "container" that matches WWNs on the SAN ports to the WWNs on your VM hosts. Anyways, as long as your zones contain the WWNs for the ports you want to connect, the FC protocol will take care of everything else.... including multipathing/login/etc

I hate to be a pedantic dickface but hard zoning/soft zoning really refers to how zoning gets enforced (is it just via FCNS or actually making sure the ASIC doesn't forward the traffic). Even when you do WWNN/alias based zoning it's still enforced on hardware on brocade and MDS switches (and probably QLogic and other OEM brands as well.) People often mistake hard zoning for port based zoning (which I agree do not do!.)

That's the end of my pedantry!

edit 2: here's an article that explains for me! http://searchstorage.techtarget.com/tip/Zoning-part-2-Hard-zoning-vs-soft-zoning

1000101 fucked around with this message at 18:30 on May 7, 2015

Strife
Apr 20, 2001

What the hell are YOU?

So there's really nothing more to it than creating zones and adding the WWNs to each zone?

Seems simple enough, really. I create zones for each one of my hosts' HBAs to the 3PAR, and then add the additional WWN for the other port on the 3PAR when I uplink it to the Brocade.

Like this?



Using the GUI because I don't know what the hell language the command line is in and it's confusing as hell.

1000101
May 14, 2003

BIRTHDAY BIRTHDAY BIRTHDAY BIRTHDAY BIRTHDAY BIRTHDAY FRUITCAKE!

Brocade FabricOS! In some ways I like it better than MDS but yeah you've got the gist of it. Brocade's GUI fabric manager is leaps and bounds better than DCNM though.

If you can, consider adding some aliases for your devices to make those screens a little readable.

Also a reminder that you probably have 2 switches that aren't connected to each other so you'll need to repeat the exercise on both sides. This is by design to make sure you have redundancy in your storage network (in case you screw up zoning somehow for example your servers will still have a path available)

Cidrick
Jun 10, 2001

Praise the siamese


1000101 posted:

If you can, consider adding some aliases for your devices to make those screens a little readable.

I highly recommend this as well. Nothing's more annoying when you're trying to troubleshoot FC issues and you're looking at a bunch of WWPNs with zero idea on what host that actually is without having to log onto every single attached box and comparing hex addresses.

Here's my Brocade FabricOS cheat sheet I use since the CLI is much much faster to do bulk changes, if you're feeling up to it. Remember you have to do this on each side, so change it up to "path1" for your second switch, or however you choose to name it.

code:
# Show a list of attached ports and WWNs
switchshow
# Find a port attached to the switch
# You can give a full WWPN or a partial one works too
nodefind 50:01:0c
# Create a new alias
alicreate "foohost_path0", "WWPN1"
# Add to existing alias
aliadd "foohost_path0", "WWPN1"
# Add alias to a new zone
zonecreate "foohost_path0_zone", "foohost_path0"
# Add alias to existing zone
zoneadd "foohost_path0_zone", "foohost_path0"
# Add the new zone to the config
cfgadd "Name_Of_My_SAN_Config", "foohost_path0_zone"
# Save the config
cfgsave
# Enable this new config, applying changes and loading it into the running config
cfgenable "Name_Of_My_SAN_Config"
As 1000101 said, in a nutshell, create an alias for every WWN, create a zone and add the alias for your hosts along with the alias for your SAN, and then save and enable your changes.

Strife
Apr 20, 2001

What the hell are YOU?

1000101 posted:

Also a reminder that you probably have 2 switches that aren't connected to each other so you'll need to repeat the exercise on both sides. This is by design to make sure you have redundancy in your storage network (in case you screw up zoning somehow for example your servers will still have a path available)

Oh, yeah, of course. I'm going to enable the zones on one switch, make sure my host can still see the storage on the HBAs connected to that switch, and then do the other.


Sweet.

Thanks for the info guys.

Syano
Jul 13, 2005


1000101 posted:



I hate to be a pedantic dickface but hard zoning/soft zoning really refers to how zoning gets enforced (is it just via FCNS or actually making sure the ASIC doesn't forward the traffic. Even when you do WWNN/alias based zoning it's still enforced on hardware on brocade and MDS switches (and probably QLogic and other OEM brands as well.) People often mistake hard zoning for port based zoning (which I agree do not do!.)

That's the end of my pedantry!

edit 2: here's an article that explains for me! http://searchstorage.techtarget.com/tip/Zoning-part-2-Hard-zoning-vs-soft-zoning

Oh gosh he gets the idea you pedant! Thanks for clarifying though :tipshat:

mattisacomputer
Jul 13, 2007

Philadelphia Sports: Classy and Sophisticated.



Also note that adding three connections to the switch for one device just gives it three separate paths at that speed, not a combined aggregate. If your VMWare path selection policy isn't set to something that utilizes multiple active ports, adding additional connections is just creating additional redundancy, but not providing any additional bandwidth.

mattisacomputer fucked around with this message at 18:22 on May 7, 2015

Pile Of Garbage
May 28, 2007





The only thing I miss about my previous job was working with Brocades and FC fabrics. At my current job it's all NFS/iSCSI :sigh:

Cidrick
Jun 10, 2001

Praise the siamese


cheese-cube posted:

The only thing I miss about my previous job was working with Brocades and FC fabrics. At my current job it's all NFS/iSCSI :sigh:

I'm kind of jealous, to be honest. FC is a great, robust protocol, but with 10GB ethernet being so cheap and simple to work with, and the fact that a lot of shops are moving away from big expensive SANs to a more appliance or commodity-based storage platform, I've been touching it a lot less and less these days. My next dream is to set up distributed ceph or gluster distributed storage platform across several rows of 2U servers packed with drives and using something like opendedup on top of it.

It'll probably never happen, but I can dream.

Strife
Apr 20, 2001

What the hell are YOU?

So, maybe a stupid question, maybe not, but should I have each host hba be in a zone with both 3PAR controllers, or should I make a separate zone for each host to each controller? Either way the host sees twice as many paths back to the storage as it did earlier.

Syano
Jul 13, 2005


Strife posted:

So, maybe a stupid question, maybe not, but should I have each host hba be in a zone with both 3PAR controllers, or should I make a separate zone for each host to each controller? Either way the host sees twice as many paths back to the storage as it did earlier.

We do per hba zoning.... On every switch 1 zone per hba per host and then add one port per controller to that zone. Then its just like old school point to point which FC loves except with full multipath features

Edit: unless its our vmax fabric in which zoning is built to match the port group configuration

Syano fucked around with this message at 01:46 on May 8, 2015

Strife
Apr 20, 2001

What the hell are YOU?

Syano posted:

We do per hba zoning.... On every switch 1 zone per hba per host and then add one port per controller to that zone. Then its just like old school point to point which FC loves except with full multipath features

Alright cool. At least I know what I'm doing is a thing. VMware handles the path back with RR, and as long as it has multiple paths back to each LUN I assume it's redundant. So now I can suffer a controller failure and a brocade switch failure.

GrandMaster
Aug 15, 2004
laidback

Strife posted:

So, maybe a stupid question, maybe not, but should I have each host hba be in a zone with both 3PAR controllers, or should I make a separate zone for each host to each controller? Either way the host sees twice as many paths back to the storage as it did earlier.

Check with your array vendor.. EMC best practice is 1 HBA, 1 controller port per zone.
So as a minimum each host would have the following added to the zone set:

Fabric A
HBA0 -> SPA P0
HBA0 -> SPB P0

Fabric B
HBA 1 -> SPA P1
HBA 1 -> SPB P1

I'm not an FC expert, but i believe this reduces the impact of RSCN notifications for fabric changes, and also VNX/CX arrays crack the shits and spam error messages as their controller ports can either be an initiator or a target if they are using mirrorview/sancopy and they keep trying to log into each other.

Adbot
ADBOT LOVES YOU

YOLOsubmarine
Oct 19, 2004

When asked which Pokemon he evolved into, Kamara pauses.

"Motherfucking, what's that big dragon shit? That orange motherfucker. Charizard."



If it's a newer version of the firmware you should have port persistence (NPIV) available/enabled so you would connect a port and its backup port on the other controller to the same fabric and then add the virtual wwpn to the zone along with a single host wwpn.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply