|
Misogynist posted:Yahoo! just announced plans yesterday to use Ceph to underpin the media storage for Flickr and Tumblr: http://www.theregister.co.uk/2015/04/15/yahoo_plans_storage_service_on_ceph/ Funny. I work at a Flickr competitor and one of our sysadmins really, REALLY wanted us to switch to Ceph for image and video storage. He built a few POC's over the last couple years and honestly it looked really promising on paper. The CTO shot it down every time, though, and I think it was the right call. Yahoo presumably has dozens of engineers they can devote to this project. For us it would have been, like, one. For a cluster storing billions of objects adding up to many petabytes of data. We just didn't have the manpower to build and support something like that in-house. This guy is brilliant but he had a giant blind spot around financial tradeoffs. Only looking at the amount of money we'd have to pay for hardware and support. Not including the salaries of everyone who would have to spend huge amounts of their time building and running the system vs just opening a ticket and moving on to something productive. He very seriously proposed we build our own nationwide CDN, too, for example, because it would lower our monthly bill to $vendor. He ended up getting poached to build another company's Ceph solution (not Yahoo) instead and is now making $$$$$$$$$$ so he got the last laugh there, I guess.
|
#
?
Apr 16, 2015 16:10
|
|
|
|
| # ? Apr 18, 2024 22:22 |
|
|
Docjowles posted:Funny. I work at a Flickr competitor and one of our sysadmins really, REALLY wanted us to switch to Ceph for image and video storage. He built a few POC's over the last couple years and honestly it looked really promising on paper. The CTO shot it down every time, though, and I think it was the right call. Yahoo presumably has dozens of engineers they can devote to this project. For us it would have been, like, one. For a cluster storing billions of objects adding up to many petabytes of data. We just didn't have the manpower to build and support something like that in-house. This guy is brilliant but he had a giant blind spot around financial tradeoffs. Only looking at the amount of money we'd have to pay for hardware and support. Not including the salaries of everyone who would have to spend huge amounts of their time building and running the system vs just opening a ticket and moving on to something productive. He very seriously proposed we build our own nationwide CDN, too, for example, because it would lower our monthly bill to $vendor. Can I ask what you actually use now?
|
|
#
?
Apr 16, 2015 17:40
|
|
|
We currently run good old NFS on a loooooooot of NetApp filers. Due to some of our executives having relationships with NetApp we got them for a song and it was great for a time. But the situation has changed and we can't really afford to keep refreshing those. We're now in the process of moving maybe 3/4 of our storage footprint to a cloud object storage vendor. Only data that's very sensitive to performance/latency will remain onsite.
|
|
#
?
Apr 16, 2015 17:46
|
|
|
Docjowles posted:We currently run good old NFS on a loooooooot of NetApp filers. Due to some of our executives having relationships with NetApp we got them for a song and it was great for a time. But the situation has changed and we can't really afford to keep refreshing those. Seems like onsite swift would be perfect, but I guess most of your objects probably don't get accessed very often...
|
|
#
?
Apr 16, 2015 19:10
|
|
|
evol262 posted:Seems like onsite swift would be perfect, but I guess most of your objects probably don't get accessed very often... We (as of a couple years ago) store a very high res copy of the photo which users can use to order prints and that sort of thing. Those are relatively huge, and accessed very infrequently. That's the bulk of what we're moving to cloud storage. Then we have a "web size" copy which is what you generally see browsing around the site or linked out on blogs/Reddit/whatever. Those are accessed very frequently, although of course there is an enormous long-tail. We did look at onsite Swift alongside Ceph. It just had the same problem, though. Our Ops team isn't big enough to support a hand-rolled system like that, it's kind of ridiculous the amount of traffic we serve with a handful of staff. And management wasn't interested in adding headcount. So we went with a hosted solution and let someone else worry about it.
|
|
#
?
Apr 16, 2015 19:20
|
|
|
Just wondering, about how many filers and TB's are you all monitoring? I am somewhere around 6PB, with 20ish filers and a few more 8040's due to arrive this month.
|
|
#
?
Apr 16, 2015 19:26
|
|
|
What do you guys use for naming conventions on virtual disks and luns? I got a DAS SAS storage device from Dell the other day, but I don't have any great ideas for naming. I'm thinking some combination of raid level, storage pool and connected endpoint. That's probably overkill. The appliance is only connected to two hosts right now with no shared storage pool between the hosts. There is a raid 10 on SSD and some 15k SAS that will be raid 5.
|
|
#
?
Apr 22, 2015 15:57
|
|
|
SpaceRangerJoe posted:What do you guys use for naming conventions on virtual disks and luns? I got a DAS SAS storage device from Dell the other day, but I don't have any great ideas for naming. I'm thinking some combination of raid level, storage pool and connected endpoint. That's probably overkill. The appliance is only connected to two hosts right now with no shared storage pool between the hosts. I'm not sure if its the best naming convention, but we do something like LUN###-<role>-<storage_characteristic+> and it seems to be serving us well. Some examples: code:
|
|
#
?
Apr 22, 2015 16:33
|
|
|
Does anyone have any experience with Scality's RING software defined storage solution? They're giving one of my coworkers a pretty aggressive sales push.
|
|
#
?
Apr 23, 2015 16:23
|
|
|
Well this week I learned what 2.8 PB of Iscilon costs (X and HD)- and we're probably ordering it next week. So much for Gluster/Ceph.
|
|
#
?
Apr 24, 2015 18:18
|
|
|
the spyder posted:Well this week I learned what 2.8 PB of Iscilon costs (X and HD)- and we're probably ordering it next week. So much for Gluster/Ceph. Can you ball park it? I am really curious.
|
|
#
?
Apr 24, 2015 18:30
|
|
|
mayodreams posted:Can you ball park it? I am really curious. Ended up being closer to 2.5PB Raw/ 2.1PB Usable after removing a x410. Let's just say it was just over $1mil with 3yr warranty. We will most likely be buying an identical unit for our other office.
|
|
#
?
Apr 24, 2015 23:29
|
|
|
the spyder posted:Ended up being closer to 2.5PB Raw/ 2.1PB Usable after removing a x410. Let's just say it was just over $1mil with 3yr warranty. We will most likely be buying an identical unit for our other office. Is this that new job with a bunch of different old arrays everywhere?
|
|
#
?
Apr 25, 2015 00:26
|
|
|
Yes. We actually have a 700TB Iscilon, but due to several issues, we can't upgrade it. Plus the maintenance is over $350k the next 3 years. We literally have had almost everything. I found some 3par and Bluearc this week, buried in some decom'd racks.
|
|
#
?
Apr 25, 2015 17:43
|
|
|
I would love and hate your job at the same time.
|
|
#
?
Apr 25, 2015 17:45
|
|
|
Moey posted:I would love and hate your job at the same time. Welcome to my realization. I won't rant here, but it's a mess in many ways.
|
|
#
?
Apr 25, 2015 21:16
|
|
|
FISHMANPET posted:Physical server, attached with Fibre Channel to a Hitachi SAN. But apparently the same issue is happening on a local 10k SAS disk and also a FusionIO card. An ending to this story. Backup guy is in fact a poo poo, we burnt a ton of Microsoft support hours for nothing (though storage team is paying for them), other than to determine that Windows is working as expected. Backup team is going to have to get off his rear end and engineer a system that isn't garbage rather than unendingly blaming his performance problems on other people. He wants to try and replicate this on a 2008 server for his own personal edification, but at this point I don't think anybody has his back.
|
|
#
?
Apr 30, 2015 21:25
|
|
|
I have a weird setup and question here: We got an old Dell PowerEdge R710 to use as a file server. We requested it with the Dell SAS 6/iR RAID card instead of the Dell H700 card because we wanted direct access to the disks for ZFS. The "nicer" Dell cards do not let you disable RAID to use JBOD. Their "cheaper" RAID cards do. I get the server and fill it with a bunch of new HDDs. I notice that only 2.0TB of the drives are showing up (they are 5TB drives). I look up more info, and I see that while the SAS 6/iR gives us "direct" access to the drives, it cannot access more than 2.0 TB. The H700 would let us work with the whole drive, but we'd lose direct-access (JBOD). Someone suggested the LSI SAS9211-8i card. It cheap and works like the SAS 6/iR, but supports >2TB drives. So I pop that in, and it seems to work fine. All drives are available with their full capacity... But now my HDD power LEDs are not on. Activity LEDs flashes when the drive is in use, but the power LED is off. Is this some Dell backplane + Non Dell backplane issue? Is there any way to make the HDD power LEDs come on?
|
|
#
?
May 4, 2015 23:34
|
|
|
FISHMANPET posted:An ending to this story. Backup guy is in fact a poo poo, we burnt a ton of Microsoft support hours for nothing (though storage team is paying for them), other than to determine that Windows is working as expected. Backup team is going to have to get off his rear end and engineer a system that isn't garbage rather than unendingly blaming his performance problems on other people. Good to hear this came to a close. From the outset, it sounded like the Win host was being asked to do something it shouldn't be asked to do at the OS level. Was it really that Commvalut was literally copying tens of thousands of files with each job, and the backup engineer didn't see that as a scaling problem?
|
|
#
?
May 6, 2015 05:20
|
|
|
Exactly that. He's still not convinced that it's not a problem, but nobody is willing to devote resources to tracking down this supposed problem. He kept going back to "well the SAN was only showing 16 IOPS during the copy" but it's not really an IOPS problem. At least now I have a "backup guy is poo poo" story just like everyone else in the office does. He keeps trying to blame this problem on other people. "Oh it's a Windows problem because Windows can't handle it." "Oh its VMware team problem because they made me consolidate backup groups." Yeah, sure, you do you.
|
|
#
?
May 6, 2015 05:40
|
|
|
FISHMANPET posted:Exactly that. He's still not convinced that it's not a problem, but nobody is willing to devote resources to tracking down this supposed problem. He kept going back to "well the SAN was only showing 16 IOPS during the copy" but it's not really an IOPS problem. At least now I have a "backup guy is poo poo" story just like everyone else in the office does. He keeps trying to blame this problem on other people. "Oh it's a Windows problem because Windows can't handle it." "Oh its VMware team problem because they made me consolidate backup groups." Yeah, sure, you do you. Just because the solution that worked a decade ago doesn't work today doesn't make him or her poo poo outright -- just means s/he needs a kick in the rear end to look at another solution. Meh, who am I kidding, fucker can't see the problem clear in front of him/her. You said you had something on the order of petabytes of data to back up? I'd have to think that justifies having at least another backup engineer on the team. If you went with an out-of-the-box solution like that which is avilable from Unitrends, for example, you'd end up buying 35 of their largest backup devices (Recovery-943S) for 2 PB of data. That much data is no small demon to slay.
|
|
#
?
May 6, 2015 06:10
|
|
|
Guy probably needs help from someone who understands modern distributed storage and backup methods at a very deep level. I only know enough that I wouldn't really have a clue where to start for that volume of data and would instead need to hire someone.
|
|
#
?
May 6, 2015 06:12
|
|
|
This particular system may only be backing up our VM infrastructure, but that's thousands of machines, and not our huge pile of data on our Isilon. But I'm only vaguely aware of the details of all of that. E: I see in the VM thread that you are also an employee of a large public university. So I'm sure you can relate to all sorts of things I'm going through here. FISHMANPET fucked around with this message at 07:23 on May 6, 2015 |
|
#
?
May 6, 2015 07:16
|
|
|
Fiber zoning question. Maybe this is the place for this. I inherited a fiber channel 3PAR SAN that's connected to two brocade switches, which are then connected to three hosts running VMware. Each of the controllers on the 3PAR has multiple fiber ports, but only one of those ports per controller is connected to a brocade switch. I want to increase the bandwidth to the SAN but I know precisely dick about fiber zoning beyond "it's similar to a VLAN." The Brocade switches are only used for connecting the hosts to the 3PAR, so do I need to do anything beyond relying on their default configs? Is there some fiber equivalent of port aggregation or can I just plug them in?
|
|
#
?
May 7, 2015 14:47
|
|
|
You need to know how zoning is configured on your switches. You are most likely using soft zoning (cause no one uses hard/port zoning.. but be sure). With soft zoning, you are going to build a zone on your switch which will be a "container" that matches WWNs on the SAN ports to the WWNs on your VM hosts. Anyways, as long as your zones contain the WWNs for the ports you want to connect, the FC protocol will take care of everything else.... including multipathing/login/etc EDIT: Also look to see if you have an ISL link between your switches
|
|
#
?
May 7, 2015 15:02
|
|
|
And make sure you're using the correct pathing policies on your data stores per the VMware HCL, bit me in the rear end big time a few weeks ago.
|
|
#
?
May 7, 2015 16:10
|
|
|
Strife posted:I want to increase the bandwidth to the SAN but I know precisely dick about fiber zoning beyond "it's similar to a VLAN." The Brocade switches are only used for connecting the hosts to the 3PAR, so do I need to do anything beyond relying on their default configs? Is there some fiber equivalent of port aggregation or can I just plug them in? Zoning is more closely related to creating access lists than it is a VLAN. You generally want to make sure you're using single initiator zones with either single targets or multiple targets. By that I mean an initiator would basically be a server and a target would be some sort of storage device, tape library, whatever. So if I have say 2 servers and a storage array I would define 2 zones using a naming convention not unlike below: SERVERHOSTNAME1_HBA0__3PARHOSTNAME member SERVERHOSTNAME1_HBA0 member 3PARHOSTNAME_CONTROLLER1_PORT1 member 3PARHOSTNAME_CONTROLLER1_PORT2 member 3PARHOSTNAME_CONTROLLER2_PORT1 member 3PARHOSTNAME_CONTROLLER2_PORT2 SERVERHOSTNAME2_HBA0__3PARHOSTNAME member SERVERHOSTNAME2_HBA0 member 3PARHOSTNAME_CONTROLLER1_PORT1 member 3PARHOSTNAME_CONTROLLER1_PORT2 member 3PARHOSTNAME_CONTROLLER2_PORT1 member 3PARHOSTNAME_CONTROLLER2_PORT2 Then add those zones to the zoneset and activate it. Brocade should support the notion of device aliases or faceless.. Use them or you will hate life when you go to troubleshoot and you find yourself staring at WWPNs instead of something that is meaningful to you. Repeat this process for the B fabric. Don't be tempted to throw all your servers in one big happy zone and call it a day as this can cause problems for you down the road. edit: I could give you CLI syntax for a Cisco MDS but I haven't touched a brocade in a good long while. Should be mostly the same though. quote:You need to know how zoning is configured on your switches. You are most likely using soft zoning (cause no one uses hard/port zoning.. but be sure). With soft zoning, you are going to build a zone on your switch which will be a "container" that matches WWNs on the SAN ports to the WWNs on your VM hosts. Anyways, as long as your zones contain the WWNs for the ports you want to connect, the FC protocol will take care of everything else.... including multipathing/login/etc I hate to be a pedantic dickface but hard zoning/soft zoning really refers to how zoning gets enforced (is it just via FCNS or actually making sure the ASIC doesn't forward the traffic). Even when you do WWNN/alias based zoning it's still enforced on hardware on brocade and MDS switches (and probably QLogic and other OEM brands as well.) People often mistake hard zoning for port based zoning (which I agree do not do!.) That's the end of my pedantry! edit 2: here's an article that explains for me! http://searchstorage.techtarget.com/tip/Zoning-part-2-Hard-zoning-vs-soft-zoning 1000101 fucked around with this message at 19:30 on May 7, 2015 |
|
#
?
May 7, 2015 18:03
|
|
|
So there's really nothing more to it than creating zones and adding the WWNs to each zone? Seems simple enough, really. I create zones for each one of my hosts' HBAs to the 3PAR, and then add the additional WWN for the other port on the 3PAR when I uplink it to the Brocade. Like this?  Using the GUI because I don't know what the hell language the command line is in and it's confusing as hell.
|
|
#
?
May 7, 2015 18:26
|
|
|
Brocade FabricOS! In some ways I like it better than MDS but yeah you've got the gist of it. Brocade's GUI fabric manager is leaps and bounds better than DCNM though. If you can, consider adding some aliases for your devices to make those screens a little readable. Also a reminder that you probably have 2 switches that aren't connected to each other so you'll need to repeat the exercise on both sides. This is by design to make sure you have redundancy in your storage network (in case you screw up zoning somehow for example your servers will still have a path available)
|
|
#
?
May 7, 2015 18:55
|
|
|
1000101 posted:If you can, consider adding some aliases for your devices to make those screens a little readable. I highly recommend this as well. Nothing's more annoying when you're trying to troubleshoot FC issues and you're looking at a bunch of WWPNs with zero idea on what host that actually is without having to log onto every single attached box and comparing hex addresses. Here's my Brocade FabricOS cheat sheet I use since the CLI is much much faster to do bulk changes, if you're feeling up to it. Remember you have to do this on each side, so change it up to "path1" for your second switch, or however you choose to name it. code:
|
|
#
?
May 7, 2015 19:06
|
|
|
1000101 posted:Also a reminder that you probably have 2 switches that aren't connected to each other so you'll need to repeat the exercise on both sides. This is by design to make sure you have redundancy in your storage network (in case you screw up zoning somehow for example your servers will still have a path available) Oh, yeah, of course. I'm going to enable the zones on one switch, make sure my host can still see the storage on the HBAs connected to that switch, and then do the other. Cidrick posted:cfg Sweet. Thanks for the info guys.
|
|
#
?
May 7, 2015 19:07
|
|
|
1000101 posted:
Oh gosh he gets the idea you pedant! Thanks for clarifying though 
|
|
#
?
May 7, 2015 19:12
|
|
|
Also note that adding three connections to the switch for one device just gives it three separate paths at that speed, not a combined aggregate. If your VMWare path selection policy isn't set to something that utilizes multiple active ports, adding additional connections is just creating additional redundancy, but not providing any additional bandwidth.
mattisacomputer fucked around with this message at 19:22 on May 7, 2015 |
|
#
?
May 7, 2015 19:19
|
|
|
The only thing I miss about my previous job was working with Brocades and FC fabrics. At my current job it's all NFS/iSCSI 
|
|
#
?
May 7, 2015 19:46
|
|
|
cheese-cube posted:The only thing I miss about my previous job was working with Brocades and FC fabrics. At my current job it's all NFS/iSCSI I'm kind of jealous, to be honest. FC is a great, robust protocol, but with 10GB ethernet being so cheap and simple to work with, and the fact that a lot of shops are moving away from big expensive SANs to a more appliance or commodity-based storage platform, I've been touching it a lot less and less these days. My next dream is to set up distributed ceph or gluster distributed storage platform across several rows of 2U servers packed with drives and using something like opendedup on top of it. It'll probably never happen, but I can dream.
|
|
#
?
May 7, 2015 20:00
|
|
|
So, maybe a stupid question, maybe not, but should I have each host hba be in a zone with both 3PAR controllers, or should I make a separate zone for each host to each controller? Either way the host sees twice as many paths back to the storage as it did earlier.
|
|
#
?
May 8, 2015 02:33
|
|
|
Strife posted:So, maybe a stupid question, maybe not, but should I have each host hba be in a zone with both 3PAR controllers, or should I make a separate zone for each host to each controller? Either way the host sees twice as many paths back to the storage as it did earlier. We do per hba zoning.... On every switch 1 zone per hba per host and then add one port per controller to that zone. Then its just like old school point to point which FC loves except with full multipath features Edit: unless its our vmax fabric in which zoning is built to match the port group configuration Syano fucked around with this message at 02:46 on May 8, 2015 |
|
#
?
May 8, 2015 02:37
|
|
|
Syano posted:We do per hba zoning.... On every switch 1 zone per hba per host and then add one port per controller to that zone. Then its just like old school point to point which FC loves except with full multipath features Alright cool. At least I know what I'm doing is a thing. VMware handles the path back with RR, and as long as it has multiple paths back to each LUN I assume it's redundant. So now I can suffer a controller failure and a brocade switch failure.
|
|
#
?
May 8, 2015 02:46
|
|
|
Strife posted:So, maybe a stupid question, maybe not, but should I have each host hba be in a zone with both 3PAR controllers, or should I make a separate zone for each host to each controller? Either way the host sees twice as many paths back to the storage as it did earlier. Check with your array vendor.. EMC best practice is 1 HBA, 1 controller port per zone. So as a minimum each host would have the following added to the zone set: Fabric A HBA0 -> SPA P0 HBA0 -> SPB P0 Fabric B HBA 1 -> SPA P1 HBA 1 -> SPB P1 I'm not an FC expert, but i believe this reduces the impact of RSCN notifications for fabric changes, and also VNX/CX arrays crack the shits and spam error messages as their controller ports can either be an initiator or a target if they are using mirrorview/sancopy and they keep trying to log into each other.
|
|
#
?
May 8, 2015 06:57
|
|
|
|
| # ? Apr 18, 2024 22:22 |
|
|
If it's a newer version of the firmware you should have port persistence (NPIV) available/enabled so you would connect a port and its backup port on the other controller to the same fabric and then add the virtual wwpn to the zone along with a single host wwpn.
|
|
#
?
May 8, 2015 20:03
|
|
