|
Does anyone have experience with 1+ PB storage systems? We have a project that may require up to 7.5TB of data collection per day.
|
#
?
May 3, 2012 00:38
|
|
|
|
| # ? Apr 25, 2024 12:38 |
|
|
Get SC 6. It's been out for months. That's admittedly about a year later than it should be, but it supports the full VAAI feature set. A bit surprised to hear it's been so bad -- the only previous hate in the thread was from the guy who works for EMC. Currently have 120 TB of Compellent over 3 arrays and 80+ of it is VMware. While I'm sad they've fallen behind on a few things (flash cache), it has been really solid for the last year and a half. Intraveinous posted:(their lab had identified a problem in some limited cases with Emulex 8Gb FC cards, so they proactively replaced them with Qlogic 8Gb FC cards) Their lab didn't identify poo poo -- we discovered it. It caused a rather long outage here due to an obscure Emulex TSB and a conflict with Brocade FW 6.3+. Small world  . But Copilot was awesome (Same engineer was leading the effort for 16+ hours) and the part was there within 2 hours of dispatch once they figured out the cause.
KS fucked around with this message at 02:00 on May 3, 2012 |
|
#
?
May 3, 2012 01:52
|
|
|
the spyder posted:Does anyone have experience with 1+ PB storage systems? We have a project that may require up to 7.5TB of data collection per day. I know some guys collecting data about this rate, say 5TB/day. Unstructured files on an IBM SONAS. If at all possible, lean on technology that will reduce your needed real disk space. Thin provision, online dedupe, data compression, etc.
|
|
#
?
May 3, 2012 02:35
|
|
|
the spyder posted:Does anyone have experience with 1+ PB storage systems? We have a project that may require up to 7.5TB of data collection per day. Any major storage vendor can provide a system, or scale out systems, to provide a petabyte or more. You'll need to give a lot more detail regarding what kind of data you're collecting, how it's accessed, what level of protection you want, backup requirements, etc.
|
|
#
?
May 3, 2012 05:35
|
|
|
NippleFloss posted:Any major storage vendor can provide a system, or scale out systems, to provide a petabyte or more. You'll need to give a lot more detail regarding what kind of data you're collecting, how it's accessed, what level of protection you want, backup requirements, etc. How do you even back up that much data? I'd assume just mirror the disks to another location? LTO-5 drives only write at 140MB/s off site for that much data would take a staggering amount of tapes and drives to be effective.
|
|
#
?
May 3, 2012 16:51
|
|
|
KS posted:Get SC 6. It's been out for months. That's admittedly about a year later than it should be, but it supports the full VAAI feature set. That's right, I remember our conversation now. You gave me your SR number which I relayed to them. My bad, I guess I should have said, "Their lab was able to reproduce and verify the problem discovered by KS." Credit where credit is due, thanks for your help. I'm just about to add more disk to Tier 3, at which point I'll be up to about 115TB total between two arrays. Offer still stands to take the array off your hands, three.
|
|
#
?
May 3, 2012 18:09
|
|
|
the spyder posted:Does anyone have experience with 1+ PB storage systems? We have a project that may require up to 7.5TB of data collection per day. I have nothing of value to offer other than I would love to know the details of the system you end up using. I have long been fascinated with storing immense amounts of data and the technologies required to do so.
|
|
#
?
May 3, 2012 18:18
|
|
|
Syano posted:I have nothing of value to offer other than I would love to know the details of the system you end up using. I have long been fascinated with storing immense amounts of data and the technologies required to do so.
|
|
#
?
May 3, 2012 18:23
|
|
|
I don't think the actual storage of the data is as important as to how the data is going to be used and accessed. If you want just bulk storage of data a BackBlaze style setup is going to be the cheapest way to store all that data. Now if you need to be able to access and run complex poo poo against 7.5TB of data from 3 months ago... that's a different ballgame. Either way, it's going to be an expensive project best served by overpaid but hopefully knowledgeable consultants.
|
|
#
?
May 3, 2012 18:33
|
|
|
Intraveinous posted:That's right, I remember our conversation now. You gave me your SR number which I relayed to them. My bad, I guess I should have said, "Their lab was able to reproduce and verify the problem discovered by KS." Credit where credit is due, thanks for your help. I imagine the array itself has hardware issues, which Compellent seems unable to see. I'd bet if I had a different unit it would probably be fine; however, the experience with a bad unit has just shown has poor Compellent and Copilot is. Any support that accidentally reboots the wrong controller and brings down the storage is bad. They've even been unable to even diagnose some issues (e.g. one controller hanging on a firmware update). Their response times to difficult tickets is pretty terrible, too. SC 6.0 finally implementing VAAI is nice. At least you can rest assure technologies will be supported eventually. In any case, we'll likely use this for junk projects and keep critical machines on a vendor that isn't terrible.
|
|
#
?
May 3, 2012 18:43
|
|
|
MC Fruit Stripe posted:Yeah poo poo, I can't even wrap my head around 7.5TB per day - more info, both specific to that implementation and just general "how this is done", would be amazing. 7.5TB a day is only 90 MB/s which isn't a an outrageous throughput. Of course if you write that 7.5TB over the course of one hour, once a day, then it's a different ballgame, which is why it's not possible to recommend something without more detail. But the basic throughput requirement of 7.5TB a day could be achieved on even mid-range gear pretty easily, even running 7.2k disks (depending on how your SAN vendor handles writes). Of course you have to be able to do something with the data once you write it, which is where it gets tricky. Is the data written once and read many times? Or is it written in bulk and then over-written randomly at a high rate? Are the reads random or sequential, and what is the size of the working set that you're reading at any one time? How long does the data need to be available for immediate reading, meaning are you going to need to keep a years worth of data (almost 3PB) available an a stated level of performance, are are you able to archive data after a week, or a month, or whatever to slower storage? Can the application address and potentially balance across multiple filesystems, or does it require a single large block of contiguous storage? There's a ton of questions that need to be answered before you can begin to architect something like this but the throughput and capacity requirements aren't all that outlandish.
|
|
#
?
May 3, 2012 20:54
|
|
|
Just to reinforce a point that has been made in this thread over and over again: We have a backup server populated with 12x 2tb 7.2krpm drives. When we purchased it several years ago we had it configured from the vendor as a raid 5 array. Mistake. We had a drive fail yesterday. No problem usually. We pulled the bad drive and hot swapped a spair in and popped in to the server manager to ensure the rebuild was taking place. Holy smokes. By the speed it is currently going the array will be finished rebuilding this time next week. Now here we are crossing our fingers that another drive doesnt flop in the next 6 days. ALWAYS configure huge arrays as raid 6 or similar. Lesson learned.
|
|
#
?
May 4, 2012 13:37
|
|
|
Quote for an EqualLogic PS4100E with 6 1TB NL-SAS drives came in at $14k. Does that seem high to anyone else or is it just me?
|
|
#
?
May 4, 2012 23:15
|
|
|
So, I am starting to have buyers remorse. We bought a NetApp 3240 HA pair, each with io expanders so we could reuse some ds14mk2 shelves we have. Would I have been able to buy a 3210 HA pair in a single chassis and ~30TB raw of SATA disk for the same price we paid for the 3240? Each unit having complete pack.
|
|
#
?
May 4, 2012 23:18
|
|
|
How would new SATA be better than the old stuff? What did you pay for the new controllers? (PM if you must, but I don't believe pricing should ever be privileged info)
|
|
#
?
May 4, 2012 23:23
|
|
|
evil_bunnY posted:How would new SATA be better than the old stuff?
|
|
#
?
May 4, 2012 23:42
|
|
|
I'm about ready to tear my hair out with DFS. I'm trying to create the first Namespace, and I get the error "RPC server not available." I have no idea what RPC server it's trying to contact, I can't find any failures in any logs, and googling hasn't come up with any useful solutions. I did this on another domain and it worked just fine, so I don't know what's different here.
|
|
#
?
May 4, 2012 23:44
|
|
|
FISHMANPET posted:I'm about ready to tear my hair out with DFS. I'm trying to create the first Namespace, and I get the error "RPC server not available." I have no idea what RPC server it's trying to contact, I can't find any failures in any logs, and googling hasn't come up with any useful solutions. I did this on another domain and it worked just fine, so I don't know what's different here. Its querying for a DC and the one it is attempting to connect to either does not exist any longer or is not reachable. Make sure all your DCs are online. Make sure you do not have a dc still listed in AD which has been taken offline without proper demotion. If you do, use ntdsutil to get rid of all references to it.
|
|
#
?
May 5, 2012 00:10
|
|
|
Cpt.Wacky posted:Quote for an EqualLogic PS4100E with 6 1TB NL-SAS drives came in at $14k. Does that seem high to anyone else or is it just me? Seems about right for MSRP. http://www.milesconsultingcorp.com/Buy-Dell-EqualLogic-PS4100E.aspx That is MSRP. 6 NL-SAS drives will be absolutely terrible for IOPs. I'd get them to do 12 drives for the same price if you could.
|
|
#
?
May 5, 2012 00:19
|
|
|
Alright, I've got a relatively dumb problem that has me stumped. We have several servers that do iSCSI booting off our old Equallogic SAN and I need to get them moved over to our new EMC SAN. I've got everything set up and figured out, but I am having a hell of a time doing a disk copy and keeping the disk bootable. I have basically been taking a snapshot of the Equallogic volume, then mounting it on a server as a drive, and then mounting the EMC volume that I want to copy to. I've tried a copy different methods and software with no luck. Anyone do this with any frequency?
|
|
#
?
May 5, 2012 00:55
|
|
|
Syano posted:Its querying for a DC and the one it is attempting to connect to either does not exist any longer or is not reachable. Make sure all your DCs are online. Make sure you do not have a dc still listed in AD which has been taken offline without proper demotion. If you do, use ntdsutil to get rid of all references to it. In my case I had 2 DCs, and for poltical reasons I had to rename them, and because I'm stupid, I just renamed them rather than demoting them, renaming, and repromoting. This week I one at a time demoted and repromoted them, but that hasn't fixed it either. So now I have to figure out ntdsutil, but it's been my suspicion all along that this was the problem, I just had no idea how to go about diagnosing that.
|
|
#
?
May 5, 2012 00:56
|
|
|
adorai posted:Pricing sent, new sata would be better because I would still have my old shelves for use on another filer we have. The 3240 is worth it purely for the 8GB of ram per controller. You'd run out of headroom on the 3210 a lot sooner.
|
|
#
?
May 5, 2012 01:08
|
|
|
Internet Explorer posted:Seems about right for MSRP. Internet Explorer posted:. I've tried a copy different methods and software with no luck. Anyone do this with any frequency? FISHMANPET posted:I have no idea what RPC server it's trying to contact, I can't find any failures in any logs, and googling hasn't come up with any useful solutions. FISHMANPET posted:In my case I had 2 DCs, and for poltical reasons I had to rename them, and because I'm stupid, I just renamed them rather than demoting them, renaming, and repromoting. This week I one at a time demoted and repromoted them, but that hasn't fixed it either. evil_bunnY fucked around with this message at 09:37 on May 5, 2012 |
|
#
?
May 5, 2012 09:27
|
|
|
evil_bunnY posted:
Short of starting the domain from scratch I'm not sure this would do any good, whatever errant records are in AD wouldn't be removed by a demotion because it's not linked to any actual computer.
|
|
#
?
May 5, 2012 09:37
|
|
|
evil_bunnY posted:Good old dd usually works for stuff like that. It'd help to know what you've tried already. I tried DriveImage XML and was going to try the trial of Acronis Disk Director, but the trial does not do copies and then I saw it was 600 a license for 1 server, which is rather outrageous. I was looking at DD for Windows yesterday, but then it was quiting time and time for a beer. Will look at it today, thanks.
|
|
#
?
May 5, 2012 15:54
|
|
|
the spyder posted:Does anyone have experience with 1+ PB storage systems? We have a project that may require up to 7.5TB of data collection per day. I've got a couple systems at this scale. Can you elaborate on what kind of data it is, how it's collected, and how it'll be accessed for reads? If it's not something you can discuss in the open thread you can msg me.
|
|
#
?
May 5, 2012 18:01
|
|
|
So who was complaining about the 255 snapshot limit on DataONTAP 
|
|
#
?
May 10, 2012 05:13
|
|
|
Is the free MS iSCSI initiator any good? I need to add more storage to our backup location in our DR site and instead of replacing the server I'm looking at one of the Rackmount netgear units (we use one here in our office and it's solid). IOPS are not a concern at all as this will be a target for Crashplan/rsync.
Matt Zerella fucked around with this message at 15:59 on May 10, 2012 |
|
#
?
May 10, 2012 15:56
|
|
|
Cpt.Wacky posted:Quote for an EqualLogic PS4100E with 6 1TB NL-SAS drives came in at $14k. Does that seem high to anyone else or is it just me? I've seen a 4100X (whatever the 2.5in 24 bay SAS model is) half populated with 12x 600gb drives for 19k. I think 14 for that is too much.
|
|
#
?
May 10, 2012 16:01
|
|
|
marketingman posted:So who was complaining about the 255 snapshot limit on DataONTAP Spill the beans! re: MS iSCSI initiator - yeah it's awesome, anyone who uses iSCSI on Windows is using the built-in initiator.
|
|
#
?
May 10, 2012 16:03
|
|
|
madsushi posted:Spill the beans! But you need to install the addon for 2003, right?
|
|
#
?
May 10, 2012 16:06
|
|
|
marketingman posted:So who was complaining about the 255 snapshot limit on DataONTAP madsushi posted:re: MS iSCSI initiator - yeah it's awesome, anyone who uses iSCSI on Windows is using the built-in initiator. LmaoTheKid posted:But you need to install the addon for 2003, right? evil_bunnY fucked around with this message at 16:15 on May 10, 2012 |
|
#
?
May 10, 2012 16:12
|
|
|
evil_bunnY posted:It's got a terrible, terrible UI but it works well. Thanks. I'm not worried about the UI as this will be a set once and forget it kind of thing.
|
|
#
?
May 10, 2012 16:16
|
|
|
evil_bunnY posted:Talk now or shut up forever. I guess I never see the built-in iSCSI UI because I'm spoiled with SnapDrive.
|
|
#
?
May 10, 2012 18:11
|
|
|
I just saw our pricing on an IBM V7000 with a single disk enclosure, laden with 900GB 10K 2.5" SAS drives. I frankly do not understand how other vendors are even still in business.
|
|
#
?
May 10, 2012 18:33
|
|
|
Misogynist posted:I just saw our pricing on an IBM V7000 with a single disk enclosure, laden with 900GB 10K 2.5" SAS drives. I frankly do not understand how other vendors are even still in business. You will when you log a support call in any kind of complex case. At least if you are in europe.
|
|
#
?
May 10, 2012 19:31
|
|
|
Misogynist posted:I just saw our pricing on an IBM V7000 with a single disk enclosure, laden with 900GB 10K 2.5" SAS drives. I frankly do not understand how other vendors are even still in business.
|
|
#
?
May 10, 2012 19:50
|
|
|
Misogynist posted:I just saw our pricing on an IBM V7000 with a single disk enclosure, laden with 900GB 10K 2.5" SAS drives. I frankly do not understand how other vendors are even still in business. There are a lot of holes in the V7000 offering. They only support file protocols via a bolt-on solution similar to what EMC has been doing for years. This costs more money and adds complexity. The NAS gateway portion of the appliance is less stable and well supported than the SAN side. There is also a lack of redundancy in the disk stack that should be worrying. If an enclosure fails anywhere in the stack all shelves below it also become inaccessible. Most SAN vendors have at least two paths from each controller to the storage stack to prevent this. They still utilize copy-on-write snapshots which have some associated performance overhead, and that overhead grows as your number of snapshots grows. FlashCopy and other features also inherit this penalty on writes. Thin-provisioning also has some performance overhead as it requires additional interaction with volume metadata during I/O. They also don't support any form of compression or deduplication, which are both standard for most SAN vendors these days. The automated tiering only works on block based protocols, so no NFS or CIFS, and it works in 256MB chunks, which isn't terribly efficient. If I had to pick any storage vendor that has a product that really concerns me as a NetApp guy it would be Nimble. They appear to be building a nice, well integrated product with a lot of forethought. They've had the advantage of watching the evolution of redirect-on-write filesystems and coding theirs from the ground up to avoid many of the pitfalls. If they manage not to get bought by EMC and expand their product line and feature set I think they could be very competitive in a few years.
|
|
#
?
May 10, 2012 22:00
|
|
|
Totally agree. And wrt to Nimble, personally, I'm taking them seriously enough that I've started looking into getting some units in and getting some training under my belt.
|
|
#
?
May 11, 2012 05:24
|
|
|
|
| # ? Apr 25, 2024 12:38 |
|
|
Here's the situation. We have an HP MSA2000fc serving as the storage for our database cluster. It was setup for us by an outside party. I don't have much knowledge on how to setup a fibre channel storage network since I've never gotten to screw around with it when it WASN'T hosting our database backend. However, we are going to be setting up a new database cluster soon and the outside party is no longer with us (for reasons I think will become apparent.) So, I'm getting the lay of the land and trying to get up to speed on what I need to know. Some more background. We have two Brocade Silkworm 200e switches with 12 licensed ports each. Each server has two HBAs, the SAN has two controllers with two connections each, and we will eventually have 4 servers connected to this SAN. What we asked for: Redundant switching fabric. What we got: One switch not plugged in at all. A0 and B0 ports on the SAN connected to one switch. One HBA in each server plugged into the switch, the 2nd one not connected to anything. I start digging into it a bit deeper. I fire up HBAnyware on the database server and take a look at the connections. A0 and B0 of the SAN are listed with both presenting drives. I go into server manager (2008 R2 machine) and take a look at the disks. I see all the disks that are being used online and mapped with a drive letter. But then I see them all listed again (only offline.) Hmm. I'm suspecting at this point if I went in an yanked the A0 connection, everything would go offline. So that's when I started trying to learn about the proper setup and this is where I need guidance from what I figured out today. I'm just trying to see if I'm on the right path and to confirm just how hilariously they hosed up this setup: First off, ports A1 and B1 should be connected to the other switch. 2nd HBA in each server should also be connected to the other switch. At that point, we'll have the physical redundancy down. Then, if I fired up HBAnywhere, I should see each set of drives 4 times under each HBA on the server (one for each of the connections the SAN has). After I had that figured out, I started thinking about the duplicate disks showing up in server manager. That just didn't seem right to me so I dug deeper. I'm guessing the issue is that they never installed the Multipath I/O feature in Windows Server 2008 R2. Because of that, the server has no idea that it's seeing multiple paths to each of the disks rather than different disks. So, does that sound about right? 1) Ensure all physical connections are in place 2) Setup multipath I/O so that all those extra connections are utilized properly This would be the absolute first time I've ever touched configuration of a SAN, so I'm just double checking myself. bull3964 fucked around with this message at 06:11 on May 11, 2012 |
|
#
?
May 11, 2012 06:08
|
|
