Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
H110Hawk
Dec 28, 2006


Misogynist posted:

I just moved part of my ESXi development environments off of local storage and onto a 48TB Sun x4500 I had lying around, shared via ZFS+NFS on OpenSolaris 2009.06 over a 10GbE link.

I was worried about performance because it's SATA disk, but holy poo poo this thing screams with all those disks. I have never seen a Linux distro install so fast ever in my life. The bottleneck seems to be the 10GbE interface, which apparently maxes out around 6 gig.

What zpool configuration are you using? Any specific kernel tweaks?

Adbot
ADBOT LOVES YOU

Vulture Culture
Jul 14, 2003

I was never enjoying it. I only eat it for the nutrients.


Bluecobra posted:

Are you using these 10GbE cards? They have been working extremely well for us but on some servers we added another card and turned on link aggregation. There is also this new feature in Solaris 10 Update 7 that may or may not be in OpenSolaris:
The Thumpers are PCI-X, so we're using an Intel 82597EX. I haven't even considered link aggregation on this thing, since it's first-gen and probably out of warranty soon. We're a lot more likely to go x4540 if we put these in production.

I also don't think our non-HPCC switching could handle it well. We're running some impressive Force10 gear on the HPCC side, but I think we're bottlenecked down to 10gig to any given rack group for our regular network.

H110Hawk posted:

What zpool configuration are you using? Any specific kernel tweaks?
RAID-Z1, 6 disks per array, single gigantic zpool and ZFS version 16. No real performance tweaks yet, though I've looked at some stuff with the I/O schedulers in OpenSolaris. I'm still playing with it, because I'm experimenting with iSCSI vs. NFS and iSCSI's performance is kind of poo poo with the default configuration on the VMware and Solaris sides.

bmoyles
Feb 15, 2002

United Neckbeard Foundation of America

Fun Shoe

Pretty neat deal
http://www.xiotech.com/Products-and-Services_Cash-for-Disk-Clunkers.aspx?elq=ea858ca57c4f46c3a1cd0dd94711be7d

KS
Jun 10, 2003


Outrageous Lumpwad

Misogynist posted:

If I can find some sane way to replicate this to another Thumper, I will be a very, very happy man.

I talked to a guy at a USENIX conference last year whose company was using dozens of X4500s as the storage backend for one of the biggest ESX deployments I've ever heard of (300k+ IOPS). At the time I was dealing with HP SAN issues and was very jealous. Now not so much, but NFS definitely seems to scale better than fiber channel for big clusters. 4540s are probably the best storage possible for ESX.

Weird Uncle Dave
Sep 2, 2003

I could do this all day.



Buglord

This is probably an invitation to all sorts of weird PMs, but do any of you do SAN consulting and/or sales?

I'm pretty sure I'm in over my head with my boss's request to virtualize several of our bigger physical servers. The virtualizing part is easy enough, but I don't really know enough about how to measure my hardware requirements, or how to shop intelligently for a low-end SAN that will meet those requirements, and I don't want to clutter the thread with all my newbie-level questions.

paperchaseguy
Feb 21, 2002

THEY'RE GONNA SAY NO

Weird Uncle Dave posted:

This is probably an invitation to all sorts of weird PMs, but do any of you do SAN consulting and/or sales?

I'm pretty sure I'm in over my head with my boss's request to virtualize several of our bigger physical servers. The virtualizing part is easy enough, but I don't really know enough about how to measure my hardware requirements, or how to shop intelligently for a low-end SAN that will meet those requirements, and I don't want to clutter the thread with all my newbie-level questions.

I work for EMC, there's at least one other person here who does, and several other professionals. You really won't clutter up the thread, it's not super active. Fire away with any questions.

Vulture Culture
Jul 14, 2003

I was never enjoying it. I only eat it for the nutrients.


Weird Uncle Dave posted:

This is probably an invitation to all sorts of weird PMs, but do any of you do SAN consulting and/or sales?

I'm pretty sure I'm in over my head with my boss's request to virtualize several of our bigger physical servers. The virtualizing part is easy enough, but I don't really know enough about how to measure my hardware requirements, or how to shop intelligently for a low-end SAN that will meet those requirements, and I don't want to clutter the thread with all my newbie-level questions.
It depends a lot on the type of platform you're running. What are the operating systems and major applications you're trying to profile?

Profiling Windows and the various MS apps is pretty easy -- your best bet is to download the 180-day trial of Systems Center Operations Manager, load it onto a VM, point it at your AD/Exchange/SQL/whatever servers and let it go to town. Within a couple of days you should have most of the relevant performance information you need. Microsoft also has a lot of best practices guides for how to obtain relevant performance information out of applications like Exchange, just Google for the whitepapers.

Linux is a lot trickier, especially if you're running on RHEL/CentOS or something else that doesn't have iotop and the other nice stuff that have made it into the system in the last couple of years. You'll have to babysit iostat for awhile.

Weird Uncle Dave
Sep 2, 2003

I could do this all day.



Buglord

We're so small, we don't even have a Windows domain. The Windows servers to be virtualized (all three of them) are standalone.

Is Windows perfmon, monitoring number of disk operations per second, a decent approximation of IOPS? I'm only really worried about disk speed for one server (about 4000 email addresses); the others (a little-used Web server and a small database server) aren't a big problem. Network bandwidth and RAM are cheap by comparison.

Most of our Linux servers (yes, they're pretty much all CentOS) have already been virtualized, so I should be able to get the data I'll need from VMware ESXi itself. Right now, there's about twenty guests on one host's local storage.

paperchaseguy
Feb 21, 2002

THEY'RE GONNA SAY NO

Weird Uncle Dave posted:

Is Windows perfmon, monitoring number of disk operations per second, a decent approximation of IOPS? I'm only really worried about disk speed for one server (about 4000 email addresses); the others (a little-used Web server and a small database server) aren't a big problem. Network bandwidth and RAM are cheap by comparison.

Exchange 2007 average user profile estimates .4 IOPS per user account. I like to use .5 IOPS because that gives some wiggle room and is easy to calculate: 2000 IOPS. A 15k FC drive maxes out on typical IO sizes at 180 IOPS. You could probably do all this on one 15 disk enclosure of FC drives, maybe a little more depending on the size.

perfmon would give you a realistic estimate of IOPS.

Weird Uncle Dave
Sep 2, 2003

I could do this all day.



Buglord

That sounds like way more than what I'd need. Right now, the mail server is an eight-year-old Dell PowerEdge, and all the email is on three 15krpm SCSI drives, RAID-5'd, and everything works perfectly well.

Left perfmon running over lunch, and the drive with all the mailboxes peaked at about 100 IOPS. Granted, that was lunch, and I'll leave things running for a couple days and peek in on them occasionally.

Most of the other servers are very light on disk use (static Web pages, DNS, very small databases, things that are more memory-bound than anything, fortunately RAM is cheap). So I should be okay in terms of disk speed; looks like even the cheapest EqualLogic iSCSI SAN will probably have plenty of that, both for now and for later. (I'm not wedded to EqualLogic by any means, just that they're Dell, and we've generally had good luck with Dell in the past.)

Assuming we go iSCSI (FC is probably way out of my budget), do I need any special magic gigabit Ethernet switch, or will any decent one do?

da sponge
May 24, 2004

..and you've eaten your pen. simply stunning.

Weird Uncle Dave posted:

That sounds like way more than what I'd need. Right now, the mail server is an eight-year-old Dell PowerEdge, and all the email is on three 15krpm SCSI drives, RAID-5'd, and everything works perfectly well.

Left perfmon running over lunch, and the drive with all the mailboxes peaked at about 100 IOPS. Granted, that was lunch, and I'll leave things running for a couple days and peek in on them occasionally.

Most of the other servers are very light on disk use (static Web pages, DNS, very small databases, things that are more memory-bound than anything, fortunately RAM is cheap). So I should be okay in terms of disk speed; looks like even the cheapest EqualLogic iSCSI SAN will probably have plenty of that, both for now and for later. (I'm not wedded to EqualLogic by any means, just that they're Dell, and we've generally had good luck with Dell in the past.)

Assuming we go iSCSI (FC is probably way out of my budget), do I need any special magic gigabit Ethernet switch, or will any decent one do?


A decent one will do. I'm partial to the HP Procurve line, although if you want jumbo frames + flow control you need a higher end switch. The 2824 would probably suit you fine (I have 2848s and they work flawlessly). It might be possible to go with a lower cost, unmanaged Procuve, but I don't know what would be capable off the top of my head.

paperchaseguy
Feb 21, 2002

THEY'RE GONNA SAY NO

Weird Uncle Dave posted:

That sounds like way more than what I'd need. Right now, the mail server is an eight-year-old Dell PowerEdge, and all the email is on three 15krpm SCSI drives, RAID-5'd, and everything works perfectly well.

Left perfmon running over lunch, and the drive with all the mailboxes peaked at about 100 IOPS. Granted, that was lunch, and I'll leave things running for a couple days and peek in on them occasionally.

If you have 4000 addresses, but many aliases, unused accounts, etc, that will obviously lower your requirements. Maybe a 4+1 SATA appliance would be better for you. SATA does 50-80 IOPS as a rule of thumb.

Vanilla
Feb 24, 2002

Hay guys what's going on in th

paperchaseguy posted:

I work for EMC, there's at least one other person here who does, and several other professionals. You really won't clutter up the thread, it's not super active. Fire away with any questions.

Ahhh another EMC brother :downs: and joined three days before me! Weird.

Weird Uncle Dave - on the MS site there's a tool that will give you their recommendations for disk based on users, workload, etc. However if it's running fine on three disks and everyone's happy then three disks it is.

Leave perfmon running for longer - 9AM is going to be the key time for things like Exchange. If you can also use perfmon to get items such as the read / write ratio of your exchange environment that would help translate the IOPS that is actually hitting the backend (writes have an overhead with R5 due to the parity calc and additional IO required).

Weird Uncle Dave
Sep 2, 2003

I could do this all day.



Buglord

We're not even running Exchange. It's a weird old mail package (Ipswitch Imail), that stores users' mail in per-user files (Unix .mbx style, as opposed to "one file per piece of email" or "Exchange-style giant database for everyone").

Perfmon will be running for a few days at least, both here and on the other servers I'm looking to consolidate (though they basically don't count - our billing database averages about 10 IOPS, and no that's not a typo).

Vulture Culture
Jul 14, 2003

I was never enjoying it. I only eat it for the nutrients.


Haha, oh man, I pity you remaining IMail admins. :(

adorai
Nov 2, 2002

10/27/04 Never forget

Grimey Drawer

We are currently looking at replacing our aging ibm san with something new. The top two on our list are a pair of Netapp 3020s and a 2050 for our offsite or a pair of EMC Clarions. I am interested in looking at a dual head Sun Unified Storage 7310 and a lower end sun at our offsite. The numbers seem to be literally half for the Sun solution, so I feel like I have to missing something on it.

For usage, the primary purpose will be backend storage for about 100 VMs, some cifs, and some iSCSI/fibre storage for a few database servers.

Any thoughts from you guys?

HorusTheAvenger
Nov 7, 2005


adorai posted:

I am interested in looking at a dual head Sun Unified Storage 7310 and a lower end sun at our offsite. The numbers seem to be literally half for the Sun solution, so I feel like I have to missing something on it.

We use the Sun Storage 7410 where I work. We bought it back in December when they didn't have the 7310 yet. We bought just the single head system. We use it for NFS/CIFS and iSCSI. It does the job and fit our price point (non-profit). My feelings are we purchased it 6 months too early. The Sun Storage line only hit the market back in November I think, so it is still fairly new technology. I don't remember them supporting fibre SAN... I remember promises about it being worked on, so if they support it now then it's relatively new.

I don't know if Sun still has it, but there was a try before you buy program. You had to be using one of their pre-built configurations approved for the program though.

If you go the Sun route, I would suggest against OS upgrades until at least a week has passed from the release date. I would also suggest you never upgrade to a base quarterly release. We were screwed hardcore with kernel panicks on 2009.Q2.0.0 (fixed the slow MS Office document issue on CIFS that management was so adament got resolved ASAP though). I would also recommend the dual-head clusterred version for this reason.

The Fishworks interface is really cool though. Before the Sun Storage I never knew how many IOPs we needed :-). The fishworks people keep a wiki of release notes here:
http://wikis.sun.com/display/FishWorks/Sun+Storage+7000+Series+Software+Updates

Edit: forgot to mention it's much faster than our antiquated 4 year old NetApp FAS270 was.

HorusTheAvenger fucked around with this message at 00:24 on Aug 26, 2009

Weird Uncle Dave
Sep 2, 2003

I could do this all day.



Buglord

Misogynist posted:

Haha, oh man, I pity you remaining IMail admins. :(

Hey, there's nothing wrong with Imail. Well, that's a lie, there are lots of little things wrong with Imail, but we've gotten used to them. Stockholm syndrome.

Anyway!

Now that I have a better idea of what I need, what's the best way to go about buying it? I'm pretty sure we're comfortably on the low end - Dell EqualLogic PS4000, Dot Hill 2330, something like that. Nobody has pricing listed publicly on their Web sites, of course, and I still don't know all that much about the intricacies of these products (or how they'll work in the real world). How do I make sure I'm getting something I need but I'm not getting completely ripped off?

Chances are, whatever I buy will cost more than my car, which to date is the largest purchase I've ever made (and my car was bought with my own money, whereas this is the boss' checkbook, so I figure I should be even more prudent).

If any of you sell something suitably low-end (just a couple terabytes, iSCSI, probably 800-1000 IOPS), feel free to PM me and we can talk.

1000101
May 14, 2003

BIRTHDAY BIRTHDAY BIRTHDAY BIRTHDAY BIRTHDAY BIRTHDAY FRUITCAKE!

adorai posted:

We are currently looking at replacing our aging ibm san with something new. The top two on our list are a pair of Netapp 3020s and a 2050 for our offsite or a pair of EMC Clarions. I am interested in looking at a dual head Sun Unified Storage 7310 and a lower end sun at our offsite. The numbers seem to be literally half for the Sun solution, so I feel like I have to missing something on it.

For usage, the primary purpose will be backend storage for about 100 VMs, some cifs, and some iSCSI/fibre storage for a few database servers.

Any thoughts from you guys?

Is your offsite storage intended to be a replication target?

What applications are you looking at running on the storage? MSSQL? Exchange, etc?

A NetApp box will do CIFS, NFS, iSCSI, and FCP all in one unit. I think the EMC Clariion only does iSCSI/FCP out of the box but you can throw a celerra in front of it or build out a fileserver.

What licensed options did you include with your storage from the other two vendors?

Vanilla
Feb 24, 2002

Hay guys what's going on in th

adorai posted:

We are currently looking at replacing our aging ibm san with something new. The top two on our list are a pair of Netapp 3020s and a 2050 for our offsite or a pair of EMC Clarions. I am interested in looking at a dual head Sun Unified Storage 7310 and a lower end sun at our offsite. The numbers seem to be literally half for the Sun solution, so I feel like I have to missing something on it.

For usage, the primary purpose will be backend storage for about 100 VMs, some cifs, and some iSCSI/fibre storage for a few database servers.

Any thoughts from you guys?

Personally I say avoid SUN. Can't go wrong with Netapp or EMC, both good kit.

EMC Clariion will do as 1000101 says but EMC has a Unified array that will do the whole shebang - FC, iSCSI, CIFS, NFS.

TobyObi
Mar 7, 2005
Ain't nobody Obi like Toby ;)

What are your issues with the Sun kit?

oblomov
Jun 20, 2002

Meh... #overrated

TobyObi posted:

What are your issues with the Sun kit?

Does Sun storage have any VSS snapshot software for Windows now? Back in December they did not and neither did they have any vmware compatible snapshotting or anything else. I really liked their hardware, but their software was just way too raw for my liking.

I went with Equallogic instead for that particular project and we got bunch of NetApp boxes since for a few other projects. However, if Sun software stack improved (or you don't need it and NFS/iSCSI on Linux/Solaris is good enough), I'd definitely take a look. I did hear bad things about support, and heck, even their pre-sales tech support was meh.

HorusTheAvenger
Nov 7, 2005


oblomov posted:

Does Sun storage have any VSS snapshot software for Windows now? Back in December they did not and neither did they have any vmware compatible snapshotting or anything else. I really liked their hardware, but their software was just way too raw for my liking.

They have supported Shadow Copy on CIFS shares since the 2009.Q2 release:

http://wikis.sun.com/display/FishWorks/ak-2009.04.10.0.0+Release+Notes

quote:

Microsoft shadow copy for shared folders

The CIFS server now supports viewing snapshots from Windows clients. Filesystem snapshots are automatically made available through the "Previous Versions" tab of the folder properties. For more information on this feature, see the Microsoft help page.

No tie-ins like SnapManager for Exchange like what NetApp has for snapshotting iSCSI luns yet afaict though.

Bluecobra
Sep 11, 2001

The Future's So Bright I Gotta Wear Shades

TobyObi posted:

What are your issues with the Sun kit?
Probably because he works for EMC? We had no issues with our 6140 controller that we recently upgraded to a 6580. Though I found that these controllers are actually made by LSI and that IBM uses the same hardware in some of their storage lines. I recently built a poor-man's Sun Storage 7310 with a 24TB J4400 array and an existing 8-core X4100 M2 server with 32GB of memory and it works well.

Vanilla
Feb 24, 2002

Hay guys what's going on in th

Bluecobra posted:

Probably because he works for EMC?

Yes, but if that was the only reason why would I recommend Netapp in the same breath? Do we wub them <3 <3? :)

TobyObi posted:

What are your issues with the Sun kit?



So my issues with the SUN kit.... :crossarms:

Firstly there’s a lot missing in the SUN range that people have come to expect and that the EMC and Netapp boxes are offering.

Replication – Not too sure about Netapp but EMC replicates based on RPO. Got a file system that’s pretty important? Set a 1 minute RPO. It’ll even tell you if it can’t maintain this. With the Sun box it’s the most basic replication ever. No throttling or any form of control – I think it’s free. It’s also only Asynchronous, no sync.

Then there are a few things that the Netapp and EMC arrays do have – such as deduplication (at no extra charge on either), the ability to have a wide range of drive sizes and type (not just 1TB drives or a very limited selection), Fibre Channel, RAID 1, RAID 10, RAID 3.

I could go on forever. No kind off application snapshot integration such as with NTAP Snapmanger or EMC RM. Performance monitoring that only goes from CPu to User (what about the actual backend?? The file system isnt everything).

They’ll have a new range out eventually. Naturally they’ll have Fibre Channel by then and you’ll probably find the usual story – no upgrade path, gotta buy new.

Then onto Sun as a business. Never have I seen a company have so many fantastic ideas and fail to bring them to the market. They bring these items out to market, persuade a few thinkers to adopt these revolutionary products and then drop them. Leaving their key sponsors holding a product that is end of life. People just have no confidence in them.

Vanilla fucked around with this message at 16:40 on Aug 27, 2009

1000101
May 14, 2003

BIRTHDAY BIRTHDAY BIRTHDAY BIRTHDAY BIRTHDAY BIRTHDAY FRUITCAKE!

To clarify some things on the NetApp:

It supports RAID 4 and RAID 6 (NetApp calls it RAID DP).

It in fact does support replication based on your RPO or it can do synchronous.

One thing I found with some Sun Storage is that it doesn't appear to do active/active very well. Which is to say if I have a LUN presented off controller A and somebody tries to access that same LUN via controller B, B ends up usurping the LUN.

I had a customer who build their SAN design where controller A was entirely on fabric A, and controller B was entirely on fabric B. When one of his ESX hosts had a path failure the LUNs kept hopping back and forth between controllers which drug performance down.

Not sure if this is still an issue (it's simple enough to fix) so if someone has a similar setup and could confirm that would be great.

TobyObi
Mar 7, 2005
Ain't nobody Obi like Toby ;)

Vanilla posted:

the ability to have a wide range of drive sizes and type (not just 1TB drives or a very limited selection), Fibre Channel, RAID 1, RAID 10, RAID 3.
To be honest, this sounds more like a gripe with a certain model, rather than an entire company. I have a pair of Sun arrays, that are "entry level" that support all of these things.

I mean, I have an EMC AX150i. I haven't written off the entirety of EMC, just the shitheel who thought that he did a good job whipping that up in about 20 minutes.

I'm not disagreeing on some of your other points though.

Vanilla
Feb 24, 2002

Hay guys what's going on in th

TobyObi posted:

To be honest, this sounds more like a gripe with a certain model, rather than an entire company. I have a pair of Sun arrays, that are "entry level" that support all of these things.

Indeed it is a gripe - this is supposed to be their super product to destroy EMC and Netapp but is so far off the mark and missing so many key, simple things that it's laughable!

It's the typical SUN home grown build hardware around ZFS. It's so good it'll sell itself! :suicide:


quote:

I mean, I have an EMC AX150i. I haven't written off the entirety of EMC, just the shitheel who thought that he did a good job whipping that up in about 20 minutes.

35 minutes*.



*included 7 minutes cigarette break.

lilbean
Oct 2, 2003



A while back I mentioned filling up a 4540 with lovely consumer disks, and I went ahead last Friday and did just that. Well, not full yet, but I've replaced to begin with one ZFS vdev of 6 disks plus one of the available hot spares with Seagate 1TB 7200.12 drives (100CDN each). In the last weekend and week we've pounded the poo poo out of them since our 4540 is our primary backup server and acts as a disk-to-disk staging area. No problems whatsoever so far.

Also, I think my manager is ready to blow me for being able to expand at 20 cents per usable gigabyte (after RAIDZ2 and a spare are taken into account).

da sponge
May 24, 2004

..and you've eaten your pen. simply stunning.

Has anyone heard of backblaze? Seems like a personal backup service, but their blog post on their infrastructure is pretty cool - http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/

optikalus
Apr 17, 2008


da sponge posted:

Has anyone heard of backblaze? Seems like a personal backup service, but their blog post on their infrastructure is pretty cool - http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/

Someone posted that blog entry to my forums a few days ago. I see several issues with it, mainly cooling and power. Only 6 fans to cool 45 drives? Those are going to bake.

The fact that it spikes 14A on power up is going to cause major issues. Real RAID hardware would have delayed start on all those drives.

I have delayed start on Areca RAID cards in 6 SuperMicro SuperServers with 8x 250gb SATA drives, and I have to be very careful powering them on otherwise I'll pop my cabinet's breaker :/ I just run a few as cold spares now.

If that set up had a real RAID adapter (not fakeRAID), or at least something like ZFS, I'd give it props. Doing a fakeRAID 6 is just asking for failures. I played with fakeRAID on BSD and RHEL for years, before writing both implementations off for production use. I personally wouldn't bother using any RAID at all, but just set them up JBOD. My reasoning being that it is a backup service -- if a drive fails, yes, that backup is gone, but the client should be able to see that the checksum is different and resync.

I wanted to build something similar, but with systems instead of drives. I even built a prototype, but power stability was a problem. However, I had /much/ more airflow, and no drives (netboot).

Edit: also, anyone remember that whitepaper about SATA drives reaching 2TB in a RAID setting will almost always be guaranteed to fail upon rebuild due to the size of the drive exceeding the drive's own bit error MTBF?

optikalus fucked around with this message at 18:11 on Sep 2, 2009

da sponge
May 24, 2004

..and you've eaten your pen. simply stunning.

optikalus posted:

Edit: also, anyone remember that whitepaper about SATA drives reaching 2TB in a RAID setting will almost always be guaranteed to fail upon rebuild due to the size of the drive exceeding the drive's own bit error MTBF?

Yep. I remember it on a different site, but here's the link I found - http://blogs.zdnet.com/storage/?p=162

Also, remember that google paper a while back, showing little correlation between hdd temperature and failure rate - http://labs.google.com/papers/disk_failures.html.

da sponge fucked around with this message at 18:32 on Sep 2, 2009

Bob Morales
Aug 18, 2006


Just wear the fucking mask, Bob

I don't care how many people I probably infected with COVID-19 while refusing to wear a mask, my comfort is far more important than the health and safety of everyone around me!



$5/user/month for unlimited backup? I wonder how much they count on the average user actually using, to turn a profit?

Each pod is $8,000 and that's not counting employees or bandwith/cooling etc.

H110Hawk
Dec 28, 2006


optikalus posted:

Someone posted that blog entry to my forums a few days ago. I see several issues with it, mainly cooling and power. Only 6 fans to cool 45 drives? Those are going to bake.

I think you're wrong here, fan count is a lot less important than CFM forced over the disks. Sure the disks may be hotter than a 2U server with 4 disks in it, but they will be consistently the same temperature. The disks will probably suffer some premature failure, but that is the whole point of RAID. Get a cheapo seagate AS disk with 5 year warranty and just replace them as they fail.

Non-delayed startup and initial power surge are certainly very valid problems. If you're doing 14A on boot you can only have ~1 of those per 20A circuit. It makes me wonder if they wander around with an extension cord on a dedicated circuit to turn them on, then migrate it over to the running circuits. Reading the blog it sounds like what they do, turn on the disks, then turn on the cpu/etc

Makes me wonder where are they finding a datacenter that can keep themselves even moderately warm for that many watts/sq ft. I imagine with proper ventilation on the hot rows it should be doable, but I can't imagine how many heat exchangers they need. You don't need to keep it really cold, you just have to keep up with the heat put out by the devices and makes sure hot air isn't getting circulated back over to the intake fans in the cold rows.

lilbean
Oct 2, 2003



It's neat but each power supply runs about half of the components. That's doubling a point a failure and a pretty crappy compromise.

adorai
Nov 2, 2002

10/27/04 Never forget

Grimey Drawer

optikalus posted:

Edit: also, anyone remember that whitepaper about SATA drives reaching 2TB in a RAID setting will almost always be guaranteed to fail upon rebuild due to the size of the drive exceeding the drive's own bit error MTBF?
The URE rate on a 1TB SATA drive is high enough that it becomes likely you will encounter one while rebuilding a large array containing them. If you are running RAID5, that means you will probably lose the array. RAID6 is still safe.

StabbinHobo
Oct 18, 2002

by Jeffrey of YOSPOS


its absolutely insane for anyone without two very niche aspects to their storage needs:
- can be slower than poo poo sliding up hill
- can afford a 67TB outage to replace one drive

I think I'd trust mogilefs's redundancy policies more than linux's software raid6.

ragzilla
Sep 9, 2005
don't ask me, i only work here




lilbean posted:

It's neat but each power supply runs about half of the components. That's doubling a point a failure and a pretty crappy compromise.

Depends on their environment- if they also implement redundancy at the network or application levels (storing the same file on multiples of these boxes) ala waffleimages/lefthand it doesn't really matter if one of the boxes dies on rebuild, just get copies redistributed and bring the node back into the cluster when it's fixed. But at that point you're just reinventing mogilefs/googlefs.

lilbean
Oct 2, 2003



ragzilla posted:

Depends on their environment- if they also implement redundancy at the network or application levels (storing the same file on multiples of these boxes) ala waffleimages/lefthand it doesn't really matter if one of the boxes dies on rebuild, just get copies redistributed and bring the node back into the cluster when it's fixed. But at that point you're just reinventing mogilefs/googlefs.
Oh sure, I know you can replicate everything at a higher-than-device level. It just seems to me that I'd rather add a few thousand dollars for a higher grade power system.

Adbot
ADBOT LOVES YOU

adorai
Nov 2, 2002

10/27/04 Never forget

Grimey Drawer

StabbinHobo posted:

its absolutely insane for anyone without two very niche aspects to their storage needs:
- can be slower than poo poo sliding up hill
- can afford a 67TB outage to replace one drive

I think I'd trust mogilefs's redundancy policies more than linux's software raid6.
since it's apparently commodity hardware, i think you could probably run opensolaris w/ raidz2 on it if you wanted to.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply