rage-saq posted:

I'm pretty sure synchronous replication (where it ACKs a mirrored write before continuing on the source side) and 70ms of latency would destroy nearly any production database performance. At a certain point you can't overcome latency.

H110Hawk posted:

Call up that dell rep who made promises and get him to tell you how. Withhold payment if that is still possible. If you just need hourly crash-equivalent data because your application is fully transaction aware then it shouldn't be that bad. Play hardball with him. Have him send you a unit which works if this one doesn't, and send back the old one. "You promised me X gigabytes of storage with the ability to make a consistent Oracle snapshot for $N."

In theory you can make a backup based on a transaction view without locking your entire database for writes, but I have never used Oracle.

H110Hawk posted:

Misogynist posted:

I had absolutely no difficulty reaching wire speed with both NFS and iSCSI in our test environment (we eventually deployed on FC anyway at the behest of our director) and neither did VMware. Your consultant is feeding you a crock of poo poo. I did find that we had a 15% drop in iSCSI performance when routing through the core, whereas we had near 100% of wire speed with the initiators and targets on the same VLAN. We might have been able to tune this out by tweaking TCP settings to accommodate the extra latency, but we were only using it in dev. We were not using jumbo frames. This probably made the performance hit much worse than it might have been if our network was properly segmented out for routing storage traffic.

In terms of 10 gigabit: I imagine this is the area where you will find performance relative to wire speed is very, very heavily vendor-dependent (and obviously heavily dependent on your backend spindles and layout as well). I'm under the impression that iSCSI at 10 gigabit is still rather CPU-intensive and this is one circumstance where you might want to consider one of the new Broadcom iSCSI 10GbE HBAs. Then again, if you need 10 gigabits of iSCSI throughput on a single box, you probably have somewhere in the vicinity of 32 cores anyway.

The real sell of 10 gigabit for VMware environments at this point is reduced cabling -- use two 10 gigabit uplinks as your network connections to each VMware box and use the 1000V's QoS features to sort out and prioritize management, VMotion/FT, and VM traffic. It's still pretty rare to see a system that will suck down all that iSCSI bandwidth. Remember that most disk throughput is measured in IOPS, not gigabits.

adorai posted:

What was he there to consult on? Just the speed issue? If so, I would handle it by not paying the bill until he did his job. In testing we have been able to saturate a single gigabit link (around 95MBps, close enough) with iSCSI, so he's full of poo poo.

Are you able to add a second link, just to see if that helps?

H110Hawk posted:

First I would make sure your setup is indeed capable of performing faster. Double check that your disks are bored, your server isn't backed up elsewhere, and that the network itself is capable of filling the pipe. Throwing 10gbe at a problem without verifying the source cause is a great way to get 45MBps (bits? Bytes?) of throughput on a 10Gbps pipe.

Have your network guy sign off on the network after actually checking the ports. Make sure your NIC's are the right brand (Intel, Broadcom if you must), etc.

Nebulis01 posted:

I could see the 45mbps limit with a a large queue depth depending on the read/write size, and array setup perhaps? We get poo poo performance with a QD of 32 in CrystalMark but since our SAN is a glorified file store anyway, I'm not really worried about it.

EDIT:
I was curious I ran an ATTO 2.46 run with 4KB blocks and a Queue Depth of 10.
1GB Length - Read 42MB/Sec, Write 1.1MB/Sec.
2GB Length - Read 52MB/Sec, Write 5.2MB/Sec

Nebulis01 posted:

The only thing I can think of (not a SAN/Storage guy myself) would be that you're having issues using just one link and having such small reads/writes. Is there any way you can utilize the dell MPIO driver and add another link or two to that box?


Also, I've got our setup running on Broadcom and they seem to run just fine. What firmware revision are you running? There have been some pretty decent fixes in the last few firmware/MPIO driver releases

Misogynist posted:

This is a very valid point -- I'm talking about wire speed and your numbers aren't coming from the wire. Can you get utilization numbers from your server's iSCSI switchport rather than the storage processor?

Mausi posted:

If this is the Equalogic server he's talking about they're designed to work like this. He mentioned the Equalogic MPIO driver, so that's probably the case.
If it's the random Windows box on the other end of the iSCSI link then it would be more of a concern.

Misogynist posted:

I'd wager incompetence over malice, though the end result is the same.

quote:

CONCLUSION
It is the conclusion of the consultingCompany team that during the times of high disk utilization, the four (4) 1Gbe iSCSI connections are experiencing periods of link saturation. Due to the overhead associated with the TCP/IP protocol, 1Gbe connections typically have a sustained throughput ceiling of approximately 40MB (megabytes) per second, as is reflected in the collected data. This limitation is causing disk contention as the demand for disk access is higher than the available throughput. The high level of disk queuing during these periods of increased utilization further supports this finding.

quote:

RECOMMENDATIONS AND NEXT STEPS
The performance data shows that the current four (4) 1Gbe connections to the iSCSI SAN does not offer enough throughput for the hosted workload during times of high utilization. The configuration of the underlying iSCSI implementation adheres to most of Microsoft's best practices. Due to the throughput limitations associated with implementing iSCSI over 1Gbe, there are few options for increasing the disk performance with this current infrastructure. Setting the Delayed Acknowledgement option on DB1 and DB2 to “0” could reduce the number of packet retransmits and enabling Jumbo Frames (MTU size increased to 9000 bytes) will provide a lower number of packets sent across the network. These two options could provide better efficiency over the iSCSI network; however, the 1Gbe links will still pose a limitation to the total available throughput to the storage system.

As the system's workload continues to increase, it would be recommended to consider upgrading the existing iSCSI implementation to 10Gbe or converting the storage system to 4Gb or 8Gb Fiber Channel. Both alternatives provide greater throughput to the storage system and can better accommodate the increased workload.
It was noted while on site that companyName has a number of similar storage system within the production environment. It would be further recommended to investigate the performance of these systems as well and determine if an enterprise class consolidated storage system would better suit the business needs. An enterprise level storage systems can also provide multiple connection options, such as iSCSI, FCP, CIFS and NFS and provides the flexibility to allocate storage based on service requirements and performance demand while lowering the associated administrative overhead.

HorusTheAvenger posted:

Depends on your Dell rep. Many reps would enjoy the prospects of selling 10Gb infrastructure.

Is that how the consultants gave the perfmon screenshot? It's kind of hard to read.

Nomex posted:

Is there a time of the day or month when this DB has low I/O? If there is, you should try copying a very large file from the server to the storage. This will at least show you if it is actually the DB causing the issue or if it's a problem with the setup.

1000101 posted:

1. You have enough disks to support the workload

quote:

2. You're using an appropriate RAID level for the workload

quote:

3. Enable flow control on your switches if your storage supports it

quote:

6. What are your server's CPUs doing at these peak times? Its possible that your application is choking out the CPU which means less time spent on a software iSCSI initiator.

quote:

3. Enable flow control on your switches if your storage supports it
4. Make sure your storage is on the same switch as your database servers (multiple hops with iSCSI can be bad depending on network topology)
7. What are your switches doing?

Mausi posted:

Everyone has been talking about iSCSI throughput on devices other than Equalogic - does anyone know what the throughput overhead of their load balancing algorithm is?

I'm not saying the consultant is right - but it could be endemic to the Equalogic array.

Nomex posted:

How many disks did you say were in this array? Also what size and speed are they?

SQLIO posted:

B:\>sqlio -kR -s360 -fsequential -o8 -b128 -LS -Fparam.txt B:\testfile.dat
sqlio v1.5.SG
using system counter for latency timings, -1900967296 counts per second
parameter file used: param.txt
file b:\testfile.dat with 2 threads (0-1) using mask 0x0 (0)
2 threads reading for 360 secs from file b:\testfile.dat
using 128KB sequential IOs
enabling multiple I/Os per thread with 8 outstanding
size of file b:\testfile.dat needs to be: 104857600 bytes
current file size: 8388608 bytes
need to expand by: 96468992 bytes
expanding b:\testfile.dat ... done.
using specified size: 100 MB for file: b:\testfile.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 1723.94
MBs/sec: 215.49
latency metrics:
Min_Latency(ms): 2
Avg_Latency(ms): 8
Max_Latency(ms): 3026
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 0 0 0 0 1 2 9 23 34 12 5 3 2 2 1 1 1 1 0 0 0 0 0 0 1

B:\>

hackedaccount posted:

Hey three, any updates on the consultant that says iSCSI is too much overhead, wants to you rip out everything and install FC, etc?

Misogynist posted:

Is there any particular reason you're avoiding VMDKs? Storage vMotion would basically eliminate all of your problems here.

adorai posted:

I would guest side iscsi and allocate 30% more space than you are currently using. Then I would resize volume (hopefully your storage supports this) as needed

skipdogg posted:

I had some engineers looking for a SAN, but they wanted granular control of the disks and what LUN etc they went to. They wanted to say disks 0,1,2,3 are part of this, and 4-10 are here, etc. The LeftHand boxes don't do that. They spread the data out automagically.

adorai posted:

HA and DR are completely different concepts.

If you want an easy, drop-in, push one button for DR solution in a VMware environment with all VMDKs you would probably use SRM. It's like $3k a proc.

1000101 posted:

I think you're walking down the right road for your budget. I assume the DR site is probably running free ESXi so you'll have to do a couple things:

1. Enable sshd/remote support shell on the ESXi server
2. Create yourself a handy recovery script to script bringing the volumes up! Now you have something you can type in, get a cup of coffee then come back.

Things you'll want to know how to use in the script:

Check with the VSA to figure out how to present your replicated LUN to your server. It probably entails breaking replication and making it writable. Ideally this can be scripted via the CLI.

Next some VMware specific stuff:
'esxcfg-volumes' -This command lets you tell VMware its okay to mount a replicated volume. You'll want to let it resignature the LUNs in question.

'esxcfg-rescan' -Use this to rescan the iSCSI initiator after you present the LUN and allow for a re-sign (don't recall if this is 100% required in 4.X anymore, the last script I wrote for this was for 3.5.)

Since its ESXi you're going to want to fiddle about in 'vmware-vim-cmd' (this is a way to get into the VMware management agent via the CLI) and feed it arguments via a find of .vmx files in your new datastore after the re-scan.

At this point you can actually use vmware-vim-cmd to power everything on for you and answer the question "hey did you move this or copy this?" (you probably just want to say you moved it.)

I had to build something like this for one of my customers who's outsourced IT is probably worse than herding cats. I use ssh key authentication for everything and all some guy has to do is run "StartDR.sh" and the script does everything he needs.

InferiorWang posted:

After reading it in that context, it makes sense now.

The local inside rep at HP refuses to call me back despite leaving multiple messages. I think it would be worthwhile looking to an outside source to help me out with this, especially if I can't get anything from HP.

szlevi posted:

You're right - hey, even I designed our system the same way - but I think you're forgetting the fact these iSCSI-boxes are the low-end SAN systems, mainly bought by SMB; you can argue about it but almost every time I talked to someone about MPIO they all said the only reason they use it is the higher throughput and no, they didn't have a second switch...

Vanilla posted:

Depends but yes, is quite possible.

In a read environment you can get around 130 IOPS (worst case) of random read IO from a 10k disk and 180 IO from a 15k disk.

However, once you add writes into the equation then you are upping the number of IOPS. Each write in RAID 10 creates 2 IOPS (write to one disk then the other). Each write in a RAID 5 environment means 4 IOPS (two reads to make a parity and two writes to write the parity).

So a RAID 5 environment you may have faster disks but it's possible a lot more IOPS are required for the RAID 5 overhead.

skipdogg posted:

What solution did you end up using for Virtual Desktops? How happy are you with it?

Mierdaan posted:

I've been getting Autosupport alerts all night about failing power supplies and spare disks for a FAS3170 at a credit union in California. I've never heard of this credit union; I have no clue why I'm listed as a contact for it.

I hope their actual storage admin knows

Intraveinous posted:

Crap... sorry about that. I should have just uploaded it somewhere.

Before too much longer, the Compellent array will be just that: a couple of Dell Servers with MDxxxx drive shelves. Currently, the controllers are 3U Supermicro boxen. The shelves are Xyratex OEM, who make quite a few different array vendor's shelves. The difference will be that the secret sauce is the Storage Center software rather than the ZFS software.

If you have an experienced admin who can spend time keeping it running/tuned to the workload; advantage may go to ZFS whitebox due to the ability to use cheaper SSDs for L2ARC/ZIL. I'd still advocate using "enterprise" SSD over a Vertex 2, but at least you could use ~$700 SSDs that are approved/supported by the server manufacturer, vs the $10K+ most array vendors seem to charge per SSD.

If array maintenance is going to just be another task on a list of CJ duties, having someone you can call at 3AM to help you through recovery from a problem is obviously a plus. The advantage goes to Compellent there. CoPilot is outstanding, especially for the company that can't afford a dedicated storage admin.

This assumes iSCSI connection, since last time I set up any ZFS/commodity hardware based "SAN", you didn't have an option for presenting it out over fibre channel. That may have changed by now, so I might be wrong.

Best of luck to you, I'm sure whatever you end up with will work fine, and it'd be (it was) a harder decision against anyone other than Oracle.

Nukelear v.2 posted:

Tape is awesome, however don't actually buy Dell autoloaders. Both powervaults we've owned have died in only a couple years, and when they were working they changed slowly and made horrible grinding noises. We've replaced ours with HP G2 loaders and I have zero complaints about them. Still quite cheap as well.

Edit: Small sample I know but we opened them up and the mechanical build quality is on par with consumer ink jets. I can't imagine them actually surviving anywhere.