Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
luminalflux
May 27, 2005



Anyone have any hard numbers on performance gains/losses using jumbo frames on iSCSI VMware storage and vMotion over GbE?

I found some good numbers (like single percent increase in performance), but they were invalidated by the guy having a dodgy switch. The documentation for my HP P4000 says "likely not needed" but I'm curious about how slight the gains are.

Adbot
ADBOT LOVES YOU

luminalflux
May 27, 2005



I went to one a few months back when they held the first one in Stockholm. It boiled down to boring IT people caring a lot about very boring stuff (from my perspective as a dev/sysadmin working with web apps on linux). It's one of those events where the event itself was boring as gently caress but the network was really really interesting and useful. Then again, they've promised more specific meets geared towards different areas of virtualization so it might be useful.

In this case, the event itself was free, dunno if all are. Walked out of there with a cool vMug, some good contacts, eval license for VEAAM. Also a free lunch and free beer :toot:

luminalflux
May 27, 2005



Whoever came up with the VMWare SDK needs to seriously reconsider their life choices. Such a huge pile of enterprise garbage. I'm really glad that someone at VMWare came to their senses and wrote a ruby wrapper around it (rbvmomi) to reduce most of the suck that is trying to automate deploying VMs.

luminalflux
May 27, 2005



Yes, people that automate everything with unix tools (like Chef and Ruby in my case). The only windows machine I'm running is the vCenter server. Also someone wrote a pretty sweet plugin to Chef's knife client for automating vSphere (knife-vsphere) which I'm using as a starting point for my scripts.

luminalflux
May 27, 2005



How much are the disks used on an ESXi host if all my datastore is over iSCSI? Should I be overly concerned with disk performance?

luminalflux
May 27, 2005



Really? It doesn't swap to local disk at all?

luminalflux
May 27, 2005



Is there seriously no way to set the iLO password of an ESXi 4.1 host on HP hardware via software? In Linux there's hponcfg that can do this, and a .vib is provided by HP for hponcfg for ESXi 5.0, but I can't find one for 4.1. Don't really want to upgrade to ESXi 5 either.

luminalflux
May 27, 2005



DL-series :smith:

luminalflux
May 27, 2005



Well yeah but I'm getting tired of all these trips to the colo facility on the other side of town, and one host is running vms that aren't on shared storage for performance reasons (but downtime for these is begrudgingly tolerated). Guess which host has an iLO password tat nobody remembers? :v:

luminalflux
May 27, 2005



Erwin posted:

Are you licensed for storage vMotion? Also, what in the world shared storage do you have that under-performs local storage?

Shared storage is iSCSI over 2x1 Gbe, local storage is SAS. The VMs that are on local storage are for our testing environment, we moved them there since their disk traffic was interfering with production disk bandwidth for the websites I run.

I'm licensed for Storage vMotion, but to be honest moving 300+ GB to the SAN and then back takes longer than just rebooting and fixing the problems with the apps not restarting correctly.

luminalflux
May 27, 2005



evil_bunnY posted:

And yeah svMotion would have trivialized the whole thing.

Doing that many hosts in one go (select all -> migrate storage) doesn't work that well, it needs to be micromanaged (at least on that combination of controller and SAN). Of course it's scriptable, but so is fixing the reboot problems so I just reboot instead.

luminalflux
May 27, 2005



Why shouldn't I migrate from vSwitches to dvSwitch? What shouldn't be moved to vSwitch

Currently I have 3 vSwitches per ESXi: one for management traffic and vMotion, one for VM "production" traffic and one for iSCSI (VMs don't see iSCSI, they only see datastores). Each vSwitch is teamed to 2 NICs. I've got 4 ESXis currently in the same cluster, all with Enterprise+ licenses connected to vCenter.

luminalflux
May 27, 2005



Is there a way to extract performance metrics from vCenter to another graphing tool? I'd like to take the graphs I can see in vSphere Client under the "Performance" tab and get them into something like Graphite or Munin. I looked a bit at SNMP, but vCenter only has traps, no performance counters at all.

luminalflux
May 27, 2005



Erwin posted:

You can export to Excel, and from there go pretty much anywhere (you might be able to do CSV directly). Of course, it's a manual process. Also, you could probably query SQL directly.

I was thinking more along the line of real-time information, so that's not really useful. Any ideas on where I could find the counters in SQL?

Splunk seems to have a statistics forwarder for vCenter, so it's definitely doable somehow.

luminalflux
May 27, 2005



Mausi posted:

I know I'm a little late here, but PowerCLI has the get-stat command, if you do a little googling there's some great posts by LucD on using it.
You can also use get-view on virtualmachine and gather the VM summary, which will always have quickstats and is very fast to return. There are also pretty simple techniques to gather the realtime stats as well.

I wrote a bespoke script with it to gather VM cpu/mem stats from 17 vCenters and dig out what needed rightsizing, so it'll probably provide you what you need.

I later found a post on how to get stats from PowerCLI into Graphite, which is exactly what I was looking for. I would have preferred VMware exposing that over SNMP or the Ruby interface having support for get-stat, instead of having to use yet another system to get stats in, but this works.

luminalflux
May 27, 2005



Mausi posted:

I'm glad you found a way to get it working, but that dude's script is massively inefficient. He shouldn't be running a get-stat for each host individually unless that's the only way to get realtime stats; much faster to run it for all hosts by feeding get-stat a list of entities, then running a group-object by entityId afterwards to process on.
Most of the effort with PowerCli isn't getting it working, it's making it run in less than 4 hrs and 4GB of ram.

Looks like I can feed a collection, but as for extraction from the collection, no idea. I'm a total powershell newbie and I can't even program in it effeciently since for some reason typing $ and | through the RDP session from my mac gets mangled. I found LucDs posts though and probably should get a windows VM installed so I can play with this stuff.

luminalflux
May 27, 2005



evil_bunnY posted:

Put the vSphere client on your bootcamp partition and put the vm on its own desktop, and pronto!

Then ctrl+arrows to switch desktops.

I actually caved and did this. I feel sooo dirty. (Fusion has seamless windowing as well, which is still pretty cool even though it's been around for a long time)

luminalflux
May 27, 2005



What magic rubber chicken do I need to wave to get vCenter to realize that yes, I do have redundant paths to my storage? I have 2 LeftHand boxes, each with 2 NICs connected to 2 different switches. Each ESXi has 2 interfaces in the iSCSI vSwitch, each going to a different switch. The iSCSI vSwitch was created according to HPs best practices doc: 2 vmknics, with one IF as active and the other as unused (shifted so vmk1 has nic2 active, nic3 unused & vmk2 has nic3 active, nic2 unused).

Yet when I view the storage views tab, it says "Partial/No Redundancy". :wtc:

luminalflux
May 27, 2005



It's 4.1. Dunno if this should be in the storage thread, but when I inspect the datastore paths, I have 2 IQNs to the same IP address, which feels wrong.

luminalflux
May 27, 2005



I'm having issues when deploying a Centos 6.3 VM from a template in vCenter on ESXi 4.1 U2. The resulting VM doesn't have the hostname changed at all (it gets the same as the template had) and the network device isn't marked as connected. Anyone else run into this and fix it? As it is I can't deploy Centos 6 VMs which makes me kinda sad.

edit: nevermind, i'm retarded, vmware-tools wasn't correctly installed.

luminalflux fucked around with this message at 11:41 on Jul 20, 2012

luminalflux
May 27, 2005



Moey posted:

I can finally upgrade to 5!

Same here. Also I now have a case for adding more memory cluster-wide. We had Enterprise Plus already, so the 96GB limit was a kick in the balls if I ever wanted to upgrade past the cheap 72GB on a DL360 (or slightly more expensive 96GB).

vvvv If it was me I'd be kicking my sales rep instead vvvv

luminalflux fucked around with this message at 23:22 on Aug 23, 2012

luminalflux
May 27, 2005



:toot: No more vRAM licensing, no move to per-core licensing :toot:

luminalflux
May 27, 2005



Honestly this means I can actually migrate to 5.1, or at least think about it. The vDS changes look sweet as well (LACP, mirroring), but I'm kinda waiting for the Nicira tech to be integrated.

luminalflux
May 27, 2005



DevNull posted:

Have you not used the console because it sucked, or because there was never a need to?

Both: I'm on a mac so I need to RDP in to a windows machine to use the vCenter client, and once the machine is cloned we do everything over SSH.

The key repeat sucks so bad over RDP and the client so even logging in is hard.

luminalflux
May 27, 2005



I just love it when ESXi becomes catatonic, but still appears up to HA. I can ping the host, but can't do poo poo: VMs aren't responding and management requests from vCenter does nothing. Is there a way to tell vCenter to drop a host that is misbehaving like this, without having to do a 2am cab ride to reboot it?

luminalflux
May 27, 2005



Mierdaan posted:

I'd check out your logs and see if you can figure out what's going on. If the host is talking to the HA master and it's writing to heartbeat datastores (if you're on 5) HA's not going to know any better. I don't think even VM HA is going to do anything if the VMs wouldn't respond to a restart request anyways.

Nope, it's 4.1. As soon as I rebooted via iLO the VMs started up on other hosts. I don't have logs on the ESXi since I neglected to configure a persistent scratch storage and/or set up a syslog target. Spent part of my sunday doing exactly that.

Is it worth even talking to support if I don't have logs from the hypervisor, or should I just upgrade to U3 that just came out and see if it happens again? This is the second time in a couple weeks this has happened (huge load spike, this specific machine goes catatonic and needs some iLO love).

luminalflux
May 27, 2005



Misogynist posted:

I'm noticing you're not following through with "set up a syslog server" as part of the action plan you're describing.

Sorry if I was unclear - I set up both persistent scratch storage AND a syslog server, so I have logs in both places (verified that they're being written in both places as well). And I more or less only deal with Unix servers.

luminalflux
May 27, 2005



quote:

1) Were the VMs online and responsive (yet stuck on that box - try ping, ssh, rdp, vnc, etc - reach out to known services in the VMs)?
Nope (found this out by nagios going apeshit)
2) Did the box ping?
Yep.
3) Can vCenter server still ping and resolve that host by FQDN and hostname?
Yep.

As for console access, I don't have the relevant licenses for text-console (only serial port access).

luminalflux
May 27, 2005



Storage is on iSCSI, and the other 3 ESXis could reach the datastores. Everything's on GbE with NIC teaming into 2 switches. So it could very well be the NIC - both iSCSI links are on the on-board NIC, whereas the management and production networks are split over the on-board Broadcom and an Intel in the PCI slot.

luminalflux
May 27, 2005



Kachunkachunk posted:

Alrighty, so perhaps a remote syslog server would successfully receive logs during a recurrence. Are you able to set up iSCSI so it uses different physical adapters?

Yep, that's the plan. Someone else set up these NICs up, I have no idea why iSCSI isn't split over the different NICs like the other nets (each ESXi has an on-board 4-port Broadcom and a 4-port intel in the PCI slot)

luminalflux
May 27, 2005



I just wish I could ditch vCenter's SQL Server for Postgres, since that's the only database I know how to run.

luminalflux
May 27, 2005



At what point do I actually need to upgrade from SQL Server Express to SQL Server for vCenter? Currently we're running Express 2005 for 4 hosts and 150 VMs with about 2GB data. I'm looking at upgrading to 5.0, but i need to upgrade to 2008 first and the document says I shouldn't use Express after I exceed either 5 hosts, 4GB or 50 VMs :ohdear:

luminalflux
May 27, 2005



So no performance issues or anything like that?

Do I need to upgrade to 2008 first, or will the installer take care of that upgrade as well?

luminalflux
May 27, 2005



Corvettefisher posted:

If you can't afford SQL then you might want to get 2008 r2, which has a DB max size of 10GB instead of 4 like 2005 has

I need to get 2008 anyway as 2005 isn't supported for vCenter 5. 10GB should probably be fine.
(It's not really an afford question, I just want to hold out as long as possible before shelling out more bucks to MS and hoping they'll make Postgresql a compatible database soon)

luminalflux
May 27, 2005



From what I can tell, VUM is supported, it just needs to be run in a windows guest instead. So if the DB size becomes a problem I'll guess I'll just deploy the appliance and deal with that hassle instead of shelling out.

luminalflux
May 27, 2005



three posted:

It makes me super sad VUM requires Windows, but I guess VCSA isn't ready for primetime anyway.

I've already paid for a Server 2008 license so I can manage that, but yeah it sucks big time. Almost makes me want to experiment a bit just to see if I can get this up and running in a lab environment.

luminalflux
May 27, 2005



Corvettefisher posted:

http://www.supermicro.com/products/nfo/FatTwin.cfm

Just purely asking, does anyone use or know people who use super micro? I can't help stop thinking how awesome that FatTwin looks, then I found out the price and can't stop thinking about it

We used to like 5 years ago, and ran the gently caress away after waaaay too many random issues.

luminalflux
May 27, 2005



Mierdaan posted:

How long should I spend troubleshooting terrible datastore latency when using Dell's R610 integrated broadcom NICs before I just replace them? Do these things work OK for anyone?

I used to build network hardware at my previous gig. We bought Broadcom PHYs and implemented our own MAC in VHDL. No, of course they didn't work for us and we got little documentation and support despite paying boatloads for the PHYs. gently caress Broadcom.

Corvettefisher posted:

Fixing it, I did a straight copy and paste of the first paragraph guess I just never fixed it

Should at least mention VirtualBox and Vagrant (get someone else to write about this). Maybe some stuff about Xen, KVM, Solaris/SmartOS Zones perhaps?
edit: And then there's stuff like OpenVZ as well.

luminalflux fucked around with this message at 12:47 on Sep 14, 2012

luminalflux
May 27, 2005



Mierdaan posted:

Anyone updated to vCenter or ESXi 5.1 yet? Any trip reports?

I'm wondering as well. Might go ahead with it next week.

Adbot
ADBOT LOVES YOU

luminalflux
May 27, 2005



Seriously, gently caress VMwares licensing department. I have one license can't be moved and another can't be upgrade, and support has a very lax interpretation of how long an hour is. Hi I've paid you a boatload, please fix this.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply