|
Wicaeed posted:This guy considers almost every aspect of virtualization to be a security vulnerability or flaw. Something about the fact that you have shared storage (ideally) with servers running in (potentially) the same memory sticks (!?!) rubs him wrong. vCenter is a four letter word to this guy. His number one job (it seems to me, probably unfairly) has been to try his damnest to make my teams job (Infrastructure Ops) harder to do. /rant
|
#
?
Nov 21, 2013 01:31
|
|
|
|
| # ? May 8, 2024 00:47 |
|
|
adorai posted:Technically it's possible. As far as I know, it's only been demonstrated one time and that exploit was fixed long ago. Here are a bunch of documents on it: http://www.blackhat.com/presentations/bh-usa-09/KORTCHINSKY/BHUSA09-Kortchinsky-Cloudburst-SLIDES.pdf http://www.blackhat.com/presentations/bh-usa-09/KORTCHINSKY/BHUSA09-Kortchinsky-Cloudburst-PAPER.pdf http://www.youtube.com/watch?v=NnYNaLSiOxY My co-workers just did a paper and presentation talking about the hardening of the SVGA device that has occurred since Cloudburst. The work that went into Cloudburst is actually really impressive, and the documentation and explanation of the exploit are really well done.
|
|
#
?
Nov 21, 2013 04:11
|
|
|
There are a bunch of ways to exploit a hypervisor, but most of which if you take some standard sec procedures and poo poo are easy to manage. You need to be proactive in managing and monitoring your environment. Realize things that leave the host are still bound to the same network security rules of your physical environment. So don't make a 100% flat network with iscsi, vmotion, and management.DevNull posted:Here are a bunch of documents on it: I haven't been able to recreate this on AMD procs but eh http://eprint.iacr.org/2013/448.pdf Some goon pointed this out Dilbert As FUCK fucked around with this message at 04:18 on Nov 21, 2013 |
|
#
?
Nov 21, 2013 04:16
|
|
|
Dilbert As gently caress posted:I haven't been able to recreate this on AMD procs but eh http://eprint.iacr.org/2013/448.pdf We actually just had a researcher at VMware on Monday giving a presentation about an attack very similar to this. While it is interesting at an academic level, it would be very difficult to actually exploit. Researchers do this type of attacked in very contrived conditions. Something like Cloudburst was very easy to exploit. In addition, it doesn't just break and RSA key used by a different guest it allows the guest to execute arbitrary code on the host.
|
|
#
?
Nov 21, 2013 04:30
|
|
|
I't an interesting topic to say the least
|
|
#
?
Nov 21, 2013 04:45
|
|
|
Docjowles posted:Vendor: "Oh that's an old chassis model and doesn't support virtualization at all. It has a ghetto homebrewed networking stack that doesn't support ARP ( Oh my gods, please tell us which vendor this is. Please don't say Supermicro. I work for a Supermicro VAR, and I know their blades are wrong and there are all sorts of reasons we don't sell them any more, but still...
|
|
#
?
Nov 21, 2013 06:34
|
|
|
McGlockenshire posted:Oh my gods, please tell us which vendor this is. Please don't say Supermicro. I work for a Supermicro VAR, and I know their blades are wrong and there are all sorts of reasons we don't sell them any more, but still... SeaMicro. I am pretty sure the product in question is discontinued as I can't even find it on their website, so don't take that as a comment on their current offerings. But still, argh.
|
|
#
?
Nov 21, 2013 16:39
|
|
|
So I'm working on configuring my first new host at work and am running into an issue with the DNS configuration setting in ESXi. The settings on the remaining hosts I have all are set to automatic and seem to resolve to the proper hostname, but whenever I reboot this host the setting seems to revert back to manual and the Hostname changes back to localhost, causing it to not join to the domain and apply a bunch of its host profile settings and various other things. All of my DHCP and DNS settings seem to be fine, and I can't find anything particularly weird with this host profile, so I'm trying to figure out where things are going wrong. Anyone seen this before or have any suggestions about what to check?
|
|
#
?
Nov 21, 2013 16:55
|
|
|
CtrlMagicDel posted:So I'm working on configuring my first new host at work and am running into an issue with the DNS configuration setting in ESXi. The settings on the remaining hosts I have all are set to automatic and seem to resolve to the proper hostname, but whenever I reboot this host the setting seems to revert back to manual and the Hostname changes back to localhost, causing it to not join to the domain and apply a bunch of its host profile settings and various other things. All of my DHCP and DNS settings seem to be fine, and I can't find anything particularly weird with this host profile, so I'm trying to figure out where things are going wrong. Anyone seen this before or have any suggestions about what to check? Wait, what? Are you handing out hostnames from AD over DHCP? What does reverse DNS say for the ESXi host's IP address? What does `vicfg-authconfig --authscheme AD --currentdomain` say? Docjowles posted:SeaMicro. I am pretty sure the product in question is discontinued as I can't even find it on their website, so don't take that as a comment on their current offerings. But still, argh. No sarcasm: Thanks for doing the legwork and not filing a bug upstream so some poor developer can try to track it down. "Vendor X's product has problem Y only in EFI mode when booting from an iSCSI HBA from this vendor" is always awful.
|
|
#
?
Nov 21, 2013 17:11
|
|
|
Wicaeed posted:Battling me every step of the way has been our Network Manager/IT Manager. This guy is somewhat of a 'Security Buff' (he has been tasked with securing our NEW network so that it doesn't get jacked up like our old one did), only his idea and my idea of security don't exactly mesh together. This guy considers almost every aspect of virtualization to be a security vulnerability or flaw. Something about the fact that you have shared storage (ideally) with servers running in (potentially) the same memory sticks (!?!) rubs him wrong. vCenter is a four letter word to this guy. His number one job (it seems to me, probably unfairly) has been to try his damnest to make my teams job (Infrastructure Ops) harder to do. /rant If you've got "attacks" (what does this even mean?) coming in from an old network, you've got much bigger problems than worrying about virtual memory crosstalk exploits. Tell him he's an idiot and is probably the reason you are in such a dismal situation in the first place. Edit: trying to be helpful instead of just snarky... Look at correctly isolating your vlans (don't use host based firewalls or granular ip-to-ip allows if you can avoid it!), maybe have a network sniffer, create and standardize local accounts for each user. Disable common exploitable poo poo like having remote root logins and default apache sites up everywhere. Patch your OS. Now, harden and standardize it. Then, set up tripwire/aide and a log munger and READ THE SECURITY LOGS at least weekly. You know, basic enterprise security. Bhodi fucked around with this message at 17:30 on Nov 21, 2013 |
|
#
?
Nov 21, 2013 17:22
|
|
|
evol262 posted:Wait, what? I have what appear to be standard DHCP reservations and DNS entries for this host (compared to all of the other hosts anyway). Reverse DNS seems to have the correct IP address. I'm not really familiar with ESXCLI and I can't find it installed on any of our servers, the previous guy used a lot of PowerCLI if there is an equivalent command there? The host doesn't join back up with AD after I reboot it until I change the hostname in ESXi and rejoin it in Virtual Center if that was info that command is supposed to yield.
|
|
#
?
Nov 21, 2013 19:15
|
|
|
CtrlMagicDel posted:I have what appear to be standard DHCP reservations and DNS entries for this host (compared to all of the other hosts anyway). Reverse DNS seems to have the correct IP address. `nslookup ip.address.of.esxi` and it should return the hostname. This is how the host itself should be discovering whether or not it has a hostname it should be taking from DNS -- it resolves whatever name maps to that IP (canonically; not every operating system works this way, but RDNS is frequently broken and the cause of a lot of pain). CtrlMagicDel posted:I'm not really familiar with ESXCLI and I can't find it installed on any of our servers, the previous guy used a lot of PowerCLI if there is an equivalent command there? The host doesn't join back up with AD after I reboot it until I change the hostname in ESXi and rejoin it in Virtual Center if that was info that command is supposed to yield. It's installed on the ESXi server. Log into it, whether over some kind of OOB management, SSH, or a physical console. It'll tell you whether or not the ESXi server thinks it's joined to a domain or not. You're going to need to get onto the actual ESXi servers on a CLI to check the hostname and other info if you want to get anywhere other than guessing at DNS. What do you mean whan you say "rejoin it"? Does it create a new computer object? Does the SID change? Does it just update the hostname? What changes in AD when you rejoin it?
|
|
#
?
Nov 21, 2013 19:35
|
|
|
evol262 posted:Not how reverse DNS works. Oh, ok. nslookup resolves correctly. That is what the Reverse Lookup Zone entry in Windows DNS Manager references, correct? evol262 posted:It's installed on the ESXi server. Log into it, whether over some kind of OOB management, SSH, or a physical console. It'll tell you whether or not the ESXi server thinks it's joined to a domain or not. I can't seem to get the syntax right to get this command to work on the ESXi server via Putty, is it under some specific namespace? I need to preface it with esxcli (namespace) command, right? evol262 posted:What do you mean whan you say "rejoin it"? Does it create a new computer object? Does the SID change? Does it just update the hostname? What changes in AD when you rejoin it? By rejoining it I mean in vCenter via "Authentication Services -> Properties -> Change to Active Directory". That at least gets the "host is not joined in any domain currently" message to go away under profile compliance. The SID does not change.
|
|
#
?
Nov 21, 2013 20:33
|
|
|
CtrlMagicDel posted:Oh, ok. nslookup resolves correctly. That is what the Reverse Lookup Zone entry in Windows DNS Manager references, correct? CtrlMagicDel posted:I can't seem to get the syntax right to get this command to work on the ESXi server via Putty, is it under some specific namespace? I need to preface it with esxcli (namespace) command, right? CtrlMagicDel posted:By rejoining it I mean in vCenter via "Authentication Services -> Properties -> Change to Active Directory". That at least gets the "host is not joined in any domain currently" message to go away under profile compliance. The SID does not change. evol262 fucked around with this message at 21:40 on Nov 21, 2013 |
|
#
?
Nov 21, 2013 20:57
|
|
|
I got an email from a coworker today complaining that one of our services was running a bit slow, so I logged into our vCenter server and saw this Just for giggles, this host has only 16 logical processors (2x Quad Core E5620):     According to my coworker, the 8vCPU configuration of some of the VMs was recommended by one of our Sr. SysAdmins whom I would rate as "brilliant". CPU utilization on the host has been running steady around 80%-90% for at LEAST a few months now and it also appears as if the workload itself is actually taking advantage of multi-threading, at least according to the VM CPU usage chart (each vCPU is showing around 30% utilized). From my (non-VCP) understanding of CPU scheduling in VMware, the %Ready time is time that a vCPU spent waiting to execute a command, but was unable to because of CPU over provisioning. It also appears as if the %CSTP column would indicate that there is some vast over-provisioning going on here as well. Does a CPUs overall usage have any impact on the %Ready time, or is that specifically a performance metric that is impacted by the number of vCPUs on a host? Also according to my coworkers, people (testers) have been complaining of slow performance of these machines. I WONDER WHY
|
|
#
?
Nov 21, 2013 21:36
|
|
|
 
|
|
#
?
Nov 21, 2013 21:43
|
|
|
Wicaeed posted:the 8vCPU configuration of some of the VMs was recommended by one of our Sr. SysAdmins whom I would rate as "brilliant".
|
|
#
?
Nov 21, 2013 21:56
|
|
|
I've read enough to know that this is bad, but I'm not 100% sure exactly why. Am I close when I read this as there are 24 VM's, 16 vCPU's available. If you had provisioned 1 to each machine, it'd be splitting 16 vCPU's 24 ways, a bit of a stretch (or not?). Instead, it's set up as needing 109v vCPU's, so those 16 vCPUs are way overburdened.
|
|
#
?
Nov 21, 2013 22:02
|
|
|
Dr. Arbitrary posted:I've read enough to know that this is bad, but I'm not 100% sure exactly why. Because every one of those vCPUs needs hardware interrupts, and while relaxed co-scheduling somewhat mitigates this problem, you can still end up with the entire VM paused while one of its CPUs contends for interrupt time with some other 8 vCPU monster. You should read this.
|
|
#
?
Nov 21, 2013 22:14
|
|
|
Dr. Arbitrary posted:I've read enough to know that this is bad, but I'm not 100% sure exactly why. From my (albiet) limited understanding, it works somewhat like this: IF the 24 VMs were all using roughly the same amount of processing power (each vCPU using 2GHz), and only had 1 vCPU assigned to them, then yes, there would be a serious contention of resources on this host, simply because there would not be enough available processing speed to service all those vCPUs. It's not actually something I've seen yet, but I think that the alternative case (many vCPUs being relatively underutilized) is a little worse, simply because you can think that "Hey my CPU usage is low, why are my VMs underperforming so much?! I know, I'll just throw more vCPU's at it to alleviate the problem!" and thereby making the problem a whole lot worse. If a resource is unavailable to the vCPU when it needs it, this shows up in the %RDY column. The %RDY stat is really a measurement for the time delay between a hardware interrupt request and the time it was actually ready for execution by the physical processor, from what I understand. Edit: Just curious, but is it possible that by manually controlling the CPU affinity for each of the overloaded VMs to distrubute the load across all available physical processing cores, it would be possible to reduce the %RDY time? double edit: haha nope, apparently you can't force a VMs workload to take place on two logical processors when it has 8 vCPUs. Wicaeed fucked around with this message at 22:39 on Nov 21, 2013 |
|
#
?
Nov 21, 2013 22:30
|
|
|
I'm kinda 'fighting' with a software vendor right now (Strongview) about the specifications of a Strongmail virtual server.quote:Our recommended requirements are 4-8 VCPUs, and 8-24GB of RAM. That’s minimum and recommended values on the low and high side. Our old server is an 8 year old dell 1950, dual core dual CPU. I've never seen it go above 10% cpu utilization. Unless the new software requirements have gone through the roof, I really don't think we need 4vCPUs (let alone 8) for a mailing server. Our virtual hosts have dual E5-2660 Xeons (Sandy Bridge 8 core Xeons.) I think 1vCPU wil be plenty of power for our mailings, MAYBE 2 if it's sufficiently multithreaded. bull3964 fucked around with this message at 22:45 on Nov 21, 2013 |
|
#
?
Nov 21, 2013 22:42
|
|
|
Haha it gets better, I got the following from my coworker. He wants me to change the following settings "To speed up the rate of memory exchange on host server": Mem.ShareScanGHZ = 32 Mem.ShareScanTIme = 600 Mem.ShareRateMax = 32768 Mem.ShareVmkThresholdPct = 100 He's totally convinced that this problem isn't caused by CPU's being over-provisioned. It's gonna be a long day.
|
|
#
?
Nov 21, 2013 22:50
|
|
|
evol262 posted:I have no idea what Windows' tools reference, honestly. Not taking anything judgmentally at all, I appreciate any help. I'm no master but I can get around in Unix, I technically owned a couple of Solaris servers in my last position but didn't have to do much with them honestly. Here's what I get when I try that command after SSH'ing to the host: ~ # vicfg-authconfig --authscheme AD --currentdomain -sh: vicfg-authconfig: not found I'm able to see that the SID on the compute object in AD has not changed, though whenever I rejoin the object's modified date updates. Beyond that my Active Directory troubleshooting skills are pretty limited, though I got my home lab domain controller up yesterday! So that's hopefully a start down the right path.
|
|
#
?
Nov 21, 2013 23:08
|
|
|
Wicaeed posted:Haha it gets better, I got the following from my coworker. He wants me to change the following settings "To speed up the rate of memory exchange on host server": How much of a cowboy are you? Cut it down to 2 or 4 CPUs, don't tell him, and see what happens. DERP: I meant DON'T tell him. Just do it and see if it fixes the problem. hackedaccount fucked around with this message at 23:19 on Nov 21, 2013 |
|
#
?
Nov 21, 2013 23:14
|
|
|
CtrlMagicDel posted:Not taking anything judgmentally at all, I appreciate any help. I'm no master but I can get around in Unix, I technically owned a couple of Solaris servers in my last position but didn't have to do much with them honestly. This is why I asked, I guess. `which vicfg-authcfg` or `vicfg<TAB><TAB>` to complete it if which doesn't find anything. If you have access to the AD tools themselves, it's easy to look, otherwise, you'll be looking for a *nix box and something like: ldapsearch -v -Y DIGEST-MD5 -h your.domain -U yourusername -R your.domain -b "DC=your,DC=domain" "(sAMAccountName=esxiserver$)" (the $ at the end says "this is a computer object") But LDAP gets very messy very fast if you're not familiar with it. Alternatively, you could put the switchport it's on into mirroring or monitoring or tcpdump from an AD server (but this is risky, since nslookup your.domain will return every AD server in the org and you have no idea which one it'll pick, but you're probably not an AD admin and you can't go look at the real logs), but a packet capture would let you interrogate what's happening on the wire. What should be happening is the server coming up with the right hostname, telling AD "here's my IP and my kerberos principal -- update DNS and rDNS to match", and going from there. What's going wrong is probably somewhere in DNS, but it's very hard to say without checking on a server that may have VMs on it. If you set the hostname and reboot it without joining to AD, does it come back up with the right hostname? Is it breaking somewhere after it joins, or is rebooting enough? Is it a static IP or DHCP reservation? Do you rely on DHCP to set the hostname?
|
|
#
?
Nov 22, 2013 00:02
|
|
|
bull3964 posted:I'm kinda 'fighting' with a software vendor right now (Strongview) about the specifications of a Strongmail virtual server.
|
|
#
?
Nov 22, 2013 00:25
|
|
|
Yeah we did the same with an ENTERPRISE APPLICATION. The thing that newer IT guys need to understand is the vendors have no skin in the game - they aren't the ones paying for the hardware so they can set whatever insane specs they want and it costs them nothing.
|
|
#
?
Nov 22, 2013 00:41
|
|
|
adorai posted:Why bother fighting. Give them what they want for the deployment, then dial it down to 1vCPU and 2GB of RAM afterward. Every time a vendor makes those kinds of requests I start by saying no. Then when they press, I relent, and then remove the resources later. It's always fine. That's what I'm planning on doing. The only issue is, they are going to be providing ongoing support so I'm sure I'll hear about it later "oh you knocked down the resources, that's not going to work."
|
|
#
?
Nov 22, 2013 01:28
|
|
|
bull3964 posted:That's what I'm planning on doing. The only issue is, they are going to be providing ongoing support so I'm sure I'll hear about it later "oh you knocked down the resources, that's not going to work." This is the most annoying thing. At a past job we had to keep a couple paid RHEL subscriptions just because a vendor outright refused to so much as pick up the phone for support if you were on CentOS. I'm pretty sure the reason your shithouse Java app crashes twice a day isn't due to "s/Red Hat/CentOS/", Motorola 
|
|
#
?
Nov 22, 2013 01:42
|
|
|
Docjowles posted:This is the most annoying thing. At a past job we had to keep a couple paid RHEL subscriptions just because a vendor outright refused to so much as pick up the phone for support if you were on CentOS. I'm pretty sure the reason your shithouse Java app crashes twice a day isn't due to "s/Red Hat/CentOS/", Motorola This is always my favorite thing.  cat /etc/redhat-release CentOS 6.3 (Final) 
|
|
#
?
Nov 22, 2013 17:01
|
|
|
Alright, finally got to the bottom of the problem. The issue had to do with enabling the coredump partition in the host profile before successfully booting up with the rest of the profile (which I guess creates all the proper partitions on the local disk? I'm still unclear on how exactly this part works). We disabled the coredump partition, successfully applied the host profile, joined to the domain, then reenabled the partition and were able to reboot successfully into a new healthy looking ESXi host. Now working on cleaning up my documentation so hopefully I can run through this easier next time and aren't chasing DNS ghosts.
|
|
#
?
Nov 22, 2013 22:26
|
|
|
Anyone else here force small page tables for VDI? It has a slighter higher CPU overhead but I am seeing 120-175% TPS improvements. I realize ESXi 5.x has extended page tabling on by default, which seems to have some improvements for large memory apps. Just wondering what some rebuttles are to not use it in a VDI where CPU is not overly constrained, as I got into a somewhat lengthily discussion with an engineer earlier today Dilbert As FUCK fucked around with this message at 00:38 on Nov 23, 2013 |
|
#
?
Nov 23, 2013 00:35
|
|
|
Dilbert As gently caress posted:Anyone else here force small page tables for VDI? Keep in mind that EPT (or AMD's RVI) doesn't necessarily mean you're using large pages; you can actually use it with small pages as well. What EPT does give you is the ability to have guest memory managed in hardware as opposed to via software. If a little extra latency doesn't hurt your apps then it may be worth forcing small pages to get higher consolidation ratios. http://www.vmware.com/pdf/Perf_ESX_Intel-EPT-eval.pdf As a side note when the ESXi host becomes memory constrained it should start breaking large pages up into small pages by default. EPT is REALLY handy for Java apps and basically broke down the last barrier to get some of my customers to adopt VMware. You almost always want EPT/RVI enabled.
|
|
#
?
Nov 23, 2013 05:03
|
|
|
1000101 posted:http://www.vmware.com/pdf/Perf_ESX_Intel-EPT-eval.pdf' quote:What EPT does give you is the ability to have guest memory managed in hardware as opposed to via software. If a little extra latency doesn't hurt your apps then it may be worth forcing small pages to get higher consolidation ratios. That is the basis of what I have been talking about to a few people I work/teach with. Good to know I have been thinking about LPT vs. SPT correctly. Damnit I kinda I wish I could work with you. Dilbert As FUCK fucked around with this message at 05:28 on Nov 23, 2013 |
|
#
?
Nov 23, 2013 05:21
|
|
|
Does anyone have any idea how I could detect a bridged network adapter as opposed to NAT? Have a virtual appliance that I distribute throughout the company, if someone switches their adapter from NAT to Bridged it screws with everyone elses VM. IT is not the least bit interested in using a global query blacklist so we're left to figure it out on our own. I'm using bginfo4x to render the wallpaper so my current plan is to write up a script that would display an alert on there.
|
|
#
?
Nov 23, 2013 21:51
|
|
|
Ashex posted:Does anyone have any idea how I could detect a bridged network adapter as opposed to NAT? Have a virtual appliance that I distribute throughout the company, if someone switches their adapter from NAT to Bridged it screws with everyone elses VM.
|
|
#
?
Nov 23, 2013 22:19
|
|
|
Sounds like the MAC address of the VM is the same on each clone. Can you not fix that instead of working around it?
|
|
#
?
Nov 24, 2013 01:58
|
|
|
I know windows 2008 r2 is now supported for linked clones, but outside of using vButt anyone here dish out Linux linked clones? Specifically RHEL5/Centos/SELinx? I know it will "fail", but the point is to spin up roughly ~200 Linux VM's for a semester, then tear down and recompose. I'm not a huge fan of shoehorning it into View; but vButt ain't going to happen. Can Orchestrator do this? I'll admit I don't into it as much as I probably should.
|
|
#
?
Nov 28, 2013 20:58
|
|
|
Dilbert As gently caress posted:but vButt ain't going to happen. Might want to turn off cloud 2 butt before posting/
|
|
#
?
Nov 28, 2013 21:19
|
|
|
|
| # ? May 8, 2024 00:47 |
|
|
Dilbert As gently caress posted:I know windows 2008 r2 is now supported for linked clones, but outside of using vButt anyone here dish out Linux linked clones? Specifically RHEL5/Centos/SELinx? I know it will "fail", but the point is to spin up roughly ~200 Linux VM's for a semester, then tear down and recompose. I'm not a huge fan of shoehorning it into View; but vButt ain't going to happen. There's no reason it should be a problem. Take HWADDR out of /etc/sysconfig/network-scripts/ifcfg-eth0 (DEVICE=eth0 should be enough to get it to come up no matter what the MAC is), change the hostname to something generic and automate scripting it (or assign hostnames via DHCP, grab the MAC address from PowerCLI, and dump it into dhcpd.conf), and go. You shouldn't have other problems other than AD registration, but that's a whole mess you'd have to deal with on Linux no matter what. You can certainly still put a kerberos key on which has permissions to create computer objects and have a one-off script in /etc/rc.local which registers to AD then takes itself out of rc.local I don't even want to ask why you want RHEL5 instead of 6, but the steps are basically the same.
|
|
#
?
Nov 28, 2013 22:27
|
|