Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
Dr. Arbitrary
Mar 15, 2006

Bleak Gremlin
Please start testing zero day fixes at least a week in advance.

Adbot
ADBOT LOVES YOU

Docjowles
Apr 9, 2009

Saukkis posted:

Even more annoying are the security updates. A new critical kernel bug is published, Ubuntu releases the update in couple hours and Red Hat is right behind in about a week. Is there a channel to get the untested RPMs? I'd rather take my chances with an un-QAd kernel update that shut down a general use shell server for several days.

What environment are you running in that an unpatched local kernel exploit requires you to completely shut down your servers until a fix is available? I don't mean that in a mocking way, just genuinely curious. Is this like a shared hosting business where you offer shell access to random users who can't be trusted to not be assholes? Or worse...students? :)

Dr. Arbitrary posted:

Please start testing zero day fixes at least a week in advance.

Symantec, now with -7 Day Protection :smuggo:

astral
Apr 26, 2004

when the exploits are -7days every week-later patch is a 0day patch!

Saukkis
May 16, 2003

Unless I'm on the inside curve pointing straight at oncoming traffic the high beams stay on and I laugh at your puny protest flashes.
I am Most Important Man. Most Important Man in the World.

Docjowles posted:

What environment are you running in that an unpatched local kernel exploit requires you to completely shut down your servers until a fix is available? I don't mean that in a mocking way, just genuinely curious. Is this like a shared hosting business where you offer shell access to random users who can't be trusted to not be assholes? Or worse...students? :)

Students. And the worst of all, professors. Hundreds of people with access to those servers.

nem
Jan 4, 2003

panel.dev
apnscp: cPanel evolved

jre posted:

This is a reason never to use Ubuntu, not a reason to avoid redhat

Have there been any prevailing instances where Canonical's rapid release schedule caused more problems than addressed?

evol262
Nov 30, 2010
#!/usr/bin/perl

nem posted:

Have there been any prevailing instances where Canonical's rapid release schedule caused more problems than addressed?

There have been a ton of instances where the complete lack of QA and "let's just use the Debian patch verbatim" have caused major breakage.

This allows for a "rapid release cycle", but when there's a 0-day kernel CVE and they release same day, you have to ask "was there any regression testing on this at all?"

And the answer to that question is no. It's a matter of time until they break a major component instead of minor ones.

xzzy
Mar 5, 2009

Ubuntu's totally rad on a desktop where a blind update and reboot breaking things doesn't hurt too bad, but in the server world where you got thousands of machines at risk of not doing their job, you need more reliability.

Unless you really, really enjoy updating the dozens of tickets that roll in. And have bosses that don't mind the company not doing any work.

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

I have no idea how distributions are put together, but as a software developer it seems crazy to me that they can't whip the patch in and run a many-hours long test suite and then go "ok tis good, roll it out".

RFC2324
Jun 7, 2012

http 418

Thermopyle posted:

I have no idea how distributions are put together, but as a software developer it seems crazy to me that they can't whip the patch in and run a many-hours long test suite and then go "ok tis good, roll it out".

for a stable distro it needs to be tested against anything else in the system that might be affected. that's quite a few things for a kernel patch

jre
Sep 2, 2011

To the cloud ?



Thermopyle posted:

I have no idea how distributions are put together, but as a software developer it seems crazy to me that they can't whip the patch in and run a many-hours long test suite and then go "ok tis good, roll it out".

As a software developer it seems crazy to me that you think you could test a kernel patch on a significant set of hardware combinations in hours. Also if the patch is a security update for a subtle bug, doing a rapid bodge has a decent chance of introducing a worse problem.

nem posted:

Have there been any prevailing instances where Canonical's rapid release schedule caused more problems than addressed?

If you are happy installing obviously untested changes to your production server environment then go hog wild, I feel sorry for anyone using your service though.

xzzy
Mar 5, 2009

Try 'rpm --erase python' on a redhat system sometime and look at the list of dependencies it spews out. Good loving luck testing all that in a couple hours. :shepface:

nem
Jan 4, 2003

panel.dev
apnscp: cPanel evolved

jre posted:

If you are happy installing obviously untested changes to your production server environment then go hog wild, I feel sorry for anyone using your service though.

CentOS crew :whatup:

It's to assess new players in the hosting panel market now, both RunCloud and ServerPilot rely on Ubuntu LTS. This feedback strengthens selling a solution built on CentOS/RHEL.

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

jre posted:

As a software developer it seems crazy to me that you think you could test a kernel patch on a significant set of hardware combinations in hours.

But I don't think that?

It sounded to me like people were saying there was no testing being done.

RFC2324
Jun 7, 2012

http 418

Thermopyle posted:

But I don't think that?

It sounded to me like people were saying there was no testing being done.

That would be the case on Ubuntu, hence the very short release period. No, or so little that it may as well be no

evol262
Nov 30, 2010
#!/usr/bin/perl

nem posted:

CentOS crew :whatup:

It's to assess new players in the hosting panel market now, both RunCloud and ServerPilot rely on Ubuntu LTS. This feedback strengthens selling a solution built on CentOS/RHEL.

Nothing wrong with LTS.

But a kernel build on RHEL runs a full regression against almost 1000 different pieces of hardware.

Take every OEM multiplied by local+FC+iscsi multiplied by every model the OEMs ship. Once all of those pass, then manual QA happens. Then the release.

But "Debian posted a patch so we crossed our fingers and shipped it -- good luck, customers" isn't the same level of safety. Different strokes. But this is why EL distros take 4-5 days after a zero day, not 4-5 hours.

nem
Jan 4, 2003

panel.dev
apnscp: cPanel evolved

evol262 posted:

But a kernel build on RHEL runs a full regression against almost 1000 different pieces of hardware.

If I understand correctly, wouldn't this be less of a concern on guest machines with virtio devices, since that serves as an abstraction layer dependent upon the host kernel? Testing a kernel against virtio devices would be the same irrespective hardware provided the host kernel has no regressions.

Edit \/\/\/: thanks for clearing that up!

nem fucked around with this message at 05:44 on Oct 15, 2017

evol262
Nov 30, 2010
#!/usr/bin/perl
No, since virtio still uses the entire storage subsystem, and the network subsystem, and the scheduler, etc. Plus all the variations in possible libvirt CPU models. And testing host-passthrough.

But the point of regression testing is to check whether the patch failed anywhere, from scheduler deadlocks to a failure in an ancient cciss controller. If any test fails, the entire suite fails, and any exceptions are known for documentation or additional patches.

E: I work on rhev and libvirt. There are a huge number of kernel bugs which are only reproducible on specific hardware, which is why most vendors also have a lab where you can go request a DL380 G6 or whatever.

Even a 0-day in USB mass storage can't be tested in VMs, since some flash drives present as multipath. Testing on real hardware is necessary. We do include a large suite of virtualized sanity tests, but passing that is not sufficient for a regression pass. It's only part.

Canonical doesn't test at all, including on virt. They rely on upstream to have tested, and assume a successful kernel build for whatever Arch is good enough

evol262 fucked around with this message at 20:28 on Oct 14, 2017

Saukkis
May 16, 2003

Unless I'm on the inside curve pointing straight at oncoming traffic the high beams stay on and I laugh at your puny protest flashes.
I am Most Important Man. Most Important Man in the World.
So you people are of the opinion, that it's better to shut down a service for several days than using a non-QA'd kernel that would most likely work just fine without any issues. Unless you have better information about how many attempts Red Hat usually requires before they manage to produce a kernel that doesn't fail catastrophically on a common hardware?

And I'm not planning to stuff it in hundreds of servers, I just want it for the couple that have untrusted users. Of course it shouldn't be available through standard channels, just give me a maze of half a dozen links and with as many warnings and disclaimers as you want, as long as there are beta RPM downloads at the end. Really, how much worse it can be than the fully-QA'd kernels which fail to boot every now and then anyway.

And on the iptables issue, how long can it take to QA a patch that teaches iptables-restore how to use -w? In the mean time whenever we start a RHEL 7.4 server there is about 1 in 3 chance that it won't have a firewall.

Saukkis fucked around with this message at 19:47 on Oct 16, 2017

Vulture Culture
Jul 14, 2003

I was never enjoying it. I only eat it for the nutrients.

Saukkis posted:

So you people are of the opinion, that it's better to shut down a service for several days than using a non-QA'd kernel that would most likely work just fine without any issues. Unless you have better information about how many attempts Red Hat usually requires before they manage to produce a kernel that doesn't fail catastrophically on a common hardware?

And I'm not planning to stuff it in hundreds of servers, I just want it for the couple that have untrusted users. Of course it shouldn't be available through standard channels, just give me a maze of half a dozen links and with as many warnings and disclaimers as you want, as long as there are beta RPM downloads at the end. Really, how much worse it can be than the fully-QA'd kernels which fail to boot every now and then anyway.

And on the iptables issue, how long can it take to QA a patch that teaches iptables-restore how to use -w? In the mean time whenever we start a RHEL 7.4 server there is about 1 in 3 chance that it won't have a firewall.
You might want to consider taking this up with your TAM rather than unloading at strangers on the Internet. This is a pretty standard way of getting access to hotfixes ahead of GA.

anthonypants
May 6, 2007

by Nyc_Tattoo
Dinosaur Gum

Saukkis posted:

So you people are of the opinion, that it's better to shut down a service for several days than using a non-QA'd kernel that would most likely work just fine without any issues. Unless you have better information about how many attempts Red Hat usually requires before they manage to produce a kernel that doesn't fail catastrophically on a common hardware?

And I'm not planning to stuff it in hundreds of servers, I just want it for the couple that have untrusted users. Of course it shouldn't be available through standard channels, just give me a maze of half a dozen links and with as many warnings and disclaimers as you want, as long as there are beta RPM downloads at the end. Really, how much worse it can be than the fully-QA'd kernels which fail to boot every now and then anyway.

And on the iptables issue, how long can it take to QA a patch that teaches iptables-restore how to use -w? In the mean time whenever we start a RHEL 7.4 server there is about 1 in 3 chance that it won't have a firewall.
It seems like wanting kernel updates, no matter the cost, is how you got into the issue with iptables in the first place.

evol262
Nov 30, 2010
#!/usr/bin/perl

Saukkis posted:

So you people are of the opinion, that it's better to shut down a service for several days than using a non-QA'd kernel that would most likely work just fine without any issues. Unless you have better information about how many attempts Red Hat usually requires before they manage to produce a kernel that doesn't fail catastrophically on a common hardware?
I don't think you understand what a regression test is, and maybe not a test harness.

If you feel like you need to shut down a service, you've done something wrong. Banking kept operating after multiple public CVEs. Even if it took some banks a couple of days to apply patches for Heartbleed (despite the it being a coordinated release with responsible disclosure, meaning packages for RHEL were available within minutes of every other distro), they didn't "shut down" until they could schedule a maintenance window.

And it's not about taking multiple attempts to "produce a kernel that doesn't fail catastrophically". The RHEL kernel does through a full regression test on PPC, s390, x86_64, and sometimes aarch64. A regression test here meaning "every kernel runs through a couple thousand tests of various subsystems on every possible piece of hardware", and some of those tests take a while. Kernels rarely fail, because they also go through the testing harness as part of development. But it still takes a while to say "ensure that bug#12345 didn't regress"

Note that for critical CVEs, we get them out as fast as possible. This means war rooms, maintainers working during PTO, etc.

Saukkis posted:

And I'm not planning to stuff it in hundreds of servers, I just want it for the couple that have untrusted users. Of course it shouldn't be available through standard channels, just give me a maze of half a dozen links and with as many warnings and disclaimers as you want, as long as there are beta RPM downloads at the end. Really, how much worse it can be than the fully-QA'd kernels which fail to boot every now and then anyway.
If kernels fail to boot, file a support case so the bug can be fixed. If you have done this, get your TAM/CEE rep to escalate it. If you haven't, there's no way the kernel engineering team can possibly be aware of it (or a failure in their testing suite).

If you want an unsupported kernel, which those RPMs surely would be, feel free to use ELrepo or apply a patch to the last released SRPM yourself. The patches are public. The SRPMs are public. This isn't hard to do if it's really urgent for you; so urgent that you feel the need to rail about it on an internet comedy forum.

Saukkis posted:

And on the iptables issue, how long can it take to QA a patch that teaches iptables-restore how to use -w? In the mean time whenever we start a RHEL 7.4 server there is about 1 in 3 chance that it won't have a firewall.
I work in Red Hat engineering. I am not omniscient. I have no idea what bug you're talking about, and I have no information about it.

It's far more likely that whatever bug you're talking about isn't a Z-stream, which means it will stay "ON_QA" until whenever RHEL 7.5 comes out. Want it to be a Z-stream? Talk to your TAM and express that this is important. The fact that it's presumably targeted to a Y-stream with or without a TAM presumably means that it's very hard to reproduce (much more than 33%), has few users (like the rancid bug from the last page, not iptables/firewalld) or has an easy workaround. Have you opened a support case? Open a support case. Nobody can help you here.

I'm not saying that there aren't problems or bugs with RHEL. I'm saying that there's no way they can be fixed unless they're reported. And that if there is a bug you want to know the schedule for, you can talk to CEE or your TAM.

I'd also suggest that, given the size of the RHEL install base and the fact that these bugs are presumably not urgent enough for a Z-stream or asynchronous release, that there's a problem with your environment or your kickstart somewhere if there's a 1/3 chance a system won't have a firewall up.

anthonypants
May 6, 2007

by Nyc_Tattoo
Dinosaur Gum

evol262 posted:

I work in Red Hat engineering. I am not omniscient. I have no idea what bug you're talking about, and I have no information about it.

It's far more likely that whatever bug you're talking about isn't a Z-stream, which means it will stay "ON_QA" until whenever RHEL 7.5 comes out. Want it to be a Z-stream? Talk to your TAM and express that this is important. The fact that it's presumably targeted to a Y-stream with or without a TAM presumably means that it's very hard to reproduce (much more than 33%), has few users (like the rancid bug from the last page, not iptables/firewalld) or has an easy workaround. Have you opened a support case? Open a support case. Nobody can help you here.

I'm not saying that there aren't problems or bugs with RHEL. I'm saying that there's no way they can be fixed unless they're reported. And that if there is a bug you want to know the schedule for, you can talk to CEE or your TAM.

I'd also suggest that, given the size of the RHEL install base and the fact that these bugs are presumably not urgent enough for a Z-stream or asynchronous release, that there's a problem with your environment or your kickstart somewhere if there's a 1/3 chance a system won't have a firewall up.
I think it's this one, but there's a bunch of associated bugs in RedHat's bugzilla, in various stages of ON_QA or VERIFIED or CLOSED ERRATA. One of those related bugs is from a year ago and is for 7.3. Another bug has a comment that suggests that a commenter, "contact our support where you can request the package as a hotfix".

evol262
Nov 30, 2010
#!/usr/bin/perl
That bug is VERIFIED and a z-stream. Which means it'll be out with the next RHEL z-stream. Which means batch updates.

These are on a regular cadence. It is not a mystery to guess when it's going to be (soon)

For the 7.3 bug, you can tell from the doc text being upgraded that it'll be available very soon.

Yes, there are hotfixes available for both of these.

anthonypants
May 6, 2007

by Nyc_Tattoo
Dinosaur Gum

evol262 posted:

That bug is VERIFIED and a z-stream. Which means it'll be out with the next RHEL z-stream. Which means batch updates.

These are on a regular cadence. It is not a mystery to guess when it's going to be (soon)

For the 7.3 bug, you can tell from the doc text being upgraded that it'll be available very soon.

Yes, there are hotfixes available for both of these.
Yeah, that errata advisory for OpenStack is from a few hours ago.

mike12345
Jul 14, 2008

"Whether the Earth was created in 7 days, or 7 actual eras, I'm not sure we'll ever be able to answer that. It's one of the great mysteries."





I'm thinking of setting up a new system next year, but I'm a bit skeptical about dual boot Linux - Windows 10. Is there (still) a risk that Windows decides to erase the Linux disk because it thinks it's faulty, or was that FUD?

taqueso
Mar 8, 2004


:911:
:wookie: :thermidor: :wookie:
:dehumanize:

:pirate::hf::tinfoil:

mike12345 posted:

I'm thinking of setting up a new system next year, but I'm a bit skeptical about dual boot Linux - Windows 10. Is there (still) a risk that Windows decides to erase the Linux disk because it thinks it's faulty, or was that FUD?

I've been dual booting for years and that hasn't ever happened to me. If I'm being cautious I will disconnect the other OS's disk during install, but that is mostly so I don't fat finger disk selection.

IIRC, windows can sometimes overwrite the MBR if you are dual booting from different partitions on the same disk, but that might just be older versions of Windows. It is pretty easy to fix and won't hurt your data if it does happen.

Volguus
Mar 3, 2009

mike12345 posted:

I'm thinking of setting up a new system next year, but I'm a bit skeptical about dual boot Linux - Windows 10. Is there (still) a risk that Windows decides to erase the Linux disk because it thinks it's faulty, or was that FUD?

Well, that's a new one, never heard of it. I've always had a windows partition on my home computer and that goes back to ... 1998 or so. Unless I had way too many beers and selected the wrong thing to install an os on (being it windows or linux or whatever) none of them ever gave me any trouble. Sure, in the old days a windows reinstallation would overwrite the MBR, but that wasn't a problem to fix and reinstall lilo or grub.

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

mike12345 posted:

I'm thinking of setting up a new system next year, but I'm a bit skeptical about dual boot Linux - Windows 10. Is there (still) a risk that Windows decides to erase the Linux disk because it thinks it's faulty, or was that FUD?

I always feel the need to point out that you may be happier not dual booting and instead running one of the operating systems in a virtual machine.

It works really well.

taqueso
Mar 8, 2004


:911:
:wookie: :thermidor: :wookie:
:dehumanize:

:pirate::hf::tinfoil:

It's even possible to run linux and passthrough the video card to a gaming Windows VM if gaming is the reason you want to dual boot.

Double Punctuation
Dec 30, 2009

Ships were made for sinking;
Whiskey made for drinking;
If we were made of cellophane
We'd all get stinking drunk much faster!
Windows Subsystem for Linux just got released, so you don’t even need a VM. You would have to start any daemons from Task Scheduler and supply your own display server, but none of that is hard.

xzzy
Mar 5, 2009

I tinkered around with WSL last night, didn't get as much time as I wanted because it took longer than I expected to get the fall update installed but after that was done it took maybe 10 minutes to get an X server running and I was off tweaking my terminator config.

If it means I never again need to be tempted to buy another macbook just to get a comfortable work environment on a laptop, I'm sold.

Even better if I can do it without cygwin.

mike12345
Jul 14, 2008

"Whether the Earth was created in 7 days, or 7 actual eras, I'm not sure we'll ever be able to answer that. It's one of the great mysteries."





Yeah I'm running a Linux VM on Windows right now, but it's more about switching to Linux as a host. I just need Windows for the occasional game, and don't mind buying a second sdd. I guess that pass-through pci is an option, too, but rebooting every now and then is no biggie. I was just worried that Windows 10 is more aggressive when it comes to non-ntfs disks.

gourdcaptain
Nov 16, 2012

Does anyone know if there's a good way to set up udisks or something to change the default mount options of all NTFS disks? By default NTFS-3G allows the creation of files with names technically legal under NTFS (such as question marks) that are illegal under windows (and produce hilarious "this file does not exist" messages when interacted with.) This has tripped me up a few times recently, and you can apparently use the windows_names mount flag to change the behavior, but I would need a way to do it automatically for it to be useful.

gourdcaptain fucked around with this message at 18:21 on Oct 20, 2017

thebigcow
Jan 3, 2001

Bully!
If I install Fedora without a separate /home will I be kicking myself when it's time to upgrade to the next version?

anthonypants
May 6, 2007

by Nyc_Tattoo
Dinosaur Gum

thebigcow posted:

If I install Fedora without a separate /home will I be kicking myself when it's time to upgrade to the next version?
I upgraded a Fedora 24 install on an RPi3 to Fedora 26, it doesn't have a separate /home, and I didn't lose anything. YMMV, of course, but imho dnf system-upgrade is really cool.

Volguus
Mar 3, 2009

thebigcow posted:

If I install Fedora without a separate /home will I be kicking myself when it's time to upgrade to the next version?

No, but it can help if you want for whatever reason to re-install or install another distro (just choose the same /home). That's the only thing. Of course, you can backup your /home prior to installing/reinstalling things but that's :effort:

post hole digger
Mar 21, 2011

I'm not sure this is the right place to ask this but I'm working with ansible and trying to learn a little about facts, and I'm wondering why this:

code:
ansible myserver -m setup -a "filter=ansible_default_ipv4.alias"
myserver.com | SUCCESS => {
    "ansible_facts": {},
    "changed": false
}
doesn't work, but making a playbook with this info:

- debug: msg="{{ ansible_default_ipv4.alias }}"

does work, eg:

code:
TASK [debug] **************************************************************
ok: [myserver.com] => {
    "msg": "eth0"
}
I'm sure there's a simple explanation for this... but what is it?

jre
Sep 2, 2011

To the cloud ?



my bitter bi rival posted:

I'm not sure this is the right place to ask this but I'm working with ansible and trying to learn a little about facts, and I'm wondering why this:

code:
ansible myserver -m setup -a "filter=ansible_default_ipv4.alias"
myserver.com | SUCCESS => {
    "ansible_facts": {},
    "changed": false
}
doesn't work, but making a playbook with this info:

- debug: msg="{{ ansible_default_ipv4.alias }}"

does work, eg:

code:
TASK [debug] **************************************************************
ok: [myserver.com] => {
    "msg": "eth0"
}
I'm sure there's a simple explanation for this... but what is it?

If you take the filter out and list all facts is it there ?

post hole digger
Mar 21, 2011

Yep. I've noticed that running that type of filter, looking for a "nested" (not sure if thats the right word) fact within a fact doesn't seem to work as an ad-hoc command for anything I've tried.

eg this works fine:

code:
$ ansible myserver -m setup -a "filter=ansible_default_ipv4"
myserver.com | SUCCESS => {
    "ansible_facts": {
        "ansible_default_ipv4": {
            "address": "172.31.XX.XXX",
            "alias": "eth0",
            "broadcast": "172.31.31.255",
            "gateway": "172.31.16.1",
            "interface": "eth0",
            "macaddress": "06:d1:6f:e5:a9:a0",
            "mtu": 9001,
            "netmask": "255.255.240.0",
            "network": "172.31.16.0",
            "type": "ether"
        }
    },
    "changed": false
}
but i cant successfully zero in on anything within 'ansible_default_ipv4' when running an ad-hoc command. is the syntax different when working with ad-hoc? a difference in how debug and setup modules process fact filtering?

post hole digger fucked around with this message at 00:11 on Oct 21, 2017

Adbot
ADBOT LOVES YOU

Tigren
Oct 3, 2003

my bitter bi rival posted:

Yep. I've noticed that running that type of filter, looking for a "nested" (not sure if thats the right word) fact within a fact doesn't seem to work as an ad-hoc command for anything I've tried.

eg this works fine:

code:
$ ansible myserver -m setup -a "filter=ansible_default_ipv4"
myserver.com | SUCCESS => {
    "ansible_facts": {
        "ansible_default_ipv4": {
            "address": "172.31.22.125",
            "alias": "eth0",
            "broadcast": "172.31.31.255",
            "gateway": "172.31.16.1",
            "interface": "eth0",
            "macaddress": "06:d1:6f:e5:a9:a0",
            "mtu": 9001,
            "netmask": "255.255.240.0",
            "network": "172.31.16.0",
            "type": "ether"
        }
    },
    "changed": false
}
but i cant successfully zero in on anything within 'ansible_default_ipv4' when running an ad-hoc command.

The filter option filters only the first level subkey below ansible_facts. When you run the playbook version, all of the facts are gathered and then your debug task filters the dictionary.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply