Search Amazon.com:
Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us $3,400 per month for bandwidth bills alone, and since we don't believe in shoving popup ads to our registered users, we try to make the money back through forum registrations.
«684 »
  • Post
  • Reply
Dr. Arbitrary
Mar 15, 2006

You're trying to say that you like DOS better then me, right?



Bleak Gremlin

Please start testing zero day fixes at least a week in advance.

Adbot
ADBOT LOVES YOU

Docjowles
Apr 9, 2009



Saukkis posted:

Even more annoying are the security updates. A new critical kernel bug is published, Ubuntu releases the update in couple hours and Red Hat is right behind in about a week. Is there a channel to get the untested RPMs? I'd rather take my chances with an un-QAd kernel update that shut down a general use shell server for several days.

What environment are you running in that an unpatched local kernel exploit requires you to completely shut down your servers until a fix is available? I don't mean that in a mocking way, just genuinely curious. Is this like a shared hosting business where you offer shell access to random users who can't be trusted to not be assholes? Or worse...students?

Dr. Arbitrary posted:

Please start testing zero day fixes at least a week in advance.

Symantec, now with -7 Day Protection

astral
Apr 26, 2004

Out there.


when the exploits are -7days every week-later patch is a 0day patch!

Saukkis
May 16, 2003



Docjowles posted:

What environment are you running in that an unpatched local kernel exploit requires you to completely shut down your servers until a fix is available? I don't mean that in a mocking way, just genuinely curious. Is this like a shared hosting business where you offer shell access to random users who can't be trusted to not be assholes? Or worse...students?

Students. And the worst of all, professors. Hundreds of people with access to those servers.

nem
Jan 4, 2003

Hostineer: Elevate Your Performance

The hosting provider formerly known as "Apis Networks". No relation to Prince.


jre posted:

This is a reason never to use Ubuntu, not a reason to avoid redhat

Have there been any prevailing instances where Canonical's rapid release schedule caused more problems than addressed?

evol262
Nov 30, 2010
#!/usr/bin/perl

nem posted:

Have there been any prevailing instances where Canonical's rapid release schedule caused more problems than addressed?

There have been a ton of instances where the complete lack of QA and "let's just use the Debian patch verbatim" have caused major breakage.

This allows for a "rapid release cycle", but when there's a 0-day kernel CVE and they release same day, you have to ask "was there any regression testing on this at all?"

And the answer to that question is no. It's a matter of time until they break a major component instead of minor ones.

xzzy
Mar 5, 2009

wakey wakey to
this bowl of tasty


Yams Fan

Ubuntu's totally rad on a desktop where a blind update and reboot breaking things doesn't hurt too bad, but in the server world where you got thousands of machines at risk of not doing their job, you need more reliability.

Unless you really, really enjoy updating the dozens of tickets that roll in. And have bosses that don't mind the company not doing any work.

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell


I have no idea how distributions are put together, but as a software developer it seems crazy to me that they can't whip the patch in and run a many-hours long test suite and then go "ok tis good, roll it out".

RFC2324
Jun 7, 2012



Thermopyle posted:

I have no idea how distributions are put together, but as a software developer it seems crazy to me that they can't whip the patch in and run a many-hours long test suite and then go "ok tis good, roll it out".

for a stable distro it needs to be tested against anything else in the system that might be affected. that's quite a few things for a kernel patch

jre
Sep 2, 2011

To the cloud ?





Thermopyle posted:

I have no idea how distributions are put together, but as a software developer it seems crazy to me that they can't whip the patch in and run a many-hours long test suite and then go "ok tis good, roll it out".

As a software developer it seems crazy to me that you think you could test a kernel patch on a significant set of hardware combinations in hours. Also if the patch is a security update for a subtle bug, doing a rapid bodge has a decent chance of introducing a worse problem.

nem posted:

Have there been any prevailing instances where Canonical's rapid release schedule caused more problems than addressed?

If you are happy installing obviously untested changes to your production server environment then go hog wild, I feel sorry for anyone using your service though.

xzzy
Mar 5, 2009

wakey wakey to
this bowl of tasty


Yams Fan

Try 'rpm --erase python' on a redhat system sometime and look at the list of dependencies it spews out. Good loving luck testing all that in a couple hours.

nem
Jan 4, 2003

Hostineer: Elevate Your Performance

The hosting provider formerly known as "Apis Networks". No relation to Prince.


jre posted:

If you are happy installing obviously untested changes to your production server environment then go hog wild, I feel sorry for anyone using your service though.

CentOS crew

It's to assess new players in the hosting panel market now, both RunCloud and ServerPilot rely on Ubuntu LTS. This feedback strengthens selling a solution built on CentOS/RHEL.

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell


jre posted:

As a software developer it seems crazy to me that you think you could test a kernel patch on a significant set of hardware combinations in hours.

But I don't think that?

It sounded to me like people were saying there was no testing being done.

RFC2324
Jun 7, 2012



Thermopyle posted:

But I don't think that?

It sounded to me like people were saying there was no testing being done.

That would be the case on Ubuntu, hence the very short release period. No, or so little that it may as well be no

evol262
Nov 30, 2010
#!/usr/bin/perl

nem posted:

CentOS crew

It's to assess new players in the hosting panel market now, both RunCloud and ServerPilot rely on Ubuntu LTS. This feedback strengthens selling a solution built on CentOS/RHEL.

Nothing wrong with LTS.

But a kernel build on RHEL runs a full regression against almost 1000 different pieces of hardware.

Take every OEM multiplied by local+FC+iscsi multiplied by every model the OEMs ship. Once all of those pass, then manual QA happens. Then the release.

But "Debian posted a patch so we crossed our fingers and shipped it -- good luck, customers" isn't the same level of safety. Different strokes. But this is why EL distros take 4-5 days after a zero day, not 4-5 hours.

nem
Jan 4, 2003

Hostineer: Elevate Your Performance

The hosting provider formerly known as "Apis Networks". No relation to Prince.


evol262 posted:

But a kernel build on RHEL runs a full regression against almost 1000 different pieces of hardware.

If I understand correctly, wouldn't this be less of a concern on guest machines with virtio devices, since that serves as an abstraction layer dependent upon the host kernel? Testing a kernel against virtio devices would be the same irrespective hardware provided the host kernel has no regressions.

Edit \/\/\/: thanks for clearing that up!

nem fucked around with this message at Oct 15, 2017 around 04:44

evol262
Nov 30, 2010
#!/usr/bin/perl

No, since virtio still uses the entire storage subsystem, and the network subsystem, and the scheduler, etc. Plus all the variations in possible libvirt CPU models. And testing host-passthrough.

But the point of regression testing is to check whether the patch failed anywhere, from scheduler deadlocks to a failure in an ancient cciss controller. If any test fails, the entire suite fails, and any exceptions are known for documentation or additional patches.

E: I work on rhev and libvirt. There are a huge number of kernel bugs which are only reproducible on specific hardware, which is why most vendors also have a lab where you can go request a DL380 G6 or whatever.

Even a 0-day in USB mass storage can't be tested in VMs, since some flash drives present as multipath. Testing on real hardware is necessary. We do include a large suite of virtualized sanity tests, but passing that is not sufficient for a regression pass. It's only part.

Canonical doesn't test at all, including on virt. They rely on upstream to have tested, and assume a successful kernel build for whatever Arch is good enough

evol262 fucked around with this message at Oct 14, 2017 around 19:28

Saukkis
May 16, 2003



So you people are of the opinion, that it's better to shut down a service for several days than using a non-QA'd kernel that would most likely work just fine without any issues. Unless you have better information about how many attempts Red Hat usually requires before they manage to produce a kernel that doesn't fail catastrophically on a common hardware?

And I'm not planning to stuff it in hundreds of servers, I just want it for the couple that have untrusted users. Of course it shouldn't be available through standard channels, just give me a maze of half a dozen links and with as many warnings and disclaimers as you want, as long as there are beta RPM downloads at the end. Really, how much worse it can be than the fully-QA'd kernels which fail to boot every now and then anyway.

And on the iptables issue, how long can it take to QA a patch that teaches iptables-restore how to use -w? In the mean time whenever we start a RHEL 7.4 server there is about 1 in 3 chance that it won't have a firewall.

Saukkis fucked around with this message at Oct 16, 2017 around 18:47

Vulture Culture
Jul 14, 2003

I was never enjoying it. I only eat it for the nutrients.


Saukkis posted:

So you people are of the opinion, that it's better to shut down a service for several days than using a non-QA'd kernel that would most likely work just fine without any issues. Unless you have better information about how many attempts Red Hat usually requires before they manage to produce a kernel that doesn't fail catastrophically on a common hardware?

And I'm not planning to stuff it in hundreds of servers, I just want it for the couple that have untrusted users. Of course it shouldn't be available through standard channels, just give me a maze of half a dozen links and with as many warnings and disclaimers as you want, as long as there are beta RPM downloads at the end. Really, how much worse it can be than the fully-QA'd kernels which fail to boot every now and then anyway.

And on the iptables issue, how long can it take to QA a patch that teaches iptables-restore how to use -w? In the mean time whenever we start a RHEL 7.4 server there is about 1 in 3 chance that it won't have a firewall.
You might want to consider taking this up with your TAM rather than unloading at strangers on the Internet. This is a pretty standard way of getting access to hotfixes ahead of GA.

anthonypants
May 6, 2007



Dinosaur Gum

Saukkis posted:

So you people are of the opinion, that it's better to shut down a service for several days than using a non-QA'd kernel that would most likely work just fine without any issues. Unless you have better information about how many attempts Red Hat usually requires before they manage to produce a kernel that doesn't fail catastrophically on a common hardware?

And I'm not planning to stuff it in hundreds of servers, I just want it for the couple that have untrusted users. Of course it shouldn't be available through standard channels, just give me a maze of half a dozen links and with as many warnings and disclaimers as you want, as long as there are beta RPM downloads at the end. Really, how much worse it can be than the fully-QA'd kernels which fail to boot every now and then anyway.

And on the iptables issue, how long can it take to QA a patch that teaches iptables-restore how to use -w? In the mean time whenever we start a RHEL 7.4 server there is about 1 in 3 chance that it won't have a firewall.
It seems like wanting kernel updates, no matter the cost, is how you got into the issue with iptables in the first place.

evol262
Nov 30, 2010
#!/usr/bin/perl

Saukkis posted:

So you people are of the opinion, that it's better to shut down a service for several days than using a non-QA'd kernel that would most likely work just fine without any issues. Unless you have better information about how many attempts Red Hat usually requires before they manage to produce a kernel that doesn't fail catastrophically on a common hardware?
I don't think you understand what a regression test is, and maybe not a test harness.

If you feel like you need to shut down a service, you've done something wrong. Banking kept operating after multiple public CVEs. Even if it took some banks a couple of days to apply patches for Heartbleed (despite the it being a coordinated release with responsible disclosure, meaning packages for RHEL were available within minutes of every other distro), they didn't "shut down" until they could schedule a maintenance window.

And it's not about taking multiple attempts to "produce a kernel that doesn't fail catastrophically". The RHEL kernel does through a full regression test on PPC, s390, x86_64, and sometimes aarch64. A regression test here meaning "every kernel runs through a couple thousand tests of various subsystems on every possible piece of hardware", and some of those tests take a while. Kernels rarely fail, because they also go through the testing harness as part of development. But it still takes a while to say "ensure that bug#12345 didn't regress"

Note that for critical CVEs, we get them out as fast as possible. This means war rooms, maintainers working during PTO, etc.

Saukkis posted:

And I'm not planning to stuff it in hundreds of servers, I just want it for the couple that have untrusted users. Of course it shouldn't be available through standard channels, just give me a maze of half a dozen links and with as many warnings and disclaimers as you want, as long as there are beta RPM downloads at the end. Really, how much worse it can be than the fully-QA'd kernels which fail to boot every now and then anyway.
If kernels fail to boot, file a support case so the bug can be fixed. If you have done this, get your TAM/CEE rep to escalate it. If you haven't, there's no way the kernel engineering team can possibly be aware of it (or a failure in their testing suite).

If you want an unsupported kernel, which those RPMs surely would be, feel free to use ELrepo or apply a patch to the last released SRPM yourself. The patches are public. The SRPMs are public. This isn't hard to do if it's really urgent for you; so urgent that you feel the need to rail about it on an internet comedy forum.

Saukkis posted:

And on the iptables issue, how long can it take to QA a patch that teaches iptables-restore how to use -w? In the mean time whenever we start a RHEL 7.4 server there is about 1 in 3 chance that it won't have a firewall.
I work in Red Hat engineering. I am not omniscient. I have no idea what bug you're talking about, and I have no information about it.

It's far more likely that whatever bug you're talking about isn't a Z-stream, which means it will stay "ON_QA" until whenever RHEL 7.5 comes out. Want it to be a Z-stream? Talk to your TAM and express that this is important. The fact that it's presumably targeted to a Y-stream with or without a TAM presumably means that it's very hard to reproduce (much more than 33%), has few users (like the rancid bug from the last page, not iptables/firewalld) or has an easy workaround. Have you opened a support case? Open a support case. Nobody can help you here.

I'm not saying that there aren't problems or bugs with RHEL. I'm saying that there's no way they can be fixed unless they're reported. And that if there is a bug you want to know the schedule for, you can talk to CEE or your TAM.

I'd also suggest that, given the size of the RHEL install base and the fact that these bugs are presumably not urgent enough for a Z-stream or asynchronous release, that there's a problem with your environment or your kickstart somewhere if there's a 1/3 chance a system won't have a firewall up.

anthonypants
May 6, 2007



Dinosaur Gum

evol262 posted:

I work in Red Hat engineering. I am not omniscient. I have no idea what bug you're talking about, and I have no information about it.

It's far more likely that whatever bug you're talking about isn't a Z-stream, which means it will stay "ON_QA" until whenever RHEL 7.5 comes out. Want it to be a Z-stream? Talk to your TAM and express that this is important. The fact that it's presumably targeted to a Y-stream with or without a TAM presumably means that it's very hard to reproduce (much more than 33%), has few users (like the rancid bug from the last page, not iptables/firewalld) or has an easy workaround. Have you opened a support case? Open a support case. Nobody can help you here.

I'm not saying that there aren't problems or bugs with RHEL. I'm saying that there's no way they can be fixed unless they're reported. And that if there is a bug you want to know the schedule for, you can talk to CEE or your TAM.

I'd also suggest that, given the size of the RHEL install base and the fact that these bugs are presumably not urgent enough for a Z-stream or asynchronous release, that there's a problem with your environment or your kickstart somewhere if there's a 1/3 chance a system won't have a firewall up.
I think it's this one, but there's a bunch of associated bugs in RedHat's bugzilla, in various stages of ON_QA or VERIFIED or CLOSED ERRATA. One of those related bugs is from a year ago and is for 7.3. Another bug has a comment that suggests that a commenter, "contact our support where you can request the package as a hotfix".

evol262
Nov 30, 2010
#!/usr/bin/perl

That bug is VERIFIED and a z-stream. Which means it'll be out with the next RHEL z-stream. Which means batch updates.

These are on a regular cadence. It is not a mystery to guess when it's going to be (soon)

For the 7.3 bug, you can tell from the doc text being upgraded that it'll be available very soon.

Yes, there are hotfixes available for both of these.

Adbot
ADBOT LOVES YOU

anthonypants
May 6, 2007



Dinosaur Gum

evol262 posted:

That bug is VERIFIED and a z-stream. Which means it'll be out with the next RHEL z-stream. Which means batch updates.

These are on a regular cadence. It is not a mystery to guess when it's going to be (soon)

For the 7.3 bug, you can tell from the doc text being upgraded that it'll be available very soon.

Yes, there are hotfixes available for both of these.
Yeah, that errata advisory for OpenStack is from a few hours ago.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply
«684 »