Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
Carthag Tuek
Oct 15, 2005

Tider skal komme,
tider skal henrulle,
slægt skal følge slægters gang



CPColin posted:

Guess I'll go function-by-function!
            /

Adbot
ADBOT LOVES YOU

MrMoo
Sep 14, 2000

tef posted:

stripped out protobufs from python code and made them go faster and smaller with gzipped json (very string heavy data)

Outside of C/C++ this is almost always going to be the case. Every serialization performance test, no matter how flawed, will show JSON performing faster than others. The gotcha is the size of the message and the transit costs that means, so say using a reverse proxy to transparent `gzip` and if the client is a browser, you get "free" decompression.

Carthag Tuek
Oct 15, 2005

Tider skal komme,
tider skal henrulle,
slægt skal følge slægters gang



to add: if your transport is http and you gzip in code, you hosed up

Sapozhnik
Jan 2, 2005

Nap Ghost
is that brotli compression thing worth bothering with

it was created by googleoids so i assume no

12 rats tied together
Sep 7, 2006

we use a lot of zstd at current job and i don't think i've heard anyone complaining about it yet

Twerk from Home
Jan 17, 2009

This avatar brought to you by the 'save our dead gay forums' foundation.

Sapozhnik posted:

is that brotli compression thing worth bothering with

it was created by googleoids so i assume no

Brotli is excellent for compressing static web text and you should do it. It is not a general purpose compressor. It has a big dictionary in the spec, so every browser has a giant brotli reference set shipping with it that lets you do insanely effective compression.

Zstd is insanely good and general purpose.

hitchens
Oct 24, 2012

harlequin macaw


Spent half the day getting the applet used for the build system to run after corporate policy hosed around with our Java installs again, but now it’s Friday and my poo poo is heading off for prod.

Powerful Two-Hander
Mar 10, 2004

Mods please change my name to "Tooter Skeleton" TIA.



I forgot about this lol

CPColin
Sep 9, 2003

Big ol' smile.
Crossposting from Coding Horrors because I figured it out:

CPColin posted:

gently caress you, Groovy (highlighting as Java for readability):
Java code:
    private Object singleValueFromString(ReportDefinitionParam rdp, String s) {
       if (rdp?.isTypeDateTime()) {
           try { 
               return new DateTime(Date.parse(rdp.pattern ?: 'MM/dd/yyyy HH:mm', s)) 
           } catch (Exception e) { 
               try { 
                   // try date portion only
                   return new DateTime(Date.parse(rdp.pattern ?: 'MM/dd/yyyy', s)) 
               } catch (Exception e2) { return null }
           }
        } else if (rdp?.isTypeDate()) {
            try { return Date.parse(rdp.pattern ?: 'MM/dd/yyyy', s) } catch (Exception e) { return null }
        } else if (rdp?.isTypeInteger()) {
            try { return new BigDecimal(s) as Integer } catch (Exception e) { return null }
        } else if (rdp?.isTypeDecimal()) {
            try { return new BigDecimal(s) } catch (Exception e) { return null }
        } else {
            return s
        }
    }
This code started breaking because I removed the ReportDefinitionParam.getPattern() function and didn't realize it was being used here. I couldn't tell it was being used here because, when the Groovy runtime encounters a property name that doesn't exist in the class, it throws something that extends Exception, not Error, so it was being silently discarded by one of these catch blocks.

The extra kicker was that nothing was ever calling setPattern() or addPattern(), so getPattern() was always returning null, before I clobbered it. Great!

(I should note that I inherited this code)

cinci zoo sniper
Mar 15, 2013




brotli (usually) is quicker to compress and compresses marginally better (compare brotli level 5 to zstd level 9 for a similar reference of “decently compressed”), and the differences we're talking about is a variable couple of % of compression ratio difference at very variable compression speed difference. you can also do the dictionary thing with zstd if you're benefitting from min-maxing the numbers after comma on really small files.

zstd decompresses much faster in literally any scenario, and is not a streamed format so you get such innovative features like checksums for your compressed files

in other words, brotli is great at sending a webpage into a web browser, but for everything else zstd should be the default choice these days

cinci zoo sniper
Mar 15, 2013




in terms of real world users that aren't their home companies, iirc cloudflare does brotli on everythign, and zstd ive heard is used by some bigger aws teams for, e.g., internal tooling log storage

Share Bear
Apr 27, 2004

tef posted:

stripped out protobufs from python code and made them go faster and smaller with gzipped json (very string heavy data)

amazing and this also applies to me. was this a “this might be faster” hunch or you just tried it on a lark?

Qtotonibudinibudet
Nov 7, 2011



Omich poluyobok, skazhi ty narkoman? ya prosto tozhe gde to tam zhivu, mogli by vmeste uyobyvat' narkotiki

MrMoo posted:

Outside of C/C++ this is almost always going to be the case. Every serialization performance test, no matter how flawed, will show JSON performing faster than others. The gotcha is the size of the message and the transit costs that means, so say using a reverse proxy to transparent `gzip` and if the client is a browser, you get "free" decompression.



wait what json is _faster_ to (de)serialize than protobuf? is it cheaper in memory too?

perf dumps of our kubernetes poo poo shows that we spend a ton of resources converting to and from JSON for API server interaction, and i was thinking "surely there must be a better way and this poo poo doesn't reallllllllly need json" but

MrMoo
Sep 14, 2000

It should not be read literally, what happens in higher level languages is that you end up with a dual level of serialization, between internal format to JSON, then to text. Thus in JavaScript land, objects are already JS objects thus that eliminates an entire stage of the pipeline in comparison. "Products" like MsgPack try to benefit from the lower memory usage in encoded form.

Ultimately with encoded messages there is a massive performance and convenience trade off. Random access to a field is an incredibly expensive operation and usually means (1) conversion to a map, or (2) reparsing on each access. The highest performance is given by single pass through a structure. However this only means something if your language has direct access to unpacked fields, any slow interpreted language is likely to require a map to simply access any variable, ideally JITs eliminating this.

gnatalie
Jul 1, 2003

blasting women into space

Powerful Two-Hander posted:

I guarantee you they do not know what that is or what a deploy tool is either. if they did they could have said "ok deploy the latest prod artefact build to the test env and also refresh the database back from production" which is actually dead simple to do but I'm not gonna do it unless asked and we have other stuff in that env anyway

last week they took screenshots of some code in gitlab as evidence of ??? and it was literally a wait loop that is used to just go "nothing to do, sleeping"

also these *are* the sox auditors lmao

loooooool auditors looking at code, fantastic.

VikingofRock
Aug 24, 2008




cinci zoo sniper posted:

brotli (usually) is quicker to compress and compresses marginally better (compare brotli level 5 to zstd level 9 for a similar reference of “decently compressed”), and the differences we're talking about is a variable couple of % of compression ratio difference at very variable compression speed difference. you can also do the dictionary thing with zstd if you're benefitting from min-maxing the numbers after comma on really small files.

zstd decompresses much faster in literally any scenario, and is not a streamed format so you get such innovative features like checksums for your compressed files

in other words, brotli is great at sending a webpage into a web browser, but for everything else zstd should be the default choice these days

For data at rest, does it still make sense to use lzma2 (e.g. with xz)? IIRC that had a significantly better compression ratio than zstd, but with significantly slower compression / decompression.

cinci zoo sniper
Mar 15, 2013




VikingofRock posted:

For data at rest, does it still make sense to use lzma2 (e.g. with xz)? IIRC that had a significantly better compression ratio than zstd, but with significantly slower compression / decompression.

imo don't use xz or lzma2. https://www.nongnu.org/lzip/xz_inadequate.html explains why better than i will be able to. and https://lists.archlinux.org/pipermail/arch-dev-public/2019-March/029520.html as a real world supplement i guess

lzma, however, is fine, and still gets you the 2nd best compression ratio. due to threading support and the number of mit phds involved into coding these, you're basically looking at like 5% compression ratio reductions going from zpaq to lzma, and then maybe 10% going from lzma to zstd (pulling random numbers out of my rear end because this is source material-dependant). however, and it's up to you to decided if this however matters to you, lzma is several orders of magnitude faster at de/compressing than zpaq, whereas zstd is several orders of magnitude faster at de/compressing than lzma. yeah, it's one of those "big tech spends big money to do one thing well" things.

Jabor
Jul 16, 2010

#1 Loser at SpaceChem
lzma balls

Bloody
Mar 3, 2013

zstd balls

necrotic
Aug 2, 2005
I owe my brother big time for this!
one part of zstd that intrigues me is the training and dictionary you can provide to improve compression. it theoretically fits a use case of ours very nicely, but haven’t had the opportunity to actually try it out yet.

are there any other compression algorithms that have something similar?

cinci zoo sniper
Mar 15, 2013




necrotic posted:

one part of zstd that intrigues me is the training and dictionary you can provide to improve compression. it theoretically fits a use case of ours very nicely, but haven’t had the opportunity to actually try it out yet.

are there any other compression algorithms that have something similar?

brotli comes with a very large, pre-packaged dictionary, and with tooling to make your own dictionaries as well

Twerk from Home
Jan 17, 2009

This avatar brought to you by the 'save our dead gay forums' foundation.
Does Intel QAT hardware accelerate zstd yet? With this new generation of Xeons, almost every Intel server out there is going to have gzip hardware acceleration, which may tilt the scales back towards good ol gzip unless zstd can use it too.

necrotic
Aug 2, 2005
I owe my brother big time for this!

cinci zoo sniper posted:

brotli comes with a very large, pre-packaged dictionary, and with tooling to make your own dictionaries as well

cool, will add that to my list to check. for us the training is enticing because we have a huge set of event types, where some of the smaller events may benefit from some training. might end up being a total wash given what the data is, but worth a look at least.

Armitag3
Mar 15, 2020

Forget it Jake, it's cybertown.


cinci zoo sniper posted:

brotli comes with a very large, pre-packaged dictionary, and with tooling to make your own dictionaries as well

drat straight



(USER WAS PUT ON PROBATION FOR THIS POST)

cinci zoo sniper
Mar 15, 2013




Twerk from Home posted:

Does Intel QAT hardware accelerate zstd yet? With this new generation of Xeons, almost every Intel server out there is going to have gzip hardware acceleration, which may tilt the scales back towards good ol gzip unless zstd can use it too.

yes, zstd and brotli both

Carthag Tuek
Oct 15, 2005

Tider skal komme,
tider skal henrulle,
slægt skal følge slægters gang



Bloody posted:

zstd balls

id rather not

mystes
May 31, 2006

Bloody posted:

zstd balls
Is zstd pronounced "zested"?

cinci zoo sniper
Mar 15, 2013




mystes posted:

Is zstd pronounced "zested"?



no, the name of the algo is "zstandard"

cinci zoo sniper
Mar 15, 2013




also please stop scraping that lime, wtf

pokeyman
Nov 26, 2006

That elephant ate my entire platoon.
zuh steed

outhole surfer
Mar 18, 2003

cinci zoo sniper posted:

no, the name of the algo is "zstandard"

required pronunciation is an american impersonating a german tho

Zee Standard!

Powerful Two-Hander
Mar 10, 2004

Mods please change my name to "Tooter Skeleton" TIA.


I'm a proponent of zstding come ask me about life on the z free from government control

redleader
Aug 18, 2005

Engage according to operational parameters

cinci zoo sniper posted:

also please stop scraping that lime, wtf

you have a problem with zest?

cinci zoo sniper
Mar 15, 2013




nudgenudgetilt posted:

required pronunciation is an american impersonating a german tho

Zee Standard!

as oposed to what, "zed standard"? lomarf

redleader posted:

you have a problem with zest?

that looks very different from zesting stuff i'm used to, e.g., microplane

mystes
May 31, 2006

cinci zoo sniper posted:

as oposed to what, "zed standard"? lomarf
I think they just mean that you should say it with a german accent so "zee" will sound like it's supposed to be "the" with a german accent

VikingofRock
Aug 24, 2008




mystes posted:

I think they just mean that you should say it with a german accent so "zee" will sound like it's supposed to be "the" with a german accent

That's zjoke

Osmosisch
Sep 9, 2007

I shall make everyone look like me! Then when they trick each other, they will say "oh that Coyote, he is the smartest one, he can even trick the great Coyote."



Grimey Drawer
Tar, eXtracten ze files

hitchens
Oct 24, 2012

harlequin macaw


I get so tired of this poo poo. i don't do java but a bit ago a servlet would crash. Logs printed a ton about threadlocals being created and not closed, causing a possible memory leak. Asking our java devs I was met with big ??? reactions but the ops team wanted me to check it out.

Checked out the recent changes, found a threadlocal (very few of them around), made sure it was closed and asked the java peeps to make sure it all looked ok. It ran fine in dev now so all good right?

Of course not. They started digging into the issue and eventually came to the conclusion that "it's obviously a bug in java.lang.error()", reverted the change and disabled logging. Pushed it to prod which caused the servlet to eat poo poo over the weekend. Suggested we add in the quick fix so senior java dev adds it back in, delivers the new .class files and it's broken again which I've got to hear about all morning.

Pulled down the .class files w/ yesterday's change date into a decompiler and the change isn't there. It's the same stupid file we delivered last week. jfc

Sagacity
May 2, 2003
Hopefully my epitaph will be funnier than my custom title.

Astroid posted:

came to the conclusion that "it's obviously a bug in java.lang.error()"
classic

"It can't be *our* code that is buggy, it must be the code that's deployed on hundreds of millions of devices instead!!!"

Adbot
ADBOT LOVES YOU

pokeyman
Nov 26, 2006

That elephant ate my entire platoon.
well that one time there was a bug in binary search so you see

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply