New around here? Register your SA Forums Account here!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
redleader
Aug 18, 2005

Engage according to operational parameters

cinci zoo sniper posted:

i mean, i love reading this and pl thread and hacker news etc even when its like hackbunny's 69th 800 word post in a thread about some subtleties of windows 95 gui radio menu api due to attempted kernel level mitigation of impact of gamma rays hitting the cpu, but i struggle to find it engaging to actually code something up at home, code or not

:hfive:

i need to get over this and start doing stuff at home because i have no cs fundamentals and cannot algorithm/data structure

Adbot
ADBOT LOVES YOU

gonadic io
Feb 16, 2011

>>=

redleader posted:

:hfive:

i need to get over this and start doing stuff at home because i have no cs fundamentals and cannot algorithm/data structure

I find I'm the most productive at home when I instead should be doing housework or going to bed before work. Procrastination as a productivity technique!

cinci zoo sniper
Mar 14, 2013




redleader posted:

:hfive:

i need to get over this and start doing stuff at home because i have no cs fundamentals and cannot algorithm/data structure

this

gonadic io posted:

I find I'm the most productive at home when I instead should be doing housework or going to bed before work. Procrastination as a productivity technique!

i usually go into full Cinderella mode procrastinating and polish my apartment better than cats balls :v:

DONT THREAD ON ME
Oct 1, 2002

by Nyc_Tattoo
Floss Finder
turns out the dumb embedding of HKTs I did in Rust also works in Swift:

code:
func takes_applicative<F: Applicative, A, B>(fa: Lifted<F, A, F.Z>, fb: Lifted<F, B, F.Z>) -> Lifted<F, (A,B), F.Z> {
    return F.product(fa: fa, fb: fb)
}

let opt_a = OptionKind.point(a: 1) // Lifted<OptionKind, Int, Nothing>(Option<Int>.Some(1))
let opt_b = OptionKind.point(a: "string") // Lifted<OptionKind, String, Nothing>.Option(Option<String>.Some("string"))

let res_a = ResultKind<String>.point(a: 1) // Lifted<ResultKind, Int, String>.Result(Result<Int, String>.Ok(1))
let res_b = ResultKind<String>.point(a: "string") // Lifted<ResultKind, String, String>.Result(Result<String, String>.Ok("string"))

takes_applicative(fa: res_a, fb: res_b).unlift() // Result(ok: (1,"string"))
takes_applicative(fa: opt_a, fb: opt_b).unlift() // Some(item: (1, "string"))
https://gist.github.com/daviswahl/c4603cff8ddbc5753ef4a079fd48392b

DONT THREAD ON ME fucked around with this message at 19:55 on Dec 18, 2018

Powerful Two-Hander
Mar 9, 2004

Mods please change my name to "Tooter Skeleton" TIA.


oh cool they didn't understand db functions either so there are two identical scalar functions that run the exact same query but return differne columns from the table.

this also could have been done with a sub query and avoided making GBS threads up the database with even more pointless crap.

edit:both functions are used to return data in the same stored procedure :suicide:

Powerful Two-Hander fucked around with this message at 22:15 on Dec 18, 2018

Cold on a Cob
Feb 6, 2006

i've seen so much, i'm going blind
and i'm brain dead virtually

College Slice

Powerful Two-Hander posted:


*this is my favourite: performed every single logical test by returning a magic string from the db and checking its value, not even just "true" or whatever but poo poo like if(caniDoIt = "yes you can go ahead")
lmao

did you outsource to galaxybrain consulting llc??

Oneiros
Jan 12, 2007



Powerful Two-Hander posted:

*this is my favourite: performed every single logical test by returning a magic string from the db and checking its value, not even just "true" or whatever but poo poo like if(caniDoIt = "yes you can go ahead")

Not gonna lie, someone wrote one of our half dozen feature flag / toggle / AB testing systems to operate this way and I've had a lot of fun with the magic string values.

TheFluff
Dec 13, 2006

FRIENDS, LISTEN TO ME
I AM A SEAGULL
OF WEALTH AND TASTE

Corla Plankun posted:

is it possible to post about the context of that lateral join or is it too hard to talk generally about without doxxing your job?

thats one of those things that i kinda know of but have never had to use or even consider using so i don't know what the good applications of it are

the most obvious use case for lateral joins (which is pretty much exaclly what i had going on in this case) is when you need a regular inner join with a complex condition that usually looks something like "join only the first three rows of the right hand table ordered by some criteria". if you are selecting only one column from one row from the right hand table, you can do this with a subquery instead (or with an aggregation, like max()/GROUP BY), but if you want something more complex than that, then a lateral join is usually the way to go.

in my case we have a table that should really be 1-to-1 with the users table, but because of terrible programming it's soft-deletable and that constraint can't be enforced, so i was writing something like (bowdlerized and probably makes no sense taken out of context)

SQL code:
SELECT posters_with_avatars.id
FROM users
CROSS JOIN LATERAL (
  SELECT forumsposters.id, avatars.uploaded_at
  FROM forumsposters
  INNER JOIN avatars ON (forumsposters.id = avatars.poster_id)
  WHERE forumsposters.user_id = users.id
  -- use non-deleted forums poster if one exists, otherwise most recently deleted one
  ORDER BY forumsposters.deleted_at DESC NULLS FIRST
  LIMIT 1
) AS posters_with_avatars
WHERE
  users.is_disabled = FALSE AND
  posters_with_avatars.uploaded_at BETWEEN ('2018-01-01' AND '2018-06-01') AND
  EXISTS (
    SELECT *
    FROM probations
    WHERE
      probations.user_id = users.id AND
      probations.created_at BETWEEN (
        posters_with_avatars.uploaded_at AND
        posters_with_avatars.uploaded_at + INTERVAL '3 days'
      )
  )
e: https://explainextended.com/2009/07/16/inner-join-vs-cross-apply/

TheFluff fucked around with this message at 09:08 on Dec 19, 2018

Powerful Two-Hander
Mar 9, 2004

Mods please change my name to "Tooter Skeleton" TIA.


Cold on a Cob posted:

lmao

did you outsource to galaxybrain consulting llc??

we offshore solely on the basis of cost so yes pretty much

entirely rewriten now. four useless duplicate database processes removed, everything neatly contained in a single model that loads its child objects on access instead of hurling around huge blobs of data in the session, and no more magic strings

a good days work.


oh wait there are two more copy and paste app sections written by the same person that I loving guarantee have the same problem, here we go again!

Shaman Linavi
Apr 3, 2012

put in my first pull request. a framework to do another teams user stories. also when my team noticed how ducked up the story was we talked to the other team and oops, looks like there are several thousand fields that area missing a dollar sign so lol not my teams problem

edit: also found out today that code freeze is Friday lol kill me

Shaman Linavi fucked around with this message at 22:09 on Dec 19, 2018

distortion park
Apr 25, 2011


TheFluff posted:

the most obvious use case for lateral joins (which is pretty much exaclly what i had going on in this case) is when you need a regular inner join with a complex condition that usually looks something like "join only the first three rows of the right hand table ordered by some criteria". if you are selecting only one column from one row from the right hand table, you can do this with a subquery instead (or with an aggregation, like max()/GROUP BY), but if you want something more complex than that, then a lateral join is usually the way to go.

in my case we have a table that should really be 1-to-1 with the users table, but because of terrible programming it's soft-deletable and that constraint can't be enforced, so i was writing something like (bowdlerized and probably makes no sense taken out of context)

SQL code:
SELECT posters_with_avatars.id
FROM users
CROSS JOIN LATERAL (
  SELECT forumsposters.id, avatars.uploaded_at
  FROM forumsposters
  INNER JOIN avatars ON (forumsposters.id = avatars.poster_id)
  WHERE forumsposters.user_id = users.id
  -- use non-deleted forums poster if one exists, otherwise most recently deleted one
  ORDER BY forumsposters.deleted_at DESC NULLS FIRST
  LIMIT 1
) AS posters_with_avatars
WHERE
  users.is_disabled = FALSE AND
  posters_with_avatars.uploaded_at BETWEEN ('2018-01-01' AND '2018-06-01') AND
  EXISTS (
    SELECT *
    FROM probations
    WHERE
      probations.user_id = users.id AND
      probations.created_at BETWEEN (
        posters_with_avatars.uploaded_at AND
        posters_with_avatars.uploaded_at + INTERVAL '3 days'
      )
  )
e: https://explainextended.com/2009/07/16/inner-join-vs-cross-apply/

If I understood this correctly I think this is similar to a t sql window function?

redleader
Aug 18, 2005

Engage according to operational parameters
the tsql equivalent is CROSS/OUTER APPLY

DONT THREAD ON ME
Oct 1, 2002

by Nyc_Tattoo
Floss Finder
i really need to take the time to learn sql properly. i was getting there once but i havent actually had to use it since my first job.

cinci zoo sniper
Mar 14, 2013




DONT THREAD ON ME posted:

i really need to take the time to learn sql properly. i was getting there once but i havent actually had to use it since my first job.

good news is that it’s dead simple, i imagine you’ll get over it in a week.

AWWNAW
Dec 30, 2008

SQL? yeah. it’s good

floatman
Mar 17, 2009
SQL good. But sometimes things done with SQL very bad.

redleader
Aug 18, 2005

Engage according to operational parameters

floatman posted:

SQL good. But sometimes things done with SQL very bad.

Ciaphas
Nov 20, 2005

> BEWARE, COWARD :ovr:


floatman posted:

SQL good. But sometimes things done with SQL very bad.

i wonder what language causes the most revulsion in devs when they see examples of something truly foul being done

i don't think it's sql but it's sure up there

Corla Plankun
May 8, 2007

improve the lives of everyone
im sure i've posted about this before but i once found a query in the wild that was being used "for financial modelling" that was like

code:
SELECT
  DAYOFMONTH(date) as day_of_month,
  butt_type,
  butt_size,
  butt_shape,
  avg(butt_volume),
  sum(butt_users)
FROM insanely_big_hive_table_that_took_ninety_minutes_to_scan
WHERE
  various_butt_conditions 
  AND DAYOFMONTH(date) = 1

UNION ALL

SELECT
  DAYOFMONTH(date) as day_of_month,
  butt_type,
  butt_size,
  butt_shape,
  avg(butt_volume),
  sum(butt_users)
FROM insanely_big_hive_table_that_took_ninety_minutes_to_scan
WHERE
  various_butt_conditions 
  AND DAYOFMONTH(date) = 2

UNION ALL 

-- etc, etc, etc

UNION ALL

SELECT
  DAYOFMONTH(date) as day_of_month,
  butt_type,
  butt_size,
  butt_shape,
  avg(butt_volume),
  sum(butt_users)
FROM insanely_big_hive_table_that_took_ninety_minutes_to_scan
WHERE
  various_butt_conditions 
  AND DAYOFMONTH(date) = 31

distortion park
Apr 25, 2011


redleader posted:

the tsql equivalent is CROSS/OUTER APPLY

You're right. I've always joined then done row_number to achieve this and have no idea why

TheFluff
Dec 13, 2006

FRIENDS, LISTEN TO ME
I AM A SEAGULL
OF WEALTH AND TASTE
you can usually achieve the same thing in a variety of more or less convoluted ways but the lateral join/cross apply usually performs significantly better than an aggregation would (whether it's done with grouping or with partitioning/window functions)

it feels great to be good at sql and replace like a hundred lines of convoluted legacy application code with 15 lines of sql that performs better.

cinci zoo sniper
Mar 14, 2013




that union stuff is impressively bad

redleader
Aug 18, 2005

Engage according to operational parameters
i've never actually used a cross apply (i.e. lateral join) on anything other than a table valued function. weirdly this thread has made me understand cross apply a bit better, although i still don't know how i'd recognise an opportunity to use one...

i have however used window functions to write a query so convoluted that i couldn't understand it a mere day later

cinci zoo sniper
Mar 14, 2013




redleader posted:

i've never actually used a cross apply (i.e. lateral join) on anything other than a table valued function. weirdly this thread has made me understand cross apply a bit better, although i still don't know how i'd recognise an opportunity to use one...

i have however used window functions to write a query so convoluted that i couldn't understand it a mere day later

lateral join is when your sub select needs access to the outer scope variables, e.g. select i.* from inner i where i.device_id = o.device_id and i.created_at::date < o.event_date

redleader
Aug 18, 2005

Engage according to operational parameters
yeah yeah. i just don't see how i'd be able to spot an opportunity to use one, since that sort of query is unusual at $job

... or is it, and i just haven't noticed? :ghost:

Corla Plankun
May 8, 2007

improve the lives of everyone

redleader posted:

i have however used window functions to write a query so convoluted that i couldn't understand it a mere day later

lmao, extremely same

cinci zoo sniper
Mar 14, 2013




Corla Plankun posted:

lmao, extremely same

i have a query waiting for me to return from vacation that changes its output on minute scale when every time stamp component involved should be on day scale, also i realised where the problem was like 4 words into the post lol :v:

raminasi
Jan 25, 2005

a last drink with no ice
i feel like sql proficiency is bipolar, like i feel comfortable with basic concepts like joins and indices and i can mostly read a query plan but i'm worried that if i put sql on my resume i'll walk into an interview where i'll have to use lateral joins and window functions and all kinds of other shenanigans that i have no proficiency with

Corla Plankun
May 8, 2007

improve the lives of everyone
i can do just about anything in sql but i have been unable to answer interview questions about it before if they're in the style of "what is a <random function from sql>" instead of "write a query that uses these tables to make <some kind of dataset>"

but being forthright about this in interviews has gone really well. I think it would only fail me out of an interview for a job that was a really bad fit

30 TO 50 FERAL HOG
Mar 2, 2005



Corla Plankun posted:

im sure i've posted about this before but i once found a query in the wild that was being used "for financial modelling" that was like

code:
SELECT
  DAYOFMONTH(date) as day_of_month,
  butt_type,
  butt_size,
  butt_shape,
  avg(butt_volume),
  sum(butt_users)
FROM insanely_big_hive_table_that_took_ninety_minutes_to_scan
WHERE
  various_butt_conditions 
  AND DAYOFMONTH(date) = 1

UNION ALL

SELECT
  DAYOFMONTH(date) as day_of_month,
  butt_type,
  butt_size,
  butt_shape,
  avg(butt_volume),
  sum(butt_users)
FROM insanely_big_hive_table_that_took_ninety_minutes_to_scan
WHERE
  various_butt_conditions 
  AND DAYOFMONTH(date) = 2

UNION ALL 

-- etc, etc, etc

UNION ALL

SELECT
  DAYOFMONTH(date) as day_of_month,
  butt_type,
  butt_size,
  butt_shape,
  avg(butt_volume),
  sum(butt_users)
FROM insanely_big_hive_table_that_took_ninety_minutes_to_scan
WHERE
  various_butt_conditions 
  AND DAYOFMONTH(date) = 31

:stare:

30 TO 50 FERAL HOG
Mar 2, 2005



wish i could dig up that post about "big data is just consulting telling you to give your virtual server more ram" because holy lmao

cinci zoo sniper
Mar 14, 2013




BIGFOOT EROTICA posted:

wish i could dig up that post about "big data is just consulting telling you to give your virtual server more ram" because holy lmao

https://twitter.com/garybernhardt/status/600783770925420546

30 TO 50 FERAL HOG
Mar 2, 2005



namaste

pseudorandom name
May 6, 2007

hey tef you’re famous
https://twitter.com/id_aa_carmack/status/1075857848016560130?s=21

Powerful Two-Hander
Mar 9, 2004

Mods please change my name to "Tooter Skeleton" TIA.


i wrote a data load process using a merge statement once and I'm pretty sure thta nobody else in the entire org knows what the gently caress that is. I kind of can't remember myself tbh but it worked and is still in production 6 years later.

iirc it replaced the previous effort which was "truncate entire 2 million row table and copy all rows from the loader table in one mega transaction". this took 20+ minutes minimum if it even actually worked. every day. during that time nothing worked that relied on that db.

Powerful Two-Hander fucked around with this message at 22:44 on Dec 20, 2018

Nomnom Cookie
Aug 30, 2009



Powerful Two-Hander posted:

i wrote a data load process using a merge statement once and I'm pretty sure thta nobody else in the entire org knows what the gently caress that is. I kind of can't remember myself tbh but it worked and is still in production 6 years later.

iirc it replaced the previous effort which was "truncate entire 2 million row table and copy all rows from the loader table in one mega transaction". this took 20+ minutes minimum if it even actually worked. every day. during that time nothing worked that relied on that db.

is that oracle? about all i remember about oracle is a million copies of COALESCE(fart, '') everywhere like every nullable column a query touched would get a coalesce. i was a junior at the time and moaned to myself about the idiots around me writing bad sql and how long the queries took, now i'm just impressed by oracle taking the poo poo we threw at it and returning results at all. a good backing store for oracle apps

floatman
Mar 17, 2009

Ciaphas posted:

i wonder what language causes the most revulsion in devs when they see examples of something truly foul being done

i don't think it's sql but it's sure up there

There was recently a jobs queue system implemented in SQL.
Jobs went in a table, then something picked jobs out. Jobs were then done.
Because there was multiple users who could queue jobs, a random picking algorithm was required so that the job system doesn't get stuck doing all the jobs for user A before user B (gotta be fair).
BUT then there was the concept that some jobs were of higher priority than other jobs, so there must be a random picking of jobs, but "randomly" pick the higher priority jobs first. And it was decided that random was not fair enough, so we must pick user jobs by round robin (i.e. high priority job for user A, high priority job for userB, normal job for user A, etc).

So how did this pan out? Some loving poo poo giant rear end SQL statement embedded in the codebase that used recursive CTEs. Since jobs were either waiting status or completed status, whenever a job was picked, the job record needed a lock to lock the job so if the SQL did run again while this job was running it wouldn't pick up the same job. This lock also touched the other tables that were involved in the job (I E. Updating of user details, etc you can imagine the loving shitshow). The recursive CTE query also had this group/order by statement that did JACK poo poo and was not needed but formed 90% of the query plan. Database just loving mysteriously died one-day once the amount of data that needed to be scanned through surpassed the query planned limits and the SQL server just loving gave up and decided to chug along row by row.
And here's the kicker:
The codebase called that SQL query to pick up a "semi-random" job if it wanted to get the next job to process. But if the codebase KNEW the primary key of the job it wanted to process and needed to get that job, it would still call this idiot hellfucker SQL query but with the primary key specified so even though you're getting just that one record you could have gotten with a straight up select you're still going to pay the costs of the recursive CTE and the 90% group/order by statement.

All these loving problems could have been avoided if we just did a naive random select from a queue of jobs. "But what about higher priority jobs," You may ask.
Welp, JUST USE TWO SEPARATE QUEUES and empty the priority one first duhhhhhhhhhhhhhh

Gazpacho
Jun 18, 2004

by Fluffdaddy
Slippery Tilde
:wth:

Slurps Mad Rips
Jan 25, 2009

There's No Guarantee

Kevin Mitnick P.E. posted:

c++ is supposed to get modules eventually but millenials killed it. at least i assume we have succeeded by now cause that post is from a year ago. sorry

I'm in the process of updating the style of my site which is why I haven't posted an update on where we are as of san diego, and nothing else has been posted there since the millenials post. We're getting "merged modules", which are a combination of the modules TS and the ATOM proposal. If we can get P1302 added to an appendix in some way, it might even be fun to write modularized c++ :v:

Adbot
ADBOT LOVES YOU

Powerful Two-Hander
Mar 9, 2004

Mods please change my name to "Tooter Skeleton" TIA.


Kevin Mitnick P.E. posted:

is that oracle? about all i remember about oracle is a million copies of COALESCE(fart, '') everywhere like every nullable column a query touched would get a coalesce. i was a junior at the time and moaned to myself about the idiots around me writing bad sql and how long the queries took, now i'm just impressed by oracle taking the poo poo we threw at it and returning results at all. a good backing store for oracle apps

nah mssql, though actually hella same for coalesce operations everywhere

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply