awful programming: Somebody set up us the BOM

The Something Awful Forums > Discussion > Serious Hardware / Software Crap > YOSPOS > awful programming: Somebody set up us the BOM

«‹›1288 »

Finster Dexter: Oct 20, 2014; Beyond is Finster's mad vision of Earth transformed.

Powerful Two-Hander posted:

"what God has INNER JOINED together let no man put asunder"

INNER JOIN? pffft sounds like CROSS-JOIN-phobic nonsense

# ? Oct 23, 2018 00:40

Adbot: ADBOT LOVES YOU

# ? Nov 17, 2025 10:33

my homie dhall: Dec 9, 2010; honey, oh please, it's just a machine

prisoner of waffles posted:

#63 seems like a pretty small ask from someone donating their time to sqlite

# ? Oct 23, 2018 01:09

CRIP EATIN BREAD: Jun 24, 2002; Hey stop worrying bout my acting bitch, and worry about your WACK ass music. In the mean time... Eat a hot bowl of Dicks! Ice T; Soiled Meat

AggressivelyStupid posted:

what flavor of Christianity is this

the weird one

# ? Oct 23, 2018 02:13

CRIP EATIN BREAD: Jun 24, 2002; Hey stop worrying bout my acting bitch, and worry about your WACK ass music. In the mean time... Eat a hot bowl of Dicks! Ice T; Soiled Meat

cinci zoo sniper posted:

https://www.sqlite.org/codeofconduct.html

*nails github pull request to front door*

# ? Oct 23, 2018 02:29

Toady: Jan 12, 2009

AggressivelyStupid posted:

what flavor of Christianity is this

Diet TempleOS

# ? Oct 23, 2018 02:40

CRIP EATIN BREAD: Jun 24, 2002; Hey stop worrying bout my acting bitch, and worry about your WACK ass music. In the mean time... Eat a hot bowl of Dicks! Ice T; Soiled Meat

Toady posted:

Diet TempleOS

# ? Oct 23, 2018 02:43

pseudorandom name: May 6, 2007

CRIP EATIN BREAD posted:

the weird one

nah, this is bog-standard christianity

# ? Oct 23, 2018 05:18

CRIP EATIN BREAD: Jun 24, 2002; Hey stop worrying bout my acting bitch, and worry about your WACK ass music. In the mean time... Eat a hot bowl of Dicks! Ice T; Soiled Meat

pseudorandom name posted:

nah, this is bog-standard christianity

# ? Oct 23, 2018 05:25

animist: Aug 28, 2018

rip terry, and also the time cube guy. less terrible programmers than the rest of us

# ? Oct 23, 2018 07:05

galenanorth: May 19, 2016

Trying out a program that tries to systematically scrape from a store locator page that requires address input. I'm trying an approach that involves making requests over a grid of latitude/longitude coordinates, and using Google to automatically convert it to an address. The problem is that some of the coordinates of the grid will be in the ocean and not have addresses, so that scenario will probably require a long workaround.

For now, I'm skipping those coordinates and hoping that won't cause the scan to come up short. The workaround is probably going to involve automatically finding the nearest ZIP code from this spreadsheet which has a coordinate associated with each

Edit: I found the workaround. It involves splitting the grid unit centered in the ocean over and over until reaching a grid unit of a specified width, keeping the subunits that are on land, and discarding the ocean grid units once they've been split too much so the width is below the minimum

galenanorth fucked around with this message at 10:28 on Oct 23, 2018

# ? Oct 23, 2018 07:44

cinci zoo sniper: Mar 14, 2013

hahaha. so, we have a sql-backed service talk to an external service via json requests. i just found a table with all shipped json requests. they all include plain text auth in their body, where login is a deveoper�s surname and password is 123<their name>

# ? Oct 23, 2018 08:18

Powerful Two-Hander: Mar 9, 2004; Mods please change my name to "Tooter Skeleton" TIA.

today I managed to work around my block on implementing this feature that I was "sure" I could fit into the data structure I designed by using a combination of exploiting the use of display/editor templates in .net mvc to create a display template that is actually used to passively edit an object that is populated using html returned from an editor in a modal that is bounced via a controller and appended to the original display list.

there's also reflection and accessing of fields to get paths to templates to render default html and oh god what have I done am I now the monster???

Powerful Two-Hander fucked around with this message at 21:30 on Oct 23, 2018

# ? Oct 23, 2018 21:28

gonadic io: Feb 16, 2011; >>=

Powerful Two-Hander posted:

today I managed to work around my block on implementing this feature that I was "sure" I could fit into the data structure I designed by using a combination of exploiting the use of display/editor templates in .net mvc to create a display template that is actually used to passively edit an object that is populated using html returned from an editor in a modal that is bounced via a controller and appended to the original display list.

there's also reflection and accessing of fields to get paths to templates to render default html and oh god what have I done am I now the monster???

admitting it is the first step

# ? Oct 23, 2018 21:39

galenanorth: May 19, 2016

The worst thing about web scraping is having to wait as long as 20 hours to see if a program collected all the records so that the number of records matches up with what it should be. I'm waiting for it to finish so I can see if my fix for the "no address exists for a latitude-longitude coordinate in the ocean" problem works

I guess I could have told the program to collect all the locations in a boxed area bounding the borders of Maine instead of the entire U.S., but it's been long enough I might as well finish it now

galenanorth fucked around with this message at 22:20 on Oct 23, 2018

# ? Oct 23, 2018 22:13

Powerful Two-Hander: Mar 9, 2004; Mods please change my name to "Tooter Skeleton" TIA.

gonadic io posted:

admitting it is the first step

idk if this is terrible or genius....the data structure in the db is a completely flat, single table that represents multiple types of thing, those things can parent one another to create a (theoretically) infinitely deep tree that is renders to the UI by iterating through each one. a "thing" can be any one of a list of predefined elements like a dropdown list, text field etc, so effectively you've got a dynamic form that will save into a common data model.

ive then added the ability for a "thing" to parent a list of things that can be dynamically added to, on page, such that when you submit the entire page back, all your things get iterated over and saved into the model without the MVC binder blowing up. i think.

this has involved about 3 solid days of staring at the screen to get this far and i dont know if im now the idiot or what

# ? Oct 23, 2018 22:23

Powerful Two-Hander: Mar 9, 2004; Mods please change my name to "Tooter Skeleton" TIA.

like, generally the mvc razor views and model binder are really loving good, but once you get into nested items in your models it either works "as is" or.....you end up handcrafting a load of poo poo to create the element structure your model needs so that when you post back it all binds properly and provides the properties you need to do whatever it is you need with it.

it all makes sense but god drat does it make for some complicated view code.

# ? Oct 23, 2018 22:33

Finster Dexter: Oct 20, 2014; Beyond is Finster's mad vision of Earth transformed.

galenanorth posted:

The worst thing about web scraping is

I assure you that's not the worst part of web scraping.

# ? Oct 23, 2018 22:46

Nomnom Cookie: Aug 30, 2009

galenanorth posted:

The worst thing about web scraping is having to wait as long as 20 hours to see if a program collected all the records so that the number of records matches up with what it should be. I'm waiting for it to finish so I can see if my fix for the "no address exists for a latitude-longitude coordinate in the ocean" problem works

I guess I could have told the program to collect all the locations in a boxed area bounding the borders of Maine instead of the entire U.S., but it's been long enough I might as well finish it now

i hope you are rate limiting

# ? Oct 23, 2018 22:48

galenanorth: May 19, 2016

Yes, I am rate limiting at about one second per request. Nearly all the requests are to Google and they have a 40,000 free requests/month cap so I'm hoping the scan completes before reaching that amount. I realized that I should start saving every Google response in a file, so when I repeat the scan to check whether anything has changed, I won't have to send the requests again. That'll allow the scan to progress 100x faster after the initial scan.

# ? Oct 23, 2018 23:55

bob dobbs is dead: Oct 8, 2017; Peeps are idiots, Leslie.; Nap Ghost

next time, do small samples first, do the caching, do cache clearance. dunno which scraping lib you're using

# ? Oct 24, 2018 00:05

my homie dhall: Dec 9, 2010; honey, oh please, it's just a machine

tps: go 1.10�s http client is used in an application I do not control and does not support cidr IP ranges in the no_proxy environment variable, it only supports single IPs

i�m considering passing all 65k IPs in the 192/16 block in as a list for it to work on our internal network :getin:

# ? Oct 24, 2018 02:22

Corla Plankun: May 8, 2007; improve the lives of everyone

galenanorth posted:

The worst thing about web scraping is having to wait as long as 20 hours to see if a program collected all the records so that the number of records matches up with what it should be.

I'm not trying to be a dick (because i have totally done worse stuff than this and had it fail, waisting even more time) but you have just described the exact situation that unit tests are designed to address

# ? Oct 24, 2018 02:58

redleader: Aug 18, 2005; Engage according to operational parameters

Powerful Two-Hander posted:

today I managed to work around my block on implementing this feature that I was "sure" I could fit into the data structure I designed by using a combination of exploiting the use of display/editor templates in .net mvc to create a display template that is actually used to passively edit an object that is populated using html returned from an editor in a modal that is bounced via a controller and appended to the original display list.

there's also reflection and accessing of fields to get paths to templates to render default html and oh god what have I done am I now the monster???

this sounds like a thing you'll regret writing in 3 months

# ? Oct 24, 2018 03:11

Stringent: Dec 22, 2004; by HopperUK

Powerful Two-Hander posted:

the data structure in the db is a completely flat, single table that represents multiple types of thing

i think i spotted your problem op

# ? Oct 24, 2018 03:14

30 TO 50 FERAL HOG: Mar 2, 2005

suffix posted:

it's terrible in python as well, maybe worse since python code tends to be less overload-crazy in general

your non-async version is technically 'better' here because it can run all the someAsyncFunction calls in parallel, which may or may not matter more than code clarity
i would expect something maybe like
code:
var promises = items.map(item => someAsyncFunction(item)
    .then(result => ({a: result, b: item.someProperty})));
return Promise.all(promises);

I like this, Ill have to work it in

# ? Oct 24, 2018 04:55

Powerful Two-Hander: Mar 9, 2004; Mods please change my name to "Tooter Skeleton" TIA.

Stringent posted:

i think i spotted your problem op

yeah that bit wasn't actually me, that's been around for like 10 years, I just added stuff to it. like the rest of this, it's either really good or really bad depending on how you look at it.

there are other tables involved to define the structure but there is one big table that stores the actual data. like if you have a record in tblThings called "butt" it could have child records in tblThings called "fart1, fart2 etc", but the actual values of all your butts and farts would be flattened out into tblvalues.

like I said, it's either really good or really bad depending on how you look at it

# ? Oct 24, 2018 07:51

Corla Plankun: May 8, 2007; improve the lives of everyone

i can't think of any reason why a huge polymorphic table of all the things would be good

# ? Oct 24, 2018 08:20

Sagacity: May 2, 2003; Hopefully my epitaph will be funnier than my custom title.

Powerful Two-Hander posted:

like I said, it's either really good or really bad depending on how you look at it

can you explain how you should be looking at it for this to be considered 'good'?

# ? Oct 24, 2018 08:29

Powerful Two-Hander: Mar 9, 2004; Mods please change my name to "Tooter Skeleton" TIA.

Well, it's more like a key value table. the related 'normal' tables produce a concrete hierarchical model that then has its values "filled" from the shared value table that just stores a reference to the model element and its value

the view then just iterates over the elements in the model rendering a pre defined html chunk for each. so it's a templating engine basically, by defining the structure you want in the DB the ui will render a form that captures all of the data and saves it back into a single dataset.

this has been running for like 10 years, originally with some horrid database to xml to javascript to html and back conversion layer, now thats all done in the mvc framework with templates... these have just got a bit complicated when you start dynamically adding elements to the page because you have to make sure that they conform to what the model binder expects and debugging binding issues loving suuuckkks.

also keeping it all generic requires all sorts of tricks to extract properties from anonymous objects using reflection because although I know that a particular IButt implementation has an additional property of "ButtTitle", as its not part of the interface it isn't accessible directly.

oh yeah did I mention the data is also extracted out into a nested xml structure and distributed as well??

i should probably try and get permission to do a proper write up online or something

# ? Oct 24, 2018 09:24

floatman: Mar 17, 2009

MongoDB

# ? Oct 24, 2018 09:27

Sagacity: May 2, 2003; Hopefully my epitaph will be funnier than my custom title.

Powerful Two-Hander posted:

Well, it's more like a key value table

ah, the Ultimate Extensibility pattern.

it's a nice pattern that disallows:
* database optimizations
* any kind of sharding
* intellisense
* refactoring your code

good

i look forward to your horror novel

# ? Oct 24, 2018 09:50

redleader: Aug 18, 2005; Engage according to operational parameters

eav schemas have their place. yours sounds like a monstrosity, and you might have a case of stockholm syndrome about it

# ? Oct 24, 2018 09:57

Powerful Two-Hander: Mar 9, 2004; Mods please change my name to "Tooter Skeleton" TIA.

Sagacity posted:

ah, the Ultimate Extensibility pattern.

it's a nice pattern that disallows:
* database optimizations
* any kind of sharding
* intellisense
* refactoring your code

good

i look forward to your horror novel

ha, not far off but no xml stored in the DB thank god. if I hadn't inherited this thing I'd have done it different for sure, but compared to the models used by consumers of the data where they have either ridiculously wide tables with column names like fart1 fart2 , fart3 etc. (and yes, this has caused their systems to blow up when fart4 turns up because our model supports lists of any length) or piles and piles of tiny tables that are hard bound to a specific thing.

you can create some absolutely heinous locking issues or mega queries though

floatman posted:

MongoDB

c'mon I have some standards

# ? Oct 24, 2018 10:00

Powerful Two-Hander: Mar 9, 2004; Mods please change my name to "Tooter Skeleton" TIA.

redleader posted:

eav schemas have their place. yours sounds like a monstrosity, and you might have a case of stockholm syndrome about it

probably yeah.

i mean, I've learned a hell of a lot from this thing, hence why if I was doing it from scratch I would do it differently.

Edit : this has just reminded me that we deal with a vendor who provide the same sort of data and their "schema" consists of a 1200 column wide flat file because they don't understand the concept of one to many relationships so have just created columns for as many child things as they think they might ever need.

oh and some of those columns contain comma separated data as well. I'll take our key value table over that any day, at least we can go back and forth between flat and nested structures easily.

Powerful Two-Hander fucked around with this message at 10:07 on Oct 24, 2018

# ? Oct 24, 2018 10:01

Sagacity: May 2, 2003; Hopefully my epitaph will be funnier than my custom title.

Powerful Two-Hander posted:

c'mon I have some standards

i thought i'd never say this but mongodb these days isn't half-bad

i mean, i'd still just use postgres, but it isn't the clown car that it used to be (i guess since the acquisition of wiredtiger)

# ? Oct 24, 2018 11:33

cinci zoo sniper: Mar 14, 2013

xmls in db have their uses :colbert:

# ? Oct 24, 2018 11:37

AggressivelyStupid: Jan 9, 2012

I'm having to mash some xml into and out of a db right now and I gotta say it's not great

# ? Oct 24, 2018 12:25

Powerful Two-Hander: Mar 9, 2004; Mods please change my name to "Tooter Skeleton" TIA.

cinci zoo sniper posted:

xmls in db have their uses

it's OK as long as you never have to query it

like we store blobs to avoid having to generate them from the db every time but we never try to actually query them

the html this thing is generating is loving mind bending

# ? Oct 24, 2018 12:32

Sagacity: May 2, 2003; Hopefully my epitaph will be funnier than my custom title.

'can you please add field X to the orders table'
'uh, sure let me just load/save all the rows of xml we have instead of just running ALTER TABLE like a sane person'

no it's great, xml

# ? Oct 24, 2018 12:41

Adbot: ADBOT LOVES YOU

# ? Nov 17, 2025 10:33

AggressivelyStupid: Jan 9, 2012

I'm at least parsing the xml, using the data to populate a series of tables, then spitting it back out as xml later (unseen by the user) after modifications have been made

I'm not sure if this is any better frankly but here we are

ctps: touching php and trying to keep my eyes closed while I do it

# ? Oct 24, 2018 12:49

The Something Awful Forums > Discussion > Serious Hardware / Software Crap > YOSPOS > awful programming: Somebody set up us the BOM

«‹›1288 »