|
Powerful Two-Hander posted:"what God has INNER JOINED together let no man put asunder" INNER JOIN? pffft sounds like CROSS-JOIN-phobic nonsense
|
# ? Oct 23, 2018 01:40 |
|
|
# ? Apr 18, 2024 06:37 |
|
prisoner of waffles posted:#63 seems like a pretty small ask from someone donating their time to sqlite
|
# ? Oct 23, 2018 02:09 |
|
AggressivelyStupid posted:what flavor of Christianity is this the weird one
|
# ? Oct 23, 2018 03:13 |
|
*nails github pull request to front door*
|
# ? Oct 23, 2018 03:29 |
|
AggressivelyStupid posted:what flavor of Christianity is this Diet TempleOS
|
# ? Oct 23, 2018 03:40 |
|
Toady posted:Diet TempleOS
|
# ? Oct 23, 2018 03:43 |
|
CRIP EATIN BREAD posted:the weird one nah, this is bog-standard christianity
|
# ? Oct 23, 2018 06:18 |
|
pseudorandom name posted:nah, this is bog-standard christianity
|
# ? Oct 23, 2018 06:25 |
|
rip terry, and also the time cube guy. less terrible programmers than the rest of us
|
# ? Oct 23, 2018 08:05 |
|
Trying out a program that tries to systematically scrape from a store locator page that requires address input. I'm trying an approach that involves making requests over a grid of latitude/longitude coordinates, and using Google to automatically convert it to an address. The problem is that some of the coordinates of the grid will be in the ocean and not have addresses, so that scenario will probably require a long workaround. For now, I'm skipping those coordinates and hoping that won't cause the scan to come up short. The workaround is probably going to involve automatically finding the nearest ZIP code from this spreadsheet which has a coordinate associated with each Edit: I found the workaround. It involves splitting the grid unit centered in the ocean over and over until reaching a grid unit of a specified width, keeping the subunits that are on land, and discarding the ocean grid units once they've been split too much so the width is below the minimum galenanorth fucked around with this message at 11:28 on Oct 23, 2018 |
# ? Oct 23, 2018 08:44 |
hahaha. so, we have a sql-backed service talk to an external service via json requests. i just found a table with all shipped json requests. they all include plain text auth in their body, where login is a deveoper’s surname and password is 123<their name>
|
|
# ? Oct 23, 2018 09:18 |
|
today I managed to work around my block on implementing this feature that I was "sure" I could fit into the data structure I designed by using a combination of exploiting the use of display/editor templates in .net mvc to create a display template that is actually used to passively edit an object that is populated using html returned from an editor in a modal that is bounced via a controller and appended to the original display list. there's also reflection and accessing of fields to get paths to templates to render default html and oh god what have I done am I now the monster??? Powerful Two-Hander fucked around with this message at 22:30 on Oct 23, 2018 |
# ? Oct 23, 2018 22:28 |
|
Powerful Two-Hander posted:today I managed to work around my block on implementing this feature that I was "sure" I could fit into the data structure I designed by using a combination of exploiting the use of display/editor templates in .net mvc to create a display template that is actually used to passively edit an object that is populated using html returned from an editor in a modal that is bounced via a controller and appended to the original display list. admitting it is the first step
|
# ? Oct 23, 2018 22:39 |
|
The worst thing about web scraping is having to wait as long as 20 hours to see if a program collected all the records so that the number of records matches up with what it should be. I'm waiting for it to finish so I can see if my fix for the "no address exists for a latitude-longitude coordinate in the ocean" problem works I guess I could have told the program to collect all the locations in a boxed area bounding the borders of Maine instead of the entire U.S., but it's been long enough I might as well finish it now galenanorth fucked around with this message at 23:20 on Oct 23, 2018 |
# ? Oct 23, 2018 23:13 |
|
gonadic io posted:admitting it is the first step idk if this is terrible or genius....the data structure in the db is a completely flat, single table that represents multiple types of thing, those things can parent one another to create a (theoretically) infinitely deep tree that is renders to the UI by iterating through each one. a "thing" can be any one of a list of predefined elements like a dropdown list, text field etc, so effectively you've got a dynamic form that will save into a common data model. ive then added the ability for a "thing" to parent a list of things that can be dynamically added to, on page, such that when you submit the entire page back, all your things get iterated over and saved into the model without the MVC binder blowing up. i think. this has involved about 3 solid days of staring at the screen to get this far and i dont know if im now the idiot or what
|
# ? Oct 23, 2018 23:23 |
|
like, generally the mvc razor views and model binder are really loving good, but once you get into nested items in your models it either works "as is" or.....you end up handcrafting a load of poo poo to create the element structure your model needs so that when you post back it all binds properly and provides the properties you need to do whatever it is you need with it. it all makes sense but god drat does it make for some complicated view code.
|
# ? Oct 23, 2018 23:33 |
|
galenanorth posted:The worst thing about web scraping is I assure you that's not the worst part of web scraping.
|
# ? Oct 23, 2018 23:46 |
|
galenanorth posted:The worst thing about web scraping is having to wait as long as 20 hours to see if a program collected all the records so that the number of records matches up with what it should be. I'm waiting for it to finish so I can see if my fix for the "no address exists for a latitude-longitude coordinate in the ocean" problem works i hope you are rate limiting
|
# ? Oct 23, 2018 23:48 |
|
Yes, I am rate limiting at about one second per request. Nearly all the requests are to Google and they have a 40,000 free requests/month cap so I'm hoping the scan completes before reaching that amount. I realized that I should start saving every Google response in a file, so when I repeat the scan to check whether anything has changed, I won't have to send the requests again. That'll allow the scan to progress 100x faster after the initial scan.
|
# ? Oct 24, 2018 00:55 |
|
next time, do small samples first, do the caching, do cache clearance. dunno which scraping lib you're using
|
# ? Oct 24, 2018 01:05 |
|
tps: go 1.10’s http client is used in an application I do not control and does not support cidr IP ranges in the no_proxy environment variable, it only supports single IPs i’m considering passing all 65k IPs in the 192/16 block in as a list for it to work on our internal network
|
# ? Oct 24, 2018 03:22 |
|
galenanorth posted:The worst thing about web scraping is having to wait as long as 20 hours to see if a program collected all the records so that the number of records matches up with what it should be. I'm not trying to be a dick (because i have totally done worse stuff than this and had it fail, waisting even more time) but you have just described the exact situation that unit tests are designed to address
|
# ? Oct 24, 2018 03:58 |
|
Powerful Two-Hander posted:today I managed to work around my block on implementing this feature that I was "sure" I could fit into the data structure I designed by using a combination of exploiting the use of display/editor templates in .net mvc to create a display template that is actually used to passively edit an object that is populated using html returned from an editor in a modal that is bounced via a controller and appended to the original display list. this sounds like a thing you'll regret writing in 3 months
|
# ? Oct 24, 2018 04:11 |
|
Powerful Two-Hander posted:the data structure in the db is a completely flat, single table that represents multiple types of thing i think i spotted your problem op
|
# ? Oct 24, 2018 04:14 |
|
suffix posted:it's terrible in python as well, maybe worse since python code tends to be less overload-crazy in general I like this, Ill have to work it in
|
# ? Oct 24, 2018 05:55 |
|
Stringent posted:i think i spotted your problem op yeah that bit wasn't actually me, that's been around for like 10 years, I just added stuff to it. like the rest of this, it's either really good or really bad depending on how you look at it. there are other tables involved to define the structure but there is one big table that stores the actual data. like if you have a record in tblThings called "butt" it could have child records in tblThings called "fart1, fart2 etc", but the actual values of all your butts and farts would be flattened out into tblvalues. like I said, it's either really good or really bad depending on how you look at it
|
# ? Oct 24, 2018 08:51 |
|
i can't think of any reason why a huge polymorphic table of all the things would be good
|
# ? Oct 24, 2018 09:20 |
|
Powerful Two-Hander posted:like I said, it's either really good or really bad depending on how you look at it
|
# ? Oct 24, 2018 09:29 |
|
Well, it's more like a key value table. the related 'normal' tables produce a concrete hierarchical model that then has its values "filled" from the shared value table that just stores a reference to the model element and its value the view then just iterates over the elements in the model rendering a pre defined html chunk for each. so it's a templating engine basically, by defining the structure you want in the DB the ui will render a form that captures all of the data and saves it back into a single dataset. this has been running for like 10 years, originally with some horrid database to xml to javascript to html and back conversion layer, now thats all done in the mvc framework with templates... these have just got a bit complicated when you start dynamically adding elements to the page because you have to make sure that they conform to what the model binder expects and debugging binding issues loving suuuckkks. also keeping it all generic requires all sorts of tricks to extract properties from anonymous objects using reflection because although I know that a particular IButt implementation has an additional property of "ButtTitle", as its not part of the interface it isn't accessible directly. oh yeah did I mention the data is also extracted out into a nested xml structure and distributed as well?? i should probably try and get permission to do a proper write up online or something
|
# ? Oct 24, 2018 10:24 |
|
MongoDB
|
# ? Oct 24, 2018 10:27 |
|
Powerful Two-Hander posted:Well, it's more like a key value table it's a nice pattern that disallows: * database optimizations * any kind of sharding * intellisense * refactoring your code good i look forward to your horror novel
|
# ? Oct 24, 2018 10:50 |
|
eav schemas have their place. yours sounds like a monstrosity, and you might have a case of stockholm syndrome about it
|
# ? Oct 24, 2018 10:57 |
|
Sagacity posted:ah, the Ultimate Extensibility pattern. ha, not far off but no xml stored in the DB thank god. if I hadn't inherited this thing I'd have done it different for sure, but compared to the models used by consumers of the data where they have either ridiculously wide tables with column names like fart1 fart2 , fart3 etc. (and yes, this has caused their systems to blow up when fart4 turns up because our model supports lists of any length) or piles and piles of tiny tables that are hard bound to a specific thing. you can create some absolutely heinous locking issues or mega queries though floatman posted:MongoDB c'mon I have some standards
|
# ? Oct 24, 2018 11:00 |
|
redleader posted:eav schemas have their place. yours sounds like a monstrosity, and you might have a case of stockholm syndrome about it probably yeah. i mean, I've learned a hell of a lot from this thing, hence why if I was doing it from scratch I would do it differently. Edit : this has just reminded me that we deal with a vendor who provide the same sort of data and their "schema" consists of a 1200 column wide flat file because they don't understand the concept of one to many relationships so have just created columns for as many child things as they think they might ever need. oh and some of those columns contain comma separated data as well. I'll take our key value table over that any day, at least we can go back and forth between flat and nested structures easily. Powerful Two-Hander fucked around with this message at 11:07 on Oct 24, 2018 |
# ? Oct 24, 2018 11:01 |
|
Powerful Two-Hander posted:c'mon I have some standards i mean, i'd still just use postgres, but it isn't the clown car that it used to be (i guess since the acquisition of wiredtiger)
|
# ? Oct 24, 2018 12:33 |
xmls in db have their uses
|
|
# ? Oct 24, 2018 12:37 |
|
I'm having to mash some xml into and out of a db right now and I gotta say it's not great
|
# ? Oct 24, 2018 13:25 |
|
cinci zoo sniper posted:xmls in db have their uses it's OK as long as you never have to query it like we store blobs to avoid having to generate them from the db every time but we never try to actually query them the html this thing is generating is loving mind bending
|
# ? Oct 24, 2018 13:32 |
|
'can you please add field X to the orders table' 'uh, sure let me just load/save all the rows of xml we have instead of just running ALTER TABLE like a sane person' no it's great, xml
|
# ? Oct 24, 2018 13:41 |
|
|
# ? Apr 18, 2024 06:37 |
|
I'm at least parsing the xml, using the data to populate a series of tables, then spitting it back out as xml later (unseen by the user) after modifications have been made I'm not sure if this is any better frankly but here we are ctps: touching php and trying to keep my eyes closed while I do it
|
# ? Oct 24, 2018 13:49 |