|
does "postman: for structured serialization formats" exist? like i want an app like postman where i can point it to my schema defs in whatever supported format and then be able to compile and run requests against a server something like this surely exists in a more unix-y flavor, right?
|
#
?
Aug 10, 2018 17:34
|
|
|
|
| # ? Apr 25, 2024 09:41 |
|
|
MALE SHOEGAZE posted:does "postman: for structured serialization formats" exist? shoegaze make the app I'll start the wiki
|
|
#
?
Aug 10, 2018 17:45
|
|
|
MALE SHOEGAZE posted:does "postman: for structured serialization formats" exist? there are clients for WSDL or swagger/openapi in most mainstream development tools. swagger's is usually built into a website
|
|
#
?
Aug 10, 2018 17:59
|
|
|
Shaggar posted:there are clients for WSDL or swagger/openapi in most mainstream development tools. swagger's is usually built into a website sorry i should have been more specific, i meant binary formats specifically, although it wouldn't need to be restricted as such.
|
|
#
?
Aug 10, 2018 18:02
|
|
|
cinci zoo sniper posted:9 months in, is absolutely baffled by and unable to perform such complex devops manipulations as "ssh into centos box and type htop to see ram use or cat a config file in known path to check the variables" (decade of professional it experience) i'm working with another organization who has their "AWS IT Specialist" as our point of contact, and the guy didn't even know what an AWS access key was, where to configure the AWS client credentials, or how to use the AWS cli to assume a role and grab an auth token. the guy is in charge of their AWS infrastructure. i once watched him struggle with why their server couldn't reach some endpoint we had and its because he decided to put an EC2 instance in a subnet with no routes through a NAT or to the internet. he spent 30 minutes looking at his iptables configuration before finally agreeing that what he was doing wasn't going to work. some people are so loving stupid, i feel like contacting his employer and throwing him under the bus for being so incompetent.
|
|
#
?
Aug 10, 2018 18:03
|
|
|
MALE SHOEGAZE posted:sorry i should have been more specific, i meant binary formats specifically, although it wouldn't need to be restricted as such. antlr maybe? not really my area.
|
|
#
?
Aug 10, 2018 18:04
|
|
|
Shaggar posted:antlr maybe? not really my area. i'm just talking about protobufs/capnproto/whatever. so i spent the last 30 minutes trying to figure out why this thing keeps rejecting all of my requests. json? nope. text? nope. xml? nope. i finally dug deeper into the code: it's an entirely custom bespoke rpc protocol sitting atop a bespoke binary serialization format. as far as i can tell it's being used for communication between a javascript runtime and the rust runtime. it's not talking to some custom c++ interface or whatever. DONT THREAD ON ME fucked around with this message at 18:12 on Aug 10, 2018 |
|
#
?
Aug 10, 2018 18:08
|
|
|
MALE SHOEGAZE posted:sorry i should have been more specific, i meant binary formats specifically, although it wouldn't need to be restricted as such. it'll need to be custom built, unless the format is braindead simple at which point it's not very useful. also where are you sending such binary objects? would the tool need to support RS-232, JTAG, some other transport mechanism, etc? i work with a few satellite comm providers and they provide some tools that lets you define a schema for messages and then generate the binary representation from a UI to send/read for testing. same thing with a lot of the embedded hardware we deal with. the reason why postman works is because http is a standard.
|
|
#
?
Aug 10, 2018 18:11
|
|
|
CRIP EATIN BREAD posted:i'm working with another organization who has their "AWS IT Specialist" as our point of contact, and the guy didn't even know what an AWS access key was, where to configure the AWS client credentials, or how to use the AWS cli to assume a role and grab an auth token. nah, i with no qualms with acknowledge that the guy smart and a rock solid dwh architect/database anything really senior in general. i just wish he'd at least try to step outside of his comfort zone, he's extremely bad at doing that, and so non-windows things are still a black box to him, and so is communication with non-programmers
|
|
#
?
Aug 10, 2018 18:16
|
|
|
CRIP EATIN BREAD posted:it'll need to be custom built, unless the format is braindead simple at which point it's not very useful. also where are you sending such binary objects? would the tool need to support RS-232, JTAG, some other transport mechanism, etc? i'm thinking in the context of webdev where things like grpc and thrift are becoming pretty common. you'd need custom support for each protocol but it would work, i think.
|
|
#
?
Aug 10, 2018 18:16
|
|
|
cinci zoo sniper posted:nah, i with no qualms with acknowledge that the guy smart and a rock solid dwh architect/database anything really senior in general. i just wish he'd at least try to step outside of his comfort zone, he's extremely bad at doing that, and so non-windows things are still a black box to him, and so is communication with non-programmers i meant making GBS threads on the guy i was talking about, not your dude. my blood just started boiling when thinking of an "AWS system admin" who doesn't know AWS at all, besides logging into the GUI and following tutorials for EVERY. SINGLE. THING.
|
|
#
?
Aug 10, 2018 18:23
|
|
|
turns out the big important project ive been working on for the last 2 years is about 30% slower then the thing it is supposed to replace. oops
|
|
#
?
Aug 10, 2018 18:31
|
|
|
Mahatma Goonsay posted:turns out the big important project ive been working on for the last 2 years is about 30% slower then the thing it is supposed to replace. oops how much has your thing been optimized and how much was the other thing optimized? did you use an orm?
|
|
#
?
Aug 10, 2018 18:33
|
|
|
literally the only two guys he communicates with reasonably well are his former coworker/protege (who is in my team), and our dba clown from sysops. speaking of dba - we had an emergency his team sysops could act on so i drop by sysops emergency channel and inform them. a week later no one has still responded, so i drop by once again with "so, any of you bothering to take a look at that" since there's no telling when the prod catches fire again. cue dba (who is unrelated to failure or necessary actions, at all) meltdown (like 800 words of thinly veiled ad hominem) on an empty spot. week later, i put a task in jira (normal priority, no particular deadline - nothing special really) that falls to him. task takes about 2 minutes, 5 minutes if you don't remember what projects you were on in the last 3 weeks. instead he and his jira manager argue with me for about 50 minutes how that is not an approriate task, and then try to forcibly delete it. ended up escalating that through my boss with cto and wasting like 2 hours total time that day (+ involvement of 2 execs for an hour or so combined). fast forward a week or two, my boss is visiting their office. cto rolls up cto: have u seen how cinci bullies my boys for no reason boss: yes he sent me screenshots of that entire conversation cto:  so now we are at the stage where dba is ghosting my jira tasks at all, so there are 5 of them total hanging since late july. 3 are business critical, none of them are being visibly actioned upon, so i need to learn what's happening by accidentally overhearing skype chats between our dwh guy and our dba. at this point im basically waiting for our idiot ceo to announce the reorganisation so we all become part of one org and i can just chew through our idiot dba and get a normal hire instead. i would understand if we has actually like the best dba east of ural monuntains, but as i've likely ranted here before he struggles with such problems as replicating a single mysql database. the solution for literally any problem is to drop replica and create a new one. our dba is literally "so i have this prob~" "REINSTALL WINDOWS" only on enterprise scale. i wonder when will he figure out some solution to the problem where he needs to be reminded the list of replica users each time he "reinstalls" a replica or creates a new one.
|
|
#
?
Aug 10, 2018 18:36
|
|
|
okay that's a page tall tps rant about workplace politics. i should go shoot some fortnites to vent out
|
|
#
?
Aug 10, 2018 18:37
|
|
|
cinci zoo sniper posted:c j/tp s: we have a new analyst joining our team so my dbt pet project will actually be useful since following the best practices has me lay our data model out in small, easily comprehensible and naturally connected chunks. I've never heard of dbt before. From the basic docs you just write some selects and it generates a data model for those selects? But then you can iteratively add select statements as you go? Hmm, sounds like magic (like ORM) and therefore bad, but it sounds potentially better than manually fiddling with db schema or liquibase or whatever.
|
|
#
?
Aug 10, 2018 18:39
|
|
|
*normal hire: this one was my initiative to be hired but i got hard blindsided by the fact that our cto just keeps hiring his personal friends and coworkers at the other jobs he currently works in into any possible position. so he shoehorned another of hsi friends into the job. i guess ill have to prepare for technical interviewing with the next guy if that happens, or at least steal some dba takehome exam from github since im sure this guy couldnt explain how github transaction work even if that would decide if mongol horde horse archers execute him in tundra or not
|
|
#
?
Aug 10, 2018 18:40
|
|
|
Finster Dexter posted:I've never heard of dbt before. From the basic docs you just write some selects and it generates a data model for those selects? But then you can iteratively add select statements as you go? Hmm, sounds like magic (like ORM) and therefore bad, but it sounds potentially better than manually fiddling with db schema or liquibase or whatever. it was mentioned by a fellow thread poster a few pages back first, so i'm toying with it. right now i agree that it is like the T bit in ETL. the actual tech, basically, is Jinja-SQL compiler with some (havent experimented yet) schema testing capacity. since i work with analytics stuff, this is pretty good poo poo for our use case at least since isntead of writing like 2000 sloc sql script with a dozen levels deep subqueries you just lay poo poo down in a logical sequence, write basic validation tests for intermediate steps, and then you can just populate a schema on dwh with it or use it somewhere else. i really like the ability to abstract things like names and values into configs - we have a case where the same database can look differently in two different places, and i can just cross compile the script batches with different targets and provide people with wrokings cripts for everything tl;dr in my current understanding (2 hours of work) its nothing complicated, magial, or grountbreaking. sql with jinja syntax sugar and barebones schema testing framework. my primary usecase for it is a presentable and explainable sql that can be easily replicated afterwards, since schemas are dime a dozen and representing some recursive garbage just in a bunch of straightforward selects is real nice when you need to deal with people who are not necessarily very strong in sql, or familiar with your data model, or have time and energy needed to traverse the atlass shrugged of sql scripts
|
|
#
?
Aug 10, 2018 18:47
|
|
|
cinci zoo sniper posted:it was mentioned by a fellow thread poster a few pages back first, so i'm toying with it. right now i agree that it is like the T bit in ETL. That makes sense. My real problem (I'm just realizing) is that I don't know anything about data warehousing. Explaining dbt as the "T" in ETL clears things up a lot, though.
|
|
#
?
Aug 10, 2018 18:51
|
|
|
i mean it's kinda bad that i am sort-of building some portions of the bi portion of our dwh on my own from grounds up, but that allows me to tailor model to my project and not try to make use of data model that is a partially verified generalisation of our different products. and boy it does need verification since apparently "current expiry date" field in one of primary dimension tables does not include subscription cycles, only the primary purchase???
|
|
#
?
Aug 10, 2018 18:52
|
|
|
i had never heard of dbt either but i looked into it last night. it's not really an ORM as much as it is a way to organize a bunch of select statements via templates and then compile them, with some built in helpers to make iteratively copying data easier to handle in a standard way. seems like its cool.
|
|
#
?
Aug 10, 2018 18:53
|
|
|
Finster Dexter posted:That makes sense. My real problem (I'm just realizing) is that I don't know anything about data warehousing. Explaining dbt as the "T" in ETL clears things up a lot, though. it's an intermediate T, standard E from your current database, and L into a single schema in that same database. that's a single dbt project encapsulated, so it's basically something not too technical people (analysts who write sql) can decently start with to get something going on in your data lake
|
|
#
?
Aug 10, 2018 18:55
|
|
|
Finster Dexter posted:That makes sense. My real problem (I'm just realizing) is that I don't know anything about data warehousing. Explaining dbt as the "T" in ETL clears things up a lot, though. as for dwh stuff, just shoot the questions. i was mostly an armchair architect on our project (nevertheless i was one of the two guys who are actually familiar with our data model, and none of them was the real architect), but i have picked some things up through osmosis i reckon
|
|
#
?
Aug 10, 2018 18:58
|
|
|
cinci zoo sniper posted:it's an intermediate T, standard E from your current database, and L into a single schema in that same database. that's a single dbt project encapsulated, so it's basically something not too technical people (analysts who write sql) can decently start with to get something going on in your data lake Yeah the dbt docs keep referring to "analysts" and I'm like wtf is that
|
|
#
?
Aug 10, 2018 18:59
|
|
|
Finster Dexter posted:Yeah the dbt docs keep referring to "analysts" and I'm like wtf is that usually bi, ba, or ds guys
|
|
#
?
Aug 10, 2018 19:03
|
|
|
so in other words the tech folks at non-it departments
|
|
#
?
Aug 10, 2018 19:04
|
|
|
MALE SHOEGAZE posted:how much has your thing been optimized and how much was the other thing optimized? oh yeah the old one is definitely more optimized. someone in another group ran some tests on it and now my director is all worried. what is funny is that i brought the same thing up a couple months ago and it just got hand waved away. oh well
|
|
#
?
Aug 10, 2018 19:05
|
|
|
Mahatma Goonsay posted:oh yeah the old one is definitely more optimized. someone in another group ran some tests on it and now my director is all worried. what is funny is that i brought the same thing up a couple months ago and it just got hand waved away. oh well I hope they referred to your concerns as "premature optimization"
|
|
#
?
Aug 10, 2018 19:46
|
|
|
Did it take 2 years to realise the performance difference? 
|
|
#
?
Aug 10, 2018 22:25
|
|
|
MrMoo posted:Did it take 2 years to realise the performance difference? well we have been building it while supporting the existing system. really with the new one we are at the point of making it work vs making it fast. of course now someone has seen the word “slower” so it should be an entertaining couple weeks
|
|
#
?
Aug 10, 2018 23:54
|
|
|
cinci zoo sniper posted:okay that's a page tall tps rant about workplace politics. i should go shoot some fortnites to vent out i feel your pain in relation to silos. like, one of our test envs is down because the db is out of space but to get it extended I need a purchase order, two changes and a request ticket because our org is split into silos that are solely driven by reducing any responsibility for anything by pushing it onto everyone else an example conversation is: "I need this thing to be run on a domain controller to set this account up for SSO" "why? We don't understand the request? have you run it on the qa domain first?" "we have no servers on the qa domain, these are standard commands, can you confirm this can be done?" "hmm I don't know, raise a ticket for next week and we'll see" cue a week of ticket chasing for then someone to go "oh this is all wrong, raise a new one.." if they'd just done it in the first place we'd save hours...
|
|
#
?
Aug 11, 2018 00:12
|
|
|
pls don't install stuff on a dc.
|
|
#
?
Aug 11, 2018 00:58
|
|
|
Tankakern posted:if i were your boss and found out you were janitoring white space for a week, i'd fire you when you're spending any serious amount of time in a dynamic garbage language like python, a pep8 linter will save u an untold amount of loving headaches from poo poo that a normal real language compiler should be catching for you anyway
|
|
#
?
Aug 11, 2018 01:08
|
|
|
CRIP EATIN BREAD posted:i'm working with another organization who has their "AWS IT Specialist" as our point of contact, and the guy didn't even know what an AWS access key was, where to configure the AWS client credentials, or how to use the AWS cli to assume a role and grab an auth token. it’s me, even after years of doing the same AWS stuff over and over it’s never any easier the next time. i just do everything in my power to avoid it. minio, linode, etc
|
|
#
?
Aug 11, 2018 02:27
|
|
|
GenJoe posted:when you're spending any serious amount of time in a dynamic garbage language like python, a pep8 linter will save u an untold amount of loving headaches from poo poo that a normal real language compiler should be catching for you anyway
|
|
#
?
Aug 11, 2018 02:31
|
|
|
CRIP EATIN BREAD posted:it'll need to be custom built, unless the format is braindead simple at which point it's not very useful. also where are you sending such binary objects? would the tool need to support RS-232, JTAG, some other transport mechanism, etc? a couple decades ago there was a Mac a tool called General Edit by Quadrivio where you started out with a hex view in one pane and could play around with an almost-C data structure definition in the other and see the data reinterpreted almost as you typed since it was almost-C it was an extremely useful reverse engineering tool and I wish I could convince its author to give me the source or convince somebody to rewrite it it’d be especially awesome if someone built something like that which could also handle packet/stream captures
|
|
#
?
Aug 11, 2018 05:35
|
|
|
wireshark/wireshark with plugins?
|
|
#
?
Aug 11, 2018 05:56
|
|
|
Powerful Two-Hander posted:i feel your pain in relation to silos. like, one of our test envs is down because the db is out of space but to get it extended I need a purchase order, two changes and a request ticket because our org is split into silos that are solely driven by reducing any responsibility for anything by pushing it onto everyone else goddamn that sucks. at least they do something, not like my guys “yeah we saw [but didn’t do anything until you asked] that the database went out of sync”
|
|
#
?
Aug 11, 2018 08:10
|
|
|
can anyone recommend exercises for getting comfortable with hex, binary, and their associated operations? They make me sick to my stomach but I just need practice. I need something like that vim adventure but for binary and hex.
|
|
#
?
Aug 11, 2018 17:15
|
|
|
|
| # ? Apr 25, 2024 09:41 |
|
|
in other news i'm am learning so much from this emulator101 tutorial it's crazy. i keep having big eureka moments and i've only finished the disassembler. everyone should go learn a bit of assembly
|
|
#
?
Aug 11, 2018 17:54
|
|