|
Kevin Mitnick P.E. posted:SQS is super robust. job has been relying on SQS for about five years now and had 0 problems. billions of messages per month, 1000x traffic spikes, once 300 misconfigured workers short polling in a loop, never had a problem sqs is ok as long as you understand it's not really a queue, because receiving a message doesn't dequeue it
|
#
?
Dec 22, 2018 02:26
|
|
|
|
| # ? Oct 11, 2024 05:36 |
|
|
DELETE CASCADE posted:sqs is ok as long as you understand it's not really a queue, because receiving a message doesn't dequeue it ugh why would you want that
|
|
#
?
Dec 22, 2018 02:27
|
|
|
why not delete that db yet please
|
|
#
?
Dec 22, 2018 02:45
|
|
|
Lmao just wait till the people at work hear that MSSQL actually has queues. It's ridiculous that the "chief architect" who's claim to fame was being a database wizzzzzzard choose to implement some idiot rubbish obscure custom bespoke queue system instead of just using the built in idiot rubbish obscure queue system.
|
|
#
?
Dec 22, 2018 09:10
|
|
|
just use rabbitmq op
|
|
#
?
Dec 22, 2018 10:31
|
|
|
not gonna lie, i picked the first queue that came to mind + i haven't given a single thought to the use case under discussion gently caress db queues though. we have a few queue or queue-like tables, and i don't trust anyone to have implemented them correctly
|
|
#
?
Dec 22, 2018 10:34
|
|
|
We use rabbitmq at work and it's been solid af.
|
|
#
?
Dec 22, 2018 10:42
|
|
|
Shaggar posted:its easier to just handle control and statusing in a table vs an opaque message queue and ive never seen a purpose built message queue that isn't fragile as hell and relies on the ability to dump everything in the queue and restart when things go wrong. i have literally never had to do that but since i am updating an operational table with statuses when creating messages i could rebuild if i have to whether I use sql server broker queue or a custom queue table. we’ve had queues survive loss of power situations without losing any messages and just pick up where they left off when restored vOv mixing state with event messages is not a good design no matter how you slice it, and if your queue doesn’t drain (as in op example I was ultimately addressing) you’re gonna eventually hit some hard limits more discussion here if you really want to do your own queue table (in particular Remus rusani’s comments): https://dba.stackexchange.com/questions/20399/fifo-queue-table-for-multiple-workers-in-sql-server re:rabbitmq, we looked at rabbitmq but my boss loves mssql and we wanted a solution where a dequeue operation would write to a table within a database txn so never lose a message to an error and always have something to go back and report on and reprocess
|
|
#
?
Dec 22, 2018 11:49
|
|
|
Stringent posted:We use rabbitmq at work and it's been solid af. Same, kafka is fine too but for slightly different use cases
|
|
#
?
Dec 22, 2018 13:37
|
|
|
This all seems like a slightly roundabout and unnecessarily complex way of summoning tef
|
|
#
?
Dec 22, 2018 14:57
|
|
|
Phobeste posted:This all seems like a slightly roundabout and unnecessarily complex way of summoning tef lol
|
|
#
?
Dec 22, 2018 17:00
|
|
|
Phobeste posted:This all seems like a slightly roundabout and unnecessarily complex way of summoning tef lol before he arrives to school us all, i should reiterate that most of the time you think you need a queue, you don't, and it won't help you deal with a true overload scenario because you still have a choke point somewhere i.e. instead of being overwhelmed at your api maybe you'll be overwhelmed at your database i pretty much only use them for pub/sub activation, smoothing out processing message volume peaks, and creating sinks that can hold messages when a target system goes down e: i guess i could have just wrote "i use them as buffers" 
Cold on a Cob fucked around with this message at 17:20 on Dec 22, 2018 |
|
#
?
Dec 22, 2018 17:09
|
|
|
in addition to that mononcqc article i posted earlier, just read what tef has to say about decomposing a monolith, it's good stuff https://programmingisterrible.com/post/162346490883/how-do-you-cut-a-monolith-in-half
|
|
#
?
Dec 22, 2018 17:19
|
|
|
i used to be of the opinion that queues between services let you reasonably distribute work to n number of consumers of m types and decouple services but all you end up doing is adding an unbounded, ephemeral transport between services which means you have to continually monitor scaling and queue depth to identify bottlenecks and can't actually apply back-pressure from the consumer contrast this with http/rpc/whatever calls to a load balancer with each services maintaining internal queues for re-work in case the downstream service applies back-pressure and imo it's pretty clear that all a queue does is provide non-durable storage outside of individual transaction handling at the operational expense of maintaining an opaque system that's really easy to tip over (i'm looking at you rabbitmq)
|
|
#
?
Dec 22, 2018 17:24
|
|
|
Eh I don't know what to think already. I do agree with the points you said about what the purpose of queues should be, but nowadays after $JOB I'm just so jaded. Like I'm starting to think that it's all fine and dandy to keep in mind "best practices" or theoretical good design and all that, but sometimes an anti pattern is necessary because if you leave the clowns to their own devices, you get poo poo like recursive CTE SQL queries for bad queue implementation or wrapping csvs in Json to be sent as XML strings. It's not even good enough to reach the standard of bad. And most Devs here have Stockholm and they actually think everything is fine. To put it this way I would have rather had a badly managed rabbitmq implementation to fix and debug rather than some idiot bespoke MSSQL server queue that one guy was so loving proud he did but didn't stay in the company long enough to see first hand the destruction it wrought
|
|
#
?
Dec 22, 2018 18:05
|
|
|
Blinkz0rz posted:i used to be of the opinion that queues between services let you reasonably distribute work to n number of consumers of m types and decouple services but all you end up doing is adding an unbounded, ephemeral transport between services which means you have to continually monitor scaling and queue depth to identify bottlenecks and can't actually apply back-pressure from the consumer fwiw you can use database transactions with a sql server service broker queue i.e. receive and push to a table in a transaction (though you have to be careful to not throw exceptions because a poison message can turn off your queue) they're pretty durable and not opaque, they use tables internally, you can query them without receiving, and you can set them to retain received messages as well if you need to i.e. debugging scenarios they're are a lot of annoyances for using them too; they're not as turn-key as using something like rabbitmq nor as popular so documentation is scarce*, so you'll have to bootstrap a bunch of poo poo yourself if you use them *if anyone reading this does decide to try service broker, Remus Rusanu is probably the best resource out there
|
|
#
?
Dec 22, 2018 18:09
|
|
|
floatman posted:To put it this way I would have rather had a badly managed rabbitmq implementation to fix and debug rather than some idiot bespoke MSSQL server queue that one guy was so loving proud he did but didn't stay in the company long enough to see first hand the destruction it wrought yeah if you were using rabbitmq at least then you can pay a consultant to fix your poo poo instead of, say, ripping it all out and starting from scratch collect that salary but always be looking for something better
|
|
#
?
Dec 22, 2018 18:12
|
|
|
floatman posted:Eh I don't know what to think already. I do agree with the points you said about what the purpose of queues should be, but nowadays after $JOB I'm just so jaded. at this point i'm not even sure recursive anything should be allowed in sql. Just write out what you're attempting so the next person in line doesn't have to guess and guess
|
|
#
?
Dec 22, 2018 18:18
|
|
|
ctes are awesome and it makes me sad that people use them in obtuse and confusing ways because they can really clean up a code base if used correctly i've only used them for recursion a few times, but they work really well non-recursively too and i pretty much always use them instead of sub-queries now because i can chain them together nicely: with foo as ( /* query */ ), bar1 as ( /* query using foo */ ), bar2 as ( /* different query using foo or maybe bar1*/ ) select * from bar1 join bar2 on /* join etc */
|
|
#
?
Dec 22, 2018 18:32
|
|
|
oh no the queue for my irrelevant fart app is growing whatever will i do
|
|
#
?
Dec 22, 2018 21:03
|
|
|
redleader posted:oh no the queue for my irrelevant fart app is growing whatever will i do Send "PULL" to localhost:79
|
|
#
?
Dec 23, 2018 01:44
|
|
|
redleader posted:oh no the queue for my irrelevant fart app is growing whatever will i do 
|
|
#
?
Dec 23, 2018 01:49
|
|
|
floatman posted:Eh I don't know what to think already. I do agree with the points you said about what the purpose of queues should be, but nowadays after $JOB I'm just so jaded. theres actually zero difference between good & bad things. you imbecile. you loving moron
|
|
#
?
Dec 23, 2018 02:22
|
|
|
CPColin posted:Send "PULL" to localhost:79 
|
|
#
?
Dec 23, 2018 02:23
|
|
|
Cold on a Cob posted:ctes are awesome and it makes me sad that people use them in obtuse and confusing ways because they can really clean up a code base if used correctly i like ctes too but i learned a couple months ago that they're dumber than subselects because postgres doesnt use indexes in the same way or something when its a cte
|
|
#
?
Dec 23, 2018 06:47
|
|
|
Kevin Mitnick P.E. posted:theres actually zero difference between good & bad things. you imbecile. you loving moron I'm sorry did I do something that offended you personally?
|
|
#
?
Dec 23, 2018 08:27
|
|
|
Boiled Water posted:at this point i'm not even sure recursive anything should be allowed in sql. Just write out what you're attempting so the next person in line doesn't have to guess and guess We actually paid a database consultant to come in to fix db problems. He wanted more context on the recursive CTE so we dug up the pull request which had the original writers plans and intents. His comments was, "wow what he says he sets out to do is entirely different from what was done."
|
|
#
?
Dec 23, 2018 08:29
|
|
|
floatman posted:I'm sorry did I do something that offended you personally? its a tweet
|
|
#
?
Dec 23, 2018 09:40
|
|
|
CPColin posted:Send "PULL" to localhost:79 nice
|
|
#
?
Dec 23, 2018 10:00
|
|
|
Kevin Mitnick P.E. posted:its a tweet Ah okay I'll go check it out for context thanks
|
|
#
?
Dec 23, 2018 10:25
|
|
|
Kevin Mitnick P.E. posted:better than handling trillions of large rows? NihilCredo posted:i think he's a perfect fit for https://www.timescale.com/how-it-works cinci zoo sniper posted:if it's a non-nullable, monotonically increasing or decreasing field that will be used in every single query, then in postgres you can use last_switched as partitioning key or brin index key. that should make this table decently well manageable at least until low-mid billions of records Just because I know sometimes people don't give any updates when they get good advice but I went ahead and added Timescaledb as another datastore option and initial testing is that it's real good. Thanks!
|
|
#
?
Dec 23, 2018 11:13
|
|
|
abigserve posted:Just because I know sometimes people don't give any updates when they get good advice but I went ahead and added Timescaledb as another datastore option and initial testing is that it's real good. yeah I'm gonna throw this at some guys at work that were looking at using elastic, if their main query mode is time based this might be a better solution. mind you our infrastructure/platform team have a severe case of "not invented here" syndrome and usually refuse point blank to consider anything that wasn't their idea so we'll see.
|
|
#
?
Dec 23, 2018 12:49
|
|
|
Corla Plankun posted:i like ctes too but i learned a couple months ago that they're dumber than subselects because postgres doesnt use indexes in the same way or something when its a cte i was actually talking from an mssqlserver perspective but good to know
|
|
#
?
Dec 23, 2018 15:26
|
|
|
Cold on a Cob posted:i was actually talking from an mssqlserver perspective but good to know should be more or less universal behaviour, cte return temporary views and if you want indexes there you should make normal indexed views for your ctes. one lovely hack you can do here is write your ctes with appropriate order by/group by clauses to help the execution of other queries, but that’s a really hacky strategy
|
|
#
?
Dec 23, 2018 15:59
|
|
|
folks I think writing verilog like it’s C might not be the best approach
|
|
#
?
Dec 23, 2018 23:53
|
|
|
Luigi Thirty posted:folks I think writing verilog like it’s C might not be the best approach  *posts "why is my verilog for loop not working" on stack overflow*
|
|
#
?
Dec 24, 2018 00:15
|
|
|
gonadic io posted:
no the problem is my cpu consists of a giant bunch of if thens I’m fixing lol turns out that turns into a bunch of priority encoded flip flops rather than muxes which is what I wanted
|
|
#
?
Dec 24, 2018 00:18
|
|
|
Powerful Two-Hander posted:yeah I'm gonna throw this at some guys at work that were looking at using elastic, if their main query mode is time based this might be a better solution. i would fight tooth and nail at work to use elasticsearch over timescaledb unless elasticsearch was totally unfit. that's down to us having decent experience running ES and 0 with timescaledb or even postgresql. learning how to operate a new db and dealing with its poo poo when you put a nontrivial workload on it (i.e. more than fits on a dev's laptop) is extremely expensive in man-hours does your infra team get a lot of suggestions to implement DBs that reached 1.0 less than two months ago?
|
|
#
?
Dec 24, 2018 04:02
|
|
|
just use elasticsearch unless you already know 100% that it won't fit your usecase already.
|
|
#
?
Dec 24, 2018 06:08
|
|
|
|
| # ? Oct 11, 2024 05:36 |
|
|
graylog and elasticsearch are amazing and can be setup pretty drat quickly. I love them.
|
|
#
?
Dec 24, 2018 06:22
|
|
