|
if you have workflow infrastructure why not just deploy them all on their own workflows? it sounds like something a tiny little k8s worker could handle and you could scale however you wanted
|
#
?
Jun 22, 2021 01:39
|
|
|
|
| # ? Apr 25, 2024 18:01 |
|
|
ime optimizing run time inside a python script is a waste. either write it in a faster lang if it matters, or deploy twenty baby worker instances with different params if it doesn't
|
|
#
?
Jun 22, 2021 01:40
|
|
|
a nice thing about doing lots of workers is when one of the steps crashes, which it will eventually because of pythons crap typing, the rest won't even notice
|
|
#
?
Jun 22, 2021 01:42
|
|
|
theres a lot of low hanging fruit you can knock out quickly but yeah you quickly hit diminishing returns trying to make normal python (esp. cpython) go fast in a single process. op you should rewrite it in cython
|
|
#
?
Jun 22, 2021 01:43
|
|
|
shoeberto posted:I'm not really sure what it's doing, it seems to get blocked randomly when there are no resources being used, either on the DB or disk IO. I'm still just playing with it though. We've used this pattern with PHP code before without much trouble so part of me expected it to just work. Could you leverage something like AWS Lambda or Batch/ECS to split up the work more?
|
|
#
?
Jun 22, 2021 02:10
|
|
|
Corla Plankun posted:a nice thing about doing lots of workers is when one of the steps crashes, which it will eventually because of pythons crap typing, the rest won't even notice drat i felt this
|
|
#
?
Jun 22, 2021 02:42
|
|
|
Imagine writing new software in python. lol. also lmao.
|
|
#
?
Jun 22, 2021 04:24
|
|
|
Sapozhnik posted:Imagine writing new software in python. lol. also lmao.
|
|
#
?
Jun 22, 2021 04:28
|
|
|
Sapozhnik posted:Imagine writing new software in python. lol. also lmao.
|
|
#
?
Jun 22, 2021 05:16
|
|
|
Sapozhnik posted:Imagine writing new software in python. lol. also lmao. If you don't write it in python, you have to write it in Java. Java 6.
|
|
#
?
Jun 22, 2021 05:19
|
|
|
MrQueasy posted:If you don't write it in python, you have to write it in Java. can I have mypy? if no, java 6 sounds fine
|
|
#
?
Jun 22, 2021 06:03
|
|
|
pokeyman posted:can I have mypy? No, only jython.
|
|
#
?
Jun 22, 2021 06:03
|
|
|
pokeyman posted:can I have mypy? Mypy and black make Python perfectly acceptable.
|
|
#
?
Jun 22, 2021 06:03
|
|
|
MrQueasy posted:No, only jython. oh well in that case it’s fine, jython is the one python implementation that is actually capable of running two threads at once
|
|
#
?
Jun 22, 2021 07:01
|
|
|
do it in erlang for maximum parallelization. (or just use separate OS processes that's what they're there for lol).
|
|
#
?
Jun 22, 2021 07:29
|
|
|
Soricidus posted:oh well in that case it’s fine, jython is the one python implementation that is actually capable of running two threads at once god drat it... all I've got left to break your enthusiasm is that you have to use Dr. Java as your ide.
|
|
#
?
Jun 22, 2021 07:46
|
|
|
Soricidus posted:oh well in that case it’s fine, jython is the one python implementation that is actually capable of running two threads at once at this stage you might as well write your program in a jvm language
|
|
#
?
Jun 22, 2021 09:23
|
|
|
Boiled Water posted:at this stage you might as well write your program in a jvm language jython is a jvm language tho
|
|
#
?
Jun 22, 2021 09:32
|
|
|
there's a dialect of python that's literally a hobby project to provide a python-like language on a hobby unix-like OS that has better multiprogramming in the base language than actual python does. python multiprogramming is so bad it makes mass vfork/exec look like a modern paradigm
|
|
#
?
Jun 22, 2021 09:34
|
|
|
I just plug a function that does the work and an iterator that spits out jobs for the function into a Pool from python's multiprocessing module, and I haven't needed more than that so far. Multithreading though…just don't. Don't do Python multithreading.
|
|
#
?
Jun 22, 2021 09:45
|
|
|
also python multiprocessing just doesn’t work on lambdas because of the environment constraints so even if you have modern libraries that work with multiprocessing you get a fun surprise when you deploy
|
|
#
?
Jun 22, 2021 09:51
|
|
|
disclaimer: i know nothing about python if python's parallelism is so poo poo, how do python webapps even work (django/flask/etc.)? do they spin up a new interpreter process for every request?
|
|
#
?
Jun 22, 2021 10:30
|
|
|
NihilCredo posted:disclaimer: i know nothing about python flask uses threading, most people use gunicorn to deploy a flask app in prod. here’s how gunicorn works to parallelize: https://docs.gunicorn.org/en/stable/design.html Python is cool and good Destroyenator posted:also python multiprocessing just doesn’t work on lambdas because of the environment constraints so even if you have modern libraries that work with multiprocessing you get a fun surprise when you deploy you mean because lambdas only get one vCPU or some other limitation? you get more than one depending on the memory you allocate to the lambda iirc CarForumPoster fucked around with this message at 10:51 on Jun 22, 2021 |
|
#
?
Jun 22, 2021 10:47
|
|
|
Python is garbage and the gil is terrible.
|
|
#
?
Jun 22, 2021 10:53
|
|
|
CarForumPoster posted:you mean because lambdas only get one vCPU or some other limitation? you get more than one depending on the memory you allocate to the lambda iirc lambdas don’t have access to some shared memory device /dev/shm and multiprocessing needs that to function
|
|
#
?
Jun 22, 2021 11:00
|
|
|
Destroyenator posted:lambdas don’t have access to some shared memory device /dev/shm and multiprocessing needs that to function oh weird. haven’t stepped on that landmine yet so thanks. step functions are what I’ve used to idiot proof concurrency with lambdas, working good so far
|
|
#
?
Jun 22, 2021 11:17
|
|
|
NihilCredo posted:if python's parallelism is so poo poo, how do python webapps even work (django/flask/etc.)? do they spin up a new interpreter process for every request? pretty much, yeah, the optimization is they fork ahead of time lol
|
|
#
?
Jun 22, 2021 12:08
|
|
|
Soricidus posted:that doesn’t take long it’s python, it will.
|
|
#
?
Jun 22, 2021 12:11
|
|
|
Sapozhnik posted:Imagine writing new software in python. lol. also lmao. a real problem is that python is just too drat accessible. there's an interpreter on every platform and its so easy to just spew out across everything. and pretty much anyone with any programming background at all can manage to bang out whatever they need to do with it, even if it's not "good python"
|
|
#
?
Jun 22, 2021 12:15
|
|
|
see also: javascript
|
|
#
?
Jun 22, 2021 12:19
|
|
|
my homie dhall posted:a real problem is that python is just too drat accessible. there's an interpreter on every platform and its so easy to just spew out across everything. and pretty much anyone with any programming background at all can manage to bang out whatever they need to do with it, even if it's not "good python" yes this is problem software is about the code, guys its about linting its about types dont forget execution time
|
|
#
?
Jun 22, 2021 12:33
|
|
|
CarForumPoster posted:yes this is problem
|
|
#
?
Jun 22, 2021 12:46
|
|
|
CarForumPoster posted:yes this is problem what I'm trying to say is that everyone "knows enough to be dangerous" when it comes to python. the existence of code inherently has negative value and python makes it very easy to produce
|
|
#
?
Jun 22, 2021 12:56
|
|
|
use c#, op
|
|
#
?
Jun 22, 2021 13:40
|
|
|
Corla Plankun posted:ime optimizing run time inside a python script is a waste. either write it in a faster lang if it matters, or deploy twenty baby worker instances with different params if it doesn't MrQueasy posted:Could you leverage something like AWS Lambda or Batch/ECS to split up the work more? Sapozhnik posted:Imagine writing new software in python. lol. also lmao.
|
|
#
?
Jun 22, 2021 13:45
|
|
|
my homie dhall posted:a real problem is that python is just too drat accessible. there's an interpreter on every platform and its so easy to just spew out across everything. and pretty much anyone with any programming background at all can manage to bang out whatever they need to do with it, even if it's not "good python" We have very specific needs that Python suits very well, and try to do solid development on top of it, but yeah, I've seen this. I had to hire someone to help us with our ETL stuff and the application process was a shitshow. I know that people have Feelings about homework for job applications, but it was 100% necessary for us to filter out the majority of "data scientists" that don't understand what a data structure is.
|
|
#
?
Jun 22, 2021 13:49
|
|
|
using anything other than ssis for etl is insane.
|
|
#
?
Jun 22, 2021 13:51
|
|
|
Shaggar posted:using anything other than ssis for etl is insane. Does it work with Postgres or Hadoop clusters?
|
|
#
?
Jun 22, 2021 14:05
|
|
|
we have all sorts of etl farming going on in Informatica instances and I swear at least 50% of it must be worthless file transforms and shifting from a to b
|
|
#
?
Jun 22, 2021 14:09
|
|
|
|
| # ? Apr 25, 2024 18:01 |
|
|
ssis will work with any db, you just need an appropriate driver installed. for example: a good use of ssis would be to get all of your poo poo out of postgres and into sql server so you dont need to deal with postgres anymore.
|
|
#
?
Jun 22, 2021 14:10
|
|