Python information and short questions megathread.

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›2 »

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

Is there any really good standard library reference for Python? Compared to Ruby, the official documentation absolutely sucks, and it's really frustrating me because I absolutely adore the Pylons framework and then doing more menial Python stuff in the framework is comparatively painful.

# ¿ Nov 12, 2007 13:50

Adbot: ADBOT LOVES YOU

# ¿ Apr 25, 2024 06:02

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

Has anyone used the Django ORM bindings to MongoDB? How well do they work? I need to write a frontend to the network monitoring system I'm building.

# ¿ May 26, 2011 04:26

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

Is anyone aware of a pre-existing synchronized priority queue implementation for gevent? I'm writing a Nagios->Graphite perfdata forwarder and I want to make sure that new files coming in on the inotify greenlet are given priority over old files lingering in the spool (queued on a different greenlet).

Edit: Never mind, it seems that gevent has an undocumented priority queue that I found by going through queue.py :downs:

Vulture Culture fucked around with this message at 19:27 on Oct 30, 2012

# ¿ Oct 30, 2012 15:01

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

Captain Capacitor posted:

Something like this?

Yeah, found that already (see edit). Thanks!

# ¿ Oct 30, 2012 20:45

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

I just finished my first legitimate (non-toy) Python project, Metricinga, a daemon that forwards Nagios/Icinga performance data to Graphite. Would anyone mind taking a minute to look over my code and tear it apart so I can write better code?

# ¿ Nov 1, 2012 21:41

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

Captain Capacitor posted:

I personally would find it a bit annoying if I had to consult a few different places for logging output (sys.stderr output versus whatever is given). Just a personal nitpick.

Also it seems that there's no place to specify log files anyway.

Can you clarify what you mean? Logs go to syslog, except when run in debug mode (app doesn't daemonize and writes output to stderr).

# ¿ Nov 2, 2012 01:26

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

Titan Coeus posted:

Python code:
# this
try:
    pf = file(self.pidfile, 'r')
    pid = int(pf.read().strip())    
    pf.close()
except IOError:
    pid = None
# should be
try:
    with file(self.pidfile, 'r') as pf:
        pid = int(pf.read().strip())    
except IOError:
    pid = None
Python code:
# this
while 1:
# should be
while True:
Python code:
# this
file(self.pidfile,'w+').write("%s\n" % pid)
# should be
with file(self.pidfile,'w+') as pid_file:
    pid_file.write("%s\n" % pid)
In stop() you call str(err) twice, the second time it will always already be a string.

And this regarding your use of file vs. open: http://stackoverflow.com/questions/112970/python-when-to-use-file-vs-open

Thanks. This was public domain daemon boilerplate that I copied from somewhere else because python-daemon has issues with gevent that even gevent.reinit() following the fork was unable to fix. This works okay until I can wrap my head around those problems, though.

I guess I should be happy that many of your concerns focused on someone else's code, though. I'll still fix it up to be less... 2002.

Titan Coeus posted:

Python code:

# this
if daemonize is True:
# should be
if daemonize:

I wasn't really sure about this as I was writing it, but you're right.

Titan Coeus posted:

Python code:

# this
(opts, args) = parser.parse_args()
# should be
opts, args = parser.parse_args()

How come? (I ask this legitimately, as someone who probably wrote that as idiomatic Perl rather than idiomatic Python.)

Titan Coeus posted:

I'm not sure about send_metric(self, metric). If you have no internet connection, this is going to loop indefinitely. You should probably set some limit (20?) on the number of failed attempts.

Absolutely not. The entire purpose of this tool is to daemonize and spool data over that socket. If the Carbon service on the other end goes down, this should parse and queue data until it comes back. Since the socket I/O is all evented, this won't even block file parsing while the socket is down.

I do need a more graceful way of deleting the files only after the metrics are sent. Propagating control messages down several layers in gevent (or anything) is frustrating.

Titan Coeus posted:

In run(), your finally block has a "shutdown successfully" message. If joinall throws some other non-expected error, that message will be false.

This is a very good catch, thanks.

Titan Coeus posted:

See the following regarding string formatting: http://docs.python.org/2/tutorial/inputoutput.html#old-string-formatting

Will do, thanks.

Vulture Culture fucked around with this message at 04:00 on Nov 2, 2012

# ¿ Nov 2, 2012 03:58

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

BigRedDot posted:

A few weeks ago my company put on PyData NYC, a conference dedicated to data analytics with python. Authors and contributors of numpy, scipy, pandas, pytables, ipython, and other projects all gave great talks to several hundred attendees. Today all the talks were made available on Vimeo, for anyone interested!

http://vimeo.com/channels/pydata/videos/sort:preset/format:detail

Oh, this is loving perfect for what I'm working on right now. Thanks very much!

# ¿ Nov 10, 2012 19:30

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

BigRedDot posted:

A few weeks ago my company put on PyData NYC, a conference dedicated to data analytics with python. Authors and contributors of numpy, scipy, pandas, pytables, ipython, and other projects all gave great talks to several hundred attendees. Today all the talks were made available on Vimeo, for anyone interested!

http://vimeo.com/channels/pydata/videos/sort:preset/format:detail

I know I thanked you already for posting this, but I just got around to watching the AppNexus talk and stumbled upon so many things immediately relevant to my current time-series work that I needed to thank you again for posting this (and your company for hosting the conference). So thanks again!

# ¿ Nov 11, 2012 21:27

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

stubblyhead posted:

I'm in the process of learning python, and I'm having trouble with array slice syntax. So foo[1:4] will return indices 1-3, which is really counter-intuitive to me. What's the reasoning for the second number being the first index not included instead of the last index that is? The latter seems more logical to me, but I'm sure there are things I haven't taken into consideration about why that would be a bad idea.

For consistency, it uses the same start and end values as the range() function. The single-argument form of range(n), according to the docs, generates a list of exactly n values.

This is one of those things that constantly frustrates the poo poo out of people who regularly switch between Ruby and Python for different projects.

# ¿ Nov 20, 2012 04:42

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

Is there a thin database abstraction layer for Python that basically just takes user-supplied connection strings using whatever database driver a user feels like without having to do a bunch of manual legwork to import and instantiate some arbitrary class name in my own code? I don't want an ORM.

# ¿ Nov 28, 2012 05:14

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

Suspicious Dish posted:

SQLAlchemy

Argh! I glossed over it because I was explicitly not looking for a large ORM, but it seems like SQLAlchemy Core is exactly what I'm looking for. Thanks.

# ¿ Nov 28, 2012 05:28

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

duck monster posted:

I was always fond of ADOdb on PHP

So this is the python implementation:

http://phplens.com/lens/adodb/adodb-py-docs.htm

e: Whats your issue with using an ORM by the way? Performance, "impedence", or learning curve?

This script literally needs to run one database query. But that one query might be run against any of a variety of databases the user runs, so being able to dump a connection string in a config file is a nice thing to have.

# ¿ Nov 30, 2012 19:37

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

I need a tool for yet another project that, given a reply email containing a quoted part, will extract out the portion of the message that's just the reply. Does anyone know anything that would fit the bill?

Edit: Nice! https://github.com/zapier/email-reply-parser

Vulture Culture fucked around with this message at 17:14 on Dec 1, 2012

# ¿ Dec 1, 2012 16:45

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

Does logging call flush() on registered log handlers when a program exits, or is that something that needs to be done manually in the application?

# ¿ Dec 5, 2012 07:25

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

Delicate Stranger posted:

Sounds like the consensus is to learn a framework like Django or Flask and then, piece by piece, learn the magic it's doing under the hood. As necessary.

My impression had been that web frameworks were more for reducing basic, highly common stuff like creating a blog to extremely few lines, rather than helping build arbitrary, random web apps. I thought for something like that I'd have to learn the plumbing, since my recipe app idea wasn't a cms or comment box or something common. I'd been googling down blind alleys with search terms like cgi etc.

I wasn't really interested in learning PHP if I didn't have to after skimming http://me.veekun.com/blog/2012/04/09/php-a-fractal-of-bad-design/, but I was hoping that it would lead to more pro-python responses by mentioning it, haha.

There's no general rule with web frameworks, because different ones are completely different have completely different goals. Some, like web.py, Sinatra, or Flask, are very bare-bones and aim to stay out of your way as much as possible and let you code things however you like. Others, like Django or Rails, are considered "opinionated" and offer tighter integration with a full stack of pre-fabricated components and libraries at the cost of some flexibility or ease-of-use if you need to do weird or non-standard stuff. More others, like Pyramid, fall somewhere in the middle, providing a "full-stack framework" that's a thin shim around a pile of pluggable components. Yet more others, like Tornado, are purpose-built to scale to hundreds or thousands of concurrent requests and have a completely different architecture and programming style than anything else in order to accommodate that.

PHP was invented so HTML authors could easily add backend functionality to their pages using a style that was convenient and comfortable. This is fine for Joe's Homepage (the "H" in "PHP/FI"), but this approach obviously started to have problems as more entire websites started to be built as dynamic applications.

Vulture Culture fucked around with this message at 20:17 on Dec 8, 2012

# ¿ Dec 8, 2012 20:15

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

Caddrel posted:

I'm not very proficient with python yet, but of the the free books out there Dive Into Python is helping me get up to speed quickly. I think it's best when you are already familiar with similar data structures and OO in other languages, and just need to know how python does it.

Dive Into Python is very old even for a Python 2 reference, and a lot of the conventions in that book (e.g. string formatting) have been deprecated for quite awhile, and it doesn't make any mention of important language features like decorators.

It's an okay thing to teach you the absolute basics, but make sure to read lots of Python code actually written in the past several years if you're going to take it seriously.

# ¿ Dec 10, 2012 01:25

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

MeramJert posted:

I kind of like project Euler when I'm learning the syntax of a new language. I know all the solutions to the first 10 problems or so, which is all I generally do. I do think it can be helpful to type them in anyway in a new language to get a feel for it.

This is how I learned Erlang and re-learned Python, actually.

# ¿ Dec 11, 2012 01:18

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

duck monster posted:

Use this thing: http://kivy.org/

It makes incredibly cool UIs.

Thanks for this, it looks perfect for the PyKaraoke megafork I'm working on.

# ¿ Dec 15, 2012 21:13

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

Hanpan posted:

I have some new scripts that need to run periodically, once an hour, once a day etc. Supervisor seems to be really backwards at starting process at intervals so I've been looking into alternatives. Obviously, cron would be ideal for this but does require a crontab, something I've tried to avoid because it makes the app less encapsulated.

Just have the app drop something in cron.hourly or cron.d or whatever. This is a problem that's been solved with package management for decades.

# ¿ Dec 23, 2012 04:18

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

I'm becoming more interested in TDD, but I'm largely clueless because most of what I use Python for is async with gevent. Anyone have any recommendations for starting points with testing concurrent clusterfuck applications?

# ¿ Apr 21, 2013 00:16

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

Dominoes posted:

The tutorials I learned from would use statements like "for parameter in parameters" or "for day in range". I don't like them because I have the phrase "parameter(s)" and "day(s)" several other places in the program, and want to make it easy to distinguish.

This means two things:

Your functions are too long and mix too many concerns, so you're not able to properly leverage variable scope
Your variable names are not descriptive in the first place

Both of these things also make it harder to unit test your code, which starts the death spiral of "this code is unreadable and I don't want to touch it because I might break something."

Dominoes posted:

The short names also make the code easier to read, although harder for someone else to interpret. I'm open to changing and suggestions on alternate ways of doing this. "loc1" was a typo.

Also harder for you to interpret once you take a few weeks off from this code, which honestly is the bigger concern to most developers.

# ¿ Apr 22, 2013 01:02

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

Suspicious Dish posted:

Don't do that. Your package's payload should have all the stuff it needs. Offline installation should be supported.

Sometimes this isn't possible due to licensing issues (think of the video driver or MS corefont installers in most Linux distros)

For basically all other instances though, agreed.

# ¿ Apr 22, 2013 21:48

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

Rohaq posted:

So I'm writing a Django app that needs to talk to our RequestTracker system: No problem, the python-rtkit module does a pretty good job of talking to the RT API.

Except I found that it uses urllib2 and the BasicAuthenticator (the only method I have of connecting to our current RT setup), and when there's a a bad username/password, urllib2 goes into a recursive loop:
code:
Exception Type: 	RuntimeError
Exception Value: 	maximum recursion depth exceeded
Exception Location: 	/usr/lib/python2.6/urllib2.py in do_open, line 1118
I mean, I don't plan on having bad login credentials in production, but on the offchance the service account is disabled some day, I'd like to return a useful error message regarding the fact.

Can anyone suggest a good way to catch the bad login the first time around, rather than letting urllib2 go into its loop?

What you're doing shouldn't be necessary and suggests you have a series of bad redirects on the server side that's causing this loop in the first place. Fire up Wireshark and see what's going on.

Vulture Culture fucked around with this message at 16:17 on Apr 23, 2013

# ¿ Apr 23, 2013 16:15

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

ARACHNOTRON posted:

That is actually literally the case. RabbitMQ/pika message queue consumption :/

Before anyone else chimes in I'd like to point out that while there is an AMQP library for Twisted, its performance is really, really bad.

# ¿ Apr 24, 2013 02:42

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

QuarkJets posted:

When you're done with an object that takes up a lot of memory (for instance, a 1k by 1k by 1k numpy array), is it considered better practice to delete it with del or just leave it for garbage collection?

In terms of GC, del doesn't do anything but remove a reference to the object. If it's about to go out of scope already anyway, it's pointless to explicitly call it unless you're deliberately breaking a cycle somewhere.

Objects in Python don't have any defined destructor semantics and inherit them from the runtime, which makes object destruction sort of an onerous thing. In CPython objects are reference-counted and __del__ is called as soon as the last reference goes out of scope, but in IronPython and Jython they pick up the finalizer semantics from the CLR and JRE mark-and-sweep GCs, respectively.

# ¿ Apr 28, 2013 21:20

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

Suspicious Dish posted:

CPython has a cycle collector too which means that destruction isn't deterministic if you have a reference cycle. I do not advocate using __del__ for anything as it can gently caress up your program in various, subtle ways.

Cycle collecting itself isn't even deterministic if I recall correctly, because if any of the objects has a __del__ defined, the runtime goes "I don't know the right order to clean these up in, so gently caress it" and just leaves them around in perpetuity.

# ¿ Apr 28, 2013 21:29

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

Can someone point me towards a larger Python project that they think has really good unit tests?

# ¿ May 9, 2013 13:14

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

From a unit testing perspective, each file should contain all the imports it needs to test the classes defined in that file. Anything else is going to be a trainwreck once you start actually writing test cases.

# ¿ May 13, 2013 16:19

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

Hammerite posted:

Is it considered poor form to use the fact that loop variables are still set after the loop?

I have to repeatedly loop over elements of a dictionary and unset an element at each iteration, but can't unset in the loop because that's not allowed. So I came up with
Python code:
while len(myDict):
    for k in myDict:
        if condition(myDict[k]):
            do_things()
            break
    else:
        raise MyCustomException('badly formed dict!')
    del myDict[k]
Here I use the fact that k is still set after the loop is broken out of.

Why are you using an inner loop when each run through the loop will execute 0 or 1 times? :raise:

Python code:

for k in myDict.keys():
    if condition(myDict[k]):
        do_things()
    else:
        raise MyCustomException('badly formed dict!')

    del myDict[k]

# ¿ May 14, 2013 21:42

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

Hammerite posted:

At each step in the process, at least one of the elements of the dictionary should satisfy the if clause (otherwise the dictionary shall be considered badly-formed by definition). However, I don't know which one(s).

This has a serious code smell. Can you go into a little more detail about what's actually going on here, how that dict gets populated, etc.?

# ¿ May 15, 2013 15:45

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

Hammerite posted:

Is it good style to use assert statements as a form of documentation? i.e. "I know that at this point in the function, condition X must hold because it follows from something I did a few lines ago. But the reader trying to understand the function might benefit from having this pointed out to them, so put in an 'assert X'."

Documentation is likely a better form of documentation when the code isn't clear enough on its own.

# ¿ Jun 5, 2013 12:00

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

QuarkJets posted:

That's usually due to all of the additional layers of security and monitoring software that are installed; most corporate/government computers are just going to be Windows workstations and won't be running custom code at all

Except whatever the Chinese government put there :laugh:

# ¿ Jun 9, 2013 21:13

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

Pollyanna posted:

How come the get_timeframe() function doesn't know what ask_timeframe() is? I defined it before I called get_timeframe().

Scope only exists within a file; move your function into another module and import it everywhere you want to call it from.

# ¿ Sep 25, 2013 03:01

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

a lovely poster posted:

PySide is my personal recommendation

TkInter is another popular one, never personally used it

https://wiki.python.org/moin/GuiProgramming has a decent list, you're probably going to have to filter through a lot of trash though

Kivy is also interesting.

# ¿ Sep 26, 2013 01:09

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

Hammerite posted:

The Game of Life occurs on an infinite plane, so given that you don't have enough memory to keep the whole thing in memory, how do you get around that limitation? I appreciate that there are other ways to manage things than a simple two-dimensional array of on/off values, for example you could instead have a list of cells that are "on", but (1) the number of cells that are "on" can still grow without bound, (2) your implementation is still incomplete on some level if it doesn't support an arbitrary, potentially infinite set of "on" cells in the initial state.

You're possibly looking at a sparse matrix, which is the data structure typically used to implement spreadsheets. Since your growth is unbounded, though, this will eventually consume all the memory in your system beyond a certain number of iterations.

Vulture Culture fucked around with this message at 20:20 on Oct 1, 2013

# ¿ Oct 1, 2013 20:16

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

And this is a pretty awful algorithm to begin with. The Sieve of Eratosthenes could be implemented in a few lines more, and the Sieve of Atkin in a handful more than that. Each has way less runtime complexity.

# ¿ Oct 20, 2013 20:36

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

Most of the dynamic content (polls, visualizations, etc.) on the Washington Post's website is also Django. They've got a pile of extensions on their GitHub account.

# ¿ Nov 4, 2013 06:37

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

Pollyanna posted:

So, you know those web games where you click a button, a number goes up, and you feel a fleeting sense of accomplishment before the high fades away and you need a new fix? (Don't worry, I'm just using this as a little project.) Is that something I can program in Python and attach to Django or Flask, or does it rely on stuff like Javascript instead?

Those games wouldn't be very exciting if you clicked something and it triggered a complete page refresh like browsing around a website. Luckily, the JavaScript involved is very easy.

# ¿ Nov 4, 2013 19:28

Adbot: ADBOT LOVES YOU

# ¿ Apr 25, 2024 06:02

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

Pollyanna posted:

You have got to be kidding me

So I upgraded to Mavericks last night. Now, whenever I want to run anything in Python, this happens:
code:
Rebeccas-MacBook-Pro:~ rebecca$ python
Python 2.7.5 (v2.7.5:ab05e7dd2788, May 13 2013, 13:18:45) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 2 + 2
4
>>> 2 + 2
Segmentation fault: 11
Inputting any more than one command causes it to crash. Did upgrading my OS seriously just break my Python installation?

The computer also says X11 is no longer on my system which is certainly bullshit.

edit: Oh, good. There's a new version out that fixes this. Never mind!

Yeah, basically the issue was that the version of readline shipped with older versions of OS X had a nasty bug that the Python shell developers needed to work around. The version in Mavericks has the bug fixed, but Python is still trying to use the workaround, so it explodes with a segfault as soon as readline has history.

# ¿ Nov 4, 2013 23:40

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›2 »