Python information and short questions megathread.

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›484 »

Habnabit: Dec 30, 2007; lift your skinny fists like
antennas in germany.

Marzzle posted:

Try this:
$ open -e `which easy_install`

Then edit the first line so that it says:
#!/usr/bin/python
instead of whatever else was there.

# ? Mar 3, 2008 08:41

Adbot: ADBOT LOVES YOU

# ? Apr 27, 2024 23:29

JoeNotCharles: Mar 3, 2005; Yet beyond each tree there are only more trees.

Even better is "#!/usr/bin/env python", which uses the "env" program to find out where Python is actually installed. That way if it'll work whether it's installed in /usr/bin or in .../Frameworks/MacPython or whatever that was.

(Type "/usr/bin/env python" in the Terminal to see if it'll work.)

# ? Mar 3, 2008 16:53

SkyLander: Dec 27, 2007

I have a couple Python questions if anyone could help.

I have to write a command line interface type program, basically what are considered simple things and I decided to mess around with Python to do it instead of C++. So far things seem a lot simpler and working in python is a lot easier than in C++. Just takes some getting used to in that everything is a little more loose.

Anyway basically I have to do basic commands and one of them is to do a dir command which just lists what is in a directory. So I found listdirectory() in the OS module. So I basically just do input from the command line into a string and I just put that into listdirectory. Now does that just create a list of strings? Or how do I then output that? Sorry if that seems confusing.

Basically what I have is

def directory(command)
directoryOut = os.listdirectory(command)
print directoryOut

And that doesn't do anything so I assume I am missing something.

Thanks in advanced.

# ? Mar 3, 2008 20:49

SmirkingJack: Nov 27, 2002

SkyLander posted:

I have a couple Python questions if anyone could help.

I have to write a command line interface type program, basically what are considered simple things and I decided to mess around with Python to do it instead of C++. So far things seem a lot simpler and working in python is a lot easier than in C++. Just takes some getting used to in that everything is a little more loose.

Anyway basically I have to do basic commands and one of them is to do a dir command which just lists what is in a directory. So I found listdirectory() in the OS module. So I basically just do input from the command line into a string and I just put that into listdirectory. Now does that just create a list of strings? Or how do I then output that? Sorry if that seems confusing.

Basically what I have is

def directory(command)
directoryOut = os.listdirectory(command)
print directoryOut

And that doesn't do anything so I assume I am missing something.

Thanks in advanced.

First I would enter the interactive editor and type "import os" followed by "dir(os)" to print a list of methods in the os module.

# ? Mar 3, 2008 21:01

nonathlon: Jul 9, 2004; And yet, somehow, now it's my fault ...

SkyLander posted:

Anyway basically I have to do basic commands and one of them is to do a dir command which just lists what is in a directory. So I found listdirectory() in the OS module. So I basically just do input from the command line into a string and I just put that into listdirectory. Now does that just create a list of strings? Or how do I then output that? Sorry if that seems confusing.

You're on the right track. The call is 'os.listdir (path)', which returns a set of strings. Apart from looking at the standard library docs (which are all online), you might want to install ipython, which lets you introspect Python objects. For example

>>> import os
>>> os??
Type: module
Base Class: <type 'module'>
String Form: <module 'os' from ...

>>> os.listdir??
Type: builtin_function_or_method
Base Class: <type 'builtin_function_or_method'>
String Form: <built-in function listdir>
Namespace: Interactive
Docstring [source file open failed]:
listdir(path) -> list_of_strings

Return a list containing the names of the entries in the directory.

...

# ? Mar 3, 2008 21:30

SkyLander: Dec 27, 2007

So I would have to do a for loop to output it? I tried that and it gave me a compiler error.

Did

for x in directoryOut
print x

Or I'm still a little fuzzy on how the for loop works in Python.

# ? Mar 3, 2008 21:50

Habnabit: Dec 30, 2007; lift your skinny fists like
antennas in germany.

SkyLander posted:

So I would have to do a for loop to output it? I tried that and it gave me a compiler error.

Did

for x in directoryOut
print x

Or I'm still a little fuzzy on how the for loop works in Python.

I'm not sure if you're using colons in your code or not because your def statement didn't have a colon either. For loops in python are always for-each loops; they take a sequence and iterate over it. Also, your directoryOut function isn't returning a value, so "for x in directoryOut(...):" wouldn't work anyway.

Python can print any object, though. I'm not sure why your code isn't working. Try this in the interactive interpreter:

code:

import os
os.listdir('.')

It might be listdirectory for you since that's what you said before; I'm on the MacOS and it's listdir in my os module.

# ? Mar 3, 2008 22:03

SmirkingJack: Nov 27, 2002

SkyLander posted:

...
Basically what I have is

def directory(command)
directoryOut = os.listdirectory(command)
print directoryOut

And that doesn't do anything so I assume I am missing something.

Thanks in advanced.

Are you calling your function? What does the following do?

code:

def directory(command):
  directoryOut = os.listdir(command)
  print directoryOut

directory("c:")

Replace "c:" with "/" if you are not running Windows.

SmirkingJack fucked around with this message at 23:08 on Mar 3, 2008

# ? Mar 3, 2008 23:04

Allie: Jan 17, 2004

SmirkingJack posted:

First I would enter the interactive editor and type "import os" followed by "dir(os)" to print a list of methods in the os module.

That's silly, it'll a list a whole bunch of useless poo poo he doesn't need or care about and won't tell him what any of it means. help(os) is much more helpful (that's why it's called help, after all).

If you want to print out a list of strings, use ''.join():

code:

>>> import os
>>> print '\n'.join(os.listdir('/usr/local'))
bin
info
lib
man
sbin

That joins each string in the list that os.listdir returns by a newline.

# ? Mar 3, 2008 23:05

SmirkingJack: Nov 27, 2002

Milde posted:

That's silly, it'll a list a whole bunch of useless poo poo he doesn't need or care about and won't tell him what any of it means. help(os) is much more helpful (that's why it's called help, after all).
...

The idea was to quickly help him realize that he was not calling the right method.

# ? Mar 3, 2008 23:10

SkyLander: Dec 27, 2007

Using the .join() function worked. Thanks guys. And yeah I was lazy at work and forgot the :. Also figured out I wasn't parsing it correctly. Thanks again for all the help.

SkyLander fucked around with this message at 01:55 on Mar 4, 2008

# ? Mar 4, 2008 01:46

deimos: Nov 30, 2006; Forget it man this bat is whack, it's got poobrain!

So with Sun hiring Leung and Wierzbicki that makes what 3 major companies (Google, Microsoft, Sun) with vested interests in Python? :woop:

# ? Mar 4, 2008 21:47

porkface: Dec 29, 2000

deimos posted:

So with Sun hiring Leung and Wierzbicki that makes what 3 major companies (Google, Microsoft, Sun) with vested interests in Python?

Yahoo!, NASA

What I love most about Python's current growth rate is that it's almost entirely due to things Python has been doing all along, not just some new flashy framework or idea.

# ? Mar 4, 2008 22:18

dorkanoid: Dec 21, 2004

porkface posted:

Yahoo!, NASA

What I love most about Python's current growth rate is that it's almost entirely due to things Python has been doing all along, not just some new flashy framework or idea.

Indeed; I remember some map editor for Quake implemented scripting in Python back...in the old days (fake edit: it was called Quake Army Knife (ninja edit to my fake edit: 1996? drat

)), and I'm sure tons of places used it at that time, but not as "visible" as it is today

# ? Mar 4, 2008 22:55

deimos: Nov 30, 2006; Forget it man this bat is whack, it's got poobrain!

porkface posted:

Yahoo!, NASA

What I love most about Python's current growth rate is that it's almost entirely due to things Python has been doing all along, not just some new flashy framework or idea.

Err yes Yahoo! too, forgot. Who of the big contributors to python development does NASA payroll?

# ? Mar 4, 2008 23:29

gabensraum: Sep 16, 2003; LOAD "NICE!",8,1

In early 1996 I was sixteen and browsing a local BBS (at the time the Internet was a long-distance call from me) where I found a folder called Python. I downloaded a few files because I thought it was Monty Python.

# ? Mar 4, 2008 23:42

porkface: Dec 29, 2000

deimos posted:

Err yes Yahoo! too, forgot. Who of the big contributors to python development does NASA payroll?

Sorry, by "vested interest" I figured using it for robots and various other systems counted.

# ? Mar 5, 2008 00:59

FrontLine: Sep 17, 2003; I gave Mr. Adequate hot, steamy man love and all he bought me was a custom title!

Does anybody know how/where to get the Python language pack for VS2005?

# ? Mar 6, 2008 09:41

deimos: Nov 30, 2006; Forget it man this bat is whack, it's got poobrain!

FrontLine posted:

Does anybody know how/where to get the Python language pack for VS2005?

AFAIK you can't integrate it, instead you get to use: IronPython Studio

# ? Mar 6, 2008 21:23

FrontLine: Sep 17, 2003; I gave Mr. Adequate hot, steamy man love and all he bought me was a custom title!

deimos posted:

AFAIK you can't integrate it, instead you get to use: IronPython Studio

Shock and awe!

Wikipedia lied to me.

Wikipedia, the lying encyclopedia posted:

Support for other languages such as F#, Python, and Ruby among others has been made available via language services which are to be installed separately.

# ? Mar 6, 2008 23:22

gabensraum: Sep 16, 2003; LOAD "NICE!",8,1

For people like me who go weeks merely checking their subscribed threads, ATLbeer started a Django thread.

# ? Mar 7, 2008 15:52

VirtuaSpy: Jul 26, 2002; OMG WTF FIREEEE!!

For my continuation of newbie learning, I wrote a script to take input from a CSV file and replace strings in a template file while making new output files. What I don't like is how every method I tried to read in the template file (plaintext) as the variable 'template' gets returned and nulled out when I call it outside of the loop. Thus my solution was to read the template file each time in the for loop. Along with that problem, I'd like commentary on what else I could improve to make the code more proper (the imports are from me playing with different methods):

code:

import csv, sys, os, re

input = csv.reader(open("input.csv","r"))

counter = 1

for i, j, k in input:
	template = open("fdsfiletoedit.fds", "r")
	output = open("fdsout" + str(counter) + ".fds", "w")
	for lines in template:
		output.write(lines.replace("IREP,JREP,KREP",(str(i) + "," + str(j) + "," + str(k))))
	counter += 1

CSV file:

1,2,3
2,3,4
3,4,5
4,5,6

Pertinent line in template file:

&MESH IJK=IREP,JREP,KREP, XB=-0.15,0.15,-0.15,0.15,0.0,0.4 / Mesh is just to make FDS run

Thanks.

VirtuaSpy fucked around with this message at 09:27 on Mar 10, 2008

# ? Mar 10, 2008 09:24

Scaevolus: Apr 16, 2007

code:

import csv, sys, os, re

input = csv.reader(open("input.csv","r"))

counter = 1

template = open("fdsfiletoedit.fds", "r").readlines()

for i, j, k in input:
	output = open("fdsout" + str(counter) + ".fds", "w")
	for lines in template:
		output.write(lines.replace("IREP,JREP,KREP",(str(i) + "," + str(j) + "," + str(k))))
        output.close()
	counter += 1

you probably don't need output.close(), but it's a good habit regardless.

# ? Mar 10, 2008 12:09

JoeNotCharles: Mar 3, 2005; Yet beyond each tree there are only more trees.

VirtuaSpy posted:

For my continuation of newbie learning, I wrote a script to take input from a CSV file and replace strings in a template file while making new output files. What I don't like is how every method I tried to read in the template file (plaintext) as the variable 'template' gets returned and nulled out when I call it outside of the loop. Thus my solution was to read the template file each time in the for loop.

I have no idea what that means.

However:

To avoid reading the template over and over again, keep it in memory. (If the template is huge, this may not be a good idea, but in that case you'd have to get into some serious rewriting anyway to be able to do anything with it.)

code:

import csv, sys, os, re

# read the template first and keep its contents around
[b]template = open("fdsfiletoedit.fds", "r")
lines = template.readlines()[/b]

# now process the input file
input = csv.reader(open("input.csv","r"))

counter = 1

for i, j, k in input:
	output = open("fdsout" + str(counter) + ".fds", "w")
	[b]for line in lines:[/b]
		output.write([b]line[/b].replace("IREP,JREP,KREP",(str(i) + "," + str(j) + "," + str(k))))
	counter += 1

# ? Mar 10, 2008 15:51

VirtuaSpy: Jul 26, 2002; OMG WTF FIREEEE!!

JoeNotCharles posted:

However:

To avoid reading the template over and over again, keep it in memory. (If the template is huge, this may not be a good idea, but in that case you'd have to get into some serious rewriting anyway to be able to do anything with it.)

That's what I meant, I just couldn't get the syntax just right as you did.

Thanks for the help.

# ? Mar 10, 2008 17:47

Lancer383: Aug 15, 2007; Say what you want about the tenants of national socialism, Dude, at least it was an ethos.

Does anyone here have experiencing working with the PyExcelerator python module or any other modules geared towards writing directly to Excel xls files?

I have created a program to read batch of Oracle queries in off of a file, grab all the results, and then write to sheets in a workbook, and it is working well with only one issue: the module (at least as I'm using it at the moment) holds all of the Excel sheets (sometimes over 100 Excel sheets, but it's also an issue if I'm just writing 20 sheets) in memory throughout the program, and then finally let's go at the end when I write to the workbook.

For small jobs this isn't an issue, but some of our queries return 40,000 to 60,000 rows, and the program's RAM usage gets up to 200-300 megabytes. We would eventually like this program to be accessible through a scheduling tool we have, so I need to find a way to get the program to use less RAM -- not a problem if I need to switch over to a different Excel module.

Thanks in advance!

# ? Mar 11, 2008 14:34

oRenj9: Aug 3, 2004; Who loves oRenj soda?!?; College Slice

JoeNotCharles posted:

To avoid reading the template over and over again, keep it in memory. (If the template is huge, this may not be a good idea, but in that case you'd have to get into some serious rewriting anyway to be able to do anything with it.)

It's honestly not that hard to read in a file in 4KB buffers. The only thing you have to do is split the buffer on each newline char then save the last element in the array because it's not complete. When you read the next buffer in, split it on newlines then just put the overflow from the last read at the beginning of the first element in the array.

This is around a hundred times faster than reading it in line by line when you're working with a large file.

You can read it in all at once using read lines, but I'm not sure if that as slow as doing file.readline().

# ? Mar 11, 2008 15:23

tef: May 30, 2004; -> some l-system crap ->

Lancer383 posted:

Does anyone here have experiencing working with the PyExcelerator python module or any other modules geared towards writing directly to Excel xls files?

I have used OLE with Excel to write and modify excel files. Run screaming Away.

Instead, have you considered writing to CSV files and then opening them in excel later?

# ? Mar 11, 2008 16:58

Lancer383: Aug 15, 2007; Say what you want about the tenants of national socialism, Dude, at least it was an ethos.

tef posted:

I have used OLE with Excel to write and modify excel files. Run screaming Away.

Instead, have you considered writing to CSV files and then opening them in excel later?

Nice. Yeah, this has been a nightmare on all parts dealing with Excel, but a great learning experience otherwise.

I have considered CSV, but the problem is that we're looking to bundle all of the results up in one container, so to speak. It is much easier to send someone one Excel workbook with 150 worksheets than 150 CSV files. I put in internal hyperlinking (and god what a pain that was with pyExcelerator) so that all the sheets would have links back to a summary page, and the summary page would have links back to all the results sheets.

I would be more than happy to have a non-Excel solution for this, but would need something that is able to package all of our query results up, and, optimally, be able to be opened by Excel / provide the same usability that Excel provides.

# ? Mar 11, 2008 17:20

duck monster: Dec 15, 2004

I can't help thinking that if you need 150 worksheets, you havent really put enough thought into how the data is being represented. Could this be easier done with a disposable database, maybe in an MS Access container so the client can open it up and noodle around with some macroed up forms and poo poo?

Anyway, if its the way its gotta be, wheres the leak? Pythons reference counted, so if you get circular references or whatever, you can lose memory with stuff if your not careful. Manually delete objects when your finished and see how that works out. If its at the Excel side, you might be poo poo out of luck, but if its at the database side, I'd suggest getting all funky with cursors and break the processing up a bit.

duck monster fucked around with this message at 20:24 on Mar 11, 2008

# ? Mar 11, 2008 20:22

Lancer383: Aug 15, 2007; Say what you want about the tenants of national socialism, Dude, at least it was an ethos.

duck monster posted:

I can't help thinking that if you need 150 worksheets, you havent really put enough thought into how the data is being represented. Could this be easier done with a disposable database, maybe in an MS Access container so the client can open it up and noodle around with some macroed up forms and poo poo?

Anyway, if its the way its gotta be, wheres the leak? Pythons reference counted, so if you get circular references or whatever, you can lose memory with stuff if your not careful. Manually delete objects when your finished and see how that works out. If its at the Excel side, you might be poo poo out of luck, but if its at the database side, I'd suggest getting all funky with cursors and break the processing up a bit.

Yeah, I didn't become part of the process until the endless list of queries had already been built -- really, it should be 15 queries pulling data off a matrix of codes rather than everything being hard-coded in 10 different ways.

Either way, there's still a good amount of data being written, and by doing a little bit of debugging, the big issue is that all of the Excel data is held in memory until the workbook is saved. Also, at least with this module, saving must be done all in one fell swoop, so all the worksheets have to stay in memory until the entire workbook is written.

Lastly, this is a tool that we want to be able to use across the business, and the skillset isn't there for being able to use Access both for our employees and for the clients'. We are just looking for an easy way to run data audits, both in large batches and scheduled to run daily, weekly, etc. If anyone has had a similar business need and found a different solution, I'd love to hear about it.

Thanks!

# ? Mar 11, 2008 21:10

tef: May 30, 2004; -> some l-system crap ->

Lancer383 posted:

I have considered CSV, but the problem is that we're looking to bundle all of the results up in one container, so to speak.

Instead of returning an excel sheet or something that can be opened directly within excel, could you distribute (in advance) an excel spreadsheet with a macro (and a button) for loading and creating the final excel version, and generate some sort of csv from the server end.

# ? Mar 12, 2008 02:25

TK422: Dec 22, 2007; I hate being right

Is there any easy way to exec interactive system commands from a python script?

Some time ago I coded a script to provide a reverse shell to access firewalled machines (for emergency purposes).

I got stuck being able to only execute non-interactive commands using commands.getoutput function. For the purpose of the script this was enough to be fairly useful, but now I was wondering if some enhancement could be made to provide basic interactive support (sudo commands for example).

# ? Mar 12, 2008 18:51

bitprophet: Jul 22, 2004; Taco Defender

You probably want the subprocess module, check docs.python.org for your Python version and you'll see it in the listing (the docs.python.org/lib/ page + your browser's quicksearch function = very easy access to almost all of the Python docs).

It's a newer replacement for older stuff such as the popen module, which is what used to provide the functionality you're looking for - the ability to define the stdin/stdout/stderr pipes to the command in question.

You can use it like this (going from memory here, don't copy/paste this, look at the docs):

code:

from subprocess import Popen, PIPE

command = Popen("/path/to/binary -args",use_shell=True,stdin=PIPE,stdout=PIPE)
command.stdin.write("text to input to the command")
print command.stdout.read() # prints the output of the command

That's equivalent to, in the shell,

code:

$ echo "text to input to the command" | /path/to/binary -args

# ? Mar 12, 2008 19:10

JoeNotCharles: Mar 3, 2005; Yet beyond each tree there are only more trees.

bitprophet posted:

You probably want the subprocess module, check docs.python.org for your Python version and you'll see it in the listing (the docs.python.org/lib/ page + your browser's quicksearch function = very easy access to almost all of the Python docs).

Subprocess is awesome - it saved my rear end just yesterday. I was spawning a DOS program that insisted on asking "Do you really want to modify this file (Y/N)?" for every file it updated, on thousands of files. I thought it was going to be a huge pain in the rear end making it send a Y over and over, but it turns out "Subprocess(...).communicate("Y")" was all I needed. If it didn't modify a file, it throws the Y away.

# ? Mar 13, 2008 02:23

bitprophet: Jul 22, 2004; Taco Defender

JoeNotCharles posted:

Subprocess is awesome - it saved my rear end just yesterday. I was spawning a DOS program that insisted on asking "Do you really want to modify this file (Y/N)?" for every file it updated, on thousands of files. I thought it was going to be a huge pain in the rear end making it send a Y over and over, but it turns out "Subprocess(...).communicate("Y")" was all I needed. If it didn't modify a file, it throws the Y away.

Unix has had a "yes" binary for probably a very long time; with no arguments it simply sends 'y' + a carriage return, over and over, to stdout. Very useful for piping to commands like the one you were using, which are retarded enough to ask Y/N type questions without accepting a --force sort of option.

You can also give "yes" any string as an argument and it'll repeat that instead. In the more juvenile-minded this typically causes behavior such as gleefully typing "yes gently caress" upon discovering the program for the first time.

# ? Mar 13, 2008 04:04

JoeNotCharles: Mar 3, 2005; Yet beyond each tree there are only more trees.

bitprophet posted:

Unix has had a "yes" binary for probably a very long time; with no arguments it simply sends 'y' + a carriage return, over and over, to stdout. Very useful for piping to commands like the one you were using, which are retarded enough to ask Y/N type questions without accepting a --force sort of option.

You can also give "yes" any string as an argument and it'll repeat that instead. In the more juvenile-minded this typically causes behavior such as gleefully typing "yes gently caress" upon discovering the program for the first time.

Well, yes, that's exactly what I would have used on Unix - and exactly the type of thing that makes me say, "Oh, gently caress, Windows has no such useful automation features, this is going to be a pain in the rear end." ("Port yes to Windows" was what I thought I was gonna have to do, which isn't terrible but more work than should be necessary to run a shell script; subprocess gave me a one-liner, which is about the amount of effort I was looking for.)

# ? Mar 13, 2008 04:38

VirtuaSpy: Jul 26, 2002; OMG WTF FIREEEE!!

Running OS X Leopard and I'm trying to install pygame to follow a tutorial. But when I use 'easy_install pygame' I get the following: "error: Setup script exited with Pygame requires PyObjC version 1.1 or above."

After much Googling, I tried:
$ easy_install pyobjc - error: Could not find suitable distribution for Requirement.parse('pyobjc')

Also, on the PyObjC page I see: "On Mac OS X 10.5 (Leopard) the stable release is PyObjC 2.0, which is included in the OS distribution."

What am I doing wrong here? Is there some path that I need to add somewhere so that easy_install sees the installed PyObjC?

# ? Mar 14, 2008 12:10

Lancer383: Aug 15, 2007; Say what you want about the tenants of national socialism, Dude, at least it was an ethos.

Hey guys, any experience with Portable Python (http://www.portablepython.com/)? Are there any drawbacks / instability with it? I want to end up having this on a USB, and sync all my installed modules, scripts I'm working on, etc. with some sort of USB drive syncing/backup program.

# ? Mar 14, 2008 17:14

Adbot: ADBOT LOVES YOU

# ? Apr 27, 2024 23:29

hey mom its 420: May 12, 2007

Here's a cool thing I've just found out while doing some stuff for school. The zip function works as its own inverse. So you can do stuff like this

code:

>>> a = [1,2,3,4,5]
>>> b = ['a','b','c','d','e']
>>> zipped = zip(a, b)
>>> zipped
[(1, 'a'), (2, 'b'), (3, 'c'), (4, 'd'), (5, 'e')]
>>> c, d = zip(*zipped) #this is the magic
>>> c
(1, 2, 3, 4, 5)
>>> d
('a', 'b', 'c', 'd', 'e')

I was kind of racking my brain about why this works actually, but then I thought about how it's best to imagine zip works. If you have two collections, we'll name them collections 1 and 2

code:

1: A B C D E
2: a b c d e

and when you zip that, zip just flips it about, so you end up with collections 1, 2, 3, 4 and 5

code:

1: A a
2: B b
3: C c 
4: D d
5: E e

So zipping that up again gives you back the original. So zip is its own inverse. f(x) = f^-1(x)

hey mom its 420 fucked around with this message at 17:45 on Mar 14, 2008

# ? Mar 14, 2008 17:41

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›484 »