From peter at osuosl.org Mon Jan 3 17:43:25 2011 From: peter at osuosl.org (Peter Krenesky) Date: Mon, 03 Jan 2011 08:43:25 -0800 Subject: [portland] OSU Open Source Lab is hiring Message-ID: <4D21FCAD.10100@osuosl.org> Hello everyone, My employer, Oregon State University's Open Source Lab, is hiring a full-time software developer. We need someone to help us build out numerous open source projects, including Ganeti Web Manager. Our ideal candidate has excellent Django and Python programming skills, some Java and Ruby preferred. http://osuosl.org/about/news/hiring-analyst-programmer thanks for your time, - kreneskyp From grace at lolapps.com Tue Jan 4 23:30:53 2011 From: grace at lolapps.com (Grace Law) Date: Tue, 4 Jan 2011 14:30:53 -0800 Subject: [portland] Can you help with Server/Scalability challenges at huge social gaming site? In-Reply-To: References: Message-ID: Hi there, Want to move to San Francisco and work with a group of smart, fun people and LOL at the office? My HR manager said we can relocate you :) We are a 2 year old, cash flow positive social gaming / Facebook App company. About 40 people now and plan to get to 60-70 in the next 6 to 12 months. Big Python/Pylons shop building high quality Flash games. You can find out how we scaled from 0 to 50 million users from this video at the last PyCon. http://us.pycon.org/2010/conference/schedule/event/135/ :) We're looking for a seasoned server/performance engineer to do more of that. More details here: *Python Server/Scalability Engineer * http://lolapps.com is looking for a seasoned performance engineer. You know the thrill and the terror of an unexpected traffic storm that's railed your application. You think on your feet, adapt and make a genius patch that let's your servers hold to see out the storm, then hit the whiteboard to start architecting a solution that will handle the next storm with ease. Ideally, you: * Love python and can code it in your sleep. * Working knowledge of Linux, scripting, and SQL. * Understand when MySQL is great and experiment with NoSQL solutions (Memcached/Mongo/Redis/Cassandra) * Know how to put together a web-application stack. (We use Pylons/Paste.) * Enjoy bouncing ideas of your teammates to build up solutions no one person could of thought up by themselves. * Care about your implementations and find yourself compulsively checking that your latest experimental deploy is working the way you thought it would. You'll get to: * Work in an innovative space that is expanding into a billion dollar industry. * Design and implement large chunks of scalability features. * Help make key infrastructure decisions (databases, replication layouts, caching solutions, etc.). * Experiment with the newest emerging open-source technologies. * Test your ideas and strategies out on millions of users and enormous data sets. * Head up a small team of experienced engineers (if you are willing and able). * Have fun. Play ping pong, foosball, video games. * Eat. We buy your lunches. Want to find out more? Send me an email or Click here to apply Cheers, Grace http://lolapps.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From michelle at pdxpython.org Tue Jan 11 00:50:43 2011 From: michelle at pdxpython.org (Michelle Rowley) Date: Mon, 10 Jan 2011 15:50:43 -0800 Subject: [portland] Reminder: PDX Python Tomorrow @ NEW LOCATION! Message-ID: Hey Pythoneers, Happy 2011! We're very excited to be kicking off the new year with a PDX Python meeting in a brand new space: Urban Airship's *new* office at 11th and NW Flanders. Michael Schurter will share a bit about deploying WSGI apps with nginx, and Jason Kirtland will give a presentation on Blinker entitled, "Decouple Your Everything with Publish/Subscribe." Head over at 6:30pm. Urban Airship is at 334 NW 11th Ave, in the Pearl District: http://goo.gl/maps/U6mC The main door will probably be locked, but the back door, which leads directly to the event space, will be propped open. The back door is right around the corner on NW Flanders, next to the loading dock: http://goo.gl/maps/Ikbh We'll put up signs, but if you get lost you can call Adam at 503-866-0663. See you there! Michelle -------------- next part -------------- An HTML attachment was scrubbed... URL: From lgellert at gmail.com Tue Jan 11 23:05:03 2011 From: lgellert at gmail.com (L G) Date: Tue, 11 Jan 2011 14:05:03 -0800 Subject: [portland] Reminder: PDX Python Tomorrow @ NEW LOCATION! In-Reply-To: References: Message-ID: How is the parking situation down there? Laurence On Mon, Jan 10, 2011 at 3:50 PM, Michelle Rowley wrote: > Hey Pythoneers, > > Happy 2011! We're very excited to be kicking off the new year with a PDX > Python meeting in a brand new space: Urban Airship's *new* office at 11th > and NW Flanders. Michael Schurter will share a bit about deploying WSGI > apps > with nginx, and Jason Kirtland will give a presentation on Blinker > entitled, > "Decouple Your Everything with Publish/Subscribe." Head over at 6:30pm. > > Urban Airship is at 334 NW 11th Ave, in the Pearl District: > http://goo.gl/maps/U6mC > > The main door will probably be locked, but the back door, which leads > directly to the event space, will be propped open. The back door is right > around the corner on NW Flanders, next to the loading dock: > http://goo.gl/maps/Ikbh > > We'll put up signs, but if you get lost you can call Adam at 503-866-0663. > > See you there! > Michelle > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://mail.python.org/pipermail/portland/attachments/20110110/f5f39225/attachment.html > > > _______________________________________________ > Portland mailing list > Portland at python.org > http://mail.python.org/mailman/listinfo/portland > -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael at susens-schurter.com Tue Jan 11 23:12:07 2011 From: michael at susens-schurter.com (Michael Schurter) Date: Tue, 11 Jan 2011 14:12:07 -0800 Subject: [portland] Reminder: PDX Python Tomorrow @ NEW LOCATION! In-Reply-To: References: Message-ID: On Tue, Jan 11, 2011 at 2:05 PM, L G wrote: > How is the parking situation down there? Normal downtown metered street parking. Not sure about nearby garages/lots. Might have to search a few blocks before finding a space, but I think it's probably better than it was around Web Trends. From adam at therobots.org Tue Jan 11 23:15:35 2011 From: adam at therobots.org (Adam Lowry) Date: Tue, 11 Jan 2011 14:15:35 -0800 Subject: [portland] Reminder: PDX Python Tomorrow @ NEW LOCATION! In-Reply-To: References: Message-ID: There's street parking, but the meters run until 7pm -- you might have luck around the NW Park Blocks, which can be a little quieter. There are several parking garages, including one at NW Davis & NW 11th (probably the closest). It's on the streetcar line, and 5 blocks from the Yellow/Green MAX lines. On Jan 11, 2011, at 2:05 PM, L G wrote: > How is the parking situation down there? > > Laurence > > On Mon, Jan 10, 2011 at 3:50 PM, Michelle Rowley wrote: > >> Hey Pythoneers, >> >> Happy 2011! We're very excited to be kicking off the new year with a PDX >> Python meeting in a brand new space: Urban Airship's *new* office at 11th >> and NW Flanders. Michael Schurter will share a bit about deploying WSGI >> apps >> with nginx, and Jason Kirtland will give a presentation on Blinker >> entitled, >> "Decouple Your Everything with Publish/Subscribe." Head over at 6:30pm. >> >> Urban Airship is at 334 NW 11th Ave, in the Pearl District: >> http://goo.gl/maps/U6mC >> >> The main door will probably be locked, but the back door, which leads >> directly to the event space, will be propped open. The back door is right >> around the corner on NW Flanders, next to the loading dock: >> http://goo.gl/maps/Ikbh >> >> We'll put up signs, but if you get lost you can call Adam at 503-866-0663. >> >> See you there! >> Michelle >> -------------- next part -------------- >> An HTML attachment was scrubbed... >> URL: < >> http://mail.python.org/pipermail/portland/attachments/20110110/f5f39225/attachment.html >>> >> _______________________________________________ >> Portland mailing list >> Portland at python.org >> http://mail.python.org/mailman/listinfo/portland >> > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > _______________________________________________ > Portland mailing list > Portland at python.org > http://mail.python.org/mailman/listinfo/portland From bergamot at gmail.com Tue Jan 11 23:19:52 2011 From: bergamot at gmail.com (Joel Bernstein) Date: Tue, 11 Jan 2011 14:19:52 -0800 Subject: [portland] Reminder: PDX Python Tomorrow @ NEW LOCATION! In-Reply-To: References: Message-ID: <5F4F5557-29FC-4658-9CCA-6BD94E338870@gmail.com> I park in a lot on the corner of Burnside and 6th. $4 for the night, and pretty close to Bailey's, where we usually end up. On Jan 11, 2011, at 2:15 PM, Adam Lowry wrote: > There's street parking, but the meters run until 7pm -- you might have luck around the NW Park Blocks, which can be a little quieter. There are several parking garages, including one at NW Davis & NW 11th (probably the closest). It's on the streetcar line, and 5 blocks from the Yellow/Green MAX lines. > > On Jan 11, 2011, at 2:05 PM, L G wrote: > >> How is the parking situation down there? >> >> Laurence >> >> On Mon, Jan 10, 2011 at 3:50 PM, Michelle Rowley wrote: >> >>> Hey Pythoneers, >>> >>> Happy 2011! We're very excited to be kicking off the new year with a PDX >>> Python meeting in a brand new space: Urban Airship's *new* office at 11th >>> and NW Flanders. Michael Schurter will share a bit about deploying WSGI >>> apps >>> with nginx, and Jason Kirtland will give a presentation on Blinker >>> entitled, >>> "Decouple Your Everything with Publish/Subscribe." Head over at 6:30pm. >>> >>> Urban Airship is at 334 NW 11th Ave, in the Pearl District: >>> http://goo.gl/maps/U6mC >>> >>> The main door will probably be locked, but the back door, which leads >>> directly to the event space, will be propped open. The back door is right >>> around the corner on NW Flanders, next to the loading dock: >>> http://goo.gl/maps/Ikbh >>> >>> We'll put up signs, but if you get lost you can call Adam at 503-866-0663. >>> >>> See you there! >>> Michelle >>> -------------- next part -------------- >>> An HTML attachment was scrubbed... >>> URL: < >>> http://mail.python.org/pipermail/portland/attachments/20110110/f5f39225/attachment.html >>>> >>> _______________________________________________ >>> Portland mailing list >>> Portland at python.org >>> http://mail.python.org/mailman/listinfo/portland >>> >> -------------- next part -------------- >> An HTML attachment was scrubbed... >> URL: >> _______________________________________________ >> Portland mailing list >> Portland at python.org >> http://mail.python.org/mailman/listinfo/portland > > _______________________________________________ > Portland mailing list > Portland at python.org > http://mail.python.org/mailman/listinfo/portland From jek at discorporate.us Thu Jan 13 23:38:08 2011 From: jek at discorporate.us (jason kirtland) Date: Thu, 13 Jan 2011 14:38:08 -0800 Subject: [portland] Front End Coder at Idealist.org Message-ID: Hi all, Idealist.org is looking for a front-end developer to join our web team, working with a Python web stack. Listing below. Cheers! -Jason ---- http://www.idealist.org/view/job/P8hWSFPsZ6W4 Idealist.org is looking for a detail-oriented Front End Coder to join our development team in our Portland, Oregon office. You will contribute scalable production code for user-facing features on a highly trafficked global website. Job Responsibilities: - Provides CSS, HTML, and Javascript code for the site - Helps create solutions for challenging design implementation issues - Provides technical guidance and support during product planning and development - Participates in planning meetings with developers and project managers - Continually upgrades and maintains the design of the site Skills needed: - Fluency in HTML, CSS, Javascript (specifically the jQuery library), HTML5 is a plus - Experience with template-driven dynamic sites, Python-specific template systems a big plus (Django templates, Jinja, Genshi, Mako, etc.) - Some experience with the server-side programming language of your choice (Python, Ruby, Perl, etc.) - Some experience with distributed version control workflow, Mercurial a plus. - Understanding of browser capabilities (and cross-browser incompatibilities) - Flexible, adaptable, and ability to learn quickly - While not being responsible for the overall look and feel of the site, you should have an eye for design and usability. - Minimum of two years related experience - Commitment to our mission Desired skills: - Experience with mobile app development - Experience with rapid prototyping (wireframes and comps) - Experience with SEO and coding for accessibility Salary and Benefits: We offer a competitive salary and comprehensive benefits package including 20 days vacation in the first year, 10 paid holidays, 401K retirement program, and health, dental, vision, and disability insurance. Deadline for applications is January 24th. To Apply: Please prepare one document (either a .pdf, .doc or text file) that includes your resume, and in lieu of a cover letter, your answers to the questions listed below. Save the file as yourlastname_FrontEndDev and email it to careers at idealist.org. Thank you! 1. Why do you want to work with us? 2. Why do you think you would excel at this job? 3. Please provide 3-5 specific examples of previous work that you?re proud of. (URLs, code samples, github/bitbucket, etc.) About Idealist.org: Idealist.org is a nonprofit organization that connects people, organizations, and resources to help build a world where all people can live free and dignified lives. Launched in 1995, Idealist.org and its sister sites Idealistas.org and Idealiste.org have become the most comprehensive online resources for the nonprofit/NGO sector worldwide, with more than 55,000 nonprofit organizations from 180 countries posting their job openings, and volunteer and internship opportunities on these sites. From python at dylanreinhardt.com Wed Jan 19 00:53:24 2011 From: python at dylanreinhardt.com (Dylan Reinhardt) Date: Tue, 18 Jan 2011 15:53:24 -0800 Subject: [portland] Django teaching opportunity Message-ID: Slightly off-topic, but possibly the perfect thing for someone here: PCC has a Django class on their schedule and students registered... but due to a communication snafu, they still need an instructor. :-) This is a pretty exciting opportunity, particularly for anyone looking to add a bit of oomph to their independent contractor resume. Here's the catch: the Django class is scheduled to start next week. So obviously this is a time-limited opportunity. In related news, they may also have an opportunity opening opening up for an instructor for their intro Python course. I would guess the right person could pick up both classes and probably propose others. If you're interested, please shoot an email to John Buesseler (copied). He'll be happy to answer any questions you may have. I've taught there the last year or so and I'm happy to vouch for the program if you want to contact me off-list. Dylan -------------- next part -------------- An HTML attachment was scrubbed... URL: From kirby.urner at gmail.com Wed Jan 19 01:07:25 2011 From: kirby.urner at gmail.com (kirby urner) Date: Tue, 18 Jan 2011 16:07:25 -0800 Subject: [portland] Django teaching opportunity In-Reply-To: References: Message-ID: Cool Dylan. Do you know if PCC does any computer classes at the facility near OMSI? I interviewed for your job (a long time ago) but wasn't crazy about the commute out to Intel or wherever that was. I'm currently doing distance ed as a part time instructor for O'Reilly School of Technology plus sometimes teach Python through Saturday Academy ( saturdayacademy.org). "Martian Math" with VPython put me on the map as an innovative curriculum writer in some circles (re this summer @ Reed College, googlable). Sorry I missed the PPUG debut @ Urban Airship ya'll. Gossip: Holden Web seeking to incorporate here in Oregon, Steve looking forward to continuing friendships, starting new ones. Kirby c.v.: http://grunch.net/kirby-urner (I don't see myself as good enough at Django to lead a course in it, so passing on that one). On Tue, Jan 18, 2011 at 3:53 PM, Dylan Reinhardt wrote: > Slightly off-topic, but possibly the perfect thing for someone here: > > PCC has a Django class on their schedule and students registered... but due > to a communication snafu, they still need an instructor. :-) > > This is a pretty exciting opportunity, particularly for anyone looking to > add a bit of oomph to their independent contractor resume. > > Here's the catch: the Django class is scheduled to start next week. So > obviously this is a time-limited opportunity. > > In related news, they may also have an opportunity opening opening up for > an > instructor for their intro Python course. I would guess the right person > could pick up both classes and probably propose others. > > If you're interested, please shoot an email to John Buesseler (copied). > He'll be happy to answer any questions you may have. > > I've taught there the last year or so and I'm happy to vouch for the > program > if you want to contact me off-list. > > Dylan > -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at dylanreinhardt.com Wed Jan 19 01:38:17 2011 From: python at dylanreinhardt.com (Dylan Reinhardt) Date: Tue, 18 Jan 2011 16:38:17 -0800 Subject: [portland] Django teaching opportunity In-Reply-To: References: Message-ID: Hey Kirby... thanks for the gentle reminder that I forgot to mention the location. :-) The Django class is scheduled at Willow Creek Center (SW 185th). http://www.pcc.edu/about/locations/willow-creek/ I imagine there might be some flexibility to use other PCC facilities for future classes, but you should contact John to discuss that if it's a make-or-break consideration. Dylan On Tue, Jan 18, 2011 at 4:07 PM, kirby urner wrote: > Cool Dylan. > > Do you know if PCC does any computer classes at the facility near OMSI? > > I interviewed for your job (a long time ago) but wasn't crazy about the > commute out to Intel or wherever that was. > > I'm currently doing distance ed as a part time instructor for O'Reilly > School of Technology plus sometimes teach Python through Saturday Academy ( > saturdayacademy.org). > > "Martian Math" with VPython put me on the map as an innovative curriculum > writer in some circles (re this summer @ Reed College, googlable). > > Sorry I missed the PPUG debut @ Urban Airship ya'll. > > Gossip: Holden Web seeking to incorporate here in Oregon, Steve looking > forward to continuing friendships, starting new ones. > > Kirby > > c.v.: http://grunch.net/kirby-urner (I don't see myself as good enough > at > Django to lead a course in it, so passing on that one). > > On Tue, Jan 18, 2011 at 3:53 PM, Dylan Reinhardt > wrote: > > > Slightly off-topic, but possibly the perfect thing for someone here: > > > > PCC has a Django class on their schedule and students registered... but > due > > to a communication snafu, they still need an instructor. :-) > > > > This is a pretty exciting opportunity, particularly for anyone looking to > > add a bit of oomph to their independent contractor resume. > > > > Here's the catch: the Django class is scheduled to start next week. So > > obviously this is a time-limited opportunity. > > > > In related news, they may also have an opportunity opening opening up for > > an > > instructor for their intro Python course. I would guess the right person > > could pick up both classes and probably propose others. > > > > If you're interested, please shoot an email to John Buesseler (copied). > > He'll be happy to answer any questions you may have. > > > > I've taught there the last year or so and I'm happy to vouch for the > > program > > if you want to contact me off-list. > > > > Dylan > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://mail.python.org/pipermail/portland/attachments/20110118/b47b6f78/attachment.html > > > _______________________________________________ > Portland mailing list > Portland at python.org > http://mail.python.org/mailman/listinfo/portland > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rbednark at gmail.com Fri Jan 21 01:47:02 2011 From: rbednark at gmail.com (Rob Bednark) Date: Thu, 20 Jan 2011 16:47:02 -0800 Subject: [portland] Hiring: developer / QA engineer to write automated tests in python Message-ID: Portland Python Users Group, I'm managing software testing of a web application and mobile application for a large telecommunications company, and am looking to hire someone to develop automated tests. I'm looking for someone with fairly strong programming experience and skills. Experience in python and Selenium is a plus, but I will also consider someone with strong skills in other languages that is a quick and eager learner. The automated testing will initially be mostly end-user functional tests, and then also doing some load/stress testing. Some manual testing may be required as well. I've started writing some tests using Selenium and python, which seem to be good tools for the job. I could possibly be talked into using Ruby instead. Experience with the following is a plus: - experience with Selenium or other tools for automated web application testing - experience with load/performance/stress testing tools for web applications like Selenium Grid or JMeter This is a work-from-home position, with other members of the culturally-diverse team scattered around the US, England, Russia, and Ukraine. Right now the testing team is just myself along with a couple of part-time manual testers in England. I live in the Portland area. This is an hourly contract position, 40 hours per week, paid weekly. The duration of the contract depends on the success and growth of the product. It will likely last weeks, if not months. Submit a resume and your hourly rate, along with anything else you think might be helpful (code samples, videos, ...). Rob [image: Rob.100px.JPG] Rob Bednark | H: 503.877.2336| Facebook| LinkedIn | Twitter| rbednark at gmail.com | 10013 SE Eastmont Dr, Damascus, OR 97089 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Rob.100px.JPG Type: image/jpeg Size: 16353 bytes Desc: not available URL: From thomas at metaphorlab.com Wed Jan 26 00:24:59 2011 From: thomas at metaphorlab.com (Thomas Hamlin) Date: Tue, 25 Jan 2011 15:24:59 -0800 Subject: [portland] Django cloud project with startup role potential Message-ID: <666A4C49-67B7-482F-B471-A65AB96DD85E@metaphorlab.com> Hi fellow PDX Pythoneers, [You have dibs, I may post this to a wider audience soon...] I am developing a few web sites this spring for a number of bicycle and active commuting campaigns around the country. I am looking for one or two developers that have experience with Django and cloud-based operations. Ideally, you are/have most of these: - able to work on a contract basis and do not need a full time/permanent gig - enjoy working when and where you like - experienced with Django, Git, PostgreSQL, Ubuntu - experienced implementing social network features such as incorporating Twitter feeds or Facebook features into a Django site - Amazon EC2 or other cloud experience - some devops experience would be great - interested in meaningful work, in advancing bicycle commuting and other active forms of transport! Lastly, these projects will lead to the launching of a large scale web service offering by next year. You have the chance to be there at the pre-startup stage! Interested? Please email me with some info about yourself and your experience. Thomas Hamlin http://metaphorlab.com Meaningful user experience, handcrafted from the finest materials. thomas at metaphorlab.com From brian.curtin at gmail.com Thu Jan 27 21:49:02 2011 From: brian.curtin at gmail.com (Brian Curtin) Date: Thu, 27 Jan 2011 14:49:02 -0600 Subject: [portland] PSF Sprints - Call For Applications Message-ID: Hello Portland Python Users! On behalf of the Python Software Foundation?s sponsored sprint group, I wanted to drop your group a quick note introducing us. If you?re already familiar with our sponsored sprints, you?ll be happy to know we made a few changes to help both sprint groups and Python even more. The PSF recently set aside funding to be distributed to groups who spend time contributing to the Python ecosystem, often in the form of development sprints. Our goal is to help you help Python, so whether it?s buying meals or renting meeting space for your all-day hackathon, we have a budget set aside to reimburse your expenses up to $300 USD (up from $250). If your goal is to make the Python world a better place, and you work on the problems facing Python today, we want to help you. We?re looking for groups of hackers that spend their time fixing and expanding the wide variety of Python interpreters, libraries, tools, and anything else affecting the community.We?re also looking for groups who want to help and get started but don?t have the resources to get together. Whether your group is separated by a train ride or lacking a shared space, we want to help you. On-boarding new contributors to open source Python projects is an especially important area that we?d like to work with.This means if you have a Python project and you want to sprint -- we want to help you.Some sprints we?ve sponsored include the porting of Genshi to Python 3, improvements to packaging (Distribute/distutils), and most recently, the PyPy winter sprint in Switzerland. If your group is interested in hosting a sprint, check out the full details of our call for applications at http://www.pythonsprints.com/cfa/ and contact us at sprints at python.org. Thanks for your time, and happy sprinting! Brian Curtin Jesse Noller http://www.pythonsprints.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcoder at gmail.com Fri Jan 28 19:39:58 2011 From: rcoder at gmail.com (Lennon Day-Reynolds) Date: Fri, 28 Jan 2011 10:39:58 -0800 Subject: [portland] Python/web/mobile coding gig at Dark Horse Comics Message-ID: Fellow PDX Pythonistas: As some of you may have already heard, I will soon be leaving our fair city to take a new job in San Francisco. That opens up a spot on our awesome dev team here at Dark Horse Comics for a Python-loving coder who wants to help build our next generation of e-commerce and digital publishing tools. Ideally, we're looking for someone who's willing to tackle a mix of Python/Django web development, deployment/ops/automation, mobile app development, and even a bit of legacy PHP maintenance (I know, I know, *eeeeewww*). More details are up on our jobs page: http://www.darkhorse.com/Company/Jobs I'd also be happy to answer any questions directly, if you want the inside scoop on what kind of projects we have going, what our development process is like, or just how awesome it is to get free comics at work. Cheers, Lennon Day-Reynolds / @rcoder From rshepard at appl-ecosys.com Fri Jan 28 20:44:39 2011 From: rshepard at appl-ecosys.com (Rich Shepard) Date: Fri, 28 Jan 2011 11:44:39 -0800 (PST) Subject: [portland] Reformatting Data Files Message-ID: I'll open with a mea culpa: I've done no python coding in more than a year, have lost most of my knowledge, and need to regain (and exceed) the proficiency I once had. My immediate need is to restructure data exported from spreadsheets as .csv files. I'm trying to use the csv module, but keep going around in circles trying to index each column in each row so they can be written to an output file with a different organization. Here are the first three lines of an exported file (10b20b.txt): :'JCM-10B':'JCM-10B':'JCM-10B':'JCM-10B':'JCM-10B':'JCM-10B':'JCM-10B':'JCM-10B':'JCM-10B':'JCM-10B':'JCM-10B':'JCM-10B':'JCM-20B':'JCM-20B':'JCM-20B':'JCM-20B':'JCM-20B':'JCM-20B':'JCM-20B':'JCM-20B':'JCM-20B':'JCM-20B':'JCM-20B':'JCM-20B' :2004-08-16:2005-06-07:2005-08-31:2006-06-12:2006-09-21:2007-06-21:2007-12-11:2008-06-09:2008-12-04:2009-06-23:2009-10-15::2004-08-16:2005-06-08:2005-08-16:2006-06-02:2006-12-29:2007-06-08:2007-12-11:2008-06-09:2008-11-13:2009-06-30:2009-12-09: 'Depth to Water':76.75:77.51:82.15:73.17:72.66:84.44:96.41:92.15:97.95::::90.72:90.74:91.65:87.41:86.81:91.72:96.42:97.03:97.08::: The first two rows have NULL values for the first column (which I'll call param). The goal of the exercise is to write an output .csv file suitable for import to a database table. This means concatenating row 0, column 1 with row 1, column 1 and row 2, column 0, row 2, column 1, row 3, col 0, row 3, col 1, etc. In other words, I want the output to look like this: ::'JCM-10B':'2004-08-16':'Depth to water':76.75:: ::'JCM-10B':'2004-08-16':::: What I've written so far doesn't do the job, and I'm not seeing the proper approach to this. Using the csv.reader() method does not allow subscripting so all I'be been able to print is the first column of each row. Not very good. datacon.py: #!/usr/bin/env python """ This file converts the exported wq excel files to have each row contain the location, date, parameter, and value. """ import csv reader = csv.reader(open("10b20b.txt","rb"), delimiter=":", quotechar="'") output = csv.writer(open("out.txt","wb")) try: for row in reader: param = row[0] site = row[1] print param, site #output.writerow(row[0]) except csv.Error, e: sys.exit('file %s, line %d: %s' % (output, reader.line_num, e)) Because I have several dozen such files to convert, this exercise is a great way for my getting my python back into working memory. I think the problem is integrating the csv module with manipulating a 2-dimensional array [row][col] and adding the NULL separators so the output can be imported to an existing database table. Pointers are needed; thanks in advance. Rich From jheasly at earthlink.net Fri Jan 28 21:02:54 2011 From: jheasly at earthlink.net (John Heasly) Date: Fri, 28 Jan 2011 12:02:54 -0800 Subject: [portland] Reformatting Data Files In-Reply-To: References: Message-ID: Hello Rich, Python-excel may be of some use: http://www.python-excel.org/ HTH, John From rshepard at appl-ecosys.com Fri Jan 28 21:23:43 2011 From: rshepard at appl-ecosys.com (Rich Shepard) Date: Fri, 28 Jan 2011 12:23:43 -0800 (PST) Subject: [portland] Reformatting Data Files In-Reply-To: References: Message-ID: On Fri, 28 Jan 2011, John Heasly wrote: > Python-excel may be of some use: http://www.python-excel.org/ John, Perhaps. The files were opened in OO.o and saved at .csv from there. Rich From python at dylanreinhardt.com Fri Jan 28 21:29:25 2011 From: python at dylanreinhardt.com (Dylan Reinhardt) Date: Fri, 28 Jan 2011 12:29:25 -0800 Subject: [portland] Reformatting Data Files In-Reply-To: References: Message-ID: On Fri, Jan 28, 2011 at 11:44 AM, Rich Shepard wrote: > The goal of the exercise is to write an output .csv file suitable for > import to a database table. This means concatenating row 0, column 1 with > row 1, column 1 and row 2, column 0, row 2, column 1, row 3, col 0, row 3, > col 1, etc. In other words, I want the output to look like this: > > To get a list of the concatenated values for column 0, 1, 2... how about: --- column_values = list() for col in range(3): # or whatever column_values.append(' '.join([row[col] for row in reader])) --- If you have a very large amount of data, it might be worth the extra time to do it in one pass... but the above might be good enough for the task at hand. HTH, Dylan -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Fri Jan 28 21:41:30 2011 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 28 Jan 2011 12:41:30 -0800 Subject: [portland] Reformatting Data Files In-Reply-To: References: Message-ID: <4D4329FA.1080202@stoneleaf.us> Rich Shepard wrote: > I'll open with a mea culpa: I've done no python coding in more than a > year, have lost most of my knowledge, and need to regain (and exceed) the > proficiency I once had. My immediate need is to restructure data exported > from spreadsheets as .csv files. I'm trying to use the csv module, but keep > going around in circles trying to index each column in each row so they can > be written to an output file with a different organization. > > Here are the first three lines of an exported file (10b20b.txt): > > :'JCM-10B':'JCM-10B':'JCM-10B':'JCM-10B':'JCM-10B':'JCM-10B':'JCM-10B': > :2004-08-16:2005-06-07:2005-08-31:2006-06-12:2006-09-21:2007-06-21:2007-12-11: > 'Depth to Water':76.75:77.51:82.15:73.17:72.66:84.44:96.41: 'HeightToSky':8917.72:7812.28:9153.72:8411.91:8837.51:8784.22:9244.76: 'SomethingElse':182.98:817.82:352.99:721.57:292.38:379.28:447.92: > > > The first two rows have NULL values for the first column (which I'll call > param). > > The goal of the exercise is to write an output .csv file suitable for > import to a database table. This means concatenating row 0, column 1 with > row 1, column 1 and row 2, column 0, row 2, column 1, row 3, col 0, row 3, > col 1, etc. In other words, I want the output to look like this: > > ::'JCM-10B':'2004-08-16':'Depth to water':76.75:: > ::'JCM-10B':'2004-08-16':::: I'm having a hard time understanding your description of what the second, third, etc, lines should look like... if I carry out your table further, is it correct? ::'JCM-10B':'2004-08-16':'Depth to water':76.75:: ::'JCM-10B':'2004-08-16':'HeightToSky':8917.72:: ::'JCM-10B':'2004-08-16':'SomethingElse':182.89:: If that's correct, what happens to the remaining values on lines 3, 4, etc? ~Ethan~ From rshepard at appl-ecosys.com Fri Jan 28 23:31:12 2011 From: rshepard at appl-ecosys.com (Rich Shepard) Date: Fri, 28 Jan 2011 14:31:12 -0800 (PST) Subject: [portland] Reformatting Data Files In-Reply-To: References: Message-ID: On Fri, 28 Jan 2011, Dylan Reinhardt wrote: > To get a list of the concatenated values for column 0, 1, 2... how about: > > --- > > column_values = list() > for col in range(3): # or whatever > column_values.append(' '.join([row[col] for row in reader])) > If you have a very large amount of data, it might be worth the extra time to > do it in one pass... but the above might be good enough for the task at > hand. Dylan, Thanks. I'll give this some thought. My initial idea was to read each line into a separate list, then extract items from each list in sequence and write them out as a row in the output file. Because the first two rows of the input file lack values in the first column the indexing would not be simple, but could be figured out by trial-and-error. Rich From rshepard at appl-ecosys.com Fri Jan 28 23:41:10 2011 From: rshepard at appl-ecosys.com (Rich Shepard) Date: Fri, 28 Jan 2011 14:41:10 -0800 (PST) Subject: [portland] Reformatting Data Files In-Reply-To: <4D4329FA.1080202@stoneleaf.us> References: <4D4329FA.1080202@stoneleaf.us> Message-ID: On Fri, 28 Jan 2011, Ethan Furman wrote: >> :'JCM-10B':'JCM-10B':'JCM-10B':'JCM-10B':'JCM-10B':'JCM-10B':'JCM-10B': >> :2004-08-16:2005-06-07:2005-08-31:2006-06-12:2006-09-21:2007-06-21:2007-12-11: >> 'Depth to Water':76.75:77.51:82.15:73.17:72.66:84.44:96.41: > 'HeightToSky':8917.72:7812.28:9153.72:8411.91:8837.51:8784.22:9244.76: > 'SomethingElse':182.98:817.82:352.99:721.57:292.38:379.28:447.92: Ethan, That's close enough. The spreadsheet was created with, in this case two locations (JCM-10B and JCM-20B) as the top row. The second row contains dates, and succeeding rows each contain a water quality paramter and the value measured for that parameter at that location on that date. > I'm having a hard time understanding your description of what the second, > third, etc, lines should look like... if I carry out your table further, > is it correct? > ::'JCM-10B':'2004-08-16':'Depth to water':76.75:: > ::'JCM-10B':'2004-08-16':'HeightToSky':8917.72:: > ::'JCM-10B':'2004-08-16':'SomethingElse':182.89:: > If that's correct, what happens to the remaining values on lines 3, 4, etc? Yes, this is correct. Instead of the organization above, I want each row to be a complete record of location, date, parameter and value. Somewhat analogous to transposing rows and columns in the spreadsheet, but that's not a solution. In the re-written file, there will be one row for each unique set of location, date, parameter, and value. For this file, there are 2 locations, 22 dates, and 32 parameters (each with 1 value for that parameter on that date at that location) or 1408 distinct rows. Does this make it clearer? Unfortunately, no one now or before at the client's site knows anything about databases or data storage so the spreadsheets are a horrible mess in terms of extracting the data. Rich From ethan at stoneleaf.us Sat Jan 29 01:07:22 2011 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 28 Jan 2011 16:07:22 -0800 Subject: [portland] Reformatting Data Files In-Reply-To: References: <4D4329FA.1080202@stoneleaf.us> Message-ID: <4D435A3A.3090709@stoneleaf.us> Rich Shepard wrote: > Yes, this is correct. Instead of the organization above, I want each row > to be a complete record of location, date, parameter and value. Somewhat > analogous to transposing rows and columns in the spreadsheet, but that's > not > a solution. > > In the re-written file, there will be one row for each unique set of > location, date, parameter, and value. For this file, there are 2 locations, > 22 dates, and 32 parameters (each with 1 value for that parameter on that > date at that location) or 1408 distinct rows. Wow. Here's an attempt -- not sure how general it will be, but hopefully it will help. ====== input ============= :'JCM-10B':'JCM-10B':'JCM-10B':'JCM-20B':'JCM-20B':'JCM-20B' :2004-08-16:2005-06-07::2006-06-12::2009-12-09 'Depth to Water':76.75:77.51:82.15:73.17:72.66: 'Clarity':6.35:9.75:4.75:3.26:4.07:7.69 'Salinity':0.38:0.69:0.16:0.79::0.05 'Temperature':41.55:34.87:42.73:40.80:45.14: ========================== ------ output ------------ ::JCM-10B:2004-08-16:Depth to Water:76.75:: ::JCM-10B:2004-08-16:Clarity:6.35:: ::JCM-10B:2004-08-16:Salinity:0.38:: ::JCM-10B:2004-08-16:Temperature:41.55:: ::JCM-10B:2005-06-07:Depth to Water:77.51:: ::JCM-10B:2005-06-07:Clarity:9.75:: ::JCM-10B:2005-06-07:Salinity:0.69:: ::JCM-10B:2005-06-07:Temperature:34.87:: ::JCM-10B::Depth to Water:82.15:: ::JCM-10B::Clarity:4.75:: ::JCM-10B::Salinity:0.16:: ::JCM-10B::Temperature:42.73:: ::JCM-20B:2006-06-12:Depth to Water:73.17:: ::JCM-20B:2006-06-12:Clarity:3.26:: ::JCM-20B:2006-06-12:Salinity:0.79:: ::JCM-20B:2006-06-12:Temperature:40.80:: ::JCM-20B::Depth to Water:72.66:: ::JCM-20B::Clarity:4.07:: ::JCM-20B::Salinity::: ::JCM-20B::Temperature:45.14:: ::JCM-20B:2009-12-09:Depth to Water::: ::JCM-20B:2009-12-09:Clarity:7.69:: ::JCM-20B:2009-12-09:Salinity:0.05:: ::JCM-20B:2009-12-09:Temperature::: ---------------------------- ~~~~~~~~~~~~~~~~~~~~~ code ~~~~~~~~~~~~~~~~~~~~~~~~~~ #!/usr/bin/env python """ This file converts the exported wq excel files to have each row contain the location, date, parameter, and value. """ import csv reader = csv.reader(open("10b20b.txt","rb"), delimiter=":", quotechar="'") output = csv.writer(open("out.txt","wb"), delimiter=":", quotechar="'") # records are in columnar, not row, format # read in all data into format # [fieldname, [value1, value2, value3, ..., valueN]] all_data = list() for row in reader: row = list(row) # keep fieldname separate from values all_data.append((row[0], row[1:])) # if fieldname is empty, assume header row # once a fieldname/value row is found, stop # looking for anymore header rows header_rows = list() for i, fieldname_row in enumerate(all_data): fieldname, row = fieldname_row if not fieldname: header_rows.append(row) else: break # assume all remaining rows are the fields and values # i will be the index of the first row with a fieldname first_field_row = i # assemble headers for j, _ in enumerate(header_rows[0]): headers = list() for row in header_rows: headers.append(row[j]) # now step through field:value rows for fieldname, row in all_data[first_field_row:]: output.writerow([None, None] + headers + [fieldname, row[j], None, None]) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Hope this helps! ~Ethan~ From kyle at bucebuce.com Sat Jan 29 01:01:09 2011 From: kyle at bucebuce.com (Kyle Jones) Date: Fri, 28 Jan 2011 16:01:09 -0800 Subject: [portland] Reformatting Data Files In-Reply-To: References: <4D4329FA.1080202@stoneleaf.us> Message-ID: Rich, It isn't anything pretty, but I think this might be similar: http://paste.pocoo.org/show/328545/ - Kyle On Fri, Jan 28, 2011 at 2:41 PM, Rich Shepard wrote: > On Fri, 28 Jan 2011, Ethan Furman wrote: > > :'JCM-10B':'JCM-10B':'JCM-10B':'JCM-10B':'JCM-10B':'JCM-10B':'JCM-10B': >>> >>> :2004-08-16:2005-06-07:2005-08-31:2006-06-12:2006-09-21:2007-06-21:2007-12-11: >>> 'Depth to Water':76.75:77.51:82.15:73.17:72.66:84.44:96.41: >>> >> 'HeightToSky':8917.72:7812.28:9153.72:8411.91:8837.51:8784.22:9244.76: >> 'SomethingElse':182.98:817.82:352.99:721.57:292.38:379.28:447.92: >> > > Ethan, > > That's close enough. The spreadsheet was created with, in this case two > locations (JCM-10B and JCM-20B) as the top row. The second row contains > dates, and succeeding rows each contain a water quality paramter and the > value measured for that parameter at that location on that date. > > > I'm having a hard time understanding your description of what the second, >> third, etc, lines should look like... if I carry out your table further, >> is it correct? >> > > ::'JCM-10B':'2004-08-16':'Depth to water':76.75:: >> ::'JCM-10B':'2004-08-16':'HeightToSky':8917.72:: >> ::'JCM-10B':'2004-08-16':'SomethingElse':182.89:: >> > > If that's correct, what happens to the remaining values on lines 3, 4, >> etc? >> > > Yes, this is correct. Instead of the organization above, I want each row > to be a complete record of location, date, parameter and value. Somewhat > analogous to transposing rows and columns in the spreadsheet, but that's > not > a solution. > > In the re-written file, there will be one row for each unique set of > location, date, parameter, and value. For this file, there are 2 locations, > 22 dates, and 32 parameters (each with 1 value for that parameter on that > date at that location) or 1408 distinct rows. > > Does this make it clearer? > > Unfortunately, no one now or before at the client's site knows anything > about databases or data storage so the spreadsheets are a horrible mess in > terms of extracting the data. > > Rich > > > > _______________________________________________ > Portland mailing list > Portland at python.org > http://mail.python.org/mailman/listinfo/portland > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rshepard at appl-ecosys.com Sat Jan 29 01:36:10 2011 From: rshepard at appl-ecosys.com (Rich Shepard) Date: Fri, 28 Jan 2011 16:36:10 -0800 (PST) Subject: [portland] Reformatting Data Files In-Reply-To: References: <4D4329FA.1080202@stoneleaf.us> Message-ID: On Fri, 28 Jan 2011, Kyle Jones wrote: > It isn't anything pretty, but I think this might be similar: > http://paste.pocoo.org/show/328545/ Thank you, Kyle. I'll ponder and play with the code until I really understand it and see what it's doing. It looks just like what I want/need. Interestingly, I had to specify binary mode when reading ("rb" rather than "r") because csv.reader() needs binary mode for some reason I don't understand. It is a text file, but that's just the way it works I guess. Much grasses, Rich From kyle at bucebuce.com Sat Jan 29 01:41:34 2011 From: kyle at bucebuce.com (Kyle Jones) Date: Fri, 28 Jan 2011 16:41:34 -0800 Subject: [portland] Reformatting Data Files In-Reply-To: References: <4D4329FA.1080202@stoneleaf.us> Message-ID: Glad I could help. If you have any specific questions, or would like me to walk you through it (off-list), I'd be happy to. - Kyle On Fri, Jan 28, 2011 at 4:36 PM, Rich Shepard wrote: > On Fri, 28 Jan 2011, Kyle Jones wrote: > > It isn't anything pretty, but I think this might be similar: >> http://paste.pocoo.org/show/328545/ >> > > Thank you, Kyle. I'll ponder and play with the code until I really > understand it and see what it's doing. It looks just like what I want/need. > > Interestingly, I had to specify binary mode when reading ("rb" rather than > "r") because csv.reader() needs binary mode for some reason I don't > understand. It is a text file, but that's just the way it works I guess. > > Much grasses, > > > Rich > _______________________________________________ > Portland mailing list > Portland at python.org > http://mail.python.org/mailman/listinfo/portland > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rshepard at appl-ecosys.com Sat Jan 29 01:49:21 2011 From: rshepard at appl-ecosys.com (Rich Shepard) Date: Fri, 28 Jan 2011 16:49:21 -0800 (PST) Subject: [portland] Reformatting Data Files In-Reply-To: <4D435A3A.3090709@stoneleaf.us> References: <4D4329FA.1080202@stoneleaf.us> <4D435A3A.3090709@stoneleaf.us> Message-ID: On Fri, 28 Jan 2011, Ethan Furman wrote: > Wow. Ethan, This is a small file. I already have > 15,000 rows in the postgres table from spreadsheets that had the proper format. With ~30 years of data the final database will consist of very large tables (in terms of row numbers). > Here's an attempt -- not sure how general it will be, but hopefully it will > help. I'll examine it closely tomorrow so I fully understand what's going on. Many thanks, Rich From pacopablo at pacopablo.com Sat Jan 29 02:51:45 2011 From: pacopablo at pacopablo.com (John Hampton) Date: Fri, 28 Jan 2011 17:51:45 -0800 Subject: [portland] Reformatting Data Files In-Reply-To: References: <4D4329FA.1080202@stoneleaf.us> Message-ID: <4D4372B1.4000604@pacopablo.com> On 1/28/11 4:36 PM, Rich Shepard wrote: > Interestingly, I had to specify binary mode when reading ("rb" rather than > "r") because csv.reader() needs binary mode for some reason I don't > understand. It is a text file, but that's just the way it works I guess. I'm assuming that you're running this code on a Windows machine. On *nix [1], 'r' == 'rb'. However, on Windows, you have to specify binary so that the OS doesn't do any line-ending munging. -John [1] At least on all *nix that I've used. From rshepard at appl-ecosys.com Sat Jan 29 03:22:25 2011 From: rshepard at appl-ecosys.com (Rich Shepard) Date: Fri, 28 Jan 2011 18:22:25 -0800 (PST) Subject: [portland] Reformatting Data Files In-Reply-To: <4D4372B1.4000604@pacopablo.com> References: <4D4329FA.1080202@stoneleaf.us> <4D4372B1.4000604@pacopablo.com> Message-ID: On Fri, 28 Jan 2011, John Hampton wrote: > I'm assuming that you're running this code on a Windows machine. John, Wrong assumption. We've used nothing Microsoft since early 1997. > On *nix [1], 'r' == 'rb'. However, on Windows, you have to specify binary > so that the OS doesn't do any line-ending munging. That's supposed to be the case. But python threw an error message to me when I didn't specify the b. This on Slackware-13.1 with python-2.6.4. Rich From python at dylanreinhardt.com Sat Jan 29 15:00:20 2011 From: python at dylanreinhardt.com (Dylan Reinhardt) Date: Sat, 29 Jan 2011 06:00:20 -0800 Subject: [portland] Reformatting Data Files In-Reply-To: References: <4D4329FA.1080202@stoneleaf.us> Message-ID: On Fri, Jan 28, 2011 at 2:41 PM, Rich Shepard wrote: > Yes, this is correct. Instead of the organization above, I want each row > to be a complete record of location, date, parameter and value. Somewhat > analogous to transposing rows and columns in the spreadsheet, but that's > not > a solution. > Ah.. I was just pivoting the table. Updating my last and adding the ability to specify number of header rows or label columns: --- import csv num_header_rows = 2 num_label_cols = 1 input = list(csv.reader(open('/path/to/file', 'r'))) output = list() headers = [':'.join(t) for t in zip(*input[:num_header_rows])[num_label_cols:]] for row in input: label = ':'.join(row[:num_label_cols]) for header,value in zip(headers, row[num_label_cols:]): output.append(':'.join([header, label, value])) --- HTH, Dylan -------------- next part -------------- An HTML attachment was scrubbed... URL: