[Tutor] Downloading S3 Logs
Lao Mao
laomao1975 at googlemail.com
Thu Feb 11 17:23:20 CET 2010
Hello,
I've written the below to get the previous day's logs from an Amazon S3
bucket.
#!/usr/bin/python
import time
from datetime import datetime
import boto
daily_s3_log = open("/tmp/s3logs", "w+")
now = datetime.now()
connection = boto.connect_s3()
bucket = connection.get_bucket("downloads.sekrit.co.uk")
todays_keys = []
for key in bucket:
time_difference = (now -
datetime(*time.strptime(key.last_modified.split(".")[0], "%Y-%m-%dT%H:
%M:%S")[0:6])).days
if time_difference < 1 and key.name.startswith("log/access_log"):
todays_keys.append(key)
for key in todays_keys:
key.get_file(daily_s3_log)
daily_s3_log.close()
This takes about 2 mins to download a day's logs (about 25M).
I'd appreciate any improvements or feedback on the above.
For example, would it make sense to make the first loop into a generator
function that yields the interesting keys? Also is there a cleaner way to
do the date comparison in Python 2.4?
Thanks,
Laomao
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/tutor/attachments/20100211/564db211/attachment.htm>
More information about the Tutor
mailing list