Is Python Suitable for Large Find & Replace Operations?
Steve Holden
steve at holdenweb.com
Tue Jun 14 13:01:58 EDT 2005
Brian wrote:
> Hi Rbt,
>
> To give an example of processing a lot of data, I used Python to read
> and process every word in a single text file that contained the entire
> King James Bible version. It processed it within about one second --
> split the words, etc. Worked quite well.
>
> Hope this helps,
>
Hardly "big" by the OP's standards at only 4.5 MB (and roughly 790,000
words, if anyone's interested).
The problem with processing terabytes is frequently the need to
accomodate the data with techniques that allow the virtual memory to
avoid becoming swamped. The last thing you need to do under such
circumstances is create an in-memory copy of hte entire data set, since
on most real computers this will inflict swapping behavior.
regards
Steve
--
Steve Holden +1 703 861 4237 +1 800 494 3119
Holden Web LLC http://www.holdenweb.com/
Python Web Programming http://pydish.holdenweb.com/
More information about the Python-list
mailing list