Joining Big Files

beginner zyzhu2000 at gmail.com
Sat Aug 25 20:13:30 EDT 2007


On Aug 25, 4:57 am, mosscliffe <mcl.off... at googlemail.com> wrote:
> I have 4 text files each approx 50mb.
>
> I need to join these into one large text file.
>
> I only need to do this very occasionally, as the problem has occurred
> because of upload limitations.
>
> Bearing in mind filesize and memory useage, would I be better reading
> every line in every file and writing each line to the output file or
> is there someway I could execute some shell command.
>
> I am working on a hosted website with no direct telnet or similar
> access.
>
> I would appreciate any tips, best code, as I do not want to break
> anything on the website.
>
> Richard

I would probably open the files in binary mode and copy 128 KB - 512
KB at a time. I wouldn't copy a line at a time as it is possible that
one of the files contains a very long line.





More information about the Python-list mailing list