HTTP or alternative upload for large files

[Jon Ribbens]

In article <0gofdukc6c10c0lg2t6qcadov2mqt2a7nu at 4ax.com>, robin at execulink.com wrote:
> This may not be specifically a Python problem, so apologies in
> advance.
> 
> I am attempting to upload large files (think 500 MB) through a web
> form using a Python CGI process.
> 
> It is easy enough for my program to get the file handle of the
> user-submitted file, and then write this out in chunks to the server.
> But there are two major problems with this.
> 
> 1. Apache first uploads the entire file itself, consuming all
> available memory, before handing over control to the CGI process.

I do not believe this to be true. It doesn't make any sense, and from
an inspection of the Apache code it doesn't appear to be the case.
Apache both reads the input to the script and the output from the
script in 8k chunks. Are you really seeing the httpd processes expand
to over 500MB?

Are you talking about a 500MB file being sent from the web server to
the web browser, or the web browser to the web server? You appear to
contradict yourself within your post as to which you are doing.

Oh, and the obvious answer is still of course "don't send 500MB files
by HTTP". It doesn't how much you want to, you're *always* going to
have problems.