HTTP or alternative upload for large files

[Steve Holden]

<robin at execulink.com> wrote ...
> This may not be specifically a Python problem, so apologies in
> advance.
>
You're right, the fact that it's a Python CGI makes no difference.

> I am attempting to upload large files (think 500 MB) through a web
> form using a Python CGI process.
>
!!!!

> It is easy enough for my program to get the file handle of the
> user-submitted file, and then write this out in chunks to the server.
> But there are two major problems with this.
>
> 1. Apache first uploads the entire file itself, consuming all
> available memory, before handing over control to the CGI process. Why
> it does this is beyond me, since I am not hip to server internals.
> Apache could at least use a restricted amount of memory to do the
> upload, but this, unfortunately, is not the case. Bad things happen
> when memory gets low.
>
> 2. The HTTP connection times out, so the entire file upload is lost.
>
Are you sure it's the HTTP connection that times out, or is the server
timing the process out (because it's including the file transfer time as a
part of the CGI processing time)? The reason I ask is because you may be
able to modify the latter. However there's no guarantee that the client
won't have problems being sufficiently responsive with a 0.5GB upload.

I'm not that familiar with Apache internals, but I do know that the server
expects to have the data ready for the CGI process before it starts it up.
The file content is simply encoded as part of a form/multipart response from
the client.

> I imagine the answer to this problem is "don't use HTTP", but I do
> need a seemless solution from a web application.
>
That's very tricky indeed. And I'm sure you're aware it isn't what HTTP was
designed for...

> Any pointers? I am sure this is a know problem. Hopefully there's a
> known solution!
>
Can't think of one, but fortunately this doesn't mean that one doesn't
exist.

regards
 Steve
--

Steve Holden: http://www.holdenweb.com/ ; Python Web Programming:
http://pydish.holdenweb.com/pwp/