HTTP or alternative upload for large files

[robin at execulink.com]

This may not be specifically a Python problem, so apologies in
advance.

I am attempting to upload large files (think 500 MB) through a web
form using a Python CGI process.

It is easy enough for my program to get the file handle of the
user-submitted file, and then write this out in chunks to the server.
But there are two major problems with this.

1. Apache first uploads the entire file itself, consuming all
available memory, before handing over control to the CGI process. Why
it does this is beyond me, since I am not hip to server internals.
Apache could at least use a restricted amount of memory to do the
upload, but this, unfortunately, is not the case. Bad things happen
when memory gets low.

2. The HTTP connection times out, so the entire file upload is lost.

I imagine the answer to this problem is "don't use HTTP", but I do
need a seemless solution from a web application.

Any pointers? I am sure this is a know problem. Hopefully there's a
known solution!

-- robin, media artist and python hacker