[python-uk] Handling large file uploads

[Hansel Dunlop]

Hello all

I'm working on an application that has to accept large uploads. Think ~
2GB+ size files getting uploaded over slowish connections. These files are
eventually going to end up in S3.

Uploading smallish files is not a problem. But things get a bit complicated
when you're dealing with large files and load balanced servers. Servers
that can be replaced at any time. Has anyone done something similar?

My current plan is:

1. Accept chunked uploads. So the app/browser sends individual POST
requests with ~10mb chunks. Once that upload is complete the server
responds with a chunk id and the current offset
2. The server stores each intermediate chunk in a temporary S3 bucket
3. Once the final chunk has been uploaded the server kicks off another
process that stitches the pieces together and puts the whole file into it's
final location. And then deletes the intermediate pieces.

I think I have to do the file in chunks like this but maybe there is some
way to stream the files somewhere?

-- 

                                Hansel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-uk/attachments/20170411/4794f6e4/attachment.html>