[python-uk] Handling large file uploads

[Roger Gammans]

Indeed.

Both the rsync and bittorrent protocols split files into chunks. There
are a lot of similarities bwtween these protocols.

Also it might be worth looking at are de-duping backup tools like
duplicity.

On Tue, 2017-04-11 at 11:33 +0100, Javier Llopis wrote:
> I haven't done anything similar, but this upload in chunks then
> reassemble idea bears some similarity to peer to peer file sharing
> software. I would look there for ideas in case I got stuck with some
> problem.
> 
> 
>  
> 
> 
> J
> 
> 
>  
> 
> 
>  
> 
> 
> 
> From: "Hansel Dunlop" <hansel at interpretthis.org>
> 
> Sent: 11 April 2017 11:22
> 
> To: "UK Python Users" <python-uk at python.org>
> 
> Subject: [python-uk] Handling large file uploads
> 
>  
> 
> Hello all
>  
> 
> I'm working on an application that has to accept large uploads. Think
> ~ 2GB+ size files getting uploaded over slowish connections. These
> files are eventually going to end up in S3.
> 
>  
> 
> Uploading smallish files is not a problem. But things get a bit
> complicated when you're dealing with large files and load balanced
> servers. Servers that can be replaced at any time. Has anyone done
> something similar? 
> 
>  
> 
> My current plan is:
> 
>  
> 
> 1. Accept chunked uploads. So the app/browser sends individual POST
> requests with ~10mb chunks. Once that upload is complete the server
> responds with a chunk id and the current offset
> 
> 2. The server stores each intermediate chunk in a temporary S3 bucket
> 
> 3. Once the final chunk has been uploaded the server kicks off
> another process that stitches the pieces together and puts the whole
> file into it's final location. And then deletes the intermediate
> pieces. 
> 
>  
> 
> I think I have to do the file in chunks like this but maybe there is
> some way to stream the files somewhere? 
> 
> 
>  
> --
> 
> 
>  
> 
>                                 Hansel
> 
> 
> 
> 
> _______________________________________________
> python-uk mailing list
> python-uk at python.org
> https://mail.python.org/mailman/listinfo/python-uk

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-uk/attachments/20170411/f6dbeb3a/attachment.html>