[Flask] Flask Digest, Vol 24, Issue 8

Sun Jun 25 12:45:45 EDT 2017

Just check the docs (https://docs.mongodb.com/manual/core/gridfs/) for
filesystem vs Gridfs:

In some situations, storing large files may be more efficient in a MongoDB
database than on a system-level filesystem.

   - If your filesystem limits the number of files in a directory, you can
   use GridFS to store as many files as needed.
   - When you want to access information from portions of large files
   without having to load whole files into memory, you can use GridFS to
   recall sections of files without reading the entire file into memory.
   - When you want to keep your files and metadata automatically synced and
   deployed across a number of systems and facilities, you can use GridFS.
   When using geographically distributed replica sets
   <https://docs.mongodb.com/manual/core/replica-set-architecture-geographically-distributed/#replica-set-geographical-distribution>,
   MongoDB can distribute files and their metadata automatically to a number of
   mongod instances and facilities.

On 25 Jun 2017 5:38 pm, "G.S.Vijay Raajaa" <gsvijayraajaa at gmail.com> wrote:

> Hi UB,
>
> Thanks for the idea. I shall probe more into the same. But I have quick
> clarifications, even with mongo/gridfs and having the connectionI/O
> delegated to the database driver makes the transaction blocking? Kindly
> correct me if I am wrong. Is there a async way of handling the upload to
> mongo and keep flask serve other incoming request?
>
> On the same note, w.r.t mongodb-gridFS, how do you think the write
> performance is when compared to storing them directly on the filesystem,
> same w.r.t to read for serving?
>
> I like to idea of scalability when you have file store within DB.
>
> Regards,
> Vijay Raajaa GS
>
> On Sun, Jun 25, 2017 at 9:30 PM, <flask-request at python.org> wrote:
>
>> Send Flask mailing list submissions to
>>         flask at python.org
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>>         https://mail.python.org/mailman/listinfo/flask
>> or, via email, send a message with subject or body 'help' to
>>         flask-request at python.org
>>
>> You can reach the person managing the list at
>>         flask-owner at python.org
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of Flask digest..."
>>
>>
>> Today's Topics:
>>
>>    1. REST based File Sharing server Implementation in Flask
>>       (G.S.Vijay Raajaa)
>>    2. Re: REST based File Sharing server Implementation in Flask
>>       (ub at artfacts.net)
>>
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Sun, 25 Jun 2017 20:08:34 +0530
>> From: "G.S.Vijay Raajaa" <gsvijayraajaa at gmail.com>
>> To: flask at python.org
>> Subject: [Flask] REST based File Sharing server Implementation in
>>         Flask
>> Message-ID:
>>         <CAB9NB9aNRbZTdLNMWO3k=gDxGW-E0gcHZF0d4X=_b41CPTSFAA at mail.gm
>> ail.com>
>> Content-Type: text/plain; charset="utf-8"
>>
>> Hi,
>>
>> I am trying to build a REST based file sharing server on top of flask. I
>> am
>> looking for a scalable stack.
>>
>> The requirement goes as follows;
>>
>> 1) Cater multiple file uploads.
>> 2) Files can be of the order of 30-100GB. The server hosting the python
>> webserver should not read the file to memory and spill to disk.
>> 3) Non blocking - when user uploads multiple files.
>> 4) Non blocking - when multiple user try to upload files.
>>
>> Pointers to any existing open source project is also much appreciated.
>>
>> Idea inspired from https://filebin.net/
>>
>> Seeing if celery + Flask can come handy. Kindly suggest the WSGI to use it
>> for production too.
>>
>> Looking forward to your valuable suggestions.
>>
>> Regards,
>> Vijay Raajaa GS
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: <http://mail.python.org/pipermail/flask/attachments/20170625
>> /b824c18b/attachment-0001.html>
>>
>> ------------------------------
>>
>> Message: 2
>> Date: Sun, 25 Jun 2017 17:47:07 +0200
>> From: "ub at artfacts.net" <ub at artfacts.net>
>> To: flask at python.org
>> Subject: Re: [Flask] REST based File Sharing server Implementation in
>>         Flask
>> Message-ID: <d0a2d934-452f-5471-fd0f-422918cc4c71 at artfacts.net>
>> Content-Type: text/plain; charset=utf-8
>>
>> hey vijay,
>>
>> you could use mongodb's gridfs. that way you hand over connection/IO
>> control to the database driver and scaling would be determined by
>> mongodb's scaling potential.
>>
>> there's multiple exmaples online:
>> https://gist.github.com/artisonian/492713
>> https://stackoverflow.com/questions/32492501/list-and-serve-
>> files-from-gridfs-with-flask
>> https://github.com/RedBeard0531/python-gridfs-server/blob/
>> master/gridfs_server.py
>> ...
>>
>> i personally haven't used mongodb for quite some time, so this is just
>> an idea.
>>
>> cheers,
>> ub
>>
>>
>> On 25.06.2017 16:38, G.S.Vijay Raajaa wrote:
>> > Hi,
>> >
>> > I am trying to build a REST based file sharing server on top of flask. I
>> > am looking for a scalable stack.
>> >
>> > The requirement goes as follows;
>> >
>> > 1) Cater multiple file uploads.
>> > 2) Files can be of the order of 30-100GB. The server hosting the python
>> > webserver should not read the file to memory and spill to disk.
>> > 3) Non blocking - when user uploads multiple files.
>> > 4) Non blocking - when multiple user try to upload files.
>> >
>> > Pointers to any existing open source project is also much appreciated.
>> >
>> > Idea inspired from https://filebin.net/
>> >
>> > Seeing if celery + Flask can come handy. Kindly suggest the WSGI to use
>> > it for production too.
>> >
>> > Looking forward to your valuable suggestions.
>> >
>> > Regards,
>> > Vijay Raajaa GS
>> >
>> >
>> > _______________________________________________
>> > Flask mailing list
>> > Flask at python.org
>> > https://mail.python.org/mailman/listinfo/flask
>> >
>>
>>
>>
>> ------------------------------
>>
>> Subject: Digest Footer
>>
>> _______________________________________________
>> Flask mailing list
>> Flask at python.org
>> https://mail.python.org/mailman/listinfo/flask
>>
>>
>> ------------------------------
>>
>> End of Flask Digest, Vol 24, Issue 8
>> ************************************
>>
>
>
> _______________________________________________
> Flask mailing list
> Flask at python.org
> https://mail.python.org/mailman/listinfo/flask
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/flask/attachments/20170625/0a421b8f/attachment-0001.html>