From amundsen.craig at gene.com Fri Jun 2 09:48:13 2017 From: amundsen.craig at gene.com (Craig Amundsen) Date: Fri, 2 Jun 2017 06:48:13 -0700 Subject: [Flask] g/session/request Message-ID: Hi - I'm trying to implement a sortable table using Flask-Table. My dummy test works great. But querying my database and building the list of dictionaries that the Table instance uses takes a certain amount of time. I'd like to keep that list of dictionaries around so when the User sorts on a different column I don't have to build it again. Currently I have one page where the User indicates which information they want in the table. When they click on the submit button I query the database, construct the dictionary, store the dictionary in the session, and then call view that shows the table. That method grabs the dictionary out of the session and constructs the table which is then sent to the html template. This all works great. Clicking on a column heading ends up calling the view that shows the table again. This time, though, the session doesn't have the dictionary and so an error gets thrown. I tried messing about with g, but I can't get that to work either. Does anyone have any hints on how I can save list so it's still available when I sort the table? Here's some snippets to show what I'm doing: @main.route('/select-data', methods = ['GET', 'POST']) @login_required @view_data_required def select_data(): form = SelectDataForm() if form.validate_on_submit(): colsToShow = { 'showName' : True, 'showX' : form.showX.data, 'showY' : form.showY.data, 'showZ' : form.showZ.data } session.dataFlags = colsToShow # Build the list that Table wants session.theList = theList return show_data() return render_template(select_data.html", form = form) @main.route('/show-data') @login_required @view_data_required def show_data(): colsToShow = session.dataFlags TableClass = create_table() for key, attr, header in [('showName', 'name', "Name"), ('showX', 'x', "X"), ("showY", "y", "Y"), ("showZ", "z", "Z")]: if colsToShow[key]: TableClass.add_column(attr, Col(header)) TableClass.allow_sort = True TableClass.sort_url = sort_url TableClass.classes = ["table", "table-hover", "followers"] sort = request.args.get('sort', 'name') direction = request.args.get('direction', 'asc') reverse = (request.args.get('direction', 'asc') == 'desc') theList = session.theList theList = sorted(theList, key = lambda x: x[sort], reverse = reverse) table = TableClass(things, sort_by = sort, sort_reverse = reverse) return render_template("dump_data.html", table = table, sort = sort, direction = direction) def sort_url(self, col_key, reverse=False): if reverse: direction = 'desc' else: direction = 'asc' return url_for('.show_data', sort=col_key, direction=direction) I've tried re-assigning session.theList and session.colsToShow at the bottom of show_data, but that doesn't work. I've tried messing around with with g instead of session, but I don't seem to grasp its use any better. If anyone has any hints on making the session/g/request know about the two things I need to store there the second time I call show_data (by clicking on a column header), I'd be most appreciative. Thanks, - Craig -------------- next part -------------- An HTML attachment was scrubbed... URL: From coreybrett at gmail.com Fri Jun 2 10:07:50 2017 From: coreybrett at gmail.com (Corey Boyle) Date: Fri, 2 Jun 2017 10:07:50 -0400 Subject: [Flask] g/session/request In-Reply-To: References: Message-ID: I could be wrong about this, but... The default session storage is done with cookies, so if you store the data in the session, all that data will get shipped back and forth with each request made by the client. Also, cookies have a storage limit that may be an issue. The "g" proxy object is tied to the request context, so anything stored on it will be garbage collected when the request is fulfilled. Option 1 - Create a new function outside your view function used solely for building your data. Either write some custom caching code inside that function, or use something like Flask-Caching. Then call that function inside your view function. Option 2 - Build the table in HTML as you are now, and allow the sorting to be done on the client side using JS. DataTables (https://datatables.net) is AMAZING and actually pretty easy to figure out. Option 3 - Option 1 + Option 2... Doing the sorting on the client side would be much more efficient, and caching the data on the server side would be advisable is you are expecting to handle many requests simultaneously. On Fri, Jun 2, 2017 at 9:48 AM, Craig Amundsen wrote: > Hi - > > I'm trying to implement a sortable table using Flask-Table. My dummy test > works great. But querying my database and building the list of dictionaries > that the Table instance uses takes a certain amount of time. I'd like to > keep that list of dictionaries around so when the User sorts on a different > column I don't have to build it again. > > Currently I have one page where the User indicates which information they > want in the table. When they click on the submit button I query the > database, construct the dictionary, store the dictionary in the session, and > then call view that shows the table. That method grabs the dictionary out of > the session and constructs the table which is then sent to the html > template. This all works great. > > Clicking on a column heading ends up calling the view that shows the table > again. This time, though, the session doesn't have the dictionary and so an > error gets thrown. > > I tried messing about with g, but I can't get that to work either. Does > anyone have any hints on how I can save list so it's still available when I > sort the table? > > Here's some snippets to show what I'm doing: > > @main.route('/select-data', methods = ['GET', 'POST']) > @login_required > @view_data_required > def select_data(): > form = SelectDataForm() > if form.validate_on_submit(): > colsToShow = { 'showName' : True, > 'showX' : form.showX.data, > 'showY' : form.showY.data, > 'showZ' : form.showZ.data } > session.dataFlags = colsToShow > > # Build the list that Table wants > > session.theList = theList > return show_data() > return render_template(select_data.html", form = form) > > @main.route('/show-data') > @login_required > @view_data_required > def show_data(): > colsToShow = session.dataFlags > > TableClass = create_table() > for key, attr, header in [('showName', 'name', "Name"), ('showX', 'x', > "X"), > ("showY", "y", "Y"), ("showZ", "z", "Z")]: > if colsToShow[key]: > TableClass.add_column(attr, Col(header)) > > TableClass.allow_sort = True > TableClass.sort_url = sort_url > TableClass.classes = ["table", "table-hover", "followers"] > > sort = request.args.get('sort', 'name') > direction = request.args.get('direction', 'asc') > reverse = (request.args.get('direction', 'asc') == 'desc') > > theList = session.theList > theList = sorted(theList, key = lambda x: x[sort], reverse = reverse) > table = TableClass(things, sort_by = sort, sort_reverse = reverse) > > return render_template("dump_data.html", table = table, sort = sort, > direction = direction) > > def sort_url(self, col_key, reverse=False): > if reverse: direction = 'desc' > else: direction = 'asc' > return url_for('.show_data', sort=col_key, direction=direction) > > > I've tried re-assigning session.theList and session.colsToShow at the bottom > of show_data, but that doesn't work. I've tried messing around with with g > instead of session, but I don't seem to grasp its use any better. > > If anyone has any hints on making the session/g/request know about the two > things I need to store there the second time I call show_data (by clicking > on a column header), I'd be most appreciative. > > Thanks, > - Craig > > _______________________________________________ > Flask mailing list > Flask at python.org > https://mail.python.org/mailman/listinfo/flask > From gergely at polonkai.eu Sat Jun 3 01:45:34 2017 From: gergely at polonkai.eu (Gergely Polonkai) Date: Sat, 03 Jun 2017 05:45:34 +0000 Subject: [Flask] g/session/request In-Reply-To: References: Message-ID: On Fri, Jun 2, 2017, 16:08 Corey Boyle wrote: > I could be wrong about this, but... > > The default session storage is done with cookies, so if you store the > data in the session, all that data will get shipped back and forth > with each request made by the client. Also, cookies have a storage > limit that may be an issue. > > The "g" proxy object is tied to the request context, so anything > stored on it will be garbage collected when the request is fulfilled. > Not anymore. With 0.10, it is tied to app context, so it doesn't get collected, but introduces another problem: every user of the system will see the same data. > Option 1 - Create a new function outside your view function used > solely for building your data. Either write some custom caching code > inside that function, or use something like Flask-Caching. Then call > that function inside your view function. > +1 on Flask-Caching, but beware if you fetch your data through an ORM like SQLAlchemy. They don't play nice with caching. > Option 2 - Build the table in HTML as you are now, and allow the > sorting to be done on the client side using JS. DataTables > (https://datatables.net) is AMAZING and actually pretty easy to figure > out. > I was about to suggest the same, although there's probably a reason OP didn't do it that way. If this is a graphical browser only app, though, it's not a problem. > Option 3 - Option 1 + Option 2... Doing the sorting on the client side > would be much more efficient, and caching the data on the server side > would be advisable is you are expecting to handle many requests > simultaneously. > > On Fri, Jun 2, 2017 at 9:48 AM, Craig Amundsen > wrote: > > Hi - > > > > I'm trying to implement a sortable table using Flask-Table. My dummy test > > works great. But querying my database and building the list of > dictionaries > > that the Table instance uses takes a certain amount of time. I'd like to > > keep that list of dictionaries around so when the User sorts on a > different > > column I don't have to build it again. > > > > Currently I have one page where the User indicates which information they > > want in the table. When they click on the submit button I query the > > database, construct the dictionary, store the dictionary in the session, > and > > then call view that shows the table. That method grabs the dictionary > out of > > the session and constructs the table which is then sent to the html > > template. This all works great. > > > > Clicking on a column heading ends up calling the view that shows the > table > > again. This time, though, the session doesn't have the dictionary and so > an > > error gets thrown. > > > > I tried messing about with g, but I can't get that to work either. Does > > anyone have any hints on how I can save list so it's still available > when I > > sort the table? > > > > Here's some snippets to show what I'm doing: > > > > @main.route('/select-data', methods = ['GET', 'POST']) > > @login_required > > @view_data_required > > def select_data(): > > form = SelectDataForm() > > if form.validate_on_submit(): > > colsToShow = { 'showName' : True, > > 'showX' : form.showX.data, > > 'showY' : form.showY.data, > > 'showZ' : form.showZ.data } > > session.dataFlags = colsToShow > > > > # Build the list that Table wants > > > > session.theList = theList > > return show_data() > > return render_template(select_data.html", form = form) > > > > @main.route('/show-data') > > @login_required > > @view_data_required > > def show_data(): > > colsToShow = session.dataFlags > > > > TableClass = create_table() > > for key, attr, header in [('showName', 'name', "Name"), ('showX', > 'x', > > "X"), > > ("showY", "y", "Y"), ("showZ", "z", "Z")]: > > if colsToShow[key]: > > TableClass.add_column(attr, Col(header)) > > > > TableClass.allow_sort = True > > TableClass.sort_url = sort_url > > TableClass.classes = ["table", "table-hover", "followers"] > > > > sort = request.args.get('sort', 'name') > > direction = request.args.get('direction', 'asc') > > reverse = (request.args.get('direction', 'asc') == 'desc') > > > > theList = session.theList > > theList = sorted(theList, key = lambda x: x[sort], reverse = reverse) > > table = TableClass(things, sort_by = sort, sort_reverse = reverse) > > > > return render_template("dump_data.html", table = table, sort = sort, > > direction = direction) > > > > def sort_url(self, col_key, reverse=False): > > if reverse: direction = 'desc' > > else: direction = 'asc' > > return url_for('.show_data', sort=col_key, direction=direction) > > > > > > I've tried re-assigning session.theList and session.colsToShow at the > bottom > > of show_data, but that doesn't work. I've tried messing around with with > g > > instead of session, but I don't seem to grasp its use any better. > > > > If anyone has any hints on making the session/g/request know about the > two > > things I need to store there the second time I call show_data (by > clicking > > on a column header), I'd be most appreciative. > > > > Thanks, > > - Craig > > > > _______________________________________________ > > Flask mailing list > > Flask at python.org > > https://mail.python.org/mailman/listinfo/flask > > > _______________________________________________ > Flask mailing list > Flask at python.org > https://mail.python.org/mailman/listinfo/flask > -------------- next part -------------- An HTML attachment was scrubbed... URL: From coreybrett at gmail.com Sat Jun 3 08:50:19 2017 From: coreybrett at gmail.com (Corey Boyle) Date: Sat, 3 Jun 2017 08:50:19 -0400 Subject: [Flask] g/session/request In-Reply-To: References: Message-ID: Replies inline below. __ Corey On Jun 3, 2017 1:45 AM, "Gergely Polonkai" wrote: On Fri, Jun 2, 2017, 16:08 Corey Boyle wrote: > I could be wrong about this, but... > > The default session storage is done with cookies, so if you store the > data in the session, all that data will get shipped back and forth > with each request made by the client. Also, cookies have a storage > limit that may be an issue. > > The "g" proxy object is tied to the request context, so anything > stored on it will be garbage collected when the request is fulfilled. > Not anymore. With 0.10, it is tied to app context, so it doesn't get collected, but introduces another problem: every user of the system will see the same data. *Yep, I messed up the details on g. However, it still can't be used for storing data between requests.* > Option 1 - Create a new function outside your view function used > solely for building your data. Either write some custom caching code > inside that function, or use something like Flask-Caching. Then call > that function inside your view function. > +1 on Flask-Caching, but beware if you fetch your data through an ORM like SQLAlchemy. They don't play nice with caching. *If I understood correctly, the OP is building a Python dictionary from his data source, so the dictionary should be easily serialized by Flask-Caching. The reason I suggested moving that code into an isolated function, was that he could potentially use one of the provided decorators from that extension.* > Option 2 - Build the table in HTML as you are now, and allow the > sorting to be done on the client side using JS. DataTables > (https://datatables.net) is AMAZING and actually pretty easy to figure > out. > I was about to suggest the same, although there's probably a reason OP didn't do it that way. If this is a graphical browser only app, though, it's not a problem. > Option 3 - Option 1 + Option 2... Doing the sorting on the client side > would be much more efficient, and caching the data on the server side > would be advisable is you are expecting to handle many requests > simultaneously. > > On Fri, Jun 2, 2017 at 9:48 AM, Craig Amundsen > wrote: > > Hi - > > > > I'm trying to implement a sortable table using Flask-Table. My dummy test > > works great. But querying my database and building the list of > dictionaries > > that the Table instance uses takes a certain amount of time. I'd like to > > keep that list of dictionaries around so when the User sorts on a > different > > column I don't have to build it again. > > > > Currently I have one page where the User indicates which information they > > want in the table. When they click on the submit button I query the > > database, construct the dictionary, store the dictionary in the session, > and > > then call view that shows the table. That method grabs the dictionary > out of > > the session and constructs the table which is then sent to the html > > template. This all works great. > > > > Clicking on a column heading ends up calling the view that shows the > table > > again. This time, though, the session doesn't have the dictionary and so > an > > error gets thrown. > > > > I tried messing about with g, but I can't get that to work either. Does > > anyone have any hints on how I can save list so it's still available > when I > > sort the table? > > > > Here's some snippets to show what I'm doing: > > > > @main.route('/select-data', methods = ['GET', 'POST']) > > @login_required > > @view_data_required > > def select_data(): > > form = SelectDataForm() > > if form.validate_on_submit(): > > colsToShow = { 'showName' : True, > > 'showX' : form.showX.data, > > 'showY' : form.showY.data, > > 'showZ' : form.showZ.data } > > session.dataFlags = colsToShow > > > > # Build the list that Table wants > > > > session.theList = theList > > return show_data() > > return render_template(select_data.html", form = form) > > > > @main.route('/show-data') > > @login_required > > @view_data_required > > def show_data(): > > colsToShow = session.dataFlags > > > > TableClass = create_table() > > for key, attr, header in [('showName', 'name', "Name"), ('showX', > 'x', > > "X"), > > ("showY", "y", "Y"), ("showZ", "z", "Z")]: > > if colsToShow[key]: > > TableClass.add_column(attr, Col(header)) > > > > TableClass.allow_sort = True > > TableClass.sort_url = sort_url > > TableClass.classes = ["table", "table-hover", "followers"] > > > > sort = request.args.get('sort', 'name') > > direction = request.args.get('direction', 'asc') > > reverse = (request.args.get('direction', 'asc') == 'desc') > > > > theList = session.theList > > theList = sorted(theList, key = lambda x: x[sort], reverse = reverse) > > table = TableClass(things, sort_by = sort, sort_reverse = reverse) > > > > return render_template("dump_data.html", table = table, sort = sort, > > direction = direction) > > > > def sort_url(self, col_key, reverse=False): > > if reverse: direction = 'desc' > > else: direction = 'asc' > > return url_for('.show_data', sort=col_key, direction=direction) > > > > > > I've tried re-assigning session.theList and session.colsToShow at the > bottom > > of show_data, but that doesn't work. I've tried messing around with with > g > > instead of session, but I don't seem to grasp its use any better. > > > > If anyone has any hints on making the session/g/request know about the > two > > things I need to store there the second time I call show_data (by > clicking > > on a column header), I'd be most appreciative. > > > > Thanks, > > - Craig > > > > _______________________________________________ > > Flask mailing list > > Flask at python.org > > https://mail.python.org/mailman/listinfo/flask > > > _______________________________________________ > Flask mailing list > Flask at python.org > https://mail.python.org/mailman/listinfo/flask > -------------- next part -------------- An HTML attachment was scrubbed... URL: From amundsen.craig at gene.com Mon Jun 5 11:16:41 2017 From: amundsen.craig at gene.com (Craig Amundsen) Date: Mon, 5 Jun 2017 08:16:41 -0700 Subject: [Flask] g/session/request In-Reply-To: References: Message-ID: OP here. I didn't initially try using DataTables because during my initial search for flask sortable table I found the Flask-Table library, but did not see the DataTables extension of jquery. It does seem like a better solution to my problem, now I just have to figure out how to get jquery up and running in my app... Thanks for the responses. - Craig On Sat, Jun 3, 2017 at 5:50 AM, Corey Boyle wrote: > Replies inline below. > > > __ > Corey > > On Jun 3, 2017 1:45 AM, "Gergely Polonkai" wrote: > > > > On Fri, Jun 2, 2017, 16:08 Corey Boyle wrote: > >> I could be wrong about this, but... >> >> The default session storage is done with cookies, so if you store the >> data in the session, all that data will get shipped back and forth >> with each request made by the client. Also, cookies have a storage >> limit that may be an issue. >> >> The "g" proxy object is tied to the request context, so anything >> stored on it will be garbage collected when the request is fulfilled. >> > > Not anymore. With 0.10, it is tied to app context, so it doesn't get > collected, but introduces another problem: every user of the system will > see the same data. > > > *Yep, I messed up the details on g. However, it still can't be used for > storing data between requests.* > > > > >> Option 1 - Create a new function outside your view function used >> solely for building your data. Either write some custom caching code >> inside that function, or use something like Flask-Caching. Then call >> that function inside your view function. >> > > +1 on Flask-Caching, but beware if you fetch your data through an ORM like > SQLAlchemy. They don't play nice with caching. > > > *If I understood correctly, the OP is building a Python dictionary from > his data source, so the dictionary should be easily serialized by > Flask-Caching. The reason I suggested moving that code into an isolated > function, was that he could potentially use one of the provided decorators > from that extension.* > > > > >> Option 2 - Build the table in HTML as you are now, and allow the >> sorting to be done on the client side using JS. DataTables >> (https://datatables.net) is AMAZING and actually pretty easy to figure >> out. >> > > I was about to suggest the same, although there's probably a reason OP > didn't do it that way. If this is a graphical browser only app, though, > it's not a problem. > > >> Option 3 - Option 1 + Option 2... Doing the sorting on the client side >> would be much more efficient, and caching the data on the server side >> would be advisable is you are expecting to handle many requests >> simultaneously. >> >> On Fri, Jun 2, 2017 at 9:48 AM, Craig Amundsen >> wrote: >> > Hi - >> > >> > I'm trying to implement a sortable table using Flask-Table. My dummy >> test >> > works great. But querying my database and building the list of >> dictionaries >> > that the Table instance uses takes a certain amount of time. I'd like to >> > keep that list of dictionaries around so when the User sorts on a >> different >> > column I don't have to build it again. >> > >> > Currently I have one page where the User indicates which information >> they >> > want in the table. When they click on the submit button I query the >> > database, construct the dictionary, store the dictionary in the >> session, and >> > then call view that shows the table. That method grabs the dictionary >> out of >> > the session and constructs the table which is then sent to the html >> > template. This all works great. >> > >> > Clicking on a column heading ends up calling the view that shows the >> table >> > again. This time, though, the session doesn't have the dictionary and >> so an >> > error gets thrown. >> > >> > I tried messing about with g, but I can't get that to work either. Does >> > anyone have any hints on how I can save list so it's still available >> when I >> > sort the table? >> > >> > Here's some snippets to show what I'm doing: >> > >> > @main.route('/select-data', methods = ['GET', 'POST']) >> > @login_required >> > @view_data_required >> > def select_data(): >> > form = SelectDataForm() >> > if form.validate_on_submit(): >> > colsToShow = { 'showName' : True, >> > 'showX' : form.showX.data, >> > 'showY' : form.showY.data, >> > 'showZ' : form.showZ.data } >> > session.dataFlags = colsToShow >> > >> > # Build the list that Table wants >> > >> > session.theList = theList >> > return show_data() >> > return render_template(select_data.html", form = form) >> > >> > @main.route('/show-data') >> > @login_required >> > @view_data_required >> > def show_data(): >> > colsToShow = session.dataFlags >> > >> > TableClass = create_table() >> > for key, attr, header in [('showName', 'name', "Name"), ('showX', >> 'x', >> > "X"), >> > ("showY", "y", "Y"), ("showZ", "z", "Z")]: >> > if colsToShow[key]: >> > TableClass.add_column(attr, Col(header)) >> > >> > TableClass.allow_sort = True >> > TableClass.sort_url = sort_url >> > TableClass.classes = ["table", "table-hover", "followers"] >> > >> > sort = request.args.get('sort', 'name') >> > direction = request.args.get('direction', 'asc') >> > reverse = (request.args.get('direction', 'asc') == 'desc') >> > >> > theList = session.theList >> > theList = sorted(theList, key = lambda x: x[sort], reverse = >> reverse) >> > table = TableClass(things, sort_by = sort, sort_reverse = reverse) >> > >> > return render_template("dump_data.html", table = table, sort = >> sort, >> > direction = direction) >> > >> > def sort_url(self, col_key, reverse=False): >> > if reverse: direction = 'desc' >> > else: direction = 'asc' >> > return url_for('.show_data', sort=col_key, direction=direction) >> > >> > >> > I've tried re-assigning session.theList and session.colsToShow at the >> bottom >> > of show_data, but that doesn't work. I've tried messing around with >> with g >> > instead of session, but I don't seem to grasp its use any better. >> > >> > If anyone has any hints on making the session/g/request know about the >> two >> > things I need to store there the second time I call show_data (by >> clicking >> > on a column header), I'd be most appreciative. >> > >> > Thanks, >> > - Craig >> > >> > _______________________________________________ >> > Flask mailing list >> > Flask at python.org >> > https://mail.python.org/mailman/listinfo/flask >> > >> _______________________________________________ >> Flask mailing list >> Flask at python.org >> https://mail.python.org/mailman/listinfo/flask >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gergely at polonkai.eu Wed Jun 21 11:27:42 2017 From: gergely at polonkai.eu (Gergely Polonkai) Date: Wed, 21 Jun 2017 15:27:42 +0000 Subject: [Flask] [ANN] Flask-SQLAlchemy-WebQuery Message-ID: Hello, I?ve started working on a Flask and SQLAlchemy based library to query SQLAlchemy models through a web interface. It can be downloaded here: https://gitlab.com/gergelypolonkai/flask-sqlalchemy-webquery I developed it with Python3 in mind; it may or may not work with Python2. Also, the web interface currently depends on Flask-Bootstrap and uses CSRF tokens from Flask-WTF, but these could be easily eliminated. I would be grateful to get some comments and ideas about it. Also, if you find some code in it that makes you scream, don?t keep it in yourself! And if you know about a library that does the same (and thus, I should stop working on this one instead of contributing to that one), please let me know! Best, Gergely -------------- next part -------------- An HTML attachment was scrubbed... URL: From coreybrett at gmail.com Wed Jun 21 14:17:51 2017 From: coreybrett at gmail.com (Corey Boyle) Date: Wed, 21 Jun 2017 14:17:51 -0400 Subject: [Flask] [ANN] Flask-SQLAlchemy-WebQuery In-Reply-To: References: Message-ID: How would it compare to Flask-Admin? On Wed, Jun 21, 2017 at 11:27 AM, Gergely Polonkai wrote: > Hello, > > I?ve started working on a Flask and SQLAlchemy based library to query > SQLAlchemy models through a web interface. It can be downloaded here: > https://gitlab.com/gergelypolonkai/flask-sqlalchemy-webquery > > I developed it with Python3 in mind; it may or may not work with Python2. > Also, the web interface currently depends on Flask-Bootstrap and uses CSRF > tokens from Flask-WTF, but these could be easily eliminated. > > I would be grateful to get some comments and ideas about it. Also, if you > find some code in it that makes you scream, don?t keep it in yourself! And > if you know about a library that does the same (and thus, I should stop > working on this one instead of contributing to that one), please let me > know! > > Best, > Gergely > > _______________________________________________ > Flask mailing list > Flask at python.org > https://mail.python.org/mailman/listinfo/flask > From gergely at polonkai.eu Wed Jun 21 15:12:19 2017 From: gergely at polonkai.eu (Gergely Polonkai) Date: Wed, 21 Jun 2017 19:12:19 +0000 Subject: [Flask] [ANN] Flask-SQLAlchemy-WebQuery In-Reply-To: References: Message-ID: Flask-Admin is a generic administration interface, providing access to SQLAlchemy models, files, forms, etc. The goal of this module is to provide an interface to query SQLAlchemy models, and allow the users to build complex(ish) queries taking all related models into account. The workflow is (roughly) as follows: ? you choose a base model ? you are then presented with the model's fields, together with all the related models and their fields, and the models related to those, etc. ? you build the list of fields you want in the output ? you set up some filters * ? you see a bunch of rows while you are selecting fields (10 by default) ? when you are done, you can view all the records (not just the 10 "example" rows), paginated * ? ?and you can save the results in CSV format * * not implemented yet The idea was to present my colleagues with such an interface, so they can do all the "complex" queries they request from me on a daily basis. Best, Gergely On Wed, Jun 21, 2017, 20:18 Corey Boyle wrote: > How would it compare to Flask-Admin? > > On Wed, Jun 21, 2017 at 11:27 AM, Gergely Polonkai > wrote: > > Hello, > > > > I?ve started working on a Flask and SQLAlchemy based library to query > > SQLAlchemy models through a web interface. It can be downloaded here: > > https://gitlab.com/gergelypolonkai/flask-sqlalchemy-webquery > > > > I developed it with Python3 in mind; it may or may not work with Python2. > > Also, the web interface currently depends on Flask-Bootstrap and uses > CSRF > > tokens from Flask-WTF, but these could be easily eliminated. > > > > I would be grateful to get some comments and ideas about it. Also, if you > > find some code in it that makes you scream, don?t keep it in yourself! > And > > if you know about a library that does the same (and thus, I should stop > > working on this one instead of contributing to that one), please let me > > know! > > > > Best, > > Gergely > > > > _______________________________________________ > > Flask mailing list > > Flask at python.org > > https://mail.python.org/mailman/listinfo/flask > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From coreybrett at gmail.com Wed Jun 21 15:25:01 2017 From: coreybrett at gmail.com (Corey Boyle) Date: Wed, 21 Jun 2017 15:25:01 -0400 Subject: [Flask] [ANN] Flask-SQLAlchemy-WebQuery In-Reply-To: References: Message-ID: Sounds very interesting. I like the row previews via ajax. On Wed, Jun 21, 2017 at 3:12 PM, Gergely Polonkai wrote: > Flask-Admin is a generic administration interface, providing access to > SQLAlchemy models, files, forms, etc. > > The goal of this module is to provide an interface to query SQLAlchemy > models, and allow the users to build complex(ish) queries taking all related > models into account. > > The workflow is (roughly) as follows: > > ? you choose a base model > ? you are then presented with the model's fields, together with all the > related models and their fields, and the models related to those, etc. > ? you build the list of fields you want in the output > ? you set up some filters * > ? you see a bunch of rows while you are selecting fields (10 by default) > ? when you are done, you can view all the records (not just the 10 "example" > rows), paginated * > ? ?and you can save the results in CSV format * > > * not implemented yet > > The idea was to present my colleagues with such an interface, so they can do > all the "complex" queries they request from me on a daily basis. > > Best, > Gergely > > > On Wed, Jun 21, 2017, 20:18 Corey Boyle wrote: >> >> How would it compare to Flask-Admin? >> >> On Wed, Jun 21, 2017 at 11:27 AM, Gergely Polonkai >> wrote: >> > Hello, >> > >> > I?ve started working on a Flask and SQLAlchemy based library to query >> > SQLAlchemy models through a web interface. It can be downloaded here: >> > https://gitlab.com/gergelypolonkai/flask-sqlalchemy-webquery >> > >> > I developed it with Python3 in mind; it may or may not work with >> > Python2. >> > Also, the web interface currently depends on Flask-Bootstrap and uses >> > CSRF >> > tokens from Flask-WTF, but these could be easily eliminated. >> > >> > I would be grateful to get some comments and ideas about it. Also, if >> > you >> > find some code in it that makes you scream, don?t keep it in yourself! >> > And >> > if you know about a library that does the same (and thus, I should stop >> > working on this one instead of contributing to that one), please let me >> > know! >> > >> > Best, >> > Gergely >> > >> > _______________________________________________ >> > Flask mailing list >> > Flask at python.org >> > https://mail.python.org/mailman/listinfo/flask >> > From kikocorreoso at gmail.com Thu Jun 22 02:24:08 2017 From: kikocorreoso at gmail.com (Kiko) Date: Thu, 22 Jun 2017 08:24:08 +0200 Subject: [Flask] Flask-security or other alternatives Message-ID: Hi all, After reading most used tutorials [1] [2] and most of the books about flask [3] [4] [5] [6] [7] [8] [9] I can't see the use of Flask-security on any of them even it provides basic authentication, mail confirmation, reset/change password,..., out of the box. Is there any reason for that? Do you discourage the use of this extension for some reason? Do you recommend it for the typical login/regster needs of a website (basic authentication, mail confirmation, reset/change password) in order to not reinventing the wheel? If you recommend it, do you know of good examples implementing this stuff? Thanks in advance. kind regards. [1] https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-now-with-python-3-support [2] https://realpython.com/blog/python/handling-email-confirmation-in-flask/ [3] http://shop.oreilly.com/product/0636920031116.do [4] https://www.packtpub.com/web-development/flask-example [5] https://www.packtpub.com/web-development/flask-blueprints [6] https://www.packtpub.com/web-development/learning-flask-framework [7] https://www.packtpub.com/web-development/mastering-flask [8] https://www.packtpub.com/web-development/building-web-applications-flask [9] https://www.packtpub.com/web-development/flask-framework-cookbook -------------- next part -------------- An HTML attachment was scrubbed... URL: From projetmbc at gmail.com Thu Jun 22 04:11:08 2017 From: projetmbc at gmail.com (Christophe Bal) Date: Thu, 22 Jun 2017 10:11:08 +0200 Subject: [Flask] Flask-security or other alternatives In-Reply-To: References: Message-ID: Take a look here : https://blog.miguelgrinberg.com/post/handling- authentication-secrets-in-the-browser Christophe BAL Enseignant Agr?g? de Math?matiques Programmeur Python Amateur Le 22 juin 2017 08:24, "Kiko" a ?crit : > Hi all, > > After reading most used tutorials [1] [2] and most of the books about > flask [3] [4] [5] [6] [7] [8] [9] I can't see the use of Flask-security on > any of them even it provides basic authentication, mail confirmation, > reset/change password,..., out of the box. > > Is there any reason for that? Do you discourage the use of this extension > for some reason? Do you recommend it for the typical login/regster needs of > a website (basic authentication, mail confirmation, reset/change password) > in order to not reinventing the wheel? If you recommend it, do you know of > good examples implementing this stuff? > > Thanks in advance. > > kind regards. > > [1] https://blog.miguelgrinberg.com/post/the-flask-mega- > tutorial-now-with-python-3-support > [2] https://realpython.com/blog/python/handling-email- > confirmation-in-flask/ > [3] http://shop.oreilly.com/product/0636920031116.do > [4] https://www.packtpub.com/web-development/flask-example > [5] https://www.packtpub.com/web-development/flask-blueprints > [6] https://www.packtpub.com/web-development/learning-flask-framework > [7] https://www.packtpub.com/web-development/mastering-flask > [8] https://www.packtpub.com/web-development/building-web- > applications-flask > [9] https://www.packtpub.com/web-development/flask-framework-cookbook > > _______________________________________________ > Flask mailing list > Flask at python.org > https://mail.python.org/mailman/listinfo/flask > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kikocorreoso at gmail.com Thu Jun 22 04:34:28 2017 From: kikocorreoso at gmail.com (Kiko) Date: Thu, 22 Jun 2017 10:34:28 +0200 Subject: [Flask] Flask-security or other alternatives In-Reply-To: References: Message-ID: 2017-06-22 10:11 GMT+02:00 Christophe Bal : > Take a look here : https://blog.miguelgrinberg. > com/post/handling-authentication-secrets-in-the-browser > Thanks, pretty nice slides. But it doesn't answer my doubts about reinventing the wheel or using flask-security :-( > > > Christophe BAL > Enseignant Agr?g? de Math?matiques > Programmeur Python Amateur > > Le 22 juin 2017 08:24, "Kiko" a ?crit : > >> Hi all, >> >> After reading most used tutorials [1] [2] and most of the books about >> flask [3] [4] [5] [6] [7] [8] [9] I can't see the use of Flask-security on >> any of them even it provides basic authentication, mail confirmation, >> reset/change password,..., out of the box. >> >> Is there any reason for that? Do you discourage the use of this extension >> for some reason? Do you recommend it for the typical login/regster needs of >> a website (basic authentication, mail confirmation, reset/change password) >> in order to not reinventing the wheel? If you recommend it, do you know of >> good examples implementing this stuff? >> >> Thanks in advance. >> >> kind regards. >> >> [1] https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial >> -now-with-python-3-support >> [2] https://realpython.com/blog/python/handling-email-confirmati >> on-in-flask/ >> [3] http://shop.oreilly.com/product/0636920031116.do >> [4] https://www.packtpub.com/web-development/flask-example >> [5] https://www.packtpub.com/web-development/flask-blueprints >> [6] https://www.packtpub.com/web-development/learning-flask-framework >> [7] https://www.packtpub.com/web-development/mastering-flask >> [8] https://www.packtpub.com/web-development/building-web-applic >> ations-flask >> [9] https://www.packtpub.com/web-development/flask-framework-cookbook >> >> _______________________________________________ >> Flask mailing list >> Flask at python.org >> https://mail.python.org/mailman/listinfo/flask >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.christoph.goetze at gmail.com Thu Jun 22 05:14:07 2017 From: paul.christoph.goetze at gmail.com (=?UTF-8?Q?Paul_G=c3=b6tze?=) Date: Thu, 22 Jun 2017 11:14:07 +0200 Subject: [Flask] Flask-security or other alternatives In-Reply-To: References: Message-ID: Hi Kiko, I am using flask-security successfully in a production app, and it just works fine. To be honest I used the fork (https://pypi.org/project/Flask-Security-Fork/) which some months ago had quite some fixes that were not in the original repo. There was a merging process going on over quite some time, because the original repo seemed to be not maintained (enough) to keep up with issues and the fork authors didn?t have maintainer access. See https://github.com/mattupstate/flask-security/issues/559 for the discussion and related issues. It looks like they managed to reintegrate the fork now (https://github.com/inveniosoftware/flask-security-fork/issues/40) and they are going to close down the fork. Also there was a new version of flask-security 3.0.0 released a couple of days ago. So, I'm pretty happy with it and I can definitely recommend using flask-security. It comes with all the standard authentication things (register, signup, password change/reset, email confirmation, etc.). The last version I used (flask-security-fork 2.0.1) didn?t provide a couple of things I needed to use it with a JSON-API, but it was pretty easy to adjust these things to my needs. Hope this is helpful. Cheers, Paul Am 22.06.2017 um 10:34 schrieb Kiko > > > 2017-06-22 10:11 GMT+02:00 Christophe Bal >: > > Take a look here > : https://blog.miguelgrinberg.com/post/handling-authentication-secrets-in-the-browser > > > > Thanks, pretty nice slides. > > But it doesn't answer my doubts about reinventing the wheel or using > flask-security :-( > > > > > Christophe BAL > Enseignant Agr?g? de Math?matiques > Programmeur Python Amateur > > Le 22 juin 2017 08:24, "Kiko" > a ?crit : > > Hi all, > > After reading most used tutorials [1] [2] and most of the > books about flask [3] [4] [5] [6] [7] [8] [9] I can't see the > use of Flask-security on any of them even it provides basic > authentication, mail confirmation, reset/change password,..., > out of the box. > > Is there any reason for that? Do you discourage the use of > this extension for some reason? Do you recommend it for the > typical login/regster needs of a website (basic > authentication, mail confirmation, reset/change password) in > order to not reinventing the wheel? If you recommend it, do > you know of good examples implementing this stuff? > > Thanks in advance. > > kind regards. > > [1] > https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-now-with-python-3-support > > [2] > https://realpython.com/blog/python/handling-email-confirmation-in-flask/ > > [3] http://shop.oreilly.com/product/0636920031116.do > > [4] https://www.packtpub.com/web-development/flask-example > > [5] https://www.packtpub.com/web-development/flask-blueprints > > [6] > https://www.packtpub.com/web-development/learning-flask-framework > > [7] https://www.packtpub.com/web-development/mastering-flask > > [8] > https://www.packtpub.com/web-development/building-web-applications-flask > > [9] > https://www.packtpub.com/web-development/flask-framework-cookbook > > > _______________________________________________ > Flask mailing list > Flask at python.org > https://mail.python.org/mailman/listinfo/flask > > > > > > _______________________________________________ > Flask mailing list > Flask at python.org > https://mail.python.org/mailman/listinfo/flask -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 455 bytes Desc: OpenPGP digital signature URL: From kikocorreoso at gmail.com Thu Jun 22 05:28:30 2017 From: kikocorreoso at gmail.com (Kiko) Date: Thu, 22 Jun 2017 11:28:30 +0200 Subject: [Flask] Flask-security or other alternatives In-Reply-To: References: Message-ID: 2017-06-22 11:14 GMT+02:00 Paul G?tze : > Hi Kiko, > > I am using flask-security successfully in a production app, and it just > works fine. To be honest I used the fork (https://pypi.org/project/ > Flask-Security-Fork/) which some months ago had quite some fixes that > were not in the original repo. There was a merging process going on over > quite some time, because the original repo seemed to be not maintained > (enough) to keep up with issues and the fork authors didn?t have maintainer > access. > > See https://github.com/mattupstate/flask-security/issues/559 for the > discussion and related issues. It looks like they managed to reintegrate > the fork now (https://github.com/inveniosoftware/flask- > security-fork/issues/40) and they are going to close down the fork. Also > there was a new version of flask-security 3.0.0 released a couple of days > ago. > So, I'm pretty happy with it and I can definitely recommend using > flask-security. It comes with all the standard authentication things > (register, signup, password change/reset, email confirmation, etc.). The > last version I used (flask-security-fork 2.0.1) didn?t provide a couple of > things I needed to use it with a JSON-API, but it was pretty easy to adjust > these things to my needs. > Thanks, Paul. I saw the new release and the authors of the latest commits so I thought the fork was integrated into the main repo. Thanks for sharing your experience. > Hope this is helpful. > > Cheers, > Paul > > > Am 22.06.2017 um 10:34 schrieb Kiko > > > > 2017-06-22 10:11 GMT+02:00 Christophe Bal : > >> Take a look here : https://blog.miguelgrinberg. >> com/post/handling-authentication-secrets-in-the-browser >> > > Thanks, pretty nice slides. > > But it doesn't answer my doubts about reinventing the wheel or using > flask-security :-( > > >> >> >> Christophe BAL >> Enseignant Agr?g? de Math?matiques >> Programmeur Python Amateur >> >> Le 22 juin 2017 08:24, "Kiko" a ?crit : >> >>> Hi all, >>> >>> After reading most used tutorials [1] [2] and most of the books about >>> flask [3] [4] [5] [6] [7] [8] [9] I can't see the use of Flask-security on >>> any of them even it provides basic authentication, mail confirmation, >>> reset/change password,..., out of the box. >>> >>> Is there any reason for that? Do you discourage the use of this >>> extension for some reason? Do you recommend it for the typical >>> login/regster needs of a website (basic authentication, mail confirmation, >>> reset/change password) in order to not reinventing the wheel? If you >>> recommend it, do you know of good examples implementing this stuff? >>> >>> Thanks in advance. >>> >>> kind regards. >>> >>> [1] https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial >>> -now-with-python-3-support >>> [2] https://realpython.com/blog/python/handling-email-confirmati >>> on-in-flask/ >>> [3] http://shop.oreilly.com/product/0636920031116.do >>> [4] https://www.packtpub.com/web-development/flask-example >>> [5] https://www.packtpub.com/web-development/flask-blueprints >>> [6] https://www.packtpub.com/web-development/learning-flask-framework >>> [7] https://www.packtpub.com/web-development/mastering-flask >>> [8] https://www.packtpub.com/web-development/building-web-applic >>> ations-flask >>> [9] https://www.packtpub.com/web-development/flask-framework-cookbook >>> >>> _______________________________________________ >>> Flask mailing list >>> Flask at python.org >>> https://mail.python.org/mailman/listinfo/flask >>> >>> > > > _______________________________________________ > Flask mailing listFlask at python.orghttps://mail.python.org/mailman/listinfo/flask > > > > _______________________________________________ > Flask mailing list > Flask at python.org > https://mail.python.org/mailman/listinfo/flask > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gsvijayraajaa at gmail.com Sun Jun 25 10:38:34 2017 From: gsvijayraajaa at gmail.com (G.S.Vijay Raajaa) Date: Sun, 25 Jun 2017 20:08:34 +0530 Subject: [Flask] REST based File Sharing server Implementation in Flask Message-ID: Hi, I am trying to build a REST based file sharing server on top of flask. I am looking for a scalable stack. The requirement goes as follows; 1) Cater multiple file uploads. 2) Files can be of the order of 30-100GB. The server hosting the python webserver should not read the file to memory and spill to disk. 3) Non blocking - when user uploads multiple files. 4) Non blocking - when multiple user try to upload files. Pointers to any existing open source project is also much appreciated. Idea inspired from https://filebin.net/ Seeing if celery + Flask can come handy. Kindly suggest the WSGI to use it for production too. Looking forward to your valuable suggestions. Regards, Vijay Raajaa GS -------------- next part -------------- An HTML attachment was scrubbed... URL: From ub at artfacts.net Sun Jun 25 11:47:07 2017 From: ub at artfacts.net (ub at artfacts.net) Date: Sun, 25 Jun 2017 17:47:07 +0200 Subject: [Flask] REST based File Sharing server Implementation in Flask In-Reply-To: References: Message-ID: hey vijay, you could use mongodb's gridfs. that way you hand over connection/IO control to the database driver and scaling would be determined by mongodb's scaling potential. there's multiple exmaples online: https://gist.github.com/artisonian/492713 https://stackoverflow.com/questions/32492501/list-and-serve-files-from-gridfs-with-flask https://github.com/RedBeard0531/python-gridfs-server/blob/master/gridfs_server.py ... i personally haven't used mongodb for quite some time, so this is just an idea. cheers, ub On 25.06.2017 16:38, G.S.Vijay Raajaa wrote: > Hi, > > I am trying to build a REST based file sharing server on top of flask. I > am looking for a scalable stack. > > The requirement goes as follows; > > 1) Cater multiple file uploads. > 2) Files can be of the order of 30-100GB. The server hosting the python > webserver should not read the file to memory and spill to disk. > 3) Non blocking - when user uploads multiple files. > 4) Non blocking - when multiple user try to upload files. > > Pointers to any existing open source project is also much appreciated. > > Idea inspired from https://filebin.net/ > > Seeing if celery + Flask can come handy. Kindly suggest the WSGI to use > it for production too. > > Looking forward to your valuable suggestions. > > Regards, > Vijay Raajaa GS > > > _______________________________________________ > Flask mailing list > Flask at python.org > https://mail.python.org/mailman/listinfo/flask > From gsvijayraajaa at gmail.com Sun Jun 25 12:37:49 2017 From: gsvijayraajaa at gmail.com (G.S.Vijay Raajaa) Date: Sun, 25 Jun 2017 22:07:49 +0530 Subject: [Flask] Flask Digest, Vol 24, Issue 8 In-Reply-To: References: Message-ID: Hi UB, Thanks for the idea. I shall probe more into the same. But I have quick clarifications, even with mongo/gridfs and having the connectionI/O delegated to the database driver makes the transaction blocking? Kindly correct me if I am wrong. Is there a async way of handling the upload to mongo and keep flask serve other incoming request? On the same note, w.r.t mongodb-gridFS, how do you think the write performance is when compared to storing them directly on the filesystem, same w.r.t to read for serving? I like to idea of scalability when you have file store within DB. Regards, Vijay Raajaa GS On Sun, Jun 25, 2017 at 9:30 PM, wrote: > Send Flask mailing list submissions to > flask at python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman/listinfo/flask > or, via email, send a message with subject or body 'help' to > flask-request at python.org > > You can reach the person managing the list at > flask-owner at python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Flask digest..." > > > Today's Topics: > > 1. REST based File Sharing server Implementation in Flask > (G.S.Vijay Raajaa) > 2. Re: REST based File Sharing server Implementation in Flask > (ub at artfacts.net) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sun, 25 Jun 2017 20:08:34 +0530 > From: "G.S.Vijay Raajaa" > To: flask at python.org > Subject: [Flask] REST based File Sharing server Implementation in > Flask > Message-ID: > gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hi, > > I am trying to build a REST based file sharing server on top of flask. I am > looking for a scalable stack. > > The requirement goes as follows; > > 1) Cater multiple file uploads. > 2) Files can be of the order of 30-100GB. The server hosting the python > webserver should not read the file to memory and spill to disk. > 3) Non blocking - when user uploads multiple files. > 4) Non blocking - when multiple user try to upload files. > > Pointers to any existing open source project is also much appreciated. > > Idea inspired from https://filebin.net/ > > Seeing if celery + Flask can come handy. Kindly suggest the WSGI to use it > for production too. > > Looking forward to your valuable suggestions. > > Regards, > Vijay Raajaa GS > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: 20170625/b824c18b/attachment-0001.html> > > ------------------------------ > > Message: 2 > Date: Sun, 25 Jun 2017 17:47:07 +0200 > From: "ub at artfacts.net" > To: flask at python.org > Subject: Re: [Flask] REST based File Sharing server Implementation in > Flask > Message-ID: > Content-Type: text/plain; charset=utf-8 > > hey vijay, > > you could use mongodb's gridfs. that way you hand over connection/IO > control to the database driver and scaling would be determined by > mongodb's scaling potential. > > there's multiple exmaples online: > https://gist.github.com/artisonian/492713 > https://stackoverflow.com/questions/32492501/list-and- > serve-files-from-gridfs-with-flask > https://github.com/RedBeard0531/python-gridfs-server/blob/master/gridfs_ > server.py > ... > > i personally haven't used mongodb for quite some time, so this is just > an idea. > > cheers, > ub > > > On 25.06.2017 16:38, G.S.Vijay Raajaa wrote: > > Hi, > > > > I am trying to build a REST based file sharing server on top of flask. I > > am looking for a scalable stack. > > > > The requirement goes as follows; > > > > 1) Cater multiple file uploads. > > 2) Files can be of the order of 30-100GB. The server hosting the python > > webserver should not read the file to memory and spill to disk. > > 3) Non blocking - when user uploads multiple files. > > 4) Non blocking - when multiple user try to upload files. > > > > Pointers to any existing open source project is also much appreciated. > > > > Idea inspired from https://filebin.net/ > > > > Seeing if celery + Flask can come handy. Kindly suggest the WSGI to use > > it for production too. > > > > Looking forward to your valuable suggestions. > > > > Regards, > > Vijay Raajaa GS > > > > > > _______________________________________________ > > Flask mailing list > > Flask at python.org > > https://mail.python.org/mailman/listinfo/flask > > > > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > Flask mailing list > Flask at python.org > https://mail.python.org/mailman/listinfo/flask > > > ------------------------------ > > End of Flask Digest, Vol 24, Issue 8 > ************************************ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gsvijayraajaa at gmail.com Sun Jun 25 12:40:42 2017 From: gsvijayraajaa at gmail.com (G.S.Vijay Raajaa) Date: Sun, 25 Jun 2017 22:10:42 +0530 Subject: [Flask] Flask Digest, Vol 24, Issue 8 In-Reply-To: References: Message-ID: Sry, Wrong reply on the digest mail !! On Sun, Jun 25, 2017 at 10:07 PM, G.S.Vijay Raajaa wrote: > Hi UB, > > Thanks for the idea. I shall probe more into the same. But I have quick > clarifications, even with mongo/gridfs and having the connectionI/O > delegated to the database driver makes the transaction blocking? Kindly > correct me if I am wrong. Is there a async way of handling the upload to > mongo and keep flask serve other incoming request? > > On the same note, w.r.t mongodb-gridFS, how do you think the write > performance is when compared to storing them directly on the filesystem, > same w.r.t to read for serving? > > I like to idea of scalability when you have file store within DB. > > Regards, > Vijay Raajaa GS > > > On Sun, Jun 25, 2017 at 9:30 PM, wrote: > >> Send Flask mailing list submissions to >> flask at python.org >> >> To subscribe or unsubscribe via the World Wide Web, visit >> https://mail.python.org/mailman/listinfo/flask >> or, via email, send a message with subject or body 'help' to >> flask-request at python.org >> >> You can reach the person managing the list at >> flask-owner at python.org >> >> When replying, please edit your Subject line so it is more specific >> than "Re: Contents of Flask digest..." >> >> >> Today's Topics: >> >> 1. REST based File Sharing server Implementation in Flask >> (G.S.Vijay Raajaa) >> 2. Re: REST based File Sharing server Implementation in Flask >> (ub at artfacts.net) >> >> >> ---------------------------------------------------------------------- >> >> Message: 1 >> Date: Sun, 25 Jun 2017 20:08:34 +0530 >> From: "G.S.Vijay Raajaa" >> To: flask at python.org >> Subject: [Flask] REST based File Sharing server Implementation in >> Flask >> Message-ID: >> > ail.com> >> Content-Type: text/plain; charset="utf-8" >> >> Hi, >> >> I am trying to build a REST based file sharing server on top of flask. I >> am >> looking for a scalable stack. >> >> The requirement goes as follows; >> >> 1) Cater multiple file uploads. >> 2) Files can be of the order of 30-100GB. The server hosting the python >> webserver should not read the file to memory and spill to disk. >> 3) Non blocking - when user uploads multiple files. >> 4) Non blocking - when multiple user try to upload files. >> >> Pointers to any existing open source project is also much appreciated. >> >> Idea inspired from https://filebin.net/ >> >> Seeing if celery + Flask can come handy. Kindly suggest the WSGI to use it >> for production too. >> >> Looking forward to your valuable suggestions. >> >> Regards, >> Vijay Raajaa GS >> -------------- next part -------------- >> An HTML attachment was scrubbed... >> URL: > /b824c18b/attachment-0001.html> >> >> ------------------------------ >> >> Message: 2 >> Date: Sun, 25 Jun 2017 17:47:07 +0200 >> From: "ub at artfacts.net" >> To: flask at python.org >> Subject: Re: [Flask] REST based File Sharing server Implementation in >> Flask >> Message-ID: >> Content-Type: text/plain; charset=utf-8 >> >> hey vijay, >> >> you could use mongodb's gridfs. that way you hand over connection/IO >> control to the database driver and scaling would be determined by >> mongodb's scaling potential. >> >> there's multiple exmaples online: >> https://gist.github.com/artisonian/492713 >> https://stackoverflow.com/questions/32492501/list-and-serve- >> files-from-gridfs-with-flask >> https://github.com/RedBeard0531/python-gridfs-server/blob/ >> master/gridfs_server.py >> ... >> >> i personally haven't used mongodb for quite some time, so this is just >> an idea. >> >> cheers, >> ub >> >> >> On 25.06.2017 16:38, G.S.Vijay Raajaa wrote: >> > Hi, >> > >> > I am trying to build a REST based file sharing server on top of flask. I >> > am looking for a scalable stack. >> > >> > The requirement goes as follows; >> > >> > 1) Cater multiple file uploads. >> > 2) Files can be of the order of 30-100GB. The server hosting the python >> > webserver should not read the file to memory and spill to disk. >> > 3) Non blocking - when user uploads multiple files. >> > 4) Non blocking - when multiple user try to upload files. >> > >> > Pointers to any existing open source project is also much appreciated. >> > >> > Idea inspired from https://filebin.net/ >> > >> > Seeing if celery + Flask can come handy. Kindly suggest the WSGI to use >> > it for production too. >> > >> > Looking forward to your valuable suggestions. >> > >> > Regards, >> > Vijay Raajaa GS >> > >> > >> > _______________________________________________ >> > Flask mailing list >> > Flask at python.org >> > https://mail.python.org/mailman/listinfo/flask >> > >> >> >> >> ------------------------------ >> >> Subject: Digest Footer >> >> _______________________________________________ >> Flask mailing list >> Flask at python.org >> https://mail.python.org/mailman/listinfo/flask >> >> >> ------------------------------ >> >> End of Flask Digest, Vol 24, Issue 8 >> ************************************ >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gsvijayraajaa at gmail.com Sun Jun 25 12:43:21 2017 From: gsvijayraajaa at gmail.com (G.S.Vijay Raajaa) Date: Sun, 25 Jun 2017 22:13:21 +0530 Subject: [Flask] REST based File Sharing server Implementation in Flask In-Reply-To: References: Message-ID: Hi UB, Thanks for the idea. I shall probe more into the same. But I have quick clarifications, even with mongo/gridfs and having the connectionI/O delegated to the database driver makes the transaction blocking? Kindly correct me if I am wrong. Is there a async way of handling the upload to mongo and keep flask serve other incoming request? On the same note, w.r.t mongodb-gridFS, how do you think the write performance is when compared to storing them directly on the filesystem, same w.r.t to read for serving? I like to idea of scalability when you have file store within DB. Regards, Vijay Raajaa GS On Sun, Jun 25, 2017 at 8:08 PM, G.S.Vijay Raajaa wrote: > Hi, > > I am trying to build a REST based file sharing server on top of flask. I > am looking for a scalable stack. > > The requirement goes as follows; > > 1) Cater multiple file uploads. > 2) Files can be of the order of 30-100GB. The server hosting the python > webserver should not read the file to memory and spill to disk. > 3) Non blocking - when user uploads multiple files. > 4) Non blocking - when multiple user try to upload files. > > Pointers to any existing open source project is also much appreciated. > > Idea inspired from https://filebin.net/ > > Seeing if celery + Flask can come handy. Kindly suggest the WSGI to use it > for production too. > > Looking forward to your valuable suggestions. > > Regards, > Vijay Raajaa GS > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertlagrant at gmail.com Sun Jun 25 12:45:45 2017 From: robertlagrant at gmail.com (Robert Grant) Date: Sun, 25 Jun 2017 17:45:45 +0100 Subject: [Flask] Flask Digest, Vol 24, Issue 8 In-Reply-To: References: Message-ID: Just check the docs (https://docs.mongodb.com/manual/core/gridfs/) for filesystem vs Gridfs: In some situations, storing large files may be more efficient in a MongoDB database than on a system-level filesystem. - If your filesystem limits the number of files in a directory, you can use GridFS to store as many files as needed. - When you want to access information from portions of large files without having to load whole files into memory, you can use GridFS to recall sections of files without reading the entire file into memory. - When you want to keep your files and metadata automatically synced and deployed across a number of systems and facilities, you can use GridFS. When using geographically distributed replica sets , MongoDB can distribute files and their metadata automatically to a number of mongod instances and facilities. On 25 Jun 2017 5:38 pm, "G.S.Vijay Raajaa" wrote: > Hi UB, > > Thanks for the idea. I shall probe more into the same. But I have quick > clarifications, even with mongo/gridfs and having the connectionI/O > delegated to the database driver makes the transaction blocking? Kindly > correct me if I am wrong. Is there a async way of handling the upload to > mongo and keep flask serve other incoming request? > > On the same note, w.r.t mongodb-gridFS, how do you think the write > performance is when compared to storing them directly on the filesystem, > same w.r.t to read for serving? > > I like to idea of scalability when you have file store within DB. > > Regards, > Vijay Raajaa GS > > On Sun, Jun 25, 2017 at 9:30 PM, wrote: > >> Send Flask mailing list submissions to >> flask at python.org >> >> To subscribe or unsubscribe via the World Wide Web, visit >> https://mail.python.org/mailman/listinfo/flask >> or, via email, send a message with subject or body 'help' to >> flask-request at python.org >> >> You can reach the person managing the list at >> flask-owner at python.org >> >> When replying, please edit your Subject line so it is more specific >> than "Re: Contents of Flask digest..." >> >> >> Today's Topics: >> >> 1. REST based File Sharing server Implementation in Flask >> (G.S.Vijay Raajaa) >> 2. Re: REST based File Sharing server Implementation in Flask >> (ub at artfacts.net) >> >> >> ---------------------------------------------------------------------- >> >> Message: 1 >> Date: Sun, 25 Jun 2017 20:08:34 +0530 >> From: "G.S.Vijay Raajaa" >> To: flask at python.org >> Subject: [Flask] REST based File Sharing server Implementation in >> Flask >> Message-ID: >> > ail.com> >> Content-Type: text/plain; charset="utf-8" >> >> Hi, >> >> I am trying to build a REST based file sharing server on top of flask. I >> am >> looking for a scalable stack. >> >> The requirement goes as follows; >> >> 1) Cater multiple file uploads. >> 2) Files can be of the order of 30-100GB. The server hosting the python >> webserver should not read the file to memory and spill to disk. >> 3) Non blocking - when user uploads multiple files. >> 4) Non blocking - when multiple user try to upload files. >> >> Pointers to any existing open source project is also much appreciated. >> >> Idea inspired from https://filebin.net/ >> >> Seeing if celery + Flask can come handy. Kindly suggest the WSGI to use it >> for production too. >> >> Looking forward to your valuable suggestions. >> >> Regards, >> Vijay Raajaa GS >> -------------- next part -------------- >> An HTML attachment was scrubbed... >> URL: > /b824c18b/attachment-0001.html> >> >> ------------------------------ >> >> Message: 2 >> Date: Sun, 25 Jun 2017 17:47:07 +0200 >> From: "ub at artfacts.net" >> To: flask at python.org >> Subject: Re: [Flask] REST based File Sharing server Implementation in >> Flask >> Message-ID: >> Content-Type: text/plain; charset=utf-8 >> >> hey vijay, >> >> you could use mongodb's gridfs. that way you hand over connection/IO >> control to the database driver and scaling would be determined by >> mongodb's scaling potential. >> >> there's multiple exmaples online: >> https://gist.github.com/artisonian/492713 >> https://stackoverflow.com/questions/32492501/list-and-serve- >> files-from-gridfs-with-flask >> https://github.com/RedBeard0531/python-gridfs-server/blob/ >> master/gridfs_server.py >> ... >> >> i personally haven't used mongodb for quite some time, so this is just >> an idea. >> >> cheers, >> ub >> >> >> On 25.06.2017 16:38, G.S.Vijay Raajaa wrote: >> > Hi, >> > >> > I am trying to build a REST based file sharing server on top of flask. I >> > am looking for a scalable stack. >> > >> > The requirement goes as follows; >> > >> > 1) Cater multiple file uploads. >> > 2) Files can be of the order of 30-100GB. The server hosting the python >> > webserver should not read the file to memory and spill to disk. >> > 3) Non blocking - when user uploads multiple files. >> > 4) Non blocking - when multiple user try to upload files. >> > >> > Pointers to any existing open source project is also much appreciated. >> > >> > Idea inspired from https://filebin.net/ >> > >> > Seeing if celery + Flask can come handy. Kindly suggest the WSGI to use >> > it for production too. >> > >> > Looking forward to your valuable suggestions. >> > >> > Regards, >> > Vijay Raajaa GS >> > >> > >> > _______________________________________________ >> > Flask mailing list >> > Flask at python.org >> > https://mail.python.org/mailman/listinfo/flask >> > >> >> >> >> ------------------------------ >> >> Subject: Digest Footer >> >> _______________________________________________ >> Flask mailing list >> Flask at python.org >> https://mail.python.org/mailman/listinfo/flask >> >> >> ------------------------------ >> >> End of Flask Digest, Vol 24, Issue 8 >> ************************************ >> > > > _______________________________________________ > Flask mailing list > Flask at python.org > https://mail.python.org/mailman/listinfo/flask > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gsvijayraajaa at gmail.com Sun Jun 25 12:50:56 2017 From: gsvijayraajaa at gmail.com (G.S.Vijay Raajaa) Date: Sun, 25 Jun 2017 22:20:56 +0530 Subject: [Flask] Flask Digest, Vol 24, Issue 8 Message-ID: Thats an interesting point. Sure, let me try it out. Any view on making the writes asynchronous while front facing it with Flask? Regards, Vijay Raajaa GS On Sun, Jun 25, 2017 at 10:15 PM, Robert Grant wrote: > Just check the docs (https://docs.mongodb.com/manual/core/gridfs/) for > filesystem vs Gridfs: > > In some situations, storing large files may be more efficient in a MongoDB > database than on a system-level filesystem. > > - If your filesystem limits the number of files in a directory, you > can use GridFS to store as many files as needed. > - When you want to access information from portions of large files > without having to load whole files into memory, you can use GridFS to > recall sections of files without reading the entire file into memory. > - When you want to keep your files and metadata automatically synced > and deployed across a number of systems and facilities, you can use GridFS. > When using geographically distributed replica sets > , > MongoDB can distribute files and their metadata automatically to a number of > mongod instances and facilities. > > > On 25 Jun 2017 5:38 pm, "G.S.Vijay Raajaa" > wrote: > >> Hi UB, >> >> Thanks for the idea. I shall probe more into the same. But I have quick >> clarifications, even with mongo/gridfs and having the connectionI/O >> delegated to the database driver makes the transaction blocking? Kindly >> correct me if I am wrong. Is there a async way of handling the upload to >> mongo and keep flask serve other incoming request? >> >> On the same note, w.r.t mongodb-gridFS, how do you think the write >> performance is when compared to storing them directly on the filesystem, >> same w.r.t to read for serving? >> >> I like to idea of scalability when you have file store within DB. >> >> Regards, >> Vijay Raajaa GS >> >> On Sun, Jun 25, 2017 at 9:30 PM, wrote: >> >>> Send Flask mailing list submissions to >>> flask at python.org >>> >>> To subscribe or unsubscribe via the World Wide Web, visit >>> https://mail.python.org/mailman/listinfo/flask >>> or, via email, send a message with subject or body 'help' to >>> flask-request at python.org >>> >>> You can reach the person managing the list at >>> flask-owner at python.org >>> >>> When replying, please edit your Subject line so it is more specific >>> than "Re: Contents of Flask digest..." >>> >>> >>> Today's Topics: >>> >>> 1. REST based File Sharing server Implementation in Flask >>> (G.S.Vijay Raajaa) >>> 2. Re: REST based File Sharing server Implementation in Flask >>> (ub at artfacts.net) >>> >>> >>> ---------------------------------------------------------------------- >>> >>> Message: 1 >>> Date: Sun, 25 Jun 2017 20:08:34 +0530 >>> From: "G.S.Vijay Raajaa" >>> To: flask at python.org >>> Subject: [Flask] REST based File Sharing server Implementation in >>> Flask >>> Message-ID: >>> >> ail.com> >>> Content-Type: text/plain; charset="utf-8" >>> >>> Hi, >>> >>> I am trying to build a REST based file sharing server on top of flask. I >>> am >>> looking for a scalable stack. >>> >>> The requirement goes as follows; >>> >>> 1) Cater multiple file uploads. >>> 2) Files can be of the order of 30-100GB. The server hosting the python >>> webserver should not read the file to memory and spill to disk. >>> 3) Non blocking - when user uploads multiple files. >>> 4) Non blocking - when multiple user try to upload files. >>> >>> Pointers to any existing open source project is also much appreciated. >>> >>> Idea inspired from https://filebin.net/ >>> >>> Seeing if celery + Flask can come handy. Kindly suggest the WSGI to use >>> it >>> for production too. >>> >>> Looking forward to your valuable suggestions. >>> >>> Regards, >>> Vijay Raajaa GS >>> -------------- next part -------------- >>> An HTML attachment was scrubbed... >>> URL: >> /b824c18b/attachment-0001.html> >>> >>> ------------------------------ >>> >>> Message: 2 >>> Date: Sun, 25 Jun 2017 17:47:07 +0200 >>> From: "ub at artfacts.net" >>> To: flask at python.org >>> Subject: Re: [Flask] REST based File Sharing server Implementation in >>> Flask >>> Message-ID: >>> Content-Type: text/plain; charset=utf-8 >>> >>> hey vijay, >>> >>> you could use mongodb's gridfs. that way you hand over connection/IO >>> control to the database driver and scaling would be determined by >>> mongodb's scaling potential. >>> >>> there's multiple exmaples online: >>> https://gist.github.com/artisonian/492713 >>> https://stackoverflow.com/questions/32492501/list-and-serve- >>> files-from-gridfs-with-flask >>> https://github.com/RedBeard0531/python-gridfs-server/blob/ma >>> ster/gridfs_server.py >>> ... >>> >>> i personally haven't used mongodb for quite some time, so this is just >>> an idea. >>> >>> cheers, >>> ub >>> >>> >>> On 25.06.2017 16:38, G.S.Vijay Raajaa wrote: >>> > Hi, >>> > >>> > I am trying to build a REST based file sharing server on top of flask. >>> I >>> > am looking for a scalable stack. >>> > >>> > The requirement goes as follows; >>> > >>> > 1) Cater multiple file uploads. >>> > 2) Files can be of the order of 30-100GB. The server hosting the python >>> > webserver should not read the file to memory and spill to disk. >>> > 3) Non blocking - when user uploads multiple files. >>> > 4) Non blocking - when multiple user try to upload files. >>> > >>> > Pointers to any existing open source project is also much appreciated. >>> > >>> > Idea inspired from https://filebin.net/ >>> > >>> > Seeing if celery + Flask can come handy. Kindly suggest the WSGI to use >>> > it for production too. >>> > >>> > Looking forward to your valuable suggestions. >>> > >>> > Regards, >>> > Vijay Raajaa GS >>> > >>> > >>> > _______________________________________________ >>> > Flask mailing list >>> > Flask at python.org >>> > https://mail.python.org/mailman/listinfo/flask >>> > >>> >>> >>> >>> ------------------------------ >>> >>> Subject: Digest Footer >>> >>> _______________________________________________ >>> Flask mailing list >>> Flask at python.org >>> https://mail.python.org/mailman/listinfo/flask >>> >>> >>> ------------------------------ >>> >>> End of Flask Digest, Vol 24, Issue 8 >>> ************************************ >>> >> >> >> _______________________________________________ >> Flask mailing list >> Flask at python.org >> https://mail.python.org/mailman/listinfo/flask >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From ub at artfacts.net Sun Jun 25 12:55:34 2017 From: ub at artfacts.net (ub at artfacts.net) Date: Sun, 25 Jun 2017 18:55:34 +0200 Subject: [Flask] REST based File Sharing server Implementation in Flask In-Reply-To: References: Message-ID: <9db8a6c0-a5ba-3de6-c1c7-b7b04c62577f@artfacts.net> On 25.06.2017 18:43, G.S.Vijay Raajaa wrote: > Hi UB, > > Thanks for the idea. I shall probe more into the same. But I have quick > clarifications, even with mongo/gridfs and having the connectionI/O > delegated to the database driver makes the transaction blocking? Kindly > correct me if I am wrong. Is there a async way of handling the upload to > mongo and keep flask serve other incoming request? well, there will be workers. if you deploy your app in uwsgi or gunicorn, you set the number of workers in you config. these will then handle requests (more or less) indepently. pymongo will have a connection pool to also handle multiple connections. [2] > On the same note, w.r.t mongodb-gridFS, how do you think the write > performance is when compared to storing them directly on the filesystem, > same w.r.t to read for serving? i guess here's the bad news. it might be slower than writing directly to the file system, but it might be better when you have app and storage separated and need to write your files over the network. but you need to consult benchmarks to be sure. > I like to idea of scalability when you have file store within DB. i think here's the upside. also seems, mongodb can help you to store and organize 8search) file metadata. if that is a concern at all. [2] http://api.mongodb.com/python/current/faq.html#how-does-connection-pooling-work-in-pymongo > > Regards, > Vijay Raajaa GS > > On Sun, Jun 25, 2017 at 8:08 PM, G.S.Vijay Raajaa > > wrote: > > Hi, > > I am trying to build a REST based file sharing server on top of > flask. I am looking for a scalable stack. > > The requirement goes as follows; > > 1) Cater multiple file uploads. > 2) Files can be of the order of 30-100GB. The server hosting the > python webserver should not read the file to memory and spill to disk. > 3) Non blocking - when user uploads multiple files. > 4) Non blocking - when multiple user try to upload files. > > Pointers to any existing open source project is also much appreciated. > > Idea inspired from https://filebin.net/ > > Seeing if celery + Flask can come handy. Kindly suggest the WSGI to > use it for production too. > > Looking forward to your valuable suggestions. > > Regards, > Vijay Raajaa GS > > > > > _______________________________________________ > Flask mailing list > Flask at python.org > https://mail.python.org/mailman/listinfo/flask > From gsvijayraajaa at gmail.com Sun Jun 25 13:31:45 2017 From: gsvijayraajaa at gmail.com (G.S.Vijay Raajaa) Date: Sun, 25 Jun 2017 23:01:45 +0530 Subject: [Flask] REST based File Sharing server Implementation in Flask ub at artfacts.net Message-ID: The number of non-block servings will correspond to the max number of workers in that case? Hypothetically if one user uploads multiple files that keeps all the workers busy, I am blocked when rest of the users online trying to upload files. Any thoughts to delegate it asynchronously? Regards, Vijay Raajaa GS On 25.06.2017 18:43, G.S.Vijay Raajaa wrote: >* Hi UB, *> >* Thanks for the idea. I shall probe more into the same. But I have quick *>* clarifications, even with mongo/gridfs and having the connectionI/O *>* delegated to the database driver makes the transaction blocking? Kindly *>* correct me if I am wrong. Is there a async way of handling the upload to *>* mongo and keep flask serve other incoming request? *well, there will be workers. if you deploy your app in uwsgi or gunicorn, you set the number of workers in you config. these will then handle requests (more or less) indepently. pymongo will have a connection pool to also handle multiple connections. [2] >* On the same note, w.r.t mongodb-gridFS, how do you think the write *>* performance is when compared to storing them directly on the filesystem, *>* same w.r.t to read for serving? *i guess here's the bad news. it might be slower than writing directly to the file system, but it might be better when you have app and storage separated and need to write your files over the network. but you need to consult benchmarks to be sure. >* I like to idea of scalability when you have file store within DB. *i think here's the upside. also seems, mongodb can help you to store and organize 8search) file metadata. if that is a concern at all. [2]http://api.mongodb.com/python/current/faq.html#how-does-connection-pooling-work-in-pymongo -------------- next part -------------- An HTML attachment was scrubbed... URL: From gsvijayraajaa at gmail.com Sun Jun 25 13:47:15 2017 From: gsvijayraajaa at gmail.com (G.S.Vijay Raajaa) Date: Sun, 25 Jun 2017 23:17:15 +0530 Subject: [Flask] REST based File Sharing server Implementation in Flask In-Reply-To: References: Message-ID: The number of non-block servings will correspond to the max number of workers in that case? Hypothetically if one user uploads multiple files that keeps all the workers busy, I am blocked when rest of the users online trying to upload files. Any thoughts to delegate it asynchronously? Regards, Vijay Raajaa GS On Sun, Jun 25, 2017 at 10:13 PM, G.S.Vijay Raajaa wrote: > Hi UB, > > Thanks for the idea. I shall probe more into the same. But I have quick > clarifications, even with mongo/gridfs and having the connectionI/O > delegated to the database driver makes the transaction blocking? Kindly > correct me if I am wrong. Is there a async way of handling the upload to > mongo and keep flask serve other incoming request? > > On the same note, w.r.t mongodb-gridFS, how do you think the write > performance is when compared to storing them directly on the filesystem, > same w.r.t to read for serving? > > I like to idea of scalability when you have file store within DB. > > Regards, > Vijay Raajaa GS > > On Sun, Jun 25, 2017 at 8:08 PM, G.S.Vijay Raajaa > wrote: > >> Hi, >> >> I am trying to build a REST based file sharing server on top of flask. I >> am looking for a scalable stack. >> >> The requirement goes as follows; >> >> 1) Cater multiple file uploads. >> 2) Files can be of the order of 30-100GB. The server hosting the python >> webserver should not read the file to memory and spill to disk. >> 3) Non blocking - when user uploads multiple files. >> 4) Non blocking - when multiple user try to upload files. >> >> Pointers to any existing open source project is also much appreciated. >> >> Idea inspired from https://filebin.net/ >> >> Seeing if celery + Flask can come handy. Kindly suggest the WSGI to use >> it for production too. >> >> Looking forward to your valuable suggestions. >> >> Regards, >> Vijay Raajaa GS >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From coreybrett at gmail.com Sun Jun 25 13:54:52 2017 From: coreybrett at gmail.com (Corey Boyle) Date: Sun, 25 Jun 2017 13:54:52 -0400 Subject: [Flask] REST based File Sharing server Implementation in Flask In-Reply-To: References: Message-ID: You might also consider storing files in AWS S3 or some other object store. Dreamhost has a service for that as well. __ Corey On Jun 25, 2017 1:47 PM, "G.S.Vijay Raajaa" wrote: > The number of non-block servings will correspond to the max number of workers in that case? Hypothetically if one user uploads multiple files that keeps all the workers busy, I am blocked when rest of the users online trying to upload files. Any thoughts to delegate it asynchronously? > > Regards, > > Vijay Raajaa GS > > > On Sun, Jun 25, 2017 at 10:13 PM, G.S.Vijay Raajaa < > gsvijayraajaa at gmail.com> wrote: > >> Hi UB, >> >> Thanks for the idea. I shall probe more into the same. But I have quick >> clarifications, even with mongo/gridfs and having the connectionI/O >> delegated to the database driver makes the transaction blocking? Kindly >> correct me if I am wrong. Is there a async way of handling the upload to >> mongo and keep flask serve other incoming request? >> >> On the same note, w.r.t mongodb-gridFS, how do you think the write >> performance is when compared to storing them directly on the filesystem, >> same w.r.t to read for serving? >> >> I like to idea of scalability when you have file store within DB. >> >> Regards, >> Vijay Raajaa GS >> >> On Sun, Jun 25, 2017 at 8:08 PM, G.S.Vijay Raajaa < >> gsvijayraajaa at gmail.com> wrote: >> >>> Hi, >>> >>> I am trying to build a REST based file sharing server on top of flask. I >>> am looking for a scalable stack. >>> >>> The requirement goes as follows; >>> >>> 1) Cater multiple file uploads. >>> 2) Files can be of the order of 30-100GB. The server hosting the python >>> webserver should not read the file to memory and spill to disk. >>> 3) Non blocking - when user uploads multiple files. >>> 4) Non blocking - when multiple user try to upload files. >>> >>> Pointers to any existing open source project is also much appreciated. >>> >>> Idea inspired from https://filebin.net/ >>> >>> Seeing if celery + Flask can come handy. Kindly suggest the WSGI to use >>> it for production too. >>> >>> Looking forward to your valuable suggestions. >>> >>> Regards, >>> Vijay Raajaa GS >>> >> >> > > _______________________________________________ > Flask mailing list > Flask at python.org > https://mail.python.org/mailman/listinfo/flask > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gsvijayraajaa at gmail.com Sun Jun 25 13:58:32 2017 From: gsvijayraajaa at gmail.com (G.S.Vijay Raajaa) Date: Sun, 25 Jun 2017 23:28:32 +0530 Subject: [Flask] REST based File Sharing server Implementation in Flask In-Reply-To: References: Message-ID: This issue is that the project will be deployed on-premise with no internet connectivity from the server. So any 3rd party cloud service is ruled out. Regards, Vijay Raajaa GS On Sun, Jun 25, 2017 at 11:24 PM, Corey Boyle wrote: > You might also consider storing files in AWS S3 or some other object > store. Dreamhost has a service for that as well. > > > __ > Corey > > On Jun 25, 2017 1:47 PM, "G.S.Vijay Raajaa" > wrote: > >> The number of non-block servings will correspond to the max number of workers in that case? Hypothetically if one user uploads multiple files that keeps all the workers busy, I am blocked when rest of the users online trying to upload files. Any thoughts to delegate it asynchronously? >> >> Regards, >> >> Vijay Raajaa GS >> >> >> On Sun, Jun 25, 2017 at 10:13 PM, G.S.Vijay Raajaa < >> gsvijayraajaa at gmail.com> wrote: >> >>> Hi UB, >>> >>> Thanks for the idea. I shall probe more into the same. But I have quick >>> clarifications, even with mongo/gridfs and having the connectionI/O >>> delegated to the database driver makes the transaction blocking? Kindly >>> correct me if I am wrong. Is there a async way of handling the upload to >>> mongo and keep flask serve other incoming request? >>> >>> On the same note, w.r.t mongodb-gridFS, how do you think the write >>> performance is when compared to storing them directly on the filesystem, >>> same w.r.t to read for serving? >>> >>> I like to idea of scalability when you have file store within DB. >>> >>> Regards, >>> Vijay Raajaa GS >>> >>> On Sun, Jun 25, 2017 at 8:08 PM, G.S.Vijay Raajaa < >>> gsvijayraajaa at gmail.com> wrote: >>> >>>> Hi, >>>> >>>> I am trying to build a REST based file sharing server on top of flask. >>>> I am looking for a scalable stack. >>>> >>>> The requirement goes as follows; >>>> >>>> 1) Cater multiple file uploads. >>>> 2) Files can be of the order of 30-100GB. The server hosting the python >>>> webserver should not read the file to memory and spill to disk. >>>> 3) Non blocking - when user uploads multiple files. >>>> 4) Non blocking - when multiple user try to upload files. >>>> >>>> Pointers to any existing open source project is also much appreciated. >>>> >>>> Idea inspired from https://filebin.net/ >>>> >>>> Seeing if celery + Flask can come handy. Kindly suggest the WSGI to use >>>> it for production too. >>>> >>>> Looking forward to your valuable suggestions. >>>> >>>> Regards, >>>> Vijay Raajaa GS >>>> >>> >>> >> >> _______________________________________________ >> Flask mailing list >> Flask at python.org >> https://mail.python.org/mailman/listinfo/flask >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From ub at artfacts.net Sun Jun 25 14:01:22 2017 From: ub at artfacts.net (ub at artfacts.net) Date: Sun, 25 Jun 2017 20:01:22 +0200 Subject: [Flask] REST based File Sharing server Implementation in Flask In-Reply-To: References: Message-ID: <7ec6b29a-8dd8-50ac-a0b0-334a7114f38e@artfacts.net> On 25.06.2017 19:31, G.S.Vijay Raajaa wrote: > The number of non-block servings will correspond to the max number of workers in that case? Hypothetically if one user uploads multiple files that keeps all the workers busy, I am blocked when rest of the users online trying to upload files. Any thoughts to delegate it asynchronously? deployment with Tornado might be interesting for you as pointed it out in the answer here: https://stackoverflow.com/questions/18525856/flask-alternatives-to-achieve-true-multi-threading?noredirect=1&lq=1 > > Regards, > > Vijay Raajaa GS > > On 25.06.2017 18:43, G.S.Vijay Raajaa wrote: > >>/Hi UB, />//>/Thanks for the idea. I shall probe more into the same. But I have quick />/clarifications, even with mongo/gridfs and having the connectionI/O />/delegated to the database driver makes the transaction blocking? Kindly />/correct me if I am wrong. Is there a async way of handling the upload to />/mongo and keep flask serve other incoming request? /well, there will be workers. if you deploy your app in uwsgi or gunicorn, > you set the number of workers in you config. these will then handle > requests (more or less) indepently. > pymongo will have a connection pool to also handle multiple connections. [2] > > >>/On the same note, w.r.t mongodb-gridFS, how do you think the write />/performance is when compared to storing them directly on the filesystem, />/same w.r.t to read for serving? /i guess here's the bad news. it might be slower than writing directly to > the file system, but it might be better when you have app and storage > separated and need to write your files over the network. but you need to > consult benchmarks to be sure. > >>/I like to idea of scalability when you have file store within DB. /i think here's the upside. also seems, mongodb can help you to store and > organize 8search) file metadata. if that is a concern at all. > > [2] > http://api.mongodb.com/python/current/faq.html#how-does-connection-pooling-work-in-pymongo > > > > _______________________________________________ > Flask mailing list > Flask at python.org > https://mail.python.org/mailman/listinfo/flask > From coreybrett at gmail.com Sun Jun 25 14:03:06 2017 From: coreybrett at gmail.com (Corey Boyle) Date: Sun, 25 Jun 2017 14:03:06 -0400 Subject: [Flask] REST based File Sharing server Implementation in Flask In-Reply-To: References: Message-ID: I see. You could run an object store locally, but that may be more moving parts than you want at this point. __ Corey On Jun 25, 2017 1:58 PM, "G.S.Vijay Raajaa" wrote: > This issue is that the project will be deployed on-premise with no > internet connectivity from the server. So any 3rd party cloud service is > ruled out. > > Regards, > Vijay Raajaa GS > > On Sun, Jun 25, 2017 at 11:24 PM, Corey Boyle > wrote: > >> You might also consider storing files in AWS S3 or some other object >> store. Dreamhost has a service for that as well. >> >> >> __ >> Corey >> >> On Jun 25, 2017 1:47 PM, "G.S.Vijay Raajaa" >> wrote: >> >>> The number of non-block servings will correspond to the max number of workers in that case? Hypothetically if one user uploads multiple files that keeps all the workers busy, I am blocked when rest of the users online trying to upload files. Any thoughts to delegate it asynchronously? >>> >>> Regards, >>> >>> Vijay Raajaa GS >>> >>> >>> On Sun, Jun 25, 2017 at 10:13 PM, G.S.Vijay Raajaa < >>> gsvijayraajaa at gmail.com> wrote: >>> >>>> Hi UB, >>>> >>>> Thanks for the idea. I shall probe more into the same. But I have quick >>>> clarifications, even with mongo/gridfs and having the connectionI/O >>>> delegated to the database driver makes the transaction blocking? Kindly >>>> correct me if I am wrong. Is there a async way of handling the upload to >>>> mongo and keep flask serve other incoming request? >>>> >>>> On the same note, w.r.t mongodb-gridFS, how do you think the write >>>> performance is when compared to storing them directly on the filesystem, >>>> same w.r.t to read for serving? >>>> >>>> I like to idea of scalability when you have file store within DB. >>>> >>>> Regards, >>>> Vijay Raajaa GS >>>> >>>> On Sun, Jun 25, 2017 at 8:08 PM, G.S.Vijay Raajaa < >>>> gsvijayraajaa at gmail.com> wrote: >>>> >>>>> Hi, >>>>> >>>>> I am trying to build a REST based file sharing server on top of flask. >>>>> I am looking for a scalable stack. >>>>> >>>>> The requirement goes as follows; >>>>> >>>>> 1) Cater multiple file uploads. >>>>> 2) Files can be of the order of 30-100GB. The server hosting the >>>>> python webserver should not read the file to memory and spill to disk. >>>>> 3) Non blocking - when user uploads multiple files. >>>>> 4) Non blocking - when multiple user try to upload files. >>>>> >>>>> Pointers to any existing open source project is also much appreciated. >>>>> >>>>> Idea inspired from https://filebin.net/ >>>>> >>>>> Seeing if celery + Flask can come handy. Kindly suggest the WSGI to >>>>> use it for production too. >>>>> >>>>> Looking forward to your valuable suggestions. >>>>> >>>>> Regards, >>>>> Vijay Raajaa GS >>>>> >>>> >>>> >>> >>> _______________________________________________ >>> Flask mailing list >>> Flask at python.org >>> https://mail.python.org/mailman/listinfo/flask >>> >>> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gsvijayraajaa at gmail.com Sun Jun 25 14:06:16 2017 From: gsvijayraajaa at gmail.com (G.S.Vijay Raajaa) Date: Sun, 25 Jun 2017 23:36:16 +0530 Subject: [Flask] REST based File Sharing server Implementation in Flask In-Reply-To: References: Message-ID: Sure, Thanks for pointing out to stackoverflow. Let me try Tornado out and see the performance of concurrent requests. Regards, Vijay Raajaa GS On Sun, Jun 25, 2017 at 11:31 PM, ub at artfacts.net wrote: > On 25.06.2017 19:31, G.S.Vijay Raajaa wrote: > > The number of non-block servings will correspond to the max number of > workers in that case? Hypothetically if one user uploads multiple files > that keeps all the workers busy, I am blocked when rest of the users online > trying to upload files. Any thoughts to delegate it asynchronously? > deployment with Tornado might be interesting for you as pointed it out > in the answer here: > https://stackoverflow.com/questions/18525856/flask-alternati > ves-to-achieve-true-multi-threading?noredirect=1&lq=1 > > > > > > Regards, > > > > Vijay Raajaa GS > > > > On 25.06.2017 18:43, G.S.Vijay Raajaa wrote: > > > >>/Hi UB, />//>/Thanks for the idea. I shall probe more into the same. But > I have quick />/clarifications, even with mongo/gridfs and having the > connectionI/O />/delegated to the database driver makes the transaction > blocking? Kindly />/correct me if I am wrong. Is there a async way of > handling the upload to />/mongo and keep flask serve other incoming > request? /well, there will be workers. if you deploy your app in uwsgi or > gunicorn, > > you set the number of workers in you config. these will then handle > > requests (more or less) indepently. > > pymongo will have a connection pool to also handle multiple connections. > [2] > > > > > >>/On the same note, w.r.t mongodb-gridFS, how do you think the write > />/performance is when compared to storing them directly on the filesystem, > />/same w.r.t to read for serving? /i guess here's the bad news. it might > be slower than writing directly to > > the file system, but it might be better when you have app and storage > > separated and need to write your files over the network. but you need to > > consult benchmarks to be sure. > > > >>/I like to idea of scalability when you have file store within DB. /i > think here's the upside. also seems, mongodb can help you to store and > > organize 8search) file metadata. if that is a concern at all. > > > > [2] > > http://api.mongodb.com/python/current/faq.html#how-does-conn > ection-pooling-work-in-pymongo > > > > > > > > _______________________________________________ > > Flask mailing list > > Flask at python.org > > https://mail.python.org/mailman/listinfo/flask > > > > _______________________________________________ > Flask mailing list > Flask at python.org > https://mail.python.org/mailman/listinfo/flask > -------------- next part -------------- An HTML attachment was scrubbed... URL: