From taushifkhan at gmail.com  Fri Jul  1 06:03:17 2016
From: taushifkhan at gmail.com (taushif khan)
Date: Fri, 1 Jul 2016 15:33:17 +0530
Subject: [Flask] [Deploying Flask with apache-wsgi] Problem
Message-ID: <CAMxno56d4RkLPreATh6Gq1NBbnss2W7VbeFCV-uLAiMfkxXVXA@mail.gmail.com>

Hi,
I am trying to put my flask-application to the web by deploying it with
apache2-wsgi. But there is a problem of permissions to read a file from the
root user.

My appliaction takes a input from the user. the input is a 3D coordinate
file. Then my codes in the backend compiles the results and writes them ia
a json file. The flask application then reades the json file and gives
result.

During the development it was smooth and both upload and download by the
server was performed with out any issues. But once I but the app in the
deployment its giving a pain.

Any suggestion to fix the permission issue during deployment in production
server. Wll be helpful.

-- 
With regards,

Taushif Khan
Ph.D.
PostDoc Research
Interuniversity Institute of Bioinformatics in Brussels
<http://ibsquare.be/>
Brussels, Belgium


*The true sign of intelligence is not knowledge but imagination.
<http://www.brainyquote.com/quotes/quotes/a/alberteins148802.html> *[*Einstein
<http://www.brainyquote.com/quotes/quotes/a/alberteins148802.html> A]*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/flask/attachments/20160701/36cb9e30/attachment.html>

From ziirish at ziirish.info  Fri Jul  1 06:21:28 2016
From: ziirish at ziirish.info (Ziirish)
Date: Fri, 1 Jul 2016 12:21:28 +0200
Subject: [Flask] [Deploying Flask with apache-wsgi] Problem
In-Reply-To: <CAMxno56d4RkLPreATh6Gq1NBbnss2W7VbeFCV-uLAiMfkxXVXA@mail.gmail.com>
References: <CAMxno56d4RkLPreATh6Gq1NBbnss2W7VbeFCV-uLAiMfkxXVXA@mail.gmail.com>
Message-ID: <20160701102127.GE3187@mail.ziirish.info>

Hi,

What is the exact error message?

I guess your Apache server is running as www-data (or apache) user.
Does this user have rights on the files/folders managed by your application?


* On Friday, July 01, 2016 at 03:33 PM +0530, taushif khan <taushifkhan at gmail.com> wrote:
> Hi,
> I am trying to put my flask-application to the web by deploying it with
> apache2-wsgi. But there is a problem of permissions to read a file from the
> root user.
> 
> My appliaction takes a input from the user. the input is a 3D coordinate
> file. Then my codes in the backend compiles the results and writes them ia
> a json file. The flask application then reades the json file and gives
> result.
> 
> During the development it was smooth and both upload and download by the
> server was performed with out any issues. But once I but the app in the
> deployment its giving a pain.
> 
> Any suggestion to fix the permission issue during deployment in production
> server. Wll be helpful.
> 
> -- 
> With regards,
> 
> Taushif Khan
> Ph.D.
> PostDoc Research
> Interuniversity Institute of Bioinformatics in Brussels
> <http://ibsquare.be/>
> Brussels, Belgium
> 
> 
> *The true sign of intelligence is not knowledge but imagination.
> <http://www.brainyquote.com/quotes/quotes/a/alberteins148802.html> *[*Einstein
> <http://www.brainyquote.com/quotes/quotes/a/alberteins148802.html> A]*

From tim at shisaa.jp  Wed Jul  6 02:40:57 2016
From: tim at shisaa.jp (Tim van der Linden)
Date: Wed, 6 Jul 2016 15:40:57 +0900
Subject: [Flask] uWSGI and Flask - Slow and varying response times
Message-ID: <20160706154057.83b8a86b5472ef3ffd232c4b@shisaa.jp>

Hi All

I'm in a bit of a head scratching situation and need some fresh pair of eyes on this.

I have chosen to post this on the Flask mailing list as my application is pure Flask ... but I think the problem might involve the whole server stack.

I'm hoping you guys can give me some pointers as to where I should look for solutions.

The situation: I have a freshly deployed Flask application running under uWSGI with Nginx in front and PostgreSQL 9.5 as the database backend. Nothing special there, a typical setup.

This stack is running on a large, dedicated server (not a VPS) with 64 cores, 130 Gb of RAM and 1.4 TB of SSDs. Also not too bad.

The application is finished and I am now benchmarking it and optimizing where needed.

The problem:

On my local development machine the most heavy pages return in around ~200 ms, which is quite fast considering what is happening on such a page. Some database queries need to be pulled, some JSON needs to be decoded and Jinja2 needs to construct and render a fairly large page (without Jinja caching). Flask rocks!

On the production machine (the huge machine described above), however, I get response times that are all over the place. The lowest is ~500 ms the longest is well over 2 seconds.

Even the lowest time is more than double of what I can get out of my development machine which is less than 10% of the server specs and running with the built in Werkzeug development sever.

Now, of course, the production server is not as isolated as my development machine but it is not sweating either. It currently gets around ~10 hits per second on uWSGI (static content not counted as this is directly handled by Nginx). 

The database itself is handling a few dozen queries a second and is almost not showing up in the process list. Only the uWSGI workers (about 10 of them, the rest are idle) consume around 30~60% CPU with some peaks to 90%.

In other words: the whole server isn't doing a whole lot but the response times are not only much slower than local but also greatly vary in between requests.

I even profiled the server with Werkzeug's ProfilerMiddleware and SnakeViz which doesn't give me much direction. The function load is spread quite identically in every request - if the request takes four times as long, all functions take four times as long as well. 

This would indicate that Python in general is slow ... ?

Do you guys have any experience with running Flask behind uWSGI/Nginx and having slow and varying response times?

Some extra info:

The current vassal configuration for uWSGI is running with 20 processes and 50 threads and talking directly with Nginx over a UNIX socket (no TCP).

PostgreSQL is set to handle around 250 concurrent connections with shared buffer of 48Gb.

All insane values.

Cheers,
Tim

From alejoar at gmail.com  Wed Jul  6 04:26:28 2016
From: alejoar at gmail.com (Alejo Arias)
Date: Wed, 6 Jul 2016 10:26:28 +0200
Subject: [Flask] uWSGI and Flask - Slow and varying response times
In-Reply-To: <20160706154057.83b8a86b5472ef3ffd232c4b@shisaa.jp>
References: <20160706154057.83b8a86b5472ef3ffd232c4b@shisaa.jp>
Message-ID: <CAMfrFKQk9ad=F0v5OpgJ7tBothfENa8FwnoWXZ6u=DVv8FF4Hw@mail.gmail.com>

Can you ssh into the server and make requests to localhost? (for example
with httpie or curl) If those show no delay you have a connection issue.

Other thing I can think of is the server might be running other heavy
processes. Maybe something related to log processing that's consuming lots
of resources, but with such a beast of a server I doubt it.

Nothing comes to mind that might be causing this that's directly related to
python or flask other than the code itself.

Regards,
Alejo

On 6 July 2016 at 08:40, Tim van der Linden <tim at shisaa.jp> wrote:

> Hi All
>
> I'm in a bit of a head scratching situation and need some fresh pair of
> eyes on this.
>
> I have chosen to post this on the Flask mailing list as my application is
> pure Flask ... but I think the problem might involve the whole server stack.
>
> I'm hoping you guys can give me some pointers as to where I should look
> for solutions.
>
> The situation: I have a freshly deployed Flask application running under
> uWSGI with Nginx in front and PostgreSQL 9.5 as the database backend.
> Nothing special there, a typical setup.
>
> This stack is running on a large, dedicated server (not a VPS) with 64
> cores, 130 Gb of RAM and 1.4 TB of SSDs. Also not too bad.
>
> The application is finished and I am now benchmarking it and optimizing
> where needed.
>
> The problem:
>
> On my local development machine the most heavy pages return in around ~200
> ms, which is quite fast considering what is happening on such a page. Some
> database queries need to be pulled, some JSON needs to be decoded and
> Jinja2 needs to construct and render a fairly large page (without Jinja
> caching). Flask rocks!
>
> On the production machine (the huge machine described above), however, I
> get response times that are all over the place. The lowest is ~500 ms the
> longest is well over 2 seconds.
>
> Even the lowest time is more than double of what I can get out of my
> development machine which is less than 10% of the server specs and running
> with the built in Werkzeug development sever.
>
> Now, of course, the production server is not as isolated as my development
> machine but it is not sweating either. It currently gets around ~10 hits
> per second on uWSGI (static content not counted as this is directly handled
> by Nginx).
>
> The database itself is handling a few dozen queries a second and is almost
> not showing up in the process list. Only the uWSGI workers (about 10 of
> them, the rest are idle) consume around 30~60% CPU with some peaks to 90%.
>
> In other words: the whole server isn't doing a whole lot but the response
> times are not only much slower than local but also greatly vary in between
> requests.
>
> I even profiled the server with Werkzeug's ProfilerMiddleware and SnakeViz
> which doesn't give me much direction. The function load is spread quite
> identically in every request - if the request takes four times as long, all
> functions take four times as long as well.
>
> This would indicate that Python in general is slow ... ?
>
> Do you guys have any experience with running Flask behind uWSGI/Nginx and
> having slow and varying response times?
>
> Some extra info:
>
> The current vassal configuration for uWSGI is running with 20 processes
> and 50 threads and talking directly with Nginx over a UNIX socket (no TCP).
>
> PostgreSQL is set to handle around 250 concurrent connections with shared
> buffer of 48Gb.
>
> All insane values.
>
> Cheers,
> Tim
> _______________________________________________
> Flask mailing list
> Flask at python.org
> https://mail.python.org/mailman/listinfo/flask
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/flask/attachments/20160706/d9c0cc16/attachment.html>

From tim at shisaa.jp  Wed Jul  6 05:02:52 2016
From: tim at shisaa.jp (Tim van der Linden)
Date: Wed, 6 Jul 2016 18:02:52 +0900
Subject: [Flask] uWSGI and Flask - Slow and varying response times
In-Reply-To: <CAMfrFKQk9ad=F0v5OpgJ7tBothfENa8FwnoWXZ6u=DVv8FF4Hw@mail.gmail.com>
References: <20160706154057.83b8a86b5472ef3ffd232c4b@shisaa.jp>
 <CAMfrFKQk9ad=F0v5OpgJ7tBothfENa8FwnoWXZ6u=DVv8FF4Hw@mail.gmail.com>
Message-ID: <20160706180252.1ffb7eda068f72ab2121cd39@shisaa.jp>

On Wed, 6 Jul 2016 10:26:28 +0200
Alejo Arias <alejoar at gmail.com> wrote:

Hi Alejo

Thanks for the reply.

> Can you ssh into the server and make requests to localhost? (for example
> with httpie or curl) If those show no delay you have a connection issue.

Good point. However, the response time comes from the Werkzeug profiler directly done on the production server so there is no network overhead.

> Other thing I can think of is the server might be running other heavy
> processes. Maybe something related to log processing that's consuming lots
> of resources, but with such a beast of a server I doubt it.

Hmm, well uWSGI and PostgreSQL have quite the free range on the server.
There is not much else going on on that machine.

> Nothing comes to mind that might be causing this that's directly related to
> python or flask other than the code itself.

But if it was the coding, it would also show up on development profiling, no?

Thanks for your insight!

> Regards,
> Alejo

Cheers,
Tim

From mark at pors.net  Wed Jul  6 05:49:34 2016
From: mark at pors.net (Mark Pors)
Date: Wed, 6 Jul 2016 12:49:34 +0300
Subject: [Flask] uWSGI and Flask - Slow and varying response times
In-Reply-To: <20160706154057.83b8a86b5472ef3ffd232c4b@shisaa.jp>
References: <20160706154057.83b8a86b5472ef3ffd232c4b@shisaa.jp>
Message-ID: <CA+9AWjE0hNLsoEmNhP_Wm2=frcVybLU+rsOiYt6Y+U2Ery8c_g@mail.gmail.com>

On Wed, Jul 6, 2016 at 9:40 AM, Tim van der Linden <tim at shisaa.jp> wrote:

> Hi All
>
> I'm in a bit of a head scratching situation and need some fresh pair of
> eyes on this.
>
> I have chosen to post this on the Flask mailing list as my application is
> pure Flask ... but I think the problem might involve the whole server stack.
>
> I'm hoping you guys can give me some pointers as to where I should look
> for solutions.
>
> The situation: I have a freshly deployed Flask application running under
> uWSGI with Nginx in front and PostgreSQL 9.5 as the database backend.
> Nothing special there, a typical setup.
>
> This stack is running on a large, dedicated server (not a VPS) with 64
> cores, 130 Gb of RAM and 1.4 TB of SSDs. Also not too bad.
>
> The application is finished and I am now benchmarking it and optimizing
> where needed.
>
> The problem:
>
> On my local development machine the most heavy pages return in around ~200
> ms, which is quite fast considering what is happening on such a page. Some
> database queries need to be pulled, some JSON needs to be decoded and
> Jinja2 needs to construct and render a fairly large page (without Jinja
> caching). Flask rocks!
>
> On the production machine (the huge machine described above), however, I
> get response times that are all over the place. The lowest is ~500 ms the
> longest is well over 2 seconds.
>
> Even the lowest time is more than double of what I can get out of my
> development machine which is less than 10% of the server specs and running
> with the built in Werkzeug development sever.
>
> Now, of course, the production server is not as isolated as my development
> machine but it is not sweating either. It currently gets around ~10 hits
> per second on uWSGI (static content not counted as this is directly handled
> by Nginx).
>
> The database itself is handling a few dozen queries a second and is almost
> not showing up in the process list. Only the uWSGI workers (about 10 of
> them, the rest are idle) consume around 30~60% CPU with some peaks to 90%.
>
> In other words: the whole server isn't doing a whole lot but the response
> times are not only much slower than local but also greatly vary in between
> requests.
>
> I even profiled the server with Werkzeug's ProfilerMiddleware and SnakeViz
> which doesn't give me much direction. The function load is spread quite
> identically in every request - if the request takes four times as long, all
> functions take four times as long as well.
>
>

All functions? Also if there is no DB access or other I/O initiated by
these functions (or waiting on it)?



> This would indicate that Python in general is slow ... ?
>
> Do you guys have any experience with running Flask behind uWSGI/Nginx and
> having slow and varying response times?
>
> Some extra info:
>
> The current vassal configuration for uWSGI is running with 20 processes
> and 50 threads and talking directly with Nginx over a UNIX socket (no TCP).
>
> PostgreSQL is set to handle around 250 concurrent connections with shared
> buffer of 48Gb.
>
> All insane values.
>
> Cheers,
> Tim
> _______________________________________________
> Flask mailing list
> Flask at python.org
> https://mail.python.org/mailman/listinfo/flask
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/flask/attachments/20160706/af50edb5/attachment-0001.html>

From unai at sysbible.org  Wed Jul  6 05:52:00 2016
From: unai at sysbible.org (Unai Rodriguez)
Date: Wed, 06 Jul 2016 17:52:00 +0800
Subject: [Flask] uWSGI and Flask - Slow and varying response times
In-Reply-To: <20160706180252.1ffb7eda068f72ab2121cd39@shisaa.jp>
References: <20160706154057.83b8a86b5472ef3ffd232c4b@shisaa.jp>
 <CAMfrFKQk9ad=F0v5OpgJ7tBothfENa8FwnoWXZ6u=DVv8FF4Hw@mail.gmail.com>
 <20160706180252.1ffb7eda068f72ab2121cd39@shisaa.jp>
Message-ID: <1467798720.439156.658232313.2A5E7454@webmail.messagingengine.com>

Hi Tim,

I think your top priority should be to get consistent results first and
for the that you want to stop all other activity on your production
server and benchmark your application alone. Would it be feasible to
declare downtime in all other apps for 30 minutes? Alternatively you
might want to run all apps on your prodution server somewhere else while
you benchmark your app.

This will tell you if the inconsistencies come from the other apps, or
it is an issue with your code or server configuration.  

-- unai

On Wed, Jul 6, 2016, at 05:02 PM, Tim van der Linden wrote:
> On Wed, 6 Jul 2016 10:26:28 +0200
> Alejo Arias <alejoar at gmail.com> wrote:
> 
> Hi Alejo
> 
> Thanks for the reply.
> 
> > Can you ssh into the server and make requests to localhost? (for example
> > with httpie or curl) If those show no delay you have a connection issue.
> 
> Good point. However, the response time comes from the Werkzeug profiler
> directly done on the production server so there is no network overhead.
> 
> > Other thing I can think of is the server might be running other heavy
> > processes. Maybe something related to log processing that's consuming lots
> > of resources, but with such a beast of a server I doubt it.
> 
> Hmm, well uWSGI and PostgreSQL have quite the free range on the server.
> There is not much else going on on that machine.
> 
> > Nothing comes to mind that might be causing this that's directly related to
> > python or flask other than the code itself.
> 
> But if it was the coding, it would also show up on development profiling,
> no?
> 
> Thanks for your insight!
> 
> > Regards,
> > Alejo
> 
> Cheers,
> Tim
> _______________________________________________
> Flask mailing list
> Flask at python.org
> https://mail.python.org/mailman/listinfo/flask

From tim at shisaa.jp  Wed Jul  6 06:16:25 2016
From: tim at shisaa.jp (Tim van der Linden)
Date: Wed, 6 Jul 2016 19:16:25 +0900
Subject: [Flask] uWSGI and Flask - Slow and varying response times
In-Reply-To: <1467798720.439156.658232313.2A5E7454@webmail.messagingengine.com>
References: <20160706154057.83b8a86b5472ef3ffd232c4b@shisaa.jp>
 <CAMfrFKQk9ad=F0v5OpgJ7tBothfENa8FwnoWXZ6u=DVv8FF4Hw@mail.gmail.com>
 <20160706180252.1ffb7eda068f72ab2121cd39@shisaa.jp>
 <1467798720.439156.658232313.2A5E7454@webmail.messagingengine.com>
Message-ID: <20160706191625.c6c547c2b3d585fea6bc13da@shisaa.jp>

On Wed, 06 Jul 2016 17:52:00 +0800
Unai Rodriguez <unai at sysbible.org> wrote:

> Hi Tim,

Hi Unai

Thanks for the response!

> I think your top priority should be to get consistent results first and
> for the that you want to stop all other activity on your production
> server and benchmark your application alone. Would it be feasible to
> declare downtime in all other apps for 30 minutes? Alternatively you
> might want to run all apps on your prodution server somewhere else while
> you benchmark your app.
> 
> This will tell you if the inconsistencies come from the other apps, or
> it is an issue with your code or server configuration.  

Very good suggestion and I just had the timing to do so. I allowed only, my local IP access to the server and nobody else. The server fell even more silent and no requests came in except mine.

I ran the profiler again which gave me very similar results to before: average response time ~700 ms. Average response time on my local machine: ~200 ms (exact same database and code).

So even when this beast of a server is fully isolated and only answers to my requests, it does so 3 to 4 times slower than on my development machine. 

>From my perspective, if the code were the bottleneck then I would also get the same "slow" performance on my local machine ... no? 

My gut feeling says that there is some sort of misconfiguration going on.

> -- unai

Cheers,
Tim

From alejoar at gmail.com  Wed Jul  6 06:19:42 2016
From: alejoar at gmail.com (Alejo Arias)
Date: Wed, 6 Jul 2016 12:19:42 +0200
Subject: [Flask] uWSGI and Flask - Slow and varying response times
In-Reply-To: <20160706191625.c6c547c2b3d585fea6bc13da@shisaa.jp>
References: <20160706154057.83b8a86b5472ef3ffd232c4b@shisaa.jp>
 <CAMfrFKQk9ad=F0v5OpgJ7tBothfENa8FwnoWXZ6u=DVv8FF4Hw@mail.gmail.com>
 <20160706180252.1ffb7eda068f72ab2121cd39@shisaa.jp>
 <1467798720.439156.658232313.2A5E7454@webmail.messagingengine.com>
 <20160706191625.c6c547c2b3d585fea6bc13da@shisaa.jp>
Message-ID: <CAMfrFKShUXNybrbY1W3Jkf_eiLz5aqaY7KKagSThxSh8N0WhRw@mail.gmail.com>

Is the DB hosted in the same server?

On 6 July 2016 at 12:16, Tim van der Linden <tim at shisaa.jp> wrote:

> On Wed, 06 Jul 2016 17:52:00 +0800
> Unai Rodriguez <unai at sysbible.org> wrote:
>
> > Hi Tim,
>
> Hi Unai
>
> Thanks for the response!
>
> > I think your top priority should be to get consistent results first and
> > for the that you want to stop all other activity on your production
> > server and benchmark your application alone. Would it be feasible to
> > declare downtime in all other apps for 30 minutes? Alternatively you
> > might want to run all apps on your prodution server somewhere else while
> > you benchmark your app.
> >
> > This will tell you if the inconsistencies come from the other apps, or
> > it is an issue with your code or server configuration.
>
> Very good suggestion and I just had the timing to do so. I allowed only,
> my local IP access to the server and nobody else. The server fell even more
> silent and no requests came in except mine.
>
> I ran the profiler again which gave me very similar results to before:
> average response time ~700 ms. Average response time on my local machine:
> ~200 ms (exact same database and code).
>
> So even when this beast of a server is fully isolated and only answers to
> my requests, it does so 3 to 4 times slower than on my development machine.
>
> From my perspective, if the code were the bottleneck then I would also get
> the same "slow" performance on my local machine ... no?
>
> My gut feeling says that there is some sort of misconfiguration going on.
>
> > -- unai
>
> Cheers,
> Tim
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/flask/attachments/20160706/848067f4/attachment.html>

From aco at miza.org  Wed Jul  6 06:24:09 2016
From: aco at miza.org (Aco Strkalj)
Date: Wed, 6 Jul 2016 05:24:09 -0500
Subject: [Flask] uWSGI and Flask - Slow and varying response times
In-Reply-To: <20160706191625.c6c547c2b3d585fea6bc13da@shisaa.jp>
References: <20160706154057.83b8a86b5472ef3ffd232c4b@shisaa.jp>
 <CAMfrFKQk9ad=F0v5OpgJ7tBothfENa8FwnoWXZ6u=DVv8FF4Hw@mail.gmail.com>
 <20160706180252.1ffb7eda068f72ab2121cd39@shisaa.jp>
 <1467798720.439156.658232313.2A5E7454@webmail.messagingengine.com>
 <20160706191625.c6c547c2b3d585fea6bc13da@shisaa.jp>
Message-ID: <FEAABD08-F76D-4151-A4AF-F4289CEC6C49@miza.org>

What version of Python are you running on both setups?

Ever tried looking into PyPy JIT? 

Odds are your server isn't configured properly to take advantage this beauty system. 

What OS / version do you run on your dev and prod machines ?

Sent from my iPhone

> On Jul 6, 2016, at 5:16 AM, Tim van der Linden <tim at shisaa.jp> wrote:
> 
> On Wed, 06 Jul 2016 17:52:00 +0800
> Unai Rodriguez <unai at sysbible.org> wrote:
> 
>> Hi Tim,
> 
> Hi Unai
> 
> Thanks for the response!
> 
>> I think your top priority should be to get consistent results first and
>> for the that you want to stop all other activity on your production
>> server and benchmark your application alone. Would it be feasible to
>> declare downtime in all other apps for 30 minutes? Alternatively you
>> might want to run all apps on your prodution server somewhere else while
>> you benchmark your app.
>> 
>> This will tell you if the inconsistencies come from the other apps, or
>> it is an issue with your code or server configuration.  
> 
> Very good suggestion and I just had the timing to do so. I allowed only, my local IP access to the server and nobody else. The server fell even more silent and no requests came in except mine.
> 
> I ran the profiler again which gave me very similar results to before: average response time ~700 ms. Average response time on my local machine: ~200 ms (exact same database and code).
> 
> So even when this beast of a server is fully isolated and only answers to my requests, it does so 3 to 4 times slower than on my development machine. 
> 
> From my perspective, if the code were the bottleneck then I would also get the same "slow" performance on my local machine ... no? 
> 
> My gut feeling says that there is some sort of misconfiguration going on.
> 
>> -- unai
> 
> Cheers,
> Tim
> _______________________________________________
> Flask mailing list
> Flask at python.org
> https://mail.python.org/mailman/listinfo/flask

From tim at shisaa.jp  Wed Jul  6 06:27:57 2016
From: tim at shisaa.jp (Tim van der Linden)
Date: Wed, 6 Jul 2016 19:27:57 +0900
Subject: [Flask] uWSGI and Flask - Slow and varying response times
In-Reply-To: <CAMfrFKShUXNybrbY1W3Jkf_eiLz5aqaY7KKagSThxSh8N0WhRw@mail.gmail.com>
References: <20160706154057.83b8a86b5472ef3ffd232c4b@shisaa.jp>
 <CAMfrFKQk9ad=F0v5OpgJ7tBothfENa8FwnoWXZ6u=DVv8FF4Hw@mail.gmail.com>
 <20160706180252.1ffb7eda068f72ab2121cd39@shisaa.jp>
 <1467798720.439156.658232313.2A5E7454@webmail.messagingengine.com>
 <20160706191625.c6c547c2b3d585fea6bc13da@shisaa.jp>
 <CAMfrFKShUXNybrbY1W3Jkf_eiLz5aqaY7KKagSThxSh8N0WhRw@mail.gmail.com>
Message-ID: <20160706192757.61a68d0cb0bb3966aea5634e@shisaa.jp>

On Wed, 6 Jul 2016 12:19:42 +0200
Alejo Arias <alejoar at gmail.com> wrote:

Hi Alejo

> Is the DB hosted in the same server?

Yep, the database cluster is on the same SSD drive.

I also just checked one of the slowest pages and the database queries total up to around ~90 ms of the response time including every transaction start and commit. That is for the amount of queries done also very acceptable.

I think this rules out the database as the bottleneck since my local machine clocks in at roughly the same amount of time for database work, meaning all other Python work is done in the remaining ~100 ms.

The server needs ~600 ms for this remaining work.

Cheers,
Tim

> On 6 July 2016 at 12:16, Tim van der Linden <tim at shisaa.jp> wrote:
> 
> > On Wed, 06 Jul 2016 17:52:00 +0800
> > Unai Rodriguez <unai at sysbible.org> wrote:
> >
> > > Hi Tim,
> >
> > Hi Unai
> >
> > Thanks for the response!
> >
> > > I think your top priority should be to get consistent results first and
> > > for the that you want to stop all other activity on your production
> > > server and benchmark your application alone. Would it be feasible to
> > > declare downtime in all other apps for 30 minutes? Alternatively you
> > > might want to run all apps on your prodution server somewhere else while
> > > you benchmark your app.
> > >
> > > This will tell you if the inconsistencies come from the other apps, or
> > > it is an issue with your code or server configuration.
> >
> > Very good suggestion and I just had the timing to do so. I allowed only,
> > my local IP access to the server and nobody else. The server fell even more
> > silent and no requests came in except mine.
> >
> > I ran the profiler again which gave me very similar results to before:
> > average response time ~700 ms. Average response time on my local machine:
> > ~200 ms (exact same database and code).
> >
> > So even when this beast of a server is fully isolated and only answers to
> > my requests, it does so 3 to 4 times slower than on my development machine.
> >
> > From my perspective, if the code were the bottleneck then I would also get
> > the same "slow" performance on my local machine ... no?
> >
> > My gut feeling says that there is some sort of misconfiguration going on.
> >
> > > -- unai
> >
> > Cheers,
> > Tim
> >


-- 
Tim van der Linden <tim at shisaa.jp>

From unai at sysbible.org  Wed Jul  6 06:28:15 2016
From: unai at sysbible.org (Unai Rodriguez)
Date: Wed, 06 Jul 2016 18:28:15 +0800
Subject: [Flask] uWSGI and Flask - Slow and varying response times
In-Reply-To: <20160706191625.c6c547c2b3d585fea6bc13da@shisaa.jp>
References: <20160706154057.83b8a86b5472ef3ffd232c4b@shisaa.jp>
 <CAMfrFKQk9ad=F0v5OpgJ7tBothfENa8FwnoWXZ6u=DVv8FF4Hw@mail.gmail.com>
 <20160706180252.1ffb7eda068f72ab2121cd39@shisaa.jp>
 <1467798720.439156.658232313.2A5E7454@webmail.messagingengine.com>
 <20160706191625.c6c547c2b3d585fea6bc13da@shisaa.jp>
Message-ID: <1467800895.445316.658254993.2C4F7D55@webmail.messagingengine.com>

Hi Tim,

Great , it's now time of testing the different parts of the server. You
could start testing the database, uwsgi/Python and maybe the network ?

You could build two queries, a simple and one more complicated and run
it multiple times on your dev and prod boxes and compare. Different
times? Consistent ?     


For uwsgi/Python you could do the same, build a simple page and a more
complicated one and compare (no database queries to isolate).

Lastly for the network, was this isolated already from your tests? I
think so but if it wasnt you'll need to look at some tools
(munin?cacti?)

The server/os will have to be looked at some point as well. 

Also, are all other app currently running  on your production behaving
better in terms of performance and consistency ?                
-- unai

On Wed, Jul 6, 2016, at 06:16 PM, Tim van der Linden wrote:
> On Wed, 06 Jul 2016 17:52:00 +0800
> Unai Rodriguez <unai at sysbible.org> wrote:
> 
> > Hi Tim,
> 
> Hi Unai
> 
> Thanks for the response!
> 
> > I think your top priority should be to get consistent results first and
> > for the that you want to stop all other activity on your production
> > server and benchmark your application alone. Would it be feasible to
> > declare downtime in all other apps for 30 minutes? Alternatively you
> > might want to run all apps on your prodution server somewhere else while
> > you benchmark your app.
> > 
> > This will tell you if the inconsistencies come from the other apps, or
> > it is an issue with your code or server configuration.  
> 
> Very good suggestion and I just had the timing to do so. I allowed only,
> my local IP access to the server and nobody else. The server fell even
> more silent and no requests came in except mine.
> 
> I ran the profiler again which gave me very similar results to before:
> average response time ~700 ms. Average response time on my local machine:
> ~200 ms (exact same database and code).
> 
> So even when this beast of a server is fully isolated and only answers to
> my requests, it does so 3 to 4 times slower than on my development
> machine. 
> 
> From my perspective, if the code were the bottleneck then I would also
> get the same "slow" performance on my local machine ... no? 
> 
> My gut feeling says that there is some sort of misconfiguration going on.
> 
> > -- unai
> 
> Cheers,
> Tim

From tim at shisaa.jp  Wed Jul  6 06:39:06 2016
From: tim at shisaa.jp (Tim van der Linden)
Date: Wed, 6 Jul 2016 19:39:06 +0900
Subject: [Flask] uWSGI and Flask - Slow and varying response times
In-Reply-To: <FEAABD08-F76D-4151-A4AF-F4289CEC6C49@miza.org>
References: <20160706154057.83b8a86b5472ef3ffd232c4b@shisaa.jp>
 <CAMfrFKQk9ad=F0v5OpgJ7tBothfENa8FwnoWXZ6u=DVv8FF4Hw@mail.gmail.com>
 <20160706180252.1ffb7eda068f72ab2121cd39@shisaa.jp>
 <1467798720.439156.658232313.2A5E7454@webmail.messagingengine.com>
 <20160706191625.c6c547c2b3d585fea6bc13da@shisaa.jp>
 <FEAABD08-F76D-4151-A4AF-F4289CEC6C49@miza.org>
Message-ID: <20160706193906.4f5c3b8cb621824717bb1ef0@shisaa.jp>

On Wed, 6 Jul 2016 05:24:09 -0500
Aco Strkalj <aco at miza.org> wrote:

Hi Aco

> What version of Python are you running on both setups?

Both are running Python 2.7.11 and both are built directly on the systems.

> Ever tried looking into PyPy JIT? 

Hmm, I am a bit wary of PyPy (mostly because of the unknowns) so first want to see what I can get out of regular CPython. Especially seeing that on my machine it does perform very well.

> Odds are your server isn't configured properly to take advantage this beauty system. 
> 
> What OS / version do you run on your dev and prod machines ?

Both systems run the same Debian 8 OS (64 bits) and the production server is fully setup and allocated by Softlayer.

Both web stacks are setup in the same Virtualenv version and all stack pieces are the same version:

- Nginx: 1.6.2
- uWSGI:2.0.13.1
- Python: 2.7.11
- Flask: 0.10.1
- Werkzeug: 0.11.4

> Sent from my iPhone

Cheers,
Tim

From tim at shisaa.jp  Wed Jul  6 06:44:54 2016
From: tim at shisaa.jp (Tim van der Linden)
Date: Wed, 6 Jul 2016 19:44:54 +0900
Subject: [Flask] uWSGI and Flask - Slow and varying response times
In-Reply-To: <1467800895.445316.658254993.2C4F7D55@webmail.messagingengine.com>
References: <20160706154057.83b8a86b5472ef3ffd232c4b@shisaa.jp>
 <CAMfrFKQk9ad=F0v5OpgJ7tBothfENa8FwnoWXZ6u=DVv8FF4Hw@mail.gmail.com>
 <20160706180252.1ffb7eda068f72ab2121cd39@shisaa.jp>
 <1467798720.439156.658232313.2A5E7454@webmail.messagingengine.com>
 <20160706191625.c6c547c2b3d585fea6bc13da@shisaa.jp>
 <1467800895.445316.658254993.2C4F7D55@webmail.messagingengine.com>
Message-ID: <20160706194454.68933bf55d1cf30a8385939c@shisaa.jp>

On Wed, 06 Jul 2016 18:28:15 +0800
Unai Rodriguez <unai at sysbible.org> wrote:

> Hi Tim,

Hi Unai

> Great , it's now time of testing the different parts of the server. You
> could start testing the database, uwsgi/Python and maybe the network ?

Sounds like a plan.

> You could build two queries, a simple and one more complicated and run
> it multiple times on your dev and prod boxes and compare. Different
> times? Consistent ?     
> 
> For uwsgi/Python you could do the same, build a simple page and a more
> complicated one and compare (no database queries to isolate).

Yep, I think this is the next step I need to take, create isolated test cases.
Especially for the Python code as it seems that the database is not the problem.

Let me get to that tomorrow ... getting late over in this neck of the woods :)

> Lastly for the network, was this isolated already from your tests? I
> think so but if it wasnt you'll need to look at some tools
> (munin?cacti?)

Yep, all network overhead was mitigated as the response time came from the CProfiler running directly on the server.

> The server/os will have to be looked at some point as well. 
> 
> Also, are all other app currently running  on your production behaving
> better in terms of performance and consistency ?   

As far as web applications go this is the only application on the server.
This whole machine is only for this application ... which makes working on this application fun for me as a developer :)

In terms of applications like PostgreSQL, these behave very stable.
Querying directly in the CLI via SSH is lighting fast, even under load.
The web server and uWSGI also seem to be stable.
             
> -- unai

Cheers,
Tim

From tim at shisaa.jp  Wed Jul  6 19:18:11 2016
From: tim at shisaa.jp (Tim van der Linden)
Date: Thu, 7 Jul 2016 08:18:11 +0900
Subject: [Flask] uWSGI and Flask - Slow and varying response times
In-Reply-To: <CA+9AWjE0hNLsoEmNhP_Wm2=frcVybLU+rsOiYt6Y+U2Ery8c_g@mail.gmail.com>
References: <20160706154057.83b8a86b5472ef3ffd232c4b@shisaa.jp>
 <CA+9AWjE0hNLsoEmNhP_Wm2=frcVybLU+rsOiYt6Y+U2Ery8c_g@mail.gmail.com>
Message-ID: <20160707081811.f15e0cb95bfc87aee0a65458@shisaa.jp>

On Wed, 6 Jul 2016 12:49:34 +0300
Mark Pors <mark at pors.net> wrote:

> On Wed, Jul 6, 2016 at 9:40 AM, Tim van der Linden <tim at shisaa.jp> wrote:

Hi Mark

> > ...
> > I even profiled the server with Werkzeug's ProfilerMiddleware and SnakeViz
> > which doesn't give me much direction. The function load is spread quite
> > identically in every request - if the request takes four times as long, all
> > functions take four times as long as well.
> >
> >
> 
> All functions? Also if there is no DB access or other I/O initiated by
> these functions (or waiting on it)?

All or at least most of the functions seem stretched in time. Many of them are indeed database related functions which led me to believe it was a typical database issue at first.

However, there are not only database related functions but also functions such as the template rendering which equally slow down when the response time is higher. This brought me on the current track that there might be an issue with Python processing being abnormally slow.

Also querying the database via its CLI was very fast and did not show any lag, the database itself is also not under any particularity heavy load. Connection between Python and the database is done via PsycoPG2 (not ORM) using direct sockets.

> > ...

Cheers,
Tim

From tim at shisaa.jp  Wed Jul  6 23:40:30 2016
From: tim at shisaa.jp (Tim van der Linden)
Date: Thu, 7 Jul 2016 12:40:30 +0900
Subject: [Flask] uWSGI and Flask - Slow and varying response times
In-Reply-To: <1467800895.445316.658254993.2C4F7D55@webmail.messagingengine.com>
References: <20160706154057.83b8a86b5472ef3ffd232c4b@shisaa.jp>
 <CAMfrFKQk9ad=F0v5OpgJ7tBothfENa8FwnoWXZ6u=DVv8FF4Hw@mail.gmail.com>
 <20160706180252.1ffb7eda068f72ab2121cd39@shisaa.jp>
 <1467798720.439156.658232313.2A5E7454@webmail.messagingengine.com>
 <20160706191625.c6c547c2b3d585fea6bc13da@shisaa.jp>
 <1467800895.445316.658254993.2C4F7D55@webmail.messagingengine.com>
Message-ID: <20160707124030.68c3f7bf5bfdbb0467100ac4@shisaa.jp>

On Wed, 06 Jul 2016 18:28:15 +0800
Unai Rodriguez <unai at sysbible.org> wrote:

> Hi Tim,

Hi Unai

> For uwsgi/Python you could do the same, build a simple page and a more
> complicated one and compare (no database queries to isolate).

First a simple page on uWSGI using Flask and Jinja. The template only consists out of the bare minimals for a HTML page and a single string variable to be printed and the whole application is a single Python file with only one view. The timings are done using the Werkzeug profiler:

Local: 7 ms for initial request, 0.5ms for all following requests
Server: 160 ms for initial request, 1ms for all following requests

I will count that as identical. The Snakeviz graphs also look identical for both.

Then I added a few range(1, 1000) loops to be printed in Jinja2 and a load of simple text:

Local: 9 ms for initial request, 3 ms for all following requests
Server: 24 ms for initial request, 10 ms for all following requests

Slowly a difference between the two start to form.

I added some more elements to the view and template such as a WTF form, but the timings did not went up much. The difference stayed however: server being three times slower then local.

I also added a couple of (fast) database queries in the mix, opening and closing both cursors and connections per query:

Local: 16 ms for initial request, 11 ms for all following requests
Server: 51 ms for initial request, 33 ms for all following requests

> The server/os will have to be looked at some point as well. 

This was interesting but maybe less important since this is testing pure CPU throughput:, I yanked a primes Python script of the interwebs to test the raw processing speed and ran it on both machines.

Local: 3.3879199028 s
Server: 9.26460504532 s

This is a script without any extras, one Python function tested with Timeit and ran directly via the Python CLI. My local machine is three times faster than the server.

Installing Sysbench and doing a primers test (20000) outside of Python:

Local: 18.1605 seconds
Server: 42.1942 seconds

CPU info:

Local: 1 x i74790K (4 Ghz)
Server: 8 x Xeon E7-4820 v2 (2 Ghz)

> -- unai

Cheers,
Tim

From ziirish at ziirish.info  Thu Jul  7 02:42:27 2016
From: ziirish at ziirish.info (Ziirish)
Date: Thu, 07 Jul 2016 08:42:27 +0200
Subject: [Flask] uWSGI and Flask - Slow and varying response times
In-Reply-To: <20160707124030.68c3f7bf5bfdbb0467100ac4@shisaa.jp>
References: <20160706154057.83b8a86b5472ef3ffd232c4b@shisaa.jp>
 <CAMfrFKQk9ad=F0v5OpgJ7tBothfENa8FwnoWXZ6u=DVv8FF4Hw@mail.gmail.com>
 <20160706180252.1ffb7eda068f72ab2121cd39@shisaa.jp>
 <1467798720.439156.658232313.2A5E7454@webmail.messagingengine.com>
 <20160706191625.c6c547c2b3d585fea6bc13da@shisaa.jp>
 <1467800895.445316.658254993.2C4F7D55@webmail.messagingengine.com>
 <20160707124030.68c3f7bf5bfdbb0467100ac4@shisaa.jp>
Message-ID: <5AF87A5B-83A1-4A28-A49B-1100670332B9@ziirish.info>

Hello,

I think you have your answer right with the last piece of information you gave us.

Your CPU clock is two times faster than your servers one.

Now your code is probably not multi threaded, it means your server will perform at least two times slower than your development machine to execute the same piece of code.

The x3 you are measuring is probably due to some context switch or something (your kernel is probably spitting the task in several pieces to let other processes perform well so these pieces of processes then need to be synced or something)

In conclusion, the delay you are observing is due to a lower CPU clock on the server. So unless you can multi thread your application there's not much to do.
In the other hand, because you have a lot of cores, your server will be able to serve (at least) 8 simultaneous clients within the same time range.

You can also have a look at PyPy as already suggested to try to optimize the generated "bytecode". It can probably save you a few CPU cycles and thus decrease your whole compute time.




Le 7 juillet 2016 05:40:30 GMT+02:00, Tim van der Linden <tim at shisaa.jp> a ?crit :
>On Wed, 06 Jul 2016 18:28:15 +0800
>Unai Rodriguez <unai at sysbible.org> wrote:
>
>> Hi Tim,
>
>Hi Unai
>
>> For uwsgi/Python you could do the same, build a simple page and a
>more
>> complicated one and compare (no database queries to isolate).
>
>First a simple page on uWSGI using Flask and Jinja. The template only
>consists out of the bare minimals for a HTML page and a single string
>variable to be printed and the whole application is a single Python
>file with only one view. The timings are done using the Werkzeug
>profiler:
>
>Local: 7 ms for initial request, 0.5ms for all following requests
>Server: 160 ms for initial request, 1ms for all following requests
>
>I will count that as identical. The Snakeviz graphs also look identical
>for both.
>
>Then I added a few range(1, 1000) loops to be printed in Jinja2 and a
>load of simple text:
>
>Local: 9 ms for initial request, 3 ms for all following requests
>Server: 24 ms for initial request, 10 ms for all following requests
>
>Slowly a difference between the two start to form.
>
>I added some more elements to the view and template such as a WTF form,
>but the timings did not went up much. The difference stayed however:
>server being three times slower then local.
>
>I also added a couple of (fast) database queries in the mix, opening
>and closing both cursors and connections per query:
>
>Local: 16 ms for initial request, 11 ms for all following requests
>Server: 51 ms for initial request, 33 ms for all following requests
>
>> The server/os will have to be looked at some point as well. 
>
>This was interesting but maybe less important since this is testing
>pure CPU throughput:, I yanked a primes Python script of the interwebs
>to test the raw processing speed and ran it on both machines.
>
>Local: 3.3879199028 s
>Server: 9.26460504532 s
>
>This is a script without any extras, one Python function tested with
>Timeit and ran directly via the Python CLI. My local machine is three
>times faster than the server.
>
>Installing Sysbench and doing a primers test (20000) outside of Python:
>
>Local: 18.1605 seconds
>Server: 42.1942 seconds
>
>CPU info:
>
>Local: 1 x i74790K (4 Ghz)
>Server: 8 x Xeon E7-4820 v2 (2 Ghz)
>
>> -- unai
>
>Cheers,
>Tim
>_______________________________________________
>Flask mailing list
>Flask at python.org
>https://mail.python.org/mailman/listinfo/flask


From unai at sysbible.org  Thu Jul  7 05:35:12 2016
From: unai at sysbible.org (Unai Rodriguez)
Date: Thu, 07 Jul 2016 17:35:12 +0800
Subject: [Flask] uWSGI and Flask - Slow and varying response times
In-Reply-To: <5AF87A5B-83A1-4A28-A49B-1100670332B9@ziirish.info>
References: <20160706154057.83b8a86b5472ef3ffd232c4b@shisaa.jp>
 <CAMfrFKQk9ad=F0v5OpgJ7tBothfENa8FwnoWXZ6u=DVv8FF4Hw@mail.gmail.com>
 <20160706180252.1ffb7eda068f72ab2121cd39@shisaa.jp>
 <1467798720.439156.658232313.2A5E7454@webmail.messagingengine.com>
 <20160706191625.c6c547c2b3d585fea6bc13da@shisaa.jp>
 <1467800895.445316.658254993.2C4F7D55@webmail.messagingengine.com>
 <20160707124030.68c3f7bf5bfdbb0467100ac4@shisaa.jp>
 <5AF87A5B-83A1-4A28-A49B-1100670332B9@ziirish.info>
Message-ID: <1467884112.618070.659319025.109A7457@webmail.messagingengine.com>

Fantastic collaborative effort!!! :) 

-- unai

On Thu, Jul 7, 2016, at 02:42 PM, Ziirish wrote:
> Hello,
> 
> I think you have your answer right with the last piece of information you
> gave us.
> 
> Your CPU clock is two times faster than your servers one.
> 
> Now your code is probably not multi threaded, it means your server will
> perform at least two times slower than your development machine to
> execute the same piece of code.
> 
> The x3 you are measuring is probably due to some context switch or
> something (your kernel is probably spitting the task in several pieces to
> let other processes perform well so these pieces of processes then need
> to be synced or something)
> 
> In conclusion, the delay you are observing is due to a lower CPU clock on
> the server. So unless you can multi thread your application there's not
> much to do.
> In the other hand, because you have a lot of cores, your server will be
> able to serve (at least) 8 simultaneous clients within the same time
> range.
> 
> You can also have a look at PyPy as already suggested to try to optimize
> the generated "bytecode". It can probably save you a few CPU cycles and
> thus decrease your whole compute time.
> 
> 
> 
> 
> Le 7 juillet 2016 05:40:30 GMT+02:00, Tim van der Linden <tim at shisaa.jp>
> a ?crit :
> >On Wed, 06 Jul 2016 18:28:15 +0800
> >Unai Rodriguez <unai at sysbible.org> wrote:
> >
> >> Hi Tim,
> >
> >Hi Unai
> >
> >> For uwsgi/Python you could do the same, build a simple page and a
> >more
> >> complicated one and compare (no database queries to isolate).
> >
> >First a simple page on uWSGI using Flask and Jinja. The template only
> >consists out of the bare minimals for a HTML page and a single string
> >variable to be printed and the whole application is a single Python
> >file with only one view. The timings are done using the Werkzeug
> >profiler:
> >
> >Local: 7 ms for initial request, 0.5ms for all following requests
> >Server: 160 ms for initial request, 1ms for all following requests
> >
> >I will count that as identical. The Snakeviz graphs also look identical
> >for both.
> >
> >Then I added a few range(1, 1000) loops to be printed in Jinja2 and a
> >load of simple text:
> >
> >Local: 9 ms for initial request, 3 ms for all following requests
> >Server: 24 ms for initial request, 10 ms for all following requests
> >
> >Slowly a difference between the two start to form.
> >
> >I added some more elements to the view and template such as a WTF form,
> >but the timings did not went up much. The difference stayed however:
> >server being three times slower then local.
> >
> >I also added a couple of (fast) database queries in the mix, opening
> >and closing both cursors and connections per query:
> >
> >Local: 16 ms for initial request, 11 ms for all following requests
> >Server: 51 ms for initial request, 33 ms for all following requests
> >
> >> The server/os will have to be looked at some point as well. 
> >
> >This was interesting but maybe less important since this is testing
> >pure CPU throughput:, I yanked a primes Python script of the interwebs
> >to test the raw processing speed and ran it on both machines.
> >
> >Local: 3.3879199028 s
> >Server: 9.26460504532 s
> >
> >This is a script without any extras, one Python function tested with
> >Timeit and ran directly via the Python CLI. My local machine is three
> >times faster than the server.
> >
> >Installing Sysbench and doing a primers test (20000) outside of Python:
> >
> >Local: 18.1605 seconds
> >Server: 42.1942 seconds
> >
> >CPU info:
> >
> >Local: 1 x i74790K (4 Ghz)
> >Server: 8 x Xeon E7-4820 v2 (2 Ghz)
> >
> >> -- unai
> >
> >Cheers,
> >Tim
> >_______________________________________________
> >Flask mailing list
> >Flask at python.org
> >https://mail.python.org/mailman/listinfo/flask
> 

From unai at sysbible.org  Thu Jul  7 07:07:36 2016
From: unai at sysbible.org (Unai Rodriguez)
Date: Thu, 07 Jul 2016 19:07:36 +0800
Subject: [Flask] uWSGI and Flask - Slow and varying response times
In-Reply-To: <1467884112.618070.659319025.109A7457@webmail.messagingengine.com>
References: <20160706154057.83b8a86b5472ef3ffd232c4b@shisaa.jp>
 <CAMfrFKQk9ad=F0v5OpgJ7tBothfENa8FwnoWXZ6u=DVv8FF4Hw@mail.gmail.com>
 <20160706180252.1ffb7eda068f72ab2121cd39@shisaa.jp>
 <1467798720.439156.658232313.2A5E7454@webmail.messagingengine.com>
 <20160706191625.c6c547c2b3d585fea6bc13da@shisaa.jp>
 <1467800895.445316.658254993.2C4F7D55@webmail.messagingengine.com>
 <20160707124030.68c3f7bf5bfdbb0467100ac4@shisaa.jp>
 <5AF87A5B-83A1-4A28-A49B-1100670332B9@ziirish.info>
 <1467884112.618070.659319025.109A7457@webmail.messagingengine.com>
Message-ID: <1467889656.1232213.659390993.60FF2E22@webmail.messagingengine.com>

What happened to the inconsistent timings though?

-- unai

On Thu, Jul 7, 2016, at 05:35 PM, Unai Rodriguez wrote:
> Fantastic collaborative effort!!! :) 
> 
> -- unai
> 
> On Thu, Jul 7, 2016, at 02:42 PM, Ziirish wrote:
> > Hello,
> > 
> > I think you have your answer right with the last piece of information you
> > gave us.
> > 
> > Your CPU clock is two times faster than your servers one.
> > 
> > Now your code is probably not multi threaded, it means your server will
> > perform at least two times slower than your development machine to
> > execute the same piece of code.
> > 
> > The x3 you are measuring is probably due to some context switch or
> > something (your kernel is probably spitting the task in several pieces to
> > let other processes perform well so these pieces of processes then need
> > to be synced or something)
> > 
> > In conclusion, the delay you are observing is due to a lower CPU clock on
> > the server. So unless you can multi thread your application there's not
> > much to do.
> > In the other hand, because you have a lot of cores, your server will be
> > able to serve (at least) 8 simultaneous clients within the same time
> > range.
> > 
> > You can also have a look at PyPy as already suggested to try to optimize
> > the generated "bytecode". It can probably save you a few CPU cycles and
> > thus decrease your whole compute time.
> > 
> > 
> > 
> > 
> > Le 7 juillet 2016 05:40:30 GMT+02:00, Tim van der Linden <tim at shisaa.jp>
> > a ?crit :
> > >On Wed, 06 Jul 2016 18:28:15 +0800
> > >Unai Rodriguez <unai at sysbible.org> wrote:
> > >
> > >> Hi Tim,
> > >
> > >Hi Unai
> > >
> > >> For uwsgi/Python you could do the same, build a simple page and a
> > >more
> > >> complicated one and compare (no database queries to isolate).
> > >
> > >First a simple page on uWSGI using Flask and Jinja. The template only
> > >consists out of the bare minimals for a HTML page and a single string
> > >variable to be printed and the whole application is a single Python
> > >file with only one view. The timings are done using the Werkzeug
> > >profiler:
> > >
> > >Local: 7 ms for initial request, 0.5ms for all following requests
> > >Server: 160 ms for initial request, 1ms for all following requests
> > >
> > >I will count that as identical. The Snakeviz graphs also look identical
> > >for both.
> > >
> > >Then I added a few range(1, 1000) loops to be printed in Jinja2 and a
> > >load of simple text:
> > >
> > >Local: 9 ms for initial request, 3 ms for all following requests
> > >Server: 24 ms for initial request, 10 ms for all following requests
> > >
> > >Slowly a difference between the two start to form.
> > >
> > >I added some more elements to the view and template such as a WTF form,
> > >but the timings did not went up much. The difference stayed however:
> > >server being three times slower then local.
> > >
> > >I also added a couple of (fast) database queries in the mix, opening
> > >and closing both cursors and connections per query:
> > >
> > >Local: 16 ms for initial request, 11 ms for all following requests
> > >Server: 51 ms for initial request, 33 ms for all following requests
> > >
> > >> The server/os will have to be looked at some point as well. 
> > >
> > >This was interesting but maybe less important since this is testing
> > >pure CPU throughput:, I yanked a primes Python script of the interwebs
> > >to test the raw processing speed and ran it on both machines.
> > >
> > >Local: 3.3879199028 s
> > >Server: 9.26460504532 s
> > >
> > >This is a script without any extras, one Python function tested with
> > >Timeit and ran directly via the Python CLI. My local machine is three
> > >times faster than the server.
> > >
> > >Installing Sysbench and doing a primers test (20000) outside of Python:
> > >
> > >Local: 18.1605 seconds
> > >Server: 42.1942 seconds
> > >
> > >CPU info:
> > >
> > >Local: 1 x i74790K (4 Ghz)
> > >Server: 8 x Xeon E7-4820 v2 (2 Ghz)
> > >
> > >> -- unai
> > >
> > >Cheers,
> > >Tim
> > >_______________________________________________
> > >Flask mailing list
> > >Flask at python.org
> > >https://mail.python.org/mailman/listinfo/flask
> > 
> _______________________________________________
> Flask mailing list
> Flask at python.org
> https://mail.python.org/mailman/listinfo/flask

From tim at shisaa.jp  Thu Jul  7 10:33:19 2016
From: tim at shisaa.jp (Tim van der Linden)
Date: Thu, 7 Jul 2016 23:33:19 +0900
Subject: [Flask] uWSGI and Flask - Slow and varying response times
In-Reply-To: <1467884112.618070.659319025.109A7457@webmail.messagingengine.com>
References: <20160706154057.83b8a86b5472ef3ffd232c4b@shisaa.jp>
 <CAMfrFKQk9ad=F0v5OpgJ7tBothfENa8FwnoWXZ6u=DVv8FF4Hw@mail.gmail.com>
 <20160706180252.1ffb7eda068f72ab2121cd39@shisaa.jp>
 <1467798720.439156.658232313.2A5E7454@webmail.messagingengine.com>
 <20160706191625.c6c547c2b3d585fea6bc13da@shisaa.jp>
 <1467800895.445316.658254993.2C4F7D55@webmail.messagingengine.com>
 <20160707124030.68c3f7bf5bfdbb0467100ac4@shisaa.jp>
 <5AF87A5B-83A1-4A28-A49B-1100670332B9@ziirish.info>
 <1467884112.618070.659319025.109A7457@webmail.messagingengine.com>
Message-ID: <20160707233319.3c9d75a55dc453f2388396d3@shisaa.jp>

On Thu, 07 Jul 2016 17:35:12 +0800
Unai Rodriguez <unai at sysbible.org> wrote:

Hi Unai (and everyone else!)

> Fantastic collaborative effort!!! :) 

Indeed. Thanks to all for giving a fresh set of eyes into this issue.
A big factor, as Ziirish pointed out, seems to be the difference in clock speed indeed.

As Unai pointed out it still does not explain the difference in response.
Maybe there is a lot of context switching going on ... ?

Another thing I noticed as well is the fact that on the server the profiler tells me a request is finished in ~700 ms (for example) but the browser has to wait a total of ~ 1100 ms before receiving back the data.

There seems to be something happening between Python being finished within those ~700 ms and uWSGI/Nginx generating a response back to the client (~ 400ms lost somewhere).

Maybe this is the cause for the difference in response rates. Python performance (yet slower slower than local) is stable, the response overhead varies.

I wonder what might be happening in uWSGI/Nginx after Python is finished with its work...

> -- unai

Cheers,
Tim

From aco at miza.org  Thu Jul  7 11:04:17 2016
From: aco at miza.org (Aco Strkalj)
Date: Thu, 7 Jul 2016 10:04:17 -0500
Subject: [Flask] uWSGI and Flask - Slow and varying response times
In-Reply-To: <20160707233319.3c9d75a55dc453f2388396d3@shisaa.jp>
References: <20160706154057.83b8a86b5472ef3ffd232c4b@shisaa.jp>
 <CAMfrFKQk9ad=F0v5OpgJ7tBothfENa8FwnoWXZ6u=DVv8FF4Hw@mail.gmail.com>
 <20160706180252.1ffb7eda068f72ab2121cd39@shisaa.jp>
 <1467798720.439156.658232313.2A5E7454@webmail.messagingengine.com>
 <20160706191625.c6c547c2b3d585fea6bc13da@shisaa.jp>
 <1467800895.445316.658254993.2C4F7D55@webmail.messagingengine.com>
 <20160707124030.68c3f7bf5bfdbb0467100ac4@shisaa.jp>
 <5AF87A5B-83A1-4A28-A49B-1100670332B9@ziirish.info>
 <1467884112.618070.659319025.109A7457@webmail.messagingengine.com>
 <20160707233319.3c9d75a55dc453f2388396d3@shisaa.jp>
Message-ID: <4CBDFB7E-5C48-47D7-9512-185AA116BFB6@miza.org>

If it?s not too much work can you run it though gunicorn?


> On Jul 7, 2016, at 9:33 AM, Tim van der Linden <tim at shisaa.jp> wrote:
> 
> On Thu, 07 Jul 2016 17:35:12 +0800
> Unai Rodriguez <unai at sysbible.org> wrote:
> 
> Hi Unai (and everyone else!)
> 
>> Fantastic collaborative effort!!! :) 
> 
> Indeed. Thanks to all for giving a fresh set of eyes into this issue.
> A big factor, as Ziirish pointed out, seems to be the difference in clock speed indeed.
> 
> As Unai pointed out it still does not explain the difference in response.
> Maybe there is a lot of context switching going on ... ?
> 
> Another thing I noticed as well is the fact that on the server the profiler tells me a request is finished in ~700 ms (for example) but the browser has to wait a total of ~ 1100 ms before receiving back the data.
> 
> There seems to be something happening between Python being finished within those ~700 ms and uWSGI/Nginx generating a response back to the client (~ 400ms lost somewhere).
> 
> Maybe this is the cause for the difference in response rates. Python performance (yet slower slower than local) is stable, the response overhead varies.
> 
> I wonder what might be happening in uWSGI/Nginx after Python is finished with its work...
> 
>> -- unai
> 
> Cheers,
> Tim
> _______________________________________________
> Flask mailing list
> Flask at python.org
> https://mail.python.org/mailman/listinfo/flask


From ziirish at ziirish.info  Thu Jul  7 11:19:25 2016
From: ziirish at ziirish.info (Ziirish)
Date: Thu, 7 Jul 2016 17:19:25 +0200
Subject: [Flask] uWSGI and Flask - Slow and varying response times
In-Reply-To: <20160707233319.3c9d75a55dc453f2388396d3@shisaa.jp>
References: <20160706154057.83b8a86b5472ef3ffd232c4b@shisaa.jp>
 <CAMfrFKQk9ad=F0v5OpgJ7tBothfENa8FwnoWXZ6u=DVv8FF4Hw@mail.gmail.com>
 <20160706180252.1ffb7eda068f72ab2121cd39@shisaa.jp>
 <1467798720.439156.658232313.2A5E7454@webmail.messagingengine.com>
 <20160706191625.c6c547c2b3d585fea6bc13da@shisaa.jp>
 <1467800895.445316.658254993.2C4F7D55@webmail.messagingengine.com>
 <20160707124030.68c3f7bf5bfdbb0467100ac4@shisaa.jp>
 <5AF87A5B-83A1-4A28-A49B-1100670332B9@ziirish.info>
 <1467884112.618070.659319025.109A7457@webmail.messagingengine.com>
 <20160707233319.3c9d75a55dc453f2388396d3@shisaa.jp>
Message-ID: <20160707151925.GN3187@mail.ziirish.info>

What is your uWSGI configuration/starting parameters?

Are they the same on both your development machine and your server?

Could you send a summary of your benchmark so far like:

      | local | server |
test1 | 100ms |  700ms |
test2 |  50ms |  500ms |
...

Because there are now a lot of informations in a lot of emails.

Thanks

* On Thursday, July 07, 2016 at 11:33 PM +0900, Tim van der Linden <tim at shisaa.jp> wrote:
> On Thu, 07 Jul 2016 17:35:12 +0800
> Unai Rodriguez <unai at sysbible.org> wrote:
> 
> Hi Unai (and everyone else!)
> 
> > Fantastic collaborative effort!!! :) 
> 
> Indeed. Thanks to all for giving a fresh set of eyes into this issue.
> A big factor, as Ziirish pointed out, seems to be the difference in clock speed indeed.
> 
> As Unai pointed out it still does not explain the difference in response.
> Maybe there is a lot of context switching going on ... ?
> 
> Another thing I noticed as well is the fact that on the server the profiler tells me a request is finished in ~700 ms (for example) but the browser has to wait a total of ~ 1100 ms before receiving back the data.
> 
> There seems to be something happening between Python being finished within those ~700 ms and uWSGI/Nginx generating a response back to the client (~ 400ms lost somewhere).
> 
> Maybe this is the cause for the difference in response rates. Python performance (yet slower slower than local) is stable, the response overhead varies.
> 
> I wonder what might be happening in uWSGI/Nginx after Python is finished with its work...
> 
> > -- unai
> 
> Cheers,
> Tim

From guymatz at gmail.com  Thu Jul  7 17:27:20 2016
From: guymatz at gmail.com (Guy Matz)
Date: Thu, 7 Jul 2016 17:27:20 -0400
Subject: [Flask] Logging from non-flask objects
Message-ID: <CABnTgtXj5zdVvH+LYqFoROAfrQZBcwz9Vq=8rDLMuQgSXr5CZQ@mail.gmail.com>

Hi!  Can anyone recommend a way to log from objects that are not flask
objects?  I.e. if my flask code creates a new object that does some
processing, etc., how can I get access to the logger from that object?  Do
I need to pass in the logger to the new object?  Or is there a better way?

Thanks!
Guy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/flask/attachments/20160707/d8a832fa/attachment.html>

From ziirish at ziirish.info  Fri Jul  8 05:49:21 2016
From: ziirish at ziirish.info (Ziirish)
Date: Fri, 8 Jul 2016 11:49:21 +0200
Subject: [Flask] Logging from non-flask objects
In-Reply-To: <CABnTgtXj5zdVvH+LYqFoROAfrQZBcwz9Vq=8rDLMuQgSXr5CZQ@mail.gmail.com>
References: <CABnTgtXj5zdVvH+LYqFoROAfrQZBcwz9Vq=8rDLMuQgSXr5CZQ@mail.gmail.com>
Message-ID: <20160708094921.GO3187@mail.ziirish.info>

Hello,

As usual, there is more than one way to do it.
I don't know what is the best way to do it though so here are a few examples.

1. Use the current_app object. In the module defining your object you can do
this:

from flask import current_app

class MyObject:
    def foo(self):
        current_app.logger.info('bar')


2. Like you suggested, you could pass to your object the flask app (that's
basically what most of the flask extensions do):

class MyObject:
    def __init__(self, app=None):
        self.app = app

    def init_app(self, app):
        self.app = app

    def foo(self):
        self.app.logger.info('bar')


There are probably other ways but those are the two most common IMHO.


* On Thursday, July 07, 2016 at 05:27 PM -0400, Guy Matz <guymatz at gmail.com> wrote:
> Hi!  Can anyone recommend a way to log from objects that are not flask
> objects?  I.e. if my flask code creates a new object that does some
> processing, etc., how can I get access to the logger from that object?  Do
> I need to pass in the logger to the new object?  Or is there a better way?
> 
> Thanks!
> Guy

From scott.werner.vt at gmail.com  Fri Jul  8 09:35:50 2016
From: scott.werner.vt at gmail.com (Scott Werner)
Date: Fri, 8 Jul 2016 09:35:50 -0400
Subject: [Flask] Logging from non-flask objects
In-Reply-To: <20160708094921.GO3187@mail.ziirish.info>
References: <CABnTgtXj5zdVvH+LYqFoROAfrQZBcwz9Vq=8rDLMuQgSXr5CZQ@mail.gmail.com>
 <20160708094921.GO3187@mail.ziirish.info>
Message-ID: <CAMtsL6hrmijBCkReOVtWnUT+QpOTe0JVxYaSV_WrqNHyHJyN=Q@mail.gmail.com>

Expanding on Ziirish's reply, I do the following:

import logging


class MyObject(object):

    def __init__(self, logger=None):

        self.logger = logger or logging.getLogger(__name__)

    def hello(self):

        self.logger.info('hi')


my = MyObject(logger=current_app.logger)


On Fri, Jul 8, 2016 at 5:49 AM, Ziirish <ziirish at ziirish.info> wrote:

> Hello,
>
> As usual, there is more than one way to do it.
> I don't know what is the best way to do it though so here are a few
> examples.
>
> 1. Use the current_app object. In the module defining your object you can
> do
> this:
>
> from flask import current_app
>
> class MyObject:
>     def foo(self):
>         current_app.logger.info('bar')
>
>
> 2. Like you suggested, you could pass to your object the flask app (that's
> basically what most of the flask extensions do):
>
> class MyObject:
>     def __init__(self, app=None):
>         self.app = app
>
>     def init_app(self, app):
>         self.app = app
>
>     def foo(self):
>         self.app.logger.info('bar')
>
>
> There are probably other ways but those are the two most common IMHO.
>
>
> * On Thursday, July 07, 2016 at 05:27 PM -0400, Guy Matz <
> guymatz at gmail.com> wrote:
> > Hi!  Can anyone recommend a way to log from objects that are not flask
> > objects?  I.e. if my flask code creates a new object that does some
> > processing, etc., how can I get access to the logger from that object?
> Do
> > I need to pass in the logger to the new object?  Or is there a better
> way?
> >
> > Thanks!
> > Guy
> _______________________________________________
> Flask mailing list
> Flask at python.org
> https://mail.python.org/mailman/listinfo/flask
>



-- 
Scott Werner
scott.werner.vt at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/flask/attachments/20160708/4e0f541f/attachment.html>

From guymatz at gmail.com  Fri Jul  8 11:22:19 2016
From: guymatz at gmail.com (Guy Matz)
Date: Fri, 8 Jul 2016 11:22:19 -0400
Subject: [Flask] Logging from non-flask objects
In-Reply-To: <20160708094921.GO3187@mail.ziirish.info>
References: <CABnTgtXj5zdVvH+LYqFoROAfrQZBcwz9Vq=8rDLMuQgSXr5CZQ@mail.gmail.com>
 <20160708094921.GO3187@mail.ziirish.info>
Message-ID: <CABnTgtU0LJ6_yCH3goWw1VfvPe0K9c4+1z1g_5eGqyRgipYjSw@mail.gmail.com>

Thanks!  So I tried something like this:

from flask import current_app
class MyObject:
  def __init__(self):
    self.logger = current_app.logger
    self.logger.info('sdfdf")

And now I'm getting something like this:
ERROR/MainProcess] Task
main.async_process_on_startup[27608e0c-c9a5-4ef0-ac47-244759acd465] raised
unexpected: RuntimeError('Working outside of application context.\n\nThis
typically means that you attempted to use functionality that needed\nto
interface with the current application object in a way.  To solve\nthis set
up an application context with app.app_context().  See the\ndocumentation
for more information.',)

Any idea what I did wrong?

Thanks again!!

On Fri, Jul 8, 2016 at 5:49 AM, Ziirish <ziirish at ziirish.info> wrote:

> Hello,
>
> As usual, there is more than one way to do it.
> I don't know what is the best way to do it though so here are a few
> examples.
>
> 1. Use the current_app object. In the module defining your object you can
> do
> this:
>
> from flask import current_app
>
> class MyObject:
>     def foo(self):
>         current_app.logger.info('bar')
>
>
> 2. Like you suggested, you could pass to your object the flask app (that's
> basically what most of the flask extensions do):
>
> class MyObject:
>     def __init__(self, app=None):
>         self.app = app
>
>     def init_app(self, app):
>         self.app = app
>
>     def foo(self):
>         self.app.logger.info('bar')
>
>
> There are probably other ways but those are the two most common IMHO.
>
>
> * On Thursday, July 07, 2016 at 05:27 PM -0400, Guy Matz <
> guymatz at gmail.com> wrote:
> > Hi!  Can anyone recommend a way to log from objects that are not flask
> > objects?  I.e. if my flask code creates a new object that does some
> > processing, etc., how can I get access to the logger from that object?
> Do
> > I need to pass in the logger to the new object?  Or is there a better
> way?
> >
> > Thanks!
> > Guy
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/flask/attachments/20160708/9112e54f/attachment-0001.html>

From ziirish at ziirish.info  Fri Jul  8 11:39:41 2016
From: ziirish at ziirish.info (Ziirish)
Date: Fri, 8 Jul 2016 17:39:41 +0200
Subject: [Flask] Logging from non-flask objects
In-Reply-To: <CABnTgtU0LJ6_yCH3goWw1VfvPe0K9c4+1z1g_5eGqyRgipYjSw@mail.gmail.com>
References: <CABnTgtXj5zdVvH+LYqFoROAfrQZBcwz9Vq=8rDLMuQgSXr5CZQ@mail.gmail.com>
 <20160708094921.GO3187@mail.ziirish.info>
 <CABnTgtU0LJ6_yCH3goWw1VfvPe0K9c4+1z1g_5eGqyRgipYjSw@mail.gmail.com>
Message-ID: <20160708153940.GR3187@mail.ziirish.info>

Well, the error speaks for itself.

How is your object instantiated/called in your application?



* On Friday, July 08, 2016 at 11:22 AM -0400, Guy Matz <guymatz at gmail.com> wrote:
> Thanks!  So I tried something like this:
> 
> from flask import current_app
> class MyObject:
>   def __init__(self):
>     self.logger = current_app.logger
>     self.logger.info('sdfdf")
> 
> And now I'm getting something like this:
> ERROR/MainProcess] Task
> main.async_process_on_startup[27608e0c-c9a5-4ef0-ac47-244759acd465] raised
> unexpected: RuntimeError('Working outside of application context.\n\nThis
> typically means that you attempted to use functionality that needed\nto
> interface with the current application object in a way.  To solve\nthis set
> up an application context with app.app_context().  See the\ndocumentation
> for more information.',)
> 
> Any idea what I did wrong?
> 
> Thanks again!!
> 
> On Fri, Jul 8, 2016 at 5:49 AM, Ziirish <ziirish at ziirish.info> wrote:
> 
> > Hello,
> >
> > As usual, there is more than one way to do it.
> > I don't know what is the best way to do it though so here are a few
> > examples.
> >
> > 1. Use the current_app object. In the module defining your object you can
> > do
> > this:
> >
> > from flask import current_app
> >
> > class MyObject:
> >     def foo(self):
> >         current_app.logger.info('bar')
> >
> >
> > 2. Like you suggested, you could pass to your object the flask app (that's
> > basically what most of the flask extensions do):
> >
> > class MyObject:
> >     def __init__(self, app=None):
> >         self.app = app
> >
> >     def init_app(self, app):
> >         self.app = app
> >
> >     def foo(self):
> >         self.app.logger.info('bar')
> >
> >
> > There are probably other ways but those are the two most common IMHO.
> >
> >
> > * On Thursday, July 07, 2016 at 05:27 PM -0400, Guy Matz <
> > guymatz at gmail.com> wrote:
> > > Hi!  Can anyone recommend a way to log from objects that are not flask
> > > objects?  I.e. if my flask code creates a new object that does some
> > > processing, etc., how can I get access to the logger from that object?
> > Do
> > > I need to pass in the logger to the new object?  Or is there a better
> > way?
> > >
> > > Thanks!
> > > Guy
> >

From guymatz at gmail.com  Mon Jul 11 10:51:46 2016
From: guymatz at gmail.com (Guy Matz)
Date: Mon, 11 Jul 2016 10:51:46 -0400
Subject: [Flask] Logging from non-flask objects
In-Reply-To: <20160708153940.GR3187@mail.ziirish.info>
References: <CABnTgtXj5zdVvH+LYqFoROAfrQZBcwz9Vq=8rDLMuQgSXr5CZQ@mail.gmail.com>
 <20160708094921.GO3187@mail.ziirish.info>
 <CABnTgtU0LJ6_yCH3goWw1VfvPe0K9c4+1z1g_5eGqyRgipYjSw@mail.gmail.com>
 <20160708153940.GR3187@mail.ziirish.info>
Message-ID: <CABnTgtW2PtHuqVkr=Mk8cPK_4LN6+VSOvXE5OVr89jwrFOcU+A@mail.gmail.com>

Silly me.  I am creating a flask app that uses celery, so I actually needed
the logger from celery in this case.  Sorry for the bother!

On Fri, Jul 8, 2016 at 11:39 AM, Ziirish <ziirish at ziirish.info> wrote:

> Well, the error speaks for itself.
>
> How is your object instantiated/called in your application?
>
>
>
> * On Friday, July 08, 2016 at 11:22 AM -0400, Guy Matz <guymatz at gmail.com>
> wrote:
> > Thanks!  So I tried something like this:
> >
> > from flask import current_app
> > class MyObject:
> >   def __init__(self):
> >     self.logger = current_app.logger
> >     self.logger.info('sdfdf")
> >
> > And now I'm getting something like this:
> > ERROR/MainProcess] Task
> > main.async_process_on_startup[27608e0c-c9a5-4ef0-ac47-244759acd465]
> raised
> > unexpected: RuntimeError('Working outside of application context.\n\nThis
> > typically means that you attempted to use functionality that needed\nto
> > interface with the current application object in a way.  To solve\nthis
> set
> > up an application context with app.app_context().  See the\ndocumentation
> > for more information.',)
> >
> > Any idea what I did wrong?
> >
> > Thanks again!!
> >
> > On Fri, Jul 8, 2016 at 5:49 AM, Ziirish <ziirish at ziirish.info> wrote:
> >
> > > Hello,
> > >
> > > As usual, there is more than one way to do it.
> > > I don't know what is the best way to do it though so here are a few
> > > examples.
> > >
> > > 1. Use the current_app object. In the module defining your object you
> can
> > > do
> > > this:
> > >
> > > from flask import current_app
> > >
> > > class MyObject:
> > >     def foo(self):
> > >         current_app.logger.info('bar')
> > >
> > >
> > > 2. Like you suggested, you could pass to your object the flask app
> (that's
> > > basically what most of the flask extensions do):
> > >
> > > class MyObject:
> > >     def __init__(self, app=None):
> > >         self.app = app
> > >
> > >     def init_app(self, app):
> > >         self.app = app
> > >
> > >     def foo(self):
> > >         self.app.logger.info('bar')
> > >
> > >
> > > There are probably other ways but those are the two most common IMHO.
> > >
> > >
> > > * On Thursday, July 07, 2016 at 05:27 PM -0400, Guy Matz <
> > > guymatz at gmail.com> wrote:
> > > > Hi!  Can anyone recommend a way to log from objects that are not
> flask
> > > > objects?  I.e. if my flask code creates a new object that does some
> > > > processing, etc., how can I get access to the logger from that
> object?
> > > Do
> > > > I need to pass in the logger to the new object?  Or is there a better
> > > way?
> > > >
> > > > Thanks!
> > > > Guy
> > >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/flask/attachments/20160711/75856e03/attachment.html>

From John.Robson at usp.br  Tue Jul 12 19:04:21 2016
From: John.Robson at usp.br (John Robson)
Date: Tue, 12 Jul 2016 19:04:21 -0400
Subject: [Flask] Flask Structure
Message-ID: <nm3t1m$97b$1@ger.gmane.org>

I created an structure that is working, but I wonder if this is correct 
or not: (summary)

~/MyApp
     |-- run.py
     |__ /app
          |-- __init__.py
          |-- file1.py
          |-- file2.py

~/MyApp/run.py
     from app import app
     app.run()

~/MyApp/app/__init__.py
     from flask import Flask, render_template, session, Blueprint, 
current_app
     app = Flask(__name__)
     sess = Session()
     sess.init_app(app)
     db = SQLAlchemy()
     db.init_app(app)
     db.app = app

~/MyApp/app/file1.py
     from app import app, db, render_template, session
     db.session.add(......)
     db.session.commit() # Works !!!

     @app.route('/test/', methods=["GET","POST"])
     def test():
         return render_template('works.html')

~/MyApp/app/file2.py (same as file1.py and everything works fine)

In file1.py and file2.py I don't import Flask objects, I import the 
objects that I Imported (render_template, session) or Created (app, db, 
sess) in __init__.py

My question is: "from app import app, db, sess" allows me to use all 
objects from "__init__.py" in every other file.

So, why use Blueprint? This approach is correct? Importing "app", "db", 
"sess", "render_template", etc from __init__.py ?

Thank you,
John
https://bpaste.net/show/9f7b889e9450


From decipher.hk at gmail.com  Tue Jul 12 22:13:44 2016
From: decipher.hk at gmail.com (Rodrigo =?ISO-8859-1?Q?Ram=EDrez?= Norambuena)
Date: Tue, 12 Jul 2016 22:13:44 -0400
Subject: [Flask] Handler for reloading
Message-ID: <1468376024.3232.31.camel@gmail.com>

Hello everyone,


I tried to find some way for handling a reload event for an app in
Flask.

someone can give me a tips?, I read something about signals, i think
could be help to do this.

1: http://flask.pocoo.org/docs/0.11/signals/

Regards,

-- Rodrigo Ram?rez Norambuena
http://www.rodrigoramirez.com


From John.Robson at usp.br  Thu Jul 14 01:51:44 2016
From: John.Robson at usp.br (John Robson)
Date: Thu, 14 Jul 2016 01:51:44 -0400
Subject: [Flask] Sharing a variable to all HTML pages
Message-ID: <nm799h$o7t$1@ger.gmane.org>

Hi everyone,

I would like to pass several variables to all HTML pages, how can I send 
a variable to all HTML pages, without pass it all the time through 
"*render_template*"? eg: sending "time" to all HTML pages

/*site.html*/*
**Last Update: {{ update }}**
**
*/*app.py*/*
**  update = '2016-07-14 01:37:56'**
**  return render_template('site.html', _update=update_)*

But how to pass this value every time to all HTML pages automatically? Just:
* return render_template('site.html')*

And the value 'update' will be there.

Thank you for helping,
John

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/flask/attachments/20160714/3b2b57a6/attachment.html>

From graham.cleven at gmail.com  Thu Jul 14 02:11:48 2016
From: graham.cleven at gmail.com (Graham Cleven)
Date: Thu, 14 Jul 2016 02:11:48 -0400
Subject: [Flask] Sharing a variable to all HTML pages
In-Reply-To: <nm799h$o7t$1@ger.gmane.org>
References: <nm799h$o7t$1@ger.gmane.org>
Message-ID: <CAJPgfo-G-sboDvPPa_w27=pgTVi=bO=_q_1WN_ysv1WasfnQ8A@mail.gmail.com>

Just a thought, the g variable is available to all render templates. Maybe
try saving your vars to that?
-Graham

On Thu, Jul 14, 2016 at 1:51 AM, John Robson <John.Robson at usp.br> wrote:

> Hi everyone,
>
> I would like to pass several variables to all HTML pages, how can I send a
> variable to all HTML pages, without pass it all the time through "
> *render_template*"? eg: sending "time" to all HTML pages
>
> *site.html*
> *Last Update: {{ update }}*
>
> *app.py*
> *  update = '2016-07-14 01:37:56'*
> *  return render_template('site.html', update=update)*
>
> But how to pass this value every time to all HTML pages automatically?
> Just:
> * return render_template('site.html')*
>
> And the value 'update' will be there.
>
> Thank you for helping,
> John
>
> _______________________________________________
> Flask mailing list
> Flask at python.org
> https://mail.python.org/mailman/listinfo/flask
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/flask/attachments/20160714/fbad9624/attachment.html>

From mark at pors.net  Thu Jul 14 02:31:25 2016
From: mark at pors.net (Mark Pors)
Date: Thu, 14 Jul 2016 09:31:25 +0300
Subject: [Flask] Sharing a variable to all HTML pages
In-Reply-To: <nm799h$o7t$1@ger.gmane.org>
References: <nm799h$o7t$1@ger.gmane.org>
Message-ID: <CA+9AWjFOcX1OxGPLmAR73SPfS0pW3UVAqJfW_His0rowxia9mA@mail.gmail.com>

Have a look at context processors. You can inject variables or functions to
templates on the app or blueprint level.

See here at the bottom http://flask.pocoo.org/docs/0.11/templating/

Mark


On Thursday, July 14, 2016, John Robson <John.Robson at usp.br> wrote:

> Hi everyone,
>
> I would like to pass several variables to all HTML pages, how can I send a
> variable to all HTML pages, without pass it all the time through "
> *render_template*"? eg: sending "time" to all HTML pages
>
> *site.html*
> *Last Update: {{ update }}*
>
> *app.py*
> *  update = '2016-07-14 01:37:56'*
> *  return render_template('site.html', update=update)*
>
> But how to pass this value every time to all HTML pages automatically?
> Just:
> * return render_template('site.html')*
>
> And the value 'update' will be there.
>
> Thank you for helping,
> John
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/flask/attachments/20160714/3d9d953e/attachment.html>

From jani.sumak at gmail.com  Thu Jul 14 06:13:57 2016
From: jani.sumak at gmail.com (=?UTF-8?Q?Jani_=C5=A0umak?=)
Date: Thu, 14 Jul 2016 10:13:57 +0000
Subject: [Flask] Flask Structure
In-Reply-To: <nm3t1m$97b$1@ger.gmane.org>
References: <nm3t1m$97b$1@ger.gmane.org>
Message-ID: <CAHpkYizCada1QeFZ4NDmd6WEMiya52p7wpZiE6X0heHNpAEHDw@mail.gmail.com>

Hi John,

great to have you here! IMHO there is no "correct" Flask structure. You
could put it all in one file and it would still be good Flask app. The only
thing that caught my attention is your run.py, I'd recommend that you
replace app.run() with

if __name__ == "__main__":
    app.run()

so you could run your app with a WSGI server, like Gunicorn or uWSGI. More
on this here
<http://stackoverflow.com/questions/419163/what-does-if-name-main-do>.

But to address your question about structure: Miguel Grinberg
<http://blog.miguelgrinberg.com/index> had a great talk/workshop at PyCon
2016 titled "Flask at Scale" <https://www.youtube.com/watch?v=tdIIJuPh3SI>.
He starts his Flask app with just one file and gradually breaks it up into
modules. The talk is a bit long, but worth your while, because he explains
why breaking up your app into modules and using blueprints is a good thing.
If this talk is too long, I'd highly recommend his book *Flask Web
Development
<http://shop.oreilly.com/product/0636920031116.do?cmp=af-webplatform-books-videos-product_cj_9781449372620_%25zp>
*or some if his previous talks/tutorials
<http://blog.miguelgrinberg.com/post/flask-at-scale-tutorial-at-pycon-2016-in-portland>

*. *
That being said*, *you are free to chose and advocate a different
structure. If you look at some flask cookiecutter repos on Github, e.g.
cookiecutter-flask <https://github.com/sloria/cookiecutter-flask>, you
might see a different app layout, that still adheres to best practices.

Hope this helps and sorry for the late response.

Take care, Jani

On Wed, Jul 13, 2016 at 2:15 AM John Robson <John.Robson at usp.br> wrote:

> I created an structure that is working, but I wonder if this is correct
> or not: (summary)
>
> ~/MyApp
>      |-- run.py
>      |__ /app
>           |-- __init__.py
>           |-- file1.py
>           |-- file2.py
>
> ~/MyApp/run.py
>      from app import app
>      app.run()
>
> ~/MyApp/app/__init__.py
>      from flask import Flask, render_template, session, Blueprint,
> current_app
>      app = Flask(__name__)
>      sess = Session()
>      sess.init_app(app)
>      db = SQLAlchemy()
>      db.init_app(app)
>      db.app = app
>
> ~/MyApp/app/file1.py
>      from app import app, db, render_template, session
>      db.session.add(......)
>      db.session.commit() # Works !!!
>
>      @app.route('/test/', methods=["GET","POST"])
>      def test():
>          return render_template('works.html')
>
> ~/MyApp/app/file2.py (same as file1.py and everything works fine)
>
> In file1.py and file2.py I don't import Flask objects, I import the
> objects that I Imported (render_template, session) or Created (app, db,
> sess) in __init__.py
>
> My question is: "from app import app, db, sess" allows me to use all
> objects from "__init__.py" in every other file.
>
> So, why use Blueprint? This approach is correct? Importing "app", "db",
> "sess", "render_template", etc from __init__.py ?
>
> Thank you,
> John
> https://bpaste.net/show/9f7b889e9450
>
> _______________________________________________
> Flask mailing list
> Flask at python.org
> https://mail.python.org/mailman/listinfo/flask
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/flask/attachments/20160714/9b091898/attachment-0001.html>

From John.Robson at usp.br  Thu Jul 21 19:07:56 2016
From: John.Robson at usp.br (John Robson)
Date: Thu, 21 Jul 2016 19:07:56 -0400
Subject: [Flask] Sharing a variable to all HTML pages
In-Reply-To: <CA+9AWjFOcX1OxGPLmAR73SPfS0pW3UVAqJfW_His0rowxia9mA@mail.gmail.com>
References: <nm799h$o7t$1@ger.gmane.org>
 <CA+9AWjFOcX1OxGPLmAR73SPfS0pW3UVAqJfW_His0rowxia9mA@mail.gmail.com>
Message-ID: <3e09b201-1b09-e2fc-a5f8-8382f4494258@usp.br>

Thank you everyone I used "context processors" and works perfectly.

On 07/14/2016 02:31 AM, Mark Pors wrote:
> Have a look at context processors. You can inject variables or functions
> to templates on the app or blueprint level.
>
> See here at the bottom http://flask.pocoo.org/docs/0.11/templating/
>
> Mark
>
>
> On Thursday, July 14, 2016, John Robson
> <John.Robson at usp.br
> <mailto:John.Robson at usp.br>> wrote:
>
>     Hi everyone,
>
>     I would like to pass several variables to all HTML pages, how can I
>     send a variable to all HTML pages, without pass it all the time
>     through "*render_template*"? eg: sending "time" to all HTML pages
>
>     /*site.html*/*
>     **Last Update: {{ update }}**
>     **
>     */*app.py*/*
>     **  update = '2016-07-14 01:37:56'**
>     **  return render_template('site.html', _update=update_)*
>
>     But how to pass this value every time to all HTML pages
>     automatically? Just:
>     * return render_template('site.html')*
>
>     And the value 'update' will be there.
>
>     Thank you for helping,
>     John
>
>
>
> _______________________________________________
> Flask mailing list
> Flask at python.org
> https://mail.python.org/mailman/listinfo/flask
>


From John.Robson at usp.br  Thu Jul 21 20:14:35 2016
From: John.Robson at usp.br (John Robson)
Date: Thu, 21 Jul 2016 20:14:35 -0400
Subject: [Flask] SQLAlchemy - One Class - Several Tables
Message-ID: <nmrohb$ej5$1@ger.gmane.org>

Hi everyone,

I want to store the minute price of all 500 stocks from S&P 500.  This 
means more than 500 million of rows + almost 200K updates per day + 
(sometimes) past values must be updated (to adjust ex-dividend, splits).

Instead of creating a BIG table, I would like to create 500 tables (one 
for each stock). They have the same structure/columns (open, high, low, 
close, volume).

There is some way of creating only one Class to map all those 500 
tables, or I need to create 500 classes?

* I found this code of a Dynamically table: 
https://gist.github.com/munro/3686e3b060c2cd7959350ea8bf77ff2c

Is this the best approach for CRUDing this 500 tables and still using 
all the SQLAlchemy features?

* I also found about "Single Table Inheritance": 
http://docs.sqlalchemy.org/en/latest/orm/inheritance.html#single-table-inheritance

So, I wonder what is the best approach to create and manage this DB with 
500 tables.

Thank you,
John


From and.damore at gmail.com  Fri Jul 22 01:57:48 2016
From: and.damore at gmail.com (Andrea D'Amore)
Date: Fri, 22 Jul 2016 07:57:48 +0200
Subject: [Flask] SQLAlchemy - One Class - Several Tables
In-Reply-To: <nmrohb$ej5$1@ger.gmane.org>
References: <nmrohb$ej5$1@ger.gmane.org>
Message-ID: <CAMonFrTrM0ZG6e95+j8FxVeF0UqprZOOeHOfFCzR5Ao_=_nnhQ@mail.gmail.com>

On 22 July 2016 at 02:14, John Robson <John.Robson at usp.br> wrote:
> I want to store the minute price of all 500 stocks from S&P 500.  This means
> more than 500 million of rows + almost 200K updates per day + (sometimes)
> past values must be updated (to adjust ex-dividend, splits).

> Instead of creating a BIG table, I would like to create 500 tables (one for
> each stock). They have the same structure/columns (open, high, low, close,
> volume).

Why? Did you hit some limit or performance issue already or is it a
case of early optimization?

Even sqlite3 should be able to handle that.

-- 
Andrea

From John.Robson at usp.br  Fri Jul 22 04:26:23 2016
From: John.Robson at usp.br (John Robson)
Date: Fri, 22 Jul 2016 04:26:23 -0400
Subject: [Flask] SQLAlchemy - One Class - Several Tables
In-Reply-To: <CAMonFrTrM0ZG6e95+j8FxVeF0UqprZOOeHOfFCzR5Ao_=_nnhQ@mail.gmail.com>
References: <nmrohb$ej5$1@ger.gmane.org>
 <CAMonFrTrM0ZG6e95+j8FxVeF0UqprZOOeHOfFCzR5Ao_=_nnhQ@mail.gmail.com>
Message-ID: <c01c7576-3985-78ce-a935-407c84460e2c@usp.br>

Is not "early optimization", I'm more comfortable with multiple tables, 
furthermore, I have plans to expand to thousands of stocks and 
high-frequency operations with more than 10K events per second (for some 
stocks). Maybe store/read hundreds of thousands of rows per second.

But my question is just, what's is the best approach to do CRUD with 
multiple (dynamic) tables. (one class to map multiple tables)

Thank you for helping.
John

On 07/22/2016 01:57 AM, Andrea D'Amore wrote:
> On 22 July 2016 at 02:14, John Robson <John.Robson at usp.br> wrote:
>> I want to store the minute price of all 500 stocks from S&P 500.  This means
>> more than 500 million of rows + almost 200K updates per day + (sometimes)
>> past values must be updated (to adjust ex-dividend, splits).
>
>> Instead of creating a BIG table, I would like to create 500 tables (one for
>> each stock). They have the same structure/columns (open, high, low, close,
>> volume).
>
> Why? Did you hit some limit or performance issue already or is it a
> case of early optimization?
>
> Even sqlite3 should be able to handle that.
>


From giannis.pin at gmail.com  Fri Jul 22 05:05:23 2016
From: giannis.pin at gmail.com (Ioannis Pinakoulakis)
Date: Fri, 22 Jul 2016 11:05:23 +0200
Subject: [Flask] SQLAlchemy - One Class - Several Tables
In-Reply-To: <c01c7576-3985-78ce-a935-407c84460e2c@usp.br>
References: <nmrohb$ej5$1@ger.gmane.org>
 <CAMonFrTrM0ZG6e95+j8FxVeF0UqprZOOeHOfFCzR5Ao_=_nnhQ@mail.gmail.com>
 <c01c7576-3985-78ce-a935-407c84460e2c@usp.br>
Message-ID: <77C8DC0A-5696-489D-89AB-C573F9221FF5@gmail.com>


If you plan for huge load why don't you consider some nosql database?

Here you can find most of them. 

http://nosql-database.org/

22 juli 2016, 10:26, ?/? John Robson <John.Robson at usp.br> ??????:

> Is not "early optimization", I'm more comfortable with multiple tables, furthermore, I have plans to expand to thousands of stocks and high-frequency operations with more than 10K events per second (for some stocks). Maybe store/read hundreds of thousands of rows per second.
> 
> But my question is just, what's is the best approach to do CRUD with multiple (dynamic) tables. (one class to map multiple tables)
> 
> Thank you for helping.
> John
> 
>> On 07/22/2016 01:57 AM, Andrea D'Amore wrote:
>>> On 22 July 2016 at 02:14, John Robson <John.Robson at usp.br> wrote:
>>> I want to store the minute price of all 500 stocks from S&P 500.  This means
>>> more than 500 million of rows + almost 200K updates per day + (sometimes)
>>> past values must be updated (to adjust ex-dividend, splits).
>> 
>>> Instead of creating a BIG table, I would like to create 500 tables (one for
>>> each stock). They have the same structure/columns (open, high, low, close,
>>> volume).
>> 
>> Why? Did you hit some limit or performance issue already or is it a
>> case of early optimization?
>> 
>> Even sqlite3 should be able to handle that.
> 
> _______________________________________________
> Flask mailing list
> Flask at python.org
> https://mail.python.org/mailman/listinfo/flask
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/flask/attachments/20160722/ecd60e39/attachment.html>

From Stephen.Mazzei at asrcfederal.com  Fri Jul 22 09:25:20 2016
From: Stephen.Mazzei at asrcfederal.com (Mazzei, Stephen Andrew)
Date: Fri, 22 Jul 2016 13:25:20 +0000
Subject: [Flask] Saving file to desktop
Message-ID: <FA1CCD1F7381504ABB7081B0462B9DDC01E2BB47@AFSY-MXDBP2.asrcfh.local>

Good morning,

Right now I am currently working on a dataframe project. Part of this project is to save off the results to an excel document. My current code is

        bookname = request.form['bookname']
        sheetname = request.form['sheetname']
        writer = pd.ExcelWriter(bookname + '.xlsx')
        validation_df.to_excel(writer,sheetname)
        writer.save()


This is generating the correct report, but storing it local to machine. Can someone help guide me in the correct direction to either directly save the file to the user desktop instead of the local flask machine?

Thank you


---
Stephen A. Mazzei
Systems Administrator | AFDS, ASRC Federal Data Solutions - P&G HPC Account | 513-634-9965


________________________________

The preceding message (including attachments) is covered by the Electronic Communication Privacy Act, 18 U.S.C. sections 2510-2512, is intended only for the person or entity to which it is addressed, and may contain information that is confidential, protected by attorney-client or other privilege, or otherwise protected from disclosure by law. If you are not the intended recipient, you are hereby notified that any retention, dissemination, distribution, or copying of this communication is strictly prohibited. Please reply to the sender that you have received the message in error and destroy the original message and all copies.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/flask/attachments/20160722/17995d04/attachment-0001.html>

From ziirish at ziirish.info  Fri Jul 22 09:40:10 2016
From: ziirish at ziirish.info (Ziirish)
Date: Fri, 22 Jul 2016 15:40:10 +0200
Subject: [Flask] Saving file to desktop
In-Reply-To: <FA1CCD1F7381504ABB7081B0462B9DDC01E2BB47@AFSY-MXDBP2.asrcfh.local>
References: <FA1CCD1F7381504ABB7081B0462B9DDC01E2BB47@AFSY-MXDBP2.asrcfh.local>
Message-ID: <20160722134010.GD5365@mail.ziirish.info>

You probably just need to add:

    return send_file(bookname + '.xlsx', as_attachement=True)

at the end of your function.

Of course you need to import the function (from flask import send_file)


This will prompt the user to download/save the generated xlsx on his local
machine.


* On Friday, July 22, 2016 at 01:25 PM +0000, Mazzei, Stephen Andrew <Stephen.Mazzei at asrcfederal.com> wrote:
> Good morning,
> 
> Right now I am currently working on a dataframe project. Part of this project is to save off the results to an excel document. My current code is
> 
>         bookname = request.form['bookname']
>         sheetname = request.form['sheetname']
>         writer = pd.ExcelWriter(bookname + '.xlsx')
>         validation_df.to_excel(writer,sheetname)
>         writer.save()
> 
> 
> This is generating the correct report, but storing it local to machine. Can someone help guide me in the correct direction to either directly save the file to the user desktop instead of the local flask machine?
> 
> Thank you
> 
> 
> ---
> Stephen A. Mazzei
> Systems Administrator | AFDS, ASRC Federal Data Solutions - P&G HPC Account | 513-634-9965
> 
> 
> ________________________________
> 
> The preceding message (including attachments) is covered by the Electronic Communication Privacy Act, 18 U.S.C. sections 2510-2512, is intended only for the person or entity to which it is addressed, and may contain information that is confidential, protected by attorney-client or other privilege, or otherwise protected from disclosure by law. If you are not the intended recipient, you are hereby notified that any retention, dissemination, distribution, or copying of this communication is strictly prohibited. Please reply to the sender that you have received the message in error and destroy the original message and all copies.

From John.Robson at usp.br  Fri Jul 22 11:27:08 2016
From: John.Robson at usp.br (John Robson)
Date: Fri, 22 Jul 2016 11:27:08 -0400
Subject: [Flask] SQLAlchemy - One Class - Several Tables
In-Reply-To: <77C8DC0A-5696-489D-89AB-C573F9221FF5@gmail.com>
References: <nmrohb$ej5$1@ger.gmane.org>
 <CAMonFrTrM0ZG6e95+j8FxVeF0UqprZOOeHOfFCzR5Ao_=_nnhQ@mail.gmail.com>
 <c01c7576-3985-78ce-a935-407c84460e2c@usp.br>
 <77C8DC0A-5696-489D-89AB-C573F9221FF5@gmail.com>
Message-ID: <5e2c29e7-8022-0086-2b20-4446b1144a95@usp.br>

Thank you for your excellent suggestion. This DB is for study purposes 
and PostgreSQL has some No-SQL features.

I wonder if this what is the best approach for this "dynamic" tables:

https://gist.github.com/munro/3686e3b060c2cd7959350ea8bf77ff2c

or

http://docs.sqlalchemy.org/en/latest/orm/inheritance.html#single-table-inheritance

or another thing...

On 07/22/2016 05:05 AM, Ioannis Pinakoulakis wrote:
>
> If you plan for huge load why don't you consider some nosql database?
>
> Here you can find most of them.
>
> http://nosql-database.org/
>
> 22 juli 2016, 10:26, ?/? John Robson
> <John.Robson at usp.br
> <mailto:John.Robson at usp.br>> ??????:
>
>> Is not "early optimization", I'm more comfortable with multiple
>> tables, furthermore, I have plans to expand to thousands of stocks and
>> high-frequency operations with more than 10K events per second (for
>> some stocks). Maybe store/read hundreds of thousands of rows per second.
>>
>> But my question is just, what's is the best approach to do CRUD with
>> multiple (dynamic) tables. (one class to map multiple tables)
>>
>> Thank you for helping.
>> John
>>
>> On 07/22/2016 01:57 AM, Andrea D'Amore wrote:
>>> On 22 July 2016 at 02:14, John Robson <John.Robson at usp.br
>>> <mailto:John.Robson at usp.br>> wrote:
>>>> I want to store the minute price of all 500 stocks from S&P 500.
>>>>  This means
>>>> more than 500 million of rows + almost 200K updates per day +
>>>> (sometimes)
>>>> past values must be updated (to adjust ex-dividend, splits).
>>>
>>>> Instead of creating a BIG table, I would like to create 500 tables
>>>> (one for
>>>> each stock). They have the same structure/columns (open, high, low,
>>>> close,
>>>> volume).
>>>
>>> Why? Did you hit some limit or performance issue already or is it a
>>> case of early optimization?
>>>
>>> Even sqlite3 should be able to handle that.
>>>
>>
>> _______________________________________________
>> Flask mailing list
>> Flask at python.org <mailto:Flask at python.org>
>> https://mail.python.org/mailman/listinfo/flask
>
>
> _______________________________________________
> Flask mailing list
> Flask at python.org
> https://mail.python.org/mailman/listinfo/flask
>


From Stephen.Mazzei at asrcfederal.com  Fri Jul 22 15:39:21 2016
From: Stephen.Mazzei at asrcfederal.com (Mazzei, Stephen Andrew)
Date: Fri, 22 Jul 2016 19:39:21 +0000
Subject: [Flask] Creating a bar chart with std and displaying
Message-ID: <FA1CCD1F7381504ABB7081B0462B9DDC01E2CBFE@AFSY-MXDBP2.asrcfh.local>

Good afternoon, another quick question.

I have the following
Out[83]: grouped.head()

Date    Wall Clock
0       2016-01-24      3348
1       2016-01-24      3337
2       2016-01-24      3370
3       2016-01-24      3343
4       2016-01-24      3345
6       2016-05-15      3381
7       2016-05-15      3432
8       2016-05-15      3417
9       2016-05-15      3444
10      2016-05-15      3412

Where date is the first column and the "Wall Clock" is the second column

I am trying to mark a bar chart showing the mean of the days wall clock times, with their std. I have the following

means = grouped.mean()
errors = grouped.std()
fig, ax = plt.subplots()
means.plot.bar(yerr=errors, ax=ax, legend=None))

Which does work, though kind of hard to see. Is there a better way to do this? If this is a fine way of doing this, my next question is how do I present this to flask front end? Do I need to save this to a .png and then load the .png on the server?

Thank you


---
Stephen A. Mazzei
Systems Administrator | AFDS, ASRC Federal Data Solutions - P&G HPC Account | 513-634-9965

________________________________

The preceding message (including attachments) is covered by the Electronic Communication Privacy Act, 18 U.S.C. sections 2510-2512, is intended only for the person or entity to which it is addressed, and may contain information that is confidential, protected by attorney-client or other privilege, or otherwise protected from disclosure by law. If you are not the intended recipient, you are hereby notified that any retention, dissemination, distribution, or copying of this communication is strictly prohibited. Please reply to the sender that you have received the message in error and destroy the original message and all copies.

From jani.sumak at gmail.com  Fri Jul 22 16:33:48 2016
From: jani.sumak at gmail.com (=?UTF-8?Q?Jani_=C5=A0umak?=)
Date: Fri, 22 Jul 2016 20:33:48 +0000
Subject: [Flask] Creating a bar chart with std and displaying
In-Reply-To: <FA1CCD1F7381504ABB7081B0462B9DDC01E2CBFE@AFSY-MXDBP2.asrcfh.local>
References: <FA1CCD1F7381504ABB7081B0462B9DDC01E2CBFE@AFSY-MXDBP2.asrcfh.local>
Message-ID: <CAHpkYiwHow-gV5D_wJkYAzaYAV9KrJn5EUOPL7=_L7teDTDa0Q@mail.gmail.com>

Hi Stephen,

saving the plot to a .png and than rendering a jinja2 template would be one
way to do it, but it'd recommend something like this:
http://bokeh.pydata.org/en/latest/

The documentation has some examples for working with Pandas dataframes
<http://bokeh.pydata.org/en/0.12.0/docs/user_guide/charts.html> and you can
find examples using Bokeh with Flask on github.

Hope that helps!

Cheers, Jani

On Fri, Jul 22, 2016 at 9:39 PM Mazzei, Stephen Andrew <
Stephen.Mazzei at asrcfederal.com> wrote:

> Good afternoon, another quick question.
>
> I have the following
> Out[83]: grouped.head()
>
> Date    Wall Clock
> 0       2016-01-24      3348
> 1       2016-01-24      3337
> 2       2016-01-24      3370
> 3       2016-01-24      3343
> 4       2016-01-24      3345
> 6       2016-05-15      3381
> 7       2016-05-15      3432
> 8       2016-05-15      3417
> 9       2016-05-15      3444
> 10      2016-05-15      3412
>
> Where date is the first column and the "Wall Clock" is the second column
>
> I am trying to mark a bar chart showing the mean of the days wall clock
> times, with their std. I have the following
>
> means = grouped.mean()
> errors = grouped.std()
> fig, ax = plt.subplots()
> means.plot.bar(yerr=errors, ax=ax, legend=None))
>
> Which does work, though kind of hard to see. Is there a better way to do
> this? If this is a fine way of doing this, my next question is how do I
> present this to flask front end? Do I need to save this to a .png and then
> load the .png on the server?
>
> Thank you
>
>
> ---
> Stephen A. Mazzei
> Systems Administrator | AFDS, ASRC Federal Data Solutions - P&G HPC
> Account | 513-634-9965
>
> ________________________________
>
> The preceding message (including attachments) is covered by the Electronic
> Communication Privacy Act, 18 U.S.C. sections 2510-2512, is intended only
> for the person or entity to which it is addressed, and may contain
> information that is confidential, protected by attorney-client or other
> privilege, or otherwise protected from disclosure by law. If you are not
> the intended recipient, you are hereby notified that any retention,
> dissemination, distribution, or copying of this communication is strictly
> prohibited. Please reply to the sender that you have received the message
> in error and destroy the original message and all copies.
> _______________________________________________
> Flask mailing list
> Flask at python.org
> https://mail.python.org/mailman/listinfo/flask
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/flask/attachments/20160722/dddb4b6f/attachment.html>

From nik.gen at gmx.de  Sat Jul 23 06:50:30 2016
From: nik.gen at gmx.de (Nikolaus Neusser)
Date: Sat, 23 Jul 2016 12:50:30 +0200
Subject: [Flask] folder structure and script-src-path
Message-ID: <fa9af51a-9ab1-dacc-1851-bc267dc74d77@gmx.de>

Hi everyone,

i am having troubles in organizing my files in sub-folders and adding them to my html files.
I am serving 4 static html files with custom .css and .js files.
What is important to me, is that i can open the html files locally with my webbrowser without the server as well.

folder structure:
   static\css -> custom.css
   static\js   -> custom.js, jquery.js, ..
   templates  -> 4x html files


index.html:
   If i deliver the file via the webserver i need the following code in index.html.
   <script type="text/javascript" src="/static/js/jquery-2.2.3.js"></script>

   If i open index.html locally with my webbrowser i need the following code.
   script type="text/javascript" src="../static/js/jquery-2.2.3.js"></script>

The difference is "/static/" vs. "../static/"

Thanks,
Ben




From davidnieder at gmx.de  Sat Jul 23 07:44:59 2016
From: davidnieder at gmx.de (David Nieder)
Date: Sat, 23 Jul 2016 13:44:59 +0200
Subject: [Flask] folder structure and script-src-path
In-Reply-To: <fa9af51a-9ab1-dacc-1851-bc267dc74d77@gmx.de>
References: <fa9af51a-9ab1-dacc-1851-bc267dc74d77@gmx.de>
Message-ID: <d8edcbee-a2fa-23ee-d68a-bd4a8ce60c6d@gmx.de>

On 23.07.2016 12:50, Nikolaus Neusser wrote:
> Hi everyone,
>
> i am having troubles in organizing my files in sub-folders and adding them to my html files.
> I am serving 4 static html files with custom .css and .js files.
> What is important to me, is that i can open the html files locally with my webbrowser without the server as well.

If those files are static html and not templates, just put them in your 
static folder ;)

>
> folder structure:
>    static\css -> custom.css
>    static\js   -> custom.js, jquery.js, ..
>    templates  -> 4x html files
>
>
> index.html:
>    If i deliver the file via the webserver i need the following code in index.html.
>    <script type="text/javascript" src="/static/js/jquery-2.2.3.js"></script>
>
>    If i open index.html locally with my webbrowser i need the following code.
>    script type="text/javascript" src="../static/js/jquery-2.2.3.js"></script>
>
> The difference is "/static/" vs. "../static/"
>
> Thanks,
> Ben
>
>
>
> _______________________________________________
> Flask mailing list
> Flask at python.org
> https://mail.python.org/mailman/listinfo/flask
>


From and.damore at gmail.com  Sat Jul 23 08:23:19 2016
From: and.damore at gmail.com (Andrea D'Amore)
Date: Sat, 23 Jul 2016 14:23:19 +0200
Subject: [Flask] Creating a bar chart with std and displaying
In-Reply-To: <FA1CCD1F7381504ABB7081B0462B9DDC01E2CBFE@AFSY-MXDBP2.asrcfh.local>
References: <FA1CCD1F7381504ABB7081B0462B9DDC01E2CBFE@AFSY-MXDBP2.asrcfh.local>
Message-ID: <CAMonFrQnjfwMrF-3aqWq09DqwA_WYEt7R3JHtuf0F77knvPOOg@mail.gmail.com>

On 22 July 2016 at 21:39, Mazzei, Stephen Andrew
<Stephen.Mazzei at asrcfederal.com> wrote:
> means = grouped.mean()
> errors = grouped.std()
> fig, ax = plt.subplots()
> means.plot.bar(yerr=errors, ax=ax, legend=None))

> Which does work, though kind of hard to see. Is there a better way to do this?

What do you mean by hard to see, is the figure too small?
And what kind of "better" are you looking for?

If you are looking for more compact code the following line should be
equivalent to what you wrote since DataFrame.plot() is able to create
its own figure and axis:

    grouped.mean().plot(kind='bar', yerr=grouped.std(), legend=False)


> If this is a fine way of doing this, my next question is how do I present this to flask front end? Do I need to save this to a .png and then load the .png on the server?

Isn't the code already running on the server? What do you mean by
"load on the server"?


-- 
Andrea

From and.damore at gmail.com  Sat Jul 23 08:27:33 2016
From: and.damore at gmail.com (Andrea D'Amore)
Date: Sat, 23 Jul 2016 14:27:33 +0200
Subject: [Flask] folder structure and script-src-path
In-Reply-To: <fa9af51a-9ab1-dacc-1851-bc267dc74d77@gmx.de>
References: <fa9af51a-9ab1-dacc-1851-bc267dc74d77@gmx.de>
Message-ID: <CAMonFrRbWvM3DAuxWPJKxKbEaaF6Rne02=0etsLzQyc57MKA+Q@mail.gmail.com>

On 23 July 2016 at 12:50, Nikolaus Neusser <nik.gen at gmx.de> wrote:

>   If i open index.html locally with my webbrowser i need the following code.
>   script type="text/javascript" src="../static/js/jquery-2.2.3.js"></script>
>
> The difference is "/static/" vs. "../static/"

If "locally" means the URL has by file:// protocol then it's the
correct behavior.

Locally should mean that you're running a local webserver (like
flask's builtin) and the URL for resources should indeed start with
the webserver root, e.g "/static/js/jquery?" .
If the latter is the case then provide a minimal example of your
layout that reproduces the issue and someone will find the reason.


-- 
Andrea

From 451517996 at qq.com  Sat Jul 23 08:28:54 2016
From: 451517996 at qq.com (=?gb18030?B?zuLD+g==?=)
Date: Sat, 23 Jul 2016 20:28:54 +0800
Subject: [Flask] a opensource rpc framework
Message-ID: <tencent_4A6A32F43B03CAEB5F17E37E@qq.com>

i write a game server opensource at github https://github.com/qianqians/abelkhan
 it is based at a rpc framework start by myself https://github.com/qianqians/abelkhan/tree/master/juggle
 welcome everyone involved in my project
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/flask/attachments/20160723/09006a6e/attachment.html>

From nik.gen at gmx.de  Sun Jul 24 15:47:23 2016
From: nik.gen at gmx.de (Nikolaus Neusser)
Date: Sun, 24 Jul 2016 21:47:23 +0200
Subject: [Flask] folder structure and script-src-path
In-Reply-To: <CAJ3SPdxyn1108pRWjm1LcF5ezq5nhUoE1D+ZNNYhFW-opbtYFw@mail.gmail.com>
References: <fa9af51a-9ab1-dacc-1851-bc267dc74d77@gmx.de>
 <CAJ3SPdxyn1108pRWjm1LcF5ezq5nhUoE1D+ZNNYhFW-opbtYFw@mail.gmail.com>
Message-ID: <64a3d850-a771-6e94-032d-789b210e5fc8@gmx.de>

I changed the includes to "/static" and "/templates" (without "..").
I am going to use a local webserver to test my webpages and not open them directly in the browser (file://..)

Thanks for the hints,
Ben


Am 23.07.2016 um 15:54 schrieb linnchord:
> Run Flask buildin server on local like http://localhost:8000 <http://localhost:8080>, do not open html template 
> directly by file://...
>
> Nikolaus Neusser <nik.gen at gmx.de <mailto:nik.gen at gmx.de>>?2016?7?23??? ??7:15???
>
>     Hi everyone,
>
>     i am having troubles in organizing my files in sub-folders and adding them to my html files.
>     I am serving 4 static html files with custom .css and .js files.
>     What is important to me, is that i can open the html files locally with my webbrowser without the server as well.
>
>     folder structure:
>        static\css -> custom.css
>        static\js   -> custom.js, jquery.js, ..
>        templates  -> 4x html files
>
>
>     index.html:
>        If i deliver the file via the webserver i need the following code in index.html.
>        <script type="text/javascript" src="/static/js/jquery-2.2.3.js"></script>
>
>        If i open index.html locally with my webbrowser i need the following code.
>        script type="text/javascript" src="../static/js/jquery-2.2.3.js"></script>
>
>     The difference is "/static/" vs. "../static/"
>
>     Thanks,
>     Ben
>
>
>
>     _______________________________________________
>     Flask mailing list
>     Flask at python.org <mailto:Flask at python.org>
>     https://mail.python.org/mailman/listinfo/flask
>
> -- 
> >> linnchord at gmail.com <mailto:linnchord at gmail.com>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/flask/attachments/20160724/70e23262/attachment.html>

From rsukhlec at visa.com  Tue Jul 26 13:46:19 2016
From: rsukhlec at visa.com (Sukhlecha, Raagi)
Date: Tue, 26 Jul 2016 17:46:19 +0000
Subject: [Flask] Flask + mongodb
Message-ID: <496ab1abad484cad8e837d104c5cabd1@SW550EXP004.visa.com>

Hey All

I working on developing a RESTful api using flask. I want to use mongodb with flask and I looked at couple of available options:

1)      Mongoengine<https://github.com/givp/Flask-MongoDB-Project> - ORM

2)      Flask-Pymongo<https://flask-pymongo.readthedocs.io/en/latest/> - ORM

3)      Flask-Mongokit - Simplifies the use of Mongokit with Flask applictions. in the flask extension list here<http://flask.pocoo.org/extensions/>

4)      Flask-MongoAlchemy - Flask support for mongoAlchemy. in the flask extension list here<http://flask.pocoo.org/extensions/>

5)      Mongoclient - of pymongo

I am not able to decide how are they different and which one to use.

Also, there are some extensions which are not listed on the official flask page like 1,2 above and others like flask-mongorest /eve which provide support for both RESTful api and mongodb.  Is it okay to go ahead with extensions not listed on the flask page? Suggestions please.

Thanks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/flask/attachments/20160726/cc36f6d2/attachment.html>

From badrihippo at gmail.com  Fri Jul 29 07:09:43 2016
From: badrihippo at gmail.com (Badri Sunderarajan)
Date: Fri, 29 Jul 2016 16:39:43 +0530
Subject: [Flask] Flask + mongodb
In-Reply-To: <496ab1abad484cad8e837d104c5cabd1@SW550EXP004.visa.com>
References: <496ab1abad484cad8e837d104c5cabd1@SW550EXP004.visa.com>
Message-ID: <579B3977.50503@gmail.com>

Any library should be fine, as long as you're comfortable with it. Don't 
worry about it not being on the extensions page?many of the newer ones 
may have just not got there yet.

If you're concerned about how active/supported the project is, you could 
look at (for example) the number of stars on GitHub or the date of the 
latest release.

Personally, I found MongoEngine simple (along with the flask-mongoengine 
wrapper), because it works the same way as peewee and other ORMs that 
I'm used to. If your API fits with what flask-mongorest provides, that 
might be a better option for you.

My suggestion would be to try flask-mongorest for a day or two and see 
if it works out. If not, you can switch to a normal ODM like MongoEngine 
or MongoAlchemy. As far as I know they all have similar features, so 
which one you settle on is just a matter of preference. (Though if 
there's a specific feature you need, it would be a good idea to verify 
that your chosen library supports it).

Hope that helps!

From nollroberts at gmail.com  Sun Jul 31 13:16:50 2016
From: nollroberts at gmail.com (Noll)
Date: Sun, 31 Jul 2016 11:16:50 -0600
Subject: [Flask] post
Message-ID: <CAGWM=wqm8BYPwT-NDMSTcHj4c1h0oBUy+6P3XU1JOYAvEmbhmg@mail.gmail.com>

To anyone interested in developing democratic software, consider reading:
https://amodicumofdemocracy.wordpress.com/2016/07/29/population-based-democratic-majority-decision-making-rough-draft/
? and post any comments, questions, or concerns.?

-- 
Noll Roberts
nollroberts at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/flask/attachments/20160731/cccf3508/attachment.html>