[Flask] File upload breaks for files with size >64kb on ARM based apache wsgi servers

[Michael Bayer]

Hi Scott,

thanks a lot for the suggestion to save it in chunks. I like the 
approach and I will take it over :)

Unfortunately it did not resolve the issue, I still get the Bad request 
error.

On the Raspi, I have your version in place.

Best regards,

Michael


Am 03.10.2018 um 16:47 schrieb Scott Werner:
> Michael,
>
> Is it possible that the uploaded file size is causing the server to 
> run out of memory when saving? You can try saving it in chunks:
>
> def read_in_chunks(file_object, block_size=1024):
>     """Return a generator which yields data in chunks.
>
>     Source: `read-file-in-chunks-ram-usage-read-strings-from-binary-file
>     <http://stackoverflow.com/questions/17056382/
> read-file-in-chunks-ram-usage-read-strings-from-binary-files>`_
>
>     :param file_object: File object to read in chunks.
>     :type file_object: file object
>
>     :param block_size: (optional) Chunk size.
>     :type block_size: int
>
>     :yield: The next chunk in file object.
>     :yield type: `bytes`
>     """
>     for chunk in iter(lambda: file_object.read(block_size), b''):
>         yield chunk
>
> @app.route("/add", methods=["GET", "POST"])
> def add():
>     ...
>     # gpx_file.save(gpx_fspath)
> with open(gpx_fspath, 'wb') as file_:
>         for data in read_in_chunks(gpx_file.stream):
>             file_.write(data)
>
> https://github.com/scottwernervt/flask-upload-issue-arm/blob/master/views.py
>
> On Wed, Oct 3, 2018 at 9:42 AM Michael Bayer <lists at mjbayer.de 
> <mailto:lists at mjbayer.de>> wrote:
>
>
>     Am 02.10.2018 um 23:39 schrieb Geert Stappers:
>>     On Tue, Oct 02, 2018 at 05:05:47PM +0200, Michael Bayer wrote:
>>>     Hi everyone,
>>>
>>>     I discovered a funny behavior of my flask app, when running on ARM
>>>     based apache wsgi servers. Could you please give me some hints to
>>>     debug this?
>>>
>>>     I have a flask application, that supports file upload.
>>>     It runs on an ARM based debian stable server with apache wsgi.
>>>     If I upload small files, everything is fine. The problem appears,
>>>     when I upload files with a file size >64kb. The famous "Bad request
>>>     - The browser (or proxy) sent a request that this server could not
>>>     understand." pops up.
>>>
>>>     Things I tried out so far:
>>>     - Deploying my flask app to my productive ARM based webserver
>>>     (debian
>>>        stable): Here I discovered the issue.
>>>     - Running in flask's development wsgi server on my Development PC:
>>>         No problem, I can upload files with any size I want.
>>>     - Clean debian stable installation in a virtualbox on my
>>>     Development
>>>        PC (AMD64) same packages as my productive server, but AMD64
>>>     instead
>>>        of armhf (apache, mod_wsgi): Again no problems at all.
>>>     - Clean installation of raspian on a spare ARM Raspberry Pi
>>>     (based on
>>>        debian 9) with apache, mod_wsgi: Here comes the problem again!
>>>     - Increase apache LogLevel to debug -> no error logged
>>>     - Activate FileLogging inside my flask app -> no error logged
>>>
>>>     Have you seen anything like this?
>>     No.
>>
>>     Thing that intriges me, is that 64K   is a 16-bit limit.
>>     ARMHF versus AMD64  is 32-bit vs 64-bit ...
>     Thanks for your answer!
>     Yes, this is strange. I also don't have an explanation
>>
>>>     I unfortunately do not have much experience with web
>>>     servers/development
>>>     so I'm stuck with my debugging approaches :/
>>>
>>>     What could I do besides that?
>>     Consider it a none flask problem, it will allow you to look at
>>     Apache,
>>     uWSGI, Python and other links in the chain. Is a webproxy involved??
>
>     There is no proxy in the network.
>
>
>>>     What information do you need from my side?
>>>     I can provide any code/configuration you need.
>>     That will help reproducing the problem.
>     I created a minimal example to reproduce the error:
>     https://github.com/mjbayer/flask-upload-issue-arm
>
>     This repo contains code, configuration and example files that can
>     be used for upload.
>
>     Here is a demo Raspberry Pi (just for that purpose) to try it out:
>     http://rgb.mjbayer.de/trackdb/
>     (I could also give ssh access to the Raspi if required)
>
>     If you run the code in flasks development wsgi server or apache
>     mod_wsgi on AMD64, you will not see the issue.
>
>
>     Michael
>
>     _______________________________________________
>     Flask mailing list
>     Flask at python.org <mailto:Flask at python.org>
>     https://mail.python.org/mailman/listinfo/flask
>
>
>
> -- 
> Scott Werner
> scott.werner.vt at gmail.com <mailto:scott.werner.vt at gmail.com>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/flask/attachments/20181003/5a697979/attachment.html>