[Flask] File upload breaks for files with size >64kb on ARM based apache wsgi servers

[Scott Werner]

Michael,

Is it possible that the uploaded file size is causing the server to run out
of memory when saving? You can try saving it in chunks:

def read_in_chunks(file_object, block_size=1024):
    """Return a generator which yields data in chunks.

    Source: `read-file-in-chunks-ram-usage-read-strings-from-binary-file
    <http://stackoverflow.com/questions/17056382/
    read-file-in-chunks-ram-usage-read-strings-from-binary-files>`_

    :param file_object: File object to read in chunks.
    :type file_object: file object

    :param block_size: (optional) Chunk size.
    :type block_size: int

    :yield: The next chunk in file object.
    :yield type: `bytes`
    """
    for chunk in iter(lambda: file_object.read(block_size), b''):
        yield chunk

@app.route("/add", methods=["GET", "POST"])
def add():
    ...
    # gpx_file.save(gpx_fspath)
    with open(gpx_fspath, 'wb') as file_:
        for data in read_in_chunks(gpx_file.stream):
            file_.write(data)

https://github.com/scottwernervt/flask-upload-issue-arm/blob/master/views.py

On Wed, Oct 3, 2018 at 9:42 AM Michael Bayer <lists at mjbayer.de> wrote:

>
> Am 02.10.2018 um 23:39 schrieb Geert Stappers:
>
> On Tue, Oct 02, 2018 at 05:05:47PM +0200, Michael Bayer wrote:
>
> Hi everyone,
>
> I discovered a funny behavior of my flask app, when running on ARM
> based apache wsgi servers. Could you please give me some hints to
> debug this?
>
> I have a flask application, that supports file upload.
> It runs on an ARM based debian stable server with apache wsgi.
> If I upload small files, everything is fine. The problem appears,
> when I upload files with a file size >64kb. The famous "Bad request
> - The browser (or proxy) sent a request that this server could not
> understand." pops up.
>
> Things I tried out so far:
> - Deploying my flask app to my productive ARM based webserver (debian
>    stable): Here I discovered the issue.
> - Running in flask's development wsgi server on my Development PC:
>     No problem, I can upload files with any size I want.
> - Clean debian stable installation in a virtualbox on my Development
>    PC (AMD64) same packages as my productive server, but AMD64 instead
>    of armhf (apache, mod_wsgi): Again no problems at all.
> - Clean installation of raspian on a spare ARM Raspberry Pi (based on
>    debian 9) with apache, mod_wsgi: Here comes the problem again!
> - Increase apache LogLevel to debug -> no error logged
> - Activate FileLogging inside my flask app -> no error logged
>
> Have you seen anything like this?
>
> No.
>
> Thing that intriges me, is that 64K   is a 16-bit limit.
> ARMHF versus AMD64  is 32-bit vs 64-bit ...
>
> Thanks for your answer!
> Yes, this is strange. I also don't have an explanation
>
>
> I unfortunately do not have much experience with web servers/development
> so I'm stuck with my debugging approaches :/
>
> What could I do besides that?
>
> Consider it a none flask problem, it will allow you to look at Apache,
> uWSGI, Python and other links in the chain. Is a webproxy involved??
>
>
> There is no proxy in the network.
>
>
> What information do you need from my side?
> I can provide any code/configuration you need.
>
> That will help reproducing the problem.
>
> I created a minimal example to reproduce the error:
> https://github.com/mjbayer/flask-upload-issue-arm
>
> This repo contains code, configuration and example files that can be used
> for upload.
>
> Here is a demo Raspberry Pi (just for that purpose) to try it out:
> http://rgb.mjbayer.de/trackdb/
> (I could also give ssh access to the Raspi if required)
>
> If you run the code in flasks development wsgi server or apache mod_wsgi
> on AMD64, you will not see the issue.
>
>
> Michael
>
> _______________________________________________
> Flask mailing list
> Flask at python.org
> https://mail.python.org/mailman/listinfo/flask
>


-- 
Scott Werner
scott.werner.vt at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/flask/attachments/20181003/ca4f5b1e/attachment-0001.html>