Fwd: Fwd: ftplib sending data out of order

[Charles Wilt]

Just to close out this thread in case anybody finds it...

We're reasonably sure the issue was cause by version 14.6 MP1 of Symantec
Vontu; which is a Data Loss Prevention (DLP) product that "inspects"
packets before they leave the PC.

I believe the issue was reported to Symantec.

In the meantime, we worked around the issue by extending the base ftplib.FTP
 class and adding a new function:

#stortext combines block read from STORBINARY with ASCII transfer mode
from STORLINEdef stortext(self, cmd, fp, blocksize=8192,
callback=None, rest=None):
    self.voidcmd('TYPE A')
    conn = self.transfercmd(cmd, rest)
    while 1:
        buf = fp.read(blocksize)
        if not buf: break
        conn.sendall(buf)
        if callback: callback(buf)
    conn.close()
    return self.voidresp()

We stopped having issues with scrambled code. As a bonus, the transfer run
considerably faster than they did with storline.

So we've continued to use our new function.


On Mon, Jul 10, 2017 at 7:23 AM, Charles Wilt <charles.wilt at gmail.com>
wrote:

> I downloaded the 3.6.1 source and added some debug logging to
> sock_sendall() and sock_send_impl() in socketmodule.c.
>
> As I understand it, sock_send_impl() is where the data is passed to the
> OS.
>
> I can see that sock_send_impl() is getting the data in the correct order.
>
>
>
>
> On Wed, Jul 5, 2017 at 2:03 PM, Dieter Maurer <dieter at handshake.de> wrote:
>
>> Charles Wilt wrote at 2017-7-5 11:08 -0600:
>> >On Mon, Jul 3, 2017 at 4:24 PM, Dieter Maurer <dieter at handshake.de>
>> wrote:
>> >
>> >>
>> >> It is not the "readline" that changes depending on the callback.
>> >> Keep in mind that "conn.sendall" can return before the bytes are
>> actually
>> >> sent.
>> >>
>> >
>> >?Not according to this answer
>> >https://stackoverflow.com/questions/34252273/what-is-the-di
>> fference-between-socket-send-and-socket-sendall
>>
>> "sendall" internally calls "send" until all the bytes passed as argument
>> are accepted by "send". Below "send", the things are packeted, transmitted
>> and reassembled.
>>
>> If send returns, this does *not* mean that the data has been successfully
>> transfered. It only means that a lower level has taken over responsibilty
>> for the transfer.
>>
>> As a consequence, "sendall" may return before the data has been
>> successfully transfered; it only means that the complete data was
>> accepted by the lower (network) level.
>>
>>
>> >And if sendall can return before the data is sent....wouldn't that make
>> it
>> >the wrong choice for an FTP client?
>>
>> Why?
>>
>> The data is transfered to the lower level and this has taken
>> over responsibility for the transfer. "ftplib" is using a TCP socket.
>> TCP promises to ensure a reliable transfer (by using retransmissions, if
>> necessary and possible). Under normal conditions, the data will be
>> transfered;
>> thus, why wait until the transfer is complete.
>>
>>
>>
>> --
>> Dieter
>>
>
>