[Catalog-sig] Problems with pep381client

[Richard Jones]

Sorry Brian, I've been busy with other aspects of PyPI. I hope to
discuss the mirroring issues at PyCon in a couple of weeks' time, and
hopefully come up with at least an interim solution during the sprints
there.


     Richard

On 28 February 2013 06:35, Brian Long <briandlong at gmail.com> wrote:
> Hello all,
>
> I've heard nothing in response to my email and with all the discussion
> taking place on this list regarding the security posture of Pypi, I
> assume private mirrors need to hold off until all the issues are
> resolved.  Am I correct in the assumption that I should avoid using
> pep381client to privately mirror Pypi content?
>
> Thank you.
>
> /Brian/
>
> On Wed, Feb 20, 2013 at 11:37 AM, Brian Long <briandlong at gmail.com> wrote:
>> Hello,
>> I've been trying to set up a private Pypi mirror at my place of
>> employment using the pep381client v1.5.  The script has bombed at
>> various places while trying to download packages over the last three
>> weeks.  I've changed __init__.py to specify a.pypi.python.org (and
>> others), but it still fails.
>>
>> After a couple of weeks of trying to mirror, fixing problems (changing
>> mirrors), etc. I've not been able to get past the following error:
>> Synchronizing Quotient
>> Copying /packages/source/Q/Quotient/Quotient-0.3.0.tar.gz
>> Traceback (most recent call last):
>>   File "/usr/local/pep381client-1.5/scripts/pep381run", line 30, in <module>
>>     state.synchronize()
>>   File "/usr/local/pep381client-1.5/scripts/../pep381client/__init__.py",
>> line 119, in synchronize
>>     self._synchronize()
>>   File "/usr/local/pep381client-1.5/scripts/../pep381client/__init__.py",
>> line 159, in _synchronize
>>     self.maybe_copy_file(project, file)
>>   File "/usr/local/pep381client-1.5/scripts/../pep381client/__init__.py",
>> line 247, in maybe_copy_file
>>     data = r.read()
>>   File "/usr/lib64/python2.6/httplib.py", line 529, in read
>>     s = self._safe_read(self.length)
>>   File "/usr/lib64/python2.6/httplib.py", line 619, in _safe_read
>>     chunk = self.fp.read(min(amt, MAXAMOUNT))
>>   File "/usr/lib64/python2.6/socket.py", line 383, in read
>>     data = self._sock.recv(left)
>> socket.error: [Errno 104] Connection reset by peer
>>
>> I'm not sure if my problems could be caused by a transparent proxy or
>> if a.pypi.python.org is refusing my connection for another reason.  My
>> source IP is 64.102.53.91.  So far, my Pypi directory has 27GB
>> downloaded.
>>
>> I'm more familiar with mirroring Linux distributions using rsync.  If
>> there were a way to set up the initial Pypi mirror using rsync and
>> then fall back to pep381client to keep things in sync, that would be
>> great.
>>
>> Thank you for any assistance in troubleshooting this problem.
>>
>> /Brian/
> _______________________________________________
> Catalog-SIG mailing list
> Catalog-SIG at python.org
> http://mail.python.org/mailman/listinfo/catalog-sig