[Distutils] PyPI v2 (Was: PyPI pull request #7)

[Donald Stufft]

On Nov 8, 2013, at 7:21 AM, anatoly techtonik <techtonik at gmail.com> wrote:

> First some background questions:
> 1. Everything for core development is in HG. Why Git now? Why
> Mercurial suxx (three major personal annoyances will do)? Why
> BitBucket suxx?

I’m most familiar with git, so I used git when I started it.

> 2. Why Apache 2.0? Is it because everybody is using that? Why not try
> CC0/CC-BY/MIT/ISC - code under these licenses is easier to copy/paste,
> and is a better investment for your time as a coder studying the stuff
> and filling up the space of your brain.

Apache 2.0 is a good license, it is similar to MIT/ISC except it has a
patent clause.

> 
> 3. Why the raw SQL? It cripples the ability to scale, to host package
> indexes on GAE or OpenStack. And it is right at the core. Why not to
> "port NDB" to SQL or use HyperDB from Roundup?

It was just recently switched to raw sql from using the SQL Alchemy
SQL expression layer. The reason for the switch was we were not
using the portability features and it was more complicated to understand
the expression layer than just plain SQL.

> 4. Why not Django/TG/... framework? (+pinax, +openid, +other_stuff)

I’ll be adding a section to the documentation on this eventually, but
basically.

Django’s ORM wasn’t powerful enough to represent the existing Schema. When
I had to throw out the ORM I found it difficult to integrate other pieces without
instating more global state. Ultimately I felt that a Fraken-Django that used Django
on the surface but with pieces replaced was going to require as much or more
of  a learning curve, even for Django developers, that it didn’t make sense to
do it.

I then took a look at Flask, but Flask’s reliance on thread locals everywhere is
not something I’m a fan of.

At that point I was a bit frustrated and fed up and turned to using just Werkzeug
as a library. This enabled a rapid increase in pace (I got more down over the
initial weekend then I had in a month previously).

> 
>>> You also need
>>> some designer guy in a team.
>> 
>> Have one who’ll have time in a few weeks to go over everything and make it really great :)
> 
> That's cool. Will it be an iterative process or just one shot? Do you
> care about public feedback? I mean of course you do, but is there a
> public channel to allow users set status if they liked the previous
> version more, and what should be changed for them to change their
> opinion?

Design by committee doesn’t work. People are free to give feedback through
either this list, the IRC channel, github issues, or to me privately but ultimately
I trust the designer to do a good job, to listen to any feedback if we get any,
and to execute well.

Feedback is good, but it’ll be tempered to prevent http://theoatmeal.com/comics/design_hell

> 
>>> You also need a backlog for
>>> collaboration. My ETA for new PyPI is no earlier than PyCon 2014 if
>>> Donald and Richard will be working on it full time.
>> 
>> Possibly! I’m unsure of how long it will take, it’s primarily Richard and I but we’ve a
>> few domain experts in particular pieces who have offered to help out as well when
>> we’re ready for their pieces.
> 
> PostgreSQL is killing all the fun. It takes a few hours to get
> acquainted with its way of doing things - groups as users, per table
> grants and insufficient DB permissions. There should be an option to
> run on SQlite for development, like in Trac, Roundup etc.

Warehouse only runs on PostgreSQL. It’s not that hard to have a local copy of
PostgreSQL and It’ll enable using the advanced feature set of PostgreSQL
instead of being limited to the lowest common denominator.

> 
>>> So, instead of
>>> all-or-nothing scenario I can try to find some help with incremental
>>> approach.
>> 
>> Mostly the problem with improving the current base is every change is particularly
>> dangerous. The code base is large, it’s untested, and it’s not very nice. It’s extremely
>> easy to break things by accident with seemingly unrelated change. Richard and I
>> have both done this multiple times.
> 
> This smells like a bad spaghetti. Do you think it will be maintainable
> if it is already so fragile on the early stages of development?

This is talking about the current code base, not about Warehouse. Warehouse has
and requires 100% unit test coverage and will be gaining a functional test suite
on top of that to test high level user stories.

> 
>> So every PR we accept has a danger of breaking
>> things. This incurs a high cognitive overload for accepting a PR because it means we
>> have to spin up a copy of the site and manually go through and test any feature we
>> think *might* be affected which typically catches most things but not always. It’s a time
>> and labor intensive process which none of us enjoy and which we’ve decided not to
>> prioritize unless the pay offs are large.
> 
> If you're not doing this only for yourself and your own pleasure, then
> dedicating more time to improve the process and making it more
> inclusive will definitely pay off for the rest of pydotorg projects
> and active part of Python community, who is still here.

This was again about the existing code base and not Warehouse. Warehouse
is (hopefully) easy to iterate on, has what I believe is a well laid out
code base, and has an extensive unit test suite.

> 
> 
> Do you have a diagram of the system what are you trying to build, or
> is it just an experiment? It may worth to put IDE aside and try to
> de-couple some things first on the whiteboard. I am afraid that I will
> never be ready to help with reinventing Django. At least not without
> some research and art coverage.

It’s not a reinvention of Django. The “framework” parts of Warehouse
are very small and are mostly glue that holds together pieces like
Werkzeug and SQLAlchemy.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131108/a3f45de2/attachment-0001.sig>