[Pandas-dev] New website infrastructure

Tue Sep 24 08:59:51 EDT 2019

Just a quick update on the infrastructure for the pandas hosting. We
(Sylvain and myself) just had a call with the people from OVH, to discuss
what was said here.

A quick summary:
- We shared more in detail what we need with them, and the credit we
discussed (having them in the sponsors page, and in the corner of the
Binder executable widgets)
- They seem happy about it, but need to discuss internally, they expect to
have a final answer in couple of weeks
- It may make more sense to reuse the current Binder infrastructure (if the
Binder team agrees, of course)
- They'll have to set a limit on the number resources we can use, and if we
ever exceed them, we'll discuss again the conditions (more like a
formalism, I don't expect the limit to be something to worry about)
- They are happy to consider doing the same with the whole ecosystem. But
we'll first start with pandas to not make things more complex, and if they
host multiple projects we'll manage it at the NumFOCUS level, so they don't
need to deal with many projects individually

I think those are the main points, if I missed something, or anything is
not clear, please feel free to add to that Sylvain.

Regarding Discourse, any progress in your side with that Andy?

Cheers!

On Fri, Sep 20, 2019 at 12:18 PM Marc Garcia <garcia.marc at gmail.com> wrote:

> I'm fine with that conceptually, but I think Discourse will make things
> quite tricky to find things then.
>
> We already got our discourse approved, if you want to join it an
> experiment with the setting. But it's the first thing I tried, and after
> you join a category (project), everything feels like it's in the same place
> (even if subcategories and tags exist). And I think we need at least a
> clear separation between pandas/users pandas/contributors discussions.
>
> May be I just couldn't find the settings, let me know if you manage to get
> a multi-project set up that makes sense.
>
> On Fri, Sep 20, 2019 at 12:07 PM Tom Augspurger <
> tom.augspurger88 at gmail.com> wrote:
>
>> I'd prefer to join a discourse along with NumPy, Dask, and other PyData
>> or NumFOCUS projects, rather than going out on our own.
>>
>> On Fri, Sep 20, 2019 at 4:47 AM Marc Garcia <garcia.marc at gmail.com>
>> wrote:
>>
>>> I don't know much about discourse, but why do we want to self-host it?
>>> Seems like Discourse does it for free for open source projects:
>>> https://free.discourse.group/ And I don't think we want another system
>>> to maintain. Am I missing something?
>>>
>>> I applied for https://pandas.discourse.group, so we can give it a try.
>>> We should have it approved and working in couple of days.
>>>
>>> For what I saw, Discourse has one level of categories, so I guess we
>>> want one per project, so we can have categories for "Users",
>>> "Contributors", "Ecosystem"... or something similar. I guess if we have a
>>> single Discourse for NumFOCUS, every project will be a category, and it'll
>>> be difficult to group conversations.
>>>
>>> If anyone already has experience with Discourse and disagrees with my
>>> guesses, please let me know.
>>>
>>> On Wed, Sep 18, 2019 at 4:32 PM Andy Terrel <andy at numfocus.org> wrote:
>>>
>>>> Sounds great to me. Just let me know where everything goes.
>>>>
>>>> NumPy wants me to help host a discourse for them, maybe OVH would be a
>>>> good place to do that as well, (although I would be more inclinded if it
>>>> was pydata and we had pandas, scipy, and numpy on it).
>>>>
>>>> -- Andy
>>>>
>>>> On Wed, Sep 18, 2019 at 8:51 AM Tom Augspurger <
>>>> tom.augspurger88 at gmail.com> wrote:
>>>>
>>>>> Sounds good w.r.t crediting OVH on those pages.
>>>>>
>>>>> For the ASV results at pandas.pydata.org/speed (which I now notice is
>>>>> currently broken for pandas), the only thing on the webserver is a
>>>>> cron job doing a `git pull` from
>>>>> https://github.com/asv-runner/asv-collection, from within
>>>>> `/usr/share/nginx`.
>>>>>
>>>>> Tom
>>>>>
>>>>>
>>>>> On Wed, Sep 18, 2019 at 8:18 AM Marc Garcia <garcia.marc at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> An update on the new website infrastructure. We need to finish
>>>>>> discussing the details, but OVH is happy to provide the hosting for the
>>>>>> pandas infrastructure we need.
>>>>>>
>>>>>> My initial idea is to credit them in the page with the rest of the
>>>>>> sponsors in the new website:
>>>>>> https://datapythonista.github.io/pandas-web/community/team.html#institutional-partners and
>>>>>> also in the top right corner of the runnable code widgets (see for example
>>>>>> where Binder is credited here: https://spacy.io/).
>>>>>>
>>>>>> What I'd like to ask is:
>>>>>>
>>>>>> 1. For the production website and docs (static content only, for the
>>>>>> traffic we need):
>>>>>> https://us.ovhcloud.com/products/public-cloud/object-storage
>>>>>> 2. For our tools and processes, like the benchmarks, builds, CI stuff
>>>>>> (temporary publish the docs for every PR,...):
>>>>>> https://www.ovh.co.uk/vps/vps-ssd.xml (VPS SSD 3)
>>>>>> 3. For BinderHub (runnable code in our docs, launch tutorials on
>>>>>> Binder...): https://www.ovh.co.uk/public-cloud/kubernetes/
>>>>>>
>>>>>> For the BinderHub, QuantStack offered help with the set up (which is
>>>>>> great, because I don't know much about Binder myself, and I'm not sure if
>>>>>> anyone else does or wants to take care of this). I don't think it'll be
>>>>>> easy to estimate how big is the cluster we need beforehand, but I guess we
>>>>>> can add things to Binder iteratively, and have more info as we grow.
>>>>>>
>>>>>> OVH gave us a 200 euros voucher to experiment with the different
>>>>>> services. Let me know how all this sounds, and if there are no objections,
>>>>>> I'll create an account and buy those services with the voucher, and I'll
>>>>>> start to prototype and see how everything works.
>>>>>>
>>>>>> Cheers!
>>>>>>
>>>>>> On Tue, Aug 20, 2019 at 11:06 PM Marc Garcia <garcia.marc at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Somehow related to the work on the new website (
>>>>>>> https://github.com/pandas-dev/pandas/pull/28014), I've been
>>>>>>> discussing with the Binder team, and looks like should be quite easy soon
>>>>>>> (with a Sphinx extension) to make all the documentation pages runnable with
>>>>>>> Binder, directly from the website (without opening the page as a Jupyter in
>>>>>>> mybinder).
>>>>>>>
>>>>>>> While they are very happy with the idea of having this is pandas,
>>>>>>> it's uncertain if the current infrastructure Binder has got, is able to
>>>>>>> handle all the traffic we would send. And scikit-learn is working on it too
>>>>>>> (today they added to the dev docs a link to mybinder to run the examples).
>>>>>>>
>>>>>>> I'm discussing with OVH (their infrastructure provider) on whether
>>>>>>> they'd be happy to provide a dedicated BinderHub specific to pandas (or may
>>>>>>> be we can have one for all NumFOCUS projects). We'll see how it goes, but
>>>>>>> wanted to let you know, so you're updated, and in case anyone is interested
>>>>>>> in participating in the discussions. Of course before any decision is made
>>>>>>> I'll open a discussion here or on GitHub.
>>>>>>>
>>>>>>> As part of the discussion I'm also trying to get a server for the
>>>>>>> website, and one for development stuff. Specfically for the dev docs
>>>>>>> (including rendered docs of every PR) and the GitHub app that will generate
>>>>>>> them. I guess it should be very easy to find a sponsor for these two
>>>>>>> servers (in exchange of a small note in the footer of the website, or
>>>>>>> something like that).
>>>>>>>
>>>>>>> Let me know if you have any comment, want to be involved or whatever.
>>>>>>>
>>>>>>> Cheers!
>>>>>>>
>>>>>> _______________________________________________
>>>>>> Pandas-dev mailing list
>>>>>> Pandas-dev at python.org
>>>>>> https://mail.python.org/mailman/listinfo/pandas-dev
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Andy R. Terrel, PhD
>>>> President
>>>> NumFOCUS
>>>> andy at numfocus.org
>>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pandas-dev/attachments/20190924/327809c6/attachment.html>