[Pandas-dev] New website infrastructure

Wed Sep 25 08:32:34 EDT 2019

On Tue, Sep 24, 2019 at 8:00 AM Marc Garcia <garcia.marc at gmail.com> wrote:

> Just a quick update on the infrastructure for the pandas hosting. We
> (Sylvain and myself) just had a call with the people from OVH, to discuss
> what was said here.
>
> A quick summary:
> - We shared more in detail what we need with them, and the credit we
> discussed (having them in the sponsors page, and in the corner of the
> Binder executable widgets)
> - They seem happy about it, but need to discuss internally, they expect to
> have a final answer in couple of weeks
> - It may make more sense to reuse the current Binder infrastructure (if
> the Binder team agrees, of course)
> - They'll have to set a limit on the number resources we can use, and if
> we ever exceed them, we'll discuss again the conditions (more like a
> formalism, I don't expect the limit to be something to worry about)
> - They are happy to consider doing the same with the whole ecosystem. But
> we'll first start with pandas to not make things more complex, and if they
> host multiple projects we'll manage it at the NumFOCUS level, so they don't
> need to deal with many projects individually
>
> I think those are the main points, if I missed something, or anything is
> not clear, please feel free to add to that Sylvain.
>
> Regarding Discourse, any progress in your side with that Andy?
>

We moved that to another thread, but for status, I applied for a free
account but haven't heard back.

>
> Cheers!
>
> On Fri, Sep 20, 2019 at 12:18 PM Marc Garcia <garcia.marc at gmail.com>
> wrote:
>
>> I'm fine with that conceptually, but I think Discourse will make things
>> quite tricky to find things then.
>>
>> We already got our discourse approved, if you want to join it an
>> experiment with the setting. But it's the first thing I tried, and after
>> you join a category (project), everything feels like it's in the same place
>> (even if subcategories and tags exist). And I think we need at least a
>> clear separation between pandas/users pandas/contributors discussions.
>>
>> May be I just couldn't find the settings, let me know if you manage to
>> get a multi-project set up that makes sense.
>>
>> On Fri, Sep 20, 2019 at 12:07 PM Tom Augspurger <
>> tom.augspurger88 at gmail.com> wrote:
>>
>>> I'd prefer to join a discourse along with NumPy, Dask, and other PyData
>>> or NumFOCUS projects, rather than going out on our own.
>>>
>>> On Fri, Sep 20, 2019 at 4:47 AM Marc Garcia <garcia.marc at gmail.com>
>>> wrote:
>>>
>>>> I don't know much about discourse, but why do we want to self-host it?
>>>> Seems like Discourse does it for free for open source projects:
>>>> https://free.discourse.group/ And I don't think we want another system
>>>> to maintain. Am I missing something?
>>>>
>>>> I applied for https://pandas.discourse.group, so we can give it a try.
>>>> We should have it approved and working in couple of days.
>>>>
>>>> For what I saw, Discourse has one level of categories, so I guess we
>>>> want one per project, so we can have categories for "Users",
>>>> "Contributors", "Ecosystem"... or something similar. I guess if we have a
>>>> single Discourse for NumFOCUS, every project will be a category, and it'll
>>>> be difficult to group conversations.
>>>>
>>>> If anyone already has experience with Discourse and disagrees with my
>>>> guesses, please let me know.
>>>>
>>>> On Wed, Sep 18, 2019 at 4:32 PM Andy Terrel <andy at numfocus.org> wrote:
>>>>
>>>>> Sounds great to me. Just let me know where everything goes.
>>>>>
>>>>> NumPy wants me to help host a discourse for them, maybe OVH would be a
>>>>> good place to do that as well, (although I would be more inclinded if it
>>>>> was pydata and we had pandas, scipy, and numpy on it).
>>>>>
>>>>> -- Andy
>>>>>
>>>>> On Wed, Sep 18, 2019 at 8:51 AM Tom Augspurger <
>>>>> tom.augspurger88 at gmail.com> wrote:
>>>>>
>>>>>> Sounds good w.r.t crediting OVH on those pages.
>>>>>>
>>>>>> For the ASV results at pandas.pydata.org/speed (which I now notice
>>>>>> is currently broken for pandas), the only thing on the webserver is a
>>>>>> cron job doing a `git pull` from
>>>>>> https://github.com/asv-runner/asv-collection, from within
>>>>>> `/usr/share/nginx`.
>>>>>>
>>>>>> Tom
>>>>>>
>>>>>>
>>>>>> On Wed, Sep 18, 2019 at 8:18 AM Marc Garcia <garcia.marc at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> An update on the new website infrastructure. We need to finish
>>>>>>> discussing the details, but OVH is happy to provide the hosting for the
>>>>>>> pandas infrastructure we need.
>>>>>>>
>>>>>>> My initial idea is to credit them in the page with the rest of the
>>>>>>> sponsors in the new website:
>>>>>>> https://datapythonista.github.io/pandas-web/community/team.html#institutional-partners and
>>>>>>> also in the top right corner of the runnable code widgets (see for example
>>>>>>> where Binder is credited here: https://spacy.io/).
>>>>>>>
>>>>>>> What I'd like to ask is:
>>>>>>>
>>>>>>> 1. For the production website and docs (static content only, for the
>>>>>>> traffic we need):
>>>>>>> https://us.ovhcloud.com/products/public-cloud/object-storage
>>>>>>> 2. For our tools and processes, like the benchmarks, builds, CI
>>>>>>> stuff (temporary publish the docs for every PR,...):
>>>>>>> https://www.ovh.co.uk/vps/vps-ssd.xml (VPS SSD 3)
>>>>>>> 3. For BinderHub (runnable code in our docs, launch tutorials on
>>>>>>> Binder...): https://www.ovh.co.uk/public-cloud/kubernetes/
>>>>>>>
>>>>>>> For the BinderHub, QuantStack offered help with the set up (which is
>>>>>>> great, because I don't know much about Binder myself, and I'm not sure if
>>>>>>> anyone else does or wants to take care of this). I don't think it'll be
>>>>>>> easy to estimate how big is the cluster we need beforehand, but I guess we
>>>>>>> can add things to Binder iteratively, and have more info as we grow.
>>>>>>>
>>>>>>> OVH gave us a 200 euros voucher to experiment with the different
>>>>>>> services. Let me know how all this sounds, and if there are no objections,
>>>>>>> I'll create an account and buy those services with the voucher, and I'll
>>>>>>> start to prototype and see how everything works.
>>>>>>>
>>>>>>> Cheers!
>>>>>>>
>>>>>>> On Tue, Aug 20, 2019 at 11:06 PM Marc Garcia <garcia.marc at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Somehow related to the work on the new website (
>>>>>>>> https://github.com/pandas-dev/pandas/pull/28014), I've been
>>>>>>>> discussing with the Binder team, and looks like should be quite easy soon
>>>>>>>> (with a Sphinx extension) to make all the documentation pages runnable with
>>>>>>>> Binder, directly from the website (without opening the page as a Jupyter in
>>>>>>>> mybinder).
>>>>>>>>
>>>>>>>> While they are very happy with the idea of having this is pandas,
>>>>>>>> it's uncertain if the current infrastructure Binder has got, is able to
>>>>>>>> handle all the traffic we would send. And scikit-learn is working on it too
>>>>>>>> (today they added to the dev docs a link to mybinder to run the examples).
>>>>>>>>
>>>>>>>> I'm discussing with OVH (their infrastructure provider) on whether
>>>>>>>> they'd be happy to provide a dedicated BinderHub specific to pandas (or may
>>>>>>>> be we can have one for all NumFOCUS projects). We'll see how it goes, but
>>>>>>>> wanted to let you know, so you're updated, and in case anyone is interested
>>>>>>>> in participating in the discussions. Of course before any decision is made
>>>>>>>> I'll open a discussion here or on GitHub.
>>>>>>>>
>>>>>>>> As part of the discussion I'm also trying to get a server for the
>>>>>>>> website, and one for development stuff. Specfically for the dev docs
>>>>>>>> (including rendered docs of every PR) and the GitHub app that will generate
>>>>>>>> them. I guess it should be very easy to find a sponsor for these two
>>>>>>>> servers (in exchange of a small note in the footer of the website, or
>>>>>>>> something like that).
>>>>>>>>
>>>>>>>> Let me know if you have any comment, want to be involved or
>>>>>>>> whatever.
>>>>>>>>
>>>>>>>> Cheers!
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Pandas-dev mailing list
>>>>>>> Pandas-dev at python.org
>>>>>>> https://mail.python.org/mailman/listinfo/pandas-dev
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Andy R. Terrel, PhD
>>>>> President
>>>>> NumFOCUS
>>>>> andy at numfocus.org
>>>>>
>>>>

-- 
Andy R. Terrel, PhD
President
NumFOCUS
andy at numfocus.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pandas-dev/attachments/20190925/f43c46e2/attachment.html>