From josh at levykramer.co.uk Mon Nov 9 08:27:02 2020 From: josh at levykramer.co.uk (Josh Levy-Kramer) Date: Mon, 9 Nov 2020 13:27:02 +0000 Subject: [Pandas-dev] Pandas style guide and best practices Message-ID: Hi all, I'm currently writing a Pandas style guide and excited to share it with you. It's an accumulation of ideas I have picked up from 5+ years of using Pandas at work in a variety of projects. I would love to get feedback, opinion and contributions from the wider Pandas community. Please have a read: https://github.com/joshlk/pandas_style_guide Josh -------------- next part -------------- An HTML attachment was scrubbed... URL: From andy.terrel at gmail.com Mon Nov 16 10:58:41 2020 From: andy.terrel at gmail.com (Andy Ray Terrel) Date: Mon, 16 Nov 2020 09:58:41 -0600 Subject: [Pandas-dev] NASA Funding opportunity In-Reply-To: References: Message-ID: Hi all, The NASA grant will require a Notice of Intent submitted this Thursday. I think Pandas has a high likelihood of being accepted as it is right in line with the work being proposed. How should we proceed? A zoom call or some other mechanism? Really need to identify who would be funded and what activities would be proposed. Really seems they don't want new features per se but things that make the software more stable. -- Andy On Tue, Oct 20, 2020 at 12:02 PM Andy Ray Terrel wrote: > FWIW, NumFOCUS is running a few grants with NASA right now. I can > definitely help with a proposal. > > On Tue, Oct 20, 2020 at 9:45 AM Tom Augspurger > wrote: > >> Hi all, >> >> NASA has a new CFP that's focused on supporting existing open-source >> projects. >> https://nspires.nasaprs.com/external/solicitations/summary.do?solId=%7b958CF134-D655-E512-B5AD-84501D14A0C1%7d >> >> 1.1 Motivation >> Open source software tools, libraries, and frameworks play an >> increasingly prominent >> role in SMD-related science research and applications. As the adoption of >> open >> software accelerates the rate of scientific discovery, the National >> Academies? has >> recognized the growing need among the NASA science community to provide >> sustained >> support and maintenance of these types of software in their 2018 report >> Open Source >> Software Policy Options for NASA Earth and Space Sciences. This program >> is designed >> to provide support to the sustainable development of open source >> software, tools, >> libraries, and framework that are critical for SMD scientific objectives. >> >> 1.2 Objectives >> SMD seeks proposals for the improvement and sustainment of high-value, >> open source >> tools, frameworks, and libraries that have made significant impacts to >> the SMD science >> community. We are seeking proposals that satisfy the following objectives: >> ? Open source software tools, libraries, and frameworks that have >> significant >> usage in the NASA science community, developed following open and >> collaborative practices, and are aligned with the scientific vision and >> data >> strategic plan of SMD. >> ? Proposals should look to improve the sustainability and utility of >> these packages >> through improvements to adding extensions, documentation, infrastructure, >> and >> maintenance of the software. >> This program seeks to support projects under active development and >> usage, and it >> does not support updating of legacy software that is no longer supported, >> which can be >> supported under other calls. We are not soliciting the development of new >> open source >> tools, frameworks, or libraries with this call. >> >> I'm not sure exactly what our chances to get funded will be compared to >> projects more squarely in the science domain, but this may be worth >> pursing. Is anyone interested in collaborating on a proposal? >> >> Timeline, we'd file a notice of intent by November 19th, and the proposal >> by January 19th. >> >> Tom >> >> Tom >> >> _______________________________________________ >> Pandas-dev mailing list >> Pandas-dev at python.org >> https://mail.python.org/mailman/listinfo/pandas-dev >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shishaozhong at gmail.com Mon Nov 16 11:19:04 2020 From: shishaozhong at gmail.com (Shaozhong SHI) Date: Mon, 16 Nov 2020 16:19:04 +0000 Subject: [Pandas-dev] Loop over and add new columns Message-ID: Hi, I got a very strange error with such a simple loop. Can anyone explain why? x={u'brandId': u'BD398', u'brandName': u'BRAND Innomary', u'careHome': u'Y', u'constituency': u'Hastings and Rye', u'dormancy': u'N', u'inspectionAreas': [], u'inspectionDirectorate': u'Adult social care', u'localAuthority': u'East Sussex', u'locationId': u'1-1000210669', u'mainPhoneNumber': u'01424716303', u'name': u'Kingswood House Nursing Home', u'numberOfBeds': 22, u'odsCode': u'VM4G6', u'onspdCcgCode': u'E38000235', u'onspdCcgName': u'NHS East Sussex CCG', u'onspdLatitude': 50.857239, u'onspdLongitude': 0.561998, u'organisationType': u'Location', u'postalAddressCounty': u'East Sussex', u'postalAddressLine1': u'21-23 Chapel Park Road', u'postalAddressTownCity': u'St Leonards On Sea', u'postalCode': u'TN37 6HR', u'providerId': u'1-877912132', u'region': u'South East', u'registrationDate': u'2013-12-12', u'registrationStatus': u'Registered', u'type': u'Social Care Org', u'uprn': u'100062576733'} import pandas as pd my_df =pd.DataFrame() for col_name in x.items(): my_df[col_name]=x[col_name] my_df Much obliged. Regards, David -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Mon Nov 16 11:44:46 2020 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Mon, 16 Nov 2020 16:44:46 +0000 Subject: [Pandas-dev] NASA Funding opportunity In-Reply-To: References: Message-ID: On Mon, Nov 16, 2020 at 3:59 PM Andy Ray Terrel wrote: > Hi all, > > The NASA grant will require a Notice of Intent submitted this Thursday. I > think Pandas has a high likelihood of being accepted as it is right in line > with the work being proposed. How should we proceed? A zoom call or some > other mechanism? Really need to identify who would be funded and what > activities would be proposed. Really seems they don't want new features per > se but things that make the software more stable. > Agreed that this is a good fit. Jeff reached out to me and it looks like we'd want to aim for a combined NumPy, Pandas, SciPy proposal. We checked in with the program manager, and a combined proposal would be good - and save a lot of energy on proposal writing time. I was planning on giving the contents of the letter or intent a push today. Andy, we could use your help - experience with NASA will be valuable here. Want to collaborate on this? Cheers, Ralf > -- Andy > > On Tue, Oct 20, 2020 at 12:02 PM Andy Ray Terrel > wrote: > >> FWIW, NumFOCUS is running a few grants with NASA right now. I can >> definitely help with a proposal. >> >> On Tue, Oct 20, 2020 at 9:45 AM Tom Augspurger < >> tom.augspurger88 at gmail.com> wrote: >> >>> Hi all, >>> >>> NASA has a new CFP that's focused on supporting existing open-source >>> projects. >>> https://nspires.nasaprs.com/external/solicitations/summary.do?solId=%7b958CF134-D655-E512-B5AD-84501D14A0C1%7d >>> >>> 1.1 Motivation >>> Open source software tools, libraries, and frameworks play an >>> increasingly prominent >>> role in SMD-related science research and applications. As the adoption >>> of open >>> software accelerates the rate of scientific discovery, the National >>> Academies? has >>> recognized the growing need among the NASA science community to provide >>> sustained >>> support and maintenance of these types of software in their 2018 report >>> Open Source >>> Software Policy Options for NASA Earth and Space Sciences. This program >>> is designed >>> to provide support to the sustainable development of open source >>> software, tools, >>> libraries, and framework that are critical for SMD scientific objectives. >>> >>> 1.2 Objectives >>> SMD seeks proposals for the improvement and sustainment of high-value, >>> open source >>> tools, frameworks, and libraries that have made significant impacts to >>> the SMD science >>> community. We are seeking proposals that satisfy the following >>> objectives: >>> ? Open source software tools, libraries, and frameworks that have >>> significant >>> usage in the NASA science community, developed following open and >>> collaborative practices, and are aligned with the scientific vision and >>> data >>> strategic plan of SMD. >>> ? Proposals should look to improve the sustainability and utility of >>> these packages >>> through improvements to adding extensions, documentation, >>> infrastructure, and >>> maintenance of the software. >>> This program seeks to support projects under active development and >>> usage, and it >>> does not support updating of legacy software that is no longer >>> supported, which can be >>> supported under other calls. We are not soliciting the development of >>> new open source >>> tools, frameworks, or libraries with this call. >>> >>> I'm not sure exactly what our chances to get funded will be compared to >>> projects more squarely in the science domain, but this may be worth >>> pursing. Is anyone interested in collaborating on a proposal? >>> >>> Timeline, we'd file a notice of intent by November 19th, and the >>> proposal by January 19th. >>> >>> Tom >>> >>> Tom >>> >>> _______________________________________________ >>> Pandas-dev mailing list >>> Pandas-dev at python.org >>> https://mail.python.org/mailman/listinfo/pandas-dev >>> >> _______________________________________________ > Pandas-dev mailing list > Pandas-dev at python.org > https://mail.python.org/mailman/listinfo/pandas-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shishaozhong at gmail.com Mon Nov 16 11:51:48 2020 From: shishaozhong at gmail.com (Shaozhong SHI) Date: Mon, 16 Nov 2020 16:51:48 +0000 Subject: [Pandas-dev] Drop all columns which are not normal numeric nor text values Message-ID: After applying myDF = json_normalize(result) We get brandId brandName careHome constituency currentRatings.overall.keyQuestionRatings currentRatings.overall.rating currentRatings.overall.reportDate currentRatings.overall.reportLinkId currentRatings.reportDate dormancy ... providerId region registrationDate registrationStatus regulatedActivities relationships reports specialisms type uprn 0 BD510 BRAND MACC Care Y Birmingham, Northfield [{u'reportDate': u'2020-10-01', u'rating': u'R... Requires improvement 2020-10-01 1157c975-c2f1-423e-a2b4-66901779e014 2020-10-01 N ... 1-101641521 West Midlands 2013-12-16 Registered [{u'code': u'RA2', u'name': u'Accommodation fo... [] [{u'reportDate': u'2020-10-01', u'linkId': u'1... [{u'name': u'Caring for adults over 65 yrs'}, ... Social Care Org 100070537642 How to replace or drop all columns which are neither numeric nor text values? What is the fastest way? Regards, David -------------- next part -------------- An HTML attachment was scrubbed... URL: From andy.terrel at gmail.com Mon Nov 16 12:05:54 2020 From: andy.terrel at gmail.com (Andy Ray Terrel) Date: Mon, 16 Nov 2020 11:05:54 -0600 Subject: [Pandas-dev] NASA Funding opportunity In-Reply-To: References: Message-ID: On Mon, Nov 16, 2020 at 10:44 AM Ralf Gommers wrote: > > > On Mon, Nov 16, 2020 at 3:59 PM Andy Ray Terrel > wrote: > >> Hi all, >> >> The NASA grant will require a Notice of Intent submitted this Thursday. I >> think Pandas has a high likelihood of being accepted as it is right in line >> with the work being proposed. How should we proceed? A zoom call or some >> other mechanism? Really need to identify who would be funded and what >> activities would be proposed. Really seems they don't want new features per >> se but things that make the software more stable. >> > > Agreed that this is a good fit. > > Jeff reached out to me and it looks like we'd want to aim for a combined > NumPy, Pandas, SciPy proposal. We checked in with the program manager, and > a combined proposal would be good - and save a lot of energy on proposal > writing time. > > I was planning on giving the contents of the letter or intent a push > today. Andy, we could use your help - experience with NASA will be valuable > here. Want to collaborate on this? > That makes sense. Definitely let me know where I can be useful. -- Andy > > Cheers, > Ralf > > > >> -- Andy >> >> On Tue, Oct 20, 2020 at 12:02 PM Andy Ray Terrel >> wrote: >> >>> FWIW, NumFOCUS is running a few grants with NASA right now. I can >>> definitely help with a proposal. >>> >>> On Tue, Oct 20, 2020 at 9:45 AM Tom Augspurger < >>> tom.augspurger88 at gmail.com> wrote: >>> >>>> Hi all, >>>> >>>> NASA has a new CFP that's focused on supporting existing open-source >>>> projects. >>>> https://nspires.nasaprs.com/external/solicitations/summary.do?solId=%7b958CF134-D655-E512-B5AD-84501D14A0C1%7d >>>> >>>> 1.1 Motivation >>>> Open source software tools, libraries, and frameworks play an >>>> increasingly prominent >>>> role in SMD-related science research and applications. As the adoption >>>> of open >>>> software accelerates the rate of scientific discovery, the National >>>> Academies? has >>>> recognized the growing need among the NASA science community to provide >>>> sustained >>>> support and maintenance of these types of software in their 2018 report >>>> Open Source >>>> Software Policy Options for NASA Earth and Space Sciences. This program >>>> is designed >>>> to provide support to the sustainable development of open source >>>> software, tools, >>>> libraries, and framework that are critical for SMD scientific >>>> objectives. >>>> >>>> 1.2 Objectives >>>> SMD seeks proposals for the improvement and sustainment of high-value, >>>> open source >>>> tools, frameworks, and libraries that have made significant impacts to >>>> the SMD science >>>> community. We are seeking proposals that satisfy the following >>>> objectives: >>>> ? Open source software tools, libraries, and frameworks that have >>>> significant >>>> usage in the NASA science community, developed following open and >>>> collaborative practices, and are aligned with the scientific vision and >>>> data >>>> strategic plan of SMD. >>>> ? Proposals should look to improve the sustainability and utility of >>>> these packages >>>> through improvements to adding extensions, documentation, >>>> infrastructure, and >>>> maintenance of the software. >>>> This program seeks to support projects under active development and >>>> usage, and it >>>> does not support updating of legacy software that is no longer >>>> supported, which can be >>>> supported under other calls. We are not soliciting the development of >>>> new open source >>>> tools, frameworks, or libraries with this call. >>>> >>>> I'm not sure exactly what our chances to get funded will be compared to >>>> projects more squarely in the science domain, but this may be worth >>>> pursing. Is anyone interested in collaborating on a proposal? >>>> >>>> Timeline, we'd file a notice of intent by November 19th, and the >>>> proposal by January 19th. >>>> >>>> Tom >>>> >>>> Tom >>>> >>>> _______________________________________________ >>>> Pandas-dev mailing list >>>> Pandas-dev at python.org >>>> https://mail.python.org/mailman/listinfo/pandas-dev >>>> >>> _______________________________________________ >> Pandas-dev mailing list >> Pandas-dev at python.org >> https://mail.python.org/mailman/listinfo/pandas-dev >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tom.augspurger88 at gmail.com Mon Nov 16 14:18:06 2020 From: tom.augspurger88 at gmail.com (Tom Augspurger) Date: Mon, 16 Nov 2020 13:18:06 -0600 Subject: [Pandas-dev] Drop all columns which are not normal numeric nor text values In-Reply-To: References: Message-ID: <856410ED-2CBB-4F0B-BC75-AD4F487075B9@gmail.com> > On Nov 16, 2020, at 10:51 AM, Shaozhong SHI wrote: > > After applying myDF = json_normalize(result) > > We get > brandId brandName careHome constituency currentRatings.overall.keyQuestionRatings currentRatings.overall.rating currentRatings.overall.reportDate currentRatings.overall.reportLinkId currentRatings.reportDate dormancy ... providerId region registrationDate registrationStatus regulatedActivities relationships reports specialisms type uprn > 0 BD510 BRAND MACC Care Y Birmingham, Northfield [{u'reportDate': u'2020-10-01', u'rating': u'R... Requires improvement 2020-10-01 1157c975-c2f1-423e-a2b4-66901779e014 2020-10-01 N ... 1-101641521 West Midlands 2013-12-16 Registered [{u'code': u'RA2', u'name': u'Accommodation fo... [] [{u'reportDate': u'2020-10-01', u'linkId': u'1... [{u'name': u'Caring for adults over 65 yrs'}, ... Social Care Org 100070537642 > > > > > > > How to replace or drop all columns which are neither numeric nor text values? > > What is the fastest way? > > Regards, > > David Hi David, This mailing list is for pandas development. We recommend stack overflow for usage questions. Tom > _______________________________________________ > Pandas-dev mailing list > Pandas-dev at python.org > https://mail.python.org/mailman/listinfo/pandas-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeffreback at gmail.com Mon Nov 16 22:46:40 2020 From: jeffreback at gmail.com (Jeff Reback) Date: Mon, 16 Nov 2020 22:46:40 -0500 Subject: [Pandas-dev] NASA Funding opportunity In-Reply-To: References: Message-ID: happy to help - lmk what is needed > On Nov 16, 2020, at 12:06 PM, Andy Ray Terrel wrote: > > ? > > >> On Mon, Nov 16, 2020 at 10:44 AM Ralf Gommers wrote: >> >> >>> On Mon, Nov 16, 2020 at 3:59 PM Andy Ray Terrel wrote: >>> Hi all, >>> >>> The NASA grant will require a Notice of Intent submitted this Thursday. I think Pandas has a high likelihood of being accepted as it is right in line with the work being proposed. How should we proceed? A zoom call or some other mechanism? Really need to identify who would be funded and what activities would be proposed. Really seems they don't want new features per se but things that make the software more stable. >> >> Agreed that this is a good fit. >> >> Jeff reached out to me and it looks like we'd want to aim for a combined NumPy, Pandas, SciPy proposal. We checked in with the program manager, and a combined proposal would be good - and save a lot of energy on proposal writing time. >> >> I was planning on giving the contents of the letter or intent a push today. Andy, we could use your help - experience with NASA will be valuable here. Want to collaborate on this? > > That makes sense. Definitely let me know where I can be useful. > > -- Andy > >> >> Cheers, >> Ralf >> >> >>> >>> -- Andy >>> >>>> On Tue, Oct 20, 2020 at 12:02 PM Andy Ray Terrel wrote: >>>> FWIW, NumFOCUS is running a few grants with NASA right now. I can definitely help with a proposal. >>>> >>>>> On Tue, Oct 20, 2020 at 9:45 AM Tom Augspurger wrote: >>>>> Hi all, >>>>> >>>>> NASA has a new CFP that's focused on supporting existing open-source projects. https://nspires.nasaprs.com/external/solicitations/summary.do?solId=%7b958CF134-D655-E512-B5AD-84501D14A0C1%7d >>>>> >>>>> 1.1 Motivation >>>>> Open source software tools, libraries, and frameworks play an increasingly prominent >>>>> role in SMD-related science research and applications. As the adoption of open >>>>> software accelerates the rate of scientific discovery, the National Academies? has >>>>> recognized the growing need among the NASA science community to provide sustained >>>>> support and maintenance of these types of software in their 2018 report Open Source >>>>> Software Policy Options for NASA Earth and Space Sciences. This program is designed >>>>> to provide support to the sustainable development of open source software, tools, >>>>> libraries, and framework that are critical for SMD scientific objectives. >>>>> >>>>> 1.2 Objectives >>>>> SMD seeks proposals for the improvement and sustainment of high-value, open source >>>>> tools, frameworks, and libraries that have made significant impacts to the SMD science >>>>> community. We are seeking proposals that satisfy the following objectives: >>>>> ? Open source software tools, libraries, and frameworks that have significant >>>>> usage in the NASA science community, developed following open and >>>>> collaborative practices, and are aligned with the scientific vision and data >>>>> strategic plan of SMD. >>>>> ? Proposals should look to improve the sustainability and utility of these packages >>>>> through improvements to adding extensions, documentation, infrastructure, and >>>>> maintenance of the software. >>>>> This program seeks to support projects under active development and usage, and it >>>>> does not support updating of legacy software that is no longer supported, which can be >>>>> supported under other calls. We are not soliciting the development of new open source >>>>> tools, frameworks, or libraries with this call. >>>>> >>>>> I'm not sure exactly what our chances to get funded will be compared to projects more squarely in the science domain, but this may be worth pursing. Is anyone interested in collaborating on a proposal? >>>>> >>>>> Timeline, we'd file a notice of intent by November 19th, and the proposal by January 19th. >>>>> >>>>> Tom >>>>> >>>>> Tom >>>>> >>>>> _______________________________________________ >>>>> Pandas-dev mailing list >>>>> Pandas-dev at python.org >>>>> https://mail.python.org/mailman/listinfo/pandas-dev >>> _______________________________________________ >>> Pandas-dev mailing list >>> Pandas-dev at python.org >>> https://mail.python.org/mailman/listinfo/pandas-dev > _______________________________________________ > Pandas-dev mailing list > Pandas-dev at python.org > https://mail.python.org/mailman/listinfo/pandas-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From tom.augspurger88 at gmail.com Fri Nov 20 15:23:46 2020 From: tom.augspurger88 at gmail.com (Tom Augspurger) Date: Fri, 20 Nov 2020 14:23:46 -0600 Subject: [Pandas-dev] Additional Funding from CZI Message-ID: <26844403-A56C-4C38-B3BC-5FB5BA9142DF@gmail.com> Hi all, I wanted to share that pandas has received additional funding from CZI, as part of the 3rd round of their EOSS program. We wrote a post at describing the proposal: http://dev.pandas.io/pandas-blog/pandas-receives-czi-funding.html Tom -------------- next part -------------- An HTML attachment was scrubbed... URL: From arj.python at gmail.com Sat Nov 21 00:31:31 2020 From: arj.python at gmail.com (Abdur-Rahmaan Janhangeer) Date: Sat, 21 Nov 2020 09:31:31 +0400 Subject: [Pandas-dev] Faster .apply natively Message-ID: Greetings list, I would like to know if there are plans on working on a faster .apply . A normal NLP function of reducing a sentence to it's essential lowercase version in 10 lines of list-comprehension processing takes an eternity for the ten of thousands rows. There are some 6 solutions like pandarallel etc which allows faster processing. But any hints if there is ongoing works in that direction? Kind Regards, Abdur-Rahmaan Janhangeer about | blog github Mauritius -------------- next part -------------- An HTML attachment was scrubbed... URL: From me at pietrobattiston.it Sat Nov 21 13:50:23 2020 From: me at pietrobattiston.it (Pietro Battiston) Date: Sat, 21 Nov 2020 19:50:23 +0100 Subject: [Pandas-dev] Faster .apply natively In-Reply-To: References: Message-ID: <4046f5bfb2641ff31f8ab05d26a3a92ad04b5c8d.camel@pietrobattiston.it> Dear Abdur, I never tried pandarallel, so I don't know how it compares, but it's definitely possible to do a parallel .apply with 5 lines of code and the standard multiprocessing library: https://stackoverflow.com/a/29281494 Does this answer your question? Pietro Il giorno sab, 21/11/2020 alle 09.31 +0400, Abdur-Rahmaan Janhangeer ha scritto: > Greetings list, > > I would like to know if there are plans on working > on a faster .apply . A normal NLP function of > reducing a sentence to it's essential lowercase version > in 10 lines of list-comprehension processing takes an > eternity for the ten of thousands rows. > > There are some 6 solutions like pandarallel etc which > allows faster processing. But any hints if there is ongoing > works in that direction? > > > Kind Regards, > > Abdur-Rahmaan Janhangeer > about | blog > github > Mauritius > _______________________________________________ > Pandas-dev mailing list > Pandas-dev at python.org > https://mail.python.org/mailman/listinfo/pandas-dev From arj.python at gmail.com Mon Nov 23 01:04:00 2020 From: arj.python at gmail.com (Abdur-Rahmaan Janhangeer) Date: Mon, 23 Nov 2020 10:04:00 +0400 Subject: [Pandas-dev] Faster .apply natively In-Reply-To: <4046f5bfb2641ff31f8ab05d26a3a92ad04b5c8d.camel@pietrobattiston.it> References: <4046f5bfb2641ff31f8ab05d26a3a92ad04b5c8d.camel@pietrobattiston.it> Message-ID: Greetings, Thanks for your response. It's not about knowing how to parallelise. It's about suggesting the Pandas lib doing it natively. If the devs can find a way to enhance the option, it might be very nice! Kind Regards, Abdur-Rahmaan Janhangeer about | blog github Mauritius -------------- next part -------------- An HTML attachment was scrubbed... URL: From rth.yurchak at gmail.com Mon Nov 23 16:36:21 2020 From: rth.yurchak at gmail.com (Roman Yurchak) Date: Mon, 23 Nov 2020 22:36:21 +0100 Subject: [Pandas-dev] Faster .apply natively In-Reply-To: References: Message-ID: <91549b59-b18c-68e3-341a-7c1c8b3d6f4b@gmail.com> On 21/11/2020 06:31, Abdur-Rahmaan Janhangeer wrote: > A normal NLP function of > reducing a sentence to it's essential lowercase version > in 10 lines of list-comprehension processing takes an > eternity for the ten of thousands rows. Calling .apply on 10k rows has an overhead of a few ms as far as I can tell. If it takes much longer it means that the bottleneck is in your function. Then the question is more how to make that function faster, with the typical answer of optimizing it in Python, rewriting in a lower level language (Cython or maybe using numba), parallelization over rows or here possibly caching. See https://pandas.pydata.org/docs/user_guide/enhancingperf.html for more details. The .apply function cannot really make any arbitrary python functions faster, and even parallelization has its limits in pure Python. -- Roman From shishaozhong at gmail.com Mon Nov 23 18:08:10 2020 From: shishaozhong at gmail.com (Shaozhong SHI) Date: Mon, 23 Nov 2020 23:08:10 +0000 Subject: [Pandas-dev] Can a script developed in Jupyter Notebook be simply run as a .py file? Message-ID: Hi, Can a script developed in Jupyter Notebook be simply run as a .py file? Can one copy and paste the content of a notebook and run as a .py file, once it is fully developed? Regarcs, David -------------- next part -------------- An HTML attachment was scrubbed... URL: From shishaozhong at gmail.com Mon Nov 23 18:31:26 2020 From: shishaozhong at gmail.com (Shaozhong SHI) Date: Mon, 23 Nov 2020 23:31:26 +0000 Subject: [Pandas-dev] How to export Jupyter notebook to a .py file? Message-ID: We would like to save Jupyter notebook and run it as a .py file. However, it appears not to be that simple. I did the following, and got errors such as the following. python cqc.py Traceback (most recent call last): File "cqc.py", line 4, in import pandas as pd ImportError: No module named pandas What else should I ask our system administrator to do, so that we can simply save Jupyter Notebook content and run it as a .py file. Regards, David -------------- next part -------------- An HTML attachment was scrubbed... URL: From jorisvandenbossche at gmail.com Tue Nov 24 03:25:30 2020 From: jorisvandenbossche at gmail.com (Joris Van den Bossche) Date: Tue, 24 Nov 2020 09:25:30 +0100 Subject: [Pandas-dev] How to export Jupyter notebook to a .py file? In-Reply-To: References: Message-ID: Hi, This is a list for discussions related to the development of pandas. I would recommend to ask your question in a different forum (eg StackOverflow). Best, Joris On Tue, 24 Nov 2020 at 00:31, Shaozhong SHI wrote: > We would like to save Jupyter notebook and run it as a .py file. > > However, it appears not to be that simple. > > I did the following, and got errors such as the following. > > > python cqc.py > Traceback (most recent call last): > File "cqc.py", line 4, in > import pandas as pd > ImportError: No module named pandas > > > What else should I ask our system administrator to do, so that we can > simply save Jupyter Notebook content and run it as a .py file. > > Regards, > > David > _______________________________________________ > Pandas-dev mailing list > Pandas-dev at python.org > https://mail.python.org/mailman/listinfo/pandas-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: