From skaller at internode.on.net Sat Feb 1 00:00:07 2020 From: skaller at internode.on.net (John Skaller2) Date: Sat, 1 Feb 2020 16:00:07 +1100 Subject: [Cython] linkage In-Reply-To: <6EBDBA6D-4C98-4DF2-AC00-609A5E462635@internode.on.net> References: <1CE7B193-D458-4C9F-8136-1D7645692DEB@internode.on.net> <9F9FA658-CA8E-4DB2-BC62-3D978D54B3A9@internode.on.net> <6E79F14A-BF42-4AA8-A030-5E5797D43765@internode.on.net> <4c347e76-3f4c-cb24-27f3-c6fc1a11cbd1@canterbury.ac.nz> <8D1932F6-3858-4F23-A075-5DF96B94C0E8@internode.on.net> <8F478954-6F00-4A14-A641-0D654FE60E24@internode.on.net> <07D8E733-EB80-4FA9-B0A0-0BD2ED744FE5@internode.on.net> <2129c6a8-add5-2a6c-8c10-40a3cdc3c2db@canterbury.ac.nz> <435F1F1C-5A6E-4607-89A3-9132E858E8B0@internode.on.net> <6EBDBA6D-4C98-4DF2-AC00-609A5E462635@internode.on.net> Message-ID: <7DFE51A9-AA9E-44F5-82DF-8FBB4C907B6B@internode.on.net> OMG. Python is moving backwards. Some people just have no understanding of tech. As of 3.8, extensions must not be dynamically linked to libpython. This doesn?t apply to Windows or MacOS because that?s the only way on those platforms. But Debian/Ubuntu was always wrong and now the error is being made canonical. If anyone here knows a way on Linux to fix this, with some sort of stub loader for example, I?d be interested. All my code is linked with visibility=default, and all dynamic loads use two level namespaces, i.e, the symbol table of a shared library being imported is only visible to the importer. The may be some impact on Cython, since its primary job is building Python extensions. BTW: its all due to a stupid bug in ld which links shared libraries without bothering to check external references are satisfiable. Until load time, maybe.. :-) ? John Skaller skaller at internode.on.net From greg.ewing at canterbury.ac.nz Sat Feb 1 04:00:52 2020 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 01 Feb 2020 22:00:52 +1300 Subject: [Cython] Status In-Reply-To: <69B56638-F69D-4499-B113-46F903B98068@internode.on.net> References: <9F9FA658-CA8E-4DB2-BC62-3D978D54B3A9@internode.on.net> <6E79F14A-BF42-4AA8-A030-5E5797D43765@internode.on.net> <4c347e76-3f4c-cb24-27f3-c6fc1a11cbd1@canterbury.ac.nz> <8D1932F6-3858-4F23-A075-5DF96B94C0E8@internode.on.net> <8F478954-6F00-4A14-A641-0D654FE60E24@internode.on.net> <07D8E733-EB80-4FA9-B0A0-0BD2ED744FE5@internode.on.net> <2129c6a8-add5-2a6c-8c10-40a3cdc3c2db@canterbury.ac.nz> <69B56638-F69D-4499-B113-46F903B98068@internode.on.net> Message-ID: <2e251d30-97d5-78e3-c856-8f64f38002f1@canterbury.ac.nz> On 1/02/20 3:29 pm, John Skaller2 wrote: > But the all hell breaks loose for pointers. Your hack only > works for rvalues. Yes, using these it's possible for Cython to accept something that will later be rejected by the C compiler. It's not perfect, but it gets the job done most of the time. -- Greg From stefan_ml at behnel.de Sat Feb 1 04:22:30 2020 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 1 Feb 2020 10:22:30 +0100 Subject: [Cython] Size of output In-Reply-To: References: <6E79F14A-BF42-4AA8-A030-5E5797D43765@internode.on.net> <4c347e76-3f4c-cb24-27f3-c6fc1a11cbd1@canterbury.ac.nz> <8D1932F6-3858-4F23-A075-5DF96B94C0E8@internode.on.net> <8F478954-6F00-4A14-A641-0D654FE60E24@internode.on.net> <07D8E733-EB80-4FA9-B0A0-0BD2ED744FE5@internode.on.net> <2129c6a8-add5-2a6c-8c10-40a3cdc3c2db@canterbury.ac.nz> <435F1F1C-5A6E-4607-89A3-9132E858E8B0@internode.on.net> Message-ID: Robert Bradshaw schrieb am 01.02.20 um 00:49: > Taking a quick glance at an auto-generated file for an empty .pyx, we have > [...] > ~300 lines module setup code. Even for trivial modules, we still > declare and call functions for creating globals, preparing types, etc. > even if we don't have any globals, types, etc. :) > I agree there's some fat that could be trimmed there, but not sure > it'd be worth the effort. +1 In any non-trivial module where the size really starts to matter, the portions of unused code that we generate will probably make no difference at all. I'll also add that it can sometimes be very difficult to avoid generating unused code due to target (CPython x.y, PyPy, ?) specific utility code that comes with its own dependencies, which then may or may not be used on a given target. Removing the final 0.5% of code that is "unused in some cases" is really not something I would want to invest days into, each time we make a change in Cython. Stefan From skaller at internode.on.net Sat Feb 1 09:32:47 2020 From: skaller at internode.on.net (John Skaller2) Date: Sun, 2 Feb 2020 01:32:47 +1100 Subject: [Cython] Size of output In-Reply-To: References: <6E79F14A-BF42-4AA8-A030-5E5797D43765@internode.on.net> <4c347e76-3f4c-cb24-27f3-c6fc1a11cbd1@canterbury.ac.nz> <8D1932F6-3858-4F23-A075-5DF96B94C0E8@internode.on.net> <8F478954-6F00-4A14-A641-0D654FE60E24@internode.on.net> <07D8E733-EB80-4FA9-B0A0-0BD2ED744FE5@internode.on.net> <2129c6a8-add5-2a6c-8c10-40a3cdc3c2db@canterbury.ac.nz> <435F1F1C-5A6E-4607-89A3-9132E858E8B0@internode.on.net> Message-ID: <4B6E2D96-FD7A-47D3-87A2-002E2F1D3EB4@internode.on.net> > given target. Removing the final 0.5% of code that is "unused in some > cases" is really not something I would want to invest days into, each time > we make a change in Cython. But its not 0.5%. My problem is trying to read the generated code. A possibility might be an option which puts the boilerplate in one file and the variant code in another that then #includes the boilerplate. So the code is the same, but the main file is much easier to inspect to see what got generated. ? John Skaller skaller at internode.on.net From skaller at internode.on.net Sat Feb 1 09:38:20 2020 From: skaller at internode.on.net (John Skaller2) Date: Sun, 2 Feb 2020 01:38:20 +1100 Subject: [Cython] Status In-Reply-To: <2e251d30-97d5-78e3-c856-8f64f38002f1@canterbury.ac.nz> References: <9F9FA658-CA8E-4DB2-BC62-3D978D54B3A9@internode.on.net> <6E79F14A-BF42-4AA8-A030-5E5797D43765@internode.on.net> <4c347e76-3f4c-cb24-27f3-c6fc1a11cbd1@canterbury.ac.nz> <8D1932F6-3858-4F23-A075-5DF96B94C0E8@internode.on.net> <8F478954-6F00-4A14-A641-0D654FE60E24@internode.on.net> <07D8E733-EB80-4FA9-B0A0-0BD2ED744FE5@internode.on.net> <2129c6a8-add5-2a6c-8c10-40a3cdc3c2db@canterbury.ac.nz> <69B56638-F69D-4499-B113-46F903B98068@internode.on.net> <2e251d30-97d5-78e3-c856-8f64f38002f1@canterbury.ac.nz> Message-ID: > On 1 Feb 2020, at 20:00, Greg Ewing wrote: > > On 1/02/20 3:29 pm, John Skaller2 wrote: >> But the all hell breaks loose for pointers. Your hack only >> works for rvalues. > > Yes, using these it's possible for Cython to accept something > that will later be rejected by the C compiler. It's not perfect, > but it gets the job done most of the time. My concern is that the C compiler wont reject it, the program will corrupt data or crash: int32_t *x=..; *x = 42; int64_t *x=..; *x = 42; The C compiler will overwrite 4 or 8 bytes. Which one matters. In an rvalue context it probably doesn?t matter, in an lvalue context it does. But probably this doesn?t happen in Cython. ? John Skaller skaller at internode.on.net From stefan_ml at behnel.de Sat Feb 1 10:23:22 2020 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 1 Feb 2020 16:23:22 +0100 Subject: [Cython] Size of output In-Reply-To: <4B6E2D96-FD7A-47D3-87A2-002E2F1D3EB4@internode.on.net> References: <6E79F14A-BF42-4AA8-A030-5E5797D43765@internode.on.net> <4c347e76-3f4c-cb24-27f3-c6fc1a11cbd1@canterbury.ac.nz> <8D1932F6-3858-4F23-A075-5DF96B94C0E8@internode.on.net> <8F478954-6F00-4A14-A641-0D654FE60E24@internode.on.net> <07D8E733-EB80-4FA9-B0A0-0BD2ED744FE5@internode.on.net> <2129c6a8-add5-2a6c-8c10-40a3cdc3c2db@canterbury.ac.nz> <435F1F1C-5A6E-4607-89A3-9132E858E8B0@internode.on.net> <4B6E2D96-FD7A-47D3-87A2-002E2F1D3EB4@internode.on.net> Message-ID: <74f98ec4-fe6e-169a-e699-9462e83f2943@behnel.de> John Skaller2 schrieb am 01.02.20 um 15:32: > My problem is trying to read the generated code. You might be looking for "cython -a". Stefan From stefan_ml at behnel.de Sat Feb 1 10:26:49 2020 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 1 Feb 2020 16:26:49 +0100 Subject: [Cython] Status In-Reply-To: References: <6E79F14A-BF42-4AA8-A030-5E5797D43765@internode.on.net> <4c347e76-3f4c-cb24-27f3-c6fc1a11cbd1@canterbury.ac.nz> <8D1932F6-3858-4F23-A075-5DF96B94C0E8@internode.on.net> <8F478954-6F00-4A14-A641-0D654FE60E24@internode.on.net> <07D8E733-EB80-4FA9-B0A0-0BD2ED744FE5@internode.on.net> <2129c6a8-add5-2a6c-8c10-40a3cdc3c2db@canterbury.ac.nz> <69B56638-F69D-4499-B113-46F903B98068@internode.on.net> <2e251d30-97d5-78e3-c856-8f64f38002f1@canterbury.ac.nz> Message-ID: <6c3f45a6-6240-a678-ecef-df4ed20120c0@behnel.de> John Skaller2 schrieb am 01.02.20 um 15:38: > On 1 Feb 2020, at 20:00, Greg Ewing wrote: >> On 1/02/20 3:29 pm, John Skaller2 wrote: >>> But the all hell breaks loose for pointers. Your hack only >>> works for rvalues. >> >> Yes, using these it's possible for Cython to accept something >> that will later be rejected by the C compiler. It's not perfect, >> but it gets the job done most of the time. > > My concern is that the C compiler wont reject it, the program > will corrupt data or crash: > > int32_t *x=..; *x = 42; > int64_t *x=..; *x = 42; > > The C compiler will overwrite 4 or 8 bytes. Which one matters. As stated before, the C compiler will see the correct definition of the types in the header files. And the generated C code will use the same type names that the user had in their source code. "int64_t" will not magically become "int" there. Stefan From skaller at internode.on.net Sat Feb 1 10:41:11 2020 From: skaller at internode.on.net (John Skaller2) Date: Sun, 2 Feb 2020 02:41:11 +1100 Subject: [Cython] Size of output In-Reply-To: <74f98ec4-fe6e-169a-e699-9462e83f2943@behnel.de> References: <6E79F14A-BF42-4AA8-A030-5E5797D43765@internode.on.net> <4c347e76-3f4c-cb24-27f3-c6fc1a11cbd1@canterbury.ac.nz> <8D1932F6-3858-4F23-A075-5DF96B94C0E8@internode.on.net> <8F478954-6F00-4A14-A641-0D654FE60E24@internode.on.net> <07D8E733-EB80-4FA9-B0A0-0BD2ED744FE5@internode.on.net> <2129c6a8-add5-2a6c-8c10-40a3cdc3c2db@canterbury.ac.nz> <435F1F1C-5A6E-4607-89A3-9132E858E8B0@internode.on.net> <4B6E2D96-FD7A-47D3-87A2-002E2F1D3EB4@internode.on.net> <74f98ec4-fe6e-169a-e699-9462e83f2943@behnel.de> Message-ID: <635AF890-2F00-4FC0-B342-EBA327404A80@internode.on.net> > On 2 Feb 2020, at 02:23, Stefan Behnel wrote: > > John Skaller2 schrieb am 01.02.20 um 15:32: >> My problem is trying to read the generated code. > > You might be looking for "cython -a". OK. I see these: ~/felix>../cython/cython.py usage: cython.py [-h] [-V] [-l] [-I INCLUDE_PATH] [-o OUTPUT_FILE] [-t] [-f] [-v] [-p] [--cleanup GENERATE_CLEANUP_CODE] [-w WORKING_PATH] [--gdb] [--gdb-outdir GDB_OUTDIR] [-D] [-a] [--annotate-fullc] [--annotate-coverage ANNOTATE_COVERAGE_XML] [--line-directives] [-+] [--embed] [-2] [-3] [--3str] [--lenient] [--capi-reexport-cincludes] [--fast-fail] [-Werror] [-Wextra] [-X NAME=VALUE,...] [-E NAME=VALUE,...] [sources [sources ...]] cython.py: error: cython: Need at least one source file What does -a do? I couldn?t find complete docs on the compiler switches although some are explained in the user guide. ? John Skaller skaller at internode.on.net From skaller at internode.on.net Sat Feb 1 10:45:21 2020 From: skaller at internode.on.net (John Skaller2) Date: Sun, 2 Feb 2020 02:45:21 +1100 Subject: [Cython] Status In-Reply-To: <6c3f45a6-6240-a678-ecef-df4ed20120c0@behnel.de> References: <6E79F14A-BF42-4AA8-A030-5E5797D43765@internode.on.net> <4c347e76-3f4c-cb24-27f3-c6fc1a11cbd1@canterbury.ac.nz> <8D1932F6-3858-4F23-A075-5DF96B94C0E8@internode.on.net> <8F478954-6F00-4A14-A641-0D654FE60E24@internode.on.net> <07D8E733-EB80-4FA9-B0A0-0BD2ED744FE5@internode.on.net> <2129c6a8-add5-2a6c-8c10-40a3cdc3c2db@canterbury.ac.nz> <69B56638-F69D-4499-B113-46F903B98068@internode.on.net> <2e251d30-97d5-78e3-c856-8f64f38002f1@canterbury.ac.nz> <6c3f45a6-6240-a678-ecef-df4ed20120c0@behnel.de> Message-ID: <19C54494-142B-4663-9522-71E5B1B2594B@internode.on.net> >> >> My concern is that the C compiler wont reject it, the program >> will corrupt data or crash: >> >> int32_t *x=..; *x = 42; >> int64_t *x=..; *x = 42; >> >> The C compiler will overwrite 4 or 8 bytes. Which one matters. > > As stated before, the C compiler will see the correct definition of the > types in the header files. And the generated C code will use the same type > names that the user had in their source code. "int64_t" will not magically > become "int" there. Ah. Ok, I get it now. Thanks. The ?int? is just telling Cython the type has the same operations as ?int?, i.e. +, - *, / etc. Like I said, a poor mans version of a ?type class?. But, the compiler emits the symbols the user wrote when required. ? John Skaller skaller at internode.on.net From nathan.goldbaum at gmail.com Sat Feb 1 11:01:41 2020 From: nathan.goldbaum at gmail.com (Nathan) Date: Sat, 1 Feb 2020 09:01:41 -0700 Subject: [Cython] Size of output In-Reply-To: <635AF890-2F00-4FC0-B342-EBA327404A80@internode.on.net> References: <6E79F14A-BF42-4AA8-A030-5E5797D43765@internode.on.net> <4c347e76-3f4c-cb24-27f3-c6fc1a11cbd1@canterbury.ac.nz> <8D1932F6-3858-4F23-A075-5DF96B94C0E8@internode.on.net> <8F478954-6F00-4A14-A641-0D654FE60E24@internode.on.net> <07D8E733-EB80-4FA9-B0A0-0BD2ED744FE5@internode.on.net> <2129c6a8-add5-2a6c-8c10-40a3cdc3c2db@canterbury.ac.nz> <435F1F1C-5A6E-4607-89A3-9132E858E8B0@internode.on.net> <4B6E2D96-FD7A-47D3-87A2-002E2F1D3EB4@internode.on.net> <74f98ec4-fe6e-169a-e699-9462e83f2943@behnel.de> <635AF890-2F00-4FC0-B342-EBA327404A80@internode.on.net> Message-ID: Interesting it?s not documented well, it definitely should be. It corresponds to this compiler option: https://cython.readthedocs.io/en/latest/src/userguide/source_files_and_compilation.html#Cython.Compiler.Options.annotate When you turn that option on or pass -a or ?annotate to the compiler, cython generates a sidecar html file you can open in a browswer. The document initially just contains the cython source code, with each line highlighted shades of yellow to indicate the amount of python overhead that line invokes. You can also click on each line to see the corresponding generated C. It?s very helpful to understand how changes in cython syntax or even compiler options can influence the performance and level of safety of the generated C code (usually in an inverse relationship). On Sat, Feb 1, 2020 at 8:41 AM John Skaller2 wrote: > > > > On 2 Feb 2020, at 02:23, Stefan Behnel wrote: > > > > John Skaller2 schrieb am 01.02.20 um 15:32: > >> My problem is trying to read the generated code. > > > > You might be looking for "cython -a". > > OK. I see these: > > ~/felix>../cython/cython.py > usage: cython.py [-h] [-V] [-l] [-I INCLUDE_PATH] [-o OUTPUT_FILE] [-t] > [-f] > [-v] [-p] [--cleanup GENERATE_CLEANUP_CODE] [-w > WORKING_PATH] > [--gdb] [--gdb-outdir GDB_OUTDIR] [-D] [-a] > [--annotate-fullc] > [--annotate-coverage ANNOTATE_COVERAGE_XML] > [--line-directives] [-+] [--embed] [-2] [-3] [--3str] > [--lenient] [--capi-reexport-cincludes] [--fast-fail] > [-Werror] [-Wextra] [-X NAME=VALUE,...] [-E > NAME=VALUE,...] > [sources [sources ...]] > cython.py: error: cython: Need at least one source file > > What does -a do? I couldn?t find complete docs on the compiler switches > although some are explained in the user guide. > > ? > John Skaller > skaller at internode.on.net > > > > > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From newt0311 at gmail.com Sat Feb 1 11:06:28 2020 From: newt0311 at gmail.com (Prakhar Goel) Date: Sat, 1 Feb 2020 11:06:28 -0500 Subject: [Cython] Size of output In-Reply-To: References: <6E79F14A-BF42-4AA8-A030-5E5797D43765@internode.on.net> <4c347e76-3f4c-cb24-27f3-c6fc1a11cbd1@canterbury.ac.nz> <8D1932F6-3858-4F23-A075-5DF96B94C0E8@internode.on.net> <8F478954-6F00-4A14-A641-0D654FE60E24@internode.on.net> <07D8E733-EB80-4FA9-B0A0-0BD2ED744FE5@internode.on.net> <2129c6a8-add5-2a6c-8c10-40a3cdc3c2db@canterbury.ac.nz> <435F1F1C-5A6E-4607-89A3-9132E858E8B0@internode.on.net> <4B6E2D96-FD7A-47D3-87A2-002E2F1D3EB4@internode.on.net> <74f98ec4-fe6e-169a-e699-9462e83f2943@behnel.de> <635AF890-2F00-4FC0-B342-EBA327404A80@internode.on.net> Message-ID: Isn't it? https://cython.readthedocs.io/en/latest/src/tutorial/cython_tutorial.html#primes: Jump down to the part where it has "annotate=True" and it describes it in some detail. Also this takes all of thirty-seconds to try out and the result is pretty self explanatory. -- ________________________ Warm Regards Prakhar Goel From skaller at internode.on.net Sat Feb 1 12:28:03 2020 From: skaller at internode.on.net (John Skaller2) Date: Sun, 2 Feb 2020 04:28:03 +1100 Subject: [Cython] Size of output In-Reply-To: References: <6E79F14A-BF42-4AA8-A030-5E5797D43765@internode.on.net> <4c347e76-3f4c-cb24-27f3-c6fc1a11cbd1@canterbury.ac.nz> <8D1932F6-3858-4F23-A075-5DF96B94C0E8@internode.on.net> <8F478954-6F00-4A14-A641-0D654FE60E24@internode.on.net> <07D8E733-EB80-4FA9-B0A0-0BD2ED744FE5@internode.on.net> <2129c6a8-add5-2a6c-8c10-40a3cdc3c2db@canterbury.ac.nz> <435F1F1C-5A6E-4607-89A3-9132E858E8B0@internode.on.net> <4B6E2D96-FD7A-47D3-87A2-002E2F1D3EB4@internode.on.net> <74f98ec4-fe6e-169a-e699-9462e83f2943@behnel.de> <635AF890-2F00-4FC0-B342-EBA327404A80@internode.on.net> Message-ID: <90B24365-D74A-47FF-8AB7-9244C5372534@internode.on.net> > On 2 Feb 2020, at 03:06, Prakhar Goel wrote: > > Isn't it? https://cython.readthedocs.io/en/latest/src/tutorial/cython_tutorial.html#primes: > Jump down to the part where it has "annotate=True" and it describes it > in some detail. > > Also this takes all of thirty-seconds to try out and the result is > pretty self explanatory. Ah, I missed that. The options are documented. What?s missing is the heading. The documentation is under ?Jupyter Notebook?, whatever that is. I may have skipped reading that since I have no idea what it is. After that there is a heading, Compiler Options, but that actually refers to options set in setup.py rather than command line switches. https://cython.readthedocs.io/en/latest/src/userguide/source_files_and_compilation.html#Cython.Compiler.Options.annotate In any case whilst useful it?s not really what I would have liked to see, which is a file with just the ?yellow? code in it, and no boilerplate. Be good to suppress the refnanny stuff too. Is that possible? That seems to be for debugging, yes? ? John Skaller skaller at internode.on.net From robertwb at gmail.com Sun Feb 2 02:19:19 2020 From: robertwb at gmail.com (Robert Bradshaw) Date: Sat, 1 Feb 2020 23:19:19 -0800 Subject: [Cython] Size of output In-Reply-To: <4B6E2D96-FD7A-47D3-87A2-002E2F1D3EB4@internode.on.net> References: <6E79F14A-BF42-4AA8-A030-5E5797D43765@internode.on.net> <4c347e76-3f4c-cb24-27f3-c6fc1a11cbd1@canterbury.ac.nz> <8D1932F6-3858-4F23-A075-5DF96B94C0E8@internode.on.net> <8F478954-6F00-4A14-A641-0D654FE60E24@internode.on.net> <07D8E733-EB80-4FA9-B0A0-0BD2ED744FE5@internode.on.net> <2129c6a8-add5-2a6c-8c10-40a3cdc3c2db@canterbury.ac.nz> <435F1F1C-5A6E-4607-89A3-9132E858E8B0@internode.on.net> <4B6E2D96-FD7A-47D3-87A2-002E2F1D3EB4@internode.on.net> Message-ID: On Sat, Feb 1, 2020 at 6:33 AM John Skaller2 wrote: > > > given target. Removing the final 0.5% of code that is "unused in some > > cases" is really not something I would want to invest days into, each time > > we make a change in Cython. > > But its not 0.5%. My problem is trying to read the generated code. > > A possibility might be an option which puts the boilerplate in one file > and the variant code in another that then #includes the boilerplate. > So the code is the same, but the main file is much easier to inspect > to see what got generated. The primary target of the generated code is a C compiler, though we do try to generate code that's reasonably comprehensible for a human as well. However, there's often a tradeoff to be made in terms of conciseness and readability of the generated C code and that of the Cython codebase itself, and the latter is arguably more important. Also, as mentioned earlier, the generated code also makes heavy use of macros (including much of the refnanny stuff) so that various decisions can be deferred to C compile time. I'm also sure there's room for improvement, but that might not be the highest priority for where to spend limited time. Hopefully the -a option is useful. - Robert From dw-git at d-woods.co.uk Thu Feb 6 07:51:13 2020 From: dw-git at d-woods.co.uk (da-woods) Date: Thu, 6 Feb 2020 12:51:13 +0000 Subject: [Cython] Optimizing fastcall Message-ID: Related to https://github.com/cython/cython/issues/3343 One issue with the current code generated for fastcall function is that for a function like: def f(a, *args, **kwds): ??? # something... the wrapper function immediately generates a tuple for `args` and a dict for `kwds`, which presumably misses most of the benefits of the more efficient calling method. It therefore seems like a good idea to have an optional optimization so that the more efficient representation can be used (close to) directly. I've created an initial attempt at it https://github.com/cython/cython/pull/3346. This defines a complier directive so that: @cython.fastcall_args("*" or "**" or "both") def function... is converted to a relatively simple structure type. The tuple equivalent is typedef struct { ??? PyObject *const *args; ??? Py_ssize_t nargs; } __Pyx_FastcallTuple_obj; and the dict equivalent a little more complicated is typedef struct { ??? PyObject *const *args; // start of the keyword args values ??? PyObject *object;????? // either a dict, a tuple or NULL } __Pyx_FastcallDict_obj; // exists in one of three states: // * args is NULL, "object" is NULL, meaning no keyword arguments // * args is NULL, in which case "object" is actually a dict, and this just defers to the dict methods // * args is non-null, kwnames is a tuple These then support simple operations (indexing, iterating, passing directly to a fastcall function call etc). For most non-simple operations they will just be coerced to a Python object with a warning (but trying to put them in a closure will explicitly fail). ------------------------------------- I suspect my proposed code is still a bit of a mess, so I'm not looking for too much scrutiny of that right now. What I'm interested in is: * is the syntax to use this sensible? (it's probably easily changed) * is the level of warnings appropriate? * any operations that I've missed that'd be sensible to support? * anything else? David -------------- next part -------------- An HTML attachment was scrubbed... URL: