[Cython] Utility Codes and templates

[mark florisson]

On 22 July 2011 16:31, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 22.07.2011 16:11:
>>
>> On 22 July 2011 15:54, Stefan Behnel wrote:
>>>
>>> mark florisson, 22.07.2011 15:07:
>>>>
>>>> On 22 July 2011 14:38, Stefan Behnel wrote:
>>>>>
>>>>> mark florisson, 22.07.2011 13:45:
>>>>>>
>>>>>> On 22 July 2011 13:10, Stefan Behnel wrote:
>>>>>>>
>>>>>>> mark florisson, 22.07.2011 12:12:
>>>>>>>>
>>>>>>>> For my work on the _memview branch (and also on fused types) I
>>>>>>>> noticed
>>>>>>>> that UtilityCodes started weighing heavily on me in their current
>>>>>>>> form, so I wrote a little loader in the _memview branch:
>>>>>>>>
>>>>>>>>
>>>>>>>> https://github.com/markflorisson88/cython/commit/e13debed2db78680ec0bd8c343433a2b73bd5e64#L2R110
>>>>>>>>
>>>>>>>> The idea is simple: you put your utility codes in Cython/Utility in
>>>>>>>> .pyx, .c, .h files etc, and then load them. It works for both
>>>>>>>> prototypes and implementations, for UtilityCode and
>>>>>>>> CythonUtilityCode:
>>>>>>>>
>>>>>>>> myutility.c
>>>>>>>>
>>>>>>>> // UtilityProto: MyUtility
>>>>>>>> header code here
>>>>>>>>
>>>>>>>> // UtilityCode: MyUtility
>>>>>>>> implementation code here
>>>>>>>>
>>>>>>>> You can add as many other utilities as you like to the same file.
>>>>>>>> You
>>>>>>>> can then load it using
>>>>>>>>
>>>>>>>>     UtilityCode.load_utility_from_file("myutility.c", "MyUtility")
>>>>>>>
>>>>>>> Why not have exactly one per file? They can't (or at least shouldn't)
>>>>>>> be
>>>>>>> interdependent anyway, since they're always loaded and injected
>>>>>>> separately.
>>>>>>> Having one per file makes it easy to take the file name and grep for
>>>>>>> it.
>>>>>>
>>>>>> Putting them in one file does not make them interdependent
>>>>>
>>>>> I meant the opposite direction. They shouldn't depend on each other
>>>>> anyway,
>>>>> so there's no need to put them in one file.
>>>>
>>>> Say we have one struct definition, and two utilities that depend on
>>>> it, but not directly on one another. I want all of them in the same
>>>> file, because they are *related*, just not all dependent on each
>>>> other.
>>>
>>> Ok, I see the use case now. How do you mark the dependencies within the
>>> file? And how do you mark dependencies towards other code in other files?
>>> I.e. how do you map the parameters that UtilityCode() current takes into
>>> the
>>> file format?
>>>
>>> You sure don't want to keep those in the Python code that injects (or
>>> loads)
>>> the utility code, do you?
>>
>> That's what I currently do. You load the utilities with the normal
>> arguments, except that the implementation and prototypes are
>> automatically fetched from a utility code file. So you do
>>
>> my_utility = UtilityCode.load("MyUtility", proto_block='...',
>> requires=[my_other_utility])
>
> Ah, ok, I hadn't noticed that in your commit. That doesn't make much sense.
> It's the implementation that knows what it depends on, not the code that
> uses it. It's not uncommon for utility code to be used in multiple places,
> it's bad to require all of them to agree on the same configuration for that
> code. That's why I'm saying that the external file should be self-contained
> in this regard. The dependencies must be explicit and right where the code
> is.
>
Point taken, it was mainly implemented this way because it was easiest
and still premature. I'll support at least the requires clause.

>> It wouldn't be hard to support requirements (and other options) in
>> utility code files though. Do you think we should?
>
> Sure. The code that injects utility code should only have to know its name
> and what cnames (functions etc.) it provides, and shouldn't have to know
> anything about its implementation details. Exactly as it works now.
>

This is about loading the utility code, not about injecting them. You
usually assign them to global variables, just like before.

>>>>>> You don't want to enforce one utility per file as many utilities
>>>>>> are pretty small, and you want to group utilities based on their
>>>>>> purpose.
>>>>>
>>>>> It substantially complicates the file handling, though.
>>>>
>>>> Complicates? In what way?
>>>
>>> In the sense that you have to do something to split the file into
>>> separate
>>> sections before being able to apply the templating engine to it. So you
>>> have
>>> several levels of marker semantics in the file format, one to separate
>>> the
>>> implementations, one to separate the parts of an implementation, and one
>>> level for the templating language. I call that "complicating the file
>>> handling".
>>>
>>> However, given that you always need to express metadata in some way (e.g.
>>> for dependencies), I don't think you can do better than with two levels
>>> anyway, so a third won't hurt *that* much.
>>
>> Actually, the prototype and the implementation do not need to be
>> grouped. i.e. it is valid to first list all prototypes and then all
>> implementations.
>
> I understand that, and it makes sense to support that.
>
>
>> So yeah, there would be two levels instead of one,
>
> Well, three instead of two. But I see how you want to mingle the third level
> into the second. That should work.
>
>
>> but that's what you want, because you want different parameters for
>> different utilities in the file, and you may load them at different
>> times or even multiple times with different parameters.
>
> Right, that's why it would be easier with separate files.

Unless you think passing in a filename is hard, separating the
utilities in different files doesn't make it easier in any way. For my
utilities I simply created a function load_memview_cy_utility() that
eats the name and parameters and loads it from the right file. I think
this two-line function is well-worth the convenience of the grouping.

> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>