[Tutor] Implicit passing of argument select functions being called

[avi.e.gross at gmail.com]

It would be helpful to understand a little more, Alexander.

Just FYI, although I know you are trying to make something simple for people, I hope you are aware of the sklearn pipeline as it may help you figure out how to make your own. Pandas has its own way too. And of course we have various ways to create a pipeline in TensorFlow and modules build on top of it like Keras and at least 4 others including some that look a bit like what you are showing. Your application may not be compatible and you want to avoid complexity but how others do things can give you ideas.

Your question though seems to be focused not on how to make a pipeline, but how you can evaluate one function after another and first load initial data and then pass that data as an argument to each successive function, replacing the data for each input with the output of the previous, and finally return the last output.

So, are you using existing functions or supplying your own?  If you want a generic data_filter to be available within all these functions, there seem to be quite a few ways you might do that if you control everything. For example, you could pass it as an argument to each function directly. Or have it available in some name space.

So could you clarify what exactly is not working now and perhaps give a concrete example? 

-----Original Message-----
From: Tutor <tutor-bounces+avi.e.gross=gmail.com at python.org> On Behalf Of alexander-rodis
Sent: Sunday, July 10, 2022 2:51 AM
To: tutor at python.org
Subject: [Tutor] Implicit passing of argument select functions being called

I'm working on a project, where accessibility is very important as it's addressed to non - specialists, who may even have no knowledge of coding of Python.

In a specific section, I've come up with this API to make data transformation pipeline:

make_pipeline(

     load_data(fpath,...),

     transform1(arg1,arg2,....),

     ....,

     transform2(arg1,arg2,....),

     transform3(arg1,arg2,....),

     transformN(arg1,arg2,....),

)

transformN are all callables (writing this in a functional style), decorated with a partial call in the background. make_pipeline just uses a forloop to call the functions

successively like so e = arg(e). Which transforms from the many available in the library I'm writting, are used will vary between uses. 
load_data returns a pandas

DataFrame. Some transforms may need to operate on sub-arrays. I've written functions (also decorated with a partial call) to select the sub array

and return it to its original shape.  The problem is, I currently have to pass the data filtering function explicitly as an argument to each function that needs it, but

this seems VERY error prone, even if I cautiously document it. I want the the filter function to be specified in one place and made automatically  available to all

transforms that need it. Something roughly equivalent to:

make_pipeline(

     load_data(fpath,...),

     transform1(arg1,arg2,....),

     ....,

     transform2(arg1,arg2,....),

     transform3(arg1,arg2,....),

     transformN(arg1,arg2,....),

     , data_filter = simple_filter(start=0,) )

I thought all aliases local to caller make_pipelines becomes automatically available to the called functions. This seems to work but only on some small toy examples, not in

this case. global is usually considered bad practice, so I'm trying to avoid it and I'm not using any classes so not OOP. I've also tried using inspect.signature to check if each

callable accepts a certain argument and pass it if that's the case, however this raises an "incorrect signature error" which I could find documented anywhere. I've also considered

passing it to all functions with a try/except and ignore thrown errors, but it seems this could also be error prone, namely it could catch other errors too. So my question is, is there an

elegant and Pythonic way to specify data_filter in only one place and implicitly pass it to all functions that need it without global or classes?


Thanks

_______________________________________________
Tutor maillist  -  Tutor at python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor