A data transformation framework. A presentation inviting commentary.

[F.R.]

On 08/21/2013 06:29 PM, F.R. wrote:
> Hi all,
>
> In an effort to do some serious cleaning up of a hopelessly cluttered 
> working environment, I developed a modular data transformation system 
> that pretty much stands. I am very
. . . etc


Chris, Terry, Dieter, thanks for your suggestions.

Chris: If my Transformer looks like a function, that's because it is 
(__call__). My idea was to have something like an erector set of 
elementary transformation machines that can be assembled into chains. 
There may be some processing overhead in managing the data flow, but I'm 
not even sure of that, because the flow needs to be managed somehow and 
throwing one's stones into someone else's garden doesn't get rid of the 
stones. My idea was to simplify, generalize and automate in order to 
deal with the kind of overhead that matters most to me: my own mental 
overhead.

Terry: I am aware of the memory-load aspect. It is no constraint for the 
things I do. If it became one, I'd develop a translation assembly using 
a small data sample and when it reaches the stage of reliability, I'd 
add a line to have each Translator delete its input the moment it is 
done. I shall certainly look at itertools. Thanks for your suggestions 
and explanations.

Dieter: I wish I could respond to the points you raise. I am unfamiliar 
with the details and they don't seem like they can be looked up in five 
minutes. I do make a note of your thoughts.


Frederic