[C++-sig] Pyste: problems with inheritance across headers.

[Prabhu Ramachandran]

Hi,

>>>>> "N" == nicodemus  <nicodemus at globalite.com.br> writes:

[snip bug details]

As I mentioned and Nicodemus confirmed here are the current problems.

    N> 1. exported_names is not working correctly, it depends on the
    N>    order that
    N> the exporters generate the code, and that order is basically
    N> randomical.

    N> 2. The order that the classes are being instantiated is not
    N>    correct.

    N> Yes, and here we have a hairy problem, more below.

[snip other details]

Nicodemus and I had a very productive discussion on IRC
(irc.debian.org/#pyste) and here is a brief summary of what we
discussed for the benefit of the list.

There are two problems as outlined above.  Nicodemus also mentioned
that in an earlier version of Pyste he'd parse all the headers and
store the resulting XML representation in memory and this would make
memory usage go up to even 600+MB.  So we essentially had three
problems.

 1. Need parsed header information but can't keep all of the XML file
    related information in memory due to excessive memory usage.

 2. Cannot group exporters via headers and need to generate all
    exported_names and only then generate the wrapper code.  If this
    is not done the class bases information is incorrect.

 3. Order of the class instantiation can potentially be incorrect.

Finally, incremental generation of wrapper code also needed to be
addressed.

As I mentioned and Nicodemus confirmed in his reply to me, problem 2
is easily solved by first generating all the exported_names.  This
problem is not tied to problem 1 since the exported_names come from
the Pyste interface files which are user generated and do not depend
on GCCXML.

The third problem was more complicated and after some discussion we
came up with what believe to be a decent solution.  In --multiple
mode, we generate the XML for each of the headers.  So for the purpose
of discussion we'll assume that all the XML files exist.

 1. Based on the known generated_names (that is obtained from the
    interface files) we parse the various XML files and generate the
    necessary code in the Single/MultipleCodeUnit for the respective
    classes, functions etc.  As each file is parsed and this code is
    generated memory is freed thus eliminating problem 1.

 2. Using the classexporters we get the class base information.  This
    is already known and infact this information is also refined
    thanks to the exported names.
 
 3. Order the execution of the Export_ClassName() function calls in
    the module initialization based on the class/base information and
    write this to the main.cpp.  This code might get a little
    complicated but should not be too hard.

The only trouble we saw with this approach was for the case without
--multiple.  This will require some modifications to the
SingleCodeUnit class but Nicodemus said this should not be a big
problem.  Essentially before saving the SingleCodeUnit we'd need to
order the methods correctly.  

As regards incremental generation generating the xml file is usually
quite a slow operation and the approach we have chosen to go in for is
to add two options and is best described with a simple example.

 pyste.py --xml-dir=xml --generate-xml file1.hpp file2.hpp ...

This command merely generates the xml data from the headers and puts
them inside the directory xml.

 pyste.py --xml-dir=xml --out=. --module=module --use-xml \
 file1.pyste  file2.pyste ...
 
This command uses the generated XML files, if some of them dont exist
it generates them.  Once the XML files are available it uses the
information in the interface files passed and then generates the
wrapper code and also a _main.cpp.

Nicodemus said he'd try and get the ordering of the module init
functions correct since he already has the base information working
correctly.  I'll try to get the --use-xml and --generate-xml options
working.

Thats about it.  If these bugs are resolved we hope that Pyste should
be quite useable.

cheers,
prabhu