Return str to a callback raise a segfault if used in string formating

[Thomas Jollans]

On 2017-10-13 11:07, Vincent Vande Vyvre wrote:
> Le 13/10/17 à 09:23, Chris Angelico a écrit :
>> On Fri, Oct 13, 2017 at 4:46 PM, Vincent Vande Vyvre
>> <vincent.vande.vyvre at telenet.be> wrote:
>>> Simplified code:
>>>
>>> ---%<------------------------------------------
>>> ...
>>> ---%<------------------------------------------
>>>
>>> If I place self.callback() at the end of the func process that doesn't
>>> change anything.
>> First off, exactly what version of Python are you running this under?
>> With a segfault, you need to be pretty specific - platform, version,
>> word size, as much as you can gather.
>>
>> Secondly: Can you create a version of this that doesn't comment out
>> part of the work? If you can make a script where, any time you run it,
>> Python segfaults, that would be extremely helpful.
>>
>> Your code currently looks a bit weird. You create a thread, start it,
>> and then immediately wait for it (join()). When you add a task, if
>> it's the first task you've added, you process tasks, thus removing
>> that task. So the class isn't actually doing anything, and logically,
>> you could simply process the files directly in the loop. I'm guessing
>> that something in there (probably the threading) is causing the crash,
>> but without a complete and testable demo, it's hard to be sure.
>>
>> ChrisA
> 
> I'm using 3.6.1 in venv
> 
> It's not easy to write a runnable code because in this example the
> method process() is too simplified and the code don't reproduce the
> segfault.

That is what you'll have to do, though.
If you strip out as much as possible to create a minimal, working,
complete example, you will have a much easier time reasoning about what
is happening. This will also allow other people to reproduce the
problem, and help.

> The code is an image processing in two parts: the processing himself in
> CPython and an api in Python.

If you have custom C code, it's likely that there is a memory management
problem there. It's not unusual for incorrect memory management to cause
problems in a completely different part of the code, where something
tries to access freed memory or something.

If removing a call to C fixes it, I recommend you carefully check the
memory management logic of your C function.


> 
> I have written the two parts.
> 
> In the real code this is the equivalent of the method process():
> ----------------------------------------------
>     def unraw(self, index=0, dest=""):
>         """Run the demosaication process.
> 
>         Args:
>         index -- the index of the image or "all" if there's more than
> one image
>                  into the file
>         dest -- the absolute file name for the image decoded.  If a file
> with
>                 the same name already exists, it will be overwritten.
> 
>         Raise IndexError if index >= self.image_count
>         """
>         if not self.is_raw:
>             fname = os.path.basename(self.filename)
>             raise TypeError("RAW file %s not supported!" % fname)
> 
>         if index >= self.data["image_count"]:
>             raise IndexError("Index of image %s out of range(%s)"
>                              %(index, self.data["image_count"]))
> 
>         multi = 0
>         if index == "all":
>             multi = 1
>             index = 0
> 
>         if not dest:
>             dest = self.filename
> 
>         target = os.path.splitext(dest)[0]
>         res = self.raw.demosaicate(index, multi, target)    # This call
> the CPython code
>         self.out_filename = self.raw.out_file
> 
>     def run_multitasks(self):
>         def process(*args):
>             fname, idx, oname = args
>             self.identify()
>             self.unraw(idx, oname)
> 
>         while self.tasks:
>             t = Thread(target=process, args=self.tasks.pop(0))
>             t.start()
>             t.join()
>             if self.callback:
>                 self.callback(self.out_filename)
> ----------------------------------------------------------------
> 
> I use a list of 20 files and all files are converted, one by one,
> because the CPython part is not thread safe.
> 
> But the real interesting thing is the behaviour in the slot() function.
> 
> Examples:
>  1 -------------------------------
> def slot(name):
>     if os.path.isfile(name):
>         print("Created:", name, type(name))
>         print("%s created" % name)
> 
> result:
> add: /home/vincent/Images/RAW_orig/RAW_CANON_1DSM2.CR2
> Scaling with darkness 127, saturation 3712, and
> multipliers 2.310656 1.000000 1.257116 1.000000
> AHD interpolation...
> Converting to sRGB colorspace...
> Created:
> /home/vincent/CPython/py361_venv/pyunraw/unraws/RAW_CANON_1DSM2.tiff
> <class 'str'>
> Erreur de segmentation (core dumped)
> ----------------------------------
> 
>  2 --------------------------------
> def slot(name):
>     new = name[:]
>     if os.path.isfile(name):
>         print("Created:", name, type(name))
>         print("New:", new)
>         print("%s created" % new)
> 
> result:
> add /home/vincent/Images/RAW_orig/RAW_CANON_1DSM2.CR2
> Scaling with darkness 127, saturation 3712, and
> multipliers 2.310656 1.000000 1.257116 1.000000
> AHD interpolation...
> Converting to sRGB colorspace...
> Created:
> /home/vincent/CPython/py361_venv/pyunraw/unraws/RAW_CANON_1DSM2.tiff
> <class 'str'>
> New: /home/vincent/CPython/py361_venv/pyunraw/unraws/RAW_CANON_1DSM2.tiff
> Erreur de segmentation (core dumped)
> ----------------------------------
> 
> The most important, of course is the name of the new file created and I
> can be satisfied with that, I just need the file name, but it's not a
> reason to ignore the problem.
> 
> Vincent
>