Return str to a callback raise a segfault if used in string formating

[Paul Moore]

As a specific suggestion, I assume the name of the created file is a
string object constructed in the C extension code, somehow. The fact
that you're getting the segfault with some uses of that string
(specifically, passing it to %-formatting) suggests that there's a bug
in the C code that constructs that string. That's where I'd start by
looking. Maybe something isn't zero-terminated that should be? Maybe
your code doesn't set up the character encoding information correctly?

Paul

On 13 October 2017 at 11:15, Thomas Jollans <tjol at tjol.eu> wrote:
> On 2017-10-13 11:07, Vincent Vande Vyvre wrote:
>> Le 13/10/17 à 09:23, Chris Angelico a écrit :
>>> On Fri, Oct 13, 2017 at 4:46 PM, Vincent Vande Vyvre
>>> <vincent.vande.vyvre at telenet.be> wrote:
>>>> Simplified code:
>>>>
>>>> ---%<------------------------------------------
>>>> ...
>>>> ---%<------------------------------------------
>>>>
>>>> If I place self.callback() at the end of the func process that doesn't
>>>> change anything.
>>> First off, exactly what version of Python are you running this under?
>>> With a segfault, you need to be pretty specific - platform, version,
>>> word size, as much as you can gather.
>>>
>>> Secondly: Can you create a version of this that doesn't comment out
>>> part of the work? If you can make a script where, any time you run it,
>>> Python segfaults, that would be extremely helpful.
>>>
>>> Your code currently looks a bit weird. You create a thread, start it,
>>> and then immediately wait for it (join()). When you add a task, if
>>> it's the first task you've added, you process tasks, thus removing
>>> that task. So the class isn't actually doing anything, and logically,
>>> you could simply process the files directly in the loop. I'm guessing
>>> that something in there (probably the threading) is causing the crash,
>>> but without a complete and testable demo, it's hard to be sure.
>>>
>>> ChrisA
>>
>> I'm using 3.6.1 in venv
>>
>> It's not easy to write a runnable code because in this example the
>> method process() is too simplified and the code don't reproduce the
>> segfault.
>
> That is what you'll have to do, though.
> If you strip out as much as possible to create a minimal, working,
> complete example, you will have a much easier time reasoning about what
> is happening. This will also allow other people to reproduce the
> problem, and help.
>
>> The code is an image processing in two parts: the processing himself in
>> CPython and an api in Python.
>
> If you have custom C code, it's likely that there is a memory management
> problem there. It's not unusual for incorrect memory management to cause
> problems in a completely different part of the code, where something
> tries to access freed memory or something.
>
> If removing a call to C fixes it, I recommend you carefully check the
> memory management logic of your C function.
>
>
>>
>> I have written the two parts.
>>
>> In the real code this is the equivalent of the method process():
>> ----------------------------------------------
>>     def unraw(self, index=0, dest=""):
>>         """Run the demosaication process.
>>
>>         Args:
>>         index -- the index of the image or "all" if there's more than
>> one image
>>                  into the file
>>         dest -- the absolute file name for the image decoded.  If a file
>> with
>>                 the same name already exists, it will be overwritten.
>>
>>         Raise IndexError if index >= self.image_count
>>         """
>>         if not self.is_raw:
>>             fname = os.path.basename(self.filename)
>>             raise TypeError("RAW file %s not supported!" % fname)
>>
>>         if index >= self.data["image_count"]:
>>             raise IndexError("Index of image %s out of range(%s)"
>>                              %(index, self.data["image_count"]))
>>
>>         multi = 0
>>         if index == "all":
>>             multi = 1
>>             index = 0
>>
>>         if not dest:
>>             dest = self.filename
>>
>>         target = os.path.splitext(dest)[0]
>>         res = self.raw.demosaicate(index, multi, target)    # This call
>> the CPython code
>>         self.out_filename = self.raw.out_file
>>
>>     def run_multitasks(self):
>>         def process(*args):
>>             fname, idx, oname = args
>>             self.identify()
>>             self.unraw(idx, oname)
>>
>>         while self.tasks:
>>             t = Thread(target=process, args=self.tasks.pop(0))
>>             t.start()
>>             t.join()
>>             if self.callback:
>>                 self.callback(self.out_filename)
>> ----------------------------------------------------------------
>>
>> I use a list of 20 files and all files are converted, one by one,
>> because the CPython part is not thread safe.
>>
>> But the real interesting thing is the behaviour in the slot() function.
>>
>> Examples:
>>  1 -------------------------------
>> def slot(name):
>>     if os.path.isfile(name):
>>         print("Created:", name, type(name))
>>         print("%s created" % name)
>>
>> result:
>> add: /home/vincent/Images/RAW_orig/RAW_CANON_1DSM2.CR2
>> Scaling with darkness 127, saturation 3712, and
>> multipliers 2.310656 1.000000 1.257116 1.000000
>> AHD interpolation...
>> Converting to sRGB colorspace...
>> Created:
>> /home/vincent/CPython/py361_venv/pyunraw/unraws/RAW_CANON_1DSM2.tiff
>> <class 'str'>
>> Erreur de segmentation (core dumped)
>> ----------------------------------
>>
>>  2 --------------------------------
>> def slot(name):
>>     new = name[:]
>>     if os.path.isfile(name):
>>         print("Created:", name, type(name))
>>         print("New:", new)
>>         print("%s created" % new)
>>
>> result:
>> add /home/vincent/Images/RAW_orig/RAW_CANON_1DSM2.CR2
>> Scaling with darkness 127, saturation 3712, and
>> multipliers 2.310656 1.000000 1.257116 1.000000
>> AHD interpolation...
>> Converting to sRGB colorspace...
>> Created:
>> /home/vincent/CPython/py361_venv/pyunraw/unraws/RAW_CANON_1DSM2.tiff
>> <class 'str'>
>> New: /home/vincent/CPython/py361_venv/pyunraw/unraws/RAW_CANON_1DSM2.tiff
>> Erreur de segmentation (core dumped)
>> ----------------------------------
>>
>> The most important, of course is the name of the new file created and I
>> can be satisfied with that, I just need the file name, but it's not a
>> reason to ignore the problem.
>>
>> Vincent
>>
>
>
>
> --
> https://mail.python.org/mailman/listinfo/python-list