__import__ fails randomly due to missing file

[Paul Moore]

On 12 March 2018 at 09:12, Arkadiusz Bulski <arek.bulski at gmail.com> wrote:
> Hi,
>
> For the record, I am the developer of Construct library. And the question
> relates to it.
> https://construct.readthedocs.io/en/latest/#
>
> I have an issue of Travis failing randomly at test cases. The cause is
> always __import__ failing to import a dynamically generated code. Library
> generates a string, saves it to a file (closes the file too), then
> __import__ it. I dont think there is anything wrong with the (generated
> imported) code itself. It seems like its some sort of issue with the Python
> runtime (or filesystem) not detecting the file because it was created only
> a moment ago. I tried to solve it by adding os.sync() between writing the
> file and importing it, but it seems to have only decreased the error rate,
> not solved it. Frankly, this is a heisenbug. Re-running the Travis job
> usually fixes it. Its random and happens very rarely. I havent pinpointed
> it to any particular version of CPython or Pypy (although I dont remember
> it ever happening on Pypy).
>
> An example:
> https://travis-ci.org/construct/construct/jobs/352213556
>
> The code in question:
> https://github.com/construct/construct/blob/d9c645ed0a75b5a2b64318113cf69cac822fd8f5/construct/core.py#L402-L414
> https://github.com/construct/construct/blob/d9c645ed0a75b5a2b64318113cf69cac822fd8f5/construct/core.py#L179-L180

The import system caches os.stat information, so it's possible that if
you're creating the file and then relatively quickly doing the import,
there may be some cache effect going on (granularity of file
modification timestamps?) I don't know the details, but it might be
worth looking in that area...

Paul