[New-bugs-announce] [issue36658] Py_Initialze() throws error 'unable to load the file system encoding' when calling Py_SetPath with a path to a directory

[RimacV]

New submission from RimacV <rimacvaldez at gmx.de>:

I compiled the source of CPython 3.7.3 myself on Windows with Visual Studio 2017 together with some packages like e.g numpy. When I start the Python Interpreter I am able to import and use numpy. However when I am running the same script via the C-API I get an ModuleNotFoundError. 

So the first thing I did, was to check if numpy is in my site-packages directory and indeed there is a folder named numpy-1.16.2-py3.7-win-amd64.egg. (Makes sense because the python interpreter can find numpy)

The next thing I did was get some information about the sys.path variable created when running the script via the C-API. 

##### sys.path content ####
C:\Work\build\product\python37.zip
C:\Work\build\product\DLLs
C:\Work\build\product\lib
C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO\2017\PROFESSIONAL\COMMON7\IDE\EXTENSIONS\TESTPLATFORM
C:\Users\rvq\AppData\Roaming\Python\Python37\site-packages

Examining the content of sys.path I noticed two things. 

1. 
C:\Work\build\product\python37.zip has the correct path 'C:\Work\build\product\'. There was just no zip file. All my files and directory were unpacked. So I zipped the files to an archive named python37.zip and this resolved the import error.

2. C:\Users\rvq\AppData\Roaming\Python\Python37\site-packages is wrong it should be C:\Work\build\product\Lib\site-packages but I dont know how this wrong path is created. 


The next thing I tried was to use Py_SetPath(L"C:/Work/build/product/Lib/site-packages") before calling Py_Initialize(). This led to the 

Fatal Python Error 'unable to load the file system encoding' 
ModuleNotFoundError: No module named 'encodings'


I created a minimal c++ project with exact these two calls and started to debug Cpython. 

int main()
{
  Py_SetPath(L"C:/Work/build/product/Lib/site-packages");
  Py_Initialize();
}

I tracked the call of Py_Initialize() down to the call of 

static int
zipimport_zipimporter___init___impl(ZipImporter *self, PyObject *path)

inside of zipimport.c

The comment above this function states the following: 

Create a new zipimporter instance.
'archivepath' must be a path-like object to a zipfile, or to a specific path
inside a zipfile. For example, it can be '/tmp/myimport.zip', or
'/tmp/myimport.zip/mydirectory', if mydirectory is a valid directory inside
the archive.
'ZipImportError' is raised if 'archivepath' doesn't point to a valid Zip
archive.
The 'archive' attribute of the zipimporter object contains the name of the
zipfile targeted.


So for me it seems that the C-API expects the path set with Py_SetPath to be a path to a zipfile. Is this expected behaviour or is it a bug? 
If it is not a bug is there a way to changes this so that it can also detect directories? 

PS: The ModuleNotFoundError did not occur for me when using Python 3.5.2+, which was the version I used in my project before. I also checked if I had set any PYTHONHOME or PYTHONPATH environment variables but I did not see one of them on my system.

----------
components: Library (Lib)
files: Capture.PNG
messages: 340494
nosy: rvq
priority: normal
severity: normal
status: open
title: Py_Initialze() throws error 'unable to load the file system encoding' when calling Py_SetPath with a path to a directory
type: behavior
versions: Python 3.7
Added file: https://bugs.python.org/file48274/Capture.PNG

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue36658>
_______________________________________