[code-quality] Namespaced packages and relative imports

[Sylvain Thénault]

On 16 mai 11:16, François Vanderkelen wrote:
> Hi experts !

Hi François,
 
> I am struggling with Pylint with an issue that might be a problem coming
> from my code. Since I am relatively new to python (~1year) I would like to
> have your input on the matter.
> 
> I am trying to split a big python project I am doing for my company into
> several packages, allowing me to update each part without impacting the
> others. Doing that, we decided to go with Jenkins to improve our code
> quality which lead me to implement pylint for code violations checks.
> 
> Currently my build process is the following :
> 
> - Create a new virtualenv with Python version 2.7.6
> - Install all dependencies needed by my package
> - Download the last version of the package from the master branch in our
> internal gitlab
> - Run nosetests with coverage reports
> - Run pylint and export the result to have some violations reports in
> Jenkins.
> 
> At first I had a lot of pylint errors since I wasn't installing all
> dependencies, resulting in something like 200 import errors ... Now it is
> reduced to some 60 errors but when I look a the build output log, I can see
> that pylint is actually crashing in the middle of the process.
> 
> ************* Module company.project.core.security.app
> company/project/core/security/app.py:1: [C0111(missing-docstring), ]
> Missing module docstring
> load_entry_point('pylint==1.2.1', 'console_scripts', 'pylint')()
>   File "/var/lib/jenkins/.pyenv/versions/2.7.6/lib/python2.7/site-packages/pylint/__init__.py",
> line 577, in _search_zip
>     raise ImportError('No module named %s' % '.'.join(modpath))
> ImportError: No module named project

Would you create an issue in pylint's bitbucket tracker for this? In any case
it shouldn't crash like that?
 
> This is how my package is constructed :
> 
> company/
> -- __init__.py
> -- project/
>    -- __init__.py
>    -- core/
>       -- __init__.py
>       -- app.py
>       -- security/
>          -- __init__.py
>          -- app.py
> 
> In company.project.core.security.app I am trying to import
> company.project.core.app which contains some BaseClass for my project using:
> 
> from ..app import BaseApplication

This is probably another issue that deserves a ticket as well.

> This is working fine with python, although pylint is crying out loud on
> this specific line. I tried to replace the relative import by core.app
> which worked fine with pylint but my code is not working anymore. I tried
> also to use the full path (company.project.core.app) but I have the exact
> same import error.
> 
> Be aware that both company and project are just namespace packages.

what do you call namespace package ? Do you mean part of them are installed in
different directories? 

> Also I am forced to use the path to run pylint because it looks like my
> top-level folder is not a package (even if it contains a __init__ file).
> 
> pylint company.project.core
> No module named project.core (fatal)

and this work if you set a PYTHONPATH?
 
> pylint company/project/core is working.
> 
> When I try to run pylint directly from the top-level package, I have the
> exact same result as from the path.
> 
> A workaround I can try is to install the package before running pylint on
> it but I reckon since my cwd is supposed to be added to the path I can't
> see how it would help (although I have other company packages installed).

It should work.

[syt at orion ~]$ mkdir x
[syt at orion ~]$ mkdir x/y
[syt at orion ~]$ touch x/__init__.py
[syt at orion ~]$ touch x/y/__init__.py
[syt at orion ~]$ pylint -rn x.y
************* Module x.y
C:  1, 0: Missing module docstring (missing-docstring)
 
> What do you think I should do ? Is my design a poorly chosen one and I
> don't have any solution ?

At a quick glance I would say you seem to abuse a bit from packages, leading to
a deep structure which may be over-complicated. For instance, do you really want
a 'company' first level package? Anyway, that doesn't seem to be the cause of
your problems with pylint. At least, it shouldn't.

-- 
Sylvain Thénault, LOGILAB, Paris (01.45.32.03.12) - Toulouse (05.62.17.16.42)
Formations Python, Debian, Méth. Agiles: http://www.logilab.fr/formations
Développement logiciel sur mesure:       http://www.logilab.fr/services
CubicWeb, the semantic web framework:    http://www.cubicweb.org