[pypy-issue] [issue1591] py3k: Django's "syncdb" fails when using sqlite3

[lvella]

New submission from lvella <lvella at gmail.com>:

Command:
$ pypy manage.py syncdb

Fails with the following message (without stack trace):
AttributeError: 'str' object has no attribute 'decode'

Digging a little deeper, I managed to find this line on SQLite3 backend
(django/db/backends/sqlite3/base.py):

if Database.version_info >= (2, 4, 1):
    # Starting in 2.4.1, the str type is not accepted anymore, therefore,
    # we convert all str objects to Unicode
    # As registering a adapter for a primitive type causes a small
    # slow-down, this adapter is only registered for sqlite3 versions
    # needing it (Python 2.6 and up).
    Database.register_adapter(str, lambda s: s.decode('utf-8'))
    Database.register_adapter(SafeBytes, lambda s: s.decode('utf-8'))

and that lambda registered with "str" type is throwing the exception. This code
itself seems to be wrong, because it doesn't take into account Python 3 changes
(Django's support for py3k is experimental), but what is really intriguing is
that it actually works on CPython.

I patched that code and replaced the lambda with a function that tries to
intercept the AttributeError exception, just to see if the exception was
occurring with CPython 3, too. The exception is raised, but somewhere in the
stack it is swallowed, what allows the execution to resume. The same is not true
on Pypy, and the exception manages to reach top-level and terminates the process
(without a stack trace).

To reproduce, install Django 1.5.2 on a PyPy3 2.1 beta 1 virtualenv, create a
project:
$ django-admin.py startproject test

modify "test/test/settings.py" to use sqlite3:
...
        'ENGINE': 'django.db.backends.sqlite3', # Add 'postgresql_psycopg2',
'mysql', 'sqlite3' or 'oracle'.
        'NAME': '/tmp/db',  # Or path to database file if using sqlite3.
...

and run syncdb inside project directory:
$ cd test
$ pypy ./manage.py syncdb

The error occurs after creating all the tables for the authentication
application (used by default), when calling the post-sync callbacks. By patching
"django/core/management/sql.py" with a traceback print:
...
        try:
            models.signals.post_syncdb.send(sender=app, app=app,
                created_models=created_models, verbosity=verbosity,
                interactive=interactive, db=db)
        except:
            traceback.print_exc()
            raise
...

I managed to get this traceback from pypy:
$ ./manage.py syncdb
Creating tables ...
Creating table auth_permission
Creating table auth_group_permissions
Creating table auth_group
Creating table auth_user_groups
Creating table auth_user_user_permissions
Creating table auth_user
Creating table django_content_type
Creating table django_session
Creating table django_site
Traceback (most recent call last):
  File
"/home/vella/projetos/chroni9/venv/site-packages/django/contrib/contenttypes/models.py",
line 39, in get_for_model
    ct = self._get_from_cache(opts)
  File
"/home/vella/projetos/chroni9/venv/site-packages/django/contrib/contenttypes/models.py",
line 29, in _get_from_cache
    return self.__class__._cache[self.db][key]
KeyError: 'default'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File
"/home/vella/projetos/chroni9/venv/site-packages/django/core/management/sql.py",
line 197, in emit_post_sync_signal
    interactive=interactive, db=db)
  File
"/home/vella/projetos/chroni9/venv/site-packages/django/dispatch/dispatcher.py",
line 170, in send
    response = receiver(signal=self, sender=sender, **named)
  File
"/home/vella/projetos/chroni9/venv/site-packages/django/contrib/auth/management/__init__.py",
line 77, in create_permissions
    ctype = ContentType.objects.db_manager(db).get_for_model(klass)
  File
"/home/vella/projetos/chroni9/venv/site-packages/django/contrib/contenttypes/models.py",
line 47, in get_for_model
    defaults = {'name': smart_text(opts.verbose_name_raw)},
  File
"/home/vella/projetos/chroni9/venv/site-packages/django/db/models/manager.py",
line 146, in get_or_create
    return self.get_query_set().get_or_create(**kwargs)
  File
"/home/vella/projetos/chroni9/venv/site-packages/django/db/models/query.py",
line 484, in get_or_create
    return self.get(**lookup), False
  File
"/home/vella/projetos/chroni9/venv/site-packages/django/db/models/query.py",
line 398, in get
    num = len(clone)
  File
"/home/vella/projetos/chroni9/venv/site-packages/django/db/models/query.py",
line 106, in __len__
    self._result_cache = list(self.iterator())
  File
"/home/vella/projetos/chroni9/venv/site-packages/django/db/models/query.py",
line 317, in iterator
    for row in compiler.results_iter():
  File
"/home/vella/projetos/chroni9/venv/site-packages/django/db/models/sql/compiler.py",
line 775, in results_iter
    for rows in self.execute_sql(MULTI):
  File
"/home/vella/projetos/chroni9/venv/site-packages/django/db/models/sql/compiler.py",
line 840, in execute_sql
    cursor.execute(sql, params)
  File
"/home/vella/projetos/chroni9/venv/site-packages/django/db/backends/util.py",
line 41, in execute
    return self.cursor.execute(sql, params)
  File
"/home/vella/projetos/chroni9/venv/site-packages/django/db/backends/sqlite3/base.py",
line 362, in execute
    return Database.Cursor.execute(self, query, params)
  File "/home/vella/progs/pypy3-2.1-beta1-linux64/lib_pypy/_sqlite3.py", line
887, in wrapper
    return func(self, *args, **kwargs)
  File "/home/vella/progs/pypy3-2.1-beta1-linux64/lib_pypy/_sqlite3.py", line
1046, in execute
    return self.__execute(False, sql, [params])
  File "/home/vella/progs/pypy3-2.1-beta1-linux64/lib_pypy/_sqlite3.py", line
1009, in __execute
    self.__statement._set_params(params)
  File "/home/vella/progs/pypy3-2.1-beta1-linux64/lib_pypy/_sqlite3.py", line
1288, in _set_params
    rc = self.__set_param(i + 1, params[i])
  File "/home/vella/progs/pypy3-2.1-beta1-linux64/lib_pypy/_sqlite3.py", line
1244, in __set_param
    param = adapt(param)
  File "/home/vella/progs/pypy3-2.1-beta1-linux64/lib_pypy/_sqlite3.py", line
1502, in adapt
    return adapter(val)
  File
"/home/vella/projetos/chroni9/venv/site-packages/django/db/backends/sqlite3/base.py",
line 82, in <lambda>
    Database.register_adapter(str, lambda s: s.decode('utf-8'))
AttributeError: 'str' object has no attribute 'decode'
AttributeError: 'str' object has no attribute 'decode'

----------
messages: 6101
nosy: lvella, pypy-issue
priority: bug
release: 2.1
status: unread
title: py3k: Django's "syncdb" fails when using sqlite3

________________________________________
PyPy bug tracker <tracker at bugs.pypy.org>
<https://bugs.pypy.org/issue1591>
________________________________________