From stonebig34 at gmail.com Mon Jun 3 20:05:32 2013 From: stonebig34 at gmail.com (big stone) Date: Mon, 3 Jun 2013 20:05:32 +0200 Subject: [DB-SIG] Make Python/SQL integration even smoother Message-ID: Hello, I discovered recently the power of simplifying the integration between python and its provided database sqlite3. In my dream, I would like to be able to querry in "sql syntax" all objects of python environnement. So, I would suggest the next evolution of the Database API to encourage/normalize the following evolution : - authorize also the use of parameters in place of table names, - the provided 'parameter for a table' should be : . a python list, . or a dictionary, . or query result, . or something similar like a pandas object. - this parameter could be used anywhere in sql, but not authorised as an "update". Any "table like" object passed as a parameter will have : - by default column names "c0", "c1", ... "cN" for each of it's N columns, - unless it has an attribute ".description" which contains a list of string names to be used as column title, it will be used instead of the default values Example : ************ #my_list is a python table my_list=[('a', 1), ('b',2)] # note the usage of ':mylist' cur=con.execute("select :mylist.* from :my_list",{'my_list':my_list}) rows = cur.fetchall() columns = [col_desc[0] for col_desc in cur.description] print(columns) print(rows) (gives:) c0 c1 a 1 b 2 # usage for a more complex work # copy that table in a real sql table cur=con.execute("create table keep_me as select :mylist.* from :my_list",{'my_list':my_list}) Example of forbidden usage : ******************************* # this is forbidden, fails : cur=con.execute("update :mylist.c0 set c1=c1+1 ",{'my_list':my_list}) Implementation : ******************** - the case where the list given contains a constant number of columns and a constant type in each columns is the 'normal' one, - If the number of columns varies, only the number of columns in the first columns is considered (or an error can be raised), - default types are unknown (as it works best with sqlite3), - if any type is provided in the object (like it may be found in a query result), it is taken. - if a non-simple object is passed, it is transform by "cpickle", or as a blob, or an exception is raised, as prefered for a simple implementation. - typically for sqlite : . the :my_list table is created as a 'create temporary table _tmp_my_list(c0,c1) . before the creation a "drop table if exists _tmp_my_list" is generated, . this temporary table is destroyed at the end of the execute. Regards, -------------- next part -------------- An HTML attachment was scrubbed... URL: From fog at dndg.it Tue Jun 4 09:28:53 2013 From: fog at dndg.it (Federico Di Gregorio) Date: Tue, 04 Jun 2013 09:28:53 +0200 Subject: [DB-SIG] Make Python/SQL integration even smoother In-Reply-To: References: Message-ID: <51AD9735.7070809@dndg.it> On 03/06/2013 20:05, big stone wrote: > Hello, > > I discovered recently the power of simplifying the integration between > python and its provided database sqlite3. > > In my dream, I would like to be able to querry in "sql syntax" all > objects of python environnement. You want LINQ [http://en.wikipedia.org/wiki/Language_Integrated_Query], and there is something similar in Python: https://github.com/heynemann/pynq/wiki but, IMHO, this is definitely something outside the scope of the dbapi. federico From vernondcole at gmail.com Tue Jun 4 21:30:09 2013 From: vernondcole at gmail.com (Vernon D. Cole) Date: Tue, 4 Jun 2013 13:30:09 -0600 Subject: [DB-SIG] adodbapi version 2.5.0 released Message-ID: Because I need it for continued work on the django-mssql upgrade, I have released adobapi in its current state (including the optional feature of being able to auto-select between 'qmark' and 'named' paramstyles by requesting a paramstyle of 'auto'.) It continues to be fully dbapi 2.0 compliant. All prospective dbapi 3 features are either defaulted 'off' or are proper extensions to the 2.0 api. The remote and proxy/server modules are working fairly well. Today, I created and populated a 38,000 record SQL Server data table using the django ORM running on my Ubuntu workstation. Interested persons can get the package from PyPi or http://sf.net/projects/adodbapi Version 2.4.3 will continue to ship with pywin32 until it drops support for Python 2.4. The new version requires pywin32 and Python 2.5 or later, or IronPython 2.7. Use of the remote and server functionality require Pyro4. [Pywin32 is not require on Linux clients, of course.] -- Vernon Cole -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at python.org Tue Jun 4 22:31:38 2013 From: mal at python.org (M.-A. Lemburg) Date: Tue, 04 Jun 2013 22:31:38 +0200 Subject: [DB-SIG] Make Python/SQL integration even smoother In-Reply-To: References: Message-ID: <51AE4EAA.3090406@python.org> I think you are suggesting a new Python DB-API module for accessing and querying Python namespaces. This could probably be done by leveraging techniques from Gadfly: http://gadfly.sourceforge.net/ Alternatively, you could mirror the namespaces into sqlite and then use SQLite's SQL engine for the queries. On 03.06.2013 20:05, big stone wrote: > Hello, > > I discovered recently the power of simplifying the integration between > python and its provided database sqlite3. > > In my dream, I would like to be able to querry in "sql syntax" all objects > of python environnement. > > So, I would suggest the next evolution of the Database API to > encourage/normalize the following evolution : > - authorize also the use of parameters in place of table names, > - the provided 'parameter for a table' should be : > . a python list, > . or a dictionary, > . or query result, > . or something similar like a pandas object. > - this parameter could be used anywhere in sql, but not authorised as an > "update". > > Any "table like" object passed as a parameter will have : > - by default column names "c0", "c1", ... "cN" for each of it's N columns, > - unless it has an attribute ".description" which contains a list of string > names to be used as column title, it will be used instead of the default > values > > > Example : > ************ > #my_list is a python table > my_list=[('a', 1), ('b',2)] > > # note the usage of ':mylist' > cur=con.execute("select :mylist.* from :my_list",{'my_list':my_list}) > > rows = cur.fetchall() > columns = [col_desc[0] for col_desc in cur.description] > > print(columns) > print(rows) > > (gives:) > c0 c1 > a 1 > b 2 > > # usage for a more complex work > # copy that table in a real sql table > cur=con.execute("create table keep_me as select :mylist.* from > :my_list",{'my_list':my_list}) > > > Example of forbidden usage : > ******************************* > # this is forbidden, fails : > cur=con.execute("update :mylist.c0 set c1=c1+1 ",{'my_list':my_list}) > > Implementation : > ******************** > - the case where the list given contains a constant number of columns and a > constant type in each columns is the 'normal' one, > - If the number of columns varies, only the number of columns in the first > columns is considered (or an error can be raised), > - default types are unknown (as it works best with sqlite3), > - if any type is provided in the object (like it may be found in a query > result), it is taken. > - if a non-simple object is passed, it is transform by "cpickle", or as a > blob, or an exception is raised, as prefered for a simple implementation. > - typically for sqlite : > . the :my_list table is created as a 'create temporary table > _tmp_my_list(c0,c1) > . before the creation a "drop table if exists _tmp_my_list" is generated, > . this temporary table is destroyed at the end of the execute. > > > > Regards, > > > > _______________________________________________ > DB-SIG maillist - DB-SIG at python.org > http://mail.python.org/mailman/listinfo/db-sig > -- Marc-Andre Lemburg Director Python Software Foundation http://www.python.org/psf/ From mal at python.org Wed Jun 5 21:07:18 2013 From: mal at python.org (M.-A. Lemburg) Date: Wed, 05 Jun 2013 21:07:18 +0200 Subject: [DB-SIG] Make Python/SQL integration even smoother In-Reply-To: References: <51AE4EAA.3090406@python.org> Message-ID: <51AF8C66.9070405@python.org> On 05.06.2013 20:55, big stone wrote: > Hello M.-A., > > Afaik : > "Linq" is not pure/transparent SQL Linq is SQL embedded into a programming language. C# and VB are the only ones supporting this at the moment, AFAIK. > "Gadfly" is not SQL power (of true SQL motors). Gadfly comes with its own SQL engine. It works in memory, so provides excellent performance, and it's written in Python (with a few C extensions for speed), so could be used as basis for what you have in mind. > My suggestion is : > - use a true SQL to query transparently a mix of python objects (lists, > tuples...) AND already SQL tables, > - simplify the upload of datas to SQL, as the upload is the painfull part > of the sql standard : > > Today : > query 1 million rows in sql= "select * from a_sql_table" > upload1 million rowsto sql = ...(more complex than 3 words in a > string)... > query 1 million rows of datas half in a sql table and half in a python > table = ... (even more complex).. > > With the suggestion = > "select * from a_sql_table as s inner join :a_python_object as p on > s.field0= p.c0" > > > And all that being performant because : > - the proposal would still use the SQL motors , > - to reduce the upload start : > . just begin your sql sequences by "create table x from select * from > :a_python_object", > . use a SQL in memory database, what is just by luck SQlite with the > ":"memry:" option. I'm not sure whether people would really want to use SQL to manipulate Python objects, but who knows... perhaps you'll find a user community who'd love to work that way. I think the closes we have to what you're suggesting is pandas, which sort of provides ORM style SQL operations on tables: http://pandas.pydata.org/ I guess you could also write a SQL parser which transforms the SQL or some Linq like dialect for Python into the method calls for pandas. > Regards, > > > 2013/6/4 M.-A. Lemburg > >> I think you are suggesting a new Python DB-API module for >> accessing and querying Python namespaces. >> >> This could probably be done by leveraging techniques from >> Gadfly: http://gadfly.sourceforge.net/ >> >> Alternatively, you could mirror the namespaces into sqlite and >> then use SQLite's SQL engine for the queries. >> >> >> On 03.06.2013 20:05, big stone wrote: >>> Hello, >>> >>> I discovered recently the power of simplifying the integration between >>> python and its provided database sqlite3. >>> >>> In my dream, I would like to be able to querry in "sql syntax" all >> objects >>> of python environnement. >>> >>> So, I would suggest the next evolution of the Database API to >>> encourage/normalize the following evolution : >>> - authorize also the use of parameters in place of table names, >>> - the provided 'parameter for a table' should be : >>> . a python list, >>> . or a dictionary, >>> . or query result, >>> . or something similar like a pandas object. >>> - this parameter could be used anywhere in sql, but not authorised as an >>> "update". >>> >>> Any "table like" object passed as a parameter will have : >>> - by default column names "c0", "c1", ... "cN" for each of it's N >> columns, >>> - unless it has an attribute ".description" which contains a list of >> string >>> names to be used as column title, it will be used instead of the default >>> values >>> >>> >>> Example : >>> ************ >>> #my_list is a python table >>> my_list=[('a', 1), ('b',2)] >>> >>> # note the usage of ':mylist' >>> cur=con.execute("select :mylist.* from :my_list",{'my_list':my_list}) >>> >>> rows = cur.fetchall() >>> columns = [col_desc[0] for col_desc in cur.description] >>> >>> print(columns) >>> print(rows) >>> >>> (gives:) >>> c0 c1 >>> a 1 >>> b 2 >>> >>> # usage for a more complex work >>> # copy that table in a real sql table >>> cur=con.execute("create table keep_me as select :mylist.* from >>> :my_list",{'my_list':my_list}) >>> >>> >>> Example of forbidden usage : >>> ******************************* >>> # this is forbidden, fails : >>> cur=con.execute("update :mylist.c0 set c1=c1+1 ",{'my_list':my_list}) >>> >>> Implementation : >>> ******************** >>> - the case where the list given contains a constant number of columns >> and a >>> constant type in each columns is the 'normal' one, >>> - If the number of columns varies, only the number of columns in the >> first >>> columns is considered (or an error can be raised), >>> - default types are unknown (as it works best with sqlite3), >>> - if any type is provided in the object (like it may be found in a query >>> result), it is taken. >>> - if a non-simple object is passed, it is transform by "cpickle", or as a >>> blob, or an exception is raised, as prefered for a simple implementation. >>> - typically for sqlite : >>> . the :my_list table is created as a 'create temporary table >>> _tmp_my_list(c0,c1) >>> . before the creation a "drop table if exists _tmp_my_list" is >> generated, >>> . this temporary table is destroyed at the end of the execute. >>> >>> >>> >>> Regards, >>> >>> >>> >>> _______________________________________________ >>> DB-SIG maillist - DB-SIG at python.org >>> http://mail.python.org/mailman/listinfo/db-sig >>> >> >> -- >> Marc-Andre Lemburg >> Director >> Python Software Foundation >> http://www.python.org/psf/ >> > -- Marc-Andre Lemburg Director Python Software Foundation http://www.python.org/psf/ From fog at dndg.it Thu Jun 6 09:37:44 2013 From: fog at dndg.it (Federico Di Gregorio) Date: Thu, 06 Jun 2013 09:37:44 +0200 Subject: [DB-SIG] Make Python/SQL integration even smoother In-Reply-To: <51AF8C66.9070405@python.org> References: <51AE4EAA.3090406@python.org> <51AF8C66.9070405@python.org> Message-ID: <51B03C48.4040906@dndg.it> On 05/06/2013 21:07, M.-A. Lemburg wrote: > On 05.06.2013 20:55, big stone wrote: >> Hello M.-A., >> >> Afaik : >> "Linq" is not pure/transparent SQL > > Linq is SQL embedded into a programming language. C# and VB are > the only ones supporting this at the moment, AFAIK. LINQ is much much more than that and does much more than execute SQL-like queries on collections of objects. That's why I cited it in my original reply: you one want queries over objects taking inspiration from LINQ will probably yeld something more useful than just an SQL-applied-to-objects parser. But we're going out of the scope of this ML, so I'll stop here. federico From michaelklein4685 at outlook.com Tue Jun 11 13:58:44 2013 From: michaelklein4685 at outlook.com (Michael Klein) Date: Tue, 11 Jun 2013 16:28:44 +0430 Subject: ACÍLIO JOÃO CARLOS MOREIRA DE CARVALHO ANDRADE Message-ID: <13709676739185558a200c5576d41055ff20794079@outlook.com> GRAFER JULIANA RODRIGUES ALVES, DIONE MARIA DA SILVA, JO?O CARLOS MOREIRA DE CARVALHO, ROBERTO BEZERRA FERREIRA, IANN MARZZO SAMPAIO LIBOS, MAYSA RAQUEL DA SILVA, THALITA JESSICA FERREIRA DA ROCHA, ERICH DOUGLAS MOREIRA CHAVES. PAULO HENRIQUE LIMA ALVARENGA. AM?VEL CLODOMIRO, ELEUT?RIO LEBARBENCHON, LEBARBENCHON CORRADINI, CLODOMIRO AC?LIO LEBARBENCHON. ELEUT?RIO GRAFER, VICENTE LEPORACE, JARDEL MELLO, N?LSON CARUSO, DANIEL DANTAS, RICARDO BLAT, FL?VIO MIGLIACCIO, LU?S DELFINO, , SELTON MELLO, HAMILTON RICARDO, MARCOS CARUSO, CARLOS VEREZA, PAULO FRIEBE, ELIEZER GOMES, JUCA DE OLIVEIRA, ANDERSON DI RIZZI APAR?CIO, AC?LIO. ?GUA DOCE DO NORTE, INDIAN?POLIS, FLOR?NIA IGUATAMA. AM?VEL APAR?CIO, AC?LIO. ?GUA DOCE DO NORTE, INDIAN?POLIS, FLOR?NIA JO?O CARLOS MOREIRA DE CARVALHO PADRE CARVALHO PAULO HENRIQUE LIMA ALVARENGA BALDU?NO GRAFER, CLODOMIRO LEBARBENCHON. From vernondcole at gmail.com Thu Jun 13 20:39:42 2013 From: vernondcole at gmail.com (Vernon D. Cole) Date: Thu, 13 Jun 2013 19:39:42 +0100 Subject: [DB-SIG] Notice: Restriction on adodbapi remote with Pyro4 v 2.20 Message-ID: Anyone trying the new ADO remote/proxy feature of adodbapi: Pyro 2.20 (just released) uses a new serializer which does not handle decimal or datetime data types. If you update your Pyro4, you need to switch back to the old pickle serializer by defining the environment variable PYRO_SERIALIZER=pickle -- Vernon Cole -------------- next part -------------- An HTML attachment was scrubbed... URL: From lhoyem at nuodb.com Fri Jun 28 21:55:08 2013 From: lhoyem at nuodb.com (Lindsey Hoyem) Date: Fri, 28 Jun 2013 15:55:08 -0400 Subject: [DB-SIG] Fwd: Check out the Techblog about NuoDB's Python driver: Pynuodb! In-Reply-To: References: Message-ID: To whom it may concern, I am a community manager for NuoDB, and we are really interested in hearing what developers think about NuoDB capabilities with Python! As a little background, NuoDB is a company leading the industry with a proven NewSQL solution to solve challenges coping with application performance, maintaining business continuity and gaining operational intelligence in real time by providing a unique combination of scale-out performance, zero downtime and geo-distributed data management. We would love to have this tech blog in your read only mailing list: http://bit.ly/14XXLjD Thank you for your consideration, Lindsey Hoyem Lindsey Hoyem Marketing Intern *www.nuodb.com* p: 617-500-0001 -------------- next part -------------- An HTML attachment was scrubbed... URL: