From rburnham@cri-inc.com Mon Mar 1 05:49:51 1999 From: rburnham@cri-inc.com (Roger Burnham) Date: Sun, 28 Feb 1999 21:49:51 -800 Subject: [DB-SIG] (no subject) Message-ID: <199903010551.XAA00863@mail1.gte.net> unsubscribe froglette rburnham@world.std.com From mal@lemburg.com Mon Mar 1 10:23:53 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 01 Mar 1999 11:23:53 +0100 Subject: [DB-SIG] API Enhancements References: <36A32169.2ECE2BD@lemburg.com> <36D93B76.1143DEE1@lyra.org> Message-ID: <36DA6AB9.4DCF10BF@lemburg.com> Greg Stein wrote: > > M.-A. Lemburg wrote: > > > > Hi everybody, > > > > It's been a long time since we've last had some discussion > > about a novelation of the API. As you probably remember I have > > a proposed spec available at: > > > > http://starship.skyport.net/~lemburg/DatabaseAPI-1.1.html > > Is it possible to get this document (or a version of it) red-lined so > that we can see the differences from the 1.0 document? I can generally > see the differences, but it would be better to clarify it. > [ I'd be willing to construct the red-lined version if you'd like > assistance ] [Meta: is there a tool available to aid in doing this, e.g. a HTML aware diff ?] I would appreciate you helping with it, since I'm pretty much burried in work. > > 1. Threading > > > > The API spec should make some notes about the scope of thread > > safety imposed on the interfaces. Thread safety could be given > > at module level (threads all use the same module, but maintain > > their own connections), connection level (threads share modules > > and connections, cursors are not shared between threads) and > > cursor level (threads share module, connections and cursors). > > > > The last level is probably not feasable, but I think connection > > level could be reached. > > A discussion in the API would be good. I might even say that a module > could export an attribute that states which level is supported. For > example, I know that many Win32 ODBC drivers are NOT thread-safe at all. > You must open a connection per thread. How about a constant defined as module global: threadsafety: 0 - module level 1 - connection level 2 - cursor level If it's not available, then module level should be assumed. > > > > 2. Stored procedures > > > > Basically I want to revisit the discussion. The 1.1 proposal > > defines this interface: > > > > callproc(procname,list_of_parameters) > > Should be a sequence of params, and probably should be optional. > > > This method is optional since not all databases > > provide stored procedures. > > > > Call a stored database procedure with the given > > name. The list of parameters must contain one > > entry for each argument that the procedure > > expects. The result of the call is returned by > > modifying the list contents in place. Input > > parameters are left untouched, output and > > input/output parameters replaced with the new > > values. > > This should be clarified that a copy of the params is made, then > modified. The wording sounds like the method mucks with the argument > value (which it may not be able to do if the argument is a tuple or > other sequence). Right. I'm currently tending towards this interface description: The sequence of parameters is passed to the stored procedure as copy. The possibly modified result is returned to the caller. Ordering of the sequence entries remains the same. Entries that bind to output variables will be overwritten with new values. Well, something along those lines... the basic idea is that the sequence will look the same on input and output, making it very easy to grab the output variables after the call is made. > > The procedure may also provide a result set as > > output. This must then be made available through > > the standard fetch-methods. > > > > Is this general enough to fit everybody's needs ? I know that > > Nobody has said "no" yet, so why don't we just say it is and be done > with it. It is certainly a step better than the 1.0 spec. > > > Jim Fulton would rather like an interface which returns a callable > > type... but it seems overkill to ask module writers to implement > > this just to be DB API conform. > > The hope is that the modules are easy to code, and then Python stuff is > layered on top if more functionality is needed. It is "The Python Way" > :-) > > > 3. A standard catalog interface > > > > There is often a need to connect to a database without knowing > > in advance what tables it contains and how the table columns > > are named. > > > > The API should define a catalog method for this purpose, > > I'd say "no" and leave that to a Python-level module. In many databases, > you can get this information from special tables. Custom APIs are not > required. For those databases that require a custom API, then I'd say > the module should expose those functions as a module-specific extension. > > Creating a "unified" API can then be done at the Python level. Few > programs, however, require that unified API, so it seems burdensome to > impose a unified API upon the module writers. Hmm. This takes us to another project: that of turning database support for Python into packages... I'm currently building all my extensions this way: there is usually one C extension and a whole bunch of Python scripts which build upon its functionality. The user always accesses the package through the main door :-) and thus does not see what actually happens at the back end. Database interfaces could use a similar approach, e.g. code the catalog interface in Python using the C extension to access those special tables. > >... > > 4. Optional named cursors > > > > The cursor constructor should be allowed to have an optional > > argument for the cursor name. This is useful for UPDATEs. > > I do not understand the use or operation of this, and how it would > benefit an UPDATE. Could you elaborate? In ODBC, for example, the ODBC driver uses generated names for all cursors. Now, it is sometimes useful to be able to refer to the cursors under known names, e.g. to do UPDATE ... WHERE CURRENT OF . The optional argument allows this by providing an explicit name. Database interfaces that do not support naming cursors should probably raise an exception in case they receive an argument to the cursor constructor. Actually, they will if the follow the 1.0 spec closely ;-) > META QUESTION: > > What should the process be for moving to a new rev of the API? Marc has > been championing his changes for well over a year now, but closure > hasn't been reached. > > I would recommend a "Last Call" type of approach. Set the date as (say) > March 12, receive comments until then, fold in the comments, then issue > a "Final Call" for the updated version for March 26. > > The DBAPI 1.1 would be posted by Monday the 29th. Sounds like a reasonable approach. I'll post a note about those dates later on. > META-META :-) ... is this approach acceptable? If not, then speak up. If > nobody has an issue with the approach, then let's assume that we've > begun the Last Call phase. Right. -- Marc-Andre Lemburg Y2000: 305 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : --------------------------------------------------------- From mal@lemburg.com Mon Mar 1 10:27:04 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 01 Mar 1999 11:27:04 +0100 Subject: [DB-SIG] ctsybasemodule problem and dbi question References: <001601be6311$9273a980$23ffd4d4@cameron> <36D933B0.229F5D4B@lyra.org> <36DA5ED3.77F92BA1@lemburg.com> Message-ID: <36DA6B78.789F5564@lemburg.com> Greg Stein wrote: > > Cameron Mellor wrote: > > > > From: M.-A. Lemburg > > >... > > >I think dbi is not really needed anymore: The only two types I can > > >think of where it has some use are date/time values (use mxDateTime > > >for those) and RAW input values (don't know how important it is > > >to be able to differentiate between plain strings and RAW input -- > > >plain Python strings work just fine for mxODBC in both modes). > > > > I thought they were there as placeholders: that is, you knew from the data > > that it was "special" and needed appropriate handling. Anyway, I'll take a > > closer look and see what I can find. I may end up using ctsybase anyway > > (since I'm playing with Sybase at the moment) and I don't really want it > > dumping core. :-/ > > The dbiRAW type is mostly for *input* binding rather than output. As I > recall, the Oracle database requires you to bind LONG / LONG RAW columns > differently than VARCHAR columns. The dbiRAW type allows the execute() > method to use the appropriate binding mechanism -- based on the argument > being a string or a dbiRAW object. > > Note: at the time the Oracle code was written, there was no way for the > execute() method to figure out the right way to bind the input. Maybe > Oracle has provided a way to prepare an INSERT statement, check the > implied input types, and then perform the bind. Somehow, I doubt it > though. Ok, I'm a little biased: ODBC let's you query the types of the to-be-bound parameters after statemente prepare and prior to the actual execute. Still, I think that these cases should be indicated by using (standard) Python types rather than reinventing the wheel for every database interface out there. So I guess appropriate approach would be replacing the dbi types with ones that are not only mere wrappers but provide useful functionality such as the mxDateTime types. A proper raw type would be one useable as generic read/write buffer complying to the buffer interface. For monetary types, a fixed decimal point format would be nice along with currency information to be able to implement automatic currency conversion. > The dbiDATE type exists for a similar reason: there was no way for the > DB modules to determine that a particular input was a date. Presuming > that a standard type is established for time/date handling (such as > mxDateTime), then the DB module can do the input binding properly. > > A similar situation would arise for money types or other special column > types. If the underlying database requires that a column must be bound > in a special way for that type, then the DB module must figure that out > from the type of the arguments (since it has nowhere else to turn to for > the info). > > Marc: did I see somewhere that Guido might incorporate mxDateTime as a > standard module in 1.6 or something like that? That might be a possibility, but it's up to Guido to decide on that. Inclusion of the above two types would then also be a good idea (monetary and buffer type). Or we could use a different approach: wrap the types up in a seperate distribution along with all kinds of useful documentation and maybe even precomiled binaries... then we could point people at one other package to download instead of giving them three or four different pointers. Cheers, -- Marc-Andre Lemburg Y2000: 305 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : --------------------------------------------------------- From gstein@lyra.org Mon Mar 1 12:55:28 1999 From: gstein@lyra.org (Greg Stein) Date: Mon, 01 Mar 1999 04:55:28 -0800 Subject: [DB-SIG] ctsybasemodule problem and dbi question References: <001601be6311$9273a980$23ffd4d4@cameron> <36D933B0.229F5D4B@lyra.org> <36DA5ED3.77F92BA1@lemburg.com> <36DA6B78.789F5564@lemburg.com> Message-ID: <36DA8E40.4D6EB7EF@lyra.org> M.-A. Lemburg wrote: > ... > Ok, I'm a little biased: ODBC let's you query the types of the > to-be-bound parameters after statemente prepare and prior to > the actual execute. Still, I think that these cases should be > indicated by using (standard) Python types rather than reinventing > the wheel for every database interface out there. "should" ... but recall this was three years ago. The standard python types for raw sequences of bytes were (are!) strings. Dates are integers. The DBI types merely wrap these to provide additional info to the modules, while the actual value is found in the "value" attribute. Sure, as Python picks up new types, then things can change. Until then, I believe the DBI style is appropriate. There shouldn't be N copies/varieties of the module, however. That has been a mistake. Does ODBC really allow you to query *input* bindings? Output bindings... sure. But querying the types of the input would surprise me. Regardless, Oracle barfs if you attempt to bind a string (in the Oracle sense) for input, but it requires a LONG (RAW). > So I guess appropriate approach would be replacing the dbi types > with ones that are not only mere wrappers but provide useful > functionality such as the mxDateTime types. > > A proper raw type would be one useable as generic read/write buffer > complying to the buffer interface. These would be handy, and would be an improvement. Python 1.5.2 has a buffer type built into it now, although there isn't a Python-level constructor for it (yet). That may be an option for replacing dbiRaw. Using mxDateTime would seem to be fine. > For monetary types, a fixed decimal point format would be nice > along with currency information to be able to implement automatic > currency conversion. Most databases allow binding of an integer/float against a currency column. This wasn't been the case for dates and LONG columns, though, so only those two had DBI types. I might argue that a currency type should be relegated to Python, and leave the module simple. > ... > Or we could use a different approach: wrap the types up in a > seperate distribution along with all kinds of useful documentation > and maybe even precomiled binaries... then we could point people > at one other package to download instead of giving them three or > four different pointers. This was the hope with the dbi module. If new, useful types replace it... sure. I might suggest deferring this kind of change to a 1.2 spec. I'm not really convinced that anything is broken, beyond the multiple copies of dbi out there (and how the modules link/find dbi). Cheers, -g -- Greg Stein, http://www.lyra.org/ From gstein@lyra.org Mon Mar 1 13:11:55 1999 From: gstein@lyra.org (Greg Stein) Date: Mon, 01 Mar 1999 05:11:55 -0800 Subject: [DB-SIG] API Enhancements References: <36A32169.2ECE2BD@lemburg.com> <36D93B76.1143DEE1@lyra.org> <36DA6AB9.4DCF10BF@lemburg.com> Message-ID: <36DA921B.E8DDA85@lyra.org> M.-A. Lemburg wrote: > > Greg Stein wrote: > >... > > Is it possible to get this document (or a version of it) red-lined so > > that we can see the differences from the 1.0 document? I can generally > > see the differences, but it would be better to clarify it. > > [ I'd be willing to construct the red-lined version if you'd like > > assistance ] > > [Meta: is there a tool available to aid in doing this, e.g. a > HTML aware diff ?] Don't know of one. > I would appreciate you helping with it, since I'm pretty much > burried in work. Sure. > > > > 1. Threading > > > > > > The API spec should make some notes about the scope of thread > > > safety imposed on the interfaces. Thread safety could be given > > > at module level (threads all use the same module, but maintain > > > their own connections), connection level (threads share modules > > > and connections, cursors are not shared between threads) and > > > cursor level (threads share module, connections and cursors). > > > > > > The last level is probably not feasable, but I think connection > > > level could be reached. > > > > A discussion in the API would be good. I might even say that a module > > could export an attribute that states which level is supported. For > > example, I know that many Win32 ODBC drivers are NOT thread-safe at all. > > You must open a connection per thread. > > How about a constant defined as module global: > > threadsafety: > > 0 - module level > 1 - connection level > 2 - cursor level > > If it's not available, then module level should be assumed. Strings constants are easy in Python. Would strings make more sense? 'module', 'connection', and 'cursor' ? A module global stating the specification level would also be handy. '1.1' for modules conforming to the DBAPI 1.1 spec. If the global is not present, then the module is a 1.0 module. > ... > Hmm. This takes us to another project: that of turning database > support for Python into packages... I'm currently building all my > extensions this way: there is usually one C extension and a whole > bunch of Python scripts which build upon its functionality. The user > always accesses the package through the main door :-) and thus does > not see what actually happens at the back end. > > Database interfaces could use a similar approach, e.g. code > the catalog interface in Python using the C extension to access > those special tables. Sure, but I'd say that the spec for any package is separate from the DBAPI spec for C modules. In general, it seems more difficult for C-level support to occur, than Python-level. IMO, providing a clean spec which is easy to implement really helps. > > >... > > > 4. Optional named cursors > > > > > > The cursor constructor should be allowed to have an optional > > > argument for the cursor name. This is useful for UPDATEs. > > > > I do not understand the use or operation of this, and how it would > > benefit an UPDATE. Could you elaborate? > > In ODBC, for example, the ODBC driver uses generated names for > all cursors. Now, it is sometimes useful to be able to refer to > the cursors under known names, e.g. to do UPDATE ... WHERE CURRENT > OF . The optional argument allows this by providing > an explicit name. But this update is passed to the execute() method of the cursor object that you're talking about, isn't it? Or do you create a second cursor, which updates the record currently indicated by the first cursor? Isn't that kind of technique a bit difficult without a rigorous specification of the module's fetching behavior? Just because I called fetchone() doesn't mean the cursor is sitting on the first record... an array-fetch may have occurred, and the cursor is 90 rows later. By naming a cursor, it would seem that any array-fetching would need to be disabled so that a program can actually determine what record is being referenced by the cursor. For safety, it may even be prudent to disable fetchmany() and fetchall(). Finally: is there a performance gain to the named-cursor technique? (compared to an UPDATE ... WHERE = ) Oh... a question about the nextresult() method. Why is that exposed? Shouldn't the module just automatically move to the next result set inside the fetch* methods? I don't see the benefit of exposing that to the user. It seems an artifact of the underlying database interface, rather than a useful user-level concept. Cheers, -g -- Greg Stein, http://www.lyra.org/ From mal@lemburg.com Mon Mar 1 21:43:48 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 01 Mar 1999 22:43:48 +0100 Subject: [DB-SIG] Redifining Python References: <199902270205.RAA30355@mailhost.akcache.com> Message-ID: <36DB0A14.735AF711@lemburg.com> Timothy Johnson wrote: > > I am just now investigating Python. I just had a converastion with a web > page designer who DOES NOT design in Perl, but has Perl modules written for > her, but then maintains and modifies code. > As a "C" programmer, I use pre-processor defines to create templates for > processing web page input (CGI program source). These templates are easy to > maintain and recompile by a "para-programmer" such as the individual I > mention above. > Having stated that, here's my question: Can one write a sophisticated > application with Python, but provide a simple, maintainable interface to > enable ongoing maintainance by a someone of less programming skills? > ("Para-programmer"). Just as I do with my C++ programs. And the beauty of > this is that it would not have to be run through a compiler. Not sure what this has to do with databases, but anyway: Check out the CGI topic guide on www.python.org. It has lots of useful pointers to HTMLgen, Zope, Bobo, etc. which all offer some sort of separation of "programming" and "webpage design". I'm using a Python-HTML intergration in such a (commercial) system myself with much success. Cheers, -- Marc-Andre Lemburg Y2000: 305 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : --------------------------------------------------------- From mal@lemburg.com Tue Mar 2 11:58:12 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Tue, 02 Mar 1999 12:58:12 +0100 Subject: [DB-SIG] DB API Spec 1.1 Roadmap Message-ID: <36DBD254.17220B43@lemburg.com> Following Greg's proposal, I suggest the following schedule for standardizing the 1.1 version of the official DB API Spec: Last Call deadline: 1999-03-12 Final Call deadline: 1999-03-26 Posting of the Spec: 1999-03-29 Comments will be received until the 12th and folded into the Spec. The updated Spec. will then go into the final round until the 26th. The current working version of the Spec. can always be found at: http://starship.skyport.net/~lemburg/DatabaseAPI-1.1.html The 1.0 Spec. is here: http://www.python.org/sigs/db-sig/DatabaseAPI.html -- Marc-Andre Lemburg Y2000: 304 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : --------------------------------------------------------- From Tod Olson Tue Mar 2 15:00:53 1999 From: Tod Olson (Tod Olson) Date: Tue, 02 Mar 1999 09:00:53 -0600 Subject: [DB-SIG] API Enhancements In-Reply-To: Your message of "Mon, 01 Mar 1999 05:11:55 -0800" References: <36DA921B.E8DDA85@lyra.org> Message-ID: <199903021500.JAA07578@stone.lib.uchicago.edu> >>>>> "Greg" == Greg Stein writes: Greg> Oh... a question about the nextresult() method. Why is that Greg> exposed? Shouldn't the module just automatically move to the Greg> next result set inside the fetch* methods? I don't see the Greg> benefit of exposing that to the user. It seems an artifact of Greg> the underlying database interface, rather than a useful Greg> user-level concept. [I assume you mean the nextset() method, else ignore me.] The module should not just automatically move to the next result set inside the fetch* methods. The next result set could look very different or need to be interpreted differently. Take output from Sybase's sp_helpdb as a trivial example (as displayed by isql or sqsh, my ellipse): name db_size owner dbid created status -------- ------------- ------- ------ -------------- -------------- horizon 31958.0 MB sa 5 May 10, 1998 no options set (1 row affected) device_fragments size usage free kbytes ------------------ ---------- ----------- ----------- sybdata1 1999.0 MB data only 123840 sybdata2 1990.0 MB data only 26496 sybdata3 1990.0 MB data only 59984 [...] syblog1 1999.0 MB log only 1967440 (return status = 0) The user needs an indication when a result set is done, which is currently provided by the fetch* methods returning None or []. The program signalling it's ready for the next result set by calling nextset() is analogous to calling fetchone() for the next row: the program is interested in a new, meaningful chunk of data. Silently moving to the next result set means that a programmer would have to do some kind of control break processing on each row's description, in the case of fetchone(), in order to figure out when the result set changed. I don't even want to conjecture how fetchmany() or fetchall() would or could behave in this context. A pleasant side-effect of nextset() is that the programmer can skip all or part of a result set without having to retrieve the whole thing. (Well, this *may* vary across database vendors who support multiple result sets, but I'd be surprised.) Tod A. Olson "How do you know I'm mad?" said Alice. ta-olson@uchicago.edu "If you weren't mad, you wouldn't have The University of Chicago Library come here," said the Cat. From mal@lemburg.com Wed Mar 3 09:00:12 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 03 Mar 1999 10:00:12 +0100 Subject: [DB-SIG] API Enhancements References: <36A32169.2ECE2BD@lemburg.com> <36D93B76.1143DEE1@lyra.org> <36DA6AB9.4DCF10BF@lemburg.com> <36DA921B.E8DDA85@lyra.org> Message-ID: <36DCFA1C.74C40BA0@lemburg.com> Greg Stein wrote: > > > > A discussion in the API would be good. I might even say that a module > > > could export an attribute that states which level is supported. For > > > example, I know that many Win32 ODBC drivers are NOT thread-safe at all. > > > You must open a connection per thread. > > > > How about a constant defined as module global: > > > > threadsafety: > > > > 0 - module level > > 1 - connection level > > 2 - cursor level > > > > If it's not available, then module level should be assumed. > > Strings constants are easy in Python. Would strings make more sense? > 'module', 'connection', and 'cursor' ? Hmm, strings wouldn't indicate order. Cursor level thread safety implies connection level and module level. > A module global stating the specification level would also be handy. > '1.1' for modules conforming to the DBAPI 1.1 spec. If the global is not > present, then the module is a 1.0 module. How about: apilevel ?! > > ... > > Hmm. This takes us to another project: that of turning database > > support for Python into packages... I'm currently building all my > > extensions this way: there is usually one C extension and a whole > > bunch of Python scripts which build upon its functionality. The user > > always accesses the package through the main door :-) and thus does > > not see what actually happens at the back end. > > > > Database interfaces could use a similar approach, e.g. code > > the catalog interface in Python using the C extension to access > > those special tables. > > Sure, but I'd say that the spec for any package is separate from the > DBAPI spec for C modules. In general, it seems more difficult for > C-level support to occur, than Python-level. IMO, providing a clean spec > which is easy to implement really helps. Well, that's why I said it's another project. The C level stuff should be fairly easy to implement; all fancy extensions can then be done in Python. > > > > 4. Optional named cursors > > > > > > > > The cursor constructor should be allowed to have an optional > > > > argument for the cursor name. This is useful for UPDATEs. > > > > > > I do not understand the use or operation of this, and how it would > > > benefit an UPDATE. Could you elaborate? > > > > In ODBC, for example, the ODBC driver uses generated names for > > all cursors. Now, it is sometimes useful to be able to refer to > > the cursors under known names, e.g. to do UPDATE ... WHERE CURRENT > > OF . The optional argument allows this by providing > > an explicit name. > > But this update is passed to the execute() method of the cursor object > that you're talking about, isn't it? Or do you create a second cursor, > which updates the record currently indicated by the first cursor? Both are possible. It depends on the database capabilities whether this technique is of much use: e.g. you could update all the records starting from the 10th row in a result set. Some DBs support scrolling cursors in backward direction too, making updates relying on external resources (e.g. file in a file system) possible. > Isn't that kind of technique a bit difficult without a rigorous > specification of the module's fetching behavior? Just because I called > fetchone() doesn't mean the cursor is sitting on the first record... an > array-fetch may have occurred, and the cursor is 90 rows later. Good point. fetchone() ought do what the name implies: fetch exactly one record. Otherwise usage of cursors would be pointless anyway. > By naming a cursor, it would seem that any array-fetching would need to > be disabled so that a program can actually determine what record is > being referenced by the cursor. For safety, it may even be prudent to > disable fetchmany() and fetchall(). Leave that to the user of the database interface. It should be mentioned in the spec. though. fetchmany() may actually fetch more records than requested. fetchall() wouldn't make sense in this context anyway. > Finally: is there a performance gain to the named-cursor technique? > (compared to an UPDATE ... WHERE = ) Hard to say... it could safe a temporary table though. > Oh... a question about the nextresult() method. Why is that exposed? > Shouldn't the module just automatically move to the next result set > inside the fetch* methods? I don't see the benefit of exposing that to > the user. It seems an artifact of the underlying database interface, > rather than a useful user-level concept. As Tod already pointed out, the nextset() method does have its use for stored procedures: result sets are needed to group data. If the interface would just silently move to the next set, there would be no way to identify the set boundary. I've uploaded an update of the spec. It's now at 1.1a4: http://starship.skyport.net/~lemburg/DatabaseAPI-1.1.html -- Marc-Andre Lemburg Y2000: 303 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : --------------------------------------------------------- From billtut@microsoft.com Wed Mar 3 09:09:55 1999 From: billtut@microsoft.com (Bill Tutt) Date: Wed, 3 Mar 1999 01:09:55 -0800 Subject: [DB-SIG] API Enhancements Message-ID: <4D0A23B3F74DD111ACCD00805F31D8100DB909C5@RED-MSG-50> > From: M.-A. Lemburg [mailto:mal@lemburg.com] > > > Greg Stein wrote: > > Oh... a question about the nextresult() method. Why is that exposed? > > Shouldn't the module just automatically move to the next result set > > inside the fetch* methods? I don't see the benefit of > exposing that to > > the user. It seems an artifact of the underlying database interface, > > rather than a useful user-level concept. > > As Tod already pointed out, the nextset() method does have its > use for stored procedures: result sets are needed to group data. > If the interface would just silently move to the next set, there > would be no way to identify the set boundary. > There's nothing preventing the spec from specifying something along the following: If there are multiple result sets: At the end of the first result set, fetch*() returns None. The next call to fetch*() will return the first row of the next resultset. (if any). The user definately needs to see a demarkation between the result sets as the different result sets typically have different row layouts. The prototypical example for doing things like this is returning 1:1 information in the first result set, with the second result set containing N:N information sorted on the reference mode (PK or AK)of the first result set. Bill From mal@lemburg.com Wed Mar 3 14:10:30 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 03 Mar 1999 15:10:30 +0100 Subject: [DB-SIG] API Enhancements References: <4D0A23B3F74DD111ACCD00805F31D8100DB909C5@RED-MSG-50> Message-ID: <36DD42D6.4B47FD12@lemburg.com> Bill Tutt wrote: > > > From: M.-A. Lemburg [mailto:mal@lemburg.com] > > > > > > Greg Stein wrote: > > > Oh... a question about the nextresult() method. Why is that exposed? > > > Shouldn't the module just automatically move to the next result set > > > inside the fetch* methods? I don't see the benefit of > > exposing that to > > > the user. It seems an artifact of the underlying database interface, > > > rather than a useful user-level concept. > > > > As Tod already pointed out, the nextset() method does have its > > use for stored procedures: result sets are needed to group data. > > If the interface would just silently move to the next set, there > > would be no way to identify the set boundary. > > > > There's nothing preventing the spec from specifying something along the > following: > If there are multiple result sets: > At the end of the first result set, fetch*() returns None. > The next call to fetch*() will return the first row of the next > resultset. (if any). This would work with .fetchone(), but not with fetchmany() and .fetchall() since the latter always return (possibly empty) sequences. It would also make the needed code look a bit cumbersome; with .nextset() you can write: cursor.callproc(...) while 1: ...fetch result set... if not cursor.nextset(): break > The user definately needs to see a demarkation between the result sets as > the different result sets typically have different row layouts. > > The prototypical example for doing things like this is returning 1:1 > information in the first result set, with the second result set containing > N:N information sorted on the reference mode (PK or AK)of the first result > set. -- Marc-Andre Lemburg Y2000: 303 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : --------------------------------------------------------- From gstein@lyra.org Wed Mar 3 14:27:04 1999 From: gstein@lyra.org (Greg Stein) Date: Wed, 03 Mar 1999 06:27:04 -0800 Subject: [DB-SIG] API Enhancements References: <36A32169.2ECE2BD@lemburg.com> <36D93B76.1143DEE1@lyra.org> <36DA6AB9.4DCF10BF@lemburg.com> <36DA921B.E8DDA85@lyra.org> <36DCFA1C.74C40BA0@lemburg.com> Message-ID: <36DD46B8.40E05F89@lyra.org> M.-A. Lemburg wrote: > ... > Hmm, strings wouldn't indicate order. Cursor level thread safety > implies connection level and module level. Sounds reasonable. > > > A module global stating the specification level would also be handy. > > '1.1' for modules conforming to the DBAPI 1.1 spec. If the global is not > > present, then the module is a 1.0 module. > > How about: apilevel ?! Sure. >... > > But this update is passed to the execute() method of the cursor object > > that you're talking about, isn't it? Or do you create a second cursor, > > which updates the record currently indicated by the first cursor? > > Both are possible. It depends on the database capabilities whether > this technique is of much use: e.g. you could update all the records > starting from the 10th row in a result set. Some DBs support scrolling > cursors in backward direction too, making updates relying on external > resources (e.g. file in a file system) possible. No... The semantics of execute() are that it wipes the cursor and starts a fresh operation. I *really* don't think you can change that. Therefore, you must use a second cursor to derive any benefit from named cursors. > > Isn't that kind of technique a bit difficult without a rigorous > > specification of the module's fetching behavior? Just because I called > > fetchone() doesn't mean the cursor is sitting on the first record... an > > array-fetch may have occurred, and the cursor is 90 rows later. > > Good point. fetchone() ought do what the name implies: fetch exactly > one record. Otherwise usage of cursors would be pointless anyway. NO!!! fetchone() is free to fetch as many as it likes. It merely needs to *return* one at a time. The name "fetch" is from the viewpoint of the DBAPI client. NOT the underlying module. If these named cursors are added, then fetchone() would need to alter its operation and semantics to rigorously specify where the cursor is. IMO, this is getting really ugly. I don't see that we should allow these kinds of varying semantics simply to get something of unknown benefit. > > By naming a cursor, it would seem that any array-fetching would need to > > be disabled so that a program can actually determine what record is > > being referenced by the cursor. For safety, it may even be prudent to > > disable fetchmany() and fetchall(). > > Leave that to the user of the database interface. It should > be mentioned in the spec. though. fetchmany() may actually fetch > more records than requested. fetchall() wouldn't make sense > in this context anyway. All the methods can fetch as many as they'd like. The only semantics are on the return amounts. The API provides a way to give the module array-size hints, but the module is still free to ignore them and optimize differently. > > Finally: is there a performance gain to the named-cursor technique? > > (compared to an UPDATE ... WHERE = ) > > Hard to say... it could safe a temporary table though. I am beginning to believe that these named cursors don't make a lot of sense, unless there is a way to rigorously specify fetching behavior. However, we have specifically *avoided* doing this to allow the module implementor much more freedom. I would recommend that named cursors and the associated fetching behavior specification are NOT incorporated into the API specification. >... > As Tod already pointed out, the nextset() method does have its > use for stored procedures: result sets are needed to group data. > If the interface would just silently move to the next set, there > would be no way to identify the set boundary. Agreed. Cheers, -g -- Greg Stein, http://www.lyra.org/ From mal@lemburg.com Wed Mar 3 16:26:31 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 03 Mar 1999 17:26:31 +0100 Subject: [DB-SIG] API Enhancements References: <36A32169.2ECE2BD@lemburg.com> <36D93B76.1143DEE1@lyra.org> <36DA6AB9.4DCF10BF@lemburg.com> <36DA921B.E8DDA85@lyra.org> <36DCFA1C.74C40BA0@lemburg.com> <36DD46B8.40E05F89@lyra.org> Message-ID: <36DD62B7.9A2584B@lemburg.com> Greg Stein wrote: > [Named cursors] > > I would recommend that named cursors and the associated fetching > behavior specification are NOT incorporated into the API specification. Ok, it's gone. Module implementors are free to support named cursors as extension (a note is included in the spec to point out why the omission was done). -- Marc-Andre Lemburg Y2000: 303 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : --------------------------------------------------------- From hoeink@uni-muenster.de Thu Mar 4 09:43:43 1999 From: hoeink@uni-muenster.de (Tobias Höink) Date: Thu, 04 Mar 1999 10:43:43 +0100 Subject: [DB-SIG] Python and database problems Message-ID: <36DE55CF.5748@uni-muenster.de> Dear "Phythons", my name is Tobias Hoeink. I am programming for the German Society of Physicists in Muenster, Germany. For registration we use CGI-Scripts written in Python. We read information from HTML forms and put them into an Access database. So far there is no problem, everything works fine. We use two fields for dates in this Access database and this is where something very strange happens: While reading dates out our database the date somehow changes into the date of the day before. As this seems to be a systematical error which occurs somewhere between Python and ODBC we were not able to handle it. To get in tough with our problem please look at http://dpg-akf.uni-muenster.de/dpg/dpg_liste.html and search e.g. for member number (Teilnehmernummer) 478. The returning page will show you two dates. Both are exact one day before the day saved in our database. If you search the name "Nink" (field Nachname) an other listing will be presented with the same effect. I asure you that the original dates in this recordset are saved with anmeldedatum = 02/25/199 and kartendatum = 03/25/1999. This problem occurs on any date we read out of the database! Please let us know what the mistake is and how we can solve the problems that are concerned with that. I would also be interested if we are the only ones presented to this effect. Thank you very much. I hope to hear from you soon. Sincerely Yours, Tobias Hoeink -- ************************************************* Tobias Höink Heekweg 12, E408 48161 Münster GERMANY phone: +49-251-868852 email: hoeink@uni-muenster.de ************************************************* From HOEINK@nwz.uni-muenster.de Thu Mar 4 10:22:47 1999 From: HOEINK@nwz.uni-muenster.de (Tobias Hoeink) Date: Thu, 4 Mar 1999 11:22:47 +0100 Subject: [DB-SIG] Python and database problems Message-ID: Dear "pythons", this is an additionally mail to the first one sent a few minutes ago. Greg Stein wrote me and asked if it could be a problem of timezones. Indeed it was. Editing the time field of our database with a time of 12.00 (noon) instead of 0.00 (midnight) the correct dates are displayed when we read them. Thanks to all of you would have answered. This was my first experience with python help mailing and I am impressed. Really! Good luck to all of you and many greetings from Muenster, Germany, sincerely your Tobias Hoeink From gstein@lyra.org Thu Mar 4 10:40:14 1999 From: gstein@lyra.org (Greg Stein) Date: Thu, 04 Mar 1999 02:40:14 -0800 Subject: [DB-SIG] annotated 1.1 spec / feedback Message-ID: <36DE630E.360AEE62@lyra.org> I've prepared an annotated 1.1 spec (it lists changes since 1.0). It is located at: http://www.lyra.org/greg/anno-DatabaseAPI-1.1.html Please let me know if I've missed any changes. I'll update the page to follow Marc-Andre's releases of the API spec. Just watch for the version string at the top to see if I've updated yet. While annotating, I've also prepared some feedback items: * Connect() should be lower-cased to connect(). Most Python module-level functions are all lower case; we should follow suit. * the connect() parameters should be called "examples" rather than "usually". I've seen a db module or two where they try to follow suit and munge together parameters into a string, simply to have to tear it apart. We should not try to avoid imposing on the connection parameters. For example: MySQL should take four params: host, user, password, database. "dsn" is not "typical" for it, so we should confuse people into attempting to abide by it. * the parameters should be positional, rather than keyword-based. keywords can be optional, but we should not require the params to be keyword-based. * I think the exception should only be available thru dbi. Having them in two places may lead to using different classes, thus breaking the inheritance/catching of the exceptions. I understand that the module implementor can get the other class object and expose it as a value, but there'd be hard-to-find problems if they skipped that step. * maybe we should discuss punting dbi. module distribution may be easier without it. While it is handy for clients that use more than one DB module, I can easily envision a python module that exports the appropriate set for the DB being used. * cursor.rowcount is new, and is extraneous. The result of execute() should be used. * the params to callproc should be: callproc(procname [, params]). * I think we should clean up the description attribute if possible. The 7-tuple is/was very Oracle-specific. * The old description for execute() returning None was clearer. Note that a SELECT can return no columns/rows (where your description says > 0). * For the >=0 return value from execute(), it should be clarified with the word "affect" since you could delete them or insert them (which are not "touching" rows). * in general, I think the 1.1 change to the return value description didn't really clear things up. Basing the descriptions on returning columns/rows doesn't make much sense to me. Instead, I liked the description based on the *type* of statement that was performed. Under this style, a SELECT that doesn't produce columns/rows would return 1, but that is just wrong. * execute()'s description should change back to the terms "sequence" and "sequence of sequences" for the parameters. There is no reason to force them to be tuples or lists. * the part about cursor movement in fetchone() should be totally removed. fetchone() is about the CLIENT fetching a single row. It should NOT affect how the underlying module performs its fetches. * the fetch* methods should say they return a sequence or a sequence of sequences. We should not force them to be tuples or lists of tuples. Some DB modules may want to have additional semantics or capabilities on their results. As long as it obeys sequence behavior, then the spec should be happy. The arraysize attribute's description also refers to a list of tuples, and should be updated. * dbiDate values probably shouldn't *contain* mxDateTime values. Instead, mxDateTime should be used in place of dbiDate. The point of dbiDate was to provide a way to discriminate between an integer and a date (unix ticks). mxDateTime can fill the same role. * we may want to strike the "ending in db" comment. seems silly in retrospec and (empirically) has not been followed anyhow. Cheers, -g -- Greg Stein, http://www.lyra.org/ From mal@lemburg.com Thu Mar 4 13:08:34 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 04 Mar 1999 14:08:34 +0100 Subject: [DB-SIG] annotated 1.1 spec / feedback References: <36DE630E.360AEE62@lyra.org> Message-ID: <36DE85D2.FD1B6AE@lemburg.com> Greg Stein wrote: > > I've prepared an annotated 1.1 spec (it lists changes since 1.0). It is > located at: > > http://www.lyra.org/greg/anno-DatabaseAPI-1.1.html > > Please let me know if I've missed any changes. I'll update the page to > follow Marc-Andre's releases of the API spec. Just watch for the version > string at the top to see if I've updated yet. Thanks. > While annotating, I've also prepared some feedback items: > > * Connect() should be lower-cased to connect(). Most Python module-level > functions are all lower case; we should follow suit. Hmm, this is probably a religous question: I tend to always use names starting with a capital letter for classes and constructors. > * the connect() parameters should be called "examples" rather than > "usually". I've seen a db module or two where they try to follow suit > and munge together parameters into a string, simply to have to tear it > apart. We should not try to avoid imposing on the connection parameters. > For example: MySQL should take four params: host, user, password, > database. "dsn" is not "typical" for it, so we should confuse people > into attempting to abide by it. Ok. > * the parameters should be positional, rather than keyword-based. > keywords can be optional, but we should not require the params to be > keyword-based. But keywords are so much simpler to maintain: the order doesn't matter, only the wording. They also serve as nice way to document what the parameter stands for, e.g. connect('degas','miro','kline') isn't too easy to comprehend without first looking at the interface's docs, yet connect(host='degas',user='miro',password='kline') is. If you fear the programming overhead, I can post a few C macros that make dealing with keywords in C really simple. > * I think the exception should only be available thru dbi. Having them > in two places may lead to using different classes, thus breaking the > inheritance/catching of the exceptions. I understand that the module > implementor can get the other class object and expose it as a value, but > there'd be hard-to-find problems if they skipped that step. It is actually quite simple to add references to those object in dbi. The reason for having them exposed through the module is that when using multiple databases, execption catching is likely to fail (the dbi approach doesn't handle this case too well anyway: packages might help here). > * maybe we should discuss punting dbi. module distribution may be easier > without it. While it is handy for clients that use more than one DB > module, Actually, it makes things harder for those cases (see above). mxODBC supports opening connections to many different databases and each has its own version of the dbi module because each subpackage uses its own set of exception classes. > I can easily envision a python module that exports the > appropriate set for the DB being used. Right: stick it all into the database interface package. > * cursor.rowcount is new, and is extraneous. The result of execute() > should be used. The execute() method returns None in case there are rows available. It does not return the number of rows in the result set. > * the params to callproc should be: callproc(procname [, params]). Right. > * I think we should clean up the description attribute if possible. The > 7-tuple is/was very Oracle-specific. Ok, things like display_size and internal_size are not really relevant, but why break existing code expecting these 7 entries (even if it probably does not use all of them) ? > * The old description for execute() returning None was clearer. Note > that a SELECT can return no columns/rows (where your description says > > 0). The None return value is to indicate that data is available to be fetched vie the fetchXXX() methods. If no rows where produced, something else should be returned, e.g. 1. The reason I changed the description was that the previous one wasn't easy to implement: it is not clear whether a statement is DDL, DML or DQL without parsing it... > * For the >=0 return value from execute(), it should be clarified with > the word "affect" since you could delete them or insert them (which are > not "touching" rows). Ok. > * in general, I think the 1.1 change to the return value description > didn't really clear things up. Basing the descriptions on returning > columns/rows doesn't make much sense to me. Instead, I liked the > description based on the *type* of statement that was performed. Under > this style, a SELECT that doesn't produce columns/rows would return 1, > but that is just wrong. See above. Without parsing there isn't much you can do as module implementor to get the return value right. Maybe we should drop the return value altogether: rowcount gives the number of rows in the result set (even 0 for SELECTs that didn't produce any rows) or the number of affected rows. > * execute()'s description should change back to the terms "sequence" and > "sequence of sequences" for the parameters. There is no reason to force > them to be tuples or lists. I thought about this: there is no way for the method to tell whether it was passed a sequence of sequences that is meant to be a sequence of parameter sequences or just a single sequence with sequence entries, e.g. ('abc',) would break under that definition. Is there really such a big need to be able to pass *any* sequence as parameter ? > * the part about cursor movement in fetchone() should be totally > removed. fetchone() is about the CLIENT fetching a single row. It should > NOT affect how the underlying module performs its fetches. That renders cursors useless... are you sure you want this ? > * the fetch* methods should say they return a sequence or a sequence of > sequences. We should not force them to be tuples or lists of tuples. > Some DB modules may want to have additional semantics or capabilities on > their results. As long as it obeys sequence behavior, then the spec > should be happy. The arraysize attribute's description also refers to a > list of tuples, and should be updated. Just thought it would be a good idea to keep execute() parameter input and fetchXXX() output in sync. > * dbiDate values probably shouldn't *contain* mxDateTime values. > Instead, mxDateTime should be used in place of dbiDate. The point of > dbiDate was to provide a way to discriminate between an integer and a > date (unix ticks). mxDateTime can fill the same role. I'll clarify that... > * we may want to strike the "ending in db" comment. seems silly in > retrospec and (empirically) has not been followed anyhow. Ok. Note: I've updated the spec to 1.1a6. -- Marc-Andre Lemburg Y2000: 302 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : --------------------------------------------------------- From Max.Slimmer@RightWorks.com Thu Mar 4 20:15:51 1999 From: Max.Slimmer@RightWorks.com (Max Slimmer) Date: Thu, 4 Mar 1999 12:15:51 -0800 Subject: [DB-SIG] Python and database problems Message-ID: <7BB27EA0B07AD211B7E800A0C9B2143D0C0464@kirk2.corp.mediak.com> This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01BE667B.CCE96B14 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable you might be having a problem in the displaying the dates, have you = tried to execute a query from the interactive python window >>> import dbi, odbc >>> db =3D odbc.ODBC('xxx/xxx/xxx') >>> cur =3D db.cursor() >>> r =3D cur.execute("Select * from someTable") >>> ans =3D cur.fetchone() then look at values in ans, you have to convert dbiDate values to = readable date first. > -----Original Message----- > From: Tobias H=F6ink [mailto:hoeink@uni-muenster.de] > Sent: Thursday, March 04, 1999 1:44 AM > To: db-sig@python.org > Subject: [DB-SIG] Python and database problems >=20 >=20 > Dear "Phythons", >=20 > my name is Tobias Hoeink. I am programming for the German Society of=20 > Physicists in Muenster, Germany. For registration we use CGI-Scripts=20 > written in Python. We read information from HTML forms and=20 > put them into=20 > an Access database. So far there is no problem, everything=20 > works fine.=20 > We use two fields for dates in this Access database and this is where = > something very strange happens: > While reading dates out our database the date somehow changes=20 > into the=20 > date of the day before. As this seems to be a systematical=20 > error which=20 > occurs somewhere between Python and ODBC we were not able to=20 > handle it. >=20 > To get in tough with our problem please look at > http://dpg-akf.uni-muenster.de/dpg/dpg_liste.html > and search e.g. for member number (Teilnehmernummer) 478. The=20 > returning > page=20 > will show you two dates. Both are exact one day before the=20 > day saved in=20 > our database. If you search the name "Nink" (field Nachname) an other = > listing will be presented with the same effect. I asure you that the=20 > original dates in this recordset are saved with anmeldedatum=20 > =3D 02/25/199 > and=20 > kartendatum =3D 03/25/1999.=20 > This problem occurs on any date we read out of the database! >=20 > Please let us know what the mistake is and how we can solve=20 > the problems=20 > that are concerned with that.=20 > =20 > I would also be interested if we are the only ones presented to this > effect. >=20 > Thank you very much. >=20 > I hope to hear from you soon. =20 >=20 > Sincerely Yours, >=20 > Tobias Hoeink > --=20 > ************************************************* > Tobias H=F6ink =09 > Heekweg 12, E408 =09 > 48161 M=FCnster=09 > GERMANY =09 > phone: +49-251-868852 =09 > email: hoeink@uni-muenster.de=09 > ************************************************* >=20 > _______________________________________________ > DB-SIG maillist - DB-SIG@python.org > http://www.python.org/mailman/listinfo/db-sig >=20 ------_=_NextPart_001_01BE667B.CCE96B14 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable RE: [DB-SIG] Python and database problems

you might be having a problem in the displaying the = dates, have you tried to execute a query from the interactive python = window

>>> import dbi, odbc
>>> db =3D odbc.ODBC('xxx/xxx/xxx')
>>> cur =3D db.cursor()
>>> r =3D cur.execute("Select * from = someTable")
>>> ans =3D cur.fetchone()

then look at values in ans, you have to convert = dbiDate values to readable date first.

> -----Original Message-----
> From: Tobias H=F6ink [mailto:hoeink@uni-muenster.de= ]
> Sent: Thursday, March 04, 1999 1:44 AM
> To: db-sig@python.org
> Subject: [DB-SIG] Python and database = problems
>
>
> Dear "Phythons",
>
> my name is Tobias Hoeink. I am programming for = the German Society of
> Physicists in Muenster, Germany. For = registration we use CGI-Scripts
> written in Python. We read information from = HTML forms and
> put them into
> an Access database. So far there is no problem, = everything
> works fine.
> We use two fields for dates in this Access = database and this is where
> something very strange happens:
> While reading dates out our database the date = somehow changes
> into the
> date of the day before. As this seems to be a = systematical
> error which
> occurs somewhere between Python and ODBC we = were not able to
> handle it.
>
> To get in tough with our problem please look = at
> http://dpg-akf.uni-muenster.de/dpg/dpg_liste.html<= /FONT>
> and search e.g. for member number = (Teilnehmernummer) 478. The
> returning
> page
> will show you two dates. Both are exact one day = before the
> day saved in
> our database. If you search the name = "Nink" (field Nachname) an other
> listing will be presented with the same effect. = I asure you that the
> original dates in this recordset are saved with = anmeldedatum
> =3D 02/25/199
> and
> kartendatum =3D 03/25/1999.
> This problem occurs on any date we read out of = the database!
>
> Please let us know what the mistake is and how = we can solve
> the problems
> that are concerned with that.
> 
> I would also be interested if we are the only = ones presented to this
> effect.
>
> Thank you very much.
>
> I hope to hear from you soon. 
>
>         = Sincerely Yours,
>
>          = ;       Tobias Hoeink
> --
> = *************************************************
>       Tobias = H=F6ink        =            

>       Heekweg 12, = E408      =        
>       48161 = M=FCnster      
>       = GERMANY          
>       phone: = +49-251-868852        
>       email: = hoeink@uni-muenster.de      
> = *************************************************
>
> = _______________________________________________
> DB-SIG maillist  -  = DB-SIG@python.org
> http://www.python.org/mailman/listinfo/db-sig
>

------_=_NextPart_001_01BE667B.CCE96B14-- From mal@lemburg.com Wed Mar 10 11:36:26 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 10 Mar 1999 12:36:26 +0100 Subject: [DB-SIG] ANN: mxDateTime Package - Version 1.2.0 Message-ID: <36E6593A.23DBBF28@lemburg.com> ANNOUNCING: mxDateTime Version 1.2.0 A Python Extension Package providing generic Date/Time Types WHAT IT IS: mxDateTime is an extension package that provides three new object types, DateTime, DateTimeDelta and RelativeDateTime, which let you store and handle date/time values in a much more natural way than by using Unix ticks (seconds since 1.1.70 0:00 UTC; the encoding used by the time module). You can add, subtract and even multiply instances, pickle and copy them and convert the results to strings, COM dates, ticks and some other more esoteric values. In addition, there are several convenient construtors, formatters and parsers at hand to greatly simplify dealing with dates and times in real-world applications. In addition to providing an easy-to-use Python interface the package also exports a comfortable C API interface for other extensions to build upon. This is especially interesting for database applications which often have to deal with date/time values (the mxODBC package is one example of an extension using this interface). WHAT'S NEW ? Version 1.2.0 introduces lots of new features. Most noteably, an experimental date/time parser, calendar support (Julian and Gregorian) and experimental support for B.C.E. dates, giving you a date range of -5851455-01-01 00:00:00.00 to 5867440-12-31 00:00:00.00 (should be enough for most needs and certainly does away with Y2K type problems ;). Also included is a work-around for platforms that do not have the timegm() C lib API. WHERE CAN I GET IT ? The full documentation and instructions for downloading and installing can be found at: http://starship.skyport.net/~lemburg/mxDateTime.html WHAT DOES IT COST ? It comes with a Python-style license, but is otherwise free for commercial and non-commercial use. WHERE CAN I GET SUPPORT ? I amd offering commercial support for this package through Python Professional Services Inc. (http://www.pythonpros.com). Look on their support pages for details or contact me directly. REFERENCE:

mxODBC 1.0.1 - ODBC Interface. (11-Mar-1999) -- Marc-Andre Lemburg Y2000: 295 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : --------------------------------------------------------- From adustman@comstar.net Thu Mar 11 17:34:34 1999 From: adustman@comstar.net (Andy Dustman) Date: Thu, 11 Mar 1999 12:34:34 -0500 (EST) Subject: [DB-SIG] DB API 1.1 Spec In-Reply-To: <36E7932B.51592852@lemburg.com> Message-ID: Okay, a few last minute comments as I skim through there: Thread safety: All those values need to be bumped up a notch to make room for the real 0: 0: Threads may not share the module. And perhaps the point needs to be made that "share" means "two threads using the same resource without a mutex". With a mutex, and the right user locking code, anything ought to be shareable. -- Andy Dustman (ICQ#32922760) You should always say "spam" and "eggs" ComStar Communications Corp. instead of "foo" and "bar" (706) 549-7689 | PGP KeyID=0xC72F3F1D in Python examples. (Mark Lutz) From sjturner@ix.netcom.com Thu Mar 11 19:29:33 1999 From: sjturner@ix.netcom.com (Stephen J. Turner) Date: Thu, 11 Mar 1999 14:29:33 -0500 Subject: [DB-SIG] DB API 1.1 Spec References: <36E7932B.51592852@lemburg.com> Message-ID: <36E8199C.63457DC@ix.netcom.com> Hello all, Calling the new spec "1.1" would to me suggest backward compatibility with the 1.0 spec. If this were true, then I would expect that applications originally written to use a 1.0 implementation should work unmodified with a 1.1 implementation. However, some of the proposed 1.1 changes seem to be incompatible with 1.0, including: - renaming the connection function from to Connect - requiring keyword args for the connection function - the new exception class hierarchy - moving the callproc method from the Connection object to the Cursor object Should we therefore call the new spec "2.0" instead? Regards, Stephen -- Stephen J. Turner From mal@lemburg.com Thu Mar 11 21:41:23 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 11 Mar 1999 22:41:23 +0100 Subject: [DB-SIG] DB API 1.1 Spec References: Message-ID: <36E83883.3B13278F@lemburg.com> Andy Dustman wrote: > > Okay, a few last minute comments as I skim through there: > > Thread safety: All those values need to be bumped up a notch to make room > for the real 0: > > 0: Threads may not share the module. Ok. Forgot that one ;-) > And perhaps the point needs to be made that "share" means "two threads > using the same resource without a mutex". With a mutex, and the right user > locking code, anything ought to be shareable. Hmm, not sure about that one: the mutex could not be coarse grained enough to lock out potential problems ;-) Anyway, I'll add a note about the code "being thread safe all by itself". -- Marc-Andre Lemburg Y2000: 295 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : --------------------------------------------------------- From gstein@lyra.org Thu Mar 11 23:23:27 1999 From: gstein@lyra.org (Greg Stein) Date: Thu, 11 Mar 1999 15:23:27 -0800 Subject: [DB-SIG] DB API 1.1 Spec References: <36E83883.3B13278F@lemburg.com> Message-ID: <36E8506F.4285F5B6@lyra.org> M.-A. Lemburg wrote: > Andy Dustman wrote: > >... > > And perhaps the point needs to be made that "share" means "two threads > > using the same resource without a mutex". With a mutex, and the right user > > locking code, anything ought to be shareable. > > Hmm, not sure about that one: the mutex could not be coarse grained > enough to lock out potential problems ;-) Anyway, I'll add a note > about the code "being thread safe all by itself". Even with a mutex, the MSFT SQL Server connector cannot be used from a different thread. If a thread creates a connection, then that is the only thread that can use it. So: the doc shouldn't simply say that a mutex can solve the threading issues. It could/should reflect one of three possibilities: fully thread safe, usuable across threads with a mutex, not usable across threads. Cheers, -g -- Greg Stein, http://www.lyra.org/ From mal@lemburg.com Fri Mar 12 09:41:29 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Fri, 12 Mar 1999 10:41:29 +0100 Subject: [DB-SIG] DB API 1.1 Spec References: <36E7932B.51592852@lemburg.com> <36E8199C.63457DC@ix.netcom.com> Message-ID: <36E8E149.6F7C1154@lemburg.com> Stephen J. Turner wrote: > > Hello all, > > Calling the new spec "1.1" would to me suggest backward compatibility > with the 1.0 spec. If this were true, then I would expect that > applications originally written to use a 1.0 implementation should work > unmodified with a 1.1 implementation. > > However, some of the proposed 1.1 changes seem to be incompatible with > 1.0, including: > - renaming the connection function from to Connect It's been lower-cased again... connect(). AFAIK, the convention of calling the connector was not always adhered to, so this may not be a big change. > - requiring keyword args for the connection function Note that the module implementor can put the keywords into a certain order that makes the constructor behave just like it did for 1.0, e.g. dsn='',user='',passwd=''. Hmm, maybe we should fix an order for the most important keyword arguments. > - the new exception class hierarchy The old "error" exception could be aliased to "Error". > - moving the callproc method from the Connection object to the Cursor > object This may look like a big change, but since most interfaces did not implement callproc() in the first place, I guess it's as bad as it looks. > Should we therefore call the new spec "2.0" instead? To be honest: I don't really care about the version number. Note that in Python's own numbering scheme small changes like the ones above are indeed allowed for minor revisions. Other opinions ? Maybe we should add an appendix to the spec about the changes from 1.0 to 1.1. -- Marc-Andre Lemburg Y2000: 294 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : --------------------------------------------------------- From mal@lemburg.com Fri Mar 12 09:43:56 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Fri, 12 Mar 1999 10:43:56 +0100 Subject: [DB-SIG] DB API 1.1 Spec References: <36E83883.3B13278F@lemburg.com> <36E8506F.4285F5B6@lyra.org> Message-ID: <36E8E1DC.5C28C02A@lemburg.com> Greg Stein wrote: > > M.-A. Lemburg wrote: > > Andy Dustman wrote: > > >... > > > And perhaps the point needs to be made that "share" means "two threads > > > using the same resource without a mutex". With a mutex, and the right user > > > locking code, anything ought to be shareable. > > > > Hmm, not sure about that one: the mutex could not be coarse grained > > enough to lock out potential problems ;-) Anyway, I'll add a note > > about the code "being thread safe all by itself". > > Even with a mutex, the MSFT SQL Server connector cannot be used from a > different thread. If a thread creates a connection, then that is the > only thread that can use it. > > So: the doc shouldn't simply say that a mutex can solve the threading > issues. It could/should reflect one of three possibilities: fully thread > safe, usuable across threads with a mutex, not usable across threads. I added a note about this to the spec: Sharing in the above context means that two threads may use a resource without wrapping it using a mutex semaphore to implement resource locking. Note that you cannot always make external resources thread safe by managing access using a mutex: the resource may rely on global variables or other external sources that are beyond your control. Is this ok with both of you ? As always the updated copy is at: http://starship.skyport.net/~lemburg/DatabaseAPI-1.1.html -- Marc-Andre Lemburg Y2000: 294 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : --------------------------------------------------------- From gstein@lyra.org Fri Mar 12 11:07:09 1999 From: gstein@lyra.org (Greg Stein) Date: Fri, 12 Mar 1999 03:07:09 -0800 Subject: [DB-SIG] DB API 1.1 Spec References: <36E7932B.51592852@lemburg.com> Message-ID: <36E8F55D.7AFCDB3A@lyra.org> M.-A. Lemburg wrote: > > Just wanted to remind you that the deadline for the last call is > this Friday. Reminder: Friday is last call for this round. The spec then receives all updates to produce a "final draft" for consensus sign-off on March 26. The new spec is posted on March 29. > The latest version of the proposal can be found at: > > http://starship.skyport.net/~lemburg/DatabaseAPI-1.1.html > > Feel free to comment on it. Greg Stein has created an annotated > version which is available at: > > http://www.lyra.org/greg/anno-DatabaseAPI-1.1.html I have updated the annotated version to match 1.1a7 (Marc's most recent version). > Some things still open for discussion (some of which may go into > 1.2): > > · punting dbi (by folding its features into the database modules > themselves) Let's do this. This would clean things up quite a bit. I do not believe the dbi thing provided all the benefits that it could have. And compared to the cost, we should nuke it. > · developing a few more common sets of types, e.g. for passing > raw data, monetary data, etc. which may be used by the interface > modules to bind the correct type I'm somewhat ambivalent here. What might be interesting is to define how an object must respond, and leave it to implementors to use a class instance or builtin type. For example: class MyDBValue: pass v = MyDBValue() v.dbtype = 'RAW' v.value = 'abcdef' The DBAPI would specify that "object.dbtype == 'RAW'" means that the object represents a RAW value, which is a read-buffer in the .value attribute. == 'CURRENCY' means a monetary unit where .value is an integer or float or something. etc. > · designing a common package layout for the database packages If we nuke dbi, then a package is not required. However, I believe it would be fair to spec a "typical" pair of .py and .so modules (e.g. somedb.py and _somedb.so). Cheers, -g -- Greg Stein, http://www.lyra.org/ From gstein@lyra.org Fri Mar 12 11:52:09 1999 From: gstein@lyra.org (Greg Stein) Date: Fri, 12 Mar 1999 03:52:09 -0800 Subject: [DB-SIG] annotated 1.1 spec / feedback References: <36DE630E.360AEE62@lyra.org> <36DE85D2.FD1B6AE@lemburg.com> Message-ID: <36E8FFE9.2930E453@lyra.org> M.-A. Lemburg wrote: >... > > While annotating, I've also prepared some feedback items: > > > > * Connect() should be lower-cased to connect(). Most Python module-level > > functions are all lower case; we should follow suit. > > Hmm, this is probably a religous question: I tend to always use names > starting with a capital letter for classes and constructors. As a reference for the "Python-way", I looked at anydbm.py. The open() method returns a new object, and it is lower-cased. Since the spec does not state that connect() is a class name, then IMO it should be lower-cased. If we said it was a class, then I'd agree with the upper-case. Moot anyhow, as I see you've lower-cased it in the current spec. thx. > > * the parameters should be positional, rather than keyword-based. > > keywords can be optional, but we should not require the params to be > > keyword-based. > > But keywords are so much simpler to maintain: the order doesn't > matter, only the wording. They also serve as nice way to document > what the parameter stands for, e.g. > > connect('degas','miro','kline') > > isn't too easy to comprehend without first looking at the interface's > docs, yet > > connect(host='degas',user='miro',password='kline') > > is. > > If you fear the programming overhead, I can post a few C macros > that make dealing with keywords in C really simple. I believe we can only require keywords if we prepare an appendix to the specification. As a module implementor, I do not do keywords... I always use positional because they are much simpler for C-implemented modules. If we do not have a simple reference, then I believe we cannot require keywords. Even then, I am hesitant to do so. [and yes, I fully understand their benefit for the user of the module; let the user fix it in Python if it bugs them] >... > > * maybe we should discuss punting dbi. module distribution may be easier > > without it. While it is handy for clients that use more than one DB > > module, > > Actually, it makes things harder for those cases (see above). mxODBC > supports opening connections to many different databases and each > has its own version of the dbi module because each subpackage uses > its own set of exception classes. If we tossed the dbi module, then each database would incorporate its dbi into the module itself. I don't understant what the problem is here. I'm definitely leaning towards axing dbi. > > I can easily envision a python module that exports the > > appropriate set for the DB being used. > > Right: stick it all into the database interface package. I'd like to see: import mysqldb ... except mysqldb.IntegrityError: ... if curs.description[0][0] == mysqldb.STRING: ... > > * cursor.rowcount is new, and is extraneous. The result of execute() > > should be used. > > The execute() method returns None in case there are rows available. > It does not return the number of rows in the result set. I am not sure that all databases can provide you the rowcount after an execute(). You may need to fetch them first. Therefore, the .rowcount couldn't even be implemented. This means that the only real way to know how many rows there are is to fetch them -- rowcount could not be relied upon. > > * I think we should clean up the description attribute if possible. The > > 7-tuple is/was very Oracle-specific. > > Ok, things like display_size and internal_size are not really > relevant, but why break existing code expecting these 7 entries (even > if it probably does not use all of them) ? I believe we should alter the specification, but that databases can continue to return an appropriate number of fields. For example: can we require two fields -- name and type. Beyond that, it is up to the database module. Implementors can choose to continue with the 7-tuple or bring it down to what they really support. > > * The old description for execute() returning None was clearer. Note > > that a SELECT can return no columns/rows (where your description says > > > 0). > > The None return value is to indicate that data is available > to be fetched vie the fetchXXX() methods. If no rows where produced, > something else should be returned, e.g. 1. I do not believe that a database can always tell that rows were or were not produced. You may need to fetch them first. Therefore, returning 1 is not possible for certain modules. Okay. I just looked at DCOracle. The execute() method does just that... an execute. No fetching, so no way to tell what the row count is. Admittedly, there may be something in the OCI interface, or in the V8 Oracle interfaces to do so. Unless sometime can point out that the Oracle interfaces support knowing the row count at execute() time, then we cannot rely on your .rowcount and return-value changes. Even so, I do not like seeing a SELECT returning "1" *ever*. That just doesn't fit right. > The reason I changed the description was that the previous one > wasn't easy to implement: it is not clear whether a statement > is DDL, DML or DQL without parsing it... Untrue. It is easy to implement -- you can always tell from the first word in all cases. > > * in general, I think the 1.1 change to the return value description > > didn't really clear things up. Basing the descriptions on returning > > columns/rows doesn't make much sense to me. Instead, I liked the > > description based on the *type* of statement that was performed. Under > > this style, a SELECT that doesn't produce columns/rows would return 1, > > but that is just wrong. > > See above. Without parsing there isn't much you can do as module > implementor to get the return value right. Maybe we should drop the > return value altogether: rowcount gives the number of rows in the > result set (even 0 for SELECTs that didn't produce any rows) or the > number of affected rows. As I mentioned, I think the rowcount is not possible for all databases that use a pure execute/fetch model like Oracle. > > * execute()'s description should change back to the terms "sequence" and > > "sequence of sequences" for the parameters. There is no reason to force > > them to be tuples or lists. > > I thought about this: there is no way for the method to tell whether > it was passed a sequence of sequences that is meant to be a > sequence of parameter sequences or just a single sequence with > sequence entries, e.g. ('abc',) would break under that definition. > > Is there really such a big need to be able to pass *any* sequence > as parameter ? Yes. What if you have a very large dataset which is encapsulated within a class instance? Let's say it responds properly to the sequence behavior, specifically so that it can be inserted without having to construct a big ol' list to do so. I also can easily see rows that identified by class instances or other builtin types, rather than always being tuples or lists. It certainly does hurt the implementor to use PySequence_GetItem() rather than PyTuple_GetItem(). An alternative would be an "insert" method that specifically takes a sequences of rows (where each row is a sequence). This would eliminate the confusion between a sequence-of-sequence being a single row or a set of rows. I believe the INSERT is the only case where a sequence input makes sense. > > * the part about cursor movement in fetchone() should be totally > > removed. fetchone() is about the CLIENT fetching a single row. It should > > NOT affect how the underlying module performs its fetches. > > That renders cursors useless... are you sure you want this ? Absolutely. Positively. There is no way that we can or should require single-row fetching. Look at DCOracle. It uses the API nearly to its full extent (it doesn't use input or output sizes to optimize its binding, though). Requiring fetchone() to disable the array fetching would be a mistake, let alone the fact that when you execute() the array-fetch there is no way to know whether the person will be using fetchone or not! > > * the fetch* methods should say they return a sequence or a sequence of > > sequences. We should not force them to be tuples or lists of tuples. > > Some DB modules may want to have additional semantics or capabilities on > > their results. As long as it obeys sequence behavior, then the spec > > should be happy. The arraysize attribute's description also refers to a > > list of tuples, and should be updated. > > Just thought it would be a good idea to keep execute() parameter > input and fetchXXX() output in sync. Right. And I think they should all sync up to be a sequence-of-sequences. I can see an implementor wanting to create a high-performance binding against a database that serves up values directly from the output buffers rather than jamming them into lists of tuples (with the requisite allocations). Jim Fulton has argued for a long while that the module should be able to return a highly functional result set. As long as it obeys the sequence-of-sequence behavior, then he could now do so, yet still "follow the spec". thx! -g -- Greg Stein, http://www.lyra.org/ From mal@lemburg.com Fri Mar 12 12:43:40 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Fri, 12 Mar 1999 13:43:40 +0100 Subject: [DB-SIG] DB API 1.1 Spec References: <36E7932B.51592852@lemburg.com> <36E8F55D.7AFCDB3A@lyra.org> Message-ID: <36E90BFC.70D08828@lemburg.com> Greg Stein wrote: > > > The latest version of the proposal can be found at: > > > > http://starship.skyport.net/~lemburg/DatabaseAPI-1.1.html > > > > Feel free to comment on it. Greg Stein has created an annotated > > version which is available at: > > > > http://www.lyra.org/greg/anno-DatabaseAPI-1.1.html > > I have updated the annotated version to match 1.1a7 (Marc's most recent > version). > > > Some things still open for discussion (some of which may go into > > 1.2): > > > > · punting dbi (by folding its features into the database modules > > themselves) > > Let's do this. This would clean things up quite a bit. I do not believe > the dbi thing provided all the benefits that it could have. And compared > to the cost, we should nuke it. Ok, I'll fold the dbi symbols into the interface spec. > > · developing a few more common sets of types, e.g. for passing > > raw data, monetary data, etc. which may be used by the interface > > modules to bind the correct type > > I'm somewhat ambivalent here. > > What might be interesting is to define how an object must respond, and > leave it to implementors to use a class instance or builtin type. For > example: > > class MyDBValue: > pass > v = MyDBValue() > v.dbtype = 'RAW' > v.value = 'abcdef' > > The DBAPI would specify that "object.dbtype == 'RAW'" means that the > object represents a RAW value, which is a read-buffer in the .value > attribute. > > == 'CURRENCY' means a monetary unit where .value is an integer or float > or something. > > etc. It would be a lot easier for the module implementor to just do a type check, but I agree this would also make the setup a little less flexible. I'd propose to leave the decision to the implementor and merely fix the names of the constructors to be used so that a programmer can rely on those being available through the interface module, e.g. Date(year,month,day) for date values Time(hour,minute,second) for time values Timestamp(year,month,day,hour,minute,second) for time stamps Raw(string_data) for raw values Currency(value,currency_name_as_string='') for monetary values etc. These constructors will then produce objects that the module can handle as input values. Output values *should* be of the same type, they need not necessarily be though (may be a little to open...). > > · designing a common package layout for the database packages > > If we nuke dbi, then a package is not required. However, I believe it > would be fair to spec a "typical" pair of .py and .so modules (e.g. > somedb.py and _somedb.so). True. Though I can picture an OO abstraction layer on top of the API, which would then be distributed as package... that's for a future revision... -- Marc-Andre Lemburg Y2000: 294 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : --------------------------------------------------------- From mal@lemburg.com Sat Mar 13 20:56:28 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Sat, 13 Mar 1999 21:56:28 +0100 Subject: [DB-SIG] annotated 1.1 spec / feedback References: <36DE630E.360AEE62@lyra.org> <36DE85D2.FD1B6AE@lemburg.com> <36E8FFE9.2930E453@lyra.org> Message-ID: <36EAD0FC.65B8B3AF@lemburg.com> Greg Stein wrote: > > M.-A. Lemburg wrote: > > > * the parameters should be positional, rather than keyword-based. > > > keywords can be optional, but we should not require the params to be > > > keyword-based. > > > > But keywords are so much simpler to maintain: the order doesn't > > matter, only the wording. They also serve as nice way to document > > what the parameter stands for, e.g. > > > > connect('degas','miro','kline') > > > > isn't too easy to comprehend without first looking at the interface's > > docs, yet > > > > connect(host='degas',user='miro',password='kline') > > > > is. > > > > If you fear the programming overhead, I can post a few C macros > > that make dealing with keywords in C really simple. > > I believe we can only require keywords if we prepare an appendix to the > specification. As a module implementor, I do not do keywords... I always > use positional because they are much simpler for C-implemented modules. > > If we do not have a simple reference, then I believe we cannot require > keywords. Even then, I am hesitant to do so. > > [and yes, I fully understand their benefit for the user of the module; > let the user fix it in Python if it bugs them] Actually, they are just as simple to implement as positional parameters. Here is some code from an extension I wrote: Py_C_Function_WithKeywords( mxBeeIndex_BeeStringIndex, "BeeStringIndex(filename,keysize,dupkeys=0,filemode=0,sectorsize=512)\n\n" ) { char *filename; int keysize; int sectorsize = 512; int dupkeys = 0; int filemode = 0; Py_KeywordsGet5Args("si|iii", filename,keysize,dupkeys,filemode,sectorsize); return (PyObject *)mxBeeIndex_New(filename,filemode, keysize+1,sectorsize, mxBeeIndex_CompareStrings, mxBeeIndex_StringFromKey, mxBeeIndex_KeyFromString, dupkeys); onError: return NULL; } ...plus: static PyMethodDef Module_methods[] = { Py_MethodWithKeywordsListEntry("BeeStringIndex",mxBeeIndex_BeeStringIndex), Py_MethodWithKeywordsListEntry("BeeIntegerIndex",mxBeeIndex_BeeIntegerIndex), Py_MethodWithKeywordsListEntry("BeeFloatIndex",mxBeeIndex_BeeFloatIndex), {NULL,NULL} /* end of list */ }; The macros used are: #define Py_KeywordGet5Args(keywords,format,a1,a2,a3,a4,a5) {if (!PyArg_ParseTupleAndKeywords(args,kw,format,keywords,&a1,&a2,&a3,&a4,&a5)) goto onError;} #define Py_C_Function_WithKeywords(fct,docstr) \ static char fct##_docstring[] = docstr;\ static PyObject *fct(PyObject *self, PyObject *args, PyObject *kw) #define Py_MethodWithKeywordsListEntry(pyname,cname) {pyname,(PyCFunction)cname,METH_VARARGS | METH_KEYWORDS,cname##_docstring} It's really no big deal... > >... > > > * maybe we should discuss punting dbi. module distribution may be easier > > > without it. While it is handy for clients that use more than one DB > > > module, > > > > Actually, it makes things harder for those cases (see above). mxODBC > > supports opening connections to many different databases and each > > has its own version of the dbi module because each subpackage uses > > its own set of exception classes. > > If we tossed the dbi module, then each database would incorporate its > dbi into the module itself. I don't understant what the problem is here. > > I'm definitely leaning towards axing dbi. Me too. So the problem is gone anyway... > > > I can easily envision a python module that exports the > > > appropriate set for the DB being used. > > > > Right: stick it all into the database interface package. > > I'd like to see: > > import mysqldb > ... > except mysqldb.IntegrityError: > ... > if curs.description[0][0] == mysqldb.STRING: > ... Right. > > > * cursor.rowcount is new, and is extraneous. The result of execute() > > > should be used. > > > > The execute() method returns None in case there are rows available. > > It does not return the number of rows in the result set. > > I am not sure that all databases can provide you the rowcount after an > execute(). You may need to fetch them first. Therefore, the .rowcount > couldn't even be implemented. This means that the only real way to know > how many rows there are is to fetch them -- rowcount could not be relied > upon. Ehm, is that a problem ? If the database interface cannot determine the exact row count, it should set it to -1 (I'll add a note about this). Note that the 1.0 spec didn't give you the rowcount information in any way for simple SELECT statements (only for DML statements). > > > * I think we should clean up the description attribute if possible. The > > > 7-tuple is/was very Oracle-specific. > > > > Ok, things like display_size and internal_size are not really > > relevant, but why break existing code expecting these 7 entries (even > > if it probably does not use all of them) ? > > I believe we should alter the specification, but that databases can > continue to return an appropriate number of fields. > > For example: can we require two fields -- name and type. Beyond that, it > is up to the database module. Implementors can choose to continue with > the 7-tuple or bring it down to what they really support. Hmm, but that would introduce database interface dependency: the module can always provide some sensible default values for the missing items, but the size and ordering of the tuple should not change from one DB interface to another. > > > * The old description for execute() returning None was clearer. Note > > > that a SELECT can return no columns/rows (where your description says > > > > 0). > > > > The None return value is to indicate that data is available > > to be fetched vie the fetchXXX() methods. If no rows where produced, > > something else should be returned, e.g. 1. > > I do not believe that a database can always tell that rows were or were > not produced. You may need to fetch them first. Therefore, returning 1 > is not possible for certain modules. > > Okay. I just looked at DCOracle. The execute() method does just that... > an execute. No fetching, so no way to tell what the row count is. > > Admittedly, there may be something in the OCI interface, or in the V8 > Oracle interfaces to do so. Unless sometime can point out that the > Oracle interfaces support knowing the row count at execute() time, then > we cannot rely on your .rowcount and return-value changes. > > Even so, I do not like seeing a SELECT returning "1" *ever*. That just > doesn't fit right. How about this: the return value of .execute remains undefined. If the database module can figure out how many rows were affected or are available via fetchXXX() it sets the .rowcount attribute accordingly. Otherwise, .rowcount is set to -1. That way we clear up the specification and still provide a standard way to report row count information. > > The reason I changed the description was that the previous one > > wasn't easy to implement: it is not clear whether a statement > > is DDL, DML or DQL without parsing it... > > Untrue. It is easy to implement -- you can always tell from the first > word in all cases. I'm not sure I follow you there: the SQL dialects allow for many different ways to express DDL, DML or DQL each with its own set of extensions... Apart from that I don't understand the use of being able to differentiate between those three categories: as far as I can tell it does not gain you any knowledge, since you normally know in advance what king of statement you are executing. >From the programmers view it is definitely easier to implement the spec 1.1 return value. I have been in exactly this situation when I implemented mxODBC: there is just no clean way of telling DDL, DML and DQL apart because each database has its own set of extensions and even the ODBC levels provide different sets of SQL dialects. > > > * in general, I think the 1.1 change to the return value description > > > didn't really clear things up. Basing the descriptions on returning > > > columns/rows doesn't make much sense to me. Instead, I liked the > > > description based on the *type* of statement that was performed. Under > > > this style, a SELECT that doesn't produce columns/rows would return 1, > > > but that is just wrong. > > > > See above. Without parsing there isn't much you can do as module > > implementor to get the return value right. Maybe we should drop the > > return value altogether: rowcount gives the number of rows in the > > result set (even 0 for SELECTs that didn't produce any rows) or the > > number of affected rows. > > As I mentioned, I think the rowcount is not possible for all databases > that use a pure execute/fetch model like Oracle. With the relaxed definition of .rowcount there should be no problem. > > > * execute()'s description should change back to the terms "sequence" and > > > "sequence of sequences" for the parameters. There is no reason to force > > > them to be tuples or lists. > > > > I thought about this: there is no way for the method to tell whether > > it was passed a sequence of sequences that is meant to be a > > sequence of parameter sequences or just a single sequence with > > sequence entries, e.g. ('abc',) would break under that definition. > > > > Is there really such a big need to be able to pass *any* sequence > > as parameter ? > > Yes. > > What if you have a very large dataset which is encapsulated within a > class instance? Let's say it responds properly to the sequence behavior, > specifically so that it can be inserted without having to construct a > big ol' list to do so. I also can easily see rows that identified by > class instances or other builtin types, rather than always being tuples > or lists. > > It certainly does hurt the implementor to use PySequence_GetItem() > rather than PyTuple_GetItem(). > > An alternative would be an "insert" method that specifically takes a > sequences of rows (where each row is a sequence). This would eliminate > the confusion between a sequence-of-sequence being a single row or a set > of rows. > > I believe the INSERT is the only case where a sequence input makes > sense. I think just about any DML statement fits the sequence of sequences category, not just INSERT... I'm not opposing the sequence argument, it's only that the module implementor has no chance of telling whether he's seeing a sequence of sequences or a plain old sequence of strings... How about adding a .executemany() API that is intended for sequence of sequences parameters only ? (or some other way of stating that the parameters sequence is in fact a seq of sequences). > > > * the part about cursor movement in fetchone() should be totally > > > removed. fetchone() is about the CLIENT fetching a single row. It should > > > NOT affect how the underlying module performs its fetches. > > > > That renders cursors useless... are you sure you want this ? > > Absolutely. Positively. There is no way that we can or should require > single-row fetching. > > Look at DCOracle. It uses the API nearly to its full extent (it doesn't > use input or output sizes to optimize its binding, though). Requiring > fetchone() to disable the array fetching would be a mistake, let alone > the fact that when you execute() the array-fetch there is no way to know > whether the person will be using fetchone or not! So cursors are deemed non-portable... given the fact that cursor implementations differ greatly from DB to DB that is probably an arguable approach. I'll remove the note from .fetchone(). > > > * the fetch* methods should say they return a sequence or a sequence of > > > sequences. We should not force them to be tuples or lists of tuples. > > > Some DB modules may want to have additional semantics or capabilities on > > > their results. As long as it obeys sequence behavior, then the spec > > > should be happy. The arraysize attribute's description also refers to a > > > list of tuples, and should be updated. > > > > Just thought it would be a good idea to keep execute() parameter > > input and fetchXXX() output in sync. > > Right. And I think they should all sync up to be a > sequence-of-sequences. I can see an implementor wanting to create a > high-performance binding against a database that serves up values > directly from the output buffers rather than jamming them into lists of > tuples (with the requisite allocations). > > Jim Fulton has argued for a long while that the module should be able to > return a highly functional result set. As long as it obeys the > sequence-of-sequence behavior, then he could now do so, yet still > "follow the spec". If we find a suitable way to implement this, fine. Just stating that the parameters could be a sequence of sequences is not enough though. The updated spec (1.1a8) is at: http://starship.skyport.net/~lemburg/DatabaseAPI-1.1.html I've folded dbi into the module interface and added constructors Date, Time and Timestamp (as well as corresponding type codes). Note that only the constructor names are defined -- not whether the result from calling these specifies a certain type or instance. Hmm. Maybe it'll be 2.0 after all... punting dbi, new interfaces, different semantics... -- Marc-Andre Lemburg Y2000: 293 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : --------------------------------------------------------- From james_northrup@iridium.com Mon Mar 15 12:04:51 1999 From: james_northrup@iridium.com (James Northrup) Date: Mon, 15 Mar 1999 07:04:51 -0500 Subject: [DB-SIG] annotated 1.1 spec / feedback References: <199903141701.MAA29819@python.org> Message-ID: <36ECF763.C410A830@iridium.com> > The updated spec (1.1a8) is at: > > http://starship.skyport.net/~lemburg/DatabaseAPI-1.1.html > > I've folded dbi into the module interface and added constructors > Date, Time and Timestamp (as well as corresponding type codes). > Note that only the constructor names are defined -- not whether > the result from calling these specifies a certain type or instance. > > Hmm. Maybe it'll be 2.0 after all... punting dbi, new > interfaces, different semantics... > reading through your 1.1a8 spec I have the following remarks hoping to offer in some convenience functions with the python time module: Where we have: Date(year,month,day) This function constructs an object holding a date value. Time(hour,minute,second) This function constructs an object holding a time value. Timestamp(year,month,day,hour,minute,second) This function constructs an object holding a time stamp value. Add: TimeStamp (string[, format]) *** (time.strptime compatible) This function constructs an object holding a time stamp value. Parse a string representing a time according to a format. The format parameter uses the same directives as those used by time.strptime(); it defaults to "%a %b %d %H:%M:%S %Y" which matches the formatting returned by ctime(). Add: TimeStamp (tuple) *** (time.mktime or time.asctime compatible) This function constructs an object holding a time stamp value. Its argument is the full 9-tuple (since the dst flag is needed -- pass -1 as the dst flag if it is unknown) which expresses the time in local time, not UTC. Add: gmtime (TimeStamp) *** (time.gmtime identical return) Convert a TimeStamp to a time tuple in UTC in which the dst flag is always zero. Add: localtime (TimeStamp) *** (time.localtime identical return) Converts to local time. The dst flag is set to 1 when DST applies to the given time. Retract: *** mxTime as an inherent piece of db-sig discussion. Use mxTime as is necesary for implemenation purposes, but keep the python learning curve slim and to the point. mxTime's presence should be escalated to the python kernel concerns (we want better time), however, this db-sig has become tangental and worrisome with no substantial core interface adherence in it's 1.0 incarnation. while DB-sig seems to have made a great case for a more reliable time library, that thread should be processed in a general-python-improvement sig. I see this SIG moving very swiftly towards an idealistic pinnacle. I would like to contribute my 2 cents worth, I want a simple and direct DB interface class to adhere to. I want to be able to teach a neophyte DBA who just learned SQL everything *except* OO syntactic sugar. I want their experience with maintaining my python application to be as simple as looking up the python core api and modules and adding a new function call. furthermore, I want to see python have the full benefit of mxTime as a default transparent upgrade, and would love to help make this a reality. I'm doing significant amounts of multi-dbms python work presently and am in fact writing my own middle-interface drivers in lack of standardized conventions. I would love to help push through something simple, solid, perhaps not idealistic, but at least published and distributed with working implementation rsn. Thanks -jim From adustman@comstar.net Mon Mar 15 19:09:31 1999 From: adustman@comstar.net (Andy Dustman) Date: Mon, 15 Mar 1999 14:09:31 -0500 (EST) Subject: [DB-SIG] annotated 1.1 spec / feedback In-Reply-To: <36EAD0FC.65B8B3AF@lemburg.com> Message-ID: The only other thing I worry about somewhat is the handling of transactions. On a db without transactions (MySQL), commit() can simply do nothing. rollback(), however, is another matter. I tend to think that rollback() on a database without transactions should raise some kind of exception. Perhaps it simply should not be defined so that an AttributeError is raised, so the application can at least try to take some corrective action. Another possibility is some kind of capabilities object which the application can use to determine what features are available. One obvious capability is the presence or absence of transactions. Another might be the type of parameter substitution used by the query language. I'm not really sure how useful this would be. Every database has it's own quirks, it seems, and it's probably not possible to work around all of them. However, at least knowing that transactions aren't available might at least let some alternate code work around this. -- Andy Dustman (ICQ#32922760) You should always say "spam" and "eggs" ComStar Communications Corp. instead of "foo" and "bar" (706) 549-7689 | PGP KeyID=0xC72F3F1D in Python examples. (Mark Lutz) From mal@lemburg.com Tue Mar 16 08:58:15 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Tue, 16 Mar 1999 09:58:15 +0100 Subject: [DB-SIG] annotated 1.1 spec / feedback References: Message-ID: <36EE1D27.21CA23B7@lemburg.com> Andy Dustman wrote: > > The only other thing I worry about somewhat is the handling of > transactions. On a db without transactions (MySQL), commit() can simply do > nothing. rollback(), however, is another matter. I tend to think that > rollback() on a database without transactions should raise some kind of > exception. Perhaps it simply should not be defined so that an > AttributeError is raised, so the application can at least try to take some > corrective action. > > Another possibility is some kind of capabilities object which the > application can use to determine what features are available. One obvious > capability is the presence or absence of transactions. Another might be > the type of parameter substitution used by the query language. I'm not > really sure how useful this would be. Every database has it's own quirks, > it seems, and it's probably not possible to work around all of them. > However, at least knowing that transactions aren't available might at > least let some alternate code work around this. We could add a module global 'transactionlevel' to make this information available with values: 0 - database does not support transactions non-0 - bit field containing database specific information about how transactions are implemented ODBC has an API to query database/driver capabilities called SQLGetInfo which provides the above under the id SQL_DEFAULT_TXN_ISOLATION. You may want to look at the ODBC docs at: http://www.solidtech.com/support/pg/httoc.htm To get an impression of what the above bit field could look like. As for .rollback() I'm not sure whether raising an exception would help much: transactions are a very basic concept and a work-around is likely not going to be possible by writing a simple try-except handler. -- Marc-Andre Lemburg Y2000: 290 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : --------------------------------------------------------- From mal@lemburg.com Tue Mar 16 09:13:06 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Tue, 16 Mar 1999 10:13:06 +0100 Subject: [DB-SIG] annotated 1.1 spec / feedback References: <199903141701.MAA29819@python.org> <36ECF763.C410A830@iridium.com> Message-ID: <36EE20A2.3BD223C8@lemburg.com> James Northrup wrote: > > > The updated spec (1.1a8) is at: > > > > http://starship.skyport.net/~lemburg/DatabaseAPI-1.1.html > > > > I've folded dbi into the module interface and added constructors > > Date, Time and Timestamp (as well as corresponding type codes). > > Note that only the constructor names are defined -- not whether > > the result from calling these specifies a certain type or instance. > > > > Hmm. Maybe it'll be 2.0 after all... punting dbi, new > > interfaces, different semantics... > > > > reading through your 1.1a8 spec I have the following remarks hoping to offer in some convenience functions with the python time module: > > Where we have: > Date(year,month,day) > This function constructs an object holding a date value. > > Time(hour,minute,second) > This function constructs an object holding a time value. > > Timestamp(year,month,day,hour,minute,second) > This function constructs an object holding a time stamp value. > > Add: > TimeStamp (string[, format]) > *** (time.strptime compatible) > This function constructs an object holding a time stamp value. > Parse a string representing a time according to a format. The format parameter uses the same directives as those used by time.strptime(); it defaults to > "%a %b %d %H:%M:%S %Y" which matches the formatting returned by ctime(). > > Add: > TimeStamp (tuple) > *** (time.mktime or time.asctime compatible) > This function constructs an object holding a time stamp value. > Its argument is the full 9-tuple (since the dst flag is needed -- pass -1 as the dst flag if it is unknown) which expresses the time in local time, not > UTC. Note that we'd need three different names for the above since Python does do allow type inference. > Add: > gmtime (TimeStamp) > *** (time.gmtime identical return) > Convert a TimeStamp to a time tuple in UTC in which the dst flag is always zero. > > Add: > localtime (TimeStamp) > *** (time.localtime identical return) > Converts to local time. The dst flag is set to 1 when DST applies to the given time. All of the above are easily done using the existing mxDateTime constructors. Why should we add them to the DB API specification ? If a module writer were not to use mxDateTime then it would only make things more complicated for him. > Retract: > *** > mxTime as an inherent piece of db-sig discussion. > > Use mxTime as is necesary for implemenation purposes, but keep the python learning curve slim and to the point. > > mxTime's presence should be escalated to the python kernel concerns (we want better time), however, this db-sig has become tangental and worrisome with no > substantial core interface adherence in it's 1.0 incarnation. I don't think we can convince Guido of adding mxDateTime to the Python core. Still, installing that package is not any more difficult than installing a typcial database interface. I don't see a point in omitting useful software that is readily available just because it does not belong to the language core. > while DB-sig seems to have made a great case for a more reliable time library, that thread should be processed in a general-python-improvement sig. > > I see this SIG moving very swiftly towards an idealistic pinnacle. I would like to contribute my 2 cents worth, I want a simple and direct DB interface > class to adhere to. I want to be able to teach a neophyte DBA who just learned SQL everything *except* OO syntactic sugar. I want their experience with > maintaining my python application to be as simple as looking up the python core api and modules and adding a new function call. > > furthermore, I want to see python have the full benefit of mxTime as a default transparent upgrade, and would love to help make this a reality. > > I'm doing significant amounts of multi-dbms python work presently and am in fact writing my own middle-interface drivers in lack of standardized > conventions. I would love to help push through something simple, solid, perhaps not idealistic, but at least published and distributed with working > implementation rsn. Agreed. The API should stay simple and an OO layer should be built on top of it in Python; still, for a few things we'll have to resort to objects for the interface specification simply because it is the only way of providing vital information to the module. -- Marc-Andre Lemburg Y2000: 290 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : --------------------------------------------------------- From gstein@lyra.org Tue Mar 16 09:40:41 1999 From: gstein@lyra.org (Greg Stein) Date: Tue, 16 Mar 1999 01:40:41 -0800 Subject: [DB-SIG] annotated 1.1 spec / feedback References: <36EE1D27.21CA23B7@lemburg.com> Message-ID: <36EE2719.19CB753B@lyra.org> M.-A. Lemburg wrote: > > Andy Dustman wrote: > > > > The only other thing I worry about somewhat is the handling of > > transactions. On a db without transactions (MySQL), commit() can simply do > > nothing. rollback(), however, is another matter. I tend to think that > > rollback() on a database without transactions should raise some kind of > > exception. Perhaps it simply should not be defined so that an > > AttributeError is raised, so the application can at least try to take some > > corrective action. > > > > Another possibility is some kind of capabilities object which the > > application can use to determine what features are available. One obvious > > capability is the presence or absence of transactions. Another might be > > the type of parameter substitution used by the query language. I'm not > > really sure how useful this would be. Every database has it's own quirks, > > it seems, and it's probably not possible to work around all of them. > > However, at least knowing that transactions aren't available might at > > least let some alternate code work around this. > > We could add a module global 'transactionlevel' to make this > information available with values: > > 0 - database does not support transactions > non-0 - bit field containing database specific information > about how transactions are implemented > > ODBC has an API to query database/driver capabilities called > SQLGetInfo which provides the above under the id > SQL_DEFAULT_TXN_ISOLATION. You may want to look at the ODBC > docs at: > http://www.solidtech.com/support/pg/httoc.htm > To get an impression of what the above bit field could look > like. > > As for .rollback() I'm not sure whether raising an exception > would help much: transactions are a very basic concept and > a work-around is likely not going to be possible by writing > a simple try-except handler. I think it would be nicer to omit the commit/rollback methods if they are not supported. A simple hasattr() test will provide the information that you need. The "os" module operates the same way: if a function is not supported, then it is simply omitted (rather than raising some kind of error). Starting to define bitfields will just get us in trouble, I think. A question does arise: for the ODBC module, it doesn't know until a connection is made whether commit/rollback is supported. In that case, how do we reset commit/rollback based on the dynamic information queried via the connection? hmm... I would recommend that if a DB module needs dynamic determination of rollback capability, then it should dynamically make commit/rollback available. For example, if those two methods are in C, then the module checks a flag (set at connection-instantiation, or lazily at the first getattr for commit/rollback) in the getattr function. If the methods are going to be in Python, then a __getattr__ is necessary. Note that something like transactionlevel would be connection-based, not module-based. I don't recommend going for yet another flag like this, when we have a "capabilities" style via the presence/absence of a method. The query substitution is a good one. I don't have any particular ideas there. I do think that we need to clarify the execute() statement, though, that the parameters imply input binding. Not string substitution. There's a big difference in behavior between the two (e.g. does the client need to quote parameter values?). Cheers, -g -- Greg Stein, http://www.lyra.org/ From gstein@lyra.org Tue Mar 16 10:02:08 1999 From: gstein@lyra.org (Greg Stein) Date: Tue, 16 Mar 1999 02:02:08 -0800 Subject: [DB-SIG] annotated 1.1 spec / feedback References: <199903141701.MAA29819@python.org> <36ECF763.C410A830@iridium.com> <36EE20A2.3BD223C8@lemburg.com> Message-ID: <36EE2C20.37352069@lyra.org> M.-A. Lemburg wrote: > > James Northrup wrote: > >... > > Where we have: > > Date(year,month,day) > > This function constructs an object holding a date value. > > > > Time(hour,minute,second) > > This function constructs an object holding a time value. > > > > Timestamp(year,month,day,hour,minute,second) > > This function constructs an object holding a time stamp value. Why do we need three variations? Do we actually have issues with input binding where the discrimination matters? We never had any problems before, when we just had a date/time object. >... > > Retract: > > *** > > mxTime as an inherent piece of db-sig discussion. > > > > Use mxTime as is necesary for implemenation purposes, but keep the python learning curve slim and to the point. > > > > mxTime's presence should be escalated to the python kernel concerns (we want better time), however, this db-sig has become tangental and worrisome with no > > substantial core interface adherence in it's 1.0 incarnation. > > I don't think we can convince Guido of adding mxDateTime to the > Python core. Still, installing that package is not any more > difficult than installing a typcial database interface. I don't > see a point in omitting useful software that is readily available > just because it does not belong to the language core. Creating a hard dependency on mxDateTime will place us right back into the problem that we had with "dbi". People like simplicity and they'll cut corners. I've been using mysqldb quite a bit lately, and it doesn't use dbi. I doubt that it would change to use mxDateTime. People want to distribute a single .c file and be done with it. That is definitely easier on the module implementor than using mxDateTime. I understand that you want to see people using your module, and I generally agree with it, but I don't think it is a good idea for us to mandate it. The API must support a DateTime(ticks) function which can map onto a simple dbiDate-like object. A module implementor must be able to use a simple ticks-based object (e.g. copy the code out of the (deprecated) dbi module). Cheers, -g -- Greg Stein, http://www.lyra.org/ From mal@lemburg.com Tue Mar 16 10:32:15 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Tue, 16 Mar 1999 11:32:15 +0100 Subject: [DB-SIG] annotated 1.1 spec / feedback References: <36EE1D27.21CA23B7@lemburg.com> <36EE2719.19CB753B@lyra.org> Message-ID: <36EE332F.754BA421@lemburg.com> Greg Stein wrote: > > I think it would be nicer to omit the commit/rollback methods if they > are not supported. A simple hasattr() test will provide the information > that you need. > > The "os" module operates the same way: if a function is not supported, > then it is simply omitted (rather than raising some kind of error). > > Starting to define bitfields will just get us in trouble, I think. All true, but due to the way methods are "published" in C dynamically making them available is not easily done. > A question does arise: for the ODBC module, it doesn't know until a > connection is made whether commit/rollback is supported. In that case, > how do we reset commit/rollback based on the dynamic information queried > via the connection? hmm... I would recommend that if a DB module needs > dynamic determination of rollback capability, then it should dynamically > make commit/rollback available. For example, if those two methods are in > C, then the module checks a flag (set at connection-instantiation, or > lazily at the first getattr for commit/rollback) in the getattr > function. If the methods are going to be in Python, then a __getattr__ > is necessary. > > Note that something like transactionlevel would be connection-based, not > module-based. I don't recommend going for yet another flag like this, > when we have a "capabilities" style via the presence/absence of a > method. Right. In ODBC you can even set the transaction level on a per connect basis. The attribute based approach is still easier to implement, though... Note that for ODBC the standard transaction behaviour is AUTO COMMIT, meaning that the transaction is committed for every executed statement. Calling .rollback() on such a connection wouldn't show any effect even if the database does support transactions. > The query substitution is a good one. I don't have any particular ideas > there. I do think that we need to clarify the execute() statement, > though, that the parameters imply input binding. Not string > substitution. There's a big difference in behavior between the two (e.g. > does the client need to quote parameter values?). Agreed. This is not clear enough yet. People tend to forget the possibility of using bind parameters: they simple use 'statement %s' % (myparam,). Of course, this can cause lots of trouble with e.g. binary data or multi KB sized data. How should the parameter handling be indicated ? AFAIK, there are two common ways: ?,?,? and :1, :2, :3... .marker: '?' or ':%i' ??? -- Marc-Andre Lemburg Y2000: 290 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : --------------------------------------------------------- From mal@lemburg.com Tue Mar 16 10:46:46 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Tue, 16 Mar 1999 11:46:46 +0100 Subject: [DB-SIG] annotated 1.1 spec / feedback References: <199903141701.MAA29819@python.org> <36ECF763.C410A830@iridium.com> <36EE20A2.3BD223C8@lemburg.com> <36EE2C20.37352069@lyra.org> Message-ID: <36EE3696.5FF48FFC@lemburg.com> Greg Stein wrote: > > M.-A. Lemburg wrote: > > > > James Northrup wrote: > > >... > > > Where we have: > > > Date(year,month,day) > > > This function constructs an object holding a date value. > > > > > > Time(hour,minute,second) > > > This function constructs an object holding a time value. > > > > > > Timestamp(year,month,day,hour,minute,second) > > > This function constructs an object holding a time stamp value. > > Why do we need three variations? Do we actually have issues with input > binding where the discrimination matters? We never had any problems > before, when we just had a date/time object. These are the standard date/time objects defined by SQL. It seemed natural to define these constructors at the module level too. Date() is not capable of representing time only information. > >... > > > Retract: > > > *** > > > mxTime as an inherent piece of db-sig discussion. > > > > > > Use mxTime as is necesary for implemenation purposes, but keep the python learning curve slim and to the point. > > > > > > mxTime's presence should be escalated to the python kernel concerns (we want better time), however, this db-sig has become tangental and worrisome with no > > > substantial core interface adherence in it's 1.0 incarnation. > > > > I don't think we can convince Guido of adding mxDateTime to the > > Python core. Still, installing that package is not any more > > difficult than installing a typcial database interface. I don't > > see a point in omitting useful software that is readily available > > just because it does not belong to the language core. > > Creating a hard dependency on mxDateTime will place us right back into > the problem that we had with "dbi". People like simplicity and they'll > cut corners. I've been using mysqldb quite a bit lately, and it doesn't > use dbi. I doubt that it would change to use mxDateTime. Sorry. That was a misunderstanding: I don't intend to hard code mxDateTime into the spec. The spec only defines constructor names; the module implementor can then code these constructors in any way he likes. mxDateTime is only meant as offering, since it already provides all necessary functionality. > People want to distribute a single .c file and be done with it. That is > definitely easier on the module implementor than using mxDateTime. Agreed; though things are moving on the distutils list, so that may soon not be such a big problem anymore. > I understand that you want to see people using your module, and I > generally agree with it, but I don't think it is a good idea for us to > mandate it. The API must support a DateTime(ticks) function which can > map onto a simple dbiDate-like object. A module implementor must be able > to use a simple ticks-based object (e.g. copy the code out of the > (deprecated) dbi module). I think we should get rid off Unix ticks in the specification completely. They introduce too many errors and quirks (e.g. only think of dates prior to 1.1.70). If a module implementor wants to use ticks anyway, he can use time.mktime to create the ticks integer in his module's Date() constructor, e.g. def Date(year,month,day): import time return time.mktime(year,month,day,0,0,0,0,0,-1) While ticks are easy to use, many database can't handle them in the APIs anyway, so there's really no need for them to go explicitly into the spec. -- Marc-Andre Lemburg Y2000: 290 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : --------------------------------------------------------- From james_northrup@iridium.com Tue Mar 16 14:50:35 1999 From: james_northrup@iridium.com (James Northrup) Date: Tue, 16 Mar 1999 09:50:35 -0500 Subject: [DB-SIG] annotated 1.1 spec / feedback References: <199903141701.MAA29819@python.org> <36ECF763.C410A830@iridium.com> <36EE20A2.3BD223C8@lemburg.com> Message-ID: <36EE6FBB.8155CDEE@iridium.com> > I don't think we can convince Guido of adding mxDateTime to the > Python core. Still, installing that package is not any more > difficult than installing a typcial database interface. I don't > see a point in omitting useful software that is readily available > just because it does not belong to the language core. I'm coming from a production support angle here. I have non-hacker people asking me why I'm using each and every package and why they are paying for it. They want to know who to turn to when undocumented features appear, and where they can find fixes. They want to be content with the knowldge that a veteran architect-developer can work within guidelines that a novice maintenance-developer can easily grasp and take over. When I roll a product into production I want to be able to acheive the following recomendation (in the hopefully near future): "Python 1.x.x base distribution includes a core DBMS api that supports plugin adapters and drivers for the following platforms: ... " Perhaps I am on a different mission than the general consensus here, but presently as I roll an application into production (and potentially never alter it again) I must justify and include the following support information : Base Python (build instructions, deployment instructions) (operational specifications, diagnostic procedures, pager number) MySQLModule (where to find it, which version, build instructions, deployment instructions, support?) DCOracle (where to find it, which version, build instructions, deployment instructions, support?) (operational specifications, diagnostic procedures, pager number) I'd like to offer into this discussion that for my own benefit and the viability of Python as a world-class contender in the world of high-paranoia production systems support, db-sig should unify, freeze code on a 1.0 or 1.1 python base includion, and push it through. If this means that xyz time package is unfit for base python distribution inclusion, or any other module, compromise; then sign off on a DB api interface so that python can move from DB-modules to DB-drivers in the paradigm of distribution. this sig is venerable, yet glaringly absent in base python representation next to the likes of generic interface/driver examples such as sgml/html/http and ui/gui interfaces. Although I'm new to python and db-sig, I would like to know what are the requirements for a db-api python baseline formalization and inclusion. I think we are very close, and I would really like to help and lend a fresh eye. Thanks -jim From mal@lemburg.com Tue Mar 16 17:24:18 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Tue, 16 Mar 1999 18:24:18 +0100 Subject: [DB-SIG] annotated 1.1 spec / feedback References: <199903141701.MAA29819@python.org> <36ECF763.C410A830@iridium.com> <36EE20A2.3BD223C8@lemburg.com> <36EE6FBB.8155CDEE@iridium.com> Message-ID: <36EE93C2.32CF01A4@lemburg.com> James Northrup wrote: > > > I don't think we can convince Guido of adding mxDateTime to the > > Python core. Still, installing that package is not any more > > difficult than installing a typcial database interface. I don't > > see a point in omitting useful software that is readily available > > just because it does not belong to the language core. > > I'm coming from a production support angle here. I have non-hacker people asking me why I'm using each and every package and why they are paying for it. They > want to know who to turn to when undocumented features appear, and where they can find fixes. They want to be content with the knowldge that a veteran > architect-developer can work within guidelines that a novice maintenance-developer can easily grasp and take over. As I already pointed out in my reply to Greg, mxDateTime is not an requirement, it's only a readily useable plugin for the DB API date/time specification part. You can use any implementation you like as long as you provide the defined constructor APIs. As for support, I'm offering commercial support for all mx* modules via the PythonPros. BTW: they also provide support for all other parts of the standard Python distribution as well as Mark Hammond's Win32 components. > When I roll a product into production I want to be able to acheive the following recomendation (in the hopefully near future): > > "Python 1.x.x base distribution includes a core DBMS api that supports plugin adapters and drivers for the following platforms: ... " > > Perhaps I am on a different mission than the general consensus here, but presently as I roll an application into production (and potentially never alter it again) > I must justify and include the following support information : > > Base Python (build instructions, deployment instructions) > (operational specifications, diagnostic procedures, pager number) > MySQLModule (where to find it, which version, build instructions, deployment instructions, support?) > DCOracle (where to find it, which version, build instructions, deployment instructions, support?) > (operational specifications, diagnostic procedures, pager number) Hmm. We could try getting the DB API specification into the standard docs for Python. That would add a little more "official" flavor to our work. -- Marc-Andre Lemburg Y2000: 290 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : --------------------------------------------------------- From james_northrup@iridium.com Wed Mar 17 16:46:51 1999 From: james_northrup@iridium.com (James Northrup) Date: Wed, 17 Mar 1999 11:46:51 -0500 Subject: [DB-SIG] annotated 1.1 spec / feedback References: <199903141701.MAA29819@python.org> <36ECF763.C410A830@iridium.com> <36EE20A2.3BD223C8@lemburg.com> <36EE6FBB.8155CDEE@iridium.com> <36EE93C2.32CF01A4@lemburg.com> Message-ID: <36EFDC7B.A72A7A81@iridium.com> > Hmm. We could try getting the DB API specification into the standard > docs for Python. That would add a little more "official" flavor to > our work. questions regarding this sig: 1) Is there an issue in publishing a module of abstract classes to adhere to ? 2) Why stop at the docs for python if we can publish self documented base classes as well? I have looked frantically for a dbapi.py and all I have found are disparate c modules that don't have parent class interfaces. If such an animal exists, I would love a pointer. I would gladly step out of this discussion and into one about improving such a module. 3) Java has a comfortable "Interface/Impl" flavor of libraries, why is the db-api an ethereal specification of convention rather than an entry point in code interface? From gstein@lyra.org Wed Mar 17 17:59:10 1999 From: gstein@lyra.org (Greg Stein) Date: Wed, 17 Mar 1999 09:59:10 -0800 Subject: [DB-SIG] annotated 1.1 spec / feedback References: <199903141701.MAA29819@python.org> <36ECF763.C410A830@iridium.com> <36EE20A2.3BD223C8@lemburg.com> <36EE6FBB.8155CDEE@iridium.com> <36EE93C2.32CF01A4@lemburg.com> <36EFDC7B.A72A7A81@iridium.com> Message-ID: <36EFED6E.3EEC99DF@lyra.org> James Northrup wrote: > > > Hmm. We could try getting the DB API specification into the standard > > docs for Python. That would add a little more "official" flavor to > > our work. > > questions regarding this sig: > > 1) Is there an issue in publishing a module of abstract classes to adhere to ? The DB modules aren't going to be able to subclass from them since the modules must be implemented in C. Therefore, publishing something like this doesn't really provide any more information than the specification that we currently have. > 2) Why stop at the docs for python if we can publish self documented base classes as well? > I have looked frantically for a dbapi.py and all I have found are disparate c modules that don't have parent class interfaces. > If such an animal exists, I would love a pointer. I would gladly step out of this discussion and into one about improving such a module. There is no such beast. The C modules can't have a parent class interface (without undue strain, and with little benefit). This is simply the way Python works. > 3) Java has a comfortable "Interface/Impl" flavor of libraries, why is the db-api an ethereal specification of convention rather than an entry point in code interface? Goody for Java. Python doesn't have strict interface mechanisms, so that programming methodology does not apply. Publishing a specification for people to conform to seems to work. There are people that do not conform to the spec; however, have a base class would not improve the situation... they just wouldn't use it if they didn't want to conform. Cheers, -g -- Greg Stein, http://www.lyra.org/ From mal@lemburg.com Wed Mar 17 18:13:50 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 17 Mar 1999 19:13:50 +0100 Subject: [DB-SIG] Last round for DB API 1.1 Message-ID: <36EFF0DE.14C814C@lemburg.com> Hi everybody, Thanks for all your comments on the 1.1 DB API specification update so far. There are still some open questions which need to be discussed prior to the final call: · What to do about Andy Dustmans proposal to have .rollback() raise an exception in case transactions are not supported by the database ? IMHO, we should leave the situation as it is: some DBs do not support transactions (e.g. MySQL), some do but you can turn transactions off (e.g. this is the ODBC default)... all in all its a feature that is very dynamic, specific to connections. If the API were to raise an exception in case tranactions are not supported, the underlying interface would have to query the current state prior to every .rollback() and even prior to garbage collecting the connection object -- at a time where there could be no connection available any more. The only way I see to support this "capability" feature is to define a standard method for querying capabilities in general, e.g. .capability('transactions'), but that would introduce a set of constants which would also have to go into the spec... or maybe a set of methods .has_this() and .has_that()... I'm not sure whether it's worth defining these things in the spec at all: e.g. switching from a database with transaction to one without them is hardly doable in case your code needs them. · Andy Dustman also asked for a standard way to find out the parameter marker syntax used by the database. There are at least two possibilities: ?,?,? in the order of the given parameters :1,:3,:2 in any order only referring to the parameter position Not sure how to get this information into a constant or a method... Any ideas ? · Should we add additional constructor signatures to the API spec ? So far we have Raw,Date,Time,Timestamp. Maybe we should rename Raw to BLOB or Binary to be in sync with SQL ?! What about a monetary constructor... does anyone have experience with such types ? · We still need to settle the sequence of sequence problem with .execute(). Since e.g. ('abc',) fits this definition but surely isn't intended, we'll need some other way to handle sequences of parameter/data sequences. How about depreciating the use of those lists in .execute() and defining a .executemany() method for this purpose instead ? Other ideas ? Cheers, -- Marc-Andre Lemburg Y2000: 289 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : --------------------------------------------------------- From adustman@comstar.net Wed Mar 17 19:26:11 1999 From: adustman@comstar.net (Andy Dustman) Date: Wed, 17 Mar 1999 14:26:11 -0500 (EST) Subject: [DB-SIG] Last round for DB API 1.1 In-Reply-To: <36EFF0DE.14C814C@lemburg.com> Message-ID: On Wed, 17 Mar 1999, M.-A. Lemburg wrote: > · What to do about Andy Dustmans proposal to have .rollback() > raise an exception in case transactions are not supported by > the database ? > > IMHO, we should leave the situation as it is: some DBs do not > support transactions (e.g. MySQL), some do but you can turn > transactions off (e.g. this is the ODBC default)... all in all > its a feature that is very dynamic, specific to connections. > > If the API were to raise an exception in case tranactions are > not supported, the underlying interface would have to query the > current state prior to every .rollback() and even prior to garbage > collecting the connection object -- at a time where there could > be no connection available any more. Well, look at it this way. If you were using a database that supported transactions, and it was operating in a transactional mode, you'd expect an exception if, for some reason, the rollback couldn't be completed. (Are rollbacks guaranteed to succeed from an SQL standpoint?) It need not be dynamic in the sense that it is sensitive to the transaction isolation; presumably the programmer has set this and will only use rollbacks in non-auto-commit mode. All I am suggesting is that if rollback is called on a database which cannot possibly roll back a transaction, that some exception be raised, which is probably preferable to having some program which depends on rollback continuing on as if the rollback was successful. This simplest way to do this is to simply not implement the rollback method if no support is available, which will raise an AttributeError. It seems harmless to allow commit to always succeed and do nothing on a transactionless database, though, since all statements are essentially committed once they are executed. > · Andy Dustman also asked for a standard way to find out the > parameter marker syntax used by the database. > > There are at least two possibilities: > > ?,?,? in the order of the given parameters > :1,:3,:2 in any order only referring to the parameter position > > Not sure how to get this information into a constant or a method... > Any ideas ? Well, there seem to be three styles, AFAIK: PARAMETER_QM = 1 # parameters represented by question marks PARAMETER_CP = 2 # parameters represented by colon+position number PARAMETER_CF = 3 # parameters represented by C format string (i.e. %s) The main problem, of course, is how to write code to make use of this in a portable way. This may be beyond the scope of what the DB API can handle, since it is a function of the query language in use, and the API is not tied to any particular query language. I suspect there are a lot of human-written queries out there that will just have to be re-written, and I'm not sure this will really be useful. > · Should we add additional constructor signatures to the API spec ? > > So far we have Raw,Date,Time,Timestamp. Maybe we should rename > Raw to BLOB or Binary to be in sync with SQL ?! What about a > monetary constructor... does anyone have experience with such > types ? Binary would be good, but should be the same as Raw, which would be deprecated. > · We still need to settle the sequence of sequence problem > with .execute(). > > Since e.g. ('abc',) fits this definition but surely isn't intended, > we'll need some other way to handle sequences of parameter/data > sequences. How about depreciating the use of those lists in > .execute() and defining a .executemany() method for this purpose > instead ? Other ideas ? Right now it says "tuple or list of tuples". I'm happy with that. Another problem: The spec currently says that when calling the connection.cursor() method, "An exception may be thrown if the database does not support a cursor concept." However, if the database does not support cursors, then you really can't do jack squat with the API. I think the old version said you could do cursor methods on the connection, but a better solution is probably to require the database module to emulate cursors. Take for an example, MySQL. MySQL does not have cursors. It has the MYSQL_RES structure, which contains a result, but queries are executed on the connection. However, it is not too hard to emulate a cursor in a Python wrapper. And yes, I have just about finished yet another MySQL interface. This one is thread-friendly (wraps blocking calls with Py_BEGIN_ALLOW_THREADS ... Py_END_ALLOW_THREADS), based on the latest MySQL 3.22.19a API, and is in two parts: A _mysqlmodule.so which implements most of the MySQL API (has connection objects and result objects; result.describe() returns a DB API compatible description), and MySQLdb.py which implements the current DB API. There is a type_conv dictionary which maps MySQL FIELD_TYPEs to Python functions, so the user can specify how SQL types should be converted; unspecified types are returned as strings. By default the C module does not convert the various time types; the Python module sets the dictionary so they are converted to DateTime objects. On writing to the database, the interface converts all input parameters to strings and does the necessary quoting. Note that due to the design, you can subclass both the connection and cursor objects, as they are implemented in Python. Compared to MySQLmodule-1.4, the C source is slightly smaller, the .so library is 40% smaller (i386, YMMV), and the Python interface is 50% smaller. Probably this is mostly due to the fact that MySQLmodule-1.4 was originally written to emulate an icky perl module, and then the Python API is written around that. I hope to release this soon under a Python-style license. It is not yet well-tested, but appears to work pretty well. -- Andy Dustman (ICQ#32922760) You should always say "spam" and "eggs" ComStar Communications Corp. instead of "foo" and "bar" (706) 549-7689 | PGP KeyID=0xC72F3F1D in Python examples. (Mark Lutz) From Tod Olson Wed Mar 17 20:33:27 1999 From: Tod Olson (Tod Olson) Date: Wed, 17 Mar 1999 14:33:27 -0600 Subject: [DB-SIG] Last round for DB API 1.1 In-Reply-To: Your message of "Wed, 17 Mar 1999 14:26:11 -0500 (EST)" References: Message-ID: <199903172033.OAA02593@stone.lib.uchicago.edu> >>>>> "AD" == Andy Dustman writes: AD> On Wed, 17 Mar 1999, M.-A. Lemburg wrote: >> What to do about Andy Dustmans proposal to have .rollback() >> raise an exception in case transactions are not supported by >> the database ? >> IMHO, we should leave the situation as it is: some DBs do not >> support transactions (e.g. MySQL), some do but you can turn >> transactions off [...] >> If the API were to raise an exception in case tranactions are not >> supported, the underlying interface would have to query the current >> state prior to every .rollback() [...] AD> Well, look at it this way. If you were using a database that AD> supported transactions, and it was operating in a transactional AD> mode, you'd expect an exception if, for some reason, the rollback AD> couldn't be completed. [...The] simplest way to do this is to AD> simply not implement the rollback method if no support is AD> available, which will raise an AttributeError. In a similar vein, if I were porting, say, a Sybase application to MySQL, I would want some indication that a rollback did/could/would not happen. That is, if the DBAPI is to be thought of as an aid to portability, I want some clue when a DB module has unexpected behavior. "def rollback(): pass" seems the worst possible case. I think Andy's notion is pretty sound, and probably generalizable. My 2 cents. (Oh, for a cent sign!) Tod A. Olson "How do you know I'm mad?" said Alice. ta-olson@uchicago.edu "If you weren't mad, you wouldn't have The University of Chicago Library come here," said the Cat. From mal@lemburg.com Wed Mar 17 21:06:31 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 17 Mar 1999 22:06:31 +0100 Subject: [DB-SIG] Last round for DB API 1.1 References: Message-ID: <36F01957.5B288682@lemburg.com> Andy Dustman wrote: > > On Wed, 17 Mar 1999, M.-A. Lemburg wrote: > > > · What to do about Andy Dustmans proposal to have .rollback() > > raise an exception in case transactions are not supported by > > the database ? > > > > IMHO, we should leave the situation as it is: some DBs do not > > support transactions (e.g. MySQL), some do but you can turn > > transactions off (e.g. this is the ODBC default)... all in all > > its a feature that is very dynamic, specific to connections. > > > > If the API were to raise an exception in case tranactions are > > not supported, the underlying interface would have to query the > > current state prior to every .rollback() and even prior to garbage > > collecting the connection object -- at a time where there could > > be no connection available any more. > > Well, look at it this way. If you were using a database that supported > transactions, and it was operating in a transactional mode, you'd expect > an exception if, for some reason, the rollback couldn't be completed. (Are > rollbacks guaranteed to succeed from an SQL standpoint?) It need not be > dynamic in the sense that it is sensitive to the transaction isolation; > presumably the programmer has set this and will only use rollbacks in > non-auto-commit mode. All I am suggesting is that if rollback is called on > a database which cannot possibly roll back a transaction, that some > exception be raised, which is probably preferable to having some program > which depends on rollback continuing on as if the rollback was successful. > This simplest way to do this is to simply not implement the rollback > method if no support is available, which will raise an AttributeError. Hmm, but what if I connect to say MySQL via ODBC... the database does not support transactions but still allows .commit() and .rollback() without raising an exception. If the spec would say that the interface *has* to raise an error, I'd have to check prior to every .rollback() to see if the connection actually does do something with it. Of course, we could add a suggestion to the spec along the lines of "In case the database does not support transactions the interface may throw an exception in case .rollback() is used. This can either be an AttributeError because the interface does not provide the .rollback() method or an OperationalError to indicate the non-ability to perform the roll back." > It seems harmless to allow commit to always succeed and do nothing on a > transactionless database, though, since all statements are essentially > committed once they are executed. True. > > · Andy Dustman also asked for a standard way to find out the > > parameter marker syntax used by the database. > > > > There are at least two possibilities: > > > > ?,?,? in the order of the given parameters > > :1,:3,:2 in any order only referring to the parameter position > > > > Not sure how to get this information into a constant or a method... > > Any ideas ? > > Well, there seem to be three styles, AFAIK: > > PARAMETER_QM = 1 # parameters represented by question marks > PARAMETER_CP = 2 # parameters represented by colon+position number > PARAMETER_CF = 3 # parameters represented by C format string (i.e. %s) > > The main problem, of course, is how to write code to make use of this in a > portable way. This may be beyond the scope of what the DB API can handle, > since it is a function of the query language in use, and the API is not > tied to any particular query language. I suspect there are a lot of > human-written queries out there that will just have to be re-written, and > I'm not sure this will really be useful. I think it is beyond the scope of the API spec. Ideal would be one syntax that could be reformatted to whatever the database uses, e.g. something with keywords much like '... %(id)s ...' % {'id':'123'} but I think that putting it into the 1.1 version goes a little too far. Maybe for 2.0 ;-) > > · Should we add additional constructor signatures to the API spec ? > > > > So far we have Raw,Date,Time,Timestamp. Maybe we should rename > > Raw to BLOB or Binary to be in sync with SQL ?! What about a > > monetary constructor... does anyone have experience with such > > types ? > > Binary would be good, but should be the same as Raw, which would be > deprecated. Ok, so I'll rename Raw to Binary (since we've dropped dbi this is no big deal anyway) and add a note to provide aliases to dbiDate and dbiRaw for backward compatibility. > > · We still need to settle the sequence of sequence problem > > with .execute(). > > > > Since e.g. ('abc',) fits this definition but surely isn't intended, > > we'll need some other way to handle sequences of parameter/data > > sequences. How about depreciating the use of those lists in > > .execute() and defining a .executemany() method for this purpose > > instead ? Other ideas ? > > Right now it says "tuple or list of tuples". I'm happy with that. Greg's not and he's got a point there. I am basically in favor of "sequences of sequences" too; it's just that the ambiguity has to be solved somehow. ".executemany()" would sound much like ".fetchmany"... > Another problem: The spec currently says that when calling the > connection.cursor() method, "An exception may be thrown if the database > does not support a cursor concept." However, if the database does not > support cursors, then you really can't do jack squat with the API. I think > the old version said you could do cursor methods on the connection, but a > better solution is probably to require the database module to emulate > cursors. Ah, that's an oversight on my part. I'll add a note about the emulation. > Take for an example, MySQL. MySQL does not have cursors. It has the > MYSQL_RES structure, which contains a result, but queries are executed on > the connection. However, it is not too hard to emulate a cursor in a > Python wrapper. > > And yes, I have just about finished yet another MySQL interface. This one > is thread-friendly (wraps blocking calls with Py_BEGIN_ALLOW_THREADS ... > Py_END_ALLOW_THREADS), based on the latest MySQL 3.22.19a API, and is in > two parts: A _mysqlmodule.so which implements most of the MySQL API (has > connection objects and result objects; result.describe() returns a DB API > compatible description), and MySQLdb.py which implements the current DB > API. There is a type_conv dictionary which maps MySQL FIELD_TYPEs to > Python functions, so the user can specify how SQL types should be > converted; unspecified types are returned as strings. By default the C > module does not convert the various time types; the Python module sets the > dictionary so they are converted to DateTime objects. On writing to the > database, the interface converts all input parameters to strings and does > the necessary quoting. Note that due to the design, you can subclass both > the connection and cursor objects, as they are implemented in Python. Sounds great. The subclassing part is swell ;-) and the callback concept a really useful approach (with new types popping up for every MySQL release this really helps). BTW: I've noticed that MySQL support BLOBs but does not handle binary data in those BinaryLongOBjects... is that a bug in their ODBC driver or by design ? [would render the Binary() constructor meaningless I guess...] > Compared to MySQLmodule-1.4, the C source is slightly smaller, the .so > library is 40% smaller (i386, YMMV), and the Python interface is 50% > smaller. Probably this is mostly due to the fact that MySQLmodule-1.4 was > originally written to emulate an icky perl module, and then the Python API > is written around that. > > I hope to release this soon under a Python-style license. It is not yet > well-tested, but appears to work pretty well. There's a preliminary test script in mxODBC you may want to adapt for your package. I'm using it to test for memory leaks and to check database capabilities (esp. supported data types and SQL syntax for them). -- Marc-Andre Lemburg Y2000: 289 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : --------------------------------------------------------- From adustman@comstar.net Wed Mar 17 21:54:52 1999 From: adustman@comstar.net (Andy Dustman) Date: Wed, 17 Mar 1999 16:54:52 -0500 (EST) Subject: [DB-SIG] Last round for DB API 1.1 In-Reply-To: <36F01957.5B288682@lemburg.com> Message-ID: On Wed, 17 Mar 1999, M.-A. Lemburg wrote: > Hmm, but what if I connect to say MySQL via ODBC... the database > does not support transactions but still allows .commit() and .rollback() > without raising an exception. If the spec would say that the interface > *has* to raise an error, I'd have to check prior to every .rollback() > to see if the connection actually does do something with it. That's a bit of a tricky situation. I'd have to say the when you use MySQL through ODBC, then it's a matter of the ODBC interface supporting rollback. I'm looking through the myODBC source code and in info.c (implementing SQLGetInfo) I see: case SQL_CURSOR_COMMIT_BEHAVIOR: /* Lie */ case SQL_CURSOR_ROLLBACK_BEHAVIOR: *((SWORD*) rgbInfoValue)=SQL_CB_PRESERVE; *pcbInfoValue=sizeof(SWORD); break; I guess "Lie" says it all. The ODBC interface says it the database supports rollback (a lie), so it does. I think this is a mistake, but one we don't have to perpetuate. > Of course, we could add a suggestion to the spec along the lines > of "In case the database does not support transactions the interface > may throw an exception in case .rollback() is used. This can either > be an AttributeError because the interface does not provide the > .rollback() method or an OperationalError to indicate the non-ability > to perform the roll back." I can probably live with this. ODBC is somewhat of a special case, as it is a sort of intermediate interface to a lot of different databases. Note that according to the Solid docs for SQLTransact, a number of errors can occur during either commit or rollback, including "The driver or data source does not support the ROLLBACK operation". So a proper ODBC interface ought to produce an error resulting in a Python exception (I guess an OperationalError) when used with a database that does not support it. In fact, I'm willing to drop the AttributeError part and go with rollback always generating a OperationError on databases where rollback is not supported. > BTW: I've noticed that MySQL support BLOBs but does not handle > binary data in those BinaryLongOBjects... is that a bug in their ODBC > driver or by design ? [would render the Binary() constructor meaningless > I guess...] I really don't know very much about their ODBC driver. The main "quirk" with MySQL seems to be that all queries are literal, i.e. when they are submitted with mysql_real_query(), the whole query with all values is one (possibly very large if you are doing multi-row inserts) byte-counted string. It is also necessary to escape all strings, whether binary or not, with mysql_escape_string, since it must escape ', \r, \n, and \0, and ' may certainly appear in a non-binary string. It is also fortunate that numeric values may be submitted in quotes, since as far as the Python API is concerned, DateTime objects are numbers... > There's a preliminary test script in mxODBC you may want to adapt > for your package. I'm using it to test for memory leaks and to > check database capabilities (esp. supported data types and SQL > syntax for them). I'd like to give it a try. -- Andy Dustman (ICQ#32922760) You should always say "spam" and "eggs" ComStar Communications Corp. instead of "foo" and "bar" (706) 549-7689 | PGP KeyID=0xC72F3F1D in Python examples. (Mark Lutz) From mal@lemburg.com Wed Mar 17 22:17:14 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 17 Mar 1999 23:17:14 +0100 Subject: [DB-SIG] Last round for DB API 1.1 References: Message-ID: <36F029EA.185110DE@lemburg.com> Andy Dustman wrote: > > On Wed, 17 Mar 1999, M.-A. Lemburg wrote: > > > Hmm, but what if I connect to say MySQL via ODBC... the database > > does not support transactions but still allows .commit() and .rollback() > > without raising an exception. If the spec would say that the interface > > *has* to raise an error, I'd have to check prior to every .rollback() > > to see if the connection actually does do something with it. > > That's a bit of a tricky situation. I'd have to say the when you use MySQL > through ODBC, then it's a matter of the ODBC interface supporting > rollback. I'm looking through the myODBC source code and in info.c > (implementing SQLGetInfo) I see: > > case SQL_CURSOR_COMMIT_BEHAVIOR: /* Lie */ > case SQL_CURSOR_ROLLBACK_BEHAVIOR: > *((SWORD*) rgbInfoValue)=SQL_CB_PRESERVE; > *pcbInfoValue=sizeof(SWORD); > break; > > I guess "Lie" says it all. The ODBC interface says it the database > supports rollback (a lie), so it does. I think this is a mistake, but one > we don't have to perpetuate. I've just implemented a check in mxODBC to check for transactions prior to actually performing the ROLLBACK. It'll raise an OperationalError instead (MySQL doesn't cheat on that one ;). > > Of course, we could add a suggestion to the spec along the lines > > of "In case the database does not support transactions the interface > > may throw an exception in case .rollback() is used. This can either > > be an AttributeError because the interface does not provide the > > .rollback() method or an OperationalError to indicate the non-ability > > to perform the roll back." > > I can probably live with this. ODBC is somewhat of a special case, as it > is a sort of intermediate interface to a lot of different databases. Note > that according to the Solid docs for SQLTransact, a number of errors can > occur during either commit or rollback, including "The driver or data > source does not support the ROLLBACK operation". So a proper ODBC > interface ought to produce an error resulting in a Python exception (I > guess an OperationalError) when used with a database that does not support > it. In fact, I'm willing to drop the AttributeError part and go with > rollback always generating a OperationError on databases where rollback is > not supported. I've added a note that gives the module implementor the two exception possibilities mentioned above: AttributeError (.rollback() not implemented) and OperationalError (.rollback() implemented, but the test is done at execution time). .commit() should always be implemented -- even for databases that do not provide transactions. > > BTW: I've noticed that MySQL support BLOBs but does not handle > > binary data in those BinaryLongOBjects... is that a bug in their ODBC > > driver or by design ? [would render the Binary() constructor meaningless > > I guess...] > > I really don't know very much about their ODBC driver. The main "quirk" > with MySQL seems to be that all queries are literal, i.e. when they are > submitted with mysql_real_query(), the whole query with all values is one > (possibly very large if you are doing multi-row inserts) byte-counted > string. It is also necessary to escape all strings, whether binary or not, > with mysql_escape_string, since it must escape ', \r, \n, and \0, and ' > may certainly appear in a non-binary string. It is also fortunate that > numeric values may be submitted in quotes, since as far as the Python API > is concerned, DateTime objects are numbers... Ah, something else I've noticed with the latest version of MySQL: something in the way of handling time and timestamps changed causing those values to be corrupted in case they include a seconds fraction part, e.g. '12:30:45.5' won't make it into the database, '12:30:45.00' will show up as '00:00:00'. Since the DateTime types produce strings *with* fraction by default, you'll probably have to use the .strftime() methods to pass strings without the fraction part to MySQL. > > There's a preliminary test script in mxODBC you may want to adapt > > for your package. I'm using it to test for memory leaks and to > > check database capabilities (esp. supported data types and SQL > > syntax for them). > > I'd like to give it a try. The latest version is here: http://starship.skyport.net/~lemburg/mxODBC-pre1.0.2.zip The updated DB API spec at: http://starship.skyport.net/~lemburg/DatabaseAPI-1.1.html -- Marc-Andre Lemburg Y2000: 289 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : --------------------------------------------------------- From harri.pasanen@trema.com Wed Mar 17 22:22:13 1999 From: harri.pasanen@trema.com (Harri Pasanen) Date: Wed, 17 Mar 1999 22:22:13 +0000 Subject: [DB-SIG] Anyone working on Sybase module? Message-ID: <36F02B15.5E000F6A@trema.com> Is anyone currently working on a DB-Api compliant Sybase module? I've had fairly good success with Peter Godman's ctsybase module, but got some coredumps with a more complicated thing I played with. Every once in a while I've entertained the idea of starting to work on it, but other priorities have always pushed it back. In the remote chance that I might actually have time to do something, is anyone working on it? With the advent of DB-API 1.1, I'm silently hoping someone will beat me to it. I'm ready to do heavy-duty testing for an API compliant Sybase module. (Sorry, ODBC won't do for my purposes.) Regards, Harri From adustman@comstar.net Wed Mar 17 22:29:25 1999 From: adustman@comstar.net (Andy Dustman) Date: Wed, 17 Mar 1999 17:29:25 -0500 (EST) Subject: [DB-SIG] Last round for DB API 1.1 In-Reply-To: <36F029EA.185110DE@lemburg.com> Message-ID: On Wed, 17 Mar 1999, M.-A. Lemburg wrote: > Ah, something else I've noticed with the latest version of MySQL: > something in the way of handling time and timestamps changed causing > those values to be corrupted in case they include a seconds fraction > part, e.g. '12:30:45.5' won't make it into the database, '12:30:45.00' > will show up as '00:00:00'. Since the DateTime types produce > strings *with* fraction by default, you'll probably have to use > the .strftime() methods to pass strings without the fraction part > to MySQL. This doesn't seem to happen with the TIMESTAMP type, if I send it str(). It truncates the fractional seconds, but otherwise it's okay. Produces a warning as well (on my interface, you can set whether or not Warning should be raised on a per-cursor basis). I have not tried other time types, but could this by a myODBC problem? -- Andy Dustman (ICQ#32922760) You should always say "spam" and "eggs" ComStar Communications Corp. instead of "foo" and "bar" (706) 549-7689 | PGP KeyID=0xC72F3F1D in Python examples. (Mark Lutz) From Tod Olson Wed Mar 17 22:56:49 1999 From: Tod Olson (Tod Olson) Date: Wed, 17 Mar 1999 16:56:49 -0600 Subject: [DB-SIG] Anyone working on Sybase module? In-Reply-To: Your message of "Wed, 17 Mar 1999 22:22:13 +0000" References: <36F02B15.5E000F6A@trema.com> Message-ID: <199903172256.QAA09714@stone.lib.uchicago.edu> >>>>> "HP" == Harri Pasanen writes: HP> Is anyone currently working on a DB-Api compliant Sybase module? HP> I've had fairly good success with Peter Godman's ctsybase module, HP> but got some coredumps with a more complicated thing I played HP> with. HP> Every once in a while I've entertained the idea of starting to HP> work on it, but other priorities have always pushed it back. HP> In the remote chance that I might actually have time to do HP> something, is anyone working on it? With the advent of DB-API HP> 1.1, I'm silently hoping someone will beat me to it. I'm ready to HP> do heavy-duty testing for an API compliant Sybase module. (Sorry, HP> ODBC won't do for my purposes.) I've got one that I started some time ago, built on Ct-Lib. Not all datatypes are well supported, but array binding works. I was working from the guts out, and wanted to get array binding early on. Work stopped when I started on a Java / Sybase project. I can share code with you, if you're interested. I'd love to see something come of this work. Tod A. Olson "How do you know I'm mad?" said Alice. ta-olson@uchicago.edu "If you weren't mad, you wouldn't have The University of Chicago Library come here," said the Cat. From james_northrup@iridium.com Wed Mar 17 22:59:28 1999 From: james_northrup@iridium.com (James Northrup) Date: Wed, 17 Mar 1999 17:59:28 -0500 Subject: [DB-SIG] interogating C langauge drivers References: <199903141701.MAA29819@python.org> <36ECF763.C410A830@iridium.com> <36EE20A2.3BD223C8@lemburg.com> <36EE6FBB.8155CDEE@iridium.com> <36EE93C2.32CF01A4@lemburg.com> <36EFDC7B.A72A7A81@iridium.com> <36EFED6E.3EEC99DF@lyra.org> Message-ID: <36F033D0.77BE53CD@iridium.com> Greg Stein wrote: > James Northrup wrote: > > > > > Hmm. We could try getting the DB API specification into the standard > > > docs for Python. That would add a little more "official" flavor to > > > our work. > > > > questions regarding this sig: > > > > 1) Is there an issue in publishing a module of abstract classes to adhere to ? > > The DB modules aren't going to be able to subclass from them since the > modules must be implemented in C. Therefore, publishing something like > this doesn't really provide any more information than the specification > that we currently have. The specification that we currently have tells me nothing about xyz module that I download in the hopes of interfacing my dbms. It just prescribes doctrine. Without a common parent or wrapper or factory interface or some sort of abstract object, using more than a single db-api module is to double check each step so that one can be assured a generalized interface can be used in core application functionality without rewrite or tuning. and if the check fails? you absolutely will do rewrite and tuning. this is bad practice, regardless of flexibility or constraints of a C-written python module. C is a systems development language. It works well for systems drivers. this db-sig seems to speak of driver specification of various C modules that cannot be subclasses. I am trying address the full application maintenance standpoint of having benefit from this C driver convention. I contest the futility of an abstraction higher than raw calls. python is an application development language. we should not have to suffer for the sins of binary libraries. > > 2) Why stop at the docs for python if we can publish self documented base classes as well? > > I have looked frantically for a dbapi.py and all I have found are disparate c modules that don't have parent class interfaces. > > If such an animal exists, I would love a pointer. I would gladly step out of this discussion and into one about improving such a module. > > There is no such beast. The C modules can't have a parent class > interface (without undue strain, and with little benefit). This is > simply the way Python works. I re-iterate, for the undertaking of strain and little benefit, I dare to tread out of here and into something constructive that lends a stronger application side interface. > > 3) Java has a comfortable "Interface/Impl" flavor of libraries, why is the db-api an ethereal specification of convention rather than an entry point in code interface? > > Goody for Java. > > Python doesn't have strict interface mechanisms, so that programming > methodology does not apply. Publishing a specification for people to > conform to seems to work. There are people that do not conform to the > spec; however, have a base class would not improve the situation... they > just wouldn't use it if they didn't want to conform. As I have tussled about in this sig it occurs to me none have yet spoken from the standpoint of building a kernel or drivers for n*m devices. Everyone wants to write a driver that conforms, only does it better than the spec. So many creative endeavors. As time permits, I shall build a pure python module that performs the task of interrogating modules (or driver, hereafter). These drivers it interrogates will belong to a registry. The user will "register" drivers at runtime. The interrogation results shall provide a driver-capability model for each subject driver. These driver capabilities shall be fashioned directly after the [DB-SIG] annotated 1.1 spec. (initially) As this undertaking nears completion, I will gladly offer this small bit of purpose-written abstraction into a larger base of usage and support. cheers -jim From mal@lemburg.com Thu Mar 18 09:02:24 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 18 Mar 1999 10:02:24 +0100 Subject: [DB-SIG] Last round for DB API 1.1 References: Message-ID: <36F0C120.4E45E477@lemburg.com> Andy Dustman wrote: > > On Wed, 17 Mar 1999, M.-A. Lemburg wrote: > > > Ah, something else I've noticed with the latest version of MySQL: > > something in the way of handling time and timestamps changed causing > > those values to be corrupted in case they include a seconds fraction > > part, e.g. '12:30:45.5' won't make it into the database, '12:30:45.00' > > will show up as '00:00:00'. Since the DateTime types produce > > strings *with* fraction by default, you'll probably have to use > > the .strftime() methods to pass strings without the fraction part > > to MySQL. > > This doesn't seem to happen with the TIMESTAMP type, if I send it > str(). It truncates the fractional seconds, but otherwise > it's okay. Well, a .strftime() approach is probably more flexible. The str() way is really only meant as convenience when printing date/time values. > Produces a warning as well (on my interface, you can set > whether or not Warning should be raised on a per-cursor basis). That's a good one: I've had some trouble lately with drivers (e.g. the MS SQL Server driver) sending warnings at connection time. The result was that no connection was possible because mxODBC kept raising a Warning exception... I guess a callback approach is worth considering for warnings. Or maybe the interface should just remember the Warning exception and raise it *after* it has finished processing the request (a little hackish I'm afraid). > I have not tried other time types, but could this by a myODBC problem? Nope. I've checked it using the mysql direct query interface. It does the same thing: no warnings, truncated or even non-sense data in the database. -- Marc-Andre Lemburg Y2000: 288 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : --------------------------------------------------------- From mal@lemburg.com Thu Mar 18 09:19:31 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 18 Mar 1999 10:19:31 +0100 Subject: [DB-SIG] interogating C langauge drivers References: <199903141701.MAA29819@python.org> <36ECF763.C410A830@iridium.com> <36EE20A2.3BD223C8@lemburg.com> <36EE6FBB.8155CDEE@iridium.com> <36EE93C2.32CF01A4@lemburg.com> <36EFDC7B.A72A7A81@iridium.com> <36EFED6E.3EEC99DF@lyra.org> <36F033D0.77BE53CD@iridium.com> Message-ID: <36F0C523.7016104D@lemburg.com> James Northrup wrote: > > As I have tussled about in this sig it occurs to me none have yet spoken from the standpoint of building a kernel or drivers for n*m devices. Everyone wants to write a > driver that conforms, only does it better than the spec. So many creative endeavors. > > As time permits, I shall build a pure python module that performs the task of interrogating modules (or driver, hereafter). > Andy Dustman's approach has much of this flexibility: it provides a core database C level wrapper and then extends it using a normal Python class to fit the DB API. You can then subclass it, etc. Note that it is also easily possible to put an abstraction layer on top of the DB API definitions to have the subclassing feature for all interface modules out there (and to possibly fix some DB specific quirks). But things tend to get slower with every layer, so this may not be an option to suit everyone. > These drivers it interrogates will belong to a registry. The user will "register" drivers at runtime. > > The interrogation results shall provide a driver-capability model for each subject driver. > > These driver capabilities shall be fashioned directly after the [DB-SIG] annotated 1.1 spec. (initially) > > As this undertaking nears completion, I will gladly offer this small bit of purpose-written abstraction into a larger base of usage and support. > Note that ODBC already offers such a flexible setup. You use one of the available Python interfaces (win32's odbc, mxODBC, Sam Rushing's calldll odbc wrapper or DC's swigged ODBC) and then leave the driver part to the system administrator. It's how I am working in production and so far I have managed quite well (let aside the memory leakage problems that occur with some ODBC drivers). -- Marc-Andre Lemburg Y2000: 288 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : --------------------------------------------------------- From gstein@lyra.org Thu Mar 18 10:32:54 1999 From: gstein@lyra.org (Greg Stein) Date: Thu, 18 Mar 1999 02:32:54 -0800 Subject: [DB-SIG] Last round for DB API 1.1 References: <36F029EA.185110DE@lemburg.com> Message-ID: <36F0D656.340A676D@lyra.org> M.-A. Lemburg wrote: > ... > I've added a note that gives the module implementor the two > exception possibilities mentioned above: AttributeError (.rollback() > not implemented) and OperationalError (.rollback() implemented, > but the test is done at execution time). The *very* strong preference should go towards not implementing .rollback() (by omission or dynamically making it (un)available). This will allow client software to do a hasattr(), rather than attempt a rollback only to find they couldn't... thus leaving their database screwed. > .commit() should always be implemented -- even for databases > that do not provide transactions. yes. Marc: could you include a list of open issues on the web page? It's becoming a pain to keep tracking down the right email :-) Cheers, -g -- Greg Stein, http://www.lyra.org/ From gstein@lyra.org Thu Mar 18 10:44:52 1999 From: gstein@lyra.org (Greg Stein) Date: Thu, 18 Mar 1999 02:44:52 -0800 Subject: [DB-SIG] Last round for DB API 1.1 References: <36EFF0DE.14C814C@lemburg.com> Message-ID: <36F0D924.14AFADA7@lyra.org> M.-A. Lemburg wrote: >... > · What to do about Andy Dustmans proposal to have .rollback() > raise an exception in case transactions are not supported by > the database ? This seems almost closed now... > · Andy Dustman also asked for a standard way to find out the > parameter marker syntax used by the database. > > There are at least two possibilities: > > ?,?,? in the order of the given parameters > :1,:3,:2 in any order only referring to the parameter position > > Not sure how to get this information into a constant or a method... > Any ideas ? I think it behooves us to have a way that a module can state *which* it uses, but we don't *require* a specific format. I will suggest that we add a module global named "paramstyle" with the following values: 'qmark' -- question mark style 'numeric' -- numeric, positional style 'named' -- :name style 'percent' -- ANSI C printf format codes 'xpercent' -- Python extended format codes (e.g. %(name)s) However, we should also note that the API does not have a mechanism for named parameters. That obviates the 'named' and 'xpercent' styles at the moment. Anyways: having this global will allow a higher-level module to do something intelligent. > · Should we add additional constructor signatures to the API spec ? > > So far we have Raw,Date,Time,Timestamp. Maybe we should rename > Raw to BLOB or Binary to be in sync with SQL ?! What about a > monetary constructor... does anyone have experience with such > types ? Per later emails, the Raw was renamed to Binary. I disagree with the need for seperate Date, Time, and Timestamp. I am unaware of an occurrance where the difference was needed. For the past three years, dbiDate() was sufficient, so I question the need for three varieties here. Note: aliases for dbiRaw, dbiDate are not needed since those were names in the "dbi" module, not the database module. > · We still need to settle the sequence of sequence problem > with .execute(). > > Since e.g. ('abc',) fits this definition but surely isn't intended, > we'll need some other way to handle sequences of parameter/data > sequences. How about depreciating the use of those lists in > .execute() and defining a .executemany() method for this purpose > instead ? Other ideas ? Ah. I like executemany() (better than my insert() suggestion). I'll vote for this, with the associated deprecation of passing a seq-of-seq to execute(). Cheers, -g -- Greg Stein, http://www.lyra.org/ From gstein@lyra.org Thu Mar 18 10:52:51 1999 From: gstein@lyra.org (Greg Stein) Date: Thu, 18 Mar 1999 02:52:51 -0800 Subject: [DB-SIG] Last round for DB API 1.1 References: <36F01957.5B288682@lemburg.com> Message-ID: <36F0DB03.56EC3AB9@lyra.org> M.-A. Lemburg wrote: > ... > > > · We still need to settle the sequence of sequence problem > > > with .execute(). > > > > > > Since e.g. ('abc',) fits this definition but surely isn't intended, > > > we'll need some other way to handle sequences of parameter/data > > > sequences. How about depreciating the use of those lists in > > > .execute() and defining a .executemany() method for this purpose > > > instead ? Other ideas ? > > > > Right now it says "tuple or list of tuples". I'm happy with that. > > Greg's not and he's got a point there. I am basically in favor of > "sequences of sequences" too; it's just that the ambiguity has to > be solved somehow. ".executemany()" would sound much like > ".fetchmany"... Yes, the issue is a database module that returns a sophisticated datatype from fetchmany() or fetchall(). That datatype *must* support sequence-of-sequence behavior. We also want to allow that datatype to be passed back to an executemany() call (for example, if you're copying rows between databases). >... > BTW: I've noticed that MySQL support BLOBs but does not handle > binary data in those BinaryLongOBjects... is that a bug in their ODBC > driver or by design ? [would render the Binary() constructor meaningless > I guess...] MySQL does handle long binary data. The Binary() constructor will be a no-op since MySQL doesn't need hints to differentiate the input binding between a VARCHAR and a LONG (like Oracle does). Dates are very tricky in MySQL, however. dbiDate arose out of the need to differentiate between an integer and a Unix ticks value. The database would then prepare the appropriate input binding. A similar need would still exist, so a MySQL database module would probably still require a date type. Note: this comes back to my point about Unix ticks. Marc: you said we should try to get rid of them. Sorry, but that is going to be quite difficult. Python's concept of a date is a Unix ticks value or a 9-tuple. We must have a constructor that accepts one of those things (since we can't pass a bare integer as a date, and I don't think we want to allow passing a 9-tuple to mean a date). Since most Python installations deal with dates in these formats, we must cooperate. Cheers, -g -- Greg Stein, http://www.lyra.org/ From gstein@lyra.org Thu Mar 18 11:06:03 1999 From: gstein@lyra.org (Greg Stein) Date: Thu, 18 Mar 1999 03:06:03 -0800 Subject: [DB-SIG] DBAPI philosophy (was Re: [DB-SIG] Last round for DB API 1.1) References: <199903172033.OAA02593@stone.lib.uchicago.edu> Message-ID: <36F0DE1B.7264C529@lyra.org> Tod Olson wrote: >... > In a similar vein, if I were porting, say, a Sybase application to > MySQL, I would want some indication that a rollback did/could/would > not happen. That is, if the DBAPI is to be thought of as an aid to > portability, I want some clue when a DB module has unexpected > behavior. "def rollback(): pass" seems the worst possible case. Historically, the position has been thus: It is a very complex problem to attempt to provide a uniform interface to all possible databases. The DB-SIG and the DBAPI will not attempt to take on such a task, but will instead create an API specification that brings *consistency* to database client applications. While the APIs will be consistent, they will not be guaranteed to be replaceable. ODBC does a pretty good job of being a uniform interface, and Python has several bindings to ODBC. However, there are just too many tweaky things in a database to be able to build truly portable applications (differences in column types, SQL syntax, available SQL functions, index behavior, stored procedures, etc). Since you (typically) cannot have an application that is truly portable, there is not a lot of reason to guarantee portability at the DBAPI level (when you recode the app, you may need to recode the API calls). I believe the DBAPI has largely met its goals. Python programmers familiar with the DBAPI can rapidly come up to speed against new databases. I just installed MySQL for the first time a month ago. Took me all of 10 minutes to write some Python code to call it. Sure, the parameter binding was different from my experience with Oracle, Informix, and ODBC modules, but I knew the API already. I was able to get something done... quickly. While it is perfectly reasonable for us to change the philosophy behind the DBAPI and move towards stricter conformance, I do not think that we should delude ourselves into believing that we can reach that goal. I think that we can continue using our present philosophy and bring continued, tremendous benefits to the Python/database community. Over time, the modules will increase their conformance naturally (e.g. Andy's new MySQL module is targetted at better conformance than the previous version). Back to Tod's pseudo-question: yes, the DBAPI *aids* portability, but even strict conformance, by the author and client, will not and cannot guarantee portability. Mostly, I view the DBAPI as aiding *understanding* and *learnability* of the individual DB modules. Cheers, -g -- Greg Stein, http://www.lyra.org/ From gstein@lyra.org Thu Mar 18 11:08:48 1999 From: gstein@lyra.org (Greg Stein) Date: Thu, 18 Mar 1999 03:08:48 -0800 Subject: [DB-SIG] Last round for DB API 1.1 References: Message-ID: <36F0DEC0.45E2F49E@lyra.org> Andy Dustman wrote: >... > I can probably live with this. ODBC is somewhat of a special case, as it > is a sort of intermediate interface to a lot of different databases. Note > that according to the Solid docs for SQLTransact, a number of errors can > occur during either commit or rollback, including "The driver or data > source does not support the ROLLBACK operation". So a proper ODBC > interface ought to produce an error resulting in a Python exception (I > guess an OperationalError) when used with a database that does not support > it. In fact, I'm willing to drop the AttributeError part and go with > rollback always generating a OperationError on databases where rollback is > not supported. As I stated in another email: I would strongly suggest that an AttributeError is highly preferable over an OperationError. > > BTW: I've noticed that MySQL support BLOBs but does not handle > > binary data in those BinaryLongOBjects... is that a bug in their ODBC > > driver or by design ? [would render the Binary() constructor meaningless > > I guess...] > > I really don't know very much about their ODBC driver. The main "quirk" > with MySQL seems to be that all queries are literal, i.e. when they are > submitted with mysql_real_query(), the whole query with all values is one > (possibly very large if you are doing multi-row inserts) byte-counted > string. It is also necessary to escape all strings, whether binary or not, > with mysql_escape_string, since it must escape ', \r, \n, and \0, and ' > may certainly appear in a non-binary string. It is also fortunate that This is really bogus, unfortunately. I think there is also a limit on the length of the SQL statement, meaning you can't insert those 537k blobs :-( Cheers, -g -- Greg Stein, http://www.lyra.org/ From mal@lemburg.com Thu Mar 18 12:26:37 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 18 Mar 1999 13:26:37 +0100 Subject: [DB-SIG] Last round for DB API 1.1 References: <36F029EA.185110DE@lemburg.com> <36F0D656.340A676D@lyra.org> Message-ID: <36F0F0FD.72E3BB79@lemburg.com> Greg Stein wrote: > > M.-A. Lemburg wrote: > > ... > > I've added a note that gives the module implementor the two > > exception possibilities mentioned above: AttributeError (.rollback() > > not implemented) and OperationalError (.rollback() implemented, > > but the test is done at execution time). > > The *very* strong preference should go towards not implementing > .rollback() (by omission or dynamically making it (un)available). > > This will allow client software to do a hasattr(), rather than attempt a > rollback only to find they couldn't... thus leaving their database > screwed. Ok. > > .commit() should always be implemented -- even for databases > > that do not provide transactions. > > yes. > > Marc: could you include a list of open issues on the web page? It's > becoming a pain to keep tracking down the right email :-) Ok. > > · Andy Dustman also asked for a standard way to find out the > > parameter marker syntax used by the database. > > > > There are at least two possibilities: > > > > ?,?,? in the order of the given parameters > > :1,:3,:2 in any order only referring to the parameter position > > > > Not sure how to get this information into a constant or a method... > > Any ideas ? > > I think it behooves us to have a way that a module can state *which* it > uses, but we don't *require* a specific format. > > I will suggest that we add a module global named "paramstyle" with the > following values: > > 'qmark' -- question mark style > 'numeric' -- numeric, positional style > 'named' -- :name style > 'percent' -- ANSI C printf format codes > 'xpercent' -- Python extended format codes (e.g. %(name)s) > > However, we should also note that the API does not have a mechanism for > named parameters. That obviates the 'named' and 'xpercent' styles at the > moment. > > Anyways: having this global will allow a higher-level module to do > something intelligent. Right. I'll add that global with your proposed values. > > · Should we add additional constructor signatures to the API spec ? > > > > So far we have Raw,Date,Time,Timestamp. Maybe we should rename > > Raw to BLOB or Binary to be in sync with SQL ?! What about a > > monetary constructor... does anyone have experience with such > > types ? > > Per later emails, the Raw was renamed to Binary. > > I disagree with the need for seperate Date, Time, and Timestamp. I am > unaware of an occurrance where the difference was needed. For the past > three years, dbiDate() was sufficient, so I question the need for three > varieties here. Well, I'd say it doesn't hurt to have the variety and on the other hand it may even be necessary in case the database screws up on specific time formats. > Note: aliases for dbiRaw, dbiDate are not needed since those were names > in the "dbi" module, not the database module. Hmm. It might make porting a little easier. > > · We still need to settle the sequence of sequence problem > > with .execute(). > > > > Since e.g. ('abc',) fits this definition but surely isn't intended, > > we'll need some other way to handle sequences of parameter/data > > sequences. How about depreciating the use of those lists in > > .execute() and defining a .executemany() method for this purpose > > instead ? Other ideas ? > > Ah. I like executemany() (better than my insert() suggestion). I'll vote > for this, with the associated deprecation of passing a seq-of-seq to > execute(). Anybody else in favor of this change ? If there are no objections, I'll add it to the spec. > > BTW: I've noticed that MySQL support BLOBs but does not handle > > binary data in those BinaryLongOBjects... is that a bug in their ODBC > > driver or by design ? [would render the Binary() constructor meaningless > > I guess...] > > MySQL does handle long binary data. The direct interface seems to do this. The output of the test script Andy sent me indicates that his interface does in fact also handle binary data correctly. The MyODBC driver screws this up though as the test script complains with my setup (mxODBC-> MyODBC->MySQL). I guess another bug report is due... > [...] > Dates are very tricky in MySQL, however. dbiDate arose out of the need > to differentiate between an integer and a Unix ticks value. The database > would then prepare the appropriate input binding. A similar need would > still exist, so a MySQL database module would probably still require a > date type. > > Note: this comes back to my point about Unix ticks. Marc: you said we > should try to get rid of them. Sorry, but that is going to be quite > difficult. Python's concept of a date is a Unix ticks value or a > 9-tuple. We must have a constructor that accepts one of those things > (since we can't pass a bare integer as a date, and I don't think we want > to allow passing a 9-tuple to mean a date). Since most Python > installations deal with dates in these formats, we must cooperate. Hmm, I really don't like to get those ticks into the spec again (after all they were the reason why I started with mxDateTime in the first place). SQL defines date ranges to 0001 - 9999: that's a range not reliably handleable using Unix ticks. But there's a simple solution: provide constructors that do the ticks (or seconds since midnight integers) conversion to Date,Time,Timestamp constructor input, e.g. def DateFromTicks(ticks): import time return apply(Date,time.localtime(ticks)[:3]) These helpers should be left to the module implementor though (or we could provide a standard dbapi.py module providing these). Just please leave those date/time issues out of the spec. Next, someone will want COM dates ;-) All of these things are handled by mxDateTime for those that want the full monty of date/time formats. I'll update the spec page later today. -- Marc-Andre Lemburg Y2000: 288 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : --------------------------------------------------------- From james_northrup@iridium.com Thu Mar 18 17:30:32 1999 From: james_northrup@iridium.com (James Northrup) Date: Thu, 18 Mar 1999 12:30:32 -0500 Subject: [DB-SIG] interogating C language drivers References: <199903141701.MAA29819@python.org> <36ECF763.C410A830@iridium.com> <36EE20A2.3BD223C8@lemburg.com> <36EE6FBB.8155CDEE@iridium.com> <36EE93C2.32CF01A4@lemburg.com> <36EFDC7B.A72A7A81@iridium.com> <36EFED6E.3EEC99DF@lyra.org> <36F033D0.77BE53CD@iridium.com> <36F0C523.7016104D@lemburg.com> Message-ID: <36F13838.82460B35@iridium.com> Mark, you have presented a veritable shopping list of independant efforts to examine and coalesce into something like a least-common-denominator wrapper/capabilities-model. seems very similar to the task of writing a configure script with autoconf, a series test/examine/remark iterations. e.g. boolean capability_node=function named 'x' of module 'z' returns result 'y' with type 'y.type' with value 'y.value' given parameters p[...] with types p[...].type and values p[...].value as for performance issues of wrapper classes, I like the flexibility provided by python to interrogate a module and pass off a function pointer, if one decides to build a structure of lean and mean objects. (These kind of performance enhancements wouldn't seem appropriate to an API specification in a scripting language, just an idiom for use at an application level. My goal is to reduce application maintenance overhead for using a dbms in production.) Thanks for your input -jim p.s. the ending remarks below concerning ODBC are exactly what I hope can apply to db-api, in an out-of-the-box python base module. "M.-A. Lemburg" wrote: > James Northrup wrote: > > > > As I have tussled about in this sig it occurs to me none have yet spoken from the standpoint of building a kernel or drivers for n*m devices. Everyone wants to write a > > driver that conforms, only does it better than the spec. So many creative endeavors. > > > > As time permits, I shall build a pure python module that performs the task of interrogating modules (or driver, hereafter). > > > > Andy Dustman's approach has much of this flexibility: it provides > a core database C level wrapper and then extends it using a normal > Python class to fit the DB API. You can then subclass it, etc. > > Note that it is also easily possible to put an abstraction layer on top > of the DB API definitions to have the subclassing feature for > all interface modules out there (and to possibly fix some DB specific > quirks). But things tend to get slower with every layer, so this > may not be an option to suit everyone. > > These drivers it interrogates will belong to a registry. The user will "register" drivers at runtime. > > > > The interrogation results shall provide a driver-capability model for each subject driver. > > > > These driver capabilities shall be fashioned directly after the [DB-SIG] annotated 1.1 spec. (initially) > > > > As this undertaking nears completion, I will gladly offer this small bit of purpose-written abstraction into a larger base of usage and support. > > > > Note that ODBC already offers such a flexible setup. You use > one of the available Python interfaces (win32's odbc, mxODBC, > Sam Rushing's calldll odbc wrapper or DC's swigged ODBC) and > then leave the driver part to the system administrator. > > It's how I am working in production and so far I have managed > quite well (let aside the memory leakage problems that occur > with some ODBC drivers). From adustman@comstar.net Thu Mar 18 19:13:24 1999 From: adustman@comstar.net (Andy Dustman) Date: Thu, 18 Mar 1999 14:13:24 -0500 (EST) Subject: [DB-SIG] Last round for DB API 1.1 In-Reply-To: <36F0D656.340A676D@lyra.org> Message-ID: On Thu, 18 Mar 1999, Greg Stein wrote: > The *very* strong preference should go towards not implementing > .rollback() (by omission or dynamically making it (un)available). > > This will allow client software to do a hasattr(), rather than attempt a > rollback only to find they couldn't... thus leaving their database > screwed. I was somewhat ambivalent about what exception should be raised, but I think this is the best argument. Plus, it means I get to delete a line of code:) def rollback(self): raise OperationalError, "transactions not supported" -- Andy Dustman (ICQ#32922760) You should always say "spam" and "eggs" ComStar Communications Corp. instead of "foo" and "bar" (706) 549-7689 | PGP KeyID=0xC72F3F1D in Python examples. (Mark Lutz) From mal@lemburg.com Wed Mar 24 15:58:17 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 24 Mar 1999 16:58:17 +0100 Subject: [DB-SIG] New update for DB API 1.1 Message-ID: <36F90B99.513E833F@lemburg.com> I've (finally) updated the DB API 1.1 proposal with all the new enhancements and also added a list of open issues: http://starship.skyport.net/~lemburg/DatabaseAPI-1.1.html -- Marc-Andre Lemburg Y2000: 282 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : --------------------------------------------------------- From adustman@comstar.net Wed Mar 24 18:22:00 1999 From: adustman@comstar.net (Andy Dustman) Date: Wed, 24 Mar 1999 13:22:00 -0500 (EST) Subject: [DB-SIG] New update for DB API 1.1 In-Reply-To: <36F90B99.513E833F@lemburg.com> Message-ID: On Wed, 24 Mar 1999, M.-A. Lemburg wrote: > I've (finally) updated the DB API 1.1 proposal with all the new > enhancements and also added a list of open issues: Okay, hopefully this will end up being the last round of comments and we can wrap this up soon: paramstyle: Historically, because of the way MySQL works, the interfaces have used the "percent" style. However, it would also be fairly easy to also implement the "xpercent" style. From an implementation standpoint, this could mean having an alternate executedict method which is like execute but expects a dictionary. Or, executeobject which is like executedict but expects an object and uses it's __dict__. (This last one, unfortunately, avoids use of __getattr__, which is probably a bad thing.) Possibly execute could test to see if a dictionary is present and use the "xpercent" style instead of "percent". Anyway, the real point is: Multiple paramstyles are possile for a given database. The two obvious combinations are "percent" and "xpercent" and possibly "qmark" or "numeric" and "named". Might I also suggest a renaming? Use "format" for the ANSI C type format strings and "dictformat" for Python type named format strings. Another question is: Does anyone think it would be particularly beneficial to allow execute or a varient to work on a dictionary? I think it could be, though I'm not sure I would personally use it. I have a module (SQLDict) that lets me define Python objects based on a list of column names, and these objects can "register" with a SQLDict connection which then creates the necessary SQL for inserting, updating, and deletion. So the dictionary feature would not be much use to me, but could be useful for other applications. Lastly, I'm not sure why the buffer object should be the preferred object for binary data. I did "discover" them at one point. Basically they seem to be mutable strings. Well, really just a C-style buffer. I guess I don't see the benefit over using strings. If the normal string/sequence operations are available, I suppose it's not a big problem. BTW, I do have permission to release the new MySQL interface. I am waiting on word that the lawyers have approved the licensing agreement. Don't worry, it's the Python license with the words "Stichting Mathematisch Centrum" scratched out and "ComStar Communications Corporation" written in with a crayon, and we get rights to your internal organs (by reading this message, you have consented to be bound to the license agreement and other terms we may think up later). -- Andy Dustman (ICQ#32922760) You should always say "spam" and "eggs" ComStar Communications Corp. instead of "foo" and "bar" (706) 549-7689 | PGP KeyID=0xC72F3F1D in Python examples. (Mark Lutz) From adustman@comstar.net Wed Mar 24 22:18:30 1999 From: adustman@comstar.net (Andy Dustman) Date: Wed, 24 Mar 1999 17:18:30 -0500 (EST) Subject: [DB-SIG] New update for DB API 1.1 In-Reply-To: <36F90B99.513E833F@lemburg.com> Message-ID: Okay, one more thing that bugs me, and this is somewhat more serious. We still have the various type indicators which are inherited from the defunct dbi module, i.e. BINARY (was RAW), STRING, NUMBER, DATE, TIME, TIMESTAMP, and ROWID. (The constructors seem fine.) DATE, TIME, and TIMESTAMP directly correspond to MySQL types. MySQL doesn't have anything that corresponds to ROWID. And there are several types which would qualify as NUMBER or STRING: >>> dir(_mysql.FIELD_TYPE) ['BLOB', 'CHAR', 'DATE', 'DATETIME', 'DECIMAL', 'DOUBLE', 'ENUM', 'FLOAT', 'INT24', 'INTERVAL', 'LONG', 'LONGLONG', 'LONG_BLOB', 'MEDIUM_BLOB', 'NEWDATE', 'NULL', 'SET', 'SHORT', 'STRING', 'TIME', 'TIMESTAMP', 'TINY', 'TINY_BLOB', 'VAR_STRING', 'YEAR', '__doc__', '__module__'] In addition, design of the module has obviated (word for the day) for the dbi constants. At the lower level (_mysql), there are result objects, and they have a describe() method which produces the description for the result. At present, it uses the MySQL type codes. But the application really never has to look at these codes, because I also employ a module-wide dictionary that maps MySQL types to Python functions which convert a string (what MySQL always returns) into a Python type. There are some reasonable defaults for numbers and they are also pluggable at run-time, and MySQLdb does this for the date/time types (and anything not mapped is returned as a string): >>> MySQLdb.type_conv {13: , 12: , 11: , 10: , 9: , 8: , 7: , 5: , 4: , 3: , 2: , 1: } So the old dbi types are really not useful at all for MySQL. In my experience, they are not useful for mxODBC, at least with Solid. I realize other databases may need some help doing the type conversion. But, they still don't seem very useful. Hmmm, should that NUMBER column be converted to an int, a long, or a float? Is that BINARY column a BLOB, LONG_BLOB, MEDIUM_BLOB, or a TINY_BLOB? Generally, these are things you need to know before you can carry out type conversion. This is what I would propose: 1) Make type code field in description database-dependent. 2) Provide a set of truth functions which take the place of comparing against fixed type code. For example, instead of: if typecode == NUMBER: ... to: if isnumber(typecode): ... I'm not really even sure that #2 is even necessary. I think the database ought to convert columns into standard Python types, or other types if possible, and otherwise return strings (or buffers) for "weird" types, like ENUM or SET. (And NULL -> None, obviously.) Converting from Python types to column types is another matter. For my MySQLdb, it's been sufficient to call str() on each parameter, make sure special characters are quoted, and then add the necessary quotation marks. Maybe I'm just not understanding this section, or the section doesn't really mean what it says: "A Cursor Object's description attribute returns information about each of the result columns of a query. The type_code must be equal to one of type codes defined below. "Note: The values returned in the description tuple may not necessarily be identical to the defined type codes, i.e. while coltype == STRING should always work, coltype is STRING may fail. This is necessary because the database interface may want to return more specific type codes than the ones defined below." But if the database is returning a more specific type code, how can it be equal to one of the defined type codes if several of the specific type codes map to a defined type code, such as BLOB, TINY_BLOB, MEDIUM_BLOB, LONG_BLOB all mapping to BINARY? I suppose this would work: if typecode in BINARY: ... assuming BINARY = (BLOB, TINY_BLOB, MEDIUM_BLOB, LONG_BLOB). Is it useful? Not particularly, at least for me, but it seems more workable than the current API... My database experience with the API is limited to Solid with mxODBC and MySQL, so if someone else out there has some perspective of why we need this stuff for some other database, let's hear it. The last bit, what to do with NUMERIC/DECIMAL SQL fields (i.e. fixed-point)? Obviously converting them to integers is RIGHT OUT, and converting them to floats is probably not a good idea either. At the moment I just leave 'em as strings and let the application figure it out. This may need to just stay an open issue for awhile unless someone wants to come up with a fixed-point type for Python (perhaps a wrapper around long?). -- Andy Dustman (ICQ#32922760) You should always say "spam" and "eggs" ComStar Communications Corp. instead of "foo" and "bar" (706) 549-7689 | PGP KeyID=0xC72F3F1D in Python examples. (Mark Lutz) From mal@lemburg.com Thu Mar 25 12:32:42 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 25 Mar 1999 13:32:42 +0100 Subject: [DB-SIG] New update for DB API 1.1 References: Message-ID: <36FA2CEA.247A62DC@lemburg.com> Andy Dustman wrote: > > On Wed, 24 Mar 1999, M.-A. Lemburg wrote: > > > I've (finally) updated the DB API 1.1 proposal with all the new > > enhancements and also added a list of open issues: > > Okay, hopefully this will end up being the last round of comments and we > can wrap this up soon: > > paramstyle: Historically, because of the way MySQL works, the interfaces > have used the "percent" style. However, it would also be fairly easy to > also implement the "xpercent" style. From an implementation standpoint, > this could mean having an alternate executedict method which is like > execute but expects a dictionary. Or, executeobject which is like > executedict but expects an object and uses it's __dict__. (This last one, > unfortunately, avoids use of __getattr__, which is probably a bad thing.) Hmm, .execute() allows the params argument to be a sequence. Now in C a Python sequence needs to provide the tp_as_sequence slot, yet in Python most APIs only rely on a certain interface to be available, e.g. __len__ and __getitem__. We could loosen the requirement in the spec to objects having a length and a __getitem__ slot. That way, both traditional sequences such as lists, tuples, etc. would fit the definition, but also mappings such as dictionaries. If the parameter style is name based, the given names would be passed to __getitem__, for position based parameters, the index would serve as argument. The extract() function in mxTools is an example of how useful this can be: >>> extract(sys.modules,('sys','os')) [, ] >>> extract(sys.modules.keys(),(0,2,1)) ['os', 'posix', 'exceptions'] > Possibly execute could test to see if a dictionary is present and use the > "xpercent" style instead of "percent". Anyway, the real point is: Multiple > paramstyles are possile for a given database. The two obvious combinations > are "percent" and "xpercent" and possibly "qmark" or "numeric" and > "named". So maybe "paramstyle" should be a sequence of such strings ?! How would the .execute method then check which style is being used ? > Might I also suggest a renaming? Use "format" for the ANSI C type format > strings and "dictformat" for Python type named format strings. How about "format" and "pyformat" ?! > Another question is: Does anyone think it would be particularly beneficial > to allow execute or a varient to work on a dictionary? I think it could > be, though I'm not sure I would personally use it. I have a module > (SQLDict) that lets me define Python objects based on a list of column > names, and these objects can "register" with a SQLDict connection which > then creates the necessary SQL for inserting, updating, and deletion. So > the dictionary feature would not be much use to me, but could be useful > for other applications. I think widing the definition of "sequence" as proposed above would make some nifty things possible, e.g. you could pass an object with __getitem__() method to it which then decides how to fetch the arguments: · if the __getitem__ argument is an integer, positional reference is needed · if the argument is a non-integer, named reference is intended > Lastly, I'm not sure why the buffer object should be the preferred object > for binary data. I did "discover" them at one point. Basically they seem > to be mutable strings. Well, really just a C-style buffer. I guess I don't > see the benefit over using strings. If the normal string/sequence > operations are available, I suppose it's not a big problem. The module may need to recognize the binary object type and since strings are the preferred type for any other string based data, the buffers fit in nicely as "alternative strings" ;-) > BTW, I do have permission to release the new MySQL interface. I am waiting > on word that the lawyers have approved the licensing agreement. Don't > worry, it's the Python license with the words "Stichting Mathematisch > Centrum" scratched out and "ComStar Communications Corporation" written in > with a crayon, and we get rights to your internal organs (by reading this > message, you have consented to be bound to the license agreement and > other terms we may think up later). Cool. -- Marc-Andre Lemburg Y2000: 282 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : --------------------------------------------------------- From mal@lemburg.com Thu Mar 25 12:32:44 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 25 Mar 1999 13:32:44 +0100 Subject: [DB-SIG] New update for DB API 1.1 References: Message-ID: <36FA2CEC.577BF014@lemburg.com> Andy Dustman wrote: > > Okay, one more thing that bugs me, and this is somewhat more serious. > > We still have the various type indicators which are inherited from the > defunct dbi module, i.e. BINARY (was RAW), STRING, NUMBER, DATE, TIME, > TIMESTAMP, and ROWID. (The constructors seem fine.) DATE, TIME, and > TIMESTAMP directly correspond to MySQL types. MySQL doesn't have anything > that corresponds to ROWID. Then ROWID should not be equal to any description field type code, e.g. see the example from mxODBC below. > And there are several types which would qualify > as NUMBER or STRING > > [...] > > This is what I would propose: > > 1) Make type code field in description database-dependent. This is already possible. Note that I changed the wording of the definition of type objects (the parts you quoted). In mxODBC for example I provide all the different SQL data type codes (as integers) in the description field; the type objects are of the form: # Helper for coltypes class _dbiSet: def __init__(self,*values): self.values = values def __cmp__(self,other): if other in self.values: return 0 if other < self.values: return 1 else: return -1 # Note that tests like 'coltype is STRING' don't work, you have # use 'coltype == STRING'. STRING = _dbiSet(SQL.CHAR, SQL.LONGVARCHAR) BINARY = _dbiSet(SQL.BINARY, SQL.LONGVARBINARY) NUMBER = _dbiSet(SQL.DECIMAL, SQL.DOUBLE, SQL.FLOAT, SQL.INTEGER) DATE = _dbiSet(SQL.DATE) # mxODBC doesn't support RowID columns; use cursor.specialcolumns() to # find out which columns are best suited to uniquely identify a row. # This object is not equal to any other object. ROWID = _dbiSet() > 2) Provide a set of truth functions which take the place of comparing > against fixed type code. For example, instead of: > > if typecode == NUMBER: ... > > to: > > if isnumber(typecode): ... You can use the above approach to meet the DB API requirements and still provide more detailed database dependent information in the description field. > Maybe I'm just not understanding this section, or the section doesn't > really mean what it says: > > "A Cursor Object's description attribute returns information about each of > the result columns of a query. The type_code must be equal to one of type > codes defined below. > > "Note: The values returned in the description tuple may not necessarily be > identical to the defined type codes, i.e. while coltype == STRING should > always work, coltype is STRING may fail. This is necessary because the > database interface may want to return more specific type codes than the > ones defined below." I'm not a too good writer and even worse have a mathematical background, so maybe someone could edit the two sections to clearify them... > The last bit, what to do with NUMERIC/DECIMAL SQL fields (i.e. > fixed-point)? Obviously converting them to integers is RIGHT OUT, and > converting them to floats is probably not a good idea either. At the > moment I just leave 'em as strings and let the application figure it out. > This may need to just stay an open issue for awhile unless someone wants > to come up with a fixed-point type for Python (perhaps a wrapper around > long?). mxODBC converts them to Python floats; but you're right: a fixed point type a la the one that Tim Peters posted on c.l.p written in C wouldn't be a bad idea (it could then also be used for monetary values). For now, I think floats are the way to go: at least they preserve the fraction part. -- Marc-Andre Lemburg Y2000: 282 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : --------------------------------------------------------- From mal@lemburg.com Fri Mar 26 15:25:08 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Fri, 26 Mar 1999 16:25:08 +0100 Subject: [DB-SIG] Remaining issues with DB API 1.1 Message-ID: <36FBA6D4.A6511A1@lemburg.com> Hi everybody, According to the proposed schedule for the DB API, today (1999-03-26) is the Final Call deadline. We made some very good progress so far and I would like to thank everybody who joined in. Some open issues remain which should be settled before going public with the new API spec (and my comments): · Require a constructor to take Unix ticks as input for timestamp values ? What about time (without date part) and date (without time part) values ? Well, you know my POV: ticks are a bad thing for serious DB interfacing and there are ways to convert them to the broken down values needed for the constructors. · Should we add additional constructor signatures to the API spec ? E.g. for monetary values (not defined in standard SQL btw). I suggest postponing this decision to the next spec release. · Are setinputsize, setoutputsize optional in the sense that the APIs itself may be left out by the module implementor or rather that they must be provided but may be implemented with void functionality. I guess they should always be available. · Naming of some of the values for the .paramstyle attribute: Andy Dustman proposed to change "percent" and "xpercent" to "format" and "dictformat". I prefer "pyformat" instead of "dictformat"... · Should we extend the notion of "sequences" in .execute() to "objects providing __len__ and __getitem__ methods" ? See my reply to Andy Dustman for a detailed explanation of the benefits. · Add a section pointing out the changes from 1.0 to 1.1 and ways to port existing scripts to the updated API spec. Could someone help with this one ?! · Spell correct and grammar check the spec... Some additional infos and URLs: Timeline: Final Call deadline: 1999-03-26 Posting of the Spec: 1999-03-29 The current working version of the Spec. can always be found at: http://starship.skyport.net/~lemburg/DatabaseAPI-1.1.html The 1.0 Spec. is here: http://www.python.org/sigs/db-sig/DatabaseAPI.html Greg's annotated version 1.1: http://www.lyra.org/greg/anno-DatabaseAPI-1.1.html -- Marc-Andre Lemburg Y2000: 280 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : --------------------------------------------------------- From adustman@comstar.net Sat Mar 27 06:43:23 1999 From: adustman@comstar.net (Andy Dustman) Date: Sat, 27 Mar 1999 01:43:23 -0500 (EST) Subject: [DB-SIG] Remaining issues with DB API 1.1 In-Reply-To: <36FBA6D4.A6511A1@lemburg.com> Message-ID: On Fri, 26 Mar 1999, M.-A. Lemburg wrote: > · Require a constructor to take Unix ticks as input for timestamp > values ? What about time (without date part) and date (without > time part) values ? > > Well, you know my POV: ticks are a bad thing for serious DB interfacing > and there are ways to convert them to the broken down values needed for > the constructors. OTOH, ticks are pretty ubiquitous. Supplying a constructor that uses them isn't a big deal; mxDateTime has one, right? Certainly it should not be the only constructor, but available as an alternative. I'm agreeable to everything else. I would help on the spelling/grammar but I will probably be spending this weekend working on a Python intro for the local ACM chapter that I'm presenting on tuesday. -- Andy Dustman (ICQ#32922760) You should always say "spam" and "eggs" ComStar Communications Corp. instead of "foo" and "bar" (706) 549-7689 | PGP KeyID=0xC72F3F1D in Python examples. (Mark Lutz) From mal@lemburg.com Sat Mar 27 09:52:14 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Sat, 27 Mar 1999 10:52:14 +0100 Subject: [DB-SIG] Remaining issues with DB API 1.1 References: Message-ID: <36FCAA4E.1A600D6A@lemburg.com> Andy Dustman wrote: > > On Fri, 26 Mar 1999, M.-A. Lemburg wrote: > > > · Require a constructor to take Unix ticks as input for timestamp > > values ? What about time (without date part) and date (without > > time part) values ? > > > > Well, you know my POV: ticks are a bad thing for serious DB interfacing > > and there are ways to convert them to the broken down values needed for > > the constructors. > > OTOH, ticks are pretty ubiquitous. Supplying a constructor that uses them > isn't a big deal; mxDateTime has one, right? Certainly it should not be > the only constructor, but available as an alternative. Sure, mxDateTime has constructors that take ticks as argument and there is also a generic solution: def DateFromTicks(ticks): return apply(Date,time.localtime(ticks)[:3]) def TimeFromTicks(ticks): return apply(Time,time.localtime(ticks)[3:6]) def TimestampFromTicks(ticks): return apply(Timestamp,time.localtime(ticks)[:6]) Still, I think we should not put any emphasis on using them for database interfacing by requiring the above constructors in the spec. A module implementor is always free to provide the constructors and a user can supply them herself in case she wants to use ticks. > I'm agreeable to everything else. I would help on the spelling/grammar but > I will probably be spending this weekend working on a Python intro for the > local ACM chapter that I'm presenting on tuesday. Ok. Another issue that I've overlooked previously... The 1.0 API spec for the fetchmany() method is: fetchmany([size]) Fetch the next set of rows of a query result, returning as a list of tuples. An empty list is returned when no more rows are available. The number of rows to fetch is specified by the parameter. If it is None, then the cursor's arraysize determines the number of rows to be fetched. That last part is bogus, IMHO. Also, the spec does not state how many rows to fetch if size is completely omitted. How about specifying size as optional argument that defaults to cursor.arraysize ?! ... fetchmany([size=cursor.arraysize]) Fetch the next set of rows of a query result, returning as a list of tuples. An empty list is returned when no more rows are available. The number of rows to fetch is specified by the parameter. If it is not given, the cursor's arraysize determines the number of rows to be fetched. A similar change should be done for setoutputsize(size[,col]) w/r to the col argument. I've updated the spec according the above lines (which, of course, doesn't mean you can't object ;-) : http://starship.skyport.net/~lemburg/DatabaseAPI-1.1.html I've also updated all the references to tuples/lists to sequences and mappings for maximum flexibility. Tell me what you think about it (BTW, Jim Fulton's result objects should now fit in nicely with this spec). Cheers, -- Marc-Andre Lemburg Y2000: 279 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : --------------------------------------------------------- From MHammond@skippinet.com.au Sun Mar 28 01:54:59 1999 From: MHammond@skippinet.com.au (Mark Hammond) Date: Sun, 28 Mar 1999 11:54:59 +1000 Subject: [DB-SIG] Remaining issues with DB API 1.1 In-Reply-To: <36FCAA4E.1A600D6A@lemburg.com> Message-ID: <006301be78bd$fd203110$0801a8c0@bobcat> > available. The number of rows to fetch is specified by the > parameter. If it is None, then the cursor's > arraysize determines > the number of rows to be fetched. > > That last part is bogus, IMHO. Also, the spec does not state how > many rows to fetch if size is completely omitted. How about specifying > size as optional argument that defaults to cursor.arraysize ?! ... I think the spec is fine - maybe just needs to be changed to "if it is None or not specified ...". Then again, explicitely stating the use of "None" makes it a PITA for implementations in C - it pretty much means you can't use PyArg_ParseTuple(args, "|i", &size); So to remove some burden on the implementors, it could be stated simply as "If it is not specified, then the cursor's arraysize determines the number of rows to be fetched.". This leaves implementors using C to use PyArg_ParseTuple, but implementors in Python to use "def xx(size = None):" - they are keeping to the spec, but users are not free to make an assumption about what the default is - they know the behaviour, not the implementation. No need to specify in the spec exactly _what_ the default should be - it seems sufficient to describe the behaviour - ie, "if not specified", rather than "the default value is xxx" > I've updated the spec according the above lines (which, of course, > doesn't mean you can't object ;-) : OK :-) Not a serious objection, tho - just 2c worth. [Hey - finally I have something to say here :-] Mark. From gstein@lyra.org Sun Mar 28 09:29:29 1999 From: gstein@lyra.org (Greg Stein) Date: Sun, 28 Mar 1999 01:29:29 -0800 Subject: [DB-SIG] Remaining issues with DB API 1.1 References: <36FCAA4E.1A600D6A@lemburg.com> Message-ID: <36FDF679.481618D6@lyra.org> M.-A. Lemburg wrote: > > Andy Dustman wrote: > > > > On Fri, 26 Mar 1999, M.-A. Lemburg wrote: > > > > > · Require a constructor to take Unix ticks as input for timestamp > > > values ? What about time (without date part) and date (without > > > time part) values ? > > > > > > Well, you know my POV: ticks are a bad thing for serious DB interfacing > > > and there are ways to convert them to the broken down values needed for > > > the constructors. > > > > OTOH, ticks are pretty ubiquitous. Supplying a constructor that uses them > > isn't a big deal; mxDateTime has one, right? Certainly it should not be > > the only constructor, but available as an alternative. > > Sure, mxDateTime has constructors that take ticks as argument and > there is also a generic solution: > > def DateFromTicks(ticks): > > return apply(Date,time.localtime(ticks)[:3]) > > def TimeFromTicks(ticks): > > return apply(Time,time.localtime(ticks)[3:6]) > > def TimestampFromTicks(ticks): > > return apply(Timestamp,time.localtime(ticks)[:6]) > > Still, I think we should not put any emphasis on using them for > database interfacing by requiring the above constructors in the > spec. A module implementor is always free to provide the constructors > and a user can supply them herself in case she wants to use ticks. Marc: two of the three people that have been participating in this discussion have said that having a "ticks" constructor is important. I think it is fair to say that you've been out-voted, and the constructor should be added. Without it, everybody is going to need to replicate the functions you list above. Also, what if a module implementor *doesn't* have a fancy date object? They may want to use ticks. It has worked for Python programmers for eight years, and C programmers for almost 30. > ... > That last part is bogus, IMHO. Also, the spec does not state how > many rows to fetch if size is completely omitted. How about specifying > size as optional argument that defaults to cursor.arraysize ?! ... yes, call it an original spec error. The change you propose is fine. Cheers, -g -- Greg Stein, http://www.lyra.org/ From gstein@lyra.org Sun Mar 28 10:26:34 1999 From: gstein@lyra.org (Greg Stein) Date: Sun, 28 Mar 1999 02:26:34 -0800 Subject: [DB-SIG] Remaining issues with DB API 1.1 References: <36FBA6D4.A6511A1@lemburg.com> Message-ID: <36FE03DA.1646520@lyra.org> M.-A. Lemburg wrote: > > Hi everybody, > > According to the proposed schedule for the DB API, today (1999-03-26) > is the Final Call deadline. We made some very good progress so far and I > would like to thank everybody who joined in. I believe we're getting close. Let's not publish too early -- dates are great to get things moving and we're seeing that, but we don't have any obligation to stick to it as long as we continue to see forward progress and good discussion. I'd say let's simply wrap up the effort this week. We certainly aren't seeing a lot of new people/issues jumping in lately. Can we state that discussion will continue this week, producing a final edit on Friday, April 2nd? Publish on the 5th? >... > · Should we add additional constructor signatures to the API > spec ? E.g. for monetary values (not defined in standard SQL > btw). > > I suggest postponing this decision to the next spec release. And I have suggested that we remove the DATETIME and TIMESTAMP constructors. I still do not understand why we have the difference. There are only a few general types of data: STRING, BLOB, DATE, NUMBER, and ROWID. DATETIME and TIMESTAMP are simply specializations of a generic DATE type. They should be handled using the "==" thing for types that you introduced. Module implementors can extend the set of constructors if they'd like, but the baseline spec should be narrow and provide a constructor that takes the standard six parameters: year, month, day, hour, minute, and second. > · Are setinputsize, setoutputsize optional in the sense that the > APIs itself may be left out by the module implementor or rather > that they must be provided but may be implemented with void > functionality. > > I guess they should always be available. Agreed. [ and it is in the spec like this ] > · Naming of some of the values for the .paramstyle attribute: > Andy Dustman proposed to change "percent" and "xpercent" > to "format" and "dictformat". I prefer "pyformat" instead of > "dictformat"... format and pyformat seem fine. [ and it is in the spec like this ] The part about multiple paramstyles should not be included. A database should take just one style and be done with it. Higher levels can perform the appropriate mappings. > · Should we extend the notion of "sequences" in .execute() > to "objects providing __len__ and __getitem__ methods" ? > See my reply to Andy Dustman for a detailed explanation > of the benefits. This seems fine, but I would leave the description of execute() and executemany() as taking a sequence or a dictionary, with a reference to a footnote. In the footnote, state that an implementation should limit its use of the sequence/mapping interface to getitem and length functionality. > · Add a section pointing out the changes from 1.0 to 1.1 and ways > to port existing scripts to the updated API spec. > > Could someone help with this one ?! Sure. We should probably rename the spec to 2.0 because of the big changes. (1) it is incompatible in a number of ways, and (2) we flat-out dropped "dbi" from the spec. ----- I have some additional comments on the spec from a final read-thru: * the connect() method appears to state "definitive" parameters. It should NOT do that, as I described before. For example, it seems to imply that a parameter named "dsn" is required. That is not the case for most databases. While your keyword code for C modules is fine, why can't we simply state that the connect() method takes an appropriate number of parameters (possibly keyword-based) which allows a connection to be made? I don't believe we need to legislate keyword params or anything like that. We don't anywhere else, so let's not do it here. For example: a DBM-style database is going to take a single parameter: a filename. It doesn't need a keyword, and it doesn't match the ones that you list in the doc. * if you want to provide an example, then place it into a footnote/appendix and reference it from the connect() method. * the 'qmark' paramstyle should not be assumed. All 2.0 DBAPI modules must specify the paramstyle attribute (along with apilevel and threadsafety). Further, the 'qmark' style is not the best style. Named or positional are best, because they allow you to use the same parameter multiple times without requiring the user to respecify it over and over. For example: "select column_a1, column_b1 from a, b where a.key = :1 and b.key = :1". If anything, we should recommend not using the qmark style if a module implementor has a choice. * the comment in the paramstyle text about 'named' and 'xpercent' no longer applies with the recent change to execute(). * a note needs to be added to execute() and executemany() to state that parameters should not require any preprocessing before being passed. To the client programmer, the interface should appear as if a value is going to be "bound" to the input, rather than substituted into the query string. * the apilevel should move up to the top, along with a reference to the 1.0 spec. "if this constant is not specified, then a 1.0 DBAPI should be assumed. see http://... for more information." (we'll leave the old spec at a URL that includes its version number; same for this spec; the old DatabaseAPI.html document can be a page listing each version with a provided URL) * I do not understand the difference between OperationalError and ProgrammingError. The examples seem to divide the two exceptions arbitrarily. What is the "rule" for deciding which is which? * I'm not sure why we have the part about named cursors in the spec, since it specifically states they are not part of the spec. Could we move that to an appendix that lists ways to extend the functionality, but which have not been fully specified (yet) ? * I still do not understand the presence of the rowcount attribute. *Especially* since the execute() method now says "return value undefined." That's bogus. Let's nuke the rowcount attribute and return *that* value from execute. No reason to do an execute and then go somewhere else for this information. * for the execute() return value (merged from the rowcount thing), we should discriminate between "I know the number and it is X" and "I have no idea". The latter should be None. Note that -1 no longer applies since the value is only available as part of an execute, rather than a random-access. I agree with your earlier comments about needing to remove the bit about detecting DML vs DDL vs DQL. I like your current text, with the additions of "0 or None for DDL statements" and "None for cases where the count cannot be determined". * the comment in fetchone() about the underlying cursor should be removed. The Python programmer can never access that thing, so the comment is irrelevant. The comment about using array fetches and whatnot still apply -- in particular, the possible use of the arraysize attribute. * in all methods that state "an exception is raised..." we should state *which* exception. We have a complete set defined at the beginning, so this makes sense to do. I would recommend InterfaceError (it is based on the interface we provide, rather than the underlying database). Hrm. Maybe not. I can see a case where the DB module will simply attempt to call the underlying DB function and generate a DB-related exception. Thoughts anyone? * on nextset(), it might be nice to define a return value that allows the user to discriminate these types of results: no more sets (None right now); another set with N rows; another set with unknown number of rows. The only thing really available for the last one is a negative number (e.g. -1), but that seems a bit hokey. Any ideas? * on setinputsizes() and setoutputsizes(), the text kind of implies that an implementation may omit it. Maybe it could read, "Implementors are free to have this method do nothing, and users are free to not use it." That's all :-) Cheers, -g -- Greg Stein, http://www.lyra.org/ From mal@lemburg.com Sun Mar 28 12:24:55 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Sun, 28 Mar 1999 14:24:55 +0200 Subject: [DB-SIG] Remaining issues with DB API 1.1 References: <006301be78bd$fd203110$0801a8c0@bobcat> Message-ID: <36FE1F97.74CFF0E9@lemburg.com> Mark Hammond wrote: > > > [changing the spec for argument of fetchmany()] > > I think the spec is fine - maybe just needs to be changed to "if > it is None or not specified ...". > > Then again, explicitely stating the use of "None" makes it a PITA > for implementations in C - it pretty much means you can't use > PyArg_ParseTuple(args, "|i", &size); > > So to remove some burden on the implementors, it could be stated > simply as "If it is not specified, then the cursor's arraysize > determines the number of rows to be fetched.". This > leaves implementors using C to use PyArg_ParseTuple, but > implementors in Python to use "def xx(size = None):" - they are > keeping to the spec, but users are not free to make an assumption > about what the default is - they know the behaviour, not the > implementation. The updated spec now says: """ fetchmany([size=cursor.arraysize]) Fetch the next set of rows of a query result, returning a sequence of sequences (e.g. a list of tuples). An empty sequence is returned when no more rows are available. The number of rows to fetch is specified by the parameter. If it is not given, the cursor's arraysize determines the number of rows to be fetched. """ > No need to specify in the spec exactly _what_ the default should > be - it seems sufficient to describe the behaviour - ie, "if not > specified", rather than "the default value is xxx" Hmm, I don't quite follow you here. Why shouldn't the default be defined ? [After all, the 1.0 spec also defined the "default" to be cursor.arraysize.] If we were not to define the default value, then the definition of cursor.arraysize would be obsolete w/r to fetchmany(): """ arraysize This read/write attribute specifies the number of rows to fetch at a time with fetchmany(). It defaults to 1 meaning to fetch a single row at a time. This value may also be used in the implementation of executemany(). Implementations must observe it with respect to the fetchmany() method, but are free to interact with the database a single row at a time. """ Should we drop the reference to fetchmany() in the above definition ? > [Hey - finally I have something to say here :-] Everybody should feel free to drop in their 2 *0.01$ :-) ... after all, the spec update will only become popular if people see some use in it. -- Marc-Andre Lemburg Y2000: 278 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : --------------------------------------------------------- From mal@lemburg.com Sun Mar 28 12:25:08 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Sun, 28 Mar 1999 14:25:08 +0200 Subject: [DB-SIG] Remaining issues with DB API 1.1 References: <36FCAA4E.1A600D6A@lemburg.com> <36FDF679.481618D6@lyra.org> Message-ID: <36FE1FA4.42E1B46C@lemburg.com> Greg Stein wrote: > > M.-A. Lemburg wrote: > > > > Andy Dustman wrote: > > > > > > On Fri, 26 Mar 1999, M.-A. Lemburg wrote: > > > > > > > · Require a constructor to take Unix ticks as input for timestamp > > > > values ? What about time (without date part) and date (without > > > > time part) values ? > > > > > > > > Well, you know my POV: ticks are a bad thing for serious DB interfacing > > > > and there are ways to convert them to the broken down values needed for > > > > the constructors. > > > > > > OTOH, ticks are pretty ubiquitous. Supplying a constructor that uses them > > > isn't a big deal; mxDateTime has one, right? Certainly it should not be > > > the only constructor, but available as an alternative. > > > > Sure, mxDateTime has constructors that take ticks as argument and > > there is also a generic solution: > > > > def DateFromTicks(ticks): > > > > return apply(Date,time.localtime(ticks)[:3]) > > > > def TimeFromTicks(ticks): > > > > return apply(Time,time.localtime(ticks)[3:6]) > > > > def TimestampFromTicks(ticks): > > > > return apply(Timestamp,time.localtime(ticks)[:6]) > > > > Still, I think we should not put any emphasis on using them for > > database interfacing by requiring the above constructors in the > > spec. A module implementor is always free to provide the constructors > > and a user can supply them herself in case she wants to use ticks. > > Marc: two of the three people that have been participating in this > discussion have said that having a "ticks" constructor is important. I > think it is fair to say that you've been out-voted, and the constructor > should be added. Without it, everybody is going to need to replicate the > functions you list above. Ok, they're in :-/ > Also, what if a module implementor *doesn't* have a fancy date object? > They may want to use ticks. It has worked for Python programmers for > eight years, and C programmers for almost 30. I wonder how they managed to get those dates of birth into the databases (and even more important: back out of them ;-). Never mind; I'll add the above constructors to the spec and include the above source code as implementation hint. > > That last part is bogus, IMHO. Also, the spec does not state how > > many rows to fetch if size is completely omitted. How about specifying > > size as optional argument that defaults to cursor.arraysize ?! ... > > yes, call it an original spec error. The change you propose is fine. Ok. -- Marc-Andre Lemburg Y2000: 278 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : --------------------------------------------------------- From mal@lemburg.com Sun Mar 28 13:37:54 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Sun, 28 Mar 1999 15:37:54 +0200 Subject: [DB-SIG] Remaining issues with DB API 1.1 References: <36FCAA4E.1A600D6A@lemburg.com> <36FDF679.481618D6@lyra.org> Message-ID: <36FE30B2.1BB63EC1@lemburg.com> Greg: Are there going to be generic conversion C APIs for buffer types in 1.5.2 ? And is there some resource I could point to for more documentation on the types and their usage ? I'm asking this because I the current CVS version of Python 1.5.2 doesn't have to much docs about the types and seems to be missing a way to extract information out of the buffer types other than going through PyArg_ParseTuple(). Another issue: How should we define the mapping of Unix ticks to date/time/timestamp values ? ... or should we define them at all ? Since ticks always refer to timestamp like values, converting them to date and time values will have to cause some kind of factoring. In the examples I gave, local time broken down values are used as basis. This could, of course, also be done using GM time. I've update the spec to 1.1a12. The next release of mxDateTime will include the new constructors for ticks based values. Cheers, -- Marc-Andre Lemburg Y2000: 278 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : --------------------------------------------------------- From gstein@lyra.org Sun Mar 28 14:11:07 1999 From: gstein@lyra.org (Greg Stein) Date: Sun, 28 Mar 1999 06:11:07 -0800 Subject: [DB-SIG] Remaining issues with DB API 1.1 References: <36FCAA4E.1A600D6A@lemburg.com> <36FDF679.481618D6@lyra.org> <36FE30B2.1BB63EC1@lemburg.com> Message-ID: <36FE387B.7CA8C990@lyra.org> M.-A. Lemburg wrote: > > Greg: Are there going to be generic conversion C APIs for buffer types > in 1.5.2 ? And is there some resource I could point to for more > documentation on the types and their usage ? Some alternatives: 1) str = PyObject_Str(buf); 2) use buf->ob_type->tp_as_buffer-> ... The former creates a new string object based on the buffer contents. The latter provides direct access to the contents without allocating additional memory. The tp_as_buffer stuff is not documented (beyond "the code is the spec/doc"). The buffer() builtin function is documented. Your meta-comment about the lack of "Abstract Functions" is well-taken, though. There are abstract functions for most of the type slots in an object. However: this was done because if you're operating against a class instance, then it would need to invoke a method rather than use a type slot. There are no equivalent methods for the buffer interface, so the need for abstract functions is arguable. The functions would merely save you a bit of typing, rather than encapsulate any real functionality. > I'm asking this because I the current CVS version of Python 1.5.2 > doesn't have to much docs about the types and seems to be missing > a way to extract information out of the buffer types other than > going through PyArg_ParseTuple(). That is a slow way to do it. People should use the tp_as_buffer slots' functions. For example, you could bind right against the address returned using the getreadbuffer type slot function. Interesting note: memory-mapped files export the buffer interface. You could mmap a file and then start shoving the bytes into a database by passing the file as a column value. hehe... > Another issue: > > How should we define the mapping of Unix ticks to date/time/timestamp > values ? ... or should we define them at all ? IMO, that is module-dependent. A ticks value specifies a distinct point in time. If a database is interacting with a column type which has less precision, then some kind of truncation will occur in a database-dependent fashion. Part of my pushback against the three varieties is that they attempt to provide coverage, when that really isn't the case. For example, MySQL and Informix allow you to specify DATE columns with near-arbitrary precision. Want just a year? No problem. How about a year and a month? Sure. Just an hour and minute? Yup. Providing a generic "type" of column called a "DATE" can encompass all these varieties. Subtypes (per the scheme you posted which uses __cmp__) can refine the generic type down to whatever precision or subtype is required. > Since ticks always refer to timestamp like values, converting them > to date and time values will have to cause some kind of factoring. > In the examples I gave, local time broken down values are used > as basis. This could, of course, also be done using GM time. Historically, the ticks value was broken into specific parts using localtime() or its C equivalent. (and mktime() on the other side) If a module implementor is using a sophisticated datatype like mxDateTime, then they don't need to worry... it will Do The Right Thing. For those who don't use it, they'll have a small paragraph of C code to write. > I've update the spec to 1.1a12. Cool. I should get to updating the annotated version. It is woefully behind. > The next release of mxDateTime > will include the new constructors for ticks based values. I thought it already had that. I looked thru the spec the other day. Or did it only have *output*, rather than constructors? Cheers, -g -- Greg Stein, http://www.lyra.org/ From gstein@lyra.org Sun Mar 28 14:19:30 1999 From: gstein@lyra.org (Greg Stein) Date: Sun, 28 Mar 1999 06:19:30 -0800 Subject: [DB-SIG] Remaining issues with DB API 1.1 References: <006301be78bd$fd203110$0801a8c0@bobcat> <36FE1F97.74CFF0E9@lemburg.com> Message-ID: <36FE3A72.4ED4880A@lyra.org> M.-A. Lemburg wrote: >... > The updated spec now says: > """ > fetchmany([size=cursor.arraysize]) > Fetch the next set of rows of a query result, returning a > sequence of sequences (e.g. a list of tuples). An empty > sequence is returned when no more rows are available. The > number of rows to fetch is specified by the parameter. If it is not > given, the cursor's arraysize determines the number of rows to > be fetched. > """ This is good, although we should probably elaborate a bit more: it is possible that *fewer* than the requested number of rows (via the parameter or cursor.arraysize) will be returned. Specifically, it will almost always happen at the "end" of a query. IMO, it should also be legal mid-query, but we don't need to explicitly state that. (although we should be clear that returning fewer does *not* mean the end-of-query... the only determining factor for that is an empty sequence) > > No need to specify in the spec exactly _what_ the default should > > be - it seems sufficient to describe the behaviour - ie, "if not > > specified", rather than "the default value is xxx" > > Hmm, I don't quite follow you here. Why shouldn't the default > be defined ? [After all, the 1.0 spec also defined the "default" > to be cursor.arraysize.] It should be defined. He's smoking something :-) Your current text looks good. > If we were not to define the default value, then the definition > of cursor.arraysize would be obsolete w/r to fetchmany(): > """ > arraysize > ... > """ > > Should we drop the reference to fetchmany() in the above definition ? Nope. Cheers, -g -- Greg Stein, http://www.lyra.org/ From mal@lemburg.com Sun Mar 28 14:59:07 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Sun, 28 Mar 1999 16:59:07 +0200 Subject: [DB-SIG] Remaining issues with DB API 2.0 References: <36FBA6D4.A6511A1@lemburg.com> <36FE03DA.1646520@lyra.org> Message-ID: <36FE43BB.6A1F45A2@lemburg.com> Note: The URL for the API spec has changed: http://starship.skyport.net/~lemburg/DatabaseAPI-2.0.html Greg Stein wrote: > > M.-A. Lemburg wrote: > > > > Hi everybody, > > > > According to the proposed schedule for the DB API, today (1999-03-26) > > is the Final Call deadline. We made some very good progress so far and I > > would like to thank everybody who joined in. > > I believe we're getting close. Let's not publish too early -- dates are > great to get things moving and we're seeing that, but we don't have any > obligation to stick to it as long as we continue to see forward progress > and good discussion. I'd say let's simply wrap up the effort this week. > We certainly aren't seeing a lot of new people/issues jumping in lately. > > Can we state that discussion will continue this week, producing a final > edit on Friday, April 2nd? Publish on the 5th? Ok. > >... > > · Should we add additional constructor signatures to the API > > spec ? E.g. for monetary values (not defined in standard SQL > > btw). > > > > I suggest postponing this decision to the next spec release. > > And I have suggested that we remove the DATETIME and TIMESTAMP > constructors. I still do not understand why we have the difference. > There are only a few general types of data: STRING, BLOB, DATE, NUMBER, > and ROWID. > > DATETIME and TIMESTAMP are simply specializations of a generic DATE > type. They should be handled using the "==" thing for types that you > introduced. Module implementors can extend the set of constructors if > they'd like, but the baseline spec should be narrow and provide a > constructor that takes the standard six parameters: year, month, day, > hour, minute, and second. The differentiation between date, time and timestamp is needed simply because they point to three different informations. A single date constructor + corresponding type object is not enough for a module to decide whether to take only the date, the time, or both parts to format some data to pass to the database. Take e.g. a module that uses literal SQL statement formatting for interfacing. Such a module would not be able to tell whether to format a date as time only value, timestamp or date only value. If it were to always pass in timestamp values, the database would probably raise an error for DATE and TIME columns. So dropping the two constructors renders that kind of interface useless. Becides, explicitly using the constructors in you database interface script will surely make them more readable :-) As for the type objects: we could group date/time values collectively under DATETIME. Would that be in your interest ? > > · Naming of some of the values for the .paramstyle attribute: > > Andy Dustman proposed to change "percent" and "xpercent" > > to "format" and "dictformat". I prefer "pyformat" instead of > > "dictformat"... > > format and pyformat seem fine. > [ and it is in the spec like this ] > > The part about multiple paramstyles should not be included. A database > should take just one style and be done with it. Higher levels can > perform the appropriate mappings. Andy ? > > · Should we extend the notion of "sequences" in .execute() > > to "objects providing __len__ and __getitem__ methods" ? > > See my reply to Andy Dustman for a detailed explanation > > of the benefits. > > This seems fine, but I would leave the description of execute() and > executemany() as taking a sequence or a dictionary, with a reference to > a footnote. In the footnote, state that an implementation should limit > its use of the sequence/mapping interface to getitem and length > functionality. That's a document formatting question. I'll look into it. > > · Add a section pointing out the changes from 1.0 to 1.1 and ways > > to port existing scripts to the updated API spec. > > > > Could someone help with this one ?! > > Sure. We should probably rename the spec to 2.0 because of the big > changes. (1) it is incompatible in a number of ways, and (2) we flat-out > dropped "dbi" from the spec. I guess you're right. So the next spec update will be versioned 2.0a13. > ----- > > I have some additional comments on the spec from a final read-thru: > > * the connect() method appears to state "definitive" parameters. It > should NOT do that, as I described before. For example, it seems to > imply that a parameter named "dsn" is required. That is not the case for > most databases. While your keyword code for C modules is fine, why can't > we simply state that the connect() method takes an appropriate number of > parameters (possibly keyword-based) which allows a connection to be > made? I don't believe we need to legislate keyword params or anything > like that. We don't anywhere else, so let's not do it here. For example: > a DBM-style database is going to take a single parameter: a filename. It > doesn't need a keyword, and it doesn't match the ones that you list in > the doc. Hmm, I guess you've got a point there. I'll make the parameter list a guideline then (this part of the spec isn't going to be portable anyway...). Still, I'll leave the keyword stuff in -- people should start using keyword based functions in C more often... > * if you want to provide an example, then place it into a > footnote/appendix and reference it from the connect() method. Dito. > * the 'qmark' paramstyle should not be assumed. All 2.0 DBAPI modules > must specify the paramstyle attribute (along with apilevel and > threadsafety). Further, the 'qmark' style is not the best style. Named > or positional are best, because they allow you to use the same parameter > multiple times without requiring the user to respecify it over and over. > For example: "select column_a1, column_b1 from a, b where a.key = :1 and > b.key = :1". If anything, we should recommend not using the qmark style > if a module implementor has a choice. True. I'll take those comments out. > * the comment in the paramstyle text about 'named' and 'xpercent' no > longer applies with the recent change to execute(). Right. > * a note needs to be added to execute() and executemany() to state that > parameters should not require any preprocessing before being passed. To > the client programmer, the interface should appear as if a value is > going to be "bound" to the input, rather than substituted into the query > string. Not sure, why you want this. The spec already says: "Parameters may be provided as sequence or mapping and will be bound to variables in the operation." > * the apilevel should move up to the top, along with a reference to the > 1.0 spec. "if this constant is not specified, then a 1.0 DBAPI should be > assumed. see http://... for more information." (we'll leave the old > spec at a URL that includes its version number; same for this spec; the > old DatabaseAPI.html document can be a page listing each version with a > provided URL) Ok. > * I do not understand the difference between OperationalError and > ProgrammingError. The examples seem to divide the two exceptions > arbitrarily. What is the "rule" for deciding which is which? OperationalErrors are all types of errors that are related to things that are not necessarily under the control of the programmer, e.g. lost connections, log full, transaction could not be performed due to lack of memory or lack of functionality, etc. ProgrammingError refer to things that are under the programmers control, e.g. he uses a table that does not exist, he specifies the wrong number of parameters for a statement, uses wrong SQL syntax, etc. Hmm, maybe OperationalError wasn't the right choice for the dynamic .rollback() error after all... There is some overlap and in fact some DBs raise ProgrammingErrors for certain things while others use OperationalError. Maybe we should make one a subclass of the other... ? > * I'm not sure why we have the part about named cursors in the spec, > since it specifically states they are not part of the spec. Could we > move that to an appendix that lists ways to extend the functionality, > but which have not been fully specified (yet) ? I turned it into a footnote. > * I still do not understand the presence of the rowcount attribute. > *Especially* since the execute() method now says "return value > undefined." That's bogus. Let's nuke the rowcount attribute and return > *that* value from execute. No reason to do an execute and then go > somewhere else for this information. The "return value undefined" was done to allow the .execute() method return the old style return value (which is not easily implementable for various reasons I already pointed out in previous mails). The module implementor is free to return whatever he likes. The rowcount definition is somewhat different from the 1.0 return value spec of .execute(). That's why it's a new attribute. Again, module implementors could go and simply have .execute() return whatever the value of .rowcount would be if they like... > * for the execute() return value (merged from the rowcount thing), we > should discriminate between "I know the number and it is X" and "I have > no idea". The latter should be None. Note that -1 no longer applies > since the value is only available as part of an execute, rather than a > random-access. I agree with your earlier comments about needing to > remove the bit about detecting DML vs DDL vs DQL. I like your current > text, with the additions of "0 or None for DDL statements" and "None for > cases where the count cannot be determined". The problem with None is that it compares false which would be ok for DDL (0 or None both are false), but fails to indicate "don't know" for DQL where 0 could be a possible known row count. As for the random-access vs. return value: some DBs could be able to calculate the row count only if at least one fetch was done. So the information might not be available right after the execute. How about defining the return value .execute() as being the current .rowcount attribute's value as you propose and provide the .rowcount as additional, possibly optional feature ? > * the comment in fetchone() about the underlying cursor should be > removed. The Python programmer can never access that thing, so the > comment is irrelevant. The comment about using array fetches and whatnot > still apply -- in particular, the possible use of the arraysize > attribute. The comment is simply a warning to those who might expect the fetchone() method to move the cursor by one row. This may be true for some interfaces (e.g. mxODBC) while being false for others. > * in all methods that state "an exception is raised..." we should state > *which* exception. We have a complete set defined at the beginning, so > this makes sense to do. I would recommend InterfaceError (it is based on > the interface we provide, rather than the underlying database). Hrm. > Maybe not. I can see a case where the DB module will simply attempt to > call the underlying DB function and generate a DB-related exception. > Thoughts anyone? This is hard to do, since the types of errors may very well be the whole range of possible Python exceptions + the ones defined in the DB API, e.g. a conversion routine might raise an OverflowError in case of numeric overflow while raising an InterfaceError in case a certain internal assertion failed. > * on nextset(), it might be nice to define a return value that allows > the user to discriminate these types of results: no more sets (None > right now); another set with N rows; another set with unknown number of > rows. The only thing really available for the last one is a negative > number (e.g. -1), but that seems a bit hokey. Any ideas? I've got no experience with multiple result sets, but do that there was a point to define the return value in the fuzzy way it is written in the spec. > * on setinputsizes() and setoutputsizes(), the text kind of implies that > an implementation may omit it. Maybe it could read, "Implementors are > free to have this method do nothing, and users are free to not use it." Ok. Cheers, -- Marc-Andre Lemburg Y2000: 278 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : --------------------------------------------------------- From mal@lemburg.com Sun Mar 28 15:16:49 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Sun, 28 Mar 1999 17:16:49 +0200 Subject: [DB-SIG] Remaining issues with DB API 1.1 References: <36FCAA4E.1A600D6A@lemburg.com> <36FDF679.481618D6@lyra.org> <36FE30B2.1BB63EC1@lemburg.com> <36FE387B.7CA8C990@lyra.org> Message-ID: <36FE47E1.356239F9@lemburg.com> Greg Stein wrote: > > M.-A. Lemburg wrote: > > > > Greg: Are there going to be generic conversion C APIs for buffer types > > in 1.5.2 ? And is there some resource I could point to for more > > documentation on the types and their usage ? > > Some alternatives: > > 1) str = PyObject_Str(buf); > 2) use buf->ob_type->tp_as_buffer-> ... Na, we (the C programmers) don't want to fiddle with type slots... as you said: there should be an abstract API for either calling the type slot (if available) and reverting to the string method otherwise. This should return a char* read-only buffer, something like PyBuffer_AsStringAndSize(PyObject *buffer, char **mem, int **len). > The former creates a new string object based on the buffer contents. The > latter provides direct access to the contents without allocating > additional memory. The tp_as_buffer stuff is not documented (beyond "the > code is the spec/doc"). > > The buffer() builtin function is documented. > > Your meta-comment about the lack of "Abstract Functions" is well-taken, > though. There are abstract functions for most of the type slots in an > object. However: this was done because if you're operating against a > class instance, then it would need to invoke a method rather than use a > type slot. There are no equivalent methods for the buffer interface, so > the need for abstract functions is arguable. The functions would merely > save you a bit of typing, rather than encapsulate any real > functionality. Ah, it would make things more "abstract" with all the nice features that go with it. That's what I'm interested in. There might some useful update to Python in the future and if I were to hard code the current setting into my extension then I could get into trouble a later point or simply be missing that new nifty feature in 2.1 ;-) > > I'm asking this because I the current CVS version of Python 1.5.2 > > doesn't have to much docs about the types and seems to be missing > > a way to extract information out of the buffer types other than > > going through PyArg_ParseTuple(). > > That is a slow way to do it. People should use the tp_as_buffer slots' > functions. For example, you could bind right against the address > returned using the getreadbuffer type slot function. > > Interesting note: memory-mapped files export the buffer interface. You > could mmap a file and then start shoving the bytes into a database by > passing the file as a column value. hehe... Now that's cool ;-) > > Another issue: > > > > How should we define the mapping of Unix ticks to date/time/timestamp > > values ? ... or should we define them at all ? > > IMO, that is module-dependent. A ticks value specifies a distinct point > in time. If a database is interacting with a column type which has less > precision, then some kind of truncation will occur in a > database-dependent fashion. > > Part of my pushback against the three varieties is that they attempt to > provide coverage, when that really isn't the case. For example, MySQL > and Informix allow you to specify DATE columns with near-arbitrary > precision. Want just a year? No problem. How about a year and a month? > Sure. Just an hour and minute? Yup. > > Providing a generic "type" of column called a "DATE" can encompass all > these varieties. Subtypes (per the scheme you posted which uses __cmp__) > can refine the generic type down to whatever precision or subtype is > required. This works fine on the output side (for which I proposed the DATETIME type object wrapping DATE, TIME and TIMESTAMP), but on the input side the constructors are simply needed to distinguish between the different aspects. As always, the implementor can go and map all three constructors to just one in Python using e.g. a string in some database specific format... really no big deal. So should we nuke the DATE, TIME and TIMESTAMP type objects from the spec and have a generic DATETIME type object instead ?! > > Since ticks always refer to timestamp like values, converting them > > to date and time values will have to cause some kind of factoring. > > In the examples I gave, local time broken down values are used > > as basis. This could, of course, also be done using GM time. > > Historically, the ticks value was broken into specific parts using > localtime() or its C equivalent. (and mktime() on the other side) > > If a module implementor is using a sophisticated datatype like > mxDateTime, then they don't need to worry... it will Do The Right Thing. > For those who don't use it, they'll have a small paragraph of C code to > write. So the local time assumption is correct ?! > > I've update the spec to 1.1a12. > > Cool. I should get to updating the annotated version. It is woefully > behind. > > > The next release of mxDateTime > > will include the new constructors for ticks based values. > > I thought it already had that. I looked thru the spec the other day. Or > did it only have *output*, rather than constructors? It did already have APIs for converting ticks to instances, it's just that the naming was different. I've added a few new Python constructors to make mxDateTime provide the right names from the start. -- Marc-Andre Lemburg Y2000: 278 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : --------------------------------------------------------- From mal@lemburg.com Sun Mar 28 15:19:06 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Sun, 28 Mar 1999 17:19:06 +0200 Subject: [DB-SIG] Remaining issues with DB API 1.1 References: <006301be78bd$fd203110$0801a8c0@bobcat> <36FE1F97.74CFF0E9@lemburg.com> <36FE3A72.4ED4880A@lyra.org> Message-ID: <36FE486A.48E79391@lemburg.com> Greg Stein wrote: > > M.-A. Lemburg wrote: > >... > > The updated spec now says: > > """ > > fetchmany([size=cursor.arraysize]) > > Fetch the next set of rows of a query result, returning a > > sequence of sequences (e.g. a list of tuples). An empty > > sequence is returned when no more rows are available. The > > number of rows to fetch is specified by the parameter. If it is not > > given, the cursor's arraysize determines the number of rows to > > be fetched. > > """ > > This is good, although we should probably elaborate a bit more: it is > possible that *fewer* than the requested number of rows (via the > parameter or cursor.arraysize) will be returned. Specifically, it will > almost always happen at the "end" of a query. IMO, it should also be > legal mid-query, but we don't need to explicitly state that. (although > we should be clear that returning fewer does *not* mean the > end-of-query... the only determining factor for that is an empty > sequence) Good point. I'll add a comment. > > > No need to specify in the spec exactly _what_ the default should > > > be - it seems sufficient to describe the behaviour - ie, "if not > > > specified", rather than "the default value is xxx" > > > > Hmm, I don't quite follow you here. Why shouldn't the default > > be defined ? [After all, the 1.0 spec also defined the "default" > > to be cursor.arraysize.] > > It should be defined. He's smoking something :-) Your current text > looks good. > > > If we were not to define the default value, then the definition > > of cursor.arraysize would be obsolete w/r to fetchmany(): > > """ > > arraysize > > ... > > """ > > > > Should we drop the reference to fetchmany() in the above definition ? > > Nope. Good ;-) -- Marc-Andre Lemburg Y2000: 278 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : --------------------------------------------------------- From mal@lemburg.com Sun Mar 28 19:14:19 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Sun, 28 Mar 1999 21:14:19 +0200 Subject: [DB-SIG] Remaining issues with DB API 2.0 References: <36FBA6D4.A6511A1@lemburg.com> <36FE03DA.1646520@lyra.org> <36FE43BB.6A1F45A2@lemburg.com> Message-ID: <36FE7F8B.67E06C09@lemburg.com> Greg: > > * in all methods that state "an exception is raised..." we should state > > *which* exception. We have a complete set defined at the beginning, so > > this makes sense to do. I would recommend InterfaceError (it is based on > > the interface we provide, rather than the underlying database). Hrm. > > Maybe not. I can see a case where the DB module will simply attempt to > > call the underlying DB function and generate a DB-related exception. > > Thoughts anyone? > > This is hard to do, since the types of errors may very well be > the whole range of possible Python exceptions + the ones defined > in the DB API, e.g. a conversion routine might raise an OverflowError > in case of numeric overflow while raising an InterfaceError in > case a certain internal assertion failed. Looking at the places you probably meant, the overall generality is probably not needed. We could probably restrict the exception range to subclasses of Error. Would that be ok ? Another quirk I noticed: right now, the .nextset() method only has optional functionality, always returning None in case the database does not support multiple result sets. Maybe we should make the method optional altogether, so that a script writer can test whether multiple sets are supported or not ?! -- Marc-Andre Lemburg Y2000: 278 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : --------------------------------------------------------- From gstein@lyra.org Sun Mar 28 22:45:48 1999 From: gstein@lyra.org (Greg Stein) Date: Sun, 28 Mar 1999 14:45:48 -0800 Subject: [DB-SIG] Remaining issues with DB API 2.0 References: <36FBA6D4.A6511A1@lemburg.com> <36FE03DA.1646520@lyra.org> <36FE43BB.6A1F45A2@lemburg.com> <36FE7F8B.67E06C09@lemburg.com> Message-ID: <36FEB11C.363EB5E9@lyra.org> M.-A. Lemburg wrote: > > Greg: > > > * in all methods that state "an exception is raised..." we should state > > > *which* exception. We have a complete set defined at the beginning, so > > > this makes sense to do. I would recommend InterfaceError (it is based on > > > the interface we provide, rather than the underlying database). Hrm. > > > Maybe not. I can see a case where the DB module will simply attempt to > > > call the underlying DB function and generate a DB-related exception. > > > Thoughts anyone? > > > > This is hard to do, since the types of errors may very well be > > the whole range of possible Python exceptions + the ones defined > > in the DB API, e.g. a conversion routine might raise an OverflowError > > in case of numeric overflow while raising an InterfaceError in > > case a certain internal assertion failed. > > Looking at the places you probably meant, the overall generality > is probably not needed. We could probably restrict the exception > range to subclasses of Error. Would that be ok ? Yes. It would be nice to be more specific, but truthfully... I just realized "who cares?" The theory is that it simply doesn't happen in a correct program, so we don't need to load the spec up with information on how to tell somebody they wrote their program incorrectly. > Another quirk I noticed: right now, the .nextset() method only > has optional functionality, always returning None in case the > database does not support multiple result sets. Maybe we should > make the method optional altogether, so that a script writer > can test whether multiple sets are supported or not ?! Well, this is a bit different from, say, rollback(). If somebody tries to call rollback(), then they better get a bad error because they're screwed if it doesn't really exist. Calling nextset(), setinputsizes(), or setoutputsizes() doesn't really hurt anything. Much like commit() for transaction-less databases. Therefore, I'd say there is much of a need for us to create additional exceptions or feature test requirements for users. Cheers, -g -- Greg Stein, http://www.lyra.org/ From adustman@comstar.net Sun Mar 28 23:36:17 1999 From: adustman@comstar.net (Andy Dustman) Date: Sun, 28 Mar 1999 18:36:17 -0500 (EST) Subject: [DB-SIG] Remaining issues with DB API 2.0 In-Reply-To: <36FE43BB.6A1F45A2@lemburg.com> Message-ID: On Sun, 28 Mar 1999, M.-A. Lemburg wrote: > Note: The URL for the API spec has changed: > > http://starship.skyport.net/~lemburg/DatabaseAPI-2.0.html > > Greg Stein wrote: > > > > · Naming of some of the values for the .paramstyle attribute: > > > Andy Dustman proposed to change "percent" and "xpercent" > > > to "format" and "dictformat". I prefer "pyformat" instead of > > > "dictformat"... > > > > format and pyformat seem fine. > > [ and it is in the spec like this ] > > > > The part about multiple paramstyles should not be included. A database > > should take just one style and be done with it. Higher levels can > > perform the appropriate mappings. > > Andy ? Yeah, sure. There's no database that supports multiple styles at this time, though the new MySQLdb could, since it is using the % string operator anyway. Maybe I'll slip that in as a separate method or something. > Hmm, I guess you've got a point there. I'll make the parameter > list a guideline then (this part of the spec isn't going to be > portable anyway...). Still, I'll leave the keyword stuff in -- > people should start using keyword based functions in C more > often... I make pretty good use of keyword parameters in the underlying _mysql module, i.e. c=_mysql.connect(host, user, passwd, db, port, unix_socket, client_flag) where all those are keyword parameters which mostly default to NULL. Usually I can get away with: c=_mysql.connect(passwd='pwd') The current positional parameters really sorta cramp my style a bit, though I have worked around it. If it were legal for me to define my connect() to only take keyword parameters, I'd be happy, though I need to renamed the password parameter somehow; my design for the C module has been to keep it very close to the C MySQL API. Well, I'll work it out. -- Andy Dustman (ICQ#32922760) You should always say "spam" and "eggs" ComStar Communications Corp. instead of "foo" and "bar" (706) 549-7689 | PGP KeyID=0xC72F3F1D in Python examples. (Mark Lutz) From gstein@lyra.org Mon Mar 29 06:13:10 1999 From: gstein@lyra.org (Greg Stein) Date: Sun, 28 Mar 1999 22:13:10 -0800 Subject: [DB-SIG] Remaining issues with DB API 2.0 References: Message-ID: <36FF19F6.23171F0C@lyra.org> Andy Dustman wrote: > ... > > > The part about multiple paramstyles should not be included. A database > > > should take just one style and be done with it. Higher levels can > > > perform the appropriate mappings. > > > > Andy ? > > Yeah, sure. There's no database that supports multiple styles at this > time, though the new MySQLdb could, since it is using the % string > operator anyway. Maybe I'll slip that in as a separate method or > something. That just doesn't feel right, but hey... it's your module :-) (IMO, the Python way of "one way to do things" keeps things clean and simple) >... > The current positional parameters really sorta cramp my style a bit, > though I have worked around it. If it were legal for me to define my > connect() to only take keyword parameters, I'd be happy, though I need to > renamed the password parameter somehow; my design for the C module has > been to keep it very close to the C MySQL API. Well, I'll work it out. This is what I've been pushing for. You SHOULD be able to define the connect() HOWEVER you want. It changes so dramatically from one database to the next that we should NOT attempt to specify anything about it beyond "connect() takes some parameters". Keyword, positional, what names, what defaults, etc should not be specified. My simple counterpoint is a DBM database that simply takes a filename. And I hope that I don't have to point out that DBM databases *are* in the scope of the DB-SIG efforts and (hopefully) within the design space of DBAPI. Cheers, -g -- Greg Stein, http://www.lyra.org/ From adustman@comstar.net Mon Mar 29 06:34:58 1999 From: adustman@comstar.net (Andy Dustman) Date: Mon, 29 Mar 1999 01:34:58 -0500 (EST) Subject: [DB-SIG] Remaining issues with DB API 2.0 In-Reply-To: <36FF19F6.23171F0C@lyra.org> Message-ID: On Sun, 28 Mar 1999, Greg Stein wrote: > Andy Dustman wrote: > > ... > > > > The part about multiple paramstyles should not be included. A database > > > > should take just one style and be done with it. Higher levels can > > > > perform the appropriate mappings. > > > > > > Andy ? > > > > Yeah, sure. There's no database that supports multiple styles at this > > time, though the new MySQLdb could, since it is using the % string > > operator anyway. Maybe I'll slip that in as a separate method or > > something. > > That just doesn't feel right, but hey... it's your module :-) > > (IMO, the Python way of "one way to do things" keeps things clean and > simple) So long as the API definition allows me to pass execute a dictionary (or other mapping object), I'd rather do it that way. Assuming I even implement it. It's not hard, but I don't have a compelling reason to do it... > This is what I've been pushing for. You SHOULD be able to define the > connect() HOWEVER you want. It changes so dramatically from one database > to the next that we should NOT attempt to specify anything about it > beyond "connect() takes some parameters". Keyword, positional, what > names, what defaults, etc should not be specified. > > My simple counterpoint is a DBM database that simply takes a filename. > And I hope that I don't have to point out that DBM databases *are* in > the scope of the DB-SIG efforts and (hopefully) within the design space > of DBAPI. I would be happy if the parameters to connect were undefined (completely database-dependent) so I could do whatever I want with them. There's too much database diversity to come up with one set of parameters. With everything else, though, it seems like we need to aim for an idealized database interface that emulates stuff that doesn't exist where feasible and raises exceptions when impossible. The examples previously mentioned were cursors, which MySQL doesn't have, but which are easily emulated, and rollback, which must always fail somehow. Even so, I'm not sure how we would wrap dbm, gdbm, or bsddb to be compatible with the API, as they have no query language, no (or one) table, and no columns! Perhaps the wrapper could parse a little SQL, ignore the specified columns, and just pickle whatever parameters it finds. It'd be tough, though. I think I'd stick with the shelf interface... (I actually have a module, SQLDict, that wraps a dictionary-like interface around a DBI-compatible connection object... That direction of abstraction is a little easier to do...) -- Andy Dustman (ICQ#32922760) You should always say "spam" and "eggs" ComStar Communications Corp. instead of "foo" and "bar" (706) 549-7689 | PGP KeyID=0xC72F3F1D in Python examples. (Mark Lutz) From mal@lemburg.com Mon Mar 29 08:46:47 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 29 Mar 1999 10:46:47 +0200 Subject: [DB-SIG] Remaining issues with DB API 2.0 References: <36FBA6D4.A6511A1@lemburg.com> <36FE03DA.1646520@lyra.org> <36FE43BB.6A1F45A2@lemburg.com> <36FE7F8B.67E06C09@lemburg.com> <36FEB11C.363EB5E9@lyra.org> Message-ID: <36FF3DF7.788D370A@lemburg.com> Greg Stein wrote: > > M.-A. Lemburg wrote: > > > > Greg: > > > > * in all methods that state "an exception is raised..." we should state > > > > *which* exception. We have a complete set defined at the beginning, so > > > > this makes sense to do. I would recommend InterfaceError (it is based on > > > > the interface we provide, rather than the underlying database). Hrm. > > > > Maybe not. I can see a case where the DB module will simply attempt to > > > > call the underlying DB function and generate a DB-related exception. > > > > Thoughts anyone? > > > > > > This is hard to do, since the types of errors may very well be > > > the whole range of possible Python exceptions + the ones defined > > > in the DB API, e.g. a conversion routine might raise an OverflowError > > > in case of numeric overflow while raising an InterfaceError in > > > case a certain internal assertion failed. > > > > Looking at the places you probably meant, the overall generality > > is probably not needed. We could probably restrict the exception > > range to subclasses of Error. Would that be ok ? > > Yes. > > It would be nice to be more specific, but truthfully... I just realized > "who cares?" The theory is that it simply doesn't happen in a correct > program, so we don't need to load the spec up with information on how to > tell somebody they wrote their program incorrectly. You've got a point there. I've added the above restriction to Error subclasses anyway; doesn't hurt, I think, and gives the module implementor a warm fuzzy feeling that he chose the right exception. > > Another quirk I noticed: right now, the .nextset() method only > > has optional functionality, always returning None in case the > > database does not support multiple result sets. Maybe we should > > make the method optional altogether, so that a script writer > > can test whether multiple sets are supported or not ?! > > Well, this is a bit different from, say, rollback(). If somebody tries > to call rollback(), then they better get a bad error because they're > screwed if it doesn't really exist. I thought of .callproc() as comparison. The two probably come in pairs anyway since result sets are usually produced by stored procedures and sometimes also by array based SELECTs (if the database supports these). > Calling nextset(), setinputsizes(), or setoutputsizes() doesn't really > hurt anything. Much like commit() for transaction-less databases. That's true, but it would be a nice way to tell wether the database supports these beasts (which seems impossible otherwise). Otherwise we would have to nuke the comment about the exception raising, BTW. > Therefore, I'd say there is much of a need for us to create additional > exceptions or feature test requirements for users. Feature testing can be done with hasattr() as you proposed. More elaborate schemes will most likely turn out to be database dependent or would result in long lists of constants in the spec. As for additional exceptions: it might be a good idea to add a NotSupportedError exception. This could then be used for things like the dynamic .rollback() capability test (which would then raise a NotSupportedError instead of an OperationalError). It should be a subclass of OperationalError. Maybe we should also add a note that module implementors are free to build upon the given exception classes. Since portable scripts would catch the exceptions using the base classes this wouldn't hurt and give the implementors a lot more freedom. -- Marc-Andre Lemburg Y2000: 277 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : --------------------------------------------------------- From mal@lemburg.com Mon Mar 29 08:52:15 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 29 Mar 1999 10:52:15 +0200 Subject: [DB-SIG] Remaining issues with DB API 2.0 References: Message-ID: <36FF3F3F.3824BAEA@lemburg.com> Andy Dustman wrote: > > > > > · Naming of some of the values for the .paramstyle attribute: > > > > Andy Dustman proposed to change "percent" and "xpercent" > > > > to "format" and "dictformat". I prefer "pyformat" instead of > > > > "dictformat"... > > > > > > format and pyformat seem fine. > > > [ and it is in the spec like this ] > > > > > > The part about multiple paramstyles should not be included. A database > > > should take just one style and be done with it. Higher levels can > > > perform the appropriate mappings. > > > > Andy ? > > Yeah, sure. There's no database that supports multiple styles at this > time, though the new MySQLdb could, since it is using the % string > operator anyway. Maybe I'll slip that in as a separate method or > something. Ok. > > Hmm, I guess you've got a point there. I'll make the parameter > > list a guideline then (this part of the spec isn't going to be > > portable anyway...). Still, I'll leave the keyword stuff in -- > > people should start using keyword based functions in C more > > often... > > I make pretty good use of keyword parameters in the underlying _mysql > module, i.e. > > c=_mysql.connect(host, user, passwd, db, port, unix_socket, client_flag) > > where all those are keyword parameters which mostly default to NULL. > Usually I can get away with: > > c=_mysql.connect(passwd='pwd') > > The current positional parameters really sorta cramp my style a bit, > though I have worked around it. If it were legal for me to define my > connect() to only take keyword parameters, I'd be happy, though I need to > renamed the password parameter somehow; my design for the C module has > been to keep it very close to the C MySQL API. Well, I'll work it out. It should be legal. The spec now says: """ connect(parameters...) Constructor for creating a connection to the database. Returns a Connection Object. It takes a number of parameters which are database dependent. [1] """ with the footnote pointing to a set of parameters that can be used as guideline (either positional, keyword based or both). -- Marc-Andre Lemburg Y2000: 277 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : --------------------------------------------------------- From murray_williams@merck.com Wed Mar 31 16:19:51 1999 From: murray_williams@merck.com (Murray Todd Williams) Date: Wed, 31 Mar 1999 11:19:51 -0500 Subject: [DB-SIG] Help with compiling the mxODBC package under Windows Message-ID: <199903311620.LAA20020@python.org> My kingdom for a Makefile! I'll be the first to admit I don't know much about the MS VC++ that I have. I'm usually working in Linux, and the only time I use VC++ is through the command line. The instructions provided for building the mxODBC are a bit beyond me. (I) I don't know how to "point the compiler to Python/Include" or "point the linker to Python/Lib". (II) When I setup a blank DLL project and add the appropriate files, the target is a single DLL (I named the project mxODBC which I assume would be the right thing.) but the instructions mention renaming MULTIPLE *.dll files to *.pyd. (III) When I do my best to get some sort of command line compilation with cl /I..\..\Include mxODBC.cpp /DL I get the following errors: mxODBC.cpp(3559) : error C2086: 'mxODBCCursor_Type" : redefinition mxODBC.cpp(4016) : error C2086: "mxODBC_Type" : redefinition I'm stuck at this point. Could someone please give me a clue? The (prerequisite) mxDateTime was (almost) completely painless. Help! Murray Todd Williams Merck & Co., Inc. From mal@lemburg.com Wed Mar 31 16:51:30 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 31 Mar 1999 18:51:30 +0200 Subject: [DB-SIG] Help with compiling the mxODBC package under Windows References: <199903311620.LAA20020@python.org> Message-ID: <37025292.439E1FE8@lemburg.com> Murray Todd Williams wrote: > > My kingdom for a Makefile! > > I'll be the first to admit I don't know much about the MS VC++ that I > have. I'm usually working in Linux, and the only time I use VC++ is > through the command line. The instructions provided for building the > mxODBC are a bit beyond me. > > (I) I don't know how to "point the compiler to Python/Include" or "point > the linker to Python/Lib". > > (II) When I setup a blank DLL project and add the appropriate files, the > target is a single DLL (I named the project mxODBC which I assume would > be the right thing.) but the instructions mention renaming MULTIPLE > *.dll files to *.pyd. As the docs say, the instructions are """adapted from generic instructions by Gordon McMillan""". You actually don't need to rename any file (Python will load the DLL file just as it would the PYD file), it's only a little less troublesome having PYD files around instead of DLLs, because of naming issues. There's only one DLL being generated for mxODBC. > (III) When I do my best to get some sort of command line compilation > with > > cl /I..\..\Include mxODBC.cpp /DL > > I get the following errors: > mxODBC.cpp(3559) : error C2086: 'mxODBCCursor_Type" : redefinition > mxODBC.cpp(4016) : error C2086: "mxODBC_Type" : redefinition > > I'm stuck at this point. Could someone please give me a clue? The > (prerequisite) mxDateTime was (almost) completely painless. Try the current pre-release version of mxODBC: it includes a set of project files for VC6 donated by Stephen Ng. http://starship.skyport.net/~lemburg/mxODBC-pre1.0.2.zip Note that mxODBC is currently undergoing some rather big changes because I want to make it DB API 2.0 compatible, so the next release might be named 1.1.0 or even 2.0.0 -- in a few weeks maybe. -- Marc-Andre Lemburg Y2000: 275 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : --------------------------------------------------------- From mal@lemburg.com Wed Mar 31 16:58:28 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 31 Mar 1999 18:58:28 +0200 Subject: [DB-SIG] Remaining issues with DB API 2.0 References: <36FBA6D4.A6511A1@lemburg.com> <36FE03DA.1646520@lyra.org> <36FE43BB.6A1F45A2@lemburg.com> <36FE7F8B.67E06C09@lemburg.com> <36FEB11C.363EB5E9@lyra.org> <36FF3DF7.788D370A@lemburg.com> Message-ID: <37025434.7B432227@lemburg.com> Ok, we are nearing the finishing line with API 2.0. Here's what's left to decide (well at least for now): · Should .nextset() be an optional method or have optional functionality ? To make testing for "multiple result sets supported" possible, I think this API should be an optional method like .rollback: either not be defined (giving an AttributeError) or raise a NotSupportError... which brings us to the next point ;-) · Should we add NotSupportedError as subclass of OperationalError ? · Is there a strong need for "capability testing" other than using hasattr() ? Cheers, -- Marc-Andre Lemburg Y2000: 275 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : --------------------------------------------------------- From adustman@comstar.net Wed Mar 31 18:46:33 1999 From: adustman@comstar.net (Andy Dustman) Date: Wed, 31 Mar 1999 13:46:33 -0500 (EST) Subject: [DB-SIG] Remaining issues with DB API 2.0 In-Reply-To: <37025434.7B432227@lemburg.com> Message-ID: eOn Wed, 31 Mar 1999, M.-A. Lemburg wrote: > To make testing for "multiple result sets supported" possible, > I think this API should be an optional method like .rollback: > either not be defined (giving an AttributeError) or raise > a NotSupportError... which brings us to the next point ;-) > > · Should we add NotSupportedError as subclass of OperationalError ? > > · Is there a strong need for "capability testing" other than using > hasattr() ? I have no objections to making it an optional method. As for other capability testing, I need to think that any program that relies upon some capabilty to function is simply going to break (hopefully in an obvious way) when it uses a database which doesn't have the capability in question. For example, if I have an application which depends upon rollback() being there, and I move my application to a database with no rollback(), it should fail in such a way that I become clueful that there is a problem. I then need to redesign my application. Hopefully if I've done the design properly, I can make a subclass which doesn't depend on the feature in question. All of which leads us back to connect(), which we have now established (yay) is completely database-dependent, so we acknowledge that interface can't cover everything, which is no real embarassment. This really means that there should be some design principles involved in writing database applications. For another example: I do have an application (which does in fact use rollback()) that I am moving from ODBC.Solid to MySQL, which of course doesn't have it. The design is such that I have a couple of abstraction layers between my application and the actual database interface. The important part of this is that I have a specialized API for my schema. It wraps a connection object and adds various methods to do specific queries, passing in and returning objects. (There's actually a layer between this and the database interface that converts between my objects and suitable query parameters/rows.) The result of this is, I really only have one module in the system that I have to modify. If I really wanted to, I could make a new module (or convert to a package) and subclass only a few methods to make it compatible with the new database. But then, I shouldn't be depending on either hasattr() for capability testing; I should RTFM. Some rewriting when switching databases is going to be inevitable, even if I'm using the same interface (i.e. mxODBC). This should not actually be part of the specification, but perhaps we need a design note, or guidelines for designing applications so that they can ported between different databases with a minimum of effort. This makes your application more resilient to changes in the DB API as well (though it's hardly affecting my application, if at all, and the API shouldn't change much more in the future). -- Andy Dustman (ICQ#32922760) You should always say "spam" and "eggs" ComStar Communications Corp. instead of "foo" and "bar" (706) 549-7689 | PGP KeyID=0xC72F3F1D in Python examples. (Mark Lutz) From mal@lemburg.com Wed Mar 31 20:05:38 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 31 Mar 1999 22:05:38 +0200 Subject: [DB-SIG] Remaining issues with DB API 2.0 References: Message-ID: <37028012.3AB3D8A7@lemburg.com> Andy Dustman wrote: > > eOn Wed, 31 Mar 1999, M.-A. Lemburg wrote: > > > To make testing for "multiple result sets supported" possible, > > I think this API should be an optional method like .rollback: > > either not be defined (giving an AttributeError) or raise > > a NotSupportError... which brings us to the next point ;-) > > > > · Should we add NotSupportedError as subclass of OperationalError ? > > > > · Is there a strong need for "capability testing" other than using > > hasattr() ? > > I have no objections to making it an optional method. As for other > capability testing, I need to think that any program that relies upon some > capabilty to function is simply going to break (hopefully in an obvious > way) when it uses a database which doesn't have the capability in > question. Ok. Greg ? > [...] > > This should not actually be part of the specification, but perhaps we need > a design note, or guidelines for designing applications so that they can > ported between different databases with a minimum of effort. This makes > your application more resilient to changes in the DB API as well (though > it's hardly affecting my application, if at all, and the API shouldn't > change much more in the future). Would make a nice addendum to the spec... or maybe a separate document. Volunteers ? Andy :-) ? -- Marc-Andre Lemburg Y2000: 275 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : --------------------------------------------------------- From adustman@comstar.net Wed Mar 31 21:14:46 1999 From: adustman@comstar.net (Andy Dustman) Date: Wed, 31 Mar 1999 16:14:46 -0500 (EST) Subject: [DB-SIG] Remaining issues with DB API 2.0 In-Reply-To: <37028012.3AB3D8A7@lemburg.com> Message-ID: On Wed, 31 Mar 1999, M.-A. Lemburg wrote: > > This should not actually be part of the specification, but perhaps we need > > a design note, or guidelines for designing applications so that they can > > ported between different databases with a minimum of effort. This makes > > your application more resilient to changes in the DB API as well (though > > it's hardly affecting my application, if at all, and the API shouldn't > > change much more in the future). > > Would make a nice addendum to the spec... or maybe a separate > document. Volunteers ? Andy :-) ? I'll see what I can do, try to consolidate what I said earlier in some generalized way. -- Andy Dustman (ICQ#32922760) You should always say "spam" and "eggs" ComStar Communications Corp. instead of "foo" and "bar" (706) 549-7689 | PGP KeyID=0xC72F3F1D in Python examples. (Mark Lutz)