Using Python for processing of large datasets (convincing managment)

[Alex Martelli]

Paul Rubin wrote:

> Alex Martelli <aleax at aleax.it> writes:
>> > I'm a little scared though about using SQL too extensivly.
>> 
>> Joe Celko's "SQL for Smarties" might be just what you need to move you
>> to the next level of SQL usage (or, to turn you off it for life,
>> depending:-).
> 
> Philip Greenspun's SQL for Web Nerds
> 
>    http://philip.greenspun.com/sql/index.html
> 
> is pretty good (at least for non-SQL-whizzes like me) and has a
> reasonable amount of info on performance tuning and optimization.

Indeed, a fast scan seems to show that Philip's coverage is quite
different from Joe's -- Joe talks about SQL (lots of SQL, very
advanced SQL, SQL to do things you wouldn't think could be done
in SQL [and maybe in some cases shouldn't!-)]), while Philip does
a reasonably wide-ranging survey of Oracle issues, including the very
best piece of advice of them all -- *hire an expert*.

There's a think3 application that uses a RDBMS and supports many
different RDBMS brands.  Towards the end of my long stay in think3,
most substantial installations of this application were being
deployed on either MS SQL Server, or Oracle.  It seems the customers
using MS SQL Server were more or less "OK, not happy, but OK" with
our performance.  OTOH, the customers using Oracle tended to fall
in two widely separated camps -- a group perfectly happy with our
performance, a group deeply dissatisfied with it.  Funny -- why...?

Solution: MS SQL Server ain't all that tunable -- it gives you
reasonable performance if you program and administer it reasonably,
and that's about it... no superb performance, no horrors either.
Oracle, OTOH, _is_ very highly tunable, and *demands* such kind
of tuning -- if you give it proper care and feeding you get stellar
performance and scalability, otherwise it can truly be the pits.

The happy Oracle-using customers were large firms who had hired
(or home-grown) full-time experts at Oracle tuning, working full
time to keep the datases humming, including those used by our
application.  The unhappy ones were mostly small and middling
firms who _thought_ they could just buy Oracle, perhaps pay for
a once-off installation / tuning / training for low-skilled or
overworked/wide-responsibility sysadm staff, and forget it.

Wrong.  Very wrong.  If you can afford Oracle, you can afford to
hire (or at least regularly retain a freelance) expert help to
tune it, keep it tuned, make it perform to its potential.  If you
can't, you may be better off with cheaper stuff (not necessarily
MS SQL Server, which has the big minus of constraining you to
wimpy Windows servers, but, say, SAP/DB, nee Adabas -- free, and
there's a lot of expertise around for it from the times when it
was a costly, enterprise-grade commercial RDBMS).


Disclaimer: this is basically hearsay -- I couldn't tune an Oracle
DB myself any more than I could tune a fish -- AND it may be based
on now-obsolete versions of the products named (for all I _know_, the
Oracle you can buy today is self-tuning, and/or the MS SQL Server
you can buy today requires half a dozen full-time staff with degrees
in nuclear engineering -- I just find such hypotheses unlikely:-).



Alex