custom data warehouse in python vs. out-of-the-box ETL tool

[snfctech]

@Martin:  I originally thought that there was nothing "magical" about
building a data warehouse, but then I did a little research and
received all sorts of feedback about how data warehouse projects have
notorious failure rates, that data warehouse design IS different than
normal RDBMS - and then there's the whole thing about data marts vs.
warehouses, Kimball vs. Inmon, star schemas, EAV tables, and so on.
So I started to think that maybe I needed to get a little better read
on the subject.

On Sep 23, 3:15 am, "Martin P. Hellwig" <martin.hell... at dcuktec.org>
wrote:
> snfctech wrote:
> > Thanks for your replies, Sean and Martin.
>
> > I agree that the ETL tools are complex in themselves, and I may as
> > well spend that learning curve on a lower-level tool-set that has the
> > added value of greater flexibility.
>
> > Can you suggest a good book or tutorial to help me build a data
> > warehouse in python?  Bill Inmon's "Building the Data Warehouse" is 17
> > years old, and I've been cautioned against Kimball.
>
> > Thanks.
>
> <cut>
>
> Data warehouse isn't something magical, it is just another database,
> albeit containing multiple datasets gathered from foreign resources in
> possibly multiple formats.
>
> Depending on your purpose of what you want, you design your tables the
> way you usually do. For example if you only want reporting, you might
> want to build your tables in such a way so it makes your life easier to
> build the actual report.
>
> Now you have an empty database containing the fields you wish for the
> report and have filled database(s) containing data from the user
> application. Now you use Python to fill the empty database, tada, you
> have a Data warehouse and used Python for ETL processing.
>
> So if you already have some insights in creating tables in a database,
> you are all set. Most likely you will go through a number of iterations
> before you are happy with the result though.
>
> There is no book substitute for applying theory, experience and common
> sense to a problem you want to solve, unless you write it yourself for
> that specific situation.
>
> --
> MPHhttp://blog.dcuktec.com
> 'If consumed, best digested with added seasoning to own preference.'