[Ncr-Python.in] [ILUGD] Which technology is better for large database calculations

[Gora Mohanty]

On 12 May 2014 23:47, Raakesh kumar <kumar3180 at gmail.com> wrote:
> Hi All,
> I am seeking help in terms of DB technology for a specific requirement. I
> have done some research but can not conclude which one is best.
> So the requirement is:
> I have an application and have 3-4 excel documents in specified
> format(expecting 1 Million of rows). I have to upload it and save into some
> tables and run some calculation against data base values. After that it will
> present me with some numbers/chart based on calculations.
>
> So my first question is, how should i select my database technology which
> will stand better in this scenario. I found that MySql can perform read
> operations perfectly in these scenario but if someone can help me understand
> it properly? I also read that Postgre is a better choice in terms of
> reliability and for a better structure, although i am not sure.
>
> Second question is how much time will it take to upload 1 million rows of
> data from Excel, considering there will be 6 columns? Or how can i optimize
> to speed up read and write operations for such a large data.
>
> Third, What are the technologies at application development level
> (programming level), i can use to achieve a better performance. One solution
> i can think of is to use Hadoop but need guidance on this too.

Your question makes little sense. 1 million rows in 3-4 Excel documents? Your
bottle-neck would be Excel, which would very likely crash and burn at well below
the capacity you want. You could use postgres, mysql, mariadb, sqlite, JimBob's
excellent DB, and probably even stone tablets as the database back-end :-)
Would not make any difference.

<flamewar>If you want a real free/open DB, use Postgres</flamewar>

Regards