Data storing

[Steven D'Aprano]

On Tue, 2 Feb 2016 01:19 pm, Anup reni wrote:

> Hi I'm new to python and planning to make a web app which contains a large
> amount of data and involves comparing two data sets. Which is better to
> use in python, a Dictionary or Mysql?

"Hi, I'm planning to transport a large number of objects from one place to
another. Which is better for me to use, a ship or an airplane?"

:-)

Well, it depends.

How much data? You might think 100 MB is a lot of data, but Python won't.

How are you planning to compare them? What sort of comparisons do you intend
to do?

If all you have is a set of key:value pairs, then dicts are the obvious
choice. They are built-in and very fast, but unordered. They'll be good for
millions or tens of millions of records, but they aren't persistent data
storage. You will have to load them to and from disk each time the
application starts. Perhaps the "shelve" module might help you with that.

If you have many different relationships, and you want, or need, to use SQL
commands to do some initial data processing, then you will certainly need a
real database, not dicts.

I would prefer Postgresql rather than MySQL, **especially** if you need to
support non-ASCII (Unicode) data.

But really, to decide which would be best for you, we would need to know a
lot more about your application and its requirements.


-- 
Steven