Is there any library for indexing binary data?

[Irmen de Jong]

On 3/25/10 4:28 AM, 甜瓜 wrote:
> Howdy,
>
> Recently, I am finding a good library for build index on binary data.
> Xapian&  Lucene for python binding focus on text digestion rather than
> binary data. Could anyone give me some recommendation? Is there any
> library for indexing binary data no matter whether it is written in
> python?
>
> In my case, there is a very big datatable which stores structured
> binary data, eg:
> struct Item
> {
>      long id; // used as key
>      double value;
> };
>
> I want to build the index on "id" field to speed on searching. Since
> this datatable is not constant, the library should support incremental
> indexing. If there is no suitable library, I have to do the index by
> myself...
>
> Thank you in advance.
>
> --
> ShenLei

Put it into an Sqlite database? Or something else from 
http://docs.python.org/library/persistence.html.
Or maybe http://www.pytables.org/ is more suitable to your needs (never 
used that one myself though).
Or install a bank or 2 of memory in your box and read everything into 
memory in one big hashtable.

Btw if you already have a big datatable in which the data is stored, I'm 
guessing that already is in some form of database format. Can't you 
write something that understands that database format.

But I think you need to provide some more details about your data set.

-irmen