search speed

[Justin Wyer]

On Fri, Jan 30, 2009 at 1:51 AM, anders <anders.u.persson at gmail.com> wrote:

> Hi!
> I have written a Python program that serach for specifik customer in
> files (around 1000 files)
> the trigger is LF01 + CUSTOMERNO
>
> So a read all fils with dirchached
>
> Then a loop thru all files each files is read with readLines() and
> after that scaned
>
> Today this works fine, it saves me a lot of manuall work, but a seach
> takes around 5 min,
> so my questin is is there another way of search in a file
> (Today i step line for line and check)


Do you require this information in a python application, seems like you did
this manually before?

If not then python is the wrong tool for this job, you can simply use this
command on a unix-like environment (install cygwin, if you are on windows)

$ find <path_to_dirs_containing_files> -name "*" -exec grep -nH "LF01" {} \;
| cut -d ":" -f 1 | sort | uniq

Now if you do require this information inside a python app, I would just do
the above in python

filenames = []
searchCmd = "find <path_to_dirs_containing_files> -name \"*\" -exec grep -nH
\"LF01\" {} \; | cut -d \":\" -f 1 | sort | uniq"
searchp = Popen(searchCmd, shell=True, bufsize=4096, stdout=PIPE)
    for line in searchp.stdout:
      filenames.append(line.strip())

Thats my advise anyway, guess you can try some search libraries don't know
of any mysql tho, the above will probably be faster than anything else.

Cheers and good luck.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20090130/1ad2f169/attachment-0001.html>