Comparing two book chapters (text files)

[Nick Matzke]

Chris Rebert wrote:
> On Wed, Feb 4, 2009 at 5:20 PM, Nick Matzke <matzke at berkeley.edu> wrote:
>> Hi all,
>>
>> So I have an interesting challenge.  I want to compare two book chapters,
>> which I have in plain text format, and find out (a) percentage similarity
>> and (b) what has changed.
>>
>> Some features make this problem different than what seems to be the standard
>> text-matching problem solvable with e.g. difflib.  Here is what I mean:
>>
>> * there is no guarantee that single lines from each file will be directly
>> comparable -- e.g., if a few words are inserted into a sentence, then a
>> chunk of the sentence will be moved to the next line, then a chunk of that
>> line moved to the next, etc.
>>
>> * Also, there are cases where paragraphs have been moved around, sections
>> re-ordered, etc.  So it can't just be a "linear" match.
>>
>> I imagine this kind of thing can't be all that hard in the grand scheme of
>> things, but I couldn't find an easily applicable solution readily available.
>>  I have advanced beginner python skills but am not quite where I could do
>> this kind of thing from scratch without some guidance about the likely
>> functions, libraries etc. to use.
>>
>> PS: I am going to have to do this for multiple book chapters so various
>> software packages, e.g. for windows, are not really usable.
> 
> Though not written in Python, wdiff
> (http://www.gnu.org/software/wdiff/wdiff.html) might be a good
> starting point.


Wow -- this is actually amazingly effective.  And fast!   Simple to run 
from python & then use python to parse the output.

Thanks!
Nick


> 
> Cheers,
> Chris
> 

-- 
====================================================
Nicholas J. Matzke
Ph.D. student, Graduate Student Researcher
Huelsenbeck Lab
Center for Theoretical Evolutionary Genomics
4151 VLSB (Valley Life Sciences Building)
Department of Integrative Biology
University of California, Berkeley

Lab websites:
http://ib.berkeley.edu/people/lab_detail.php?lab=54
http://fisher.berkeley.edu/cteg/hlab.html
Dept. personal page: 
http://ib.berkeley.edu/people/students/person_detail.php?person=370
Lab personal page: http://fisher.berkeley.edu/cteg/members/matzke.html
Lab phone: 510-643-6299
Dept. fax: 510-643-6264
Cell phone: 510-301-0179
Email: matzke at berkeley.edu

Mailing address:
Department of Integrative Biology
3060 VLSB #3140
Berkeley, CA 94720-3140

-----------------------------------------------------
"[W]hen people thought the earth was flat, they were wrong. When people 
thought the earth was spherical, they were wrong. But if you think that 
thinking the earth is spherical is just as wrong as thinking the earth 
is flat, then your view is wronger than both of them put together."

Isaac Asimov (1989). "The Relativity of Wrong." The Skeptical Inquirer, 
14(1), 35-44. Fall 1989.
http://chem.tufts.edu/AnswersInScience/RelativityofWrong.htm
====================================================