Merge/append CSV files with different headers

[Mark Lawrence]

On 24/03/2014 17:50, Vincent Davis wrote:
> I have several csv file I need to append (vertically). They have
> different but overlapping headers. For example;
> file1 headers ['a', 'b', 'c']
> file2 headers ['d', 'e']
> file3 headers ['c', 'd']
>
> Is there a better way than this
> import csv
> def merge_csv(fileList, newFileName):
>      allHeaders = set([])
>      for afile in fileList:
>          with open(afile, 'rb') as csvfilesin:
>              eachheader = csv.reader(csvfilesin, delimiter=',').next()
>              allHeaders.update(eachheader)
>      print(allHeaders)
>      with open(newFileName, 'wb') as csvfileout:
>          outfile = csv.DictWriter(csvfileout, allHeaders)
>          outfile.writeheader()
>          for afile in fileList:
>              print('***'+afile)
>              with open(afile, 'rb') as csvfilesin:
>                  rows = csv.DictReader(csvfilesin, delimiter=',')
>                  for r in rows:
>                      print(allHeaders.issuperset(r.keys()))
>                      outfile.writerow(r)
>
> Vincent Davis
>

I haven't looked too hard at your code but guess you could simplify it 
by using the fieldnames parameter to the DictReader class or making use 
of the Sniffer class.  See 
http://docs.python.org/3/library/csv.html#module-csv


-- 
My fellow Pythonistas, ask not what our language can do for you, ask 
what you can do for our language.

Mark Lawrence

---
This email is free from viruses and malware because avast! Antivirus protection is active.
http://www.avast.com