Merge/append CSV files with different headers

[Chris Angelico]

On Tue, Mar 25, 2014 at 4:50 AM, Vincent Davis <vincent at vincentdavis.net> wrote:
> I have several csv file I need to append (vertically). They have different
> but overlapping headers. For example;
> file1 headers ['a', 'b', 'c']
> file2 headers ['d', 'e']
> file3 headers ['c', 'd']
>
> Is there a better way than this

Summary of your code:

1) Build up a set of all headers used, by opening each file and
reading the headers.
2) Go through each file a second time and write them out.

That seems like the best approach, broadly. You might be able to
improve it a bit (it might be tidier to open each file once, but since
you're using two different CSV readers, it'd probably not be), but by
and large, I'd say you have the right technique. Your processing time
here is going to be dominated by the actual work of copying.

The only thing you might want to consider is order. The headers all
have a set order to them, and it'd make sense to have the output come
out as ['a', 'b', 'c', 'd', 'e'] - the first three from the first
file, then adding in everything from subsequent files in the order
they were found. Could be done easily enough by using 'in' and
.append() on a list, rather than using a set. But if that doesn't
matter to you, or if something simple like "sort the headers
alphabetically" will do, then I think you basically have what you
want.

ChrisA