[Tutor] CSV Read

[Peter Otten]

Giulia Marcoux wrote:

> Hello,
> 
> I am a student learning python, and i am running into some difficulties. I
> am tryign to read a csv file that has different delimiters for different
> rows: Example:
> 
> Format:V1.1
> Model: R
> Step Size: 10mm
> Distance: 10cm
> Gain: 1000
> 
> X,Y
> 1,3
> 2,5
> 6,5
> 5,7
> 
> For the X,Y data, I have no problem getting it into the right cells,
> however I cannot separate the values after the colon for the first few
> rows. Its either one or the other. I am using pandas to read my CSV file.
> So far I have:
> 
> d=pd.read_csv('read.csv',delimiter=',',skiprows=0)
> 
> When I replace the delimiter for ':', it no longer works, and if i put
> sep=',|:' it does not work either. In the end I need to plot the x,y
> values and i will need some of the info in the rows above like the gain to
> manipulate the y data. If you have any tips for me I would be very
> grateful!

Rather than forcing both blocks into the same data structure I would read 
the header with the "key: value" lines into a separate object.

If the header and the X,Y pairs are separated by an empty line read until 
that and then feed the file object to pandas.read_csv() to read those pairs:

import itertools
import pandas as pd


def header(f):
    """Pass on lines from f; stop at the first whitespace-only line.
    """
    return itertools.takewhile(str.strip, f)


def pairs(lines, sep=":"):
    """Split lines into (key, value) pairs.

    Remove leading/trailing whitespace from value.
    """
    for line in lines:
        key, value = line.split(sep, 1)
        yield key, value.strip()


with open("data.txt") as f:
    metadata = dict(pairs(header(f)))
    df = pd.read_csv(f)

print(metadata)
print(df)