Multiple files reading

[MRAB]

On 2016-06-14 17:53, Joaquin Alzola wrote:
> Hi Guys
>
> I am doing  program that reads into a directory for the files that were created the last 5 mins. (working)
>
> Inside those files there are 242 fields in each line separated by | (pipe). Each file has about 5k records and there are about 5 files per 5 mins.
>
> I will look for field 29 and 200 (Country, Diameter Error code). (split)
>
> I have 6 different countries (which I differentiate by field 29 which is the CountryCode).
>
> The thing is that I make it work but it goes slow. On the parent class I read the files all over for each Country 150k lines read (because I read 25k for each country which makes a total of 150k lines read).
> So the code even working is inefficient.
>
> I created classes such as:
>
> Read Files (Parent) - Country -- Service
>
> The dictionary that I am using in the classes: {'Country':'Empty','Service':'Empty','TimeStamp':'Empty','Ocg':'see3','DiameterCodes':{'2001':0,'4010':0,'4012':0,'4998':0,'4999':0,'5007':0,'5012':0}}
>
> Wanted help from your side on how to focus this just because I want to read the files once (not 6 times) and then use classes to get back the dictionary value ...
>
> I need just advice on steps to follow...
>
Use a dict (or defaultdict) where the key is the country and the value 
is info (class?) related to that country.