[Tutor] Load Entire File into memory

[Amal Thomas]

@Steven: Thank you...My input data is basically AUGC and newlines... I
would like to know about bytearray technique. Please suggest me some links
or reference.. I will go through the profiler and check whether the code
maintains linearity with the input files.




> > It's probably worth putting some more timing statements into your code
> > to see where the time is going because it's not the reading from the
> > disk that's the problem.
>
> The first thing I would do is run the code on three smaller sample
> files:
>
> 50MB
> 100MB
> 200MB
>
> The time taken should approximately double as you double the size of the
> file: say it takes 2 hours to process the 50MB file, 4 hours for the
> 100MB file and 8 hours for the 200 MB file, that's linear performance
> and isn't too bad.
>
> But if performance isn't linear, say 2 hours, 4 hours, 16 hours, then
> you're in trouble and you *desperately* need to reconsider the algorithm
> being used. Either that, or just accept that this is an inherently slow
> calculation and it will take a week or two.
>
> Amal, another thing you should try is use the Python profiler on your
> code (again, on a smaller sample file). The profiler will show you where
> the time is being spent.
>
> Unfortunately the profiler may slow your code down, so it is important
> to use it on manageable sized data. The profiler is explained here:
>
> http://docs.python.org/3/library/profile.html
>
> If you need any help, don't hesitate to ask.
>
>
> > >trying to optimize my code to get the outputs in less time and memory
> > >efficiently.
> >
> > Memory efficiency is easy, do it line by line off the disk.
>
> This assumes that you can process one line at a time, sequentially. I
> expect that is not the case.
>
>
> --
> Steven
> _______________________________________________
> Tutor maillist  -  Tutor at python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor
>



-- 


*AMAL THOMAS*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/tutor/attachments/20131104/3da3c04e/attachment.html>