Most space-efficient way to store log entries

[Mark Lawrence]

On 28/10/2015 22:53, Tim Chase wrote:
> On 2015-10-29 09:38, Chris Angelico wrote:
>> On Thu, Oct 29, 2015 at 9:30 AM, Marc Aymerich
>> <glicerinu at gmail.com> wrote:
>>> I'm writting an application that saves historical state in a log
>>> file. I want to be really efficient in terms of used bytes.
>>
>> Why, exactly?
>>
>> By zipping the state, you make it utterly opaque.
>
> If it's only zipped, it's not opaque.  Just `zcat` or `zgrep` and
> process away.  The whole base64+minus_newlines thing does opaquify
> and doesn't really save all that much for the trouble.
>
>> Disk space is not expensive. Even if you manage to cut your file by
>> a factor of four (75% compression, which is entirely possible if
>> your content is plain text, but far from guaranteed)
>
> Though one also has to consider the speed of reading it off the drive
> for processing.  If you have spinning-rust drives, it's pretty slow
> (and SSD is still not like accessing RAM), and reading zipped
> content can shovel a LOT more data at your CPU than if it is coming
> off the drive uncompressed.  Logs aren't much good if they aren't
> being monitored and processed for the information they contain.  If
> nobody is monitoring the logs, just write them to /dev/null for 100%
> compression. ;-)
>
> -tkc
>

Can you get better than 100% compression if you write them to somewhere 
other than /dev/null/ ?

-- 
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence