Most space-efficient way to store log entries

[Cameron Simpson]

On 29Oct2015 23:16, Laura Creighton <lac at openend.se> wrote:
>In a message of Fri, 30 Oct 2015 08:28:07 +1100, Cameron Simpson writes:
>>On 29Oct2015 09:15, Laura Creighton <lac at openend.se> wrote:
>>>Did the OP say he wanted to keep his compressed logfiles on a
>>>local disk?  What if he wants to send them across the internet
>>>to some other machine and would like the transfer to happen as
>>>quickly as possible?
>>
>>Then he's still better off keeping them uncompressed and using compression in
>>the transfer. "ssh -o compression=yes" or "rsync -z", etc.
>>
>>Cheers,
>>Cameron Simpson <cs at zip.com.au>
>
>"ssh -o compression=yes" is what I am using.
>But I am all ears for a better idea.
>
>We can do no better?

Depends on your criteria for "better". Absent other constraints I'm broadly 
for:

  keeping logs uncompressed and unencoded, for ease of eyeballing and processing with text tools

  rotating and compressing logs if feasible

  avoiding compression and other encoding within log lines

Another post suggests that the OP is transferring log info in UDP packets and 
hopes to keep the state within a maximum packet size, hence his desire for 
compact representation. I suspect that personally I'd be going for some 
efficient text encoding of the state and putting whatever compression he 
intends in the UDP throw/catch:

  take text log line
  compress
  send over UDP
  receive UDP packet
  decompress
  store in clear text

or bypass UDP altogether, but I imagine the OP has his reasons.

Cheers,
Cameron Simpson <cs at zip.com.au>