[Tutor] looking but not finding

[Mats Wichmann]

On 7/12/23 09:26, o1bigtenor wrote:
> Thank you for your patience mr Alan.
> 
> What I'm finding is that the computing world uses terms in a different way than
> does the sensor world than does the storage world than does the rest of us.
> That seriously complicates things.

I'm guessing you have no idea how bad the jargon problem is... and it 
certainly doesn't help when terms are "overloaded" - given different 
special meanings by different groups.

Logging:  what you've found has to do with event logging - where a 
program wants to record things that have happened in the running of the 
program itself.  What you're after is what some call data logging.   You 
can even look for projects that self-declare being related to that term 
like this:

https://github.com/topics/data-logging

 >> There are many, many different types of storage available
 >> from structured files(shelves, JSON, YAML, XML, etc to
 >> databases both SQL and NoSQL based.

 > This is where I start getting 'lost'. I have no idea which of the three
 > listed formats will work well with what I'm doing - - - dunno even how to
 > find out either!

The storage format is something you might consider an implementation 
detail, that you don't have to worry about yet. The first steps are the 
ones you're taking: to flesh out what your requirements are, and move to 
a conceptual design. When you think you have a vision of what it will 
look like, you can see if some particular existing library provides an 
easy to use implementation of one of these that would fit your needs, 
with some room to group.  If you design it right, the piece that drives 
that could also drive a different thing, if you found you chose poorly, 
or outgrew it.

JSON, YAML and XML are text-based formats, with some markup to help both 
you as a human reader and in some cases, computer code, to decode it. If 
there's not a ton of data, and/or there's not a lot of interrelationship 
between different types of data (suggesting a "relational" database), 
one of these will probably be fine. XML is a lot chattier, and the XML 
libraries in Python are quite fiddly to use so recommend you cross XML 
off your candidate list unless there are really good reasons to needs 
its level of structure.  You can even use comma-separated-values (like a 
spreadsheet program would dump put), which is quite inflexible - one 
line is one record, and every field has to be present in every record, 
you don't have the option of extra or omitted fields, or it just breaks. 
  Pretty sure each one of those has an eloquent wikipedia page, e.g.

https://en.wikipedia.org/wiki/JSON#Syntax

If you want to use a database, rather than a text file, Alan laid out 
some choices. For reasonably small amounts of data, sqlite is usually 
fine. It's perhaps closer to the text-based interchange formats in that 
there's no separate database program that you communicate with, data is 
read from and written to a disk files in-process, but the interaction 
with that data is through SQL statements so it's also like the "big 
databases" - mysql/mariadb, SQL Server, Postgres.  Then the "noSQL" 
databases (that's not a terribly precise term) are the ones that have 
less structure, to aid in scalability - a problem you're not likely to 
have from your description so won't say more here.