From leamhall at gmail.com Sun Jul 2 08:21:40 2023 From: leamhall at gmail.com (Leam Hall) Date: Sun, 2 Jul 2023 07:21:40 -0500 Subject: [Tutor] async learning Message-ID: Until recently, I haven't had a lot of reason to do async stuff. Now I'm trying to figure it out, but I'm missing something. The commented out section is my attempt at async, but it runs slower than the non-async version. Thoughts? The parameters are: Python 3.9 or 3.10 Import from Standard Library only Later, add_stuff() will include running python processes to gather data Code so far: import asyncio import urllib.request site = "https://en.wikipedia.org/wiki/" def add_stuff(name, site): url = site + name result = dict() result['name'] = name.replace('_', ' ') result['details'] = urllib.request.urlopen(url).read() return result def show_stuff(thing): info = thing[1] name = info['name'] details = info['details'] result = "Let's look at {}, with {} bits and bytes of detail.".format(name, len(details)) return result #async def main(names): # team = { name: add_stuff(name, site) for name in names} # return team if __name__ == "__main__": names = ["Frodo_Baggins", "Samwise_Gamgee", "Merry_Brandybuck", "Pippin_Took"] #team = asyncio.run(main(names)) team = { name: add_stuff(name, site) for name in names} for t in sorted(team.items()): print(show_stuff(t)) -- Software Engineer (reuel.net/resume) Scribe: The Domici War (domiciwar.net) General Ne'er-do-well (github.com/LeamHall) From alan.gauld at yahoo.co.uk Tue Jul 4 18:02:26 2023 From: alan.gauld at yahoo.co.uk (Alan Gauld) Date: Tue, 4 Jul 2023 23:02:26 +0100 Subject: [Tutor] async learning In-Reply-To: References: Message-ID: On 02/07/2023 13:21, Leam Hall wrote: > Until recently, I haven't had a lot of reason to do async stuff. ...> but it runs slower than the non-async version. Thoughts? I'm no expert but trivial tasks often run slower on async architecture than when in a single process, even a single thread. I don;t know how you measured things or what kind of tasks you were doing, but if a single task takes less than a few milliseconds to execute it's likely that the overheads in async are higher than the benefits of concurrency. Try creating a task that takes some time such as opening a file and processing its contents in some way(search/sort/convert/calculate something per line, say). Then create many such files and have your code process them all. That should show a gain of some sort. Concurrency is hard! And the right kind of concurrency depends greatly on the problem. async is best suited to server type scenarios. > > The parameters are: > Python 3.9 or 3.10 > Import from Standard Library only > Later, add_stuff() will include running python processes to gather data > > Code so far: > > import asyncio > import urllib.request > > site = "https://en.wikipedia.org/wiki/" > > def add_stuff(name, site): > url = site + name > result = dict() > result['name'] = name.replace('_', ' ') > result['details'] = urllib.request.urlopen(url).read() > return result > > def show_stuff(thing): > info = thing[1] > name = info['name'] > details = info['details'] > result = "Let's look at {}, with {} bits and bytes of detail.".format(name, len(details)) > return result > > #async def main(names): > # team = { name: add_stuff(name, site) for name in names} > # return team > > if __name__ == "__main__": > names = ["Frodo_Baggins", "Samwise_Gamgee", "Merry_Brandybuck", "Pippin_Took"] > > #team = asyncio.run(main(names)) > team = { name: add_stuff(name, site) for name in names} > > for t in sorted(team.items()): > print(show_stuff(t)) -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos From mats at wichmann.us Tue Jul 4 19:21:26 2023 From: mats at wichmann.us (Mats Wichmann) Date: Tue, 4 Jul 2023 17:21:26 -0600 Subject: [Tutor] async learning In-Reply-To: References: Message-ID: <550b5386-6d4c-0733-ed44-dac69f7e8897@wichmann.us> On 7/4/23 16:02, Alan Gauld via Tutor wrote: > On 02/07/2023 13:21, Leam Hall wrote: >> Until recently, I haven't had a lot of reason to do async stuff. ...> but it runs slower than the non-async version. Thoughts? > > I'm no expert but trivial tasks often run slower on async > architecture than when in a single process, even a single > thread. > > I don;t know how you measured things or what kind of tasks > you were doing, but if a single task takes less than a few > milliseconds to execute it's likely that the overheads in > async are higher than the benefits of concurrency. > > Try creating a task that takes some time such as opening a > file and processing its contents in some > way(search/sort/convert/calculate something per line, say). Then create > many such > files and have your code process them all. That should show > a gain of some sort. This one does (fetches from a url, the classic example of a "slow" operation in computer terms), but... it's not happening in an async function. As a first step, "add_stuff" needs to be an "async def". > > Concurrency is hard! And the right kind of concurrency > depends greatly on the problem. async is best suited to > server type scenarios. > > >> >> The parameters are: >> Python 3.9 or 3.10 >> Import from Standard Library only >> Later, add_stuff() will include running python processes to gather data >> >> Code so far: >> >> import asyncio >> import urllib.request >> >> site = "https://en.wikipedia.org/wiki/" >> >> def add_stuff(name, site): >> url = site + name >> result = dict() >> result['name'] = name.replace('_', ' ') >> result['details'] = urllib.request.urlopen(url).read() >> return result >> >> def show_stuff(thing): >> info = thing[1] >> name = info['name'] >> details = info['details'] >> result = "Let's look at {}, with {} bits and bytes of detail.".format(name, len(details)) >> return result >> >> #async def main(names): >> # team = { name: add_stuff(name, site) for name in names} >> # return team >> >> if __name__ == "__main__": >> names = ["Frodo_Baggins", "Samwise_Gamgee", "Merry_Brandybuck", "Pippin_Took"] >> >> #team = asyncio.run(main(names)) >> team = { name: add_stuff(name, site) for name in names} >> >> for t in sorted(team.items()): >> print(show_stuff(t)) > > From leamhall at gmail.com Tue Jul 4 20:34:53 2023 From: leamhall at gmail.com (Leam Hall) Date: Tue, 4 Jul 2023 19:34:53 -0500 Subject: [Tutor] async learning In-Reply-To: <550b5386-6d4c-0733-ed44-dac69f7e8897@wichmann.us> References: <550b5386-6d4c-0733-ed44-dac69f7e8897@wichmann.us> Message-ID: <07a7929b-1c6c-c166-088c-f74a34f887fa@gmail.com> On 7/4/23 18:21, Mats Wichmann wrote: > On 7/4/23 16:02, Alan Gauld via Tutor wrote: >> On 02/07/2023 13:21, Leam Hall wrote: >>> Until recently, I haven't had a lot of reason to do async stuff. ...> but it runs slower than the non-async version. Thoughts? >> >> I'm no expert but trivial tasks often run slower on async >> architecture than when in a single process, even a single >> thread. >> >> I don;t know how you measured things or what kind of tasks >> you were doing, but if a single task takes less than a few >> milliseconds to execute it's likely that the overheads in >> async are higher than the benefits of concurrency. >> >> Try creating a task that takes some time such as opening a >> file and processing its contents in some >> way(search/sort/convert/calculate something per line, say). Then create >> many such >> files and have your code process them all. That should show >> a gain of some sort. > > This one does (fetches from a url, the classic example of a "slow" operation in computer terms), but... it's not happening in an async function.? As a first step, "add_stuff" needs to be an "async def". Thanks! I'll give that a go. The code I posted was the minimum I could find to demonstrate what I was trying to do. Leam -- Software Engineer (reuel.net/resume) Scribe: The Domici War (domiciwar.net) General Ne'er-do-well (github.com/LeamHall) From dcasher at usnx.com Tue Jul 11 13:20:31 2023 From: dcasher at usnx.com (dcasher at usnx.com) Date: Tue, 11 Jul 2023 12:20:31 -0500 Subject: [Tutor] Python script (beginner) Message-ID: <00b701d9b41b$fec72140$fc5563c0$@usnx.com> Good afternoon, I am trying to run a script that converts jpegs to pdfs. I currently have installed the img2pdf command line tool. My script will still not execute. Any help will be greatly appreciated. I have a attched a snip of the output. Derrick Casher 601-956-4770 x227 www.usnx.com From alan.gauld at yahoo.co.uk Tue Jul 11 15:34:31 2023 From: alan.gauld at yahoo.co.uk (Alan Gauld) Date: Tue, 11 Jul 2023 20:34:31 +0100 Subject: [Tutor] Python script (beginner) In-Reply-To: <00b701d9b41b$fec72140$fc5563c0$@usnx.com> References: <00b701d9b41b$fec72140$fc5563c0$@usnx.com> Message-ID: On 11/07/2023 18:20, dcasher at usnx.com wrote: > I am trying to run a script that converts jpegs to pdfs. I currently have > installed the img2pdf command line tool. Does the command line tool work as you want it to from the command line? If not that's the first step. > My script will still not execute. We need to see the script to be able to comment. And if there are any error messages please podt them in full too. > Any help will be greatly appreciated. I have a attched a snip of the > output. The output is probably of limited value but we really need to see code and any errors. Also, we can't see screenshots because the server will strip them as potential security threats. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos From o1bigtenor at gmail.com Tue Jul 11 08:06:01 2023 From: o1bigtenor at gmail.com (o1bigtenor) Date: Tue, 11 Jul 2023 07:06:01 -0500 Subject: [Tutor] looking but not finding Message-ID: Greetings I am new to programming (although I've been using computers for quite a while) - - - have a project. Situation - have a fluid collection system - fluid system is on a load cell system want to not only weigh system say every 0.5 sec (I know that's slow in electronics) but I want to record that information (as well as other information around it). I've tried looking for data log software - - - not much luck so far except for commercial stuff. Tried looking to see if there were python libraries that could be manipulated into service - - - not finding much joy either - - - - is there such software out there - - libraries? (Running on linux if that makes a difference.) TIA From alan.gauld at yahoo.co.uk Tue Jul 11 16:21:38 2023 From: alan.gauld at yahoo.co.uk (Alan Gauld) Date: Tue, 11 Jul 2023 21:21:38 +0100 Subject: [Tutor] looking but not finding In-Reply-To: References: Message-ID: On 11/07/2023 13:06, o1bigtenor wrote: > Greetings > > I am new to programming (although I've been using computers for quite > a while) - - - have a project. > > Situation > - have a fluid collection system > - fluid system is on a load cell system I googled load call sysem and it seems to be a hardware based system, possibly proprietary. I couldn't find any details on how to access it via software. You will need to give us much more information about how you connect your python code to the sensors? > but I want to record that information (as well as other information around it). > > I've tried looking for data log software I would expect the hardware supplier to have some kind of libraries - possibly in C but that's OK Python can probably access those. Alternatively they may provide command line tools (which OS?) and again Python can probably drive those. But without more details on the interface you will need to hope somebody here has prior experience and thats a long shot! Better if you can describe it or at least provide a link to a description. (But thats asking a lot from volunteer helpers!) > stuff. Tried looking to see if there were python libraries that could > be manipulated into service We need to know more. There are lots of possibilities from low level IP sockets to http/html, or JSON etc If its an industry standard such as HPIB then there may well be existing high level libraries. But we don't know your hardware setup. > (Running on linux if that makes a difference.) Yes the OS will be very significant here. The distribution might matter too. Also the Python version you are using. The closer to the hardware you get the more specific the details need to be! -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos From o1bigtenor at gmail.com Tue Jul 11 17:08:54 2023 From: o1bigtenor at gmail.com (o1bigtenor) Date: Tue, 11 Jul 2023 16:08:54 -0500 Subject: [Tutor] looking but not finding In-Reply-To: References: Message-ID: On Tue, Jul 11, 2023 at 3:22?PM Alan Gauld via Tutor wrote: > > On 11/07/2023 13:06, o1bigtenor wrote: > > Greetings > > > > I am new to programming (although I've been using computers for quite > > a while) - - - have a project. > > > > Situation > > - have a fluid collection system > > - fluid system is on a load cell system > > I googled load call sysem and it seems to be a hardware based system, > possibly proprietary. I couldn't find any details on how to access it > via software. You will need to give us much more information about how > you connect your python code to the sensors? > The load cell system does NOT have to be proprietary. Your response is headed in quite a different direction than my question. > > > but I want to record that information (as well as other information around it). > > > > I've tried looking for data log software > > I would expect the hardware supplier to have some kind of libraries - > possibly in C but that's OK Python can probably access those. > Alternatively they may provide command line tools (which OS?) > and again Python can probably drive those. > > But without more details on the interface you will need to hope > somebody here has prior experience and thats a long shot! Better > if you can describe it or at least provide a link to a description. > (But thats asking a lot from volunteer helpers!) What I specifically need assistance on is software that enables one to log data. What I find mountainous reams of information is on syslog - - - useful - - - - but when it comes to logging data - - - - well - - - quite NOT. > > > stuff. Tried looking to see if there were python libraries that could > > be manipulated into service > > We need to know more. There are lots of possibilities from low level > IP sockets to http/html, or JSON etc If its an industry standard > such as HPIB then there may well be existing high level libraries. > But we don't know your hardware setup. Hardware should be immaterial. Sensors are available for a plethora of measurables. I want to grab a value from sensor A and deposit that value in a data log over there. If you want I could give you a list of all the different kinds of sensors that I'm dealing with but that would be obfuscating the point. I understand how to read the sensors (there are a number of different ways depending upon a number of different factors as you intimate - - - but that isn't what I am asking about) - - - I think - - - problems here would go to a much more hardware slanted list - - - - not software. > > > (Running on linux if that makes a difference.) > > Yes the OS will be very significant here. > The distribution might matter too. > Also the Python version you are using. AFAIK right now it would be Python 3.11.x - - - I lose track of the exact version just know that ists 3.11 at this point. > > The closer to the hardware you get the more specific > the details need to be! > The Python info https://docs.python.org/3/howto/logging.html is quite slanted toward syslog although it does not state that. I was thinking that someone just might have produced something like pyplot for producing graphs or scipy which gets close to what I want in its statistics section (for the optimisation routines for a PID controller I think I would find a way to call a fortran routine that I've found that is scary good instead of using Python as in scipy) but I haven't been able to find anything when I include the term data log (logger/logging etc). What I'm storing - - - 1. item identification 2. time/date (routine is called every 0.5 seconds at this point in the planning) 3. value (from the weighing sensor system) above is data most likely shipping over tcp/ip. Its possible that data logging software is a 'roll your own everytime' kind of function but I was hoping that something better than (https://docs.python.org/3/howto/logging.html) is available. After all - - - - any time there is a process there needs to be a way to monitor said process - - - - and I would think its sorta normal to store at least some of the values - - - - that being the core value of a data logging system - - - now if I could only find some kind of python libraries that could help get me close to such - - - - that would be great. Regards From trent.shipley at gmail.com Tue Jul 11 17:26:42 2023 From: trent.shipley at gmail.com (trent shipley) Date: Tue, 11 Jul 2023 14:26:42 -0700 Subject: [Tutor] ModuleNotFoundError in Terminal, PyCharm OK In-Reply-To: <8b906754-77ca-4167-8384-3dbde839b9d0@DancesWithMice.info> References: <8b906754-77ca-4167-8384-3dbde839b9d0@DancesWithMice.info> Message-ID: It took me forever to figure out I could use the Python path to call the executable on the command line instead of the file path. That was after getting an error message which led to the great insight that all Python package examples were singly rooted with a single source--while I was trying to have sibling src and tests directories. I would up using setuptools in an effort to get absolute Python path names to work. I'm not sure it was needed, but absolute Python path names for modules are now working. Please confirm the setuptools build artifacts should go in .gitignore. Kivy has been provisionally selected as my GUI framework. I'm working through the documentation and tutorials prior to jumping into the RPG-GUI version. (There is no demand for that. The CLI suits my needs, but the RPG-GUI is a personal stretch goal.) (venv) trent at trent-virtual-machine:~/pythonworkspace/hackable_dice_roller$ python3 -m src.cli.shdroll --base 1 --sides 6 --dice 3 --rolls 3 d6_0 d6_1 d6_2 3_*_d6_roll_total grand_total 0 1 4 4 9 37 1 1 6 4 11 37 2 6 6 5 17 37 (venv) trent at trent-virtual-machine:~/pythonworkspace/hackable_dice_roller$ tree --gitignore . ??? build ? ??? bdist.linux-x86_64 ? ??? lib ? ??? api ? ? ??? core.py ? ? ??? __init__.py ? ??? cli ? ? ??? __init__.py ? ? ??? shdroll_cli_parser.py ? ? ??? shdroll.py ? ??? gui ? ? ??? __init__.py ? ? ??? simple_gui ? ? ??? __init_.py ? ? ??? __init__.py ? ??? tests ? ??? api ? ? ??? __init__.py ? ? ??? test_hdr_core.py ? ? ??? test_hdr_core_visually.py ? ??? cli ? ? ??? __init__.py ? ? ??? test_shdroll_visual.py ? ??? gui ? ? ??? __init__.py ? ? ??? simple_gui ? ? ??? __init__.py ? ??? __init__.py ??? dist ? ??? src-0.0.1-py3.11.egg ? ??? src-0.0.1-py3-none-any.whl ? ??? src-0.0.1.tar.gz ??? hackable_dice_roller.toml ??? README ??? setup.cfg ??? setup.py ??? src ??? api ? ??? core.py ? ??? __init__.py ? ??? __pycache__ ??? cli ? ??? __init__.py ? ??? __pycache__ ? ??? shdroll_cli_parser.py ? ??? shdroll.py ??? gui ? ??? __init__.py ? ??? __pycache__ ? ??? simple_gui ? ??? __init__.py ? ??? __pycache__ ??? __init__.py ??? __pycache__ ??? src.egg-info ? ??? dependency_links.txt ? ??? PKG-INFO ? ??? SOURCES.txt ? ??? top_level.txt ??? tests ??? api ? ??? __init__.py ? ??? __pycache__ ? ??? test_hdr_core.py ? ??? test_hdr_core_visually.py ??? cli ? ??? __init__.py ? ??? __pycache__ ? ??? test.csv ? ??? test_shdroll_visual.py ? ??? test.xlsx ??? gui ? ??? __init__.py ? ??? simple_gui ? ??? __init__.py ??? __init__.py ??? __pycache__ 32 directories, 46 files On Wed, Jun 28, 2023 at 3:00?PM dn via Tutor wrote: > On 29/06/2023 08.53, trent shipley wrote: > > Hi dn, > > > > #1 was not very useful, but #5 > > > https://docs.python.org/3/reference/import.html#:~:text=5.4.5.-,module.__path__,search%20for%20modules%20during%20import > . > > was > > quite useful > > > > I discovered that bash had no PYTHONPATH environment variable, and that > > Did you discover this by looking from BASH, or did you use Python (while > it is running) to survey the paths? > > (Python sets it up when the interpreter starts) > > > > PyCharm had obligingly intelligently populated it. (And if I changed the > > file structure, I had to manually inform PyCharm I had changed it, even > if > > I used the "refactor" feature.) > > > > I wound up with this: > > > > # shdroll.py > > import sys > > sys.path.extend(['../../..', # Important > > '../../../src', # Stuff > > '../../../src/core']) # Here > > from simple_hdroll_cli_parser import SimpleHDRollCliParser > > from src.core import core > > > > parse_cli_args = SimpleHDRollCliParser() > > kwargs = parse_cli_args.parse() > > > > if kwargs.add is not None: > > transform = core.add_currying(kwargs.add) > > elif kwargs.mult is not None: > > transform = core.multiply_currying(kwargs.mult) > > else: > > transform = None # not strictly necessary, but it makes my brain > hurt > > less > > > > # Create a die > > die = core.IntegerDie(transform_fn=transform, > > sides=kwargs.sides, > > bottom=kwargs.base) > > > > > > * > > > > * > > > > * > > > > > > > > print(rolls.to_string()) > > > > ------- > > > > I also wound up with a src.py file in the ./hackable_dice_roller/src > > directory so it would stop complaining about a missing module in the src > > package. src.py has one line. > > > > from src.core import core > > > > Was that what you had in mind, or did I come up with a singularly ugly > and > > unPythonic solution? > > Yes it does seem ugly, doesn't it. However, that is a good sign. The > very next sentence should be a question: "is there a better way?". > > Congratulations! (?) You are now ready to discuss Modules > (https://docs.python.org/3/tutorial/modules.html) which will lead into > "packages", and the exact scenario under discussion... > > NB whilst much discussion centers around mechanisms to deliver (your) > libraries to A.N.Other, you can also be your own 'customer'. > > > That said, there does seem to be many 'layers' of directories. If you > suspect so, perhaps you could draw-up a chart to show how the code-units > have been separated/laid-out, to aid discussion? > > -- > Regards, > =dn > _______________________________________________ > Tutor maillist - Tutor at python.org > To unsubscribe or change subscription options: > https://mail.python.org/mailman/listinfo/tutor > From alan.gauld at yahoo.co.uk Tue Jul 11 21:14:15 2023 From: alan.gauld at yahoo.co.uk (Alan Gauld) Date: Wed, 12 Jul 2023 02:14:15 +0100 Subject: [Tutor] looking but not finding In-Reply-To: References: Message-ID: On 11/07/2023 22:08, o1bigtenor wrote: >>> - fluid system is on a load cell system >> >> I googled load call sysem and it seems to be a hardware based system, > The load cell system does NOT have to be proprietary. > > Your response is headed in quite a different direction than my question. Sorry, but it sounded like you were having trouble *recording* the data. It was not obvious that you had managed to read the data and were interested in storing it. Quite a different problem I agree. >>> but I want to record that information (as well as other information around it). >>> >>> I've tried looking for data log software There are many, many different types of storage available from structured files(shelves, JSON, YAML, XML, etc to databases both SQL and NoSQL based. Much depends on what you want to do with the data and how regular it is. If its regular a SQL database may be the best option - SQLite comes with python and is lightweight and easily portable. If the data is irregular then a NoSQL database like Mongo may be better. But if you only need the storage for persistence then a structured file would be simpler. > What I specifically need assistance on is software that enables one to log > data. What I find mountainous reams of information is on syslog - - - > useful - - - - but when it comes to logging data - - - - well - - - quite NOT. The logging module and its cousins are intended for logging program progress/output/errors etc not for storing data. You could use them for that but its not their designed purpose. >>> stuff. Tried looking to see if there were python libraries that could >>> be manipulated into service It sounds like you either want to persist data between program runs or to store data for processing/analysis. The former is probably best met with a structured file format such as JSON. (Or even good old CSV!) If you want to slice/dice and perform calculations on the data, especially if it's a large volume, you may be better with a proper database - it's what they are designed for after all. If the data is regular go with SQL if irregular go NoSQL. > Hardware should be immaterial. > Sensors are available for a plethora of measurables. > I want to grab a value from sensor A and deposit that value in a > data log over there. OK, but that can be as simple as writing a string to a file. Or it can mean a complex SQL query into a database. We don't know what your data looks like or what you intend to do with it, or how much data there is to store. (We have a clue in the data capture rate of 2 samples per second (but from how many sensors?) but we don't know the length of a measurement period - minutes, hours, days, "infinite"? If there are several sensor types are the sensor data similar for each? Or do you need a separate data format(table or file?) for each sensor type 9or even each sensor!) And if that's the case do you need to combine those data later for analysis, based on which keys? Also will concurrent reading/writing be required? ie Will there be reporting/analysis being performed at the same time as the sensors are being read and their data stored? What about locking issues? There are a myriad factors to consider when choosing an appropriate storage mechanism. > I understand how to read the sensors That was the bit that was not apparent in your original message, and usually the hardest bit by far! > ... problems here would go to a much more > hardware slanted list - - - - not software. We have had several hardware interface discussions on the tutor list. Often involving USB or RS232 serial links and such. The list caters for experienced programmers new to Python as well as new programmers. > The Python info https://docs.python.org/3/howto/logging.html is quite > slanted toward syslog although it does not state that. I agree, I don't think that's the best route for what I believe you want. Logging is quite a well defined operation in software engineering so I guess the authors of the docs just assumed folks would think in terms of system logs. > someone just might have produced something like pyplot for producing > graphs or scipy which gets close to what I want in its statistics section But this is a completely different issue again. This is moving into how you analyse and report on the data you have stored. That's a completely different toolset. You essentially have at least 3 separate problems: 1) get the raw data from the sensor 2) format it and store it 3) analyze it and report the results It sounds like you have cracked number 1. I've discussed some options for 2 but maybe you have cracked that too? For number 3 you've already mentioned the obvious sources with Pandas and RPy being perhaps the most obvious candidates for analysis and pyplot (or maybe gnuplot) for generating graphical output. (But there are many other options from using Excel to building SVG graphics in a web page) > (for the optimisation routines for a PID controller I think I would find a > way to call a fortran routine that I've found that is scary good instead of > using Python as in scipy) I'm not sure where the optimisation fits in your workflow - is that before or after the analysis/reporting bit? Perhaps based on the analysis results? Calling external routines is certainly possible, either directly from Python (have a look at ctypes as one possibility) or via command line utilities. > but I haven't been able to find anything when > I include the term data log (logger/logging etc). I suspect what you are trying to do is not what most programmers would think of as logging. It sounds more like data storage. > What I'm storing - - - > 1. item identification > 2. time/date (routine is called every 0.5 seconds at this point in the planning) > 3. value (from the weighing sensor system) > > above is data most likely shipping over tcp/ip. See above. That looks simple and regular enough that a simple formatted text string is all that is needed. But for potential future complexity(*) I'd suggest that a CSV or JSON file would be better for persistent storage. A SQL database could handle it easily too if you need more complex processing of the data post storage, or concurrent access. (*)From my experience of building telemetry applications it isn't long before people want geographical location, and error detection/correction keys etc. The amount of data per event tends to grow over time. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos From PythonList at DancesWithMice.info Tue Jul 11 22:28:28 2023 From: PythonList at DancesWithMice.info (dn) Date: Wed, 12 Jul 2023 14:28:28 +1200 Subject: [Tutor] ModuleNotFoundError in Terminal, PyCharm OK In-Reply-To: References: <8b906754-77ca-4167-8384-3dbde839b9d0@DancesWithMice.info> Message-ID: <4f4cf834-70d5-2568-05b0-5c22ed74c62a@DancesWithMice.info> Not using setuptools myself, am not competent to respond. Perhaps A.N.Other will offer guidance? Am glad to see that the project is coming-together - that you have achieved success(es) in some areas, and have a plan to continue (and continue learning). Good one! On 12/07/2023 09.26, trent shipley wrote: > It took me forever to figure out I could use the Python path to call the > executable on the command line instead of the file path. That was after > getting an error message which led to the great insight that all Python > package examples were singly rooted with a single source--while I was > trying to have sibling src and tests directories. I would up using > setuptools in an effort to get absolute Python path names to work. I'm not > sure it was needed, but absolute Python path names for modules are now > working. > > Please confirm the setuptools build artifacts should go in .gitignore. > > Kivy has been provisionally selected as my GUI framework. I'm working > through the documentation and tutorials prior to jumping into the RPG-GUI > version. (There is no demand for that. The CLI suits my needs, but the > RPG-GUI is a personal stretch goal.) > > (venv) trent at trent-virtual-machine:~/pythonworkspace/hackable_dice_roller$ > python3 -m src.cli.shdroll --base 1 --sides 6 --dice 3 --rolls 3 > d6_0 d6_1 d6_2 3_*_d6_roll_total grand_total > 0 1 4 4 9 37 > 1 1 6 4 11 37 > 2 6 6 5 17 37 > (venv) trent at trent-virtual-machine:~/pythonworkspace/hackable_dice_roller$ > tree --gitignore > . > ??? build > ? ??? bdist.linux-x86_64 > ? ??? lib > ? ??? api > ? ? ??? core.py > ? ? ??? __init__.py > ? ??? cli > ? ? ??? __init__.py > ? ? ??? shdroll_cli_parser.py > ? ? ??? shdroll.py > ? ??? gui > ? ? ??? __init__.py > ? ? ??? simple_gui > ? ? ??? __init_.py > ? ? ??? __init__.py > ? ??? tests > ? ??? api > ? ? ??? __init__.py > ? ? ??? test_hdr_core.py > ? ? ??? test_hdr_core_visually.py > ? ??? cli > ? ? ??? __init__.py > ? ? ??? test_shdroll_visual.py > ? ??? gui > ? ? ??? __init__.py > ? ? ??? simple_gui > ? ? ??? __init__.py > ? ??? __init__.py > ??? dist > ? ??? src-0.0.1-py3.11.egg > ? ??? src-0.0.1-py3-none-any.whl > ? ??? src-0.0.1.tar.gz > ??? hackable_dice_roller.toml > ??? README > ??? setup.cfg > ??? setup.py > ??? src > ??? api > ? ??? core.py > ? ??? __init__.py > ? ??? __pycache__ > ??? cli > ? ??? __init__.py > ? ??? __pycache__ > ? ??? shdroll_cli_parser.py > ? ??? shdroll.py > ??? gui > ? ??? __init__.py > ? ??? __pycache__ > ? ??? simple_gui > ? ??? __init__.py > ? ??? __pycache__ > ??? __init__.py > ??? __pycache__ > ??? src.egg-info > ? ??? dependency_links.txt > ? ??? PKG-INFO > ? ??? SOURCES.txt > ? ??? top_level.txt > ??? tests > ??? api > ? ??? __init__.py > ? ??? __pycache__ > ? ??? test_hdr_core.py > ? ??? test_hdr_core_visually.py > ??? cli > ? ??? __init__.py > ? ??? __pycache__ > ? ??? test.csv > ? ??? test_shdroll_visual.py > ? ??? test.xlsx > ??? gui > ? ??? __init__.py > ? ??? simple_gui > ? ??? __init__.py > ??? __init__.py > ??? __pycache__ > > 32 directories, 46 files > > On Wed, Jun 28, 2023 at 3:00?PM dn via Tutor wrote: > >> On 29/06/2023 08.53, trent shipley wrote: >>> Hi dn, >>> >>> #1 was not very useful, but #5 >>> >> https://docs.python.org/3/reference/import.html#:~:text=5.4.5.-,module.__path__,search%20for%20modules%20during%20import >> . >>> was >>> quite useful >>> >>> I discovered that bash had no PYTHONPATH environment variable, and that >> >> Did you discover this by looking from BASH, or did you use Python (while >> it is running) to survey the paths? >> >> (Python sets it up when the interpreter starts) >> >> >>> PyCharm had obligingly intelligently populated it. (And if I changed the >>> file structure, I had to manually inform PyCharm I had changed it, even >> if >>> I used the "refactor" feature.) >>> >>> I wound up with this: >>> >>> # shdroll.py >>> import sys >>> sys.path.extend(['../../..', # Important >>> '../../../src', # Stuff >>> '../../../src/core']) # Here >>> from simple_hdroll_cli_parser import SimpleHDRollCliParser >>> from src.core import core >>> >>> parse_cli_args = SimpleHDRollCliParser() >>> kwargs = parse_cli_args.parse() >>> >>> if kwargs.add is not None: >>> transform = core.add_currying(kwargs.add) >>> elif kwargs.mult is not None: >>> transform = core.multiply_currying(kwargs.mult) >>> else: >>> transform = None # not strictly necessary, but it makes my brain >> hurt >>> less >>> >>> # Create a die >>> die = core.IntegerDie(transform_fn=transform, >>> sides=kwargs.sides, >>> bottom=kwargs.base) >>> >>> >>> * >>> >>> * >>> >>> * >>> >>> >>> >>> print(rolls.to_string()) >>> >>> ------- >>> >>> I also wound up with a src.py file in the ./hackable_dice_roller/src >>> directory so it would stop complaining about a missing module in the src >>> package. src.py has one line. >>> >>> from src.core import core >>> >>> Was that what you had in mind, or did I come up with a singularly ugly >> and >>> unPythonic solution? >> >> Yes it does seem ugly, doesn't it. However, that is a good sign. The >> very next sentence should be a question: "is there a better way?". >> >> Congratulations! (?) You are now ready to discuss Modules >> (https://docs.python.org/3/tutorial/modules.html) which will lead into >> "packages", and the exact scenario under discussion... >> >> NB whilst much discussion centers around mechanisms to deliver (your) >> libraries to A.N.Other, you can also be your own 'customer'. >> >> >> That said, there does seem to be many 'layers' of directories. If you >> suspect so, perhaps you could draw-up a chart to show how the code-units >> have been separated/laid-out, to aid discussion? >> >> -- >> Regards, >> =dn >> _______________________________________________ >> Tutor maillist - Tutor at python.org >> To unsubscribe or change subscription options: >> https://mail.python.org/mailman/listinfo/tutor >> > _______________________________________________ > Tutor maillist - Tutor at python.org > To unsubscribe or change subscription options: > https://mail.python.org/mailman/listinfo/tutor -- Regards, =dn From mats at wichmann.us Wed Jul 12 09:14:10 2023 From: mats at wichmann.us (Mats Wichmann) Date: Wed, 12 Jul 2023 07:14:10 -0600 Subject: [Tutor] looking but not finding In-Reply-To: References: Message-ID: On 7/11/23 15:08, o1bigtenor wrote: > What I specifically need assistance on is software that enables one to log > data. What I find mountainous reams of information is on syslog - - - > useful - - - - but when it comes to logging data - - - - well - - - quite NOT. ... > The Python info https://docs.python.org/3/howto/logging.html is quite > slanted toward syslog although it does not state that. I was thinking that > someone just might have produced something like pyplot for producing > graphs or scipy which gets close to what I want in its statistics section > (for the optimisation routines for a PID controller I think I would find a > way to call a fortran routine that I've found that is scary good instead of > using Python as in scipy) but I haven't been able to find anything when > I include the term data log (logger/logging etc). > > What I'm storing - - - > 1. item identification > 2. time/date (routine is called every 0.5 seconds at this point in the planning) > 3. value (from the weighing sensor system) Well, a quick glance turns up a fair bit of material. Here's something that sounds like what you're talking about: http://www.steves-internet-guide.com/simple-python-mqtt-data-logger/ It's written from an IoT prespective, so rather than raw sensor data, he's dealing with smart enough "things" that they can report in a structured way over a data bus, but it's the logging concepts you were asking about. I believe there's a followon article in the series that shows using a database, rather than a plain file, for logging. Here's a project that's more on the analysis side: https://github.com/whylabs/whylogs The world is full of tools for data analysis and visualization so I will assume you can find those if you don't want to roll your own (matplotlib is probably the "gold standard" for plotting - pyplot is just an interface to that). From mats at wichmann.us Wed Jul 12 12:40:28 2023 From: mats at wichmann.us (Mats Wichmann) Date: Wed, 12 Jul 2023 10:40:28 -0600 Subject: [Tutor] ModuleNotFoundError in Terminal, PyCharm OK In-Reply-To: References: <8b906754-77ca-4167-8384-3dbde839b9d0@DancesWithMice.info> Message-ID: On 7/11/23 15:26, trent shipley wrote: > Please confirm the setuptools build artifacts should go in .gitignore. What goes in .gitignore has no absolute answer - it's what you don't want git to track by default. Convention is things that will be regenerated are not checked in, and you set those to ignore so you don't get nagged about files that are present but not tracked. For Python code, a good example is the saved bytecode. $ git status ... Untracked files: (use "git add ..." to include in what will be committed) ... From o1bigtenor at gmail.com Wed Jul 12 10:36:42 2023 From: o1bigtenor at gmail.com (o1bigtenor) Date: Wed, 12 Jul 2023 09:36:42 -0500 Subject: [Tutor] looking but not finding In-Reply-To: References: Message-ID: On Wed, Jul 12, 2023 at 8:20?AM Mats Wichmann wrote: > > On 7/11/23 15:08, o1bigtenor wrote: > > > What I specifically need assistance on is software that enables one to log > > data. What I find mountainous reams of information is on syslog - - - > > useful - - - - but when it comes to logging data - - - - well - - - quite NOT. > > ... > > > The Python info https://docs.python.org/3/howto/logging.html is quite > > slanted toward syslog although it does not state that. I was thinking that > > someone just might have produced something like pyplot for producing > > graphs or scipy which gets close to what I want in its statistics section > > (for the optimisation routines for a PID controller I think I would find a > > way to call a fortran routine that I've found that is scary good instead of > > using Python as in scipy) but I haven't been able to find anything when > > I include the term data log (logger/logging etc). > > > > What I'm storing - - - > > 1. item identification > > 2. time/date (routine is called every 0.5 seconds at this point in the planning) > > 3. value (from the weighing sensor system) > > Well, a quick glance turns up a fair bit of material. Here's something > that sounds like what you're talking about: > > http://www.steves-internet-guide.com/simple-python-mqtt-data-logger/ dunno about finding 'a fair bit of material' - - - I seem to have been using the wrong search terms. > > It's written from an IoT prespective, so rather than raw sensor data, > he's dealing with smart enough "things" that they can report in a > structured way over a data bus, but it's the logging concepts you were > asking about. I believe there's a followon article in the series that > shows using a database, rather than a plain file, for logging. This is in the right direction but that follow article doesn not address a number of the points that mr Alan makes. (see my response to his note) > > Here's a project that's more on the analysis side: > > https://github.com/whylabs/whylogs > > The world is full of tools for data analysis and visualization so I will > assume you can find those if you don't want to roll your own (matplotlib > is probably the "gold standard" for plotting - pyplot is just an > interface to that). > Agreed - - - this is an area where I have not trouble finding ideas or techniques its that mid point of not only grabbing the sensor info but also stashing it! Thanks for the pointers! From o1bigtenor at gmail.com Wed Jul 12 11:26:34 2023 From: o1bigtenor at gmail.com (o1bigtenor) Date: Wed, 12 Jul 2023 10:26:34 -0500 Subject: [Tutor] looking but not finding In-Reply-To: References: Message-ID: Thank you for your patience mr Alan. What I'm finding is that the computing world uses terms in a different way than does the sensor world than does the storage world than does the rest of us. That seriously complicates things. (Details in the following.) On Tue, Jul 11, 2023 at 8:15?PM Alan Gauld via Tutor wrote: > > On 11/07/2023 22:08, o1bigtenor wrote: > > >>> - fluid system is on a load cell system > >> > >> I googled load call sysem and it seems to be a hardware based system, > > The load cell system does NOT have to be proprietary. > > > > Your response is headed in quite a different direction than my question. > > Sorry, but it sounded like you were having trouble *recording* the data. > It was not obvious that you had managed to read the data and were > interested in storing it. Quite a different problem I agree. > > >>> but I want to record that information (as well as other information around it). > >>> > >>> I've tried looking for data log software > > There are many, many different types of storage available > from structured files(shelves, JSON, YAML, XML, etc to > databases both SQL and NoSQL based. This is where I start getting 'lost'. I have no idea which of the three listed formats will work well with what I'm doing - - - dunno even how to find out either! > > Much depends on what you want to do with the data and > how regular it is. If its regular a SQL database may be > the best option - SQLite comes with python and is > lightweight and easily portable. The first point with the data is to store it. At the time of its collection the data will also be used to trigger other events so that will be (I think that is the proper place) be done in the capturing/organizing/supervising program (in python). > > If the data is irregular then a NoSQL database like > Mongo may be better. But if you only need the storage > for persistence then a structured file would be simpler. Went looking at MongoDB - - - honestly cannot tell what makes data 'irregular' - - - - perhaps you might advise - - - I cannot find a reasonable definition - - - -yet! I am going to try to delineate at least some of what I'm thinking. (Each point has data associated with it.) 1. reset weighing system 2. data is captured from the identification sub-system 3. data is captured from the calendar system (date/time) 4. (this one is not in initial plans but possibly in later test vacuum in system for level) 5. get weighing system value 6. zero value before starting measurement 7. poll sensor every 0.5 sec using step $8 if there is a change from #6 8. (assuming change in sensor value - - store data 9. when difference between values shows that the value change is less than 250 g/min continue for another 20 seconds 10. after #9 is fulfilled issue 'final weight' 11. 2 seconds (may change) after #10 open valve #1 12. after 30 seconds (this value may change) close valve #1 13. after valve #1 closes over valve #2 connecting system to wash/rinse system 14. 30 sec after valve #2 is opened valve #3 is opened (This may be opened to the rinse system at the same time as valve #2 is design not yet complete - - - needs fabbing and testing!!!) 15. 5 seconds after rinse is complete valves connecting to the rinse system (and its drain) are closed. 16. see #1 Each weighing subsystem will have its own control. There will be multiple such subsystems (initially at least 3 more likely 12 - - - further (future) - - - up to a couple hundred all operating at the same time). > > > What I specifically need assistance on is software that enables one to log > > data. What I find mountainous reams of information is on syslog - - - > > useful - - - - but when it comes to logging data - - - - well - - - quite NOT. > > The logging module and its cousins are intended for > logging program progress/output/errors etc not for > storing data. You could use them for that but its > not their designed purpose. > > >>> stuff. Tried looking to see if there were python libraries that could > >>> be manipulated into service > > It sounds like you either want to persist data between > program runs or to store data for processing/analysis. > The former is probably best met with a structured file > format such as JSON. (Or even good old CSV!) > > If you want to slice/dice and perform calculations on > the data, especially if it's a large volume, you may be > better with a proper database - it's what they are > designed for after all. > > If the data is regular go with SQL if irregular go NoSQL. Now if I only knew what regular and/or irregular data was. Have been considering using Postgresql as a storage engine. AFAIK it has the horsepower to deal with serious large amounts of data. This weighing routine will be running for up to 3.5 minutes. That doesn't sound like much - - - some 400 'chunks' (meaning all the data that comprises what is being stored) of data. (Then there may be a repeat of this function every 4.5 to 6 minutes for say up to 180 minutes - - - multiple times (varying up to 3x) per day. Its not serious huge data amounts but its definitely non- trivial. > > > Hardware should be immaterial. > > Sensors are available for a plethora of measurables. > > I want to grab a value from sensor A and deposit that value in a > > data log over there. > > OK, but that can be as simple as writing a string to a file. > Or it can mean a complex SQL query into a database. We don't > know what your data looks like or what you intend to do > with it, or how much data there is to store. (We have a > clue in the data capture rate of 2 samples per second > (but from how many sensors?) but we don't know the length > of a measurement period - minutes, hours, days, "infinite"? Sorry - - - previous - - - > > If there are several sensor types are the sensor data similar > for each? Or do you need a separate data format(table or file?) > for each sensor type 9or even each sensor!) And if that's > the case do you need to combine those data later for analysis, > based on which keys? This is part of what I hope I was able to answer in my list (1-16). > > Also will concurrent reading/writing be required? ie Will > there be reporting/analysis being performed at the same > time as the sensors are being read and their data stored? > What about locking issues? Here you have opened another pandora's box for me. Dunno much about any of this. Concurrent - - - yes, there are multiple sensors per station and multiple stations and hopefully they're all working (problems with the system if they're not!) There is a small amount of analysis being done as the data is stored but that's to drive the system (ie at point x you stop this, at point y you do this, point y does this for z time then a happens etc - - - I do not consider that heavy duty analysis. That happens on a different system (storage happens first to the local governing system. After a round of data (one cycle) has completed that data is shipped in a burst to the long term storage system. (there may be a cascade of storage systems but I haven't got that far yet!) This system is quite capable of handling a LOT of information - - - or that's the goal anyway. Maybe One system will store the info and another will do analysis and then return that analysis to the storing system for accumulation - - - also not decided.) Locking - - - how, when, where and why - - - the why I think I understand ( to make sure data doesn't get corrupted) but the rest - - - would love some guidance on. > > There are a myriad factors to consider when choosing > an appropriate storage mechanism. > > > I understand how to read the sensors > > That was the bit that was not apparent in your original > message, and usually the hardest bit by far! > > > ... problems here would go to a much more > > hardware slanted list - - - - not software. > > We have had several hardware interface discussions on > the tutor list. Often involving USB or RS232 serial > links and such. The list caters for experienced programmers > new to Python as well as new programmers. Hmmmmmmmmmmm - - - Modbus, Probus and RS-485 are all being considered - - - likely using a tcp/ip backbone. > > > The Python info https://docs.python.org/3/howto/logging.html is quite > > slanted toward syslog although it does not state that. > > I agree, I don't think that's the best route for > what I believe you want. Logging is quite a well defined > operation in software engineering so I guess the authors > of the docs just assumed folks would think in terms > of system logs. Can understand that but think that those functions might also be used for what I'm trying to do - - - but I'm really not sure how to. > > > someone just might have produced something like pyplot for producing > > graphs or scipy which gets close to what I want in its statistics section > > But this is a completely different issue again. This is > moving into how you analyse and report on the data > you have stored. That's a completely different toolset. snip The two mentioned items were examples of code (and/or analysis) - - - - not functions I wished to use. > > > but I haven't been able to find anything when > > I include the term data log (logger/logging etc). > > I suspect what you are trying to do is not what > most programmers would think of as logging. > It sounds more like data storage. > > > What I'm storing - - - > > 1. item identification > > 2. time/date (routine is called every 0.5 seconds at this point in the planning) > > 3. value (from the weighing sensor system) > > > > above is data most likely shipping over tcp/ip. > > See above. That looks simple and regular enough that > a simple formatted text string is all that is needed. > But for potential future complexity(*) I'd suggest > that a CSV or JSON file would be better for persistent > storage. A SQL database could handle it easily > too if you need more complex processing of the data > post storage, or concurrent access. > > (*)From my experience of building telemetry applications > it isn't long before people want geographical location, > and error detection/correction keys etc. The amount > of data per event tends to grow over time. > Oh yes - - - - I'm quite greedy with trying to get as much data as I can. There are a lot of uses for such and if its easy to put together then there are less excuses for the not using. I don't think I'll I'll be starting with csv - - - - really need to figure out what I should be using for what you termed 'structured files'. Do you have any suggestions as to where I find information that will help me figure out a good or a great way to do that part of things? (Choosing yaml/xml/???? for starters.) Thanking you for your patience and assistance. Regards From mats at wichmann.us Wed Jul 12 14:52:22 2023 From: mats at wichmann.us (Mats Wichmann) Date: Wed, 12 Jul 2023 12:52:22 -0600 Subject: [Tutor] looking but not finding In-Reply-To: References: Message-ID: <227607cb-4fb2-11cc-d3ec-54bff36ffbb5@wichmann.us> On 7/12/23 09:26, o1bigtenor wrote: > Thank you for your patience mr Alan. > > What I'm finding is that the computing world uses terms in a different way than > does the sensor world than does the storage world than does the rest of us. > That seriously complicates things. I'm guessing you have no idea how bad the jargon problem is... and it certainly doesn't help when terms are "overloaded" - given different special meanings by different groups. Logging: what you've found has to do with event logging - where a program wants to record things that have happened in the running of the program itself. What you're after is what some call data logging. You can even look for projects that self-declare being related to that term like this: https://github.com/topics/data-logging >> There are many, many different types of storage available >> from structured files(shelves, JSON, YAML, XML, etc to >> databases both SQL and NoSQL based. > This is where I start getting 'lost'. I have no idea which of the three > listed formats will work well with what I'm doing - - - dunno even how to > find out either! The storage format is something you might consider an implementation detail, that you don't have to worry about yet. The first steps are the ones you're taking: to flesh out what your requirements are, and move to a conceptual design. When you think you have a vision of what it will look like, you can see if some particular existing library provides an easy to use implementation of one of these that would fit your needs, with some room to group. If you design it right, the piece that drives that could also drive a different thing, if you found you chose poorly, or outgrew it. JSON, YAML and XML are text-based formats, with some markup to help both you as a human reader and in some cases, computer code, to decode it. If there's not a ton of data, and/or there's not a lot of interrelationship between different types of data (suggesting a "relational" database), one of these will probably be fine. XML is a lot chattier, and the XML libraries in Python are quite fiddly to use so recommend you cross XML off your candidate list unless there are really good reasons to needs its level of structure. You can even use comma-separated-values (like a spreadsheet program would dump put), which is quite inflexible - one line is one record, and every field has to be present in every record, you don't have the option of extra or omitted fields, or it just breaks. Pretty sure each one of those has an eloquent wikipedia page, e.g. https://en.wikipedia.org/wiki/JSON#Syntax If you want to use a database, rather than a text file, Alan laid out some choices. For reasonably small amounts of data, sqlite is usually fine. It's perhaps closer to the text-based interchange formats in that there's no separate database program that you communicate with, data is read from and written to a disk files in-process, but the interaction with that data is through SQL statements so it's also like the "big databases" - mysql/mariadb, SQL Server, Postgres. Then the "noSQL" databases (that's not a terribly precise term) are the ones that have less structure, to aid in scalability - a problem you're not likely to have from your description so won't say more here. From threesomequarks at proton.me Wed Jul 12 15:09:27 2023 From: threesomequarks at proton.me (ThreeBlindQuarks) Date: Wed, 12 Jul 2023 19:09:27 +0000 Subject: [Tutor] looking but not finding In-Reply-To: References: Message-ID: It looks like the data gathering algorithm is fairly complex using many parts and that it also has to compare recent to present values to decide what, if anything, needs to be recorded. My personal goal here is to help with some stumbling block the user has with the language rather than help them build it. So some smaller problems asked carefully and clearly might elicit a response. But can I ask a design question? Is this a long-running program that checks things for days or months at a time? Must it log things almost immediately, or can it buffer data internally and periodically write something out? What happens if things crash? Some designs may work better than others. Is anything else going to be reading these logs immediately, or can you make one set of logs that nobody reads that can later be massaged into a form suitable for sharing? A simple-minded approach may be to log everything using a design that drops one or more records that may be linked by something like the same date-time or sequence number. Don't worry if a measure has not changed and so on, just log it. A second program can (maybe later) read in the contents of a log and apply many kinds of logic such as ignoring repeated same results, and perhaps use the logged data to populate a database of whatever kind in ways that can be easily queried. As to how to save data, there are too many ways and for some purposes, simple trivial ones could be best. If you have a fixed number of simple fields in a single record, using some commas to add a line in a CSV file may be adequate. If you need to save multiple types of records, you could add them to different CSV as in tables, or have a wider record with room for many kinds of info and only populate the ones relevant for that item. Or, if your data effectively can be viewed as being encapsulated in some kind of object you build, as is done in some languages, you may actually convert it to JSON or the like and save that, albeit perhaps using other technologies that support object storage and retrieval. Again, a less broad question well described may get better answers. Sent with Proton Mail secure email. ------- Original Message ------- On Wednesday, July 12th, 2023 at 11:26 AM, o1bigtenor wrote: > Thank you for your patience mr Alan. > > What I'm finding is that the computing world uses terms in a different way than > does the sensor world than does the storage world than does the rest of us. > That seriously complicates things. (Details in the following.) > > On Tue, Jul 11, 2023 at 8:15?PM Alan Gauld via Tutor tutor at python.org wrote: > > > On 11/07/2023 22:08, o1bigtenor wrote: > > > > > > > - fluid system is on a load cell system > > > > > > > > I googled load call sysem and it seems to be a hardware based system, > > > > The load cell system does NOT have to be proprietary. > > > > > > Your response is headed in quite a different direction than my question. > > > > Sorry, but it sounded like you were having trouble recording the data. > > It was not obvious that you had managed to read the data and were > > interested in storing it. Quite a different problem I agree. > > > > > > > but I want to record that information (as well as other information around it). > > > > > > > > > > I've tried looking for data log software > > > > There are many, many different types of storage available > > from structured files(shelves, JSON, YAML, XML, etc to > > databases both SQL and NoSQL based. > > > This is where I start getting 'lost'. I have no idea which of the three > listed formats will work well with what I'm doing - - - dunno even how to > find out either! > > > Much depends on what you want to do with the data and > > how regular it is. If its regular a SQL database may be > > the best option - SQLite comes with python and is > > lightweight and easily portable. > > > The first point with the data is to store it. At the time of its collection > the data will also be used to trigger other events so that will be (I think > that is the proper place) be done in the capturing/organizing/supervising > program (in python). > > > If the data is irregular then a NoSQL database like > > Mongo may be better. But if you only need the storage > > for persistence then a structured file would be simpler. > > > Went looking at MongoDB - - - honestly cannot tell what makes data > 'irregular' - - - - perhaps you might advise - - - I cannot find a reasonable > definition - - - -yet! > > I am going to try to delineate at least some of what I'm thinking. > > (Each point has data associated with it.) > 1. reset weighing system > 2. data is captured from the identification sub-system > 3. data is captured from the calendar system (date/time) > 4. (this one is not in initial plans but possibly in later > test vacuum in system for level) > 5. get weighing system value > 6. zero value before starting measurement > 7. poll sensor every 0.5 sec using step $8 if there is a change from #6 > 8. (assuming change in sensor value - - store data > 9. when difference between values shows that the value change is less > than 250 g/min continue for another 20 seconds > 10. after #9 is fulfilled issue 'final weight' > 11. 2 seconds (may change) after #10 open valve #1 > 12. after 30 seconds (this value may change) close valve #1 > 13. after valve #1 closes over valve #2 connecting system to wash/rinse system > 14. 30 sec after valve #2 is opened valve #3 is opened > (This may be opened to the rinse system at the same time as valve #2 is > design not yet complete - - - needs fabbing and testing!!!) > 15. 5 seconds after rinse is complete valves connecting to the rinse system > (and its drain) are closed. > 16. see #1 > > Each weighing subsystem will have its own control. There will be multiple > such subsystems (initially at least 3 more likely 12 - - - further > (future) - - - up to > a couple hundred all operating at the same time). > > > > What I specifically need assistance on is software that enables one to log > > > data. What I find mountainous reams of information is on syslog - - - > > > useful - - - - but when it comes to logging data - - - - well - - - quite NOT. > > > > The logging module and its cousins are intended for > > logging program progress/output/errors etc not for > > storing data. You could use them for that but its > > not their designed purpose. > > > > > > > stuff. Tried looking to see if there were python libraries that could > > > > > be manipulated into service > > > > It sounds like you either want to persist data between > > program runs or to store data for processing/analysis. > > The former is probably best met with a structured file > > format such as JSON. (Or even good old CSV!) > > > > If you want to slice/dice and perform calculations on > > the data, especially if it's a large volume, you may be > > better with a proper database - it's what they are > > designed for after all. > > > > If the data is regular go with SQL if irregular go NoSQL. > > > Now if I only knew what regular and/or irregular data was. > Have been considering using Postgresql as a storage engine. > AFAIK it has the horsepower to deal with serious large amounts of data. > This weighing routine will be running for up to 3.5 minutes. That doesn't > sound like much - - - some 400 'chunks' (meaning all the data that comprises > what is being stored) of data. (Then there may be a repeat of this function > every 4.5 to 6 minutes for say up to 180 minutes - - - multiple times (varying > up to 3x) per day. Its not serious huge data amounts but its definitely non- > trivial. > > > > Hardware should be immaterial. > > > Sensors are available for a plethora of measurables. > > > I want to grab a value from sensor A and deposit that value in a > > > data log over there. > > > > OK, but that can be as simple as writing a string to a file. > > Or it can mean a complex SQL query into a database. We don't > > know what your data looks like or what you intend to do > > with it, or how much data there is to store. (We have a > > clue in the data capture rate of 2 samples per second > > (but from how many sensors?) but we don't know the length > > of a measurement period - minutes, hours, days, "infinite"? > > > Sorry - - - previous - - - > > > If there are several sensor types are the sensor data similar > > for each? Or do you need a separate data format(table or file?) > > for each sensor type 9or even each sensor!) And if that's > > the case do you need to combine those data later for analysis, > > based on which keys? > > > This is part of what I hope I was able to answer in my list (1-16). > > > Also will concurrent reading/writing be required? ie Will > > there be reporting/analysis being performed at the same > > time as the sensors are being read and their data stored? > > What about locking issues? > > > Here you have opened another pandora's box for me. > Dunno much about any of this. > Concurrent - - - yes, there are multiple sensors per station and > multiple stations and hopefully they're all working (problems > with the system if they're not!) > > There is a small amount of analysis being done as the data is stored > but that's to drive the system (ie at point x you stop this, at point > y you do this, point y does this for z time then a happens etc - - - I > do not consider that heavy duty analysis. That happens on a different > system (storage happens first to the local governing system. After > a round of data (one cycle) has completed that data is shipped in a burst > to the long term storage system. (there may be a cascade of storage systems > but I haven't got that far yet!) This system is quite capable of handling a LOT > of information - - - or that's the goal anyway. Maybe One system will store the > info and another will do analysis and then return that analysis to the storing > system for accumulation - - - also not decided.) > > Locking - - - how, when, where and why - - - the why I think I understand ( to > make sure data doesn't get corrupted) but the rest - - - would love some > guidance on. > > > There are a myriad factors to consider when choosing > > an appropriate storage mechanism. > > > > > I understand how to read the sensors > > > > That was the bit that was not apparent in your original > > message, and usually the hardest bit by far! > > > > > ... problems here would go to a much more > > > hardware slanted list - - - - not software. > > > > We have had several hardware interface discussions on > > the tutor list. Often involving USB or RS232 serial > > links and such. The list caters for experienced programmers > > new to Python as well as new programmers. > > > Hmmmmmmmmmmm - - - Modbus, Probus and RS-485 are all > being considered - - - likely using a tcp/ip backbone. > > > > The Python info https://docs.python.org/3/howto/logging.html is quite > > > slanted toward syslog although it does not state that. > > > > I agree, I don't think that's the best route for > > what I believe you want. Logging is quite a well defined > > operation in software engineering so I guess the authors > > of the docs just assumed folks would think in terms > > of system logs. > > > Can understand that but think that those functions might also be used > for what I'm trying to do - - - but I'm really not sure how to. > > > > someone just might have produced something like pyplot for producing > > > graphs or scipy which gets close to what I want in its statistics section > > > > But this is a completely different issue again. This is > > moving into how you analyse and report on the data > > you have stored. That's a completely different toolset. > > > snip > The two mentioned items were examples of code (and/or analysis) - - - - not > functions I wished to use. > > > > but I haven't been able to find anything when > > > I include the term data log (logger/logging etc). > > > > I suspect what you are trying to do is not what > > most programmers would think of as logging. > > It sounds more like data storage. > > > > > What I'm storing - - - > > > 1. item identification > > > 2. time/date (routine is called every 0.5 seconds at this point in the planning) > > > 3. value (from the weighing sensor system) > > > > > > above is data most likely shipping over tcp/ip. > > > > See above. That looks simple and regular enough that > > a simple formatted text string is all that is needed. > > But for potential future complexity(*) I'd suggest > > that a CSV or JSON file would be better for persistent > > storage. A SQL database could handle it easily > > too if you need more complex processing of the data > > post storage, or concurrent access. > > > > (*)From my experience of building telemetry applications > > it isn't long before people want geographical location, > > and error detection/correction keys etc. The amount > > of data per event tends to grow over time. > > Oh yes - - - - I'm quite greedy with trying to get as much data > as I can. There are a lot of uses for such and if its easy to put > together then there are less excuses for the not using. > > I don't think I'll I'll be starting with csv - - - - really need to figure out > what I should be using for what you termed 'structured files'. > Do you have any suggestions as to where I find information that > will help me figure out a good or a great way to do that part of things? > (Choosing yaml/xml/???? for starters.) > > Thanking you for your patience and assistance. > > Regards > _______________________________________________ > Tutor maillist - Tutor at python.org > To unsubscribe or change subscription options: > https://mail.python.org/mailman/listinfo/tutor From alan.gauld at yahoo.co.uk Wed Jul 12 19:49:59 2023 From: alan.gauld at yahoo.co.uk (Alan Gauld) Date: Thu, 13 Jul 2023 00:49:59 +0100 Subject: [Tutor] looking but not finding In-Reply-To: References: Message-ID: This is late at night so I won;t reply to all the points but I'll pick up a few easier ones! :-) On 12/07/2023 16:26, o1bigtenor wrote: > What I'm finding is that the computing world uses terms in a different way than > does the sensor world than does the storage world than does the rest of us. Sadly true. I come from an electronic/telecomms background but most of my career was in software engineering. SE is a very new field and doesn't have the well established vocabulary of more traditional engineering fields. Often one term can have multiple meanings depending on context. And other times multiple terms are used for the exact same thing. >> from structured files(shelves, JSON, YAML, XML, etc to >> databases both SQL and NoSQL based. > > This is where I start getting 'lost'. I have no idea which of the three > listed formats will work well I actually listed 4! :-) shelve is a Python proprietary format that will take arbitrary Python data structures and save them to a file that can then be accessed like a dictionary. Its very easy to use and works well if your data can be found via a simple key. For more general use it has issues. JSON, YAML, XML are all text based files using structured text to define the nature of the data. JSON looks a lot like a Python dictionary format. XML is good for complex data but is correspondingly complex to work with. YAML is somewhere in the middle. For your purposes JSON is probably a good choice. Wikipedia describes all of them in much more detail. > The first point with the data is to store it. At the time of its collection > the data will also be used to trigger other events so that will be (I think > that is the proper place) be done in the capturing/organizing/supervising > program (in python). That raises a whole bunch of other questions such as is the data stateful? Does the processing of sample Sn depend on the values of Sn-1, Sn-2... If so you need to store a "window" of data in memory and then write out the oldest item to storage as it expires. If not you can probably just read, process, store. > honestly cannot tell what makes data > 'irregular' It just means that different data items have different attributes. Some fields may be optional and different readings will contain different numbers of fields. Or the same fields may contain different types of data, for example one sensor reports on/off while another gives a magnitude and another gives a status message. That would be irregular data and SQL databases don't like it much (although there are techniques to get round it) Regular data just means every reading looks the same so you can define a table with a set number of columns each of a known type. > Now if I only knew what regular and/or irregular data was. > Have been considering using Postgresql as a storage engine. > AFAIK it has the horsepower to deal with serious large amounts of data. It will do the job but so would something much lighter like SQLite. The really critical factor is how much parallel access you need. SQLite is great if only a single program is reading and writing the data. Especially if there is only a single thread of execution. The server based databases like Postgres come into their own if you have multiple clients accessing the database at once. They can perform tricks like record level locking (rather than table level or even database level) during writes. > Its not serious huge data amounts but its definitely non- > trivial. Modern storage solutions have made even Gigabytes of data almost trivial. Big volumes would be more of an issue if using flat files because they need to be read from disk and that takes time for big volumes - even using SSDs. > Concurrent - - - yes, there are multiple sensors per station and > multiple stations and hopefully they're all working (problems > with the system if they're not!) You can deal with a relatively small number of sensors in a single threaded Python application, just by polling each one in sequence. But if the read/process/store time gets bigger then that will limit how many cycles you can perform in your 0.5 second window. One option is to have multiple threads, each reading a sensor(or sensor group). Another is to split the processing out to a separate program that reads the recorded data from storage and processes it before issuing updates to the sensors as needed. This takes us into the thorny world of systems architecture and concurrency management both of which are complex and depend on detailed analysis of requirements. Probably at a level beyond the scope of this list! (Although that was my day job before I retired!) > There is a small amount of analysis being done as the data is stored > but that's to drive the system (ie at point x you stop this, at point > y you do this, point y does this for z time then a happens etc - - - I > do not consider that heavy duty analysis. That happens on a different > system (storage happens first to the local governing system. OK that sounds like the second of my options above. That's a perfectly viable approach. > a round of data (one cycle) has completed that data is shipped in a burst > to the long term storage system. That again is viable but does run the risk of losing all data for the cycle if anything breaks. But if the data are interdependant that may not matter and may even be a good thing. > of information - - - or that's the goal anyway. Maybe One system will store the > info and another will do analysis and then return that analysis to the storing > system for accumulation - - - also not decided.) That's fine you need very careful analysis to determine the architecture and it will need to be based on throughput calculations etc. (This is where the old software engineering meme about only considering performance after you find there is a problem does not apply. The cost of refactoring a whole architecture is very high!) > what I should be using for what you termed 'structured files'. The more I read your posts the more I think you should go with a database and probably a server based one because i think you'll wind up with several independent programs reading/writing data concurrently. I'm still not 100% sure if it's a SQL or NoSQL solution, but my gut says SQL will be adequate. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos From o1bigtenor at gmail.com Wed Jul 12 18:59:45 2023 From: o1bigtenor at gmail.com (o1bigtenor) Date: Wed, 12 Jul 2023 17:59:45 -0500 Subject: [Tutor] looking but not finding In-Reply-To: <227607cb-4fb2-11cc-d3ec-54bff36ffbb5@wichmann.us> References: <227607cb-4fb2-11cc-d3ec-54bff36ffbb5@wichmann.us> Message-ID: Some interesting ideas mr Mats On Wed, Jul 12, 2023 at 1:53?PM Mats Wichmann wrote: > > On 7/12/23 09:26, o1bigtenor wrote: > > Thank you for your patience mr Alan. > > > > What I'm finding is that the computing world uses terms in a different way than > > does the sensor world than does the storage world than does the rest of us. > > That seriously complicates things. > > I'm guessing you have no idea how bad the jargon problem is... and it > certainly doesn't help when terms are "overloaded" - given different > special meanings by different groups. I have some idea - - - general engineering sorta started that and anything to do with engineering in any way shape or form (plus the medical/dental fields) seems to glory in obfuscation - - - seems to make far too many think that they know something where in fact this kind of usage - - - - well - - - - I'll stop - - - 'cause I'm a supposed to be at least a little polite! (Will admit to having great fun showing 30 year practitioners with the highest level of education how they really need to get back to hands on - - - - instead of paper manipulation - - - - except that never works with a bureaucrat - - :-( ! ) > > Logging: what you've found has to do with event logging - where a > program wants to record things that have happened in the running of the > program itself. What you're after is what some call data logging. You > can even look for projects that self-declare being related to that term > like this: Well - - - - sorta - - - - I want to do both. I need to record the results from the sensors because some of that information is used in the running of the program but almost all of the sensor data is going to be stored as well. SDo this isn't a clean store only nor is it where the data isn't used within the management program - - - its both! > > https://github.com/topics/data-logging In the first example listed I find the using of the python 'logging' libraries. > > >> There are many, many different types of storage available > >> from structured files(shelves, JSON, YAML, XML, etc to > >> databases both SQL and NoSQL based. > > > This is where I start getting 'lost'. I have no idea which of the three > > listed formats will work well with what I'm doing - - - dunno even how to > > find out either! > > The storage format is something you might consider an implementation > detail, that you don't have to worry about yet. The first steps are the > ones you're taking: to flesh out what your requirements are, and move to > a conceptual design. When you think you have a vision of what it will > look like, you can see if some particular existing library provides an > easy to use implementation of one of these that would fit your needs, > with some room to group. If you design it right, the piece that drives > that could also drive a different thing, if you found you chose poorly, > or outgrew it. > > JSON, YAML and XML are text-based formats, with some markup to help both > you as a human reader and in some cases, computer code, to decode it. If > there's not a ton of data, and/or there's not a lot of interrelationship > between different types of data (suggesting a "relational" database), > one of these will probably be fine. XML is a lot chattier, and the XML > libraries in Python are quite fiddly to use so recommend you cross XML > off your candidate list unless there are really good reasons to needs > its level of structure. You can even use comma-separated-values (like a > spreadsheet program would dump put), which is quite inflexible - one > line is one record, and every field has to be present in every record, > you don't have the option of extra or omitted fields, or it just breaks. > Pretty sure each one of those has an eloquent wikipedia page, e.g. > > https://en.wikipedia.org/wiki/JSON#Syntax With some digging it seems that yaml is pretty much a superset of of json and with your notes I'm leaning very much toward using yaml. > > If you want to use a database, rather than a text file, Alan laid out > some choices. For reasonably small amounts of data, sqlite is usually > fine. It's perhaps closer to the text-based interchange formats in that > there's no separate database program that you communicate with, data is > read from and written to a disk files in-process, but the interaction > with that data is through SQL statements so it's also like the "big > databases" - mysql/mariadb, SQL Server, Postgres. Then the "noSQL" > databases (that's not a terribly precise term) are the ones that have > less structure, to aid in scalability - a problem you're not likely to > have from your description so won't say more here. > Its looking quite like postgresql is going to be the storage apparatus. Spent some time looking to see the difference between regular and irregular data - - - another set of terms used commonly and just not defined! (Or at least not well or perhaps only hard to find!) Thanking you for your assistance. From o1bigtenor at gmail.com Thu Jul 13 09:02:23 2023 From: o1bigtenor at gmail.com (o1bigtenor) Date: Thu, 13 Jul 2023 08:02:23 -0500 Subject: [Tutor] looking but not finding In-Reply-To: References: Message-ID: On Wed, Jul 12, 2023 at 6:51?PM Alan Gauld via Tutor wrote: > > This is late at night so I won;t reply to all the points > but I'll pick up a few easier ones! :-) > > > On 12/07/2023 16:26, o1bigtenor wrote: > > What I'm finding is that the computing world uses terms in a different way than > > does the sensor world than does the storage world than does the rest of us. > > Sadly true. I come from an electronic/telecomms background but > most of my career was in software engineering. SE is a very new > field and doesn't have the well established vocabulary of more > traditional engineering fields. Often one term can have > multiple meanings depending on context. And other times multiple > terms are used for the exact same thing. > The phrase "divided by a common language" comes to mind. A pity that practitioners seem to believe that using such arcana makes them seem knowledgeable - - - imo it does the opposite. Clarity promotes understanding and eases usage - - - and as I think I've seen - - - drives sales (but not maybe profits). > > > >> from structured files(shelves, JSON, YAML, XML, etc to > >> databases both SQL and NoSQL based. > > > > This is where I start getting 'lost'. I have no idea which of the three > > listed formats will work well > > I actually listed 4! :-) > shelve is a Python proprietary format that will take arbitrary Python > data structures and save them to a file that can then be accessed like a > dictionary. Its very easy to use and works well if your data can be > found via a simple key. For more general use it has issues. > > JSON, YAML, XML are all text based files using structured text > to define the nature of the data. JSON looks a lot like a Python > dictionary format. XML is good for complex data but is correspondingly > complex to work with. YAML is somewhere in the middle. > For your purposes JSON is probably a good choice. > > Wikipedia describes all of them in much more detail. Seems that YAML is a superset to JSON - - - so at this point I'm thinking of using YAML. > > > > The first point with the data is to store it. At the time of its collection > > the data will also be used to trigger other events so that will be (I think > > that is the proper place) be done in the capturing/organizing/supervising > > program (in python). > > That raises a whole bunch of other questions such as is the data > stateful? Does the processing of sample Sn depend on the values > of Sn-1, Sn-2... If so you need to store a "window" of data in > memory and then write out the oldest item to storage as it expires. > > If not you can probably just read, process, store. Found something called MQTT and it would seem that this 'subsystem' should be useful for what I'm trying to do. Comments? > > > honestly cannot tell what makes data > > 'irregular' > > It just means that different data items have different attributes. > Some fields may be optional and different readings will contain > different numbers of fields. > Or the same fields may contain different types of data, for example > one sensor reports on/off while another gives a magnitude and > another gives a status message. That would be irregular data > and SQL databases don't like it much (although there are > techniques to get round it) > > Regular data just means every reading looks the same so you can > define a table with a set number of columns each of a known type. Your explanation is what I would have thought but I've learnt the hard way that logical and programming really don't necessarily work together. I wonder what ever happened to the concept of mathematically provable constructs in programming (ie Byte article where programs could be proven bug free). > > > Now if I only knew what regular and/or irregular data was. > > Have been considering using Postgresql as a storage engine. > > AFAIK it has the horsepower to deal with serious large amounts of data. > > It will do the job but so would something much lighter like SQLite. > The really critical factor is how much parallel access you need. > SQLite is great if only a single program is reading and writing > the data. Especially if there is only a single thread of execution. > The server based databases like Postgres come into their own if > you have multiple clients accessing the database at once. They can > perform tricks like record level locking (rather than table level > or even database level) during writes. As I'm starting with probably over 10 different systems logging and future possibilities of perhaps 200 I'm starting with the server based database system (Postgresql is the plan). > > > Its not serious huge data amounts but its definitely non- > > trivial. > > Modern storage solutions have made even Gigabytes of data almost > trivial. Big volumes would be more of an issue if using flat files > because they need to be read from disk and that takes time for big > volumes - even using SSDs. Storage isn't the issue - - - - its making sure that the data is handled well and accurately. (I can remember buy a hand lettered 40 MB HDD and it was some considered huge - - at the time 10 MB was considered decent sized - - - grin!) I've already got one circa 2 TB and another at 6TB both raid 10. Storage isn't that expensive today. (Remember working at a facility early 82 and the 14" HDD was 5 MB - - - that was HUGE at the time.) > > > Concurrent - - - yes, there are multiple sensors per station and > > multiple stations and hopefully they're all working (problems > > with the system if they're not!) > > You can deal with a relatively small number of sensors in a single > threaded Python application, just by polling each one in sequence. > But if the read/process/store time gets bigger then that will limit > how many cycles you can perform in your 0.5 second window. One option > is to have multiple threads, each reading a sensor(or sensor group). > Another is to split the processing out to a separate program that > reads the recorded data from storage and processes it before issuing > updates to the sensors as needed. I am using a number of microcontrollers coupled to one SoC for every 'point' so the issue is more in the storing and getting it there. That's why the questions regarding that. > > This takes us into the thorny world of systems architecture and > concurrency management both of which are complex and depend on > detailed analysis of requirements. Probably at a level beyond > the scope of this list! (Although that was my day job before > I retired!) > > > There is a small amount of analysis being done as the data is stored > > but that's to drive the system (ie at point x you stop this, at point > > y you do this, point y does this for z time then a happens etc - - - I > > do not consider that heavy duty analysis. That happens on a different > > system (storage happens first to the local governing system. > > OK that sounds like the second of my options above. That's a > perfectly viable approach. > > > a round of data (one cycle) has completed that data is shipped in a burst > > to the long term storage system. > > That again is viable but does run the risk of losing all data for the > cycle if anything breaks. But if the data are interdependant that may > not matter and may even be a good thing. > > > of information - - - or that's the goal anyway. Maybe One system will store the > > info and another will do analysis and then return that analysis to the storing > > system for accumulation - - - also not decided.) > > That's fine you need very careful analysis to determine the > architecture and it will need to be based on throughput > calculations etc. (This is where the old software engineering > meme about only considering performance after you find there > is a problem does not apply. The cost of refactoring a > whole architecture is very high!) > > > what I should be using for what you termed 'structured files'. > > The more I read your posts the more I think you should go with a > database and probably a server based one because i think you'll > wind up with several independent programs reading/writing data concurrently. > > I'm still not 100% sure if it's a SQL or NoSQL solution, > but my gut says SQL will be adequate. > Well - - - I think Postgresql should handle what I want it to do adequately. (Now - - - I'm looking at on-line in-line spectrographic analysis for a further implementation but that's not today - - - that will drive the data up considerably!) Thanking you for your assistance. Regards From hariganivada81 at gmail.com Thu Jul 13 09:11:25 2023 From: hariganivada81 at gmail.com (hari ganivada) Date: Thu, 13 Jul 2023 18:41:25 +0530 Subject: [Tutor] (no subject) Message-ID: Sir I am hari Ganivada. def is _power_off_two(number) number=2 while number ? 2==0: number = number/2 If number == 1: return true return False Call to the function print(is_power_of_two(0)) # should be False print(is_power_of_two(1))# should be true print(is_power_of_two(8))#should be true print(is_power_of_two(9))#should be false. Please help me sir From hariganivada81 at gmail.com Thu Jul 13 09:13:18 2023 From: hariganivada81 at gmail.com (hari ganivada) Date: Thu, 13 Jul 2023 18:43:18 +0530 Subject: [Tutor] Code In-Reply-To: References: Message-ID: Sir I am hari Ganivada. def is _power_off_two(number) number=2 while number ? 2==0: number = number/2 If number == 1: return true return False Call to the function print(is_power_of_two(0)) # should be False print(is_power_of_two(1))# should be true print(is_power_of_two(8))#should be true print(is_power_of_two(9))#should be false. Please help me sir ---------- Forwarded message --------- From: hari ganivada Date: Thu, 13 Jul, 2023, 18:41 Subject: To: Sir I am hari Ganivada. def is _power_off_two(number) number=2 while number ? 2==0: number = number/2 If number == 1: return true return False Call to the function print(is_power_of_two(0)) # should be False print(is_power_of_two(1))# should be true print(is_power_of_two(8))#should be true print(is_power_of_two(9))#should be false. Please help me sir From wlfraed at ix.netcom.com Thu Jul 13 13:03:16 2023 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Thu, 13 Jul 2023 13:03:16 -0400 Subject: [Tutor] looking but not finding References: <227607cb-4fb2-11cc-d3ec-54bff36ffbb5@wichmann.us> Message-ID: On Wed, 12 Jul 2023 17:59:45 -0500, o1bigtenor declaimed the following: >Spent some time looking to see the difference between regular and irregular >data - - - another set of terms used commonly and just not defined! >(Or at least not well or perhaps only hard to find!) "Regular", as used here, means that EVERY record has the same layout. You may have columns that permit NULLs, but all the columns are in each record, and the columns are in the same order from record to record. Classic relational database theory (besides SQL, you'd want to study the first three Codd Normal Forms -- short summary, not in NF order: no column has repeating data; if you did have repeating data it gets split into a second relation with a foreign key [link] back to the parent relation unique ID, and if order matters, a sequence value added to the second relation for sorting; all data should be dependent upon the primary key, if you find you are repeating the same information over and over it should go into another relation and link to it replaces the column(s) in the main relation} uniqueID | timestamp | sensornumber | sensorreading | etc. {My practice is to always put an auto-incrementing ID field, even if [timestamp|sensornumber] might be sufficient to uniquely identify one reading} At the most gross level, irregular data would be a system that might store a Word document next to an Excel spreadsheet next to... There is no regularity (no tabular representation) of the data. Some such may be described as "document" databases. Per https://en.wikipedia.org/wiki/MongoDB MongoDB is a document oriented system using JSON format "documents" https://en.wikipedia.org/wiki/Zope_Object_Database ZODB is a persistence scheme for Python objects -- uses pickle so essentially only Python code can read/write the contents. SQLite, (Windows) Access/JET, M$ SQL Server, MySQL/MariaDB, Oracle, PostgreSQL, Firebird are all relational database management systems. Most offer command line tools for simple queries. SQLite and Access/JET are "file server" RDBMs -- your code actually links to the database runtime and directly opens the database files. SQL Server, MySQL/MariaDB, Oracle, PostgreSQL, and Firebird are "client/server" RDBMs -- there is a server/engine running on some machine which is responsible for opening the database files, and your code sends SQL statement to that server for processing. From wlfraed at ix.netcom.com Thu Jul 13 13:14:53 2023 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Thu, 13 Jul 2023 13:14:53 -0400 Subject: [Tutor] Code References: Message-ID: On Thu, 13 Jul 2023 18:43:18 +0530, hari ganivada declaimed the following: >Sir > > I am hari Ganivada. > def is _power_off_two(number) > number=2 > while number ? 2==0: > number = number/2 > If number == 1: > return true > return False >Call to the function >print(is_power_of_two(0)) # should be False >print(is_power_of_two(1))# should be true >print(is_power_of_two(8))#should be true >print(is_power_of_two(9))#should be false. > >Please help me sir The most crucial thing is that Python is indentation sensitive. The code you show above has no consistent indentation and is unusable as shown. It is missing a : on the "def" line, you have a blank space between "is" and "_power". I have no idea what character you were trying to put in that while statement (modulo?). "true" is not the same as "True". Don't retype what you think you are running: cut&paste the exact TEXT (not an image) from your editing window. Do the same with any error messages you get when trying to run your code. From threesomequarks at proton.me Thu Jul 13 16:13:38 2023 From: threesomequarks at proton.me (ThreeBlindQuarks) Date: Thu, 13 Jul 2023 20:13:38 +0000 Subject: [Tutor] Code In-Reply-To: References: Message-ID: Hari, We can help better when we understand the problem and can read and run the code. As Dennis pointed out, you code is not right for any purpose. Before writing code it can be helpful to write down the algorithm you want to use. What you show makes no sense to me. The goal seems to be to check if a number supplied is a power of two. Besides 2,4,8,16,... do you want to accept two raised to the zero power (1) or negative powers? So what is your algorithm? How does your code reflect it? Your code shown starts with: def is _power_off_two(number) Suggestion, a function definition ends with a colon and usually carriage return. In English, the words "off" and "of" mean different things. It is perfectly legal to call your function is_power_off_two but not if you have a space after "if". Clearly the code you are showing us is NOT what you told python as it will not run here and you sow sample output. Your next line floors me: number=2 This HAS TO be indented. And it makes no sense to change what is an input variable to your function as once changed, it will now do the same thing every time you call it with any number. Your next line is also nonsense. It too needs indentation and is not valid code: while number ? 2==0: What is the purpose of the question mark? Do you mean to use whatever symbols it takes to ask if the number modulo 2 returns 0 as a remainder? I am guessing that is what you want and you need the right way to do that. You then divide the number by 2. THe line is not indented. Depending on how you indent, it can be part of the while loop or back in the main body of the function or ??? number = number/2 And note you asked for regular division rather than integer division so 3/2 can be 1.5. Then you have an IF statement badly formatted. Is it inside the while or not? Normally I expect an ELSE clause but in your case, it looks like you have an unconditional False return if the code is reached which is valid. > If number == 1: > return true > return False You then ask why it does not work? We have no idea as the code you show us cannot work at all. But looking at your examples, if this is for a class as an assignment, you seem to consider valid powers of two to include 1 and 2 and 4 and so on. I do not see you showing what did happen, just what you expect. So I am guessing your question is not why the algorithm failed but why the code was not accepted at all by Python. So why not supply the error messages received. Consider fixing some of the things Dennis and I are talking about like not having a space after is and having a colon at the end of the line, and see what breaks next. Then consider what happens in your code if a number comes in that makes the while loop run forever and so on. Good Luck. Sent with Proton Mail secure email. ------- Original Message ------- On Thursday, July 13th, 2023 at 9:13 AM, hari ganivada wrote: > Sir > > I am hari Ganivada. > def is _power_off_two(number) > number=2 > while number ? 2==0: > number = number/2 > If number == 1: > return true > return False > Call to the function > print(is_power_of_two(0)) # should be False > print(is_power_of_two(1))# should be true > print(is_power_of_two(8))#should be true > print(is_power_of_two(9))#should be false. > > Please help me sir > > > > ---------- Forwarded message --------- > From: hari ganivada hariganivada81 at gmail.com > > Date: Thu, 13 Jul, 2023, 18:41 > Subject: > To: tutor at python.org > > > > Sir > > I am hari Ganivada. > def is _power_off_two(number) > number=2 > while number ? 2==0: > number = number/2 > If number == 1: > return true > return False > Call to the function > print(is_power_of_two(0)) # should be False > print(is_power_of_two(1))# should be true > print(is_power_of_two(8))#should be true > print(is_power_of_two(9))#should be false. > > Please help me sir > _______________________________________________ > Tutor maillist - Tutor at python.org > To unsubscribe or change subscription options: > https://mail.python.org/mailman/listinfo/tutor From o1bigtenor at gmail.com Thu Jul 13 16:17:06 2023 From: o1bigtenor at gmail.com (o1bigtenor) Date: Thu, 13 Jul 2023 15:17:06 -0500 Subject: [Tutor] looking but not finding In-Reply-To: References: <227607cb-4fb2-11cc-d3ec-54bff36ffbb5@wichmann.us> Message-ID: On Thu, Jul 13, 2023 at 12:04?PM Dennis Lee Bieber wrote: > > On Wed, 12 Jul 2023 17:59:45 -0500, o1bigtenor > declaimed the following: > > > >Spent some time looking to see the difference between regular and irregular > >data - - - another set of terms used commonly and just not defined! > >(Or at least not well or perhaps only hard to find!) > > "Regular", as used here, means that EVERY record has the same layout. > You may have columns that permit NULLs, but all the columns are in each > record, and the columns are in the same order from record to record. > Classic relational database theory (besides SQL, you'd want to study the > first three Codd Normal Forms -- short summary, not in NF order: no column > has repeating data; if you did have repeating data it gets split into a > second relation with a foreign key [link] back to the parent relation > unique ID, and if order matters, a sequence value added to the second > relation for sorting; all data should be dependent upon the primary key, if > you find you are repeating the same information over and over it should go > into another relation and link to it replaces the column(s) in the main > relation} > Thank you for this definition!!!! > uniqueID | timestamp | sensornumber | sensorreading | etc. > > {My practice is to always put an auto-incrementing ID field, even if > [timestamp|sensornumber] might be sufficient to uniquely identify one > reading} > > At the most gross level, irregular data would be a system that might > store a Word document next to an Excel spreadsheet next to... There is no > regularity (no tabular representation) of the data. Some such may be > described as "document" databases. > > Per https://en.wikipedia.org/wiki/MongoDB MongoDB is a document oriented > system using JSON format "documents" > > https://en.wikipedia.org/wiki/Zope_Object_Database ZODB is a persistence > scheme for Python objects -- uses pickle so essentially only Python code > can read/write the contents. Excellent - - - and now I can understand where MongoDB might be useful. > > SQLite, (Windows) Access/JET, M$ SQL Server, MySQL/MariaDB, Oracle, > PostgreSQL, Firebird are all relational database management systems. Most > offer command line tools for simple queries. SQLite and Access/JET are > "file server" RDBMs -- your code actually links to the database runtime and > directly opens the database files. SQL Server, MySQL/MariaDB, Oracle, > PostgreSQL, and Firebird are "client/server" RDBMs -- there is a > server/engine running on some machine which is responsible for opening the > database files, and your code sends SQL statement to that server for > processing. > Great - - - thank you very much!!! Regards From farazpak at gmail.com Thu Jul 13 13:24:19 2023 From: farazpak at gmail.com (Faraz Ahmad) Date: Thu, 13 Jul 2023 22:24:19 +0500 Subject: [Tutor] (no subject) In-Reply-To: References: Message-ID: Dear hari: def is_power_of_two(number): if number <= 0: return False while number % 2 == 0: number = number // 2 return number == 1 # Call the function print(is_power_of_two(0)) # should be False print(is_power_of_two(1)) # should be True print(is_power_of_two(8)) # should be True print(is_power_of_two(9)) # should be False On Thu, Jul 13, 2023 at 8:35?PM hari ganivada wrote: > Sir > > I am hari Ganivada. > def is _power_off_two(number) > number=2 > while number ? 2==0: > number = number/2 > If number == 1: > return true > return False > Call to the function > print(is_power_of_two(0)) # should be False > print(is_power_of_two(1))# should be true > print(is_power_of_two(8))#should be true > print(is_power_of_two(9))#should be false. > > Please help me sir > _______________________________________________ > Tutor maillist - Tutor at python.org > To unsubscribe or change subscription options: > https://mail.python.org/mailman/listinfo/tutor > From threesomequarks at proton.me Fri Jul 14 14:09:47 2023 From: threesomequarks at proton.me (ThreeBlindQuarks) Date: Fri, 14 Jul 2023 18:09:47 +0000 Subject: [Tutor] Tungsten teaching method Message-ID: I have been wondering about some of the newer python learners in terms of how they approach learning. This is not really about just python but all kinds of languages. Do you start with the basics as in the kinds of constructs many earlier programming languages have. I mean define basic types like integer, floating point and character/text. Teach how to do simple assignment statements, then IF statements (or even weirder GOTO) and loop constructs like WHILE or FOR or whatever, then learn how to create functions, perhaps make simple objects and maybe eventually how to treat functions as first class objects. In other words, do you teach loosely simple parts of procedural programming then perhaps introduce object oriented and functional and perhaps other methodologies, including teaching about assorted libraries/packages/modules that extend the language? I think some here have learned vaguely along these lines and some quite differently. Some skip learning much and just look at snippets of code, perhaps in internet searches and try to figure things out. Some would prefer to search for pre-built software, as in publicly shared modules, that already do 90% of what they want. So I decided to learn another language called WOLFRAM. I am not suggesting anyone here do so. I just am using it to illustrate some points. In a sense, this language is a successor to Mathematica and is a very high-level language that is hard to characterize. It comes with at least 6,000 functions built-in as in not needing to be imported. It is by Stephen Wolfram, hence the name that means Tungsten which is why the chemical symbol is W. The book is from 2015 and is called An Elementary Introduction to the Wolfram Language. What got me is that for an ELEMENTARY text, it skips all the elementary parts. It seems to practically jump into functional programming from the start and only about 2/3 of the way through the book is there a mention of a kind of IF statement. No real mention of loops as everything sort of had loops built-in one way or another. I still have not seen examples on how to setup a variable and assign to it or create a function other than an immediate inline (anonymous) function let alone how to create an object. I know it all has to be THERE but you can apparently do an amazing amount in this language by simply calling one of the many built-in functions appropriately, often in a nested fashion. So, no, not suggesting it as a first language. The point here is that we often have little idea about the person asking a question and their background. There can be a temptation to simply give them an answer, sometimes an elegant answer using advanced features. Several of us recently tried coaching someone to fix their code and work through learning how to solve a problem. Someone else just supplied working code. Now if someone asked Wolfram how to say determine if a number is prime, rather than suggesting some technique using an algorithm, they would just suggest using PrimeQ[number] and voila! (In that language, you use square brackets in function calls.) If they were then asked how to check a whole list of numbers that way, do you think they would show any visible loops? Nope. I will spare the details but in effect they would apply a function to a list returning the list of results. If you want just the ones that are prime, you can use a sort of embedded if statement. One-liners seem to be the norm. And perhaps odder, the language is merely a part of a larger entity that includes a gigantic knowledge base on the internet that it automatically taps into so your program can literally ask in plain English for things like the populations of countries embedded in a request to make some complex plot displaying them. Python though is actually many paradigms in one. You can actually do quite a bit similar to what I described if you focus on a combination of functional programming and some object-oriented programming and are a bit more verbose. But if you want to learn how computers work on a basic level and then get into higher abstractions, it is very good. Some of what I describe is in there if you know where to look. You can use plain loops, then as syntactic sugar you can use comprehensions or perhaps generators and of course you can use functional programming, including the kinds of tools you can get in the itertools module that automate lots of stuff for you. In this forum, I suspect that unless asked otherwise, we ought to remain in the lower levels. And I don?t mean the meaning of Elementary of someone like Stephen Wolfram! LOL! From alan.gauld at yahoo.co.uk Fri Jul 14 15:33:25 2023 From: alan.gauld at yahoo.co.uk (Alan Gauld) Date: Fri, 14 Jul 2023 20:33:25 +0100 Subject: [Tutor] Tungsten teaching method In-Reply-To: References: Message-ID: On 14/07/2023 19:09, ThreeBlindQuarks via Tutor wrote: > Do you start with the basics as in ...basic types... IF statements... That is the traditional approach to 3G procedural languages that most people use. But... > So I decided to learn another language called WOLFRAM. Yes, it comes on the RaspberryPi microcomputer version of Linux. It is a 4G type language similar, as you say to Mathematica. > What got me is that for an ELEMENTARY text, it skips all the elementary parts. Thats because you have to approach it differently to 3G languages. You express your problem and let the language solve it. Much of it is done using symbolic math equations etc. In that sense it is similar to Prolog which is also typically taught in a more "exemplary" style than 3G languages. (Or indeed SQL which also doesn't explicitly have concepts like variables and loops etc) > ..how to create an object. I know it all has to be THERE Not necessarily. 4G (and higher) languages often don't provide those explicit concepts because they don;t expect the user to be deciding how to solve the problem so the internals of the language implementation create variables(as in temporary storage) build loops and make major decisions. Usually you can get in through a back door to do those things if you must (Prolog for example does allow such things) but they are advanced concepts. For example, its hard to envision how OOP fits into a declarative style of programming as used in Prolog. (It can be done and I've seen an ObjectProlog interpreter somewhere on the net but its not obvious, especially to a normal Prolog user.) > ... functions appropriately, often in a nested fashion. And that nesting is the key. The values stored in variables are simply passed as inputs to outer functions. (Javascript is often used in a similar way with code like function f(value, function(another_value), function()...) And the parametrized functions can be defined inline which leads to fairly tricky code to read but relatively few explicit variables, certainly at the global level. > The point here is that we often have little idea about > the person asking a question and their background. Indeed. > ...determine if a number is prime, rather than suggesting > some technique using an algorithm, they would just suggest > using PrimeQ[number] and voila! And that is typical of 4G and 5G languages. You give the interpreter the problem to solve and don't worry too much about how its done. > ...in effect they would apply a function to a list returning > the list of results. If you want just the ones that are prime, > you can use a sort of embedded if statement. That sounds like functional programming. Even in Python we have similar tools like generators and map(), reduce(), and others. Modules like itertools, functools, and some others remove or reduce the need for explicit loops and branches. > Python though is actually many paradigms in one. Indeed it is. > ... learn how computers work on a basic level and then > get into higher abstractions, it is very good. And that was (back in the 1990s) what Guido had in mind. A language that was useful for real world tasks but also suitable for teaching programming, in all its various guises. It also had to be easily extendible and easily embedded in other systems. > Some of what I describe is in there if you know where And that's true of most 5G languages. Even SQL(but you have to look very hard!) > In this forum, I suspect that unless asked otherwise, > we ought to remain in the lower levels. Yes. If someone knows enough to ask about other approaches we will oblige but in 90%+ of cases we are dealing with "students" who are learning conventional 3G programming. 4G languages are very good at what they do, but they tend to be focused on specific domains. It's not easy to build a GUI application in Wolfram or Prolog or to write a web server, say. But if a student is only interested in that niche area the 5G approach may be the best(*). The problems arise when they want to step outside the niche and discover they need to learn a whole new programming style. One that may initially seem very primitive and complex. Its like a Python programmer suddenly having to learn assembler... (*)Although 4G languages often have their own steep learning curve. But it tends to kick in later. Basic usage can be picked up easily but the huge "vocabulary" of builtin functions can take a long time to master. Compare that with "primitive" C which originally only had about 2 dozen keywords. Everything else was functions in libraries. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos From threesomequarks at proton.me Fri Jul 14 21:45:37 2023 From: threesomequarks at proton.me (ThreeBlindQuarks) Date: Sat, 15 Jul 2023 01:45:37 +0000 Subject: [Tutor] Tungsten teaching method In-Reply-To: References: Message-ID: <_9CNIAi4oJJ-uu0NRE6IpUxgv3SX0csbyZ0lCk2vusz_XLTl2wqZwoXkFhzj9bKQPPN_R3J6GQ1xiiXuPSqcfW_ZfHSxBaZRg_265Xphit8=@proton.me> Interesting Alan and I want to just slightly correct one thing, not that I am an expert. There are two somewhat related things called Wolfram|Alpha and the Wolfram Language. You may be combining both in your thoughts. The first has a sort of Natural Language Interface and the second is the formal language I was talking about. Of course, one aspect of the Wolfram Language is that you can call on the first in various ways so it interrogates a knowledge base and seamlessly returns objects or info you can drill down in. I, personally, am thinking of how to teach the language. As I continue reading the book I am seeing more data structures that seem to be variations on dictionaries and sets with rather different names and notations and approaches. My main point is that people who design languages often have one or more themes in mind. I know in a language like R, so much is vectorized that it feels natural to skip most loops as you can just use operations on complete vectors that sort of do everything at once. Similarly, it is very common to apply a function to something like a list that returns a list of the results. Python has that to an extent but because it was built on lists as a main component, some people feel more comfortable using the techniques on the data structures added in numpy/pandas. I think the point you make about levels of languages includes doing things at higher levels and that can be anything from ideas like the above to just having a huge library of flexible functions that handle most of your needs so you rarely HAVE TO provide your own detailed designs. But I keep suspecting the rest of the stuff is in there just as in python where you have all kinds of pre-defined objects and functions you can use, but sometimes feel a need to subclass or apply a decorator to tweak it to your needs. My point though was that anyone answering a question here may find their response is challenging. A recent example wanted an iterative solution and might have not been ready for a similar recursive solution. Sent with Proton Mail secure email. ------- Original Message ------- On Friday, July 14th, 2023 at 3:33 PM, Alan Gauld via Tutor wrote: > On 14/07/2023 19:09, ThreeBlindQuarks via Tutor wrote: > > > Do you start with the basics as in ...basic types... IF statements... > > > That is the traditional approach to 3G procedural languages > that most people use. But... > > > So I decided to learn another language called WOLFRAM. > > > Yes, it comes on the RaspberryPi microcomputer version of Linux. > It is a 4G type language similar, as you say to Mathematica. > > > What got me is that for an ELEMENTARY text, it skips all the elementary parts. > > > Thats because you have to approach it differently to 3G languages. > You express your problem and let the language solve it. Much of > it is done using symbolic math equations etc. In that sense it > is similar to Prolog which is also typically taught in a more > "exemplary" style than 3G languages. (Or indeed SQL which > also doesn't explicitly have concepts like variables and > loops etc) > > > ..how to create an object. I know it all has to be THERE > > > Not necessarily. 4G (and higher) languages often don't provide > those explicit concepts because they don;t expect the user to > be deciding how to solve the problem so the internals of the > language implementation create variables(as in temporary storage) > build loops and make major decisions. Usually you can get in > through a back door to do those things if you must (Prolog > for example does allow such things) but they are advanced concepts. > For example, its hard to envision how OOP fits into a declarative > style of programming as used in Prolog. (It can be done and I've > seen an ObjectProlog interpreter somewhere on the net but its not > obvious, especially to a normal Prolog user.) > > > ... functions appropriately, often in a nested fashion. > > > And that nesting is the key. The values stored in variables > are simply passed as inputs to outer functions. (Javascript is often > used in a similar way with code like > > function f(value, function(another_value), function()...) > > And the parametrized functions can be defined inline which > leads to fairly tricky code to read but relatively few explicit > variables, certainly at the global level. > > > The point here is that we often have little idea about > > the person asking a question and their background. > > > Indeed. > > > ...determine if a number is prime, rather than suggesting > > some technique using an algorithm, they would just suggest > > using PrimeQ[number] and voila! > > > And that is typical of 4G and 5G languages. > You give the interpreter the problem to solve and don't > worry too much about how its done. > > > ...in effect they would apply a function to a list returning > > the list of results. If you want just the ones that are prime, > > you can use a sort of embedded if statement. > > > That sounds like functional programming. Even in Python we have > similar tools like generators and map(), reduce(), and others. > Modules like itertools, functools, and some others remove or > reduce the need for explicit loops and branches. > > > Python though is actually many paradigms in one. > > > Indeed it is. > > > ... learn how computers work on a basic level and then > > get into higher abstractions, it is very good. > > > And that was (back in the 1990s) what Guido had in mind. A > language that was useful for real world tasks but also > suitable for teaching programming, in all its various guises. > It also had to be easily extendible and easily embedded > in other systems. > > > Some of what I describe is in there if you know where > > > And that's true of most 5G languages. Even SQL(but you have > to look very hard!) > > > In this forum, I suspect that unless asked otherwise, > > we ought to remain in the lower levels. > > > Yes. If someone knows enough to ask about other approaches > we will oblige but in 90%+ of cases we are dealing with > "students" who are learning conventional 3G programming. > > 4G languages are very good at what they do, but they tend > to be focused on specific domains. It's not easy to build > a GUI application in Wolfram or Prolog or to write a web > server, say. But if a student is only interested in that > niche area the 5G approach may be the best(). > > The problems arise when they want to step outside the > niche and discover they need to learn a whole new > programming style. One that may initially seem very > primitive and complex. Its like a Python programmer > suddenly having to learn assembler... > > ()Although 4G languages often have their own steep learning > curve. But it tends to kick in later. Basic usage can be > picked up easily but the huge "vocabulary" of builtin > functions can take a long time to master. Compare that > with "primitive" C which originally only had about > 2 dozen keywords. Everything else was functions in > libraries. > > -- > Alan G > Author of the Learn to Program web site > http://www.alan-g.me.uk/ > http://www.amazon.com/author/alan_gauld > Follow my photo-blog on Flickr at: > http://www.flickr.com/photos/alangauldphotos > > > > _______________________________________________ > Tutor maillist - Tutor at python.org > To unsubscribe or change subscription options: > https://mail.python.org/mailman/listinfo/tutor From PythonList at DancesWithMice.info Wed Jul 26 01:04:20 2023 From: PythonList at DancesWithMice.info (dn) Date: Wed, 26 Jul 2023 17:04:20 +1200 Subject: [Tutor] Tungsten teaching method In-Reply-To: References: Message-ID: This is a quarky/quirky conversation. (hah!) On 15/07/2023 06.09, ThreeBlindQuarks via Tutor wrote: > I have been wondering about some of the newer python learners in terms of how they approach learning. This is not really about just python but all kinds of languages. Actually, the idea of "language" embodies an assumption. There are major differences between the abstractions of Romance languages, and the 'click-based' Khoisan and similar - even Romance cf Chinese and Japanese. Over here, Te Reo Maori was a spoken language (pre-white man and 'civilisation') and 'writing' had quite a different meaning (and limited application*) when compared to English (of the period (!) ). Yet there are permanent 'records'. > Do you start with the basics as in the kinds of constructs many earlier programming languages have. I mean define basic types like integer, floating point and character/text. Teach how to do simple assignment statements, then IF statements (or even weirder GOTO) and loop constructs like WHILE or FOR or whatever, then learn how to create functions, perhaps make simple objects and maybe eventually how to treat functions as first class objects. A program[me] can be considered in two modes: control-flow and data-flow. Those of us who started back in the ?good, old, days were taught (lectured at) both FORTRAN-II/IV and COBOL-68. We quickly noticed similarities, eg both allowed one to add two values. Equally, it was starkly obvious that the languages had quite different approaches and philosophies - FORTRAN diving into its formulae/s, but COBOL requiring one to build a DATA DIVISION's worth of data-structures before even (linearly) reaching the PROCEDURE DIVISION. Recently, I've been taking opportunities to ask various groups of pythonista about the data-structures they use, and gain some idea of the frequency with which they reach 'beyond the basics'. A surprising (to me!) number rarely go beyond the basic built-ins (strings, lists, dicts - notice the omission of sets and tuples) and (perhaps) the Collections Library. Thus, many prefer not to utilise custom-class(es), dataclass(es), even named-tuples - and many have never even read about enums. Why? The most rational explanation is 'I can do it with what I've got'. In other words, the pain which may be incurred by having to manually-code functionality is seen as 'lesser', than the cognitive effort of indulging curiosity ("there must be a better way") and reading-up. NB a recent addition/quirk to this was someone who enthusiastically revealed his version: having the curiosity, but then asking ChatGPT to suggest the 'better way'. Good one! Question: has he added to his cognition, or is it more likely that ChatGPT gave some immediate cut-and-paste which formed a cognitive-bypass (in other words: the problem was 'solved', but no learning took place. Thus, the next time a similar problem arises ...) > In other words, do you teach loosely simple parts of procedural programming then perhaps introduce object oriented and functional and perhaps other methodologies, including teaching about assorted libraries/packages/modules that extend the language? There are two competing 'tensions' in traditional teaching: creating a basic foundational knowledge, and maintaining motivation. Remember the term (above) "lectured at" - were both present, or were 'we' required to maintain one - in the face of the other? I'll leave you to decide if 'modern students' are any more/less capable... > I think some here have learned vaguely along these lines and some quite differently. Some skip learning much and just look at snippets of code, perhaps in internet searches and try to figure things out. Some would prefer to search for pre-built software, as in publicly shared modules, that already do 90% of what they want. Which by indulging the motivation side, can lead to 'holes' in the foundations - unless part of a structured curriculum. That said, I prefer the approach of showing working examples BEFORE sourcing 'the facts' - somewhat the opposite of sitting through a lecture and then being given an assignment to 'prove' learning. (I followed this technique at last Wednesday's PUG-meeting: "Properties and Descriptors", and am 'famous' for introductions (for often nervous adults/slightly-fearful folk) in first sessions of both HTML5 and SQL/RDBMS) Again, there is a danger: SODD (StackOverflow-Driven Development) tempts cognitive-bypass, ie the learning you (convince yourself you) are doing without doing any actual learning. (there's a yawning chasm between reading something and thinking one understands the material, and actual learning - the observation which underlies the advice we all give: WRITE CODE to learn! Coming back to the "tension". There is a terrible temptation to hoodwink oneself into believing that 'progress has been made', when it hasn't. This is why external 'testing' is better than anything we come-up with by ourselves... > So I decided to learn another language called WOLFRAM. I am not suggesting anyone here do so. I just am using it to illustrate some points. In a sense, this language is a successor to Mathematica and is a very high-level language that is hard to characterize. It comes with at least 6,000 functions built-in as in not needing to be imported. It is by Stephen Wolfram, hence the name that means Tungsten which is why the chemical symbol is W. The book is from 2015 and is called An Elementary Introduction to the Wolfram Language. The implicit comparison is between Python and Wolfram, and is inappropriate (?unfair). Wolfram is unashamedly a Domain-Specific Language - would one use it for a typical COBOL-based record processing over-night batch job? (equally: would you use COBOL for 3D matrix processing?) The virtue of DSLs is two-fold, and these illustrate your points beautifully: 1 domain: they (can) presume that the user is also a 'member' of the domain. In 'W' it is a mathematical background. This was also presumed (to a lesser extent) in FORTRAN - which name, for others' benefit, was a contraction of FORmula TRANslation; where the purpose and even the expression of functions was 'taken as read'. Books often stated "just as in mathematics" or "mathematical proofs"! Key to any domain, is language. A word for a specialised (sub-set) language is "jargon". W assumes this for mathematics - indeed even as an applied mathematician (statistics) there are whole swathes of W functionality which zip over the top of my head. This also exists in Python, eg 'normal school kids' (in as much as I) talk of "whole numbers", "real numbers", and "letters"; whereas Python 'demands' we think of "integers", "floats", and "strings" (albeit wider than letters - and might be longer than one) - and we haven't even started to write code, yet! 2 implementation of concepts: the purpose of jargon is to enable rapid communication of (foundational and advanced) concepts rapidly - more rapidly than 'plain English'. (Sadly) it's effect (on someone from outside the domain, particularly a Python-Apprentice, is to present a 'wall of incomprehensibility'. What, eh? If a language can be presented to someone who has been 'admitted into the priesthood', ie knows the jargon; then it can communicate at such a higher-level, as does W! * (as promised, I return to the concept of 'writing') The Maori people did record the legacies of their ancestors. Various forms of carving communicated history and performed particular functions when the carving was part of specific structures (https://www.newzealand.com/int/maori-carving/) - North Americans may think of similarities with "monumental poles" (when I was at school, the term was "totem poles" - not sure how PC these days). Such carving says less to me than W! However, the imagery 'sparks' recitations (and education/training) from adult 'story-tellers' who thereby pass-on an oral history to their younger generations. (for example) In this situation, my phrasing of "a priesthood" to a domain, might be a literal description! The 'problem' comes when considering the 'specificity'. As before, W is little/no use outside its (narrow) domain. (reminds me of my early days 'learning' German - being presented with a (virus laden) PC running MS-Windows. It rapidly improved my vocabulary because I quickly noticed that there was a "Hilfe" sub-menu where (normally) there would be a "Help" menu (on my (English-language) version). However, this knowledge was of little use when I went out to buy dinner from the supermarket...) > What got me is that for an ELEMENTARY text, it skips all the elementary parts. It seems to practically jump into functional programming from the start and only about 2/3 of the way through the book is there a mention of a kind of IF statement. No real mention of loops as everything sort of had loops built-in one way or another. I still have not seen examples on how to setup a variable and assign to it or create a function other than an immediate inline (anonymous) function let alone how to create an object. I know it all has to be THERE but you can apparently do an amazing amount in this language by simply calling one of the many built-in functions appropriately, often in a nested fashion. Once again, we are comparing an apparently "functional" DSL with what started as a general-purpose language - and has since been extended in a variety of directions to suit proponents in (at times) wildly different fields. Narrow and powerful, cf wide and flexible! (see also (active and regular/periodic) discussions about what should be 'in' and what excised or added, to the Python Standard Library (PSL)) Dispute: what got you there was a combination of the text, your pre-existing mathematics knowledge, and your experience of other programming languages. You were NOT a raw-beginner/blank-slate! NB the latter achievement may sometimes also contribute negative factors! It seems that you have quickly absorbed W and have enjoyed a rapid rate of 'success'. See elsewhere "intrinsic motivation" - and my mantra when it comes to training approaches: "nothing succeeds like success"! Conversely, (and matching your thesis) the more (apparent) road-blocks placed in the path/way of a trainee, the less likely (s)he will persevere... Also, those of us who have been 'around' for some time have had to adapt from (perhaps0 'spaghetti coding' practices, to "Modular Programming", thence "Structured Programming", thereafter we had to consider "Object-Oriented Programming". Lately, we've been exposed to "Functional Programming". With such history - or even a part thereof, it easy to assume that each is an improvement on the previous, perhaps some newer generation of programming techniques. Such is not true. Accordingly, Functional Programming is something that can be practised immediately and without any need to have gone-through, or even to know about all those others. Accordingly, the "jump into" is not as big a deal as (perhaps) the words (above) may make it seem. In fact, we could take a Python newcomer and introduce lists of numbers, and jump straight into min(), max(), even map() - am not sure why a non-math programmer would find that advantageous, but it could be done... > So, no, not suggesting it as a first language. Dispute (although unlikely): why not use it as a 'first-language' (for mathematicians). Although not my first, at uni we were introduced to a matrix processing language (think SPSS or a higher-level numpy/pandas) which served as a 'first language' for many of my colleagues. All our stats-assignments had to be completed using it - even when I asked about using FORTRAN! I suspect though, your point is: "not suggesting it as a first language", means: not using it as a spring-board into learning (the likes of) Python. Although... > The point here is that we often have little idea about the person asking a question and their background. There can be a temptation to simply give them an answer, sometimes an elegant answer using advanced features. Several of us recently tried coaching someone to fix their code and work through learning how to solve a problem. Someone else just supplied working code. Now if someone asked Wolfram how to say determine if a number is prime, rather than suggesting some technique using an algorithm, they would just suggest using PrimeQ[number] and voila! (In that language, you use square brackets in function calls.) If they were then asked how to check a whole list of numbers that way, do you think they would show any visible loops? Nope. I will spare the details but in effect they would apply a function to a list returning the list of results. If you want just the ones that are prime, you can use a sort of embedded if statement. One-liners seem to be the norm. And perhaps odder, the language is merely a part of a larger entity that includes a gigantic knowledge base on the internet that it automatically taps into so your program can literally ask in plain English for things like the populations of countries embedded in a request to make some complex plot displaying them. Ah, now you've got me furiously riding my 'hobby horse' through the wilds of Cognitive Psychology. (Health warning = TLDR;) Despite being involved in virtual training initiatives for many more years than COVID-19 has entered the common lexicon*, I've long regretted the lesser-quality of "communication" when compared with in-person interaction. * as we, 'admitted to the priesthood', say, instead of "parlance" The process of "learning" involves the formation of "mental models". One might liken these to a skeleton of bones. One 'hangs' new-knowledge from the bones, and extending/covering the skeleton of existing-knowledge. For example, lists and 'primitives' - the former being a "collection" or "sequence" of the latter (and the latter not being part of the Python jargon - yet you understand the term, immediately!). If one's mental-model of how Python works is congruent to the (actual) way Python works, then we have (a degree of) success. If however it is not, then when the person comes to apply his/her knowledge, exceptions and errors ensue (plus, possibly, weeping, wailing, and gnashing of teeth - or in extreme cases, even sociopathy: "the computer made a mistake"). - cue perennial list-question revealing puzzlement over the binary representation of floating-point numbers... Thus, when it comes to the "questions" discussed here, a first reaction should not be 'what is the answer?', but more: 'why is the person asking THIS question' and perhaps: 'in this way?'. In other words, establishing WHY the person's mental model is causing the query in the first place. For example, why is 4 / 3 "1.333 (etc)" and not "1" - after all, both "4" and "3" are integers? Both the person and Python are 'correct'. However, the person's logic, whilst eminently reasonable, does not match Python's - and Python 'wins'/the person is confused. Thus, (as you...) the short answer is "float[ing-point] result". The longer answer is to discuss floats and integers, division and (integer) floor-division, number-types, etc, etc. Dispute: yes, one can "ask in plain English for..." but although (apparently) plain language, once-again these words are in-fact specialised, skewed, and domain-specific, ie jargon. You are asking "how many" type questions about populations, eg "how many people in France speak German?". Thus, it is more data concerned with 'the numbers', rather than anything to do with France, or German-speakers. For you (with your background, experience, and expertise) it might seem like "plain English". This assessment is skewed by your bias! (how's that for talking-math? BTW: no 'judgement'!) However, for many it is 'hidden knowledge' which resides within a particular 'domain', ie they would not know where to start, even when asking that question in their minds (again: no 'judgement'). > Python though is actually many paradigms in one. You can actually do quite a bit similar to what I described if you focus on a combination of functional programming and some object-oriented programming and are a bit more verbose. But if you want to learn how computers work on a basic level and then get into higher abstractions, it is very good. Some of what I describe is in there if you know where to look. You can use plain loops, then as syntactic sugar you can use comprehensions or perhaps generators and of course you can use functional programming, including the kinds of tools you can get in the itertools module that automate lots of stuff for you. Agreed. Again, a 'tension'. When asked about these, usually in the context of 'readability' (whatever that is!); my response involves assessing 'the team'. Thus, if 'we' are all perfectly capable of writing comprehensions and generators, then 'we' (all) consider them 'readable'. However, if 'we' includes folk who are still Python-Apprentices, perhaps we should consider such abstractions 'less readable' and take-the-long-way-home? OR... should we make a point of upgrading our colleagues' mental-models to include such powerful features, improve their (personal) skills, and move the team to some 'higher plane'? (worthy consideration: what would such a move, properly implemented, do for group-cohesion, etc?) Perhaps some of the observations (above) are applicable because it only needs to apply to an individual. The 'rules' change when consideration of the team is paramount. Let's return to 'training', and the phenomenon often visualised as a "Learning Curve", and specifically the issues of "cognitive over-load" (trying to absorb too much information and/or too many concepts over a short space of time). This is a BIG problem - and not just in Python or Computer Programming! Which is why Curriculum Design is not a trivial matter. I'm observing the side-project of one of our recent PUG-speakers called "The When of Python" (https://whenof.python.nz/blog/) wherein the idea is to try to identify which Python-constructs one needs to learn, at various stages in one's Python-career. Similarly, this problem is revealed in the many approaches taken by the authors of 'intro to Python' books. Which one is correct? I fear that the answer is both 'all' and 'none'! Certainly, an individual learner may (think to) prefer one over another, and an individual trainer similarly. However, (and one of your points) there is not one-size-fits-all (ie no 'silver bullet'). NB don't get me started on the pop-psy fallacies implicit in the idea of different types of 'learner'... Coming back to 'mental models'. The key to scraping the 'rough corners' forming errors in an individual's model is saying/seeing/trying the same thing in a different manner. > In this forum, I suspect that unless asked otherwise, we ought to remain in the lower levels. And I don?t mean the meaning of Elementary of someone like Stephen Wolfram! LOL! Yes. But... We have different types of enquirer. All SHOULD be made to feel welcome, per the aphorism "I came for the language. I stayed for the community". - the homeworker: yes, many simply want an answer and consider 'us' to be Stack-Overflow or ChatGPT, and the path to a (rapidly) completed assignment (see also "essay mills"). Yes, that attitude is wrong. No, I don't like it. We often respond with 'won't do your homework for you' and whilst this seems uncooperative (even rude or cruel?), the reason is 'cognitive bypass', ie the 'homework' was given to promote learning and NOT to produce a (correct, numeric or whatever) answer. - the baffled: the person doesn't understand why something happened, because his/her mental-model is incompatible with Python. This is difficult to detect because the person seldom realises where the divergence occurs/occurred. Such necessitates responding to a question with a question - often regarded as rude/impolite in many cultures. However, it is the 'correct' approach (but only seen to be so) in the long-run. - the learner: such a person appreciates that (s)he doesn't know 'the topic' and is providing (some) background information in order to establish sufficient context. This person will be very happy to be given some web-references (or similar), and left to continue on his/her learning-journey. Sometimes it is just-as-easy to explain the concept. In this case, correcting 4 / 3 to 4 // 3 doesn't actually ANSWER the question - because the question is about the concept not the syntax. - others? It is a fact of life that contributors here don't have the same motivations (or "domain") as I/we/the rest of us. Accordingly, differ in the opinion of how a question should be answered - indeed whether an answer is 'satisfactory'. There have been criticisms (overt or implicit) in response to the 'won't do your homework' response - evidence that different motivations (of both question-er and answer-er) have been interpreted by 'reading between the lines'. All of which is quite another essay, and is rooted in the issues and short-comings of written communication (email in this case) and the ability of each of us to 'communicate'... (mea culpa!) -- Regards, =dn From leamhall at gmail.com Wed Jul 26 06:39:18 2023 From: leamhall at gmail.com (Leam Hall) Date: Wed, 26 Jul 2023 05:39:18 -0500 Subject: [Tutor] Tungsten teaching method In-Reply-To: References: Message-ID: On 7/26/23 00:04, dn via Tutor wrote: > This is a quarky/quirky conversation. > (hah!) > > On 15/07/2023 06.09, ThreeBlindQuarks via Tutor wrote: >> I have been wondering about some of the newer python learners in terms of how they approach learning. This is not really about just python but all kinds of languages. [edited to conserve innocent electrons] Some of my favorite topics are rapid skill acquisition (https://www.amazon.com/The-First-20-Hours-Josh-Kaufman-audiobook/dp/B00D24BEQU) and deeper learning (https://www.amazon.com/Real-World-Gunfight-Training-Cutting-Edge/dp/B09M74KWF1). I disagree with dn about learner types being pop-psych, I find videos to be nearly useless. I read somewhere that the first time you are exposed to a topic, you gain about 17% of it. The next time, you get 44%. The "nearly" uselessness of videos is useful here, I'll sometimes watch a video for the 17%, and then when my brain is primed for the topic, read a book and maximize the 44%. Most of my career has been built that way. There's a reason some of us don't deeply care about sets, or enums, or ; we don't need them. Why keep trivia in you brain if you're trying to get work done? You can do a lot with the basics. Of course, if your work set requires sets, enums, etc, then they are not trivia. :) I was reading a PEP about a new pip option yesterday, and "meh". Maybe it's needed by some, but pip works for me. I have limited time and no real need to try all of the options. My focus is getting stuff done, not exploring the nooks and crannies of the work tools. Artful craftsmanship often comes from using a few tools very well. The "language to brain" fit is a great point! I've done a small number (set!) of languages, and Python is one of the closest to the way I think. Ruby was closer, but not really a great career choice. Smalltalk left me confused, and most of my spaghetti code was in PHP. I wrote about the best programming language (https://www.reuel.net/blog/20180814_best_programming_language.html), and how to get the language into your brain faster (https://github.com/LeamHall/conversationalCoding). Are new Python learners less capable? Doubtful, you should have seen me when I first tried to learn it. I'm glad you didn't, it would be embarrassing. Do we need better teaching methods? Always! Learning should move forward, and learning about learning is still learning. Leam -- Software Engineer (reuel.net/resume) Scribe: The Domici War (domiciwar.net) General Ne'er-do-well (github.com/LeamHall) From bossladyofthefuture at gmail.com Tue Jul 25 20:30:12 2023 From: bossladyofthefuture at gmail.com (Fatima Mehak) Date: Tue, 25 Jul 2023 19:30:12 -0500 Subject: [Tutor] Tutoring help automating with python Message-ID: Hello, I've been getting errors when trying to run the attached text file. I have indicated the error at the bottom of the text file for review. Can you please assist? There is also an instructions file in word format. -Fatima From threesomequarks at proton.me Wed Jul 26 13:26:19 2023 From: threesomequarks at proton.me (ThreeBlindQuarks) Date: Wed, 26 Jul 2023 17:26:19 +0000 Subject: [Tutor] Tungsten teaching method In-Reply-To: References: Message-ID: Dave, You have a lot to say on this and related topics and I do not have any idea on where to even start in response and perhaps not here. You and I both have years of varied experience with programming with lots of overlap. My point is that this forum attracts several kinds of people and the ones asking questions may be the main focus as they have their own purposes in mind. Some are less interested in the philosophy or history than you and I. I happen to be interested in a very general set of topics that include Computer SCIENCE as in understanding on multiple levels how things work and on innovations to see if they offer something worthwhile. I am currently less focused on the kind of day job or earning a degree that many others see as a primary goal. This bias is something I need to push back when trying to help some here as they mostly want to get something DONE. Sure, many are curious and may be interested in multiple ways to do things or doing serious learning, but I bet often the goal is more focused on getting it done SOON and correctly. My son-in-law recently suggested I look into ChatGPT as they find it helpful to supply some code they can use. I see such programs as part of a continuum in terms of finding information and deciding what is of value. I go far enough back to when you consulted a librarian and then maybe got directed to a more specialist form of librarian or scholar who helped you find books or other materials that might broach the topic and then perhaps find references from there to follow. The process was not quick or easy and often ended in failure. But in the very early days of the internet, there really was not much yet online. Heck, finding things before web crawlers required being GIVEN info like the URL for a home page. Sending email at one point required mapping out a path between a set of machines that locally knew the next one in the path. Yahoo was initially just a set of curated bookmarks and early search engines were rather primitive. We had improvements steadily, and all kinds of protocols led to just asking for something and letting some invisible entities figure it out and silently do it, so you no longer had to send email to ...!ihnp4!...!...!machine666!user but could just say user at machine666 and forget about it. So what we have now is tons of info out there and ever smarter ways to access it, way beyond the Page Rank algorithm and deeply into machine learning methods. And this leads to an interesting thing for me. Often a question is asked and I just open something like a browser and with a few clicks find a reasonable result. I then ask myself why the one asking the question could not as easily do something similar BEFORE asking, LOL! I then realize there is a level of skill involved that many have not really learned yet either and that recognizing if what you are shown applies to your case, takes some learning and skill too. If you were to ask ChatGTP how to do something it may return info on how to do it using some computer language you are not even aware of and not setup to use for your purpose. You may need to tell it you want to do something using Python and using a particular module or two as a way to help zero in. And when you get a result, be very cautious. The info it uses is sometimes wrong and often misleading. What if the program it supplies was gleaned from ages ago when someone discussed how to do something in a 2.x version of Python and perhaps using a module that is no longer used. Let me say that natural languages are not really natural and often quite confusing so more rigorous structured computing languages have some advantages. When I say Wolfram has access to a natural language interface, I may need to explain. At one level simply allows you to specify a sort of query in text that returns data structures the program then continues to use the normal way. In some ways this is not much different than someone using a function call (in any language) that given a URL, reads the embedded HTML and searches for things like lists or tables or images and then returns some data structure that contains something like a list or dictionary containing a sort of copy of the selected contents. Your program may then simply loop over this data structure instead of trying the hard way to work directly with the web page. So I note articles like this one which do some comparison and analysis of how some of these programs work differently: https://writings.stephenwolfram.com/2023/03/chatgpt-gets-its-wolfram-superpowers/ It looks like Wolfram, given its mathematical underpinnings that you note, does quite a bit of calculation work and chatgtp is a tad more about parroting what it finds. He gives examples of times it is just plain wrong as it does not really know much and accepts what it has been given and how it could benefit from consulting HIS knowledge base that is quite different. My personal view is that human brains partially work by having many somewhat independent units evaluate stimuli from different facets and contribute upward to areas that combine info and analyses and perhaps assign weights till at some level near consciousness, some are improved and selected as the choice you are going with. As such, I see a place for many computerized algorithms that work independently and also cooperate to produce better results than any one of them might have been able to do alone. So if a user asks in some natural language "how far apart are China and Brazil" then one source might find some web site that mentions traveling from one to the other for some distance with a stopover in iceland to refuel. Obviously, the mileage in that case may be wrong. Another source would look up China and Brazil in a knowledge base and get the names of cities such as Beijing and Brazilia as well as their coordinates on a map. It would know how to deal with spherical geometry and plot a minimal length flight between them, perhaps at minimal altitude, or at 30,000 feet. It might report an airport to airport distance. But an interactive source, might then query the user as to what they really want. Do they want to specific the specific cities or do they want the distance from the nearest point in Brazil to the nearest point in China. Heck, do they want the nearest distance heading from the Pacific or over the Atlantic? This is wandering but perhaps makes the point that for a significant set of users, many interesting questions may not involve wanting to know HOW to make your own answer just to get one. If I see someone interesting on TV and wonder about some aspect of their background such as NATIONALITY, I can do a search like "John Dow bio NATIONALITY" and look at some of the answers and see if there is info. It may well say he is part NATIONALITY on his mother's side. But for many purposes, I can stop there and not demand an exhaustive family tree showing many generations. Of course, if my question is whether s/he is descended from some Kind/Queen of England, I might need a more precise genealogy tracing them back step by step. Your points about actual use of a language like Python are illustrative. Yes, many people rarely venture into the more interesting (to me) parts of the language. The reality is that some fairly simple programs can be written in, well, ANYTHING. Many modern shell programs like BASH can actually handle some fairly simple arithmetic and algorithms. Heck, I often go to my browser and type something in the URL box like "=1024*1024" and it returns a calculator of sorts with the answer shown and I can then go on using it online. Languages are way more interesting if you first are told something about their purpose and intent and maybe even a tad of history. BUT if someone has negligible experience then I agree trying to tell them much of that may not be helpful. My point with Wolfram is that if the goal was to teach a NEW person from scratch, perhaps the book was doing it right. If the goal was someone more like me, then perhaps an introduction outlining things would have been very helpful. It could have said that Wolfram has a full amount of the kind of language features found in many other languages BUT the goal was to rarely use them as it had more interesting and powerful paradigms. Only when writing your own lower-level functions, might they be needed or perhaps when trying to write faster code. Then, as an experience programmer, I might sit back and enjoy. Python has a rather loose concept we often discuss about what may be the Pythonic way. It can be both be enlightening and confusing. Is it really wrong to check values to avoid dividing by zero? I say no. But, of course, if you have a paradigm where exceptions can be triggered AND caught, and dividing by zero may be very unlikely, then sure, write your code with a "try" that does not test first but deals with the consequences when and if it happens. From PythonList at DancesWithMice.info Wed Jul 26 13:44:23 2023 From: PythonList at DancesWithMice.info (dn) Date: Thu, 27 Jul 2023 05:44:23 +1200 Subject: [Tutor] Tutoring help automating with python In-Reply-To: References: Message-ID: <426437d9-b641-e740-95c5-fe7082bea053@DancesWithMice.info> Salaam Fatima, On 26/07/2023 12.30, Fatima Mehak wrote: > Hello, > I've been getting errors when trying to run the attached text file. I have > indicated the error at the bottom of the text file for review. Can you > please assist? There is also an instructions file in word format. Will be happy to help, but regret that this list will not forward attachments. Please copy-paste from the output into an email-message. If you can reduce the code to its shortest possible form - which still displays the error, then copy-paste that too. Is this a course-assignment? In which case, perhaps indicate the source and URL if online. -- Regards, =dn From bossladyofthefuture at gmail.com Wed Jul 26 16:17:54 2023 From: bossladyofthefuture at gmail.com (Fatima Mehak) Date: Wed, 26 Jul 2023 15:17:54 -0500 Subject: [Tutor] Tutoring help automating with python Message-ID: Hello, I've been getting errors when trying to run the below text file. I have indicated the error at the bottom for review. Can you please assist? #!/usr/bin/env python3 from PIL import Image import os, sys def resize_rename_rotate(srcfile, targetdir="", size=(128,128)): targetfile = os.path.splitext(srcfile)[0] extension = os.path.splitext(srcfile)[1] if srcfile != tgtfile: try : im = Image.open(srcfile) # open file im.rotate(90) # degrees counter-clockwise im.resize((128, 128)) # resize the file im.save(targetdir+targetfile+extension,"jpeg") except IOError: print ("cannot change image for ", srcfile) if __name__=="__main__": targetdir = "../output/" srcdir = "../input/" for file in os.listdir(srcdir): resize_rename_rotate(file,targetdir) #*Traceback (most recent call last): # File "/home/student-03-61dc276e6247/./fix_image.py", line 23, in # for file in os.listdir(srcdir): #FileNotFoundError: [Errno 2] No such file or directory: '../input/' From threesomequarks at proton.me Wed Jul 26 19:52:30 2023 From: threesomequarks at proton.me (ThreeBlindQuarks) Date: Wed, 26 Jul 2023 23:52:30 +0000 Subject: [Tutor] Tutoring help automating with python In-Reply-To: References: Message-ID: Fatima, Check what folder the program thinks it is in. Then check if the parent directory has a subdirectory with the name needed and contents. My guess is the current directory has not been set to where you want it to be or that one level up has not been initialized as anticipated. Q Sent with Proton Mail secure email. ------- Original Message ------- On Wednesday, July 26th, 2023 at 4:17 PM, Fatima Mehak wrote: > Hello, > I've been getting errors when trying to run the below text file. I have > indicated the error at the bottom for review. Can you please assist? > > > #!/usr/bin/env python3 > > from PIL import Image > import os, sys > > def resize_rename_rotate(srcfile, targetdir="", size=(128,128)): > targetfile = os.path.splitext(srcfile)[0] > extension = os.path.splitext(srcfile)[1] > > if srcfile != tgtfile: > try : > im = Image.open(srcfile) # open file > im.rotate(90) # degrees counter-clockwise > im.resize((128, 128)) # resize the file > im.save(targetdir+targetfile+extension,"jpeg") > except IOError: > print ("cannot change image for ", srcfile) > > if name=="main": > targetdir = "../output/" > srcdir = "../input/" > > for file in os.listdir(srcdir): > resize_rename_rotate(file,targetdir) > > > > > > #*Traceback (most recent call last): > # File "/home/student-03-61dc276e6247/./fix_image.py", line 23, in > > # for file in os.listdir(srcdir): > #FileNotFoundError: [Errno 2] No such file or directory: '../input/' > _______________________________________________ > Tutor maillist - Tutor at python.org > To unsubscribe or change subscription options: > https://mail.python.org/mailman/listinfo/tutor From cs at cskk.id.au Wed Jul 26 19:45:01 2023 From: cs at cskk.id.au (Cameron Simpson) Date: Thu, 27 Jul 2023 09:45:01 +1000 Subject: [Tutor] Tutoring help automating with python In-Reply-To: References: Message-ID: On 26Jul2023 15:17, Fatima Mehak wrote: >I've been getting errors when trying to run the below text file. I have >indicated the error at the bottom for review. Can you please assist? [...] >``` > targetdir = "../output/" > srcdir = "../input/" > for file in os.listdir(srcdir): > resize_rename_rotate(file,targetdir) >``` > >``` >#*Traceback (most recent call last): ># File "/home/student-03-61dc276e6247/./fix_image.py", line 23, in ># for file in os.listdir(srcdir): >#FileNotFoundError: [Errno 2] No such file or directory: '../input/' >``` This means that there's no such directory `../input/`. Because that is a relative path, it is important that when you invoke Python that it is _running_ in the right directory for that path to work. Try putting this: import os print(os.getcwd()) before the `for...` line. That will tell you the current working directory Python is using, and hopefully that will make clear why the relative path is not working - the relative path must be valid from that directory from `os.getcwd()`. Cheers, Cameron Simpson From phillor9 at gmail.com Thu Jul 27 01:03:49 2023 From: phillor9 at gmail.com (Phil) Date: Thu, 27 Jul 2023 15:03:49 +1000 Subject: [Tutor] Tkinter iconbitmap question Message-ID: <02e89c59-e100-8843-6203-4589d58a5c09@gmail.com> I'm playing with an example from ttkbootstrap and I'm having a problem with the iconbitmap function. I realise that .ico files are not compatible with Linux so I've chosen an xbm file instead but I still have the same error as follows: root = tb.Window(themename="superhero") # root = Tk() root.title("TTK Bootstrap! DateEntry") root.iconbitmap('wind.xbm') root.geometry('500x350') Traceback (most recent call last): ? File ??? root.iconbitmap('wind.xbm') ??? ^^^^^^^^^^^^^^^^^^^^^^^^^^^ ? File "/usr/lib/python3.11/tkinter/__init__.py", line 2136, in wm_iconbitmap ??? return self.tk.call('wm', 'iconbitmap', self._w, bitmap) ?????????? ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _tkinter.TclError: bitmap "wind.xbm" not defined Wind.xbm is in the same virtual environment directory as the python file. I've searched the Internet for an answer and found the same question but no answers that makes much sense to me. It's not overly important. The desktop manager that I use doesn't have window icons, perhaps it's just an incompatibility problem? -- Regards, Phil From roel at roelschroeven.net Thu Jul 27 03:48:43 2023 From: roel at roelschroeven.net (Roel Schroeven) Date: Thu, 27 Jul 2023 09:48:43 +0200 Subject: [Tutor] Tutoring help automating with python In-Reply-To: References: Message-ID: Op 26/07/2023 om 22:17 schreef Fatima Mehak: > #*Traceback (most recent call last): > # File "/home/student-03-61dc276e6247/./fix_image.py", line 23, in > # for file in os.listdir(srcdir): > #FileNotFoundError: [Errno 2] No such file or directory: '../input/' Hi, Others have already answered why '../input/' is not found. Once that is solved, though, you're going to have the next issue: if I'm not mistaken, Image.open(srcfile) will not find srcfile. That's because os.listdir() returns the names of the files, with no reference to the directory they're in. One solution is to do something like this: ??? for file in os.listdir(srcdir): ??????? resize_rename_rotate(os.path.join(srcdir, file), targetdir) Regards, Roel -- "You can fool some of the people all the time, and all of the people some of the time, but you cannot fool all of the people all of the time." -- Abraham Lincoln "You can fool too many of the people too much of the time." -- James Thurber From alan.gauld at yahoo.co.uk Thu Jul 27 19:36:52 2023 From: alan.gauld at yahoo.co.uk (Alan Gauld) Date: Fri, 28 Jul 2023 00:36:52 +0100 Subject: [Tutor] Tutoring help automating with python In-Reply-To: References: Message-ID: The immediate issue has been identified. Here are some other things to consider... On 26/07/2023 21:17, Fatima Mehak wrote: > def resize_rename_rotate(srcfile, targetdir="", size=(128,128)): > targetfile = os.path.splitext(srcfile)[0] > extension = os.path.splitext(srcfile)[1] > > if srcfile != tgtfile: What is tgtfile? Where is it defined? > try : > im = Image.open(srcfile) # open file > im.rotate(90) # degrees counter-clockwise > im.resize((128, 128)) # resize the file Shouldn't this use the size parameter instead of a hard coded (128,128)? > im.save(targetdir+targetfile+extension,"jpeg") > except IOError: > print ("cannot change image for ", srcfile) -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos