From alexkleider at gmail.com Mon Jun 6 16:04:32 2022 From: alexkleider at gmail.com (Alex Kleider) Date: Mon, 6 Jun 2022 13:04:32 -0700 Subject: [Tutor] unicode question Message-ID: I've been playing around with unicode a bit and found that the following code doesn't behave as I might have expected: Python 3.9.2 (default, Feb 28 2021, 17:03:44) [GCC 10.2.1 20210110] on linux Type "help", "copyright", "credits" or "license" for more information. >>> print("\N{middle dot}") ? >>> >>> middle_dot = '0140', '013F', '00B7', '2027' >>> ucode = ['\\u' + dot for dot in middle_dot] >>> for dot in ucode: ... print(dot) ... \u0140 \u013F \u00B7 \u2027 >>> print("\u0140") ? >>> print("\u013f") ? >>> print("\u00b7") # the one I want ? >>> print("\u2027") ? >>> I was expecting the for loop to output the same as the last four print statements but, alas, not so. Comments would be welcome. Sincerely, Alex -- alex at kleider.ca (sent from my current gizmo) From __peter__ at web.de Mon Jun 6 17:56:29 2022 From: __peter__ at web.de (Peter Otten) Date: Mon, 6 Jun 2022 23:56:29 +0200 Subject: [Tutor] unicode question In-Reply-To: References: Message-ID: On 06/06/2022 22:04, Alex Kleider wrote: > I've been playing around with unicode a bit and found that the > following code doesn't behave as I might have expected: > > Python 3.9.2 (default, Feb 28 2021, 17:03:44) > [GCC 10.2.1 20210110] on linux > Type "help", "copyright", "credits" or "license" for more information. >>>> print("\N{middle dot}") > ? >>>> >>>> middle_dot = '0140', '013F', '00B7', '2027' >>>> ucode = ['\\u' + dot for dot in middle_dot] >>>> for dot in ucode: > ... print(dot) > ... > \u0140 > \u013F > \u00B7 > \u2027 >>>> print("\u0140") > ? >>>> print("\u013f") > ? >>>> print("\u00b7") # the one I want > ? >>>> print("\u2027") > ? >>>> > > I was expecting the for loop to output the same as the last four print > statements but, alas, not so. "\\u" is a string containing the backslash followed by a "u" -- and that won't change when you concatenate another string like "0140". The easiest way to realize the loop would be to use integers: >>> for i in 0x140, 0x13f: print(chr(i)) ? ? The obvious way when you want to start with strings is >>> for c in "0140", "013f": print(eval(f"'\\u{c}'")) # dangerous, may execute arbitrary code ? ? with the safe alternative >>> for c in "0140", "013f": print(codecs.decode(f"\\u{c}", "unicode-escape")) ? ? From alan.gauld at yahoo.co.uk Mon Jun 6 18:22:57 2022 From: alan.gauld at yahoo.co.uk (Alan Gauld) Date: Mon, 6 Jun 2022 23:22:57 +0100 Subject: [Tutor] unicode question In-Reply-To: References: Message-ID: On 06/06/2022 21:04, Alex Kleider wrote: >>>> middle_dot = '0140', '013F', '00B7', '2027' >>>> ucode = ['\\u' + dot for dot in middle_dot] This creates a list of literal strings like "\u0140",.... Which is identical to writing: ucode = ["\\u0140",...] So you have the \u as characters within the string which is not what you want. Instead you want to get the character values from the hex values in middle dot which you can do using chr() and int(). for dot in middle_dot: print(chr(int(dot,16))) However it would be easier just to store the hex values directly: middle_dot = [0x0140, 0x013F, 0x00B7, 0x2027) for dot in middle_dot: print(chr(dot)) -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos From mats at wichmann.us Mon Jun 6 19:30:53 2022 From: mats at wichmann.us (Mats Wichmann) Date: Mon, 6 Jun 2022 17:30:53 -0600 Subject: [Tutor] unicode question In-Reply-To: References: Message-ID: <56dfac24-29a1-5ada-cd90-2c65f368d98b@wichmann.us> On 6/6/22 14:04, Alex Kleider wrote: > I've been playing around with unicode a bit and found that the > following code doesn't behave as I might have expected: > > Python 3.9.2 (default, Feb 28 2021, 17:03:44) > [GCC 10.2.1 20210110] on linux > Type "help", "copyright", "credits" or "license" for more information. >>>> print("\N{middle dot}") > ? >>>> >>>> middle_dot = '0140', '013F', '00B7', '2027' Why not directly enter these in a usable form? >>> middle_dot = "\N{LATIN SMALL LETTER L WITH MIDDLE DOT}", "\N{LATIN CAPITAL LETTER L WITH MIDDLE DOT}" >>> for n in middle_dot: ... print(n) ... ... ? ? (too lazy to go look up the other two, whatever they may be) From drgauravshukla at gmail.com Tue Jun 7 06:33:09 2022 From: drgauravshukla at gmail.com (Gaurav Shukla) Date: Tue, 7 Jun 2022 16:03:09 +0530 Subject: [Tutor] Gaussian to image Message-ID: Hello, Is there a way to convert Gaussian curves to a greyscale image simulating a CCD image? So basically my Gauss has an x-value and y-value with multiple Gaussian's, which one generally get from a CCD/sCMOS experimentally. I have generated a Gaussian spectrum (with four peaks) based on atomic physics calculations, I would then try to read with another code that would read the image and give back the Gaussian spectrum. I am very new so any help would be highly appreciated. Thanks DG From wlfraed at ix.netcom.com Tue Jun 7 17:04:22 2022 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Tue, 07 Jun 2022 17:04:22 -0400 Subject: [Tutor] Gaussian to image References: Message-ID: On Tue, 7 Jun 2022 16:03:09 +0530, Gaurav Shukla declaimed the following: >Hello, >Is there a way to convert Gaussian curves to a greyscale image simulating a >CCD image? >So basically my Gauss has an x-value and y-value with multiple Gaussian's, >which one generally get from a CCD/sCMOS experimentally. > I have no idea of exactly what that sentence is describing... "x-value and y-value" to me imply ONE POINT on a Cartesian plane, so "multiple Gaussian's" is a meaningless term. A 2D array/matrix (x-axis, y-axis) in which the each point has a value representing the grey-level resulting from (possibly overlapping) gaussian data, OTOH, I can understand. However, in general, have you studied the capabilities of the MatPlotLib package? https://matplotlib.org/ -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From alan.gauld at yahoo.co.uk Tue Jun 7 18:35:22 2022 From: alan.gauld at yahoo.co.uk (Alan Gauld) Date: Tue, 7 Jun 2022 23:35:22 +0100 Subject: [Tutor] Gaussian to image In-Reply-To: References: Message-ID: On 07/06/2022 11:33, Gaurav Shukla wrote: > Hello, > Is there a way to convert Gaussian curves to a greyscale image simulating a > CCD image? Probably, but its not really a Pthon question but a mathematical one. You may be better off asking on one of the Python scientific computing forums. sciPy might be a good start. > So basically my Gauss has an x-value and y-value with multiple Gaussian's, > which one generally get from a CCD/sCMOS experimentally. > > I have generated a Gaussian spectrum (with four peaks) based on atomic > physics calculations, I would then try to read with another code that would > read the image and give back the Gaussian spectrum. That is all very domain specific. But this is a general programming forum concerned with the Python language and its standard library. It sounds like your request is very domain specific so either you need to find a forum for that domain(see above) or you have to go back to basics and exolain what those 2 paragraphs mean in simple terms a layman can understand. > I am very new so any help would be highly appreciated. We are happy to help with any Python programming questions. But how you apply math to this specific question is a bit beyond our remit. If you are very lucky one or two of our members will recognise what you are saying, but you are much more likely to find that on the Scipy fora -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos From PythonList at DancesWithMice.info Tue Jun 7 18:56:40 2022 From: PythonList at DancesWithMice.info (dn) Date: Wed, 8 Jun 2022 10:56:40 +1200 Subject: [Tutor] Gaussian to image In-Reply-To: References: Message-ID: On 07/06/2022 22.33, Gaurav Shukla wrote: > Hello, > Is there a way to convert Gaussian curves to a greyscale image simulating a > CCD image? > So basically my Gauss has an x-value and y-value with multiple Gaussian's, > which one generally get from a CCD/sCMOS experimentally. > > I have generated a Gaussian spectrum (with four peaks) based on atomic > physics calculations, I would then try to read with another code that would > read the image and give back the Gaussian spectrum. > > I am very new so any help would be highly appreciated. Might OpenCV help? (https://opencv.org/opencv-python-is-now-an-official-opencv-project/ and plenty of tutorials on the web) -- Regards, =dn From aliyan.navaid at gmail.com Fri Jun 10 08:59:25 2022 From: aliyan.navaid at gmail.com (Aliyan Navaid) Date: Fri, 10 Jun 2022 17:59:25 +0500 Subject: [Tutor] Learning Object Oriented Programming Message-ID: <594D68F1-6265-4F6B-A95F-61BE6E85C7CF@hxcore.ol> Hello, I aspire to learn AI with python and ?learned the basics of python such as datatypes, iteration, selection. However now I?m stuck as there is so much in the field of AI, what should my next move be? Learning OOP? Libraries such as SciPy? Tensor flow? Statistics? Thanks in advance. From leamhall at gmail.com Fri Jun 10 15:51:45 2022 From: leamhall at gmail.com (Leam Hall) Date: Fri, 10 Jun 2022 14:51:45 -0500 Subject: [Tutor] Learning Object Oriented Programming In-Reply-To: <594D68F1-6265-4F6B-A95F-61BE6E85C7CF@hxcore.ol> References: <594D68F1-6265-4F6B-A95F-61BE6E85C7CF@hxcore.ol> Message-ID: <02fafe43-51d3-60e9-f57b-99000635182c@gmail.com> Learning Python's object system will probably help, since most of your other material will assume you know it. As you said, the AI field is huge. Maybe mention one or two things that really excite you and see if folks can make suggestions for those specific things. On 6/10/22 07:59, Aliyan Navaid wrote: > Hello, > > I aspire to learn AI with python and ?learned the basics of python such as > datatypes, iteration, selection. > > However now I?m stuck as there is so much in the field of AI, what should > my next move be? Learning OOP? Libraries such as SciPy? Tensor flow? > Statistics? > > Thanks in advance. -- Automation Engineer (reuel.net/resume) Scribe: The Domici War (domiciwar.net) General Ne'er-do-well (github.com/LeamHall) From aliyan.navaid at gmail.com Sat Jun 11 04:40:01 2022 From: aliyan.navaid at gmail.com (Aliyan Navaid) Date: Sat, 11 Jun 2022 13:40:01 +0500 Subject: [Tutor] Learning Object Oriented Programming In-Reply-To: <02fafe43-51d3-60e9-f57b-99000635182c@gmail.com> References: <594D68F1-6265-4F6B-A95F-61BE6E85C7CF@hxcore.ol> <02fafe43-51d3-60e9-f57b-99000635182c@gmail.com> Message-ID: To be honest every AI project I've seen amazes me whether it is about finding insights in data, NLP, or Image generation. However, could you list down stuff (in order) that is essential to all these applications? On Sat, 11 Jun 2022 at 00:53, Leam Hall wrote: > Learning Python's object system will probably help, since most of your > other material will assume you know it. As you said, the AI field is huge. > Maybe mention one or two things that really excite you and see if folks can > make suggestions for those specific things. > > > On 6/10/22 07:59, Aliyan Navaid wrote: > > Hello, > > > > I aspire to learn AI with python and learned the basics of python > such as > > datatypes, iteration, selection. > > > > However now I?m stuck as there is so much in the field of AI, what > should > > my next move be? Learning OOP? Libraries such as SciPy? Tensor flow? > > Statistics? > > > > Thanks in advance. > > -- > Automation Engineer (reuel.net/resume) > Scribe: The Domici War (domiciwar.net) > General Ne'er-do-well (github.com/LeamHall) > _______________________________________________ > Tutor maillist - Tutor at python.org > To unsubscribe or change subscription options: > https://mail.python.org/mailman/listinfo/tutor > From avi.e.gross at gmail.com Fri Jun 10 16:52:48 2022 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Fri, 10 Jun 2022 16:52:48 -0400 Subject: [Tutor] Learning Object Oriented Programming In-Reply-To: <02fafe43-51d3-60e9-f57b-99000635182c@gmail.com> References: <594D68F1-6265-4F6B-A95F-61BE6E85C7CF@hxcore.ol> <02fafe43-51d3-60e9-f57b-99000635182c@gmail.com> Message-ID: <00f501d87d0c$0bda8720$238f9560$@gmail.com> My two cents is that the question is way too broad and this is not the place to discuss it. [Most people here who are focused on learning the basics of python probably should skip this message.] The subject was about Object Oriented Programming. The questions asked are NOT really. They are about an assortment of modules that internally use all kinds of techniques that do often do create and manage a series of objects and functions and to use them the user may have to use Object Oriented techniques and often other things like functional programming techniques. I have studied base Python a dozen or more ways to gain some fluency and only then did I study the modules you mention and many others. You may be able to use some much better if you learn incrementally. You may need to study the numpy module that is heavily used by everything including YOUR code that has to put data into a form that can be processed well. You may want to study the pandas module that sort of extends numpy. After that, it depends on what you want to do with AI. I am not sure you need SciPy for many things as much as Scikit-learn. As for TensorFlow, that is doing things at a relatively low level and many others have built on top of it some ways to do things in possibly simpler and more powerful ways such as Keras or Theano and others, albeit some take very different paths with some not feeling natural unless you understand many styles of python programming. But this is NOT the place for tutoring about specific and esoteric questions and as you have not asked such a question, I will not talk about this more here. There are oodles of resources on-line and in books that you can consult and so much depends on your background. AI is a vast subject and getting vaster and you may actually be more interested in topics like Machine Learning and end up working in a place where others use specific tools among the many available. Python as a language is just one of many tools. It is commonly used for the general purposes mentioned but some places use others. Good Luck. -----Original Message----- From: Tutor On Behalf Of Leam Hall Sent: Friday, June 10, 2022 3:52 PM To: tutor at python.org Subject: Re: [Tutor] Learning Object Oriented Programming Learning Python's object system will probably help, since most of your other material will assume you know it. As you said, the AI field is huge. Maybe mention one or two things that really excite you and see if folks can make suggestions for those specific things. On 6/10/22 07:59, Aliyan Navaid wrote: > Hello, > > I aspire to learn AI with python and learned the basics of python such as > datatypes, iteration, selection. > > However now I?m stuck as there is so much in the field of AI, what should > my next move be? Learning OOP? Libraries such as SciPy? Tensor flow? > Statistics? > > Thanks in advance. -- Automation Engineer (reuel.net/resume) Scribe: The Domici War (domiciwar.net) General Ne'er-do-well (github.com/LeamHall) _______________________________________________ Tutor maillist - Tutor at python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor From alan.gauld at yahoo.co.uk Sat Jun 11 07:55:44 2022 From: alan.gauld at yahoo.co.uk (Alan Gauld) Date: Sat, 11 Jun 2022 12:55:44 +0100 Subject: [Tutor] Learning Object Oriented Programming In-Reply-To: References: <594D68F1-6265-4F6B-A95F-61BE6E85C7CF@hxcore.ol> <02fafe43-51d3-60e9-f57b-99000635182c@gmail.com> Message-ID: On 11/06/2022 09:40, Aliyan Navaid wrote: > To be honest every AI project I've seen amazes me whether it is about > finding insights in data, NLP, or Image generation. However, could you list > down stuff (in order) that is essential to all these applications? > I see Avi has already posted a reply suggesting you won't get your answes here. That is true in terms of the ultimate and specific answers to your question however I do think its a fair question for a beginner to ask here. The reason is, and your question and follow-up suggest it is true of you too, that most beginners see an interesting field but have no idea of the complexity involved. In your case AI is a whole sub genre of computing with many specialisms within it. Until you know which specific sub-branch you want to explore it is impossible to give specific advice. What we can say is a that a good knowledge of general programming techniques will never go amiss and Functional Programming (which is not the same thing as just writing/using functions!) and the use of classes and objects are both worth studying. I don't think you need to study the full OOP paradigm since most AI projects seem to be more likely to be functional in nature with just a few classes or objects used as data carriers. This list and other python resources are certainly able to help with that. Other areas of general programming that may be worth considering include data storage(RDBMS, NO-SQL, and other "Big Data" solutions) as well as generic algorithms for searching/sorting etc. Wikipedia is probably a good starting place for these areas. Once you get into AI domain specific subjects you will probably find domain-specific fora that are better placed to help. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos From avi.e.gross at gmail.com Sat Jun 11 17:42:06 2022 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Sat, 11 Jun 2022 17:42:06 -0400 Subject: [Tutor] Learning Object Oriented Programming In-Reply-To: References: <594D68F1-6265-4F6B-A95F-61BE6E85C7CF@hxcore.ol> <02fafe43-51d3-60e9-f57b-99000635182c@gmail.com> Message-ID: <00e301d87ddc$191540d0$4b3fc270$@gmail.com> Alan, My first post was moderated and I have not received a copy here showing me it was shared. My main point was not that some aspects could not be discussed here but that others would be better done elsewhere or 1:1 with someone as this forum is dedicated for other purposes. I am adding or expanding on some of what you wrote as I have no disagreement. The part I found relevant was asking what aspects of Python the language to focus on for some purpose, albeit I find such questions hard to answer. Should you learn all about using sets in Python? Some people never find a need to use them and many languages have no such construct. But not learning them may confuse you later when you encounter what to you looks like a very badly formed dictionary that seems to be missing values, or is it keys? AI is not only a very broad field but a moving target that also keeps evolving. If you are interested in the AI I was studying while working on a Ph.D. in AI at Rutgers in the early 80's, it is barely recognizable and definitely did not use Python. Much of what was seen as new and interesting ended up migrating out from the umbrella of AI into other aspects of computer science and has become so commonplace that it is not considered challenging any more. Other aspects have fallen out of favor or been shown to not be a great approach. I have revisited many aspects of what is now AI in recent years and am especially focused on many kinds of statistical methods that overlap AI and I Machine Learning. Much of that is done using Python extensions but I have done much work you would consider to be aspects of AI in other languages such as R or stand-alone applications. So the question here struck me as having several parts and that the person asking was either not very clear in their mind and sort of hoping to find some direction to get started. And, of course, sometimes it may simply be that they are just not that proficient yet in English. So if the question is whether learning a bit about Object-Oriented techniques in Python if your plan is to study AI, then the answer might be YES. But only up to a point. No need to be an expert at first. Many aspects of AI are mostly done using modules and packages others have already written and often tested. They come with pre-made objects that often are usable as black boxes with details well hidden within and the manuals tell you how to access or modify them, albeit good ones often hide much of that too and provide you with all the interfaces you normally need. Of course if you later want to tweak some things, you may need more, let alone if you want to share your own creations. Consider one of many scenarios in the scikit-learn module. You may want to do SOMETHING (and I am being abstract) that has variations. Typically you ask for a new object to be created that encapsulates what you want to do. You call code like: MyProblemSolver = doSomething(initializers) You now have an object which has used one of the initializers defined within that you do not need to see anything unless you feed it something wrong. It can set all kinds of parameters, maybe check validity, maybe creates accessory objects within and who knows what. You may also tweak it using methods supplied in additional lines of code like: MyProblemSolver.dosomething(...) I mean objectName.fit(...), objectName.transform(...), objectName.predict(...) and so on. One key part is giving it the data you want to use. For that, you may end up calling it with a method you hand something to that can range from a python list or some exotic object like a numpy Series or pandas DataFrame and so on. There are quite a few such object types used for various purposes and you can make many of them and tweak each and compare results and so on. But you can use many of these tools without having the slightest idea what an object is. A well designed tool may be easy to use by setting reasonable defaults so if you want one of many other possible tweaks, like adjusting the level of an alpha variable or choosing a different algorithm for some part of a transaction by name or even supplying your own function for use in comparing two things internally, you can tweak it quite a bit and even make your own customizations. Beginners rarely need to. As I mentioned, if you want to do something with some Machine Learning such as a sort of neural network, a beginner is often better off skipping learning powerful multi-purpose packages like TensorFlow and going straight to Keras which is built on top of TensorFlow and pandas and other things and largely removes you even further from seeing the endless details beneath BUT may let you dip lower if you wish. Much depends on whether you are just interested in the topic, are a student hoping to major in something, or even looking at specific jobs. Many skills are not worthwhile to spend lots of time on except as a hobby. And, as noted, some of the modules written that encapsulate TensorFlow have a rather different but still pythonic aspect as what they do is less object-oriented and more function-oriented as you add layers by encapsulating functions within functions. To use those you need to focus on other aspects of python. But here is an important point. Generally you do NOT do machine learning or other forms of AI alone. You often do lots of things in base python or other modules to read in the data, clean it up or do calculations on it, perhaps present graphics about it, and much more. Then you do one or more analyses as described above and often do more calculations on the various parts created in the process such as combining several and choosing which model made better results on your test cases and perhaps more analyses and graphs and ... So your overall knowledge of other aspects of python and the many modules people like for these tasks is important. I personally like to do some things using other languages and tools and often do programming using environments that create documents (as in a PDF or HTML or WORD documents) that include marking up all kinds of text as well as including snippets of python and R code that inter-operate with each other on the same data where each does what it is good at or provides what I like. This is what happens not at the beginning of a learning process but way later. You sound like you need some baby steps headed in the right direction. So, as others have said, learn enough about python without worrying if you have mastered ALL the many ways to create formatted text and so on. I suggest that much of AI is data intensive and that learning numpy decently (but not expertly) should come next. Depending on your future needs, pandas would come next, or might be studied only if it ever is needed. You can always come back and learn missing things such as how to use sets. Man/most languages have no such thing built in and yet people make do using things like vectors of items along with abilities to make the contents unique() or do various union() and intersection() and setdiff() operations OR creating your own object-like creations or using packages from others. But you do not need to do this linearly. You can use later stages to help guide you by seeing what code they discuss that you have not understood. You can fill in the gaps. And you would be amazed how much (along with misleading nonsense) you can pick up with a web search or two if you include things like the name of the language and module as in "python numpy remove duplicates" or whatever you can phrase well enough. Frankly, it can be a mistake to study python in a multi-volume way when the vast majority of what you learn is overkill. An intro book or two and maybe something intermediate but focused such as found using "book machine learning in python" or one about graphics and python may let you make progress. Now if you have specific questions about some code you read about use of objects and want to understand, this could be the place, but not if you have read nothing and want one on one attention to learn as that may require a real full-time dedicated person as a tutor. Again, good luck. Questions deeper than basic python, though, often have a better place, or even better, find resources you can read and practice with on your own as such learning can be longer-lasting. -----Original Message----- From: Tutor On Behalf Of Alan Gauld via Tutor Sent: Saturday, June 11, 2022 7:56 AM To: tutor at python.org Subject: Re: [Tutor] Learning Object Oriented Programming On 11/06/2022 09:40, Aliyan Navaid wrote: > To be honest every AI project I've seen amazes me whether it is about > finding insights in data, NLP, or Image generation. However, could you > list down stuff (in order) that is essential to all these applications? > I see Avi has already posted a reply suggesting you won't get your answes here. That is true in terms of the ultimate and specific answers to your question however I do think its a fair question for a beginner to ask here. The reason is, and your question and follow-up suggest it is true of you too, that most beginners see an interesting field but have no idea of the complexity involved. In your case AI is a whole sub genre of computing with many specialisms within it. Until you know which specific sub-branch you want to explore it is impossible to give specific advice. What we can say is a that a good knowledge of general programming techniques will never go amiss and Functional Programming (which is not the same thing as just writing/using functions!) and the use of classes and objects are both worth studying. I don't think you need to study the full OOP paradigm since most AI projects seem to be more likely to be functional in nature with just a few classes or objects used as data carriers. This list and other python resources are certainly able to help with that. Other areas of general programming that may be worth considering include data storage(RDBMS, NO-SQL, and other "Big Data" solutions) as well as generic algorithms for searching/sorting etc. Wikipedia is probably a good starting place for these areas. Once you get into AI domain specific subjects you will probably find domain-specific fora that are better placed to help. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos _______________________________________________ Tutor maillist - Tutor at python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor From rakesh7biswas at gmail.com Sat Jun 11 21:41:49 2022 From: rakesh7biswas at gmail.com (Rakesh Biswas) Date: Sun, 12 Jun 2022 07:11:49 +0530 Subject: [Tutor] Learning Object Oriented Programming In-Reply-To: <00e301d87ddc$191540d0$4b3fc270$@gmail.com> References: <594D68F1-6265-4F6B-A95F-61BE6E85C7CF@hxcore.ol> <02fafe43-51d3-60e9-f57b-99000635182c@gmail.com> <00e301d87ddc$191540d0$4b3fc270$@gmail.com> Message-ID: It was shared to this list and it was a great email but this one was even better. best, rb On Sun, Jun 12, 2022, 4:14 AM wrote: > Alan, > > My first post was moderated and I have not received a copy here showing me > it was shared. My main point was not that some aspects could not be > discussed here but that others would be better done elsewhere or 1:1 with > someone as this forum is dedicated for other purposes. I am adding or > expanding on some of what you wrote as I have no disagreement. > > The part I found relevant was asking what aspects of Python the language to > focus on for some purpose, albeit I find such questions hard to answer. > > Should you learn all about using sets in Python? Some people never find a > need to use them and many languages have no such construct. But not > learning > them may confuse you later when you encounter what to you looks like a very > badly formed dictionary that seems to be missing values, or is it keys? > > AI is not only a very broad field but a moving target that also keeps > evolving. If you are interested in the AI I was studying while working on a > Ph.D. in AI at Rutgers in the early 80's, it is barely recognizable and > definitely did not use Python. Much of what was seen as new and interesting > ended up migrating out from the umbrella of AI into other aspects of > computer science and has become so commonplace that it is not considered > challenging any more. Other aspects have fallen out of favor or been shown > to not be a great approach. > > I have revisited many aspects of what is now AI in recent years and am > especially focused on many kinds of statistical methods that overlap AI and > I Machine Learning. Much of that is done using Python extensions but I > have > done much work you would consider to be aspects of AI in other languages > such as R or stand-alone applications. > > So the question here struck me as having several parts and that the person > asking was either not very clear in their mind and sort of hoping to find > some direction to get started. And, of course, sometimes it may simply be > that they are just not that proficient yet in English. > > So if the question is whether learning a bit about Object-Oriented > techniques in Python if your plan is to study AI, then the answer might be > YES. But only up to a point. No need to be an expert at first. > > Many aspects of AI are mostly done using modules and packages others have > already written and often tested. They come with pre-made objects that > often > are usable as black boxes with details well hidden within and the manuals > tell you how to access or modify them, albeit good ones often hide much of > that too and provide you with all the interfaces you normally need. Of > course if you later want to tweak some things, you may need more, let alone > if you want to share your own creations. > > Consider one of many scenarios in the scikit-learn module. You may want to > do SOMETHING (and I am being abstract) that has variations. > > Typically you ask for a new object to be created that encapsulates what you > want to do. You call code like: > > MyProblemSolver = doSomething(initializers) > > You now have an object which has used one of the initializers defined > within > that you do not need to see anything unless you feed it something wrong. It > can set all kinds of parameters, maybe check validity, maybe creates > accessory objects within and who knows what. > > You may also tweak it using methods supplied in additional lines of code > like: > > MyProblemSolver.dosomething(...) > > I mean objectName.fit(...), objectName.transform(...), > objectName.predict(...) and so on. > > One key part is giving it the data you want to use. For that, you may end > up > calling it with a method you hand something to that can range from a python > list or some exotic object like a numpy Series or pandas DataFrame and so > on. > > There are quite a few such object types used for various purposes and you > can make many of them and tweak each and compare results and so on. > > But you can use many of these tools without having the slightest idea what > an object is. A well designed tool may be easy to use by setting reasonable > defaults so if you want one of many other possible tweaks, like adjusting > the level of an alpha variable or choosing a different algorithm for some > part of a transaction by name or even supplying your own function for use > in > comparing two things internally, you can tweak it quite a bit and even make > your own customizations. Beginners rarely need to. > > As I mentioned, if you want to do something with some Machine Learning such > as a sort of neural network, a beginner is often better off skipping > learning powerful multi-purpose packages like TensorFlow and going straight > to Keras which is built on top of TensorFlow and pandas and other things > and > largely removes you even further from seeing the endless details beneath > BUT > may let you dip lower if you wish. > > Much depends on whether you are just interested in the topic, are a student > hoping to major in something, or even looking at specific jobs. Many skills > are not worthwhile to spend lots of time on except as a hobby. > > And, as noted, some of the modules written that encapsulate TensorFlow have > a rather different but still pythonic aspect as what they do is less > object-oriented and more function-oriented as you add layers by > encapsulating functions within functions. To use those you need to focus > on > other aspects of python. > > But here is an important point. Generally you do NOT do machine learning or > other forms of AI alone. You often do lots of things in base python or > other > modules to read in the data, clean it up or do calculations on it, perhaps > present graphics about it, and much more. Then you do one or more analyses > as described above and often do more calculations on the various parts > created in the process such as combining several and choosing which model > made better results on your test cases and perhaps more analyses and graphs > and ... > > So your overall knowledge of other aspects of python and the many modules > people like for these tasks is important. I personally like to do some > things using other languages and tools and often do programming using > environments that create documents (as in a PDF or HTML or WORD documents) > that include marking up all kinds of text as well as including snippets of > python and R code that inter-operate with each other on the same data where > each does what it is good at or provides what I like. This is what happens > not at the beginning of a learning process but way later. You sound like > you need some baby steps headed in the right direction. > > So, as others have said, learn enough about python without worrying if you > have mastered ALL the many ways to create formatted text and so on. I > suggest that much of AI is data intensive and that learning numpy decently > (but not expertly) should come next. Depending on your future needs, pandas > would come next, or might be studied only if it ever is needed. You can > always come back and learn missing things such as how to use sets. Man/most > languages have no such thing built in and yet people make do using things > like vectors of items along with abilities to make the contents unique() or > do various union() and intersection() and setdiff() operations OR creating > your own object-like creations or using packages from others. > > But you do not need to do this linearly. You can use later stages to help > guide you by seeing what code they discuss that you have not understood. > You > can fill in the gaps. And you would be amazed how much (along with > misleading nonsense) you can pick up with a web search or two if you > include > things like the name of the language and module as in "python numpy remove > duplicates" or whatever you can phrase well enough. Frankly, it can be a > mistake to study python in a multi-volume way when the vast majority of > what > you learn is overkill. An intro book or two and maybe something > intermediate > but focused such as found using "book machine learning in python" or one > about graphics and python may let you make progress. > > Now if you have specific questions about some code you read about use of > objects and want to understand, this could be the place, but not if you > have > read nothing and want one on one attention to learn as that may require a > real full-time dedicated person as a tutor. > > Again, good luck. Questions deeper than basic python, though, often have a > better place, or even better, find resources you can read and practice with > on your own as such learning can be longer-lasting. > > > -----Original Message----- > From: Tutor On Behalf Of > Alan Gauld via Tutor > Sent: Saturday, June 11, 2022 7:56 AM > To: tutor at python.org > Subject: Re: [Tutor] Learning Object Oriented Programming > > On 11/06/2022 09:40, Aliyan Navaid wrote: > > To be honest every AI project I've seen amazes me whether it is about > > finding insights in data, NLP, or Image generation. However, could you > > list down stuff (in order) that is essential to all these applications? > > > > I see Avi has already posted a reply suggesting you won't get your answes > here. That is true in terms of the ultimate and specific answers to your > question however I do think its a fair question for a beginner to ask here. > The reason is, and your question and follow-up suggest it is true of you > too, that most beginners see an interesting field but have no idea of the > complexity involved. > In your case AI is a whole sub genre of computing with many specialisms > within it. Until you know which specific sub-branch you want to explore it > is impossible to give specific advice. > > What we can say is a that a good knowledge of general programming > techniques > will never go amiss and Functional Programming (which is not the same thing > as just writing/using functions!) and the use of classes and objects are > both worth studying. I don't think you need to study the full OOP paradigm > since most AI projects seem to be more likely to be functional in nature > with just a few classes or objects used as data carriers. This list and > other python resources are certainly able to help with that. > > Other areas of general programming that may be worth considering include > data storage(RDBMS, NO-SQL, and other "Big Data" solutions) as well as > generic algorithms for searching/sorting etc. Wikipedia is probably a good > starting place for these areas. > > Once you get into AI domain specific subjects you will probably find > domain-specific fora that are better placed to help. > > > -- > Alan G > Author of the Learn to Program web site > http://www.alan-g.me.uk/ > http://www.amazon.com/author/alan_gauld > Follow my photo-blog on Flickr at: > http://www.flickr.com/photos/alangauldphotos > > > _______________________________________________ > Tutor maillist - Tutor at python.org > To unsubscribe or change subscription options: > https://mail.python.org/mailman/listinfo/tutor > > _______________________________________________ > Tutor maillist - Tutor at python.org > To unsubscribe or change subscription options: > https://mail.python.org/mailman/listinfo/tutor > From anmol.ece at gmail.com Sun Jun 12 00:10:40 2022 From: anmol.ece at gmail.com (Anmol Sharma) Date: Sun, 12 Jun 2022 09:40:40 +0530 Subject: [Tutor] Issue with running iPython Message-ID: I just installed python on win 10 and installed Anaconda distribution. I get the following error while using ipython.Python error - self._poll(timeout) RuntimeError: still has pending operation at deallocation, process may crash This error is coming without any code also. SO, if i just load ipython and it taken me to ipython prompt, then If I just press "enter" multiple times, it gives this error. I also imported the necessary packages - import platform, asyncio if platform.system() == 'Windows': asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy()) And then I just load ipython and without even writing any line of code, I just press enter 2-3 times and I get this error - (base) C:\Users\03987N744\Training_notebook>ipython -i --no-banner enumerate_vs_range.py In [1]: In [1]: In [1]: Traceback (most recent call last): File "C:\Users\Anaconda3\lib\asyncio\windows_events.py", line 439, in select self._poll(timeout) RuntimeError: <_overlapped.Overlapped object at 0x0000022127E19DB0> still has pending operation at deallocation, the process may crash -- Also, this seems to be specific to *i*python and jupyter notebooks... It does not occur on python.... Regards Anmol Sharma From alan.gauld at yahoo.co.uk Sun Jun 12 07:04:46 2022 From: alan.gauld at yahoo.co.uk (Alan Gauld) Date: Sun, 12 Jun 2022 12:04:46 +0100 Subject: [Tutor] Issue with running iPython In-Reply-To: References: Message-ID: On 12/06/2022 05:10, Anmol Sharma wrote: > And then I just load ipython and without even writing any line of code, I > just press enter 2-3 times and I get this error - > > (base) C:\Users\03987N744\Training_notebook>ipython -i --no-banner > enumerate_vs_range.py What happens if you start ipython without the enumerate_vs_range.py file? Does it work OK then? And what happens if you start it without the -i and --no_banner flags? That might help isolate where the error is coming from. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos From nathan-tech at hotmail.com Tue Jun 14 01:31:56 2022 From: nathan-tech at hotmail.com (Nathan Smith) Date: Tue, 14 Jun 2022 06:31:56 +0100 Subject: [Tutor] writing to an address in memory Message-ID: Hi! I'm working with a library which written in C and has a callback function for listening for data, basically. Essentially any time it wants new data, it calls the callback with arguments: callback(handle, buffer, length) Where bufffer is a pointer to the buffer to write the data to. I know how to read the data (using ctypes.from_address) but I don't know how to write data to it? I've tried Googling, but I think my research is being hampered that I may not be using the correct termonology? The library I am working with is: http://www.un4seen.com/doc/#bass/STREAMPROC.html -- Best Wishes, Nathan Smith, BSC My Website: https://nathantech.net From wlfraed at ix.netcom.com Tue Jun 14 11:37:35 2022 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Tue, 14 Jun 2022 11:37:35 -0400 Subject: [Tutor] writing to an address in memory References: Message-ID: <15ahahdcdq4imse4ab8l9052kq22546dem@4ax.com> On Tue, 14 Jun 2022 06:31:56 +0100, Nathan Smith declaimed the following: >Hi! > > >I'm working with a library which written in C and has a callback >function for listening for data, basically. > >Essentially any time it wants new data, it calls the callback with >arguments: > >callback(handle, buffer, length) > >Where bufffer is a pointer to the buffer to write the data to. > > >I know how to read the data (using ctypes.from_address) but I don't know >how to write data to it? Do https://docs.python.org/3/library/ctypes.html#passing-pointers-or-passing-parameters-by-reference https://docs.python.org/3/library/ctypes.html#ctypes.byref https://docs.python.org/3/library/ctypes.html#callback-functions provide any hints (caveat: I've not used ctypes, I just read documentation ) -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From phillor9 at gmail.com Sat Jun 18 01:04:50 2022 From: phillor9 at gmail.com (Phil) Date: Sat, 18 Jun 2022 15:04:50 +1000 Subject: [Tutor] Event loop logic question Message-ID: Thank you for reading this. What I'm trying to achieve here is have the word 'stop' printed once whenever no keys are pressed and 'forward' printed once even when the up arrow key is held down. I don't think this question needs any knowledge of Pygame Zero to solve and it's quite likely that I've? misunderstood the logic required. The following code is my most recent version which seems to work the first time through the loop but fails after subsequent 'up' key presses. Debugging the event loop is difficult. Any hints or alternative methods will be, as always, greatly appreciated. By the way, this project sends commands to a controller which is easily overwhelmed if too many commands are received per second. import pgzrun WIDTH = 20 HEIGHT = 20 direction = 'stop' stop_flag = True forward_flag = True def on_key_down(key): ??? global direction ??? if key == keys.UP: ??????? print('up pressed') ??????? direction = 'forward' ??????? forward_flag = True ??????? print('forward_flag ', forward_flag) ??? elif key == keys.SPACE: ??????? direction = 'quit' def on_key_up(key): ??? global direction ??? #print('stop') ??? direction = 'stop' ??? stop_flag = True def update(): ??? global direction, stop_flag, forward_flag ??? if direction == 'stop' and stop_flag: ??????? print('stop') ??????? stop_flag = False ??? elif direction == 'forward' and forward_flag: ??????? print('forward') ??????? forward_flag = False ??????? stop_flag = True ??? elif direction == 'quit': ??????? quit() pgzrun.go() -- Regards, Phil From PythonList at DancesWithMice.info Sat Jun 18 01:14:26 2022 From: PythonList at DancesWithMice.info (dn) Date: Sat, 18 Jun 2022 17:14:26 +1200 Subject: [Tutor] Event loop logic question In-Reply-To: References: Message-ID: On 18/06/2022 17.04, Phil wrote: > Thank you for reading this. > > What I'm trying to achieve here is have the word 'stop' printed once > whenever no keys are pressed and 'forward' printed once even when the up > arrow key is held down. > > I don't think this question needs any knowledge of Pygame Zero to solve > and it's quite likely that I've? misunderstood the logic required. The > following code is my most recent version which seems to work the first > time through the loop but fails after subsequent 'up' key presses. > Debugging the event loop is difficult. > > Any hints or alternative methods will be, as always, greatly appreciated. > > By the way, this project sends commands to a controller which is easily > overwhelmed if too many commands are received per second. You may like to study the games (start with snake) at https://simplegametutorials.github.io/pygamezero/ to see how they do it. -- Regards, =dn From phillor9 at gmail.com Sat Jun 18 01:45:57 2022 From: phillor9 at gmail.com (Phil) Date: Sat, 18 Jun 2022 15:45:57 +1000 Subject: [Tutor] Event loop logic question In-Reply-To: References: Message-ID: <13dd409b-5240-0a03-6d22-883a63f5684a@gmail.com> On 18/6/22 15:14, dn wrote: > > You may like to study the games (start with snake) at > https://simplegametutorials.github.io/pygamezero/ to see how they do it. Thanks dn. I had already looked through the examples but they didn't seem to relate to what I'm trying to achieve. I'll have another look. -- Regards, Phil From alan.gauld at yahoo.co.uk Sat Jun 18 06:05:10 2022 From: alan.gauld at yahoo.co.uk (Alan Gauld) Date: Sat, 18 Jun 2022 11:05:10 +0100 Subject: [Tutor] Event loop logic question In-Reply-To: References: Message-ID: On 18/06/2022 06:04, Phil wrote: > Thank you for reading this. > > What I'm trying to achieve here is have the word 'stop' printed once > whenever no keys are pressed and 'forward' printed once even when the up > arrow key is held down. I understand the second part, but not the first. How do you determine that no keys are pressed - how you detect the absence of an event? Usually there is some kind of dummy "idle" event that you could use to trigger it, but I don't know your toolkit. However you seem to be using the key_up event which will trigger between every key press whioch is not, I think, what you want? > I don't think this question needs any knowledge of Pygame Zero to solve On the contrary, knowing what events are available from the toolkit seems to be crucial to how you solve this. I can do it in tkinter (I think!) but have no idea about PygameZero, which i'd not heard of till now! > following code is my most recent version which seems to work the first > time through the loop but fails after subsequent 'up' key presses. Can you be more specific? What does "fail" mean? What does it actually do? Does it print anything? Print too much(of what?) or crash? Is there an error message? > Debugging the event loop is difficult. The usual way is to insert print satements into the event handlers or to catch the event handlers in a debugger using breakpoints. > direction = 'stop' > stop_flag = True > forward_flag = True > > def on_key_down(key): > ??? global direction > > ??? if key == keys.UP: > ??????? print('up pressed') > ??????? direction = 'forward' > ??????? forward_flag = True > ??????? print('forward_flag ', forward_flag) Its usual practice not to do UI changes(ie print) in the event handlers but to do that in the draw/update/refresh method. Just set the values in the handlers. Another question - what should happen if another key other than UP (or space) is pressed? Would you want to cancel the forward flag? > ??? elif key == keys.SPACE: > ??????? direction = 'quit' Why set direction to quit? It's not really a direction. Why not just quit right here? If you did want a farewell message then fair enough, although I'd tend to use a flag rather than abuse direction... > > def on_key_up(key): > ??? global direction > > ??? #print('stop') > ??? direction = 'stop' > ??? stop_flag = True In other words should forward_flag be set to false here? > def update(): > ??? global direction, stop_flag, forward_flag > > ??? if direction == 'stop' and stop_flag: > ??????? print('stop') > ??????? stop_flag = False > > ??? elif direction == 'forward' and forward_flag: > ??????? print('forward') > ??????? forward_flag = False > ??????? stop_flag = True > > ??? elif direction == 'quit': > ??????? quit() This feels wrong to me. I'd expect it to look more like: def update() if direction == forward and not forward_flag print direction forward_flag = True # only set flag after first print elif direction == stop and not stop_flag print direction stop_flag = true # same again elif direction == quit # do you really want this in update? quit I hope these random musings are helpful... My main suggestions are: Take the print and setting of the flags out of the handlers and put that in update. Only set direction in the handlers. And also make sure you set all of the state flags to their correct values when you change state. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos From avi.e.gross at gmail.com Sat Jun 18 01:22:57 2022 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Sat, 18 Jun 2022 01:22:57 -0400 Subject: [Tutor] Event loop logic question In-Reply-To: References: Message-ID: <050f01d882d3$789af080$69d0d180$@gmail.com> Phil, I am not familiar with the package you are using and have to assume you have some code somewhere not shown that binds your functions to the task as event handlers. But can multiple functions be called and active at the same time. I do not see any kind of lock in place such as a semaphore that stops two or more changes to the global variables as might happen if an arrow key is held down or whatever. And your on_key_up() function does not make stop_flag global and so any changes it makes are to a local variable that immediately disappears. It is not my place to mention that this may be an advanced question and someone may want to direct you to resources showing the care that must be taken when a global variable is being handled by multiple threads. Ditto for forward_flag in on_key_down() -----Original Message----- From: Tutor On Behalf Of Phil Sent: Saturday, June 18, 2022 1:05 AM To: tutor at python.org Subject: [Tutor] Event loop logic question Thank you for reading this. What I'm trying to achieve here is have the word 'stop' printed once whenever no keys are pressed and 'forward' printed once even when the up arrow key is held down. I don't think this question needs any knowledge of Pygame Zero to solve and it's quite likely that I've misunderstood the logic required. The following code is my most recent version which seems to work the first time through the loop but fails after subsequent 'up' key presses. Debugging the event loop is difficult. Any hints or alternative methods will be, as always, greatly appreciated. By the way, this project sends commands to a controller which is easily overwhelmed if too many commands are received per second. import pgzrun WIDTH = 20 HEIGHT = 20 direction = 'stop' stop_flag = True forward_flag = True def on_key_down(key): global direction if key == keys.UP: print('up pressed') direction = 'forward' forward_flag = True print('forward_flag ', forward_flag) elif key == keys.SPACE: direction = 'quit' def on_key_up(key): global direction #print('stop') direction = 'stop' stop_flag = True def update(): global direction, stop_flag, forward_flag if direction == 'stop' and stop_flag: print('stop') stop_flag = False elif direction == 'forward' and forward_flag: print('forward') forward_flag = False stop_flag = True elif direction == 'quit': quit() pgzrun.go() -- Regards, Phil _______________________________________________ Tutor maillist - Tutor at python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor From wlfraed at ix.netcom.com Sat Jun 18 15:19:39 2022 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Sat, 18 Jun 2022 15:19:39 -0400 Subject: [Tutor] Event loop logic question References: Message-ID: On Sat, 18 Jun 2022 15:04:50 +1000, Phil declaimed the following: >Any hints or alternative methods will be, as always, greatly appreciated. > >By the way, this project sends commands to a controller which is easily >overwhelmed if too many commands are received per second. > For the last matter, one solution would be to use an elapsed time counter. rate = 1.0 / commands_per_second last_command_time = 0.0 #or get the time (fractional seconds) #at start of program Then, when you issue a command, some logic like if last_command_time + rate < time_now: issue_command last_command_time = time_now ... optional if you return a status that "command skipped -- rate limited" For the former/general... I see too many flags that should (as I understand it) be mutually exclusive (only one should be set/cleared [depending on overall logic] at any moment in the program). Instead I'd suggest using a "state" variable (maybe check on what Python has for "enums" -- I've not used them in my dribblings, so the following is just using "constants" STATE_QUIT, STATE_FORWARD, STATE_STOPPED = range(3) current_state = STATE_STOPPED requested_state = STATE_STOPPED ... key-press if key == keys.UP: requested_state = STATE_FORWARD elif key == keys.SPACE: requested_state = STATE_QUIT else: pass #log an unused key press ... key-release requested_state = STATE_STOPPED ... update section if requested_state != current_state: #if the requested state is the same as current_state, #it may signal a repeating key (but only if the OS doesn't #issue a key release and key press at the start of each repeat -- #if it does -- you'll need to incorporate some nasty rate timer #to determine what is a repeat vs a real release/press) if requested_state == STATE_QUIT: #initiate shutdown? #log change? elif requested_state == STATE_FORWARD: #issue forward command to device #log change elif requested_state == STATE_STOPPED: #issue stop command to device #log change else: #unexpected STATE! current_state = requested_state #update state -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From tempest.amogh at gmail.com Sun Jun 19 06:13:32 2022 From: tempest.amogh at gmail.com (tempest.amogh ) Date: Sun, 19 Jun 2022 15:43:32 +0530 Subject: [Tutor] Help with brute force algorithm Message-ID: <78101919-6461-4009-b172-8996e3b23304@Canary> As a personal project, I was inspired by Erik Demaine. He has created a font based on tetris shapes [http://erikdemaine.org/fonts/], however I would like to emulate this but based on another reference. I want to write an algorithm that brute force solves all the positions for these pieces in the english alphabet. I am having trouble making a grid that can be solved with these pieces, despite a book saying so. I have attached the book that shows the shapes, and some pseudocode. I would appreciate any possible guidance. Thank you very much :) ? Regards, Amogh Atwe -------------- next part -------------- program start import tetrominoes import trominoes import dominoes input("Please write a word that you would like to see written").upper() function(input): for input{letter from alphabet}: backtracking algorithm, work from left to right: from filled letter to respective tetrominoes place(tetromino): for: if tetromino does not fit in grid: place in checked position: place(tetromino) repeat until pattern is found for letter, continue with rest of the letters print(letter from alphabet) in respective position with tetrominoes, trominoes and dominoes until no more letters output (GUI of written word in combination of tetrominoes, trominoes, dominoes) From manpritsinghece at gmail.com Sun Jun 19 13:11:43 2022 From: manpritsinghece at gmail.com (Manprit Singh) Date: Sun, 19 Jun 2022 22:41:43 +0530 Subject: [Tutor] make a sqlite3 database from an ordinary text file Message-ID: Dear Sir, I was trying to make a sqlite3 data base from an ordinary text file named abcd.txt on my computer. it has data as given below in the form of 2 columns Time Pieces 1 10 2 15 3 25 4 31 5 40 6 45 7 53 8 54 9 65 10 75 Below given is the code that makes sqlite3 database work.db, with a table named workdata, which will be having 2 columns Time & Pieces . The code lateron fetches all data from the table. import sqlite3 def data_generator(workfile): with open(workfile) as obj: obj.readline() for line in obj: yield tuple(line.split()) con = sqlite3.connect("work.db") cur = con.cursor() cur.execute("create table workdata(Time INT, Pieces INT)") cur.executemany("insert into workdata values (?, ?)", data_generator("abcd.txt")) con.commit() cur.execute("select Time, Pieces from workdata") print(cur.fetchall()) cur.close() con.close() In this example i am using generator function for iterator to be used in executemany(). I will not prefer a list comprehension in this case as a file may contain a large number of rows . Is there any better way to do this task ? Kindly suggest. Regards Manprit Singh From manpritsinghece at gmail.com Sun Jun 19 13:52:05 2022 From: manpritsinghece at gmail.com (Manprit Singh) Date: Sun, 19 Jun 2022 23:22:05 +0530 Subject: [Tutor] Fwd: make a sqlite3 database from an ordinary text file In-Reply-To: References: Message-ID: Dear sir , One more thing : def data_generator(workfile): with open(workfile) as obj: obj.readline() for line in obj: yield tuple(line.split()) Instead of yield tuple(line.split()) i can write yield line.split() too. Am i right ? ---------- Forwarded message --------- From: Manprit Singh Date: Sun, Jun 19, 2022 at 10:41 PM Subject: make a sqlite3 database from an ordinary text file To: Dear Sir, I was trying to make a sqlite3 data base from an ordinary text file named abcd.txt on my computer. it has data as given below in the form of 2 columns Time Pieces 1 10 2 15 3 25 4 31 5 40 6 45 7 53 8 54 9 65 10 75 Below given is the code that makes sqlite3 database work.db, with a table named workdata, which will be having 2 columns Time & Pieces . The code lateron fetches all data from the table. import sqlite3 def data_generator(workfile): with open(workfile) as obj: obj.readline() for line in obj: yield tuple(line.split()) con = sqlite3.connect("work.db") cur = con.cursor() cur.execute("create table workdata(Time INT, Pieces INT)") cur.executemany("insert into workdata values (?, ?)", data_generator("abcd.txt")) con.commit() cur.execute("select Time, Pieces from workdata") print(cur.fetchall()) cur.close() con.close() In this example i am using generator function for iterator to be used in executemany(). I will not prefer a list comprehension in this case as a file may contain a large number of rows . Is there any better way to do this task ? Kindly suggest. Regards Manprit Singh From alan.gauld at yahoo.co.uk Sun Jun 19 14:04:12 2022 From: alan.gauld at yahoo.co.uk (Alan Gauld) Date: Sun, 19 Jun 2022 19:04:12 +0100 Subject: [Tutor] make a sqlite3 database from an ordinary text file In-Reply-To: References: Message-ID: On 19/06/2022 18:11, Manprit Singh wrote: > cur.execute("create table workdata(Time INT, Pieces INT)") This will fail the second time it runs. You should check if the table already exists and either drop it or use the existing table. However, I rarely create tables from Python, it's generally easier to create a sql file and run that directly using the sqlite interpreter. You only need Python if the format of the table has to be deduced from the data. and that then leads to all sorts of follow-on issues over accessing names of columns etc. Once you have the table you can then use Python to load the data, although in your case you could do that directly from SQL too since it looks like a tab separated file. As always use the most appropriate tool. If you can do it directly in SQL there is no point in wrapping it in Python. That just adds complexity and slows things down. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos From alan.gauld at yahoo.co.uk Sun Jun 19 14:09:02 2022 From: alan.gauld at yahoo.co.uk (Alan Gauld) Date: Sun, 19 Jun 2022 19:09:02 +0100 Subject: [Tutor] Fwd: make a sqlite3 database from an ordinary text file In-Reply-To: References: Message-ID: On 19/06/2022 18:52, Manprit Singh wrote: > One more thing : > def data_generator(workfile): > with open(workfile) as obj: > obj.readline() > for line in obj: > yield tuple(line.split()) > > Instead of yield tuple(line.split()) i can write yield line.split() > too. > > Am i right ? It depends what you want. The first one gives a tuple back, the second gives a list. Often it won't make any difference but if you were using the value as a dictionary key, say, then only the first would work. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos From alan.gauld at yahoo.co.uk Sun Jun 19 14:07:01 2022 From: alan.gauld at yahoo.co.uk (Alan Gauld) Date: Sun, 19 Jun 2022 19:07:01 +0100 Subject: [Tutor] Help with brute force algorithm In-Reply-To: <78101919-6461-4009-b172-8996e3b23304@Canary> References: <78101919-6461-4009-b172-8996e3b23304@Canary> Message-ID: On 19/06/2022 11:13, tempest.amogh wrote: > As a personal project, I was inspired by Erik Demaine. > He has created a font based on tetris shapes > > I am having trouble making a grid that can be solved > with these pieces, despite a book saying so. So show us the code and sample data and any error messages. > > I have attached the book that shows the shapes, and some pseudocode. Non-text attachments get stripped by the server for security reasons. Copy/paste the relevant text into the email or provide a link to a paste bin or other web site. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos From manpritsinghece at gmail.com Sun Jun 19 14:33:27 2022 From: manpritsinghece at gmail.com (Manprit Singh) Date: Mon, 20 Jun 2022 00:03:27 +0530 Subject: [Tutor] make a sqlite3 database from an ordinary text file In-Reply-To: References: Message-ID: Yes sir , I do agree it will fail for second time , if table exists . One treatment could be writing the command like create table table_name if not exist (--------) I do agree with your further comments too . Regards Manprit Singh On Sun, 19 Jun, 2022, 23:36 Alan Gauld via Tutor, wrote: > On 19/06/2022 18:11, Manprit Singh wrote: > > > cur.execute("create table workdata(Time INT, Pieces INT)") > > This will fail the second time it runs. You should check > if the table already exists and either drop it or use > the existing table. > > However, I rarely create tables from Python, it's generally > easier to create a sql file and run that directly using > the sqlite interpreter. You only need Python if the format > of the table has to be deduced from the data. and that then > leads to all sorts of follow-on issues over accessing names > of columns etc. > > Once you have the table you can then use Python to load > the data, although in your case you could do that directly > from SQL too since it looks like a tab separated file. > > As always use the most appropriate tool. If you can do it > directly in SQL there is no point in wrapping it in > Python. That just adds complexity and slows things > down. > > -- > Alan G > Author of the Learn to Program web site > http://www.alan-g.me.uk/ > http://www.amazon.com/author/alan_gauld > Follow my photo-blog on Flickr at: > http://www.flickr.com/photos/alangauldphotos > > > _______________________________________________ > Tutor maillist - Tutor at python.org > To unsubscribe or change subscription options: > https://mail.python.org/mailman/listinfo/tutor > From manpritsinghece at gmail.com Sun Jun 19 14:37:42 2022 From: manpritsinghece at gmail.com (Manprit Singh) Date: Mon, 20 Jun 2022 00:07:42 +0530 Subject: [Tutor] make a sqlite3 database from an ordinary text file In-Reply-To: References: Message-ID: Dear sir , One more query .... How can i use column name in text file as colum names in sqlite3 database . I tried using python but failed. Can you put some light over it ? Regards Manprit Singh On Sun, 19 Jun, 2022, 23:36 Alan Gauld via Tutor, wrote: > On 19/06/2022 18:11, Manprit Singh wrote: > > > cur.execute("create table workdata(Time INT, Pieces INT)") > > This will fail the second time it runs. You should check > if the table already exists and either drop it or use > the existing table. > > However, I rarely create tables from Python, it's generally > easier to create a sql file and run that directly using > the sqlite interpreter. You only need Python if the format > of the table has to be deduced from the data. and that then > leads to all sorts of follow-on issues over accessing names > of columns etc. > > Once you have the table you can then use Python to load > the data, although in your case you could do that directly > from SQL too since it looks like a tab separated file. > > As always use the most appropriate tool. If you can do it > directly in SQL there is no point in wrapping it in > Python. That just adds complexity and slows things > down. > > -- > Alan G > Author of the Learn to Program web site > http://www.alan-g.me.uk/ > http://www.amazon.com/author/alan_gauld > Follow my photo-blog on Flickr at: > http://www.flickr.com/photos/alangauldphotos > > > _______________________________________________ > Tutor maillist - Tutor at python.org > To unsubscribe or change subscription options: > https://mail.python.org/mailman/listinfo/tutor > From alan.gauld at yahoo.co.uk Sun Jun 19 15:16:39 2022 From: alan.gauld at yahoo.co.uk (Alan Gauld) Date: Sun, 19 Jun 2022 20:16:39 +0100 Subject: [Tutor] make a sqlite3 database from an ordinary text file In-Reply-To: References: Message-ID: On 19/06/2022 19:37, Manprit Singh wrote: > How can i use column name in text file as colum names in sqlite3 database . > I tried using python but failed. I'm not sure if this is what you mean but.... sqlite> .help import .import FILE TABLE Import data from FILE into TABLE Options: --ascii Use \037 and \036 as column and row separators --csv Use , and \n as column and row separators --skip N Skip the first N rows of input -v "Verbose" - increase auxiliary output Notes: * If TABLE does not exist, it is created. The first row of input determines the column names. * If neither --csv or --ascii are used, the input mode is derived from the ".mode" output mode * If FILE begins with "|" then it is a command that generates the input text. sqlite> -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos From wlfraed at ix.netcom.com Sun Jun 19 15:17:06 2022 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Sun, 19 Jun 2022 15:17:06 -0400 Subject: [Tutor] make a sqlite3 database from an ordinary text file References: Message-ID: On Mon, 20 Jun 2022 00:03:27 +0530, Manprit Singh declaimed the following: >Yes sir , >I do agree it will fail for second time , if table exists . > >One treatment could be writing the command like create table table_name if >not exist (--------) > The syntax for that is CREATE TABLE IF NOT EXISTS ... -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From wlfraed at ix.netcom.com Sun Jun 19 15:26:46 2022 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Sun, 19 Jun 2022 15:26:46 -0400 Subject: [Tutor] make a sqlite3 database from an ordinary text file References: Message-ID: <7ituahdsjskfm8pv9dn80sph2kiqrtevv5@4ax.com> On Mon, 20 Jun 2022 00:07:42 +0530, Manprit Singh declaimed the following: >Dear sir , > >One more query .... >How can i use column name in text file as colum names in sqlite3 database . >I tried using python but failed. Show us the code, and the traceback... Otherwise we have no real idea of what you might be doing... Not to mention -- you are opening yourself up for an SQL injection attack by doing what I think you mean. ANYTHING involving the database schema should NEVER rely upon user input values. Only the DATA, and that via the adapter's form of parameterization (SQLite3 using ?, others have different syntax). This way the adapter can do escaping and quoting if needed to ensure that the value is treated solely as data, and not something that could be interpreted as another SQL statement. If you need to generate SQL statements from user specified fields, you should present a menu to the user of the fields that are defined in the database, and use the user input (position/index #) to select from the known field names. -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From avi.e.gross at gmail.com Sun Jun 19 19:43:14 2022 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Sun, 19 Jun 2022 19:43:14 -0400 Subject: [Tutor] Fwd: make a sqlite3 database from an ordinary text file In-Reply-To: References: Message-ID: <007a01d88436$58d53da0$0a7fb8e0$@gmail.com> I may not be looking at this right since much happens in functions not seen but does cur.execute(many) expect two items of text or two items of type integer or will it handle a list or tuple with arbitrary numbers of contents and so on, is what makes the design decision. Deeper within that function, it presumable maps your passed function to code that may do, First, Second = f() And so on. Or it may just take a single result or other choices. It is a bit confusing to pass a filename as someone pointed out. In what follows, either I am confused or the one asking the question is. The goal of the iterator seems to be to avoid reading all the data in at once, right. Otherwise, why bother when you can do something like reading in the entire file and splitting and putting the results in something like a list structure, or perhaps in something from numpy. So why does the code use readline() (no 's') to read in a single line and only once? As I read the somewhat confusing code, the goal is to open the file the first time and read a line and throw the results away as it is not clear where it is saved. Then the goal is to loop on nothing (or maybe one line) and yield the results of splitting it and pause till called again. But since only no or maybe one line have been received, the call to iterate should just exit as the loop is done. So I think reading a single line should be moved down within the loop! And you need some kind of test to know when there are no remaining lines in the file or the current line is empty and instead of yielding a non-existent result, you should return what the iteration protocol needs and close the file, albeit the with does that for you silently as part of that protocol. But as others have noted, it is not clear other than as an exercise, if this iterator is needed. Your other modules may supply something for you. Where a generator like this is even more useful is when at some point you want to stop and perhaps resume a bit later elsewhere. For example, if your code wanted random numbers that had been precalculated and use it for various purposes until done, the iterator can in principle be called from multiple places and keep marching along and eventually stop without reading the entire file. Another example would be a generator that yields all primes needed in order, and generally that never goes long enough to run out of primes, LOL! -----Original Message----- From: Tutor On Behalf Of Alan Gauld via Tutor Sent: Sunday, June 19, 2022 2:09 PM To: tutor at python.org Subject: Re: [Tutor] Fwd: make a sqlite3 database from an ordinary text file On 19/06/2022 18:52, Manprit Singh wrote: > One more thing : > def data_generator(workfile): > with open(workfile) as obj: > obj.readline() > for line in obj: > yield tuple(line.split()) > > Instead of yield tuple(line.split()) i can write yield line.split() > too. > > Am i right ? It depends what you want. The first one gives a tuple back, the second gives a list. Often it won't make any difference but if you were using the value as a dictionary key, say, then only the first would work. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos _______________________________________________ Tutor maillist - Tutor at python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor From avi.e.gross at gmail.com Sun Jun 19 19:57:51 2022 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Sun, 19 Jun 2022 19:57:51 -0400 Subject: [Tutor] make a sqlite3 database from an ordinary text file In-Reply-To: References: Message-ID: <007e01d88438$63156090$294021b0$@gmail.com> Manprit, You may be approaching this topic differently than you could. What you keep calling a TEXT file is what others might call structured files like .CSV or .DAT types that contain values separated by some delimiter like a comma or a tab or arbitrary whitespace or other choices. Lots of python functionalities are available to read these in to structures used in modules like numpy and pandas that many people use on top of or alongside regular built-in python commands. In these, there can be (and usually are) HEADER names for columns and sometimes also rows. The functionality that reads in such tables, Dataframes, or whatever you want to call it, will often optionally take the first line of the file to have headers specifying names. And, you can always add or change the names as well. So if passing the entire structure to another function, it can choose to access the name of any column (or often row) when it wants. Or, if you need to, you can ask for it and pass it your way as in interpolating the name into an SQL command. You may be asking a slightly different question here about just your data. Does your "text" file contain a first line saying ""Time Pieces" or not? If it DOES, you may need to read it ONCE at the start before giving your iterator for the numbers to be read and store the results as your names to use as you wish. If the file has no header, then your code is the one attaching a name. To me the real issue is your design seems to be of a limited nature. You want to read in your data in a small batch (single line) and presumably are opening a connection to the database and doing login and so on, then sending a command to create a table, unless it already exists, then sending umpteen commands to insert yet another row into the table and then who knows? This seems like such a common operation that I have to guess there is a way to do it fairly trivially and my guess is way more efficiently. -----Original Message----- From: Tutor On Behalf Of Manprit Singh Sent: Sunday, June 19, 2022 2:38 PM Cc: tutor at python.org Subject: Re: [Tutor] make a sqlite3 database from an ordinary text file Dear sir , One more query .... How can i use column name in text file as colum names in sqlite3 database . I tried using python but failed. Can you put some light over it ? Regards Manprit Singh On Sun, 19 Jun, 2022, 23:36 Alan Gauld via Tutor, wrote: > On 19/06/2022 18:11, Manprit Singh wrote: > > > cur.execute("create table workdata(Time INT, Pieces INT)") > > This will fail the second time it runs. You should check if the table > already exists and either drop it or use the existing table. > > However, I rarely create tables from Python, it's generally easier to > create a sql file and run that directly using the sqlite interpreter. > You only need Python if the format of the table has to be deduced from > the data. and that then leads to all sorts of follow-on issues over > accessing names of columns etc. > > Once you have the table you can then use Python to load the data, > although in your case you could do that directly from SQL too since it > looks like a tab separated file. > > As always use the most appropriate tool. If you can do it directly in > SQL there is no point in wrapping it in Python. That just adds > complexity and slows things down. > > -- > Alan G > Author of the Learn to Program web site http://www.alan-g.me.uk/ > http://www.amazon.com/author/alan_gauld > Follow my photo-blog on Flickr at: > http://www.flickr.com/photos/alangauldphotos > > > _______________________________________________ > Tutor maillist - Tutor at python.org > To unsubscribe or change subscription options: > https://mail.python.org/mailman/listinfo/tutor > _______________________________________________ Tutor maillist - Tutor at python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor From manpritsinghece at gmail.com Sun Jun 19 20:59:58 2022 From: manpritsinghece at gmail.com (Manprit Singh) Date: Mon, 20 Jun 2022 06:29:58 +0530 Subject: [Tutor] make a sqlite3 database from an ordinary text file In-Reply-To: <007e01d88438$63156090$294021b0$@gmail.com> References: <007e01d88438$63156090$294021b0$@gmail.com> Message-ID: Dear all, Learning is a lifelong process . At this point i still do not feel i am absolutely ok with python or programming. In the past i was very active in this mailing list and will appreciate a few - Alan sir, Dennis Lee Bieber, dn and few others, they always answered my queries . As i wrote in the starting , i have a simple text file named abcd.txt. (I am not using a csv file or any other structured file) . The contents are given below : Time Pieces 1 10 2 15 3 25 4 31 5 40 6 45 7 53 8 54 9 65 10 75 We can clearly see the first line of the file is Time Pieces Now I have to read this text into an sqlite3 database table , So it is clear that the database table will have 2 columns, Column names of the sqlite3 table are to be picked from the first line of the text file abcd.txt. (First column name will be Time, second column name will be Pieces ). See i know this can be done very easily using numpy and Pandas. But I am trying to do it with Core Python only. cur.execute("create table workdata(? INT, ? INT)", ("Time", "Pieces")) I just want to insert column names (where these column names are read from text file abcd.txt 1st line) using a question mark style placeholder. Pls let me know if anything is possible . Time Pieces On Mon, Jun 20, 2022 at 5:34 AM wrote: > Manprit, > > You may be approaching this topic differently than you could. > > What you keep calling a TEXT file is what others might call structured > files > like .CSV or .DAT types that contain values separated by some delimiter > like > a comma or a tab or arbitrary whitespace or other choices. Lots of python > functionalities are available to read these in to structures used in > modules like numpy and pandas that many people use on top of or alongside > regular built-in python commands. In these, there can be (and usually are) > HEADER names for columns and sometimes also rows. The functionality that > reads in such tables, Dataframes, or whatever you want to call it, will > often optionally take the first line of the file to have headers specifying > names. And, you can always add or change the names as well. > > So if passing the entire structure to another function, it can choose to > access the name of any column (or often row) when it wants. Or, if you need > to, you can ask for it and pass it your way as in interpolating the name > into an SQL command. > > You may be asking a slightly different question here about just your data. > Does your "text" file contain a first line saying ""Time Pieces" or > not? > > If it DOES, you may need to read it ONCE at the start before giving your > iterator for the numbers to be read and store the results as your names to > use as you wish. If the file has no header, then your code is the one > attaching a name. > > To me the real issue is your design seems to be of a limited nature. You > want to read in your data in a small batch (single line) and presumably are > opening a connection to the database and doing login and so on, then > sending > a command to create a table, unless it already exists, then sending umpteen > commands to insert yet another row into the table and then who knows? This > seems like such a common operation that I have to guess there is a way to > do > it fairly trivially and my guess is way more efficiently. > > > > -----Original Message----- > From: Tutor On Behalf Of > Manprit Singh > Sent: Sunday, June 19, 2022 2:38 PM > Cc: tutor at python.org > Subject: Re: [Tutor] make a sqlite3 database from an ordinary text file > > Dear sir , > > One more query .... > How can i use column name in text file as colum names in sqlite3 database . > I tried using python but failed. > > Can you put some light over it ? > Regards > Manprit Singh > > On Sun, 19 Jun, 2022, 23:36 Alan Gauld via Tutor, > wrote: > > > On 19/06/2022 18:11, Manprit Singh wrote: > > > > > cur.execute("create table workdata(Time INT, Pieces INT)") > > > > This will fail the second time it runs. You should check if the table > > already exists and either drop it or use the existing table. > > > > However, I rarely create tables from Python, it's generally easier to > > create a sql file and run that directly using the sqlite interpreter. > > You only need Python if the format of the table has to be deduced from > > the data. and that then leads to all sorts of follow-on issues over > > accessing names of columns etc. > > > > Once you have the table you can then use Python to load the data, > > although in your case you could do that directly from SQL too since it > > looks like a tab separated file. > > > > As always use the most appropriate tool. If you can do it directly in > > SQL there is no point in wrapping it in Python. That just adds > > complexity and slows things down. > > > > -- > > Alan G > > Author of the Learn to Program web site http://www.alan-g.me.uk/ > > http://www.amazon.com/author/alan_gauld > > Follow my photo-blog on Flickr at: > > http://www.flickr.com/photos/alangauldphotos > > > > > > _______________________________________________ > > Tutor maillist - Tutor at python.org > > To unsubscribe or change subscription options: > > https://mail.python.org/mailman/listinfo/tutor > > > _______________________________________________ > Tutor maillist - Tutor at python.org > To unsubscribe or change subscription options: > https://mail.python.org/mailman/listinfo/tutor > > _______________________________________________ > Tutor maillist - Tutor at python.org > To unsubscribe or change subscription options: > https://mail.python.org/mailman/listinfo/tutor > From PythonList at DancesWithMice.info Sun Jun 19 22:39:14 2022 From: PythonList at DancesWithMice.info (dn) Date: Mon, 20 Jun 2022 14:39:14 +1200 Subject: [Tutor] make a sqlite3 database from an ordinary text file In-Reply-To: References: <007e01d88438$63156090$294021b0$@gmail.com> Message-ID: <9b0f5a6e-7f9d-1bef-8a71-658bb3b75d45@DancesWithMice.info> On 20/06/2022 12.59, Manprit Singh wrote: > Dear all, > Learning is a lifelong process . At this point i still do not feel i am > absolutely ok with python or programming. In the past i was very active in > this mailing list and will appreciate a few - Alan sir, Dennis Lee Bieber, > dn and few others, they always answered my queries . > As i wrote in the starting , i have a simple text file named abcd.txt. (I > am not using a csv file or any other structured file) . The contents are > given below : > > Time Pieces > 1 10 > 2 15 > 3 25 > 4 31 > 5 40 > 6 45 > 7 53 > 8 54 > 9 65 > 10 75 > > We can clearly see the first line of the file is > > Time Pieces but, as advised earlier, whilst "we can clearly see" is true IN THIS CASE; what happens when someone brings another table to the party? Ref: https://xkcd.com/327/ BTW my favorite joke from this genre is: ? = 3.141592653589793helpimtrappedinapiefactory7108914... I'm +1 after @Alan. Although we often see it in programming texts, I've NEVER initialised a DB/TBL this way. Easy for me to say: I learned RDBMS and SQL very early in my career, and therefore know to use the tools from that eco-system. (the rationale in books is the exact opposite - presuming the reader has never used DB-tools). Use the best tool for the job! With regard to "Learning is a lifelong process", most?all of us here will whole-heartedly agree. Those who don't have little reason to subscribe to this list! However, rather than spending years (it must surely be) working on small and simple (and at-times, artificial or academic) problems - even to the point of seeking the most efficient of solutions (to same); there comes a time to stop "learning" per-se, and start 'doing'! The best way to learn how to USE Python to solve 'real-world' problems is to use it - in 'the real-world'! Pick a project that will require multiple classes, several modules, or hundreds of lines of code; and dive in! Given that we've ventured into a multi-media discussion. Here is the very first PyBites podcast (out of more than 70, over the years). It espouses a solid 'learn by doing' mind-set and (IIRC) talks of "JiT-learning" (start from a solid base, and then learn what you need-to, when you need to): https://www.youtube.com/watch?v=NoTkCSP3A68 - or in a component of my understanding of "refactoring": 'there must be a better way'... -- Regards, =dn From alan.gauld at yahoo.co.uk Mon Jun 20 06:54:03 2022 From: alan.gauld at yahoo.co.uk (Alan Gauld) Date: Mon, 20 Jun 2022 11:54:03 +0100 Subject: [Tutor] make a sqlite3 database from an ordinary text file In-Reply-To: References: <007e01d88438$63156090$294021b0$@gmail.com> Message-ID: On 20/06/2022 01:59, Manprit Singh wrote: > As i wrote in the starting , i have a simple text file named abcd.txt. (I > am not using a csv file or any other structured file) . The contents are > given below : > > Time Pieces > 1 10 > 2 15 > 3 25 > 4 31 > 5 40 > 6 45 > 7 53 > 8 54 > 9 65 > 10 75 Sticking with the theme of using the right tool for the right job here is how I did this on Unix(after cutting and pasting your daata into a file called data.txt): ~/src/sql $ tr -cs "[:alnum:]\n" "," data2.txt ~/src/sql $ sqlite3 sqlite> .import -csv data2.txt DATA sqlite> .schema DATA CREATE TABLE IF NOT EXISTS "DATA"( "Time" TEXT, "Pieces" TEXT ); sqlite> Select Time from DATA where Pieces=40; 5 ie. I used tr to convert the text file to a CSV. (-cs means translate everything except alphanum and newlines and "squeeze" to one replacement character) Then I imported the csv file into SQLite. To test it, I checked the schema create statement and did a select query. If you don't like tr you could of course use a simple python (or perl or awk or sed) script to replace the spaces with , or even use a spreadsheet such as Excel. In Python: >>> with open('data.txt') as inf: with open('data2.txt",'w') as outf: for line in inf: line = re.sub(" +", ",", line) outf.write(line) The only problem with this approach is that it makes the two columns TEXT rather than INT. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos From alan.gauld at yahoo.co.uk Mon Jun 20 07:07:48 2022 From: alan.gauld at yahoo.co.uk (Alan Gauld) Date: Mon, 20 Jun 2022 12:07:48 +0100 Subject: [Tutor] make a sqlite3 database from an ordinary text file In-Reply-To: References: <007e01d88438$63156090$294021b0$@gmail.com> Message-ID: On 20/06/2022 01:59, Manprit Singh wrote: > clear that the database table will have 2 columns, Column names of the > sqlite3 table are to be picked from the first line of the text file > abcd.txt. (First column name will be Time, second column name will be > Pieces ). > See i know this can be done very easily using numpy and Pandas. But I am > trying to do it with Core Python only. > > cur.execute("create table workdata(? INT, ? INT)", ("Time", "Pieces")) > > I just want to insert column names (where these column names are read > from text file abcd.txt 1st line) using a question mark style > placeholder. OK, Having shown how I would really do it, I'll now try to answer the question for pure Python: def get_data(open_file): for line in open_file: yield line.split() # open the database/cursor here.... with open('data.txt') as inf: heads = inf.readline().split() cur.execute("create table DATA(? INT, ? INT)",heads) cur.executemany("insert into workdata values (?, ?)",get_data(inf)) cur.execute("select * from DATA") for row in cur.fetchall(): print(row) Note, I didn't actually run this but I think it should be close. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos From manpritsinghece at gmail.com Mon Jun 20 10:17:51 2022 From: manpritsinghece at gmail.com (Manprit Singh) Date: Mon, 20 Jun 2022 19:47:51 +0530 Subject: [Tutor] make a sqlite3 database from an ordinary text file In-Reply-To: References: <007e01d88438$63156090$294021b0$@gmail.com> Message-ID: Dear Sir, Many thanks for this . This mail has a lot of things for me . I Will do all the exercises on my own and will revert you soon. Regards Manprit Singh On Mon, Jun 20, 2022 at 4:39 PM Alan Gauld via Tutor wrote: > On 20/06/2022 01:59, Manprit Singh wrote: > > > clear that the database table will have 2 columns, Column names of the > > sqlite3 table are to be picked from the first line of the text file > > abcd.txt. (First column name will be Time, second column name will be > > Pieces ). > > See i know this can be done very easily using numpy and Pandas. But I am > > trying to do it with Core Python only. > > > > cur.execute("create table workdata(? INT, ? INT)", ("Time", "Pieces")) > > > > I just want to insert column names (where these column names are read > > from text file abcd.txt 1st line) using a question mark style > > placeholder. > > OK, Having shown how I would really do it, I'll now try to > answer the question for pure Python: > > def get_data(open_file): > for line in open_file: > yield line.split() > > > # open the database/cursor here.... > > with open('data.txt') as inf: > heads = inf.readline().split() > cur.execute("create table DATA(? INT, ? INT)",heads) > cur.executemany("insert into workdata values (?, ?)",get_data(inf)) > > cur.execute("select * from DATA") > for row in cur.fetchall(): > print(row) > > > Note, I didn't actually run this but I think it should be close. > > -- > Alan G > Author of the Learn to Program web site > http://www.alan-g.me.uk/ > http://www.amazon.com/author/alan_gauld > Follow my photo-blog on Flickr at: > http://www.flickr.com/photos/alangauldphotos > > > _______________________________________________ > Tutor maillist - Tutor at python.org > To unsubscribe or change subscription options: > https://mail.python.org/mailman/listinfo/tutor > From manpritsinghece at gmail.com Mon Jun 20 10:34:52 2022 From: manpritsinghece at gmail.com (Manprit Singh) Date: Mon, 20 Jun 2022 20:04:52 +0530 Subject: [Tutor] make a sqlite3 database from an ordinary text file In-Reply-To: References: <007e01d88438$63156090$294021b0$@gmail.com> Message-ID: Dear Sir , cur.execute("create table DATA(? INT, ? INT)",heads) As in earlier mail I pointed out that the above command will not work because column names are not written inside the single quotes in the command create table. Hence we can not use (?} place holders . Regards Manprit Singh On Mon, Jun 20, 2022 at 7:47 PM Manprit Singh wrote: > Dear Sir, > > Many thanks for this . This mail has a lot of things for me . I Will do > all the exercises on my own and will revert you soon. > > Regards > Manprit Singh > > On Mon, Jun 20, 2022 at 4:39 PM Alan Gauld via Tutor > wrote: > >> On 20/06/2022 01:59, Manprit Singh wrote: >> >> > clear that the database table will have 2 columns, Column names of the >> > sqlite3 table are to be picked from the first line of the text file >> > abcd.txt. (First column name will be Time, second column name will be >> > Pieces ). >> > See i know this can be done very easily using numpy and Pandas. But I am >> > trying to do it with Core Python only. >> > >> > cur.execute("create table workdata(? INT, ? INT)", ("Time", "Pieces")) >> > >> > I just want to insert column names (where these column names are read >> > from text file abcd.txt 1st line) using a question mark style >> > placeholder. >> >> OK, Having shown how I would really do it, I'll now try to >> answer the question for pure Python: >> >> def get_data(open_file): >> for line in open_file: >> yield line.split() >> >> >> # open the database/cursor here.... >> >> with open('data.txt') as inf: >> heads = inf.readline().split() >> cur.execute("create table DATA(? INT, ? INT)",heads) >> cur.executemany("insert into workdata values (?, ?)",get_data(inf)) >> >> cur.execute("select * from DATA") >> for row in cur.fetchall(): >> print(row) >> >> >> Note, I didn't actually run this but I think it should be close. >> >> -- >> Alan G >> Author of the Learn to Program web site >> http://www.alan-g.me.uk/ >> http://www.amazon.com/author/alan_gauld >> Follow my photo-blog on Flickr at: >> http://www.flickr.com/photos/alangauldphotos >> >> >> _______________________________________________ >> Tutor maillist - Tutor at python.org >> To unsubscribe or change subscription options: >> https://mail.python.org/mailman/listinfo/tutor >> > From manpritsinghece at gmail.com Mon Jun 20 11:32:56 2022 From: manpritsinghece at gmail.com (Manprit Singh) Date: Mon, 20 Jun 2022 21:02:56 +0530 Subject: [Tutor] make a sqlite3 database from an ordinary text file In-Reply-To: References: <007e01d88438$63156090$294021b0$@gmail.com> Message-ID: Dear Sir, The most convenient way that i found to read that text file into an sqlite3 database is using pandas as given below: import sqlite3 import pandas as pd df = pd.read_table("abcd.txt", sep="\s+") # will read the entire file to a dataframe conn = sqlite3.connect("work.db") # will make a database work.db cur = conn.cursor() df.to_sql(name="worktime", con=conn, if_exists="replace", index=False) # Entire text file data, which is into a dataframe is now written to sqlite table cur.execute("select Time from worktime where Pieces=40") # select query print(cur.fetchall()) gives output : [(5,)] On Mon, Jun 20, 2022 at 8:04 PM Manprit Singh wrote: > Dear Sir , > > > > cur.execute("create table DATA(? INT, ? INT)",heads) > > As in earlier mail I pointed out that the above command will not work > because column names are not written inside the single quotes in the > command create table. Hence we can not use (?} place holders . > > Regards > Manprit Singh > > On Mon, Jun 20, 2022 at 7:47 PM Manprit Singh > wrote: > >> Dear Sir, >> >> Many thanks for this . This mail has a lot of things for me . I Will do >> all the exercises on my own and will revert you soon. >> >> Regards >> Manprit Singh >> >> On Mon, Jun 20, 2022 at 4:39 PM Alan Gauld via Tutor >> wrote: >> >>> On 20/06/2022 01:59, Manprit Singh wrote: >>> >>> > clear that the database table will have 2 columns, Column names of the >>> > sqlite3 table are to be picked from the first line of the text file >>> > abcd.txt. (First column name will be Time, second column name will be >>> > Pieces ). >>> > See i know this can be done very easily using numpy and Pandas. But I >>> am >>> > trying to do it with Core Python only. >>> > >>> > cur.execute("create table workdata(? INT, ? INT)", ("Time", "Pieces")) >>> > >>> > I just want to insert column names (where these column names are read >>> > from text file abcd.txt 1st line) using a question mark style >>> > placeholder. >>> >>> OK, Having shown how I would really do it, I'll now try to >>> answer the question for pure Python: >>> >>> def get_data(open_file): >>> for line in open_file: >>> yield line.split() >>> >>> >>> # open the database/cursor here.... >>> >>> with open('data.txt') as inf: >>> heads = inf.readline().split() >>> cur.execute("create table DATA(? INT, ? INT)",heads) >>> cur.executemany("insert into workdata values (?, ?)",get_data(inf)) >>> >>> cur.execute("select * from DATA") >>> for row in cur.fetchall(): >>> print(row) >>> >>> >>> Note, I didn't actually run this but I think it should be close. >>> >>> -- >>> Alan G >>> Author of the Learn to Program web site >>> http://www.alan-g.me.uk/ >>> http://www.amazon.com/author/alan_gauld >>> Follow my photo-blog on Flickr at: >>> http://www.flickr.com/photos/alangauldphotos >>> >>> >>> _______________________________________________ >>> Tutor maillist - Tutor at python.org >>> To unsubscribe or change subscription options: >>> https://mail.python.org/mailman/listinfo/tutor >>> >> From manpritsinghece at gmail.com Mon Jun 20 11:41:51 2022 From: manpritsinghece at gmail.com (Manprit Singh) Date: Mon, 20 Jun 2022 21:11:51 +0530 Subject: [Tutor] make a sqlite3 database from an ordinary text file In-Reply-To: References: <007e01d88438$63156090$294021b0$@gmail.com> Message-ID: Dear Sir, Also the column names in the text file, becomes the column name in sqlite3 table. I have verified it while opening the table in column mode from sqlite3 terminal On Mon, Jun 20, 2022 at 9:02 PM Manprit Singh wrote: > Dear Sir, > > The most convenient way that i found to read that text file into an > sqlite3 database is using pandas as given below: > > import sqlite3 > import pandas as pd > > df = pd.read_table("abcd.txt", sep="\s+") > # will read the entire file to a > dataframe > conn = sqlite3.connect("work.db") > # will make a database work.db > cur = conn.cursor() > df.to_sql(name="worktime", con=conn, if_exists="replace", > index=False) # Entire text file data, which is into a > dataframe is now written to sqlite table > cur.execute("select Time from worktime where > Pieces=40") # select query > print(cur.fetchall()) > > gives output : > > [(5,)] > > > On Mon, Jun 20, 2022 at 8:04 PM Manprit Singh > wrote: > >> Dear Sir , >> >> >> >> cur.execute("create table DATA(? INT, ? INT)",heads) >> >> As in earlier mail I pointed out that the above command will not work >> because column names are not written inside the single quotes in the >> command create table. Hence we can not use (?} place holders . >> >> Regards >> Manprit Singh >> >> On Mon, Jun 20, 2022 at 7:47 PM Manprit Singh >> wrote: >> >>> Dear Sir, >>> >>> Many thanks for this . This mail has a lot of things for me . I Will do >>> all the exercises on my own and will revert you soon. >>> >>> Regards >>> Manprit Singh >>> >>> On Mon, Jun 20, 2022 at 4:39 PM Alan Gauld via Tutor >>> wrote: >>> >>>> On 20/06/2022 01:59, Manprit Singh wrote: >>>> >>>> > clear that the database table will have 2 columns, Column names of the >>>> > sqlite3 table are to be picked from the first line of the text file >>>> > abcd.txt. (First column name will be Time, second column name will be >>>> > Pieces ). >>>> > See i know this can be done very easily using numpy and Pandas. But I >>>> am >>>> > trying to do it with Core Python only. >>>> > >>>> > cur.execute("create table workdata(? INT, ? INT)", ("Time", "Pieces")) >>>> > >>>> > I just want to insert column names (where these column names are read >>>> > from text file abcd.txt 1st line) using a question mark style >>>> > placeholder. >>>> >>>> OK, Having shown how I would really do it, I'll now try to >>>> answer the question for pure Python: >>>> >>>> def get_data(open_file): >>>> for line in open_file: >>>> yield line.split() >>>> >>>> >>>> # open the database/cursor here.... >>>> >>>> with open('data.txt') as inf: >>>> heads = inf.readline().split() >>>> cur.execute("create table DATA(? INT, ? INT)",heads) >>>> cur.executemany("insert into workdata values (?, ?)",get_data(inf)) >>>> >>>> cur.execute("select * from DATA") >>>> for row in cur.fetchall(): >>>> print(row) >>>> >>>> >>>> Note, I didn't actually run this but I think it should be close. >>>> >>>> -- >>>> Alan G >>>> Author of the Learn to Program web site >>>> http://www.alan-g.me.uk/ >>>> http://www.amazon.com/author/alan_gauld >>>> Follow my photo-blog on Flickr at: >>>> http://www.flickr.com/photos/alangauldphotos >>>> >>>> >>>> _______________________________________________ >>>> Tutor maillist - Tutor at python.org >>>> To unsubscribe or change subscription options: >>>> https://mail.python.org/mailman/listinfo/tutor >>>> >>> From wlfraed at ix.netcom.com Mon Jun 20 12:18:08 2022 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Mon, 20 Jun 2022 12:18:08 -0400 Subject: [Tutor] make a sqlite3 database from an ordinary text file References: <007e01d88438$63156090$294021b0$@gmail.com> Message-ID: On Mon, 20 Jun 2022 06:29:58 +0530, Manprit Singh declaimed the following: >See i know this can be done very easily using numpy and Pandas. But I am >trying to do it with Core Python only. > >cur.execute("create table workdata(? INT, ? INT)", ("Time", "Pieces")) > >I just want to insert column names (where these column names are read >from text file abcd.txt 1st line) using a question mark style >placeholder. Pls let me know if anything is possible . > You DON'T! As stated in my previous post, that is manipulating the database schema -- which should never rely upon values found in data files due to the possibility of an SQL injection attack. https://www.explainxkcd.com/wiki/index.php/327:_Exploits_of_a_Mom The purpose of using the parameterized placeholders is that the interface can "sanitize" (escape/quote) characters in the DATA that have meaning in SQL. The SQLite3 adapter appears to inherently block using parameterized queries to modify the database schema. If you REALLY must generate the schema using data from a file, you have to do all of the that in Python, BEFORE passing it to .execute(). >>> import sqlite3 as db >>> con = db.connect("junk.db") >>> cur = con.cursor() >>> cur.execute("create table workdata (? INT, ? INT)", ("Time", "Pieces")) Traceback (most recent call last): File "", line 1, in sqlite3.OperationalError: near "?": syntax error >>> cur.execute("create table workdata (%s INT, %s INT)" % ("Time", "Pieces")) >>> cur.execute("insert into workdata (Time, Pieces) values (?, ?)", (123, "who")) >>> Note that the second .execute() is using Python string interpolation to create the schema definition SQL, not parameterized SQLite3 operations. The third .execute() shows that parameterized statements work with DATA. And, per the warning about SQL injection attacks, you should NEVER use string interpolation (or other formatting operations) in Python to work with data for the database. -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From alan.gauld at yahoo.co.uk Mon Jun 20 13:32:20 2022 From: alan.gauld at yahoo.co.uk (Alan Gauld) Date: Mon, 20 Jun 2022 18:32:20 +0100 Subject: [Tutor] make a sqlite3 database from an ordinary text file In-Reply-To: References: <007e01d88438$63156090$294021b0$@gmail.com> Message-ID: On 20/06/2022 17:18, Dennis Lee Bieber wrote: > The SQLite3 adapter appears to inherently block using parameterized > queries to modify the database schema. If you REALLY must generate the > schema using data from a file, you have to do all of the that in Python, Ah! I didn't know that. Precisely because I never use Python to create a database! > per the warning about SQL injection attacks, you should NEVER use string > interpolation (or other formatting operations) in Python to work with data > for the database. That's true, although I might not say NEVER, since often you write one-off scripts for database work that nobody else ever uses. But the official mechanism works well enough for most real cases that there's rarely any reason not to use it. And creating tables and columns dynamically is frought with problems too. It's like trying to create variables in code from user input - how does the rest of the code refer to those values? For variables we can use a dictionary as a partial solution but for a database it gets very messy with the need to look at the meta data before performing any action. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos From avi.e.gross at gmail.com Sun Jun 19 23:03:15 2022 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Sun, 19 Jun 2022 23:03:15 -0400 Subject: [Tutor] make a sqlite3 database from an ordinary text file In-Reply-To: References: <007e01d88438$63156090$294021b0$@gmail.com> Message-ID: <046401d88452$49aa1e60$dcfe5b20$@gmail.com> With all due respect, Manprit, your text file may not end in .csv or .dat but is just as structured as files that are. Many such files now have no funny headers or anything and can be edited using a flat text editor of any kind and you can enter data one at a time on each line with your designated delimiter (space, tab, comma, colon, whatever) between items and not at the end of a line. If all lines have the same number Of items and you save the file, you can often read it in no matter what the extension is. Nobody is arguing with you about this, but trying to educate you. Just about any programming language, not just python, has available facilities to read in such files for you or let you read it a line at a time and use some built-in method to split the line by those separators. If it turn out that your problem in reading it in using SQL or python is that the name of the file ends in .txt, then do something to tell it what it is in a way that over-rides the guess it makes OR rename or copy the file (you can even do that in python) from xyy.txt to xyz.tsv or xyz.tab or whatever makes the software happy! I repeat. Most such files are indistinguishable from text files albeit there may be some that add some kind of meta-data as something like comments that software can use or ignore. Most of the functionality people use to read in a file either lets you specify that you expect the first line to be a header or that you do not want it to be read. Some make a guess if the first line is not numeric and the rest are. In your case, I believe you have an answer already and maybe have not figured it out. If you want us to write the code for you, hire someone. We are here more for hints and education. But in English, you seem to want a multistep process and to do it YOURSELF. Fine. Write your function and call it ONCE like this: Col1Name, col2Name <- func("file.xt") You now swallowed one line of header and have the text, THEN you can call your other function and let it call func() in iteration till done. Every call will return a tuple of strings that contain a number as text. You now have the names in one place and a growing list of "values" in another. Presumably the way you call other code results in one place asking to create or use a table with those column names, and the second keeps inserting entries into that table by supplying data in the same order as the columns. I pointed out earlier that the code may not be right unless you made a spelling mistake in showing it to us and you should read one line at a time IN THE LOOP and not read one line once before. I am not convinced your function is correct. I am curious what you consider a big file. If your machine has enough memory, you can process quite a large file and since python has garbage collection, you can easily reclaim it by something as simple as putting something new into the variable that held it, including deleting it entirely. I find your suggestion about python confusing. Your application is straightforward to do in just about ANY language I have used and that is well over a hundred I have studied. Feel free to use another, but NOT because you claim it is hard in python. Mind you, ideas like generators are not everywhere you look. In some more limited languages, you might simply read a line at a time and perhaps open a temporary file in which you place SQL statements to start the process, then in a loop read your .txt and produce a line or two vaguely like this: INSERT INTO table_name (column1, column2, column3, ...) VALUES (value1, value2, value3, ...); Finally you could throw that file at your SQL provider and wait a week for it to finish. Many SQL databases allow you to put in many at once with some syntax like this: INSERT INTO MyTable ( Column1, Column2, Column3 ) VALUES ('John', 123, 'Lloyds Office'), ('Jane', 124, 'Lloyds Office'), ('Billy', 125, 'London Office'), ('Miranda', 126, 'Bristol Office'); You can imagine making those by yourself. But you are using some modules that do this for you and that probably is a great idea. So work with it and see what the manual says is available. I took a look as I have never had reason to use what you are talking about and see your code: cur.execute("create table workdata(Time INT, Pieces INT)") The above is doing nothing but passing along what you created exactly. So what you seem to want to do is take a string of text and interpolate the words "Time" and "Pieces" into a template after reading them from the first line of the text file, right? Python has at least 5 major ways to do that and quite a few more. If you know how, fine. If not, ask a specific question, please. The next line of your code is: cur.executemany("insert into workdata values (?, ?)", data_generator("abcd.txt")) The above is actually fairly simple shorthand for doing this by yourself in pseudocode AFTER the table has been created using the first line of the data-file: Forever until out of data: Col1, Col2 = next_line_read_split MyCommand = interpolate Col1 and Col2 into "insert into workdata values (%s, %s)" using one of many methods such as replacing the two instances of %s Run this: cur.execute(MyCommand) The more complex loop you feed a generator to can be easily avoided if that seems more complex. Heck, you do not need to have a generator at all, just keep reading a line till done in the loop. If the above makes no sense to you, fine. Ask specifically what step you do not know how to do or give an example of your code and ask why what you did is not working. Note the problem in cases like this often is not in the code where you think but maybe in the code it calls. If you want to solve a problem, this one is not hard in any number of ways. If you want to solve it using cute shortcuts like the above that ask for a generator, it may be a tad harder. -----Original Message----- From: Tutor On Behalf Of Manprit Singh Sent: Sunday, June 19, 2022 9:00 PM To: tutor at python.org Subject: Re: [Tutor] make a sqlite3 database from an ordinary text file Dear all, Learning is a lifelong process . At this point i still do not feel i am absolutely ok with python or programming. In the past i was very active in this mailing list and will appreciate a few - Alan sir, Dennis Lee Bieber, dn and few others, they always answered my queries . As i wrote in the starting , i have a simple text file named abcd.txt. (I am not using a csv file or any other structured file) . The contents are given below : Time Pieces 1 10 2 15 3 25 4 31 5 40 6 45 7 53 8 54 9 65 10 75 We can clearly see the first line of the file is Time Pieces Now I have to read this text into an sqlite3 database table , So it is clear that the database table will have 2 columns, Column names of the sqlite3 table are to be picked from the first line of the text file abcd.txt. (First column name will be Time, second column name will be Pieces ). See i know this can be done very easily using numpy and Pandas. But I am trying to do it with Core Python only. cur.execute("create table workdata(? INT, ? INT)", ("Time", "Pieces")) I just want to insert column names (where these column names are read from text file abcd.txt 1st line) using a question mark style placeholder. Pls let me know if anything is possible . Time Pieces On Mon, Jun 20, 2022 at 5:34 AM wrote: > Manprit, > > You may be approaching this topic differently than you could. > > What you keep calling a TEXT file is what others might call structured > files like .CSV or .DAT types that contain values separated by some > delimiter like a comma or a tab or arbitrary whitespace or other > choices. Lots of python functionalities are available to read these > in to structures used in modules like numpy and pandas that many > people use on top of or alongside regular built-in python commands. In > these, there can be (and usually are) HEADER names for columns and > sometimes also rows. The functionality that reads in such tables, > Dataframes, or whatever you want to call it, will often optionally > take the first line of the file to have headers specifying names. And, > you can always add or change the names as well. > > So if passing the entire structure to another function, it can choose > to access the name of any column (or often row) when it wants. Or, if > you need to, you can ask for it and pass it your way as in > interpolating the name into an SQL command. > > You may be asking a slightly different question here about just your data. > Does your "text" file contain a first line saying ""Time Pieces" or > not? > > If it DOES, you may need to read it ONCE at the start before giving > your iterator for the numbers to be read and store the results as your > names to use as you wish. If the file has no header, then your code is > the one attaching a name. > > To me the real issue is your design seems to be of a limited nature. > You want to read in your data in a small batch (single line) and > presumably are opening a connection to the database and doing login > and so on, then sending a command to create a table, unless it already > exists, then sending umpteen commands to insert yet another row into > the table and then who knows? This seems like such a common operation > that I have to guess there is a way to do it fairly trivially and my > guess is way more efficiently. > > > > -----Original Message----- > From: Tutor On Behalf > Of Manprit Singh > Sent: Sunday, June 19, 2022 2:38 PM > Cc: tutor at python.org > Subject: Re: [Tutor] make a sqlite3 database from an ordinary text > file > > Dear sir , > > One more query .... > How can i use column name in text file as colum names in sqlite3 database . > I tried using python but failed. > > Can you put some light over it ? > Regards > Manprit Singh > > On Sun, 19 Jun, 2022, 23:36 Alan Gauld via Tutor, > wrote: > > > On 19/06/2022 18:11, Manprit Singh wrote: > > > > > cur.execute("create table workdata(Time INT, Pieces INT)") > > > > This will fail the second time it runs. You should check if the > > table already exists and either drop it or use the existing table. > > > > However, I rarely create tables from Python, it's generally easier > > to create a sql file and run that directly using the sqlite interpreter. > > You only need Python if the format of the table has to be deduced > > from the data. and that then leads to all sorts of follow-on issues > > over accessing names of columns etc. > > > > Once you have the table you can then use Python to load the data, > > although in your case you could do that directly from SQL too since > > it looks like a tab separated file. > > > > As always use the most appropriate tool. If you can do it directly > > in SQL there is no point in wrapping it in Python. That just adds > > complexity and slows things down. > > > > -- > > Alan G > > Author of the Learn to Program web site http://www.alan-g.me.uk/ > > http://www.amazon.com/author/alan_gauld > > Follow my photo-blog on Flickr at: > > http://www.flickr.com/photos/alangauldphotos > > > > > > _______________________________________________ > > Tutor maillist - Tutor at python.org > > To unsubscribe or change subscription options: > > https://mail.python.org/mailman/listinfo/tutor > > > _______________________________________________ > Tutor maillist - Tutor at python.org > To unsubscribe or change subscription options: > https://mail.python.org/mailman/listinfo/tutor > > _______________________________________________ > Tutor maillist - Tutor at python.org > To unsubscribe or change subscription options: > https://mail.python.org/mailman/listinfo/tutor > _______________________________________________ Tutor maillist - Tutor at python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor From avi.e.gross at gmail.com Mon Jun 20 12:46:55 2022 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Mon, 20 Jun 2022 12:46:55 -0400 Subject: [Tutor] make a sqlite3 database from an ordinary text file In-Reply-To: References: <007e01d88438$63156090$294021b0$@gmail.com> Message-ID: <012b01d884c5$5a1c0880$0e541980$@gmail.com> Manprit, When something is not working, think about it before asking others to. The variable called "heads" is what? Try printing it. It came from " yield line.split()" and it probably is a list containing two items of text as in ["title1", "title2"] So verify what is in it using print() or other tools before continuing. Then look to see what your cur.execute() wants there. Does it want one argument that is a list or does it want two individual arguments? OR, does it need any arguments there if you DO IT YOURSELF! What if you made a line for yourself I which you interpolated heads[1] and heads[2] into a string and executed that? A little search shows examples like: cur.execute("select md5(?)", (b"foo",)) cur.execute("SELECT * FROM stocks WHERE symbol = '%s'" % symbol) Forget the Select part and focus on how it puts something dynamically into the string. See if you can make your code in a way that passes the right thing. The second example above is one of many ways of interpolating by replacing the %s within quotes v=by the value of "symbol" and you have two things to replace. Time to go live my real life while you do what seems like homework. -----Original Message----- From: Tutor On Behalf Of Manprit Singh Sent: Monday, June 20, 2022 10:35 AM To: tutor at python.org Subject: Re: [Tutor] make a sqlite3 database from an ordinary text file Dear Sir , cur.execute("create table DATA(? INT, ? INT)",heads) As in earlier mail I pointed out that the above command will not work because column names are not written inside the single quotes in the command create table. Hence we can not use (?} place holders . Regards Manprit Singh On Mon, Jun 20, 2022 at 7:47 PM Manprit Singh wrote: > Dear Sir, > > Many thanks for this . This mail has a lot of things for me . I Will > do all the exercises on my own and will revert you soon. > > Regards > Manprit Singh > > On Mon, Jun 20, 2022 at 4:39 PM Alan Gauld via Tutor > > wrote: > >> On 20/06/2022 01:59, Manprit Singh wrote: >> >> > clear that the database table will have 2 columns, Column names of >> > the >> > sqlite3 table are to be picked from the first line of the text >> > file abcd.txt. (First column name will be Time, second column name >> > will be Pieces ). >> > See i know this can be done very easily using numpy and Pandas. But >> > I am trying to do it with Core Python only. >> > >> > cur.execute("create table workdata(? INT, ? INT)", ("Time", >> > "Pieces")) >> > >> > I just want to insert column names (where these column names are >> > read from text file abcd.txt 1st line) using a question mark style >> > placeholder. >> >> OK, Having shown how I would really do it, I'll now try to answer the >> question for pure Python: >> >> def get_data(open_file): >> for line in open_file: >> yield line.split() >> >> >> # open the database/cursor here.... >> >> with open('data.txt') as inf: >> heads = inf.readline().split() >> cur.execute("create table DATA(? INT, ? INT)",heads) >> cur.executemany("insert into workdata values (?, >> ?)",get_data(inf)) >> >> cur.execute("select * from DATA") >> for row in cur.fetchall(): >> print(row) >> >> >> Note, I didn't actually run this but I think it should be close. >> >> -- >> Alan G >> Author of the Learn to Program web site http://www.alan-g.me.uk/ >> http://www.amazon.com/author/alan_gauld >> Follow my photo-blog on Flickr at: >> http://www.flickr.com/photos/alangauldphotos >> >> >> _______________________________________________ >> Tutor maillist - Tutor at python.org >> To unsubscribe or change subscription options: >> https://mail.python.org/mailman/listinfo/tutor >> > _______________________________________________ Tutor maillist - Tutor at python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor From roel at roelschroeven.net Mon Jun 20 04:37:45 2022 From: roel at roelschroeven.net (Roel Schroeven) Date: Mon, 20 Jun 2022 10:37:45 +0200 Subject: [Tutor] Fwd: make a sqlite3 database from an ordinary text file In-Reply-To: <007a01d88436$58d53da0$0a7fb8e0$@gmail.com> References: <007a01d88436$58d53da0$0a7fb8e0$@gmail.com> Message-ID: Op 20/06/2022 om 1:43 schreef avi.e.gross at gmail.com: > I may not be looking at this right since much happens in functions not seen > but does cur.execute(many) expect two items of text or two items of type > integer or will it handle a list or tuple with arbitrary numbers of contents > and so on, is what makes the design decision. cur.executemany() takes an iterable which on each iteration yields a number of items corresponding to the number of placeholders ("?") in the query. You can use any datatype that sqlite3 understands, but care should be taken to use the type you need. Other than other databases, sqlite3 stores data as the type you give it, rather than the type specified in the database definition (in a sense sqlite3 is dynamically typed like Python, instead of statically typed like other SQL implementations). So AFAICS the code is wrong on that point: it should convert the data to integers before handing them over to the database. > Deeper within that function, it presumable maps your passed function to code > that may do, > > First, Second = f() > > And so on. Or it may just take a single result or other choices. It is a bit > confusing to pass a filename as someone pointed out. This cur.executemany() call is the equivalent of: ??? for time, nr_pieces in data_generator('abcd.txt'): ??????? cur.execute("insert into workdata values (?, ?)", (time, nr_pieces)) But a bit shorter and with presumably less overhead. > In what follows, either I am confused or the one asking the question is. > > The goal of the iterator seems to be to avoid reading all the data in at > once, right. Otherwise, why bother when you can do something like reading in > the entire file and splitting and putting the results in something like a > list structure, or perhaps in something from numpy. > > So why does the code use readline() (no 's') to read in a single line and > only once? > > As I read the somewhat confusing code, the goal is to open the file the > first time and read a line and throw the results away as it is not clear > where it is saved. Then the goal is to loop on nothing (or maybe one line) > and yield the results of splitting it and pause till called again. But since > only no or maybe one line have been received, the call to iterate should > just exit as the loop is done. > > So I think reading a single line should be moved down within the loop! That readline() call is to skip the first line, because it contains column names instead of actual data. After that the code iterates over the file object which yields each remaining line on each iteration, which is then split and yielded for use by cur.executemany(). > And you need some kind of test to know when there are no remaining lines in > the file or the current line is empty and instead of yielding a non-existent > result, you should return what the iteration protocol needs and close the > file, albeit the with does that for you silently as part of that protocol. No need, that's all taken care of automatically. "for line in obj" stops if there are no more lines, and as you say the with statement closes the file. > But as others have noted, it is not clear other than as an exercise, if this > iterator is needed. Your other modules may supply something for you. Why not use an iterator? Once you get the hang of it, using iterators is often just as easy as using lists; with the advantage that a program that uses iterators where it matters works for small and large datasets without any problem because it doesn't need large amounts of memory to store its data. -- "Peace cannot be kept by force. It can only be achieved through understanding." -- Albert Einstein From roel at roelschroeven.net Mon Jun 20 04:45:06 2022 From: roel at roelschroeven.net (Roel Schroeven) Date: Mon, 20 Jun 2022 10:45:06 +0200 Subject: [Tutor] make a sqlite3 database from an ordinary text file In-Reply-To: <9b0f5a6e-7f9d-1bef-8a71-658bb3b75d45@DancesWithMice.info> References: <007e01d88438$63156090$294021b0$@gmail.com> <9b0f5a6e-7f9d-1bef-8a71-658bb3b75d45@DancesWithMice.info> Message-ID: Op 20/06/2022 om 4:39 schreef dn: > I'm +1 after @Alan. Although we often see it in programming texts, I've > NEVER initialised a DB/TBL this way. Easy for me to say: I learned RDBMS > and SQL very early in my career, and therefore know to use the tools > from that eco-system. (the rationale in books is the exact opposite - > presuming the reader has never used DB-tools). Use the best tool for the > job! I don't really agree with this. I quite often use Python to read data from some file, perhaps process it a bit, then store it in SQL. If any kind of processing on the data is needed, Python is much easier and flexible to do it than SQL (at least to me). And if no processing is needed, well, I often use a Python script anyway because I know how to do that; doing it using the RDBMS tools would involve me starting to look up things left and right, and would be different between different RDBMS systems. -- "Peace cannot be kept by force. It can only be achieved through understanding." -- Albert Einstein From alan.gauld at yahoo.co.uk Mon Jun 20 14:01:15 2022 From: alan.gauld at yahoo.co.uk (Alan Gauld) Date: Mon, 20 Jun 2022 19:01:15 +0100 Subject: [Tutor] make a sqlite3 database from an ordinary text file In-Reply-To: References: <007e01d88438$63156090$294021b0$@gmail.com> <9b0f5a6e-7f9d-1bef-8a71-658bb3b75d45@DancesWithMice.info> Message-ID: On 20/06/2022 09:45, Roel Schroeven wrote: >> NEVER initialised a DB/TBL this way. Easy for me to say: I learned RDBMS >> and SQL very early in my career, and therefore know to use the tools >> from that eco-system. (the rationale in books is the exact opposite - >> presuming the reader has never used DB-tools). Use the best tool for the >> job! > I don't really agree with this. I quite often use Python to read data > from some file, perhaps process it a bit, then store it in SQL. Roel, We weren't saying not to use Python for processing SQL data. It was specifically about creating the schema in Python (and to a lesser degree loading it with initial data). You can do it but it's usually much easier to do it directly in SQL (which you have to embed in the Python code anyway!) Once you have some data in a database using Python to process the data makes sense. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos From wlfraed at ix.netcom.com Mon Jun 20 15:55:23 2022 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Mon, 20 Jun 2022 15:55:23 -0400 Subject: [Tutor] make a sqlite3 database from an ordinary text file References: <007e01d88438$63156090$294021b0$@gmail.com> Message-ID: <28j1bhhdrgrrc860lblqn974sarmrejd1l@4ax.com> On Mon, 20 Jun 2022 18:32:20 +0100, Alan Gauld via Tutor declaimed the following: >On 20/06/2022 17:18, Dennis Lee Bieber wrote: > >> The SQLite3 adapter appears to inherently block using parameterized >> queries to modify the database schema. If you REALLY must generate the >> schema using data from a file, you have to do all of the that in Python, > >Ah! I didn't know that. Precisely because I never use Python >to create a database! > I'm not ambitious enough to try testing with Firebird or SQL-Server to see if they object. I suspect SQLite3 probably objects to any DDL statement that is parameterized -- I think it goes through the "prepare" (compile the SQL statement)/"execute" (run the compiled statement with parameters). Schema entities (database, table, fields...) probably have to be KNOWN during the prepare/compile phase. One of the older MySQL adapters did not use "prepared statements" -- and internally it used Python string interpolation (with %s placeholders) to plug in data values (and .executemany() might have been a loop issuing statements for each set of data values). On that adapter, the OP's attempt to define the schema from data values probably worked. As I recall, MySQL in the 3.x series did not support prepared statements even from C. The latest versions do support prepared statements (and other complex features) and newer adapters may take advantage of such. >That's true, although I might not say NEVER, since often you write >one-off scripts for database work that nobody else ever uses. But the >official mechanism works well enough for most real cases that there's >rarely any reason not to use it. > My throw-away efforts still have me looking at the input data file to determine field names and data types (though SQLite3 doesn't really care -- if a field is defined INT and the data is a string that can be interpreted as INT it converts, otherwise it will happily store it as text). >And creating tables and columns dynamically is frought with problems >too. It's like trying to create variables in code from user input - how >does the rest of the code refer to those values? For variables we can >use a dictionary as a partial solution but for a database it gets very >messy with the need to look at the meta data before performing any action. And that leads back to my prior post about "menus". If the application needs the user to specify fields for matching in a WHERE clause, say, I'd present a menu of field names obtained from the schema, and the user would select by menu position, not by entering the name. That way I know the field name(s) can not contain garbage, and it is safe to use Python string interpolation to build the SQL statement, while using parameterization for the user provided values to be matched. -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From roel at roelschroeven.net Mon Jun 20 15:18:25 2022 From: roel at roelschroeven.net (Roel Schroeven) Date: Mon, 20 Jun 2022 21:18:25 +0200 Subject: [Tutor] make a sqlite3 database from an ordinary text file In-Reply-To: References: <007e01d88438$63156090$294021b0$@gmail.com> <9b0f5a6e-7f9d-1bef-8a71-658bb3b75d45@DancesWithMice.info> Message-ID: <9434478e-a71d-733d-18cb-bce1eab0c10f@roelschroeven.net> Alan Gauld via Tutor schreef op 20/06/2022 om 20:01: > On 20/06/2022 09:45, Roel Schroeven wrote: > > >> NEVER initialised a DB/TBL this way. Easy for me to say: I learned RDBMS > >> and SQL very early in my career, and therefore know to use the tools > >> from that eco-system. (the rationale in books is the exact opposite - > >> presuming the reader has never used DB-tools). Use the best tool for the > >> job! > > I don't really agree with this. I quite often use Python to read data > > from some file, perhaps process it a bit, then store it in SQL. > > Roel, > > We weren't saying not to use Python for processing SQL data. > It was specifically about creating the schema in Python > (and to a lesser degree loading it with initial data). You > can do it but it's usually much easier to do it directly > in SQL (which you have to embed in the Python code anyway!) Ah, I misunderstood. My apologies. -- "Experience is that marvelous thing that enables you to recognize a mistake when you make it again." -- Franklin P. Jones From mats at wichmann.us Mon Jun 20 17:22:12 2022 From: mats at wichmann.us (Mats Wichmann) Date: Mon, 20 Jun 2022 15:22:12 -0600 Subject: [Tutor] make a sqlite3 database from an ordinary text file In-Reply-To: References: <007e01d88438$63156090$294021b0$@gmail.com> <9b0f5a6e-7f9d-1bef-8a71-658bb3b75d45@DancesWithMice.info> Message-ID: On 6/20/22 12:01, Alan Gauld via Tutor wrote: > On 20/06/2022 09:45, Roel Schroeven wrote: > >>> NEVER initialised a DB/TBL this way. Easy for me to say: I learned RDBMS >>> and SQL very early in my career, and therefore know to use the tools >>> from that eco-system. (the rationale in books is the exact opposite - >>> presuming the reader has never used DB-tools). Use the best tool for the >>> job! >> I don't really agree with this. I quite often use Python to read data >> from some file, perhaps process it a bit, then store it in SQL. > > Roel, > > We weren't saying not to use Python for processing SQL data. > It was specifically about creating the schema in Python > (and to a lesser degree loading it with initial data). You > can do it but it's usually much easier to do it directly > in SQL (which you have to embed in the Python code anyway!) This isn't as true (especially the "have to embed sql anyway") if you're using an ORM. From alan.gauld at yahoo.co.uk Mon Jun 20 18:55:02 2022 From: alan.gauld at yahoo.co.uk (Alan Gauld) Date: Mon, 20 Jun 2022 23:55:02 +0100 Subject: [Tutor] make a sqlite3 database from an ordinary text file In-Reply-To: References: <007e01d88438$63156090$294021b0$@gmail.com> <9b0f5a6e-7f9d-1bef-8a71-658bb3b75d45@DancesWithMice.info> Message-ID: On 20/06/2022 22:22, Mats Wichmann wrote: >> We weren't saying not to use Python for processing SQL data. >> It was specifically about creating the schema in Python > > This isn't as true (especially the "have to embed sql anyway") if you're > using an ORM. Thats true, but then an ORM is there to pretend that a SQL RDBMS is actually an OODB. So using SQL for anything is likely to break stuff because the ORM controls how the relationships are set up and mapped. But I confess I've rarely used ORMs, I'd rather just use a real Object Database if I need to maage persistent objects. But I' haven't done much of that either, in fact the only OODB I'd say I'd recommend is Objectivity, although I'd like to have a play with ZODB someday. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos From phillor9 at gmail.com Tue Jun 21 00:17:06 2022 From: phillor9 at gmail.com (Phil) Date: Tue, 21 Jun 2022 14:17:06 +1000 Subject: [Tutor] Event loop logic question In-Reply-To: References: Message-ID: On 19/6/22 05:19, Dennis Lee Bieber wrote: > On Sat, 18 Jun 2022 15:04:50 +1000, Phil declaimed the > following: > > >> Any hints or alternative methods will be, as always, greatly appreciated. >> >> By the way, this project sends commands to a controller which is easily >> overwhelmed if too many commands are received per second. >> Thank you avi, Alan and Dennis for your time. I've been in an area without mobile phone reception for the past couple of days and it could be awhile before I'm in an area with reasonable reception again. In the meantime I'll give some thought to the suggestions offered and see what I can come up with. The controller firmware is known to contain bugs (crashing because of commands in quick succession is one of them). The print statements at seemingly random places are just my feeble attempt at debugging the code. -- Regards, Phil From avi.e.gross at gmail.com Mon Jun 20 18:02:42 2022 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Mon, 20 Jun 2022 18:02:42 -0400 Subject: [Tutor] make a sqlite3 database from an ordinary text file In-Reply-To: References: <007e01d88438$63156090$294021b0$@gmail.com> Message-ID: <018201d884f1$778aa7b0$669ff710$@gmail.com> Dennis, While your comments are quite valid and reasonable and make me wonder about all sorts of techniques hackers can use that often must be defended against, wouldn't it be nice if people doing homework would actually show us the problem rather than trying to give a few parts? If this is meant to be serious code to be heavily used in real world situations, such as for a job at Google, it does often require some bulletproofing as you describe. But is it that or just solving a problem I a sandbox to learn how to use a few features, or is it more akin to homework? I am not saying the case in point is homework but it has some signs I interpret that way. It may be the kind of homework where you do independent study from a book and are trying the questions at the end of the chapter and really want to learn, not where you want to get someone to just do it for you in a class and not learn to do it yourself. It seems to both be open to some packages and advanced ideas while we get steered away from others. Some of the replies probably are way off base if we only knew the context. I had to go out for the afternoon at this point and note that with amusement that a book I was reading about web development happened to cover various ways to store and query data from a browser on the local machine and had a section where a similar javascript interface tp something like SQL LITE was used that clearly showed how the API expected multiple names to be given as a single argument with something like a list. So I am not sure why our persistent questioner said the code he tried did not work. I look for patterns and anomalies that might help me GUESS things and I saw some here that may, and may not, mean anything. For example, why is it required to use a text file with what looks like a space or tab or whitespace separator? Alan has suggested ways to Change the file to something like a CSV that can be done directly in the non-python arena of the database. Well, yes, that is a perfectly reasonable approach if you are not told the goal is to do it all in a circumscribed subset of python! In real life, other than perhaps efficiency considerations, many people would start with existing tools such as numpy and pandas and read the darn data in natively, specifying a delimiter as in: pd.read_csv('file.csv', delimiter=' ') is a fairly easy way to read his "text file" unless a requirement is made to read it in a line at a time. Functions like the above typically read quite a bit of the file to determine what types the columns of data should be, so they probably are not designed to act much like generators. But I suspect they could easily be if the goal was not to read in all of the file. You could read in a thousand lines or so and serve those until you ran low and got another batch and so on. So is it really a requirement to read a line at a time and not to make sure all columns are consistent? Apparently not. Nor were we told of any requirements that this be SQL safe. Yes, tools can ensure that but if this was a beginners course, I might add requirements like take the two text items you got from the split and checks them several ways before using them. They should all be digits, and no longer than some length of say 8 characters. If you allow commas or underscores, strip them. If they can be decimals, or scientific notation, it gets more complex but sometimes something as simple as calling a function that converts the contents to a number and then converting them back, can reject or repair many such strings and possibly make them safe to use by interpolating the contents directly. I can not be sure but I think the main problem in this example was poor programming based on what may not have been a well-planned design. Loops need some things done before and some within and some after and you need to segregate parts of your plan so each is done as often as needed (such as once) and in the right place. In my view, this assignment of self-motivate project is not that complicated if approached systematically with a knowledge of how things can be done and some minor skills at debugging such as strategically placed print statement. The fact that several of us are going back and forth may partially be due to some English language issues and might work better if we communicated in their native language. I suspect none of us participating does! LOL! I do not speak Punjabi or even Hindi. So patience and some awareness the other is working hard to try to communicate is necessary. Where are we? Several possible methods have been offered using advice/pseudocode or actual python code and if none of them works, we need more than just being told the user could not get it to work. WHERE did it fail? What were the context of variables? What error message? Did they cut/paste and adjust indentation correctly, or perhaps type something directly but wrong? There is only so much time and effort being offered here and it is, in my case, more about teaching someone so they can then do it again on their own, than producing unpaid work or something that shows off what we can do but not what they need. -----Original Message----- From: Tutor On Behalf Of Dennis Lee Bieber Sent: Monday, June 20, 2022 12:18 PM To: tutor at python.org Subject: Re: [Tutor] make a sqlite3 database from an ordinary text file On Mon, 20 Jun 2022 06:29:58 +0530, Manprit Singh declaimed the following: >See i know this can be done very easily using numpy and Pandas. But I >am trying to do it with Core Python only. > >cur.execute("create table workdata(? INT, ? INT)", ("Time", "Pieces")) > >I just want to insert column names (where these column names are read >from text file abcd.txt 1st line) using a question mark style >placeholder. Pls let me know if anything is possible . > You DON'T! As stated in my previous post, that is manipulating the database schema -- which should never rely upon values found in data files due to the possibility of an SQL injection attack. https://www.explainxkcd.com/wiki/index.php/327:_Exploits_of_a_Mom The purpose of using the parameterized placeholders is that the interface can "sanitize" (escape/quote) characters in the DATA that have meaning in SQL. The SQLite3 adapter appears to inherently block using parameterized queries to modify the database schema. If you REALLY must generate the schema using data from a file, you have to do all of the that in Python, BEFORE passing it to .execute(). >>> import sqlite3 as db >>> con = db.connect("junk.db") >>> cur = con.cursor() >>> cur.execute("create table workdata (? INT, ? INT)", ("Time", >>> "Pieces")) Traceback (most recent call last): File "", line 1, in sqlite3.OperationalError: near "?": syntax error >>> cur.execute("create table workdata (%s INT, %s INT)" % ("Time", >>> "Pieces")) >>> cur.execute("insert into workdata (Time, Pieces) values (?, ?)", >>> (123, "who")) >>> Note that the second .execute() is using Python string interpolation to create the schema definition SQL, not parameterized SQLite3 operations. The third .execute() shows that parameterized statements work with DATA. And, per the warning about SQL injection attacks, you should NEVER use string interpolation (or other formatting operations) in Python to work with data for the database. -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ _______________________________________________ Tutor maillist - Tutor at python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor From avi.e.gross at gmail.com Mon Jun 20 18:18:15 2022 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Mon, 20 Jun 2022 18:18:15 -0400 Subject: [Tutor] Fwd: make a sqlite3 database from an ordinary text file In-Reply-To: References: <007a01d88436$58d53da0$0a7fb8e0$@gmail.com> Message-ID: <01a801d884f3$a3874420$ea95cc60$@gmail.com> Roel, I appreciate your explanations. I wonder about one thing here. From your description, it sounds like the fully-text file being read needs to have the first line read in as text and remain as text. But in this case, all subsequent files should be sent to the database as integers or perhaps another numeric format. If that is correct, the generator used normally cannot return both of those easily. For this one case, you could open the darn file once to get the column names without using the generator, to get the column names as text, then reopen the file in the generator and throw away the first result as the generator will now return an integer form of the data and the first line will be nonsense. Several variations are possible, and since a generator can keep state, it can be set to effectively skip the first line. The designs so far have not shown any idea of making integers, albeit a wrapper around the generator could perhaps do the conversion. Some of this makes me appreciate all the hidden level of detail that read_csv() and similar functions deal with that may not be needed every time or may not be perfect, but can be relied on to deal with lots of detail you might not even think about. -----Original Message----- From: Tutor On Behalf Of Roel Schroeven Sent: Monday, June 20, 2022 4:38 AM To: tutor at python.org Subject: Re: [Tutor] Fwd: make a sqlite3 database from an ordinary text file Op 20/06/2022 om 1:43 schreef avi.e.gross at gmail.com: > I may not be looking at this right since much happens in functions not > seen but does cur.execute(many) expect two items of text or two items > of type integer or will it handle a list or tuple with arbitrary > numbers of contents and so on, is what makes the design decision. cur.executemany() takes an iterable which on each iteration yields a number of items corresponding to the number of placeholders ("?") in the query. You can use any datatype that sqlite3 understands, but care should be taken to use the type you need. Other than other databases, sqlite3 stores data as the type you give it, rather than the type specified in the database definition (in a sense sqlite3 is dynamically typed like Python, instead of statically typed like other SQL implementations). So AFAICS the code is wrong on that point: it should convert the data to integers before handing them over to the database. > Deeper within that function, it presumable maps your passed function > to code that may do, > > First, Second = f() > > And so on. Or it may just take a single result or other choices. It is > a bit confusing to pass a filename as someone pointed out. This cur.executemany() call is the equivalent of: for time, nr_pieces in data_generator('abcd.txt'): cur.execute("insert into workdata values (?, ?)", (time, nr_pieces)) But a bit shorter and with presumably less overhead. > In what follows, either I am confused or the one asking the question is. > > The goal of the iterator seems to be to avoid reading all the data in > at once, right. Otherwise, why bother when you can do something like > reading in the entire file and splitting and putting the results in > something like a list structure, or perhaps in something from numpy. > > So why does the code use readline() (no 's') to read in a single line > and only once? > > As I read the somewhat confusing code, the goal is to open the file > the first time and read a line and throw the results away as it is not > clear where it is saved. Then the goal is to loop on nothing (or maybe > one line) and yield the results of splitting it and pause till called > again. But since only no or maybe one line have been received, the > call to iterate should just exit as the loop is done. > > So I think reading a single line should be moved down within the loop! That readline() call is to skip the first line, because it contains column names instead of actual data. After that the code iterates over the file object which yields each remaining line on each iteration, which is then split and yielded for use by cur.executemany(). > And you need some kind of test to know when there are no remaining > lines in the file or the current line is empty and instead of yielding > a non-existent result, you should return what the iteration protocol > needs and close the file, albeit the with does that for you silently as part of that protocol. No need, that's all taken care of automatically. "for line in obj" stops if there are no more lines, and as you say the with statement closes the file. > But as others have noted, it is not clear other than as an exercise, > if this iterator is needed. Your other modules may supply something for you. Why not use an iterator? Once you get the hang of it, using iterators is often just as easy as using lists; with the advantage that a program that uses iterators where it matters works for small and large datasets without any problem because it doesn't need large amounts of memory to store its data. -- "Peace cannot be kept by force. It can only be achieved through understanding." -- Albert Einstein _______________________________________________ Tutor maillist - Tutor at python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor From PythonList at DancesWithMice.info Tue Jun 21 05:07:37 2022 From: PythonList at DancesWithMice.info (dn) Date: Tue, 21 Jun 2022 21:07:37 +1200 Subject: [Tutor] make a sqlite3 database from an ordinary text file In-Reply-To: <9434478e-a71d-733d-18cb-bce1eab0c10f@roelschroeven.net> References: <007e01d88438$63156090$294021b0$@gmail.com> <9b0f5a6e-7f9d-1bef-8a71-658bb3b75d45@DancesWithMice.info> <9434478e-a71d-733d-18cb-bce1eab0c10f@roelschroeven.net> Message-ID: On 21/06/2022 07.18, Roel Schroeven wrote: > > > Alan Gauld via Tutor schreef op 20/06/2022 om 20:01: >> On 20/06/2022 09:45, Roel Schroeven wrote: >> >> >> NEVER initialised a DB/TBL this way. Easy for me to say: I learned >> RDBMS >> >> and SQL very early in my career, and therefore know to use the tools >> >> from that eco-system. (the rationale in books is the exact opposite - >> >> presuming the reader has never used DB-tools). Use the best tool >> for the >> >> job! >> > I don't really agree with this. I quite often use Python to read >> data > from some file, perhaps process it a bit, then store it in SQL. >> Roel, >> >> We weren't saying not to use Python for processing SQL data. >> It was specifically about creating the schema in Python >> (and to a lesser degree loading it with initial data). You >> can do it but it's usually much easier to do it directly >> in SQL (which you have to embed in the Python code anyway!) > > Ah, I misunderstood. My apologies. There's plenty of room for that, just as there is plenty of room for using Python, SQL-tools, or something else. (I wasn't offended) The different SQL eco-systems, eg DB2, Oracle, MySQL, SQLite; offer very different ranges of tools beyond the RDBMS 'engine' and command-line client. Thus, it is all-very-well for me/us to say 'use the SQL-tools' if we are talking about one system and for you/A.N.Other to question such if using something less feature-full. Then there are the points you have made. If there is quite a bit of 'data-cleaning' going-on, then such will start to exceed the capability of the tools I/we've espoused. In which case, Python is the 'Swiss Army Knife' to use - no debate. There are plenty of rules and warnings about sanitising data before INSERT-ing or UPDATE-ing. Anyone unaware of that should stop coding and do some reading *immediately*! That someone would dare to use user-data to formulate a schema is far to rich (risky) for my blood! I'm trying to imagine a reasonable use-case, but... Forced into some sort of situation like this, I'd split the task into parts, with a manual inspection of the generated schema before I'd let it run. (and once again, I'd input that schema through an SQL-tool - but allow for the fact that my background enables me to achieve such quickly. Whereas YMMV and thus you/A.N.Other might prefer to use the Python interface) Another consideration worth some thought is that key-value databases are 'schema-less'. The fact that in this scenario we don't know what the schema will be until the first phase of the system 'designs it', becomes irrelevant! The other nifty advantage to using something like MongoDB is that its mechanisms are virtually indistinguishable from Python dicts (etc) - so I'll often prototype with Python built-ins and only later worry about persistence. -- Regards, =dn From tempest.amogh at gmail.com Tue Jun 21 01:47:26 2022 From: tempest.amogh at gmail.com (tempest.amogh ) Date: Tue, 21 Jun 2022 11:17:26 +0530 Subject: [Tutor] Help with brute force approach to solving grid combinations Message-ID: As a personal project, I was inspired by Erik Demaine. He has created a font based on tetris shapes [http://erikdemaine.org/fonts/], however I would like to emulate this but based on another reference. I want to write an algorithm that brute force solves all the positions for these pieces in the english alphabet. I am having trouble making a grid that can be solved with these pieces, despite a book saying so. I have attached the image that shows the shapes [http://erikdemaine.org/fonts/tetris/kadon_fonts.jpg]. My code: --- import numpy as np default_grid = np.array([[0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0] ]) a_grid = np.array([[0, 0, 0, 0, 0, 0], [0, 0, -1, -1, 0, 0], [0, 0, -1, -1, 0, 0], [0, 0, 0, 0, 0, 0], [0, 0, -1, -1, 0, 0], [0, 0, -1, -1, 0, 0] ]) # 1. SHAPES # 1.1 DOMINOES shape_straight2 = np.array([[7, 0, 0, 0, 0, 0], [7, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0] ]) shape_flat = np.array([[0, 0, 0, 0, 8, 8], [0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0] ]) # 1.2 TRIMINOES shape_corner = np.array([[6, 0], [6, 6] ]) shape_straight3 = np.array([[7], [7], [7] ]) # 1.3 TETRIMINOES shape_straight = np.array([[1, 1, 1, 1] ]) shape_straight1 = np.array([[1], [1], [1], [1] ]) shape_square = np.array([[2, 2], [2, 2] ]) shape_l = np.array([[3, 0], [3, 0], [3, 3] ]) shape_l2 = np.array([[0, 3], [0, 3], [3, 3] ]) shape_l3 = np.array([[3, 3, 3], [0, 0, 3] ]) shape_l4 = np.array([[3, 3, 3], [3, 0, 0] ]) shape_skew = np.array([[4, 4, 0], [0, 4, 4] ]) shape_skew2 = np.array([[4, 0], [4, 4], [0, 4] ]) shape_skew3 = np.array([[0, 4, 4], [4, 4, 0] ]) shape_skew4 = np.array([[0, 4], [4, 4], [4, 0] ]) shape_t = np.array([[5, 0], [5, 5], [5, 0] ]) shape_t2 = np.array([[5, 5, 5], [0, 5, 0] ]) shape_t3 = np.array([[0, 5], [5, 5], [0, 5] ]) shape_t4 = np.array([[0, 5, 0], [5, 5, 5] ]) new_shape = np.array(default_grid + shape_t4) --- I ran a test with just inserting one shape into a grid. But the following error messages comes up. In the final code, I want there to be a way to check the coordinates and insert the shapes there will many for loops and if elif statements. Error Message: 120, in new_shape = np.array(default_grid + shape_t4) ValueError: operands could not be broadcast together with shapes (6,6) (2,3) ? Regards, Amogh Atwe From phillor9 at gmail.com Thu Jun 23 20:57:15 2022 From: phillor9 at gmail.com (Phil) Date: Fri, 24 Jun 2022 10:57:15 +1000 Subject: [Tutor] Event loop logic question In-Reply-To: References: Message-ID: <74a97182-bacb-deb1-479b-6d7a792504d0@gmail.com> Problem solved. I suddenly realised, while watching the output of the print statements, that I don't need the update function. Instead the control statements should be in the keyboard reading functions. Advice for self; don't bother the mailing list but instead give the problem a rest for a day or two. -- Regards, Phil From marcus.luetolf at bluewin.ch Sat Jun 25 14:45:37 2022 From: marcus.luetolf at bluewin.ch (marcus.luetolf at bluewin.ch) Date: Sat, 25 Jun 2022 20:45:37 +0200 Subject: [Tutor] problem solving with lists: final (amateur) solution Message-ID: <000f01d888c3$c32e6eb0$498b4c10$@bluewin.ch> Hello Experts, hello dn, it's a while since I - in terms of Mark Lawrence - bothered you with my problem. Thanks to your comments, especially to dn's structured guidance I've come up with the code below, based on repeatability. I am shure there is room for improvement concerning pythonish style but for the moment the code serves my purposes. A commented version can be found on https://github.com/luemar/player_startlist. def startlist(all_players, num_in_flight): c_all_players = all_players[:] history = {'a':[], 'b':[],'c':[],'d':[],'e':[],'f':[],'g':[],'h':[],\ 'i':[],'j':[],'k':[],'l':[],'m':[],'n':[],'o':[],'p':[]} print('...............day_1................') def day_1_flights(): key_hist = list(history.keys()) c_key_hist = key_hist[:] for dummy_i in c_key_hist: print('flights_day_1: ', c_key_hist[:num_in_flight]) for key in c_key_hist[:num_in_flight]: [history[key].append(player)for player in c_all_players[0:num_in_flight]] del c_key_hist[:num_in_flight] del c_all_players[0:num_in_flight] day_1_flights() def day_2_to_5_flights(): flights = {} for i in range(2,6): print('...............day_' + str(i)+'................') flights['a_flight_day_'+str(i)]= [] flights['b_flight_day_'+str(i)]= [] flights['c_flight_day_'+str(i)]= [] flights['d_flight_day_'+str(i)]= [] lead = list('abcd') flight_list = [flights['a_flight_day_'+str(i)], flights['b_flight_day_'+str(i)],\ flights['c_flight_day_'+str(i)], flights['d_flight_day_'+str(i)]] for j in range(len(flight_list)): def flight(cond, day): for player in all_players: if player not in cond: day.extend(player) cond.extend(history[player]) history[lead[j]].extend(player) day.extend(lead[j]) day.sort() [history[pl].extend(day) for pl in day[1:]] return lead[j]+'_flight_day_'+str(i)+ ': ' + str(flight_list[j]) conditions = [history[lead[j]], history[lead[j]] + flights['a_flight_day_'+str(i)],\ history[lead[j]] + flights['a_flight_day_'+str(i)] + \ flights['b_flight_day_'+str(i)], \ history[lead[j]] + flights['a_flight_day_'+str(i)] + \ flights['b_flight_day_'+str(i)]+ flights['c_flight_day_'+str(i)]] print(flight(list(set(conditions[j])), flight_list[j])) day_2_to_5_flights() startlist(list('abcdefghijklmnop'), 4) Many thanks, Marcus. -----Urspr?ngliche Nachricht----- Von: Tutor Im Auftrag von dn Gesendet: Mittwoch, 25. Mai 2022 22:02 An: tutor at python.org Betreff: Re: [Tutor] problem solving with lists: preliminary solution Hi Marcus, This is a bit of 'blast from the past' - and I haven't gone 'back' to look at our previous correspondence/this message hasn't "threaded". May I offer a few criticisms and suggestions:- On 26/05/2022 05.19, marcus.luetolf at bluewin.ch wrote: > ....sorry, forgott correct e_mail address..... > > Hello Experts, hello dn, and not forgetting @wlfraed who mentioned university research papers dealing with the intricacies (and insolubles) of the "SGP". Did you try following-up on any of those? > In resuming my task to write code for a special constellation of the > "SocialGolferProblem" (SGP) : > number_of_days (number_of_weeks) = 5 > number_of_flights (number_of_groupings) = 4 seize of flight > (players_per_grouping) = 4 Criticism: although each of these terms is perfectly understandable, and easily converted into a 'readable' pythonic identifier, later the code uses "n" (for example) - which of the two values commencing "*n*umber_" is it? > and given the solution below (draws for week 1 through 5) I've come up > with a code of which I only post part of it (for it gets to big): > for day 1( hardcoded) and for day 2 (as a function to be used through > day 5). The word "hardcoded" immediately stopped me in my tracks! The whole point of using the computer is to find 'repetition' and have the machine/software save us from such boredom (or nit-picking detail in which we might make an error/become bored). That said, may I suggest that you grab a whiteboard/blackboard, or a convenient wall and a bunch of Post-It notes, and try to solve the problem manually - first for week-one (which is largely trivial), but then recording the 'history' and moving-on to the extra decision-layer for week-two. If your brain doesn't go 'click'*, go on to deal with week-three and see how the algorithm develops... * keeping my 'pocket handkerchief' lawn tidy doesn't require 'heavy machinery', but after many years of service my little electric mower 'died'. I bought a new Bosch model which needed to be partly assembled from the box. The accompanying manual featured explanations in dozens of languages. However, the (single/all-language) diagram showing where the back-wheels were to be joined to the axles featured (only) the English explanation: "Click". Perhaps other languages do use the word (?spelling), but the term "going click in your mind" has long been used for the idea explained in today's idiom as an "ahah! moment". > The crucial part of my functions for day 2 to 5 are the conditional > statements. These conditional statements do not follow a consistent pattern. > This inconsistency casts some doubt on the effectiveness of my code > below and I'd appreciate critical comments. > ... > a_flight_day_1 = [] > b_flight_day_1 = [] > c_flight_day_1 = [] ... > history = {'a':[], > 'b':[],'c':[],'d':[],'e':[],'f':[],'g':[],'h':[],'i':[],'j':[],'k':[],'l':[] ,'m':[],'n':[],'o':[],'p':[]} ... Following-on from talking about looping (repetition), yes we need to use multiple conditional expressions to ensure that history is taken into account (consistently). The other 'side' of both of these code-constructs is the data-construct. Code-loops require data-collections! The hard-coded "a" and "day_1" made me shudder. (not a pretty sight - the code, nor me shuddering!) Would you benefit from spending a little time, putting aside the SGP for a moment, and looking at a tutorial on "nesting" lists (and dictionaries) in Python? Again, the above-mentioned 'whiteboard' exercise may help 'reveal' the patterns in the data, as well as identifying an algorithm. Sadly, the 'hard-coded' parts may 'help' sort-out week-one, but (IMHO) have made things impossibly-difficult to proceed into week-two (etc). (and I'm back to mentioning that I haven't searched for our previous discussions - specifically whether we talked-about a data-construct for 'history', and also to referring-back to the 'whiteboard' suggestion) We were warned! The SGP can be made to work for this particular combination of weeks/flights/players. Which gives credence to the first-thought: "it looks so easy". However, the wider problem is more complex than one at-first imagines! This is the reason why 'combinatorial problems' are so interesting. Did I say "interesting"? Perhaps "frustrating" and/or "frustratingly difficult to solve" - or even: 'cannot (always) be solved'! This is no 'little kitten' or some old piece of rope, you have "a tiger by the tail"... -- Regards, =dn _______________________________________________ Tutor maillist - Tutor at python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor From alan.gauld at yahoo.co.uk Sat Jun 25 18:59:40 2022 From: alan.gauld at yahoo.co.uk (Alan Gauld) Date: Sat, 25 Jun 2022 23:59:40 +0100 Subject: [Tutor] Event loop logic question In-Reply-To: <74a97182-bacb-deb1-479b-6d7a792504d0@gmail.com> References: <74a97182-bacb-deb1-479b-6d7a792504d0@gmail.com> Message-ID: On 24/06/2022 01:57, Phil wrote: > Problem solved. > > I suddenly realised, while watching the output of the print statements, > that I don't need the update function. Instead the control statements > should be in the keyboard reading functions. I don't know your particular framework but as a general rule UI changes should be in the paint/draw/update function and only state data changes in the event handlers. So based purely on general principles I'd say you are completely wrong and ALL of the print tatements should be in the update and none of them in the key hanling functions. But that assumes that somewhere your framework calls update regularly to redraw the UI. If that's not the case then all bets are off. But if it is true you will save yourself an awful lot of grief later by keeping the print statements in one place. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos From phillor9 at gmail.com Sat Jun 25 20:06:25 2022 From: phillor9 at gmail.com (Phil) Date: Sun, 26 Jun 2022 10:06:25 +1000 Subject: [Tutor] Event loop logic question In-Reply-To: References: <74a97182-bacb-deb1-479b-6d7a792504d0@gmail.com> Message-ID: <89410b66-f8cb-2712-398a-86a3af319b67@gmail.com> On 26/6/22 08:59, Alan Gauld via Tutor wrote: > > I don't know your particular framework but as a general rule UI > changes should be in the paint/draw/update function and only > state data changes in the event handlers. Thank you Alan once again for your time and your comments are always welcome. I was looking for an easy way to repeatedly read the keyboard and pygamezero fills the bill. The problem that I initially had was that the update function is called many times per second and sending control commands from there caused the receiving hardware to be overwhelmed and then to crash. The UI is a 20 * 20 square and servers no use at all, it's just a requirement as far as I can tell, and so does not need updating. All I need is to read the arrow keys and to be able to quit the programme. There is no need for any user output to be displayed on the transmitting laptop, the print statements were just a debugging aid. Briefly, I SSH from my laptop to a raspberry pi via my mobile phone created WiFi network. No doubt it's an amateurish solution, however, it's simple and does exactly what I had intended. -- Regards, Phil From PythonList at DancesWithMice.info Sat Jun 25 20:34:54 2022 From: PythonList at DancesWithMice.info (dn) Date: Sun, 26 Jun 2022 12:34:54 +1200 Subject: [Tutor] problem solving with lists: final (amateur) solution In-Reply-To: <000f01d888c3$c32e6eb0$498b4c10$@bluewin.ch> References: <000f01d888c3$c32e6eb0$498b4c10$@bluewin.ch> Message-ID: On 26/06/2022 06.45, marcus.luetolf at bluewin.ch wrote: > Hello Experts, hello dn, > it's a while since I - in terms of Mark Lawrence - bothered you with my > problem. > Thanks to your comments, especially to dn's structured guidance I've come up > with the code below, based on repeatability. > I am shure there is room for improvement concerning pythonish style but for > the moment the code serves my purposes. > A commented version can be found on > https://github.com/luemar/player_startlist. > > def startlist(all_players, num_in_flight): > c_all_players = all_players[:] > history = {'a':[], 'b':[],'c':[],'d':[],'e':[],'f':[],'g':[],'h':[],\ > 'i':[],'j':[],'k':[],'l':[],'m':[],'n':[],'o':[],'p':[]} > print('...............day_1................') > def day_1_flights(): > key_hist = list(history.keys()) > c_key_hist = key_hist[:] > for dummy_i in c_key_hist: > print('flights_day_1: ', c_key_hist[:num_in_flight]) > for key in c_key_hist[:num_in_flight]: > [history[key].append(player)for player in > c_all_players[0:num_in_flight]] > del c_key_hist[:num_in_flight] > del c_all_players[0:num_in_flight] > day_1_flights() > > def day_2_to_5_flights(): > flights = {} > for i in range(2,6): > print('...............day_' + str(i)+'................') > flights['a_flight_day_'+str(i)]= [] > flights['b_flight_day_'+str(i)]= [] > flights['c_flight_day_'+str(i)]= [] > flights['d_flight_day_'+str(i)]= [] > lead = list('abcd') > flight_list = [flights['a_flight_day_'+str(i)], > flights['b_flight_day_'+str(i)],\ > flights['c_flight_day_'+str(i)], > flights['d_flight_day_'+str(i)]] > > for j in range(len(flight_list)): > def flight(cond, day): > for player in all_players: > if player not in cond: > day.extend(player) > cond.extend(history[player]) > history[lead[j]].extend(player) > day.extend(lead[j]) > day.sort() > [history[pl].extend(day) for pl in day[1:]] > return lead[j]+'_flight_day_'+str(i)+ ': ' + > str(flight_list[j]) > > conditions = [history[lead[j]], history[lead[j]] + > flights['a_flight_day_'+str(i)],\ > history[lead[j]] + > flights['a_flight_day_'+str(i)] + \ > flights['b_flight_day_'+str(i)], \ > history[lead[j]] + > flights['a_flight_day_'+str(i)] + \ > flights['b_flight_day_'+str(i)]+ > flights['c_flight_day_'+str(i)]] > print(flight(list(set(conditions[j])), flight_list[j])) > day_2_to_5_flights() > startlist(list('abcdefghijklmnop'), 4) > > Many thanks, Marcus. ... > The word "hardcoded" immediately stopped me in my tracks! > > The whole point of using the computer is to find 'repetition' and have the > machine/software save us from such boredom (or nit-picking detail in which > we might make an error/become bored). ... > The other 'side' of both of these code-constructs is the data-construct. > Code-loops require data-collections! The hard-coded "a" and "day_1" made me > shudder. > (not a pretty sight - the code, nor me shuddering!) ... > Sadly, the 'hard-coded' parts may 'help' sort-out week-one, but (IMHO) have > made things impossibly-difficult to proceed into week-two (etc). ... It works. Well done! What could be better than that? [Brutal] critique: - ?pythonish? in German becomes ?pythonic? in English (but I'm sure we all understood) - position the two inner-functions outside and before startlist() - whereas the ?4?, ie number of players per flight (num_in_flight), is defined as a parameter in the call to startlist(), the five ?times or days? is a 'magic constant' (worse, it appears in day_2_to_5_flights() as part of ?range(2,6)? which 'disguises' it due to Python's way of working) - the comments also include reference to those parameters as if they are constants (which they are - if you only plan to use the algorithm for this 16-4-5 configuration of the SGP). Thus, if the function were called with different parameters, the comments would be not only wrong but have the potential to mislead the reader - in the same vein (as the two points above), the all_players (variable) argument is followed by the generation of history as a list of constants (?constant? cf ?variable?) - on top of which: day_1_flights() generates key_hist from history even though it already exists as all_players - the Python facility for a 'dummy value' (that will never be used, or perhaps only 'plugged-in' to 'make things happen') is _ (the under-score/under-line character), ie for _ in c_key_hist: - an alternative to using a meaningless 'placeholder' with no computational-purpose, such as _ or dummy_i, is to choose an identifier which aids readability, eg for each_flight in c_key_hist - well done for noting that a list-comprehension could be used to generate history/ies. Two thoughts: 1 could the two for-loops be combined into a single nested list-comprehension? 2 does the reader's mind have to 'change gears' between reading the outer for-loop as a 'traditional-loop' structure, and then the inner-loop as a list-comprehension? ie would it be better to use the same type of code-construct for both? - both the code- and data-structures of day_1_flights() seem rather tortured (and tortuous), and some are unused and therefore unnecessary. Might it be possible to simplify, if the control-code commences with: for first_player_index in range( 0, len( all_players ), num_in_flight ): print( first_player_index, all_players[ first_player_index: first_player_index+num_in_flight ] ) NB the print() is to make the methodology 'visible'. - the docstring for day_1_flights() is only partly-correct. Isn't the function also creating and building the history set? - that being the case, should the initial set-definition be moved inside the function? - functions should not depend upon global values. How does the history 'pass' from one function to another - which is allied to the question: how do the functions know about values such as _all_players and num_in_flight? To make the functions self-contained and ?independent?, these values should be passed-in as parameters/arguments and/or return-ed - much of the above also applies to day_2_to_5_flights() - chief concern with day_2_to_5_flights() is: what happens to d_flight_day_N if there are fewer/more than four players per flight, or what if there are fewer/more than 5 flights? - the observation that the same players would always be the 'lead' of a flight, is solid. Thus, could the lead-list be generated from a provided-parameter, rather than stated as a constant? Could that construct (also) have been used in the earlier function? - we know (by definition) that flight() is an unnecessary set of conditions to apply during day_1, but could it be used nonetheless? If so, could day_1_flights() be 'folded into' day_2_to_5_flights() instead of having separate functions? (yes, I recall talking about the essential differences in an earlier post - and perhaps I'm biased because this was how I structured the draft-solution) [More than] enough for now? -- Regards, =dn From wlfraed at ix.netcom.com Sat Jun 25 22:55:14 2022 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Sat, 25 Jun 2022 22:55:14 -0400 Subject: [Tutor] Event loop logic question References: <74a97182-bacb-deb1-479b-6d7a792504d0@gmail.com> <89410b66-f8cb-2712-398a-86a3af319b67@gmail.com> Message-ID: On Sun, 26 Jun 2022 10:06:25 +1000, Phil declaimed the following: WARNING: LONG HARANGUE FOLLOWS > >I was looking for an easy way to repeatedly read the keyboard and >pygamezero fills the bill. The problem that I initially had was that the >update function is called many times per second and sending control >commands from there caused the receiving hardware to be overwhelmed and >then to crash. The UI is a 20 * 20 square and servers no use at all, >it's just a requirement as far as I can tell, and so does not need >updating. All I need is to read the arrow keys and to be able to quit >the programme. > In most GUI frameworks -- update would be called whenever the framework has processed an event (some may have an ability to process a couple of events and collect changes for one update call). Pygame Zero apparently uses update() to render changes to the GUI buffer, and a separate draw() operation to copy the buffer to the screen. There is no inherent means of rate-limiting a GUI framework. YOU will have to write code to do rate limiting either within the event handlers, or in the update operation. That likely means you need to track system elapsed time for each event, issue an actual update to your device only if enough elapsed time has passed and there has been no change in event type; OR you issue a command when the event type changes. Looking for repeated key events can be troublesome. The OS handles the keyboard repeat rate, and sends them (events) to whatever application/framework is asking for them. Most likely, that means you might receive a keydown/keyup event pair for each repetition. The only way to confirm is to have a test application log (probably to a file, as it can buffer entries rather than slowly updating a screen) each event it sees. Don't know if it applies but... https://www.mattlayman.com/blog/2019/teach-kid-code-pygame-zero/ """ The common pieces that you must handle yourself in Pygame are pre-wired in Pygame Zero. This dramatically lowers the barrier to entry, at the cost of flexibility. It?s absolutely the right trade-off for teaching. """ ... could mean stuff you are trying to do is not really available. However, from the documentation, the best option for rate limiting is to use clock.schedule_interval() configured to call a function that issues commands to the end device, it will use an interval that will not overload the device (you may have to experiment -- 1.0/20.0 would produce 20 commands per second). You will NOT issue commands from .update() or .draw() (though you might use them to have that "unused" GUI window produce a heartbeat display). Your .on_key_down() and .on_key_up() callbacks just set a state flag (I'm assuming more than binary state -- EXIT, IDLE, FORWARD, whatever else. But you do need to verify that holding a key down doesn't result in multiple down/up calls as the system repeats the key. If it does, you will have more difficulties since the key handler state could go from, say FORWARD to IDLE before the scheduler checks for command sending. I'm tempted to suggest that you ignore key_up events. You set the state for the depressed key, and have the scheduler clear the state to "IDLE" (but don't issue the idle command on this invocation). If no key has been pressed since the scheduler cleared it, the next scheduled time will see the IDLE and issue the proper command. Otherwise, a repeating key will be another key_down event on return from the scheduler, and the event handler will just reset the state to the proper value for that key. If the "GUI" is not really being used, why even invoke a GUI framework. Just run a console application. On Windows, there is the msvcrt module in the standard library. It has functions for """ msvcrt.kbhit() Return True if a keypress is waiting to be read. msvcrt.getch() Read a keypress and return the resulting character as a byte string. Nothing is echoed to the console. This call will block if a keypress is not already available, but will not wait for Enter to be pressed. If the pressed key was a special function key, this will return '\000' or '\xe0'; the next call will return the keycode. The Control-C keypress cannot be read with this function. """ For Linux systems, there are a few modules to terminal IO control (but understanding what you'd need to get characters as entered. Might be easier to create a curses application (which should be available for Linux, Windows ports are more problematic). """ curses.cbreak() Enter cbreak mode. In cbreak mode (sometimes called ?rare? mode) normal tty line buffering is turned off and characters are available to be read one by one. However, unlike raw mode, special characters (interrupt, quit, suspend, and flow control) retain their effects on the tty driver and calling program. Calling first raw() then cbreak() leaves the terminal in cbreak mode. """ Note that doesn't mention key down/key up. """ curses.noecho() Leave echo mode. Echoing of input characters is turned off. """ """ window.getch([y, x]) Get a character. Note that the integer returned does not have to be in ASCII range: function keys, keypad keys and so on are represented by numbers higher than 255. In no-delay mode, return -1 if there is no input, otherwise wait until a key is pressed. window.getkey([y, x]) Get a character, returning a string instead of an integer, as getch() does. Function keys, keypad keys and other special keys return a multibyte string containing the key name. In no-delay mode, raise an exception if there is no input. """ Don't see an equivalent for .kbhit() -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From phillor9 at gmail.com Sat Jun 25 23:56:52 2022 From: phillor9 at gmail.com (Phil) Date: Sun, 26 Jun 2022 13:56:52 +1000 Subject: [Tutor] Event loop logic question In-Reply-To: References: <74a97182-bacb-deb1-479b-6d7a792504d0@gmail.com> <89410b66-f8cb-2712-398a-86a3af319b67@gmail.com> Message-ID: On 26/6/22 12:55, Dennis Lee Bieber wrote: > On Sun, 26 Jun 2022 10:06:25 +1000, Phil declaimed the > following: > > WARNING: LONG HARANGUE FOLLOWS > > If the "GUI" is not really being used, why even invoke a GUI framework. > Just run a console application. Thank you Dennis, I have used ncurses in the past, and it's no doubt the best option, but I couldn't remember how to read the keyboard but I do remember that it wasn't straight forward so I chose the simplest option. I've completed the project and about to move onto something else but I'll have another look at ncurses just as an exercise. Maybe there's an alternative to ncurses for Linux, I'll check. -- Regards, Phil From alexkleider at gmail.com Sun Jun 26 15:37:43 2022 From: alexkleider at gmail.com (Alex Kleider) Date: Sun, 26 Jun 2022 12:37:43 -0700 Subject: [Tutor] Feedback on coding style In-Reply-To: References: Message-ID: I don't see this code on your github account. (https://github.com/LeamHall?tab=repositories) I'd be interested in seeing the current version having done something somewhat related. (https://github.com/alexKleider/blood-pressure-record) My code needs a little updating (with regard to the README at least) which I'll do if prodded a bit:-) a PS Disclaimer: python for me is an avocation, not a vocation! On Mon, May 9, 2022 at 5:03 AM Leam Hall wrote: > > Hey all, > > I'm looking for general Python code critique, feel free to share snarky comments. :) The parameters are: > 1. Code to Python 3.6 or so. > 2. Use only the standard library or locally created modules. > 3. Keep it clean and simple so new Pythonistas can understand and contribute. > > Let me know how to make this better. > > Leam > -- > Automation Engineer (reuel.net/resume) > Scribe: The Domici War (domiciwar.net) > General Ne'er-do-well (github.com/LeamHall) > > ### > > #!/usr/bin/env python3 > > # name: bp_tracker.py > # version: 0.0.1 > # date: 20220509 > # author: Leam Hall > # desc: Track and report on blood pressure numbers. > > # Notes: > # Datafile expects three ints and one float, in order. > > # TODO > # Add statistical analysis for standard deviation. > # Report based on time of day (early, midmorning, afternoon, evening) > # (?) Add current distance from goal? > > import argparse > from datetime import datetime > import os.path > > def array_from_file(report_file): > data = [] > with open(report_file, 'r') as file: > for line in file: > line.strip() > datum = line.split() > if len(datum) == 4: > data.append(datum) > else: > continue > return data > > def report(report_data): > highest_systolic = 0 > highest_diastolic = 0 > highest_pulse = 0 > latest = -1.0 > for datum in report_data: > systolic = int(datum[0]) > diastolic = int(datum[1]) > pulse = int(datum[2]) > date = float(datum[3]) > if systolic > highest_systolic: > highest_systolic = systolic > highest_systolic_event = datum > if diastolic > highest_diastolic: > highest_diastolic = diastolic > highest_diastolic_event = datum > if pulse > highest_pulse: > highest_pulse = pulse > highest_pulse_event = datum > if date > latest: > latest_record = datum > > print("Highest Systolic: {}/{} {} {}".format(*highest_systolic_event)) > print("Highest Diastolic: {}/{} {} {}".format(*highest_diastolic_event)) > print("Highest Pulse: {}/{} {} {}".format(*highest_pulse_event)) > print("Latest Record: {}/{} {} {}".format(*latest_record)) > > def result_string(report_list): > return "{} {} {} {}".format(*report_list) > > report_file = "bp_numbers.txt" > > parser = argparse.ArgumentParser() > parser.add_argument("-a", "--add", nargs=3, > help = "Add in the order of systolic, diastolic, pulse") > parser.add_argument("-f", "--file", help = "Report file") > args = parser.parse_args() > > if args.file: > report_file = args.file > > if args.add: > # This format allows sequencing now and parsing later. > timestamp = datetime.now().strftime("%Y%m%d.%H%M") > this_report = args.add > this_report.append(timestamp) > with open(report_file, 'a') as file: > file.write(result_string(this_report) + "\n") > else: > # Default behavior is to report. > if os.path.exists(report_file): > try: > report_data = array_from_file(report_file) > report(report_data) > except: > print("Error processing report data") > else: > print("Cannot find ", report_file) > > ### > _______________________________________________ > Tutor maillist - Tutor at python.org > To unsubscribe or change subscription options: > https://mail.python.org/mailman/listinfo/tutor -- alex at kleider.ca (sent from my current gizmo) From leamhall at gmail.com Sun Jun 26 15:53:32 2022 From: leamhall at gmail.com (Leam Hall) Date: Sun, 26 Jun 2022 14:53:32 -0500 Subject: [Tutor] Feedback on coding style In-Reply-To: References: Message-ID: <7e75e30d-7a48-3098-7cc2-63a5950aca63@gmail.com> Hey Alex, here's the code: https://github.com/LeamHall/admin_tools/blob/master/bp_tracker.py I try to keep things simple, it lets you specify a file "-f", and you can use "-a" to add numbers. The added data gets appended to a file, with the timestamp of the add as a data point. The plan is to allow a "for the past X days" as a range, without forcing the file to remain in order. For example: 150 93 73 20220512.0832 That's systolic, diastolic, pulse, and the automatically appended timestamp in "YYYYMMDD.HHMM" format. Leam On 6/26/22 14:37, Alex Kleider wrote: > I don't see this code on your github account. > (https://github.com/LeamHall?tab=repositories) > I'd be interested in seeing the current version having done something > somewhat related. > (https://github.com/alexKleider/blood-pressure-record) > My code needs a little updating (with regard to the README at least) > which I'll do if prodded a bit:-) > a > PS Disclaimer: python for me is an avocation, not a vocation! > > On Mon, May 9, 2022 at 5:03 AM Leam Hall wrote: >> >> Hey all, >> >> I'm looking for general Python code critique, feel free to share snarky comments. :) The parameters are: >> 1. Code to Python 3.6 or so. >> 2. Use only the standard library or locally created modules. >> 3. Keep it clean and simple so new Pythonistas can understand and contribute. >> >> Let me know how to make this better. >> >> Leam >> -- >> Automation Engineer (reuel.net/resume) >> Scribe: The Domici War (domiciwar.net) >> General Ne'er-do-well (github.com/LeamHall) >> >> ### >> >> #!/usr/bin/env python3 >> >> # name: bp_tracker.py >> # version: 0.0.1 >> # date: 20220509 >> # author: Leam Hall >> # desc: Track and report on blood pressure numbers. >> >> # Notes: >> # Datafile expects three ints and one float, in order. >> >> # TODO >> # Add statistical analysis for standard deviation. >> # Report based on time of day (early, midmorning, afternoon, evening) >> # (?) Add current distance from goal? >> >> import argparse >> from datetime import datetime >> import os.path >> >> def array_from_file(report_file): >> data = [] >> with open(report_file, 'r') as file: >> for line in file: >> line.strip() >> datum = line.split() >> if len(datum) == 4: >> data.append(datum) >> else: >> continue >> return data >> >> def report(report_data): >> highest_systolic = 0 >> highest_diastolic = 0 >> highest_pulse = 0 >> latest = -1.0 >> for datum in report_data: >> systolic = int(datum[0]) >> diastolic = int(datum[1]) >> pulse = int(datum[2]) >> date = float(datum[3]) >> if systolic > highest_systolic: >> highest_systolic = systolic >> highest_systolic_event = datum >> if diastolic > highest_diastolic: >> highest_diastolic = diastolic >> highest_diastolic_event = datum >> if pulse > highest_pulse: >> highest_pulse = pulse >> highest_pulse_event = datum >> if date > latest: >> latest_record = datum >> >> print("Highest Systolic: {}/{} {} {}".format(*highest_systolic_event)) >> print("Highest Diastolic: {}/{} {} {}".format(*highest_diastolic_event)) >> print("Highest Pulse: {}/{} {} {}".format(*highest_pulse_event)) >> print("Latest Record: {}/{} {} {}".format(*latest_record)) >> >> def result_string(report_list): >> return "{} {} {} {}".format(*report_list) >> >> report_file = "bp_numbers.txt" >> >> parser = argparse.ArgumentParser() >> parser.add_argument("-a", "--add", nargs=3, >> help = "Add in the order of systolic, diastolic, pulse") >> parser.add_argument("-f", "--file", help = "Report file") >> args = parser.parse_args() >> >> if args.file: >> report_file = args.file >> >> if args.add: >> # This format allows sequencing now and parsing later. >> timestamp = datetime.now().strftime("%Y%m%d.%H%M") >> this_report = args.add >> this_report.append(timestamp) >> with open(report_file, 'a') as file: >> file.write(result_string(this_report) + "\n") >> else: >> # Default behavior is to report. >> if os.path.exists(report_file): >> try: >> report_data = array_from_file(report_file) >> report(report_data) >> except: >> print("Error processing report data") >> else: >> print("Cannot find ", report_file) >> >> ### >> _______________________________________________ >> Tutor maillist - Tutor at python.org >> To unsubscribe or change subscription options: >> https://mail.python.org/mailman/listinfo/tutor > > > -- Automation Engineer (reuel.net/resume) Scribe: The Domici War (domiciwar.net) General Ne'er-do-well (github.com/LeamHall) From alexkleider at gmail.com Sun Jun 26 16:10:26 2022 From: alexkleider at gmail.com (Alex Kleider) Date: Sun, 26 Jun 2022 13:10:26 -0700 Subject: [Tutor] Feedback on coding style In-Reply-To: <7e75e30d-7a48-3098-7cc2-63a5950aca63@gmail.com> References: <7e75e30d-7a48-3098-7cc2-63a5950aca63@gmail.com> Message-ID: Thanks for (so promptly!) getting back to me. I already tried that and was unsuccessful: (perci) alex at t460:~/Notes/Py$ cd ~/Git/LH/ (perci) alex at t460:~/Git/LH$ git clone https://github.com/LeamHall/admin_tools/blob/master/bp_tracker.py Cloning into 'bp_tracker.py'... fatal: repository 'https://github.com/LeamHall/admin_tools/blob/master/bp_tracker.py/' not found (perci) alex at t460:~/Git/LH$ I was able (successfully) to clone your resume_writer: $ git clone https://github.com/LeamHall/resume_writer I'm not very proficient with git and github (which is probably pretty apparent:-) The bp_tracker seems to be embedded within something else so I'm guessing that's what's leading to the error. a On Sun, Jun 26, 2022 at 12:55 PM Leam Hall wrote: > > Hey Alex, here's the code: > > https://github.com/LeamHall/admin_tools/blob/master/bp_tracker.py > > I try to keep things simple, it lets you specify a file "-f", and you can use "-a" to add numbers. The added data gets appended to a file, with the timestamp of the add as a data point. The plan is to allow a "for the past X days" as a range, without forcing the file to remain in order. For example: > > 150 93 73 20220512.0832 > > That's systolic, diastolic, pulse, and the automatically appended timestamp in "YYYYMMDD.HHMM" format. > > Leam > > > On 6/26/22 14:37, Alex Kleider wrote: > > I don't see this code on your github account. > > (https://github.com/LeamHall?tab=repositories) > > I'd be interested in seeing the current version having done something > > somewhat related. > > (https://github.com/alexKleider/blood-pressure-record) > > My code needs a little updating (with regard to the README at least) > > which I'll do if prodded a bit:-) > > a > > PS Disclaimer: python for me is an avocation, not a vocation! > > > > On Mon, May 9, 2022 at 5:03 AM Leam Hall wrote: > >> > >> Hey all, > >> > >> I'm looking for general Python code critique, feel free to share snarky comments. :) The parameters are: > >> 1. Code to Python 3.6 or so. > >> 2. Use only the standard library or locally created modules. > >> 3. Keep it clean and simple so new Pythonistas can understand and contribute. > >> > >> Let me know how to make this better. > >> > >> Leam > >> -- > >> Automation Engineer (reuel.net/resume) > >> Scribe: The Domici War (domiciwar.net) > >> General Ne'er-do-well (github.com/LeamHall) > >> > >> ### > >> > >> #!/usr/bin/env python3 > >> > >> # name: bp_tracker.py > >> # version: 0.0.1 > >> # date: 20220509 > >> # author: Leam Hall > >> # desc: Track and report on blood pressure numbers. > >> > >> # Notes: > >> # Datafile expects three ints and one float, in order. > >> > >> # TODO > >> # Add statistical analysis for standard deviation. > >> # Report based on time of day (early, midmorning, afternoon, evening) > >> # (?) Add current distance from goal? > >> > >> import argparse > >> from datetime import datetime > >> import os.path > >> > >> def array_from_file(report_file): > >> data = [] > >> with open(report_file, 'r') as file: > >> for line in file: > >> line.strip() > >> datum = line.split() > >> if len(datum) == 4: > >> data.append(datum) > >> else: > >> continue > >> return data > >> > >> def report(report_data): > >> highest_systolic = 0 > >> highest_diastolic = 0 > >> highest_pulse = 0 > >> latest = -1.0 > >> for datum in report_data: > >> systolic = int(datum[0]) > >> diastolic = int(datum[1]) > >> pulse = int(datum[2]) > >> date = float(datum[3]) > >> if systolic > highest_systolic: > >> highest_systolic = systolic > >> highest_systolic_event = datum > >> if diastolic > highest_diastolic: > >> highest_diastolic = diastolic > >> highest_diastolic_event = datum > >> if pulse > highest_pulse: > >> highest_pulse = pulse > >> highest_pulse_event = datum > >> if date > latest: > >> latest_record = datum > >> > >> print("Highest Systolic: {}/{} {} {}".format(*highest_systolic_event)) > >> print("Highest Diastolic: {}/{} {} {}".format(*highest_diastolic_event)) > >> print("Highest Pulse: {}/{} {} {}".format(*highest_pulse_event)) > >> print("Latest Record: {}/{} {} {}".format(*latest_record)) > >> > >> def result_string(report_list): > >> return "{} {} {} {}".format(*report_list) > >> > >> report_file = "bp_numbers.txt" > >> > >> parser = argparse.ArgumentParser() > >> parser.add_argument("-a", "--add", nargs=3, > >> help = "Add in the order of systolic, diastolic, pulse") > >> parser.add_argument("-f", "--file", help = "Report file") > >> args = parser.parse_args() > >> > >> if args.file: > >> report_file = args.file > >> > >> if args.add: > >> # This format allows sequencing now and parsing later. > >> timestamp = datetime.now().strftime("%Y%m%d.%H%M") > >> this_report = args.add > >> this_report.append(timestamp) > >> with open(report_file, 'a') as file: > >> file.write(result_string(this_report) + "\n") > >> else: > >> # Default behavior is to report. > >> if os.path.exists(report_file): > >> try: > >> report_data = array_from_file(report_file) > >> report(report_data) > >> except: > >> print("Error processing report data") > >> else: > >> print("Cannot find ", report_file) > >> > >> ### > >> _______________________________________________ > >> Tutor maillist - Tutor at python.org > >> To unsubscribe or change subscription options: > >> https://mail.python.org/mailman/listinfo/tutor > > > > > > > > -- > Automation Engineer (reuel.net/resume) > Scribe: The Domici War (domiciwar.net) > General Ne'er-do-well (github.com/LeamHall) > _______________________________________________ > Tutor maillist - Tutor at python.org > To unsubscribe or change subscription options: > https://mail.python.org/mailman/listinfo/tutor -- alex at kleider.ca (sent from my current gizmo) From leamhall at gmail.com Sun Jun 26 16:21:43 2022 From: leamhall at gmail.com (Leam Hall) Date: Sun, 26 Jun 2022 15:21:43 -0500 Subject: [Tutor] Feedback on coding style In-Reply-To: References: <7e75e30d-7a48-3098-7cc2-63a5950aca63@gmail.com> Message-ID: <54cf42fc-bbed-63f0-e096-0e3481b14def@gmail.com> Happy to help! My apologies, I sent you the direct link to the code. You should be able to clone: https://github.com/LeamHall/admin_tools and then it'll be in there. You need to create the "data" directory, and then run it with an "add" option to create the file. Of course, if you don't care about the timestamps at first, you can just copy in the data, where each line is: systolic diastolic pulse 0 The last 0 is the timestamp position, and each element in the line is white space delimited. you can manually enter the timestamp, if you like. Leam On 6/26/22 15:10, Alex Kleider wrote: > Thanks for (so promptly!) getting back to me. > I already tried that and was unsuccessful: > (perci) alex at t460:~/Notes/Py$ cd ~/Git/LH/ > (perci) alex at t460:~/Git/LH$ git clone > https://github.com/LeamHall/admin_tools/blob/master/bp_tracker.py > Cloning into 'bp_tracker.py'... > fatal: repository > 'https://github.com/LeamHall/admin_tools/blob/master/bp_tracker.py/' > not found > (perci) alex at t460:~/Git/LH$ > > I was able (successfully) to clone your resume_writer: > $ git clone https://github.com/LeamHall/resume_writer > I'm not very proficient with git and github (which is probably pretty > apparent:-) > The bp_tracker seems to be embedded within something else so I'm > guessing that's what's leading to the error. > a > > On Sun, Jun 26, 2022 at 12:55 PM Leam Hall wrote: >> >> Hey Alex, here's the code: >> >> https://github.com/LeamHall/admin_tools/blob/master/bp_tracker.py >> >> I try to keep things simple, it lets you specify a file "-f", and you can use "-a" to add numbers. The added data gets appended to a file, with the timestamp of the add as a data point. The plan is to allow a "for the past X days" as a range, without forcing the file to remain in order. For example: >> >> 150 93 73 20220512.0832 >> >> That's systolic, diastolic, pulse, and the automatically appended timestamp in "YYYYMMDD.HHMM" format. >> >> Leam >> >> >> On 6/26/22 14:37, Alex Kleider wrote: >>> I don't see this code on your github account. >>> (https://github.com/LeamHall?tab=repositories) >>> I'd be interested in seeing the current version having done something >>> somewhat related. >>> (https://github.com/alexKleider/blood-pressure-record) >>> My code needs a little updating (with regard to the README at least) >>> which I'll do if prodded a bit:-) >>> a >>> PS Disclaimer: python for me is an avocation, not a vocation! >>> >>> On Mon, May 9, 2022 at 5:03 AM Leam Hall wrote: >>>> >>>> Hey all, >>>> >>>> I'm looking for general Python code critique, feel free to share snarky comments. :) The parameters are: >>>> 1. Code to Python 3.6 or so. >>>> 2. Use only the standard library or locally created modules. >>>> 3. Keep it clean and simple so new Pythonistas can understand and contribute. >>>> >>>> Let me know how to make this better. >>>> >>>> Leam >>>> -- >>>> Automation Engineer (reuel.net/resume) >>>> Scribe: The Domici War (domiciwar.net) >>>> General Ne'er-do-well (github.com/LeamHall) >>>> >>>> ### >>>> >>>> #!/usr/bin/env python3 >>>> >>>> # name: bp_tracker.py >>>> # version: 0.0.1 >>>> # date: 20220509 >>>> # author: Leam Hall >>>> # desc: Track and report on blood pressure numbers. >>>> >>>> # Notes: >>>> # Datafile expects three ints and one float, in order. >>>> >>>> # TODO >>>> # Add statistical analysis for standard deviation. >>>> # Report based on time of day (early, midmorning, afternoon, evening) >>>> # (?) Add current distance from goal? >>>> >>>> import argparse >>>> from datetime import datetime >>>> import os.path >>>> >>>> def array_from_file(report_file): >>>> data = [] >>>> with open(report_file, 'r') as file: >>>> for line in file: >>>> line.strip() >>>> datum = line.split() >>>> if len(datum) == 4: >>>> data.append(datum) >>>> else: >>>> continue >>>> return data >>>> >>>> def report(report_data): >>>> highest_systolic = 0 >>>> highest_diastolic = 0 >>>> highest_pulse = 0 >>>> latest = -1.0 >>>> for datum in report_data: >>>> systolic = int(datum[0]) >>>> diastolic = int(datum[1]) >>>> pulse = int(datum[2]) >>>> date = float(datum[3]) >>>> if systolic > highest_systolic: >>>> highest_systolic = systolic >>>> highest_systolic_event = datum >>>> if diastolic > highest_diastolic: >>>> highest_diastolic = diastolic >>>> highest_diastolic_event = datum >>>> if pulse > highest_pulse: >>>> highest_pulse = pulse >>>> highest_pulse_event = datum >>>> if date > latest: >>>> latest_record = datum >>>> >>>> print("Highest Systolic: {}/{} {} {}".format(*highest_systolic_event)) >>>> print("Highest Diastolic: {}/{} {} {}".format(*highest_diastolic_event)) >>>> print("Highest Pulse: {}/{} {} {}".format(*highest_pulse_event)) >>>> print("Latest Record: {}/{} {} {}".format(*latest_record)) >>>> >>>> def result_string(report_list): >>>> return "{} {} {} {}".format(*report_list) >>>> >>>> report_file = "bp_numbers.txt" >>>> >>>> parser = argparse.ArgumentParser() >>>> parser.add_argument("-a", "--add", nargs=3, >>>> help = "Add in the order of systolic, diastolic, pulse") >>>> parser.add_argument("-f", "--file", help = "Report file") >>>> args = parser.parse_args() >>>> >>>> if args.file: >>>> report_file = args.file >>>> >>>> if args.add: >>>> # This format allows sequencing now and parsing later. >>>> timestamp = datetime.now().strftime("%Y%m%d.%H%M") >>>> this_report = args.add >>>> this_report.append(timestamp) >>>> with open(report_file, 'a') as file: >>>> file.write(result_string(this_report) + "\n") >>>> else: >>>> # Default behavior is to report. >>>> if os.path.exists(report_file): >>>> try: >>>> report_data = array_from_file(report_file) >>>> report(report_data) >>>> except: >>>> print("Error processing report data") >>>> else: >>>> print("Cannot find ", report_file) >>>> >>>> ### >>>> _______________________________________________ >>>> Tutor maillist - Tutor at python.org >>>> To unsubscribe or change subscription options: >>>> https://mail.python.org/mailman/listinfo/tutor >>> >>> >>> >> >> -- >> Automation Engineer (reuel.net/resume) >> Scribe: The Domici War (domiciwar.net) >> General Ne'er-do-well (github.com/LeamHall) >> _______________________________________________ >> Tutor maillist - Tutor at python.org >> To unsubscribe or change subscription options: >> https://mail.python.org/mailman/listinfo/tutor > > > -- Automation Engineer (reuel.net/resume) Scribe: The Domici War (domiciwar.net) General Ne'er-do-well (github.com/LeamHall) From PythonList at DancesWithMice.info Sun Jun 26 19:10:13 2022 From: PythonList at DancesWithMice.info (dn) Date: Mon, 27 Jun 2022 11:10:13 +1200 Subject: [Tutor] Feedback on coding style In-Reply-To: References: Message-ID: On 10/05/2022 00.01, Leam Hall wrote: > Hey all, > > I'm looking for general Python code critique, feel free to share snarky > comments.? :)? The parameters are: > ?1. Code to Python 3.6 or so. > ?2. Use only the standard library or locally created modules. > ?3. Keep it clean and simple so new Pythonistas can understand and > contribute. > > Let me know how to make this better. Apologies: was 'otherwise occupied' when this first hit the list. (and you're not Swiss, German, or ... so will try to stay away from what may seem like the 'snarky' way to reply) - as a reader, will I be more-interested in the location of your PC's python3 interpreter, the program[me]'s name, its version, origin-date, author, or its desc[ription]? - the university requires me to establish 'ownership' of all materials in case they are (later) used in courseware, articles, books, etc. Thus a rather legalistic line! Here is the standard 'base' used in a typical toy-example used to illustrate a particular concept during a local PUG-meeting: <<< """ Generator expression. """ __author__ = "David Neil, IT&T Consultant" __copyright__ = "Copyright ? 2022~" __license__ = "May not be copied" __python__ = "3.9" >>> Thus, an alternative to earlier suggestions of 'bundling' everything into a single docstring: the module docstring is separate, and each of these items is retrievable. Herewith: >>> import tuple_comprehension_demo as tcd >>> help( tcd ) Help on module tuple_comprehension_demo: NAME tuple_comprehension_demo - Generator expression. DATA __copyright__ = 'Copyright ? 2022~' __license__ = 'May not be copied' __python__ = '3.9' student_names = ('fred', 'wiLma', 'bARNEy', 'bettY', 'Pete', 'dn') tuple_of_names = ('Fred', 'Wilma', 'Barney', 'Betty', 'Pete', 'Dn') AUTHOR David Neil, IT&T Consultant FILE /home/dn/Projects/AuckPUG-2022-04-20/tuple_comprehension_demo.py >>> PS I'll be interested to see what 'normal people' do for this... - emphasising @Alan's comments: we (old-timers - me, not Alan!) used to talk about Input-Process-Output, ie in order to produce the output required, what processing will need to be done, and thus: what input(s) will be needed. I suggest that this is still a valid approach to decomposing a problem, and applies particularly well to the (two modes) way this code is to work - in fact, consider adding a fourth 'phase': 'Establish the Environment' (all the argparse stuff which will define which mode the user wishes to operate). Accordingly, the four 'phases' guide us into a 'first-layer' thinking-about/designing, and then coding solutions. More importantly (especially given the discussions about flat-file, CSV, and SQL storage options), if the code is split into *independent* sections, it becomes easy to change (or experiment with changes/upgrades) because only the one section will be affected - and the rest of the code should continue to operate (correctly, untouched, in perfect splendor...) - just because there was talk about options for persistence, I'll throw-in the suggestion of MongoDB (or similar). NoSQL DBs are a lot simpler to use than SQL (and I learned SQL before Oracle or SQL/DS (IBM DB2) were even commercial products, and train in the subject to this day!). More to the point, their data appears as if it was a Python data-structure (a list in this case), so no need to bend-your-head around much that is 'new'! (similar argument applies to JSON, YAML, etc) - for SCM (source code management) there are several good tools (I don't do CI/CD). I use git on my local machines, and for security-copies and sharing, push and pull to my GitLab or others'. One of the perqs of joining the local Open Source Society is access to a number of servers and services. One of which is a GitLab instance. That Microsoft cannot claim 'ownership' or otherwise use/make use of 'my' code (see legalism, above) is an important legal, if not, moral component to my decision (or that imposed upon me). GitLab.com tells us how much better (more mature) their offering is than GitHub (perhaps 'the other guys' do the same, to their benefit?), and outlines their $free/$paid service plans. - should you move to SCM, then suggest review of the header-comments, because dates and version-numbers become a function of the SCM rather than a manual coding-process! - recommend researching named-tuples (https://docs.python.org/3/library/collections.html?highlight=namedtuple#collections.namedtuple) then the datum can be given a less-generic name, eg blood_pressure_reading; and the three components similarly identified. Because each element of the tuple can be given its own name (hence the "named", hah!) there is no need for packing/unpacking between int(s) and the collection data-structure! - reiterating: 'names' are important. (yes, this is a short and simple module, but habits are hard to form - and all-to easy to slip) Thus, generic terms such as "report", "result_string", and "report_list" seem obvious at the time of coding - but will require more comprehension-effort (than necessary) at some (six-months in the) future time-of-reading! With today's text-editors able to pop-up suggestions for what we might be intending to type, identifier length (and thus precision-naming) is hardly an issue (cf temptations towards laziness/saving my worn-down fingers, ...) - unlabelled lists of 'anything' decrease readability! Accordingly, when it comes to function declarations (obvious example), after about two parameters (IMHO) we should be using keyword-arguments. The problem with format() (particularly in this example-code) is that it is preceded by a bunch of empty braces, and these have to be positionally-related to some code 'further over on the right' (of the code-line). Admittedly in this case things are not too bad because tuple-unpacking is in-use - so we don't have to keep track of four pairs of braces (on the left), and four identifiers (on the right). [it's a lot of effort for aged eyes!] Recommend researching f-strings (https://docs.python.org/3/reference/lexical_analysis.html#f-strings). The process becomes an expression (cf a function) but doesn't take much learning because the same format() protocol is applied. - finally, if you can persuade my at-home BP-machine to 'talk' to my computer, sign me up! At this time, I would have to copy from my bed-side scratch-pad into the PC, manually. (am too lazy for that!) Which does lead to the observation that now() is only applicable if the input is carried-out immediately after the BP-test - which saves adding another input field, but there's a human involved... -- Regards, =dn From leamhall at gmail.com Sun Jun 26 20:12:19 2022 From: leamhall at gmail.com (Leam Hall) Date: Sun, 26 Jun 2022 19:12:19 -0500 Subject: [Tutor] Feedback on coding style In-Reply-To: References: Message-ID: No apologies needed! Your input is habitually useful and welcome. It has taken me years, literally (I'm a bit slow), to figure out my coding style. While Python is adept in a lot of markets, I'm more of a tools person than an application programmer. My best work is usually to write something, make sure it's documented, train the gaining team on how it works, and then hand it off. In that environment, third-party libraries, venvs, and external application overhead (like MongoDB) don't really fit well. Plan text rules, and there's little reason for complex features. For the record, I passed one of the MongoDB U Python developer's classes. Fun stuff, and I've been considering doing more. Sadly, MongoDB v5 needs a CPU flag that my old Dell doesn't have. So I'm on the obsolescence track already. Of course, us tools folks are often tasked to write in different languages. The "#" symbol is a common, if not universal comment designator, and that's why I use it vice language specific things like "__author__". Keeping the interpreter and those lines up top really makes life easy when I have to sort through a few thousand files to track names and the last time it had major modifications. Some years ago my brother, a better programmer than I, told me about "input-output-process" and I try to stick to that. In this case, the input is technically just three items, BP, pulse, and timestamp. Usually we talk about the two BP numbers as a single unit, and the only reason they are separate here is to make parsing easier. The pulse is almost ancillary, and the timestamp is for sorting if I ever get to "last X number of days", or maybe chart across morning, midday, and evening. Because the data is stored in a text file, I can edit (vim, thank you. None of that pretty stuff) the timestamp if it needs changing. I'm on the fence about f-strings. Ruby has been doing that for years, and I usually prefer something that's common-ish between languages. Using printf or format() is less visually noisy, for me, than f-strings. f-strings have added a lot of usefulness, like Ruby, but much of what I do can be done before the print statement. Does that make sense? Leam On 6/26/22 18:10, dn wrote: > On 10/05/2022 00.01, Leam Hall wrote: >> Hey all, >> >> I'm looking for general Python code critique, feel free to share snarky >> comments.? :)? The parameters are: >> ?1. Code to Python 3.6 or so. >> ?2. Use only the standard library or locally created modules. >> ?3. Keep it clean and simple so new Pythonistas can understand and >> contribute. >> >> Let me know how to make this better. > > > Apologies: was 'otherwise occupied' when this first hit the list. (and > you're not Swiss, German, or ... so will try to stay away from what may > seem like the 'snarky' way to reply) > > > - as a reader, will I be more-interested in the location of your PC's > python3 interpreter, the program[me]'s name, its version, origin-date, > author, or its desc[ription]? > > - the university requires me to establish 'ownership' of all materials > in case they are (later) used in courseware, articles, books, etc. Thus > a rather legalistic line! Here is the standard 'base' used in a typical > toy-example used to illustrate a particular concept during a local > PUG-meeting: > > <<< > """ Generator expression. """ > > __author__ = "David Neil, IT&T Consultant" > __copyright__ = "Copyright ? 2022~" > __license__ = "May not be copied" > __python__ = "3.9" >>>> > > Thus, an alternative to earlier suggestions of 'bundling' everything > into a single docstring: the module docstring is separate, and each of > these items is retrievable. Herewith: > >>>> import tuple_comprehension_demo as tcd >>>> help( tcd ) > > Help on module tuple_comprehension_demo: > > NAME > tuple_comprehension_demo - Generator expression. > > DATA > __copyright__ = 'Copyright ? 2022~' > __license__ = 'May not be copied' > __python__ = '3.9' > student_names = ('fred', 'wiLma', 'bARNEy', 'bettY', 'Pete', 'dn') > tuple_of_names = ('Fred', 'Wilma', 'Barney', 'Betty', 'Pete', 'Dn') > > AUTHOR > David Neil, IT&T Consultant > > FILE > /home/dn/Projects/AuckPUG-2022-04-20/tuple_comprehension_demo.py >>>> > PS I'll be interested to see what 'normal people' do for this... > > - emphasising @Alan's comments: we (old-timers - me, not Alan!) used to > talk about Input-Process-Output, ie in order to produce the output > required, what processing will need to be done, and thus: what input(s) > will be needed. I suggest that this is still a valid approach to > decomposing a problem, and applies particularly well to the (two modes) > way this code is to work - in fact, consider adding a fourth 'phase': > 'Establish the Environment' (all the argparse stuff which will define > which mode the user wishes to operate). Accordingly, the four 'phases' > guide us into a 'first-layer' thinking-about/designing, and then coding > solutions. More importantly (especially given the discussions about > flat-file, CSV, and SQL storage options), if the code is split into > *independent* sections, it becomes easy to change (or experiment with > changes/upgrades) because only the one section will be affected - and > the rest of the code should continue to operate (correctly, untouched, > in perfect splendor...) > > - just because there was talk about options for persistence, I'll > throw-in the suggestion of MongoDB (or similar). NoSQL DBs are a lot > simpler to use than SQL (and I learned SQL before Oracle or SQL/DS (IBM > DB2) were even commercial products, and train in the subject to this > day!). More to the point, their data appears as if it was a Python > data-structure (a list in this case), so no need to bend-your-head > around much that is 'new'! (similar argument applies to JSON, YAML, etc) > > - for SCM (source code management) there are several good tools (I don't > do CI/CD). I use git on my local machines, and for security-copies and > sharing, push and pull to my GitLab or others'. One of the perqs of > joining the local Open Source Society is access to a number of servers > and services. One of which is a GitLab instance. That Microsoft cannot > claim 'ownership' or otherwise use/make use of 'my' code (see legalism, > above) is an important legal, if not, moral component to my decision (or > that imposed upon me). GitLab.com tells us how much better (more mature) > their offering is than GitHub (perhaps 'the other guys' do the same, to > their benefit?), and outlines their $free/$paid service plans. > > - should you move to SCM, then suggest review of the header-comments, > because dates and version-numbers become a function of the SCM rather > than a manual coding-process! > > - recommend researching named-tuples > (https://docs.python.org/3/library/collections.html?highlight=namedtuple#collections.namedtuple) > then the datum can be given a less-generic name, eg > blood_pressure_reading; and the three components similarly identified. > Because each element of the tuple can be given its own name (hence the > "named", hah!) there is no need for packing/unpacking between int(s) and > the collection data-structure! > > - reiterating: 'names' are important. (yes, this is a short and simple > module, but habits are hard to form - and all-to easy to slip) Thus, > generic terms such as "report", "result_string", and "report_list" seem > obvious at the time of coding - but will require more > comprehension-effort (than necessary) at some (six-months in the) future > time-of-reading! With today's text-editors able to pop-up suggestions > for what we might be intending to type, identifier length (and thus > precision-naming) is hardly an issue (cf temptations towards > laziness/saving my worn-down fingers, ...) > > - unlabelled lists of 'anything' decrease readability! Accordingly, when > it comes to function declarations (obvious example), after about two > parameters (IMHO) we should be using keyword-arguments. The problem with > format() (particularly in this example-code) is that it is preceded by a > bunch of empty braces, and these have to be positionally-related to some > code 'further over on the right' (of the code-line). Admittedly in this > case things are not too bad because tuple-unpacking is in-use - so we > don't have to keep track of four pairs of braces (on the left), and four > identifiers (on the right). [it's a lot of effort for aged eyes!] > Recommend researching f-strings > (https://docs.python.org/3/reference/lexical_analysis.html#f-strings). > The process becomes an expression (cf a function) but doesn't take much > learning because the same format() protocol is applied. > > - finally, if you can persuade my at-home BP-machine to 'talk' to my > computer, sign me up! At this time, I would have to copy from my > bed-side scratch-pad into the PC, manually. (am too lazy for that!) > Which does lead to the observation that now() is only applicable if the > input is carried-out immediately after the BP-test - which saves adding > another input field, but there's a human involved... -- Automation Engineer (reuel.net/resume) Scribe: The Domici War (domiciwar.net) General Ne'er-do-well (github.com/LeamHall) From cs at cskk.id.au Sun Jun 26 18:40:14 2022 From: cs at cskk.id.au (Cameron Simpson) Date: Mon, 27 Jun 2022 08:40:14 +1000 Subject: [Tutor] Feedback on coding style In-Reply-To: References: Message-ID: On 26Jun2022 13:10, Alex Kleider wrote: >Thanks for (so promptly!) getting back to me. >I already tried that and was unsuccessful: >(perci) alex at t460:~/Notes/Py$ cd ~/Git/LH/ >(perci) alex at t460:~/Git/LH$ git clone >https://github.com/LeamHall/admin_tools/blob/master/bp_tracker.py >Cloning into 'bp_tracker.py'... >fatal: repository >'https://github.com/LeamHall/admin_tools/blob/master/bp_tracker.py/' >not found >(perci) alex at t460:~/Git/LH$ bp_tracker.py is not a repository, so it cannot be cloned. You can clone https://github.com/LeamHall/admin_tools instead. Or go to https://github.com/LeamHall/admin_tools/blob/master/bp_tracker.py and use the download link on that page. Cheers, Cameron Simpson From PythonList at DancesWithMice.info Sun Jun 26 21:02:11 2022 From: PythonList at DancesWithMice.info (dn) Date: Mon, 27 Jun 2022 13:02:11 +1200 Subject: [Tutor] Feedback on coding style In-Reply-To: References: Message-ID: <9082cb7b-83e2-b6b9-a8ce-ccc39b881352@DancesWithMice.info> On 27/06/2022 12.12, Leam Hall wrote: > No apologies needed! Your input is habitually useful and welcome. > > It has taken me years, literally (I'm a bit slow), to figure out my This is not at all unusual. The phenomenon experienced (largely unknowingly) by most of us, is that we start in a course (which only requires short coding examples, and it is only sometime later that we realise the virtue of some of this 'other stuff', like "style", after we've some miles under our belts and the size of our tasks has increased (markedly), 'complexity' has raised its ugly head... (see also 'structure', 'objects', 'testing', ...) > coding style. While Python is adept in a lot of markets, I'm more of a > tools person than an application programmer. My best work is usually to > write something, make sure it's documented, train the gaining team on > how it works, and then hand it off. In that environment, third-party > libraries, venvs, and external application overhead (like MongoDB) don't > really fit well. Plan text rules, and there's little reason for complex > features. IMHO the opposite applies! The tools I build are used by trainers and educators. There is no such thing as 'one size fits all', and we are constantly innovating. A tool is discarded/replaced before anyone says "it's done"! Accordingly, a need to keep things tidy and 'professional'. Admittedly one reason is that someone else might pick-up the tool and code the extension - so the more 'help' they have, the better our team will perform. (see also earlier wisdom (not mine) about "habits"...) This is also a very good argument for SCM. Whilst it might seem like an "overhead" (and yes, there is bound to be some learning to do), it adds extra layers of flexibility (as well as coping with version-numbering for you). For example, your first version involved a flat-file; then someone else came-along and suggested SQLite. What if you had delivered the first version to some 'client' and (s)he was perfectly happy - perhaps doesn't have SQLite on his/her PC (as if). Now you have two distinct versions to track. Just as you thought it was safe to get back into the water, I come along with MongoDB. Then there were three! SCMs offer features such as "branches" which will help to manage this. Building upon the earlier 'structure' conversation, the SCM will also help when upgrading some 'process' code, and creates a situation where that can be "shipped" to every user, of every 'version' (because the 'process' code has been structured to be "independent" of the storage-method!). This sort of thing requires all-manner of mental-gymnastics to perform manually (or more likely, certainly in my experience of this business, 'we' will keep-quiet and the 'other users' will simply miss-out...). The other handy side-effect is that documentation associated with "branches" will enable you to keep-track of which users/groups use which branch, and when they come to you with a problem/proposal (as they inevitably will), you 'know' (can reproduce) their exact system, ie what they think of as "the system"! I guess it all depends upon one's experience of "hand it off"? Yes, "features" should be countered by YAGNI. However, much of what was discussed has the ultimate aim of making life easier for me, oh wait a minute: it's for you! Something I find helpful, you may not. Fair enough - there's that other 'principle': YMMV! > For the record, I passed one of the MongoDB U Python developer's > classes. Fun stuff, and I've been considering doing more. Sadly, MongoDB > v5 needs a CPU flag that my old Dell doesn't have. So I'm on the > obsolescence track already. Warning: this comment may contain heresy! Who says that we have to use the latest version? (can you hear the collective sucking-in of breath? A little frisson of excitement?) Often we don't use the latest version until 'they' have had time to make sure it works properly, and then, only after we've gone through and checked our various 'versions' of things to make sure that our stuff still works as-advertised. (it's not just 'our code' that should be 'tested'!) An alternative (which may be in the "it's OK for you" category*) is to use a cloud service. The likes of AWS, IBM, Oracle, etc, usually offer a 'free tier' and offer a range of RDBMS or NoSQL options. * such outfits are generally keen to gain our business, and 'can be persuaded' to throw freebies our way. Whilst some see Oracle as 'the death star', etc, etc, I think their free-stuff might be the most generous of the bunch. Again, sadly, some learning to be done first... > Of course, us tools folks are often tasked to write in different > languages. The "#" symbol is a common, if not universal comment > designator, and that's why I use it vice language specific things like > "__author__". Keeping the interpreter and those lines up top really > makes life easy when I have to sort through a few thousand files to > track names and the last time it had major modifications. Once again, an 'advert' for SCM. Such systems offer 'search' facilities, and certainly track mods, dates, etc! Doing such manually over hundreds of projects is too much like hard work for this little boy! As to trying to make one language look like another: nope, not going there - what is the saying about the British and the Americans being "divided by the use of a common language"? My philosophy (such as it is): if you're going to use a tool (language), then use it - don't play with your food! (compare Python with JavaScript and 'comments' translate, now the rest of HTML5 (HTML and CSS)? What about SQL, from earlier discussion 'here'? It ain't goin' to fly, Wilbur!) The OP asked about "style". Is there such a thing as a multi-lingual style? > Some years ago my brother, a better programmer than I, told me about > "input-output-process" and I try to stick to that. In this case, the > input is technically just three items, BP, pulse, and timestamp. Usually > we talk about the two BP numbers as a single unit, and the only reason > they are separate here is to make parsing easier. The pulse is almost > ancillary, and the timestamp is for sorting if I ever get to "last X > number of days", or maybe chart across morning, midday, and evening. > Because the data is stored in a text file, I can edit (vim, thank you. > None of that pretty stuff) the timestamp if it needs changing. > > I'm on the fence about f-strings. Ruby has been doing that for years, > and I usually prefer something that's common-ish between languages. > Using printf or format() is less visually noisy, for me, than f-strings. > f-strings have added a lot of usefulness, like Ruby, but much of what I > do can be done before the print statement. > > Does that make sense? Perfectly! What is right for you, may not be for me - and vice-versa. When introducing organisations/teams/colleagues to new tools (or when it is me being the introduce-ee), my motto is: "nothing succeeds like success". Thus, if I can show you something working - and working well, and you say "hey, there's something in that for me", then you'll adopt/adapt/put-in whatever effort may be necessary. Until then, we're just having a good time talking... -- Regards, =dn From alexkleider at gmail.com Mon Jun 27 00:56:03 2022 From: alexkleider at gmail.com (Alex Kleider) Date: Sun, 26 Jun 2022 21:56:03 -0700 Subject: [Tutor] Feedback on coding style In-Reply-To: References: Message-ID: Thanks, Cameron. Leam already sorted that for me. a On Sun, Jun 26, 2022 at 5:45 PM Cameron Simpson wrote: > > On 26Jun2022 13:10, Alex Kleider wrote: > >Thanks for (so promptly!) getting back to me. > >I already tried that and was unsuccessful: > >(perci) alex at t460:~/Notes/Py$ cd ~/Git/LH/ > >(perci) alex at t460:~/Git/LH$ git clone > >https://github.com/LeamHall/admin_tools/blob/master/bp_tracker.py > >Cloning into 'bp_tracker.py'... > >fatal: repository > >'https://github.com/LeamHall/admin_tools/blob/master/bp_tracker.py/' > >not found > >(perci) alex at t460:~/Git/LH$ > > bp_tracker.py is not a repository, so it cannot be cloned. > You can clone https://github.com/LeamHall/admin_tools instead. > Or go to > https://github.com/LeamHall/admin_tools/blob/master/bp_tracker.py and > use the download link on that page. > > Cheers, > Cameron Simpson > _______________________________________________ > Tutor maillist - Tutor at python.org > To unsubscribe or change subscription options: > https://mail.python.org/mailman/listinfo/tutor -- alex at kleider.ca (sent from my current gizmo) From __peter__ at web.de Mon Jun 27 04:44:24 2022 From: __peter__ at web.de (Peter Otten) Date: Mon, 27 Jun 2022 10:44:24 +0200 Subject: [Tutor] problem solving with lists: final (amateur) solution In-Reply-To: <000f01d888c3$c32e6eb0$498b4c10$@bluewin.ch> References: <000f01d888c3$c32e6eb0$498b4c10$@bluewin.ch> Message-ID: <3fd442a8-0578-5683-044c-4aac65ccccde@web.de> On 25/06/2022 20:45, marcus.luetolf at bluewin.ch wrote: dn gave you a long list of things that are bad style or outright broken in your code, a list that may look rather intimidating. > def day_1_flights(): > key_hist = list(history.keys()) > c_key_hist = key_hist[:] > for dummy_i in c_key_hist: > print('flights_day_1: ', c_key_hist[:num_in_flight]) > for key in c_key_hist[:num_in_flight]: > [history[key].append(player)for player in > c_all_players[0:num_in_flight]] > del c_key_hist[:num_in_flight] > del c_all_players[0:num_in_flight] If you want to fix your script and are looking for a starting point I suggest that you begin with the function above and turn it into a so-called "pure" one. In simple words this means it returns a result that depends only on its arguments, does not print anything and does not modify mutable arguments. Calling the function twice with the same input will produce the same output twice. Pure functions are the easiest to reason about and to test. def day_one_flights(players, num_in_flight): """Create flights for the first day. >>> day_one_flights(["Peter", "Paul", "Mary", "Marcus"], 2) [['Peter', 'Paul'], ['Mary', 'Marcus']] """ ... # your code The paragraph in the docstring that looks like an interactive Python session is a "doctest". You can run it in a terminal window with py -m doctest players_startlist_commented.py -v and get a nice report on whether the expected output of the doctests conforms with the actual, i. e. whether your script is bug free to the extent that you tested it. Now allow me to mention one issue specifically: > for dummy_i in c_key_hist: ... > del c_key_hist[:num_in_flight] Here you modify the list you are iterating over. If you are smart enough to understand what that does you should be too smart to do it ;) Do not ever write such code! From marcus.luetolf at bluewin.ch Mon Jun 27 06:03:36 2022 From: marcus.luetolf at bluewin.ch (marcus.luetolf at bluewin.ch) Date: Mon, 27 Jun 2022 12:03:36 +0200 Subject: [Tutor] problem solving with lists: final (amateur) solution In-Reply-To: <3fd442a8-0578-5683-044c-4aac65ccccde@web.de> References: <000f01d888c3$c32e6eb0$498b4c10$@bluewin.ch> <3fd442a8-0578-5683-044c-4aac65ccccde@web.de> Message-ID: <00f201d88a0d$2b4ffd20$81eff760$@bluewin.ch> Hello Experts, dn and Peter, I am very grateful for you "scrutinizing" my code and I'll take your critiques as incentive for improving it. Nevertheless, It's quite a moutfull. But I'm learning much more about programming and python than by courses over the internet !!! Just one replay to Peter's last issue: I used a copy of the original all_players list because I wanted to keep the original list unaltered. It will take awhile til my next "presentation". Regards and thanks, Marcus. ............................................................................ ............................................................................ ............................................................................ ...... -----Urspr?ngliche Nachricht----- Von: Tutor Im Auftrag von Peter Otten Gesendet: Montag, 27. Juni 2022 10:44 An: tutor at python.org Betreff: Re: [Tutor] problem solving with lists: final (amateur) solution On 25/06/2022 20:45, marcus.luetolf at bluewin.ch wrote: dn gave you a long list of things that are bad style or outright broken in your code, a list that may look rather intimidating. > def day_1_flights(): > key_hist = list(history.keys()) > c_key_hist = key_hist[:] > for dummy_i in c_key_hist: > print('flights_day_1: ', c_key_hist[:num_in_flight]) > for key in c_key_hist[:num_in_flight]: > [history[key].append(player)for player in > c_all_players[0:num_in_flight]] > del c_key_hist[:num_in_flight] > del c_all_players[0:num_in_flight] If you want to fix your script and are looking for a starting point I suggest that you begin with the function above and turn it into a so-called "pure" one. In simple words this means it returns a result that depends only on its arguments, does not print anything and does not modify mutable arguments. Calling the function twice with the same input will produce the same output twice. Pure functions are the easiest to reason about and to test. def day_one_flights(players, num_in_flight): """Create flights for the first day. >>> day_one_flights(["Peter", "Paul", "Mary", "Marcus"], 2) [['Peter', 'Paul'], ['Mary', 'Marcus']] """ ... # your code The paragraph in the docstring that looks like an interactive Python session is a "doctest". You can run it in a terminal window with py -m doctest players_startlist_commented.py -v and get a nice report on whether the expected output of the doctests conforms with the actual, i. e. whether your script is bug free to the extent that you tested it. Now allow me to mention one issue specifically: > for dummy_i in c_key_hist: ... > del c_key_hist[:num_in_flight] Here you modify the list you are iterating over. If you are smart enough to understand what that does you should be too smart to do it ;) Do not ever write such code! _______________________________________________ Tutor maillist - Tutor at python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor From __peter__ at web.de Tue Jun 28 04:22:10 2022 From: __peter__ at web.de (Peter Otten) Date: Tue, 28 Jun 2022 10:22:10 +0200 Subject: [Tutor] problem solving with lists: final (amateur) solution In-Reply-To: <00f201d88a0d$2b4ffd20$81eff760$@bluewin.ch> References: <000f01d888c3$c32e6eb0$498b4c10$@bluewin.ch> <3fd442a8-0578-5683-044c-4aac65ccccde@web.de> <00f201d88a0d$2b4ffd20$81eff760$@bluewin.ch> Message-ID: <758c874e-53f3-0d4f-12a8-b6da0b074c4e@web.de> On 27/06/2022 12:03, marcus.luetolf at bluewin.ch wrote: > Just one replay to Peter's last issue: > I used a copy of the original all_players list because I wanted to keep the > original list unaltered. [untested code below] Technically you can alter a function def chunks(items, chunksize): result = [] while items: # look, ma, no for-loop ;) result.append(items[:chunksize]) del items[:chunksize] return result into one that avoids the side effect of removing the entries from the items argument by making a copy: def chunked(items, chunksize): return chunks(list(items)) But I doubt that any experienced programmer would write such code. You are far more likely to see def chunked(items, chunksize): return [ items[start: start + chunksize] for start in range(0, len(items), chunksize) ] While you may find that a little more complex at first a general strategy that avoids "make a copy, then alter it" in favor of "build a new object from the current one(s)" makes for cleaner, more readable code. The two things that made your script hard to understand for me were (1) name mangling (2) multiple just-in-case copies It was not just me, though. The name mangling is what makes it hard to impossible for you to allow arbitrary player names. The copies increase the number of variable names and the perceived complexity of your code. I can tell that you lost track of what was what, too, as there are places where you have lists that remain identical over the course of your script, and sometimes mistake a string for a list and write players.extend(player) # /seems/ to work for single-character # player names Finally: an excellent means to avoid design choices that make your code hard to maintain is to write tests early on in the design process. If it's hard to test it's a bad idea. That's why I suggested doctest in my previous post. From __peter__ at web.de Tue Jun 28 04:22:10 2022 From: __peter__ at web.de (Peter Otten) Date: Tue, 28 Jun 2022 10:22:10 +0200 Subject: [Tutor] problem solving with lists: final (amateur) solution In-Reply-To: <00f201d88a0d$2b4ffd20$81eff760$@bluewin.ch> References: <000f01d888c3$c32e6eb0$498b4c10$@bluewin.ch> <3fd442a8-0578-5683-044c-4aac65ccccde@web.de> <00f201d88a0d$2b4ffd20$81eff760$@bluewin.ch> Message-ID: <758c874e-53f3-0d4f-12a8-b6da0b074c4e@web.de> On 27/06/2022 12:03, marcus.luetolf at bluewin.ch wrote: > Just one replay to Peter's last issue: > I used a copy of the original all_players list because I wanted to keep the > original list unaltered. [untested code below] Technically you can alter a function def chunks(items, chunksize): result = [] while items: # look, ma, no for-loop ;) result.append(items[:chunksize]) del items[:chunksize] return result into one that avoids the side effect of removing the entries from the items argument by making a copy: def chunked(items, chunksize): return chunks(list(items)) But I doubt that any experienced programmer would write such code. You are far more likely to see def chunked(items, chunksize): return [ items[start: start + chunksize] for start in range(0, len(items), chunksize) ] While you may find that a little more complex at first a general strategy that avoids "make a copy, then alter it" in favor of "build a new object from the current one(s)" makes for cleaner, more readable code. The two things that made your script hard to understand for me were (1) name mangling (2) multiple just-in-case copies It was not just me, though. The name mangling is what makes it hard to impossible for you to allow arbitrary player names. The copies increase the number of variable names and the perceived complexity of your code. I can tell that you lost track of what was what, too, as there are places where you have lists that remain identical over the course of your script, and sometimes mistake a string for a list and write players.extend(player) # /seems/ to work for single-character # player names Finally: an excellent means to avoid design choices that make your code hard to maintain is to write tests early on in the design process. If it's hard to test it's a bad idea. That's why I suggested doctest in my previous post.