From vinay_sajip at yahoo.co.uk Tue Dec 13 23:56:37 2011 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 13 Dec 2011 22:56:37 +0000 (UTC) Subject: [Python-porting] lib2to3 and matching of fixers Message-ID: I'm trying to write a custom lib2to3 fixer which, when processing the source x = b'foo' b'bar' will match the right hand side of the expression. From what I can see I should be able to match it with 'atom' or 'power', but it doesn't seem to match either of those. The individual byte-string tokens are matched using STRING, as expected. What should I use to match the right-hand side, for example if I want to do constant folding and transform to something like b'foobar'? Regards, Vinay Sajip From benjamin at python.org Wed Dec 14 02:30:16 2011 From: benjamin at python.org (Benjamin Peterson) Date: Tue, 13 Dec 2011 20:30:16 -0500 Subject: [Python-porting] lib2to3 and matching of fixers In-Reply-To: References: Message-ID: 2011/12/13 Vinay Sajip : > I'm trying to write a custom lib2to3 fixer which, when processing the source > > x = b'foo' b'bar' > > will match the right hand side of the expression. From what I can see I should > be able to match it with 'atom' or 'power', but it doesn't seem to match either > of those. The individual byte-string tokens are matched using STRING, as > expected. What are you using exactly? I would expect "atom< STRING+>" to work. > > What should I use to match the right-hand side, for example if I want to do > constant folding and transform to something like b'foobar'? -- Regards, Benjamin From vinay_sajip at yahoo.co.uk Wed Dec 14 12:43:10 2011 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 14 Dec 2011 11:43:10 +0000 (GMT) Subject: [Python-porting] Fw: lib2to3 and matching of fixers In-Reply-To: <1323854603.84819.YahooMailNeo@web25802.mail.ukl.yahoo.com> References: <1323854603.84819.YahooMailNeo@web25802.mail.ukl.yahoo.com> Message-ID: <1323862990.653.YahooMailNeo@web25807.mail.ukl.yahoo.com> >> What are you using exactly? I would expect "atom< STRING+>" to >> work. > >Thanks, that worked. I tried "atom" and "STRING+" separately - sorry, I'm unfamiliar with the pattern-matching > language, but hopefully I'll get more familiar as I work with it! I'm now trying to match a tuple using the following pattern: PATTERN = "atom< '(' args=testlist_gexp< any+ > ')' >" This is a subset of the pattern used in the isinstance fixer that Armin Ronacher wrote. However, the pattern fails to match the source x = (int, long) The parse tree for this source looks like this: Node(file_input, [Node(simple_stmt, [Node(expr_stmt, [Leaf(1, 'x'), Leaf(22, '='), Node(atom, [Leaf(7, '('), Node(testlist_gexp, [Leaf(1, 'int'), Leaf(12, ','), Leaf(1, 'long')]), Leaf(8, ')')])]), Leaf(4, '\n')]), Leaf(0, '')]) Since it contains an "atom" node with children consisting of a parenthesised "testlist_gexp" node, I'm not sure why there is no match. What am I missing? Thanks and regards, Vinay Sajip From benjamin at python.org Wed Dec 14 18:35:25 2011 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 14 Dec 2011 12:35:25 -0500 Subject: [Python-porting] Fw: lib2to3 and matching of fixers In-Reply-To: <1323862990.653.YahooMailNeo@web25807.mail.ukl.yahoo.com> References: <1323854603.84819.YahooMailNeo@web25802.mail.ukl.yahoo.com> <1323862990.653.YahooMailNeo@web25807.mail.ukl.yahoo.com> Message-ID: 2011/12/14 Vinay Sajip : >>> What are you using exactly? I would expect "atom< STRING+>" to > >>> work. >> >>Thanks, that worked. I tried "atom" and "STRING+" separately - sorry, I'm unfamiliar with the pattern-matching >> language, but hopefully I'll get more familiar as I work with it! > > > I'm now trying to match a tuple using the following pattern: > > PATTERN = "atom< '(' args=testlist_gexp< any+ > ')' >" > > > This is a subset of the pattern used in the isinstance fixer that Armin Ronacher wrote. However, the pattern fails to match the source > > x = (int, long) > > > The parse tree for this source looks like this: > > Node(file_input, [Node(simple_stmt, [Node(expr_stmt, [Leaf(1, 'x'), Leaf(22, '='), Node(atom, [Leaf(7, '('), Node(testlist_gexp, [Leaf(1, 'int'), Leaf(12, ','), Leaf(1, 'long')]), Leaf(8, ')')])]), Leaf(4, '\n')]), Leaf(0, '')]) You have to account for the newline. -- Regards, Benjamin From vinay_sajip at yahoo.co.uk Wed Dec 14 21:38:35 2011 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 14 Dec 2011 20:38:35 +0000 (UTC) Subject: [Python-porting] Fw: lib2to3 and matching of fixers References: <1323854603.84819.YahooMailNeo@web25802.mail.ukl.yahoo.com> <1323862990.653.YahooMailNeo@web25807.mail.ukl.yahoo.com> Message-ID: Benjamin Peterson writes: > > You have to account for the newline. > Thanks for the response, but it's not clear how or why. As to the "why": the portion of the tree I get by looking at tree.children[0].children[0].children[2] is Node(atom, [ Leaf(7, '('), Node(testlist_gexp, [ Leaf(1, 'int'), Leaf(12, ','), Leaf(1, 'long') ]), Leaf(8, ')') ]) which is the bit I expect to match. It does not contain the newline (that's present as the last child the enclosing grandparent node (simple_stmt) obtained by looking at tree.children[0]: Node(simple_stmt, [ Node(expr_stmt, [ Leaf(1, 'x'), Leaf(22, '='), Node(atom, [ Leaf(7, '('), Node(testlist_gexp, [ Leaf(1, 'int'), Leaf(12, ','), Leaf(1, 'long') ]), Leaf(8, ')') ]) ]), Leaf(4, '\n') ]) The pattern I used appears consistent with the subset of the pattern used in the is_instance fixer to match a tuple in the isinstance call, other than whitespace in the pattern string itself: power< 'isinstance' trailer< '(' arglist< any ',' atom< '(' args=testlist_gexp< any+ > ')' > > ')' > > in the is_instance fixer, as against my fixer's atom< '(' args=testlist_gexp< any+ > ')' > So I can't see why the newline needs to be accounted for, nor how to do it. Arranging the whitespace in my pattern to exactly match that segment in the is_instance fixer's pattern made no difference, as I would have expected. So, how do I account for the newline, and why do I need to when it's two levels up in the tree? Thanks for your patience with lib2to3 newbie. Regards, Vinay Sajip From benjamin at python.org Wed Dec 14 23:18:19 2011 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 14 Dec 2011 17:18:19 -0500 Subject: [Python-porting] Fw: lib2to3 and matching of fixers In-Reply-To: References: <1323854603.84819.YahooMailNeo@web25802.mail.ukl.yahoo.com> <1323862990.653.YahooMailNeo@web25807.mail.ukl.yahoo.com> Message-ID: 2011/12/14 Vinay Sajip : > Benjamin Peterson writes: > >> >> You have to account for the newline. >> > > Thanks for the response, but it's not clear how or why. As to the "why": the > portion of the tree I get by looking at Sorry. I was trying to read the parse tree from the linear version in your last mail and failed. You are correct. Does your fixer us BM_compatible = True? Try setting that to false. -- Regards, Benjamin From vinay_sajip at yahoo.co.uk Thu Dec 15 00:01:52 2011 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 14 Dec 2011 23:01:52 +0000 (GMT) Subject: [Python-porting] Fw: lib2to3 and matching of fixers In-Reply-To: References: <1323854603.84819.YahooMailNeo@web25802.mail.ukl.yahoo.com> <1323862990.653.YahooMailNeo@web25807.mail.ukl.yahoo.com> Message-ID: <1323903712.50651.YahooMailNeo@web25801.mail.ukl.yahoo.com> > > Does your fixer us BM_compatible = True? Try setting that to false. > With BM_compatible = False, the match is indeed made, and my fixer's transform method called. Thanks! Is the meaning of the BM_compatible flag, or the overall matching and transformation algorithm, documented somewhere? Most of the fixers I looked at had BM_compatible = True, and I took my cue from the fix_isinstance fixer as it appeared closest to what I was trying to do. Regards, Vinay Sajip From benjamin at python.org Thu Dec 15 00:05:31 2011 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 14 Dec 2011 18:05:31 -0500 Subject: [Python-porting] Fw: lib2to3 and matching of fixers In-Reply-To: <1323903712.50651.YahooMailNeo@web25801.mail.ukl.yahoo.com> References: <1323854603.84819.YahooMailNeo@web25802.mail.ukl.yahoo.com> <1323862990.653.YahooMailNeo@web25807.mail.ukl.yahoo.com> <1323903712.50651.YahooMailNeo@web25801.mail.ukl.yahoo.com> Message-ID: 2011/12/14 Vinay Sajip : >> > >> Does your fixer us BM_compatible = True? Try setting that to false. >> > > With BM_compatible = False, the match is indeed made, and my fixer's transform method called. Thanks! > > Is the meaning of the BM_compatible flag, or the overall matching and transformation algorithm, documented somewhere? Most of the fixers I looked at had BM_compatible = True, and I took my cue from the fix_isinstance fixer as it appeared closest to what I was trying to do. No, unfortunately. The BM matcher is faster, but has weird behavior on some things (as you can see). -- Regards, Benjamin From vinay_sajip at yahoo.co.uk Wed Dec 14 10:23:23 2011 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 14 Dec 2011 09:23:23 +0000 (GMT) Subject: [Python-porting] lib2to3 and matching of fixers In-Reply-To: References: Message-ID: <1323854603.84819.YahooMailNeo@web25802.mail.ukl.yahoo.com> > What are you using exactly? I would expect "atom< STRING+>" to > work. Thanks, that worked. I tried "atom" and "STRING+" separately - sorry, I'm unfamiliar with the pattern-matching language, but hopefully I'll get more familiar as I work with it! Regards, Vinay Sajip -------------- next part -------------- An HTML attachment was scrubbed... URL: