Documentation incomplete about the order of testing rules.

[mrolle45]

In doc/ply.md, it states

When building the master regular expression, rules are added in the following order:

1. All tokens defined by functions are added in the same order as they appear in the lexer file.
2. Tokens defined by strings are added next by sorting them in order of decreasing regular expression length (longer expressions are added first).

What is missing from this is what happens if (1) several function rules are defined in the same place, or (2) several string rules have same regex length.

The answer depends on whether the rules are local variables in a function that calls lex.lex(), or else attributes of an external object specified by lex.lex(module=obj). In the first case, rules are taken in the same order that they were introduced into the caller's local namespace. In the second case, rules are taken in alphabetical order, as returned by the library function dir(obj).
For string rules, the program should not depend on which regex is longer, except if the longer regex starts with the shorter regex. Otherwise, if it is possible for multiple regexes to match the input text, functions should be used instead of strings, to guarantee the order of matching.
For function rules, correct lexing may depend on the order in which the rules are applied. Usually, each rule is defined separately, and the rules are applied in order of their line number in the source file. However, a rule might be created within a method which is called multiple times, and so these rules all have the same code line number. This can save a lot of writing similar def statements.

For example, in my own code I have a method which is the equivalent of a string rule, but creates a function rule in order for it to be applied before all string rules.

class MyLexer:
    def t_A(t) :"++"; return t
    def makerule(self, token, regex):
        def rule(t: LexToken) -> LexToken:
            return t
        rule.regex = regex
        setattr(self,  't_' + token, rule)
    def t_B(t) :"+"; return t
    def build(self):
        return lex.lex(module=self)
    self.tokens = 'A B X Y'.split()
    self.makerule('Y', '==')
    self.makerule('X', '=')

lexer = MyLexer().build()

This makes a lexer with t_X and t_Y rules. However, since these rules are functions, they are applied in alphabetical order. Both rules use the function rule() within MyLexer.makerule(), hence the same line number.
This is not what is wanted here. We want the lexer to place t_Y before t_X, which is their order in self.__dict__. The complete order of rules is t_A, t_X, t_Y, t_B.
In order to place t_Y before t_X, we could rewrite MyLexer.build() so that the rules become local variables assigned in the desired order. Then lex.lex(), without the module=self keyword, will do the job:

    def build(self) -> lex.Lexer:
        """ Makes the lex.Lexer for given variables, preserving order of rules. """
        for key, value in type(self).__dict__.items():
            exec(f"{key} = value")
        for key, value in self.__dict__.items():
            exec(f"{key} = value")
        return lex.lex()

The order of rules is t_A, t_Y, t_X, t_B.

The important point to be made is this: If you are using an external module= argument, and your module uses nested functions for some of its rules, then you need copy the class and instance variables into another function, in their dictionary order, and then make the lexer from the local context.