Switchable Tokenizers

[PeterNerlich]

Allow Grammars to specify which Tokenizer to use and automatically adapt import statement

The parser generator from pegen experiments assumes all grammars work with the same tokens. To allow for more general languages, the grammar should be able to specify which token definition it depends on.

Research is needed in how far this is dealt with usually.