Lines Matching full:tokenizer
890 class Tokenizer(object):891 """A simple string tokenizer that chops expressions into variables,1047 tokens = Tokenizer(expr).Tokenize()
