Lines Matching refs:ENCODING
5 determining source file encoding.
19 operators. Additionally, all token lists start with an ENCODING token
20 which tells you which encoding was used to decode the bytes stream.
42 "NL", "untokenize", "ENCODING", "TokenInfo"]
49 ENCODING = N_TOKENS + 2
50 tok_name[ENCODING] = 'ENCODING'
228 self.encoding = None
252 if tok_type == ENCODING:
253 self.encoding = token
288 if toknum == ENCODING:
289 self.encoding = tokval
319 It returns a bytes object, encoded using the ENCODING
339 if ut.encoding is not None:
340 out = out.encode(ut.encoding)
357 The detect_encoding() function is used to detect the encoding that should
361 It will call readline a maximum of twice, and return the encoding used
364 It detects the encoding from the presence of a utf-8 bom or an encoding
366 but disagree, a SyntaxError will be raised. If the encoding cookie is an
370 If no encoding is specified, then the default of 'utf-8' will be returned.
377 encoding = None
387 # Decode as UTF-8. Either the line is an encoding declaration,
389 # per default encoding.
392 msg = "invalid or missing encoding declaration"
400 encoding = _get_normal_name(match.group(1))
402 codec = lookup(encoding)
406 msg = "unknown encoding: " + encoding
408 msg = "unknown encoding for {!r}: {}".format(filename,
409 encoding)
413 if encoding != 'utf-8':
416 msg = 'encoding problem: utf-8'
418 msg = 'encoding problem for {!r}: utf-8'.format(filename)
420 encoding += '-sig'
421 return encoding
431 encoding = find_cookie(first)
432 if encoding:
433 return encoding, [first]
441 encoding = find_cookie(second)
442 if encoding:
443 return encoding, [first, second]
449 """Open a file in read only mode using the encoding detected by
454 encoding, lines = detect_encoding(buffer.readline)
456 text = TextIOWrapper(buffer, encoding, line_buffering=True)
480 The first token sequence will always be an ENCODING token
481 which tells you which encoding was used to decode the bytes stream.
486 encoding, consumed = detect_encoding(readline)
489 return _tokenize(chain(consumed, rl_gen, empty).__next__, encoding)
492 def _tokenize(readline, encoding):
505 if encoding is not None:
506 if encoding == "utf-8-sig":
508 encoding = "utf-8"
509 yield TokenInfo(ENCODING, encoding, (0, 0), (0, 0), '')
516 if encoding is not None:
517 line = line.decode(encoding)
