解决python读取含BOM码文本问题

Saturday, January 30, 2010

在处理utf8文本格式的时候,由于这些文本是含BOM(byte order
mark)的utf8文本,编译时产生一个Error,"UnicodeEncodeError: 'gbk' codec can't encode
character u'ufeff' in position 0: illegal multibyte sequence"
原来,某些软件,如notepad,在保存一个以UTF-8编码的文件时,会在文件开始的地方插入三个不可见的字符(0xEF 0xBB
0xBF,即BOM)。 因此我们在读取时需要自己去掉这些字符,python中的codecs module定义了这个常量:

import codecs
data = open("Test.txt").read()
if data[:3] == codecs.BOM_UTF8:
data = data[3:]
print data.decode("utf-8")

This entry was tagged Python

comments powered by Disqus

© 2009-2013 lxneng.com. All rights reserved. Powered by Pyramid

go to Top