python - UTF-8 编码文件被 chardetect 选为 ASCII

Question

我正在编写一个组合文件夹中存在的所有文件的单个文件。我希望文本文件是 UTF-8 编码的。我的代码如下

import os
import codecs
import re
def file_concatenation(path):
    with codecs.open('C:/Users/JAYASHREE/Documents/NLP/text-corpus.txt', 'w',encoding='utf8') as outfile:
        for root, dirs, files in os.walk(path):            
                    for dir_name in dirs:    
                        for fname in os.listdir(root+"/"+dir_name):
                            with open(root+"/"+dir_name+"/"+fname) as infile:
                                for line in infile:                                    
                                    new_line = re.sub('[^a-zA-Z]', ' ',line)                                      
                                    outfile.write(re.sub("\s\s+", " ", new_line.lstrip()))
file_concatenation('C:/Users/JAYASHREE/Documents/NLP/bbc-fulltext/bbc')

当我使用 chardetect 查找我的编码时，它显示为 ASCII 有信心 1.0

C:\Users\JAYASHREE>chardetect "C:/Users/JAYASHREE/Documents/NLP/text-corpus.txt"
C:/Users/JAYASHREE/Documents/NLP/text-corpus.txt: ascii with confidence 1.0

请解决问题。谢谢

Answer 1

用于在文件开头encoding='utf-8-sig"'强制使用BOM 。它应该被 chardetect 拾取。