python - UnicodeDecodeError：“ascii”编解码器无法解码位置 1 的字节 0xef

Question

我在尝试将字符串编码为 UTF-8 时遇到了一些问题。我尝试了很多事情，包括使用string.encode('utf-8')and unicode(string)，但我得到了错误：

UnicodeDecodeError：“ascii”编解码器无法解码位置 1 的字节 0xef：序数不在范围内（128）

这是我的字符串：

(｡･ω･｡)ﾉ

我不知道出了什么问题，知道吗？

编辑：问题是按原样打印字符串不能正确显示。另外，当我尝试转换它时出现这个错误：

Python 2.7.1+ (r271:86832, Apr 11 2011, 18:13:53)
[GCC 4.5.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> s = '(\xef\xbd\xa1\xef\xbd\xa5\xcf\x89\xef\xbd\xa5\xef\xbd\xa1)\xef\xbe\x89'
>>> s1 = s.decode('utf-8')
>>> print s1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode characters in position 1-5: ordinal not in range(128)

score 72 · Accepted Answer

这与您的终端编码未设置为 UTF-8 有关。这是我的终端

$ echo $LANG
en_GB.UTF-8
$ python
Python 2.7.3 (default, Apr 20 2012, 22:39:59) 
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> s = '(\xef\xbd\xa1\xef\xbd\xa5\xcf\x89\xef\xbd\xa5\xef\xbd\xa1)\xef\xbe\x89'
>>> s1 = s.decode('utf-8')
>>> print s1
(｡･ω･｡)ﾉ
>>>

在我的终端上，该示例适用于上述示例，但如果我摆脱该LANG设置，它将无法正常工作

$ unset LANG
$ python
Python 2.7.3 (default, Apr 20 2012, 22:39:59) 
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> s = '(\xef\xbd\xa1\xef\xbd\xa5\xcf\x89\xef\xbd\xa5\xef\xbd\xa1)\xef\xbe\x89'
>>> s1 = s.decode('utf-8')
>>> print s1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode characters in position 1-5: ordinal not in range(128)
>>>

请查阅有关您的 linux 变体的文档，以了解如何使此更改永久生效。

score 25 · Accepted Answer

尝试：

string.decode('utf-8')  # or:
unicode(string, 'utf-8')

编辑：

'(\xef\xbd\xa1\xef\xbd\xa5\xcf\x89\xef\xbd\xa5\xef\xbd\xa1)\xef\xbe\x89'.decode('utf-8')给出u'(\uff61\uff65\u03c9\uff65\uff61)\uff89'，这是正确的。

所以你的问题一定出在其他地方，可能如果你试图用它做一些事情，那么有一个隐式转换正在进行（可能是打印，写入流......）

要说更多，我们需要查看一些代码。

score 22 · Accepted Answer

22

于 2012-05-12T11:43:09.373 回答

score 10 · Accepted Answer

If you are working on a remote host, look at /etc/ssh/ssh_config on your local PC.

When this file contains a line:

SendEnv LANG LC_*

comment it out with adding # at the head of line. It might help.

With this line, ssh sends language related environment variables of your PC to the remote host. It causes a lot of problems.

score 10 · Accepted Answer

Try setting the system default encoding as utf-8 at the start of the script, so that all strings are encoded using that.

# coding: utf-8
import sys
reload(sys)
sys.setdefaultencoding('utf-8')

score 5 · Accepted Answer

It's fine to use the below code in the top of your script as Andrei Krasutski suggested.

import sys
reload(sys)
sys.setdefaultencoding('utf-8')

But I will suggest you to also add # -*- coding: utf-8 -* line at very top of the script.

Omitting it throws below error in my case when I try to execute basic.py.

$ python basic.py
  File "01_basic.py", line 14
SyntaxError: Non-ASCII character '\xd9' in file basic.py on line 14, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details

The following is the code present in basic.py which throws above error.

code with error

from pylatex import Document, Section, Subsection, Command, Package
from pylatex.utils import italic, NoEscape

import sys
reload(sys)
sys.setdefaultencoding('utf-8')

def fill_document(doc):
    with doc.create(Section('ِش سثؤفهخى')):
        doc.append('إخع ساخعمي شمصشغس سحثشن فاث فقعفا')
        doc.append(italic('فشمهؤ ؤخىفثىفس شقث شمسخ ىهؤث'))

        with doc.create(Subsection('آثص ٍعلاسثؤفهخى')):
            doc.append('بشةخعس ؤقشئغ ؤاشقشؤفثقس: $&#{}')


if __name__ == '__main__':
    # Basic document
    doc = Document('basic')
    fill_document(doc)

Then I added # -*- coding: utf-8 -*- line at very top and executed. It worked.

code without error

# -*- coding: utf-8 -*-
from pylatex import Document, Section, Subsection, Command, Package
from pylatex.utils import italic, NoEscape

import sys
reload(sys)
sys.setdefaultencoding('utf-8')

def fill_document(doc):
    with doc.create(Section('ِش سثؤفهخى')):
        doc.append('إخع ساخعمي شمصشغس سحثشن فاث فقعفا')
        doc.append(italic('فشمهؤ ؤخىفثىفس شقث شمسخ ىهؤث'))

        with doc.create(Subsection('آثص ٍعلاسثؤفهخى')):
            doc.append('بشةخعس ؤقشئغ ؤاشقشؤفثقس: $&#{}')


if __name__ == '__main__':
    # Basic document
    doc = Document('basic')
    fill_document(doc)

Thanks.

score 4 · Accepted Answer

No problems with my terminal. The above answers helped me looking in the right directions but it didn't work for me until I added 'ignore':

fix_encoding = lambda s: s.decode('utf8', 'ignore')

As indicated in the comment below, this may lead to undesired results. OTOH it also may just do the trick well enough to get things working and you don't care about losing some characters.

score 2 · Accepted Answer

2

this works for ubuntu 15.10:

sudo locale-gen "en_US.UTF-8"
sudo dpkg-reconfigure locales

于 2016-07-11T13:07:00.753 回答

score 1 · Accepted Answer

看起来您的字符串被编码为utf-8，那么到底是什么问题？或者你想在这里做什么..？

Python 2.7.3 (default, Apr 20 2012, 22:39:59) 
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> s = '(\xef\xbd\xa1\xef\xbd\xa5\xcf\x89\xef\xbd\xa5\xef\xbd\xa1)\xef\xbe\x89'
>>> s1 = s.decode('utf-8')
>>> print s1
(｡･ω･｡)ﾉ
>>> s2 = u'(｡･ω･｡)ﾉ'
>>> s2 == s1
True
>>> s2
u'(\uff61\uff65\u03c9\uff65\uff61)\uff89'

score 1 · Accepted Answer

In my case, it was caused by my Unicode file being saved with a "BOM". To solve this, I cracked open the file using BBEdit and did a "Save as..." choosing for encoding "Unicode (UTF-8)" and not what it came with which was "Unicode (UTF-8, with BOM)"

score 0 · Accepted Answer

I was getting the same type of error, and I found that the console is not capable of displaying the string in another language. Hence I made the below code changes to set default_charset as UTF-8.

data_head = [('\x81\xa1\x8fo\x89\xef\x82\xa2\x95\xdb\x8f\xd8\x90\xa7\x93x\x81\xcb3\x8c\x8e\x8cp\x91\xb1\x92\x86(\x81\x86\x81\xde\x81\x85)\x81\xa1\x8f\x89\x89\xf1\x88\xc8\x8aO\x81A\x82\xa8\x8b\xe0\x82\xcc\x90S\x94z\x82\xcd\x88\xea\x90\xd8\x95s\x97v\x81\xa1\x83}\x83b\x83v\x82\xcc\x82\xa8\x8e\x8e\x82\xb5\x95\xdb\x8c\xaf\x82\xc5\x8fo\x89\xef\x82\xa2\x8am\x92\xe8\x81\xa1', 'shift_jis')]
default_charset = 'UTF-8' #can also try 'ascii' or other unicode type
print ''.join([ unicode(lin[0], lin[1] or default_charset) for lin in data_head ])

score -1 · Accepted Answer

This is the best answer: https://stackoverflow.com/a/4027726/2159089

in linux:

export PYTHONIOENCODING=utf-8

so sys.stdout.encoding is OK.

score -1 · Accepted Answer

BOM, it's so often BOM for me

vi the file, use

:set nobomb

and save it. That nearly always fixes it in my case

score -1 · Accepted Answer

I had the same error, with URLs containing non-ascii chars (bytes with values > 128)

url = url.decode('utf8').encode('utf-8')

Worked for me, in Python 2.7, I suppose this assignment changed 'something' in the str internal representation--i.e., it forces the right decoding of the backed byte sequence in url and finally puts the string into a utf-8 str with all the magic in the right place. Unicode in Python is black magic for me. Hope useful

score -2 · Accepted Answer

i solve that problem changing in the file settings.py with 'ENGINE': 'django.db.backends.mysql', don´t use 'ENGINE': 'mysql.connector.django',

score -2 · Accepted Answer

-2

Just convert the text explicitly to string using str(). Worked for me.

于 2016-02-05T08:22:33.497 回答

python - UnicodeDecodeError：“ascii”编解码器无法解码位置 1 的字节 0xef

16 回答 16

code with error

code without error

Related

Reference