UnicodeDecodeError Python/Django应用程序

Question

我收到此错误

Unicode解码器位于/select_text'utf-8'编解码器无法解码位置92的字节0xe7：无效的延续字节请求方法：POST请求URL：http://agata.pgie.ufrgs.br/select_text Django版本：2.0.1异常类型：Unicode解码错误异常值：'utf-8'编解码器无法解码位置92的字节0xe7：无效的延续字节异常位置：/home/metis/public_html/AGATA/agataenv/lib/python3.4/编解码器.py解码，第319行Python 可执行文件： /usr/bin/python3 Python 版本： 3.4.3 Python Path： [/home/metis/public_html/AGATA'， “/home/metis/public_html/AGATA/agataenv/lib/python3.4”， “/home/metis/public_html/AGATA/agataenv/lib/lib/python3.4/plat-x86_64-linux-gnu'， '/home/metis/public_html/AGATA/agataenv/lib/python3.4/lib-dynload'， '/usr/lib/python3.4'， '/home/metis/python3.4/plat-x86_64-linux-gnu'， '/home/metis/public_html/AGATA/agataenv/lib/python3.4/site-packages'] Server Time： 星期四， 22 Feb 2018 12：29：51 0000 Unicode 错误提示 无法编码/解码的字符串是： Varia es nvironment：

Request Method: POST
Request URL: http://agata.pgie.ufrgs.br/select_text

Django Version: 2.0.1
Python Version: 3.4.3
Installed Applications:
['django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'textMining',
 'bootstrapform']
Installed Middleware:
['django.middleware.security.SecurityMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware']



Traceback:

File "/home/metis/public_html/AGATA/agataenv/lib/python3.4/site-packages/django/core/handlers/exception.py" in inner
  35.             response = get_response(request)

File "/home/metis/public_html/AGATA/agataenv/lib/python3.4/site-packages/django/core/handlers/base.py" in _get_response
  128.                 response = self.process_exception_by_middleware(e, request)

File "/home/metis/public_html/AGATA/agataenv/lib/python3.4/site-packages/django/core/handlers/base.py" in _get_response
  126.                 response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/home/metis/public_html/AGATA/textMining/views.py" in select_text
  59.     text_mining = TextMining(file_path, keywords)

File "/home/metis/public_html/AGATA/textMining/TextMining.py" in __init__
  15.         self.separete_file_sentences()

File "/home/metis/public_html/AGATA/textMining/TextMining.py" in separete_file_sentences
  31.             file_text = text_file.read().decode('string-escape').decode("utf-8")

File "/home/metis/public_html/AGATA/agataenv/lib/python3.4/codecs.py" in decode
  319.         (result, consumed) = self._buffer_decode(data, self.errors, final)

Exception Type: UnicodeDecodeError at /select_text
Exception Value: 'utf-8' codec can't decode byte 0xe7 in position 92: invalid continuation byte

在我的Django应用程序上，已经在Apache..上，无法弄清楚这里的问题是什么，因为我正在处理编码（至少我认为是这样..）

我的代码（按照顺序）：

def select_text(request):

    book_file = request.FILES['book']
    fs = FileSystemStorage()
    file_name = fs.save(book_file.name, book_file)
    uploaded_file_url = fs.url(file_name)
    print(uploaded_file_url)

    keywords = [
        request.POST['keyword_1'],
        request.POST['keyword_2'],
        request.POST['keyword_3'],
    ]

    blank_optional_keywords = {
        'keyword_2' : False,
        'keyword_3' : False
    }

    if keywords[1] == "":
        blank_optional_keywords['keyword_2'] = True
    if keywords[2] == "":
        blank_optional_keywords['keyword_3'] = True

    request.session["blank_optional_keywords"] = blank_optional_keywords

    #file_name = "LivroMA4_P1_formatado(1).txt"

    #file_path = get_file_path(file_name, 'text')

    file_path = get_file_path(uploaded_file_url, 'upload')

    text_mining = TextMining(file_path, keywords)
    text_mining.get_keywords_sentences()

    sentences = text_mining._keyword_sentences

    sentences_info = generate_sentences_info(sentences)

    request.session["sentences_info"] = sentences_info

    return render(request, 'textMining/select_text.html', {'sentences_info': sentences_info})

TextMining类函数:

class TextMining(object):
    def __init__(self, file_path, keywords):
        self._file_path = file_path
        self._keywords = keywords
        self._sentences = list()
        self._keyword_sentences = dict()

        self.lower_keywords()
        self.separete_file_sentences()
...
    def separete_file_sentences(self):
        with open(self._file_path, "r", encoding='utf-8') as text_file:
            file_text = text_file.read()
            sentences = nltk.tokenize.sent_tokenize(file_text)

            for i in range(len(sentences)):
                if(len(sentences[i]) > 0):
                    self._sentences.append(sentences[i])

我已经处理这个几天了，尝试了很多东西，但都不起作用…

urls.py（文本挖掘应用程序）

urlpatterns = [
        url(r'^$', views.index, name='index'),
        url(r'^select_text', views.select_text, name = 'select_text'),
        url(r'^edit_text', views.edit_text, name = 'edit_text'),
        url(r'^generate_aiml', views.generate_aiml, name = 'generate_aiml'),
]

urls.py（文本挖掘项目）

urlpatterns = [
    url(r'^admin/', admin.site.urls),
    url(r'^', include('textMining.urls')),
] + static(settings.STATIC_URL, document_root=settings.STATIC_ROOT)

if settings.DEBUG is True:
    urlpatterns += static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)

Answer 1

我遇到了同样的问题，奇怪的是，问题的根源是数据库，但Django说问题出在122位置的模板中，我一直从模板中删除行，但Django将错误移动到下一行！我在数据库中发现有一个UTF8字段，类型为TINYTEXT

`md` tinytext COLLATE utf8_unicode_ci DEFAULT NULL COMMENT 'metadata', 

我正在从另一个表中复制数据，并带有“插入到表中选择...”语句，源表字段类型为 TEXT，当数据被复制时，一些长文本在 UTF8 字符的中间被截断，这让 Django 感到困惑。解决方案是将字段的数据类型更改为 TEXT，然后再次复制数据。

`md` text COLLATE utf8_unicode_ci DEFAULT NULL COMMENT 'metadata',