4

给定一个混合编码的损坏文件（例如 utf-8 和 latin-1），我如何配置 Emacs 以在保存文件时将其所有符号“投影”到单一编码（例如 utf-8）？

我做了以下功能来自动化一些清洁，但我想我可以在某处找到将一个编码中的符号“é”映射到 utf-8 中的“é”的信息，以改进此功能（或有人已经写过这样的功能）。

  (defun jyby/cleanToUTF ()
    "Cleaning to UTF"
    (interactive)
    (progn
         (save-excursion (replace-regexp "अ" ""))
         (save-excursion (replace-regexp "आ" ""))
         (save-excursion (replace-regexp "ॆ" ""))
       )
  )

  (global-unset-key [f11])
  (global-set-key [f11] 'jyby/cleanToUTF)

我有许多混合编码“损坏”的文件（由于从具有错误字体配置的浏览器复制粘贴），产生以下错误。有时我会手动清理它们，方法是用“”或适当的字符搜索和替换每个有问题的符号，或者更快地将“utf-8-unix”指定为编码（下次编辑和保存时会提示相同的消息文件）。这已成为一个问题，因为在任何此类损坏的文件中，任何重音字符都会被每次保存时大小翻倍的序列替换，最终文件大小翻倍。我正在使用 GNU Emacs 24.2.1

These default coding systems were tried to encode text
in the buffer `test_accents.org':
(utf-8-unix (30 . 4194182) (33 . 4194182) (34 . 4194182) (37
. 4194182) (40 . 4194181) (41 . 4194182) (42 . 4194182) (45
. 4194182) (48 . 4194182) (49 . 4194182) (52 . 4194182))
However, each of them encountered characters it couldn't encode:
utf-8-unix cannot encode these:           ...

Click on a character (or switch to this window by `C-x o'
and select the characters by RET) to jump to the place it appears,
where `C-u C-x =' will give information about it.

Select one of the safe coding systems listed below,
or cancel the writing with C-g and edit the buffer
to remove or modify the problematic characters,
or specify any other coding system (and risk losing
the problematic characters).

raw-text emacs-mule no-conversion
4

2 回答 2

2

我在 emacs 中为此苦苦挣扎了很多次。当我有一个文件被弄乱时，例如在 raw-text-unix 模式下，并保存为 utf-8，emacs 甚至会抱怨已经是干净的 utf-8 的文本。我还没有找到一种方法让它只抱怨非 utf-8。

我刚刚找到了一种使用重新编码的合理的半自动化方法：

f=mixed-file
recode -f ..utf-8 $f > /tmp/recode.out
diff $f recode.out | cat -vt

# manually fix lines of text that can't be converted to utf-8 in $f,
# and re-run recode and diff until the output diff is empty.

一路上一个有用的工具是http://www.ltg.ed.ac.uk/~richard/utf-8.cgi?input=342+200+224&mode=obytes

然后我只是在emacs中重新打开文件，它被识别为干净的unicode。

于 2014-10-02T17:55:16.473 回答
1

这里有一些东西可以让你开始：

(put 'eof-error 'error-conditions '(error eof-error))
(put 'eof-error 'error-message "End of stream")
(put 'bad-byte 'error-conditions '(error bad-byte))
(put 'bad-byte 'error-message "Not a UTF-8 byte")

(defclass stream ()
  ((bytes :initarg :bytes :accessor bytes-of)
   (position :initform 0 :accessor position-of)))

(defun logbitp (byte bit) (not (zerop (logand byte (ash 1 bit)))))

(defmethod read-byte ((this stream) &optional eof-error eof)
  (with-slots (bytes position) this
    (if (< position (length bytes))
        (prog1 (aref bytes position) (incf position))
      (if eof-error (signal eof-error (list position)) eof))))

(defmethod unread-byte ((this stream))
  (when (> (position-of this) 0) (decf (position-of this))))

(defun read-utf8-char (stream)
  (let ((byte (read-byte stream 'eof-error)))
    (if (not (logbitp byte 7)) byte
      (let ((numbytes
             (cond
              ((not (logbitp byte 5))
               (setf byte (logand #2r11111 byte)) 1)
              ((not (logbitp byte 4))
               (setf byte (logand #2r1111 byte)) 2)
              ((not (logbitp byte 3))
               (setf byte (logand #2r111 byte)) 3))))
        (dotimes (b numbytes byte)
          (let ((next-byte (read-byte stream 'eof-error)))
            (if (and (logbitp next-byte 7) (not (logbitp next-byte 6)))
                (setf byte (logior (ash byte 6) (logand next-byte #2r111111)))
              (signal 'bad-byte (list next-byte)))))
        (signal 'bad-byte (list byte))))))

(defun load-corrupt-file (file)
  (interactive "fFile to load: ")
  (with-temp-buffer
    (set-buffer-multibyte nil)
    (insert-file-literally file)
    (with-output-to-string
      (set-buffer-multibyte t)
      (loop with stream = (make-instance 'stream :bytes (buffer-string))
            for next-char =
            (condition-case err
                (read-utf8-char stream)
              (bad-byte (message "Fix this byte %d" (cdr err)))
              (eof-error nil))
            while next-char
            do (write-char next-char)))))

这段代码的作用 - 它加载一个没有转换的文件并尝试读取它，就好像它是使用 UTF-8 编码的一样，一旦它遇到一个看起来不属于 UTF-8 的字节，它就会出错，并且你需要以某种方式处理它，它是"Fix this byte"消息所在的位置）。但是你需要对如何修复它有创造性......

于 2013-04-13T14:40:25.033 回答