7

我总是处理阿拉伯文本文件，为了避免编码问题，我根据 Buckwalter 的方案将阿拉伯字符音译成英文 (http://www.qamus.org/transliteration.htm)

这是我的代码，但即使是 400 kb 这样的小文件，它也非常慢。让它更快的想法？

谢谢

     def transliterate(file):
          data = open(file).read()
          buckArab = {"'":"ء", "|":"آ", "?":"أ", "&":"ؤ", "<":"إ", "}":"ئ", "A":"ا", "b":"ب", "p":"ة", "t":"ت", "v":"ث", "g":"ج", "H":"ح", "x":"خ", "d":"د", "*":"ذ", "r":"ر", "z":"ز", "s":"س", "$":"ش", "S":"ص", "D":"ض", "T":"ط", "Z":"ظ", "E":"ع", "G":"غ", "_":"ـ", "f":"ف", "q":"ق", "k":"ك", "l":"ل", "m":"م", "n":"ن", "h":"ه", "w":"و", "Y":"ى", "y":"ي", "F":"ً", "N":"ٌ", "K":"ٍ", "~":"ّ", "o":"ْ", "u":"ُ", "a":"َ", "i":"ِ"}    
          for char in data: 
               for k, v in arabBuck.iteritems():
                     data = data.replace(k,v)                 
      return data
4

5 回答 5

8

编辑 2021 年 10 月

最近发布的一个 python 包可以做到这一点（还有更多），所以现在阅读这篇文章的任何人都应该忽略所有其他答案，只使用 Camel 工具。（Nizar Habash 和他在纽约大学阿布扎比分校的团队非常棒地开发了这个并让它变得如此易于访问！）

::python
from camel_tools.utils.charmap import CharMapper
sentence = "ذهبت إلى المكتبة."
print(sentence)

ar2bw = CharMapper.builtin_mapper('ar2bw')

sent_bw = ar2bw(sentence)
print(sent_bw)

输出：

هبت إلى المكتبة.
*hbt <lY Almktbp.

您可以在此处找到安装说明和教程：https ://github.com/CAMeL-Lab/camel_tools

旧答案 顺便说一句，有人已经编写了一个执行此操作的脚本，因此您可能需要在自己花费太多时间之前检查一下： buckwalter2unicode.py

它可能比您需要的更多，但您不必全部使用它：我只复制了两个字典和 transliterateString 函数（我认为进行了一些调整），并在我的网站上使用它。

编辑： 上面的脚本是我一直在使用的，但我刚刚发现它比使用替换慢得多，尤其是对于大型语料库。这是我最终得到的代码，它似乎更简单更快（这引用了字典 buck2uni）：

def transString(string, reverse=0):
    '''Given a Unicode string, transliterate into Buckwalter. To go from
    Buckwalter back to Unicode, set reverse=1'''

    for k, v in buck2uni.items():
        if not reverse:
            string = string.replace(v, k)
        else:
            string = string.replace(k, v)

    return string
于 2012-10-16T19:03:32.537 回答
5

每当您必须进行音译str.translate时，使用的方法是：

>>> import timeit
>>> buckArab = {"'":"ء", "|":"آ", "?":"أ", "&":"ؤ", "<":"إ", "}":"ئ", "A":"ا", "b":"ب", "p":"ة", "t":"ت", "v":"ث", "g":"ج", "H":"ح", "x":"خ", "d":"د", "*":"ذ", "r":"ر", "z":"ز", "s":"س", "$":"ش", "S":"ص", "D":"ض", "T":"ط", "Z":"ظ", "E":"ع", "G":"غ", "_":"ـ", "f":"ف", "q":"ق", "k":"ك", "l":"ل", "m":"م", "n":"ن", "h":"ه", "w":"و", "Y":"ى", "y":"ي", "F":"ً", "N":"ٌ", "K":"ٍ", "~":"ّ", "o":"ْ", "u":"ُ", "a":"َ", "i":"ِ"}
>>> def repl(data, table):
...     for k,v in table.iteritems():
...         data = data.replace(k, v)
... 
>>> def trans(data, table):
...     return data.translate(table)
... 
>>> T = u'This is a test to see how fast is translitteration'
>>> timeit.timeit('trans(T, buckArab)', 'from __main__ import trans, T, buckArab', number=10**6)
6.766200065612793
>>> T = 'This is a test to see how fast is translitteration' #in python2 requires ASCII string
>>> timeit.timeit('repl(T, buckArab)', 'from __main__ import repl, T, buckArab', number=10**6)
12.668706893920898

正如您所看到的，即使是小字符串str.translate也快 2 倍。

于 2012-10-06T08:45:51.467 回答
4

每当我str.translate在 unicode 对象上使用它时，它都会返回相同的对象。这可能是由于Martijn Peters 提到的行为变化。

如果还有其他人正在努力将诸如阿拉伯语之类的 unicode 音译为 ascii，我发现将序数映射到 unicode 文字效果很好。

>>> buckArab = {"'":"ء", "|":"آ", "?":"أ", "&":"ؤ", "<":"إ", "}":"ئ", "A":"ا", "b":"ب", "p":"ة", "t":"ت", "v":"ث", "g":"ج", "H":"ح", "x":"خ", "d":"د", "*":"ذ", "r":"ر", "z":"ز", "s":"س", "$":"ش", "S":"ص", "D":"ض", "T":"ط", "Z":"ظ", "E":"ع", "G":"غ", "_":"ـ", "f":"ف", "q":"ق", "k":"ك", "l":"ل", "m":"م", "n":"ن", "h":"ه", "w":"و", "Y":"ى", "y":"ي", "F":"ً", "N":"ٌ", "K":"ٍ", "~":"ّ", "o":"ْ", "u":"ُ", "a":"َ", "i":"ِ"}
>>> ordbuckArab = {ord(v.decode('utf8')): unicode(k) for (k, v) in buckArab.iteritems()}
>>> ordbuckArab
{1569: u"'", 1570: u'|', 1571: u'?', 1572: u'&', 1573: u'<', 1574: u'}', 1575: u'A', 1576: u'b', 1577: u'p', 1578: u't', 1579: u'v', 1580: u'g', 1581: u'H', 1582: u'x', 1583: u'd', 1584: u'*', 1585: u'r', 1586: u'z', 1587: u's', 1588: u'$', 1589: u'S', 1590: u'D', 1591: u'T', 1592: u'Z', 1593: u'E', 1594: u'G', 1600: u'_', 1601: u'f', 1602: u'q', 1603: u'k', 1604: u'l', 1605: u'm', 1606: u'n', 1607: u'h', 1608: u'w', 1609: u'Y', 1610: u'y', 1611: u'F', 1612: u'N', 1613: u'K', 1614: u'a', 1615: u'u', 1616: u'i', 1617: u'~', 1618: u'o'}
>>> u'طعصط'.translate(ordbuckArab)
u'TEST'
于 2014-01-30T17:00:46.087 回答
3

您正在为每个角色重做相同的工作。当您这样做时data = data.replace(k, v)，它将替换整个文件中所有出现的给定字符。但是当您只需要为每个音译对执行一次时，您会在循环中一遍又一遍地执行此操作。只需删除最外层的循环，它就会极大地加快您的代码速度。

如果您需要进一步优化它，您可以查看 string translate 方法。我不确定这在性能方面如何。

于 2012-10-06T06:47:57.327 回答
2

扩展@larapsodia 的答案，这是带有字典的完整代码：

# -*- coding: utf-8 -*-

# Arabic Transliteration based on Buckwalter
# dictionary source is buckwalter2unicode.py http://www.redhat.com/archives/fedora-extras-commits/2007-June/msg03617.html 

buck2uni = {"'": u"\u0621", # hamza-on-the-line
            "|": u"\u0622", # madda
            ">": u"\u0623", # hamza-on-'alif
            "&": u"\u0624", # hamza-on-waaw
            "<": u"\u0625", # hamza-under-'alif
            "}": u"\u0626", # hamza-on-yaa'
            "A": u"\u0627", # bare 'alif
            "b": u"\u0628", # baa'
            "p": u"\u0629", # taa' marbuuTa
            "t": u"\u062A", # taa'
            "v": u"\u062B", # thaa'
            "j": u"\u062C", # jiim
            "H": u"\u062D", # Haa'
            "x": u"\u062E", # khaa'
            "d": u"\u062F", # daal
            "*": u"\u0630", # dhaal
            "r": u"\u0631", # raa'
            "z": u"\u0632", # zaay
            "s": u"\u0633", # siin
            "$": u"\u0634", # shiin
            "S": u"\u0635", # Saad
            "D": u"\u0636", # Daad
            "T": u"\u0637", # Taa'
            "Z": u"\u0638", # Zaa' (DHaa')
            "E": u"\u0639", # cayn
            "g": u"\u063A", # ghayn
            "_": u"\u0640", # taTwiil
            "f": u"\u0641", # faa'
            "q": u"\u0642", # qaaf
            "k": u"\u0643", # kaaf
            "l": u"\u0644", # laam
            "m": u"\u0645", # miim
            "n": u"\u0646", # nuun
            "h": u"\u0647", # haa'
            "w": u"\u0648", # waaw
            "Y": u"\u0649", # 'alif maqSuura
            "y": u"\u064A", # yaa'
            "F": u"\u064B", # fatHatayn
            "N": u"\u064C", # Dammatayn
            "K": u"\u064D", # kasratayn
            "a": u"\u064E", # fatHa
            "u": u"\u064F", # Damma
            "i": u"\u0650", # kasra
            "~": u"\u0651", # shaddah
            "o": u"\u0652", # sukuun
            "`": u"\u0670", # dagger 'alif
            "{": u"\u0671", # waSla
}

def transString(string, reverse=0):
    '''Given a Unicode string, transliterate into Buckwalter. To go from
    Buckwalter back to Unicode, set reverse=1'''

    for k, v in buck2uni.items():
      if not reverse:
            string = string.replace(v, k)
      else:
            string = string.replace(k, v)

    return string


>>> print(transString(u'مرحبا'))
mrHbA
>>> print(transString('mrHbA', 1))
مرحبا
>>>

这取自http://www.redhat.com/archives/fedora-extras-commits/2007-June/msg03617.html，它在 GPLv2 或更高版本下获得许可。

于 2016-10-21T22:21:50.237 回答