nlp - 是否有任何可用于印度语言的词干分析器

Question

是否有任何用于印度语言的词干分析器的实现，例如（印地语，泰卢固语）......

Answer 1

带有词干分析器的印地语分析器在 Lucene 中可用。它基于此算法(pdf)。

Answer 2

Hindi_stemmer是Ananthakrishnan Ramanathan 和 Durgesh D Rao在“ A Lightweight Stemmer for Hindi ”中描述的印地语词干分析器的 Python 实现。

Answer 3

import java.util.Map;
import java.util.WeakHashMap;

/** * Hindi light stemmer- 从名词和形容词中删除数字、性别和格后缀

public class HindiStemmerLight{

/**
 * A cache of words and their stems
 */
static private Map<String, String> cache = new WeakHashMap<String, String>();

/**
 * A buffer of the current word being stemmed
 */
private StringBuilder sb = new StringBuilder();

/**
 * Default constructor
 */
public HindiStemmerLight() {
}

public String stem(String word) {
    String result = cache.get(word);

    if (result != null)
        return result;

    // 
    sb.delete(0, sb.length());

    // 
    sb.append(word);

    /* remove the case endings from nouns and adjectives */
    remove_suffix(sb);



    result = sb.toString();
    cache.put(word, result);

    return result;
}

private void remove_suffix(StringBuilder word) {
    int len = word.length() - 1;

    /* article */

    if (len > 4) {
        if (word.substring( len- 2, len+1).equals("िया")) {
            word.delete(len-2 , len + 1);
            return;
        }
        if (word.substring( len- 2, len+1).equals("ियो")) {
            word.delete(len-2 , len + 1);
            return;
        }

    } /* end if len >4 */
    if (len > 3) {
        if (word.substring(len-1, len+1).equals("ाए")) {
            word.delete(len - 1, len + 1);
            return;
        }
        if (word.substring(len-1, len+1).equals(" ाओ")) {
            word.delete(len - 1, len + 1);
            return;
        }
        if (word.substring(len-1, len+1).equals(" ुआ")) {
            word.delete(len - 1, len + 1);
            return;
        }
        if (word.substring(len-1, len+1).equals(" ुओ")) {
            word.delete(len - 1, len + 1);
            return;
        }
        if (word.substring( len- 1, len+1).equals("ये")) {
            word.delete(len-1 , len + 1);
            return;
        }
        if (word.substring(len-1, len+1).equals(" ेन")) {
            word.delete(len - 1, len + 1);
            return;
        }
        if (word.substring(len-1, len+1).equals(" ेण")) {
            word.delete(len - 1, len + 1);
            return;
        }
        if (word.substring( len- 1, len+1).equals(" ीय")) {
            word.delete(len-1 , len + 1);
            return;
        }
        if (word.substring(len-1, len+1).equals("टी")) {
            word.delete(len - 1, len + 1);
            return;
        }
        if (word.substring(len-1, len+1).equals("ार")) {
            word.delete(len - 1, len + 1);
            return;
        }
        if (word.substring(len-1, len+1).equals("ाई")) {
            word.delete(len - 1, len + 1);
            return;
        }

    } /* end if len > 3 */
    if (len > 2) {
        if (word.substring(len, len+1).equals(" ा")) {
            word.delete(len , len + 1);
            return;
        }
        if (word.substring(len, len+1).equals(" े")) {
            word.delete(len , len + 1);
            return;
        }
        if (word.substring(len, len+1).equals(" ी")) {
            word.delete(len , len + 1);
            return;
        }
        if (word.substring(len, len+1).equals(" ो")) {
            word.delete(len , len + 1);
            return;
        }
        if (word.substring(len, len+1).equals("ि ")) {
            word.delete(len , len + 1);
            return;
        }
        if (word.substring(len, len+1).equals("अ")) {
            word.delete(len , len + 1);
            return;
        }

    } /* end if len > 2 */
    return;
}

}

Answer 4

我们创建了原始代码Python Hindi Stemmer的 java 版本

虽然原始印地语词干分析器的作者使用变量 L 的方式还不是很清楚，但有一个完整的代码可以工作

导入 java.util.ArrayList；导入 org.apache.commons.lang.StringUtils；

public class SimpleHindiStemmer {

/* This is not coming while pasting the code better copy from Python Code as given in suffixes[1]  */
private static String [] stem1 = new String [] { "ो", "े", "ू", "ु", "ी", "ि", "ा" };

/* This is not coming while pasting the code better copy from Python Code as given in suffixes[2]  */
private static String [] stem2 = new String [] { "कर", "ाओ", "िए", "ाई", "ाए", "ने", "नी", "ना", "ते", "ीं", "ती", "ता", "ाँ", "ां", "ों", "ें" };

private static String [] stem3 = new String [] { "ाकर", "ाइए", "ाईं", "ाया", "ेगी", "ेगा", "ोगी", "ोगे", "ाने", "ाना", "ाते", "ाती", "ाता", "तीं", "ाओं", "ाएं", "ुओं", "ुएं", "ुआं" };

private static String [] stem4 = new String [] { "ाएगी", "ाएगा", "ाओगी", "ाओगे", "एंगी", "ेंगी", "एंगे", "ेंगे", "ूंगी", "ूंगा", "ातीं", "नाओं", "नाएं", "ताओं", "ताएं", "ियाँ", "ियों", "ियां" };

private static String [] stem5 = new String [] { "ाएंगी", "ाएंगे", "ाऊंगी", "ाऊंगा", "ाइयाँ", "ाइयों", "ाइयां" };

private static ArrayList<String []> stemList = new ArrayList<String []>(5);

static int [] cut = new int [] { 0, 1, 1, 1, 2, 2 };

static {

    stemList.add(stem5);
    stemList.add(stem4);
    stemList.add(stem3);
    stemList.add(stem2);
    stemList.add(stem1);

}

public SimpleHindiStemmer(){

}

public static void main(String [] argv){

    SimpleHindiStemmer sm = new SimpleHindiStemmer();
    String word = "रास्ते";
    System.out.println(sm.stemprocess(word));
}

public String stemprocess(String word){


    int wlen = word.length();
    int wordlen = wlen*3;
    int icnt = 5;
    for (String [] stemwords : stemList){
        if(wordlen > (icnt + 1)){
            for(String sw: stemwords){
                if(StringUtils.endsWith(word, sw)){
                    return StringUtils.substring(word, 0, wlen - cut[icnt]);

                }
            }

        }


        icnt--;
    }

    return word;
}
}

如您所见，在某些情况下 utf-8 字符不能很好地捕获。查看原始 python 代码，然后从那里复制后缀值。