16

使用机器翻译,我可以获得一个非常压缩的句子版本,例如。我真的很想喝一杯美味可口的咖啡会被翻译成我想要咖啡 是否有任何 NLP 引擎提供这样的功能?

我得到了一些研究论文,这些论文进行了副相生成句子压缩。但是有没有任何图书馆已经实现了这一点?

4

4 回答 4

10

如果你的目的是让你的句子简短而不丢失句子中的重要思想,那么你可以通过提取三元组主谓宾来做到这一点。

谈到工具/引擎,我推荐你使用斯坦福 NLP。它的依赖解析器输出已经提供了主题和对象(如果有的话)。但是您仍然需要进行一些调整才能获得所需的结果。

您可以在此处下载斯坦福 NLP 并学习示例用法

我找到了与您的问题相关的论文。看看使用类型依赖的文本简化:不同生成策略的鲁棒性的比较

于 2012-01-09T12:46:59.490 回答
2

这是我发现的:

Clarke 和 Lapata,2008 年,“句子压缩的全局推理:整数线性规划方法”中描述的模型的修改实现。

论文:https ://www.jair.org/media/2433/live-2433-3731-jair.pdf

来源:https ://github.com/cnap/sentence-compression (用JAVA写的)

输入: 在营地,叛军受到了横幅的欢迎,上面写着“欢迎回家”。

输出: 在营地,军队受到了欢迎。

更新: 用于文本摘要的带有注意力模型的序列到序列。

https://github.com/tensorflow/models/tree/master/textsum

https://arxiv.org/abs/1509.00685

于 2015-04-24T05:45:16.840 回答
1

首先尝试使用 watson NaturalLanguageUnderstanding/Alchemy 库。使用它我能够从我的陈述中提取重要的关键字,例如:

输入:嘿!我的笔记本电脑屏幕有问题

输出:笔记本电脑屏幕问题硬件。

不仅是改写而且使用 NLU,您可以获得输入语句的以下详细信息,如上述语句,您可以获得以下类别的详细信息:

像“en”这样的语言、实体、概念、像“笔记本电脑屏幕”这样的关键字、带有相关性、文本、关键字情感、情绪等细节的“问题”。带有标签、相关性分数等细节的类别。带有句子、主题等细节的语义角色,动作和对象

除此之外,您还可以使用语气分析器来获得陈述的突出语气,例如恐惧、愤怒、快乐、厌恶等。

以下是 watson 库的代码示例 注意:waston 库不是免费的,但提供一个月的试用期,因此您可以从这个开始,然后一旦掌握了这些概念,然后切换到其他开源库并找出类似的库和功能

NaturalLanguageUnderstanding service = new NaturalLanguageUnderstanding(
    NaturalLanguageUnderstanding.VERSION_DATE_2017_02_27,
    WatsonConfiguration.getAlchemyUserName(),
    WatsonConfiguration.getAlchemyPassword());


//ConceptsOptions
ConceptsOptions conceptOptions = new ConceptsOptions.Builder()
    .limit(10)
    .build();

//CategoriesOptions
    CategoriesOptions categoriesOptions = new CategoriesOptions();

//SemanticOptions
SemanticRolesOptions semanticRoleOptions = new SemanticRolesOptions.Builder()
    .entities(true)
    .keywords(true)
    .limit(10)
    .build();

EntitiesOptions entitiesOptions = new EntitiesOptions.Builder()
    .emotion(true)
    .sentiment(true)
    .limit(10)
    .build();

KeywordsOptions keywordsOptions = new KeywordsOptions.Builder()
    .emotion(true)
    .sentiment(true)
    .limit(10)
    .build();

Features features = new Features.Builder()
    .entities(entitiesOptions)
    .keywords(keywordsOptions)
    .concepts(conceptOptions)
    .categories(categoriesOptions)
    .semanticRoles(semanticRoleOptions)
    .build();

AnalyzeOptions parameters = new AnalyzeOptions.Builder()
    .text(inputText)
    .features(features)
    .build();

AnalysisResults response = service
    .analyze(parameters)
    .execute();
System.out.println(response);
于 2017-07-06T06:02:33.527 回答
0

您可以结合使用“去除停用词”和“词干和词形还原”。Stemming and lemmatization 是将文本中的所有单词返回到其基本词根的过程,您可以在此处找到完整的解释,我正在使用 Porter stemmer 在 google 中查找。在词干和词形还原之后,停用词的去除非常容易,这是我的停用词去除方法:

public static String[] stopwords ={"a", "about", "above", "across", "after", "afterwards", "again", "against", "all", "almost", 
    "alone", "along", "already", "also","although","always","am","among", "amongst", "amoungst", "amount",  "an", "and", 
    "another", "any","anyhow","anyone","anything","anyway", "anywhere", "are", "around", "as",  "at", "back","be","became", 
    "because","become","becomes", "becoming", "been", "before", "beforehand", "behind", "being", "below", "beside", "besides", 
    "between", "beyond", "bill", "both", "bottom","but", "by", "call", "can", "cannot", "cant", "co", "con", "could", "couldnt",
    "cry", "de", "describe", "detail", "do", "done", "down", "due", "during", "each", "eg", "eight", "either", "eleven","else",
    "elsewhere", "empty", "enough", "etc", "even", "ever", "every", "everyone", "everything", "everywhere", "except", "few", 
    "fifteen", "fify", "fill", "find", "fire", "first", "five", "for", "former", "formerly", "forty", "found", "four", "from", 
    "front", "full", "further", "get", "give", "go", "had", "has", "hasnt",
    "have", "he", "hence", "her", "here", "hereafter", "hereby", "herein", "hereupon", "hers", "herself", 
    "him", "himself", "his", "how", "however", "hundred", "ie", "if", "in", "inc", "indeed", "interest", "into", 
    "is", "it", "its", "itself", "keep", "last", "latter", "latterly", "least", "less", "ltd", "made", "many", 
    "may", "me", "meanwhile", "might", "mill", "mine", "more", "moreover", "most", "mostly", "move", "much", "must", 
    "my", "myself", "name", "namely", "neither", "never", "nevertheless", "next", "nine", "no", "nobody", "none", 
    "noone", "nor", "not", "nothing", "now", "nowhere", "of", "off", "often", "on", "once", "one", "only", "onto", 
    "or", "other", "others", "otherwise", "our", "ours", "ourselves", "out", "over", "own","part", "per", "perhaps",
    "please", "put", "rather", "re", "same", "see", "seem", "seemed", "seeming", "seems", "serious", "several", "she",
    "should", "show", "side", "since", "sincere", "six", "sixty", "so", "some", "somehow", "someone", "something", 
    "sometime", "sometimes", "somewhere", "still", "such", "system", "take", "ten", "than", "that", "the", "their", 
    "them", "themselves", "then", "thence", "there", "thereafter", "thereby", "therefore", "therein", "thereupon", 
    "these", "they", "thickv", "thin", "third", "this", "those", "though", "three", "through", "throughout", "thru", 
    "thus", "to", "together", "too", "top", "toward", "towards", "twelve", "twenty", "two", "un", "under", "until", 
    "up", "upon", "us", "very", "via", "was", "we", "well", "were", "what", "whatever", "when", "whence", "whenever",
    "where", "whereafter", "whereas", "whereby", "wherein", "whereupon", "wherever", "whether", "which", "while", 
    "whither", "who", "whoever", "whole", "whom", "whose", "why", "will", "with", "within", "without", "would", "yet",
    "you", "your", "yours", "yourself", "yourselves","1","2","3","4","5","6","7","8","9","10","1.","2.","3.","4.","5.","6.","11",
    "7.","8.","9.","12","13","14","A","B","C","D","E","F","G","H","I","J","K","L","M","N","O","P","Q","R","S","T","U","V","W","X","Y","Z",
    "terms","CONDITIONS","conditions","values","interested.","care","sure","!","@","#","$","%","^","&","*","(",")","{","}","[","]",":",";",",","<",">","/","?","_","-","+","=",
    "a","b","c","d","e","f","g","h","i","j","k","l","m","n","o","p","q","r","s","t","u","v","w","x","y","z",
    "contact","grounds","buyers","tried","said,","plan","value","principle.","forces","sent:","is,","was","like",
    "discussion","tmus","diffrent.","layout","area.","thanks","thankyou","hello","bye","rise","fell","fall","psqft.","http://","km","miles"};

在我的项目中,我使用段落作为文本输入:

public static String removeStopWords(String paragraph) throws IOException{
    Scanner paragraph1=new Scanner( paragraph );
    String newtext="";
    Map map = new TreeMap();
    Integer ONE = new Integer(1);
    while(paragraph1.hasNext()) {
        int flag=1;
        fixString=paragraph1.next();
        fixString=fixString.toLowerCase();
        for(int i=0;i<stopwords.length;i++) {
            if(fixString.equals(stopwords[i])) {
                flag=0;
            }
        }
        if(flag!=0) {
            newtext=newtext+fixString+" ";  
        }
            if (fixString.length() > 0) {
            Integer frequency = (Integer) map.get(fixString);
            if (frequency == null) {
                frequency = ONE;
            } else {
                int value = frequency.intValue();
                frequency = new Integer(value + 1);
            }
            map.put(fixString, frequency);                 
            }                     
    }
    return newtext;
}

我使用了斯坦福 NLP 库,你可以从这里下载。我希望我在某种程度上帮助了你。

于 2015-03-08T19:04:17.840 回答