使用Lucene 4.0为单词添加词干并创建不带停止词的索引

Question

我有以下问题：有几个文本文档需要解析和创建索引，但没有停止词和词干。我可以手动操作，但我从一位同事那里听说Lucene可以自动操作。我在网上搜索了很多我尝试过的例子，但是每个例子都使用了不同版本的lucene和不同的方法，没有一个例子是完整的。在这个过程结束时，我需要计算集合中每个术语的tf/idf。

更新：我现在已经用一个文档创建了一个索引。doc没有停止词，并且有词干。如何使用lucenc计算此文档的tf/idf？（我将在计算出如何进行计算后添加更多文档）

对lucene的任何帮助都将不胜感激。谢谢

import java.io.*;
    import java.util.HashSet;
    import org.apache.lucene.analysis.*;
    import org.apache.lucene.analysis.tokenattributes.*;
    import org.apache.lucene.analysis.standard.*;
    import org.apache.lucene.store.Directory;
    import org.apache.lucene.store.FSDirectory;
    import org.apache.lucene.util.*;
    import org.apache.lucene.analysis.snowball.*;
    import org.apache.lucene.document.Document;
    import org.apache.lucene.document.Field;
    import org.apache.lucene.index.IndexWriter;


public class Stemmer
{
    static HashSet<String> stopWordsList = null;

    public static String Stem(String text, String language) throws IOException
    {
        parse p = new parse();
        stopWordsList = p.readStopWordsFile();
        StringBuffer result = new StringBuffer();
        if (text!=null && text.trim().length()>0)
        {
            StringReader tReader = new StringReader(text);
            // Analyzer analyzer = new StopAnalyzer(Version.LUCENE_36,stopWordsList);
            @SuppressWarnings("deprecation")
            Analyzer analyzer = new SnowballAnalyzer(Version.LUCENE_35,"English",stopWordsList);
            // disk index storage
            Directory directory = FSDirectory.open(new File("d:/index")); 

            @SuppressWarnings("deprecation")
            IndexWriter writer = new IndexWriter(directory, analyzer, true, new IndexWriter.MaxFieldLength(25000));

            TokenStream tStream = analyzer.tokenStream("contents", tReader);
            @SuppressWarnings("deprecation")
            TermAttribute term = tStream.addAttribute(TermAttribute.class);

            try {
                while (tStream.incrementToken())
                    {
                        result.append(term.term());
                        result.append(" ");
                    }

                Document doc = new Document();
                String title = "DocID";
                // adding title field
                doc.add(new Field("title", title, Field.Store.YES, Field.Index.ANALYZED)); 
                String content = result.toString();
                // adding content field
                doc.add(new Field("content", content, Field.Store.YES, Field.Index.ANALYZED));
                // writing new document to the index
                writer.addDocument(doc);  
                writer.close();
                System.out.println("Reult is: " + result);  
            } 
            catch (IOException ioe)
                {
                    System.out.println("Error: "+ioe.getMessage());
                }
        }

        // If, for some reason, the stemming did not happen, return the original text
        if (result.length()==0)
            result.append(text);
        return result.toString().trim();

    } //end stem

    public static void main (String[] args) throws IOException
        {
            Stemmer.Stem("Michele Bachmann amenities pressed her allegations that the former head of her Iowa presidential bid was bribed by the campaign of rival Ron Paul to endorse him, even as one of her own aides denied the charge.", "English");
        }
}//end class    

Answer 1

要过滤掉停止字，请使用StopAnalyzer。它将删除以下文字：

  "a", "an", "and", "are", "as", "at", "be", "but", "by",
  "for", "if", "in", "into", "is", "it",
  "no", "not", "of", "on", "or", "such",
  "that", "the", "their", "then", "there", "these",
  "they", "this", "to", "was", "will", "with"

如果您使用addDocument（Iterable），则可以提供Analyzer

关于词干分析，请看这篇文章。

很难给出更多建议，因为你没有解释到底是什么失败了。