Solr自定义筛选器标记流问题

Question

<tokenizer class="solr.WhitespaceTokenizerFactory"/>

    <filter class="solr.LowerCaseFilterFactory" />
    <filter class="solr.TrimFilterFactory" />
    <filter class="solr.TrimFilterFactory" pattern="([^a-z])" replacement="" replace="all" />
    <filter class="solr.StopFilterFactory" ignoreCase="true" words="custom_stop_words.txt"/>
    <filter class="intuit.ripple.solr.ConcatFilterFactory" />
    <filter class="solr.NGramFilterFactory" minGramSize="3" maxGramSize="3" />

1. Input value: "foo Bar Baz qux"
2. WhitespaceTokenizerFactory: "foo", "Bar", "Baz", "qux"
3. LowerCaseFilterFactory: "foo", "bar", "baz", "qux"
4. TrimFilterFactory, TrimFilterFactory and StopFilterFactory have nothing to do in this case.
5. ConcatFilterFactory: "foobarbazqux". It should concatenate the tokens.
6. NGramFilterFactory: This will generate the token.

@Override
public boolean incrementToken()  throws IOException  {

    StringBuilder builder = new StringBuilder();

    while (input.incrementToken()) {
        int len = charTermAtt.length();
        char buffer[] = charTermAtt.buffer();
        builder.append(buffer, 0, len);
        System.out.println("Tokens: " + new String(buffer, 0, len));
        clearAttributes();
        charTermAtt.setEmpty();
    }
    System.out.println("Concat tokens: " + builder.toString());

    charTermAtt.copyBuffer(builder.toString().toCharArray(), 0, builder.length());
    charTermAtt.setLength(builder.length());
    posIncAtt.setPositionIncrement(1);
    setOffsetAttr.setOffset(0, builder.length());

    input.end();
    input.close();
    return false;
}

Answer 1

我想你想做一些其他的事情，而不是你实现的：D

在incrementToken方法中所做的只是遍历整个输入（在本例中是StopFilter的输出）。在每次调用增量令牌时，您只需从输入中获取一个（如果需要的话，还需要更多）令牌，然后生成一个令牌输出。

所以我想您不想在这里使用“while”循环，并且在每次中间调用“clearattributes（）”。

Tokens: foo
Tokens: bar
Concat tokens: foo bar