Question

I have been trying to implement a custom analyzer in Lucene. I think I am really close to finishing but I face two strange issues.

First, my filter works as expected for every term in the tokenstream except the last one. (Though I am trying to handle it).

Second, I wouldn’t have a problem using that TokenFilter (even missing the last term). But although the indexing is working perfectly (checked the resulting index with Luke), when I try to use my analyzer to parse user queries = the resulting Query is blank(!) Could this be due to the missing term?

I have posted the incrementToken() method of filter below. Any help would be really welcome. Thank you in advance.

P.S. I now that in terms of contribution this question is not good, but i could not find something specific elsewhere.

public boolean incrementToken() throws IOException {
    if (!input.incrementToken()) {
        if (previousTokenFlag) {
            tempPreviousToken.attSource.copyTo(this);
            previousTokenFlag = false;
            this.incrementToken();
            return false;
        } else {
            return false;
        }
    }
    if (previousTokenFlag) {
        if (CheckIfMainName(this.termAtt.term())) {
            if (CheckIfMainName(tempPreviousToken.termAtt.term())) {
                termAtt.setTermBuffer(tempPreviousToken.termAtt.term() + 
                       TOKEN_SEPARATOR + this.termAtt.term());
                this.setPreviousTokenFlag(false);
                return true;
            } else {
                tempHelpingToken = new TempToken(this.input.cloneAttributes());
            }
            tempPreviousToken.attSource.copyTo(this);
            tempHelpingToken.attSource.copyTo(tempPreviousToken.attSource);
            return true;
        } else {
            if (CheckIfMainName(tempPreviousToken.termAtt.term())) {
                tempHelpingToken = new TempToken(this.input.cloneAttributes());
                tempPreviousToken.attSource.copyTo(this);
                tempHelpingToken.attSource.copyTo(tempPreviousToken.attSource);
                tempHelpingToken.attSource.clearAttributes();
                return true;
            } else {
                tempHelpingToken = new TempToken(this.input.cloneAttributes());
                tempPreviousToken.attSource.copyTo(this);
                tempHelpingToken.attSource.copyTo(tempPreviousToken.attSource);
                tempHelpingToken.attSource.clearAttributes();
                return true;
            }
        }
    } else {
        tempPreviousToken = new TempToken(this.input.cloneAttributes());
        tempPreviousToken.termAtt.setTermBuffer(this.termAtt.term());
        this.setPreviousTokenFlag(true);
        this.incrementToken();
        return true;
    }
}

Answer 1

I am not sure what you are trying to do, but the first lines of you filter look wrong:

        tempPreviousToken.attSource.copyTo(this);
        previousTokenFlag = false;
        this.incrementToken();
        return false;

First, you copy your tempPreviousToken to the current token (tempPreviousToken.attSource.copyTo(this)), and then you always return false, giving no chance to the current token to be ever used. You should either return false directly, or return true/false depending on the token you tried to copy.