Question

I am trying to make a custom filter in Lucene which simply recognizes whether two consequent words in a text start with a capital letter and have the rest as lower case, in which case the two words are to be joined as one token.

The overriden incrementToken method has the following code

@Override
    public boolean incrementToken() throws IOException {
        if(!input.incrementToken()){ 
     return false;}
     //Case were the previous token WAS NOT starting with capital letter and the     rest small   
     if(previousTokenCanditateMainName==false)
     {
            if(CheckIfMainName(termAtt.term()))
            {
                previousTokenCanditateMainName=true;
                tempString=this.termAtt.term() ;                           /*This is the*/
               // myToken.offsetAtt=this.offsetAtt;                             /*Token i need to "delete"*/
            tempStartOffset=this.offsetAtt.startOffset();
            tempEndOffset=this.offsetAtt.endOffset();
            return true;
        }
        else
        {
            return true;
        }
 }
 //Case were the previous token WAS a Proper name (starting with Capital and continuiing with small letters)
     else
         {
         if(CheckIfMainName(termAtt.term()))
         {
             previousTokenCanditateMainName=false;
             posIncrAtt.setPositionIncrement(0);
             termAtt.setTermBuffer(tempString+TOKEN_SEPARATOR+this.termAtt.term());
             offsetAtt.setOffset(tempStartOffset, this.offsetAtt.endOffset());
             return true;
         }
         else
         {
             previousTokenCanditateMainName=false;
             return true;
         }
     }

}

My question is how once i find the first Token that meets my requirements can i “ignore” it.
Currently the code works perfectly with joining the two tokens but i also get an extra token with the first one of the two that I identified.
I tried using the same method setEnableIncrementsPosition(true) as does the built-in stopFilter but in that case my filter needs to be a TokenFilter type which does not allow me to override the incrementToken method.

I hope i phrased my problem properly

Answer 1

You might have a custom method:

private void tokenize() 

where you do the splitting and the custom joins. The resulting List<String> tokens need to be held as an attribute of the tokenizer.

In the incrementToken method you simply check if this attribute is null and initialize it if necessary.

You also need to add the tokens in the incrementToken() method to the termAttribute

termAttribute.append(tokens.get(tokenIndex));

this includes that your Tokenizer needs to have an attribute like this:

private CharTermAttribute termAttribute = addAttribute(CharTermAttribute.class);

Probably you need also some fine tuning. But thats only a draft on how this can be achieved in a pretty simple way.