Question

Working with a Lucene index, I have a standard document format that looks something like this:

Name: John Doe 
Job: Plumber 
Hobby: Fishing

My goal is to append a payload to the job field that would hold additional information about Plumbing, for instance, a wikipedia link to the plumbing article. I do not want to put payloads anywhere else. Initially, I found an example that covered what I’d like to do, but it used Lucene 2.2, and has no updates to reflect the changes in the token stream api.
After some more research, I came up with this little monstrosity to build a custom token stream for that field.

public static TokenStream tokenStream(final String fieldName, Reader reader, Analyzer analyzer, final String item) {
        final TokenStream ts = analyzer.tokenStream(fieldName, reader) ;
        TokenStream res = new TokenStream() {
            CharTermAttribute termAtt = addAttribute(CharTermAttribute.class);
            PayloadAttribute payAtt = addAttribute(PayloadAttribute.class);

            public boolean incrementToken() throws IOException {
                while(true) {
                    boolean hasNext = ts.incrementToken();
                    if(hasNext) {
                        termAtt.append("test");
                        payAtt.setPayload(new Payload(item.getBytes()));
                    }
                    return hasNext;
                }
            }

        };
        return res;
    } 

When I take the token stream and iterate over all the results, prior to adding it to a field, I see it successfully paired the term and the payload. After calling reset() on the stream, I add it to a document field and index the document. However, when I print out the document and look at the index with Luke, my custom token stream didn’t make the cut. The field name appears correctly, but the term value from the token stream does not appear, nor does either indicate the successful attachment of a payload.

This leads me to 2 questions. First, did I use the token stream correctly and if so, why doesn’t it tokenize when I add it to the field? Secondly, if I didn’t use the stream correctly, do I need to write my own analyzer. This example was cobbled together using the Lucene standard analyzer to generate the token stream and write the document. I’d like to avoid writing my own analyzer if possible because I only wish to append the payload to one field!

Edit:

Calling code

TokenStream ts = tokenStream("field", new StringReader("value"), a, docValue);
        CharTermAttribute cta = ts.getAttribute(CharTermAttribute.class);
        PayloadAttribute payload = ts.getAttribute(PayloadAttribute.class);
        while(ts.incrementToken()) {
            System.out.println("Term = " + cta.toString());
            System.out.println("Payload = " + new String(payload.getPayload().getData()));

        }
        ts.reset();

Answer 1

It’s very hard to tell why the payloads are not saved, the reason may lay in the code that uses the method that you presented.

The most convenient way to set payloads is in a TokenFilter — I think that taking this approach will give you much cleaner code and in turn make your scenario work correctly. I think that it’s most illustrative to take a look at some filter of this type in Lucene source, e.g. TokenOffsetPayloadTokenFilter. You can find an example of how it should be used in the test for this class.

Please also consider if there is no better place to store these hyperlinks than in payloads. Payloads have very special application for e.g. boosting some terms depending on their location or formatting in the original document, part of speech… Their main purpose is to affect how the search is performed, so they are normally numeric values, efficiently packed to cut down the index size.