Home/ Questions/Q 3799840
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-19T13:47:55+00:00

For a toy project, I want to implement an automated question answering system with

  • 0

For a toy project, I want to implement an automated question answering system with Lucene and I’m trying to figure out a reasonable way to implement it. The basic operation is as follows:

1) The user will enter a question.

2) The system will identify the keywords in the question.

3) The keywords will be searched in a large knowledgebase and matching sentences will be shown as answers.

My knowledgebase (i.e., corpus) is not structured. It is just a large, continuous text (say, a user manual without any chapters). I mean that the only structure is that sentences and paragraphs are identified.

I plan to treat each sentence or paragraph as a separate document. To present the answer in a context, I may consider keeping one sentence/paragraph before/after the indexed one as payload. I would like to know if that makes sense. Also, I’m wondering if there are other tried and well-known approaches for that kind of systems. As an example, another approach that comes to mind is to index large chunks of the corpus as documents with the token positions, then process the vicinity of found keywords to construct my answers.

I would appreciate direct recommendations based on experience or intuition, but also tutorials or introductory materials to question-answering systems with Lucene in mind.

Thanks.

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Indexing each sentence as a document will give you some problems. You’ve pointed out one: you would need to store the surrounding texts a payloads. That means you’ll need to store each sentence three times (before, during and after), and you’ll have to manually get into the payload.

    If you want to go the route of each sentence being a document, I would recommend coming up with an ID for each sentence and storing that as a separate field. Then you can display [ID-1, ID, ID+1] in each result.

    The bigger question though is: how should you break up the text into documents? Identifying semantically related areas seems difficult, so doing it by sentence/paragraph might be the only way to go. A better way would be if you could find which text is the header of a section, and then put everything in that section as a document.

    You might also want to use the index (if your corpus has one). The terms there could be boosted, as they are presumably more important.

    • 0