I suspect my corpus is polluted with duplicate files with different filenames, so my responses are polluted with ‘duplicate’ response texts. Can the API remove duplicate entries from the ‘response’ payload? if the text matches in both - its a duplicate as far as I’m concerned.
would I be safe deduplicating by ‘score’? does that have enough decimals that any exact ‘tie’ is likely a duplicate content?
Hey @David_Cawley
Of course the best way would be to identify these cases if possible and remove old documents if they do not apply anymore and have been replaced by an updated document.
But there is also a very cool functionality that’s available in the API to remove duplicates. IT’s called MMR and stands for Max Marginal Relevance. See Get Diverse Results and Comprehensive Summaries with Vectara’s MMR Reranker - Vectara for more details, and here in the docs: Reranking | Vectara Docs