Metadata Filter in Flowise

[Ed_Moore]

Hi there,

I am trying to use the metadata filter in Flowise to filter a specific part of my corpa but I keep getting error code 400. Does anyone know how to solve this?

Here is how I am trying it right now.

Thanks

[aahmad]

Hi Ed, have you tried logging into the admin console and issuing the query directly in the corpus search tab? Does that work?

If that works correctly, please share your account number and we can try checking the logs.

[Ed_Moore]

Hey Amin,

Yes I have and it works just fine in there.

My customer ID is 1803420155.

Would you expect the way I added the metadata filter in Flowise to work? I also tried part.part_company being they key and Barclays Bank being the value but that didn’t work either.

Thanks again!

[aahmad]

The fact that it’s working fine in the console suggests to me that there is a problem with the integration between Flowise and Vectara. I will forward the issue internally to the engineer who developed it. I’ll keep you updated.

[Ed_Moore]

Ok got it, thanks again!

[seif]

Hello Ed,

It appears that there was an internal bug causing the issue you encountered. I’ve fixed the issue, and the fix should soon be live on the Flowise repository. I’ll let you know once it’s live. Apologies for any inconvenience this might have caused.

[Ed_Moore]

Hey Seif,

Great thanks so much! No inconvenience at all.

Thanks
Ed

[seif]

Hi Ed,

Thanks for your patience. The issue should be resolved now! Make sure you update your Flowise installation to receive the fix.

We’ve also added some documentation on specifying Vectara filters in Flowise: Vectara - FlowiseAI

Please let me know if you’re still running into issues.

[Ed_Moore]

Hey Seif,

Thanks for this, the docs make it much clearer.

Having said that, I am still having an issue with Vectara via Flowise and I am not 100% sure what the issue is.

I am using Anthropic and the Conversational Retrieval QA Chain in Flowise. Yesterday, I had it set up without Metadata for a corpora that had only information about one company in it (Tesco) and it was working perfectly.

However, now (with or without) metadata filters the results are drastically diffierent (much worse) than I get in the Vecata console.

Below are 4 images of the results I get from the search ‘Tell me about DevOps at Tesco’ in two corpora directly in the Vectara console & via Flowise.

• Corpora with only information about Tesco
• Corpora with only information about Tesco via Flowise
• Corpora with information about 3 companies, including Tesco using the Metadata tag
• Corpora with information about 3 companies, including Tesco using the Metadata tag via Flowise

Corpora with only information about Tesco:

• Corpora with only information about Tesco via Flowise

• Corpora with information about 3 companies, including Tesco using the Metadata tag
  

  

• Corpora with information about 3 companies, including Tesco using the Metadata tag via Flowise
  

Here is a screenshot of the ‘Source’ Document from the search results for the last image.

To me this suggests:

1. it is not accessing much of the ‘pagecontent’ and so cannot come up with an accurate answer via Flowise.
2. When I am applying the Metadata tag it is having some type of impact on the information retrieval outside of just the part_company. Otherwise I would expect the results to not be drastically different.

Is there anything I can do to solve these issues? Via Flowise / directly?

In an API call the charsbefore/after & sentencesbefore/after worked really well - is it possible to do this via Flowise in some way?

[Ed_Moore]

To add to this, I did a quick test and in a Corpora that only has information about Tesco with or without a part_company filter it gets the same result.

So it seems to be that if the Corpora has information that is not about that specific company then the search functionality does not work properly (for me at least).

Any help would be massively appreciated on both topics!

[seif]

Hi Ed,

Thanks for providing this information. I’ll try reproducing the issue you’re having to identify the problem.

In the meantime, could you provide additional details about your Flowise installation? Did you install it using npm install -g flowise? If so, what version of the package is currently installed? Alternatively, did you install it directly from the Flowise repository using git?

And yes we can add a configuration to allow edits to the charsbefore/after & sentencesbefore/after config.

[Ed_Moore]

Hey Seif,

I copied the git repo and have it running on Render, I updated it yesterday when you said it had been updated and I am currently 4 commits behind FlowiseAI:main.

I have managed to sort the issue using Pinecone which worked fine for all the tasks that I couldn’t do on Vectara so I do think it is either an issue with how my JSON was structured or the Flowise X Vectara connection.

Thanks anyway

[seif]

Hey Ed,

I’m sorry you couldn’t get the results you wanted using the Flowise and Vectara connection.

We’ve recently added an option to configure the number for sentences before and after in Flowise. This configuration might help address the issue you encountered. If you’re still open to it, I definitely recommend giving this option a try.

Best,
Seif

[Ed_Moore]

That’s excellent Seif, thank you!

I was playing around with it earlier and I think the issue may have had something to do with the amount of metadata I have per sectionZ

I will explore this tomorrow though, thanks again!

Ed

[Ed_Moore]

Hello again,

Just a quick question on this, I am now trying to filter Vectara via the Flowise API endpoint but it doesn’t seem to be working - am I doing something wrong?

The same thing works in the vectara portal so I am not sure if it is me or Flowise…

Thanks very much!
Ed

[Ed_Moore]

[Ed_Moore]

I figured it out via the Langchain doc so posting for anyone who comes across this…

“overrideConfig”: {
“filter”: “part.part_company = ‘HSBC’”
}