Deadline Exceeded using hybrid search

Vectara Platform Q&A
1

I found multiple timeouts yesterday in our logs alongside a couple of “connection reset by peer” errors.
Can you have a look and see what is causing the issue?
Customer ID: 1353344576
Corpus ID: 34
Hybrid search enabled with lambda = 0.005
Timeout is defined at 5 seconds

2024-03-22 03:51:13,867 Deadline Exceeded
2024-03-22 03:51:13,868 Query: facing
2024-03-22 03:51:13,868 Request duration: 5.001059293746948 seconds
2024-03-22 03:52:25,092 recvmsg:Connection reset by peer
2024-03-22 03:52:25,093 Query: 👯Soranet👏bog𝟤.t𝗈𝘱👐Pangyo rest area🚚Iksan racing girl🎩Daegu sex partner search🎑Samsung gunma👠Ochang SM Wonchu
2024-03-22 03:52:25,093 Request duration: 0.3337595462799072 seconds
2024-03-22 04:04:52,990 recvmsg:Connection reset by peer
2024-03-22 04:04:52,990 Query: al jatsiyah
2024-03-22 04:04:52,990 Request duration: 0.027812480926513672 seconds
2024-03-22 16:27:56,314 Deadline Exceeded
2024-03-22 16:27:56,315 Query: monotheism
2024-03-22 16:27:56,315 Request duration: 5.001343488693237 seconds
2024-03-22 18:37:51,430 Deadline Exceeded
2024-03-22 18:37:51,431 Query: circumcision
2024-03-22 18:37:51,431 Request duration: 5.0016186237335205 seconds
2024-03-22 18:38:03,630 Deadline Exceeded
2024-03-22 18:38:03,631 Query: shm
2024-03-22 18:38:03,631 Request duration: 5.001489877700806 seconds
2024-03-22 18:38:14,176 Deadline Exceeded
2024-03-22 18:38:14,177 Query: apparently
2024-03-22 18:38:14,177 Request duration: 5.000914812088013 seconds
2024-03-22 18:38:16,903 Deadline Exceeded
2024-03-22 18:38:16,904 Query: Baraqa
2024-03-22 18:38:16,904 Request duration: 5.118109941482544 seconds
2024-03-23 13:15:39,292 Deadline Exceeded
2024-03-23 13:15:39,293 Query: the belivers
2024-03-23 13:15:39,293 Request duration: 5.001176357269287 seconds
2024-03-23 13:16:21,187 Deadline Exceeded
2024-03-23 13:16:21,188 Query: the belivers
2024-03-23 13:16:21,188 Request duration: 5.001652956008911 seconds
2024-03-23 13:16:32,003 Deadline Exceeded
2024-03-23 13:16:32,004 Query: Duaik
2024-03-23 13:16:32,004 Request duration: 5.0013816356658936 seconds
2024-03-23 13:16:36,253 Deadline Exceeded
2024-03-23 13:16:36,253 Query: belivers
2024-03-23 13:16:36,254 Request duration: 5.225629806518555 seconds
2024-03-23 13:16:48,383 Deadline Exceeded
2024-03-23 13:16:48,384 Query: duaaekam
2024-03-23 13:16:48,384 Request duration: 5.001326322555542 seconds
2024-03-23 13:16:59,594 Deadline Exceeded
2024-03-23 13:16:59,595 Query: duaaekmsur
2024-03-23 13:16:59,595 Request duration: 5.001165390014648 seconds
2024-03-23 13:17:46,816 Deadline Exceeded
2024-03-23 13:17:46,816 Query: hear it
2024-03-23 13:17:46,817 Request duration: 5.001149415969849 seconds
2024-03-23 13:17:46,858 Deadline Exceeded
2024-03-23 13:17:46,858 Query: al falq
2024-03-23 13:17:46,859 Request duration: 5.001379013061523 seconds
2024-03-23 13:17:56,565 Deadline Exceeded
2024-03-23 13:17:56,566 Query: al falq
2024-03-23 13:17:56,566 Request duration: 5.00127649307251 seconds
2024-03-23 13:17:59,882 Deadline Exceeded
2024-03-23 13:17:59,883 Query: to your heart's content
2024-03-23 13:17:59,883 Request duration: 5.0010316371917725 seconds
2024-03-23 13:18:06,898 Deadline Exceeded
2024-03-23 13:18:06,899 Query: al musalat
2024-03-23 13:18:06,899 Request duration: 5.000871896743774 seconds
2024-03-23 13:18:50,766 Deadline Exceeded
2024-03-23 13:18:50,766 Query: 💦Shincheonji Game Land🍏𝗋𝚣d626.𝚝𝗈𝑝🌿Access to mobile reel game🍲Reel game scam guarantee💓Sea Story Mobile⚾Free Golden Castle📣Playground Reel Box
2024-03-23 13:18:50,767 Request duration: 5.000943899154663 seconds
2024-03-23 13:18:58,725 Deadline Exceeded
2024-03-23 13:18:58,726 Query: Alalak
2024-03-23 13:18:58,726 Request duration: 5.001136779785156 seconds
2024-03-23 13:19:25,757 Deadline Exceeded
2024-03-23 13:19:25,758 Query: 👕Sora Net🌼b𝚘𝚐1.𝗍𝘰𝑝🏈Western Net Site👰Honey Barnet Domain😊Korean Porn Site🌙avpop Domain🍄Jilsa.com Address
2024-03-23 13:19:25,758 Request duration: 5.001207590103149 seconds
2024-03-23 13:19:58,909 Deadline Exceeded
2024-03-23 13:19:58,910 Query: الانشقاق
2024-03-23 13:19:58,910 Request duration: 5.000959157943726 seconds
2024-03-23 13:20:08,193 Deadline Exceeded
2024-03-23 13:20:08,193 Query: greetings
2024-03-23 13:20:08,194 Request duration: 5.001113176345825 seconds
2024-03-23 13:21:07,419 Deadline Exceeded
2024-03-23 13:21:07,420 Query: hunting age
2024-03-23 13:21:07,420 Request duration: 5.001566171646118 seconds
2024-03-23 13:22:31,702 Deadline Exceeded
2024-03-23 13:22:31,703 Query: al
2024-03-23 13:22:31,703 Request duration: 5.001520872116089 seconds
2024-03-23 13:22:40,534 Deadline Exceeded
2024-03-23 13:22:40,535 Query: alkahf
2024-03-23 13:22:40,535 Request duration: 5.001166582107544 seconds
2024-03-23 13:22:45,099 Deadline Exceeded
2024-03-23 13:22:45,100 Query: Shirah
2024-03-23 13:22:45,100 Request duration: 5.222245454788208 seconds
2024-03-23 13:42:43,302 Deadline Exceeded
2024-03-23 13:42:43,302 Query: ibris
2024-03-23 13:42:43,302 Request duration: 5.001041412353516 seconds
2024-03-23 13:43:53,663 Deadline Exceeded
2024-03-23 13:43:53,664 Query: alkahf
2024-03-23 13:43:53,664 Request duration: 5.001100063323975 seconds
2024-03-23 13:44:00,813 Deadline Exceeded
2024-03-23 13:44:00,814 Query: fattah
2024-03-23 13:44:00,814 Request duration: 5.281635522842407 seconds
2024-03-23 13:44:37,921 Deadline Exceeded
2024-03-23 13:44:37,922 Query: al falq
2024-03-23 13:44:37,922 Request duration: 5.1197662353515625 seconds
2024-03-23 13:45:31,190 Deadline Exceeded
2024-03-23 13:45:31,190 Query: दसुरः रा रा'
2024-03-23 13:45:31,190 Request duration: 5.001297235488892 seconds
2024-03-23 14:11:44,573 Deadline Exceeded
2024-03-23 14:11:44,574 Query: lyeech
2024-03-23 14:11:44,574 Request duration: 5.0018310546875 seconds
2024-03-23 14:11:55,745 Deadline Exceeded
2024-03-23 14:11:55,745 Query: lyeech
2024-03-23 14:11:55,746 Request duration: 5.001041889190674 seconds
2024-03-23 14:11:57,542 Deadline Exceeded
2024-03-23 14:11:57,543 Query: two
2024-03-23 14:11:57,543 Request duration: 5.001358509063721 seconds
2024-03-23 14:11:58,880 Deadline Exceeded
2024-03-23 14:11:58,880 Query: tehreem
2024-03-23 14:11:58,881 Request duration: 5.0010974407196045 seconds
2024-03-23 14:17:41,416 Deadline Exceeded
2024-03-23 14:17:41,416 Query: Surah Mulk
2024-03-23 14:17:41,417 Request duration: 5.000994682312012 seconds
2024-03-23 14:18:19,521 Deadline Exceeded
2024-03-23 14:18:19,521 Query: Surah Mulk
2024-03-23 14:18:19,521 Request duration: 5.0014588832855225 seconds
2024-03-23 14:18:56,232 Deadline Exceeded
2024-03-23 14:18:56,233 Query: d
2024-03-23 14:18:56,233 Request duration: 5.001508474349976 seconds
2024-03-23 14:19:11,650 Deadline Exceeded
2024-03-23 14:19:11,650 Query: Casino Toto|r̔o̩w᷃8̞4̏9̉ㆍt̉òp̍/Best Baccarat Site-Live Super Casino Noliter/Evolution Work Betting|How to Play Slot Machines|Midas Phone Betting
2024-03-23 14:19:11,650 Request duration: 5.000849008560181 seconds
2024-03-23 14:19:50,641 Deadline Exceeded
2024-03-23 14:19:50,642 Query: لازواجكم
2024-03-23 14:19:50,642 Request duration: 5.26725697517395 seconds
2024-03-23 14:19:54,001 Deadline Exceeded
2024-03-23 14:19:54,001 Query: al baqrah
2024-03-23 14:19:54,002 Request duration: 5.001590251922607 seconds
2024-03-23 14:20:03,921 Deadline Exceeded
2024-03-23 14:20:03,922 Query: Azawajman
2024-03-23 14:20:03,922 Request duration: 5.0018134117126465 seconds
2024-03-23 14:20:26,954 Deadline Exceeded
2024-03-23 14:20:26,954 Query: kinzeer
2024-03-23 14:20:26,954 Request duration: 5.001352310180664 seconds
2024-03-23 14:20:31,362 Deadline Exceeded
2024-03-23 14:20:31,363 Query: Casino site⸤gm632,com⸣ 𓊈Code 451942𓊉 ↻﹙If you can't connect, KakaoTalk gm52﹚Baccarat site address♟︎Evolution Casino address≅Baccarat site address Casino coupon ꗃLA Casino⍢Joy Casino
2024-03-23 14:20:31,363 Request duration: 5.000988483428955 seconds
2024-03-23 14:20:55,949 Deadline Exceeded
2024-03-23 14:20:55,950 Query: rachm
2024-03-23 14:20:55,950 Request duration: 5.001011848449707 seconds
2024-03-23 14:20:56,055 Deadline Exceeded
2024-03-23 14:20:56,056 Query: Suraj Almulk
2024-03-23 14:20:56,056 Request duration: 5.001739263534546 seconds
2024-03-23 14:21:03,970 Deadline Exceeded
2024-03-23 14:21:03,971 Query: 😡Sea Story Free🐰r𝑠𝖽𝟻36.𝘵𝗈𝚙🎓Pachinko Site🚧Golden Star Game👰Sea Story PC Version👋Sea Story Season 7💾Golden Star Reel
2024-03-23 14:21:03,971 Request duration: 5.001146554946899 seconds
2024-03-23 14:21:24,301 Deadline Exceeded
2024-03-23 14:21:24,302 Query: Ramezan
2024-03-23 14:21:24,302 Request duration: 5.0010926723480225 seconds
2024-03-23 14:21:27,497 Deadline Exceeded
2024-03-23 14:21:27,498 Query: Ban:
2024-03-23 14:21:27,498 Request duration: 5.001168489456177 seconds
2024-03-23 14:21:39,350 Deadline Exceeded
2024-03-23 14:21:39,351 Query: bayy
2024-03-23 14:21:39,351 Request duration: 5.001195192337036 seconds
2

HI Hossam,

I do not see any such timeouts on our end. One guess is that there is some connectivity issue between your machine and our end-point. The first two errors (" Connection reset by peer) suggest the same.

Are you still facing this issue? Does the same happen when querying from the Web Console? Also, does it still happen when you set lamdba to zero?

/tallat

3

Hi Tallat,

No, it doesn’t always happen, any of these queries if I run it again with the same request body I get the response in less than 0.5 seconds.
It seems that the issue happens during a window of time, for example the last window in the logs I sent was from 14:17 to 14:21, like your endpoint was unreachable during this window of time.
We hit other APIs in the backend and none of them seems to show an exception during this time window, so I have strong confidence that there were no network issues from our side.

When you mentioned that you didn’t see any timeouts, did this mean that you never saw the requests reaching you or you saw the requests but no timeout occurs and it returned successfully ?

Let me know if you need other information to debug this issue further

4

from 14:17 to 14:21

Can you please mention date and timezone? It’ll help us look at things at our end. Thanks.

5

The window started at 2024-03-23 14:17:41 GMT timezone

6

Thanks for sharing the exact window. I looked at the logs quite a bit but couldn’t find any such errors. I also checked our servers; they weren’t restarted during this time window.

Can you share the code snippet used for handling timeouts (sending request, and processing response)? It might give a hint if the timeout was returned by our servers, or whether this was a client side error.

7

Sure, here is the code snippet

timeout = 120 if benchmark else 5
try:
    query_stub = services_pb2_grpc.QueryServiceStub(self.channel)
    packed_customer_id = struct.pack(">q", config.ZIR_CUSTOMER_ID)
    response = query_stub.Query(batch_request,
                                credentials=grpc.access_token_call_credentials(self.get_access_token()),
                                metadata=[("customer-id-bin", packed_customer_id)],
                                timeout=timeout)
    # some parsing code
    return parsed_results
    
except grpc.RpcError as rpc_error:
    self.logger.critical(rpc_error.details())
    self.logger.critical("Query: " + query)
    self.logger.critical("Request duration: %s seconds" % (time.time() - start))
    return []

As you can see a grpc.RpcError exception is thrown when the error happens

8

Looking at your code, I think this was a timeout that occurred on the client side. When the request times out on the server side, the timeout error is returned inside the query response in the statue object. The fact that this threw a timeout exception suggests it occurred on the client side. The “Connection” errors before the timeouts also suggests there was some network disruption, which resulted in timeouts for subsequent requests as well.

When I looked at requests in our analytics db, I see records of queries before and after your time window, but not during. This further supports the theory that the requests never reached the server.

┌          timestamp     ─┬ elapsedMs ┬lambdas
│ 2024-03-23 14:24:47.027 │       133 │ [0.005] │
│ 2024-03-23 14:24:41.653 │       139 │ [0.005] │
│ 2024-03-23 14:24:34.123 │       133 │ [0.005] │
│ 2024-03-23 14:24:18.015 │       130 │ [0.005] │
│ 2024-03-23 14:24:11.693 │       185 │ [0.005] │
│ 2024-03-23 14:23:18.688 │       142 │ [0.005] │
│ 2024-03-23 14:23:14.151 │       164 │ [0.005] │
│ 2024-03-23 14:22:58.039 │       172 │ [0.005] │
│ 2024-03-23 14:22:12.644 │       143 │ [0.005] │
│ 2024-03-23 14:17:30.651 │       132 │ [0.005] │
│ 2024-03-23 14:15:58.766 │       287 │ [0.005] │
│ 2024-03-23 14:15:23.223 │       134 │ [0.005] │
│ 2024-03-23 14:15:21.561 │       134 │ [0.005] │
...