I'm receiving the error: "Some data remained in the queue for too long and so was expired (consumer too slow)"

If your data destination is file based and consuming data too slowly, you will receive the error message in your /push/log, or Task log:

"Some data remained in the queue for too long and expired (consumer too slow)"

This indicates that there has been data loss due to Push's buffer being exhausted. Data that remains in the buffer for longer than an hour will be dropped.

The rate of consumption for a Push subscription is dictated by two parameters: Delivery Frequency and Max Size. To avoid expiring data, it is generally suggested that Delivery Frequency be set to continuous delivery whenever possible ( output_params.delivery_frequency=0 ), and Max Size to the largest size that your endpoint can accept per delivery.

To ensure that you are consuming fast enough, monitor your 'remaining_bytes'. You can see this value when you hit the /push/get API endpoint. This value tells you how many bytes are in DataSift's Push buffer, waiting to be delivered. If this value is increasing faster than the rate at which you are consuming data, your data may well expire in the buffer before it is delivered to you.

Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request