Dory Discards Issue

[20Rishabh20]

Getting a lot of dory discards with error logs:-

1. Discarding message due to rate limit
2. Discarding message because failed delivery attempt limit reached
3. Discarding message due to buffer space cap

Current Configuration is:-

            <!-- Set 500 milliseconds max batching delay.  Here, you must
                 specify integer values directly rather than using syntax
                 such as "10k".  Specifying "disable" for this setting is
                 not recommended, since it can cause Dory to batch a
                 message indefinitely if not enough other messages arrive
                 to exceed one of the other two batching thresholds below.
              -->
            <time value="500" />

            <!-- No limit on message count.  You can specify a value here
                 such as "10" or "10k".  A value of "10k" is interpreted as
                 (10 * 1024) messages. -->
            <messages value="disable" />

            <!-- Somewhat arbitrary upper bound on batch data size.  As
                 above, "256k" is interpreted as (256 * 1024) bytes.  A
                 simple integer value such as "10000" can also be
                 specified.  This value applies to the actual message
                 content (keys and values).  The total size of a batch will
                 be a bit larger due to header overhead.  The size of an
                 empty message is counted as 1 byte.  This prevents
                 batching an unbounded number of empty messages in the case
                 where no limit is placed on batching delay or message
                 count. -->
            <bytes value="256k" />
        </config>

        <config name="default_latency">
            <time value="10000" />
            <messages value="disable" />
            <bytes value="256k" />
        </config>
    </namedConfigs>

    <!-- Somewhat arbitrary upper bound on produce request size.  As above,
         a value such as "100k" is interpreted as (100 * 1024) bytes.  You
         can also specify simple integer values such as "100000" directly.
         Here, you can not specify "disable".  A nonnegative integer value
         must be specified.  This value applies to the actual message
         content (keys and values).  The total size of a produce request
         will be a bit larger due to header overhead. -->
    <produceRequestDataLimit value="1024k" />

    <!-- This value should be exactly the same as the message.max.bytes
         value in the Kafka broker configuration.  A larger value will
         cause Kafka to send MessageSizeTooLarge error ACKs to Dory for
         large compressed message sets, which will cause Dory to discard
         them.  A smaller value will not cause data loss, but will
         unnecessarily restrict the size of a compressed message set.  As
         above, you can supply a value such as "1024k".  Specifying
         "disable" here is not permitted. -->
    <messageMaxBytes value="1000000" />

    <!-- This specifies the configuration for combined topics batching
         (where a single batch may contain multiple topics).  Setting
         "enable" to false is strongly discouraged due to performance
         considerations.
      -->
    <combinedTopics enable="true" config="default_latency" />

    <!-- This specifies how batching is handled for topics not specified in
         "topicConfigs" below.  Allowed values for the "action" attribute
         are as follows:

             "perTopic": This setting will cause each topic not listed in
                 "topicConfigs" to be batched individually on a per-topic
                 basis, with the "config" attribute specifying a batching
                 configuration from "namedConfigs" above.

             "combinedTopics": This setting will cause all topics not
                 listed in "topicConfigs" to be batched together in mixed
                 topic batches.  In this case, "config" must be either
                 missing or set to the empty string, and the batching
                 configuration is determined by the "combinedTopics"
                 element above.  This is the setting that most people will
                 want.

             "disable": This setting will cause batching to be disabled for
                 all topics not listed in "topicConfigs".  In this case,
                 the "config" attribute is optional and ignored.  This
                 setting is strongly discouraged due to performance
                 considerations.
      -->
    <defaultTopic action="combinedTopics" config="" />

    <topicConfigs>
        <!-- Uncomment and customize the settings in here if you wish to
             have batching configurations that differ on a per-topic basis.

             As above, allowed settings for the "action" attribute are
             "perTopic", "combinedTopics", and "disable".  The "disable"
             setting is strongly discouraged due to performance
             considerations.

        <topic name="low_latency_topic_1" action="perTopic"
               config="low_latency" />
        <topic name="low_latency_topic_2" action="perTopic"
               config="low_latency" />
          -->
    </topicConfigs>
</batching>

<compression>
    <namedConfigs>
        <!-- Don't bother to compress a message set whose total size is
             less than minSize bytes.  As above, a value such as "1k" is
             interpreted as (1 * 1024) bytes.  Here, a value of "disable"
             is not recognized, but you can specify "0".  The value of 128
             below is somewhat arbitrary, and not based on experimental
             data.  Currently the only allowed values for "type" are
             "snappy" and "none".
          -->
        <config name="snappy_config" type="snappy" minSize="128" />

        <!-- "minSize" is ignored (and optional) if type is "none". -->
        <config name="no_compression" type="none" />
    </namedConfigs>

    <!-- This must be an integer value at least 0 and at most 100.  If the
         compressed size of a message set is greater than this percentage
         of the uncompressed size, then Dory sends the data uncompressed,
         so the Kafka brokers don't waste CPU cycles dealing with the
         compression.  The value below is somewhat arbitrary, and not based
         on experimental data. -->
    <sizeThresholdPercent value="75" />

    <!-- This specifies the compression configuration for all topics not
         listed in "topicConfigs" below.
      -->
    <defaultTopic config="snappy_config" />

    <topicConfigs>
        <!-- Uncomment and customize the settings in here if you wish to
             configure compression on a per-topic basis.

        <topic name="no_compression_topic_1" config="no_compression" />
        <topic name="no_compression_topic_2" config="no_compression" />
          -->
    </topicConfigs>
</compression>

<topicRateLimiting>
    <namedConfigs>
        <!-- This configuration specifies that all messages should be
             discarded. -->
        <config name="zero" interval="1" maxCount="0" />

        <!-- This configuration specifies no rate limit (i.e. don't discard
             any messages regardless of their arrival rate). -->
        <config name="infinity" interval="1" maxCount="unlimited" />

        <!-- This configuration specifies a limit of at most 1000 messages
             every 10000 milliseconds.  Messages that would exceed this
             limit are discarded. -->
        <config name="config1" interval="10000" maxCount="1000" />

        <!-- This configuration specifies a limit of at most (4 * 1024)
             messages every 15000 milliseconds.  Messages that would exceed
             this limit are discarded. -->
        <config name="config2" interval="15000" maxCount="4k" />
    </namedConfigs>

    <!-- This specifies a default configuration for topics not listed in
         <topicConfigs> below.  Each such topic is rate-limited
         individually.  In other words, with this configuration, topic
         "topic_a" would be allowed 1000 messages every 10000 milliseconds,
         and "topic_b" would also be allowed 1000 messages every 10000
         milliseconds. -->
    <defaultTopic config="infinity" />

    <topicConfigs>
        <!-- Rate limit configurations for individual topics go here. -->
        <topic name="topic1" config="zero" />
        <topic name="topic2" config="infinity" />
        <topic name="topic3" config="config1" />
        <topic name="topic4" config="config2" />
    </topicConfigs>
</topicRateLimiting>

<initialBrokers>
    <!-- When Dory starts, it chooses a broker in this list to contact for
         metadata.  If Dory cannot get metadata from the host it chooses,
         it tries other hosts until it succeeds.  Once Dory successfully
         gets metadata, the broker list in the metadata determines which
         brokers Dory will connect to for message transmission and future
         metadata requests.  Specifying a single host is ok, but multiple
         hosts are recommended to guard against the case where a single
         specified host is down.
      -->
<broker host="ts005i" port="9092" />

<broker host="ts008i" port="9092" />
</initialBrokers>

Current Dory Version:- 2.0.5
OS version:- centos 6.10
Please help with the issue.

[dspeterson]

Hi 20Rishabh20,

I'm currently on vacation and won't have easy access to a computer until next weekend, so until then my ability to provide help is limited. The per-topic message rate limiting feature was implemented quite a while ago, and the intent was to prevent buggy clients from overwhelming the Kafka brokers with too many messages. I believe recent versions of Kafka have pretty good built-in throttling functionality to deal with this sort of problem, so the rate limiting feature is probably not as useful as it once it was. To stop the rate limit discards, you can loosen or eliminate the rate limits in the config file. For instance, to eliminate them completely, you can edit the topic configs under the section to look like this:

    <topicConfigs>
    </topicConfigs>

or like this:

    <topicConfigs>
        <!-- Rate limit configurations for individual topics go here. -->
        <topic name="topic1" config="infinity" />
        <topic name="topic2" config="infinity" />
        <topic name="topic3" config="infinity" />
        <topic name="topic4" config="infinity" />
    </topicConfigs>

The discards due to failed delivery attempt limit reached and buffer space cap indicate some issue interacting with the Kafka brokers. Does Dory's logfile have any error messages that may indicate why the failed delivery attempts are occurring? You can post log excerpts here and I'll take a look. Another useful diagnostic tool is the counters Dory provides through its web interface as documented here. The failed delivery attempts should cause one or more counters to increment, indicating why they are occurring. You can match up the counter increments with specific places in the code where the counters are being incremented to get more information on what's going on. If you want, you can post counter output here and I'll take a look. I hope this helps!

Dave