Set the Retention Period in Kafka using the bitnami Helm Chart

Permanently update your helm deployment using the extraConfigs property

Kafka is made to re-play messages. This is relevant if, e.g., a consumer disconnects temporarily.

The default setting for storing messages is 1 week.

So we’re sending 150 messages per second from one data source and experienced memory and performance issues. If those messages are not processed within 5 minutes, they are worthless.

Thus, we reduce the retention period, i.e., the time messages are stored. Whether this improves performance (our cluster is quite small in size so we might experience at least some improvement) we will see after the weekend.

The issue I faced now was how to properly set the retention.ms value given a helm-chart deployment. Setting the value via the CLI was useless as anytime the pod restarts, the setting would be gone.

Setting the values.yaml file for a proper helm deployment with updated retention values.

We’re using this template.

To add retention.ms values, you have multiple options.

1. per topic. In this case, you browse to the topics-section under provisioning in the values.yaml. In the example you see that we set the retention.ms value to 10000ms. This value overrides all global settings. So if you have a specific topic that you want to control individually (and permamently), you need to update the values.yaml file like this.

provisioning:  
  topics:
      - name: provisioned_test_input_topic
        partitions: 1
        replicationFactor: 1
        # https://kafka.apache.org/documentation/#topicconfigs
        config:
          max.message.bytes: 64000
      - name: provisioned_test_output_topic
        partitions: 1
        replicationFactor: 1
        # https://kafka.apache.org/documentation/#topicconfigs
        config:
          max.message.bytes: 64000
          retention.ms: 100000

2. globally. This was our goal as we have a lot of topics, also sometimes new ones, and want to make sure our whole setup stays slim and neat.

Here, you do not go to the broker section in the values.yaml file. You rather set the value at the very top under the global extraConfig property.

There is an extraConfig property under the broker-section. However, this does not set the retention properly.

config: ""
existingConfigmap: ""
## @param extraConfig Additional configuration to be appended at the end of the generated Kafka configuration file.
##
extraConfig: |-
    log.retention.ms=600000

Please not the string-notation and that on topic level, you omit the “log.”

According to the release notes, you need to set retention values in the extraConfig property.

This value is pulled whenever no topic-individual value is set.

Verification

To verify if your setup works, you can run:

bin/kafka-configs.sh --bootstrap-server path-to-your-server:port --entity-type topics --entity-name provisioned_test_output_topic --describe

The output tells you:

 retention.ms=100000 sensitive=false synonyms={DYNAMIC_TOPIC_CONFIG:retention.ms=100000, STATIC_BROKER_CONFIG:log.retention.ms=600000, DEFAULT_CONFIG:log.retention.hours=168}

This tells you that 10000ms will be used for this topic, overwriting the default value of 600000 (so our setting worked) and even further overruling the default value of 1 week.

However, if you did not set topic-individual values, you can further verify your generated config files.

Download the file from your cluster using kubctl. Then verify the server.properties file. There you should find the entry at the very bottom.

# Custom Kafka Configuration
log.retention.ms=600000

In a similar way, you could permanently set retention.bytes if you rather want to control size than time.

Hope this helps. Happy streaming.

Henriette