feat(source): use transformers in pod informers to reduce memory footprint #5596

valerian-roche · 2025-07-01T23:12:00Z

What does it do ?

This PR adds transformers to the pod informers in the pod and service sources to reduce the resulting memory footprint.

Motivation

As described in the issue, the pod informers currently lead to a large memory usage of the external-dns pods. As the controller only requires a few specific fields we can leverage transformers in the informers to greatly reduce the footprint.
In our environment this reduces the average memory usage by ~10 times, and once using watch list (disabled by default) and the next client-go release, we now have a peak memory usage 10 times smaller, greatly reducing our controller footprint.

This PR does not enable watchlist (but I'll raise a PR to optionally do so) as it is currently not really valuable without a fix in the next release of client-go.

More

Yes, this PR title follows Conventional Commits
Yes, I added unit tests
~~Yes, I updated end user documentation accordingly~~ Not applicable, this change is purely technical within the informer

Refs: #5595

k8s-ci-robot · 2025-07-01T23:12:09Z

Welcome @valerian-roche!

It looks like this is your first PR to kubernetes-sigs/external-dns 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/external-dns has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2025-07-01T23:12:09Z

Hi @valerian-roche. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

ivankatliarchuk · 2025-07-02T05:29:19Z

/ok-to-test

ivankatliarchuk

There is a PR open #5583. Not yet merged, as w8 for confirmation.

I think is going to be quite nice to move transfromers under source/informers/transformers.go file. We could identify some cool patterns and parity acros sources.

source/annotations/annotations.go

ivankatliarchuk · 2025-07-02T05:41:22Z

source/pod.go

I wonder how this is going to work with FQDN template, we could do some pre-processing and attach it as a field ?

I'll take a look at what is needed here.

Looking at the code I think the informer can be fully split:

when fqdnTemplate is unset this change will be used

when fqdnTemplate is set we actually do not need the pods at all, only informer events and we can drop even more content. I will update this PR to not use the transformer when the template is set for now.

valerian-roche · 2025-07-02T14:43:08Z

There is a PR open #5583. Not yet merged, as w8 for confirmation.

I think is going to be quite nice to move transfromers under source/informers/transformers.go file. We could identify some cool patterns and parity acros sources.

The shared transformers are an interesting question as each source may use different parts of the pod. I can add one that's the superset (it will still drop a lot of the blueprint) but I'm unsure how this will evolve long term

ivankatliarchuk

Probably missing evidences that there is a memory reduction. Rest looks quite nice, going to apply similar transfomers elsewhere soon

ivankatliarchuk · 2025-07-08T08:18:36Z

source/pod.go

Do we need annotation kubectl.kubernetes.io/last-applied-configuration? Just a question, not a big deal

Good remark, we could at least drop this one. We don't use it internally so had no impact but definitely a potential memory horder for people using apply.
Will add this

ivankatliarchuk · 2025-07-08T08:23:25Z

source/pod.go

Worth to rebase with master branch. This most likely not going to behave correctly in cases when

annotationFilter is present

labelSelector is present

Let me rebase and assess what we need to store

valerian-roche · 2025-07-31T21:16:37Z

Sorry for the delay, I actually discovered a missed detail when testing the upstream client-go that is needed for this, which led me to prioritize this fix first.

Probably missing evidences that there is a memory reduction.
This PR by itself will improve "average" memory usage over time but not peak memory usage until the fix above is merged in and external-dns go.mod is updating k8s.io/client-go to a version incorporating it (which will hopefully make it for the 0.34 versions).

From internal forks, those are examples of gains with and without the gains from client-go:
without

solid is actual memory usage, and we can see a major drop when transformers are added
dashes is maximum memory usage on the container lifetime (and therefore the memory limit required), which does not change due to client-go ignoring the transformer during the initial list

with (different cluster hence initial state is worse)

solid also shows an improvement. It's more efficient in this cluster due to a lot of pods with very long templates, but same transformer, showing that the gains will vary significantly based on pod average footprint
dashes is maximum memory usage and is now barely higher than average state as client-go applies the transformer on each incoming object as it handles them (this relies on watchlist being active on the cluster, otherwise it will behave as above)

ivankatliarchuk · 2025-08-02T08:03:03Z

source/service.go

How this will behave in a scenario

someone decided to execute kubectl patch pod .... and modify/change annotaions, labels or some other value of the pod?

The informer will receive a new object that will be transformed from scratch in this case. Client-go does not alter objects after them being passed to informers

mloiseleur · 2025-08-05T12:26:40Z

@valerian-roche It looks really great. Do you think you can rebase this PR ?

ivankatliarchuk · 2025-08-05T13:08:49Z

I think this PR will require

The pr to be merged [client-go #1415] Embed proper interface in TransformingStore to ensure DeltaFIFO and RealFIFO are implementing it kubernetes/kubernetes#133263
new release for https://github.com/kubernetes/kubernetes

I think this is a fantastic solution. I'm already looking at how we can apply these improvements to our other sources as well.

Add a transformer to the pods informer of the pod and service sources. Refs: kubernetes-sigs#5595 Signed-off-by: Valerian Roche <[email protected]>

valerian-roche · 2025-08-05T14:27:50Z

I think this PR will require

The pr to be merged [client-go #1415] Embed proper interface in TransformingStore to ensure DeltaFIFO and RealFIFO are implementing it kubernetes/kubernetes#133263

new release for https://github.com/kubernetes/kubernetes

I think this is a fantastic solution. I'm already looking at how we can apply these improvements to our other sources as well.

I rebased the PR. This should be mergeable without the other PR from client-go being available, even if it will bring limited benefits for now

ivankatliarchuk · 2025-08-05T15:01:31Z

/lgtm

source/pod_test.go

Co-authored-by: Michel Loiseleur <[email protected]>

ivankatliarchuk

/lgtm
/approve

k8s-ci-robot · 2025-08-09T08:50:21Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ivankatliarchuk

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [ivankatliarchuk]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

…print (kubernetes-sigs#5596) * feat: use transformers in pod informers to reduce memory footprint Add a transformer to the pods informer of the pod and service sources. Refs: kubernetes-sigs#5595 Signed-off-by: Valerian Roche <[email protected]> * Do not use transformer when fqdnTemplate is set * Update source/pod_test.go Co-authored-by: Michel Loiseleur <[email protected]> --------- Signed-off-by: Valerian Roche <[email protected]> Co-authored-by: Michel Loiseleur <[email protected]>

k8s-ci-robot requested a review from ivankatliarchuk July 1, 2025 23:12

k8s-ci-robot added the source label Jul 1, 2025

k8s-ci-robot requested a review from mloiseleur July 1, 2025 23:12

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jul 1, 2025

k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jul 1, 2025

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jul 2, 2025

ivankatliarchuk suggested changes Jul 2, 2025

View reviewed changes

k8s-ci-robot assigned ivankatliarchuk Jul 2, 2025

ivankatliarchuk mentioned this pull request Jul 5, 2025

fix(source/service): disable node informer when not required #5613

Merged

3 tasks

valerian-roche force-pushed the vr/transformer branch from 8638370 to c6772e5 Compare July 8, 2025 01:57

ivankatliarchuk reviewed Jul 8, 2025

View reviewed changes

ivankatliarchuk suggested changes Jul 8, 2025

View reviewed changes

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 9, 2025

ivankatliarchuk reviewed Aug 2, 2025

View reviewed changes

valerian-roche added 2 commits August 5, 2025 10:24

feat: use transformers in pod informers to reduce memory footprint

59d5415

Add a transformer to the pods informer of the pod and service sources. Refs: kubernetes-sigs#5595 Signed-off-by: Valerian Roche <[email protected]>

Do not use transformer when fqdnTemplate is set

2ed7bb3

valerian-roche force-pushed the vr/transformer branch from c6772e5 to 2ed7bb3 Compare August 5, 2025 14:27

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 5, 2025

valerian-roche requested a review from ivankatliarchuk August 5, 2025 14:58

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 5, 2025

mloiseleur reviewed Aug 5, 2025

View reviewed changes

source/pod_test.go Outdated Show resolved Hide resolved

mloiseleur changed the title ~~feat: use transformers in pod informers to reduce memory footprint~~ feat(source): use transformers in pod informers to reduce memory footprint Aug 5, 2025

ivankatliarchuk mentioned this pull request Aug 8, 2025

refactor(source/istio): add transformers #5728

Merged

3 tasks

Update source/pod_test.go

409a5a7

Co-authored-by: Michel Loiseleur <[email protected]>

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 8, 2025

valerian-roche requested a review from mloiseleur August 8, 2025 21:54

ivankatliarchuk approved these changes Aug 9, 2025

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 9, 2025

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 9, 2025

k8s-ci-robot merged commit 111df9f into kubernetes-sigs:master Aug 9, 2025
14 checks passed

feat(source): use transformers in pod informers to reduce memory footprint #5596

feat(source): use transformers in pod informers to reduce memory footprint #5596

Uh oh!

Conversation

valerian-roche commented Jul 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does it do ?

Motivation

More

Uh oh!

k8s-ci-robot commented Jul 1, 2025

Uh oh!

k8s-ci-robot commented Jul 1, 2025

Uh oh!

ivankatliarchuk commented Jul 2, 2025

Uh oh!

ivankatliarchuk left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

valerian-roche Jul 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

valerian-roche commented Jul 2, 2025

Uh oh!

ivankatliarchuk left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

valerian-roche commented Jul 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mloiseleur commented Aug 5, 2025

Uh oh!

ivankatliarchuk commented Aug 5, 2025

Uh oh!

valerian-roche commented Aug 5, 2025

Uh oh!

ivankatliarchuk commented Aug 5, 2025

Uh oh!

Uh oh!

ivankatliarchuk left a comment

Choose a reason for hiding this comment

Uh oh!

k8s-ci-robot commented Aug 9, 2025

Uh oh!

Uh oh!

Uh oh!

valerian-roche commented Jul 1, 2025 •

edited

Loading

valerian-roche Jul 7, 2025 •

edited

Loading

valerian-roche commented Jul 31, 2025 •

edited

Loading