-
Notifications
You must be signed in to change notification settings - Fork 179
Description
My company (singlestore.com) has an extremely large and diverse kubernetes infrastructure which is managed by Tanka. At the time of this writing, we maintain 1,830 separate Tanka envrionments, comprised of 61 unique environments deployed across 96 Kubernetes clusters.
To avoid mass duplication of environment configuration, we use Tanka inline environments to render the configs for each cluster/environment. Our inline environment code is roughly structured as follows:
- Each Kubernetes cluster's config is stored in its own libsonnet file, which contains parameters such as cluster name, kube api-server, and a set of identifying parameters such as cluster type (what the cluster is used for, i.e prod, staging, control-plane, etc), the CSP the cluster is deployed on (we deploy on EKS, GKE, and AKS), and various feature flags to indicate certain functionaltiy
- When a tanka environment is instantiated, it is configured for all clusters by default, but can be filtered by the identifying parameters outlined above (and many more, omitted for brevity). Individual resources can also be filtered by these parameters.
This approach has served us well, but we have run into some issues that we would like to address:
- It is difficult to automate the deployment of a new cluster, because we have to dynamically insert an
import
call for the cluster's config as Jsonnet has no functionality for dynamically computing imports. - We are starting to deploy inside external/restricted customer environments, and we would rather not ship the configs for all of our clusters when doing this.
To resolve this, I am proposing the addition of a new jsonnet native function that is capable of importing and evaluating all files in a given directory.
This way:
- The addition of a new cluster is as simple as adding the cluster's config file.
- When deploying to restricted environments, we can simply deploy our same code base without the other cluster configs and it will just work without patching existing code
The only downside to this approach is the rendered object doesn't support late binding with directly imported resources, but this is fine for our use case as we can make sure the cluster config resources are rendered separately. I considered doing this with raw json instead, but this is less ideal for us as we have a bunch of standardized attribute names defined in a constants libsonnet file, and we wouldn't be able to use those if our cluster configs are stored as json
I put together a PR for this: #1374 and verified that it works as expected. If this change is accepted, I will happily add the necessary library support to jsonnet-libs
Metadata
Metadata
Assignees
Labels
Type
Projects
Status