DocsEnterprise Edition API ReferenceSecrets Manager

Secrets Manager​Secrets ​Manager

Available on: Enterprise Edition

How to configure the secrets manager.

Kestra integrates with various secret managers to provide secure storage and handling of sensitive data.

Kestra respects your privacy. Therefore, Secrets are persisted externally in a backend of your choice. They are accessed by workers at runtime and stored only in memory.

You can add, modify or delete secrets from the Secrets tab of any given namespace in the Kestra UI, or programmatically via Terraform.

AWS Secret Manager Configuration

In order to use AWS Secret Manager as a secrets backend, make sure that your AWS IAM user or role have the required permissions including CreateSecret, DeleteSecret, DescribeSecret, GetSecretValue, ListSecrets, PutSecretValue, RestoreSecret, TagResource, UpdateSecret.

You can configure the authentication to AWS Cloud in multiple ways:

  • Using accessKeyId, secretKeyId, and region properties.
  • Including a sessionToken alongside the above credentials.
  • If the above properties are not set, Kestra will use the default AWS authentication, in the same way as AWS CLI handles it (i.e. trying to use the AWS CLI profile or the default environment variables AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_DEFAULT_REGION).
yaml
kestra:
  secret:
    type: aws-secret-manager
    aws-secret-manager:
      accessKeyId: mysuperaccesskey
      secretKeyId: mysupersecretkey
      sessionToken: mysupersessiontoken
      region: us-east-1

Additionally, you can configure the following properties:

  • Prefix: kestra.secret.aws-secret-manager.prefix is an optional property to store secrets separately for a different namespace, tenant, or instance. If configured, Kestra will prefix all Secret keys using that prefix. The main purpose of a prefix is to share the same secret manager between multiple Kestra instances.
  • Endpoint Override: kestra.secret.aws-secret-manager.endpointOverride is an optional property to replace AWS default endpoint by an AWS-compatible service such as MinIO.

Azure Key Vault Configuration

To configure Azure Key Vault as your secrets backend, make sure that kestra's user or service principal (clientId) has the necessary permissions, including "Get", "List", "Set", "Delete", "Recover", "Backup", "Restore", "Purge". Then, paste the clientSecret from the Azure portal to the clientSecret property in the configuration below.

yaml
kestra:
  secret:
    type: azure-secret-manager
    azure-key-vault:
      clientSecret:
        tenantId: "id"
        clientId: "id"
        clientSecret: "secret"

If no credentials are set in the above configuration, Kestra will use the default Azure authentication akin to the Azure CLI.

Additionally, you can configure the following properties:

  • Vault Name: kestra.secret.azure-key-vault.vaultName is the name of the Azure Key Vault.
  • Key Vault URI: kestra.secret.azure-key-vault.keyVaultUri is an optional property allowing you to replace the Azure Key Vault name with a full URL.
  • Prefix: kestra.secret.azure-key-vault.prefix is an optional property to store secrets separately for a different namespace, tenant, or instance. If configured, Kestra will prefix all Secret keys using that prefix. The main purpose of a prefix is to share the same secret manager between multiple Kestra instances.

Elasticsearch Configuration

Elasticsearch backend stores secrets with an additional layer of security using AES encryption. You will need to provide a cryptographic key (at least 32 characters-long string) in order to encrypt and decrypt secrets stored in Elasticsearch.

yaml
kestra:
  secret:
    type: elasticsearch
    elasticsearch:
      secret: "a-secure-32-character-minimum-key"

Google Secret Manager Configuration

To leverage Google Secret Manager as your secrets backend, you will need to create a service account with the roles/secretmanager.admin permission. Paste the contents of the service account JSON key file to the serviceAccount property in the configuration below. Alternatively, set the GOOGLE_APPLICATION_CREDENTIALS environment variable to point to the credentials file.

yaml
kestra:
  secret:
    type: google-secret-manager
    google-secret-manager:
      project: gcp-project-id
      serviceAccount: |
        Paste here the contents of the service account JSON key file

If you opt for authentication using the GOOGLE_APPLICATION_CREDENTIALS environment variable, make sure that it's set on all worker nodes. Keep in mind that this authentication method is less secure than using the serviceAccount property.

If no credentials are set in the above configuration, Kestra will use the default Google authentication akin to the Google Cloud SDK.

Additionally, you can configure the kestra.secret.google-secret-manager.prefix property to store secrets separately for a different namespace, tenant, or instance. If configured, Kestra will prefix all Secret keys using that prefix. The main purpose of a prefix is to share the same secret manager between multiple Kestra instances.

Vault Configuration

Kestra also supports the KV Secrets Engine - Version 2 as a secrets backend.

To authenticate Kestra with HashiCorp Vault, you can use Userpass, Token or AppRole Auth Methods, all of which requires full read and write policies. You can optionally change root-engine or namespace (if you use Vault Enterprise).

  1. Here is how you can set up Userpass Auth Method in your Kestra configuration:
yaml
kestra:
  secret:
    type: vault
    vault:
      address: "http://localhostt:8200"
      password:
        user: john
        password: foo
  1. Here is how you can set up Token Auth Method in your Kestra configuration:
yaml
kestra:
  secret:
    type: vault
    vault:
      address: "http://localhostt:8200"
      token:
        token: your-secret-token
  1. Finally, here is how you can set up AppRole Auth Method in your Kestra configuration:
yaml
kestra:
  secret:
    type: vault
    vault:
      address: "http://localhostt:8200"
      app-role:
        path: approle
        role-id: your-role-id
        secret-id: your-secret-id

Additionally, you can configure the following properties:

  • Address: kestra.secret.vault.address is a fully qualified address with scheme and port to your Vault instance.
  • Namespace: kestra.secret.vault.namespace is an optional configuration available on Vault Enterprise Pro allowing you to set a global namespace for the Vault server instance.
  • Engine Version: kestra.secret.vault.engine-version is an optional property allowing you to set the KV Secrets Engine version of the Vault server instance. Default is 2.
  • Root Engine: kestra.secret.vault.root-engine is an optional property allowing you to set the KV Secrets Engine of the Vault server instance. Default is secret.

JDBC (Postgres, H2, MySQL) Secret Manager

Kestra also supports internal secret backend. For the JDBC backend (H2, Postgres or MySQL), the following configuration allows you to set secret backend:

yaml
kestra:
  secret:
    type: jdbc
    jdbc:
      secret: "your-secret-key"

Your secret key should be encrypted. You can find an example of encryption key here.

Elastic Secret Manager

For Kestra instance deployed using the Kafka/Elastic backend, the secret backend can be configured like this:

yaml
kestra:  
  secret:
    type: elasticsearch
    elasticsearch:
      secret: "your-secret-key"

Your secret key should be encrypted. You can find an example of encryption key here.

Was this page helpful?