​Task ​Runner ​Benefits

How Task Runners can help with resource allocation and environment management.

Docker in development, Kubernetes in production

Many Kestra users develop their scripts locally in Docker containers and then run the same code in a production environment as Kubernetes pods. Thanks to the taskRunner property, setting this up is a breeze. Below is an example showing how you can combine pluginDefaults with the taskRunner property to use Docker in the development environment and Kubernetes in production — all without changing anything in your code.

1. Development namespace/tenant/instance:
   yaml

   pluginDefaults:
     - type: io.kestra.plugin.scripts
       values:
         taskRunner:
           type: io.kestra.plugin.scripts.runner.docker.Docker
           pullPolicy: IF_NOT_PRESENT # in dev, only pull the image when needed
           cpu:
             cpus: 1
           memory:
             memory: 512Mi
   

2. Production namespace/tenant/instance:
   yaml

   pluginDefaults:
     - type: io.kestra.plugin.scripts
       values:
         taskRunner:
           type: io.kestra.plugin.kubernetes.runner.Kubernetes
           namespace: company.team
           pullPolicy: ALWAYS # Always pull the latest image in production
           config:
             username: "{{ secret('K8S_USERNAME') }}"
             masterUrl: "{{ secret('K8S_MASTER_URL') }}"
             caCert: "{{ secret('K8S_CA_CERT') }}"
             clientCert: "{{ secret('K8S_CLIENT_CERT') }}"
             clientKey: "{{ secret('K8S_CLIENT_KEY') }}"
           resources: # can be overriden by a specific task if needed
             request: # The resources the container is guaranteed to get
               cpu: "500m" # Request 1/2 of a CPU (500 milliCPU)
               memory: "256Mi" # Request 256 MB of memory
   

Centralized configuration management

The combination of pluginDefaults and taskRunner properties allows you to centrally manage your task runner configuration. For example, you can use pluginDefaults on a namespace level to centrally manage your AWS credentials for the Batch task runner plugin.

pluginDefaults:
  - type: io.kestra.plugin.aws
    values:
      accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
      secretKeyId: "{{ secret('AWS_SECRET_ACCESS_KEY') }}"
      region: "us-east-1"

This configuration will apply to all components of the AWS plugin incl. tasks, triggers and task runners. This includes the io.kestra.plugin.aws.runner.Batch.

Documentation and autocompletion

Each task runner is a plugin with its own icon, documentation, and schema to validate its properties. The built-in code editor in the Kestra UI provides autocompletion and syntax validation for all runner properties, and when you click on the runner's name in the editor, you can see its documentation on the right side of the screen.

Full customization: create your own Task Runner

You can create a custom task runner plugin for your specific environment, build it as a JAR file, and add that file to the plugins directory. Once you restart Kestra, your custom runner plugin will be available on any script task in the system.

You can contribute your custom task runner plugin to the Kestra community, and if you do, we'll be happy to include it on the plugins page.