​Task ​Runner ​Use ​Cases

Use cases that can benefit from Task Runners.

Overview

With Task Runners, you can offload compute-intensive tasks to dedicated workers. This capability is particularly valuable in scenarios where tasks require significant CPU, memory, or GPU resources. Below are example use cases where Task Runners can provide significant value.

Data Analytics

• Data Warehousing Loads: you can increase resources for heavy data ingestion phases, and decrease afterwards
• Batch Processing Jobs: scheduled (e.g. nightly) batch data transformation and data cleaning jobs may require significantly more resources than usual (e.g. hourly) jobs
• Machine Learning & AI: model training and inference are resource-intensive, especially when working with large datasets or complex models; Task Runners can dynamically allocate as much GPUs, CPUs and memory to these tasks as needed
• Large-scale Backfills: reprocessing some data during backfill requires significant resources for a short period of time. Task Runners can be used to dynamically allocate more resources than usual to complete the backfill and then scale down to save costs
• Batch Reporting: sometimes you may need to generate complex reports from large datasets at scheduled times, requiring increased CPU and memory.

Financial Services

• Financial Data Processing: scaling up resources during trading hours when data volume is high
• Financial Risk Simulations: scaling up resources to run a large number of simulations to assess risk
• End-of-day Stock Market Analysis: allocating more CPUs for analyzing end-of-day stock prices in batch mode
• Fraud Detection: processing large volumes of transaction data to detect fraud patterns.

Healthcare & Life Sciences

• Genomic Data Processing: sequencing or analyzing large genomic datasets
• Geospatial Data Processing: processing satellite images or GIS data in batches, which can be highly memory-intensive
• Healthcare Data Analysis: scaling during health crises or outbreaks when rapid data analysis is critical for decision-making
• Weather Forecasting Models: dynamically allocating more resources during severe weather events when models must process more data points quickly.

Software Development

• Database Migrations: scaling up during database migration tasks that require transferring large volumes of data
• Application Migrations: scaling up when switching from one application to another, which may require significant resources for moving and transforming data
• Backups: increasing resources during backup operations to ensure they are completed within a predefined maintenance window
• Video Transcoding Jobs: allocating enough CPU and memory during bulk video transcoding operations
• Image Processing Tasks: increasing GPU resources for high-throughput image processing, such as batch image resizing
• IoT Device Data Management: allocating more resources as device count and data volume grow.