← richiebachala.com

AI-Based Data Engineering · Chapter 3

Five Workflow Patterns

Each pattern solves a different class of data engineering problem. Start with the simplest that works.

← Pattern decision tree
1 Prompt Chaining
Metadata Draft Business YAML
Use when: Steps depend sequentially on each other's output; number of steps is fixed
Avoid when: Steps are independent — use Parallelization instead
Cost: Low–medium
OpsPulse: Column documentation pipeline
See animation →
2 Routing
Classifier Null-Rate Handler Schema-Drift Handler Distribution Handler
Use when: Different input types need different prompts or tools
Avoid when: One general-purpose prompt handles all inputs adequately
Cost: Low
OpsPulse: Anomaly triage dispatcher
See animation →
3 Parallelization
Input Column 1 Column 2 Column 3 Output
Use when: Input splits into independent chunks, or multiple runs build confidence
Avoid when: Steps depend on each other's output
Cost: Low wall-clock
OpsPulse: 60 columns described concurrently
See animation →
4 Orchestrator-Workers
Orchestrator Worker 1 Worker 2 Worker 3
Use when: Heterogeneous subtasks where decomposition itself requires judgment
Avoid when: Subtasks are identical — use Parallelization instead
Cost: Medium–high
OpsPulse: Schema migration impact assessment
See animation →
5 Evaluator-Optimizer use sparingly
Generator Output Evaluator
Use when: Quality is critical AND first-pass output is reliably insufficient
Avoid when: A well-crafted single-pass prompt meets the quality bar — try that first
Cost: High — reserve for quality-critical tasks
OpsPulse: dbt data_tests generation loop
See animation →