Most SE leaders came up the SE track. I spent a decade as the enterprise customer first — building petabyte-scale data platforms at Sherwin-Williams and Weatherford — before leading SE and SA teams at YugaByte and Snowflake. I understand what buyers actually care about because I’ve been the buyer. That loop changes how I build teams, run technical deals, and design go-to-market.
A 5-part technical series published in the Snowflake Builders Blog covering the architecture, patterns, and AI integration for running Snowflake and Databricks together — without migration, without duplication. Built from real-world enterprise deployments across Financial Services, Philanthropy, Retail, and Technology verticals.
A detailed technical comparison of governance capabilities: data discovery, lineage, access control, and compliance across both platforms.
Read Article → Snowflake Blog · 2026How Snowflake Horizon and Unity Catalog handle open table format governance, catalog interoperability, and multi-engine reads.
Read Article →How I built the first enterprise-wide DG&DQ program at Weatherford — 100+ stakeholder interviews, NIST/GDPR/ISO alignment, Executive Council ratification.
Weatherford International, 2024Benchmarking Azure Premium SSD, Premium SSD v2, and Ultra SSD under TPC-C workloads with YugabyteDB — practical guidance on disk type selection.
Published Feb 1, 2024Five takeaways from Money 20/20 USA — AI ethics in banking, quantum computing's threat to encryption, embedded finance, and the balance of automation vs. human oversight.
Published Nov 14, 202310 rules for effective dimensional modeling in the cloud era — from star schemas to semantic layers.
Read → Medium · 2022Data-driven comparison of the three dominant cloud data warehouses across performance, cost, and enterprise fit.
Read → Medium · 2022YugabyteDB's architecture and how distributed SQL enables data durability and global consistency at scale.
Read → Medium · Towards Data Science · 2021How to structure, scale, and operate a modern data engineering organization — frameworks, team design, and operating models drawn from building a 70+ person global DE team at Sherwin-Williams.
Read on Medium → Fivetran · Aug 2021"To build all of the data sources and models would have taken three months. Fivetran cut that to four days." Feature by Theo Hopkinson on the Sherwin-Williams DaaS transformation.
Read Feature → Medium · 2018Strengths and weaknesses of the two dominant BI tools — focused on enterprise analytics use cases and business size fit.
Read →Open to conversations about enterprise AI architecture, Snowflake Interoperability, building high-performing SE teams, or speaking opportunities across Retail, CPG, Energy, and Finance.
Most data engineering books stop at moving bytes. This one starts where the hard problems actually live: coordinating meaning, managing change, and maintaining trust as the business evolves faster than the documentation can keep up — and AI agents are now consuming your data at scale.
AI-Based Data Engineering builds a complete platform from the ground up — context pipelines that give models what they need to reason correctly, structured generation patterns that constrain what models can produce, and evaluation frameworks that catch failures before they reach production. Every pattern is grounded in OpsPulse, a single running case study that threads through all 14 chapters so you see every concept applied to the same real system.
The platform starts at an AI-readiness score of 4 out of 18 and ends at 18 out of 18 — one dimension at a time.
"OpsPulse starts in a realistic initial state: incomplete documentation, divergent metric definitions, no golden datasets, and an AI-readiness score of 4 out of 18. By the end of the book, you will have built the context infrastructure, agentic workflows, and evaluation systems that turn it into a production-grade AI data platform."
OpsPulse is a global operational analytics initiative combining ERP, CRM, IoT telemetry, and support signals. Four teams each have their own definition of "active customer" — producing four different numbers from the same source data:
Every chapter of the book resolves one more dimension of this problem — until the platform can answer "what is our current active customer count?" with a single, traceable, reproducible number.
Richie spent a decade as the enterprise customer — building petabyte-scale data platforms at Sherwin-Williams and Weatherford International — before moving to SE and SA leadership at YugaByte and Snowflake. He built Snowflake's Interoperability offering, co-created Blueprint Manager, and has led AI production deployments across Financial Services, Healthcare, and Retail. He writes at the Snowflake Builders Blog and speaks on enterprise AI architecture and data platform design.
Six dimensions. Eighteen points. One arc from raw data platform to production-grade AI infrastructure. Every point is a concrete, measurable property of your data estate — not a maturity label.
Data platform can support human workflows but AI agents will fail silently — wrong answers, unauditable outputs, no quality gates.
Selective AI use cases are viable. Specific dimensions are blocking production; targeted fixes unlock the next capability tier.
AI agents can reason correctly, produce auditable outputs, operate within governance boundaries, and be evaluated for correctness.
Each dimension is scored 0–3 (None / Partial / Mostly / Complete). The six are independent — a weakness in any one undermines the others, because AI systems expose all six failure modes simultaneously.
data_tests: or equivalent quality assertions. Quality must be a gate before AI inference, not a monitor after the fact.OpsPulse starts at 4/18 — a realistic baseline for most enterprise data platforms. Each chapter closes one or more dimension gaps. The score is the exact state of the platform at the end of that chapter.
Shows each dimension's score (0–3) at the end of each chapter. Darker blue = higher score. Scroll horizontally on small screens.
| Dimension | Ch1 | Ch2 | Ch3 | Ch4 | Ch5 | Ch6 | Ch7 | Ch8 | Ch9 | Ch10 | Ch11 | Ch12 | Ch13 | Ch14 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| D1 Schema Freshness | 1 | 2 | 2 | 2 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 |
| D2 Semantic Coverage | 0 | 1 | 1 | 2 | 2 | 2 | 2 | 3 | 3 | 3 | 3 | 3 | 3 | 3 |
| D3 Test Coverage | 1 | 1 | 1 | 2 | 2 | 2 | 2 | 2 | 3 | 3 | 3 | 3 | 3 | 3 |
| D4 Eval Readiness | 1 | 1 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 3 | 3 | 3 | 3 |
| D5 Governance Scope | 1 | 1 | 1 | 1 | 1 | 1 | 2 | 2 | 2 | 2 | 2 | 3 | 3 | 3 |
| D6 Lineage Coverage | 0 | 2 | 2 | 2 | 2 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 |
| Total | 4 | 8 | 9 | 11 | 12 | 13 | 14 | 15 | 16 | 16 | 17 | 18 | 18 | 18 |
* Scores represent approximate OpsPulse state at chapter end. Intermediate chapters hold scores while building enabling infrastructure.
Select the level (0–3) that best describes your current data platform for each dimension. Your score updates live. This is the same 18-point checklist used in AI-Based Data Engineering Chapter 1.
Each chapter closes specific dimension gaps in a deliberate order — the OpsPulse arc takes you from wherever you are to 18/18, one concrete pattern at a time.