tl;dr:“Until recently, we’ve been using an in-house orchestration system that’s provided the foundation for Discord’s data analytics over the last five years. As our data organization grew, it became apparent that both self-service and top-notch observability would be key for our ability to effectively scale as a team. The team embraced an ambitious project: to overhaul our data orchestration infrastructure using modern, open-source tools sharing the candid lessons learned along the way, and how our new system is powering over 2000 dbt tables today.”