/Scale

Scaling Causal's Spreadsheet Engine From Thousands To Billions Of Cells: From Maps To Arrays

- Simon Hørup Eskildsen tl;dr: "How can we scale the calculation engine 100x, from millions to billions of cells? By moving from maps to arrays. That may seem like an awfully pedestrian observation, but it certainly wasn’t obvious to us at the outset that this was the crux of the problem!" 

featured in #332

How Live Is Your Stream? Measure Live Latency At Scale With Mux Data

tl;dr: Beta test the new HLS Live Stream Latency metric in Mux Data for free! Understand the live streaming experience, and find opportunities to improve.

featured in #316

How Live Is Your Stream? Measure Live Latency At Scale With Mux Data

tl;dr: Beta test the new HLS Live Stream Latency metric in Mux Data for free! Understand the live streaming experience, and find opportunities to improve.

featured in #308

Bottleneck #02: Talent

- Tim Cochran Roni Smith tl;dr: Common tech debt bottlenecks by companies entering growth. The authors cover the common signs you are approaching a scaling bottleneck i.e. frustration from employees, stretching to hit deadlines, dependency on people and more. And strategies on how to get out of the bottleneck i.e. Use your technology and innovation as a hiring differentiator, hire more T-shaped technologists than specialists, utilize non-senior developers, and more. 

featured in #304

On Building Scalable Systems

- Kislay Verma tl;dr: "Scalability is the idea that a system should be able to handle an increase in workload by employing more computing resources without significant changes to its design." The key axes of scalability are latency, throughput & capacity. Kislay discusses each, as well as how to quantify scalability, and more.

featured in #303

Architecture Patterns: Caching

- Kislay Verma tl;dr: "Depending on the type of application, the type of data, and the expectation of failure, there are several strategies that can be applied for caching." Kislay discusses the levels in a systems architecture where caching commonly occurs and various caching strategies, such as read through, write through, write behind. 

featured in #302

Why We Switched Our Data Orchestration Service

- Guillaume Perchais tl;dr: "Within Spotify, we run 20,000 batch data pipelines defined in 1,000+ repositories, owned by 300+ teams — daily. The majority of our pipelines rely on two tools: Luigi (Python) and Flo (Java). The data orchestration team decided to move away from these tools, and in this post, the team details why the decision was made, and the journey they took to make the transition."

featured in #301

Bottleneck #01

- Tim Cochran Carl Nygard tl;dr: This first post in a series studies common technical debt bottlenecks by companies entering growth. The authors look at causes of tech debt bottlenecks, various types of debt, signs you are approaching this bottleneck, and strategies to get out of it, include setting a quality bar, damage limitation to the business, collaboration with product, and more.

featured in #299

Four Rules for Scaling

- Varun Srinivasan tl;dr: Coinbase's Director of Eng: (1) Shape your org chart so your product reflects what you want to ship. (2) Push part of the strategic decision-making process down the org to gain speed and flexibility. (3) Think in 3D to visualize what's missing i.e, think about how the work gets done, not just who does it. (4) Put people over org structures and leverage emerging leaders.

featured in #298

The Four Innovation Phases Of Netflix’s Trillions Scale Real-time Data Infrastructure

- Zhenzhong Xu tl;dr: "I hope this post will help platform engineers develop their cloud-native, self-serve streaming data platforms and scale use cases across many business functions (not necessarily from our success but maybe more from our failures)."

featured in #294