/Database

Teréga Replaced Its Legacy Data Historian with InfluxDB, AWS, And IO-Base

- Jessica Wachtel tl;dr: Teréga, a French gas company, faced challenges with outdated IT systems. Recognizing a gap in available cloud-native data historians, they turned to InfluxDB. With InfluxDB, they developed Indabox for efficient data collection and IO-Base, hosted on AWS, for robust data storage. This InfluxDB-centric solution significantly modernized Teréga's IT landscape.

featured in #446

This Is How Quora Shards MySQL To Handle 13+ Terabytes

tl;dr: With data storage requirements in the tens of terabytes and 100,000 queries per second, Quora chose MySQL for its improved read performance. To manage rapid data growth and high write queries, Quora implemented both vertical and horizontal sharding techniques. Vertical sharding involves moving different tables to different servers, improving write scalability. Horizontal sharding, on the other hand, splits a large table into multiple smaller tables. Quora opted to build its sharding solution instead of using third-party service for low latency and easy reuse of existing logic.

featured in #445

Fuzz Testing Is the Best Thing To Happen To Our Application Tests

- Andrei Pechkurov tl;dr: The team at QuestDB faced challenges with segfaults, data corruption, and concurrency bugs. To address these, the team implemented fuzz testing, an automated software testing technique that provides invalid or unexpected data to a program to monitor for exceptions. This article details the process of introducing fuzz testing, revealing critical issues and leading to more robust database performance. The team also collaborated with SQLancer, a tool for testing SQL Database Management Systems, to uncover issues in their SQL engine.

featured in #441

NetApp Leverages Time Series Data for Real-Time Resource Trending and Alerting

tl;dr: NetApp, a leader in cloud data services, storage systems and software, uses time series data for real-time resource trending, SLO/SLI calculations, and alerting. Their SRE team identifies trends in resource consumption for critical Linux servers, DB monitoring, and custom resource monitoring. The team appreciates the high ingest, tool integration, and performance of InfluxDB, their chosen time series database. They also value its integration with Grafana for dashboards and Slack for global team communication.

featured in #440

A Strategic Approach To Replacing Data Historians

- Jason Myers tl;dr: Transitioning from legacy data historians to modern technologies in IoT / OT stacks can be achieved strategically. This involves automating manual processes, managing changes in legacy technology, and considering new tools during equipment upgrades and operational growth. Start small, scale sensibly.

featured in #436

Joins 13 Ways

- Justin Jaffray tl;dr: “Relational inner joins are really common in the world of databases, and one weird thing about them is that it seems like everyone has a different idea of what they are. In this post I’ve aggregated a bunch of different definitions, ways of thinking about them, and ways of implementing them that will hopefully be interesting. They’re not without redundancy, some of them are arguably the same, but I think they’re all interesting perspectives nonetheless.”

featured in #429

What Is A Vector Database?

- Roie Schwaber-Cohen tl;dr: This post reviews key aspects of a vector database — how it works, algorithms it uses, and the additional features that make it operationally ready for production scenarios.

featured in #422

You Don't Always Need Indexes

- Jeff Kaufman tl;dr: “Sometimes you have a lot of data, and one approach to support quick searches is pre-processing it to build an index so a search can involve only looking at a small fraction of the total data. The threshold at which it's worth switching to indexing, though, might be higher than you'd guess.” Jeff illustrates cases where full scans were better engineering choices.

featured in #418

Is 20M Of Rows Still A Valid Soft Limit Of MySQL Table In 2023?

- Yisheng Gong tl;dr: “There’s rumor around the internet that we should avoid having > 20M rows in a single MySQL table. Otherwise, the table’s performance will be downgraded, you will find SQL query much slower than usual when it’s above the soft limit. These judgements were made on HDD many years ago. I’m wondering if it’s still true for MySQL on SSD in 2023, and if true, why is that?”

featured in #417

Factors To Consider In Database Selection

- Alex Xu tl;dr: Alex examines key factors that influence the decision-making process of database selection such as scalability, performance, data consistency.

featured in #409