Unlocking the power of Distributed SQL: Introduction to YugabyteDB

The excitement around distributed SQL is increasing as more and more apps move to the cloud. However, there are also a lot of questions that come up when working with these databases. Questions like: What are the use cases for distributed SQL? How does backup and restore work? How do you handle sharding? What are the deployment considerations? How do you investigate performance issues?

As for the founding story of YugabyteDB, it was created by a group of engineers who had previously built databases at Facebook. Despite the abundance of databases available, they found that none of them were universally applicable. The cloud offers infinite resources, but they found that the existing databases were not able to take full advantage of this. And that is why they created YugabyteDB to be a more universally applicable database solution for cloud-based applications and services.

The best thing about the cloud is the ability to quickly and easily scale resources. Before the cloud, the biggest impediment to software development was the time it took to get machines set up and ready to go. In the past, teams would have to order machines months in advance, even before the application or database was built. This was a huge risk, as there was no guarantee that the machines would actually be used by the application.

With the cloud, resources are readily available in a matter of minutes. However, it’s important to remember that these machines are still prone to failure. To counter this, it’s important to build a database that is always available, regardless of whether it’s a failure or a software upgrade. Additionally, with the cloud, there are now hundreds of data centers available, making it possible to replicate data across different locations with varying SLAs and features.

YugabyteDB was created to take advantage of these cloud-based features. It combines the powerful relational feature set of a database like PostgreSQL with the scalability, always-available, and data replication capabilities of the cloud. This allows developers to retain the power of SQL while also gaining the benefits of the cloud. The architecture is modular and it has a few different layers, one of which is the query processing layer which is similar to AWS Aurora in that it gives users the full power of PostgreSQL and integrates with the other features such as scale on demand, and being always available.

Underneath the query processing layer, there is a distributed scalability layer that provides all of the features required for a distributed database such as sharding, replication, failover, and storage management on each node. This layer also supports distributed transactions and it is the core of the distributed system. It can be compared to Google Spanner. One unique feature of YugabyteDB’s architecture is that it offers multiple different query layers. Apart from SQL, it also supports a Cassandra-compatible API called YCQL and an API inspired by radius. This allows developers to use the APIs they are comfortable with while still taking advantage of a horizontally scalable database.

As for the complexity of replicating the feature set of a relational database like MySQL, PostgreSQL, or Oracle DB in a distributed SQL system, it is a challenging task. Relational databases offer a great deal of data modeling flexibility and query flexibility, which come with the power of SQL. These features, like indexing, multi-row, multi-table operations, and point-in-time restore, are considered core building blocks for any database system. Without them, developers would have to jump through hoops to perform basic operations like moving data from one row to another in an atomic transaction. The power of SQL and these standard features enable developers to write productive applications.

At YugabyteDB, we have focused on bringing all of the developer productivity that comes with SQL to a distributed architecture. Additionally, we have made certain trade-offs in order to innovate, such as the decision to use the open-source model.

When building the database, we wanted to make sure it was cloud-native and the most simple way to start was to decide which language API to emulate. We considered building our own or emulating an existing one. But as we mentioned earlier, one of the biggest challenges of building a new database is answering the question of why it’s necessary. We realised that we weren’t going to innovate much on the API, as there are already so many databases out there. Our goal was to take existing APIs and extend them to work well in a cloud-native environment.

They decided to use an existing flavour of a relational database, which narrowed it down to either PostgreSQL or MySQL. They also knew from the start that they wanted to make it open-source, so they did a careful analysis of the feature set that both databases offered and compared it to other powerful databases like Oracle, DB2, and SQL Server. After considering all the factors, they ultimately chose to base YugabyteDB on the PostgreSQL layer. This was due to its extensibility, features and adoption, and license compatibility to the Open-source model. This was not an obvious decision back in 2016, but they believed it was the right one. They wanted to re-write it in C++, the way they wanted it to be completely cloud native and thus YugabyteDB was born.

After five or six months of trying to rewrite PostgreSQL in C++, they realized it was not going to be feasible. It would take forever to build the language layer, and it would be just as hard as building a new database from scratch. So, they decided to reuse the existing PostgreSQL codebase, but that presented a new type of risk: would they even be able to complete the project? They spent the first part of the project de-risking this piece to make sure it would work, which it fortunately did.

The choice of programming language was pretty obvious, as PostgreSQL is written in C++ and they also chose RocksDB, which is also in C++, as a starting point. They wanted to ensure they had high-performance, so they went with C++. One of the biggest challenges was figuring out how to synchronize clocks across different regions and data centers. They looked at Google Spanner’s atomic blocks with GPS, but knew that wasn’t a practical solution for most users. So, they decided to use the hybrid logical clock algorithm and made improvements and tweaks to it to make it safe and high-performing.

Another decision was to start as an open-core company. At the time, it seemed like the only business model for open-source companies was to make a portion of the product proprietary. However a feature called cross-datacenter replication. So these were some of the most challenging features that they had to build. Ensuring PostgreSQL compatibility, integrating the query processing layer with their distributed transactions, implementing enhancements to the Raft protocol, and implementing a safe and efficient cross-datacenter replication system were all very challenging tasks that required a lot of careful design, testing and debugging to ensure they worked correctly.

Network compression and also made other performance optimisations to the rough protocol and these are just a few examples of the many challenges that they faced and continue to face as they build their database. But overall, it’s been a great journey and they’re constantly learning and improving as they go, always striving to make their database the best it can be for their users.

YugabyteDB is a distributed database designed for cloud native environments and is built on top of the popular open-source relational database PostgreSQL. It is intended to be used for online transactional applications that require horizontal scalability, active-active resilience, and geo-distribution. The team behind YugabyteDB has focused on bringing all the developer productivity that comes with SQL to a distributed architecture. However, it is important to note that YugabyteDB is not a one-size-fits-all solution and may not be the best fit for certain use cases. It is primarily an OLTP (Online Transactional Processing) database, and is not suitable for use cases primarily focused on analytics or reporting. Other use cases such as data warehousing or batch processing may also not be a good fit for YugabyteDB.

Leave a Reply

Your email address will not be published. Required fields are marked *