Turso is rewriting SQLite in Rust (Interview)
What’s up and welcome back to the Changelog! We feature the hackers, the leaders, and those replacing SQLite. Today, we’re joined by Glauber Costa, co-founder and CEO of Turso. Turso is an open-source distributed database powered by libSQL, which is described as the open contribution fork of SQLite. We discussed their efforts with libSQL, the challenge of SQLite being in the public domain and not having an opportunity for contribution, their choice to rewrite everything with Limbo, how this all plays into the future of the Turso platform, how they test Limbo with Deterministic Simulation Testing (DST), and their plan to replace SQLite. A massive thank you to our friends and partners over at Fly.io. Fly is the public cloud built for developers who ship. Over three million apps have launched on Fly, and you can too. Deploy your app in five minutes. Learn more at Fly.io. Okay, let’s see what it takes to replace SQLite.
Before the show, I’m here with my good friend David Shu over at Retool. Now, David, I’ve known about Retool for a very long time. You’ve been working with us for many, many years, and speaking of many, many years, Brex is one of your oldest customers. You’ve been in business almost seven years. I think they’ve been a customer of yours for almost all those seven years to my knowledge. But share the story, what do you do for Brex? How does Brex leverage Retool, and why have they stayed with you all these years?
What’s really interesting about Brex is that they are an extremely operationally heavy company. So for them, the quality of the internal tools is so important because you can imagine they have to deal with fraud, they have to deal with underwriting, they have to deal with so many problems. Basically, they have a giant team internally basically just using internal tools day in and day out. So they have a very high bar for internal tools, and when they first started, we were in the same YC batch, actually. We were both at Winter ‘17, and they were, yeah, I think maybe customer number five or something like that for us. I think DoorDash was a little bit before them, but they were pretty, pretty early.
The problem they had was they had so many internal tools they needed to go and build, but not enough time or engineers to go build all of them. Even if they did have the time or engineers, they wanted their engineers focused on building external-facing software because that is what would drive the business forward. Brex’s mobile app, for example, is awesome. The Brex website, for example, is awesome. The expense flow, all really, you know, really great external-facing software. So they wanted their engineers focused on that as opposed to building internal CRUD UIs. That’s why they came to us, and it was honestly a wonderful partnership. It has been for seven, eight years now. Today, I think Brex has probably around a thousand Retool apps they use in production every week, which is awesome, and their whole business effectively runs now on Retool, and we are so privileged to be a part of their journey. To me, what’s really cool about all this is that we’ve managed to allow them to move so fast. So whether it’s launching new product lines, whether it’s responding to customers faster, whatever it is, if they need an app for that, they can get an app for it in a day, which is a lot better than, you know, in six months or a year, for example, having to schlep through spreadsheets, etc. So I’m really, really proud of our partnership with Brex. Retool is the best way to build, maintain, and deploy internal software. Seamlessly connect databases, build with elegant components, and customize with code. Accelerate mundane tasks and free up time for the work that really matters for you and your team. Learn more at Retool.com, start for free, book a demo again, Retool.com.
What would happen if you took SQLite, the most widely deployed database in the entire world, and you rewrote it in Rust with some new twists and new ideas? We are all going to find out because Glauber Costa and his team at Turso are working on it. It’s called Limbo. Glauber, welcome to the Changelog!
Thank you, it’s great to be here.
It’s great to have Adam here despite flu season hitting hard in the Stachowiak house.
Oh man, it sounds great though Adam, you sound great. You’ve mentioned you on the tail end of the flu but don’t sound like I’ve been trying so hard to sound better so thank you. Where should we start? I think this story goes back quite a ways I remember lib SQL or lib SQL which was a couple years ago now your guys’s effort to create a different open source SQLite I thought SQLite was open source and public domain can you tell that story?
Oh happy, happy to do that and the story in a sense it’s still the same story. So I think you hit the nail in the head there, it’s just really how it started. I don’t feel this is a completely different thing what we’re doing now. The story there is that we were using SQLite in our company, so Pekka and I, Pekka is my co-founder, we founded a company for, if you happen to stumble upon it just mentioning here by name called ChiselStrike. It was in the API space. We were doing a bunch of things, and we were using SQLite very heavily in that company. The reasons we were using SQLite were reasons I think that a lot of your audience will understand and relate to. Look, you would just do npm install the name of the package, and then everything’s there. Your database is there, everything is there, everything works. You don’t have to install anything. SQLite for us always had this appeal of a database that is just always working, and then you can build stuff around it, and it’s just always there.
There was always something that was very appealing to us. However, we also stumbled upon this thing a little bit about SQLite not being open source. Let me address that. SQLite is a public domain software, which is technically a difference that doesn’t matter. I think only a lawyer would be concerned with what is the difference between public domain and open source. For all intents and purposes, it’s the same difference between, like, the BSD license and the MIT license. It’s minor. Nobody cares. I don’t care and I think nobody else does either. But SQLite is, according to their own website, by the way, this is something that they put in big bold letters in their website, open source. Again, I’m not claiming that they’re wrong to say that. It’s all good, open source but not open contribution. When people think of an open source project, there are two things that come to your mind, and the first one is, like, the code is open, and SQLite is that. But you also think about instinctively, well, the code is going to be on GitHub somehow, and then if I find an issue, I can go contribute to this project, and I have some source of ownership to the code, right? It’s not just they can see the code, they can also modify the code. That’s usually classically one of the tenets of open source. Now, SQLite is not like that. The maintainers of SQLite, the creators of SQLite, do not take contributions. This is not something that we are saying, this is something that they’re saying. I think it happened in the past that people managed to contribute to SQLite in very special occasions, but it’s not a project that is designed to take contributions from other people.
That is how SQLite is. We started running into a couple of hurdles with SQLite. We wanted to deploy our SQLite databases to the edge. We wanted to deploy it in a way that was replicated. We wanted to do things like read replicas, and we wanted to put it to run on Cloudflare workers and all things like that. We consider this a good thing for our project, so technically we knew we could do it. For context here, Pekka and I have been working together. Pekka is my co-founder, we’ve been working together for 15 years now. We met each other when we were both working on the Linux kernel. Pekka was one of the maintainers. We worked, each one of us, in Linux for almost 10 years. The first five without knowing each other. I think we knew of each other but we never met. In the last five years very closely to each other where we developed a good friendship. After that, we actually joined a startup that also became a database company. We spent eight years doing that. We felt like SQLite is, all things considered, pretty simple, given the scope of things that we used to do. Technically, we can make the changes that we believe we can. Let’s just come up with our own version. We saw, at the time, a lot of people dancing around the subject, that’s what we saw.
There were some projects like Lighter and Fast that were super interesting projects, Decolite. We saw like lots of people trying to get SQLite to work as a distributed system. The problem that we saw with all of those projects is that they were all again dancing around the subject that SQLite cannot, SQLite cannot be changed. We’re going to create layers and layers and layers around SQLite to help with that. For us it was, technically it’s actually a pretty simple problem. You just have to change SQLite here, here, and here, and you’re good to go.
The last piece of the puzzle, maybe the two last pieces of the puzzle, number one is that we were very concerned with who’s going to want to run our SQLite plus patches because SQLite is a very trusted project. You make random modifications to SQLite, everybody starts distrusting what you’re doing. The solution to this problem is that we should just create an open contribution version of SQLite because now it’s not like SQLite plus fingers, it’s its own project, and then people come to trust that this one project. With that, the diagnosis that we had at the time is, hey, SQLite is great code-wise. The thing that is holding it back in our reading is exactly the fact that it cannot take advantage of the dynamism of a truly open source community. Let’s create that. The last piece of the puzzle is that at the time we were actually discussing how do we do this, and one of the options was, maybe we should just rewrite SQLite. We had a lot of experience in our previous company, Sila. Sila was a reimplementation of Apache Cassandra in C++, fully compatible. Maybe we over-indexed in the fears that we had going that route. The problem with Sila is that it took a long time for us to put in the market. SQLite is much smaller, so maybe we can do it. Maybe because we had just come from an experience where we wrote a database, we figured that forking was a better strategy. At the time our decision, after much deliberation, was we’re not going to rewrite SQLite. We are going to fork it instead because then the main advantage is that you have something tomorrow, because you start from a code base that is already working, and then we’re going to start making changes to that. So that was libSQL. libSQL was essentially, at the time, the story of Limbo. If I had to tell the sanitized version in which the fork never existed, it would be the same up until this point because all of those things, like we were all of that, it’s still valid. It’s just that at the time, a year and a half ago, we decided the forking was the best alternative. Now I’m happy to go into the why, but now we decided to try, okay, what if we had done that one of those options that we consider. What if we just rewrote it? When we put it out there, the results were, I’ve never seen anything like that before.
The public domain of SQLite is unique. I think the not open to contribution is also quite unique, very valid in the in quotes open source world. Even they say on this public domain page they have where they say warranty of title, so if you are a company that needed something where you needed to be indemnified against copyright infringement or anything happening inside this code base, it gets to be a little bit hairy trying to use SQLite in unique ways beyond. It kind of makes sense some of the challenges you had and then maybe the fork made the most sense, but ultimately the rewrite seems to make the most sense because what you want is the attributes of what SQLite offers the world, but not the, and I don’t want to say like a negative thing because it’s great but it’s dominated very well in the marketplace. It’s used on Mars, I believe, right?
It is on mars, it has trillions of databases deployed.
It’s not, and Dr. Richard Hipp is one of our friends. We love him. We think he’s an amazing person, but I think you’re all right, though, that the public domainness and the not open to contribution makes it challenging because, Dr. Richard Hipp even said on this podcast in the past, the test suite is proprietary and the fact that it’s not in the open, you can’t see it. Making these large-scale changes to build on top of what is SQLite, if that’s your desire, becomes problematic.
This was one of the big contributors for us to revisit that decision was exactly the test suits. It’s not that we didn’t know about it, it’s just that we thought things would go one way. I always love to, if I may, to use opportunities like this to clarify something because when you’re telling a story, when you’re saying something, it’s easy for people to misunderstand you. I always want to make this 100% clear. There’s nothing wrong with SQLite and the way they manage their community. Sometimes people assume that because we went this direction we believe that what they’re doing is wrong. I want to clarify that that’s not the case. The beauty of having different people with different points of view coming from different backgrounds and doing things differently is exactly that, you can experiment with a lot of models and those models have advantages and disadvantages and it’s fine. We don’t want, we truly believe and the goal we have with Limbo now is really to replace SQLite, so we believe that we can build something much better if we’re able to tap into the creativity of a modern open source community.
I’m glad you make that explicit because as an observer, I remember your initial announcement and maybe it’s because of maybe the social baggage of what a fork implies but exactly my initial impression was and I don’t remember the words that you use anymore, it was two years ago but I was like they see you and your team seemed a little bit upset or you know you’re kind of disappointed with this and so we’re going to fork and do it a different way and it’s hard to do that without ruffling feathers or without getting even if you don’t mean to, people think that you’re angry or mad or whatever.
We spent a lot of time crafting that message, but regardless of how much there’s always some room for misunderstanding, which is all which is why I always appreciate. I also don’t want to run from it. I want to make something clear. We disagree technically with that decision. It doesn’t mean we think it’s wrong. They clearly created something very successful. The way Pekka and I always run open source communities and the way we always participated in open source communities for us this component of being able to tap into the creativity of the community was always something very important. You have to put it in context as well. Pekka and I grew up in the Linux kernel community. A lot of people say this about SQLite. SQLite has to, and with that I disagree, SQLite being what it is has to be kept as this very small thing because that’s the whole beauty of SQLite and again I disagree with that because if you look at Linux, Linux started as this project that would only run on x86 computers and today it is an operating system that runs everywhere. It runs very well on the server.
We worked together in memory management for very big box machines like how to make sure that when you have two terabytes of memory in 2010, which was a lot of memory back then, it’s still a lot of memory today, but those algorithms to manage memory don’t take three minutes and hang up the system. It’s a problem that doesn’t happen in embedded devices, and Linux still runs on embedded devices. Through this big community you actually can create something that runs well everywhere and don’t necessarily have to make those compromises. We don’t agree that SQLite has to be protected and has to be maintained by this single individual to be this the success it is. Maybe I’m wrong, maybe I’m right, but we act based on what we believe. There’s nothing we don’t believe there is anything wrong with what they’re doing. We just think that, you know, it could benefit from a different with a different direction.
You do believe that you can replace, you said, we believe we can replace SQLite 100% and we will, and I want to clarify that now, because with libSQL it became very clear to us in maybe a month that that was not going to happen. With libSQL we had this goal of replacing SQLite for 10 minutes, because it’s a dream. It’s such a big prize. As I wrote in my blog post, were we bummed about it? No, because we built a pretty successful business on top of libSQL, which is today our company Turso. For us, it was this exercise of, if you heard the expression shoot for the stars because even if you miss you’re going to hit the moon. It turns out I don’t think it’s going to be possible for us to replace SQLite with the fork, but that’s us hitting for the stars. We didn’t hit the stars, we hit the moon. The moon’s great. We built a super super great project. We’re having fun. libSQL brought something to SQLite that that wasn’t there, which was, like, the serverless and running this on serverless environments on the cloud platform, syncing databases. One of the things that we do it through is that you can sync your databases between different devices and the platform and backups and all of that, and this is going great.
We were never sad or disappointed that this thing that crossed, I can imagine how amazing it will be if people just start using our fork instead of SQLite. We’re never disappointed at that. When we put the rewrite out, it became very clear. I’ve been using this analogy from chemistry that the reaction was right. libSQL was something that the market was clamoring for. Seeing the reaction of Limbo now is that it’s just that the activation energy is not enough to make the reaction happen. After talking to a lot of people, I talked to a lot of the people who came, started contributing to the project, that they all wanted, like, they all agree with us that we need a better version of SQLite. It will be fantastic if you have something like SQLite, but does this all our original vision. The fork just wasn’t differentiated enough. It just wasn’t ambitious enough. It just wasn’t bold enough to get people to come and contribute. Furthermore, it is actually very hard to do because one of the things you did mention, like the SQLite’s test suit is fully proprietary, so once you start making changes that are very deep into the core, you start hitting a bunch of issues and some of them is just like a yourself saying I know that the best way to solve this problem is this but I’m gonna solve it in this other way which is almost as good but not great but it’s less risky so you start having this on your mind. The fork never really unshackles you fully to go pursue this, which is why when we announced the rewrite what we saw was something I’ve never seen before.
We got 8,000 GitHub stars in a week. We had for the first time in the history of computer science a Hacker News comment section that was mostly positive with a couple comments only that were trying to denigrate us, but the rest very supportive. We had this fantastic inspiring story of this individual who is in prison at the moment. He was one of the first people in the United States to be granted access to the internet in prison. He can access Discord but not X. He is now the fourth top contributor to the Limbo project. The level of contributions that we started to get. We started getting people running this on the browser a week after we announced. It was running on the browser already. It was compiled to WebAssembly. All from contributions from third-party people.
If libSQL, we did get a lot of contributors, but they were all contributing to the drivers to the server, things at the margin, which they’re important as well for an open source project, but with Limbo, we actually managed to see our dream come true but we’re going to build a community of people coming here and writing this database with us. It is no longer a 10-minute dream that went away. We truly believe now that once we get the momentum going, we can replace SQLite and that’s the goal.
I know my gut response to Limbo was quite a bit more positive than mine was to libSQL just by reading the announcement post. I’m like okay this makes a lot of sense. It seems like with a rewrite there’s just so much more to do. It has its own identity, there’s new ideas and whereas a fork almost comes from a different place. On the testing front, though, I would imagine it’s got to be just as hard or maybe I would expect it to be harder with a rewrite. What makes it easier with a rewrite than with a fork if you still don’t have a test suite?
Your intuition is correct, by the way, which is one of the things that all things consider we went with the fork because at least there’s some level of guard rails that come with the fact that at least you know that this code that you’re importing was tested by this proprietary test. I don’t think your intuition is incorrect. Something amazing happened. I don’t actually think that we made a mistake with the fork because if we were rewriting back then, there were many decisions that we made today that we would not have made, but now we did because we have the experience. We’ve been running the Turso platform and we learn a lot of things. It actually allowed, I almost see it as a prototype and it actually allow us to to make tremendously good decisions with the Limbo project and one of them is exactly the testing. Pekka and I got hooked into something called deterministic simulation testing. Deterministic simulation testing is a very niche thing that most people have never heard about. I had heard about it a couple of times but I didn’t truly quite understand what it was until I met Joren. Joren is the founder of the most amazing database company I’ve ever seen. Tiger Beetle is a database just to keep it very brief that is designed for financial transactions. It’s completely written in Zig. Their goal is to replace entire systems, the bespoke systems that most financial institutions have to process transactions. If you’re selling to banks or in financial institutions if you if your goal is to like replace the backbone of world’s commerce it’s it’s pretty hard to do.
One of the challenges that that you have is just this like how do we trust this thing and you’re unfound the solution for that. The solution that you’re unfound was to write Tiger Beetle entirely with something with a technique called deterministic simulation testing. Deterministic simulation testing is essentially a fuzzer, so it will generate a bunch of inputs. It randomly generates as many inputs as it can on the space of possible inputs. This is the disadvantage. You kind of have to write software in a special way to to lend itself to deterministic simulation testing. It’s very hard to bolt it to an existing code base, but you write it in a way that every single operation that you do on IO on you know thread scheduling everything that happens goes through a special interface that abstracts all IO and then you plug a simulator. The simulator does is that it’s going to explore the space of possibilities, create the most arcane impossible situations ever. When something breaks it gives you the exact steps deterministically that everything in the system had up until that point. Debugging those problems become very easy. In record time, Tiger Beetle managed to create a system based on deterministic simulation testing. We found this bug that would only happen if you would call F-sync on a disk and then F-sync would return an incorrect result at the same time a packet will come from the network. The simulator gives you a seed. You type that seed into the simulator. Now you have every single step that happened to make that happen. We knew that we could now try to rewrite using deterministic simulation testing and then we also partner with a company called Antithesis that offers almost like the the integration test analogy for deterministic simulation testing which is a full system version of that that will simulate things between machines it will simulate the network it will simulate like hardware failures and stuff like that.
Whatever bugs our simulator does not catch usually you give it to Antithesis, the next day Antithesis catches the bug. We will be able to actually create something that probably even surpasses the level of testing the SQLite has, but here’s the catch. It is easier to do this on the rewrite because you have to write the system with this in mind from the get-go. Is that popular enough that you can go out and there’s Rust crates that will give you this functionality?
For people who are interested in DST but don’t want to go through the pains of just rewriting all sorts of things like that I truly recommend taking a look at Antithesis because Antithesis is amazing. It’s the second you know it’s not there’s the second next best thing. We use it in conjunction it’s not an either or the analogy that I have is that our own DST is like unit tests. We can run centuries of possibilities in two days and it’s very fast all things considered and Antithesis is like integration tests, so you want to have both.
The problem with deterministic simulation testing is that once you start importing other crates in Rust, for example, you have no idea what those crates are doing. Those crates are calling IO. You have to write everything, so it’s not even that you have a crate for deterministic simulation testing. You might not even want to import something super simple but you don’t want to import anything that does IO that has a timer. You want to write those things so it’s something very hard to do for a general purpose testing for a database like Tiger Beetle for a database like Limbo you know it’s worth it and and the scope is quite limited. Tiger Beetle is way more insane than we are. They have a policy that they just don’t have dependencies. They write every single piece of code. We try to be a little bit more flexible, but we will not import a crate. Also depends because SQL comes with a CLI. If you want to import a crate that does whatever crazy stuff to implement callers in the CLI, but for for the core of the database like we try not to import anything that could potentially do IO because we want to make sure that everything goes through the simulator.
You can manage failures, network outages, flaky endpoints, long-running processes and so much more, ensuring your workflows and your applications never fail. Temporal allows you to build business logic not plumbing. They deliver durable execution and abstracts away the complexity of building scalable distributed systems and lets you focus on what matters delivering reliable systems that are faster. Masari are the Bloomberg for crypto, they provide market intelligence products to help investors navigate digital assets and they recently turned to temporal to help them improve the reliability of their data ingestion pipeline. This pipeline collects massive amounts of data from various sources and then they enrich it with AI. This process previously relied heavily on cron jobs and background jobs and cues and the design worked well however these jobs were difficult to debug at scale because they needed more controls and more observability and as they looked to rethink this ingestion flow they wanted to avoid cron jobs background jobs cues they didn’t want to create a custom orchestration for the system to oversee and to ensure these jobs and work was being done reliably.
Before temporal we had to code for dead letter queues circuit breakers etc to ensure we were resilient to potential system failures now we eliminate these complexities the headache of maintaining custom retry logic has vanished by using temporal end quote so if you’re ready to build invincible applications and you’re ready to learn why companies like netflix doordash doordash and stripe trust temporal as their secure and scalable way to build and innovate go to temporal.io.
This DST sounds like magic.
The simulator just writes a couple of scenarios. The simulator will include a fuzzing element to that. So we will instead of generate this query that I wrote in the unit test, you give it almost like a query generator and then it starts like generating random queries and then you start like in the simulator itself what happens. Imagine this like if you want to write to a file in any software like you would call a operating system API like right.
When you’re writing software for DST you don’t do this like you have your own IO interface abstraction and then all of your IO goes through this so when you’re running this in production mode your abstraction for write just call the operating system write but when you’re running deterministic simulation testing mode your abstraction for write runs runs the simulator code that will start injecting failures into this and injecting failures in a deterministic way because then if the query fails because you injected a IO failure at that moment you will be able to replay that session piece by piece.
Is that the guy that’s barely inconvenient?
Very easy, barely an inconvenience. Super easy I love that guy man.
How do you rewrite SQLite with the confidence that it will actually have the level of trust that SQLite acquire? All you have to lose I love it just throw a DST at it so when you talk about SQLite compatible that’s Limbo’s goal, there’s a lot of different fronts that that has to be on, like, are you talking file structure, are you talking syntax language, are there performance compatibility there’s all kinds of things that SQLite is. What’s Limbo’s goals with regard to these different areas of compatibility?
Compatibility in a project like this has to be like a one-way street. You don’t want to shackle yourself and say I will always because sometimes to implement a new feature you have to do it in a different way. If you we’re going to offer you the same feature set as SQLite we’re gonna read your SQLite files we’re gonna execute your SQLite code and if you’re not using any of the specific features that we have we can generate SQLite files as well.
Limbo so far wants to keep the language the same. We want to keep the ABI. We want to be able to load SQLite extensions. The file format obviously it’s the thing that defines SQLite so obviously again we might add to it in the future but we will be reading SQLite files normally and we bytecode compatible as well. So one of the other ways in which we test Limbo is by generating random SQL statements and then seeing that the bytecode that is generated by Limbo is the same bytecode set of instructions that is generated by SQLite.
I mentioned at the top that it also has some ambitious goals, one of those fully asynchronous IO, which is quite a bit different than SQLite. How do you accomplish that but maintain this compatibility?
Async does not mean that we’re going to use an Async runtime like Tokyo. If you look at the Limbo code it’s just sync rust it’s not async rust. All the Async means is that when you call an operation if that operation is not ready it will return to you instead of blocking and the main now I’m getting a little bit too technical but let me just give you the full context the SQLite C API has a bunch of functions but the most important of them is called SQLite step and SQLite step is essentially like you take another step take another step in processing this query for me.
What that means is that if you call SQLite step it will block until it resolves. In Limbo, if you detect that you’re not ready to execute those bytecode instructions at that time you just return something saying no right call me back later. It becomes very easy to plug something like Tokyo on top in Rust or run it on the browser because you can call those Async functions right and if it’s not ready it will not block that’s all that means.
What does that unlock for you because SQLite historically is sync but super fast because it’s right there in process. It isn’t that fast. Queries that are much more complex that you can run in dashboards and things like that and also like for the serverless environments Turso being one example it allows you to do like serverless SQLite on the cloud.
Our product in the Turso cloud allows you to do like serverless SQLite on the cloud. It allows you for example to host your data on S3 because if you have your data hosted on S3 your query now is not necessarily super fast because you may be hitting a page that is not local so it allows you for example to run SQLite with partial storage like with most of your data on S3 and some of the data locally. Browser. If you block you block everything.
You put a lot of thought behind this. Why were you so surprised it was so well received?
We consider rewriting SQLite from the get-go anyway, so this was always on our mind. Our goal was, if this project keeps going, maybe in two years, we can make something out of it. The presentation was just like, yeah, whatever man. This was on Becca’s personal GitHub. The logo he just asked Chat GPT to generate. There was not a lot of thought on the presentation and publishing, but there was a lot of thought about the technical side. Maybe in two years there is a play for us to tackle rewriting c-collide. What we do not expect is that as we were we decide what we decided to do is just make it a official Turso experiment.
We don’t have a roadmap for this.
It did pretty well based on two metrics. We got a thousand GitHub stars and two incredibly good engineers that started. We hired those those two engineers. So we knew there was something there. Let’s just now publish it as a official Turso experiment. I would expect maybe this to gain 2,000 more stars. We saw three times more four times more than what we anticipated. I mean it sounds it seems very clear to us now that the world at large really wants a evolution of SQL like. The signals are very strong so I think we just need to to get behind it right.
What do you think made those two contributors that were so pivotal contribute?
The story is always the same. They were very excited about the prospect of like a better sickle light. They knew about lib sickle but lip sickle never enticed them because it just wasn’t crazy enough. I love the vision. The vision is the right amount of ambitious that that I want to start devoting my time to that and I want to be a part of that as well so that’s exactly and we wanted we wanted to one of the things that we said a lot in the in the early lip sickle days is that we said that a lot because we heard from a couple of people a couple of companies even we love the idea of what you’re doing because it gives us a seat at the table. My interpretation of that is that it turns out it’s actually pretty hard to give people a seat at the table if you don’t own the table so Limbo gave us this I mean it’s our table now it’s a table that is modeled after a another table but we own the table and now we can really truly give those people a seat at the table. They’re reviewing code they’re they’re helping with the direction they come out with ideas things that we never considered things that we never considered things that will not be a priority the thing a sequel already Limbo runs on the browser though somebody showed up today today oh I got I got lib SQL to compile to run this the Limbo CLI in web assembly. The person took an interest because of it doesn’t have a ceiling you essentially can take this anywhere.
Does lib SQL just go away then or be replaced outright? The leap SQL project today is a fork of SQLite with some of those changes like vector search. It is also, and it’s all in the same repository, it’s all part of the same project, it is also a server implementation to do serverless SQLite over the wire, client side part is going to be full Limbo. We don’t intend. We intend to eventually get all of those things we’ve done like vector search all of that lip SQL is still maintained for the time being because again we have a business around it we have features that we depend on but our goal is to eventually port all of those features to Limbo and then the client is that and lip SQL can become just the server implementation as an open source alternative.
We love Notion. It’s saving us so much time. I don’t have to kind of think where is this in my massive notion workspace i just notion ai it and it comes up it’s so cool if you know notion, they combine docs notes projects all into a single space that you can design yourself. It is your one place for your team to connect with your tools your knowledge and you’re empowered to do your most meaningful work and unlike other tools out there that make you bounce from one thing to the next to the next notion is seamlessly integrated infinitely flexible and it’s beautiful and easy to use so notion ai helps us work faster we’re writing better thinking bigger we’re doing tasks that normally take hours and we’re doing those things in minutes sometimes even seconds and yes we’re not a fortune 500 company but notion is used by over half of fortune 500 companies and teams that use notion like us send less emails they cancel more meetings they save time searching for their work and they reduce their spending on tools which helps everyone stay on the same page so try notion for free today when you go to notion.com/changelog that’s all lowercase letters notion.com/changelog and try the powerful easy to use notion ai today and when you use our link of course you are supporting this podcast which you love and we love that too so notion.com
Is there trailblazing left to do or have you done the trailblazing and now it’s just a matter of work? One of the things that we heard as we launched leap sequel, give me better schema changes and I’m switching to this tomorrow. One of the areas SQLite is really bad at is schema changes. You can make schema changes, but you cannot alter the type of a column. There’s all sorts of limitations like that which is something by the way that we made better in leap SQL so leap SQL does better schema changes than SQLite but it’s not better enough. We were limited in in what we can do. We want to run replication like native replication to the browser which is something people have been asking us for a long time imagine you have this SQLite database running on your browser that can then sync with an external server or SG or whatever and just get pages on demand that’s one of the things we want to do. We want to tackle the problem of schema changes. We want to tackle the problem of right throughput because SQLite is a very bad database for right operations. We want to make SQLite much better for analytical workloads.
How much time are we talking months years? Deterministic simulation testing allows you to move with a lot more confidence. If you get into the thought process of a database writer, deterministic simulation testing just allows it to make a lot faster. It just allows you to move so much faster. SQLite is not that complex. If we were rewriting Postgres it’s a completely different story we’re writing SQLite it’s actually doable in in a year. We’re going to put the whole we’re not going to do this in Q1. We’re going to have at least seven eight engineers just working on it full time. It’s a very reasonable timeline like nine months to a year.
Next January are we talking about is it production grade what 1.0 what does it look like next? We want to release a 1.0 much earlier than that because we want to release often. A lot of people are going to be early adopters. A lot of people can already use it. We want to be very aggressive with making releases. The moment we’re going to say hey like this is stable and look we can actually even give us more time to say something is stable I mean you you know people trust you that this is going to work so we can take even more time than that to say hey this now has our seal of approval we’re going to support all of that but much earlier than that it can already be production ready.
How does this all affect Turso the business and how does it fit in? Our business is going to change. What we need to do is that we need to simplify our product a lot. We had a very very hard decisions to make. A couple of features that we have, users got quite upset with that. Those features we will discontinue. The way we’re doing this is that if you’re a paid user up after a certain cutout date you’re going to be allowed to keep using those features but new users new signups anybody else who is not a paid user at that point we will no longer have access to those features.
We think that having a single person just running the platform is will be enough and we’re not going to be investing in having new features into the platform. The Turso cloud will essentially be a place where you can still run. What we do today is two things very well. You can access SQLite over the wire from serverless environments so you have a serverless managed SQLite database and you have syncing of databases between devices servers etc so you can start with us with with your SQLite file and you can upload that file to Turso and then replicate that to other SQLite files that you own that’s the platform. It’s a position that we can afford to do this and then for one year we’re going to go all in in getting Limbo to replace SQLite and that’s it. Limbo then becomes the thing that we run on the platform and we have the money we have the runway to do this I mean we it’s all accounted for so if we manage to which I believe it’s very doable to get Limbo in a production ready state in 15 months our plan still works but 9 to 12 is what we’re we’re thinking here.
When you get to that place whenever Limbo becomes Turso and Turso cloud goes away how will you differentiate this new launch like how will you? Turso cloud doesn’t go away. Turso the embedded database just becomes Turso. It’s very hard to create two brands. The reason we kept those things separate and the reason lib SQL wasn’t called Turso was exactly because we wanted to give people the sense of this is a welcoming community. This idea that we had that we have to keep things very separate to create the welcoming community we learned that nobody cares. We’re gonna we’re gonna take the opportunity here to just to fix it and the people in the know they know but people who are just hearing about it like now you have to explain that. We’re going to just consolidate the names and Turso today again is the cloud offering that is going to be renamed to Turso cloud and then what what is Limbo which is the client offering will just be renamed to Turso and that’s it and we’re still welcoming we still want people to to come in and and build this with us we just learned that the name doesn’t matter right.
How do you manage that relationship between the open source Turso the the entrance of the interests of Turso cloud third-party contributors who may want Turso to go a different direction that maybe you don’t want it to? That’s why what we decided to do is never port the server code to Limbo. The server can be kept completely separate. This has to be just the client-side library exactly because of this because we don’t want to be in a position and the name was a part of that. We want to design things in a way that we are never ever ever in a position to even think about we would like to a contributor coming with this code and maybe we don’t want to merge it. We want to make people in the community maintainers as well. Structure things to make that happen is that our business now has to be just the cloud and then all the code for the cloud become that is a separate project.
How will you know when Limbo Turso has replaced SQLite? I think if this was a consumer business is when your mom calls you to say that she’s using it without knowing that this is the software they work. The criteria for us is when we can see somehow through some fuzzy metric that we’ve got like we have a billion databases out there. We’re not going to replace SQLite in a year that’s not that’s not the goal. But we think that in a year we can get to this point where hey we got our first billion. People wanted to play with it today how much is there what could you play with if anything a lot of the read stuff works. He wrote on day one. There is a compatibility matrix so if you go to if you go to the repository there is a file there compact.md that is linked into readme and there is a full compatibility matrix with everything so what you’re going to see there is like the read stuff from SQLite already works pretty well. The basics of reading from a SQLite file they work already.
Adam any other questions for globber before we let him go? Client side open source server side closed source and so lib SQL is Turso cloud and if you want to self-host more or less because. We already had for other reasons we already had a proof of concept of a new server implementation that is not based on lib SQL. We also want our server to have deterministic simulation testing. The reason we’re doing this is that we want to have this very clear separation without concerns of what goes where and we think the best way to do this is if the server is fully proprietary.
Lib SQL still exists as a reference implementation if you truly want to run an open source version. The embedded database is 100 percent open source. In the future when somebody runs Limbo which will be renamed Turso they can run that on their own but if they want this massive scale multi-tenant world they’ve got to host with you. You’re long betting on hosting a lot of databases in the future. Everybody needs money so no need to raise more now though to get to this goal. Let’s talk in a year I guess. There you go send me an invitation today for a year in the future January 22nd 26 same time same date. Thanks globber hey we’re rooting for you this is really exciting stuff thank you yeah
Well the future of SQLite or SQLite is not necessarily in jeopardy but it is being primed for disruption if globber and his co-founder has it their way they will unseat and they will replace SQLite and Limbo is full steam ahead with lots of inertia lots of momentum lots of contributions and there’s lots to do for you for me for others so you can dig in it is open source and it’s also open to contributions and it’s being rewritten which means there’s new things to do new innovation to be had and that’s exciting make sure you go to terso.tech t-u-r-s-o.tech to learn more about what they’re doing and to learn more about Limbo and the future of this database and this platform okay big thank you to our friends and our partners over at fly we love them fly you’re awesome fly.io launch your app in five minutes like we do it is the cloud for developers who ship that’s us that’s you fly.io and to our friends over at retool retool.com and our friends over at temporal temporal.io and our friends over at timescale timescale.com and to the beat freak in residence brake master cylinder those beats are banging banging okay friends it’s on friday stay tuned it’s awesome the show’s done we’ll see you on friday
Can you tell us more about this uh press is it Preston Thorpe do you have more information on this fella contributing from prison that you can tell a little bit more of the story yeah he he wrote a blog about his situation I actually mentioned that on x the primogen which is this big streamer that I’m sure a lot of your audience knows read his article on stream based on my tweet so there’s a lot if you know if you want to see a long account and and hear from him I do recommend you you you look and into his blog I’m happy to give you the link here but long long story short uh as far as I understand I mean it’s this guy that was uh in prison from non-violent drug offenses uh so he’s in prison somewhere in Maine as part of a pilot program he was allowed to have a remote job and for his remote job uh he has access to the internet monitor and with restrictions of course but when Limbo when Limbo was announced he took a tremendous interest and I actually spoke to him I called him the other day and the story that he told me is that it just so happened we announced Limbo in December right December 2025 so he was on a break so he was on vacation from his work and when you and I go on vacation you know maybe you go travel somewhere or I go play with my kids et cetera Preston’s vacation is just like I sit here in prison for 12 hours so change love plus plus