SQLite Data Flow

An interactive walkthrough of how SQLite transforms a SQL string into rows on disk — from the tokenizer through the virtual machine down to the filesystem.

Complete Data Flow Pipeline
SQL Text
e.g. "SELECT * FROM users WHERE id=1"
input
characters
tokenize.c · sqlite3RunParser():600
parse
token stream (TK_SELECT, TK_FROM…)
parse.y · sqlite3RunParser():600
parse
AST — Select, Expr, SrcList…
select.c · insert.c · where.c · expr.c
codegen
VDBE opcode array (OP_OpenRead, OP_Next…)
vdbe.c · sqlite3VdbeExec():881
execute
BtCursor calls
btree.c · sqlite3BtreeCursor():4775
storage
page reads/writes
pager.c · sqlite3PagerGet():5726
storage
VFS calls
os.c · os_unix.c · sqlite3OsRead():88
I/O
syscalls (read/write/fsync)
Filesystem / Disk
database.db (4096-byte pages)
output
Parse phase
Code generation
Execution
Storage
I/O
Public API Entry Points

Applications interact with SQLite through three primary functions that wrap the full pipeline:

FunctionFileRole
sqlite3_exec() legacy.c:30 One-shot: prepare → step loop → finalize, with row callback
sqlite3_prepare_v2() prepare.c:943 Compile SQL text → prepared statement (Vdbe*)
sqlite3_step() vdbeapi.c:913 Advance VM one step; returns SQLITE_ROW or SQLITE_DONE
legacy.c:30 — sqlite3_exec() loops prepare → step → finalize
int sqlite3_exec(
  sqlite3 *db,                /* The database on which the SQL executes */
  const char *zSql,           /* The SQL to be executed */
  sqlite3_callback xCallback, /* Invoke this callback routine */
  void *pArg,                 /* First argument to xCallback() */
  char **pzErrMsg             /* Write error messages here */
){
  int rc = SQLITE_OK;
  const char *zLeftover;
  sqlite3_stmt *pStmt = 0;

  sqlite3_mutex_enter(db->mutex);
  while( rc==SQLITE_OK && zSql[0] ){
    rc = sqlite3_prepare_v2(db, zSql, -1, &pStmt, &zLeftover);
    if( rc!=SQLITE_OK ) continue;
    while( 1 ){
      rc = sqlite3_step(pStmt);          // ← drives the VDBE
      if( xCallback && SQLITE_ROW==rc ){
        /* invoke user callback with row data */
      }
      if( rc!=SQLITE_ROW ) break;
    }
    sqlite3_finalize(pStmt);
    zSql = zLeftover;
  }
  ...
}
Component Overview
Key Data Structures
StructureDefined inPurpose
sqlite3 sqliteInt.h Database connection; holds schema, pager handle, flags, mutex
Parse sqliteInt.h Parse/compile context; threads through tokenizer → parser → codegen
Select sqliteInt.h AST for a SELECT statement; linked list for compound queries
Expr sqliteInt.h Expression tree node (binary tree); carries type, operator, literal value
Vdbe vdbeInt.h Prepared statement / VM instance; holds opcode array, register bank
BtCursor btreeInt.h Cursor positioned within a B-tree; tracks page stack and cell index
Pager pager.h Page cache and journal manager; one instance per open database file