Skip to content

sqlite3.Connection.backup() blocks indefinitely on a locked source DB; connect(timeout) does not apply #162414

Open

Description

@daneeller

Environment

  • CPython 3.13.14 (Windows 11), sqlite3 module. The behavior also appears in earlier CPython versions; I only runtime-tested 3.13.

What happens

sqlite3.Connection.backup() can block indefinitely when the source database is locked by another connection. The timeout argument of sqlite3.connect() does not bound this wait — it applies to locked table operations, not to the backup retry loop.

In the repro below, a parent process holds BEGIN EXCLUSIVE on the source database. The child calls src.backup(dst) and never returns while the lock is held (killed after 4 seconds). Once the lock is released, the identical child code finishes in 0.06 s.

Repro

# parent.py: hold an exclusive write lock for 4 seconds, then release
import sqlite3, subprocess, sys, time

CHILD = """
import sqlite3, sys
src = sqlite3.connect(sys.argv[1], timeout=1)
dst = sqlite3.connect(sys.argv[2], timeout=1)
print('child: calling backup()', flush=True)
src.backup(dst)
print('child: backup() returned', flush=True)
"""

conn = sqlite3.connect("locked.db")
conn.execute("CREATE TABLE t(x)")
conn.commit()
conn.execute("BEGIN EXCLUSIVE")
conn.execute("INSERT INTO t VALUES (1)")

p = subprocess.Popen([sys.executable, "-c", CHILD, "locked.db", "backup.db"],
                     stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
try:
    p.communicate(timeout=4)
except subprocess.TimeoutExpired:
    p.kill()
    p.communicate()
    print("child was still blocked after 4 s")

conn.rollback()
conn.close()

Observed:

child: calling backup()
child was still blocked after 4 s
child: calling backup()      # control run, lock released
child: backup() returned

Documentation gap

The current backup() docs say:

Works even if the database is being accessed by other clients or concurrently by the same connection.

and define sleep as:

The number of seconds to sleep between successive attempts to back up remaining pages.

They do not say that backup() retries on SQLITE_BUSY/SQLITE_LOCKED until it succeeds or a real error occurs, nor that connect(timeout=...) has no effect on this wait. A reader can reasonably assume the connection timeout applies, which is not the case.

Suggested documentation change

Add a note to the sqlite3.Connection.backup() section, e.g.:

backup() retries internally when the source database is locked; the timeout passed to connect() does not limit this wait. To bound the total time, pass a progress callback that checks your own deadline and raises, or call backup() with a bounded number of pages per step.

If a code change is considered instead, an optional deadline/timeout parameter on backup() would be useful, but I understand that is an API change and may need broader discussion.

Impact

A background backup thread that relies on connect(timeout=...) can hang forever if another process holds a write lock, with no exception and no timeout to trigger cleanup or retry logic.

No project-specific or private data is included in this report; the repro uses only the standard library.

Activity

  1. daneeller commented on Nov 22, 2026

    @daneeller
    Author

    A verified workaround that could be worth adding to the backup() docs, including one non-obvious detail.

    A bounded wait can be implemented with the progress parameter of backup() itself, using a monotonic clock:

    import sqlite3, time
    
    def backup_with_deadline(src_path, dst_path, seconds=30.0):
        deadline = time.monotonic() + seconds
    
        def check(status, remaining, total):
            if time.monotonic() >= deadline:
                raise TimeoutError(f"backup exceeded {seconds}s")
    
        src = sqlite3.connect(src_path, timeout=1)
        dst = sqlite3.connect(dst_path)
        try:
            src.backup(dst, progress=check)
        finally:
            src.close()
            dst.close()

    Verified on 3.13 with another connection holding BEGIN EXCLUSIVE: the above aborted in ~1.8 s with a 1 s deadline and completed in 0.01 s once the lock was released, instead of blocking indefinitely.

    The non-obvious part: Connection.set_progress_handler() does not fire during backup()'s busy-wait. I first tried that API and it hung for over 2 minutes — the backup step loop retries on SQLITE_BUSY without executing VM instructions, so the connection-level progress handler never gets a chance to run. Only the progress= argument to backup() is invoked on those retries.

    Suggested addition to the Connection.backup() docs:

    backup() may block indefinitely when the source database is locked by another connection; the timeout passed to connect() does not bound this wait. If a bounded wait is required, pass a progress callback that checks time.monotonic() and raises when a deadline is reached. Note that Connection.set_progress_handler() is not called during backup()'s retries on a locked source.

  2. picnixz commented on Nov 22, 2026

    @picnixz
    Member

    I think this is the expected behavior and that the docs are sufficiently clear. At least, this is wgat I would expect if something is locked.

  3. daneeller commented on Nov 22, 2026

    @daneeller
    Author

    Thanks for the clarification — that makes sense, I'll keep the progress-callback workaround on my side rather than pushing for a doc change. Appreciated you taking the time.

  4. picnixz commented on Nov 22, 2026

    @picnixz
    Member

    That being said, we may consider making the timeout handled by the backup retry loop. Or add another timeout parameter to backup().

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions