TikTok Data Backup and Disaster Recovery for Production Apps

## Backup vs Disaster Recovery for TikTok-Data Apps Backend teams running production TikTok scrapers blur two concepts that should stay separate. **Backup** is the act of copying data to a second location so you can recreate state after corruption or loss. **Disaster recovery (DR)** is the documented process of rebuilding service after that loss, with measured time and data targets. A team can have nightly database snapshots and still have no DR plan, because nobody has rehearsed the restore. For TikTok-data products the picture is unusual. Most of your business value sits in **derived data** computed from TikTok responses: engagement timelines, hashtag rankings, creator graphs, music trend tables. The source (TikTok itself) is not under your control and is not addressable for restore. If a creator deletes their account today, no backup of TikTok exists for you to query tomorrow. The only copy of that data that will ever exist is the one you took when you had the chance. That single fact reshapes the entire backup strategy: your **raw API response archive** becomes your most important asset, sometimes more important than your warehouse, because the warehouse can be recomputed from the archive but the archive can never be recomputed from anywhere. ## What You Can Never Recover From TikTok It is worth writing this list on a wall. - **Deleted videos.** Once a creator deletes a post, `/post-detail/` returns a not-found state. Play counts, caption text, music binding, the original `hdplay` URL: gone unless you snapshotted them. - **Deleted accounts.** When a user is removed, `/userinfo-by-username/` stops resolving. Their `secUid`, follower history, and bio link disappear from the live source. - **Privacy flips.** A user toggling liked videos to private kills future `/user-liked/` results. Past snapshots are the only record. - **Edited captions and bios.** TikTok keeps no changelog. If you did not record yesterday, yesterday is gone. - **Comment deletions.** A removed comment vanishes from `/post-comments/` and from comment-reply trees. What stays archivable forever is **your own copy** of what was returned during the window when those entities were public. That is the entire backup target. ## The Three Things to Back Up ### 1. Raw API responses (cheap insurance) Write every successful response to object storage before you parse it. Compressed JSON is small, schema-flexible, and replayable.

import gzip, json, os, datetime as dt, requests, boto3

API = "https://api.tikliveapi.com"
KEY = os.environ["TIKLIVE_KEY"]
s3  = boto3.client("s3")

def fetch_and_archive(endpoint: str, params: dict, slug: str):
    r = requests.get(f"{API}{endpoint}",
                     params=params,
                     headers={"X-Api-Key": KEY},
                     timeout=20)
    r.raise_for_status()
    body = r.json()
    today = dt.date.today().strftime("%Y/%m/%d")
    key = f"tiklive-raw/{today}/{endpoint.strip('/')}/{slug}.json.gz"
    s3.put_object(
        Bucket="tiklive-archive",
        Key=key,
        Body=gzip.compress(json.dumps(body).encode()),
        ContentType="application/json",
        ContentEncoding="gzip",
    )
    return body

Every call costs you a fraction of a cent in storage and gives you a perfect replay source. ### 2. Derived data in your warehouse This is whatever you compute from the raw responses: Postgres tables, ClickHouse rollups, BigQuery marts. Your cloud provider's standard snapshot mechanism covers this. Schedule daily snapshots, retain 30 days, and replicate at least one to a second region. The warehouse is **reproducible** from the raw archive, so this layer is convenience, not insurance. ### 3. Ingestion code and cursor state The third layer is the part teams forget. Your scrapers carry state: the last cursor returned by `/user-posts/`, the last `time` value from `/user-followers/`, the last `cursor` from `/post-comments/`. Lose those and you either re-pull everything (expensive) or skip a window (data gap). - Code in **Git**, tagged per deploy. - Cursors and ingestion bookmarks in **Redis**, with `BGSAVE` snapshots shipped to S3 hourly.

import redis, boto3, time
r  = redis.Redis()
s3 = boto3.client("s3")

def snapshot_cursors():
    r.bgsave()
    time.sleep(5)
    with open("/var/lib/redis/dump.rdb", "rb") as f:
        s3.put_object(Bucket="tiklive-archive",
                      Key=f"cursors/{int(time.time())}.rdb",
                      Body=f.read())

## RPO and RTO Targets - **Recovery Point Objective (RPO):** how much data you can afford to lose. For TikTok-data products **24 hours is usually acceptable**, because TikTok itself is the upstream and any gap can be re-pulled (modulo deletions, see above). - **Recovery Time Objective (RTO):** how fast user-facing endpoints must come back. **1 hour** is a common bar for a paid dashboard. Backend batch jobs can tolerate 4-8 hours. If your contract or customer expectation is tighter than 24h RPO, increase archive cadence and snapshot frequency. Tighter than 1h RTO usually means a hot standby, not just backups. ## Archive Layout Use date partitions in the key prefix so you can replay a single day cheaply and so lifecycle rules can age data out.

s3://tiklive-archive/tiklive-raw/2026/05/29/userid/cristiano.json.gz
s3://tiklive-archive/tiklive-raw/2026/05/29/user-posts/107955-cursor0.json.gz
s3://tiklive-archive/tiklive-raw/2026/05/29/post-detail/7234.json.gz
s3://tiklive-archive/tiklive-raw/2026/05/29/post-comments/7234-cursor0.json.gz
s3://tiklive-archive/tiklive-raw/2026/05/29/user-followers/107955-time0.json.gz

Two notes that match the wire format: `/user-followers/` paginates with a `time` parameter (not `cursor`), and `/user-following/` returns its list under the key `followings` (with the trailing s). Bake these into your archive slugs so a replay knows which paginator to use. `/post-comments/` items carry their identifier under the field `id`, and `/post-detail/` flat responses include `hdplay` alongside `play` and `wmplay`; preserve all three when you snapshot, because TikTok's CDN URLs expire even when the video does not. ## Backup Verification A backup you have never restored is a rumour. Run a **nightly restore test**: pick a random day from the last 30, replay it into a scratch database, and assert that row counts match the warehouse to within tolerance.

import random, datetime as dt, gzip, json, boto3, psycopg2
s3 = boto3.client("s3")

def verify_random_day():
    day = dt.date.today() - dt.timedelta(days=random.randint(1, 30))
    prefix = f"tiklive-raw/{day:%Y/%m/%d}/"
    paginator = s3.get_paginator("list_objects_v2")
    count = 0
    for page in paginator.paginate(Bucket="tiklive-archive", Prefix=prefix):
        for obj in page.get("Contents", []):
            blob = s3.get_object(Bucket="tiklive-archive", Key=obj["Key"])["Body"].read()
            json.loads(gzip.decompress(blob))  # parse must not throw
            count += 1
    assert count > 0, f"empty archive for {day}"
    print(f"verified {count} objects for {day}")

Wire this to an alert. A silent backup that started failing three weeks ago is the second-worst incident class after losing the production database. ## DR Scenarios **1. Your database fails.** Restore the latest warehouse snapshot. Identify the time window between the snapshot and the failure. Replay the raw archive for that window through your normal parser. Cursor state from Redis snapshots tells you where the resumed live ingestion should pick up. **2. Your TikLiveAPI key is revoked.** Sign in at [/profile/](/profile/) and rotate the key. Verify the new key with a one-shot call to `/userid/` (cheap, single result). Update the secret in your secret manager, restart workers, confirm `/userinfo-by-id/` returns a `user` and `stats` block. See [user docs](/documentation/users/) for the full request shape. **3. Your ingestion pipeline drifts.** Symptoms: cursor stuck, duplicate rows, or a worker silently caught in a `hasMore=true` loop. Stop the worker. Pull the last cursor snapshot from Redis that matches a known-good run (compare timestamp and row counts). Reset the worker to that cursor and let it walk forward. The raw archive guarantees you can identify the last clean state. ## Retention and Right to Be Forgotten GDPR Article 17 gives EU users the right to erasure. Your archive is in scope. Two mitigations: - **Index your archive by TikTok user id.** When you receive a deletion request, you must be able to locate and remove every object referencing that id. Without an index you cannot comply. - **Adopt a tiered retention policy.** Hot raw archive at 30 days, warm at 12 months, cold thereafter. Apply this via S3 Lifecycle rules. Document the policy publicly on [/privacy/](https://www.tikliveapi.com/privacy/). Make sure the policy is the same in the warehouse and in the raw archive. A deletion that succeeds in Postgres but leaves the JSON in S3 is still a violation. ## Legal Hold Litigation, subpoena, or a regulator inquiry can override your retention policy and require you to **preserve data you would otherwise delete**. Build a legal-hold flag into your retention job that suspends lifecycle rules for a prefix or a tag. Document the flag in your runbook. Separately, your archive must respect TikTok's own ToS: do not redistribute raw responses, keep access internal, and never expose the archive as a public mirror. ## Choosing an S3-Compatible Store For TikTok archive workloads the choice usually comes down to egress cost, since you write a lot and read rarely except during DR drills. - **AWS S3.** Highest egress cost, deepest ecosystem, easiest IAM integration if the rest of your stack is on AWS. - **Cloudflare R2.** Zero egress fee. Strong fit if your warehouse or restore target lives outside AWS, or if DR drills are frequent. - **Backblaze B2.** Cheapest at-rest cost, simple S3-compatible API, lowest cost for write-mostly workloads. Slower than the other two for high concurrency restores. A common pattern: primary archive in R2 (cheap egress for restores), secondary copy in B2 (cheapest insurance), both fed by the same boto3 client with different endpoint URLs. ## Small SaaS DR Plan Template A worked example for a two-engineer team running a TikTok analytics dashboard: - **RPO:** 24h. **RTO:** 2h for dashboard, 8h for batch. - **Raw archive:** R2 bucket `tiklive-archive`, date-partitioned, gzipped. - **Warehouse:** Postgres on a managed provider, nightly snapshot, 30-day retention, replica in second region. - **Cursor state:** Redis with hourly `BGSAVE` to R2 under `cursors/`. - **Code:** GitHub, tagged on every prod deploy. - **Verification:** nightly restore test of one random day, alerts to PagerDuty. - **Runbook:** living doc covering the three DR scenarios above with exact CLI commands. Reviewed quarterly. Test restore once per quarter. - **Key rotation:** documented in runbook, tied to [/profile/](/profile/) self-service rotation. New key validated with [/documentation/users/user-id/](https://www.tikliveapi.com/documentation/users/user-id/). - **Pricing math:** budget for archive storage scales with daily call volume. Sketch it against your plan at [/pricing/](/pricing/) and revisit when you cross 1M calls per day. Before you ship the plan, walk every page on [/documentation/](/documentation/), confirm your archive captures the response shape for each endpoint you depend on (especially [/documentation/posts/detail/](https://www.tikliveapi.com/documentation/posts/detail/) with `hdplay`, and the two non-obvious paginators in [/documentation/users/followers/](https://www.tikliveapi.com/documentation/users/followers/) and [/documentation/users/following/](https://www.tikliveapi.com/documentation/users/following/)), and dry-run a single-day replay end to end. ## FAQ **Do I really need both warehouse snapshots and a raw archive?** Yes. The warehouse snapshot restores quickly but only contains derived data shaped by the current parser. The raw archive lets you reprocess history when your schema changes, and is the only record of entities later deleted on TikTok. **How long should I keep raw JSON?** At least 12 months for trend analysis, longer if you sell historical reports. Anchor the answer in your privacy policy and your customer contracts, not in storage cost. **Can I skip Redis snapshots if I store cursors in Postgres?** Yes. The point is durability of cursor state, not the specific store. Pick one place, snapshot it, and document which one is authoritative. **What about live data and the playground?** The interactive [/playground/](/playground/) is for ad-hoc testing and is not a backup tool. Production ingestion must hit `https://api.tikliveapi.com` directly with `X-Api-Key` and write to your own archive. **Who do I contact if the API key rotation is stuck?** Use [/contact/](/contact/) for support, and check [/blog/](/blog/) for any active incident notes before opening a ticket.