Capacity Planning Playbook for TikTok Data Pipelines

Most TikTok data pipelines do not fall over because of a bug. They fall over because somebody forgot to do arithmetic. The team ships a new feature, traffic doubles, the credit pack empties on day 22, the DB volume hits 90 percent on a Saturday, and the on-call engineer spends the weekend pasting numbers into a spreadsheet that should have existed a quarter earlier.

This post is that spreadsheet, written out. It walks through the four resource bottlenecks every TikTok pipeline hits, how to forecast each one from feature usage patterns, and how to set early warning thresholds so capacity reviews stop being fire drills. It is aimed at data engineers and platform engineers who are running pipelines on top of TikLiveAPI in production.

The Four Resource Bottlenecks

Capacity planning fails when teams treat the system as one big bucket. It is not. A TikTok data pipeline has four independent resources, each with its own cost curve, its own refill mechanic, and its own failure mode. Treat them separately.

1. TikLiveAPI Credits

Credits are the easiest resource to model because they are discrete and metered. Every call to an endpoint such as /userinfo-by-username/ or /user-posts/ consumes credits, and the meter resets when you buy a new pack on /pricing/. The hard part is that credit consumption is not uniform across features. A user profile sync hits one endpoint per tenant per day. A "scan every video for a hashtag" feature hits /user-posts/ and then /post-detail/ per video. The second pattern can burn ten thousand credits for one tenant in an afternoon.

2. Database IOPS

The second bottleneck is usually invisible until it bites. Most teams size their database on storage and CPU and forget that snapshot inserts are write-heavy. If you write a row per user per day into a user_snapshots table, you are doing one insert per active tenant per scheduled run, plus index maintenance. At ten thousand tenants on a five-minute cadence, that is more sustained write throughput than a default RDS instance comfortably handles.

3. Queue Depth

Whether you use SQS, RabbitMQ, Redis lists, or Kafka, the queue between your scheduler and your workers is a buffer with a finite ceiling. Queue depth is a leading indicator. When it grows faster than workers can drain it, every downstream SLA breaks in the order they were written.

4. ASR and LLM Compute

For advanced apps that transcribe TikTok audio or summarize comments with an LLM, the GPU or model-API budget is often the most expensive resource in the stack. A single hour of Whisper-large on a busy account can cost more than a month of TikLiveAPI credits. Plan it separately.

Forecasting Credit Consumption

The trick to forecasting credits is to project bottom-up from feature usage patterns, not top-down from "we used 800k last month." Top-down forecasts hide the tenants who are about to triple.

Build a small model with three inputs per feature: the endpoint it calls, the per-tenant call rate, and the active-tenant count. Then sum across features. A worked example for a typical multi-feature product:

Feature              Endpoint                       Calls/tenant/day   Active tenants   Daily credits
Profile sync         /userinfo-by-username/         1                  4,800            4,800
Video discovery      /user-posts/                   4                  4,800            19,200
Watermark removal    /post-detail/                  30                 1,200            36,000
Comment ingest       /post-comments/                12                 800              9,600
                                                                                        -------
Total daily                                                                              69,600
Monthly projection   x 30                                                                2,088,000

This is the per-tenant projection. The valuable column is "calls per tenant per day" because it is the one that grows when product ships a new feature. Track it per feature in a dashboard, not just at the global level. If watermark removal goes from thirty calls to forty-five calls per tenant after a UX change, that is a 50 percent increase on the largest line item and you need to know within hours.

DB Sizing for Snapshot Tables

Snapshot tables grow linearly in time and linearly in tenants. The combined growth is the product. A row in user_snapshots typically holds the flattened fields from /userinfo-by-username/ response: id, uniqueId, nickname, followerCount, followingCount, heartCount, videoCount, plus a timestamp.

Call that row 400 bytes on disk after indexes. For ten thousand tracked TikTok accounts snapshotted daily, that is 4 MB per day, 1.4 GB per year. Cheap. For ten thousand tracked accounts snapshotted hourly, it is 33 GB per year. Still fine. For one hundred thousand accounts snapshotted every fifteen minutes, you are at 1.4 TB per year and you need partitioning, retention policy, and an answer to the question "do we actually query data older than 90 days?"

Three rules that keep snapshot tables sane:

• Partition by date from day one. Postgres native partitioning or a time-bucketed table per month. Retroactive partitioning is painful.
• Decide retention up front. If product only displays a 30-day chart, archive everything older to S3 as Parquet.
• Compute aggregates ahead of time. A daily rollup table with followers_added_today is two orders of magnitude smaller than the raw snapshot and answers 90 percent of dashboard queries.

Kafka Topic Sizing if Streaming

If you pipe TikTok events through Kafka, two numbers matter: peak throughput per partition and retention window. A partition handles a few MB per second comfortably. A single TikLiveAPI response from /user-posts/ serialized as JSON sits around 30 to 80 KB depending on the video count. At a thousand fetches per minute, you are at about 1 MB per second per partition, which is fine for one partition but leaves no headroom for replay.

Size partitions for peak throughput plus replay traffic, not steady state. If you ever need to rebuild a downstream materialized view, you will read the topic from the earliest offset at maximum consumer throughput, which can be five to ten times steady-state write rate. The retention window is the other lever. Seven days of retention on a 1 MB per second topic is roughly 600 GB. That is real money on managed Kafka.

Autoscaling Worker Pools

Workers that consume the queue should autoscale on queue depth, not on CPU. CPU-based autoscaling for an IO-bound TikTok worker is the wrong signal because the worker spends most of its life waiting on a TikLiveAPI response. Queue depth tells you the truth: if it is growing, add workers. If it is shrinking faster than your target drain rate, remove them.

A simple target-tracking policy: keep queue depth below the number that gives you a five-minute drain time at current worker throughput. If a worker processes one job per second and you have ten workers, you can drain 3,000 jobs in five minutes. Scale up when queue depth exceeds 3,000.

Set a hard ceiling on the worker pool. Without one, a runaway tenant can pull the cluster into a state where the workers themselves become the bottleneck on TikLiveAPI rate limits, and every other tenant suffers. The ceiling is your circuit breaker.

Scenario Planning

Capacity reviews should rehearse two scenarios on a regular cadence: 10x customer growth and 100x customer growth. The first is the realistic ambition for the next year. The second is the "what if we get featured" stress test.

10x Growth Scenario

Take the daily credit projection, multiply by ten, and check that it fits inside the largest plan on /pricing/ or that the math works for stacked packs. Take the daily snapshot row count, multiply by ten, and check that the partition strategy still works at one year of retention. Take the peak queue depth, multiply by ten, and check that the autoscaling ceiling can drain it inside the SLA.

100x Growth Scenario

This is the scenario where you find the real bottleneck. At 100x, credit packs need pre-purchase planning weeks in advance. The DB needs a sharding strategy or a move to a different engine. The queue probably needs to be split by tenant tier so a single heavy tenant cannot starve the small ones. ASR and LLM compute needs a per-tenant budget enforced at the application layer, because there is no plan you can buy that absorbs 100x audio transcription for free.

You will not implement the 100x plan today. The point of running the scenario is to know which decisions are reversible and which are not. Sharding choices are not reversible cheaply. Queue topology is. Make the irreversible decisions early.

Early Warning Thresholds

Forecasting only helps if alerts fire before the system breaks. Three thresholds that have saved real outages:

• Credit budget at day 20. If you have burned 75 percent of the monthly credit pack by day 20, you will overshoot. Alert and trigger a top-up review. Do not wait for the pack to empty before deciding.
• DB free space at 30 percent. Not 10 percent. By the time you hit 10 percent, you do not have time to provision a bigger volume safely.
• Queue depth above five-minute drain. If queue depth exceeds what your current worker count can drain in five minutes, page someone. This is the most actionable signal because the response is always the same: scale workers, throttle ingest, or kill a runaway tenant.

Pair the credit threshold with the /status/ page for any infrastructure incidents on the API side. Sometimes the credits-per-call ratio is healthy and the problem is retry storm against a degraded upstream. Knowing which one you are in changes the response.

Pre-Purchasing Credit Packs for Predicted Spikes

Credit packs are cheaper per credit at higher tiers. If your forecast says you will burn 2 million credits next month with 80 percent confidence, buying a pack sized for 2.5 million now is cheaper than buying the base pack and topping up twice mid-month. Pre-purchasing also smooths cash flow projections for finance, who tend to prefer one large monthly line item over five unpredictable ones.

Two events justify pre-purchasing above the steady-state forecast: a product launch that touches a credit-heavy endpoint, and a known marketing push that will spike new tenants. Both are predictable a week or more in advance. Add 20 to 30 percent buffer for either and revisit pack sizing on /pricing/ before the spike, not during it.

Quarterly Capacity Reviews

The discipline that holds all of this together is the quarterly capacity review. Sixty minutes, four people in a room or on a call, one document. The document covers: actual versus forecast credit consumption per feature, DB growth versus projection, queue depth percentiles for the quarter, ASR and LLM spend per tenant, and the two scenario projections updated with new tenant counts.

The output of the review is three decisions: what to pre-purchase, what to rearchitect, and what to monitor more closely. Anything that does not become one of those three is noise. Send the document to engineering and finance the same day. Finance cares more about capacity planning than engineering usually realizes, and looping them in turns "we need a bigger DB" into a budget line, not a fight.

Cost of Capacity vs Cost of Overflow

Every capacity decision is a tradeoff between paying for headroom you might not use and paying for an outage you might not have. The right ratio depends on the business. A B2B SaaS with annual contracts and SLA penalties should overprovision aggressively because an overflow costs customer trust plus contractual credits. A free-tier product can run hot because the overflow costs a Twitter complaint.

A practical heuristic: if the cost of one hour of downtime exceeds one month of the headroom buffer, buy the headroom. If it does not, run hot and invest the difference in better autoscaling. Most teams err on the side of running too hot because the cost of headroom is visible in the bill and the cost of overflow shows up as engineer hours that do not have a line item.

Sample Capacity Planning Spreadsheet

The spreadsheet that runs all of this has six tabs. Tab one is the feature-to-endpoint mapping, with one row per feature, its endpoint, and current calls per tenant per day. Tab two is tenants by tier, with current count and projected count at 30, 60, and 90 days. Tab three multiplies the two and produces a daily credit projection per feature. Tab four is the DB sizing model, with rows per day, bytes per row, and rollup ratios. Tab five is the queue and worker sizing model, with peak throughput and drain time calculations. Tab six is the scenario runner, which takes a multiplier and applies it to tabs three through five. Test it in the /playground/ by running a sample feature for a single tenant and feeding the measured credit cost back into tab one. The model is only as good as that one calibration.

FAQ

How often should we recalibrate the credit forecast?

Monthly at minimum, weekly during a product launch. Every shipped feature changes calls per tenant per day. If you only recalibrate quarterly, you will be three months behind reality by the time the forecast looks wrong.

Should we cache TikLiveAPI responses to reduce credit consumption?

Yes for slow-changing endpoints like /userinfo-by-username/ and /userinfo-by-id/ where a 15-minute cache is fine for most product features. No for /post-comments/ where freshness is the value. The decision belongs in the feature spec, not the infrastructure layer.

What is the right alert threshold for credit burn rate?

Two thresholds. A soft alert at 75 percent of budget by day 20 of the month for the review trigger. A hard alert at 90 percent at any point in the month for the immediate top-up decision.

How do we plan for ASR and LLM compute if usage is bursty?

Run it on a separate budget from steady-state pipeline cost and meter it per tenant. A per-tenant quota that the application enforces is the only way to prevent one tenant from consuming the entire monthly LLM budget in one weekend.

Where do we go for help if a capacity prediction starts going sideways?

Send a message via /contact/ with your forecast and current burn rate. Capacity planning questions get faster answers than "the API is slow" tickets because they come with numbers attached.

The pipelines that survive growth are the ones with a spreadsheet that gets updated before the alerts fire. Build it now, calibrate it monthly, and the on-call weekends get a lot quieter.