Your tracker says a keyword is "not found." You open an incognito window, set the location, and there it is at position six. Nothing happened to your page. Something happened to the check.
The takeaway before the detail: rank tracking is automated search at scale, the precise behaviour anti-bot systems exist to stop, so being challenged isn't a malfunction — it's the expected state. The real problem is that a blocked check rarely looks like a failure. It comes back as a 200 OK with no results in it, your pipeline parses zero positions and writes down "not ranking," and every trend line then sits on an unknown number of silent holes.
What your tracker looks like from the other side
Stop thinking about "getting caught doing something wrong" and think of it as a scoring system. Anti-bot layers assemble a confidence score from dozens of cheap signals, and automated position checking scores badly on most of them by its very nature.
The connection fingerprint. Before a byte of your request is read, the TLS handshake has already described you: cipher suite order, extensions, elliptic curves. That combination (commonly hashed as a JA3/JA4 fingerprint) differs between Chrome, curl, Python's requests, and a Go client. Headers then confirm it — browsers send a predictable set in a stable order (sec-ch-ua, sec-fetch-site, accept-language) with that browser's exact values, while libraries send fewer, differently ordered, often with a giveaway Accept: */*. Announcing Chrome in your headers while your handshake says Python is a contradiction the scorer notices immediately.
IP reputation and ASN. Datacenter ranges from the big cloud providers are trivially identifiable by ASN, and they carry history: if hundreds of other scrapers have run through that subnet, you inherit their reputation the moment you rent the IP.
Behavioural rhythm and query pattern. Humans are irregular — they pause, re-query, mistype, and load a page's images and fonts. A tracker fires a query, parses the HTML, and immediately fires the next, with no sub-resources and near-identical intervals. Add two hundred unrelated commercial keywords from one location, all paginated, none ever clicking a result: no single signal is damning, but the combination has no plausible human explanation.
Missing JavaScript state. Many challenges never render a puzzle. They ship a script that measures the environment — canvas rendering, timing quirks, whether navigator.webdriver is set — and post the result back. A plain HTTP client never runs it, so the token is absent, and absence is itself an answer.
You won't pass all of these, and that isn't the goal. The goal is knowing when you failed them, because that's where the data corruption starts.
Why blocked checks go missing instead of erroring
If blocks returned HTTP 429 or 403 every time, this would be a solved problem — your client would raise an exception, log it, and retry. Modern defences avoid that deliberately: a clean error code is free feedback telling an automated system exactly what to change. Ambiguity is more effective.
So instead you get a 200 OK containing a challenge, an interstitial, a consent wall, a redirect to an "unusual traffic" page, or — most insidiously — a degraded result page: correctly structured but thinner, reordered, or missing the SERP features you were counting. The HTTP layer reports success, your parser finds nothing matching the result selector and returns an empty list, and upstream that empty list becomes a row in your database.
That row then takes one of four shapes, and each is a lie with a different flavour:
- Position 0 or 101 — reads as a catastrophic drop and fires a false alert. Someone spends an afternoon on a page that never moved.
nullsilently skipped — today's average now measures a different keyword set than yesterday's, so the portfolio "improves" because the hard keywords dropped out.- Carried forward from yesterday — the worst one: it papers over a real decline, so the alert that should have fired never does.
- Row dropped entirely — the chart interpolates across the gap, drawing a smooth line through data that doesn't exist.
This is the failure mode that makes rank movement hard to read honestly, so the method for separating real movement from measurement artefacts is worth having first — see is that ranking change real or just noise. A proxy pool or a solving step only helps once you can tell a bad check from a bad week.
The fix is a response classifier, not a better parser
The highest-value change to a homegrown tracker is refusing to let any response reach the parser unclassified. Before extracting positions, decide what you're holding:
def classify(resp):
"""Return one of: OK, CHALLENGE, RATELIMIT, CONSENT, EMPTY, MALFORMED."""
body = resp.text
if resp.status_code in (403, 429, 503):
return "RATELIMIT"
# Challenge widgets and interstitials often arrive with a 200.
for marker in ("g-recaptcha", "cf-turnstile", "/sorry/", "unusual traffic",
"challenge-platform", "hcaptcha"):
if marker in body:
return "CHALLENGE"
if "consent." in str(resp.url):
return "CONSENT"
results = extract_results(body)
if not results:
# Zero results from a page that otherwise looks normal is suspicious,
# not authoritative. Never write this as "not ranking".
return "EMPTY" if looks_like_serp(body) else "MALFORMED"
return "OK"
Then store the classification beside the position, and let the schema force the distinction:
ALTER TABLE rank_checks
ADD COLUMN outcome ENUM('ok','challenge','ratelimit','consent','empty','malformed'),
ADD COLUMN position INT NULL; -- NULL means unknown, never "not ranking"
Now "position unknown" and "position 101" are different facts, charts can show coverage alongside rankings, and you can compute the one number that says whether you have a blocking problem: completion rate, clean reads divided by scheduled checks, per day and per target. At 96% you're fine. At 70%, nearly a third of your trend line is fiction.
What actually changes when you run your own tracker
Teams build their own for locations a vendor doesn't offer, or competitor coverage on a budget. Legitimate reasons — but be clear about what you're taking on, ordered by how much time each consumes:
- The fingerprint becomes your maintenance burden. Vendors amortise it across every customer; you carry it alone, and it changes without notice. First because it's the cost people most underestimate.
- You own IP acquisition and rotation. Reputable rotating or residential proxies at a respectful rate — not the cheapest pool available, which is cheap because it's already burnt.
- You own cadence policy. Positions move over days, not minutes. Twice daily on a jittered schedule gives a trend as usable as hourly hammering for a fraction of the challenges — slower is frequently more accurate.
- You own challenge handling. Some checks get challenged however well-behaved you are, and each has two honest outcomes: complete it, or record it as unknown.
Before any of that, the cheapest wins don't involve reading a live SERP. Search Console and Bing Webmaster Tools give query-level position data for sites you own, aggregated across real users — no scraping, no challenges, and arguably truer than one machine's view from one location. Reserve direct reading for what they can't tell you: competitor positions, specific locales, SERP-feature presence. And read the target's robots.txt and terms of service first; respecting rate limits and stated rules is the line between measurement and abuse.
Handling the challenges you still hit
For the checks that are legitimate, respectful, and still challenged, a solving service turns a hole in your dataset into a completed read. CaptchaAI is worth knowing about here for one practical reason: it exposes a proxy-per-task parameter, which matters for rank tracking specifically — the challenge has to be solved from the same egress the check runs through, or the token won't validate. It also speaks the legacy 2Captcha-shaped protocol, so an existing integration is largely a host swap. The flow is a submit and a poll against ocr.captchaai.com:
curl -s "https://ocr.captchaai.com/in.php" \
-d "key=YOUR_API_KEY" \
-d "method=userrecaptcha" \
-d "googlekey=SITE_KEY_FROM_PAGE" \
-d "pageurl=https://example.com/results" \
-d "proxy=user:[email protected]:8080" \
-d "proxytype=HTTP" \
-d "json=1"
# {"status":1,"request":"2122988149"}
# 2. Poll roughly every 5s until it stops returning CAPCHA_NOT_READY.
curl -s "https://ocr.captchaai.com/res.php?key=YOUR_API_KEY&action=get&id=2122988149&json=1"
# {"status":0,"request":"CAPCHA_NOT_READY"}
# {"status":1,"request":"03AGdBq26...token..."}
Two notes that save debugging time. Always send json=1; without it you get legacy plain-text responses that are painful to parse reliably. And treat the documented error codes as branches, not exceptions: CAPCHA_NOT_READY means keep polling, ERROR_UNSOLVABLE means record the check as challenge and move on, and ERROR_ZERO_BALANCE means your monitoring is silently degrading right now and should page someone.
The cost model is unusual for this category: pricing is thread-based — you buy concurrent threads rather than individual solves, unlimited solves per thread, no per-CAPTCHA fee, no surcharge by challenge type, with published tiers from BASIC $15/month for 5 threads up to ENTERPRISE $300/month for 200. That suits a scheduled tracker, because concurrency is the thing you can predict: a nightly run of N parallel workers needs N threads, and the bill doesn't move when a target starts challenging harder. Its published per-type figures include >99.5% success under 60s for reCAPTCHA v2 and 100% under 10s for Cloudflare Turnstile, the latter returning a clearance cookie plus the matching user agent — vendor claims, so benchmark them on your own targets.
One boundary, stated plainly: this recovers reads that were blocked. It does not invent positions for keywords you never reached, and it has no business anywhere near authentication, account creation, or paywalls. If a check didn't complete, the dataset says unknown — coverage you can defend beats coverage that looks good.
FAQ
Why does my rank tracker return "not found" for a keyword I can see ranking?
Almost always because the check was challenged rather than answered: the response arrived with a 200 status but held an interstitial or a stripped-down page, your parser found no results, and the empty list was recorded as "not ranking." Classify responses before parsing and the distinction becomes visible immediately.
Do proxies alone fix rank tracker blocking?
No. Proxies address IP reputation, one signal among many. If your TLS fingerprint, header order, and request rhythm still read as automated, rotating addresses just spreads the same detectable pattern across more IPs. Pace and fingerprint consistency matter at least as much.
How often can I check rankings without getting blocked?
There's no published safe number, and anyone quoting one is guessing. The direction is reliable, though: positions move over days, so a jittered once- or twice-daily cadence provokes far fewer challenges than hourly checking and produces an equally usable trend. Tune it against your own completion rate.
Is scraping SERPs for rank tracking allowed?
It depends on the target's terms of service and robots.txt — those are the documents to read, not a blog post. For your own site, Search Console gives position data through a sanctioned interface with no scraping at all. For everything else, stay within stated rules and rate limits; measurement at a respectful pace is a different activity from circumventing protections for fraud or abuse, and only the former belongs in an SEO workflow.
Start with the number you don't have yet
Most trackers report positions and nothing about how many checks succeeded. Add that first: classify every response, store unknown as NULL rather than zero, and put completion rate on the same dashboard as the rankings. The blocking problem is usually there already, quietly flattening your trend lines.
Then fix it in cost order — sanctioned APIs where they exist, a calmer jittered cadence, clean rotating IPs — and route whatever still gets challenged through a solving step so the check completes instead of leaving a hole. To size that last piece, CaptchaAI is a reasonable place to benchmark: run one scheduled batch with it and one without, then compare completion rates before committing. And if maintaining that pipeline isn't where you want your time going, track your keyword rankings daily with SBRanker and let collection be someone else's problem.
