How we cut p95 latency by 43% using anycast routing

We traced high time-to-first-frame for European users to suboptimal BGP path selection on upstream transit providers. Here's what we found and how we fixed it.

---

At LightStream we track time-to-first-frame (TTFF) as our primary quality-of-experience metric. A video player that takes more than two seconds to start playing loses a measurable share of its audience before the first frame appears — and that loss compounds at scale.

In January we noticed something worrying in our RUM data: p95 TTFF for users in Germany and Poland had crept to 3.1 seconds, while our US East users were consistently at 0.8 seconds. Both groups were hitting edge nodes within 20 ms of them geographically, so the culprit wasn't physical distance.

Following the packets

We added detailed traceroute logging to our player SDK and collected data from 12,000 affected sessions over 72 hours. The pattern was clear: traffic originating from DE-CIX and AMS-IX was making unnecessary hops through our US backbone before reaching our Frankfurt PoP, adding 80–120 ms of round-trip time.

The root cause was a BGP community misconfiguration on two of our upstream transit providers. We had inadvertently allowed them to re-advertise our prefixes with lower local-preference values, causing distant routers to prefer longer paths to our origin instead of routing to the nearest edge.

The fix

We corrected the community strings on all four of our European PoP announcements and added explicit no-export policies to prevent prefixes that should be served locally from leaking to other regions. We also deployed a BGP path monitoring pipeline built on route-collector feeds combined with synthetic traceroute probes that fire every five minutes from 18 vantage points.

Within 48 hours of the fix, p95 TTFF for European users dropped from 3.1 s to 1.8 s — a 43% improvement. We've since applied the same audit to our Asian PoPs and resolved two smaller path-selection issues that had been quietly adding 25–40 ms for users in South Korea and Japan.

What we'd do differently

• Establish BGP community baselines at PoP launch time, not retroactively. A routing misconfiguration can persist for months before it shows up in aggregate performance metrics.
• Instrument the player with routing metadata from the first request. CDN performance logs alone don't tell you whether traffic arrived at the right edge node.
• Treat anycast as requiring continuous validation, not a one-time configuration. BGP routes change as transit providers update their policies.

← Back to blog