Live streaming on AWS: signal, DRM and a load test for two and a half million viewers

Live broadcast is one of the few things in IT you cannot fix after the fact. If the picture freezes during a match, nobody cares that everything was back half an hour later. Which is why a working delivery chain does not get swapped for a new one lightly, and rightly so.

Our consultants worked on a project where the question was framed differently. Not “should we replace a working chain”, but “can we build a second one in the cloud, run it in parallel and move traffic onto it whenever we choose”. That is a question a proof of concept can answer honestly.

Below we share what we learned: what that chain actually does, where it trips, and what came out of the load test. We do not say for whom or where.

What had to be proven

Four things, in this order:

1. Whether the studio signal reaches the cloud and comes back out as an adaptive stream without losing broadcast quality.
2. Whether we can apply the DRM that the rights department requires, and whether the player will decode it.
3. Whether the player survives viewers arriving in the millions over a quarter of an hour.
4. Whether traffic can be switched between the cloud chain and the existing one on demand, at a set hour.

Points one and four went smoothly. Point two ate the most time. Point three taught us something about load tests rather than about the system.

The signal: MediaConnect, MediaLive, MediaPackage

The AWS Elemental chain looks familiar to anyone who has worked with hardware encoders, except the boxes now sit in the cloud.

MediaConnect takes the signal from the studio. In our case over SRT, a protocol built for the public internet: it runs on UDP but recovers lost packets itself and encrypts the transport. Access is entitlement based, so knowing the address is not enough to receive the stream.

MediaLive is the encoder. This is where a decision is made that follows you through the whole project: the channel class. A STANDARD channel is two independent pipelines in two availability zones with automatic failover between them. A SINGLE_PIPELINE channel is one. The price difference is roughly double, and it cannot be changed after the channel is created, only by recreating it. Primary chain STANDARD, backup chain SINGLE_PIPELINE, unless the event is premium.

The second decision is the ABR ladder, the set of quality renditions. We used four: 1080p at 6 Mbps, 720p at 3 Mbps, 480p at 1.5 Mbps and 240p at 700 kbps. That ladder is a fixed cost. MediaLive bills per channel running hour, not per minute watched, so a channel left on overnight costs the same with no viewers as with a full house.

MediaPackage packages the encoder output on demand, into HLS and DASH at once, and acts as the origin for the CDN. Just in time packaging means we do not hold two full sets of segments, only one CMAF set from which manifests are generated on the fly. The same component handles content encryption, which comes next.

One thing worth knowing up front: between MediaLive and MediaPackage we send CMAF rather than classic HLS with TS segments. That way segments in both chains carry identical names and identical boundaries, which is the precondition for any sensible failover.

Architecture

DRM, the place where a PoC usually stalls

Content protection is the part that occupies one rectangle on the diagram and two weeks in the calendar.

The standard that ties it together is called SPEKE. It is an agreement about how a packager talks to a key provider. MediaPackage asks the key server for the material needed to encrypt a given piece of content, receives the keys, receives a ready PSSH block to place in the manifest, and receives the licence server address. The player then asks that same server for a licence, and only then does the picture start.

The second version of the standard handles multiple keys in a single request, which matters when you want to encrypt video and audio separately, or give different keys to different quality tiers. We encrypted Widevine on DASH in a CMAF container, with one key shared across the ladder.

Four things that consume time without ever being described as difficult:

• Authenticating to the key server. Our provider issued OAuth2 client credentials. Those credentials have to live somewhere, and the only sensible place is Secrets Manager, not an environment variable in a channel definition.
• The player needs a real decryption module. Widevine works in the browser through a CDM, and a CDM requires a secure context. On a test domain without a certificate you will see nothing, and you will spend the first hour looking for a fault in the key configuration.
• You have to be able to see what is happening. We ran a Shaka player with diagnostics on, because without logs from the encryption layer the difference between “no key”, “wrong key” and “PSSH never reached the manifest” looks identical: a black screen.
• Key rotation is tied to failover. If keys change over time, both chains have to change them on the same segment boundary. Which is why the aligned segmentation from the previous section is not cosmetic.

Covering every device really takes three systems at once: Widevine for Android and Chrome, PlayReady for Windows and many television sets, FairPlay for Apple. Each wants a different format, which is why providers sell this as a single multi DRM service. In the PoC we deliberately did one, so as not to confuse an integration problem with a device coverage problem.

A player on a cluster, not on S3

A player is a static file. The natural reflex is to drop it in S3 and serve it through CloudFront, and in many cases that is the right answer.

We ran ours in a container on Amazon EKS in Auto Mode, behind an application load balancer with WAF. The reason is not that a static file needs Kubernetes. The reason is that everything around the static file was not static: stream and licence server addresses varying by environment, custom headers, access rules, and room for logic the team wanted to add later. Once an application stops being one file and starts being a service, a cluster stops being overkill.

Auto Mode removes most of the administrative work. We do not manage node groups or system image versions; the cluster picks and replaces machines under load by itself. Horizontal scaling is the standard Kubernetes mechanism, and traffic arrives through an ALB carrying WAF with rate limits and OWASP rules.

If the player really is one file and nobody is planning logic around it, S3 with CloudFront is cheaper and simpler. It is worth making that decision deliberately rather than inheriting it.

The load test for two and a half million viewers

The goal was to find out whether viewers arriving in the millions over a quarter of an hour would put the application down.

We built the load generator on Locust in distributed mode, 189 worker processes. The scenario mixed several behaviour patterns, because a real viewer does not refresh a page in a loop: a sharp arrival, background traffic, fetching the player page, fetching secondary assets.

The target of two and a half million virtual users was reached in full. The run issued more than 2.4 million requests, of which 12,285 failed, half a percent. Peaks went above three thousand requests per second.

And now the interesting part, the one we did not expect.

The fastest responses came back in 6 milliseconds, the median stayed low throughout, and the 95th percentile climbed into the hundreds of seconds. Responses taking over five minutes. An application that genuinely needed five minutes to return a page would be dead, yet half the requests were coming back instantly.

The conclusion is that at that point the bottleneck stopped being the application and became the generator. At that number of virtual users per worker, Locust cannot service its own coroutines fast enough and counts the wait before issuing a request as part of the response time. What we were measuring was our own tool.

Hence three rules we have applied ever since:

1. The server side tells the truth. Origin request counts, cache hit ratio, status codes and latency from CloudFront, not percentiles from the generator. The generator tells you how much traffic it emitted, not how the system behaved.
2. The generator has to be scaled and measured too. If CPU on the load generating machines is pinned, everything after that point is a story about the generator.
3. A test without video does not test video. Our scenario deliberately did not pull segments, to avoid burning transfer. It was a test of the application layer and has to be reported as one, because a CDN under real consumption behaves differently.

One procedural note as well: large scale load tests are notified to AWS in advance. It is a formality, but done on the day of the test it stops being one.

What the cross-region variant adds

The chain described here was single region, with resilience inside that region, because that was the question asked. For premium services AWS describes a variant that goes further, across two regions at once, and it is worth knowing what it rests on, because it is not simply duplicated resources.

The key is aligning both regions in time. The studio encoder stamps matching timecode on corresponding frames in both chains, and the MediaLive channels run with output locking based on universal time. As a result both regions produce segments with identical boundaries and identical names.

CloudFront performs the switch through an origin group. MediaPackage on the primary chain can report its own bad state and start answering with 404 when the manifest has gone stale, has gaps, is missing DRM keys, or when the signal has dropped to slate. CloudFront sees those responses and moves to the backup origin, then returns on its own once the primary stops reporting an error.

There is one hard precondition: without time alignment at the source, a viewer will see the picture jump backwards or forwards on switchover. Service redundancy alone does not solve that.

AWS states plainly that for most use cases redundancy within a single region is sufficient. We agree. The two region variant is a cost decision, not a default safety level.

What this project taught us

Item	What turned out to be true
Signal into the cloud	The easiest part. SRT into MediaConnect over a public link works.
MediaLive channel class	An irreversible, cost bearing decision. Taken before anyone sees a picture.
DRM	The biggest time sink. Three quarters of the problems sat in the player, not the keys.
Player on a cluster	Justified by the logic around the file, not by the file.
Load test	Needs designing and measuring itself. Otherwise it measures the generator.
Traffic switchover	It worked. The difficulty is segment alignment, not redirection.

The biggest lesson, though, is not technical. The PoC was worth what it was worth precisely because not knowing something was allowed inside it. Had the same scope arrived straight as a production rollout, two weeks spent on a black screen in a player would have been a project failure. Inside a PoC they were simply the answer to the question of why you run one.

Our consultants have experience with projects of this kind, from signal ingest through content protection to testing at scale. We are happy to talk if you have a similar decision ahead of you.

Sources

• Build a resilient cross-region live streaming architecture on AWS, AWS for M&E blog
• What is Secure Packager and Encoder Key Exchange, SPEKE documentation
• Plan the channel class, AWS Elemental MediaLive documentation
• What is AWS Elemental MediaPackage, AWS documentation
• Amazon EKS Auto Mode, Amazon EKS documentation
• Locust, the tool used for load testing