The Beginner’s Guide To Header Bidding
If you
follow advertising technology, you have no doubt encountered the term
“header bidding” in the last year or two. Yet, when I ask people how much they
know about it, most reply “very little.” This is unfortunate, because I believe
header bidding is the most important advancement in ad tech since the
introduction of real-time bidding (RTB).
Like many concepts in ad tech,
truly understanding how new innovations work, why they matter, and how they fit
into the existing landscape can be difficult. Much of the information is
scattered across the Internet. So, to shorten the learning curve, I created
this guide for people who want a crash course on the essentials, all in one
place.
In this extensive guide, you
will learn the fundamentals of header bidding in depth: what it is, why it
matters, how it works, how to implement it, and much more. Because header
bidding is so conceptual, explaining it with only words can be challenging.
That is why we will also lean heavily on illustrations to help us wrap our heads
around it.
By the end, you will know more
about header bidding than 99 percent of people in ad tech.
Part 1: What Is Header Bidding?
Header bidding is a new,
unified auction conducted by publishers outside of their primary ad server,
which allows advertisers to cherry-pick impressions at the highest priority.
This is sometimes called “first look.”
Before header bidding was introduced, ad space was auctioned off and delivered only once the ad placements began to load on a web page. In other words, once a web page’s ad placements began to load, the publisher’s ad server was called up—in most cases DoubleClick for Publishers (DFP)—and from there, the so-called waterfall of ad serving would begin:
When a person would visit a
website, direct orders would be served first, because they sat at the highest
priority in the ad server. Direct orders guaranteed a certain number of
impressions for the advertiser. Both of these benefits—reservation of inventory
and priority—are functions of the price that advertisers pay for such
privilege.
If the visitor views enough
pages on the website so that the frequency cap on the direct orders is
exhausted, the ad server cascades down to the programmatic line items, where
the visitor enters a real-time auction environment (i.e., RTB), where
multitudes of advertisers bid to serve their ads to visitors.
As of today, this is how most
publishers still serve ads. Unfortunately, this approach does not properly
optimize the publisher’s ad inventory based on price.
Header bidding is an
additional auction that takes place outside of the ad server, in the header of
a web page, which loads before anything else on the page. The header typically
contains metadata about the page and calls scripts used for formatting the
style of the page, tracking, and so on. Because of this, it’s an ideal area to
conduct a new auction.
Even though this new auction happens in the visitor’s browser, the publisher essentially controls this auction. (This fact marks an important shift in power, which we will discuss later.)
This new header auction allows the publisher to field bids on every page load, well before ad placements load on the page. With these bid values, publishers can prioritize delivering these ads ahead of direct orders, assuming that:
1. The bids are high enough and
2. They don’t disrupt the
delivery of direct orders.
Let’s examine why this
ad-serving advancement is so important.
Part 2: Why Is Header Bidding
Important?
Header bidding is an important
development for both advertisers and publishers. Let’s begin with why
publishers should care.
INCREASED REVENUE FOR PUBLISHERS
In the old waterfall approach,
each exchange partner would have a corresponding line item in publisher ad
server, and each would sit at a different priority. If the first exchange
partner in the priority sequence did not purchase the impression, it would
cascade down to the next exchange and to the next, until someone bought it—or
until it was finally passed to Google AdSense or to a line item for house ads.
Typically, each exchange
partner would be configured at a different floor price, so that the cost at
which the impression could be bought would keep dropping as it cascaded down. For example:
·
Exchange 1 (e.g., Index) @ floor price = $3.40
·
Exchange 2 (e.g., Rubicon) @ floor price = $3.00
· Exchange 3 (e.g., OpenX) @ floor price = $2.50
This approach is inefficient
and ultimately hurts both publishers and advertisers.
It’s like offering a bag of
apples to four different people, one after another, lowering the price whenever
someone rejects the apples. Eventually someone might say yes because of the
bargain, but that isn’t how you get the best price for your apples.
Furthermore, it gives the impression that you’re selling low-grade
apples—apples that many people already rejected. And given the reality of
discrepancies when using this cascading passback approach, it’s like losing a
few apples between each pitch.
But with header bidding, every impression is auctioned off to all demand partners simultaneously, before the publisher’s ad server is even called. Advertisers not only get to look at every single impression, which wasn’t possible before but also get the first look at every impression, which is like getting the pick of the litter. Whether they win the impression depends on how much they are willing to pay.
As a result, publishers are
finally able to sell their ad inventory in a way that ensures they get the best
price. And putting all demand partners on the same level to determine an impression’s
true value creates an efficiency that’s not possible with the sequential
waterfall approach.
So, header bidding’s most
important impact on publishers is the significant increase of revenue. People
have reported anywhere from 30% to 60% increases in revenue with header
bidding. However, many factors go in to such calculations, such as geography,
vertical, audience size, and so on.
Additionally, since publishers
now allow advertisers to cherry-pick impressions at the highest priority in the
ad server, it also drives up the scarcity of a publisher’s unsold inventory,
which goes to the exchanges in a “last look” sort of way. This outcome has the
potential to drive up rates for unsold inventory, further increasing revenue
for publishers.
PREMIUM INVENTORY FOR ADVERTISERS
As for advertisers, they now
have access to “premium” inventory that was previously only available via
direct deals with the publishers. Ad buyers using programmatic channels are
finally able to get a real look at all of a publisher’s impressions, not just
those that went unsold in the traditional waterfall setup.
When RTB first came on the
scene, it had a negative reputation. Because it was mostly unsold inventory
being traded, it had negative connotations associated with low quality (e.g.,
remnant, or unsold stuff nobody else wanted). There were even clever
redefinitions of RTB, such as “race to the bottom,” that critics liked to toss
around.
In some ways, this was true.
Most RTB inventory was only available at the bottom of the waterfall, so the
session depths were lower (i.e., last look). But this was a result of how
publishers chose to implement their supply-side platform (SSP) partners in
their ad servers. Instead of using their SSPs the way the SSPs wanted to be
used—as the primary ad server for the publisher—publishers relegated them to
line items at the bottom of the ad server.
With header bidding, RTB
demand can now have a first look at inventory, even above direct orders, which
is a giant leap in the evolution of ad serving.
Because of the revenue
benefits publishers are realizing from header bidding and the premium inventory
now available to advertisers, programmatic advertising is finally able to shake
off the reputation of being a low-quality, direct-response channel. As a result,
header bidding is likely to fuel the accelerated adoption of programmatic ad
tech among publishers.
Part 3: How
Does Header Bidding Work?
Header bidding happens as soon
as a page begins to load. The header bidding code (which lives in the page
header) executes, and the visitor’s browser calls all the demand partners
(e.g., AppNexus, Index, OpenX, Amazon, Criteo) simultaneously and says, “Hey,
here I am! How much will you bid for me?” Within a few hundred milliseconds
(usually less than 500 milliseconds), each makes a bid.
Keep in mind: During this split-second window, many of these demand partners are holding their own auctions to determine the top bid they send. Amazon and Criteo have one auction. Exchanges, on the other hand, have two auctions to wait for: theirs and those held by all their demand side partners.
A couple of important notes
about the diagram above:
1. As you can see in the
example, the highest bidder from the DSPs did not end up winning the auction,
due to the second-price auction model used by most exchanges. (To recap,
second-price auctions send the price of the second highest bidder plus one
cent.) In a header bidding scenario like the one above, the second-price
auction model is not optimal, since the highest bid was prevented from winning.
The consequence is lost revenue to the publisher, and a handicapped environment
for advertisers. The obvious solution would be to switch to a first-price
auction, where the highest bids are passed through without reduction. (Update:
As of March 21, 2019, most of the major exchanges have moved to the first-price
model for the aforementioned reasons, including
Google.)
2. In the scenario depicted
in this illustration, the header bidding wrapper is configured to be
“mediating”, which only passes the top bid to the ad server. Header bidding
wrappers, like Prebid.js for example, allow you to choose whether you want it
to be a “mediating wrapper”, which is the default and sends only the top bid,
or a “non-mediating wrapper”, which passes every single bid that comes through
from every single header bidding partner. When all the bids are passed through,
the ad server then decides which bids win what ad slots.
To see how this works in real
life, just install the Headerbid
Expert plugin for Chrome. It will show you all the demand partners that a
publisher uses in their header bidding implementation, and each partner’s
response time.
The highest bid values from each partner are then passed from the visitor’s browser to the publisher’s ad server. The bid value is matched with a corresponding line item inside the publisher’s ad server, which is prioritized against direct orders to ensure guaranteed delivery is not compromised. If the bid from the header is chosen to serve, the winning advertiser’s ad is shown.
EFFECTIVELY A FIRST-PRICE AUCTION
The beauty of header bidding
is that publishers take the highest bids from all their demand partners—and
take them at face value.
While ad exchanges conduct
second-price auctions on their platforms, where the highest bidders pay only
$0.01 more than the second-highest bidder, header bidding takes the highest
bids from each demand partner to determine the winning price, which is
effectively a first-price auction.
This means that the overall
auction economics are now more favorable to the publisher. With RTB, the
second-price auction was more efficient price-wise (i.e., inexpensive), which
was naturally more favorable to advertisers. As a result, though, it left money
on the table from the publisher’s perspective.
Part 4: How To Implement Header
Bidding
Since this is a beginner’s guide,
diving into the technical details of header bidding implementation is beyond
our scope. Instead, we will cover some important topics to consider when
implementing header bidding on a website.
(For a detailed implementation
guide, I highly recommend the “Getting Started” documentation on the
Prebid.js website.)
CHOOSING DEMAND PARTNERS
Having strong demand partners
is key to maximizing revenue through header bidding. As a publisher, if you
already have relationships with ad exchanges or SSPs, then you are already in a
good position to add them as header bidding demand partners. If you don’t
already have these relationships, then a key initial step is to forge them.
In the header auction, demand
partners can include massive retargeters like Amazon and Criteo, as well as ad
exchanges like AppNexus, Index, and Rubicon, who essentially aggregate demand
from all the major DSPs.
The process for actually evaluating ad tech partners is beyond the scope of this guide, but here is a list of the top demand sources by popularity, according to Adzerk:
MANAGING HEADER LATENCY
One challenge of header
bidding is the latency added to page loading. The header auction process takes
longer than a typical RTB auction on a single exchange, which generally times
out at 100 milliseconds. However, since many publishers use multiple exchange
partners in a traditional waterfall setup, such latency has always been there.
It just hasn’t been in the header.
Header bidding asks for
multiple demand sources to put forth their best price, so several RTB auctions
happen simultaneously. As a result, publishers need to manage their timeouts so
that no one partner will hold up the header auction and jeopardize page
loading. Ideally, the overall timeout should be kept below 500 milliseconds, or
half a second.
HEADER BIDDING WRAPPERS
In the early days of header
bidding, publishers would manage their header auction partners manually. This
proved to be cumbersome and inefficient, so they moved to using header bidding
wrappers (or containers) to make the process more efficient.
Header bidding wrappers are
like tag management systems but for header bidding partners. They allow
publishers to manage their demand partners easily, adding or removing them as
necessary. They translate each partner’s unique parameters into a common value
that can be passed to the ad server uniformly and allow important settings,
such as a centralized timeout, to be configured, enforcing a hard deadline for
the header auction. They also ensure that the header auction happens
asynchronously, so that it won’t negatively affect the rest of the page
loading.
The most popular open-source
solution is Prebid.js, for reasons we will cover
later, while proprietary solutions can be found from companies like Index
Exchange.
(For additional reading, I recommend
Ben Kneen’s “Guide
to Header Bidding Wrappers.”)
PUBLISHER AD SERVER CONFIGURATION
Many people have referred to
header bidding as a hack, and rightly so.
Once a publisher has
configured its header bidding wrapper and associated demand partners, it then
needs to create hundreds of line items in its ad server to capture each bid
value that comes from the header auction. This is hands down the most
time-consuming and cumbersome part of the header bidding setup process. It’s
also the part where you realize how much of a hack it currently is. It’s a
workaround to create a unified auction outside of the publisher’s primary ad
server. To give you an example of how this process looks, check out this
tutorial video on YouTube.
(For a guide on streamlining
the setup of line items, check out “How to
Simplify Line Item Setup” on the Prebid.js website.)
That covers some of the major
topics to think about when implementing header bidding. Now let’s explore what
header bidding means for the ad tech industry as a whole.
Part 5: What Are The Implications Of
Header Bidding?
Header bidding is an evolution
of RTB that provides more value to the publisher while providing advertisers
with important benefits. Yet, there are a number of significant implications
for the industry as a whole.
DECREASED REPORTING DISCREPANCIES
Some degree of discrepancy is
always inevitable in the online advertising space. But discrepancies are
exacerbated when impressions are constantly passed across multiple ad servers
and JavaScript tags, where latency is usually introduced.
Because header bidding is a
single auction that happens across multiple partners simultaneously, there is
no sequential daisy chaining of partners (and their associated ad tags), which
drastically reduces reporting discrepancies. Not to mention the amount of
JavaScript a visitor’s browser needs to load, which further adds to the
importance and appeal of header bidding technology.
INCREASED INFRASTRUCTURE COSTS
For the two major categories
of vendors in the ad tech landscape — demand-side and supply-side platforms —
infrastructure costs are considerably high. In order to process billions of ad
impressions per day, DSPs and SSPs need hundreds, in some cases thousands, of
servers to support the load.
Before header bidding,
exchanges would generally see impressions only once — when it was their turn to
fill it in the waterfall. (We will ignore edge cases for now.) As for DSPs,
they might see that same impression two or three times, as it cascades down the
waterfall from one exchange to the next. However, with the addition of a header
auction, exchanges now see the same impression twice, which almost doubles the
number of calls going to their servers. It’s no surprise that Rubicon spent $25
million on just hardware infrastructure last year. And with their new
commitment to header bidding, expect that number to rise sharply.
For DSPs, it’s even worse. For
every exchange in the header auction, a new impression is being “offered” to
the exchange’s DSP partners. So now, instead of seeing the same impression two
or three times, there is now a real possibility that DSPs are seeing that same
impression twice as much, or more. Now multiply that by billions and it translates
into hefty incremental hosting fees. And unless those new impressions are being
won, they are merely a new expense with little to no corresponding revenue.
POTENTIAL FOR DISINTERMEDIATION
With header bidding,
publishers are now the ones conducting the ultimate unified auction for their
inventory. And it’s not just SSPs and ad exchanges who get a seat at this
meta-auction. Other sources of demand, like DSPs and third parties
like Amazon and Criteo, can also have a seat. Everyone competes at the same
level.
Think about that for a second.
As a publisher, accepting a bid directly from a DSP, for example, means that
you no longer have to forfeit 15% to 20% as a technology fee to your exchange
or SSP partner. That’s a great thing for publishers, who have reportedly been
seeing only 30% to 50% of every ad dollar spent on their sites. It’s also great
for advertisers, because it means more of their ad budgets are going to the
media owner, not intermediaries. In this scenario, exchanges and SSPs appear to
be an expensive channel for demand.
Even if overall yield remained
the same with header bidding, which doesn’t appear to be the case, choosing a
demand partner that allows you to keep 15% to 20% more of every dollar is a big
win. This is excellent for publishers and ad tech platforms with lots of
demand. But it’s a potentially bad sign for aggregators of demand, like smaller
exchanges and SSPs, who are at risk of becoming costly partners.
POTENTIAL FOR CONSOLIDATION
One side effect of header
bidding is that it creates finite room for demand partners in the header,
driven largely by the latency issue. As a result, demand partners must have
enough demand to make it worthwhile for the publisher to add them to the header
auction. If a demand partner is added to the header auction but consistently
fails to put forth competitive bids and frequently loses, publishers would be
justified in removing them from the header auction, thereby putting smaller
players at a real disadvantage.
However, some newer header
bidding solutions are server based (aka server-to-server). Server-side
solutions mitigate latency issues by conducting the auction on a server instead
of in the visitor’s browser, largely eliminating these hard limits on partners.
The potential downside is that auctions on servers are opaque by default,
reducing the amount of transparency in the whole header auction process, unless
otherwise provided.
NEW COMPETITION FOR HEADER CONTROL
In the competitive industry of
advertising technology, most companies want to dominate. With header bidding,
the real power comes from controlling the wrapper that manages all the header
partners. Controlling the wrapper gives the wrapper owner the ability to
collect data about the publisher’s inventory and the auction’s dynamics, which
results in a great deal of knowledge about the overall economics.
Because of this power, and for
the reasons just mentioned, an open source solution seems to be the most
reasonable course. By using an open source solution, publishers can audit the
wrapper code to ensure that nothing shady is happening that may not be in their
interests. That’s why Prebid.js has quickly become the most popular header
bidding wrapper. Conversely, a proprietary solution from a single ad tech
company offers no such ability or comfort.
Part 6: What
Does The Future Hold For Header Bidding?
Header bidding has exposed an
Achilles’ heel in the sequential ad-serving waterfall. Clearly a unified
auction is superior, as demonstrated by header bidding implementations to date.
But those implementations are largely workarounds, built on top of legacy
ad-serving systems like Google’s DFP.
These workarounds pose a
direct threat to Google’s hegemony in the ad-serving space. After all, header
bidding was introduced to create more competition with Google’s ad exchange.
DOUBLECLICK DEFICIENCY, GOOGLE GREED
Google has a strong hold on the
majority of publishers with its DoubleClick ad server, and it takes advantage
of it. A perfect example was the introduction of Dynamic Allocation. This
feature allows Google’s Ad Exchange (AdX) to win impressions above direct
orders if the bids are high enough. And this preferential treatment of its own
technology is understandable for Google shareholders but is far from ideal for
almost everyone else.
Header bidding has exposed the
glaring deficiency in Google’s DFP ad server. It was designed as a waterfall,
and the Dynamic Allocation feature was obviously introduced for self-serving
reasons. Google has no direct financial incentive to open up header bidding
inside DFP to outside demand partners, even though it’s played lip service to
such openness by introducing “Exchange Bidding in Dynamic Allocation” (EBDA),
which, like most Google products, is a black box.
In short, header bidding puts
Google’s stranglehold on the publisher ad server at risk.
THE THREAT: ALTERNATIVE AD SERVERS
The primary threat comes from
alternative ad servers designed to be more holistic solutions with openness and real-time bidding as core features, not afterthoughts.
An ad server that allows publishers to implement a unified auction, with
server-to-server connections to the exchanges — and without the absurd task of
manually creating hundreds or thousands of line items — is surely where things
are heading.
However, as long as publishers
continue to implement header bidding on top of their DFP ad servers, Google
will retain power and ultimate control over ad serving.
As a publisher, replacing your
primary ad server is not a trivial task. Think of it like doing a mid-flight
engine swap on an airplane. Except that it’s your revenue engine. It’s hard to
imagine many publishers wanting to take such a risk. But those that do and that
prove better solutions are feasible will greatly influence any potential
migration from Google’s DFP.
DESPITE PROGRESS, BETTER TOOLS ARE NEEDED
Header bidding raises many
important questions: How much longer until header bidding goes from hack to
mainstream adoption? Right now, it’s very much in the early adoption phase
among publishers. Once the majority of publishers adopt it, how much will it
change the power dynamics in the ad tech world? How much longer will it be a
necessary hack? How will publisher ad servers change? Who will pose the most
credible threat to Google?
Publishers clearly need better tools to manage their ad inventory in an efficient, holistic manner. Header bidding is a major leap in the right direction, not the ideal solution. But it hints toward a better future. And like most innovations, it’s forcing the ad tech industry to evolve. Where this evolution will lead looks like very much like a redesign of the publisher ad server – where programmatic execution is central and not an afterthought. By Ratko Vidakovic
