AdTech: Header Bidding on fire

 

The Beginner’s Guide To Header Bidding  

If you follow advertising technology, you have no doubt encountered the term “header bidding” in the last year or two. Yet, when I ask people how much they know about it, most reply “very little.” This is unfortunate, because I believe header bidding is the most important advancement in ad tech since the introduction of real-time bidding (RTB).

Like many concepts in ad tech, truly understanding how new innovations work, why they matter, and how they fit into the existing landscape can be difficult. Much of the information is scattered across the Internet. So, to shorten the learning curve, I created this guide for people who want a crash course on the essentials, all in one place.

In this extensive guide, you will learn the fundamentals of header bidding in depth: what it is, why it matters, how it works, how to implement it, and much more. Because header bidding is so conceptual, explaining it with only words can be challenging. That is why we will also lean heavily on illustrations to help us wrap our heads around it.

By the end, you will know more about header bidding than 99 percent of people in ad tech.

Part 1: What Is Header Bidding?

Header bidding is a new, unified auction conducted by publishers outside of their primary ad server, which allows advertisers to cherry-pick impressions at the highest priority. This is sometimes called “first look.”

Before header bidding was introduced, ad space was auctioned off and delivered only once the ad placements began to load on a web page. In other words, once a web page’s ad placements began to load, the publisher’s ad server was called up—in most cases DoubleClick for Publishers (DFP)—and from there, the so-called waterfall of ad serving would begin:

When a person would visit a website, direct orders would be served first, because they sat at the highest priority in the ad server. Direct orders guaranteed a certain number of impressions for the advertiser. Both of these benefits—reservation of inventory and priority—are functions of the price that advertisers pay for such privilege.

If the visitor views enough pages on the website so that the frequency cap on the direct orders is exhausted, the ad server cascades down to the programmatic line items, where the visitor enters a real-time auction environment (i.e., RTB), where multitudes of advertisers bid to serve their ads to visitors.

As of today, this is how most publishers still serve ads. Unfortunately, this approach does not properly optimize the publisher’s ad inventory based on price.

Header bidding is an additional auction that takes place outside of the ad server, in the header of a web page, which loads before anything else on the page. The header typically contains metadata about the page and calls scripts used for formatting the style of the page, tracking, and so on. Because of this, it’s an ideal area to conduct a new auction.

Even though this new auction happens in the visitor’s browser, the publisher essentially controls this auction. (This fact marks an important shift in power, which we will discuss later.)

This new header auction allows the publisher to field bids on every page load, well before ad placements load on the page. With these bid values, publishers can prioritize delivering these ads ahead of direct orders, assuming that:

1.     The bids are high enough and

2.     They don’t disrupt the delivery of direct orders.

Let’s examine why this ad-serving advancement is so important.

Part 2: Why Is Header Bidding Important?

Header bidding is an important development for both advertisers and publishers. Let’s begin with why publishers should care.

INCREASED REVENUE FOR PUBLISHERS

In the old waterfall approach, each exchange partner would have a corresponding line item in publisher ad server, and each would sit at a different priority. If the first exchange partner in the priority sequence did not purchase the impression, it would cascade down to the next exchange and to the next, until someone bought it—or until it was finally passed to Google AdSense or to a line item for house ads.

Typically, each exchange partner would be configured at a different floor price, so that the cost at which the impression could be bought would keep dropping as it cascaded down. For example:

·        Exchange 1 (e.g., Index) @ floor price = $3.40

·        Exchange 2 (e.g., Rubicon) @ floor price = $3.00

·        Exchange 3 (e.g., OpenX) @ floor price = $2.50

This approach is inefficient and ultimately hurts both publishers and advertisers.

It’s like offering a bag of apples to four different people, one after another, lowering the price whenever someone rejects the apples. Eventually someone might say yes because of the bargain, but that isn’t how you get the best price for your apples. Furthermore, it gives the impression that you’re selling low-grade apples—apples that many people already rejected. And given the reality of discrepancies when using this cascading passback approach, it’s like losing a few apples between each pitch.

But with header bidding, every impression is auctioned off to all demand partners simultaneously, before the publisher’s ad server is even called. Advertisers not only get to look at every single impression, which wasn’t possible before but also get the first look at every impression, which is like getting the pick of the litter. Whether they win the impression depends on how much they are willing to pay.

As a result, publishers are finally able to sell their ad inventory in a way that ensures they get the best price. And putting all demand partners on the same level to determine an impression’s true value creates an efficiency that’s not possible with the sequential waterfall approach.

So, header bidding’s most important impact on publishers is the significant increase of revenue. People have reported anywhere from 30% to 60% increases in revenue with header bidding. However, many factors go in to such calculations, such as geography, vertical, audience size, and so on.

Additionally, since publishers now allow advertisers to cherry-pick impressions at the highest priority in the ad server, it also drives up the scarcity of a publisher’s unsold inventory, which goes to the exchanges in a “last look” sort of way. This outcome has the potential to drive up rates for unsold inventory, further increasing revenue for publishers.

PREMIUM INVENTORY FOR ADVERTISERS

As for advertisers, they now have access to “premium” inventory that was previously only available via direct deals with the publishers. Ad buyers using programmatic channels are finally able to get a real look at all of a publisher’s impressions, not just those that went unsold in the traditional waterfall setup.

When RTB first came on the scene, it had a negative reputation. Because it was mostly unsold inventory being traded, it had negative connotations associated with low quality (e.g., remnant, or unsold stuff nobody else wanted). There were even clever redefinitions of RTB, such as “race to the bottom,” that critics liked to toss around.

In some ways, this was true. Most RTB inventory was only available at the bottom of the waterfall, so the session depths were lower (i.e., last look). But this was a result of how publishers chose to implement their supply-side platform (SSP) partners in their ad servers. Instead of using their SSPs the way the SSPs wanted to be used—as the primary ad server for the publisher—publishers relegated them to line items at the bottom of the ad server.

With header bidding, RTB demand can now have a first look at inventory, even above direct orders, which is a giant leap in the evolution of ad serving.

Because of the revenue benefits publishers are realizing from header bidding and the premium inventory now available to advertisers, programmatic advertising is finally able to shake off the reputation of being a low-quality, direct-response channel. As a result, header bidding is likely to fuel the accelerated adoption of programmatic ad tech among publishers.

Part 3: How Does Header Bidding Work?

Header bidding happens as soon as a page begins to load. The header bidding code (which lives in the page header) executes, and the visitor’s browser calls all the demand partners (e.g., AppNexus, Index, OpenX, Amazon, Criteo) simultaneously and says, “Hey, here I am! How much will you bid for me?” Within a few hundred milliseconds (usually less than 500 milliseconds), each makes a bid.

Keep in mind: During this split-second window, many of these demand partners are holding their own auctions to determine the top bid they send. Amazon and Criteo have one auction. Exchanges, on the other hand, have two auctions to wait for: theirs and those held by all their demand side partners.

A couple of important notes about the diagram above:

1. As you can see in the example, the highest bidder from the DSPs did not end up winning the auction, due to the second-price auction model used by most exchanges. (To recap, second-price auctions send the price of the second highest bidder plus one cent.) In a header bidding scenario like the one above, the second-price auction model is not optimal, since the highest bid was prevented from winning. The consequence is lost revenue to the publisher, and a handicapped environment for advertisers. The obvious solution would be to switch to a first-price auction, where the highest bids are passed through without reduction. (Update: As of March 21, 2019, most of the major exchanges have moved to the first-price model for the aforementioned reasons, including Google.)

2. In the scenario depicted in this illustration, the header bidding wrapper is configured to be “mediating”, which only passes the top bid to the ad server. Header bidding wrappers, like Prebid.js for example, allow you to choose whether you want it to be a “mediating wrapper”, which is the default and sends only the top bid, or a “non-mediating wrapper”, which passes every single bid that comes through from every single header bidding partner. When all the bids are passed through, the ad server then decides which bids win what ad slots.

To see how this works in real life, just install the Headerbid Expert plugin for Chrome. It will show you all the demand partners that a publisher uses in their header bidding implementation, and each partner’s response time.

The highest bid values from each partner are then passed from the visitor’s browser to the publisher’s ad server. The bid value is matched with a corresponding line item inside the publisher’s ad server, which is prioritized against direct orders to ensure guaranteed delivery is not compromised. If the bid from the header is chosen to serve, the winning advertiser’s ad is shown.

EFFECTIVELY A FIRST-PRICE AUCTION

The beauty of header bidding is that publishers take the highest bids from all their demand partners—and take them at face value.

While ad exchanges conduct second-price auctions on their platforms, where the highest bidders pay only $0.01 more than the second-highest bidder, header bidding takes the highest bids from each demand partner to determine the winning price, which is effectively a first-price auction.

This means that the overall auction economics are now more favorable to the publisher. With RTB, the second-price auction was more efficient price-wise (i.e., inexpensive), which was naturally more favorable to advertisers. As a result, though, it left money on the table from the publisher’s perspective.

Part 4: How To Implement Header Bidding

Since this is a beginner’s guide, diving into the technical details of header bidding implementation is beyond our scope. Instead, we will cover some important topics to consider when implementing header bidding on a website.

(For a detailed implementation guide, I highly recommend the “Getting Started” documentation on the Prebid.js website.)

CHOOSING DEMAND PARTNERS

Having strong demand partners is key to maximizing revenue through header bidding. As a publisher, if you already have relationships with ad exchanges or SSPs, then you are already in a good position to add them as header bidding demand partners. If you don’t already have these relationships, then a key initial step is to forge them.

In the header auction, demand partners can include massive retargeters like Amazon and Criteo, as well as ad exchanges like AppNexus, Index, and Rubicon, who essentially aggregate demand from all the major DSPs.

The process for actually evaluating ad tech partners is beyond the scope of this guide, but here is a list of the top demand sources by popularity, according to Adzerk:

MANAGING HEADER LATENCY

One challenge of header bidding is the latency added to page loading. The header auction process takes longer than a typical RTB auction on a single exchange, which generally times out at 100 milliseconds. However, since many publishers use multiple exchange partners in a traditional waterfall setup, such latency has always been there. It just hasn’t been in the header.

Header bidding asks for multiple demand sources to put forth their best price, so several RTB auctions happen simultaneously. As a result, publishers need to manage their timeouts so that no one partner will hold up the header auction and jeopardize page loading. Ideally, the overall timeout should be kept below 500 milliseconds, or half a second.

HEADER BIDDING WRAPPERS

In the early days of header bidding, publishers would manage their header auction partners manually. This proved to be cumbersome and inefficient, so they moved to using header bidding wrappers (or containers) to make the process more efficient.

Header bidding wrappers are like tag management systems but for header bidding partners. They allow publishers to manage their demand partners easily, adding or removing them as necessary. They translate each partner’s unique parameters into a common value that can be passed to the ad server uniformly and allow important settings, such as a centralized timeout, to be configured, enforcing a hard deadline for the header auction. They also ensure that the header auction happens asynchronously, so that it won’t negatively affect the rest of the page loading.

The most popular open-source solution is Prebid.js, for reasons we will cover later, while proprietary solutions can be found from companies like Index Exchange.

(For additional reading, I recommend Ben Kneen’s “Guide to Header Bidding Wrappers.”)

PUBLISHER AD SERVER CONFIGURATION

Many people have referred to header bidding as a hack, and rightly so.

Once a publisher has configured its header bidding wrapper and associated demand partners, it then needs to create hundreds of line items in its ad server to capture each bid value that comes from the header auction. This is hands down the most time-consuming and cumbersome part of the header bidding setup process. It’s also the part where you realize how much of a hack it currently is. It’s a workaround to create a unified auction outside of the publisher’s primary ad server. To give you an example of how this process looks, check out this tutorial video on YouTube.

(For a guide on streamlining the setup of line items, check out “How to Simplify Line Item Setup” on the Prebid.js website.)

That covers some of the major topics to think about when implementing header bidding. Now let’s explore what header bidding means for the ad tech industry as a whole.

Part 5: What Are The Implications Of Header Bidding?

Header bidding is an evolution of RTB that provides more value to the publisher while providing advertisers with important benefits. Yet, there are a number of significant implications for the industry as a whole.

DECREASED REPORTING DISCREPANCIES

Some degree of discrepancy is always inevitable in the online advertising space. But discrepancies are exacerbated when impressions are constantly passed across multiple ad servers and JavaScript tags, where latency is usually introduced.

Because header bidding is a single auction that happens across multiple partners simultaneously, there is no sequential daisy chaining of partners (and their associated ad tags), which drastically reduces reporting discrepancies. Not to mention the amount of JavaScript a visitor’s browser needs to load, which further adds to the importance and appeal of header bidding technology.

INCREASED INFRASTRUCTURE COSTS

For the two major categories of vendors in the ad tech landscape — demand-side and supply-side platforms — infrastructure costs are considerably high. In order to process billions of ad impressions per day, DSPs and SSPs need hundreds, in some cases thousands, of servers to support the load.

Before header bidding, exchanges would generally see impressions only once — when it was their turn to fill it in the waterfall. (We will ignore edge cases for now.) As for DSPs, they might see that same impression two or three times, as it cascades down the waterfall from one exchange to the next. However, with the addition of a header auction, exchanges now see the same impression twice, which almost doubles the number of calls going to their servers. It’s no surprise that Rubicon spent $25 million on just hardware infrastructure last year. And with their new commitment to header bidding, expect that number to rise sharply.

For DSPs, it’s even worse. For every exchange in the header auction, a new impression is being “offered” to the exchange’s DSP partners. So now, instead of seeing the same impression two or three times, there is now a real possibility that DSPs are seeing that same impression twice as much, or more. Now multiply that by billions and it translates into hefty incremental hosting fees. And unless those new impressions are being won, they are merely a new expense with little to no corresponding revenue.

POTENTIAL FOR DISINTERMEDIATION

With header bidding, publishers are now the ones conducting the ultimate unified auction for their inventory. And it’s not just SSPs and ad exchanges who get a seat at this meta-auction. Other sources of demand, like DSPs and third parties like Amazon and Criteo, can also have a seat. Everyone competes at the same level.

Think about that for a second. As a publisher, accepting a bid directly from a DSP, for example, means that you no longer have to forfeit 15% to 20% as a technology fee to your exchange or SSP partner. That’s a great thing for publishers, who have reportedly been seeing only 30% to 50% of every ad dollar spent on their sites. It’s also great for advertisers, because it means more of their ad budgets are going to the media owner, not intermediaries. In this scenario, exchanges and SSPs appear to be an expensive channel for demand.

Even if overall yield remained the same with header bidding, which doesn’t appear to be the case, choosing a demand partner that allows you to keep 15% to 20% more of every dollar is a big win. This is excellent for publishers and ad tech platforms with lots of demand. But it’s a potentially bad sign for aggregators of demand, like smaller exchanges and SSPs, who are at risk of becoming costly partners.

POTENTIAL FOR CONSOLIDATION

One side effect of header bidding is that it creates finite room for demand partners in the header, driven largely by the latency issue. As a result, demand partners must have enough demand to make it worthwhile for the publisher to add them to the header auction. If a demand partner is added to the header auction but consistently fails to put forth competitive bids and frequently loses, publishers would be justified in removing them from the header auction, thereby putting smaller players at a real disadvantage.

However, some newer header bidding solutions are server based (aka server-to-server). Server-side solutions mitigate latency issues by conducting the auction on a server instead of in the visitor’s browser, largely eliminating these hard limits on partners. The potential downside is that auctions on servers are opaque by default, reducing the amount of transparency in the whole header auction process, unless otherwise provided.

NEW COMPETITION FOR HEADER CONTROL

In the competitive industry of advertising technology, most companies want to dominate. With header bidding, the real power comes from controlling the wrapper that manages all the header partners. Controlling the wrapper gives the wrapper owner the ability to collect data about the publisher’s inventory and the auction’s dynamics, which results in a great deal of knowledge about the overall economics.

Because of this power, and for the reasons just mentioned, an open source solution seems to be the most reasonable course. By using an open source solution, publishers can audit the wrapper code to ensure that nothing shady is happening that may not be in their interests. That’s why Prebid.js has quickly become the most popular header bidding wrapper. Conversely, a proprietary solution from a single ad tech company offers no such ability or comfort.

Part 6: What Does The Future Hold For Header Bidding?

Header bidding has exposed an Achilles’ heel in the sequential ad-serving waterfall. Clearly a unified auction is superior, as demonstrated by header bidding implementations to date. But those implementations are largely workarounds, built on top of legacy ad-serving systems like Google’s DFP.

These workarounds pose a direct threat to Google’s hegemony in the ad-serving space. After all, header bidding was introduced to create more competition with Google’s ad exchange.

DOUBLECLICK DEFICIENCY, GOOGLE GREED

Google has a strong hold on the majority of publishers with its DoubleClick ad server, and it takes advantage of it. A perfect example was the introduction of Dynamic Allocation. This feature allows Google’s Ad Exchange (AdX) to win impressions above direct orders if the bids are high enough. And this preferential treatment of its own technology is understandable for Google shareholders but is far from ideal for almost everyone else.

Header bidding has exposed the glaring deficiency in Google’s DFP ad server. It was designed as a waterfall, and the Dynamic Allocation feature was obviously introduced for self-serving reasons. Google has no direct financial incentive to open up header bidding inside DFP to outside demand partners, even though it’s played lip service to such openness by introducing “Exchange Bidding in Dynamic Allocation” (EBDA), which, like most Google products, is a black box.

In short, header bidding puts Google’s stranglehold on the publisher ad server at risk.

THE THREAT: ALTERNATIVE AD SERVERS

The primary threat comes from alternative ad servers designed to be more holistic solutions with openness and real-time bidding as core features, not afterthoughts. An ad server that allows publishers to implement a unified auction, with server-to-server connections to the exchanges — and without the absurd task of manually creating hundreds or thousands of line items — is surely where things are heading.

However, as long as publishers continue to implement header bidding on top of their DFP ad servers, Google will retain power and ultimate control over ad serving.

As a publisher, replacing your primary ad server is not a trivial task. Think of it like doing a mid-flight engine swap on an airplane. Except that it’s your revenue engine. It’s hard to imagine many publishers wanting to take such a risk. But those that do and that prove better solutions are feasible will greatly influence any potential migration from Google’s DFP.

DESPITE PROGRESS, BETTER TOOLS ARE NEEDED 

Header bidding raises many important questions: How much longer until header bidding goes from hack to mainstream adoption? Right now, it’s very much in the early adoption phase among publishers. Once the majority of publishers adopt it, how much will it change the power dynamics in the ad tech world? How much longer will it be a necessary hack? How will publisher ad servers change? Who will pose the most credible threat to Google?

Publishers clearly need better tools to manage their ad inventory in an efficient, holistic manner. Header bidding is a major leap in the right direction, not the ideal solution. But it hints toward a better future. And like most innovations, it’s forcing the ad tech industry to evolve. Where this evolution will lead looks like very much like a redesign of the publisher ad server – where programmatic execution is central and not an afterthought. By Ratko Vidakovic