HTTPS is Hard

This blog post is the first in a regular tech series from the Yell engineering team looking at challenges they face and problems they solve across Yell’s various digital solutions.

Here, Yell’s Head of Web Engineering, Steve Workman, looks back over‘s seven-month transition to HTTPS, (a secure version of the HTTP protocol – which sends data between a browser and a website) to raise awareness of the issues with the move in the industry and to make the adoption process easier for other engineering teams.


“We’re a business directory, why do we need to be secure?” I said, talking to Daniel Applequist (W3C Technical Architecture Group co-chair) in the pub after London Web Standard’s January 2015 event (full disclosure, I help to run this event) where he spoke about HTTPS and web security. He replied:

“Think about what information your users put through that every day, searching for legal counsel and family planning clinics, as well as regular plumbers and hairdressers. They search for it locally, and that is all personally identifiable. If I were a hacker intercepting this traffic I could work out some pretty interesting stuff about you.”

It’s a solid standpoint — in this post-Snowden world, the person snooping on you may not be reaching you through that insecure coffee-shop router that you’ve logged on to whilst waiting for your latte. When traffic is unencrypted, it opens the potential for anyone to identify you and your internet footprint, and Yell could have been a part of this. Therefore, we were determined to increase website security across

Having HTTPS is also proving increasingly important as a search ranking factor. According to Daniel Applequist:

“Google is pushing hard on this, they made it a ranking factor to encourage the big guys to change. If you’re selling this transition to your boss, that’s what you’ll major on.”

The directory provides value to businesses across the country through our listing pages and by linking to their websites. For Yell, ranking well in search engine result pages is important and, even if the effect of a HTTPS transition on ranking was very small, this would be a big differentiator between competing directories.

With all this in mind, our engineering team was tasked with moving to HTTPS.

I mean, how hard can it be?

In 2015, the year of hoverboards, this should have been simple; it’s just a few certificates in the right places. Put those on the servers, update your links and add some redirects to point to the HTTPS URL , right?


Implementing HTTPS


Firstly, is old, even in internet terms . It was  first registered in 1996 and has had nearly 20 years of patches, upgrades, new components and re-writes, producing what you see today. Whilst we have many modern systems, including our consumer-facing systems, there are less-used (but no-less important) structures that do not use that same technology, or even the same programming language.

We have traditional servers in data centres that are completely in our control, we have some in the cloud, and there are some that are not in our control — services provided by other parties that need to be modified in order to meet our HTTPS requirements. We have a laundry list of third-party suppliers that all need to support HTTPS in order for us to keep that big green padlock in the browser, and then we’ve got to set up efficient rules to take our customers to the right place, including those with 20-years-worth of links around the internet (read, a few hundred million).

We had to make sure we could do this all without downtime, without impacting our users, our website performance and our search engine rankings.

The process

Before starting this journey,  I expected I would have to talk to the internal teams, listed in the diagram below to get HTTPS implemented on In smaller organisations, it could even be fewer teams than this, but for Yell, by the end of the journey, this model couldn’t have been further from the truth.


Teams I thought I'd have to talk to at the start of this process

I began by identifying all of the different components of our system that used HTTP as a hard-coded link, and tried to switch them over to HTTPS. The easiest way to do that was to set up a self-signed certificate on my local development machine and see what failed. There are lots of good tutorials on setting up self-signed certificates on different web servers to help with this. I used the Mozilla SSL Configuration Generator to get going, as well as this handy article on getting the green padlock locally on OS X. It’s important to also test the configuration against the Qualys SSL Server Test, to make sure you’ve got a secure connection, serving the right ciphers.

With those set up, I ran the site and looked for errors, and there were a lot of them, including:

  • Adobe Analytics
  • Adtech display ads
  • The entire reviews section of the site
  • Quite a few internal URLs, including the canonical URLs of each page

In general, most things worked, and once I’d added an “S” to most of the URLs (or removed the protocol entirely, as is the simpler case) we were getting somewhere. This was the technically easy section, where generally everything worked and the fixes were easy. Then I had to move on to try and fix the other issues.


AOL’s Adtech platform is one of the largest advertising networks in the world, along with Google’s DoubleClick, and they both support serving adverts over HTTPS. A short discussion with our contact at Adtech and a read through their documentation meant the changes could be made and adverts would be served over a secure connection. At least, that’s what should have happened.

The issue here was that even though Adtech serves images and JavaScript libraries over HTTPS, the adverts that they inject into your page may not be served over HTTPS. This produced a mixed content warning in the best case. In the worst case, there was a 300×250 pixel gap where the advert should have appeared.

Solving this was a human problem —  I sat down with Yell’s advertising manager and requested that all adverts be served over HTTPS. They were more than happy to oblige, and ticked all the appropriate boxes for us. They also changed the processes for the advertising sales team to ensure that new advertising is served over HTTPS by default. Even further than this, they agreed to monitor advertising providers and talk to Adtech to change the list of suppliers of general advertising content to those that supported HTTPS — not all of them do. Our team was really happy with the way we were able to work together with Adtech to reach a mutually beneficial outcome here.

This is a challenge that the IAB (Interactive Advertising Beaurau) have written about, citing that 80% of the ad industry supports HTTPS. That’s a lot, but not enough for many, and this can be a major reason for websites to delay their migration to HTTPS. If your website is entirely reliant on advertising revenue, then this migration can be very difficult to justify.

My team released this update by March 2015, well before our main HTTPS transition. By this point in the process, the diagram had grown to include:

HTTPS process 2


Adobe Analytics

Yell makes great use of Adobe’s analytics solution Omniture, (an alternative to Google Analytics), and it is an essential part of our business . Using Omniture we provide valuable data to our merchants on how their listings are performing on In short, it’s a really important tool for Yell, and there are a lot of people in our team working with it.

To make sure there were no issues here, I contacted Adobe’s support team to enable the HTTPS beacon collection for our domain, rather than use a third-party domain. They came back with a form to fill in, a request for a certificate and a recommendation that Yell upgrade our implementation as well as move to a closer data-centre. This all made sense, especially the regional data collection, meaning we’d be sending data 50 miles, rather than 5,000 miles.

This part of the process took time – nearly two months. It involved testing, trialling, upgrading libraries across multiple systems and prioritising this amongst other changes and priorities. This was all quite a logistical challenge, made tougher because it came under a lot of scrutiny. I found myself writing more reports than code in order to complete this task.

At the end of the process, is better off in many ways: the upgrade reduced page load times by around 100 milliseconds (ms) and thanks to a new library, our overall analytics capabilities have improved.

Since we went through this process, Adobe have now started offering free security certificates for their first-party collection domains, removing part of this barrier to HTTPS adoption.

This change went live to in April 2015 – three months after we began our transition to HTTPS. By now, the departments involved included:

Who I had to talk to after resolving the Adobe Analytics issues

YELL Reviews

Yell’s reviews platform is one of the older components in the stack, and was inherited from an acquisition a few years ago ( have since introduced a new, improved ratings system as of December 2015). This meant that some of the technical work for HTTPS had to be repeated, and a few gremlins eliminated. This process wasn’t as simple as it could have been, but we were able to deploy those changes in April 2015. I mention this as a separate section because in any upgrade there will often be one or two systems that just don’t play nicely, and take a significant amount of extra engineering time to create a workable solution. This was one of those components.

By this stage in the transition, we’d reached the point where the bulk of the technical challenges were out of the way and are live (continuously deploying and testing this was one of the best decisions we made on this project). The last thing to do was get the certificates onto the servers and redirect our users.

Obtaining certificates

Buying an SSL certificate is relatively simple for an individual to do. For a single website I’d recommend going to StartSSL to get a free certificate to install. There’s also the LetsEncrypt project, which provides free certificates for services through a simple command-line interface. Yell is a large company, buying certificates in volume for our customers, so we have existing agreements and a procurement process to go through to complete this. This meant the process took longer, but ultimately was a simple matter of talking to our security team. Here, we also wanted an extended validation (EV) certificate, for a few reasons:

  • It’s a mark of trust in the organisation
  • It’s not much more expensive than a regular certificate
  • It’s the only type of certificate that turns the padlock green in Microsoft’s Edge web browser, and provides larger amounts of green in other browsers, which is important for the perception of security

The process for obtaining an EV certificate involves not only proving that you are the owner of the site digitally, but that you exist physically too. For Yell, the trickiest part of this was the third-party company validation, as our domain name was registered for Yell, though we are legally registered under our parent company hibu (UK) Limited. To do this I spoke to our legal team and asked them to adjust the name of the business in our domain name records to match the records at Companies House. I’m pretty sure this is an exceptional step but it is worth noting for businesses who could be registered under alternative legal or international group names. This was relatively simple to solve, but the whole process took four weeks.

By June 2015, the project and the number of departments consulted had grown significantly:

Who I had to talk to after I had obtained certificates

Other technology providers

Yell work with two other technology providers that also came into the process. One is an anti-scraping service and the other is a content delivery network (CDN) for our video products. Both of these requested copies of the certificates we would buy and would charge us a few thousand dollars a year for this process. The actual implementation with these suppliers progressed smoothly, but this was another chink in the chain.

This also required a business case for sign off by the company. Writing the case, taking input from and getting time with all of the stakeholders, and presenting it to the budget holder, came in at a total of three weeks from start to finish. Happily, we were able to purchase the appropriate certificate and update agreements with our suppliers.

For a lot of engineers, writing a business case is a daunting process that they will have never done before. We are very lucky to have team members who have written them before and supported us through the process, but this can be a significant hurdle if you are a lone engineer or don’t have the right level of support within your organisation. One of the few resources I found to help with this was Making the Software Business Case by Donald Reifer (available on Amazon) which was written in 2001. If you have used any more up to date resources in your organisations, please let us know in the comments.

Performance concerns

During the business case review, risks were raised regarding the performance of the system. This was taken up with our operations team. who looked into areas which would be effected by the change.

We would be terminating the HTTPS request at our load balancer, which makes it an important point in the chain. This review highlighted improvements which could be made on the load balancer’s TLS performance. We therefore agreed to perform a software update to enable us to handle these requests at speed.

We also agreed to monitor and report on the performance of the website in detail during the transition, using our real user measurement solution, mPulse. This only provides accurate TLS data from Navigation Timing v2 enabled browsers (Chrome, and recently iOS 9) but this is enough for our purposes.


On launch day itself, we updated the following items, all of which are critical to’s SEO:

  • Our sitemaps, and every URL inside them (> 10M links)
  • Robots.txt file, pointing to the updated HTTPS sitemaps
  • Canonical URLs for every page, with HTTPS links
  • Register Google Search Console for HTTPS and all of the sub-properties, so we can monitor the transition
  • 301 redirects to HTTPS for HTTP traffic

My team spent pretty much all day waiting for things to fall over, but they didn’t! The transition was a success, and there is now a green padlock for every user.

Process summary

Altogether, getting the HTTPS project from ideation to go-live took 7 months and involved over 10 internal departments and five external agencies, visable in the diagram below:

Who I had to talk to to successfully get a business case through

The aftermath

With a transition of this magnitude and a product this large it was no surprise that we didn’t think of everything right at the start. After the launch, we spent a couple of days fixing bugs like e-mail sending, publishing premium adverts and our performance testing suite. The main culprit here was Java.

Java is written in Java, which maintains a list of trusted root Certificate Authorities (CAs). Depending on the version of Java you are using, certain CAs aren’t trusted and therefore the system throws up a security error if it tries to get a resource from a HTTPS URL with an untrusted certificate. Here we discovered that not all of our systems had our certificate provider in their key stores.

It should have been a simple matter of updating the keystore, but for a while this didn’t work. We noticed this only after we had started redirecting internal traffic from HTTP to HTTPS, and at that time, we were redirecting our users too. There was a key lesson here: you may need an internal HTTP service that isn’t upgraded, to deal with any applications that simply don’t want to work with HTTPS.

TLS Performance

We have been told that HTTPS is fast, and that it makes very little difference for modern browsers. You no longer need dedicated hardware to perform decryption within a reasonable amount of time — but just how much of a difference does it actually make to your bottom line? Well, looking at our real user monitoring, we can tell you:

HTTPS adds 150ms for desktops and 200ms for smartphones at the median, and over half a second at the 95th percentile

The impact of HTTPS on desktop website performance
The impact of HTTPS on desktop website performance
The impact of HTTPS on smartphone performance
The impact of HTTPS on smartphone performance

These graphs show TCP connection time for Chrome at the point of redirecting traffic to HTTPS.

That’s a relatively large gap to have to make up, especially at the extremes where 5% of users are taking an additional two seconds to connect while their web clients negotiate the HTTPS connection. If we’re trying to achieve a first render time of less than 1,000ms, then adding another 200ms for TLS negotiation takes up 20% of your entire budget. Add in latency for 3G mobile requests – 300ms at the minimum – and you only have half a second to get everything to the browser. An additional 200ms from TLS negotiation is likely to make that very, very difficult indeed.

To mitigate this, we performed some TCP Profile tuning on our load balancer. The load balancer in question (a Citrix Netscaler) is well-established and commonly used in large organisations in the industry, though there is very little information on how to tune it for performance, save for this slide deck by AOL. So, we looked for tuning options in line with Ilya Grigorik’s High Performance Browser Networking to guide us.

Of the 7 critical features and concepts (which you can see in this table on Is TLS Fast Yet) the Netscaler doesn’t support the ones that provide the best performance, namely OCSP stapling and dynamic record sizing. It also only supports HTTP/2 in the very latest version of the software, which is a killer feature of HTTPS that we weren’t able to implement in time. These are significant limitations of the load balancer, and the feature support table on Is TLS Fast Yet is a great guide to the features of each of the competitors should you be looking to get started with load balancing or looking to move from one to another.

Despite all of this, we were guided to a profile (TLS false start on, record size optimisation, correct window sizing) that saved us about 20ms at the median, and a little more at the extremes. Frankly, that’s not a lot of benefit from the profile — but it’s better than nothing.

In the future, we will look to upgrade to technologies like the HTTP/2 protocol to improve TLS negotiation and reduce the RTT (round-trip time), turning a negative impact into a positive performance benefit.

HTTP Referrer

You may not know this, but a lot of analytics information relies upon HTTP referrers to identify where traffic came from. The referrer tells you the previous page that a visit came from, so it’s useful for businesses of all sizes to identify if a marketing campaign is working, or to track their links around the web.

The HTTP 1.1 spec defines when to pass a referrer:

HTTP Pass Pass
HTTPS Do not pass Pass

So, with now on HTTPS, for the referrer information to be passed to Google Analytics for example, the destination site must also be on HTTPS. While the rest of the internet transitions to HTTPS, things aren’t looking good. 99% of our customer’s websites are served over HTTP, and of our 1% who are on HTTPS, one quarter are Facebook pages.

This has wide-ranging ramifications for

Although Yell does provide click-through metrics to our merchants directly, Google Analytics is still the tool of choice for many of our customers tracking wide range of business campaigns across the web. So, if our referrer data stops sending information to their dashboards, it looks like we’ve just stopped working. Of course, their overall traffic hasn’t dipped, but looking broken is sometimes worse than actually being broken.

There are other ways to allow the passage of referral data from HTTPS to HTTP through a technology called Referrer Policy. This allows you to control referrers in a fine-grained way, and if you set it to “unsafe”, the referrer will always be passed. However, Internet Explorer does not support this feature, so we would lose a significant proportion of the traffic – whilst it would look like traffic had returned, it would be less than before – so better to not provide in this case.

Alternatively, we could control the referrer using the Content Security Policy, which is how Twitter’s link-shortening service sends referrals. This has even poorer browser support, with Chrome and Firefox being the only supporters, so this was also not an option. For Yell, the only option available was education.

The engineering team spent weeks working with our sales, customer services and marketing teams to help guide customers through this transition. We’re not going back to HTTP (the nuclear option), and this option has never been raised  — Google’s position on HTTPS as a ranking factor has always been a strong factor for Yell. Now, all departments are working together to communicate the benefits of the transition to our customers, and to encourage them to consider an upgrade to HTTPS, or to use campaign tags in their URLs — these provide a greater degree of referrer accuracy throughout reporting data, especially in Google Analytics.

This element was the most difficult part of Yell’s transition. It’s unlikely to affect anyone but a directory company, but there will be cases where this really matters to your business.

So, after all that, this ‘simple’ web change has led me to work with the majority of the company:

The final list of everyone I had to talk to to make HTTPS succeed at Yell

HTTPS is Hard

This article is written with the best of intentions, to highlight how difficult it can be to implement HTTPS across a large organisation such as Yell and the challenges other businesses can encounter through the process.

For me, the whole exercise was worth it. For all the times my head hit the desk at the stupidity of some technology, and the times where things just took far too long, seeing that shiny EV certificate in my browser is something I’m proud of bring to We resolved far more small issues with all of our systems than I’ve detailed here  – in fact,  I probably made changes to every single component in our stack to complete this change. For all those challenges, Yell is now in a better position that we’ve ever been in terms of resilience and security.

In summary: implementing HTTPS isn’t just hampered by technology problems, it also has a people problem – namely a lack of incentives.

For many websites, there currently aren’t enough strong enticements to balance out the headache of migrating a site to HTTPS. This imbalance in trade-offs to benefits can be clearly seen in this post. Across the internet, there needs to be a shift towards positive motivations to improve adoption.

How can we make HTTPS easier?

Based on everything Yell has been through to implement HTTPS, I suggest the following:

  • Eliminate the cost of standard certificatesLetsEncrypt is showing how this can be made easier and free, Dreamhost is undercutting themselves and Amazon Web Services is providing free certificates for their East-Coast data centre. All of these companies are setting the trends that others need to follow.
  • Reduce the cost of setup: I understand that private IP addresses can be costly, but this step needs to get cheaper (or become free) to be successful. The alternative to this, SNI, is not broadly available from service providers at the moment.
  • SaaS (software as a service) providers should offer HTTPS by default
  • Google, Bing and Yahoo need to make HTTPS a stronger ranking factor. Having it as a tiebreaker at the moment is nice, but it doesn’t have as big an effect as it could. If Google really wants the web to go HTTPS, there needs to be a stronger signal from them on this.
  • Improve education on what HTTPS will actually do for you— especially in regards to referrers, performance and certificate issues. Google have recently created some HTTPS transition guidelines, and released their HTTPS transparency report, though currently, there’s not a lot of information out there.
  • Implement the privileged contexts spec for developers – there are plans to deprecate access to geolocation, user media and others, whilst service worker will be HTTPS-only. This is known as “privileged contexts”, or more commonly, “powerful features”. Deprecation of features may not be enough here, and this is only viable once the first few items in this list have been made a success or this may have the opposite effect. I look forward to this advancing more in the near future.
  • Finally, browser manufacturers need to be sending strong signals to the public, and online companies, that HTTP is insecure. Both Google and Mozilla have plans to action this in 2016, though to what extent it is still unknown.

Overall, the future is bright for HTTPS, though right now, it’s hard.

Thanks for reading. I’d love to hear your experiences working with HTTPS and comments below or on Twitter: @steveworkman or @yelldev

The Washington Post‘s recent article about moving to HTTPS is also well worth a read.

Steve Workman is discussing HTTPS and the recent transition at Front End London on 31 March, 2016. Register for tickets at

24 thoughts on “HTTPS is Hard”

  1. Steve, thanks for the great deep-dive! It’s really useful to have case studies like this one, both as a resource to folks planning the move, and for those of us working on the infrastructure bits to make it simpler and easier.

    With that in mind, a couple of comments on the performance section:

    > Where 5% of users are taking an additional two seconds to connect while their web clients negotiate the HTTPS connection. If we’re trying to achieve a first render time of less than 1,000ms, then adding another 200ms for TLS negotiation takes up 20% of your entire budget.

    I’m willing to bet that your long tail is dominated by users that have to validate your EV certs. Chrome (and other browsers) have to pause the handshake (see [1]) and query the OCSP responders ( and, in this instance), before they can validate the certificate and proceed with sending application data. OCSP stapling would, obviously, be a huge win here.. I hope Citrix gets their act together. Or, perhaps time to consider an alternative?

    Also, it looks like you’re sending a root cert as part of your chain, which you don’t need to do. Related, I can’t validate this with certainty, but looking at the TCP packet trace of the handshake, I’m a bit suspect.. is the NetScaler still using IW4? If you look at the attached image, you’ll see a 1-RTT pause between the cert and “server key exchange” records, which leads me to believe that you may overflowing the 4 packet CWND — worth looking into.

    Last but not least, you’re also stuck on a 2-RTT TLS handshake; TLS false start is not enabled. To make that work, at least for Chrome, you need to enable ALPN and Forward Secrecy — for the latter, SSL Labs is failing [2] the check due to weak key exchange.

    In short, you’re taking a hit of EV validation and 2-RTT handshakes. Resolving those issues would be a big win. Enabling HTTP/2, which would allow you to reduce number of connections, and get better utilization on the wire, would be another obvious win.


    1. @igrigorik:disqus thank you so much for the kind words!

      As you can see from the labs test, there’s a MITM performing anti-scrape functionality as @__apf__ noticed. I only briefly touch on this in the post, but that means we’re only as strong as the weakest link. The exchange behind this, which gets an A on SSL labs [1] is the one I’ve been able to optimise, and has some of the modifications that you’ve suggested. I’ll pass on the feedback to our supplier, and see what can be done.

      This is the sad reality of not quite being in control of the edge of the network, and I’m willing to bet this isn’t the only kind of network edge termination that has these kind of optimisation issues.


      1. Ah, interesting! Looking at their site ( they’re not on HTTPS themselves (always a bad sign), and I don’t see TLS mentioned anywhere. At the same time, looks like they might be serving a number of large sites… Perhaps we should add them to If nothing else, it would help raise visibility into these kinds of issues.

  2. Have you looked at adding something like CloudFlare or Akamai before your anti scraping solution? It would be an extra hop, but they would be able to handle SSL termination close to the user and then will more quickly be able to establish an SSL connection to distil (assuming you need end to end encryption for everything).

    Thanks for the write up!

    1. Hi EJ,

      Using a CDN or similar would be possible, reducing the number of terminations. You’d want to secure that connection between the CDN, the middle-man and our own servers with a VPN otherwise that traffic will travel unencrypted along the network as soon as it leaves the CDN towards the servers.

      If you were to get the CDN to do the termination, then reencrypt and send on, we’d be performing a triple SSL termination (CDN, MITM and Yell) which would be even worse for performance. We’re also very UK-centric, so our traffic won’t benefit very much from having a closer node. Interesting thoughts though!


      1. I would still test it. Termination from Cloudflare / Akamai to Distill might be much better at high percentiles.

  3. Wished I could have used this as a reference to navigate our own https migration late last year. Great analysis.

    Obviously we ‘re on a much smaller and technically simpler scale, but we’ve actually found the migration process a lot less scary and painful than anticipated. All previous case studies seemed to be littered with tales of huge traffic drop offs, So we decided to run http and https in parallel, but directing crawlers by using canonical tags.

    The https site now appears to be fully indexed, 3rd parties and APIs are now updated. Still doing some more testing, but the time is approaching for the final redirect. One thing we did miss were users embedded YouTube videos, which was hard coded to http.

    Our take away would be planning, testing and don’t rush. Don’t cut straight over to https, let Google re-index you first, but tell it what your doing.

  4. Hey @disqus_PmdOi1Vgnv:disqus, awesome article!

    I noticed that you’ve switched to using certificates from Let’s Encrypt and I was curious about the motivation behind that. Was cost a factor and/or was the point made by @igrigorik:disqus regarding the additional overhead on the client side of EV certificates involved?

    Keep up the awesome work 🙂

    1. Hi Scott,

      We’re still using an EV certificate on the top-level domain and will continue to do so. This blog, and other smaller applications, will be using a variety of DV certificates, for two reasons – ease of procurement and cost. Cost isn’t a big issue, the best thing about LetsEncrypt is that I don’t have to ask anyone else to procure or renew a certificate, meaning I can get apps live faster.


      1. The procurement aspect is something I hadn’t considered for Let’s Encrypt before now, thanks for the response!

  5. Interesting that you choose EV certificate, to my eyes, it seems confusing to a user to be visiting, but seeing Hibu Ltd in the bar, which he has never heard of.

    1. Hibu is the legal name of the company in the UK, though we trade as Yell. It has the same effect on the Apple and Play stores with our native apps. If we ever change our company name formally, we’ll regenerate the certificate.

    2. I agree – if you go to Barclays you see “Barclays Bank PLC” if you go to Apple Support you see “Apple Inc.”

      I (a software engineer) knew what was up and from the Wikipedia page that Hibu is the parent group of Yell, but ordinary users will think it odd and in the worst case will even think it is a fake site.

      Looks like the cert expires in June 2018 so I would advise a re-issue with “Yell Ltd” as the EV subject.

      @Steve Workman – excellent post, very well-written and informative – many thanks for sharing your experience.

Comments are closed.