This blog post is the first in a regular tech series from the Yell engineering team looking at challenges they face and problems they solve across Yell’s various digital solutions.
Here, Yell’s Head of Web Engineering, Steve Workman, looks back over Yell.com‘s seven-month transition to HTTPS, (a secure version of the HTTP protocol – which sends data between a browser and a website) to raise awareness of the issues with the move in the industry and to make the adoption process easier for other engineering teams.
“We’re a business directory, why do we need to be secure?” I said, talking to Daniel Applequist (W3C Technical Architecture Group co-chair) in the pub after London Web Standard’s January 2015 event (full disclosure, I help to run this event) where he spoke about HTTPS and web security. He replied:
“Think about what information your users put through that every day, searching for legal counsel and family planning clinics, as well as regular plumbers and hairdressers. They search for it locally, and that is all personally identifiable. If I were a hacker intercepting this traffic I could work out some pretty interesting stuff about you.”
It’s a solid standpoint — in this post-Snowden world, the person snooping on you may not be reaching you through that insecure coffee-shop router that you’ve logged on to whilst waiting for your latte. When traffic is unencrypted, it opens the potential for anyone to identify you and your internet footprint, and Yell could have been a part of this. Therefore, we were determined to increase website security across Yell.com.
Having HTTPS is also proving increasingly important as a search ranking factor. According to Daniel Applequist:
“Google is pushing hard on this, they made it a ranking factor to encourage the big guys to change. If you’re selling this transition to your boss, that’s what you’ll major on.”
The Yell.com directory provides value to businesses across the country through our listing pages and by linking to their websites. For Yell, ranking well in search engine result pages is important and, even if the effect of a HTTPS transition on ranking was very small, this would be a big differentiator between competing directories.
With all this in mind, our engineering team was tasked with moving Yell.com to HTTPS.
I mean, how hard can it be?
In 2015, the year of hoverboards, this should have been simple; it’s just a few certificates in the right places. Put those on the servers, update your links and add some redirects to point to the HTTPS URL , right?
Firstly, Yell.com is old, even in internet terms . It was first registered in 1996 and has had nearly 20 years of patches, upgrades, new components and re-writes, producing what you see today. Whilst we have many modern systems, including our consumer-facing systems, there are less-used (but no-less important) structures that do not use that same technology, or even the same programming language.
We have traditional servers in data centres that are completely in our control, we have some in the cloud, and there are some that are not in our control — services provided by other parties that need to be modified in order to meet our HTTPS requirements. We have a laundry list of third-party suppliers that all need to support HTTPS in order for us to keep that big green padlock in the browser, and then we’ve got to set up efficient rules to take our customers to the right place, including those with 20-years-worth of links around the internet (read, a few hundred million).
We had to make sure we could do this all without downtime, without impacting our users, our website performance and our search engine rankings.
Before starting this journey, I expected I would have to talk to the internal teams, listed in the diagram below to get HTTPS implemented on Yell.com. In smaller organisations, it could even be fewer teams than this, but for Yell, by the end of the journey, this model couldn’t have been further from the truth.
I began by identifying all of the different components of our system that used HTTP as a hard-coded link, and tried to switch them over to HTTPS. The easiest way to do that was to set up a self-signed certificate on my local development machine and see what failed. There are lots of good tutorials on setting up self-signed certificates on different web servers to help with this. I used the Mozilla SSL Configuration Generator to get going, as well as this handy article on getting the green padlock locally on OS X. It’s important to also test the configuration against the Qualys SSL Server Test, to make sure you’ve got a secure connection, serving the right ciphers.
With those set up, I ran the site and looked for errors, and there were a lot of them, including:
- Adobe Analytics
- Adtech display ads
- The entire reviews section of the site
- Quite a few internal URLs, including the canonical URLs of each page
In general, most things worked, and once I’d added an “S” to most of the URLs (or removed the protocol entirely, as is the simpler case) we were getting somewhere. This was the technically easy section, where generally everything worked and the fixes were easy. Then I had to move on to try and fix the other issues.
AOL’s Adtech platform is one of the largest advertising networks in the world, along with Google’s DoubleClick, and they both support serving adverts over HTTPS. A short discussion with our contact at Adtech and a read through their documentation meant the changes could be made and adverts would be served over a secure connection. At least, that’s what should have happened.
Solving this was a human problem — I sat down with Yell’s advertising manager and requested that all adverts be served over HTTPS. They were more than happy to oblige, and ticked all the appropriate boxes for us. They also changed the processes for the advertising sales team to ensure that new advertising is served over HTTPS by default. Even further than this, they agreed to monitor advertising providers and talk to Adtech to change the list of suppliers of general advertising content to those that supported HTTPS — not all of them do. Our team was really happy with the way we were able to work together with Adtech to reach a mutually beneficial outcome here.
This is a challenge that the IAB (Interactive Advertising Beaurau) have written about, citing that 80% of the ad industry supports HTTPS. That’s a lot, but not enough for many, and this can be a major reason for websites to delay their migration to HTTPS. If your website is entirely reliant on advertising revenue, then this migration can be very difficult to justify.
My team released this update by March 2015, well before our main HTTPS transition. By this point in the process, the diagram had grown to include:
Yell makes great use of Adobe’s analytics solution Omniture, (an alternative to Google Analytics), and it is an essential part of our business . Using Omniture we provide valuable data to our merchants on how their listings are performing on Yell.com. In short, it’s a really important tool for Yell, and there are a lot of people in our team working with it.
To make sure there were no issues here, I contacted Adobe’s support team to enable the HTTPS beacon collection for our domain, rather than use a third-party domain. They came back with a form to fill in, a request for a certificate and a recommendation that Yell upgrade our implementation as well as move to a closer data-centre. This all made sense, especially the regional data collection, meaning we’d be sending data 50 miles, rather than 5,000 miles.
This part of the process took time – nearly two months. It involved testing, trialling, upgrading libraries across multiple systems and prioritising this amongst other changes and priorities. This was all quite a logistical challenge, made tougher because it came under a lot of scrutiny. I found myself writing more reports than code in order to complete this task.
At the end of the process, Yell.com is better off in many ways: the upgrade reduced page load times by around 100 milliseconds (ms) and thanks to a new library, our overall analytics capabilities have improved.
Since we went through this process, Adobe have now started offering free security certificates for their first-party collection domains, removing part of this barrier to HTTPS adoption.
This change went live to Yell.com in April 2015 – three months after we began our transition to HTTPS. By now, the departments involved included:
Yell’s reviews platform is one of the older components in the stack, and was inherited from an acquisition a few years ago (Yell.com have since introduced a new, improved ratings system as of December 2015). This meant that some of the technical work for HTTPS had to be repeated, and a few gremlins eliminated. This process wasn’t as simple as it could have been, but we were able to deploy those changes in April 2015. I mention this as a separate section because in any upgrade there will often be one or two systems that just don’t play nicely, and take a significant amount of extra engineering time to create a workable solution. This was one of those components.
By this stage in the transition, we’d reached the point where the bulk of the technical challenges were out of the way and are live (continuously deploying and testing this was one of the best decisions we made on this project). The last thing to do was get the certificates onto the servers and redirect our users.
Buying an SSL certificate is relatively simple for an individual to do. For a single website I’d recommend going to StartSSL to get a free certificate to install. There’s also the LetsEncrypt project, which provides free certificates for services through a simple command-line interface. Yell is a large company, buying certificates in volume for our customers, so we have existing agreements and a procurement process to go through to complete this. This meant the process took longer, but ultimately was a simple matter of talking to our security team. Here, we also wanted an extended validation (EV) certificate, for a few reasons:
- It’s a mark of trust in the organisation
- It’s not much more expensive than a regular certificate
- It’s the only type of certificate that turns the padlock green in Microsoft’s Edge web browser, and provides larger amounts of green in other browsers, which is important for the perception of security
The process for obtaining an EV certificate involves not only proving that you are the owner of the site digitally, but that you exist physically too. For Yell, the trickiest part of this was the third-party company validation, as our domain name was registered for Yell, though we are legally registered under our parent company hibu (UK) Limited. To do this I spoke to our legal team and asked them to adjust the name of the business in our domain name records to match the records at Companies House. I’m pretty sure this is an exceptional step but it is worth noting for businesses who could be registered under alternative legal or international group names. This was relatively simple to solve, but the whole process took four weeks.
By June 2015, the project and the number of departments consulted had grown significantly:
Other technology providers
Yell work with two other technology providers that also came into the process. One is an anti-scraping service and the other is a content delivery network (CDN) for our video products. Both of these requested copies of the certificates we would buy and would charge us a few thousand dollars a year for this process. The actual implementation with these suppliers progressed smoothly, but this was another chink in the chain.
This also required a business case for sign off by the company. Writing the case, taking input from and getting time with all of the stakeholders, and presenting it to the budget holder, came in at a total of three weeks from start to finish. Happily, we were able to purchase the appropriate certificate and update agreements with our suppliers.
For a lot of engineers, writing a business case is a daunting process that they will have never done before. We are very lucky to have team members who have written them before and supported us through the process, but this can be a significant hurdle if you are a lone engineer or don’t have the right level of support within your organisation. One of the few resources I found to help with this was Making the Software Business Case by Donald Reifer (available on Amazon) which was written in 2001. If you have used any more up to date resources in your organisations, please let us know in the comments.
During the business case review, risks were raised regarding the performance of the system. This was taken up with our operations team. who looked into areas which would be effected by the change.
We would be terminating the HTTPS request at our load balancer, which makes it an important point in the chain. This review highlighted improvements which could be made on the load balancer’s TLS performance. We therefore agreed to perform a software update to enable us to handle these requests at speed.
We also agreed to monitor and report on the performance of the website in detail during the transition, using our real user measurement solution, mPulse. This only provides accurate TLS data from Navigation Timing v2 enabled browsers (Chrome, and recently iOS 9) but this is enough for our purposes.
On launch day itself, we updated the following items, all of which are critical to Yell.com’s SEO:
- Our sitemaps, and every URL inside them (> 10M links)
- Robots.txt file, pointing to the updated HTTPS sitemaps
- Canonical URLs for every page, with HTTPS links
- Register Google Search Console for HTTPS and all of the sub-properties, so we can monitor the transition
- 301 redirects to HTTPS for HTTP traffic
My team spent pretty much all day waiting for things to fall over, but they didn’t! The transition was a success, and there is now a green padlock for every user.
Altogether, getting the HTTPS project from ideation to go-live took 7 months and involved over 10 internal departments and five external agencies, visable in the diagram below:
With a transition of this magnitude and a product this large it was no surprise that we didn’t think of everything right at the start. After the launch, we spent a couple of days fixing bugs like e-mail sending, publishing premium adverts and our performance testing suite. The main culprit here was Java.
Yell.com is written in Java, which maintains a list of trusted root Certificate Authorities (CAs). Depending on the version of Java you are using, certain CAs aren’t trusted and therefore the system throws up a security error if it tries to get a resource from a HTTPS URL with an untrusted certificate. Here we discovered that not all of our systems had our certificate provider in their key stores.
It should have been a simple matter of updating the keystore, but for a while this didn’t work. We noticed this only after we had started redirecting internal traffic from HTTP to HTTPS, and at that time, we were redirecting our users too. There was a key lesson here: you may need an internal HTTP service that isn’t upgraded, to deal with any applications that simply don’t want to work with HTTPS.
We have been told that HTTPS is fast, and that it makes very little difference for modern browsers. You no longer need dedicated hardware to perform decryption within a reasonable amount of time — but just how much of a difference does it actually make to your bottom line? Well, looking at our real user monitoring, we can tell you:
HTTPS adds 150ms for desktops and 200ms for smartphones at the median, and over half a second at the 95th percentile
These graphs show TCP connection time for Chrome at the point of redirecting traffic to HTTPS.
That’s a relatively large gap to have to make up, especially at the extremes where 5% of users are taking an additional two seconds to connect while their web clients negotiate the HTTPS connection. If we’re trying to achieve a first render time of less than 1,000ms, then adding another 200ms for TLS negotiation takes up 20% of your entire budget. Add in latency for 3G mobile requests – 300ms at the minimum – and you only have half a second to get everything to the browser. An additional 200ms from TLS negotiation is likely to make that very, very difficult indeed.
To mitigate this, we performed some TCP Profile tuning on our load balancer. The load balancer in question (a Citrix Netscaler) is well-established and commonly used in large organisations in the industry, though there is very little information on how to tune it for performance, save for this slide deck by AOL. So, we looked for tuning options in line with Ilya Grigorik’s High Performance Browser Networking to guide us.
Of the 7 critical features and concepts (which you can see in this table on Is TLS Fast Yet) the Netscaler doesn’t support the ones that provide the best performance, namely OCSP stapling and dynamic record sizing. It also only supports HTTP/2 in the very latest version of the software, which is a killer feature of HTTPS that we weren’t able to implement in time. These are significant limitations of the load balancer, and the feature support table on Is TLS Fast Yet is a great guide to the features of each of the competitors should you be looking to get started with load balancing or looking to move from one to another.
Despite all of this, we were guided to a profile (TLS false start on, record size optimisation, correct window sizing) that saved us about 20ms at the median, and a little more at the extremes. Frankly, that’s not a lot of benefit from the profile — but it’s better than nothing.
In the future, we will look to upgrade to technologies like the HTTP/2 protocol to improve TLS negotiation and reduce the RTT (round-trip time), turning a negative impact into a positive performance benefit.
You may not know this, but a lot of analytics information relies upon HTTP referrers to identify where traffic came from. The referrer tells you the previous page that a visit came from, so it’s useful for businesses of all sizes to identify if a marketing campaign is working, or to track their links around the web.
The HTTP 1.1 spec defines when to pass a referrer:
|HTTPS||Do not pass||Pass|
So, with Yell.com now on HTTPS, for the referrer information to be passed to Google Analytics for example, the destination site must also be on HTTPS. While the rest of the internet transitions to HTTPS, things aren’t looking good. 99% of our customer’s websites are served over HTTP, and of our 1% who are on HTTPS, one quarter are Facebook pages.
This has wide-ranging ramifications for Yell.com.
Although Yell does provide click-through metrics to our merchants directly, Google Analytics is still the tool of choice for many of our customers tracking wide range of business campaigns across the web. So, if our referrer data stops sending information to their dashboards, it looks like we’ve just stopped working. Of course, their overall traffic hasn’t dipped, but looking broken is sometimes worse than actually being broken.
There are other ways to allow the passage of referral data from HTTPS to HTTP through a technology called Referrer Policy. This allows you to control referrers in a fine-grained way, and if you set it to “unsafe”, the referrer will always be passed. However, Internet Explorer does not support this feature, so we would lose a significant proportion of the traffic – whilst it would look like traffic had returned, it would be less than before – so better to not provide in this case.
Alternatively, we could control the referrer using the Content Security Policy, which is how Twitter’s link-shortening t.co service sends referrals. This has even poorer browser support, with Chrome and Firefox being the only supporters, so this was also not an option. For Yell, the only option available was education.
The engineering team spent weeks working with our sales, customer services and marketing teams to help guide customers through this transition. We’re not going back to HTTP (the nuclear option), and this option has never been raised — Google’s position on HTTPS as a ranking factor has always been a strong factor for Yell. Now, all departments are working together to communicate the benefits of the transition to our customers, and to encourage them to consider an upgrade to HTTPS, or to use campaign tags in their URLs — these provide a greater degree of referrer accuracy throughout reporting data, especially in Google Analytics.
This element was the most difficult part of Yell’s transition. It’s unlikely to affect anyone but a directory company, but there will be cases where this really matters to your business.
So, after all that, this ‘simple’ web change has led me to work with the majority of the company:
HTTPS is Hard
This article is written with the best of intentions, to highlight how difficult it can be to implement HTTPS across a large organisation such as Yell and the challenges other businesses can encounter through the process.
For me, the whole exercise was worth it. For all the times my head hit the desk at the stupidity of some technology, and the times where things just took far too long, seeing that shiny EV certificate in my browser is something I’m proud of bring to Yell.com. We resolved far more small issues with all of our systems than I’ve detailed here – in fact, I probably made changes to every single component in our stack to complete this change. For all those challenges, Yell is now in a better position that we’ve ever been in terms of resilience and security.
In summary: implementing HTTPS isn’t just hampered by technology problems, it also has a people problem – namely a lack of incentives.
For many websites, there currently aren’t enough strong enticements to balance out the headache of migrating a site to HTTPS. This imbalance in trade-offs to benefits can be clearly seen in this post. Across the internet, there needs to be a shift towards positive motivations to improve adoption.
How can we make HTTPS easier?
Based on everything Yell has been through to implement HTTPS, I suggest the following:
- Eliminate the cost of standard certificates. LetsEncrypt is showing how this can be made easier and free, Dreamhost is undercutting themselves and Amazon Web Services is providing free certificates for their East-Coast data centre. All of these companies are setting the trends that others need to follow.
- Reduce the cost of setup: I understand that private IP addresses can be costly, but this step needs to get cheaper (or become free) to be successful. The alternative to this, SNI, is not broadly available from service providers at the moment.
- SaaS (software as a service) providers should offer HTTPS by default
- Google, Bing and Yahoo need to make HTTPS a stronger ranking factor. Having it as a tiebreaker at the moment is nice, but it doesn’t have as big an effect as it could. If Google really wants the web to go HTTPS, there needs to be a stronger signal from them on this.
- Improve education on what HTTPS will actually do for you— especially in regards to referrers, performance and certificate issues. Google have recently created some HTTPS transition guidelines, and released their HTTPS transparency report, though currently, there’s not a lot of information out there.
- Implement the privileged contexts spec for developers – there are plans to deprecate access to geolocation, user media and others, whilst service worker will be HTTPS-only. This is known as “privileged contexts”, or more commonly, “powerful features”. Deprecation of features may not be enough here, and this is only viable once the first few items in this list have been made a success or this may have the opposite effect. I look forward to this advancing more in the near future.
- Finally, browser manufacturers need to be sending strong signals to the public, and online companies, that HTTP is insecure. Both Google and Mozilla have plans to action this in 2016, though to what extent it is still unknown.
Overall, the future is bright for HTTPS, though right now, it’s hard.
The Washington Post‘s recent article about moving to HTTPS is also well worth a read.
Steve Workman is discussing HTTPS and the recent Yell.com transition at Front End London on 31 March, 2016. Register for tickets at www.frontendlondon.co.uk