On 5 June 2017, Google released Chrome 59. The tech giant incorporated a number of security changes into its web browser's latest version. Among them, it added Venafi's Gen2 CT log. This trust means Venafi accepts certificates from all roots trusted by Chrome. Stability issues had disrupted Venafi's first-generation log, which prompted Google to remove its trust in March of 2017.
Certificate Transparency logs maintain a record of SSL certificates. They are publically auditable and cryptographically assured using a mechanism known as Merkle Tree Hash. This cryptographic process helps prove whether a log operator has tampered with certificates in their log, such as by inserting back-dated certificates. The log server ultimately signs the Merkle Tree Hash, thereby making it a signed tree head (STH).
CT logs rely on a STH as proof of their trust. If it serves two or more STH that are inconsistent, the log could be corrupt. Such a scenario oftentimes spurs web browsers and other entities to remove their trust for the log.
That's exactly what happened between Google and Venafi after two availability events caused the security firm's first-generation (Gen1) CT log to publish inconsistent STHs.
Venafi's Deyan Bektchiev and Steve Topletz explain that this removal of trust stretches all the way back to 2015 when the key management software provider deployed its first log:
"In September 2015, we initiated our CT log server with a conservative implementation. This led us to rely on non-distributed, third-party infrastructure for our limited initial release. Specifically, the architecture of the Venafi CT server assumed periodic back-ups of the certificate log data in AWS S3, which left us susceptible to the possibility of incomplete archives in the event of outages."
Sure enough, Amazon’s S3 web-based storage service suffered an outage on 28 February 2017. The incident caused the log to publish an incorrect STH for two minutes. Thereafter, the log signed a correct log head, thereby creating inconsistencies. A similar outage on March 13, 2017 created similar issues.
Venafi responded to these availability events by reaching out to those affected by the conflicting STHs. The company also contacted Google, which removed its trust for Venafi's Gen1 log in March 2017. As part of Google’s announcement, the tech giant praised Venafi for its "unparalleled degree of transparency" and for helping to "set a new standard with respect to disclosure and information sharing that we hope all logs will emulate."
Down but not out, Venafi launched into gaining Google's trust for a new and improved CT log. Bektchiev and Topletz elaborate on this point:
"Fortunately, at the time of these incidents, we had already submitted our second-generation CT log to Google in early 2017. It’s built on a distributed CT log infrastructure that is not predisposed to the same types of outages and the resulting inconsistencies that we experienced in our first implementation. However, we’ll look at what improvements we need to make to our second-generation CT log and the lessons learned we need to apply if we want to successfully maintain it moving forward."
Is your organization prepared for transparent and rapid response to incidents so that you can maintain the trust of customer and partners?