Verizon's PCI DSS Report: Bad Comparisons?Careful Staging of Data Leads Verizon to a Predictable Conclusion
On Thursday, Verizon released its fifth study of the Payment Card Industry Data Security Standard, or PCI DSS, compliance landscape, called the 2017 Payment Security Report. It's a dense, 44-page report that dives into how organizations are keeping up - or not - with the complex recommendations to keep payment card details secured from hackers.
But studies performed by vendors are invariably done for one purpose: driving their own business case. That doesn't necessarily mean the conclusions are invalid or should be dismissed outright. But a close look at Verizon's report reveals the company makes suspect logical stretches.
We don't know for sure if Verizon's PCI compliance validation customers have never been breached. We only know that Verizon says that they haven't reported a breach.
As a PCI Qualified Security Assessor, or QSA, Verizon is a giant in PCI assessments, the lucrative consulting field that emerged as a result of organizations trying to follow byzantine PCI DSS mandates and recommendations.
The report broadly contends that organizations often nail a PCI DSS assessment but tend to slip out of compliance. This is well-known: A simple network change can throw an organization out of compliance or at least into a state of lesser compliance.
Verizon attempts to show how this compliance slippage puts an organization at greater risk of a breach. This is uncontestable and makes sense. But how Verizon goes about illustrating it is questionable.
Starting on page 35 of the lengthy report, Verizon compares two groups, concluding that one has slipped more than the other when it comes to PCI. The first group is composed of organizations that have contracted with Verizon for forensic investigators following a breach. The other group is organizations that have contracted with Verizon for a QSA interim compliance validation, which is sort of a check-up on PCI compliance. This is what Verizon labels as the control group.
"We see very clear indicators and correlations between these two data sets," the report says. "Our analysis identifies common breach vectors and extrapolates the control(s) that would prevent similar breaches from being successful."
Fish in a Barrel
Finding an out-of-whack PCI control in any organization, whether breached or not, is like shooting fish in a barrel. And that determination is often subjective.
I shared Verizon's report with Avivah Litan, Gartner's resident payments guru. She tells me that "measuring PCI compliance is more art than science."
"Compliance and checklists are a leaky sieve," she says. "It's not that difficult to find non-compliance when there are hundreds of controls mapped against thousands of moving parts. As someone from a card brand once told me years ago - 'companies who are breached are by default non-compliant with PCI'."
But we all know that some of the most significant payment card breaches - think Heartland Payment Systems, Target, Neiman Marcus - occurred when the companies were PCI compliant on paper.
Verizon seems to not acknowledge that this scenario has happened before, or at least not with its own assessment customers. "Were a compliant entity to be breached, it would probably indicate circumvention of multiple control layers by the attackers and/or exploitations of ineffectively implemented controls - and it would make a fascinating case study."
Fascinating, indeed. Verizon conveniently adds on page 35 that "none of Verizon's PCI customers have reported a payment card compromise after being assessed by Verizon [for interim PCI compliance validation] and thus are not included in the confirmed compromise data set."
Clearly, the subtle message that Verizon is attempting to make here is this: Use Verizon as your PCI QSA. But it must be pointed out that, in fact, we don't know for sure if Verizon's PCI compliance validation customers have never been breached. We only know that Verizon says that they haven't reported a breach. And if any of those organizations in fact do end up reporting a breach, It's pretty much assured a forensic investigation is going to turn up some kind of PCI-related failure.
Too Many Assumptions
In a briefing with me, Ashish Thapar, a managing principal with Verizon's Investigative Response unit in Singapore, acknowledges that organizations sometimes simply don't report breaches.
As for those that have been breached, it's highly unlikely any PCI analysts would certify them as perfectly compliant after they've had an incident. Thapar assured me that of the 300 plus forensic investigations that Verizon has done between 2010 and 2016, all involved some sort of PCI stumble.
Verizon's press release also addresses this, contending that none of the organizations where it did post-breach investigations was "fully compliant at the time of the breach, and showed lower compliance with 10 out of the 12 PCI DSS key requirements."
"Lower compliance" is where Verizon wanders into squishy territory, especially when it presents the two groups in a sparkling contrast. Clearly, the group of organizations that had experienced a breach was never going to win. After further questioning, Thapar backed off somewhat on the allegedly sure-fire link between lax PCI and breaches.
"When we do PCI assessments, there is a finite scope of the assessment," he says. "Now when entities get breached, sometimes that finite scope overlaps with the breached asset; sometimes it doesn't. So it's difficult to say that the breach was a direct outcome of the PCI non-compliance because PCI compliance is code-based and a breach could be completely difficult to scope in an organization."
Verizon does insert the proper caveats in its report. On page 37, it says: "Being compliant with PCI DSS does not guarantee security - though it certainly helps."
Verizon appears to have anticipated that its press release may come across as too ham-fisted. It bluntly writes that the "aim of the 2017 PSR is not to convince readers of the need for PCI compliance but track that measurable performance of PCI compliance."
But one parallel goal is the former, of course.
Both Sides of the Wall
Litan tells me that Verizon's data on PCI compliance and breached companies is much appreciated by the industry. But she says that as far back as 2008, she and then-fellow analyst John Pescatore wrote about the conflicts of interest that exist in companies, including Verizon, that are in the PCI QSA business and also provide breach remediation services.
"PCI assessors should not be able to cross-sell managed security services (e.g. threat detection, device monitoring) without a 'Chinese Wall' between those two parts of their business, much like the walls that exist in Wall Street banks between the investment banking and brokerage services of their institutions," she says.
Verizon's access to information on both sides of that wall - and careful staging of it in this report - unfortunately undercuts a point that no one would really contest: PCI DSS helps to better secure payment card data. So why needlessly pit the "losers" (breach victims) versus the questionable "winners," which use Verizon's PCI assessors?
Unfortunately, this muddles the point that PCI DSS does help ensure that organizations better secure their payment card data. Verizon would have done better to end its PCI services pitch on that real-world note.