Jeremy Kirk: On New Year’s Eve 2019, Don Gibson was on the decks. He’s a trance DJ. Well, he’s not always a trance DJ. He’s also an information security professional, who has held security architect and higher positions for more than 20 years. At the time, he was a security architect with Travelex, the global foreign exchange and remittance company. But on that night, he was just Don1 - the number 1 - on the decks. He thinks he was DJing the Vandalism remix of Back Once Again by DJ Jeroenski when his phone started lighting up.
Don Gibson: It started kicking off. We really started noticing it in the evening, because it had started in the Far East. And as it started in the Far East at 12 o'clock, then the time zones meant that we were starting to get reports of stuff when we still could react. So it was a case of phone started lighting up as I must be New Year stuff, have a look at it and started working on the incident while still on the decks.
Kirk: The incident was ransomware. A note appeared on Travelex computers, which had just been encrypted. It was from the REvil gang, one of the most prolific ransomware groups that’s now defunct. A ransom note read: “It is just business. We absolutely do not care about you or your details, except getting benefits.”
For Don, that night started a turbulent period that lasted throughout the rest of the year. Don’s name became publicly linked with the Travelex incident, and the attention was completely undesired. His story is one of how social media, a frantic incident response and stress contributed to a nearly tragic health outcome. He went from IR, or incident response, to the ER - the hospital’s emergency room.
This is The Ransomware Files. I’m Jeremy Kirk.
In this podcast mini-series, I'm speaking with those who have navigated their way through a ransomware incident and learn how they fought back and what tips they can pass on to others. No ransomware infection is ever welcomed. But there's invaluable knowledge gained. There should be no shame in getting infected, but it's important to share the lessons.
Kirk: There are limits on what Don can say about Travelex’s ransomware infection and its recovery. He’s still bound a nondisclosure agreement. But anyone who follows information security news remembers that the Travelex incident was probably one of the most mentioned data security incidents of that year. It gained notoriety because of Travelex’s footprint: it is one of the largest retail foreign exchange and remittance companies in the world. Just about anyone who has ever passed through an international airport has probably walked past a Travelex desk, where behind the counter is cash from around the world. It operated in at least 70 countries and had more than 1,200 branches. It had its own 1,000-strong ATM fleet, and it had high-security vaults where it stored currency. It was a complicated business and information technology touched all parts of it.
Then, ransomware enters the picture on New Year’s Eve 2019.
Some services go down, others are intentionally taken down. Its websites go offline. Customers can’t order currency. The Travelex Money card, a prepaid Mastercard, can’t be reloaded. Its internal network is partially damaged and encrypted. And it affected the operations of those physicals vaults too by disrupting the movement of cash. And those vaults weren’t just for travelers who wanted, say, some euros to head off from Heathrow to Malaga. Travelex helped supply cash to wholesale clients that the Wall Street Journal reported included central banks, casinos and hotels. In just a few short months, Travelex went into administration, in part because of the ransomware attack and in part due to the effects that the Covid-19 pandemic had on the travel industry. At least 1,300 people in the U.K. even ended up losing their jobs. Travelex is, however, still around today. In August 2020, it completed a restructuring with new investor funds.
But for Don, all of that mess was in his near future. He started with Travelex just eight months prior to the attack.
Gibson: I was there as a security architect to do a large transformation piece across the board. So they had a lot of - they didn't like calling it a legacy - classic systems.
Kirk: That's certainly one way of putting it.
Gibson: That'd be one way of putting it. I'd been brought in to help usher through certain things. For example, moving to Office 365, making sure that's entirely up and working. They wanted to transform rather than lift and shift, because obviously, they're two very different things. It was a case of "No, we want to go through the business system, we want to understand the data flow, we want to understand the actual requirements of the business. And we want to design the system itself. And this brand new system that can be evergreen, it can be designed properly, the security the first way, the first time, etc. and therefore we will have a very solid foundation for the business to use for the next 20 years."
Kirk: Don didn’t have anything to do with the day-to-day security operations - the usual monitoring and investigation to keeping everything tidy and secure. But after the infection kicked off, everyone helped with incident response.
Gibson: It was really all hands to the pump. Everybody who knew anything, get in and help. January 2020, I worked over 375 hours. That's over three months worth of work in one. My boss, I believed did more. Towards the end of the month, it was getting ugly. Personally getting ugly.
Kirk: Don can’t talk a lot about what happened during incident response due to the NDA. But the bits he worked on, luckily, were OK. That included Office 365, which ran in the cloud. And that was great because it meant the organization as a whole could communicate and use email. REvil’s ransomware did have some slight effects on his patch. But Don says he’d put in place two backup schemes. Microsoft would backup about last 14 days, and Travelex also used Barracuda for 365 backups.
Gibson: So we were able to just wipe those files and recover them, just like that.
Kirk: So Don’s part of the network was alright. But a broader question was, how were Travelex’s systems infected in the first place?
Gibson: The automatic question is, if your systems are so good, how come you were attacked? You didn't get alerts or whatever else? I can't answer that, or that. I'm not allowed to answer that.
Kirk: Don says to this day, he still doesn’t definitely know the initial infection vector. The incident response report has never been released. But there is a plausible theory for how it happened.
Around five days after the attack, a likely source for Travelex’s infection emerged on Twitter. A Chicago-based threat intelligence company called Bad Packets tweeted that it had sent a warning to Travelex in September 2019. Travelex had seven servers running Pulse Secure VPN software that did not have a critical patch.
Here’s a bit of background on what all that means. VPN stands for virtual private network. It’s software that makes a secure connection between your computer and another one, or in the case of Travelex, its corporate network. Companies often use VPNs as a gateway to their internal networks, but they’re falling out of fashion in favor of more secure alternatives. Attackers who obtained login credentials for someone’s VPN could get access to a company’s entire network. The Pulse Security vulnerability, which was CVE-2019-11510, was really bad and easy to exploit. A remote attack could result in an “arbitrary file reading.” In this case, that arbitrary file could contain hashed MD5 user passwords, cached plaintext passwords, cookies, private keys and configurations for other things like LDAP and SAML.
Bad Packets was founded by Troy Mursch, and his company is one of the good guys when it comes to defending against attacks like ransomware. It has a worldwide network of what are called honeypots. Honeypots are servers or computers that are designed to look appealing to malicious hackers and criminals. These online traps can give an indication when bad actors are scanning the internet for new systems to attack. By laying these traps, Bad Packets can collect intelligence on botnet activity, vulnerability exploitation trends and other data that’s useful in defense.
In 2019, ransomware crews were had a field day with VPN vulnerabilities. That year, there were severe vulnerabilities found in VPN software made not only by Pulse Secure, but also Palo Alto Networks and Fortinet. Pulse Secure issued an emergency patch in April 2019 for CVE-2019-11510. But many organizations were not patching their vulnerable applications despite many warnings.
In August 2019, Bad Packets detected that someone or some group was mass scanning the internet and trying to find unpatched instances of Pulse Secure and other VPN software.
On Aug. 19, 2019, Bad Packets found about 14,428 Pulse Secure VPN endpoints on the internet that had not patched for CVE-2019-11510. Those affected included government agencies, schools, hospitals, utilities, financial institutions, media outlets and Fortune 500 companies, including Travelex.
Bad Packets then undertook a massive, unpaid effort to try to warn organizations of the dangers of leaving Pulse Secure unpatched. It sent out emails to thousands of organizations. It was a herculean, good Samaritan effort. And, in part, it worked. After 11 days, scanning showed that more than 4,000 previously vulnerable Pulse Secure servers had now been patched. But not Travelex.
Fast forward to early January. Travelex is about five days into its incident response. On Jan. 5, 2020, Bad Packets tweets the actual warning email it had sent Travelex on Sept. 13, 2019. The tweet says: “We notified Travelex of their vulnerable Pulse Secure servers on Sept. 13, 2019. No response.” Meaning no response from the company.
Three full Travelex email addresses are in plain view in the screenshot. If you want to niggle about it, two of the email addresses are considered personal data according to Europe’s General Data Protection Regulation. Don’s email address is on there, and it immediately starts causing him problems.
Gibson: It gets alerted to me by a member of my team who followed him. They say: "Don, I think you need to see this." That was Jan. 6 or 6, and I'd have had maybe four hours of sleep since the new year's day. And I'm going "what now?" and that was the answer. So yes, not great.
Kirk: Of the three Travelex email addresses, only one was actually monitored. That was Don’s. And the message from Bad Packets didn’t clear the spam filter. It went to his junk mail. But Don managed to catch it.
Gibson: I genuinely look at my junk folder a couple of times a week. I do remember when I saw it. I was in a very boring meeting and my attention was wandering. So I sent the email off to the person it should have gone to. I don't even know how they picked out my email address.
Kirk: As he says, Don forwarded the email to the appropriate security team at Travelex that handles patching. He says he doesn’t respond to people he doesn’t know over email, which is why Bad Packets didn’t get a response. That was the end of it for Don, until the email surfaced again on Twitter on Jan. 5, 2020.
Don says he then starts getting contacted by journalists. Threat intelligence companies eventually use the screenshot of the email in their literature as an example of what happens to companies that fail to promptly patch. His name becomes the closest direct one attached to a failure to patch even though he had nothing to do with patching. And overall, this came as Travelex was just getting battered on social media.
Gibson: The team is going through so much. I am personally going through so much. Because all of a sudden, there's a name to this, and it's my name. Which means I'm getting contacted by press. And I don't need that. I'm already in a really bad place. Because the company I'm working for is in trouble. And I'm trying to fix it.
Kirk: I contacted Troy Mursch of Bad Packets. He declined to comment. I don’t think there’s actually a lot of daylight between Don and Troy, however. Troy was trying to help when he initially sent the email, and Don was trying to help his own company by forwarding it on. They both were doing the right thing.
But, in retrospect, it might have been prudent to at least redact the email addresses before posting that email just days after the attack. Maybe saying “I told you” so soon after an attack is something that could have waited. Don’s still a bit sore about it, but he also doesn’t want that to take away from the good work that Bad Packets has done.
Gibson:I do have some admiration for Troy Mursch and Bad Packets for the fact that he's actively going out and trying to help. That is why I'd be willing to give him a pass on this one.
Kirk: How much blame should be apportioned to organizations that get infected with ransomware? It’s often cast as a shameful situation, and no one feels worse than those on the IT security team. We have to remember that Travelex - like any other organization that is hit by ransomware - was a victim of a crime.
And while it is possible to spot fault from the outside, only people who worked inside Travelex know what went down. Did the patch team see the email that Don forwarded from Bad Packets? What were Travelex’s patching policies? Was there someone on Travelex’s IT security team jumping up and down about how important it was to patch Pulse Secure, but a manager dismissed the warning? To this day, we still have an incomplete picture.
To be sure, Travelex didn’t help itself either. It was a bit slow to acknowledge the obvious, which is that it had been struck by a ransomware. And then shortly after the attack, the story got even messier. REvil claimed it had also stolen 5 gigabytes of personal data, including birth dates and Social Security Numbers, before encrypting Travelex’s systems.
This is often referred to as double extortion. Ransomware actors grab as much data as possible from a victim’s network prior to encrypting it. Then, if a victim has good backups and doesn’t need to pay to decrypt their data, they might pay to prevent the release of that sensitive data. Travelex, however, said about a week after the attack that there was no evidence that data had been taken.
Around Jan. 17, 2020, Travelex’s top executive finally got in front of a camera and recorded a short statement on the state of the company.
Tony D'Souza: "Hello, I'm Tony D'Souza, CEO of Travelex. This is the first opportunity I've had to speak to you directly about the cyberattack that impacted us from Dec. 31."
Kirk: Well, he probably should have fronted the public sooner than 18 days or so after the attack. But, he continues.
D'Souza: “At all times, we remain focused on protecting our customers' data and containing the virus. We engaged internationally renowned cyber experts to run forensic analysis, and we have not to date uncovered any evidence to suggest that any customer data has left the organization.”
Kirk: So that’s important because Tony repeats there’s no evidence personal data was taken. We know that REvil often stole data before actually launching the ransomware, so that' great for people who use Travelex’s services and provided their personal data.
Then in April that year, the Wall Street Journal reported that Travelex paid around $2.3 million in bitcoin to the REvil attackers. The source was anonymous, but the Journal wrote that it was someone who was familiar with transaction. The Wall Street Journal also said the REvil group itself had claimed to it in January that it had received payment.
It appears that the payment wasn’t to stop the release of customers’ personal data, and it appears that REvil was bluffing. The U.K.’s data protection authority is the Information Commissioner’s Office. The ICO told me it was satisfied that no personal information was breached based on information Travelex provided.
So why did Travelex pay REvil? Was it to get a decryption key for data that maybe wasn’t backed up? Maybe Travelex panicked in the early days after the attack and believed REvil’s bluff that it had a lot of personal data. We don’t really know.
There’s one last bit to this part of the story. A Travelex customer named Robert Picon filed a class action suit in New York federal court against Travelex in June 2020. The lawsuit alleged that Robert and 100 other class members were at an increased risk of identity theft as a result of the attack. Travelex challenged the suit on several grounds. It contended that Robert hadn’t demonstrated what personal data of his was compromised as a result of the ransomware attack and what harm came of it. The court record shows the lawsuit was voluntarily dismissed by Robert’s legal team in September 2020.
By September 2020, Don had already had a very long year. His health was wavering.
Gibson: The team and I were all absolutely spent. Even the small IR team. The CISO and I alone had put in over half a year's worth of work in a month; just the two of us. There are others that were just as ragged. The company didn't apply any time in lieu or anything. That sounds absolutely terrible, and it's not great.
My heart has started messing around. To the point where it was really, really weird. An arrhythmia was found - atrial fibrillation is the name of it. It's where there's an anomalous current going between the top two ventricles of the heart and instead of getting squish squish, squish squish, they start getting all messed up. And so I ended up getting a little heart monitor.
Kirk: Don’s a gardener. In September 2020, Don went down to a machinery shop that was having a sale.
Gibson: I popped in to see, and their shelves were more or less bare, except for a wood chipper. And I'm sitting there, going f**k yeah.
Kirk:He got to work with the wood chipper, which worked brilliantly. Until...
Gibson: Until I started not feeling so good. And my vision started going slightly grey around the edges. And I didn't feel good. So I thought, "let me take myself in and check my heart rate." It was around 280 BPM. I told myself, "I'm a trance DJ. The fastest I've ever recorded a track was exactly half that, and that was some pretty pounding stuff.
Kirk: He didn’t go the hospital. He went inside and took his heart pills. He called his dad the next day, who is a medical doctor. He sent his dad a picture of his heart trace. His dad told him to go to the hospital immediately.
Gibson: I was immediately put into the ICU. About a month later, I was on the slab, having a four-and-a-half hour heart operation, which was not on my to-do list at 44. Definitely not in my life plan. And I can tell you heart surgery sucks. But it's better than the alternative.
Kirk: He left Travelex at the end of 2020. He’s now the head of cyber for the U.K.’s Department of International Trade. Two of the people who worked at Travelex did reward him for the very long year, but not quite in the traditional way. It was in a way that meant a lot more to him.
Gibson: And this is where the really beautiful thing about my boss and my boss's boss come out. I left Travelex. I was going to be a civil servant and civil servants don't get private medical. And so as part of my leaving package, they gave me a year's medical from Travelex to look after me because of everything that was going on. And for that, I can never thank them enough.
Kirk: The Travelex experience has left a lasting impression on him. He speaks at conferences about the importance of taking mental and physical health into account not only for CISOs but for anyone on a security team. Don himself is neurodiverse, and everyone has different thresholds for stress. He says he remembers an exchange just days after the ransomware struck with Travelex CEO Tony D’Souza, who sat behind him in an open plan office.
Gibson: He started to understand the cyber ethos, maybe the third or the fourth day since the attack. We were the only two in the office. I turned around and went: "How are you doing?" And he went, "Pardon?" I said: "You're the head of this. Everyone's looking to you. How are you doing? Are you alright?" And at that point, he understood that all of us were trying to look after everyone.
It's a tough place to be. When the brown stuff starts hitting the revolving objects, it's a lonely place to be. You're trying to liaise between the NCSC, the police incident response teams, your board… there's a lot going on. And health very quickly slips to the bottom of the priority list. You're running on adrenaline. Yeah, adrenaline only takes you so far.
Kirk: Don says people need rest during incidents even when the work appears like it is never going to end. There should be at least two teams that can rotate so the other gets rest. Companies should practice their IR plans to have a clear idea of what they’re going to need to do and how difficult those tasks may be. And lastly, he says that major responses to an incident such as ransomware aren’t just a IT or cyber thing - he says it’s an everybody thing.
Gibson: This is one thing that I've been very vocal about. Ransomware is not a cyber problem. It's a technology problem. It's an HR problem. It's a commercial problem. It's a board problem. Yes, it's a cyber problem. It's an everybody problem.
Kirk: This episode of The Ransomware Files was written, researched, edited and produced by Jeremy Kirk. The production coordinator is Rashmi Ramesh. The Ransomware Files theme song and other original music in this episode is by Chris Gilbert of Ordinary Weirdos Records. If you enjoyed this episode of The Ransomware Files, please share it and leave a review. It will help keep this project going. The series has its own Twitter handle @ransomwarefiles, which tweets news and happenings about ransomware. I’m on Twitter @jeremy_kirk. If you would like to participate in this project or have an idea for it, please get in touch. The project is looking for other people, organizations and companies that can share their unique experiences for the benefit of all until ransomware, hopefully, becomes a thing of the past.