Looting of the Fox: The Story of Sabotage at ShapeShift

Looting of the Fox: The Story of Sabotage at ShapeShift

65254
62
SHARE
ShapeShift Erik Voorhees

Bitcoin, as any system of man, exhibits together both the highest ideals of utopia, and the lowest residual trash of society.

[Note: some names & sensitive details have been changed]

Erik VoorheesThis is the story of how ShapeShift, a leading blockchain asset exchange platform, was betrayed. Not once, not twice, but three times in less than a month.

In total, nearly two-hundred thousand dollars in cryptocurrency was stolen by thieves within and without, not to mention the significant resources expended in its wake. Nevertheless, no customer funds were ever lost or at risk, a milestone for an industry pocked with past tragedy, and ShapeShift itself has adapted and rebuilt, humbled by the experience learned, and ever more resolute in its mission of safe, frictionless asset exchange.

In the spirit of Bitcoin’s openness, we wanted to share this story with the community; may you be informed, entertained, reflective, and ever-diligent in your own affairs.

The Backstory

Since its inception in the Spring of 2014, ShapeShift has been an evolving creature. What began as a quick experimental way to swap between Bitcoin and Litecoin grew into an advanced engine for the effortless exchange of all major blockchain assets, each one into the other, with no user friction. No user accounts. No signup process. It is the Google Translate of cryptocurrency.

And we’ve always been playing catch-up. Trying to build at the speed of this industry, not only along the vertical of Bitcoin proper, but along the breadth of all crypto, is a challenge.

Last Fall, we realized the “minimum viable product” server architecture established originally for ShapeShift was insufficient. We needed a professional to join the small team, and craft a scalable, and secure, server apparatus upon which our technology could grow.

We hired such a person, and patted ourselves on the back for our proactive decision. On paper, he looked great; the reference we called confirmed his prior role and responsibility. He’d even been into Bitcoin since 2011/2012 and had built miners in his room. Awesome. We’ll call this new employee Bob… indeed his real name starts with a B.

Over the next months, Bob built and managed ShapeShift’s infrastructure. He did okay, nothing special, but we were content to have a professional taking care of devops at least well enough to enable our engineers to build upon the architecture.

In the first quarter of this year, as the market discovered what we already knew – that our world will be one of many blockchain assets each needing liquidity with the other – exchange volumes surged at ShapeShift. Ethereum was on the rise, specifically. Our infrastructure was not ready for the pace of growth. It was like riding a bicycle upon which jet engines suddenly appear full-thrust

Unfortunately, Bob did little to be helpful. He puttered around aimlessly while the team worked long hours to keep the ship together.

Scratch that, actually, Bob was not aimless.

He was preparing to steal from us.

The Genesis Betrayal

On the morning of March 14th, in the midst of one of our heaviest volume weeks ever, I get a call from our Head of Operations, Greg. “Erik, our hot wallet is missing 315 Bitcoin.” Why did we have so much in a hot wallet, you ask? Well, with volumes surging, our hot wallet would be drained through normal business in an hour at that level, which then required constant manual rebalancing. Are there ways to automate and reduce that risk? Absolutely… but hindsight of one’s development priorities is always 20/20.

So 315 Bitcoin was gone.

To those who have experienced such incidents, the feeling of sickness is profound. It’s a deep, dismal state, that doesn’t stop at the edge of financial loss, but permeates down to one’s core. When systems are breached, systems that one has engineered and cared for deeply, obsessively, that violation of what one considers safe and secure is very, very uncomfortable. And then there’s the loss itself. 315 Bitcoin… roughly $130,000. That’s college tuition, part of a house, food for ten years… a couple months of payroll. It’s a lot of money for a pre-profit startup.

I rushed to the office, hoping there was some mistake. The only comforting thought was that the loss was only our own money. With no customer accounts, neither customer funds nor personal information were at risk from the hack. That was by design from the beginning of ShapeShift; one of our tenets. But even if nobody nearby is harmed, a punch in the face still hurts like hell.

Myself, Greg, and our two lead engineers poured through logs and servers, trying frantically to figure out what had happened. The 315 BTC went to an unfamiliar Bitcoin address, and was sitting there.

Indeed, it sits there still: https://blockchain.info/address/1LchKFYxkugq3EPMoJJp5cvUyTyPMu1qBR

Despite our note to all employees to come into the office urgently, Bob, our head IT guy, the one responsible for security and infrastructure, arrives at 11:30am.

We ask Bob to join our discussion. We reveal the hack to him. We ask him if he had logged in at all that morning, to which he responded no (on several occasions). On the new of the theft, he seems neither particularly shocked nor outraged, yet it was his security that failed us. Immediately, he starts pointing to red herring explanations, “It must be one of the exchanges that got hacked, that happens all the time.” Umm, our exchange accounts are fine, Bob.

“Well, look at the IP address, it happened somewhere off west Africa.” Umm, IP addresses on block explorers indicate only the first node that noticed a transaction, and are generally meaningless in the context of Bitcoin, Bob. (What kind of Bitcoiner doesn’t know that?)

Very quickly, we realize he is pretty much useless. Here we have our “server guy” and he has zero intelligent comments about a hack against his own infrastructure.

While pouring over logs we noticed, however, a couple SSH keys (belonging to Bob) that had logged into the breached server that morning an hour before the rogue transaction, and then logged off two minutes after. Not nefarious, necessarily, for indeed Bob’s keys would be expected to log in periodically, though the timing was strange (6am-ish in the morning). We also discovered the breach occurred over the VPN, meaning someone in the office, or someone with access to our VPN, committed the theft.

We ask everyone with server access to provide the fingerprints of their SSH keys so we can start comparing them to logs. Everyone does so, but another strange thing: the fingerprint of the key handed in by Bob doesn’t appear in any logs. It appears brand new. Strange that the key of the server admin would never have been seen on any server…

Soon after, Bob decides it’s time for his lunch break, and we don’t see him for an hour, during the worst incident in ShapeShift’s history. We frankly didn’t care that much, he wasn’t helpful and suspicions were starting to creep in. He tells all of us that he’s leaving his laptop open to download some logs, and makes sure we see that the laptop is left open. He’s being a little weird.

Upon his return an hour later, he is sitting down with other engineers still investigating what occurred. I’m in the other room on a call. When I finish my call, I come check on the progress. Bob appears to receive a call “from his mother who needs to go to the hospital.” He packs up his stuff, grabs his dog who was at the office, and heads out. We’re all half relieved for his departure and half in awe… did our server admin really just leave for the second time during our investigation, which he should be leading?

He says, “I’ll be back within an hour.” This was at about 3pm, March 14.

We never saw him again

Shortly after he leaves, one of our engineers pulls myself and Greg aside, and says, “While you were on your call, we were all sitting around the table, and we saw in the logs that Bob deleted two SSH keys while he was sitting there with us, then he grep’d several times for them [a server command to find specific text], and then he left. Those two keys matched the two keys we saw in the log this morning which accessed the Bitcoin server just prior to the hack.”

He just deleted his keys from the server?? Well fuck. Guns don’t get any smokier than that.

We all immediately move to the assumption that Bob stole the funds. He is out of the building, and so we start locking everything down. All keys are changed in haste (well, almost all).

We work for a few more hours, no word from Bob. No calls, no texts, nothing. By the end of the day, it had been 3-4 hours since he left to “take his mother to the hospital.” We decide to call him, without letting on our suspicions just yet.

“Hey Bob, where are you?.”

“Oh hey, I just decided to go home.”

“You’re at home?”

“Yeah, just here, working on some stuff.”

WTF?

That call is innocuous, but we recorded it. We also recorded the next one 30 mins later, in which we confront him with some of the evidence.

“So Bob, it looks like you deleted your SSH keys, and gave us a new key that had never accessed any servers.”

“Yeah, well I deleted them because I didn’t think they were important.”

Yes, he actually said that. Our server admin, in the midst of an investigation into a $130,000 theft, deletes his two keys, and only these two keys, without telling anyone, and then admits on our call that he did it because “they weren’t important.”

It just so happens those two keys were the exact ones logged into the Bitcoin server that morning, and which logged off two minutes after the theft transaction. Not important indeed!

He gives no explanation of his behavior or actions that day, but dances around questions and implies, subtly at first, and then more explicitly, that we’re being racist.

“Umm Bob, we’re targeting you because your keys were on the server, and you deleted them and left, during an active investigation.”

It goes on like that for 45 mins. He says other ridiculous stuff, all recorded.

We uncover further evidence details, and there is a sense of relief after knowing exactly what happened and who was responsible. We spend the rest of the evening documenting everything, and preparing to file civil and criminal charges against Bob.

I give him a final chance that evening for redemption. In a message to all employees, so as not to force him to implicate himself by responding,

This is your chance to walk away, learn a lesson, and let this be closed. We will not pursue legal action if 315 Bitcoin are found in this address by 10am. No further questions will be asked, and we can part ways amicably. Send 315 BTC here: 35JBgzjyCUPswjRP9iqrUTkkX76QwrKkB9 -Erik

I get a response message from Bob at 4:36am, “I didn’t delete any keys and I regularly log into servers to check them out.”

Right, except that we have him already on record saying he did delete the keys and hadn’t logged on that morning. His ineptitude at lying appears outmatched only by his incompetence in server administration.

He goes on, with charming adolescent flare…

“Of course blaming me is the racist thing to do… you were basically looking for an excuse to satisfy your racism. I have no criminal history unlike you with the SEC.”

The next morning, our general counsel writes a formal letter (via email and post) to Bob, outlining some of the evidence that we knew, and demanding the stolen property be returned. It also notified Bob that his employment was terminated (I think that was fair, considering). In response, Bob emails back to the lawyer, addressing none of the evidence whatsoever,  “Your clients are racist so make sure you know who you’re dealing with.”

It’s like he was wearing his internet troll hat in real life. Did he not even understand the seriousness of the situation? Well… the absurdity was just getting started.

Over the next days, we file the formal civil complaint. The address Bob had given us was a PO box, though we had his legal name, his bank info, and his social serfdom number. We hired a private investigator. We found his apartment within a couple days. Several attempts at service failed, though the investigator heard a dog barking behind the door. One of his cars was found; he drives two unmarked retired police cruisers.

I have investors to whom I owe a level of protocol diligence, so, we also made arrangements for a criminal case, and herein the theft constitutes a Class 3 Felony, with 4-12 years in prison. Honestly, I don’t care whether he is punished. I care whether we are made whole, and whether he realizes his error and changes his life to become a better person. No sign yet, of that.

We learn some more things. Bob has prior police records in Florida, where he’s from. Incidentally, the records indicate he’s white, after all.

With civil and criminal cases proceeding against him, and with further discovery that Bob fled to Florida (leaving his dog to be temporarily cared for by his neighbor… who is now wondering where he is and hasn’t heard from him in weeks), we thought the case was basically closed. We’d get him somewhere, sooner or later. And, hopefully, we’d get our stolen property returned, or the fiat equivalent.

Rovion

We’d worked to build a new server infrastructure in Bob’s wake, assuming his work in our system to be largely compromised. We set up a new cloud architecture with a company we’ll call CloudCo.

It’s now the week of April 4th, and we were about ready to go live with this new cloud infrastructure. Then all hell breaks loose. Again.

On Thursday April 7th, around midday, we notice a bunch of Ethereum had left the hot wallet on the new infrastructure at CloudCo. The NEW infrastructure. The infrastructure that was not even public yet. At first, we believed our code had done something weird, perhaps sweeping funds to a development server address or similar. Then we noticed a bunch of Bitcoin was also missing. And then Litecoin also.

Thief’s Bitcoin address: 14Kt9i5MdQCKvjX6HS2hEevVgbPhK13SKD

Thief’s Ethereum address: 0xC26B321d50910f2f990EF92A8Effd8EC38aDE8f5

Thief’s Litecoin address: LL9jqgXVqxUbWbWVaJocBcF9Vm8uS3NaTd

And very quickly reality hits you, and that’s what flashback feels like. The horrible sinking feeling sets in immediately, once again. What the fuck happened?

Keys that were not even on publicly known servers had been compromised, somehow. We shut the system down, including our live production site, while we investigated. We didn’t lose as much as the hack a month prior, because we’d be keeping wallets somewhat conservative, but it was still quite a bit. We couldn’t believe it. How could brand new keys, generated with brand new infrastructure, be compromised?

After several hours of fruitless investigation, we decide that one of the most likely explanations is that the cloud company itself was compromised. This has happened before in Bitcoinland. We thought CloudCo was reputable, but who knows? Clouds are very convenient and scalable, but on some level you’re trusting that company with your infrastructure. We decided we had to keep the site down for at least 24 hours, and bust our asses to prepare, yet again, an entirely new infrastructure on an entirely new set of servers.

What was nearly as bad as the money lost was not knowing how it happened. Logs were not done as well as they should have been, so they proved fruitless. Indeed, they had been wiped.

Despite that, we watched the blockchains for the hacked funds. We tracked some to an exchange account. We got profile information of the depositor.

Name: Rovion Vavilov

Email: rovion.vavilov@riseup.net

Address: Chayanova St. 15, Moscow

DOB: Feb 2, 1980

Phone: +7 9625148445

That profile information was probably fake, but I emailed him that night.

From: Erik Voorhees erik@shapeshift.io

To: rovion.vavilov@riseup.net

Subject: ShapeShift Hack…

“Nice job on the hack. How did you do it? -Erik”

Pro Tip: Black hats like to be recognized for their skill, regardless of how immoral their deeds may be. Talk to them calmly, as adults. They may reveal information, or help in some way. It’s weird, but it happens. In any case, I didn’t expect anything to come of my email.

The rest of that night, and into the next day (Friday, the 8th), the team worked feverishly to rebuild everything on new infrastructure, once again, in a wholly clean environment on a wholly separate host.

Now to many, ShapeShift appears to be a simple web service. It’s taken a lot of work by our engineers to keep up that appearance. Behind the scenes, the platform is complex. Over 1,400 direct asset trading pairs, integrations with half a dozen exchange API’s requiring real-time price information on all offered cryptocurrencies, low-latency service API’s to several dozen partners, the monitoring and calculation of constantly changing exchange rates and order book depth in some of the most volatile markets on Earth, and incorporation of what can only be described as alpha-level software in various states of disarray (coin daemons…bleh).

And in Bitcoinland, indeed, and there is no guide book.

Admittedly, as a non-engineer myself, I can only occasionally glimpse the magnificence of what we’re building. I wish I could take credit. To our team reading this, you have engineered an amazing machine and should be very proud of it.

And now here is where the story deepens

Around mid-day on Friday, the hacker responds to my email (remember I had asked him how he did it…)

From: rovion.vavilov@fastmail.com (noted new domain)

To: Erik Voorhees erik@shapeshift.io

Subject: ShapeShift Hack…

“One word: Bob”

That was the entirety of that first email, but we were stunned. For a moment, we thought, “Is Bob the hacker?” Quickly, that notion gave way to the more likely answer: that Bob sold or gave away our information to a hacker, who then exploited it.

Bob betrayed us. He betrayed his privileged position, profiting directly from the destruction of those who trusted him. He stole, lied, ran away, and then after being afforded a period of time long enough to reflect upon his actions, decided to betray us again for a few more scraps in his pathetic bowl. Hackers gonna hack, but it takes a certain variety of bastard to ascend to a trusted position, work face to face with a team, receive a salary and confidence from that team, and then screw them all for barely enough money to buy a Tesla. Oh yeah, and then abandon a dog to starve alone, likely soon to be put down by animal services.

Watch out for these people in your lives. If you suspect them, sever ties quickly.

Anyway, after herculean efforts, we had everything ready by Friday night, 24 hrs later. We launched the site on yet a new provider, who we’ll call HostCo. Despite a couple glitchy bugs, the system was running. We had told the public about the hack and decided to release more details once we studied the compromised environment in more detail later.

Exchange orders started up immediately. We breathed a sigh of relief. I fell asleep around 1am and slept peacefully, exhausted from the ordeal and very proud of the team.

Then it was Saturday 9am, and I start emerging from slumber. My phone rings. It was Greg.

“We were hacked again. Bitcoin and Ethereum taken from the HostCo hot wallets.”

I’m silent on the phone. I’m thinking only, “Is this the fucking apocalypse?!?”

It didn’t seem possible. The hack two days prior didn’t seem possible, and this now was just immensely confusing and depressing. I tell Greg to take the site down again and I’ll call him back in 30 minutes. How the hell are we going to explain this to the community, to our customers… to our investors? How do we even explain it to ourselves?

I get out of bed, not panicked, but just feeling utterly defeated. I take the worst shower of my life. Anger surrounds me… we knew Bob was involved from the hacker’s email, and we knew Bob committed a Class 3 felony against us, which the authorities knew about three weeks ago, and our private investigator had provided all the information needed for an immediate conviction. And now this happens.

As I gather my thoughts, I decide it’s time to call in some professional resources.

Michael Perklin, Head of Security and Investigative Services at Ledger Labs, and chairman of the Steering Committee for the Board of CCSS, is first on my list. He’s in Toronto, and agrees to fly out to meet us that evening. He was on his way to the hospital; he had a toe broken in an event he’d prefer not to discuss. He changes course and heads to the airport. What a champion.

I also chat further with heads of several leading exchanges. None of them like thieves, and are eager to help. Despite its hectic pace and diversity of opinions and interests, this industry comes together when it needs to.

1500 ETH recovered, and exchanges are hunting for more. The thief is probably upset by this… it sucks to be stolen from, after all.

Fireside Chats with the Thief

In parallel to all that, I hear again from the thief via email. I had responded to his “One word: Bob” message by asking if he would provide more info. He mentions that for a price, he may.

“hi” he says.

I arrange to pay him 2 BTC for information.

“I need to know what your relation to bob is” I ask. I tried to avoid pre-empting details.

He replies, “I got information that Bob “hacked” you while I was trying to hack you too. I had some access before Bob hacked you but not enough to get the coins myself.”

“What do you know about Bob hacking us?” I ask

“Inside job. 315 BTC.” he replies. “I talked to Bob after he took the coins, asked him about how I could hack it too. He gave me more information about the infrastructure and some keys.”

I ask, “Why would he give you information and what did he give you?”

Rovion responds, “Because I offered BTC. IP addresses, server roles, users, a working SSH key. Does not work anymore.”

We chat further, and he reveals Bob’s email that he communicated with: m0money@gmail.com.

While I had not seen that email before, it seemed familiar. I thought for a while, and then realized that Bob often substituted 0’s for o’s, including on one of the two keys which he had deleted from the server (the specific key was named something which, if displayed, would give away Bob’s real name). That, and the fact that one of Bob’s common password variations was “m0m0ney.” Our security guy used l33tspeak for his passwords. Real secure.

As clear as it had been that Bob had stolen our funds a few weeks prior, it was now clear that this hacker, Rovion, was giving us information related to Bob that only Bob or those with whom he had actually interacted would know.

Another thought, could this hacker have actually framed Bob from the beginning? Sure, perhaps, but every action of Bob’s back on March 14th points away from that explanation, specifically Bob deleting his own keys right under our nose and then leaving the office, never to return. Other evidence not listed here further counters that theory.

Back to the chat with Rovion… I ask which “working SSH key” he had obtained. “None of your business,” he responds, “but he told me he got it from a coworker’s open laptop.”

Wow. If true, that means Bob, while working at ShapeShift, accessed a coworkers computer and copied a key (or more?), at some point before he stole the funds. Did he premeditate the whole thing, I wonder?

I try to get more information, but Rovion is unforthcoming. His last message…

“Your millions will save you, Erik Voorhees. Goodbye, I will be on email.”

By the early evening, our forensic investigator, Michael Perklin, had arrived. I picked him up from the airport. We had decided to hold off on poking around in our servers until he was there. While the hacker gave a vague sense of how he came upon secret information, we didn’t really know the specifics of the breach. Keys had been changed after Bob’s departure, and while we found one key we hadn’t remembered to change, it only had access to a server that could not have stolen the funds on the preceding Thursday. And again, it wouldn’t at all explain how the Saturday morning theft occurred. Both CloudCo and HostCo had funds stolen off them, despite them being built as entirely new environments with wholly new keys.

Michael asked me to convey to him the whole story of the past month. He proceeded through his investigative protocol, which included the assumption that nobody at the company was trustworthy. It was hard to argue that the team was trustworthy, given the fact that this all started with a rogue employee. It was a depressing feeling.

Many interesting details could be added here about how such forensic work is done, but space is limited and it’s probably unwise to reveal every such method. After a while, we dove into the logs themselves, attacking the Saturday logs first. They were deleted, most of them. How were they deleted? We weren’t sure.

We know now how to prevent that… indeed, the experience we’ve received throughout this incident has been immensely valuable. Though it sounds cliché, if your startup is involved in securing information or servers whatsoever, do yourself a favor and bring in 3rd party professional help very early. We hadn’t needed it at first, because we were small. But growth creeps up on you, and before you know it you are securing significant assets with sub-standard methods.

While much of the logs were gone, we in fact recovered a great portion of them off the “empty” disk space itself using forensic techniques. This was just lucky. Perhaps the Ghost of Satoshi was looking out for us (could have used his help a week ago, of course!)

From the recovered data, we discovered the malware, if that’s the right term. There was a program, written in Go, installed on a crucial server which communicated with coins. This program had its dates changed to appear consistent with the setup of the server, and its filename made to look innocuous. But it was the direct tool by which funds were stolen.

udevd-bridge it was called

We were glad to find it (and yes, the same thing appeared in both server environments, CloudCo and HostCo). However, it still didn’t explain how it was put there. We had a lot of information, but not the whole story.

And we wouldn’t have the whole story for a couple more days. But then the stars aligned.

Out of the blue, the hacker, Rovion, emails me again on Wednesday, April 13th.

From: Rovion Vavilov rovion.vavilov@fastmail.com

To: Erik Voorhees erik@shapeshift.io Subject:

Re: ShapeShift

“Would you be interested in buying the ETH that I currently hold back at a highly discounted rate in exchange for BTC? I’d be willing to trade in small quantities since you have no reason to trust me.”

Yes, it appears the hacker has gotten annoyed that his Ethereum kept getting frozen at exchanges. So he comes back to the store he robbed from, and asks us if we’ll trade for a more liquid asset. We’d be essentially buying back our own Ethereum, and paying him Bitcoin.

Obviously worth it, if we can obtain more information. Since neither of us trust the other, we establish a protocol:

1) We pay 2 BTC to get the conversation started

2) Rovion gives us half the relevant information

3) We exchange, in increments of 250, 2000 ETH for BTC at 0.02 BTC/ETH rate

4) Rovion gives us second half of the relevant information

5) We exchange, in the same increments, the remaining 2500 ETH for BTC at same rate

6) We cease communication (this last one was Rovion’s suggestion)

He asks us to send the BTC to his already known BTC address: 14Kt9i5MdQCKvjX6HS2hEevVgbPhK13SKD

After the initial 2 BTC payment, Rovion begins with description of April 7th hack:

“We contacted Bob. He gave us the ShapeShift core source code, core server IP address, an SSH key, and [redacted]. I logged in to the core server with the SSH key provided, installed a backdoor and took the coins since the core server had SSH access to the coins server.”

“What’s the fingerprint of the SSH key mentioned above?” I ask

“9c:3f:4b:ad:d6:43:ec:9a:55:de:b9:0b:d8:f5:0a:cb”

We see that it’s Greg’s key, newly created for the CloudCo environment. It was not even in existence until more than a week after Bob had stolen the funds in March and disappeared. How on Earth did this hacker get a new key, post Bob?

I also ask about the “[redacted]” mentioned but Rovion says that is part of the second batch of information. We proceed with the incremental exchange of the second batch of funds.

Then Rovion says,

“[redacted] was access to an RDP installed on a coworker’s machine by Bob. That’s how I hacked you the second time.”

Wow, now it’s starting to come together, each revelation peeling back a layer of Bob’s treachery. Bob had installed an RDP (remote desktop protocol – basically a screen viewer or controller) on Greg’s computer. And perhaps on others, we must assume.

Then Rovion shares via pastebin an email from Bob (the info he purchased):

“hi,

i received your 50 bitcoin. gh source and ssh priv key as attachments”

core ip: XX.XX.XX.XX

router for forwarding: XX.XX.XX.XX:XXXX

admin:[redacted password]

rdp internal ip: XX.XX.XX.XX

acadmin:pass

thanks for your business.

[2 attachments listed]

(specific IP’s redacted by us)

And there it is. Bob sold information on the production servers, access to ShapeShift’s internal network, part of ShapeShift’s source code, and access to an RDP client he had installed on a co-worker’s computer, to Rovion, for 50 Bitcoin. The IP and internal router info checked out.

This explained almost everything. With access to Greg’s computer (and perhaps others), via RDP, the new server environments could be witnessed and the new SSH keys could be used. It wasn’t the cloud service provider’s fault, it was our own.

We had changed almost everything, but hadn’t scrapped our personal computers used while Bob had been part of the team. Would that have been the paranoid thing to do? Yes. Would it have been the right thing to do?

Clearly.

And one of the last things Rovion said before we ended the discussion,

“Even though I said cease communication, can you still send me an email when Bob gets sued/whatever it is you’re going to do? I feel it’s really shitty to steal from your own employer.”

Cleaning Up a Mess

We imagine this information will assist in demonstrating criminal intent on the part of Bob. This was not a spur-of-the-moment taking, but an orchestrated treachery. I’ve lost count of the number of felonies involved at this point.

We also know that while the story from Rovion checks out, it may well not be the full story. We have to assume other details are relevant to the case, and to our infrastructure. This is why ShapeShift has been offline for longer than any of us would have liked. We are being very careful, and very paranoid.

Nonetheless, I have been immensely proud of my team. Working in a startup, in the Bitcoin industry, is stressful enough, and then to deal with a series of layered betrayals like this and all the damage (financially, technically, psychologically) it causes… that is hard. You guys have done an amazing job and I am immensely encouraged seeing the team’s cohesion and fortitude.

It didn’t help that we had just brought on four new employees in the very week of the two incidents (nearly doubling our development staff). They were thrown into the fray without mercy, and they’ve been incredible.

#ShapeShiftUserNotAffected

To survive in Bitcoin, one has to be an optimist. While the betrayal and loss and clean up effort has been horribly taxing, there are some silver linings.

First, no person or organization is perfect. We learned some of our own vulnerabilities, and our own mistakes. We are correcting them, and improving upon them wherever possible. Such improvement doesn’t come cheap, but the ShapeShift of today is made better than the ShapeShift of yesterday. The steel is tempered, the machine refined. Though no single organization can ultimately achieve it, we try to approach anti-fragility, and exemplify it as an ideal in our work.

Second, no customers lost money throughout multiple hacks orchestrated even by an insider. Through decentralization, through code, through innovation, through structure… consumer protection by design is one of this industry’s most important contributions to society – something that a century of legacy banking has failed to achieve, as noted by Satoshi’s infamous line in the Genesis Block.

ShapeShift will always work to develop upon this platform of consumer protection. Many others in this community are doing the same along different avenues. Thank you for the tools you are building, and the work you have done. And indeed, there is still much to do.

To our customers, I would like to personally apologize for our downtime. While we can ensure your funds are not at risk, I know many rely on our service, and it has been unavailable. Redundancy, even in the face of disaster, will be one of our primary development goals going forward.

Further, thank you sincerely to those in the community who reached out and offered all manner of support, and to our investors who were immensely kind and understanding.

And finally, as with all intense episodes one endures, we must appreciate the room and opportunity for growth, for experience, and for one of life’s most precious luxuries, reflection.

Never a dull day in Bitcoinland

-Erik Voorhees

CEO ShapeShift.io

And to Bob… Note that your real name and identifying information were not divulged. Consider that a final, tenuous courtesy.


Images courtesy of ShapeShift.

  • Danielle Wall

    All I have to say is fuck Bob

  • Well written Erik, great job on the investigation. If that isnt going to compel Bob to cough up, no civil suit will. Great job and good luck.

  • Vladimir Marchenko

    This sux. It is very difficult to defend against insider attack, even more difficult against your own security guy.

    However, your description of events suggests that your infrastructure was in really bad shape security wise. Not using best practices, using cloud providers, using windows machines to store private keys and it seems in plain text form. Using server OS which is not really suitable for secure servers, no information security awareness among staff and management, leaving open laptops around with a possibility to steal keys from them etc…

    It should have been obvious early on that Bob, regardless of the talk, does not walk the walk. Or perhaps you, Erik, yourself, in your understandable “startapiness” and MVP eagerness chose to pay only lip service to information security.

    Should have not fallen for pseudosecurity technobuble talk. Should have gotten second opinions. Should have used established information security frameworks and be extremely suspicious on why they are not being used, should have made your security person to explain why some things are being done and some not, how risks are assessed and mitigated etc, and be extremely suspicious if your security guy is being secretive and does not act as an educator first and foremost.

    It is a common misconception that to defend against hacking you must hire a hacker. Perhaps you’ve been watching that old movie “hacker” and lots of other Hollywood produce too much and believed that crap. In reality, to defend you must hire good people who do information security professionally, not those who go about how great hackers they are.

    If you do not change YOUR ways, this horrible story is bound to repeat itself.

    In any case, I wish you the best of luck, I can imagine how hard last few weeks were for you and your team. Stay strong.

    P.S. And please, Erik, forgive me if I made any unfair assertions above. It is just what I immediately thought after reading your article.

    • erikvoorhees

      PWs were not stored in plaintext, and windows machines were not used. Computers were all encrypted. It’s easy to list of “should have done’s” in hindsight… when you’re a startup looking at a hundred grand in expenses for such expertise, the right decision isn’t necessarily clear. There were some low hanging fruit that we didn’t pick, and hopefully our story can help others with those.

      • Vladimir Marchenko

        Of course you are right. Though, all those things could have been as easily said in foresight. Not sure about “PW’s”. But with regard to “all encrypted and windows machines not being used” your text has mentioned RDS (a windows specific protocol which is rarely used on other operating systems) and lifting ssh keys (have public and private parts). This is consistent with windows computers and not encrypted private keys. If I made a wrong conclusion based just on that, I might have been mistaken.

        Surely I hope this story will help others. The others in bitcoin space (and elsewhere) should pay close attention to information security.

        • Donald

          Erik Voorhees shouldn’t be trusted. He’s always involved with rubbish cryptocoins. ShapeShift is stupid. I’m happy to see him look like a fool today!

          • gphx

            Shut up Bob. Bob this.

    • Itsko

      Such a strong and right words, every “Bitcoin” company should adopt military architecture and habits, and not less than that…

  • Homero Garza

    Clowns, you used keys without a passphrase? Shitcoiners I guess

    • erikvoorhees

      The SSH key is the passphrase… a passphrase on top would be good 2-factor, and yes we should have done that.

      • LaLu

        Erik, I would recommend your team to use 2FA on ssh connections (works with yubikey or google authenticator). I know some other business in the industry are using this too.

      • You said keys were encrypted in a previous comment, in your reply to Vladimir.

      • Homero Garza

        Ssh keys usually make you use a password to unlock the key

      • Kampenauto
      • Jonathan Wilkins

        An ssh key is many things, but a passphrase isn’t one of them. It provides better security than a passphrase (given proper generation) as it doesn’t get transmitted, but is used to sign a message and the server verifies the signature instead of seeing a secret directly. You can use the newer private key file format to ensure that grinding is significantly harder.

        eg
        ssh-keygen -t ed25519 -a 2000 -o -f ~/.ssh/new_ed25519_key

        -o tells keygen to use the new file format
        -a specifies the number of rounds the KDF will use

        Trezor or the new Ledger Blue are better in many ways. Being able to confirm use of the key is a major step up. Having it completely off device is good in either case but the trezor protects against fewer hardware level attacks than the blue will.

  • EvilDave

    This sucks mightily indeed. The betrayal of trust is just staggering, but hopefully ShapeShift can set a precedent and get Bob busted. Kudos to Erik and the SS team, btw. A little bit of professionalism goes a long way in crypto.

  • why bother

    It’s another reason why your machines should have a trusted platform module and LUKS encryption of your root partition, it can be done. It is not a cure all but helps a lot in preventing malware.

    • And your phone should have the Clipper chip, it can be done.

      Don’t forget to update the antivirus. To prevent malware.

      • why bother

        This is a misconception of the TPM. All based on superstition and no proof. Let’s assume there is a key in a backdoor. Well so what? You can stil determine if malware infects your machine. If you are important enough that the consortium of manufacturers will get your data then okay. But for most of us it will require too much of a cost compared with the info they would get

  • be_free

    Waow, it reads like a crime novel. I’m sorry what you had to go through, and welcome back!

  • Marco Maltese

    Cryptos are like the far west: a frontier land, full of thieves and pioneers. Trust nobody, and I mean NOBODY, bind the unlock with another key that only YOU have so that funds movement are only authorized when YOU unlock it.
    When playing with hundreds of thousands of dollars there’s no other way.

  • Barbierir

    This story and other similar hackings in crypto would made a great book. Well done the way this one was handled.

  • jason wright

    Did Bob need physical access to your laptops to pull this off?

  • Simon

    I guess next recruits will have more in depth interviews …

    • James M. Ray

      Or background checks…

  • Patrick Doan

    Erik,
    While I get that you are trying to take the high road and not divulge his information, you are in some ways doing a dis-service to all potential employers who are going to google his name. The reality is I do not think he is going to return the Bitcoins and as a means of protecting the community as a whole, you should think about releasing his information. This was not a mistake, or a lapse in judgement, this was a planned, conscious effort and all he has to do is strip his time with you from his resume and he is free to steal again.

    If it was me and my company, I can’t say that I would refrain from posting his name everywhere to try to protect other people in the industry. Can you divulge on why you decided to protect someone who essentially stole hundreds of thousands of dollars from you?

    • Eric LW

      wait for the indictment

    • Martin Samuel

      He’s not posting the name because all charges are pending. If this is all there is to it, no charges will ever be filed. It sucks that he lost the money, but you really need to read this article more critically. Had he mentioned Bob by name, he would be opening himself up to three or more civil counts, and at least one criminal count if pursued in court. Two of them can be litigated anyway, now that he’s admitted to it publicly. He’s wise not to say anything more than he has. If you’re willing to post names about your employees on grounds this shaky, I (and most reasonable people) would have serious reservations about both your and your company, Patrick.

  • Dan

    Great writeup Erik. I’m thinking about ways for a high volume business like yours to mitigate the risk, even against insiders. It’s a tricky problem because automated sending of funds is a business requirement. A hot wallet of some type is therefore necessary, so the question is how to limit the funds in that hot-wallet at any moment while also keeping it liquid enough for customers.

    Many people will say multi-sig for all outgoing hot-wallet tx, perhaps with an independent 3rd party such
    as bitgo. That’s a good start, but may not be sufficient because:

    1) The 3rd party will likely rely on information from your system to know whether it should sign the tx or not. If your system is compromised by an insider (or clever attacker) who knows this, then they can compromise your internal DB or oracle to lie to the 3rd party.

    2) if the 3rd party instead relies on heuristics such as “do not sign if tx is over X amount” then the attacker can simply split up into many smaller tx or otherwise construct tx that seem OK.

    3) If the 3rd party goes down for some reason or cannot keep up with your volume, then your platform may also become unavailable for a period. Though this could be mitigated with an extra “emergency use only” signing key.

    Using Multi-sig without a 3rd party has the same issues. Plus the extra signing key is more likely to be compromised than if stored behind a 3rd party’s separate security precautions.

    So, multi-sig is better than nothing for a hot-wallet, but not a cure-all.

    How can these risks be mitigated?

    You may benefit from the addition of a warm wallet that is hosted on a seperate network, locked down to a small set of users/fiduciaries responsible for the funds, and all it does is monitor the blockchain itself to determine when a hot-wallet is low on funds and then re-fill it, within acceptable amount per minute bands and subject to manual multi-sig approval. So it is automated in the sense that it determines when a refill tx is needed for the hot-wallet and it prepares it, but then a message pops up on the 2+ fiduciary’s screens and they must manually sign/authorize the tx. In this way, the human agent(s) may perform any needed checks to verify all funds are accounted for before authorizing refill tx. The warm wallet itself would be re-filled periodically from cold storage, ideally by separate personnel.

  • Joshua Davis

    thank you for this

  • Only a doubt. What would it be the RDP client in Greg notebook? Like a TeamViewer? Machine dont work with slow speed, it is not perceptible? Saludos from Latinoamerica.

    • Martin Samuel

      It’s there so that Rovio can read the emails going back and forth. Really amazed that he just glosses over this detail. Draw whatever conclusions you like.

  • If its any consolation Erik, PayPal lost a large sum of money shortly after raising capital. It was a simple exploit enabled by a few lax procedures. It was also a big lesson. After that, a lot of internal controls were developed to limit, if not eliminate, the scale of any one breach. Good luck with business.

    • James M. Ray

      The bank Paypal acquired before they went to credit card processing instead of banking, X.com, was saved from a very large theft by tiny e-gold, which at the time had not even split up into e-gold & OmniPay yet. Everyone kept it quiet at the time, but it was shocking. And that wasn’t the only one, it was just a big one.

  • Peter Herbst

    Whow, what a story, tx for the Transparency in this !! Well done from the managing Shapeshift side !!
    sorry you had to go through with this , should be a great movie one day ; )

  • dantavares

    In bitcoin business, be paranoic is a prerequisite.

    • Adella Toulon-Foerster

      It’s not paranoia if they really are out to get you. 🙂

  • toxip

    woah, what a story! This whole incident must’ve been a great learning experience. At least you did something right by designing a system where stealing from the customers is (almost) impossible. Wish all the best to you guys. May you be tempered by this experience and build ever more secure systems!

  • Adella Toulon-Foerster

    Wow. This just read like some kind of movie script.

    How can you trust *anyone* after this?

  • Itsko

    That was fucking interesting! Thank you very much Erik for the sincerity.
    We are all going to learn a lot from that.
    And for all of you guys, I am developing some tool that may help for this kind of Incidents.
    Check Riders – The Bitcoin ranking engine : https://riders.io/

  • TysonandKelly Wall

    Something as simple as a IP white list in your firewall would have stopped this. 2fa (Not google or androis, IOS based app junk. cell phones are extremely insecure) is recommended no doubt, and also real time network monitoring and alerts.

  • SH

    I like Rovion’s charisma. I feel like he got into cryptocurrency theft to make friends.

  • Swapster_com

    You should still negotiate to get back the stolen Bitcoin.

  • DoesntMatter

    Dig deeper. How did you this Bob guy got into your team?

    Wasn’t he possibly a planted saboteur? I’m shocked by his persistence even after getting caught.

  • Dave Dann

    I’m amazed that monkies are allowed to manage so much.

    • Martin Samuel

      My thinking exactly.

    • Bitcoins and Gravy

      I’m amazed that people hire monkeys.
      : )

  • Martin Samuel

    First and foremost, if Bob had an out of state arrest record… why didn’t Erik know about it? That’s pretty breathtaking, considering that he’s hiring someone to deal directly with money. And it speaks to Erik’s complete lack of expertise as a CEO. This is not Bill Gates we’re dealing with, obviously.

    Bob, being an administrator, did the right thing by deleting his keys after the hack. That’s not the least bit suspicious, because all SSH keys can and should be treated as if they are compromised, and reissued. Bob should have done this for everyone, but was probably just not thinking. In either case, not thinking is not betrayal, and shouldn’t be treated that way. Everyone else should have done the same thing. It’s fucking insane that Erik thinks this is cause for alarm after losing $130k, and it speaks to the rest of his absurd mistreatment of the situation.

    Erik wants to finger Bob for betrayal from the first day. Bob, reasonably, freaks the fuck out, knowing full well there’s nothing he can say in his defense, and disappears. Not necessarily evidence of guilt, but it is evidence that Erik is willing to believe without evidence, as we continue to see later.

    Rovion really wants to finger Bob, and goes out of his way to do so. Over and over again, the information he provides looks too perfect, and nobody questions why. So, why? Do you think for a minute that this guy is really interested in a couple of 2btc transactions?

    Come on now. It doesn’t add up. Think about it.

    If I was capable of doing something like this, I would go out of my way to finger the guy that they already think did it, if not me. If Rovion is the culprit, then Rovion had a rat installed on multiple systems in the company. He knows what’s going on internally, and he knows what they’ve been saying back and forth. That would mean that he knows Bob is the prime suspect, which works for him because he knows nobody will come looking for him. Duh.

    Chances are that Rovion is someone on the inside, and that he still works for the company. Probably not Erik, but someone with access to everything. Maybe someone who was involved early on, before Bob got there. Otherwise, there would be no need for a RAT.

    Go is an uncommon language for a RAT. A good, lightweight choice for performance, certainly, but not the conventional fare. Go is also a language with a high learning curve, and very brittle syntax. I know this, because I work in Go. Does Bob know enough about Golang to write a RAT? Normally, you would expect a RAT to be written in Java or C++. What do we know about Bob’s skillset? If he does not advertise that he’s a Golang expert, and he wasn’t hired to work in Go, the chances that he’s the culprit are slim to none. Even installing it well enough to compile third party code is tricky, with this kind of thing; because so many Go third party go sources are obsolete, and need to be rewired in order to work. You simply can’t know that unless you’re an expert. Go is probably not the choice of platform for the bigger app. Especially after all the issues we’ve seen with the Golang implementation of Bitcoin protocol. It also falls into the category of what I would consider “yet to be mainstream” or even marginal or fringe languages. Other Go devs may disagree with me on that.

    The whole case against Bob is based on wild conjecture and hearsay. It would never hold up in court if Bob has an even half way competent lawyer. Erik is wise not to finger Bob by name, because if he did, and this pans out the way I think it might, Erik would have a libel suit on his hands. If you look at some of the other things Erik admits to doing here, you might also have a strong case for violation of privacy, HIPPA, breach of employment agreements, trespassing, and a few others. Even without fingering Bob directly. I would tread with caution.

    If I were an investor in this company, I would be lobbying the board for a new CEO after reading this article. Seriously. The buck has to stop somewhere. This is his account of gross negligence and mishandling of a bad situation. Not mine.

    But it’s like I keep saying. The big problem with Bitcoin world is that people who can’t handle pressure, jump to conclusions, and completely lack professionalism, not to mention technical skill all seem to think that Bitcoin is a space where they can do business and get recognized. And crypto-currencies are going nowhere, and I mean absolutely nowhere, as long as idiots like Erik Voorhees are running these companies.

    Knowing the way Bob was treated, and how eager he was to jump to conclusions without even bothering to know the facts… I wouldn’t consider working for these guys under any circumstances.

    Just my two cents.

    • Norman Khine

      +1 – Bob is just a patsy

      • Bitcoins and Gravy

        Don’t the Samuels and the Khines get together every year at that little greasy spoon on the East side?

        • Martin Samuel

          Whatever Erik.

      • Martin Samuel

        It’s the only explanation that makes any sense.

    • Bitcoins and Gravy

      Martin Samuel no offense but your post reads like something written by Bob himself – or by someone very close to him. It sounds to me like you have something against Erik. I think most people in the bitcoin world who know Erik can vouch for his good reputation and for his integrity. On the other hand, no one knows who you are. Put that in your pipe and smoke it – along with whatever you’re smoking over there.
      : )

      • Martin Samuel

        Wow, that’s impressive, B&G. I’m going to be the better man, and ignore than unjustified smear on my character. One thing I would point out, B&G, is that unlike you, I’m willing to post with my real name. Nobody knows who you are. I don’t have anything against Erik, other than the fact that he’s advertising that he’s completely incompetent, and it’s a level of complete incompetence that’s truly frightening. The fact that he’s willing to jump to those kinds of conclusions without knowing the facts is proof by itself that he’s not qualified to be doing the job he’s doing. Just read what he said about this. If I was an investor in this company, and he posted this, I would be gunning for his head. He’s the one jumping to conclusions, not me. This is a story of his gross mishandling of a bad situation. I’m not involved. I don’t know Bob. You should read these things more critically, brother.

      • Martin Samuel

        If you’re leaving comments like this, you’re either not capable, or not interested in a serious discussion about the topic. But I will tell you, that you should probably be at least the slightest bit careful about believing what hackers with RATs on your system tell you, about who did what. To take the word of someone who can read your email at face value… you’re either an idiot, or you’re a fool. Either way, it’s nothing good. If you yourself would have treated the situation the same way, I would challenge your professional qualifications too.

  • Bo joe

    You’re gonna have a hell of a time convicting him.

  • Bitcoins and Gravy

    This was a great read! Please someone put this in a bitcoin novel and publish it!
    I will say however that you guys at Shapeshift are pretty naive.

    When hiring new employees it is VERY standard to do an extensive background search – including criminal. You can hire a licensed PI,(private investigator), or any number of services that will go deeper in depth into that person’s life than you could possibly imagine. This should cost between $100 and $500. Landlords and property managers do this all the time when checking out prospective tenants. They take it seriously because they don’t want to lease to a crook or a con. Very, very common, very, very easy and very, very naive of you guys for not coughing up the dough to do this right. Again – this is a very standard practice.

    I still think it’s funny and sad how bitcoin businesses somehow think they can live and function in some sort of bitcoin vacuum . . . as if their problems have NOT already been addressed by tens of thousands of companies for decades now OR as if their problems don’t overlap with “the real world”.

    Truth is EVERY bitcoin company should hire a communications expert to keep communication among employees open and honest and above board. But guess what? None of them will because for the most part geeks have really, really poor communication skills.

    I spent my life kicking around the streets of neighborhoods and cities and jobs and bars and you name it! And guess what I learned? I learned to have a keen understanding of human nature and the American mind. I also learned to speak and write well. Yes a communications expert is JUST AS important as your dev and systems geeks.

    Guys just because you are doing something unique does not mean you can ignore standard business and security practices.
    I hope you guys learned from this and may the force (or the schwartz) be with you!

    John Barrett

  • Why did not use EMCSSL and EMCSSH for your server and login? You will be much safe!

  • George Freeman

    Basically (bestethicalhackers@gmail.com)he just helps you out with whatever hacking or spying activity
    Stay classified stay certified, call (302) 365-0294
    Thank me later..

  • Ade

    I don’t normally read long articles, I read this one, it reads a lot like my life to date, I’m sorry to have to report, there are lots of Bobs out there.
    I’ve often wondered if in a business like yours or an exchange whether a tiny percentage, let’s say 0.1% or even lower could be set aside in Escrow or Paper wallet address, perhaps held in a Bank vault (not that I trust bankers any more than Bob ) hopefully, if the business has not suffered a catastrophic loss immediately, 6 months to a year later, those 0.1% transaction fees have accumulated to the point where any customer funds lost in an incident can be refunded. If there’s been no breach after two yrs, or so, and the funds far exceed any potential loss, it might even be decided that 10% of the accumulated funds could be given to staff as a bonus, charity, or to customers (who after all have been paying the extra 0.1% as a fee). Anyway, keep soldiering on, Shapeshift is a Brilliant piece of Software engineering.

  • Crystal

    I am gladly referring and recommending ( your-safety@gmx.us ) for a all clean good job. I have used this Ethical Hacker & Pentester Pro for my company 2 Times. He is relisble and swift. You will come back to thank me later…

    N: B

    Please be watchful of who you contact here… so many fake people up here lol.. some self proclaimed hackers here don’t even spell check lol .. Choose Wisely!!

    CRYSTAL.