Namecheap DNS Hijacking

A few days ago, a former client of mine contacted me and informed me that a user of their website had reported seeing pornographic (!!) advertisements and content while visiting the site. Immediately, I suspected that the site might have been compromised somehow, and gingerly visited it in my browser. Everything looked perfectly fine. I checked a few pages, logged in, even checked the security-related logs on the server for any suspicious activity. Nothing. I put it down to a malware problem on the user’s computer and told my client to advise the user to check their computer for malicious software.

I thought nothing more of it.

A few hours later, I received another message from the same client, saying they were seeing odd error pages when visiting the site. “500 Internal Server Error” …Something, something… “nginx”. The complaint was backed up by a screenshot this time, and apparently they’d heard that a couple of their users were seeing the same thing.

“Wait a minute…“, I thought. “nginx? That site doesn’t use nginx. It’s not even installed on the box.”

I started paying attention right about now.

DNS Digging…

I couldn’t replicate the problem locally at all, but clearly for some users the domain was somehow resolving to the wrong IP address.

I used dig to query some trusted DNS servers (OpenDNS - 208.67.222.222, Google - 8.8.8.8). Most answers contained the correct IP address, but a couple did not. They contained an unknown IP address that indeed served up a “500 Internal Server Error” nginx error message. These records had very long (7-day +) TTLs. Worrying. Looking at the IP address on VirusTotal showed that it has a history of malicious use.

Now, this particular domain was originally registered by the client at their own choice of registrar, and the nameservers were set to Namecheap’s FreeDNS service. I became suspicious that my Namecheap account might have been compromised and malicious records added, so I logged in and checked the host records in Namecheap’s DNS record editor UI. Everything looked fine. There was one record for the origin (@) host pointing to the correct IP address.

At a loss, I started a live support session with Namecheap. In the past, their live support people have been really responsive and helpful, so I was optimistic that they’d have some answers for me. Their response was disappointingly less than informative:

Unfortunately, the inconvenience is caused by the issue that occurred with our FreeDNS service. Kindly re-point the domain to our FreeDNS nameservers to have the issue resolved.

I pressed for more details, but their live support attendant was cagey at best. They did promise that there would be an investigation and I look forward to reading that if/when it is made available.

It looks like some 3rd party had managed to insert malicious records at Namecheap’s DNS servers. Quite how widespread this was remains to be seen, but a number of tweets were published that point to this being more than an incident isolated to my account/domain:

@jelveh still having problems? we experienced what appeared to be a dns hijacking this morning using namecheaps freeDNS service

— Rob Collier (@robcollier) July 28, 2016

Conclusion

My immediate reaction to this incident was to move the DNS hosting away from Namecheap. While they’re a great registrar, this made me wonder about their competency as a DNS provider. I opted for Amazon’s Route 53 instead.

The caching nature of DNS makes this kind of incident particularly troublesome. It’s paramount that you trust your DNS provider - any malicious modifications can have consequences that stick around for a long time and really hurt your users. I’m still waiting for the TTLs to expire on those malicious records so that they’ll disappear completely out of DNS caches. Thankfully the malicious IP address the record pointed to seems to be dead for now.

N.B. Shortly after writing this post, I found an excellent report by Denis Sinegubko. Definitely worth a read if you were affected by this incident.