Communications

Communicating with clients during cyber attacks

On the morning of October 21^st, the world awoke to a deviously clever Distributed Denial of Service attack. It was to date the largest ever witnessed, with the attack load exceeding 1.2 terabits per second.

Distributed Denial of Service (“DDoS”) attacks have always been popular in the hacker underground. The idea is simple; quietly compromise a large number of computers, without letting the owner know his computer is compromised creating a botnet. Then after you’ve compromised enough systems, direct your botnet army to launch an attack.

The genius of the DDoS is that attacks arrive in such large quantity the victim’s Internet connection is overwhelmed. And because the attacks come from all over the Internet, the victim is unable to filter the attack away. They are left immobilized, often having to suffer the onslaught until the attacker finally relents.

What was particularly clever about the October DDoS incident was that the attackers choose Dyn, Inc as their target. Dyn provides DNS infrastructure services to some of the world’s largest Internet sites. By targeting Dyn, hackers were able to bring down Dyn customers. Customers such as Amazon, CNN, Comcast, Github, Netflix,  Paypal, Twitter, Verizon – and many more !

The most insidiously clever part of the October DDoS was the makeup of the botnet army was everyday devices found in people’s homes and offices. Using the recently open sourced Mirai malware, hackers had compromised hundreds of thousands of web cameras, home printers, Internet routers, DVRs and baby monitors – the so called Internet-Of-Things. The hackers used these everyday devices to perpetrate the largest DDoS attack of all time.

The attack came in three waves, with each wave separated by several hours. By spreading the attack over time, it made the perceived affect worse; with no one sure if what they were seeing was part of the attack, or part of some local issue with their ISP.  At the end of the day, the attack had spread from the East Coast, to the West Coast covering most of the major population centers of the United States.

As an IT Provider, your job just got harder. Besides the full time task of keeping your client’s networks up and operational, you now need to keep abreast of overall Internet health. This means keeping your customers informed about DDoS events during a rapidly evolving attack.

How to Recognize a DDoS Attack

For busy IT professionals a global Internet attack is not your go-to-thought when a customer calls you to report a problem.  Knowing the difference between a local problem vs. a wide scale issue is a mixture of process of elimination and seat of the pants intuition.

One concept that can help is the “Rule of three by three”.  The Rule is to always consider any problem report to be a local problem until you hear three reports from three different clients.  While there are still network elements that can be shared between different customers; such as each using the same Cable company or the same router manufacturer; after three reports from three separate clients, you should start to question whether they are local problems or perhaps what your clients are experiencing is wider then their own network.

Once your start to suspect the source might be a wide scale event, the question becomes how you can get confirmation.  With all the information on the Internet, you’d think this would be easy – but it’s not. Too many sites offer post-mortem analysis days after the attack. Getting information on current attacks can be a challenge.

Where to Turn?

There are three sites  that excel at real-time dissemination of attacks.  You should consider printing the list and tacking it to a bulletin board in your office – remember, if a DDoS attack is wide scale, it could even affect your network.  Here’ the quick list

US-CERT

Number one on the list is US-CERT. The US-CERT is the United States Computer Emergency Readiness Team and its run by the Department of Homeland Security.  The current activity page https://www.us-cert.gov/ncas/current-activity is where they track high-impact types of security incidents currently being reported to the US-CERT. It acts as a clearing house of coordinate cyber events occurring across the entire United States.

Here’s an interesting fact, if you had checked US-CERT during the October DDoS attack, you would NOT have found any information about the attack. It’s not clear why US-CERT was silent on such a record setting attack.

What is interesting however, is that US-CERT did publish a warning about the potential for DDoS attacks from Mirai a full 7-days before Mirai was used exactly in the way the US-CERT warned – https://www.us-cert.gov/ncas/alerts/TA16-288A

ABC News: Cyber Attack Reporting

ABC News is one of the few mainstream media sites that have a special area of their website to report on cyber attacks.  http://abcnews.go.com/topics/news/cyber-attack.htm.

The site often cover cyber news from a political perspective which diminishes some of it’s value, but they also cover developing stories when the Internet is under actual attacks.  If you had gone to the ABC News site during the October DDoS attack you would have found several news articles, each with useful information such as http://abcnews.go.com/US/us-monitoring-repeated-cyber-attacks-twitter-major-sites/story?id=42971910

The Hacker News

Arguably the best sites for tracking DDoS attacks as they are evolving is http://thehackernews.com.  This site provides some of the timeliest and accurate technical reporting as a cyber attacks evolves.  Their attack page breaks down attacks by type, and they even have a special section for DDoS attacks.

During the October DDoS attack, they kept users updated with information as it became available. They also hosted a user comments section on their news articles that allowed system administrators to communicate the impact as they were experiencing it. This was a valuable service and assisted the Internet community to exchange vital details with each other.

What to Tell Your Clients?

Once you’ve confirmed that a wide scale issue is affecting your clients, the next issue to consider is how you should communicate the facts of the event to your clients.  How much detail should you provide and how frequently should they be updated?

In general, you want to keep the communication brief and to the point. You want to give your users a breakdown of what’s happening and describe what services are impacted. In your communications, try to itemize impacted elements, and if available, provide a link to the highest profile source you can find that allows them to read more details.

Here’s a sample message sent to clients the morning of October 21^st that demonstrates the principles:

Sent: Friday, October 21, 2016 10:15 AM
To: All Clients
Subject: DDoS Attack in progress creating wide scale Internet outages

We wanted to make you aware of an Internet attack currently on going. The attack is not against your company, but it may affect services you are trying to access.

The attack is against Dyn. They provide DNS services to a number of other Internet sites. DNS is a critical infrastructure component, and if it is not working correctly, can cripple a website and/or a network.

This morning, hackers mounted a large scale distribute-denial-of-service attack (called DDoS) against Dyn.  The nature of DDoS attacks are quite complex to cope with. While Dyn is claiming they are getting things under control, the issue is still occurring, and many sites are still dark.

You might see problems if you try to access a company who uses Dyn as their DNS provider. Here is a partial list of some of the Dyn customers:

* Amazon
* Netflix
* Twitter
* PayPal
* Verizon
* Comcast

And a number of hosted phone service providers

The issue is now large enough to be reported in the mainstream media. CNN has an article about the current attack: http://money.cnn.com/2016/10/21/technology/ddos-attack-popular-sites/index.html

As does ABC News http://abcnews.go.com/US/us-monitoring-repeated-cyber-attacks-twitter-major-sites/story?id=42971910

As of 10:15 am Friday, the issue is still going on. We will provide more details as the situation evolves.

There are a couple of tactical issues to mention.

• You should send the message from a support address at your company. This enables a customer to hit reply and ask followup questions.
• You should BCC your customer list, so if one of them hit’s reply all, their questions of followup comments do not go to the entire list.
• We recommend you send the message as plain-text formatted (not as HTML nor Rich Text) to avoid the need for their email client to have to make Internet requests to render the information.

In the event the issue is impacting the client’s email services, you can send an abbreviated version of the message via SMS/Text Messaging:

DDoS Attack in progress creating wide scale Internet outages. Hackers are attacking Internet infrastructure provider Dyn causing black out to major Internet websites. More details  http://money.cnn.com/2016/10/21/technology/ddos-attack-popular-sites/index.html

If the event goes on for more than four hours, you should consider sending a brief followup message and provide the latest information.  This message should focus on any estimates about when the issue will be resolved along with any steps clients can take to mitigate the impact.

Sent: Friday, October 21, 2016 2:00 PM
To: All Clients
Subject: DDoS Update

The DDoS attack reported this morning is still ongoing.  The company Dyn has reported some success in controlling the issue, but a second wave of attacks started at 12pm causing another round of outages. At this time, Dyn has no estimated time of resolution.

If you try to reach a website and your browser reports a time out, try hitting refresh.  A second or third attempt to retrieve the site will likely succeed.  Also, make sure you are typing the address directly into your browser and not surfing through links you find in search engines – which will slow down the refresh request.

More details:

• http://money.cnn.com/2016/10/21/technology/ddos-attack-popular-sites/index.html
• http://abcnews.go.com/US/us-monitoring-repeated-cyber-attacks-twitter-major-sites/story?id=42971910

 
We will keep you posted.

Once the issue is resolved, a simple “all-clear” message will make sure clients know that if they are still seeing a problem, it is not related to the event you reported earlier.

Sent: Friday, October 21, 2016 2:00 PM
To: All Clients
Subject: RESOLVED: DDoS Attack

The DDoS attack reported this morning has been resolved. All Internet services and website should be restored to normal operations.  If you are continuing to see Internet issues, please give us a call so we can evaluate the situation.

Dyn has begun the process of analyzing the event.  In the coming days and weeks this will certainly be discussed in the press and Industry news outlets. If you’d like further details, just let us know, and we’ll forward on insight we uncover as the post mortem process evolves.