So what happened? Friday, October 21st, 2016 began with many across the U.S. being unable to access a large number of popular web sites including Twitter, PayPal, CNN, Reddit, Netflix, Github, Iheart Radio, Pinterest, Spotify, Wired, and Yelp. This outage was caused by a massive distributed denial of service attack (DDoS) that targeted the authoritative name servers for the impacted domains. These name services were being provided by Dyn, a DNS service provider. The attack started about 7:10 AM EDT and initially impacted the east coast of the U.S. Later, waves of attacks moved strategically across the U.S. and E.U. The attacks continued to impact site accessibility well into the evening on Friday. You can see the progression of the attack here from Gizmodo.
Analysis of the attack indicates the perpetrators utilized newly release attack vector software called Mirai. This code enables an attacker to take advantage of several flaws in the firmware and software installed on Internet of Things (IoT) devices like cameras, DVRs, thermostats, etc. Commandeering a single device is not really of any benefit aside from gaining a foothold in the network where that device resides. Commandeering hundreds of thousands of these these IoT devices and instructing them to act in concert (aka a DDoS attack) is a different ball game. That is exactly what happened in Friday’s attacks.
Malicious attackers coordinated a massive collective of these vulnerable devices and programmed them to flood their intended target with massive numbers of DNS requests amplified through open DNS cache resolvers located around the Internet. The DNS requests are formatted so that the IP address of the requester is spoofed, targeting in this case the IP addresses of the Dyn DNS servers. This causes the DNS cache servers to send all responses to the spoofed source IP. This combined with using DNS amplification tricks, effectively makes the target IP addresses unresponsive to DNS requests and unreachable due to massive bandwidth utilization. These sorts of attacks can generate bandwidth in the 500-1000 Gbps range. This level of traffic can overwhelm even the Tier 1 network providers like Level3, Sprint, Verizon and Century Link. The result is a cascading impact across the Internet.
These attacks are successful because manufacturers of many IoT devices fail to properly secure the devices they sell or they fail to provide the means for end users to effectively secure these devices. One manufacturer, Hangzhou Xiongmai Technology, which produces DVRs and internet-connected cameras, announced on Sunday that many of their products were exploited to participate in the attack. They have since instituted a recall for the impacted devices.
This is the third large attack we have seen that used the Mirai malware. The first attack was more fine tuned to target the Krebs on Security website and occurred on September 21, 2016.
Give the massive number of IoT devices out there, what can we do to insure that we are not the target of these attacks as well as make sure we are not unknowingly participating in these types of attacks? Let’s look at some simple steps you can take to lessen the likelihood of these DDoS attacks impacting your business.
Preventing Your Network From Participating in DDoS
- Configure simple outbound firewall rules. Since these attacks rely on spoofing source IP addresses in outbound packets, your firewall should be set to explicitly drop and log any outbound packet the has a source IP not found within your internal network. Set another rule that limits outbound UDP port 53 (DNS) traffic to only approved DNS server IP addresses. Ideally, your network has a pair of DNS cache servers that serve DNS for your internal network. Only allow DNS requests from your internal network to hit these servers. Drop and log all other Internet bound DNS traffic.
- Configure your internal DNS cache servers to log all DNS queries and report these logs to some sort of log aggregation like Graylog. Be sure to alert on anomalous behavior. Know what is normal. If you notice large numbers of DNS ANY queries for a particular domain, you might be compromised. DNS is the key to communication so by logging and understanding your DNS traffic, you can uncover compromise and mitigate malicious activity.
- Have your DNS servers forward DNS requests to a DNS filtering service like AppRiver’s SecureSurf service. These types of services will dynamically assist in dramatically reducing the likelihood of users accessing compromised sites as well as severely limiting botnet command and control traffic.
- Remove root forwarding from your DNS cache servers if already forwarding to a filtering DNS server as in item 3 above. This prevents your DNS servers from bypassing the DNS filtering should those servers be unavailable. Your DNS now fails secure.
- Configure your firewall to allow only your DNS server IPs outbound DNS access and allow those to only access the configured forwarding servers. This will block inside DNS requests from reaching anything but approved DNS resolvers.
- DO NOT expose DNS services on your internal network to the outside world. Nobody on the outside of your LAN should be able to resolve DNS by hitting your WAN IP address.
- Implement a reliable SPAM filtering solution like AppRiver’s SecureTide to greatly reduce the likelihood that users will be exposed to malicious email content designed to infect and ultimately gain covert access to your network resources.
- Inventory all devices on your network. Use some sort of network scanning tool to gather and analyze MAC addresses on the network. Be sure you know what each device is and make sure each device is properly configured, patched and secured. This means changing all default passwords and accounts.
- Use a network scanner such as nmap or masscan to scan you internal IP range for make sure there are not unexpected TCP or UDP ports open and listening for connections on network. Pay particular attention to TCP ports 23 (Telnet) and 22 (SSH). Telnet should be disabled everywhere! It is not secure and credentials are passed in the clear. If you can, allow SSH to devices only from limited internal IP addresses.
Lowering the Likelihood of Being a DDoS Target
Ultimately, if a malicious actor wants to deliberately target your organization, preventing them from doing so will be an uphill climb. However, if you do nothing, there is nothing that will protect you from a DDoS attack when needed. Bottom line, don’t be the low hanging fruit. Most of the mitigation strategies involve proper configuration of your network edge routing gear, such as these:
- Configure router firewall rules on the WAN connection to drop any IP address that is listed in RFC 1918. Be careful if you have IPSEC or DMVPN tunnels terminating on that interface as you might block LAN traffic over the VPN. Your access list might need to be honed down to allow your local LAN IP ranges.
- Make sure your router blocks IP directed broadcasts.
- Be sure IP unreachables are disabled on all WAN interfaces.
- If you can, set your router to black hole bogon IP address ranges. You can get these via BGP from Team Cmyru.
- Set set rules to block all inbound port connections directed at ports for which you have no public-facing services.
- Set firewall rules to drop all inbound traffic not destined for your WAN IP.
- Limit incoming connections from NTP servers to only those that you specifically need to access.
- Outsource public-facing services. Host those services with providers who specialize in providing those services.
- Be sure Telnet (TCP 23) is explicitly blocked and ideally disabled everywhere. Also block or limit SSH (TCP 22) from the Internet. The Mirai botnet recruits devices by scanning for open and vulnerable Telnet and SSH end points.
- Check all incoming connections particularly directed at TCP 9001, 80 and 443 as this could be indicative of botnet C&C traffic.
Employing these simple steps can dramatically reduce your exposure and alert you when things start to deviate from what is normal. Please exercise caution when implementing any security control. Consult with proper parties, document your changes and monitor for any unexpected behavior. Remember to balance deployment of security controls against availability and functionality.
Jim Nitterauer is a senior security specialist at AppRiver. Read more AppRiver blogs here.
