Contribute  :  Web Resources  :  Site Statistics  :  About  :  RSS  
    Bart's Soapbox talk about technology and politics    

FreeBSD, dummynet and a nameserver

   

*BSDA few days ago, someone tried a bandwidth exhaustion type denial of service attack against my home connection. They did this by flooding my (public) dns with requests for the nameservers for the . zone.

Background

My nameservers refuse such queries, but I still found the cpu use and upstream bandwidth use to be annoying (8mbit downstream, only 1mbit upstream, so the 'refused' messages would exhaust the upstream bandwidth) so I decided to see if I could reduce both by using a bit of filtering.

The attack lasted for approx 10 hours, flooding my nameserver with over 1 gigabye worth of requests (actually just repeating the same one as fast as the uplink on the other side allowed), comming from a single client.

Filtering out the requests

The obvious solution is to just drop all traffic from that client and be done with it (well, and inform the people responsible for the netblock and law enforcement if you feel like it).

The solution I opted for is a bit different in that it works against any such attack from any ip. The idea is to have a per client limit on dns traffic, ie, allow only so many requests/time/client and drop the rest.

The machine running the nameserver runs FreeBSD 5.3 and has ipfw and dummynet in the kernel already, if you don't have that you will have to load the kernel module for it. Add a line to /boot/loader.conf like:

dummynet_load="YES"

Be aware that ipfw will be loaded as a dependency and will default to denying all traffic, so you will also have to ensure proper ipfw rules are added on boot. If you are not very familiar with ipfw don't try this without having physical access to the console.

When you have the line in loader.conf and have ensured you can access the machine once ipfw is loaded, you can reboot the machine.

Alternatively, you can load dummynet manually by typing 'kldload dummynet'. Loading it on boot is recomended however.

After doing some measurements on my local network, I determined that for my nameserver that serves some 12 domains, I needed approx 4kbit bandwidth/client in order to not cause any noticable lag.

This can be done with dummynet on FreeBSD, you'll have to create a 'pipe' with the correct bandwidth, number of slots and source/destination masks.

ipfw pipe 1 config bw 4kbit/s queue 15 mask src-ip 0xffffffff

You may have to set the number of buckets also. 64 is the default and should do for most situations, but if you run a really really big nameserver, you may need more buckets (you need one for each concurrent dns client). Also, I serve some 12 domains from my dns, if you have substantially more then 12 domains, you will have to allow for a somewhat bigger queue, ensure it is a little bit bigger then the total number of meaningfull requests a client could be doing, 1.2 times the number of domains you serve seems reasonable as the number of slots, 1024 * domains/3 seems to be reasonable for bandwidth.

Also, mind the mask on the src-ip, this is what makes this work on a per client basis.

Now you have to direct traffic for the dns into this pipe:

ipfw add pipe 1 udp from any to me 53 in

This assumes you are not using ipfw for anything, if you are using it already, make sure to insert the second line at a proper location so that it actually gets to see the dns traffic.

Also, when already using ipfw, you may have to turn off 'one pass' for the firewall:

sysctl net.inet.ip.fw.one_pass=0

From man ipfw:

     Pass packet to a dummynet(4) ``pipe'' (for bandwidth limitation,
     delay, etc.).  See the TRAFFIC SHAPER (DUMMYNET) CONFIGURATION
     Section for further information.  The search terminates; however,
     on exit from the pipe and if the sysctl(8) variable
     net.inet.ip.fw.one_pass is not set, the packet is passed again to
     the firewall code starting from the next rule.

In other words, not disabling one_pass will result in the packet leaving the firewall when it matches the pipe rule, and no other rules being applied to the packet. This is often NOT what you want.

Instead of the above sysctl command, you can try 'ipfw disable one_pass', supposedly that does the same thing.

How does it work?

First of all it limits the bandwidth a client can use. On top of that, it limits the total number of outstanding requests/client. WHen a client sends too many requests too quickly, excess requests will simply be dropped.

So, how well does it work?

It dropped approx 95% of all incomming requests from the machine trying to do the DOS attack, only answering some 5% of those requests. It did not drop any of the thousands of normal requests from other nameservers during that time. This means to me that it works pretty well, it reduces the DOS attack substantially while not hindering normal traffic. Also, it does not depend on the source IP of the attacker. It will not completely protect against a distributed denial of service attack but it will still reduce its effects on the nameserver.

When implementing a setup like this, use ipfw pipe show at regular intervals to see if it is dropping valid requests, if so, slightly increase the number of slots and/or bandwidth.

Notes:
  • The numbers for bandwidth and slots are based on observation and not on 'solid' calculations. If you have rather large zones, you may need more bandwidth and slots. The assumption is that a client will only have one or at times two outstanding requests for each zone.
  • The allowed bandwidth is higher then strictly required. This is to ensure the nameserver stays reasonably responsive to valid requests. I have aimed for answering less then 1 out of 15 requests from a client that is flooding the nameserrver at 1mbit/sec without dropping requests from 'normal' clients or causing more then 10% latency increase for a directly conencted client (100 mbit ethernet)
  • This setup only makes sense for a public nameserver that is authoritive for some zones, but doesn't answer queries for domains it is not authoritive for. If you also use your nameserver to do hostname lookups for your lan clients, you need to make sure they are excluded from the filter/pipe. Such clients can do lots of valid requests in a very short time, and using a bandwidth/rate limiter will cause problems for this.
  • This setup protects your nameserver and upstream link, it does not protect your downstreamlink. You may have to talk to your provider for filtering on their side for this
  • This setup protects very well against a single client trying to flood your nameserver, it provides only limited protection against distributed attacks

Last but not least, I am still wondering what they were trying to achieve with attacking a DNS that serves a few private domains (and that has a good secondary). No important infrastructure there so it seems utterly pointless... but asking for the point of what happens on the net seems a bit much anyway.




What's Related

Story Options

FreeBSD, dummynet and a nameserver | 10 comments | Create New Account
The following comments are owned by whomever posted them. This site is not responsible for what they say.
FreeBSD, dummynet and a nameserver
Authored by: Anonymous on Saturday, March 12 2005 @ 07:04 PM CET
Good info, thanks!
FreeBSD, dummynet and a nameserver
Authored by: Anonymous on Sunday, March 13 2005 @ 03:32 AM CET
Cool idea, and good article. Will keep this one in mind when I finally get around to set up bandwidth management on my firewall box. Thanks!
FreeBSD, dummynet and a nameserver
Authored by: Anonymous on Sunday, March 13 2005 @ 01:07 PM CET
Intersting article, good info. Well done!
FreeBSD, dummynet and a nameserver
Authored by: bart on Sunday, March 13 2005 @ 05:41 PM CET
Thanks all for the nice feedback. Is there anything you think is missing or unclear?
FreeBSD, dummynet and a nameserver
Authored by: Anonymous on Monday, March 14 2005 @ 04:50 AM CET
A good read on dummynet. I look forward to seeing a longer article about bandwidth management on both upstream and download in the future. Oh, and if it has some more details on multi-homing, multi-default-gateway network. All of these can be done with dummynet.
FreeBSD, dummynet and a nameserver
Authored by: bart on Monday, March 14 2005 @ 05:20 PM CET
I look forward to seeing a longer article about bandwidth management on both upstream and download in the future. Oh, and if it has some more details on multi-homing, multi-default-gateway network. All of these can be done with dummynet.

There is way too little documentation about those things indeed. I could (and maybe will) write quite a bit more about bandwidth management, but I have no experience setting up multi-homing and multi-default-gateway networks with dummynet. If you have the experience and feel like writing soemthign about it, I'd be glad to add it to my site.

FreeBSD, dummynet and a nameserver
Authored by: Anonymous on Monday, March 14 2005 @ 05:35 AM CET
Hi,

I have a question. The use of queues and bandwidth control
mechanisms is great if the conection is from your network.
If the connection is started from Internet to your systems
there is nothing you can do. I mean, once the paquet has
already reach the TCP/IP of your dns server, that paquet
has already consum bandwitdh.
So despite you can control the bandwidth from your network
to internet you cannot do the same in the other way.
Am I right?
FreeBSD, dummynet and a nameserver
Authored by: Anonymous on Monday, March 14 2005 @ 06:00 AM CET
You can't do much about what comes in from the internet unless you can stop it at your provider or further up. Not letting the server deal with it still lowers resource use however and may be usefull just because of that.

In the case of an attack like the one described in this article, the issue is that the answers from the dns consume quite a bit more bandwidth then the requests going to it. As a result, someone with less bandwidth then you may not be able to exhaust your downstream bandwidth but can still exhaust your upstream bandwidth.

The author seems to have a dsl connection with a lot more downstream then upstream bandwidth, making this a much bigger issue.
FreeBSD, dummynet and a nameserver
Authored by: bart on Monday, March 14 2005 @ 12:27 PM CET
I have a question. The use of queues and bandwidth control mechanisms is great if the conection is from your network. If the connection is started from Internet to your systems there is nothing you can do. I mean, once the paquet has already reach the TCP/IP of your dns server, that paquet has already consum bandwitdh.

There are 3 things to consider here:

  • downstream bandwidth
  • server load
  • upstream bandwidth

As you say, you can't do much about the first one (other then talking to your provider).

Keeping packets away from your dns is usefull despite that. First of all, tho my dns refuses queries for the root zone, not all do (often due to misconfiguration, but also at times due to havign a need for this). A request for the nameservers for the root zone is approx 17 bytes, and so is the answer when it is a 'refused', but if the answer is the actual nameservers for the root zone, then it will be a lot more then those 17 bytes, and the same applies to other requests that your dns will answer.

This makes it possible to make your dns generate a lot more traffic then an attack has to send it, causing exhaustion of the uplink.

With an asymetric connection (8mbit down, 1mbit up) even the simple 'refused' answers were already a problem, while the attacker could use approx 1/8 of my downlink, that still resulted in completely saturating my uplink.

But even when packets make it to the ip stack of the dns, it still saves both cpu use and uplink bandwidth to not answer them (and with it saving about half of the bandwidth that the attack would use, this can be very helpfull if you pay for bandwidth like many commercial users.

At any rate, you are right that at least in this case, you can do little to prevent packets from at least making it to your router, from there you are in control

The person who replied before me is wrong on that dns answers are always substantially bigger then requests, in this specific case they were not

FreeBSD, dummynet and a nameserver
Authored by: bart on Tuesday, March 22 2005 @ 05:12 PM CET
 Copyright © 2014 Bart's Soapbox
 All trademarks and copyrights on this page are owned by their respective owners.
Powered By Geeklog 
Created this page in 0.19 seconds 
Powered by Apache Powered by FreeBSD