[nycbug-talk] [announce] Tonight: NYC*BUG at the Soho Apple Store

[alex at pilosoft.com]

On Wed, 6 Jun 2007, George Rosamond wrote:

<snip>
> > Dealing with anything sub 1G of traffic is trivial enough that it
> > could be discussed on nycbug meetings. Oh wait :P
> >
> 
> Yeah, the meeting was all about the speaker's home dsl with IPCop. . .
> :-'
might as well have been.

to be honest, if the presentation was named "tuning freebsd, 5 years 
ago, 101", it'd be more appropriate.

syn cookies are 1996, fbsd implementation '98
syn cache implemented in 2001
polling is early 2002
tcp/udp blackhole sysctl is pre-2002

these are not necessarily ddos avoidance - this is basic tuning.

(the bit about disabling autoneg is factually incorrect and actually 
harmful)

ddos protection in 2007 would discuss cooperative work with your
upstream[s] in handling ddos's over 10gbit in size (see below)

> On the serious side, I thought the meeting was on the mark in terms of
> dealing with the issue, but of course it's people like the above
> replied-to that really matter.
> 
> Not because they're so smart, but rather because we sit on their
> bandwidth.
> 
> The only reason "end-user sysadmins" talk have meetings on topics like
> this is due to the shortcomings of those who do deal with the bandwidth
> for us.
>
> Therefore, Alex and all, please deal with this upstream from us, and we
> promise never to have another meeting on the topic. :)
Wrong. These are your bits, you requested them, you pay for them, you deal
with them. We'll help you to do things you *cannot* but it is your
responsibility to do everything that you can before bothering us.

Asking us to help you with ddos is similar to putting up a webserver and
not knowing how to handle large amount of traffic and saying "I didn't
expect all this traffic and don't want to pay for it".

Today, dealing with ddos means:

* Getting *large* pipes to your upstream (you want a GE port at minimum,
if you get ddos'd with 1G of traffic and you have 100M port, I'll just
tell you to upgrade first before anything else). If you have GE port and 
get 700Mbit of traffic, I'll tell you its your problem. We do our job and 
deliver bits to you.

* Scalability: If you get 1gbit of traffic and you simply can't handle 
dealing with all of it on a single box due to CPU limits, you need to 
figure out how to spread it over multiple boxes. This may be untrivial in 
certain cases.

* Distinguishing bad traffic from good traffic. This is *key*. If you
cannot tell the "good" traffic from bad traffic, you have to deal with
*all* of it. For example, syncache/syncookies will make your system deal
with all traffic - in a better manner. However, if you see all bad traffic
have same tcp SN, you can just drop it before it is handled. If you see 
specific IPs generate 10kpps of syns - drop them.

* Cooperation with upstream: Once you figured out what kind of traffic is
bad (example: list of IPs ddosing you, or UDP traffic that you actually
don't want), you can then contact upstream to put filters on your port.

These are all not rocket science, but a very manual endeavor. For each
type of attack, you have to figure out how to mitigate. Automated-ish
solutions exist (riverhead, arbor) but cost obscene amounts of money
(50K$+) and don't scale for *huge* attacks anyway. There are still ddos's 
that cannot be handled by upstream - when all traffic is potentially good. 
It's all complicated. :)


--
Alex Pilosov    | DSL, Colocation, Hosting Services
President       | alex at pilosoft.com    877-PILOSOFT x601
Pilosoft, Inc.  | http://www.pilosoft.com