The following example will drop incoming connections if IP make more than 10 connection attempts to port 80 within 100 seconds (add rules to your iptables shell script)
Code:
IPT=/sbin/iptables
# Max connection in seconds
SECONDS=100
# Max connections per IP
BLOCKCOUNT=10
# ....
# ..
# default action can be DROP or REJECT
DACTION="DROP"
$IPT -I INPUT -p tcp --dport 80 -i eth0 -m state --state NEW -m recent --set
$IPT -I INPUT -p tcp --dport 80 -i eth0 -m state --state NEW -m recent --update --seconds ${SECONDS} --hitcount ${BLOCKCOUNT} -j ${DACTION}
# ....
# ..