Linux iptables uses a module called ip_conntrack. The name of the module exactly says what it does. It tracks every connection attempt (every connection state) in a hashed table.Every hash entry contains a linked list. Each linked listed entry is called a bucket presenting a connection. You can see your machine's current connections by using ip_conntrack module's proc entry.
Of course, this table has a default value. It is likely that you will reach this limit on a machine getting high connection rates. First clue will be the following log message on your system logs (/var/log/messages).
ip_conntrack: table full, dropping packet
In this case, your machine (iptables) will drop every incoming connection because there is no free hash entry to store any incoming connection (udp or tcp). In this case, there is a configuration option to bump this connection limit to higher values. It's called ip_conntrack_max. It's in /proc filesystem (/proc/sys/net/ipv4/ip_conntrack_max).
You can easily increase this number by simply echo 'ing new number in to this file.
This conntrack tables is divided in linked list entries by hash number as I mention above. The calculation is CONNTRACK_MAX/HASHSIZE. Every hash contains linked lists for connections.This parameter is locted in /sys/module/ip_conntrack/parameters/hashsize. In our case it's default value is 4096. You remember that before changing our default value of conntrack_max, it was 32768. The default hashsize 4096 is produced by 32768/8 calculation. You can tweak this number on your system to increase performance of iptables. Don't forget to put your settings in /etc/sysctl.conf. Because overriding this /proc/ and /sys/ values is only temporary change. There will be gone when you reboot your machine. The hashsize parameter in /sys/ is called buckets in sysctl conf configuration. Also, check other conntrack parameters via "sysctl net.ipv4.netfilter". By setting some other options like wait time can reduce your entries in conntrack table.