[LARTC] big problem with HTB/CBQ and CPU for more than 1.700
customers
Acácio Alves dos Santos
adm.acacio at digi.com.br
Sat May 26 18:22:39 CEST 2007
Pablo,
Here we have HTB being used for more than 10.000 customers. The
difference, is that we use tc and u32 filters to classify the packets..
I use the same Dell PE 1850, but I have two Quad-Core Xeon (1.86GHz)
on it :)
# uptime
13:18:08 up 16 days, 12:32, 1 user, load average: 0.02, 0.02, 0.00
mpstat says:
01:19:11 PM CPU %user %nice %sys %iowait %irq %soft %
steal %idle intr/s
01:19:13 PM all 0.00 0.00 0.00 0.00 0.57 13.81
0.00 85.61 10568.88
And as you can see.. the use of cpu is not that big..
On May 26, 2007, at 6:54 AM, Pablo Fernandes Yahoo wrote:
> Hello,
>
>
>
> have HTB „rules“ in 4 different ISPs and i control for each
> customer this way:
>
>
>
> Flush and 1:0 class
>
> tc qdisc del dev eth0 root
>
> tc qdisc add dev eth0 root handle 1:0 htb
>
> tc class add dev eth0 parent 1:0 classid 1:1 htb rate 100mbit
>
> tc qdisc del dev eth1 root
>
> tc qdisc add dev eth1 root handle 1:0 htb
>
> tc class add dev eth1 parent 1:0 classid 1:1 htb rate 100mbit
>
>
>
> Upload and Download: user1
>
> tc class add dev eth0 parent 1:1 classid 1:5 htb rate 150kbit ceil
> 150kbit
>
> tc qdisc add dev eth0 parent 1:5 handle 5: sfq perturb 10
>
> tc class add dev eth1 parent 1:1 classid 1:5 htb rate 50kbit ceil
> 50kbit
>
> tc qdisc add dev eth1 parent 1:5 handle 5: sfq perturb 10
>
> iptables -t mangle -A POSTROUTING --dest x.x.x.x -o eth0 -j
> CLASSIFY --set-class 1:5
>
> iptables -t mangle -A FORWARD --src x.x.x.x -o eth1 -j CLASSIFY --
> set-class 1:5
>
>
>
> Upload and Download: user2
>
> tc class add dev eth0 parent 1:1 classid 1:8 htb rate 150kbit ceil
> 150kbit
>
> tc qdisc add dev eth0 parent 1:8 handle 8: sfq perturb 10
>
> tc class add dev eth1 parent 1:1 classid 1:8 htb rate 50kbit ceil
> 50kbit
>
> tc qdisc add dev eth1 parent 1:8 handle 8: sfq perturb 10
>
> iptables -t mangle -A POSTROUTING --dest y.y.y.y -o eth0 -j
> CLASSIFY --set-class 1:8
>
> iptables -t mangle -A FORWARD --src y.y.y.y -o eth1 -j CLASSIFY --
> set-class 1:8
>
>
>
> (…)
>
>
>
> This rules works fine, but just for less than 1.700 customers. More
> than 1.700 customers, i have my load avarage in the sky and
> Ksoftirqd process (top information) in 100% fulltime. I don’t know
> why. I used to use CBQ instead HTB because i had the same problem
> and Ron (a guy in this list) gave this rules and told me that he
> uses this for more than 3.000 customers. I tested it in more than 7
> different computers (but the same hadware specifications) and i had
> the same problem with either CBQ or HTB rules. The computers that i
> have are all of them DELL PowerEdge 1850. I will put some hardware
> iformations here:
>
>
>
> top
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>
> 3 root 39 19 0 0 0 R 100 0.0 5316:20
> ksoftirqd/0
>
>
>
> [root at fw ~]# uptime
>
> 10:38:11 up 161 days, 17:21, 3 users, load average: 1.58, 1.65,
> 1.51 (unfortunately when i took this, the load average was
> „pretty good“, but minutes ago, it was more than 11.0
>
>
>
> [root at fw ~]# lspci
>
> 00:00.0 Host bridge: Intel Corporation E7520 Memory Controller Hub
> (rev 09)
>
> 00:02.0 PCI bridge: Intel Corporation E7525/E7520/E7320 PCI Express
> Port A (rev 09)
>
> 00:04.0 PCI bridge: Intel Corporation E7525/E7520 PCI Express Port
> B (rev 09)
>
> 00:05.0 PCI bridge: Intel Corporation E7520 PCI Express Port B1
> (rev 09)
>
> 00:06.0 PCI bridge: Intel Corporation E7520 PCI Express Port C (rev
> 09)
>
> 00:1d.0 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R)
> USB UHCI Controller #1 (rev 02)
>
> 00:1d.1 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R)
> USB UHCI Controller #2 (rev 02)
>
> 00:1d.2 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R)
> USB UHCI Controller #3 (rev 02)
>
> 00:1d.7 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R)
> USB2 EHCI Controller (rev 02)
>
> 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev c2)
>
> 00:1f.0 ISA bridge: Intel Corporation 82801EB/ER (ICH5/ICH5R) LPC
> Interface Bridge (rev 02)
>
> 00:1f.1 IDE interface: Intel Corporation 82801EB/ER (ICH5/ICH5R)
> IDE Controller (rev 02)
>
> 01:00.0 PCI bridge: Intel Corporation 80332 [Dobson] I/O processor
> (A-Segment Bridge) (rev 06)
>
> 01:00.2 PCI bridge: Intel Corporation 80332 [Dobson] I/O processor
> (B-Segment Bridge) (rev 06)
>
> 02:0c.0 Ethernet controller: Intel Corporation 82545GM Gigabit
> Ethernet Controller (rev 04)
>
> 02:0e.0 RAID bus controller: Dell PowerEdge Expandable RAID
> controller 4 (rev 06)
>
> 03:0b.0 Ethernet controller: Intel Corporation 82545GM Gigabit
> Ethernet Controller (rev 04)
>
> 05:00.0 PCI bridge: Intel Corporation 6700PXH PCI Express-to-PCI
> Bridge A (rev 09)
>
> 05:00.2 PCI bridge: Intel Corporation 6700PXH PCI Express-to-PCI
> Bridge B (rev 09)
>
> 06:07.0 Ethernet controller: Intel Corporation 82541GI/PI Gigabit
> Ethernet Controller (rev 05)
>
> 07:08.0 Ethernet controller: Intel Corporation 82541GI/PI Gigabit
> Ethernet Controller (rev 05)
>
> 09:0d.0 VGA compatible controller: ATI Technologies Inc Radeon
> RV100 QY [Radeon 7000/VE]
>
>
>
> [root at fw ~]# free -m
>
> total used free shared buffers
> cached
>
> Mem: 2021 1479 542 0
> 400 654
>
> -/+ buffers/cache: 424 1597
>
> Swap: 1027 0 1027
>
>
>
> [root at fw ~]# cat /proc/cpuinfo
>
> processor : 0
>
> vendor_id : GenuineIntel
>
> cpu family : 15
>
> model : 4
>
> model name : Intel(R) Xeon(TM) CPU 3.00GHz
>
> stepping : 3
>
> cpu MHz : 2992.674
>
> cache size : 2048 KB
>
> physical id : 0
>
> siblings : 2
>
> core id : 0
>
> cpu cores : 1
>
> fdiv_bug : no
>
> hlt_bug : no
>
> f00f_bug : no
>
> coma_bug : no
>
> fpu : yes
>
> fpu_exception : yes
>
> cpuid level : 5
>
> wp : yes
>
> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr
> pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm
> pbe nx lm constant_tsc pni monitor ds_cpl cid cx16 xtpr
>
> bogomips : 5990.78
>
>
>
> processor : 1
>
> vendor_id : GenuineIntel
>
> cpu family : 15
>
> model : 4
>
> model name : Intel(R) Xeon(TM) CPU 3.00GHz
>
> stepping : 3
>
> cpu MHz : 2992.674
>
> cache size : 2048 KB
>
> physical id : 0
>
> siblings : 2
>
> core id : 0
>
> cpu cores : 1
>
> fdiv_bug : no
>
> hlt_bug : no
>
> f00f_bug : no
>
> coma_bug : no
>
> fpu : yes
>
> fpu_exception : yes
>
> cpuid level : 5
>
> wp : yes
>
> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr
> pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm
> pbe nx lm constant_tsc pni monitor ds_cpl cid cx16 xtpr
>
> bogomips : 5985.13
>
>
>
>
>
> Any help/Tipp/hint will be very welcome.
>
>
>
> Thanks in Advance!
>
>
>
> Pablo Fernandes
>
>
>
> _______________________________________________
> LARTC mailing list
> LARTC at mailman.ds9a.nl
> http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
--
Acácio Alves dos Santos
Administração de redes
Diginet Brasil
adm.acacio at digi.com.br
(+55) 84 4008-9000
Esta mensagem, incluindo seus anexos, pode conter informação
confidencial e/ou
privilegiada. Se você não for o destinatário ou a pessoa autorizada a
receber
esta mensagem, não pode usar, copiar ou divulgar as informações nela
contidas ou tomar qualquer ação baseada nessas informações. Se você
recebeu esta mensagem por engano, por favor avise imediatamente o
remetente, respondendo o e-mail e em seguida apague-o. Agradecemos sua
cooperação.
This message, including its attatchments, may contain confidential
and/or
privileged information. If you are not the recipient or authorized
person to
receive this message, you must not use, copy, disclose or take any
action based
on this message or any information herein. If you received this message
by mistake, please advise the sender immediately by replying the e-
mail and
deleting this message. Thank you for your cooperation.
More information about the LARTC
mailing list