[LARTC] HTB - not borrowing, not exceeding rate

[Kajetan Staszkiewicz]

Hello!

I have a quite complicated setup. In my network on each interface there is 
bandwidth limitation for each user. Booth outgoing (on interface itself) and 
incoming (attached IMQ) traffic. There is main HTB class which limits 
bandwidth for whole interface and HTB subclasses for each user. Filtering is 
done with hashing filters. This setup was working correctly.

But now in the network I have a DC hub (p2p) which allows user to exchange 
files. The problem is that I want in-network p2p connections to be faster 
than Internet bandwidth (and not "eating" their Internet bandwidth) and also 
I want in-network connections to have lower priority in borrowing the link 
bandwidth (I am using radio links between network segments and having "good" 
Internet access is priority over local traffic).

Here is current setup:

tc qdisc del root dev eth2.24 2>/dev/null
tc qdisc add root dev eth2.24 handle 1: htb default 1

# main rate limitation for whole connection (802.11a radio link)
tc class add dev eth2.24 parent 1: classid 1:1 htb rate 15000kbit ceil 
15000kbit burst 10kbit

# class for internet connections - this one can use nearly whole link
tc class add dev eth2.24 parent 1:1 classid 1:6667 htb rate 12000kbit ceil 
13500kbit burst 100kbit

# class for in-network p2p connections - this one has lower guaranteed rate
tc class add dev eth2.24 parent 1:1 classid 1:6666 htb rate 3000kbit ceil 
14000kbit burst 10kbit
tc qdisc add dev eth2.24 parent 1:6666 handle 6666: sfq perturb 5 quantum 
1500b

# users (htb+sfq for each):
tc class add dev eth2.24 parent 1:6667 classid 1:2 htb rate 1kbit ceil 256kbit 
quantum 2000 burst 100kbit
tc qdisc add dev eth2.24 parent 1:2 handle 2: sfq perturb 5 quantum 1500b
tc class add dev eth2.24 parent 1:6667 classid 1:3 htb rate 1kbit ceil 256kbit 
quantum 2000 burst 100kbit
tc qdisc add dev eth2.24 parent 1:3 handle 3: sfq perturb 5 quantum 1500b
....
....
tc class add dev eth2.24 parent 1:6667 classid 1:1006 htb rate 1kbit ceil 
384kbit quantum 2000 burst 100kbit
tc qdisc add dev eth2.24 parent 1:1006 handle 1006: sfq perturb 5 quantum 
1500b

(some hashing filters - which I think are working properly, so not shown here)

# traffic going to these networks goes to hashing filters
tc filter add dev eth2.24 protocol ip parent 1:0 u32 match ip dst 
192.168.1.0/24 hashkey mask 0x000000ff at 16 link 2:
$TC filter add dev eth2.24 protocol ip parent 1:0 u32 match ip dst 
192.168.3.0/24 hashkey mask 0x000000ff at 16 link 3:

# now in-network p2p traffic. if it comes from other LANs then direct it to
# 1:6666
tc filter add dev eth2.24 protocol ip parent 1:0 u32 match ip src 
192.168.5.0/24 flowid 1:6666
tc filter add dev eth2.24 protocol ip parent 1:0 u32 match ip src 
192.168.4.0/24 flowid 1:6666

Filters look OK. AFAIK the last filter is the most important so even trafiic 
to 192.168.1.0/24 but comming from 192.168.5.0/24 will go to 1:6666 instead 
of hashing filters and user bandwidth HTB. The hashing filtes were working on 
previous setup and now users still have their proper bandwidth from the 
Internet.

Here is the graph representing the traffic:
http://tuxpowered.net/lan_p2p/lan_eth1_rx_dzien.png
the colors are:
#ff00ff - htb 1:6666 (LAN p2p)
#00ff00 - htb 1:6667 (Internet)
#000000 - real interface traffic (tc -s li show ethXX)

And now description of the problem:
Class 1:6666 never has more traffic than 'rate'. AFAIK it should be having 
'rate' as minimum guaranted rate and going up to 'ceil' if there is free 
bandwidth. (directing LAN traffic to 1:1 works OK - look at the graph at 
about 14:45 - 14:55)

Bandwidth sharing setup works for me on WAN interfaces, but there I have only 
1 level of classes tree. Here are 2 levels and it is not working.

What I want:

.---------------------------------.
| 1:1 rate = ceil = link bandwidth|
`---------------------------------'
   |                      |
.--------------------. .-------------------------.
| 1:6666 p2p traffic | | 1:6667 Internet traffic |
| low priority in    | | high prioriy in         |
| badwidth sharing:  | | bandwidth sharing:      |
| ceil =~ from 1:1   | | ceil =~ from 1:1        |
| rate = small       | | rate = (from1:1)-1:6666 |
`--------------------' `-------------------------'
                         |
                         +-- 1:2 user rate=1kbit ceil=256kbit
                         +-- 1:2 user rate=1kbit ceil=256kbit
                         .....
                         +-- 1:1203 user rate=1kbit ceil=384kbit

-- 
| pozdrawiam / greetings | powered by Trustix, Gentoo and FreeBSD |
|  Kajetan Staszkiewicz  | JID: vegeta at chrome.pl                  |
|        Vegeta          | IMQ devnames: http://tuxpowered.net    |
`------------------------^----------------------------------------'