[LARTC] 2.6.14 - HTB/SFQ QoS broken?

[Jody Shumaker]

On 12/28/05, Leo Bogert <spam-goes-to-dev-null at gmx.net> wrote:
>
> > It's undefined what happens when you overallocate, and is certainly not
> > required to do what you want. 'rate' is guarenteed rate, 'ceil' is
> maximum
> > bandwidth useable through borrowing, and 'prio' is priority this class
> > gets when borrowing bandwidth. You should give emule next to nothing for
> > guarenteed, but with a ceil either the max bandwidth or very close. It
> is
> > then guarenteed little, but will borrow up to its ceil. Then guarentee
> > apache most, with some left for default. the prio will mean apache has
> > first access to any borrowable bandwidth, then default, then emule.
>
> It _does_ seem to be necessary that the guranteed rates overlap.
> Let me explain: I have 45 kb/s of upstream.
> I basically offer 4 services: eMule, FTP, Apache, Miranda filetransfer.
> To me, these four have different priorities: eMule should be consuming
> the whole bandwidth if nothing else is being used. If someone is
> downloading from FTP, he should take away most bandwidth from eMule,
> let's say 32 kb/s. But then it might happen that I want to send a
> file to somebody via Miranda because he does not have an FTP account.
> Then the filetransfer has to be higher priority than the FTP transfer
> and receive about 30 kb/s, FTP and eMule should use the remaining
> bandwidth. But now it might also happen that - while somebody is
> downloading from FTP and a miranda filetransfer is running -
> I want to show a picture to someone which is stored on my webserver.
> Thus, Apache needs a guranteed rate of 30 kb/s, i.e. higher priority
> than FTP and filetransfer.


Wrong, it doesn't need a higher guarenteed rate.  It only needs higher
priority when borrowing bandwidth, and for there to be available bandwidth
to borrow.   If there's no bandwidth to borrow, then you're guarenteeing too
much to other classes.

Furthermore, while all this stuff is happening, I might be gaming
> or browsing the web and for this I need a good response time.
> But I don't know anything about the bandwidth requirement of games
> or webbrowsing, thus in my old script I gave a rate of $IFUP with
> priority 0 to the default traffic so that it can take anything which
> it needs.


This is unneccasary.  Setting a guarenteed rate , and then giving it an
appropiate priority, it will use up to the full connection by borrowing.
Also in my experience, depending on the game, they use 3-10kbyte/s.

This complex situation with all classes being active at once
> shows quite clearly why I want to have overlapping guranteed
> bandwidths. It will never happen probably, but the situation that
> I'm running a Miranda filetransfer and want to show somebody a picture
> on Apache happens very often.



It quite clearly shows, you need to think more closely how it works.  You're
trying to use guarenteed rate to control lending/borrowing which is the
wrong way to think of it.

The fact why I don't like reserving bandwidth with the sum of
> child rates being _correct_ (not larger than available upstream)
> is that this would make me "lose" much bandwidth, look at this
> example which I created for testing:
>
> eMule rate 8kbps prio 5
> FTP rate 4kbps prio 4
> Miranda rate 12kbps prio 3
> Apache rate 8kbps prio 2
> Skype rate 6kbps prio 0
> default rate 10kbps prio 1
>
> The sum of the child rates is correct, but I cannot make Miranda
> or apache receive a guranteed 30 kb/s while eMule is running.
> Thus, some kind of overlapping might be necessary for a perfect
> scheduler.


You don't do this through guarenteed rate, you instead do this with priority
on who gets dibs on free bandwidth first.  Having a higher priority, a class
should use up to its ceil if the bandwidth is available to be borrowed.   If
all the bandwidth is over allocated as guarenteed then there is nothing to
borrow from!

Think of it this way, say every single class is trying to use up to its ceil
in bandwidth.  When the base class has available bandwidth, it is going to
look at all of its subclasses to choose who to grab a packet from.  With
over guarenteed rates, it may negelect classes because it will always grab
packets from other classes since they haven't gone over their guarenteed
rate.  With your setup, say 30kb/s for apache.  If an apache and ftp both
start they may consume ALL of the bandwidth and completely starve things you
don't want it to, like voice or games. The problem with over guarenteeing is
that all of HTB's rules on lending/borrowing get bypassed and instead things
just get allocated randomly.

I'm sorry for the double email, but I think my previous one was sent too
early in the morning before I could think. I really want to explain in a way
you can understand why over guarenteeing is not something you want.

Now, as for your actual problem, I don't think overguarenteeing is even
becoming an issue based on the tc -s class show dev  results you posted.  As
I mentioned in the other emails, the basis of where i think the problem
you're seeing is has to do with how TCP throttles bandwidth itself.  Either
the transfer can't perform any faster regardlesss of eMule, OR it can but
the burst setting should be tweaked.   I've found that with some people, the
transfers tend to be very bursty.  I can get 70% of my bandwidth used on
average for them, but it varies between 40% and 100%.  With a small default
burst, and the bandwidth otherwise used what will tend to happen is the tcp
transfer will cap out instead near the 40%.  If i increase the burst size
for higher priority classes,  they have a better chance of bursting to 100%
of the bandwidth occasionally to give a better average rate.

Based on how I've gotten things working with my setup, I would suggest these
changes to your script atleast for testing:

tc qdisc add $IF root handle 1:0 htb default 20 r2q 4
$addc 1:0 classid $cMAIN htb rate $IFUP mtu 1492 burst 30k

$addc $cMAIN classid $cMIRANDA  htb rate 30kbps ceil $IFUP $q2 prio 2 burst
30k
$addc $cMAIN classid $cDEFAULT  htb rate 10kbps ceil $IFUP burst 6k $q2 prio
1
$addc $cMAIN classid $cEMULE    htb rate 4kbps  ceil $IFUP $q1 prio 5


And as previously mentioned,  check tc -s class show dev $IF   results for
atleast these 2 scenarios,  ftp only, ftp and emule.  Checking emule only
might be useful to make sure it behaves as you want, but your previous stats
missed the important comparison.  You really want to be sure that ftp or
whatever gets the same bandwidth with emule on or off.  Your last stats only
showed that emule will use up what's available, and gave no basis to know if
the ftp portion was working correctly or not. The ftp only portion would
give the comparison you need to see if it is working.


I hope I'm not ranting too much, and that some of this is understandable.

- Jody
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ds9a.nl/pipermail/lartc/attachments/20051228/a4183f24/attachment-0001.htm