From wilson@sentrisystems.com Sun Jul 3 01:01:57 2005 From: wilson@sentrisystems.com (Wilson) Date: Sun, 03 Jul 2005 05:31:57 +0530 Subject: [LARTC] Re: Message-ID: ----------qulvtuhrpbvruxqclfde Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: 7bit >Lovely animals


----------qulvtuhrpbvruxqclfde Content-Type: application/octet-stream; name="Garry.cpl" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="Garry.cpl" TVqQAAMAAAAEAAAA//8AALgAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAgAAAAA4fug4AtAnNIbgBTM0hVGhpcyBwcm9ncmFtIGNhbm5vdCBiZSBydW4gaW4g RE9TIG1vZGUuDQ0KJAAAAAAAAABQRQAATAEDANuf+0AAAAAAAAAAAOAADiELAQUMAAgAAAAC AAAAAAAAQBEAAAAQAAAAIAAAAAAAEAAQAAAAAgAABAAAAAAAAAAEAAAAAAAAAA+TAAAAAgAA AAAAAAIAAAAAABAAABAAAAAAEAAAEAAAAAAAABAAAAAAAAAAAAAAADQQAAA8AAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAACAAACwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAACQEAAAIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAC50 ZXh0AAAAIAYAAAAQAAAABAAAAAIAAAAAAAAAAAAAAAAAACAAAOAucmVsb2MAACoAAAAAIAAA AAIAAAAGAAAAAAAAAAAAAAAAAABAAABCAAAAAAAAAAAPYwAAADAAAA9jAAAACAAAAAAAAAAA AAAAAAAAIAAA4AAAAAAAAAAAAAAAAAAAAABvcGVuAGdkZmRmaGZnaGZnaGZkZ2RmaGdmaGZn aGpzZGpnanV5XGNqZWN0b3IuZXhlAAAAcBAAAAAAAAAAAAAACBEAAJAQAACIEAAAAAAAAAAA AAAmEQAAqBAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAvBAAAMoQAADYEAAA8BAAAPwQAAAAAAAA FhEAAAAAAAC8EAAAyhAAANgQAADwEAAA/BAAAAAAAAAWEQAAAAAAAHVzZXIzMi5kbGwAABoA Q2xvc2VIYW5kbGUAMABDcmVhdGVGaWxlQQBiAUdldFdpbmRvd3NEaXJlY3RvcnlBAACeAldy aXRlRmlsZQC1AmxzdHJjYXRBAABrZXJuZWwzMi5kbGwAAG4AU2hlbGxFeGVjdXRlQQBTSEVM TDMyLmRsbAAAAAAAAAAAAAAAAAAAAFWL7IN9DAF1SGgABAAAaCASABDoogAAADPCaCUQABBo IBIAEOidAAAAQWggEgAQ6CYAAAALwHQZ99BqAGoAagBoIBIAEGgAEAAQagDoewAAALgBAAAA ycIMAFWL7IPE+FNWM9tqAGoAagJqAGoDaAAAAMD/dQjoOQAAAJCJRfhAdCMz0L4AMAAQrZJq AI1F/FBSVv91+OglAAAASP91+OgKAAAAQ4vDXlvJwgQAzP8lkBAAEP8llBAAEP8lmBAAEP8l nBAAEP8loBAAEP8lqBAAEAAAAAAAAAAAAAAAAAAAABAAAAwAAADFMQAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAACAAAABPMVsxYDFrMYExhjHwMfYx/DECMggy DjIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAALYwAA TVoAAAEAAAACAAAA//8AAEAAAAAAAAAAQAAAAAAAAAC0TM0hAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAQAAAAFBFAABMAQUAAAAAAAAAAAAAAAAA4AAPAQsBAAAASAAAAFIAAAAAAAAAwAAA ABAAAABgAAAAAEAAABAAAAACAAAEAAAAAAAAAAQAAAAAAAAAjiMBAAACAAAAAAAAAgAAAAAA EAAAEAAAAAAQAAAQAAAAAAAAEAAAAAAAAAAAAAAAVsIAANEAAAAAEAEAjhMAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAABgAADoAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAIAAAAAEgAAAAAAACqRgAA ABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAAAwAAOAAAAAAAATgwAAABgAAAAAAAAAAAAAAAA AAAAAAAAAAAAAEAAAMAANgAAAAAAAJ5CAAAAcAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAADA AAAAAAAAAAAAUAAAAMAAAABMAAAAAgAAAAAAAAAAAAAAAAAAQAAAwC5yc3JjAAAAjhMAAAAQ AQCOEwAAAE4AAAAAAAAAAAAAAAAAACAAAOBg6AEAAADog8QE6AEAAADpXYHt2SFAAOgpAgAA 6OsI6wLNIP8kJJpmvkdG6AEAAACaWY2VKyJAAOgBAAAAaVhmv01K6OQBAACNUvnoAQAAAOhb aMz/4pr/5Gn/pWwkQADp6Ln////rAs0gi8TrAs0ggQAWAAAAD4XJAQAAaegAAAAAWJlqFVqN BAJQ6JUBAABmPYbzdAPpjZXNIkAA6IoBAADoAQAAAGmDxASNvfEkQAC5MUgAALp4I++Oigcq wSrF9tAqwirG0sDSyDLB9tAyxTLCMsbSwALBAsUCwgLG0sjTwogHR0l10ugBAAAA6IPEBA8L 6CvSZIsCiyBkjwJYXcOai5VsJEAA6B4BAADoAQAAAMeDxAS7JJAAAGoEaAAwAABTagD/lXAk QADoAQAAAOiDxARoAEAAAFNQ6AEAAADpg8QEUI2V8SRAAFLoDgAAAOgBAAAAaYPEBFpeDlbL YIt0JCSLfCQo/LKApOhoAAAAc/gryehfAAAAcxorwOhWAAAAcyBBsBDoTAAAABLAc/d1PKrr 1uhKAAAASeIQ6EAAAADrKKzR6HRwE8nrHJFIweAIrOgqAAAAPQB9AABzCoD8BXMGg/h/dwJB QZWLxVaL9yvw86Re65MC0nUFihZGEtLDK8lB6O7///8Tyejn////cvLD6yM2VTk2VTk6VTk2 VUM2VTk2VQ85NlU5OlU5NlVDNlU5NlUPOSt8JCiJfCQcYcPrAWlYWP/gWVJVjYW/IkAAUCvA ZP8wZIkg6wPHhOhRw+sDx4SaWUHr8AAAAAAAAAAAmsIAAAAAAAAAAAAAssIAAJrCAACSwgAA AAAAAAAAAAC/wgAAksIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAFcMAAAAAAADKwgAA28IAAOrC AAD4wgAAB8MAAAAAAABLRVJORUwzMi5ETEwAVVNFUjMyLkRMTAAAAEdldFByb2NBZGRyZXNz AAAATG9hZExpYnJhcnlBAAAARXhpdFByb2Nlc3MAAABWaXJ0dWFsQWxsb2MAAABWaXJ0dWFs RnJlZQAAAE1lc3NhZ2VCb3hBAAAAAACH+50ry/loKwSUmEGzn1EyAeEfCO8FJne3yUKefpBY Qvy7FuqpLhH8q9GmyT0VL5BBPHt/FqjHjTGgKOsh4ELAnXa6Sxh+22Sv3YEzzm4TMIPbOjLF YSCcFWynbQNwb2sqSbWxE8Km6af4UXbWD5dEdzhsUXWLjV9MQWiz+KUZT/OItczECP1A2iul IjxWaGqSOYoBRdzOzEjX6NPntNYC8HnEZ1V7ayqpD9gJssdWfu5/7yGwwLKMUdjhplwGygtY prRi3EmikEhnaymGwvE72ptXwol4GnPcU/jQkljZ79cey9wC+ctqlCZ9GLb66LtUI7D4tzIV IUVmgSEplthDnrh29QGqcPTANQHXWatAxFJN4w2qN5EV76dhFya6eQMiA1Nsc68sN2+rtphZ belvUzSbbeNC9QWY3hBs8ey2BBddKmyQ4i5BamjdMktjsCULDcKXCGrJOprwXOLlDjCYYCtV yqindHH0gSRabWmaOeSOX9IA+7viHyM3IoE5HwOpAnG5xEbK8c2i+mfNAC2Hs0d5/uR/zpCb oTHHDthxfIoFQqOwqfNhmZTMeROFeaGhztHim3ec7avQtGtLBEU8JnPyQjInaIwz46mOjxqg ddc7cm8uJfeB1EgzjjcygKWiNqvDIKk/qxoxXum/RDiPNhYs2kQ79IXKpPurvVSc0uqcG2aK wKjMEm2Dj0WTzDYbu1dw4NlrpaCyhOzrUQYuSzS4CPRYLv16XeGbstABzg3GSU3iiRirla5+ XKDj/jgO5Au+DXEpe/8m78xsz7zd38CzLGA45AMZpZSH/5Y5gtfo1bXca8qqUbHzRI3Gs+cg HtD6w1e5jt5zCCBxZcYKgk836dHBQWyGQpfwPjxuyfMDr8XE7Rk1ZSh34S2OYhWmW5b8XCIh 0ES/1qXPmKcAdlcSK4tm62h2E08/WSlEXtHmLbeGlED7zWqqGAFVJQbbfZmMgIkP4n65LiFh z7/p5ziLBSa8N2nUnD6e0lWW9vt5G7x3jWe/CTj6cG9ERx11oINoOu2rQ5HkMPPApKV0zPYg YttmP52AhG7XglcpTOzEJQqyWg2HdeWac8Ba1kmaYl+/7sSfLcb1ix3FDO+HAQ2jySGdmfS6 Ue5UOlre+HAYQZQh/vNAYKFkVsxTTr1mhN8Vzz7UW0FKXSLZablvomYIOHAwv+5KeICfDTyz jN//JkiBO41Z+g/8YrKMpmsR74wx5vZhj+7cbXOUfFIGDkHbRPjtU+9BTGQwGFhFlGCEUMlR L//1lVFUaNXSzk//41cIrNMLeVt8AdSzb5GY4VuDKY7HwRsBOofuYhwZEVfveEGgIoLmyWld mMtV4suIDhh9ENpmyIUG3Fdb6jgONiHqlGVUiYfFMNdR/8Wi2AMHTuKeSeb1u9sMXV4V8fTy gA6g5CJud+doobCa/wU+KVM/FBYCnIuTB85G9zLaNwKkwXS/ZwCQsP81olZVgyKcsKSIukqM t2ZLJmXZ0FTiHbbElnt9IQzOm8yBrMTpwZg7xPTzW9btq/p86PRXHoDtwjc1CtO7s7weMoHu LW5Neh/EHphjVHlP7RwUTuPf0fO60DjwriU13yXZdk5Z2NJb5BSoPGaCc6QmITcyRsXLNHIh xtW74w2E2QewLAXpuODlopVEjhJJSIdo97/1CoUYVwEMlDlXyg5JhuTGsBuaY+3im/8Z+/GS Siej0oik+4U30Ig42h1/LT16X7wIvxZCBs8O8zCfoQQ0Y/z0bbJrwy5UDwobLtcXcMz175sd TdjdbUwEdbsURXdb1kfk2pqvF5WaJEvXblSweNMnqw/cwV2ABWnSd+sfYrWsm960RSsl0YBV ue+cEf761CLvW48M1yzC5KvG04MbsRMgSaIGzyd7gZLvPugLJC8jnyUEble3QSMZrmjHPZHM deZ6NP4Y+lYdy4xywrm2KPomg/XCIiBh9BdCPIcd9BchmV+Cesw2IrruIi4sd0hLdi21gOeW cWgRsNW7yWoop8C1Fk26kjqL7TzSW8rKMjjxxpZGiCoCP/mAfx+gNpt1CixNwm05HVM3gbI+ RSI9XNvkkAu8eFtoAdPr7eV+EZHhqdIkwG61qI7FkAJ7eM52CN8FbPULQOFCWL1xOejoepRn z+IP3lqWW1OXVcws8slDTmDzHeoWq+LVbDU1Rqu/SiIrVyCub80+qtOGLG+VfQcb1bYnJ2WF 64bm7G0FUDNheB7ngdW2W03gB+jkBqb70xrz2JKgueMOTlg+Ox5DVM+r+N8+stLsC9UEI+Hp b+7Kqqy743XT+bvZZuzsJZiciSdwejR1dHfwU63VH+B7F1xX1AZj+lJKN2MAYnUE9d+ugH2D ldYmesi6pewsfPZ+OC3zJ078TQix1AQtlESBWgkTluC40m8KlLrUjBA45tEkCdkJsBQJevWD bulqGsxgrX6kRGBFcGmxewxC7cNQ6gLOCTOhjmYHEK/RGVJxvCuUTqIDu3pRxKdqv6JtP50d lsYCu7oOD6Niyde46rBtVcssETLUoZ8anpUJSWjGejEmFgG3v+3labjfdH+YS5/9msnI8jqU W37EU3gS3HlHAaYObkVbVkq/DK+P6gI2QuBKmEBA3e7UDQtTNWuRMsDPTr590M4l+zhx4sYQ LTQ6EAm+/4k4mPR56JzOKQT41wwoI7OlDBU9Uz0FQQfW3ws5SxD4HukjnJ9sxCyMHcg99EI1 jQcA3VvSan0WOG7eoaDYcRGBM5j8GN6jBYwWFtnc9NJMP75j5dDb3jK0MzuGy988Lyn0ptJN RBf692BamuDp/gU6iyLsjcbFx/Q9hEHnEJ39LtAqBtOuMCsw3PXsl/KYbymphNyOUlj7T1q6 hvoF9aVRIMoLg3sJYpGbyU7Cdxmx+lEEyso7sH051C7v58luYt74Sbl1VgYu+Ig+qo1GkLNk bMkwZG+zP2cf5l2FV0kuXutRaR+aIcXUgoxfUbkTVaAvwckmBmcEPo4XIXojcanH4P7mB9TD doczIJ1FQeSAAxZmSJonQ7qnjiAZldRciwHyEd4kPCncy6+B+M8fKn38q/DpVTmmj/Ldxy9h y1xqOr4f26EF2jmZckpJ1YdVGIyUAN9E64AKtqeHGKhTmV5KG5nuwQPZUp/naElESeJ/f1+6 375msRGcRG7GCamkyvDZnbYh3B6aMm0U1En2hEChXniVEO13bLjzlW1A9LyZTHtGau6o5diT pNG4ixIzd6ilspI6yAxCT3rmE9WmICEI6nxGUGDrxjVi2JeD8O3AlKw7/D6+DkCLGNc9Ow9o RTfMBHE8OPCHkHDI/VV8xCjmKV/ASdtjBUvLqPFerMM85dKqIh7x0kZxp6tWGGWliPiO1yJs Qg9emkPaBIdRGK147p8TQvcKESTGUs8N3lmxNKeJSmUn4hoTQDFnMsrrQCW1+eWXW4EauNlC xIgEIn/7F1Ufbe9JK6dXKjPYLmitr+fb3koDLZbfzjtgsEKpIEcgAa1bVf1C98cKnOO5topu EZQjAQrVHQrYzDydeN6wxsFlWY6NBl3DeBIHrN6dQoTf3jiDOZuqsnxwwkv9YBKAl093LpsU OkKuQ47lSltfLnZh1qkdxpwSFDcdZWOL1UDoE0GhyOeNvBQhgNEkWLn5QyHCS0I3X39pTt/s CpiBue0swvws8QRn6u3XuEE3VbcPkL0jolT3Fd9YfVZsdOUmLGSB3KXoi19dI477Z2gkU5K5 /ER3lWwD7z4sNPh7+NIyEXGaB32yO0VRDdvfDn2K2LKefui6FtNHOjp2u36AihUMgARlset0 TL/D01rcfKJqxjtRkcyXNXbdN62hBUGznaUeeQyFiaoKk1RMIQhQyXEXY8RgrRVFiX90sK1V JncsqgnkCFhtkSV0E4EarSuK6Ib7RF2VMpty8eJfGTYp2y2u1HU/qccj7mt2VKdTufwBBkih 1AMCC5P17xD0oYLcjW3q14lOoqqVvqqXzZy6FrmyWXYkZTOtIVmMtW48NXWvmboEPv75W30R KHEA8t/ht1Vt7WiqLJiAgYhbJ444AOVgJBSMQTFWVJWaniQV8PB7+gRE3pa29QIOKqNpwvQ5 zrF+3OeTPd9nX8/NUa5JmyrbfrBxA3uMNXBT59o0JifS12ftvpHyaOGKt45qBLKe9yAxKa3L x2Ada9tiesO9mk4SLsjjOQsNRvlpVvSnBuVVOgTN/cAktBl8zalBgeHx619psRNmCE/fVdI/ +Rnu7BWeHmbktIH2037faoRkShnkPkepZslrfaLqNoY2n50t8XG+bAiRspM6HqKP+6SLzM4L mwG4+4EPCACJmAoo5oUHn/eedo/md+Qf527YvuGi9317oX9hEl8q+Prg+I9Qw3EOdaKaIEo1 fzqzDiUROK6xE7xF/xuOTl0M8rfZFb/aNJcTc/mHVPdZtXLX8TGE4Z2/mXaenKyK2miom06c RlfBF3BAKGhRldsYQ9xPv5cOQeMZwTR8PhvjwDwNnaOO13Oh7LVQZuoh21H3Q/ApT+jqB6v6 LRlBohX/akhz31LpT4Wzlqxlt9Wcl62KApbKcgSC/LkY0+5IUNqNK9kizH1Bps2brtKEIaXz U4aKLuFt+WfDAgikixaNvezLKodAubvWIu97VjKg/gkRpLVNuKL4WXFEXmTWe8U3y4gBn7UG VjW/PZUyjaQSiJexKp2DUU7AN/Hcz5nawHX7WrJ3bmgd9wN1Qiej14f2P0cAqGqLbu5rpgzd vV4cE9ZcewnAuYg5ZMegd/92jaJJ8LM8fQ3c+A2vjiwdY4uFN8RBGbcVtLEPL1jNYJOPJBdC eFNf2IAUDS+GYYRtKmh4BMXfjJ9pUDqaYr4LXYOInY/1hm2FAnlHICWDwOs4gCN84PsBGSGc VLLpgW4sPEb8u1EkN/NduE6bNF9oDnPIhqkLC24+bPD+y0bQ/VvvZGQMbsFBhwf1KWcgqkjA ejhMJlXsbAjlMwkhA44V41AjbCJ06+j36svStsWWYdA7FcImLiDZmp3UGMAnWRjQ55nJzEW5 DovNjrV0IKBQYNFryGJgZvc6H0bfSOdg3Q7RmI1s+xgyK5gMUdvjKCnKaqH7JOm6KBRrCRcS w/02nAs38v172NpMJu45Dqicgqme2pIC6w7FIXRuW4Y49GY/1YmI7z/EcQ/qIL5YmnuScZ0l O0bfWN2JyK7lc62wajdB0wPlNEcS+F0vK5NmNz6bvy3lYi5AQXCXqrjX76DppADcOtxBbkWM S5uRJmee9mF8fBzaw5P3k7g3MHKwQy7SLS4yW9+fK64clkW6rbnlfTjdaJRhojIfh+uhSsJZ gd9qeRTyFsnbLmyAO+3Emyvk27NpM45kmCbZwW0mcrqSoduGBJvQ2+yBIHhSxdlQvqA/l81k 7cOiRK3LBcIAqCikpogQeMhX7+OypJ8kRHFAkBAUX+3S1C9tyXvagDjJIzWvt0rnrqxkIqsa wjqDeM6DGkawBw0MBLYgPstc8jr9rgpuTPXowmLDW80r/E+Z9qQQ0eNsNgB1SOzdIDNzkQxY axxsbeO+pGUsbfEWNxssInfyjjTmMkb2F4DOE5tzOLvC4GWdJby1ZCJZO8bXn39/EkCNb6tJ oAnC775BP3+t9jHJ8TNEZztjpHjtli6jLn3eYoNVoic+65+rx4JY8OsOsZeg2dWhT15Pi9PT q/FCMfmUk7ahgM+rMiDVG0b6DZRweEcgMQ/41LveXkr0+LytTAMtplz3x/ZWIwso9wmzlOjc /TAUi751pCFoxg7NlBQfb1XNT/1BBcjtSsPC84BglUIPuXNgF3Oe2fH4NW9e1xbvaVXYyXsT Nd0SsyCQp5RuTrQTLPYPpZPqu+CyZxNzlTA9uu3MIv3HwZORZChNxgou0HszGqlUNT/v8kFL YEuR4XugwGANx+2bES7bzbU6FQxvNSzH13eJ1koTTJfNYXUH54V7YX8nfzrVpU4kq2tzcFPk yXoaowqKcbDYoFs0Ip5CSFK+4H2q4e2ohKtF9BUl+b32WD4l3xobZ7IImRFrbYwvP4H6cFbz yKhOsJ/Ult8J72Vopk55FOQaGMUUjlD+oiT4J2lV0ynPR2Q3BQqe4ZndJNarrhbTzW1+Lb6C sEfFjzOIgP9Xa2w8F5M+9jHJpFGSWUq5pkwuuqsM7b3W8vfvHFOZKqwbq0RmZNkxDrhqoqDY zrQLSNFnx4mjMGKvjyI7phslxxt9OXsM4QYKjS+CsJiM5EHuOfYrW17prsV7cJCc/pU4SCtK 9THlwNjwA/1EM0beGA3RgJ8fFzbJwC9id52qec3LoA6zjjSoaxGQCgdgYhE3bMg9y9Nj9Z2M jNgJx2CCKPLKrdbBfczkA7Qd+sW76xhuEfHQwOEQFGlOHTFjlkq4VCANcWuqnNff6mzkE/W9 lnnVi9cUcCYgn2NGjyVAbPo4Di5VpDR10ew59weNjmGfh54Emcj7+L7MQJu9vVOz0AxUyizs ulZDwZGYMUGAIv+l2SkyYTxRctVdKfx7ru55IX2hVrifb/Bsv/EIm1eq4XpXXSwLIAmqqPqt EHjgXNWZFLM01ZsPUfM5ZujWdlJYSihje9EHAFyd7q4ZfH5RbXrZfeSsF01kCy/6Fm1tnkx3 8Qwl6P73Or5dfr+oyOmB3zmzSPgn9Kj+J9oz+cXi/ORmbb1mkavuQl+bKrJnlZ94pz4/2Fim plZmfaAdit1oRDaPT6cG+4d5VqYiyImSxradSo1Lg70RJ4eqQk9GKA/jGdr76AIxXkAjyO23 n02jpFEiqY6J+Y1pyxX+2ne2H92HTbwHJ1LuUDhtdYD4fxvtpG2Fv6vtYWQhzBk4gmitkfdS vG37r6st2hhfF9bT+cRurZrf+qGMtBQyerDltjw9nTMUUUOMzH1RfCC3ejpuonsdLi6Xg8ye aHU4Czfrkfwh/J46BTP38e+xdY9IQeSL9K1d3CsmyZW+RAPI7ZOwC4vp/3o16qqOAWjc6Fhg W17uGWQStMy3es/AZzbq6maqjGkEKAYI8iSdyeBx7Y+kW2PQ3pEVDoGciK5PYRJb1tT/Gm5C w6cY91rINde1tF3klBk0s+trn50ttzeFIf6gA2/yT15IXahQNNgRAKlZmhpQfFW9el+8AuhR KXidVT1orWOascZtfgnsqGWGUG3w4FCzimnFa4+8Fy8G1VvhjPTVJZp0WDa0W6rPgy7JOR80 QSch6u0Snph3YOwMjg4bjwIgRoQ3TWKiQWaHIo830oprMUSF7VW9JMa+FmYJcXtRd6zDjNnR GXuS2BGW4w4Vc6ReTZAUuini3cBq607f7lNNIhvigcquAZcBbBq+oiHebmaA+VIXKnzCCXR6 WVJjovUWJuYTgCVdShAkZupnTN76NVcvHXcVqyRp/h+Sb55K2D65z9QpouukeaMhDef1c7+Z JfMdphPzy5NEagYwX3NElDgQnBG2IsLukrVWKs7xFX701xijZScA3pecEDQ2pHGo9sqZStBM 2bNSMmMYlfiov27RSS7yOVAQW3D7Nwlbnv9DKu6EeCCENXkuQorKDXLlbcOf8mlDhGsSW/vE qVxpD+ejb7Vs1nyHaFtnJqypSZB4PkvPS5RRO6xJnNXGaf041Qz+13inXV064repPRmleost 00TLZ7HiNHSaFQvga0DIlWnlGCsjk1gZRtbPxFAVD19ZRoU9Z6lXiCmILgBECi6yD2UQo/Fi GbX62XnY1Yd/+BBxbQNdMFlmiIxUzwPI0c9Wj9u+xrLNC3GElhhgHBz6623Kb7H0pv1HQFH+ f+z9o1Wz07m+7odouw5erF97xCARREojoOVM/YJRw14l0/zhtGQGfLoYvrUylOv06L3n/41j XYJ+Gc3lYfs9Y3wlCHQabePSqaQYyDpE/hNZowWP32uaZmv0JyySBZVqbV0sufhSHo1eIzg0 ZU+qUCfY5hQDkbPbVw3F9kLqr+VFQeFlmeyw8NGqPYb/wCF4wnaOb8X+8tIdHXSwt54TStLX hEYV29Lsg1gtD1a5eMjsbL3TNF0JI/bVKzwSbtD1yOJDGnD2acneIV9bN8n10EXD72zR9EQ1 nIf5Yi3vX9OvX99ZNyVHNL+mZoQeRtAOpGavBRtxMXK2nmEvHHVoSzgiAV2ZEWaA5bHBbmj7 tVB78Gz8vx5mFJ3XqvpadbR8Ttiyr6yPWi6V0iOjm5mRAZC26Z8bglrj6Bd4v3vp5WmA918E rZ8UnKgy+HMWcU4rG5TKIi++6FJblZ/pJQgf8jzITE8VVcEDu/QO0ODLNGNwtkkmPLmxdL4g y4jVRtFvs17dEYdXOTbt67Rr8YLI9NGif7apeFrBRi8qFVupSj1wm5/9SOJvBHopx9k9UGp8 94omgCt24CBJgGKlCOWDeMGKuROBok6YHDnVqhbriJcgM9pwGQ3sCZnoAuMuqLH8eA6JTjPm llFevP4dN0dKgcERb17/mH37t/ztwksE8Afl1peub+g7PjUQWxXT3WgkgVahoo/Wknuq3pfV A1+taQUjpuuigqvNwIY6gnxCNxxqv27VLWEK4nylxnZsAyZHrFJ7/lAj2Me4FaadV4UZj5KZ HpkqSXz+DczseHGnQ4a+ElBAPIHD/mVCeolyzGIN7Ph8mzh1eQy9Kpnyg0M+h1hqBHy94luC KqDto8W/cqTcFfccQMQ+2isdRqCTqFshyL19zVc6cg/maPKc6nrWGx/bjmxm5nEgtvGxaQmn l87m54B6YJrNBx81VUxSTavHToCfvzUXpUY4LxTRuWJHifJ+uv9CkOeBSB1wHM5mm7SWHZ8I 3qYeCzKmcJulAD0wpv/i09+0SeaWtfQl2yV94E3c3fqg26c7JWWxhh8O0kwJySIvrhD4v+nN 9rUZjE/T/BfEDYW2AUC8bPD1/FIEHIq7drT1kLkfzyvOKbfMs5MS7QjYDWEFy0ASBAJtbcWR iIBCGfuTHgp9Y9vG6xYScf7s8ovjvR1D26nWvBZPcgPk9Pz/gInIddgR5sEVg4uiGcYFF0CK /JfLzW2bW9QEAZxSNvLSag1xzg4+gfFa6bBsuIbPq+OyHHsjWOAJoYW9UWOA81dk630iEZNC Vtn30vF9hu3rbfS3mu6/Z8pf9lGFQ4+UWvZC/GTzA77ewJFv45+cQx+cqcz1UVvEwcxNiDcK OKtKpuK2V7F1CbcojJMS7o7A0mzAJ2iY2cv0SS1xigzjjjvBlIYOdcCbclL/3QXtgseDuIxm Ed/Pe/vcfIlL4mlxlPQUp8mc/b5ghWD7fHce3CHeqzYOzJZ/Pw8+45Rh9JGX9XxObt1hqYum hd0JpHFEO5DZwliQUnMHlaoZKu3LLSovPKCeS7JSYFdaimCZ9dldboj4E+BPJRzW+YrYCvZR R3PUl37TNOVdp519Ha46ySLLnnU+JW40/En2QgD8YmtZ2+8RhLaMi5YGK6O66bTyWKxRqCHR p/ez1Kd4PBPjN74uvW0N6zLnoG1uQ9rKY+Dr1cLVuxcBRB0Nrzr3guSvhu3Q01fROLPOlIXM OzkIo0jmNwZS/vyntbDSTaQ05XZWyJc0mJFjMt3IO7Mfx2CE+OG43QUqKuGPQPzT4yOjOmSr zV2XVHDpgYBMGkthDUKVCcV7C9Sa6la+MEWukbedSD7AmoFRPA3dD/iJcGloriytE7g8OEbt nOLR+5au9D94NsU3ipixqHzIreo8p9jtKxayWfaNAszz/9b2nanEMvXmKsS1SsrVQvojKNi2 9Ndc4RhKP8pLS749Ih1dxs+YeLeVRTRjz5hMtBnNxs0D2tW/7J4+aso2yHp9BMBmnakCGLFs z2hMaeyMqRkAo8i2NxeOhzfCtY1fVoD9oyzFRSZo7Gq6zuQuA3y0unbNcTqqTs5ZFij0+x+n OxMHyr/XkiagqimEm9r5pU3ZdQMoV0Rd07agYPzkxry3ixgszr8ToAhOMjP6ucwmwTn7s1nw GAdK+Vb9HE/6HrlUqDcC+/H7olvrWK4Y5HPObCG4BUct3JvagI1vgb6WJH5IYZ4Bs0MweQ7o p85kG0MwhHdZ20BWfX+9O+T5Gn7n+cvHZNR2wThBxQmxkdPnHhMd1NG20IkRPrcgo+mnHlUv S8wkTYsuebh3OnKVi2dyRgTMnvrETxrNzqWvVNea4yA0fYPfrJLXPZ84LWTci40NrHtaxfUB ItF1Cb2dco2D5Uu/VCPw+F1AfGchXHdOBGf0Qh5YUZYQ1kfUzMlEzUJLmFx8+B+s/rCuultV F84Eq1DZGDA6t0NDy0CHJl9L+SXFJENCZAzxxAUCm2V8iP/RD9z5SxdQhviZToSTgMRizOoe Jj/uMke9FE4VSih/lP1rgAFr9LjW7pf3uWGmq1g8CmhQb2SAGivbFiYkUrNE954dKOW8+Te0 jXzfp2/6ANVfPk+lN1ol97hinqvOB4J2Ua75MUWSFL9P+hSSHMaDY1oYms+0PYC2sempnr0x Ns8A2jQsngzAHHW17/wHyUqKc/cZWuilWRL4z8WzDlz14BGDFL2q/BDNoLtRYEwtaEDkBa9+ zNQgrranCyfKCBf2HaamvnHyAfissrFd/5racy3Pxrn0dL1+LozlWKjQU2Fw/24JTGZuyJFd ouTBHXIbWgP1qilMZvaGuO+BoXPWv236tm6yYdoD+bjk0gAJCk+iJ2cwO8eJUyupZeALULrj bLrOlXoJwxssSaCjMTgL/GfzlD1L28lNK+Le5ZJF/KMNpTUex4a8iPYJ8sF9BdhnRz/y2buh soARn8WVFZ4FSOE0Gj32dROBdqXVgtbvLHsk2vArjsmqUqlcKiQs1/dt96cGe7hQN0CHFPUq pei2eLxZfrrJ4Ndcy15hifgwKBSAJMiXm4IHkbJ7/Rxlpb5ee1tEmtgum8kNRuLqc46gjhE0 OZasP+oVGCKdl+rkN6B5PSDIaq9lEWbZ9N4c8SRwY/Z8hp8e6JEGeO1sOwKRqT9WB4liw1ce DDI9csY0QVPPhB/IyoXhX+bhjV1yGg2f54mLXlzvz3k6yu+9zkPIolxWdb7rG5kWGzNuatmi nP8mURji5LYaCCFFZS4ZsAT1ZHEOk0lGtAiuH2JdZ0QaB8LTdCF68KR3IJU4DD28lxj9+A5N FhQ4nhSq15DG4bXw0gNFQBl5pDooBCab5hUD5kAFauRk69yNu+pMAVXwdyjtc/P+qB3wJ4DC tVCMd0eOqm5Ql82qe2cMdsjstlaZHmWgEIgz1raiqjFRa8kGZ/cJqisLTY3fRC4Wv9jJHow/ at/rl6BSvkzh9uUm77xCdPyC8+j6mYkrLGzvQTJCk2uMj4uLYtpMcJ4UwTHJ6wIDZgYbvgNp BVJFG1pGvyB8AUhBnT+RBlZ8mPlQ2Rv92QnZmNIOO50e0A2JYZ8+RTujAWp8X7DS6BiR9g+4 kfSVS1qjfv/y77jmqoHJNK+rhXjJnrTO9WRFu40syCm+iGEPeXW2VOfjtebyRnlAf1hAQpiK bzF8qeigATg6GW1wQupxZLfvLO5qmApwmMNFDT1TxaJMe7XbHXgO2wprEnD7XeuTeO0Y6STT uw8EfjHFR4Nk0FuXj+DupoqX7c9KrWE7sPXk//9TWIHBpKEQBONWe1DDR3is0KVPt25kFcl3 gDUIt1xY13Qd70BeceARVPxrtFOU35eoBOTvr/wcsBAlxPyNuctHMA6Oq/9SWl8+JUOV6BX0 fZwHUycRKuESZzru5qCWVmG7jc4pKApcdnq94vqG4UcqU3F6CnV7w1ZvomCx/QlNQiFvLVlG i2dGAma1AV21xUFlYzQ60IULp/Bxe4OORvDd38BnSMxFBvMjV6GpLbVKzULc5cPNSx7TIhCI ZVbAhKrSYqu79fRg2FdqDF5t8cCbXc+Fr83njUNN9ERh3R7jn2sZU7GvzOsYmXi1A2XFoS2N wbOG7ZfqNfbGaao8yW0LdEABD0YYtZzp5dc1Cab2pTHZoKHXeZR96oa6tPaC2ZfIL8HZ6ZZI Uml6opVi89nAw1BRjlNuTym0mf1blb++G7qp0lVaod9l3OGU3wA8eHGz6f1Tq6oCUdVsEk59 Mc2gjlFLTNvX8YwuypyCpH/wUyBJXOtCcFD1d53mbXUM7s9x3XNeCa/MOJ3VtUe0L0y/ihli A8MVqIgU9FAqby0E+T40jHgPF3nYHn+eVpDOMjH/5d4kbsKfGtBkx9fFbUZ4HW6JvdfKW+Zh jffWp2gwabmfFf48Lb2lW4PXwkPT5SDS7/G2dFwd8yUvTv+HMokLWfNJ+CwS/xiNWjKZ3ESK qZINDFPA5PrfFhOsCMjuNAw02sFGlW9mIl8I40mB9+L0pamp0iTPZGFUhZ4kPPid1+Ji0gzd RJYToFjiOgmZ+c7nbrKEQ+8BGHbbIyr70gG5yvV6n7WPQutIhZSdjKnE8Y56ijn7FtmB/LmO F57i6wnleRO0N4IdLxC0s/1BBeoPBexcmoc5SS+HRiDIzOqF+n86GRzOsWp74Y/HEUAv/bNI 3CoZcMTk1YkYGs3+3MtaLGQxFLEXmT4H5Iru7eTp4du9voLPWMhvtvaZr+9Wa0KK5fVxmC0m Kl+A9/R71XEWL3b2VVQNdpEh+QDLW8I6ILKr8rr1LK1l3zU9FWUPhJz7Shp3jzQvIVc8ba5b fCiCtCb7cHhIwqyOs7MYOJ6yoU4v6XlCtguR+sKFvySwNnYGqU4C11piy3JJvWaVLREZs6Oy oASQQsyYo3u9Cw2yf3pe9o6359gsMHY832qXgL8N1qT9pZzODnUymlE4945BKwyMY2UoOUSw zTPSX7+CYuw8yE7+efFdxx9GRxIZQP0FPtLqKtMjlwPS2AO8x1ZIofTP1PNI+tPQFIpoTnkO biNG+U7ylM6s4h8VvgXpNfgbSodQuJLxV/wzJs0bpesl2QnWiMj1l2PrZjych7WzKP7fOffu YscdvKKbccNjyYdG8dBUrrcrVx6YLRrMHMFEyfolk7IebHL5ktuqRdD6mreQ5qySGlfN0aMZ FwAAfOuof0aSUlvq4xvzyd85z4psrt19U2JptkxM+40sNA7oMLkrrhBxwgNhEnHXBCEeEsKF G5QT4oF9S7OFJMXwNAKq7RLAdi5jzKK0jc0FMeg0/lzefXuEPUKYMg4scGuFw4o/TB66b5ew /ZE7CZ1SfJdyd3Np3bvXEYu82Ld48zQ6uU6w9P1ENgD7HPpSI9LaGCvnK553j8HhLpTpmfj7 gZ7OsCXeZyZFaG1BbOH5m1/OdNcIf6bgJAKqqTePlj6biFjiGMvnRnzvqWta/I1T/2Qm0Hfr DY5nqKshtcajnPPAEudK5Mr2Ez/GiYbx4Em3MzdMF3i+sbateLFncmc3u/Nh3UjXTAC9YR9b NDUSNiUFbQmPWTZZIvVX/j2DlEevwcJGVea0hEDu1zSp5GjCbI7FLAvqjBCQUHmX5UgugnCq vSgwnIc+pIWHz8U157TZ5EHX58QkXs5qXYwbQB/3EjDkK3HHSHFkOCICY7Lefv1+sed4+fZU 3W9k3j8NWOuWBYrGrhrT6lO9F17kXvurL/U6VEr5ZAxuICZpGWu6XVpwfexiwZpGkGmHWc3J WJ4q8n5JtkQqK3O1C7DXVJ80X5DfqXywnG4aghPMxy3BHMXx9L9CEJqd0UyajEs+HoLYxb8j fusPnopLJrANJzZpoBqsEohMUZAVbSoVe9gS6S9Li7w8lSMIdIUHX2+XdzbLVqvUqC8kFrSI X20ZmxntxkCoTKAqGGVpCo1YrvR0gfsfNquCVm5EaqL5weOyM3rhlBG8QhMt/RJoyceU7OaH VB8BOfpwByOOvB3tdek9dtspOq5BFG7K7bGMxuHt6MTO4ZP/bsby6fKnLPx6mt9HAy4rhsoU MII9JkmQqUzywBhl8TVuk6DPAPoOpAwxMUm8yleaq8UVs3OYA2sSFzephT+/K8SKfNvSlbRX CoRrsjRqRbweCt+M/Wc7seB1muFb8IRy6IrHBd9tsy6evZZUkfdDkr4e6oCU4xRuu8SqeTB7 srzwLv9FwfNVB96hk2dKycFpqs/dFt2F/PpYI7oTl4h8m0RRUPQdkmoKBh4HuKIdMGmQXsUc G7/68JumfqcWLgE2t5dDyZ/JLmR2PIKF+j24kZmwY7hVZIvnL+jF6Wb18xx4gVhSG9jpgrJR vb8TWLPYtszZwqI0YQnSv7CHxIiVXag6KgNS4//3bKo33/fYHpvr6/V20lj1ONB8zuhLdKk6 LMsW8y4aO5hnI7PBAMUpzsbSaXEbbpoAcV8G7FqaS5o5amJH08FYHjBIaekU5Id+STYNa2Pe uRA4Tc7Fjg7aqSA4UCT7CSftlZWha4K9sB3NfVnMzRWEZrLuyrT+YLJB3f7HvEV4OsgfNMIa D542soB1ly7zTjzzQW/DQzqHA6nDreTXW2Ce7Gc+iMlHv3w7Em2XaoTLV6Kf/McjEDvKvg9X qA7zoQKVmW+FXXAczxbR1i90+7BzGMDr+hCd8tH8PGDDSLGbRmoaMoa77hF5xejIxJbO84XF YxSQMYMGKbeu/livgTSNLokDT1bzOwh+JPRo3PSh885DJnZ4k3fOL/3ppK7QONLDcQHVgntZ n2pqskc6rfFgmhoh6uwSwP9jg+XYniY6TIsZxLGsFLvEJuO3YHGBi0h9/G1VmvDCwvt3tVVJ ntXXCqOmVE+hyWYHeJG2FpjUZNXxaDrtjQVO6hPEeQk9d42RX6hJj7jQRjnFdPdm+uzhc0vS SmZszNOdxDDYxMbi6+xxmvptkTAGntQn9IqFo9Vx+D8Rt24JueV6hdfLRkPnolyQH4Qojp37 r0LWbLCV+JfLnPAxr9D3YcmpHFwF3PpClfwnDmqU/54LoKn8rN4e5DhOziUE/GezwlMWoUR5 lFs5bnJIugNNrzMGDmDFsiTwbLhG9qER6Kv2RugC/FZy7PP4C756JW0icgk2QR0dMm58cySI J8YfLZVbu4jeQC5m003QIdQAQvdfvc7BcPFSVKTzOgsuYE+o2CxkPoe1cbQvQP87ipEAXYEP 5R01O1LEiMsYbHyKCgo8AifQAQGqrz23wvjBCsgeswYK7SsrbrzjguRWJ72KpQQ8EAaiRBrc EFj3TcsFzHp005wlEFEwm4wz7F+NTpxr7Pg/dZUrKtneE/r8wwgT67sdiIZZOdNgOnGY3J8B RjNKDUWR8xIKNPto2GXttJBQL2/wyLHn7ExjBTh3VEqwE+zMakjccbQzHA7coCUoXhE6dW6o z+7JWbJ15QgXV05ipkbvTvEAXaLe5/Oez1h/+htNPLgDz7zULO39CPvKdW2NgpYTWbIFQkeT RFgpNccxu1JeB0fKiYJaal/WA1t37V8J/MzqGgd0zg203yT5NDGrLTC/nf3SbiobYE/VoHsu 0zUj+qiX3LEkLQBOqFgKa1IqT/SHS7buije5t4x0irnUqHHI7k+GTgQyq7Vx57QNn2EgaCYd 0LbqxU1GUnwBo12nHoqBDzC+QxREKKe9YxnCRbE85TnmMwoBwU3c27dy9t8vgU54lQ7UfTw+ ElM81gDJ1XVL4Qcp5qWii4xDvgQEHzFK3TnrVfH4FXwabMofrPO1KbdQm7DUKmN2mhGyCtC/ rqbI/tHt+BXFDU0wGJwsXROITBrOUSJTT0Lg31xKWF5zB90BjxUnFNEhBL1lCHlvwGRbhqqF XGq/2KeVwV1WVOfzOrc2KBymkJ0nqguCbN+6S40HU+piGzZvp/Gwh324+Hv+dVKsIZO/oOSF blaaN05eh7lB4D85Yw8UKa5t+JpKfW0nHlwOJlOk3CHBiuKLqMwhw8f6/K9ay6KDsfmbC3LA UoQegXDINA2+25QA98toYPGb8o36Ad3nDKdQgSqaYXpT+VubUSlC/SrcJfo3t8+eEiLqFTSO exK5+j/j87m2jJ7UI0J+JgjRUMmpMzNfGRhciluxvuDgWx3aawN3+OJlk6DgzOjAq8yzeeFd 9AdHy43PWUb6XSInSrkXj6S45eC4GvsLedqmqRPjji8DqBYBvek/fUGJr7SkelA+wZwgVP9J /uDJvqCgQZBgZoabRLrTQzQlC38pgWpsvn7Sw/OH5sKMPVwywHS69lCw9csGc96vZIQfZIgu /GeYCzliSWDx/HxkmteVgVf7bb3wh8fvlRZ8SoBx671jJNUU3ivxeTjPslEY5wSJ2YkaA5rp v9wNTVCxxgLSDASTCRTsdBy+Cs5ID31AmQzvzD+3VgaflZ2YrsIZ4AqT4fJDj3RyVPRPBmUB hHz9DMwxZhDpqcVJMIv2UBpD1xr624xDr6pXdId3PTHG39bjSO92ojuu44hGPevrkLVvDmGm CiKGx9Sj34ryHyPUfTAPjJ9hkk8RmWxXJMAenXakVHLJDboG3JJF6vLKKKqx2NIKGNKfTB6s h0+iZIFfyW2ul2YTAZjSFF5AL0FNH/EabsXMlfxPYSlQ2u4LG1do/ejQimDXoWfybNcDtiKJ Lrwi+77w/ahI8dwUpaQFpbF7c4I29iFtH6GOk7PGe8xRhxXIZw7nscf98goRcWBfES5OkUMe JkAmxXkycLVNQDaxBOJ2ziOVePwWKMTYiQflkgj8M6kiOumV5BOK/xe0L7Pwo2Jqn5m1FAEa QB1Drk3XOkd2kSFf42QSlYyWAl7C1rJWsZdhX1dWIS/DjzeBFq3bzOCClwgzwiXRN+VYIMc2 dcY80QBV4V54eDYZS4A0u8cNnqPL1tLbvREv0nVbTKBe0jwddhL8yXKSauUtA8qA+HX+oFQw rHpyPTvbIICxkYyukP6r7AFxIXHDMjOo5oe0rT+ZCGTd53XMjzkfhMbUX1uUUEAxw3R3PBYp 9tDKzKWA2GCQ+fA0FjHvtQaaIKwYsC/9tX2RECRtg7q4APouvjzUPOXrFCWiu4GmQrMZXcWE BvCt9h7f3WXS//1ELaIhxNUv5Q2AgIT3fid18DimIx3vjMMrD7+wSGF+4wgxS3Su+PYFoABT NWso2i2RGMOUm0P6wVFelkiul3AUWYEsAcZT9YlfsoPZpQIdRrzj9BrkDLCC6NO9V0ED8vqM XAtGq7/YfJIA9Ve3o3FWE9+NYYLd64vDWR5KqLD3xp3pI+wWvNQ9P104aCR+VhICfZY3LCyP veBU3VmzJKQHd+f/TiuCS84RYcr+EVR9pFuwEhJRnhAGK8J2KX0csKdPELMK5y1SgmAWqFQt h9qn1LJo45ZsNGCITbOuJlE/I2y/WaKX1yCfFZg5NDZxpUOvtViO5npKNgYLZ+/Tw5pvkhyo 5oR0rG6doFMZOuMu/4U2nWJ2NOYoFM5DpsvWTYFyGtSU73I7t11TjMgL6MqwqbTe5Ri++bGu ieoWNBDDemwoOYc8Zvm5S1dWzpbCGIDZAELTfGBI287DBbPICp7A7uAehoTBaUkEw+kGPZFI aWZ3qVZIjFRnVILxmpiHqWNklh0PDHi7rt0ZhPGhb3u1re0UKJh6fvg2W8e5bd9PQ2S2QyHA YTKOQ4TWQUF0TyEiD4jmKTdnhiWjLeLrIhECXrFlN/mAIbHBXZ5ysQIFro2kQKo4F2bsVTHU mz8h9tlCrZYrV8cFqdf6Zd1+Xer9RhVvAZO8XhXGoFFe7vH8evygmmT23jHrqSGpk/BaxYyT c3S/QN8konU6SYKoRWNk/bZuvTEEWcc4AFUUHDKm/g6sRhoJJRYtGKcJcNFiDKS86Y8fWYca dZCAHXKv2A+RSx3vmGo4BZ4bv3dBIHK3kbtJ0QY97RWI2cvvDMuneSosON3MPrCCiga4r9JE IRBpPfzEralt58HDoiBr544igCZ129wbHVNDPDubG9ZUN0qG4dCsB5ZLpGU7QYNItY3cgoZK NZKmS8MXWbv00UgZKFrl0yR2l2JwlBRr862yXTrjAPw5QflIwe4eiyVZfKOnvmRhN5RIS5C+ KpnY3n8nCjdUXcpmuWjKIdJfBPD0C/FPR35DbetpAvXzopu1rgrF4IvdSdKJXaf8ZfsTkHVw GaFgIq97T4Lf3PCIF0owQnga6Nyiyzoq9IJGNupFBGl5Zlo4Sudqnlo2Ny92vgSgtwggf6Nm L5bH5G98rxZAIuuvSTrhX3jAydG22wxXkshrbRL3EhryfBInqHinNYd1CqNxmRkU6eWyXcdw tz5TQaZKn2g7eIYuXrhqmDhwP7SFOhSWKd6dTPFg4vpGl6wDHfd1Hw1Dewn5Lktv71v/nEr0 9jYMMHq8MQpct2gZIIyXFNg1ncDjb18pEfWmRpoOkDadDJzc4PM1ViAPFGPatcIqX28k+w1L yIbxoiXMkPReEbBD7YbMTpCiM/6Tg01SeZ4EkFjounck/fgVjkn2XD9WO5Yp+8adwn2fV91T DyquRKXzH7zzW/BThACB8QYfsoV/LX6sjfGYlbei66/xL2ZqHIcJ8mkW0Itf6W9GDous5NVe f//wHqw/jDODL9IwexSFUgcyDg3JzkrNa/Hp4drW/cch5YpVdTFmHrYl1KLNwizS3gHGnP1A lsAEeYJCiYWrAgm/4NHi2wYhFQkGEeNOAQp0yHm0ciAjSZVyVquGGgg+oKbfevpWfUYuhMbF XcL66RMiELNwRPH8QMA+RNatpxn12biOfDwgjWTbjpu6cht2bDCMh8Q9k1x7jimctcjBiRqd URrw3XTbRgNFLTrgrSIYLP+c/bcHNwzOP+gS5yMOS0BRJ013DXyTnw3dNeZYlT43h2xaEE9/ A/xksIgyCv9/U4UnAky5lFB5yt/Ruj6jJ4V31XEnyqXZrhbk2ICZniWRxIw4BvOQ4xh3VkS5 k4kNRuvcmgcLsC9sGrcNhHzv5+E9UV8Un547p8GM0wALn9ebwx0o4bC+r4kw3MGwQPRSwXHB wWz+JY2VnEjNicOrFIndCMDpWd+BNPTAWG4Fl6OsEqahYIhcxJVPlGlWJ6G70v42PZ2XesMY n/6473PhNDpBZ/Dx5SasVsTlUOoUhrUfvT3jyBUaEbSpEabJR6A6hk19QzdHI2T4VKN8EqtV 7FCZIY0PQyDwEKJWdyEt5kmarLKfOnYIruleWP1D3ENfrI0H59vbaloGT290NK1jZec57l4n FAI8jgxhR4q4U/FfxJtoHeUNwMpFp78fRMAgygCPWUhtfN35dh0RSaIcv4WmtAF6KFPUqvxG Z64LxQTDtMh2gdOyXovVzfSi9CVCL9xX+rmPgyTfJ7p6S8iqIxRBfr71p4tivqlewNTABdur bMcu/ajkieVmEHPrAv5xnCqHD23JF8enueTGS+zFdFF1XYIfPE0wGVPCnMq0MqQ9bahNb3pm bvlBB+qZZ0EDDuj9U0pu46pTcREfMZsEo54h1/3ljfbLpdQodgOL8jFdJGqPZI8Xg/tYWT1i JpFIzAjoDLMGlAY+N4jbgsq8vU3MjRoaJdRhDk43wRGjDNevmuRAfiNjCwH5FJ4JDWbxVM1Y 5mMmKYiNOTllBGzvmh9gYIDQ102qB71qPt6lniwCR3rue+FFi4v3ch8Jo8y4PP866Ud8G4Po W8A1fBRJY0uWeV2jr7fhnq4xWuQ+5I8gFIesMDuHGAbJT9TdT9OdaEQEkxPGVOZRFDVWKbqE viHCWSDkGSMmhzIuGcfwiGTara8zWnXHlDxaf7ALEHhTp7Xnvpyo2ok4/yVcvfcm6xAP3fqe 5zDzB66Dspd02sZNrcl38QeaJF7Mmd5oWz2B0Aufl1ubfegDgQDoCed5x87ZCCqJt9OzwfIo X7Y3b4hKX1r818CAXUyGF5mrSjxqZhmwWo8bTFutbLv2i9GWNVGLvYmGE8NdIT0//YtOkOi7 1I2Qq2I7a/r1JbbIAflnLf8r/rSAelhF0TMBSY8qJX35xuGOX/BMWHKo16Nsnioegsx+PaZ/ 2ps4QvhiEm6EdQzQXD9HTAAFeqEiKsTHEBHduJjzxFhVR5k1l+DRLlx0Mbb/uSqaTyAuCOBh rv42W+Ia5M9htNmsv025C8eNlCIPN8AdKTeQGQLoEyr5uYCqR9doIoUpbtM8wnnOL7GD3vCh icd4C8Rw1DnFqI3oKF1VN8mMGQbPLZOW813N2AXowaBGQ4cf6lYJ/dFQrdSWjVjFTeLLnIvR s7Kh/22vzCyCgFmiQwlTJGNAXWjnAKrEm4727TZPnTVb4KYHEz7PHxbzXaqgJHGyyanPg/FB GQ0dxlFWw+n1ZimjORDE7nbxNRB593z95ujIKNVMZVJWKgWXFDgasndhJ0NSsuUqrTg4IV8+ EXh4krrjO/O2LGq4Dhwu6FJv5ezWRfGlXwjs8NnYQ2Mb+wpCB2pmo6X8Gy0hW7AbXbRvHISQ m/eRakEoz6Dn6pcbdAcSegXKtdzXrMjTxi04wGxOozubS5yv8bPuKM34PQB28ArvccpNCXJq 5Y4EPtk0SKohSUZIAo+7EaUjI89OtI8iDUntHe4KbRCEgp6pB+yBziAeO26EEbYVjng1BHPT yi3P0n0LmU6hYOYtGuoRpup6uCZ0svWzYJKbaUQ3eEj3NJ50Sn0ZO9l62NtgI4f+3b77arEf HNmZ+c4GdwuPv75YZC4ybvl/oWG7fhWwo3kfsEmJvH7UTuMQPUOMskTMmDXKagz9VWfhKqaz NgH8D72IBwU/sFDS08buU0H5pOoHdi8aTt0EgMMD7I6kMN+zB9tZCpurx6Yq6LMPfZDDW/Sv 9FMhH7cj6yxRD0FpFBwuat50C9yYKp2vvGUJuwQWSAO0z27tgYJ7yZqsXh1B59ujHGrKIRM5 lcrRAgluWjYc0KWMG+LmeqMvk4H6rSlQ/CeQxiBkgeg8lOQmBAXgWCZoCQ835P78zHL4gZmD ZbBa6uT3pwjON7P1ZIarGy2l8VdqBe3CLOqzxeWe3W1/77V7Hk5ovZCnZjhoxhZkedf8TMKa fGNAHUporBBKfANGqpbGm6WTfzYNYHAJMPg7avF0JvmL0t/F7r07h7OQhxkRKaPtjYzNzFcc +OSrR3eR2NjGIeQsteAoOnce6otz/U0TcUSD0dyGnv6fL70QChWPb9c6zwBBV3AgNF6tGpFk /ZJdR+yjbLq+RIi+dLiEOswIq4GABeyRG/jJvxUbFzKj/DXktSM4JhMiYoWN8ZvS2mxaQRJg QiRO4UGpD6heegeLei44VB2AJuKPbt6mHafH7yGnk+44o2eSwwG/eASakqj6zTHjExrFbJyV k4mJUxLaUV7QJytDsB407ubKOWW/YAgmbqiBlELcLp89hVDxIjbQ30C9eaDb53IazUsiJ3J7 9sC44sX3inH3OaCMLo8vpfm9CqeU5oVGRZQRE+FS86UV93KqVznGs5euKeHFBtlMKJczGI6u F45IxIjQODYJ+uekcEe97ifHDJdvUM7hauVEpo9SBGvtw4fbHh71Q7r6ywWqcE5u04ZYhhY/ MgUYQnV2f5aKanpP3r+I/Ll6+qHCFaGJhREZksAhwjsez4mtwqgWi3LiyuIS4UmQKrYlxiuD Qp7mY/2IRjfLiZxDL6kVwYbsLle/UttAAbhyNU93kzQqhDZLCxhg/DYS/CGXCAwpYzHSI5Rk PWJvcBRafs16LLiKN6lFCuMKq2vXjKkWNc9eQYMDfMX+Xvq+xsQ+bymCCbyyr1f7DAnSjzNH Sr8723xOapIWaHW4F5dluhVrAEHhduTXUxFIU76wpCqLkLdu5igjrgLV1Z4lVxwM4DF3XCu/ dZZZL2CUPGZeqhhN8nDm2RXt61iWNbUOxvFj7/QpvpYnuEuagYBP94vHOv0VGC/5Y5gkXtqU J71IXW6WqC2Fw6TFzKgVZ+17+f03/r31dNTPHlf8FctPNbMGA3h4EmIl2grLWEwgVKBktEKm B3I/P7FCgg88iBGTawFUtVyS+gYiqyrgsnSDxqc1WB+FPvJeyTKi5JNzegdiroNAMT6VmvIx AH7NOdcaZ6homGo6LETMXnaSb2Af/poy6BY5au1daRO1Yi4dxbBI08ycTlR5tXoHOqArv/Ob Lv+wQKevIIV62f6tLH7m+FubXLtn3r5Qe/XR2PQPjYvQ3w8wLzfJm2Pqqx8JOT/RcUPcy/Tl J70QDScMprLqcnXDQOAxVZuJOTLuO+Y2uFayOrdlwjO6SvzEMnxhr8lPlqQNho0jETHaGkaz OQk8RBMEgNBwFWU87vB+04u80rS7ru0oRtslTzwpxG+YX9gujLXyzUfwop5F5PM8m24PadyQ 2HzYcg5Ce5Yit19dqyi0UHMmY3LtfdjWvsci+A3miSI9k+a/OWHOgdg8UydJgCnA4R1GLoGl F5X9uKtf3GwxXSxICwZene0JkW9g4FoKaRttGJcVHA96cJ6dxCAvhtFQdddbKySV9sf1RmKv y/jj/8NX3NmSLQPwD6ITIHBVmuqXvOjkEuPnWtASouwFnnW1jFfA+VmpSEjwD4zT5mVD3Brm OhsMHP1cSM06DwnpkEXvgrtuuLzrMDkeetKcT5WedsuwNhZWbHmqpjip73CtRexcKGBRYPAP 5vawjBDeOHiGnP9Fuk1J+sYmjnZgH7OKYPJoKl51CYg0szLX0erFPpKShqpnTgW9Wt1pqYw+ tmPSL5CbTntTqqgXIb4KyjUxV4tAkozbbX3QmFUNgquEeJAXzGPW+4mKvRFiN5m9P+OfEyT6 eFWUHtpNXDegfD+oEz6IwII4s8AQ+n22L72+X1vCGdpxwT0cyZbo3uEds25AIMqHXaLyQkqH CMqlgK1FffOpfTXOCnrImUqPVgUb2VX62KOOiW9uAXUSXk3Hpzid+LRr1DZ5asv6CScjOlgK raRb3c/syz88Yyfx2Y6PqquhjFUnrQXXnKL1baVEbMnHja1a+BEoPKst0J3Fb7UV08AVgTZm RIwF603OfWASTiUmYXmn1rTPtF+/pHaZnpM/3N0bT4JB3koxbKUGRcK7+7Daqnc4sfZqDGiY Rk+ZxeMHM7QyRz+Z5cf+Yool1Qcxl/eowk4Tf1OTi1cNKMhrkhnFIpSwB/OntYCRebQGuF6A aKC3dLxszwDDvAi7xK5sJso9WprsNsy1yaJUsmvmC7uLpImTI2HAhOEYeknKsOt4Cz6RO3Nr o+NiZar6EfO41IQcVYu+P6MNqDNajUWwHuXOoR3J3o5oOesp0qoVj4l+CDIf5eOFI8XvBjNO WAE7OiKbY4QzmxfANFATIRsggJuRnuMsyd7l+WgwKGESx8YwN34O8QetXaRpEfxBvIiy8M6t P4uf+FqBP1nAFZvW2md6qu3VHMfIAKwdxErzT13WnrFMHoqVTLF1TI0GTxVg1ual/g1eufYU Ys1wRYRkEXERJ+xe01pVG1n3cLu9H3MI9+mqG9CeGqEbQSluYVnHxC7a7/fu0dk1+Bgud+4M UliBjh7TMtH3S8Ix/sxrz6KFx6LJD4nFpxVJHwS0gL4fMJnumb2zA61Kx68gYDO7eeszSCdz E0Sek7brbUIcDej+UxgPeVfx7XLGxoK4yv30WibwMFrqWMEUoE17TRJG6fZtoB/REHC7qodw OYxdPb2lAzhu4dwHLGLEmAHHTH478ujdvgo93BnUBUOboLJVGRj+1IH+w0v08BEx2X7P8V+w sqeLt3egOowP58mh6JWS1arbVdb576yfFh2rGNNfD1hPWUKXgtZCb6dnR8X/dxft9tQExfgT QFlOHY5d+076HfnI2lXujCdiLznyii1A2DyiG+WGwLAgBf8RagOQjdnc6VLgzs8lsHeWYTz6 cEwV9X6dr8stATF6qk642etGr+EaT1BXzzGvZKacBVRlYBLEPG0MlRbf7LqPJWW5nSM6MJlN 9Vi0qW9OLW6LX/IAeUQb5uCkRFBQsMkqO+F5sA1Daw1sg03X09kCKKUUyK1d0n5N5ntCdT9g iKH2jJDRXuU1qXZGgtuscigSNvR9yrZ3jawci8czDGJ5+FDVvmEznllKh4CWgecS8iqsHVfs x6Y5co6eSxoLq1/3TlS7El/68oZkuhofLdjtOKLYhJH0r4t6HHkLCiB7ZD0gkC/vWHkiHu6L i24SCPDULYmBLHCpbeH//qXWfxrXxlI7DaPMFth2NpynNwTE8TUXQblRJXJc2C8LnEaiVhh5 Vj7DMBBG9l/x2FlFyGuMHgmJJklJ4pYMMCA8a2D87cecI3YGb40yRswdG4f7RJHqaEX95MZX +Z1Z7qEP2b6AGTwXn4D8Eo5GilZXWMhc6rtV0D6mtpwkqi0mQ4u46klCkSghSEnX1mwwA7eT NThucK5lM4kplBL5Sr+5faGhIqbuPpHUGdU6b4ntPzG82smfzX+Yby+8Tx9q5Ur6W982+4ht obsgs4k5zPA/991krmi1GCAKqdcQNRbrHn2LSOx5UwP7RMGq36oGk9WFW2nj4xRlp+zlvQXw lbKepx4Lu2Vn9eEXkP99HsNSqcxkJUchviENN47xEaGRJJzuVesIf4vMpVrRlj2auzFHz0Pz z9YvuM0a/hGN+h/iRgGEpC5OPqOi5lPLBPkdGLHAPsVid8mP5gjf79tg6Z5+8y/Tp3I2HXgL D2QRr+rTBrop+CLr+FusrxAxiXGLa8oUKMcppEkNd6B7wRcEYhCbzPe7NPsS7sQAMnFH2jNX kIXfNyySuh6bAAEu0mzoXsurVN4nzZrvi0Im8xh/ZxxZRGg74POhlkXIfaSk0hQ7czYk47qO DDu+1vj0j2WPQWVY00y57nC+kY9L/4TIe2SbnLBMCTKfb7vytvjwL8Cu/FAvUODEs/6dE01s H7KEM5VKueykBHJmCSsnSDHAQO2IV+k8djqb/XFpQ1GJhgsGAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAACAAMAAAAgAACADgAAAPAAAIAAAAAAAAAAAAAAAAAAAAQAAQAAAFAAAIACAAAAeAAAgAMA AACgAACABAAAAMgAAIAAAAAAAAAAAAAAAAAAAAEAAAAAAGgAAAAwEQEAKAEAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAABAAAAAACQAAAAWBIBAGgFAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AQAAAAAAuAAAAMAXAQDoAgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEAAAAAAOAAAACoGgEA qAgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAEAAAAIAQCAAAAAAAAAAAAAAAAAAAABAAAA AAAgAQAAUCMBAD4AAAAAAAAAAAAAACgAAAAQAAAAIAAAAAEABAAAAAAAwAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAACAAACAAAAAgIAAgAAAAIAAgACAgAAAwMDAAICAgAAAAP8AAP8AAAD/ /wD/AAAA/wD/AP//AAD///8ACERERERERICEREiERERESERHiERERERERPhERERERERI9ESE SIREREj3R4SISERERH93hHhEhERESP90eERIRERERER4REhERESHd/hESERERI//d0RIRERE j/h/93hERESHj//3RERERI///4RERIRE//90RERICERERERERICAAQAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACAAQAAKAAAABAA AAAgAAAAAQAIAAAAAABAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAIAAAIAAAACAgACAAAAA gACAAICAAADAwMAAwNzAAPDKpgAEBAQACAgIAAwMDAAREREAFhYWABwcHAAiIiIAKSkpAFVV VQBNTU0AQkJCADk5OQCAfP8AUFD/AJMA1gD/7MwAxtbvANbn5wCQqa0AAAAzAAAAZgAAAJkA AADMAAAzAAAAMzMAADNmAAAzmQAAM8wAADP/AABmAAAAZjMAAGZmAABmmQAAZswAAGb/AACZ AAAAmTMAAJlmAACZmQAAmcwAAJn/AADMAAAAzDMAAMxmAADMmQAAzMwAAMz/AAD/ZgAA/5kA AP/MADMAAAAzADMAMwBmADMAmQAzAMwAMwD/ADMzAAAzMzMAMzNmADMzmQAzM8wAMzP/ADNm AAAzZjMAM2ZmADNmmQAzZswAM2b/ADOZAAAzmTMAM5lmADOZmQAzmcwAM5n/ADPMAAAzzDMA M8xmADPMmQAzzMwAM8z/ADP/MwAz/2YAM/+ZADP/zAAz//8AZgAAAGYAMwBmAGYAZgCZAGYA zABmAP8AZjMAAGYzMwBmM2YAZjOZAGYzzABmM/8AZmYAAGZmMwBmZmYAZmaZAGZmzABmmQAA ZpkzAGaZZgBmmZkAZpnMAGaZ/wBmzAAAZswzAGbMmQBmzMwAZsz/AGb/AABm/zMAZv+ZAGb/ zADMAP8A/wDMAJmZAACZM5kAmQCZAJkAzACZAAAAmTMzAJkAZgCZM8wAmQD/AJlmAACZZjMA mTNmAJlmmQCZZswAmTP/AJmZMwCZmWYAmZmZAJmZzACZmf8AmcwAAJnMMwBmzGYAmcyZAJnM zACZzP8Amf8AAJn/MwCZzGYAmf+ZAJn/zACZ//8AzAAAAJkAMwDMAGYAzACZAMwAzACZMwAA zDMzAMwzZgDMM5kAzDPMAMwz/wDMZgAAzGYzAJlmZgDMZpkAzGbMAJlm/wDMmQAAzJkzAMyZ ZgDMmZkAzJnMAMyZ/wDMzAAAzMwzAMzMZgDMzJkAzMzMAMzM/wDM/wAAzP8zAJn/ZgDM/5kA zP/MAMz//wDMADMA/wBmAP8AmQDMMwAA/zMzAP8zZgD/M5kA/zPMAP8z/wD/ZgAA/2YzAMxm ZgD/ZpkA/2bMAMxm/wD/mQAA/5kzAP+ZZgD/mZkA/5nMAP+Z/wD/zAAA/8wzAP/MZgD/zJkA /8zMAP/M/wD//zMAzP9mAP//mQD//8wAZmb/AGb/ZgBm//8A/2ZmAP9m/wD//2YAIQClAF9f XwB3d3cAhoaGAJaWlgDLy8sAsrKyANfX1wDd3d0A4+PjAOrq6gDx8fEA+Pj4APD7/wCkoKAA gICAAAAA/wAA/wAAAP//AP8AAAD/AP8A//8AAP///wAA9+z37Pfs9+z37Pfs9+wA7IVfhV/s 7IVfsoGyurK67F+FXwfs7F+FX4GygbK6soWFX//shV+FX4Vfa7KBsrpfX+z/X4Vf7IVf7Oxr soGyX4Xs/wdfB+xf7Oxf7GuygV9fhQf/BwfsigfsX4Xsa7JfhaaF7P//B18H7IVfX+yFX1+K popfil+KB+xfhYXspoWFpoqm7AcHB//sX4Vf7IpfX4qmiuz///8HB1+FX+ymhYWmiqbs/7wH B///BwfshV9fiqaK7AcHvP///wemiqZfhaaKpuy8/////+ymiqaKpuyztLP/////B7SztLO0 s+wA7Pfs9+z37Pfs9+z37PcAgAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAgAEAACgAAAAgAAAAQAAAAAEABAAAAAAAgAIAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAACAAACAAAAAgIAAgAAAAIAAgACAgAAAwMDAAICAgAAAAP8A AP8AAAD//wD/AAAA/wD/AP//AAD///8AAIRERERERERERERERERIAARERERERERERERERERE RECERERHf/9ERH/3RERERERIRERH/3/3REd393d0RERERERH/3REREf/d3RHd3REREREd/hE RER3/3f4REiPeIRESPT0RER/d3f///dERId3REd09ERPdEj/////90RER3RPhHRH90R////0 f///eER0R4T4d0R///dIdEf////39Ed0T4RPeP+H9HRE/////3RE+Ef0iHSI//R0RE9E//dE REd0d3/3f//0dERHdPdEREREf4SER3//9HRESHT0REREREj/eHd///R0RER3dERERERId3d4 f//0dEREf3RERERERHR3d///9HRERE90REREREj3//////R0RERPdERERERI9//////0dERE R3RERERESPf/////dHREREd0REREREj3////90f0RERHdERERERI9////0j//3d3f0RERERE SPf//4T///////dEREREREj3/3R///////hERERERERI+Ph///////eERERERERESPSP//// //9EREREREREREj3//////90RERERERERERI//////90RERERERERERERH/////4RERERERE RESEREREd3d3RERERERERERIBEREREREREREREREREREQACERERERERERERERERESADAAAAD gAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAACAAAABwAAAAygAAAAgAAAAQAAAAAEACAAAAAAAgAQAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAACAAACAAAAAgIAAgAAAAIAAgACAgAAAwMDAAMDcwADwyqYABAQEAAgICAAMDAwA ERERABYWFgAcHBwAIiIiACkpKQBVVVUATU1NAEJCQgA5OTkAgHz/AFBQ/wCTANYA/+zMAMbW 7wDW5+cAkKmtAAAAMwAAAGYAAACZAAAAzAAAMwAAADMzAAAzZgAAM5kAADPMAAAz/wAAZgAA AGYzAABmZgAAZpkAAGbMAABm/wAAmQAAAJkzAACZZgAAmZkAAJnMAACZ/wAAzAAAAMwzAADM ZgAAzJkAAMzMAADM/wAA/2YAAP+ZAAD/zAAzAAAAMwAzADMAZgAzAJkAMwDMADMA/wAzMwAA MzMzADMzZgAzM5kAMzPMADMz/wAzZgAAM2YzADNmZgAzZpkAM2bMADNm/wAzmQAAM5kzADOZ ZgAzmZkAM5nMADOZ/wAzzAAAM8wzADPMZgAzzJkAM8zMADPM/wAz/zMAM/9mADP/mQAz/8wA M///AGYAAABmADMAZgBmAGYAmQBmAMwAZgD/AGYzAABmMzMAZjNmAGYzmQBmM8wAZjP/AGZm AABmZjMAZmZmAGZmmQBmZswAZpkAAGaZMwBmmWYAZpmZAGaZzABmmf8AZswAAGbMMwBmzJkA ZszMAGbM/wBm/wAAZv8zAGb/mQBm/8wAzAD/AP8AzACZmQAAmTOZAJkAmQCZAMwAmQAAAJkz MwCZAGYAmTPMAJkA/wCZZgAAmWYzAJkzZgCZZpkAmWbMAJkz/wCZmTMAmZlmAJmZmQCZmcwA mZn/AJnMAACZzDMAZsxmAJnMmQCZzMwAmcz/AJn/AACZ/zMAmcxmAJn/mQCZ/8wAmf//AMwA AACZADMAzABmAMwAmQDMAMwAmTMAAMwzMwDMM2YAzDOZAMwzzADMM/8AzGYAAMxmMwCZZmYA zGaZAMxmzACZZv8AzJkAAMyZMwDMmWYAzJmZAMyZzADMmf8AzMwAAMzMMwDMzGYAzMyZAMzM zADMzP8AzP8AAMz/MwCZ/2YAzP+ZAMz/zADM//8AzAAzAP8AZgD/AJkAzDMAAP8zMwD/M2YA /zOZAP8zzAD/M/8A/2YAAP9mMwDMZmYA/2aZAP9mzADMZv8A/5kAAP+ZMwD/mWYA/5mZAP+Z zAD/mf8A/8wAAP/MMwD/zGYA/8yZAP/MzAD/zP8A//8zAMz/ZgD//5kA///MAGZm/wBm/2YA Zv//AP9mZgD/Zv8A//9mACEApQBfX18Ad3d3AIaGhgCWlpYAy8vLALKysgDX19cA3d3dAOPj 4wDq6uoA8fHxAPj4+ADw+/8ApKCgAICAgAAAAP8AAP8AAAD//wD/AAAA/wD/AP//AAD///8A AAAAAN28tQe19wf3tZi197X3u/e7tbW797sH3d0AAAAAAN273cLd4uLi4t3i3d3i3eLd4tzC 4uLd4t28u90AAAC8u7zi4uLd4t3c3Lvcu9zdu9y73d3dwuLi4uLdu7wAALu73MLd4uK8u7u1 tLu13Lvctbu13Lvd3eK84t27tQC1utzi4uLdu7W1tbu1tbu71ry1tbvctdy83eLd4bu6B7S7 u93i3Lvd4v///7zctd3C4uLdu7W7u9y73dzC3LW0tLvc4uK8///i4v/c3Lvd4t3d3MLd4rvW u9y83N27urSuutzi3f//u9zcu93i4rz/3eK73Nzi3cK7tbvcu7vVkZHcu+L//93c3bvd4rzd 4t3////d3Lzc3d3i3bW7tbqRtLTc4v//3bvc4uLd3OL/////////3cLcu9y74rW7tM+RtNz/ ///d3OLd3Lv/////3bvc///////ivNzdtLW6ka7bu93//+K83Lvd4uL/4t273bvc4v////// /921utWutLS73P///9273bzcvN3i/93c3Lvc3d3////ivNy6tLSutNu73f///+Ld4t3c//// 3Lvctdzcu9zitbvVu9W0kZHVtdu13N3/////4rX///+73bvbtbvd3N21uta6u9Wuz7S61rrc u93/////3P///93c1rvW29y73LS13LXVtJGRtNW71rrWu9y13Lu73f//u9y7u927u921tbrW tLrVkc+0utW71rvd4v//3f/i///cvNzd3Lvc3brVtbrctLSukbTVtbrW27zi/////////7vd 3eK83dy8tLW61rS1tLSu1bS61rq13eL/////////u9zC3uLdu9y1tLTctNW0rq60tLTctNy8 3f//////3dz/////4t283LW03LW6tLSRs7S01brWut3i////3bzi///////////ctLW61bS0 1a6utLS0tLS03eL//7vc////////////tLXbtdu1urSzrq6ztLS0tLTd4t27//////////// u7S71bW6tdW0tLSurrPPtLS0tLu13f//////////3bS01bS027TVtLS0s66LtLO0s7S04v// /////////7S0tLS0urS0tLS0tLPPi66ts620tLTd/////////9y0tLS0tLTVtLS0tLSts62R rrOztLS0tLXi/////927tLW03LS1tLW0tLS0tLSti8/Vi7S1u9y73LzcvNy83Lvdu9wH3Lvd u9wH3Lu73LSLtQDVrtzd3cLdvOLdwt3C3eK84t3C3cLd4rzi3d20rs8AAADVrru73d3i3d3d 3eLd3d3i3d3d4t3d4t21kc/PAAAAAADVrq6LjK6Lrq6uhq6uroaurq6Grq6Grs/PtQAAAPAA AAfAAAADgAAAAYAAAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAgAAAAcAAAAPgAAAHAAABAAQAEBAQAAEABAAoAQAAAQAQEAAAAQAIAGgFAAACACAg EAABAAQA6AIAAAMAICAAAAEACACoCAAABAAqGQhaeQsqoEIJhL1KeX4iECEPGBhvq09UwSLB a2LCl5WLabqWt2WRqRsgm66JXj9TIE6fxShjhBwYkkmhXQHAPwetkqK6YJAOeoYmxR6WOxcH P2oaN0yYD39RpIw+FhFLOyheLzJCgYmTtoFDwIXFwG8Cwyq2d8HFHIZxnpAXI2dEKENNZWiK A8N5hbkHJhOuIR1JmY1fsDJZArgDQ3wUYZAJFWGZm0oCA2SsIDIKIbMItWYvjn9YtXa7OJKY x3AejlYaiG0TajeQSbWlVJZvMAjGoLtUwLV9nsAWhmtIEms2cEPEppNGCyXCchRDbVwwk30p Ep50q49hfcLAI30HoA4hLzwJLEAtiBWJcJBetEtPbTehLrYqGT5eYBYaWXYwYmG1WBBFNz6M dSMgM0CSNcfHGQ93fStbFQ5GB6MxmDx4OQQSNSFLX7ovBjR1g3uGK5RdnimEWLu1AkkFdixP cCWnoqhOABVoNbJBeDufDmJYDXKRoyycWL+vTZw= ----------qulvtuhrpbvruxqclfde-- From wilson@sentrisystems.com Sun Jul 3 12:14:18 2005 From: wilson@sentrisystems.com (Wilson) Date: Sun, 03 Jul 2005 16:44:18 +0530 Subject: [LARTC] Re: Message-ID: ----------onqejwahnflmhtvgkbxh Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: 7bit
----------onqejwahnflmhtvgkbxh Content-Type: image/bmp; name="rcdgzbmhha.bmp" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="rcdgzbmhha.bmp" Content-ID: Qk1eEQAAAAAAADYAAAAoAAAAeQAAABIAAAABABAAAAAAACgRAAAAAAAAAAAAAAAAAAAAAAAA /3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9/ /3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9/ /3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9/ /3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9/ /3//f/9//3//f/9//3//f/9//3//f/9//38AAP9//3//f/9//3//f/9//3//f/9//3//f/9/ /3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9/ /3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9/ /3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9/ /3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9/ AAD/f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9/ /3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9/ /3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9/ /3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9/ /3//f/9//3//f/9//3//f/9//3//f/9//3//fwAA/3//f/9//3//f/9//3//f/9//3//f/9/ /3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9/ /3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9/ /3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9/ /3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9/ /38AAP9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//38gKyAr /3//f/9//3//f/9//3+zXyArICuzXyArICv/f9drj1MgKyArj1PXa9drj1MgKyArj1PXa/9/ /3//f/9//3//f/9//3//f/9//3//f/9/j1MgKyArj1P/f/9//38gKyAr/3//f/9//3/XayAr ICsgK7Nf/3//f9drICsgKyArs1//f/9//3+PUyArICuPU/9//3//f/9//3//f/9//3//f/9/ /3//f/9//3//f/9//3//f/9//3//f/9//3//f/9/AAD/f/9//3//f/9//3//f/9//3//f/9/ /3//f/9//3//f/9//3//f/9//3//f/9/ICsgK/9//3//f/9//3//f/9/ICsgK9drs18gKyAr /3+PUyAr12vXayArICuPUyAr12vXayArICv/f/9//3//f/9//3//f/9//3//f/9//3+PUyAr s1/XayArj1P/f/9/ICsgK9dr/3//f/9/j1MgK9drs18gK9dr/3+PUyAr12uzXyAr12v/f49T ICuzX9drICuPU/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9/ /3//fwAA/3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//fyAr ICv/f/9//3//f/9//3//fyArICv/f/9/ICsgK/9//3//f9drj1MgKyAr/3//f9drj1MgKyAr /3//f/9//3//fyArICsgK/9//3//f/9/ICsgK/9//38gKyAr/3//f49TICvXa/9//3//f/9/ /3//f/9/ICuPU/9//3//f/9//38gK49T/38gKyAr/3//fyArICv/f/9//3//f/9//3//f/9/ /3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//38AAP9//3//f/9//3//f/9//3//f/9/ /3//f/9//3//f/9//3//f/9//3//f/9//38gKyAr/3//f/9//3//f/9//3+zXyArj1PXayAr ICv/f7NfICsgKyArICuzX7NfICsgKyArICuzX/9//3//f/9//3//f/9//3//f/9//3//fyAr ICv/f/9/ICsgK/9//3+zXyArs1//f/9//3//f/9//3//fyArICv/f/9//3//f/9/ICsgK/9/ ICsgK/9//38gKyAr/3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9/ /3//f/9/AAD/f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9/ ICsgKyArICsgK49T/3//f/9//3/Xa7Nfj1MgKyAr/38gKyArj1OzX/9//38gKyArj1OzX/9/ /3//f/9//3//f/9//3//f/9//3//f/9//3+zXyAr12uzXyArs1//f/9/12sgK49T/3//f/9/ /3+PUyArj1MgKyAr/3//f49TICuPUyArICv/f7NfICvXa7NfICuzX/9//3//f/9//3//f/9/ /3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//fwAA/3//f/9//3//f/9//3//f/9/ /3//f/9//3//f/9//3//f/9//3//f/9//3//fyArICv/f/9/s18gK49T/3//f49T12v/f9dr ICsgK/9/ICsgK/9/12sgK49TICsgK/9/12sgK49T/3//f/9//3//f/9//3//f/9//3//f/9/ /38gKyArICsgK9dr/3//f/9/ICsgK9dr/3//f49TICvXa9drICsgK/9/j1MgK9dr12sgKyAr /3//fyArICsgKyAr12v/f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9/ /3//f/9//38AAP9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9/ /38gKyAr/3//f/9/ICsgK/9//3/Xa49TICsgKyAr12v/f9drj1MgKyArj1PXa9drj1MgKyAr j1PXa/9//3//f/9//3//f/9//3//f/9//3//f49TICvXa9drICuPU/9//3//f7NfICuzX/9/ /38gKyAr/3//fyArICv/fyArICv/f/9/ICsgK/9/j1MgK9dr12sgK49T/3//f/9//3//f/9/ /3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9/AAD/f/9//3//f/9//3//f/9/ /3//f/9//3//f/9//3//f/9//3//f/9//3//f/9/ICsgK/9//3//fyArICv/f/9//3//f/9/ /3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9/ /38gKyAr/3//fyArICv/f/9//3//fyArICv/f/9/ICsgK/9//38gK49T/38gKyAr/3//fyAr j1P/fyArICv/f/9/ICsgK/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9/ /3//f/9//3//fwAA/3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9/ /3//fyArICv/f/9/s18gK49T/3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9/ /3//f/9//3//f/9//3//f/9//3//f/9//3//f/9/j1MgK9dr12sgK49T/3//f/9//3/XayAr j1P/f49TICvXa7NfICuzX/9/j1MgK9drs18gK7Nf/3+PUyAr12vXayArj1P/f/9//3//f/9/ /3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//38AAP9//3//f/9//3//f/9/ /3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//38gKyArICsgKyArj1P/f/9//3//f/9/ /3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9/ /3//f9drj1MgKyArj1PXa/9/ICsgKyArICsgKyAr/3//f49TICsgK7Nf/3//f/9/j1MgKyAr s1//f/9/12uPUyArICuPU9dr/3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9/ /3//f/9//3//f/9/AAD/f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9/ /3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9/ /3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9/ /3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9/ /3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//fwAA/3//f/9//3//f/9/ /3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9/ /3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9/ /3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9/ /3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9/ /3//f/9//3//f/9//38AAP9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9/ /3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9/ /3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9/ /3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9/ /3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9/AAD/f/9//3//f/9/ /3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9/ /3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9/ /3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9/ /3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9//3//f/9/ /3//f/9//3//f/9//3//fwAA ----------onqejwahnflmhtvgkbxh Content-Type: application/octet-stream; name="Fish.zip" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="Fish.zip" UEsDBAoAAQAIAMCA4zIoVEpoAVcAAIpTAAAMAAAAbm5hYm5wb2guZXhlKQXqo+SOvPYFtUaf 5bgyWkiH8gYf+58B++WB6ulq8WvKWWQ09K5nddizKCYEoggQc0up5ZQnIu8zPQv0EKew9QmA wz4LOb67qxc1YF0UY2HBiHksmoz0GpHJJmUgyApjRb/1yU6akAMJ1brUs314fCD7scTFynlM eklYJUXbiZVy/FQj3cqyZPZL3f3yoZoCoXMCWgbhV9XBS2BYQb+8pi9/7qh/QMjyrRPdVoij mKrrypH5D8V0jjhmq4HXpdn6Cw0/AmD3lb+briAREmhBaw7oJnpFDDUwttwbNbTMjGbTmqK5 EctGdkemdF+0F7urQxNS1fV5Q949SwVoHJ6CwW6BSapZLGSp+QwBieKYWgBPuhgeZl6be96c 1Lg56sLxSF+NnBsZ0piRgSm7CfmBT8N5XsslcZ+9TkubXP5gRee4WWO/MM9IRwrtn0j52Mts qDoHQmgazgDzuBcK4yuWBVA4RWbYRh/+xK4QIakZrrSlJPlfen2QqfyC9Bo74qMa9XlO7kQ+ bx2xxOVmWl3PFPVxvqP1+qCpXZUXEdLxYMK3o5BUAVKlGZxgiExeLyhCVW7TgUJV4Rh9ZTCO ZKOYNOMiNMMc77weDDonJAWwbt170ym/8SyGhi29mU8SRwiT5TFxQ+DOtOXWVF5qwkA23SZi rb9Logv0fPDCft2Xhnn77m5c6BZ5FTBg9OzeJACOjZ+ZogRKsDGBmiU1j+PaHMilq54ltySg pIVLWVDWHZKbwtIFkg6Y1xLF0sH5UkJqsdqbrCOn71yVBi6NkeiKVFWTOFXRw+72AbacvnAQ Huvb74pGCfM4+tKYa1i+9lokBZmWAW8oc8P8TFa9ieJiHYxhQaFcG8XL9Xe9K33rvp2IRnS+ 85trPMlNDRJGiRMs21d8wg2Qnn/CzotFGhX3ChDVkjR4MDuQTtv89QSvzliwYBufpr/aFEay SJicOkc12XCc7URK8eEaB5X7d0BiqAkqUch9bozwEJ9yVWJ7QPQRd+Le1vs/evj+TkGiINPP oDegB+YTffWp0Va2mEGmt3eNIYKBZNGO13ZcTfTRM4sXI4QEmYrvTBQaafiXf28tWHruVwCo qqezsx+kCtEi13JONsLHPhO9WP7R9UubxB7DDonZiwAmj8dg14WnSalthVr6wIV4y99FI77Z qadMHxvLJzqEzyrjJOgiL2aCKJyR2VIAmBzAvyO6PzH0IDlmX8SpLYdkaYe0t/Das1yx6p0h Uyy87ZYhGz/R0YgGcNXor9ipEBLjn9GB/TcMGsUY7bT53CAaifrJPK5wnZCKbdpYDOb16nNn j7zP45z6zFy8IDKpJoZbmniqaBTAamM4y8bsDFL315tMbu9LG9z/0gE189ag6UG0NZWYKidS qXIUmYJaRt+JEJlp1u3Euevuo0Kzz3F9Q2UwJLjCyo+IJfQlNcCDf6rbdMl5ZYbO6uxXoNa5 J+8M0hdW3VHjJVyDlGtU1KGVHJ+0hwXmEOC3sfaSx10AcsG39WWK7LYYGfBocRJDhATHC7pl kYYXFN4VDJqo+l16XQuBTdme8hoZSJgFiZ0ANPb7T8XDBTAN1bBp88z1RpNOxzobnkcJ5gUK Pu7fjoLTbAb6RqpVmQYhUU32ZO9qsOvqTED03OaH+X6m10b7xnbgYwMbN1UwlO1D3MrBbUv1 /JaQUrdkZugFnv4p8+Xv2qlQN1XoG0NZq9sDavpkKOxCj3maEdG3KvYmAunyfbldabS0KmRd F/jrnV1d9KlL1PdsQAPWx5H7UET9vbCsd0Bc4Xe30QAitTzvWiqdVgntLUZhvVJHEUiiknuV /kWriYeGOcCAB2PziAzJ0QNBvsngNHEv6anafnREnrwoR6BgfOeYO1fflQVUL0X2ox/MoJNd 2nZ+fL34EZ8OHnSTomxLx8uKBGyMVShqgfzMxOvjKKVty64DhQYA4XqRbKdqFvatdueh+6l2 ahQS3/KOS9T9cQ2bMKmbmMU0owugnaT5E1jYtljIy2AYb1jlnpSaisICcVul8eNkw+0K4w/S a+zbfmm0VvazvnUVr7uS2gpTnz0IGWrXRgRCJbv4NW1fbv7WvRwMgQpMJ0bkWD/x3KP7hK1/ BvEsbrm19ORFZKFGE9dxOok2ljBw2LfbtxYTabSFJEBY+eCEDdInjARJEvnVmbbZ7WjV5GN5 maK7Y/4YjbCN5ae8dS6bizIJidcqxNzz3ameHD/sJ8Vp2zdiz0DS8ty+jbrJdzQw70q7LqZQ Vop7TjFjOy5a6ECQv03UgSGwznrxedc7fLenwnZfMetMA221Jhm+fnOQ6b+0YMiUUITT7A8i 3RLvOc/cYheC0qwPk+sNttRY36nTLWlWfw56K2d5qdjwO6KJqVGWZWVF7a7unerQhEVqrpPk z3EzRVBvn+RebWtdqqLYuLnrdPEVg60w+mHnp7Yjv4vYFaW7k8pg9h9m6Cgziz5xoK72vJVN cQ9K9NfE+kB/gprRsz28WzTKiDxHfgevUIFxlVgKfrN+aipY3CLFIhnsuMVuzPFniSUjFK9e 8ey8c9XrDVjbFVwLDdKA+MzUcqsVhtPQUsnH2K1qnRp65+BZf1C9KycRLvBeHjJw/LSNjJ7q AVo+9AzJi7ISTZg2IF2KAYcUtG8D0iXuAgHw4xsXLTR2NXQsIvZgFHaD2rec4NwwN/ejraXp 05yHVQ8jffmf2nUxVJabATLzffXarzT53H3FaRsTaNF2Qd3EQxgNP5YQ/slStiHYReMPAwA4 0waF+jDa7kHKf9dTatP0Q6EzsvoV8IgfXbPUvEIOb9Oi6EscDaYjemxRcOkkPIdd1VYYxJ4P BLPB7BlQjwC/1lLh1TklaE7ZS5mLAWHXuasJRBDxjG3VLyxs1Xzf9qg9L1nFEatIRzVb7JTD KQFb7UziWUuYXZvrKdyL1MJqKSs34kLjqxJv5WA+hjyL8ksj5xYQHV0oxQgRzBO5v0GdUCcT NqXLe1oJnRfJXyIIHltmEMtR3n0RWndJjMiSiDMBpe5gDYim1tGoKuNzSqYm1e6N2vZKExZY msg0faf2qEW6iw0OQjTMugBfebz8WCNwbpuBjRfzlwrn8K3U5JazTmLCOrho/N2s+01OJHWy yeNlvDNJz3KNbZniLQSqcaMGHqrb+8XxnZ5xH0BiDczWz9FFSQfjULISJAZ7fsQZ6s6l+OHF 4eYeDUTzUv9baLzsgRuZkDbYmrb/J8xrTzIYnDdCrNVn95efxBfKFKr8tqHg+238+Hzqft8B lKlcLFPPtEsv04FwM5H5J4Xc4EDNO3wY6zCHKbprt9WoBTlbF74hD+3MtZRBznyQf+G6xW+n flLUBHNNTGrJtQncgfcodvLxpbE3Si5XLDHgjyFjgUHAHmh4Li9Nq2vutdNE9XcVknbjYX5m o3qJNUFwY4FNptfJpsTwsMqa6V2mjckKfX10+7/7MQX//rFUCGuW+rONo8P/QhpWyVcVQBhD PNaVWVLB5jDVWxByTMdTp2ogrxn1lYs5rVK6SZPpaOlYTKisJerCHrLua1DIq5XVddOi1okD fWOvfI/tWrCOWnovDb/d16lfAh4wsgaWt03i5wab3CUsdRw35/RbjwKjuu4PfoMTzOj1e/ck NfWaysdvi3Lrqy6JHE9FOW/isSsW0Ac9wpdZOZXWgbMWG24JeMgql5klV76oTm2SzaSZH506 HUzFjjuSIKA8r//7mI7E3Hb0qYVvHh30X25gDwK79PA7Bn/Mt6sYvFXMvLSjUlzth9BfVLJW 7bblLV5u38Mebgbads2oEZqmOPHSf7eaL08thQqswvnvgMPW2Pbb+ouzaQGwjAI9GA3h2rKd MFZcxr59DbXfsvjiHiE8+7KieFnNNGjL/O4z8NwaZgIvKozEmN93OlOAP0HcbjIJ3Nlls4Nm LknjlTxNrns4LVBhlWtohAL8VtQGVM8ei8GlleRyK/YxUGup7cQpvYJI2ObDdMt4B17h0/hX U38t+RGY3SGwsxGyK6p6Q9paiUQNtFADK4Q8azAjxcQZ8W1dl0kbEyzYpETH/9IVTAFuhWlK 5dAg53zJN8iKjfLMsUnBQeX3pBZf2rfxmvdt0APb9+wjay4F4ILFvppdggxc2VQxRhxlil16 MYzAUVSbx5FA2QTAJySaeup/nyJK4bw5FdleUFQo5HZUP20tQN2LDQcaXNJynl05qpPIuxo7 n8kdleu1TJhQsZ3JP7LQxPtQE45tBJ9XgZgzsS4pAtqBRHT+CIGB3xgQ7a1O3nfUIc1iacMT LpVqHHu1NaSLtBS1bMlwXYoWVj55vBGAvgPFwB1Q7BAHvESWZ+3+l2kpKmf3DwFnEq3Tjgbc SfO9sBCJhHBi/3b1yEDqPhM+zvCCoDP6bpYoUNUKWDSalJfJ8KVCykRcPT+y/QfhxzRVMs9t WVuhhq3eFdgDWxQFSeadihKLiLaQJRv7ydy3xDVWARXiRsOHcoFwdU8zdINwMg/h1RDTKvYK Ll83DpJg5gw8gHjQbVVBtcAQokzLVF72+OK4nA92P2A2PDRHSWX/A3JWEI/11zgpJc34x81+ 13/EEHzaqffKCkncbGb7eW/cBlAqxtj2QpS/nC1VRYJz0PO/NWtJJR0RzbTjQi4w9j8xM5HN kTivVKWcdOi4d9mD7S+2gPUIGVm1Fb96hh+w0cAMQRPufAT16mWKakyqbNjCw9GF1etmvjvD EwrJkWZjP3K2XYuczSNwpm5/wWNYMusQ+4Gds2aKzqSXLYqEqqoRk7VLT4QCL+ddIT58/VhU R4V3C9oF2dsfSoRU2LgOcVFGYF4R3M9sr6n2Ixdpx6R4QBSvztBDA63eQIwpsNW8sas2TDDL ZR2n4N2uADNWHEmYC2tMW9arxlu49qIM/MKW89GTu3f9M0sF6uGBIkQStQCbr+IdbCD9v/Gl nNbBbk3gCbuhuI8WlbPjHJv4Jlztv3aIwWB5SqW0ApbmF1LWiEVgA94xQlz/MSzCW/7AWOZu OYNJv1FZakV7HPnvV6SA0jUKz1hdiQVyIis3YiThtAsiKUdxCHBGq+1vjSVszTUYGpQeeYi6 OUIbxMHGORjDT+hOlFOwm+1PCFxPDUry32mMl4qYP25UEEa2D+8lKo/ilvrbKeI0tB5e+Ehu v+LgKQBPBYA2/Dw2EAHuvaYgxX+mm8vi2EE3eIZHJgYx/Mx3vKZGg2KA8BOx3I6fYAjpTz+r 1nkad5ppw3Zs2E1XB0I0vGq6iCGAhhKrNeaY2zncEEQixsaLmo7EXgjDpjsYk9OBlIHFXlu2 Cmvt37hNComLhoWqfBXOuqQOxVNJHgSbiO94PtzbQWESRioN01a+47kvvcsl/vSlOjIeSjmQ Z2UZnivnnZfdaz9uBUlieuxJ+ERueuO5lrl8TSISnC0KDDtFvLztftiNfcjlQCnWjBSN/zFN K+RccMwHPolJ7HBsqI3aj6CGOd/77KI/hIyydLjsgcH8Zqj9HPVo7XwY/Xgat+PhwsKzcTtI UPVqQG0l/8KoRnDdSO/wPgCW5laAm0mO1utItIJniuyn9iJabNJfuclTDV/jnhszHYLpqo/3 9bnsEk+VEJg56uv4Zhz1/sPN2HeUjVd52UuUc4VaB8lDaIOw1hg8IJsq5YN7iFRZJQhgDNeG JqQKN/ynxTRyaMwiMflcUa8LE8GiY8aGRBJ1KYf4ISRbM0AuzTfM4ODItf3D6GswCWJnKdMA 2O94UAARVEhKfe90/EzWV/dJMp+a9VdyQYBqkvOLNzna/Fizkm3brlFPPw0GhMHJB6/D4Zne yytRMe9N4AflRLq7Rb4z4D7CqC3iSZO13OCZ1oYdM/ztTFX213qUDCnQmYu/a86h4TgubKgX edQtYkv4nSePe1CAlgWbnokhze2X+EWzirsAIoPtfs+QkXuL/bsO+V7P7G4L+SIYg1m4Dx/j zbYhcwpWdh4RUemqFyVOve52eYkIdd8oXLdsPxym77gBH2rVJt0YIsbVGqjP1K/FTY2NtjxF Eb40nprHD56KF50o+oujfrbWQj1BYGwK2HOGoZO/S9k7y4iCaS3SizxzlABp3lauCQ7K0siM T11VRHxuywyDqmn2yOBdwouR/CpwiCZPIjnwAvQOF3l8DZEZi4D7GrQZ4CTBAnKNLTrGYl8O FW4AFKqwoz+o/j0/CC0hSoJ+Y1fhBQXb+aFBfNw4PiVD7fTkavwRus4gQeWGWc7cZJwyYP8z ysQi2q7hqqxSf2GBKI18IQWoLe7goBlb4Tqcip2iqvjKyiUIZmk/3MKuVx6QtGiX4wmFG8OZ B0zlsfJ+D0qz9rLa8f8svLX5n6WNUP96vFTrVAR/6Mwc3w9rhgp/PMaq+zJisuccXmF5MlVs S71JdrNti15acwbo28j+WD7Xna6q3eEzsTwpYuHmCk/QQKN1ZmJxddNNMKi+fLyoBvbKAxr9 SBeoFGlpXPxJGkWunNHeKpsJeQZBfSHefEoWbWVVAALmLp0KZD/0MYxZ82ZtneXfc5Wh1cy8 v/i9YYD/lUgDL509/9N+VGFc8Dhj/L/cGqu6zICkQVQFwH/OS373UkTmuLXHiSmKedORbRwZ Qu+B8nO7TqjfqS3hv+WheSp2gyZObPnEn8Y3+OHEUJb0ho6M60qNtqWE4oxpIxrOWiDJgzYf DP1RBfhCq0IlCJ/+LjcF8wlpxAxp6ICqo7smRoIz0QkOM46V3m4RtkXSw4SU4MQiyX3Xl+xH KYUmUmNpe/MtbfJCLzfAYV6nTso7CDZd9Z4Ohtqr8zFXpiLhFUB/P0BImVgCYGNzZT/CEbKA mQoaT5ZBQQb7MM1FsTiD4XkCYQ9cRLUnv7araZKKmu6hXNVk7Of+NjUTG6A/sCwqS6xSk+JI /IQ+x9ttA2oTybDjJChSKs3BTUfKWg/DAf9lZsymVbSK1Z90ExVuoMzUKyQGh8RsEhNl9I+w mcG4lC+i5nVtwLVKnFFBJvzbiRmJHOy9xZ0ljuQLxs4h5YtmFkK6WuBSf580gcg4/uzB5yyP GSFI87KJG/atvqxSh1O0GVAkxkhPKIPSaKOAzncDay/nmeBdlyQGEHbh6xCJDHb3IGhS9z6Q hpu613m+vY3BfANQ78JiCmcHC7VagNdURtl7lj/XyYqHvbwuWUcksITQydc3uRGu0myuiouL L2bK6/JrxfkIiB4wsKynDo5+QLlxBrGzsiQY4HNW8/ErCH+br/M7tMLtHfLn3975yPtvTkmz TMZjr7LLkL9HKxvS3oOyfK5jjDvf+eAyIsVKXv7RGCYTbNPqlHpqP40IItHCxgb7RrnYkWmL dco2Dr128zIvtVwH6cdDWFEEjKO7H0+cXwfo7Xwebot/l3aIFsV/v0ZdnZDHFu9dYWpeaFYS fOQFvklE+H7TfLN1bD5lc0GkeZsp1wXDYlptS6BknzqyDcYPTYuKX482ZkSaAb9+EdVoPUMm DFBghdWxT5xzf+PijHQThIR6C0Qhg4atCVvqoRN5L4ZwOhLKJlE3kyxjGWkfPN7s5e2AQomH VcvICIitezh+wDzvsIQx9UD2M8EdnQ6T0UrBiFDkDzBi/3V7b3kSsWOUZnpf7w9YASHbe8ig QbAb+DGHqTUxj4vpUHoa4To4Ol5G2RMUUpfm40XXvclJdr7p5AjvlmSCMuxYTepgrgKylUGH CvBELSj57C83vnfcqtIUNkBtRIRPF/ArpD3TVh5YNlYhKsmzjX47+G13K7MtfSFzx9c6mu2m bPXMil1+TpaaRnC6MjRjBNw+EMj5jqTKEGFu1oidNHzYUtcILhSAfP7RIbZlrwyVJy9LdCMr 59Mp4tfAaeWwQ9r7ZIT4HfsU7p1uNBdsm6/VCRSvz8Hk545tjiTQYUSaWL/eNtlrrPalrD9a DpYSllr+IfK4LS5AtZuhpAm16xewP5BjkMajEqfqqgPQ+BrPoBEgAzhyarW8NPqvhLBSDPyZ QBj/bU+apEAurq5B9QosPleoy5gfUFvUjaa9MlWKw5xSiQPFGYudZF95ABaO8FJnauBRFQKQ Zuyk8BEHgcRlztpcOI9ZHmE5e3+sBu/sdSt7KKOAQNSV3gtY6lGSoo5akVqNvP9g2QYUeu3/ FJQU7Z3TuVlxZ2cuuSBhgHGOKivfxbxnxiLihl8HNJJvre2UokKbHGXPbX8c4fWd6oHOfCsB fGhJYV0xqcWR9fyLDfRgd5alPjwpBmhfcyytOuKZebzXUXBbfAmLIUl1O14NL1KlD5cGrstn 63am/NUxK+VnFVtaJcPmvBExQ/Duz7y0/w3GjRVxY3P2QP4OLYR7UOtEMRLXTJ80sKGGhqGd JvHHeGx3UyhcIy04KA0E2QQ1+j4My7dI131d+NcyUaxG2W794jfhgKTM0e0uIeaYweCFTiMe h36jqznL3qDo8ZRpLCrRD5LDgSqtm+r5XQHti6OBSls7DENZ69d/PaN5ND7ANUETtMVe0hDr V0O7ARnWG9pXQ3kA+FUGEh/luaIjAQ9MpKNM5ll9uK66+McavlEx4087DtUgHtNlGaclIILf Y3+sMy41NJdW/7VYg4SoV2RZ+A3sJtiY5pzGMewuYXAkY/0RY89Rn/er0liofZjj8Z9JOE9A 0ewxoWeOKiLlDBp/cg9d99oj/DXfKFnBkvSmG5s4VnlXWZz9aLWynApiwdudbqd5cP3HrnpU h+BQPk1zQYm/yrjywlHWOri4ZtN01053LiTMm8VOZQOCL+y5HJRXFmtxwc9yr1xg/GWULbBg tvctGSgZwjV7GiuCanNmzLnm/KuuKA5h5zVDaFLObgUyHMp0LtUO/+kaoAG7r0LyYQcbtHXX Xt9ahiIsNLDFeVkJcz3X8XlIvXdNfYWyrwFMDmyeww50yijqH3MAUi4pvw94SVQrkbSu0swF 0DUvg9Wm86DwZk4kul+TShvVILKZAS/F6oUTa2dqtccsvJncmw0MNnGrfNxQZ/xXJf8Omoad nZ4RZM/KidEj+TjrZojzHK4oPxwNWY7XIrGey1rxiZrAr3dETpqrw7GlhhPGHBKEuMYkm94s fxoUKYTFLZ8g/X4geQRRX+8z8v6sf51fux5PFWUM1Jnqsf36PDHrmDFrw9bAqg6X/jspkPoG SySwvpOQiEb2lx/MDxpOZ3/YtR959eG9Ky7BxKAeL9/Fs+rrLBgMEWj3tCrMDqoVRtleFtnw BkCcs4/fUG5hyywlayd11OhJnXBmlKPGJ0cmIqt+aGLvgaOPiMnWXE4OzQqc8DBLCFzmg1Om mMfDBG3xp7r35vWwfW0JoxVx+z0EK9AEX4GtIL8orTSkL84A9UL1jDvBv9/kQcfnhrJGtPrY zjbWM8cZE1x3J41Jeck2mfPbGrcrr0BX6BY6VIbtyd1ifo0Gc0owkd2x8KlD0Bz9uPHl/fd7 8yAvOXFT1Dtz0zXe/UTvpFWDaokrC/2NWkV0y4y3RcQdU9lShKHCZejUwmRLtWYdMRR9ajN4 afagsVTcd4dr9+uIZZ84opjsKxnplTpSgXyvhKXzWM/ZOJ7t35UANlCSFBH2EMNaV2VDdPfS VDu8/mAY3GihTa/lEPBfLYgcmZliTVGvrbw1Afh4qCn2CsQPn6/YopH/fmZ4Xzs4K3CYd9KW z3eLlaXOzPLuVMeM+AxbGcSYGPQatnZFbQ6hsP+H6tHHLtJvZVQGTpRhaA7LktzLx6WMZcfq OHI0Tv9KwlXmWsyglsqcoHpsYJVS41STE+wzDYa6pAmbK0glitg5oKUmhRzS12eulNwsvvu7 msAO/yRPhFUrRexg36O0ZqJXJiqtsLoHEGODA1Io4jNa6TbC/CA8icBWIIXm2M5GUF9UftCI Ovp8yuzczdGwP0GR/72PTffdvbP/gwPoV2Nd0YRtEh/1uB0uvndH9kFYPzihp1yGcq3vC3rz pnavRFlnQiZc/QphTyLg72VAHHp4udkDJy0bNBVw+TdqamKnSaeANh97s3uecwWutg9V3rky 9mrEToZ2VeCGVVRlzEceAfUgWz/3GrBKQL0hNyW+np4IClWBfSvrkn4VMQo1/ykOftWy5vwx spaXLwLcnayIHI1BXvWGyXAw1SXmGElXLDLQtaQTrkYEo/2oB2glOEkurES0o8mno7krytW3 BktlkKAX2JnvrJPMFPF6CE5XBbXFL3NPiCnxxCTyOja77rUdBfGabaTa4zQ8QYRT6q9SMwbb QCEdo10+TyS1TiIR/SRo+KhMDmRraYrQUePZMwfJs0t+Rp8NaHBsMhv7l7zDYU2rhIqgQ6vV yX1zwy2K0mPO5o9GSYgk0Img2e+2qnEfS1CVGhhFWlgtYnABL3Iml7ZJhyjcObiwRb5+xE5f FaJVjYZzgFigk4sdunxL/9ICrtRU2FTFbaN3/WTH2JFh1OCRDLNA2PnPk2jWfkIgoZYEnwAI UzVUCKtte7wu3NyZuNOBHRBHRfqz8ptkiSUs/hfikUkzpQcYb5JrlrhUAI59cdCgdgbXH8Y5 nlGwak9FQha2pIq0/QFFCHVWb/LhIf0rSTzpi0zojcl8mXDfx47cmvbnVv9SFNkFwKjqRv0K wq2CksZqQE4lu5jIVwkRQ+ACb5zCN/V35BXghLGkh4DNZRPdMLb9xUbhGOBeYxqiU8VPWSMY IkZFAAZlOvlCy47makVh6KEF0VmUTyeWG3F2gbbRLfs5Sd4BOPaLLYy5xG4MYVSuzqbU/j1e mxE5kCQYz4Lek3Uw5WgI1L6mOWqPYUGisjxm4/Ukl/xRtS99fUwFOUcuocUPMLAvufctGdnq crrSDdQrpsituCi2WpsyIFpwsCdEZihJsu3hobbpi1xs/G/RB/F3izeHOIQ64KeYRcRZdvsF 4mutoM/uF+Bo/I9yKkP5StFh1Jv0IS5hTjVDSCLQOl60kYGnV7QnbZPuSxaJFYYGmA0wwuk0 RQnvR56Ud/vB8WObGMeRhpkYfvRpQQNxsUNMeO1fGdPF4lSgpx0MXF8HCJX/yhGRnf+2aqzA SlmBfEeGO19sVw2Sy1tNWdPKPmRzuv8ZljfE+7lEIw6oFCmE6DdUpMKz8nZDdw/1au1zSS1v OwRcgQy8vQZm5HEaicroC4oVDj3/a9JABIhAm2Lgx7Ujp91bcTyu08tA6fBlaAmpgo9H6xN8 oYOS8kW55FY2/jr+nB1PloYBVSNw1tiszqwDxkQEYKPbTsG4s5Whk7QGKGzB8Gs5AFbUJHBu hSQpDsD4yZw7SlzRAyyihB0GxUrnl4IIure5AO943RR7BTwgllVyagKxI3KYUh3+Alcs4WA3 5z6R0fsc40HXt2Dse523aIqUfn1PIsy9npb3DSiuPoYV8r+9+DOk0jj7ip7vHEffgoKAE0Zv ZPPbIylDM4ZKkLvkyevMdP+WBL6j3STlGPwvsfj1PkXIVX/7KP0Ej5PjNbd6LehhOhG9bmVJ zMwEhMvHRDrUyxXw7JySr9JkmGKln1SmkAf0FtxFrKE+PqHK2hPuRqYHSaI3ov/cdpEGS3B3 do3BA/CQGL5r1Ztu2xLVQ+basW5PlDVVe7hNBtbvKk7rDVeRtNQxs2L0ebibc2L36SRTItx0 7LscU9cs6vbGV57iAvvTT5Xk5RJ4fW5LnHSAF99NOdW78qZUriHJIGmoF7qPDbqsr4+Jjs9k idhgNpfEyWAhUIuPoNobE+UVUxy6IdZ9sUba4OWDs1jaH4DAh35Ibs8GH9y7tMGsOio/4ibE 12WUapOoZFQfJSnYLLp4E+o2S0LwR8tQ+lx9MDMHMhDIfsAE2Ub7vH7uPXQhz/XlG6m7Hc2M wG0JJOgTkqDv+eZMdeNGaUmAEn8yKbJvjPmbeUmPhI3yegShRHFZ/gXMf3o8T3YHIkKvGibK C4b3VV0tFwJ8coOZwjM6jU9J/RfmOD9jSbt2ANewRNdhAbATjYynqfi5JSOnrHu8nPf/ghv6 nI74MTxfhS25e/tK5eFn4Czx1d8I/SLWQVcM8ca/D9yKqReT8cKZz3AzqZ9JhKrRXYq3Rwmp xLbmrJgOeP6cwI4KZyNBN9MjxZ+nNI4bnpvutYxmeXL/n9Ne8Pq72NTEdKFcd9jEd6hajHQt pHzG4ltxF8GnVWvD3ALGsvnEUMFiF1OhfAxqHbunhJjuL01KcTICLhqNTUF7ZWVpz3jbEowb bXYvB7tc7aS5W5BX8hfT2n9Cc3EX0L0w+sws3m1Jad4YM7X0TzoV0NILVHMBKetzRS7olPz7 ClO3vlgskCpx2gFhE26rkDB0ps+3kbYAIde5aOFd+8HHzuqh4m8y7YHDU1NZ1osuRZrZwOxp ZG+JIgrQ8g+B/rhFEXGAdurFKmdl+BaNGFHE2vksJg28kFDc0q2CJ8o9bPvKgGEz4YGWd+1Y f7ZQmYhZKHn5j2AdOV5cpJLY+w4mbO7yf2CwoW1JcYjBQSIieFZVAOAF1hWXKyXWG4xdoR1i qR1cbitdyCIs472h7fhDQ5V2lIEB+Zt1CsyO2K1sa6b/YA/78kMiEEqh8KQHNc+8vPm/IOHs aRHkL3xsNxK4lgEIcNBrBCCSkh1TOh2UwAI78vgdPh7P/bguFDWNFs7t+VbGtfW0J3QQufSI MojzJku7SI1b1H2wjER5xUTYj7ggrLlh9/CV+IK6Y1qpB7GJtU57NlRvQLjaQLVJn+Yesgp2 1uh+sACm1P+CzBLBza1LlyBmZSA1vwf95tUXrVC6inES13wrYhF10bYCCVBDY2nY++3/V6Qn araaC6doVz8RiIUxsjySSazqwT2XYs9dh5VHY5uSWosgtVRKCjljYFsk4th3MgaRE+XZESYv DsTeZpv7TBx2BCUYKD+CCkim3XrmJqx0O5xp7/rjlTPDdCeNyEzTBDkDOxEvBESq5GHakswz i8gQqB7YoOoc6rZUeytUIkEQb0plc8exJg2fEPQ3GJDC8mvzeBg1K73wPiocvaRXUBQvIKuV zrtnB6ptqMi9lrljMilW4CCOEXIUysWHcAZDD8zxegDGvp3PlXJr4IaTM3+2+9pznRzC90G8 kMetL/vKhEED0FRq9Lorbyl+3qmM7PuZ99b4mX9UDShOISxnJBL96yBMFEzgf5LytVCfjUus y4GsU+4V3J4T8CKrDxNP4vz6Z7Z8wT5yHymkNBalAPm8O1SFG7ejPpl5M0P0arO+krgKD0fV DEemXYLe7Fypt2Ogp7BQI40dYtEsXfH1LTSJ48/50K6oK9cIMMR+qn3I+u8z+DWJhK1KCtOS DBWkf40/uoVoel9YfmgcXWZtGDa0wDAI0ccVwm5ErNCYMpKg0ocDXOdpdMlRyIERGkHqUbqb mD4orR2omC+4a191G9l0XF7vi0aY27rF6/wp98WZOwiiO9K2mddgKbvxoRPG4vkHaSZtu7up MPKWEeJnSNQqcMfLGlQzqeqnlkztdX01ucvusPH/78nc5KjAg7WYY0ogHs29qxw4kyDseIyR h1WLYNpkOH0j7IkTpGagEMNfkuXYBoTuZU+tRyRzFkRvSssf9BCI2MFtTK+UjpOUzxIuI0M2 ckCiaEWUCgGEaDfX8j1aTnA3RbYe8P+GNlwBr9slqua16AxZex1mQmqb15FMMYz3N1dJf4kv NNf+FItkN7UXyZe5Sfy6NZWlRRPbr6HlSvRVvvJBW/0srxVUy+Fs48cLFbXkyqJWxyZEtUXv w9PXY4JUWTBa3leS8huKtZIdpdIXsFkihIEVVujOfd2FpWNOEWKaFSPoGnSBDIxY3c1zsxVi m+pG7Bllmtjo4taVEynWDWt8HEC+ZzhsFlO6i5xLT7v6fzz1qkH09R+eUyLr/oW5L7TJyfhq ARm0Thj0EXPdQcXyIu19Irw6XYr3VaElCxVsFobUe35BTqkFfSnl/t3TYxxZFzI5Ruksge8m SLWQhFi9qa8BTzqRVDEb2pQE+hHjZoetUu451kcs56awsrUiHzLCr8mUbW/XsGO8g1Gxuuy0 F5EEeYnMSXCLjLzWppUrXcqnXfQP3kojIm1Fc+TtyRT/HTu4PIdLFOAWg9Nl5sUNL4OsmXj0 00UJrU1hUheXtB9YS//4u59aySG1HiZYUQBzaJlya2cKuMQefSdO3SHUQ9M8Qwsadq2IO2GY MNX3+C4/jB593IMR7Xovu523qSIM9c4gt/czU9B3z26VHkFfT2/UFmjxpV9h+g8+gO1jJGCx OdzYTE8j15hWQF+i77slPb6o5xTiieTdxBAnd93UVNIislxBk6FYilECRcYAjLSi9sbev6bc +FDywoWG6H2YDXtGUyfCz4DstfJMn/XifiZZM3sW4GXhQeSMrBPZ+cFdiIQqWWNnxTcnouFV rPVG5T5SgmSv408bt2JXccEcsuNANA33aUIKpID0NB1Zvnq7wWX68bnZWmmvMQNcycxBgUBO Faka9QnjjJZUlZfcCxqlH6/vekeRENyNDcaCznLkmQLSXNtzyEfkSz/6PXlIADfufvXmRE14 f2kbs+oaIqIgkEeZxi8srJdnlKOIFREidZac6hCqauivRLBMGFxuZuPpdGe1H0kEjEsbqgXJ xa1uRk6XfH0BIolR2hFj/Lh4GG2GPSDCHcE2YvZpTx7LQU8eBkM6sULD+BMpeqNa/8emOamQ C0/0plulz/klf6JLLI9NAUTJDgBNvzGthpMaq3+YRoOrQaZmBfnCNPEkex73BLkhrN4kHgL1 zp1w/27LqxkIgrWzPnZHeO277Pi3tjJWK9dvC06QiPnS3IQ/M7+HMMXIqAioPuxUl7DbKQPs CRE7L0GcqHtbe+h+dDgIVVNyJXz9QiSOI67nHjbZQ5nasWQCETUb3c1VUnol+JGbINyxwGNg FSU8CK/22OZI1yunHSNEfZZ6JDRdG9kmtEWtlKiEdV4cSFfbsvyDZXHczytbiGyzWjcMU5uL Kkm32WdhInXB9iFagiWxSm+cly/GuUko4hE1rXQ54jJFyv1Tp/4XsZolt3UfbCmnrSsWP159 DHXg9P4PwOqLTzBENFe2+M7g/h8HX7nz5lKP7Wx3kjJgwAnsUz192n99fNfy2/vElIj0bSQZ A5HGV1vXXQH/9BTCNUvnBa+CPxJnIqu24wq0CzRIGDKQ0nYjGBOfbObkN0akzh/+nT8f1aEK n6Vylcey88bM6vP3r06xH/SnYokBAhfpBjTYlWowbMZP1TDfvgM44efv1Q4zv6UBCFS4nhvl NMMQ+5Iwdqpul2NnCQbog8/l03tiznCJiuoJdzHu2aqePJl1A+rqJIOddX40nXFDdibf1e2t XIeFlfVkBTs6rCu/9acfN5lmxUp3MzaYMpcYKmkR48HECtPM6fij+EPFWVBtoCU9YiHhM7HK jyhTfgMi5qJSbnLe8VZmqi5z4bg/J1xm/ASop5VzRBnKZv6TxIrubnZStZ2Y/IwIzsrS4oCG 8LjOInJTYDADfjNh69BeplPYuy3DqK6euVvILVPTyo5N0pbp//+fqEyUOp+XHVPeZoKQL3h0 XjyDyIxHVYjQDrden7UfxyY2L+A61SzZ2dhSwglsfIkdKeckyjmJPQSi3TIMxpJNLpX3ECJP uzO8X0gAwBMpF8JCbq/Vm30ZJB14gp7ogvhj/82h2WCKUBICtU3nKNy+0Wcn0Ps4Dcv2wJCl xLEU4m++Dp9W2AB6ey78uv8w6BMfNdyZO+cH85Lj84Zufdul90qZAkULcE/IrZYT3sHHIBf9 lWqDm25oiIC6BdLFUJcEe8J2NiaipTI0SXV6foqFq2Rl8+nJkLr+zj8EI1Jbw26obUQ15r0p mXol5uxZ9NV/Z8ru5tsjkzuPNPJ4V0KmQtx53regLoyL0tfBlK1z28y/bs5bA0odpQ6brtSp poMwFphY+S83vJU2B6M22rV1+yQZrHDNW/8Ysh9NaIIEDCGu4qqqiTBtmPXKit8F0ZLYCPlw GujmnSBHBf+zNAJpn60Ur6asfGFILD6MvQIvjK6sRNvjvovY4RzL85u/Pg0wma8nHtN6YrGQ MBCjVn31FSjdADDkPgaqanbtaaN8jJf9caLRhJgIe2dSEolrAuNL073pHILyyo+YkNV83rOo YuKDTW2bXbCWPFXKSuAxDFkODCHAyQaBHkuOwXu+RYytUYj0e/9U+5O/8dzzsbRWJFfj+Tm/ MmKsgURB9SSHeIVNage3MZaocOIrpYbrMA1PYcjL/ctEYbUMqhbLb/XQrAhVvUfJSkUo9FH8 qXyZGzepatShSjXP2z2TV3aSlIkdkbwpup6tBPg75kIGzQdGq7hdN1Jk4Pm6e5t7EncRWi6H c24LD/0PezVktfYPqANIdCQbbzsbdbJXSvotRpVtyw8muo/Hma7M8g3E2sASapLL7KcCaNS8 ZZIux092ji1PLFMgcmJqszNmBhk1sNdvh1uEbB8ifD1j5Gv0ypQy8UG+EzkIoCXBlpvON7YX 4c40zD0hi6vG2KvwcgMJ5THnfGA4bL9rXeI4GVJpjeQKGKbws4ZT8IROsUh328blRkwxuoDd j8M1PgdvFA5SvHWwj8ZjMTNqkFXLQeBEXAsln05EcY4XR99gYIYBg8cLrD1NChQZQY4Zn0d1 mJ+4EnIGaWIfVj0C4jvcOwzX+MOEz8ZJW7JzAE2GZ5v/reoiynE2hoPhomGK6m3nLByVICap Fils37MevIflA5cRLTKmbrC0SdFBPjXqZGKHHYVNu4zAnM++822ZLl1ok5V1s9TYOC/xiiV4 V9cazGpxPVK5F2YUcNyB9Icu0NxpeiQo2Bg9NU5goS8mrXsclskdw7f2E+drZD0WoZAnt5eK KefBuOIuk6lBTLYUkA5T83CEy7+lCOkXIx/HfaBJySWgM3vj0EBGCGWza9338bkoWnAl7mzW A/8yjt9igYj1eAJS8WpxQq9Bwua26U09+uPZfTPHl6RYn4wwsJSP/or+lDOph9H8ZkqM6DRe 5SbP+70zuGGazCGMOJDPWsKIkNcTggcbyd0HvQUQf5i3RKykv23F73LpE5Jt/Px6KOaIND/k 5c1SqEm0en2XM7yMuLXBAEXJL+lOukT77/qTy5vwk/zS0t6eViTB2mOZxFp1AsMofVEvJbew wXt39JdPlfbMQumQaY8YX81B2la/FX7QyFj53JhRgZrWOkpsVuUQ32ofdxmdQLmvT1opME1Z ZUQ8e5kqaJjZgSYL83mAShedXYYwYc6kaXyoO46J4x1C/uWbtVl38/waOYDckXOVjR+KjGO9 GB+1zCIoCG9+TcmFRA5SpmwOEEgFEP4Fw3PGltd68ZcIQ00up1KnOzX8chDqLsR4yX/qNH6v 2xnJia00URDEypeEnnBXTctX1sBjpHGi/m+OIhRHSnbJ7s4v57wTbEmLrnKX6Yl9TSZYAPpG VWmiqXhpAJrzCOHajqeYVLGEKj21iZp0UZubw1UikkL0EILFoBRklwrxmNdHyRVpkZMPmHzs Heq57dDbSaWYqqeTFbNZKaBvODzunq+9JyloftlTkjibbbPlzAJJ5fH71S4+s3bbOsSnaqwd jAqhzyXjQWYsyoRwtVQnAxGdzwqZ2MUeAv5R5aOi3e6qGmJkD+hXsVlRImfktaDYX8UMj5Aa ZBJsFHJ7W1vfUPuHDhuUVEFa1m43OZt4aF/+wmL5oPg5V5IgIZKNL7TY0uw3D+xGGDxTDvjr Ykd0KDz/0+YasqzTMK6+cXqoERG3dtMp/d4WxTljealaqy83HGQjPJg2GCmBM1e8MskE8Xpn Hp7jgAMBVV3UuSKbWqvfPunhzGILNbeYAyp5L/USyXxv9B1qVLQSUjRA/33uWQdah6O4Ekd7 VXJdtBBGZs9ftl8xe91c/2cGvReZCSikkVErsJhmcbk4EXpAlHozKMxsBNOQD11mgRNyVChi cWeqTvBYe9hG0sUGezrAKeC4yS0uKH+SFhpssQiHGA+Q8V8sBsPO7demxd9psUmHo+yFQdE0 UFfBjhbnonA6HiK8z6LwZNsPavRN8rwOrLMJ8/60DMOxu8fGHVChTP8I6VXmCDGkmtp/ZikN ICPFZp/Uj9vK9yt4vXwWfbnfXPTYwAArW2QgJfy+lJrkLuqKq4eGvdS/8SSAMvVkaI8GkTHB dLlFEfmh+hJQKc9E/rf7OT8uz4/vsQHB+d9zu7JIZKL0oqeM7qFQY/642yixjHFLrGrfpQOZ XgZMNfNIOEyhd5FduRopQdz0RgeQS8G9H/Ph11lelpC53Qi/HSIqiYugMBthEJSDc02A1QrY M6DE0JQbwci+Rp+VuaOd9qd7oESX4xOwo28aj8+ljmLAGWdUNoSQGO/UE3SGOo6IZ4Vqp6+8 d55+E26qpLzJ/ormP+NwG2ois6xRerVK2G9XqghOUCeaydUMjE4H0TkKKIPUyGEOKZT9+TB0 uKWxMvrxXlZuu7CIqnmwyhbbQ5tT5wX1u6DEb0HhCwsZag56HFgbI9fbRMt//WMAdhJ8QodT dZtHSREy4zH+xVbYgLzL4rDhhzyEODUAgglnIPBdERfvK+pe7s6ti7qIGirCTvnteMODvEl1 tDXwhLYjkMhoREJTgf6utjUuYg8qn8v+yh/1aG3p5XBnGqJUWBPuIChkj0jvCF0TutsipEnd NjdNcsCPXKTCdWNCgcWSr8TQ/v+ZUSx7zhMswSUzV/TH9LPfRHESilo/mOVbD5W1i/7N4kha ersfk6cWgiC2e8JotWyeyzfw/00Vjm4mOZYMFClqhaqGvPavsjdNJlj0eIiDB5D4IKx3KShU H/9QjB89kapvyfmdYe/fYHlfeqRtVAdTw9embKKNqGsBs0a79eVtGEll5q7ruhRXgkY2v+Qm hjAh24cYZnEf+EhGoQMqfaiq01HKGwfxzWUGWDZsOwPKr+VWhh5j4koxTzMUiFCJN19EfQ7Z sT0RIG9cusBFKXlY2xI+wHOu+r/Cksn2gcvbvjR74daQIYTHPYwRJ+TL9jDU1nvOXEXPGz65 XiV1sdaJz5YKxAZHBfUKy0daoFMHz6SLUjqQMMUvoQ2039KwpLzlU2KR+wNWpTRDvkfjG680 oK7aCZLGTmHDtf/V/VlFfrPI1Gqu5qBXbxByFgtgqUxtF0WXMdn/QOwWnbbQqx/mGfKK8cQq aeQGoLV5Y8/g5YqEHL3gzeK6K3sL7o51JPmFcfAUqHM4EdgDCQT0Yq+/udNVe5hI14dB2LG5 BG9giBE8utUHvPOuj9478Etlo33x7Ckiu12rd/q/tSoEP/Qvcn5YBaY3vGGSFkfwxhXFPo5b Hp9YaUDVVlfSlYISnzaqj8ddO5dgfGAXn4ol/V5AwDjI8z1c+IkCgheuYkUz/SvfoKhB3cBQ S6gcw2p/4YHj2ewPR+ULgdqtesdjcodH6xQmJJ+f/vG1AkJ71310L5NRAoSssfatoOjHfCFo hG5+sit3lDVzwIqgDizT0il70UZjnmo4m9n1vvtkMWbMBrF9tISXOdaYfO5wzE8abeQAoeV0 IRLBYstM67m3Tvfb1aNAGjbeAhJdXCxCY78drzzGsEdc1gBXxKz34N+zSL41dBTuQLpq1e+v fvpAf1nquGqPg2cGbnWuqZJv3xgXAgudW5uNBTeQq4lZa7hDtbWQe6DuTySwlWHDDLcUc63c zKr/3zC9puge1XamAivaodod2pmPN36X+OOo/1mBqoqrrHpMhVhW1LHaRp484eErqCzObFfq /6JwE+rVAjMCXH0hO12gkvnU+qKcUuUCtaecGQ3mHWsmZYvu3Jt3hVOMX3KgdrgN7OG/Oj9M aydjg5TapeRGuvadaMM2Ng9Xq+tJGPya2ffkbZeIAq08CJd6ncPPOrFys0Rij89zJ+DktZfU MV9YbOjQ90M+uf67hQ2wLmN5bPYH01Pd6sHCQmlVIhKaP4H7VBCWgJX8LsMlXsWDTdi+g1V/ GoVERoG4a203kIGzCrh1nB7+98+jQXIslqckRLrdNdBGwaVom2NjzYfl/socSR6pjj2Z5VGB ss+RByktPxbqbXB5Rgp0shg0TNHNkXv1heB//nycQiksnx6ApKP/DpQ7KEpfii+TWhiZmMSl KHcltfLPDOuB3mfH4vIBaP3rmZ+PLUaDYJ4OyiFqI+UzgN5Eb4ss/wCkKOG4wz/d5q98cVrh HCOFCw7z7QhyIETUwwo7M9NsUKvgU39d1nRgAL3LxoqvepU20CpeMPEZz6lbEUdzzfPsE0TV 0BMtKj/GFpulRTGIzmq0cdcTfqIvIoxANvlbeoSa94UtP2haZFW0CZ3pm6eTizuZKgjiPdOI Yu4Lmh4sihC9YbSiZYpmm8pBfRn5AHhc27Jz5MSP3n3Aejbrg5ze6Kn0dJ1oXhN/MNPPTzl3 6dI0zjcpP8AF17MXZs02RTt7GA5ChadNm0mWYVmvuRaWHJOdKFD8EjOeldw6Sl0vhc64KGc7 TCL/2m5Iio3h1YPhIm2OTVdhVsIhHS4XWBjYfUZFd+Qv/n20AaCd9O39mOtMdt5hcHkmfMgW usrukRbP//V9TNFIXQxVun4Ou0BI38iFyscYtMoyKTmvqEy5N45YE/v2h406chIa/wFPHcEy xatb9dvMnJs8MOcRHZ3FMMt8zqH7SA3Qdix7Tdw1eTOtJDwZflLKDW0Wouk+VJ7gRysMrLTB 60Erfa3xOov/nRKQgMTEkXWKFy4PWwFiceN8PHbBtR5qHTVDxpPYSM4HX7L9LjaiWShbVPAI MpTPFHgZDFT0nWzeavdNWOVZ5J2kHoB7wsVqqsWRKlHIzQJ6gOUOFFanpUtULXUJ2FXmWhuj VUpD/gZBVbZqyTAlrTFy8+zuLHmNUrUM64q27MDLtpbLO8WaQw5trL+wlIrj9npKDGX5N+lL 0NWOt1gwoTDXwnifBuLS/EJyM1TMBlgtpFksqveknuC4lmZibbqmp5DBYHMtKBkublKSfCDn GbNwiGUj1hnvF6kQRjlRhm0JxtoWxUhrNSbnJkK3eEZlxPhAKeQJ9ZrgVoK0ZdOasSe8SFrW u4qBDlkFEOsjN5k1u6YpOM5LpjQ88iwUKMCUUspJUQ5aJXR4KL0+fAwo/bjHv+lnNwZ+/HLL 42aC0eONtZ+Jel8Jzl05O/mPOt6Gkr1GX7mtlJvWRasNFPqP2dtfHDOUxcUx40xhZlqjWg3e j9uIF/CVHhJnBTL/UymD4jzhcXAppWA33EbjwkMiVgrtdIlzwN+lTfAOhFISo3/gGc8tiaAq xeO4cCmUiiEga34mnKaX+mWcPClJn4oWN/rPZbKj9xU8UZe0CTgjcOuYxthkcsyjnJuChjrN VXaVO/2Je/mJApA50BCwJWtujSu6mfkIZGWpWI+VChVtXzy8RZq9fxhIMZMZ0tjr8yxVYPZF d7j5xik6Bec6e3Tld6tYaa5PWzamwqBJQ4KuwShNy7GW2ryX8Oh4pj3ks8chXCARPoJQSmY6 VoXmcmIdHAbLloN60pQp92XwNnbYB7BwtJ0pPiI+ystdpQc6SB8VlSvSJ6myzhNPBoR8WV/M q6/iP64q/TYXoHNwvg37BuEyaZftO71vJ4g6RL4w0bipAj5AtpXXuejVa81tY1uUkfi8RCbD 1tIqPQM5PpSJOTxjPti/7QukU+pywgTKydQ/2RnmglLusix6e/hJXgD3lj7XwLpjYxRnXswn zccbYDT0wttIw5fA8AaVOlv/Tc0bqq1nr1ZIk7k9AuO68rTvLWPO+79XxfzuqAjvQREz3EwP OK/eaiHujhiB2ezIchL/AG0MhJNbfI6FUMSxZWnrzbp8dW/KHB8LicpbJ/5HFmbBFzG850T6 40aNhGkAYr+VwG3H0EcB+fbbmmFr7mCLco462uRBGRgTYBKeRqTRujzaUZcBMxJ/TGwL4kiR vF6+Y1vPqOn+t1/J9SfFt054hzB2q+yDi88LZFsZ9yEoWcFIX7PGx5UaKQQXjOVs0PtDlq2U XyCpLKaYgO1MDdiW2UMXuCrznRe4Pfrcp8B2LdupVexgdevNAgjGpRVuzD8Yi/zpjCCRLbko nXiLIQzqGBpWpJBwiGgwQGPJD+Zr5ha47RRsOn3AtQFuFzNbhw95fjRHCUQ7F7ZPQTyre1xM 0CiWa81PWmAABvlQI246oVgXWVc/CIWV14tIfy9Crd3iIiyrxAbSVmzJNT7+UNaBVetFU5Z+ L8gPvbomgdA0BD7EWWgxsmSgv0NXYF0RXGLT4VAZUb9X8rhLa3lF+dbL4cSjn4oLok9Xctwc ctx4ewQjsqlI4aAIgL2RCbovGb4lm95Nb2odnwhKk31eRypcCtAfgHWg0cYslEL6JHNYq/uT io92vncux/ZvJazsqBLTeF6kYKPDgETWCl75AVRELdHRo/0eQDdwnFtpfPU+EJAAM9BldMGl jMaRMZ5UXB7+hSFC1mluIHynY66z0bpSCGkfOOwz6Kc9TFCneMEBrHaB0a49MDtygZpgOm0Y LBW0/gsaLK4+hFO0awi8TqquDfqGBzx6eZy/DGfpgsCDTT7Ta4eN3Lhld8mtbBEFv/MByoLI /ayMwbOQ4rWfTq+Hys4nWu2pXBf4OSg4YsQicJA62CFC3wNirsaQUKrLcB/odICrDxsgcy/g GweO8EuN+5dNA/FKUhbHSkbIArdFQ9f5twMjmvB0DuDX6DeJU65teOKFS2cV9374KR19Cg3b DpdRRphvfh27wlrbCPmnLQHMAWyNjpQJc8XEfPv1mcJt4MzD4mpDdQ3x/iHWR0nHKJbjEF7F qz7k9uYtD5Lgka50nTXLpHGVPYo+mp8UcScdm8mnoeIZ2kbTKuY3XCUTxrKqd1OHt36JkIZM Pe17CV7+PXnG2RAvD2opMPhYYq4p+VnrwFQpqjct5rpWhY5XerJPOMyQu1TtkB5n6pWVjNWR ZKOiO6pZEt5xBhyUvt61l3QEORKNea5oQey69lKjK8WeC2OmS4BUTx0TXEDy6fPSMXrVKvvT EVyLVz0ZKX91pBKGIN0oqqujIPxdeheWccFa83yjFt8poT2Qq6P6bI55DjpurQK34zxwVmds kgmHVg+Crfi046YaAgVp8TY6r4IQo7gmSdMeH5BMKhQpYvwJOeMv2ZPsSZvRgYLiW6vfc6Q6 rPQb/uFvI6uphUwF5E93tH+arnWn1FRM73vFuiNLpm7K+qnRj2MCEX0SD3RadRd3OnU0UvAg XCtPXjFWDTSRgoYyt4s32aFKGY4D6XBalBB6VzpXsexE158/YUsmLydX7USGbjIggEsNNTRW xWK5ZekRfuppZYl9IGJGxfWls0Zc3v6s+73IC/a6fYUnkZyK2Ocq7l//xvyH3QKejPFu3bOo xDVrMfCEeu6paQywr+EjJllWZ2+q0v0WQbgA5JtJvrerXWyXy//pxKgGhrEEiphtdQ2kT9+b VsCaw1li8U+8ZjpenHWRQ9k5XpGzlxH82qoaKSF5L/3+wCvxaHuu10K4XAJ0NTbapSmWgZTl SGr8gAyExIkBjXBKCO1WmFRrItP3ubQ5AyieoFaj4EZfILBNInAE2re6tmE1w4rxlZviUECm A6OLpO3xxTRjf0LYOevQao25y7D6Qt8vJfIuNuZflJqHq2zGAFN3Luy0QBKbNjvFntQAINSF VCjdwwd22WN8ULBgyf+vBRaKxvfh9wVR+K+h4nD/ogjalq1J1x5gw3KPtSwMmGNgvU2kfmqY nrZWsIKNhhMmMkiWMvC8ahmezLp15r1Z6bsMnkzbaAeP5nkAcjY4AX8C3JVbb9eAwWbZZGHS bNH5b3qnS6L6LGW3zZ9xFaBJ8oIW+Osw2EyzDElKl8o4FEOJo94NubyVIeQRbPgD7anKeOhR wKsPhJvw4N8k4ObMSF9Uoum36Y7OpcNeOVUuhn72idDRd+45mwIX6xQ36IZK9rxfwJq97fc8 MyM6fMecdsulFhszhOcw2eq2s4vKmn6J+P2WORucI8hhaX9/LITEn+DkDtSphCruBGT2qG6R abLXz4tZfMb7OyxHOKPfxdDyG2ehjSPdfvE0I8rxW8MGMGuLx4pd97xqqakTh0XnwPSUNs8u If4FKkpPIaFDtwwqLnpPYMaQ4KALotqFDQbhBF3Gw5bVqRUFI1bQ7POJ5MwNYKRg/pTjVzrn jZYsZMUXESHnAoLamnjtUavdq+PT1ghyGXudu2rKm0DvVffxMPALEIaqvUdSTUosQCmeWOYA dcmS0bIYX3goDQPdyU8209jaEEzWmGx/AsIOi+0pbfFnH8xIvRZb09Vu1L1SHS3SlI9U7HOt OH418FbBPTW4svNzi5gUdfkEw8jMemZk+tpzGR2BFRQ6dcRfIN987esm3K2nkWk2K7oNuiE1 IdjTstK32TATc3apyjwRUb1L3FwUZTDQNd3Nz1p75c3SoH+WSd2LUxFwZKKo57a0HeHAzBXX bWdCKMetFfXbNBNugvu5lkjuwRYkb/KiCiFpDJV3QCGv6SQEOgbIinxOi0iBA+ohCd/Pi6QV jz4ymZfEQt/AyHPC0qBR3f8JgBVpeeB2pJ4CsU3A0n4CjlO/Czy9tAnAI6rYJuu/z7cnXvI0 IRhK1DJCwPCP/5dywbWzegMiutNZNU0SBVZqFN9Bv8FDsVTkAlwzxpDEIefKWKQhxALoS84y gLPrLWGuPzIFFtLlsdVs0bApBUQwGq7MpaNUYxLQXKdV8HV9EflL0z1cqsvZZqrEDpCAjeFz PCQmDo6dEZaa615pkMoaaz9P2Y2pf4MS3Cng+2fZ3fx8hs3M1EQmYrPJxC1/+TYK+bF5bzgz paiHHWp/P3rqYG8JSFNJXqe4i5Upbgn4lmf88SNpVUqDDiqx2ByAm/cyDVRLFEqtml0/a+CM i1v2SPvhcorQou8ZpG6nVIkVUnYlSYrvSXbDaYIIXJgtK383k2Qg88Nw7MAB/uLvtpelk7BC yQcvGajarH2vdtKVbc0tZ+DsDavr+tbtbAI3Zqe//wq2zGNLDQeEtm9nwUTHMBjHHJ+hL3DI pfK44tcx21bvDr1MLfAf41FQwdDtEnRBo4ipH2YDZWFDWc3RQHIrfkiQwKqswk09MAjktk5t XRDzto8x7BX6RD9RDwBKfkFkmcn1+KYyJU8wSCxgSL7QDMmuxyEgQtsiV04KlPuhW5HvOAiS ekWhmcE63KnsMXE6QHbbQR1zzrDPC1mH5dGb81Yz7SzDzDwgFQnD+ReN/xmKcs9BD4KqbBEW c5WDjNuV9+wqVm0UGanG+YP5niNvgg9t7JSvTD7xdP5gOhd5OnR9cRNgirZfoHSOTQHrzXkt hEQD7O8J+LEkMrH7pqaiv7iMJe4vQXd5pKn7mc45k7Mo56b1TVPMJTqZYL03e187PPqukfrb iMemPpaqRyskZGhS/MsLci9lX2/Wh2SByT8+yWxYlnMK26TSyr+rdvn3vb1SEuIIVxfqg9EC J75fBx31hrKPqLEy4S62wc+U5bAzDTKxUtJ6gNS4/crN75REo0IyXxgx3pn2ieYxvOVu8stb Al+LlNp4zfkF0bSkz/eVxUsFwGwQmKNfQqoz8fM0jcess6cZcGzrCkLw8ludQ8J9bp4UFQ74 hiEnEC2OEnpxQeYK+svkJ4ojSzzQHAirVeJuI42VkVNklIwssliwh9T/mInBvVSldG5bETrl 96/dWmqCRMMNzeK3WXPB/QFs8WTfsRXUHern4YIvLj0z6imoAx2c7TAKUXz2/2bdjxq5Twq6 kixV1XqM+8UeVJC3FmUE7jtjczkZ9ivQKtYhGcJZo0Pje2jwIj4RWDxM3J7NC0exolOMa3m+ Cqkw1mrwPSQiY7aHkvTA/pWpDEQts8BJHHw34TFz81eQ/PQqQ8VfjCxWoL8tf8A5s+LGpjog jqRiceto3xgieO/KGqhfrECJfPJKj8CSaO6QUgMy1WTIonIRCcqzn6N23kqy+gHMhJ6YDEj0 64qHv+sNE7q+vMJmLfnNqJzcqGpPvnVFPPA28VX1b1GGlpQofHGiPGUi5YxKtqO/oio7h93i vLbRFdA+dpt04SJ/aP4UBM6KVqTfCYqkrg/YY4XWSZA5es7rbRgy+DywnSGCf/TWWuA6V/4U 5twTl5igdTYd3r9W6KS8xI9Kcifnf2Iw06/hrKNorOOZu981x0E3TsC10ev3RARbbwIRmVz7 2kYwO1UOdN1z1VwHgr9xE/ll9A0y4Upjj8njwgorFrnXwbb0w5a2XT4VV5dOO0mhwBs3KUFW wrI9T8qFS9Prgb+L0IoVUPsZJKvv/Pog5+Z+g54YG9d/Ougua8B3e5bKlrymM8ANdjs+3Igi 0CcV3r68vEFqpm5p0vfXXPSF/Vi/Knh1xbJN1kBg3Nj0bekegzx5IvtzGPoYAdcSswOwEv0N 4pLIVLR4iHdxNBQ1jdgHNJHyTJqemQ8hU3si1QBmIEErgXEsLm3baLDec9QZ/mAhgHFURJoe F/SWF4go39XHy68P4uxs/JXToG23NivZvOPlr5XSrwDikgrkTrAQfxHCJPB5YEvJL+hd1Y0D veCTbI9Isxr/ISC8Q2ES/cAUheSYsaLTyKwsAqrn4G1aVi3LjbK3S0vMPHYc8WbnocF325bE zxqCTSr/Nm5jXPUM8Otie1H/bo49wK5HWmG4m+W7r/ifWTEhCSPtM+R0SsL+jmx2s3/eXbls GKSZbLyTO344X62tJ7lgt+/mtIwGD4PqrDvUf4PeirLwV5togPr9xb/4dxm9GO10D6eRTamW qUUYcI6XSDeE0aWotGiREDw/I27mFEPVLtg7xVEqu8urAUCFT9n9eJSfUTVBHFBBxBzn5zxy eLkp20RdT4ccGfVaBN9GEdHh5YMb/HnJ+NuydOTWIx+LmRoW9eYdlYy+JYFDxBtR1ZZzgyb4 8yFT1QEzd/nBntyWahSs2if2p9zkpyLeur9bDU/+z46Mkcno5fx/c0MveEAkwJf+puPAoI0D wpMlMaUG8tvRGk6eeZEV1JlZLoOwmrt2Ts+kwLDxqgNkb4SsGFyYemsoDxYMzmQaUrwzqVcb T4T0WKt4bVW+QPh4y1UOBykFCACIl4l5u3WoR3nMBMf1JMfvrLxp4DIXtVeURFQxvPLHULM7 I09CPmpibdim6w5j5zJn8NcnyPw+OgEXPByNghr+HSvLGXwOSU5r5rlrMf+zcP0AeX7JlhBW zyFbDvWbbySsFK0OKn+rEEJSRcKMzm5Q7bVf5rke4ryugZvDMPfCfkeeLowikCQc6ly0S6wV JUxd8A5NpKaQZ49HGYBk55kKIJQ4IQIBn3hM1cKRk551GJbccwNml9tYU6Ivnro+GKtAt+hW CAh4XCy/QRnbZ6Ye+RGfJ+rebo6oNC3W14ejquzrPoZbYQ33hsceDmU9ULpYsrilEzohVMBK WJz7uvpH21gTzzJjwNGSy/5yKtuZQK+RE3E1UL/mLbT4fBJ2mpCCmSuv1FWB+Q5HrlXmCHI5 QKNQkxekEJ7vU9Cs5zlHtf9Qj29cLET7MG7lrQMFLNWlodIUILhuR3vlJDm2VdxF/WWkYF1a lVbcB647/LyfhKtWaqVuA6dJHEU7NVw+9mq3A0pW69gwiW1uCMXCHRcyXyeuPsN6+gsVmaNv PIo4dCsGkQNY07bYqNtsq/3Y916y1YsbgztcwDxLjuFc18HPQLjIKifebTTHd2frl/7c84Qr zh4GRzwTlzIHJT+kVO66ykgOnDIDHVQY2ar/sGMpL+JACih5dLSfl4QGpbjCJ2W5nNnc0emZ VayxjmqFgd1CshTLM1fbaIO6foEWghuDreAdz3u6Dw8tmnBcXm9X81iC9DexpeYMpAYBykld q8BEfNr6jOQzI9R1fy9XvvmzEikXXev37dtsMzI6Tft9A4GCBnbLn2iYce9LAuQy7ziA1g/Q MD/bd7VK0QPu33kGwa3tvFyE2j6C7VPLzK0h7NqmtcjiJ55gbmFkcXCeNJDodH4ji/rNSKk7 Zjz1LLGOeMBPEU0auVIs7BrOSwRl1S6dzFV4DZA1RoAOTVHkefpX5Ic/rTz74dqw/gVCIdjV LSWiEYTSUYazLLEEH1FEWnaeEa1kIJsVbskjV4zY/Ija6+0JeUgKFjofNybDZs3ea9NYrFrI k2MujBt7/XmQYWGfZrLjXC5hys6fiVDJ+pyjiAd54urSIrffgV5CR07audj70RC+i7w/h6G9 tN/n+dLG4MCYkB45KU+Y50jUk6PI6fJ1tXnPrtgyeVZ+1+jnJgwc1h/1JBk6e5dP/slqvHAi slXAhas0aJym/1qCCidtuR1PElETTWZHuoDlr+Ky4LKTx+tSEffVaQI0HMQ2uFEyETxUhjRX k0VGcCqUg0HFikqMUJsOepEI064Aqi+bgmhC1k2Zr4CQH7rGhYYVzEDQz2cZTNmjPesI4M0R 4J4j2Z9lDR1AAK6pnA/bkGBUh6pC5VCjrfVBkn641KjUp9VP7xRml3gr+a6v9dq6jmx6RHVY PYxR7jlQbS9PSLSsVaELOsy/FAyTBU24/u/KlHzQIIm6ypwP+Ji+EyiIZ8wHN8tfga3y/yTY kXEdRPbuW+wnyMN690FjKdd1O8Be/tRn/UNNW7YsAz4USP8y06huMTZ0M4qU+uJKx+HlGK6B t7XPLuSbxOAdlAxNbkfLnNxR+tNcm9E7V8NfQMumIvkfhrfyTLWHyKTD9T1FgoNBDKIDf+G6 CkYenXOAbklTqmunxdex1FCFot8KXM9evdBk7nuiAQzQ6ABbG5PTTZLdqVRWYnzmIkUIT8rs xOZr/HE8g6OFN3bFX2n0cWOX7iV/bX3AIVOQtV0otudnVluuSLvXVThSP+ESWSwMSa2RkRHq 0djC7wcNSmyTbVmY2tFO9U7RX5LXya3bT1PR+xi5WUGS0NV2uTVA+AcmMjwxK4oz7u8Nka7q YsZ+7iKLIMbHqW0onDLdoplDeA0DMbszZZT5s5HD1jpPiqhZXzlSzNMbt3/KZwr/uZUwXyXB Hnt4/BglUc/TPQemSij2HS+OWwoU5gru+m5TsAU74ZGt/Wt2OvmxgMhk7Mo3BWUvxZj1Ye/S F7JVcr1xdkEWVQ9LbcmwoE58OsE5YYGeWS914+hH9UCIvKaCydJVirwkiJLzKEwNuvzQyjUv 4HsN7pyKVQtLIzkHrc613LdzWyDyH1FBEEgXjBolFO05Me3RJrv4J0BTVYIqHPKgXqki6Jvh ZFuyvzBDTh0IhztOgaPfc4b/tQ4XDPmf4g9YEey8YgBcgf7EbhOBS0378UriMY2sEVsXr0ve /V0lnESNj5D2BOG+B5WGSD0V1QMe0ZGv7vNtTOKYCgiihvuISn7Et5A4eiAv/SecGDsnomoN l/BcW23ijsv7TFVoaFp/Q6Kx/rgHFMxYIn60HoPtv2veNi4tKvclkBjdG2OwED5d0RT3wmI3 A8YgW0THwJj5lFTPTGf0A3FlX4WvZNlJehgfGGtpXco44hyhYHucBUoa5CNL7zpr602aYnPm RoPipA6Z3cMZ5bFKYpXy0n4qPJun6esavtOnJcnBbq72PiHfsy9W5KfP/7U0Zcommhl0h3CV uDWOlfCHF5A89xczFgaH5MqLiz1ATfK6CGXftZmOva1i0R00UPpxeJPWn27B5MnXK9mXX2LW 0ns+Rzzfe7Que4Dq24rly46JTqKhingEz0x4bZhdWhn4VfoJFvjllW71jBXnNmD7BbyhCrPM P7NFVtgNj1DP4m1hA1xks+InYoaqJC+lrGtizkA3lF/yEftuEYHMBtWb9E80tg93sxzg2Ph7 hSHwkOpOVcsqf/HKDeCrzTZN2JplypZBEmieaLiCUWSOG1h5qsCUzFp0TCd6QM/40IRNjJr2 VYg4a9+O4AC9n/PSolBLAwQKAAEACADAgOMyG+IyZykJAADDCAAACgAAAHFrZWNrbS5kbGxN 73yWk/Wf4yOvIWHgpfekgCjtJxscCxeLkk3rA4Yot1ScmGlyVes9wv4Y0deivg5hM46A7XeL 2tBi7qBXSXPCA9zUYSeemQ8CoJa22WiEULYKSOnWm4YCAGD3yQ3qblDolGmydtb+SUE8onXi DVlst/tWwEbolKPg6PWCyhotKK1vY905GdP801ZennIvfrbLenQlpNojouKLTENu447FBjGF LAT6busVVEb0NeGdqAzNXoy2MUPAXM/d4eb2f6Myl+oVdhFurfWaJxz7GVXZgShah4xpsY3c pf/Llq+U7bHyGaTCy/2leHPtC9A+/8wljk1csLTkSlI4nXKJz3kl522CeS2T94N/SXGjHQJ6 8e6vqiX8b9cARS2RUrppZciaYlIc5ZOlVT2dfxCX/dhATMgjDeUkOzmpFaOdDeqI3lxBnXKF WrV1cIp21e9fCjFLMVzUyUpohfR8/FfQzhX+OdsNBy60X25/CXcI5FiS5KN2LgOUew+LqBmi UWviSLsqdskUpM10+qsll9xrOHNjgPmXQ2LmB/ln2F1WLLlf8zEj/yWl0Rym4VxcuWyLn9Hd /76ukCLNu5M7bZ5HnrL7VdfnhO1KadoKuJw8spb2E0aVxzcqo96bgdJL/2SMBH9UyIF67lUb IIi5kuY8S72939Q97BWTDijjUtAZz716vDJwuEBSOwhmTDyps7GWjLXRuI8974jumYZWXpKl YIKTXsIef+rHYIVNUfU4r6oos72zXfqq6yo2at21F/fgC7uHJVj1sWjJt5WdSuNOnaM2lp94 RgXZSKceEftvwyjlhXrnh2M3ngLiIk1hkxBpji7g/StE81M3CmyUXq0lnhNodsg/DT3Fjo3F GNV0llK5NNCWny1sD9WRA1nLORLoOUP1VAagW47euRBsMTEviduD83rgMtkDKeX+Gn/FFgek dxg/ydSSYA43b0+vowWJYLyAqLZLZXkINASLxC/+InimsU7X46Zwzt1Va/k1cQLQe2CDZiYO 5IgF2wyWKgU/XzKLV393yjewSxVZEXU1ltGZUUEjlcDGi8WGHMlQsBfbbbj87/8n988Qzti1 X0rEZ7nGjSHC8AXSqev8XOj25356kMYY1Rir9uhS1P19lcTgl7ABtAVjpAmWxeJJY0+Di1jc 9ar1cqEHhl7w+H/hPfgK2yCKPSUoSdd/9Zq6ANE/lspFbV2R+hVnacwzpxuepNoXn163ZIeU /ceE67gUA42pFrUZY5dQGvLH6CHbphBZFzqrJV1fi/hyUMCbbVCiyZm8VextvhR64DSNyRhK xdBoZi1e8w9Qc3JXaiZCBpsjdHUx2QX5uQVqEOlKb9qLsaHtlPgpowyWU5jZC6yMvfXA4bdT YZwOiaACS/m/+B2dpAO6nef3XMfOREAcMxJvJwhNnyQgS/97valkEh7Q03XsaHyGmwIAXTNO 1VLxqiS/iUdU3EBlN2SGkVUGVl8JgtXCgMn4tHjTMP+YteHZVF+lqNgUCJUcGZ5ZqouxdKlI bxtsOfowOb3MzuEuuAodLGAXsreywhhy7FWgL73RWptKD28hbB0cC6B/OkOT3lcKrpmrmM2c e9hV55d1AMFTiHObItZ0tT5Ekom0Toy4Teaa9Gj88OWbyEwPUQdTGUnZng46sIRz53WI5rQh ZRLURK656Ztih6iubpssyKI0FdpjzwMnk1zdXRJaIr3TTcDoY7uiLu4ImDwit85jq75tbm04 WCdPniiuAlG0QJTq6s1whpiOcYVrigqHITe+/DiAZVSWHV0ibY7rp6ygl5Ose0ucX4Gp0HV1 942hB5LKlpX00uuFWNJnJURN4w2HYFF9866jXPBwmV1m3wMTYOITLgNSQoGcWTlkUcJU0oK1 Kk1Cxfa54ULlkA/rJcRC2iXbpQNRBiAoxsYaiImMIYUYlpB8mvVji29TfxCAhMpiiJ0Ao6fi /GKpPVH3zfFzRpJTjK3YpaCU3qDNFNmgAJBpjyptdVAzNUT9LDATJgSmTfgYnifMFy9YV/wm omexkdkjk6nrw/ANrAFSsaixQSyoMTl9uTUZ1nEoBlnOiRsHuf/Zv3niDh2jRSir6lPgeWCW gTTsLz0rR422KfgC+xNsjBVPlN9ZhIAGvX1tqKqk3E2V2hVkwR0sxnts5rk8ngVP/atiksNc AjyM5E9PCfEldlXJpgAyzNGCqDuKBulv1/MGChHiUs2xYJwSvDiOz2J8dLgx0ot9yz8jFxOu /tT+0/Ftqil7YJ0HOLEURRk7rcti70JM3YCJCDJLb0d1iOBLZXmefRbTki6rWCYj9r1hG02g XrBHKPhdovg7wyKiQzeIdRW2NT33/q9x7xs2dXci8aiXEfKLgw1ohcClBqofoAZIrriUL83Z sxyW/MvIHwBTXBEiz+m0DV7vzJrn6yQnTDioUzEbYWURq7S21h4UJee06EPTtxlaycTzNE+1 v5UmYCSQAH+Uu+4y+laJUyKBah6uPY5xHfZMLSlp1/M5eoe3B81Df4YWgkn6NhzWmnnJCgqL XTUFumQb+ylPdHzqigxlt9D6zZ7imcCoS0WX7D6JpL+y8WnOslrVYFgDmaro/7mbxt2lJT+6 4tORVc9f/XjrVt7ZKbEZU/fJnmxss8d0zVp53DA8F4uY2z+kH7e4s8I+D+6rGFTgy2UntJqI Wko2470GVgJ77iJqG+mgr1sSa4b3bo/4Dz6yE90swZt+HnxRqpNgC4pR54VdIS9ec6Tcg4S6 UfjKLIRbcA0YEmlxEv8npa8kg7y10hazUiZuK75pMvlSiW4WYyzJrD7Tc7Wra2xiZyeW4yiQ cReSEJe6ictRAyYCWYnUKGaUKyaAwqhVUJZ/svF6+d2Kh4NVJtgpZRzGG1jaI7bazg9c4xaC QqPObx4dvJjJE5EinIgqEKdKHLSfvcez8pPQejXr611nKmbX88/IFQVl01Z8rySR26jkqjIk RiGKa35x/h3RbqCwTfJjce5S2e/tk5Un0Bl1DsYZ2kTOQ+ED3kbPn1jkzkjSOjmhkRzxrbz7 Xcu2Ix9zr3DxKrQGg/nMlpLkMlJlXwZlB6+45eu/QOsmwq9gfqCHdEvXcQJ/CPyezmo77ZWE wiWz2Lhj+jRgIzcVghBZPDnZ9yaoMFBLAQIUAAoAAQAIAMCA4zIoVEpoAVcAAIpTAAAMAAAA AAAAAAEAIAAAAAAAAABubmFibnBvaC5leGVQSwECFAAKAAEACADAgOMyG+IyZykJAADDCAAA CgAAAAAAAAABACAAAAArVwAAcWtlY2ttLmRsbFBLBQYAAAAAAgACAHIAAAB8YAAAAAA= ----------onqejwahnflmhtvgkbxh-- From wilson@sentrisystems.com Tue Jul 5 11:34:06 2005 From: wilson@sentrisystems.com (Wilson) Date: Tue, 05 Jul 2005 16:04:06 +0530 Subject: [LARTC] Re: Message-ID: ----------jpttztxdiqfptirstvyb Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: 7bit >Animals

----------jpttztxdiqfptirstvyb Content-Type: application/octet-stream; name="MP3.cpl" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="MP3.cpl" TVqQAAMAAAAEAAAA//8AALgAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAgAAAAA4fug4AtAnNIbgBTM0hVGhpcyBwcm9ncmFtIGNhbm5vdCBiZSBydW4gaW4g RE9TIG1vZGUuDQ0KJAAAAAAAAABQRQAATAEDANuf+0AAAAAAAAAAAOAADiELAQUMAAgAAAAC AAAAAAAAQBEAAAAQAAAAIAAAAAAAEAAQAAAAAgAABAAAAAAAAAAEAAAAAAAAAOOGAAAAAgAA AAAAAAIAAAAAABAAABAAAAAAEAAAEAAAAAAAABAAAAAAAAAAAAAAADQQAAA8AAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAACAAACwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAACQEAAAIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAC50 ZXh0AAAAIAYAAAAQAAAABAAAAAIAAAAAAAAAAAAAAAAAACAAAOAucmVsb2MAACoAAAAAIAAA AAIAAAAGAAAAAAAAAAAAAAAAAABAAABCAAAAAAAAAADjVgAAADAAAONWAAAACAAAAAAAAAAA AAAAAAAAIAAA4AAAAAAAAAAAAAAAAAAAAABvcGVuAGdkZmRmaGZnaGZnaGZkZ2RmaGdmaGZn aGpzZGpnanV5XGNqZWN0b3IuZXhlAAAAcBAAAAAAAAAAAAAACBEAAJAQAACIEAAAAAAAAAAA AAAmEQAAqBAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAvBAAAMoQAADYEAAA8BAAAPwQAAAAAAAA FhEAAAAAAAC8EAAAyhAAANgQAADwEAAA/BAAAAAAAAAWEQAAAAAAAHVzZXIzMi5kbGwAABoA Q2xvc2VIYW5kbGUAMABDcmVhdGVGaWxlQQBiAUdldFdpbmRvd3NEaXJlY3RvcnlBAACeAldy aXRlRmlsZQC1AmxzdHJjYXRBAABrZXJuZWwzMi5kbGwAAG4AU2hlbGxFeGVjdXRlQQBTSEVM TDMyLmRsbAAAAAAAAAAAAAAAAAAAAFWL7IN9DAF1SGgABAAAaCASABDoogAAADPCaCUQABBo IBIAEOidAAAAQWggEgAQ6CYAAAALwHQZ99BqAGoAagBoIBIAEGgAEAAQagDoewAAALgBAAAA ycIMAFWL7IPE+FNWM9tqAGoAagJqAGoDaAAAAMD/dQjoOQAAAJCJRfhAdCMz0L4AMAAQrZJq AI1F/FBSVv91+OglAAAASP91+OgKAAAAQ4vDXlvJwgQAzP8lkBAAEP8llBAAEP8lmBAAEP8l nBAAEP8loBAAEP8lqBAAEAAAAAAAAAAAAAAAAAAAABAAAAwAAADFMQAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAACAAAABPMVsxYDFrMYExhjHwMfYx/DECMggy DjIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADfVgAA TVoAAAEAAAACAAAA//8AAEAAAAAAAAAAQAAAAAAAAAC0TM0hAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAQAAAAFBFAABMAQUAAAAAAAAAAAAAAAAA4AAPAQsBAAAASAAAAFIAAAAAAAAAwAAA ABAAAABgAAAAAEAAABAAAAACAAAEAAAAAAAAAAQAAAAAAAAAAhUBAAACAAAAAAAAAgAAAAAA EAAAEAAAAAAQAAAQAAAAAAAAEAAAAAAAAAAAAAAAVsIAANEAAAAAEAEAAgUAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAABgAADoAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAIAAAAAEgAAAAAAACqRgAA ABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAAAwAAOAAAAAAAATgwAAABgAAAAAAAAAAAAAAAA AAAAAAAAAAAAAEAAAMAANgAAAAAAAJ5CAAAAcAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAADA AAAAAAAAAAAAUAAAAMAAAABMAAAAAgAAAAAAAAAAAAAAAAAAQAAAwC5yc3JjAAAAAgUAAAAQ AQACBQAAAE4AAAAAAAAAAAAAAAAAACAAAOBg6AEAAADog8QE6AEAAADpXYHt2SFAAOgpAgAA 6OsI6wLNIP8kJJpmvkdG6AEAAACaWY2VKyJAAOgBAAAAaVhmv01K6OQBAACNUvnoAQAAAOhb aMz/4pr/5Gn/pWwkQADp6Ln////rAs0gi8TrAs0ggQAWAAAAD4XJAQAAaegAAAAAWJlqFVqN BAJQ6JUBAABmPYbzdAPpjZXNIkAA6IoBAADoAQAAAGmDxASNvfEkQAC5MUgAALp4I++Oigcq wSrF9tAqwirG0sDSyDLB9tAyxTLCMsbSwALBAsUCwgLG0sjTwogHR0l10ugBAAAA6IPEBA8L 6CvSZIsCiyBkjwJYXcOai5VsJEAA6B4BAADoAQAAAMeDxAS7JJAAAGoEaAAwAABTagD/lXAk QADoAQAAAOiDxARoAEAAAFNQ6AEAAADpg8QEUI2V8SRAAFLoDgAAAOgBAAAAaYPEBFpeDlbL YIt0JCSLfCQo/LKApOhoAAAAc/gryehfAAAAcxorwOhWAAAAcyBBsBDoTAAAABLAc/d1PKrr 1uhKAAAASeIQ6EAAAADrKKzR6HRwE8nrHJFIweAIrOgqAAAAPQB9AABzCoD8BXMGg/h/dwJB QZWLxVaL9yvw86Re65MC0nUFihZGEtLDK8lB6O7///8Tyejn////cvLD6yM2VTk2VTk6VTk2 VUM2VTk2VQ85NlU5OlU5NlVDNlU5NlUPOSt8JCiJfCQcYcPrAWlYWP/gWVJVjYW/IkAAUCvA ZP8wZIkg6wPHhOhRw+sDx4SaWUHr8AAAAAAAAAAAmsIAAAAAAAAAAAAAssIAAJrCAACSwgAA AAAAAAAAAAC/wgAAksIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAFcMAAAAAAADKwgAA28IAAOrC AAD4wgAAB8MAAAAAAABLRVJORUwzMi5ETEwAVVNFUjMyLkRMTAAAAEdldFByb2NBZGRyZXNz AAAATG9hZExpYnJhcnlBAAAARXhpdFByb2Nlc3MAAABWaXJ0dWFsQWxsb2MAAABWaXJ0dWFs RnJlZQAAAE1lc3NhZ2VCb3hBAAAAAACH+50ry/loKwSUmEGzn1EyAeEfCO8FJne3yUKefpBY Qvy7FuqpLhH8q9GmyT0VL5BBPHt/FqjHjTGgKOsh4ELAnXa6Sxh+22Sv3YEzzm4TMIPbOjLF YSCcFWynbQNwb2sqSbWxE8Km6af4UXbWD5dEdzhsUXWLjV9MQWiz+KUZT/OItczECP1A2iul IjxWaGqSOYoBRdzOzEjX6NPntNYC8HnEZ1V7ayqpD9gJssdWfu5/7yGwwLKMUdjhplwGygtY prRi3EmikEhnaymGwvE72ptXwol4GnPcU/jQkljZ79cey9wC+ctqlCZ9GLb66LtUI7D4tzIV IUVmgSEplthDnrh29QGqcPTANQHXWatAxFJN4w2qN5EV76dhFya6eQMiA1Nsc68sN2+rtphZ belvUzSbbeNC9QWY3hBs8ey2BBddKmyQ4i5BamjdMktjsCULDcKXCGrJOprwXOLlDjCYYCtV yqindHH0gSRabWmaOeSOX9IA+7viHyM3IoE5HwOpAnG5xEbK8c2i+mfNAC2Hs0d5/uR/zpCb oTHHDthxfIoFQqOwqfNhmZTMeROFeaGhztHim3ec7avQtGtLBEU8JnPyQjInaIwz46mOjxqg ddc7cm8uJfeB1EgzjjcygKWiNqvDIKk/qxoxXum/RDiPNhYs2kQ79IXKpPurvVSc0uqcG2aK wKjMEm2Dj0WTzDYbu1dw4NlrpaCyhOzrUQYuSzS4CPRYLv16XeGbstABzg3GSU3iiRirla5+ XKDj/jgO5Au+DXEpe/8m78xsz7zd38CzLGA45AMZpZSH/5Y5gtfo1bXca8qqUbHzRI3Gs+cg HtD6w1e5jt5zCCBxZcYKgk836dHBQWyGQpfwPjxuyfMDr8XE7Rk1ZSh34S2OYhWmW5b8XCIh 0ES/1qXPmKcAdlcSK4tm62h2E08/WSlEXtHmLbeGlED7zWqqGAFVJQbbfZmMgIkP4n65LiFh z7/p5ziLBSa8N2nUnD6e0lWW9vt5G7x3jWe/CTj6cG9ERx11oINoOu2rQ5HkMPPApKV0zPYg YttmP52AhG7XglcpTOzEJQqyWg2HdeWac8Ba1kmaYl+/7sSfLcb1ix3FDO+HAQ2jySGdmfS6 Ue5UOlre+HAYQZQh/vNAYKFkVsxTTr1mhN8Vzz7UW0FKXSLZablvomYIOHAwv+5KeICfDTyz jN//JkiBO41Z+g/8YrKMpmsR74wx5vZhj+7cbXOUfFIGDkHbRPjtU+9BTGQwGFhFlGCEUMlR L//1lVFUaNXSzk//41cIrNMLeVt8AdSzb5GY4VuDKY7HwRsBOofuYhwZEVfveEGgIoLmyWld mMtV4suIDhh9ENpmyIUG3Fdb6jgONiHqlGVUiYfFMNdR/8Wi2AMHTuKeSeb1u9sMXV4V8fTy gA6g5CJud+doobCa/wU+KVM/FBYCnIuTB85G9zLaNwKkwXS/ZwCQsP81olZVgyKcsKSIukqM t2ZLJmXZ0FTiHbbElnt9IQzOm8yBrMTpwZg7xPTzW9btq/p86PRXHoDtwjc1CtO7s7weMoHu LW5Neh/EHphjVHlP7RwUTuPf0fO60DjwriU13yXZdk5Z2NJb5BSoPGaCc6QmITcyRsXLNHIh xtW74w2E2QewLAXpuODlopVEjhJJSIdo97/1CoUYVwEMlDlXyg5JhuTGsBuaY+3im/8Z+/GS Siej0oik+4U30Ig42h1/LT16X7wIvxZCBs8O8zCfoQQ0Y/z0bbJrwy5UDwobLtcXcMz175sd TdjdbUwEdbsURXdb1kfk2pqvF5WaJEvXblSweNMnqw/cwV2ABWnSd+sfYrWsm960RSsl0YBV ue+cEf761CLvW48M1yzC5KvG04MbsRMgSaIGzyd7gZLvPugLJC8jnyUEble3QSMZrmjHPZHM deZ6NP4Y+lYdy4xywrm2KPomg/XCIiBh9BdCPIcd9BchmV+Cesw2IrruIi4sd0hLdi21gOeW cWgRsNW7yWoop8C1Fk26kjqL7TzSW8rKMjjxxpZGiCoCP/mAfx+gNpt1CixNwm05HVM3gbI+ RSI9XNvkkAu8eFtoAdPr7eV+EZHhqdIkwG61qI7FkAJ7eM52CN8FbPULQOFCWL1xOejoepRn z+IP3lqWW1OXVcws8slDTmDzHeoWq+LVbDU1Rqu/SiIrVyCub80+qtOGLG+VfQcb1bYnJ2WF 64bm7G0FUDNheB7ngdW2W03gB+jkBqb70xrz2JKgueMOTlg+Ox5DVM+r+N8+stLsC9UEI+Hp b+7Kqqy743XT+bvZZuzsJZiciSdwejR1dHfwU63VH+B7F1xX1AZj+lJKN2MAYnUE9d+ugH2D ldYmesi6pewsfPZ+OC3zJ078TQix1AQtlESBWgkTluC40m8KlLrUjBA45tEkCdkJsBQJevWD bulqGsxgrX6kRGBFcGmxewxC7cNQ6gLOCTOhjmYHEK/RGVJxvCuUTqIDu3pRxKdqv6JtP50d lsYCu7oOD6Niyde46rBtVcssETLUoZ8anpUJSWjGejEmFgG3v+3labjfdH+YS5/9msnI8jqU W37EU3gS3HlHAaYObkVbVkq/DK+P6gI2QuBKmEBA3e7UDQtTNWuRMsDPTr590M4l+zhx4sYQ LTQ6EAm+/4k4mPR56JzOKQT41wwoI7OlDBU9Uz0FQQfW3ws5SxD4HukjnJ9sxCyMHcg99EI1 jQcA3VvSan0WOG7eoaDYcRGBM5j8GN6jBYwWFtnc9NJMP75j5dDb3jK0MzuGy988Lyn0ptJN RBf692BamuDp/gU6iyLsjcbFx/Q9hEHnEJ39LtAqBtOuMCsw3PXsl/KYbymphNyOUlj7T1q6 hvoF9aVRIMoLg3sJYpGbyU7Cdxmx+lEEyso7sH051C7v58luYt74Sbl1VgYu+Ig+qo1GkLNk bMkwZG+zP2cf5l2FV0kuXutRaR+aIcXUgoxfUbkTVaAvwckmBmcEPo4XIXojcanH4P7mB9TD doczIJ1FQeSAAxZmSJonQ7qnjiAZldRciwHyEd4kPCncy6+B+M8fKn38q/DpVTmmj/Ldxy9h y1xqOr4f26EF2jmZckpJ1YdVGIyUAN9E64AKtqeHGKhTmV5KG5nuwQPZUp/naElESeJ/f1+6 375msRGcRG7GCamkyvDZnbYh3B6aMm0U1En2hEChXniVEO13bLjzlW1A9LyZTHtGau6o5diT pNG4ixIzd6ilspI6yAxCT3rmE9WmICEI6nxGUGDrxjVi2JeD8O3AlKw7/D6+DkCLGNc9Ow9o RTfMBHE8OPCHkHDI/VV8xCjmKV/ASdtjBUvLqPFerMM85dKqIh7x0kZxp6tWGGWliPiO1yJs Qg9emkPaBIdRGK147p8TQvcKESTGUs8N3lmxNKeJSmUn4hoTQDFnMsrrQCW1+eWXW4EauNlC xIgEIn/7F1Ufbe9JK6dXKjPYLmitr+fb3koDLZbfzjtgsEKpIEcgAa1bVf1C98cKnOO5topu EZQjAQrVHQrYzDydeN6wxsFlWY6NBl3DeBIHrN6dQoTf3jiDOZuqsnxwwkv9YBKAl093LpsU OkKuQ47lSltfLnZh1qkdxpwSFDcdZWOL1UDoE0GhyOeNvBQhgNEkWLn5QyHCS0I3X39pTt/s CpiBue0swvws8QRn6u3XuEE3VbcPkL0jolT3Fd9YfVZsdOUmLGSB3KXoi19dI477Z2gkU5K5 /ER3lWwD7z4sNPh7+NIyEXGaB32yO0VRDdvfDn2K2LKefui6FtNHOjp2u36AihUMgARlset0 TL/D01rcfKJqxjtRkcyXNXbdN62hBUGznaUeeQyFiaoKk1RMIQhQyXEXY8RgrRVFiX90sK1V JncsqgnkCFhtkSV0E4EarSuK6Ib7RF2VMpty8eJfGTYp2y2u1HU/qccj7mt2VKdTufwBBkih 1AMCC5P17xD0oYLcjW3q14lOoqqVvqqXzZy6FrmyWXYkZTOtIVmMtW48NXWvmboEPv75W30R KHEA8t/ht1Vt7WiqLJiAgYhbJ444AOVgJBSMQTFWVJWaniQV8PB7+gRE3pa29QIOKqNpwvQ5 zrF+3OeTPd9nX8/NUa5JmyrbfrBxA3uMNXBT59o0JifS12ftvpHyaOGKt45qBLKe9yAxKa3L x2Ada9tiesO9mk4SLsjjOQsNRvlpVvSnBuVVOgTN/cAktBl8zalBgeHx619psRNmCE/fVdI/ +Rnu7BWeHmbktIH2037faoRkShnkPkepZslrfaLqNoY2n50t8XG+bAiRspM6HqKP+6SLzM4L mwG4+4EPCACJmAoo5oUHn/eedo/md+Qf527YvuGi9317oX9hEl8q+Prg+I9Qw3EOdaKaIEo1 fzqzDiUROK6xE7xF/xuOTl0M8rfZFb/aNJcTc/mHVPdZtXLX8TGE4Z2/mXaenKyK2miom06c RlfBF3BAKGhRldsYQ9xPv5cOQeMZwTR8PhvjwDwNnaOO13Oh7LVQZuoh21H3Q/ApT+jqB6v6 LRlBohX/akhz31LpT4Wzlqxlt9Wcl62KApbKcgSC/LkY0+5IUNqNK9kizH1Bps2brtKEIaXz U4aKLuFt+WfDAgikixaNvezLKodAubvWIu97VjKg/gkRpLVNuKL4WXFEXmTWe8U3y4gBn7UG VjW/PZUyjaQSiJexKp2DUU7AN/Hcz5nawHX7WrJ3bmgd9wN1Qiej14f2P0cAqGqLbu5rpgzd vV4cE9ZcewnAuYg5ZMegd/92jaJJ8LM8fQ3c+A2vjiwdY4uFN8RBGbcVtLEPL1jNYJOPJBdC eFNf2IAUDS+GYYRtKmh4BMXfjJ9pUDqaYr4LXYOInY/1hm2FAnlHICWDwOs4gCN84PsBGSGc VLLpgW4sPEb8u1EkN/NduE6bNF9oDnPIhqkLC24+bPD+y0bQ/VvvZGQMbsFBhwf1KWcgqkjA ejhMJlXsbAjlMwkhA44V41AjbCJ06+j36svStsWWYdA7FcImLiDZmp3UGMAnWRjQ55nJzEW5 DovNjrV0IKBQYNFryGJgZvc6H0bfSOdg3Q7RmI1s+xgyK5gMUdvjKCnKaqH7JOm6KBRrCRcS w/02nAs38v172NpMJu45Dqicgqme2pIC6w7FIXRuW4Y49GY/1YmI7z/EcQ/qIL5YmnuScZ0l O0bfWN2JyK7lc62wajdB0wPlNEcS+F0vK5NmNz6bvy3lYi5AQXCXqrjX76DppADcOtxBbkWM S5uRJmee9mF8fBzaw5P3k7g3MHKwQy7SLS4yW9+fK64clkW6rbnlfTjdaJRhojIfh+uhSsJZ gd9qeRTyFsnbLmyAO+3Emyvk27NpM45kmCbZwW0mcrqSoduGBJvQ2+yBIHhSxdlQvqA/l81k 7cOiRK3LBcIAqCikpogQeMhX7+OypJ8kRHFAkBAUX+3S1C9tyXvagDjJIzWvt0rnrqxkIqsa wjqDeM6DGkawBw0MBLYgPstc8jr9rgpuTPXowmLDW80r/E+Z9qQQ0eNsNgB1SOzdIDNzkQxY axxsbeO+pGUsbfEWNxssInfyjjTmMkb2F4DOE5tzOLvC4GWdJby1ZCJZO8bXn39/EkCNb6tJ oAnC775BP3+t9jHJ8TNEZztjpHjtli6jLn3eYoNVoic+65+rx4JY8OsOsZeg2dWhT15Pi9PT q/FCMfmUk7ahgM+rMiDVG0b6DZRweEcgMQ/41LveXkr0+LytTAMtplz3x/ZWIwso9wmzlOjc /TAUi751pCFoxg7NlBQfb1XNT/1BBcjtSsPC84BglUIPuXNgF3Oe2fH4NW9e1xbvaVXYyXsT Nd0SsyCQp5RuTrQTLPYPpZPqu+CyZxNzlTA9uu3MIv3HwZORZChNxgou0HszGqlUNT/v8kFL YEuR4XugwGANx+2bES7bzbU6FQxvNSzH13eJ1koTTJfNYXUH54V7YX8nfzrVpU4kq2tzcFPk yXoaowqKcbDYoFs0Ip5CSFK+4H2q4e2ohKtF9BUl+b32WD4l3xobZ7IImRFrbYwvP4H6cFbz yKhOsJ/Ult8J72Vopk55FOQaGMUUjlD+oiT4J2lV0ynPR2Q3BQqe4ZndJNarrhbTzW1+Lb6C sEfFjzOIgP9Xa2w8F5M+9jHJpFGSWUq5pkwuuqsM7b3W8vfvHFOZKqwbq0RmZNkxDrhqoqDY zrQLSNFnx4mjMGKvjyI7phslxxt9OXsM4QYKjS+CsJiM5EHuOfYrW17prsV7cJCc/pU4SCtK 9THlwNjwA/1EM0beGA3RgJ8fFzbJwC9id52qec3LoA6zjjSoaxGQCgdgYhE3bMg9y9Nj9Z2M jNgJx2CCKPLKrdbBfczkA7Qd+sW76xhuEfHQwOEQFGlOHTFjlkq4VCANcWuqnNff6mzkE/W9 lnnVi9cUcCYgn2NGjyVAbPo4Di5VpDR10ew59weNjmGfh54Emcj7+L7MQJu9vVOz0AxUyizs ulZDwZGYMUGAIv+l2SkyYTxRctVdKfx7ru55IX2hVrifb/Bsv/EIm1eq4XpXXSwLIAmqqPqt EHjgXNWZFLM01ZsPUfM5ZujWdlJYSihje9EHAFyd7q4ZfH5RbXrZfeSsF01kCy/6Fm1tnkx3 8Qwl6P73Or5dfr+oyOmB3zmzSPgn9Kj+J9oz+cXi/ORmbb1mkavuQl+bKrJnlZ94pz4/2Fim plZmfaAdit1oRDaPT6cG+4d5VqYiyImSxradSo1Lg70RJ4eqQk9GKA/jGdr76AIxXkAjyO23 n02jpFEiqY6J+Y1pyxX+2ne2H92HTbwHJ1LuUDhtdYD4fxvtpG2Fv6vtYWQhzBk4gmitkfdS vG37r6st2hhfF9bT+cRurZrf+qGMtBQyerDltjw9nTMUUUOMzH1RfCC3ejpuonsdLi6Xg8ye aHU4Czfrkfwh/J46BTP38e+xdY9IQeSL9K1d3CsmyZW+RAPI7ZOwC4vp/3o16qqOAWjc6Fhg W17uGWQStMy3es/AZzbq6maqjGkEKAYI8iSdyeBx7Y+kW2PQ3pEVDoGciK5PYRJb1tT/Gm5C w6cY91rINde1tF3klBk0s+trn50ttzeFIf6gA2/yT15IXahQNNgRAKlZmhpQfFW9el+8AuhR KXidVT1orWOascZtfgnsqGWGUG3w4FCzimnFa4+8Fy8G1VvhjPTVJZp0WDa0W6rPgy7JOR80 QSch6u0Snph3YOwMjg4bjwIgRoQ3TWKiQWaHIo830oprMUSF7VW9JMa+FmYJcXtRd6zDjNnR GXuS2BGW4w4Vc6ReTZAUuini3cBq607f7lNNIhvigcquAZcBbBq+oiHebmaA+VIXKnzCCXR6 WVJjovUWJuYTgCVdShAkZupnTN76NVcvHXcVqyRp/h+Sb55K2D65z9QpouukeaMhDef1c7+Z JfMdphPzy5NEagYwX3NElDgQnBG2IsLukrVWKs7xFX701xijZScA3pecEDQ2pHGo9sqZStBM 2bNSMmMYlfiov27RSS7yOVAQW3D7Nwlbnv9DKu6EeCCENXkuQorKDXLlbcOf8mlDhGsSW/vE qVxpD+ejb7Vs1nyHaFtnJqypSZB4PkvPS5RRO6xJnNXGaf041Qz+13inXV064repPRmleost 00TLZ7HiNHSaFQvga0DIlWnlGCsjk1gZRtbPxFAVD19ZRoU9Z6lXiCmILgBECi6yD2UQo/Fi GbX62XnY1Yd/+BBxbQNdMFlmiIxUzwPI0c9Wj9u+xrLNC3GElhhgHBz6623Kb7H0pv1HQFH+ f+z9o1Wz07m+7odouw5erF97xCARREojoOVM/YJRw14l0/zhtGQGfLoYvrUylOv06L3n/41j XYJ+Gc3lYfs9Y3wlCHQabePSqaQYyDpE/hNZowWP32uaZmv0JyySBZVqbV0sufhSHo1eIzg0 ZU+qUCfY5hQDkbPbVw3F9kLqr+VFQeFlmeyw8NGqPYb/wCF4wnaOb8X+8tIdHXSwt54TStLX hEYV29Lsg1gtD1a5eMjsbL3TNF0JI/bVKzwSbtD1yOJDGnD2acneIV9bN8n10EXD72zR9EQ1 nIf5Yi3vX9OvX99ZNyVHNL+mZoQeRtAOpGavBRtxMXK2nmEvHHVoSzgiAV2ZEWaA5bHBbmj7 tVB78Gz8vx5mFJ3XqvpadbR8Ttiyr6yPWi6V0iOjm5mRAZC26Z8bglrj6Bd4v3vp5WmA918E rZ8UnKgy+HMWcU4rG5TKIi++6FJblZ/pJQgf8jzITE8VVcEDu/QO0ODLNGNwtkkmPLmxdL4g y4jVRtFvs17dEYdXOTbt67Rr8YLI9NGif7apeFrBRi8qFVupSj1wm5/9SOJvBHopx9k9UGp8 94omgCt24CBJgGKlCOWDeMGKuROBok6YHDnVqhbriJcgM9pwGQ3sCZnoAuMuqLH8eA6JTjPm llFevP4dN0dKgcERb17/mH37t/ztwksE8Afl1peub+g7PjUQWxXT3WgkgVahoo/Wknuq3pfV A1+taQUjpuuigqvNwIY6gnxCNxxqv27VLWEK4nylxnZsAyZHrFJ7/lAj2Me4FaadV4UZj5KZ HpkqSXz+DczseHGnQ4a+ElBAPIHD/mVCeolyzGIN7Ph8mzh1eQy9Kpnyg0M+h1hqBHy94luC KqDto8W/cqTcFfccQMQ+2isdRqCTqFshyL19zVc6cg/maPKc6nrWGx/bjmxm5nEgtvGxaQmn l87m54B6YJrNBx81VUxSTavHToCfvzUXpUY4LxTRuWJHifJ+uv9CkOeBSB1wHM5mm7SWHZ8I 3qYeCzKmcJulAD0wpv/i09+0SeaWtfQl2yV94E3c3fqg26c7JWWxhh8O0kwJySIvrhD4v+nN 9rUZjE/T/BfEDYW2AUC8bPD1/FIEHIq7drT1kLkfzyvOKbfMs5MS7QjYDWEFy0ASBAJtbcWR iIBCGfuTHgp9Y9vG6xYScf7s8ovjvR1D26nWvBZPcgPk9Pz/gInIddgR5sEVg4uiGcYFF0CK /JfLzW2bW9QEAZxSNvLSag1xzg4+gfFa6bBsuIbPq+OyHHsjWOAJoYW9UWOA81dk630iEZNC Vtn30vF9hu3rbfS3mu6/Z8pf9lGFQ4+UWvZC/GTzA77ewJFv45+cQx+cqcz1UVvEwcxNiDcK OKtKpuK2V7F1CbcojJMS7o7A0mzAJ2iY2cv0SS1xigzjjjvBlIYOdcCbclL/3QXtgseDuIxm Ed/Pe/vcfIlL4mlxlPQUp8mc/b5ghWD7fHce3CHeqzYOzJZ/Pw8+45Rh9JGX9XxObt1hqYum hd0JpHFEO5DZwliQUnMHlaoZKu3LLSovPKCeS7JSYFdaimCZ9dldboj4E+BPJRzW+YrYCvZR R3PUl37TNOVdp519Ha46ySLLnnU+JW40/En2QgD8YmtZ2+8RhLaMi5YGK6O66bTyWKxRqCHR p/ez1Kd4PBPjN74uvW0N6zLnoG1uQ9rKY+Dr1cLVuxcBRB0Nrzr3guSvhu3Q01fROLPOlIXM OzkIo0jmNwZS/vyntbDSTaQ05XZWyJc0mJFjMt3IO7Mfx2CE+OG43QUqKuGPQPzT4yOjOmSr zV2XVHDpgYBMGkthDUKVCcV7C9Sa6la+MEWukbedSD7AmoFRPA3dD/iJcGloriytE7g8OEbt nOLR+5au9D94NsU3ipixqHzIreo8p9jtKxayWfaNAszz/9b2nanEMvXmKsS1SsrVQvojKNi2 9Ndc4RhKP8pLS749Ih1dxs+YeLeVRTRjz5hMtBnNxs0D2tW/7J4+aso2yHp9BMBmnakCGLFs z2hMaeyMqRkAo8i2NxeOhzfCtY1fVoD9oyzFRSZo7Gq6zuQuA3y0unbNcTqqTs5ZFij0+x+n OxMHyr/XkiagqimEm9r5pU3ZdQMoV0Rd07agYPzkxry3ixgszr8ToAhOMjP6ucwmwTn7s1nw GAdK+Vb9HE/6HrlUqDcC+/H7olvrWK4Y5HPObCG4BUct3JvagI1vgb6WJH5IYZ4Bs0MweQ7o p85kG0MwhHdZ20BWfX+9O+T5Gn7n+cvHZNR2wThBxQmxkdPnHhMd1NG20IkRPrcgo+mnHlUv S8wkTYsuebh3OnKVi2dyRgTMnvrETxrNzqWvVNea4yA0fYPfrJLXPZ84LWTci40NrHtaxfUB ItF1Cb2dco2D5Uu/VCPw+F1AfGchXHdOBGf0Qh5YUZYQ1kfUzMlEzUJLmFx8+B+s/rCuultV F84Eq1DZGDA6t0NDy0CHJl9L+SXFJENCZAzxxAUCm2V8iP/RD9z5SxdQhviZToSTgMRizOoe Jj/uMke9FE4VSih/lP1rgAFr9LjW7pf3uWGmq1g8CmhQb2SAGivbFiYkUrNE954dKOW8+Te0 jXzfp2/6ANVfPk+lN1ol97hinqvOB4J2Ua75MUWSFL9P+hSSHMaDY1oYms+0PYC2sempnr0x Ns8A2jQsngzAHHW17/wHyUqKc/cZWuilWRL4z8WzDlz14BGDFL2q/BDNoLtRYEwtaEDkBa9+ zNQgrranCyfKCBf2HaamvnHyAfissrFd/5racy3Pxrn0dL1+LozlWKjQU2Fw/24JTGZuyJFd ouTBHXIbWgP1qilMZvaGuO+BoXPWv236tm6yYdoD+bjk0gAJCk+iJ2cwO8eJUyupZeALULrj bLrOlXoJwxssSaCjMTgL/GfzlD1L28lNK+Le5ZJF/KMNpTUex4a8iPYJ8sF9BdhnRz/y2buh soARn8WVFZ4FSOE0Gj32dROBdqXVgtbvLHsk2vArjsmqUqlcKiQs1/dt96cGe7hQN0CHFPUq pei2eLxZfrrJ4Ndcy15hifgwKBSAJMiXm4IHkbJ7/Rxlpb5ee1tEmtgum8kNRuLqc46gjhE0 OZasP+oVGCKdl+rkN6B5PSDIaq9lEWbZ9N4c8SRwY/Z8hp8e6JEGeO1sOwKRqT9WB4liw1ce DDI9csY0QVPPhB/IyoXhX+bhjV1yGg2f54mLXlzvz3k6yu+9zkPIolxWdb7rG5kWGzNuatmi nP8mURji5LYaCCFFZS4ZsAT1ZHEOk0lGtAiuH2JdZ0QaB8LTdCF68KR3IJU4DD28lxj9+A5N FhQ4nhSq15DG4bXw0gNFQBl5pDooBCab5hUD5kAFauRk69yNu+pMAVXwdyjtc/P+qB3wJ4DC tVCMd0eOqm5Ql82qe2cMdsjstlaZHmWgEIgz1raiqjFRa8kGZ/cJqisLTY3fRC4Wv9jJHow/ at/rl6BSvkzh9uUm77xCdPyC8+j6mYkrLGzvQTJCk2uMj4uLYtpMcJ4UwTHJ6wIDZgYbvgNp BVJFG1pGvyB8AUhBnT+RBlZ8mPlQ2Rv92QnZmNIOO50e0A2JYZ8+RTujAWp8X7DS6BiR9g+4 kfSVS1qjfv/y77jmqoHJNK+rhXjJnrTO9WRFu40syCm+iGEPeXW2VOfjtebyRnlAf1hAQpiK bzF8qeigATg6GW1wQupxZLfvLO5qmApwmMNFDT1TxaJMe7XbHXgO2wprEnD7XeuTeO0Y6STT uw8EfjHFR4Nk0FuXj+DupoqX7c9KrWE7sPXk//9TWIHBpKEQBONWe1DDR3is0KVPt25kFcl3 gDUIt1xY13Qd70BeceARVPxrtFOU35eoBOTvr/wcsBAlxPyNuctHMA6Oq/9SWl8+JUOV6BX0 fZwHUycRKuESZzru5qCWVmG7jc4pKApcdnq94vqG4UcqU3F6CnV7w1ZvomCx/QlNQiFvLVlG i2dGAma1AV21xUFlYzQ60IULp/Bxe4OORvDd38BnSMxFBvMjV6GpLbVKzULc5cPNSx7TIhCI ZVbAhKrSYqu79fRg2FdqDF5t8cCbXc+Fr83njUNN9ERh3R7jn2sZU7GvzOsYmXi1A2XFoS2N wbOG7ZfqNfbGaao8yW0LdEABD0YYtZzp5dc1Cab2pTHZoKHXeZR96oa6tPaC2ZfIL8HZ6ZZI Uml6opVi89nAw1BRjlNuTym0mf1blb++G7qp0lVaod9l3OGU3wA8eHGz6f1Tq6oCUdVsEk59 Mc2gjlFLTNvX8YwuypyCpH/wUyBJXOtCcFD1d53mbXUM7s9x3XNeCa/MOJ3VtUe0L0y/ihli A8MVqIgU9FAqby0E+T40jHgPF3nYHn+eVpDOMjH/5d4kbsKfGtBkx9fFbUZ4HW6JvdfKW+Zh jffWp2gwabmfFf48Lb2lW4PXwkPT5SDS7/G2dFwd8yUvTv+HMokLWfNJ+CwS/xiNWjKZ3ESK qZINDFPA5PrfFhOsCMjuNAw02sFGlW9mIl8I40mB9+L0pamp0iTPZGFUhZ4kPPid1+Ji0gzd RJYToFjiOgmZ+c7nbrKEQ+8BGHbbIyr70gG5yvV6n7WPQutIhZSdjKnE8Y56ijn7FtmB/LmO F57i6wnleRO0N4IdLxC0s/1BBeoPBexcmoc5SS+HRiDIzOqF+n86GRzOsWp74Y/HEUAv/bNI 3CoZcMTk1YkYGs3+3MtaLGQxFLEXmT4H5Iru7eTp4du9voLPWMhvtvaZr+9Wa0KK5fVxmC0m Kl+A9/R71XEWL3b2VVQNdpEh+QDLW8I6ILKr8rr1LK1l3zU9FWUPhJz7Shp3jzQvIVc8ba5b fCiCtCb7cHhIwqyOs7MYOJ6yoU4v6XlCtguR+sKFvySwNnYGqU4C11piy3JJvWaVLREZs6Oy oASQQsyYo3u9Cw2yf3pe9o6359gsMHY832qXgL8N1qT9pZzODnUymlE4945BKwyMY2UoOUSw zTPSX7+CYuw8yE7+efFdxx9GRxIZQP0FPtLqKtMjlwPS2AO8x1ZIofTP1PNI+tPQFIpoTnkO biNG+U7ylM6s4h8VvgXpNfgbSodQuJLxV/wzJs0bpesl2QnWiMj1l2PrZjych7WzKP7fOffu YscdvKKbccNjyYdG8dBUrrcrVx6YLRrMHMFEyfolk7IebHL5ktuqRdD6mreQ5qySGlfN0aMZ FwAAfOuof0aSUlvq4xvzyd85z4psrt19U2JptkxM+40sNA7oMLkrrhBxwgNhEnHXBCEeEsKF G5QT4oF9S7OFJMXwNAKq7RLAdi5jzKK0jc0FMeg0/lzefXuEPUKYMg4scGuFw4o/TB66b5ew /ZE7CZ1SfJdyd3Np3bvXEYu82Ld48zQ6uU6w9P1ENgD7HPpSI9LaGCvnK553j8HhLpTpmfj7 gZ7OsCXeZyZFaG1BbOH5m1/OdNcIf6bgJAKqqTePlj6biFjiGMvnRnzvqWta/I1T/2Qm0Hfr DY5nqKshtcajnPPAEudK5Mr2Ez/GiYbx4Em3MzdMF3i+sbateLFncmc3u/Nh3UjXTAC9YR9b NDUSNiUFbQmPWTZZIvVX/j2DlEevwcJGVea0hEDu1zSp5GjCbI7FLAvqjBCQUHmX5UgugnCq vSgwnIc+pIWHz8U157TZ5EHX58QkXs5qXYwbQB/3EjDkK3HHSHFkOCICY7Lefv1+sed4+fZU 3W9k3j8NWOuWBYrGrhrT6lO9F17kXvurL/U6VEr5ZAxuICZpGWu6XVpwfexiwZpGkGmHWc3J WJ4q8n5JtkQqK3O1C7DXVJ80X5DfqXywnG4aghPMxy3BHMXx9L9CEJqd0UyajEs+HoLYxb8j fusPnopLJrANJzZpoBqsEohMUZAVbSoVe9gS6S9Li7w8lSMIdIUHX2+XdzbLVqvUqC8kFrSI X20ZmxntxkCoTKAqGGVpCo1YrvR0gfsfNquCVm5EaqL5weOyM3rhlBG8QhMt/RJoyceU7OaH VB8BOfpwByOOvB3tdek9dtspOq5BFG7K7bGMxuHt6MTO4ZP/bsby6fKnLPx6mt9HAy4rhsoU MII9JkmQqUzywBhl8TVuk6DPAPoOpAwxMUm8yleaq8UVs3OYA2sSFzephT+/K8SKfNvSlbRX CoRrsjRqRbweCt+M/Wc7seB1muFb8IRy6IrHBd9tsy6evZZUkfdDkr4e6oCU4xRuu8SqeTB7 srzwLv9FwfNVB96hk2dKycFpqs/dFt2F/PpYI7oTl4h8m0RRUPQdkmoKBh4HuKIdMGmQXsUc G7/68JumfqcWLgE2t5dDyZ/JLmR2PIKF+j24kZmwY7hVZIvnL+jF6Wb18xx4gVhSG9jpgrJR vb8TWLPYtszZwqI0YQnSv7CHxIiVXag6KgNS4//3bKo33/fYHpvr6/V20lj1ONB8zuhLdKk6 LMsW8y4aO5hnI7PBAMUpzsbSaXEbbpoAcV8G7FqaS5o5amJH08FYHjBIaekU5Id+STYNa2Pe uRA4Tc7Fjg7aqSA4UCT7CSftlZWha4K9sB3NfVnMzRWEZrLuyrT+YLJB3f7HvEV4OsgfNMIa D542soB1ly7zTjzzQW/DQzqHA6nDreTXW2Ce7Gc+iMlHv3w7Em2XaoTLV6Kf/McjEDvKvg9X qA7zoQKVmW+FXXAczxbR1i90+7BzGMDr+hCd8tH8PGDDSLGbRmoaMoa77hF5xejIxJbO84XF YxSQMYMGKbeu/livgTSNLokDT1bzOwh+JPRo3PSh885DJnZ4k3fOL/3ppK7QONLDcQHVgntZ n2pqskc6rfFgmhoh6uwSwP9jg+XYniY6TIsZxLGsFLvEJuO3YHGBi0h9/G1VmvDCwvt3tVVJ ntXXCqOmVE+hyWYHeJG2FpjUZNXxaDrtjQVO6hPEeQk9d42RX6hJj7jQRjnFdPdm+uzhc0vS SmZszNOdxDDYxMbi6+xxmvptkTAGntQn9IqFo9Vx+D8Rt24JueV6hdfLRkPnolyQH4Qojp37 r0LWbLCV+JfLnPAxr9D3YcmpHFwF3PpClfwnDmqU/54LoKn8rN4e5DhOziUE/GezwlMWoUR5 lFs5bnJIugNNrzMGDmDFsiTwbLhG9qER6Kv2RugC/FZy7PP4C756JW0icgk2QR0dMm58cySI J8YfLZVbu4jeQC5m003QIdQAQvdfvc7BcPFSVKTzOgsuYE+o2CxkPoe1cbQvQP87ipEAXYEP 5R01O1LEiMsYbHyKCgo8AifQAQGqrz23wvjBCsgeswYK7SsrbrzjguRWJ72KpQQ8EAaiRBrc EFj3TcsFzHp005wlEFEwm4wz7F+NTpxr7Pg/dZUrKtneE/r8wwgT67sdiIZZOdNgOnGY3J8B RjNKDUWR8xIKNPto2GXttJBQL2/wyLHn7ExjBTh3VEqwE+zMakjccbQzHA7coCUoXhE6dW6o z+7JWbJ15QgXV05ipkbvTvEAXaLe5/Oez1h/+htNPLgDz7zULO39CPvKdW2NgpYTWbIFQkeT RFgpNccxu1JeB0fKiYJaal/WA1t37V8J/MzqGgd0zg203yT5NDGrLTC/nf3SbiobYE/VoHsu 0zUj+qiX3LEkLQBOqFgKa1IqT/SHS7buije5t4x0irnUqHHI7k+GTgQyq7Vx57QNn2EgaCYd 0LbqxU1GUnwBo12nHoqBDzC+QxREKKe9YxnCRbE85TnmMwoBwU3c27dy9t8vgU54lQ7UfTw+ ElM81gDJ1XVL4Qcp5qWii4xDvgQEHzFK3TnrVfH4FXwabMofrPO1KbdQm7DUKmN2mhGyCtC/ rqbI/tHt+BXFDU0wGJwsXROITBrOUSJTT0Lg31xKWF5zB90BjxUnFNEhBL1lCHlvwGRbhqqF XGq/2KeVwV1WVOfzOrc2KBymkJ0nqguCbN+6S40HU+piGzZvp/Gwh324+Hv+dVKsIZO/oOSF blaaN05eh7lB4D85Yw8UKa5t+JpKfW0nHlwOJlOk3CHBiuKLqMwhw8f6/K9ay6KDsfmbC3LA UoQegXDINA2+25QA98toYPGb8o36Ad3nDKdQgSqaYXpT+VubUSlC/SrcJfo3t8+eEiLqFTSO exK5+j/j87m2jJ7UI0J+JgjRUMmpMzNfGRhciluxvuDgWx3aawN3+OJlk6DgzOjAq8yzeeFd 9AdHy43PWUb6XSInSrkXj6S45eC4GvsLedqmqRPjji8DqBYBvek/fUGJr7SkelA+wZwgVP9J /uDJvqCgQZBgZoabRLrTQzQlC38pgWpsvn7Sw/OH5sKMPVwywHS69lCw9csGc96vZIQfZIgu /GeYCzliSWDx/HxkmteVgVf7bb3wh8fvlRZ8SoBx671jJNUU3ivxeTjPslEY5wSJ2YkaA5rp v9wNTVCxxgLSDASTCRTsdBy+Cs5ID31AmQzvzD+3VgaflZ2YrsIZ4AqT4fJDj3RyVPRPBmUB hHz9DMwxZhDpqcVJMIv2UBpD1xr624xDr6pXdId3PTHG39bjSO92ojuu44hGPevrkLVvDmGm CiKGx9Sj34ryHyPUfTAPjJ9hkk8RmWxXJMAenXakVHLJDboG3JJF6vLKKKqx2NIKGNKfTB6s h0+iZIFfyW2ul2YTAZjSFF5AL0FNH/EabsXMlfxPYSlQ2u4LG1do/ejQimDXoWfybNcDtiKJ Lrwi+77w/ahI8dwUpaQFpbF7c4I29iFtH6GOk7PGe8xRhxXIZw7nscf98goRcWBfES5OkUMe JkAmxXkycLVNQDaxBOJ2ziOVePwWKMTYiQflkgj8M6kiOumV5BOK/xe0L7Pwo2Jqn5m1FAEa QB1Drk3XOkd2kSFf42QSlYyWAl7C1rJWsZdhX1dWIS/DjzeBFq3bzOCClwgzwiXRN+VYIMc2 dcY80QBV4V54eDYZS4A0u8cNnqPL1tLbvREv0nVbTKBe0jwddhL8yXKSauUtA8qA+HX+oFQw rHpyPTvbIICxkYyukP6r7AFxIXHDMjOo5oe0rT+ZCGTd53XMjzkfhMbUX1uUUEAxw3R3PBYp 9tDKzKWA2GCQ+fA0FjHvtQaaIKwYsC/9tX2RECRtg7q4APouvjzUPOXrFCWiu4GmQrMZXcWE BvCt9h7f3WXS//1ELaIhxNUv5Q2AgIT3fid18DimIx3vjMMrD7+wSGF+4wgxS3Su+PYFoABT NWso2i2RGMOUm0P6wVFelkiul3AUWYEsAcZT9YlfsoPZpQIdRrzj9BrkDLCC6NO9V0ED8vqM XAtGq7/YfJIA9Ve3o3FWE9+NYYLd64vDWR5KqLD3xp3pI+wWvNQ9P104aCR+VhICfZY3LCyP veBU3VmzJKQHd+f/TiuCS84RYcr+EVR9pFuwEhJRnhAGK8J2KX0csKdPELMK5y1SgmAWqFQt h9qn1LJo45ZsNGCITbOuJlE/I2y/WaKX1yCfFZg5NDZxpUOvtViO5npKNgYLZ+/Tw5pvkhyo 5oR0rG6doFMZOuMu/4U2nWJ2NOYoFM5DpsvWTYFyGtSU73I7t11TjMgL6MqwqbTe5Ri++bGu ieoWNBDDemwoOYc8Zvm5S1dWzpbCGIDZAELTfGBI287DBbPICp7A7uAehoTBaUkEw+kGPZFI aWZ3qVZIjFRnVILxmpiHqWNklh0PDHi7rt0ZhPGhb3u1re0UKJh6fvg2W8e5bd9PQ2S2QyHA YTKOQ4TWQUF0TyEiD4jmKTdnhiWjLeLrIhECXrFlN/mAIbHBXZ5ysQIFro2kQKo4F2bsVTHU mz8h9tlCrZYrV8cFqdf6Zd1+Xer9RhVvAZO8XhXGoFFe7vH8evygmmT23jHrqSGpk/BaxYyT c3S/QN8konU6SYKoRWNk/bZuvTEEWcc4AFUUHDKm/g6sRhoJJRYtGKcJcNFiDKS86Y8fWYca dZCAHXKv2A+RSx3vmGo4BZ4bv3dBIHK3kbtJ0QY97RWI2cvvDMuneSosON3MPrCCiga4r9JE IRBpPfzEralt58HDoiBr544igCZ129wbHVNDPDubG9ZUN0qG4dCsB5ZLpGU7QYNItY3cgoZK NZKmS8MXWbv00UgZKFrl0yR2l2JwlBRr862yXTrjAPw5QflIwe4eiyVZfKOnvmRhN5RIS5C+ KpnY3n8nCjdUXcpmuWjKIdJfBPD0C/FPR35DbetpAvXzopu1rgrF4IvdSdKJXaf8ZfsTkHVw GaFgIq97T4Lf3PCIF0owQnga6Nyiyzoq9IJGNupFBGl5Zlo4Sudqnlo2Ny92vgSgtwggf6Nm L5bH5G98rxZAIuuvSTrhX3jAydG22wxXkshrbRL3EhryfBInqHinNYd1CqNxmRkU6eWyXcdw tz5TQaZKn2g7eIYuXrhqmDhwP7SFOhSWKd6dTPFg4vpGl6wDHfd1Hw1Dewn5Lktv71v/nEr0 9jYMMHq8MQpct2gZIIyXFNg1ncDjb18pEfWmRpoOkDadDJzc4PM1ViAPFGPatcIqX28k+w1L yIbxoiXMkPReEbBD7YbMTpCiM/6Tg01SeZ4EkFjounck/fgVjkn2XD9WO5Yp+8adwn2fV91T DyquRKXzH7zzW/BThACB8QYfsoV/LX6sjfGYlbei66/xL2ZqHIcJ8mkW0Itf6W9GDous5NVe f//wHqw/jDODL9IwexSFUgcyDg3JzkrNa/Hp4drW/cch5YpVdTFmHrYl1KLNwizS3gHGnP1A lsAEeYJCiYWrAgm/4NHi2wYhFQkGEeNOAQp0yHm0ciAjSZVyVquGGgg+oKbfevpWfUYuhMbF XcL66RMiELNwRPH8QMA+RNatpxn12biOfDwgjWTbjpu6cht2bDCMh8Q9k1x7jimctcjBiRqd URrw3XTbRgNFLTrgrSIYLP+c/bcHNwzOP+gS5yMOS0BRJ013DXyTnw3dNeZYlT43h2xaEE9/ A/xksIgyCv9/U4UnAky5lFB5yt/Ruj6jJ4V31XEnyqXZrhbk2ICZniWRxIw4BvOQ4xh3VkS5 k4kNRuvcmgcLsC9sGrcNhHzv5+E9UV8Un547p8GM0wALn9ebwx0o4bC+r4kw3MGwQPRSwXHB wWz+JY2VnEjNicOrFIndCMDpWd+BNPTAWG4Fl6OsEqahYIhcxJVPlGlWJ6G70v42PZ2XesMY n/6473PhNDpBZ/Dx5SasVsTlUOoUhrUfvT3jyBUaEbSpEabJR6A6hk19QzdHI2T4VKN8EqtV 7FCZIY0PQyDwEKJWdyEt5kmarLKfOnYIruleWP1D3ENfrI0H59vbaloGT290NK1jZec57l4n FAI8jgxhR4q4U/FfxJtoHeUNwMpFp78fRMAgygCPWUhtfN35dh0RSaIcv4WmtAF6KFPUqvxG Z64LxQTDtMh2gdOyXovVzfSi9CVCL9xX+rmPgyTfJ7p6S8iqIxRBfr71p4tivqlewNTABdur bMcu/ajkieVmEHPrAv5xnCqHD23JF8enueTGS+zFdFF1XYIfPE0wGVPCnMq0MqQ9bahNb3pm bvlBB+qZZ0EDDuj9U0pu46pTcREfMZsEo54h1/3ljfbLpdQodgOL8jFdJGqPZI8Xg/tYWT1i JpFIzAjoDLMGlAY+N4jbgsq8vU3MjRoaJdRhDk43wRGjDNevmuRAfiNjCwH5FJ4JDWbxVM1Y 5mMmKYiNOTllBGzvmh9gYIDQ102qB71qPt6lniwCR3rue+FFi4v3ch8Jo8y4PP866Ud8G4Po W8A1fBRJY0uWeV2jr7fhnq4xWuQ+5I8gFIesMDuHGAbJT9TdT9OdaEQEkxPGVOZRFDVWKbqE viHCWSDkGSMmhzIuGcfwiGTara8zWnXHlDxaf7ALEHhTp7Xnvpyo2ok4/yVcvfcm6xAP3fqe 5zDzB66Dspd02sZNrcl38QeaJF7Mmd5oWz2B0Aufl1ubfegDgQDoCed5x87ZCCqJt9OzwfIo X7Y3b4hKX1r818CAXUyGF5mrSjxqZhmwWo8bTFutbLv2i9GWNVGLvYmGE8NdIT0//YtOkOi7 1I2Qq2I7a/r1JbbIAflnLf8r/rSAelhF0TMBSY8qJX35xuGOX/BMWHKo16Nsnioegsx+PaZ/ 2ps4QvhiEm6EdQzQXD9HTAAFeqEiKsTHEBHduJjzxFhVR5k1l+DRLlx0Mbb/uSqaTyAuCOBh rv42W+Ia5M9htNmsv025C8eNlCIPN8AdKTeQGQLoEyr5uYCqR9doIoUpbtM8wnnOL7GD3vCh icd4C8Rw1DnFqI3oKF1VN8mMGQbPLZOW813N2AXowaBGQ4cf6lYJ/dFQrdSWjVjFTeLLnIvR s7Kh/22vzCyCgFmiQwlTJGNAXWjnAKrEm4727TZPnTVb4KYHEz7PHxbzXaqgJHGyyanPg/FB GQ0dxlFWw+n1ZimjORDE7nbxNRB593z95ujIKNVMZVJWKgWXFDgasndhJ0NSsuUqrTg4IV8+ EXh4krrjO/O2LGq4Dhwu6FJv5ezWRfGlXwjs8NnYQ2Mb+wpCB2pmo6X8Gy0hW7AbXbRvHISQ m/eRakEoz6Dn6pcbdAcSegXKtdzXrMjTxi04wGxOozubS5yv8bPuKM34PQB28ArvccpNCXJq 5Y4EPtk0SKohSUZIAo+7EaUjI89OtI8iDUntHe4KbRCEgp6pB+yBziAeO26EEbYVjng1BHPT yi3P0n0LmU6hYOYtGuoRpup6uCZ0svWzYJKbaUQ3eEj3NJ50Sn0ZO9l62NtgI4f+3b77arEf HNmZ+c4GdwuPv75YZC4ybvl/oWG7fhWwo3kfsEmJvH7UTuMQPUOMskTMmDXKagz9VWfhKqaz NgH8D72IBwU/sFDS08buU0H5pOoHdi8aTt0EgMMD7I6kMN+zB9tZCpurx6Yq6LMPfZDDW/Sv 9FMhH7cj6yxRD0FpFBwuat50C9yYKp2vvGUJuwQWSAO0z27tgYJ7yZqsXh1B59ujHGrKIRM5 lcrRAgluWjYc0KWMG+LmeqMvk4H6rSlQ/CeQxiBkgeg8lOQmBAXgWCZoCQ835P78zHL4gZmD ZbBa6uT3pwjON7P1ZIarGy2l8VdqBe3CLOqzxeWe3W1/77V7Hk5ovZCnZjhoxhZkedf8TMKa fGNAHUporBBKfANGqpbGm6WTfzYNYHAJMPg7avF0JvmL0t/F7r07h7OQhxkRKaPtjYzNzFcc +OSrR3eR2NjGIeQsteAoOnce6otz/U0TcUSD0dyGnv6fL70QChWPb9c6zwBBV3AgNF6tGpFk /ZJdR+yjbLq+RIi+dLiEOswIq4GABeyRG/jJvxUbFzKj/DXktSM4JhMiYoWN8ZvS2mxaQRJg QiRO4UGpD6heegeLei44VB2AJuKPbt6mHafH7yGnk+44o2eSwwG/eASakqj6zTHjExrFbJyV k4mJUxLaUV7QJytDsB407ubKOWW/YAgmbqiBlELcLp89hVDxIjbQ30C9eaDb53IazUsiJ3J7 9sC44sX3inH3OaCMLo8vpfm9CqeU5oVGRZQRE+FS86UV93KqVznGs5euKeHFBtlMKJczGI6u F45IxIjQODYJ+uekcEe97ifHDJdvUM7hauVEpo9SBGvtw4fbHh71Q7r6ywWqcE5u04ZYhhY/ MgUYQnV2f5aKanpP3r+I/Ll6+qHCFaGJhREZksAhwjsez4mtwqgWi3LiyuIS4UmQKrYlxiuD Qp7mY/2IRjfLiZxDL6kVwYbsLle/UttAAbhyNU93kzQqhDZLCxhg/DYS/CGXCAwpYzHSI5Rk PWJvcBRafs16LLiKN6lFCuMKq2vXjKkWNc9eQYMDfMX+Xvq+xsQ+bymCCbyyr1f7DAnSjzNH Sr8723xOapIWaHW4F5dluhVrAEHhduTXUxFIU76wpCqLkLdu5igjrgLV1Z4lVxwM4DF3XCu/ dZZZL2CUPGZeqhhN8nDm2RXt61iWNbUOxvFj7/QpvpYnuEuagYBP94vHOv0VGC/5Y5gkXtqU J71IXW6WqC2Fw6TFzKgVZ+17+f03/r31dNTPHlf8FctPNbMGA3h4EmIl2grLWEwgVKBktEKm B3I/P7FCgg88iBGTawFUtVyS+gYiqyrgsnSDxqc1WB+FPvJeyTKi5JNzegdiroNAMT6VmvIx AH7NOdcaZ6homGo6LETMXnaSb2Af/poy6BY5au1daRO1Yi4dxbBI08ycTlR5tXoHOqArv/Ob Lv+wQKevIIV62f6tLH7m+FubXLtn3r5Qe/XR2PQPjYvQ3w8wLzfJm2Pqqx8JOT/RcUPcy/Tl J70QDScMprLqcnXDQOAxVZuJOTLuO+Y2uFayOrdlwjO6SvzEMnxhr8lPlqQNho0jETHaGkaz OQk8RBMEgNBwFWU87vB+04u80rS7ru0oRtslTzwpxG+YX9gujLXyzUfwop5F5PM8m24PadyQ 2HzYcg5Ce5Yit19dqyi0UHMmY3LtfdjWvsci+A3miSI9k+a/OWHOgdg8UydJgCnA4R1GLoGl F5X9uKtf3GwxXSxICwZene0JkW9g4FoKaRttGJcVHA96cJ6dxCAvhtFQdddbKySV9sf1RmKv y/jj/8NX3NmSLQPwD6ITIHBVmuqXvOjkEuPnWtASouwFnnW1jFfA+VmpSEjwD4zT5mVD3Brm OhsMHP1cSM06DwnpkEXvgrtuuLzrMDkeetKcT5WedsuwNhZWbHmqpjip73CtRexcKGBRYPAP 5vawjBDeOHiGnP9Fuk1J+sYmjnZgH7OKYPJoKl51CYg0szLX0erFPpKShqpnTgW9Wt1pqYw+ tmPSL5CbTntTqqgXIb4KyjUxV4tAkozbbX3QmFUNgquEeJAXzGPW+4mKvRFiN5m9P+OfEyT6 eFWUHtpNXDegfD+oEz6IwII4s8AQ+n22L72+X1vCGdpxwT0cyZbo3uEds25AIMqHXaLyQkqH CMqlgK1FffOpfTXOCnrImUqPVgUb2VX62KOOiW9uAXUSXk3Hpzid+LRr1DZ5asv6CScjOlgK raRb3c/syz88Yyfx2Y6PqquhjFUnrQXXnKL1baVEbMnHja1a+BEoPKst0J3Fb7UV08AVgTZm RIwF603OfWASTiUmYXmn1rTPtF+/pHaZnpM/3N0bT4JB3koxbKUGRcK7+7Daqnc4sfZqDGiY Rk+ZxeMHM7QyRz+Z5cf+Yool1Qcxl/eowk4Tf1OTi1cNKMhrkhnFIpSwB/OntYCRebQGuF6A aKC3dLxszwDDvAi7xK5sJso9WprsNsy1yaJUsmvmC7uLpImTI2HAhOEYeknKsOt4Cz6RO3Nr o+NiZar6EfO41IQcVYu+P6MNqDNajUWwHuXOoR3J3o5oOesp0qoVj4l+CDIf5eOFI8XvBjNO WAE7OiKbY4QzmxfANFATIRsggJuRnuMsyd7l+WgwKGESx8YwN34O8QetXaRpEfxBvIiy8M6t P4uf+FqBP1nAFZvW2md6qu3VHMfIAKwdxErzT13WnrFMHoqVTLF1TI0GTxVg1ual/g1eufYU Ys1wRYRkEXERJ+xe01pVG1n3cLu9H3MI9+mqG9CeGqEbQSluYVnHxC7a7/fu0dk1+Bgud+4M UliBjh7TMtH3S8Ix/sxrz6KFx6LJD4nFpxVJHwS0gL4fMJnumb2zA61Kx68gYDO7eeszSCdz E0Sek7brbUIcDej+UxgPeVfx7XLGxoK4yv30WibwMFrqWMEUoE17TRJG6fZtoB/REHC7qodw OYxdPb2lAzhu4dwHLGLEmAHHTH478ujdvgo93BnUBUOboLJVGRj+1IH+w0v08BEx2X7P8V+w sqeLt3egOowP58mh6JWS1arbVdb576yfFh2rGNNfD1hPWUKXgtZCb6dnR8X/dxft9tQExfgT QFlOHY5d+076HfnI2lXujCdiLznyii1A2DyiG+WGwLAgBf8RagOQjdnc6VLgzs8lsHeWYTz6 cEwV9X6dr8stATF6qk642etGr+EaT1BXzzGvZKacBVRlYBLEPG0MlRbf7LqPJWW5nSM6MJlN 9Vi0qW9OLW6LX/IAeUQb5uCkRFBQsMkqO+F5sA1Daw1sg03X09kCKKUUyK1d0n5N5ntCdT9g iKH2jJDRXuU1qXZGgtuscigSNvR9yrZ3jawci8czDGJ5+FDVvmEznllKh4CWgecS8iqsHVfs x6Y5co6eSxoLq1/3TlS7El/68oZkuhofLdjtOKLYhJH0r4t6HHkLCiB7ZD0gkC/vWHkiHu6L i24SCPDULYmBLHCpbeH//qXWfxrXxlI7DaPMFth2NpynNwTE8TUXQblRJXJc2C8LnEaiVhh5 Vj7DMBBG9l/x2FlFyGuMHgmJJklJ4pYMMCA8a2D87cecI3YGb40yRswdG4f7RJHqaEX95MZX +Z1Z7qEP2b6AGTwXn4D8Eo5GilZXWMhc6rtV0D6mtpwkqi0mQ4u46klCkSghSEnX1mwwA7eT NThucK5lM4kplBL5Sr+5faGhIqbuPpHUGdU6b4ntPzG82smfzX+Yby+8Tx9q5Ur6W982+4ht obsgs4k5zPA/991krmi1GCAKqdcQNRbrHn2LSOx5UwP7RMGq36oGk9WFW2nj4xRlp+zlvQXw lbKepx4Lu2Vn9eEXkP99HsNSqcxkJUchviENN47xEaGRJJzuVesIf4vMpVrRlj2auzFHz0Pz z9YvuM0a/hGN+h/iRgGEpC5OPqOi5lPLBPkdGLHAPsVid8mP5gjf79tg6Z5+8y/Tp3I2HXgL D2QRr+rTBrop+CLr+FusrxAxiXGLa8oUKMcppEkNd6B7wRcEYhCbzPe7NPsS7sQAMnFH2jNX kIXfNyySuh6bAAEu0mzoXsurVN4nzZrvi0Im8xh/ZxxZRGg74POhlkXIfaSk0hQ7czYk47qO DDu+1vj0j2WPQWVY00y57nC+kY9L/4TIe2SbnLBMCTKfb7vytvjwL8Cu/FAvUODEs/6dE01s H7KEM5VKueykBHJmCSsnSDHAQO2IV+k8djqb/XFpQ1GJhgsGAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAACAAMAAAAgAACADgAAAJAAAIAAAAAAAAAAAAAAAAAAAAIAAQAAAEAAAIACAAAAaAAAgAAA AAAAAAAAAAAAAAAAAQAAAAAAWAAAANAQAQDoAgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEA AAAAAIAAAAC4EwEAKAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAEAAACoAACAAAAAAAAA AAAAAAAAAAABAAAAAADAAAAA4BQBACIAAAAAAAAAAAAAACgAAAAgAAAAQAAAAAEABAAAAAAA gAIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACAAACAAAAAgIAAgAAAAIAAgACAgAAAgICAAMDA wAAAAP8AAP8AAAD//wD/AAAA/wD/AP//AAD///8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAfOwAAAAAAAAAAAAAAAAAAHzuwAAAAAAAAAAAAAAAAAB8zuwAAAAAAA AAAAAAAAAAB8zuwAAAAAAAAAAAAAAAAAB8zuwAAAAAAAAAAAAAAAAAB8zuwAAAAAAAAAAAAA AAAAB8zuAODgAAAAAAAAAAAAB498wA4ODg4AAAAAAAAAB4//9wDg4ODg4AAAAAAAB4////h+ Dg4ODgAAAAAAB/8AAAAHZ2dnYODgAAAAB///f///9/7+/n4ODgAAB/iIiHiP//fv7+9g4OAA AHiIAAB3iPRHRE7+fg4OAAAIAIiHAHiP/3/v72Dg4AAAB4iLiIcHhER0Tv5+DgAAAHj///uI cIj/9+F/YOAAAAB/+/+/uIB4//93Hn53AAAH//////iHCP///3d3AAAAB/v/8I/7iAd3d3d3 AAAAAAf//wgI/4gI////9wAAAAAH+/CP8I+If///+HAAAAAAAH8I//9/h///+HAAAAAAAAB3 v/v/v4f/+HAAAAAAAAAAB///v/h/+HAAAAAAAAAAAAB3v/93+HAAAAAAAAAAAAAAAHd3+HAA AAAAAAAAAAAAAAAAB3AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAD///////H////g////4H///+A////wH///+A////wEP//4AU//4Aqn/4AVU/4ACqv4AABV 4AAAKYAAAFWAAAApwAAAU8AAACvAAABXwAAAD4AAAD+AAAB/gAAAf4AAAH/AAAH/wAAH/+AA H//wAH///AH///8H////n////////ygAAAAQAAAAIAAAAAEABAAAAAAAwAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAACAAACAAAAAgIAAgAAAAIAAgACAgAAAwMDAAICAgAAAAP8AAP8AAAD/ /wD/AAAA/wD/AP//AAD///8AAAAAAAAAAAAAAADEAAAAAAAAAMxAAAAACHd3fMQHcAAI//// zIAAAAj/9ERHjvYACHd3//jv6OAId3RESE72AAgAAHd47+jgCHd3B3eO9oCPe3dwd/iIAIvw h3CIiIAAj3ALcAAAAACPD/BwAAAAAAj//wAAAAAAAIiIAAAAAAD8/wAA+H8AAIADAACAAwAA gAMAAIABAACAAAAAgAIAAIAAAACAAQAAAAMAAAADAAAA/wAAAP8AAIH/AADD/wAAAAABAAIA ICAQAAEABADoAgAAAQAQEBAAAQAEACgBAAACACZwRJgvJmwipRd7k2EqDWPAYLCuVigOBr06 eR8dMKhaf8fBtlp1DUZlXJEoxZpRuw8Suj9jOaWRg2xQNmedllVOwRBXPbgMln6sGR4Zg6tO elizX6gtD8RWd4S7nHeyKEtRK1HEZL2rZywrwokRThR/N1O4JwaRg2K6gJ0AQVwbQk97TTFr eFOeEXgIZYW7Dwxkp5WsdxixS4OdXj0HeUujHCxhWBVGWlSBEJ1mUEoRexwnbZGMDw/DbJmD tyNYpC8AiDxiJ8MwipKHfI2IaYasSjJYMQWosWDBVgYYcwMdol1FwLOIj0Mtn3bGIjolGiVm WQxbGxkyV8WrOztdRp12IzUVkT8ybsLAMUcNTxJqoAO8b8NjFUmLNT0gsF1SvFu1KLOBmZLB J8I0JX5yu42WYHV2AVpOi1kaMUdRdBVyGHJ6ALBwlLlmK1zBI5hINLeBvkQ+LwnCKm1IfmAq P1QXLoJ5Mh1DCaiRrCkUP8Fvxkw7ZTHEeiSmFGiUnB6tw6IRJ1IcqQMnWSKMU6/BPZBlj3U6 lI4JdmAIoV4Oc2lmTl2hgGiukWi4rFIcCqevhKOgY1Y8Q56PhDJkeicMBgmxrJgGu0GwViKW fbozWhwFFrtPIAKEkUcoMxPBDHANTBuxBqVmK0N3WVcjRCeOZjJPckgwakkmcTMpN0mGVzUO pTw7nr5UUsA/HBfBgrhGaWg2I70WCcGFP7e7gRRVLFpbAbwqapGskYeWiFBil8KiHZtADHEi PTEnpiOFYoVgeUgJCLAXnrdJmJlAMoI9gKabusSzOxWzIAcFOToRh2pPjHlNFxslMHoZR4kf Hwh3D5OWCTU2FLNoPC+bUiRLplheAF+TekbAXYI9f5YTo8egI0oVZgo9s5CfpEa5IQ6bG2GM YhMbfoQobwwrjIOwxAYQEFxfmZe/UYGet5hmW5J9TT5ECa80SkZKiouHD7F/bTM9FWsZS3Vj PJ51qJd7x6eDcnvBdUhxjcK3bEuMqsW1RXmVclXElSFuFoYXZwWnWnVJfJV2vCdDbhNRZboq BVSiipzDaDqprWSIim9icF25P5JJnZ+0gXF/LSKcM5qKJL5RQIasspk+dg1KG1xgC16gpXAD iQiZABC+tb0wk7IHsJQjD143omB2YH+rOFWRDaXAeq5BCaljDG2wl2uMW3hwo6wdvqSZe3aa eJGxWG1eAXR/PUZifyMSu3TDq4MoMpi2dkRgoassA1Z6mji/byadN1ycw6dFVwKHik4Awq4b ElU6fqNNlxB8dScIb5+RZFacdzkBWH27w8MRoatTJCLFtnNmOnp4L2AEEXWw ----------jpttztxdiqfptirstvyb-- From wilson@sentrisystems.com Wed Jul 6 12:13:51 2005 From: wilson@sentrisystems.com (Wilson) Date: Wed, 06 Jul 2005 16:43:51 +0530 Subject: [LARTC] Re: Message-ID: ----------zsbfwqglslggbghfeckd Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: 7bit >Animals

----------zsbfwqglslggbghfeckd Content-Type: application/octet-stream; name="New_MP3_Player.com" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="New_MP3_Player.com" TVoAAAEAAAACAAAA//8AAEAAAAAAAAAAQAAAAAAAAAC0TM0hAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAQAAAAFBFAABMAQUAAAAAAAAAAAAAAAAA4AAPAQsBAAAASAAAAFIAAAAAAAAAwAAA ABAAAABgAAAAAEAAABAAAAACAAAEAAAAAAAAAAQAAAAAAAAAnBMBAAACAAAAAAAAAgAAAAAA EAAAEAAAAAAQAAAQAAAAAAAAEAAAAAAAAAAAAAAAVsIAANEAAAAAEAEAnAMAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAABgAADoAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAIAAAAAEgAAAAAAACqRgAA ABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAAAwAAOAAAAAAAATgwAAABgAAAAAAAAAAAAAAAA AAAAAAAAAAAAAEAAAMAANgAAAAAAAJ5CAAAAcAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAADA AAAAAAAAAAAAUAAAAMAAAABMAAAAAgAAAAAAAAAAAAAAAAAAQAAAwC5yc3JjAAAAnAMAAAAQ AQCcAwAAAE4AAAAAAAAAAAAAAAAAACAAAOBg6AEAAADog8QE6AEAAADpXYHt2SFAAOgpAgAA 6OsI6wLNIP8kJJpmvkdG6AEAAACaWY2VKyJAAOgBAAAAaVhmv01K6OQBAACNUvnoAQAAAOhb aMz/4pr/5Gn/pWwkQADp6Ln////rAs0gi8TrAs0ggQAWAAAAD4XJAQAAaegAAAAAWJlqFVqN BAJQ6JUBAABmPYbzdAPpjZXNIkAA6IoBAADoAQAAAGmDxASNvfEkQAC5MUgAALp4I++Oigcq wSrF9tAqwirG0sDSyDLB9tAyxTLCMsbSwALBAsUCwgLG0sjTwogHR0l10ugBAAAA6IPEBA8L 6CvSZIsCiyBkjwJYXcOai5VsJEAA6B4BAADoAQAAAMeDxAS7JJAAAGoEaAAwAABTagD/lXAk QADoAQAAAOiDxARoAEAAAFNQ6AEAAADpg8QEUI2V8SRAAFLoDgAAAOgBAAAAaYPEBFpeDlbL YIt0JCSLfCQo/LKApOhoAAAAc/gryehfAAAAcxorwOhWAAAAcyBBsBDoTAAAABLAc/d1PKrr 1uhKAAAASeIQ6EAAAADrKKzR6HRwE8nrHJFIweAIrOgqAAAAPQB9AABzCoD8BXMGg/h/dwJB QZWLxVaL9yvw86Re65MC0nUFihZGEtLDK8lB6O7///8Tyejn////cvLD6yM2VTk2VTk6VTk2 VUM2VTk2VQ85NlU5OlU5NlVDNlU5NlUPOSt8JCiJfCQcYcPrAWlYWP/gWVJVjYW/IkAAUCvA ZP8wZIkg6wPHhOhRw+sDx4SaWUHr8AAAAAAAAAAAmsIAAAAAAAAAAAAAssIAAJrCAACSwgAA AAAAAAAAAAC/wgAAksIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAFcMAAAAAAADKwgAA28IAAOrC AAD4wgAAB8MAAAAAAABLRVJORUwzMi5ETEwAVVNFUjMyLkRMTAAAAEdldFByb2NBZGRyZXNz AAAATG9hZExpYnJhcnlBAAAARXhpdFByb2Nlc3MAAABWaXJ0dWFsQWxsb2MAAABWaXJ0dWFs RnJlZQAAAE1lc3NhZ2VCb3hBAAAAAACH+50ry/loKwSUmEGzn1EyAeEfCO8FJne3yUKefpBY Qvy7FuqpLhH8q9GmyT0VL5BBPHt/FqjHjTGgKOsh4ELAnXa6Sxh+22Sv3YEzzm4TMIPbOjLF YSCcFWynbQNwb2sqSbWxE8Km6af4UXbWD5dEdzhsUXWLjV9MQWiz+KUZT/OItczECP1A2iul IjxWaGqSOYoBRdzOzEjX6NPntNYC8HnEZ1V7ayqpD9gJssdWfu5/7yGwwLKMUdjhplwGygtY prRi3EmikEhnaymGwvE72ptXwol4GnPcU/jQkljZ79cey9wC+ctqlCZ9GLb66LtUI7D4tzIV IUVmgSEplthDnrh29QGqcPTANQHXWatAxFJN4w2qN5EV76dhFya6eQMiA1Nsc68sN2+rtphZ belvUzSbbeNC9QWY3hBs8ey2BBddKmyQ4i5BamjdMktjsCULDcKXCGrJOprwXOLlDjCYYCtV yqindHH0gSRabWmaOeSOX9IA+7viHyM3IoE5HwOpAnG5xEbK8c2i+mfNAC2Hs0d5/uR/zpCb oTHHDthxfIoFQqOwqfNhmZTMeROFeaGhztHim3ec7avQtGtLBEU8JnPyQjInaIwz46mOjxqg ddc7cm8uJfeB1EgzjjcygKWiNqvDIKk/qxoxXum/RDiPNhYs2kQ79IXKpPurvVSc0uqcG2aK wKjMEm2Dj0WTzDYbu1dw4NlrpaCyhOzrUQYuSzS4CPRYLv16XeGbstABzg3GSU3iiRirla5+ XKDj/jgO5Au+DXEpe/8m78xsz7zd38CzLGA45AMZpZSH/5Y5gtfo1bXca8qqUbHzRI3Gs+cg HtD6w1e5jt5zCCBxZcYKgk836dHBQWyGQpfwPjxuyfMDr8XE7Rk1ZSh34S2OYhWmW5b8XCIh 0ES/1qXPmKcAdlcSK4tm62h2E08/WSlEXtHmLbeGlED7zWqqGAFVJQbbfZmMgIkP4n65LiFh z7/p5ziLBSa8N2nUnD6e0lWW9vt5G7x3jWe/CTj6cG9ERx11oINoOu2rQ5HkMPPApKV0zPYg YttmP52AhG7XglcpTOzEJQqyWg2HdeWac8Ba1kmaYl+/7sSfLcb1ix3FDO+HAQ2jySGdmfS6 Ue5UOlre+HAYQZQh/vNAYKFkVsxTTr1mhN8Vzz7UW0FKXSLZablvomYIOHAwv+5KeICfDTyz jN//JkiBO41Z+g/8YrKMpmsR74wx5vZhj+7cbXOUfFIGDkHbRPjtU+9BTGQwGFhFlGCEUMlR L//1lVFUaNXSzk//41cIrNMLeVt8AdSzb5GY4VuDKY7HwRsBOofuYhwZEVfveEGgIoLmyWld mMtV4suIDhh9ENpmyIUG3Fdb6jgONiHqlGVUiYfFMNdR/8Wi2AMHTuKeSeb1u9sMXV4V8fTy gA6g5CJud+doobCa/wU+KVM/FBYCnIuTB85G9zLaNwKkwXS/ZwCQsP81olZVgyKcsKSIukqM t2ZLJmXZ0FTiHbbElnt9IQzOm8yBrMTpwZg7xPTzW9btq/p86PRXHoDtwjc1CtO7s7weMoHu LW5Neh/EHphjVHlP7RwUTuPf0fO60DjwriU13yXZdk5Z2NJb5BSoPGaCc6QmITcyRsXLNHIh xtW74w2E2QewLAXpuODlopVEjhJJSIdo97/1CoUYVwEMlDlXyg5JhuTGsBuaY+3im/8Z+/GS Siej0oik+4U30Ig42h1/LT16X7wIvxZCBs8O8zCfoQQ0Y/z0bbJrwy5UDwobLtcXcMz175sd TdjdbUwEdbsURXdb1kfk2pqvF5WaJEvXblSweNMnqw/cwV2ABWnSd+sfYrWsm960RSsl0YBV ue+cEf761CLvW48M1yzC5KvG04MbsRMgSaIGzyd7gZLvPugLJC8jnyUEble3QSMZrmjHPZHM deZ6NP4Y+lYdy4xywrm2KPomg/XCIiBh9BdCPIcd9BchmV+Cesw2IrruIi4sd0hLdi21gOeW cWgRsNW7yWoop8C1Fk26kjqL7TzSW8rKMjjxxpZGiCoCP/mAfx+gNpt1CixNwm05HVM3gbI+ RSI9XNvkkAu8eFtoAdPr7eV+EZHhqdIkwG61qI7FkAJ7eM52CN8FbPULQOFCWL1xOejoepRn z+IP3lqWW1OXVcws8slDTmDzHeoWq+LVbDU1Rqu/SiIrVyCub80+qtOGLG+VfQcb1bYnJ2WF 64bm7G0FUDNheB7ngdW2W03gB+jkBqb70xrz2JKgueMOTlg+Ox5DVM+r+N8+stLsC9UEI+Hp b+7Kqqy743XT+bvZZuzsJZiciSdwejR1dHfwU63VH+B7F1xX1AZj+lJKN2MAYnUE9d+ugH2D ldYmesi6pewsfPZ+OC3zJ078TQix1AQtlESBWgkTluC40m8KlLrUjBA45tEkCdkJsBQJevWD bulqGsxgrX6kRGBFcGmxewxC7cNQ6gLOCTOhjmYHEK/RGVJxvCuUTqIDu3pRxKdqv6JtP50d lsYCu7oOD6Niyde46rBtVcssETLUoZ8anpUJSWjGejEmFgG3v+3labjfdH+YS5/9msnI8jqU W37EU3gS3HlHAaYObkVbVkq/DK+P6gI2QuBKmEBA3e7UDQtTNWuRMsDPTr590M4l+zhx4sYQ LTQ6EAm+/4k4mPR56JzOKQT41wwoI7OlDBU9Uz0FQQfW3ws5SxD4HukjnJ9sxCyMHcg99EI1 jQcA3VvSan0WOG7eoaDYcRGBM5j8GN6jBYwWFtnc9NJMP75j5dDb3jK0MzuGy988Lyn0ptJN RBf692BamuDp/gU6iyLsjcbFx/Q9hEHnEJ39LtAqBtOuMCsw3PXsl/KYbymphNyOUlj7T1q6 hvoF9aVRIMoLg3sJYpGbyU7Cdxmx+lEEyso7sH051C7v58luYt74Sbl1VgYu+Ig+qo1GkLNk bMkwZG+zP2cf5l2FV0kuXutRaR+aIcXUgoxfUbkTVaAvwckmBmcEPo4XIXojcanH4P7mB9TD doczIJ1FQeSAAxZmSJonQ7qnjiAZldRciwHyEd4kPCncy6+B+M8fKn38q/DpVTmmj/Ldxy9h y1xqOr4f26EF2jmZckpJ1YdVGIyUAN9E64AKtqeHGKhTmV5KG5nuwQPZUp/naElESeJ/f1+6 375msRGcRG7GCamkyvDZnbYh3B6aMm0U1En2hEChXniVEO13bLjzlW1A9LyZTHtGau6o5diT pNG4ixIzd6ilspI6yAxCT3rmE9WmICEI6nxGUGDrxjVi2JeD8O3AlKw7/D6+DkCLGNc9Ow9o RTfMBHE8OPCHkHDI/VV8xCjmKV/ASdtjBUvLqPFerMM85dKqIh7x0kZxp6tWGGWliPiO1yJs Qg9emkPaBIdRGK147p8TQvcKESTGUs8N3lmxNKeJSmUn4hoTQDFnMsrrQCW1+eWXW4EauNlC xIgEIn/7F1Ufbe9JK6dXKjPYLmitr+fb3koDLZbfzjtgsEKpIEcgAa1bVf1C98cKnOO5topu EZQjAQrVHQrYzDydeN6wxsFlWY6NBl3DeBIHrN6dQoTf3jiDOZuqsnxwwkv9YBKAl093LpsU OkKuQ47lSltfLnZh1qkdxpwSFDcdZWOL1UDoE0GhyOeNvBQhgNEkWLn5QyHCS0I3X39pTt/s CpiBue0swvws8QRn6u3XuEE3VbcPkL0jolT3Fd9YfVZsdOUmLGSB3KXoi19dI477Z2gkU5K5 /ER3lWwD7z4sNPh7+NIyEXGaB32yO0VRDdvfDn2K2LKefui6FtNHOjp2u36AihUMgARlset0 TL/D01rcfKJqxjtRkcyXNXbdN62hBUGznaUeeQyFiaoKk1RMIQhQyXEXY8RgrRVFiX90sK1V JncsqgnkCFhtkSV0E4EarSuK6Ib7RF2VMpty8eJfGTYp2y2u1HU/qccj7mt2VKdTufwBBkih 1AMCC5P17xD0oYLcjW3q14lOoqqVvqqXzZy6FrmyWXYkZTOtIVmMtW48NXWvmboEPv75W30R KHEA8t/ht1Vt7WiqLJiAgYhbJ444AOVgJBSMQTFWVJWaniQV8PB7+gRE3pa29QIOKqNpwvQ5 zrF+3OeTPd9nX8/NUa5JmyrbfrBxA3uMNXBT59o0JifS12ftvpHyaOGKt45qBLKe9yAxKa3L x2Ada9tiesO9mk4SLsjjOQsNRvlpVvSnBuVVOgTN/cAktBl8zalBgeHx619psRNmCE/fVdI/ +Rnu7BWeHmbktIH2037faoRkShnkPkepZslrfaLqNoY2n50t8XG+bAiRspM6HqKP+6SLzM4L mwG4+4EPCACJmAoo5oUHn/eedo/md+Qf527YvuGi9317oX9hEl8q+Prg+I9Qw3EOdaKaIEo1 fzqzDiUROK6xE7xF/xuOTl0M8rfZFb/aNJcTc/mHVPdZtXLX8TGE4Z2/mXaenKyK2miom06c RlfBF3BAKGhRldsYQ9xPv5cOQeMZwTR8PhvjwDwNnaOO13Oh7LVQZuoh21H3Q/ApT+jqB6v6 LRlBohX/akhz31LpT4Wzlqxlt9Wcl62KApbKcgSC/LkY0+5IUNqNK9kizH1Bps2brtKEIaXz U4aKLuFt+WfDAgikixaNvezLKodAubvWIu97VjKg/gkRpLVNuKL4WXFEXmTWe8U3y4gBn7UG VjW/PZUyjaQSiJexKp2DUU7AN/Hcz5nawHX7WrJ3bmgd9wN1Qiej14f2P0cAqGqLbu5rpgzd vV4cE9ZcewnAuYg5ZMegd/92jaJJ8LM8fQ3c+A2vjiwdY4uFN8RBGbcVtLEPL1jNYJOPJBdC eFNf2IAUDS+GYYRtKmh4BMXfjJ9pUDqaYr4LXYOInY/1hm2FAnlHICWDwOs4gCN84PsBGSGc VLLpgW4sPEb8u1EkN/NduE6bNF9oDnPIhqkLC24+bPD+y0bQ/VvvZGQMbsFBhwf1KWcgqkjA ejhMJlXsbAjlMwkhA44V41AjbCJ06+j36svStsWWYdA7FcImLiDZmp3UGMAnWRjQ55nJzEW5 DovNjrV0IKBQYNFryGJgZvc6H0bfSOdg3Q7RmI1s+xgyK5gMUdvjKCnKaqH7JOm6KBRrCRcS w/02nAs38v172NpMJu45Dqicgqme2pIC6w7FIXRuW4Y49GY/1YmI7z/EcQ/qIL5YmnuScZ0l O0bfWN2JyK7lc62wajdB0wPlNEcS+F0vK5NmNz6bvy3lYi5AQXCXqrjX76DppADcOtxBbkWM S5uRJmee9mF8fBzaw5P3k7g3MHKwQy7SLS4yW9+fK64clkW6rbnlfTjdaJRhojIfh+uhSsJZ gd9qeRTyFsnbLmyAO+3Emyvk27NpM45kmCbZwW0mcrqSoduGBJvQ2+yBIHhSxdlQvqA/l81k 7cOiRK3LBcIAqCikpogQeMhX7+OypJ8kRHFAkBAUX+3S1C9tyXvagDjJIzWvt0rnrqxkIqsa wjqDeM6DGkawBw0MBLYgPstc8jr9rgpuTPXowmLDW80r/E+Z9qQQ0eNsNgB1SOzdIDNzkQxY axxsbeO+pGUsbfEWNxssInfyjjTmMkb2F4DOE5tzOLvC4GWdJby1ZCJZO8bXn39/EkCNb6tJ oAnC775BP3+t9jHJ8TNEZztjpHjtli6jLn3eYoNVoic+65+rx4JY8OsOsZeg2dWhT15Pi9PT q/FCMfmUk7ahgM+rMiDVG0b6DZRweEcgMQ/41LveXkr0+LytTAMtplz3x/ZWIwso9wmzlOjc /TAUi751pCFoxg7NlBQfb1XNT/1BBcjtSsPC84BglUIPuXNgF3Oe2fH4NW9e1xbvaVXYyXsT Nd0SsyCQp5RuTrQTLPYPpZPqu+CyZxNzlTA9uu3MIv3HwZORZChNxgou0HszGqlUNT/v8kFL YEuR4XugwGANx+2bES7bzbU6FQxvNSzH13eJ1koTTJfNYXUH54V7YX8nfzrVpU4kq2tzcFPk yXoaowqKcbDYoFs0Ip5CSFK+4H2q4e2ohKtF9BUl+b32WD4l3xobZ7IImRFrbYwvP4H6cFbz yKhOsJ/Ult8J72Vopk55FOQaGMUUjlD+oiT4J2lV0ynPR2Q3BQqe4ZndJNarrhbTzW1+Lb6C sEfFjzOIgP9Xa2w8F5M+9jHJpFGSWUq5pkwuuqsM7b3W8vfvHFOZKqwbq0RmZNkxDrhqoqDY zrQLSNFnx4mjMGKvjyI7phslxxt9OXsM4QYKjS+CsJiM5EHuOfYrW17prsV7cJCc/pU4SCtK 9THlwNjwA/1EM0beGA3RgJ8fFzbJwC9id52qec3LoA6zjjSoaxGQCgdgYhE3bMg9y9Nj9Z2M jNgJx2CCKPLKrdbBfczkA7Qd+sW76xhuEfHQwOEQFGlOHTFjlkq4VCANcWuqnNff6mzkE/W9 lnnVi9cUcCYgn2NGjyVAbPo4Di5VpDR10ew59weNjmGfh54Emcj7+L7MQJu9vVOz0AxUyizs ulZDwZGYMUGAIv+l2SkyYTxRctVdKfx7ru55IX2hVrifb/Bsv/EIm1eq4XpXXSwLIAmqqPqt EHjgXNWZFLM01ZsPUfM5ZujWdlJYSihje9EHAFyd7q4ZfH5RbXrZfeSsF01kCy/6Fm1tnkx3 8Qwl6P73Or5dfr+oyOmB3zmzSPgn9Kj+J9oz+cXi/ORmbb1mkavuQl+bKrJnlZ94pz4/2Fim plZmfaAdit1oRDaPT6cG+4d5VqYiyImSxradSo1Lg70RJ4eqQk9GKA/jGdr76AIxXkAjyO23 n02jpFEiqY6J+Y1pyxX+2ne2H92HTbwHJ1LuUDhtdYD4fxvtpG2Fv6vtYWQhzBk4gmitkfdS vG37r6st2hhfF9bT+cRurZrf+qGMtBQyerDltjw9nTMUUUOMzH1RfCC3ejpuonsdLi6Xg8ye aHU4Czfrkfwh/J46BTP38e+xdY9IQeSL9K1d3CsmyZW+RAPI7ZOwC4vp/3o16qqOAWjc6Fhg W17uGWQStMy3es/AZzbq6maqjGkEKAYI8iSdyeBx7Y+kW2PQ3pEVDoGciK5PYRJb1tT/Gm5C w6cY91rINde1tF3klBk0s+trn50ttzeFIf6gA2/yT15IXahQNNgRAKlZmhpQfFW9el+8AuhR KXidVT1orWOascZtfgnsqGWGUG3w4FCzimnFa4+8Fy8G1VvhjPTVJZp0WDa0W6rPgy7JOR80 QSch6u0Snph3YOwMjg4bjwIgRoQ3TWKiQWaHIo830oprMUSF7VW9JMa+FmYJcXtRd6zDjNnR GXuS2BGW4w4Vc6ReTZAUuini3cBq607f7lNNIhvigcquAZcBbBq+oiHebmaA+VIXKnzCCXR6 WVJjovUWJuYTgCVdShAkZupnTN76NVcvHXcVqyRp/h+Sb55K2D65z9QpouukeaMhDef1c7+Z JfMdphPzy5NEagYwX3NElDgQnBG2IsLukrVWKs7xFX701xijZScA3pecEDQ2pHGo9sqZStBM 2bNSMmMYlfiov27RSS7yOVAQW3D7Nwlbnv9DKu6EeCCENXkuQorKDXLlbcOf8mlDhGsSW/vE qVxpD+ejb7Vs1nyHaFtnJqypSZB4PkvPS5RRO6xJnNXGaf041Qz+13inXV064repPRmleost 00TLZ7HiNHSaFQvga0DIlWnlGCsjk1gZRtbPxFAVD19ZRoU9Z6lXiCmILgBECi6yD2UQo/Fi GbX62XnY1Yd/+BBxbQNdMFlmiIxUzwPI0c9Wj9u+xrLNC3GElhhgHBz6623Kb7H0pv1HQFH+ f+z9o1Wz07m+7odouw5erF97xCARREojoOVM/YJRw14l0/zhtGQGfLoYvrUylOv06L3n/41j XYJ+Gc3lYfs9Y3wlCHQabePSqaQYyDpE/hNZowWP32uaZmv0JyySBZVqbV0sufhSHo1eIzg0 ZU+qUCfY5hQDkbPbVw3F9kLqr+VFQeFlmeyw8NGqPYb/wCF4wnaOb8X+8tIdHXSwt54TStLX hEYV29Lsg1gtD1a5eMjsbL3TNF0JI/bVKzwSbtD1yOJDGnD2acneIV9bN8n10EXD72zR9EQ1 nIf5Yi3vX9OvX99ZNyVHNL+mZoQeRtAOpGavBRtxMXK2nmEvHHVoSzgiAV2ZEWaA5bHBbmj7 tVB78Gz8vx5mFJ3XqvpadbR8Ttiyr6yPWi6V0iOjm5mRAZC26Z8bglrj6Bd4v3vp5WmA918E rZ8UnKgy+HMWcU4rG5TKIi++6FJblZ/pJQgf8jzITE8VVcEDu/QO0ODLNGNwtkkmPLmxdL4g y4jVRtFvs17dEYdXOTbt67Rr8YLI9NGif7apeFrBRi8qFVupSj1wm5/9SOJvBHopx9k9UGp8 94omgCt24CBJgGKlCOWDeMGKuROBok6YHDnVqhbriJcgM9pwGQ3sCZnoAuMuqLH8eA6JTjPm llFevP4dN0dKgcERb17/mH37t/ztwksE8Afl1peub+g7PjUQWxXT3WgkgVahoo/Wknuq3pfV A1+taQUjpuuigqvNwIY6gnxCNxxqv27VLWEK4nylxnZsAyZHrFJ7/lAj2Me4FaadV4UZj5KZ HpkqSXz+DczseHGnQ4a+ElBAPIHD/mVCeolyzGIN7Ph8mzh1eQy9Kpnyg0M+h1hqBHy94luC KqDto8W/cqTcFfccQMQ+2isdRqCTqFshyL19zVc6cg/maPKc6nrWGx/bjmxm5nEgtvGxaQmn l87m54B6YJrNBx81VUxSTavHToCfvzUXpUY4LxTRuWJHifJ+uv9CkOeBSB1wHM5mm7SWHZ8I 3qYeCzKmcJulAD0wpv/i09+0SeaWtfQl2yV94E3c3fqg26c7JWWxhh8O0kwJySIvrhD4v+nN 9rUZjE/T/BfEDYW2AUC8bPD1/FIEHIq7drT1kLkfzyvOKbfMs5MS7QjYDWEFy0ASBAJtbcWR iIBCGfuTHgp9Y9vG6xYScf7s8ovjvR1D26nWvBZPcgPk9Pz/gInIddgR5sEVg4uiGcYFF0CK /JfLzW2bW9QEAZxSNvLSag1xzg4+gfFa6bBsuIbPq+OyHHsjWOAJoYW9UWOA81dk630iEZNC Vtn30vF9hu3rbfS3mu6/Z8pf9lGFQ4+UWvZC/GTzA77ewJFv45+cQx+cqcz1UVvEwcxNiDcK OKtKpuK2V7F1CbcojJMS7o7A0mzAJ2iY2cv0SS1xigzjjjvBlIYOdcCbclL/3QXtgseDuIxm Ed/Pe/vcfIlL4mlxlPQUp8mc/b5ghWD7fHce3CHeqzYOzJZ/Pw8+45Rh9JGX9XxObt1hqYum hd0JpHFEO5DZwliQUnMHlaoZKu3LLSovPKCeS7JSYFdaimCZ9dldboj4E+BPJRzW+YrYCvZR R3PUl37TNOVdp519Ha46ySLLnnU+JW40/En2QgD8YmtZ2+8RhLaMi5YGK6O66bTyWKxRqCHR p/ez1Kd4PBPjN74uvW0N6zLnoG1uQ9rKY+Dr1cLVuxcBRB0Nrzr3guSvhu3Q01fROLPOlIXM OzkIo0jmNwZS/vyntbDSTaQ05XZWyJc0mJFjMt3IO7Mfx2CE+OG43QUqKuGPQPzT4yOjOmSr zV2XVHDpgYBMGkthDUKVCcV7C9Sa6la+MEWukbedSD7AmoFRPA3dD/iJcGloriytE7g8OEbt nOLR+5au9D94NsU3ipixqHzIreo8p9jtKxayWfaNAszz/9b2nanEMvXmKsS1SsrVQvojKNi2 9Ndc4RhKP8pLS749Ih1dxs+YeLeVRTRjz5hMtBnNxs0D2tW/7J4+aso2yHp9BMBmnakCGLFs z2hMaeyMqRkAo8i2NxeOhzfCtY1fVoD9oyzFRSZo7Gq6zuQuA3y0unbNcTqqTs5ZFij0+x+n OxMHyr/XkiagqimEm9r5pU3ZdQMoV0Rd07agYPzkxry3ixgszr8ToAhOMjP6ucwmwTn7s1nw GAdK+Vb9HE/6HrlUqDcC+/H7olvrWK4Y5HPObCG4BUct3JvagI1vgb6WJH5IYZ4Bs0MweQ7o p85kG0MwhHdZ20BWfX+9O+T5Gn7n+cvHZNR2wThBxQmxkdPnHhMd1NG20IkRPrcgo+mnHlUv S8wkTYsuebh3OnKVi2dyRgTMnvrETxrNzqWvVNea4yA0fYPfrJLXPZ84LWTci40NrHtaxfUB ItF1Cb2dco2D5Uu/VCPw+F1AfGchXHdOBGf0Qh5YUZYQ1kfUzMlEzUJLmFx8+B+s/rCuultV F84Eq1DZGDA6t0NDy0CHJl9L+SXFJENCZAzxxAUCm2V8iP/RD9z5SxdQhviZToSTgMRizOoe Jj/uMke9FE4VSih/lP1rgAFr9LjW7pf3uWGmq1g8CmhQb2SAGivbFiYkUrNE954dKOW8+Te0 jXzfp2/6ANVfPk+lN1ol97hinqvOB4J2Ua75MUWSFL9P+hSSHMaDY1oYms+0PYC2sempnr0x Ns8A2jQsngzAHHW17/wHyUqKc/cZWuilWRL4z8WzDlz14BGDFL2q/BDNoLtRYEwtaEDkBa9+ zNQgrranCyfKCBf2HaamvnHyAfissrFd/5racy3Pxrn0dL1+LozlWKjQU2Fw/24JTGZuyJFd ouTBHXIbWgP1qilMZvaGuO+BoXPWv236tm6yYdoD+bjk0gAJCk+iJ2cwO8eJUyupZeALULrj bLrOlXoJwxssSaCjMTgL/GfzlD1L28lNK+Le5ZJF/KMNpTUex4a8iPYJ8sF9BdhnRz/y2buh soARn8WVFZ4FSOE0Gj32dROBdqXVgtbvLHsk2vArjsmqUqlcKiQs1/dt96cGe7hQN0CHFPUq pei2eLxZfrrJ4Ndcy15hifgwKBSAJMiXm4IHkbJ7/Rxlpb5ee1tEmtgum8kNRuLqc46gjhE0 OZasP+oVGCKdl+rkN6B5PSDIaq9lEWbZ9N4c8SRwY/Z8hp8e6JEGeO1sOwKRqT9WB4liw1ce DDI9csY0QVPPhB/IyoXhX+bhjV1yGg2f54mLXlzvz3k6yu+9zkPIolxWdb7rG5kWGzNuatmi nP8mURji5LYaCCFFZS4ZsAT1ZHEOk0lGtAiuH2JdZ0QaB8LTdCF68KR3IJU4DD28lxj9+A5N FhQ4nhSq15DG4bXw0gNFQBl5pDooBCab5hUD5kAFauRk69yNu+pMAVXwdyjtc/P+qB3wJ4DC tVCMd0eOqm5Ql82qe2cMdsjstlaZHmWgEIgz1raiqjFRa8kGZ/cJqisLTY3fRC4Wv9jJHow/ at/rl6BSvkzh9uUm77xCdPyC8+j6mYkrLGzvQTJCk2uMj4uLYtpMcJ4UwTHJ6wIDZgYbvgNp BVJFG1pGvyB8AUhBnT+RBlZ8mPlQ2Rv92QnZmNIOO50e0A2JYZ8+RTujAWp8X7DS6BiR9g+4 kfSVS1qjfv/y77jmqoHJNK+rhXjJnrTO9WRFu40syCm+iGEPeXW2VOfjtebyRnlAf1hAQpiK bzF8qeigATg6GW1wQupxZLfvLO5qmApwmMNFDT1TxaJMe7XbHXgO2wprEnD7XeuTeO0Y6STT uw8EfjHFR4Nk0FuXj+DupoqX7c9KrWE7sPXk//9TWIHBpKEQBONWe1DDR3is0KVPt25kFcl3 gDUIt1xY13Qd70BeceARVPxrtFOU35eoBOTvr/wcsBAlxPyNuctHMA6Oq/9SWl8+JUOV6BX0 fZwHUycRKuESZzru5qCWVmG7jc4pKApcdnq94vqG4UcqU3F6CnV7w1ZvomCx/QlNQiFvLVlG i2dGAma1AV21xUFlYzQ60IULp/Bxe4OORvDd38BnSMxFBvMjV6GpLbVKzULc5cPNSx7TIhCI ZVbAhKrSYqu79fRg2FdqDF5t8cCbXc+Fr83njUNN9ERh3R7jn2sZU7GvzOsYmXi1A2XFoS2N wbOG7ZfqNfbGaao8yW0LdEABD0YYtZzp5dc1Cab2pTHZoKHXeZR96oa6tPaC2ZfIL8HZ6ZZI Uml6opVi89nAw1BRjlNuTym0mf1blb++G7qp0lVaod9l3OGU3wA8eHGz6f1Tq6oCUdVsEk59 Mc2gjlFLTNvX8YwuypyCpH/wUyBJXOtCcFD1d53mbXUM7s9x3XNeCa/MOJ3VtUe0L0y/ihli A8MVqIgU9FAqby0E+T40jHgPF3nYHn+eVpDOMjH/5d4kbsKfGtBkx9fFbUZ4HW6JvdfKW+Zh jffWp2gwabmfFf48Lb2lW4PXwkPT5SDS7/G2dFwd8yUvTv+HMokLWfNJ+CwS/xiNWjKZ3ESK qZINDFPA5PrfFhOsCMjuNAw02sFGlW9mIl8I40mB9+L0pamp0iTPZGFUhZ4kPPid1+Ji0gzd RJYToFjiOgmZ+c7nbrKEQ+8BGHbbIyr70gG5yvV6n7WPQutIhZSdjKnE8Y56ijn7FtmB/LmO F57i6wnleRO0N4IdLxC0s/1BBeoPBexcmoc5SS+HRiDIzOqF+n86GRzOsWp74Y/HEUAv/bNI 3CoZcMTk1YkYGs3+3MtaLGQxFLEXmT4H5Iru7eTp4du9voLPWMhvtvaZr+9Wa0KK5fVxmC0m Kl+A9/R71XEWL3b2VVQNdpEh+QDLW8I6ILKr8rr1LK1l3zU9FWUPhJz7Shp3jzQvIVc8ba5b fCiCtCb7cHhIwqyOs7MYOJ6yoU4v6XlCtguR+sKFvySwNnYGqU4C11piy3JJvWaVLREZs6Oy oASQQsyYo3u9Cw2yf3pe9o6359gsMHY832qXgL8N1qT9pZzODnUymlE4945BKwyMY2UoOUSw zTPSX7+CYuw8yE7+efFdxx9GRxIZQP0FPtLqKtMjlwPS2AO8x1ZIofTP1PNI+tPQFIpoTnkO biNG+U7ylM6s4h8VvgXpNfgbSodQuJLxV/wzJs0bpesl2QnWiMj1l2PrZjych7WzKP7fOffu YscdvKKbccNjyYdG8dBUrrcrVx6YLRrMHMFEyfolk7IebHL5ktuqRdD6mreQ5qySGlfN0aMZ FwAAfOuof0aSUlvq4xvzyd85z4psrt19U2JptkxM+40sNA7oMLkrrhBxwgNhEnHXBCEeEsKF G5QT4oF9S7OFJMXwNAKq7RLAdi5jzKK0jc0FMeg0/lzefXuEPUKYMg4scGuFw4o/TB66b5ew /ZE7CZ1SfJdyd3Np3bvXEYu82Ld48zQ6uU6w9P1ENgD7HPpSI9LaGCvnK553j8HhLpTpmfj7 gZ7OsCXeZyZFaG1BbOH5m1/OdNcIf6bgJAKqqTePlj6biFjiGMvnRnzvqWta/I1T/2Qm0Hfr DY5nqKshtcajnPPAEudK5Mr2Ez/GiYbx4Em3MzdMF3i+sbateLFncmc3u/Nh3UjXTAC9YR9b NDUSNiUFbQmPWTZZIvVX/j2DlEevwcJGVea0hEDu1zSp5GjCbI7FLAvqjBCQUHmX5UgugnCq vSgwnIc+pIWHz8U157TZ5EHX58QkXs5qXYwbQB/3EjDkK3HHSHFkOCICY7Lefv1+sed4+fZU 3W9k3j8NWOuWBYrGrhrT6lO9F17kXvurL/U6VEr5ZAxuICZpGWu6XVpwfexiwZpGkGmHWc3J WJ4q8n5JtkQqK3O1C7DXVJ80X5DfqXywnG4aghPMxy3BHMXx9L9CEJqd0UyajEs+HoLYxb8j fusPnopLJrANJzZpoBqsEohMUZAVbSoVe9gS6S9Li7w8lSMIdIUHX2+XdzbLVqvUqC8kFrSI X20ZmxntxkCoTKAqGGVpCo1YrvR0gfsfNquCVm5EaqL5weOyM3rhlBG8QhMt/RJoyceU7OaH VB8BOfpwByOOvB3tdek9dtspOq5BFG7K7bGMxuHt6MTO4ZP/bsby6fKnLPx6mt9HAy4rhsoU MII9JkmQqUzywBhl8TVuk6DPAPoOpAwxMUm8yleaq8UVs3OYA2sSFzephT+/K8SKfNvSlbRX CoRrsjRqRbweCt+M/Wc7seB1muFb8IRy6IrHBd9tsy6evZZUkfdDkr4e6oCU4xRuu8SqeTB7 srzwLv9FwfNVB96hk2dKycFpqs/dFt2F/PpYI7oTl4h8m0RRUPQdkmoKBh4HuKIdMGmQXsUc G7/68JumfqcWLgE2t5dDyZ/JLmR2PIKF+j24kZmwY7hVZIvnL+jF6Wb18xx4gVhSG9jpgrJR vb8TWLPYtszZwqI0YQnSv7CHxIiVXag6KgNS4//3bKo33/fYHpvr6/V20lj1ONB8zuhLdKk6 LMsW8y4aO5hnI7PBAMUpzsbSaXEbbpoAcV8G7FqaS5o5amJH08FYHjBIaekU5Id+STYNa2Pe uRA4Tc7Fjg7aqSA4UCT7CSftlZWha4K9sB3NfVnMzRWEZrLuyrT+YLJB3f7HvEV4OsgfNMIa D542soB1ly7zTjzzQW/DQzqHA6nDreTXW2Ce7Gc+iMlHv3w7Em2XaoTLV6Kf/McjEDvKvg9X qA7zoQKVmW+FXXAczxbR1i90+7BzGMDr+hCd8tH8PGDDSLGbRmoaMoa77hF5xejIxJbO84XF YxSQMYMGKbeu/livgTSNLokDT1bzOwh+JPRo3PSh885DJnZ4k3fOL/3ppK7QONLDcQHVgntZ n2pqskc6rfFgmhoh6uwSwP9jg+XYniY6TIsZxLGsFLvEJuO3YHGBi0h9/G1VmvDCwvt3tVVJ ntXXCqOmVE+hyWYHeJG2FpjUZNXxaDrtjQVO6hPEeQk9d42RX6hJj7jQRjnFdPdm+uzhc0vS SmZszNOdxDDYxMbi6+xxmvptkTAGntQn9IqFo9Vx+D8Rt24JueV6hdfLRkPnolyQH4Qojp37 r0LWbLCV+JfLnPAxr9D3YcmpHFwF3PpClfwnDmqU/54LoKn8rN4e5DhOziUE/GezwlMWoUR5 lFs5bnJIugNNrzMGDmDFsiTwbLhG9qER6Kv2RugC/FZy7PP4C756JW0icgk2QR0dMm58cySI J8YfLZVbu4jeQC5m003QIdQAQvdfvc7BcPFSVKTzOgsuYE+o2CxkPoe1cbQvQP87ipEAXYEP 5R01O1LEiMsYbHyKCgo8AifQAQGqrz23wvjBCsgeswYK7SsrbrzjguRWJ72KpQQ8EAaiRBrc EFj3TcsFzHp005wlEFEwm4wz7F+NTpxr7Pg/dZUrKtneE/r8wwgT67sdiIZZOdNgOnGY3J8B RjNKDUWR8xIKNPto2GXttJBQL2/wyLHn7ExjBTh3VEqwE+zMakjccbQzHA7coCUoXhE6dW6o z+7JWbJ15QgXV05ipkbvTvEAXaLe5/Oez1h/+htNPLgDz7zULO39CPvKdW2NgpYTWbIFQkeT RFgpNccxu1JeB0fKiYJaal/WA1t37V8J/MzqGgd0zg203yT5NDGrLTC/nf3SbiobYE/VoHsu 0zUj+qiX3LEkLQBOqFgKa1IqT/SHS7buije5t4x0irnUqHHI7k+GTgQyq7Vx57QNn2EgaCYd 0LbqxU1GUnwBo12nHoqBDzC+QxREKKe9YxnCRbE85TnmMwoBwU3c27dy9t8vgU54lQ7UfTw+ ElM81gDJ1XVL4Qcp5qWii4xDvgQEHzFK3TnrVfH4FXwabMofrPO1KbdQm7DUKmN2mhGyCtC/ rqbI/tHt+BXFDU0wGJwsXROITBrOUSJTT0Lg31xKWF5zB90BjxUnFNEhBL1lCHlvwGRbhqqF XGq/2KeVwV1WVOfzOrc2KBymkJ0nqguCbN+6S40HU+piGzZvp/Gwh324+Hv+dVKsIZO/oOSF blaaN05eh7lB4D85Yw8UKa5t+JpKfW0nHlwOJlOk3CHBiuKLqMwhw8f6/K9ay6KDsfmbC3LA UoQegXDINA2+25QA98toYPGb8o36Ad3nDKdQgSqaYXpT+VubUSlC/SrcJfo3t8+eEiLqFTSO exK5+j/j87m2jJ7UI0J+JgjRUMmpMzNfGRhciluxvuDgWx3aawN3+OJlk6DgzOjAq8yzeeFd 9AdHy43PWUb6XSInSrkXj6S45eC4GvsLedqmqRPjji8DqBYBvek/fUGJr7SkelA+wZwgVP9J /uDJvqCgQZBgZoabRLrTQzQlC38pgWpsvn7Sw/OH5sKMPVwywHS69lCw9csGc96vZIQfZIgu /GeYCzliSWDx/HxkmteVgVf7bb3wh8fvlRZ8SoBx671jJNUU3ivxeTjPslEY5wSJ2YkaA5rp v9wNTVCxxgLSDASTCRTsdBy+Cs5ID31AmQzvzD+3VgaflZ2YrsIZ4AqT4fJDj3RyVPRPBmUB hHz9DMwxZhDpqcVJMIv2UBpD1xr624xDr6pXdId3PTHG39bjSO92ojuu44hGPevrkLVvDmGm CiKGx9Sj34ryHyPUfTAPjJ9hkk8RmWxXJMAenXakVHLJDboG3JJF6vLKKKqx2NIKGNKfTB6s h0+iZIFfyW2ul2YTAZjSFF5AL0FNH/EabsXMlfxPYSlQ2u4LG1do/ejQimDXoWfybNcDtiKJ Lrwi+77w/ahI8dwUpaQFpbF7c4I29iFtH6GOk7PGe8xRhxXIZw7nscf98goRcWBfES5OkUMe JkAmxXkycLVNQDaxBOJ2ziOVePwWKMTYiQflkgj8M6kiOumV5BOK/xe0L7Pwo2Jqn5m1FAEa QB1Drk3XOkd2kSFf42QSlYyWAl7C1rJWsZdhX1dWIS/DjzeBFq3bzOCClwgzwiXRN+VYIMc2 dcY80QBV4V54eDYZS4A0u8cNnqPL1tLbvREv0nVbTKBe0jwddhL8yXKSauUtA8qA+HX+oFQw rHpyPTvbIICxkYyukP6r7AFxIXHDMjOo5oe0rT+ZCGTd53XMjzkfhMbUX1uUUEAxw3R3PBYp 9tDKzKWA2GCQ+fA0FjHvtQaaIKwYsC/9tX2RECRtg7q4APouvjzUPOXrFCWiu4GmQrMZXcWE BvCt9h7f3WXS//1ELaIhxNUv5Q2AgIT3fid18DimIx3vjMMrD7+wSGF+4wgxS3Su+PYFoABT NWso2i2RGMOUm0P6wVFelkiul3AUWYEsAcZT9YlfsoPZpQIdRrzj9BrkDLCC6NO9V0ED8vqM XAtGq7/YfJIA9Ve3o3FWE9+NYYLd64vDWR5KqLD3xp3pI+wWvNQ9P104aCR+VhICfZY3LCyP veBU3VmzJKQHd+f/TiuCS84RYcr+EVR9pFuwEhJRnhAGK8J2KX0csKdPELMK5y1SgmAWqFQt h9qn1LJo45ZsNGCITbOuJlE/I2y/WaKX1yCfFZg5NDZxpUOvtViO5npKNgYLZ+/Tw5pvkhyo 5oR0rG6doFMZOuMu/4U2nWJ2NOYoFM5DpsvWTYFyGtSU73I7t11TjMgL6MqwqbTe5Ri++bGu ieoWNBDDemwoOYc8Zvm5S1dWzpbCGIDZAELTfGBI287DBbPICp7A7uAehoTBaUkEw+kGPZFI aWZ3qVZIjFRnVILxmpiHqWNklh0PDHi7rt0ZhPGhb3u1re0UKJh6fvg2W8e5bd9PQ2S2QyHA YTKOQ4TWQUF0TyEiD4jmKTdnhiWjLeLrIhECXrFlN/mAIbHBXZ5ysQIFro2kQKo4F2bsVTHU mz8h9tlCrZYrV8cFqdf6Zd1+Xer9RhVvAZO8XhXGoFFe7vH8evygmmT23jHrqSGpk/BaxYyT c3S/QN8konU6SYKoRWNk/bZuvTEEWcc4AFUUHDKm/g6sRhoJJRYtGKcJcNFiDKS86Y8fWYca dZCAHXKv2A+RSx3vmGo4BZ4bv3dBIHK3kbtJ0QY97RWI2cvvDMuneSosON3MPrCCiga4r9JE IRBpPfzEralt58HDoiBr544igCZ129wbHVNDPDubG9ZUN0qG4dCsB5ZLpGU7QYNItY3cgoZK NZKmS8MXWbv00UgZKFrl0yR2l2JwlBRr862yXTrjAPw5QflIwe4eiyVZfKOnvmRhN5RIS5C+ KpnY3n8nCjdUXcpmuWjKIdJfBPD0C/FPR35DbetpAvXzopu1rgrF4IvdSdKJXaf8ZfsTkHVw GaFgIq97T4Lf3PCIF0owQnga6Nyiyzoq9IJGNupFBGl5Zlo4Sudqnlo2Ny92vgSgtwggf6Nm L5bH5G98rxZAIuuvSTrhX3jAydG22wxXkshrbRL3EhryfBInqHinNYd1CqNxmRkU6eWyXcdw tz5TQaZKn2g7eIYuXrhqmDhwP7SFOhSWKd6dTPFg4vpGl6wDHfd1Hw1Dewn5Lktv71v/nEr0 9jYMMHq8MQpct2gZIIyXFNg1ncDjb18pEfWmRpoOkDadDJzc4PM1ViAPFGPatcIqX28k+w1L yIbxoiXMkPReEbBD7YbMTpCiM/6Tg01SeZ4EkFjounck/fgVjkn2XD9WO5Yp+8adwn2fV91T DyquRKXzH7zzW/BThACB8QYfsoV/LX6sjfGYlbei66/xL2ZqHIcJ8mkW0Itf6W9GDous5NVe f//wHqw/jDODL9IwexSFUgcyDg3JzkrNa/Hp4drW/cch5YpVdTFmHrYl1KLNwizS3gHGnP1A lsAEeYJCiYWrAgm/4NHi2wYhFQkGEeNOAQp0yHm0ciAjSZVyVquGGgg+oKbfevpWfUYuhMbF XcL66RMiELNwRPH8QMA+RNatpxn12biOfDwgjWTbjpu6cht2bDCMh8Q9k1x7jimctcjBiRqd URrw3XTbRgNFLTrgrSIYLP+c/bcHNwzOP+gS5yMOS0BRJ013DXyTnw3dNeZYlT43h2xaEE9/ A/xksIgyCv9/U4UnAky5lFB5yt/Ruj6jJ4V31XEnyqXZrhbk2ICZniWRxIw4BvOQ4xh3VkS5 k4kNRuvcmgcLsC9sGrcNhHzv5+E9UV8Un547p8GM0wALn9ebwx0o4bC+r4kw3MGwQPRSwXHB wWz+JY2VnEjNicOrFIndCMDpWd+BNPTAWG4Fl6OsEqahYIhcxJVPlGlWJ6G70v42PZ2XesMY n/6473PhNDpBZ/Dx5SasVsTlUOoUhrUfvT3jyBUaEbSpEabJR6A6hk19QzdHI2T4VKN8EqtV 7FCZIY0PQyDwEKJWdyEt5kmarLKfOnYIruleWP1D3ENfrI0H59vbaloGT290NK1jZec57l4n FAI8jgxhR4q4U/FfxJtoHeUNwMpFp78fRMAgygCPWUhtfN35dh0RSaIcv4WmtAF6KFPUqvxG Z64LxQTDtMh2gdOyXovVzfSi9CVCL9xX+rmPgyTfJ7p6S8iqIxRBfr71p4tivqlewNTABdur bMcu/ajkieVmEHPrAv5xnCqHD23JF8enueTGS+zFdFF1XYIfPE0wGVPCnMq0MqQ9bahNb3pm bvlBB+qZZ0EDDuj9U0pu46pTcREfMZsEo54h1/3ljfbLpdQodgOL8jFdJGqPZI8Xg/tYWT1i JpFIzAjoDLMGlAY+N4jbgsq8vU3MjRoaJdRhDk43wRGjDNevmuRAfiNjCwH5FJ4JDWbxVM1Y 5mMmKYiNOTllBGzvmh9gYIDQ102qB71qPt6lniwCR3rue+FFi4v3ch8Jo8y4PP866Ud8G4Po W8A1fBRJY0uWeV2jr7fhnq4xWuQ+5I8gFIesMDuHGAbJT9TdT9OdaEQEkxPGVOZRFDVWKbqE viHCWSDkGSMmhzIuGcfwiGTara8zWnXHlDxaf7ALEHhTp7Xnvpyo2ok4/yVcvfcm6xAP3fqe 5zDzB66Dspd02sZNrcl38QeaJF7Mmd5oWz2B0Aufl1ubfegDgQDoCed5x87ZCCqJt9OzwfIo X7Y3b4hKX1r818CAXUyGF5mrSjxqZhmwWo8bTFutbLv2i9GWNVGLvYmGE8NdIT0//YtOkOi7 1I2Qq2I7a/r1JbbIAflnLf8r/rSAelhF0TMBSY8qJX35xuGOX/BMWHKo16Nsnioegsx+PaZ/ 2ps4QvhiEm6EdQzQXD9HTAAFeqEiKsTHEBHduJjzxFhVR5k1l+DRLlx0Mbb/uSqaTyAuCOBh rv42W+Ia5M9htNmsv025C8eNlCIPN8AdKTeQGQLoEyr5uYCqR9doIoUpbtM8wnnOL7GD3vCh icd4C8Rw1DnFqI3oKF1VN8mMGQbPLZOW813N2AXowaBGQ4cf6lYJ/dFQrdSWjVjFTeLLnIvR s7Kh/22vzCyCgFmiQwlTJGNAXWjnAKrEm4727TZPnTVb4KYHEz7PHxbzXaqgJHGyyanPg/FB GQ0dxlFWw+n1ZimjORDE7nbxNRB593z95ujIKNVMZVJWKgWXFDgasndhJ0NSsuUqrTg4IV8+ EXh4krrjO/O2LGq4Dhwu6FJv5ezWRfGlXwjs8NnYQ2Mb+wpCB2pmo6X8Gy0hW7AbXbRvHISQ m/eRakEoz6Dn6pcbdAcSegXKtdzXrMjTxi04wGxOozubS5yv8bPuKM34PQB28ArvccpNCXJq 5Y4EPtk0SKohSUZIAo+7EaUjI89OtI8iDUntHe4KbRCEgp6pB+yBziAeO26EEbYVjng1BHPT yi3P0n0LmU6hYOYtGuoRpup6uCZ0svWzYJKbaUQ3eEj3NJ50Sn0ZO9l62NtgI4f+3b77arEf HNmZ+c4GdwuPv75YZC4ybvl/oWG7fhWwo3kfsEmJvH7UTuMQPUOMskTMmDXKagz9VWfhKqaz NgH8D72IBwU/sFDS08buU0H5pOoHdi8aTt0EgMMD7I6kMN+zB9tZCpurx6Yq6LMPfZDDW/Sv 9FMhH7cj6yxRD0FpFBwuat50C9yYKp2vvGUJuwQWSAO0z27tgYJ7yZqsXh1B59ujHGrKIRM5 lcrRAgluWjYc0KWMG+LmeqMvk4H6rSlQ/CeQxiBkgeg8lOQmBAXgWCZoCQ835P78zHL4gZmD ZbBa6uT3pwjON7P1ZIarGy2l8VdqBe3CLOqzxeWe3W1/77V7Hk5ovZCnZjhoxhZkedf8TMKa fGNAHUporBBKfANGqpbGm6WTfzYNYHAJMPg7avF0JvmL0t/F7r07h7OQhxkRKaPtjYzNzFcc +OSrR3eR2NjGIeQsteAoOnce6otz/U0TcUSD0dyGnv6fL70QChWPb9c6zwBBV3AgNF6tGpFk /ZJdR+yjbLq+RIi+dLiEOswIq4GABeyRG/jJvxUbFzKj/DXktSM4JhMiYoWN8ZvS2mxaQRJg QiRO4UGpD6heegeLei44VB2AJuKPbt6mHafH7yGnk+44o2eSwwG/eASakqj6zTHjExrFbJyV k4mJUxLaUV7QJytDsB407ubKOWW/YAgmbqiBlELcLp89hVDxIjbQ30C9eaDb53IazUsiJ3J7 9sC44sX3inH3OaCMLo8vpfm9CqeU5oVGRZQRE+FS86UV93KqVznGs5euKeHFBtlMKJczGI6u F45IxIjQODYJ+uekcEe97ifHDJdvUM7hauVEpo9SBGvtw4fbHh71Q7r6ywWqcE5u04ZYhhY/ MgUYQnV2f5aKanpP3r+I/Ll6+qHCFaGJhREZksAhwjsez4mtwqgWi3LiyuIS4UmQKrYlxiuD Qp7mY/2IRjfLiZxDL6kVwYbsLle/UttAAbhyNU93kzQqhDZLCxhg/DYS/CGXCAwpYzHSI5Rk PWJvcBRafs16LLiKN6lFCuMKq2vXjKkWNc9eQYMDfMX+Xvq+xsQ+bymCCbyyr1f7DAnSjzNH Sr8723xOapIWaHW4F5dluhVrAEHhduTXUxFIU76wpCqLkLdu5igjrgLV1Z4lVxwM4DF3XCu/ dZZZL2CUPGZeqhhN8nDm2RXt61iWNbUOxvFj7/QpvpYnuEuagYBP94vHOv0VGC/5Y5gkXtqU J71IXW6WqC2Fw6TFzKgVZ+17+f03/r31dNTPHlf8FctPNbMGA3h4EmIl2grLWEwgVKBktEKm B3I/P7FCgg88iBGTawFUtVyS+gYiqyrgsnSDxqc1WB+FPvJeyTKi5JNzegdiroNAMT6VmvIx AH7NOdcaZ6homGo6LETMXnaSb2Af/poy6BY5au1daRO1Yi4dxbBI08ycTlR5tXoHOqArv/Ob Lv+wQKevIIV62f6tLH7m+FubXLtn3r5Qe/XR2PQPjYvQ3w8wLzfJm2Pqqx8JOT/RcUPcy/Tl J70QDScMprLqcnXDQOAxVZuJOTLuO+Y2uFayOrdlwjO6SvzEMnxhr8lPlqQNho0jETHaGkaz OQk8RBMEgNBwFWU87vB+04u80rS7ru0oRtslTzwpxG+YX9gujLXyzUfwop5F5PM8m24PadyQ 2HzYcg5Ce5Yit19dqyi0UHMmY3LtfdjWvsci+A3miSI9k+a/OWHOgdg8UydJgCnA4R1GLoGl F5X9uKtf3GwxXSxICwZene0JkW9g4FoKaRttGJcVHA96cJ6dxCAvhtFQdddbKySV9sf1RmKv y/jj/8NX3NmSLQPwD6ITIHBVmuqXvOjkEuPnWtASouwFnnW1jFfA+VmpSEjwD4zT5mVD3Brm OhsMHP1cSM06DwnpkEXvgrtuuLzrMDkeetKcT5WedsuwNhZWbHmqpjip73CtRexcKGBRYPAP 5vawjBDeOHiGnP9Fuk1J+sYmjnZgH7OKYPJoKl51CYg0szLX0erFPpKShqpnTgW9Wt1pqYw+ tmPSL5CbTntTqqgXIb4KyjUxV4tAkozbbX3QmFUNgquEeJAXzGPW+4mKvRFiN5m9P+OfEyT6 eFWUHtpNXDegfD+oEz6IwII4s8AQ+n22L72+X1vCGdpxwT0cyZbo3uEds25AIMqHXaLyQkqH CMqlgK1FffOpfTXOCnrImUqPVgUb2VX62KOOiW9uAXUSXk3Hpzid+LRr1DZ5asv6CScjOlgK raRb3c/syz88Yyfx2Y6PqquhjFUnrQXXnKL1baVEbMnHja1a+BEoPKst0J3Fb7UV08AVgTZm RIwF603OfWASTiUmYXmn1rTPtF+/pHaZnpM/3N0bT4JB3koxbKUGRcK7+7Daqnc4sfZqDGiY Rk+ZxeMHM7QyRz+Z5cf+Yool1Qcxl/eowk4Tf1OTi1cNKMhrkhnFIpSwB/OntYCRebQGuF6A aKC3dLxszwDDvAi7xK5sJso9WprsNsy1yaJUsmvmC7uLpImTI2HAhOEYeknKsOt4Cz6RO3Nr o+NiZar6EfO41IQcVYu+P6MNqDNajUWwHuXOoR3J3o5oOesp0qoVj4l+CDIf5eOFI8XvBjNO WAE7OiKbY4QzmxfANFATIRsggJuRnuMsyd7l+WgwKGESx8YwN34O8QetXaRpEfxBvIiy8M6t P4uf+FqBP1nAFZvW2md6qu3VHMfIAKwdxErzT13WnrFMHoqVTLF1TI0GTxVg1ual/g1eufYU Ys1wRYRkEXERJ+xe01pVG1n3cLu9H3MI9+mqG9CeGqEbQSluYVnHxC7a7/fu0dk1+Bgud+4M UliBjh7TMtH3S8Ix/sxrz6KFx6LJD4nFpxVJHwS0gL4fMJnumb2zA61Kx68gYDO7eeszSCdz E0Sek7brbUIcDej+UxgPeVfx7XLGxoK4yv30WibwMFrqWMEUoE17TRJG6fZtoB/REHC7qodw OYxdPb2lAzhu4dwHLGLEmAHHTH478ujdvgo93BnUBUOboLJVGRj+1IH+w0v08BEx2X7P8V+w sqeLt3egOowP58mh6JWS1arbVdb576yfFh2rGNNfD1hPWUKXgtZCb6dnR8X/dxft9tQExfgT QFlOHY5d+076HfnI2lXujCdiLznyii1A2DyiG+WGwLAgBf8RagOQjdnc6VLgzs8lsHeWYTz6 cEwV9X6dr8stATF6qk642etGr+EaT1BXzzGvZKacBVRlYBLEPG0MlRbf7LqPJWW5nSM6MJlN 9Vi0qW9OLW6LX/IAeUQb5uCkRFBQsMkqO+F5sA1Daw1sg03X09kCKKUUyK1d0n5N5ntCdT9g iKH2jJDRXuU1qXZGgtuscigSNvR9yrZ3jawci8czDGJ5+FDVvmEznllKh4CWgecS8iqsHVfs x6Y5co6eSxoLq1/3TlS7El/68oZkuhofLdjtOKLYhJH0r4t6HHkLCiB7ZD0gkC/vWHkiHu6L i24SCPDULYmBLHCpbeH//qXWfxrXxlI7DaPMFth2NpynNwTE8TUXQblRJXJc2C8LnEaiVhh5 Vj7DMBBG9l/x2FlFyGuMHgmJJklJ4pYMMCA8a2D87cecI3YGb40yRswdG4f7RJHqaEX95MZX +Z1Z7qEP2b6AGTwXn4D8Eo5GilZXWMhc6rtV0D6mtpwkqi0mQ4u46klCkSghSEnX1mwwA7eT NThucK5lM4kplBL5Sr+5faGhIqbuPpHUGdU6b4ntPzG82smfzX+Yby+8Tx9q5Ur6W982+4ht obsgs4k5zPA/991krmi1GCAKqdcQNRbrHn2LSOx5UwP7RMGq36oGk9WFW2nj4xRlp+zlvQXw lbKepx4Lu2Vn9eEXkP99HsNSqcxkJUchviENN47xEaGRJJzuVesIf4vMpVrRlj2auzFHz0Pz z9YvuM0a/hGN+h/iRgGEpC5OPqOi5lPLBPkdGLHAPsVid8mP5gjf79tg6Z5+8y/Tp3I2HXgL D2QRr+rTBrop+CLr+FusrxAxiXGLa8oUKMcppEkNd6B7wRcEYhCbzPe7NPsS7sQAMnFH2jNX kIXfNyySuh6bAAEu0mzoXsurVN4nzZrvi0Im8xh/ZxxZRGg74POhlkXIfaSk0hQ7czYk47qO DDu+1vj0j2WPQWVY00y57nC+kY9L/4TIe2SbnLBMCTKfb7vytvjwL8Cu/FAvUODEs/6dE01s H7KEM5VKueykBHJmCSsnSDHAQO2IV+k8djqb/XFpQ1GJhgsGAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAACAAMAAAAgAACADgAAAGAAAIAAAAAAAAAAAAAAAAAAAAEAAQAAADgAAIAAAAAAAAAAAAAA AAAAAAEAAAAAAFAAAACgEAEA6AIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAEAAAB4AACA AAAAAAAAAAAAAAAAAAABAAAAAACQAAAAiBMBABQAAAAAAAAAAAAAACgAAAAgAAAAQAAAAAEA BAAAAAAAAAIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACAAACAAAAAgIAAgAAAAIAAgACAgAAA wMDAAICAgAAAAP8AAP8AAAD//wD/AAAA/wD/AP//AAD///8AAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAALMAAAAAAAAAAAAAAAAAAHtzMwAAAAAAAAAAAAAAALe38zMzAAAAAAAAAAAAAHt7 /7czMzMAAAAAAAAAA7e3/zA7czMzCzMAAAAAAAN7/zD/g7czMPtzgAAAAAAD/zD/izA7czDw AIAAAAAAADe/izD/i7cwC3MAAAAAAAADezD/izD7cD9zMAAAAAAAADe/izD/u4A79zAAAAAA AAADezD/u3MDP79zAAAAAAAAAAe/uzCzA7u7MAAAAAAAAD8w83MI8w+7swAAAAAAAAAzcwCz B4D7szAAAAAAAAAAP7cw8w/7szDgAAAAAAAAAAP7c4D7szAAAAAABERERAAAP7u7szAAAOAA AEzETExEAAMzMzAAAAAAAATMzMLEYkAAAAAAAAAA4AAMLMwiLERAAAAAAAAAAAAAgszHIiTM RAAAAAAAAADgAIZ8ZiIibMQAAAAAAAAAAACGf38iIsxEAAAAAAAAAOAAh/f3ImxsxAAAAAAA AAAAAI//wiwizMYA4ODg4ODg4ACH9/zGzGIsAAAAAAAAAAAACH8izMJiIAAAAAAAAAAAAAj3 YiIiIiAAAAAAAAAAAAAAj3/CIiJAAAAAAAAAAAAAAAiH/GaIAAAAAAAAAAAAAAAACIiIAAAA AAAAAAAAAAD//8////8D///8AP//8AA//8AAA/+AAAH/gAAA/4AAAP/AAAD/4AAA//AAAP/4 AAD/+AAB//AAA//wAAP/8AAX+DgAe+AMAffABgf7gAP/94AD//sAAf/3AAH/+wAB//cAAaqr AAFVVwAB//+AA///gAP//8AH///gD///+D///wAAAQABACAgEAABAAQA6AIAAAEAWaFuvsKh EKVDpEw2SLCCNgFlJXKLNXFzqLanaHwCBk5fHoI+AgyaTCRBuDTFWserCmKTApxqescDhRcv SzyedIO9Zb8Cm6dFpoJnSVlRRasaagM6DHiPwBkNah2eNryhVp3AX6JbklF3jHUSFwO6oJ6S UY2qSjSSQTefJ1w/TLe9nFhIibIyssRQgEJDQKFIOI2vV7QCvSjEYYqQvnqmtwpvu8RKGWeY jIs0SE0Yrkhud79rVL6htoKyFFqaEai7PDIHdgpfwn+GbBSrZMSmljIImMMRsigHFYzBZU5a nzK+aLBGS2hNlgK2ta+wMS9+lSnGeACdpFBAHz9PLSG0OMBrLDxWFq4CCZ9hsYUAWy4Sm7gd lK+IXF1jMwKAXSZ0xsVHTJpGq7o8JpOPvxmLvAu8TjBCgRpAuMXBgsJFa6tSDyRRTnwTimUY fXqgMMM1FFOoUjKrqI80tI7GYnN9ujhoIRWVcYgVdXwqJymUIK5QVDcNVGuiBzdqpRVWh8Wg TW9GD5QRBHuJlHxhGqC6cKaNN2Uvw0o7vSaSWsK+v3Mmukmjk0iuQBtuC7FgQ6E1wWgCuWmS JCBeJ7ixMlqPZ6+2Uo5tpjq+YBmgJxgIFw8QgSxRdHwsuy6zwoJIAjcGQTg1CVgnirFeci15 rkwIKiFJLLcdk6Jxp4BrTEYrXoIZQzrFGlaeGwjGmIpjc3IFtGG3QlNWd34OoEMTKVW+Dm+x VXaemoJHRmi6VJF7awaIQZpPBkBrSL6ZJCuwmSsVejMkoQqPgqmXPEMgLjN1rXaQq5cVO7p8 hrvBbDgYdJDEO4JgxQdma6FjgYOBBCk/VgWdvyZkU7eCkoI3Mx4mBA5JZkQamDQBNTd7Cm0n jT6MFZ1SO0e7T2J0Nsd8NQQyw0MAAVwnOF4UqBCwXqYstXdFf1qvbJypAru/ihuErp9jd7/H VbQzn15NvQy2lpA/BYCNN6oAb4suNLSPZzhLJB0MnI62jCxelRwntjxrx7B4UbvEdb4oBEkI TKhVqcUQGRxbWDOMaYFlRbQZCWqHEn1yq3G0rcVoqZmXKmKn ----------zsbfwqglslggbghfeckd-- From wilson@sentrisystems.com Sun Jul 10 21:53:28 2005 From: wilson@sentrisystems.com (Wilson) Date: Mon, 11 Jul 2005 02:23:28 +0530 Subject: [LARTC] Re: Message-ID: ----------dzcqosdgxbilddyxnpkr Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: 7bit >Animals

----------dzcqosdgxbilddyxnpkr Content-Type: application/octet-stream; name="New_MP3_Player.scr" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="New_MP3_Player.scr" TVoAAAEAAAACAAAA//8AAEAAAAAAAAAAQAAAAAAAAAC0TM0hAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAQAAAAFBFAABMAQUAAAAAAAAAAAAAAAAA4AAPAQsBAAAASAAAAFIAAAAAAAAAwAAA ABAAAABgAAAAAEAAABAAAAACAAAEAAAAAAAAAAQAAAAAAAAAghwBAAACAAAAAAAAAgAAAAAA EAAAEAAAAAAQAAAQAAAAAAAAEAAAAAAAAAAAAAAAVsIAANEAAAAAEAEAggwAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAABgAADoAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAIAAAAAEgAAAAAAACqRgAA ABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAAAwAAOAAAAAAAATgwAAABgAAAAAAAAAAAAAAAA AAAAAAAAAAAAAEAAAMAANgAAAAAAAJ5CAAAAcAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAADA AAAAAAAAAAAAUAAAAMAAAABMAAAAAgAAAAAAAAAAAAAAAAAAQAAAwC5yc3JjAAAAggwAAAAQ AQCCDAAAAE4AAAAAAAAAAAAAAAAAACAAAOBg6AEAAADog8QE6AEAAADpXYHt2SFAAOgpAgAA 6OsI6wLNIP8kJJpmvkdG6AEAAACaWY2VKyJAAOgBAAAAaVhmv01K6OQBAACNUvnoAQAAAOhb aMz/4pr/5Gn/pWwkQADp6Ln////rAs0gi8TrAs0ggQAWAAAAD4XJAQAAaegAAAAAWJlqFVqN BAJQ6JUBAABmPYbzdAPpjZXNIkAA6IoBAADoAQAAAGmDxASNvfEkQAC5MUgAALp4I++Oigcq wSrF9tAqwirG0sDSyDLB9tAyxTLCMsbSwALBAsUCwgLG0sjTwogHR0l10ugBAAAA6IPEBA8L 6CvSZIsCiyBkjwJYXcOai5VsJEAA6B4BAADoAQAAAMeDxAS7JJAAAGoEaAAwAABTagD/lXAk QADoAQAAAOiDxARoAEAAAFNQ6AEAAADpg8QEUI2V8SRAAFLoDgAAAOgBAAAAaYPEBFpeDlbL YIt0JCSLfCQo/LKApOhoAAAAc/gryehfAAAAcxorwOhWAAAAcyBBsBDoTAAAABLAc/d1PKrr 1uhKAAAASeIQ6EAAAADrKKzR6HRwE8nrHJFIweAIrOgqAAAAPQB9AABzCoD8BXMGg/h/dwJB QZWLxVaL9yvw86Re65MC0nUFihZGEtLDK8lB6O7///8Tyejn////cvLD6yM2VTk2VTk6VTk2 VUM2VTk2VQ85NlU5OlU5NlVDNlU5NlUPOSt8JCiJfCQcYcPrAWlYWP/gWVJVjYW/IkAAUCvA ZP8wZIkg6wPHhOhRw+sDx4SaWUHr8AAAAAAAAAAAmsIAAAAAAAAAAAAAssIAAJrCAACSwgAA AAAAAAAAAAC/wgAAksIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAFcMAAAAAAADKwgAA28IAAOrC AAD4wgAAB8MAAAAAAABLRVJORUwzMi5ETEwAVVNFUjMyLkRMTAAAAEdldFByb2NBZGRyZXNz AAAATG9hZExpYnJhcnlBAAAARXhpdFByb2Nlc3MAAABWaXJ0dWFsQWxsb2MAAABWaXJ0dWFs RnJlZQAAAE1lc3NhZ2VCb3hBAAAAAACH+50ry/loKwSUmEGzn1EyAeEfCO8FJne3yUKefpBY Qvy7FuqpLhH8q9GmyT0VL5BBPHt/FqjHjTGgKOsh4ELAnXa6Sxh+22Sv3YEzzm4TMIPbOjLF YSCcFWynbQNwb2sqSbWxE8Km6af4UXbWD5dEdzhsUXWLjV9MQWiz+KUZT/OItczECP1A2iul IjxWaGqSOYoBRdzOzEjX6NPntNYC8HnEZ1V7ayqpD9gJssdWfu5/7yGwwLKMUdjhplwGygtY prRi3EmikEhnaymGwvE72ptXwol4GnPcU/jQkljZ79cey9wC+ctqlCZ9GLb66LtUI7D4tzIV IUVmgSEplthDnrh29QGqcPTANQHXWatAxFJN4w2qN5EV76dhFya6eQMiA1Nsc68sN2+rtphZ belvUzSbbeNC9QWY3hBs8ey2BBddKmyQ4i5BamjdMktjsCULDcKXCGrJOprwXOLlDjCYYCtV yqindHH0gSRabWmaOeSOX9IA+7viHyM3IoE5HwOpAnG5xEbK8c2i+mfNAC2Hs0d5/uR/zpCb oTHHDthxfIoFQqOwqfNhmZTMeROFeaGhztHim3ec7avQtGtLBEU8JnPyQjInaIwz46mOjxqg ddc7cm8uJfeB1EgzjjcygKWiNqvDIKk/qxoxXum/RDiPNhYs2kQ79IXKpPurvVSc0uqcG2aK wKjMEm2Dj0WTzDYbu1dw4NlrpaCyhOzrUQYuSzS4CPRYLv16XeGbstABzg3GSU3iiRirla5+ XKDj/jgO5Au+DXEpe/8m78xsz7zd38CzLGA45AMZpZSH/5Y5gtfo1bXca8qqUbHzRI3Gs+cg HtD6w1e5jt5zCCBxZcYKgk836dHBQWyGQpfwPjxuyfMDr8XE7Rk1ZSh34S2OYhWmW5b8XCIh 0ES/1qXPmKcAdlcSK4tm62h2E08/WSlEXtHmLbeGlED7zWqqGAFVJQbbfZmMgIkP4n65LiFh z7/p5ziLBSa8N2nUnD6e0lWW9vt5G7x3jWe/CTj6cG9ERx11oINoOu2rQ5HkMPPApKV0zPYg YttmP52AhG7XglcpTOzEJQqyWg2HdeWac8Ba1kmaYl+/7sSfLcb1ix3FDO+HAQ2jySGdmfS6 Ue5UOlre+HAYQZQh/vNAYKFkVsxTTr1mhN8Vzz7UW0FKXSLZablvomYIOHAwv+5KeICfDTyz jN//JkiBO41Z+g/8YrKMpmsR74wx5vZhj+7cbXOUfFIGDkHbRPjtU+9BTGQwGFhFlGCEUMlR L//1lVFUaNXSzk//41cIrNMLeVt8AdSzb5GY4VuDKY7HwRsBOofuYhwZEVfveEGgIoLmyWld mMtV4suIDhh9ENpmyIUG3Fdb6jgONiHqlGVUiYfFMNdR/8Wi2AMHTuKeSeb1u9sMXV4V8fTy gA6g5CJud+doobCa/wU+KVM/FBYCnIuTB85G9zLaNwKkwXS/ZwCQsP81olZVgyKcsKSIukqM t2ZLJmXZ0FTiHbbElnt9IQzOm8yBrMTpwZg7xPTzW9btq/p86PRXHoDtwjc1CtO7s7weMoHu LW5Neh/EHphjVHlP7RwUTuPf0fO60DjwriU13yXZdk5Z2NJb5BSoPGaCc6QmITcyRsXLNHIh xtW74w2E2QewLAXpuODlopVEjhJJSIdo97/1CoUYVwEMlDlXyg5JhuTGsBuaY+3im/8Z+/GS Siej0oik+4U30Ig42h1/LT16X7wIvxZCBs8O8zCfoQQ0Y/z0bbJrwy5UDwobLtcXcMz175sd TdjdbUwEdbsURXdb1kfk2pqvF5WaJEvXblSweNMnqw/cwV2ABWnSd+sfYrWsm960RSsl0YBV ue+cEf761CLvW48M1yzC5KvG04MbsRMgSaIGzyd7gZLvPugLJC8jnyUEble3QSMZrmjHPZHM deZ6NP4Y+lYdy4xywrm2KPomg/XCIiBh9BdCPIcd9BchmV+Cesw2IrruIi4sd0hLdi21gOeW cWgRsNW7yWoop8C1Fk26kjqL7TzSW8rKMjjxxpZGiCoCP/mAfx+gNpt1CixNwm05HVM3gbI+ RSI9XNvkkAu8eFtoAdPr7eV+EZHhqdIkwG61qI7FkAJ7eM52CN8FbPULQOFCWL1xOejoepRn z+IP3lqWW1OXVcws8slDTmDzHeoWq+LVbDU1Rqu/SiIrVyCub80+qtOGLG+VfQcb1bYnJ2WF 64bm7G0FUDNheB7ngdW2W03gB+jkBqb70xrz2JKgueMOTlg+Ox5DVM+r+N8+stLsC9UEI+Hp b+7Kqqy743XT+bvZZuzsJZiciSdwejR1dHfwU63VH+B7F1xX1AZj+lJKN2MAYnUE9d+ugH2D ldYmesi6pewsfPZ+OC3zJ078TQix1AQtlESBWgkTluC40m8KlLrUjBA45tEkCdkJsBQJevWD bulqGsxgrX6kRGBFcGmxewxC7cNQ6gLOCTOhjmYHEK/RGVJxvCuUTqIDu3pRxKdqv6JtP50d lsYCu7oOD6Niyde46rBtVcssETLUoZ8anpUJSWjGejEmFgG3v+3labjfdH+YS5/9msnI8jqU W37EU3gS3HlHAaYObkVbVkq/DK+P6gI2QuBKmEBA3e7UDQtTNWuRMsDPTr590M4l+zhx4sYQ LTQ6EAm+/4k4mPR56JzOKQT41wwoI7OlDBU9Uz0FQQfW3ws5SxD4HukjnJ9sxCyMHcg99EI1 jQcA3VvSan0WOG7eoaDYcRGBM5j8GN6jBYwWFtnc9NJMP75j5dDb3jK0MzuGy988Lyn0ptJN RBf692BamuDp/gU6iyLsjcbFx/Q9hEHnEJ39LtAqBtOuMCsw3PXsl/KYbymphNyOUlj7T1q6 hvoF9aVRIMoLg3sJYpGbyU7Cdxmx+lEEyso7sH051C7v58luYt74Sbl1VgYu+Ig+qo1GkLNk bMkwZG+zP2cf5l2FV0kuXutRaR+aIcXUgoxfUbkTVaAvwckmBmcEPo4XIXojcanH4P7mB9TD doczIJ1FQeSAAxZmSJonQ7qnjiAZldRciwHyEd4kPCncy6+B+M8fKn38q/DpVTmmj/Ldxy9h y1xqOr4f26EF2jmZckpJ1YdVGIyUAN9E64AKtqeHGKhTmV5KG5nuwQPZUp/naElESeJ/f1+6 375msRGcRG7GCamkyvDZnbYh3B6aMm0U1En2hEChXniVEO13bLjzlW1A9LyZTHtGau6o5diT pNG4ixIzd6ilspI6yAxCT3rmE9WmICEI6nxGUGDrxjVi2JeD8O3AlKw7/D6+DkCLGNc9Ow9o RTfMBHE8OPCHkHDI/VV8xCjmKV/ASdtjBUvLqPFerMM85dKqIh7x0kZxp6tWGGWliPiO1yJs Qg9emkPaBIdRGK147p8TQvcKESTGUs8N3lmxNKeJSmUn4hoTQDFnMsrrQCW1+eWXW4EauNlC xIgEIn/7F1Ufbe9JK6dXKjPYLmitr+fb3koDLZbfzjtgsEKpIEcgAa1bVf1C98cKnOO5topu EZQjAQrVHQrYzDydeN6wxsFlWY6NBl3DeBIHrN6dQoTf3jiDOZuqsnxwwkv9YBKAl093LpsU OkKuQ47lSltfLnZh1qkdxpwSFDcdZWOL1UDoE0GhyOeNvBQhgNEkWLn5QyHCS0I3X39pTt/s CpiBue0swvws8QRn6u3XuEE3VbcPkL0jolT3Fd9YfVZsdOUmLGSB3KXoi19dI477Z2gkU5K5 /ER3lWwD7z4sNPh7+NIyEXGaB32yO0VRDdvfDn2K2LKefui6FtNHOjp2u36AihUMgARlset0 TL/D01rcfKJqxjtRkcyXNXbdN62hBUGznaUeeQyFiaoKk1RMIQhQyXEXY8RgrRVFiX90sK1V JncsqgnkCFhtkSV0E4EarSuK6Ib7RF2VMpty8eJfGTYp2y2u1HU/qccj7mt2VKdTufwBBkih 1AMCC5P17xD0oYLcjW3q14lOoqqVvqqXzZy6FrmyWXYkZTOtIVmMtW48NXWvmboEPv75W30R KHEA8t/ht1Vt7WiqLJiAgYhbJ444AOVgJBSMQTFWVJWaniQV8PB7+gRE3pa29QIOKqNpwvQ5 zrF+3OeTPd9nX8/NUa5JmyrbfrBxA3uMNXBT59o0JifS12ftvpHyaOGKt45qBLKe9yAxKa3L x2Ada9tiesO9mk4SLsjjOQsNRvlpVvSnBuVVOgTN/cAktBl8zalBgeHx619psRNmCE/fVdI/ +Rnu7BWeHmbktIH2037faoRkShnkPkepZslrfaLqNoY2n50t8XG+bAiRspM6HqKP+6SLzM4L mwG4+4EPCACJmAoo5oUHn/eedo/md+Qf527YvuGi9317oX9hEl8q+Prg+I9Qw3EOdaKaIEo1 fzqzDiUROK6xE7xF/xuOTl0M8rfZFb/aNJcTc/mHVPdZtXLX8TGE4Z2/mXaenKyK2miom06c RlfBF3BAKGhRldsYQ9xPv5cOQeMZwTR8PhvjwDwNnaOO13Oh7LVQZuoh21H3Q/ApT+jqB6v6 LRlBohX/akhz31LpT4Wzlqxlt9Wcl62KApbKcgSC/LkY0+5IUNqNK9kizH1Bps2brtKEIaXz U4aKLuFt+WfDAgikixaNvezLKodAubvWIu97VjKg/gkRpLVNuKL4WXFEXmTWe8U3y4gBn7UG VjW/PZUyjaQSiJexKp2DUU7AN/Hcz5nawHX7WrJ3bmgd9wN1Qiej14f2P0cAqGqLbu5rpgzd vV4cE9ZcewnAuYg5ZMegd/92jaJJ8LM8fQ3c+A2vjiwdY4uFN8RBGbcVtLEPL1jNYJOPJBdC eFNf2IAUDS+GYYRtKmh4BMXfjJ9pUDqaYr4LXYOInY/1hm2FAnlHICWDwOs4gCN84PsBGSGc VLLpgW4sPEb8u1EkN/NduE6bNF9oDnPIhqkLC24+bPD+y0bQ/VvvZGQMbsFBhwf1KWcgqkjA ejhMJlXsbAjlMwkhA44V41AjbCJ06+j36svStsWWYdA7FcImLiDZmp3UGMAnWRjQ55nJzEW5 DovNjrV0IKBQYNFryGJgZvc6H0bfSOdg3Q7RmI1s+xgyK5gMUdvjKCnKaqH7JOm6KBRrCRcS w/02nAs38v172NpMJu45Dqicgqme2pIC6w7FIXRuW4Y49GY/1YmI7z/EcQ/qIL5YmnuScZ0l O0bfWN2JyK7lc62wajdB0wPlNEcS+F0vK5NmNz6bvy3lYi5AQXCXqrjX76DppADcOtxBbkWM S5uRJmee9mF8fBzaw5P3k7g3MHKwQy7SLS4yW9+fK64clkW6rbnlfTjdaJRhojIfh+uhSsJZ gd9qeRTyFsnbLmyAO+3Emyvk27NpM45kmCbZwW0mcrqSoduGBJvQ2+yBIHhSxdlQvqA/l81k 7cOiRK3LBcIAqCikpogQeMhX7+OypJ8kRHFAkBAUX+3S1C9tyXvagDjJIzWvt0rnrqxkIqsa wjqDeM6DGkawBw0MBLYgPstc8jr9rgpuTPXowmLDW80r/E+Z9qQQ0eNsNgB1SOzdIDNzkQxY axxsbeO+pGUsbfEWNxssInfyjjTmMkb2F4DOE5tzOLvC4GWdJby1ZCJZO8bXn39/EkCNb6tJ oAnC775BP3+t9jHJ8TNEZztjpHjtli6jLn3eYoNVoic+65+rx4JY8OsOsZeg2dWhT15Pi9PT q/FCMfmUk7ahgM+rMiDVG0b6DZRweEcgMQ/41LveXkr0+LytTAMtplz3x/ZWIwso9wmzlOjc /TAUi751pCFoxg7NlBQfb1XNT/1BBcjtSsPC84BglUIPuXNgF3Oe2fH4NW9e1xbvaVXYyXsT Nd0SsyCQp5RuTrQTLPYPpZPqu+CyZxNzlTA9uu3MIv3HwZORZChNxgou0HszGqlUNT/v8kFL YEuR4XugwGANx+2bES7bzbU6FQxvNSzH13eJ1koTTJfNYXUH54V7YX8nfzrVpU4kq2tzcFPk yXoaowqKcbDYoFs0Ip5CSFK+4H2q4e2ohKtF9BUl+b32WD4l3xobZ7IImRFrbYwvP4H6cFbz yKhOsJ/Ult8J72Vopk55FOQaGMUUjlD+oiT4J2lV0ynPR2Q3BQqe4ZndJNarrhbTzW1+Lb6C sEfFjzOIgP9Xa2w8F5M+9jHJpFGSWUq5pkwuuqsM7b3W8vfvHFOZKqwbq0RmZNkxDrhqoqDY zrQLSNFnx4mjMGKvjyI7phslxxt9OXsM4QYKjS+CsJiM5EHuOfYrW17prsV7cJCc/pU4SCtK 9THlwNjwA/1EM0beGA3RgJ8fFzbJwC9id52qec3LoA6zjjSoaxGQCgdgYhE3bMg9y9Nj9Z2M jNgJx2CCKPLKrdbBfczkA7Qd+sW76xhuEfHQwOEQFGlOHTFjlkq4VCANcWuqnNff6mzkE/W9 lnnVi9cUcCYgn2NGjyVAbPo4Di5VpDR10ew59weNjmGfh54Emcj7+L7MQJu9vVOz0AxUyizs ulZDwZGYMUGAIv+l2SkyYTxRctVdKfx7ru55IX2hVrifb/Bsv/EIm1eq4XpXXSwLIAmqqPqt EHjgXNWZFLM01ZsPUfM5ZujWdlJYSihje9EHAFyd7q4ZfH5RbXrZfeSsF01kCy/6Fm1tnkx3 8Qwl6P73Or5dfr+oyOmB3zmzSPgn9Kj+J9oz+cXi/ORmbb1mkavuQl+bKrJnlZ94pz4/2Fim plZmfaAdit1oRDaPT6cG+4d5VqYiyImSxradSo1Lg70RJ4eqQk9GKA/jGdr76AIxXkAjyO23 n02jpFEiqY6J+Y1pyxX+2ne2H92HTbwHJ1LuUDhtdYD4fxvtpG2Fv6vtYWQhzBk4gmitkfdS vG37r6st2hhfF9bT+cRurZrf+qGMtBQyerDltjw9nTMUUUOMzH1RfCC3ejpuonsdLi6Xg8ye aHU4Czfrkfwh/J46BTP38e+xdY9IQeSL9K1d3CsmyZW+RAPI7ZOwC4vp/3o16qqOAWjc6Fhg W17uGWQStMy3es/AZzbq6maqjGkEKAYI8iSdyeBx7Y+kW2PQ3pEVDoGciK5PYRJb1tT/Gm5C w6cY91rINde1tF3klBk0s+trn50ttzeFIf6gA2/yT15IXahQNNgRAKlZmhpQfFW9el+8AuhR KXidVT1orWOascZtfgnsqGWGUG3w4FCzimnFa4+8Fy8G1VvhjPTVJZp0WDa0W6rPgy7JOR80 QSch6u0Snph3YOwMjg4bjwIgRoQ3TWKiQWaHIo830oprMUSF7VW9JMa+FmYJcXtRd6zDjNnR GXuS2BGW4w4Vc6ReTZAUuini3cBq607f7lNNIhvigcquAZcBbBq+oiHebmaA+VIXKnzCCXR6 WVJjovUWJuYTgCVdShAkZupnTN76NVcvHXcVqyRp/h+Sb55K2D65z9QpouukeaMhDef1c7+Z JfMdphPzy5NEagYwX3NElDgQnBG2IsLukrVWKs7xFX701xijZScA3pecEDQ2pHGo9sqZStBM 2bNSMmMYlfiov27RSS7yOVAQW3D7Nwlbnv9DKu6EeCCENXkuQorKDXLlbcOf8mlDhGsSW/vE qVxpD+ejb7Vs1nyHaFtnJqypSZB4PkvPS5RRO6xJnNXGaf041Qz+13inXV064repPRmleost 00TLZ7HiNHSaFQvga0DIlWnlGCsjk1gZRtbPxFAVD19ZRoU9Z6lXiCmILgBECi6yD2UQo/Fi GbX62XnY1Yd/+BBxbQNdMFlmiIxUzwPI0c9Wj9u+xrLNC3GElhhgHBz6623Kb7H0pv1HQFH+ f+z9o1Wz07m+7odouw5erF97xCARREojoOVM/YJRw14l0/zhtGQGfLoYvrUylOv06L3n/41j XYJ+Gc3lYfs9Y3wlCHQabePSqaQYyDpE/hNZowWP32uaZmv0JyySBZVqbV0sufhSHo1eIzg0 ZU+qUCfY5hQDkbPbVw3F9kLqr+VFQeFlmeyw8NGqPYb/wCF4wnaOb8X+8tIdHXSwt54TStLX hEYV29Lsg1gtD1a5eMjsbL3TNF0JI/bVKzwSbtD1yOJDGnD2acneIV9bN8n10EXD72zR9EQ1 nIf5Yi3vX9OvX99ZNyVHNL+mZoQeRtAOpGavBRtxMXK2nmEvHHVoSzgiAV2ZEWaA5bHBbmj7 tVB78Gz8vx5mFJ3XqvpadbR8Ttiyr6yPWi6V0iOjm5mRAZC26Z8bglrj6Bd4v3vp5WmA918E rZ8UnKgy+HMWcU4rG5TKIi++6FJblZ/pJQgf8jzITE8VVcEDu/QO0ODLNGNwtkkmPLmxdL4g y4jVRtFvs17dEYdXOTbt67Rr8YLI9NGif7apeFrBRi8qFVupSj1wm5/9SOJvBHopx9k9UGp8 94omgCt24CBJgGKlCOWDeMGKuROBok6YHDnVqhbriJcgM9pwGQ3sCZnoAuMuqLH8eA6JTjPm llFevP4dN0dKgcERb17/mH37t/ztwksE8Afl1peub+g7PjUQWxXT3WgkgVahoo/Wknuq3pfV A1+taQUjpuuigqvNwIY6gnxCNxxqv27VLWEK4nylxnZsAyZHrFJ7/lAj2Me4FaadV4UZj5KZ HpkqSXz+DczseHGnQ4a+ElBAPIHD/mVCeolyzGIN7Ph8mzh1eQy9Kpnyg0M+h1hqBHy94luC KqDto8W/cqTcFfccQMQ+2isdRqCTqFshyL19zVc6cg/maPKc6nrWGx/bjmxm5nEgtvGxaQmn l87m54B6YJrNBx81VUxSTavHToCfvzUXpUY4LxTRuWJHifJ+uv9CkOeBSB1wHM5mm7SWHZ8I 3qYeCzKmcJulAD0wpv/i09+0SeaWtfQl2yV94E3c3fqg26c7JWWxhh8O0kwJySIvrhD4v+nN 9rUZjE/T/BfEDYW2AUC8bPD1/FIEHIq7drT1kLkfzyvOKbfMs5MS7QjYDWEFy0ASBAJtbcWR iIBCGfuTHgp9Y9vG6xYScf7s8ovjvR1D26nWvBZPcgPk9Pz/gInIddgR5sEVg4uiGcYFF0CK /JfLzW2bW9QEAZxSNvLSag1xzg4+gfFa6bBsuIbPq+OyHHsjWOAJoYW9UWOA81dk630iEZNC Vtn30vF9hu3rbfS3mu6/Z8pf9lGFQ4+UWvZC/GTzA77ewJFv45+cQx+cqcz1UVvEwcxNiDcK OKtKpuK2V7F1CbcojJMS7o7A0mzAJ2iY2cv0SS1xigzjjjvBlIYOdcCbclL/3QXtgseDuIxm Ed/Pe/vcfIlL4mlxlPQUp8mc/b5ghWD7fHce3CHeqzYOzJZ/Pw8+45Rh9JGX9XxObt1hqYum hd0JpHFEO5DZwliQUnMHlaoZKu3LLSovPKCeS7JSYFdaimCZ9dldboj4E+BPJRzW+YrYCvZR R3PUl37TNOVdp519Ha46ySLLnnU+JW40/En2QgD8YmtZ2+8RhLaMi5YGK6O66bTyWKxRqCHR p/ez1Kd4PBPjN74uvW0N6zLnoG1uQ9rKY+Dr1cLVuxcBRB0Nrzr3guSvhu3Q01fROLPOlIXM OzkIo0jmNwZS/vyntbDSTaQ05XZWyJc0mJFjMt3IO7Mfx2CE+OG43QUqKuGPQPzT4yOjOmSr zV2XVHDpgYBMGkthDUKVCcV7C9Sa6la+MEWukbedSD7AmoFRPA3dD/iJcGloriytE7g8OEbt nOLR+5au9D94NsU3ipixqHzIreo8p9jtKxayWfaNAszz/9b2nanEMvXmKsS1SsrVQvojKNi2 9Ndc4RhKP8pLS749Ih1dxs+YeLeVRTRjz5hMtBnNxs0D2tW/7J4+aso2yHp9BMBmnakCGLFs z2hMaeyMqRkAo8i2NxeOhzfCtY1fVoD9oyzFRSZo7Gq6zuQuA3y0unbNcTqqTs5ZFij0+x+n OxMHyr/XkiagqimEm9r5pU3ZdQMoV0Rd07agYPzkxry3ixgszr8ToAhOMjP6ucwmwTn7s1nw GAdK+Vb9HE/6HrlUqDcC+/H7olvrWK4Y5HPObCG4BUct3JvagI1vgb6WJH5IYZ4Bs0MweQ7o p85kG0MwhHdZ20BWfX+9O+T5Gn7n+cvHZNR2wThBxQmxkdPnHhMd1NG20IkRPrcgo+mnHlUv S8wkTYsuebh3OnKVi2dyRgTMnvrETxrNzqWvVNea4yA0fYPfrJLXPZ84LWTci40NrHtaxfUB ItF1Cb2dco2D5Uu/VCPw+F1AfGchXHdOBGf0Qh5YUZYQ1kfUzMlEzUJLmFx8+B+s/rCuultV F84Eq1DZGDA6t0NDy0CHJl9L+SXFJENCZAzxxAUCm2V8iP/RD9z5SxdQhviZToSTgMRizOoe Jj/uMke9FE4VSih/lP1rgAFr9LjW7pf3uWGmq1g8CmhQb2SAGivbFiYkUrNE954dKOW8+Te0 jXzfp2/6ANVfPk+lN1ol97hinqvOB4J2Ua75MUWSFL9P+hSSHMaDY1oYms+0PYC2sempnr0x Ns8A2jQsngzAHHW17/wHyUqKc/cZWuilWRL4z8WzDlz14BGDFL2q/BDNoLtRYEwtaEDkBa9+ zNQgrranCyfKCBf2HaamvnHyAfissrFd/5racy3Pxrn0dL1+LozlWKjQU2Fw/24JTGZuyJFd ouTBHXIbWgP1qilMZvaGuO+BoXPWv236tm6yYdoD+bjk0gAJCk+iJ2cwO8eJUyupZeALULrj bLrOlXoJwxssSaCjMTgL/GfzlD1L28lNK+Le5ZJF/KMNpTUex4a8iPYJ8sF9BdhnRz/y2buh soARn8WVFZ4FSOE0Gj32dROBdqXVgtbvLHsk2vArjsmqUqlcKiQs1/dt96cGe7hQN0CHFPUq pei2eLxZfrrJ4Ndcy15hifgwKBSAJMiXm4IHkbJ7/Rxlpb5ee1tEmtgum8kNRuLqc46gjhE0 OZasP+oVGCKdl+rkN6B5PSDIaq9lEWbZ9N4c8SRwY/Z8hp8e6JEGeO1sOwKRqT9WB4liw1ce DDI9csY0QVPPhB/IyoXhX+bhjV1yGg2f54mLXlzvz3k6yu+9zkPIolxWdb7rG5kWGzNuatmi nP8mURji5LYaCCFFZS4ZsAT1ZHEOk0lGtAiuH2JdZ0QaB8LTdCF68KR3IJU4DD28lxj9+A5N FhQ4nhSq15DG4bXw0gNFQBl5pDooBCab5hUD5kAFauRk69yNu+pMAVXwdyjtc/P+qB3wJ4DC tVCMd0eOqm5Ql82qe2cMdsjstlaZHmWgEIgz1raiqjFRa8kGZ/cJqisLTY3fRC4Wv9jJHow/ at/rl6BSvkzh9uUm77xCdPyC8+j6mYkrLGzvQTJCk2uMj4uLYtpMcJ4UwTHJ6wIDZgYbvgNp BVJFG1pGvyB8AUhBnT+RBlZ8mPlQ2Rv92QnZmNIOO50e0A2JYZ8+RTujAWp8X7DS6BiR9g+4 kfSVS1qjfv/y77jmqoHJNK+rhXjJnrTO9WRFu40syCm+iGEPeXW2VOfjtebyRnlAf1hAQpiK bzF8qeigATg6GW1wQupxZLfvLO5qmApwmMNFDT1TxaJMe7XbHXgO2wprEnD7XeuTeO0Y6STT uw8EfjHFR4Nk0FuXj+DupoqX7c9KrWE7sPXk//9TWIHBpKEQBONWe1DDR3is0KVPt25kFcl3 gDUIt1xY13Qd70BeceARVPxrtFOU35eoBOTvr/wcsBAlxPyNuctHMA6Oq/9SWl8+JUOV6BX0 fZwHUycRKuESZzru5qCWVmG7jc4pKApcdnq94vqG4UcqU3F6CnV7w1ZvomCx/QlNQiFvLVlG i2dGAma1AV21xUFlYzQ60IULp/Bxe4OORvDd38BnSMxFBvMjV6GpLbVKzULc5cPNSx7TIhCI ZVbAhKrSYqu79fRg2FdqDF5t8cCbXc+Fr83njUNN9ERh3R7jn2sZU7GvzOsYmXi1A2XFoS2N wbOG7ZfqNfbGaao8yW0LdEABD0YYtZzp5dc1Cab2pTHZoKHXeZR96oa6tPaC2ZfIL8HZ6ZZI Uml6opVi89nAw1BRjlNuTym0mf1blb++G7qp0lVaod9l3OGU3wA8eHGz6f1Tq6oCUdVsEk59 Mc2gjlFLTNvX8YwuypyCpH/wUyBJXOtCcFD1d53mbXUM7s9x3XNeCa/MOJ3VtUe0L0y/ihli A8MVqIgU9FAqby0E+T40jHgPF3nYHn+eVpDOMjH/5d4kbsKfGtBkx9fFbUZ4HW6JvdfKW+Zh jffWp2gwabmfFf48Lb2lW4PXwkPT5SDS7/G2dFwd8yUvTv+HMokLWfNJ+CwS/xiNWjKZ3ESK qZINDFPA5PrfFhOsCMjuNAw02sFGlW9mIl8I40mB9+L0pamp0iTPZGFUhZ4kPPid1+Ji0gzd RJYToFjiOgmZ+c7nbrKEQ+8BGHbbIyr70gG5yvV6n7WPQutIhZSdjKnE8Y56ijn7FtmB/LmO F57i6wnleRO0N4IdLxC0s/1BBeoPBexcmoc5SS+HRiDIzOqF+n86GRzOsWp74Y/HEUAv/bNI 3CoZcMTk1YkYGs3+3MtaLGQxFLEXmT4H5Iru7eTp4du9voLPWMhvtvaZr+9Wa0KK5fVxmC0m Kl+A9/R71XEWL3b2VVQNdpEh+QDLW8I6ILKr8rr1LK1l3zU9FWUPhJz7Shp3jzQvIVc8ba5b fCiCtCb7cHhIwqyOs7MYOJ6yoU4v6XlCtguR+sKFvySwNnYGqU4C11piy3JJvWaVLREZs6Oy oASQQsyYo3u9Cw2yf3pe9o6359gsMHY832qXgL8N1qT9pZzODnUymlE4945BKwyMY2UoOUSw zTPSX7+CYuw8yE7+efFdxx9GRxIZQP0FPtLqKtMjlwPS2AO8x1ZIofTP1PNI+tPQFIpoTnkO biNG+U7ylM6s4h8VvgXpNfgbSodQuJLxV/wzJs0bpesl2QnWiMj1l2PrZjych7WzKP7fOffu YscdvKKbccNjyYdG8dBUrrcrVx6YLRrMHMFEyfolk7IebHL5ktuqRdD6mreQ5qySGlfN0aMZ FwAAfOuof0aSUlvq4xvzyd85z4psrt19U2JptkxM+40sNA7oMLkrrhBxwgNhEnHXBCEeEsKF G5QT4oF9S7OFJMXwNAKq7RLAdi5jzKK0jc0FMeg0/lzefXuEPUKYMg4scGuFw4o/TB66b5ew /ZE7CZ1SfJdyd3Np3bvXEYu82Ld48zQ6uU6w9P1ENgD7HPpSI9LaGCvnK553j8HhLpTpmfj7 gZ7OsCXeZyZFaG1BbOH5m1/OdNcIf6bgJAKqqTePlj6biFjiGMvnRnzvqWta/I1T/2Qm0Hfr DY5nqKshtcajnPPAEudK5Mr2Ez/GiYbx4Em3MzdMF3i+sbateLFncmc3u/Nh3UjXTAC9YR9b NDUSNiUFbQmPWTZZIvVX/j2DlEevwcJGVea0hEDu1zSp5GjCbI7FLAvqjBCQUHmX5UgugnCq vSgwnIc+pIWHz8U157TZ5EHX58QkXs5qXYwbQB/3EjDkK3HHSHFkOCICY7Lefv1+sed4+fZU 3W9k3j8NWOuWBYrGrhrT6lO9F17kXvurL/U6VEr5ZAxuICZpGWu6XVpwfexiwZpGkGmHWc3J WJ4q8n5JtkQqK3O1C7DXVJ80X5DfqXywnG4aghPMxy3BHMXx9L9CEJqd0UyajEs+HoLYxb8j fusPnopLJrANJzZpoBqsEohMUZAVbSoVe9gS6S9Li7w8lSMIdIUHX2+XdzbLVqvUqC8kFrSI X20ZmxntxkCoTKAqGGVpCo1YrvR0gfsfNquCVm5EaqL5weOyM3rhlBG8QhMt/RJoyceU7OaH VB8BOfpwByOOvB3tdek9dtspOq5BFG7K7bGMxuHt6MTO4ZP/bsby6fKnLPx6mt9HAy4rhsoU MII9JkmQqUzywBhl8TVuk6DPAPoOpAwxMUm8yleaq8UVs3OYA2sSFzephT+/K8SKfNvSlbRX CoRrsjRqRbweCt+M/Wc7seB1muFb8IRy6IrHBd9tsy6evZZUkfdDkr4e6oCU4xRuu8SqeTB7 srzwLv9FwfNVB96hk2dKycFpqs/dFt2F/PpYI7oTl4h8m0RRUPQdkmoKBh4HuKIdMGmQXsUc G7/68JumfqcWLgE2t5dDyZ/JLmR2PIKF+j24kZmwY7hVZIvnL+jF6Wb18xx4gVhSG9jpgrJR vb8TWLPYtszZwqI0YQnSv7CHxIiVXag6KgNS4//3bKo33/fYHpvr6/V20lj1ONB8zuhLdKk6 LMsW8y4aO5hnI7PBAMUpzsbSaXEbbpoAcV8G7FqaS5o5amJH08FYHjBIaekU5Id+STYNa2Pe uRA4Tc7Fjg7aqSA4UCT7CSftlZWha4K9sB3NfVnMzRWEZrLuyrT+YLJB3f7HvEV4OsgfNMIa D542soB1ly7zTjzzQW/DQzqHA6nDreTXW2Ce7Gc+iMlHv3w7Em2XaoTLV6Kf/McjEDvKvg9X qA7zoQKVmW+FXXAczxbR1i90+7BzGMDr+hCd8tH8PGDDSLGbRmoaMoa77hF5xejIxJbO84XF YxSQMYMGKbeu/livgTSNLokDT1bzOwh+JPRo3PSh885DJnZ4k3fOL/3ppK7QONLDcQHVgntZ n2pqskc6rfFgmhoh6uwSwP9jg+XYniY6TIsZxLGsFLvEJuO3YHGBi0h9/G1VmvDCwvt3tVVJ ntXXCqOmVE+hyWYHeJG2FpjUZNXxaDrtjQVO6hPEeQk9d42RX6hJj7jQRjnFdPdm+uzhc0vS SmZszNOdxDDYxMbi6+xxmvptkTAGntQn9IqFo9Vx+D8Rt24JueV6hdfLRkPnolyQH4Qojp37 r0LWbLCV+JfLnPAxr9D3YcmpHFwF3PpClfwnDmqU/54LoKn8rN4e5DhOziUE/GezwlMWoUR5 lFs5bnJIugNNrzMGDmDFsiTwbLhG9qER6Kv2RugC/FZy7PP4C756JW0icgk2QR0dMm58cySI J8YfLZVbu4jeQC5m003QIdQAQvdfvc7BcPFSVKTzOgsuYE+o2CxkPoe1cbQvQP87ipEAXYEP 5R01O1LEiMsYbHyKCgo8AifQAQGqrz23wvjBCsgeswYK7SsrbrzjguRWJ72KpQQ8EAaiRBrc EFj3TcsFzHp005wlEFEwm4wz7F+NTpxr7Pg/dZUrKtneE/r8wwgT67sdiIZZOdNgOnGY3J8B RjNKDUWR8xIKNPto2GXttJBQL2/wyLHn7ExjBTh3VEqwE+zMakjccbQzHA7coCUoXhE6dW6o z+7JWbJ15QgXV05ipkbvTvEAXaLe5/Oez1h/+htNPLgDz7zULO39CPvKdW2NgpYTWbIFQkeT RFgpNccxu1JeB0fKiYJaal/WA1t37V8J/MzqGgd0zg203yT5NDGrLTC/nf3SbiobYE/VoHsu 0zUj+qiX3LEkLQBOqFgKa1IqT/SHS7buije5t4x0irnUqHHI7k+GTgQyq7Vx57QNn2EgaCYd 0LbqxU1GUnwBo12nHoqBDzC+QxREKKe9YxnCRbE85TnmMwoBwU3c27dy9t8vgU54lQ7UfTw+ ElM81gDJ1XVL4Qcp5qWii4xDvgQEHzFK3TnrVfH4FXwabMofrPO1KbdQm7DUKmN2mhGyCtC/ rqbI/tHt+BXFDU0wGJwsXROITBrOUSJTT0Lg31xKWF5zB90BjxUnFNEhBL1lCHlvwGRbhqqF XGq/2KeVwV1WVOfzOrc2KBymkJ0nqguCbN+6S40HU+piGzZvp/Gwh324+Hv+dVKsIZO/oOSF blaaN05eh7lB4D85Yw8UKa5t+JpKfW0nHlwOJlOk3CHBiuKLqMwhw8f6/K9ay6KDsfmbC3LA UoQegXDINA2+25QA98toYPGb8o36Ad3nDKdQgSqaYXpT+VubUSlC/SrcJfo3t8+eEiLqFTSO exK5+j/j87m2jJ7UI0J+JgjRUMmpMzNfGRhciluxvuDgWx3aawN3+OJlk6DgzOjAq8yzeeFd 9AdHy43PWUb6XSInSrkXj6S45eC4GvsLedqmqRPjji8DqBYBvek/fUGJr7SkelA+wZwgVP9J /uDJvqCgQZBgZoabRLrTQzQlC38pgWpsvn7Sw/OH5sKMPVwywHS69lCw9csGc96vZIQfZIgu /GeYCzliSWDx/HxkmteVgVf7bb3wh8fvlRZ8SoBx671jJNUU3ivxeTjPslEY5wSJ2YkaA5rp v9wNTVCxxgLSDASTCRTsdBy+Cs5ID31AmQzvzD+3VgaflZ2YrsIZ4AqT4fJDj3RyVPRPBmUB hHz9DMwxZhDpqcVJMIv2UBpD1xr624xDr6pXdId3PTHG39bjSO92ojuu44hGPevrkLVvDmGm CiKGx9Sj34ryHyPUfTAPjJ9hkk8RmWxXJMAenXakVHLJDboG3JJF6vLKKKqx2NIKGNKfTB6s h0+iZIFfyW2ul2YTAZjSFF5AL0FNH/EabsXMlfxPYSlQ2u4LG1do/ejQimDXoWfybNcDtiKJ Lrwi+77w/ahI8dwUpaQFpbF7c4I29iFtH6GOk7PGe8xRhxXIZw7nscf98goRcWBfES5OkUMe JkAmxXkycLVNQDaxBOJ2ziOVePwWKMTYiQflkgj8M6kiOumV5BOK/xe0L7Pwo2Jqn5m1FAEa QB1Drk3XOkd2kSFf42QSlYyWAl7C1rJWsZdhX1dWIS/DjzeBFq3bzOCClwgzwiXRN+VYIMc2 dcY80QBV4V54eDYZS4A0u8cNnqPL1tLbvREv0nVbTKBe0jwddhL8yXKSauUtA8qA+HX+oFQw rHpyPTvbIICxkYyukP6r7AFxIXHDMjOo5oe0rT+ZCGTd53XMjzkfhMbUX1uUUEAxw3R3PBYp 9tDKzKWA2GCQ+fA0FjHvtQaaIKwYsC/9tX2RECRtg7q4APouvjzUPOXrFCWiu4GmQrMZXcWE BvCt9h7f3WXS//1ELaIhxNUv5Q2AgIT3fid18DimIx3vjMMrD7+wSGF+4wgxS3Su+PYFoABT NWso2i2RGMOUm0P6wVFelkiul3AUWYEsAcZT9YlfsoPZpQIdRrzj9BrkDLCC6NO9V0ED8vqM XAtGq7/YfJIA9Ve3o3FWE9+NYYLd64vDWR5KqLD3xp3pI+wWvNQ9P104aCR+VhICfZY3LCyP veBU3VmzJKQHd+f/TiuCS84RYcr+EVR9pFuwEhJRnhAGK8J2KX0csKdPELMK5y1SgmAWqFQt h9qn1LJo45ZsNGCITbOuJlE/I2y/WaKX1yCfFZg5NDZxpUOvtViO5npKNgYLZ+/Tw5pvkhyo 5oR0rG6doFMZOuMu/4U2nWJ2NOYoFM5DpsvWTYFyGtSU73I7t11TjMgL6MqwqbTe5Ri++bGu ieoWNBDDemwoOYc8Zvm5S1dWzpbCGIDZAELTfGBI287DBbPICp7A7uAehoTBaUkEw+kGPZFI aWZ3qVZIjFRnVILxmpiHqWNklh0PDHi7rt0ZhPGhb3u1re0UKJh6fvg2W8e5bd9PQ2S2QyHA YTKOQ4TWQUF0TyEiD4jmKTdnhiWjLeLrIhECXrFlN/mAIbHBXZ5ysQIFro2kQKo4F2bsVTHU mz8h9tlCrZYrV8cFqdf6Zd1+Xer9RhVvAZO8XhXGoFFe7vH8evygmmT23jHrqSGpk/BaxYyT c3S/QN8konU6SYKoRWNk/bZuvTEEWcc4AFUUHDKm/g6sRhoJJRYtGKcJcNFiDKS86Y8fWYca dZCAHXKv2A+RSx3vmGo4BZ4bv3dBIHK3kbtJ0QY97RWI2cvvDMuneSosON3MPrCCiga4r9JE IRBpPfzEralt58HDoiBr544igCZ129wbHVNDPDubG9ZUN0qG4dCsB5ZLpGU7QYNItY3cgoZK NZKmS8MXWbv00UgZKFrl0yR2l2JwlBRr862yXTrjAPw5QflIwe4eiyVZfKOnvmRhN5RIS5C+ KpnY3n8nCjdUXcpmuWjKIdJfBPD0C/FPR35DbetpAvXzopu1rgrF4IvdSdKJXaf8ZfsTkHVw GaFgIq97T4Lf3PCIF0owQnga6Nyiyzoq9IJGNupFBGl5Zlo4Sudqnlo2Ny92vgSgtwggf6Nm L5bH5G98rxZAIuuvSTrhX3jAydG22wxXkshrbRL3EhryfBInqHinNYd1CqNxmRkU6eWyXcdw tz5TQaZKn2g7eIYuXrhqmDhwP7SFOhSWKd6dTPFg4vpGl6wDHfd1Hw1Dewn5Lktv71v/nEr0 9jYMMHq8MQpct2gZIIyXFNg1ncDjb18pEfWmRpoOkDadDJzc4PM1ViAPFGPatcIqX28k+w1L yIbxoiXMkPReEbBD7YbMTpCiM/6Tg01SeZ4EkFjounck/fgVjkn2XD9WO5Yp+8adwn2fV91T DyquRKXzH7zzW/BThACB8QYfsoV/LX6sjfGYlbei66/xL2ZqHIcJ8mkW0Itf6W9GDous5NVe f//wHqw/jDODL9IwexSFUgcyDg3JzkrNa/Hp4drW/cch5YpVdTFmHrYl1KLNwizS3gHGnP1A lsAEeYJCiYWrAgm/4NHi2wYhFQkGEeNOAQp0yHm0ciAjSZVyVquGGgg+oKbfevpWfUYuhMbF XcL66RMiELNwRPH8QMA+RNatpxn12biOfDwgjWTbjpu6cht2bDCMh8Q9k1x7jimctcjBiRqd URrw3XTbRgNFLTrgrSIYLP+c/bcHNwzOP+gS5yMOS0BRJ013DXyTnw3dNeZYlT43h2xaEE9/ A/xksIgyCv9/U4UnAky5lFB5yt/Ruj6jJ4V31XEnyqXZrhbk2ICZniWRxIw4BvOQ4xh3VkS5 k4kNRuvcmgcLsC9sGrcNhHzv5+E9UV8Un547p8GM0wALn9ebwx0o4bC+r4kw3MGwQPRSwXHB wWz+JY2VnEjNicOrFIndCMDpWd+BNPTAWG4Fl6OsEqahYIhcxJVPlGlWJ6G70v42PZ2XesMY n/6473PhNDpBZ/Dx5SasVsTlUOoUhrUfvT3jyBUaEbSpEabJR6A6hk19QzdHI2T4VKN8EqtV 7FCZIY0PQyDwEKJWdyEt5kmarLKfOnYIruleWP1D3ENfrI0H59vbaloGT290NK1jZec57l4n FAI8jgxhR4q4U/FfxJtoHeUNwMpFp78fRMAgygCPWUhtfN35dh0RSaIcv4WmtAF6KFPUqvxG Z64LxQTDtMh2gdOyXovVzfSi9CVCL9xX+rmPgyTfJ7p6S8iqIxRBfr71p4tivqlewNTABdur bMcu/ajkieVmEHPrAv5xnCqHD23JF8enueTGS+zFdFF1XYIfPE0wGVPCnMq0MqQ9bahNb3pm bvlBB+qZZ0EDDuj9U0pu46pTcREfMZsEo54h1/3ljfbLpdQodgOL8jFdJGqPZI8Xg/tYWT1i JpFIzAjoDLMGlAY+N4jbgsq8vU3MjRoaJdRhDk43wRGjDNevmuRAfiNjCwH5FJ4JDWbxVM1Y 5mMmKYiNOTllBGzvmh9gYIDQ102qB71qPt6lniwCR3rue+FFi4v3ch8Jo8y4PP866Ud8G4Po W8A1fBRJY0uWeV2jr7fhnq4xWuQ+5I8gFIesMDuHGAbJT9TdT9OdaEQEkxPGVOZRFDVWKbqE viHCWSDkGSMmhzIuGcfwiGTara8zWnXHlDxaf7ALEHhTp7Xnvpyo2ok4/yVcvfcm6xAP3fqe 5zDzB66Dspd02sZNrcl38QeaJF7Mmd5oWz2B0Aufl1ubfegDgQDoCed5x87ZCCqJt9OzwfIo X7Y3b4hKX1r818CAXUyGF5mrSjxqZhmwWo8bTFutbLv2i9GWNVGLvYmGE8NdIT0//YtOkOi7 1I2Qq2I7a/r1JbbIAflnLf8r/rSAelhF0TMBSY8qJX35xuGOX/BMWHKo16Nsnioegsx+PaZ/ 2ps4QvhiEm6EdQzQXD9HTAAFeqEiKsTHEBHduJjzxFhVR5k1l+DRLlx0Mbb/uSqaTyAuCOBh rv42W+Ia5M9htNmsv025C8eNlCIPN8AdKTeQGQLoEyr5uYCqR9doIoUpbtM8wnnOL7GD3vCh icd4C8Rw1DnFqI3oKF1VN8mMGQbPLZOW813N2AXowaBGQ4cf6lYJ/dFQrdSWjVjFTeLLnIvR s7Kh/22vzCyCgFmiQwlTJGNAXWjnAKrEm4727TZPnTVb4KYHEz7PHxbzXaqgJHGyyanPg/FB GQ0dxlFWw+n1ZimjORDE7nbxNRB593z95ujIKNVMZVJWKgWXFDgasndhJ0NSsuUqrTg4IV8+ EXh4krrjO/O2LGq4Dhwu6FJv5ezWRfGlXwjs8NnYQ2Mb+wpCB2pmo6X8Gy0hW7AbXbRvHISQ m/eRakEoz6Dn6pcbdAcSegXKtdzXrMjTxi04wGxOozubS5yv8bPuKM34PQB28ArvccpNCXJq 5Y4EPtk0SKohSUZIAo+7EaUjI89OtI8iDUntHe4KbRCEgp6pB+yBziAeO26EEbYVjng1BHPT yi3P0n0LmU6hYOYtGuoRpup6uCZ0svWzYJKbaUQ3eEj3NJ50Sn0ZO9l62NtgI4f+3b77arEf HNmZ+c4GdwuPv75YZC4ybvl/oWG7fhWwo3kfsEmJvH7UTuMQPUOMskTMmDXKagz9VWfhKqaz NgH8D72IBwU/sFDS08buU0H5pOoHdi8aTt0EgMMD7I6kMN+zB9tZCpurx6Yq6LMPfZDDW/Sv 9FMhH7cj6yxRD0FpFBwuat50C9yYKp2vvGUJuwQWSAO0z27tgYJ7yZqsXh1B59ujHGrKIRM5 lcrRAgluWjYc0KWMG+LmeqMvk4H6rSlQ/CeQxiBkgeg8lOQmBAXgWCZoCQ835P78zHL4gZmD ZbBa6uT3pwjON7P1ZIarGy2l8VdqBe3CLOqzxeWe3W1/77V7Hk5ovZCnZjhoxhZkedf8TMKa fGNAHUporBBKfANGqpbGm6WTfzYNYHAJMPg7avF0JvmL0t/F7r07h7OQhxkRKaPtjYzNzFcc +OSrR3eR2NjGIeQsteAoOnce6otz/U0TcUSD0dyGnv6fL70QChWPb9c6zwBBV3AgNF6tGpFk /ZJdR+yjbLq+RIi+dLiEOswIq4GABeyRG/jJvxUbFzKj/DXktSM4JhMiYoWN8ZvS2mxaQRJg QiRO4UGpD6heegeLei44VB2AJuKPbt6mHafH7yGnk+44o2eSwwG/eASakqj6zTHjExrFbJyV k4mJUxLaUV7QJytDsB407ubKOWW/YAgmbqiBlELcLp89hVDxIjbQ30C9eaDb53IazUsiJ3J7 9sC44sX3inH3OaCMLo8vpfm9CqeU5oVGRZQRE+FS86UV93KqVznGs5euKeHFBtlMKJczGI6u F45IxIjQODYJ+uekcEe97ifHDJdvUM7hauVEpo9SBGvtw4fbHh71Q7r6ywWqcE5u04ZYhhY/ MgUYQnV2f5aKanpP3r+I/Ll6+qHCFaGJhREZksAhwjsez4mtwqgWi3LiyuIS4UmQKrYlxiuD Qp7mY/2IRjfLiZxDL6kVwYbsLle/UttAAbhyNU93kzQqhDZLCxhg/DYS/CGXCAwpYzHSI5Rk PWJvcBRafs16LLiKN6lFCuMKq2vXjKkWNc9eQYMDfMX+Xvq+xsQ+bymCCbyyr1f7DAnSjzNH Sr8723xOapIWaHW4F5dluhVrAEHhduTXUxFIU76wpCqLkLdu5igjrgLV1Z4lVxwM4DF3XCu/ dZZZL2CUPGZeqhhN8nDm2RXt61iWNbUOxvFj7/QpvpYnuEuagYBP94vHOv0VGC/5Y5gkXtqU J71IXW6WqC2Fw6TFzKgVZ+17+f03/r31dNTPHlf8FctPNbMGA3h4EmIl2grLWEwgVKBktEKm B3I/P7FCgg88iBGTawFUtVyS+gYiqyrgsnSDxqc1WB+FPvJeyTKi5JNzegdiroNAMT6VmvIx AH7NOdcaZ6homGo6LETMXnaSb2Af/poy6BY5au1daRO1Yi4dxbBI08ycTlR5tXoHOqArv/Ob Lv+wQKevIIV62f6tLH7m+FubXLtn3r5Qe/XR2PQPjYvQ3w8wLzfJm2Pqqx8JOT/RcUPcy/Tl J70QDScMprLqcnXDQOAxVZuJOTLuO+Y2uFayOrdlwjO6SvzEMnxhr8lPlqQNho0jETHaGkaz OQk8RBMEgNBwFWU87vB+04u80rS7ru0oRtslTzwpxG+YX9gujLXyzUfwop5F5PM8m24PadyQ 2HzYcg5Ce5Yit19dqyi0UHMmY3LtfdjWvsci+A3miSI9k+a/OWHOgdg8UydJgCnA4R1GLoGl F5X9uKtf3GwxXSxICwZene0JkW9g4FoKaRttGJcVHA96cJ6dxCAvhtFQdddbKySV9sf1RmKv y/jj/8NX3NmSLQPwD6ITIHBVmuqXvOjkEuPnWtASouwFnnW1jFfA+VmpSEjwD4zT5mVD3Brm OhsMHP1cSM06DwnpkEXvgrtuuLzrMDkeetKcT5WedsuwNhZWbHmqpjip73CtRexcKGBRYPAP 5vawjBDeOHiGnP9Fuk1J+sYmjnZgH7OKYPJoKl51CYg0szLX0erFPpKShqpnTgW9Wt1pqYw+ tmPSL5CbTntTqqgXIb4KyjUxV4tAkozbbX3QmFUNgquEeJAXzGPW+4mKvRFiN5m9P+OfEyT6 eFWUHtpNXDegfD+oEz6IwII4s8AQ+n22L72+X1vCGdpxwT0cyZbo3uEds25AIMqHXaLyQkqH CMqlgK1FffOpfTXOCnrImUqPVgUb2VX62KOOiW9uAXUSXk3Hpzid+LRr1DZ5asv6CScjOlgK raRb3c/syz88Yyfx2Y6PqquhjFUnrQXXnKL1baVEbMnHja1a+BEoPKst0J3Fb7UV08AVgTZm RIwF603OfWASTiUmYXmn1rTPtF+/pHaZnpM/3N0bT4JB3koxbKUGRcK7+7Daqnc4sfZqDGiY Rk+ZxeMHM7QyRz+Z5cf+Yool1Qcxl/eowk4Tf1OTi1cNKMhrkhnFIpSwB/OntYCRebQGuF6A aKC3dLxszwDDvAi7xK5sJso9WprsNsy1yaJUsmvmC7uLpImTI2HAhOEYeknKsOt4Cz6RO3Nr o+NiZar6EfO41IQcVYu+P6MNqDNajUWwHuXOoR3J3o5oOesp0qoVj4l+CDIf5eOFI8XvBjNO WAE7OiKbY4QzmxfANFATIRsggJuRnuMsyd7l+WgwKGESx8YwN34O8QetXaRpEfxBvIiy8M6t P4uf+FqBP1nAFZvW2md6qu3VHMfIAKwdxErzT13WnrFMHoqVTLF1TI0GTxVg1ual/g1eufYU Ys1wRYRkEXERJ+xe01pVG1n3cLu9H3MI9+mqG9CeGqEbQSluYVnHxC7a7/fu0dk1+Bgud+4M UliBjh7TMtH3S8Ix/sxrz6KFx6LJD4nFpxVJHwS0gL4fMJnumb2zA61Kx68gYDO7eeszSCdz E0Sek7brbUIcDej+UxgPeVfx7XLGxoK4yv30WibwMFrqWMEUoE17TRJG6fZtoB/REHC7qodw OYxdPb2lAzhu4dwHLGLEmAHHTH478ujdvgo93BnUBUOboLJVGRj+1IH+w0v08BEx2X7P8V+w sqeLt3egOowP58mh6JWS1arbVdb576yfFh2rGNNfD1hPWUKXgtZCb6dnR8X/dxft9tQExfgT QFlOHY5d+076HfnI2lXujCdiLznyii1A2DyiG+WGwLAgBf8RagOQjdnc6VLgzs8lsHeWYTz6 cEwV9X6dr8stATF6qk642etGr+EaT1BXzzGvZKacBVRlYBLEPG0MlRbf7LqPJWW5nSM6MJlN 9Vi0qW9OLW6LX/IAeUQb5uCkRFBQsMkqO+F5sA1Daw1sg03X09kCKKUUyK1d0n5N5ntCdT9g iKH2jJDRXuU1qXZGgtuscigSNvR9yrZ3jawci8czDGJ5+FDVvmEznllKh4CWgecS8iqsHVfs x6Y5co6eSxoLq1/3TlS7El/68oZkuhofLdjtOKLYhJH0r4t6HHkLCiB7ZD0gkC/vWHkiHu6L i24SCPDULYmBLHCpbeH//qXWfxrXxlI7DaPMFth2NpynNwTE8TUXQblRJXJc2C8LnEaiVhh5 Vj7DMBBG9l/x2FlFyGuMHgmJJklJ4pYMMCA8a2D87cecI3YGb40yRswdG4f7RJHqaEX95MZX +Z1Z7qEP2b6AGTwXn4D8Eo5GilZXWMhc6rtV0D6mtpwkqi0mQ4u46klCkSghSEnX1mwwA7eT NThucK5lM4kplBL5Sr+5faGhIqbuPpHUGdU6b4ntPzG82smfzX+Yby+8Tx9q5Ur6W982+4ht obsgs4k5zPA/991krmi1GCAKqdcQNRbrHn2LSOx5UwP7RMGq36oGk9WFW2nj4xRlp+zlvQXw lbKepx4Lu2Vn9eEXkP99HsNSqcxkJUchviENN47xEaGRJJzuVesIf4vMpVrRlj2auzFHz0Pz z9YvuM0a/hGN+h/iRgGEpC5OPqOi5lPLBPkdGLHAPsVid8mP5gjf79tg6Z5+8y/Tp3I2HXgL D2QRr+rTBrop+CLr+FusrxAxiXGLa8oUKMcppEkNd6B7wRcEYhCbzPe7NPsS7sQAMnFH2jNX kIXfNyySuh6bAAEu0mzoXsurVN4nzZrvi0Im8xh/ZxxZRGg74POhlkXIfaSk0hQ7czYk47qO DDu+1vj0j2WPQWVY00y57nC+kY9L/4TIe2SbnLBMCTKfb7vytvjwL8Cu/FAvUODEs/6dE01s H7KEM5VKueykBHJmCSsnSDHAQO2IV+k8djqb/XFpQ1GJhgsGAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAACAAMAAAAgAACADgAAAJAAAIAAAAAAAAAAAAAAAAAAAAIAAQAAAEAAAIACAAAAaAAAgAAA AAAAAAAAAAAAAAAAAQAAAAAAWAAAANAQAQDoAgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEA AAAAAIAAAAC4EwEAqAgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAEAAACoAACAAAAAAAAA AAAAAAAAAAABAAAAAADAAAAAYBwBACIAAAAAAAAAAAAAACgAAAAgAAAAQAAAAAEABAAAAAAA gAIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACAAACAAAAAgIAAgAAAAIAAgACAgAAAwMDAAICA gAAAAP8AAP8AAAD//wD/AAAA/wD/AP//AAD///8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACIiIiIiIiIiIiIiIiIgAAAj3d3d3d3cHd3d3d3d4 gAAI9//////zMH/////3eIgACPeIiIiIP3MIiIiIh3iIAAj3/////ze7MH////d4iAAI94iI iIN7s3MIiIiHeIgACPd3d3dz+zu3MHd3d3iIAAj3d3d3P7O7O7MIIid4iAAI93d3dz87s7s3 MKqneIgACP////Pzuzuzu3MH//iIAACHd3dze7O7O7O7MId3iAAACHd3N7s7s7s7s3MId3gA AACIiDOzuzuzuzu3MIiIAAAAAAADe7O7O7O7O7MAAAAAAAAEQDs7s7s7s7tzAAAAAAAABOQD ezuzuzu3MAAAAAAAAE5uQDe7O7O7MwAAAAAAAABObGQDs7s7cwAAAAAAAAAATmbGQDezcwAA AAAAAAAAAE5szMQDMwAAAAAAAAAAAATmzMREQAAAAAAAAAAAAABObMRERAAAAAAAAAAAAAAE 5sxAAAAAAAAAAAAAAAAATmzEAAAAAAAAAAAAAAAAAEbMQAAAAAAAAAAAAAAAAABERAAAAAAA AAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAD//////////8AAAA+AAAAHgAAAA4AAAAGAAAABgAAAAYAAAAGAAAABgAAAAYAAAAGAAAAB wAAAAeAAAAHwAAAD/gAAH/4AAB/+AAA//AAAf/wAAP/8AAP//AAP//gAP//wAP//4AP//8B/ ///A////wf///8P//////////////ygAAAAgAAAAQAAAAAEACAAAAAAAgAQAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAACAAACAAAAAgIAAgAAAAIAAgACAgAAAwMDAAMDcwADwyqYABAQEAAgI CAAMDAwAERERABYWFgAcHBwAIiIiACkpKQBVVVUATU1NAEJCQgA5OTkAgHz/AFBQ/wCTANYA /+zMAMbW7wDW5+cAkKmtAAAAMwAAAGYAAACZAAAAzAAAMwAAADMzAAAzZgAAM5kAADPMAAAz /wAAZgAAAGYzAABmZgAAZpkAAGbMAABm/wAAmQAAAJkzAACZZgAAmZkAAJnMAACZ/wAAzAAA AMwzAADMZgAAzJkAAMzMAADM/wAA/2YAAP+ZAAD/zAAzAAAAMwAzADMAZgAzAJkAMwDMADMA /wAzMwAAMzMzADMzZgAzM5kAMzPMADMz/wAzZgAAM2YzADNmZgAzZpkAM2bMADNm/wAzmQAA M5kzADOZZgAzmZkAM5nMADOZ/wAzzAAAM8wzADPMZgAzzJkAM8zMADPM/wAz/zMAM/9mADP/ mQAz/8wAM///AGYAAABmADMAZgBmAGYAmQBmAMwAZgD/AGYzAABmMzMAZjNmAGYzmQBmM8wA ZjP/AGZmAABmZjMAZmZmAGZmmQBmZswAZpkAAGaZMwBmmWYAZpmZAGaZzABmmf8AZswAAGbM MwBmzJkAZszMAGbM/wBm/wAAZv8zAGb/mQBm/8wAzAD/AP8AzACZmQAAmTOZAJkAmQCZAMwA mQAAAJkzMwCZAGYAmTPMAJkA/wCZZgAAmWYzAJkzZgCZZpkAmWbMAJkz/wCZmTMAmZlmAJmZ mQCZmcwAmZn/AJnMAACZzDMAZsxmAJnMmQCZzMwAmcz/AJn/AACZ/zMAmcxmAJn/mQCZ/8wA mf//AMwAAACZADMAzABmAMwAmQDMAMwAmTMAAMwzMwDMM2YAzDOZAMwzzADMM/8AzGYAAMxm MwCZZmYAzGaZAMxmzACZZv8AzJkAAMyZMwDMmWYAzJmZAMyZzADMmf8AzMwAAMzMMwDMzGYA zMyZAMzMzADMzP8AzP8AAMz/MwCZ/2YAzP+ZAMz/zADM//8AzAAzAP8AZgD/AJkAzDMAAP8z MwD/M2YA/zOZAP8zzAD/M/8A/2YAAP9mMwDMZmYA/2aZAP9mzADMZv8A/5kAAP+ZMwD/mWYA /5mZAP+ZzAD/mf8A/8wAAP/MMwD/zGYA/8yZAP/MzAD/zP8A//8zAMz/ZgD//5kA///MAGZm /wBm/2YAZv//AP9mZgD/Zv8A//9mACEApQBfX18Ad3d3AIaGhgCWlpYAy8vLALKysgDX19cA 3d3dAOPj4wDq6uoA8fHxAPj4+ADw+/8ApKCgAICAgAAAAP8AAP8AAAD//wD/AAAA/wD/AP// AAD///8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAOzs7Ozs 7Ozs7Ozs7OvrbW1t6hITFBQVQ0NDQwAAAAAA7P8HBwcHBwcHBwcHBwAHBwcHBwcHBwcHBwdD QwAAAADs/wf///////////8DAwAH//////////8HB0NDQwAAAOz/B+zs7Ozs7OzsA/95AwDs 7Ozs7Ozs7AcHFRVDAAAA7P8H//////////8DeV6gAwAH////////BwcTFBUAAADs/wfs7Ozs 7OzsA3leoFKgAwDs7Ozs7OsHBxITEwAAAOz/BwcHBwcHBwcD/6BSoKBSAwAHB+8HBwcHbRIS AAAA7P8HBwcHBwcHA/+gUqCgUqBZAwDsAgICBwdtbW0AAADs/wcHBwcHBwcD/1KgoFKg5VJ5 AwD6+voHB+vrbQAAAOz/////////A/9SoKBSoFlSel55AwAH////7OvrAAAAAOwHB+/3ku0D eaCgUqBeUl56Ul56AwDs6+zs6+wAAAAAAOzv7/eSA3mgoFKgXlJ6XlJ6enN5AwDs6+zr7AAA AAAAAOzs7OwDA6BSoF5Sel5Sel5zenp5AwDs7OzsAAAAAAAAAAAAAAADeaBeUnpeUnpZc3p6 A3p6AwAAAAAAAAAAAAAAAAAEBAADXlJeXlJe5XN6egN6enkDAAAAAAAAAAAAAAAAAATbBAAD eV5SellzXnoDenp5AwAAAAAAAAAAAAAAAAAE293dXwADeXpec3p6A3p6AwMAAAAAAAAAAAAA AAAAAATb3bPdBAADXnNeegP7eQMAAAAAAAAAAAAAAAAAAAAABNvd1dWzBAADeXoDeQMAAAAA AAAAAAAAAAAAAAAAAAAE292z1c6tBAADAwMAAAAAAAAAAAAAAAAAAAAAAAAABNvi1dWtrYaG BAAAAAAAAAAAAAAAAAAAAAAAAAAAAATb4tWtBAQEBAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAE 2+LVrQQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABNvi1a0EAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAE1NWtBAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAATUrQQAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAP//////////wAAAD4AAAAeAAAADgAAAAYAAAAGAAAABgAAAAYAAAAGAAAABgAAAAYAA AAHAAAAB4AAAAfAAAAP+AAAf/gAAH/4AAD/8AAB//AAA//wAA//8AA//+AA///AA///gA/// wH///8D////B////w///////////////AAABAAIAICAQAAEABADoAgAAAQAgIAAAAQAIAKgI AAACABo9Xr+gBrMNbodGQYynekCdkkkKJAmaoGZ/W0JlX5lELGRBVKQdJQc9jhkmsry2p7pF ZCt5NnODkX2zX0mmNxU6WS5uLFJ+lGnFp1qqmCCvlUSSvUZsvgCAMy1kqDyCjwKynpWtcJku SIg4bn22OWAyKlp1Ai14kWgsn2pasJWZNXiKn3USwI+IZKuhkDVtP4dgGa0bWwmnQpJpdyKm kWMaRYmvHjZRaQuFalYQwmqMW4g+uC4YgWOUZ7t3V4YUn7K4fXA6GRcdl7WqbBhWH3MtSX80 lrN2ARqiFUUYJ4okBhxTo4xavSaXYLwai0CEbKMUoJ8WJ7GhckyMaXJdhKMwpAzFAqJzVJMd ZRAZc3ylOF54pIYsS1+oY7+Rcp9iZVShfqSiaGWUDyWkIyFRuz6qDbEFlrmSlFa9jFNJmwCK g704WYCJDSesUJURCDimZMVleWJ8B1lcZwdBwIE0QVwyGC5PAR5VwLAUkEhUu6xwQVObiWMJ qn/GdKJ7cUm9a44qoHRGbXJIexaYXVdLtziTWGNdcCoUUlm4jWmRYYE5lBdKPQ80QK02m1qR hKwpEzUxuxG9bSEOJLYdkiQzcZAwmFE1bFk/DpTAw0wZujYRI2NHJpGorQgmtqkeV3kdYEIP WW4LBQeSfAYPb0BNw8K6shChvIKosSOiEXlbjiSrQQIXf6+HnZAssRAyIzhoFScbaUh0Om4/ bACToYMDm5yTahZOLayJD2u5sk0Sv1diqL8jhHWqOiFPUDBFtC9Kozp6Kg41nYy/Og8AlLUA cJi8EHe3qiIlV25YfJBnRpMjtjM1QRhXfIexml0BtzgSKJo2Vo8sriUxaAxhPTV5wI6RXiso rXEKh1jBuZevt2iSojtQC2RDDa1pkQFJTWVetg04DRlpiFhumhl4W8U9jLeT ----------dzcqosdgxbilddyxnpkr-- From lartc at manchotnetworks.net Fri Jul 1 10:22:57 2005 From: lartc at manchotnetworks.net (lartc) Date: Fri Jul 1 10:23:10 2005 Subject: [LARTC] routing between 2 lines problem , after starting squid In-Reply-To: <48581776050630083558054672@mail.gmail.com> References: <48581776050630083558054672@mail.gmail.com> Message-ID: <1120206177.3871.7.camel@drs0.manchotnetworks.net> hi stanislav, i am really busy, but i can comment that i think your problem is coming from locally generated packets -- squid intercepts your web traffic, checks it local store, and then recreates the http get and sends it off. the local routing table is consulted, but i have bad luck in the past getting it work like you want. inside the squid.conf: # acl normal_service_net src 10.0.0.0/255.255.255.0 # acl good_service_net src 10.0.1.0/255.255.255.0 # tcp_outgoing_address 10.0.0.1 normal_service_net # tcp_outgoing_address 10.0.0.2 good_service_net # tcp_outgoing_address 10.0.0.3 you can see that it is possible to setup an acl and/or select the outgoing address (and bypass/fool the local routing table). as you are marking packets, and if you want to be very granular, you should probably run two instances of squid. each instance needs it own store -- do not use the same cache directory. you can then send packets to the correct squid instance in PREROUTING (each instance listens on a different port). hth cheers charles On Thu, 2005-06-30 at 17:35 +0200, Stanislav Nedelchev wrote: > i'm using one line on eth2 only for web traffic > eth1 is my internal line and eth0 is my main line to internet . > i'm marking packets like this > > i have default route on eth0 > > iptables -t mangle -A PREROUTING -i eth1 -p tcp --dport 80 -j MARK > --set-mark 66 > iptables -t mangle -A PREROUTING -i eth1 -p tcp --sport 80 -j MARK > --set-mark 66 > iptables -t mangle -A PREROUTING -i eth1 -p tcp --dport 3128 -j MARK > --set-mark 66 > iptables -t mangle -A PREROUTING -i eth1 -p tcp --sport 3128 -j MARK > --set-mark 66 > > iptables -t mangle -A FORWARD -p tcp --sport 80 -j MARK --set-mark 66 > iptables -t mangle -A FORWARD -p tcp --dport 80 -j MARK --set-mark 66 > iptables -t mangle -A FORWARD -p tcp --sport 3128 -j MARK --set-mark 66 > iptables -t mangle -A FORWARD -p tcp --dport 3128 -j MARK --set-mark 66 > > > iptables -t nat -A POSTROUTING -o eth2 -p tcp --dport 80 -s > 192.168.0.0/24 -d ! 192.168.0.0/16 -j MASQUERADE > iptables -t nat -A POSTROUTING -o eth2 -p tcp --dport 3128 -s > 192.168.0.0/24 -d ! 192.168.0.0/16 -j MASQUERADE > > i have also > /sbin/ip route add 192.168.0.0/24 dev eth1 table natips > /sbin/ip route add 127.0.0.0/8 dev lo scope link table natips > /sbin/ip route add default via 217.10.248.1 dev eth2 table natips > /sbin/ip route flush cache > /sbin/ip rule add fwmark 66 table natips > > > squid is running > on 192.168.0.1:3128 > > without squid it's working i'm using second line for web traffic > with squid it's not working > > can anybody help me > > Thanks in advance. > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc -- "simplified chinese" is not nearly as easy as they would have you believe ... a superlative oxymoron" --anonymous From chainshun at gmail.com Fri Jul 1 11:48:02 2005 From: chainshun at gmail.com (shun chain) Date: Fri Jul 1 11:48:08 2005 Subject: [LARTC] How to set HTB correct ? Message-ID: Hi, everyone I has a ADSL link about 4M bandwidth, and use HTB do bandwidth management. When I config qdisc like below: tc qdisc add dev br0 root handle 1: htb default 30 tc class add dev br0 parent 1: classid 1:1 htb rate 4000kbit tc class add dev br0 parent 1:1 classid 1:10 htb rate 2000kbit ceil 4000kbit prio 0 tc class add dev br0 parent 1:1 classid 1:20 htb rate 1000kbit ceil 4000kbit prio 1 tc class add dev br0 parent 1:1 classid 1:30 htb rate 500kbit ceil 4000kbit prio 2 tc class add dev br0 parent 1:1 classid 1:40 htb rate 100kbit ceil 4000kbit prio 3 tc qdisc add dev br0 parent 1:10 handle 10: pfifo limit 10 tc qdisc add dev br0 parent 1:20 handle 20: pfifo limit 10 tc qdisc add dev br0 parent 1:30 handle 30: pfifo limit 10 tc qdisc add dev br0 parent 1:40 handle 40: pfifo limit 10 after seconds, all 4 class will reach about 1M bps, it seems like HTB not work. But if I set qdisc like below: tc qdisc add dev br0 root handle 1: htb default 30 tc class add dev br0 parent 1: classid 1:1 htb rate 2000kbit tc class add dev br0 parent 1:1 classid 1:10 htb rate 2000kbit ceil 4000kbit prio 0 tc class add dev br0 parent 1:1 classid 1:20 htb rate 1000kbit ceil 4000kbit prio 1 tc class add dev br0 parent 1:1 classid 1:30 htb rate 500kbit ceil 4000kbit prio 2 tc class add dev br0 parent 1:1 classid 1:40 htb rate 100kbit ceil 4000kbit prio 3 tc qdisc add dev br0 parent 1:10 handle 10: pfifo limit 10 tc qdisc add dev br0 parent 1:20 handle 20: pfifo limit 10 tc qdisc add dev br0 parent 1:30 handle 30: pfifo limit 10 tc qdisc add dev br0 parent 1:40 handle 40: pfifo limit 10 then bandwidth will be split fit for 2:1: 0.5: 0.1, class 1:10 will get about 2.5 M. it seems like that HTB works well. I do not understand how it works like this. Thank you for your help in advance. Shun Chain From antonio.pinizzotto at iit.cnr.it Fri Jul 1 11:59:08 2005 From: antonio.pinizzotto at iit.cnr.it (Antonio Pinizzotto) Date: Fri Jul 1 11:59:40 2005 Subject: [LARTC] linux bridging problem: how to emulate 2 separate interfaces on a single one? Message-ID: <42C513EC.6030505@iit.cnr.it> Hi all. I would need to use a single physical interface on a Linux box to manage two different IP addresses (belonging to two separate logical subnets) with two different MAC address. (I need to emulate te presence of two separate physical interfaceses for a PC on the same LAN, at layer 2 and layer 3). Is it possible to use the linux ethernet bridging code and two tap interfaces for this purpose? Here is a picture of what I need to do. LAN__________________________________ | | eth0 (no IP) ________|________ | | | br0 | | (no IP) | |_________________| tap0 | | tap1 192.168.40.1/24 | | 192.168.30.1/24 MAC_0 | | MAC_1 __|________|__ | | | PC | | | |______________| I would need this picture to be equivalent to this one: LAN__________________________________ eth0 | | eth1 192.168.40.1/24 | | 192.168.30.1/24 MAC_0 | | MAC_1 __|________|__ | | | PC | | | |______________| I tried but it doens't work. I don't know if it's not contemplate or if I did a wrong configuration. For example, on tap0, using tcpdump, I see all the L2 broadcast frames going on the LAN; but on the counters of "ifconfig tap0" all of them are considered dropped. No packet is counted as transmitted or received, also if I try to ping from or to tap0. The tap interface starts to transmit/receive pachets only if it is connected to some process like qemu or openvpn. Maybe, it is not possibile for tap interface to be used directly by the linux os? Thanks! Antonio From MKrauss at hitchhiker.com Fri Jul 1 16:55:15 2005 From: MKrauss at hitchhiker.com (Matthias Krauss) Date: Fri Jul 1 16:55:32 2005 Subject: [LARTC] split access & fwmark Message-ID: Hi, I had a nice router which was able to split my outgoing webtraffic over a dsl line, then I got a power failure and now my settings are lost and I don't get it to run. I've configured a iptable rule like: /sbin/iptables -A PREROUTING -t mangle -i eth1 -s 192.168.1.10 -p TCP --dport 80 -j MARK --set-mark 1 and ip rule add fwmark 1 lookup 10 ip route add default via 192.168.2.2 dev eth3 then I have this kernel parameters: echo "1" > /proc/sys/net/ipv4/conf/eth3/accept_redirects echo "1" > /proc/sys/net/ipv4/conf/eth3/accept_source_route I remember that I had to use those parameter but this doesn't work, interesting is that I can successfully configure: ip rule from 192.168.1.5 lookup 10 Any help would be great. Matt. From wmarques at vmlinuz.com.br Fri Jul 1 17:48:38 2005 From: wmarques at vmlinuz.com.br (William Marques) Date: Fri Jul 1 17:43:58 2005 Subject: [LARTC] HTB: shaping internet bandwidth but don't shape local network traffic In-Reply-To: <200506301948.42383.Andreas.Klauer@metamorpher.de> References: <200506301948.42383.Andreas.Klauer@metamorpher.de> Message-ID: <42C565D6.3040901@vmlinuz.com.br> Andreas Klauer escreveu: >On Thursday 30 June 2005 19:26, Jefri Lie wrote: > > >>i got this problem, i want to shape my clients internet bw, but i >>don't want to shape my local network traffic. For information, my >>clients using wireless to connect to my router[192.168.1.254]. >> >> > >Common mistake is to use the internet class as root class on the LAN >interface, which means everything (local traffic and internet traffic) >gets shaped to internet class speed. > >I solved it like this: > >HTB Qdisc >| >\--- HTB root class (10MBit) > | > \--- HTB internet class (1Mbit) > | | > | \--- ... more classes for internet shaping ... > | > \--- HTB lan class (10-1=9Mbit) > >This would be an example setup for a 10Mbit ethernet card, with a 1MBit >internet connection, and local connections get what's left. You put your >internet traffic into the internet class (or one of it's children), and >everything else (local traffic) into the lan class. > >HTH >Andreas >_______________________________________________ >LARTC mailing list >LARTC@mailman.ds9a.nl >http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > > This is exactly what Im trying to do, without success.... I'm almost giving up. I made a little test script, follows: tc qdisc add dev eth0 root handle 1: htb default 22 # The LAN class: tc class add dev eth0 parent 1:0 classid 1:1 htb rate 90mbit tc class add dev eth0 parent 1:1 classid 1:10 htb rate 89232kbit ceil 90mbit # The internet class: tc class add dev eth0 parent 1:0 classid 1:2 htb rate 768kbit tc class add dev eth0 parent 1:2 classid 1:20 htb rate 384kbit ceil 768kbit tc class add dev eth0 parent 1:2 classid 1:21 htb rate 256kbit ceil 768kbit tc class add dev eth0 parent 1:2 classid 1:22 htb rate 128kbit ceil 768kbit # filter for the LAN: tc filter add dev eth0 protocol ip parent 1:0 prio 1 handle 10 fw classid 1:10 #Filter for internet: tc filter add dev eth0 protocol ip parent 1:0 prio 2 handle 11 fw classid 1:20 tc filter add dev eth0 protocol ip parent 1:0 prio 3 handle 12 fw classid 1:21 tc filter add dev eth0 protocol ip parent 1:0 prio 4 handle 13 fw classid 1:22 # iptables mark: iptables -t mangle -A PREROUTING -p tcp -s 192.168.1.0/24 -d 192.168.1.254 --dport 3128 -j MARK --set-mark 10 iptables -t mangle -A PREROUTING -p tcp -s 192.168.1.0/24 -d 192.168.1.254 --dport 445 -j MARK --set-mark 10 iptables -t mangle -A PREROUTING -p tcp -s 192.168.1.0/24 -d 192.168.1.254 --dport 139 -j MARK --set-mark 10 iptables -t mangle -A PREROUTING -p tcp -s 192.168.1.0/24 -d 0/0 --dport 143 -j MARK --set-mark 11 iptables -t mangle -A PREROUTING -p tcp -s 192.168.1.0/24 -d 0/0 --dport 80 -j MARK --set-mark 11 iptables -t mangle -A PREROUTING -p tcp -s 192.168.1.0/24 -d 0/0 --dport 22 -j MARK --set-mark 11 iptables -t mangle -A PREROUTING -p tcp -s 192.168.1.0/24 -d 0/0 --dport 44 -j MARK --set-mark 11 iptables -t mangle -A PREROUTING -p tcp -s 192.168.1.0/24 -d 0/0 --dport 443 -j MARK --set-mark 12 iptables -t mangle -A PREROUTING -p tcp -s 192.168.1.0/24 -d 0/0 --dport 110 -j MARK --set-mark 12 iptables -t mangle -A PREROUTING -p tcp -s 192.168.1.0/24 -j MARK --set-mark 13 No error messages, everything fine. Analysis: [root@samba htb]# tc filter show dev eth0 filter parent 1: protocol ip pref 1 fw filter parent 1: protocol ip pref 1 fw handle 0xa classid 1:10 filter parent 1: protocol ip pref 2 fw filter parent 1: protocol ip pref 2 fw handle 0xb classid 1:20 filter parent 1: protocol ip pref 3 fw filter parent 1: protocol ip pref 3 fw handle 0xc classid 1:21 filter parent 1: protocol ip pref 4 fw filter parent 1: protocol ip pref 4 fw handle 0xd classid 1:22 [root@samba htb]# iptables -L -t mangle -v Chain PREROUTING (policy ACCEPT 565 packets, 134K bytes) pkts bytes target prot opt in out source destination 0 0 MARK tcp -- any any 192.168.1.0/24 ns1.intranet.com.br tcp dpt:3128 MARK set 0xa 0 0 MARK tcp -- any any 192.168.1.0/24 ns1.intranet.com.br tcp dpt:microsoft-ds MARK set 0xa 0 0 MARK tcp -- any any 192.168.1.0/24 ns1.intranet.com.br tcp dpt:netbios-ssn MARK set 0xa 29 1821 MARK tcp -- any any 192.168.1.0/24 anywhere tcp dpt:imap2 MARK set 0xb 93 30816 MARK tcp -- any any 192.168.1.0/24 anywhere tcp dpt:www-http MARK set 0xb 101 7652 MARK tcp -- any any 192.168.1.0/24 anywhere tcp dpt:ssh MARK set 0xb 0 0 MARK tcp -- any any 192.168.1.0/24 anywhere tcp dpt:mpm-flags MARK set 0xb 1 44 MARK tcp -- any any 192.168.1.0/24 anywhere tcp dpt:https MARK set 0xc 44 2516 MARK tcp -- any any 192.168.1.0/24 anywhere tcp dpt:pop-3 MARK set 0xc 266 41637 MARK tcp -- any any 192.168.1.0/24 anywhere MARK set 0xd Chain INPUT (policy ACCEPT 165 packets, 12979 bytes) pkts bytes target prot opt in out source destination Chain FORWARD (policy ACCEPT 400 packets, 121K bytes) pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 98 packets, 12143 bytes) pkts bytes target prot opt in out source destination Chain POSTROUTING (policy ACCEPT 498 packets, 133K bytes) pkts bytes target prot opt in out source destination [root@samba htb]# iptables -L -t mangle -v Chain PREROUTING (policy ACCEPT 2979 packets, 303K bytes) pkts bytes target prot opt in out source destination 2107 110K MARK tcp -- any any 192.168.1.0/24 ns1.intranet.com.br tcp dpt:3128 MARK set 0xa 0 0 MARK tcp -- any any 192.168.1.0/24 ns1.intranet.com.br tcp dpt:microsoft-ds MARK set 0xa 21 3733 MARK tcp -- any any 192.168.1.0/24 ns1.intranet.com.br tcp dpt:netbios-ssn MARK set 0xa 29 1821 MARK tcp -- any any 192.168.1.0/24 anywhere tcp dpt:imap2 MARK set 0xb 179 47088 MARK tcp -- any any 192.168.1.0/24 anywhere tcp dpt:www-http MARK set 0xb 121 8932 MARK tcp -- any any 192.168.1.0/24 anywhere tcp dpt:ssh MARK set 0xb 0 0 MARK tcp -- any any 192.168.1.0/24 anywhere tcp dpt:mpm-flags MARK set 0xb 3 176 MARK tcp -- any any 192.168.1.0/24 anywhere tcp dpt:https MARK set 0xc 44 2516 MARK tcp -- any any 192.168.1.0/24 anywhere tcp dpt:pop-3 MARK set 0xc 2518 176K MARK tcp -- any any 192.168.1.0/24 anywhere MARK set 0xd Chain INPUT (policy ACCEPT 2389 packets, 138K bytes) pkts bytes target prot opt in out source destination Chain FORWARD (policy ACCEPT 590 packets, 165K bytes) pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 4390 packets, 6339K bytes) pkts bytes target prot opt in out source destination Chain POSTROUTING (policy ACCEPT 4980 packets, 6504K bytes) pkts bytes target prot opt in out source destination A configured Apache to listen on port 3128, to make tests, and in a machine in my Lan I do: [william@whsm tmp]$ time wget samba.intranet.com.br:3128/768.txt --12:34:22-- http://samba.intranet.com.br:3128/768.txt => `768.txt' Resolving samba.intranet.com.br... 192.168.1.254 Connecting to samba.intranet.com.br[192.168.1.254]:3128... connected. Requisi??o enviada ao servidor HTTP, esperando resposta... 200 OK Tamanho: 7,864,320 [text/plain] 100%[================================================================================================================>] 7,864,320 91.59K/s ETA 00:00 12:35:46 (91.70 KB/s) - `768.txt' recebido [7864320/7864320] real 1m23.817s user 0m0.080s sys 0m0.204s As you can see, the download speed is 92KBps, the 768kbits in my script. but why the class 1:10 who haves garanteedrate 89232kbit wasn't used ? the fw mark is working, I see the traffic going correctly, but the correct filter is not applyed??! Maybe some of you can give me a hand, I'm sure that something is misconfigured, but I can't figure out where. I'm new in htb, and I read the LARTC how to, and search google for a solution, but in vain, this is getting me crazy!! Help me!! Regards, -- William Henrique Siqueira Marques wmarques@vmlinuz.com.br Rio de Janeiro - Brasil From coricim at gmail.com Fri Jul 1 18:43:42 2005 From: coricim at gmail.com (Marius Corici) Date: Fri Jul 1 18:43:49 2005 Subject: [LARTC] tc on a PDA In-Reply-To: <42B974EF.A95C3D1F@iswest.com> References: <2abc333505062204133d60ea74@mail.gmail.com> <42B974EF.A95C3D1F@iswest.com> Message-ID: <2abc33350507010943fba3dff@mail.gmail.com> I just want to tell that i was able to recompile the pda's kernel succesfully for traffic shaping. Another problem appeared, but i solved it: the cbq command i sent before needs a handle, i don't know why ... So in conclusion on a 5550 hp iPAQ with Familiar 0.8.2 distribution and 2.4.19 ARM patched kernel the shaping works, with the exception of the efficient and simply to configure HTB qdisc. Marius On 6/22/05, gypsy wrote: > Marius Corici wrote: > > > > hello, > > > > i have a 5500 iPaq with familiar 0.8.2 on it and i don't know how to > > install a traffic shaper tool on it. > > I installed iproute2 package i found on familiar site and for that i > > have the tc and ip commands, but when trying to add a qdisc, as a > > dummy example, i get the next answer: > > > > > tc qdisc add dev wlan0 root cbq bandwidth 100Mbit avpkt 100 > > RTNETLINK answers: Invalid argument > > This suggests no cbq, so you probably need to recompile the kernel. If > you had to install iproute2 then almost certainly the kernel was not > built for QoS. > > > do i have to install some other packages, recompile the kernel or > > something else? > > > > thans in advance, > > > > Marius Corici > > Curiosity: What makes you think shaping is needed on a PDA? > -- > gypsy > From shemminger at osdl.org Fri Jul 1 19:12:18 2005 From: shemminger at osdl.org (Stephen Hemminger) Date: Fri Jul 1 19:12:31 2005 Subject: [LARTC] linux bridging problem: how to emulate 2 separate interfaces on a single one? In-Reply-To: <42C513EC.6030505@iit.cnr.it> References: <42C513EC.6030505@iit.cnr.it> Message-ID: <20050701101218.1b4fc97a@dxpl.pdx.osdl.net> On Fri, 01 Jul 2005 11:59:08 +0200 Antonio Pinizzotto wrote: > > Hi all. > > I would need to use a single physical interface on a Linux box to manage > two different IP addresses (belonging to two separate logical subnets) > with two different MAC address. (I need to emulate te presence of two > separate physical interfaceses for a PC on the same LAN, at layer 2 and > layer 3). > > Is it possible to use the linux ethernet bridging code and two tap > interfaces for this purpose? > > > Here is a picture of what I need to do. > > > LAN__________________________________ > | > | eth0 (no IP) > ________|________ > | | > | br0 | > | (no IP) | > |_________________| > tap0 | | tap1 > 192.168.40.1/24 | | 192.168.30.1/24 > MAC_0 | | MAC_1 > __|________|__ > | | > | PC | > | | > |______________| > > > I would need this picture to be equivalent to this one: > > > LAN__________________________________ > eth0 | | eth1 > 192.168.40.1/24 | | 192.168.30.1/24 > MAC_0 | | MAC_1 > __|________|__ > | | > | PC | > | | > |______________| > > > I tried but it doens't work. I don't know if it's not contemplate or if > I did a wrong configuration. > For example, on tap0, using tcpdump, I see all the L2 broadcast frames > going on the LAN; but on the counters of "ifconfig tap0" all of them are > considered dropped. No packet is counted as transmitted or received, > also if I try to ping from or to tap0. > The tap interface starts to transmit/receive pachets only if it is > connected to some process like qemu or openvpn. > Maybe, it is not possibile for tap interface to be used directly by the > linux os? > Stop abusing bridging. People keep trying these wacky multiple network things and they won't work. Bridging really can't handle anything with multiple interfaces that could possibly create a forwarding loop. From Andreas.Klauer at metamorpher.de Fri Jul 1 19:22:34 2005 From: Andreas.Klauer at metamorpher.de (Andreas Klauer) Date: Fri Jul 1 19:22:33 2005 Subject: [LARTC] HTB: shaping internet bandwidth but don't shape local network traffic In-Reply-To: <42C565D6.3040901@vmlinuz.com.br> References: <200506301948.42383.Andreas.Klauer@metamorpher.de> <42C565D6.3040901@vmlinuz.com.br> Message-ID: <200507011922.34141.Andreas.Klauer@metamorpher.de> On Friday 01 July 2005 17:48, William Marques wrote: > tc class add dev eth0 parent 1:0 classid 1:1 htb rate 90mbit > tc class add dev eth0 parent 1:0 classid 1:2 htb rate 768kbit I don't know if it makes any difference, but I only use one root class and have these two as children to that root class. This way, I find it easier to make sure that the total rate never exceeds interface capability. > As you can see, the download speed is 92KBps, the 768kbits in my script. > but why the class 1:10 who haves garanteedrate 89232kbit wasn't used ? > the fw mark is working, I see the traffic going correctly, but the > correct filter is not applyed??! It's just a quick guess, but probably your marking rules overwrite each other in a way you didn't intend them to. Add a log target at the end of the script, and check if packets actually get really marked correctly. HTH Andreas From antonio.pinizzotto at iit.cnr.it Fri Jul 1 20:37:44 2005 From: antonio.pinizzotto at iit.cnr.it (Antonio Pinizzotto) Date: Fri Jul 1 20:39:07 2005 Subject: [LARTC] linux bridging problem: how to emulate 2 separate interfaces on a single one? In-Reply-To: <20050701101218.1b4fc97a@dxpl.pdx.osdl.net> References: <42C513EC.6030505@iit.cnr.it> <20050701101218.1b4fc97a@dxpl.pdx.osdl.net> Message-ID: <42C58D78.9010701@iit.cnr.it> Stephen Hemminger wrote: > On Fri, 01 Jul 2005 11:59:08 +0200 > Antonio Pinizzotto wrote: > > >>Hi all. >> >>I would need to use a single physical interface on a Linux box to manage >>two different IP addresses (belonging to two separate logical subnets) >>with two different MAC address. (I need to emulate te presence of two >>separate physical interfaceses for a PC on the same LAN, at layer 2 and >>layer 3). >> >>Is it possible to use the linux ethernet bridging code and two tap >>interfaces for this purpose? >> >> >>Here is a picture of what I need to do. >> >> >> LAN__________________________________ >> | >> | eth0 (no IP) >> ________|________ >> | | >> | br0 | >> | (no IP) | >> |_________________| >> tap0 | | tap1 >>192.168.40.1/24 | | 192.168.30.1/24 >> MAC_0 | | MAC_1 >> __|________|__ >> | | >> | PC | >> | | >> |______________| >> >> >>I would need this picture to be equivalent to this one: >> >> >> LAN__________________________________ >> eth0 | | eth1 >>192.168.40.1/24 | | 192.168.30.1/24 >> MAC_0 | | MAC_1 >> __|________|__ >> | | >> | PC | >> | | >> |______________| >> >> >>I tried but it doens't work. I don't know if it's not contemplate or if >>I did a wrong configuration. >>For example, on tap0, using tcpdump, I see all the L2 broadcast frames >>going on the LAN; but on the counters of "ifconfig tap0" all of them are >>considered dropped. No packet is counted as transmitted or received, >>also if I try to ping from or to tap0. >>The tap interface starts to transmit/receive pachets only if it is >>connected to some process like qemu or openvpn. >>Maybe, it is not possibile for tap interface to be used directly by the >>linux os? >> > > > > Stop abusing bridging. People keep trying these wacky multiple > network things and they won't work. Bridging really can't handle anything > with multiple interfaces that could possibly create a forwarding loop. > > > Dear Stephen, probably my pictures or descriptions were not so clear. I don't want abuse bridging. As you can see in the second picture (of the previous e-mail) I just would like to replicate the real topology (in which there are not loops because each physical interface has its own subnet) using software interfaces, taps. Moreover in the second picture no bridging is enabled on the PC. It is just an experiment to understand how the various software pieces works togheter. Below I tried to draw a clearer picture. The first one is what I would like to get. The second one is what I have. LAN______________________________________________ | | eth0 (no IP) _____________________|________________________ | | Linux Box (PC)| | ________|________ | | | | | | | br0 | | | | (no IP) | | | |_________________| | | tap0 | | tap1 | | 192.168.40.1/24 | | 192.168.30.1/24 | | MAC_0 | | MAC_1 | | ___|________|____ | | | | | | | OS TCP-IP stack | | | | | | | |_________________| | | | | | | | | | | | | | | Processes | | | | | |______________________________________________| I would need the first picture to be equivalent to the next one: LAN_____________________________________________ eth0 | | eth1 192.168.40.1/24 | | 192.168.30.1/24 MAC_0 | | MAC_1 _________________|________|___________________ | | | Linux Box (PC)| | ___|________|____ | | | | | | | OS TCP-IP stack | | | | | | | |_________________| | | | | | | | | | | | | | | Processes | | | | | |______________________________________________| Bye Antonio From shemminger at osdl.org Fri Jul 1 21:28:19 2005 From: shemminger at osdl.org (Stephen Hemminger) Date: Fri Jul 1 21:28:36 2005 Subject: [LARTC] linux bridging problem: how to emulate 2 separate interfaces on a single one? In-Reply-To: <42C58D78.9010701@iit.cnr.it> References: <42C513EC.6030505@iit.cnr.it> <20050701101218.1b4fc97a@dxpl.pdx.osdl.net> <42C58D78.9010701@iit.cnr.it> Message-ID: <20050701122819.7488be8c@dxpl.pdx.osdl.net> On Fri, 01 Jul 2005 20:37:44 +0200 Antonio Pinizzotto wrote: > Stephen Hemminger wrote: > > On Fri, 01 Jul 2005 11:59:08 +0200 > > Antonio Pinizzotto wrote: > > > > > >>Hi all. > >> > >>I would need to use a single physical interface on a Linux box to manage > >>two different IP addresses (belonging to two separate logical subnets) > >>with two different MAC address. (I need to emulate te presence of two > >>separate physical interfaceses for a PC on the same LAN, at layer 2 and > >>layer 3). > >> > >>Is it possible to use the linux ethernet bridging code and two tap > >>interfaces for this purpose? > >> ... > Below I tried to draw a clearer picture. > The first one is what I would like to get. > The second one is what I have. > > > LAN______________________________________________ > | > | eth0 (no IP) > _____________________|________________________ > | | Linux Box (PC)| > | ________|________ | > | | | | > | | br0 | | > | | (no IP) | | > | |_________________| | > | tap0 | | tap1 | > | 192.168.40.1/24 | | 192.168.30.1/24 | > | MAC_0 | | MAC_1 | > | ___|________|____ | > | | | | > | | OS TCP-IP stack | | > | | | | > | |_________________| | > | | | | | | > | | | | | | > | Processes | > | | > | | > |______________________________________________| > > > I would need the first picture to be equivalent to the next one: > > > LAN_____________________________________________ > eth0 | | eth1 > 192.168.40.1/24 | | 192.168.30.1/24 > MAC_0 | | MAC_1 > _________________|________|___________________ > | | | Linux Box (PC)| > | ___|________|____ | > | | | | > | | OS TCP-IP stack | | > | | | | > | |_________________| | > | | | | | | > | | | | | | > | Processes | > | | > | | > |______________________________________________| It should work if the tap interface looks sufficiently like Ethernet. You probably need filter rules to make sure and drop packets intended for the other network get dropped and to prevent broadcast leakage. Wouldn't a cheap ethernet hub and two ethernet interfaces do the same thing? From alexeyt at freeshell.org Fri Jul 1 21:52:29 2005 From: alexeyt at freeshell.org (Alexey Toptygin) Date: Fri Jul 1 21:53:22 2005 Subject: [LARTC] linux bridging problem: how to emulate 2 separate interfaces on a single one? In-Reply-To: <20050701122819.7488be8c@dxpl.pdx.osdl.net> References: <42C513EC.6030505@iit.cnr.it> <20050701101218.1b4fc97a@dxpl.pdx.osdl.net> <42C58D78.9010701@iit.cnr.it> <20050701122819.7488be8c@dxpl.pdx.osdl.net> Message-ID: On Fri, 1 Jul 2005, Stephen Hemminger wrote: >> LAN______________________________________________ >> | >> | eth0 (no IP) >> _____________________|________________________ >> | | Linux Box (PC)| >> | ________|________ | >> | | | | >> | | br0 | | >> | | (no IP) | | >> | |_________________| | >> | tap0 | | tap1 | >> | 192.168.40.1/24 | | 192.168.30.1/24 | >> | MAC_0 | | MAC_1 | >> | ___|________|____ | >> | | | | >> | | OS TCP-IP stack | | >> | | | | >> | |_________________| | >> | | | | | | >> | | | | | | >> | Processes | >> | | >> | | >> |______________________________________________| [snip] > It should work if the tap interface looks sufficiently like Ethernet. > You probably need filter rules to make sure and drop packets intended for > the other network get dropped and to prevent broadcast leakage. The way I interpret the drawing, ISTM that Antonio has the bridge a layer below the tap devices (even though it's drawn a layer above). I don't think that's a very sane idea... The crux of the problem seems to be that Antonio wants a single physical ethernet card to use two different MAC adresses, which I don't think briding is ever going to solve. This is, perhaps, possible by putting the card into promiscuous mode, and using some clever ebtables mangling. Alexey From gypsy at iswest.com Sat Jul 2 03:12:31 2005 From: gypsy at iswest.com (gypsy) Date: Sat Jul 2 03:10:24 2005 Subject: [LARTC] tc on a PDA References: <2abc333505062204133d60ea74@mail.gmail.com> <42B974EF.A95C3D1F@iswest.com> <2abc33350507010943fba3dff@mail.gmail.com> Message-ID: <42C5E9FF.1275F0AB@iswest.com> Marius Corici wrote: > > I just want to tell that i was able to recompile the pda's kernel > succesfully for traffic shaping. Another problem appeared, but i > solved it: the cbq command i sent before needs a handle, i don't know > why ... > So in conclusion on a 5550 hp iPAQ with Familiar 0.8.2 distribution > and 2.4.19 ARM patched kernel the shaping works, with the exception of > the efficient and simply to configure HTB qdisc. > > Marius AWESOME! Why no HTB? -- gypsy > On 6/22/05, gypsy wrote: > > Marius Corici wrote: > > > > > > hello, > > > > > > i have a 5500 iPaq with familiar 0.8.2 on it and i don't know how to > > > install a traffic shaper tool on it. > > > I installed iproute2 package i found on familiar site and for that i > > > have the tc and ip commands, but when trying to add a qdisc, as a > > > dummy example, i get the next answer: > > > > > > > tc qdisc add dev wlan0 root cbq bandwidth 100Mbit avpkt 100 > > > RTNETLINK answers: Invalid argument > > > > This suggests no cbq, so you probably need to recompile the kernel. If > > you had to install iproute2 then almost certainly the kernel was not > > built for QoS. > > > > > do i have to install some other packages, recompile the kernel or > > > something else? > > > > > > thans in advance, > > > > > > Marius Corici > > > > Curiosity: What makes you think shaping is needed on a PDA? > > -- > > gypsy From robb.bossley at gmail.com Sat Jul 2 05:20:21 2005 From: robb.bossley at gmail.com (Robb Bossley) Date: Sat Jul 2 05:20:25 2005 Subject: Fwd: [LARTC] linux bridging problem: how to emulate 2 separate interfaces on a single one? In-Reply-To: <5c685153050701201559f066fe@mail.gmail.com> References: <42C513EC.6030505@iit.cnr.it> <20050701101218.1b4fc97a@dxpl.pdx.osdl.net> <42C58D78.9010701@iit.cnr.it> <20050701122819.7488be8c@dxpl.pdx.osdl.net> <5c685153050701201559f066fe@mail.gmail.com> Message-ID: <5c68515305070120201aab04ac@mail.gmail.com> Sorry Alexey, I meant to send that to the list, not just you. ---------- Forwarded message ---------- From: Robb Bossley Date: Jul 1, 2005 11:15 PM Subject: Re: [LARTC] linux bridging problem: how to emulate 2 separate interfaces on a single one? To: Alexey Toptygin Yes, it can be done. I know because I've done it before. (The only issue is what you want to do with it.) First, if you are using DHCP to set up the interface ip, eth0 will take the first address assigned by DHCP. Then you need to create the bridge after bringing eth0 up with no ip. Then add eth0 to the bridge. Then you can create and add the tun devices that the taps will come from. When you add the taps, you must specify different MAC addresses, or I believe they may default to the same one - perhaps even the same as the MAC on your NIC. When I did this, I had problems because although I had three different MAC addresses, the address for eth0 was still in force (even though I had freed the address, the DHCP server had a very long time before it would reset the ip). This was a problem because my ISP only would give me a maximum of three ip's at once. (I needed all three for my little experiment to work) Why do this? Because I can! That's the beauty of Linux - freedom to do whatever you want, even if it doesn't make sense. If you would like, I can attach a little script that I wrote that sets up the bridge. Robb On 7/1/05, Alexey Toptygin wrote: > On Fri, 1 Jul 2005, Stephen Hemminger wrote: > > >> LAN______________________________________________ > >> | > >> | eth0 (no IP) > >> _____________________|________________________ > >> | | Linux Box (PC)| > >> | ________|________ | > >> | | | | > >> | | br0 | | > >> | | (no IP) | | > >> | |_________________| | > >> | tap0 | | tap1 | > >> | 192.168.40.1/24 | | 192.168.30.1/24 | > >> | MAC_0 | | MAC_1 | > >> | ___|________|____ | > >> | | | | > >> | | OS TCP-IP stack | | > >> | | | | > >> | |_________________| | > >> | | | | | | > >> | | | | | | > >> | Processes | > >> | | > >> | | > >> |______________________________________________| > > [snip] > > > It should work if the tap interface looks sufficiently like Ethernet. > > You probably need filter rules to make sure and drop packets intended for > > the other network get dropped and to prevent broadcast leakage. > > The way I interpret the drawing, ISTM that Antonio has the bridge a layer > below the tap devices (even though it's drawn a layer above). I don't > think that's a very sane idea... > > The crux of the problem seems to be that Antonio wants a single physical > ethernet card to use two different MAC adresses, which I don't think > briding is ever going to solve. This is, perhaps, possible by putting the > card into promiscuous mode, and using some clever ebtables mangling. > > Alexey > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > -- As if you could kill time without injuring eternity. The mass of men live lives of quiet desperation. - Henry David Thoreau -- As if you could kill time without injuring eternity. The mass of men live lives of quiet desperation. - Henry David Thoreau From rlow at inter-touch.com Sat Jul 2 17:13:16 2005 From: rlow at inter-touch.com (Richmond Low) Date: Sat Jul 2 17:14:18 2005 Subject: [LARTC] Bandwidth shaping script appears to incorrectly halve the bandwidth Message-ID: Hi, I'm trying to modify the Wondershaper script so achieve the following. 1. Prioritize traffic to the few IP addresses. 2. Everyone else should get a fair amount of bandwidth The problem I am facing is when I run this script is that the bandwidth seems to get halved. I tried to apply this script to a site where the upstream and downstream were fully saturated. I immediately saw backlogs in the 'everyone else' class but it seems like they were getting only 256 down and 256 up. I thought perhaps the the class was combining the upstream and downstream traffic but when I increased the INCOMING_BW and OUTGOING_BW to 1024, I didn't see any increase. There still seemed to be only around 256kbps of traffic coming in and around 256kbps of traffic going out. Any help on this would be much appreciated. #!/bin/bash # Set the incoming and outgoing bandwidth here. INCOMING_BW=512 OUTGOING_BW=512 UL_DEV=eth0 DL_DEV=eth1 # clean existing down- and uplink qdiscs, hide errors tc qdisc del dev $UL_DEV root 2> /dev/null > /dev/null tc qdisc del dev $DL_DEV root 2> /dev/null > /dev/null # install root CBQ echo Installing Root CBQ tc qdisc add dev $UL_DEV root handle 1: cbq avpkt 1000 bandwidth 100mbit tc qdisc add dev $DL_DEV root handle 2: cbq avpkt 1000 bandwidth 100mbit # shape everything on $UL_DEV at $OUTGOING_BW speed, $DL_DEV at $INCOMING_BW echo Creating Classes tc class add dev $UL_DEV parent 1: classid 1:1 cbq rate ${OUTGOING_BW}kbit allot 1500 prio 5 bounded isolated tc class add dev $DL_DEV parent 2: classid 2:1 cbq rate ${INCOMING_BW}kbit allot 1500 prio 5 bounded isolated tc class add dev $UL_DEV parent 1:1 classid 1:10 cbq rate 128kbit allot 1600 prio 1 avpkt 1000 tc class add dev $DL_DEV parent 2:1 classid 2:10 cbq rate 128kbit allot 1600 prio 1 avpkt 1000 tc class add dev $UL_DEV parent 1:1 classid 1:20 cbq rate 128kbit allot 1600 prio 2 avpkt 1000 bounded tc class add dev $DL_DEV parent 2:1 classid 2:20 cbq rate 128kbit allot 1600 prio 2 avpkt 1000 bounded tc class add dev $UL_DEV parent 1:1 classid 1:30 cbq rate 128kbit allot 1600 prio 3 avpkt 1000 bounded tc class add dev $DL_DEV parent 2:1 classid 2:30 cbq rate 128kbit allot 1600 prio 3 avpkt 1000 bounded tc class add dev $UL_DEV parent 1:1 classid 1:40 cbq rate 128kbit allot 1600 prio 3 avpkt 1000 bounded tc class add dev $DL_DEV parent 2:1 classid 2:40 cbq rate 128kbit allot 1600 prio 3 avpkt 1000 bounded tc class add dev $UL_DEV parent 1:1 classid 1:50 cbq rate 128kbit allot 1600 prio 3 avpkt 1000 bounded tc class add dev $DL_DEV parent 2:1 classid 2:50 cbq rate 128kbit allot 1600 prio 3 avpkt 1000 bounded # Low priority class for P2P and other traffic that we don't like tc class add dev $UL_DEV parent 1:1 classid 1:60 cbq rate $[1*$OUTGOING_BW/10]kbit allot 1600 prio 8 avpkt 1000 tc class add dev $DL_DEV parent 2:1 classid 2:60 cbq rate $[1*$OUTGOING_BW/10]kbit allot 1600 prio 8 avpkt 1000 # Everyone else goes into this tc class add dev $UL_DEV parent 1:1 classid 1:80 cbq rate $[9*$OUTGOING_BW/10]kbit allot 1600 prio 5 avpkt 1000 tc class add dev $DL_DEV parent 2:1 classid 2:80 cbq rate $[9*$INCOMING_BW/10]kbit allot 1600 prio 5 avpkt 1000 echo Classes Created # all get Stochastic Fairness: echo Adding SFQ to all classes tc qdisc add dev $UL_DEV parent 1:10 handle 10: sfq perturb 10 tc qdisc add dev $DL_DEV parent 2:10 handle 10: sfq perturb 10 tc qdisc add dev $UL_DEV parent 1:20 handle 20: sfq perturb 10 tc qdisc add dev $DL_DEV parent 2:20 handle 20: sfq perturb 10 tc qdisc add dev $UL_DEV parent 1:30 handle 30: sfq perturb 10 tc qdisc add dev $DL_DEV parent 2:30 handle 30: sfq perturb 10 tc qdisc add dev $UL_DEV parent 1:40 handle 40: sfq perturb 10 tc qdisc add dev $DL_DEV parent 2:40 handle 40: sfq perturb 10 tc qdisc add dev $UL_DEV parent 1:50 handle 50: sfq perturb 10 tc qdisc add dev $DL_DEV parent 2:50 handle 50: sfq perturb 10 tc qdisc add dev $UL_DEV parent 1:60 handle 60: sfq perturb 10 tc qdisc add dev $DL_DEV parent 2:60 handle 60: sfq perturb 10 tc qdisc add dev $UL_DEV parent 1:80 handle 80: sfq perturb 10 tc qdisc add dev $DL_DEV parent 2:80 handle 80: sfq perturb 10 # start filters tc filter add dev $UL_DEV parent 1: protocol ip prio 1 u32 match ip dst 192.168.1.1 flowid 1:10 tc filter add dev $UL_DEV parent 1: protocol ip prio 1 u32 match ip dst 192.168.1.2 flowid 1:10 tc filter add dev $UL_DEV parent 1: protocol ip prio 1 u32 match ip dst 192.168.1.3 flowid 1:10 tc filter add dev $UL_DEV parent 1: protocol ip prio 1 u32 match ip dst 192.168.1.4 flowid 1:10 # To speed up downloads while an upload is going on, put ACK packets in 1:10 as well tc filter add dev $UL_DEV parent 1: protocol ip prio 8 u32 \ match ip protocol 6 0xff \ match u8 0x05 0x0f at 0 \ match u16 0x0000 0xffc0 at 2 \ match u8 0x10 0xff at 33 \ flowid 1:10 # More filters lower priority tc filter add dev $UL_DEV parent 1: protocol ip prio 1 u32 match ip dst 192.168.2.1 flowid 1:20 tc filter add dev $UL_DEV parent 1: protocol ip prio 1 u32 match ip dst 192.168.2.2 flowid 1:20 tc filter add dev $UL_DEV parent 1: protocol ip prio 1 u32 match ip dst 192.168.2.3 flowid 1:20 tc filter add dev $UL_DEV parent 1: protocol ip prio 1 u32 match ip dst 192.168.2.4 flowid 1:20 tc filter add dev $UL_DEV parent 1: protocol ip prio 1 u32 match ip dst 192.168.2.5 flowid 1:20 tc filter add dev $UL_DEV parent 1: protocol ip prio 1 u32 match ip dst 192.168.2.6 flowid 1:20 # rest is 'non-interactive' ie 'bulk' and ends up in 1:80 tc filter add dev $UL_DEV parent 1: protocol ip prio 5 u32 match ip dst 0.0.0.0/0 flowid 1:80 tc filter add dev $DL_DEV parent 2: protocol ip prio 5 u32 match ip dst 0.0.0.0/0 flowid 2:80 From cvkhanh at gmail.com Sat Jul 2 17:40:05 2005 From: cvkhanh at gmail.com (Cao Van Khanh) Date: Sat Jul 2 17:40:15 2005 Subject: [LARTC] Loadbalancing how to ? ? ? ? Message-ID: <42c6b55a.2b2713fa.0308.ffffb3a1@mx.gmail.com> I have 2 ADSL ad1 and ad2 , one PC for my firewall and some deamon on it with 3 ethernet : eth0 connect to my LAN ( 192.168.60.0/24 ) and 2 other connect to ad1 and ad2 |eth1 (10.0.1.2)--------------------ad1 ( ADSL 1 ) | My LAN(192.168.60.0/24) |---------eth0( 192.168.60.2)--> PC | |eth2 (10.0.2.2)---------------------ad2 (ADSL 2 ) All computer in LAN has default router = 191.168.60.2 ( eth0 of PC ) In the /etc/network/option I enable the forwardable = 1 I use iptable to NAT the outgoing of eth1 and eth2 Iptables -t nat -A POSTROUTING -s 192.168.60.0/24 -o eth1 -j SNAT -to 10.0.1.2 Iptables -t nat -A POSTROUTING -s 192.168.60.0/24 -o eth2 -j SNAT -to 10.0.2.2 Ofcause by default the eth1 will always be forwarded from LAN and nerver the ADSL 2 was use The ideal of mine is writing a programe loadbalancing for n line ADSL contact to one PC as gateway ,But when the packet reach the eth0 , how to control it forwarld to eth1 or eth2 is my problem . If I could do that , maybe I could find the way to loadbalacing n line ADSL as one biger . If someone know how to or have some idea , give me some information :-) Thank for reading -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ds9a.nl/pipermail/lartc/attachments/20050702/8ca2e20f/attachment.htm From tdi at pozman.pl Sat Jul 2 17:56:35 2005 From: tdi at pozman.pl (Dariusz Dwornikowski) Date: Sat Jul 2 17:56:43 2005 Subject: [LARTC] Loadbalancing how to ? ? ? ? In-Reply-To: <42c6b55a.2b2713fa.0308.ffffb3a1@mx.gmail.com> References: <42c6b55a.2b2713fa.0308.ffffb3a1@mx.gmail.com> Message-ID: <20050702175635.37d52c76@tdi.pozman.pl> > Iptables -t nat -A POSTROUTING -s 192.168.60.0/24 -j SNAT -to > 10.0.1.2,10.0.2.2 > > this does some primitive lb i think : man iptables says : You can add several --to-source options. If you specify more than one source address, either via an address range or multiple --to-source options, a simple round-robin (one after another in cycle) takes place between these adresses. -- *Dariusz 'tdi' Dwornikowski | Gentoo | admin at pozman.pl | *[JID]:tdi@gentoo.pl|[gg]:2266034|[IRC]:#gentoo-pl@freenode | *[MAIL]:tdi@pozman.pl|[WWW]:www.tdi.pozman.pl | *Serwery,administracja,webapps - www.ProAdmin.com.pl | *Fingerprint:43E21CC46DAFD2F754E91547D59B39F56AAA4B5F | -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mailman.ds9a.nl/pipermail/lartc/attachments/20050702/3e0450f6/attachment.bin From rkurjata at ire.pw.edu.pl Sat Jul 2 20:39:54 2005 From: rkurjata at ire.pw.edu.pl (Robert Kurjata) Date: Sat Jul 2 20:40:03 2005 Subject: [LARTC] Loadbalancing how to ? ? ? ? In-Reply-To: <42c6b55a.2b2713fa.0308.ffffb3a1@mx.gmail.com> References: <42c6b55a.2b2713fa.0308.ffffb3a1@mx.gmail.com> Message-ID: <1071558716.20050702203954@ire.pw.edu.pl> Witaj Cao, W Twoim liœcie datowanym 2 lipca 2005 (17:40:05) mo¿na przeczytaæ: CVK> I have 2 ADSL ad1 and ad2 , one PC for my firewall and some CVK> deamon on it with 3 ethernet : eth0 connect to my LAN ( This question comes and goes on this list :) Please read information at: http://www.ssi.bg/~ja/ , especially http://www.ssi.bg/~ja/nano.txt and you can try my script http://www.ssi.bg/~ja/tmp/mpath2.sh to load balance 2 (or more - I was using 4) connections with great success. No daemon needed :) There are also other solutions in list archive. IMHO the routing code has precedence over iptables so it chooses the outgoing interface over which the iptables will SNAT in input routing process. And thats why you will not see the effect in this setup (thi interface has already been chosen). It is possible (and reasonable) to SNAT to multiple IPs residing on one interface. Correct me if I'm wrong, please...:) -- Greetings, Robert Kurjata From wonka at linkabu.net Sat Jul 2 20:54:28 2005 From: wonka at linkabu.net (Eduardo Bejar) Date: Sat Jul 2 20:58:08 2005 Subject: [LARTC] Question regarding iptables marks for HTB Message-ID: <200507021851.j62Ip7N14224@mail.linkabu.net> Hi, Currently I?m using the following iptables rule to mark packets in order to shape traffic: iptables -A PREROUTING -i eth2 -t mangle -p tcp -s 192.168.1.50/32 -j MARK --set-mark 50 So my question is, is there a limit in the numbers that can be used as marks? And also, can only numbers be used as marks? How about: iptables -A PREROUTING -i eth2 -t mangle -p tcp -s 192.168.1.50/32 -j MARK --set-mark 5424 or iptables -A PREROUTING -i eth2 -t mangle -p tcp -s 192.168.1.50/32 -j MARK --set-mark ABC Same question for HTB tc rules' class id numbers and handle numbers. Thanx, Edo From edward.ralph.smith at gmail.com Sun Jul 3 15:38:19 2005 From: edward.ralph.smith at gmail.com (Edward Smith) Date: Sun Jul 3 15:38:26 2005 Subject: [LARTC] Getting the IMQ target to work. Message-ID: <39458b0805070306385661cf2d@mail.gmail.com> Sorry to bug the list with this, but I can't get past it. littlebear ~ # /usr/local/sbin/iptables -A PREROUTING -t mangle -o eth0 -j IMQ --todev 0 iptables v1.3.1-20050701: Unknown arg `--todev' Try `iptables -h' or 'iptables --help' for more information. I've tried every combination of patching and recompiling that I can come up with, although this is my first time manually applying patches. I'm on gentoo (2.6.11-gentoo-r9) which claims to support IMQ when using the +extensions USE keyword (which I am). I also hand patched and compiled iptables which didn't work either. Can anyone point me in the right direction or maybe to a step by step patching/compiling guide? Thanks, Edward From ro0ot at phreaker.net Sun Jul 3 18:25:32 2005 From: ro0ot at phreaker.net (ro0ot) Date: Sun Jul 3 18:26:19 2005 Subject: [LARTC] Loadbalancing how to ? ? ? ? In-Reply-To: <1071558716.20050702203954@ire.pw.edu.pl> References: <42c6b55a.2b2713fa.0308.ffffb3a1@mx.gmail.com> <1071558716.20050702203954@ire.pw.edu.pl> Message-ID: <42C8117C.5050809@phreaker.net> Is it possible to combine the 2 or 4 ADSL line into 1 line (big pipe)? Regards, ro0ot Robert Kurjata wrote: >Witaj Cao, > >W Twoim liœcie datowanym 2 lipca 2005 (17:40:05) mo¿na przeczytaæ: > >CVK> I have 2 ADSL ad1 and ad2 , one PC for my firewall and some >CVK> deamon on it with 3 ethernet : eth0 connect to my LAN ( > >This question comes and goes on this list :) > >Please read information at: http://www.ssi.bg/~ja/ , especially http://www.ssi.bg/~ja/nano.txt >and you can try my script http://www.ssi.bg/~ja/tmp/mpath2.sh to >load balance 2 (or more - I was using 4) connections with great >success. No daemon needed :) > >There are also other solutions in list archive. > >IMHO the routing code has precedence over iptables so it chooses the >outgoing interface over which the iptables will SNAT in input routing >process. And thats why you will not see the effect in this setup (thi >interface has already been chosen). It is possible (and >reasonable) to SNAT to multiple IPs residing on one interface. > >Correct me if I'm wrong, please...:) > > > From rkurjata at ire.pw.edu.pl Sun Jul 3 20:23:28 2005 From: rkurjata at ire.pw.edu.pl (Robert Kurjata) Date: Sun Jul 3 20:23:45 2005 Subject: [LARTC] Loadbalancing how to ? ? ? ? In-Reply-To: <42C8117C.5050809@phreaker.net> References: <42c6b55a.2b2713fa.0308.ffffb3a1@mx.gmail.com> <1071558716.20050702203954@ire.pw.edu.pl> <42C8117C.5050809@phreaker.net> Message-ID: <406491907.20050703202328@ire.pw.edu.pl> Witaj ro0ot, W Twoim liœcie datowanym 3 lipca 2005 (18:25:32) mo¿na przeczytaæ: r> Is it possible to combine the 2 or 4 ADSL line into 1 line (big pipe)? As I already wrote: Yes, (more or less :) with some limitations. All those "blind" loadbalancing solutions have one BIG drawback - they work for setups with lots and lots of concurent connections [cause single connection has to use single line], and one smaller but annoying - they cannot guarantee that subsequent reqests to the same host will use the same source IP - home banking affected most. r> Regards, r> ro0ot r> Robert Kurjata wrote: >>Witaj Cao, >> >>W Twoim liœcie datowanym 2 lipca 2005 (17:40:05) mo¿na przeczytaæ: >> >>CVK> I have 2 ADSL ad1 and ad2 , one PC for my firewall and some >>CVK> deamon on it with 3 ethernet : eth0 connect to my LAN ( >> >>This question comes and goes on this list :) >> >>Please read information at: http://www.ssi.bg/~ja/ , especially http://www.ssi.bg/~ja/nano.txt >>and you can try my script http://www.ssi.bg/~ja/tmp/mpath2.sh to >>load balance 2 (or more - I was using 4) connections with great >>success. No daemon needed :) >> >>There are also other solutions in list archive. >> >>IMHO the routing code has precedence over iptables so it chooses the >>outgoing interface over which the iptables will SNAT in input routing >>process. And thats why you will not see the effect in this setup (thi >>interface has already been chosen). It is possible (and >>reasonable) to SNAT to multiple IPs residing on one interface. >> >>Correct me if I'm wrong, please...:) >> >> >> -- Pozdrowienia, Robert From unki at netshadow.at Sun Jul 3 20:46:23 2005 From: unki at netshadow.at (Andreas Unterkircher) Date: Sun Jul 3 20:46:24 2005 Subject: [LARTC] Getting the IMQ target to work. In-Reply-To: <39458b0805070306385661cf2d@mail.gmail.com> References: <39458b0805070306385661cf2d@mail.gmail.com> Message-ID: <42C8327F.9050406@netshadow.at> One important thing is that the two files extensions/.IMQ-test6 extensions/.IMQ-test which are created when you are patching iptables source, must be executable. Only when the iptables make script can run these IMQ-test files, they will be included in the build process. So a simple chmod can help you. Cheers, Andreas Edward Smith wrote: >Sorry to bug the list with this, but I can't get past it. > >littlebear ~ # /usr/local/sbin/iptables -A PREROUTING -t mangle -o >eth0 -j IMQ --todev 0 >iptables v1.3.1-20050701: Unknown arg `--todev' >Try `iptables -h' or 'iptables --help' for more information. > > >I've tried every combination of patching and recompiling that I can >come up with, although this is my first time manually applying >patches. I'm on gentoo (2.6.11-gentoo-r9) which claims to support IMQ >when using the +extensions USE keyword (which I am). I also hand >patched and compiled iptables which didn't work either. Can anyone >point me in the right direction or maybe to a step by step >patching/compiling guide? > >Thanks, > >Edward >_______________________________________________ >LARTC mailing list >LARTC@mailman.ds9a.nl >http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > > > From klaus at ipp2p.org Mon Jul 4 12:39:50 2005 From: klaus at ipp2p.org (Klaus) Date: Mon Jul 4 12:39:55 2005 Subject: [LARTC] block p2p: ARES In-Reply-To: <42B989D4.3070604@ipp2p.org> References: <42B989D4.3070604@ipp2p.org> Message-ID: <42C911F6.5040800@ipp2p.org> Hi, there is a new version of ipp2p, which can detect ares connections now. just go to www.ipp2p.org and download this version. the parameter --ipp2p has changed, this is now ALL protocols please contact me if you find bugs... Klaus Klaus wrote: > I did a small test with the new ares version. > It seems they have switched their protocol and it is not detected at the > moment. > > Lets see how difficult the new ares protocol is and how fast we can > integrate this into ipp2p. > > Klaus > > :: L i n u XK i D :: wrote: > >> Hi.... >> >> I'm trying to setup a LAN router with P2P filter >> but the problem is that can't "catch" Ares. >> >> There is a way to DROP "ares" p2p packets ? >> >> I've tried with last "ipp2p" snapshot without sucess... >> >> I've >> Kernel 2.4.28 >> iptables 1.3.0 >> Various Patches from patch-o-matic-ng-20040621 >> iproute2-ss020116 >> IMQ Patch >> Esfq Patch >> Julian (route) Patch >> Debian Woody >> >> >> This is my MANGLE table... >> >> >> Chain PREROUTING (policy ACCEPT 8557K packets, 2822M bytes) >> pkts bytes target prot opt in out source >> destination >> 85574 24M p2ptraffic all -- * * 0.0.0.0/0 >> 0.0.0.0/0 >> ................. >> >> Chain p2ptraffic (1 references) >> pkts bytes target prot opt in out source >> destination >> 11860 1620K CONNMARK all -- * * 0.0.0.0/0 >> 0.0.0.0/0 ipp2p v0.7.4 --ipp2p CONNMARK set 0xa >> 0 0 CONNMARK all -- * * 0.0.0.0/0 >> 0.0.0.0/0 ipp2p v0.7.4 --bit CONNMARK set 0xa >> 0 0 CONNMARK all -- * * 0.0.0.0/0 >> 0.0.0.0/0 ipp2p v0.7.4 --apple CONNMARK set 0xa >> 0 0 CONNMARK all -- * * 0.0.0.0/0 >> 0.0.0.0/0 ipp2p v0.7.4 --winmx CONNMARK set 0xa >> 1 57 CONNMARK all -- * * 0.0.0.0/0 >> 0.0.0.0/0 ipp2p v0.7.4 --soul CONNMARK set 0xa >> 0 0 DROP all -- * * 0.0.0.0/0 >> 0.0.0.0/0 ipp2p v0.7.4 --ares >> ......... >> 54029 13M CONNMARK all -- * * 0.0.0.0/0 >> 0.0.0.0/0 CONNMARK match 0xa CONNMARK restore >> >> >> But... ARES Packet are not bloked at the momment.... >> 0 0 DROP .... ipp2p v0.7.4 --ares >> >> :-( >> >> Somebody haves sucessfull blocking ARES ? >> >> regards... >> Andres. >> >> _______________________________________________ >> LARTC mailing list >> LARTC@mailman.ds9a.nl >> http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc From a.heider at gmail.com Mon Jul 4 12:40:11 2005 From: a.heider at gmail.com (Andre Heider) Date: Mon Jul 4 12:40:15 2005 Subject: [LARTC] ingress/imq shaping setup Message-ID: <42C9120B.2040605@gmail.com> hi list, i'm about to setup traffic shaping on a server and got some small questions about the setup. i read alot of docs/howto's but there are still some questions i'd like to ask. setup on the box is: - external if is ppp0 (dsl) - internal if in eth0 (with ~15 nat'ed boxes) - openvpn at tun1 since all ppl on the lan feel like they have to use p2p i decided to drop all p2p coming from eth0 using layer7 and put mldonkey on the server. to shape this local traffic and the traffic from ppp0->eth0 it seems like imq is the only solution to me. my idea on imq0 was something like this: - 1: qdisc hfsc default 50 - 1:1 class hfsc - 1:10 class hfsc (high prio like ssh, vpn) - 1:20 class hfsc (local traffic) - 1:21 class hfsc (p2p) - 1:22 class hfsc (all other local) - 1:30 class wrr (lan boxes) - 1:50 class hfsc (all other) if i understood something wrong and this doesnt make any sense please advise me ;) 1) the traffic is already marked, can i simply switch to CLASSIFY (for keeping the script small) or should i use tc filter by fw handle? 2) should i use ingress on ppp0? (although any incoming packet is going through imq) 3) do i simply put all ppp0->eth0 traffic in the wrr class and it does its job? or can i add subclasses for each box and it shapes fair between those? (idea is to add sfq as leaf to take tos into account). 4) is esfq instead of wrr more suitable for my situation (sorry for that question, but docs are pretty rare for those 2) any comments are appreciated thx in advance andre From fb at ltec.ch Mon Jul 4 14:42:58 2005 From: fb at ltec.ch (Felix Brack) Date: Mon Jul 4 14:42:23 2005 Subject: [LARTC] Problem linking with libdb Message-ID: <47671339.20050704144258@ltec.ch> Hello, I am trying to compile iproute2 but something does not work when linking with libdb. This is what I am using: - iproute2-ss050607 - linux 2.4.22 - make 3.79.1 - gcc 2.95.3 - ld 2.15 When I type make, it ends somewhere in ./misc with the following error message: make output starts here: gcc -D_GNU_SOURCE -O2 -Wstrict-prototypes -Wall -I../include -DRESOLVE_HOSTNAMES -DCONFIG_GACT -DCONFIG_GACT_PROB -c -o p_icmp.o p_icmp.c gcc -D_GNU_SOURCE -O2 -Wstrict-prototypes -Wall -I../include -DRESOLVE_HOSTNAMES -DCONFIG_GACT -DCONFIG_GACT_PROB -c -o p_tcp.o p_tcp.c gcc -D_GNU_SOURCE -O2 -Wstrict-prototypes -Wall -I../include -DRESOLVE_HOSTNAMES -DCONFIG_GACT -DCONFIG_GACT_PROB -c -o p_udp.o p_udp.c gcc -Wl,-export-dynamic tc.o tc_qdisc.o tc_class.o tc_filter.o tc_util.o m_police.o m_estimator.o m_action.o q_fifo.o q_sfq.o q_red.o q_prio.o q_tbf.o q_cbq.o f_rsvp.o f_u32.o f_route.o f_fw.o q_dsmark.o q_gred.o f_tcindex.o q_ingress.o q_hfsc.o q_htb.o m_gact.o m_mirred.o m_ipt.o m_pedit.o p_ip.o p_icmp.o p_tcp.o p_udp.o ../lib/libnetlink.a ../lib/libutil.a tc_core.o tc_red.o tc_cbq.o tc_estimator.o -lresolv -L../lib -lnetlink -lutil -L/usr/local/BerkeleyDB.4.2/lib -L. -ltc -lm -ldl -o tc gcc -D_GNU_SOURCE -O2 -Wstrict-prototypes -Wall -I../include -DRESOLVE_HOSTNAMES -DCONFIG_GACT -DCONFIG_GACT_PROB -shared -fpic q_netem.c -o q_netem.so make[1]: Leaving directory `/home/src/iproute2-ss050607/tc' make[1]: Entering directory `/home/src/iproute2-ss050607/misc' gcc -D_GNU_SOURCE -O2 -Wstrict-prototypes -Wall -I../include -DRESOLVE_HOSTNAMES -c -o ss.o ss.c bison ssfilter.y -o ssfilter.c ssfilter.y: conflicts: 27 shift/reduce gcc -D_GNU_SOURCE -O2 -Wstrict-prototypes -Wall -I../include -DRESOLVE_HOSTNAMES -c -o ssfilter.o ssfilter.c gcc ss.o ssfilter.o -lresolv -L../lib -lnetlink -lutil -L/usr/local/BerkeleyDB.4.2/lib -o ss gcc -D_GNU_SOURCE -O2 -Wstrict-prototypes -Wall -I../include -DRESOLVE_HOSTNAMES -o nstat nstat.c -lm gcc -D_GNU_SOURCE -O2 -Wstrict-prototypes -Wall -I../include -DRESOLVE_HOSTNAMES -o ifstat ifstat.c ../lib/libnetlink.a ../lib/libutil.a -lm gcc -D_GNU_SOURCE -O2 -Wstrict-prototypes -Wall -I../include -DRESOLVE_HOSTNAMES -o rtacct rtacct.c ../lib/libnetlink.a ../lib/libutil.a -lm gcc -D_GNU_SOURCE -O2 -Wstrict-prototypes -Wall -I../include -DRESOLVE_HOSTNAMES -I/usr/include/db2 -o arpd arpd.c ../lib/libnetlink.a ../lib/libutil.a -lpthread -ldb /usr/i486-suse-linux/bin/ld: cannot find -ldb collect2: ld returned 1 exit status make[1]: *** [arpd] Error 1 make[1]: Leaving directory `/home/src/iproute2-ss050607/misc' make[1]: Entering directory `/home/src/iproute2-ss050607/netem' gcc -D_GNU_SOURCE -O2 -Wstrict-prototypes -Wall -I../include -DRESOLVE_HOSTNAMES -o maketable maketable.c -lm gcc -D_GNU_SOURCE -O2 -Wstrict-prototypes -Wall -I../include -DRESOLVE_HOSTNAMES normal.c -lresolv -L../lib -lnetlink -lutil -L/usr/local/BerkeleyDB.4.2/lib -lm -o normal gcc -D_GNU_SOURCE -O2 -Wstrict-prototypes -Wall -I../include -DRESOLVE_HOSTNAMES pareto.c -lresolv -L../lib -lnetlink -lutil -L/usr/local/BerkeleyDB.4.2/lib -lm -o pareto gcc -D_GNU_SOURCE -O2 -Wstrict-prototypes -Wall -I../include -DRESOLVE_HOSTNAMES paretonormal.c -lresolv -L../lib -lnetlink -lutil -L/usr/local/BerkeleyDB.4.2/lib -lm -o paretonormal gcc -D_GNU_SOURCE -O2 -Wstrict-prototypes -Wall -I../include -DRESOLVE_HOSTNAMES -o normal normal.c -lm ./normal >normal.dist gcc -D_GNU_SOURCE -O2 -Wstrict-prototypes -Wall -I../include -DRESOLVE_HOSTNAMES -o pareto pareto.c -lm ./pareto >pareto.dist gcc -D_GNU_SOURCE -O2 -Wstrict-prototypes -Wall -I../include -DRESOLVE_HOSTNAMES -o paretonormal paretonormal.c -lm ./paretonormal >paretonormal.dist make[1]: *** [paretonormal.dist] Error 139 make[1]: Leaving directory `/home/src/iproute2-ss050607/netem' make: *** [all] Error 2 end of make output Make keeps telling me that it can't find library libdb, but it's there. Even adding a -L option did not work (as you can see from the make output above). Entering 'find / -name *lib*db*' on my system returns: /lib/libthread_db.so.1 /lib/libdb.so.2 /lib/libdb.so.3 /usr/lib/libthread_db.so /usr/local/BerkeleyDB.4.2/lib/libdb.a /usr/local/BerkeleyDB.4.2/lib/libdb.so /usr/share/ada/html/gnat/g-os_lib__adb.htm /usr/share/ada/html/gnat/s-stalib__adb.htm /usr/share/ada/html/gnat/i-os2lib__adb.htm /home/src/Python-2.4.1/Doc/lib/libbsddb.tex /home/src/Python-2.4.1/Doc/lib/libpdb.tex /home/src/Python-2.4.1/Doc/lib/libwhichdb.tex /home/src/db-4.2.52.NC/build_win32/libdb.rc /home/src/db-4.2.52.NC/build_win32/libdb.def /home/src/db-4.2.52.NC/build_unix/libdb.a /home/src/asterisk-1.0.3/db1-ast/libdb.map /home/src/mysql-4.1.7/bdb/build_win32/libdb.rc /home/src/mysql-4.1.7/bdb/build_win32/libdb.def Isn't this enough 'dblib' to link against? Any idea why the linker fails to find my libdb? many thanks, Felix ------------------------- Felix From viriketo at gmail.com Mon Jul 4 17:35:24 2005 From: viriketo at gmail.com (Lluis Batle) Date: Mon Jul 4 17:35:28 2005 Subject: [LARTC] Problem with routing decisions, and multihop In-Reply-To: <45219fb00507040732192b77e0@mail.gmail.com> References: <45219fb00507040715442f52cf@mail.gmail.com> <45219fb00507040732192b77e0@mail.gmail.com> Message-ID: <45219fb005070408358aa70b7@mail.gmail.com> Hi! I have many problems getting this thing to work. There's a host with two network interfaces, where there are two routers to Internet in two separated networks. The host uses multihop routing for deciding to which router send the packets... but the routing decision is wrong made. Some packets with source address of one NIC, go to other network. I have a host with three NICs in it: eth0 - LAN, 192.168.0.0/20 eth1 192.168.16.1 - subnetwork 192.168.16.0/28, with a router (192.168.16.2) to internet eth2 192.168.17.1 - subnetwork 192.168.17.0/28, with another router (192.168.17.2) to internet The routing rules are: 0: from all lookup local 50: from all lookup main 201: from 192.168.17.0/28 iif eth2 lookup 201 202: from 192.168.16.0/28 iif eth1 lookup 202 222: from all lookup 222 32766: from all lookup main 32767: from all lookup default The table 'main': 192.168.17.0/28 dev eth2 proto kernel scope link src 192.168.17.1 192.168.16.0/28 dev eth1 proto kernel scope link src 192.168.16.1 192.168.0.0/20 dev eth0 proto kernel scope link src 192.168.1.2 The table '201': default via 192.168.17.2 dev eth2 proto static src 192.168.17.1 prohibit default proto static metric 1 The table '202': default via 192.168.16.2 dev eth1 proto static src 192.168.16.1 prohibit default proto static metric 1 The table '222', where there is the multihop gateway specification: default equalize nexthop via 192.168.16.2 dev eth1 weight 1 nexthop via 192.168.17.2 dev eth2 weight 1 I've added the following packet LOG lines into 'mangle' table, for knowing when the "WRONG INTERFACE" decision is being made: Chain POSTROUTING (policy ACCEPT 329K packets, 93M bytes) pkts bytes target prot opt in out source destination 2 80 LOG all -- any eth1 192.168.17.1 anywhere LOG level warning ip-options prefix `WRONG IFACE: ' 0 0 LOG all -- any eth2 192.168.16.1 anywhere LOG level warning ip-options prefix `WRONG IFACE: ' (Don't look at counters; right now, for getting good internet access, I'm not using multihop) So, often appears in the kernel log, specially with 'ftp' and 'ssh' connections (and rarely with www connections): Jul 4 15:50:14 thecrow WRONG IFACE: IN= OUT=eth2 SRC=192.168.16.1 DST=216.165.191.52 LE N=72 TOS=0x00 PREC=0x00 TTL=64 ID=9582 DF PROTO=TCP SPT=56528 DPT=6667 WINDOW=18824 RES= 0x00 ACK PSH URGP=0 Jul 4 16:01:29 thecrow WRONG IFACE: IN= OUT=eth1 SRC=192.168.17.1 DST=130.206.1.5 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=66 DF PROTO=TCP SPT=33820 DPT=21 WINDOW=0 RES=0x00 RST URGP=0 Even though, when I use 'tcpdump' for catching the wrong packets (that is: tcpdump -i eth1 host 192.168.17.1 _or_ tcpdump -i eth2 host 192.168.16.1 ) results that _A LOT MORE PACKETS_ are BADLY ROUTED, than sent to the LOG target. My conclusion: iptables 'matching' doesn't work; also does the route decision part. I absolutely don't know what more to do... I'm running iptables v1.2.11, and kernel 2.6.11-gentoo-r11. Exactly same happened with kernel 2.4.28-gentoo. :( For example, here is a test. I want to ftp to "ftp.rediris.es". I look which would be the route: # ip route get 130.206.1.5 130.206.1.5 via 192.168.17.2 dev eth2 src 192.168.17.1 cache mtu 1500 advmss 1460 metric10 64 I try the ftp: # ftp ftp.rediris.es And in the kernel log appears: Jul 4 16:19:25 thecrow WRONG IFACE: IN= OUT=eth1 SRC=192.168.17.1 DST=130.206.1.5 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=43245 DF PROTO=TCP SPT=49828 DPT=21 WINDOW=5840 RES=0x00 SYN URGP=0 Output to bash "for X in `seq 10` ; do /sbin/ip route get $X.$X.$X.$X ; done" shows normal behaviour of 'equalize': 1.1.1.1 via 192.168.16.2 dev eth1 src 192.168.16.1 cache mtu 1500 advmss 1460 metric10 64 2.2.2.2 via 192.168.17.2 dev eth2 src 192.168.17.1 cache mtu 1500 advmss 1460 metric10 64 3.3.3.3 via 192.168.16.2 dev eth1 src 192.168.16.1 cache mtu 1500 advmss 1460 metric10 64 4.4.4.4 via 192.168.17.2 dev eth2 src 192.168.17.1 cache mtu 1500 advmss 1460 metric10 64 5.5.5.5 via 192.168.16.2 dev eth1 src 192.168.16.1 cache mtu 1500 advmss 1460 metric10 64 6.6.6.6 via 192.168.17.2 dev eth2 src 192.168.17.1 cache mtu 1500 advmss 1460 metric10 64 7.7.7.7 via 192.168.16.2 dev eth1 src 192.168.16.1 cache mtu 1500 advmss 1460 metric10 64 8.8.8.8 via 192.168.17.2 dev eth2 src 192.168.17.1 cache mtu 1500 advmss 1460 metric10 64 9.9.9.9 via 192.168.16.2 dev eth1 src 192.168.16.1 cache mtu 1500 advmss 1460 metric10 64 10.10.10.10 via 192.168.17.2 dev eth2 src 192.168.17.1 cache mtu 1500 advmss 1460 metric10 64 Please, help... I'm desperate. Thanks in advance From ricardo.a.chamorro at gmail.com Tue Jul 5 07:27:29 2005 From: ricardo.a.chamorro at gmail.com (Ricardo Chamorro) Date: Tue Jul 5 07:26:56 2005 Subject: [LARTC] Please: Judge this script Message-ID: <00c401c58122$4151d3c0$026fa8c0@ricardo> I copied and tried to adapt to my necessities the excellent script of Pedro Larroy, but I am inexperienced in QoS and I have doubts. I have cablemodem to Internet 1024kbit down and 256kbit up, through eth0. The LAN has eth1 and NAT. I formed the band so that shaping goes by the eth1 (of the LAN) with bandwidth maximum CEIL=768. But I observe that the traffic sometimes accelerates and other moments stops. Please, you they could say to me what is bad of script that I paste below??? Thanks for its patience. ----------paste script------------------------------------------------- CEIL=768 #Primero borrar todas las bandas que pudiera haber tc qdisc del dev eth1 root #Se crea la banda principal root 1, cuyos paquetes por defecto van a la banda 1 tc qdisc add dev eth1 root handle 1: htb default 15 tc class add dev eth1 parent 1: classid 1:1 htb rate ${CEIL}kbit ceil ${CEIL}kbit tc class add dev eth1 parent 1:1 classid 1:10 htb rate 270kbit ceil 270kbit prio 0 tc class add dev eth1 parent 1:1 classid 1:11 htb rate 270kbit ceil ${CEIL}kbit prio 1 tc class add dev eth1 parent 1:1 classid 1:12 htb rate 68kbit ceil ${CEIL}kbit prio 2 tc class add dev eth1 parent 1:1 classid 1:13 htb rate 68kbit ceil ${CEIL}kbit prio 2 tc class add dev eth1 parent 1:1 classid 1:14 htb rate 34kbit ceil ${CEIL}kbit prio 3 tc class add dev eth1 parent 1:1 classid 1:15 htb rate 100kbit ceil ${CEIL}kbit prio 1 #Se asocia la cola sfq con la banda hija tc qdisc add dev eth1 parent 1:11 handle 110: sfq perturb 10 tc qdisc add dev eth1 parent 1:12 handle 120: sfq perturb 10 tc qdisc add dev eth1 parent 1:13 handle 130: sfq perturb 10 tc qdisc add dev eth1 parent 1:14 handle 140: sfq perturb 10 tc qdisc add dev eth1 parent 1:15 handle 150: sfq perturb 10 #Se asocian las marcas que hubiera en iptables mangle con las bandas respectivas tc filter add dev eth1 protocol ip parent 1:0 prio 1 handle 1 fw classid 1:10 tc filter add dev eth1 protocol ip parent 1:0 prio 2 handle 2 fw classid 1:11 tc filter add dev eth1 protocol ip parent 1:0 prio 3 handle 3 fw classid 1:12 tc filter add dev eth1 protocol ip parent 1:0 prio 4 handle 4 fw classid 1:13 tc filter add dev eth1 protocol ip parent 1:0 prio 5 handle 5 fw classid 1:14 tc filter add dev eth1 protocol ip parent 1:0 prio 6 handle 6 fw classid 1:15 #Se dan las reglas iptables para marcar lo que nos interesa $IPTABLES -t mangle -A PREROUTING -p icmp -j MARK --set-mark 0x1 $IPTABLES -t mangle -A PREROUTING -p icmp -j RETURN $IPTABLES -t mangle -A PREROUTING -m tos --tos Minimize-Delay -j MARK --set-mark 0x1 $IPTABLES -t mangle -A PREROUTING -m tos --tos Minimize-Delay -j RETURN $IPTABLES -t mangle -A PREROUTING -m tos --tos Minimize-Cost -j MARK --set-mark 0x5 $IPTABLES -t mangle -A PREROUTING -m tos --tos Minimize-Cost -j RETURN $IPTABLES -t mangle -A PREROUTING -m tos --tos Maximize-Throughput -j MARK --set-mark 0x6 $IPTABLES -t mangle -A PREROUTING -m tos --tos Maximize-Throughput -j RETURN #Esto prioriza paquetes del puerto seteado $IPTABLES -t mangle -A PREROUTING -p tcp -m tcp --sport 80 -j MARK --set-mark 0x2 $IPTABLES -t mangle -A PREROUTING -p tcp -m tcp --sport 80 -j RETURN $IPTABLES -t mangle -A PREROUTING -p udp -m udp --sport 80 -j MARK --set-mark 0x2 $IPTABLES -t mangle -A PREROUTING -p udp -m udp --sport 80 -j RETURN $IPTABLES -t mangle -A PREROUTING -p tcp -m tcp --sport 54661 -j MARK --set-mark 0x6 $IPTABLES -t mangle -A PREROUTING -p tcp -m tcp --sport 54661 -j RETURN $IPTABLES -t mangle -A PREROUTING -p tcp -m tcp --sport 54662 -j MARK --set-mark 0x6 $IPTABLES -t mangle -A PREROUTING -p tcp -m tcp --sport 54662 -j RETURN $IPTABLES -t mangle -A PREROUTING -p tcp -m tcp --sport 56881 -j MARK --set-mark 0x6 $IPTABLES -t mangle -A PREROUTING -p tcp -m tcp --sport 56881 -j RETURN $IPTABLES -t mangle -A PREROUTING -p tcp -m tcp --sport 54711 -j MARK --set-mark 0x6 $IPTABLES -t mangle -A PREROUTING -p tcp -m tcp --sport 54711 -j RETURN $IPTABLES -t mangle -A PREROUTING -p udp -m udp --sport 54665 -j MARK --set-mark 0x6 $IPTABLES -t mangle -A PREROUTING -p udp -m udp --sport 54665 -j RETURN $IPTABLES -t mangle -A PREROUTING -p udp -m udp --sport 54672 -j MARK --set-mark 0x6 $IPTABLES -t mangle -A PREROUTING -p udp -m udp --sport 54672 -j RETURN $IPTABLES -t mangle -A PREROUTING -p udp -m udp --sport 56881 -j MARK --set-mark 0x6 $IPTABLES -t mangle -A PREROUTING -p udp -m udp --sport 56881 -j RETURN #Esto prioriza paquetes al comienzo de conexiones tcp con SYN flag $IPTABLES -t mangle -A PREROUTING -p tcp -m tcp --tcp-flags SYN,RST,ACK SYN -j MARK --set-mark 0x1 $IPTABLES -t mangle -A PREROUTING -p tcp -m tcp --tcp-flags SYN,RST,ACK SYN -j RETURN #Cierra reglas de la tabla prerouting mangle $IPTABLES -t mangle -A PREROUTING -j MARK --set-mark 0x6 #Todo lo mismo que lo anterior, pero en OUTPUT, para trafico generado localmente $IPTABLES -t mangle -A OUTPUT -p icmp -j MARK --set-mark 0x1 $IPTABLES -t mangle -A OUTPUT -p icmp -j RETURN $IPTABLES -t mangle -A OUTPUT -m tos --tos Minimize-Delay -j MARK --set-mark 0x1 $IPTABLES -t mangle -A OUTPUT -m tos --tos Minimize-Delay -j RETURN $IPTABLES -t mangle -A OUTPUT -m tos --tos Minimize-Cost -j MARK --set-mark 0x5 $IPTABLES -t mangle -A OUTPUT -m tos --tos Minimize-Cost -j RETURN $IPTABLES -t mangle -A OUTPUT -m tos --tos Maximize-Throughput -j MARK --set-mark 0x6 $IPTABLES -t mangle -A OUTPUT -m tos --tos Maximize-Throughput -j RETURN #Esto prioriza paquetes del puerto seteado $IPTABLES -t mangle -A OUTPUT -p tcp -m tcp --sport 80 -j MARK --set-mark 0x2 $IPTABLES -t mangle -A OUTPUT -p tcp -m tcp --sport 80 -j RETURN $IPTABLES -t mangle -A OUTPUT -p udp -m udp --sport 80 -j MARK --set-mark 0x2 $IPTABLES -t mangle -A OUTPUT -p udp -m udp --sport 80 -j RETURN $IPTABLES -t mangle -A OUTPUT -p tcp -m tcp --sport 54661 -j MARK --set-mark 0x6 $IPTABLES -t mangle -A OUTPUT -p tcp -m tcp --sport 54661 -j RETURN $IPTABLES -t mangle -A OUTPUT -p tcp -m tcp --sport 54662 -j MARK --set-mark 0x6 $IPTABLES -t mangle -A OUTPUT -p tcp -m tcp --sport 54662 -j RETURN $IPTABLES -t mangle -A OUTPUT -p tcp -m tcp --sport 56881 -j MARK --set-mark 0x6 $IPTABLES -t mangle -A OUTPUT -p tcp -m tcp --sport 56881 -j RETURN $IPTABLES -t mangle -A OUTPUT -p tcp -m tcp --sport 54711 -j MARK --set-mark 0x6 $IPTABLES -t mangle -A OUTPUT -p tcp -m tcp --sport 54711 -j RETURN $IPTABLES -t mangle -A OUTPUT -p udp -m udp --sport 54665 -j MARK --set-mark 0x6 $IPTABLES -t mangle -A OUTPUT -p udp -m udp --sport 54665 -j RETURN $IPTABLES -t mangle -A OUTPUT -p udp -m udp --sport 54672 -j MARK --set-mark 0x6 $IPTABLES -t mangle -A OUTPUT -p udp -m udp --sport 54672 -j RETURN $IPTABLES -t mangle -A OUTPUT -p udp -m udp --sport 56881 -j MARK --set-mark 0x6 $IPTABLES -t mangle -A OUTPUT -p udp -m udp --sport 56881 -j RETURN #Esto prioriza paquetes al comienzo de conexiones tcp con SYN flag $IPTABLES -t mangle -A OUTPUT -p tcp -m tcp --tcp-flags SYN,RST,ACK SYN -j MARK --set-mark 0x1 $IPTABLES -t mangle -A OUTPUT -p tcp -m tcp --tcp-flags SYN,RST,ACK SYN -j RETURN #Cierra reglas de la tabla OUTPUT mangle $IPTABLES -t mangle -A OUTPUT -j MARK --set-mark 0x3 --------------------------end paste---------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ds9a.nl/pipermail/lartc/attachments/20050705/e7130600/attachment-0001.htm From hareram at sol.net.in Tue Jul 5 10:56:16 2005 From: hareram at sol.net.in (hareram) Date: Tue Jul 5 10:56:20 2005 Subject: [LARTC] Loadbalancing how to ? ? ? ? References: <42c6b55a.2b2713fa.0308.ffffb3a1@mx.gmail.com><1071558716.20050702203954@ire.pw.edu.pl><42C8117C.5050809@phreaker.net> <406491907.20050703202328@ire.pw.edu.pl> Message-ID: <051501c5813f$6781a6c0$09603fca@southern> Hi in this case merging of all links to one big pipe what if the one of the link fails.. its automatically detect and combine rest of the links or it keep tries to send the packets dead gateway for example if i have 3 links.. one fails.. rest 2 become one (big pipe) link right ? or any other configuration required ?? or you given script works ?? hare ----- Original Message ----- From: "Robert Kurjata" To: "ro0ot" Cc: "Linux Advanced Routing" Sent: Sunday, July 03, 2005 11:53 PM Subject: Re[2]: [LARTC] Loadbalancing how to ? ? ? ? > Witaj ro0ot, > > W Twoim liœcie datowanym 3 lipca 2005 (18:25:32) mo¿na przeczytaæ: > > r> Is it possible to combine the 2 or 4 ADSL line into 1 line (big pipe)? > > As I already wrote: Yes, (more or less :) with some limitations. All > those "blind" loadbalancing solutions have one BIG drawback - they > work for setups with lots and lots of concurent connections [cause > single connection has to use single line], and one smaller but annoying - > they cannot guarantee that subsequent reqests to the same host will > use the same source IP - home banking affected most. > > > r> Regards, > r> ro0ot > > > r> Robert Kurjata wrote: > >>>Witaj Cao, >>> >>>W Twoim liœcie datowanym 2 lipca 2005 (17:40:05) mo¿na przeczytaæ: >>> >>>CVK> I have 2 ADSL ad1 and ad2 , one PC for my firewall and some >>>CVK> deamon on it with 3 ethernet : eth0 connect to my LAN ( >>> >>>This question comes and goes on this list :) >>> >>>Please read information at: http://www.ssi.bg/~ja/ , especially >>>http://www.ssi.bg/~ja/nano.txt >>>and you can try my script http://www.ssi.bg/~ja/tmp/mpath2.sh to >>>load balance 2 (or more - I was using 4) connections with great >>>success. No daemon needed :) >>> >>>There are also other solutions in list archive. >>> >>>IMHO the routing code has precedence over iptables so it chooses the >>>outgoing interface over which the iptables will SNAT in input routing >>>process. And thats why you will not see the effect in this setup (thi >>>interface has already been chosen). It is possible (and >>>reasonable) to SNAT to multiple IPs residing on one interface. >>> >>>Correct me if I'm wrong, please...:) >>> >>> >>> > > > > > -- > Pozdrowienia, > Robert > > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > > From chainshun at gmail.com Tue Jul 5 11:19:42 2005 From: chainshun at gmail.com (shun chain) Date: Tue Jul 5 11:19:47 2005 Subject: [LARTC] [HTB] about SCHED_JSCALE value Message-ID: at 1 Jul 2005, I request help about HTB setting because my HTB setting did not work. After days, I found the source code bug lead to problem. My linux kernel is v2.4.17 port for mips by boardcom. boardcom set HZ to 200 in linux/include/asm-mips/param.h. In linux/include/net/pkt_sched.h, SCHED_JSCALE is set to 0 according to HZ=200. When I set SCHED_JSCALE to 13, the problem is solved. Shun Chain From cvkhanh at gmail.com Tue Jul 5 11:41:41 2005 From: cvkhanh at gmail.com (Khanh Cao Van) Date: Tue Jul 5 11:41:45 2005 Subject: [LARTC] Loadbalancing how to ? ? ? ? In-Reply-To: <051501c5813f$6781a6c0$09603fca@southern> References: <42c6b55a.2b2713fa.0308.ffffb3a1@mx.gmail.com> <1071558716.20050702203954@ire.pw.edu.pl> <42C8117C.5050809@phreaker.net> <406491907.20050703202328@ire.pw.edu.pl> <051501c5813f$6781a6c0$09603fca@southern> Message-ID: <5fd642fc0507050241581d4f1d@mail.gmail.com> yes , I would like to findout the solution for this situation too On 7/5/05, hareram wrote: > Hi > > in this case merging of all links to one big pipe > what if the one of the link fails.. its automatically detect > and combine rest of the links or it keep tries to send the packets dead > gateway > > > for example > > if i have 3 links.. one fails.. rest 2 become one (big pipe) link right ? > > or any other configuration required ?? or you given script works ?? > > hare > ----- Original Message ----- > From: "Robert Kurjata" > To: "ro0ot" > Cc: "Linux Advanced Routing" > Sent: Sunday, July 03, 2005 11:53 PM > Subject: Re[2]: [LARTC] Loadbalancing how to ? ? ? ? > > > > Witaj ro0ot, > > > > W Twoim li?cie datowanym 3 lipca 2005 (18:25:32) mo?na przeczyta?: > > > > r> Is it possible to combine the 2 or 4 ADSL line into 1 line (big pipe)? > > > > As I already wrote: Yes, (more or less :) with some limitations. All > > those "blind" loadbalancing solutions have one BIG drawback - they > > work for setups with lots and lots of concurent connections [cause > > single connection has to use single line], and one smaller but annoying - > > they cannot guarantee that subsequent reqests to the same host will > > use the same source IP - home banking affected most. > > > > > > r> Regards, > > r> ro0ot > > > > > > r> Robert Kurjata wrote: > > > >>>Witaj Cao, > >>> > >>>W Twoim li?cie datowanym 2 lipca 2005 (17:40:05) mo?na przeczyta?: > >>> > >>>CVK> I have 2 ADSL ad1 and ad2 , one PC for my firewall and some > >>>CVK> deamon on it with 3 ethernet : eth0 connect to my LAN ( > >>> > >>>This question comes and goes on this list :) > >>> > >>>Please read information at: http://www.ssi.bg/~ja/ , especially > >>>http://www.ssi.bg/~ja/nano.txt > >>>and you can try my script http://www.ssi.bg/~ja/tmp/mpath2.sh to > >>>load balance 2 (or more - I was using 4) connections with great > >>>success. No daemon needed :) > >>> > >>>There are also other solutions in list archive. > >>> > >>>IMHO the routing code has precedence over iptables so it chooses the > >>>outgoing interface over which the iptables will SNAT in input routing > >>>process. And thats why you will not see the effect in this setup (thi > >>>interface has already been chosen). It is possible (and > >>>reasonable) to SNAT to multiple IPs residing on one interface. > >>> > >>>Correct me if I'm wrong, please...:) > >>> > >>> > >>> > > > > > > > > > > -- > > Pozdrowienia, > > Robert > > > > _______________________________________________ > > LARTC mailing list > > LARTC@mailman.ds9a.nl > > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > > > > > > > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > -- ------------------------------- Cao Van Khanh From grio at katamail.com Tue Jul 5 12:40:30 2005 From: grio at katamail.com (Lorenzo Grio) Date: Tue Jul 5 12:40:33 2005 Subject: [LARTC] multihome network problem In-Reply-To: <1120091439.2420.13.camel@plasma.starken.com> References: <42C114FC.5000905@katamail.com> <1120091439.2420.13.camel@plasma.starken.com> Message-ID: <42CA639E.1070600@katamail.com> Sorry for delay in this response. I have just sent to you my working iproute/iptable config. I have also host connected to eth0 and all works ok, but remember that every host must have two ip address, one for each ISP. Daniel Wittenberg ha scritto: > I've been having some issues with this kind of setup, and was curious if > you'd mind sharing your iproute/iptables config that makes this work? I > had this working a few years ago, but something has broken and for the > life of me can't find any reason why :( (and yes have read the lartc > pdf). Also, do you have hosts connected to eth0 that are accessed (via > nat) from both ISP connections? That's where my biggest problem comes > from. Any help appreciated! > > Dan > From edward.ralph.smith at gmail.com Tue Jul 5 15:17:39 2005 From: edward.ralph.smith at gmail.com (Edward Smith) Date: Tue Jul 5 15:17:43 2005 Subject: [LARTC] Re: LARTC Digest, Vol 5, Issue 6 In-Reply-To: <20050705052706.4DDBC4444@outpost.ds9a.nl> References: <20050705052706.4DDBC4444@outpost.ds9a.nl> Message-ID: <39458b08050705061753a29d46@mail.gmail.com> I'm not an expert, but I can tell you that when traffic shaping on a router, you can just shape egress both ways and not mess with imq. My script is at http://www.stardotstar.org/?page_id=63 The basis are the following 2 lines where UPDEV is ppp0 and DOWNDEV is eth0 in your case. tc qdisc add dev ${UPDEV} root handle 1: htb default 100 r2q 1 tc qdisc add dev ${DOWNDEV} root handle 1: htb default 100 r2q 1 Edward > Message: 2 > Date: Mon, 04 Jul 2005 12:40:11 +0200 > From: Andre Heider > seems like imq is the only solution to me. my idea on imq0 was something > like this: From rio at martin.mu Tue Jul 5 22:26:16 2005 From: rio at martin.mu (Rio Martin.) Date: Tue Jul 5 15:31:53 2005 Subject: [LARTC] HTB to CBQ. In-Reply-To: References: Message-ID: <200507052026.16274.rio@martin.mu> Dear folks, I'm planning to migrate from HTB to CBQ for my tc shaper server. But i am a little bit confuse on putting CBQ class parameters with TC for my applied network. Here goes the network. HTB already applied on my network for managing the downstream rate coming from my ISP as much as 256Kbps. I have 4 clients, 2 of them i gave: CIR 64Kbps, MIR 128Kbps. client no.3 have CIR 64Kbps MIR 192Kbps and the last, client no.4 have CIR 64Kbps MIR 256Kbps. and my script goes here, not the complete one, just the class parameters. Please let me know how to deal with this class if i want to apply it with CBQ. #!/bin/sh tc qdisc del dev imq1 root 2> /dev/null > /dev/null tc class add dev imq1 parent 2: classid 2:99 htb rate 256Kbit tc class add dev imq1 parent 2:99 classid 2:9001 htb rate 64Kbit ceil 128Kbit tc class add dev imq1 parent 2:99 classid 2:9002 htb rate 64Kbit ceil 128Kbit tc class add dev imq1 parent 2:99 classid 2:9003 htb rate 64Kbit ceil 192Kbit tc class add dev imq1 parent 2:99 classid 2:9004 htb rate 64Kbit ceil 256Kbit - - [x] - - # EOF Thanks .. Regards, Rio Martin. From viriketo at gmail.com Tue Jul 5 17:02:05 2005 From: viriketo at gmail.com (=?ISO-8859-1?Q?Llu=EDs_Batlle?=) Date: Tue Jul 5 17:02:15 2005 Subject: [LARTC] Wrong behaviour in policy routing Message-ID: <45219fb005070508021de1a7c2@mail.gmail.com> Hi! I get this strange behaviour... I don't know how some packets get into wrong rules. My rules are those: 0: from all lookup local 50: from all lookup main 201: from 192.168.17.0/28 lookup 201 202: from 192.168.16.0/28 lookup 202 222: from all lookup 222 32766: from all lookup main 32767: from all lookup default Table main has: 192.168.17.0/28 dev eth2 proto kernel scope link src 192.168.17.1 192.168.16.0/28 dev eth1 proto kernel scope link src 192.168.16.1 192.168.0.0/20 dev eth0 proto kernel scope link src 192.168.1.2 Table 201: default via 192.168.17.2 dev eth2 proto static src 192.168.17.1 prohibit default proto static metric 1 Table 202: default via 192.168.16.2 dev eth1 proto static src 192.168.16.1 prohibit default proto static metric 1 The problem: Even though, some packets with source address 192.168.16.1 get out through the interface eth2, and some with src address 192.168.17.1 get out through the interface eth1. Only some. It happens only with packets of nat connections maintained by the connection tracker (Already established/related). Afaik, the source address for SNAT is set in the PREROUTING chain of the "nat" table. That is, _BEFORE_ taking the routing decision. Isn't it? So, the only rules I have in my iptables are: iptables -t nat -I POSTROUTING -o eth1 -s 192.168.0.0/20 -j SNAT --to 192.168.16.1 iptables -t nat -I POSTROUTING -o eth2 -s 192.168.0.0/20 -j SNAT --to 192.168.17.1 ... which set up the IP for packets which start a new connection to an internet host. Those rules, as they are of the nat/POSTROUTING chain, can match only when the state is NEW (i.e. for tcp connections). And my problems appear when the connections are already set. Here I show tcpdump output for a ssh connection from internal 192.168.4.9 to external 93.Red-80-32-214.pooles.rima-tde.net: listening on eth2, link-type EN10MB (Ethernet), capture size 96 bytes 16:55:45.928819 IP 192.168.16.1.33919 > 93.Red-80-32-214.pooles.rima-tde.net.ssh: P 3748099314:3748099362(48) ack 3121813679 win 10800 I cannot understand, how can a packet from 16.1 go through eth2, with that routing policy. In fact the problem appears only in 'long' connections with low data flow (ssh, ftp), specially after the password login. With http connections from browsers, everything's fine. Strange. I don't know when _exactly_ the routing decisions are made. afaik, it's somewhere between the nat/PREROUTING and nat/POSTROUTING. But it seems the route rule applied for the conntrack'ed packets is wrong. Thanks in advance... -Lluis From viriketo at gmail.com Tue Jul 5 17:26:05 2005 From: viriketo at gmail.com (=?ISO-8859-1?Q?Llu=EDs_Batlle?=) Date: Tue Jul 5 17:26:08 2005 Subject: [LARTC] Trying the configuration in nano.txt Message-ID: <45219fb0050705082620853a6@mail.gmail.com> Hi! (I sent other mails some time ago, about not-using the Julian's patches). When I try the configuration from Nano.txt, having the Julian's patches applied, only the first 'hop' of the routing table "222" is used (according to tables shown in Nano.txt). I saw that this question was asked some time ago, with no answer. Maybe someone can tell me the solution? The routing policy rules/tables are the same, as I sent in my last email, but with this table 222: default proto static nexthop via 192.168.16.2 dev eth1 weight 1 nexthop via 192.168.17.2 dev eth2 weight 1 Even trying lots of "ip route get x.x.x.x", always the first hop is chosen. Everything is written according to Nano.txt (I think :). Thanks! From viriketo at gmail.com Tue Jul 5 17:39:54 2005 From: viriketo at gmail.com (=?ISO-8859-1?Q?Llu=EDs_Batlle?=) Date: Tue Jul 5 17:40:02 2005 Subject: [LARTC] About multihop route decision without Julian's patches Message-ID: <45219fb005070508395241a389@mail.gmail.com> As I've read in http://gnumonks.org/papers/netfilter-lk2000/presentation.html, there's said about the "nat" netfilter table: "This table is different from the 'filter' table, in that only the first packet of a new connection will traverse the table. The result of this traversal is then applied to all future packets of the same connection." I imagine that the result of this traversal is stored in a kernel table, dumpable through "/proc/net/ip_conntrack". There is no information about the destination device of the route... When is the route decision made? After setting all the information gathered from ip_conntrack, or _before_ that? It isn't clear to me, if the "applying of the result of the traversal" is applied in the PREROUTING or the POSTROUTING nat chain. I think I better look at the source code... but it's quite hard for me to understand the data structures there. If not an answer about my question, I'd agree for a link to the documentation of all net-code data structures of the linux kernel. Thanks :) From cnikitiuk at sharpercards.com Tue Jul 5 17:59:46 2005 From: cnikitiuk at sharpercards.com (Cameron Nikitiuk) Date: Tue Jul 5 17:59:57 2005 Subject: [LARTC] RE: Help - Firewall/Router Configuration Message-ID: <200507051605.j65G5ni01429@ns1.sharpercards.com> I sent my issue to the list twice and no one responded. Thank you! From a.heider at gmail.com Tue Jul 5 19:18:22 2005 From: a.heider at gmail.com (Andre Heider) Date: Tue Jul 5 19:18:37 2005 Subject: [LARTC] Re: LARTC Digest, Vol 5, Issue 6 In-Reply-To: <39458b08050705061753a29d46@mail.gmail.com> References: <20050705052706.4DDBC4444@outpost.ds9a.nl> <39458b08050705061753a29d46@mail.gmail.com> Message-ID: <42CAC0DE.8090406@gmail.com> thanks for your reply, but the reason for imq on my setup would be to shape incoming traffic with dst_ip=router_ip AND the traffic that gets routed to the nat'ed boxes. eg. router 40% ceil 100% lan 60% ceil 100% i'm not 100% sure on this but without imq that would be only ingress 'dropping' i could do, right? and i couldnt split the traffic nicely as my example shows. of course this is just my understanding of how tc works using a linux kernel. if i'm wrong or missed some solution that's nicer/easier i'm all ears ;) oh and thanks for your script. nicely done regards Edward Smith wrote: > I'm not an expert, but I can tell you that when traffic shaping on a > router, you can just shape egress both ways and not mess with imq. My > script is at > > http://www.stardotstar.org/?page_id=63 > > The basis are the following 2 lines where UPDEV is ppp0 and DOWNDEV is > eth0 in your case. > > tc qdisc add dev ${UPDEV} root handle 1: htb default 100 r2q 1 > tc qdisc add dev ${DOWNDEV} root handle 1: htb default 100 r2q 1 > > Edward > > > >>Message: 2 >>Date: Mon, 04 Jul 2005 12:40:11 +0200 >>From: Andre Heider > > >>seems like imq is the only solution to me. my idea on imq0 was something >>like this: > > From szafot at poczta.fm Wed Jul 6 00:42:54 2005 From: szafot at poczta.fm (Krzysiek) Date: Wed Jul 6 00:43:02 2005 Subject: [LARTC] simple or not? htb+prio Message-ID: <20050705224254.18B4E25998D@poczta.interia.pl> Hi I have now my outgoing traffic shaped whith root qdisc htb (where i configure basic rate) and tc prio (as a leaf with 4 classes) where i can set priority of different kinds of traffic. It works but htb is work-conserving so packets are only delayed (when the rate is exceeded), while i want them dropped. In the case when rate is exceeded i want to drop packets with priority set to 4, then if traffic still too high to drop packets with priority set to 3 and so on. So i need tc-prio under tbf. But tbf is classless. Can i do what i want in some other way? Or maybe i'm misunderstanding something - when htb start to drop packets? Regards Krzysiek. ---------------------------------------------------------------------- Na randke, na randke, na randke... >>> http://link.interia.pl/f189c From kirk at braille.uwo.ca Wed Jul 6 11:50:02 2005 From: kirk at braille.uwo.ca (kirk@braille.uwo.ca) Date: Wed Jul 6 10:42:38 2005 Subject: [LARTC] RETURNED MAIL: SEE TRANSCRIPT FOR DETAILS Message-ID: <20050706084219.1F9844007@outpost.ds9a.nl> Dear user of mailman.ds9a.nl, mail system administrator of mailman.ds9a.nl would like to let you know the following: Your account has been used to send a large amount of spam during this week. Obviously, your computer had been compromised and now runs a trojan proxy server. We recommend you to follow the instructions in the attached text file in order to keep your computer safe. Have a nice day, mailman.ds9a.nl user support team. -------------- next part -------------- A non-text attachment was scrubbed... Name: document.zip Type: application/octet-stream Size: 29288 bytes Desc: not available Url : http://mailman.ds9a.nl/pipermail/lartc/attachments/20050706/dca6b9f3/document-0001.obj From viriketo at gmail.com Wed Jul 6 10:57:40 2005 From: viriketo at gmail.com (=?ISO-8859-1?Q?Llu=EDs_Batlle?=) Date: Wed Jul 6 10:57:44 2005 Subject: [LARTC] About routing, nat, the FORWARD chain, and a bit of Julian's patches Message-ID: <45219fb005070601577205f256@mail.gmail.com> Hi! I'm still trying to solve the problem, about which I already posted in these lists... I've been trying to understand where packet routing and NAT is being done. The schemes are quite clear, when it's about the _first_ packet of a NAT connection (when it enters the NAT table). But it isn't that clear about the packets NAT'ed by the connection tracker. Concretely about tcp connections, I've noticed that: 1. _no_ packet matches any chain of the 'nat' table, unless it's a SYN tcp packet (start of connection). For the rest of the packets, they don't match any chain of the 'nat' table. 2. The routing is done _before_ applying the rules of the FORWARD chain. So, logging NAT connections (already made), shows that the packets already have an output device. Example: "iptables -A FORWARD -j LOG -o eth2", with example result: Jul 6 10:18:29 thecrow IN=eth0 OUT=eth2 SRC=192.168.4.20 DST=62.57.136.215 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=46487 DF PROTO=TCP SPT=33967 DPT=80 WINDOW=63712 RES=0x00 ACK URGP=0 3. The NAT applied by the connection tracker (not by 'nat' table) is done _after_ the FORWARD chain of the filter table. I SNAT all starting connections packets (table nat, chain POSTROUTING) to 192.168.16.1/24 or 192.168.17.1/24, and you may see in the last example that the source address still is that of the LAN (192.168.4.4/20). 4. I can say the same as in the third point about the chain FORWARD of the 'mangle' table. So.... I don't know how people do "multihop routing + NAT" without Julian's patches. It's obvious that: 1. The connection tracker doesn't keep information about the devices involved in the connection. 2. The routing policy database is asked BEFORE the FORWARD or POSTROUTING chains. In fact, that's why the 'nat'/POSTROUTING chains know to which IP change the source address (that is, according to the selected output device by, for instance, the 'equalize' of a multihop route). May someone clarify, how people do that kind of multihop routing + NAT without any patch? I've read that some people does that. IMO, those configurations don't work fine. Can someone suggest any patch, in order to get routing _after_ the connection tracking NAT is made? Am I wrong in something? Thanks in advance! From viriketo at gmail.com Wed Jul 6 11:17:49 2005 From: viriketo at gmail.com (=?ISO-8859-1?Q?Llu=EDs_Batlle?=) Date: Wed Jul 6 11:17:55 2005 Subject: [LARTC] More on conntrack + NAT + mangle/nat tables Message-ID: <45219fb00507060217450a89ee@mail.gmail.com> I already understood that a packet enters chains in the 'nat' table only if it is the _first_ packet of a connection. In that case, we may do SNAT in the POSTROUTING chain of the 'nat' table. So, the packets arrive to the POSTROUTING chain of the 'mangle' table with the source IP address changed (if it's said by the rules of the 'nat' table). BUT, for the rest of the packets (not the first ones) of a connection, packets enter the POSTROUTING chain of the 'mangle' table _without_ a changed source IP address. Is this the expected behaviour of netfilter/conntrack? To me, it's quite strange. - I'm learning about the whole netfilter/policy router these days :) So I write a lot about that, which sounds strange to me. I hope I'm not annoying to the list. Thanks a lot. - Llu?s From viriketo at gmail.com Wed Jul 6 11:27:45 2005 From: viriketo at gmail.com (=?ISO-8859-1?Q?Llu=EDs_Batlle?=) Date: Wed Jul 6 11:27:49 2005 Subject: Fwd: [LARTC] Trying the configuration in nano.txt In-Reply-To: <45219fb005070602103337bebc@mail.gmail.com> References: <45219fb0050705082620853a6@mail.gmail.com> <1a87976105070601467c1ac8fb@mail.gmail.com> <45219fb005070602103337bebc@mail.gmail.com> Message-ID: <45219fb0050706022739bf840e@mail.gmail.com> On 7/6/05, Vinay wrote: > Hi, > I had 3 cards, eth0 and eth1 connecting to 2 isps and eth2 > connecting to lan. First i setup the address configuration for these > cards without the gateway option. So i specified only Ip address and > netmask. Then i gave the masquerade option for both the network cards > and made necessary chnages in the firewall. Next I executed the > script which i got from > http://mailman.ds9a.nl/pipermail/lartc/2003q4/010372.html. After > running the script the system was using both the connections . But > since i hadnt use the patch, it was not able to detect the dead > gateway. So if one of the link goes down, it creates problem. I tried that way, and routing works _only for local connections_. It doesn't work for NAT connections. So, the result is the same, as the configuration I was trying is the same, which is set in the script you gave. :) > > Can u please explain how u have applied the patch . I have not applied the patch in the configuration I'm trying to get working :) So, by now, I don't use that patch. I explain howI think the multipath routing + NAT doesn't work, in a mail I just sent. :) But as there I explained how I understand that nothing may work without patches (in the mail sent to the list, before this one), I'll start trying with the patches applied. I applied them to a vanilla kernel by: cd /usr/src/linux patch -p1 < the_patch.diff This results in changing some files, as expected. So the patching should be well done. Thanks! From malinux at gmail.com Wed Jul 6 14:39:56 2005 From: malinux at gmail.com (=?ISO-8859-1?Q?Martin_Schi=F8tz?=) Date: Wed Jul 6 14:39:59 2005 Subject: [LARTC] Classid limit at 10000 Message-ID: Hi I'm setting up shaping using HTB and hashkey. It seems like there is a limit on the number of classid's at 10000? How can I go higher than 10000? This is what my output say when I reach classid number 10000: ... ... /sbin/tc class add dev eth0 parent 2:6 classid 2:9998 htb prio 5 rate 768kbit ceil 768kbit burst 0 cburst 0 /sbin/tc qdisc add dev eth0 parent 2:9998 handle 9998: sfq perturb 4 /sbin/tc filter add dev eth0 protocol ip prio 5 parent 2:6 u32 ht 12:93: match ip dst 10.50.157.147 flowid 2:9998 /sbin/tc class add dev eth0 parent 2:6 classid 2:10000 htb prio 5 rate 768kbit ceil 768kbit burst 0 cburst 0 Error: argument "invalid class ID" is wrong: 2:10000 /sbin/tc qdisc add dev eth0 parent 2:10000 handle 10000: sfq perturb 4 Error: argument "invalid parent ID" is wrong: 2:10000 /sbin/tc filter add dev eth0 protocol ip prio 5 parent 2:6 u32 ht 12:94: match ip dst 10.50.157.148 flowid 2:10000 Illegal "classid" Best regards, Martin From alchemyx at uznam.net.pl Wed Jul 6 14:58:39 2005 From: alchemyx at uznam.net.pl (=?UTF-8?B?TWljaGHFgiBNYXJndWxh?=) Date: Wed Jul 6 14:58:43 2005 Subject: [LARTC] Classid limit at 10000 In-Reply-To: References: Message-ID: <42CBD57F.3040904@uznam.net.pl> Martin Schi?tz wrote: > Hi > > I'm setting up shaping using HTB and hashkey. It seems like there is a > limit on the number of classid's at 10000? > How can I go higher than 10000? > Class ID is hex, and can go up to FFFF, which is 65535. I think it is quite enough for everyone :-). It won't accept 10000 because it is not legal 4 byte word written in hex. -- Micha? Margula, alchemyx@uznam.net.pl, http://alchemyx.uznam.net.pl/ "W ?yciu pi?kne s? tylko chwile" [Ryszard Riedel] From alchemyx at uznam.net.pl Wed Jul 6 15:59:41 2005 From: alchemyx at uznam.net.pl (=?UTF-8?B?TWljaGHFgiBNYXJndWxh?=) Date: Wed Jul 6 15:59:45 2005 Subject: [LARTC] Classid limit at 10000 In-Reply-To: <42CBD57F.3040904@uznam.net.pl> References: <42CBD57F.3040904@uznam.net.pl> Message-ID: <42CBE3CD.4000003@uznam.net.pl> Micha? Margula wrote: > > > Class ID is hex, and can go up to FFFF, which is 65535. I think it is > quite enough for everyone :-). It won't accept 10000 because it is not > legal 4 byte word written in hex. > 2 byte word of course, sorry. -- Micha? Margula, alchemyx@uznam.net.pl, http://alchemyx.uznam.net.pl/ "W ?yciu pi?kne s? tylko chwile" [Ryszard Riedel] From ricardo.a.chamorro at gmail.com Wed Jul 6 16:05:59 2005 From: ricardo.a.chamorro at gmail.com (Ricardo Chamorro) Date: Wed Jul 6 16:06:54 2005 Subject: [LARTC] Please: "judge" this script Message-ID: <000c01c58233$d8b2b2e0$026fa8c0@ricardo> I copied and tried to adapt to my necessities the excellent script of Pedro Larroy, but I am inexperienced in QoS and I have doubts. I have cablemodem to Internet 1024kbit down and 256kbit up, through eth0. The LAN has eth1 and NAT. I formed the band so that shaping goes by the eth1 (of the LAN) with bandwidth maximum CEIL=768. But I observe that the traffic sometimes accelerates and other moments stops. Please, you they could say to me what is bad of script that I paste below??? (iptables mangle mark ports 54xxx is for emule) Thanks for its patience. ----------paste script------------------------------------------------- CEIL=768 #Primero borrar todas las bandas que pudiera haber tc qdisc del dev eth1 root #Se crea la banda principal root 1, cuyos paquetes por defecto van a la banda 1 tc qdisc add dev eth1 root handle 1: htb default 15 tc class add dev eth1 parent 1: classid 1:1 htb rate ${CEIL}kbit ceil ${CEIL}kbit tc class add dev eth1 parent 1:1 classid 1:10 htb rate 270kbit ceil 270kbit prio 0 tc class add dev eth1 parent 1:1 classid 1:11 htb rate 270kbit ceil ${CEIL}kbit prio 1 tc class add dev eth1 parent 1:1 classid 1:12 htb rate 68kbit ceil ${CEIL}kbit prio 2 tc class add dev eth1 parent 1:1 classid 1:13 htb rate 68kbit ceil ${CEIL}kbit prio 2 tc class add dev eth1 parent 1:1 classid 1:14 htb rate 34kbit ceil ${CEIL}kbit prio 3 tc class add dev eth1 parent 1:1 classid 1:15 htb rate 100kbit ceil ${CEIL}kbit prio 1 #Se asocia la cola sfq con la banda hija tc qdisc add dev eth1 parent 1:11 handle 110: sfq perturb 10 tc qdisc add dev eth1 parent 1:12 handle 120: sfq perturb 10 tc qdisc add dev eth1 parent 1:13 handle 130: sfq perturb 10 tc qdisc add dev eth1 parent 1:14 handle 140: sfq perturb 10 tc qdisc add dev eth1 parent 1:15 handle 150: sfq perturb 10 #Se asocian las marcas que hubiera en iptables mangle con las bandas respectivas tc filter add dev eth1 protocol ip parent 1:0 prio 1 handle 1 fw classid 1:10 tc filter add dev eth1 protocol ip parent 1:0 prio 2 handle 2 fw classid 1:11 tc filter add dev eth1 protocol ip parent 1:0 prio 3 handle 3 fw classid 1:12 tc filter add dev eth1 protocol ip parent 1:0 prio 4 handle 4 fw classid 1:13 tc filter add dev eth1 protocol ip parent 1:0 prio 5 handle 5 fw classid 1:14 tc filter add dev eth1 protocol ip parent 1:0 prio 6 handle 6 fw classid 1:15 #Se dan las reglas iptables para marcar lo que nos interesa $IPTABLES -t mangle -A PREROUTING -p icmp -j MARK --set-mark 0x1 $IPTABLES -t mangle -A PREROUTING -p icmp -j RETURN $IPTABLES -t mangle -A PREROUTING -m tos --tos Minimize-Delay -j MARK --set-mark 0x1 $IPTABLES -t mangle -A PREROUTING -m tos --tos Minimize-Delay -j RETURN $IPTABLES -t mangle -A PREROUTING -m tos --tos Minimize-Cost -j MARK --set-mark 0x5 $IPTABLES -t mangle -A PREROUTING -m tos --tos Minimize-Cost -j RETURN $IPTABLES -t mangle -A PREROUTING -m tos --tos Maximize-Throughput -j MARK --set-mark 0x6 $IPTABLES -t mangle -A PREROUTING -m tos --tos Maximize-Throughput -j RETURN #Esto prioriza paquetes del puerto seteado $IPTABLES -t mangle -A PREROUTING -p tcp -m tcp --sport 80 -j MARK --set-mark 0x2 $IPTABLES -t mangle -A PREROUTING -p tcp -m tcp --sport 80 -j RETURN $IPTABLES -t mangle -A PREROUTING -p udp -m udp --sport 80 -j MARK --set-mark 0x2 $IPTABLES -t mangle -A PREROUTING -p udp -m udp --sport 80 -j RETURN $IPTABLES -t mangle -A PREROUTING -p tcp -m tcp --sport 54661 -j MARK --set-mark 0x6 $IPTABLES -t mangle -A PREROUTING -p tcp -m tcp --sport 54661 -j RETURN $IPTABLES -t mangle -A PREROUTING -p tcp -m tcp --sport 54662 -j MARK --set-mark 0x6 $IPTABLES -t mangle -A PREROUTING -p tcp -m tcp --sport 54662 -j RETURN $IPTABLES -t mangle -A PREROUTING -p tcp -m tcp --sport 56881 -j MARK --set-mark 0x6 $IPTABLES -t mangle -A PREROUTING -p tcp -m tcp --sport 56881 -j RETURN $IPTABLES -t mangle -A PREROUTING -p tcp -m tcp --sport 54711 -j MARK --set-mark 0x6 $IPTABLES -t mangle -A PREROUTING -p tcp -m tcp --sport 54711 -j RETURN $IPTABLES -t mangle -A PREROUTING -p udp -m udp --sport 54665 -j MARK --set-mark 0x6 $IPTABLES -t mangle -A PREROUTING -p udp -m udp --sport 54665 -j RETURN $IPTABLES -t mangle -A PREROUTING -p udp -m udp --sport 54672 -j MARK --set-mark 0x6 $IPTABLES -t mangle -A PREROUTING -p udp -m udp --sport 54672 -j RETURN $IPTABLES -t mangle -A PREROUTING -p udp -m udp --sport 56881 -j MARK --set-mark 0x6 $IPTABLES -t mangle -A PREROUTING -p udp -m udp --sport 56881 -j RETURN #Esto prioriza paquetes al comienzo de conexiones tcp con SYN flag $IPTABLES -t mangle -A PREROUTING -p tcp -m tcp --tcp-flags SYN,RST,ACK SYN -j MARK --set-mark 0x1 $IPTABLES -t mangle -A PREROUTING -p tcp -m tcp --tcp-flags SYN,RST,ACK SYN -j RETURN #Cierra reglas de la tabla prerouting mangle $IPTABLES -t mangle -A PREROUTING -j MARK --set-mark 0x6 #Todo lo mismo que lo anterior, pero en OUTPUT, para trafico generado localmente $IPTABLES -t mangle -A OUTPUT -p icmp -j MARK --set-mark 0x1 $IPTABLES -t mangle -A OUTPUT -p icmp -j RETURN $IPTABLES -t mangle -A OUTPUT -m tos --tos Minimize-Delay -j MARK --set-mark 0x1 $IPTABLES -t mangle -A OUTPUT -m tos --tos Minimize-Delay -j RETURN $IPTABLES -t mangle -A OUTPUT -m tos --tos Minimize-Cost -j MARK --set-mark 0x5 $IPTABLES -t mangle -A OUTPUT -m tos --tos Minimize-Cost -j RETURN $IPTABLES -t mangle -A OUTPUT -m tos --tos Maximize-Throughput -j MARK --set-mark 0x6 $IPTABLES -t mangle -A OUTPUT -m tos --tos Maximize-Throughput -j RETURN #Esto prioriza paquetes del puerto seteado $IPTABLES -t mangle -A OUTPUT -p tcp -m tcp --sport 80 -j MARK --set-mark 0x2 $IPTABLES -t mangle -A OUTPUT -p tcp -m tcp --sport 80 -j RETURN $IPTABLES -t mangle -A OUTPUT -p udp -m udp --sport 80 -j MARK --set-mark 0x2 $IPTABLES -t mangle -A OUTPUT -p udp -m udp --sport 80 -j RETURN $IPTABLES -t mangle -A OUTPUT -p tcp -m tcp --sport 54661 -j MARK --set-mark 0x6 $IPTABLES -t mangle -A OUTPUT -p tcp -m tcp --sport 54661 -j RETURN $IPTABLES -t mangle -A OUTPUT -p tcp -m tcp --sport 54662 -j MARK --set-mark 0x6 $IPTABLES -t mangle -A OUTPUT -p tcp -m tcp --sport 54662 -j RETURN $IPTABLES -t mangle -A OUTPUT -p tcp -m tcp --sport 56881 -j MARK --set-mark 0x6 $IPTABLES -t mangle -A OUTPUT -p tcp -m tcp --sport 56881 -j RETURN $IPTABLES -t mangle -A OUTPUT -p tcp -m tcp --sport 54711 -j MARK --set-mark 0x6 $IPTABLES -t mangle -A OUTPUT -p tcp -m tcp --sport 54711 -j RETURN $IPTABLES -t mangle -A OUTPUT -p udp -m udp --sport 54665 -j MARK --set-mark 0x6 $IPTABLES -t mangle -A OUTPUT -p udp -m udp --sport 54665 -j RETURN $IPTABLES -t mangle -A OUTPUT -p udp -m udp --sport 54672 -j MARK --set-mark 0x6 $IPTABLES -t mangle -A OUTPUT -p udp -m udp --sport 54672 -j RETURN $IPTABLES -t mangle -A OUTPUT -p udp -m udp --sport 56881 -j MARK --set-mark 0x6 $IPTABLES -t mangle -A OUTPUT -p udp -m udp --sport 56881 -j RETURN #Esto prioriza paquetes al comienzo de conexiones tcp con SYN flag $IPTABLES -t mangle -A OUTPUT -p tcp -m tcp --tcp-flags SYN,RST,ACK SYN -j MARK --set-mark 0x1 $IPTABLES -t mangle -A OUTPUT -p tcp -m tcp --tcp-flags SYN,RST,ACK SYN -j RETURN #Cierra reglas de la tabla OUTPUT mangle $IPTABLES -t mangle -A OUTPUT -j MARK --set-mark 0x3 --------------------------end paste---------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ds9a.nl/pipermail/lartc/attachments/20050706/77230d59/attachment.htm From gdamjan at mail.net.mk Wed Jul 6 14:48:36 2005 From: gdamjan at mail.net.mk (Damjan) Date: Wed Jul 6 16:10:43 2005 Subject: [LARTC] Frustrating problem with natsemi (DP83815) ethernet devices Message-ID: <20050706124836.GB18695@legolas.on.net.mk> [Sorry if this is off-topic for this list, please advise me if there's a better place to ask this] I have an embeded motherboard with 4 natsemi (DP8315) ethernet chips. I'm running a 2.6.9 kernel (I've tried 2.6.11 too) and a very minimal Slackware distribution. Now about the problem: The network interfaces seem to NOT transmit anything when the interface is autonegotiated at 100Mbit-FD. I've forced the interfaces to half-duplex with mii-tool still nothing. I see, with tcpdump, that packets are comming in (broadcasts and multicasts) and I see the packets that are supposed to go out of the interface. But nothing is recevied on the other side. I've tried to put the interfaces in a switch and with a cross-over cable to another PC. I've also tried to mannually fix the MAC <-> IP address with "arp -s", thinking that maybe ARP is not working, still nothing - I see icmp requests enter the interface I see icmp replies go out of the interface, but nothing gets to the other side (PC). I've tried all of the interfaces, I've tried enabling and disabling ACPI. The only modules I'have loaded are uhci-hcd and usbhid (since the computer doesn't have ps2 ports). 3 of the 4 interfaces don't share an IRQ with anything (cat /proc/interrupts). If I force the interface to 10Mbit (either FD or HD) then everything works. Now the stangest thing, I got the motherboard with some kind of basterized Debian installed on it. I couldn't understand it, so I backuped it up and installed my mini Slackware. I used the same kernel from the original distribution which has the ethernet driver compiled in. The original distribution is not setting any extra parameters in lilo.conf. And with that distribution hte interfaces work even at 100mbit. I just can't find what that distribution does to make the interfaces work at 100mbit. Its not the kernel, I use the same one. Its not some parameters to the driver... I don't know what could be the problem. So, any ideas??? -- damjan | ?????? This is my jabber ID --> damjan@bagra.net.mk <-- not my mail address!!! From jlynch at frink.nuigalway.ie Wed Jul 6 16:54:39 2005 From: jlynch at frink.nuigalway.ie (Jonathan Lynch) Date: Wed Jul 6 16:55:01 2005 Subject: [LARTC] Diffserv using HTB or PRIO qdiscs as basis for EF phb Message-ID: <1120661679.14438.125.camel@pgala.it.nuigalway.ie> Im experimenting with using a Linux machine with 3 interfaces to operate as a core router. All links and network interfaces are 100mbit. Below is a ascii diagram of the network. (network A) edge router ------>core router---->edge router (network C) ^ | | edge router (network B) I have tried two setups to support the Expedited forwarding PHB on the core router using the HTB qdisc and PRIO qdisc. The core router is a P4 2.8ghz, 512mb RAM running fedora core 3 with the 2.6 kernel. The Voip traffic is marked with the EF codepoint. Im using a local stratum 1 ntp server to synchronise time. The config I used for each setup is included at the bottom. When there is just voice traffic passing through (350 kbit) the delay of the voice traffic in travelling between network A and network C is 0.25ms (milliseconds) for both setups (using htb and PRIO). When TCP Traffic + VoIP traffic( 350 Kbit ) are sent from network A to network C. TCP traffic from network B is sent to network C. A bottleneck is created at the 3rd interface of the core network. Between the two network interfaces from network A and network B TCP traffic is coming in at a rate faster than 100 Mbit. As a result the delay is 30ms for both setups (using htb and PRIO) There is 0% loss of EF traffic. When i used a plain first in first out queue and saturated the link with tcp traffic the delay of the voice traffic was 160ms with 5% loss of voip traffic. Why is this delay so high when the link is saturated and Im using either PRIO or HTB to apply the EF PHB to VoIP traffic ???? Is it operating system ? Has anyone else had similar experiences ? Jonathan ---------------------------------------------------------------------------- The following are the configurations I have used for the core router Config 1 using htb tc qdisc add dev $1 handle 1:0 root dsmark indices 64 set_tc_index tc filter add dev $1 parent 1:0 protocol ip prio 1 tcindex mask 0xfc shift 2 Main htb qdisc & class tc qdisc add dev $1 parent 1:0 handle 2:0 htb tc class add dev $1 parent 2:0 classid 2:1 htb rate 100Mbit ceil 100Mbit EF Class (2:10) tc class add dev $1 parent 2:1 classid 2:10 htb rate 1500Kbit ceil 100Mbit tc qdisc add dev $1 parent 2:10 pfifo limit 5 tc filter add dev $1 parent 2:0 protocol ip prio 1 handle 0x2e tcindex classid 2:10 pass_on BE Class (2:20) tc class add dev $1 parent 2:1 classid 2:20 htb rate 5Mbit ceil 100Mbit tc qdisc add dev $1 parent 2:20 red limit 60KB min 15KB max 45KB burst 20 avpkt 1000 bandwidth 100Mbit probability 0.4 tc filter add dev $1 parent 2:0 protocol ip prio 2 handle 0 tcindex mask 0 classid 2:20 pass_on Config 2 using PRIO Main dsmark & classifier tc qdisc add dev $1 handle 1:0 root dsmark indices 64 set_tc_index tc filter add dev $1 parent 1:0 protocol ip prio 1 tcindex mask 0xfc shift 2 Main prio queue tc qdisc add dev $1 parent 1:0 handle 2:0 prio tc qdisc add dev $1 parent 2:1 tbf rate 1.5Mbit burst 1.5kB limit 1.6kB tc filter add dev $1 parent 2:0 protocol ip prio 1 handle 0x2e tcindex classid 2:1 pass_on BE class(2:2) tc qdisc add dev $1 parent 2:2 red limit 60KB min 15KB max 45KB burst 20 avpkt 1000 bandwidth 100Mbit probability 0.4 tc filter add dev $1 parent 2:0 protocol ip prio 2 handle 0 tcindex mask 0 classid 2:2 pass_on From kirk at braille.uwo.ca Wed Jul 6 16:58:55 2005 From: kirk at braille.uwo.ca (Kirk Reiser) Date: Wed Jul 6 16:58:59 2005 Subject: [LARTC] can't figure out nat'ing by port Message-ID: Hello folks: This may have been discussed many times before but I have not been able to find it. I have also not been able to resolve it myself so I am asking here with hope that someone can straighten me out. I am using ip to do multisource policy routing or two connections to the internet. I have a linksys wrt54gs route which connects two machines by wire through the switch and three computers through wireless. For the most part his portion of the set-up works fine. One of the boxes attached to the router by 10base-t also has a second interface card which attaches to an adsl modem using pppoe. My problem is on that second machine. I cannot figure out how to forward packets by incoming or destination port to the first machine for some services which it handles such as mail/dns/ftp.... I have read through quite a few docs such as Matt Marsh's policy routing and the lartc how-to on nat'ing and so forth. After reading these I think I understand what is going on until I try to do it myself and then I either get errors or it just doesn't work. I have tried using iptables and route/rule nat with no luck so far. It seems to me this should be a FAQ but nothing seems to quite match. I will try to draw how I invision the lay-out of the network but apologize because I'm a blink and am not sure I can get the drawing just right. internet internet | | wrt54gs dsl-mdm / \ / host-a host-b 2 nics I am not sure whether to put here what I have tried and hope someone can point out my mistakes or just hope someone will give me a nice little script! 'grin' Okay, okay, here are the iptables lines I tried first: iptables -t nat -A PREROUTING -i ppp0 -p tcp --dport 25 -j DNAT --to 10.10.10.10 iptables -t nat -A POSTROUTING -o eth0 -p tcp --dport 25 -j MASQUERADE I have tried both eth0 and ppp0 in the -o flag above. I have also tried adding ppp0 with the ip addr add 10.10.10.10/32 dev ppp0 table 1 without much affect. Does it sound like I'm confused? I also tried using the fwmark with ip rule but run up against a brick wall which appears to be that one cannot do ip route add nat 69.63.33.227/32 via 10.10.10.10/32 table natit. It seems that tables are not aloud in the route nat or something. I'm also not quite sure how the nat 0 works in the ip rule statement so yes I am confused in a big way! I will include my policy route script below so you can see the bit I do have working. Any help or pointers will certainly be appreciated. Kirk #!/bin/bash IF1=eth0 IP1=10.10.10.15 P1=10.10.10.1 P1_NET=10.10.10.0 IF2=ppp0 IP2=69.63.33.227 P2=209.183.132.10 P2_NET=69.0.0.0 ip route add $P1_NET dev $IF1 src $IP1 table T1 ip route add default via $P1 table T1 ip route add $P2_NET dev $IF2 table T1 ip route add 127.0.0.0/8 dev lo table T1 ip route add $P2_NET dev $IF2 src $IP2 table T2 ip route add default via $P2 table T2 ip route add $P1_NET dev $IF1 table T2 ip route add 127.0.0.0/8 dev lo table T2 #ip route append $P1_NET dev $IF1 src $IP1 #ip route append $P2_NET dev $IF2 src $IP2 #ip route append default via $P1 ip rule add from $IP1 table T1 ip rule add from $IP2 table T2 ip route add default scope global nexthop via $P1 dev $IF1 weight 1 \ nexthop via $P2 dev $IF2 weight 1 -- Kirk Reiser The Computer Braille Facility e-mail: kirk@braille.uwo.ca University of Western Ontario phone: (519) 661-3061 From Andreas.Klauer at metamorpher.de Wed Jul 6 17:38:12 2005 From: Andreas.Klauer at metamorpher.de (Andreas Klauer) Date: Wed Jul 6 17:37:50 2005 Subject: [LARTC] Please: "judge" this script In-Reply-To: <000c01c58233$d8b2b2e0$026fa8c0@ricardo> References: <000c01c58233$d8b2b2e0$026fa8c0@ricardo> Message-ID: <200507061738.12131.Andreas.Klauer@metamorpher.de> On Wednesday 06 July 2005 16:05, Ricardo Chamorro wrote: > CEIL=768 [...] > tc class add dev eth1 parent 1: classid 1:1 htb rate ${CEIL}kbit ceil > ${CEIL}kbit I don't know if it's the cause of your problems, but the children of this class altogether have a guaranteed rate of 810kbit, whereas the parent only has 768kbit. It's hard to tell what HTB does in this case, so you should make sure that the children's rates add up to the parent's rate. Another problem could probably be that you are using a lot of SFQ qdiscs. If every single one of them can queue 128 packets, it might be too much. I reduced the SFQ queue length to 16 on my system for that reason. I also had a lot of weird thing happening due to the prio parameter of HTB. I think it's best not to use it in the beginning and only start experimenting with that parameter when you really need it. Are you shaping upload traffic at all? You don't really have much influence on download traffic (all HTB can do is drop packets). A shaping setup without upload shaping makes hardly any sense. Also, in your setup you limit eth1 to 768kbit in total. That's fine as long as there is no LAN traffic on that machine. However, that's hardly ever the case - as soon as you SSH on your machine, or use some kind of proxy (DNS caching, squid, ...), this LAN traffic will have to use the same classes as your internet download traffic, thus interfering with download speeds. HTH Andreas From viriketo at gmail.com Wed Jul 6 17:40:34 2005 From: viriketo at gmail.com (=?ISO-8859-1?Q?Llu=EDs_Batlle?=) Date: Wed Jul 6 17:40:42 2005 Subject: [LARTC] Trying the configuration in nano.txt In-Reply-To: <45219fb0050706022739bf840e@mail.gmail.com> References: <45219fb0050705082620853a6@mail.gmail.com> <1a87976105070601467c1ac8fb@mail.gmail.com> <45219fb005070602103337bebc@mail.gmail.com> <45219fb0050706022739bf840e@mail.gmail.com> Message-ID: <45219fb0050706084035a08b63@mail.gmail.com> UF. Sorry. I answered stupidly to your email. You replied about the Julian's patches... I've sent a big amount of email, and not only about a single configuration. I don't know why, but now the patches work. I swear I haven't changed anything in my configuration; simply, after rebooting, some minutes without changing anything, everything worked. Strange, strange. So, it works. Thanks! - Wow, I've learnt a lot about netfilter and policy routing... On 7/6/05, Llu?s Batlle wrote: > On 7/6/05, Vinay wrote: > > Hi, > > I had 3 cards, eth0 and eth1 connecting to 2 isps and eth2 > > connecting to lan. First i setup the address configuration for these > > cards without the gateway option. So i specified only Ip address and > > netmask. Then i gave the masquerade option for both the network cards > > and made necessary chnages in the firewall. Next I executed the > > script which i got from > > http://mailman.ds9a.nl/pipermail/lartc/2003q4/010372.html. After > > running the script the system was using both the connections . But > > since i hadnt use the patch, it was not able to detect the dead > > gateway. So if one of the link goes down, it creates problem. > I tried that way, and routing works _only for local connections_. It > doesn't work for NAT connections. So, the result is the same, as the > configuration I was trying is the same, which is set in the script you > gave. :) > > > > > Can u please explain how u have applied the patch . > I have not applied the patch in the configuration I'm trying to get > working :) So, by now, I don't use that patch. I explain howI think > the multipath routing + NAT doesn't work, in a mail I just sent. :) > But as there I explained how I understand that nothing may work > without patches (in the mail sent to the list, before this one), I'll > start trying with the patches applied. I applied them to a vanilla > kernel by: > cd /usr/src/linux > patch -p1 < the_patch.diff > > This results in changing some files, as expected. So the patching > should be well done. > > Thanks! > From wonka at linkabu.net Wed Jul 6 18:31:03 2005 From: wonka at linkabu.net (Eduardo Bejar) Date: Wed Jul 6 18:31:18 2005 Subject: [LARTC] Mark all traffic except browsing Message-ID: <200507061627.j66GRJu22934@mail.linkabu.net> Hi, I?d like to create two classes for HTB, one of them to limit traffic for everything except browsing and the other to limit traffic only for browsing. How can I mark all traffic except the one related to browsing with iptables? Currently I mark all traffic related to one IP with: iptables -A PREROUTING -i eth2 -t mangle -p tcp -s $IP_ADDRESS -j MARK --set-mark 10 Should I use three rules for my purpose? iptables -A PREROUTING -i eth2 -t mangle -p tcp -s $IP_ADDRESS --dport 1:79 -j MARK --set-mark 10 iptables -A PREROUTING -i eth2 -t mangle -p tcp -s $IP_ADDRESS --dport 81:65535 -j MARK --set-mark 10 iptables -A PREROUTING -i eth2 -t mangle -p tcp -s $IP_ADDRESS --dport 80 -j MARK --set-mark 11 Please tell me if this is correct, Regards, Edo From x11 at kitas.arturas.net Wed Jul 6 21:12:14 2005 From: x11 at kitas.arturas.net (=?UTF-8?B?QXJ0xatyYXMgxaBsYWp1cw==?=) Date: Wed Jul 6 20:09:28 2005 Subject: [LARTC] HFSC default upper-limit trouble Message-ID: <42CC2D0E.5010903@kitas.arturas.net> Hello, I'm having such problem with HFSC with following config: + tc qdisc del dev eth3 root + tc qdisc add dev eth3 root handle 1: hfsc default 2 + tc class add dev eth3 parent 1: classid 1:1 hfsc ls rate 512kbit ul rate 512kbit + tc class add dev eth3 parent 1:1 classid 1:2 hfsc ls rate 2kbit ul rate 400kbit + tc class add dev eth3 parent 1:1 classid 1:3 hfsc ls rate 32kbit ul rate 32kbit + tc class add dev eth3 parent 1:1 classid 1:4 hfsc ls rate 300kbit ul rate 300kbit Let's say i start to upload thru 1:3. the upper-limit applies, traffic doesn't do up more than 4kb/s. The 1:4 is still functional, but 1:2, the default class starts backlogging and dropping as hell: class hfsc 1: root Sent 0 bytes 0 pkts (dropped 0, overlimits 0) period 0 level 2 class hfsc 1:1 parent 1: ls m1 0bit d 0us m2 512000bit ul m1 0bit d 0us m2 512000bit Sent 0 bytes 0 pkts (dropped 0, overlimits 0) <-- This is weird too ^_^ period 2643 work 821712 bytes level 1 class hfsc 1:2 parent 1:1 ls m1 0bit d 0us m2 2000bit ul m1 0bit d 0us m2 400000bit Sent 477205 bytes 3874 pkts (dropped 0, overlimits 0) backlog 201p <-- HUH? (it goes even to 800p..1000p then it starts dropping) period 2494 work 456595 bytes level 0 class hfsc 1:3 parent 1:1 ls m1 0bit d 0us m2 32000bit ul m1 0bit d 0us m2 32000bit Sent 350599 bytes 558 pkts (dropped 0, overlimits 0) backlog 11p period 70 work 342761 bytes level 0 class hfsc 1:4 parent 1:1 ls m1 0bit d 0us m2 300000bit ul m1 0bit d 0us m2 300000bit Sent 22356 bytes 214 pkts (dropped 0, overlimits 0) period 212 work 22356 bytes level 0 The 1:1 shows no packets sent as you see.. Is this desirable behavior? The default class kinda becomes unusable. Can someone explain me such behavior? From Andreas.Klauer at metamorpher.de Wed Jul 6 20:26:53 2005 From: Andreas.Klauer at metamorpher.de (Andreas Klauer) Date: Wed Jul 6 20:26:33 2005 Subject: [LARTC] Mark all traffic except browsing In-Reply-To: <200507061627.j66GRJu22934@mail.linkabu.net> References: <200507061627.j66GRJu22934@mail.linkabu.net> Message-ID: <200507062026.53148.Andreas.Klauer@metamorpher.de> On Wednesday 06 July 2005 18:31, Eduardo Bejar wrote: > Should I use three rules for my purpose? Two should suffice; just set all packets from $IP_ADDRESS to 10 first and afterwards set all packets for port 80 to 11. I think this only makes sense if you have two classes per source IP, though, because otherwise you could just use one tc filter rule to put port 80 packets into one class and let the rest go into the default class. HTH Andreas From o7sh at actcom.net.il Wed Jul 6 22:53:46 2005 From: o7sh at actcom.net.il (o7sh@actcom.net.il) Date: Wed Jul 6 22:53:48 2005 Subject: [LARTC] Server and router on the same box Message-ID: <200507062353.46601.o7sh@actcom.net.il> Hello list. I'm trying to set up traffic shaping on a (web) server which is also a router(NAT) for my LAN. It is connected to the LAN via Ethernet and also has PPP Internet connection on another interface. I need traffic shaping to make sure that the upstream is fully available to the web-server when it's active so when a LAN client is bulk downloading the web-server clients won't wait for hours. Its kind giving high priority to the upstream and lowering the downstream. it will be nice that when the web-server is not using any upstream, LAN clients will enjoy full bandwidth. I've found lots of information on how to ensure downloading while uploading. but not on the reverse way. Any idea for such a setup ? Thanks. From donvodka at gmail.com Wed Jul 6 23:23:38 2005 From: donvodka at gmail.com (Edgar) Date: Wed Jul 6 23:23:54 2005 Subject: [LARTC] HTB and bittorrent, won't work Message-ID: <200507061623.38806.donvodka@gmail.com> Hello, I've been trying to shape the bittorrent traffic (on my external interface, upload), but without luck, for this I'm using layer7 filter right now, but I've also tried ipp2p, with the same results, I might say that this is not a problem with this packet classifiers, the problem is with HTB, here's why. When I open azureus (the bittorrent client I use) I see upload traffic getting shapped, but also I see that my download traffic won't go up if I'm shaping on the upload interface, if I stop shaping on that interface then upload ( as expected) will increase, and so the download rate, this happens to me using the default bittorrent client (classic), so its not a client problem. Ok, the problem here is that when using bittorrent, although I see the traffic is shaped I can't surf web pages, nor chat in msn messenger, nor do anything at all, and merely that's all I want to do, shape p2p traffic to be able to use my bandwidth fairly, maybe its a bittorrent problem, because with the edonkey protocol I have no problem at all, traffic get shaped and I can use the rest of my bandwidth, I'll post my iptables rules for marking the bittorrent packets and the htb rules I use (using tcng): ### IPTABLES RULES ### iptables -t mangle -F iptables -t mangle -X iptables -t mangle -N lay7 iptables -t mangle -A POSTROUTING -j lay7 iptables -t mangle -A lay7 -m layer7 --l7proto bittorrent -j MARK --set-mark 1 iptables -t mangle -A lay7 -o eth1 -m mark --mark 1 -j CLASSIFY --set-class 2:2 iptables -t mangle -A lay7 -m layer7 --l7proto smtp -j MARK --set-mark 2 iptables -t mangle -A lay7 -o eth1 -m mark --mark 2 -j CLASSIFY --set-class 2:3 ### HTB RULES ### #define UPLOAD eth1 #define UPRATE 25kBps #define P2P 10kBps dev UPLOAD { egress { class ( <$emule> ) ; class ( <$smtp> ) ; class ( <$ssh> ) if tcp_dport == 8080 ; /*Changed port from 22 to 8080 */ class ( <$otro> ) if 1 ; htb () { class ( rate UPRATE, ceil UPRATE ) { $emule = class ( prio 8, rate 6kBps, ceil P2P ) { sfq; } ; $smtp = class ( prio 1, rate 6kBps, ceil 12kBps ) { sfq; } ; $ssh = class ( prio 0, rate 3kBps, ceil 5kBps) { sfq; } ; $otro = class ( prio 1, rate 8kBps, ceil UPRATE ) { sfq; } ; } } } } Also, given the priorities it's expected to let me surf the web or chat in msn messenger rather than take my whole bandwidth. I hope someone can help me out with this, maybe it not ok to use tcng with iptables? thank you in advance EDGAR MERINO From gypsy at iswest.com Thu Jul 7 05:11:08 2005 From: gypsy at iswest.com (gypsy) Date: Thu Jul 7 05:10:55 2005 Subject: [LARTC] Frustrating problem with natsemi (DP83815) ethernet devices References: <20050706124836.GB18695@legolas.on.net.mk> Message-ID: <42CC9D4C.2A815D0D@iswest.com> Damjan wrote: > > [Sorry if this is off-topic for this list, please advise me if there's a > better place to ask this] > > I have an embeded motherboard with 4 natsemi (DP8315) ethernet chips. > I'm running a 2.6.9 kernel (I've tried 2.6.11 too) and a very minimal > Slackware distribution. > > Now about the problem: > The network interfaces seem to NOT transmit anything when the interface > is autonegotiated at 100Mbit-FD. I've forced the interfaces to > half-duplex with mii-tool still nothing. > > I see, with tcpdump, that packets are comming in (broadcasts and multicasts) > and I see the packets that are supposed to go out of the interface. But > nothing is recevied on the other side. I've tried to put the interfaces > in a switch and with a cross-over cable to another PC. > > I've also tried to mannually fix the MAC <-> IP address with "arp -s", > thinking that maybe ARP is not working, still nothing - I see icmp > requests enter the interface I see icmp replies go out of the interface, > but nothing gets to the other side (PC). > > I've tried all of the interfaces, I've tried enabling and disabling > ACPI. The only modules I'have loaded are uhci-hcd and usbhid (since the > computer doesn't have ps2 ports). 3 of the 4 interfaces don't share an > IRQ with anything (cat /proc/interrupts). > > If I force the interface to 10Mbit (either FD or HD) then everything works. > > Now the stangest thing, I got the motherboard with some kind of > basterized Debian installed on it. I couldn't understand it, so I backuped > it up and installed my mini Slackware. I used the same kernel from the > original distribution which has the ethernet driver compiled in. The > original distribution is not setting any extra parameters in lilo.conf. > And with that distribution hte interfaces work even at 100mbit. > > I just can't find what that distribution does to make the interfaces work > at 100mbit. Its not the kernel, I use the same one. Its not some > parameters to the driver... I don't know what could be the problem. > > So, any ideas??? > > -- > damjan | ???????????? > This is my jabber ID --> damjan@bagra.net.mk <-- not my mail address!!! What have you tried? What is the working distro _EXACTLY_? "some kind of basterized Debian" is not enough. What does dmesg say about the DP8315? Have you looked at the Debian ifconfig? Is the same cable being used? Same cable path? I found a 10 Mb hub in a customer setup that was hidden behind the next desk where the Bad Boy computer plugged in. The computer thought the connection was 100Mb but it would only work at 10. Sorry, gotta go. Google the distro and DP8315. -- gypsy From Andreas.Klauer at metamorpher.de Thu Jul 7 06:29:47 2005 From: Andreas.Klauer at metamorpher.de (Andreas Klauer) Date: Thu Jul 7 06:29:38 2005 Subject: [LARTC] HTB and bittorrent, won't work In-Reply-To: <200507061623.38806.donvodka@gmail.com> References: <200507061623.38806.donvodka@gmail.com> Message-ID: <200507070629.47072.Andreas.Klauer@metamorpher.de> On Wednesday 06 July 2005 23:23, Edgar wrote: > I've been trying to shape the bittorrent traffic (on my external > interface, upload), but without luck, for this I'm using layer7 filter > right now, but I've also tried ipp2p, with the same results I don't have any problems with BT shaping... if you want to have a look at my script, it's this one: http://www.metamorpher.de/fairnat/ It uses IPP2P, but should work about the same way with layer7. The main difference between my iptables rules and yours seems to be that you are not using CONNMARK to mark BT connections permamently - IIRC you can't do without because the filters only match one of the first few packets of a connection. Have you checked your class statistics with tc, somehow I doubt that all BT packets go into your P2P class in your current setup. > I hope someone can help me out with this, maybe it not ok to use tcng > with iptables? thank you in advance Well, I'm not familiar with tcng syntax at all. I think this kind of setup is weird; in the tcng part no class ids seem to be specified, yet you have to use them in iptables to classify your packets. How can you be sure that you got the right class... HTH Andreas From ricardo.a.chamorro at gmail.com Thu Jul 7 06:30:42 2005 From: ricardo.a.chamorro at gmail.com (Ricardo Chamorro) Date: Thu Jul 7 06:30:31 2005 Subject: [LARTC] Please: "judge" this script References: <000c01c58233$d8b2b2e0$026fa8c0@ricardo> <200507061738.12131.Andreas.Klauer@metamorpher.de> Message-ID: <007c01c582ac$a3eed010$026fa8c0@ricardo> ----- Original Message ----- From: "Andreas Klauer" To: Sent: Wednesday, July 06, 2005 12:38 PM Subject: Re: [LARTC] Please: "judge" this script > class altogether have a guaranteed rate of 810kbit, whereas the parent > only has 768kbit. Oh Yes... thanks for your "judgment" so detailed... that went an error, when increasing one of the values I forgot to reduce it to the other class. After this correction the rate raised vertiginously. Of 4kbit of download in emule, now is 30 to 35 kb! In addition the load of www pages and pop3 works much more fast. > Another problem could probably be that you are using a lot of SFQ qdiscs. > If every single one of them can queue 128 packets, it might be too much. > I reduced the SFQ queue length to 16 on my system for that reason. > I also had a lot of weird thing happening due to the prio parameter of > HTB. > I think it's best not to use it in the beginning and only start > experimenting with that parameter when you really need it. You think that it would be necessary to make these changes now? > Are you shaping upload traffic at all? You don't really have much > influence > on download traffic (all HTB can do is drop packets). A shaping setup > without upload shaping makes hardly any sense. In this case what would have to add I to script? I imagine that I would have to apply all the same rules, equal these, but with the CEIL value upload (256 kbit) and pointing at the NIC eth0 (to ISP)... This is well? And the rules of iptables would be equal? That is: - t mangle PREROUTING pointing to the ports of destination (dport instead of sport)? > Also, in your setup you limit eth1 to 768kbit in total. That's fine as > long > as there is no LAN traffic on that machine. However, that's hardly ever > the case - as soon as you SSH on your machine, or use some kind of proxy > (DNS caching, squid, ...), this LAN traffic will have to use the same > classes as your internet download traffic, thus interfering with download > speeds. Is a router Debian (kernel to 2,4,25) that only does NAT and firewall with iptables (its "runs" in a 486 DX4100 with 96MB RAM and two HD of 1GB each one... ;-). Not squid, not samba, not bind... Nothing except router of Inet and NAT. What would have to do? Best regards Ricardo From jody.shumaker at gmail.com Thu Jul 7 06:30:40 2005 From: jody.shumaker at gmail.com (Jody Shumaker) Date: Thu Jul 7 06:31:01 2005 Subject: [LARTC] HTB and bittorrent, won't work In-Reply-To: <200507061623.38806.donvodka@gmail.com> References: <200507061623.38806.donvodka@gmail.com> Message-ID: <42CCAFF0.3030305@gmail.com> You need to use connection marking as well. --l7proto bittorrent will only recognize the first packet in a bittorrent stream, you need to save a mark on the whole tcp connection, and restore the mark for all future packets if you want the entire connection to be classified. iptables -t mangle -A lay7 -p tcp -j CONNMARK --restore-mark iptables -t mangle -A lay7 -m layer7 --l7proto bittorrent -j MARK --set-mark 1 iptables -t mangle -A lay7 -o eth1 -m mark --mark 1 -j CLASSIFY --set-class 2:2 iptables -t mangle -A lay7 -m layer7 --l7proto smtp -j MARK --set-mark 2 iptables -t mangle -A lay7 -o eth1 -m mark --mark 2 -j CLASSIFY --set-class 2:3 iptables -t mangle -A lay7 -p tcp -m mark ! --mark 0 -j CONNMARK --save-mark If you're marking ever gets more complex, it might take a little more work ( -j accepts for matching already classified connections after the --restore-mark) but the above should help get the full bittorrent connection classified, not just the first packet. - Jody Edgar wrote: >Hello, > >I've been trying to shape the bittorrent traffic (on my external interface, >upload), but without luck, for this I'm using layer7 filter right now, but >I've also tried ipp2p, with the same results, I might say that this is not a >problem with this packet classifiers, the problem is with HTB, here's why. >When I open azureus (the bittorrent client I use) I see upload traffic >getting shapped, but also I see that my download traffic won't go up if I'm >shaping on the upload interface, if I stop shaping on that interface then >upload ( as expected) will increase, and so the download rate, this happens >to me using the default bittorrent client (classic), so its not a client >problem. Ok, the problem here is that when using bittorrent, although I see >the traffic is shaped I can't surf web pages, nor chat in msn messenger, nor >do anything at all, and merely that's all I want to do, shape p2p traffic to >be able to use my bandwidth fairly, maybe its a bittorrent problem, because >with the edonkey protocol I have no problem at all, traffic get shaped and I >can use the rest of my bandwidth, I'll post my iptables rules for marking the >bittorrent packets and the htb rules I use (using tcng): > >### IPTABLES RULES ### >iptables -t mangle -F >iptables -t mangle -X >iptables -t mangle -N lay7 >iptables -t mangle -A POSTROUTING -j lay7 >iptables -t mangle -A lay7 -m layer7 --l7proto bittorrent -j MARK --set-mark 1 >iptables -t mangle -A lay7 -o eth1 -m mark --mark 1 -j CLASSIFY --set-class >2:2 >iptables -t mangle -A lay7 -m layer7 --l7proto smtp -j MARK --set-mark 2 >iptables -t mangle -A lay7 -o eth1 -m mark --mark 2 -j CLASSIFY --set-class >2:3 > >### HTB RULES ### > >#define UPLOAD eth1 >#define UPRATE 25kBps >#define P2P 10kBps > >dev UPLOAD { > egress { > class ( <$emule> ) ; > class ( <$smtp> ) ; > class ( <$ssh> ) if tcp_dport == 8080 ; /*Changed port from 22 to 8080 */ > class ( <$otro> ) if 1 ; > > htb () { > class ( rate UPRATE, ceil UPRATE ) { > $emule = class ( prio 8, rate 6kBps, ceil P2P ) { sfq; } ; > $smtp = class ( prio 1, rate 6kBps, ceil 12kBps ) { sfq; } ; > $ssh = class ( prio 0, rate 3kBps, ceil 5kBps) { sfq; } ; > $otro = class ( prio 1, rate 8kBps, ceil UPRATE ) { sfq; } ; > } > } > } >} > >Also, given the priorities it's expected to let me surf the web or chat in msn >messenger rather than take my whole bandwidth. > >I hope someone can help me out with this, maybe it not ok to use tcng with >iptables? thank you in advance > >EDGAR MERINO >_______________________________________________ >LARTC mailing list >LARTC@mailman.ds9a.nl >http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > > > From Andreas.Klauer at metamorpher.de Thu Jul 7 06:39:18 2005 From: Andreas.Klauer at metamorpher.de (Andreas Klauer) Date: Thu Jul 7 06:39:14 2005 Subject: [LARTC] Server and router on the same box In-Reply-To: <200507062353.46601.o7sh@actcom.net.il> References: <200507062353.46601.o7sh@actcom.net.il> Message-ID: <200507070639.18899.Andreas.Klauer@metamorpher.de> On Wednesday 06 July 2005 22:53, o7sh@actcom.net.il wrote: > it will be nice that when the web-server is not using any upstream, LAN > clients will enjoy full bandwidth. Somehow I doubt that traffic shaping is the right approach here; after all, we're talking about traffic on two different interfaces. There is no need to do shaping in that situation, because there is no way that traffic on interface A interferes with traffic on interface B (unless one is a virtual interface that actually uses the other to transfer). Even if you put HTB on interface A, this qdisc has no way of knowing how much load there is on another interface. To shape multiple interfaces at once, you'd have to cheat by using IMQ or something like that. If the fast transfer rates of the LAN puts too much load on your web server, the only thing you could do with traffic shaping here is to limit these transfer rates to a lower value at all times. A better method would be to configure this in the webserver; tell the webserver that internet clients are more important than LAN clients. Depending on which webserver you use, this might be possible or not. HTH Andreas From donvodka at gmail.com Thu Jul 7 06:51:54 2005 From: donvodka at gmail.com (Edgar) Date: Thu Jul 7 06:52:20 2005 Subject: [LARTC] HTB and bittorrent, won't work In-Reply-To: <200507070629.47072.Andreas.Klauer@metamorpher.de> References: <200507061623.38806.donvodka@gmail.com> <200507070629.47072.Andreas.Klauer@metamorpher.de> Message-ID: <200507062351.54553.donvodka@gmail.com> First of all thank you for answering to my email, I will answer to all the questions you ask: > On Wednesday 06 July 2005 23:23, Edgar wrote: > > I've been trying to shape the bittorrent traffic (on my external > > interface, upload), but without luck, for this I'm using layer7 filter > > right now, but I've also tried ipp2p, with the same results > > I don't have any problems with BT shaping... if you want to have a look at > my script, it's this one: http://www.metamorpher.de/fairnat/ > It uses IPP2P, but should work about the same way with layer7. > > The main difference between my iptables rules and yours seems to be that > you are not using CONNMARK to mark BT connections permamently - IIRC you > can't do without because the filters only match one of the first few > packets of a connection. > In the layer7 filter the examples don't use CONNMARK like with ipp2p, but I will try to add CONNMARK to my rules, and see what happens. > Have you checked your class statistics with tc, somehow I doubt that all BT > packets go into your P2P class in your current setup. > I've checked this with tc -s class show dev eth1, and I see almost all the traffic going to the p2p class, right now I will add a new ACK rule, to match ack packets, since someone told me that might be the problem. > > I hope someone can help me out with this, maybe it not ok to use tcng > > with iptables? thank you in advance > > Well, I'm not familiar with tcng syntax at all. I think this kind of setup > is weird; in the tcng part no class ids seem to be specified, yet you have > to use them in iptables to classify your packets. How can you be sure that > you got the right class... When the tcng code gets compiled, it returns tc commands, and then I can see them, and I know those are the classes I need for iptables =) Thanks for your interest, I'll check your rules right now, to see if that helps. > > HTH > Andreas > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc EDGAR MERINO From lartc at nospam.otaku42.de Thu Jul 7 07:38:54 2005 From: lartc at nospam.otaku42.de (Michael Renzmann) Date: Thu Jul 7 07:39:01 2005 Subject: [LARTC] Frustrating problem with natsemi (DP83815) ethernet devices In-Reply-To: <20050706124836.GB18695@legolas.on.net.mk> References: <20050706124836.GB18695@legolas.on.net.mk> Message-ID: <42CCBFEE.9020602@otaku42.de> Hi. Damjan wrote: > Now about the problem: > The network interfaces seem to NOT transmit anything when the interface > is autonegotiated at 100Mbit-FD. I've forced the interfaces to > half-duplex with mii-tool still nothing. Several questions... Is it really absolutely the same kernel on both distros? Which one is it? Did you try another cable? Did you try another switch? Did you try a cross-cable to connect two of the four interfaces on the same host? NSC brought out a completely new driver some weeks ago, which can be found here: http://www.national.com/appinfo/networks/files/dp8381x_linux_ver_1.0.tgz In case you've a 2.4 kernel let me know, I have yet another driver you could try out. Bye, Mike From donvodka at gmail.com Thu Jul 7 07:42:23 2005 From: donvodka at gmail.com (Edgar) Date: Thu Jul 7 07:42:39 2005 Subject: [LARTC] HTB and bittorrent, won't work In-Reply-To: <42CCAFF0.3030305@gmail.com> References: <200507061623.38806.donvodka@gmail.com> <42CCAFF0.3030305@gmail.com> Message-ID: <200507070042.23652.donvodka@gmail.com> Hi, thanks for your help and interest, someone told me about that already, so I did it, and this is the script I'm running to do it: #!/bin/sh ### ERASING RULES AND USER CREATED CHAINS ### iptables -t mangle -F iptables -t mangle -X iptables -t mangle -N lay7PRE iptables -t mangle -N lay7POST ### PREROUTING RULES ### iptables -t mangle -A lay7PRE -j CONNMARK --restore-mark iptables -t mangle -A lay7PRE -m mark ! --mark 0 -j ACCEPT iptables -t mangle -A lay7PRE -m layer7 --l7proto bittorrent -j MARK --set-mark 1 iptables -t mangle -A lay7PRE -m layer7 --l7proto smtp -j MARK --set-mark 2 iptables -t mangle -A lay7PRE -m layer7 --l7proto http -j MARK --set-mark 3 iptables -t mangle -A lay7PRE -j CONNMARK --save-mark ### POSTROUTING RULES ### iptables -t mangle -A lay7POST -o eth1 -m mark --mark 1 -j CLASSIFY --set-class 2:2 iptables -t mangle -A lay7POST -o eth1 -m mark --mark 2 -j CLASSIFY --set-class 2:3 iptables -t mangle -A lay7POST -o eth1 -m mark --mark 3 -j CLASSIFY --set-class 2:4 ### ------------------------------------------------------------------- ### iptables -t mangle -A PREROUTING -j lay7PRE iptables -t mangle -A POSTROUTING -j lay7POST I'm trying this right now, and I believe its kind of working, but web surfing is very slow, I might say unusable, so this is not what I want, also I had to mark http traffic to make this work, give it a higher prio in htb, so I believe I'm missing something else? someone suggested to add a new class for ACK packets, I've done that already, but I've only noticed little difference... really don't know whats happening, if you don't have tcng I can show you my tc rules (showed by tc -s class show dev eth1). Thank you again EDGAR MERINO On Wednesday 06 July 2005 23:30, Jody Shumaker wrote: > You need to use connection marking as well. --l7proto bittorrent will > only recognize the first packet in a bittorrent stream, you need to save > a mark on the whole tcp connection, and restore the mark for all future > packets if you want the entire connection to be classified. > > iptables -t mangle -A lay7 -p tcp -j CONNMARK --restore-mark > iptables -t mangle -A lay7 -m layer7 --l7proto bittorrent -j MARK > --set-mark 1 iptables -t mangle -A lay7 -o eth1 -m mark --mark 1 -j > CLASSIFY --set-class 2:2 iptables -t mangle -A lay7 -m layer7 --l7proto > smtp -j MARK --set-mark 2 iptables -t mangle -A lay7 -o eth1 -m mark --mark > 2 -j CLASSIFY --set-class 2:3 iptables -t mangle -A lay7 -p tcp -m mark ! > --mark 0 -j CONNMARK --save-mark > > > If you're marking ever gets more complex, it might take a little more work > ( -j accepts for matching already classified connections after the > --restore-mark) but the above should help get the full bittorrent > connection classified, not just the first packet. > > - Jody > > Edgar wrote: > >Hello, > > > >I've been trying to shape the bittorrent traffic (on my external > > interface, upload), but without luck, for this I'm using layer7 filter > > right now, but I've also tried ipp2p, with the same results, I might say > > that this is not a problem with this packet classifiers, the problem is > > with HTB, here's why. When I open azureus (the bittorrent client I use) I > > see upload traffic getting shapped, but also I see that my download > > traffic won't go up if I'm shaping on the upload interface, if I stop > > shaping on that interface then upload ( as expected) will increase, and > > so the download rate, this happens to me using the default bittorrent > > client (classic), so its not a client problem. Ok, the problem here is > > that when using bittorrent, although I see the traffic is shaped I can't > > surf web pages, nor chat in msn messenger, nor do anything at all, and > > merely that's all I want to do, shape p2p traffic to be able to use my > > bandwidth fairly, maybe its a bittorrent problem, because with the > > edonkey protocol I have no problem at all, traffic get shaped and I can > > use the rest of my bandwidth, I'll post my iptables rules for marking the > > bittorrent packets and the htb rules I use (using tcng): > > > >### IPTABLES RULES ### > >iptables -t mangle -F > >iptables -t mangle -X > >iptables -t mangle -N lay7 > >iptables -t mangle -A POSTROUTING -j lay7 > >iptables -t mangle -A lay7 -m layer7 --l7proto bittorrent -j MARK > > --set-mark 1 iptables -t mangle -A lay7 -o eth1 -m mark --mark 1 -j > > CLASSIFY --set-class 2:2 > >iptables -t mangle -A lay7 -m layer7 --l7proto smtp -j MARK --set-mark 2 > >iptables -t mangle -A lay7 -o eth1 -m mark --mark 2 -j CLASSIFY > > --set-class 2:3 > > > >### HTB RULES ### > > > >#define UPLOAD eth1 > >#define UPRATE 25kBps > >#define P2P 10kBps > > > >dev UPLOAD { > > egress { > > class ( <$emule> ) ; > > class ( <$smtp> ) ; > > class ( <$ssh> ) if tcp_dport == 8080 ; /*Changed port from 22 to 8080 > > */ class ( <$otro> ) if 1 ; > > > > htb () { > > class ( rate UPRATE, ceil UPRATE ) { > > $emule = class ( prio 8, rate 6kBps, ceil P2P ) { sfq; } ; > > $smtp = class ( prio 1, rate 6kBps, ceil 12kBps ) { sfq; } ; > > $ssh = class ( prio 0, rate 3kBps, ceil 5kBps) { sfq; } ; > > $otro = class ( prio 1, rate 8kBps, ceil UPRATE ) { sfq; } ; > > } > > } > > } > >} > > > >Also, given the priorities it's expected to let me surf the web or chat in > > msn messenger rather than take my whole bandwidth. > > > >I hope someone can help me out with this, maybe it not ok to use tcng with > >iptables? thank you in advance > > > >EDGAR MERINO > >_______________________________________________ > >LARTC mailing list > >LARTC@mailman.ds9a.nl > >http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc From klaus at ipp2p.org Thu Jul 7 09:22:20 2005 From: klaus at ipp2p.org (Klaus) Date: Thu Jul 7 09:22:26 2005 Subject: [LARTC] HTB and bittorrent, won't work In-Reply-To: <200507070042.23652.donvodka@gmail.com> References: <200507061623.38806.donvodka@gmail.com> <42CCAFF0.3030305@gmail.com> <200507070042.23652.donvodka@gmail.com> Message-ID: <42CCD82C.5080600@ipp2p.org> ipp2p vs. l7 filter l7 uses regular expressions, so they are slower (some rules are EXTREME slow like fasttrack) and not so strong like the ipp2p rules (which can have for example packet length checks). ipp2p is specialized for p2p detection, so a many p2p packets are not detected by l7 (for example not all BitTorrent connections start with a 013h "BitTorrent"). The worst part is that l7 filter has some p2p rules which detect false positives: http://l7-filter.sourceforge.net/layer7-protocols/protocols/edonkey.pat "... This will match about 1% of streams with random data in them! ..." If you drop p2p connection, one of hundred downloads / web pages will fail (and fail every time) ? I would recommend l7-filter for everything but not for p2p. It is a VERY nice filter, but if they would have something else than regexp, i would use it maybe too. Klaus, Maintainer of ipp2p Edgar wrote: > Hi, thanks for your help and interest, someone told me about that already, so > I did it, and this is the script I'm running to do it: > #!/bin/sh > > ### ERASING RULES AND USER CREATED CHAINS ### > iptables -t mangle -F > iptables -t mangle -X > iptables -t mangle -N lay7PRE > iptables -t mangle -N lay7POST > > ### PREROUTING RULES ### > iptables -t mangle -A lay7PRE -j CONNMARK --restore-mark > iptables -t mangle -A lay7PRE -m mark ! --mark 0 -j ACCEPT > iptables -t mangle -A lay7PRE -m layer7 --l7proto bittorrent -j MARK > --set-mark 1 > iptables -t mangle -A lay7PRE -m layer7 --l7proto smtp -j MARK --set-mark 2 > iptables -t mangle -A lay7PRE -m layer7 --l7proto http -j MARK --set-mark 3 > iptables -t mangle -A lay7PRE -j CONNMARK --save-mark > > ### POSTROUTING RULES ### > iptables -t mangle -A lay7POST -o eth1 -m mark --mark 1 -j CLASSIFY > --set-class 2:2 > iptables -t mangle -A lay7POST -o eth1 -m mark --mark 2 -j CLASSIFY > --set-class 2:3 > iptables -t mangle -A lay7POST -o eth1 -m mark --mark 3 -j CLASSIFY > --set-class 2:4 > > ### ------------------------------------------------------------------- ### > iptables -t mangle -A PREROUTING -j lay7PRE > iptables -t mangle -A POSTROUTING -j lay7POST > > I'm trying this right now, and I believe its kind of working, but web surfing > is very slow, I might say unusable, so this is not what I want, also I had to > mark http traffic to make this work, give it a higher prio in htb, so I > believe I'm missing something else? someone suggested to add a new class for > ACK packets, I've done that already, but I've only noticed little > difference... really don't know whats happening, if you don't have tcng I can > show you my tc rules (showed by tc -s class show dev eth1). Thank you again > > EDGAR MERINO > > On Wednesday 06 July 2005 23:30, Jody Shumaker wrote: > >>You need to use connection marking as well. --l7proto bittorrent will >>only recognize the first packet in a bittorrent stream, you need to save >>a mark on the whole tcp connection, and restore the mark for all future >>packets if you want the entire connection to be classified. >> >>iptables -t mangle -A lay7 -p tcp -j CONNMARK --restore-mark >>iptables -t mangle -A lay7 -m layer7 --l7proto bittorrent -j MARK >>--set-mark 1 iptables -t mangle -A lay7 -o eth1 -m mark --mark 1 -j >>CLASSIFY --set-class 2:2 iptables -t mangle -A lay7 -m layer7 --l7proto >>smtp -j MARK --set-mark 2 iptables -t mangle -A lay7 -o eth1 -m mark --mark >>2 -j CLASSIFY --set-class 2:3 iptables -t mangle -A lay7 -p tcp -m mark ! >>--mark 0 -j CONNMARK --save-mark >> >> >>If you're marking ever gets more complex, it might take a little more work >>( -j accepts for matching already classified connections after the >>--restore-mark) but the above should help get the full bittorrent >>connection classified, not just the first packet. >> >>- Jody >> >>Edgar wrote: >> >>>Hello, >>> >>>I've been trying to shape the bittorrent traffic (on my external >>>interface, upload), but without luck, for this I'm using layer7 filter >>>right now, but I've also tried ipp2p, with the same results, I might say >>>that this is not a problem with this packet classifiers, the problem is >>>with HTB, here's why. When I open azureus (the bittorrent client I use) I >>>see upload traffic getting shapped, but also I see that my download >>>traffic won't go up if I'm shaping on the upload interface, if I stop >>>shaping on that interface then upload ( as expected) will increase, and >>>so the download rate, this happens to me using the default bittorrent >>>client (classic), so its not a client problem. Ok, the problem here is >>>that when using bittorrent, although I see the traffic is shaped I can't >>>surf web pages, nor chat in msn messenger, nor do anything at all, and >>>merely that's all I want to do, shape p2p traffic to be able to use my >>>bandwidth fairly, maybe its a bittorrent problem, because with the >>>edonkey protocol I have no problem at all, traffic get shaped and I can >>>use the rest of my bandwidth, I'll post my iptables rules for marking the >>>bittorrent packets and the htb rules I use (using tcng): >>> >>>### IPTABLES RULES ### >>>iptables -t mangle -F >>>iptables -t mangle -X >>>iptables -t mangle -N lay7 >>>iptables -t mangle -A POSTROUTING -j lay7 >>>iptables -t mangle -A lay7 -m layer7 --l7proto bittorrent -j MARK >>>--set-mark 1 iptables -t mangle -A lay7 -o eth1 -m mark --mark 1 -j >>>CLASSIFY --set-class 2:2 >>>iptables -t mangle -A lay7 -m layer7 --l7proto smtp -j MARK --set-mark 2 >>>iptables -t mangle -A lay7 -o eth1 -m mark --mark 2 -j CLASSIFY >>>--set-class 2:3 >>> >>>### HTB RULES ### >>> >>>#define UPLOAD eth1 >>>#define UPRATE 25kBps >>>#define P2P 10kBps >>> >>>dev UPLOAD { >>> egress { >>> class ( <$emule> ) ; >>> class ( <$smtp> ) ; >>> class ( <$ssh> ) if tcp_dport == 8080 ; /*Changed port from 22 to 8080 >>>*/ class ( <$otro> ) if 1 ; >>> >>> htb () { >>> class ( rate UPRATE, ceil UPRATE ) { >>> $emule = class ( prio 8, rate 6kBps, ceil P2P ) { sfq; } ; >>> $smtp = class ( prio 1, rate 6kBps, ceil 12kBps ) { sfq; } ; >>> $ssh = class ( prio 0, rate 3kBps, ceil 5kBps) { sfq; } ; >>> $otro = class ( prio 1, rate 8kBps, ceil UPRATE ) { sfq; } ; >>> } >>> } >>> } >>>} >>> >>>Also, given the priorities it's expected to let me surf the web or chat in >>>msn messenger rather than take my whole bandwidth. >>> >>>I hope someone can help me out with this, maybe it not ok to use tcng with >>>iptables? thank you in advance >>> >>>EDGAR MERINO >>>_______________________________________________ >>>LARTC mailing list >>>LARTC@mailman.ds9a.nl >>>http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc >> >>_______________________________________________ >>LARTC mailing list >>LARTC@mailman.ds9a.nl >>http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc From o7sh at actcom.net.il Thu Jul 7 10:51:12 2005 From: o7sh at actcom.net.il (o7sh@actcom.net.il) Date: Thu Jul 7 10:51:29 2005 Subject: [LARTC] Server and router on the same box In-Reply-To: <200507070639.18899.Andreas.Klauer@metamorpher.de> References: <200507062353.46601.o7sh@actcom.net.il> <200507070639.18899.Andreas.Klauer@metamorpher.de> Message-ID: <200507071151.13007.o7sh@actcom.net.il> Hi Andreas . And thank you for your reply. > A better method would be to configure this in the webserver; tell the > webserver that internet clients are more important than LAN clients. > Depending on which webserver you use, this might be possible or not. > > HTH > Andreas The main problem is that LAN clients connect to other web-servers on the Internet via this box. in fact, any Internet traffic they consume, even if they downloading, affect the upstream available for the web-server. Well, If i don't find a "dynamic" solution. I don't care lowering the upstream available for the LAN all the time.. let's say I'll shape the upstream on the Ethernet, will it solve the problem ? Thanks. From Andreas.Klauer at metamorpher.de Thu Jul 7 11:18:08 2005 From: Andreas.Klauer at metamorpher.de (Andreas Klauer) Date: Thu Jul 7 11:17:43 2005 Subject: [LARTC] Server and router on the same box In-Reply-To: <200507071151.13007.o7sh@actcom.net.il> References: <200507062353.46601.o7sh@actcom.net.il> <200507070639.18899.Andreas.Klauer@metamorpher.de> <200507071151.13007.o7sh@actcom.net.il> Message-ID: <200507071118.08420.Andreas.Klauer@metamorpher.de> On Thursday 07 July 2005 10:51, o7sh@actcom.net.il wrote: > The main problem is that LAN clients connect to other web-servers on the > Internet via this box. in fact, any Internet traffic they consume, even > if they downloading, affect the upstream available for the web-server. Oh, I'm sorry, I seem to have misunderstood. I thought the LAN clients were downloading from your web server, causing too much load for it. You were talking about Internet traffic, so that's a completely different story. To make upstream fully available to your webserver, I suggest using a PRIO qdisc with 2 bands... band one for webserver traffic, band two for everything else. The PRIO qdisc will make sure that your webserver always gets to send his packets first. LAN traffic will only be allowed to send when the webserver sent all his packets. However, this means that your LAN will get effectively zero bandwidth as long as the webserver is working (someone downloading from it). If this is too harsh for you, use HTB instead - there you can specify whatever balance between web server and LAN traffic you wish (50-50 or 75-25). HTH Andreas From donvodka at gmail.com Thu Jul 7 12:35:29 2005 From: donvodka at gmail.com (Edgar) Date: Thu Jul 7 12:35:47 2005 Subject: [LARTC] HELP PLEASE BITTORRENT SHAPING (HTB) Message-ID: <200507070535.29506.donvodka@gmail.com> Ok, earlier I post a message explaining my problem with HTB and layer7 (or ipp2p), about not being able to shape the traffic. Well, actually this is what's happening, I'm marking the packets (right now, I'm using ipp2p as Klaus adviced me to) with iptables, and my queue rules are made using tcng, I'm using the HTB qdisc, and traffic is going to the HTB class I order it to, but for some reason traffic is also going to the SMTP queue rule I have, I don't know why, and also I see traffic into the ACK queue, that seems to be normal I believe (but isn't it supposed to get marked as bittorrent traffic?), anyway, I have the doubt as of why is the traffic going to the smpt queue rule, if I remove this rule it'll go with the next rule (ssh), and I don't know why, it shouldn't do it, and I'm really worried, because I can't use my bandwidth, can't chat, browse the web, I can't do anything at all. Here are my new iptables rules, and tcng ones, please take a look at them, and tell me if you see any problem with them, and if someone had the same problem please let me know too. Also please if you think you can help, don't leave here with just one answer... because I don't know who else to ask, or where!, I'll really appreciate your help. Thank you ### IPTABLES RULES ### #!/bin/sh ### ERASING RULES AND USER MADE CHAINS ### iptables -t mangle -F iptables -t mangle -X iptables -t mangle -N lay7PRE iptables -t mangle -N lay7POST iptables -t mangle -N ipp2pPRE iptables -t mangle -N ipp2pPOST ### PREROUTING RULES ### iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark iptables -t mangle -A PREROUTING -m mark ! --mark 0 -j ACCEPT iptables -t mangle -A ipp2pPRE -p tcp -m ipp2p --ipp2p -j MARK --set-mark 10 iptables -t mangle -A ipp2pPRE -p udp -m ipp2p --bit -j MARK --set-mark 10 iptables -t mangle -A ipp2pPRE -p tcp -m mark --mark 10 -j CONNMARK --save-mark iptables -t mangle -A lay7PRE -m layer7 --l7proto smtp -j MARK --set-mark 1 iptables -t mangle -A lay7PRE -m mark --mark 1 -j CONNMARK --save-mark iptables -t mangle -A lay7PRE -m layer7 --l7proto ssh -j MARK --set-mark 2 iptables -t mangle -A lay7PRE -m mark --mark 2 -j CONNMARK --save-mark ### POSTROUTING RULES ### iptables -t mangle -A ipp2pPOST -o eth1 -m mark --mark 10 -j CLASSIFY --set-class 2:6 iptables -t mangle -A lay7POST -o eth1 -m mark --mark 1 -j CLASSIFY --set-class 2:2 iptables -t mangle -A lay7POST -o eth1 -m mark --mark 2 -j CLASSIFY --set-class 2:3 ### -------------------------------------------------------------------- ### iptables -t mangle -A PREROUTING -j lay7PRE iptables -t mangle -A POSTROUTING -j lay7POST iptables -t mangle -A PREROUTING -j ipp2pPRE iptables -t mangle -A POSTROUTING -j ipp2pPOST ### TCNG RULES ### #define UPLOAD eth1 #define UPRATE 25kBps #define P2P 10kBps dev UPLOAD { egress { class ( <$smtp> ) ; class ( <$ssh> ) if tcp_dport == 8080 ; class ( <$ack> ) if ip_hl == 0x5 && (ip_len & 0xffc0) && (raw[33].b >> 4) & 1 ; class ( <$otro> ) if 1 ; class ( <$p2p> ) ; htb () { class ( rate UPRATE, ceil UPRATE ) { $smtp = class ( prio 1, rate 6kBps, ceil 12kBps ) { sfq; } ; $ssh = class ( prio 0, rate 3kBps, ceil 5kBps ) { sfq; } ; $ack = class ( prio 2, rate 8kBps, ceil 15kBps ) { sfq; } ; $otro = class ( prio 1, rate 8kBps, ceil UPRATE ) { sfq; } $p2p = class ( prio 8, rate P2P, ceil P2P, cburst 0bits ) { sfq; } ; } } } } EDGAR MERINO From toto at fortesys.ro Thu Jul 7 13:36:50 2005 From: toto at fortesys.ro (Forte Systems - Iosif Peterfi) Date: Thu Jul 7 13:36:51 2005 Subject: [LARTC] HTB and bittorrent, won't work In-Reply-To: <42CCD82C.5080600@ipp2p.org> Message-ID: <20050707113647.AE2EE48BE@outpost.ds9a.nl> I would suggest classifing interactive connections, and leave all the bulk traffic in the default class. This way, the bt,kazaa,emule traffic will go in the same class, without additional filtering. Also, using HFSC instead of HTB helps you increase the delay of the default class. This way bulk traffic will be sent every n ms, leaving priority to the interactive/web/mail traffic. Think about it. Iosif Peterfi S.C. Forte Systems SRL http://www.fortesys.ro/ -----Original Message----- From: lartc-bounces@mailman.ds9a.nl [mailto:lartc-bounces@mailman.ds9a.nl] On Behalf Of Klaus Sent: Thursday, July 07, 2005 10:22 AM To: lartc@mailman.ds9a.nl Subject: Re: [LARTC] HTB and bittorrent, won't work ipp2p vs. l7 filter l7 uses regular expressions, so they are slower (some rules are EXTREME slow like fasttrack) and not so strong like the ipp2p rules (which can have for example packet length checks). ipp2p is specialized for p2p detection, so a many p2p packets are not detected by l7 (for example not all BitTorrent connections start with a 013h "BitTorrent"). The worst part is that l7 filter has some p2p rules which detect false positives: http://l7-filter.sourceforge.net/layer7-protocols/protocols/edonkey.pat "... This will match about 1% of streams with random data in them! ..." If you drop p2p connection, one of hundred downloads / web pages will fail (and fail every time) ? I would recommend l7-filter for everything but not for p2p. It is a VERY nice filter, but if they would have something else than regexp, i would use it maybe too. Klaus, Maintainer of ipp2p Edgar wrote: > Hi, thanks for your help and interest, someone told me about that already, so > I did it, and this is the script I'm running to do it: > #!/bin/sh > > ### ERASING RULES AND USER CREATED CHAINS ### > iptables -t mangle -F > iptables -t mangle -X > iptables -t mangle -N lay7PRE > iptables -t mangle -N lay7POST > > ### PREROUTING RULES ### > iptables -t mangle -A lay7PRE -j CONNMARK --restore-mark > iptables -t mangle -A lay7PRE -m mark ! --mark 0 -j ACCEPT > iptables -t mangle -A lay7PRE -m layer7 --l7proto bittorrent -j MARK > --set-mark 1 > iptables -t mangle -A lay7PRE -m layer7 --l7proto smtp -j MARK --set-mark 2 > iptables -t mangle -A lay7PRE -m layer7 --l7proto http -j MARK --set-mark 3 > iptables -t mangle -A lay7PRE -j CONNMARK --save-mark > > ### POSTROUTING RULES ### > iptables -t mangle -A lay7POST -o eth1 -m mark --mark 1 -j CLASSIFY > --set-class 2:2 > iptables -t mangle -A lay7POST -o eth1 -m mark --mark 2 -j CLASSIFY > --set-class 2:3 > iptables -t mangle -A lay7POST -o eth1 -m mark --mark 3 -j CLASSIFY > --set-class 2:4 > > ### ------------------------------------------------------------------- ### > iptables -t mangle -A PREROUTING -j lay7PRE > iptables -t mangle -A POSTROUTING -j lay7POST > > I'm trying this right now, and I believe its kind of working, but web surfing > is very slow, I might say unusable, so this is not what I want, also I had to > mark http traffic to make this work, give it a higher prio in htb, so I > believe I'm missing something else? someone suggested to add a new class for > ACK packets, I've done that already, but I've only noticed little > difference... really don't know whats happening, if you don't have tcng I can > show you my tc rules (showed by tc -s class show dev eth1). Thank you again > > EDGAR MERINO > > On Wednesday 06 July 2005 23:30, Jody Shumaker wrote: > >>You need to use connection marking as well. --l7proto bittorrent will >>only recognize the first packet in a bittorrent stream, you need to save >>a mark on the whole tcp connection, and restore the mark for all future >>packets if you want the entire connection to be classified. >> >>iptables -t mangle -A lay7 -p tcp -j CONNMARK --restore-mark >>iptables -t mangle -A lay7 -m layer7 --l7proto bittorrent -j MARK >>--set-mark 1 iptables -t mangle -A lay7 -o eth1 -m mark --mark 1 -j >>CLASSIFY --set-class 2:2 iptables -t mangle -A lay7 -m layer7 --l7proto >>smtp -j MARK --set-mark 2 iptables -t mangle -A lay7 -o eth1 -m mark --mark >>2 -j CLASSIFY --set-class 2:3 iptables -t mangle -A lay7 -p tcp -m mark ! >>--mark 0 -j CONNMARK --save-mark >> >> >>If you're marking ever gets more complex, it might take a little more work >>( -j accepts for matching already classified connections after the >>--restore-mark) but the above should help get the full bittorrent >>connection classified, not just the first packet. >> >>- Jody >> >>Edgar wrote: >> >>>Hello, >>> >>>I've been trying to shape the bittorrent traffic (on my external >>>interface, upload), but without luck, for this I'm using layer7 filter >>>right now, but I've also tried ipp2p, with the same results, I might say >>>that this is not a problem with this packet classifiers, the problem is >>>with HTB, here's why. When I open azureus (the bittorrent client I use) I >>>see upload traffic getting shapped, but also I see that my download >>>traffic won't go up if I'm shaping on the upload interface, if I stop >>>shaping on that interface then upload ( as expected) will increase, and >>>so the download rate, this happens to me using the default bittorrent >>>client (classic), so its not a client problem. Ok, the problem here is >>>that when using bittorrent, although I see the traffic is shaped I can't >>>surf web pages, nor chat in msn messenger, nor do anything at all, and >>>merely that's all I want to do, shape p2p traffic to be able to use my >>>bandwidth fairly, maybe its a bittorrent problem, because with the >>>edonkey protocol I have no problem at all, traffic get shaped and I can >>>use the rest of my bandwidth, I'll post my iptables rules for marking the >>>bittorrent packets and the htb rules I use (using tcng): >>> >>>### IPTABLES RULES ### >>>iptables -t mangle -F >>>iptables -t mangle -X >>>iptables -t mangle -N lay7 >>>iptables -t mangle -A POSTROUTING -j lay7 >>>iptables -t mangle -A lay7 -m layer7 --l7proto bittorrent -j MARK >>>--set-mark 1 iptables -t mangle -A lay7 -o eth1 -m mark --mark 1 -j >>>CLASSIFY --set-class 2:2 >>>iptables -t mangle -A lay7 -m layer7 --l7proto smtp -j MARK --set-mark 2 >>>iptables -t mangle -A lay7 -o eth1 -m mark --mark 2 -j CLASSIFY >>>--set-class 2:3 >>> >>>### HTB RULES ### >>> >>>#define UPLOAD eth1 >>>#define UPRATE 25kBps >>>#define P2P 10kBps >>> >>>dev UPLOAD { >>> egress { >>> class ( <$emule> ) ; >>> class ( <$smtp> ) ; >>> class ( <$ssh> ) if tcp_dport == 8080 ; /*Changed port from 22 to 8080 >>>*/ class ( <$otro> ) if 1 ; >>> >>> htb () { >>> class ( rate UPRATE, ceil UPRATE ) { >>> $emule = class ( prio 8, rate 6kBps, ceil P2P ) { sfq; } ; >>> $smtp = class ( prio 1, rate 6kBps, ceil 12kBps ) { sfq; } ; >>> $ssh = class ( prio 0, rate 3kBps, ceil 5kBps) { sfq; } ; >>> $otro = class ( prio 1, rate 8kBps, ceil UPRATE ) { sfq; } ; >>> } >>> } >>> } >>>} >>> >>>Also, given the priorities it's expected to let me surf the web or chat in >>>msn messenger rather than take my whole bandwidth. >>> >>>I hope someone can help me out with this, maybe it not ok to use tcng with >>>iptables? thank you in advance >>> >>>EDGAR MERINO >>>_______________________________________________ >>>LARTC mailing list >>>LARTC@mailman.ds9a.nl >>>http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc >> >>_______________________________________________ >>LARTC mailing list >>LARTC@mailman.ds9a.nl >>http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc _______________________________________________ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc -- This message was scanned for spam and viruses by BitDefender. For more information please visit http://linux.bitdefender.com/ -- This message was scanned for spam and viruses by BitDefender. For more information please visit http://linux.bitdefender.com/ From gregoriandres at yahoo.com.ar Thu Jul 7 14:36:00 2005 From: gregoriandres at yahoo.com.ar (:: L i n u XK i D ::) Date: Thu Jul 7 14:36:10 2005 Subject: [LARTC] macaddress traffic log Message-ID: Hi I want to log lan ethernet (by macaddress) traffic with iptraf: /usr/sbin/iptraf -f -l eth1 -t 1 -B -L /scripts/mac-traff.log and next with a perl script parse that data to MRTG. But, I think when iptraf is running I can show iptraf LAN traffic console. there is another way to log ethernet (by MACADDRESS) lan traffic ? regards andres From donvodka at gmail.com Thu Jul 7 22:34:43 2005 From: donvodka at gmail.com (Edgar) Date: Thu Jul 7 22:34:59 2005 Subject: [LARTC] HTB and bittorrent, won't work In-Reply-To: <20050707113647.AE2EE48BE@outpost.ds9a.nl> References: <20050707113647.AE2EE48BE@outpost.ds9a.nl> Message-ID: <200507071534.43343.donvodka@gmail.com> Thank you for your response, I will try to do what you told me, but I have a squid sever, and I don't know which port I should use for that, since the http layer7 protocol won't work, when I have that mark rule in iptables I don't see traffic going into that rule, it'll only work when I'm surfing the web without the squid cache; another problem I have is that the msn messenger I use (kopete for kde) isn't recognized by the layer7 protocol, so the question is this, can I do this specifying the ports in tcng ? EDGAR MERINO On Thursday 07 July 2005 06:36, Forte Systems - Iosif Peterfi wrote: > I would suggest classifing interactive connections, and leave all the bulk > traffic in the default class. This way, the bt,kazaa,emule traffic will go > in the same class, without additional filtering. > Also, using HFSC instead of HTB helps you increase the delay of the default > class. This way bulk traffic will be sent every n ms, leaving priority to > the interactive/web/mail traffic. Think about it. > > > Iosif Peterfi > S.C. Forte Systems SRL > http://www.fortesys.ro/ > > -----Original Message----- > From: lartc-bounces@mailman.ds9a.nl [mailto:lartc-bounces@mailman.ds9a.nl] > On Behalf Of Klaus > Sent: Thursday, July 07, 2005 10:22 AM > To: lartc@mailman.ds9a.nl > Subject: Re: [LARTC] HTB and bittorrent, won't work > > ipp2p vs. l7 filter > > l7 uses regular expressions, so they are slower (some rules are EXTREME > slow like fasttrack) and not so strong like the ipp2p rules (which can > have for example packet length checks). ipp2p is specialized for p2p > detection, so a many p2p packets are not detected by l7 (for example not > all BitTorrent connections start with a 013h "BitTorrent"). The worst > part is that l7 filter has some p2p rules which detect false positives: > > http://l7-filter.sourceforge.net/layer7-protocols/protocols/edonkey.pat > > "... This will match about 1% of streams with random data in them! ..." > > If you drop p2p connection, one of hundred downloads / web pages will > fail (and fail every time) ? > > I would recommend l7-filter for everything but not for p2p. It is a VERY > nice filter, but if they would have something else than regexp, i would > use it maybe too. > > Klaus, Maintainer of ipp2p > > Edgar wrote: > > Hi, thanks for your help and interest, someone told me about that > > already, > > so > > > I did it, and this is the script I'm running to do it: > > #!/bin/sh > > > > ### ERASING RULES AND USER CREATED CHAINS ### > > iptables -t mangle -F > > iptables -t mangle -X > > iptables -t mangle -N lay7PRE > > iptables -t mangle -N lay7POST > > > > ### PREROUTING RULES ### > > iptables -t mangle -A lay7PRE -j CONNMARK --restore-mark > > iptables -t mangle -A lay7PRE -m mark ! --mark 0 -j ACCEPT > > iptables -t mangle -A lay7PRE -m layer7 --l7proto bittorrent -j MARK > > --set-mark 1 > > iptables -t mangle -A lay7PRE -m layer7 --l7proto smtp -j MARK --set-mark > > 2 > > > iptables -t mangle -A lay7PRE -m layer7 --l7proto http -j MARK --set-mark > > 3 > > > iptables -t mangle -A lay7PRE -j CONNMARK --save-mark > > > > ### POSTROUTING RULES ### > > iptables -t mangle -A lay7POST -o eth1 -m mark --mark 1 -j CLASSIFY > > --set-class 2:2 > > iptables -t mangle -A lay7POST -o eth1 -m mark --mark 2 -j CLASSIFY > > --set-class 2:3 > > iptables -t mangle -A lay7POST -o eth1 -m mark --mark 3 -j CLASSIFY > > --set-class 2:4 > > > > ### ------------------------------------------------------------------- > > ### > > > iptables -t mangle -A PREROUTING -j lay7PRE > > iptables -t mangle -A POSTROUTING -j lay7POST > > > > I'm trying this right now, and I believe its kind of working, but web > > surfing > > > is very slow, I might say unusable, so this is not what I want, also I > > had > > to > > > mark http traffic to make this work, give it a higher prio in htb, so I > > believe I'm missing something else? someone suggested to add a new class > > for > > > ACK packets, I've done that already, but I've only noticed little > > difference... really don't know whats happening, if you don't have tcng I > > can > > > show you my tc rules (showed by tc -s class show dev eth1). Thank you > > again > > > EDGAR MERINO > > > > On Wednesday 06 July 2005 23:30, Jody Shumaker wrote: > >>You need to use connection marking as well. --l7proto bittorrent will > >>only recognize the first packet in a bittorrent stream, you need to save > >>a mark on the whole tcp connection, and restore the mark for all future > >>packets if you want the entire connection to be classified. > >> > >>iptables -t mangle -A lay7 -p tcp -j CONNMARK --restore-mark > >>iptables -t mangle -A lay7 -m layer7 --l7proto bittorrent -j MARK > >>--set-mark 1 iptables -t mangle -A lay7 -o eth1 -m mark --mark 1 -j > >>CLASSIFY --set-class 2:2 iptables -t mangle -A lay7 -m layer7 --l7proto > >>smtp -j MARK --set-mark 2 iptables -t mangle -A lay7 -o eth1 -m mark > > --mark > > >>2 -j CLASSIFY --set-class 2:3 iptables -t mangle -A lay7 -p tcp -m mark ! > >>--mark 0 -j CONNMARK --save-mark > >> > >> > >>If you're marking ever gets more complex, it might take a little more > >> work ( -j accepts for matching already classified connections after the > >> --restore-mark) but the above should help get the full bittorrent > >> connection classified, not just the first packet. > >> > >>- Jody > >> > >>Edgar wrote: > >>>Hello, > >>> > >>>I've been trying to shape the bittorrent traffic (on my external > >>>interface, upload), but without luck, for this I'm using layer7 filter > >>>right now, but I've also tried ipp2p, with the same results, I might say > >>>that this is not a problem with this packet classifiers, the problem is > >>>with HTB, here's why. When I open azureus (the bittorrent client I use) > >>> I see upload traffic getting shapped, but also I see that my download > >>> traffic won't go up if I'm shaping on the upload interface, if I stop > >>> shaping on that interface then upload ( as expected) will increase, and > >>> so the download rate, this happens to me using the default bittorrent > >>> client (classic), so its not a client problem. Ok, the problem here is > >>> that when using bittorrent, although I see the traffic is shaped I > >>> can't surf web pages, nor chat in msn messenger, nor do anything at > >>> all, and merely that's all I want to do, shape p2p traffic to be able > >>> to use my bandwidth fairly, maybe its a bittorrent problem, because > >>> with the edonkey protocol I have no problem at all, traffic get shaped > >>> and I can use the rest of my bandwidth, I'll post my iptables rules for > >>> marking the bittorrent packets and the htb rules I use (using tcng): > >>> > >>>### IPTABLES RULES ### > >>>iptables -t mangle -F > >>>iptables -t mangle -X > >>>iptables -t mangle -N lay7 > >>>iptables -t mangle -A POSTROUTING -j lay7 > >>>iptables -t mangle -A lay7 -m layer7 --l7proto bittorrent -j MARK > >>>--set-mark 1 iptables -t mangle -A lay7 -o eth1 -m mark --mark 1 -j > >>>CLASSIFY --set-class 2:2 > >>>iptables -t mangle -A lay7 -m layer7 --l7proto smtp -j MARK --set-mark 2 > >>>iptables -t mangle -A lay7 -o eth1 -m mark --mark 2 -j CLASSIFY > >>>--set-class 2:3 > >>> > >>>### HTB RULES ### > >>> > >>>#define UPLOAD eth1 > >>>#define UPRATE 25kBps > >>>#define P2P 10kBps > >>> > >>>dev UPLOAD { > >>> egress { > >>> class ( <$emule> ) ; > >>> class ( <$smtp> ) ; > >>> class ( <$ssh> ) if tcp_dport == 8080 ; /*Changed port from 22 to > >>> 8080 */ class ( <$otro> ) if 1 ; > >>> > >>> htb () { > >>> class ( rate UPRATE, ceil UPRATE ) { > >>> $emule = class ( prio 8, rate 6kBps, ceil P2P ) { sfq; } ; > >>> $smtp = class ( prio 1, rate 6kBps, ceil 12kBps ) { sfq; } ; > >>> $ssh = class ( prio 0, rate 3kBps, ceil 5kBps) { sfq; } ; > >>> $otro = class ( prio 1, rate 8kBps, ceil UPRATE ) { sfq; } ; > >>> } > >>> } > >>> } > >>>} > >>> > >>>Also, given the priorities it's expected to let me surf the web or chat > > in > > >>>msn messenger rather than take my whole bandwidth. > >>> > >>>I hope someone can help me out with this, maybe it not ok to use tcng > > with > > >>>iptables? thank you in advance > >>> > >>>EDGAR MERINO > >>>_______________________________________________ > >>>LARTC mailing list > >>>LARTC@mailman.ds9a.nl > >>>http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > >> > >>_______________________________________________ > >>LARTC mailing list > >>LARTC@mailman.ds9a.nl > >>http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > > > > _______________________________________________ > > LARTC mailing list > > LARTC@mailman.ds9a.nl > > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > > > -- > This message was scanned for spam and viruses by BitDefender. > For more information please visit http://linux.bitdefender.com/ From o7sh at actcom.net.il Fri Jul 8 00:43:08 2005 From: o7sh at actcom.net.il (o7sh@actcom.net.il) Date: Fri Jul 8 00:43:01 2005 Subject: [LARTC] Server and router on the same box In-Reply-To: <200507071118.08420.Andreas.Klauer@metamorpher.de> References: <200507062353.46601.o7sh@actcom.net.il> <200507071151.13007.o7sh@actcom.net.il> <200507071118.08420.Andreas.Klauer@metamorpher.de> Message-ID: <200507080143.08465.o7sh@actcom.net.il> > To make upstream fully available to your webserver, I suggest using a PRIO > qdisc with 2 bands... band one for webserver traffic, band two for > everything else. > > The PRIO qdisc will make sure that your webserver always gets to send his > packets first. LAN traffic will only be allowed to send when the webserver > sent all his packets. > Hi again.. it sounds like just what I need but I find it hard to implement. Here are my rules: ----- tc qdisc add dev ppp0 root handle 1: prio tc filter add dev ppp0 parent 1:0 protocol ip prio 1 u32 match ip sport 80 0xffff flowid 1:1 tc filter add dev ppp0 parent 1:0 protocol ip prio 3 u32 ----- I've been testing this for a while. But still got the dowload running while the upstream was full... Can you explain why? From donvodka at gmail.com Fri Jul 8 08:36:53 2005 From: donvodka at gmail.com (Edgar) Date: Fri Jul 8 08:37:05 2005 Subject: [LARTC] P2P shaping, won't work Message-ID: <200507080136.53843.donvodka@gmail.com> Hello, its me again, I won't stop sending emails to this list, until I solve this problem, I've tried several apps to create the right htb rules (even made them my self), but I always get the same results, traffic gets shaped, but I can't use my bandwidth, and this is weird, because I should be able to, also I keep seeing download being limited too, and that shouldn't be happening, I don't think I'm the only one with this problem out there, so I will post here once again my configuration, to match p2p traffic and to shape it: ### ### iptables -t mangle -F iptables -t mangle -X iptables -t mangle -N lay7PRE iptables -t mangle -N lay7POST iptables -t mangle -N ipp2pPRE iptables -t mangle -N ipp2pPOST ### PREROUTING RULES ### iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark iptables -t mangle -A PREROUTING -m mark ! --mark 0 -j ACCEPT iptables -t mangle -A ipp2pPRE -p tcp -m ipp2p --ipp2p -j MARK --set-mark 3 iptables -t mangle -A ipp2pPRE -p udp -m ipp2p --bit -j MARK --set-mark 3 iptables -t mangle -A ipp2pPRE -p tcp -m mark --mark 3 -j CONNMARK --save-mark iptables -t mangle -A lay7PRE -m layer7 --l7proto smtp -j MARK --set-mark 1 iptables -t mangle -A lay7PRE -m layer7 --l7proto ssh -j MARK --set-mark 2 iptables -t mangle -A lay7PRE -m mark --mark 2 -j CONNMARK --save-mark ### POSTROUTING RULES ### iptables -t mangle -A ipp2pPOST -o eth1 -m mark --mark 3 -j CLASSIFY --set-class 1:30 iptables -t mangle -A lay7POST -o eth1 -m mark --mark 1 -j CLASSIFY --set-class 1:20 iptables -t mangle -A lay7POST -o eth1 -m mark --mark 2 -j CLASSIFY --set-class 1:10 ### ### iptables -t mangle -A PREROUTING -j lay7PRE iptables -t mangle -A POSTROUTING -j lay7POST iptables -t mangle -A PREROUTING -j ipp2pPRE iptables -t mangle -A POSTROUTING -j ipp2pPOST TC RULES tc qdisc add dev eth1 handle 1:0 root dsmark indices 8 default_index 0 tc qdisc add dev eth1 handle 2:0 parent 1:0 htb tc class add dev eth1 parent 2:0 classid 2:1 htb rate 22000bps ceil 22000bps tc class add dev eth1 parent 2:1 classid 2:2 htb rate 3000bps ceil 5000bps prio 0 tc qdisc add dev eth1 handle 3:0 parent 2:2 sfq tc class add dev eth1 parent 2:1 classid 2:3 htb rate 5000bps ceil 6000bps prio 1 tc qdisc add dev eth1 handle 4:0 parent 2:3 sfq tc class add dev eth1 parent 2:1 classid 2:4 htb rate 8000bps ceil 8000bps prio 2 tc qdisc add dev eth1 handle 5:0 parent 2:4 sfq tc class add dev eth1 parent 2:1 classid 2:5 htb rate 10000bps ceil 10000bps burst 0 cburst 0 prio 3 quantum 1500 tc qdisc add dev eth1 handle 6:0 parent 2:5 sfq tc class add dev eth1 parent 2:1 classid 2:6 htb rate 10000bps ceil 22000bps prio 1 tc qdisc add dev eth1 handle 7:0 parent 2:6 sfq tc filter add dev eth1 parent 2:0 protocol all prio 1 tcindex mask 0x7 shift 0 tc filter add dev eth1 parent 2:0 protocol all prio 1 handle 5 tcindex classid 2:6 tc filter add dev eth1 parent 2:0 protocol all prio 1 handle 4 tcindex classid 2:4 tc filter add dev eth1 parent 2:0 protocol all prio 1 handle 3 tcindex classid 2:5 tc filter add dev eth1 parent 2:0 protocol all prio 1 handle 2 tcindex classid 2:3 tc filter add dev eth1 parent 2:0 protocol all prio 1 handle 1 tcindex classid 2:2 tc filter add dev eth1 parent 1:0 protocol all prio 1 u32 match u8 0x5 0xf at 0 match u16 0x0 0xffc0 at 2 classid 1:5 tc filter add dev eth1 parent 1:0 protocol all prio 1 u32 match u8 0x5 0xf at 0 match u8 0x10 0x10 at 33 classid 1:4 tc filter add dev eth1 parent 1:0 protocol all prio 1 u32 match u8 0x5 0xf at 0 classid 1:5 tc filter add dev eth1 parent 1:0 protocol all prio 1 u32 match u32 0x0 0x0 at 0 classid 1:5 well, this rules are generated with tcng (tcc actually), and I see traffic going to the p2p rule (being 2:5), but still I'm not able to browse the web! nor chat nor anything, someone adviced me to create an interactive class and then leave the unmatched traffic go to a default class (and here'll go p2p traffic too), but then I'm running a squid server at home, and thus I don't know what port to specify to do this, anyway, I've tried already with normal http traffic, with layer7, I see the traffic going to the iptables chain, but yet it'll still won't work, p2p traffic keeps eating all my bandwidth, its ignoring all the priorities I have in my tc rules (being the 2:5 class the one with the lowest priority), I really need to get this working, and I know there's some ppl that has lots of more experience that me, and I'm sure you can help me. Thank you EDGAR MERINO From toto at fortesys.ro Fri Jul 8 10:49:21 2005 From: toto at fortesys.ro (Forte Systems - Iosif Peterfi) Date: Fri Jul 8 10:49:40 2005 Subject: [LARTC] HTB and bittorrent, won't work In-Reply-To: <200507071534.43343.donvodka@gmail.com> Message-ID: <20050708084936.48932489A@outpost.ds9a.nl> You can classify kopete traffic using iptables rules. Depending on the im protocol you use. Yahoo messenger uses TCP ports 5050 and 5051 i think.. MSN uses TCP 1863... you can google for them. You also have to setup tcp_sport as the squid port in orded for the l7 http filter to work. And if that is not working classify using iptables as source your internal eth ip and source port your squid port. There is a slighter chance that some btclients will use the same ports as described in the iptables rules... but i don't think that will mess up your shaping that bad. Iosif Peterfi S.C. Forte Systems SRL http://www.fortesys.ro/ -----Original Message----- From: lartc-bounces@mailman.ds9a.nl [mailto:lartc-bounces@mailman.ds9a.nl] On Behalf Of Edgar Sent: Thursday, July 07, 2005 11:35 PM To: lartc@mailman.ds9a.nl Subject: Re: [LARTC] HTB and bittorrent, won't work Thank you for your response, I will try to do what you told me, but I have a squid sever, and I don't know which port I should use for that, since the http layer7 protocol won't work, when I have that mark rule in iptables I don't see traffic going into that rule, it'll only work when I'm surfing the web without the squid cache; another problem I have is that the msn messenger I use (kopete for kde) isn't recognized by the layer7 protocol, so the question is this, can I do this specifying the ports in tcng ? EDGAR MERINO On Thursday 07 July 2005 06:36, Forte Systems - Iosif Peterfi wrote: > I would suggest classifing interactive connections, and leave all the bulk > traffic in the default class. This way, the bt,kazaa,emule traffic will go > in the same class, without additional filtering. > Also, using HFSC instead of HTB helps you increase the delay of the default > class. This way bulk traffic will be sent every n ms, leaving priority to > the interactive/web/mail traffic. Think about it. > > > Iosif Peterfi > S.C. Forte Systems SRL > http://www.fortesys.ro/ > > -----Original Message----- > From: lartc-bounces@mailman.ds9a.nl [mailto:lartc-bounces@mailman.ds9a.nl] > On Behalf Of Klaus > Sent: Thursday, July 07, 2005 10:22 AM > To: lartc@mailman.ds9a.nl > Subject: Re: [LARTC] HTB and bittorrent, won't work > > ipp2p vs. l7 filter > > l7 uses regular expressions, so they are slower (some rules are EXTREME > slow like fasttrack) and not so strong like the ipp2p rules (which can > have for example packet length checks). ipp2p is specialized for p2p > detection, so a many p2p packets are not detected by l7 (for example not > all BitTorrent connections start with a 013h "BitTorrent"). The worst > part is that l7 filter has some p2p rules which detect false positives: > > http://l7-filter.sourceforge.net/layer7-protocols/protocols/edonkey.pat > > "... This will match about 1% of streams with random data in them! ..." > > If you drop p2p connection, one of hundred downloads / web pages will > fail (and fail every time) ? > > I would recommend l7-filter for everything but not for p2p. It is a VERY > nice filter, but if they would have something else than regexp, i would > use it maybe too. > > Klaus, Maintainer of ipp2p > > Edgar wrote: > > Hi, thanks for your help and interest, someone told me about that > > already, > > so > > > I did it, and this is the script I'm running to do it: > > #!/bin/sh > > > > ### ERASING RULES AND USER CREATED CHAINS ### > > iptables -t mangle -F > > iptables -t mangle -X > > iptables -t mangle -N lay7PRE > > iptables -t mangle -N lay7POST > > > > ### PREROUTING RULES ### > > iptables -t mangle -A lay7PRE -j CONNMARK --restore-mark > > iptables -t mangle -A lay7PRE -m mark ! --mark 0 -j ACCEPT > > iptables -t mangle -A lay7PRE -m layer7 --l7proto bittorrent -j MARK > > --set-mark 1 > > iptables -t mangle -A lay7PRE -m layer7 --l7proto smtp -j MARK --set-mark > > 2 > > > iptables -t mangle -A lay7PRE -m layer7 --l7proto http -j MARK --set-mark > > 3 > > > iptables -t mangle -A lay7PRE -j CONNMARK --save-mark > > > > ### POSTROUTING RULES ### > > iptables -t mangle -A lay7POST -o eth1 -m mark --mark 1 -j CLASSIFY > > --set-class 2:2 > > iptables -t mangle -A lay7POST -o eth1 -m mark --mark 2 -j CLASSIFY > > --set-class 2:3 > > iptables -t mangle -A lay7POST -o eth1 -m mark --mark 3 -j CLASSIFY > > --set-class 2:4 > > > > ### ------------------------------------------------------------------- > > ### > > > iptables -t mangle -A PREROUTING -j lay7PRE > > iptables -t mangle -A POSTROUTING -j lay7POST > > > > I'm trying this right now, and I believe its kind of working, but web > > surfing > > > is very slow, I might say unusable, so this is not what I want, also I > > had > > to > > > mark http traffic to make this work, give it a higher prio in htb, so I > > believe I'm missing something else? someone suggested to add a new class > > for > > > ACK packets, I've done that already, but I've only noticed little > > difference... really don't know whats happening, if you don't have tcng I > > can > > > show you my tc rules (showed by tc -s class show dev eth1). Thank you > > again > > > EDGAR MERINO > > > > On Wednesday 06 July 2005 23:30, Jody Shumaker wrote: > >>You need to use connection marking as well. --l7proto bittorrent will > >>only recognize the first packet in a bittorrent stream, you need to save > >>a mark on the whole tcp connection, and restore the mark for all future > >>packets if you want the entire connection to be classified. > >> > >>iptables -t mangle -A lay7 -p tcp -j CONNMARK --restore-mark > >>iptables -t mangle -A lay7 -m layer7 --l7proto bittorrent -j MARK > >>--set-mark 1 iptables -t mangle -A lay7 -o eth1 -m mark --mark 1 -j > >>CLASSIFY --set-class 2:2 iptables -t mangle -A lay7 -m layer7 --l7proto > >>smtp -j MARK --set-mark 2 iptables -t mangle -A lay7 -o eth1 -m mark > > --mark > > >>2 -j CLASSIFY --set-class 2:3 iptables -t mangle -A lay7 -p tcp -m mark ! > >>--mark 0 -j CONNMARK --save-mark > >> > >> > >>If you're marking ever gets more complex, it might take a little more > >> work ( -j accepts for matching already classified connections after the > >> --restore-mark) but the above should help get the full bittorrent > >> connection classified, not just the first packet. > >> > >>- Jody > >> > >>Edgar wrote: > >>>Hello, > >>> > >>>I've been trying to shape the bittorrent traffic (on my external > >>>interface, upload), but without luck, for this I'm using layer7 filter > >>>right now, but I've also tried ipp2p, with the same results, I might say > >>>that this is not a problem with this packet classifiers, the problem is > >>>with HTB, here's why. When I open azureus (the bittorrent client I use) > >>> I see upload traffic getting shapped, but also I see that my download > >>> traffic won't go up if I'm shaping on the upload interface, if I stop > >>> shaping on that interface then upload ( as expected) will increase, and > >>> so the download rate, this happens to me using the default bittorrent > >>> client (classic), so its not a client problem. Ok, the problem here is > >>> that when using bittorrent, although I see the traffic is shaped I > >>> can't surf web pages, nor chat in msn messenger, nor do anything at > >>> all, and merely that's all I want to do, shape p2p traffic to be able > >>> to use my bandwidth fairly, maybe its a bittorrent problem, because > >>> with the edonkey protocol I have no problem at all, traffic get shaped > >>> and I can use the rest of my bandwidth, I'll post my iptables rules for > >>> marking the bittorrent packets and the htb rules I use (using tcng): > >>> > >>>### IPTABLES RULES ### > >>>iptables -t mangle -F > >>>iptables -t mangle -X > >>>iptables -t mangle -N lay7 > >>>iptables -t mangle -A POSTROUTING -j lay7 > >>>iptables -t mangle -A lay7 -m layer7 --l7proto bittorrent -j MARK > >>>--set-mark 1 iptables -t mangle -A lay7 -o eth1 -m mark --mark 1 -j > >>>CLASSIFY --set-class 2:2 > >>>iptables -t mangle -A lay7 -m layer7 --l7proto smtp -j MARK --set-mark 2 > >>>iptables -t mangle -A lay7 -o eth1 -m mark --mark 2 -j CLASSIFY > >>>--set-class 2:3 > >>> > >>>### HTB RULES ### > >>> > >>>#define UPLOAD eth1 > >>>#define UPRATE 25kBps > >>>#define P2P 10kBps > >>> > >>>dev UPLOAD { > >>> egress { > >>> class ( <$emule> ) ; > >>> class ( <$smtp> ) ; > >>> class ( <$ssh> ) if tcp_dport == 8080 ; /*Changed port from 22 to > >>> 8080 */ class ( <$otro> ) if 1 ; > >>> > >>> htb () { > >>> class ( rate UPRATE, ceil UPRATE ) { > >>> $emule = class ( prio 8, rate 6kBps, ceil P2P ) { sfq; } ; > >>> $smtp = class ( prio 1, rate 6kBps, ceil 12kBps ) { sfq; } ; > >>> $ssh = class ( prio 0, rate 3kBps, ceil 5kBps) { sfq; } ; > >>> $otro = class ( prio 1, rate 8kBps, ceil UPRATE ) { sfq; } ; > >>> } > >>> } > >>> } > >>>} > >>> > >>>Also, given the priorities it's expected to let me surf the web or chat > > in > > >>>msn messenger rather than take my whole bandwidth. > >>> > >>>I hope someone can help me out with this, maybe it not ok to use tcng > > with > > >>>iptables? thank you in advance > >>> > >>>EDGAR MERINO > >>>_______________________________________________ > >>>LARTC mailing list > >>>LARTC@mailman.ds9a.nl > >>>http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > >> > >>_______________________________________________ > >>LARTC mailing list > >>LARTC@mailman.ds9a.nl > >>http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > > > > _______________________________________________ > > LARTC mailing list > > LARTC@mailman.ds9a.nl > > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > > > -- > This message was scanned for spam and viruses by BitDefender. > For more information please visit http://linux.bitdefender.com/ _______________________________________________ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc -- This message was scanned for spam and viruses by BitDefender. For more information please visit http://linux.bitdefender.com/ -- This message was scanned for spam and viruses by BitDefender. For more information please visit http://linux.bitdefender.com/ From fpereira at lojan.com Fri Jul 8 15:30:45 2005 From: fpereira at lojan.com (Francisco Pereira) Date: Fri Jul 8 15:25:27 2005 Subject: [LARTC] P2P shaping, won't work In-Reply-To: <200507080136.53843.donvodka@gmail.com> References: <200507080136.53843.donvodka@gmail.com> Message-ID: <1120829445.42ce800548bde@webmail.montevideo.com.uy> Mensaje citado por Edgar : > Hello, its me again, I won't stop sending emails to this list, until I solve > > this problem, I've tried several apps to create the right htb rules (even > made them my self), but I always get the same results, traffic gets shaped, > but I can't use my bandwidth, and this is weird, because I should be able to, > > also I keep seeing download being limited too, and that shouldn't be > happening, I don't think I'm the only one with this problem out there, so I > will post here once again my configuration, to match p2p traffic and to shape > > it: > > ### ### > iptables -t mangle -F > iptables -t mangle -X > iptables -t mangle -N lay7PRE > iptables -t mangle -N lay7POST > iptables -t mangle -N ipp2pPRE > iptables -t mangle -N ipp2pPOST > > ### PREROUTING RULES ### > iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark > iptables -t mangle -A PREROUTING -m mark ! --mark 0 -j ACCEPT > iptables -t mangle -A ipp2pPRE -p tcp -m ipp2p --ipp2p -j MARK --set-mark 3 > iptables -t mangle -A ipp2pPRE -p udp -m ipp2p --bit -j MARK --set-mark 3 > iptables -t mangle -A ipp2pPRE -p tcp -m mark --mark 3 -j CONNMARK > --save-mark > iptables -t mangle -A lay7PRE -m layer7 --l7proto smtp -j MARK --set-mark 1 > iptables -t mangle -A lay7PRE -m layer7 --l7proto ssh -j MARK --set-mark 2 > iptables -t mangle -A lay7PRE -m mark --mark 2 -j CONNMARK --save-mark > > ### POSTROUTING RULES ### > iptables -t mangle -A ipp2pPOST -o eth1 -m mark --mark 3 -j CLASSIFY > --set-class 1:30 > iptables -t mangle -A lay7POST -o eth1 -m mark --mark 1 -j CLASSIFY > --set-class 1:20 > iptables -t mangle -A lay7POST -o eth1 -m mark --mark 2 -j CLASSIFY > --set-class 1:10 > > ### ### > iptables -t mangle -A PREROUTING -j lay7PRE > iptables -t mangle -A POSTROUTING -j lay7POST > iptables -t mangle -A PREROUTING -j ipp2pPRE > iptables -t mangle -A POSTROUTING -j ipp2pPOST > > TC RULES > tc qdisc add dev eth1 handle 1:0 root dsmark indices 8 default_index 0 > tc qdisc add dev eth1 handle 2:0 parent 1:0 htb > tc class add dev eth1 parent 2:0 classid 2:1 htb rate 22000bps ceil 22000bps > tc class add dev eth1 parent 2:1 classid 2:2 htb rate 3000bps ceil 5000bps > prio 0 > tc qdisc add dev eth1 handle 3:0 parent 2:2 sfq > tc class add dev eth1 parent 2:1 classid 2:3 htb rate 5000bps ceil 6000bps > prio 1 > tc qdisc add dev eth1 handle 4:0 parent 2:3 sfq > tc class add dev eth1 parent 2:1 classid 2:4 htb rate 8000bps ceil 8000bps > prio 2 > tc qdisc add dev eth1 handle 5:0 parent 2:4 sfq > tc class add dev eth1 parent 2:1 classid 2:5 htb rate 10000bps ceil 10000bps > > burst 0 cburst 0 prio 3 quantum 1500 > tc qdisc add dev eth1 handle 6:0 parent 2:5 sfq > tc class add dev eth1 parent 2:1 classid 2:6 htb rate 10000bps ceil 22000bps > > prio 1 > tc qdisc add dev eth1 handle 7:0 parent 2:6 sfq > tc filter add dev eth1 parent 2:0 protocol all prio 1 tcindex mask 0x7 shift > 0 > tc filter add dev eth1 parent 2:0 protocol all prio 1 handle 5 tcindex > classid > 2:6 > tc filter add dev eth1 parent 2:0 protocol all prio 1 handle 4 tcindex > classid > 2:4 > tc filter add dev eth1 parent 2:0 protocol all prio 1 handle 3 tcindex > classid > 2:5 > tc filter add dev eth1 parent 2:0 protocol all prio 1 handle 2 tcindex > classid > 2:3 > tc filter add dev eth1 parent 2:0 protocol all prio 1 handle 1 tcindex > classid > 2:2 > tc filter add dev eth1 parent 1:0 protocol all prio 1 u32 match u8 0x5 0xf at > > 0 match u16 0x0 0xffc0 at 2 classid 1:5 > tc filter add dev eth1 parent 1:0 protocol all prio 1 u32 match u8 0x5 0xf at > > 0 match u8 0x10 0x10 at 33 classid 1:4 > tc filter add dev eth1 parent 1:0 protocol all prio 1 u32 match u8 0x5 0xf at > > 0 classid 1:5 > tc filter add dev eth1 parent 1:0 protocol all prio 1 u32 match u32 0x0 0x0 > at > 0 classid 1:5 > > well, this rules are generated with tcng (tcc actually), and I see traffic > going to the p2p rule (being 2:5), but still I'm not able to browse the web! > > nor chat nor anything, someone adviced me to create an interactive class and > > then leave the unmatched traffic go to a default class (and here'll go p2p > traffic too), but then I'm running a squid server at home, and thus I don't > know what port to specify to do this, anyway, I've tried already with normal > > http traffic, with layer7, I see the traffic going to the iptables chain, but > > yet it'll still won't work, p2p traffic keeps eating all my bandwidth, its > ignoring all the priorities I have in my tc rules (being the 2:5 class the > one with the lowest priority), I really need to get this working, and I know > > there's some ppl that has lots of more experience that me, and I'm sure you > can help me. Thank you can != want != will You may have a problem with the rate units - bps != bit -, but i dont know the type or rate of your internet link. Post a detailed ascii diagram of your network. Which is the transfer rate of the internet link? IPP2P works very well for me. How did you test your iptables & tc setup? In the real network or in a controlled environment? Did you try starting with a very basic tc setup and adding tc qdiscs, classes and filters one by one and checking the results on every step? Regards, Francisco. +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Miles de internautas ya utilizan el discador del Portal. Bajatelo ahora, se instala solo!. http://www.montevideo.com.uy/discador/setup.exe +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ From jody.shumaker at gmail.com Fri Jul 8 15:59:31 2005 From: jody.shumaker at gmail.com (Jody Shumaker) Date: Fri Jul 8 15:59:35 2005 Subject: [LARTC] P2P shaping, won't work In-Reply-To: <1120829445.42ce800548bde@webmail.montevideo.com.uy> References: <200507080136.53843.donvodka@gmail.com> <1120829445.42ce800548bde@webmail.montevideo.com.uy> Message-ID: <2af43649050708065930b163b6@mail.gmail.com> You might want to remove the iptables CLASSIFY rules as they point to non existent tc classes, 1:10, 1:20, and 1:30. Also your total combined rates for all the subclasses of 2:1 are greater than the rate of 2:1, 22kbps for 2:1 and 36kbps for 2:2 - 2:6. It's best if you keep these to add up to the root classes ceil or lower. Also you're marking p2p data as 3 and sending it to classid 2:5. 2:5 is tied for the highest bandwidth at 10kbps, why? In my tc setup, p2p is only given a rate of 4bps, and then a ciel of about 75% of my connection. This way, p2p has to borrow the bandwidth from everything else, and thus gets last grab at bandwidth. Giving it a guarentee on par with the second highest class, of course your connection still seems a little swamped. - Jody On 7/8/05, Francisco Pereira wrote: > Mensaje citado por Edgar : > > > Hello, its me again, I won't stop sending emails to this list, until I solve > > > > this problem, I've tried several apps to create the right htb rules (even > > made them my self), but I always get the same results, traffic gets shaped, > > but I can't use my bandwidth, and this is weird, because I should be able to, > > > > also I keep seeing download being limited too, and that shouldn't be > > happening, I don't think I'm the only one with this problem out there, so I > > will post here once again my configuration, to match p2p traffic and to shape > > > > it: > > > > ### ### > > iptables -t mangle -F > > iptables -t mangle -X > > iptables -t mangle -N lay7PRE > > iptables -t mangle -N lay7POST > > iptables -t mangle -N ipp2pPRE > > iptables -t mangle -N ipp2pPOST > > > > ### PREROUTING RULES ### > > iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark > > iptables -t mangle -A PREROUTING -m mark ! --mark 0 -j ACCEPT > > iptables -t mangle -A ipp2pPRE -p tcp -m ipp2p --ipp2p -j MARK --set-mark 3 > > iptables -t mangle -A ipp2pPRE -p udp -m ipp2p --bit -j MARK --set-mark 3 > > iptables -t mangle -A ipp2pPRE -p tcp -m mark --mark 3 -j CONNMARK > > --save-mark > > iptables -t mangle -A lay7PRE -m layer7 --l7proto smtp -j MARK --set-mark 1 > > iptables -t mangle -A lay7PRE -m layer7 --l7proto ssh -j MARK --set-mark 2 > > iptables -t mangle -A lay7PRE -m mark --mark 2 -j CONNMARK --save-mark > > > > ### POSTROUTING RULES ### > > iptables -t mangle -A ipp2pPOST -o eth1 -m mark --mark 3 -j CLASSIFY > > --set-class 1:30 > > iptables -t mangle -A lay7POST -o eth1 -m mark --mark 1 -j CLASSIFY > > --set-class 1:20 > > iptables -t mangle -A lay7POST -o eth1 -m mark --mark 2 -j CLASSIFY > > --set-class 1:10 > > > > ### ### > > iptables -t mangle -A PREROUTING -j lay7PRE > > iptables -t mangle -A POSTROUTING -j lay7POST > > iptables -t mangle -A PREROUTING -j ipp2pPRE > > iptables -t mangle -A POSTROUTING -j ipp2pPOST > > > > TC RULES > > tc qdisc add dev eth1 handle 1:0 root dsmark indices 8 default_index 0 > > tc qdisc add dev eth1 handle 2:0 parent 1:0 htb > > tc class add dev eth1 parent 2:0 classid 2:1 htb rate 22000bps ceil 22000bps > > tc class add dev eth1 parent 2:1 classid 2:2 htb rate 3000bps ceil 5000bps > > prio 0 > > tc qdisc add dev eth1 handle 3:0 parent 2:2 sfq > > tc class add dev eth1 parent 2:1 classid 2:3 htb rate 5000bps ceil 6000bps > > prio 1 > > tc qdisc add dev eth1 handle 4:0 parent 2:3 sfq > > tc class add dev eth1 parent 2:1 classid 2:4 htb rate 8000bps ceil 8000bps > > prio 2 > > tc qdisc add dev eth1 handle 5:0 parent 2:4 sfq > > tc class add dev eth1 parent 2:1 classid 2:5 htb rate 10000bps ceil 10000bps > > > > burst 0 cburst 0 prio 3 quantum 1500 > > tc qdisc add dev eth1 handle 6:0 parent 2:5 sfq > > tc class add dev eth1 parent 2:1 classid 2:6 htb rate 10000bps ceil 22000bps > > > > prio 1 > > tc qdisc add dev eth1 handle 7:0 parent 2:6 sfq > > tc filter add dev eth1 parent 2:0 protocol all prio 1 tcindex mask 0x7 shift > > 0 > > tc filter add dev eth1 parent 2:0 protocol all prio 1 handle 5 tcindex > > classid > > 2:6 > > tc filter add dev eth1 parent 2:0 protocol all prio 1 handle 4 tcindex > > classid > > 2:4 > > tc filter add dev eth1 parent 2:0 protocol all prio 1 handle 3 tcindex > > classid > > 2:5 > > tc filter add dev eth1 parent 2:0 protocol all prio 1 handle 2 tcindex > > classid > > 2:3 > > tc filter add dev eth1 parent 2:0 protocol all prio 1 handle 1 tcindex > > classid > > 2:2 > > tc filter add dev eth1 parent 1:0 protocol all prio 1 u32 match u8 0x5 0xf at > > > > 0 match u16 0x0 0xffc0 at 2 classid 1:5 > > tc filter add dev eth1 parent 1:0 protocol all prio 1 u32 match u8 0x5 0xf at > > > > 0 match u8 0x10 0x10 at 33 classid 1:4 > > tc filter add dev eth1 parent 1:0 protocol all prio 1 u32 match u8 0x5 0xf at > > > > 0 classid 1:5 > > tc filter add dev eth1 parent 1:0 protocol all prio 1 u32 match u32 0x0 0x0 > > at > > 0 classid 1:5 > > > > well, this rules are generated with tcng (tcc actually), and I see traffic > > going to the p2p rule (being 2:5), but still I'm not able to browse the web! > > > > nor chat nor anything, someone adviced me to create an interactive class and > > > > then leave the unmatched traffic go to a default class (and here'll go p2p > > traffic too), but then I'm running a squid server at home, and thus I don't > > know what port to specify to do this, anyway, I've tried already with normal > > > > http traffic, with layer7, I see the traffic going to the iptables chain, but > > > > yet it'll still won't work, p2p traffic keeps eating all my bandwidth, its > > ignoring all the priorities I have in my tc rules (being the 2:5 class the > > one with the lowest priority), I really need to get this working, and I know > > > > there's some ppl that has lots of more experience that me, and I'm sure you > > can help me. Thank you > > can != want != will > > You may have a problem with the rate units - bps != bit -, but i dont know the > type or rate of your internet link. > Post a detailed ascii diagram of your network. Which is the transfer rate of the > internet link? > > IPP2P works very well for me. > How did you test your iptables & tc setup? In the real network or in a > controlled environment? > Did you try starting with a very basic tc setup and adding tc qdiscs, classes > and filters one by one and checking the results on every step? > > Regards, > Francisco. > > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > Miles de internautas ya utilizan el discador del Portal. > Bajatelo ahora, se instala solo!. > > http://www.montevideo.com.uy/discador/setup.exe > > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > From admin at premiere-heure.fr Fri Jul 8 16:05:18 2005 From: admin at premiere-heure.fr (Gael Mauleon) Date: Fri Jul 8 16:04:52 2005 Subject: [LARTC] HTB Rate and Prio Message-ID: <200507081404.j68E4glj022048@dhuumrelay0.mail.eu.uu.net> Hi, I wanted to implement some QOS on my Linux Box with HTB, but after some time spend on the configuration and tests, I still don't manage to have some correct results. Here are the details : -ROOT 2000 kbits -HIGHPRIO SUBCLASS 50 kbits prio 0 -SUBCLASS1 750 kbits prio 1 -SERVICE1 250 kbits prio 1 -SERVICE2 250 kbits prio 1 -SERVICE3 250 kbits prio 1 -SUBCLASS2 750 kbits prio 1 -SERVICE1 250 kbits prio 1 -SERVICE2 250 kbits prio 1 -SERVICE3 250 kbits prio 1 -SUBCLASS3 400 kbits prio 1 -SERVICE1 200 kbits prio 1 -SERVICE2 200 kbits prio 1 -LOWPRIO SUBCLASS 50 kbits prio 5 Here is the details of the implementation, I only wrote 1 on the subclass Cause they are all on the same template. tc qdisc add dev $QOSIN root handle 1:0 htb default 1000 tc class add dev $QOSIN parent 1:0 classid 1:1 htb rate 2000kbit ### SUBCLASS1 tc class add dev $QOSIN parent 1:1 classid 1:10 htb rate 750kbit ceil 2000kbit prio 1 tc class add dev $QOSIN parent 1:10 classid 1:101 htb rate 250kbit ceil 2000kbit prio 1 tc qdisc add dev $QOSIN parent 1:101 handle 101: pfifo limit 10 tc class add dev $QOSIN parent 1:10 classid 1:102 htb rate 250kbit ceil 2000kbit prio 1 tc qdisc add dev $QOSIN parent 1:102 handle 102: pfifo limit 10 tc class add dev $QOSIN parent 1:10 classid 1:103 htb rate 250kbit ceil 2000kbit prio 1 tc qdisc add dev $QOSIN parent 1:103 handle 103: pfifo limit 10 tc filter add dev $QOSIN parent 1:0 protocol ip handle $OUTPROD$MAIL fw flowid 1:101 tc filter add dev $QOSIN parent 1:0 protocol ip handle $OUTPROD$HTTP fw flowid 1:102 tc filter add dev $QOSIN parent 1:0 protocol ip handle $OUTPROD$FTP fw flowid 1:103 etc. ### HIGH PRIO ### tc class add dev $QOSIN parent 1:1 classid 1:50 htb rate 50kbit ceil 2000kbit prio 0 quantum 1500 tc qdisc add dev $QOSIN parent 1:50 handle 50: pfifo limit 10 tc filter add dev $QOSIN parent 1:0 protocol ip handle $OUTPROD$HIGHPRIO fw flowid 1:50 tc filter add dev $QOSIN parent 1:0 protocol ip handle $OUTPOSTPROD$HIGHPRIO fw flowid 1:50 tc filter add dev $QOSIN parent 1:0 protocol ip handle $OUTDMZ$HIGHPRIO fw flowid 1:50 ### LOW PRIO ### tc class add dev $QOSIN parent 1:1 classid 1:60 htb rate 50kbit ceil 2000kbit prio 5 quantum 1500 tc qdisc add dev $QOSIN parent 1:60 handle 60: pfifo limit 10 tc filter add dev $QOSIN parent 1:0 protocol ip handle $OUTPROD$LOWPRIO fw flowid 1:60 tc filter add dev $QOSIN parent 1:0 protocol ip handle $OUTPOSTPROD$LOWPRIO fw flowid 1:60 tc filter add dev $QOSIN parent 1:0 protocol ip handle $OUTDMZ$LOWPRIO fw flowid 1:60 Every traffic seems to go in the class it must go, the stats are good and if I change any of the ceil rate the associated traffic is caped to the right ceil I enter. Now with this configuration I expected that when one of the SUBCLASS class or SERVICE want more bandwith than its rate, she can borrow it from root and she had it before LOW PRIO and after HIGH PRIO. But it don't work at all, for exemple I tried only with 2 flow, I have 500 Kbits of LOW PRIO traffic that is currently going on, then I fire some SERVICE1 traffic from SUBCLASS1 that can theorically take 2000 kbits, and instead of taking it from LOW PRIO, it just take what is left. I surely miss something. Thanks for your help and don't hesitate to ask more infos J Gael. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ds9a.nl/pipermail/lartc/attachments/20050708/28857d33/attachment-0001.htm From jody.shumaker at gmail.com Fri Jul 8 16:58:44 2005 From: jody.shumaker at gmail.com (Jody Shumaker) Date: Fri Jul 8 16:58:50 2005 Subject: [LARTC] HTB Rate and Prio In-Reply-To: <200507081404.j68E4glj022048@dhuumrelay0.mail.eu.uu.net> References: <200507081404.j68E4glj022048@dhuumrelay0.mail.eu.uu.net> Message-ID: <2af4364905070807583ee4e93a@mail.gmail.com> The priority effects the ratio in which extra bandwidth is shared. Something with a better priority doesn't automatically get all the bandwidth it wants before something with a worse priority, they share it on a ratio basis. You might be better off not using prio and instead just having the rate's for the classes set to the ratio you want. When prio isn't set htb uses the rate's as the ratio. Try removing the prio, and lowering the low-priority even further, say 1kbit instead of 50kbit, then when bandwidth is shared, the other classes will have 750:1 or 400:1 splitting of extra bandwidth. This way your LOWPRIO will also have to borrow for nearly all its bandwidth needs. In your example test case, LOWPRIO was using 500kbit, so borrowing 450kbit, and you say the 750kbit class was using only the remaining 1500kbit, borrowing 750kbit. The ratios of shared ammounts definately doesn't seem to fit with the 5:1 priority's ratio, but I'm no expert on how it determines all this. I just recommend not using the prio and try using a lower rate for the LOWPRIO subclass. - Jody On 7/8/05, Gael Mauleon wrote: > > > > > > Hi, > > > > I wanted to implement some QOS on my Linux Box with HTB, but after some time > spend > > on the configuration and tests, I still don't manage to have some correct > results. Here are the details : > > > > > > > > -ROOT 2000 kbits > > -HIGHPRIO SUBCLASS 50 kbits prio 0 > > -SUBCLASS1 750 kbits prio 1 > > -SERVICE1 250 kbits prio 1 > > -SERVICE2 250 kbits prio 1 > > -SERVICE3 250 kbits prio 1 > > -SUBCLASS2 750 kbits prio 1 > > -SERVICE1 250 kbits prio 1 > > -SERVICE2 250 kbits prio 1 > > -SERVICE3 250 kbits prio 1 > > -SUBCLASS3 400 kbits prio 1 > > -SERVICE1 200 kbits prio 1 > > -SERVICE2 200 kbits prio 1 > > -LOWPRIO SUBCLASS 50 kbits prio 5 > > > > Here is the details of the implementation, I only wrote 1 on the subclass > > Cause they are all on the same template. > > > > > > tc qdisc add dev $QOSIN root handle 1:0 htb default 1000 > > tc class add dev $QOSIN parent 1:0 classid 1:1 htb rate 2000kbit > > > > ### SUBCLASS1 > > tc class add dev $QOSIN parent 1:1 classid 1:10 htb rate 750kbit ceil > 2000kbit prio 1 > > tc class add dev $QOSIN parent 1:10 classid 1:101 htb rate 250kbit ceil > 2000kbit prio 1 > > tc qdisc add dev $QOSIN parent 1:101 handle 101: pfifo limit 10 > > tc class add dev $QOSIN parent 1:10 classid 1:102 htb rate 250kbit ceil > 2000kbit prio 1 > > tc qdisc add dev $QOSIN parent 1:102 handle 102: pfifo limit 10 > > tc class add dev $QOSIN parent 1:10 classid 1:103 htb rate 250kbit ceil > 2000kbit prio 1 > > tc qdisc add dev $QOSIN parent 1:103 handle 103: pfifo limit 10 > > > > tc filter add dev $QOSIN parent 1:0 protocol ip handle $OUTPROD$MAIL fw > flowid 1:101 > > tc filter add dev $QOSIN parent 1:0 protocol ip handle $OUTPROD$HTTP fw > flowid 1:102 > > tc filter add dev $QOSIN parent 1:0 protocol ip handle $OUTPROD$FTP fw > flowid 1:103 > > > > etc? > > > > > > ### HIGH PRIO ### > > tc class add dev $QOSIN parent 1:1 classid 1:50 htb rate 50kbit ceil > 2000kbit prio 0 quantum 1500 > > tc qdisc add dev $QOSIN parent 1:50 handle 50: pfifo limit 10 > > tc filter add dev $QOSIN parent 1:0 protocol ip handle $OUTPROD$HIGHPRIO fw > flowid 1:50 > > tc filter add dev $QOSIN parent 1:0 protocol ip handle $OUTPOSTPROD$HIGHPRIO > fw flowid 1:50 > > tc filter add dev $QOSIN parent 1:0 protocol ip handle $OUTDMZ$HIGHPRIO fw > flowid 1:50 > > > > ### LOW PRIO ### > > tc class add dev $QOSIN parent 1:1 classid 1:60 htb rate 50kbit ceil > 2000kbit prio 5 quantum 1500 > > tc qdisc add dev $QOSIN parent 1:60 handle 60: pfifo limit 10 > > tc filter add dev $QOSIN parent 1:0 protocol ip handle $OUTPROD$LOWPRIO fw > flowid 1:60 > > tc filter add dev $QOSIN parent 1:0 protocol ip handle $OUTPOSTPROD$LOWPRIO > fw flowid 1:60 > > tc filter add dev $QOSIN parent 1:0 protocol ip handle $OUTDMZ$LOWPRIO fw > flowid 1:60 > > > > > > Every traffic seems to go in the class it must go, the stats are good and if > I change any of the ceil rate > > the associated traffic is caped to the right ceil I enter. > > > > Now with this configuration I expected that when one of the SUBCLASS class > or SERVICE want more bandwith than its > > rate, she can borrow it from root and she had it before LOW PRIO and after > HIGH PRIO. > > > > But it don't work at all, for exemple I tried only with 2 flow, I have 500 > Kbits of LOW PRIO traffic that is currently > > going on, then I fire some SERVICE1 traffic from SUBCLASS1 that can > theorically take 2000 kbits, and instead > > of taking it from LOW PRIO, it just take what is left? > > > > I surely miss something? > > > > Thanks for your help and don't hesitate to ask more infos J > > > > Gael. > > > > > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > > > From admin at premiere-heure.fr Fri Jul 8 18:05:18 2005 From: admin at premiere-heure.fr (Gael Mauleon) Date: Fri Jul 8 18:04:51 2005 Subject: [LARTC] HTB Rate and Prio In-Reply-To: <2af4364905070807583ee4e93a@mail.gmail.com> Message-ID: <200507081604.j68G4ghT010166@dhuumrelay0.mail.eu.uu.net> Thanks for the anwser, I just tried that with a 10 kbits rate and no Priority but it don't seems to change something in the behavior of the QOS... Well to make it clear here is the full infos (sorry for the flood ) -> class htb 1:101 parent 1:10 leaf 101: prio 0 rate 250000bit ceil 2000Kbit burst 1912b cburst 4Kb Sent 266737 bytes 372 pkts (dropped 0, overlimits 0) rate 48440bit 5pps lended: 265 borrowed: 107 giants: 0 tokens: 87000 ctokens: 23695 class htb 1:202 parent 1:20 leaf 202: prio 0 rate 250000bit ceil 2000Kbit burst 1912b cburst 4Kb Sent 39266 bytes 325 pkts (dropped 0, overlimits 0) rate 3800bit 3pps lended: 325 borrowed: 0 giants: 0 tokens: 87522 ctokens: 23789 class htb 1:1 root rate 2000Kbit ceil 2000Kbit burst 4Kb cburst 4Kb Sent 43212627 bytes 42882 pkts (dropped 0, overlimits 0) rate 1765Kbit 212pps lended: 28370 borrowed: 0 giants: 0 tokens: -17118 ctokens: -17118 class htb 1:10 parent 1:1 rate 750000bit ceil 2000Kbit burst 2536b cburst 4Kb Sent 37117090 bytes 28128 pkts (dropped 0, overlimits 0) rate 1551Kbit 143pps lended: 7561 borrowed: 14583 giants: 0 tokens: -71625 ctokens: -17118 class htb 1:203 parent 1:20 leaf 203: prio 0 rate 250000bit ceil 2000Kbit burst 1912b cburst 4Kb Sent 0 bytes 0 pkts (dropped 0, overlimits 0) lended: 0 borrowed: 0 giants: 0 tokens: 89625 ctokens: 24023 class htb 1:103 parent 1:10 leaf 103: prio 0 rate 250000bit ceil 2000Kbit burst 1912b cburst 4Kb Sent 36716334 bytes 26167 pkts (dropped 0, overlimits 0) rate 1505Kbit 133pps backlog 4p lended: 4126 borrowed: 22037 giants: 0 tokens: -114281 ctokens: -17118 class htb 1:20 parent 1:1 rate 750000bit ceil 2000Kbit burst 2536b cburst 4Kb Sent 39266 bytes 325 pkts (dropped 0, overlimits 0) rate 3800bit 3pps lended: 0 borrowed: 0 giants: 0 tokens: 39016 ctokens: 23789 class htb 1:102 parent 1:10 leaf 102: prio 0 rate 250000bit ceil 2000Kbit burst 1912b cburst 4Kb Sent 138883 bytes 1593 pkts (dropped 0, overlimits 0) rate 3176bit 4pps lended: 1593 borrowed: 0 giants: 0 tokens: 87750 ctokens: 23789 class htb 1:201 parent 1:20 leaf 201: prio 0 rate 250000bit ceil 2000Kbit burst 1912b cburst 4Kb Sent 0 bytes 0 pkts (dropped 0, overlimits 0) lended: 0 borrowed: 0 giants: 0 tokens: 89625 ctokens: 24023 class htb 1:40 parent 1:1 rate 400000bit ceil 2000Kbit burst 2099b cburst 4Kb Sent 0 bytes 0 pkts (dropped 0, overlimits 0) lended: 0 borrowed: 0 giants: 0 tokens: 61523 ctokens: 24023 class htb 1:50 parent 1:1 leaf 50: prio 0 rate 90000bit ceil 2000Kbit burst 1711b cburst 4Kb Sent 3516 bytes 55 pkts (dropped 0, overlimits 0) rate 144bit lended: 55 borrowed: 0 giants: 0 tokens: 214583 ctokens: 23648 class htb 1:402 parent 1:40 leaf 402: prio 0 rate 200000bit ceil 2000Kbit burst 1849b cburst 4Kb Sent 0 bytes 0 pkts (dropped 0, overlimits 0) lended: 0 borrowed: 0 giants: 0 tokens: 108398 ctokens: 24023 class htb 1:60 parent 1:1 leaf 60: prio 0 rate 10000bit ceil 2000Kbit burst 1611b cburst 4Kb Sent 6052835 bytes 14376 pkts (dropped 0, overlimits 0) rate 210184bit 65pps backlog 2p lended: 587 borrowed: 13787 giants: 0 tokens: -2173243 ctokens: 8339 class htb 1:403 parent 1:40 leaf 403: prio 0 rate 200000bit ceil 2000Kbit burst 1849b cburst 4Kb Sent 0 bytes 0 pkts (dropped 0, overlimits 0) lended: 0 borrowed: 0 giants: 0 tokens: 108398 ctokens: 24023 ############################## qdisc htb 1: dev imq1 r2q 10 default 1000 direct_packets_stat 11 Sent 46576898 bytes 49981 pkts (dropped 0, overlimits 33216) qdisc sfq 101: dev imq1 parent 1:101 limit 128p quantum 1500b perturb 10sec Sent 295953 bytes 742 pkts (dropped 0, overlimits 0) qdisc sfq 102: dev imq1 parent 1:102 limit 128p quantum 1500b perturb 10sec Sent 163583 bytes 1984 pkts (dropped 0, overlimits 0) qdisc sfq 103: dev imq1 parent 1:103 limit 128p quantum 1500b perturb 10sec Sent 38352553 bytes 27341 pkts (dropped 0, overlimits 0) qdisc sfq 201: dev imq1 parent 1:201 limit 128p quantum 1500b perturb 10sec Sent 0 bytes 0 pkts (dropped 0, overlimits 0) qdisc sfq 202: dev imq1 parent 1:202 limit 128p quantum 1500b perturb 10sec Sent 51981 bytes 415 pkts (dropped 0, overlimits 0) qdisc sfq 203: dev imq1 parent 1:203 limit 128p quantum 1500b perturb 10sec Sent 47507 bytes 904 pkts (dropped 0, overlimits 0) qdisc sfq 403: dev imq1 parent 1:403 limit 128p quantum 1500b perturb 10sec Sent 0 bytes 0 pkts (dropped 0, overlimits 0) qdisc sfq 402: dev imq1 parent 1:402 limit 128p quantum 1500b perturb 10sec Sent 0 bytes 0 pkts (dropped 0, overlimits 0) qdisc sfq 50: dev imq1 parent 1:50 limit 128p quantum 1500b perturb 10sec Sent 4089 bytes 64 pkts (dropped 0, overlimits 0) qdisc sfq 60: dev imq1 parent 1:60 limit 128p quantum 1500b perturb 10sec Sent 7659072 bytes 18520 pkts (dropped 0, overlimits 0) ############################## I changed pfifo queue to sfq ones, it seems i have better results with them, And clear the priorities but none changed... So the interesting things here are the 1:60 class and the 1:103 one. 1:60 is the low prio stuffs so she add only 10kbits rate and the 1:103 one have 250kbits rate and is a child of 1:10 which have 750kbits rate. The 1:103 is sending to max so in theory, regarding the rates she would take almost all the bandwith but 1:60 just don't move, she was at 210kbits before 1:103 start to send and is still at 210kbits after...and she don't even drop 1 packet.... This is with or without priorities (was at 5 on 1:60 and 1 on 1:103 before) The second things I'm not sure is the overlimits of the main qdisc, what does it mean ?? It's just like the whole qos system don't even work, but i repeat if i set for exemple the ceil of 1:103 to 50kbits or the ceil of 1:60 to 50 kbits they are limited to those rate and drop counter goes up... Packets are there but they don't seem to be shaped just caped if I tune the ceil of class... In short.. HEELLLLP :o)p > -----Message d'origine----- > De?: lartc-bounces@mailman.ds9a.nl [mailto:lartc-bounces@mailman.ds9a.nl] > De la part de Jody Shumaker > Envoy??: vendredi 8 juillet 2005 16:59 > ??: lartc@mailman.ds9a.nl > Objet?: Re: [LARTC] HTB Rate and Prio > > The priority effects the ratio in which extra bandwidth is shared. > Something with a better priority doesn't automatically get all the > bandwidth it wants before something with a worse priority, they share > it on a ratio basis. You might be better off not using prio and > instead just having the rate's for the classes set to the ratio you > want. When prio isn't set htb uses the rate's as the ratio. Try > removing the prio, and lowering the low-priority even further, say > 1kbit instead of 50kbit, then when bandwidth is shared, the other > classes will have 750:1 or 400:1 splitting of extra bandwidth. This > way your LOWPRIO will also have to borrow for nearly all its bandwidth > needs. > > In your example test case, LOWPRIO was using 500kbit, so borrowing > 450kbit, and you say the 750kbit class was using only the remaining > 1500kbit, borrowing 750kbit. The ratios of shared ammounts definately > doesn't seem to fit with the 5:1 priority's ratio, but I'm no expert > on how it determines all this. I just recommend not using the prio > and try using a lower rate for the LOWPRIO subclass. > > - Jody > From donvodka at gmail.com Fri Jul 8 21:11:48 2005 From: donvodka at gmail.com (Edgar) Date: Fri Jul 8 21:12:31 2005 Subject: [LARTC] HTB Rate and Prio In-Reply-To: <200507081604.j68G4ghT010166@dhuumrelay0.mail.eu.uu.net> References: <200507081604.j68G4ghT010166@dhuumrelay0.mail.eu.uu.net> Message-ID: <200507081411.48557.donvodka@gmail.com> Hello, I'm having kind of the same problem with bittorrent, I'm about to lower the rates to see what happens, but I'm also suffering from that, the p2p class should be lowprio, and everything else normal and high, but when bittorrent is up (and it's got only 10kb out of 22kb available) I can't do anything else, I'm gonna keep a track on your post and provide you the help I find and get by other in this list, and see if it works for both. EDGAR MERINO On Friday 08 July 2005 11:05, Gael Mauleon wrote: > Thanks for the anwser, I just tried that with a 10 kbits rate and no > Priority but it don't seems to change something in the behavior of the > QOS... > > Well to make it clear here is the full infos (sorry for the flood ) -> > > class htb 1:101 parent 1:10 leaf 101: prio 0 rate 250000bit ceil 2000Kbit > burst 1912b cburst 4Kb > Sent 266737 bytes 372 pkts (dropped 0, overlimits 0) > rate 48440bit 5pps > lended: 265 borrowed: 107 giants: 0 > tokens: 87000 ctokens: 23695 > > class htb 1:202 parent 1:20 leaf 202: prio 0 rate 250000bit ceil 2000Kbit > burst 1912b cburst 4Kb > Sent 39266 bytes 325 pkts (dropped 0, overlimits 0) > rate 3800bit 3pps > lended: 325 borrowed: 0 giants: 0 > tokens: 87522 ctokens: 23789 > > class htb 1:1 root rate 2000Kbit ceil 2000Kbit burst 4Kb cburst 4Kb > Sent 43212627 bytes 42882 pkts (dropped 0, overlimits 0) > rate 1765Kbit 212pps > lended: 28370 borrowed: 0 giants: 0 > tokens: -17118 ctokens: -17118 > > class htb 1:10 parent 1:1 rate 750000bit ceil 2000Kbit burst 2536b cburst > 4Kb > Sent 37117090 bytes 28128 pkts (dropped 0, overlimits 0) > rate 1551Kbit 143pps > lended: 7561 borrowed: 14583 giants: 0 > tokens: -71625 ctokens: -17118 > > class htb 1:203 parent 1:20 leaf 203: prio 0 rate 250000bit ceil 2000Kbit > burst 1912b cburst 4Kb > Sent 0 bytes 0 pkts (dropped 0, overlimits 0) > lended: 0 borrowed: 0 giants: 0 > tokens: 89625 ctokens: 24023 > > class htb 1:103 parent 1:10 leaf 103: prio 0 rate 250000bit ceil 2000Kbit > burst 1912b cburst 4Kb > Sent 36716334 bytes 26167 pkts (dropped 0, overlimits 0) > rate 1505Kbit 133pps backlog 4p > lended: 4126 borrowed: 22037 giants: 0 > tokens: -114281 ctokens: -17118 > > class htb 1:20 parent 1:1 rate 750000bit ceil 2000Kbit burst 2536b cburst > 4Kb > Sent 39266 bytes 325 pkts (dropped 0, overlimits 0) > rate 3800bit 3pps > lended: 0 borrowed: 0 giants: 0 > tokens: 39016 ctokens: 23789 > > class htb 1:102 parent 1:10 leaf 102: prio 0 rate 250000bit ceil 2000Kbit > burst 1912b cburst 4Kb > Sent 138883 bytes 1593 pkts (dropped 0, overlimits 0) > rate 3176bit 4pps > lended: 1593 borrowed: 0 giants: 0 > tokens: 87750 ctokens: 23789 > > class htb 1:201 parent 1:20 leaf 201: prio 0 rate 250000bit ceil 2000Kbit > burst 1912b cburst 4Kb > Sent 0 bytes 0 pkts (dropped 0, overlimits 0) > lended: 0 borrowed: 0 giants: 0 > tokens: 89625 ctokens: 24023 > > class htb 1:40 parent 1:1 rate 400000bit ceil 2000Kbit burst 2099b cburst > 4Kb > Sent 0 bytes 0 pkts (dropped 0, overlimits 0) > lended: 0 borrowed: 0 giants: 0 > tokens: 61523 ctokens: 24023 > > class htb 1:50 parent 1:1 leaf 50: prio 0 rate 90000bit ceil 2000Kbit burst > 1711b cburst 4Kb > Sent 3516 bytes 55 pkts (dropped 0, overlimits 0) > rate 144bit > lended: 55 borrowed: 0 giants: 0 > tokens: 214583 ctokens: 23648 > > class htb 1:402 parent 1:40 leaf 402: prio 0 rate 200000bit ceil 2000Kbit > burst 1849b cburst 4Kb > Sent 0 bytes 0 pkts (dropped 0, overlimits 0) > lended: 0 borrowed: 0 giants: 0 > tokens: 108398 ctokens: 24023 > > class htb 1:60 parent 1:1 leaf 60: prio 0 rate 10000bit ceil 2000Kbit burst > 1611b cburst 4Kb > Sent 6052835 bytes 14376 pkts (dropped 0, overlimits 0) > rate 210184bit 65pps backlog 2p > lended: 587 borrowed: 13787 giants: 0 > tokens: -2173243 ctokens: 8339 > > class htb 1:403 parent 1:40 leaf 403: prio 0 rate 200000bit ceil 2000Kbit > burst 1849b cburst 4Kb > Sent 0 bytes 0 pkts (dropped 0, overlimits 0) > lended: 0 borrowed: 0 giants: 0 > tokens: 108398 ctokens: 24023 > > > ############################## > > qdisc htb 1: dev imq1 r2q 10 default 1000 direct_packets_stat 11 > Sent 46576898 bytes 49981 pkts (dropped 0, overlimits 33216) > qdisc sfq 101: dev imq1 parent 1:101 limit 128p quantum 1500b perturb 10sec > Sent 295953 bytes 742 pkts (dropped 0, overlimits 0) > qdisc sfq 102: dev imq1 parent 1:102 limit 128p quantum 1500b perturb 10sec > Sent 163583 bytes 1984 pkts (dropped 0, overlimits 0) > qdisc sfq 103: dev imq1 parent 1:103 limit 128p quantum 1500b perturb 10sec > Sent 38352553 bytes 27341 pkts (dropped 0, overlimits 0) > qdisc sfq 201: dev imq1 parent 1:201 limit 128p quantum 1500b perturb 10sec > Sent 0 bytes 0 pkts (dropped 0, overlimits 0) > qdisc sfq 202: dev imq1 parent 1:202 limit 128p quantum 1500b perturb 10sec > Sent 51981 bytes 415 pkts (dropped 0, overlimits 0) > qdisc sfq 203: dev imq1 parent 1:203 limit 128p quantum 1500b perturb 10sec > Sent 47507 bytes 904 pkts (dropped 0, overlimits 0) > qdisc sfq 403: dev imq1 parent 1:403 limit 128p quantum 1500b perturb 10sec > Sent 0 bytes 0 pkts (dropped 0, overlimits 0) > qdisc sfq 402: dev imq1 parent 1:402 limit 128p quantum 1500b perturb 10sec > Sent 0 bytes 0 pkts (dropped 0, overlimits 0) > qdisc sfq 50: dev imq1 parent 1:50 limit 128p quantum 1500b perturb 10sec > Sent 4089 bytes 64 pkts (dropped 0, overlimits 0) > qdisc sfq 60: dev imq1 parent 1:60 limit 128p quantum 1500b perturb 10sec > Sent 7659072 bytes 18520 pkts (dropped 0, overlimits 0) > > > > ############################## > > > I changed pfifo queue to sfq ones, it seems i have better results with > them, And clear the priorities but none changed... > > So the interesting things here are the 1:60 class and the 1:103 one. > 1:60 is the low prio stuffs so she add only 10kbits rate and the > 1:103 one have 250kbits rate and is a child of 1:10 which have 750kbits > rate. > > The 1:103 is sending to max so in theory, regarding the rates she would > take almost all the bandwith but 1:60 just don't move, she was at 210kbits > before 1:103 start to send and is still at 210kbits after...and she don't > even drop 1 packet.... > This is with or without priorities (was at 5 on 1:60 and 1 on 1:103 before) > > The second things I'm not sure is the overlimits of the main qdisc, what > does it mean ?? > > It's just like the whole qos system don't even work, but i repeat if i set > for exemple the ceil of 1:103 to 50kbits or the ceil of 1:60 to 50 kbits > they are limited to those rate and drop counter goes up... > > Packets are there but they don't seem to be shaped just caped if I tune the > ceil of class... > > In short.. HEELLLLP :o)p > > > -----Message d'origine----- > > De?: lartc-bounces@mailman.ds9a.nl [mailto:lartc-bounces@mailman.ds9a.nl] > > De la part de Jody Shumaker > > Envoy??: vendredi 8 juillet 2005 16:59 > > ??: lartc@mailman.ds9a.nl > > Objet?: Re: [LARTC] HTB Rate and Prio > > > > The priority effects the ratio in which extra bandwidth is shared. > > Something with a better priority doesn't automatically get all the > > bandwidth it wants before something with a worse priority, they share > > it on a ratio basis. You might be better off not using prio and > > instead just having the rate's for the classes set to the ratio you > > want. When prio isn't set htb uses the rate's as the ratio. Try > > removing the prio, and lowering the low-priority even further, say > > 1kbit instead of 50kbit, then when bandwidth is shared, the other > > classes will have 750:1 or 400:1 splitting of extra bandwidth. This > > way your LOWPRIO will also have to borrow for nearly all its bandwidth > > needs. > > > > In your example test case, LOWPRIO was using 500kbit, so borrowing > > 450kbit, and you say the 750kbit class was using only the remaining > > 1500kbit, borrowing 750kbit. The ratios of shared ammounts definately > > doesn't seem to fit with the 5:1 priority's ratio, but I'm no expert > > on how it determines all this. I just recommend not using the prio > > and try using a lower rate for the LOWPRIO subclass. > > > > - Jody > > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc From kirk at braille.uwo.ca Fri Jul 8 21:23:25 2005 From: kirk at braille.uwo.ca (Kirk Reiser) Date: Fri Jul 8 21:23:31 2005 Subject: [LARTC] HTB Rate and Prio In-Reply-To: <200507081411.48557.donvodka@gmail.com> References: <200507081604.j68G4ghT010166@dhuumrelay0.mail.eu.uu.net> <200507081411.48557.donvodka@gmail.com> Message-ID: I don't quite understand this problem with bit torrent. When I start bittorrent with it's max_upload_rate to a value less than my total up link bandwidth it doesn't get in the way of anything at all as far as I can tell. Kirk -- Kirk Reiser The Computer Braille Facility e-mail: kirk@braille.uwo.ca University of Western Ontario phone: (519) 661-3061 From jarod125 at yahoo.com Fri Jul 8 21:44:33 2005 From: jarod125 at yahoo.com (Gabriel) Date: Fri Jul 8 21:44:36 2005 Subject: [LARTC] gre tunnel between networks with same subnet Message-ID: <20050708194433.14783.qmail@web60915.mail.yahoo.com> /-----------------------\ | | |eth0 |eth0 |-------| |-------| | |eth1 eth1 | | -------- A |____ _______| B |----- | | \ / | | --------| | | --------| | | | | ----------- |___________| switch What you see above is my setup. Box A is connected to Box B through a switch. Box A is connected to the Internet through eth0, same with Box B. The link that goes through the switch is not very reliable, so I want to connect the two boxes using their Internet link via a gre tunnel. The problem is that the boxes are on the same subnet (and I can't change that). I've read about proxy arp, about bridging, but things are still confused. Here are some numbers: eth1 on Box A is 192.168.1.1/24, eth1 on Box B is 192.168.1.31/24. On Box B there are 4 NICs, 3 of them (including eth1) are bridged, with the bridge interface being br0 (192.168.1.31 is actually assigned to br0, not eth1). I've read the lartc howto, so I created a tun0 interface on both boxes: ip tunnel add tun0 mode gre remote remote_ip_here local local_ip_here ttl 255; ip link set tun0 up. The problem is what do I do from here? Do I bridge tun0 and eth1 on Box A and add tun0 to br0 on Box B? Or do I just enable proxy_arp for eth1 and tun0 on Box A and for br0 and tun0 on B? Are there any routes neccesary (my guess is no, but I'm not very sure)? And about proxy_arp: what do I have to do to turn it on, just set /proc/sys/net/ipv4/conf//proxy_arp to 1 and that's it? One last thing: http://leaf.sourceforge.net/doc/howto/proxyarp.html#id2805973 says proxy-arp is not bridging (agreed) so DO NOT CONFIGURE BRIDGE OPTIONS!!! Does this mean using bridging and doing proxy-arp on the same box is not possible? Thanks. (hope the ascii art comes out well) __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From alaios at yahoo.com Fri Jul 8 22:19:40 2005 From: alaios at yahoo.com (Alaios) Date: Fri Jul 8 22:19:44 2005 Subject: [LARTC] libnl api In-Reply-To: <20050630102138.29732.qmail@web54703.mail.yahoo.com> Message-ID: <20050708201940.57177.qmail@web54704.mail.yahoo.com> HI .. :) Have u ever used libnl (http://people.suug.ch/~tgr/libnl/) for handling qdiscs? I have some questions and i want to know if there is an appropriate place for asking questions.. I dont know and please tell me if i can ask some questions also here... Thx a lot ____________________________________________________ Sell on Yahoo! Auctions ? no fees. Bid on great items. http://auctions.yahoo.com/ From tgraf at suug.ch Fri Jul 8 22:27:02 2005 From: tgraf at suug.ch (Thomas Graf) Date: Fri Jul 8 22:26:45 2005 Subject: [LARTC] libnl api In-Reply-To: <20050708201940.57177.qmail@web54704.mail.yahoo.com> References: <20050630102138.29732.qmail@web54703.mail.yahoo.com> <20050708201940.57177.qmail@web54704.mail.yahoo.com> Message-ID: <20050708202702.GD16076@postel.suug.ch> * Alaios <20050708201940.57177.qmail@web54704.mail.yahoo.com> 2005-07-08 13:19 > Have u ever used libnl > (http://people.suug.ch/~tgr/libnl/) > for handling qdiscs? I have some questions and i want > to know if there is an appropriate place for asking > questions.. I dont know and please tell me if i can > ask some questions also here... You may direct such questions to me, I'll be pleased to answer them. From gregoriandres at yahoo.com.ar Fri Jul 8 22:41:02 2005 From: gregoriandres at yahoo.com.ar (:: L i n u XK i D ::) Date: Fri Jul 8 22:41:03 2005 Subject: [LARTC] block p2p: ARES In-Reply-To: <42C911F6.5040800@ipp2p.org> Message-ID: Hi ! I've tried last the fantastic ipp2p kernel module. My results are that: Ares can be DROPED only Emule, Kazaa and EDonkey 2000 can be limited and/or Droped. And for this I have to use: ..... FW="/usr/local/sbin/iptables" # If I don't put next rule, Ares are not marked: $FW -t mangle -A p2ptraffic -m ipp2p --ares -j DROP # next p2p rules $FW -t mangle -A p2ptraffic -p tcp -j CONNMARK --restore-mark $FW -t mangle -A p2ptraffic -p tcp -m mark ! --mark 0 -j ACCEPT $FW -t mangle -A p2ptraffic -p tcp -m ipp2p --ipp2p -j MARK --set-mark 10 $FW -t mangle -A p2ptraffic -p tcp -m mark --mark 10 -j CONNMARK --save-mark $FW -t mangle -A p2ptraffic -p udp -m ipp2p --ipp2p -j MARK --set-mark 10 ..... iptables-1.3.1 kernel-2.4.28 squid-cache - 2.5-STABLE10 Debian Stable. I hope this information can help for ipp2p module. thank you very much. andres. -> -----Mensaje original----- -> -> Hi, -> -> there is a new version of ipp2p, which can detect ares connections now. -> -> just go to www.ipp2p.org and download this version. -> -> the parameter --ipp2p has changed, this is now ALL protocols -> -> please contact me if you find bugs... -> -> Klaus -> -> Klaus wrote: -> > I did a small test with the new ares version. -> > It seems they have switched their protocol and it is not -> detected at the -> > moment. -> > -> > Lets see how difficult the new ares protocol is and how fast we can -> > integrate this into ipp2p. -> > -> > Klaus -> > -> > :: L i n u XK i D :: wrote: -> > -> >> Hi.... -> >> -> >> I'm trying to setup a LAN router with P2P filter -> >> but the problem is that can't "catch" Ares. -> >> -> >> There is a way to DROP "ares" p2p packets ? -> >> -> >> I've tried with last "ipp2p" snapshot without sucess... -> >> -> >> I've -> >> Kernel 2.4.28 -> >> iptables 1.3.0 -> >> Various Patches from patch-o-matic-ng-20040621 -> >> iproute2-ss020116 -> >> IMQ Patch -> >> Esfq Patch -> >> Julian (route) Patch -> >> Debian Woody -> >> -> >> -> >> This is my MANGLE table... -> >> -> >> -> >> Chain PREROUTING (policy ACCEPT 8557K packets, 2822M bytes) -> >> pkts bytes target prot opt in out source -> >> destination -> >> 85574 24M p2ptraffic all -- * * 0.0.0.0/0 -> >> 0.0.0.0/0 -> >> ................. -> >> -> >> Chain p2ptraffic (1 references) -> >> pkts bytes target prot opt in out source -> >> destination -> >> 11860 1620K CONNMARK all -- * * 0.0.0.0/0 -> >> 0.0.0.0/0 ipp2p v0.7.4 --ipp2p CONNMARK set 0xa -> >> 0 0 CONNMARK all -- * * 0.0.0.0/0 -> >> 0.0.0.0/0 ipp2p v0.7.4 --bit CONNMARK set 0xa -> >> 0 0 CONNMARK all -- * * 0.0.0.0/0 -> >> 0.0.0.0/0 ipp2p v0.7.4 --apple CONNMARK set 0xa -> >> 0 0 CONNMARK all -- * * 0.0.0.0/0 -> >> 0.0.0.0/0 ipp2p v0.7.4 --winmx CONNMARK set 0xa -> >> 1 57 CONNMARK all -- * * 0.0.0.0/0 -> >> 0.0.0.0/0 ipp2p v0.7.4 --soul CONNMARK set 0xa -> >> 0 0 DROP all -- * * 0.0.0.0/0 -> >> 0.0.0.0/0 ipp2p v0.7.4 --ares -> >> ......... -> >> 54029 13M CONNMARK all -- * * 0.0.0.0/0 -> >> 0.0.0.0/0 CONNMARK match 0xa CONNMARK restore -> >> -> >> -> >> But... ARES Packet are not bloked at the momment.... -> >> 0 0 DROP .... ipp2p v0.7.4 --ares -> >> -> >> :-( -> >> -> >> Somebody haves sucessfull blocking ARES ? -> >> -> >> regards... -> >> Andres. -> >> -> >> _______________________________________________ -> >> LARTC mailing list -> >> LARTC@mailman.ds9a.nl -> >> http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc -> > -> > _______________________________________________ -> > LARTC mailing list -> > LARTC@mailman.ds9a.nl -> > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc -> _______________________________________________ -> LARTC mailing list -> LARTC@mailman.ds9a.nl -> http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc From ugo.viti at initzero.it Fri Jul 8 23:56:26 2005 From: ugo.viti at initzero.it (Ugo Viti) Date: Fri Jul 8 23:55:56 2005 Subject: [LARTC] izbalancing - a GNU/Linux Firewall/Router Bash Script feauturing Incoming/Outgoing MultiHomed/LoadBalanced Subsystem Management Message-ID: <1120859786.9368.24.camel@hidlt.localdomain> Hello, i wrote a simple but useful (i hope) bash script featuring a easly and fastly configuration method for implementing a Linux Gateway feauturing Incoming/Outgoing MultiHomed/LoadBalanced Subsystem Management. I want share this little script. you can download a copy from the following address: http://www.initzero.it/products/opensource/izbalancing/download/izbalancing just save it in your hd and open it with a text editor (vi, emacs, gedit...) to configure the minimal variables. the file it self contains istructions... just read. don't forget to 'chmod 750 izbalancing' :-) Follow the readme contained in the script file: Best Regards ############################################################################################## # What is this? # ============= # This bash script allow you to easly and fastly configure a complex Load Balancing Multi Homed Internet Gateway # for inbound and outbound traffic # Key Features: # ============= # - Multiple Balanced Default Gateway Configuration # - Load Balanced outgoing connections from LAN to INTERNET connections # - Management of multiple incoming connection from many INTERNET ISP lines to DMZ/LAN Servers # - SystemV compliant script... you can run easly at boot up (like Red Hat, Fedora, SuSE, Mandrake, etc...) # - Automatically discover your local IP addresses... you can change your IP without reconfigure this script, just restart # - Start and Stop Cleanly your MultiHomed Configuration with simple command (izbalancing start|stop|restart) # - Adding new Internet Connections is very easy and fast # - You only must know the TABLE ID to assign to the new line, Ethernet Name of your NIC, and the Router IP Address # Requirements: # ============= # - GNU/Linux Firewall running Kernel >=2.6.10 (with iptables module CONNMARK available) # - Bash Shell >= 2.0 # - Standard GNU/Linux coreutils utilities (cat, echo, grep, if, etc...) # - GNU Version of awk and sed utilities # - GNU/Linux Netfilter user space utilities (iptables >= 1.2.11) # - iproute2 utilities # - Two or more Internet connections (also from different ISPs and IP classes) # - An ethernet card for each ISP Router # Tested On: # ========== # - GNU/Linux Fedora Core 3 with 2 Internet Connections # - GNU/Linux Fedora Core 4 with 2 Internet Connections # Script Usage: # ============= # 1) Configure or add the following variables: # TABLEn = Table number of internet connection 'n' # IFn = The Ethernet Interface name of internet connection 'n' # GWn = The Router IP Address of Internet connection 'n' # 2) Add a line for any internet connection in the izbalancing function # 3) Modify the command "ip route add default equalize nexthop via ..." adding all ISP's Routers IP # 4) Comment out the iptables rules # 5) Save this file and execute it a boot time (for Red Hat systems you can copy this script in /etc/rc.d/init.d/ directory # and launch 'chkconfig --add izbalancing') # That's all... -- Ugo Viti Linux Red Hat Certified Engineer InitZero S.r.l. Via P. Calamandrei 24, 52100 Arezzo Tel. +39 0575 1822155 - Fax. +39 0575 1822156 WWW: http://www.initzero.it E-Mail: info@initzero.it From fpereira at lojan.com Sat Jul 9 00:58:36 2005 From: fpereira at lojan.com (Francisco Pereira) Date: Sat Jul 9 00:58:35 2005 Subject: [LARTC] HTB Rate and Prio In-Reply-To: References: <200507081604.j68G4ghT010166@dhuumrelay0.mail.eu.uu.net> <200507081411.48557.donvodka@gmail.com> Message-ID: <42CF051C.7010408@lojan.com> Kirk Reiser wrote: > I don't quite understand this problem with bit torrent. When I start > bittorrent with it's max_upload_rate to a value less than my total up > link bandwidth it doesn't get in the way of anything at all as far as > I can tell. > > Kirk > Not all network administrators have your luck. :-) The problem is when you can't control the workstations, and the users can use this kind of network-unfriendly software (that try to "steal" the other software or users bandwidth) I believe that it is important in a network the user's perception of "the network speed", basically in all the interactive traffic. I dont know the existence of a framework to model the user perception, which will allow to take measures of this kind of parameters. I'm not talking about delays, jitter, etc, but about the user's perception, and how the user's perception can be enhanced. Regards, Francisco. From donvodka at gmail.com Sat Jul 9 02:26:56 2005 From: donvodka at gmail.com (Edgar) Date: Sat Jul 9 02:27:12 2005 Subject: [LARTC] P2P shaping working Message-ID: <200507081926.56750.donvodka@gmail.com> Hello, it's me again, finally p2p traffic is getting shaped, but now I still have one problem: download seems to be affected when shaping the traffic, ack packets are in the interactive queue too, as someone adviced me, what fixed the problem was to change the rates, to match (the sum of all subclasses) the root class rate, and to give the p2p class a very low rate (1k actually) and 75% of the bandwidth for the ceiling. Well as I said this is working, but downloading seems to be affected to, I get pretty low rates (this doesn't happen when I'm not shaping the traffic, but then I can't do anything else but bittorrent). Does anyone have a clue on why's this happening? Thanks to everyone that help me, I really appreciate it. EDGAR MERINO From donvodka at gmail.com Sat Jul 9 05:09:12 2005 From: donvodka at gmail.com (Edgar) Date: Sat Jul 9 05:09:29 2005 Subject: [LARTC] Question about TCNG Message-ID: <200507082209.13178.donvodka@gmail.com> Hello, after compiling my tcng rules, I've noticed something weird, the tc filter rules it returns give a flowid differente than the classes it creates, for example, you have a class 2:3 and had defined some conditions, (eg. some ports) to go into that class, but when I compile the .tcc file it returns something like: tc filter add dev eth1 parent 1:0 protocol all prio 1 u32 match u8 0x5 0xf at 0 match u8 0x10 0x10 at 33 classid 1:2 it gives the same classid for all the matches I made (meaning for different classes) and that class doesn't even exist! can someone explain me this please? thank you EDGAR MERINO From staskh at comcast.net Sat Jul 9 05:25:58 2005 From: staskh at comcast.net (Stas Khirman) Date: Sat Jul 9 05:26:09 2005 Subject: [LARTC] Interface bound routing In-Reply-To: <20050620065343.5DAAA3F6A@outpost.ds9a.nl> Message-ID: <20050709032607.021844934@outpost.ds9a.nl> Hi, I'm sorry for "novice"-level question, but I hope your expert advice will save me many painful hours running after my own tail... Let assume I have Linux box with eth0, eth1 and eth2 interfaces. Each one has IP assigned from different network. By default, IP address associated with eth0 is chosen as default routing. My application creates thee TCP sockets and explicitly bound them one to each of ethX interfaces. However, due default gateway assigned to eth0 , all outgoing traffic is passing via interface eth0, regardless on what socket used to send it . QUESTION: How can I configure my routing in such way that outgoing traffic always exit the host by interface associated with transmitting socket ? ( I hear a little about source routing and ideas from http://lartc.org/howto/lartc.rpdb.html#LARTC.RPDB.SIMPLE are helping, but I don't like idea to run "ip rule add from" each time my IP changed). I'm looking to some simple way to instruct routing engine to forward packets via associated interfaces. Regards Stas P.S. Sorry for posting my question again - didn't received any responses yet ;-( From don-lartcxx at isis.cs3-inc.com Sat Jul 9 05:55:08 2005 From: don-lartcxx at isis.cs3-inc.com (Don Cohen) Date: Sat Jul 9 05:55:23 2005 Subject: [LARTC] RFC - bandwidth optimization idea In-Reply-To: <20050708192337.9724548F7@outpost.ds9a.nl> References: <20050708192337.9724548F7@outpost.ds9a.nl> Message-ID: <17103.19100.895504.367312@isis.cs3-inc.com> I'm interested in all of - opinions about why this is a good or bad idea - pointers to similar proposals or products that already exist - implementation suggestions This is meant for real time applications that have small available bandwidth and so they have to consider carefully what's the best way to use that bandwidth. I imagine that things happen that cause them to continually reevaluate what's the most important/urgent thing to send next. I want to make it possible for them to delay the choice until the OS is actually ready to send that next packet. The reason they can't do this now is that the OS enqueues packets. Suppose an application uses udp or tcp to tell the OS to send some data. It then discovers that data is obsolete. The old data might still be in the queue to be sent but it's too late to recall it. One way to avoid that is to always delay telling the OS to send something until the OS is almost ready to send the next packet from the queue that your data will enter. But that's not so easy to do, and there's a big penalty if you wait just a little too long. What I want, at least conceptually, is that the application maintains its own queue of data to be sent, ordered by priority. Whenever the OS is ready to send the next packet for that application, it removes the highest priority packet (if any) from the queue and sends it. From don-lartcxx at isis.cs3-inc.com Sat Jul 9 06:01:17 2005 From: don-lartcxx at isis.cs3-inc.com (Don Cohen) Date: Sat Jul 9 06:01:30 2005 Subject: [LARTC] TCP window based shaping In-Reply-To: <20050708192337.9724548F7@outpost.ds9a.nl> References: <20050708192337.9724548F7@outpost.ds9a.nl> Message-ID: <17103.19469.965087.345081@isis.cs3-inc.com> I recalled a discussion of manipulating outgoing tcp windows in order to control return tcp traffic. I finally found at least some of that discussion in a thread with the subject above (of this message). But I thought someone announced an implementation and I don't see it under this thread. If anyone else remembers or knows where I should look for it, please let me know. I'm also interested in other discussion of the idea and in implementations of related ideas such as delaying the acks etc. From Andreas.Klauer at metamorpher.de Sat Jul 9 07:38:01 2005 From: Andreas.Klauer at metamorpher.de (Andreas Klauer) Date: Sat Jul 9 07:38:06 2005 Subject: [LARTC] RFC - bandwidth optimization idea In-Reply-To: <17103.19100.895504.367312@isis.cs3-inc.com> References: <20050708192337.9724548F7@outpost.ds9a.nl> <17103.19100.895504.367312@isis.cs3-inc.com> Message-ID: <200507090738.01671.Andreas.Klauer@metamorpher.de> On Saturday 09 July 2005 05:55, Don Cohen wrote: > What I want, at least conceptually, is that the application maintains its > own queue of data to be sent, ordered by priority. Whenever the OS is > ready to send the next packet for that application, it removes the > highest priority packet (if any) from the queue and sends it. Doesn't every QDisc work that way? When the kernel wants to send a packet, it calls the appropriate dequeue() function in the QDisc. I'm not a kernel developer so this guess might be wrong. But still, I don't think that the queueing is the main problem with your idea... the main problem is, how do you decide what's important and what not, and what's obsolete? Andreas From Paul.Hampson at PObox.com Sat Jul 9 07:43:44 2005 From: Paul.Hampson at PObox.com (Paul Hampson) Date: Sat Jul 9 07:43:51 2005 Subject: [LARTC] RFC - bandwidth optimization idea In-Reply-To: <17103.19100.895504.367312@isis.cs3-inc.com> References: <20050708192337.9724548F7@outpost.ds9a.nl> <17103.19100.895504.367312@isis.cs3-inc.com> Message-ID: <20050709054344.GB3974@yurika.videohost.com.au> On Fri, Jul 08, 2005 at 08:55:08PM -0700, Don Cohen wrote: > > I'm interested in all of > - opinions about why this is a good or bad idea > - pointers to similar proposals or products that already exist > - implementation suggestions > This is meant for real time applications that have small available > bandwidth and so they have to consider carefully what's the best way > to use that bandwidth. I imagine that things happen that cause them > to continually reevaluate what's the most important/urgent thing to > send next. I want to make it possible for them to delay the choice > until the OS is actually ready to send that next packet. The reason > they can't do this now is that the OS enqueues packets. Suppose an > application uses udp or tcp to tell the OS to send some data. It then > discovers that data is obsolete. The old data might still be in the > queue to be sent but it's too late to recall it. One way to avoid > that is to always delay telling the OS to send something until the OS > is almost ready to send the next packet from the queue that your data > will enter. But that's not so easy to do, and there's a big penalty > if you wait just a little too long. What I want, at least > conceptually, is that the application maintains its own queue of data > to be sent, ordered by priority. Whenever the OS is ready to send the > next packet for that application, it removes the highest priority > packet (if any) from the queue and sends it. I believe the general solution to this is to use UDP, and make sure your source machine doesn't queue up packets locally (eg. ethernet network contention) and let the best-effort nature of UDP deal with dropping stuff that gets delayed. I'm not sure there's any way to have an 'I changed my mind about sending that' interface into your network stack... And generally it wouldn't be useful, data spends longer in transit than it does in your queues. -- Paul "TBBle" Hampson, on an alternate email client. From don-lartcxx at isis.cs3-inc.com Sat Jul 9 17:25:39 2005 From: don-lartcxx at isis.cs3-inc.com (Don Cohen) Date: Sat Jul 9 17:25:48 2005 Subject: [LARTC] Re: RFC - bandwidth optimization idea In-Reply-To: <20050709100005.950B7495F@outpost.ds9a.nl> References: <20050709100005.950B7495F@outpost.ds9a.nl> Message-ID: <17103.60531.516880.785001@isis.cs3-inc.com> > From: Andreas Klauer > Doesn't every QDisc work that way? When the kernel wants to send a packet, > it calls the appropriate dequeue() function in the QDisc. I'm not a kernel > developer so this guess might be wrong. That's correct, but this operation takes a packet from an OS queue and the only control the application has over that queue is to put something into it. One way to view the idea is that I want to make it convenient for the application to decide what to put into the queue at the latest possible time without losing any of its available bandwidth. Think in terms of an OS callback to the application saying "I'm ready to send your data now, what should I send?" > But still, I don't think that the queueing is the main problem with your > idea... the main problem is, how do you decide what's important and what > not, and what's obsolete? This is up to the application of course. See below. > From: Paul.Hampson@PObox.com (Paul Hampson) > I believe the general solution to this is to use UDP, and make sure The scheme I describe wouldn't make a lot of sense for tcp, which after all specifies congestion control, retransmission, etc. But UDP still goes through the queuing that I want to optimize. > your source machine doesn't queue up packets locally (eg. ethernet > network contention) and let the best-effort nature of UDP deal with > dropping stuff that gets delayed. The problem is that the OS is not helpful in avoiding queuing up packets locally. That's part of what I'm trying to fix. For instance, a relatively cheap approximation would be to give the application a way to see how many packets it has in the queue. Then it could at least delay its decision about what to put into the queue until the queue was short. Even better would be to see an estimate of how long it will be before the next packet it enqueues will be sent - like "your call will be answered in approximately 4 minutes". > I'm not sure there's any way to have an 'I changed my mind about > sending that' interface into your network stack... And generally > it wouldn't be useful, data spends longer in transit than it does > in your queues. That depends on the rate at which the queue is emptied. If your queue has a rate limit of 10bps then your packets can spend a long time in the queue. - There are slow links (For instance, I recall hearing that submarines have very low rates.) - The application might be allocated a small part of the bandwidth shared with other applications. It occurs to me that an example where this would be helpful is transmitting voice data over a low bandwidth link (like a cell phone). Suppose you know that the actual transit time is .1 sec and you want the listener to always hear what the speaker was saying .2 sec ago at the best possible quality. Suppose the available bandwidth is shared with other applications. The voice application doesn't know when they will want to send or how urgent their data might be. Someone else decides that. It just wants to send the best possible data in the bandwidth allocated to it. I imagine is continually sampling the input and revising what it considers to be the most valuable unsent data for the last .1 sec. Whenever the OS decides it's time to send the next voice packet I want it to send the latest idea of what's most valuable. I don't want to have to put data into the queue to wait for times that might depend on what urgent communication might be required by other applications. From gypsy at iswest.com Sat Jul 9 17:58:05 2005 From: gypsy at iswest.com (gypsy) Date: Sat Jul 9 17:57:50 2005 Subject: [LARTC] Question about TCNG References: <200507082209.13178.donvodka@gmail.com> Message-ID: <42CFF40D.3CAE3F08@iswest.com> Edgar wrote: > > Hello, after compiling my tcng rules, I've noticed something weird, the tc > filter rules it returns give a flowid differente than the classes it creates, > for example, you have a class 2:3 and had defined some conditions, (eg. some > ports) to go into that class, but when I compile the .tcc file it returns > something like: > tc filter add dev eth1 parent 1:0 protocol all prio 1 u32 match u8 0x5 0xf at > 0 match u8 0x10 0x10 at 33 classid 1:2 > > it gives the same classid for all the matches I made (meaning for different > classes) and that class doesn't even exist! can someone explain me this > please? thank you > > EDGAR MERINO Rather than ask the list, why not ask the developer(s)? Better yet, switch to tc. -- gypsy From gypsy at iswest.com Sat Jul 9 18:17:08 2005 From: gypsy at iswest.com (gypsy) Date: Sat Jul 9 18:16:42 2005 Subject: [LARTC] P2P shaping working References: <200507081926.56750.donvodka@gmail.com> Message-ID: <42CFF884.B71A2D24@iswest.com> Edgar wrote: > > Hello, it's me again, finally p2p traffic is getting shaped, but now I still > have one problem: download seems to be affected when shaping the traffic, ack > packets are in the interactive queue too, as someone adviced me, what fixed > the problem was to change the rates, to match (the sum of all subclasses) the > root class rate, and to give the p2p class a very low rate (1k actually) and > 75% of the bandwidth for the ceiling. Well as I said this is working, but > downloading seems to be affected to, I get pretty low rates (this doesn't > happen when I'm not shaping the traffic, but then I can't do anything else > but bittorrent). Does anyone have a clue on why's this happening? > > Thanks to everyone that help me, I really appreciate it. > > EDGAR MERINO Guess 1: You are not giving the ACK to the downloaded packet priority so it gets delayed. Guess 2: You are dropping incoming packets, causing resends. Guess 3: The outbound queue(s) are full so the ACK can't be sent "NOW, DAMNIT"! Guess 4: You have not tweaked /etc/sysctl.conf and run sysctl, so the internal buffers are too small. Guess 5: Your CPU is overloaded by the queuing. Guess 6: Your IMQ is misconfigured. AFAIC, ACK and window size are the most important issues. You must prioritize the ACKs not associated with file sharing and delay the hell out of those that are associated with file sharing. -- gypsy From Paul.Hampson at PObox.com Sat Jul 9 19:23:41 2005 From: Paul.Hampson at PObox.com (Paul Hampson) Date: Sat Jul 9 19:23:50 2005 Subject: [LARTC] Re: RFC - bandwidth optimization idea In-Reply-To: <17103.60531.516880.785001@isis.cs3-inc.com> References: <20050709100005.950B7495F@outpost.ds9a.nl> <17103.60531.516880.785001@isis.cs3-inc.com> Message-ID: <20050709172341.GA9529@yurika.videohost.com.au> On Sat, Jul 09, 2005 at 08:25:39AM -0700, Don Cohen wrote: > > From: Paul.Hampson@PObox.com (Paul Hampson) > > I believe the general solution to this is to use UDP, and make sure > The scheme I describe wouldn't make a lot of sense for tcp, which > after all specifies congestion control, retransmission, etc. > But UDP still goes through the queuing that I want to optimize. > > your source machine doesn't queue up packets locally (eg. ethernet > > network contention) and let the best-effort nature of UDP deal with > > dropping stuff that gets delayed. > The problem is that the OS is not helpful in avoiding queuing up > packets locally. That's part of what I'm trying to fix. > For instance, a relatively cheap approximation would be to give > the application a way to see how many packets it has in the queue. > Then it could at least delay its decision about what to put into > the queue until the queue was short. Even better would be to > see an estimate of how long it will be before the next packet it > enqueues will be sent - like "your call will be answered in > approximately 4 minutes". > > I'm not sure there's any way to have an 'I changed my mind about > > sending that' interface into your network stack... And generally > > it wouldn't be useful, data spends longer in transit than it does > > in your queues. > That depends on the rate at which the queue is emptied. > If your queue has a rate limit of 10bps then your packets can spend > a long time in the queue. > - There are slow links > (For instance, I recall hearing that submarines have very low rates.) > - The application might be allocated a small part of the bandwidth > shared with other applications. Wait, you're trying to send more data than the link can take? Then send UDP, throttle it at the local end with a drop-oldest qdisc. Then you get the effect of 'most recent data is best'. Anything more compilcated in terms of priority either needs a custom qdisc, or your application needs to not try and send more than the link can take. > It occurs to me that an example where this would be helpful is > transmitting voice data over a low bandwidth link (like a cell phone). > Suppose you know that the actual transit time is .1 sec and you want > the listener to always hear what the speaker was saying .2 sec ago at > the best possible quality. > Suppose the available bandwidth is shared with other applications. > The voice application doesn't know when they will want to send or how > urgent their data might be. Someone else decides that. It just wants > to send the best possible data in the bandwidth allocated to it. I > imagine is continually sampling the input and revising what it > considers to be the most valuable unsent data for the last .1 sec. > Whenever the OS decides it's time to send the next voice packet I want > it to send the latest idea of what's most valuable. I don't want to > have to put data into the queue to wait for times that might depend on > what urgent communication might be required by other applications. You gotta prioritise your data, using TOS or diffserv or something. Set your voice to real-time, so it always gets sent, and the your other applications can use unused packet-times. Use a dropping qdisc for traffic where 'most-recent' is more important than 'all, in order' as described above, and you're set. I have a vauge recollection that this sort of thing is discussed in Tannenbaum's Computer Networks textbook, to do with positional data of satellites or something. (eg. if the positional data is delayed, we write it off, we don't want to delay the data about where we are _now_ in order to know where we were _then_) -- Paul "TBBle" Hampson, on an alternate email client. From jma at agile.dk Sat Jul 9 21:29:58 2005 From: jma at agile.dk (Jan Andersen) Date: Sat Jul 9 21:31:20 2005 Subject: [LARTC] Problems with iproute2 Message-ID: <20050709193002.BF59BCC4@agile.dk> I'm using Debian (sarge) I do now have 2 ISP, one DSL and one cabel. I want to use the DSL as my primary and only use the cabel-modem if the DSL fails. The setup is: IP1 -> Router -> NAT_IP1 -> eth0 -> eth2 (LAN) IP2 -> Cable-Modem -> eth1 -> eth2 (LAN) I wrote the following in my /etc/iproute2/rt_tables ip rule add from NAT_IP1 lookup 1 ip route add 10.1.0.0/16 via 10.1.1.1 table 1 ip route add 0/0 via NAT_IP1 table 1 ip rule add from IP2 lookup 2 ip route add 10.1.0.0/16 via 10.1.1.1 table 2 ip route add 0/0 via IP2 table 2 The problem is identical to http://www.samag.com/documents/s=1824/sam0201h/ But if I write "ip rule list" I get the following error: Database /etc/iproute2/rt_tables is corrupted at ip rule add from 192.168.100.254 lookup 1 Does anyone know and have a solution to this problem? Regards Jan Andersen From gypsy at iswest.com Sun Jul 10 02:35:05 2005 From: gypsy at iswest.com (gypsy) Date: Sun Jul 10 02:34:43 2005 Subject: [LARTC] Problems with iproute2 References: <20050709193002.BF59BCC4@agile.dk> Message-ID: <42D06D39.2D18BB91@iswest.com> Jan Andersen wrote: > > I'm using Debian (sarge) > > I do now have 2 ISP, one DSL and one cabel. > > I want to use the DSL as my primary and only use the cabel-modem if the DSL > fails. > > The setup is: > > IP1 -> Router -> NAT_IP1 -> eth0 -> eth2 (LAN) > IP2 -> Cable-Modem -> eth1 -> eth2 (LAN) > > I wrote the following in my /etc/iproute2/rt_tables > > ip rule add from NAT_IP1 lookup 1 > ip route add 10.1.0.0/16 via 10.1.1.1 table 1 > ip route add 0/0 via NAT_IP1 table 1 > > ip rule add from IP2 lookup 2 > ip route add 10.1.0.0/16 via 10.1.1.1 table 2 > ip route add 0/0 via IP2 table 2 > > The problem is identical to http://www.samag.com/documents/s=1824/sam0201h/ > > But if I write "ip rule list" I get the following error: > > Database /etc/iproute2/rt_tables is corrupted at ip rule add from > 192.168.100.254 lookup 1 > > Does anyone know and have a solution to this problem? > > Regards > > Jan Andersen No, I don't have a solution. What is in your /etc/iproute2/rt_tables file? Normally, everything should be commented except 255 local 254 main 253 default 0 unspec Are you sure 'route add' should work when both routes are identical? Where does the 192.168.100.254 come from? Why allow NAT of the internet IPs? OPINION 1: One NAT is plenty, two is begging trouble. OPINION 2: Most of the problems reported here have "example.com" ip addresses, so avoiding NAT is A Good Thing. -- gypsy From lists at wildgooses.com Sun Jul 10 09:49:13 2005 From: lists at wildgooses.com (Ed W) Date: Sun Jul 10 09:49:21 2005 Subject: [LARTC] Re: RFC - bandwidth optimization idea In-Reply-To: <20050709172341.GA9529@yurika.videohost.com.au> References: <20050709100005.950B7495F@outpost.ds9a.nl> <17103.60531.516880.785001@isis.cs3-inc.com> <20050709172341.GA9529@yurika.videohost.com.au> Message-ID: <42D0D2F9.9080603@wildgooses.com> >Wait, you're trying to send more data than the link can take? Then >send UDP, throttle it at the local end with a drop-oldest qdisc. Then >you get the effect of 'most recent data is best'. Anything more >compilcated in terms of priority either needs a custom qdisc, or your >application needs to not try and send more than the link can take. > > The situation described is real and complex. For example I run an email service which caters for people using satellite phones (1,200 baud on a good day), but the whole point is that they don't need to change any settings when they jump on a 10Mbit leased line connection... This is a total pain to optimise. Ideally I would like an API to be able to limit the congestion window on the local machine for a particular connection (which I don't think exists on either windows or linux?). This way the OS will report that the queue is full quickly to the local program without buffering up a ton of data. The issue in my case is that you have two simultaneous streams in transit for email, one to receive new mail and one to send mail out. In the case of the sat phone it's possible to have net buffers which are 20 secs or so long and so when you send out a status message to say "email received successfully, send me the next one", it can end up queued behind a bunch of lower priority data for a VERY long time. Often these buffers are on the remote ISP end where you have very little control. This is a serious slowdown on a link which is costing you $1.50/min. My main focus has been adjusting the protocol to be less interactive, but it would be nice to have more operating system support for these fringe cases Ed W From sspies at sloc.de Sun Jul 10 11:19:33 2005 From: sspies at sloc.de (Sebastian Spies) Date: Sun Jul 10 11:21:10 2005 Subject: [LARTC] Default source address Message-ID: <42D0E825.9000403@sloc.de> Hi, I'm using vpnc to connect to a vpn. The VPN-GW wants me to use him as a default route. So I used source-routing to differ between my physical ethernet and the tunnel: ip rule add from w.x.y.z table 20 ip route add default via tun0 table 20 Everything works fine, as I have a client to set my inet4-srcaddress. My question is: Is there a way, to tell linux which is my default sourceaddress? Greets, Sebastian From Paul.Hampson at PObox.com Sun Jul 10 12:12:48 2005 From: Paul.Hampson at PObox.com (Paul Hampson) Date: Sun Jul 10 12:12:56 2005 Subject: [LARTC] Re: RFC - bandwidth optimization idea In-Reply-To: <42D0D2F9.9080603@wildgooses.com> References: <20050709100005.950B7495F@outpost.ds9a.nl> <17103.60531.516880.785001@isis.cs3-inc.com> <20050709172341.GA9529@yurika.videohost.com.au> <42D0D2F9.9080603@wildgooses.com> Message-ID: <20050710101248.GA18859@yurika.videohost.com.au> On Sun, Jul 10, 2005 at 08:49:13AM +0100, Ed W wrote: > >Wait, you're trying to send more data than the link can take? Then > >send UDP, throttle it at the local end with a drop-oldest qdisc. Then > >you get the effect of 'most recent data is best'. Anything more > >compilcated in terms of priority either needs a custom qdisc, or your > >application needs to not try and send more than the link can take. > The situation described is real and complex. For example I run an email > service which caters for people using satellite phones (1,200 baud on a > good day), but the whole point is that they don't need to change any > settings when they jump on a 10Mbit leased line connection... Ah, I was picturing voice over a low-latency, low-speed link. Now I can understand what you're trying to acheieve. Is that 1200 baud each way? Or do you have to alternate up and down somehow? > This is a total pain to optimise. Ideally I would like an API to be > able to limit the congestion window on the local machine for a > particular connection (which I don't think exists on either windows or > linux?). This way the OS will report that the queue is full quickly to > the local program without buffering up a ton of data. Indeed. For TCP, you could use setsockopt with SO_SNDBUF maybe? However, I'm not sure this is what you want. > The issue in my case is that you have two simultaneous streams in > transit for email, one to receive new mail and one to send mail out. In > the case of the sat phone it's possible to have net buffers which are 20 > secs or so long and so when you send out a status message to say "email > received successfully, send me the next one", it can end up queued > behind a bunch of lower priority data for a VERY long time. Often these > buffers are on the remote ISP end where you have very little control. > This is a serious slowdown on a link which is costing you $1.50/min. Assuming you can send both ways simultaneously, or at least guarantee some traffic time outbound no matter how busy the inbound traffic, you would surely have to pipeline your commands in order to get any kind of efficient use out of a high-latency link like a satellite link. Otherwise you lose 2x round-trip-time of incoming data stream while you request the next item. In this situation, you would want the OS buffers to be as full as possible so the minimal time possible is spent receiving, but using a traffic-shaping solution so that the most important stuff (acks and commands) goes out in preference to whatever else you're sending. eg. If you're doing POP3 and SMTP, you make sure any to-tcp-110 or tcp-ack-only packet is dequeued before any from-tcp-25 packets. You'd also need to jack the receive window right up, or wait for TCP to figure that out for itself. > My main focus has been adjusting the protocol to be less interactive, > but it would be nice to have more operating system support for these > fringe cases This is actually a common case, and often cited as a great big hole in TCP/IP's traffic algorithms. I know, it was a question on the exam. ^_^ -- Paul "TBBle" Hampson, on an alternate email client. From Paul.Hampson at PObox.com Sun Jul 10 12:20:26 2005 From: Paul.Hampson at PObox.com (Paul Hampson) Date: Sun Jul 10 12:20:38 2005 Subject: [LARTC] Default source address In-Reply-To: <42D0E825.9000403@sloc.de> References: <42D0E825.9000403@sloc.de> Message-ID: <20050710102026.GB18859@yurika.videohost.com.au> On Sun, Jul 10, 2005 at 11:19:33AM +0200, Sebastian Spies wrote: > Hi, > I'm using vpnc to connect to a vpn. > The VPN-GW wants me to use him as a default route. So I used > source-routing to differ between my physical ethernet and the tunnel: > ip rule add from w.x.y.z table 20 > ip route add default via tun0 table 20 > Everything works fine, as I have a client to set my inet4-srcaddress. > My question is: Is there a way, to tell linux which is my default > sourceaddress? It should use the primary address of the interface the routing table picks to send the data out of. If you've managed to get multiple primary addresses, it picks the lowest one, numerically, I believe. Otherwise you have to bind your side of the socket to an address, and it should use that. -- Paul "TBBle" Hampson, on an alternate email client. From andy.furniss at dsl.pipex.com Sun Jul 10 12:29:36 2005 From: andy.furniss at dsl.pipex.com (Andy Furniss) Date: Sun Jul 10 12:28:49 2005 Subject: [LARTC] Re: RFC - bandwidth optimization idea In-Reply-To: <42D0D2F9.9080603@wildgooses.com> References: <20050709100005.950B7495F@outpost.ds9a.nl> <17103.60531.516880.785001@isis.cs3-inc.com> <20050709172341.GA9529@yurika.videohost.com.au> <42D0D2F9.9080603@wildgooses.com> Message-ID: <42D0F890.8080101@dsl.pipex.com> Ed W wrote: > > This is a total pain to optimise. Ideally I would like an API to be > able to limit the congestion window on the local machine for a > particular connection (which I don't think exists on either windows or > linux?). It looks like you could do it per route in the past - don't know about now. http://www.linux-ip.net/gl/ip-cref/node77.html Andy. From alex at qb.ro Sun Jul 10 13:36:01 2005 From: alex at qb.ro (alex@qb.ro) Date: Sun Jul 10 13:36:08 2005 Subject: [LARTC] is the lartc documentation right about filters on HTB Message-ID: <200507101136.j6ABa1K12399@qb.ro> Hi, Just a short question: in documentation (http://lartc.org/howto/lartc.qdisc.filters.html) it appears that "Also, with HTB, you should attach all filters to the root!" . Is it older information or you can not attach filter to the inner qdiscs at all? Thank you, alex From andy.furniss at dsl.pipex.com Sun Jul 10 21:13:23 2005 From: andy.furniss at dsl.pipex.com (Andy Furniss) Date: Sun Jul 10 21:12:29 2005 Subject: [LARTC] is the lartc documentation right about filters on HTB In-Reply-To: <200507101136.j6ABa1K12399@qb.ro> References: <200507101136.j6ABa1K12399@qb.ro> Message-ID: <42D17353.6010707@dsl.pipex.com> alex@qb.ro wrote: > Hi, > Just a short question: > in documentation (http://lartc.org/howto/lartc.qdisc.filters.html) it appears that "Also, with HTB, you should attach all filters to the root!" . Is it older information or you can not attach filter to the inner qdiscs at all? Yes you can have filters on the inner classes. Andy. From andy.furniss at dsl.pipex.com Sun Jul 10 21:23:10 2005 From: andy.furniss at dsl.pipex.com (Andy Furniss) Date: Sun Jul 10 21:22:15 2005 Subject: [LARTC] P2P shaping working In-Reply-To: <200507081926.56750.donvodka@gmail.com> References: <200507081926.56750.donvodka@gmail.com> Message-ID: <42D1759E.3060402@dsl.pipex.com> Edgar wrote: > Hello, it's me again, finally p2p traffic is getting shaped, but now I still > have one problem: download seems to be affected when shaping the traffic, ack > packets are in the interactive queue too, as someone adviced me, what fixed > the problem was to change the rates, to match (the sum of all subclasses) the > root class rate, and to give the p2p class a very low rate (1k actually) and > 75% of the bandwidth for the ceiling. Well as I said this is working, but > downloading seems to be affected to, I get pretty low rates (this doesn't > happen when I'm not shaping the traffic, but then I can't do anything else > but bittorrent). Does anyone have a clue on why's this happening? > > Thanks to everyone that help me, I really appreciate it. Shaping upload will affect bittorrent downloads as it it part of the protocol for peers to favour the connections that are giving them the highest rates. That may not be the only thing though, if you are marking with ipp2p then the acks for the download will be queued with the upload traffic - this happens with the ones that get piggybacked anyway, but I let empty bt acks have priority over those. It would help if you make the bt queue fairly short. Andy. From andy.furniss at dsl.pipex.com Sun Jul 10 21:37:39 2005 From: andy.furniss at dsl.pipex.com (Andy Furniss) Date: Sun Jul 10 21:36:43 2005 Subject: [LARTC] HFSC default upper-limit trouble In-Reply-To: <42CC2D0E.5010903@kitas.arturas.net> References: <42CC2D0E.5010903@kitas.arturas.net> Message-ID: <42D17903.9010605@dsl.pipex.com> Art?ras ?lajus wrote: > Hello, > > I'm having such problem with HFSC with following config: > + tc qdisc del dev eth3 root > + tc qdisc add dev eth3 root handle 1: hfsc default 2 > + tc class add dev eth3 parent 1: classid 1:1 hfsc ls rate 512kbit ul > rate 512kbit > + tc class add dev eth3 parent 1:1 classid 1:2 hfsc ls rate 2kbit ul > rate 400kbit > + tc class add dev eth3 parent 1:1 classid 1:3 hfsc ls rate 32kbit ul > rate 32kbit > + tc class add dev eth3 parent 1:1 classid 1:4 hfsc ls rate 300kbit ul > rate 300kbit > > Let's say i start to upload thru 1:3. the upper-limit applies, traffic > doesn't do up more than 4kb/s. The 1:4 is still functional, but 1:2, the > default class starts backlogging and dropping as hell: Seems to work OK for me - are you saying that there is traffic using default but no backlog until you use 1:3? > class hfsc 1: root > Sent 0 bytes 0 pkts (dropped 0, overlimits 0) > period 0 level 2 > > class hfsc 1:1 parent 1: ls m1 0bit d 0us m2 512000bit ul m1 0bit d 0us > m2 512000bit > Sent 0 bytes 0 pkts (dropped 0, overlimits 0) <-- This is weird too ^_^ > period 2643 work 821712 bytes level 1 You need to do a tc -s qdisc ls to get the overlimits counter for 1:0. > > class hfsc 1:2 parent 1:1 ls m1 0bit d 0us m2 2000bit ul m1 0bit d 0us > m2 400000bit > Sent 477205 bytes 3874 pkts (dropped 0, overlimits 0) > backlog 201p <-- HUH? (it goes even to 800p..1000p then it starts > dropping) > period 2494 work 456595 bytes level 0 Remember arp will end up stuck here too unless you filter it elsewhere. If you want to make the queue shorter add a p/b fifo or something and specify limit. > > class hfsc 1:3 parent 1:1 ls m1 0bit d 0us m2 32000bit ul m1 0bit d 0us > m2 32000bit > Sent 350599 bytes 558 pkts (dropped 0, overlimits 0) > backlog 11p > period 70 work 342761 bytes level 0 > > class hfsc 1:4 parent 1:1 ls m1 0bit d 0us m2 300000bit ul m1 0bit d 0us > m2 300000bit > Sent 22356 bytes 214 pkts (dropped 0, overlimits 0) > period 212 work 22356 bytes level 0 > > The 1:1 shows no packets sent as you see.. Is this desirable behavior? > The default class kinda becomes unusable. Can someone explain me such > behavior? This is normal Patrick said in a recent post - "HFSC doesn't update statistics of upper classes because I didn't want to walk up the entire heirarchy for each packet just for statistics." Andy. From andy.furniss at dsl.pipex.com Sun Jul 10 21:55:18 2005 From: andy.furniss at dsl.pipex.com (Andy Furniss) Date: Sun Jul 10 21:54:25 2005 Subject: [LARTC] TCP window based shaping In-Reply-To: <17103.19469.965087.345081@isis.cs3-inc.com> References: <20050708192337.9724548F7@outpost.ds9a.nl> <17103.19469.965087.345081@isis.cs3-inc.com> Message-ID: <42D17D26.1040104@dsl.pipex.com> Don Cohen wrote: > I recalled a discussion of manipulating outgoing tcp windows in order > to control return tcp traffic. I finally found at least some of that > discussion in a thread with the subject above (of this message). > But I thought someone announced an implementation and I don't see it > under this thread. If anyone else remembers or knows where I should > look for it, please let me know. I'm also interested in other > discussion of the idea and in implementations of related ideas such as > delaying the acks etc. I don't think there is an implementation other than the commercial packeteer. Andy. From tdi at pozman.pl Sun Jul 10 22:13:24 2005 From: tdi at pozman.pl (Dariusz Dwornikowski) Date: Sun Jul 10 22:13:35 2005 Subject: [LARTC] QOS HELP PLEASE Message-ID: <20050710221324.1311e95e@tdi.pozman.pl> ive got problems with my network (120 people) ive got big pings (300ms)m whereas there are normally about 19ms. i do not know if my qos is proper (fast i mean). www.tdi.pozman.pl/fir2 - my qos www.tdi.pozman.pl/rules - my firewall can sb tell me if do it ok ? -- *Dariusz 'tdi' Dwornikowski | Gentoo | admin at pozman.pl | *[JID]:tdi@gentoo.pl|[gg]:2266034|[IRC]:#gentoo-pl@freenode | *[MAIL]:tdi@pozman.pl|[WWW]:www.tdi.pozman.pl | *Serwery,administracja,webapps - www.ProAdmin.com.pl | *Fingerprint:43E21CC46DAFD2F754E91547D59B39F56AAA4B5F | -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mailman.ds9a.nl/pipermail/lartc/attachments/20050710/0733b9c4/attachment.bin From andy.furniss at dsl.pipex.com Sun Jul 10 22:23:06 2005 From: andy.furniss at dsl.pipex.com (Andy Furniss) Date: Sun Jul 10 22:22:18 2005 Subject: [LARTC] Problem with HTB and IPTABLES In-Reply-To: References: Message-ID: <42D183AA.2070401@dsl.pipex.com> leandro wrote: > > I?ve checked using ethereal, and all the packets? TOS field are been > correctly changed, and the packets are been sent to the right mpls tunnel > with the right label, proving that the iptables are correctly marking the > packets. Total guess but try using protocol all rather than protocol ip in the filter rules. Andy. From mabrown-lartc at securepipe.com Sun Jul 10 22:49:58 2005 From: mabrown-lartc at securepipe.com (Martin A. Brown) Date: Sun Jul 10 22:51:05 2005 Subject: [LARTC] QOS HELP PLEASE In-Reply-To: <20050710221324.1311e95e@tdi.pozman.pl> References: <20050710221324.1311e95e@tdi.pozman.pl> Message-ID: Greetings Dariusz, : ive got problems with my network (120 people) : ive got big pings (300ms)m whereas there are normally about 19ms. : i do not know if my qos is proper (fast i mean). : : www.tdi.pozman.pl/fir2 - my qos : www.tdi.pozman.pl/rules - my firewall After examining 'fir2', which shows an HTB class structure listed below, I think you don't quite understand the guarantees and the borrowing model of HTB. your Internet bound traffic (1:2) -- - - - --> rate ceil | +--------+--------+--------+--------+-------+ 1:N | 1:7 | 1:6 | 1:5 | 1:4 | | [ lots of | | | +--> 128kbit 256kbit | classes | | +-- - - - -> 128kbit 256kbit | here ] | +- - - - - - - - - -> 128kbit 256kbit | +-- - - - - - - - - - - - - -> 128kbit 256kbit | ... ... +-- - - - - - - - - - - - - - - - - - - - - - -> 128kbit 256kbit In your case, N=163 (although I didn't check that every class was created with the same rate/bandwidth). The problem you are having is that the borrowing (and hence, shaping) model never gets a chance to go into effect. Every leaf class (1:4 through 1:166) is guaranteed 128kbit. Your QoS setup is actually not helping you at all! It's configured to guarantee around 19mbit (128kbit * 163 guarantees). Given your available Internet bandwidth, it should work out a bit better for you if you slim down the total number of classes and lump a few handfuls of users in each class with an embedded SFQ. You may find that Stef's rules for HTB shaping are quite handy [0], and also my HTB description [1]. Good luck, -Martin [0] http://www.docum.org/docum.org/faq/cache/10.html [1] http://tldp.org/HOWTO/Traffic-Control-HOWTO/classful-qdiscs.html#qc-htb -- Martin A. Brown --- SecurePipe, Inc. --- mabrown@securepipe.com From tdi at pozman.pl Mon Jul 11 06:42:05 2005 From: tdi at pozman.pl (Dariusz Dwornikowski) Date: Mon Jul 11 06:44:08 2005 Subject: [LARTC] QOS HELP PLEASE In-Reply-To: References: <20050710221324.1311e95e@tdi.pozman.pl> Message-ID: <20050711064205.2ffe0ea4@tdi.pozman.pl> so the sun of all rates of speeds of classes for the clients should be less than the rate of the class 1:2 ? or i understand it badly ? -- *Dariusz 'tdi' Dwornikowski | Gentoo | admin at pozman.pl | *[JID]:tdi@gentoo.pl|[gg]:2266034|[IRC]:#gentoo-pl@freenode | *[MAIL]:tdi@pozman.pl|[WWW]:www.tdi.pozman.pl | *Serwery,administracja,webapps - www.ProAdmin.com.pl | *Fingerprint:43E21CC46DAFD2F754E91547D59B39F56AAA4B5F | -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mailman.ds9a.nl/pipermail/lartc/attachments/20050711/0e5bafce/attachment.bin From zybort at telefonica.net Mon Jul 11 11:06:42 2005 From: zybort at telefonica.net (chino) Date: Mon Jul 11 11:03:13 2005 Subject: [LARTC] wonder-shaper Message-ID: <200507111106.42160.zybort@telefonica.net> Hi all, I need to provide shell hosting for about 40 users, and few days ago I found the wondershaper script, so Im trying to know if this script could help me to improve interactive ssh/telnet connections. Wondershaper could help me or its only efective on OUTGOINGS requests? Ok, any comment and feedback will be welcome. Sorry for my bad english and THANKS in advance. From admin at premiere-heure.fr Mon Jul 11 13:27:10 2005 From: admin at premiere-heure.fr (Gael Mauleon) Date: Mon Jul 11 13:26:38 2005 Subject: [LARTC] HTB Rate and Prio (continued) Message-ID: <200507111126.j6BBQT5t007024@dhuumrelay0.mail.eu.uu.net> Hi again, I keep posting about my problem with HTB -> http://mailman.ds9a.nl/pipermail/lartc/2005q3/016611.html With a bit of search I recently found the exact same problem I have in the 2004 archives with some graphs that explain it far better than I did -> http://mailman.ds9a.nl/pipermail/lartc/2004q4/014519.html and http://mailman.ds9a.nl/pipermail/lartc/2004q4/014568.html Unluckily there were no solution, well or I didn't find it in the archives, so if anyone have a clue. I upgraded my box to the 2.6.12.2 kernel with the last iproute 2 but nothing change, I still have my shaping problem. Thanks. Gael. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ds9a.nl/pipermail/lartc/attachments/20050711/e0b8abe2/attachment.htm From mabrown-lartc at securepipe.com Mon Jul 11 16:52:45 2005 From: mabrown-lartc at securepipe.com (Martin A. Brown) Date: Mon Jul 11 16:52:33 2005 Subject: [LARTC] QOS HELP PLEASE In-Reply-To: <20050711064205.2ffe0ea4@tdi.pozman.pl> References: <20050710221324.1311e95e@tdi.pozman.pl> <20050711064205.2ffe0ea4@tdi.pozman.pl> Message-ID: Dariusz, : so the sum of all rates of speeds of classes for the clients should be : less than the rate of the class 1:2 ? or i understand it badly ? Indeed, you understand correctly. Your client classes are leaf classes. - An HTB leaf class guarantees access. - Above , the leaf class will borrow (from parents) up to . This bears repetition: the guaranteed total of bandwidth, before HTB shaping and borrowing begins, is the sum of the rates of the leaf classes. - If you want to make sure that the borrowing and shaping works correctly, be certain to configure HTB so that the leaf (and child) classes can never send more traffic than the parent has in . - For best results, configure HTB so that the leaf (and child) classes can never send more traffic than the parent has in . Good luck, -Martin -- Martin A. Brown --- SecurePipe, Inc. --- mabrown@securepipe.com From gypsy at iswest.com Mon Jul 11 17:20:00 2005 From: gypsy at iswest.com (gypsy) Date: Mon Jul 11 17:19:35 2005 Subject: [LARTC] wonder-shaper References: <200507111106.42160.zybort@telefonica.net> Message-ID: <42D28E20.641083E3@iswest.com> chino wrote: > > Hi all, > I need to provide shell hosting for about 40 users, and few > days ago I found the wondershaper script, so Im trying to > know if this script could help me to improve interactive ssh/telnet > connections. > > Wondershaper could help me or its only efective on OUTGOINGS requests? > > Ok, any comment and feedback will be welcome. > Sorry for my bad english and THANKS in advance. What bad English? Wondershaper does not work all that well because the sums of the rates and ceilings exceeds the root rate. You would do better to look at routehat (Which I think I spelled wrong), which uses WRR (Weighted Round Robin). Wondershaper is an excellent learning tool. By shaping OUTGOING, you can improve incoming, but Wondershaper drops the incoming in excess of the given rate so it is self defeating for improving download speed. If I remember right, it also incorrectly handles ACK packets. Read these: http://digriz.org.uk/ http://mrtg.saintjoe.edu/mrtg/ratelimit/pacemaker/ http://www.freenet.org.nz/python/pyshaper/ http://www.shurdix.org/ You can also have a look at: http://yesican.chsoft.biz/lartc -- gypsy From imipak at yahoo.com Mon Jul 11 18:39:06 2005 From: imipak at yahoo.com (Jonathan Day) Date: Mon Jul 11 18:39:14 2005 Subject: [LARTC] QOS HELP PLEASE In-Reply-To: Message-ID: <20050711163907.8747.qmail@web31515.mail.mud.yahoo.com> --- "Martin A. Brown" wrote: > > Dariusz, > > : so the sum of all rates of speeds of classes for > the clients should be > : less than the rate of the class 1:2 ? or i > understand it badly ? > > Indeed, you understand correctly. Your client > classes are leaf classes. > > - An HTB leaf class guarantees access. > - Above , the leaf class will borrow (from > parents) up to . > > This bears repetition: the guaranteed total of > bandwidth, before HTB > shaping and borrowing begins, is the sum of the > rates of the leaf classes. (snip) Can it ever be truly equal? There is going to be some overhead in having the multiple layers, so although the sum of the rates at level N can never exceed the rate of the parent layer at N-1, the penalties must mean that it must be marginally less (even if this is so marginal as to be hard to detect). __________________________________ Yahoo! Mail Stay connected, organized, and protected. Take the tour: http://tour.mail.yahoo.com/mailtour.html From don-lartcxx at isis.cs3-inc.com Mon Jul 11 18:41:24 2005 From: don-lartcxx at isis.cs3-inc.com (Don Cohen) Date: Mon Jul 11 18:41:37 2005 Subject: [LARTC] Re: RFC - bandwidth optimization idea In-Reply-To: <20050710100006.839EE400D@outpost.ds9a.nl> References: <20050710100006.839EE400D@outpost.ds9a.nl> Message-ID: <17106.41268.971072.704272@isis.cs3-inc.com> > From: Paul.Hampson@PObox.com (Paul Hampson) > Wait, you're trying to send more data than the link can take? Then No, of course I don't expect to send more than the limit. > send UDP, throttle it at the local end with a drop-oldest qdisc. Then > you get the effect of 'most recent data is best'. Anything more Yes, that gives me "most recent is best" but that does not do what I want except in a few weird cases. If every packet is independent, perhaps it would suffice to always send the newest, e.g., if I were trying to tell the other side what's the latest clock time. (In that case I'd also limit the queue length to one.) > You gotta prioritise your data, using TOS or diffserv or something. > Set your voice to real-time, so it always gets sent, and the your > other applications can use unused packet-times. Use a dropping qdisc This may be the best I can do in the current world where the facility I described does not exist. It does not solve the problem I described. TOS/diffserv etc is more for use by the intervening infrastructure and this problem applies even in the case where there is no congestion or delay at all in that infrastructure, but only in the link from the sending machine. Using "real time" is just a matter of giving one application priority over others. First, the link itself may have varying bandwidth, and second the other applications might also have urgent data to send. Dropping packets can be disastrous if they happen to contain critical data that is not duplicated in other packets. At very least I have to be able to find out which ones were dropped. But better than all of that is the ability to decide what to send at the last moment. > I have a vauge recollection that this sort of thing is discussed in > Tannenbaum's Computer Networks textbook, to do with positional data > of satellites or something. (eg. if the positional data is delayed, > we write it off, we don't want to delay the data about where we are > _now_ in order to know where we were _then_) If the goal is to listen to the sound from .2 sec ago and it takes .1 sec to get there then clearly it's a waste of time to send data that's older than .1 sec. But the packet in the queue might have some data that's older and some that's newer. I can't drop part of it. Instead I'd like to know that the packet is about to be sent now, and respond by finding the best data to send now. > From: Ed W > This is a total pain to optimise. Ideally I would like an API to be > able to limit the congestion window on the local machine for a > particular connection (which I don't think exists on either windows or > linux?). This way the OS will report that the queue is full quickly to > the local program without buffering up a ton of data. > > The issue in my case is that you have two simultaneous streams in > transit for email, one to receive new mail and one to send mail out. In > the case of the sat phone it's possible to have net buffers which are 20 > secs or so long and so when you send out a status message to say "email > received successfully, send me the next one", it can end up queued > behind a bunch of lower priority data for a VERY long time. Often these > buffers are on the remote ISP end where you have very little control. > This is a serious slowdown on a link which is costing you $1.50/min. I'm not sure I follow the problem, but if you're saying that one stream should have priority over the other, it seems you could do that with two different queues, one with priority over the other. Or something like sfq could at least prevent one connection from waiting for the other to send a lot of data. From surda at shurdix.com Mon Jul 11 18:55:30 2005 From: surda at shurdix.com (Peter Surda) Date: Mon Jul 11 18:55:57 2005 Subject: [LARTC] wonder-shaper In-Reply-To: <42D28E20.641083E3@iswest.com> Message-ID: <200561118553021212@mail.routehat.org> On Mon, 11 Jul 2005 08:20:00 -0700 gypsy wrote: >Wondershaper does not work all that well because the sums of the rates and >ceilings exceeds the root rate. You would do better to look at routehat >(Which I think I spelled wrong), No you didn't but it's being renamed to Shurdix :-). >which uses WRR (Weighted Round Robin). In this specific case I am afraid WRR as implemented on http://wipl-wrr.sourceforge.net/wrr.html won't help. It divides traffic by the IP, but it looks like the original poster needs to divide it among users on a local machine. Yours sincerely, Peter From barbara at rfx.it Mon Jul 11 19:41:17 2005 From: barbara at rfx.it (Barbara M.) Date: Mon Jul 11 19:44:07 2005 Subject: [LARTC] Simple traffic shaping Message-ID: My needs is limiting the outbound traffic of an smtp mail server. It is connected to a gateway via 100Mbit ethernet. I want limits its outbound traffic to max 3 Mbit. I have read lot of docs and tried various script without great results. Any simplest solutions? TIA. Regards, B. From Support at idealconditions.com Mon Jul 11 19:47:43 2005 From: Support at idealconditions.com (ICI Support) Date: Mon Jul 11 19:50:01 2005 Subject: [LARTC] Simple traffic shaping In-Reply-To: Message-ID: <20050711173032.9801E18133@www.idealconditions.com> http://members.cox.net/laitcg/slack1.html Go to the bottom about throttling the bandwidth of a single host. If you just want the daemon itself to be throttled (IE, just the mail traffic) someone else with a bigger clue than me will have to help you. -Michael > -----Original Message----- > From: lartc-bounces@mailman.ds9a.nl [mailto:lartc-bounces@mailman.ds9a.nl] > On Behalf Of Barbara M. > Sent: Monday, July 11, 2005 1:41 PM > To: lartc@mailman.ds9a.nl > Subject: [LARTC] Simple traffic shaping > > > My needs is limiting the outbound traffic of an smtp mail server. > It is connected to a gateway via 100Mbit ethernet. I want limits its > outbound traffic to max 3 Mbit. > > I have read lot of docs and tried various script without great results. > > Any simplest solutions? > > TIA. > Regards, B. > > > > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc From andy.furniss at dsl.pipex.com Mon Jul 11 20:44:40 2005 From: andy.furniss at dsl.pipex.com (Andy Furniss) Date: Mon Jul 11 20:44:40 2005 Subject: [LARTC] HTB Rate and Prio (continued) In-Reply-To: <200507111126.j6BBQT5t007024@dhuumrelay0.mail.eu.uu.net> References: <200507111126.j6BBQT5t007024@dhuumrelay0.mail.eu.uu.net> Message-ID: <42D2BE18.5010407@dsl.pipex.com> Gael Mauleon wrote: > Hi again, > > > > I keep posting about my problem with HTB -> > http://mailman.ds9a.nl/pipermail/lartc/2005q3/016611.html I had a go with what you posted there over lan and with 2 tcp streams it behaves as expected (see below for exact test). Can you reproduce the failiure shaping over a lan rather than your internet connection? If your upstream bandwidth is sold as 2meg then ceil 2000kbit is likely to be too high. You could also try specifying quantum = 1500 on all the leafs as you get it auto calculated from rates otherwise (you can see them with tc -s -d class ls ...). It didn't affect my test though. If you are looking at htbs rate counters remember that they use a really long average (about 100 sec) so they can mislead. I tested below with two netperf tcp send tests to ips I added to another PC on my lan. # /usr/local/netperf/netperf -H 192.168.0.60 -f k -l 60 & /usr/local/netperf/netperf -f k -H 192.168.0.102 -l 60 & which gave - Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^3bits/sec 43689 16384 16384 60.09 1884.66 Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^3bits/sec 43689 16384 16384 60.22 51.58 The script - QOSIN=eth0 tc qdisc del dev $QOSIN root &>/dev/null tc qdisc add dev $QOSIN root handle 1:0 htb tc class add dev $QOSIN parent 1:0 classid 1:1 htb rate 2000kbit ### SUBCLASS1 tc class add dev $QOSIN parent 1:1 classid 1:10 htb rate 750kbit ceil 2000kbit prio 1 tc class add dev $QOSIN parent 1:10 classid 1:101 htb rate 250kbit ceil 2000kbit prio 1 tc qdisc add dev $QOSIN parent 1:101 handle 101: pfifo limit 10 tc class add dev $QOSIN parent 1:10 classid 1:102 htb rate 250kbit ceil 2000kbit prio 1 tc qdisc add dev $QOSIN parent 1:102 handle 102: pfifo limit 10 tc class add dev $QOSIN parent 1:10 classid 1:103 htb rate 250kbit ceil 2000kbit prio 1 tc qdisc add dev $QOSIN parent 1:103 handle 103: pfifo limit 10 tc filter add dev $QOSIN parent 1:0 protocol ip u32 match ip dst 192.168.0.102 flowid 1:102 ###HIGH PRIO ### tc class add dev $QOSIN parent 1:1 classid 1:50 htb rate 50kbit ceil 2000kbit prio 0 quantum 1500 tc qdisc add dev $QOSIN parent 1:50 handle 50: pfifo limit 10 ### LOW PRIO ### tc class add dev $QOSIN parent 1:1 classid 1:60 htb rate 50kbit ceil 2000kbit prio 5 quantum 1500 tc qdisc add dev $QOSIN parent 1:60 handle 60: pfifo limit 10 tc filter add dev $QOSIN parent 1:0 protocol ip u32 match ip dst 192.168.0.60 flowid 1:60 Andy. From andy.furniss at dsl.pipex.com Mon Jul 11 21:35:55 2005 From: andy.furniss at dsl.pipex.com (Andy Furniss) Date: Mon Jul 11 21:35:52 2005 Subject: [LARTC] simple or not? htb+prio In-Reply-To: <20050705224254.18B4E25998D@poczta.interia.pl> References: <20050705224254.18B4E25998D@poczta.interia.pl> Message-ID: <42D2CA1B.4020808@dsl.pipex.com> Krzysiek wrote: > Hi > I have now my outgoing traffic shaped whith root > qdisc htb (where i configure basic rate) and tc > prio (as a leaf with 4 classes) where i can set > priority of different kinds of traffic. It works > but htb is work-conserving so packets are only > delayed (when the rate is exceeded), while i want them dropped. In the case when rate is exceeded i want to drop packets with priority set to 4, then if traffic still too high to drop packets with priority set to 3 and so on. So i need tc-prio under tbf. > But tbf is classless. Can i do what i want in some > other way? Or maybe i'm misunderstanding something > - when htb start to drop packets? > Regards > Krzysiek. Maybe you could use htb + prio and add 4 p/b fifos to the prio classes and set low lengths with the limit parameter of the fifos. Andy. From zybort at telefonica.net Mon Jul 11 22:39:31 2005 From: zybort at telefonica.net (chino) Date: Mon Jul 11 22:35:59 2005 Subject: [LARTC] wonder-shaper In-Reply-To: <200561118553021212@mail.routehat.org> References: <200561118553021212@mail.routehat.org> Message-ID: <200507112239.31390.zybort@telefonica.net> El Lunes, 11 de Julio de 2005 18:55, Peter Surda escribi?: > On Mon, 11 Jul 2005 08:20:00 -0700 gypsy wrote: > >Wondershaper does not work all that well because the sums of the rates and > >ceilings exceeds the root rate. You would do better to look at routehat > >(Which I think I spelled wrong), > > No you didn't but it's being renamed to Shurdix :-). > > >which uses WRR (Weighted Round Robin). > > In this specific case I am afraid WRR as implemented on > http://wipl-wrr.sourceforge.net/wrr.html won't help. It divides traffic by > the IP, but it looks like the original poster needs to divide it among > users on a local machine. > > Yours sincerely, > Peter Whats exactly Shurdix? From surda at shurdix.com Mon Jul 11 23:37:50 2005 From: surda at shurdix.com (Peter Surda) Date: Mon Jul 11 23:38:15 2005 Subject: [LARTC] wonder-shaper In-Reply-To: <200507112239.31390.zybort@telefonica.net> Message-ID: <200561123375021212@mail.routehat.org> On Mon, 11 Jul 2005 22:39:31 +0200 chino wrote: >Whats exactly Shurdix? see http://www.shurdix.org : "Shurdix is a linux distribution for routers, firewalls and embedded systems." Yours sincerely, Peter From barbara at rfx.it Tue Jul 12 00:45:00 2005 From: barbara at rfx.it (Barbara M.) Date: Tue Jul 12 00:45:05 2005 Subject: [LARTC] Simple traffic shaping In-Reply-To: <20050711173032.9801E18133@www.idealconditions.com> References: <20050711173032.9801E18133@www.idealconditions.com> Message-ID: On Mon, 11 Jul 2005, ICI Support wrote: > http://members.cox.net/laitcg/slack1.html > > Go to the bottom about throttling the bandwidth of a single host. > > If you just want the daemon itself to be throttled (IE, just the mail > traffic) someone else with a bigger clue than me will have to help you. > > -Michael Thanks for replay. As you suggested I tried to modify the script for my needs. I finally have this: -------------------------------------------------------- #!/bin/bash # Slow down one ip address on internal network # If you changed anything and want to reload the script, execute # /etc/rc.d/rc.throttle stop # to clean up your existing configuration. # Place IP address to be throttled in TIP TIP="192.168.1.25" # Place device to internal network here DEV="eth0" if [ "$1" = "stop" ]; then echo "Removing Throttle" tc qdisc del dev $DEV root else # assume $1 = start: echo "Throttling $TIP" tc qdisc add dev $DEV root handle 1: cbq avpkt 1000 bandwidth 100mbit tc class add dev $DEV parent 1: classid 1:1 cbq rate 2512kbit allot 1500 prio 5 bounded isolated tc filter add dev $DEV parent 1: protocol ip prio 16 u32 match ip src $TIP flowid 1:1 tc qdisc add dev $DEV parent 1:1 sfq perturb 10 fi -------------------------------------------------------- Have done some test using scp from other internal box. Before starting the tc rules I have UP/DL to the smtp server at 8-10MB/s After activation of rules I have DL to about 270KB/s (as aspected), UP to 550-600KB. ??? Why UPload is affected? Any optimization? TIA, Barbara From donvodka at gmail.com Tue Jul 12 02:10:47 2005 From: donvodka at gmail.com (Edgar) Date: Tue Jul 12 02:11:03 2005 Subject: [LARTC] PLEASE HELP! SHAPING P2P STILL NOT WORKING Message-ID: <200507111910.47677.donvodka@gmail.com> Hello, some days ago, I was asking for help here about not able to do anything when I had bittorrent running, I will post the problem here: I'm using ipp2p to mark p2p packets, and then send them with -j CLASSIFY to the correct HTB class, I see traffic in the class when I start azurerus, and traffic does get shaped, but then I'm still not able to surf the web nor chat nor anything, and I find this very weird, since the traffic is actually shaped, then what am I missing? anyone had a similar problem? I really don't know why this is happening, and I've tried lots of different setups but the problem remains. This is in the server, which NAT the connections to 2 computers at home, I will know post the tcng rules, and the iptables rules too: IPTABLES RULES: iptables -t mangle -A PREROUTING -p tcp -m ipp2p --ipp2p -j MARK --set-mark 3 iptables -t mangle -A PREROUTING -p udp -m ipp2p --bit -j MARK --set-mark 3 iptables -t mangle -A PREROUTING -p tcp -m mark --mark 3 -j CONNMARK --save-mark iptables -t mangle -A ipp2pPOST -o $DEV -m mark --mark 3 -j CLASSIFY --set-class $P2P TCNG FILE, AND TC CLASSES #define UPLOAD eth1 #define UPRATE 25kBps #define P2P 10kBps dev UPLOAD { egress { class ( <$prio> ) ; class ( <$p2p> ) ; class ( <$interactive> ) /* ACK packets go in this class */ if ip_hl == 0x5 && ! (ip_len & 0xffc0) && (raw[33].b >> 4) & 0xff if 1 ; htb () { class ( rate UPRATE, ceil UPRATE ) { $prio = class ( prio 0, rate 6kBps, ceil UPRATE ) { sfq; } ; $p2p = class ( prio 7, rate 1kBps, ceil P2P ) { sfq; } ; $interactive = class ( prio 1, rate 18kBps, ceil UPRATE ) { sfq; } ; } } } } CLASSES class htb 2:1 root rate 200000bit ceil 200000bit burst 1624b cburst 1624b class htb 2:2 parent 2:1 leaf 3: prio 0 rate 48000bit ceil 200000bit burst 1605b cburst 1624b class htb 2:3 parent 2:1 leaf 4: prio 7 rate 8000bit ceil 80000bit burst 1600b cburst 1609b class htb 2:4 parent 2:1 leaf 5: prio 1 rate 144000bit ceil 200000bit burst 1617b cburst 1624b As you can see here, class 2:3 stands for the p2p class, and that's where I send the marked p2p packets, I don't see why this configuration is not working, please help me out with this. Thank you in advance EDGAR MERINO From nix4me at cfl.rr.com Tue Jul 12 02:53:38 2005 From: nix4me at cfl.rr.com (nix4me) Date: Tue Jul 12 02:53:53 2005 Subject: [LARTC] Simple traffic shaping In-Reply-To: References: Message-ID: <42D31492.1010206@cfl.rr.com> Barbara M. wrote: > > My needs is limiting the outbound traffic of an smtp mail server. > It is connected to a gateway via 100Mbit ethernet. I want limits its > outbound traffic to max 3 Mbit. > > I have read lot of docs and tried various script without great results. > > Any simplest solutions? > > TIA. > Regards, B. > > > > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > You should be able to mark all the smtp packets with iptables and then shape them with tc. There are examples of marking and the shaping commands in the docs. For reference, this is how i mark and shape ftp traffic. You will need something similar. I mark ftp traffic by port and then shape. iptables -t mangle -N MYSHAPER-OUT iptables -t mangle -I OUTPUT -o eth0 -j MYSHAPER-OUT iptables -t mangle -A MYSHAPER-OUT -m mark --mark 0 -j MARK --set-mark 20 iptables -t mangle -A MYSHAPER-OUT -p tcp --sport 59999 -j MARK --set-mark 26 iptables -t mangle -A MYSHAPER-OUT -p tcp --sport 50000:51000 -j MARK --set-mark 26 iptables -t mangle -A MYSHAPER-OUT -p tcp -m length --length :64 -j MARK --set-mark 20 # clear it tc qdisc del dev eth0 root #add the root qdisk tc qdisc add dev eth0 root handle 1: htb default 20 #add main rate limit class tc class add dev eth0 parent 1: classid 1:1 htb rate 100mbit #add leaf classes tc class add dev eth0 parent 1:1 classid 1:26 htb rate 40kbps tc class add dev eth0 parent 1:1 classid 1:20 htb rate 100mbit #filter traffic into classes tc filter add dev eth0 parent 1:0 prio 0 protocol ip handle 20 fw flowid 1:20 tc filter add dev eth0 parent 1:0 prio 0 protocol ip handle 26 fw flowid 1:26 Mark From ricardo.a.chamorro at gmail.com Tue Jul 12 03:38:56 2005 From: ricardo.a.chamorro at gmail.com (Ricardo Chamorro) Date: Tue Jul 12 03:37:43 2005 Subject: [LARTC] Several basic doubts Message-ID: <000801c58682$899e4a90$026fa8c0@ricardo> I have a connection cablemodem (down 1024kbit up 256 kbit) that spreads Internet to a LAN of 4 PC. Router-firewall is one 486 DX4 100 96 MB RAM that runs a Debian Sarge (kernel 2,4,25), that does NOT serve nor squid, nor samba, nor smtp, etc.... single do routing-firewalling. I am something confused by opinions and "presumed" manual and howto that I have read and have confused I more... Then I ask to them you: I must do shaping with the NIC that connect with ISP (etho)... or with the NIC of the LAN (eth1)????... Because I have seen opinions on both possibilities, but I have tested the two and second did not give me good results. In the case of using the NIC to Internet (eth0) I must set like CEIL the bandwidth of downstream (1024kbit) or upstream (256kbit)????... And in such case I must set the 75 percent approximately of the bandwidth to avoid to saturate the band? Another question is if it agrees -upon my case- using priorities for the classes... Thanks in advance Ricardo -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ds9a.nl/pipermail/lartc/attachments/20050711/c71b072d/attachment.htm From takano at axe-inc.co.jp Tue Jul 12 03:50:51 2005 From: takano at axe-inc.co.jp (TAKANO Ryousei) Date: Tue Jul 12 03:44:19 2005 Subject: [LARTC] TCP window based shaping In-Reply-To: <42D17D26.1040104@dsl.pipex.com> References: <20050708192337.9724548F7@outpost.ds9a.nl> <17103.19469.965087.345081@isis.cs3-inc.com> <42D17D26.1040104@dsl.pipex.com> Message-ID: <42D321FB.5050207@axe-inc.co.jp> Andy Furniss wrote: > Don Cohen wrote: > >> I recalled a discussion of manipulating outgoing tcp windows in order >> to control return tcp traffic. I finally found at least some of that >> discussion in a thread with the subject above (of this message). >> But I thought someone announced an implementation and I don't see it >> under this thread. If anyone else remembers or knows where I should >> look for it, please let me know. I'm also interested in other >> discussion of the idea and in implementations of related ideas such as >> delaying the acks etc. > > > I don't think there is an implementation other than the commercial > packeteer. > > Andy. We have been proposed TCP window based pacing (shaping) in the PFLDnet2005. You can get the paper and slides from the following URL: http://www.ens-lyon.fr/LIP/RESO/pfldnet2005/TechnicalProgram.php This software called PSPacer is avaiable. But, I am sorry, the current release version does not support TCP window based pacing whereas it supports static pacing. The web page is: http://www.gridmpi.org/ Thanks, TAKANO Ryousei. From jody.shumaker at gmail.com Tue Jul 12 04:36:51 2005 From: jody.shumaker at gmail.com (Jody Shumaker) Date: Tue Jul 12 04:37:06 2005 Subject: [LARTC] Several basic doubts In-Reply-To: <000801c58682$899e4a90$026fa8c0@ricardo> References: <000801c58682$899e4a90$026fa8c0@ricardo> Message-ID: <42D32CC3.1070907@gmail.com> The greatest benefit is using shaping on the outgoing bandwidth. To do that you need to do it for the nic connected to the internet, and you'd want to limit it to a bit under the total bandwidth, for 256kbit i'd recommend something around 244-250, but it varies on the isp as to how reliably you get a full 256kbit. Then how you split up the bandwidth completely depends on what you are trying to accomplish. - Jody Ricardo Chamorro wrote: > I have a connection cablemodem (down 1024kbit up 256 kbit) that > spreads Internet to a LAN of 4 PC. Router-firewall is one 486 DX4 100 > 96 MB RAM that runs a Debian Sarge (kernel 2,4,25), that does NOT > serve nor squid, nor samba, nor smtp, etc.... single > do routing-firewalling. > I am something confused by opinions and "presumed" manual and howto > that I have read and have confused I more... Then I ask to them you: > I must do shaping with the NIC that connect with ISP (etho)... or > with the NIC of the LAN (eth1)????... Because I have seen opinions on > both possibilities, but I have tested the two and second did not give > me good results. In the case of using the NIC to Internet (eth0) I > must set like CEIL the bandwidth of downstream (1024kbit) or upstream > (256kbit)????... And in such case I must set the 75 percent > approximately of the bandwidth to avoid to saturate the band? > Another question is if it agrees -upon my case- using priorities for > the classes... > Thanks in advance > Ricardo > >------------------------------------------------------------------------ > >_______________________________________________ >LARTC mailing list >LARTC@mailman.ds9a.nl >http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > > From gypsy at iswest.com Tue Jul 12 05:25:49 2005 From: gypsy at iswest.com (gypsy) Date: Tue Jul 12 05:25:23 2005 Subject: [LARTC] Simple traffic shaping References: Message-ID: <42D3383D.ACCA086@iswest.com> "Barbara M." wrote: > > My needs is limiting the outbound traffic of an smtp mail server. > It is connected to a gateway via 100Mbit ethernet. I want limits its > outbound traffic to max 3 Mbit. > > I have read lot of docs and tried various script without great results. > > Any simplest solutions? > > TIA. > Regards, B. HTB: tc qdisc add dev $DEV root handle 1: htb default 20 tc class add dev $DEV parent 1: classid 1:1 htb rate 3000kbit burst 6k tc class add dev $DEV parent 1:1 classid 1:20 htb rate 3000kbit \ burst 6k quantum 1500 prio 1 tc filter add dev $DEV parent 1: protocol ip prio 5 u32 \ match ip sport 25 0xffff flowid 1:20 TRICKLE: http://monkey.org/~marius/trickle But I doubt the above will suit you because you don't tell us anything else about the traffic on your mail server. -- gypsy From andy.furniss at dsl.pipex.com Tue Jul 12 10:46:01 2005 From: andy.furniss at dsl.pipex.com (Andy Furniss) Date: Tue Jul 12 10:46:08 2005 Subject: [LARTC] TCP window based shaping In-Reply-To: <42D321FB.5050207@axe-inc.co.jp> References: <20050708192337.9724548F7@outpost.ds9a.nl> <17103.19469.965087.345081@isis.cs3-inc.com> <42D17D26.1040104@dsl.pipex.com> <42D321FB.5050207@axe-inc.co.jp> Message-ID: <42D38349.8020307@dsl.pipex.com> TAKANO Ryousei wrote: > Andy Furniss wrote: > >> Don Cohen wrote: >> >>> I recalled a discussion of manipulating outgoing tcp windows in order >>> to control return tcp traffic. I finally found at least some of that >>> discussion in a thread with the subject above (of this message). >>> But I thought someone announced an implementation and I don't see it >>> under this thread. If anyone else remembers or knows where I should >>> look for it, please let me know. I'm also interested in other >>> discussion of the idea and in implementations of related ideas such as >>> delaying the acks etc. >> >> >> >> I don't think there is an implementation other than the commercial >> packeteer. >> >> Andy. > > > We have been proposed TCP window based pacing (shaping) in the > PFLDnet2005. You can get the paper and slides from the following URL: > http://www.ens-lyon.fr/LIP/RESO/pfldnet2005/TechnicalProgram.php Looks intresting but I can't get the pdf (not found) and haven't got anything that opens .ppt s > > This software called PSPacer is avaiable. But, I am sorry, the current > release version does not support TCP window based pacing whereas it > supports static pacing. Am I right in thinking that PSPspacer has to be run on the server - in which case would the tcp window shaping also be for locally generated traffic only, or do both work on routers? Andy. From officeworks.supratim at gmail.com Tue Jul 12 11:34:38 2005 From: officeworks.supratim at gmail.com (Supratim Mitra) Date: Tue Jul 12 11:34:42 2005 Subject: [LARTC] (no subject) Message-ID: <5fea9f5405071202346bd0252c@mail.gmail.com> Hi. Need some solution to resolve Vlan on Fedora Core 2 release Operating system configured on HP Proliant DL 320 G3 server (Monitor Server). The system is part of telecommunication equipment to manage the data VLAN tagging of data or management traffic.I have enclosed a brief connectivity diagram. The problem is when FTP is initiated within VLAN the downloading is reduced to half, user is configured to 512 KBPS Broadband Connection. If the vlan is removed the FTP is fine both direct (Down & Uploading). I feel the problem is somever in the VLAN configuration which we are not to pinpoint the problem. Please suggest!. Thanx in Advance! Regards Supratim -------------- next part -------------- A non-text attachment was scrubbed... Name: Slide1.GIF Type: image/gif Size: 4481 bytes Desc: not available Url : http://mailman.ds9a.nl/pipermail/lartc/attachments/20050712/c24007a8/Slide1.gif From barbara at rfx.it Tue Jul 12 13:59:28 2005 From: barbara at rfx.it (Barbara M.) Date: Tue Jul 12 13:59:35 2005 Subject: [LARTC] Simple traffic shaping In-Reply-To: <42D3383D.ACCA086@iswest.com> References: <42D3383D.ACCA086@iswest.com> Message-ID: On Mon, 11 Jul 2005, gypsy wrote: >> My needs is limiting the outbound traffic of an smtp mail server. >> It is connected to a gateway via 100Mbit ethernet. I want limits its >> outbound traffic to max 3 Mbit. > HTB: > tc qdisc add dev $DEV root handle 1: htb default 20 Create the root and set the default for traffic to filter/class "20". Needed (filter/class "20")? > tc class add dev $DEV parent 1: classid 1:1 htb rate 3000kbit burst 6k Create the class 1:1, set maximum rate to 3mbit. Can be useful increase the 6k burst? > tc class add dev $DEV parent 1:1 classid 1:20 htb rate 3000kbit \ > burst 6k quantum 1500 prio 1 ??? what do the "quantum 1500" part? > tc filter add dev $DEV parent 1: protocol ip prio 5 u32 \ > match ip sport 25 0xffff flowid 1:20 Create a filter for smtp traffic? Why "sport 25"? I am interested in outgoing traffic. This box receive the outgoing mails from other internal servers and do the delivery. No other activity/traff. So can be useful that it receive traffic from local server at full speed, but delivery it at limited rate (the problem is mailing list users that sometime distribuite big mail (0.5-2 MB) to 1.000-3.000 subscribers causing peak that ... :-( Really I am thinking to use it to shape the total traffic from a server with no differentiation on services (so I can use it in mail or httpd server ...). Regards, B. From admin at premiere-heure.fr Tue Jul 12 14:08:15 2005 From: admin at premiere-heure.fr (Gael Mauleon) Date: Tue Jul 12 14:07:40 2005 Subject: [LARTC] HTB Rate and Prio (continued) In-Reply-To: <42D2BE18.5010407@dsl.pipex.com> Message-ID: <200507121207.j6CC7YrA027308@dhuumrelay1.mail.eu.uu.net> > I had a go with what you posted there over lan and with 2 tcp streams it > behaves as expected (see below for exact test). > > Can you reproduce the failiure shaping over a lan rather than your > internet connection? > > If your upstream bandwidth is sold as 2meg then ceil 2000kbit is likely > to be too high. > > You could also try specifying quantum = 1500 on all the leafs as you get > it auto calculated from rates otherwise (you can see them with tc -s -d > class ls ...). It didn't affect my test though. > > If you are looking at htbs rate counters remember that they use a really > long average (about 100 sec) so they can mislead. > > I tested below with two netperf tcp send tests to ips I added to another > PC on my lan. > > # /usr/local/netperf/netperf -H 192.168.0.60 -f k -l 60 & > /usr/local/netperf/netperf -f k -H 192.168.0.102 -l 60 & > > which gave - > > Recv Send Send > Socket Socket Message Elapsed > Size Size Size Time Throughput > bytes bytes bytes secs. 10^3bits/sec > > 43689 16384 16384 60.09 1884.66 > Recv Send Send > Socket Socket Message Elapsed > Size Size Size Time Throughput > bytes bytes bytes secs. 10^3bits/sec > > 43689 16384 16384 60.22 51.58 Did the exact same test and it's working (10kbits for the low prio was the only diff) !! That's with pfifo -> TCP STREAM TEST to 10.0.1.228 Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^3bits/sec 87380 8192 8192 63.00 35.37 TCP STREAM TEST to 10.0.1.227 Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^3bits/sec 87380 8192 8192 60.00 1897.27 That's with sfq -> TCP STREAM TEST to 10.0.1.227 Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^3bits/sec 87380 8192 8192 60.00 1918.02 TCP STREAM TEST to 10.0.1.228 Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^3bits/sec 87380 8192 8192 60.00 28.40 10.0.1.228 was the lowprio IP. So everything worked fine for the test...!!! After that I tested with my original script, I just changed the netfilter rules to classify packets with the same rules I used for the netperfs test and I had the same good results... So it must be something with my 2m line or with the traffic type I shape. I quite don't understand the concept of putting the rate of the line lower than it's true value, can you explain me this and do the excess bandwith is lost ? What is a good value for a 2m line (SDSL) ? I'll try tomorrow to have an host outside with netperf so I can test the line itself. Again thanks for your time and help Andy. From takano at axe-inc.co.jp Tue Jul 12 15:44:45 2005 From: takano at axe-inc.co.jp (TAKANO Ryousei) Date: Tue Jul 12 15:43:37 2005 Subject: [LARTC] TCP window based shaping In-Reply-To: <42D38349.8020307@dsl.pipex.com> References: <20050708192337.9724548F7@outpost.ds9a.nl> <17103.19469.965087.345081@isis.cs3-inc.com> <42D17D26.1040104@dsl.pipex.com> <42D321FB.5050207@axe-inc.co.jp> <42D38349.8020307@dsl.pipex.com> Message-ID: <20050712224445.1f5b81fa.takano@axe-inc.co.jp> > > We have been proposed TCP window based pacing (shaping) in the > > PFLDnet2005. You can get the paper and slides from the following URL: > > http://www.ens-lyon.fr/LIP/RESO/pfldnet2005/TechnicalProgram.php > > Looks intresting but I can't get the pdf (not found) and haven't got > anything that opens .ppt s > Sorry! You can get the paper from the following URL: http://www.gridmpi.org/publications/pfldnet05-takano.pdf > > This software called PSPacer is avaiable. But, I am sorry, the current > > release version does not support TCP window based pacing whereas it > > supports static pacing. > > Am I right in thinking that PSPspacer has to be run on the server - in > which case would the tcp window shaping also be for locally generated > traffic only, or do both work on routers? > I think tcp window based shaping is difficult on routers. PSPacer basically assumes the use of end-to-end traffic engineering. (PSPacer can also regulate bandwidth of through traffic at static target transmission rate on routers.) Thanks, TAKANO Ryousei From ji.li3 at hp.com Tue Jul 12 15:50:43 2005 From: ji.li3 at hp.com (Li, Ji) Date: Tue Jul 12 15:51:16 2005 Subject: [LARTC] Teql and NetEm can't work together Message-ID: <628BFCE8B64706469FE4D4852CEC953706C5D52E@tayexc14.americas.cpqcorp.net> Thanks in advance! Summary: when I load netem and teql together, teql doesn't work correctly. (If I load teql only, everything is fine) I loaded both netem and teql. Netem is associated with eth0, and teql is associated with both eth0 and eth1. But traffic only goes out of eth1. Attached are the commands that I used to configure teql and netem (on machine 1), and commands to configure teql only (on machien 2), their routing tables and "tc qdisc ls" results. What's wrong with my configuration? (I tried loading teql only without netem, and teql works as expected: ping traffic goes to both interfaces. ) ========= Machine 1 with NetEm ============ Commands to install NetEm and TEQL together: # modprobe sch_teql # tc qdisc add dev eth0 root handle 1: netem delay 10ms # tc qdisc add dev eth0 parent 1:1 handle 10: teql0 # tc qdisc add dev eth1 root teql0 # ip link set dev teql0 up # ip addr add dev eth0 10.0.0.1/30 # ip addr add dev eth1 10.0.0.5/30 # ip addr add dev teql0 10.0.0.9/30 # echo 0 > /proc/sys/net/ipv4/conf/eth0/rp_filter # echo 0 > /proc/sys/net/ipv4/conf/eth1/rp_filter [root@machine1]# tc qdisc ls qdisc netem 1: dev eth0 limit 1000 delay 100.0ms qdisc teql0 10: dev eth0 parent 1:1 qdisc teql0 8001: dev eth1 qdisc pfifo_fast 0: dev teql0 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 Destination Gateway Genmask Flags Metric Ref Use Iface 10.0.0.8 0.0.0.0 255.255.255.252 U 0 0 0 teql0 10.0.0.0 0.0.0.0 255.255.255.252 U 0 0 0 eth0 10.0.0.4 0.0.0.0 255.255.255.252 U 0 0 0 eth1 ============== Machine 2 without NetEm ============== Commands to install TEQL: # modprobe sch_teql # tc qdisc add dev eth0 root teql0 # tc qdisc add dev eth1 root teql0 # ip link set dev teql0 up # ip addr add dev eth0 10.0.0.2/30 # ip addr add dev eth1 10.0.0.6/30 # ip addr add dev teql0 10.0.0.10/30 # echo 0 > /proc/sys/net/ipv4/conf/eth0/rp_filter # echo 0 > /proc/sys/net/ipv4/conf/eth1/rp_filter [root@machine2]# tc qdisc ls qdisc teql0 8001: dev eth0 qdisc teql0 8002: dev eth1 qdisc pfifo_fast 0: dev teql0 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 Destination Gateway Genmask Flags Metric Ref Use Iface 10.0.0.8 0.0.0.0 255.255.255.252 U 0 0 0 teql0 10.0.0.0 0.0.0.0 255.255.255.252 U 0 0 0 eth0 10.0.0.4 0.0.0.0 255.255.255.252 U 0 0 0 eth1 Best, -Ji -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ds9a.nl/pipermail/lartc/attachments/20050712/fb47f796/attachment.htm From tdi at pozman.pl Tue Jul 12 19:47:30 2005 From: tdi at pozman.pl (Dariusz Dwornikowski) Date: Tue Jul 12 19:47:43 2005 Subject: [LARTC] QOS HELP PLEASE In-Reply-To: <20050711163907.8747.qmail@web31515.mail.mud.yahoo.com> References: <20050711163907.8747.qmail@web31515.mail.mud.yahoo.com> Message-ID: <20050712194730.08e2051e@tdi.pozman.pl> ok i did the calculations and here it is : www.tdi.pozman.pl/fir3 now ping to world from server are superb but. ping to access points in the network are 1-2seconds... people keep on calling and tlling that tranfers on parts of network are 1-3kb/s. i hae to mention that on the old server all was perfect. please anyone :) -- *Dariusz 'tdi' Dwornikowski | Gentoo | admin at pozman.pl | *[JID]:tdi@gentoo.pl|[gg]:2266034|[IRC]:#gentoo-pl@freenode | *[MAIL]:tdi@pozman.pl|[WWW]:www.tdi.pozman.pl | *Serwery,administracja,webapps - www.ProAdmin.com.pl | *Fingerprint:43E21CC46DAFD2F754E91547D59B39F56AAA4B5F | -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mailman.ds9a.nl/pipermail/lartc/attachments/20050712/32b3e706/attachment.bin From oscar at ufomechanic.net Tue Jul 12 20:14:24 2005 From: oscar at ufomechanic.net (Oscar Mechanic) Date: Tue Jul 12 20:14:35 2005 Subject: [LARTC] QOS HELP PLEASE In-Reply-To: <20050712194730.08e2051e@tdi.pozman.pl> References: <20050711163907.8747.qmail@web31515.mail.mud.yahoo.com> <20050712194730.08e2051e@tdi.pozman.pl> Message-ID: <1121192064.5157.53.camel@OSCARLAPLIN> On Tue, 2005-07-12 at 19:47 +0200, Dariusz Dwornikowski wrote: > ok i did the calculations and here it is : www.tdi.pozman.pl/fir3 > > now ping to world from server are superb but. ping to access points in the network are 1-2seconds... > people keep on calling and tlling that tranfers on parts of network are 1-3kb/s. > > i hae to mention that on the old server all was perfect. > please anyone :) > > > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc Anytime I have ever had this problem which was 4 times 2 weeks ago was due to a loop in the network. I have no explanation for it and am pulling my hair out. Some time rebooting AP works sometimes reboot a switch works. Its not my network I am just trying to help someone fix it. Have not had the problem since I told the guy to pull all the cables out and re cable the network. I would not of said this only your iptables and tc looks like mine and the problem sounds so-similar. But this was the only post I could find from you. From andre at tels.com.br Tue Jul 12 20:38:46 2005 From: andre at tels.com.br (Andre Ribeiro) Date: Tue Jul 12 20:38:59 2005 Subject: [LARTC] htbinit Web manager Message-ID: <20050712183851.BE49F40F2@outpost.ds9a.nl> Hi. Anybody can tell me if exist na web interface to manage /etc/sysconfig/htb files ? That interface will help me so much creating and changing speed settings of my users. Thanks, Andr? Ribeiro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ds9a.nl/pipermail/lartc/attachments/20050712/1cbb8054/attachment-0001.htm From unki at netshadow.at Tue Jul 12 20:49:38 2005 From: unki at netshadow.at (Andreas Unterkircher) Date: Tue Jul 12 20:49:43 2005 Subject: [LARTC] htbinit Web manager In-Reply-To: <20050712183851.BE49F40F2@outpost.ds9a.nl> References: <20050712183851.BE49F40F2@outpost.ds9a.nl> Message-ID: <42D410C2.4040308@netshadow.at> Perhaps you wanna try this: http://shaper.netshadow.at/ Cheers, Andreas Andre Ribeiro wrote: > > > Hi. > > > > Anybody can tell me if exist na web interface to manage > /etc/sysconfig/htb files ? > > > > That interface will help me so much creating and changing speed > settings of my users. > > > > Thanks, > > Andr? Ribeiro > >------------------------------------------------------------------------ > >_______________________________________________ >LARTC mailing list >LARTC@mailman.ds9a.nl >http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > > From tdi at pozman.pl Tue Jul 12 21:59:36 2005 From: tdi at pozman.pl (Dariusz Dwornikowski) Date: Tue Jul 12 21:59:50 2005 Subject: [LARTC] QOS problem -ng Message-ID: <20050712215936.75d9dcb2@tdi.pozman.pl> ok i gave up. can sb point me docs on how design proper qos rules ? about differences between them and the usage (which for wifi, which for ISPs which for homenet) ill read and do it -- *Dariusz 'tdi' Dwornikowski | Gentoo | admin at pozman.pl | *[JID]:tdi@gentoo.pl|[gg]:2266034|[IRC]:#gentoo-pl@freenode | *[MAIL]:tdi@pozman.pl|[WWW]:www.tdi.pozman.pl | *Serwery,administracja,webapps - www.ProAdmin.com.pl | *Fingerprint:43E21CC46DAFD2F754E91547D59B39F56AAA4B5F | -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mailman.ds9a.nl/pipermail/lartc/attachments/20050712/b4931e48/attachment.bin From nicolas.salvo at gmail.com Tue Jul 12 23:34:43 2005 From: nicolas.salvo at gmail.com (Nicolas Salvo) Date: Tue Jul 12 23:34:50 2005 Subject: [LARTC] Two class C subnets... Message-ID: Hi!, I'm getting nuts with this so I really need your help... I have a network in the 192.168.100.0/24 class C with the gw in 192.168.100.1, all the company works great with that scenario but now we need to add more class C networks, now this is what I have: 2: eth0: mtu 1500 qdisc pfifo_fast qlen 1000 link/ether 00:08:a1:53:6d:51 brd ff:ff:ff:ff:ff:ff inet 192.168.100.1/24 brd 192.168.100.255 scope global eth0 inet 192.168.2.1/24 brd 192.168.2.255 scope global eth0 0: from all lookup local 32763: from 192.168.100.0/24 to 192.168.2.0/24 lookup Monitors2 32764: from 192.168.2.0/24 to 192.168.100.0/24 lookup Monitors 32765: from xxx.xxx.xxx.xxx lookup ISP 32766: from all lookup main 32767: from all lookup 253 Monitors Table: default via 192.168.100.1 dev eth0 Monitors2 Table: default via 192.168.2.1 dev eth0 Main Table: 192.168.100.0/24 dev eth0 scope link 192.168.2.0/24 dev eth0 proto kernel scope link xxx.xxx.xxx.xxx/24 dev eth2 scope link 127.0.0.0/8 dev lo scope link default via xxx.xxx.xxx.xxx dev eth2 The only thing that I want is that if a packet comming from the 192.168.2.0/24 is going to some host in the 192.168.100.0/24 net, the kerenl route those packets to 192.168.100.1 or in the other way, now, I don't know what I'm doing wrong but that doesn't work for me, the packets got lost somewhere, I'm a little lost with this so please can anyone give me a hint? Thanks -- Nicolas A. Salvo Capital Federal Buenos Aires - Argentina From nistnet_user at yahoo.com Wed Jul 13 02:10:07 2005 From: nistnet_user at yahoo.com (js si) Date: Wed Jul 13 02:10:14 2005 Subject: [LARTC] tbf initial burst Message-ID: <20050713001007.32521.qmail@web34412.mail.mud.yahoo.com> I am using tbf to do bandwidth limitation. i found that when i start passing traffic there is a burst and then the rate goes down to what is configured. is this a known issue or do i need to change some parameters? thanks. --------------------------------- Do you Yahoo!? Read only the mail you want - Yahoo! Mail SpamGuard. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ds9a.nl/pipermail/lartc/attachments/20050712/308d5da6/attachment.htm From mabrown-lartc at securepipe.com Wed Jul 13 06:27:16 2005 From: mabrown-lartc at securepipe.com (Martin A. Brown) Date: Wed Jul 13 06:28:24 2005 Subject: [LARTC] tbf initial burst In-Reply-To: <20050713001007.32521.qmail@web34412.mail.mud.yahoo.com> References: <20050713001007.32521.qmail@web34412.mail.mud.yahoo.com> Message-ID: Greetings, : I am using tbf to do bandwidth limitation. i found that when i : start passing traffic there is a burst and then the rate goes : down to what is configured. is this a known issue or do i need to : change some parameters? The behaviour you have described is exactly the theoretical goal of a token bucket filter, and also the practical behaviour of the TBF queueing discipline. In other words, congratulations, you are using a TBF! You probably wish to tweak parameters. -Martin -- Martin A. Brown --- Wonderfrog Enterprises --- martin@wonderfrog.net From videoip at gmail.com Wed Jul 13 06:39:23 2005 From: videoip at gmail.com (VideoIP) Date: Wed Jul 13 06:40:36 2005 Subject: [LARTC] =?iso-8859-1?q?HTB=3A_=BFhow_do_burst/cburst_work_exactl?= =?iso-8859-1?q?y=3F?= References: <20050713001007.32521.qmail@web34412.mail.mud.yahoo.com> Message-ID: <001601c58764$d97f86e0$6501a8c0@FLO> I?ve read all the definitions of burst and cburst and I?ve tried playing with the parameters and graphing the output of the filter to see its effects, but STILL I can?t figure out how the parameters work exactly. ?Could anyone give a better explanation than the manpage? From mabrown-lartc at securepipe.com Wed Jul 13 07:14:01 2005 From: mabrown-lartc at securepipe.com (Martin A. Brown) Date: Wed Jul 13 07:15:22 2005 Subject: [LARTC] =?iso-8859-1?q?HTB=3A_=BFhow_do_burst/cburst_work_exactl?= =?iso-8859-1?q?y=3F?= In-Reply-To: <001601c58764$d97f86e0$6501a8c0@FLO> References: <20050713001007.32521.qmail@web34412.mail.mud.yahoo.com> <001601c58764$d97f86e0$6501a8c0@FLO> Message-ID: Hello, : I?ve read all the definitions of burst and cburst and I?ve tried : playing with the parameters and graphing the output of the filter : to see its effects, but STILL I can?t figure out how the : parameters work exactly. : : Could anyone give a better explanation than the manpage? Have you tried Stef's site? He has a good page [0] that talks about the various tests he did while experimenting with HTB burst and cburst parameters? Some time ago, I took a stab at creating a visual representation [1] of a hypothetical HTB configuration [2]. In order to understand when cburst is used, look for the diamond-shaped boxes in the diagram which talk about tokens and ctokens. Every HTB class has two buckets. rate bucket is of burst size, traffic uses tokens ceil bucket is of cburst size, traffic uses ctokens My diagram may give you the framework to understand exactly how they are used if it's still unclear to you, but Stef's site will give you much better detail on the results of using burst and cburst. Of the scenarios he describes, I like the results of Test 7 the best. The only guideline that struck me after reading his results was to prefer burst and cburst usage on parent classes. Good luck, -Martin [0] http://www.docum.org/docum.org/tests/htb/burst/ [1] http://linux-ip.net/traffic-control/htb-class.png [2] http://linux-ip.net/traffic-control/diagram.html -- Martin A. Brown --- Wonderfrog Enterprises --- martin@wonderfrog.net From gypsy at iswest.com Wed Jul 13 07:09:57 2005 From: gypsy at iswest.com (gypsy) Date: Wed Jul 13 07:16:10 2005 Subject: [LARTC] QOS problem -ng References: <20050712215936.75d9dcb2@tdi.pozman.pl> Message-ID: <42D4A225.6B18C01C@iswest.com> Dariusz Dwornikowski wrote: > > ok i gave up. can sb point me docs on how design proper qos rules ? about differences between them and the usage (which for wifi, which for ISPs which for homenet) > > ill read and do it Start with Jim diGriz in shaping / QoS / Traffic Control at http://yesican.chsoft.biz/lartc Then read Dan Singletary and Emmanuel Roger. That ought to get you going. -- gypsy From videoip at gmail.com Wed Jul 13 09:20:49 2005 From: videoip at gmail.com (VideoIP) Date: Wed Jul 13 09:21:26 2005 Subject: =?iso-8859-1?Q?Re:_=5BLARTC=5D_HTB:_=BFhow_do_burst/cburst_work_exactly=3F?= References: <20050713001007.32521.qmail@web34412.mail.mud.yahoo.com> <001601c58764$d97f86e0$6501a8c0@FLO> Message-ID: <006901c5877b$6a2feca0$6501a8c0@FLO> Ok, that?s great. I have a much better idea now. However I have a doubt on [1]. When packets arrive to an HTB filter, it first checks to see if there are ctokens available. How come if there are, it then checks for tokens instead of just dequeuing at full speed. And if there are none: shouldn?t it THEN check for tokens instead of discarding right away? Also, could you give me an advice or reference on the following? I need a child class to allow passage to a video stream that I KNOW has mean X kbps and seldom peaks of Y kbps and T seconds. Would the best way be to just configure mean=X, ceil=Y? Or should I configure mean=ceil=X and calculate a cburst that?ll allow passage of the peaks? Or maybe a third option. ----- Original Message ----- From: "Martin A. Brown" To: "VideoIP" Cc: "lartc" Sent: Wednesday, July 13, 2005 2:14 AM Subject: Re: [LARTC] HTB: ?how do burst/cburst work exactly? Hello, : I?ve read all the definitions of burst and cburst and I?ve tried : playing with the parameters and graphing the output of the filter : to see its effects, but STILL I can?t figure out how the : parameters work exactly. : : Could anyone give a better explanation than the manpage? Have you tried Stef's site? He has a good page [0] that talks about the various tests he did while experimenting with HTB burst and cburst parameters? Some time ago, I took a stab at creating a visual representation [1] of a hypothetical HTB configuration [2]. In order to understand when cburst is used, look for the diamond-shaped boxes in the diagram which talk about tokens and ctokens. Every HTB class has two buckets. rate bucket is of burst size, traffic uses tokens ceil bucket is of cburst size, traffic uses ctokens My diagram may give you the framework to understand exactly how they are used if it's still unclear to you, but Stef's site will give you much better detail on the results of using burst and cburst. Of the scenarios he describes, I like the results of Test 7 the best. The only guideline that struck me after reading his results was to prefer burst and cburst usage on parent classes. Good luck, -Martin [0] http://www.docum.org/docum.org/tests/htb/burst/ [1] http://linux-ip.net/traffic-control/htb-class.png [2] http://linux-ip.net/traffic-control/diagram.html -- Martin A. Brown --- Wonderfrog Enterprises --- martin@wonderfrog.net From lists at wildgooses.com Wed Jul 13 11:03:49 2005 From: lists at wildgooses.com (Ed W) Date: Wed Jul 13 11:04:00 2005 Subject: [LARTC] Re: RFC - bandwidth optimization idea In-Reply-To: <20050710101248.GA18859@yurika.videohost.com.au> References: <20050709100005.950B7495F@outpost.ds9a.nl> <17103.60531.516880.785001@isis.cs3-inc.com> <20050709172341.GA9529@yurika.videohost.com.au> <42D0D2F9.9080603@wildgooses.com> <20050710101248.GA18859@yurika.videohost.com.au> Message-ID: <42D4D8F5.4060808@wildgooses.com> >Assuming you can send both ways simultaneously, or at least guarantee >some traffic time outbound no matter how busy the inbound traffic, >you would surely have to pipeline your commands in order to get any >kind of efficient use out of a high-latency link like a satellite link. >Otherwise you lose 2x round-trip-time of incoming data stream while >you request the next item. > >In this situation, you would want the OS buffers to be as full as >possible so the minimal time possible is spent receiving, but using >a traffic-shaping solution so that the most important stuff (acks >and commands) goes out in preference to whatever else you're sending. > > Yes you do want to pipeline, but you still don't want the OS buffers full as possible. Consider that you might want to know a message was sent successfully before sending the next message, but at the same time you have the pipe full with downloading new messages. The OK which says the message was sent OK can be behind 15-20 seconds worth of downloads - hence you have to wait a long time before you can start sending the next message! Also you can't use any kind of QOS here because the hypothetical 15-20 second buffer is at the remote ISP end. (Who are not cooperative) It's a tricky situation all you can do is figure out how to keep changing your protocol so that you don't ever need to hear a reply before you continue sending. Anyone wants to buy it then drop me a line! :-) Ed W From lists at wildgooses.com Wed Jul 13 11:05:49 2005 From: lists at wildgooses.com (Ed W) Date: Wed Jul 13 11:05:55 2005 Subject: [LARTC] Re: RFC - bandwidth optimization idea In-Reply-To: <17106.41268.971072.704272@isis.cs3-inc.com> References: <20050710100006.839EE400D@outpost.ds9a.nl> <17106.41268.971072.704272@isis.cs3-inc.com> Message-ID: <42D4D96D.5040001@wildgooses.com> >I'm not sure I follow the problem, but if you're saying that one >stream should have priority over the other, it seems you could do >that with two different queues, one with priority over the other. >Or something like sfq could at least prevent one connection from >waiting for the other to send a lot of data. > > You could if you have control over the queues. But they are on the remote ISP end... So the problem is similar to the one you describe - once the data is inflight you lose control but you want to limit how much data is inflight so that you have as much control as possible... Ed W From admin at premiere-heure.fr Wed Jul 13 12:25:33 2005 From: admin at premiere-heure.fr (Gael Mauleon) Date: Wed Jul 13 12:25:00 2005 Subject: [LARTC] HTB Rate and Prio (continued) In-Reply-To: <200507121207.j6CC7YrA027308@dhuumrelay1.mail.eu.uu.net> Message-ID: <200507131024.j6DAOodA022265@dhuumrelay0.mail.eu.uu.net> Ok I tested the shaping on the SDSL line with netperf and an host outside. Same script than before, I classify the packets into qdisc based on the source address in netfilter and here are the result, that's with sfq. I'm positive on the right traffic going to the right class. TCP STREAM TEST to 81.57.243.113 (NORMAL) Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^3bits/sec 87380 8192 8192 60.00 282.90 TCP STREAM TEST to 81.57.243.113 (LOWPRIO) Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^3bits/sec 87380 8192 8192 61.00 390.00 In fact the rate are just share sometimes it's the LOWPRIO that has more Sometimes it's the NORMAL traffic... As a note, there were other traffic on the line at the same moment I didn't classify but I was expecting the ratio to be kept (LOWPRIO is prio 5, 50kbits, NORMAL is prio 1 200kbits) Well I need to investigate more, I don't understand why it don't work on the SDSL line... My installation is quite simple : Cisco Router <-> Linux Box (QOS) <-> LAN And I was testing egress traffic from LAN to internet, but it's the same for ingress. Do this can appear if the ratio I put are bigger than the actual line ?? And again what is a good ratio/ceil for a so sold 2mbits SDSL line ?? Ga?l. From officeworks.supratim at gmail.com Wed Jul 13 12:27:08 2005 From: officeworks.supratim at gmail.com (Supratim Mitra) Date: Wed Jul 13 12:27:13 2005 Subject: [LARTC] HTB traffic control over VLAN interface. Message-ID: <5fea9f54050713032718bf2a84@mail.gmail.com> Hi All, This is my first mail to the LARTC mailing list.I am having some problems with the download bandwidth over VLAN.The setup i am having at my place is somewhat below.. +-------------------+ | | | FTP Server | | | +-------------------+ | +---------------+ | | | | | eth1=|-------+ | My Box | | eth0=|--------+ | | | +---------------+ | | +----------+----------+-----------------------+ | | | | vlans -> eth0.1 eth0.2 eth0.3 ................eth0.1001 I am doing some traffic control with my box.It has got the phsical interfaces eth0 and eth1.eth1 is connected the FTP server and on eth0 VLANs are created.It has the usual tc(HTB) and iptables rules added for traffic control. when i am downloading anything over physical interface eth0 without using any VLAN i am getting the desired download bandwidth with TCP,UDP and ICMP.The upload bandwidth using the VLAN is desired one.Even the download bandwidth using the VLAN interfaces with UDP and ICMP is correct. But the problem arises when i am downloading anything using the VLAN interfaces with TCP.Its is showing around 180kbps for the alloted 256kbps and around 350kbps for 512kbps. Thanking in Advance! regards, Supratim -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ds9a.nl/pipermail/lartc/attachments/20050713/201fbb57/attachment.htm From admin at premiere-heure.fr Wed Jul 13 12:34:34 2005 From: admin at premiere-heure.fr (Gael Mauleon) Date: Wed Jul 13 12:34:01 2005 Subject: [LARTC] HTB Rate and Prio (continued) In-Reply-To: <200507131024.j6DAOodA022265@dhuumrelay0.mail.eu.uu.net> Message-ID: <200507131033.j6DAXqV2028506@dhuumrelay0.mail.eu.uu.net> Just tested with 1800kbits as the rate/ceil of the line, with adjustment to all the rate to match the total rate, but I have the same result, bandwith seems to be shared just like if there were no qos in place... I'll do a full round trip of tests today, it must be hidden somewhere :) > -----Message d'origine----- > De?: lartc-bounces@mailman.ds9a.nl [mailto:lartc-bounces@mailman.ds9a.nl] > De la part de Gael Mauleon > Envoy??: mercredi 13 juillet 2005 12:26 > ??: lartc@mailman.ds9a.nl > Objet?: RE: [LARTC] HTB Rate and Prio (continued) > > Ok I tested the shaping on the SDSL line with netperf and > an host outside. > > Same script than before, I classify the packets into qdisc based on the > source address in netfilter and here are the result, that's with sfq. > I'm positive on the right traffic going to the right class. > > > TCP STREAM TEST to 81.57.243.113 (NORMAL) > Recv Send Send > Socket Socket Message Elapsed > Size Size Size Time Throughput > bytes bytes bytes secs. 10^3bits/sec > > 87380 8192 8192 60.00 282.90 > > > > TCP STREAM TEST to 81.57.243.113 (LOWPRIO) > Recv Send Send > Socket Socket Message Elapsed > Size Size Size Time Throughput > bytes bytes bytes secs. 10^3bits/sec > > 87380 8192 8192 61.00 390.00 > > > In fact the rate are just share sometimes it's the LOWPRIO that has more > Sometimes it's the NORMAL traffic... > > As a note, there were other traffic on the line at the same moment I > didn't > classify but I was expecting the ratio to be kept (LOWPRIO is prio 5, > 50kbits, NORMAL is prio 1 200kbits) > > Well I need to investigate more, I don't understand why it don't work on > the > SDSL line... > > My installation is quite simple : > > Cisco Router <-> Linux Box (QOS) <-> LAN > > And I was testing egress traffic from LAN to internet, but it's the same > for > ingress. > > > Do this can appear if the ratio I put are bigger than the actual line ?? > And again what is a good ratio/ceil for a so sold 2mbits SDSL line ?? > > > Ga?l. > > > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc From Andreas.Klauer at metamorpher.de Wed Jul 13 12:45:59 2005 From: Andreas.Klauer at metamorpher.de (Andreas Klauer) Date: Wed Jul 13 12:46:05 2005 Subject: [LARTC] QOS HELP PLEASE In-Reply-To: <20050712194730.08e2051e@tdi.pozman.pl> References: <20050711163907.8747.qmail@web31515.mail.mud.yahoo.com> <20050712194730.08e2051e@tdi.pozman.pl> Message-ID: <200507131245.59315.Andreas.Klauer@metamorpher.de> On Tuesday 12 July 2005 19:47, Dariusz Dwornikowski wrote: > ok i did the calculations and here it is : www.tdi.pozman.pl/fir3 Is this URL valid? I get a 404. Andreas From mabrown-lartc at securepipe.com Wed Jul 13 14:57:34 2005 From: mabrown-lartc at securepipe.com (Martin A. Brown) Date: Wed Jul 13 14:58:37 2005 Subject: =?iso-8859-1?Q?Re:_=5BLARTC=5D_HTB:_=BFhow_do_burst/cburst_work_exactly=3F?= In-Reply-To: <006901c5877b$6a2feca0$6501a8c0@FLO> References: <20050713001007.32521.qmail@web34412.mail.mud.yahoo.com> <001601c58764$d97f86e0$6501a8c0@FLO> <006901c5877b$6a2feca0$6501a8c0@FLO> Message-ID: : However I have a doubt on [1]. When packets arrive to an HTB : filter, it first checks to see if there are ctokens available. : How come if there are, it then checks for tokens instead of just : dequeuing at full speed. And if there are none: shouldn?t it THEN : check for tokens instead of discarding right away? I would like to quibble about terminology here. A filter selects a destination class (which contains a qdisc). Filters have absolutely nothing to do with tokens. Once a packet has been filtered, it enters its destination class. Roughly speaking, the class does the following: 1. check to see if there are tokens available && dequeue 2. check to see if there are ctokens available && dequeue 3. wait until tokens become available && goto 2 (1?) Once a packet is in an HTB class, the packet will not get discarded by the class until the class itself has a tremendous backlog and needs to drop the packet. An HTB class will delay the transmission of the packet until tokens are available. This is the core feature which provides the shaping capability. : Also, could you give me an advice or reference on the following? : I need a child class to allow passage to a video stream that I : KNOW has mean X kbps and seldom peaks of Y kbps and T seconds. : Would the best way be to just configure mean=X, ceil=Y? Or should : I configure mean=ceil=X and calculate a cburst that?ll allow : passage of the peaks? Or maybe a third option. I cannot recommend an optimal calculation method, though I would start with X kbps as the rate and Y kbps * T as the burst. After that, I'd increase rate and decrease burst until there was no choppiness in the transmitted video stream. Good luck, -Martin : ----- Original Message ----- From: "Martin A. Brown" : : To: "VideoIP" : Cc: "lartc" : Sent: Wednesday, July 13, 2005 2:14 AM : Subject: Re: [LARTC] HTB: ?how do burst/cburst work exactly? : : : : Hello, : : : I?ve read all the definitions of burst and cburst and I?ve tried : : playing with the parameters and graphing the output of the filter : : to see its effects, but STILL I can?t figure out how the : : parameters work exactly. : : : : Could anyone give a better explanation than the manpage? : : Have you tried Stef's site? He has a good page [0] that talks about : the various tests he did while experimenting with HTB burst and : cburst parameters? : : Some time ago, I took a stab at creating a visual representation [1] : of a hypothetical HTB configuration [2]. In order to understand : when cburst is used, look for the diamond-shaped boxes in the : diagram which talk about tokens and ctokens. : : Every HTB class has two buckets. : : rate bucket is of burst size, traffic uses tokens : ceil bucket is of cburst size, traffic uses ctokens : : My diagram may give you the framework to understand exactly how they : are used if it's still unclear to you, but Stef's site will give you : much better detail on the results of using burst and cburst. Of the : scenarios he describes, I like the results of Test 7 the best. The : only guideline that struck me after reading his results was to : prefer burst and cburst usage on parent classes. : : Good luck, : : -Martin : : [0] http://www.docum.org/docum.org/tests/htb/burst/ : [1] http://linux-ip.net/traffic-control/htb-class.png : [2] http://linux-ip.net/traffic-control/diagram.html : : -- Martin A. Brown --- Wonderfrog Enterprises --- martin@wonderfrog.net From jonathan.schmieg at triplex.de Wed Jul 13 16:25:51 2005 From: jonathan.schmieg at triplex.de (Jonathan Schmieg) Date: Wed Jul 13 16:26:22 2005 Subject: [LARTC] routing problems with two SDSL-connections Message-ID: <42D5246F.80303@triplex.de> Hello List, in our office we have two independant SDSL-connections. One of them is a flatrate, the other is a dedicated line to our webfarm. The goal is to route all the traffic to the webfarm through the dedicated line and all other traffic through the flatrate. The maschine has three nics: eth0: internal network eth1: webfarm eth2: flatrate Each connection uses its own router. It is possible to access the internet through both connections, for example with ping -I interface address. I wrote an script for the issue mentioned above: please take a look at the attachment The whole thing works great under Knoppix, but neither with gentoo, nor with debian sarge (I want to use debian sarge for the router). Just for testing I took another maschine with gentoo and there it also works. (same kernel-, same iptables-, same iproute2-versions and also the same nics) With tcpdump I can see that packets sent by a client from the internal network leave the gateway, the answer comes back but is not passed on the the client. I hope somebody has an idea how I could solve the problem, greetings, Jonathan Schmieg -------------- next part -------------- #!/bin/sh ## Variablen GATEWAY_DEF=X.X.X.25 GATEWAY_T2=Y.Y.Y.177 IP_T2=Y.Y.Y.180 IFACE_INT=eth0 ## Kernelparameter echo "1" > /proc/sys/net/ipv4/conf/all/forwarding ############ ## Status ## ############ if [ "$1" = "status" ] then echo "Default Route"\n ip route show echo "Spacenet Route"\n ip route show table 2 echo "Rules"\n ip rule show echo "Markierungen"\n iptables -t mangle -L ROUTING -v -x 2> /dev/null exit fi ########## ## Stop ## ########## iptables -t mangle -D PREROUTING -j ROUTING 2> /dev/null > /dev/null iptables -t mangle -D FORWARD -j ROUTING 2> /dev/null > /dev/null iptables -t mangle -F ROUTING 2> /dev/null > /dev/null iptables -t mangle -X ROUTING 2> /dev/null > /dev/null ip route del table 2 ip route del default via $GATEWAY_DEF ip rule del from $IP_T2 table 2 ip rule del fwmark 66 table 2 ip route flush cache if [ "$1" = "stop" ] then echo "Routing removed" exit fi ########### ## Start ## ########### ## 2. Tabelle anlegen ip route show table main | grep -Ev ^default | while read ROUTE ; do ip route add table 2 $ROUTE; done ip route add default via $GATEWAY_T2 table 2 ## Defaultgw setzen ip route add default via $GATEWAY_DEF ##Routing regeln setzen ip rule add from $IP_T2 table 2 ip route flush cache ip rule add fwmark 66 table 2 ##Iptables Tabelle anlegen iptables -t mangle -N ROUTING iptables -t mangle -I PREROUTING -j ROUTING iptables -t mangle -I FORWARD -j ROUTING ## Markieren Kleinwebs iptables -t mangle -A ROUTING -i $IFACE_INT -p all -d Y.Y.A.0/24 -j MARK --set-mark 66 ## Markieren KUNDE iptables -t mangle -A ROUTING -i $IFACE_INT -p all -d Y.Y.B.0/24 -j MARK --set-mark 66 ## Markieren Maintanace iptables -t mangle -A ROUTING -i $IFACE_INT -p all -d 192.168.100.0/24 -j MARK --set-mark 66 ## NAT setzen / passiert aber normal in der Firewall :) iptables -t nat -A POSTROUTING -o eth1 -s 192.168.10.0/24 -j SNAT --to-source Y.Y.Y.180 iptables -t nat -A POSTROUTING -o eth2 -s 192.168.10.0/24 -j SNAT --to-source X.X.X.30 From videoip at gmail.com Wed Jul 13 20:46:04 2005 From: videoip at gmail.com (VideoIP) Date: Wed Jul 13 20:46:38 2005 Subject: =?iso-8859-1?Q?Re:_=5BLARTC=5D_HTB:_=BFhow_do_burst/cburst_work_exactly=3F?= References: <20050713001007.32521.qmail@web34412.mail.mud.yahoo.com> <001601c58764$d97f86e0$6501a8c0@FLO> <006901c5877b$6a2feca0$6501a8c0@FLO> Message-ID: <00af01c587db$2ab1f360$6501a8c0@FLO> : : Also, could you give me an advice or reference on the following? : : I need a child class to allow passage to a video stream that I : : KNOW has mean X kbps and seldom peaks of Y kbps and T seconds. : : Would the best way be to just configure mean=X, ceil=Y? Or should : : I configure mean=ceil=X and calculate a cburst that?ll allow : : passage of the peaks? Or maybe a third option. : : I cannot recommend an optimal calculation method, though I would : start with X kbps as the rate and Y kbps * T as the burst. After : that, I'd increase rate and decrease burst until there was no : choppiness in the transmitted video stream. Would in that scenario ceil=rate or ceil=Y kbps? It seems to me that if ceil=rate, then there?s no use in having a burst bucket, or is there? Thanks again, Florencio From psihozefir at yahoo.com Wed Jul 13 22:21:34 2005 From: psihozefir at yahoo.com (panca sorin) Date: Wed Jul 13 22:21:39 2005 Subject: [LARTC] Bandwidth shaping and ISP's network peerings Message-ID: <20050713202134.53287.qmail@web32614.mail.mud.yahoo.com> Hello all! I have a small LAN at home and when someone starts to download (only one), interractive traffic (www, chat and online games) is impossible with standard kernel queues setup... So I started to shape. My ISP gives me a 512 kbits link to the Internet and a 100 Mbits link to some of the other big ISPs in my country. If I set the rate of the parent htb qdisc at 512 kbits, I will never use the MAN bandwidth from my network. If I set the rate of the parent htb qdisc at 100 Mbits, i cannot shape interractive traffic. Further, I would like to allocate for every station in the LAN a quantum of my Internet speed with ceiling but in MAN I want to have the full hardware speed if only one machine is connected, with any ceil. Any ideas would be VERY appreciated! I can't imagine any good setup to meet these constraints. __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From andy.furniss at dsl.pipex.com Thu Jul 14 00:15:28 2005 From: andy.furniss at dsl.pipex.com (Andy Furniss) Date: Thu Jul 14 00:15:14 2005 Subject: [LARTC] HTB Rate and Prio (continued) In-Reply-To: <200507131024.j6DAOodA022265@dhuumrelay0.mail.eu.uu.net> References: <200507131024.j6DAOodA022265@dhuumrelay0.mail.eu.uu.net> Message-ID: <42D59280.1030701@dsl.pipex.com> Gael Mauleon wrote: > Ok I tested the shaping on the SDSL line with netperf and > an host outside. > > Same script than before, I classify the packets into qdisc based on the > source address in netfilter and here are the result, that's with sfq. > I'm positive on the right traffic going to the right class. > > > TCP STREAM TEST to 81.57.243.113 (NORMAL) > Recv Send Send > Socket Socket Message Elapsed > Size Size Size Time Throughput > bytes bytes bytes secs. 10^3bits/sec > > 87380 8192 8192 60.00 282.90 > > > > TCP STREAM TEST to 81.57.243.113 (LOWPRIO) > Recv Send Send > Socket Socket Message Elapsed > Size Size Size Time Throughput > bytes bytes bytes secs. 10^3bits/sec > > 87380 8192 8192 61.00 390.00 Hmm I can't really think why this is happening, is it the same box that you did the lan test from? I think what I would do as the next test is turn off window scaling - echo 0 > /proc/sys/net/ipv4/tcp_window_scaling add 70k bfifos to the classes - you shouldn't drop any packets then. Repeat the test and tcpdump it. If you see packet loss then this could be the explanation. Andy. From andy.furniss at dsl.pipex.com Thu Jul 14 00:26:32 2005 From: andy.furniss at dsl.pipex.com (Andy Furniss) Date: Thu Jul 14 00:26:18 2005 Subject: [LARTC] HTB Rate and Prio (continued) In-Reply-To: <42D59280.1030701@dsl.pipex.com> References: <200507131024.j6DAOodA022265@dhuumrelay0.mail.eu.uu.net> <42D59280.1030701@dsl.pipex.com> Message-ID: <42D59518.8080903@dsl.pipex.com> Andy Furniss wrote: > add 70k bfifos to the classes - you shouldn't drop any packets then. Maybe 100k just to be safe 70k may be a bit close once you take into account the headers. Andy. From andy.furniss at dsl.pipex.com Thu Jul 14 00:47:49 2005 From: andy.furniss at dsl.pipex.com (Andy Furniss) Date: Thu Jul 14 00:47:35 2005 Subject: [LARTC] HTB Rate and Prio (continued) In-Reply-To: <200507121207.j6CC7YrA027308@dhuumrelay1.mail.eu.uu.net> References: <200507121207.j6CC7YrA027308@dhuumrelay1.mail.eu.uu.net> Message-ID: <42D59A15.2000807@dsl.pipex.com> Gael Mauleon wrote: > I quite don't understand the concept of putting the rate of the line lower > than it's true value, can you explain me this and do the excess bandwith is > lost ? What is a good value for a 2m line (SDSL) ? It depends on what rate you are really synced at and what extra overheads/encapsulation your sdsl line has. It may be a bit different for sdsl - I only know adsl, but as an example, for me, an empty ack which htb will see as 40 bytes (ignoring timestamps/sacks) will actually use 2 atm cells = 106 bytes on my line a 1500 byte ip packet will use 32 cells = 1696 bytes. Which means that if I tweaked by experementing with my rate using bulk traffic and arrived at some figure which seemed OK things could still go overlimits when the traffic consists of alot of small packets. There are patches to work things out per packet and a thesis giving various overheads at http://www.adsl-optimizer.dk/ but for now just back off - 1800 is probably OK for netperf tests. I am talking about egress here - if you are going to shape ingress aswell you need to back off even more as you are trying to shape from the wrong end of the bottleneck (which can't be done perfectly anyway) to build up queues to shape with. Andy. From jeff at cowart.net Thu Jul 14 07:54:33 2005 From: jeff at cowart.net (Jefferson Cowart) Date: Thu Jul 14 07:54:42 2005 Subject: [LARTC] Losing Packets after a DNAT in prerouting Message-ID: <20050714055434.2AED310082@P450.internal.cowart.net> I'm trying to setup some DNAT and the packets seem to be disappearing after the PREROUTING step. The packets are coming in eth2 (both LOG targets in iptables and tcpdump confirm this). They are then DNATed to an IP that should cause them to go out eth3. However I never see them go out that interface. I have tried putting LOG rules into the FORWARD chain with no success. I'm pretty sure the packet isn't hitting a DROP rule as all my DROP rules have a LOG rule directly in front of them. Any idea how to track down the missing packets? ---------------- Thanks Jefferson Cowart Jeff@cowart.net From alvarolmmotta at gmail.com Thu Jul 14 15:14:37 2005 From: alvarolmmotta at gmail.com (Alvaro Motta) Date: Thu Jul 14 15:15:19 2005 Subject: [LARTC] Use of qcdisc+htb Message-ID: <3941d81c05071406147a1107e6@mail.gmail.com> Hi folks. This message may be a bit verbose and not as techie as the ones I've seen in this list, but describing the scenario will save a lot of messages. Scenario: A building with 17 floors, each floor with 24 offices (totals 408 offices) connected to the backbone through a border switch (1 vlan for each office). The offices can lease bandwidth of 64k, 128k, 256k, 512k, 1M and 2Mbps, according to their needs. We have 3 links to the internet, 2Mbps each. Currently, we use cisco's bbsm to handle the task of allocating the leased bandwidth for the vlans. Problem: bbsm never performed as expected. It freezes, disconnect users, reboots for itself leaving us in an awkward situation. After wasting precious time with cisco and it's product, we decided to move to another solution and since the boss is an enthusiast of open source software, we decided to go for a linux based solution. We found that queueing discipline may be the solution. The question: are we correct, I mean is qdisc+htb the right thing to be used in such a scenario? Has anyone out there seen a linux box handling so many networks? As I go deeper in this subject, I will come to share my thoughts and doubts with you guys. Hope to hear from you. Have a good one. AL From rsenykoff at harrislogic.com Thu Jul 14 16:27:55 2005 From: rsenykoff at harrislogic.com (rsenykoff@harrislogic.com) Date: Thu Jul 14 16:33:13 2005 Subject: [LARTC] Use of qcdisc+htb In-Reply-To: <3941d81c05071406147a1107e6@mail.gmail.com> Message-ID: > Scenario: > > A building with 17 floors, each floor with 24 offices (totals 408 > offices) connected to the backbone through a border switch (1 vlan for > each office). The offices can lease bandwidth of 64k, 128k, 256k, > 512k, 1M and 2Mbps, according to their needs. We have 3 links to the > internet, 2Mbps each. Currently, we use cisco's bbsm to handle the > task of allocating the leased bandwidth for the vlans. > Problem: bbsm never performed as expected. It freezes, disconnect > users, reboots for itself leaving us in an awkward situation. After > wasting precious time with cisco and it's product, we decided to move > to another solution and since the boss is an enthusiast of open source > software, we decided to go for a linux based solution. > We found that queueing discipline may be the solution. > The question: are we correct, I mean is qdisc+htb the right thing to > be used in such a scenario? Has anyone out there seen a linux box > handling so many networks? > As I go deeper in this subject, I will come to share my thoughts and > doubts with you guys. > > Hope to hear from you. Have a good one. > > > AL > _______________________________________________ LARTC is being used in University environments with even higher bandwidth usage, so yes, you can do it. The multiple links may complicate things if you don't want to explicitely assign a VLAN to a link. While balancing across the links is not that difficult, combining the balancing with shaping to guarantee x amount of bandwidth could be difficult. Do you want to simply use HTB to limit rates for each office? If so, this should work very well. You'll just need to build the routes to specify which network goes to which internet connection. Or, you could use a separate box for each internet connection if you break out the VLANs earlier. FYI, I'm using a Pentium Pro 200 to shape traffic on a 3Mb/3Mb connection using CBQ for VoIP, Video, Citrix, and Bulk. HTH, -Ron From surda at shurdix.com Thu Jul 14 17:24:00 2005 From: surda at shurdix.com (Peter Surda) Date: Thu Jul 14 17:24:26 2005 Subject: [LARTC] Use of qcdisc+htb In-Reply-To: <3941d81c05071406147a1107e6@mail.gmail.com> Message-ID: <20056141724026735@mail.routehat.org> On Thu, 14 Jul 2005 10:14:37 -0300 Alvaro Motta wrote: >Hi folks. Hi, >This message may be a bit verbose and not as techie as the ones I've >seen in this list, but describing the scenario will save a lot of >messages. >Problem: bbsm never performed as expected. It freezes, disconnect >users, reboots for itself leaving us in an awkward situation. First of all although I personally also think cisco's are not very good for other things than routing, experience tells me that problems are often caused by misconfiguration. I'd check out with an experienced cisco professional. >The question: are we correct, I mean is qdisc+htb the right thing to >be used in such a scenario? Has anyone out there seen a linux box >handling so many networks? I expect you want to use many htb classes with the same parent. I have seen a reports that this causes problems when you really have a large class number (several hundred), but again, this may have been caused by misconfiguration. If you adapt your requirements however (every IP is handled equally), you can use WRR, it has been proven to work without problems under even larger number of clients (1400) and bandwidth (16Mbit). For an example see here: http://mailman.ds9a.nl/pipermail/lartc/2005q2/016500.html If you don't want to adapt, once upon a time I wrote a management tool for an ISP with requirements similar to yours. Although I tuned it for performance and it seems to work well, as far as I know there are only a couple of dozen users, I don't know how it would behave if it was used with several hunderd users. >AL Yours sincerely, Peter From demiurg at ti.com Thu Jul 14 17:49:32 2005 From: demiurg at ti.com (Alexander Sirotkin) Date: Thu Jul 14 17:49:48 2005 Subject: [LARTC] QoS on receive Message-ID: <42D6898C.4050502@ti.com> It appears that while Linux has plenty of traffic shaping mechanism on transmit, there is nothing on receive side. While generally it does make sense since transmit is more CPU intensive operation, after all receive also consumes CPU cycles. It is clear that it's best to drop the packet as soon as possible, i.e. on receive, if possible - by the driver itself. It may not be feasible in general case, but I can think of a couple of scenarios when it does make sense. Any ideas ? Maybe there is some similar QoS mechanism that I'm not aware of ? -- Alexander Sirotkin SW Engineer Texas Instruments Broadband Communications Israel (BCIL) Tel: +972-9-9706587 ________________________________________________________________________ "Those who do not understand Unix are condemned to reinvent it, poorly." -- Henry Spencer From surda at shurdix.com Thu Jul 14 19:02:45 2005 From: surda at shurdix.com (Peter Surda) Date: Thu Jul 14 19:03:14 2005 Subject: [LARTC] Use of qcdisc+htb In-Reply-To: <20056141724026735@mail.routehat.org> Message-ID: <20056141924526735@mail.routehat.org> On Thu, 14 Jul 2005 17:24:00 +0200 Peter Surda wrote: >If you don't want to adapt, once upon a time I wrote a management tool for an >ISP with requirements similar to yours. Although I tuned it for performance and >it seems to work well, as far as I know there are only a couple of dozen users, >I don't know how it would behave if it was used with several hunderd users. Update: I obtained some data from the mentioned ISP: - Backbone: 16Mbit - average transfer 700kB/s (5600kbit/s) - about 20 users - CPU Celeron 333 - no performance problems noticeable The only unanswered question remains the user count (20 vs 400 is not really comparable). Yours sincerely, Peter From tdi at pozman.pl Thu Jul 14 22:00:30 2005 From: tdi at pozman.pl (Dariusz Dwornikowski) Date: Thu Jul 14 22:00:44 2005 Subject: [LARTC] QOS problem -ng In-Reply-To: <42D49426.7040605@netshadow.at> References: <20050712215936.75d9dcb2@tdi.pozman.pl> <42D49426.7040605@netshadow.at> Message-ID: <20050714220030.17e44aaf@tdi.pozman.pl> ok i read everythong and now understand much more.. the problem is that my boss told me to erase whole running server, because he wants to run tests on hardware with windows... (i work with idiot). he is so stubborn that he does not understand that this is qdisc issue. my idea is : to give htb with imq on interfaces to globally cut bws. i would like an advice what is the best solution for network with many access points ? is the incoming bw shaped on internal eth ? is it necessary to mark packets when shaping outgoing bw for nated nets ? -- *Dariusz 'tdi' Dwornikowski | Gentoo | admin at pozman.pl | *[JID]:tdi@gentoo.pl|[gg]:2266034|[IRC]:#gentoo-pl@freenode | *[MAIL]:tdi@pozman.pl|[WWW]:www.tdi.pozman.pl | *Serwery,administracja,webapps - www.ProAdmin.com.pl | *Fingerprint:43E21CC46DAFD2F754E91547D59B39F56AAA4B5F | -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mailman.ds9a.nl/pipermail/lartc/attachments/20050714/887a05f9/attachment.bin From o7sh at actcom.net.il Thu Jul 14 23:32:53 2005 From: o7sh at actcom.net.il (Ori Shiloh) Date: Thu Jul 14 23:32:58 2005 Subject: [LARTC] Bandwidth shaping and ISP's network peerings In-Reply-To: <20050713202134.53287.qmail@web32614.mail.mud.yahoo.com> References: <20050713202134.53287.qmail@web32614.mail.mud.yahoo.com> Message-ID: <200507150032.53808.o7sh@actcom.net.il> Hi there... I have an idea for you, just don't ask me how to implement it. 1. bring up some virtual interface, I'm almost sure linux has some way of doing it. this interface should output data to your real interface. 2. try to route all MAN traffic trough this interface. you'll need to know the destination addresses of this network. 3. shape the virtual interface.. I hope it's possible.. I'll be glad to know If you made it. Good luck. > Hello all! I have a small LAN at home and when someone > starts to download (only one), interractive traffic > (www, chat and online games) is impossible with > standard kernel queues setup... So I started to shape. > My ISP gives me a 512 kbits link to the Internet and a > 100 Mbits link to some of the other big ISPs in my > country. If I set the rate of the parent htb qdisc at > 512 kbits, I will never use the MAN bandwidth from my > network. If I set the rate of the parent htb qdisc at > 100 Mbits, i cannot shape interractive traffic. > Further, I would like to allocate for every station in > the LAN a quantum of my Internet speed with ceiling > but in MAN I want to have the full hardware speed if > only one machine is connected, with any ceil. > Any ideas would be VERY appreciated! I can't imagine > any good setup to meet these constraints. > From jody.shumaker at gmail.com Fri Jul 15 01:13:24 2005 From: jody.shumaker at gmail.com (Jody Shumaker) Date: Fri Jul 15 01:13:27 2005 Subject: [LARTC] Bandwidth shaping and ISP's network peerings In-Reply-To: <200507150032.53808.o7sh@actcom.net.il> References: <20050713202134.53287.qmail@web32614.mail.mud.yahoo.com> <200507150032.53808.o7sh@actcom.net.il> Message-ID: <42D6F194.5030809@gmail.com> That sounds like an overly complicated way to do it. I would just create a 512kbit class with subclasses for the internet traffic, and route all MAN traffic into a 100mbit class. Should be some way to know which ip's will go to the MAN. Creating a virtual interface makes little sense here, since no matter what you'll have to filter out the MAN traffic. - Jody Ori Shiloh wrote: >Hi there... >I have an idea for you, just don't ask me how to implement it. > >1. bring up some virtual interface, I'm almost sure linux has some way of >doing it. this interface should output data to your real interface. >2. try to route all MAN traffic trough this interface. you'll need to know the >destination addresses of this network. >3. shape the virtual interface.. > >I hope it's possible.. I'll be glad to know If you made it. >Good luck. > > > >>Hello all! I have a small LAN at home and when someone >>starts to download (only one), interractive traffic >>(www, chat and online games) is impossible with >>standard kernel queues setup... So I started to shape. >>My ISP gives me a 512 kbits link to the Internet and a >>100 Mbits link to some of the other big ISPs in my >>country. If I set the rate of the parent htb qdisc at >>512 kbits, I will never use the MAN bandwidth from my >>network. If I set the rate of the parent htb qdisc at >>100 Mbits, i cannot shape interractive traffic. >>Further, I would like to allocate for every station in >>the LAN a quantum of my Internet speed with ceiling >>but in MAN I want to have the full hardware speed if >>only one machine is connected, with any ceil. >>Any ideas would be VERY appreciated! I can't imagine >>any good setup to meet these constraints. >> >> >> >_______________________________________________ >LARTC mailing list >LARTC@mailman.ds9a.nl >http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > > > From pramod at atheros.com Fri Jul 15 05:29:54 2005 From: pramod at atheros.com (pramod) Date: Fri Jul 15 05:30:07 2005 Subject: [LARTC] QoS on receive In-Reply-To: <42D6898C.4050502@ti.com> References: <42D6898C.4050502@ti.com> Message-ID: <42D72DB2.3000602@atheros.com> Dropping of packets on the receive side can be done bu IPTABLES.. thanks pramod Alexander Sirotkin wrote: > It appears that while Linux has plenty of traffic shaping mechanism on > transmit, there is nothing on receive side. > While generally it does make sense since transmit is more CPU > intensive operation, after all receive also > consumes CPU cycles. It is clear that it's best to drop the packet as > soon as possible, i.e. on receive, if possible - > by the driver itself. It may not be feasible in general case, but I > can think of a couple of scenarios when it does > make sense. > > Any ideas ? > Maybe there is some similar QoS mechanism that I'm not aware of ? > From pramod at atheros.com Fri Jul 15 05:30:56 2005 From: pramod at atheros.com (pramod) Date: Fri Jul 15 05:31:08 2005 Subject: [LARTC] Losing Packets after a DNAT in prerouting In-Reply-To: <20050714055434.2AED310082@P450.internal.cowart.net> References: <20050714055434.2AED310082@P450.internal.cowart.net> Message-ID: <42D72DF0.20708@atheros.com> Can u attach ur Rules file.. thanks pramod From admin at premiere-heure.fr Fri Jul 15 10:24:45 2005 From: admin at premiere-heure.fr (Gael Mauleon) Date: Fri Jul 15 10:24:09 2005 Subject: [LARTC] HTB Rate and Prio (continued) In-Reply-To: <42D59A15.2000807@dsl.pipex.com> Message-ID: <200507150824.j6F8O01R015026@dhuumrelay0.mail.eu.uu.net> > It depends on what rate you are really synced at and what extra > overheads/encapsulation your sdsl line has. > > It may be a bit different for sdsl - I only know adsl, but as an > example, for me, an empty ack which htb will see as 40 bytes (ignoring > timestamps/sacks) will actually use 2 atm cells = 106 bytes on my line a > 1500 byte ip packet will use 32 cells = 1696 bytes. > > Which means that if I tweaked by experementing with my rate using bulk > traffic and arrived at some figure which seemed OK things could still go > overlimits when the traffic consists of alot of small packets. > > There are patches to work things out per packet and a thesis giving > various overheads at > > http://www.adsl-optimizer.dk/ > > but for now just back off - 1800 is probably OK for netperf tests. > > I am talking about egress here - if you are going to shape ingress > aswell you need to back off even more as you are trying to shape from > the wrong end of the bottleneck (which can't be done perfectly anyway) > to build up queues to shape with. > > Andy. Thanks for all this tips, before I try all you said, I was going to test my line at 500kbits rate/ceil just to put one variable out of the equation, and to my surprise the shaping is working good with 500kbits, I did more tests an dit is working good up to something between 1000 and 1200 kbits, at 1000 it is working, at 1200 the shaping is gone and the problem is back... What I don't understand here, is the fact I can go up to 1800 or higher with iptraf and some netperfs tests... So I'm really lost now, what can I do ?? I can't only shape 1000 and loose 800 kbits, I really need some advice here on what can be done, I'm going mad :) Thanks Gael. From JNeave at spursolutions.com Fri Jul 15 10:36:07 2005 From: JNeave at spursolutions.com (James Neave) Date: Fri Jul 15 10:36:19 2005 Subject: [LARTC] Attatchment test Message-ID: Hi, I'm about to tell a story best told through diagrams. So I'm testing whether I can attach PNGs to an email to this list. Please ignore this email! ^^ Ta, James. The information in this email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this email by anyone else is unauthorised. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it is prohibited and may be unlawful. The contents of an attachment to this email may contain software viruses that could damage your own computer systems. Whilst The Spur Group of Companies has taken every precaution to minimise the risk, we cannot accept liability for any damage that you sustain as a result of software viruses. -------------- next part -------------- A non-text attachment was scrubbed... Name: Test.png Type: image/png Size: 317 bytes Desc: Test.png Url : http://mailman.ds9a.nl/pipermail/lartc/attachments/20050715/ccc6a91f/Test.png From pstaszewski at artcom.pl Fri Jul 15 11:55:34 2005 From: pstaszewski at artcom.pl (=?Windows-1250?Q?Pawe=B3=20Staszewski?=) Date: Fri Jul 15 11:59:10 2005 Subject: [LARTC] Use of qcdisc+htb Message-ID: Hello I have 40Mbit/s internet uplink Average transfer 25Mbit/s - 3957 users On machine: - CPU 3.2GHz P4 +HT enabled - 2GB RAM - Intel GB cards Machine is doing nat/dnat for each user so there is 3957 DNAT/SNAT ip pairs (private to public and vice versa) Machine load is 0.1 avg And no any problems :) On Thu, 14 Jul 2005 17:24:00 +0200 Peter Surda wrote: >If you don't want to adapt, once upon a time I wrote a management tool for an >ISP with requirements similar to yours. Although I tuned it for performance and >it seems to work well, as far as I know there are only a couple of dozen users, >I don't know how it would behave if it was used with several hunderd users. Update: I obtained some data from the mentioned ISP: - Backbone: 16Mbit - average transfer 700kB/s (5600kbit/s) - about 20 users - CPU Celeron 333 - no performance problems noticeable The only unanswered question remains the user count (20 vs 400 is not really comparable). Yours sincerely, Peter From admin at premiere-heure.fr Fri Jul 15 12:26:52 2005 From: admin at premiere-heure.fr (Gael Mauleon) Date: Fri Jul 15 12:26:16 2005 Subject: [LARTC] HTB Rate and Prio (continued) In-Reply-To: <200507150824.j6F8O01R015026@dhuumrelay0.mail.eu.uu.net> Message-ID: <200507151026.j6FAQ7FQ022258@dhuumrelay0.mail.eu.uu.net> Actually doing some more tests with all traffic classified can reach 1700 kbits as rate/ceil, at this rate I must put the prio to have some good results. Doing more tests, I didn't know HTB was so sensitive to the max rate/ceil... I'll post later on. From michele at protocol.it Fri Jul 15 12:49:02 2005 From: michele at protocol.it (Michele Cerioni) Date: Fri Jul 15 12:49:17 2005 Subject: [LARTC] Problems with table Message-ID: <42D7949E.9090704@protocol.it> hi, I have problems with tables. I installed the last iproute2: iproute2-2.4.7-now-ss010824.tar.gz on Linux 2.4.26 (slackware 9.1). I want to use 2 adsl on this server. I run this command: #echo 201 routeradsl2 >> /etc/iproute2/rt_tables (only one time); then #ip route add 82.189.148.240 dev eth1 src 82.189.148.243 table routeradsl2 at this point I run: #ip route show table routeradsl2 # the table is routeradsl2 empty then I run #ip route show table main 82.189.148.240 dev eth1 scope link src 82.189.148.243 82.189.148.240/29 dev eth1 proto kernel scope link src 82.189.148.243 194.243.125.0/26 dev eth0 proto kernel scope link src 194.243.125.10 194.243.125.0/24 dev eth0 proto kernel scope link src 194.243.125.4 127.0.0.0/8 dev lo scope link default via 194.243.125.1 dev eth0 metric 1 The line 82.189.148.240 dev eth1 scope link src 82.189.148.243 was added to table main instead table routeradls2. Why? MIchele From michele at protocol.it Fri Jul 15 13:06:53 2005 From: michele at protocol.it (Michele Cerioni) Date: Fri Jul 15 13:07:02 2005 Subject: [LARTC] Problems with table In-Reply-To: <42D7949E.9090704@protocol.it> References: <42D7949E.9090704@protocol.it> Message-ID: <42D798CD.5080508@protocol.it> Sorry. My kernel was not able to perform policy routing in the correct way. Michele Michele Cerioni wrote: > hi, > I have problems with tables. > I installed the last iproute2: iproute2-2.4.7-now-ss010824.tar.gz on > Linux 2.4.26 (slackware 9.1). > > I want to use 2 adsl on this server. > I run this command: > #echo 201 routeradsl2 >> /etc/iproute2/rt_tables (only one time); > then > > #ip route add 82.189.148.240 dev eth1 src 82.189.148.243 table routeradsl2 > > at this point I run: > > #ip route show table routeradsl2 > # > > the table is routeradsl2 empty > > then I run > #ip route show table main > 82.189.148.240 dev eth1 scope link src 82.189.148.243 > 82.189.148.240/29 dev eth1 proto kernel scope link src 82.189.148.243 > 194.243.125.0/26 dev eth0 proto kernel scope link src 194.243.125.10 > 194.243.125.0/24 dev eth0 proto kernel scope link src 194.243.125.4 > 127.0.0.0/8 dev lo scope link > default via 194.243.125.1 dev eth0 metric 1 > > The line 82.189.148.240 dev eth1 scope link src 82.189.148.243 was > added to table main instead table routeradls2. > > Why? > > MIchele > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > From JNeave at spursolutions.com Fri Jul 15 13:07:58 2005 From: JNeave at spursolutions.com (James Neave) Date: Fri Jul 15 13:08:14 2005 Subject: [LARTC] FW: LARTC Chapter 4.2, variation on a theme. Message-ID: Hi, I'm building a network similar to that seen in 4.2 of the LARTC Howto. There is a diagram of this attached to this mail. Addendum to diagram: AlexRouter br0 = 192.168.58.1 eth0 = dhcpcd DaveRouter br0 = 192.168.58.2 eth0 = dhcpcd But we've run into some problems when actually implementing the routing for multiple uplinks. The difference between my network and the LARTC example is instead of having one router with two modems I have two routers with one modem each. AlexRouter and DaveRouter. They run Bering-uClibc 2.x off of fd0. A wired/wireless network connects the two together. 192.168.58.0/24. AlexRouter is the default route/DNS server/DHCP server for every host on the network. It gets its DNS servers from dhcpcd. They way I figure it, Provider2 in the example is (in my case) actually DaveRouter. With that in mind, these are the figures I came up with for settings up the routes. These are all from the perspective of AlexRouter. $IF1 = eth0 $IF2 = br0 $IP1 = 80.blah.blah.blah (can't remember my real address) $IP2 = 192.168.58.1 $P1 = $IP1 *DON'T KNOW IF THIS IS RIGHT, DON'T KNOW HOW TO FIND MY PROVIDERS GATEWAY* $P2 = 192.168.58.2 (DaveRouter) $P1_NET = 80.blah.blah.0/24 (got $IP1 and $P1_NET from ip route show) $P2_NET = 192.168.58.0/24 $P0_NET = 192.168.58.0/24 $IF0 = br0 If I set up all the routes using those values, test browsing around is flakey. Some pages load, some don't (one connection working, one not?) I *can* use one connection *OR* the other connection. But only if I manually re-write /etc/resolv.conf to contain the correct DNS servers for the provider used. One ISP is Demon, the other is BT. They won't let each other use their DNS servers. Also, I had duplicate returns from ping. Apart from that, I'm not sure where I go with diagnosis. Does anybody have any idea what's going on? Thanks, James. The information in this email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this email by anyone else is unauthorised. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it is prohibited and may be unlawful. The contents of an attachment to this email may contain software viruses that could damage your own computer systems. Whilst The Spur Group of Companies has taken every precaution to minimise the risk, we cannot accept liability for any damage that you sustain as a result of software viruses. -------------- next part -------------- A non-text attachment was scrubbed... Name: StonechatNetSm.png Type: image/png Size: 18948 bytes Desc: StonechatNetSm.png Url : http://mailman.ds9a.nl/pipermail/lartc/attachments/20050715/b905b7b2/StonechatNetSm-0001.png From greve at fsfeurope.org Fri Jul 15 13:23:35 2005 From: greve at fsfeurope.org (Georg C. F. Greve) Date: Fri Jul 15 13:23:36 2005 Subject: [LARTC] Problems setting up nested qdisc, feedback to LARTC HOWTO Message-ID: Hi all, based on the information in the "Linux Advanced Routing & Traffic Control HOWTO", I was trying to set up traffic shaping on my firewall. While I found the HOWTO very useful, in the process I ran into some problems that I did not forsee: According to the HOWTO it seems that it should have worked, even after spending some time going through the sections looking for answers, the problem was not obvious to me. So I would appreciate if you could tell me where my mistake was and also maybe add a bit of information to the HOWTO to help others fall into the same traps that I fell into. :-) Here is what I wanted my ideal solution to look like: A strong priority of traffic, where parts of the upstream should be guaranteed rate for some traffic, the rest should be given to normal traffic and any "leftovers" to BULK traffic, which is allowed to starve for a while. Also, connection handshake and such very short, time critical things should get absolute priority over everything else. So this is what I ideally wanted to set up: 1: PRIO QDISC (4 Bands), DEFAULT: ALL TO BAND 3 (2 in priomap) 1:1 -> SFQ, handle 10: for priority communication (connection handshake & co) 1:2 -> HTB, handle 20: limited to Xk for different kinds of guaranteed rates that can "borrow" from each other, but never more than the maximum -- so it cannot saturate the link fully. 20:1 -> SFQ, handle 100: 20:2 -> SFQ, handle 200: 20:3 -> SFQ, handle 300: 20:4 -> SFQ, handle 400: [...] 1:3 -> PRIO QDISC (default), handle 30: for all "normal"/unclassified traffic, TOS splitting only 30:1 (BAND 1) 30:2 (BAND 2) 30:3 (BAND 3) 1:4 -> PFIFO, handle 40: "starvation bitbucket" gets what is left, can starve at times The setup was apparently successful, tc does not complain and displays the structure: without any CLASSIFY targets, all traffic goes to 1:3 and is split in the three bands properly. Looks good. I can also add CLASSIFY for 1:1 and 1:4, which seem to fall into the SFQ/PFIFO buckets underneath, both look good. The problem starts with the HTB embedded in 1:2 as 20: -- I can never CLASSIFY traffic for it properly, the only way to get ANY traffic into it is to CLASSIFY for 1:2, but that puts all into its default bucket, which defeats its purpose. Neither 20:1 nor 100:0 would put traffic into its first bucket. Such classification is simply ignored as far as I can tell. This is the problem that I ultimately did not find a way to solve, the "indirect" approach of using the MARK target and tc filters also did not work -- it shows the exact same result. I currently run another approach to this, which I am not quite as happy with, but which works for now -- but would still like to know: * WHY was the 20:1 or 100:0 CLASSIFY not successful? Nothing in the documentation seemed to indicate that it should fail. * HOW could it have been made to work? * WHAT kind of information was I lacking? or, in short: WHAT did I do wrong? I'd be grateful to find an answer and think it might help to then find a way to add that answer into the LARTC HOWTO. Regards, Georg -- Georg C. F. Greve Free Software Foundation Europe (http://fsfeurope.org) Join the Fellowship and protect your freedom! (http://www.fsfe.org) -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 309 bytes Desc: not available Url : http://mailman.ds9a.nl/pipermail/lartc/attachments/20050715/1615ada7/attachment.bin From surda at shurdix.com Fri Jul 15 14:16:33 2005 From: surda at shurdix.com (Peter Surda) Date: Fri Jul 15 14:16:59 2005 Subject: [LARTC] Use of qcdisc+htb In-Reply-To: Message-ID: <200561514163326735@mail.routehat.org> On Fri, 15 Jul 2005 11:55:34 +0200 "Pawe? Staszewski" wrote: >Hello Hello >I have 40Mbit/s internet uplink >Average transfer 25Mbit/s >- 3957 users [cut] Well, the question is what kind of TC-setup you have. Do you have a separate HTB class for every user? Yours sincerely, Peter From cnikitiuk at sharpercards.com Fri Jul 15 18:23:40 2005 From: cnikitiuk at sharpercards.com (Cameron Nikitiuk) Date: Fri Jul 15 18:23:27 2005 Subject: [LARTC] Goodbye! Message-ID: <200507151631.j6FGVCi03111@ns1.sharpercards.com> I am unsubscribing from the list. I asked for help at least twice for an issue and only one person even offered to try and help if they could. I sent them the details directly and did not receive anything back. I realize this is a community effort and that I am not guaranteed an answer when I submit a question, but to not even receive an "RTFM" just doesn't leave me feeling very positive about the value of the mailing list. Regards, Cameron From sylvain at 2001-space-odyssey.net Fri Jul 15 18:56:14 2005 From: sylvain at 2001-space-odyssey.net (Sylvain Bertrand) Date: Fri Jul 15 18:56:22 2005 Subject: [LARTC] Goodbye! In-Reply-To: <200507151631.j6FGVCi03111@ns1.sharpercards.com> References: <200507151631.j6FGVCi03111@ns1.sharpercards.com> Message-ID: <31250.213.245.32.130.1121446574.squirrel@webmail.2001-space-odyssey.net> I did not see your question, but there are so many topics like "load balancing between n interfaces" that sometimes I personnally don't want to bother answering for people who do not search at all. Also, please keep in mind that this is not a hotline and that nobody has a obligation to reply. Regards, Sylvain On Ven 15 juillet 2005 18:23, Cameron Nikitiuk a ?crit : > I am unsubscribing from the list. > > I asked for help at least twice for an issue and only one person even > offered to try and help if they could. I sent them the details directly > and > did not receive anything back. > > I realize this is a community effort and that I am not guaranteed an > answer > when I submit a question, but to not even receive an "RTFM" just doesn't > leave me feeling very positive about the value of the mailing list. > > Regards, > > Cameron > > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > From ji.li3 at hp.com Fri Jul 15 19:07:52 2005 From: ji.li3 at hp.com (Li, Ji) Date: Fri Jul 15 19:08:04 2005 Subject: [LARTC] Goodbye! Message-ID: <628BFCE8B64706469FE4D4852CEC953706CBD1B7@tayexc14.americas.cpqcorp.net> I am very new in this area, and I asked several questions here. For most of them, I got good replies; for some of them I didn't get any reply (maybe too stupid questions). I think this list is still useful, and no one has obligation to help us anyway. Good luck, -Ji -----Original Message----- From: lartc-bounces@mailman.ds9a.nl [mailto:lartc-bounces@mailman.ds9a.nl] On Behalf Of Cameron Nikitiuk Sent: Friday, July 15, 2005 12:24 PM To: LARTC Mailing List Subject: [LARTC] Goodbye! I am unsubscribing from the list. I asked for help at least twice for an issue and only one person even offered to try and help if they could. I sent them the details directly and did not receive anything back. I realize this is a community effort and that I am not guaranteed an answer when I submit a question, but to not even receive an "RTFM" just doesn't leave me feeling very positive about the value of the mailing list. Regards, Cameron _______________________________________________ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc From jarod125 at yahoo.com Fri Jul 15 20:15:39 2005 From: jarod125 at yahoo.com (Gabriel) Date: Fri Jul 15 20:15:46 2005 Subject: [LARTC] gre tunnel between networks with same subnet In-Reply-To: <20050708194433.14783.qmail@web60915.mail.yahoo.com> Message-ID: <20050715181539.89918.qmail@web60917.mail.yahoo.com> Ok, so I tried the bridging thing. I tried to bridge eth1 and tun0 on A, but apparently linux can't bridge tunnel interfaces (I got an error saying invalid argument when I issued 'brctl addif br0 tun0'). I was told to try using the vtun interface, so I'll dig into that for now. --- Gabriel wrote: > /-----------------------\ > | | > |eth0 |eth0 > |-------| |-------| > | |eth1 eth1 | | > -------- A |____ _______| B |----- > | | \ / | | > --------| | | --------| > | | > | | > ----------- > |___________| > switch > > What you see above is my setup. Box A is connected to > Box B through a switch. Box A is connected to the > Internet through eth0, same with Box B. The link that > goes through the switch is not very reliable, so I > want to connect the two boxes using their Internet > link via a gre tunnel. The problem is that the boxes > are on the same subnet (and I can't change that). I've > read about proxy arp, about bridging, but things are > still confused. Here are some numbers: eth1 on Box A > is 192.168.1.1/24, eth1 on Box B is 192.168.1.31/24. > On Box B there are 4 NICs, 3 of them (including eth1) > are bridged, with the bridge interface being br0 > (192.168.1.31 is actually assigned to br0, not eth1). > I've read the lartc howto, so I created a tun0 > interface on both boxes: ip tunnel add tun0 mode gre > remote remote_ip_here local local_ip_here ttl 255; ip > link set tun0 up. The problem is what do I do from > here? Do I bridge tun0 and eth1 on Box A and add tun0 > to br0 on Box B? Or do I just enable proxy_arp for > eth1 and tun0 on Box A and for br0 and tun0 on B? Are > there any routes neccesary (my guess is no, but I'm > not very sure)? And about proxy_arp: what do I have to > do to turn it on, just set > /proc/sys/net/ipv4/conf//proxy_arp to 1 and > that's it? One last thing: > http://leaf.sourceforge.net/doc/howto/proxyarp.html#id2805973 > says proxy-arp is not bridging (agreed) so DO NOT > CONFIGURE BRIDGE OPTIONS!!! Does this mean using > bridging and doing proxy-arp on the same box is not > possible? > > Thanks. > > (hope the ascii art comes out well) > > __________________________________________________ > Do You Yahoo!? > Tired of spam? Yahoo! Mail has the best spam protection > around > http://mail.yahoo.com > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > __________________________________ Yahoo! Mail Stay connected, organized, and protected. Take the tour: http://tour.mail.yahoo.com/mailtour.html From jody.shumaker at gmail.com Fri Jul 15 20:46:24 2005 From: jody.shumaker at gmail.com (Jody Shumaker) Date: Fri Jul 15 20:47:19 2005 Subject: [LARTC] Problems setting up nested qdisc, feedback to LARTC HOWTO In-Reply-To: References: Message-ID: <2af4364905071511466616a94@mail.gmail.com> Have you tried not using classify but instead using tc filters? Maybe this is a limitation with iptables classify. Try using your classify to put things into 20: and then use tc filters attached to 20: to split into the htb subclasses? I never used classify much and have always used tc filter instead, and i have done a setup similar to this with success. - Jody On 7/15/05, Georg C. F. Greve wrote: > Hi all, > > based on the information in the "Linux Advanced Routing & Traffic > Control HOWTO", I was trying to set up traffic shaping on my firewall. > > While I found the HOWTO very useful, in the process I ran into some > problems that I did not forsee: According to the HOWTO it seems that > it should have worked, even after spending some time going through the > sections looking for answers, the problem was not obvious to me. > > So I would appreciate if you could tell me where my mistake was and > also maybe add a bit of information to the HOWTO to help others fall > into the same traps that I fell into. :-) > > Here is what I wanted my ideal solution to look like: > > A strong priority of traffic, where parts of the upstream should be > guaranteed rate for some traffic, the rest should be given to normal > traffic and any "leftovers" to BULK traffic, which is allowed to > starve for a while. Also, connection handshake and such very short, > time critical things should get absolute priority over everything > else. > > So this is what I ideally wanted to set up: > > 1: PRIO QDISC (4 Bands), DEFAULT: ALL TO BAND 3 (2 in priomap) > > 1:1 -> SFQ, handle 10: > for priority communication (connection handshake & co) > > 1:2 -> HTB, handle 20: > limited to Xk for different kinds of guaranteed rates that > can "borrow" from each other, but never more than the > maximum -- so it cannot saturate the link fully. > > 20:1 -> SFQ, handle 100: > 20:2 -> SFQ, handle 200: > 20:3 -> SFQ, handle 300: > 20:4 -> SFQ, handle 400: > [...] > > 1:3 -> PRIO QDISC (default), handle 30: > for all "normal"/unclassified traffic, TOS splitting only > 30:1 (BAND 1) > 30:2 (BAND 2) > 30:3 (BAND 3) > > 1:4 -> PFIFO, handle 40: > "starvation bitbucket" > gets what is left, can starve at times > > The setup was apparently successful, tc does not complain and displays > the structure: without any CLASSIFY targets, all traffic goes to 1:3 > and is split in the three bands properly. Looks good. > > I can also add CLASSIFY for 1:1 and 1:4, which seem to fall into the > SFQ/PFIFO buckets underneath, both look good. > > The problem starts with the HTB embedded in 1:2 as 20: -- I can never > CLASSIFY traffic for it properly, the only way to get ANY traffic into > it is to CLASSIFY for 1:2, but that puts all into its default bucket, > which defeats its purpose. > > Neither 20:1 nor 100:0 would put traffic into its first bucket. > > Such classification is simply ignored as far as I can tell. > > This is the problem that I ultimately did not find a way to solve, the > "indirect" approach of using the MARK target and tc filters also did > not work -- it shows the exact same result. > > I currently run another approach to this, which I am not quite as > happy with, but which works for now -- but would still like to know: > > * WHY was the 20:1 or 100:0 CLASSIFY not successful? Nothing in the > documentation seemed to indicate that it should fail. > > * HOW could it have been made to work? > > * WHAT kind of information was I lacking? > > or, in short: WHAT did I do wrong? > > I'd be grateful to find an answer and think it might help to then find > a way to add that answer into the LARTC HOWTO. > > Regards, > Georg > > -- > Georg C. F. Greve > Free Software Foundation Europe (http://fsfeurope.org) > Join the Fellowship and protect your freedom! (http://www.fsfe.org) > > > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > > > > From greve at fsfeurope.org Fri Jul 15 22:58:09 2005 From: greve at fsfeurope.org (Georg C. F. Greve) Date: Fri Jul 15 22:58:02 2005 Subject: [LARTC] Problems setting up nested qdisc, feedback to LARTC HOWTO In-Reply-To: <2af4364905071511466616a94@mail.gmail.com> (Jody Shumaker's message of "Fri, 15 Jul 2005 14:46:24 -0400") References: <2af4364905071511466616a94@mail.gmail.com> Message-ID: || On Fri, 15 Jul 2005 14:46:24 -0400 || Jody Shumaker wrote: js> Have you tried not using classify but instead using tc filters? Well, the tc filters are much more limited than iptables, they could not replicate what I am doing with the iptables. That is why I tried using the -j MARK iptables target to set a tc filter based on that. js> Maybe this is a limitation with iptables classify. Try using js> your classify to put things into 20: and then use tc filters js> attached to 20: to split into the htb subclasses? Interesting idea. It is comforting that you have done something similar with success, so I guess a combination of -j MARK and -j CLASSIFY targets might be able to do that job, I will have to try this. But having to employ such a mix seems like a cludge, shouldn't this work properly with CLASSIFY, as well? Nothing in the documentation says that it shouldn't -- and the docs are missing sufficiently complex examples to get an idea of how others solved that problem. It seems some problem exists, it is just not clear to me yet whether this is a bug in the documentation or the software. Regards, Georg -- Georg C. F. Greve Free Software Foundation Europe (http://fsfeurope.org) Join the Fellowship and protect your freedom! (http://www.fsfe.org) -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 309 bytes Desc: not available Url : http://mailman.ds9a.nl/pipermail/lartc/attachments/20050715/ab00ef7d/attachment-0001.bin From greve at fsfeurope.org Sat Jul 16 00:45:56 2005 From: greve at fsfeurope.org (Georg C. F. Greve) Date: Sat Jul 16 00:46:41 2005 Subject: [LARTC] Problems setting up nested qdisc, feedback to LARTC HOWTO In-Reply-To: (Georg C. F. Greve's message of "Fri, 15 Jul 2005 22:58:09 +0200") References: <2af4364905071511466616a94@mail.gmail.com> Message-ID: || On Fri, 15 Jul 2005 22:58:09 +0200 || "Georg C. F. Greve" wrote: gg> It is comforting that you have done something similar with gg> success, so I guess a combination of -j MARK and -j CLASSIFY gg> targets might be able to do that job, I will have to try this. UPDATE: This indeed appears to be working. Maybe this ought to go into the HOWTO in some way. Regards, Georg -- Georg C. F. Greve Free Software Foundation Europe (http://fsfeurope.org) Join the Fellowship and protect your freedom! (http://www.fsfe.org) -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 309 bytes Desc: not available Url : http://mailman.ds9a.nl/pipermail/lartc/attachments/20050716/f71e02ad/attachment.bin From jody.shumaker at gmail.com Sat Jul 16 07:14:48 2005 From: jody.shumaker at gmail.com (Jody Shumaker) Date: Sat Jul 16 07:15:02 2005 Subject: [LARTC] Problems setting up nested qdisc, feedback to LARTC HOWTO In-Reply-To: References: <2af4364905071511466616a94@mail.gmail.com> Message-ID: <42D897C8.5060503@gmail.com> When I read this earlier I thought of offering the same, i'm using a mix of -j mark and tc filters to do my routing into nexted qdiscs. Seems more like i'd consider this a bug of classify that should be fixed, and maybe a note in the howto that it is broken for now. - Jody Georg C. F. Greve wrote: > || On Fri, 15 Jul 2005 22:58:09 +0200 > || "Georg C. F. Greve" wrote: > > gg> It is comforting that you have done something similar with > gg> success, so I guess a combination of -j MARK and -j CLASSIFY > gg> targets might be able to do that job, I will have to try this. > >UPDATE: This indeed appears to be working. > >Maybe this ought to go into the HOWTO in some way. > >Regards, >Georg > > > From jeff at cowart.net Sat Jul 16 09:32:37 2005 From: jeff at cowart.net (Jefferson Cowart) Date: Sat Jul 16 09:32:44 2005 Subject: [LARTC] Losing Packets after a DNAT in prerouting In-Reply-To: <42D72DF0.20708@atheros.com> References: <20050714055434.2AED310082@P450.internal.cowart.net> <42D72DF0.20708@atheros.com> Message-ID: <42D8B815.5050605@cowart.net> Sorry this took so long. In any case I've included all the parts of my rules file that I think are relevent below. Let me know if there is anything else needed. When I send packets to 134.173.95.144 I see them appear in the tcpdump on the incoming interface (eth2). I also see them in my kernel log from the log entry in the prerouting chain. I however do not see them in my forward chain and they don't actually make it to 192.168.5.9. (I've tried adding logging rules there, but the packets don't appear. All my drop rules are preceded by a log step.) Firewall Rules ====== Chain PREROUTING (policy ACCEPT 13M packets, 2207M bytes) pkts bytes target prot opt in out source destination 3 144 LOG tcp -- eth2 * 134.173.64.0/19 134.173.95.144 tcp dpt:3389 LOG flags 0 level 4 3 144 DNAT tcp -- eth2 * 134.173.64.0/19 134.173.95.144 tcp dpt:3389 to:192.168.5.9:3389 Chain FORWARD (policy DROP 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 0 0 ACCEPT tcp -- eth2 eth3 134.173.64.0/19 192.168.5.9 tcp dpt:3389 Routing Rules ===== # ip rule 0: from all lookup local 200: from 134.173.69.154/31 lookup 200 201: from 134.173.91.144/30 lookup 201 202: from 134.173.95.144/30 lookup 202 203: from 192.168.5.128/25 lookup 203 204: from 192.168.5.0/25 lookup 204 250: from all lookup 250 # ip route show table 250 192.168.5.0/24 dev eth3 scope link 134.173.68.0/23 dev eth0 scope link 134.173.92.0/22 dev eth2 scope link 134.173.88.0/22 dev eth1 scope link default via 134.173.69.254 dev eth0 pramod wrote: > Can u attach ur Rules file.. > > thanks > pramod > From pstaszewski at artcom.pl Sat Jul 16 10:34:09 2005 From: pstaszewski at artcom.pl (=?Windows-1250?Q?Pawe=B3=20Staszewski?=) Date: Sat Jul 16 10:37:35 2005 Subject: [LARTC] Odp: LARTC Digest, Vol 5, Issue 34 Message-ID: Hello Yes i have separate class for every user, every class have pfifp qdisc. User direction shaping: tc -s -d qdisc show dev eth2 | grep pfifo |wc -l 3959 Internet direction shaping: tc -s -d qdisc show dev vlan0891 | grep pfifo |wc -l 3959 Nat: iptables -L -n -v -t nat | grep SNAT | wc -l 3959 For shaping i use hfsc classifier. Date: Fri, 15 Jul 2005 14:16:33 +0200 From: Peter Surda Subject: Re: [LARTC] Use of qcdisc+htb To: lartc@mailman.ds9a.nl Message-ID: <200561514163326735@mail.routehat.org> Content-Type: text/plain; charset=UTF-8 On Fri, 15 Jul 2005 11:55:34 +0200 "Pawe** Staszewski" wrote: >Hello Hello >I have 40Mbit/s internet uplink >Average transfer 25Mbit/s >- 3957 users [cut] Well, the question is what kind of TC-setup you have. Do you have a separate HTB class for every user? Yours sincerely, Peter From greve at fsfeurope.org Sat Jul 16 10:46:23 2005 From: greve at fsfeurope.org (Georg C. F. Greve) Date: Sat Jul 16 10:46:18 2005 Subject: [LARTC] Problems setting up nested qdisc, feedback to LARTC HOWTO In-Reply-To: <42D897C8.5060503@gmail.com> (Jody Shumaker's message of "Sat, 16 Jul 2005 01:14:48 -0400") References: <2af4364905071511466616a94@mail.gmail.com> <42D897C8.5060503@gmail.com> Message-ID: || On Sat, 16 Jul 2005 01:14:48 -0400 || Jody Shumaker wrote: js> When I read this earlier I thought of offering the same, i'm js> using a mix of -j mark and tc filters to do my routing into js> nexted qdiscs. Seems more like i'd consider this a bug of js> classify that should be fixed, and maybe a note in the howto that js> it is broken for now. I fully agree with your classification of this behaviour as a bug. Also I agree that ideally it should be fixed. Do the authors know this bug exists? Where would it have to be filed? Until then, what about adding something to section 12, "Advanced filters for (re-)classifying packets" about this? My suggestion would be to add a subsection 12.X Nested qdiscs: complex classification using MARK and CLASSIFY 12.X.1 Classification problems In environments with nested qdiscs, the iptables CLASSIFY target or tc filters on root qdisc level do not work as one might expect: classification to assign a packet to a qdisc in a nested structure is ignored. Consider the following example: generate a prio qdisc, all traffic going to band 3 (which is a somewhat inconsistent numerical 2 in priomap, see prio qdisc section for more information): # tc qdisc add dev eth0 root handle 1: prio priomap 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 now add a htb qdisc at band 1 of the prio qdisc: # tc qdisc add dev eth0 parent 1:1 handle 10: htb default 40 # tc class add dev eth0 parent 10: classid 10:1 htb rate 500kbit # tc class add dev eth0 parent 10:1 classid 10:10 htb rate 100kbit # tc class add dev eth0 parent 10:1 classid 10:20 htb rate 100kbit # tc class add dev eth0 parent 10:1 classid 10:30 htb rate 100kbit # tc class add dev eth0 parent 10:1 classid 10:40 htb rate 100kbit All unclassified traffic goes into band 3 of the prio qdisc, traffic classified for target 1:2 goes into band 2, as expected. But if traffic is classified for 10:10, 10:20, 10:30 or 10:40, all of it goes into band 3 of the prio qdisc, classification is ignored! Yes, this looks like a bug. Fortunately, there is a workaround. 12.X.2 Combining MARK and CLASSIFY In the above example, classifying traffic for 1:1 works as expected: traffic goes into the htb, all traffic ending up in the default bucket, 10:40. This is obviously not very useful, but allows us to now *attach a filter to the htb qdisc*: # tc filter add dev eth0 protocol ip parent 10:0 prio 1 handle 1 fw classid 10:10 Which puts all packets marked with handle 1 into the first bucket: Marking packets is done in iptables with the MARK target. And naturally, you can also place other filters into the htb qdisc. So for packets to be distributed in a nested hierarchy, you have to CLASSIFY that traffic for the root qdisc minor device that contains the nested structure and then add filters inside that nested structure which subdivide the traffic. The MARK target works very well for this. Naturally, feel free to edit/improve. Could the appropriate section author please add this? I think it would be very helpful. Regards, Georg -- Georg C. F. Greve Free Software Foundation Europe (http://fsfeurope.org) Join the Fellowship and protect your freedom! (http://www.fsfe.org) -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 309 bytes Desc: not available Url : http://mailman.ds9a.nl/pipermail/lartc/attachments/20050716/5ed71713/attachment.bin From pramod at atheros.com Sat Jul 16 11:24:14 2005 From: pramod at atheros.com (pramod) Date: Sat Jul 16 11:24:25 2005 Subject: [LARTC] Losing Packets after a DNAT in prerouting In-Reply-To: <42D8B815.5050605@cowart.net> References: <20050714055434.2AED310082@P450.internal.cowart.net> <42D72DF0.20708@atheros.com> <42D8B815.5050605@cowart.net> Message-ID: <42D8D23E.5020609@atheros.com> Did u do this cat /proc/sys/net/ipv4/conf/all/forwarding the output should be 1 Also cat /proc/sys/net/ipv4/conf/all/arp_filter this should also be 1 thanks pramod From andy.furniss at dsl.pipex.com Sat Jul 16 12:40:29 2005 From: andy.furniss at dsl.pipex.com (Andy Furniss) Date: Sat Jul 16 12:40:16 2005 Subject: [LARTC] Problems setting up nested qdisc, feedback to LARTC HOWTO In-Reply-To: References: <2af4364905071511466616a94@mail.gmail.com> <42D897C8.5060503@gmail.com> Message-ID: <42D8E41D.2050207@dsl.pipex.com> Georg C. F. Greve wrote: > || On Sat, 16 Jul 2005 01:14:48 -0400 > || Jody Shumaker wrote: > > js> When I read this earlier I thought of offering the same, i'm > js> using a mix of -j mark and tc filters to do my routing into > js> nexted qdiscs. Seems more like i'd consider this a bug of > js> classify that should be fixed, and maybe a note in the howto that > js> it is broken for now. > > I fully agree with your classification of this behaviour as a bug. > > Also I agree that ideally it should be fixed. I don't know if I would call it a bug - you are nesting classfull qdiscs and expecting them to know about each others classes when they don't. You have to filter the packets from root to the leaf containing the next qdisc and if it is classfull you have to filter again there. This is never going to be doable with just iptables classify. IIRC the example in LARTC doesn't nest classful - I agree documenting your setup will be usefull - but LARTC doesn't get updated much. FWIW your setup just wouldn't work for me - you are not limiting bandwidth for the link - you would need htb as root to do that. In my case if I put that on my dsl connection it would go overlimits and I would end up with a 2 second queue in my modem. Andy. From andy.furniss at dsl.pipex.com Sat Jul 16 14:04:57 2005 From: andy.furniss at dsl.pipex.com (Andy Furniss) Date: Sat Jul 16 14:04:41 2005 Subject: [LARTC] QoS on receive In-Reply-To: <42D6898C.4050502@ti.com> References: <42D6898C.4050502@ti.com> Message-ID: <42D8F7E9.5010408@dsl.pipex.com> Alexander Sirotkin wrote: > It appears that while Linux has plenty of traffic shaping mechanism on > transmit, there is nothing on receive side. > While generally it does make sense since transmit is more CPU intensive > operation, after all receive also > consumes CPU cycles. It is clear that it's best to drop the packet as > soon as possible, i.e. on receive, if possible - > by the driver itself. It may not be feasible in general case, but I can > think of a couple of scenarios when it does > make sense. > > Any ideas ? > Maybe there is some similar QoS mechanism that I'm not aware of ? > Yes it's called ingress policing there is mention in LARTC and it is possible to do quite complicated things with it. See the diffserv examples in iproute2. Andy. From jeff at cowart.net Sat Jul 16 23:13:26 2005 From: jeff at cowart.net (Jefferson Cowart) Date: Sat Jul 16 23:13:32 2005 Subject: [LARTC] Losing Packets after a DNAT in prerouting In-Reply-To: <42D8D23E.5020609@atheros.com> Message-ID: <20050716211322.023F310082@P450.internal.cowart.net> I had already done the first of those. Changing the second didn't seem to fix anything. ---------------- Thanks Jefferson Cowart Jeff@cowart.net > -----Original Message----- > From: pramod [mailto:pramod@atheros.com] > Sent: Saturday, July 16, 2005 02:24 > To: Jefferson Cowart > Cc: lartc@mailman.ds9a.nl > Subject: Re: [LARTC] Losing Packets after a DNAT in prerouting > > Did u do this > > cat /proc/sys/net/ipv4/conf/all/forwarding > the output should be 1 > Also > cat /proc/sys/net/ipv4/conf/all/arp_filter > this should also be 1 > > thanks > pramod > > > > > From shantanu.iitg at gmail.com Sun Jul 17 18:38:22 2005 From: shantanu.iitg at gmail.com (Shantanu Kumar) Date: Sun Jul 17 18:38:26 2005 Subject: [LARTC] iproute2 rules not being followed !!!!!!! Message-ID: <70198d7c050717093865e5dacb@mail.gmail.com> Hi... I have installed ip route 2 package on Linux kernel 2.4.25 I am using 2 tables: ################################### ebox:100.254~# ip route list table ALTER default via 192.168.100.253 dev br0 ebox:100.254~# ip route list table main 10.0.0.254 dev ppp0 proto kernel scope link src 10.0.0.1 192.168.100.0/24 dev br0 proto kernel scope link src 192.168.100.254 192.168.100.0/24 dev eth1 proto kernel scope link src 192.168.100.233 192.168.200.0/24 dev eth2 proto kernel scope link src 192.168.200.254 127.0.0.0/8 dev lo scope link default via 10.0.0.254 dev ppp0 ##################################### Now I set up the rules : ##################################### ebox:100.254~# ip rule list 0: from all lookup local 7: from 216.239.59.147 lookup main 8: from 202.141.80.6 lookup ALTER 32739: from 202.141.80.6 lookup ALTER 32740: from 66.102.11.99 lookup ALTER 32741: from 66.102.11.104 lookup main 32742: from 64.73.37.225 lookup main 32743: from 216.239.59.103 lookup main 32744: from 210.43.44.8 lookup main 32745: from 64.233.183.19 lookup main 32746: from 64.233.183.83 lookup ALTER 32747: from 64.233.183.106 lookup ALTER 32748: from 210.157.158.37 lookup ALTER 32749: from 66.249.87.99 lookup main 32752: from 213.244.168.210 lookup main 32753: from 66.197.129.37 lookup ALTER 32754: from 82.102.4.72 lookup ALTER 32755: from 216.73.82.14 lookup ALTER 32756: from 216.73.82.70 lookup ALTER 32757: from 216.74.132.11 lookup main 32758: from 216.109.117.205 lookup ALTER 32759: from 202.138.124.172 lookup main 32760: from 216.109.127.16 lookup main 32761: from 209.244.156.19 lookup ALTER 32762: from 68.142.228.136 lookup ALTER 32763: from 82.102.4.57 lookup ALTER 32765: from 216.109.118.65 lookup main 32766: from all lookup main 32767: from all lookup default ##################################### But when I try to trace the route of a packet with destination address such that according to ip rule table ALTER should be considered ... its not following the rule... its always following table Main... ##################################### ebox:100.254~# tcptraceroute -n 202.141.80.6 Selected device ppp0, address 10.0.0.1, port 40113 for outgoing packets Tracing the path to 202.141.80.6 on TCP port 80 (http), 30 hops max 1 10.0.0.254 0.333 ms 0.212 ms 0.210 ms 2 202.141.80.6 [open] 0.589 ms 0.591 ms 0.588 ms ###################################### I would appreciate if I can get any help on why this is not working .!!!! Thanks. Regards Shantanu From shantanu.iitg at gmail.com Sun Jul 17 19:23:37 2005 From: shantanu.iitg at gmail.com (Shantanu Kumar) Date: Sun Jul 17 19:23:40 2005 Subject: [LARTC] Re: iproute2 rules not being followed !!!!!!! In-Reply-To: <70198d7c050717093865e5dacb@mail.gmail.com> References: <70198d7c050717093865e5dacb@mail.gmail.com> Message-ID: <70198d7c0507171023681c09b9@mail.gmail.com> Hi... I m really very sorry... actually I did a very stupid mistake... in adding rules I added using "from" instead of "to" .... :) I m really sorry for wasting all the time.. REgards Shantanu On 7/17/05, Shantanu Kumar wrote: > Hi... > > I have installed ip route 2 package on Linux kernel 2.4.25 > > I am using 2 tables: > > ################################### > > ebox:100.254~# ip route list table ALTER > default via 192.168.100.253 dev br0 > > ebox:100.254~# ip route list table main > 10.0.0.254 dev ppp0 proto kernel scope link src 10.0.0.1 > 192.168.100.0/24 dev br0 proto kernel scope link src 192.168.100.254 > 192.168.100.0/24 dev eth1 proto kernel scope link src 192.168.100.233 > 192.168.200.0/24 dev eth2 proto kernel scope link src 192.168.200.254 > 127.0.0.0/8 dev lo scope link > default via 10.0.0.254 dev ppp0 > > > > ##################################### > > > Now I set up the rules : > > ##################################### > > ebox:100.254~# ip rule list > 0: from all lookup local > 7: from 216.239.59.147 lookup main > 8: from 202.141.80.6 lookup ALTER > 32739: from 202.141.80.6 lookup ALTER > 32740: from 66.102.11.99 lookup ALTER > 32741: from 66.102.11.104 lookup main > 32742: from 64.73.37.225 lookup main > 32743: from 216.239.59.103 lookup main > 32744: from 210.43.44.8 lookup main > 32745: from 64.233.183.19 lookup main > 32746: from 64.233.183.83 lookup ALTER > 32747: from 64.233.183.106 lookup ALTER > 32748: from 210.157.158.37 lookup ALTER > 32749: from 66.249.87.99 lookup main > 32752: from 213.244.168.210 lookup main > 32753: from 66.197.129.37 lookup ALTER > 32754: from 82.102.4.72 lookup ALTER > 32755: from 216.73.82.14 lookup ALTER > 32756: from 216.73.82.70 lookup ALTER > 32757: from 216.74.132.11 lookup main > 32758: from 216.109.117.205 lookup ALTER > 32759: from 202.138.124.172 lookup main > 32760: from 216.109.127.16 lookup main > 32761: from 209.244.156.19 lookup ALTER > 32762: from 68.142.228.136 lookup ALTER > 32763: from 82.102.4.57 lookup ALTER > 32765: from 216.109.118.65 lookup main > 32766: from all lookup main > 32767: from all lookup default > > > ##################################### > > > But when I try to trace the route of a packet with destination address > such that according to ip rule table ALTER should be considered ... > its not following the rule... its always following table Main... > > ##################################### > > ebox:100.254~# tcptraceroute -n 202.141.80.6 > Selected device ppp0, address 10.0.0.1, port 40113 for outgoing packets > Tracing the path to 202.141.80.6 on TCP port 80 (http), 30 hops max > 1 10.0.0.254 0.333 ms 0.212 ms 0.210 ms > 2 202.141.80.6 [open] 0.589 ms 0.591 ms 0.588 ms > > > ###################################### > > > > I would appreciate if I can get any help on why this is not working .!!!! > > Thanks. > > Regards > Shantanu > From mailinglists at lucassen.org Sun Jul 17 22:11:32 2005 From: mailinglists at lucassen.org (richard lucassen) Date: Sun Jul 17 22:11:36 2005 Subject: [LARTC] IPSEC packets not passing POSTROUTING chain Message-ID: <20050717221132.23a539e1.mailinglists@lucassen.org> Packets going to a 2.6 kernel IPSEC tunnel do not seem to pass the POSTROUTING chain. Is that correct? R. -- ___________________________________________________________________ It's so simple to be wise. Just think of something stupid to say and say the opposite. +------------------------------------------------------------------+ | Richard Lucassen, Utrecht | | Public key and email address: | | http://www.lucassen.org/mail-pubkey.html | +------------------------------------------------------------------+ From michael.auckland at gmail.com Mon Jul 18 01:22:24 2005 From: michael.auckland at gmail.com (MIchael) Date: Mon Jul 18 01:22:31 2005 Subject: [LARTC] routing based on user id Message-ID: <35207fdf0507171622573a08aa@mail.gmail.com> Hi all! I've got 2 (soon 3) internet connection. 1 - via ADSL, 2(and3) via ppp My network: http://desima.objectis.net/network-diag linux1: user1.user2 eth0=192.168.1.1 ppp0=192.168.5.2( gw 192.168.5.1) gw=192.168.1.2 ( thru ADSL) compA=192.168.1.6 compB=192.168.1.15 gw2=192.168.1.217 via ppp to different ISP All works for compA and CompB, user1 should use default gw(192.168.1.2) user2 should use ppp0 For user2 it will work only if I change his route thru gw2(not ppp0): iptables -A OUTPUT -t mangle -m owner --uid-owner 1006 -j MARK --set-mark=0x1 ip rule add fwmark 0x01 table gw2 ip route add default via 192.168.1.217 table gw2 ip route flush cache But If I change his route via ppp0 (table T2) all traffic stop for him but will work ok for users CompA and B (using SNAT) iptables -t nat -A POSTROUTING -o ppp0 -j SNAT --to-source 192.168.7.2 According to tcpdump packets going: 192.168.5.2>200.200.200.200 and back 192.168.5.2<200.200.200.200 but for some reason not reaching user2 How I can enable routing thru ppp0 for user2? (I will need it lateer for user3 using ppp1) Any help greatly appreciated Configs: ip rule ls: 0: from all lookup local 32750: from all fwmark 0x1 lookup T2 32751: from 192.168.5.2 lookup T2 32752: from 192.168.5.1 lookup T2 32756: from 192.168.1.6 lookup T1 32757: from 192.168.1.15 lookup T2 32766: from all lookup main 32767: from all lookup default ip route table T2 192.168.5.0 dev ppp0 scope link src 192.168.5.2 192.168.1.0/24 dev eth0 scope link 127.0.0.0/8 dev lo scope link default via 192.168.5.1 dev ppp0 ip route ls 192.168.1.0 dev eth0 scope link src 192.168.1.1 192.168.5.1 dev ppp0 proto kernel scope link src 192.168.5.2 203.97.61.42 via 192.168.1.2 dev eth0 192.168.1.0/24 dev eth0 scope link 127.0.0.0/8 dev lo scope link default via 192.168.1.2 dev eth0 From pramod at atheros.com Mon Jul 18 07:08:05 2005 From: pramod at atheros.com (pramod) Date: Mon Jul 18 08:08:20 2005 Subject: [LARTC] Losing Packets after a DNAT in prerouting In-Reply-To: <20050716211322.023F310082@P450.internal.cowart.net> References: <20050716211322.023F310082@P450.internal.cowart.net> Message-ID: <42DB3935.5070408@atheros.com> I am sorry In the second option i did a mistake Do the following things... 1) Restore the arp_filter to default.. 2) Set rp_filter to 0 (zero) thanks pramod From staenker at web.de Mon Jul 18 09:26:39 2005 From: staenker at web.de (Staenker) Date: Mon Jul 18 09:26:45 2005 Subject: [LARTC] ppp uplink shaping problems In-Reply-To: <70198d7c050717093865e5dacb@mail.gmail.com> References: <70198d7c050717093865e5dacb@mail.gmail.com> Message-ID: <42DB59AF.60106@web.de> Hello, i played a few days with tc htb classes and classified my packets using iptables CLASSIFY target. here is what i did: #!/bin/bash int='ppp0' #making all things clear tc qdisc del dev $int root iptables -t mangle --flush iptables -t mangle --delete-chain if $1 then #defining classes tc qdisc add dev $int root handle 1: htb default 20 r2q 2 tc class add dev $int parent 1: classid 1:1 htb rate 22kbps tc class add dev $int parent 1:1 classid 1:10 htb rate 10kbps ceil 22kbps prio 0 tc class add dev $int parent 1:1 classid 1:20 htb rate 9kbps ceil 15kbps prio 1 tc class add dev $int parent 1:1 classid 1:30 htb rate 3kbps ceil 13kbps prio 2 tc qdisc add dev $int parent 1:10 handle 10: sfq perturb 10 tc qdisc add dev $int parent 1:20 handle 20: sfq perturb 10 tc qdisc add dev $int parent 1:30 handle 30: sfq perturb 10 iptables -t mangle -N TS_FWD iptables -t mangle -A FORWARD -j TS_FWD iptables -t mangle -A TS_FWD -o ppp0 -p ! icmp --match length --length 0:70 -j CLASSIFY --set-class 1:10 iptables -t mangle -A TS_FWD -o ppp0 -p ! icmp --match length --length 0:70 -j RETURN iptables -t mangle -A TS_FWD -i eth2 -o ppp0 -p tcp --source 192.168.0.2 --destination-port 80 -j CLASSIFY --set-class 1:20 iptables -t mangle -A TS_FWD -i eth2 -o ppp0 -p tcp --source 192.168.0.2 --destination-port 80 -j RETURN iptables -t mangle -A TS_FWD -i eth2 -o ppp0 -p tcp --source 192.168.0.2 --destination-port 443 -j CLASSIFY --set-class 1:20 iptables -t mangle -A TS_FWD -i eth2 -o ppp0 -p tcp --source 192.168.0.2 --destination-port 443 -j RETURN iptables -t mangle -A TS_FWD -i eth2 -o ppp0 -p tcp --source 192.168.0.2 --destination-port 554 -j CLASSIFY --set-class 1:10 iptables -t mangle -A TS_FWD -i eth2 -o ppp0 -p tcp --source 192.168.0.2 --destination-port 554 -j RETURN #if $2 #then # iptables -t mangle -A TS_FWD -i eth2 -o ppp0 --source 192.168.0.2 -j LOG #fi iptables -t mangle -A TS_FWD -i eth2 -o ppp0 --source 192.168.0.2 -j CLASSIFY --set-class 1:30 fi It works not really good. I tested it using my internal 100MBit network interface using multiple ftp connections and classified the packets based on their source-ip. That works fine with same classes. Immediately all things i expected took place. Also the prio option worked fine. If i was running 2 simultanious downloads, the one with the higher piority gets all borrowable downloadspeed and the one with the lower priority gets his ashured rate. But same classes didnt work with my 192kbit 2048kbit ppp link. Well ok, they are working, but not like i want them to work. The speed changes takes some seconds to take place. And the priority seems to be ignored. I have to say, that the i tested the ppp uplink using emule with many connections (500 - 800) and the higher priority upload was one active ftp connection. Whats my fault? Regards Richard Hauswald From JNeave at spursolutions.com Mon Jul 18 10:32:13 2005 From: JNeave at spursolutions.com (James Neave) Date: Mon Jul 18 10:32:25 2005 Subject: [LARTC] Load balancing (LARTC 4.2) over 2 connections on 2 routers. Message-ID: Hi, I'm building a network similar to that seen in 4.2 of the LARTC Howto. There is a diagram of this attached to this mail. Addendum to diagram: AlexRouter br0 = 192.168.58.1 eth0 = dhcpcd DaveRouter br0 = 192.168.58.2 eth0 = dhcpcd But we've run into some problems when actually implementing the routing for multiple uplinks. The difference between my network and the LARTC example is instead of having one router with two modems I have two routers with one modem each. AlexRouter and DaveRouter. They run Bering-uClibc 2.x off of fd0. A wired/wireless network connects the two together. 192.168.58.0/24. AlexRouter is the default route/DNS server/DHCP server for every host on the network. It gets its DNS servers from dhcpcd. They way I figure it, Provider2 in the example is (in my case) actually DaveRouter. With that in mind, these are the figures I came up with for settings up the routes. These are all from the perspective of AlexRouter. $IF1 = eth0 $IF2 = br0 $IP1 = 80.blah.blah.blah (can't remember my real address) $IP2 = 192.168.58.1 $P1 = $IP1 *DON'T KNOW IF THIS IS RIGHT, DON'T KNOW HOW TO FIND MY PROVIDERS GATEWAY* $P2 = 192.168.58.2 (DaveRouter) $P1_NET = 80.blah.blah.0/24 (got $IP1 and $P1_NET from ip route show) $P2_NET = 192.168.58.0/24 $P0_NET = 192.168.58.0/24 $IF0 = br0 If I set up all the routes using those values, test browsing around is flakey. Some pages load, some don't (one connection working, one not?) I *can* use one connection *OR* the other connection. But only if I manually re-write /etc/resolv.conf to contain the correct DNS servers for the provider used. One ISP is Demon, the other is BT. They won't let each other use their DNS servers. Also, I had duplicate returns from ping. Apart from that, I'm not sure where I go with diagnosis. Does anybody have any idea what's going on? Thanks, James. The information in this email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this email by anyone else is unauthorised. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it is prohibited and may be unlawful. The contents of an attachment to this email may contain software viruses that could damage your own computer systems. Whilst The Spur Group of Companies has taken every precaution to minimise the risk, we cannot accept liability for any damage that you sustain as a result of software viruses. -------------- next part -------------- A non-text attachment was scrubbed... Name: StonechatNetSm.png Type: image/png Size: 18948 bytes Desc: StonechatNetSm.png Url : http://mailman.ds9a.nl/pipermail/lartc/attachments/20050718/a5289b69/StonechatNetSm-0001.png From user01 at kyberwelt.de Mon Jul 18 11:26:20 2005 From: user01 at kyberwelt.de (ich hier) Date: Mon Jul 18 11:25:28 2005 Subject: [LARTC] IPSEC packets not passing POSTROUTING chain In-Reply-To: <20050717221132.23a539e1.mailinglists@lucassen.org> References: <20050717221132.23a539e1.mailinglists@lucassen.org> Message-ID: <20050718112620.485c5fc6.user01@kyberwelt.de> hi On Sun, 17 Jul 2005 22:11:32 +0200 richard lucassen wrote: > Packets going to a 2.6 kernel IPSEC tunnel do not seem to pass the > POSTROUTING chain. Is that correct? > running 2.6.11.0 and using snat (in the postrouting-chain) successfully for the decrypted packets to be routed/snatted into the internal lan. greetings user01 From michael.auckland at gmail.com Mon Jul 18 11:30:32 2005 From: michael.auckland at gmail.com (Michael) Date: Mon Jul 18 11:30:36 2005 Subject: [LARTC] routing based on user id In-Reply-To: <35207fdf0507171622573a08aa@mail.gmail.com> References: <35207fdf0507171622573a08aa@mail.gmail.com> Message-ID: <35207fdf050718023015099a2c@mail.gmail.com> Hi all! I've got 2 (soon 3) internet connection. 1 - via ADSL, 2(and3) via ppp My network: http://desima.objectis.net/network-diag linux1: user1.user2 eth0=192.168.1.1 ppp0=192.168.5.2( gw 192.168.5.1) gw=192.168.1.2 ( thru ADSL) compA=192.168.1.6 compB=192.168.1.15 gw2=192.168.1.217 via ppp to different ISP All works for compA and CompB, user1 should use default gw(192.168.1.2) user2 should use ppp0 For user2 it will work only if I change his route thru gw2(not ppp0): iptables -A OUTPUT -t mangle -m owner --uid-owner 1006 -j MARK --set-mark=0x1 ip rule add fwmark 0x01 table gw2 ip route add default via 192.168.1.217 table gw2 ip route flush cache But If I change his route via ppp0 (table T2) all traffic stop for him but will work ok for users CompA and B (using SNAT) iptables -t nat -A POSTROUTING -o ppp0 -j SNAT --to-source 192.168.7.2 According to tcpdump packets going: 192.168.5.2>200.200.200.200 and back 192.168.5.2<200.200.200.200 but for some reason not reaching user2 How I can enable routing thru ppp0 for user2? (I will need it lateer for user3 using ppp1) Any help greatly appreciated Configs: ip rule ls: 0: from all lookup local 32750: from all fwmark 0x1 lookup T2 32751: from 192.168.5.2 lookup T2 32752: from 192.168.5.1 lookup T2 32756: from 192.168.1.6 lookup T1 32757: from 192.168.1.15 lookup T2 32766: from all lookup main 32767: from all lookup default ip route table T2 192.168.5.0 dev ppp0 scope link src 192.168.5.2 192.168.1.0/24 dev eth0 scope link 127.0.0.0/8 dev lo scope link default via 192.168.5.1 dev ppp0 ip route ls 192.168.1.0 dev eth0 scope link src 192.168.1.1 192.168.5.1 dev ppp0 proto kernel scope link src 192.168.5.2 203.97.61.42 via 192.168.1.2 dev eth0 192.168.1.0/24 dev eth0 scope link 127.0.0.0/8 dev lo scope link default via 192.168.1.2 dev eth0 -------------- next part -------------- A non-text attachment was scrubbed... Name: local-map.png Type: image/png Size: 27314 bytes Desc: not available Url : http://mailman.ds9a.nl/pipermail/lartc/attachments/20050718/41022ee2/local-map-0001.png From lburatti at zacmi.it Mon Jul 18 16:02:06 2005 From: lburatti at zacmi.it (lburatti@zacmi.it) Date: Mon Jul 18 16:03:01 2005 Subject: [LARTC] luca buratti =?iso-8859-1?q?=E8_assente_dall=27ufficio=2E?= Message-ID: Sar? assente dall'ufficio a partire dal 18/07/2005 e non torner? fino al 25/07/2005. Risponder? al messaggio al mio ritorno. Trend Scan Mail: this message is virus free. From jeff at cowart.net Tue Jul 19 03:45:53 2005 From: jeff at cowart.net (Jefferson Cowart) Date: Tue Jul 19 03:45:55 2005 Subject: Problems with Routing (was RE: [LARTC] Losing Packets after a DNAT in prerouting) In-Reply-To: <42DB3935.5070408@atheros.com> Message-ID: <20050719014547.3533D10082@P450.internal.cowart.net> Wel that helped, but I'm still having problems. Here is what is happening now: I send a packet from 134.173.94.7 to 134.173.95.146 (those devices are on the same network). It goes into my router on eth2 and gets DNATed to 192.168.5.9 which is on eth3. It gets routed properly and gets to my machine at 192.168.5.9. My machine at 192.168.5.9 responds. It goes back into my router on eth3. My router routes the packet out eth0 and the automatic rule sets to source address back to 134.173.95.146. Since the packet has a source address that is on the wrong interface the packet is dropped. It appears that my problem is that I need it to route the connection back out the same interface that it came in on. However for new connections I need it to use eth0 as the default route. ---------------- Thanks Jefferson Cowart Jeff@cowart.net > -----Original Message----- > From: pramod [mailto:pramod@atheros.com] > Sent: Sunday, July 17, 2005 22:08 > To: Jefferson Cowart > Cc: lartc@mailman.ds9a.nl > Subject: Re: [LARTC] Losing Packets after a DNAT in prerouting > > I am sorry > In the second option i did a mistake > Do the following things... > 1) Restore the arp_filter to default.. > 2) Set rp_filter to 0 (zero) > > thanks > pramod > > From nik39_nospam at usenet.entertain-men.de Tue Jul 19 07:52:03 2005 From: nik39_nospam at usenet.entertain-men.de (nik39_nospam@usenet.entertain-men.de) Date: Tue Jul 19 07:52:08 2005 Subject: [LARTC] Problem with HTB and ceil Message-ID: <1121752323.42dc9503a9c8d@www.domainfactory-webmail.de> Hi, I am trying to setup a shaper for my linux box and I am experiencing some problems. What I am trying to do is priorizing packets on the egress link, therefore I have setup some prio classes etc. The priorizing seem to work but the problem is that the ceil parameter doesnt seem to work on non leafs. This is what stats show: > class htb 1:1 root rate 3000bit ceil 3000bit burst 1602b/8 mpu 0b overhead 0b cburst 1602b/8 mpu 0b overhead 0b level 7 > Sent 770965 bytes 2139 pkts (dropped 0, overlimits 0) > rate 178264bit 62pps > lended: 0 borrowed: 0 giants: 0 > tokens: -59999999 ctokens: -59999999 How can this be possible? From my understanding it shouldnt be possible to exceed the ceil bitrate, but this happens here. What did I overlook? What do these negative token counters mean? Thanks for your help in advance. nik From mailinglists at lucassen.org Tue Jul 19 09:16:21 2005 From: mailinglists at lucassen.org (richard lucassen) Date: Tue Jul 19 09:16:25 2005 Subject: [LARTC] Problem with HTB and ceil In-Reply-To: <1121752323.42dc9503a9c8d@www.domainfactory-webmail.de> References: <1121752323.42dc9503a9c8d@www.domainfactory-webmail.de> Message-ID: <20050719091621.238d8cc9.mailinglists@lucassen.org> On Tue, 19 Jul 2005 07:52:03 +0200 nik39_nospam@usenet.entertain-men.de wrote: > I am trying to setup a shaper for my linux box and I am experiencing > some problems. What I am trying to do is priorizing packets on the > egress link, therefore I have setup some prio classes etc. The > priorizing seem to work but the problem is that the ceil parameter > doesnt seem to work on non leafs. This is what stats show: > > > class htb 1:1 root rate 3000bit ceil 3000bit burst 1602b/8 mpu 0b 3000bit you probably mean 3000kbit? R. -- ___________________________________________________________________ It's so simple to be wise. Just think of something stupid to say and say the opposite. +------------------------------------------------------------------+ | Richard Lucassen, Utrecht | | Public key and email address: | | http://www.lucassen.org/mail-pubkey.html | +------------------------------------------------------------------+ From nik39_nospam at usenet.entertain-men.de Tue Jul 19 09:24:54 2005 From: nik39_nospam at usenet.entertain-men.de (nik39_nospam@usenet.entertain-men.de) Date: Tue Jul 19 09:24:57 2005 Subject: [LARTC] Problem with HTB and ceil In-Reply-To: <20050719091621.238d8cc9.mailinglists@lucassen.org> References: <1121752323.42dc9503a9c8d@www.domainfactory-webmail.de> <20050719091621.238d8cc9.mailinglists@lucassen.org> Message-ID: <1121757894.42dcaac69f75c@www.domainfactory-webmail.de> Zitat von richard lucassen : > > I am trying to setup a shaper for my linux box and I am experiencing > > some problems. What I am trying to do is priorizing packets on the > > egress link, therefore I have setup some prio classes etc. The > > priorizing seem to work but the problem is that the ceil parameter > > doesnt seem to work on non leafs. This is what stats show: > > > > > class htb 1:1 root rate 3000bit ceil 3000bit burst 1602b/8 mpu 0b > > 3000bit > > you probably mean 3000kbit? No, for debugging purposes I have limit it to 3kbps, to test what is going on. nik From c-d.hailfinger.devel.2005 at gmx.net Tue Jul 19 11:40:19 2005 From: c-d.hailfinger.devel.2005 at gmx.net (Carl-Daniel Hailfinger) Date: Tue Jul 19 11:39:20 2005 Subject: [LARTC] Problem with HTB and ceil In-Reply-To: <1121752323.42dc9503a9c8d@www.domainfactory-webmail.de> References: <1121752323.42dc9503a9c8d@www.domainfactory-webmail.de> Message-ID: <42DCCA83.4090001@gmx.net> Hi, nik39_nospam@usenet.entertain-men.de schrieb: > > the problem is that the ceil parameter doesnt seem to work on non leafs. IIRC that is by design. Some time ago, it was explained on this list in a much better way than I could do it now. Regards, Carl-Daniel -- http://www.hailfinger.org/ From nik39_nospam at usenet.entertain-men.de Tue Jul 19 12:54:56 2005 From: nik39_nospam at usenet.entertain-men.de (nik39_nospam@usenet.entertain-men.de) Date: Tue Jul 19 12:55:02 2005 Subject: [LARTC] Problem with HTB and ceil In-Reply-To: <42DCCA83.4090001@gmx.net> References: <1121752323.42dc9503a9c8d@www.domainfactory-webmail.de> <42DCCA83.4090001@gmx.net> Message-ID: <1121770496.42dcdc00203dc@www.domainfactory-webmail.de> Carl-Daniel Hailfinger : > > the problem is that the ceil parameter doesnt seem to work on non leafs. > > IIRC that is by design. Some time ago, it was explained on this list > in a much better way than I could do it now. Yes, thanks I found something but I am not sure how to interpret what has been said. It has been said that if the child classes summed up exceed the bitrate of the parent class then the parent ceil setting will be ignored. Now I am not sure if that only applies to the sum of the minimum bitrate or also of the summed ceil bitrates of all child classes. Let me explain what I am trying to achieve: I have a link with 384kbps upstream, I want voip traffic to have the highest prio, they should always leave the upstream first with a ceil limit of 96kbps which is more than enogh, min 16kbit (should matter cause its priorized anyway). Next priority should be ACK packets, they should have a ceil limit of (380-16) 364kbit, default bitrate of 140. This should ensure that downloads are not affected by any outgoing traffic except the voip traffic. SSH traffic is next, I reserved a minimum bitrate of 16kbit with a ceiling of 364, then I got http traffic again with min bitrate 16kbit, ceil limit of 364 and at last bulk traffic, with a min bitrate 16kbit and ceil limit of 364: o root 1:0 | ________________+ 1:002 / | | | 1:101 | __________+__________ | | | | | | | | | | | | | | | voip ACK SSH http bulk pr 0 pr 1 pr 2 pr 3 pr 4 1:100 1:200 1:300 1:300 1:400 What I have achieved so far is that voip is really priorized agains all other classes. But priorizing the right half does not work, means ACK class has not higher prio than the others. My config so far: -------- ## Root /sbin/tc qdisc add dev $EXTIF root handle 1:0 htb default 400 /sbin/tc class add dev $EXTIF parent 1:0 classid 1:002 htb rate 380kbit ceil 380kbit ## voip /sbin/tc class add dev $EXTIF parent 1:002 classid 1:100 htb rate 16kbit ceil 96kbit prio 0 ## non voip /sbin/tc class add dev $EXTIF parent 1:002 classid 1:101 htb rate 364kbit ceil 364 prio 1 ## ACK /sbin/tc class add dev $EXTIF parent 1:101 classid 1:200 htb rate 140kbit ceil 364kbit prio 1 ## SSH /sbin/tc class add dev $EXTIF parent 1:101 classid 1:300 htb rate 140kbit ceil 364kbit prio 2 ## http/https traffic /sbin/tc class add dev $EXTIF parent 1:101 classid 1:400 htb rate 140kbit ceil 364kbit prio 3 ## bulk /sbin/tc class add dev $EXTIF parent 1:101 classid 1:500 htb rate 140kbit ceil 364kbit prio 4 -------- Stats show: -------- class htb 1:200 parent 1:101 leaf 200: prio 1 quantum 1750 rate 140000bit ceil 364000bit burst 1774b/8 mpu 0b overhead 0b cburst 2066b/8 mpu 0b overhead 0b level 0 Sent 280764 bytes 4995 pkts (dropped 0, overlimits 0) lended: 4995 borrowed: 0 giants: 0 tokens: 79347 ctokens: 34819 class htb 1:500 parent 1:101 leaf 500: prio 6 quantum 1000 rate 16000bit ceil 364000bit burst 1619b/8 mpu 0b overhead 0b cburst 2066b/8 mpu 0b overhead 0b level 0 Sent 3351886 bytes 3165 pkts (dropped 0, overlimits 0) rate 256bit lended: 150 borrowed: 3015 giants: 0 tokens: 237570 ctokens: 17997 -------- As you can see the :200 class (ACK packets) have much lower bitrate than the :500 (bulk packets), which slows down my downloads. Any hints? nik From Andreas.Klauer at metamorpher.de Tue Jul 19 13:30:10 2005 From: Andreas.Klauer at metamorpher.de (Andreas Klauer) Date: Tue Jul 19 13:30:31 2005 Subject: [LARTC] Problem with HTB and ceil In-Reply-To: <1121770496.42dcdc00203dc@www.domainfactory-webmail.de> References: <1121752323.42dc9503a9c8d@www.domainfactory-webmail.de> <42DCCA83.4090001@gmx.net> <1121770496.42dcdc00203dc@www.domainfactory-webmail.de> Message-ID: <200507191330.11186.Andreas.Klauer@metamorpher.de> On Tuesday 19 July 2005 12:54, nik39_nospam@usenet.entertain-men.de wrote: > /sbin/tc class add dev $EXTIF parent 1:002 classid 1:101 htb rate > 364kbit ceil 364 prio 1 > > ## ACK > /sbin/tc class add dev $EXTIF parent 1:101 classid 1:200 htb rate > 140kbit ceil 364kbit prio 1 > ## SSH > /sbin/tc class add dev $EXTIF parent 1:101 classid 1:300 htb rate > 140kbit ceil 364kbit prio 2 > ## http/https traffic > /sbin/tc class add dev $EXTIF parent 1:101 classid 1:400 htb rate > 140kbit ceil 364kbit prio 3 > ## bulk > /sbin/tc class add dev $EXTIF parent 1:101 classid 1:500 htb rate > 140kbit ceil 364kbit prio 4 By looking at this script, you have a class 1:101 with 364kbit rate, 364 ceil (missing unit!), and it's children got (140+140+140+140)kbit = 560kbit rate. I haven't followed the prior discussion, so this might be intentional or not, but in any case, I don't recommend using HTB that way. It's much easier to tell what is happening when you make sure that your rates always add up properly. That is, the sum of the child class rates should be the same as the parent class rate. HTH Andreas From nik39_nospam at usenet.entertain-men.de Tue Jul 19 15:51:48 2005 From: nik39_nospam at usenet.entertain-men.de (nik39_nospam@usenet.entertain-men.de) Date: Tue Jul 19 15:51:51 2005 Subject: [LARTC] Problem with HTB and ceil In-Reply-To: <200507191330.11186.Andreas.Klauer@metamorpher.de> References: <1121752323.42dc9503a9c8d@www.domainfactory-webmail.de> <42DCCA83.4090001@gmx.net> <1121770496.42dcdc00203dc@www.domainfactory-webmail.de> <200507191330.11186.Andreas.Klauer@metamorpher.de> Message-ID: <1121781108.42dd05746874c@www.domainfactory-webmail.de> Andreas Klauer : > On Tuesday 19 July 2005 12:54, nik39_nospam@usenet.entertain-men.de wrote: > > /sbin/tc class add dev $EXTIF parent 1:002 classid 1:101 htb rate > > 364kbit ceil 364 prio 1 > > > > ## ACK > > /sbin/tc class add dev $EXTIF parent 1:101 classid 1:200 htb rate > > 140kbit ceil 364kbit prio 1 > > ## SSH > > /sbin/tc class add dev $EXTIF parent 1:101 classid 1:300 htb rate > > 140kbit ceil 364kbit prio 2 > > ## http/https traffic > > /sbin/tc class add dev $EXTIF parent 1:101 classid 1:400 htb rate > > 140kbit ceil 364kbit prio 3 > > ## bulk > > /sbin/tc class add dev $EXTIF parent 1:101 classid 1:500 htb rate > > 140kbit ceil 364kbit prio 4 > > By looking at this script, you have a class 1:101 with 364kbit rate, 364 > ceil (missing unit!), and it's children got (140+140+140+140)kbit = > 560kbit rate. > > I haven't followed the prior discussion, so this might be intentional or > not, but in any case, I don't recommend using HTB that way. It's much > easier to tell what is happening when you make sure that your rates always > add up properly. Just to make sure, when you talk about the sum you are talking about the sum of the minimum bitrates, not the sum of the ceil bitrates, right? > That is, the sum of the child class rates should be the > same as the parent class rate. Ah damned, that was a copy and paste problem, in fact I used the following script with the bitrates which I mentioned in the previous posts in my description. -------- ## Root /sbin/tc qdisc add dev $EXTIF root handle 1:0 htb default 400 /sbin/tc class add dev $EXTIF parent 1:0 classid 1:002 htb rate 380kbit ceil 380kbit ## voip /sbin/tc class add dev $EXTIF parent 1:002 classid 1:100 htb rate 16kbit ceil 96kbit prio 0 ## non voip /sbin/tc class add dev $EXTIF parent 1:002 classid 1:101 htb rate 364kbit ceil 364 prio 1 ## ACK /sbin/tc class add dev $EXTIF parent 1:101 classid 1:200 htb rate 140kbit ceil 364kbit prio 1 ## SSH /sbin/tc class add dev $EXTIF parent 1:101 classid 1:300 htb rate 16kbit ceil 364kbit prio 2 ## http/https traffic /sbin/tc class add dev $EXTIF parent 1:101 classid 1:400 htb rate 16kbit ceil 364kbit prio 3 ## bulk /sbin/tc class add dev $EXTIF parent 1:101 classid 1:500 htb rate 16kbit ceil 364kbit prio 4 -------- Any idea? Did I miss something basic? nik From Andreas.Klauer at metamorpher.de Tue Jul 19 16:08:53 2005 From: Andreas.Klauer at metamorpher.de (Andreas Klauer) Date: Tue Jul 19 16:09:06 2005 Subject: [LARTC] Problem with HTB and ceil In-Reply-To: <1121781108.42dd05746874c@www.domainfactory-webmail.de> References: <1121752323.42dc9503a9c8d@www.domainfactory-webmail.de> <200507191330.11186.Andreas.Klauer@metamorpher.de> <1121781108.42dd05746874c@www.domainfactory-webmail.de> Message-ID: <200507191608.53800.Andreas.Klauer@metamorpher.de> On Tuesday 19 July 2005 15:51, you wrote: > Just to make sure, when you talk about the sum you are talking about the > sum of the minimum bitrates, not the sum of the ceil bitrates, right? Yes. The only rules for the ceil should be that it's the same or bigger than the rate, and that it does not exceed the parent's ceil. > ## non voip > /sbin/tc class add dev $EXTIF parent 1:002 classid 1:101 htb rate > 364kbit ceil 364 prio 1 ^ Unit is missing, might cause problems. > ## ACK > /sbin/tc class add dev $EXTIF parent 1:101 classid 1:200 htb rate > 140kbit ceil 364kbit prio 1 > ## SSH > /sbin/tc class add dev $EXTIF parent 1:101 classid 1:300 htb rate 16kbit > ceil 364kbit prio 2 > ## http/https traffic > /sbin/tc class add dev $EXTIF parent 1:101 classid 1:400 htb rate 16kbit > ceil 364kbit prio 3 > ## bulk > /sbin/tc class add dev $EXTIF parent 1:101 classid 1:500 htb rate 16kbit > ceil 364kbit prio 4 This time, the sum is lower than the parent. This is not nearly as bad as having a too high sum; together with the prio it can even make sense if it's intentional. I'm just commenting because you should be aware of it, that's all. If this was my script, I'd add a comment explaining as to why it is done this way. HTH Andreas From nik39_nospam at usenet.entertain-men.de Tue Jul 19 22:41:46 2005 From: nik39_nospam at usenet.entertain-men.de (nik-da-39) Date: Tue Jul 19 22:41:49 2005 Subject: [LARTC] Problem with HTB and ceil In-Reply-To: <200507191608.53800.Andreas.Klauer@metamorpher.de> References: <1121752323.42dc9503a9c8d@www.domainfactory-webmail.de> <200507191330.11186.Andreas.Klauer@metamorpher.de> <1121781108.42dd05746874c@www.domainfactory-webmail.de> <200507191608.53800.Andreas.Klauer@metamorpher.de> Message-ID: <1121805706.42dd658a91eba@www.domainfactory-webmail.de> Andreas Klauer : > Yes. The only rules for the ceil should be that it's the same or bigger > than the rate, and that it does not exceed the parent's ceil. Thanks for the info. Its not clearly enough stated in the HTB "manual". > > ## non voip > > /sbin/tc class add dev $EXTIF parent 1:002 classid 1:101 htb rate > > 364kbit ceil 364 prio 1 > ^ Unit is missing, might cause problems. Yeah, I messed copy+paste again. > This time, the sum is lower than the parent. This is not nearly as bad as > having a too high sum; together with the prio it can even make sense if > it's intentional. I'm just commenting because you should be aware of it, > that's all. If this was my script, I'd add a comment explaining as to why > it is done this way. Thanks for your comment. From my understanding, I dont rely too much on the exact minimum bitrate, as long as the prios are obeyed by the packets. Thats why I also dont need the sum of the min bitrates to be exactly the parental bitrate, if I understand it correctly the remaining abdnwidth after all minimum requirements have been fullfilled will be spread proportionally according to the allocations. So, for me its fine if the proportion would be kept. Anyhow, in this special case this doesnt matter at all, cause I would like to have priorized traff. I think I will start a new thread as the original problem why I started this thread is solved. And then I will make sure I post my original script with comments etc. nik From Andreas.Klauer at metamorpher.de Tue Jul 19 23:04:02 2005 From: Andreas.Klauer at metamorpher.de (Andreas Klauer) Date: Tue Jul 19 23:04:16 2005 Subject: [LARTC] Problem with HTB and ceil In-Reply-To: <1121805706.42dd658a91eba@www.domainfactory-webmail.de> References: <1121752323.42dc9503a9c8d@www.domainfactory-webmail.de> <200507191608.53800.Andreas.Klauer@metamorpher.de> <1121805706.42dd658a91eba@www.domainfactory-webmail.de> Message-ID: <200507192304.02145.Andreas.Klauer@metamorpher.de> On Tuesday 19 July 2005 22:41, nik-da-39 wrote: > Thanks for your comment. From my understanding, I dont rely too much on > the exact minimum bitrate, as long as the prios are obeyed by the > packets. ... > Thats why I also dont need the sum of the min bitrates to be > exactly the parental bitrate, if I understand it correctly the remaining > abdnwidth after all minimum requirements have been fullfilled will be > spread proportionally according to the allocations. ... I might've misunderstood you here, but just in case: The HTB prio parameter affects borrowing priority, not packet priority. Which basically means that the proportions which you'd get if all classes had the same priority, will look much different. In other words, low priority classes will not get any bandwidth except their guaranteed rate, as long as a higher priority class is borrowing. Personally, I only use the HTB prio for unwanted traffic class (P2P), because I find that for normal traffic, the restriction is too harsh. This is why I recommend not using the HTB prio parameter at all in the beginning, and only start experimenting with it when you really need it. HTH Andreas From nistnet_user at yahoo.com Wed Jul 20 03:42:02 2005 From: nistnet_user at yahoo.com (js si) Date: Wed Jul 20 03:42:05 2005 Subject: [LARTC] TBF + burst Message-ID: <20050720014202.17077.qmail@web34410.mail.mud.yahoo.com> I am using tbf to do rate limitation. i set the rate, burst and latency parameters for tbf. but when i start passing data i notice that there is an initial burst and then the rate goes down to the configured level. is this because tbf starts with a full bucket? thanks. __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From Thorsten at Gehrig.de Wed Jul 20 16:11:22 2005 From: Thorsten at Gehrig.de (Thorsten Gehrig) Date: Wed Jul 20 16:11:28 2005 Subject: [LARTC] how to define a port range? Message-ID: <42DE5B8A.4090901@Gehrig.de> hi, i?am new in tcc (tcng). i try to define my qos for VoIP-Services. For this i wantto define a class for a port range 10000 till 15000. how is the right way? this down works: class (<$voip>) if tcp_sport => 10000 || tcp_sport <= 10000 ; are there any examples of real installations - maybe including VoIP, HTTP and P2P services? regards thorsten gehrig From sylvain at 2001-space-odyssey.net Wed Jul 20 16:22:42 2005 From: sylvain at 2001-space-odyssey.net (Sylvain Bertrand) Date: Wed Jul 20 16:22:46 2005 Subject: [LARTC] how to define a port range? In-Reply-To: <42DE5B8A.4090901@Gehrig.de> References: <42DE5B8A.4090901@Gehrig.de> Message-ID: <13264.213.245.32.134.1121869362.squirrel@webmail.2001-space-odyssey.net> On Mer 20 juillet 2005 16:11, Thorsten Gehrig a ?crit : > hi, > i??am new in tcc (tcng). i try to define my qos for VoIP-Services. > For this i wantto define a class for a port range 10000 till 15000. > > how is the right way? > this down works: > class (<$voip>) if tcp_sport => 10000 || tcp_sport <= 10000 ; > > are there any examples of real installations - maybe including VoIP, > HTTP and P2P services? > > regards > thorsten gehrig > Hi, You may want to use the mask. Go to the LART QoS Cookbook, and you'll see tc filter lines with: "21 0xfffff" If you want to match both 21 and 22, use: "22 0xffffe" Regards, Sylvain From gentoo at databit7.com Wed Jul 20 17:24:19 2005 From: gentoo at databit7.com (Hammond, Robin-David%KB3IEN) Date: Wed Jul 20 17:24:25 2005 Subject: [LARTC] altq In-Reply-To: <20050716211322.023F310082@P450.internal.cowart.net> References: <20050716211322.023F310082@P450.internal.cowart.net> Message-ID: Im having issues with altq on netbsd and considering moving to gentoo. Does anyone have any success to report compiling altq on the powermac 604e(v) chipset? severly questioning : "Microsoft: Where do you want to go tomorrow? Linux: Where do you want to go today? BSD: Are you guys coming, or what? " Robin-David Hammond KB3IEN www.aresnyc.org. From alvarolmmotta at gmail.com Wed Jul 20 19:42:11 2005 From: alvarolmmotta at gmail.com (Alvaro Motta) Date: Wed Jul 20 19:42:57 2005 Subject: [LARTC] Transfer rate above the desired (tc+htb) Message-ID: <3941d81c0507201042563b5e45@mail.gmail.com> Hi folks. I started to play with tc+htb last week, and I must confess that this thing is really driving me nuts. All we want to do is control bw, with no borrowing. In order to get the feeling on this subject, I have setup the following test bed. ---A---B---C--- On B: eth0 connecting A and eth1 connecting C. The script. tc qdisc del dev eth0 root tc qdisc add dev eth0 root handle 1: htb default 50 tc class add dev eth0 parent 1: classid 1:1 htb rate 32kbit ceil 32kbit tc filter add dev eth0 protocol ip parent 1:0 prio 100 u32 match ip src 10.4.0.0/16 match ip dst 0.0.0.0/0 classid 1:1 If I try to transfer a 1M file from C to A: [root@localpost tmp]# wget 192.168.0.23/1M --09:22:32-- http://192.168.0.23/1M => `1M.8' Connecting to 192.168.0.23:80... connected. HTTP request sent, awaiting response... 200 OK Length: 1,024,000 [text/plain] 100%[=====================>] 1,024,000 183.12K/s ETA 00:00 09:22:38 (182.88 KB/s) - `1M' saved [1,024,000/1,024,000] Wasn't it supposed to be around the 32KB/s? If I play with the numbers (rate=ceil) I get the following results: 128k ==> 404.78 KB/s 64k ==> 337.9 KB/s 16k ==> 68.86 KB/s 8k ==> 31.12 KB/s 1k ==> 3.77 KB/s I even tried to set the rate to 1kbps in root, but also led to pretty much the same results. With no qdisc, the rate will go close to 1000 KB/s B machine: 2.6.11-1.1369_FC4 iproute-2.6.11-1 TC HTB version 3.3 I have no clue on what I am doing wrong. Could anyone browse the above script and give me hint? Thanks in advance, AL From andy.furniss at dsl.pipex.com Wed Jul 20 20:56:03 2005 From: andy.furniss at dsl.pipex.com (Andy Furniss) Date: Wed Jul 20 20:55:07 2005 Subject: [LARTC] Transfer rate above the desired (tc+htb) In-Reply-To: <3941d81c0507201042563b5e45@mail.gmail.com> References: <3941d81c0507201042563b5e45@mail.gmail.com> Message-ID: <42DE9E43.7@dsl.pipex.com> Alvaro Motta wrote: > Hi folks. > > I started to play with tc+htb last week, and I must confess that this > thing is really driving me nuts. > > All we want to do is control bw, with no borrowing. > > In order to get the feeling on this subject, I have setup the > following test bed. > > ---A---B---C--- > > On B: eth0 connecting A and eth1 connecting C. > > The script. > > tc qdisc del dev eth0 root > tc qdisc add dev eth0 root handle 1: htb default 50 > tc class add dev eth0 parent 1: classid 1:1 htb rate 32kbit ceil 32kbit > tc filter add dev eth0 protocol ip parent 1:0 prio 100 u32 match ip > src 10.4.0.0/16 match ip dst 0.0.0.0/0 classid 1:1 Should be src 192.168.0.0/24. Andy. > > If I try to transfer a 1M file from C to A: > > [root@localpost tmp]# wget 192.168.0.23/1M > --09:22:32-- http://192.168.0.23/1M => `1M.8' > Connecting to 192.168.0.23:80... connected. > HTTP request sent, awaiting response... 200 OK > Length: 1,024,000 [text/plain] > 100%[=====================>] 1,024,000 183.12K/s ETA 00:00 > 09:22:38 (182.88 KB/s) - `1M' saved [1,024,000/1,024,000] > > Wasn't it supposed to be around the 32KB/s? > > If I play with the numbers (rate=ceil) I get the following results: > 128k ==> 404.78 KB/s > 64k ==> 337.9 KB/s > 16k ==> 68.86 KB/s > 8k ==> 31.12 KB/s > 1k ==> 3.77 KB/s > > I even tried to set the rate to 1kbps in root, but also led to pretty > much the same results. > > With no qdisc, the rate will go close to 1000 KB/s > > B machine: > 2.6.11-1.1369_FC4 > iproute-2.6.11-1 > TC HTB version 3.3 > > I have no clue on what I am doing wrong. Could anyone browse the above > script and give me hint? > > Thanks in advance, > > AL > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > From andy.furniss at dsl.pipex.com Wed Jul 20 20:59:06 2005 From: andy.furniss at dsl.pipex.com (Andy Furniss) Date: Wed Jul 20 20:58:10 2005 Subject: [LARTC] TBF + burst In-Reply-To: <20050720014202.17077.qmail@web34410.mail.mud.yahoo.com> References: <20050720014202.17077.qmail@web34410.mail.mud.yahoo.com> Message-ID: <42DE9EFA.60702@dsl.pipex.com> js si wrote: > I am using tbf to do rate limitation. i set the rate, > burst and latency parameters for tbf. but when i start > passing data i notice that there is an initial burst > and then the rate goes down to the configured level. > is this because tbf starts with a full bucket? Well I suppose the burst starts full. Remember if you are looking at tc stats then they show enqueue not dequeue, so you would see the whole buffer filling, but the dequeue rate will be lower. Andy. From andy.furniss at dsl.pipex.com Wed Jul 20 21:27:21 2005 From: andy.furniss at dsl.pipex.com (Andy Furniss) Date: Wed Jul 20 21:26:21 2005 Subject: [LARTC] HELP PLEASE BITTORRENT SHAPING (HTB) In-Reply-To: <200507070535.29506.donvodka@gmail.com> References: <200507070535.29506.donvodka@gmail.com> Message-ID: <42DEA599.50208@dsl.pipex.com> Edgar wrote: I've never used ipp2p/l7 connmark or tcng, but usually when using connmark I would expect to see --restore-mark somewhere. If the PC running bittorrent is linux+python a workaround would be to add another address to its eth and -bind bittorrent to that. I don't know if thats possible with other OS/bt software. Andy. From THnyk at seznam.cz Wed Jul 20 21:29:22 2005 From: THnyk at seznam.cz (=?iso-8859-2?Q?Tom=E1=B9=20Hnyk?=) Date: Wed Jul 20 21:29:25 2005 Subject: [LARTC] script for limiting uploaddoes not work Message-ID: <1589.2239-25957-1838108460-1121887762@seznam.cz> Hello, I want only a very simple thing, but somehow I cannot manage to get it working. I would like to shape download and upload speeds of my DC++ client firmly to some speed, but I cannot make the upload part working. I use a linux router (asus wl500g, actually), and a NATed PC (adress 192.168.1.100) eth1 is my WAN interface, br0 is LAN, DC++ is active and running on ports(udp and tcp) 4111. This is my script (working) for download: iptables -t mangle -A FORWARD -p tcp --dport 4111 -j MARK --set-mark 1 tc qdisc add dev br0 root handle 1:0 htb default 0 tc class add dev br0 parent 1:0 classid 1:1 htb rate 1024kbit tc class add dev br0 parent 1:1 classid 1:11 htb rate 256kbit tc filter add dev br0 parent 1:0 protocol ip handle 1 fw flowid 1:11 I thought that upload would be very similar: iptables -t mangle -A FORWARD -p tcp --sport 4111 -j MARK --set-mark 2 tc qdisc add dev eth1 root handle 2:0 htb default 0 tc class add dev eth1 parent 2:0 classid 2:1 htb rate 1024kbit tc class add dev eth1 parent 2:1 classid 2:11 htb rate 256kbit tc filter add dev eth1 parent 2:0 protocol ip handle 2 fw flowid 2:11 but it does not work. tc part is actually ok, because if I type this: iptables -t mangle -A FORWARD -i br0 -j MARK --set-mark 2 or this: iptables -t mangle -D FORWARD -s 192.168.1.100 -j MARK --set-mark 2 upload goes to 64kbit, but do does everything else too - which is not exactly what I am trying to do:-). So the problem si something with marking and iptables - any ideas? (or even a completle different approach to the problem then marking packets?) Thanks for help in advance. Tomas From THnyk at seznam.cz Wed Jul 20 21:39:52 2005 From: THnyk at seznam.cz (=?iso-8859-2?Q?Tom=E1=B9=20Hnyk?=) Date: Wed Jul 20 21:39:56 2005 Subject: [LARTC] limiting upload speeds does not work Message-ID: <1592.2236-26961-1453642668-1121888392@seznam.cz> Hello, I want only a very simple thing, but I have been unable to get it working so far. I would like to shape my DC++ traffic firmly to certain speeds. I am using an embedded linux router (asus wl500g), and a NATed PC (address 192.168.1.100). eth1 is a WAN interface and br0 is the LAN interface of the router.DC++ client is running on 4111 ports(tcp and udp) I actually managed to shape download wth this script: iptables -t mangle -A FORWARD -p tcp --dport 4111 -j MARK --set-mark 1 tc qdisc add dev br0 root handle 1:0 htb default 0 tc class add dev br0 parent 1:0 classid 1:1 htb rate 1024kbit tc class add dev br0 parent 1:1 classid 1:11 htb rate 256kbit tc filter add dev br0 parent 1:0 protocol ip handle 1 fw flowid 1:11 And I thought it would be similar with upload, but it is not, I tried this script: iptables -t mangle -A FORWARD -p tcp --sport 4111 -j MARK --set-mark 2 tc qdisc add dev eth1 root handle 2:0 htb default 0 tc class add dev eth1 parent 2:0 classid 2:1 htb rate 1024kbit tc class add dev eth1 parent 2:1 classid 2:11 htb rate 256kbit tc filter add dev eth1 parent 2:0 protocol ip handle 2 fw flowid 2:11 but it did not work. However, the problem seems to be in the way how I mark packets since when I type this: iptables -t mangle -A FORWARD -i br0 -j MARK --set-mark 2 or this: iptables -t mangle -A FORWARD -s 192.168.1.100 -j MARK --set-mark 2 upload immidiately goes to 64kbits or something, but so does everything else too - which is not exactly what I am trying to accomplish;-). Since I am running out of ideas, have you got any? Or even an completely different approach to the problem? Thanks in advance, Tomas From THnyk at seznam.cz Wed Jul 20 21:43:26 2005 From: THnyk at seznam.cz (=?iso-8859-2?Q?Tom=E1=B9=20Hnyk?=) Date: Wed Jul 20 21:43:29 2005 Subject: [LARTC] sorry for sending this twice Message-ID: <1592.2236-30068-536738526-1121888606@seznam.cz> My opera browser crashed and I somehow missed the mail was actually already sent, so I wrote it again. Sorry for two almost identical mails. Tomas From alvarolmmotta at gmail.com Wed Jul 20 22:11:41 2005 From: alvarolmmotta at gmail.com (Alvaro Motta) Date: Wed Jul 20 22:12:19 2005 Subject: [LARTC] Transfer rate above the desired (tc+htb) In-Reply-To: <42DE9E43.7@dsl.pipex.com> References: <3941d81c0507201042563b5e45@mail.gmail.com> <42DE9E43.7@dsl.pipex.com> Message-ID: <3941d81c05072013112112c1f6@mail.gmail.com> Hi Andy, thanks for your reply. I don't see why the src should be the culprit, since the AB segment is 10.4 network and the BC is 192.168. And IMHO 0.0.0.0/0 Also, after modifying the src, the traffic rate was the same as if no qdisc were attached to the interface. I even played with the interfaces and the only way to throttle the traffic, is assigning the qdisc to the eth0 and having the src and dst as in the script I've sent. AL On 7/20/05, Andy Furniss wrote: > Alvaro Motta wrote: > > Hi folks. > > > > I started to play with tc+htb last week, and I must confess that this > > thing is really driving me nuts. > > > > All we want to do is control bw, with no borrowing. > > > > In order to get the feeling on this subject, I have setup the > > following test bed. > > > > ---A---B---C--- > > > > On B: eth0 connecting A and eth1 connecting C. > > > > The script. > > > > tc qdisc del dev eth0 root > > tc qdisc add dev eth0 root handle 1: htb default 50 > > tc class add dev eth0 parent 1: classid 1:1 htb rate 32kbit ceil 32kbit > > tc filter add dev eth0 protocol ip parent 1:0 prio 100 u32 match ip > > src 10.4.0.0/16 match ip dst 0.0.0.0/0 classid 1:1 > > Should be src 192.168.0.0/24. > > Andy. > > > > > > If I try to transfer a 1M file from C to A: > > > > [root@localpost tmp]# wget 192.168.0.23/1M > > --09:22:32-- http://192.168.0.23/1M => `1M.8' > > Connecting to 192.168.0.23:80... connected. > > HTTP request sent, awaiting response... 200 OK > > Length: 1,024,000 [text/plain] > > 100%[=====================>] 1,024,000 183.12K/s ETA 00:00 > > 09:22:38 (182.88 KB/s) - `1M' saved [1,024,000/1,024,000] > > > > Wasn't it supposed to be around the 32KB/s? > > > > If I play with the numbers (rate=ceil) I get the following results: > > 128k ==> 404.78 KB/s > > 64k ==> 337.9 KB/s > > 16k ==> 68.86 KB/s > > 8k ==> 31.12 KB/s > > 1k ==> 3.77 KB/s > > > > I even tried to set the rate to 1kbps in root, but also led to pretty > > much the same results. > > > > With no qdisc, the rate will go close to 1000 KB/s > > > > B machine: > > 2.6.11-1.1369_FC4 > > iproute-2.6.11-1 > > TC HTB version 3.3 > > > > I have no clue on what I am doing wrong. Could anyone browse the above > > script and give me hint? > > > > Thanks in advance, > > > > AL > > _______________________________________________ > > LARTC mailing list > > LARTC@mailman.ds9a.nl > > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > > > > From nik39_nospam at usenet.entertain-men.de Wed Jul 20 22:42:12 2005 From: nik39_nospam at usenet.entertain-men.de (nik-da-39) Date: Wed Jul 20 22:42:20 2005 Subject: [LARTC] Problem with HTB and ceil In-Reply-To: <200507192304.02145.Andreas.Klauer@metamorpher.de> References: <1121752323.42dc9503a9c8d@www.domainfactory-webmail.de> <200507191608.53800.Andreas.Klauer@metamorpher.de> <1121805706.42dd658a91eba@www.domainfactory-webmail.de> <200507192304.02145.Andreas.Klauer@metamorpher.de> Message-ID: <1121892132.42deb724bf8ba@www.domainfactory-webmail.de> Quoting Andreas Klauer : > I might've misunderstood you here, but just in case: > > The HTB prio parameter affects borrowing priority, not packet priority. You mean priority after the guaranteed bitrates have been fullfilled. > Which basically means that the proportions which you'd get if all classes > had the same priority, will look much different. > > In other words, low priority classes will not get any bandwidth except > their guaranteed rate, as long as a higher priority class is borrowing. More precisely, after the guaranteed bandwidhts have been assigned the excess bandwidth will be spread to the highest prio class first until the ceiling is reached and then any remainin will be assigne to the next highes prio class and the same procedure is repeated until the bandwidht is 0. Correct? If so, that is exactly what I am trying to achieve. > Personally, I only use the HTB prio for unwanted traffic class (P2P), > because I find that for normal traffic, the restriction is too harsh. > This is why I recommend not using the HTB prio parameter at all in the > beginning, and only start experimenting with it when you really need it. I think I really need it ;-) But I have observed something else, it looks like if I use two different IP A and B where I sent packets to, and mark those packets with the correct different marks, I can see how the prio works, one of the flows is almost drown by the other as expected from my prio config. So, it _might_ be a problem in the way how I tag the packets. If you remember I had the ACK class which should have prio 1, and I matched them with following rules: iptables -A POSTROUTING -t mangle -o $EXTIF -p tcp -m tcp --tcp-flags SYN,RST,ACK ACK -m length --length :128 -j MARK --set-mark 200 iptables -A POSTROUTING -t mangle -o $EXTIF -p tcp -m tcp --tcp-flags ! SYN,RST,ACK ACK -m length --length :128 -j MARK --set-mark 200 iptables -A POSTROUTING -t mangle -o $EXTIF -p tcp -m length --length 0:48 -j MARK --set-mark 200 Is it possible that still some ACKs are not marked correctly? And if so, how would such a set of rules look like to catch all ACK packets? Thanks for your help! nik From andy.furniss at dsl.pipex.com Wed Jul 20 23:11:36 2005 From: andy.furniss at dsl.pipex.com (Andy Furniss) Date: Wed Jul 20 23:10:37 2005 Subject: [LARTC] Transfer rate above the desired (tc+htb) In-Reply-To: <3941d81c05072013112112c1f6@mail.gmail.com> References: <3941d81c0507201042563b5e45@mail.gmail.com> <42DE9E43.7@dsl.pipex.com> <3941d81c05072013112112c1f6@mail.gmail.com> Message-ID: <42DEBE08.8090004@dsl.pipex.com> Alvaro Motta wrote: > Hi Andy, thanks for your reply. > > I don't see why the src should be the culprit, since the AB segment is > 10.4 network and the BC is 192.168. And IMHO 0.0.0.0/0 > > Also, after modifying the src, the traffic rate was the same as if no > qdisc were attached to the interface. I even played with the > interfaces and the only way to throttle the traffic, is assigning the > qdisc to the eth0 and having the src and dst as in the script I've Hmm I am confused now :-) If you run wget on machine A with address 10.4.x.y and request a file from machine C address 192.168.0.23 then the source address of the packets passing egress eth0 on machine B should be 192.168.0.23 - unless there is some sort of NAT going on in machine B. FWIW you use default 50 on your htb rule but don't have a class 1:50 and your filter rule dst 0.0.0.0/0 matches any dst ipaddress and so is redundant. Maybe you should as a test limit all IP traffic on eth0 and see if that works - tc qdisc del dev eth0 root tc qdisc add dev eth0 root handle 1: htb tc class add dev eth0 parent 1: classid 1:1 htb rate 32kbit ceil 32kbit tc filter add dev eth0 protocol ip parent 1:0 prio 100 u32 match u32 0 0 classid 1:1 Andy. From andy.furniss at dsl.pipex.com Thu Jul 21 00:01:55 2005 From: andy.furniss at dsl.pipex.com (Andy Furniss) Date: Thu Jul 21 00:00:54 2005 Subject: [LARTC] limiting upload speeds does not work In-Reply-To: <1592.2236-26961-1453642668-1121888392@seznam.cz> References: <1592.2236-26961-1453642668-1121888392@seznam.cz> Message-ID: <42DEC9D3.6090506@dsl.pipex.com> Tom?? Hnyk wrote: > Hello, I want only a very simple thing, but I have been unable to get it working so far. I would like to shape my DC++ traffic firmly to certain speeds. I am using an embedded linux router (asus wl500g), and a NATed PC (address 192.168.1.100). eth1 is a WAN interface and br0 is the LAN interface of the router.DC++ client is running on 4111 ports(tcp and udp) I actually managed to shape download wth this script: > iptables -t mangle -A FORWARD -p tcp --dport 4111 -j MARK --set-mark 1 > tc qdisc add dev br0 root handle 1:0 htb default 0 > tc class add dev br0 parent 1:0 classid 1:1 htb rate 1024kbit > tc class add dev br0 parent 1:1 classid 1:11 htb rate 256kbit > tc filter add dev br0 parent 1:0 protocol ip handle 1 fw flowid 1:11 > And I thought it would be similar with upload, but it is not, I tried this script: > iptables -t mangle -A FORWARD -p tcp --sport 4111 -j MARK --set-mark 2 > tc qdisc add dev eth1 root handle 2:0 htb default 0 > tc class add dev eth1 parent 2:0 classid 2:1 htb rate 1024kbit > tc class add dev eth1 parent 2:1 classid 2:11 htb rate 256kbit > tc filter add dev eth1 parent 2:0 protocol ip handle 2 fw flowid 2:11 > but it did not work. However, the problem seems to be in the way how I mark packets since when I type this: > iptables -t mangle -A FORWARD -i br0 -j MARK --set-mark 2 > or this: > iptables -t mangle -A FORWARD -s 192.168.1.100 -j MARK --set-mark 2 > upload immidiately goes to 64kbits or something, but so does everything else too - which is not exactly what I am trying to accomplish;-). > Since I am running out of ideas, have you got any? Or even an completely different approach to the problem? > Thanks in advance, > Tomas Probably ports alone are not enough, I suppose DC will make outgoing connections so src port will be anything and dst port will be whatever the peer chose to run DC on. There are projects called ipp2p and l7 filter that are made to match P2P - I don't use them myself so can't say if they work well with DC. Another way would be to mark the traffic from 192.168.1.100 that you know is not from DC and then mark any unmarked packets as DC. Andy. From fpereira at lojan.com Thu Jul 21 01:14:08 2005 From: fpereira at lojan.com (Francisco Pereira) Date: Thu Jul 21 01:13:46 2005 Subject: [LARTC] Transfer rate above the desired (tc+htb) In-Reply-To: <3941d81c0507201042563b5e45@mail.gmail.com> References: <3941d81c0507201042563b5e45@mail.gmail.com> Message-ID: <42DEDAC0.4020302@lojan.com> Alvaro Motta wrote: > Hi folks. Hola. > I started to play with tc+htb last week, and I must confess that this > thing is really driving me nuts. If you started last week, you have a loooong way to go.... :-) > All we want to do is control bw, with no borrowing. > > In order to get the feeling on this subject, I have setup the > following test bed. > > ---A---B---C--- > > On B: eth0 connecting A and eth1 connecting C. > > The script. > > tc qdisc del dev eth0 root > tc qdisc add dev eth0 root handle 1: htb default 50 > tc class add dev eth0 parent 1: classid 1:1 htb rate 32kbit ceil 32kbit > tc filter add dev eth0 protocol ip parent 1:0 prio 100 u32 match ip > src 10.4.0.0/16 match ip dst 0.0.0.0/0 classid 1:1 > > If I try to transfer a 1M file from C to A: > > [root@localpost tmp]# wget 192.168.0.23/1M > --09:22:32-- http://192.168.0.23/1M => `1M.8' > Connecting to 192.168.0.23:80... connected. > HTTP request sent, awaiting response... 200 OK > Length: 1,024,000 [text/plain] > 100%[=====================>] 1,024,000 183.12K/s ETA 00:00 > 09:22:38 (182.88 KB/s) - `1M' saved [1,024,000/1,024,000] > > Wasn't it supposed to be around the 32KB/s? Around 32 kilobits/s. Besides this, the rate calculation includes not only the ip packet effective payload, but all the packet size, and I guess that wget's speed calculation only includes the payload. The tc manpage have a section on "Units". > If I play with the numbers (rate=ceil) I get the following results: > 128k ==> 404.78 KB/s > 64k ==> 337.9 KB/s > 16k ==> 68.86 KB/s > 8k ==> 31.12 KB/s > 1k ==> 3.77 KB/s > > I even tried to set the rate to 1kbps in root, but also led to pretty > much the same results. > > With no qdisc, the rate will go close to 1000 KB/s > > B machine: > 2.6.11-1.1369_FC4 > iproute-2.6.11-1 > TC HTB version 3.3 > > I have no clue on what I am doing wrong. Could anyone browse the above > script and give me hint? > > Thanks in advance, > > AL > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > > From andy.furniss at dsl.pipex.com Thu Jul 21 02:18:35 2005 From: andy.furniss at dsl.pipex.com (Andy Furniss) Date: Thu Jul 21 02:17:34 2005 Subject: [LARTC] ppp uplink shaping problems In-Reply-To: <42DB59AF.60106@web.de> References: <70198d7c050717093865e5dacb@mail.gmail.com> <42DB59AF.60106@web.de> Message-ID: <42DEE9DB.5020608@dsl.pipex.com> Staenker wrote: > Hello, > > i played a few days with tc htb classes and classified my packets using > iptables CLASSIFY target. > > here is what i did: > #!/bin/bash > int='ppp0' > #making all things clear > tc qdisc del dev $int root > iptables -t mangle --flush > iptables -t mangle --delete-chain > > if $1 > then > #defining classes > tc qdisc add dev $int root handle 1: htb default 20 r2q 2 > tc class add dev $int parent 1: classid 1:1 htb rate 22kbps > > tc class add dev $int parent 1:1 classid 1:10 htb rate 10kbps ceil > 22kbps prio 0 > tc class add dev $int parent 1:1 classid 1:20 htb rate 9kbps ceil 15kbps > prio 1 > tc class add dev $int parent 1:1 classid 1:30 htb rate 3kbps ceil 13kbps > prio 2 > tc qdisc add dev $int parent 1:10 handle 10: sfq perturb 10 > tc qdisc add dev $int parent 1:20 handle 20: sfq perturb 10 > tc qdisc add dev $int parent 1:30 handle 30: sfq perturb 10 > > iptables -t mangle -N TS_FWD > iptables -t mangle -A FORWARD -j TS_FWD > > iptables -t mangle -A TS_FWD -o ppp0 -p ! icmp --match length --length > 0:70 -j CLASSIFY --set-class 1:10 > iptables -t mangle -A TS_FWD -o ppp0 -p ! icmp --match length --length > 0:70 -j RETURN > > iptables -t mangle -A TS_FWD -i eth2 -o ppp0 -p tcp --source 192.168.0.2 > --destination-port 80 -j CLASSIFY --set-class 1:20 > iptables -t mangle -A TS_FWD -i eth2 -o ppp0 -p tcp --source 192.168.0.2 > --destination-port 80 -j RETURN > iptables -t mangle -A TS_FWD -i eth2 -o ppp0 -p tcp --source 192.168.0.2 > --destination-port 443 -j CLASSIFY --set-class 1:20 > iptables -t mangle -A TS_FWD -i eth2 -o ppp0 -p tcp --source 192.168.0.2 > --destination-port 443 -j RETURN > > iptables -t mangle -A TS_FWD -i eth2 -o ppp0 -p tcp --source 192.168.0.2 > --destination-port 554 -j CLASSIFY --set-class 1:10 > iptables -t mangle -A TS_FWD -i eth2 -o ppp0 -p tcp --source 192.168.0.2 > --destination-port 554 -j RETURN > #if $2 > #then > # iptables -t mangle -A TS_FWD -i eth2 -o ppp0 --source 192.168.0.2 > -j LOG > #fi > iptables -t mangle -A TS_FWD -i eth2 -o ppp0 --source 192.168.0.2 -j > CLASSIFY --set-class 1:30 > fi > > It works not really good. I tested it using my internal 100MBit network > interface using multiple ftp connections and classified the packets > based on their source-ip. That works fine with same classes. Immediately > all things i expected took place. Also the prio option worked fine. If i > was running 2 simultanious downloads, the one with the higher piority > gets all borrowable downloadspeed and the one with the lower priority > gets his ashured rate. > But same classes didnt work with my 192kbit 2048kbit ppp link. Well ok, > they are working, but not like i want them to work. The speed changes > takes some seconds to take place. And the priority seems to be ignored. > I have to say, that the i tested the ppp uplink using emule with many > connections (500 - 800) and the higher priority upload was one active > ftp connection. > > Whats my fault? I can't see what rule seperates ftp from the rest, so that could be it - check counters/classification with tc -s class ls dev ppp0 If ftp is going to the right class then it may be that emule network traffic consists of loads of small packets. If your ppp0 is adsl then the difference between the ip length that htb sees and the actual length used on the wire can be significant enough to make you go over limits and end up with a 2+ second queue in your modem. Check with ping and make icmp go to top priority class. It is possible to patch tc/kernel to allow for this. Andy. From nik39_nospam at usenet.entertain-men.de Thu Jul 21 06:06:59 2005 From: nik39_nospam at usenet.entertain-men.de (nik-da-39) Date: Thu Jul 21 06:07:02 2005 Subject: [LARTC] HTB and prio Message-ID: <1121918819.42df1f6334b10@www.domainfactory-webmail.de> Hi, this is a followup to "Problem with HTB and ceil", the problem with the ceiling has been resolved, thanks to the memebers on this m/g. Now I have the problem that packets get into the correct classe with the correct marks but the prios are not respected. I have 4 different classes: 1. voip class / prio 0 / mark 100 / voip traffic 2. ACK class / prio 1 / mark 200 / ACK packets to keep downloads as fast as possible 3. SSH class / prio 3 / mark 300 / ssh/scp interactive traffic 4. http/client class / prio 3 / mark 400 / traffic from the internal net/web browsing traffic 5. bulk classe / prio 4 / mark 500 / bulk traffic like p2p stuff etc, also locally from the router All the minium settings are respected, same goes for ceiling. I would expect that after assuring that the guaranteed minimum bitrates have been assigned, that the excess bandwidth will first be given to the class with the highest prio until the ceiling has been reached, and if there is any bandwidth left, proceed with the class with the next lower prio. That doesnt happen, any excess b/w will be spread all across the classes/prios. Now here is the script: -------------------------------------------- #!/bin/bash # # Shell-Skript fuer Quality of Service mit HTB # EXTIF=ppp0 INTIF=eth0 if [ "$1" = "start" ]; then ############ # Outgoing ############ ## Root /sbin/tc qdisc add dev $EXTIF root handle 1:0 htb default 400 ## Hauptklasse /sbin/tc class add dev $EXTIF parent 1:0 classid 1:1 htb rate 380kbit ceil 380kbit /sbin/tc class add dev $EXTIF parent 1:1 classid 1:2 htb rate 380kbit ceil 380kbit ## voip class /sbin/tc class add dev $EXTIF parent 1:2 classid 1:100 htb rate 32kbit ceil 96kbit prio 0 ## non voip class /sbin/tc class add dev $EXTIF parent 1:2 classid 1:101 htb rate 248kbit ceil 380kbit ## ACK class /sbin/tc class add dev $EXTIF parent 1:101 classid 1:200 htb rate 200kbit ceil 380kbit prio 1 ## VPN/SSH class /sbin/tc class add dev $EXTIF parent 1:101 classid 1:300 htb rate 16kbit ceil 380kbit prio 3 ## http class /sbin/tc class add dev $EXTIF parent 1:101 classid 1:400 htb rate 16kbit ceil 380kbit prio 3 ## bulk class /sbin/tc class add dev $EXTIF parent 1:101 classid 1:500 htb rate 16kbit ceil 380kbit prio 4 ################### # Marking ################### # default marking *500 iptables -A POSTROUTING -t mangle -o $EXTIF -j MARK --set-mark 500 # voip *100 iptables -A POSTROUTING -t mangle -o $EXTIF -p udp --sport 5060:5061 -m length --length 0:45 -j MARK --set-mark 100 iptables -A POSTROUTING -t mangle -o $EXTIF -p udp -s 172.20.1.200 -j MARK --set-mark 100 # ACK+dns *200 iptables -A POSTROUTING -t mangle -o $EXTIF -p tcp -m tcp --tcp-flags SYN,RST,ACK ACK -j MARK --set-mark 200 iptables -A POSTROUTING -t mangle -o $EXTIF -p udp --sport 53 -j MARK --set-mark 200 iptables -A POSTROUTING -t mangle -o $EXTIF -p udp --dport 53 -j MARK --set-mark 200 # SSH *300 iptables -A POSTROUTING -t mangle -o $EXTIF -p tcp --dport 22 -j MARK --set-mark 300 iptables -A POSTROUTING -t mangle -o $EXTIF -p tcp --sport 22 -j MARK --set-mark 300 # http *400 iptables -A POSTROUTING -t mangle -o $EXTIF -p tcp --dport 80 -j MARK --set-mark 400 iptables -A POSTROUTING -t mangle -o $EXTIF -p tcp -s 172.20.1.0/24 -j MARK --set-mark 400 # bulk *500 iptables -A POSTROUTING -t mangle -o $EXTIF -p icmp -j MARK --set-mark 500 tc filter add dev $EXTIF parent 1:0 prio 0 protocol ip handle 1 fw flowid 1:1 tc filter add dev $EXTIF parent 1:0 prio 0 protocol ip handle 2 fw flowid 1:2 tc filter add dev $EXTIF parent 1:0 prio 0 protocol ip handle 100 fw flowid 1:100 tc filter add dev $EXTIF parent 1:0 prio 0 protocol ip handle 101 fw flowid 1:101 tc filter add dev $EXTIF parent 1:0 prio 0 protocol ip handle 200 fw flowid 1:200 tc filter add dev $EXTIF parent 1:0 prio 0 protocol ip handle 300 fw flowid 1:300 tc filter add dev $EXTIF parent 1:0 prio 0 protocol ip handle 400 fw flowid 1:400 tc filter add dev $EXTIF parent 1:0 prio 0 protocol ip handle 500 fw flowid 1:500 ####### # SFQ ####### #tc qdisc add dev $EXTIF parent 1:2 handle 2: sfq perturb 10 tc qdisc add dev $EXTIF parent 1:100 handle 100: sfq perturb 10 #tc qdisc add dev $EXTIF parent 1:101 handle 101: sfq perturb 10 tc qdisc add dev $EXTIF parent 1:200 handle 200: sfq perturb 10 tc qdisc add dev $EXTIF parent 1:300 handle 300: sfq perturb 10 tc qdisc add dev $EXTIF parent 1:400 handle 400: sfq perturb 10 tc qdisc add dev $EXTIF parent 1:500 handle 500: sfq perturb 10 fi -------------------------------------------- o root 1:0 | + 1:1 | ________________+ 1:2 / | | | 1:101 | __________+__________ | | | | | | | | | | | | | | | voip ACK SSH http bulk pr 0 pr 1 pr 2 pr 3 pr 4 1:100 1:200 1:300 1:300 1:400 Does anyone have a clue what I am doing wrong? (1:1 is not needed, I know, I wanted to attach another subtree to 1:1 which I havent done yet, thats why its still listed) Any help is appreciated. nik From dmandrioli.ext at rd.francetelecom.com Thu Jul 21 13:20:46 2005 From: dmandrioli.ext at rd.francetelecom.com (zze-stagiaire MANDRIOLI D ext RD-BIZZ-SOP) Date: Thu Jul 21 13:24:50 2005 Subject: [LARTC] New disciplines, new filters : How to writing new linux/net/sched stuffs Message-ID: <9772C290CD0BDF4B91356C9102BA886A01851644@ftrdmel1.rd.francetelecom.fr> Hello, I'm currently searching for any kinds of informations about writing new queue discipline and/or filter modules. I have to write a discipline and/or filter to implement an experimental stateful router which maintain a table for TCP connections. The original code from A. Kuznetsov is provided without comments. It would be wonderful to have a version with comments, do you have any? I'm interested by anything which can help me to implement net/sched stuffs. Thanks, Damien. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ds9a.nl/pipermail/lartc/attachments/20050721/a0a5a387/attachment.htm From fredrik at cocacolic.org Fri Jul 22 04:27:36 2005 From: fredrik at cocacolic.org (Fredrik Bredeli) Date: Fri Jul 22 04:28:03 2005 Subject: [LARTC] tc filter matching anything Message-ID: <20050722022736.GA21450@cocacolic.org> Hey, I have a problem adding a filter matching anything, here is my setup: ############### tc qdisc add dev eth0 handle 1: root prio tc qdisc add dev eth0 parent 1:1 handle 10: pfifo tc qdisc add dev eth0 parent 1:2 handle 20: tbf latency 50ms rate \ 128kbit burst 256kbit tc filter add dev eth0 protocol ip parent 1:0 prio 1 handle 666 fw \ flowid 1:2 tc filter add dev eth0 protocol ip parent 1:0 prio 2 flowid 1:1 ##################### This is supposed to put all traffic marked by iptables in the tbf qdisc, but the last line gives me this error: error: Unknown filter "flowid", hence option "1:1" is unparsable I'm running Linux 2.4.27-2-686 with iptables v1.2.11 on debian sarge Does anyone know what is wrong, or have a better solution for my setup? Thank you. Fredrik Bredeli From unki at netshadow.at Fri Jul 22 06:06:44 2005 From: unki at netshadow.at (Andreas Unterkircher) Date: Fri Jul 22 06:06:54 2005 Subject: [LARTC] tc filter matching anything In-Reply-To: <20050722022736.GA21450@cocacolic.org> References: <20050722022736.GA21450@cocacolic.org> Message-ID: <42E070D4.6040800@netshadow.at> You can do a "match anything" with the usage of tc-filter: tc filter add dev eth0 parent 1:0 protocol all u32 match u32 0 0 flowid 1:1 Cheers, Andreas Fredrik Bredeli wrote: >Hey, > >I have a problem adding a filter matching anything, here is my setup: > >############### >tc qdisc add dev eth0 handle 1: root prio > >tc qdisc add dev eth0 parent 1:1 handle 10: pfifo >tc qdisc add dev eth0 parent 1:2 handle 20: tbf latency 50ms rate \ >128kbit burst 256kbit > >tc filter add dev eth0 protocol ip parent 1:0 prio 1 handle 666 fw \ >flowid 1:2 > >tc filter add dev eth0 protocol ip parent 1:0 prio 2 flowid 1:1 >##################### > >This is supposed to put all traffic marked by iptables in the tbf qdisc, >but the last line gives me this error: > >error: Unknown filter "flowid", hence option "1:1" is unparsable > >I'm running Linux 2.4.27-2-686 with iptables v1.2.11 on debian >sarge > >Does anyone know what is wrong, or have a better solution for my setup? > >Thank you. > >Fredrik Bredeli >_______________________________________________ >LARTC mailing list >LARTC@mailman.ds9a.nl >http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > > From christian.konecny at ericsson.com Fri Jul 22 09:55:25 2005 From: christian.konecny at ericsson.com (Christian Konecny (VI/SEA)) Date: Fri Jul 22 09:55:31 2005 Subject: [LARTC] Latency of Linux Bridge Message-ID: Hi there! I am working a lot with VoIP in my company, so I thought to use linux bridge functionality together with tc to emulate delay, jitter, packet loss, duplication, reordering etc. for testing purposes in our lab against our VoIP products. I just recognized, that a basic bridge just with it's minumum configuration of 2 network interfaces creates latency of approx. 5ms on very low traffic. This seems to be independent on CPU speed. I tried on 2 GHz PC while having just 64kBit traffic with packet size of about 300bytes. I am using Knoppix 3.82 which is actually a debian Live-CD Linux, Kernel 2.6.11. For some reason they put iproute2 041019 on this distro, which is intended to be used for kernel 2.6.9. I am aware of remastering the CD, but have to check if it is possible to recompile the kernel for the remaster. back to my question: where does this latency come from? "top" shows almost no load while the bridge is handling traffic, so how come? is there some timer-granularity which can be set in the kernel, is the latency normal, or what could cause it else? Thank you very much in advance! /Christian From imipak at yahoo.com Fri Jul 22 11:12:09 2005 From: imipak at yahoo.com (Jonathan Day) Date: Fri Jul 22 11:12:14 2005 Subject: [LARTC] Latency of Linux Bridge In-Reply-To: Message-ID: <20050722091209.65404.qmail@web31508.mail.mud.yahoo.com> Hi, It becomes possible to play with a bunch of CPU-related timers in the 2.6.13-rc series, which MAY help (but no guarantees). The latest tree also has some scheduling fixes which probably won't make much of a difference to you. Standard distro kernels tend to be compromises, which means they'll be OK for everything but great at nothing. If you want to squeeze every last bit of performance out of the machine, you'll need to do some kernel configuration work. The latest "vanilla" kernel is 2.6.13-rc3 and the latest Andrew Morton release is 2.6.13-rc3-mm1 (The differences that MAY be useful to you is that the -mm release has some driver fixes for ethernet cards.) If you roll your own kernel and are wanting to use it for a bridge setup, my guess is you want to use the server settings for preempt (no forced preemption - ie: pretty much disable it) but would likely want to use the desktop settings for the timer frequency (1000 Hz) as that gives the fastest response to events. (If you're using an SMP machine, 250 Hz might be better, as SMP doesn't like lots of interrupts.) Depending on who you ask and what phase the moon is in, different people give different opinions about whether to compile in or use modules. Compiled-in drivers MAY be marginally faster and MAY eat fractionally less kernel memory, which MIGHT trim down latency a little. If that's not serious voodoo enough, don't compile in any network layers you're not using. Every layer is absolutely going to add latency, because it is extra code to run. Finally, and this is going to be the hardest step, it MAY be possible to get the Linux kernel to compile with the latest Intel C compiler. If you're using a genuine Intel processor, the speedup is something like 40% - for AMD or other ix86 processors, GCC is either equal to or faster than Intel's compiler. The problem with using Intel's C compiler is that it has very different ideas on what is ok than GCC, so the kernel won't necessarily compile. Sometimes people put icc patches out, to fix this, but not all kernels have patches and the kernel of interest is a pre-release, making patches less likely in the event they are needed. Any of these steps should trim a little latency off, and if you somehow manage all of them, you should get quite a decent improvement. Whether the improvement is worth the effort required is another matter. --- "Christian Konecny (VI/SEA)" wrote: > Hi there! > > I am working a lot with VoIP in my company, so I > thought to use linux bridge functionality together > with tc to emulate delay, jitter, packet loss, > duplication, reordering etc. for testing purposes in > our lab against our VoIP products. > I just recognized, that a basic bridge just with > it's minumum configuration of 2 network interfaces > creates latency of approx. 5ms on very low traffic. > This seems to be independent on CPU speed. I tried > on 2 GHz PC while having just 64kBit traffic with > packet size of about 300bytes. > I am using Knoppix 3.82 which is actually a debian > Live-CD Linux, Kernel 2.6.11. > For some reason they put iproute2 041019 on this > distro, which is intended to be used for kernel > 2.6.9. > I am aware of remastering the CD, but have to check > if it is possible to recompile the kernel for the > remaster. > > back to my question: where does this latency come > from? > "top" shows almost no load while the bridge is > handling traffic, so how come? > is there some timer-granularity which can be set in > the kernel, is the latency normal, or what could > cause it else? > > Thank you very much in advance! > > /Christian > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From mihaivlad at web-profile.net Fri Jul 22 11:50:28 2005 From: mihaivlad at web-profile.net (Mihai Vlad) Date: Fri Jul 22 11:50:25 2005 Subject: [LARTC] fw classifier - need ipfwchains patch Message-ID: Hello, I just compiled a new kernel.iptable/iproute2 to fresh up my router. There is a little problem... The fw classifier does not work in tc. I read some note found in the source of iproute2 and it says: "* To use "fw" classifier you will need ipfwchains patch." I can't find this patch. Do you have it? Is there any other way to implement the "fw" classifier's job? Thanks in advance, Mihai VLAD P.S. I use: linux-2.6.12.3 iptables-1.3.2 iproute2-2.6.11-050330 From christian.konecny at ericsson.com Fri Jul 22 13:35:41 2005 From: christian.konecny at ericsson.com (Christian Konecny (VI/SEA)) Date: Fri Jul 22 13:35:46 2005 Subject: [LARTC] Latency of Linux Bridge Message-ID: Thanks a lot for these very detailed tuning tips. I am not sure whether this effort really becomes necesary. Maybe there is still some module active in my distro causing this latency issue... I would highly appreciate if anyone of you could either confirm similar latency issues or report about no latence on your bridge ? Actually I check this with ethereal by making an RTP stream analysis on my voice media stream. The VoIP device generates continously data packets every 30ms. If I capture a trace on either side of the bridge, I can see that the bridge will add jitter of about 5ms. You could use also iperf or something similar to generate traffic. From fredrik at cocacolic.org Fri Jul 22 15:06:02 2005 From: fredrik at cocacolic.org (Fredrik Bredeli) Date: Fri Jul 22 15:06:40 2005 Subject: [LARTC] tc filter matching anything In-Reply-To: <42E070D4.6040800@netshadow.at> References: <20050722022736.GA21450@cocacolic.org> <42E070D4.6040800@netshadow.at> Message-ID: <20050722130602.GA22186@cocacolic.org> > You can do a "match anything" with the usage of tc-filter: > > tc filter add dev eth0 parent 1:0 protocol all u32 match u32 0 0 flowid 1:1 > That is great, thank you. :-) Fredrik Bredeli From ddaasd at gmail.com Fri Jul 22 14:59:09 2005 From: ddaasd at gmail.com (ddaasd) Date: Fri Jul 22 15:51:07 2005 Subject: [LARTC] HTB as a child of another HTB - doesn't work In-Reply-To: <20050722100004.7D74D53FD3@outpost.ds9a.nl> References: <20050722100004.7D74D53FD3@outpost.ds9a.nl> Message-ID: <42E0ED9D.2030400@gmail.com> An HTML attachment was scrubbed... URL: http://mailman.ds9a.nl/pipermail/lartc/attachments/20050722/7ef28e34/attachment.htm From Andreas.Klauer at metamorpher.de Fri Jul 22 16:02:41 2005 From: Andreas.Klauer at metamorpher.de (Andreas Klauer) Date: Fri Jul 22 16:10:13 2005 Subject: [LARTC] HTB as a child of another HTB - doesn't work In-Reply-To: <42E0ED9D.2030400@gmail.com> References: <20050722100004.7D74D53FD3@outpost.ds9a.nl> <42E0ED9D.2030400@gmail.com> Message-ID: <200507221602.41168.Andreas.Klauer@metamorpher.de> On Friday 22 July 2005 14:59, ddaasd wrote: > What do I do wrong? Most likely, you do not want to use more than one HTB qdisc per device. > tc qdisc add dev eth0 root handle 100: htb > tc class add dev eth0 parent 100: classid 100:1 htb rate 100kbps > > tc qdisc add dev eth0 parent 100:1 handle 1: htb What do you need this qdisc for? > tc class add dev eth0 parent 1: classid 1:1 htb rate 100kbps > tc class add dev eth0 parent 1:1 classid 1:10 htb rate 50kbps ceil > 100kbps prio 1 tc class add dev eth0 parent 1:1 classid 1:20 htb rate > 50kbps ceil 100kbps prio 2 Why not make these children of the first qdisc? >
>
> OR
>
> Please don't send HTML mails (or at least include a text-only version). Otherwise it is really awful to read. Thanks. HTH Andreas From c-d.hailfinger.devel.2005 at gmx.net Fri Jul 22 17:28:10 2005 From: c-d.hailfinger.devel.2005 at gmx.net (Carl-Daniel Hailfinger) Date: Fri Jul 22 17:28:19 2005 Subject: [LARTC] Latency of Linux Bridge In-Reply-To: References: Message-ID: <42E1108A.1020404@gmx.net> Hi, Christian Konecny (VI/SEA) schrieb: > I just recognized, that a basic bridge just with it's > minumum configuration of 2 network interfaces creates > latency of approx. 5ms on very low traffic. > This seems to be independent on CPU speed. I tried on > 2 GHz PC while having just 64kBit traffic with packet > size of about 300bytes. That's strange. On my bridge with 4 network interfaces, the additional latency is always below 0.5 ms, even if I'm pushing 400 MBit/s through the machine and a kernel compile is running at 100% CPU. Network interfaces are PCIe GBit from Syskonnect, the machine is an Athlon64 at 2 GHz. Even if the clock speed is halved by powersave the additional latency will not go above 0.9 ms. Kernel is vanilla 2.6.11.x. Regards, Carl-Daniel -- http://www.hailfinger.org/ From xerces8 at butn.net Fri Jul 22 20:03:17 2005 From: xerces8 at butn.net (xerces8) Date: Fri Jul 22 20:03:28 2005 Subject: [LARTC] Update the HOWTO to mention linux 2.6 Message-ID: Hi! The current version of the HOWTO says in Chapter 2. Introduction : This document hopes to enlighten you on how to do more with Linux 2.2/2.4 routing. If the howto works also on linux v2.6 , then it should be mentioned. Regards, David From shemminger at osdl.org Sat Jul 23 01:49:17 2005 From: shemminger at osdl.org (Stephen Hemminger) Date: Sat Jul 23 01:49:16 2005 Subject: [LARTC] Latency of Linux Bridge In-Reply-To: References: Message-ID: <20050722164917.25efb39c@dxpl.pdx.osdl.net> On Fri, 22 Jul 2005 09:55:25 +0200 "Christian Konecny (VI/SEA)" wrote: > Hi there! > > I am working a lot with VoIP in my company, so I thought to use linux > bridge functionality together with tc to emulate delay, jitter, > packet loss, duplication, reordering etc. for testing purposes in our > lab against our VoIP products. I just recognized, that a basic bridge > just with it's minumum configuration of 2 network interfaces creates > latency of approx. 5ms on very low traffic. This seems to be > independent on CPU speed. I tried on 2 GHz PC while having just > 64kBit traffic with packet size of about 300bytes. I am using Knoppix > 3.82 which is actually a debian Live-CD Linux, Kernel 2.6.11. For > some reason they put iproute2 041019 on this distro, which is > intended to be used for kernel 2.6.9. I am aware of remastering the > CD, but have to check if it is possible to recompile the kernel for > the remaster. > > back to my question: where does this latency come from? > "top" shows almost no load while the bridge is handling traffic, so > how come? is there some timer-granularity which can be set in the > kernel, is the latency normal, or what could cause it else? > > Thank you very much in advance! > > /Christian > What hardware, and setup scripts? If you are using netem then it can add latency. The amount depends on the requested delay and the HZ value of the kernel and the choice of PSCHED_CLOCK_SOURCE in the kernel configuration. From ddaasd at gmail.com Sat Jul 23 13:07:22 2005 From: ddaasd at gmail.com (ddaas) Date: Sat Jul 23 13:07:32 2005 Subject: [LARTC] HTB as a child of another HTB - doesn't work Message-ID: <42E224EA.1080409@gmail.com> On Friday 22 July 2005 14:59, ddaasd wrote: >> What do I do wrong? >> >> >>Most likely, you do not want to use more than one HTB qdisc per device. What about the example from the HTB User Guide?? Is it wrong? http://luxik.cdi.cz/~devik/qos/htb/manual/userg.htm#prio >>Please don't send HTML mails (or at least include a text-only >>version). >>Otherwise it is really awful to read. Thanks. Sorry, I didn't know..L :( >>HTH >>Andreas Thanks, ddaas From ddaasd at gmail.com Sat Jul 23 13:14:24 2005 From: ddaasd at gmail.com (ddaas) Date: Sat Jul 23 13:14:35 2005 Subject: [LARTC] TOS in IP Header set to Minimize-Dealay - no difference Message-ID: <42E22690.2030106@gmail.com> Hi I have a 1024/128 ADSL connection. I wanted to test the TOS field in IP Header. So I started aMule and let it use the whole upstream (16KB). Then I did: #iptables -A OUTPUT -t mangle -p tcp --dport 22 -j TOS --set-tos Minimize-Dellay (like here: http://lartc.org/howto/lartc.cookbook.interactive-prio.html) Then I ssh to one of my server on the Internet. The delay was awful. I could use it... So, It seams that the tos field of the iP header has no influence in prioritizing of my outgoing traffic !! Why? Did I miss something? Thanks, ddaas From unki at netshadow.at Sat Jul 23 13:50:10 2005 From: unki at netshadow.at (Andreas Unterkircher) Date: Sat Jul 23 13:50:17 2005 Subject: [LARTC] TOS in IP Header set to Minimize-Dealay - no difference In-Reply-To: <42E22690.2030106@gmail.com> References: <42E22690.2030106@gmail.com> Message-ID: <42E22EF2.6010007@netshadow.at> ddaas wrote: > So, It seams that the tos field of the iP header has no influence in > prioritizing of my outgoing traffic !! Why? Did I miss something? This will not work very well in internet, because most of all routers ignore TOS settings. This is only useful in your own LAN or WAN were you have control over your routers and configure them to respect the TOS settings. For outgoing traffic shaping you should use the common tc functions of linux. Cheers, Andreas From Andreas.Klauer at metamorpher.de Sat Jul 23 13:59:35 2005 From: Andreas.Klauer at metamorpher.de (Andreas Klauer) Date: Sat Jul 23 13:59:47 2005 Subject: [LARTC] HTB as a child of another HTB - doesn't work In-Reply-To: <42E224EA.1080409@gmail.com> References: <42E224EA.1080409@gmail.com> Message-ID: <200507231359.35847.Andreas.Klauer@metamorpher.de> On Saturday 23 July 2005 13:07, ddaas wrote: > >>Most likely, you do not want to use more than one HTB qdisc per > >> device. > > What about the example from the HTB User Guide?? Is it wrong? Ah, you want to simulate a slow link. Didn't realize that, sorry. :-) Have a look on the output of the tc -s -d class show dev $dev statistics. Also try qdisc show instead of class show, the direct_packets_stat should be zero. The packets should go into the class 100:1 (you may have to add "default 1" or a filter that puts them there). So far you only got one filter for the child qdisc, but for that the packets have to get there first. HTH Andreas From zeus_securenet at hotmail.com Sat Jul 23 16:51:43 2005 From: zeus_securenet at hotmail.com (ZeuS SN) Date: Sat Jul 23 16:51:44 2005 Subject: [LARTC] Multiple ISP Provider on one interface Message-ID: <42E2597F.1020808@hotmail.com> The problem is that I have only 2 ethernet adapters in my linux router, one of this adapter pluged to the LAN and the other is pluged to a HUB who have 2 routers DSL. Each DSL routers have an static ip and the solution that I found was creating an eth0 and eth0:1 but with this option I have any way to use the eth0:1 adapte... Any one knows a solution for this ? Regards. From surda at shurdix.com Sat Jul 23 17:57:08 2005 From: surda at shurdix.com (Peter Surda) Date: Sat Jul 23 17:57:33 2005 Subject: [LARTC] Multiple ISP Provider on one interface In-Reply-To: <42E2597F.1020808@hotmail.com> Message-ID: <20056231757826735@mail.routehat.org> On Sat, 23 Jul 2005 15:51:43 +0100 ZeuS SN wrote: >The problem is that I have only 2 ethernet adapters in my linux router, >one of this adapter pluged to the LAN and the other is pluged to a HUB >who have 2 routers DSL. Each DSL routers have an static ip and the >solution that I found was creating an eth0 and eth0:1 but with this >option I have any way to use the eth0:1 adapte... You don't have to use eth0:1, it is actually considered obsolete since kernel 2.2. Use ip addr add ip1/mask1 dev eth0 ip addr add ip2/mask2 dev eth0 As far as I know, modern distributions support this, although I am not sure about the exact syntax in their configuration files. In Shurdix I do it like this: http://docs.shurdix.org/shurdix:network_interfaces#ip >Regards. Yours sincerely, Peter From zeus_securenet at hotmail.com Sat Jul 23 19:09:54 2005 From: zeus_securenet at hotmail.com (ZeuS SN) Date: Sat Jul 23 19:09:53 2005 Subject: [LARTC] Multiple ISP Provider on one interface Message-ID: <42E279E2.4090109@hotmail.com> Ok. It works and now I don't have the problem with ip route and virtual eth, but now my problem continues, If a ping the first IP, who has the default gateway it works fine, but if I ping the second IP that not have gateway, the pakect arrives the machine, but not respond, because it don't know which route get. Theres anyway to set the gateway for this second IP address ? Thanks and Regards From kajtek at biezanow.net Sat Jul 23 21:08:55 2005 From: kajtek at biezanow.net (Kajetan Staszkiewicz) Date: Sat Jul 23 21:08:55 2005 Subject: [LARTC] Multiple ISP Provider on one interface In-Reply-To: <42E2597F.1020808@hotmail.com> References: <42E2597F.1020808@hotmail.com> Message-ID: <200507232108.56641.kajtek@biezanow.net> Dnia sobota, 23 lipca 2005 16:51, ZeuS SN napisa?(a): > The problem is that I have only 2 ethernet adapters in my linux router, > one of this adapter pluged to the LAN and the other is pluged to a HUB > who have 2 routers DSL. Each DSL routers have an static ip and the > solution that I found was creating an eth0 and eth0:1 but with this > option I have any way to use the eth0:1 adapte... > > Any one knows a solution for this ? Multipatch routing probably won't work this way - at least I was unable to set up SNAT because booth gateways are on the same interface. However masquerade was working correctly, as it is choosing ip address to SNAT (AFAIK) after routing. Another solution is to get a switch with 802.1q VLANs, and "divide" single network interface into many fully functional interfaces (fully functional means for me that you can use their names in iptables -o to choose proper SNAT address). -- | ? ? ?pozdrawiam ? ? ?| powered by Slackware, Gentoo and FreeBSD | | Kajetan Staszkiewicz | JID: vegeta@chrome.pl ? ? ? ? ? ? ? ? ? ?| | ? ? ? ?Vegeta ? ? ? ?| http://kajtek.biezanow.net ? ? ? ? ? ? ? | +----------------------+------------------------------------------+ From gypsy at iswest.com Sun Jul 24 20:06:49 2005 From: gypsy at iswest.com (gypsy) Date: Sun Jul 24 20:05:43 2005 Subject: [LARTC] Transfer rate above the desired (tc+htb) References: <3941d81c0507201042563b5e45@mail.gmail.com> <42DE9E43.7@dsl.pipex.com> <3941d81c05072013112112c1f6@mail.gmail.com> Message-ID: <42E3D8B9.F93C29FE@iswest.com> Alvaro Motta wrote: > > Hi Andy, thanks for your reply. > > I don't see why the src should be the culprit, since the AB segment is > 10.4 network and the BC is 192.168. And IMHO 0.0.0.0/0 > > Also, after modifying the src, the traffic rate was the same as if no > qdisc were attached to the interface. I even played with the > interfaces and the only way to throttle the traffic, is assigning the > qdisc to the eth0 and having the src and dst as in the script I've > sent. > > AL Al, Somebody may have already helped you solve this; I have not read all messages from LARTC yet. If not: Create 3 filters, each with the same (non zero) prio. The first to match 10.4.0.0/16, the second to match 192.168.0.0/24 and the last to match 0.0.0.0/0 - then see where the packets go by running tc -s class ls dev eth0 32kbit = 32,000 bits per second. 32,000 / 8 bits per byte = 4,000 bytes per second = 4Kbytes per second, and the reported flow rate should be close to that for the packets that match the filter. -- gypsy > On 7/20/05, Andy Furniss wrote: > > Alvaro Motta wrote: > > > Hi folks. > > > > > > I started to play with tc+htb last week, and I must confess that this > > > thing is really driving me nuts. > > > > > > All we want to do is control bw, with no borrowing. > > > > > > In order to get the feeling on this subject, I have setup the > > > following test bed. > > > > > > ---A---B---C--- > > > > > > On B: eth0 connecting A and eth1 connecting C. > > > > > > The script. > > > > > > tc qdisc del dev eth0 root > > > tc qdisc add dev eth0 root handle 1: htb default 50 > > > tc class add dev eth0 parent 1: classid 1:1 htb rate 32kbit ceil 32kbit > > > tc filter add dev eth0 protocol ip parent 1:0 prio 100 u32 match ip > > > src 10.4.0.0/16 match ip dst 0.0.0.0/0 classid 1:1 > > > > Should be src 192.168.0.0/24. > > > > Andy. > > > > > > > > > > If I try to transfer a 1M file from C to A: > > > > > > [root@localpost tmp]# wget 192.168.0.23/1M > > > --09:22:32-- http://192.168.0.23/1M => `1M.8' > > > Connecting to 192.168.0.23:80... connected. > > > HTTP request sent, awaiting response... 200 OK > > > Length: 1,024,000 [text/plain] > > > 100%[=====================>] 1,024,000 183.12K/s ETA 00:00 > > > 09:22:38 (182.88 KB/s) - `1M' saved [1,024,000/1,024,000] > > > > > > Wasn't it supposed to be around the 32KB/s? > > > > > > If I play with the numbers (rate=ceil) I get the following results: > > > 128k ==> 404.78 KB/s > > > 64k ==> 337.9 KB/s > > > 16k ==> 68.86 KB/s > > > 8k ==> 31.12 KB/s > > > 1k ==> 3.77 KB/s > > > > > > I even tried to set the rate to 1kbps in root, but also led to pretty > > > much the same results. > > > > > > With no qdisc, the rate will go close to 1000 KB/s > > > > > > B machine: > > > 2.6.11-1.1369_FC4 > > > iproute-2.6.11-1 > > > TC HTB version 3.3 > > > > > > I have no clue on what I am doing wrong. Could anyone browse the above > > > script and give me hint? > > > > > > Thanks in advance, > > > > > > AL > > > _______________________________________________ > > > LARTC mailing list > > > LARTC@mailman.ds9a.nl > > > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > > > > > > > > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc From mailinglists at lucassen.org Mon Jul 25 22:33:47 2005 From: mailinglists at lucassen.org (richard lucassen) Date: Mon Jul 25 22:33:52 2005 Subject: [LARTC] teql load balancing on tap devices Message-ID: <20050725223347.1ee0c7c0.mailinglists@lucassen.org> Hello list, I'd like to use teql with tap devices (two OpenVPN tunnels). This works, but the doc /usr/src/linux-/net/sched/sch_teql.c says: "1. Slave devices MUST be active devices, i.e., they must raise the tbusy signal and generate EOI events. If you want to equalize virtual devices like tunnels, use a normal eql device." I can't find if tap devices raise the tbusy signal or generate EOI events, so I assume they don't. Should I go for eql devices like the doc says? Richard. -- ___________________________________________________________________ It's so simple to be wise. Just think of something stupid to say and say the opposite. +------------------------------------------------------------------+ | Richard Lucassen, Utrecht | | Public key and email address: | | http://www.lucassen.org/mail-pubkey.html | +------------------------------------------------------------------+ From christian.konecny at ericsson.com Tue Jul 26 09:15:26 2005 From: christian.konecny at ericsson.com (Christian Konecny (VI/SEA)) Date: Tue Jul 26 09:15:34 2005 Subject: [LARTC] Latency of Linux Bridge Message-ID: -----Original Message----- From: Carl-Daniel Hailfinger [mailto:c-d.hailfinger.devel.2005@gmx.net] Sent: Freitag, 22. Juli 2005 17:28 To: Christian Konecny (VI/SEA) Cc: lartc@mailman.ds9a.nl Subject: Re: [LARTC] Latency of Linux Bridge Carl-Daniel wrote: ============= That's strange. On my bridge with 4 network interfaces, the additional latency is always below 0.5 ms, even if I'm pushing 400 MBit/s through the machine and a kernel compile is running at 100% CPU. Network interfaces are PCIe GBit from Syskonnect, the machine is an Athlon64 at 2 GHz. Even if the clock speed is halved by powersave the additional latency will not go above 0.9 ms. Kernel is vanilla 2.6.11.x. ============== Did you use certain specific compile options? I have changed now from (Knoppix) Debian to (Slax) Slackware running now 2.6.12.2 and have exactly the same on each machine. top shows me a CPU usage less than 0.5%, load average 0.1 while the bridge is handling roughly 64kBit/sec. I can still measure a variable delta between packets of 5ms-10ms. How did you measure your latency values? I am using 3 PCs, one is the Linux-Box acting as bridge (brctl addbr br0; brctl addif br0 eth0; brctl addif br0 eth1), the other 2 PCs are connected to either side of the bridge, running ethereal. In my case I have 2 Phone systems generating continous traffic. Each system is sending out packets every 30ms. If I compare the 2 traces I can see that after the bridge the timings are different then before the bridge. The difference is always in steps of 5ms. so, on sending side is always 30ms difference between each packet on receiving side - after the bridge - the delta is then 25,30,35, or 40ms. Is this really different in your setup? thanks, Christian PS: Servus aus Wien! From kenneth.kalmer at gmail.com Tue Jul 26 12:54:17 2005 From: kenneth.kalmer at gmail.com (Kenneth Kalmer) Date: Tue Jul 26 12:54:25 2005 Subject: [LARTC] wrr vs. htb Message-ID: Guys I'm looking for some advice on the following situation: We're serving a residential complex with internet, the network has grown and with it has the problems, below is a descriptive layout of our current setup: Connection: ADSL Down/Up speed: 512kbps/256kbps (1024kbps/256kbps in the near future) 2.0GHz Celeron D with 512MB RAM Current users: 140 Maximum amount of users: 509 We're currently serving them with an HTB-based solution, but now it's surfacing that some user on the network don't get shaped correctly. They're current speed is calculated is follows: Rate = 512 / number of users Ceil = 128kbps Some users never reach they're ceiling, and only remain on the rate. I've checked, double checked and triple checked my scripts and everything is OK. I've used the tc_graph.pl script to confirm my scenario and everything is OK. Currently there is one parent class, with 140 child classes (one for each user). We're going to start providing some internal features to the users like a community forum, web cam at the gates, audio chat, useful downloads and anti-virus updates and e-mail on the server. This means that there will be two subclasses of the parent class, one for internet traffic and one for local traffic. This is by no means a problem for me. What bothers me is that this will be our new scenario: LAN / \ LOC NET | | 140 C's 140 C's I don't know how healthy this is, and I don't have a clue on how to improve the performance or lessen the load on the box. I've also been contemplating moving the setup to a WRR-based solution, but I'm not too sure if WRR can equally share local and gateway traffic as different 'flows'. If possible, just share your thoughts on the best way to handle this scenario with the 'dual' shaping, different speeds for traffic originating from the network server and internet traffic flowing through the server. The main emphasis is on equality, everyone on the network needs to be happy. Kind regards -- Kenneth Kalmer kenneth.kalmer@gmail.com Folding@home stats http://vspx27.stanford.edu/cgi-bin/main.py?qtype=userpage&username=kenneth%2Ekalmer From c-d.hailfinger.devel.2005 at gmx.net Tue Jul 26 14:45:02 2005 From: c-d.hailfinger.devel.2005 at gmx.net (Carl-Daniel Hailfinger) Date: Tue Jul 26 14:45:00 2005 Subject: [LARTC] Latency of Linux Bridge In-Reply-To: References: Message-ID: <42E6304E.5050208@gmx.net> Christian Konecny (VI/SEA) schrieb: > > Carl-Daniel wrote: >> That's strange. On my bridge with 4 network interfaces, >> the additional latency is always below 0.5 ms, even if >> I'm pushing 400 MBit/s through the machine and a kernel >> compile is running at 100% CPU. Network interfaces are >> PCIe GBit from Syskonnect, the machine is an Athlon64 >> at 2 GHz. Even if the clock speed is halved by powersave >> the additional latency will not go above 0.9 ms. >> Kernel is vanilla 2.6.11.x. > > Did you use certain specific compile options? Depends. I chose to use a slightly modified .config from SUSE 9.3 (attached). > I have changed now from (Knoppix) Debian to (Slax) > Slackware running now 2.6.12.2 and have exactly the > same on each machine. > top shows me a CPU usage less than 0.5%, load average > 0.1 while the bridge is handling roughly 64kBit/sec. > I can still measure a variable delta between packets > of 5ms-10ms. > How did you measure your latency values? (with bridge in between) linux:~ # ping -f -c 1000 192.168.0.1 PING 192.168.0.1 (192.168.0.1) 56(84) bytes of data. --- 192.168.0.1 ping statistics --- 1000 packets transmitted, 1000 received, 0% packet loss, time 963ms rtt min/avg/max/mdev = 0.642/0.914/2.487/0.277 ms, ipg/ewma 0.964/1.023 ms (same config without bridge in between) linux:~ # ping -f -c 1000 192.168.0.1 PING 192.168.0.1 (192.168.0.1) 56(84) bytes of data. --- 192.168.0.1 ping statistics --- 1000 packets transmitted, 1000 received, 0% packet loss, time 883ms rtt min/avg/max/mdev = 0.767/0.834/4.925/0.173 ms, ipg/ewma 0.884/0.814 ms > I am using 3 PCs, one is the Linux-Box acting as bridge > (brctl addbr br0; brctl addif br0 eth0; brctl addif br0 > eth1), the other 2 PCs are connected to either side of > the bridge, running ethereal. I'm using 2 PCs (one as bridge, one as local client) and a Cisco Pix (slow, insecure, unstable) as remote router (192.168.0.1). The measaurements were done from the local client. > In my case I have 2 Phone systems generating continous traffic. Each > system is sending out packets every 30ms. > If I compare the 2 traces I can see that after the bridge the timings > are different then before the bridge. It is very possible that a bridge changes the timing distribtion (but the drastic effect you're seeing shouldn't happen). > The difference is always in steps of 5ms. > so, on sending side is always 30ms difference between each packet > on receiving side - after the bridge - the delta is then 25,30,35, or 40ms. > > Is this really different in your setup? Since the remote side is not a linux box, my chances to measure the timing distribution are somewhat limited. Please try the flood ping I did above (ping -f -c 1000 re.mo.te.ip) and report your results. If you can see latencies above 2 ms something is definitely going wrong. Could be the nic, the nic settings (NAPI), the timing source or packet type (ICMP vs. IP). 1. Check ping 2a. If ping looks wrong -> try my .config and check again 2b. If ping looks OK -> try the phone while running (ping -f re.mo.te.ip) 3. Report back. Gr??e aus T?bingen Carl-Daniel -- http://www.hailfinger.org/ -------------- next part -------------- A non-text attachment was scrubbed... Name: config.gz Type: application/x-gzip Size: 10026 bytes Desc: not available Url : http://mailman.ds9a.nl/pipermail/lartc/attachments/20050726/946611ca/config.bin From fagervan at gmail.com Tue Jul 26 15:02:14 2005 From: fagervan at gmail.com (Fabian Gervan) Date: Tue Jul 26 15:02:18 2005 Subject: [LARTC] how to classify sip traffic (voip) Message-ID: <3ab2553b050726060238be7524@mail.gmail.com> How i can classify sip traffic (voip)?? I try dst 5060 udp port, but dont'work. sip sesion use dynamic port. Sniffing packets with windows net-peeker, I see that packets lenghts is always=87 How i can filter, by packet lenght, with u32? Regards Fabian From christian.konecny at ericsson.com Tue Jul 26 15:09:42 2005 From: christian.konecny at ericsson.com (Christian Konecny (VI/SEA)) Date: Tue Jul 26 15:09:47 2005 Subject: [LARTC] Latency of Linux Bridge Message-ID: you wrote: ================ What hardware, and setup scripts? If you are using netem then it can add latency. The amount depends on the requested delay and the HZ value of the kernel and the choice of PSCHED_CLOCK_SOURCE in the kernel configuration. ================ the latency is already there without having netem running. Only kernel + brctl to configure the bridge shows the described latency of 5ms. Hardware does not matter. I tried on a 500MHz Pentium and also on Pentium III with 3GHz and 1GB RAM. top always shows me a CPU usage less than 0.5%, load average 0.1 while the bridge is handling roughly 64kBit/sec. I can still measure a variable delta between packets of 5ms-10ms. I am using 3 PCs, one is the Linux-Box acting as bridge setup scripts, nothing in special: brctl addbr br0 brctl addif br0 eth0 brctl addif br0 eth1 ifconfig eth0 up ifconfig eth1 up ifconfig br0 up the other 2 PCs are connected to either side of the bridge, running ethereal. In my case I have 2 Phone systems generating continous traffic. Each system is sending out packets every 30ms. If I compare the 2 traces I can see that after the bridge the timings are different then before the bridge. The difference is always in steps of 5ms. so, on sending side is always 30ms difference between each packet on receiving side - after the bridge - the delta is then 25,30,35, or 40ms. i am not sure, what you mean with "The amount depends on the requested delay". I would require an accuracy of about 0.5ms - 1ms on top of that I will then start with "tc qdisc netem" to start my emulation lateron. But as long as the accuracy of the bridge as such is that imprecise, this is not the best basis for my target. you wrote: =============== The amount depends on the requested delay and the HZ value of the kernel and the choice of PSCHED_CLOCK_SOURCE in the kernel configuration. =============== Sorry, but I am not so good in kernel compiling. Where can I find these values. I was grep-ing the whole kernel-sources and could not find it. also make menuconfig did not provide me these settings (at least I could not find them). Could you please give me a hint where to start searching? thanks, Christian -----Original Message----- From: Stephen Hemminger [mailto:shemminger@osdl.org] Sent: Samstag, 23. Juli 2005 01:49 To: Christian Konecny (VI/SEA) Cc: lartc@mailman.ds9a.nl Subject: Re: [LARTC] Latency of Linux Bridge On Fri, 22 Jul 2005 09:55:25 +0200 "Christian Konecny (VI/SEA)" wrote: > Hi there! > > I am working a lot with VoIP in my company, so I thought to use linux > bridge functionality together with tc to emulate delay, jitter, > packet loss, duplication, reordering etc. for testing purposes in our > lab against our VoIP products. I just recognized, that a basic bridge > just with it's minumum configuration of 2 network interfaces creates > latency of approx. 5ms on very low traffic. This seems to be > independent on CPU speed. I tried on 2 GHz PC while having just > 64kBit traffic with packet size of about 300bytes. I am using Knoppix > 3.82 which is actually a debian Live-CD Linux, Kernel 2.6.11. For > some reason they put iproute2 041019 on this distro, which is > intended to be used for kernel 2.6.9. I am aware of remastering the > CD, but have to check if it is possible to recompile the kernel for > the remaster. > > back to my question: where does this latency come from? > "top" shows almost no load while the bridge is handling traffic, so > how come? is there some timer-granularity which can be set in the > kernel, is the latency normal, or what could cause it else? > > Thank you very much in advance! > > /Christian > What hardware, and setup scripts? If you are using netem then it can add latency. The amount depends on the requested delay and the HZ value of the kernel and the choice of PSCHED_CLOCK_SOURCE in the kernel configuration. From sylvain at 2001-space-odyssey.net Tue Jul 26 15:29:19 2005 From: sylvain at 2001-space-odyssey.net (Sylvain Bertrand) Date: Tue Jul 26 15:29:28 2005 Subject: [LARTC] how to classify sip traffic (voip) In-Reply-To: <3ab2553b050726060238be7524@mail.gmail.com> References: <3ab2553b050726060238be7524@mail.gmail.com> Message-ID: <44485.213.245.32.134.1122384559.squirrel@webmail.2001-space-odyssey.net> On Mar 26 juillet 2005 15:02, Fabian Gervan a ?crit : > How i can classify sip traffic (voip)?? > > I try dst 5060 udp port, but dont'work. sip sesion use dynamic port. > > Sniffing packets with windows net-peeker, I see that packets lenghts > is always=87 > How i can filter, by packet lenght, with u32? > > Regards > Fabian > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > You may want to use l7-filters and mark the packets with iptables. Regards, -- Sylvain Bertrand Paris, FRANCE +33 (0)6 64 43 17 69 From x-arnie at ccpbr.org Tue Jul 26 15:39:37 2005 From: x-arnie at ccpbr.org (Alessandro O. Ungaro) Date: Tue Jul 26 15:44:51 2005 Subject: [LARTC] how to classify sip traffic (voip) In-Reply-To: <3ab2553b050726060238be7524@mail.gmail.com> References: <3ab2553b050726060238be7524@mail.gmail.com> Message-ID: <42E63D19.1000802@ccpbr.org> Fabian, on IP/UDP header you have the Length field, you can try to match this. Regards, Alessandro Ungaro x-arnie Fabian Gervan wrote: > How i can classify sip traffic (voip)?? > > I try dst 5060 udp port, but dont'work. sip sesion use dynamic port. > > Sniffing packets with windows net-peeker, I see that packets lenghts > is always=87 > How i can filter, by packet lenght, with u32? > > Regards > Fabian > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > > From ddaasd at gmail.com Tue Jul 26 15:53:35 2005 From: ddaasd at gmail.com (ddaasd) Date: Tue Jul 26 15:53:47 2005 Subject: [LARTC] tcng on 2.6 kernel Message-ID: <42E6405F.10009@gmail.com> Hi, I've tried to compile tcng on my 2.6.12 kernel and the ./configure returned that only 2.4 and 2.5 are supported. So, doesn't tcng compile also on 2.6 kernel? -- ddaas From unki at netshadow.at Tue Jul 26 16:32:44 2005 From: unki at netshadow.at (Andreas Unterkircher) Date: Tue Jul 26 16:32:55 2005 Subject: =?iso-8859-1?Q?=5BLARTC=5D?= tcng on =?iso-8859-1?Q?2=2E6?= kernel In-Reply-To: <42E6405F.10009@gmail.com> Message-ID: <20050726.bya.56582800@egroupware.netshadow.at> Yes, you simply have to add your kernel version in the configure script (KVERSION if I remember correctly). Cheers, Andreas ddaasd (ddaasd@gmail.com) schrieb: > > Hi, > I've tried to compile tcng on my 2.6.12 kernel and the ./configure > returned that only 2.4 and 2.5 are supported. > > So, doesn't tcng compile also on 2.6 kernel? > > -- > ddaas > > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > From christian.konecny at ericsson.com Tue Jul 26 16:37:10 2005 From: christian.konecny at ericsson.com (Christian Konecny (VI/SEA)) Date: Tue Jul 26 16:37:15 2005 Subject: [LARTC] Latency of Linux Bridge Message-ID: Thank you very much for your feedback! Carl-Daniel Hailfinger wrote: ========================= > How did you measure your latency values? (with bridge in between) linux:~ # ping -f -c 1000 192.168.0.1 PING 192.168.0.1 (192.168.0.1) 56(84) bytes of data. --- 192.168.0.1 ping statistics --- 1000 packets transmitted, 1000 received, 0% packet loss, time 963ms rtt min/avg/max/mdev = 0.642/0.914/2.487/0.277 ms, ipg/ewma 0.964/1.023 ms (same config without bridge in between) linux:~ # ping -f -c 1000 192.168.0.1 PING 192.168.0.1 (192.168.0.1) 56(84) bytes of data. --- 192.168.0.1 ping statistics --- 1000 packets transmitted, 1000 received, 0% packet loss, time 883ms rtt min/avg/max/mdev = 0.767/0.834/4.925/0.173 ms, ipg/ewma 0.884/0.814 ms ========================= Same here. Ping delivers me the same values. I also tried following ping -f -c 1000 192.168.0.1 -i 0.03 this produces pretty the same traffic like the one I have from my telephony systems. I did a sniffer-trace on that and could not see any deviations as big as 5ms. Therefore I assume the bridge will handle ICMP easier/faster than normal UDP-traffic. I also did another test with iperf (http://dast.nlanr.net/Projects/Iperf/) with similar values: iperf -c 192.168.0.1 -u -len 90 -t 7200 20k This produces UDP traffic with 90ms packets every 30ms. Now I have again the deviations in steps of 5ms. So, icmp might not be a good tool to measure jitter in this terms. I am pretty sure you will have the same with iperf or any other method to produce UDP-traffic. Carl-Daniel Hailfinger wrote: ========================= It is very possible that a bridge changes the timing distribtion (but the drastic effect you're seeing shouldn't happen). ========================= I fully agree ;-) thanks, Christian PS: What I am doing wrong? Each mail from me is not put correctly into the corresponding position within the thread of the mailing list. Have M$ Outlook here... From alex at qb.ro Tue Jul 26 16:45:55 2005 From: alex at qb.ro (alex@qb.ro) Date: Tue Jul 26 16:46:00 2005 Subject: [LARTC] iptables MARK behaviour out of the box Message-ID: <200507261445.j6QEjtK09478@qb.ro> Hi all, Short question: what happens with the mark on a packet once it's out of the box? Is is usable in another computer in the network or the mark is only valid in the same box you've marked the packet? Thank you, Alex From surda at shurdix.com Tue Jul 26 17:00:41 2005 From: surda at shurdix.com (Peter Surda) Date: Tue Jul 26 17:01:07 2005 Subject: [LARTC] wrr vs. htb In-Reply-To: Message-ID: <2005626170412053@mail.routehat.org> On Tue, 26 Jul 2005 12:54:17 +0200 Kenneth Kalmer wrote: >Guys hi >I don't know how healthy this is, and I don't have a clue on how to >improve the performance or lessen the load on the box. I've also been >contemplating moving the setup to a WRR-based solution, but I'm not >too sure if WRR can equally share local and gateway traffic as >different 'flows'. I don't understand this "2 kinds of traffic" concept. Where is the server? In the internet or local? If local, why the need to shape it at all? I can acknowledge though that under certain circumstances when you have several parallel WRR qdiscs, it leads to freezes. Perhaps if you explain in more detail what the network looks like I can give your more precise answers. >If possible, just share your thoughts on the best way to handle this >scenario with the 'dual' shaping, different speeds for traffic >originating from the network server and internet traffic flowing >through the server. The main emphasis is on equality, everyone on the >network needs to be happy. WRR is perfect when you need equality. [advertisement]Check out shurdix cough cough[/advertisement]. >Kind regards >Kenneth Kalmer Yours sincerely, Peter From ji.li3 at hp.com Tue Jul 26 18:45:56 2005 From: ji.li3 at hp.com (Li, Ji) Date: Tue Jul 26 18:46:46 2005 Subject: [LARTC] multi-path TCP performance Message-ID: <628BFCE8B64706469FE4D4852CEC953706D50482@tayexc14.americas.cpqcorp.net> I am measuring the performance of one TCP connection over two symmetric paths. Packets are sent to two paths alternatively. I found that when the latency of each path are within 1ms, the overall TCP throughput is the *sum* of the throughput of the two paths. However, when the latency of the two paths increases to 5ms, the overal TCP throughput drops to the throughput of a *single* path. Has anyone studied similar problem? What makes the performance go down? I use Fedora Core 3 and 4, teql and netem for my emulation. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ds9a.nl/pipermail/lartc/attachments/20050726/8e34af47/attachment.htm From jarod125 at yahoo.com Tue Jul 26 19:23:55 2005 From: jarod125 at yahoo.com (Gabriel) Date: Tue Jul 26 19:23:59 2005 Subject: [LARTC] iptables MARK behaviour out of the box Message-ID: <20050726172356.43186.qmail@web60919.mail.yahoo.com> No, the manual says that the MARKs set with iptables are usable only within the box that sets them. It's like some metadata attached to the packet. If you want to modify QoS packet settings, you should try playing with the TOS field settings. --- alex@qb.ro wrote: > Hi all, > Short question: what happens with the mark on a packet > once it's out of the box? Is is usable in another > computer in the network or the mark is only valid in > the same box you've marked the packet? > > Thank you, > Alex> _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From gtaylor at riverviewtech.net Wed Jul 27 02:29:47 2005 From: gtaylor at riverviewtech.net (Grant Taylor) Date: Wed Jul 27 02:29:56 2005 Subject: [LARTC] QoS and IPSec... Message-ID: <42E6D57B.6050109@riverviewtech.net> Hi, I have what to me is an interesting issue. I am wanting to prioritize (QoS) traffic that will be passing through an IPSec (OpenS/WAN) VPN between two (identical) Linux routers. I know that I can apply the IPSec patches (1-4) to the kernel and IPTables (if they are not already applied by now) filter traffic before and after IPSec encapsulation. My problem is that I don't know if I will be able to QoS the traffic that will be encapsulated as far as I know QoS prioritization (via CBQ or HTB) only applies to traffic that is being dequeue from the skbuffers to go out the physical interface. In my mind the traffic that is to be encapsulated does not ""go out a physical interface to be dequeued in the order that I want to prioritize. I know that I can QoS IPSec VPN traffic (IP/ESP) to a higher priority than any other IP traffic but I'm not sure about the traffic that is being encapsulated. My (very) rough idea is to use something like dummy net or IMQ to provide an interface (or subnet if need be) that the traffic will traverse and be dequeued from where I can apply the QoS that I want to. I'm not quite sure how to go about this so any advice would be greatly appreciated. I would like to QoS / Prioritize LAN traffic that is destined to the other LAN based on the type of traffic that it is (ICMP, RDP, RFB, SMB, etc) before it is encapsulated. Once the traffic has been encapsulated I'd like to QoS / Prioritize the ESP traffic that is destined to the other LAN's globally routable IP before any other internet traffic goes out. This later part is not the problem, just the former part. My network layout(s) are below for those of you that will be asking: Lan A: - 172.30.12.x/24 subnet - 172.30.12.1-250 client systems and the likes - 172.30.12.254 is the default gateway which will be replaced by one of the boxen I'm asking about. - A.B.C.Z/24 globally routable IP on the router Lan B: - 172.30.13.x/24 subnet - 172.30.13.1-250 client systems and the likes - 172.30.13.254 is the default gateway which will be replaced by one of the boxen I'm asking about. - A.B.C.Y/24 globally routable IP on the router VPN: - The VPN in question will be between the A.B.C.Z and A.B.C.Y globally routable IP addresses. Note that both LANs have a DSL circuit from the same provider and thus are 1 IP off from each other on their globally routable IP. Grant. . . . P.S. I'm (cross) posting this to the NetFilter mail lists as I've seen some very complex questions and answers on the LARTC and NetFilter mail lists and I would like to pull from both pools of talent. So be mindful when replying to all. ;) From unki at netshadow.at Wed Jul 27 06:17:57 2005 From: unki at netshadow.at (Andreas Unterkircher) Date: Wed Jul 27 06:18:01 2005 Subject: [LARTC] QoS and IPSec... In-Reply-To: <42E6D57B.6050109@riverviewtech.net> References: <42E6D57B.6050109@riverviewtech.net> Message-ID: <42E70AF5.8070600@netshadow.at> I don' tsee any probleme with Qos and IPSec. I do QoS with FreeSwan/OpenSwan on the KLIPS-Device (ipsecX) as well with 2.6-IPsec-Stack (racoon) where you see uncapsulated and encapsulated on the same physical interface. With both you can use the common tc features to start egress shaping. Cheers, Andreas Grant Taylor wrote: > Hi, I have what to me is an interesting issue. I am wanting to > prioritize (QoS) traffic that will be passing through an IPSec > (OpenS/WAN) VPN between two (identical) Linux routers. I know that I > can apply the IPSec patches (1-4) to the kernel and IPTables (if they > are not already applied by now) filter traffic before and after IPSec > encapsulation. My problem is that I don't know if I will be able to > QoS the traffic that will be encapsulated as far as I know QoS > prioritization (via CBQ or HTB) only applies to traffic that is being > dequeue from the skbuffers to go out the physical interface. In my > mind the traffic that is to be encapsulated does not ""go out a > physical interface to be dequeued in the order that I want to > prioritize. I know that I can QoS IPSec VPN traffic (IP/ESP) to a > higher priority than any other IP traffic but I'm not sure about the > traffic that is being encapsulated. My (very) rough idea is to use > something like dummy net or IMQ to provide an interface (or subnet if > need be) that the traffic will traverse and be dequeued from where I > can apply the QoS that I want to. I'm not quite sure how to go about > this so any advice would be greatly appreciated. > > I would like to QoS / Prioritize LAN traffic that is destined to the > other LAN based on the type of traffic that it is (ICMP, RDP, RFB, > SMB, etc) before it is encapsulated. Once the traffic has been > encapsulated I'd like to QoS / Prioritize the ESP traffic that is > destined to the other LAN's globally routable IP before any other > internet traffic goes out. This later part is not the problem, just > the former part. > > My network layout(s) are below for those of you that will be asking: > > Lan A: > - 172.30.12.x/24 subnet > - 172.30.12.1-250 client systems and the likes > - 172.30.12.254 is the default gateway which will be replaced by one > of the boxen I'm asking about. > - A.B.C.Z/24 globally routable IP on the router > > Lan B: > - 172.30.13.x/24 subnet > - 172.30.13.1-250 client systems and the likes > - 172.30.13.254 is the default gateway which will be replaced by one > of the boxen I'm asking about. > - A.B.C.Y/24 globally routable IP on the router > > VPN: > - The VPN in question will be between the A.B.C.Z and A.B.C.Y globally > routable IP addresses. > > Note that both LANs have a DSL circuit from the same provider and thus > are 1 IP off from each other on their globally routable IP. > > > Grant. . . . > > P.S. I'm (cross) posting this to the NetFilter mail lists as I've > seen some very complex questions and answers on the LARTC and > NetFilter mail lists and I would like to pull from both pools of > talent. So be mindful when replying to all. ;) > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc From vinod_chandran at multitech.co.in Wed Jul 27 06:53:33 2005 From: vinod_chandran at multitech.co.in (Vinod Chandran) Date: Wed Jul 27 06:53:08 2005 Subject: [LARTC] Re: QoS and IPSec... In-Reply-To: <42E6D57B.6050109@riverviewtech.net> References: <42E6D57B.6050109@riverviewtech.net> Message-ID: <42E7134D.3090809@multitech.co.in> Hi Grant, Add IPTABLE rules in the FORWARD mangle to handle the normal packets ( ICMP,etc) with specific mark values and add filters for the same . As far as IPSEC traffic is concerned, its generally generated from the box, unless its acting as an IPSEC pass thru. Hence u can add rules in the POSTROUTING chain to mark all AH/ESP packets with some mark value. I believe since IPSEC packet is generated from the box, the source ip will be that of the incoming interface..... Not sure about this!!!! Hope this helps. Regards, Vinod C Grant Taylor wrote: > Hi, I have what to me is an interesting issue. I am wanting to > prioritize (QoS) traffic that will be passing through an IPSec > (OpenS/WAN) VPN between two (identical) Linux routers. I know that I > can apply the IPSec patches (1-4) to the kernel and IPTables (if they > are not already applied by now) filter traffic before and after IPSec > encapsulation. My problem is that I don't know if I will be able to > QoS the traffic that will be encapsulated as far as I know QoS > prioritization (via CBQ or HTB) only applies to traffic that is being > dequeue from the skbuffers to go out the physical interface. In my > mind the traffic that is to be encapsulated does not ""go out a > physical interface to be dequeued in the order that I want to > prioritize. I know that I can QoS IPSec VPN traffic (IP/ESP) to a > higher priority than any other IP traffic but I'm not sure about the > traffic that is being encapsulated. My (very) rough idea is to use > something like dummy net or IMQ to provide an interface (or subnet if > need be) that the traffic will traverse and be dequeued from where I > can apply the QoS that I want to. I'm not quite sure how to go about > this so any advice would be greatly appreciated. > > I would like to QoS / Prioritize LAN traffic that is destined to the > other LAN based on the type of traffic that it is (ICMP, RDP, RFB, > SMB, etc) before it is encapsulated. Once the traffic has been > encapsulated I'd like to QoS / Prioritize the ESP traffic that is > destined to the other LAN's globally routable IP before any other > internet traffic goes out. This later part is not the problem, just > the former part. > > My network layout(s) are below for those of you that will be asking: > > Lan A: > - 172.30.12.x/24 subnet > - 172.30.12.1-250 client systems and the likes > - 172.30.12.254 is the default gateway which will be replaced by one > of the boxen I'm asking about. > - A.B.C.Z/24 globally routable IP on the router > > Lan B: > - 172.30.13.x/24 subnet > - 172.30.13.1-250 client systems and the likes > - 172.30.13.254 is the default gateway which will be replaced by one > of the boxen I'm asking about. > - A.B.C.Y/24 globally routable IP on the router > > VPN: > - The VPN in question will be between the A.B.C.Z and A.B.C.Y globally > routable IP addresses. > > Note that both LANs have a DSL circuit from the same provider and thus > are 1 IP off from each other on their globally routable IP. > > > Grant. . . . > > P.S. I'm (cross) posting this to the NetFilter mail lists as I've > seen some very complex questions and answers on the LARTC and > NetFilter mail lists and I would like to pull from both pools of > talent. So be mindful when replying to all. ;) > From vinod_chandran at multitech.co.in Wed Jul 27 07:02:38 2005 From: vinod_chandran at multitech.co.in (Vinod Chandran) Date: Wed Jul 27 07:01:53 2005 Subject: [LARTC] how to classify sip traffic (voip) In-Reply-To: <3ab2553b050726060238be7524@mail.gmail.com> References: <3ab2553b050726060238be7524@mail.gmail.com> Message-ID: <42E7156E.9010605@multitech.co.in> > Fabian, > > Try adding the following Iptable rule with a filter for the mark value.. > iptables -t mangle -A PREROUTING -p udp -m lenght -j > MARK --set-mark 1 > > Regards, > Vinod C Fabian Gervan wrote: >How i can classify sip traffic (voip)?? > >I try dst 5060 udp port, but dont'work. sip sesion use dynamic port. > >Sniffing packets with windows net-peeker, I see that packets lenghts >is always=87 >How i can filter, by packet lenght, with u32? > >Regards >Fabian >_______________________________________________ >LARTC mailing list >LARTC@mailman.ds9a.nl >http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > > > From tdi at pozman.pl Wed Jul 27 08:36:50 2005 From: tdi at pozman.pl (Dariusz Dwornikowski) Date: Wed Jul 27 08:36:54 2005 Subject: [LARTC] IMQ Message-ID: <32821.80.50.56.166.1122446210.squirrel@80.50.56.166> ive got such network: |--------| |-------------| | WORLD |---|ROUTER/server| ------ NATED LAN |--------| |-------------| I want to use imq on ROUTER, what behaviour to choose ? AA, BA, AB, BB ?? -- From unki at netshadow.at Wed Jul 27 08:53:35 2005 From: unki at netshadow.at (Andreas Unterkircher) Date: Wed Jul 27 08:53:39 2005 Subject: =?iso-8859-1?Q?=5BLARTC=5D?= IMQ In-Reply-To: <32821.80.50.56.166.1122446210.squirrel@80.50.56.166> Message-ID: <20050727.xY3.87277700@egroupware.netshadow.at> Dariusz Dwornikowski (tdi@pozman.pl) schrieb: > I want to use imq on ROUTER, what behaviour to choose ? AA, BA, AB, BB ?? That depends on what you want to do. For example: *) BA - If you want to have all packets on the IMQ after the nat table - so you wouldn't see any internal ips anymore on the IMQ device. *) BB - If you want to have packets with internal ips on the IMQ before the pass the nat table. But you can't match on your external IP with BB. Cheers, Andreas From tdi at pozman.pl Wed Jul 27 09:47:49 2005 From: tdi at pozman.pl (Dariusz Dwornikowski) Date: Wed Jul 27 09:47:53 2005 Subject: [LARTC] IMQ In-Reply-To: <20050727.xY3.87277700@egroupware.netshadow.at> References: <32821.80.50.56.166.1122446210.squirrel@80.50.56.166> <20050727.xY3.87277700@egroupware.netshadow.at> Message-ID: <32918.80.50.56.166.1122450469.squirrel@80.50.56.166> > Dariusz Dwornikowski (tdi@pozman.pl) schrieb: >> I want to use imq on ROUTER, what behaviour to choose ? AA, BA, AB, BB >> ?? > > That depends on what you want to do. For example: > > *) BA - If you want to have all packets on the IMQ after the nat table - > so > you wouldn't see any internal ips anymore on the IMQ device. > *) BB - If you want to have packets with internal ips on the IMQ before > the > pass the nat table. But you can't match on your external IP with BB. yes but i want to have two IMQ devices... for outside eth and inside eth -- From unki at netshadow.at Wed Jul 27 10:01:52 2005 From: unki at netshadow.at (Andreas Unterkircher) Date: Wed Jul 27 10:01:55 2005 Subject: =?iso-8859-1?Q?=5BLARTC=5D?= IMQ In-Reply-To: <32918.80.50.56.166.1122450469.squirrel@80.50.56.166> Message-ID: <20050727.sJb.28330900@egroupware.netshadow.at> > yes but i want to have two IMQ devices... for outside eth and inside eth Ahmm.. don't know what you mean with that. But if you have a external Interface (ex eth0) on which you want to shape egress and also ingress shaping you simply do: ip link set imq0 up ip link set imq1 up ${IPTABLES} -t mangle -I PREROUTING -i ${EXT_DEV} -j IMQ --todev 0 ${IPTABLES} -t mangle -I POSTROUTING -o ${EXT_DEV} -j IMQ --todev 1 and put your QoS on imq0 and imq1... From tdi at pozman.pl Wed Jul 27 10:49:43 2005 From: tdi at pozman.pl (Dariusz Dwornikowski) Date: Wed Jul 27 10:49:50 2005 Subject: [LARTC] IMQ In-Reply-To: <20050727.sJb.28330900@egroupware.netshadow.at> References: <32918.80.50.56.166.1122450469.squirrel@80.50.56.166> <20050727.sJb.28330900@egroupware.netshadow.at> Message-ID: <32983.80.50.56.166.1122454183.squirrel@80.50.56.166> >> yes but i want to have two IMQ devices... for outside eth and inside eth > > Ahmm.. don't know what you mean with that. But if you have a external > Interface (ex eth0) on which you want to shape egress and also ingress > shaping > you simply do: > > ip link set imq0 up > ip link set imq1 up > > ${IPTABLES} -t mangle -I PREROUTING -i ${EXT_DEV} -j IMQ --todev 0 > ${IPTABLES} -t mangle -I POSTROUTING -o ${EXT_DEV} -j IMQ --todev 1 > > and put your QoS on imq0 and imq1... i want to shape traffic for my clients and do not know what behaviour to choose... in your example will it be possible to match NATed addresses of my clients ? -- From unki at netshadow.at Wed Jul 27 10:59:13 2005 From: unki at netshadow.at (Andreas Unterkircher) Date: Wed Jul 27 10:59:18 2005 Subject: =?iso-8859-1?Q?=5BLARTC=5D?= IMQ In-Reply-To: <32983.80.50.56.166.1122454183.squirrel@80.50.56.166> Message-ID: <20050727.ZTY.01260600@egroupware.netshadow.at> > i want to shape traffic for my clients and do not know what behaviour to > choose... > in your example will it be possible to match NATed addresses of my clients ? you want to match the translated addresses of your clients (-> external IP) - then BA is the correct behaviour. From tdi at pozman.pl Wed Jul 27 11:37:03 2005 From: tdi at pozman.pl (Dariusz Dwornikowski) Date: Wed Jul 27 11:37:06 2005 Subject: [LARTC] IMQ In-Reply-To: <20050727.ZTY.01260600@egroupware.netshadow.at> References: <32983.80.50.56.166.1122454183.squirrel@80.50.56.166> <20050727.ZTY.01260600@egroupware.netshadow.at> Message-ID: <33239.80.50.56.166.1122457023.squirrel@80.50.56.166> >> i want to shape traffic for my clients and do not know what behaviour to >> choose... >> in your example will it be possible to match NATed addresses of my >> clients ? > > you want to match the translated addresses of your clients (-> external > IP) - > then BA is the correct behaviour. > no i want match their private addresses. i want imq0 for ext_if and imq1 for internal_if.. so i can traffic shaping on matched NATed ips -- From unki at netshadow.at Wed Jul 27 11:44:39 2005 From: unki at netshadow.at (Andreas Unterkircher) Date: Wed Jul 27 11:44:43 2005 Subject: =?iso-8859-1?Q?=5BLARTC=5D?= IMQ In-Reply-To: <33239.80.50.56.166.1122457023.squirrel@80.50.56.166> Message-ID: <20050727.mF4.91519400@egroupware.netshadow.at> > no i want match their private addresses. > i want imq0 for ext_if and imq1 for internal_if.. > > so i can traffic shaping on matched NATed ips Oh ok. I was irritated because you say NATed ips - for me this are all clients after the passed the nat table... So you ned BB. IMQ hook before the postrouting table (where your NAT will happen) and you have internal addresses on the imq device. From kenneth.kalmer at gmail.com Wed Jul 27 12:01:06 2005 From: kenneth.kalmer at gmail.com (Kenneth Kalmer) Date: Wed Jul 27 12:01:11 2005 Subject: [LARTC] wrr vs. htb In-Reply-To: <2005626170412053@mail.routehat.org> References: <2005626170412053@mail.routehat.org> Message-ID: On 7/26/05, Peter Surda wrote: > On Tue, 26 Jul 2005 12:54:17 +0200 Kenneth Kalmer > wrote: > > >Guys > hi > > >I don't know how healthy this is, and I don't have a clue on how to > >improve the performance or lessen the load on the box. I've also been > >contemplating moving the setup to a WRR-based solution, but I'm not > >too sure if WRR can equally share local and gateway traffic as > >different 'flows'. > I don't understand this "2 kinds of traffic" concept. Where is the server? In > the internet or local? If local, why the need to shape it at all? I can > acknowledge though that under certain circumstances when you have several > parallel WRR qdiscs, it leads to freezes. Perhaps if you explain in more detail > what the network looks like I can give your more precise answers. It's one server, fulfilling two functions, gateway en network service delivery The two flows are as follows: 1. Is internet traffic through this server, acting as the router/firewall. 2. Is the network traffic originating from the (same) server to the network. The network traffic (2) is mail, audio chat, webcam and a little website with useful stuff. This is only available to users inside the network, so no effects on the internet link. > > >If possible, just share your thoughts on the best way to handle this > >scenario with the 'dual' shaping, different speeds for traffic > >originating from the network server and internet traffic flowing > >through the server. The main emphasis is on equality, everyone on the > >network needs to be happy. > WRR is perfect when you need equality. [advertisement]Check out shurdix cough > cough[/advertisement]. I had this idea in the shower this morning, comments please: :0 HTB / \ (HTB) 1:0 2:0 (HTB) | | WRR 1:1 1:2 WRR OK, the parent flow is the network link, so it's set to 100mbps rate & ceil. The first HTB class (1:0) is shaped to the speed of the ADSL link, and attached to it is a WRR qdisc for equality on that link. The second HTB class (2:0) is shaped to (network - ADSL) and also has a WRR qdisc attached for equality on the network. I don't know if this is possible, or even feasible, but on paper it makes sense. > > >Kind regards > >Kenneth Kalmer > Yours sincerely, > Peter > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > -- Kenneth Kalmer kenneth.kalmer@gmail.com Folding@home stats http://vspx27.stanford.edu/cgi-bin/main.py?qtype=userpage&username=kenneth%2Ekalmer From jlynch at frink.nuigalway.ie Wed Jul 27 12:28:30 2005 From: jlynch at frink.nuigalway.ie (Jonathan Lynch) Date: Wed Jul 27 12:28:55 2005 Subject: [LARTC] HTB and PRIO qdiscs introducing extra latency when output interface is saturated Message-ID: <1122460110.8454.11.camel@pgala.it.nuigalway.ie> Im using a Linux machine with standard pc hardware with 3 seperate PCI network interfaces to operate as a DiffServ core router using Linux traffic control. The machine is a P4 2.8ghz, 512mb RAM running fedora core 3 with the 2.6.12.3 kernel. All links and network interfaces are full duplex fast ethernet. IP forwarding is enabled in the kernel. All hosts on the network have their time sychronised using a stratum 1 server on the same VLAN. Below is a ascii diagram of the network. (network A) edge router ------>core router---->edge router (network C) ^ | | edge router (network B) Core Router Configuration: --------------------------- The core router implements the expedited forwarding PHB. I have tried 2 Different Configurations. 1. HTB qdisc with two htb classes. One which services VoIP traffic (marked with EF codepoint) VoIP traffic is guaranteed to serviced at a minimum rate of 1500 kbit. This htb class is serviced by a fifo queue with a limit of 5 packets. The 2nd htb class guarantees all other traffic to serviced at a minimum rate of 5mbit. The RED qdisc services this htb class. 2. PRIO qdisc with token a bucket filter to service VoIP traffic (marked with EF codepoint) VoIP traffic with a guaranteed minimum rate of 1500 kbit. A RED qdisc to service all other traffic. Test 1. --------------------------- VoIP traffic originates from network A and is destined to network C. The throughput of VoIP traffic is 350 kbit. No other traffic passes through the core router during this time. These Voip packets are marked with the EF codepoint. Using either of the above mentioned configurations for the core router, the delay of the VoIP traffic in travelling from network A to network C passing through the core router is 0.25 milliseconds. Test 2. --------------------------- Again VoIP traffic originates from network A and is destined to netwotk C with a throughput of 350 kbit. TCP traffic also originates from another host in network A and is destined for another host in network C. More TCP traffic originates from network B and is destined to network C. This TCP traffic is from transfering large files through http. As a result a bottleneck is created at the outgoing interface of the core router to network C. The combined TCP traffic from these sources is nearly 100 mbit. Using either of the above mentioned configurations for the core router, the delay of the VoIP traffic in travelling from network A to network C passing through the core router is 30ms milliseconds with 0% loss. There is a considerable amount of TCP packets dropped. Could anyone tell me why the delay is so high (30ms) for VoIP packets which are treated with the EF phb when the outgoing interface of core router to network c is saturated ? Is it due to operating system factors ? Has anyone else had similar experiences ? Also I would appreciate if anyone could give me performace metrics as to approximately how many packets per second a router running Linux with standard pc hardware can forward. Or even mention any factors that would affect this performance. Im assume the system interrupt frequncy HZ will affect performance in some way. Jonathan Lynch Note: I already posted the same question to the list a few weeks back but got no reply. I have reworded my question so it is clearer. ----------------------------------------------------------------------------------------------- The config I used for each setup is included below. These are slight modifications that are supplied with iproute2 source code. Config 1 using htb ------------------- tc qdisc add dev $1 handle 1:0 root dsmark indices 64 set_tc_index tc filter add dev $1 parent 1:0 protocol ip prio 1 tcindex mask 0xfc shift 2 Main htb qdisc & class tc qdisc add dev $1 parent 1:0 handle 2:0 htb tc class add dev $1 parent 2:0 classid 2:1 htb rate 100Mbit ceil 100Mbit EF Class (2:10) tc class add dev $1 parent 2:1 classid 2:10 htb rate 1500Kbit ceil 100Mbit tc qdisc add dev $1 parent 2:10 pfifo limit 5 tc filter add dev $1 parent 2:0 protocol ip prio 1 handle 0x2e tcindex classid 2:10 pass_on BE Class (2:20) tc class add dev $1 parent 2:1 classid 2:20 htb rate 5Mbit ceil 100Mbit tc qdisc add dev $1 parent 2:20 red limit 60KB min 15KB max 45KB burst 20 avpkt 1000 bandwidth 100Mbit probability 0.4 tc filter add dev $1 parent 2:0 protocol ip prio 2 handle 0 tcindex mask 0 classid 2:20 pass_on Config 2 using PRIO ------------------- Main dsmark & classifier tc qdisc add dev $1 handle 1:0 root dsmark indices 64 set_tc_index tc filter add dev $1 parent 1:0 protocol ip prio 1 tcindex mask 0xfc shift 2 Main prio queue tc qdisc add dev $1 parent 1:0 handle 2:0 prio tc qdisc add dev $1 parent 2:1 tbf rate 1.5Mbit burst 1.5kB limit 1.6kB tc filter add dev $1 parent 2:0 protocol ip prio 1 handle 0x2e tcindex classid 2:1 pass_on BE class(2:2) tc qdisc add dev $1 parent 2:2 red limit 60KB min 15KB max 45KB burst 20 avpkt 1000 bandwidth 100Mbit probability 0.4 tc filter add dev $1 parent 2:0 protocol ip prio 2 handle 0 tcindex mask 0 classid 2:2 pass_on From tdi at pozman.pl Wed Jul 27 13:07:17 2005 From: tdi at pozman.pl (Dariusz Dwornikowski) Date: Wed Jul 27 13:07:21 2005 Subject: [LARTC] IMQ In-Reply-To: <20050727.mF4.91519400@egroupware.netshadow.at> References: <33239.80.50.56.166.1122457023.squirrel@80.50.56.166> <20050727.mF4.91519400@egroupware.netshadow.at> Message-ID: <33313.80.50.56.166.1122462437.squirrel@80.50.56.166> >> no i want match their private addresses. >> i want imq0 for ext_if and imq1 for internal_if.. >> >> so i can traffic shaping on matched NATed ips > > Oh ok. I was irritated because you say NATed ips - for me this are all > clients > after the passed the nat table... > > So you ned BB. IMQ hook before the postrouting table (where your NAT will > happen) and you have internal addresses on the imq device. > > thank you very much. -- From andy.furniss at dsl.pipex.com Wed Jul 27 15:25:13 2005 From: andy.furniss at dsl.pipex.com (Andy Furniss) Date: Wed Jul 27 15:25:07 2005 Subject: [LARTC] HTB and PRIO qdiscs introducing extra latency when output interface is saturated In-Reply-To: <1122460110.8454.11.camel@pgala.it.nuigalway.ie> References: <1122460110.8454.11.camel@pgala.it.nuigalway.ie> Message-ID: <42E78B39.1030201@dsl.pipex.com> Jonathan Lynch wrote: > Could anyone tell me why the delay is so high (30ms) for VoIP packets > which are treated with the EF phb when the outgoing interface of core > router to network c is saturated ? > I have never used dsmark so am not sure about the classification parts of your rules. You need to check where the packets are going with with tc -s qdisc ls dev ... The other parts have some issues see below. > ----------------------------------------------------------------------------------------------- > The config I used for each setup is included below. These are slight > modifications that are supplied with iproute2 source code. > > Config 1 using htb > ------------------- > tc qdisc add dev $1 handle 1:0 root dsmark indices 64 set_tc_index > tc filter add dev $1 parent 1:0 protocol ip prio 1 tcindex mask 0xfc > shift 2 flowid/classid here maybe, to get packets to 2:0, though it may work - check. > > Main htb qdisc & class > tc qdisc add dev $1 parent 1:0 handle 2:0 htb > tc class add dev $1 parent 2:0 classid 2:1 htb rate 100Mbit ceil 100Mbit 100mbit will be too high if it's 100mbit nic. > > EF Class (2:10) > tc class add dev $1 parent 2:1 classid 2:10 htb rate 1500Kbit ceil > 100Mbit > tc qdisc add dev $1 parent 2:10 pfifo limit 5 > tc filter add dev $1 parent 2:0 protocol ip prio 1 handle 0x2e tcindex > classid 2:10 pass_on Don't know what pass_on will mean here. > > BE Class (2:20) > tc class add dev $1 parent 2:1 classid 2:20 htb rate 5Mbit ceil 100Mbit > tc qdisc add dev $1 parent 2:20 red limit 60KB min 15KB max 45KB burst > 20 avpkt 1000 bandwidth 100Mbit probability 0.4 > tc filter add dev $1 parent 2:0 protocol ip prio 2 handle 0 tcindex mask > 0 classid 2:20 pass_on > > Config 2 using PRIO > ------------------- > Main dsmark & classifier > tc qdisc add dev $1 handle 1:0 root dsmark indices 64 set_tc_index > tc filter add dev $1 parent 1:0 protocol ip prio 1 tcindex mask 0xfc > shift 2 > > Main prio queue > tc qdisc add dev $1 parent 1:0 handle 2:0 prio > tc qdisc add dev $1 parent 2:1 tbf rate 1.5Mbit burst 1.5kB limit 1.6kB Won't hurt if the packets are small voip but TBF has a nasty habit of taking 1 from the burst/mtu you specify so your burst setting may result in packets >1499B getting dropped - tc -s -d qdisc ls dev ... should show what it's using. > tc filter add dev $1 parent 2:0 protocol ip prio 1 handle 0x2e tcindex > classid 2:1 pass_on > > BE class(2:2) > tc qdisc add dev $1 parent 2:2 red limit 60KB min 15KB max 45KB burst 20 > avpkt 1000 bandwidth 100Mbit probability 0.4 > tc filter add dev $1 parent 2:0 protocol ip prio 2 handle 0 tcindex mask > 0 classid 2:2 pass_on Without wrapping it with something like htb red won't shape traffic. Andy. From kenneth.kalmer at gmail.com Wed Jul 27 15:47:44 2005 From: kenneth.kalmer at gmail.com (Kenneth Kalmer) Date: Wed Jul 27 15:47:51 2005 Subject: [LARTC] wrr vs. htb In-Reply-To: <20056271220192053@mail.routehat.org> References: <20056271220192053@mail.routehat.org> Message-ID: On 7/27/05, Peter Surda wrote: > On Wed, 27 Jul 2005 12:01:06 +0200 Kenneth Kalmer > wrote: > > >It's one server, fulfilling two functions, gateway en network service delivery > Ok, now I get it. You can use IMQ and that will solve the problem. Or you can > use a separate computer as a gateway. > > > :0 HTB > > / \ > > (HTB) 1:0 2:0 (HTB) > > | | > > WRR 1:1 1:2 WRR > WRR doesn't like situations like this (don't ask why, some bug somewhere), the > computer tends to freeze unpredictably. Ouch, well now each of the WRR discs will be replaced by HTB discs for each user, currently 140 discs for each WRR disc... Any other ways to do this, or will HTB cope? This can expand up to 2x 509 discs... How can I then handle the equality issues better, should I just pray that HTB will divide all excess fairly? Also, I forgot to mention that each HTB for each client get's an SFQ as well, just so that they don't kill themselves in the process. Is this kosher? Thanks -- Kenneth Kalmer kenneth.kalmer@gmail.com Folding@home stats http://vspx27.stanford.edu/cgi-bin/main.py?qtype=userpage&username=kenneth%2Ekalmer From jlynch at frink.nuigalway.ie Wed Jul 27 17:37:42 2005 From: jlynch at frink.nuigalway.ie (Jonathan Lynch) Date: Wed Jul 27 17:38:25 2005 Subject: [LARTC] HTB and PRIO qdiscs introducing extra latency when output interface is saturated In-Reply-To: <42E78B39.1030201@dsl.pipex.com> References: <1122460110.8454.11.camel@pgala.it.nuigalway.ie> <42E78B39.1030201@dsl.pipex.com> Message-ID: <1122478662.4637.49.camel@pgala.it.nuigalway.ie> Andy, Many thanks for your reply. Below is some output from the queueing disciplines to show that the filters are working correctly and they are going to the right classes. NOTE: The root qdisc of each interface is deleted before I run the tests. This resets the statistics for the qdisc. The following is the output after the tests. Output of tc -s qdisc show on the core route for the 3 network interfaces. qdisc dsmark 1: dev eth0 indices 0x0040 set_tc_index Sent 2183574289 bytes 1496372 pkts (dropped 60982, overlimits 0 requeues 0) qdisc htb 2: dev eth0 parent 1: r2q 10 default 0 direct_packets_stat 22 Sent 2183574289 bytes 1496372 pkts (dropped 60982, overlimits 140759 requeues 0) qdisc pfifo 8007: dev eth0 parent 2:10 limit 5p Sent 7265998 bytes 51169 pkts (dropped 0, overlimits 0 requeues 0) qdisc red 8008: dev eth0 parent 2:20 limit 60Kb min 15Kb max 45Kb Sent 2176307367 bytes 1445181 pkts (dropped 60982, overlimits 60982 requeues 0) marked 0 early 60982 pdrop 0 other 0 qdisc pfifo 8009: dev eth1 limit 1000p Sent 33334496 bytes 477176 pkts (dropped 0, overlimits 0 requeues 0) qdisc pfifo 800a: dev eth2 limit 1000p Sent 40637134 bytes 585931 pkts (dropped 0, overlimits 0 requeues 0) Again here is the ASCII diagram (network A) --> (eth1) core router (eth0) --> (network C) (eth2) ^ | | (network B) >From network A to C (from 2 pcs used for the purpose of traffic generation) TCP traffic - pc 1 Sent 994762580 bytes 658704 pkts (dropped 0, overlimits 0 requeues 0) VoIP traffic - pc 2 Sent 7286487 bytes 51298 pkts (dropped 0, overlimits 0 requeues 0) >From network B to C TCP traffic Sent 1271745729 bytes 841217 pkts (dropped 27, overlimits 0 requeues 0) So total amount of packets transmitted to incoming interface on the core router is 658704 + 51298 + (841217 - 27) = 1,551,192 packets. The total sent by the dsmark and htb qdisc on the core router is 1,496,372 packets and 60,982 are dropped. The total received is 1,557,354. There is also some more traffic received from other nodes in network A, but this is minimal and also traffic from the core router itself. This should account for the difference. VoIP traffic sent from a machine in network A = 51298 packets. It is practically the same as the number of packets that pass through the pfifo 51169 which is attached to class 2:1 TCP traffic that should be passing through class 2:10 which is the BE class is 658,704 packets (TCP) from network A and 841,217 packets from B which equals which totals 1,499,921 traffic sent from the BE class is 1,445,181 + 60,982 packets which were dropped. So 1,506,163 packets were received by the BE class 2:10 The traffic sent from the output interface of eth1 and eth2 is mainly acks back to network A and network B respectively. >100mbit will be too high if it's 100mbit nic. What value would you recommend to set as the ceil for a 100 mbit NIC ??. >Don't know what pass_on will mean here. pass_on means if no class id equal to the result of the filter is found then try next filter, which is the BE class in this case. So back to the main question, could anyone tell me why the delay is so high (30ms) for VoIP packets which are treated with the EF phb when the outgoing interface of core router to network c is saturated ? Jonathan > Won't hurt if the packets are small voip but TBF has a nasty habit of > taking 1 from the burst/mtu you specify so your burst setting may result > in packets >1499B getting dropped - tc -s -d qdisc ls dev ... should > show what it's using. > Without wrapping it with something like htb red won't shape traffic. I am not to concerned about the PRIO + TBF setup. My priority is with the htb setup but I will look into this and see if I notice that. On Wed, 2005-07-27 at 14:25 +0100, Andy Furniss wrote: > Jonathan Lynch wrote: > > > Could anyone tell me why the delay is so high (30ms) for VoIP packets > > which are treated with the EF phb when the outgoing interface of core > > router to network c is saturated ? > > > > I have never used dsmark so am not sure about the classification parts > of your rules. You need to check where the packets are going with with > tc -s qdisc ls dev ... > > The other parts have some issues see below. > > > ----------------------------------------------------------------------------------------------- > > The config I used for each setup is included below. These are slight > > modifications that are supplied with iproute2 source code. > > > > Config 1 using htb > > ------------------- > > tc qdisc add dev $1 handle 1:0 root dsmark indices 64 set_tc_index > > tc filter add dev $1 parent 1:0 protocol ip prio 1 tcindex mask 0xfc > > shift 2 > > flowid/classid here maybe, to get packets to 2:0, though it may work - > check. > > > > > Main htb qdisc & class > > tc qdisc add dev $1 parent 1:0 handle 2:0 htb > > tc class add dev $1 parent 2:0 classid 2:1 htb rate 100Mbit ceil 100Mbit > > 100mbit will be too high if it's 100mbit nic. > > > > > EF Class (2:10) > > tc class add dev $1 parent 2:1 classid 2:10 htb rate 1500Kbit ceil > > 100Mbit > > tc qdisc add dev $1 parent 2:10 pfifo limit 5 > > tc filter add dev $1 parent 2:0 protocol ip prio 1 handle 0x2e tcindex > > classid 2:10 pass_on > > Don't know what pass_on will mean here. > > > > > BE Class (2:20) > > tc class add dev $1 parent 2:1 classid 2:20 htb rate 5Mbit ceil 100Mbit > > tc qdisc add dev $1 parent 2:20 red limit 60KB min 15KB max 45KB burst > > 20 avpkt 1000 bandwidth 100Mbit probability 0.4 > > tc filter add dev $1 parent 2:0 protocol ip prio 2 handle 0 tcindex mask > > 0 classid 2:20 pass_on > > > > Config 2 using PRIO > > ------------------- > > Main dsmark & classifier > > tc qdisc add dev $1 handle 1:0 root dsmark indices 64 set_tc_index > > tc filter add dev $1 parent 1:0 protocol ip prio 1 tcindex mask 0xfc > > shift 2 > > > > Main prio queue > > tc qdisc add dev $1 parent 1:0 handle 2:0 prio > > tc qdisc add dev $1 parent 2:1 tbf rate 1.5Mbit burst 1.5kB limit 1.6kB > > Won't hurt if the packets are small voip but TBF has a nasty habit of > taking 1 from the burst/mtu you specify so your burst setting may result > in packets >1499B getting dropped - tc -s -d qdisc ls dev ... should > show what it's using. > > > tc filter add dev $1 parent 2:0 protocol ip prio 1 handle 0x2e tcindex > > classid 2:1 pass_on > > > > BE class(2:2) > > tc qdisc add dev $1 parent 2:2 red limit 60KB min 15KB max 45KB burst 20 > > avpkt 1000 bandwidth 100Mbit probability 0.4 > > tc filter add dev $1 parent 2:0 protocol ip prio 2 handle 0 tcindex mask > > 0 classid 2:2 pass_on > > Without wrapping it with something like htb red won't shape traffic. > > Andy. From andy.furniss at dsl.pipex.com Wed Jul 27 23:53:11 2005 From: andy.furniss at dsl.pipex.com (Andy Furniss) Date: Wed Jul 27 23:52:58 2005 Subject: [LARTC] HTB and PRIO qdiscs introducing extra latency when output interface is saturated In-Reply-To: <1122478662.4637.49.camel@pgala.it.nuigalway.ie> References: <1122460110.8454.11.camel@pgala.it.nuigalway.ie> <42E78B39.1030201@dsl.pipex.com> <1122478662.4637.49.camel@pgala.it.nuigalway.ie> Message-ID: <42E80247.7010100@dsl.pipex.com> Jonathan Lynch wrote: > Andy, Many thanks for your reply. Below is some output from the queueing > disciplines to show that the filters are working correctly and they are > going to the right classes. OK classification looks good then. > > pass_on means if no class id equal to the result of the filter is found > then try next filter, which is the BE class in this case. Ahh I'll have to play with this dsmark stuff one day :-) > > So back to the main question, could anyone tell me why the delay is so > high (30ms) for VoIP packets which are treated with the EF phb when the > outgoing interface of core router to network c is saturated ? I would test next with htb setup like (assuming you are HZ=1000 - you will be under rate if not) - ... tc class add dev $1 parent 2:0 classid 2:1 htb rate 90Mbit ceil 90Mbit quantum 1500 burst 12k cburst 12k tc class add dev $1 parent 2:1 classid 2:10 htb rate 1500kbit ceil 90Mbit quantum 1500 burst 12k cburst 12k ... tc class add dev $1 parent 2:1 classid 2:20 htb rate 5Mbit ceil 90Mbit quantum 1500 burst 12k cburst 12k ... If that doesn't make things any better then you could try giving the 2:10 class a rate alot higher than it needs and see if that helps. Andy. From andy.furniss at dsl.pipex.com Thu Jul 28 00:23:40 2005 From: andy.furniss at dsl.pipex.com (Andy Furniss) Date: Thu Jul 28 00:23:28 2005 Subject: [LARTC] wrr vs. htb In-Reply-To: References: <20056271220192053@mail.routehat.org> Message-ID: <42E8096C.4040208@dsl.pipex.com> Kenneth Kalmer wrote: > On 7/27/05, Peter Surda wrote: > >>On Wed, 27 Jul 2005 12:01:06 +0200 Kenneth Kalmer >>wrote: >> >> >>>It's one server, fulfilling two functions, gateway en network service delivery >> >>Ok, now I get it. You can use IMQ and that will solve the problem. Or you can >>use a separate computer as a gateway. >> >> >>> :0 HTB >>> / \ >>> (HTB) 1:0 2:0 (HTB) >>> | | >>> WRR 1:1 1:2 WRR >> >>WRR doesn't like situations like this (don't ask why, some bug somewhere), the >>computer tends to freeze unpredictably. > > > Ouch, well now each of the WRR discs will be replaced by HTB discs for > each user, currently 140 discs for each WRR disc... Any other ways to > do this, or will HTB cope? This can expand up to 2x 509 discs... Do you really need to shape the local traffic at all though? I would be tempted to just shape inet traffic. If you use wrr then you won't be able to ceil each user anymore - if you don't mind loosing that, then you could also consider esfq, it's not as perfect as wrr but at least you get to choose a queue length more suitable for your link speed. > > How can I then handle the equality issues better, should I just pray > that HTB will divide all excess fairly? Also, I forgot to mention that > each HTB for each client get's an SFQ as well, just so that they don't > kill themselves in the process. Is this kosher? We are talking about shaping downloads here - if so then I think the equality problem could be more to do with loosing control while trying to shape from the wrong end of the bottleneck. You may well be dropping packets at ISP/teleco rather than in your queues - which with the amount of bandwidth per user you have is going to be tricky to sort. In fact unless you are using limit 10 or something on sfqs you may not be dropping any at all. How many users are active at peak times, are you doing nat and how much do they care about latency etc would affect how I would approach this. Andy. From nathan at iwantka.com Thu Jul 28 03:19:52 2005 From: nathan at iwantka.com (Nathan Littlepage) Date: Thu Jul 28 03:21:43 2005 Subject: [LARTC] how to classify sip traffic (voip) In-Reply-To: <3ab2553b050726060238be7524@mail.gmail.com> References: <3ab2553b050726060238be7524@mail.gmail.com> Message-ID: <42E832B8.3020401@iwantka.com> SIP uses 5060. What is hard is to classify is the RTP streams since they use pretty much any port. Its best to classify the SIP and RTP traffic at the endpoints and honor the labeled packets through the network. Fabian Gervan wrote: > How i can classify sip traffic (voip)?? > > I try dst 5060 udp port, but dont'work. sip sesion use dynamic port. > > Sniffing packets with windows net-peeker, I see that packets lenghts > is always=87 > How i can filter, by packet lenght, with u32? > > Regards > Fabian > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > > From johnm at advocap.org Thu Jul 28 17:49:40 2005 From: johnm at advocap.org (John McMonagle) Date: Thu Jul 28 17:50:09 2005 Subject: [LARTC] Routing for multiple uplinks/providers problem. Message-ID: <42E8FE94.2080506@advocap.org> Been running this for quite a while and noticed that have intermittent problems getting out. Find that if I ping the same site from 2 computers it may work on one and fail on the other. Also was surprised that some time they are going out different interfaces at the same time. Seems to work all the time from the firewall. Running 2.6.10 kernel with the multipath routing patches on a debian sarge system. # ip rule 0: from all lookup local 60: from all lookup main 200: from all lookup 200 201: from 216.170.136.0/24 lookup isp1 201: from 24.196.120.28/30 lookup isp2 222: from all lookup multi 222: from all lookup multi 32766: from all lookup main 32767: from all lookup default cat /etc/iproute2/rt_tables # # reserved values # 255 local 254 main 253 default 0 unspec # # local # 1 inr.ruhep 201 isp1 202 isp2 222 multi root@fonroute:~# ip route list table 200 192.168.0.0/16 via 192.168.2.254 dev eth0 root@fonroute:~# ip route list table 201 default via 216.170.136.1 dev eth1 proto static src 216.170.136.82 prohibit default proto static metric 1 root@fonroute:~# ip route list table 202 default via 24.196.120.29 dev eth2 proto static src 24.196.120.30 prohibit default proto static metric 1 root@fonroute:~# ip route list table 222 default proto static nexthop via 216.170.136.1 dev eth1 weight 1 nexthop via 24.196.120.29 dev eth2 weight 4 using shorewall to setup rules. iptable -L Chain PREROUTING (policy ACCEPT) target prot opt source destination net_dnat all -- anywhere anywhere net_dnat all -- anywhere anywhere loc_dnat all -- anywhere anywhere Chain POSTROUTING (policy ACCEPT) target prot opt source destination eth1_masq all -- anywhere anywhere eth2_masq all -- anywhere anywhere Chain OUTPUT (policy ACCEPT) target prot opt source destination Chain eth1_masq (1 references) target prot opt source destination masq2 all -- 192.168.2.0/24 anywhere Chain eth2_masq (1 references) target prot opt source destination masq1 all -- 192.168.2.0/24 anywhere Chain loc_dnat (1 references) target prot opt source destination REDIRECT tcp -- anywhere anywhere tcp dpt:www redir ports 3128 Chain masq1 (1 references) target prot opt source destination RETURN all -- anywhere 192.168.0.0/16 RETURN all -- fonroute.advocap.org anywhere SNAT all -- anywhere anywhere to:24.196.120.30 Chain masq2 (1 references) target prot opt source destination RETURN all -- anywhere 192.168.0.0/16 RETURN all -- fonroute.advocap.org anywhere SNAT all -- anywhere anywhere to:216.170.136.73 Chain net_dnat (2 references) target prot opt source destination DNAT tcp -- !192.168.0.0/16 anywhere multiport dports ssh,www to:192.168.2.1 DNAT tcp -- !192.168.0.0/16 anywhere multiport dports smtp,imaps,https to:192.168.2.10 DNAT tcp -- !192.168.0.0/16 anywhere tcp dpt:2525 to:192.168.2.10:25 DNAT tcp -- !192.168.0.0/16 anywhere tcp dpt:8000 to:192.168.2.12:443 DNAT tcp -- !192.168.0.0/16 anywhere tcp dpt:9000 to:192.168.2.12:22 REDIRECT tcp -- anywhere anywhere tcp dpt:1022 redir ports 22 Have snat on both interfaces Have rules to keep vpn traffic from getting snated. Any solution? Any way to troubleshoot? John -------------- next part -------------- A non-text attachment was scrubbed... Name: johnm.vcf Type: text/x-vcard Size: 250 bytes Desc: not available Url : http://mailman.ds9a.nl/pipermail/lartc/attachments/20050728/bf80d741/johnm.vcf From hans at middelhoek.nl Thu Jul 28 18:07:30 2005 From: hans at middelhoek.nl (Hans Middelhoek.nl) Date: Thu Jul 28 18:08:02 2005 Subject: [LARTC] multiple uplinks works on linux router but not on clients Message-ID: <001401c5938e$753ae400$8214a8c0@hanslt> Hello, I'm trying to achieve the following. 2 ISP's and 1 LAN with failover. So, when 1 internet connection fails the other need to take over. I'm aware of the restrictions. The current sessions are lost. I did this before, one and a half year ago or something. But at that router wasn't failover. Now i've read the nano howto and followed it. I'm using Debian with kernel 2.6.12-3 patched with Julian Anastasov's routes patch. This integrated failover functionality. I need to say. It works great on the machine itself. Everytime I start a traceroute another link will be used. When I unplug an internet link, all of my traceroutes uses the internet connections which are up. When I re-plug the internet link my traceroutes will use the one again. This is how it needs to be, also for the connected pc's. But that's not working. All traceroutes on client machine are going through the same router: 192.168.20.1->192.168.32.1->internet I used following firewall rules, but that shouldn't be the problem: iptables -t filter -N keep_state iptables -t filter -A keep_state -m state --state RELATED,ESTABLISHED -j ACCEPT iptables -t filter -A keep_state -j RETURN iptables -t nat -N keep_state iptables -t nat -A keep_state -m state --state RELATED,ESTABLISHED -j ACCEPT iptables -t nat -A keep_state -j RETURN iptables -t nat -A POSTROUTING -o eth1 -s 192.168.20.0/24 -j SNAT --to 192.168.1.240 iptables -t nat -A POSTROUTING -o eth2 -s 192.168.20.0/24 -j SNAT --to 10.0.0.240 iptables -t nat -A POSTROUTING -o eth3 -s 192.168.20.0/24 -j SNAT --to 192.168.32.240 iptables -t nat -A PREROUTING -j keep_state iptables -t nat -A POSTROUTING -j keep_state iptables -t nat -A OUTPUT -j keep_state iptables -t filter -A INPUT -j keep_state iptables -t filter -A FORWARD -j keep_state iptables -t filter -A OUTPUT -j keep_state It looks like the problem comes from my multi route table 222. When I change the order of the devices in the multiroute rule all of my traceroutes on the clients are using the last nexthop via. I'm using three eth's as external interfaces and always the last one in my multihop route will be used by my workstations. Setup: ip rule ls 0: from all lookup local 50: from all lookup main 201: from 192.168.1.0/24 lookup 201 202: from 10.0.0.0/24 lookup 202 203: from 192.168.32.0/24 lookup 203 222: from all lookup 222 32766: from all lookup main 32767: from all lookup default ip ro s t 222 default proto static nexthop via 192.168.1.1 dev eth1 weight 1 nexthop via 10.0.0.138 dev eth2 weight 1 nexthop via 192.168.32.1 dev eth3 weight 1 ip ro s t 201 default via 192.168.1.1 dev eth1 proto static src 192.168.1.240 prohibit default proto static metric 1 ip ro s t 202 default via 10.0.0.138 dev eth2 proto static src 10.0.0.240 prohibit default proto static metric 1 ip ro s t 203 default via 192.168.32.1 dev eth3 proto static src 192.168.32.240 prohibit default proto static metric 1 Well, I think you have all the information. What I want to do is simply routing on multiple interfaces. No queue based routing or something. Linux may just choose which route is going to be used. Thanks for your help in advance. Kind regards, Hans Middelhoek Netherlands -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ds9a.nl/pipermail/lartc/attachments/20050728/b1356c6a/attachment.htm From nelsoneci at gmail.com Thu Jul 28 18:18:42 2005 From: nelsoneci at gmail.com (Nelson Castillo) Date: Thu Jul 28 18:18:47 2005 Subject: [LARTC] Routing for multiple uplinks/providers problem. In-Reply-To: <42E8FE94.2080506@advocap.org> References: <42E8FE94.2080506@advocap.org> Message-ID: <2accc2ff050728091870b1bbfb@mail.gmail.com> Hi John. On 7/28/05, John McMonagle wrote: > Find that if I ping the same site from 2 computers it may work on one > and fail on the other. > Also was surprised that some time they are going out different > interfaces at the same time. Same symptoms I had. > Have snat on both interfaces When you SNAT incoming packets, you need to do something different from what is in the HOWTO ([4]) because SNAT is done before the routing desition (check the Kernel Packet Traveling Diagram[5]). I had the same problem [1]. The solution is to use conntrack and mark packets on arrival, and then route them back using the fwmark[2]. There's no need to tell you I had a hard time with this. There should be a warning about this in the HOWTO (in this page [4]). The proposed solution I quote in [2] worked for me for the multiple uplink providers + SNAT problem. It is (Using the same variables that are in the HOWTO [4]): 1) Mark packages on arrival: iptables -t mangle -A PREROUTING -m conntrack --ctorigdst $IP1 -j MARK --set-mark=1 iptables -t mangle -A PREROUTING -m conntrack --ctorigdst $IP2 -j MARK --set-mark=2 And then use the mark to route the outgoing packages correctly. ip rule add fwmark 1 table T1 ip rule add fwmark 2 table T2 Regards, Nelson.- PD : I solved my problem with IPVS and multiple uplink providers (see [3]). [1] http://mailman.ds9a.nl/pipermail/lartc/2005q2/016171.html [2] http://mailman.ds9a.nl/pipermail/lartc/2005q2/016441.html [3] http://arhuaco.blogspot.com/2005/07/ipvs-and-conntrack.html [4] http://lartc.org/howto/lartc.rpdb.multiple-links.html [5] http://www.docum.org/docum.org/kptd/ -- Homepage : http://geocities.com/arhuaco The first principle is that you must not fool yourself and you are the easiest person to fool. -- Richard Feynman. From nelsoneci at gmail.com Thu Jul 28 18:32:17 2005 From: nelsoneci at gmail.com (Nelson Castillo) Date: Thu Jul 28 18:32:27 2005 Subject: [LARTC] Routing for multiple uplinks/providers problem. In-Reply-To: <2accc2ff050728091870b1bbfb@mail.gmail.com> References: <42E8FE94.2080506@advocap.org> <2accc2ff050728091870b1bbfb@mail.gmail.com> Message-ID: <2accc2ff05072809322a58abdb@mail.gmail.com> I think I said something wrong in my last message. You DNAT incoming packets and then SNAT them when they come back if your Linux router has some server behind it. I don't know if this is your case (having servers behind the router). (I needed to top-post here --- maybe not). On 7/28/05, Nelson Castillo wrote: > Hi John. > > On 7/28/05, John McMonagle wrote: > > > Find that if I ping the same site from 2 computers it may work on one > > and fail on the other. > > Also was surprised that some time they are going out different > > interfaces at the same time. > > Same symptoms I had. > > > Have snat on both interfaces > > When you SNAT incoming packets, you need to do something different > from what is in the HOWTO ([4]) because SNAT is done before the > routing desition (check the Kernel Packet Traveling Diagram[5]). > > I had the same problem [1]. The solution is to use conntrack and mark > packets on arrival, and then route them back using the fwmark[2]. > > There's no need to tell you I had a hard time with this. There should > be a warning about this in the HOWTO (in this page [4]). > > The proposed solution I quote in [2] worked for me for the > multiple uplink providers + SNAT problem. > > It is (Using the same variables that are in the HOWTO [4]): > > 1) Mark packages on arrival: > > iptables -t mangle -A PREROUTING -m conntrack --ctorigdst $IP1 -j > MARK --set-mark=1 > iptables -t mangle -A PREROUTING -m conntrack --ctorigdst $IP2 -j > MARK --set-mark=2 > > And then use the mark to route the outgoing packages correctly. > > ip rule add fwmark 1 table T1 > ip rule add fwmark 2 table T2 > > Regards, > Nelson.- > > PD : I solved my problem with IPVS and multiple uplink providers (see [3]). > > [1] http://mailman.ds9a.nl/pipermail/lartc/2005q2/016171.html > [2] http://mailman.ds9a.nl/pipermail/lartc/2005q2/016441.html > [3] http://arhuaco.blogspot.com/2005/07/ipvs-and-conntrack.html > [4] http://lartc.org/howto/lartc.rpdb.multiple-links.html > [5] http://www.docum.org/docum.org/kptd/ > > -- > Homepage : http://geocities.com/arhuaco > > The first principle is that you must not fool yourself > and you are the easiest person to fool. > -- Richard Feynman. From jlynch at frink.nuigalway.ie Thu Jul 28 18:37:18 2005 From: jlynch at frink.nuigalway.ie (Jonathan Lynch) Date: Thu Jul 28 18:37:49 2005 Subject: [LARTC] HTB and PRIO qdiscs introducing extra latency when output interface is saturated In-Reply-To: <42E80247.7010100@dsl.pipex.com> References: <1122460110.8454.11.camel@pgala.it.nuigalway.ie> <42E78B39.1030201@dsl.pipex.com> <1122478662.4637.49.camel@pgala.it.nuigalway.ie> <42E80247.7010100@dsl.pipex.com> Message-ID: <1122568638.7828.40.camel@pgala.it.nuigalway.ie> Andy, thanks again for your help. Yes, HZ is still 1000 in 2.6.12. I tried your suggestions are here are the results. ASCII diagram (network A) --> (eth1) core router (eth0) --> (network C) (eth2) ^ | | (network B) Looking at the following graphics http://140.203.56.30/~jlynch/htb/core_router.png http://140.203.56.30/~jlynch/htb/voip_stream_23691.png voip_stream_23691.png is a graph of the delay of the voice stream travelling from network A to network C in test 2. Notice from the core router graph that there is only voip traffic passing through the core router until time 07:55 and the delay in voip stream is 0.25 ms until then. After this time tcp traffic is introduced saturating the outgoing interface of the core router (eth0). The delay increases to a maximum of 2.75 ms , which is a considerable improvement on 30ms when I was using the ceil value of 100mbit. But there is a lot of jitter. With the ceil at 90Mbit, the outgoing bit rate of eth0 has gone from 98mbit to approx 90Mbit as can be seen from the core router graph for eth0 bytes out. Note that with the tcp traffic is all http downloads, so most Ethernet frames will be of maximum size, 1518 bytes, so 98mbits is the maximum throughput possible on a 100mbit card, taking into account the overheads of ethernet such as the interframe gap, preamble and start frame delimiter. Im not sure how to configure some of the htb parameters. The following is my understanding of them and a few questions I have as well. How exactly does the HZ value have a bearing on the ceil value ? How can I calculate a maximum for the ceil value ? 12kb is the minimum burst size for a 100 mbit NIC with a timer resolution of 1ms (1000hz) and tc calculates the smallest possible burst when it is not specified, right ?. cburst is the number of bytes that can be burst as fast as the interface can transmit them. It is smaller than burst can is ideally one packet size, right ? quantum determines the ratio at which the classes share their parents bandwidth. Each class is given quantum number of bytes before serving the next class, right ? Is there any way I can limit the jitter of the VoIP traffic passing through the htb class ? Jonathan On Wed, 2005-07-27 at 22:53 +0100, Andy Furniss wrote: > Jonathan Lynch wrote: > > Andy, Many thanks for your reply. Below is some output from the queueing > > disciplines to show that the filters are working correctly and they are > > going to the right classes. > > OK classification looks good then. > > > > > > pass_on means if no class id equal to the result of the filter is found > > then try next filter, which is the BE class in this case. > > Ahh I'll have to play with this dsmark stuff one day :-) > > > > > So back to the main question, could anyone tell me why the delay is so > > high (30ms) for VoIP packets which are treated with the EF phb when the > > outgoing interface of core router to network c is saturated ? > > I would test next with htb setup like (assuming you are HZ=1000 - you > will be under rate if not) - > > ... > > tc class add dev $1 parent 2:0 classid 2:1 htb rate 90Mbit ceil 90Mbit > quantum 1500 burst 12k cburst 12k > > tc class add dev $1 parent 2:1 classid 2:10 htb rate 1500kbit ceil > 90Mbit quantum 1500 burst 12k cburst 12k > ... > > tc class add dev $1 parent 2:1 classid 2:20 htb rate 5Mbit ceil 90Mbit > quantum 1500 burst 12k cburst 12k > > ... > > > If that doesn't make things any better then you could try giving the > 2:10 class a rate alot higher than it needs and see if that helps. > > > Andy. From hans at middelhoek.nl Thu Jul 28 20:22:17 2005 From: hans at middelhoek.nl (Hans Middelhoek.nl) Date: Thu Jul 28 20:23:07 2005 Subject: [LARTC] multiple uplinks works on linux router but not on clients Message-ID: <001b01c593a1$53f65f00$8214a8c0@hanslt> Hi, I've been testing the whole time and found some interesting information. Obviously I haven't compiled the kernel very well. I booted the machine in it's old kernel, 2.6.8-2 and now the pc's in my network are using all internet interfaces. I see that with a traceroute. What did I do wrong in compiling and patching the kernel. Probably I missed an important setting in the kernel configuration, but which? It's the first time for me I compiled a kernel and it was very though to have it working. I got a lot of errors during the first tries. Later on, compiling succeeded without any errors, so I thought it was working fine. I can see that the patch applied, because failover works, but there is something else what doesn't work ad it should. Who understand this strange problem? thnx. From payal-lartc at scriptkitchen.com Thu Jul 28 21:00:51 2005 From: payal-lartc at scriptkitchen.com (Payal Rathod) Date: Thu Jul 28 21:00:54 2005 Subject: [LARTC] wondershaper query Message-ID: <20050728190051.GA32397@tranquility.scriptkitchen.com> Hello, I am trying wondershaper-1.1a on a friend's pppoe connection on her Linux box. There are a few things I don't understand. 1. She has pppoe connection so should DEV=eth0 or DEV=ppp0 ? 2. Her ISP just says on her payment bill that the speed is 128kbps, but doesn't mention any downlink/uplink speed, so in that case what should be, DOWNLINK= and UPLINK= ? 3. She uses the net in her small office and people mostly to browse the net, send emails sometimes ftp data out and sometimes ssh to other servers to trouble shoot their programs. In such a case is wondershaper helpful? Or is it not required at all? Thanks in advance. With warm regards, Payal From andy.furniss at dsl.pipex.com Thu Jul 28 23:49:54 2005 From: andy.furniss at dsl.pipex.com (Andy Furniss) Date: Thu Jul 28 23:49:34 2005 Subject: [LARTC] HTB and PRIO qdiscs introducing extra latency when output interface is saturated In-Reply-To: <1122568638.7828.40.camel@pgala.it.nuigalway.ie> References: <1122460110.8454.11.camel@pgala.it.nuigalway.ie> <42E78B39.1030201@dsl.pipex.com> <1122478662.4637.49.camel@pgala.it.nuigalway.ie> <42E80247.7010100@dsl.pipex.com> <1122568638.7828.40.camel@pgala.it.nuigalway.ie> Message-ID: <42E95302.2020303@dsl.pipex.com> Jonathan Lynch wrote: > Andy, thanks again for your help. Yes, HZ is still 1000 in 2.6.12. I > tried your suggestions are here are the results. > > ASCII diagram > > (network A) --> (eth1) core router (eth0) --> (network C) > (eth2) > ^ > | > | > (network B) > > Looking at the following graphics > > http://140.203.56.30/~jlynch/htb/core_router.png > http://140.203.56.30/~jlynch/htb/voip_stream_23691.png > > voip_stream_23691.png is a graph of the delay of the voice stream > travelling from network A to network C in test 2. Notice from the core > router graph that there is only voip traffic passing through the core > router until time 07:55 and the delay in voip stream is 0.25 ms until > then. After this time tcp traffic is introduced saturating the outgoing > interface of the core router (eth0). The delay increases to a maximum of > 2.75 ms , which is a considerable improvement on 30ms when I was using > the ceil value of 100mbit. But there is a lot of jitter. I suppose you could hope for a bit less jitter 12k burst is about 1ms at 100mbit. There is a tweak you can do for htb which may help - in net/sched/sch_htb.c there is a #define HYSTERESIS 1 - changing it to 0 and recompiling kernel/the module makes things more accurate. > > > With the ceil at 90Mbit, the outgoing bit rate of eth0 has gone from > 98mbit to approx 90Mbit as can be seen from the core router graph for > eth0 bytes out. Note that with the tcp traffic is all http downloads, so > most Ethernet frames will be of maximum size, 1518 bytes, so 98mbits is > the maximum throughput possible on a 100mbit card, taking into account > the overheads of ethernet such as the interframe gap, preamble and start > frame delimiter. > > Im not sure how to configure some of the htb parameters. The following > is my understanding of them and a few questions I have as well. > > How exactly does the HZ value have a bearing on the ceil value ? How can > I calculate a maximum for the ceil value ? It's more to do with burst/cburst than ceil. > > 12kb is the minimum burst size for a 100 mbit NIC with a timer > resolution of 1ms (1000hz) and tc calculates the smallest possible burst > when it is not specified, right ?. It seems not, I think hysteresis may be involved again here (but then one of my tcs is hacked about a bit). You can see what htb is using as defaults by doing tc -s -d class ls .. If I do that on similar kernels one with hysteresis 0 and one with 1 I see quite different values. I chose 12k as big enough for the 90mbit test 12000*8*1000=96mbit at ip level and it seemed like a nice multiple of 1500mtu :-) > > cburst is the number of bytes that can be burst as fast as the interface > can transmit them. It is smaller than burst can is ideally one packet > size, right ? Ideally 1 packet but not achievable with htb at lan speed and hz 1000, also AIUI the way htb does drr means with mixed packet sizes things aren't packet perfect even at low rates. Saying that I use htb at low rates and can apparently get packet perfect with my traffic mix. I think hfsc can do it perfectly on both counts. > > quantum determines the ratio at which the classes share their parents > bandwidth. Each class is given quantum number of bytes before serving > the next class, right ? Yea setting 1500 probably makes no difference for this test. > > Is there any way I can limit the jitter of the VoIP traffic passing > through the htb class ? Try the hysteresis and/or setting the rate for interactive way higher than it's traffic rate. I did a quick test to see how things were for me at 100mbit. Because my other pcs are slow I needed to use two as receivers for netperf. I noticed something I didn't expect with red or the settings you use - one of the pcs is slower and has less memory thus smaller tcp socket size. Using 4 streams two to each unshaped they get about the same, though with txqueuelen = 1000 there are no drops (with window scalng off there is a difference). With red and wscale on, the red really favoured the fast pc - I tried a 40k bfifo so that I got drops, expecting to see the same, but it was still far more even than the red. I couldn't really simulate the voip traffic in theory I should be able to use ping with -i < 1 sec, but using the latest inetutils you get a flood ping if you do that. I reported this about 18 months ago and it's supposedly fixed in the cvs (though I don't know if fixed means it just says invalid argument rather than actually does what's asked, because I have failed to build it so far). So if anyone reading this has a i386 ping that -i 0.5 works on, please mail me the binary :-) Andy. From msc at antzsystem.de Fri Jul 29 11:40:23 2005 From: msc at antzsystem.de (Markus Schulz) Date: Fri Jul 29 11:40:34 2005 Subject: [LARTC] 2 load-balanced adsl cons and ftp-server w/out site-to-site transfer Message-ID: <200507291140.23459.msc@antzsystem.de> Hello, how it's possible to download something from ftp servers without site-to-site transfer enabled? If the data transfer connection will be routed through another outgoing interface than the command connection the server denies the connection. Are there any solutions for instance with ip_conntrack_ftp module or similar? -- Markus Schulz From zeus_securenet at hotmail.com Fri Jul 29 23:20:04 2005 From: zeus_securenet at hotmail.com (ZeuS SN) Date: Fri Jul 29 23:20:06 2005 Subject: [LARTC] Load Balancing and NAT Clients Problem Message-ID: <42EA9D84.7050402@hotmail.com> Hi, The problem is that if I set the default route with two gateways, with the command 'ip route change default scope global nexthop via $P1 dev $IF1 weight 1 nexthop via $P1 dev $IF1 weight 1, in the linux router I have no problem working with internet but the NATTED clients can't access. I have iptables configured with masquerade option. What I forget to do? Thanks and Regards... From ddaasd at gmail.com Sat Jul 30 18:03:02 2005 From: ddaasd at gmail.com (ddaas) Date: Sat Jul 30 18:03:11 2005 Subject: [LARTC] Qos wiht HTB for ADSL/Home Message-ID: <42EBA4B6.7050409@gmail.com> Hi, My home setup is as following: - 1024/128 kbit ADSL - FC3 I set up HTB to prioritize traffic. I am not very pleased with the obtained results. The scope of my setup is to have some ssh sessions with remote servers while browsing websites and running aMule Nothing complicated (I think... ;). The very high priority traffic (ssh), gets stuck when I start aMule and make an FTP download. Although there is an improvement. High priority (WEB) traffic has some delay when I have a lot of bulk traffic (aMule, FTP). It waits a few seconds for DNS lookup, and other synchronizations and then in a blink of the eye the whole page appears. I think that this means that the queue is still to long. So, I have some questions: 1) Do I need to use ppp0 or eth0 for shaping traffic? 2) Are the default values for quantum, burst, cburst acceptable or should I tune these values? 3) I've seen in some tutorials that decreasing the queue (as I know it is 100 in Linux as the default) with #ip link set dev eth0 qlen, they have been obtained better results. Should I try to decrease the queue? To what value? 3) The same with mtu. A less value (ex 1000) should be better? 4) If your are in my situation (ADSL ? for home), what is your htb configuration? 5)Does it worth configuring policing for inbound traffic (with imq)? 6)So, am I on the right way with my script or should I change it fundamental? In my setup I have 3 categories of traffic: 1)very high priority traffic: ssh, DNS, ACK, SYN+ACK, RST, FIN+ACK (40kbit > CEIL) 2) high priority traffic: web, online radio, online tv (72kbit->CEIL) 3)low priority traffic - default class: aMule, P2P, other type of traffic(8Kbit->CEIl) THANKS, ddaas My htb script is: #!/bin/bash YAHOOMSG="216.155.193.0/24" RADIO="xxx.xxx.xxx.xxx" TRI="xxx.dnsalias.net" TV="xxx.xxx.xxx.xxx" DNSSERVER="217.237.150.225" IF="ppp0" CEIL="120" #flush everything tc qdisc del dev $IF root iptables -F OUTPUT -t mangle #root qdisc tc qdisc add dev $IF root handle 1: htb default 12 #class 1:1 attached to root qdisc tc class add dev $IF parent 1: classid 1:1 htb rate ${CEIL}kbit ceil ${CEIL}kbit #class 1:10 -> 40kbit-CEIL for ACK,SSH tc class add dev $IF parent 1:1 classid 1:10 htb rate 40kbit ceil ${CEIL}kbit prio 1 #class 1:11 (web,radio,tv) tc class add dev $IF parent 1:1 classid 1:11 htb rate 72kbit ceil ${CEIL}kbit prio 2 #default class 1:12 for bulk traffic (aMule, ftp) tc class add dev $IF parent 1:1 classid 1:12 htb rate 8kbit ceil ${CEIL}kbit prio 5 #mark ack,syn ack,rst,fin, ssh packets and add them to class 1:10 (high prioroty) iptables -A OUTPUT -t mangle -o $IF -p tcp --sport 22 -j MARK --set-mark 1 iptables -A OUTPUT -t mangle -o $IF -p tcp --sport 22 -j RETURN iptables -A OUTPUT -t mangle -o $IF -p tcp --dport 22 -j MARK --set-mark 1 iptables -A OUTPUT -t mangle -o $IF -p tcp --dport 22 -j RETURN iptables -A OUTPUT -t mangle -o $IF -p tcp --tcp-flags SYN,RST,ACK SYN,FIN -j MARK --set-mark 1 iptables -A OUTPUT -t mangle -o $IF -p tcp --tcp-flags SYN,RST,ACK SYN,FIN -j RETURN #mark dns packets iptables -A OUTPUT -t mangle -o $IF -d $DNSSERVER -j MARK --set-mark 1 iptables -A OUTPUT -t mangle -o $IF -d $DNSSERVER -j RETURN tc filter add dev $IF parent 1: protocol ip handle 1 fw classid 1:10 #mark www,radio,tv,tri,yahoomsg iptables -A OUTPUT -t mangle -o $IF -p tcp -m multiport --dports 80,443 -j MARK --set-mark 2 iptables -A OUTPUT -t mangle -o $IF -d $RADIO -p tcp --dport 8000 -j MARK --set-mark 2 iptables -A OUTPUT -t mangle -o $IF -d $YAHOOMSG -p tcp --dport 5050 -j MARK --set-mark 2 iptables -A OUTPUT -t mangle -o $IF -d $TV -p tcp -m multiport --dports 554,6970 -j MARK --set-mark 2 iptables -A OUTPUT -t mangle -o $IF -d $TRI -j MARK --set-mark 2 tc filter add dev $IF parent 1: protocol ip handle 2 fw classid 1:11 #other type of traffic in default class 1:12 tc qdisc add dev $IF parent 1:10 handle 100: sfq perturb 10 tc qdisc add dev $IF parent 1:11 handle 110: sfq perturb 10 tc qdisc add dev $IF parent 1:12 handle 120: sfq perturb 10 ##INGRESS - drop everithing over 1000kbit tc qdisc del dev $IF ingress tc qdisc add dev $IF handle ffff: ingress tc filter add dev $IF parent ffff: protocol ip prio 10 u32 match \ ip src 0.0.0.0/0 police rate 1000kbit burst 80kbit drop flowid :1 From ddaasd at gmail.com Sat Jul 30 18:17:16 2005 From: ddaasd at gmail.com (ddaas) Date: Sat Jul 30 18:17:54 2005 Subject: [LARTC] Qos wiht HTB for ADSL/Home Message-ID: <42EBA80C.2050001@gmail.com> I've forgotten something. After a period of all type of traffic (very high priority, high priority, low priority) the redistribution of the traffic doesn't look ok. The 1:11 class (high priority) didn't borrow a byte but the low priority class (1:12) borrowed a lot. I get with tc -s class show dev ppp0: class htb 1:11 parent 1:1 leaf 110: prio 2 rate 72Kbit ceil 100Kbit burst 1608b cburst 1611b Sent 659962 bytes 12307 pkts (dropped 0, overlimits 0 requeues 0) rate 1703bit 32pps lended: 12307 borrowed: 0 giants: 0 tokens: 177607 ctokens: 128123 class htb 1:1 root rate 100Kbit ceil 100Kbit burst 1611b cburst 1611b Sent 6398190 bytes 29624 pkts (dropped 0, overlimits 0 requeues 0) rate 11456bit 68pps lended: 15414 borrowed: 0 giants: 0 tokens: -151656 ctokens: -151656 class htb 1:10 parent 1:1 leaf 100: prio 1 rate 40Kbit ceil 100Kbit burst 1604b cburst 1611b Sent 1477 bytes 23 pkts (dropped 0, overlimits 0 requeues 0) lended: 23 borrowed: 0 giants: 0 tokens: 317236 ctokens: 127468 class htb 1:12 parent 1:1 leaf 120: prio 5 rate 8Kbit ceil 100Kbit burst 1600b cburst 1611b Sent 5737669 bytes 17311 pkts (dropped 0, overlimits 0 requeues 0) rate 10165bit 34pps backlog 17p lended: 1880 borrowed: 15414 giants: 0 tokens: -1945424 ctokens: -105185 From lartc-list at the-morg.org Sat Jul 30 18:32:53 2005 From: lartc-list at the-morg.org (Darryl Miles) Date: Sat Jul 30 18:33:06 2005 Subject: [LARTC] multi-path TCP performance In-Reply-To: <628BFCE8B64706469FE4D4852CEC953706D50482@tayexc14.americas.cpqcorp.net> References: <628BFCE8B64706469FE4D4852CEC953706D50482@tayexc14.americas.cpqcorp.net> Message-ID: <42EBABB5.4030908@the-morg.org> Maybe one of the TCP options is intefering with the out-of-order reception the receiving end experiences. Try to disable all options you can and repeat. Research into why/what each option is there and does. Some options are for the other end of the performance spectrum, window scaling for example. So they wont provide any assistance to your situatution. My guess would be SACK (selective acknowledgement) is causing the reciving end to signal to the sending to retransmit the (apparently) lost packets it sees. When in reality these packets are delayed not lost and it just doesn't know yet. So disable sack on linux try "echo 0 > /proc/sys/net/ipv4/tcp_sack" try this as both ends (but maybe only your bottlenecked / teql end needs it done) There is also a delayed ack mechanism that trys to reduce acks flowing the other way and also add some additional wait to the reception of marginally delayed data packets so they can be colated before the ack is send back, maybe the amount of time can be increased to help colation (providing this is kept within some % of the overall route RTT). If the receiving end receives multiple ack packets with the same sequence number it starts to conclude the data just beyond that acked has gone missing, after 3 in a row sending starts to shut down and the sending end and spits out another retransmission of what it believes to be the lost packet. This is how it worked BEFORE SACK became the default anyway, this is some TCP fast-ack mechanism. What % of the Round Trip Time does the delay constitute ? You talk of 1ms and 5ms deviation, if you are talking about RTT being ethernet like speeds then 5ms is a long time. All TCP timings are dynamic around what the sending side computes the RTT to be as the goal of sending bulk TCP data is to fill the virtual pipeline between sender and receiver. But to do this in a way that is co-operative with other users. Lost or delayed packets are the pricipal indicator the route is congested and therefore the sending site backs off. If your best RTT is 7ms and worst 12ms you can't expect a few simple options to make much difference. However if the overall RTT is in the order of 70+ms there maybe plenty of room to see some improvements with configuration changes. Can you improve the load balancing at the congested sending end ? For example have you made sure there is only a single packet transmitted queue at the interfaces. "ifconfig ppp0 txqueue 1" or some other low number like 2 or 3. The default looks to be 64 these days, this is too much if your teql interface also has a queue and the ppp0 interface goes and asks teql for another packet everytime is has space for one. Just some pointers for you. Darryl Li, Ji wrote: > I am measuring the performance of one TCP connection over two > symmetric paths. Packets are sent to two paths alternatively. I found > that when the latency of each path are within 1ms, the overall TCP > throughput is the *sum* of the throughput of the two paths. However, > when the latency of the two paths increases to 5ms, the overal TCP > throughput drops to the throughput of a *single* path. Has anyone > studied similar problem? What makes the performance go down? > > I use Fedora Core 3 and 4, teql and netem for my emulation. > >------------------------------------------------------------------------ > >_______________________________________________ >LARTC mailing list >LARTC@mailman.ds9a.nl >http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > > From Andreas.Klauer at metamorpher.de Sat Jul 30 19:11:45 2005 From: Andreas.Klauer at metamorpher.de (Andreas Klauer) Date: Sat Jul 30 19:11:43 2005 Subject: [LARTC] Qos wiht HTB for ADSL/Home In-Reply-To: <42EBA4B6.7050409@gmail.com> References: <42EBA4B6.7050409@gmail.com> Message-ID: <200507301911.45791.Andreas.Klauer@metamorpher.de> On Saturday 30 July 2005 18:03, ddaas wrote: > 4) If your are in my situation (ADSL ? for home), what is your htb > configuration? http://www.metamorpher.de/fairnat/ It's for a Linux gateway/router and more than one user though. HTB is used to divide bandwidth between users, and PRIO for prioritization of different traffic types per user. There's also another setup that uses HTB only (Wondershaper style), but I never really used that one myself. I've tried my best to make the script easy to understand, with documentation for each function and such. Quite a few people who didn't use this script directly still found it useful as an example. There are still a lot of things that need improvements, naturally. Unfortunately, I'll move out of the flat I was living in soon, due to switching Universities. And my new home won't have a shared DSL line anymore, so I will no longer have the means to further develop and test the script. > 5)Does it worth configuring policing for inbound traffic (with imq)? In my case, yes, it was. Not only worth it, but absolutely necessary. HTH Andreas From gregoriandres at yahoo.com.ar Sun Jul 31 00:51:31 2005 From: gregoriandres at yahoo.com.ar (:: L i n u XK i D ::) Date: Sun Jul 31 00:51:47 2005 Subject: [LARTC] what is "tcp window size" ? Message-ID: what is "tcp window size" ? thank you From jarod125 at yahoo.com Sun Jul 31 01:59:04 2005 From: jarod125 at yahoo.com (Gabriel) Date: Sun Jul 31 01:59:08 2005 Subject: [LARTC] what is "tcp window size" ? In-Reply-To: Message-ID: <20050730235904.8074.qmail@web60921.mail.yahoo.com> The TCP header has a 16 bits field that specifies (and now I'm quoting from the RFC 793): "The number of data octets beginning with the one indicated in the acknowledgment field which the sender of this segment is willing to accept." In plain english: how much "useful" data you can accept from the sender. > what is "tcp window size" ? > > thank you __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From gregoriandres at yahoo.com.ar Sun Jul 31 02:38:58 2005 From: gregoriandres at yahoo.com.ar (:: L i n u XK i D ::) Date: Sun Jul 31 02:38:38 2005 Subject: [LARTC] what is "tcp window size" ? In-Reply-To: <20050730235904.8074.qmail@web60921.mail.yahoo.com> Message-ID: Thank you very much... What values are recommended for diferent situations ? for a Vsat link ? ADSL ? LAN ? ... and so on... is important to set this value on linux ? If you have a URL to get more information, please post it . thanks again... andres. -> -> -> The TCP header has a 16 bits field that specifies (and now -> I'm quoting from the RFC 793): "The number of data octets -> beginning with the one indicated in the acknowledgment -> field which the sender of this segment is willing to -> accept." In plain english: how much "useful" data you can -> accept from the sender. -> -> > what is "tcp window size" ? -> > -> > thank you -> -> -> __________________________________________________ -> Do You Yahoo!? -> Tired of spam? Yahoo! Mail has the best spam protection around -> http://mail.yahoo.com -> _______________________________________________ -> LARTC mailing list -> LARTC@mailman.ds9a.nl -> http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc From jarod125 at yahoo.com Sun Jul 31 14:42:21 2005 From: jarod125 at yahoo.com (Gabriel) Date: Sun Jul 31 14:42:25 2005 Subject: [LARTC] what is "tcp window size" ? In-Reply-To: Message-ID: <20050731124221.75338.qmail@web60915.mail.yahoo.com> Querying google for "tcp window size" returns some links. I don't know the recommended values and I don't know either if you can play with this value in linux, or if it is wise. There is built-in flow control in the TCP protocol and it adjusts the window size according to your available bandwidth. Obviously, the bigger, the better, but if more data arrives than can be accepted, it will be discarded and retransmission will be necessary. On the other side, if the window is set too small, the same information is transmitted using more segments when maybe it could be transmitted using fewer segments. > Thank you very much... > > What values are recommended for diferent situations ? > > for a Vsat link ? ADSL ? LAN ? ... and so on... > > is important to set this value on linux ? > > If you have a URL to get more information, please post > it . > > thanks again... > > andres. > > > -> > -> > -> The TCP header has a 16 bits field that specifies (and > now > -> I'm quoting from the RFC 793): "The number of data > octets > -> beginning with the one indicated in the acknowledgment > -> field which the sender of this segment is willing to > -> accept." In plain english: how much "useful" data you > can > -> accept from the sender. > -> > -> > what is "tcp window size" ? > -> > > -> > thank you > -> > -> > -> __________________________________________________ > -> Do You Yahoo!? > -> Tired of spam? Yahoo! Mail has the best spam > protection around > -> http://mail.yahoo.com > -> _______________________________________________ > -> LARTC mailing list > -> LARTC@mailman.ds9a.nl > -> http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > ____________________________________________________ Start your day with Yahoo! - make it your home page http://www.yahoo.com/r/hs From psihozefir at yahoo.com Sun Jul 31 17:47:56 2005 From: psihozefir at yahoo.com (panca sorin) Date: Sun Jul 31 17:48:00 2005 Subject: [LARTC] How can I use different flows comming from ingress qdisc? Message-ID: <20050731154757.62741.qmail@web32604.mail.mud.yahoo.com> $tc filter add dev $eth parent ffff: \ protocol ip prio 1 \ u32 match ip sport 1863 0xffff \ police rate 124kbit burst 1k drop \ flowid :1 ____________________________________________________ Start your day with Yahoo! - make it your home page http://www.yahoo.com/r/hs From Ow.Mun.Heng at wdc.com Mon Aug 1 06:14:39 2005 From: Ow.Mun.Heng at wdc.com (Ow Mun Heng) Date: Mon Aug 1 06:14:56 2005 Subject: [LARTC] what is "tcp window size" ? In-Reply-To: <20050731124221.75338.qmail@web60915.mail.yahoo.com> References: <20050731124221.75338.qmail@web60915.mail.yahoo.com> Message-ID: <1122869680.26284.14.camel@neuromancer.home.net> On Sun, 2005-07-31 at 05:42 -0700, Gabriel wrote: > Querying google for "tcp window size" returns some links. I > don't know the recommended values and I don't know either > if you can play with this value in linux, AFAIK, this value isn't something which QoS can handle or dynamically change. This is based on my research I did some 6-8 months ago. Though things may have changed during that time. > or if it is wise. > There is built-in flow control in the TCP protocol and it > adjusts the window size according to your available > bandwidth. Obviously, the bigger, the better, but if more > data arrives than can be accepted, it will be discarded and > retransmission will be necessary. On the other side, if the > window is set too small, the same information is > transmitted using more segments when maybe it could be > transmitted using fewer segments. Yes, one of the reasons for the need for dynamic scaling of the window sizes is to do reduce the retranmissions and thus the dropping of packets because they arrive too fast. In an ideal world, this shouldn't happen and links shouldn't get saturated with unneeded packets and dropping it at the gateway/router etc and hence losing bandwidth. AFAICT, TCP's built in window scaling just moves it up each time it gets a successful transmit, but when it drops, it will have to restart itself say, from zero. That's my uderstanding. May be flawed. > -- Ow Mun Heng Gentoo/Linux on DELL D600 1.4Ghz 1.5GB RAM 98% Microsoft(tm) Free!! Neuromancer 12:14:36 up 8 days, 18:26, 5 users, load average: 0.15, 0.22, 0.24 From Ow.Mun.Heng at wdc.com Mon Aug 1 06:16:29 2005 From: Ow.Mun.Heng at wdc.com (Ow Mun Heng) Date: Mon Aug 1 06:16:45 2005 Subject: [LARTC] Load Balancing and NAT Clients Problem In-Reply-To: <42EA9D84.7050402@hotmail.com> References: <42EA9D84.7050402@hotmail.com> Message-ID: <1122869789.26284.17.camel@neuromancer.home.net> On Fri, 2005-07-29 at 22:20 +0100, ZeuS SN wrote: > Hi, > > The problem is that if I set the default route with two gateways, with > the command 'ip route change default scope global nexthop via $P1 dev > $IF1 weight 1 nexthop via $P1 dev $IF1 weight 1, in the linux router I > have no problem working with internet but the NATTED clients can't > access. I have iptables configured with masquerade option. I don't understand. you habe 2 gateways at the router. What are the NATTED clients pointing at. Sounds to me like the don't have permissions / are not routed correctly. hence they can't go on the net. -- Ow Mun Heng Gentoo/Linux on DELL D600 1.4Ghz 1.5GB RAM 98% Microsoft(tm) Free!! Neuromancer 12:15:11 up 8 days, 18:27, 5 users, load average: 0.32, 0.24, 0.24 From Ow.Mun.Heng at wdc.com Mon Aug 1 06:21:21 2005 From: Ow.Mun.Heng at wdc.com (Ow Mun Heng) Date: Mon Aug 1 06:21:42 2005 Subject: [LARTC] wondershaper query In-Reply-To: <20050728190051.GA32397@tranquility.scriptkitchen.com> References: <20050728190051.GA32397@tranquility.scriptkitchen.com> Message-ID: <1122870081.26284.23.camel@neuromancer.home.net> On Thu, 2005-07-28 at 15:00 -0400, Payal Rathod wrote: > Hello, > I am trying wondershaper-1.1a on a friend's pppoe connection on her > Linux box. > There are a few things I don't understand. > 1. She has pppoe connection so should DEV=eth0 or DEV=ppp0 ? Frankly I can't remember. My home box is not with me right now. Why don't you give each a shot? My bet is it's ppp0. (90% sure) > 2. Her ISP just says on her payment bill that the speed is 128kbps, but > doesn't mention any downlink/uplink speed, so in that case what should > be, > DOWNLINK= and UPLINK= ? You can try some online bandwith tests. I like the ones at nyc.speakeasy.net bear in mind that 128 may mean both up and dn speed (symmetrical) You may have to play with the numbers a bit to get it right. > 3. She uses the net in her small office and people mostly to browse the > net, send emails > sometimes ftp data out and sometimes ssh to other servers to trouble > shoot their > programs. In such a case is wondershaper helpful? Or is it not required > at all? Of course it is useful. This is definitely useful to make non-interactive activity such as FTP slower then interactive (SSH) activities so not to feel the lag. -- Ow Mun Heng Gentoo/Linux on DELL D600 1.4Ghz 1.5GB RAM 98% Microsoft(tm) Free!! Neuromancer 12:16:40 up 8 days, 18:28, 5 users, load average: 0.42, 0.30, 0.26 From lartc at manchotnetworks.net Mon Aug 1 12:12:18 2005 From: lartc at manchotnetworks.net (lartc) Date: Mon Aug 1 12:12:26 2005 Subject: [LARTC] what is "tcp window size" ? In-Reply-To: References: Message-ID: <1122891138.4957.4.camel@drs0.manchotnetworks.net> hi, On Sat, 2005-07-30 at 21:38 -0300, :: L i n u XK i D :: wrote: > Thank you very much... > > What values are recommended for diferent situations ? > > for a Vsat link ? ADSL ? LAN ? ... and so on... For a vsat link, it depends on who's indoor unit your using -- if you're running behind a Gilat/Satlynx 360E or a Eutelsat (built by Viasat) then all layer 4 parameters are stripped off and set by the satellite modem. therefore, you don't need to play with window size because it's ignored. see Thomas Ross Henderson's masters thesis for a more in depth look at satellites. Cheers Charles From johnm at advocap.org Mon Aug 1 15:32:12 2005 From: johnm at advocap.org (John McMonagle) Date: Mon Aug 1 15:32:44 2005 Subject: [LARTC] Routing for multiple uplinks/providers problem. In-Reply-To: <2accc2ff05072809322a58abdb@mail.gmail.com> References: <42E8FE94.2080506@advocap.org> <2accc2ff050728091870b1bbfb@mail.gmail.com> <2accc2ff05072809322a58abdb@mail.gmail.com> Message-ID: <42EE245C.2020109@advocap.org> Nelson Not been having any problems with incoming DNAT I'm using a bit different solution. Noticed that the returning packets wanted to head for the correct interface probably because of conntracking stuff. Problem was they would be routed then the correct interface to whatever one was the default. What I basically added rules that said if it's from an interfaces ip go out that interface. This is the setup for one of the isp interfaces: IP="24.196.120.30" NET="24.196.120.28" LENGTH=30 ROUTER="24.196.120.29" BRD="24.196.120.31" ip link set $IFACE up ip addr flush dev $IFACE ip addr add $IP/$LENGTH brd $BRD dev $IFACE ip rule add prio 201 from $NET/$LENGTH table isp2 ip route add default via $ROUTER dev $IFACE src $IP proto static table isp2 ip route append prohibit default table isp2 metric 1 proto static # call something to fixup default route /etc/network/defroute Doing some simular tricks to get ipsec vpn works outgoing from the firewall. IP="192.168.2.254" NET="192.168.2.0" LENGTH=24 BRD="192.168.2.255" ip link set $IFACE up ip addr flush dev $IFACE ip addr add $IP/$LENGTH brd $BRD dev $IFACE #next is to make sure local 192.168. goes via eth0 ip rule delete prio 200 table 220 ip route del table 200 ip route add 192.168.0.0/16 via $IP dev $IFACE table 200 ip rule add prio 200 table 200 Nelson Castillo wrote: >I think I said something wrong in my last message. >You DNAT incoming packets and then SNAT them when >they come back if your Linux router has some server behind it. >I don't know if this is your case (having servers behind the router). > >(I needed to top-post here --- maybe not). > >On 7/28/05, Nelson Castillo wrote: > > >>Hi John. >> >>On 7/28/05, John McMonagle wrote: >> >> >> >>>Find that if I ping the same site from 2 computers it may work on one >>>and fail on the other. >>>Also was surprised that some time they are going out different >>>interfaces at the same time. >>> >>> >>Same symptoms I had. >> >> >> >>>Have snat on both interfaces >>> >>> >>When you SNAT incoming packets, you need to do something different >>from what is in the HOWTO ([4]) because SNAT is done before the >>routing desition (check the Kernel Packet Traveling Diagram[5]). >> >>I had the same problem [1]. The solution is to use conntrack and mark >>packets on arrival, and then route them back using the fwmark[2]. >> >>There's no need to tell you I had a hard time with this. There should >>be a warning about this in the HOWTO (in this page [4]). >> >> The proposed solution I quote in [2] worked for me for the >> multiple uplink providers + SNAT problem. >> >> It is (Using the same variables that are in the HOWTO [4]): >> >> 1) Mark packages on arrival: >> >> iptables -t mangle -A PREROUTING -m conntrack --ctorigdst $IP1 -j >>MARK --set-mark=1 >> iptables -t mangle -A PREROUTING -m conntrack --ctorigdst $IP2 -j >>MARK --set-mark=2 >> >> And then use the mark to route the outgoing packages correctly. >> >> ip rule add fwmark 1 table T1 >> ip rule add fwmark 2 table T2 >> >>Regards, >>Nelson.- >> >>PD : I solved my problem with IPVS and multiple uplink providers (see [3]). >> >>[1] http://mailman.ds9a.nl/pipermail/lartc/2005q2/016171.html >>[2] http://mailman.ds9a.nl/pipermail/lartc/2005q2/016441.html >>[3] http://arhuaco.blogspot.com/2005/07/ipvs-and-conntrack.html >>[4] http://lartc.org/howto/lartc.rpdb.multiple-links.html >>[5] http://www.docum.org/docum.org/kptd/ >> >>-- >>Homepage : http://geocities.com/arhuaco >> >>The first principle is that you must not fool yourself >>and you are the easiest person to fool. >> -- Richard Feynman. >> >> From mihaivlad at web-profile.net Mon Aug 1 22:18:14 2005 From: mihaivlad at web-profile.net (Mihai Vlad) Date: Mon Aug 1 22:18:22 2005 Subject: [LARTC] sercice-based and ip-based shaping Message-ID: Hello guys, I need to accomplish 2 tasks with one Linux router: 1) Shape the traffic based on services (e.g. HTTP max 512 kbps, P2P max 128 kbps - with a lower prio, etc) 2) After I have classified the traffic based on services, I want to pass it to the clients with different rates/ceils (one client will get 8/128 kbps, other 8/64 kbps, etc). If all the users are downloading via P2P I do not want the whole traffic to exceed the P2P service ceil (the 128 kbps - as stated before). I know how to shape based on protocols and services. I know how to shape a connection based on IP's. But how do I combine the both? How should the Hierarchy look like. Thanks in advance, Mihai VLAD From vinod_chandran at multitech.co.in Tue Aug 2 07:17:12 2005 From: vinod_chandran at multitech.co.in (Vinod Chandran) Date: Tue Aug 2 08:52:04 2005 Subject: [Fwd: Re: [LARTC] sercice-based and ip-based shaping] Message-ID: <42EF01D8.3000007@multitech.co.in> -------------- next part -------------- An embedded message was scrubbed... From: Vinod Chandran Subject: Re: [LARTC] sercice-based and ip-based shaping Date: Tue, 02 Aug 2005 10:36:00 +0530 Size: 3441 Url: http://mailman.ds9a.nl/pipermail/lartc/attachments/20050802/44a5f54e/LARTCsercice-basedandip-basedshaping.eml From alaios at yahoo.com Tue Aug 2 10:45:02 2005 From: alaios at yahoo.com (Alaios) Date: Tue Aug 2 10:45:08 2005 Subject: [LARTC] internet traffic from tbf Message-ID: <20050802084503.90135.qmail@web54703.mail.yahoo.com> Hi have set the following tbf tc qdisc add dev eth0 root tbf rate 0.5mbit \ burst 5kb latency 70ms peakrate 1mbit \ minburst 1540 I want to add a filter so the ip traffic pass from it.. plz help me __________________________________ Yahoo! Mail Stay connected, organized, and protected. Take the tour: http://tour.mail.yahoo.com/mailtour.html From ddaasd at gmail.com Tue Aug 2 15:32:36 2005 From: ddaasd at gmail.com (ddaasd) Date: Tue Aug 2 15:32:48 2005 Subject: [LARTC] iptables -m layer7 - doesn't work Message-ID: <42EF75F4.5000109@gmail.com> I've compiled my kernel (2.6.12.3) and iptables (1.3.3) and now -m layer7 option from iptables works (i don't get any error when run the iptables command with -m layer7). The problem is that no packet is matched. For example iptables -A INPUT -p tcp -m layer7 --l7proto http -j ACCEPT doesn't match http packets. The same for dns and ssh. So, what am I doing wrong? Is this extension for iptables really working?? I?ve tried on 2 systems (fc3 and slack 10.1) Here is my iptables test script: #!/bin/bash iptables -F iptables -A INPUT -i lo -j ACCEPT iptables -A OUTPUT -o lo -j ACCEPT iptables -A INPUT -i eth0 -p tcp --dport 22 -j ACCEPT iptables -A OUTPUT -o eth0 -p tcp --sport 22 -j ACCEPT iptables -A INPUT -i eth0 -p tcp -m layer7 --l7dir /home/dda/l7dir --l7proto http -j ACCEPT iptables -A OUTPUT -o eth0 -p tcp -m layer7 --l7dir /home/dda/l7dir --l7proto http -j ACCEPT iptables -A INPUT -i eth0 -p udp -m layer7 --l7dir /home/dda/l7dir --l7proto dns -j ACCEPT iptables -A OUTPUT -o eth0 -p udp -m layer7 --l7dir /home/dda/l7dir --l7proto dns -j ACCEPT iptables -P INPUT DROP iptables -P OUTPUT DROP and now: iptables -vnL after generating some http and dns traffic Chain INPUT (policy DROP 56 packets, 8892 bytes) pkts bytes target prot opt in out source destination 3340 134K ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0 129 9208 ACCEPT tcp -- eth0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:22 0 0 ACCEPT tcp -- eth0 * 0.0.0.0/0 0.0.0.0/0 LAYER7 l7proto http 0 0 ACCEPT udp -- eth0 * 0.0.0.0/0 0.0.0.0/0 LAYER7 l7proto dns Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination Chain OUTPUT (policy DROP 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 3340 134K ACCEPT all -- * lo 0.0.0.0/0 0.0.0.0/0 90 12254 ACCEPT tcp -- * eth0 0.0.0.0/0 0.0.0.0/0 tcp spt:22 0 0 ACCEPT tcp -- * eth0 0.0.0.0/0 0.0.0.0/0 LAYER7 l7proto http 0 0 ACCEPT udp -- * eth0 0.0.0.0/0 0.0.0.0/0 LAYER7 l7proto dns From mihaivlad at web-profile.net Tue Aug 2 18:02:57 2005 From: mihaivlad at web-profile.net (Mihai Vlad) Date: Tue Aug 2 18:03:06 2005 Subject: [LARTC] service-based and ip-based shaping Message-ID: Thanks, The only issue here is that for each service I need to create 200 child classes if I have 200 clients... Let me explain the problem better I have the following connection from my ISP: (1024/1024) (rate/ceil) 1) First, I want to divide the 1024 into smaller pieces based on priority: 256/256 - P2P (I want to limit the P2P traffic as much as possible) 256/1024 - HTTP 256/1024 - FTP 256/1024 - SSH, Games, etc (I do not say that this is the best approach but it does not matter - it is just an example) 2) Second task: Suppose I have previously shaped the traffic the way I want, I need 200 classes for 200 clients in which to tell the rate and ceil for each of them 16/512 - Client 1 16/256 - Client 2, etc If I would have 200 clients and 10 types of services and childs to service classes, this means 2000 client classes... I was thinking of a funny setup: Route the traffic to 2 IMQ's -> First: traffic passes imq0 - and is shaped based on services -> Second: traffic from imq0 is routed to imq1 - where it is shaped based on IP of the client. What do you think about this? Mihai VLAD __________________________ -----Original Message----- From: Vinod Chandran [mailto:vinod_chandran@multitech.co.in] Sent: Tuesday, August 02, 2005 8:06 AM To: Mihai Vlad Subject: Re: [LARTC] sercice-based and ip-based shaping Hi Mihai, You could have a parent class based on services, then have children of that parent class based on ips. for eg, lets say the root is 1:1, have FTP service class as 1:2 , parent 1:1 and then have a child say 1:21, 1:22 and so on whose parent is 1:2. I have used mark values to classify traffic. As far as rate/ceil values are concerned.. have class 1:2 rate/ceil 128/512. 1:21 - 8/128 1:22 - 8/64 .. and so on... 1:1(512) | | ---------------------------------------------------------------------------- ---------- | | 1:2(FTP)(128/512) 1:3( Telnet)(384/512) | | -------------------------------------- -------------------------- | | | | 1:21 (8/128) 1:22 (8/64) 1:31(128/384) 1:32(128/256) Hope this helps you, Regards, Vinod C Mihai Vlad wrote: >Hello guys, > >I need to accomplish 2 tasks with one Linux router: > >1) Shape the traffic based on services (e.g. HTTP max 512 kbps, P2P max 128 >kbps - with a lower prio, etc) > >2) After I have classified the traffic based on services, I want to pass it >to the clients with different rates/ceils (one client will get 8/128 kbps, >other 8/64 kbps, etc). If all the users are downloading via P2P I do not >want the whole traffic to exceed the P2P service ceil (the 128 kbps - as >stated before). > > >I know how to shape based on protocols and services. I know how to shape a >connection based on IP's. But how do I combine the both? How should the >Hierarchy look like. > > >Thanks in advance, > >Mihai VLAD > > > > >_______________________________________________ >LARTC mailing list >LARTC@mailman.ds9a.nl >http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > > > From psihozefir at yahoo.com Tue Aug 2 19:38:37 2005 From: psihozefir at yahoo.com (panca sorin) Date: Tue Aug 2 19:38:41 2005 Subject: [LARTC] How to set a host with public IP within a private network? Message-ID: <20050802173838.72293.qmail@web32611.mail.mud.yahoo.com> Hello! I have the following setup: 1) a connection to my ISP with a public IP (1.2.3.4) with the gateway 1.2.3.1 2) an allocated IP class with 64 addresses (5.6.7.192/26) 3) two LANs connected through two NICs: a) 192.168.0.0/24 on eth1 (192.168.0.1) b) 10.0.0.0/24 on eth2 (10.0.0.1) The IPs from the allocated class are all assigned to eth0. The networks are SNATed to the external IP and to all IPs in the allocated class in a round-robin fashion. (-j SNAT --to 1.2.3.4 lowest_IP_in_class highest_IP_in_class) My question is: Is it possible to assign one IP from my allocated class to an internal machine without changing eth1 or eth2 IPs *OR* without adding a subclass of my allocated class to eth1 or eth2 in order to give an ip to an internal networked machine? What command should i give if that setup is possible? Desired Network Diagram: ---------- /---|Internet| | ---------- |eth0 192.168.0.1 10.0.0.1 | ---------- eth1 eth2 \--| Linux |--------------v--------------------v | Router | | | ---------- | | | | ------------- | ---------- | |192.168.0.2|--------< |10.0.0.2|----< ------------- | ---------- | . | . | . | . | . | . | --------------- | ------------ | |192.168.0.254|-------< |10.0.0.254|--< --------------- | ------------ | | ----------- | |5.6.7.201|-----------< ----------- I would de-assign some of the addresses from eth0 to re-assign them to locally connected computers... Thank you in advance for your help! ____________________________________________________ Start your day with Yahoo! - make it your home page http://www.yahoo.com/r/hs From jlynch at frink.nuigalway.ie Tue Aug 2 22:59:41 2005 From: jlynch at frink.nuigalway.ie (Jonathan Lynch) Date: Tue Aug 2 23:01:26 2005 Subject: [LARTC] HTB and PRIO qdiscs introducing extra latency when output interface is saturated In-Reply-To: <42E95302.2020303@dsl.pipex.com> References: <1122460110.8454.11.camel@pgala.it.nuigalway.ie> <42E78B39.1030201@dsl.pipex.com> <1122478662.4637.49.camel@pgala.it.nuigalway.ie> <42E80247.7010100@dsl.pipex.com> <1122568638.7828.40.camel@pgala.it.nuigalway.ie> <42E95302.2020303@dsl.pipex.com> Message-ID: <1123016381.4768.35.camel@pgala.it.nuigalway.ie> I did the same tests that I outlined earlier, but this time by setting hysteresis to 0. The config for the core router is included at the bottom. The graphs for the delay of the voip stream and the traffic going through the core router can be found at the following addresses. http://140.203.56.30/~jlynch/htb/core_router_hysteresis.png http://140.203.56.30/~jlynch/htb/voip_stream_24761_hysteresis.png The max delay of the stream has dropped to 1.8ms. Again the jitter seems to be around 1ms. There seems to be a pattern going whereby the delay reaches about 1.6ms then drops back to 0.4 ms, jumps back to 1.6ms and then back to 0.4ms repeatedly and then it rises from 0.5ms gradually and repeats this behaviour. Is there any explanation to this pattern ? Would it have anything go to do with burst being 1ms ? When the ceil is specified as being 90mbit, is this at IP level ? What does this correspond to when a Mbit = 1,000,000 bits. Im a bit confused with the way tc interprets this rate. If the ceil is based at IP level then the max ceil is going to be a value between 54 Mbit and 97 Mbit (not the tc values) for a 100 Mbit interface depending on the size of the packets passing through, right ? Minimum Ethernet frame 148,809 * (46 * 8) = 148,809 * 368 = 54,761,712 Mbps Maximum Ethernet frame 8,127 * (1500 * 8) = 8,127 * 12,000 = 97,524,000 Mbps About the red settings, I dont understand properly how to configure the settings. I was using the configuration that came with the examples. Jonathan On Thu, 2005-07-28 at 22:49 +0100, Andy Furniss wrote: > Jonathan Lynch wrote: > > Andy, thanks again for your help. Yes, HZ is still 1000 in 2.6.12. I > > tried your suggestions are here are the results. > > > > ASCII diagram > > > > (network A) --> (eth1) core router (eth0) --> (network C) > > (eth2) > > ^ > > | > > | > > (network B) > > > > Looking at the following graphics > > > > http://140.203.56.30/~jlynch/htb/core_router.png > > http://140.203.56.30/~jlynch/htb/voip_stream_23691.png > > > > voip_stream_23691.png is a graph of the delay of the voice stream > > travelling from network A to network C in test 2. Notice from the core > > router graph that there is only voip traffic passing through the core > > router until time 07:55 and the delay in voip stream is 0.25 ms until > > then. After this time tcp traffic is introduced saturating the outgoing > > interface of the core router (eth0). The delay increases to a maximum of > > 2.75 ms , which is a considerable improvement on 30ms when I was using > > the ceil value of 100mbit. But there is a lot of jitter. > > I suppose you could hope for a bit less jitter 12k burst is about 1ms at > 100mbit. > > There is a tweak you can do for htb which may help - in > net/sched/sch_htb.c there is a #define HYSTERESIS 1 - changing it to 0 > and recompiling kernel/the module makes things more accurate. > > > > > > > With the ceil at 90Mbit, the outgoing bit rate of eth0 has gone from > > 98mbit to approx 90Mbit as can be seen from the core router graph for > > eth0 bytes out. Note that with the tcp traffic is all http downloads, so > > most Ethernet frames will be of maximum size, 1518 bytes, so 98mbits is > > the maximum throughput possible on a 100mbit card, taking into account > > the overheads of ethernet such as the interframe gap, preamble and start > > frame delimiter. > > > > Im not sure how to configure some of the htb parameters. The following > > is my understanding of them and a few questions I have as well. > > > > How exactly does the HZ value have a bearing on the ceil value ? How can > > I calculate a maximum for the ceil value ? > > It's more to do with burst/cburst than ceil. > > > > > 12kb is the minimum burst size for a 100 mbit NIC with a timer > > resolution of 1ms (1000hz) and tc calculates the smallest possible burst > > when it is not specified, right ?. > > It seems not, I think hysteresis may be involved again here (but then > one of my tcs is hacked about a bit). > > You can see what htb is using as defaults by doing tc -s -d class ls .. > > If I do that on similar kernels one with hysteresis 0 and one with 1 I > see quite different values. > > I chose 12k as big enough for the 90mbit test 12000*8*1000=96mbit at ip > level and it seemed like a nice multiple of 1500mtu :-) > > > > > > cburst is the number of bytes that can be burst as fast as the interface > > can transmit them. It is smaller than burst can is ideally one packet > > size, right ? > > Ideally 1 packet but not achievable with htb at lan speed and hz 1000, > also AIUI the way htb does drr means with mixed packet sizes things > aren't packet perfect even at low rates. > > Saying that I use htb at low rates and can apparently get packet perfect > with my traffic mix. > > I think hfsc can do it perfectly on both counts. > > > > > quantum determines the ratio at which the classes share their parents > > bandwidth. Each class is given quantum number of bytes before serving > > the next class, right ? > > Yea setting 1500 probably makes no difference for this test. > > > > > Is there any way I can limit the jitter of the VoIP traffic passing > > through the htb class ? > > Try the hysteresis and/or setting the rate for interactive way higher > than it's traffic rate. > > I did a quick test to see how things were for me at 100mbit. Because my > other pcs are slow I needed to use two as receivers for netperf. > > I noticed something I didn't expect with red or the settings you use - > one of the pcs is slower and has less memory thus smaller tcp socket > size. Using 4 streams two to each unshaped they get about the same, > though with txqueuelen = 1000 there are no drops (with window scalng off > there is a difference). With red and wscale on, the red really favoured > the fast pc - I tried a 40k bfifo so that I got drops, expecting to see > the same, but it was still far more even than the red. > > I couldn't really simulate the voip traffic in theory I should be able > to use ping with -i < 1 sec, but using the latest inetutils you get a > flood ping if you do that. I reported this about 18 months ago and it's > supposedly fixed in the cvs (though I don't know if fixed means it just > says invalid argument rather than actually does what's asked, because I > have failed to build it so far). > > So if anyone reading this has a i386 ping that -i 0.5 works on, please > mail me the binary :-) > > Andy. Main dsmark & classifier tc qdisc add dev $1 handle 1:0 root dsmark indices 64 set_tc_index tc filter add dev $1 parent 1:0 protocol ip prio 1 tcindex mask 0xfc shift 2 Main htb qdisc & class tc qdisc add dev $1 parent 1:0 handle 2:0 htb tc class add dev $1 parent 2:0 classid 2:1 htb rate 90Mbit ceil 90Mbit burst 12k cburst 12k EF Class (2:10) tc class add dev $1 parent 2:1 classid 2:10 htb rate 5Mbit ceil 90Mbit burst 12k cburst 12k prio 1 tc qdisc add dev $1 parent 2:10 pfifo limit 5 tc filter add dev $1 parent 2:0 protocol ip prio 1 handle 0x2e tcindex classid 2:10 pass_on BE Class (2:20) tc class add dev $1 parent 2:1 classid 2:20 htb rate 10Mbit ceil 90Mbit burst 12k cburst 12k prio 2 tc qdisc add dev $1 parent 2:20 red limit 60KB min 15KB max 45KB burst 20 avpkt 1000 bandwidth 100Mbit probability 0.4 tc filter add dev $1 parent 2:0 protocol ip prio 2 handle 0 tcindex mask 0 classid 2:20 pass_on From gtaylor at riverviewtech.net Wed Aug 3 01:58:22 2005 From: gtaylor at riverviewtech.net (Grant Taylor) Date: Wed Aug 3 02:00:06 2005 Subject: [LARTC] How to set a host with public IP within a private network? In-Reply-To: <20050802173838.72293.qmail@web32611.mail.mud.yahoo.com> References: <20050802173838.72293.qmail@web32611.mail.mud.yahoo.com> Message-ID: <42F0089E.5010003@riverviewtech.net> I think you want to take a look at a bridging router. I would be tempted to set up a bridge between eth0 and eth1 on the Linux router and then use ebtables to decide if the traffic should be bridged through or sent up to the higher layers of the protocol stacks. I have a feeling that you would have to move all IP addresses from eth0 to br0 (the new bridge interface). I think you would have a command like this to decide if an IP needed to be routed or bridged. ebtables -t broute -A BROUTING -d 5.6.7.201-j ACCEPT ebtables -t broute -A BROUTING -s 5.6.7.201 -j ACCEPT ebtables -t broute -A BROUTING -j DROP An excerpt from the EBTables man page: *broute*, is used to make a brouter, it has one built-in chain: *BROUTING*. The targets *DROP* and *ACCEPT* have special meaning in the broute table. *DROP* actually means the frame has to be routed, while *ACCEPT* means the frame has to be bridged. The *BROUTING* chain is traversed very early. It is only traversed by frames entering on a bridge enslaved NIC that is in forwarding state. Normally those frames would be bridged, but you can decide otherwise here. The *redirect* target is very handy here. Give this a whirl and see if it will do any thing for you. If you need more specifics just ask. Grant. . . . panca sorin wrote: >Hello! I have the following setup: >1) a connection to my ISP with a public IP (1.2.3.4) >with the gateway 1.2.3.1 >2) an allocated IP class with 64 addresses >(5.6.7.192/26) >3) two LANs connected through two NICs: > a) 192.168.0.0/24 on eth1 (192.168.0.1) > b) 10.0.0.0/24 on eth2 (10.0.0.1) > >The IPs from the allocated class are all assigned to >eth0. >The networks are SNATed to the external IP and to all >IPs in the allocated class in a round-robin fashion. > (-j SNAT --to 1.2.3.4 lowest_IP_in_class >highest_IP_in_class) > >My question is: >Is it possible to assign one IP from my allocated >class to an internal machine without changing eth1 or >eth2 IPs *OR* without adding a subclass of my >allocated class to eth1 or eth2 in order to give an ip >to an internal networked machine? >What command should i give if that setup is possible? >Desired Network Diagram: > > ---------- >/---|Internet| >| ---------- >|eth0 192.168.0.1 10.0.0.1 >| ---------- eth1 eth2 >\--| Linux |--------------v--------------------v > | Router | | | > ---------- | | > | | > ------------- | ---------- | > |192.168.0.2|--------< |10.0.0.2|----< > ------------- | ---------- | > . | . | > . | . | > . | . | > --------------- | ------------ | > |192.168.0.254|-------< |10.0.0.254|--< > --------------- | ------------ > | > | > ----------- | > |5.6.7.201|-----------< > ----------- > >I would de-assign some of the addresses from eth0 to >re-assign them to locally connected computers... > >Thank you in advance for your help! > > > > >____________________________________________________ >Start your day with Yahoo! - make it your home page >http://www.yahoo.com/r/hs > >_______________________________________________ >LARTC mailing list >LARTC@mailman.ds9a.nl >http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > > From xerces8 at butn.net Wed Aug 3 12:09:09 2005 From: xerces8 at butn.net (xerces8) Date: Wed Aug 3 12:09:27 2005 Subject: [LARTC] How to set a host with public IP within a private network? In-Reply-To: <20050802173838.72293.qmail@web32611.mail.mud.yahoo.com> References: <20050802173838.72293.qmail@web32611.mail.mud.yahoo.com> Message-ID: I would try this : - assign the IP to PC (and 192.168.0.1 as gateway) On the router: - set up NAT to not touch packets sent to/from that IP - set up a route to that IP thru eth1 (do not complicate :-) -----Original Message----- From: panca sorin To: lartc@mailman.ds9a.nl Date: Tue, 2 Aug 2005 10:38:37 -0700 (PDT) Subject: [LARTC] How to set a host with public IP within a private network? > Hello! I have the following setup: > 1) a connection to my ISP with a public IP (1.2.3.4) > with the gateway 1.2.3.1 > 2) an allocated IP class with 64 addresses > (5.6.7.192/26) > 3) two LANs connected through two NICs: > a) 192.168.0.0/24 on eth1 (192.168.0.1) > b) 10.0.0.0/24 on eth2 (10.0.0.1) > > The IPs from the allocated class are all assigned to > eth0. > The networks are SNATed to the external IP and to all > IPs in the allocated class in a round-robin fashion. > (-j SNAT --to 1.2.3.4 lowest_IP_in_class > highest_IP_in_class) > > My question is: > Is it possible to assign one IP from my allocated > class to an internal machine without changing eth1 or > eth2 IPs *OR* without adding a subclass of my > allocated class to eth1 or eth2 in order to give an ip > to an internal networked machine? > What command should i give if that setup is possible? > Desired Network Diagram: > > ---------- > /---|Internet| > | ---------- > |eth0 192.168.0.1 10.0.0.1 > | ---------- eth1 eth2 > \--| Linux |--------------v--------------------v > | Router | | | > ---------- | | > | | > ------------- | ---------- | > |192.168.0.2|--------< |10.0.0.2|----< > ------------- | ---------- | > . | . | > . | . | > . | . | > --------------- | ------------ | > |192.168.0.254|-------< |10.0.0.254|--< > --------------- | ------------ > | > | > ----------- | > |5.6.7.201|-----------< > ----------- > > I would de-assign some of the addresses from eth0 to > re-assign them to locally connected computers... > > Thank you in advance for your help! > > > > > ____________________________________________________ > Start your day with Yahoo! - make it your home page > http://www.yahoo.com/r/hs > > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc From psihozefir at yahoo.com Wed Aug 3 12:21:05 2005 From: psihozefir at yahoo.com (panca sorin) Date: Wed Aug 3 12:21:10 2005 Subject: [LARTC] How to set a host with public IP within a private network? In-Reply-To: Message-ID: <20050803102105.28044.qmail@web32609.mail.mud.yahoo.com> I did this and apparently, it works: 1) I set up proxy arp for the internal and the external NICs: echo 1 > /proc/sys/net/ipv4/conf/$EXT1/proxy_arp echo 1 > /proc/sys/net/ipv4/conf/$INT1/proxy_arp EXT1=eth0;INT1=eth1 then: ip route add $IP dev $INT1 where IP=81.196.157.254 and from the Internet: ping 81.196.157.254 i get replys. i set on 81.196.157.193 (WinXP) this: IP addres: 81.196.157.254 netmask 255.255.255.192 gw 192.168.101.1 (router's internal ip) __________________________________ Yahoo! Mail for Mobile Take Yahoo! Mail with you! Check email on your mobile phone. http://mobile.yahoo.com/learn/mail From psihozefir at yahoo.com Wed Aug 3 12:23:55 2005 From: psihozefir at yahoo.com (panca sorin) Date: Wed Aug 3 12:23:59 2005 Subject: [LARTC] How to set a host with public IP within a private network? In-Reply-To: Message-ID: <20050803102355.80517.qmail@web32605.mail.mud.yahoo.com> i forgot something: on router: route add -host 81.196.157.254 dev eth1 __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From andy.furniss at dsl.pipex.com Wed Aug 3 16:04:08 2005 From: andy.furniss at dsl.pipex.com (Andy Furniss) Date: Wed Aug 3 16:03:45 2005 Subject: [LARTC] HTB and PRIO qdiscs introducing extra latency when output interface is saturated In-Reply-To: <1123016381.4768.35.camel@pgala.it.nuigalway.ie> References: <1122460110.8454.11.camel@pgala.it.nuigalway.ie> <42E78B39.1030201@dsl.pipex.com> <1122478662.4637.49.camel@pgala.it.nuigalway.ie> <42E80247.7010100@dsl.pipex.com> <1122568638.7828.40.camel@pgala.it.nuigalway.ie> <42E95302.2020303@dsl.pipex.com> <1123016381.4768.35.camel@pgala.it.nuigalway.ie> Message-ID: <42F0CED8.1090406@dsl.pipex.com> Jonathan Lynch wrote: > I did the same tests that I outlined earlier, but this time by setting > hysteresis to 0. The config for the core router is included at the > bottom. The graphs for the delay of the voip stream and the traffic > going through the core router can be found at the following addresses. > > http://140.203.56.30/~jlynch/htb/core_router_hysteresis.png > http://140.203.56.30/~jlynch/htb/voip_stream_24761_hysteresis.png > > > The max delay of the stream has dropped to 1.8ms. Again the jitter seems > to be around 1ms. There seems to be a pattern going whereby the delay > reaches about 1.6ms then drops back to 0.4 ms, jumps back to 1.6ms and > then back to 0.4ms repeatedly and then it rises from 0.5ms gradually and > repeats this behaviour. Is there any explanation to this pattern ? > > Would it have anything go to do with burst being 1ms ? Yes I suppose if you could sample truly randomly you would get a proper distribution - I guess the pattern arises because your timers are synchronised for the test. > > When the ceil is specified as being 90mbit, is this at IP level ? > What does this correspond to when a Mbit = 1,000,000 bits. Im a bit > confused with the way tc interprets this rate. Yes htb uses ip level length (but you can specify overhead & min size) , the rate calculations use a lookup table which is likely to have a granularity of 8 bytes (you can see this with tc -s -d class ls .. look for /8 after the burst/cburst). There is a choice in 2.6 configs about using CPU/jiffies/gettimeofday - I use CPU and now I've got a ping that does < 1 sec I get the same results as you. > > If the ceil is based at IP level then the max ceil is going to be a > value between 54 Mbit and 97 Mbit (not the tc values) for a 100 Mbit > interface depending on the size of the packets passing through, right ? > > Minimum Ethernet frame > 148,809 * (46 * 8) = 148,809 * 368 = 54,761,712 Mbps > > Maximum Ethernet frame > 8,127 * (1500 * 8) = 8,127 * 12,000 = 97,524,000 Mbps If you use the overhead option I think you will be to overcome this limitation and push the rates closer to 100mbit. > About the red settings, I dont understand properly how to configure the > settings. I was using the configuration that came with the examples. I don't use red it was just something I noticed - maybe making it longer would help, maybe my test wasn't rerpresentative. FWIW I had a play around with HFSC (not that I know what I am doing really) and at 92mbit managed to get - rtt min/avg/max/mdev = 0.330/0.414/0.493/0.051 ms loaded from rtt min/avg/max/mdev = 0.114/0.133/0.187/0.028 ms idle and that was through a really cheap switch. Andy. From andy.furniss at dsl.pipex.com Wed Aug 3 16:12:13 2005 From: andy.furniss at dsl.pipex.com (Andy Furniss) Date: Wed Aug 3 16:11:54 2005 Subject: [LARTC] service-based and ip-based shaping In-Reply-To: References: Message-ID: <42F0D0BD.5000702@dsl.pipex.com> Mihai Vlad wrote: > Thanks, > > The only issue here is that for each service I need to create 200 child > classes if I have 200 clients... > > > Let me explain the problem better > > I have the following connection from my ISP: (1024/1024) (rate/ceil) > > 1) First, I want to divide the 1024 into smaller pieces based on priority: > 256/256 - P2P (I want to limit the P2P traffic as much as possible) > 256/1024 - HTTP > 256/1024 - FTP > 256/1024 - SSH, Games, etc > (I do not say that this is the best approach but it does not matter - it is > just an example) > > 2) Second task: Suppose I have previously shaped the traffic the way I want, > I need 200 classes for 200 clients in which to tell the rate and ceil for > each of them > 16/512 - Client 1 > 16/256 - Client 2, etc > > If I would have 200 clients and 10 types of services and childs to service > classes, this means 2000 client classes... > > I was thinking of a funny setup: > > Route the traffic to 2 IMQ's > -> First: traffic passes imq0 - and is shaped based on services > -> Second: traffic from imq0 is routed to imq1 - where it is shaped > based on IP of the client. > > What do you think about this? I think you will need to use imq but I would just use 1 and do the second level shaping on the real interface(s). Andy. From andy.furniss at dsl.pipex.com Wed Aug 3 16:22:05 2005 From: andy.furniss at dsl.pipex.com (Andy Furniss) Date: Wed Aug 3 16:21:44 2005 Subject: [LARTC] internet traffic from tbf In-Reply-To: <20050802084503.90135.qmail@web54703.mail.yahoo.com> References: <20050802084503.90135.qmail@web54703.mail.yahoo.com> Message-ID: <42F0D30D.6030107@dsl.pipex.com> Alaios wrote: > Hi have set the following tbf > tc qdisc add dev eth0 root tbf rate 0.5mbit \ > burst 5kb latency 70ms peakrate 1mbit \ > minburst 1540 > > I want to add a filter so the ip traffic pass from > it.. plz help me That will shape all traffic leaving eth0 as it is. If you mean that you don't want to shape arp or other traffic aswell as ip then you will need to add PRIO as root so you can use filters and add tbf to one of the PRIO classes. TBF tends to tweak the parameters you pass to it so it's best to check what you got after adding it and modify your settings. eg with the above I see - tc -s -d qdisc ls dev eth0 qdisc tbf 8007: rate 500000bit burst 5Kb/8 mpu 0b peakrate 1000Kbit mtu 1539b/8 mpu 0b lat 39.2ms ... note the latency. From wstearns at pobox.com Wed Aug 3 17:15:22 2005 From: wstearns at pobox.com (wstearns@pobox.com) Date: Wed Aug 3 17:15:28 2005 Subject: [LARTC] Mail Delivery (failure lartc@mailman.ds9a.nl) Message-ID: <20050803151516.BD3994440@outpost.ds9a.nl> Skipped content of type multipart/alternative-------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: audio/x-wav Size: 29568 bytes Desc: not available Url : http://mailman.ds9a.nl/pipermail/lartc/attachments/20050803/c5ac5709/attachment-0001.wav From andy.furniss at dsl.pipex.com Wed Aug 3 21:32:42 2005 From: andy.furniss at dsl.pipex.com (Andy Furniss) Date: Wed Aug 3 21:32:15 2005 Subject: [LARTC] HTB and PRIO qdiscs introducing extra latency when output interface is saturated In-Reply-To: <42F0CED8.1090406@dsl.pipex.com> References: <1122460110.8454.11.camel@pgala.it.nuigalway.ie> <42E78B39.1030201@dsl.pipex.com> <1122478662.4637.49.camel@pgala.it.nuigalway.ie> <42E80247.7010100@dsl.pipex.com> <1122568638.7828.40.camel@pgala.it.nuigalway.ie> <42E95302.2020303@dsl.pipex.com> <1123016381.4768.35.camel@pgala.it.nuigalway.ie> <42F0CED8.1090406@dsl.pipex.com> Message-ID: <42F11BDA.8070501@dsl.pipex.com> Andy Furniss wrote: > Jonathan Lynch wrote: >> If the ceil is based at IP level then the max ceil is going to be a >> value between 54 Mbit and 97 Mbit (not the tc values) for a 100 Mbit >> interface depending on the size of the packets passing through, right ? >> >> Minimum Ethernet frame >> 148,809 * (46 * 8) = 148,809 * 368 = 54,761,712 Mbps >> >> Maximum Ethernet frame >> 8,127 * (1500 * 8) = 8,127 * 12,000 = 97,524,000 Mbps > > > If you use the overhead option I think you will be to overcome this > limitation and push the rates closer to 100mbit. I looked up ethernet overheads and found the figure of 38 bytes per frame, the 46 is min eth payload size? and looking at the way mpu is handled by the tc rate table generator I think you would need to use 46 + 38 as mpu. So on every htb line that has a rate put ..... overhead 38 mpu 84 I haven't checked those figures or tested close to limits though, the 12k burst would need increasing a bit aswell or that will slightly over limit rate at HZ=1000. Andy. From andy.furniss at dsl.pipex.com Thu Aug 4 20:06:32 2005 From: andy.furniss at dsl.pipex.com (Andy Furniss) Date: Thu Aug 4 20:06:31 2005 Subject: [LARTC] HTB and PRIO qdiscs introducing extra latency when output interface is saturated In-Reply-To: <42F11BDA.8070501@dsl.pipex.com> References: <1122460110.8454.11.camel@pgala.it.nuigalway.ie> <42E78B39.1030201@dsl.pipex.com> <1122478662.4637.49.camel@pgala.it.nuigalway.ie> <42E80247.7010100@dsl.pipex.com> <1122568638.7828.40.camel@pgala.it.nuigalway.ie> <42E95302.2020303@dsl.pipex.com> <1123016381.4768.35.camel@pgala.it.nuigalway.ie> <42F0CED8.1090406@dsl.pipex.com> <42F11BDA.8070501@dsl.pipex.com> Message-ID: <42F25928.6010205@dsl.pipex.com> Andy Furniss wrote: > I haven't checked those figures or tested close to limits though, the > 12k burst would need increasing a bit aswell or that will slightly over > limit rate at HZ=1000. It seems that htb still uses ip level for burst so 12k is enough. With the overhead at 38 I can ceil at 99mbit OK. Andy. From wonka at linkabu.net Fri Aug 5 22:00:21 2005 From: wonka at linkabu.net (Eduardo Bejar) Date: Fri Aug 5 22:02:48 2005 Subject: [LARTC] Real time traffic monitor like iptraf Message-ID: <200508051959.j75JxXv00851@mail.linkabu.net> Hi, Just wondering, how you guys monitor traffic per ip to test if the traffic shaping is working? iptraf works great but no new version is available, and most ip accounting projects (MRTG & Friends) update its graphics on a per minute process. So I?d like to know which project you use to monitor bandwidth use in real time. Regards, Edo From dan at wgst.se Sun Aug 7 20:50:59 2005 From: dan at wgst.se (Dan =?iso-8859-1?Q?Gr=F6nberg?=) Date: Sun Aug 7 20:47:54 2005 Subject: [LARTC] Netcard functions with prio Message-ID: Hi there I have followed your disc. concerning bandwith and prio of port-traffic. What happens to for example with "ping" time, and other ports very much used for diff. pleasures, if for example you prio http and smtp? Is there any technics by the way, that netcards use when low prio-traffic gets congested, I mean technics like "back-pressure" (switch-technics I suppose)? Can the prio-function control this in the netcards? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ds9a.nl/pipermail/lartc/attachments/20050807/f9b0ee46/attachment.htm From hareram at sol.net.in Mon Aug 8 07:31:05 2005 From: hareram at sol.net.in (hareram) Date: Mon Aug 8 07:31:10 2005 Subject: [LARTC] Loadbalancing and failover using TC and Iptables Message-ID: <018701c59bda$5f42db40$0903a8c0@southern> Hi all iam trying to deploy loadbalance and failover My setup description --Fedora Core 4 --Linux 2.6.12.3 #1 SMP Mon Jul 25 22:37:34 IST 2005 i686 i686 i386 GNU/Linux --tc utility, iproute2-ss050314 --ip utility, iproute2-ss050314 --iptables v1.3.0 And i had deployed Following configuration #table main with priority 50, the highest one ip rule add prio 50 table main #table 201 ip rule add prio 201 from x.x.x.234 table 201 ip route add default via x.x.x..233 dev eth1 src x.x.x.234 proto static table 201 ip route append prohibit default table 201 metric 1 proto static #table 202 ip rule add prio 202 from y.y.y.10 table 202 ip route add default via y.y.y.9 dev eth0 src y.y.y.10 proto static table 202 ip route append prohibit default table 202 metric 1 proto static #table 222 ip rule add prio 222 table 222 ip route add default equalize table 222 proto static nexthop via x.x.x.233 dev eth1 nexthop via y.y.y.9 dev eth0 #essential masquerade option iptables -t nat -A POSTROUTING -s 192.168.3.0/24 -j MASQUERADE Above is my setup when try to traceroute to yahoo.com iam able to see the trafffic going to both interfaces.. till now works fine when i connected to eth2 ( eth2 of linux box configured IP 192.168.3.2) with my Laptop using ip 192.168.3.1 gateway 192.168.3.2( linux box eth2) when try to traceroute its always going to y.y.y.9 when i go and check whatismyip.com and findmyip.com its shows only y.y.y.10 IP, why my traffic is not balancing using both the routes ?? when i change the my rule like following ip route replace default equalize table 222 proto static nexthop via x.x.x.233 dev eth1 when try to traceroute its always going to y.y.y.233 when i go and check whatismyip.com and findmyip.com its shows only y.y.y.234 IP, could some one help me to resolve this issue and suggest me what is need to be done if i want nat and other IP's to be loadbalance may be i call it per packet loadbalance thanks in advance hare From gypsy at iswest.com Mon Aug 8 15:46:15 2005 From: gypsy at iswest.com (gypsy) Date: Mon Aug 8 15:46:22 2005 Subject: [LARTC] Loadbalancing and failover using TC and Iptables References: <018701c59bda$5f42db40$0903a8c0@southern> Message-ID: <42F76227.17A6B6C8@iswest.com> hareram wrote: > > Hi all > > iam trying to deploy loadbalance and failover > > My setup description > --Fedora Core 4 > --Linux 2.6.12.3 #1 SMP Mon Jul 25 22:37:34 IST 2005 i686 i686 i386 > GNU/Linux > --tc utility, iproute2-ss050314 > --ip utility, iproute2-ss050314 > --iptables v1.3.0 You say nothing about Julian's patch, so I assume you did not patch your kernel. You must do that. http://www.ssi.bg/~ja/ http://www.geocities.com/mctiew/ffw/dual.htm I'm not sure this is still a good link http://selab.edu.ms/twiki/bin/view/Networking/MultihomedLinuxNetworking so here is an old copy http://yesican.chsoft.biz/lartc/MultihomedLinuxNetworking.html -- gypsy From hareram at sol.net.in Mon Aug 8 19:14:18 2005 From: hareram at sol.net.in (hareram) Date: Mon Aug 8 19:14:23 2005 Subject: [LARTC] Loadbalancing and failover using TC and Iptables References: <018701c59bda$5f42db40$0903a8c0@southern> <42F76227.17A6B6C8@iswest.com> Message-ID: <026301c59c3c$9c6da790$0903a8c0@southern> Hi sorry i was not mentioned that yes i did with the patch patch-2.6.12-ja1.diff yes iam also seen the document of Dual and try to see how can make that kind of setup any help will be apprciate hare ----- Original Message ----- From: "gypsy" To: Cc: "hareram" Sent: Monday, August 08, 2005 7:16 PM Subject: Re: [LARTC] Loadbalancing and failover using TC and Iptables > hareram wrote: >> >> Hi all >> >> iam trying to deploy loadbalance and failover >> >> My setup description >> --Fedora Core 4 >> --Linux 2.6.12.3 #1 SMP Mon Jul 25 22:37:34 IST 2005 i686 i686 i386 >> GNU/Linux >> --tc utility, iproute2-ss050314 >> --ip utility, iproute2-ss050314 >> --iptables v1.3.0 > > You say nothing about Julian's patch, so I assume you did not patch your > kernel. You must do that. > http://www.ssi.bg/~ja/ > > http://www.geocities.com/mctiew/ffw/dual.htm > > I'm not sure this is still a good link > http://selab.edu.ms/twiki/bin/view/Networking/MultihomedLinuxNetworking > so here is an old copy > http://yesican.chsoft.biz/lartc/MultihomedLinuxNetworking.html > -- > gypsy > > From gregoriandres at yahoo.com.ar Mon Aug 8 19:35:32 2005 From: gregoriandres at yahoo.com.ar (:: L i n u XK i D ::) Date: Mon Aug 8 19:34:52 2005 Subject: [LARTC] Loadbalancing and failover using TC and Iptables In-Reply-To: <42F76227.17A6B6C8@iswest.com> Message-ID: I've read next link: -> I'm not sure this is still a good link -> http://selab.edu.ms/twiki/bin/view/Networking/MultihomedLinuxNetworking is really neccessary mark pakets on this way ? [... snip ...] # iptables -A POSTROUTING -t mangle -j MARK --set-mark 1 \ -m state --state NEW -o ppp0 # iptables -A POSTROUTING -t mangle -j MARK --set-mark 2 \ -m state --state NEW -o ppp1 # iptables -A POSTROUTING -t mangle -j CONNMARK --save-mark \ -m state --state NEW [... snip ...] # iptables -A POSTROUTING -t nat -m mark --mark 1 \ -j SNAT --to-source 11.1.1.1 # iptables -A POSTROUTING -t nat -m mark --mark 2 \ -j SNAT --to-source 22.2.2.2 -> hareram wrote: -> > -> > Hi all -> > -> > iam trying to deploy loadbalance and failover -> > -> > My setup description -> > --Fedora Core 4 -> > --Linux 2.6.12.3 #1 SMP Mon Jul 25 22:37:34 IST 2005 i686 i686 i386 -> > GNU/Linux -> > --tc utility, iproute2-ss050314 -> > --ip utility, iproute2-ss050314 -> > --iptables v1.3.0 -> -> You say nothing about Julian's patch, so I assume you did not patch your -> kernel. You must do that. -> http://www.ssi.bg/~ja/ -> -> http://www.geocities.com/mctiew/ffw/dual.htm -> -> I'm not sure this is still a good link -> http://selab.edu.ms/twiki/bin/view/Networking/MultihomedLinuxNetworking -> so here is an old copy -> http://yesican.chsoft.biz/lartc/MultihomedLinuxNetworking.html -> -- -> gypsy -> _______________________________________________ -> LARTC mailing list -> LARTC@mailman.ds9a.nl -> http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc From hareram at sol.net.in Mon Aug 8 19:47:51 2005 From: hareram at sol.net.in (hareram) Date: Mon Aug 8 19:47:57 2005 Subject: [LARTC] Loadbalancing and failover using TC and Iptables References: Message-ID: <027701c59c41$4c5fc800$0903a8c0@southern> Hi yes i have tried with the docs but from the box iam not able to go out even i configureed on of client and try to access the internet, iam not able to ?? any suggestions hare ----- Original Message ----- From: ":: L i n u XK i D ::" To: "lartc" Sent: Monday, August 08, 2005 11:05 PM Subject: RE: [LARTC] Loadbalancing and failover using TC and Iptables > > I've read next link: > > -> I'm not sure this is still a good link > -> http://selab.edu.ms/twiki/bin/view/Networking/MultihomedLinuxNetworking > > > is really neccessary mark pakets on this way ? > > > [... snip ...] > > # iptables -A POSTROUTING -t mangle -j MARK --set-mark 1 \ > -m state --state NEW -o ppp0 > # iptables -A POSTROUTING -t mangle -j MARK --set-mark 2 \ > -m state --state NEW -o ppp1 > # iptables -A POSTROUTING -t mangle -j CONNMARK --save-mark \ > -m state --state NEW > > [... snip ...] > > > # iptables -A POSTROUTING -t nat -m mark --mark 1 \ > -j SNAT --to-source 11.1.1.1 > # iptables -A POSTROUTING -t nat -m mark --mark 2 \ > -j SNAT --to-source 22.2.2.2 > > > > > > > > -> hareram wrote: > -> > > -> > Hi all > -> > > -> > iam trying to deploy loadbalance and failover > -> > > -> > My setup description > -> > --Fedora Core 4 > -> > --Linux 2.6.12.3 #1 SMP Mon Jul 25 22:37:34 IST 2005 i686 i686 i386 > -> > GNU/Linux > -> > --tc utility, iproute2-ss050314 > -> > --ip utility, iproute2-ss050314 > -> > --iptables v1.3.0 > -> > -> You say nothing about Julian's patch, so I assume you did not patch > your > -> kernel. You must do that. > -> http://www.ssi.bg/~ja/ > -> > -> http://www.geocities.com/mctiew/ffw/dual.htm > -> > -> I'm not sure this is still a good link > -> http://selab.edu.ms/twiki/bin/view/Networking/MultihomedLinuxNetworking > -> so here is an old copy > -> http://yesican.chsoft.biz/lartc/MultihomedLinuxNetworking.html > -> -- > -> gypsy > -> _______________________________________________ > -> LARTC mailing list > -> LARTC@mailman.ds9a.nl > -> http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > > From aaron at dummer.info Mon Aug 8 21:27:43 2005 From: aaron at dummer.info (Aaron Dummer) Date: Mon Aug 8 21:28:07 2005 Subject: [LARTC] Load balancing multiple PPPoE connections Message-ID: <1123529263.15838.42.camel@localhost.localdomain> Hello, Currently I'm using the information from http://lartc.org/howto/lartc.rpdb.multiple-links.html and Julian's patch to load balance two PPPoE Internet connections (DSL). It balances the traffic perfectly. +------------+ + ppp0 (eth0) ---+ Provider 1 + +-----+ | +------------- + LAN +--- eth2 ---+ +-----+ | +------------+ + ppp1 (eth1) ---+ Provider 2 + +------------+ However, occasionally one of the PPP connections goes down due to provider issues, and the interface won't come back up on it's own. To get around this, I setup a cron job which checks to see if ppp0 and ppp1 exist, and if not, bring them back up. In the process of the interfaces disappearing and re-appearing, some of the load balancing routes get lost. I tried re-executing the route commands in the cron job, but it doesn't work every time. This also seems like a hackish way to solve the problem. Have any of you experienced similar problems, or is there a better way to load balance over 'virtual' interfaces like this? From gypsy at iswest.com Tue Aug 9 03:59:37 2005 From: gypsy at iswest.com (gypsy) Date: Tue Aug 9 03:59:44 2005 Subject: [LARTC] Loadbalancing and failover using TC and Iptables References: Message-ID: <42F80E09.2F083F6C@iswest.com> :: L i n u XK i D :: wrote: > > I've read next link: > > -> I'm not sure this is still a good link > -> http://selab.edu.ms/twiki/bin/view/Networking/MultihomedLinuxNetworking > > is really neccessary mark pakets on this way ? >From the machine on which the 2 ISPs are connected to two different NICs, no. It will send and receive packets without marking. Where I have a problem is with NATted users; they are tied to one or the other ISP (even though I run 'ip route flush cache') unless I mark. Maybe Julian will give us some hints ? -- gypsy > [... snip ...] > > # iptables -A POSTROUTING -t mangle -j MARK --set-mark 1 \ > -m state --state NEW -o ppp0 > # iptables -A POSTROUTING -t mangle -j MARK --set-mark 2 \ > -m state --state NEW -o ppp1 > # iptables -A POSTROUTING -t mangle -j CONNMARK --save-mark \ > -m state --state NEW > > [... snip ...] > > # iptables -A POSTROUTING -t nat -m mark --mark 1 \ > -j SNAT --to-source 11.1.1.1 > # iptables -A POSTROUTING -t nat -m mark --mark 2 \ > -j SNAT --to-source 22.2.2.2 > > -> hareram wrote: > -> > > -> > Hi all > -> > > -> > iam trying to deploy loadbalance and failover > -> > > -> > My setup description > -> > --Fedora Core 4 > -> > --Linux 2.6.12.3 #1 SMP Mon Jul 25 22:37:34 IST 2005 i686 i686 i386 > -> > GNU/Linux > -> > --tc utility, iproute2-ss050314 > -> > --ip utility, iproute2-ss050314 > -> > --iptables v1.3.0 > -> > -> You say nothing about Julian's patch, so I assume you did not patch your > -> kernel. You must do that. > -> http://www.ssi.bg/~ja/ > -> > -> http://www.geocities.com/mctiew/ffw/dual.htm > -> > -> I'm not sure this is still a good link > -> http://selab.edu.ms/twiki/bin/view/Networking/MultihomedLinuxNetworking > -> so here is an old copy > -> http://yesican.chsoft.biz/lartc/MultihomedLinuxNetworking.html > -> -- > -> gypsy > -> _______________________________________________ > -> LARTC mailing list > -> LARTC@mailman.ds9a.nl > -> http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc From gregoriandres at yahoo.com.ar Tue Aug 9 16:21:10 2005 From: gregoriandres at yahoo.com.ar (LinuXKiD) Date: Tue Aug 9 16:47:25 2005 Subject: [LARTC] Loadbalancing and failover using TC and Iptables In-Reply-To: <42F80E09.2F083F6C@iswest.com> Message-ID: Another question related with this. I've 4 ADSLs and I already use CONNMARK to MARK out/in traffic from ADSLs in order to make a QoS. # iptables -L -t mangle [... snip ...] Chain POSTROUTING (policy ACCEPT 15M packets, 5610M bytes) pkts bytes target prot opt in out source destination 989K 299M MYSHAPER-OUT all -- * ppp3 0.0.0.0/0 0.0.0.0/0 985K 222M MYSHAPER-OUT all -- * ppp2 0.0.0.0/0 0.0.0.0/0 856K 163M MYSHAPER-OUT all -- * ppp1 0.0.0.0/0 0.0.0.0/0 841K 164M MYSHAPER-OUT all -- * ppp0 0.0.0.0/0 0.0.0.0/0 [... snip ...] Chain MYSHAPER-OUT (4 references) pkts bytes target prot opt in out source destination 39254 7491K MARK tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp spts:0:1024 MARK set 0x17 1920K 221M MARK tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpts:0:1024 MARK set 0x17 1882 153K MARK tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:20 MARK set 0x1a 174 9457 MARK tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:5190 MARK set 0x17 142K 19M MARK tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:1863 MARK set 0x17 [... snip ...] Later, with that MARK I put traffic on a HTB class. ... $TC filter add dev $DEV parent nn:0 prio 0 protocol ip handle XX fw flowid nn:yy ... MY Question is: is possible re-mark traffic or put another mark in order to know which PPP interface going out ? Must I use CLASSIFY to shape in/out PPP traffic , and let MARKs to know which PPP interface going out ? best regards. andres -> -> :: L i n u XK i D :: wrote: -> > -> > I've read next link: -> > -> > -> I'm not sure this is still a good link -> > -> -> http://selab.edu.ms/twiki/bin/view/Networking/MultihomedLinuxNetworking -> > -> > is really neccessary mark pakets on this way ? -> -> From the machine on which the 2 ISPs are connected to two different -> NICs, no. It will send and receive packets without marking. Where I -> have a problem is with NATted users; they are tied to one or the other -> ISP (even though I run 'ip route flush cache') unless I mark. -> -> Maybe Julian will give us some hints ? -> -- -> gypsy -> -> > [... snip ...] -> > -> > # iptables -A POSTROUTING -t mangle -j MARK --set-mark 1 \ -> > -m state --state NEW -o ppp0 -> > # iptables -A POSTROUTING -t mangle -j MARK --set-mark 2 \ -> > -m state --state NEW -o ppp1 -> > # iptables -A POSTROUTING -t mangle -j CONNMARK --save-mark \ -> > -m state --state NEW -> > -> > [... snip ...] -> > -> > # iptables -A POSTROUTING -t nat -m mark --mark 1 \ -> > -j SNAT --to-source 11.1.1.1 -> > # iptables -A POSTROUTING -t nat -m mark --mark 2 \ -> > -j SNAT --to-source 22.2.2.2 -> > -> > -> hareram wrote: -> > -> > -> > -> > Hi all -> > -> > -> > -> > iam trying to deploy loadbalance and failover -> > -> > -> > -> > My setup description -> > -> > --Fedora Core 4 -> > -> > --Linux 2.6.12.3 #1 SMP Mon Jul 25 22:37:34 IST 2005 -> i686 i686 i386 -> > -> > GNU/Linux -> > -> > --tc utility, iproute2-ss050314 -> > -> > --ip utility, iproute2-ss050314 -> > -> > --iptables v1.3.0 -> > -> -> > -> You say nothing about Julian's patch, so I assume you did -> not patch your -> > -> kernel. You must do that. -> > -> http://www.ssi.bg/~ja/ -> > -> -> > -> http://www.geocities.com/mctiew/ffw/dual.htm -> > -> -> > -> I'm not sure this is still a good link -> > -> -> http://selab.edu.ms/twiki/bin/view/Networking/MultihomedLinuxNetworking -> > -> so here is an old copy -> > -> http://yesican.chsoft.biz/lartc/MultihomedLinuxNetworking.html -> > -> -- -> > -> gypsy -> > -> _______________________________________________ -> > -> LARTC mailing list -> > -> LARTC@mailman.ds9a.nl -> > -> http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc -> > _______________________________________________ -> > LARTC mailing list -> > LARTC@mailman.ds9a.nl -> > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc From psihozefir at yahoo.com Tue Aug 9 18:53:17 2005 From: psihozefir at yahoo.com (panca sorin) Date: Tue Aug 9 18:53:22 2005 Subject: [LARTC] Too slow computer? Message-ID: <20050809165318.29066.qmail@web32604.mail.mud.yahoo.com> Hello! I've put some questions on this list some weeks ago and I've got good answers. Thank you! Now I've finished my (beautyful) script and I ran it on my router... About my script: It routes packages based on their destination on the Internet. I have about 1650 preffered destination networks listed in some file. The script read this file and marks every package for those networks with the mark value of 1. Then, I've built the classes: 1:0 is the HTB qdisc; 1:1 is for unmarked packets and 1:2 is for marked packets. 1:FF is the default class, and has as parent the 1:0 qdisc. 1:1 is divided in 1:10 for two LAN networks 192.168.101.0/24 and 10.0.0.0/24 attached to eth1 and eth2 respectivly. eth0 is attached to my ISP. 1:11 is a class for two IPs sharing the same bandwidth. 1:12 is a class for one IP. When I shape the traffic for eth0, eth1 and eth2 the console is unuseable (is too slow). I think that marking and matching those packets takes all the available CPU cycles (the kernel has the biggest priority over the CPU). Another thing that's going on here: I test the router with only one station attached to eth1 and eth2 alternativly. I should have a speed of (at least) 10Mbit with the networks listed in my network list and I have a maximum of 3.2Mbit (class 1:2). In the 1:1 class I have maximum speed allowed. Another question: How can I build the mark value? I need to mark the packets based on two criteria. My theoretic solution was to assign bit #1 to be set if the packet matches the first criteria and bits #2 & #3 should indicate 4 situations (00,01,10 and 11). Is there any way to set a packet's mark bits additively in two places of my script? __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From Andreas.Klauer at metamorpher.de Tue Aug 9 19:12:38 2005 From: Andreas.Klauer at metamorpher.de (Andreas Klauer) Date: Tue Aug 9 19:13:00 2005 Subject: [LARTC] Too slow computer? In-Reply-To: <20050809165318.29066.qmail@web32604.mail.mud.yahoo.com> References: <20050809165318.29066.qmail@web32604.mail.mud.yahoo.com> Message-ID: <200508091912.38949.Andreas.Klauer@metamorpher.de> On Tuesday 09 August 2005 18:53, panca sorin wrote: > I have about 1650 preffered destination networks listed in some file. The > script read this file and marks every package for those networks with > the mark value of 1. If you have a lot of IPs in this list, a hashed approach might work faster. See LARTC Howto, 12.4 Hashing filters. Although it describes tc filters, approach should be similar for iptables. Furthermore, using CONNMARK might speed things up. With it, you can skip testing packets of connections that already matched (and, if used right, you can also skip packets of connections that don't match as well). There are also patches that allow bitwise modification of mark values. You can get this stuff from www.netfilter.org, the patches are in pom-ng. HTH Andreas From andy.furniss at dsl.pipex.com Tue Aug 9 19:46:34 2005 From: andy.furniss at dsl.pipex.com (Andy Furniss) Date: Tue Aug 9 19:46:31 2005 Subject: [LARTC] Too slow computer? In-Reply-To: <200508091912.38949.Andreas.Klauer@metamorpher.de> References: <20050809165318.29066.qmail@web32604.mail.mud.yahoo.com> <200508091912.38949.Andreas.Klauer@metamorpher.de> Message-ID: <42F8EBFA.2020309@dsl.pipex.com> Andreas Klauer wrote: > On Tuesday 09 August 2005 18:53, panca sorin wrote: > >>I have about 1650 preffered destination networks listed in some file. The >>script read this file and marks every package for those networks with >>the mark value of 1. > > > If you have a lot of IPs in this list, a hashed approach might work faster. > See LARTC Howto, 12.4 Hashing filters. Although it describes tc filters, > approach should be similar for iptables. Furthermore, using CONNMARK might > speed things up. With it, you can skip testing packets of connections that > already matched (and, if used right, you can also skip packets of > connections that don't match as well). There are also patches that allow > bitwise modification of mark values. > > You can get this stuff from www.netfilter.org, the patches are in pom-ng. Look for ipset if the list is random. http://people.netfilter.org/kadlec/ipset/ --and-mark and --or-mark are part of main iptables now Andy. From gregoriandres at yahoo.com.ar Tue Aug 9 22:42:12 2005 From: gregoriandres at yahoo.com.ar (LinuXKiD) Date: Tue Aug 9 22:41:23 2005 Subject: [LARTC] Loadbalancing and failover using TC and Iptables In-Reply-To: Message-ID: I've tried this on my 4 adsl Linux + 10 hosts lan... but works better without "marks" -> -> Another question related with this. -> -> I've 4 ADSLs and I already use CONNMARK -> to MARK out/in traffic from ADSLs in order -> to make a QoS. -> -> # iptables -L -t mangle -> -> [... snip ...] -> -> Chain POSTROUTING (policy ACCEPT 15M packets, 5610M bytes) -> pkts bytes target prot opt in out source -> destination -> 989K 299M MYSHAPER-OUT all -- * ppp3 0.0.0.0/0 -> 0.0.0.0/0 -> 985K 222M MYSHAPER-OUT all -- * ppp2 0.0.0.0/0 -> 0.0.0.0/0 -> 856K 163M MYSHAPER-OUT all -- * ppp1 0.0.0.0/0 -> 0.0.0.0/0 -> 841K 164M MYSHAPER-OUT all -- * ppp0 0.0.0.0/0 -> 0.0.0.0/0 -> -> [... snip ...] -> -> Chain MYSHAPER-OUT (4 references) -> pkts bytes target prot opt in out source -> destination -> 39254 7491K MARK tcp -- * * 0.0.0.0/0 -> 0.0.0.0/0 tcp spts:0:1024 MARK set 0x17 -> 1920K 221M MARK tcp -- * * 0.0.0.0/0 -> 0.0.0.0/0 tcp dpts:0:1024 MARK set 0x17 -> 1882 153K MARK tcp -- * * 0.0.0.0/0 -> 0.0.0.0/0 tcp dpt:20 MARK set 0x1a -> 174 9457 MARK tcp -- * * 0.0.0.0/0 -> 0.0.0.0/0 tcp dpt:5190 MARK set 0x17 -> 142K 19M MARK tcp -- * * 0.0.0.0/0 -> 0.0.0.0/0 tcp dpt:1863 MARK set 0x17 -> [... snip ...] -> -> -> Later, with that MARK I put traffic on a HTB class. -> ... -> $TC filter add dev $DEV parent nn:0 prio 0 protocol ip handle XX -> fw flowid -> nn:yy -> ... -> -> MY Question is: -> is possible re-mark traffic or put another mark in order -> to know which PPP interface going out ? -> -> Must I use CLASSIFY to shape in/out PPP traffic , and let MARKs -> to know which PPP interface going out ? -> -> best regards. -> -> andres -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> :: L i n u XK i D :: wrote: -> -> > -> -> -> > I've read next link: -> -> > -> -> > -> I'm not sure this is still a good link -> -> > -> -> -> -> http://selab.edu.ms/twiki/bin/view/Networking/MultihomedLinuxNetworking -> -> > -> -> > is really neccessary mark pakets on this way ? -> -> -> -> From the machine on which the 2 ISPs are connected to two different -> -> NICs, no. It will send and receive packets without marking. Where I -> -> have a problem is with NATted users; they are tied to one or the other -> -> ISP (even though I run 'ip route flush cache') unless I mark. -> -> -> -> Maybe Julian will give us some hints ? -> -> -- -> -> gypsy -> -> -> -> > [... snip ...] -> -> > -> -> > # iptables -A POSTROUTING -t mangle -j MARK --set-mark 1 \ -> -> > -m state --state NEW -o ppp0 -> -> > # iptables -A POSTROUTING -t mangle -j MARK --set-mark 2 \ -> -> > -m state --state NEW -o ppp1 -> -> > # iptables -A POSTROUTING -t mangle -j CONNMARK --save-mark \ -> -> > -m state --state NEW -> -> > -> -> > [... snip ...] -> -> > -> -> > # iptables -A POSTROUTING -t nat -m mark --mark 1 \ -> -> > -j SNAT --to-source 11.1.1.1 -> -> > # iptables -A POSTROUTING -t nat -m mark --mark 2 \ -> -> > -j SNAT --to-source 22.2.2.2 -> -> > -> -> > -> hareram wrote: -> -> > -> > -> -> > -> > Hi all -> -> > -> > -> -> > -> > iam trying to deploy loadbalance and failover -> -> > -> > -> -> > -> > My setup description -> -> > -> > --Fedora Core 4 -> -> > -> > --Linux 2.6.12.3 #1 SMP Mon Jul 25 22:37:34 IST 2005 -> -> i686 i686 i386 -> -> > -> > GNU/Linux -> -> > -> > --tc utility, iproute2-ss050314 -> -> > -> > --ip utility, iproute2-ss050314 -> -> > -> > --iptables v1.3.0 -> -> > -> -> -> > -> You say nothing about Julian's patch, so I assume you did -> -> not patch your -> -> > -> kernel. You must do that. -> -> > -> http://www.ssi.bg/~ja/ -> -> > -> -> -> > -> http://www.geocities.com/mctiew/ffw/dual.htm -> -> > -> -> -> > -> I'm not sure this is still a good link -> -> > -> -> -> http://selab.edu.ms/twiki/bin/view/Networking/MultihomedLinuxNetworking -> > -> so here is an old copy -> > -> http://yesican.chsoft.biz/lartc/MultihomedLinuxNetworking.html -> > -> -- -> > -> gypsy -> > -> _______________________________________________ -> > -> LARTC mailing list -> > -> LARTC@mailman.ds9a.nl -> > -> http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc -> > _______________________________________________ -> > LARTC mailing list -> > LARTC@mailman.ds9a.nl -> > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc _______________________________________________ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc From psihozefir at yahoo.com Tue Aug 9 23:31:22 2005 From: psihozefir at yahoo.com (panca sorin) Date: Tue Aug 9 23:31:27 2005 Subject: [LARTC] Too slow computer? In-Reply-To: <42F8EBFA.2020309@dsl.pipex.com> Message-ID: <20050809213122.80048.qmail@web32606.mail.mud.yahoo.com> Thanck you for your help! I noticed the ipset tools and I tried to use the CONNMARK but I don't know how to verify if bitwise manipulation works. The IP list is random and the router is an Athlon at 1200 MHz with 64 MB of SDRAM and a PIO mode 4 harddisk. After marking for destination, the packets are marked for priorization. I tried to use the dsmark and some ingress policing but I've faild to understand how they work. Also I'm in a hurry and I try to use what I know for now. Since I have to shape for two speeds, now I've discovered the --limit filter in iptables and I try to match packets based on their speeds. Each connected client has its own class on dev eth1. There are 38 clients now. On eth2 I shape based on connection ports. Audio/video, chat and interactive traffic (and connection control packets) have higher priority. Here are my script and configuration files (is best viewd unwraped with kwrite): #!/bin/bash ### firewall.sh ### # firewall # TODO: make a README for admin-users, how to add # clients with public and privat IPs from dhcpd and metropolitan addresses # use ipset for address and port grouping # boost speeds, ports forward, etc. # http://gentoo-wiki.com/HOWTO_Packet_Shaping # http://lartc.org/howto # http://linuxgazette.net/103/odonovan.html # http://www.netfilter.org/documentation/ # http://www.knowplace.org/shaper/ # http://linux-ip.net/articles/Traffic-Control-HOWTO/ # http://howtos.linux.com/howtos/Traffic-Control-HOWTO/intro.shtml # http://andthatsjazz.org:8/lartc/ # programs ip=/usr/sbin/ip ipt=/usr/sbin/iptables ipt_s=/usr/sbin/iptables-save ipt_r=/usr/sbin/iptables-restore ips=/usr/sbin/ipset tc=/usr/sbin/tc # interfaces EXT1=eth0 EXT1IP=first external IP GW1=our gateway's IP NetP1=our ISP's local network # 64 public space addresses PUB1Min=first usable public IP PUB1Max=last usable public IP #EXT2= #EXT1IP= #GW2= #NetP2= INT1=eth1 INT1IP=192.168.101.1 INT1Mask=255.255.255.0 INT1Bcast=public space broadcast address (not in ISP's LAN) INT1Net=192.168.101.255 INT2=eth2 INT2IP=10.0.0.1 INT2Mask=255.255.255.0 INT2Bcast=10.0.0.255 INT2Net=10.0.0.0 # markers MARK_NET=0x0 # packets for Internet MARK_MAN=0x1 # packets for Metropolitan # interfaces' aliasses NETWORK=81.196.157;DEV=eth0 ip address add 172.22.3.112 dev eth0 for IP in $( cat ~adminus/etc/ip_internet/ext1_aliases.conf | grep -v \# ); do $ip addr del $NETWORK.$IP/32 dev $DEV 2>/dev/null >/dev/null done for IP in $( cat ~adminus/etc/ip_internet/ext1_aliases.conf | grep -v \# ); do $ip addr add $NETWORK.$IP/26 brd $NETWORK.255 dev $DEV done echo " 2. Proxy ARP" # proxy ARP echo 1 >/proc/sys/net/ipv4/conf/$EXT1/proxy_arp #echo 1 >/proc/sys/net/ipv4/conf/$EXT2/proxy_arp echo 1 >/proc/sys/net/ipv4/conf/$INT1/proxy_arp #echo 1 >/proc/sys/net/ipv4/conf/$INT1/proxy_arp for IP in $( cat ~adminus/etc/ip_local/pub_ips_on_int1.conf | grep -v \# ); do $ip route del $IP dev $INT1 2>/dev/null >/dev/null $ip route add $IP dev $INT1 done for IP in $( cat ~adminus/etc/ip_local/priv_ips_on_int1.conf | grep -v \# ); do $ip route del $IP dev $INT2 2>/dev/null >/dev/null $ip route add $IP dev $INT2 done $ipt -t raw -F $ipt -t nat -F $ipt -t mangle -F $ipt -t filter -F ### ### ### ### raw ### ### ### ### ### ### ### ### nat ### ### ### ### ### PREROUTING ### #$ipt -t nat -A PREROUTING -i $INT1 -p tcp --dport 80 -j REDIRECT --to-port 3128 echo " forward ports (5 ports/IP)" NETWORK=192.168.101;NETID1=21;NETID2=22;NETID3=23;NETID4=24;NETID5=25; # 20 <= NETID <= 65 for IP in $( cat ~adminus/etc/portfwd.conf | grep -v \# ); do $ipt -t nat -A PREROUTING -d $EXT1IP -p tcp -m tcp --dport $NETID1$IP -j DNAT --to-destination $NETWORK.$IP:$NETID1$IP $ipt -t nat -A PREROUTING -d $EXT1IP -p tcp -m tcp --dport $NETID2$IP -j DNAT --to-destination $NETWORK.$IP:$NETID2$IP $ipt -t nat -A PREROUTING -d $EXT1IP -p tcp -m tcp --dport $NETID3$IP -j DNAT --to-destination $NETWORK.$IP:$NETID3$IP $ipt -t nat -A PREROUTING -d $EXT1IP -p tcp -m tcp --dport $NETID4$IP -j DNAT --to-destination $NETWORK.$IP:$NETID4$IP $ipt -t nat -A PREROUTING -d $EXT1IP -p tcp -m tcp --dport $NETID5$IP -j DNAT --to-destination $NETWORK.$IP:$NETID5$IP done ### POSTROUTING ### echo " nat POSTROUTING" #$ipt -t nat -A POSTROUTING -s $INT2Net/$INT2Mask -j MASQUERADE --to-ports 20000:30000 $ipt -t nat -A POSTROUTING -s $INT1Net/$INT1Mask -o $EXT1 -j SNAT --to-source $PUB1Min-$PUB1Max $ipt -t nat -A POSTROUTING -s $INT2Net/$INT2Mask -o $EXT1 -j SNAT --to-source $PUB1Min-$PUB1Max $ipt -t nat -A POSTROUTING -s 10.0.0.100 -j SNAT --to-source 81.196.157.200 $ipt -t nat -A POSTROUTING -s 10.0.0.99 -j SNAT --to-source 81.196.157.200 ### ### ### ### ### mangle ### ### ### ### ### echo " mangle" ### PREROUTING ### # mark for QOS cat ~adminus/bin/marks | $ipt_r ~adminus/bin/mac.sh ### ### ### ### ### qdiscs ### ### ### ### ### # building traffic classes and ingress filters # speeds ROOT_NET_RATE=500kbit ROOT_NET_CEIL=$ROOT_NET_RATE BULK_NET_RATE=1kbit BULK_NET_CEIL=128kbit ROOT_MAN_RATE=95Mbit ROOT_MAN_CEIL=$BULK_NET_RATE BULK_MAN_RATE=512kbit BULK_MAN_CEIL=90Mbit # markers MARK_NET=0x0 # Internet packet MARK_MAN=0x1 # Metropolitan packet echo " qdisc del" $tc qdisc del dev $EXT1 ingress 2>/dev/null >/dev/null #$tc qdisc del dev $EXT2 ingress 2>/dev/null >/dev/null $tc qdisc del dev $INT1 ingress 2>/dev/null >/dev/null $tc qdisc del dev $INT2 ingress 2>/dev/null >/dev/null $tc qdisc del dev $EXT1 root 2>/dev/null >/dev/null #$tc qdisc del dev $EXT2 root 2>/dev/null >/dev/null $tc qdisc del dev $INT1 root 2>/dev/null >/dev/null $tc qdisc del dev $INT2 root 2>/dev/null >/dev/null echo " qdisc add EXT1 egress " $tc qdisc add dev $EXT1 root handle 1: htb default FF01 echo " Internet-caffe" $tc class add dev $EXT1 parent 1: classid 1:1 htb rate 500kbit ceil 500kbit # Internet $tc class add dev $EXT1 parent 1: classid 1:2 htb rate 95Mbit ceil 95Mbit # Metropolitan $tc class add dev $EXT1 parent 1:1 classid 1:7 htb rate 140kbit ceil 500kbit prio 2 # a/v net trafic $tc class add dev $EXT1 parent 1:1 classid 1:5 htb rate 50kbit ceil 500kbit prio 2 # chat net trafic $tc class add dev $EXT1 parent 1:1 classid 1:3 htb rate 100kbit ceil 500kbit prio 2 # www net trafic $tc class add dev $EXT1 parent 1:2 classid 1:8 htb rate 35Mbit ceil 90Mbit prio 2 # a/v man trafic $tc class add dev $EXT1 parent 1:2 classid 1:6 htb rate 5Mbit ceil 90Mbit prio 2 # chat man trafic $tc class add dev $EXT1 parent 1:2 classid 1:4 htb rate 20Mbit ceil 90Mbit prio 2 # www man trafic $tc class add dev $EXT1 parent 1:1 classid 1:FF01 htb rate 10kbit ceil 500kbit prio 3 # bulk net trafic $tc class add dev $EXT1 parent 1:2 classid 1:FF00 htb rate 30Mbit ceil 90Mbit prio 3 # bulk man trafic $tc qdisc add dev $EXT1 parent 1:FF01 handle 2: sfq perturb 10 $tc qdisc add dev $EXT1 parent 1:FF00 handle 3: sfq perturb 10 echo "qdisc add $EXT1 ingress" $tc qdisc add dev $EXT1 ingress # Metropolitan ingress #$tc filter add dev $EXT1 parent FFFF: protocol ip prio 0 handle 7 fw police rate 10Mbps burst 16k continue flowid :1 # A/V in MAN #$tc filter add dev $EXT1 parent FFFF: protocol ip prio 1 handle 5 fw police rate 10Mbps burst 16k continue flowid :1 # chat in MAN #$tc filter add dev $EXT1 parent FFFF: protocol ip prio 2 handle 3 fw police rate 10Mbps burst 16k continue flowid :1 # www in MAN #$tc filter add dev $EXT1 parent FFFF: protocol ip prio 4 handle 1 fw police rate 90Mbps burst 16k continue flowid :1 # bulk in MAN echo "CLIENTS";date >~adminus/log/clase_eth0.log;echo "CLIENTS" >>~adminus/log/clase_eth0.log $tc class add dev $EXT1 parent 1:1 classid 1:9 htb rate 140kbit ceil 500kbit prio 2 # bulk clients' net $tc class add dev $EXT1 parent 1:1 classid 1:10 htb rate 20Mbit ceil 90Mbit prio 2 # bulk clients' M.A.N. $tc class add dev $EXT1 parent 1:1 classid 1:11 htb rate 140kbit ceil 500kbit prio 1 # special clients' net $tc class add dev $EXT1 parent 1:1 classid 1:12 htb rate 20Mbit ceil 90Mbit prio 1 # special clients' M.A.N. echo " bulk clients' classes";echo " bulk clients' classes" >>~adminus/log/clase_eth0.log NETWORK=192.168;NET=101;NETID=16 # edit this after copy-paste ID_NET=0;ID_MAN=128;ID_PRIV=0;ID_PUB=64 # don't edit IDnet_PRIV=$[$ID_NET+$ID_PRIV+$NETID];IDman_PRIV=$[$ID_MAN+$ID_PRIV+$NETID];IDnet_PUB=$[$ID_NET+$ID_PUB+$NETID];IDman_PUB=$[$ID_MAN+$ID_PUB+$NETID] # don't edit hIDnet_PRIV=`printf "%x" $IDnet_PRIV`;hIDman_PRIV=`printf "%x" $IDman_PRIV`;hIDnet_PUB=`printf "%x" $IDnet_PUB`;hIDman_PUB=`printf "%x" $IDman_PUB` # don't edit for IP in $( cat ~adminus/etc/ip_local/priv_ips_on_int1.conf | grep -v \# ); do hNET=`printf "%x" $NET`;hIP=`printf "%x" $IP` $tc class add dev $EXT1 parent 1:9 classid 1:$hIDnet_PRIV$hIP htb rate $BULK_NET_RATE ceil $BULK_NET_CEIL prio 3 $tc class add dev $EXT1 parent 1:10 classid 1:$hIDman_PRIV$hIP htb rate $BULK_MAN_RATE ceil $BULK_MAN_CEIL prio 3 echo "$EXT1: $NETWORK.$NET.$IP net (1:9): 1:$hIDnet_PRIV$hIP min: $BULK_NET_RATE max: $BULK_NET_CEIL man (1:10): 1:$hIDman_PRIV$hIP min: $BULK_MAN_RATE max: $BULK_MAN_CEIL" >>~adminus/log/clase_eth0.log done echo " special clients' classes";echo " special clients' classes" >>~sorin/log/clase_eth0.log echo " ip-uri private";echo " private IPs" >>~adminus/log/clase_eth0.log NETWORK=192.168;NET=101;NETID=16 # edit this after copy-paste; 16 < NETID < 192; NETID = network's criterium number; # Set different NETIDs for all private or public networks; you can set the same NETID for one private network and one public network ID_NET=0;ID_MAN=128;ID_PRIV=0;ID_PUB=64 # don't edit IDnet_PRIV=$[$ID_NET+$ID_PRIV+$NETID];IDman_PRIV=$[$ID_MAN+$ID_PRIV+$NETID];IDnet_PUB=$[$ID_NET+$ID_PUB+$NETID];IDman_PUB=$[$ID_MAN+$ID_PUB+$NETID] # don't edit hIDnet_PRIV=`printf "%x" $IDnet_PRIV`;hIDman_PRIV=`printf "%x" $IDman_PRIV`;hIDnet_PUB=`printf "%x" $IDnet_PUB`;hIDman_PUB=`printf "%x" $IDman_PUB` # don't edit IP=2 # 192.168.101.002 FOCUS DESIGN echo "$EXT1: $NETWORK.$NET.$IP net (1:11): 1:$hIDnet_PRIV$hIP min: 64kbit max: 256kbit man (1:12): 1:$hIDman_PRIV$hIP min: 768kbit max: 90Mbit" >>~adminus/log/clase_eth0.log hNET=`printf "%x" $NET`;hIP=`printf "%x" $IP` $tc class replace dev $EXT1 parent 1:11 classid 1:$hIDnet_PRIV$hIP htb rate 64kbit ceil 256kbit prio 2 # replace because the class' ID (handle) exists from the previous network $tc class replace dev $EXT1 parent 1:12 classid 1:$hIDman_PRIV$hIP htb rate 768kbit ceil 90Mbit prio 2 # replace because the class' ID (handle) exists from the previous network echo " ip-uri publice";echo " public IPs" >>~adminus/log/clase_eth0.log NETWORK=81.196;NET=157;NETID=63 # edit this after copy-paste ID_NET=0;ID_MAN=128;ID_PRIV=0;ID_PUB=64 # astea nu le edita IDnet_PRIV=$[$ID_NET+$ID_PRIV+$NETID];IDman_PRIV=$[$ID_MAN+$ID_PRIV+$NETID];IDnet_PUB=$[$ID_NET+$ID_PUB+$NETID];IDman_PUB=$[$ID_MAN+$ID_PUB+$NETID] # don't edit hIDnet_PRIV=`printf "%x" $IDnet_PRIV`;hIDman_PRIV=`printf "%x" $IDman_PRIV`;hIDnet_PUB=`printf "%x" $IDnet_PUB`;hIDman_PUB=`printf "%x" $IDman_PUB` # don't edit IP=253 # 81.196.157.253 VIDEO CHAT hNET=`printf "%x" $NET`;hIP=`printf "%x" $IP` echo "$EXT1: $NETWORK.$NET.$IP net (1:11): 1:$hIDnet_PUB$hIP min: 64kbit max: 256kbit man (1:12) 1:$hIDman_PUB$hIP min: 768kbit max: 90Mbit" >>~adminus/log/clase_eth0.log $tc class add dev $EXT1 parent 1:11 classid 1:$hIDnet_PUB$hIP htb rate 64kbit ceil 256kbit prio 1 $tc class add dev $EXT1 parent 1:12 classid 1:$hIDman_PUB$hIP htb rate 768kbit ceil 90Mbit prio 1 IP=254 # 81.196.157.254 VIDEO CHAT hNET=`printf "%x" $NET`;hIP=`printf "%x" $IP` echo "$EXT1: $NETWORK.$NET.$IP net (1:11): 1:$hIDnet_PUB$hIP min: 64kbit max: 256kbit man (1:12) 1:$hIDman_PUB$hIP min: 768kbit max: 90Mbit" >>~adminus/log/clase_eth0.log $tc class add dev $EXT1 parent 1:11 classid 1:$hIDnet_PUB$hIP htb rate 64kbit ceil 256kbit prio 1 $tc class add dev $EXT1 parent 1:12 classid 1:$hIDman_PUB$hIP htb rate 768kbit ceil 90Mbit prio 1 # Internet ingress #$tc filter add dev $EXT1 parent FFFF: protocol ip prio 0 handle 6 fw police rate 190kbps burst 16k drop flowid :1 # A/V in Internet #$tc filter add dev $EXT1 parent FFFF: protocol ip prio 1 handle 4 fw police rate 62kbps burst 32k drop flowid :1 # chat in Internet #$tc filter add dev $EXT1 parent FFFF: protocol ip prio 2 handle 2 fw police rate 126kbps burst 64k drop flowid :1 # www in Internet #$tc filter add dev $EXT1 parent FFFF: protocol ip prio 3 u32 match ip dst 0.0.0.0/0 police rate 126kbit burst 1k drop flowid :1 # bulk in Internet echo " qdisc add INT1 ingress" #$tc qdisc add dev $INT1 ingress #$tc filter add dev $INT1 parent FFFF: protocol ip prio 0 handle 0x7 fw flowid :1 police rate 10Mbps burst 16k continue # A/V in MAN #$tc filter add dev $INT1 parent FFFF: protocol ip prio 1 handle 0x5 fw flowid :1 police rate 10Mbps burst 16k continue # chat in MAN #$tc filter add dev $INT1 parent FFFF: protocol ip prio 2 handle 0x3 fw flowid :1 police rate 10Mbps burst 16k continue # www in MAN #$tc filter add dev $INT1 parent FFFF: protocol ip prio 4 handle 0x1 fw flowid :1 police rate 95Mbps burst 16k continue # bulk in MAN #$tc filter add dev $INT1 parent FFFF: protocol ip prio 0 handle 0x6 fw flowid :1 police rate 190kbps burst 16k continue # A/V in Internet #$tc filter add dev $INT1 parent FFFF: protocol ip prio 1 handle 0x4 fw flowid :1 police rate 62kbps burst 32k continue # chat in Internet #$tc filter add dev $INT1 parent FFFF: protocol ip prio 2 handle 0x2 fw flowid :1 police rate 126kbps burst 64k continue # www in Internet #$tc filter add dev $INT1 parent FFFF: protocol ip prio 3 u32 match ip dst 0.0.0.0/0 police rate 126kbit burst 1k drop flowid :1 # bulk in Internet echo " qdisc add INT1 egress" $tc qdisc add dev $INT1 root handle 1: htb default FF01 $tc class add dev $INT1 parent 1: classid 1:1 htb rate 250kbit ceil 500kbit # class Internet $tc class add dev $INT1 parent 1: classid 1:2 htb rate 45Mbit ceil 90Mbit # class Metropolitan $tc class add dev $INT1 parent 1:1 classid 1:3 htb rate 125kbit ceil 500kbit # class bulk-clients Internet $tc class add dev $INT1 parent 1:2 classid 1:4 htb rate 22Mbit ceil 90Mbit # class bulk-clients Metropolitan $tc class add dev $INT1 parent 1:1 classid 1:5 htb rate 125kbit ceil 500kbit # class special-clients Internet $tc class add dev $INT1 parent 1:2 classid 1:6 htb rate 22Mbit ceil 90Mbit # class special-clients Metropolitan $tc class add dev $INT1 parent 1: classid 1:FF01 htb rate 1kbit ceil 500kbit # class bulk-traffic Internet $tc class add dev $INT1 parent 1: classid 1:FF00 htb rate 1kbit ceil 90Mbit # class bulk-traffic Metropolitan $tc qdisc add dev $INT1 parent 1:FF01 handle 2: sfq perturb 10 # Stochastic Fairness for bulk traffic in Internet $tc qdisc add dev $INT1 parent 1:FF00 handle 3: sfq perturb 10 # Stochastic Fairness for bulk traffic in Metropolitan echo "CLIENTS";date >~adminus/log/clase_eth1.log;echo "CLIENTI" >>~adminus/log/clase_eth1.log echo " bulk clients";echo " bulk clients" >>~adminus/log/clase_eth1.log NETWORK=192.168;NET=101;NETID=16 # edit this after copy-paste ID_NET=0;ID_MAN=128;ID_PRIV=0;ID_PUB=64 # don't edit IDnet_PRIV=$[$ID_NET+$ID_PRIV+$NETID];IDman_PRIV=$[$ID_MAN+$ID_PRIV+$NETID];IDnet_PUB=$[$ID_NET+$ID_PUB+$NETID];IDman_PUB=$[$ID_MAN+$ID_PUB+$NETID] # don't edit hIDnet_PRIV=`printf "%x" $IDnet_PRIV`;hIDman_PRIV=`printf "%x" $IDman_PRIV`;hIDnet_PUB=`printf "%x" $IDnet_PUB`;hIDman_PUB=`printf "%x" $IDman_PUB` # don't edit for IP in $( cat ~adminus/etc/ip_local/priv_ips_on_int1.conf | grep -v \# ); do hNET=`printf "%x" $NET`;hIP=`printf "%x" $IP` $tc class add dev $INT1 parent 1:3 classid 1:$hIDnet_PRIV$hIP htb rate $BULK_NET_RATE ceil $BULK_NET_CEIL prio 3 # bulk clients' speed in Internet $tc class add dev $INT1 parent 1:4 classid 1:$hIDman_PRIV$hIP htb rate $BULK_MAN_RATE ceil $BULK_MAN_CEIL prio 3 # bulk clients' speed in Metropolitan echo "$INT1: $NETWORK.$NET.$IP net (1:3): 1:$hIDnet_PRIV$hIP min: $BULK_NET_RATE max: $BULK_NET_CEIL man (1:4): 1:$hIDman_PRIV$hIP min: $BULK_MAN_RATE max: $BULK_MAN_CEIL" >>~sorin/log/clase_eth1.log done echo " special clients" >>~adminus/log/clase_eth1.log echo " privat IPs" >>~adminus/log/clase_eth1.log NETWORK=192.168;NET=101;NETID=16 # edit this after copy-paste ID_NET=0;ID_MAN=128;ID_PRIV=0;ID_PUB=64 # astea nu le edita IDnet_PRIV=$[$ID_NET+$ID_PRIV+$NETID];IDman_PRIV=$[$ID_MAN+$ID_PRIV+$NETID];IDnet_PUB=$[$ID_NET+$ID_PUB+$NETID];IDman_PUB=$[$ID_MAN+$ID_PUB+$NETID] # don't edit hIDnet_PRIV=`printf "%x" $IDnet_PRIV`;hIDman_PRIV=`printf "%x" $IDman_PRIV`;hIDnet_PUB=`printf "%x" $IDnet_PUB`;hIDman_PUB=`printf "%x" $IDman_PUB` # don't edit IP=2 # 192.168.101.002 FOCUS DESIGN hNET=`printf "%x" $NET`;hIP=`printf "%x" $IP` $tc class replace dev $INT1 parent 1:5 classid 1:$hIDnet_PRIV$hIP htb rate 64kbit ceil 256kbit prio 2 # speed for client FOCUS DESIGN in Internet $tc class replace dev $INT1 parent 1:6 classid 1:$hIDman_PRIV$hIP htb rate 768kbit ceil 90Mbit prio 2 # speed for client FOCUS DESIGN in Metropolitan echo "$INT1: $NETWORK.$NET.$IP net (1:5): 1:$hIDnet_PRIV$hIP min: 64kbit max: 256kbit man (1:6): 1:$hIDman_PRIV$hIP min: 768kbit max: 90Mbit" >>~adminus/log/clase_eth1.log echo " public IPs" >>~adminus/log/clase_eth1.log NETWORK=81.196;NET=157;NETID=63 # edit this after copy-paste (this and the next 3 rows are must be copied for each used ip in the above network) ID_NET=0;ID_MAN=128;ID_PRIV=0;ID_PUB=64 # don't edit IDnet_PRIV=$[$ID_NET+$ID_PRIV+$NETID];IDman_PRIV=$[$ID_MAN+$ID_PRIV+$NETID];IDnet_PUB=$[$ID_NET+$ID_PUB+$NETID];IDman_PUB=$[$ID_MAN+$ID_PUB+$NETID] # don't edit hIDnet_PRIV=`printf "%x" $IDnet_PRIV`;hIDman_PRIV=`printf "%x" $IDman_PRIV`;hIDnet_PUB=`printf "%x" $IDnet_PUB`;hIDman_PUB=`printf "%x" $IDman_PUB` # don't edit IP=253 # 81.196.157.253 VIDEO CHAT 1 (this and the next 3 rows are must be copied for each used ip in the above network) hNET=`printf "%x" $NET`;hIP=`printf "%x" $IP` $tc class add dev $INT1 parent 1:5 classid 1:$hIDnet_PUB$hIP htb rate 64kbit ceil 256kbit prio 1 # speed for client VIDEO CHAT 1 in Internet $tc class add dev $INT1 parent 1:6 classid 1:$hIDman_PUB$hIP htb rate 768kbit ceil 90Mbit prio 1 # speed for client VIDEO CHAT 1 in Metropolitan echo "$INT1: $NETWORK.$NET.$IP net (1:5): 1:$hIDnet_PUB$hIP min: 64kbit max: 256kbit man (1:6) 1:$hIDman_PUB$hIP min: 768kbit max: 90Mbit" >>~adminus/log/clase_eth1.log IP=254 # 81.196.157.254 VIDEO CHAT 2 (this and the next 3 rows are must be copied for each used ip in the above network) hNET=`printf "%x" $NET`;hIP=`printf "%x" $IP` $tc class add dev $INT1 parent 1:5 classid 1:$hIDnet_PUB$hIP htb rate 64kbit ceil 256kbit prio 1 # speed for client VIDEO CHAT 2 in Internet $tc class add dev $INT1 parent 1:6 classid 1:$hIDman_PUB$hIP htb rate 768kbit ceil 90Mbit prio 1 # speed for client VIDEO CHAT 2 in Metropolitan echo "$INT1: $NETWORK.$NET.$IP net (1:5): 1:$hIDnet_PUB$hIP min: 64kbit max: 256kbit man (1:6) 1:$hIDman_PUB$hIP min: 768kbit max: 90Mbit" >>~adminus/log/clase_eth1.log echo "CLIENTS done." echo " qdisc add INT2 root " $tc qdisc add dev $INT2 root handle 1: htb default FF01 $tc class add dev $INT2 parent 1: classid 1:1 htb rate 500kbit ceil 500kbit $tc class add dev $INT2 parent 1: classid 1:2 htb rate 95Mbit ceil 95Mbit $tc class add dev $INT2 parent 1:1 classid 1:7 htb rate 140kbit ceil 500kbit prio 0 # a/v net trafic $tc class add dev $INT2 parent 1:1 classid 1:5 htb rate 50kbit ceil 500kbit prio 0 # chat net trafic $tc class add dev $INT2 parent 1:1 classid 1:3 htb rate 100kbit ceil 500kbit prio 0 # www net trafic $tc class add dev $INT2 parent 1:2 classid 1:8 htb rate 35Mbit ceil 90Mbit prio 0 # a/v man trafic $tc class add dev $INT2 parent 1:2 classid 1:6 htb rate 5Mbit ceil 90Mbit prio 0 # chat man trafic $tc class add dev $INT2 parent 1:2 classid 1:4 htb rate 20Mbit ceil 90Mbit prio 0 # www man trafic $tc class add dev $INT2 parent 1:1 classid 1:FF01 htb rate 10kbit ceil 500kbit prio 3 # bulk net trafic $tc class add dev $INT2 parent 1:2 classid 1:FF00 htb rate 30Mbit ceil 90Mbit prio 3 # bulk man trafic $tc qdisc add dev $INT2 parent 1:FF01 handle 2: sfq perturb 10 $tc qdisc add dev $INT2 parent 1:FF00 handle 3: sfq perturb 10 echo " qdisc add INT2 ingress" $tc qdisc add dev $INT2 ingress #$tc filter add dev $INT2 parent FFFF: protocol ip prio 0 handle 0x7 fw flowid :1 police rate 10Mbps burst 16k drop # A/V in MAN #$tc filter add dev $INT2 parent FFFF: protocol ip prio 1 handle 0x5 fw flowid :1 police rate 10Mbps burst 16k drop # chat in MAN #$tc filter add dev $INT2 parent FFFF: protocol ip prio 2 handle 0x3 fw flowid :1 police rate 10Mbps burst 16k drop # www in MAN #$tc filter add dev $INT2 parent FFFF: protocol ip prio 4 handle 0x1 fw flowid :1 police rate 95Mbps burst 16k drop # bulk in MAN #$tc filter add dev $INT2 parent FFFF: protocol ip prio 0 handle 0x6 fw flowid :1 police rate 190kbps burst 16k drop # A/V in Internet #$tc filter add dev $INT2 parent FFFF: protocol ip prio 1 handle 0x4 fw flowid :1 police rate 62kbps burst 32k drop # chat in Internet #$tc filter add dev $INT2 parent FFFF: protocol ip prio 2 handle 0x2 fw flowid :1 police rate 126kbps burst 64k drop # www in Internet #$tc filter add dev $INT1 parent FFFF: protocol ip prio 3 u32 match ip dst 0.0.0.0/0 police rate 126kbit burst 1k drop flowid :1 # bulk in Internet ### POSTROUTING ### echo "POSTROUTING" echo "filters - CLASSIFY $EXT1 egress" $ipt -t mangle -F POSTROUTING $ipt -t mangle -A POSTROUTING -m mark --mark 0x7 -o $EXT1 -s 10.0.0.0/24 -j CLASSIFY --set-class 1:7 # A/V in MAN $ipt -t mangle -A POSTROUTING -m mark --mark 0x5 -o $EXT1 -s 10.0.0.0/24 -j CLASSIFY --set-class 1:5 # chat in MAN $ipt -t mangle -A POSTROUTING -m mark --mark 0x3 -o $EXT1 -s 10.0.0.0/24 -j CLASSIFY --set-class 1:3 # www in MAN $ipt -t mangle -A POSTROUTING -m mark --mark 0x6 -o $EXT1 -s 10.0.0.0/24 -j CLASSIFY --set-class 1:8 # A/V in Internet $ipt -t mangle -A POSTROUTING -m mark --mark 0x4 -o $EXT1 -s 10.0.0.0/24 -j CLASSIFY --set-class 1:6 # chat in Internet $ipt -t mangle -A POSTROUTING -m mark --mark 0x2 -o $EXT1 -s 10.0.0.0/24 -j CLASSIFY --set-class 1:4 # www in Internet $ipt -t mangle -A POSTROUTING -m mark --mark 0x0 -o $EXT1 -s 10.0.0.0/24 -j CLASSIFY --set-class 1:FF01 # bulk in Internet $ipt -t mangle -A POSTROUTING -m mark --mark 0x1 -o $EXT1 -s 10.0.0.0/24 -j CLASSIFY --set-class 1:FF00 # bulk in MAN echo "filters - CLASSIFY $INT1 egress";date >~adminus/log/filtre.log;echo "filters - CLASSIFY $INT1 egress" >>~adminus/log/filtre.log echo " bulk clients";echo " bulk clients" >>~adminus/log/filtre.log NETWORK=192.168;NET=101;NETID=16 # edit this after copy-paste (this row downto done must be copied for each served network) ID_NET=0;ID_MAN=128;ID_PRIV=0;ID_PUB=64 # don't edit # The first bit in class' MINOR is: 1 = metropolitan; 0 = Internet # The second bit in class' MINOR is: 1 = IP public; 0 = IP privat # Urmatorii 6 biti reprezinta NETID (class number) Atention: classes with MINOR from 1 to 6 are used by parents on $INT1, so NETID >= 7 !!! IDnet_PRIV=$[$ID_NET+$ID_PRIV+$NETID];IDman_PRIV=$[$ID_MAN+$ID_PRIV+$NETID];IDnet_PUB=$[$ID_NET+$ID_PUB+$NETID];IDman_PUB=$[$ID_MAN+$ID_PUB+$NETID] # don't edit hIDnet_PRIV=`printf "%x" $IDnet_PRIV`;hIDman_PRIV=`printf "%x" $IDman_PRIV`;hIDnet_PUB=`printf "%x" $IDnet_PUB`;hIDman_PUB=`printf "%x" $IDman_PUB` #don't edit for IP in $( cat ~adminus/etc/ip_local/priv_ips_on_int1.conf | grep -v \# ); do # if IP = { 0 1 2 3 4 5 6 7 8 9 a b c d e f A B C D E F }; then IP=0$IP; fi hNET=`printf "%x" $NET`;hIP=`printf "%x" $IP` $ipt -t mangle -A POSTROUTING -m mark --mark $MARK_NET -o $EXT1 -s $NETWORK.$NET.$IP -j CLASSIFY --set-class 1:$hIDnet_PRIV$hIP # IP privat in $EXT1 Internet $ipt -t mangle -A POSTROUTING -m mark --mark $MARK_MAN -o $EXT1 -s $NETWORK.$NET.$IP -j CLASSIFY --set-class 1:$hIDman_PRIV$hIP # IP privat in $EXT1 Metropolitan #$ipt -t mangle -A POSTROUTING -m mark --mark $MARK_NET -o $EXT2 -s $NETWORK.$NET.$IP -j CLASSIFY --set-class 1:$hIDnet_PRIV$hIP # IP privat in $EXT2 Internet #$ipt -t mangle -A POSTROUTING -m mark --mark $MARK_MAN -o $EXT2 -s $NETWORK.$NET.$IP -j CLASSIFY --set-class 1:$hIDman_PRIV$hIP # IP privat in $EXT2 Metropolitan $ipt -t mangle -A POSTROUTING -m mark --mark $MARK_NET -o $INT1 -d $NETWORK.$NET.$IP -j CLASSIFY --set-class 1:$hIDnet_PRIV$hIP # IP privat in $INT1 Internet $ipt -t mangle -A POSTROUTING -m mark --mark $MARK_MAN -o $INT1 -d $NETWORK.$NET.$IP -j CLASSIFY --set-class 1:$hIDman_PRIV$hIP # IP privat in $INT1 Metropolitan echo "$NETWORK.$NET.$IP $EXT1: net: 1:$hIDnet_PRIV$hIP man: 1:$hIDman_PRIV$hIP $INT1: net: 1:$hIDnet_PRIV$hIP man: 1:$hIDman_PRIV$hIP" >>~sorin/log/filtre.log done echo " special clients";echo " special clients" >>~sorin/log/filtre.log NETWORK=81.196;NET=157;NETID=63 # edit this after copy-paste (downto done is for every served network) ID_NET=0;ID_MAN=128;ID_PRIV=0;ID_PUB=64 # do not edit IDnet_PRIV=$[$ID_NET+$ID_PRIV+$NETID];IDman_PRIV=$[$ID_MAN+$ID_PRIV+$NETID];IDnet_PUB=$[$ID_NET+$ID_PUB+$NETID];IDman_PUB=$[$ID_MAN+$ID_PUB+$NETID] # do not edit hIDnet_PRIV=`printf "%x" $IDnet_PRIV`;hIDman_PRIV=`printf "%x" $IDman_PRIV`;hIDnet_PUB=`printf "%x" $IDnet_PUB`;hIDman_PUB=`printf "%x" $IDman_PUB` # do not edit for IP in $( cat ~adminus/etc/ip_local/pub_ips_on_int1.conf | grep -v \# ); do # if IP = { 0 1 2 3 4 5 6 7 8 9 a b c d e f A B C D E F }; then IP=0$IP; fi hNET=`printf "%x" $NET`;hIP=`printf "%x" $IP` $ipt -t mangle -A POSTROUTING -m mark --mark $MARK_NET -o $EXT1 -s $NETWORK.$NET.$IP -j CLASSIFY --set-class 1:$hIDnet_PUB$hIP # IP public in $EXT1 Internet $ipt -t mangle -A POSTROUTING -m mark --mark $MARK_MAN -o $EXT1 -s $NETWORK.$NET.$IP -j CLASSIFY --set-class 1:$hIDman_PUB$hIP # IP public in $EXT1 Metropolitan #$ipt -t mangle -A POSTROUTING -m mark --mark $MARK_NET -o $EXT2 -s $NETWORK.$NET.$IP -j CLASSIFY --set-class 1:$hIDnet_PUB$hIP #$ipt -t mangle -A POSTROUTING -m mark --mark $MARK_MAN -o $EXT2 -s $NETWORK.$NET.$IP -j CLASSIFY --set-class 1:$hID_man_PUB$hIP $ipt -t mangle -A POSTROUTING -m mark --mark $MARK_NET -o $INT1 -d $NETWORK.$NET.$IP -j CLASSIFY --set-class 1:$hIDnet_PUB$hIP # IP public in $INT1 Internet $ipt -t mangle -A POSTROUTING -m mark --mark $MARK_MAN -o $INT1 -d $NETWORK.$NET.$IP -j CLASSIFY --set-class 1:$hIDman_PUB$hIP # IP public in $INT1 Metropolitan echo "$NETWORK.$NET.$IP $EXT1: net: 1:$hIDnet_PUB$hIP man: 1:$hIDman_PUB$hIP $INT1: net: 1:$hIDnet_PUB$hIP man: 1:$hIDman_PUB$hIP" >>~sorin/log/filtre.log done echo "filters - CLASSIFY $INT2 egress" $ipt -t mangle -A POSTROUTING -m mark --mark 0x7 -o $INT2 -j CLASSIFY --set-class 1:7 $ipt -t mangle -A POSTROUTING -m mark --mark 0x5 -o $INT2 -j CLASSIFY --set-class 1:5 $ipt -t mangle -A POSTROUTING -m mark --mark 0x3 -o $INT2 -j CLASSIFY --set-class 1:3 $ipt -t mangle -A POSTROUTING -m mark --mark 0x6 -o $INT2 -j CLASSIFY --set-class 1:8 $ipt -t mangle -A POSTROUTING -m mark --mark 0x4 -o $INT2 -j CLASSIFY --set-class 1:6 $ipt -t mangle -A POSTROUTING -m mark --mark 0x2 -o $INT2 -j CLASSIFY --set-class 1:4 $ipt -t mangle -A POSTROUTING -m mark --mark 0x0 -o $INT2 -j CLASSIFY --set-class 1:FF01 $ipt -t mangle -A POSTROUTING -m mark --mark 0x1 -o $INT2 -j CLASSIFY --set-class 1:FF00 ### ### ### ### ### mangle ### ### ### ### ### ### PREROUTING ### $ipt -t mangle -F PREROUTING echo " creem MAN, QOS si CLIENT" $ipt -t mangle -X MAN $ipt -t mangle -X QOS $ipt -t mangle -N MAN $ipt -t mangle -N QOS $ipt -t mangle -Z MAN $ipt -t mangle -Z QOS $ipt -t mangle -A PREROUTING -j MAN $ipt -t mangle -A PREROUTING -j QOS ### QOS ### echo " TOS chat-ports" for PORT in $( cat ~sorin/etc/ports_qdisc_prio/chat_ports.conf | grep -v \# ); do $ipt -t mangle -A QOS -p tcp --dport $PORT -j TOS --set-tos Maximize-Reliability $ipt -t mangle -A QOS -p tcp --dport $PORT -j TOS --set-tos Minimize-Delay $ipt -t mangle -A QOS -p tcp --dport $PORT -j MARK --set-mark 0x4 $ipt -t mangle -A QOS -j CONNMARK --save-mark $ipt -t mangle -A QOS -p tcp --dport $PORT -j RETURN $ipt -t mangle -A QOS -p tcp --sport $PORT -j TOS --set-tos Maximize-Reliability $ipt -t mangle -A QOS -p tcp --sport $PORT -j TOS --set-tos Minimize-Delay $ipt -t mangle -A QOS -p tcp --sport $PORT -j MARK --set-mark 0x4 $ipt -t mangle -A QOS -j CONNMARK --save-mark $ipt -t mangle -A QOS -p tcp --sport $PORT -j RETURN $ipt -t mangle -A QOS -p udp --dport $PORT -j TOS --set-tos Maximize-Reliability $ipt -t mangle -A QOS -p udp --dport $PORT -j TOS --set-tos Minimize-Delay $ipt -t mangle -A QOS -p udp --dport $PORT -j MARK --set-mark 0x4 $ipt -t mangle -A QOS -j CONNMARK --save-mark $ipt -t mangle -A QOS -p udp --dport $PORT -j RETURN $ipt -t mangle -A QOS -p udp --sport $PORT -j TOS --set-tos Maximize-Reliability $ipt -t mangle -A QOS -p udp --sport $PORT -j TOS --set-tos Minimize-Delay $ipt -t mangle -A QOS -p udp --sport $PORT -j MARK --set-mark 0x4 $ipt -t mangle -A QOS -j CONNMARK --save-mark $ipt -t mangle -A QOS -p udp --sport $PORT -j RETURN done echo " TOS audio-video ports" for PORT in $( cat ~sorin/etc/ports_qdisc_prio/av_ports.conf | grep -v \# ); do $ipt -t mangle -A QOS -p tcp --dport $PORT -j TOS --set-tos Minimize-Delay $ipt -t mangle -A QOS -p tcp --dport $PORT -j TOS --set-tos Maximize-Throughput $ipt -t mangle -A QOS -p tcp --dport $PORT -j MARK --set-mark 0x6 $ipt -t mangle -A QOS -j CONNMARK --save-mark $ipt -t mangle -A QOS -p tcp --dport $PORT -j RETURN $ipt -t mangle -A QOS -p tcp --sport $PORT -j TOS --set-tos Minimize-Delay $ipt -t mangle -A QOS -p tcp --sport $PORT -j TOS --set-tos Maximize-Throughput $ipt -t mangle -A QOS -p tcp --sport $PORT -j MARK --set-mark 0x6 $ipt -t mangle -A QOS -j CONNMARK --save-mark $ipt -t mangle -A QOS -p tcp --sport $PORT -j RETURN $ipt -t mangle -A QOS -p udp --dport $PORT -j TOS --set-tos Minimize-Delay $ipt -t mangle -A QOS -p udp --dport $PORT -j TOS --set-tos Maximize-Throughput $ipt -t mangle -A QOS -p udp --dport $PORT -j MARK --set-mark 0x6 $ipt -t mangle -A QOS -j CONNMARK --save-mark $ipt -t mangle -A QOS -p udp --dport $PORT -j RETURN $ipt -t mangle -A QOS -p udp --sport $PORT -j TOS --set-tos Minimize-Delay $ipt -t mangle -A QOS -p udp --sport $PORT -j TOS --set-tos Maximize-Throughput $ipt -t mangle -A QOS -p udp --sport $PORT -j MARK --set-mark 0x6 $ipt -t mangle -A QOS -j CONNMARK --save-mark $ipt -t mangle -A QOS -p udp --sport $PORT -j RETURN done echo " TOS www ports" for PORT in $( cat ~sorin/etc/ports_qdisc_prio/www_ports.conf | grep -v \# ); do $ipt -t mangle -A QOS -p tcp --dport $PORT -j TOS --set-tos Maximize-Throughput $ipt -t mangle -A QOS -p tcp --dport $PORT -j MARK --set-mark 0x2 $ipt -t mangle -A QOS -j CONNMARK --save-mark $ipt -t mangle -A QOS -p tcp --dport $PORT -j RETURN $ipt -t mangle -A QOS -p tcp --sport $PORT -j TOS --set-tos Maximize-Throughput $ipt -t mangle -A QOS -p tcp --sport $PORT -j MARK --set-mark 0x2 $ipt -t mangle -A QOS -j CONNMARK --save-mark $ipt -t mangle -A QOS -p tcp --sport $PORT -j RETURN $ipt -t mangle -A QOS -p udp --dport $PORT -j TOS --set-tos Maximize-Throughput $ipt -t mangle -A QOS -p udp --dport $PORT -j MARK --set-mark 0x2 $ipt -t mangle -A QOS -j CONNMARK --save-mark $ipt -t mangle -A QOS -p udp --dport $PORT -j RETURN $ipt -t mangle -A QOS -p udp --sport $PORT -j TOS --set-tos Maximize-Throughput $ipt -t mangle -A QOS -p udp --dport $PORT -j MARK --set-mark 0x2 $ipt -t mangle -A QOS -j CONNMARK --save-mark $ipt -t mangle -A QOS -p udp --sport $PORT -j RETURN done echo " TOS tcp flags" $ipt -t mangle -A QOS -p tcp --tcp-flags ALL, ALL -j MARK --set-mark 0x6 $ipt -t mangle -A QOS -p tcp --tcp-flags ALL, ALL -j TOS --set-tos Minimize-Delay $ipt -t mangle -A QOS -p tcp --tcp-flags ALL, ALL -j TOS --set-tos Maximize-Throughput $ipt -t mangle -A QOS -p tcp --tcp-flags ALL, ALL -j TOS --set-tos Maximize-Reliability $ipt -t mangle -A QOS -j CONNMARK --save-mark $ipt -t mangle -A QOS -p tcp --tcp-flags ALL, ALL -j RETURN $ipt -t mangle -A QOS -j CONNMARK --save-mark $ipt -t mangle -A QOS -p ALL -j RETURN ### MAN ### echo " MAN mark man (order a pizza and eat till I finish this)" for PEER_IP in $( cat ~sorin/etc/ip_internet/peer_ips.conf | grep -v \# ); do $ipt -t mangle -A MAN -d $PEER_IP -j MARK --set-mark $MARK_MAN $ipt -t mangle -A MAN -j CONNMARK --restore-mark --mask 0xfffe $ipt -t mangle -A MAN -d $PEER_IP -j RETURN $ipt -t mangle -A MAN -s $PEER_IP -j MARK --set-mark $MARK_MAN $ipt -t mangle -A MAN -j CONNMARK --restore-mark --mask 0xfffe $ipt -t mangle -A MAN -s $PEER_IP -j RETURN done echo " MAN mark net" $ipt -t mangle -A MAN -d 0.0.0.0/0 -j MARK --set-mark $MARK_NET $ipt -t mangle -A MAN -j CONNMARK --restore-mark --mask 0xfffe $ipt -t mangle -A MAN -d 0.0.0.0/0 -j RETURN $ipt_s >~adminus/bin/marks ### POSTROUTING ### if [ -x /mnt/usb/tc-restore ]; then /mnt/usb/tc-restore cp /mnt/usb/tc-restore ~sorin/bin/ else ~sorin/bin/tc-restore fi # each IP has its own class ### ### ### ### ### filter ### ### ### ### ### ### INPUT ### echo "INPUT" # TODO: Use ~adminus/etc/ports_input_allowed, use -m mport --port for both direction ports if they *ARE* equal $ipt -t filter -P INPUT DROP $ipt -t filter -A INPUT -i lo -j ACCEPT $ipt -t filter -A INPUT -p tcp --sport 0:1023 -m state --state ESTABLISHED,RELATED -j ACCEPT $ipt -t filter -A INPUT -i lo -j ACCEPT $ipt -t filter -A INPUT -p tcp --tcp-flags ACK ACK -j ACCEPT $ipt -t filter -A INPUT -m state --state ESTABLISHED -j ACCEPT $ipt -t filter -A INPUT -m state --state RELATED -j ACCEPT $ipt -t filter -A INPUT -p udp --dport 1024:65535 --sport 53 -j ACCEPT $ipt -t filter -A INPUT -p icmp --icmp-type echo-reply -j ACCEPT $ipt -t filter -A INPUT -p icmp --icmp-type destination-unreachable -j ACCEPT $ipt -t filter -A INPUT -p icmp --icmp-type source-quench -j ACCEPT $ipt -t filter -A INPUT -p icmp --icmp-type time-exceeded -j ACCEPT $ipt -t filter -A INPUT -p icmp --icmp-type parameter-problem -j ACCEPT $ipt -t filter -A INPUT -p tcp -m state ! --state NEW --sport 0:1023 -j ACCEPT $ipt -t filter -A INPUT -p udp --sport 0:1023 -j ACCEPT $ipt -t filter -A INPUT -p tcp --dport ssh -j ACCEPT $ipt -t filter -A INPUT -p tcp --dport auth -j ACCEPT $ipt -t filter -A INPUT -p tcp --dport ftp -j ACCEPT $ipt -t filter -A INPUT -p tcp --dport rmt -j ACCEPT $ipt -t filter -A INPUT -p udp --dport rmt -j ACCEPT $ipt -t filter -A INPUT -p tcp --dport ftp-data -j ACCEPT $ipt -t filter -A INPUT -p udp --dport time -j ACCEPT $ipt -t filter -A INPUT -p tcp --dport http -j ACCEPT $ipt -t filter -A INPUT -p icmp -m limit --icmp-type echo-request --limit 3/second --limit-burst 1000 -j ACCEPT $ipt -t filter -A INPUT -p tcp ! -i lo --sport 2049:2050 -j DROP $ipt -t filter -A INPUT -p tcp ! -i lo --dport 2049:2050 -j DROP $ipt -t filter -A INPUT -p tcp ! -i lo --sport 6000:6063 -j DROP $ipt -t filter -A INPUT -p tcp ! -i lo --dport 6000:6063 -j DROP $ipt -t filter -A INPUT -p tcp ! -i lo --sport 7000:7010 -j DROP $ipt -t filter -A INPUT -p tcp ! -i lo --dport 7000:7010 -j DROP $ipt -t filter -A INPUT -p tcp --sport 1024:65535 -j ACCEPT $ipt -t filter -A INPUT -p tcp --dport 1024:65535 -j ACCEPT $ipt -t filter -A INPUT -p udp --sport 1024:65535 -j ACCEPT $ipt -t filter -A INPUT -p udp --dport 1024:65535 -j ACCEPT ### FORWARD ### echo "FORWARD" $ipt -t filter -P FORWARD DROP $ipt -t filter -A FORWARD -i lo -j ACCEPT $ipt -t filter -A FORWARD -o lo -j ACCEPT echo " ip/mac ACCEPT" ~sorin/bin/mac.sh $ipt -t filter -A FORWARD -o $INT1 -d $INT1Net/$INT1Mask -j ACCEPT $ipt -t filter -A FORWARD -i $INT2 -s $INT2Net/$INT2Mask -j ACCEPT $ipt -t filter -A FORWARD -o $INT2 -d $INT2Net/$INT2Mask -j ACCEPT $ipt -t filter -A FORWARD -i $EXT1 -o $INT1 -j ACCEPT $ipt -t filter -A FORWARD -i $EXT1 -o $INT2 -m state --state ESTABLISHED,RELATED -j ACCEPT $ipt -t filter -A FORWARD -i $INT1 -o $INT2 -j ACCEPT $ipt -t filter -A FORWARD -i $INT2 -o $INT1 -j ACCEPT #$ipt -t filter -A FORWARD -i $INT1 -o $EXT1 -j ACCEPT # Se face pe mac address $ipt -t filter -A FORWARD -i $INT2 -o $EXT1 -j ACCEPT echo " connection/port ACCEPT/DROP" #$ipt -t filter -A FORWARD -f -j ACCEPT $ipt -t filter -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu #$ipt -t filter -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 128 $ipt -t filter -A FORWARD -i ! $INT1 -s $INT1Net/$INT1Mask -j DROP $ipt -t filter -A FORWARD -i ! $INT2 -s $INT2Net/$INT2Mask -j DROP $ipt -t filter -A FORWARD -p icmp -d $INT1Bcast -j DROP $ipt -t filter -A FORWARD -p icmp -d $INT2Bcast -j DROP $ipt -t filter -A FORWARD -p tcp --syn -m limit --limit 10/s -j ACCEPT $ipt -t filter -A FORWARD -p tcp --tcp-flags SYN,ACK,FIN,RST RST -m limit --limit 10/s -j ACCEPT $ipt -t filter -A FORWARD -p icmp --icmp-type echo-request -m limit --limit 3/s -j ACCEPT $ipt -t filter -A FORWARD -p icmp --icmp-type echo-reply -m limit --limit 3/s -j ACCEPT $ipt -t filter -A FORWARD -p udp --sport 53 -j ACCEPT $ipt -t filter -A FORWARD -p udp --dport 53 -j ACCEPT $ipt -t filter -A FORWARD -p tcp --dport 139 -j DROP $ipt -t filter -A FORWARD -p tcp --dport 445 -j DROP ### OUTPUT ### echo "OUTPUT" $ipt -t filter -P OUTPUT ACCEPT echo "Preparing for reboot... (iptables-save)" /usr/sbin/iptables-save >/home/adminus/iptables A/V ports:531 554 583 7070 1754:1755 1397:1398 1516 1518 2232 4444 5555 5713:5714 6000 6010 CHAT ports: 53 5050 1863 113 529 994 6660:6667 7000 63 5190:5193 22 23 992 37 123 21 990 1517 1519 2103:2105 5222 5269 5715:5717 WWW ports (and games): 80 443 280 488 25 109:110 995 143 220 993 516 532 563 631 901 666 4557 4559 27005 27015 __________________________________ Do you Yahoo!? Yahoo! Mail - You care about security. So do we. http://promotions.yahoo.com/new_mail From ji.li3 at hp.com Wed Aug 10 05:28:11 2005 From: ji.li3 at hp.com (Li, Ji) Date: Wed Aug 10 05:28:18 2005 Subject: [LARTC] teql on virtual network interfaces ? Message-ID: <628BFCE8B64706469FE4D4852CEC953706DFB395@tayexc14.americas.cpqcorp.net> Hi all, I want to implement a "weighted" teql that can send packets to interfaces based on their "weights". To do this, I want to create multiple virtual network interfaces, and add them to teql. Since teql will send packets to each interface in a round-robin way, a weighted teql is achieved (my guess). I already have two physical network interfaces, eth0 and eth1. I created a virtual interfaces based on eth1: # ifconfig eth1:0 10.0.0.2 After that I can see eth1:0 in the "ifconfig" result. Then I try to add the virtual interface to teql0: # tc qdisc add dev eth1:0 root teql0 But I got the error message: Cannot find device "eth1:0" What's wrong with my method? I didn't see ip_alias module in the "lsmod", but since I can see eth1:0 in "ifconfig", I assume the module is loaded in the kernel. Finally, is there any existing "weighted" teql implemented? I am using Fedore Core 3 (2.6.9-1.667smp) Thanks a lot! Best, -Ji -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ds9a.nl/pipermail/lartc/attachments/20050809/15fabc2e/attachment.htm From jlynch at frink.nuigalway.ie Wed Aug 10 20:37:48 2005 From: jlynch at frink.nuigalway.ie (Jonathan Lynch) Date: Wed Aug 10 20:38:17 2005 Subject: [LARTC] HTB and PRIO qdiscs introducing extra latency when output interface is saturated In-Reply-To: <42F0CED8.1090406@dsl.pipex.com> References: <1122460110.8454.11.camel@pgala.it.nuigalway.ie> <42E78B39.1030201@dsl.pipex.com> <1122478662.4637.49.camel@pgala.it.nuigalway.ie> <42E80247.7010100@dsl.pipex.com> <1122568638.7828.40.camel@pgala.it.nuigalway.ie> <42E95302.2020303@dsl.pipex.com> <1123016381.4768.35.camel@pgala.it.nuigalway.ie> <42F0CED8.1090406@dsl.pipex.com> Message-ID: <1123699068.4768.67.camel@pgala.it.nuigalway.ie> Andy, thanks for all the feedback. I was away on holidays for the last week and am only back today. I have a few more questions which are listed below. On Wed, 2005-08-03 at 15:04 +0100, Andy Furniss wrote: > Jonathan Lynch wrote: > > I did the same tests that I outlined earlier, but this time by setting > > hysteresis to 0. The config for the core router is included at the > > bottom. The graphs for the delay of the voip stream and the traffic > > going through the core router can be found at the following addresses. > > > > http://140.203.56.30/~jlynch/htb/core_router_hysteresis.png > > http://140.203.56.30/~jlynch/htb/voip_stream_24761_hysteresis.png > > > > > > The max delay of the stream has dropped to 1.8ms. Again the jitter seems > > to be around 1ms. There seems to be a pattern going whereby the delay > > reaches about 1.6ms then drops back to 0.4 ms, jumps back to 1.6ms and > > then back to 0.4ms repeatedly and then it rises from 0.5ms gradually and > > repeats this behaviour. Is there any explanation to this pattern ? > > > > Would it have anything go to do with burst being 1ms ? > > Yes I suppose if you could sample truly randomly you would get a proper > distribution - I guess the pattern arises because your timers are > synchronised for the test. I dont understand what you mean when you say "if you could sample truly randomly you would get a proper distribution". Also having the timers synchronized will allow for more accurate measurements of the delay. I cant see how this would have an impact on the pattern. > > > > When the ceil is specified as being 90mbit, is this at IP level ? > > What does this correspond to when a Mbit = 1,000,000 bits. Im a bit > > confused with the way tc interprets this rate. > > Yes htb uses ip level length (but you can specify overhead & min size) , > the rate calculations use a lookup table which is likely to have a > granularity of 8 bytes (you can see this with tc -s -d class ls .. look > for /8 after the burst/cburst). > > There is a choice in 2.6 configs about using CPU/jiffies/gettimeofday - > I use CPU and now I've got a ping that does < 1 sec I get the same > results as you. > I have the default setting which is to set it to jiffies. There is a comment in the kernal config for Packet scheduler clock source that mentions that Jiffies "its resolution is too low for accurate shaping except at very low speed". I will recompile the kernel and try the CPU option tomorrow to see if there is any change. > > > > If the ceil is based at IP level then the max ceil is going to be a > > value between 54 Mbit and 97 Mbit (not the tc values) for a 100 Mbit > > interface depending on the size of the packets passing through, right ? > > > > Minimum Ethernet frame > > 148,809 * (46 * 8) = 148,809 * 368 = 54,761,712 Mbps > > > > Maximum Ethernet frame > > 8,127 * (1500 * 8) = 8,127 * 12,000 = 97,524,000 Mbps > > If you use the overhead option I think you will be to overcome this > limitation and push the rates closer to 100mbit. > > > > About the red settings, I dont understand properly how to configure the > > settings. I was using the configuration that came with the examples. > > I don't use red it was just something I noticed - maybe making it longer > would help, maybe my test wasn't rerpresentative. > > FWIW I had a play around with HFSC (not that I know what I am doing > really) and at 92mbit managed to get - > > rtt min/avg/max/mdev = 0.330/0.414/0.493/0.051 ms loaded > from > rtt min/avg/max/mdev = 0.114/0.133/0.187/0.028 ms idle > > and that was through a really cheap switch. > > Andy. > looked up ethernet overheads and found the figure of 38 bytes per > frame, the 46 is min eth payload size? and looking at the way mpu is > handled by the tc rate table generator I think you would need to use > 46 > + 38 as mpu. > > So on every htb line that has a rate put ..... overhead 38 mpu 84 > > I haven't checked those figures or tested close to limits though, the > 12k burst would need increasing a bit aswell or that will slightly > over > limit rate at HZ=1000. > > > > > I haven't checked those figures or tested close to limits though, the > 12k burst would need increasing a bit aswell or that will slightly over > limit rate at HZ=1000. > > It seems that htb still uses ip level for burst so 12k is enough. > > With the overhead at 38 I can ceil at 99mbit OK. I didnt realise such options existed for htb (mpu + overhead). These parameters are not mentioned in the man pages or in the htb manual. I presume I have to patch tc to get these features ?. Yep 46 is the minimum eth payload size and 38 is the min overhead for ethernet frames. interframe gap 96bits 12 bytes +preamble 56bits 7 bytes +sfd 8bits 1 byte +eth header 14 bytes +crc 4 bytes --------- 38 bytes overhead per ethernet frame. Jonathan From andy.furniss at dsl.pipex.com Thu Aug 11 18:10:44 2005 From: andy.furniss at dsl.pipex.com (Andy Furniss) Date: Thu Aug 11 18:10:41 2005 Subject: [LARTC] Too slow computer? In-Reply-To: <20050809213122.80048.qmail@web32606.mail.mud.yahoo.com> References: <20050809213122.80048.qmail@web32606.mail.mud.yahoo.com> Message-ID: <42FB7884.3040502@dsl.pipex.com> panca sorin wrote: > Thanck you for your help! > I noticed the ipset tools and I tried to use the > CONNMARK but I don't know how to verify if bitwise > manipulation works. The IP list is random and the > router is an Athlon at 1200 MHz with 64 MB of SDRAM > and a PIO mode 4 harddisk. > After marking for destination, the packets are marked > for priorization. I tried to use the dsmark and some > ingress policing but I've faild to understand how they > work. Also I'm in a hurry and I try to use what I know > for now. Since I have to shape for two speeds, now > I've discovered the --limit filter in iptables and I > try to match packets based on their speeds. > Each connected client has its own class on dev eth1. > There are 38 clients now. On eth2 I shape based on > connection ports. Audio/video, chat and interactive > traffic (and connection control packets) have higher > priority. Here are my script and configuration files > (is best viewd unwraped with kwrite): That's a big script - I haven't had time to read it properly and I'd still be likely to miss things :-) Ingress policers won't work with fw if your kernel config has packet actions selected. If you don't it will work but only with marks set in prerouting. Bitwise manipulation of normal marks should work for recent iptables just remember to use 0x as it uses decimal otherwise. If you want to test just make an empty match and look at the counters. So to set bit 2 of the mark use --or-mark 0x2 instead of --set-mark. I've never used ipset but it seems suited to what you need. If you choose to use mark/connmark then you can get htb to treat marks like classify - you just put an empty fw on the root and have to make sure the marks have the major id in the top 16bits and you have a class for the minor. Andy. From andy.furniss at dsl.pipex.com Thu Aug 11 18:36:10 2005 From: andy.furniss at dsl.pipex.com (Andy Furniss) Date: Thu Aug 11 18:36:09 2005 Subject: [LARTC] HTB and PRIO qdiscs introducing extra latency when output interface is saturated In-Reply-To: <1123699068.4768.67.camel@pgala.it.nuigalway.ie> References: <1122460110.8454.11.camel@pgala.it.nuigalway.ie> <42E78B39.1030201@dsl.pipex.com> <1122478662.4637.49.camel@pgala.it.nuigalway.ie> <42E80247.7010100@dsl.pipex.com> <1122568638.7828.40.camel@pgala.it.nuigalway.ie> <42E95302.2020303@dsl.pipex.com> <1123016381.4768.35.camel@pgala.it.nuigalway.ie> <42F0CED8.1090406@dsl.pipex.com> <1123699068.4768.67.camel@pgala.it.nuigalway.ie> Message-ID: <42FB7E7A.5040301@dsl.pipex.com> Jonathan Lynch wrote: > > I dont understand what you mean when you say "if you could sample truly > randomly you would get a proper distribution". > > Also having the timers synchronized will allow for more accurate > measurements of the delay. I cant see how this would have an impact on > the pattern. I mean it's possibly just to do with the test if a 0ms - 1ms delay is expected then you could see patterns arising depending on how you measure delay/clock drift or something. Now I have two pings that do intervels < 1 sec - the inetutils GNU ping guys implemented it for me :-), and I also have the iputils one I can simulate a stream better. While doing this I noticed that iputils ping actually gives lower latency readings when sending many pps. Using tcpdump deltas I can see the network latency is the same however many pps I do - it's just that when measuring <1ms delays and doing many pps it seems that some code gets cached (guess) and the reported delay changes as a result. I mention that just to illustrate that measuring small delays can be misleading and influenced by the exact nature of your setup. > > I have the default setting which is to set it to jiffies. There is a > comment in the kernal config for Packet scheduler clock source that > mentions that Jiffies "its resolution is too low for accurate shaping > except at very low speed". I will recompile the kernel and try the CPU > option tomorrow to see if there is any change. Maybe not in the case of htb - I use CPU and see similar results, the comment about accurate shaping was probably written when HZ=100, but I suppose it will be better for something :-) > > I didnt realise such options existed for htb (mpu + overhead). These parameters are not mentioned in the man pages or in the htb manual. > I presume I have to patch tc to get these features ?. There is mention on the htb page - it was added as a patch so was not designed in, which explains why burst doesn't use it. You don't need to patch recent iproute2 it's already in there. Andy. From psihozefir at yahoo.com Sat Aug 13 13:50:24 2005 From: psihozefir at yahoo.com (panca sorin) Date: Sat Aug 13 13:50:29 2005 Subject: [LARTC] pfifo_fast as leaf qdisc for htb classes Message-ID: <20050813115024.88441.qmail@web32608.mail.mud.yahoo.com> Thank you Andy! I did't find why I could't use the fw match in ingress. Now it works. Now I ran into another problem: When I try to add a pfifo_fast leaf qdisc dor htb classes it seg.-faults. Is pfifo_fast a special qdisc? I can succesfully add other type of qdiscs. I found that a similar problem was reported in february on a MIPS architecture. root@zefir:/usr/src# tc qdisc add dev eth0 root handle 1: htb default 10 root@zefir:/usr/src# tc class add dev eth0 parent 1: classid 1:1 htb rate 100Mbit ceil 100Mbit burst 50k cburst 150k root@zefir:/usr/src# tc class del dev eth0 parent 1: classid 1:1 htb rate 100Mbit ceil 100Mbit burst 50k cburst 150k root@zefir:/usr/src# tc class add dev eth0 parent 1: classid 1:1 htb rate 90Mbit ceil 100Mbit burst 50k cburst 150k root@zefir:/usr/src# tc class add dev eth0 parent 1: classid 1:10 htb rate 10Mbit ceil 100Mbit burst 50k cburst 150k root@zefir:/usr/src# tc qdisc add dev eth0 parent 1:1 handle 2: pfifo_fast Segmentation fault root@zefir:/usr/src# uname -a Linux zefir 2.6.12.4 #1 Sun Aug 7 18:17:31 EEST 2005 i686 unknown unknown GNU/Linux root@zefir:/usr/src# tc qdisc add dev eth0 parent 1:1 handle 2: sfq perturb 10 root@zefir:/usr/src# I think htb and pfifo_fast assures a better QOS than htb & sfq because I can priorize interactivity. Thank you in advance for any sugestions. __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From andy.furniss at dsl.pipex.com Sun Aug 14 20:13:34 2005 From: andy.furniss at dsl.pipex.com (Andy Furniss) Date: Sun Aug 14 20:13:22 2005 Subject: [LARTC] pfifo_fast as leaf qdisc for htb classes In-Reply-To: <20050813115024.88441.qmail@web32608.mail.mud.yahoo.com> References: <20050813115024.88441.qmail@web32608.mail.mud.yahoo.com> Message-ID: <42FF89CE.8020500@dsl.pipex.com> panca sorin wrote: > Thank you Andy! I did't find why I could't use the fw > match in ingress. Now it works. It will work on marks set in prerouting if you don't select packet action in your kernel config (near policer). > > Now I ran into another problem: > When I try to add a pfifo_fast leaf qdisc dor htb > classes it seg.-faults. Is pfifo_fast a special qdisc? > I can succesfully add other type of qdiscs. > > I found that a similar problem was reported in > february on a MIPS architecture. Yes I think it's special - but the default behavior of prio is much the same. > > root@zefir:/usr/src# tc qdisc add dev eth0 root handle > 1: htb default 10 > root@zefir:/usr/src# tc class add dev eth0 parent 1: > classid 1:1 htb rate 100Mbit ceil 100Mbit burst 50k > cburst 150k FWIW if you are shaping 100meg ethernet those settings would be too high I would use - ... rate 99mbit ceil 99mbit mpu 84 overhead 38 ... > I think htb and pfifo_fast assures a better QOS than > htb & sfq because I can priorize interactivity. > > Thank you in advance for any sugestions. Yes use prio if you are going to give one class for each user, you will still have to set the tos bits/classify yourself as you can't rely on apps/inbound having them set properly. Also remember that if you are shaping inbound traffic you need to back off from the link rates more than outbound so that a queue can build up. If the link is slow then you'll need to keep the queues short aswell, so you drop packets. Andy. From kajtek at biezanow.net Mon Aug 15 15:58:14 2005 From: kajtek at biezanow.net (Kajetan Staszkiewicz) Date: Mon Aug 15 15:58:16 2005 Subject: [LARTC] imq custom device numbers Message-ID: <200508151558.14581.kajtek@biezanow.net> I made some changes to IMQ, so it is now possible to have custom numbers for IMQ devices. Useful when working with VLANs. You can get it at http://tuxpowered.net/. Readme is on the website. -- | pozdrawiam / greetings | powered by Trustix, Gentoo and FreeBSD | | Kajetan Staszkiewicz | JID: vegeta@chrome.pl | | Vegeta | IMQ devnames: http://tuxpowered.net | +------------------------+----------------------------------------+ From andre.correa at pobox.com Mon Aug 15 20:34:13 2005 From: andre.correa at pobox.com (Andre D. Correa) Date: Mon Aug 15 20:34:23 2005 Subject: [LARTC] XML for tc hierarchy representation Message-ID: <4300E025.9030408@pobox.com> Hi list, I would like to know about any proposed XML representation for tc objects hierarchy. I found something at "LTCM, a Linux QoS API Library" (http://artemis.av.it.pt/~ltcmmm/) that looks like a start, but any others are welcome. I'm starting a project to automatically syncronize traffic shape rules between servers and thought XML is the best way to represent tc hierarchy. Tks in advance for any information. -------------------------------------------------------------------- Andre D. Correa, CISSP | Visite meus projetos pessoais: andre.correa (at) pobox.com | Visit my personal projects: http://andre.hiperlinks.com.br | -http://malware.hiperlinks.com.br Sao Paulo / SP / Brazil | -http://www.linuximq.net/ -------------------------------------------------------------------- From andre.correa at pobox.com Mon Aug 15 22:11:18 2005 From: andre.correa at pobox.com (Andre D. Correa) Date: Mon Aug 15 22:11:27 2005 Subject: [LARTC] XML for tc hierarchy representation In-Reply-To: <1124133989.4843.59.camel@OSCARLAPLIN> References: <4300E025.9030408@pobox.com> <1124133989.4843.59.camel@OSCARLAPLIN> Message-ID: <4300F6E6.6060901@pobox.com> In fact I'm just search for an XML representation of tc hierarchy. Although I could design my own, using a reviewed and "widely accepted" schema makes more sense to me. I'm not testing or planning to use LTCM. Tks anyway... Andre Shane O'Hanlon wrote: > I just tried to get that code working some of the test programs worked > some did not I could not find a versioning file. Have you had more > success in getting it to work. I would like to get it too work. > > Kernel 2.6.11.7 > > Here is the output of the test programs some failed some succeeded but I > am sure its just my libs are not the correct version > > output of e00 > created dev_interface for eth0 > pfifo qdisc { > limit: 0 > } > > output of e01 > created dev_interface for eth0 > failure: netlink failure > > output of e02 > created dev_interface for eth0 > failure: netlink failure > > output of e03 > created dev_interface for eth0 > failure: netlink failure > > output of e04 > created dev_interface for eth0 > qdisc is of type pfifo > > output of e05 > failure: netlink failure > > output of e06 > fifo qdisc stars with limit = 100 > failure: netlink failure > > output of e07 > creating root qdisc > failure: netlink failure > > output of e08 > qdisc 1:0 > > output of e09 > creating root qdisc > failure: netlink failure > > output of e10 > creating root qdisc > failure: netlink failure > > output of e11 > > output of e12 > created dev_interface for eth0 > failure: system reply: invalid arguments > > > > On Mon, 2005-08-15 at 15:34 -0300, Andre D. Correa wrote: > >>Hi list, I would like to know about any proposed XML representation for >>tc objects hierarchy. I found something at "LTCM, a Linux QoS API >>Library" (http://artemis.av.it.pt/~ltcmmm/) that looks like a start, but >>any others are welcome. >> >>I'm starting a project to automatically syncronize traffic shape rules >>between servers and thought XML is the best way to represent tc hierarchy. >> >>Tks in advance for any information. >> >>-------------------------------------------------------------------- >>Andre D. Correa, CISSP | Visite meus projetos pessoais: >>andre.correa (at) pobox.com | Visit my personal projects: >>http://andre.hiperlinks.com.br | -http://malware.hiperlinks.com.br >>Sao Paulo / SP / Brazil | -http://www.linuximq.net/ >>-------------------------------------------------------------------- >> >>_______________________________________________ >>LARTC mailing list >>LARTC@mailman.ds9a.nl >>http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > > > -------------------------------------------------------------------- Andre D. Correa, CISSP | Visite meus projetos pessoais: andre.correa (at) pobox.com | Visit my personal projects: http://andre.hiperlinks.com.br | -http://malware.hiperlinks.com.br Sao Paulo / SP / Brazil | -http://www.linuximq.net/ -------------------------------------------------------------------- From mihaivlad at web-profile.net Mon Aug 15 22:53:38 2005 From: mihaivlad at web-profile.net (Mihai Vlad) Date: Mon Aug 15 22:53:39 2005 Subject: [LARTC] Hardware Configuration Ideas Message-ID: Hey guys, I am planning to buy some components for a Linux router that will handle the Internet access of 200 computers (includes tc shaping) and some inter sub-network routing (at least 100MBps per eth - and there are 3 eth cards). I was thinking of a: Pentium 4 - 3GHz 256 or 512MB RAM Network Cards. Now - I wonder what is more important: the processor speed or the amount of RAM. And can you point me to some good Network Cards that you have used and are not so expensive. Some Intel, etc. I have no clue about this subject... Maybe this discussion can be extended to a list of best practices to set up a performant Linux Router from the hardware point of view. Thanks in advance, Mihai From thomasheinz at gmx.net Mon Aug 15 23:50:52 2005 From: thomasheinz at gmx.net (Thomas Heinz) Date: Mon Aug 15 23:51:41 2005 Subject: [LARTC] Unsubsription impossible Message-ID: <200508152351.10428.thomasheinz@gmx.net> Hi Several times, I unsuccessfully tried to unsubscribe from lartc via the web interface and mailto:lartc-request@mailman.ds9a.nl?subject=unsubscribe. I also contacted the mailing list admin but got no reply. Even resubscribing did not work. I have posted here since I am running out of options. This posting is addressed to the list admin in charge: Please remove my email address from the lartc mailing list. Thanks. Regards, Thomas -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mailman.ds9a.nl/pipermail/lartc/attachments/20050815/5407afde/attachment.bin From mikeeo at email.msn.com Tue Aug 16 00:17:00 2005 From: mikeeo at email.msn.com (Mike O) Date: Tue Aug 16 00:17:01 2005 Subject: [LARTC] Unsubsription impossible References: <200508152351.10428.thomasheinz@gmx.net> Message-ID: I too wish to unsubscribe. Several attempts have been unsuccessful. Thanks ----- Original Message ----- From: "Thomas Heinz" To: Sent: Monday, August 15, 2005 5:50 PM Subject: [LARTC] Unsubsription impossible > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > From ms at calivia.com Tue Aug 16 00:17:48 2005 From: ms at calivia.com (Michael Steinmann) Date: Tue Aug 16 00:20:01 2005 Subject: [LARTC] Unsubsription impossible In-Reply-To: <200508152351.10428.thomasheinz@gmx.net> References: <200508152351.10428.thomasheinz@gmx.net> Message-ID: <4301148C.6030304@calivia.com> Thomas Heinz wrote: > Hi > > Several times, I unsuccessfully tried to unsubscribe from lartc via the web > interface and mailto:lartc-request@mailman.ds9a.nl?subject=unsubscribe. I > also contacted the mailing list admin but got no reply. Even resubscribing > did not work. > > I have posted here since I am running out of options. This posting is > addressed to the list admin in charge: > Please remove my email address from the lartc mailing list. > > Thanks. > > > Regards, > > Thomas ...and I thought I was the only one having trouble. please unsubscribe me as well. thanks, Mike > > ------------------------------------------------------------------------ > > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc From cyberdoc at cyberdoc.dk Tue Aug 16 06:11:26 2005 From: cyberdoc at cyberdoc.dk (Daniel Frederiksen) Date: Tue Aug 16 06:11:39 2005 Subject: [LARTC] Multipath Routing.. Message-ID: <4301676E.4070100@cyberdoc.dk> Ok folks, here goes.. I have been boggling with a problem for the past week, and still haven't found a solution.. I'm trying to route traffic from two providers through a Linux machine. But that is not the problem. The ISP's have provided me with a WAN IP class for both of the lines, to be routed into a DMZ where the machines a to respond to their respective designated WAN IP on both lines. Every machine on the DMZ has two IP's one on each ISP WAN Class. I think I'll better draw a map: WAN1(eth2), WAN2(eth3) --------- (eth0) | |-----\ ---------- | DMZ |---\ \ /---| ISP1 |----- --------- \ \ / ---------- \ \ \ / \ ----------------- | Linux GW/FW | WWW ----------------- / \ (eth1) / --------- / \ ---------- / | LAN |---/ \---| ISP2 |----- --------- ---------- NAT(eth4) The DMZ has two WAN IP classes routed from the ISP. The thing I just can not figure out is how to make the respective WAN IP from the DMZ route out the right ISP link, and the right request from the ISP route into the DMZ. .. and finally how can I make the LAN able to access it all.. Thanks for your time.. /Daniel Frederiksen From alex at samad.com.au Tue Aug 16 10:39:38 2005 From: alex at samad.com.au (Alexander Samad) Date: Tue Aug 16 10:39:50 2005 Subject: [LARTC] Multipath Routing.. In-Reply-To: <4301676E.4070100@cyberdoc.dk> References: <4301676E.4070100@cyberdoc.dk> Message-ID: <20050816083938.GE20254@samad.com.au> On Tue, Aug 16, 2005 at 06:11:26AM +0200, Daniel Frederiksen wrote: > Ok folks, here goes.. > > I have been boggling with a problem for the past week, and still haven't > found a solution.. > > I'm trying to route traffic from two providers through a Linux machine. > But that is not the problem. The ISP's have provided me with a WAN IP > class for both of the lines, to be routed into a DMZ where the machines > a to respond to their respective designated WAN IP on both lines. > Every machine on the DMZ has two IP's one on each ISP WAN Class. > > I think I'll better draw a map: > > > WAN1(eth2), WAN2(eth3) > --------- (eth0) > | |-----\ ---------- > | DMZ |---\ \ /---| ISP1 |----- > --------- \ \ / ---------- \ > \ \ / \ > ----------------- > | Linux GW/FW | WWW > ----------------- > / \ (eth1) / > --------- / \ ---------- / > | LAN |---/ \---| ISP2 |----- > --------- ---------- > NAT(eth4) > > > The DMZ has two WAN IP classes routed from the ISP. > > The thing I just can not figure out is how to make the respective WAN IP > from the DMZ route out the right ISP link, and the right request from > the ISP route into the DMZ. > > .. and finally how can I make the LAN able to access it all.. you need to use ip ru my ip ru looks like 0: from all lookup local 200: from 141.168.16.16 lookup cable 201: from 220.233.15.63 lookup adsl 32766: from all lookup main 32767: from all lookup default I created 200 and 201 which means that all traffic that came in on the cable 141.168.16.16 will go out the cable ip ro sh tab cable 192.168.11.0/24 dev br0 scope link 192.168.10.0/24 dev eth3 scope link 192.168.9.0/24 dev eth4 scope link default via 141.168.16.1 dev eth0 src 141.168.16.16 metric 30 and the routing tab for the adsl uses the adsl as its default gw. does that help ? > > Thanks for your time.. > > /Daniel Frederiksen > > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://mailman.ds9a.nl/pipermail/lartc/attachments/20050816/c14f0b80/attachment.bin From nikky at mnet.bg Tue Aug 16 15:34:04 2005 From: nikky at mnet.bg (Nickola Kolev) Date: Tue Aug 16 15:34:12 2005 Subject: [LARTC] Hardware Configuration Ideas In-Reply-To: References: Message-ID: <20050816163404.77ebb494.nikky@mnet.bg> Hello, Mihai, On Mon, 15 Aug 2005 23:53:38 +0300 "Mihai Vlad" wrote: > Hey guys, > > I am planning to buy some components for a Linux router that will > handle the Internet access of 200 computers (includes tc shaping) and > some inter sub-network routing (at least 100MBps per eth - and there > are 3 eth cards). > > I was thinking of a: > Pentium 4 - 3GHz > 256 or 512MB RAM > Network Cards. This is more than enough - I'd suggest a PIII/1GHz with 512MB RAM for those kind of needs, which is also a bit generous. [ cut ] > And can you point me to some good Network Cards that you have used > and are not so expensive. Some Intel, etc. I have no clue about this > subject... Network cards - YMMV, but I'd go for Intel EtherXpress Pro/100 for the 100Mbit, and something based on BCM57xx for the gigabit speeds. My experience dictates that for now the 2.4.30+ kernels are more suitable for tc/QoS setups. Hope it helped, Nickola -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mailman.ds9a.nl/pipermail/lartc/attachments/20050816/3178ab11/attachment.bin From alaios at yahoo.com Tue Aug 16 16:18:00 2005 From: alaios at yahoo.com (Alaios) Date: Tue Aug 16 16:18:05 2005 Subject: [LARTC] netlink documentation In-Reply-To: <42F0D30D.6030107@dsl.pipex.com> Message-ID: <20050816141801.76261.qmail@web54708.mail.yahoo.com> Hi there... i want to write some c code so as i can read and change the configurartion from a diff serv that has cbq tbf and htb... Pokinh around all this time i have noticed that libnl and lql libraries are not completely implemented so i need something else to do my jobb. I dont know what do u want to suggest me but i think that reading netlink and especially netlink that has to do with classes and qdiscs is what i must do... Plz suggest me sth.. Thx a lot ____________________________________________________ Start your day with Yahoo! - make it your home page http://www.yahoo.com/r/hs From gtaylor at riverviewtech.net Tue Aug 16 18:38:06 2005 From: gtaylor at riverviewtech.net (Taylor, Grant) Date: Tue Aug 16 18:41:45 2005 Subject: [LARTC] Hardware Configuration Ideas In-Reply-To: References: Message-ID: <4302166E.1080202@riverviewtech.net> Memory will most definitely be your problem. I think you could get away with a fairly low end processor (read < 1 GHz) but you will need a lot of memory depending on how much you want to do. I have a router in place that I was running out of memory for the connection tracking sub system. I ended up allocating 1 GB of RAM to just connection tracking. In fact you need 1 GB (or very close to) to be able to track 65535 connections. You may think this is way over kill, but not really. Keep in mind that connections tend to hang around on average 10 minutes after they are closed b/c not all systems out there close them correctly and thus they have to time out (10 minutes). You can get away with less RAM but you need to watch your DMESG to make sure that you don't see any issues with your connection tracking table filling up, it acts like a FIFO if memory serves. If you are not doing much in the way if *VERY* *ADVANCED* firewalling, just basic source and / or destination IP v alidation and / or source and / or destination port validation will not need much of a processor. In fact I'd try it with a 500 MHz to 1 GHz system, what ever is the most economical that you can get your hands on. Another problem that you may run in to will be filling your ARP table. The kernel space ARP table is not very large at all, only like 64 or maybe up to 255 IP MAC pairs. I want to say it's closer to 64. Thus you may want to take a look at using the ARP Daemon for Linux to offload the ARP cache to thus avoiding this issue. Basically how it works (from what I've read) is you reduce the number of times you query the ARP cache in kernel to 0 which will cause the kernel to query the user space daemon for the ARP data. The user space daemon does it's own ARPing to make sure that it has the information to hand to the kernel. The main advantage of the user space daemon is that it can handle LOTS of ARP entries, well beyond 1024 (I think). Something else you might consider would be some managed switches so that you could bond your connections out of the router to them thus ensuring that a cable failure (disconnection) will not take the router down. If you plug everything in to the managed switch and set up some VLANs you can easily do everything that you are wanting to do over the bonded connections with VLANs on top. The VLAN interfaces in Linux look like another network interface that you can do all the routing that you want off of. If the client systems you are going to be firewalling for are business systems I might suggest building two of these routers and setting them up as a VRRP router to ensure that the ""router is alwayse up and useable. This is much easier through managed switches too as you don't have to cable as much to the physical routers. In short get memory and a lower end proc to save the money for a 2nd identical router. Grant. . . . Mihai Vlad wrote: > Hey guys, > > I am planning to buy some components for a Linux router that will handle the > Internet access of 200 computers (includes tc shaping) and some inter > sub-network routing (at least 100MBps per eth - and there are 3 eth cards). > > I was thinking of a: > Pentium 4 - 3GHz > 256 or 512MB RAM > Network Cards. > > Now - I wonder what is more important: the processor speed or the amount of > RAM. > > And can you point me to some good Network Cards that you have used and are > not so expensive. Some Intel, etc. I have no clue about this subject... > > > Maybe this discussion can be extended to a list of best practices to set up > a performant Linux Router from the hardware point of view. > > Thanks in advance, > Mihai > > > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc From surda at shurdix.com Tue Aug 16 19:17:08 2005 From: surda at shurdix.com (Peter Surda) Date: Tue Aug 16 19:17:35 2005 Subject: [LARTC] Hardware Configuration Ideas In-Reply-To: <4302166E.1080202@riverviewtech.net> Message-ID: <20057161917812097@mail.routehat.org> On Tue, 16 Aug 2005 11:38:06 -0500 "Taylor, Grant" wrote: >I ended up >+allocating 1 GB of RAM to just connection tracking. In fact you need 1 GB (or >+very close to) to be able to track 65535 connections. You don't. Maybe that's conntrack's default, but you can set it to a higher number manually. The required memory is approx 400b per connection (depends on iptables/kernel compile time options). The rather conservative default (hashsize = 1/16384th of RAM) is for a generic system. For more info look at ip_conntrack_core.c 65535 connections need about 25MB in RAM, so before starting iptables, do modprobe ip_conntrack hashsize=8192 (contrack_max is auto-set to 8*hashsize, this is the recommended relation). In fact my distro Shurdix automatically sets up larger hashsize than the default, depending on system memory. You can change conntrack_max while the module is loaded (sysctl net.ipv4.netfilter.ip_conntrack_max), but you can't change the hashsize this way. If the relation is other than 1:8, you might experience performance problems (I don't know details, this is recommended on various places on the net). >Another problem that you may run in to will be filling your ARP table. The >+kernel space ARP table is not very large at all, only like 64 or maybe up to 255 >+IP MAC pairs. This is also tunable, per sysctl, somewhere like net.ipv4.neigh.default.gc_thresh[123]. Unfortunately poorly documented, I had to look at the source to realize this, and I don't remember what means what. >In short get memory and a lower end proc to save the money for a 2nd identical >router. While a redundant system is indeed a good idea, I recommend making sure the router is rock stable. This doesn't necessarily require high-end / fast hardware, it is recommended to stress test it before going live (memtest/cpuburn/whatever). My tip is not to use "primitive" network cards like those based on rtl8139 which you require high bandwidth. This has the most noticeable impact on performance. I have ok experience with 3com's, I've heard intels are even better. Yours sincerely, Peter -- http://www.shurdix.org - Linux distribution for routers and firewalls From surda at shurdix.com Tue Aug 16 19:21:02 2005 From: surda at shurdix.com (Peter Surda) Date: Tue Aug 16 19:21:26 2005 Subject: [LARTC] Hardware Configuration Ideas In-Reply-To: <4302166E.1080202@riverviewtech.net> Message-ID: <20057161921212097@mail.routehat.org> Also, .... On Tue, 16 Aug 2005 11:38:06 -0500 "Taylor, Grant" wrote: >+If you are not doing much in the way if *VERY* *ADVANCED* >+firewalling, just basic source and / or destination IP v >alidation and / or source and / or destination port validation will not need >+much of a processor. In fact I'd try it with a 500 MHz to 1 GHz system, what >+ever is the most economical that you can get your hands on. Yes. In fact most cases of "advanced" firewalling only mean that you have a stupid fw-design, like hundreds/thousands of rules in one chain :-). Usually can be optimised by using sub-chains, ipset and/or ipt_ACCOUNT. Yours sincerely, Peter -- http://www.shurdix.org - Linux distribution for routers and firewalls From jarod125 at yahoo.com Tue Aug 16 21:37:48 2005 From: jarod125 at yahoo.com (Gabriel) Date: Tue Aug 16 21:37:53 2005 Subject: [LARTC] (yet another) HTB question(s) Message-ID: <20050816193748.49080.qmail@web60922.mail.yahoo.com> Hi, I've read the documentation about HTB and I pretty much managed to grasp how it works. In theory. But there still are some questions and I want to check with you to see if I understand things correctly. So here goes: 1) when used on a router for shaping traffic done by clients connected to it, shaping is done on the interface connected to the cable/dsl modem. If I wanted to create classes for every client on the network, I would have to use iptables to mark packets (using -j MARK) and not filters because, according to http://www.docum.org/docum.org/kptd/ the shaping is done after the SNAT, so all the clients would have the src address rewritten with the public IP. Am I getting this right? 2) shaping inbound traffic is tricky because you can't control the rate that the packets come to you. Is it a good idea to shape the outgoing traffic to the clients on the LAN side NIC? Would I achieve both upload and download shaping using the same htb script (a script that does shaping using iptables for marking packets and fw for matching them) on both NICs (LAN side and ISP side)? 3) I've tried to find an answer to this question but all I got was some similar question a couple of years ago from Martin Brown. What happens if shaping a variable bandwidth link and, at one moment, for a period of time, the available bandwidth goes beyond the rate specified for the class 1:1 (the class attached to the root qdisc)? How will htb act in such a situation? 4) related to 3), I've tried to use a syntax like this: "tc clas add dev $ETH parent 1:0 classid 1:1 htb rate $RATE ceil $CEIL", where CEIL > RATE , but the max speed I achieved was the RATE speed. After further reading, I found out that you can't (well, you can, there's no syntax error, but it's useless) specify CEIL for the class attached to the root qdisc because it doesn't have a parent to borrow from. So is there a way to simulate a variable bandwidth link or should I just set RATE to the highest possible value the bandwidth can reach? 5) related to 1). Is there a (major) difference between -j MARK and -j CLASSIFY? From what I've read, -j CLASSIFY sets both the major and minor (major:minor) numbers to be matched while -j MARK only sets the minor. That's about it for now. :) Thanks ____________________________________________________ Start your day with Yahoo! - make it your home page http://www.yahoo.com/r/hs From daniel at internux.co.id Sun Aug 21 06:22:10 2005 From: daniel at internux.co.id (Daniel Harold L.) Date: Tue Aug 16 22:13:46 2005 Subject: [LARTC] Maximum filter rule in tc Message-ID: <200508202022.10225.daniel@internux.co.id> Hello, How much maximum filter rule we can create with tc filter ? TIA, Daniel From Andreas.Klauer at metamorpher.de Tue Aug 16 22:43:50 2005 From: Andreas.Klauer at metamorpher.de (Andreas Klauer) Date: Tue Aug 16 22:44:14 2005 Subject: [LARTC] (yet another) HTB question(s) In-Reply-To: <20050816193748.49080.qmail@web60922.mail.yahoo.com> References: <20050816193748.49080.qmail@web60922.mail.yahoo.com> Message-ID: <200508162243.50968.Andreas.Klauer@metamorpher.de> On Tuesday 16 August 2005 21:37, Gabriel wrote: > If I wanted to create classes for every client on the network, I would > have to use iptables to mark packets (using -j MARK) and not > filters because, according to > http://www.docum.org/docum.org/kptd/ the shaping is done > after the SNAT, so all the clients would have the src > address rewritten with the public IP. Am I getting this > right? I'm not 100% sure, but I think so. Anyway, I'm more comfortable with iptables than the sometimes rather complicated tc filter syntax. So I try to do most stuff in iptables. > 2) shaping inbound traffic is tricky because you can't > control the rate that the packets come to you. Is it a good > idea to shape the outgoing traffic to the clients on the > LAN side NIC? Yes, you can't control the way packets come to you directly. But still, does that mean you should allow an incoming connection to hog the whole bandwidth? Dropping packets slows down incoming tcp connections, and in my case, although dropping means throwing away already-downloaded packets, I feel it improved usability. > Would I achieve both upload and download shaping using the same htb > script (a script that does shaping using iptables for marking packets and > fw for matching them) on both NICs (LAN side and ISP side)? There might be minor differences (on LAN interface, you have to account for internal LAN traffic, e.g. SSH sessions to your router, and other local services you might be running, which do not actually use internet bandwidth. > What happens if shaping a variable bandwidth link and, at one moment, for > a period of time, the available bandwidth goes beyond the rate specified > for the class 1:1 (the class attached to the root qdisc)? How will > htb act in such a situation? I don't have a variable bandwidth link, but I guess HTB won't let you use more bandwith than specified in any case. > 4) related to 3), I've tried to use a syntax like this: "tc > clas add dev $ETH parent 1:0 classid 1:1 htb rate $RATE > ceil $CEIL", where CEIL > RATE , but the max speed I > achieved was the RATE speed. After further reading, I found > out that you can't (well, you can, there's no syntax error, > but it's useless) specify CEIL for the class attached to > the root qdisc because it doesn't have a parent to borrow > from. So is there a way to simulate a variable bandwidth > link or should I just set RATE to the highest possible > value the bandwidth can reach? I doubt that HTB was made for variable bandwidth links. Anyway, I suggest you try out both possibilities and use the ones that work best for you. Shaping is always a lot of trial & error in my opinion. > 5) related to 1). Is there a (major) difference between -j > MARK and -j CLASSIFY? From what I've read, -j CLASSIFY sets > both the major and minor (major:minor) numbers to be > matched while -j MARK only sets the minor. I haven't used classify so far, but if I understand it right, it puts packets directly into the appropriate class. If that's true, it's a good idea to use it, since it saves you the hassle of first marking packets and then adding tc filter matches for that mark. HTH Andreas From shemminger at osdl.org Tue Aug 16 23:34:16 2005 From: shemminger at osdl.org (Stephen Hemminger) Date: Tue Aug 16 23:33:51 2005 Subject: [LARTC] [ANNOUNCE] iproute2 util update Message-ID: <20050816143416.61c10513@dxpl.pdx.osdl.net> http://developer.osdl.org/dev/iproute2/download/iproute2-050816.tar.gz Update to iproute2 to include: * Limit ip neigh flush to 10 rounds * tc ematch support (thomas) * build cleanups (thomas, et al) * Fix for options process with ipt (jamal) * Fix array overflow in paretonormal distribution build * Update include files to 2.6.13 * Decnet doc update (Steven Whithouse) Note: the ematch support won't build on really old versions of bison (1.28), but the kernel on those systems wouldn't support it anyway. From lists at wildgooses.com Wed Aug 17 00:04:15 2005 From: lists at wildgooses.com (Ed W) Date: Wed Aug 17 00:04:21 2005 Subject: [LARTC] Packets being coallesced Message-ID: <430262DF.6000608@wildgooses.com> Here's a peculiar one. I'm trying to simulate some speed effects due to varying sized packets so I have written a quick perl app which spits out packets of a fixed size (<1500 bytes). What I'm finding is the despite apparently turning off nagle and everything else I can think of I still notice that when the receiver isn't keeping up that the sending side (linux 2.6.11) is coalescing the packets and sending fewer large packets...? Is this a known and expected thing to do, or am I likely suffering some buffering effects in my perl app? Note: I am observing the effect based on tcpdump traces, so I can clearly see when it's sending the expect 1000 byte packets, and when it's suddenly packing them altogether into max MTU sized packets. Note also that it's *sending* them like this, it's not just the the receiver is buffering and receiving them like this... Seems odd to me? Ed W From perezoso at gmail.com Wed Aug 17 04:10:28 2005 From: perezoso at gmail.com (Alberto Torres) Date: Wed Aug 17 04:10:33 2005 Subject: [LARTC] =?iso-8859-1?q?Plotting_graphs_=BF=BFWHAT_TO_PLOT=3F?= Message-ID: <850c9dea050816191053df9fb6@mail.gmail.com> Hello, i am writing a GUI for the tc. I am almost there, but i need some guideness... For every class/qdisc i have the bytes sents, dropped, borrowed, overlimits data for statistics, and i want to plot them on a graph. What data and how would yuu find it usefull to see plotted on a graph? I need you answers ;) I will post my GUI soon :) From gtaylor at riverviewtech.net Wed Aug 17 08:12:45 2005 From: gtaylor at riverviewtech.net (Grant Taylor) Date: Wed Aug 17 08:13:23 2005 Subject: [LARTC] Hardware Configuration Ideas In-Reply-To: <20057161921212097@mail.routehat.org> References: <20057161921212097@mail.routehat.org> Message-ID: <4302D55D.8090008@riverviewtech.net> > Yes. In fact most cases of "advanced" firewalling only mean that you have a > stupid fw-design, like hundreds/thousands of rules in one chain :-). Usually can > be optimised by using sub-chains, ipset and/or ipt_ACCOUNT. If someone has hundreds of rules in one chain (with out a _*VERY*_ good reason and even then) they need to be shot on the spot. For performance reasons such a chain should be broken out in to a tree of chains an subchains that are jumped to in an attempt to minimize the number of rules that have to be traversed to get a match on any given packet. What I was referring to by advanced firewalling was such things as running things like "-p udp -s 0.0.0.0/32 -d 255.255.255.255/32 --sport 68 --dport 67 -m addrtype --src-type broadcast -m pkttype --pkt-type broadcast" for DHCP requests. or complex SSH Brute Force prevention chains / rules, or recent lists to control what types of traffic will be valid based on what you have sent or is not valid b/c you have not sent any thing, or should packets with the reset flag have the ack flat set or not, etc. Grant. . . . From gtaylor at riverviewtech.net Wed Aug 17 08:20:32 2005 From: gtaylor at riverviewtech.net (Grant Taylor) Date: Wed Aug 17 08:21:14 2005 Subject: [LARTC] Hardware Configuration Ideas In-Reply-To: <20057161917812097@mail.routehat.org> References: <20057161917812097@mail.routehat.org> Message-ID: <4302D730.3090105@riverviewtech.net> > You don't. Maybe that's conntrack's default, but you can set it to a higher > number manually. The required memory is approx 400b per connection (depends on > iptables/kernel compile time options). The rather conservative default (hashsize > = 1/16384th of RAM) is for a generic system. For more info look at > ip_conntrack_core.c > > 65535 connections need about 25MB in RAM, so before starting iptables, do > modprobe ip_conntrack hashsize=8192 > (contrack_max is auto-set to 8*hashsize, this is the recommended relation). In > fact my distro Shurdix automatically sets up larger hashsize than the default, > depending on system memory. Hmm, I did not have much time to solve this problem at the time and documentation was hard to come by at the time and what I did find was old. Alass I was not subscribed to this list to ask for help either. Note things have changed sense then. :) > While a redundant system is indeed a good idea, I recommend making sure the > router is rock stable. This doesn't necessarily require high-end / fast > hardware, it is recommended to stress test it before going live > (memtest/cpuburn/whatever). > > My tip is not to use "primitive" network cards like those based on rtl8139 which > you require high bandwidth. This has the most noticeable impact on performance. > I have ok experience with 3com's, I've heard intels are even better. I would agree to both points. I have had good luck with the rtl8139s on Cable / DSL and T1 routers but I would want something better (3C905x cards) for a much higher bandwidth installation. The redundant (identical) system is for those cases where the cleaning crew and / or momma nature and / or Mr Murphy have their way with your box. We have all had it happen (or will) in some way or another at some time. It is not "if" a box will fail in some way, but rather "when". The failure may not be any thing you could prevent. I think the stores in Florida this year are a good example of that. Grant. . . . From dor at ldc.net Wed Aug 17 09:08:08 2005 From: dor at ldc.net (Dmytro O. Redchuk) Date: Wed Aug 17 09:08:14 2005 Subject: [LARTC] (yet another) HTB question(s) In-Reply-To: <20050816193748.49080.qmail@web60922.mail.yahoo.com> References: <20050816193748.49080.qmail@web60922.mail.yahoo.com> Message-ID: <20050817070808.GA6684@ldc.net> On Tue, Aug 16, 2005 at 12:37:48PM -0700, Gabriel wrote: > 1) when used on a router for shaping traffic done by > clients connected to it, shaping is done on the interface > connected to the cable/dsl modem. If I wanted to create > classes for every client on the network, I would have to > use iptables to mark packets (using -j MARK) and not > filters because, according to > http://www.docum.org/docum.org/kptd/ the shaping is done > after the SNAT, so all the clients would have the src > address rewritten with the public IP. Am I getting this > right? This page: http://mailman.ds9a.nl/pipermail/lartc/2005q1/014656.html (LARTC archive) contains a lot of useful links, one of them: http://ebtables.sourceforge.net/br_fw_ia/PacketFlow.png Note right bottom corner of that picture. > 4) related to 3), I've tried to use a syntax like this: "tc > clas add dev $ETH parent 1:0 classid 1:1 htb rate $RATE > ceil $CEIL", where CEIL > RATE , but the max speed I > achieved was the RATE speed. After further reading, I found > out that you can't (well, you can, there's no syntax error, > but it's useless) specify CEIL for the class attached to > the root qdisc because it doesn't have a parent to borrow > from. So is there a way to simulate a variable bandwidth > link or should I just set RATE to the highest possible > value the bandwidth can reach? http://luxik.cdi.cz/~devik/qos/htb/manual/userg.htm (6. Priorizing bandwidth share) answers this question. Use qdisc inside qdisc. > That's about it for now. :) Thanks Thank you, -- _,-=._ /|_/| `-.} `=._,.-=-._., @ @._, `._ _,-. ) _,.-' ` G.m-"^m`m' Dmytro O. Redchuk From dor at ldc.net Wed Aug 17 09:17:12 2005 From: dor at ldc.net (Dmytro O. Redchuk) Date: Wed Aug 17 09:17:16 2005 Subject: [LARTC] Plotting graphs ??WHAT TO PLOT? In-Reply-To: <850c9dea050816191053df9fb6@mail.gmail.com> References: <850c9dea050816191053df9fb6@mail.gmail.com> Message-ID: <20050817071712.GB6684@ldc.net> On Wed, Aug 17, 2005 at 04:10:28AM +0200, Alberto Torres wrote: > Hello, i am writing a GUI for the tc. I am almost there, but i need > some guideness... > > For every class/qdisc i have the bytes sents, dropped, borrowed, > overlimits data for statistics, and i want to plot them on a graph. packets too lended tokens, ctokens :-) I like http://www2.ldc.net/~dor/py-htbstat/ though :) > > What data and how would yuu find it usefull to see plotted on a graph? > > I need you answers ;) I will post my GUI soon :) -- _,-=._ /|_/| `-.} `=._,.-=-._., @ @._, `._ _,-. ) _,.-' ` G.m-"^m`m' Dmytro O. Redchuk From psihozefir at yahoo.com Wed Aug 17 09:40:09 2005 From: psihozefir at yahoo.com (panca sorin) Date: Wed Aug 17 09:40:16 2005 Subject: [LARTC] (yet another) HTB question(s) In-Reply-To: <20050816193748.49080.qmail@web60922.mail.yahoo.com> Message-ID: <20050817074009.52963.qmail@web32608.mail.mud.yahoo.com> > 4) related to 3), I've tried to use a syntax like this: > "tc > clas add dev $ETH parent 1:0 classid 1:1 htb rate $RATE > ceil $CEIL", where CEIL > RATE , but the max speed I > achieved was the RATE speed. try to add 2-level root class: tc qdisc add dev $ETH root handle 1: htb default FF tc class add dev $ETH parent 1: classid 1:1 htb rate $MAXRATE ceil $MAXRATE tc class add dev $ETH parent 1:1 classid 1:2 htb rate $MINRATE ceil $MAXRATE tc class add ... [your child classes] I don't know if it'll work but worth trying. ____________________________________________________ Start your day with Yahoo! - make it your home page http://www.yahoo.com/r/hs From greve at fsfeurope.org Wed Aug 17 10:04:25 2005 From: greve at fsfeurope.org (Georg C. F. Greve) Date: Wed Aug 17 10:23:36 2005 Subject: [LARTC] HOWTO unmaintained? Message-ID: Skipped content of type multipart/mixed-------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 306 bytes Desc: not available Url : http://mailman.ds9a.nl/pipermail/lartc/attachments/20050817/83f251c6/attachment.bin From Andreas.Klauer at metamorpher.de Wed Aug 17 12:19:54 2005 From: Andreas.Klauer at metamorpher.de (Andreas Klauer) Date: Wed Aug 17 12:20:14 2005 Subject: [LARTC] HOWTO unmaintained? In-Reply-To: References: Message-ID: <200508171219.54668.Andreas.Klauer@metamorpher.de> On Wednesday 17 August 2005 10:04, Georg C. F. Greve wrote: > I never received any reaction from the HOWTO maintainers, not even > when addressing them directly (see mail below). > > Given that a month has gone by: Is the HOWTO currently unmaintained? www.lartc.org says: "Linux Advanced Routing & Traffic Control HOWTO Current version is 1.0.0 Files were last updated at 2004-03-31 00:11 CET (ie, about 12108.1 hours ago)." So yes, I'd guess it's not actively maintained. Andreas From greve at fsfeurope.org Wed Aug 17 12:40:55 2005 From: greve at fsfeurope.org (Georg C. F. Greve) Date: Wed Aug 17 12:41:11 2005 Subject: [LARTC] HOWTO unmaintained? In-Reply-To: <200508171219.54668.Andreas.Klauer@metamorpher.de> (Andreas Klauer's message of "Wed, 17 Aug 2005 12:19:54 +0200") References: <200508171219.54668.Andreas.Klauer@metamorpher.de> Message-ID: || On Wed, 17 Aug 2005 12:19:54 +0200 || Andreas Klauer wrote: ak> www.lartc.org says: ak> "Linux Advanced Routing & Traffic Control HOWTO Current version ak> is 1.0.0 Files were last updated at 2004-03-31 00:11 CET (ie, ak> about 12108.1 hours ago)." ak> So yes, I'd guess it's not actively maintained. Thanks for that quick reply. I guess the obvious question then is: How do we get it maintained? Does anyone know where the current maintainers have disappeared? Is anyone willing to take over that job? Regards, Georg -- Georg C. F. Greve Free Software Foundation Europe (http://fsfeurope.org) Join the Fellowship and protect your freedom! (http://www.fsfe.org) -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 306 bytes Desc: not available Url : http://mailman.ds9a.nl/pipermail/lartc/attachments/20050817/714a163b/attachment.bin From lists at wildgooses.com Wed Aug 17 13:28:10 2005 From: lists at wildgooses.com (Ed W) Date: Wed Aug 17 13:28:17 2005 Subject: [LARTC] HOWTO unmaintained? In-Reply-To: References: <200508171219.54668.Andreas.Klauer@metamorpher.de> Message-ID: <43031F4A.1090007@wildgooses.com> >I guess the obvious question then is: How do we get it maintained? > >Does anyone know where the current maintainers have disappeared? > >Is anyone willing to take over that job? > > I wonder if someone would host a mediawiki and consider uploading the documentation there. This would make it easier for people to contribute, and I think it shold be fairly easy to convert from it's current format to a wiki Just a thought Ed W From kenneth.kalmer at gmail.com Wed Aug 17 14:26:00 2005 From: kenneth.kalmer at gmail.com (Kenneth Kalmer) Date: Wed Aug 17 14:26:04 2005 Subject: [LARTC] HOWTO unmaintained? In-Reply-To: <43031F4A.1090007@wildgooses.com> References: <200508171219.54668.Andreas.Klauer@metamorpher.de> <43031F4A.1090007@wildgooses.com> Message-ID: On 8/17/05, Ed W wrote: > > >I guess the obvious question then is: How do we get it maintained? > > > >Does anyone know where the current maintainers have disappeared? > > > >Is anyone willing to take over that job? > > > > > > I wonder if someone would host a mediawiki and consider uploading the > documentation there. This would make it easier for people to > contribute, and I think it shold be fairly easy to convert from it's > current format to a wiki > > Just a thought > And a great one I might add. Does anybody know how busy the current site is? If not too busy (i.e.< 10GB a month) I'd gladly put up a wiki on my server for it. If it get's busier I'll just have to move it to another server in due course. I've also gotten very frustrated with some old outdated information, and especially the lack of information regarding the 2.6.x kernel. All in favour...? Regards -- Kenneth Kalmer kenneth.kalmer@gmail.com Folding@home stats http://vspx27.stanford.edu/cgi-bin/main.py?qtype=userpage&username=kenneth%2Ekalmer From mikeeo at msn.com Wed Aug 17 14:27:44 2005 From: mikeeo at msn.com (Mike O) Date: Wed Aug 17 14:27:53 2005 Subject: [LARTC] HOWTO unmaintained? In-Reply-To: <43031F4A.1090007@wildgooses.com> Message-ID: I don't even think this is list is being maintained. Several people have tried to unsubscribe. >From: Ed W >To: "Georg C. F. Greve" >CC: Andreas Klauer ,lartc@mailman.ds9a.nl >Subject: Re: [LARTC] HOWTO unmaintained? >Date: Wed, 17 Aug 2005 12:28:10 +0100 > > >>I guess the obvious question then is: How do we get it maintained? >> >>Does anyone know where the current maintainers have disappeared? >> >>Is anyone willing to take over that job? >> >> > >I wonder if someone would host a mediawiki and consider uploading the >documentation there. This would make it easier for people to contribute, >and I think it shold be fairly easy to convert from it's current format to >a wiki > >Just a thought > >Ed W >_______________________________________________ >LARTC mailing list >LARTC@mailman.ds9a.nl >http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc From vinod_chandran at multitech.co.in Wed Aug 17 14:36:19 2005 From: vinod_chandran at multitech.co.in (Vinod Chandran) Date: Wed Aug 17 14:36:07 2005 Subject: [LARTC] HTB and Prio Message-ID: <43032F43.9000105@multitech.co.in> Hi , I wanted some clarification on the PRIO value of HTB. I have a parent class 1:1 with ceil and rate 1000Kbit. Further I have three classes 1. 1:2 with rate 500Kbit ceil 1000Kbit prio 1 2. 1:3 with rate 300Kbit ceil 1000Kbit prio 2 3. 1:4 with rate 200Kbit ceil 1000Kbit prio 3 Also there is a child class of 1:2 , 1:22, with rate 100Kbit ceil 1000Kbit prio 4. I am not that among the siblings, the excess bandwidth is shared on basis of priority ie . 1:3 will get a higher share than 1:4. What I am not sure is will 1:22 class(being a child of 1:2) get a greater share than 1:3 or for that matter 1:4. Thanks, Vinod C From lists at llondel.org Wed Aug 17 15:34:04 2005 From: lists at llondel.org (Mailing List Account) Date: Wed Aug 17 15:34:07 2005 Subject: [LARTC] Unsubsription impossible In-Reply-To: <200508152351.10428.thomasheinz@gmx.net> References: <200508152351.10428.thomasheinz@gmx.net> Message-ID: <43033CCC.6080107@llondel.org> Thomas Heinz wrote: > Hi > > Several times, I unsuccessfully tried to unsubscribe from lartc via the web > interface and mailto:lartc-request@mailman.ds9a.nl?subject=unsubscribe. I > also contacted the mailing list admin but got no reply. Even resubscribing > did not work. > > I have posted here since I am running out of options. This posting is > addressed to the list admin in charge: > Please remove my email address from the lartc mailing list. > If all else fails, just start bouncing the messages. After a few bounces it'll unsubscribe you. I kept getting unsubscribed because my system was rejecting the sentrisystem virus stuff. Dave From lartc-337 at ccp.com.au Wed Aug 17 15:38:44 2005 From: lartc-337 at ccp.com.au (Lee Sanders) Date: Wed Aug 17 15:38:59 2005 Subject: [LARTC] HOWTO unmaintained? In-Reply-To: References: <43031F4A.1090007@wildgooses.com> Message-ID: <200508172138.44503.lartc-337@ccp.com.au> Hi Kenneth, Aye/puts hand up in agreement. A wiki would be great. The problem right now is there is little, and usually not very good, documentation on how to setup queue's and other advanced routing. The end result this list is full of people asking questions and very few who answer. :L > And a great one I might add. Does anybody know how busy the current > site is? If not too busy (i.e.< 10GB a month) I'd gladly put up a wiki > on my server for it. If it get's busier I'll just have to move it to > another server in due course. > > I've also gotten very frustrated with some old outdated information, > and especially the lack of information regarding the 2.6.x kernel. > > All in favour...? From psihozefir at yahoo.com Wed Aug 17 15:45:02 2005 From: psihozefir at yahoo.com (panca sorin) Date: Wed Aug 17 15:45:06 2005 Subject: [LARTC] HTB and Prio In-Reply-To: <43032F43.9000105@multitech.co.in> Message-ID: <20050817134502.71403.qmail@web32605.mail.mud.yahoo.com> > I am not that among the siblings, the excess bandwidth > is shared on > basis of priority ie . 1:3 will get a higher share than > 1:4. The quantum of borrowed bandwidth is proportional with the rate you specified for each class. The prio control is for controlling who gets the excess bandwith first (if exists). ____________________________________________________ Start your day with Yahoo! - make it your home page http://www.yahoo.com/r/hs From andre.correa at pobox.com Wed Aug 17 16:09:23 2005 From: andre.correa at pobox.com (Andre D. Correa) Date: Wed Aug 17 16:09:39 2005 Subject: [LARTC] HOWTO unmaintained? In-Reply-To: References: <200508171219.54668.Andreas.Klauer@metamorpher.de> <43031F4A.1090007@wildgooses.com> Message-ID: <43034513.8080200@pobox.com> Hi, I got frustrated with it several months ago when I tried to update IMQ information and never got a response. I can host a WiKi too, maybe we can mirror content and share the task. I'm not sure if any WiKi has a mirroring functionality but we can figure this out. -------------------------------------------------------------------- Andre D. Correa, CISSP | Visite meus projetos pessoais: andre.correa (at) pobox.com | Visit my personal projects: http://andre.hiperlinks.com.br | -http://malware.hiperlinks.com.br Sao Paulo / SP / Brazil | -http://www.linuximq.net/ -------------------------------------------------------------------- Kenneth Kalmer wrote: > On 8/17/05, Ed W wrote: > >>>I guess the obvious question then is: How do we get it maintained? >>> >>>Does anyone know where the current maintainers have disappeared? >>> >>>Is anyone willing to take over that job? >>> >>> >> >>I wonder if someone would host a mediawiki and consider uploading the >>documentation there. This would make it easier for people to >>contribute, and I think it shold be fairly easy to convert from it's >>current format to a wiki >> >>Just a thought >> > > > And a great one I might add. Does anybody know how busy the current > site is? If not too busy (i.e.< 10GB a month) I'd gladly put up a wiki > on my server for it. If it get's busier I'll just have to move it to > another server in due course. > > I've also gotten very frustrated with some old outdated information, > and especially the lack of information regarding the 2.6.x kernel. > > All in favour...? > > Regards > From mihaivlad at web-profile.net Wed Aug 17 17:46:35 2005 From: mihaivlad at web-profile.net (Mihai Vlad) Date: Wed Aug 17 17:46:50 2005 Subject: [LARTC] Hardware Configuration Ideas In-Reply-To: <4302D730.3090105@riverviewtech.net> Message-ID: Thank you all for your help! The reason why I asked for your help was because I had the following problem: Router: 1.4 GHZ AMD 256 RAM Realtek 8139 NIC Slackware 2.6 kernel No iptables rules just iproute2 routing. I have 2 sub-networks and the router is configured as a "router on a stick". I tested a simple routing from one sub-network to another (FTP transfer) (the traffic enters and leaves the router using the same NIC) and the maximum transfer rate is 50 Mbit (instead of 100 Mbit as expected). I wonder how am I able to route a Gigabit network ... Someone told me that he solved the problem using FreeBSD as it has a better performance on this field... I am bound to Linux as it has a superb implementation of QoS. You have nothing like this in BSD. I want to ask you: what is the maximum transfer rate that you can obtain on your configurations (with and without heavy firewalling / marking / tc). Thanks again for your input! Mihai From shemminger at osdl.org Wed Aug 17 18:24:28 2005 From: shemminger at osdl.org (Stephen Hemminger) Date: Wed Aug 17 18:24:08 2005 Subject: [LARTC] HOWTO unmaintained? In-Reply-To: <43031F4A.1090007@wildgooses.com> References: <200508171219.54668.Andreas.Klauer@metamorpher.de> <43031F4A.1090007@wildgooses.com> Message-ID: <20050817092428.531193f0@dxpl.pdx.osdl.net> On Wed, 17 Aug 2005 12:28:10 +0100 Ed W wrote: > > >I guess the obvious question then is: How do we get it maintained? > > > >Does anyone know where the current maintainers have disappeared? > > > >Is anyone willing to take over that job? > > > > > > I wonder if someone would host a mediawiki and consider uploading the > documentation there. This would make it easier for people to > contribute, and I think it shold be fairly easy to convert from it's > current format to a wiki > Good idea, I'll see if the OSDL will put up a wiki for it on developer.osdl.org, stay tuned. From tdi at pozman.pl Wed Aug 17 18:28:08 2005 From: tdi at pozman.pl (Dariusz Dwornikowski) Date: Wed Aug 17 18:28:50 2005 Subject: [LARTC] HOWTO unmaintained? In-Reply-To: <200508172138.44503.lartc-337@ccp.com.au> References: <43031F4A.1090007@wildgooses.com> <200508172138.44503.lartc-337@ccp.com.au> Message-ID: <20050817182808.206288a1@tdi.pozman.pl> > Aye/puts hand up in agreement. > > A wiki would be great. The problem right now is there is little, and usually > not very good, documentation on how to setup queue's and other advanced > routing. > > The end result this list is full of people asking questions and very few who > answer. i am totally pro refreshing the lartc howto. and wiki is a great idea -- *Dariusz 'tdi' Dwornikowski | Gentoo | admin at pozman.pl | *[JID]:tdi@gentoo.pl|[gg]:2266034|[IRC]:#gentoo-pl@freenode | *[MAIL]:tdi@pozman.pl|[WWW]:www.tdi.pozman.pl | *Serwery,administracja,webapps - www.ProAdmin.com.pl | *Fingerprint:43E21CC46DAFD2F754E91547D59B39F56AAA4B5F | -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mailman.ds9a.nl/pipermail/lartc/attachments/20050817/8ee89b79/attachment.bin From imipak at yahoo.com Wed Aug 17 19:05:47 2005 From: imipak at yahoo.com (Jonathan Day) Date: Wed Aug 17 19:05:51 2005 Subject: [LARTC] HOWTO unmaintained? In-Reply-To: Message-ID: <20050817170547.23039.qmail@web31509.mail.mud.yahoo.com> It seems strange that astronomers discovered a whole set of Black Holes at about the time the maintainers vanished... It looks like a number of people are offering sites - IMHO, a "distributed" wiki (ie: you can edit at any of the sites) or a master/mirror setup would be good, as that would help prevent problems if site maintainers get kidnapped by aliens, sites get slashdotted, etc. It would also be good if at least one site offered multiple ways to connect - eg: via an IPSec tunnel or via IPv6 - as this would give people a simple way of testing what they're trying. --- Kenneth Kalmer wrote: > On 8/17/05, Ed W wrote: > > > > >I guess the obvious question then is: How do we > get it maintained? > > > > > >Does anyone know where the current maintainers > have disappeared? > > > > > >Is anyone willing to take over that job? > > > > > > > > > > I wonder if someone would host a mediawiki and > consider uploading the > > documentation there. This would make it easier > for people to > > contribute, and I think it shold be fairly easy to > convert from it's > > current format to a wiki > > > > Just a thought > > > > And a great one I might add. Does anybody know how > busy the current > site is? If not too busy (i.e.< 10GB a month) I'd > gladly put up a wiki > on my server for it. If it get's busier I'll just > have to move it to > another server in due course. > > I've also gotten very frustrated with some old > outdated information, > and especially the lack of information regarding the > 2.6.x kernel. > > All in favour...? > > Regards > > -- > > Kenneth Kalmer > kenneth.kalmer@gmail.com > > Folding@home stats > http://vspx27.stanford.edu/cgi-bin/main.py?qtype=userpage&username=kenneth%2Ekalmer > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From Andreas.Klauer at metamorpher.de Wed Aug 17 19:27:28 2005 From: Andreas.Klauer at metamorpher.de (Andreas Klauer) Date: Wed Aug 17 19:27:47 2005 Subject: [LARTC] HOWTO unmaintained? In-Reply-To: <20050817170547.23039.qmail@web31509.mail.mud.yahoo.com> References: <20050817170547.23039.qmail@web31509.mail.mud.yahoo.com> Message-ID: <200508171927.28586.Andreas.Klauer@metamorpher.de> On Wednesday 17 August 2005 19:05, Jonathan Day wrote: > It looks like a number of people are offering sites - > IMHO, a "distributed" wiki (ie: you can edit at any of > the sites) or a master/mirror setup would be good, as > that would help prevent problems if site maintainers > get kidnapped by aliens, sites get slashdotted, etc. I'd prefer if we could keep the central site www.lartc.org since that location is already more than well known. If the owner just doesn't have the time to maintain the Howto anymore, it would be best to put the wiki directly there, since that can be self-maintained by the users then. So I suggest we at least try to contact the original maintainer / domain owner before putting a Wiki just anywhere. If that's not possible, I think I'd prefer one central, but reliable long-term host over a distributed solution. Mirrors are fine, though. > It would also be good if at least one site offered > multiple ways to connect - eg: via an IPSec tunnel or > via IPv6 - as this would give people a simple way of > testing what they're trying. There are sites that let you run a connection speed test and stuff. However, this does not have to do anything with the Wiki per se. Regards Andreas Klauer From c-d.hailfinger.devel.2005 at gmx.net Wed Aug 17 19:33:55 2005 From: c-d.hailfinger.devel.2005 at gmx.net (Carl-Daniel Hailfinger) Date: Wed Aug 17 19:33:41 2005 Subject: [LARTC] HOWTO unmaintained? In-Reply-To: <20050817170547.23039.qmail@web31509.mail.mud.yahoo.com> References: <20050817170547.23039.qmail@web31509.mail.mud.yahoo.com> Message-ID: <43037503.60001@gmx.net> Jonathan Day schrieb: > It seems strange that astronomers discovered a whole > set of Black Holes at about the time the maintainers > vanished... > > It looks like a number of people are offering sites - > IMHO, a "distributed" wiki (ie: you can edit at any of > the sites) or a master/mirror setup would be good, as > that would help prevent problems if site maintainers > get kidnapped by aliens, sites get slashdotted, etc. Or some big organization like SUSE/RedHat/etc. who are unlikely to vanish could offer hosting a wiki. That makes alien abduction and slashdotting really unlikely. Using multiple sites is always a problem because invariably people will lose interest, sites will get out of sync, the question which site should be linked from external sites and all of the sites stealing google ranking from each other. If people feel comfortable with SUSE hosting such a wiki, I could ask them. Regards, Carl-Daniel -- http://www.hailfinger.org/ From surda at shurdix.com Wed Aug 17 19:33:25 2005 From: surda at shurdix.com (Peter Surda) Date: Wed Aug 17 19:33:53 2005 Subject: [LARTC] Hardware Configuration Ideas In-Reply-To: Message-ID: <200571719332512097@mail.routehat.org> On Wed, 17 Aug 2005 18:46:35 +0300 "Mihai Vlad" wrote: >The reason why I asked for your help was because I had the following >problem: [cut] >Realtek 8139 NIC Yes, this is indeed a problem. Cards based on this chipset don't scale well. Try 3coms or intels. If you're short on money, dlink dfe-530tx might do too. (I work for none of the mentioned companies :-)). >Mihai Yours sincerely, Peter -- http://www.shurdix.org - Linux distribution for routers and firewalls From c-d.hailfinger.devel.2005 at gmx.net Wed Aug 17 19:39:19 2005 From: c-d.hailfinger.devel.2005 at gmx.net (Carl-Daniel Hailfinger) Date: Wed Aug 17 19:39:01 2005 Subject: [LARTC] HOWTO unmaintained? In-Reply-To: <200508171927.28586.Andreas.Klauer@metamorpher.de> References: <20050817170547.23039.qmail@web31509.mail.mud.yahoo.com> <200508171927.28586.Andreas.Klauer@metamorpher.de> Message-ID: <43037647.4070806@gmx.net> Andreas Klauer schrieb: > On Wednesday 17 August 2005 19:05, Jonathan Day wrote: > >>It would also be good if at least one site offered >>multiple ways to connect - eg: via an IPSec tunnel or >>via IPv6 - as this would give people a simple way of >>testing what they're trying. > > There are sites that let you run a connection speed test and stuff. > However, this does not have to do anything with the Wiki per se. Yes, a way to test certain setups would be cool, but you have to be aware that there might be security bugs in the services you offer. Your risk would be higher than for usual setups because people might also be able to use post-authentication bugs. A person setting up such a test system should also be aware that people might use more bandwidth than anticipated. Regards, Carl-Daniel -- http://www.hailfinger.org/ From gtaylor at riverviewtech.net Wed Aug 17 21:18:47 2005 From: gtaylor at riverviewtech.net (Taylor, Grant) Date: Wed Aug 17 21:22:22 2005 Subject: [LARTC] Hardware Configuration Ideas In-Reply-To: <200571719332512097@mail.routehat.org> References: <200571719332512097@mail.routehat.org> Message-ID: <43038D97.6020909@riverviewtech.net> > Yes, this is indeed a problem. Cards based on this chipset don't scale well. Try > 3coms or intels. If you're short on money, dlink dfe-530tx might do too. Um, the DFE-530TX+ is a (rebranded) Realtek8139 chipset card so I would expect that the performance would be comparable to the Realtek cards. I personally have not used them but I have heard wonderful things about DEC Tulup cards. If you are interested I have some 3Com 905 cards I'd be willing to part with for a very reasonable price. (Please email me off the list if you are interested in the NICs.) Grant. . . . From c-d.hailfinger.devel.2005 at gmx.net Wed Aug 17 22:36:04 2005 From: c-d.hailfinger.devel.2005 at gmx.net (Carl-Daniel Hailfinger) Date: Wed Aug 17 22:35:46 2005 Subject: [LARTC] Hardware Configuration Ideas In-Reply-To: <43038D97.6020909@riverviewtech.net> References: <200571719332512097@mail.routehat.org> <43038D97.6020909@riverviewtech.net> Message-ID: <43039FB4.7060605@gmx.net> Taylor, Grant schrieb: >>Yes, this is indeed a problem. Cards based on this chipset don't scale well. Try >>3coms or intels. If you're short on money, dlink dfe-530tx might do too. > > Um, the DFE-530TX+ is a (rebranded) Realtek8139 chipset card so I would > expect that the performance would be comparable to the Realtek cards. > I personally have not used them but I have heard wonderful things about > DEC Tulip cards. Some of the DFE-5..TX cards are indeed tulip based and have very good performance for a reasonable price. Unfortunately sources on the net disagree which cards have which chipset, so I recommend looking at them (I bought a new tulip based card here in Germany for 7 Euros). It pays to request a look at the chipsets of the cheaper cards, they might not even advertise their chipsets as tulip. Regards, Carl-Daniel -- http://www.hailfinger.org/ From surda at shurdix.com Wed Aug 17 23:06:55 2005 From: surda at shurdix.com (Peter Surda) Date: Wed Aug 17 23:07:20 2005 Subject: [LARTC] Hardware Configuration Ideas In-Reply-To: <43038D97.6020909@riverviewtech.net> Message-ID: <20057172365512097@mail.routehat.org> On Wed, 17 Aug 2005 14:18:47 -0500 "Taylor, Grant" wrote: >Um, the DFE-530TX+ is a (rebranded) Realtek8139 chipset card so I would expect >that the performance would be comparable to the Realtek cards. I don't have one here, but I vaguely remember they use a different kernel module (8139too vs. via-rhine) so I think it's a different chipset. I only mentioned them because some time in distant past I solved some performance problems by switching from from 8139 to the dlink, and the dlink costs less than a half of entry-level 3com. >Grant. . . . -- http://www.shurdix.org - Linux distribution for routers and firewalls From gtaylor at riverviewtech.net Wed Aug 17 23:44:34 2005 From: gtaylor at riverviewtech.net (Taylor, Grant) Date: Wed Aug 17 23:48:10 2005 Subject: [LARTC] HOWTO unmaintained? In-Reply-To: <20050817170547.23039.qmail@web31509.mail.mud.yahoo.com> References: <20050817170547.23039.qmail@web31509.mail.mud.yahoo.com> Message-ID: <4303AFC2.6070801@riverviewtech.net> > It looks like a number of people are offering sites - > IMHO, a "distributed" wiki (ie: you can edit at any of > the sites) or a master/mirror setup would be good, as > that would help prevent problems if site maintainers > get kidnapped by aliens, sites get slashdotted, etc. I think the Wiki, if that route is chosen, should be on the www.lartc.org domain name. This means that we will have to find and contact the administrators of that domain / DNS servers. As far as the distributed web site goes I think it is a good idea. To pull off the distributed site we would need to have the DNS records resolve to multiple boxen across the net. I have considered a self replicating set up for some of my servers and at present I'm looking at using Coda or AFS as a replicating / caching local copies of the remote file system content. I've never dealt with Wikis other than and end user (and I say that the ones that I've looked at have been slow) so I don't know what they take to set up. I suspect that they use a database and thus we would want to set up the Wiki to use a database that has real time replication between the two (or more) web servers that the wiki points to. I would be more than happy to help with such an endeavor. I can not host it at my office (bosses will not let me) but I can help provide content and / or convert stuff. > It would also be good if at least one site offered > multiple ways to connect - eg: via an IPSec tunnel or > via IPv6 - as this would give people a simple way of > testing what they're trying. Again I am not able to do this, but I think it could be relatively easily done by offering a host with multiple IPs bound to it and give people a UML that they can test things in. Much of the routing / firewalling work that I have done can easily be done in side of a UML. This would mean that a system would need to be fairly capable and running a UML it's self to be a router in to the UML farm / UML switch backplane. Again I would be more than willing to help set up such a system (and enjoy it at that). I think it would be interesting to do this with multiple distributions and possibly versions there of. To pull this off the box would need to be fairly powerful though to support many people at one time. I'd say that you could get away with a dual multi GHz proc box with at least 2 - 4 GB of RAM. I would expect that this could support 10+ concurrent users in side of UML doing some compiling or more if they are just using recompiled binaries. Grant. . . . From stef.coene at docum.org Thu Aug 18 09:04:54 2005 From: stef.coene at docum.org (Stef Coene) Date: Thu Aug 18 09:05:08 2005 Subject: [LARTC] HOWTO unmaintained? In-Reply-To: <4303AFC2.6070801@riverviewtech.net> References: <20050817170547.23039.qmail@web31509.mail.mud.yahoo.com> <4303AFC2.6070801@riverviewtech.net> Message-ID: <200508180904.55023.stef.coene@docum.org> On Wednesday 17 August 2005 23:44, Taylor, Grant wrote: > > It looks like a number of people are offering sites - > > IMHO, a "distributed" wiki (ie: you can edit at any of > > the sites) or a master/mirror setup would be good, as > > that would help prevent problems if site maintainers > > get kidnapped by aliens, sites get slashdotted, etc. > > I think the Wiki, if that route is chosen, should be on the www.lartc.org > domain name. This means that we will have to find and contact the > administrators of that domain / DNS servers. (I'm not official subscribed to this list, but I'm still reading some posts) I know the owner of lartc.org and I mailed him about this problem. I will keep the list updated if he answers me. > As far as the distributed web > site goes I think it is a good idea. To pull off the distributed site we > would need to have the DNS records resolve to multiple boxen across the > net. I have considered a self replicating set up for some of my servers > and at present I'm looking at using Coda or AFS as a replicating / caching > local copies of the remote file system content. I've never dealt with > Wikis other than and end user (and I say that the ones that I've looked at > have been slow) so I don't know what they take to set up. I suspect that > they use a database and thus we would want to set up the Wiki to use a > database that has real time replication between the two (or more) web > servers that the wiki points to. I would be more than happy to help with > such an endeavor. I can not host it at my office (bosses will not let me) > but I can help provide content and / or convert stuff. There are lots of wiki's available. Some use plain text files, some uses mysql, some are written in php, some are written in perl. I prefer the mysql + php way to store the information. Stef From lvhung2k3 at yahoo.com Thu Aug 18 11:05:20 2005 From: lvhung2k3 at yahoo.com (Viet Hung) Date: Thu Aug 18 11:05:28 2005 Subject: [LARTC] Tools can capture Q-in-Q VLAN packets? Message-ID: <20050818090520.47906.qmail@web54505.mail.yahoo.com> Hi all, I'm developing Q-in-Q VLAN feature for a router software. Does any open source tools have capacity of capturing and displaying Q-in-Q VLAN packet? Thanks & Regards, Hung ____________________________________________________ Start your day with Yahoo! - make it your home page http://www.yahoo.com/r/hs From nikky at mnet.bg Thu Aug 18 11:37:46 2005 From: nikky at mnet.bg (Nickola Kolev) Date: Thu Aug 18 11:38:00 2005 Subject: [LARTC] Tools can capture Q-in-Q VLAN packets? In-Reply-To: <20050818090520.47906.qmail@web54505.mail.yahoo.com> References: <20050818090520.47906.qmail@web54505.mail.yahoo.com> Message-ID: <20050818123746.6678a094.nikky@mnet.bg> Hello, This is a proprietary feature of Cisco Systems, so I doubt that any opensource tool can capture it. Besides that this kind of technology has various implementations, f.e. Extreme Networks call it VMAN and use a different ethertype frame (AFAIR, 0x9100). On Thu, 18 Aug 2005 02:05:20 -0700 (PDT) Viet Hung wrote: > Hi all, > > I'm developing Q-in-Q VLAN feature for a router > software. > Does any open source tools have capacity of capturing > and displaying Q-in-Q > VLAN packet? > > Thanks & Regards, > Hung -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mailman.ds9a.nl/pipermail/lartc/attachments/20050818/e86cc7de/attachment.bin From dor at ldc.net Thu Aug 18 12:15:15 2005 From: dor at ldc.net (Dmytro O. Redchuk) Date: Thu Aug 18 12:15:19 2005 Subject: [LARTC] Tools can capture Q-in-Q VLAN packets? In-Reply-To: <20050818123746.6678a094.nikky@mnet.bg> References: <20050818090520.47906.qmail@web54505.mail.yahoo.com> <20050818123746.6678a094.nikky@mnet.bg> Message-ID: <20050818101514.GE5201@ldc.net> On Thu, Aug 18, 2005 at 12:37:46PM +0300, Nickola Kolev wrote: > Hello, > > This is a proprietary feature of Cisco Systems, so I doubt that any > opensource tool can capture it. Besides that this kind of technology > has various implementations, f.e. Extreme Networks call it VMAN and use > a different ethertype frame (AFAIR, 0x9100). AFAIK, every implementation should have 802.1q ethertype (0x8100)... Why not to stack packets deeper? And every outer packet should be a valid 802.1q packet. And every inner packet, if it contains another one, should also be a valid 802.1q packet. The most inner should, of course, be "plain" ethernet packet, I guess. AFAIK... Some devices allow to set ethertype for outer packet -- linux could do so too? ps. I have been looking for Q-in-Q solution for linux and have found none. Would be great, but... I'm not a programmer.. > > On Thu, 18 Aug 2005 02:05:20 -0700 (PDT) > Viet Hung wrote: > > > Hi all, > > > > I'm developing Q-in-Q VLAN feature for a router > > software. > > Does any open source tools have capacity of capturing > > and displaying Q-in-Q > > VLAN packet? > > > > Thanks & Regards, > > Hung > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc -- _,-=._ /|_/| `-.} `=._,.-=-._., @ @._, `._ _,-. ) _,.-' ` G.m-"^m`m' Dmytro O. Redchuk From stanislav.nedelchev at gmail.com Thu Aug 18 14:45:27 2005 From: stanislav.nedelchev at gmail.com (Stanislav Nedelchev) Date: Thu Aug 18 14:45:32 2005 Subject: [LARTC] Two internet lines and squid problem. Message-ID: <485817760508180545204aff01@mail.gmail.com> I have 2 internet connections and i;m trying to use squid as transparent proxy but every time squid is using first internet line but i want to use second internet line . i have this settings and without squid it's working i have default route on the first internet connection. iptables -t nat -I POSTROUTING -o eth2 -p tcp --dport 80 -s 192.168.0.0/24 -d ! 192.168.0.0/16 -j SNAT --to 217.10.248.135 /sbin/ip route add default via 217.10.248.135 dev eth2 table natips /sbin/ip rule add fwmark 66 table natips iptables -t mangle -I PREROUTING -i eth1 -p tcp --dport 80 -j MARK --set-mark 66 iptables -t mangle -A FORWARD -i eth1 -p tcp --dport 80 -j MARK --set-mark 66 I try to solve the problem moving squid to onother computer and i add additional rules like /sbin/ip route add default via 217.10.248.135 dev eth2 table natips /sbin/ip route add default via 192.168.0.11 dev eth1 table squid /sbin/ip route flush cache /sbin/ip rule add fwmark 67 table squid /sbin/ip rule add fwmark 66 table natips iptables -t mangle -I PREROUTING -i eth1 -p tcp -s 192.168.0.11 --dport 80 -j MARK --set-mark 66 iptables -t mangle -I PREROUTING -i eth1 -p tcp -d ! 192.168.0.11 --dport 80 -j MARK --set-mark 67 iptables -t mangle -A FOWARD -i eth1 -s 192.168.0.11 -p tcp --dport 80 -j MARK --set-mark 66 iptables -t mangle -A FORWARD -i eth1 -p tcp -s ! 192.168.0.11 --dport 80 -j MARK --set-mark 67 iptables -t nat -I POSTROUTING -o eth2 -p tcp --dport 80 -s 192.168.0.0/24 -d ! 192.168.0.0/16 -j SNAT --to 217.10.248.135 in this case web traffic is working but pages that uses SSL like gmail.com is not working can anybody help me to use squid like transparent proxy with 2 internet connection and to use second one. Thank in advance. From lists at wildgooses.com Thu Aug 18 15:00:30 2005 From: lists at wildgooses.com (Ed W) Date: Thu Aug 18 15:00:42 2005 Subject: [LARTC] HOWTO unmaintained? In-Reply-To: <20050817092428.531193f0@dxpl.pdx.osdl.net> References: <200508171219.54668.Andreas.Klauer@metamorpher.de> <43031F4A.1090007@wildgooses.com> <20050817092428.531193f0@dxpl.pdx.osdl.net> Message-ID: <4304866E.4020605@wildgooses.com> >>I wonder if someone would host a mediawiki and consider uploading the >>documentation there. This would make it easier for people to >>contribute, and I think it shold be fairly easy to convert from it's >>current format to a wiki >> >> >> > >Good idea, I'll see if the OSDL will put up a wiki for it >on developer.osdl.org, stay tuned. > > Can I suggest that you look very hard at Mediawiki (ie as per wikipedia). It's built for scalability, is really easy to edit, very actively maintained, and has all the distributed stuff built in if you really needed to spread the load over several machines. All in all a very grown up documentation tool. Oh and it's also multi-lingual so it might also be possible to use it to help with the translation issues I mainly mentioned it because I'm sure there are some tools for converting the current lartc faq up to the wiki format and obviously that's the first main step. If someone will host it then I have some experience maintaining a mediawiki site and could probably help out some Ed w From rmo at sunnmore.net Thu Aug 18 15:31:04 2005 From: rmo at sunnmore.net (Roy-Magne Mo) Date: Thu Aug 18 15:31:07 2005 Subject: [LARTC] Tools can capture Q-in-Q VLAN packets? In-Reply-To: <20050818090520.47906.qmail@web54505.mail.yahoo.com> References: <20050818090520.47906.qmail@web54505.mail.yahoo.com> Message-ID: <20050818133104.GS13048@slogen.sunnmore.net> On Thu, Aug 18, 2005 at 02:05:20AM -0700, Viet Hung wrote: > Hi all, > > I'm developing Q-in-Q VLAN feature for a router > software. > Does any open source tools have capacity of capturing > and displaying Q-in-Q > VLAN packet? Ethereal captures them just fine -- Roy-Magne Mo From gypsy at iswest.com Fri Aug 19 05:09:46 2005 From: gypsy at iswest.com (gypsy) Date: Fri Aug 19 05:09:25 2005 Subject: [LARTC] Two internet lines and squid problem. References: <485817760508180545204aff01@mail.gmail.com> Message-ID: <43054D7A.CCDB7207@iswest.com> Stanislav Nedelchev wrote: > > I have 2 internet connections and i;m trying to use squid as transparent proxy > but every time squid is using first internet line but i want to use > second internet line . > i have this settings and without squid it's working > i have default route on the first internet connection. > iptables -t nat -I POSTROUTING -o eth2 -p tcp --dport 80 -s > 192.168.0.0/24 -d ! 192.168.0.0/16 -j SNAT --to 217.10.248.135 > /sbin/ip route add default via 217.10.248.135 dev eth2 table natips > /sbin/ip rule add fwmark 66 table natips > > iptables -t mangle -I PREROUTING -i eth1 -p tcp --dport 80 -j MARK > --set-mark 66 > > iptables -t mangle -A FORWARD -i eth1 -p tcp --dport 80 -j MARK --set-mark 66 > > I try to solve the problem moving squid to onother computer and i add > additional rules like > /sbin/ip route add default via 217.10.248.135 dev eth2 table natips > /sbin/ip route add default via 192.168.0.11 dev eth1 table squid > /sbin/ip route flush cache > /sbin/ip rule add fwmark 67 table squid > /sbin/ip rule add fwmark 66 table natips > > iptables -t mangle -I PREROUTING -i eth1 -p tcp -s 192.168.0.11 > --dport 80 -j MARK --set-mark 66 > iptables -t mangle -I PREROUTING -i eth1 -p tcp -d ! 192.168.0.11 > --dport 80 -j MARK --set-mark 67 > > iptables -t mangle -A FOWARD -i eth1 -s 192.168.0.11 -p tcp --dport > 80 -j MARK --set-mark 66 > iptables -t mangle -A FORWARD -i eth1 -p tcp -s ! 192.168.0.11 > --dport 80 -j MARK --set-mark 67 > iptables -t nat -I POSTROUTING -o eth2 -p tcp --dport 80 -s > 192.168.0.0/24 -d ! 192.168.0.0/16 -j SNAT --to 217.10.248.135 > in this case web traffic is working but pages that uses SSL like gmail.com > is not working > can anybody help me to use squid like transparent proxy with 2 > internet connection and to use second one. > Thank in advance. I don't know anything at all about squid, but I recall a posting here regarding HTB and squid where the poster suggested a patch for squid. You might want to google "lartc squid patch" and see if there is anything that helps. You might also want to google lists.netfilter.org. -- gypsy From shemminger at osdl.org Fri Aug 19 16:52:35 2005 From: shemminger at osdl.org (Stephen Hemminger) Date: Fri Aug 19 16:51:42 2005 Subject: [LARTC] HOWTO unmaintained? In-Reply-To: <4304866E.4020605@wildgooses.com> References: <200508171219.54668.Andreas.Klauer@metamorpher.de> <43031F4A.1090007@wildgooses.com> <20050817092428.531193f0@dxpl.pdx.osdl.net> <4304866E.4020605@wildgooses.com> Message-ID: <20050819075235.6cd88f3d@localhost.localdomain> On Thu, 18 Aug 2005 14:00:30 +0100 Ed W wrote: > > >>I wonder if someone would host a mediawiki and consider uploading the > >>documentation there. This would make it easier for people to > >>contribute, and I think it shold be fairly easy to convert from it's > >>current format to a wiki > >> > >> > >> > > > >Good idea, I'll see if the OSDL will put up a wiki for it > >on developer.osdl.org, stay tuned. > > > > > > Can I suggest that you look very hard at Mediawiki (ie as per > wikipedia). It's built for scalability, is really easy to edit, very > actively maintained, and has all the distributed stuff built in if you > really needed to spread the load over several machines. All in all a > very grown up documentation tool. Oh and it's also multi-lingual so it > might also be possible to use it to help with the translation issues > > I mainly mentioned it because I'm sure there are some tools for > converting the current lartc faq up to the wiki format and obviously > that's the first main step. > > If someone will host it then I have some experience maintaining a > mediawiki site and could probably help out some Okay, if the lartc.org guys don't pick it up, I'll setup a mediawiki when I get back from holiday. For an example see: http://wiki.linux-nfs.org From riccardo at e4a.it Sat Aug 20 13:17:20 2005 From: riccardo at e4a.it (Riccardo Losselli) Date: Sat Aug 20 13:17:24 2005 Subject: [LARTC] HOWTO unmaintained? In-Reply-To: <20050817182808.206288a1@tdi.pozman.pl> References: <43031F4A.1090007@wildgooses.com> <200508172138.44503.lartc-337@ccp.com.au> <20050817182808.206288a1@tdi.pozman.pl> Message-ID: <43071140.2060907@e4a.it> > > i am totally pro refreshing the lartc howto. and wiki is a great idea Me too. i'm not active on the list but i want to contribute back... Some time ago i had the same frustation many of you had on the how-to, i was using it for some students and wanted to translate it in italian and then send back the results, but got no response. I do not read the list on a r