Actually - I just posted something similar - I traced my issue to the firewall in my router.
Have you tried turning off the firewall to see if that might be the issue. I'm trying to find alternative routers that have a better in-unit firewall or faster processor I don't want to put a standalone box out there for that purpose - I will if necessary - but don't want to.
I drop from 9.8Mb to 3.5Mb when the firewall is turned on (it's always on btw - I just turn it off when testing for this problem).
I'd say that the problem is bad memory management in router firmware. Basically whenever router needs to have lots of connections open it uses up its RAM which is normal.
When a connection is closed that RAM should be freed up. It looks like in your case it isn't. As router has less and less RAM available it becomes slower and slower and in the end you have to reboot it.
When it is rebooted contents of the memory are cleared and you have fresh memory to use. If that's the case then there's very little you can do. Your router may be supported by third party firmware like DD-WRT or OpenWRT (where's the rest of its name? I can't tell without it) so you could try installing one of then.
Otherwise you could hope for new version of official firmware which could fix the problem. I had a similar problem on a Netgear FVG318 router and solved it by getting a router which supports OpenWRT. I haven't had any problems since then.
Why Is My Buffalo Router So Slow All Of A Sudden?