I wouldn't think you're hurting anything by just leaving it. If you see the leaves start to deteriorate, you could cut them then. The plant knows best and maybe by leaving them, you'll get quicker growth next year. I cut mine back but I'm in Mi where winter is more serious. But as Mrs C said; wouldn't hurt to cut, either. It's a good idea to fertilize if you're going to clean up as it'll provide better flowering next season. I use Osmocote when I repot/trim them up.