Well, custodians are not really a workaround -- they're intended just for situations where you want to close up all resources that belong to a server thread, for example.
But in any case, I don't know about such a leak... AFAIK, if you're closing the port, then everything should work properly. Is there some example code that shows the problem? (Better to mail me, btw, since keeping track of this thing is difficult and there's no email alerts...)