If you are having problems with collections taking a long time to refresh membership, it is very possible that you have incremental updates or collection membership evaluation schedule set of happen frequently on a large number of collections.
Rule of thumb: