Currently the main LoadState state machine iterates through all LoadStates every frame. Generally it only needs to iterate over states that may change. I think avoiding processing of Loaded assets will be sufficient, and can probably be achieved by processing only when a ref is added or removed.