Partly revert ad3824a90261 and add comment about reference ownership