Adding attributes to a uniqued set has become expensive, don't do it more often
than necessary. No functionality change.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@181662 91177308-0d34-0410-b5e6-96231b3b80d8