Revert "More SkColorSpaceXformer caching"

This reverts commit 8d1fc16ba6161afc1174e29e30bae4fc9939264f.

Reason for revert: looks like it broke a couple of Chrome unit tests - https://uberchromegw.corp.google.com/i/chromium.win/builders/Win%207%20Tests%20x64%20%281%29/builds/26292

Original change's description:
> More SkColorSpaceXformer caching
> 
>   * apply(SkColorFilter*)
>   * apply(SkImage*)
> 
> Change-Id: I9b55632edd73dbbc5edb8b5ca9bb5bead1131260
> Reviewed-on: https://skia-review.googlesource.com/21736
> Reviewed-by: Mike Reed <reed@google.com>
> Commit-Queue: Florin Malita <fmalita@chromium.org>

TBR=mtklein@google.com,herb@google.com,fmalita@chromium.org,reed@google.com

Change-Id: I86bc61fca370796125b66419ad89bfe3610b996f
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://skia-review.googlesource.com/21980
Reviewed-by: Florin Malita <fmalita@chromium.org>
Commit-Queue: Florin Malita <fmalita@chromium.org>
2 files changed