hwc/overlay: Fix decimation / bwc bugs
1) While setting BWC check for 90 transform and swap crop
2) Calculate decimation in the getDecimationFactor API itself.
Currently, a residual downscale is calculated and the clients have
to take logs and apply forcible decimation like in MdpCtrl
3) Fix bug where transform gets typecasted to bool
b/14225977
Change-Id: I3c99c571e02e2cf7822342516b6a87d97be553d1
diff --git a/liboverlay/overlayUtils.h b/liboverlay/overlayUtils.h
index c2649f3..39507f2 100644
--- a/liboverlay/overlayUtils.h
+++ b/liboverlay/overlayUtils.h
@@ -413,8 +413,8 @@
int getDownscaleFactor(const int& src_w, const int& src_h,
const int& dst_w, const int& dst_h);
void getDecimationFactor(const int& src_w, const int& src_h,
- const int& dst_w, const int& dst_h, float& horDscale,
- float& verDscale);
+ const int& dst_w, const int& dst_h, uint8_t& horzDeci,
+ uint8_t& vertDeci);
/* flip is upside down and such. V, H flip
* rotation is 90, 180 etc