Tremolo: Fix integer overflows in codebook.c am: bbf9d256df am: e1fea432c9

Original change: https://googleplex-android-review.googlesource.com/c/platform/external/tremolo/+/14336180

Change-Id: I5f8a874590303feb037c7648160d500a667f98d5