Refactoring to remove some duplication of probability
tables between tokenization and detokenization.

Change-Id: I2fc6a6497f9c0410021a9b41f828bc58a864e466