added support for token stream filtering and preprocessing.

--HG--
branch : trunk
7 files changed