Move token length calculation out of the diagnostics machinery into 
the lexer, where it can be shared.


git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@43090 91177308-0d34-0410-b5e6-96231b3b80d8
4 files changed