| # |
| # Comments should be ignored |
| # |
| -option key # end of line comment |
| +reset imbedded#hash # <- not a comment, thought this is |
| |
| This\ is' a 'single" token" |
| |
| And\\\ \''even '\'\""more \""complex |
| |
| "Backslash chars \n are returned as is" |
| 'regardless \n of quoting' |
| |
| 'Single quote escapes' |
| \' "'" |
| |
| "Double quote escapes" |
| \" '"' "\"" |
| |
| Back\ slash\ escapes |
| \\ '\' "\\" # NOTE that backslash in single quotes are literial! |
| |
| 'Space Character Escapes' |
| \ ' ' " " |
| |
| 'Empty Tokens, using quotes' |
| '' "" |
| |
| "Unicode charcaters are handled" |
| "° ' ² ³ ` ´" |
| "µ ¶ ⨀ ⨁ ⨂" |
| 测试用的汉字 |
| |
| Lines__\ |
| __Continuation |
| |
| 'quoted_newlines__ |
| __are_part_of_token' |
| |
| "Last Token before EOF" |
| |
| |