Improving tokenizer performance by initialising input array beforehand

[BlackBuck]

The tokenizer function was computing the length of the input on the fly i.e. for current < len([]rune(input)). While it's not that big of a deal, it forces the compiler to re-compute the array for each iteration of the loop. Instead, what I did was that I calculated the array beforehand and then passed it to the loop as well as the line that required the current character in the input.