[Improve] use token_get_all instead of tokenize method

[tinpont]

Thanks for your awesome project.

Can we improve performance by replacing tokenize to token_get_all ?

tinpont@local:~/expression-test$ php -f compare.php
token_get_all: 0.077315092086792
StringCalc: 2.0649530887604

Source code:

<?php

require 'vendor/autoload.php';

$limit = 100000;
$expression = 'INT(W*0.013)*0.014';
$startTime = microtime(true);
for ($i = 0; $i < $limit; $i ++) {
    $tokens = token_get_all("<?php $expression;");
}
echo 'token_get_all: ' . (microtime(true) - $startTime) . PHP_EOL;

$stringCalc = new ChrisKonnertz\StringCalc\StringCalc();
$startTime = microtime(true);
for ($i = 0; $i < $limit; $i ++) {
    $tokens = $stringCalc->tokenize($expression);
}
echo 'StringCalc: ' . (microtime(true) - $startTime) . PHP_EOL;

[chriskonnertz]

Hello,
did you experience actual performance issues when using this library?

[tinpont]

In fact it did not. But I think that it still have a chance to improve performance.