Original Post
Hi all. I'm working on a game, and in my game I have written a custom GLSL preprocessor to add some new features not supported by barebones GLSL (input/output semantics kind of stuff). To correctly implement this I need the ability to interpret #define/#ifdef/#if on the C++ side of my program and not rely on my OpenGL driver's preprocessor, so I've written a basic preprocessor myself to run against the GLSL source code before uploading it.
My complaint right now is that it's really painfully slow, even for something that only needs to run occasionally. I've profiled the issue down to the tokenizer, I'm using std::tr1::regex to tokenize the source code, and its often taking something like 100+ ms to tokenize a 30-50 line shader, which seems pretty extreme.
My caveat is that I'm running with debug optimizations on, but seeing as how I'm still in development I kind of need that. I'm sure that the regex performance would be hugely better in release mode, but right now it's just annoying me when my program locks up for 500ms every time I need to compile a few new shaders with some different defines set.
I'm trying to come up with some ways to make this less painful, I'm thinking something along the lines of:
1) Rewrite the code to not use regex. This would probably be a lot faster as I'm guessing tr1::regex is some super-templated/iterator'd mess that makes things slow when it has to run with debug information. I'd be kind of disappointed to do this as it seems a much less elegant solution.
2) Some visual studio option to compile the regex to run with speed optimization, while letting the rest of the program run in debug mode? I'm not really aware of what options are available here, maybe someone could explain if there's any way to do something like this. I tried enabling /O2 (optimize for speed) by right clicking on the .cpp file that contained all of the regex searches, but it didn't make a difference in the regex speed, so I guess that didn't do much.
This is the code I used to tokenize the file, I think it is not too complex
My complaint right now is that it's really painfully slow, even for something that only needs to run occasionally. I've profiled the issue down to the tokenizer, I'm using std::tr1::regex to tokenize the source code, and its often taking something like 100+ ms to tokenize a 30-50 line shader, which seems pretty extreme.
My caveat is that I'm running with debug optimizations on, but seeing as how I'm still in development I kind of need that. I'm sure that the regex performance would be hugely better in release mode, but right now it's just annoying me when my program locks up for 500ms every time I need to compile a few new shaders with some different defines set.
I'm trying to come up with some ways to make this less painful, I'm thinking something along the lines of:
1) Rewrite the code to not use regex. This would probably be a lot faster as I'm guessing tr1::regex is some super-templated/iterator'd mess that makes things slow when it has to run with debug information. I'd be kind of disappointed to do this as it seems a much less elegant solution.
2) Some visual studio option to compile the regex to run with speed optimization, while letting the rest of the program run in debug mode? I'm not really aware of what options are available here, maybe someone could explain if there's any way to do something like this. I tried enabling /O2 (optimize for speed) by right clicking on the .cpp file that contained all of the regex searches, but it didn't make a difference in the regex speed, so I guess that didn't do much.
This is the code I used to tokenize the file, I think it is not too complex
static const std::tr1::regex tokenRegexFull("([+\\-=*/<>^%!&|]+|[()]|[#\\w_]+|\\s+)");
static const std::tr1::regex tokenRegexSimple("([\\w]+)");
void Tokenizer::TokenizeString(std::string string, std::vector<Token>& tokens){
std::match_results<std::string::const_iterator> result;
bool simple = string.find("#") == std::string::npos; //if no preprocessor directives, use the simple regex to look for words only
while(string.length() != 0){
Token token;
if(std::tr1::regex_search(string,result,simple ? tokenRegexSimple : tokenRegexFull)){
token.text = result.prefix();
if(token.text.length() > 0){
tokens.push_back(token);
}
token.text = result[1].str();
tokens.push_back(token);
string = result.suffix();
} else {
token.text = string;
tokens.push_back(token);
break;
}
}
tokens.push_back(newlineToken);
}
