🔒 Locked

Tokenizing infinite loop

Started by Endar Jan 9, 2005 at 2:53 AM 6 replies 1.3k views

Original Post

Jan 9, 2005 at 2:53 AM


// tokenize
while( index != string::npos ){
	// push word
	string::size_type next_space = m_commandLine.find(' ');
	arguments.push_back(m_commandLine.substr(index, next_space) );

	// increment index
	if( next_space != string::npos )
		index = next_space + 1;
	else
		break;
}

Now, with this, if m_commandLine has a space character in it, then it goes into an infinite loop. Can anyone give me a quick tip that doens't include me scrapping the function and writing my own?

Kibble

Jan 9, 2005 at 3:02 AM

You aren't changing where it starts looking from, so it will find the first space every time. Try this:

// tokenizewhile( index != string::npos ){	// push word	string::size_type next_space = m_commandLine.find(' ', index);	arguments.push_back(m_commandLine.substr(index, next_space) );	// increment index	if( next_space != string::npos )		index = next_space + 1;	else		break;}

Fruny

Jan 9, 2005 at 4:13 AM

If your delimiter is just whitespace, there is a simple way to tokenize, by using a stringstream and istream_iterators.

#include <iostream>#include <vector>#include <iterator>#include <sstream>int main(){   std::string str = "Now, with this, if m_commandLine has a "                     "space character in it, then it goes "                     "into an infinite loop.";   // stream containing the string   std::istringstream iss(str);   // iterators reading std::string objects off the specified stream   std::istream_iterator<std::string> iss_begin(iss);   std::istream_iterator<std::string> iss_end;   // initialize the vector with the data contained in the iterator range   std::vector<std::string> tokens(iss_begin, iss_end);   for(size_t i=0; i<tokens.size(); ++i)      std::cout << "Token #" << i << " = " << tokens << std::endl;}

Endar

Jan 9, 2005 at 11:39 PM

int main(){   std::string str = "Now, with this, if m_commandLine has a "                     "space character in it, then it goes "                     "into an infinite loop.";	// stream containing the string	istringstream iss(str);	// iterators reading std::string objects off the specified stream	istream_iterator<string> iss_begin(iss);	istream_iterator<string> iss_end;	// initialize the vector with the data contained in the iterator range	vector<string> tokens(iss_begin, iss_end);    ///////// Error at this line	for(size_t i=0; i<tokens.size(); ++i)		cout << "Token #" << i << " = " << tokens << endl;return 0;}

I tried to compile this code from Fruny, but I'm getting an error "can't convert parameter 1 from 'istream_iterator' to 'unsigned int'"

I've never used these functions before, and while I've read on MSDN about them, thats about it.

Zahlman

Jan 10, 2005 at 12:24 AM

Strange, it looks OK to me. Anyway here's a non-iterator version:

#include <iostream>#include <vector>#include <sstream>int main() {  std::string str = "Now, with this, if m_commandLine has a "                    "space character in it, then it goes "                    "into an infinite loop.";  // stream containing the string  std::istringstream iss(str);  std::string token;  // initialize the vector with the data contained in the iterator range  std::vector<std::string> tokens;  while(iss >> token) tokens.push_back(token);  for(size_t i=0; i<tokens.size(); ++i)    std::cout << "Token #" << i << " = " << tokens << std::endl;}

Fruny

Jan 10, 2005 at 12:49 AM

Quote:
Original post by Endar
I tried to compile this code from Fruny, but I'm getting an error "can't convert parameter 1 from 'istream_iterator' to 'unsigned int'"

You are using a crappy library (guess: VC6), which doesn't have the appropriate overloads for the constructor. Try using assign() instead.

vector<string> tokens;tokens.assign(iss_begin, iss_end);

Endar

Jan 10, 2005 at 1:17 AM

Thanks. :D

I don't know whats wrong with me. I should have seen that.

Anyway, thanks Zahlman and Fruny.

nprz

Jan 10, 2005 at 1:22 AM

Quote:
Original post by Fruny
Quote:
Original post by Endar
I tried to compile this code from Fruny, but I'm getting an error "can't convert parameter 1 from 'istream_iterator' to 'unsigned int'"

You are using a crappy library (guess: VC6), which doesn't have the appropriate overloads for the constructor. Try using assign() instead.
vector<string> tokens;tokens.assign(iss_begin, iss_end);

It compiled fine on my VC6. Maybe the SP fixed it.

Topic Locked

This topic has been locked by a moderator. New replies are not allowed.