Storing a text document

OK - I dont do projects. Usually I end up not completing them if they are too big, and I have tonnes of non-reusable code. Thus I do "libraries" working toward a goal. If goal is not met, I still have lots of useful libraries left over. What I want to do is read in a text file into an easily modifyable structure in memory. For this I have written a mini regular expression parser (ill be adding it to my site soon - if you want the souce, tell me and ill send it to you). What I considered was making bunches of tokens which would be linked together to form lines, which would be linked together to form a document. Thus for editing purposes, it is easy to add lines/move lines, and do searches for phrases. Each token is going to be a regular expression, and only accept characters that fit the expression. If a character is to be inserted at some position in the token, and the character does NOT match the expression.. then the token is split into 2 tokens. (One token for whats before where the character is being inserted, and one for after). Then the character is tried to be inserted at the end of the first token, or start of the second. If it cant, it must become a new token. Yes it sounds like massive work, but the above is now easy to implement. It means I can have a token which accepts all whitespace, and then one which accepts decimal values, and one which accepts hexadecimal values (0x????) The problem comes in when a token is split into multiple tokens, and this occurs say a few times. Then the user deletes the characters. The program must find a way of combining the tokens again back into their original form. One way is to just backtrack a bit, take the text from there onward for some number of bytes and reconvert it into tokens. This would work, but would be quite a bit of processing IMHO just for a backspace. (not that a person usually backspaces fast enough to chomp up even 5% of their CPU)... Another issue is how far back do you start reconverting the text? If you go back 1 token, it could just happen that going back 2 tokens would mean a whole bunch of tokens just becoming 1. And with that in mind, how many bytes do you choose to reconvert? Or how many tokens forward does it look in "optimising" multiple tokens into as few as possible? --- In a slightly unrelated question: What if the user types "\*" ? (assuming its used for editing code) Then the program has to go through ALL tokens setting a flag in each one to mark it as being part of a comment? This would be acceptable in small files, but if the program file is 200k, then you dont want the user waiting 3 or 4 secs while the whole program is updated to be commented out ? Ofcourse a lot of this optimisation and parsing could be done in another thread. But that means even when the editor is idle, there is a constant CPU use for the parsing, optimising and such. I have been grappling with this problem for a week or so now, and every solution I can think of has a fast use in one scenario, but then makes up for it by being quite slow in another (ie: adding and then removing characters). Any comments greatly appreciated!! regards, GeniX