Original Post
Hello
I am trying to figure out if I could make my algorithm more efficient for the problem below.
I am setting myself a task of parsing emails that are about a specific computer problem & then suggesting the possible solution according to the key words in the email.
But I am unsure how I can make this efficient so I have these questions:
- For each email I parse; will I have to compare EVERY solutions tag list to my emails inverted index OR can I weed some out so I dont have a 0^n algorithm?
- For each comparision between a solutions tag list & the emails inverted index will I have to will I have to check & compare each element in the tag list to each element in the inverted_index (so the time will = tag_list_size * inverted_ind_size)?
Also I am writting this in python, so a solution is a class containing a tuple of string tags. And the emails inverted index is a Dictionary.
Algorithm:
- I was thinking of having each tag as a constant & each tag has a numerical value... eg, switch = 20000
- Then each solutions tag list will be ordered in ascending numerical order
- Then each solutions tag list will have a hash value that is the addition of all tags put together
- I will store my solution objects in a list & arrange them in ascending hash value order
- Then I will generate the emails hash value by iterating(0^n speed) over the emails inverted index, & for any constant tags identified I will add its value to the emails hash value. The following refers to building the emails hash value. Will this mean if I have 250 constant tag words, for every word in email I will have to perform 250 checks therefore to generate the emails hash value it will take 250 * inverted_index.size()? That isn;t efficient is it? Is there a better way to find all the tags that occur in the emails inverted index?
- Using the emails hash value I will then search the solution_list to find the 5 closest solution hash values & these solutions will become my suggested solutions. Can you recommend a algorithm to do this step
Are there easier algorithms I could use, & do you think my algorithm can improve or needs to change anywhere?
I am trying to figure out if I could make my algorithm more efficient for the problem below.
I am setting myself a task of parsing emails that are about a specific computer problem & then suggesting the possible solution according to the key words in the email.
Quote:
So my overall algorithm is to parse each email, build an inverted index of the words in the email(where the key will be the word, & the value will be the number of times that word appears in the email). And I also have a list of solutions & each solution has some tag words related to it; so for example a Router Solution would contain the tags ["router", "internet", "switch", "connection"...]
I will then iterate through my list of solutions & compare the solution tag array to my emails inverted index. The solution that has the most number of hits (highest number of similar words in the email) will be my suggested solution.
But I am unsure how I can make this efficient so I have these questions:
- For each email I parse; will I have to compare EVERY solutions tag list to my emails inverted index OR can I weed some out so I dont have a 0^n algorithm?
- For each comparision between a solutions tag list & the emails inverted index will I have to will I have to check & compare each element in the tag list to each element in the inverted_index (so the time will = tag_list_size * inverted_ind_size)?
Also I am writting this in python, so a solution is a class containing a tuple of string tags. And the emails inverted index is a Dictionary.
Algorithm:
- I was thinking of having each tag as a constant & each tag has a numerical value... eg, switch = 20000
- Then each solutions tag list will be ordered in ascending numerical order
- Then each solutions tag list will have a hash value that is the addition of all tags put together
- I will store my solution objects in a list & arrange them in ascending hash value order
- Then I will generate the emails hash value by iterating(0^n speed) over the emails inverted index, & for any constant tags identified I will add its value to the emails hash value. The following refers to building the emails hash value. Will this mean if I have 250 constant tag words, for every word in email I will have to perform 250 checks therefore to generate the emails hash value it will take 250 * inverted_index.size()? That isn;t efficient is it? Is there a better way to find all the tags that occur in the emails inverted index?
- Using the emails hash value I will then search the solution_list to find the 5 closest solution hash values & these solutions will become my suggested solutions. Can you recommend a algorithm to do this step
Are there easier algorithms I could use, & do you think my algorithm can improve or needs to change anywhere?