🔒 Locked

[web] Web based search engine wiki

Started by Nice Coder Apr 13, 2006 at 11:57 PM 4 replies 1.1k views

Original Post

Apr 13, 2006 at 11:57 PM

I'm currently developing a search engine, using the "brain" off my latest bot, and a few ideas off wikipedia. The gist if it is that people add entries in a wiki like fasion, as a responce to a question. When a user then asks the site a question, it looks through the answers it knows, and responds with a couple that should be the best. Link: www.searchbase.uni.cc The problems that i'm having come in about three groups. One -> Recognising particular individuals (Can't use cookies, because i am using cgi, and i can't find the way to access them. Hidden values could be hacked) Two -> Recognising and changing particular posts. (i'm storing the database as a file, querying it with cgi. Maybe an sql server?) Three -> Stoping people from spamming the site, and using proxies to bypass bans. Any comments? Suggestions? [Edited by - Nice Coder on April 14, 2006 12:29:10 AM]

Sander

Apr 14, 2006 at 12:16 AM

Quote:
Original post by Nice Coder
One -> Recognising particular individuals (Can't use cookies, because they are cgi. Hidden values could be hacked)

Eh? Care to explain? Cookies are CGI? Anyway, sounds like you need support for browsing sessions.

Quote:

Two -> Recognising and changing particular posts. (i'm storing the database as a file, querying it with cgi. Maybe an sql server?)

Yes, a database would be easier than a flat file, especially when the size of the database increases or it's structure gets more complicated.

Quote:

Three -> Stoping people from spamming the site, and using proxies to bypass bans.

There are lists of public proxies. You could use those and disallow access from public proxies based on IP. As for stopping spamming on publicly editable documents, that's hard to do. You could do some keyword evaluation to catch the most common forms of smap (porn et. al.) but it's mostly moderating. Add a "this search result is spam" button so users can report spam to you.

Nice Coder

Apr 14, 2006 at 12:32 AM

Hey sander.

Hmmm. What i meant to say is that i'm using cgi, and the library that i'm using to use cgi doesn't have any way to use cookies. (that i've seen).

As per proxies, i was thinking of portscanning people as they log in. Just a few common ports and a couple hundred random ones.

As per moderation, how does wikipedia do their moderation? They don't seem to have any controlls over who edits what...

markr

Apr 14, 2006 at 5:39 AM

Quote:
Original post by Nice Coder
Hmmm. What i meant to say is that i'm using cgi, and the library that i'm using to use cgi doesn't have any way to use cookies. (that i've seen).

Cookies can be used fine in CGI- they are just headers like anything else (Cookie: for receiving cookies and Set-Cookie: for setting them).

However it's annoying without a library because non-session cookies require an RFC822 date, which is a silly format.

Mark

Nice Coder

Apr 14, 2006 at 6:50 AM

Hmmm.

In the library that i'm using (which basically sits between the application and the webserver, so i get a nice civilised interface to work with.

The only problem is that there is no way to grab the cookie header as its beings sent to the server :(

Any other ways? (like ip tagging?)

Sander

Apr 14, 2006 at 11:44 AM

Quote:
Original post by Nice Coder
As per proxies, i was thinking of portscanning people as they log in. Just a few common ports and a couple hundred random ones.

Excuse me?! That's the most offensive thing I've ever heared. Plus, you'll run into tarpits.

Quote:
As per moderation, how does wikipedia do their moderation? They don't seem to have any controlls over who edits what...

No they don't. Since everyone edits things, anyone can also remove spam and junk. It works for wikipedia because there are lots of users willing to contribute positively.

Topic Locked

This topic has been locked by a moderator. New replies are not allowed.