I am working on a project with a group, and we are making an experimental site that involves heavy user interaction. In a nutshell, the nature of the site involves heavy user posting and commenting. Based on the theme of our site, we are expecting to get controversial posts and most likely offensive material.
My question is what algorithms, methods, etc. we can use to monitor and handle these “bad user” interactions with our website.
Right now, we have really only come up with checking the posts against a database of people, college and business names. This would make the posts anonymous somewhat and would take a sense of offense out of the post. What else should/can we implement into our design that will accomplish this?
Solution:
Everybody had really good suggestions that I’m going to research a little more. In reference to the making a list, I have been experimenting with a small script I wrote that is taking a collection of websites which contain directories of names with a substantial amount of data(3000-4000 names), and I am parsing the HTML, and storing each value in a database to be ran against the user posts. This is a little “makeshift” but it will serve as a good tester for the time being.
For some good background to the problem, with some general suggestions, check out this transcript of a speech by Clay Shirky: A group is its own worst enemy
To steal directly from the StackOverflow podcast, rate limiting is one of the most effective methods. Put reasonable limits on how much time may elapse between comments, and if the limit is exceeded, put that user into a temporary “cool-off” period where they can’t interact for a few minutes. If they keep bouncing against this limit, you may have a pathological abuser, and might cool them off for longer, ask them nicely to refrain, etc.
Rate limiting will reduce flaming because one of the primary contributors to flame wars is people get angry and start posting personal attacks rather than rational arguments. Rate limiting will reduce this behavior somewhat.
Allowing people to flag offensive material is also valuable (and only allow each user to flag an item once), but I would only show flagged items to moderators where there is a fairly high rate of flagging. You need to filter out the “background noise” because almost anything you post is going to offend someone.