Question

I’m using the congomongo library in two processes. One of the processes interfaces with the databases without error. The other process is having more trouble. If I try to interact with the database from that processes slime repl, it too is has no problem, but if I try to interact through telnet things work badly.

Command Line:

joshua@joshua-Aspire-5251:/data/db$ telnet localhost 4576
Trying ::1...
Connected to localhost.localdomain.
Escape character is '^]'.
http://news.ycombinator.com/item?id=2286907
Connection closed by foreign host.

Server’s Reaction

http://news.ycombinator.com/item?id=2286907
Processing link.
Validating link.
Scraping item.
Wikifying items.
Uploading items.

From the REPL:

(process-link "http://news.ycombinator.com/item?id=2286907")

Server’s Reaction:

http://news.ycombinator.com/item?id=2286907
Processing link.
Validating link.
Scraping item.
Wikifying items.
Uploading items.

The only functional difference? One actually puts the items in the database and the other doesn’t. Both seem to be accomplishing everything else.

(ns hnparser.main
  (:use somnium.congomongo
    [hnparser.core :only [scrape-item]]))

(mongo! :db "hacker-archives")

(defn valid-link?
  [link]
  (do
    (println "Validating link.")
    (not= nil (re-matches #"http://news\.ycombinator\.com/item\?id=\d+" link))))

(defn wikify [item]
  (dissoc (assoc item :wiki [{:title (:title item)
                  :author (:user item)
                  :body (:body item)
                  :date (:date item)
                  :reason "Original post."}])
      :title :user :body :date))

(defn wikify-items [items]
  (do
    (println "Wikifying items.")
    (map wikify items)))

(defn upload-item
  [item]
  (if (nil? (fetch-one :items :where {:id (:id item)}))
    (insert! :items (assoc item :scrape-date (java.util.Date.)))))

(defn upload-items
  [items]
  (do
    (println "Uploading items.")
    (map upload-item items)))

(defn process-link
  [link]
  (do
    (println "Processing link.")
    (if (valid-link? link)
      (upload-items (wikify-items (scrape-item link))))))


(ns hnparser.server
  (:use clojure.contrib.server-socket
    hnparser.main)
  (:import [java.io PushbackReader InputStreamReader]))

(defn read-links
  [is os]
  (let [in-reader (PushbackReader. (InputStreamReader. is))]
    (let [input (str (read in-reader))]
      (do
    (println input)
    (process-link input)))))

(defn -main
  []
  (def *server* (create-server 4576 read-links)))

Note that this successfully scrapes (verified through print statements). It just doesn’t upload to the database.

Answer 1

Map is lazy: you should never use map when you want side effects instead of return values. You can wrap a map with dorun, but in most cases it’s easier to use doseq.

Also defn has an implicit do, so (fn [x] (do (println x) (inc x))) is the same as (fn [x] (println x) (inc x)).