Question

I am researching the possibility of starting a data mining project which will include intensive calculations and transformation on data, and should be relatively easy to scale.

In your experience, is the choice of programming language critical for said project?

For example, if I am already working on a JVM environment, should I prefer Clojure over plain Java? Does the functional environment guarantee easier scalability? Better performance?

Put aside other factors such as familiarity with the language, toolchain, etc. In your experience, is the choice of language a critical one?

Answer 1

There are a few good reasons for choosing functional programming for data mining projects.

1. Usually data mining projects
   involve algorithmics and mathematics
   (than other types of systems) which
   can be more easily expressed in
   functional programming
2. Data
   mining projects would involve
   aggregate functions – which are better in functional
   programming, say Clojure
3. Data
   mining programs also would be more
   suitable to parallelism – definitely
   data parallelism and could even be
   task parallelism, again a forte of
   functional programming
4. And
   functional languages like Clojure
   can interface with java anyway for I/O, file read and write
5. I
   think one can learn the tool chain
   easily; it is not that different and so that shouldn’t be a factor.

I was asking the same question myself and came with a big Yes for Clojure – am still thinking through how to include R in the mix.