To begin, not only are there two main dialects of the language (Common Lisp and Scheme), but each of the dialects has many individual implementations. For example, Chicken Scheme, Bigloo, etc… each with slight differences.
From a modern point of view this is strange, as languages these days tend to have definitive implementations/specs. Think Java, C#, Python, Ruby, etc, where each has a single definitive site you can go to for API docs, downloads, and such. Of course Lisp predates all of these languages. But then again, even C/C++ are standardized (more or less).
Is the fragmentation of this community due to the age of Lisp? Or perhaps different implementations/dialects are intended to solve different problems? I understand there are good reasons why Lisp will never be as united as languages that have grown up around a single definitive implementation, but at this point is there any good reason why the Lisp community should not move in this direction?
The Lisp community is fragmented, but everything else is too.
Why are there so many Linux distributions?
Why are there so many BSD variants? OpenBSD, NetBSD, FreeBSD, … even Mac OS X.
Why are there so many scripting languages? Ruby, Python, Rebol, TCL, PHP, and countless others.
Why are there so many Unix shells? sh, csh, bash, ksh, …?
Why are there so many implementations of Logo (>100), Basic (>100), C (countless), …
Why are there so many variants of Ruby? Ruby MRI, JRuby, YARV, MacRuby, HotRuby?
Python may have a main site, but there are several slightly different implementations: CPython, IronPython, Jython, Python for S60, PyPy, Unladen Swallow, CL-Python, …
Why is there C (Clang, GCC, MSVC, Turbo C, Watcom C, …), C++, C#, Cilk, Objective-C, D, BCPL, … ?
Just let some of them get fifty and see how many dialects and implementations it has then.
I guess Lisp is diverse, because every language is diverse or gets diverse. Some start with a single implementation (McCarthy’s Lisp) and after fifty years you got a zoo. Common Lisp even started with multiple implementations (for different machine types, operating systems, with different compiler technology, …).
Nowadays Lisp is a family of languages, not a single language. There is not even consensus what languages belong to that family or not. There might be some criteria to check (s-expressions, functions, lists, …), but not every Lisp dialect supports all these criteria. The language designers have experimented with different features and we got many, more or less, Lisp-like languages.
If you look at Common Lisp, there are about three or four different active commercial vendors. Try to get them behind one offering! Won’t work. Then you have a bunch of active open source implementations with different goals: one compiles to C, another one is written in C, one tries to have a fast optimizing compiler, one tries to have some middlle ground with native compilation, one is targeting the JVM … and so on. Try to tell the maintainers to drop their implementations!
Scheme has around 100 implementations. Many are dead, or mostly dead. At least ten to twenty are active. Some are hobby projects. Some are university projects, some are projects by companies. The users have diverse needs. One needs a real-time GC for a game, another one needs embedding in C, one needs only barebones constructs for educational purposes, and so on. How to tell the developers to keep from hacking their implementation.
Then there are some who don’t like Commmon Lisp (too big, too old, not functional enough, not object oriented enough, too fast, not fast enough, …). Some don’t like Scheme (too academic, too small, does not scale, too functional, not functional enough, no modules, the wrong modules, not the right macros, …).
Then somebody needs a Lisp combined with Objective-C, then you get Nu. Somebody hacks some Lisp for .net. Then you get some Lisp with concurrency and fresh ideas, then you have Clojure.
It’s language evolution at work. It is like the cambrian explosion (when lots of new animals appeared). Some will die, others will live on, some new will appear. At some point in time some dialects appear that pick up the state of art (Scheme for everything with functional programming in Lisp in the 70s/80s and Common Lisp for everything MacLisp-like in the 80s) – that causes some dialects to disappear mostly (namely Standard Lisp, InterLisp, and others).
Common Lisp is the alligator of Lisp dialects. It is a very old design (hundred million years) with little changes, looks a little bit frightening, and from time to time it eats some young…
If you want to know more, The Evolution of Lisp (and the corresponding slides) is a very good start!