I’ve forked this question from a similar question I made in a comment I made to one of the many great answers I recieved. I was originally asking about AST macros, which mostly provoked very detailed and thoughtful responses from Lispers. Thanks.
The question I made in a comment was whether project-specific DSLs are actually a good idea. Of course, this is completely subjective — After all, when you are writing in a really expressive language, where do you draw the line between an expressive API and an actual DSL? For example, I think what most Rubyists call ‘DSLs’ are actually just well-designed APIs and nothing more.
Note that I say project-specific APIs. I don’t think many will argue against using regular expressions or SQL where it makes sense to do so.
But despite this, I think we can all draw a vauge, hazy line between an API and a DSL. Of course they’re both really APIs, but whatever.
On one extreme you have Lisp, where DSLs seem to be actively encouraged via macros. On the other you have the likes of Java where DSLs are pretty much impossible.
Proponents of DSLs would argue that they increase flexibility, expressiveness, and increase consistency (for example, a custom number object using the same operators as the language’s own numbers).
Detractors would say that they can lead to sub-languages that nobody except the DSL writer knows, kills the point of having different programming languages in the first place, and leads to code nobody can understand because the way of interfacing with API is different.
I gotta say, I agree with both sides in many ways. Some Java APIs are just plain nasty due to the lack of expressiveness. Despite this, I can generally always work out what’s going on without reading the documentation — Which can’t be said about custom DSLs in the slightest. Maybe DSL proponents argue that you should always read API documentation. I disagree, but I also digress.
But let’s look at some of the big languages at the moment. C# and Java, namely. Neither of them really ‘do’ DSLs, yet they’re massively popular. Is this precisely because they don’t allow things like DSLs, allowing mediocre coders to churn out code that’s still comprehensible?
Is the fact that DSLs allow mediocre coders to produce impenetrable garbage the reason why Lisp is not used as much as it should be, despite what a DSL can look like in the right hands?
There are of course arguments in favor of DSLs and against them, and there’s of course a vague line between “a library” or “an API” and “a DSL”. That part you’ve covered well in the question, so I’ll avoid those subjective points and focus on just the question of whether they’re a liability.
A good project to consider for this is Racket which puts language construction as its main feature. It’s easy to slap up a language for any definition of “a language”: DSL or not, made up from near scratch via an interpreter or (more commonly) done via macros that define the new language and possibly a parser for a different syntax. As a result, the Racket source tree has a bunch of languages — some of them have fundamentally different execution semantics. Some examples:
Lazy Racket is what you’d expect the name to mean,
Typed Racket is a statically typed language,
Scribble is a language for writing documentation and other prose,
Slideshow is a language for writing … slideshows,
RackLog/DataLog are languages that are even more different in both semantics and syntax.
In fact, making up languages is so easy in Racket, that you easily slap up a language even if it’s something that fits a very limited use, or even just a single one. For example, we have such “small languages” that are used to create our web pages, decide which files are included in distributed installers, and many many more. See this tutorial for a description of how to come up with a language.
It’s true that there’s a fine line between a useful DSL that many people can use and one that only one person uses — but still, the kind of abstractions that you can get when you define a language rather than a library are substantial, to the point that it’s a useful concept even when it’s a “one-man’s language”. One hard problem in this are is considering interoperability — Racket allows each module to be written in it’s own language, which brings up issues of what happens when several of these modules are supposed to talk to each other. For example, how does evaluation proceeds when there’s interaction of functions in the lazy language and in the default one; or how does the typed make sure that it can interact with the default untyped language and still get the usual benefits of a statically typed language.