Question

I’ve had a long standing interest in developing an OS UI tailored to my needs and interests. When I was younger, I went crazy playing with the Stardock skinning tools for Windows. I loved developing my own window styles and layout ideas. Since then I’ve had an unscratchable itch.

• I’m not particularly interested in fiddling with the lower level kernel operations, just the presentational aspects. I’ve been considering some flavor of Linux.
• While I would love to have some basic integrated tools/applications that I create myself, I’m primarily interested in the presentations of basic windowing, toolbars, desktop, and file management.
• I don’t want to be locked out from using existing applications, if possible. I, for example, have no interest in writing my own web browser, but having the ability to work existing tools into a design would be a major plus.
• My ideas are sometimes very visual in nature, so the ability to produce more complex transparency effects and unusual window shapes would be useful. More-or-less the ability to reproduce the graphic fidelity of current UI designs for OS X or Windows would be ideal.
• An OS flavor and tools that lets me start with a blank screen and work from the ground up or include basics are encouraged. The idea of a “desktop” in the traditional sense shouldn’t be mandatory. Flexibility to do different things is important to me.

What I’m looking for:

I’m interested in what Linux/OS flavors might make sense, what tools / frameworks would be useful for accomplishing the task ( Qt seems to be a popular suggestion here ), and basic info on how / where one might start developing / testing such a UI. I’d also be interested reading about any “personal” UI’s that people may have developed.

This is an interest far outside my normal development work, so as a disclaimer, forgive my naivete if I appear to be confused. Corrective insight welcomed.

I realize this is a big question, so thanks to those for taking the time to make suggestions.

Answer 1

You want to build your own DE (desktop environment). Common examples are GNOME and KDE, although more of them were popular. It actually used to be very popular to build your own DE, and dedicated scripting tools were often created.

You will usually need to combine multiple tools: a window manager, a toolbar program, a desktop manager (icons), session manager, possibly more. Out of that only a window manager is a required part (it usually doesn’t make any sense not to run any WM inside X server), and others are optional. You will usually not write these tools from scratch (it is a lot of work), but use already available components.

Note that both GNOME and KDE actually consists of these elements (well integrated). For example GNOME consists of a window manager (metacity), toolbars (gnome-panel), desktop manager (nautilus) and so on. You can change one element into another if you want: it used to be popular to use sawfish as a window manager instead of metacity, keeping all the other elements intact. Now it is popular to use compiz instead of metacity.

I used to use FVWM, which is a window manager with an additional (quite good) scripting capabilities that could be used to create toolbars, menus and other things. One of nice examples is FVWM-Crystal, which is a complete DE built on FVWM and few other tools, written in a mix of FVWM scripting and Python. You could actually change any kind of behaviour at runtime by just opening FVWM’s scripting console and typing commands. FVWM-Crystal can be a good starting point if you want to start from something already usable; bare FVWM is good if you want to build your DE from scratch.

[UPDATE: it seems that gnome-shell is very similar in this regard to FVWM. Most of its behavior is scripted using JavaScript (instead of a custom language in FVWM), and is very easy to change. It is also very actively developed, as it is an important part of GNOME3, with very good support for composition. It’s a hacker’s heaven ;-)]

[UPDATE2: so it seems I was right on gnome-shell. There’s a Cinnamon, which uses the same framework (mutter window manager) as gnome-shell, yet builds a desktop which works in a different way than gnome-shell.]

There was a website that was a good source of ideas to implement: lynucs.org. It is down for two years now, but it got archived.

Warning: lots of these components are not maintained any more, especially since GNOME/KDE started to become actually usable. Development in this area is usually driven more by specific requirements of embedded devices: phones or netbooks.

Now, to answer your actual questions:

• Linux/OS flaver: anything easily customizable: Debian, Arch, Gentoo, LFS…
• Tools/frameworks: it depends on what do you want to achieve. If you want to write a custom toolbar you might just use some scripting languages (like FVWM’s), custom tools like adesklets, or write your own in Qt (if most of your other components is in Qt) or GTK (if most of your other components is in GTK).
• Personal UIs: there were lots of them, and sites like lynucs.org was a nice compilation of them (lynucs listed components used in each desktop).
• Complex transparency effects: older tools usually do not handle it natively, or do a simple things like just reusing part of desktop wallpaper as its own background (so-called fake transparency). Most probably you’ll need to write your own code, f.e. as a plugin to compiz.