Below UI is something I’d like to aim for. But I have no idea, how they have the “skin” of the app. On my end, the Java application looks like it was made in 1990s. I want to change a look to a more modern style.
What components are they using possibly here? JSplitpane for one. but I’m not sure how they created that “Dokument/Vorschau” tabs.

First using a look and feel like Nimbus, can easily change the look of your application to a have a more modern feel.
Second, you will need to customize the font, color, borders, node icons, etc. of each component type to achieve a non-standard look. Some of these can be changed with updates to the
UIDefaultsof the look and feel but many will be made by calling methods on the specific instance of the component you are dealing with.