I have NSSplitView with divider style set as thin and it looks great.

However, if I replace one of a subviews with this line of code
[self.horizontalSplitView replaceSubview:[[self.horizontalSplitView subviews] objectAtIndex:0] with:myGreatView];
The divider looks kind of bad, how can I fix this issue?

The problem is that your view (
myGreatView) has a border. Set the border type toNSNoBorder.