Question

In vim, type :sh will switch to shell, and exit can exit the shell and get back to vim. Is there any command to check if it’s in vim shell mode? So that I won’t accidentally vim to open the same file again. I want to avoid below scenario:

vim myfile > :sh > exit > vim myfile // get warning of another vim instance is editing the same file

These are the normal scenario:
vim myfile > :sh > exit // keep editing
vim myfile > :wq > vim myfile // keep editing

Answer 1

In addition to @a3nm answer, what you can do is

1. use pstree -h: it will output process tree with current branch highligted and all you need to check is whether there is vim in the highlight.
2. Another possibility is ps t: it will show all processes using the current terminal and should show vim in a list when you are inside :sh. ps -oargs t may be more useful in case you want to know arguments you ran vim with.

These methods are more reliable because VIMRUNTIME, VIM and MYVIMRC environment variables may be overrided by you in order to do some customizations (BTW, they are defined by vim for use in vimscripts, not by :sh). They also work for other processes that allow you to run a subshell, but do not define any environment variables.

I would also suggest to consider using <C-z> in normal mode or :suspend/:stop in Ex because these use shell vim was launched from instead of creating new. This behavior gives you access to history of commands you typed before running vim and also makes you able to write more complex and time-consuming shell configuration without needing to wait every time.

In case you use <C-z> both methods still work, but first method won’t highlight vim because it will be at the same level (has the same parent) as pstree itself, likely just below or above pstree in graph. This also enables third method: jobs shell builtin.

In order to restore from <C-z> you should use fg (a single % in zsh and bash also works) which is less to type then exit (but more then <C-d>).