I want a project directory like this:
app/ | src/ Makefile
All of the .cpp and .hpp, .h files are in the src subdirectory. How can I configure Makefile.am or configure.ac to let the generated Makefile using the directory /src as the source directory? And how to let the generated executable into a specified directory like /usr/opt?
Another question is that If I want to make my program as a .so library, how to achieve my goal to make it install into /usr/lib or /usr/local/lib? And how to make the header files into like /usr/local/include/app?
Usually, you just add
to the toplevel Makefile.am, and mention src/Makefile in AC_CONFIG_FILES. There are some reasonable complaints about recursive make, but if all of your source is in src then the single level of recursion in make is not really an issue. Putting the executable in /usr/opt is done by the user when setting prefix at configure time, and should not be a consideration when building the package. To build an executable, use bin_PROGRAMS in the Makefile.am, and to build a library you should use either lib_LIBRARIES or lib_LTLIBRARIES (personally, I tend to use libtool and go with the latter.)
I think your questions about installing in /usr/lib or in /usr/opt indicate a fundamental misunderstanding. When building the package, you absolutely do not care about that. As the package maintainer, you indicate that the program should be installed in $(prefix)/bin (or $exec_prefix/bin, or $bindir) and that libraries should be in $libdir, but the user determines the location of $prefix and $libdir and $bindir when they run configure. (Usually, people just set prefix, and bindir defaults to $prefix/bin while libdir defaults to $prefix/lib.)