Question

how can I call a tinymce plugin function?

 tinymce.activeEditor.plugins.customplugin.customfunction(customvar);

not working!

Answer 1

tinymce.activeEditor.plugins.customplugin.customfunction(customvar);

is the correct way to call such a function.
Be aware that tinymce.activeEditor needs to be set already in order to use it.
tinymce.activeEditor gets set when the user clicks into the editor for example.
Otherwise use

tinymce.get('your_editor_id_here').plugins.customplugin.customfunction(customvar);

There might be another reason for your function call not to work:
The function you want to call needs to be defined like the functions getInfo, _save and _nodeChange in the save plugin (see the developer build of tinymce to inspect this plugin in the plugins directory).

The save plugin shortened here:

(function() {
    tinymce.create('tinymce.plugins.Save', {
        init : function(ed, url) {
           ...
        },

        getInfo : function() {
                   ...
        },

        // Private methods

        _nodeChange : function(ed, cm, n) {
                   ...
        },

        // Private methods
                   ...
        _save : function() {

        }
    });

    // Register plugin
    tinymce.PluginManager.add('save', tinymce.plugins.Save);
})();

You may call the getInfo function of this plugin using the following javascript call:

tinymce.get('your_editor_id_here').plugins.save.getInfo();