Our Ecommerce website utilizes search engine crawlable faceted navigation links. According to Google Webmaster Tools, googlebot has crawled but chosen not to select, hundreds of thousands of urls and we have only ~1500 products. Googlebot seems to be getting lost in the huge crawl space afforded by our faceted navigation. How can I calculate the number of unique urls that are possible with our faceted navigation?
For example, on one category page, we have 6 different attributes with a total of 51 possible values.
attributeA has 2 selectable values
attributeB has 7 selectable values
attributeC has 7 selectable values
attributeD has 4 selectable values
attributeE has 4 selectable values
attributeF has 27 selectable values
Each value selection adds a parameter and value to the url. How many possible urls are there, given that none of the attributes are multiple select (i.e. if a value from attributeB was selected, another attributeB value cannot be added to it) and the url parameters are always put in alphabetical order?
I would like a formulaic answer so that I can calculate the crawl space for every one of our category pages.
There are 42,636 possible combinations if at least one option from each attribute must be selected, or there are 134,400 if each can be left blank.
This is simply obtained by multiplying together the amount of choices available for each attribute: the number of selectable values in the first case, and the number of selectable values + 1 (for null) in the latter.
The latter seems comparable to number of urls googlebot has visited.