My company is engaging with a multivariate testing vendor (I can’t disclose which one just yet), and I am being asked to integrate their system into our flagship B2C commerce site.
Usually, the word “integrate” is a heavy-duty term. However, here it means to add a <script> tag on several views, and then be out of the loop from that point forward. The mulitvariate experiments will be set up by our marketing department (and/or the vendor). Our development team will not be involved in that, and may not even be aware when it’s happening.
Essentially, I am being told to deliberately add a JavaScript-injection attack vector to our flagship application… and hope for the best. My concerns are not so much with malicious code as they are with inadvertent screw-ups. Our CSS and basic page layout is a mess already, and I anticipate catiching the heat when someone else’s multivariate experiment blows up the home page look-n-feel, etc. Much more importantly, there is a lot of ugly AJAX and server-side cruft that depends on predictable HTML elements, id attributes, etc… and so if an experiment changes HTML elements too much, core commerce functionality will simply break altogether in unpredictable ways.
My concerns are amplified by the lack of a real test environment. The vendor does filtering, so that only AJAX calls from our production URL are processed. Calls from our Dev or QA environments are ignored. However, that’s not the same thing as having a test environment. Rather, it means that production is now our test environment for each experiment.
Given all of the above, are there any practices available to mitigate some of these risks? Multivariate testing is such a new field, the first several pages of a Google search consists of shady consultants selling buzzwords to CIO’s. There’s not much written for the in-house technical audience ultimately responsible for keeping an eye on risk. Are there any ways to properly QA under the above restraints, or is there simply no alternative but to push back(*) in such a scenario?
(*) Note: Given the politics of the vendor relationship, we would need to build an extraordinarily airtight case for push-back, and it may be ignored anyway.
You need to remember that multi-variate testing is not testing in the sense of what you think (or, it shouldn’t be). You are not supposed to be testing the code or content. You are testing visitor reactions to that content.
Meaning… the content to be tested by your vendor should still pass through some quality assurance process before being deployed in their platform. This should suggest, then, that the best way to mitigate the risks you are concerned about is to get yourself injected into their testing/pre-release cycle to ensure the security and stability of their code before it goes out. You don’t need to (and don’t want to) belabor their efforts with full regression tests or anything like that. But you can take the initiative to set up a highly-specific set of tests for the dynamic content to pass before it goes to prod.
If the vendor is worth their weight in [insert whatever material here] then they should already have accounted for this in their processes, or at least be able to accommodate such a request. It’s not too much to ask that you get some birds-eye sign-off before any intrusion-risk code is injected.
Or… get a waiver signed relieving you of any security or performance responsibility during the vendor’s exercises. 🙂