User Test with Task Scenarios That Have Unachievable Goals?
I intend to do quantitative user tests of an existing tool and compare its usability performance to a new design that I'd propose.
Let's say after doing user research I come up with task scenarios (based on real needs) that include goals which can't be achieved with the current tool.
How does one proceed in that case?
It'd surely feel wrong to test a scenario with an unachievable goal. It would also feel wrong to test different scenarios for the current and new design (while also eliminating comparability).
Would a valid approach be defaulting to a 0% success rate for such task scenarios? Or would it be better to leave out such scenarios completely and only test tasks that have achievable goals in both, the old and new design?