When working on visionary systems or systems desig...
# share-your-work
b
When working on visionary systems or systems designed for experts, the academic publishing review process can sometimes be brutal. Reviewers sometimes expect user studies, but complex systems, particularly systems built for experts, cannot be compared head-to-head with prior work in 1-2hr user study sessions—fluency with a tool could require 6 months practice or more! How then do you show that the system you’ve built is actually an improvement, if you can’t perform traditional human testing on it? In 2007, Dan Olsen addressed this problem, identifying the common reviewer mistakes (“Evaluation Errors”), outlining a framework for justifying the contribution of a complex system (“STU Context”), and offering possible claims a paper could make in leu of human usability tests. To help me keep track of these ideas, I pulled them into a one-page summary. I hope others find this summary helpful as well.
(PDF)