CS6310: Design Studies

Designers often build and compare physical models in order to better understand a problem. Examples are building architects and automobile designers. For the purposes of this course, a design study is a systematic comparison of multiple versions of a program with the intent of better understanding it.

You can think of a design study as an empirical scientific experiment. That is, there are research questions, subjects of study, experimental conditions, methods, tools, metrics, independent and dependent variables, data collection, statistical analyses, and conclusions. An overall goal is repeatability. That is, someone else should be able to take your study report, use it to recreate the study and reach the same conclusions. It is one of the goals of this class that you learn the skills to produce and present an industrial-quality design study.

Here are the topics you should consider in planning a design study. They should appear as separate labeled sections in the resulting report.

  • Context: What is your study examining?For CS6310, this means determining the role of variousnon-functional constraints on the quality of a simulationprogram. Give enough background so that a reader unfamiliarwith this class and its projects can understand what youare talking about. Also, in this section, define anyspecialized vocabulary that you will be using in the report.
  • Research questions: You shouldformulate specific research questions that you will study.For example, What is the effect of increased precisionon execution time? There should be questions discussingeach tradeoff among the problem constraints
  • Subjects: A subject issomething that you are studying, for example, a program. Adesign study compares multiple subjects. Each subject shouldbe briefly described, differentiating it from the othersubjects
  • Experimental conditions: What is theenvironment in which the study will take place. This includeselements such as machines, operating systems, programminglanguages, virtual machines, network (where relevant), buildand execution parameters, input files, and confoundingfactors (other users or processes). Remember that the goalis repeatability
  • Independent variables: Explicitlydescribe factors that you can vary, such as precision. Givea distinct name to each variable
  • Dependent variables: Explicit statementof what factors you expect to be affected by variations inthe independent variables, such as performance.Give a distinct name to each variable
  • Metrics: For each variable (independentand dependent), state how will you measure it. This shouldinclude the unit of measurement, such as seconds, and aprecise definition, such as a line of code is a maximalsequence of non-line-break characters that includes somenon-whitespace content occurring in a Java source file".For some factors, such as time, this is straightforward.For others, such as maintainability, you must thinkcarefully about the factor and invent an appropriate wayto measure it
  • Method: A description of how youwill conduct the experiment. This includes number of trials,measurement devices and tools, randomization technique(where appropriate), significant digits, etc. This shouldalso include an explicit statement of which subjects willbe run with the arguments used. For example, if you werestudying the relation of performance to grid size, you wouldwant to specify what different grid sizes you will be using
  • Analysis techniques: How will youanalyze the data? What statistics will you use? For example,means vs. medians? Are you performing regression analysisor computing correlations? What will you do about outliers?How will you judge significance? How will you report errorbounds? Are you doing any interpolation or curve fitting?
  • Presentation: How do you plan topresent the results? Graphs? Tables?
  • Results: The data collected(empirical results) and the results of analysis(analytical results)
  • Discussion: Reflection on theexperimentation itself, including explanations for anomalousphenomena and suggestions for improving the study process.In particular, any anomalous results should be thoroughlyexplored and explained
  • Conclusions: Specific answers to theresearch questions

A design study is presented in a report. The report is a carefully formatted presentation of the items listed above. It may include charts, tables, graphs and screen shots as well as descriptive text. It is not a narrative, but a dispassionate description of a systematic exploration. The report should be professional in quality. This means that you should treat it like you would if you were preparing to show it to customers or submit it for publications. Its spelling and grammar should be checked, and it should be carefully proofread by a team member other than the author.

Here is an MS Word file providing a starting point for your report.

ncG1vNJzZmirY2Ourq3ZqKWar6NjsLC5jpymp6yVo8FvwcOamqKsqWKxosDAZ5qopV%2BYvLa%2B0p6qaJ%2BkYrC0gpJqZ2iooqS3pq%2FTrGadnaOetK%2Bf066bop2jY7W1ucs%3D