Supplementary Material for "Identifying Explanation Needs: Towards a Catalog of User-based Indicators"

Identifying Explanation Needs: Towards a Catalog of User-based IndicatorsCoding Guidelines: behavior-based and event-based indicatorsSummary of ArtifactThe uploaded dataset contains the supplementary material for the RE '25 paper "Identifying Explanation Needs: Towards a Catalog of User-based Indicators". The material can be used to reproduce our survey study.Authors informationH. Deters, L. Reinhardt, J. Droste, M.Obaidi and K. Schneider (2025). Supplementary Material for Research Paper "Identifying Explanation Needs: Towards a Catalog of User-based Indicators".IEEE 33th Requirements Engineering Conference (RE'25), Hannover, Germany. https://doi.org/10.5281/zenodo.15798316Description of ArtifactThe pdf-file titled "Survey.pdf" presents the survey questions used for the survey of the paper.The pdf-file titled "Coding-Guidelines_Phyiscal-Reactions_Emotional-State.pdf" presents the coding guidelines for the physical reaction and emotional state indicators.The pdf-file titled "Coding-Guidelines_behavior-based_event-based.pdf" presents the coding guidelines for the behavior-based and event-based indicators.The spreadsheet "CodesForQuestionsQ2Q3Q4.xlsx" presents all participants responses for the second, third and fourth question of the survey. The spreadsheet "CodesForQuestionsQ5.xlsx" presents all participants responses for the fifth question of the survey.System RequirementsNot applicable as our supplementary material is only a dataset.Installation InstructionsNot applicable as our supplementary material is only a dataset.Usage InstructionsThe dataset presents the study material required to replicate our survey study. The "Survey.pdf" contains all the questions asked in the online survey and the supposed type of answer that participants had to give. Answer types were either free-text answer fields or single-choice. The survey can be reproduced with an online-survey tool or on paper. The "Coding-Guidelines_Physical-Reactions_Emotional-State.pdf" contains all the coding guidelines for physical reaction and emotional state indicators. By providing definitions, inclusion and exclusion criteria for each indicator the guidelines assist researchers with categorizing qualitative data from open ended survey responses about physical reactions and emotional states. The survey responses contain participants descriptions of physical reactions and emotional states resulting from an explanation need in a software system. Physical reaction indicators include descriptions of specific physical behaviors that participants might have reported in their responses and how these behaviors need to be assigned to a physical reaction indicator. Emotional state indicators describe different negative or positive emotions participants might have described in their answers. Each emotional state indicator lists multiple synonyms or similar emotions that can be assigned to one specific indicator. Each response can have 0 or more indicators assigned to their reported physical reactions and emotional states. Researchers can use the coding guidelines with our survey responses or new responses from a reproduced survey study. Researcher also might expand the coding guidelines, if they find new inclusion or exclusion criteria for the existing indicators or if they find new indicators.The "Coding-Guidelines_behavior-based_event-based.pdf" contains all the coding guidelines for behavior- and event-based indicators. By providing definitions, inclusion and exclusion criteria for each indicator the guidelines assist researchers with categorizing qualitative data from open ended survey responses. The survey responses contain participants descriptions of their own user behavior or event-based behavior of a software system resulting from an explanation need in this software system.  Some indicators also include keywords that are associated with the indicator. The categorization of the indiciators is splitted in two coding levels. The first-level coding guidelines are applied to identify the indicators. With the second-level coding the already identified indicators can be further specified. Each response can have 0 or more indicators assigned from the first- and second-level of the coding guidelines. Researchers can use the coding guidelines with our survey responses or new responses from a reproduced survey study. Researcher also might expand the coding guidelines, if they find new inclusion or exclusion criteria for the existing indicators or if they find new indicators.The "CodesForQuestionsQ2Q3Q4.xlsx" contains all participants responses describring their need in a software system, their physical reaction and emotional state from the second to fourth question of the survey and the determined indicator codes. The columns of the spreadsheet are splitted in answer ID, software, need category, answer need translated from german, need original response, behavior-based/event-based indicator code, answer behavior response translated from german, behavior original response, physical reaction indicator code, answer physical behavior translated from german, physical behavior original response, emotional state indicator code, answer emotional state translated from german, original emotional state response. Each line contains all the responses and codes for one named software system, where each coding cell contains 0 or more indicators determined for a response. Every participant response has its own cell. Researchers can use the whole spreadsheet without our indicator codes to reproduce the coding results or use the spreadsheet as a template for their own survey responses and a completely new coding process. Similar to the other spreadsheet "CodesForQuestionsQ5.xlsx" contains all participants responses describing their own ideas of needs in different software systems and the determined indicator codes. The columns of the spreadsheet are splitted in answer ID, original participants response, participants response translated from german and the behavior-based/event-based indicator code. Each line contains all the responses and codes for one named software system, where each coding cell contains 0 or more indicators determined for a response. Every participant response has its own cell. Researchers can use the whole spreadsheet without our indicator codes to reproduce the coding results or use the spreadsheet as a template for their own survey responses and a completely new coding process.Steps to ReproduceNot applicable as our supplementary material is only a dataset.