This page will present general resources for gathering metrics around the activities of facilitation described in the Leading Practices of Facilitation. For specific impacts and measurement of the ACI-REF project, see the “Impact” tab or the summary of project accomplishments on the Project Evaluation and Metrics page.
Metrics over the course of ACI-REF
At the start of the ACI-REF project, facilitation evaluation was limited to addressing the goals of the ACI-REF project. However, over the course of the project, it became clear that it would be valuable (and in line with the goals of the ACI-REF project) to develop more general standards for evaluating facilitation at a campus or compute center. There were two major products of the ACI-REF project:
- Surveys: The metrics team first looked at how to get feedback from the users of CI resources through the format of surveys or in-person interviews.
- Monthly Data Collection: Later, based on the major activities from the Leading Practices of Facilitation, the metrics team identified major data categories and sub-categories for tracking facilitator activities and impact. These were developed into a series of spreadsheets for recording this data, with an end goal of making it easy to aggregate and summarize data over time.
Both the surveys and data collection tables are available in the Appendices of the Leading Practices of Facilitation, and the tables are also presented below.
Facilitation Data Collection
The following data categories are meant as a starting point for collecting data around the activities of facilitation. Depending on your institution or center, it may not be relevant or possible to report on all of the categories listed in the templates. These can be removed or simply left blank when reporting data. Groups are also welcome to add site-specific categories/sub-categories relevant to data that they are trying to track. The presented data categories primarily focus on facilitation of research computing, but could be adapted to data, programming, and other services.
You can access spreadsheets that have been pre-populated with these data categories in two ways:
- as Google Sheets in this Google Drive folder
- as a single Excel file, downloadable from the ACI-REF website
To use this reporting structure, create a copy of each template spreadsheet. Fill in with the appropriate information on a regular basis.
Facilitation Data Categories
Quantitative Table
Categories for tracking quantitative data related to facilitation activities.
| Category | Sub-Category | Notes |
|---|---|---|
| meetings | general/total | number of researchers with in-person facilitator engagement |
| meetings | initial | number of researchers with initial facilitator engagement (e.g. new accounts, new group, first time having an individual meeting) |
| meetings | follow-up | number of researchers with ongoing facilitator engagement (e.g. after first meeting) |
| meetings | assistance/office-hours | number of researchers who attended office hours or received other in-person assistance |
| email/tickets | whose email/tickets you report is a judgment call; it seems that most sites are reporting numbers for a team or center, not ACI-REFs specifically | |
| email/tickets | updated | number of updated tickets/email threads during the month |
| email/tickets | closed | number of resolved or closed tickets/email threads during the month |
| training | sessions | number of training sessions |
| training | attendees | total number of attendees across training sessions |
| training | hours | number of hours of training offered |
| training | attendee-hours | number of hours spent by attendees in training (ex: 6 people at a 2 hour workshop is 12 attendee-hours) |
| outreach | sessions | number of outreach (presentation) activities |
| outreach | attendees | total number of attendees at outreach activities |
| accounts | newly-created | number of new user accounts (for new users with hours, see users/new) |
| accounts | new-projects | number of new projects with accounts |
| users | new | number of new users with compute hours or use of resources |
| users | all | number of users with hours |
| users | domain | number of users with hours by domain |
| departments | all | number of departments represented by users with hours |
| departments | domain | number of departments represented by users with hours, by domain |
| hours | all | number of hours supported |
| hours | domain | number of hours supported by domain |
Stories Table
Categories for recording noteworthy researcher/user stories, interventions, publications, or feedback.
| Category | Notes |
|---|---|
| story | Notable user story or interaction |
| quote | Quote or “love note” from user |
| press | University or other press release |
| publication | Publication information from researcher who used your resources / benefited from facilitators |
| presentation/poster | Poster or presentation by a researcher who used your resources / benefited from facilitators |
Activities Table
Categories for recording general facilitation activities. Note that “external” is outside your institution and “internal” is on-campus or within institution.
| Category | Sub-Category | Notes |
|---|---|---|
| external | conference-participation | name of conferences attended |
| external | presentation/panel | presentations at conferences or meetings |
| external | service/collaboration | participation in non-conference community events outside your campus (e.g. campus champion calls or emails) |
| internal | training | training for local users |
| internal | outreach | outreach on campus (e.g. include visiting a class, presentations, resource fairs, developing materials) |
| internal | documentation | significant user guide or documentation efforts |
| internal | team-projects | involvement with projects of your group/center (testing, new service) |
| internal | collaborations | collaborations or meetings with researchers and/or research-supporting units |
| pro-devel | learning | professional development opportunities |