Calibrated Peer Review Tool
This project was created within a software design class at SUNY Oswego during the spring 2023 semester. This was an existing project and was handed to us to continue development and improve the previous design. The project was done in collaboration with IBM.
The CPR Tool is the product of a software design project that functions to facilitate the peer review process in an academic setting. A calibrated peer review refers to the process in which an instructor creates an assignment, distributes the assignment to be completed by students, reviews their solutions, disperses the solution to other students for peer review, and then quality checks each review before assigning a final grade.
The software design team utilized an agile development methodology to create an anonymous peer review platform, which reduced biases and increased the reliability of reviews.
Our Goal:
Distributing peer reviews anonymously, moderating the peer reviews to ensure that they are fair, and redistributing the peer reviews back to the original team is an extremely lengthy and tedious process for instructors that can feel overwhelming.
With the development of the CPR Tool, this process becomes automated, greatly simplifying the efforts required to complete peer reviews by both students and instructors.
What We’ve Achieved
Complete visual overhaul to meet the standards of the stakeholders.
Developed a live and functional product.
Increased average student satisfaction rating of the peer review process from 2.4 when performing it manually, to 3.7 when using the CPR Tool. (Likert Scale 1-5)
Increased average instructor satisfaction rating of the peer review process from 1.9 when performing it manually, to 4.0 when using the CPR Tool. (Likert Scale 1-5)
Reduced the average time it takes a student to complete the peer review process from 8 minutes and 32 seconds when completing it manually, to 3 minutes and 50 seconds when using the CPR Tool
Reduced the average time it takes an instructor to complete the peer review process from 18 minutes and 12 seconds when completing it manually to 10 minutes and 7 seconds when using the CPR Tool
Created a detailed list of recommendations and findings from user testing into a handoff document for the potential that this project could be continued by another team in the future.
Figma Link: CPR Personal – Figma
CPR Tool Link*: moxie.cs.oswego.edu
Handoff Document: Handoff Doc - Google Docs
*SUNY Oswego is constantly updating the moxie servers so there may be issues accessing the tool at times.
CPR Tool Empathy Map
An empathy map was created to gain an understanding of what users might say, think, do, or feel while interacting with the CPR Tool. This provides an organized manner to understand the complexity of user needs and goals, along with their pain points and displeasures when using the tool.
Hierarchical Task Analysis
A hierarchical task analysis was created for both the student role and the instructor role in order to map out the tasks and smaller subtasks required to complete the desired goal.
Style Guide
With a GUI team comprised of seven members, it was imperative to create a style guide so that the design between each page is uniform. The style guide includes every button needed within the CPR Tool, each type of input field, the typography, the product logo, the colors, shadows, pop-ups, tool tips, tables, and charts.
Old Instructor Home Page
The old instructor homepage design provided course instructors with a basic navigation route to the courses they had available for that current semester but lacked an easy way for users to navigate between courses, students within courses, and the various assignments found within each course.
Instructor Home Page
The instructor home page highlights the layout and navigation of the tool under the instructor role. The instructor has direct access to their available classes within the semester, with an additional navigation bar on the left allowing access to archived courses, as well as the ability to edit any current course information.
Old Student Home Page
The old student home page immediately presents users with one of their courses, showcasing anonymous teams they may join for peer reviews. Students are unable to navigate to specific sections within a course directly from this page and must proceed one page at a time.
Student Home Page
The student home page highlights the layout and navigation of the tool under the student role. Students are presented with their courses from the home page and are able to navigate between courses and sections through the use of the navigation bar on the left. Users can also directly navigate to a specific assignment from a course using the navigation bar.
Admin Home Page
The admin role was created by the request of the stakeholders for the purpose of allowing administrators at institutions to be able to moderate all user roles, and courses available at the institution, as well as edit profanity settings so that the system may detect any forbidden words that may be contained in a document when it is uploaded for submission.
Old Peer Reviews Page
The old design included both assignments and peer reviews under the same page, with the assignments being color-coded to represent whether it was a new assignment or a peer review to be completed. This made it more difficult for students to quickly sort between their own assignments and peer reviews that they have completed. Although this may not be an immediate issue, further into the semester when there are dozens of assignments and peer reviews, users may get overwhelmed without an option to filter content.
Peer Reviews Page
This page highlights the central functionality of the tool, which is peer reviews. Students are able to navigate to this section through the dropdown menu in the navigation bar and are able to see any peer reviews they have performed for another team as well as those that still need to be completed, through the “PR Given” tab. Teams are able to view the peer reviews they have received from other teams using the “PR Received” tab.