Week 11 [from Wed Mar 25 noon] - Project

PE Overview

Objectives:

• The primary objective of the PE is to increase the rigor of project grading. Assessing most aspects of the project involves an element subjectivity. As the project counts for a large percentage of the final grade, it is not prudent to rely on evaluations of tutors alone as there can be significant variations between how different tutors assess projects. That is why we collect more data points via the PE so as to minimize the chance of your project being affected by evaluator-bias.
• PE is also used to evaluate your manual testing skills, product evaluation skills, effort estimation skills etc.
• Note that significant project components are not graded solely based on peer ratings. Rather, PE data are mostly used to cross-validate tutors' grades and identify cases that need further investigation. When peer inputs are used for grading, they are usually combined with tutor evaluations with appropriate weight for each. In some cases ratings from team members are given a higher weight compared to ratings from other peers, if that is appropriate.
• PE is not a means of pitting you against each other. Developers and testers play for the same side; they need to push each other to improve the quality of their work -- not bring down each other.

Grading:

• Your performance in the practical exam will affect your final grade and your peers', as explained in Admin: Project Grading section.
• As such, we have put in measures to identify and penalize insincere/random evaluations.
• Also see:

Grading bugs found in the PE

• Of Developer Testing component, based on the bugs found in your code3A and System/Acceptance Testing component, based on the bugs found in others' code3B above, the one you do better will be given a 70% weight and the other a 30% weight so that your total score is driven by your strengths rather than weaknesses.
• Bugs rejected by the dev team, if the rejection is approved by the teaching team, will not affect marks of the tester or the developer.
• The penalty/credit for a bug varies based on,
  • The severity of the bug: severity.High > severity.Medium > severity.Low > severity.VeryLow
  • The type of the bug: type.FunctionalityBug > type.DocumentationBug > type.FeatureFlaw
• The penalty for a bug is divided equally among assignees.
• Developers are not penalized for duplicate bug reports they received but the testers earn credit for duplicate bug reports they submitted as long as the duplicates are not submitted by the same tester.
• i.e., the same bug reported by many testersObvious bugs earn less credit for the tester and slightly more penalty for the developer.
• If the team you tested has a low bug count i.e., total bugs found by all testers is low, we will fall back on other means (e.g., performance in PE dry run) to calculate your marks for system/acceptance testing.
• Your marks for developer testing depends on the bug density rather than total bug count. Here's an example:
  • n bugs found in your feature; it is a difficult feature consisting of lot of code → 4/5 marks
  • n bugs found in your feature; it is a small feature with a small amount of code → 1/5 marks
• You don't need to find all bugs in the product to get full marks. For example, finding half of the bugs of that product or 4 bugs, whichever the lower, could earn you full marks.
• Excessive incorrect downgrading/rejecting/marking as duplicatesduplicate-flagging, if deemed an attempt to game the system, will be penalized.

PE Preparation

• It's similar to,

PE Phase 1: Bug Reporting

PE Phase 1 is an 'exam', to be done under exam conditions:

• You are not allowed to communicate with others (except the invigilators).
• You are not allowed to share your work with others.
• Violators will be reported to the university for disciplinary action.

Online channels used for the PE:

• Gitter:
  • Prof will be online in Gitter.
  • Direct all your queries to prof as private messages to @damithc. Do not use the public chat channel . If you use the main channel, you might accidentally reveal which team you are testing.
• Zoom:
  • There will be a Zoom meeting (details will be announced later) to provide you with audio announcements.
  • Join it and keep the speaker on during the PE session so that you can hear our announcements.
  • You can use it to chat with the prof (but Gitter is preferred for chatting as it's harder to post screenshots in Zoom chats).
  • You will not be able to share your screen, share your cam, or talk to us.

• When, where: Week 13 lecture slot

PE Phase 1 - Part I Product Testing [60 minutes]

• Test the product and report bugs as described below. You may report both product bugs and documentation bugs during this period.

PE Phase 1 - Part II Evaluating Documents [30 minutes]

• This slot is for reporting documentation bugs only. You may report bugs related to the UG and the DG.
• For each bug reported, cite evidence and justify. For example, if you think the explanation of a feature is too brief, explain what information is missing and why the omission hinders the reader.

PE Phase 1 - Part III Overall Evaluation [15 minutes]

• To be submitted via TEAMMATES. You are recommended to complete this during the PE session itself, but you have until the end of the day to submit (or revise) your submissions.

PE Phase 2: Developer Response

This phase is for you to respond to the bug reports you received.

Duration: The review period will start around 1 day after the PE (exact time to be announced) and will last until the following Saturday midnight (i.e., within 3 days). However, you are recommended to finish this task ASAP, to minimize cutting into your exam preparation work.

Bug reviewing is recommended to be done as a team as some of the decisions need team consensus.

Instructions for Reviewing Bug Reports

• Don't freak out if there are lot of bug reports. Many can be duplicates and some can be false positives. In any case, we anticipate that all of these products will have some bugs and our penalty for bugs is not harsh. Furthermore, it depends on the severity of the bug. Some bug may not even be penalized.

• CATcher does not come with a UG, but the UI is fairly intuitive (there are tool tips too). Do post in the forum if you need any guidance with its usage.
• Also note that CATcher hasn't been battle-tested for this phase, in particular, w.r.t. multiple team members editing the same issue concurrently. It is ideal if the team members get together and work through the issues together. If you think others might be editing the same issues at the same time, use the Sync button at the top to force-sync your view with the latest data from GitHub.

Launch CATcher, and login to the profile CS2113/T PE. It will show all the bugs assigned to your team, divided into three sections:

1. Issues Pending Responses - Issues that your team has not processed yet.
2. Issues Responded - Your job is to get all issues to the second category.
3. Faulty Issues - e.g., Bugs marked as duplicates of each other, or causing circular duplicate relationships. Fix the problem given so that no issues remain in this category.

Respond to the bug reports shown.

You must use CATcher. You are strictly prohibited from editing PE bug reports using the GitHub Web interface as it can can render bug reports unprocessable by CATcher, sometimes in an irreversible ways, and can affect the entire class. Please contact the prof if you are unable to use CATcher for some reason.

• If a bug seems to be for a different product (i.e. wrongly assigned to your team), let us know ASAP.
• If the bug is reported multiple times,
  • Mark all copies EXCEPT one as duplicates of the one left out (let's call that one the original) using the A Duplicate of tick box..
  • If the duplicates have different severity levels, you should keep the one with the highest severity as the original. But you can downgrade the severity of the original or the duplicates.
  • For each group of duplicates, all duplicates should point to one original i.e., no multiple levels of duplicates, and no cyclical duplication relationships.
  • If the duplication status is eventually accepted, all duplicates will be assumed to have inherited the type.* and severity.* from the original.

• Apply one of these labels (if missing, we assign: response.Accepted)

Response Labels:

• response.Accepted: You accept it as a bug.
• response.NotInScope: It is a valid issue but not something the team should be penalized for e.g., it was not related to features delivered in v2.1.
• response.Rejected: What tester treated as a bug is in fact the expected behavior. You can reject bugs that you inherited from AB3.
• response.CannotReproduce: You are unable to reproduce the behavior reported in the bug after multiple tries.
• response.IssueUnclear: The issue description is not clear. Don't post comments asking the tester to give more info. The tester will not be able to see those comments because the bug reports are anonymous.

• Apply one of these labels (if missing, we assign: type.FunctionalityBug)

Type labels:

• type.FunctionalityBug: A functionality does not work as specified/expected.
• type.FeatureFlaw: Some functionality missing from a feature delivered in v2.1 in a way that the feature becomes less useful to the intended target user for normal usage. i.e., the feature is not 'complete'. In other words, an acceptance testing bug that falls within the scope of v2.1 features. These issues are counted against the 'depth and completeness' of the feature.
• type.DocumentationBug: A flaw in the documentation that can potentially affect the user e.g., a missing step, a wrong instruction, typos that affect users

• If you disagree with the original severity assigned to the bug, you may change it to the correct level.

Bug Severity labels:

• severity.VeryLow : A flaw that is purely cosmetic and does not affect usage e.g., a typo/spacing/layout/color/font issues in the docs or the UI that doesn't affect usage.
• severity.Low : A flaw that is unlikely to affect normal operations of the product. Appears only in very rare situations and causes a minor inconvenience only.
• severity.Medium : A flaw that causes occasional inconvenience to some users but they can continue to use the product.
• severity.High : A flaw that affects most users and causes major problems for users. i.e., makes the product almost unusable for most users.

• If you need teaching team's inputs when deciding on a bug e.g., if you are not sure if the UML notation is correct, post in the forum. Remember to quote the issue number shown in CATcher (it appears at the end of the issue title).

• Broken links in UG/DG: Severity can be low or medium depending on how many such cases and how much inconvenience they cause to the reader.
• UML notation variations caused by the diagramming tool: Can be rejected if not contradicting the standard notation (as given by the textbook) i.e., extra decorations that are not misleading.
  If the same problem is reported for multiple diagrams, can be flagged as duplicates.
  Omit optional notations is not a bug as long it doesn't hinder understanding.
• Minor typos and grammar errors: These are still considered as severity.VeryLow type.DocumentationBug bugs (even if it is in the actual UI) which carry a very tiny penalty.
• How to prove that something is 'not in scope': The following (at least one) can be used to prove that a feature was left out deliberately because it was not in scope:
  • The UG specifies it as not supported or coming in a future version.
  • The user cannot attempt to use the missing feature or when the user does so, the software fails gracefully, possibly with a suitable error message i.e., the software should not crash.
• If a missing feature is essential for the app to be reasonably useful, its omission can be considered a feature flaw even if it can be proven as not in scope as given in the previous point.

• Decide who should fix the bug. Use the Assignees field to assign the issue to that person(s). There is no need to actually fix the bug though. It's simply an indication/acceptance of responsibility. If there is no assignee, we will distribute the penalty for that bug (if any) among all team members.
  • If it is not easy to decide the assignee(s), we recommend (but not enforce) that the feature owner should be assigned bugs related to the feature, Reason: The feature owner should have defended the feature against bugs using automated tests and defensive coding techniques.
  • It is expected that PR reviewers will take part of the responsibilities if they were given a reasonable time to review the PR (even if they never reviewed it), provided the bug should have been reasonably apparent to a reviewer. For example, if someone put up a PR for review but merged it un-reviewed after two days because no one bothered to review it, it's only fair that the whole team (or whoever was assigned to review the PR) take a share of the blame.

• As far as possible, choose the correct type.*, severity.*, response.*, assignees, and duplicate status even for bugs you are not accepting or marking as duplicates. Reason: your non-acceptance or duplication status may be rejected in a later phase, in which case we need to grade it as an accepted/non-duplicate bug.

• Justify your response. For all of the following cases, you must add a comment justifying your stance. Testers will get to respond to all those cases and will be double-checked by the teaching team in later phases. Indiscriminate/unreasonable dev/tester responses, if deemed as a case of trying to game the system, will be penalized.

  • downgrading severity
  • non-acceptance of a bug
  • changing the bug type
  • non-obvious duplicate

PE Phase 3: Tester Response

• In this phase you will get to state whether you agree or disagree with the dev team's response to the bugs you reported. If a bug reported has been subjected to any of the below by the receiving dev team, you can record your objections and the reason for the objection.
  • not accepted
  • severity downgraded
  • bug type changed
• As before, consider carefully before you object to a dev team's response. If many of your objections were overruled by the teaching team later, you will lose marks for not being able to evaluate a bug report properly.
• You can also refer to the below guidelines:

• This phase is optional. If you do not respond to a dev response, we'll assume that you agree with it.
• Deadline: Reading week Tuesday 2359 (i.e., within 3 days)
• Procedure:

PE Phase 4: Tutor Moderation

• In this phase tutors will look through all dev responses you objected to in the previous phase and decide on a final outcome.
• In the unlikely we need your inputs, the tutor will contact you.

At the end of the project each student is required to submit a Project Portfolio Page.

PPP Objectives

• For you to use (e.g. in your resume) as a well-documented data point of your SE experience
• For evaluators to use as a data point to evaluate your project contributions

PPP Sections to include

• Overview: A short overview of your product to provide some context to the reader. The opening 1-2 sentences may be reused by all team members. If your product overview extends beyond 1-2 sentences, the remainder should be written by yourself.
• Summary of Contributions --Suggested items to include:
  • Code contributed: Give a link to your code on tP Code Dashboard. The link is available in the Project List Page -- linked to the icon under your photo.
  • Enhancements implemented: A summary of the enhancements you implemented.
  • Contributions to documentation: Which sections did you contribute to the UG?
  • Contributions to the DG: Which sections did you contribute to the DG? Which UML diagrams did you add/updated?
  • Contributions to team-based tasks :
  • Review/mentoring contributions: Links to PRs reviewed, instances of helping team members in other ways
  • Contributions beyond the project team:
    • Evidence of helping others e.g. responses you posted in our forum, bugs you reported in other team's products,
    • Evidence of technical leadership e.g. sharing useful information in the forum

Team-tasks are the tasks that someone in the team has to do. Marks allocated to team-tasks will be divided among team members based on how much each member contributed to those tasks.

Here is a non-exhaustive list of team-tasks:

1. Necessary general code enhancements
2. Setting up tools e.g., GitHub, Gradle
3. Maintaining the issue tracker
4. Release management
5. Updating user/developer docs that are not specific to a feature e.g. documenting the target user profile
6. Incorporating more useful tools/libraries/frameworks into the product or the project workflow (e.g. automate more aspects of the project workflow using a GitHub plugin)

• [Optional] Contributions to the User Guide (Extracts): Reproduce the parts in the User Guide that you wrote. This can include features you implemented as well as features you propose to implement.
  The purpose of allowing you to include proposed features is to provide you more flexibility to show your documentation skills. e.g. you can bring in a proposed feature just to give you an opportunity to use a UML diagram type not used by the actual features.

• [Optional] Contributions to the Developer Guide (Extracts): Reproduce the parts in the Developer Guide that you wrote. Ensure there is enough content to evaluate your technical documentation skills and UML modelling skills. You can include descriptions of your design/implementations, possible alternatives, pros and cons of alternatives, etc.

• [Optional] If you plan to use the PPP in your Resume, you can also include your SE work outside of the module (will not be graded).

PPP Format

• File name (i.e., in the repo): docs/team/githbub_username_in_lower_case.md e.g., docs/team/goodcoder123.md
• Follow the example in the AddressBook-Level3

PPP Page Limit

• The page limits given above are after converting to PDF format. The actual amount of content you require is actually less than what these numbers suggest because the HTML → PDF conversion adds a lot of spacing around content.