Evaluation

We know that there are many different factors that make contributions valuable to an open source project. We want to reward well-written, well-functioning contributions that provide meaningful progress and that represent improvement beyond what would be expected from others attempting the same contribution.

The processes and panels below have been thoughtfully assembled to account for the multiple goals that this community has for project contributions. A system of Points will be used as a tool for tracking contribution effort and importance. Points awarded to contributors will be displayed throughout the challenge on the live Leaderboard. For more on how these points will eventually inform how prizes are awarded, see the Awards page.

Remember, the ultimate goal of this challenge is to help physicians save lives. Points and prizes are not the central purpose of the competition and serve only as a way to make this challenge more fun and attract the broadest array of talented contributors. As with any collaborative technical project, we expect there to be differences in opinion regarding design decisions and implementation. We are happy to receive constructive suggestions and questions directed to project staff; however, public debate in technical forums or excessive public litigation over points violates the spirit of the challenge and bogs down development, and does not have a place in this challenge. For more on this topic see the Contributor Code of Conduct.


Pre-designated points

This project is organized as a series of issues on GitHub. Many of the issues will be designated in advance by project maintainers, informed by the project design document. The points will generally follow the level of effort chart below.

As contributions are accepted, contributors will be awarded the point values for those issues on a provisional basis, and smaller point values (under 10 points) will be immediately assigned. At the end of each project sprint, project maintainers will make recommendations to the Expert Technical Panel about how points should be distributed for any contribution of 10 points or above, which the panel will vote to accept, modify, or send back for reconsideration.

Points Contribution type

100

  • Major feature

40

  • Medium feature
  • Major enhancement
  • Major refactoring
  • Major structure or tooling improvement
  • Major documentation contribution

20

  • Minor enhancement
  • Minor feature
  • Minor documentation contribution
  • Major bugfix

5

  • Minor refactoring
  • Minor structure or tooling improvement
  • Minor bugfix or documentation edit
  • Community participation/QA
  • Opening an issue that is converted to “official” with a label

1-3

  • Very minor but worthy contribution in any of the above categories
  • Definitions:

  • A feature is a patch providing new functionality that is coded from scratch to achieve one or many functional requirements outlined in the design doc (e.g. implementing the model training pipeline in new source files which were stubbed out before but not actually coded).
  • An enhancement is a patch that improves or extends an existing *feature* (e.g. making improvements to the existing DICOM viewer to that image features are more easily zoomable).
  • A bugfix is a patch that corrects an officially recognized and non-trivial problem tracked by an existing issue (e.g. fixing an out-of-memory crash when a new DICOM image is loaded before the old is unloaded).
  • Refactoring is "the process of restructuring existing computer code—changing the factoring—without changing its external behavior. Refactoring improves nonfunctional attributes of the software."[1] (e.g. taking duplicated code in two parts of the application and creating a clean interface which both parts now use).
  • A structure or tooling improvement is a meta improvement to the project which makes development easier or more productive (e.g. setting up a new Dockerfile which isolated a testing environment for developer reproducibility).
  • NOTE: Point estimates in the table give a rough order of magnitude for the points awarded for issues in the GitHub repository. All issues have a relative level of effort and work stream to help figure out about how many points fixing certain issues will earn you. The exact number of points will be decided by the technical panel upon review and acceptance of the PR.

Discretionary points

We know that many contributions will not necessarily respond to pre-designated issues, but are still valuable (in fact essential) to any successful open source project. Those issues that are opened by contributors or other meaningful contributions made to the project that were not assigned in the pre-designated pool (for example, writing tests, filing issues, helping in community forums, promoting the project on social media) but add meaningful value to the project will have the opportunity to earn points through this discretionary pool instead.

During each sprint, project maintainers will track these additional contributions, and may award smaller point values (under 10 points) on a rolling basis. At the end of the sprint project maintainers will make recommendation to the Expert Technical Panel to assign points at a point level fitting the contribution for any contribution of 10 points or above, which the technical panel will vote to accept, modify, or send back for reconsideration.

Collaborative contributions

As described in the rules, a contribution is assumed to come from the participant submitting it, unless others are specifically listed in the commit history or in the comment field that you fill in when opening the pull request. If multiple participants are listed, the default method for dividing credit is to split any points in equal shares.

If a different allocation is desired, all listed participants must agree to the different breakdown in writing to DrivenData immediately upon submitting the pull request, either by e-mail or in comments on the pull request.

Expert technical panel →

Project managers →