diff --git a/METRICS.md b/METRICS.md new file mode 100644 index 0000000..f4f77bb --- /dev/null +++ b/METRICS.md @@ -0,0 +1,197 @@ +## Project Health Metrics + +This is the list of the metrics generated by this tool: + +### Pony factor + +The metric is defined as the number of individuals, who produce up to the +first 50% of the total number of code contributions (in descending order) +within a given time period. + +A low Pony Factor implies a high dependency on these individuals, making the +project vulnerable if they were to leave. + +*Also known as: Lottery Factor, Bus Factor, Contributor Absence Factor.* + +CHAOSS definition of the [Contributor Abscence Factor](https://www.chaoss.community/kb/metric-contributor-absence-factor/). + +### Elephant factor + +The metric is defined as the number of unique organizations producing up to the +first 50% of the total number of code contributions (in descending order) +within a given time period. + +It was first defined by Bitergia, and it applies the concept of the +Pony Factor metric and takes it to contributing Organizations. + +Contributions are focused on Git commits, and the organization +is determined by the email address of the commit author. + +CHAOSS definition of the [Elephant Factor](https://www.chaoss.community/kb/metric-elephant-factor/). + +### Number of contributing organizations + +This metric quantifies the total number of distinct organizations whose members +have made contributions to an open source project over a specified period. + +Contributions are focused on Git commits, and the organization +is determined by the email address of the commit author. + +CHAOSS definition of the [Organizational Diversity](https://www.chaoss.community/kb/metric-organizational-diversity/). + +### Number of organizations contributing recently + +This metric quantifies the number of unique organizations whose members have +actively made contributions to an open source project within the last 90 days. + +Contributions are focused on Git commits, and the organization +is determined by the email address of the commit author. + +CHAOSS definition of the [Organizational Diversity](https://www.chaoss.community/kb/metric-organizational-diversity/) +filtered by a specific timeframe. + +### Number of recent contributors + +This metric quantifies the total count of unique individuals who contributed +within the last 90 days. + +CHAOSS definition of [Contributors](https://www.chaoss.community/kb/metric-contributors/) filtered by a +specific timeframe. + + +### Number of recent commits + +This metric counts the total number of commits made to the project within the +last 90 days. + +CHAOSS definition of [Code Changes Commits](https://www.chaoss.community/kb/metric-code-changes-commits/) filtered by +the last 90 days. + +### Contributor Growth Rate + +This metric measures the growth rate of active contributors, defined as the +number of people sending one or more code contributions (in this case, +Git commits) in a given period. + +To calculate it, the period is split into two halves, and the number of active +contributors in each half is compared. The growth rate is the difference +between the second half and the first half, divided by the number of active +contributors in the first half. + +```math +GrowthRate (t_1, t_2) = \frac{C_a(t_2) - C_a(t_1)}{C_a(t_1)} +``` + +### Contributor Growth + +This metric measures the growth of active contributors, defined as the +number of people sending one or more code contributions (in this case, +Git commits) in a given period. + +To calculate it, the period is split into two halves, and the number of active +contributors in each half is compared. Growth is the difference between the +second half and the first half. + +```math +Growth (t_1, t_2) = C_a(t_2) - C_a(t_1) +``` + +### Number of active branches + +This metric refers to the count of branches within a project's version control +repository (for this case, Git) that have seen recent development activity, +usually indicated by new commits. + +CHAOSS definition of [Branch Lifecycle](https://www.chaoss.community/kb/metric-branch-lifecycle/). + +### Days since last commit + +This metric shows the number of days since the last commit was submitted to the +repository or the project. + +CHAOSS definition of [Code Changes Commits](https://www.chaoss.community/kb/metric-code-changes-commits/) +taking into account the last commit activity. + +### Presence of an adopters file in a standard location + +This metric verifies the existence of a file containing the full text of +the project's adopters. + +Standard practice dictates that this file is named `ADOPTERS`, `ADOPTERS.md`, +or `ADOPTERS.txt`, and is located in the root directory of the project's +source code repository. + +### Presence of a license file in a standard location + +This metric verifies the existence of a file containing the full text of +the project's chosen open source license. + +Standard practice dictates that this file is named `LICENSE`, `LICENSE.md`, +`LICENSE.txt`, or `COPYING` (a convention historically used by GNU projects) +and is located in the root directory of the project's source code repository. + +CHAOSS definition of [License Declared](https://www.chaoss.community/kb/metric-licenses-declared/). + +### Rate of contributors contributing infrequently vs. regularly + +This metric involves categorizing contributors based on the frequency, +consistency, and intensity of their contributions over a defined period. It aims +to distinguish between individuals who contribute episodically +(infrequent contributors) and those who engage with the project consistently +and often substantially (regular or core contributors). + +The CHAOSS community, for instance, defines +"[Occasional Contributors](https://www.chaoss.community/kb/metric-occasional-contributors/)" as +"people who make contributions to a project on an irregular basis". + +### Number of contributors who have contributed in previous periods + +This metric, often referred to as "returning contributors" or as an indicator +of "contributor retention," counts the number of unique individuals who were +active contributors in the last 90 days and had also made contributions in one +or more defined previous periods. + +### Rate of commits over specified periods + +This metrics calculates the rate of commits between the last 90 days and +the last year. The purpose of this metrics is having an estimation of how +distributed contributions are, and the "momentum" of the project. + +### Number of commits per repository + +Activity for each of the Git repositories analyzed. + +CHAOSS definition of [Code Changes Commits](https://www.chaoss.community/kb/metric-code-changes-commits/) filtered +by repository. + +### Number of developers per repository + +Unique participants producing commits in a given repository. + +CHAOSS definition of [Contributors](https://www.chaoss.community/kb/metric-contributors/) filtered +by repository. + +### File type metrics (code, documentation, or others) + +Type of activity done by developers, mainly split into code, documentation, and others. + +### Commit size metrics (added and removed lines) + +This provides an overview of the usual size of the code review processes and good practices +when submitting code. + +CHAOSS uses the metric [Change Request Commits](https://www.chaoss.community/kb/metric-change-request-commits/) +as a way to use this as a filter. + +### Message size metrics (total, mean, and median) + +This metrics provides information about how extensive a developer is providing information about the +change in the commit. + +### Frequency metrics for commits + +This allows to understand the consistency of developers when producing code. + +### Developer categories (core, regular and casual) + +This metric structures the developers by their activity and consistency. The core developers produce diff --git a/README.md b/README.md index 666b956..d9922d8 100644 --- a/README.md +++ b/README.md @@ -1,343 +1,315 @@ -# grimoirelab-metrics +# HealthyCode + +Open source project health metrics and scores, built on an open model, open +methodology, with open data. + +HealthyCode attempts to tell you how healthy an open source projects is, so you +can make more informed, data-driven decisions about which packages to adopt, +upgrade, or replace. HealthyCode collect a project's development history events +(like commits, contributors, and other activities) and turns these into health +"metrics". The metrics are then used to compute a health score using a scoring +procedure where each metrics is weighted, and weights are tuned from a training +data sets. The score is backed by these metrics, enabling to see why a package +scored and how, and to set own policy thresholds. + +HealthyCode uses [GrimoireLab](https://github.com/chaoss/grimoirelab) to collect +data, [ScanCode.io](https://github.com/aboutcode-org/scancode.io) to run +pipelines, and [PurlDB](https://github.com/aboutcode-org/purldb) to store the +data. + +## Why project health matters + +Most software is assembled from open source packages. Security scanners are good +at flagging a package with a known vulnerability, but they say nothing about a +package whose last maintainer has quietly moved on... + +> HealthyCode starts with npm. Other ecosystems will follow. +> + +> npm is the largest package registry in the world, and its packages depend on +> each other heavily. A small library can sit underneath thousands of projects. +> + +> With npms, this risk can spread quickly as the JavaScript developers prefer +> publishing many smaller packages, and many small unmaintained library can become +> a single point of failure for everything built on it, and are also easier to +> take over. When a maintainer's account or email domain expires, an attacker can +> claim it and publish a malicious release. + +HealthyCode helps you spot these packages before you depend on them, and helps +you keep watching the ones you already use as dependencies. + +## What's provided + +For this first iteration, HealthyCode focus is on npm packages. We will extend +support to other ecosystems, and we designed the scording model to be specific +to one open source packaging ecosystem. + +Given a PURL (Package-URL) for a package, HealthyCode's API returns: + +- Health metrics from the project's Git history, such as recent commits, +contributor activity and growth, days since last commit, pony factor (how many +people do half the work), and the elephant factor (how many organizations do +half the work). The full list is in [METRICS.md](METRICS.md). + +- A health score is calculted from the metrics, between 0 and 1 that is the +model's estimate of how likely the project is to be healthy. 0 means unhealthy +or risky, and 1 means healthier and less risky. + +- The settings used for the run: time window, thresholds, and the versions of +HealthyCode and the scoring model. + +The API results are JSON, so you can pull these into a dashboard, a report, or +in your software supply processing and management pipelines, including policies +to allow or block or alert in your own tooling about the health of a software +page. HealthyCode gives you the evidence, but the policy decision stays with +you. + +Here is a sample result for the `pkg:npm/semver` package: +```json +{ + "purl": "pkg:npm/semver", + "source_purl": "pkg:github/npm/node-semver", + "vcs_url": "https://github.com/npm/node-semver.git", + "scoring_model": "npm-health-0.2", + "score": 1.0, + "commit_range": { + "last_commit": "3484e1785e18a7ae4f06365f35c7b6eaec05088a", + "first_commit": "a25789b09b1192fa8414c35f2cd679ae2e1d5192", + "last_commit_date": "2026-09-10T19:51:52+00:00", + "first_commit_date": "2025-10-07T10:26:26-07:00" + }, + "run_start_date": "2026-10-07T21:22:06.088852Z", + "run_end_date": "2026-10-07T21:22:06.799342Z", + "metrics": { + "pony_factor": 2, + "total_commits": 45, + "recent_commits": 45, + "active_branches": 10, + "elephant_factor": 1, + "file_types_code": 37, + "commits_per_week": 0.8630136986301369, + "commits_per_year": 45.0, + "file_types_other": 61, + "commits_per_month": 3.6986301369863015, + "file_types_binary": 0, + "message_size_mean": 2538.8444444444444, + "contributor_growth": 5, + "found_file_license": 1, + "message_size_total": 114248, + "total_contributors": 16, + "found_file_adopters": 0, + "message_size_median": 593, + "recent_contributors": 16, + "total_organizations": 5, + "recent_organizations": 5, + "days_since_last_commit": 26, + "returning_contributors": 0, + "commit_size_added_lines": 930, + "contributor_growth_rate": 0.7142857142857143, + "coefficient_of_variation": 1.0786365421788087, + "commit_size_removed_lines": 137, + "commits_over_periods_rate": 1.0, + "developer_categories_core": 7, + "developer_categories_casual": 0, + "developer_categories_regular": 9, + "casual_regular_contributors_rate": 0.0 + }, + "date_collected": "2026-10-07T21:22:07.178202Z" +} -Client to generate GrimoireLab metrics for Project Health using the -software analytics platform [GrimoireLab](https://github.com/chaoss/grimoirelab). - -![grimoirelab_metrics_schema.jpg](docs/images/grimoirelab_metrics_schema.jpg) - -## Installation - -### Prerequisites - -Instances of GrimoireLab 2.x and OpenSearch must be running before launching this tool. -Please check the [GrimoireLab](https://github.com/chaoss/grimoirelab/blob/2.x/README.md) -documentation in order to deploy the platform. - -To get this tool running, we also recommend using [poetry](https://python-poetry.org/). -This package manager will install the tool and all its dependencies. -You can install `poetry` by following [this guide](https://python-poetry.org/docs/#installing-with-pipx). - -Once you have `poetry` running, move to the next section. - -### Steps - -1. Clone the repository: - - ```bash - git clone git@github.com:Bitergia/grimoirelab-metrics.git - cd grimoirelab_metrics - ``` - -2. Install dependencies and tool: - - ```bash - poetry update - poetry install - ``` - - This will install the tool inside of a virtual environment managed by - poetry. To use the tool you will have to activate it first with the - command `eval $(poetry env activate)` (poetry >= 2.x) or - `poetry shell` (poetry < 2.x). - - For development mode, install the tool with: `poetry install --with dev` - -## Usage - -Given a SPDX SBOM file with git repositories as input, this tool will generate -a set of Project Health metrics. These metrics are calculated using the data -stored on GrimoireLab about those repositories. If any of the repositories -is not available on GrimoireLab, the tool will add it to GrimoireLab to have -it analyzed. - -```bash -grimoirelab-metrics spdx.xml \ - --grimoirelab-url http://localhost:8000 \ - --grimoirelab-user user --grimoirelab-password password \ - --opensearch-url https://127.0.0.1:9200 \ - --opensearch-index events \ - --opensearch-user 'admin' --opensearch-password 'admin' \ - --verify-certs --opensearch-ca-certs /path/to/ca.pem \ - --from-date 2024-01-01 --to-date 2025-01-01 \ - --repository-timeout 3600 \ - --code-file-pattern "\.py$|\.js$" \ - --binary-file-pattern "\.exe$|\.tar$" \ - --pony-threshold 0.5 \ - --elephant-threshold 0.5 \ - --dev-categories-thresholds 0.8 0.95 \ - --grimoirelab-ecosystem "npm-training-set" \ - --grimoirelab-project "npm-popular-components" \ - --output metrics.json -``` - -The parameters needed to run the tool are: - -- A valid SPDX file -- GrimoireLab instance address -- OpenSearch instance address -- OpenSearch index name, where GrimoireLab events data are stored -- Output filename, where metrics will be written. - -This is an example of a valid SPDX file: - -```xml - - - SPDXRef-DOCUMENT - - 2025-02-07T00:00:01Z - Organization: Bitergia - - CC0-1.0 - GrimoireLab - SPDX-2.3 - mynamespace - - SPDXRef-bootstrap-gnu-config.bst-0 - Product: gnu-config - https://github.com/chaoss/grimoirelab-perceval.git - false - bootstrap/gnu-config.bst - git - - - SPDXRef-bootstrap-gnu-config.bst-0 - Product: gnu-config - https://github.com/chaoss/grimoirelab-core.git - false - bootstrap/gnu-config.bst - git - - ``` -### Running with Docker -The tool can also be run using the published Docker image. This is useful when you -do not want to install Poetry and the tool's dependencies locally. - -Build the Docker image from the repository: -```bash -docker build -t healthycode . -``` - -Then run the tool with a Git repository as the input: - -```bash -docker run --rm \ - healthycode \ - /opt/healthycode/.venv/bin/grimoirelab-metrics https://github.com/aboutcode/example.git \ - --grimoirelab-url http://localhost:8000 \ - --grimoirelab-user user --grimoirelab-password password \ - --opensearch-url https://127.0.0.1:9200 \ - --opensearch-index events \ - --opensearch-user 'admin' --opensearch-password 'admin' \ - --verify-certs --opensearch-ca-certs /path/to/ca.pem \ - --from-date 2024-01-01 --to-date 2025-01-01 \ - --repository-timeout 3600 \ - --code-file-pattern "\.py$|\.js$" \ - --binary-file-pattern "\.exe$|\.tar$" \ - --pony-threshold 0.5 \ - --elephant-threshold 0.5 \ - --dev-categories-thresholds 0.8 0.95 \ - --grimoirelab-ecosystem "npm-training-set" \ - --grimoirelab-project "npm-popular-components" \ - --output metrics.json -``` - -## Project Health Metrics - -This is the list of the metrics generated by this tool: +## Who is HealthyCode for? -### Pony factor +- Software and engineering teams deciding whether to adopt, upgrade, replace, or +drop a dependency in their codebases -The metric is defined as the number of individuals, who produce up to the -first 50% of the total number of code contributions (in descending order) -within a given time period. +- Open source program offices (OSPO) and software supply chains teams tracking +the packages used in their organization apps, systems and products -A low Pony Factor implies a high dependency on these individuals, making the -project vulnerable if they were to leave. +- Maintainers who want an outside view of their own project -*Also known as: Lottery Factor, Bus Factor, Contributor Absence Factor.* +- Researchers studying open source sustainability -CHAOSS definition of the [Contributor Abscence Factor](https://www.chaoss.community/kb/metric-contributor-absence-factor/). +## When not to use HealthyCode -### Elephant factor +HealthyCode measures or rather approximates an open source project's health. It +does not scan for vulnerabilities or check license compliance or check the +configuration or security posture of a project. For those, see: -The metric is defined as the number of unique organizations producing up to the -first 50% of the total number of code contributions (in descending order) -within a given time period. +- [OpenSSF ScoreCard](https://github.com/ossf/scorecard) for security posture and configuration +- [VulnerableCode](https://github.com/aboutcode-org/vulnerablecode) for vulnerability lookup +- [ScanCode.io](https://github.com/aboutcode-org/scancode.io) to orchestrate scans including health scans +- [ScanCode Toolkit](https://github.com/aboutcode-org/scancode-toolkit) for origin, license, copyright and dependencies +- [PurlDB](https://github.com/aboutcode-org/purldb) that also hosts the health/ API endpoint. +- [ClearlyDefined](https://https://github.com/clearlydefined/) that pre-scans, and curates open source packages. -It was first defined by Bitergia, and it applies the concept of the -Pony Factor metric and takes it to contributing Organizations. -Contributions are focused on Git commits, and the organization -is determined by the email address of the commit author. +## Getting started -CHAOSS definition of the [Elephant Factor](https://www.chaoss.community/kb/metric-elephant-factor/). +You access health data by Package-URL (PURL) using the [PurlDB](https://github.com/aboutcode-org/purldb) /health API. +This is accessible at https://health.purldb.io/api/health for demonstration. +For instance, check https://health.purldb.io/api/health/?purl=pkg:npm/semver or https://health.purldb.io/api/health/?purl=pkg:npm/lodash + +You call the health API for a package using its PURL (for example, +`pkg:npm/semver`) and get JSON back. If the package has already been analyzed, +the answer comes back immediately. If not, you can poll the same URL until the +metrics and scores are collected and computed. The request will return first a +"new" status and then a "submitted" status when the job is processed. +The results are cached for a week, and the JSON is returned last. -### Number of contributing organizations - -This metric quantifies the total number of distinct organizations whose members -have made contributions to an open source project over a specified period. - -Contributions are focused on Git commits, and the organization -is determined by the email address of the commit author. - -CHAOSS definition of the [Organizational Diversity](https://www.chaoss.community/kb/metric-organizational-diversity/). - -### Number of organizations contributing recently - -This metric quantifies the number of unique organizations whose members have -actively made contributions to an open source project within the last 90 days. - -Contributions are focused on Git commits, and the organization -is determined by the email address of the commit author. - -CHAOSS definition of the [Organizational Diversity](https://www.chaoss.community/kb/metric-organizational-diversity/) -filtered by a specific timeframe. - -### Number of recent contributors - -This metric quantifies the total count of unique individuals who contributed -within the last 90 days. - -CHAOSS definition of [Contributors](https://www.chaoss.community/kb/metric-contributors/) filtered by a -specific timeframe. - - -### Number of recent commits - -This metric counts the total number of commits made to the project within the -last 90 days. - -CHAOSS definition of [Code Changes Commits](https://www.chaoss.community/kb/metric-code-changes-commits/) filtered by -the last 90 days. - -### Contributor Growth Rate - -This metric measures the growth rate of active contributors, defined as the -number of people sending one or more code contributions (in this case, -Git commits) in a given period. - -To calculate it, the period is split into two halves, and the number of active -contributors in each half is compared. The growth rate is the difference -between the second half and the first half, divided by the number of active -contributors in the first half. - -```math -GrowthRate (t_1, t_2) = \frac{C_a(t_2) - C_a(t_1)}{C_a(t_1)} -``` - -### Contributor Growth - -This metric measures the growth of active contributors, defined as the -number of people sending one or more code contributions (in this case, -Git commits) in a given period. - -To calculate it, the period is split into two halves, and the number of active -contributors in each half is compared. Growth is the difference between the -second half and the first half. - -```math -Growth (t_1, t_2) = C_a(t_2) - C_a(t_1) -``` +The processing goes from PurlDB `/health` api endpoint through ScanCode.io +`scan_repo_health` pipeline to the HealthyCode npm-health scoring model (for +now, for npms only), that calls GrimoireLab to collect project metrics. -### Number of active branches +## Access health data through ScanCode.io pipelines -This metric refers to the count of branches within a project's version control -repository (for this case, Git) that have seen recent development activity, -usually indicated by new commits. +Run HealthyCode using ScanCode.io `scan_repo_health` next to your other scans: +https://github.com/aboutcode-org/scancode.io/blob/main/scanpipe/pipelines/scan_repo_health.py -CHAOSS definition of [Branch Lifecycle](https://www.chaoss.community/kb/metric-branch-lifecycle/). +That pipeline takes a Git repository URL, collects the data with GrimoireLab, +and saves the metrics and score with the project results. Your ScanCode.io needs +access to a configured GrimoireLab instance and HealthCode models. -### Days since last commit +## Run HealthCode locally -This metric shows the number of days since the last commit was submitted to the -repository or the project. +HealthyCode needs a running [GrimoireLab 2.x](https://github.com/chaoss/grimoirelab/blob/2.x/README.md) with +OpenSearch. GrimoireLab collects the data, and HealthyCode turns the data into metrics +(like the menagerie of ponies and elephants) and computes a score. -CHAOSS definition of [Code Changes Commits](https://www.chaoss.community/kb/metric-code-changes-commits/) -taking into account the last commit activity. +The installation, setup, and command-line usage guidance for GrimoireLab is in +[grimoirelab-guide.md]/(grimoirelab-guide.md). -### Presence of an adopters file in a standard location -This metric verifies the existence of a file containing the full text of -the project's adopters. +## How Healthy works +1. HealthyCode is given a Git repository URL, or an SPDX SBOM that lists Git repositories. +2. HealthyCode asks GrimoireLab to collect each repository's history. Repositories GrimoireLab hasn't seen yet are added and analyzed. +3. When the data is ready, HealthyCode computes the metrics over a time window. The default is 12 months. +4. The npm health model turns those metrics into a score. +5. The metrics, score, and run settings are written to a JSON file. -Standard practice dictates that this file is named `ADOPTERS`, `ADOPTERS.md`, -or `ADOPTERS.txt`, and is located in the root directory of the project's -source code repository. +#### High-level overview of HealthyCode components -### Presence of a license file in a standard location +![High-level components of HealthyCode](https://github.com/user-attachments/assets/9fa54e45-a6f9-43b5-8c55-5d7159aa71ed) -This metric verifies the existence of a file containing the full text of -the project's chosen open source license. +#### Drilling down on PurlDB and ScanCode.io -Standard practice dictates that this file is named `LICENSE`, `LICENSE.md`, -`LICENSE.txt`, or `COPYING` (a convention historically used by GNU projects) -and is located in the root directory of the project's source code repository. +![PurlDB and ScanCode.io overview](https://github.com/user-attachments/assets/46d2c199-ddbc-43e0-8963-1a438bdcc20a) -CHAOSS definition of [License Declared](https://www.chaoss.community/kb/metric-licenses-declared/). +#### Drilling down on ScanCode.io + GrimoireLab -### Rate of contributors contributing infrequently vs. regularly +![ScanCode.io + GrimoireLab overview](https://github.com/user-attachments/assets/9d5d3bff-1e2d-4be3-8b86-132f02d9151e) -This metric involves categorizing contributors based on the frequency, -consistency, and intensity of their contributions over a defined period. It aims -to distinguish between individuals who contribute episodically -(infrequent contributors) and those who engage with the project consistently -and often substantially (regular or core contributors). -The CHAOSS community, for instance, defines -"[Occasional Contributors](https://www.chaoss.community/kb/metric-occasional-contributors/)" as -"people who make contributions to a project on an irregular basis". +## Methodology for selecting metrics -### Number of contributors who have contributed in previous periods +HealthyCode uses the Goal-Question-Metric (GQM) approach. You start with what +you care about (the goal), then work out what you need to know (the questions), +and only then choose what you can measure (the metrics). -This metric, often referred to as "returning contributors" or as an indicator -of "contributor retention," counts the number of unique individuals who were -active contributors in the last 90 days and had also made contributions in one -or more defined previous periods. +For example: -### Rate of commits over specified periods +- Goal: The project stays maintained +- Question: Would it survive losing its top contributor? +- Metrics: Pony factor (CHAOSS Contributor Absence Factor aka. "Bus Factor" ), +e.g., top author's share of commits over the last 12 months. -This metrics calculates the rate of commits between the last 90 days and -the last year. The purpose of this metrics is having an estimation of how -distributed contributions are, and the "momentum" of the project. +## (npm-health) scoring model -### Number of commits per repository +The model was trained on about 1,150 popular npm packages drawn from Census II, +Census III, and deps.dev. As open source experts, we reviewed about 200 of these +npms and classified them as "healthy" or "unhealthy" to create a training +dataset. We computed a logistic regression to determine and learn learned which +metrics best dsicriminate between the two groups and how much weight each metric +would get in the score computation. -Activity for each of the Git repositories analyzed. +All the backing data, the Jupyter notebook, a workflow diagram, and our expert +classifications guidelines are in the [model/npm](model/npm) directory for +reference. -CHAOSS definition of [Code Changes Commits](https://www.chaoss.community/kb/metric-code-changes-commits/) filtered -by repository. +This is a first version and the weights and thresholds will be tuned as more +packages are analyzed; we will define new models fo other ecosystems using the +same approch; and new questions and metrics will be added, drawing on +[CHAOSS](https://www.chaoss.community/), +[OpenSSF Scorecard](https://github.com/ossf/scorecard), +and foundation maturity models. -### Number of developers per repository +Eventually we plan to enrich back the [OpenSSF +Scorecard](https://github.com/ossf/scorecard) with these metrics as additionals +checks. -Unique participants producing commits in a given repository. -CHAOSS definition of [Contributors](https://www.chaoss.community/kb/metric-contributors/) filtered -by repository. +## How this project compares to other approaches and alterives -### File type metrics (code, documentation, or others) +We have published a review of the state-of-the-art here: +https://aboutcode.org/blog/npm-health-state-of-the-art -Type of activity done by developers, mainly split into code, documentation, and others. +People have been working on open source health from these three directions: -### Commit size metrics (added and removed lines) +1. Organizations improving how they reuse and contribute to open source, through +work such as OpenChain, the FINOS Open Source Maturity Model, the Good +Governance Initiative, and the TODO Group. -This provides an overview of the usual size of the code review processes and good practices -when submitting code. +2. Open source Foundations grading their own projects, such as the Apache +Software Foundation maturity model, the Eclipse Foundation Development Process, +and the CNCF project lifecycle. -CHAOSS uses the metric [Change Request Commits](https://www.chaoss.community/kb/metric-change-request-commits/) -as a way to use this as a filter. +3. Community projects that work across ecosystems, such as the CHAOSS project +that defines open health metrics and models (and where the GrimoireLab projects +lives), and the OpenSSF Scorecard that scores security practices, project +configuration and security posture. -### Message size metrics (total, mean, and median) +Multiple commercial companies also sell or publish open source package health +scores, but in most cases these are opaque, and you get the score without the +underlying data or weight calculation, making results hard to review, trace or +reproduce. + +## Most importantly, our results can be audited and traced -This metrics provides information about how extensive a developer is providing information about the -change in the commit. +The code, the collected data, and the model weights are open and public. Each +score is backed and comed by its supporting metrics; metrics comes from public +project data and history that can be verified. -### Frequency metrics for commits +And you can also rerun the analysis on your own infrastructure to verify, and +reproduce the results. The metrics follow CHAOSS definitions where they exist, +and data collection is run by GrimoireLab, a CHAOSS project. + +## Project status -This allows to understand the consistency of developers when producing code. +HealthyCode is under active development with npm as the first supported +ecosystem. The selected metrics and models will change as more packages are +analyzed, and if a result looks wrong to you, please open an issue with the +Package-URL and a hint of what you expected. That feedback will go directlyinto +improving the model with new and improved training data! -### Developer categories (core, regular and casual) +## HealthyCode in the larger AboutCode context -This metric structures the developers by their activity and consistency. The core developers produce +HealthyCode is an initiative that is part of [AboutCode](https://aboutcode.org), +a family of open source tools, open data, and open standards for healthy and +safe software supply chains, together with ScanCode, VulnerableCode, PurlDB, +DejaCode, and Package-URL. + +You can get results in the PurlDB health API by PURL, the standard package +identifier used in SBOMs and vulnerability databases and across software supply +chains. Exposing the /health endpoint in PurlDB also puts a package's health +data next to its license and origin data collected in PurlDB from ScanCode, +eventually delivering better visibility into the packages from multiple angles. + +HealthyCode is developed by the AboutCode and GrimoireLab community and +community contributors like you! + +## Contributing + +Issues and pull requests are welcome. +Please follow the [Code of Conduct](CODE_OF_CONDUCT.rst). + +Join the [AboutCode community Slack](https://join.slack.com/t/aboutcode-org/shared_invite/zt-31uzazd7l-tBHcqKUKkX6jUEPRLswiNw) +to chat with maintainers, contributors, and users. + +## License + +The code is licensed under GPL-3.0-or-later. See [LICENSE](LICENSE). + +Data produced by HealthyCode is licensed under +[CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/). diff --git a/grimoirelab-guide.md b/grimoirelab-guide.md new file mode 100644 index 0000000..457da98 --- /dev/null +++ b/grimoirelab-guide.md @@ -0,0 +1,145 @@ +# GrimoireLab Guide for HealthyCode + +Client to generate GrimoireLab metrics for project health using the +software analytics platform [GrimoireLab](https://github.com/chaoss/grimoirelab). + +![grimoirelab_metrics_schema.jpg](docs/images/grimoirelab_metrics_schema.jpg) + +## Installation + +### Prerequisites + +Instances of GrimoireLab 2.x and OpenSearch must be running before launching this tool. +Please check the [GrimoireLab](https://github.com/chaoss/grimoirelab/blob/2.x/README.md) +documentation in order to deploy the platform. + +To get this tool running, we also recommend using [poetry](https://python-poetry.org/). +This package manager will install the tool and all its dependencies. +You can install `poetry` by following [this guide](https://python-poetry.org/docs/#installing-with-pipx). + +Once you have `poetry` running, move to the next section. + +### Steps + +1. Clone the repository: + + ```bash + git clone git@github.com:Bitergia/grimoirelab-metrics.git + cd grimoirelab_metrics + ``` + +2. Install dependencies and tool: + + ```bash + poetry update + poetry install + ``` + + This will install the tool inside of a virtual environment managed by + poetry. To use the tool you will have to activate it first with the + command `eval $(poetry env activate)` (poetry >= 2.x) or + `poetry shell` (poetry < 2.x). + + For development mode, install the tool with: `poetry install --with dev` + +## Usage + +Given a SPDX SBOM file with git repositories as input, this tool will generate +a set of Project Health metrics. These metrics are calculated using the data +stored on GrimoireLab about those repositories. If any of the repositories +is not available on GrimoireLab, the tool will add it to GrimoireLab to have +it analyzed. + +```bash +grimoirelab-metrics spdx.xml \ + --grimoirelab-url http://localhost:8000 \ + --grimoirelab-user user --grimoirelab-password password \ + --opensearch-url https://127.0.0.1:9200 \ + --opensearch-index events \ + --opensearch-user 'admin' --opensearch-password 'admin' \ + --verify-certs --opensearch-ca-certs /path/to/ca.pem \ + --from-date 2024-01-01 --to-date 2025-01-01 \ + --repository-timeout 3600 \ + --code-file-pattern "\.py$|\.js$" \ + --binary-file-pattern "\.exe$|\.tar$" \ + --pony-threshold 0.5 \ + --elephant-threshold 0.5 \ + --dev-categories-thresholds 0.8 0.95 \ + --grimoirelab-ecosystem "npm-training-set" \ + --grimoirelab-project "npm-popular-components" \ + --output metrics.json +``` + +The parameters needed to run the tool are: + +- A valid SPDX file +- GrimoireLab instance address +- OpenSearch instance address +- OpenSearch index name, where GrimoireLab events data are stored +- Output filename, where metrics will be written. + +This is an example of a valid SPDX file: + +```xml + + + SPDXRef-DOCUMENT + + 2025-02-07T00:00:01Z + Organization: Bitergia + + CC0-1.0 + GrimoireLab + SPDX-2.3 + mynamespace + + SPDXRef-bootstrap-gnu-config.bst-0 + Product: gnu-config + https://github.com/chaoss/grimoirelab-perceval.git + false + bootstrap/gnu-config.bst + git + + + SPDXRef-bootstrap-gnu-config.bst-0 + Product: gnu-config + https://github.com/chaoss/grimoirelab-core.git + false + bootstrap/gnu-config.bst + git + + +``` + +### Running with Docker +The tool can also be run using the published Docker image. This is useful when you +do not want to install Poetry and the tool's dependencies locally. + +Build the Docker image from the repository: +```bash +docker build -t healthycode . +``` + +Then run the tool with a Git repository as the input: + +```bash +docker run --rm \ + healthycode \ + /opt/healthycode/.venv/bin/grimoirelab-metrics https://github.com/aboutcode/example.git \ + --grimoirelab-url http://localhost:8000 \ + --grimoirelab-user user --grimoirelab-password password \ + --opensearch-url https://127.0.0.1:9200 \ + --opensearch-index events \ + --opensearch-user 'admin' --opensearch-password 'admin' \ + --verify-certs --opensearch-ca-certs /path/to/ca.pem \ + --from-date 2024-01-01 --to-date 2025-01-01 \ + --repository-timeout 3600 \ + --code-file-pattern "\.py$|\.js$" \ + --binary-file-pattern "\.exe$|\.tar$" \ + --pony-threshold 0.5 \ + --elephant-threshold 0.5 \ + --dev-categories-thresholds 0.8 0.95 \ + --grimoirelab-ecosystem "npm-training-set" \ + --grimoirelab-project "npm-popular-components" \ + --output metrics.json +```