MoonDetEval is a MoonBit library and native CLI for detector-independent bounding-box and instance-mask evaluation. It complements MBMOT, which measures multi-object tracking.
Choose an entry point:
| Need | Start here |
|---|---|
| Evaluate in MoonBit code | Install and call the library (moon add python123-ops/moondeteval@0.1.0) |
| Evaluate one or two prediction sets from files | Run the source examples (moon run cli --target native -- --help) |
| Inspect an existing JSON report | Read-only browser viewer |
| Compare more than two prediction sets | Batch evaluator |
The CLI help and argument diagnostics described here are in the current source checkout; the published Mooncakes library remains the frozen 0.1.0 release. For a reproducible problem, open a GitHub issue with the version, target, minimal non-sensitive input, expected behavior, and actual output.
The library accepts validated in-memory images, categories, ground truths, and detections, or COCO-style JSON files. It computes the 12 standard COCO bbox summary metrics and per-category values. Scores are ranked in descending order; equal scores preserve input order within an image/category, while cross-image accumulation follows sorted image IDs, matching the COCO reference. Crowd boxes use intersection over detection area and do not count as positive ground truths. Values of -1 mean a metric has no eligible ground truths. Boxes use continuous xywh coordinates, without pixel-endpoint +1 or automatic clipping.
The source checkout also provides CocoRle for mask evaluation. CocoRle::new(height, width, counts) accepts uncompressed COCO RLE counts: column-major pixels, alternating background/foreground runs, beginning with background (an initial zero run means the first pixel is foreground). It validates that the runs exactly cover one positive-size image of at most 2,147,483,647 pixels. area() returns foreground pixels; iou(other) returns ordinary mask IoU; crowd_iou(other) treats self as the detection and other as crowd ground truth, dividing intersection by detection area. Both IoU methods reject mismatched image dimensions and return zero for an empty denominator. The implementation processes runs directly and copies constructor input.
CocoRle::from_compressed(height, width, counts) reads COCO ASCII compressed counts, including signed run differences; to_compressed() exports the stored run sequence in the same format. Reading rejects invalid characters, truncated groups, runs needing more than seven characters, negative reconstructed runs, and lengths exceeding the image. Character positions in errors are zero-based. parse_coco_rle(source) accepts one segmentation RLE JSON object with size: [height, width] and counts as either a run array or a compressed string. Sizes and array entries must decode to nonnegative integer-valued numbers within the Int range. It uses the core JSON parser's numeric and duplicate-key behavior; producers should emit integer tokens and unique keys. The parser handles JSON escaping, including backslashes within counts.
These source-only APIs are absent from the published 0.1.0 package. The source checkout now evaluates RLE mask AP/AR through the library and native CLI. COCO polygon rings can also be rasterized into RLE using declared image dimensions. The differential script checks 35 fixed/seeded mask pairs in three input forms against pycocotools==2.0.7, including byte-for-byte compressed exports. Int-limit codec cases use explicit MoonBit fixtures: the Windows reference C decoder mishandles some large signed differences, so they are outside this script's reference geometry claim.
///|
let detection = @moondeteval.CocoRle::new(3, 4, [1, 2, 2, 3, 4])
///|
let truth = @moondeteval.CocoRle::new(3, 4, [0, 3, 2, 1, 6])
// detection.area() == 5; detection.iou(truth) == 0.5
// detection.crowd_iou(truth) == 0.6
///|
let compressed = @moondeteval.parse_coco_rle(
"{\"size\":[3,4],\"counts\":\"032N4\"}",
)
// compressed.to_compressed() == "032N4"; detection.iou(compressed) == 0.5COCO annotation IDs use signed 64-bit integers because real validation annotations can exceed the 32-bit range. The parser accepts integer JSON numbers within that range, rejects fractional or out-of-range IDs, and preserves the exact ID in diagnostics. Report IDs remain JSON numbers. Library callers constructing GroundTruth directly now pass an Int64 ID.
All 12 summary fields are unitless fractions in [0,1] when defined, in this order: ap, ap50, ap75, ap_small, ap_medium, ap_large, ar1, ar10, ar100, ar_small, ar_medium, ar_large. ap and ar* average over the configured IoU thresholds; ap50 and ap75 are fixed probes. Each field is -1 when its category/area has no eligible ground truth, including a dataset with no categories or only crowd truths. In comparison metrics arrays use this same order; an undefined side and its delta are JSON null, while the nested single-model reports retain -1. Categories and per-class comparisons follow dataset category order, image summaries follow ascending image ID, and PR curves follow category order then configured ascending IoU threshold. Optional arrays are present but empty when disabled; enabling them does not change summary metrics. JSON object key order is not an API guarantee.
COCO JSON errors identify the source file (in CLI messages), record index, and decoded field path where available. Duplicate image/category/annotation IDs and unknown annotation references report their record and field. The underlying JSON parser resolves repeated keys within one object to the last value before model validation; duplicate object keys are not rejected. Producers should emit unique keys. A JSON number larger than JavaScript's exact integer range remains exact in MoonDetEval's annotation ID output, but a JavaScript consumer must use a lossless JSON parser to preserve it.
The implementation is under compatibility validation. The differential script checks all 12 summary metrics, per-image matches, and exported 101-point precision/recall at all 10 default IoU thresholds against pycocotools==2.0.7 on nine synthetic scenarios. These include explicit ignore, crowd, empty detections, maxDets=100, area and IoU boundaries, score ties within and across images, out-of-image boxes, and fixed-seed mixed images. This is stronger evidence for the covered bbox cases, but does not establish full COCO equivalence.
In the default coco_bbox mode, COCO bbox evaluation uses iscrowd to determine whether a ground truth is ignored, even when the source annotation contains ignore=1. The input flag remains available in the model. Callers who need that flag to affect matching can construct EvalConfig::new(..., ignore_policy="respect_explicit"); the chosen policy is recorded in the report. The latter is a MoonDetEval extension, not a COCO compatibility claim.
Use parse_coco_mask_dataset, parse_coco_mask_detections_with_dataset, and evaluate_masks for segmentation-only COCO JSON, or construct MaskGroundTruth, MaskDetection, and MaskDataset from validated CocoRle values. Compressed strings, uncompressed run arrays and COCO polygon rings are supported. Polygon predictions use parse_coco_mask_detections_with_dataset(source, dataset) so canvas size comes from the declared image. RLE objects must have size=[image.height,image.width]; polygons use the declared image canvas. malformed RLE, duplicate IDs, unknown references and missing segmentation are rejected. Scores must be finite. Empty masks are valid; their overlap is zero, so an empty regular truth remains eligible and cannot match.
Mask matching uses foreground intersection/union, or intersection/detection foreground area for crowd. It uses the same stable ranking, ignored-truth priority, maxDets 1/10/100, area intervals, ten default IoUs and 101-point interpolation as bbox evaluation. GT area is required and retained from the annotation, even when it differs from the RLE pixel count. Prediction area always comes from the RLE; supplied bbox and area fields are ignored. For comparable pycocotools reference inputs, remove prediction bbox: COCO.loadRes prioritizes its bbox branch and otherwise overwrites detection area with box area.
evaluate_masks_with_config and compare_mask_predictions_with_config support custom IoU thresholds, diagnostics, image summaries and PR curves. The default EvalConfig::coco() is normalized to ignore_policy="coco_segm" in a mask report; both COCO policies use iscrowd to ignore GT. You may explicitly construct EvalConfig::new(..., ignore_policy="coco_segm"), or opt into respect_explicit. Bbox entry points reject the segm policy. Reports reuse schema 0.1.0 and add iou_type="segm"; bbox JSON retains its original fields, and report.iou_type() returns bbox or segm. Comparisons contain this marker in both nested reports. The viewer labels mask AP and rejects mixed bbox/segm comparisons. Python batch evaluation supports both bbox and segm, with one task per batch.
///|
let masks = @moondeteval.parse_coco_mask_dataset(ground_truth_json)
///|
let predictions = @moondeteval.parse_coco_mask_detections(detections_json)
///|
let report = @moondeteval.evaluate_masks(masks, predictions)
///|
let comparison = @moondeteval.compare_mask_predictions(
masks, predictions, other_predictions,
)moon run cli --target native -- examples/mask-ground-truth.json examples/mask-baseline.json --iou-type segm --compare examples/mask-candidate.json --diagnostics --image-ranking --pr-curves --min-ap50 1The checked-in 2×2 example uses masks with identical enclosing boxes and disjoint foregrounds. Its baseline AP is 0, candidate AP is 1, and candidate-minus-baseline image error count is -2. Add --output PATH to write a new report. All existing CLI limits and output/quality gates apply; --iou-type defaults to bbox, and segm rejects YOLO inputs. This is evaluation of supplied masks, with no inference or external polygon conversion dependency.
The mask AP differential script verifies 44 synthetic cases (20 fixed boundary cases and 24 seeded cases, seed 20261012) against COCOeval(iouType="segm") from pycocotools==2.0.7. It checks all 12 summary and per-class metrics, 69,690 exported PR samples, 7,746 diagnostic entries and per-image results, at absolute tolerance 1e-9, plus real-process CLI failures and paired output. This establishes the covered RLE behavior, with polygon validation described below. These APIs are unreleased and are not in the Mooncakes 0.1.0 package.
CocoRle::from_polygons(height, width, polygons) rasterizes a COCO array of flat coordinate rings ([[x0,y0,x1,y1,...], ...]) directly into RLE. parse_coco_segmentation(source, height, width) accepts this polygon JSON or a compressed/uncompressed RLE object and checks the canvas dimensions. A polygon ring has at least three x/y pairs and finite coordinates. Both clockwise and counterclockwise rings are accepted; all rings form a union, including nested rings, so winding does not define holes. Use RLE for a mask with holes. Empty ring lists, short/odd coordinate lists, nonnumeric coordinates and malformed JSON are rejected; collinear or collapsed rings may produce valid empty masks. Consecutive equal quantized vertices and an explicit closing vertex are normalized.
The conversion uses COCO's 5× coordinate quantization, truncation toward zero, edge sampling, boundary downsampling and clipping to the image canvas. It streams boundary points and builds RLE runs without allocating an image-sized pixel array. Negative and out-of-image coordinates are allowed when within the coordinate/work bounds. This is a COCO rasterization contract, not a generic continuous-area polygon calculation: GT annotation area is retained while prediction area comes from foreground pixels.
There are fixed per-mask limits: at most 100,000 input vertices, 2,000,000 boundary samples after consecutive-point normalization, and 2,000,000 cumulative RLE union run steps. 5*coordinate+0.5 must fit the signed 32-bit range. These limits reject oversized work before large sampling loops or during cumulative merges; they do not promise a hard process-memory cap. Errors identify the record and polygon ring, with coordinate indices for invalid values.
parse_coco_mask_dataset accepts polygon and RLE annotations together. For predictions, call parse_coco_mask_detections_with_dataset(source, dataset) to parse either form, check image/category references and rasterize at each image's declared size. The original parse_coco_mask_detections(source) remains an RLE-only API and reports the dataset-aware entry point when given polygons. parse_coco_rle also remains RLE-only. The native CLI automatically uses the dataset-aware parser in --iou-type segm mode, including comparisons.
moon run cli --target native -- examples/polygon-ground-truth.json examples/polygon-detections.json --iou-type segm --compare examples/mask-candidate.json --pr-curves --image-ranking --min-ap50 1This synthetic example compares polygon predictions with equivalent compressed RLE: both AP values are 1 and defined metric deltas are 0. The polygon reference script checks 178 fixed/seeded geometries against exact pycocotools==2.0.7 pixels, area and IoU; it also checks 42 polygon-input AP cases (42,420 PR samples and 1,330 diagnostic entries), mixed polygon/RLE comparison, output protection and malformed/oversized CLI input. Seed is 20261013; metric tolerance is 1e-9. Cases cover subpixel rounding, negative coordinates, clipping, concave/self-intersecting shapes, multiple/nested rings, repeated points and degenerate rings. This evidence covers these tested cases and bounds; it does not establish arbitrary geometry equivalence.
The polygon boundary algorithm is adapted from COCO's rleFrPoly and carries its BSD-2-Clause attribution in THIRD_PARTY_NOTICES.md, included in the source archive. All polygon APIs are unreleased and absent from the published Mooncakes 0.1.0 package.
There is real feature overlap with moonbit-visual-debug and moonbit-synthetic-vision. In the versions inspected on 2026-10-02 (0.2.2 and 0.3.0), the former exposes single-threshold detection precision/recall/F1, confusion data, and COCO/YOLO adapters; the latter exposes synthetic fixtures and integer-scale detection precision/recall. MoonDetEval focuses on the 12 COCO bbox AP/AR summaries, maxDets/crowd/area behavior checked against pycocotools, and a two-prediction native CLI with diagnostics and quality gates. This describes the inspected APIs, not a claim that no other package can evaluate detections.
Version 0.1.0 is published on Mooncakes. Add the exact version to your own MoonBit module with moon add python123-ops/moondeteval@0.1.0, then add "python123-ops/moondeteval" to that package's moon.pkg imports. This minimal in-memory consumer was compiled and run both against the release archive and from a separate module using the exact registry version, producing AP50=1:
///|
fn main {
let box = @moondeteval.Box::from_xywh(0.0, 0.0, 10.0, 10.0) catch {
error => abort(error.message())
}
let dataset = @moondeteval.Dataset::new(
[@moondeteval.Image::new(1, 20, 20)],
[@moondeteval.Category::new(7, "object")],
[@moondeteval.GroundTruth::new(9L, 1, 7, box, area=100.0)],
) catch {
error => abort(error.message())
}
let prediction = @moondeteval.Detection::new(0, 1, 7, box, 0.9) catch {
error => abort(error.message())
}
let report = @moondeteval.evaluate(dataset, [prediction]) catch {
error => abort(error.message())
}
println("AP50=\{report.summary().ap50()}")
}The optional read-only report viewer opens directly in a browser from a checkout. Generate a JSON file with --output report.json, open viewer/index.html, and choose that file. It accepts bbox or RLE mask single-model and --compare reports with schema_version=0.1.0. It shows the existing 12 metrics, class results, per-image changes, PR samples, and paged matching diagnostics; optional sections say when their CLI flag was not enabled. The viewer reads the file locally, sends no data to a server, and never recalculates AP/AR or comparison deltas. It accepts files up to 128 MiB; a browser may need more memory than the file size. It rejects numeric report IDs outside JavaScript's exact integer range rather than displaying rounded IDs; use a lossless JSON reader for those reports. The viewer is distributed with the source repository, outside the Mooncakes library archive.
For more than two COCO bbox or segmentation prediction files, the source-checkout batch evaluator evaluates every model against one frozen copy of the same ground-truth JSON and emits an index.json plus one full report per model. It records SHA-256 hashes of inputs and reports, exact source paths, replay commands, configuration, 12 metrics, and per-model differences from a named baseline. Undefined -1 metrics have null differences; metrics are never averaged across models. This tool does not infer comparability from unrelated existing report files. Use a new output directory, for example:
python examples/batch_eval.py --ground-truth examples/compare-ground-truth.json --model baseline examples/compare-baseline.json --model candidate examples/compare-candidate.json --baseline baseline --image-ranking --pr-curves --output-dir ../moondeteval-batch-resultsFor instance masks, use the same tool with one shared task selection:
python examples/batch_eval.py --ground-truth examples/polygon-ground-truth.json --iou-type segm --model polygon examples/polygon-detections.json --model rle examples/mask-candidate.json --model miss examples/mask-baseline.json --baseline polygon --diagnostics --image-ranking --pr-curves --output-dir ../moondeteval-segm-batch-resultsThis example produces AP values 1, 1, 0 and baseline-relative AP differences 0, 0, -1. Polygon and RLE encodings can coexist within a segm batch. --iou-type accepts bbox (default) or segm and applies to every model in the run. Every report must match that type; missing type markers are accepted only for legacy bbox reports. A segm report requires explicit iou_type="segm". Unknown/mismatched markers and conflicting COCO ignore policies are rejected before publishing the bundle, as are differences in configuration, ground-truth counts or evaluator version.
New indexes use format_version="2" and record iou_type at the top level and in every model entry. Format 1 was bbox-only; consumers should check the version and task before comparing indexes. The tool generates one comparable batch from frozen inputs; it does not merge unrelated indexes or existing reports. Each segm replay command retains --iou-type segm. Default bbox commands omit the new CLI flag to remain compatible with the published 0.1.0 native executable; segm requires a source CLI with mask support. Run python -m unittest examples/batch_eval_test.py -v for seven integration/boundary checks including exact report replay, three-input mixed encoding, failed-input cleanup and rejected report types.
Real-model validation also used fixed Ultralytics 8.3.0 YOLO11n-seg weights on 64 cached COCO val2017 images, with CPU inference at imgsz=640 and 320, conf=0.001, NMS IoU 0.7, max_det=100, and retina_masks=True. Masks came from original-resolution binary masks.data, encoded as RLE; category IDs were mapped by COCO class names. A third input filtered the 640 predictions at confidence 0.25. All three reports matched COCOeval(segm) summary/per-class metrics, 242,400 PR samples, 111,844 diagnostic entries and image counts at tolerance 1e-9; batch deltas matched direct comparison, and GT polygon-to-RLE conversion/removing supplied detection box area preserved metrics. Reference inputs remove detection bbox/area to select COCO's mask-area branch. These are subset validation results, not official full-validation accuracy or runtime ML dependencies.
The index and all reports appear only when the full batch succeeds. Existing output directories are refused. This standard-library Python tool is a development companion to the native CLI, not part of the Mooncakes package.
From a checkout of this repository, run:
moon run cli --target native -- examples/ground-truth.json examples/detections.json
moon run cli --target native -- examples/ground-truth.json examples/detections.json --diagnostics --min-ap50 0.8
moon run cli --target native -- examples/ground-truth.json examples/detections.json --output report.json
python examples/compare_demo.py
python examples/showcase_demo.pyThe first command's actual schema_version and summary fields are:
{"schema_version":"0.1.0","summary":{"ap":1,"ap50":1,"ap75":1,"ap_small":1,"ap_medium":-1,"ap_large":-1,"ar1":0,"ar10":1,"ar100":1,"ar_small":1,"ar_medium":-1,"ar_large":-1}}The command prints a JSON report with schema_version, evaluator_version, config, counts, summary, classes, diagnostics, and quality_warnings. The example's AP and AP50 are 1, while AR@1 is 0: the highest-scored prediction lands inside an ignored crowd box and occupies the single-detection slot. The second prediction matches the regular ground truth. counts includes raw input totals and evaluated_detections, the predictions retained after the all-area maxDets=100 cap for each image/category. A retained prediction need not be a true positive. Warnings identify out-of-image ground-truth and detection boxes by source type and record/image/category IDs. They appear even without --diagnostics, and never clip boxes or alter AP/AR. --diagnostics includes decisions for each configured IoU threshold using the all-area range and maxDets=100. Each entry has source IDs, status (tp, fp, ignored, or fn), reason, and matched IoU (zero when unmatched). FP reasons use fixed priority: duplicate, wrong_class, localization (same-class IoU at least 0.1), then background. These explanations do not affect metrics.
For a direct two-detector comparison, run:
moon run cli --target native -- examples/compare-ground-truth.json examples/compare-baseline.json --compare examples/compare-candidate.json --diagnostics
moon run cli --target native -- examples/compare-ground-truth.json examples/compare-baseline.json --compare examples/compare-candidate.json --image-ranking
moon run cli --target native -- examples/compare-ground-truth.json examples/compare-baseline.json --compare examples/compare-candidate.json --pr-curves--image-ranking adds image_summaries to each evaluation: per image tp, fp, fn_count, and ignored at fixed IoU 0.50, all areas, and maxDets=100 per image/category. These are detection counts, not per-image AP. The option works without --diagnostics and does not alter AP/AR. In comparison mode, image_changes ranks images by error_delta = candidate(fp + fn_count) - baseline(fp + fn_count), largest regression first, then image ID for ties. An image with no truths or predictions still has a zero-count row; crowd matches are ignored, not errors. The option is available to library callers through EvalConfig::new(..., include_image_summaries=true). Without it the arrays are empty. This ranking is a review aid, not an official COCO metric.
--pr-curves adds a shared recall_thresholds grid (0.00 to 1.00, 101 values) and pr_curves entries for each category and configured IoU threshold, in dataset category order and ascending threshold order. Each curve contains category_id, iou_threshold, 101 sampled precision values, max_recall, and ap. It uses the same all-area, maxDets=100 precision envelope as AP; the mean of a defined curve's samples equals its ap. With no eligible ground truths, all 101 precision values, max_recall, and ap are -1; with eligible truths but no detections they are zero. A custom config exports only its configured IoU thresholds, even though AP50/AP75 remain fixed summary probes. Library callers use EvalConfig::new(..., include_pr_curves=true). The option works without diagnostics or image ranking; in comparison mode each nested report carries its own curves. Without the option both arrays are empty. Existing --max-report-bytes also bounds the larger JSON output.
--compare evaluates both prediction files against the same ground truths and configuration. The comparison JSON contains complete baseline and candidate reports, then 12 overall metric changes and 12 changes per class in stable metric order. Every delta is candidate minus baseline; a COCO -1 undefined value becomes null in the comparison entries and its delta is null. The full nested reports still retain their original -1 representation. In the included synthetic example, baseline AP50 is 1, candidate AP50 is 0.25, and the delta is -0.75; at IoU 0.5 candidate detection IDs 0, 1, and 3 are wrong_class, localization, and duplicate. python examples/compare_demo.py checks these values and prints a short readable trace. The diagnostic reason labels are MoonDetEval explanations, not official COCO categories.
The native CLI can evaluate six-column YOLO prediction TXT files directly through a JSON manifest. The manifest's category_ids maps zero-based YOLO class indices to the ground-truth category IDs; each images entry names a ground-truth image ID and a TXT file relative to the manifest directory:
{
"category_ids": [42, 7],
"images": [
{"image_id": 12, "file": "labels/image-12.txt"},
{"image_id": 9, "file": "labels/image-9.txt"}
]
}These commands use the checked-in two-image fixture. The second command compares a COCO baseline against a YOLO candidate; the third deliberately fails its AP50 gate with exit code 3 while keeping the complete report in failed-gate.json.
moon run cli --target native -- examples/yolo/ground-truth.json examples/yolo/manifest.json --prediction-format yolo --diagnostics
moon run cli --target native -- examples/yolo/ground-truth.json examples/yolo/detections.json --compare examples/yolo/manifest.json --compare-format yolo --diagnostics
moon run cli --target native -- examples/yolo/ground-truth.json examples/yolo/manifest.json --prediction-format yolo --min-ap50 0.8 --output failed-gate.json--prediction-format and --compare-format independently accept coco (default) or yolo; --compare-format requires --compare. A YOLO TXT row is class_index center_x center_y width height confidence. Class/coordinate/score validation is the same as parse_yolo_detections. Blank TXT files are valid, and an image absent from the manifest has zero predictions. Detection IDs start at zero and follow manifest order, then nonblank TXT row order. Duplicate image IDs or file names, unknown image/category IDs, invalid field types, and relative paths containing empty, . or .. segments, backslashes or drive prefixes are rejected. A referenced file must exist and contain valid UTF-8; errors identify its path and, for malformed rows, the line. Path checks are lexical; the CLI operates on local files the caller supplies.
python examples/yolo_manifest_demo.py checks the 12 summary metrics, per-class results, diagnostics, and counts against equivalent COCO JSON, plus mixed-format zero deltas and failure behavior. This is a file-ingestion parity test, not an additional claim of COCO equivalence.
--min-ap50 N accepts a finite value in [0,1] and checks the candidate in comparison mode. --fail-on-warning fails when any evaluated input has an out-of-image box. Either failed gate keeps the complete report and exits with code 3; invalid options or evaluation input exit with code 2. When both gates fail, the AP50 failure is reported first.
With --output, the CLI writes the JSON file and prints a one-line summary. It refuses to overwrite an existing path, creates an exclusive temporary file beside the target, syncs it, then renames it without replacement. A failed evaluation creates no report; a failed gate keeps the complete report. A pre-existing .moondeteval.tmp file is never deleted automatically. The default CLI limits are 64 MiB per input file (including each YOLO TXT), 128 MiB total input bytes across ground truth, prediction files/manifests, and referenced TXT files, 100,000 total image/category/ground-truth/detection records across both models, an upper bound of 100,000 diagnostic entries across both reports, and 64 MiB of serialized report text. Use --max-input-bytes, --max-total-input-bytes, --max-records, --max-diagnostics, and --max-report-bytes with positive integers to change them. These guard against accidental oversized work; they are not a hard process-memory limit.
For performance measurements, build the native CLI with moon build cli --target native --release and add --timings. Successful runs print input, evaluation, serialization, and output durations in microseconds to stderr while leaving the JSON on stdout unchanged. input includes reading, parsing, and input validation; output includes report-size validation and writing to stdout or the requested file. The durations exclude process startup. In particular, --max-report-bytes is checked after serialization, so a large diagnostic report can consume substantial memory before the limit rejects it.
///|
let dataset = @moondeteval.parse_coco_dataset(ground_truth_json)
///|
let predictions = @moondeteval.parse_coco_detections(detections_json)
///|
let report = @moondeteval.evaluate(dataset, predictions)
///|
let ap50 = report.summary().ap50()
///|
let config = @moondeteval.EvalConfig::new([0.5, 0.75], include_diagnostics=true)
///|
let custom = @moondeteval.evaluate_with_config(dataset, predictions, config)
///|
let comparison = @moondeteval.compare_predictions(
dataset, predictions, other_predictions,
)
///|
let ap50_change = comparison.metrics()[1].delta()
///|
let yolo_predictions = @moondeteval.parse_yolo_detections(
yolo_txt,
dataset,
9,
[42, 7],
first_detection_id=100,
)
///|
let manifest = @moondeteval.parse_yolo_manifest(manifest_json, dataset)
///|
let retained = report.counts().evaluated_detections()
///|
let warnings = report.quality_warnings()
///|
let explicit_ignore = @moondeteval.EvalConfig::new(
[0.5],
ignore_policy="respect_explicit",
)The public API also supports Image::new, Category::new, Box::from_xywh, GroundTruth::new, Detection::new, and Dataset::new for callers with records already in memory. Detection IDs must be unique per evaluation; the JSON adapter assigns zero-based IDs from source order. Coordinates and scores must be finite, and boxes must have positive dimensions. Image and category IDs may be sparse or zero. Custom IoU thresholds must be strictly increasing in (0,1]; AP and AR average over them, while AP50 and AP75 remain fixed probes. Area bins and maxDets remain the COCO defaults. With no --output, the CLI prints JSON to standard output.
parse_yolo_detections accepts one image's six-column YOLO prediction TXT (class_index center_x center_y width height confidence). Ultralytics save_txt omits confidence by default, so use save_conf=True; five-column training labels and seven-column tracking rows are intentionally rejected here because AP requires a score and tracking IDs are outside this adapter's scope. Pass the ground-truth Dataset, its image ID, and an explicit category map: [42, 7] maps YOLO class 0 to category 42 and class 1 to category 7. It uses that image's width and height to convert normalized center coordinates to continuous boxes, skips blank lines, and gives nonblank rows sequential IDs starting at first_detection_id. Use a nonoverlapping ID range when combining multiple image files. Class indices must be mapped, center/score fields must lie in [0,1], and normalized width/height in (0,1]; errors identify the source line and field. A derived box that crosses the image edge is retained for evaluation and produces a quality warning. parse_yolo_manifest validates the ordered multi-image file list against a dataset; the native CLI then loads those files and assigns consecutive detection IDs.
The consumer example is a separate MoonBit module linked to this checkout by its moon.work file. Its MBMOT bridge converts detector frames into MoonDetEval predictions. The multi-frame form takes distinct target image_id values, an explicit MBMOT class-ID to dataset category-ID map, and a starting detection ID; it allocates consecutive IDs across frames and rejects unmapped classes, repeated frame IDs, ambiguous mappings, and ID overflow. It evaluates detector outputs only: MBMOT track IDs and MOT metrics are not reused. Run moon test --target all --deny-warn from examples/consumer for the source-workspace tests.
The same consumer also imports the moon-cv-geometry package at version 0.2.2. Its geometry adapter converts a Rect2 to a validated MoonDetEval Box, or maps four corners through a homography and evaluates the target-image envelope. The integration test gets AP50 1 for a known transform and rejects a projective horizon crossing or zero-area rectangle. No geometry dependency is added to the MoonDetEval library; applications opt into the bridge when they need cross-view coordinates. The geometry repository's examples/detection_bounds independently shows the source side of the workflow.
To verify the published versions together, run python examples/consumer/registry_smoke.py. It creates a temporary module with no moon.work, checks the resolved dependency tree for registry-sourced MoonDetEval 0.1.0, MBMOT 0.3.0, and moon-cv-geometry 0.2.2, runs the adapter tests on all four stable targets, then runs a native two-frame pipeline through geometry, MBMOT detections, and MoonDetEval (AP50=1). Pass --output-dir PATH to retain the independent module and registry-result.json; the destination must not exist. The target-image envelope is not clipped to the image boundary, and an axis-aligned envelope is not a rotated-box or mask evaluation.
moon fmt --check
moon check --target all --deny-warn
moon test --target all --deny-warn
python examples/compare_demo.py
python examples/yolo_manifest_demo.py
python -m pip install numpy==1.26.4 pycocotools==2.0.7
python reference_compare.py
python examples/mask_rle_reference.py
python examples/mask_ap_reference.py
python examples/polygon_reference.pypycocotools is used only for development comparison; the MoonBit library does not depend on Python. The native CLI uses moonbitlang/async@0.21.3 for file access.
开发 MoonDetEval 时，我使用了 OpenAI Codex，帮助我梳理功能、编写和修改 MoonBit 与 Python 代码、补充测试、排查问题以及整理文档。AI 参与了实际代码的辅助开发。
整体开发由我把控技术路线和质量边界。我根据项目需求逐步确定和调整开发目标，并通过实际运行测试、检查边界情况，以及与 pycocotools 对照结果来验证功能。
项目参考或改编的 COCO 算法会保留相应的来源和许可证说明。AI 工具不属于评测库的运行依赖，真实分割模型的推理用于外部验收。
MoonDetEval is Apache-2.0. COCO's official evaluator and mask encoding API are behavioral and format references for the original bbox and RLE implementations. The polygon rasterization rules are adapted from COCO rleFrPoly; its BSD-2-Clause notice is retained in THIRD_PARTY_NOTICES.md. The COCO API license is separate from this repository's license. pycocotools==2.0.7 and NumPy are development-only reference tools, while the CLI's MoonBit runtime dependency is moonbitlang/async@0.21.3. The checked-in example data is synthetic. Real COCO images/annotations and YOLO model weights used for external validation are not bundled or redistributed in this repository.