Repository navigation
Add language neutral protocol conformance fixtures for concore core behavior #440
Description
Hi @pradeeban ,
I’ve been looking through the recent CI and testing improvements (like #196, #193, and #210), and the test setup is looking really solid but then it got me thinking about a follow up improvement that could make maintaining the project much easier in the long run.
As the project evolves, there's always a risk that protocol behavior might drift slightly across different implementations. Right now, we don’t really have a single, central "source of truth" that defines exactly how the protocol must behave, regardless of the language it's written in.
what i am proposing is a Language Neutral Conformance Suite ,which is basically adding a fixture based conformance layer. Essentially, we would create a set of standardized test cases (fixtures) that make the expected behavior explicit and easy to validate across the board.
What this would look like:
- Protocol Fixtures: A collection of JSON input/expected output pairs covering core behaviors (e.g.,
initvalhandling, read/write framing semantics, param parsing edge cases, and fallback behaviors for malformed inputs). - A
pytestConformance Runner: A tool to validate our current Python behavior against these new fixtures. - Implementation Agnostic Format: Keeping the test vectors completely language neutral so that other runtimes can reuse the exact same files later to prove compatibility.
Why i think it would be a very beneficial thing to add because it creates :
- Zero Ambiguity: It creates a crystal clear baseline for how the protocol should act.
- Future-Proofing: Gives us a strict compatibility standard for any new runtime implementations we build down the line.
- Safety: Catches weird edge-case regressions early.
If this sounds like a good direction to you, I’d be happy to open a focused PR to get it started.
For the first iteration, i would just introduce the fixture schema, add an initial set of tests, and wire it into our existing pytest runs. There would be absolutely no runtime behavior changes in this first step, i will be just laying the groundwork.
Let me know what you think! I'm happy to pivot if you have a different vision for the testing architecture.
Activity
pradeeban commentedon Feb 20, 2026
@Titas-Ghosh I like the idea. I just fear the practical challenges. Language bindings were contributed by different folks. There could be reasons for the way things diverging across languages. Making changes to change the conformance could break certain implementations (say, Verilog.) How do you plan to implement this without incurring such issues?
Titas-Ghosh commentedon Feb 20, 2026
Hi @pradeeban , what you have said is a completely valid concern which even i thought of and i totally agree, especially with bindings like Verilog that have very different architectural needs, forcing them into a rigid box could cause major headaches.
My goal isn't to force all languages to blindly mimic Python. Instead, I want to use this to map out what the protocol behavior actually is across the board. To ensure we don't break existing implementations, I'd like to propose a strict "observe first, enforce later" approach.
Here is what i have thought of planning out to avoid even minimum risks:
Phase 1: Foundation & Observation (Zero Risk)
We establish the test structure without changing any runtime code or failing any builds.
tests/protocol_fixtures/*.json) and a simple conformance runner for Python.continue-on-error). It will not fail any existing builds; it will simply publish a report of how Python behaves against the expected standard.Phase 2: Mapping the Territory
We expand the tests to other bindings(no forced compliance) just to document reality.
required(it's a genuine bug we all agree needs fixing)implementation_defined(a completely valid, language-specific design choice)known_deviation(a quirk we decide to document and leave alone)Phase 3: Controlled Hardening (Opt-in)
We only touch runtime code if we explicitly agreed it was necessary in Phase 2.
This phase will only be done after review/approval of Phase 2
requiredbugs.known_deviation.I think this phased approach is the best way to go to for safely implementing this, i’d be happy to just draft PR 1 so you can see what the JSON schema looks like in practice. If it feels too risky once you see the code, we can easily pause or scrap it.
How does it sound? I happy to iterate based on your suggestions .
Again thanks a lot for the feedback
pradeeban commentedon Feb 20, 2026
@Titas-Ghosh ok, go ahead. Having a PR will help me understand the implications better.
Titas-Ghosh commentedon Feb 20, 2026
Thanks a lot @pradeeban!
I’ll get started on PR 1 right away. i’ll keep it strictly focused on the JSON schema and the Python runner in report only mode. That way, you can see exactly how the infrastructure looks and feels without any risk to the existing builds.
I'll ping you as soon as it's up for review!
pradeeban commentedon Feb 20, 2026
Solid plan, @Titas-Ghosh.
Titas-Ghosh commentedon Feb 21, 2026
Hi @pradeeban, thanks a lot again for approving my plan!
I have just opened the Phase 1 PR here: #443
As promised, it is strictly the baseline infrastructure and the Python runner in report only mode, i have not touched any runtime code. Let me know if this structure makes sense to you once you have a chance to take a look!
Titas-Ghosh commentedon Feb 24, 2026
Hi @pradeeban,
Following up on the implementation in Phase 1, i’ve just raised the PR for Phase 2: #467
I have implemented the mapping/audit matrix to track protocol status across the different runtimes (Python, C++, Verilog, etc.). As we discussed, this remains strictly report only and non breaking, the non-Python bindings are currently marked as
not_auditedto ensure we aren't forcing any behavior changes yet.I'd love to get your thoughts on this matrix structure once you have a moment.
Also on a more personal note, i also wanted to extend a big thanks to you for merging the Phase 1 PR. It really boosted my confidence to see that foundation get in, and it’s definitely motivated me to keep working on this plan as efficiently as possible. I’m looking forward to your thoughts on the Phase 2 structure.
Titas-Ghosh commentedon Mar 1, 2026
Thanks for merging Phase 2, @pradeeban
Based on your earlier guidance in discussions #172 that protocol stabilization is a great fit for the actual GSoC coding period, i’m planning to keep the Phase 3 scope (matrix enforcement + cross-binding audit + Julia integration) strictly aligned with my proposal milestones.
will pause further implementation on this feature to focus on finalizing the proposal drafts, unless you feel a small Phase 3 groundwork PR is absolutely necessary before the application window opens.
pradeeban commentedon Mar 1, 2026
@Titas-Ghosh sounds good. Excellent contributions so far. Yes, you do not have to implement everything at this point. You just need a strategy and a plan for the GSoC coding period. Contributions are always welcome - but excessive contributions during the application period are not expected. :)
Titas-Ghosh commentedon Mar 1, 2026
Thank you, @pradeeban i really appreciate the guidance and the positive feedback, will shift my full focus now to detailing the strategy and milestones in the proposal draft, will be sharing the first draft soon....
test: audit cpp runtime against protocol conformance matrix (Phase 2 …