Add Python objects: instances of Python classes owned by VHDL

README.md

@@ -12,7 +12,9 @@ the selected simulator by a foreign language interface the package builds itself
 called through VHPIDIRECT (NVC, GHDL) or the FLI (Questa/ModelSim), or a VHPI application built with
 the simulator's own compiler driver (Riviera-PRO/Active-HDL). Values cross the interface with their
 VHDL types: `integer`, `real`, `string`, `boolean`, `std_ulogic`, `unsigned`/`signed`, the vector
-types, and `integer_array_t` as a NumPy array.
+types, and `integer_array_t` as a NumPy array. A `python_object_t` is an instance of a Python class
+owned by VHDL, for example the backend of a verification component, with a namespace of its own for
+every instance.
 
 ## Installation
 
@@ -88,8 +90,8 @@ environment and its installed packages.
 
 ## Documentation
 
-The user guide is in [docs/user_guide.rst](docs/user_guide.rst): sessions, `exec`, `eval`, `call`
-and its argument forms, `exec_file`, `import_run_script`, the type mapping, `integer_array_t` and
+The user guide is in [docs/user_guide.rst](docs/user_guide.rst): sessions, Python objects as the
+backends of verification components, `exec`, `eval`, `call` and its argument forms, `exec_file`, `import_run_script`, the type mapping, `integer_array_t` and
 NumPy, error reporting, and how the bridge works. A complete example covering all three simulator
 families is in [examples/embedded_python](examples/embedded_python).
 

docs/user_guide.rst

@@ -229,14 +229,17 @@ session, keeping the mutable model state on ``self`` rather than in globals:
 
 .. code-block:: vhdl
 
-    import_module_from_file(join(tb_path(runner_cfg), "accumulator.py"), "accumulator", session);
+    import_module_from_file(model_file, "accumulator", session);
     exec("model = accumulator.Accumulator()", session);
 
     y <= call("model.accumulate", arg(x), session => session);
 
 The module, and so the class, is shared by all sessions (see the caveats
 below), while every session has a ``model`` object of its own. Like all
 non-default sessions, this needs the Python bridge: NVC, GHDL or Questa.
+:ref:`Python objects <python_bridge:objects>` package this pattern: an object
+is an instance of a class in a session of its own, with no module loading or
+session handling left to the component.
 
 Caveats
 ~~~~~~~
@@ -261,6 +264,275 @@ functions, classes and variables, are separate. Everything else is shared:
 * All sessions end with the simulation. Test cases run in the same simulation
   (``run_all_in_same_sim``) share the sessions.
 
+.. _python_bridge:objects:
+
+Python objects
+--------------
+
+A ``python_object_t`` is an instance of a Python class owned by VHDL. Every
+object lives in a :ref:`session <python_bridge:sessions>` of its own, where it
+is bound to ``self``, so several objects, of the same class or not, never
+share names or state. This makes an object the natural backend of a
+verification component: every instance of the component gets an object of its
+own, and a testbench can create one and give it to a component.
+
+.. code-block:: vhdl
+
+    constant model : python_object_t := new_python_object(
+      "my_models.uart.UartModel", kwarg("baud", 115200), id => get_id("uart")
+    );
+
+    ...
+
+    count := call(model, "push", arg(byte));   -- calls self.push(byte)
+    call(model, "reset");                      -- no return value
+    check_true(eval_boolean(model, "self.is_idle()"));
+    exec(model, "self.trace = True");
+
+``new_python_object(class_name, args, id)`` names the class and the arguments
+of its constructor, given like the arguments of ``call``, including groups made
+with ``&``. The class is
+
+* a dotted name like ``"package.module.Class"``, imported from the module it
+  names, which is found like any other import: see :ref:`python_bridge:finding_code`.
+  No file name is involved, so the class is found the same way wherever the
+  object is created.
+* the name of a class defined in the default session, like ``"Class"``, for
+  example by ``exec`` or ``exec_file`` in the testbench, or reached through a
+  name defined there, like ``"np.ndarray"`` after ``exec("import numpy as np")``.
+
+A name that is neither, or that is not a class, is reported as an error of
+the object.
+
+``call(object, method, ...)`` calls a method, and ``eval(object, expr)`` and
+``exec(object, code)`` evaluate an expression and execute code in the session
+of the object, where the object is ``self``. They have the same arguments,
+result types and aliases as ``call``, ``eval`` and ``exec``, including the
+procedure forms taking a ``std_ulogic_vector``, ``signed`` or ``unsigned``
+result as an ``out`` parameter.
+
+new_python_object or create_python_object
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Both make an object and take the same arguments. They differ in when Python
+constructs it:
+
+.. list-table::
+   :header-rows: 1
+   :widths: 30 35 35
+
+   * -
+     - ``new_python_object``
+     - ``create_python_object``
+   * - Constructed
+     - On first use: the first ``call``, ``eval`` or ``exec``
+     - By the call itself
+   * - Use it for
+     - Constants, generics and the handles of verification components, made
+       during elaboration
+     - Variables in a process, for example in a test case
+   * - A wrong class or constructor argument
+     - Is reported at the first use
+     - Is reported by ``create_python_object``
+   * - During elaboration
+     - Works: nothing runs in Python
+     - Does not work: the simulator cannot run Python during elaboration
+
+Use ``new_python_object`` wherever the object is declared outside a process: a
+constant in an architecture or a package, a generic, or the record a
+verification component's constructor returns. Nothing happens in Python until
+the object is used, so it does not depend on the order of the processes or on
+``test_runner_setup``.
+
+Use ``create_python_object`` in a process when the object should exist, and
+its errors be reported, at a known point, for example when a test case starts:
+
+.. code-block:: vhdl
+
+    if run("Test a transfer") then
+      model := create_python_object("my_models.uart.UartModel", kwarg("baud", 115200));
+
+Identity
+~~~~~~~~
+
+The ``id`` of an object names its session and its logger,
+``get_logger(object)``, which reports the errors of its operations. It is
+optional and mainly makes the log readable. Without an
+``id``, objects are enumerated the way VUnit enumerates verification
+components: ``python_bridge:python:object:1``,
+``python_bridge:python:object:2`` and so on. The session of an object must
+be its own, so ``new_python_object`` with an identity already used by another
+object, or by a session made with ``new_session``, is reported as a failure on
+the logger of the identity and returns ``null_python_object``. Using
+``null_python_object`` is reported as a failure on ``python_logger``.
+
+From a model with module state to an object
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A model loaded with ``exec_file`` and keeping its state in module globals is
+shared by everything that calls it. Move the state into a class and give every
+user an object:
+
+.. code-block:: python
+
+    # Before: packet_model.py, loaded with exec_file
+    _packets = []
+
+    def make_packets(seed, n):
+        ...
+
+    # After: the state belongs to the instance
+    class PacketSource:
+        def __init__(self, width):
+            self.width = width
+            self.packets = []
+
+        def make_packets(self, seed, n):
+            ...
+
+.. code-block:: vhdl
+
+    -- Before
+    exec_file("packet_model.py");
+    call("make_packets", arg(seed), arg(n));
+
+    -- After: one object per source, found by module name
+    constant source : python_object_t := new_python_object(
+      "packet_model.PacketSource", kwarg("width", 32)
+    );
+    ...
+    call(source, "make_packets", arg(seed), arg(n));
+
+The class is found by its module name, so its directory must be on
+``sys.path``, which for a model next to its testbench means adding that
+directory to ``PYTHONPATH`` in the run script: see
+:ref:`python_bridge:finding_code`.
+
+Backends of verification components
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A verification component usually makes its object in the constructor of its
+handle, the way VUnit's own components make everything they need there.
+``get_python_object(object, class_name, args, id)`` returns ``object`` when one
+is given and makes a new one otherwise, like ``get_logger`` and ``get_id`` do
+for loggers and identities. Every instance then gets a backend of its own,
+unless the testbench gives it one:
+
+.. code-block:: vhdl
+
+    type uart_master_t is record
+      p_std_cfg : std_cfg_t;
+      p_model : python_object_t;
+    end record;
+
+    impure function new_uart_master(
+      baud : positive; model : python_object_t := null_python_object; id : id_t := null_id
+    ) return uart_master_t is
+      constant std_cfg : std_cfg_t := create_std_cfg(id, "my_vcs", "uart_master");
+    begin
+      return (
+        p_std_cfg => std_cfg,
+        p_model => get_python_object(model, "my_vcs.uart.UartModel", kwarg("baud", baud), get_id(std_cfg))
+      );
+    end;
+
+The component itself only calls the object of its handle:
+
+.. code-block:: vhdl
+
+    entity uart_master is
+      generic(master : uart_master_t);
+      ...
+
+      process
+      begin
+        wait until rising_edge(clk) and start = '1';
+        call(master.p_model, "transmit", arg(to_integer(data)));
+      end process;
+
+The handle is a constant made during elaboration, and so is its object, but
+nothing happens in Python until the first call. ``create_std_cfg`` numbers the
+instances made without an ``id``, ``my_vcs:uart_master:1``,
+``my_vcs:uart_master:2`` and so on, and their objects get the same identities.
+
+``filter_vc`` of the `embedded_python example
+<https://github.com/VUnit/vunit-python-bridge/tree/main/examples/embedded_python>`__
+is a complete component of this kind: the handle is in ``filter_vc_pkg.vhd``,
+the component in ``filter_vc.vhd`` and the Python model in ``filter_model.py``.
+Its ``Test Python objects as backends of verification components`` test case
+drives three instances, two with objects of their own and one sharing an object
+with the testbench.
+
+A component without a handle can take the path of its instance, ``'path_name``,
+as the identity, in the statement part of a process: GHDL leaves the instance
+labels out of ``'path_name`` and ``'instance_name`` when they are evaluated
+during elaboration, which would give all instances the same identity.
+``tests/path_counter_vc.vhd`` shows this:
+
+.. code-block:: vhdl
+
+    process
+      variable vc_id : id_t := id;
+      variable backend : python_object_t;
+    begin
+      if vc_id = null_id then
+        vc_id := get_id(path_counter_vc'path_name);
+      end if;
+      backend := get_python_object(model, "models.counter_model.Counter", kwarg("step", step), vc_id);
+      ...
+
+Several components can share an object, and so can a component and the
+testbench: the testbench makes the object and gives it to each of them. A
+scoreboard filled by a monitor and checked by the test is a typical case:
+
+.. code-block:: vhdl
+
+    constant scoreboard : python_object_t := new_python_object("my_models.Scoreboard");
+
+    ...
+
+    monitor_inst : entity work.my_monitor
+      generic map (
+        monitor => new_my_monitor(model => scoreboard)
+      )
+      port map (
+        ...
+      );
+
+    ...
+
+    check_equal(integer'(call(scoreboard, "num_mismatches")), 0);
+
+In the same way, a component can let its user choose the class, by taking its
+name as a ``string`` generic.
+
+Errors and limitations
+~~~~~~~~~~~~~~~~~~~~~~
+
+* Errors are reported on the logger of the object, named after what failed:
+  ``call("push") failed:`` for every form of ``call``, ``eval("<expr>", ...)
+  failed:`` for ``eval`` and ``exec(...) failed:`` for ``exec``, followed by
+  the Python error.
+* An object that cannot be created, for example since its class is not
+  found, reports the error once, as ``new_python_object("<class>") failed:``
+  or ``create_python_object("<class>") failed:``.
+  Every later use of the object, calls and assignments alike, fails with
+  ``The <class> object could not be created: <error>``.
+* The caveats of sessions apply: the module of a class, and any state kept in
+  it, is shared by all objects.
+* String arguments are passed to Python verbatim, see
+  :ref:`python_bridge:semantics`, which applies to constructor arguments too.
+* An ``integer_array_t`` argument is transferred to Python when it is made, so
+  it cannot be a constructor argument of an object made during elaboration,
+  before the simulation starts. Give the values to the object in a call
+  instead: ``call(model, "load", arg(values))``. See `#17
+  <https://github.com/VUnit/vunit-python-bridge/issues/17>`__.
+* An object that holds resources, such as files or threads, is closed by a
+  call like any other method: ``call(model, "close")``.
+* Objects need sessions, and so the Python bridge: NVC, GHDL or Questa. On
+  Riviera-PRO/Active-HDL, ``new_python_object`` reports that as a failure on
+  the logger of the object and returns ``null_python_object``.
+
 exec
 ----
 
@@ -727,12 +999,9 @@ How a file name is resolved
        ``pathlib.Path`` and so on
      - The current working directory of Python, which is the working
        directory of the simulator. See the warning below.
-   * - Python module imported by name, ``import model``
-     - ``sys.path``, which holds the directory of the run script first, like
-       ``python run.py`` does, the directories of ``PYTHONPATH`` (the
-       simulator inherits the environment of VUnit, so a ``PYTHONPATH`` set
-       in the run script before ``vu.main()`` is seen), and, while
-       ``exec_file`` executes a file, the directory of that file.
+   * - Python module imported by name, ``import model``, and the class of a
+       :ref:`Python object <python_bridge:objects>`
+     - ``sys.path``: see :ref:`python_bridge:finding_code` below.
 
 .. warning::
 
@@ -744,6 +1013,34 @@ How a file name is resolved
    ``Path(__file__).parent`` in the model, or pass an absolute name from VHDL,
    built with ``base_path`` or ``tb_path``.
 
+.. _python_bridge:finding_code:
+
+Python modules
+~~~~~~~~~~~~~~
+
+The interpreter in the simulator imports modules the way Python started on the
+run script does. Without any setup it finds:
+
+* modules in the directory of the run script, the first entry of ``sys.path``.
+  This is also how ``import_run_script`` imports the run script;
+* the packages of the Python environment VUnit runs in, for example an active
+  virtual environment, including packages installed with ``pip install -e``;
+* while ``exec_file`` executes a file, the modules next to that file.
+
+Python code anywhere else is found once its directory is on ``PYTHONPATH``,
+set in the run script before ``vu.main()``: the simulations inherit the
+environment of VUnit (see the recipes below). A verification component used by
+many testbenches can so keep its model in a directory of its own and name its
+class, for example ``"my_vc_model.Model"`` for a :ref:`Python object
+<python_bridge:objects>`. Installing the model as a package, for example with
+``pip install -e``, has the same effect without changing the run script.
+
+There is one ``sys.path`` and one ``sys.modules`` per simulation, shared by all
+:ref:`sessions <python_bridge:sessions>`. Two modules with the same name are
+therefore the same module, so a package is best named after the component it
+belongs to, and state kept at module level is shared by everything that
+imports the module.
+
 Recipes
 ~~~~~~~
 
@@ -965,9 +1262,10 @@ arguments, a 20 register status dump, wide
 ``call``, Python files executed with ``exec_file`` or imported with
 ``import_module_from_file``, two models loaded into a session each, and a
 Python model failing with the logger of the default session mocked. Its last
-test case drives ``python_model``, a verification component whose behaviour is
-the Python function in ``python_model.py`` rather than VHDL. The component finds
-that file by joining its name to ``base_path``, the directory of the run script, which
+test cases drive ``python_model``, a verification component whose behaviour is
+the Python function in ``python_model.py`` rather than VHDL, and ``filter_vc``,
+whose instances have Python objects as backends. ``python_model`` finds its file
+by joining its name to ``base_path``, the directory of the run script, which
 ``run.py`` sets as the base directory, while the testbench gives the files it
 loads itself relative to its own directory. See :ref:`python_bridge:file_paths`.