Parameters with continuously dynamically-sized setpoints

[thangleiter]

I would like to implement a new measurement class of the API as a parameter for the Swabian Instruments TimeTagger driver.

The measurement generates a histogram of time differences between subsequent clicks. Since these can be arbitrarily far apart, the bins are dynamic with only the smallest bin width being defined at start time. Hence, when getting the data at any point after the measurement has started, the array returned will (most likely) have a different shape than the last time it was fetched. To account for this, both the time bins (the setpoints in qcodes lingo) and the counts within (the actual data) are returned together.

The qcodes parameter most naturally suited for this (an array with setpoints) would of course be a ParameterWithSetpoints. However, that requires both an Arrays validator and setpoints with an Arrays validator with a given shape -- which is unknown at setup time. To avoid a chicken-egg problem when making the shape dynamic, i.e., a callback, one would have to call the getData() method of the API for each validator to determine the shape of the data. But then the actual get on the parameter might already have a new shape, because the measurement is ongoing in between calls.

Next, one could think about using MultiParameter. It requires some hacking to make the setpoints and shapes dynamic, but this sort of works:

class DynamicMultiParameter(MultiParameter):
    @property
    def shapes(self):
        return tuple(array.shape for array in self.get_latest())

    @shapes.setter
    def shapes(self, val):
        # the constructor wants to set this
        pass

    @property
    def setpoints(self):
        return tuple(tuple(np.arange(n) for n in shape) for shape in self.shapes)

    @setpoints.setter
    def setpoints(self, val):
        # the constructor wants to set this
        pass


class MyParam(DynamicMultiParameter):
    def get_raw(self):
        # api returns array of shape (n, 2)
        return tuple(self.instrument.api.getData().T)

While it works, it's not perfect because one of the parameters in this MultiParameter is the others' setpoint, and we'd like to reflect that relationship. The same goes of course for the trivial solution Parameter(..., get_cmd=api.getData).

My questions now are:

• does anybody have a neater solution?
• if not, can we adapt ParameterWithSetpoints to allow for dynamically sized setpoints/data somehow?
• or else, what would a new parameter type look like that reflects this sort of relationship, and would it have a place in the qcodes core?

[paul-lehmann-unibas]

Let me first say, that I fully second improvements like this to either ParameterWithSetpoints or MultiParameter.

I am assuming you would like to use the parameter with dond functions? Because, if not you could just work with the plain Parameter as you suggested.

I believe it's possible to reflect the dependency using MultiParameter. I think you can dynamically change the setpoints as you change the shape. Here is an example I adapted from one of my hacks to MultiParameter.

def api_call(num_indep):
    size = np.random.randint(10, 100)
    return np.vstack((np.arange(size), np.random.random((num_indep, size)))).T


class DynamicMulitParameter(qc.MultiParameter):
    """
    Parameter for getting an odmr sweep with the cw power.
    """

    def __init__(
        self,
        name: str,
        names: Sequence[str],
        instrument: InstrumentBase | None = None,
        labels: Sequence[str] | None = None,
        units: Sequence[str] | None = None,
        setpoints: Sequence[Sequence[Any]] | None = None,
        setpoint_names: Sequence[Sequence[str]] | None = None,
        setpoint_labels: Sequence[Sequence[str]] | None = None,
        setpoint_units: Sequence[Sequence[str]] | None = None,
        docstring: str | None = None,
        snapshot_get: bool = True,
        snapshot_value: bool = False,
        snapshot_exclude: bool = False,
        metadata: Mapping[Any, Any] | None = None,
        **kwargs: Any,
    ) -> None:
        super().__init__(
            name,
            names,
            tuple([() for _ in names]),
            instrument,
            labels,
            units,
            setpoints,
            setpoint_names,
            setpoint_labels,
            setpoint_units,
            docstring,
            snapshot_get,
            snapshot_value,
            snapshot_exclude,
            metadata,
            **kwargs,
        )

        self._shapes = tuple([() for _ in names])

        self.api_values = None

    def get_raw(self):
        self.api_values = api_call(num_indep=len(self.names)).T
        self._update_interdeps()
        return tuple(self.api_values[1:])

    # rename to update interdeps
    def _update_interdeps(self):
        """Update all the meta info for succesful get."""
        if self.api_values is None:
            self._shapes = tuple([() for _ in self.names])
            self.setpoints = None
        else:
            num_points = self.api_values.shape[1]
            self._shapes = tuple([(num_points,) for _ in self.names])
            # note that EACH item needs a sequence of setpoint arrays
            # so a 1D item has its setpoints wrapped in a length-1 tuple
            self.setpoints = tuple([(tuple(self.api_values[0]),) for _ in self.names])

    # This is needed because doNd detect shapes before get is called so it is important
    # to update before. Still pretty hacky and can fail I bet.
    @property
    def shapes(self):
        self._update_interdeps()
        return self._shapes

    @shapes.setter
    def shapes(self, new_shapes):
        self._shapes = new_shapes

Note that running this in a do1d loop I got warnings regarding shapes, but the resulting dataset looked ok as far as I could tell.

You can also return the setpoints as part of the tuple in get_raw and just set the setpoint_names to the name of the other subparameter of the MultiParameter.
Here as an example

class MultiParameterWithGetSetpoints(MultiParameter):
    def __init__(self, **kwargs):
        super().__init__(
            "test",
            names=("dep1", "dep2", "indep1", "indep2"),
            labels=("dep1", "dep2", "indep1", "indep2"),
            shapes=((10, 1), (10, 1), (), ()),
            setpoints=(
                (np.zeros(10), np.zeros((10, 1))),
                (np.zeros(10), np.zeros((10, 1))),
                (),
                (),
            ),
            setpoint_names=(("indep1", "indep2"), ("indep1", "indep2"), (), ()),
            setpoint_labels=(("indep1", "indep2"), ("indep1", "indep2"), (), ()),
            **kwargs,
        )

        self._shapes = {}

    def get_raw(self):
        return (np.arange(10), np.arange(10), np.arange(10), np.arange(10))

    @property
    def shapes(self):
        for f in inspect.stack():
            if f.function == "_get_shapes_of_multi_parameter":
                # return ((10,), (10,), (10,), (10,))
                raise TypeError("This is a test")
        return ((10, ()), (10, ()), (), ())
        # return (None

    @shapes.setter
    def shapes(self, new_shapes):
        self._shapes = new_shapes

    @property
    def setpoint_full_names(self):
        return (("indep1", "indep2"), ("indep1", "indep2"), None, None)

    @property
    def setpoint_labels(self):
        return (("indep1", "indep2"), ("indep1", "indep2"), None, None)

    @setpoint_labels.setter
    def setpoint_labels(self, new_sp_labels):
        pass

I find that by hacking MultiParameter you can make it to many things (it is not designed to do) like dynamically changing shape and setpoints or having multi-dimensional non-gridded setpoints. The only problem is shapes. It seems that different parts of dond (register_parameter, add_results, ...) require different values for shapes to not produce an error and it is not possible to satisfy them all just like that. In the example above I inspect the call stack to give back different answers depending what called shapes.

[thangleiter]

I am assuming you would like to use the parameter with dond functions? Because, if not you could just work with the plain Parameter as you suggested.

I don't actually work with dond as the sweep definitions are not flexible enough for my use case. The reason why I don't want to use Parameter is because I'd like the parameter to reflect the relationship between the two columns of the array returned by the API, which is: one column contains the setpoints of the other column.

The API call returns an array of size (n, 2) which corresponds to np.stack([times, counts], axis=1). Ideally, I'd like a parameter that represents the counts data to have times as its setpoints (and another array-valued parameter times). The problem now arises from the fact that n is dynamic, i.e., in increases with time. That's why ParameterWithSetpoints does not work as it currently is implemented.

If I adjust your code snippet a bit, it does what I want, mostly (one caveat is that the setpoints are not Parameter s and therefore don't have metadata like units).

DynamicMultiParameter

def api_call(num_indep):
    size = np.random.randint(10, 100)
    return np.random.random((size, num_indep))


class DynamicMulitParameter(qc.MultiParameter):
    """
    Parameter for getting an odmr sweep with the cw power.
    """

    def __init__(
        self,
        name: str,
        names: Sequence[str],
        instrument: InstrumentBase | None = None,
        labels: Sequence[str] | None = None,
        units: Sequence[str] | None = None,
        setpoints: Sequence[Sequence[Any]] | None = None,
        setpoint_names: Sequence[Sequence[str]] | None = None,
        setpoint_labels: Sequence[Sequence[str]] | None = None,
        setpoint_units: Sequence[Sequence[str]] | None = None,
        docstring: str | None = None,
        snapshot_get: bool = True,
        snapshot_value: bool = False,
        snapshot_exclude: bool = False,
        metadata: Mapping[Any, Any] | None = None,
        **kwargs: Any,
    ) -> None:
        super().__init__(
            name,
            names,
            tuple([() for _ in names]),
            instrument,
            labels,
            units,
            setpoints,
            setpoint_names,
            setpoint_labels,
            setpoint_units,
            docstring,
            snapshot_get,
            snapshot_value,
            snapshot_exclude,
            metadata,
            **kwargs,
        )

        self._shapes = tuple([() for _ in names])

        self.api_values = None

    def get_raw(self):
        self.api_values = api_call(num_indep=len(self.names)).T
        self._update_interdeps()
        return tuple(self.api_values)

    # rename to update interdeps
    def _update_interdeps(self):
        """Update all the meta info for succesful get."""
        if self.api_values is None:
            self._shapes = tuple(() for _ in self.names)
            self.setpoints = None
        else:
            num_points = self.api_values.shape[1]
            self._shapes = tuple((num_points,) for _ in self.names)
            self.setpoints = ((np.arange(num_points),), (self.api_values[1],))

    # This is needed because doNd detect shapes before get is called so it is important
    # to update before. Still pretty hacky and can fail I bet.
    @property
    def shapes(self):
        self._update_interdeps()
        return self._shapes

    @shapes.setter
    def shapes(self, new_shapes):
        self._shapes = new_shapes

>>> p = DynamicMulitParameter('p', names=['time', 'data'], units=['s', 'cts'])
>>> ds, *_ = do0d(p)

However, this does not work consistently and roughly every other time there is an error in dataset:

Traceback

ValueError                                Traceback (most recent call last)
Cell In[142], line 1
----> 1 do0d(p, write_period=1, use_threads=False)

File ~\AppData\Local\miniforge3\envs\py312\Lib\contextlib.py:81, in ContextDecorator.__call__.<locals>.inner(*args, **kwds)
     78 @wraps(func)
     79 def inner(*args, **kwds):
     80     with self._recreate_cm():
---> 81         return func(*args, **kwds)

File \src\qcodes\dataset\dond\do_0d.py:89, in do0d(write_period, measurement_name, exp, do_plot, use_threads, log_info, *param_meas)
     86 _set_write_period(meas, write_period)
     88 with meas.run() as datasaver:
---> 89     datasaver.add_result(*process_params_meas(param_meas, use_threads=use_threads))
     90     dataset = datasaver.dataset
     92 return _handle_plotting(dataset, do_plot)

File \src\qcodes\dataset\measurements.py:224, in DataSaver.add_result(self, *res_tuple)
    221 self._validate_result_shapes(results_dict)
    222 self._validate_result_types(results_dict)
--> 224 self.dataset._enqueue_results(results_dict)
    226 if perf_counter() - self._last_save_time > self.write_period:
    227     self.flush_data_to_database()

File \src\qcodes\dataset\data_set.py:1291, in DataSet._enqueue_results(self, result_dict)
   1286             new_results[st.name] = {
   1287                 st.name: self._reshape_array_for_cache(st, result_dict[st])
   1288             }
   1290 if self._in_memory_cache:
-> 1291     self.cache.add_data(new_results)

File \src\qcodes\dataset\data_set_cache.py:125, in DataSetCache.add_data(self, new_data)
    121 for param_name, single_param_dict in new_data.items():
    122     expanded_data[param_name] = _expand_single_param_dict(single_param_dict)
    124 (self._write_status, self._data) = (
--> 125     append_shaped_parameter_data_to_existing_arrays(
    126         self.rundescriber,
    127         self._write_status,
    128         self._data,
    129         new_data=expanded_data,
    130     )
    131 )
    133 if not all(status is None for status in self._write_status.values()):
    134     self._live = True

File \src\qcodes\dataset\data_set_cache.py:285, in append_shaped_parameter_data_to_existing_arrays(rundescriber, write_status, existing_data, new_data)
    281     else:
    282         shape = None
    284     (merged_data[meas_parameter], updated_write_status[meas_parameter]) = (
--> 285         _merge_data(
    286             existing_data_1_tree,
    287             new_data_1_tree,
    288             shape,
    289             single_tree_write_status=write_status.get(meas_parameter),
    290             meas_parameter=meas_parameter,
    291         )
    292     )
    293 return updated_write_status, merged_data

File \src\qcodes\dataset\data_set_cache.py:314, in _merge_data(existing_data, new_data, shape, single_tree_write_status, meas_parameter)
    307     raise RuntimeError(
    308         "Trying to add unexpected key to cache."
    309         "The following keys were unexpected: "
    310         f"{set(new_data.keys() - existing_data.keys())}"
    311     )
    313 new_write_status: int | None
--> 314 single_param_merged_data, new_write_status = _merge_data_single_param(
    315     existing_data.get(meas_parameter),
    316     new_data.get(meas_parameter),
    317     shape,
    318     single_tree_write_status,
    319 )
    320 if single_param_merged_data is not None:
    321     subtree_merged_data[meas_parameter] = single_param_merged_data

File \src\qcodes\dataset\data_set_cache.py:351, in _merge_data_single_param(existing_values, new_values, shape, single_tree_write_status)
    347     (merged_data, new_write_status) = _insert_into_data_dict(
    348         existing_values, new_values, single_tree_write_status, shape=shape
    349     )
    350 elif new_values is not None:
--> 351     (merged_data, new_write_status) = _create_new_data_dict(new_values, shape)
    352 elif existing_values is not None:
    353     merged_data = existing_values

File \src\qcodes\dataset\data_set_cache.py:375, in _create_new_data_dict(new_values, shape)
    372     elif new_values.dtype.kind == "c":
    373         data[:] = np.nan + 1j * np.nan
--> 375     data.ravel()[0:n_values] = new_values.ravel()
    376     return data, n_values
    377 else:

ValueError: could not broadcast input array from shape (96,) into shape (63,)

That unfortunately hints at a deeper incompatibility between the dataset and dynamically sized parameters.

[paul-lehmann-unibas]

Sorry my code snippet was a bit all over the place. I adapted it from a similar case in my own experiments and in the hurry made a couple of mistakes / made it unclear how to use the code.

I think there was a misunderstanding regarding the usage of DynamicMultiParameter. You should specify time as a setpoint:

class DynamicMultiParameter(qc.MultiParameter):
    """
    Parameter for getting an odmr sweep with the cw power.
    """

    def __init__(
        self,
        name: str,
        names: Sequence[str],
        instrument: InstrumentBase | None = None,
        labels: Sequence[str] | None = None,
        units: Sequence[str] | None = None,
        setpoints: Sequence[Sequence[Any]] | None = None,
        setpoint_names: Sequence[Sequence[str]] | None = None,
        setpoint_labels: Sequence[Sequence[str]] | None = None,
        setpoint_units: Sequence[Sequence[str]] | None = None,
        docstring: str | None = None,
        snapshot_get: bool = True,
        snapshot_value: bool = False,
        snapshot_exclude: bool = False,
        metadata: Mapping[Any, Any] | None = None,
        **kwargs: Any,
    ) -> None:
        super().__init__(
            name,
            names,
            tuple([(1,) for _ in names]),
            instrument,
            labels,
            units,
            setpoints,
            setpoint_names,
            setpoint_labels,
            setpoint_units,
            docstring,
            snapshot_get,
            snapshot_value,
            snapshot_exclude,
            metadata,
            **kwargs,
        )

        self._shapes = tuple([() for _ in names])

        self.api_values = None

    def get_raw(self):
        self.api_values = api_call(num_indep=len(self.names)).T
        self._update_interdeps()
        return tuple(self.api_values[1:])

    # rename to update interdeps
    def _update_interdeps(self):
        """Update all the meta info for succesful get."""
        if self.api_values is None:
            self._shapes = tuple([() for _ in self.names])
            self.setpoints = None
        else:
            num_points = self.api_values.shape[1]
            self._shapes = tuple([(num_points,) for _ in self.names])
            # note that EACH item needs a sequence of setpoint arrays
            # so a 1D item has its setpoints wrapped in a length-1 tuple
            self.setpoints = tuple([(self.api_values[0],) for _ in self.names])

    # This is needed because doNd detect shapes before get is called so it is important
    # to update before. Still pretty hacky and can fail I bet.
    @property
    def shapes(self):
        self._update_interdeps()
        return self._shapes

    @shapes.setter
    def shapes(self, new_shapes):
        self._shapes = new_shapes

dmp = DynamicMultiParameter(
    "test_dmp",
    names=["data"],
    setpoint_names=(("time",),),
    setpoint_units=(("s",),),
)
dmp.get()

indep = qc.Parameter("indep", set_cmd=None, get_cmd=None)
do1d(indep, 0, 10, 11, 0, dmp)

does this make more sense to you?

In case you don't want to use dond functions, you could just register the parameters using either register_custom_parameter or register_parameter with dummy parameters, which are only used to specify the metainformation. Then in the measurement context you just call the api and then call add_data with the numpy arrays:

Using register_custom_parameter

meas = qc.Measurement(name="histogram")

# specify independent parameters, i.e. things you set.
# Use "register_custom_parameter" to register a parameter to the measurement using strings instead of a qcodes.Parameter object.
meas.register_custom_parameter("time", "Time 0", "s")

# specify dependent parameters and which parameters they depend on, i.e. things you measure.
meas.register_custom_parameter("data", "Data, "cts", setpoints=("time",))
with meas.run() as datasaver:
    # Generate data with some arbitrary code
    values = api_call()

    # Save the data
    # Data comes in pairs (name_of_parameter, values).
    # The values may be array things, e.g. numpy.ndarray or pandas.Series. This is not so apparent from the docs.
    # When a parameter appears in add_result, all its setpoints must also appear and the values
    # must have the same shape.
    # You can play around and look at the documentation to understand it better.
    datasaver.add_result(
        ("time", values[0]),
        ("data", values[1]),
    )

using dummy parameters:

time = qc.Parameter("time", unit="s", set_cmd=False, get_cmd=False)
data = qc.Parameter("data", unit="cts", set_cmd=False, get_cmd=False)

meas = qc.Measurement(name="histogram")

qcodes.Parameter object.
meas.register_parameter(time)
meas.register_parameter(data, setpoints=(time,))

with meas.run() as datasaver:
    values = api_call()

    datasaver.add_result(
        (time, values[0]),
        (data, values[1]),
    )

Again, I adapted this from my own examples / code. Hope it is useful to you.

[thangleiter]

I think there was a misunderstanding regarding the usage of DynamicMultiParameter. You should specify time as a setpoint:

Ah I see. That was indeed not clear to me, thanks. In any case, the issue with do0d persists and even though I don't use it, it rules out using this in a public driver.

For completeness, the problem, likely also with the warnings in do1d, is because they set the shapes ahead of measurement time:

Qcodes/src/qcodes/dataset/dond/do_nd_utils.py

Line 73 in 4cfa6e9

meas.set_shapes(shapes=filtered_shapes)

If the maintainers think parameters with dynamic shapes are a worthwhile addition, I'm sure that would be an easy fix though. Maybe some private flag marking a parameter as dynamic.