Python containers conversion to Fish

Hi everyone!

We are extensively using the Python API in our 3DEC computations, so first of all, thanks for that!

As of today (and at least in 3DEC), conversion of Python containers to Fish objects are not supported, except for length 2 list/tuple/numpy.arrays (+ vec.vec2) that would convert to vector2 and length 3 of the same objects (+ vec.vec3) that would convert to vector3.

Interestingly, conversion is more flexible on the other way around:

Fish Python
list (of any length) tuple
map dict
vector2 vec.vec2
vector3 vec.vec3
array not supported
matrix not supported
tensor not supported

I have no idea of how much work it would require, but would it be possible to complete the conversion of containers between Python and Fish in a near future update?

Maybe not all conversions, but being able to convert tuples and numpy.arrays from Python to Fish, and Fish arrays to numpy.arrays in Python would be great.

Thanks a lot!



Hello Théophile,
We added the conversions for more containers in the recent version (list python → list fish, dict python → map fish).
We can add
matrix <=> numpy array
tensor <=> vec.tens3
For array, I think it’s more like multi-dimensional lists.

Hi @Huy !

Ok, thanks.

As of 3DEC versions 7.00.161 and 9.00.167, setting a Fish list from a Python’s one still raises a ValueError: unknown type in conversion from PyObject to QVariant tuple.

In the meantime, I wrote Python functions that I give at the end of this post in case it could be of any help to some users. They deal with the following Python to Fish conversions:

  • list/tuple to list
  • dict to map
  • numpy.ndarray to array

Notice it is only a temporary solution until your native implementation since it is underoptimized (e.g., 1 s for instantiating à 100,000 elements array from Python to Fish versus 0.5 ms in pure Python). Besides, since it uses a bypass through strings, the float format precision might falsify the values in Fish.




PS: the code

Temporary solution to pass containers to Fish

Just a workaround until container conversion is natively dealt with through
Itasca's Python API. This workaround is underoptimized, e.g., a random 
(100, 1000)-shaped array takes 0.5ms to run in Python console, and
conversion to Fish takes ~1s (depends of course on the machine it's running on,
but it's the 2000x ratio that matters).


import itasca as it

import numpy as np

def format_fish_list(iterator, transformers={}):
    Formatting a Python iterator to a Fish list
    iterator: iterator (preferably list or tuple)
        The sequence to convert to Fish list
        Nested iterators are supported
    transformers: dict
        Custom type-to-string formatters with (key, val) = (type to format,
        formatting function), see examples
    out: str
        The formatted iterator as a Fish list constructor
    >>> test = (1, 'foo', 'bar', ('nested', True), 3.2548e-10)
    >>> format_fish_list(test)
    'list.seq(1, "foo", "bar", list.seq("nested", true), 3.2548e-10)'
    Custom converters
    >>> test = (99, {'a': 1, 'b': 2})
    >>> format_fish_list(test)
    KeyError: <class 'dict'>
    >>> def my_dict_conv(d):
    ...     return format_fish_list(d.values())
    >>> format_fish_list(test, {dict: my_dict_conv})
    >>> test = (99, {'a': 1, 'b': 2})
    'list.seq(99, list.seq(1, 2))'
    transform = {
        bool: lambda x: str(x).lower(),
        int: lambda x: str(x),
        float: lambda x: str(x),
        list: format_fish_list,
        str: lambda x: f'"{x}"',
        tuple: format_fish_list,
    items = ', '.join([
        for item in iterator
    return f'list.seq({items})'

def format_fish_array(arr):
    Formatting a numpy array to a Fish array
    arr: np.ndarray
        The array to format
    out: str
        The formatted array as a Fish array constructor
    >>> import numpy as np
    >>> test = np.array(((1,2,3), (4,5,6)))
    >>> format_fish_array(test)
    'array(list.seq(1, 4, 2, 5, 3, 6), 2, 3)'

    items = arr.flatten('F').tolist()
    shape = ', '.join([str(n) for n in arr.shape])
    return f'array({format_fish_list(items)}, {shape})'

def format_fish_map(dictionary):
    Formatting a dictionary a Fish map
    dictionary: dict
        The dictionary to format
    out: str
        The formatted dict as a Fish map constructor
    >>> test = {'hello': 'world', -62: 1.e10, True: False}
    >>> format_fish_map(test)
    'map(list.seq("hello", -62, true), list.seq("world", 10000000000.0, false))'
    keys, values = zip(*dictionary.items())
    return f'map({format_fish_list(keys)}, {format_fish_list(values)})'
def format_fish_container(
    container, var_name=None, instantiate=False, transformers={}
    Formatting a Python container to a Fish constructor
    container: list/tuple/dict/np.ndarray
        The container to set in Fish
    var_name: str
        The name of the Fish variable for storing the container.
        If None, the formatted container has no associated variable.
    instantiate: bool
        Whether the container should be instantiated on the fly in Fish
        (var_name must not be None).
    transformers: dict
        Custom type-to-string formatters with (key, val) = (type to format,
        formatting function), see doc from format_fish_list()
    >>> test = (1, -15., 'foo', ('nested', True), 3.2548e-10)
    >>> format_fish_container(test)
    'list.seq(1, -15.0, "foo", list.seq("nested", true), 3.2548e-10)'
    >>> format_fish_container(demo_seq, 'test')
    'test=list.seq(1, -15.0, "foo", list.seq("nested", true), 3.2548e-10)'
    Setting variable on-the-fly
    >>> format_fish_container(demo_seq, 'test', instantiate=True)
    'test=list.seq(1, -15.0, "foo", list.seq("nested", true), 3.2548e-10)'
    (1, -15.0, 'foo', ('nested', True), 3.2548e-10)
    >>> test_arr = np.array(((1.,2.,3.),(4.,5.,6.)))
    >>> format_fish_container(test_arr, 'test_arr', instantiate=True)
    'test_arr=array(list.seq(1.0, 4.0, 2.0, 5.0, 3.0, 6.0), 2, 3)'
    >>> test_dict = {'hey': 'yo', -62: 1.e3, True: False}
    >>> format_fish_container(test_dict, 'test_dict', instantiate=True)
    'test_dict=map(list.seq("hey", -62, true), list.seq("yo", 1000.0, false))'

    if isinstance(container, np.ndarray):
        items = format_fish_array(container)
    elif isinstance(container, dict):
        items = format_fish_map(container)
        items = format_fish_list(container, transformers)
    if var_name is not None:
        items = f'{var_name}={items}'
        if instantiate:
        assert not instantiate, 'Cannot instantiate with no var_name'
    return items

if __name__ == '__main__':
    # List/tuple
    test = (1, -15., 'foo', ('nested', True), 3.2548e-10)
    print(format_fish_container(test, 'test'))
    assert not'test')
    format_fish_container(test, 'test', instantiate=True)
    # Array: test with a silly formatting
    test_arr = np.array((((1,),(2,),(3,)),((4,),(5,),(6,))))
    format_fish_container(test_arr, 'test_arr', instantiate=True)
    # Dict
    test_dict = {'hey': 'yo', -62: 1.e3, True: False}
    format_fish_container(test_dict, 'test_dict', instantiate=True)

You can check maybe the next subversion.
Note that the fish array is not homogeneous as ndarray. So I think ndarray should be equivalent to fish matrix

Oh, ok, thanks for the hint.

In previous versions of 3DEC, I think arrays stored homogeneous types.
Converting to a matrix would restrict to 2-dimensional arrays, but it’s true that it’s their most common use.