Arrow support (data representations of jagged arrays)
d
drahnreb
Working on nested arrays is a crucial task in most scientific fields. I think TileDB could perfectly leverage its strengths to support the community working in that field: https://youtu.be/jvt4v2LTGK0?t=1366
Working with Data Management in TileDB and Data Wrangling in awkward-array (https://github.com/scikit-hep/awkward-1.0) or other libraries with arrow support would be extremely beneficial workhorse.
Any updates on when Arrow will be supported?
Isaiah Norton
Hi drahnreb, we will add Arrow support in the 2.1 release, which we are planning for the end of August. TileDB supports variable-length subarrays, so for jagged arrays of scalars (at the very least), there should be a simple mapping. Awkward looks very interesting, thanks - I will take a look.
d
drahnreb
Isaiah Norton: I have seen your PR#1816 (https://github.com/TileDB-Inc/TileDB/pull/1816)
Great work. I understand it is an experimental C++ API implementation.
Is there plans for a python implementation or a more detailed documentation apart from the tests?
I would love to test it would be great if you could point me to further plans for arrow support.