ndtbl๏ƒ

License Build Documentation codecov pre-commit.ci C++

ndtbl is an n-dimensional table format and toolkit.

This repository currently contains two user-facing parts:

  • a header-only C++14 library in include/ndtbl/

  • a separate pure-Python package in python/ndtbl/ for reading, writing, inspecting, querying, and generating .ndtbl files

The C++ reader can also be built with optional POSIX mmap memory mapping support so payloads can stay file-backed instead of always being copied into heap memory entirely.

๐Ÿ—‚๏ธ Layout๏ƒ

  • app/: C++ command-line tool for inspecting .ndtbl files

  • benchmarks/: C++ benchmarks

  • cmake/: CMake modules

  • doc/: Sphinx and Doxygen documentation

  • include/ndtbl/: public C++ headers

  • python/ndtbl/: pure-Python package and ndtbl CLI

  • tests/: Catch2-based C++ test suite

๐Ÿ–ฅ๏ธ Which interface to use๏ƒ

Use the C++ library when you want to integrate ndtblโ€™s lookup logic directly into a C++ application.

Use the Python package when you want a pip-installable Python interface for creating .ndtbl files from NumPy arrays, inspecting existing .ndtbl files, or running queries against them. The Python package also provides a convenient CLI for inspecting, querying, and generating .ndtbl files. See python/ndtbl/README.md for Python package details.

If you want to inspect .ndtblfiles with a C++ command-line tool, use the C++ ndtbl-inspect executable built from app/. It is not prebuilt; it becomes available only after running the local CMake build.

๐Ÿ“‹ Prerequisites๏ƒ

Building the C++ project requires:

  • a C++14-compliant compiler for the header-only library interface

  • a compiler with C++20 support for the executables in app/ and tests in tests/

  • CMake >= 3.23

  • Doxygen only if you want to build the documentation

  • Catch2 only if you want to build the optional C++ test suite

๐Ÿ› ๏ธ Build๏ƒ

From the top-level ndtbl/ directory:

cmake -B build -Dndtbl_BUILD_TESTING=OFF -Dndtbl_BUILD_DOCS=OFF
cmake --build build

This produces the C++ command-line tool in build/app/:

  • build/app/ndtbl-inspect

Relevant CMake options:

  • ndtbl_BUILD_TESTING: build the C++ test suite, default OFF

  • ndtbl_BUILD_BENCHMARKS: build developer lookup benchmarks, default OFF

  • ndtbl_BUILD_DOCS: build the documentation, default ON for top-level builds

  • ndtbl_ENABLE_MMAP: enable POSIX-only mmap-backed payload reads, default OFF

  • ndtbl_ENABLE_MMAP_POPULATE: add Linux-only MAP_POPULATE to mmap-backed payload reads, default OFF; requires ndtbl_ENABLE_MMAP=ON

  • ndtbl_ENABLE_MMAP_DIAGNOSTICS: enable mmap payload residency diagnostics through mincore, default OFF; requires ndtbl_ENABLE_MMAP=ON

When ndtbl_ENABLE_MMAP=OFF (the default), read_field_group() and read_runtime_field_group() read payload data into owned heap storage. When ndtbl_ENABLE_MMAP=ON, supported POSIX builds use read-only memory mapping instead, which can reduce heap usage for large tables and enables shared memory access in multi-process environments. On Linux, ndtbl_ENABLE_MMAP_POPULATE=ON adds MAP_POPULATE to the mapping flags so the kernel faults the mapped payload in during mmap() instead of on first access.

When ndtbl_ENABLE_MMAP_DIAGNOSTICS=ON, mmap-loaded field groups can report OS page residency for their payload:

const auto info = group.payload_residency();
if (info.available) {
  // inspect info.resident_pages, info.total_pages, info.resident_fraction
}

The diagnostic is page-granular: it reports what mincore exposes for the mapped payload pages, not exact application RSS or exact bytes touched.

If you want to install the C++ headers and CMake package metadata:

cmake --install build --prefix /desired/prefix

To enable optional mmap-backed reads on supported POSIX platforms:

cmake -B build -Dndtbl_ENABLE_MMAP=ON
cmake --build build

To additionally pre-populate mapped payload pages on Linux:

cmake -B build -Dndtbl_ENABLE_MMAP=ON -Dndtbl_ENABLE_MMAP_POPULATE=ON
cmake --build build

To enable mmap payload residency diagnostics on Linux:

cmake -B build -Dndtbl_ENABLE_MMAP=ON -Dndtbl_ENABLE_MMAP_DIAGNOSTICS=ON
cmake --build build

โš™๏ธ C++ Tool Workflow๏ƒ

Inspect existing .ndtbl files:

./build/app/ndtbl-inspect output.ndtbl

๐Ÿ“ Interpolation๏ƒ

The standard C++ lookup path uses multilinear interpolation through explicit evaluate_all_linear() and Grid::prepare_linear() calls. This path uses 2^Dim table points per query and keeps the hot path allocation-free.

The C++ API also exposes local tensor-product cubic interpolation through explicit evaluate_all_cubic() and Grid::prepare_cubic() calls. Cubic interpolation uses 4^Dim table points, can be much more expensive in high dimensions, and may overshoot smooth-looking table data enough to produce unwanted values. Bounds handling is independent of interpolation order: queries outside the table domain can either clamp or throw according to the selected bounds_policy.

C++ Error Handling๏ƒ

The C++ API distinguishes invalid table data, system I/O failures, and invalid object state through ndtbl::FormatError, ndtbl::IOError, and ndtbl::StateError. All three derive from ndtbl::Error and provide the standard what() message interface:

try {
  const auto group = ndtbl::read_runtime_field_group<2>(path);
  // Use group.
} catch (const ndtbl::FormatError& error) {
  // The file is malformed, unsupported, truncated, or incompatible.
} catch (const ndtbl::IOError& error) {
  // Opening, reading, mapping, or another system operation failed.
} catch (const ndtbl::StateError& error) {
  // An operation required a populated runtime field group.
} catch (const ndtbl::Error& error) {
  // Any other ndtbl-specific runtime failure.
}

Argument, bounds, and size failures continue to use the standard std::invalid_argument, std::out_of_range, and std::overflow_error categories.

๐Ÿ Python Package๏ƒ

The repository also ships a separate Python package in python/ndtbl/.

That package installs a different CLI executable named ndtbl, with the subcommands:

  • inspect

  • query

  • generate

Install it from the package directory:

cd python/ndtbl
python -m pip install .

After that, the Python CLI is available:

ndtbl --help

See python/ndtbl/README.md for usage examples and Python API details.

๐Ÿงช Testing๏ƒ

Enable the C++ test suite during configuration:

cmake -B build -Dndtbl_BUILD_TESTING=ON
cmake --build build

Then run:

cd build
ctest --output-on-failure

โฑ๏ธ Benchmarks๏ƒ

The lookup-time benchmarks use Google Benchmark and measure query preparation, prepared evaluation, typed combined lookup, and runtime-erased combined lookup for representative 2D, 4D, and 6D tables. See benchmarks/README.md for the benchmark case definitions and interpretation.

Build the benchmark target:

cmake -B build -DCMAKE_BUILD_TYPE=Release -Dndtbl_BUILD_BENCHMARKS=ON
cmake --build build --target ndtbl_lookup_benchmarks

Run a benchmark:

./build/benchmarks/ndtbl_lookup_benchmarks

๐Ÿ“– Documentation๏ƒ

Online documentation is available at ndtbl.readthedocs.io.

To build the docs locally, first install the documentation requirements from the top-level ndtbl/ directory:

python -m pip install -r doc/requirements.txt

Then build the Sphinx target:

cmake -B build -Dndtbl_BUILD_DOCS=ON -Dndtbl_BUILD_TESTING=OFF
cmake --build build --target sphinx-doc

Open build/doc/sphinx/index.html in a browser to inspect the generated site.