Open Source Data Tools

What it is

A catalog is the component that tells a query engine which tables exist and where their metadata lives, and that arbitrates commits when several writers touch the same table. Pangolin implements the Apache Iceberg REST specification, so PyIceberg, Spark, and other Iceberg clients can point at it directly.

Beyond the REST surface, it adds the pieces an operator tends to want around a catalog:

• Multi-tenancy with tenant-scoped namespaces and warehouses, checked by isolation tests running against the real authentication middleware.
• Git-style branching, tags and 3-way merge over catalog state, for experimenting against a branch before promoting it.
• Credential vending for AWS STS, Azure SAS and GCP downscoped credentials, so engines get scoped access rather than long-lived keys.
• RBAC, service users and API keys for CI, ETL and automated pipelines.
• Audit logging across more than 40 actions and 19 resource types, including authentication events.
• OpenID Connect with PKCE, id_token signature validation through JWKS, discovery, and rate-limited key rotation, for Google, Microsoft, Okta or any IdP you give an issuer.
• Warehouse cloud credentials encrypted at rest with AES-256-GCM, and rate limiting on the authentication endpoints per source address and per account.
• Pluggable metadata backends: PostgreSQL, SQLite, MongoDB or in-memory.
• Federated catalogs, proxying an external Iceberg REST catalog through the same endpoint.
• A management UI in SvelteKit, two CLIs, and PyPangolin, a Python SDK.

Current status

Version 0.8.0, released 11 August 2026. Pre-1.0, and a long way from where it was two days earlier. 0.7.0 and 0.8.0 were a security and production-readiness push: 0.7.0 closed a privilege escalation that let any authenticated caller mint a Root token, and 0.8.0 did the operational work. The published artifacts are a GitHub release, the alexmerced/pangolin-api:0.8.0 container image, and PyPangolin 0.8.0 on PyPI.

Almost everything I would have warned you about a week ago is now closed:

• Warehouse cloud credentials are encrypted at rest with AES-256-GCM. They used to sit in the catalog database in plaintext.
• The authentication endpoints are rate limited, per source address and per account. The login endpoint used to be freely brute-forceable.
• OpenID Connect is implemented: PKCE, id_token validation via JWKS, iss, aud, exp and nonce checks, and discovery.
• Branch creation by copy is transactional, and the API no longer returns 200 on a copy that failed partway through.
• The missing Iceberg operations landed: registerTable, listViews, viewExists and dropView. Before this, DROP VIEW appeared to succeed against a catalog that had never heard of the operation.
• Backup and restore is drilled, not described. A script dumps, destroys the schema, restores and verifies. Measured against PostgreSQL 15 with 1,345 rows: 7 seconds to back up, 53 seconds to restore.
• There are real performance figures, measured server-side: 0.018ms for a readiness check, 0.023ms for /v1/config, and 0.060ms for an authenticated catalog list.
• MongoDB manages its index set and enforces the uniqueness the SQL backends get from primary keys.

The claim is backed by tests rather than inspection: 415 tests across 63 targets with zero failures, and 19 green CI jobs including an authorization matrix and a four-backend parity suite, run against live PostgreSQL, MongoDB and MinIO.

What is still open, in the order it would block a production deployment:

• GitHub logins cannot be OIDC-validated. GitHub issues no id_token and publishes no JWKS, so that path still rests on the userinfo endpoint. Setting PANGOLIN_OIDC_REQUIRE=true refuses any provider that cannot be validated.
• Multi-replica operation is constrained and unproven. It works on PostgreSQL, but OAuth needs session affinity, rate limiting is per replica, and it has not been load tested or soaked.
• Token revocation fails open. If the revocation check errors, the request proceeds, so a database blip means revoked tokens are accepted again.
• commitTransaction is absent on purpose, because there is no cross-table transaction underneath it and pretending otherwise would be worse.
• Smaller gaps: no tamper-evident audit trail or SIEM export, HS256 JWTs with no rotation, no MFA or account lockout, no point-in-time recovery, and session tokens stored unhashed at rest.

One caveat on the repository itself: the README still opens with an older alpha framing and a 0.6.0 version line. STATUS.md is the reconciled view and is the file to trust. Anything running a version before 0.7.0 should be upgraded and its tokens rotated.

Use cases

0.8.0 widens this from "development only" to a catalog you could run for real, within a posture the project spells out.

• An open catalog for an Iceberg lakehouse you control. Not tied to a vendor, covering the endpoints the common engines actually call, with credential vending so engines get scoped access rather than long-lived keys.
• A self-hosted catalog under a constrained posture. The project names the smallest credible one: PostgreSQL, a single replica, OAuth disabled, network restricted, PANGOLIN_ENCRYPTION_KEY set, and a backup you have actually restored using the drill script against a copy of your own data.
• Development and team environments. A catalog for building and testing Iceberg pipelines, where the SQLite and in-memory backends make a throwaway instance cheap.
• Multi-tenant internal platforms where the tenants are trusted. Tenant scoping is a required parameter throughout and is covered by an authorization matrix in CI.
• Learning how a catalog works. The commit path, requirement enforcement, credential vending and multi-tenancy are readable Rust in one repository.

Where I would still hesitate: untrusted multi-tenant use, or anything needing proven multi-replica scale, since that path is documented but not load tested. For those, a managed catalog or Apache Polaris remains the safer call.

What it is

SencilloDB sits between a JSON file you read and write by hand and a real database server. No daemon, no ports, no schema migrations to run against a service. You point it at a file or a folder and write transactions against it, and the data on disk stays readable in an editor.

npm install sencillodb

import { SencilloDB } from "sencillodb";

const db = new SencilloDB({ file: "./app.json" });

const user = await db.transaction(async (tx) => {
  await tx.ensureIndex({ collection: "users", field: "email", unique: true });

  return tx.create({
    collection: "users",
    data: { name: "Alice", email: "alice@example.com", age: 30 },
  });
});

• Transactions that are serialized and all-or-nothing. A throw inside the callback discards every change.
• Queries with $eq $ne $gt $gte $lt $lte $in $nin $regex $exists $and $or $nor $not, dot paths, sorting, limit, skip and count.
• Secondary indexes with optional unique constraints, used for equality, $in and range lookups.
• Three storage modes: one file, one file per collection loaded on demand, or one file per index bucket when a collection gets large.
• Durability options: atomic temp-file-and-rename writes, an optional append only log with an appendfsync policy, and an advisory cross-process lock.
• Operational basics: export, import, snapshot, versioned migrations, TTL expiry, change events, gzip compression and an LRU cache with a memory ceiling.
• Relations through populate, resolving foreign keys against another collection.

It is ESM only, needs Node 18 or later, ships TypeScript types, and has one runtime dependency.

Current status

Version 0.7.0, live on npm, pre-1.0 but the most settled of the three projects here. The API surface is stable in practice and the library does a small job completely.

0.7.0 shipped on August 10, 2026 as an audit release: the source was split into modules, twelve roadmap features landed, and eighteen bugs were fixed, several of which were real correctness problems. A document in an unloaded shard crashed update and destroy. Folder mode with the append only log enabled wrote a log it never replayed, so a fresh instance saw an empty store. Collection and index names went into file paths unsanitised, so a name like ../foo escaped the store folder. Those are fixed.

Two things to know before upgrading. Returned documents are now copies rather than live references into the in-memory store, which was the source of silent state corruption; pass clone: false for the old behaviour. And stores written by 0.7.0 cannot be read by 0.6.x, because secondary indexes gained a sorted key list. Upgrades in the other direction are handled in place.

The scope is deliberately narrow, and I would rather name the boundary than let you find it. It is one writer process, data sized to what you can comfortably scan and back up as JSON, and no server-side access control, replication or remote sync. Past that, SQLite, Postgres or MongoDB is the right answer. Avoid it on network filesystems such as NFS and SMB, where the atomic rename and lock files it depends on are not dependable.

Use cases

• CLI tools that need durable local state. It installs with your package and stores data next to the project or in a user config folder, with no setup step for whoever installs your tool.
• Electron and Tauri desktop apps where the database lives beside the application and the app snapshots, migrates and compacts its own store.
• Prototypes and small internal tools that have outgrown a hand-managed JSON file but do not justify choosing and running infrastructure yet.
• Test fixtures and integration tests that should exercise real persistence, including transactions and rollback, without booting Docker or a service.
• Bots, automation scripts and build tooling that need a durable cache, a manifest or task state with one writer and a predictable data size.

Reach for something else when many processes write frequently, when you need replication or access control, or when queries need joins, aggregations or full-text search.