Smelter: frequently asked questions

Smelter is a data harmonization platform. It brings every source, internal and external, into one governed record: ingested, transformed, validated, approved and published, on an open lakehouse inside your own environment. Every team reads the same trusted numbers, and nothing reaches the record without passing your rules and, where it matters, a person.

Every team holding its own figure. When ERP, CRM, spreadsheets and external feeds disagree, people reconcile by hand and argue about whose number is right. Smelter builds one record from every system, holds bad data for review instead of publishing it silently, and keeps an audit trail of every rule, exception and approval.

Data engineers build pipelines in SQL and Python; analysts query and build dashboards; data stewards review quarantined rows and approve what is published; platform operators run the environment and onboard tenants. Business teams consume the trusted record through their BI tools and through Assembler.

Neither. Smelter is the governed record, not the dashboard: keep your BI tools on top through SQL. It is also not a cloud warehouse: it runs inside your perimeter, on open table formats any engine can read.

Bringing sources with different shapes, keys and quality into one consistent model: cleaning and typing values, normalising formats such as emails and phone numbers, de-duplicating and merging on business keys, conforming entities across systems and validating the result against your business rules before anyone relies on it.

Smelter refines data in stages. Ore is raw data exactly as it arrived. Alloy is cleaned, typed, de-duplicated and joined. Ingot is validated, business-ready data. Published is approved data pushed to operational systems such as Assembler. Data only moves forward, and each step is recorded.

Smelter is built on Meltano, with access to 300+ community connectors for databases, warehouses, CRM, finance, commerce, marketing, support and HR systems. You can also upload files (CSV, Excel, JSON, Parquet and more), import from URLs and shared spreadsheets, consume Kafka streams, and query external PostgreSQL, MySQL and SQL Server databases in place without copying them.

Yes. Loads can replace, append, merge on keys, or run incrementally from a watermark, with change-data capture where the source supports log-based replication. Smelter corrects unsafe combinations automatically, for example never appending a full reload on top of existing data.

In plain, versioned SQL with dbt models running on Trino, with shared macros for common clean-up, and in Python with Polars for logic SQL cannot express, plus Spark for large jobs. Notebooks combine SQL, Python and documentation. There is no proprietary language to learn.

Every pipeline runs on Dagster as a graph of assets. Jobs chain connector syncs, dbt builds, SQL statements and notebooks in sequence or in parallel branches, with parameters, per-step retries, timeouts, cron schedules with time zones, and re-running only the failed steps.

Business rules are declared in versioned YAML next to the pipeline: not null, unique, allowed values, ranges, required columns, row-count bounds, cross-field comparisons and compound uniqueness, plus dbt tests and reference lookups. Rows that pass are promoted; rows that fail are quarantined. If quarantining fails, nothing is promoted, so no row is ever dropped silently.

Rows that fail validation are held in a queue with every rule they failed and why. A data steward reviews them, fixes the cause, re-validates and promotes them, or discards them with a note. Every steward action is recorded.

Publishing to operational systems, backfills, schema changes, new-source onboarding and bulk quarantine decisions can all require sign-off. Once approved, the action runs automatically, exactly once, and its outcome is recorded against the approval.

Agreements between the team producing a dataset and the teams consuming it: freshness, row counts, completeness, null levels, required columns and types. Smelter checks them on every run and raises an alert when a contract is breached.

Yes. Lineage events from every run show where each dataset came from and what it feeds, dbt models carry column-level lineage, and the audit trail records every step and steward action from source to published record.

In Apache Iceberg tables on S3-compatible object storage that you own, on premises or in your private cloud. Iceberg is an open table format any compatible engine can read, so your data is never locked into a proprietary store.

Smelter's catalog, Project Nessie, lets you branch, compare and merge data like code. You can test a change on a branch before it reaches the record, and roll back with time travel if needed.

Yes. SQL over every zone runs through Trino, and BI tools such as Power BI, Tableau, Looker, Metabase and Excel connect over JDBC and ODBC, under each tenant's permissions. Smelter Studio also includes dashboards, scheduled reports and a semantic layer of shared business metrics.

Data that is harmonized is loaded into the lakehouse inside your perimeter, and never leaves it. For sources you would rather not copy, federation queries external databases in place.

Each tenant has its own catalog, data branch, roles and audit trail, and projects within a tenant have their own zones. The query engine only lets each tenant reach its own catalogs.

Users sign in through the Ory identity stack and are invited by administrators. Roles are monitor (read), operate (build, run and steward) and admin (approve and administer). Row-level security limits which records each user sees, and sensitive columns are tagged and masked: unmasking is a permission, not a default. API keys are hashed, scoped and can expire.

Yes. Smelter runs entirely inside your perimeter, on infrastructure you control, with no dependency on any external service.

SQL queries and schema changes, uploads, quarantine and approval decisions, publishing runs, access denials, tenant and membership changes and maintenance operations, each with who, what and when, searchable and exportable.

A browser workspace for the whole data team: SQL workbench, notebooks, data explorer, ingestion, quality, quarantine, approvals, jobs, lineage, data branches, catalog, contracts, dashboards, metrics, alerts and administration, in one place.

Yes. An MCP server exposes schemas, SQL, quarantine and approvals to AI agents under the same permissions as a user. Studio also offers AI assistance, such as text-to-SQL and generating dbt or Python transforms, using the model provider you choose.

A streaming add-on lands Kafka streams continuously into Iceberg tables for data that cannot wait for a batch; a machine-learning add-on adds experiment tracking, a model registry and AutoML on the governed record; and a disaster-recovery add-on provides a warm standby environment.

Installable use-case bundles: connector settings, transforms, quality rules, dashboards and alert rules for a specific problem, such as discovering unapproved AI tools from workspace audit logs, or sales and pipeline analytics.

They are sibling platforms, deployed independently. Quarry's completion events trigger ingestion of external facts into Smelter's raw zone; Smelter publishes approved data into Assembler's entities, and Assembler's change events flow back into Smelter. Smelter is the system the other two trust for clean, governed data.

Inside your environment: a single host for evaluation and smaller estates, or Kubernetes for production scale, in your private cloud or on premises, air-gapped where required. Only Studio is exposed to users. RUSH can operate Smelter for you inside your walls as a managed service.

Meltano, dbt, Polars, Dagster, Apache Iceberg, Project Nessie, Trino, Apache Kafka, Apache Spark, PostgreSQL, SeaweedFS, MLflow and the Ory identity stack. Every layer of the underlying stack is open source, and your data stays in open formats.

Smelter is licensed for a term of 3, 5 or 10 years in Starter, Growth and Enterprise editions. Each edition includes a defined number of named users, a volume of data under management, a number of connected sources and query capacity, with support coverage that increases by edition. Extra users, data and sources, and the streaming, machine-learning and disaster-recovery add-ons, are available above the allowance. Annual support and updates accompany the licence, and source onboarding, data-model and rules packages and managed operations are optional services.