Artie integration

Artie is a fully managed change data capture (CDC) platform. It streams changes from your PlanetScale Postgres database into warehouses and lakehouses such as Snowflake, BigQuery, Databricks, Redshift, and ClickHouse. This document will walk you through connecting your PlanetScale Postgres database to Artie.

Connect to Artie

Requirements

A PlanetScale Postgres database
Database or organization administrator access, to change cluster parameters
Access to your branch’s default role
An Artie account

Network access

If your database uses IP restrictions, add Artie’s fixed IP addresses to the allowlist before you connect. IP restrictions are set per database. To connect over a private network instead, set up AWS PrivateLink with Artie’s PrivateLink guide. Use your branch’s Private Host (*.private-pg.psdb.cloud) as the host when you connect.

Configure your PlanetScale cluster

Set cluster parameters

To enable logical replication, configure these parameters in your branch’s Clusters > Parameters tab:

Parameter	Required Value	Description
`wal_level`	`logical`	Enables logical replication.
`max_replication_slots`	2 x replicas and CDC consumers	Each Artie source reader uses its own slot. Count them alongside your replicas and any other CDC tools.
`max_wal_senders`	2 x replicas and CDC consumers	Not less than `max_replication_slots`.
`max_slot_wal_keep_size`	> 4GB	The most WAL a slot can hold back. Tune for your change rate and available disk space.
`sync_replication_slots`	`on`	Synchronizes Artie’s slot to your replicas so it survives a switchover or failover.
`hot_standby_feedback`	`on`	Required alongside `sync_replication_slots`.

wal_level, max_replication_slots, and max_wal_senders require a restart. Queue all three changes with “Queue parameter changes”, then click “Apply changes” once so the cluster restarts once. On production clusters, PlanetScale performs a rolling restart, which terminates direct connections during a brief switchover. After the cluster restarts, verify the WAL level:

SHOW wal_level;

The result should show logical.

Create a replication user

Connect to the primary as the default role on port 5432 and create a dedicated user for Artie:

CREATE USER artie_transfer WITH REPLICATION PASSWORD 'strong_password';

GRANT USAGE ON SCHEMA public TO artie_transfer;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO artie_transfer;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO artie_transfer;

Repeat the grants for each schema you want to replicate. If another role owns your tables, such as a migration role, run the GRANT SELECT and ALTER DEFAULT PRIVILEGES statements as that role too. PlanetScale routes connections by branch, so append your branch ID to log in: artie_transfer.<branch_id>. It’s the suffix on your default role’s username, such as postgres.12345678. In SQL, use artie_transfer without it.

Roles created via SQL do not appear on the Settings > Roles page and must be managed directly in Postgres. A user-defined role can only have the REPLICATION attribute if it inherits postgres, so this SQL user is the least-privileged option.

Create a heartbeat table (optional)

Artie’s replication slot only advances when a published table changes. If those tables can go hours without writes while the rest of the database keeps writing, the slot holds back WAL. To prevent this, create a heartbeat table, which Artie updates every 5 minutes. Skip this if your published tables change regularly. As the default role, run:

CREATE TABLE test_heartbeat_table (id text PRIMARY KEY, ts timestamp);
GRANT SELECT, UPDATE ON TABLE test_heartbeat_table TO artie_transfer;
INSERT INTO test_heartbeat_table (id, ts) VALUES ('1', NOW());

Create a publication

Artie only creates the publication itself when its user is a superuser. The PlanetScale role isn’t one, so create it as the default role. Artie uses a publication named dbz_publication unless you pick another one when you connect.

Once a table is in a publication, Postgres rejects UPDATE and DELETE on it unless the table has a primary key or a replica identity. Check your tables before you create the publication.

List the tables that would be affected:

SELECT n.nspname AS table_schema, c.relname AS table_name
FROM pg_class c
JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE c.relkind = 'r'
  AND c.relpersistence = 'p'
  AND n.nspname NOT IN ('pg_catalog', 'information_schema')
  AND n.nspname NOT LIKE 'pg_toast%'
  AND (
    c.relreplident = 'n'
    OR (c.relreplident = 'd' AND NOT EXISTS (
      SELECT 1 FROM pg_index i WHERE i.indrelid = c.oid AND i.indisprimary))
    OR (c.relreplident = 'i' AND NOT EXISTS (
      SELECT 1 FROM pg_index i WHERE i.indrelid = c.oid AND i.indisreplident))
  )
ORDER BY 1, 2;

If this returns any tables, set the replica identity to FULL on each one, or leave them out of a publication of specific tables:

ALTER TABLE table1 REPLICA IDENTITY FULL;

Then create a publication for all tables or for specific tables. To publish all tables, run:

CREATE PUBLICATION dbz_publication FOR ALL TABLES WITH (publish_via_partition_root = true);

This includes tables you create later, so give every new table a primary key or replica identity. To limit what Artie receives, publish specific tables instead. To publish specific tables, list them. If you created a heartbeat table, include it:

CREATE PUBLICATION dbz_publication
  FOR TABLE public.orders, public.customers, test_heartbeat_table
  WITH (publish_via_partition_root = true);

Remember to update your publication before adding new tables to your Artie pipeline:

ALTER PUBLICATION dbz_publication ADD TABLE public.refunds;

Set up Artie

In PlanetScale, navigate to your database and branch, and click the “Connect” button. Leave this tab open, as you’ll need the host shortly.
In the Artie dashboard, create a new pipeline and select “PostgreSQL” as the source.
Artie’s PlanetScale source is for PlanetScale Vitess databases. For PlanetScale Postgres, use the PostgreSQL source.
Fill in the connection information. TLS is enforced automatically, so no additional TLS configuration is required.
- Host: Paste in the host from the PlanetScale Connect page, for example xxxxxxxxxx-useast1-1.horizon.psdb.cloud. If you use PrivateLink, use the Private Host instead.
- Port: 5432
- User: artie_transfer.<branch_id>
- Password: The password you set for artie_transfer
- Database name: Select the database your tables are in (postgres by default)
Connect directly to the primary on port 5432. Don’t use the PgBouncer port (6432) or add |replica to the username.
For Publication, select dbz_publication from the list. Artie sets Publication mode and Publish via partition root to match. If you set them by hand, they must match the publication, or deployment fails validation, for example with please set the publication mode to 'all tables' to match the existing publication.
Under Advanced options:
- Turn on Failover slot, so Artie creates its replication slot with failover = true. PlanetScale requires this for the slot to survive a switchover or failover. This setting doesn’t change a slot that already exists. See Enable failover on an existing slot.
- If you created a heartbeat table, turn on Heartbeats. Artie shows SQL for the heartbeat table, which you already ran, so you can skip it.
Select the tables you want to replicate. For a table without a primary key, set Override primary keys in its table settings and set the table to REPLICA IDENTITY FULL. Otherwise, deployment fails with the following tables are missing primary keys.
Select your destination, review the pipeline settings, and click “Deploy.”

Sync overview

Artie starts the initial copy and change capture together:

Backfill: Artie reads the selected tables and loads their existing rows into your destination.
Streaming: Artie reads inserts, updates, and deletes from its replication slot while the backfill is running and afterward.

Changes captured during the backfill are queued. When a table’s backfill finishes, Artie applies those changes in order so the destination does not retain stale backfill values. Artie creates destination tables automatically and adds new columns as your schema changes.

Manage your pipeline

Enable failover on an existing slot

If PlanetScale detects a logical replication slot with failover = false, you will be notified and given 3 days to update the configuration. If the slot is not reconfigured within that window, PlanetScale will resume performing planned failovers as normal, and Artie’s slot won’t survive the next switchover or failover.

A slot that Artie created before Failover slot was turned on has failover = false. Using the default role, check Artie’s slot:

SELECT slot_name, active, failover FROM pg_replication_slots WHERE slot_type = 'logical';

If the failover column shows f:

In Artie, pause the pipeline with Pause reading & delivery so it releases the slot. The default, Pause delivery only, stays connected to the slot.
Follow Enabling failover on an existing replication slot.
Resume the pipeline. Artie continues from its last acknowledged position.

Pausing a pipeline

While a pipeline is paused with Pause reading & delivery, its slot holds back WAL. Keep pauses short:

If Artie falls further behind than max_slot_wal_keep_size, Postgres invalidates the slot. Artie can no longer resume from it, and the pipeline needs a new slot and a backfill.
A failover can proceed before an idle slot is synced to the new primary, which risks losing the slot. logical_slot_sync_timeout bounds how long a failover waits for slot sync. For more details, see Postgres High Availability with CDC.

Monitor replication lag and WAL retention with PlanetScale’s CDC monitoring queries and metrics.

Removing a pipeline

Artie doesn’t drop its replication slot or your publication when you delete a pipeline. After the pipeline has disconnected, check whether its slot still exists and drop it so it stops holding WAL. As the default role, run:

SELECT pg_drop_replication_slot('artie');
DROP PUBLICATION dbz_publication;

Replace artie and dbz_publication with your own names if you changed them in Artie. Postgres can’t drop a slot while it’s active. Only drop the publication if no other pipeline uses it.

Branches

Artie’s credentials target one branch, so each branch needs its own pipeline. A new, empty branch doesn’t include the parameters, user, heartbeat table (if you use one), or publication from these steps, so repeat them on that branch and create a separate Artie pipeline with its credentials. The branch ID in the username changes too. A branch restored from a backup includes the backup’s data, so artie_transfer, the heartbeat table, and the publication may already exist there. Check them before you repeat these steps.

Need help?

For help with Artie, contact Artie support or see the Artie documentation.

Need help?

Get help from the PlanetScale Support team, or join our Discord community to see how others are using PlanetScale.

Onboarding

Platform

Guides

Connect to Artie

Requirements

Network access

Configure your PlanetScale cluster

Set cluster parameters

Create a replication user

Create a heartbeat table (optional)

Create a publication

Set up Artie

Sync overview

Manage your pipeline

Enable failover on an existing slot

Pausing a pipeline

Removing a pipeline

Branches

Need help?

Need help?

​Connect to Artie

​Requirements

​Network access

​Configure your PlanetScale cluster

​Set cluster parameters

​Create a replication user

​Create a heartbeat table (optional)

​Create a publication

​Set up Artie

​Sync overview

​Manage your pipeline

​Enable failover on an existing slot

​Pausing a pipeline

​Removing a pipeline

​Branches

​Need help?

​Need help?

Connect to Artie

Requirements

Network access

Configure your PlanetScale cluster

Set cluster parameters

Create a replication user

Create a heartbeat table (optional)

Create a publication

Set up Artie

Sync overview

Manage your pipeline

Enable failover on an existing slot

Pausing a pipeline

Removing a pipeline

Branches

Need help?

Need help?