Skip to main content
This FAQ focuses on using MySQL as a source in streamkap, including general self-hosted setups and cloud variants (AWS RDS/Aurora, Google Cloud SQL, Azure MySQL Database). Streamkap’s MySQL connector provides real-time CDC with managed features like automatic scaling, UI setup, and ETL transformations.
A MySQL source in streamkap enables real-time Change Data Capture (CDC) from MySQL databases, capturing row-level inserts, updates, and deletes with sub-second latency. It uses the binary log (binlog) to stream changes to destinations, supporting snapshots for initial loads, schema evolution, and handling for schema changes. Streamkap offers a serverless setup via UI or API.What MySQL versions are supported as sources?Streamkap: MySQL 5.7+ for basic CDC; 8.0+ for advanced features like GTIDs and query log events; 8.4+ for latest enhancements. Compatible with MySQL 5.6.5+ in limited modes.
Streamkap supports:
  • Self-hosted/Generic (on-prem/VM)
  • AWS RDS MySQL (including Aurora)
  • Google Cloud SQL for MySQL
  • Azure MySQL Database
  • Any other MySQL compatible platform
Streamkap also supports standalone, primary/replica, high availability clusters, and multi-primary topologies.
  • CDC: Binlog-based for inserts/updates/deletes; captures schema changes
  • Snapshots: Ad-hoc/initial using global/table locks; incremental or blocking
  • Schema Evolution: Automatic DDL tracking; emits change events
  • Data Types: Standard mappings (numerics, temporals, strings, binary configurable, JSON, ENUM/SET); temporal/decimal/binary handling modes
  • Ingestion Modes: Inserts (append) or upserts
  • Security: SSL, authentication
  • Monitoring: Latency, lag, binlog metrics
  • Heartbeats: For low-traffic databases to advance binlog position via a heartbeat table
  • Streamkap adds transaction metadata, row filtering, and original SQL queries if enabled
Streamkap reads MySQL’s binary log (binlog) to capture committed operations, emitting changes as events.Requires binlog_format=ROW and binlog_row_image=FULL.
Uses GTID mode for position tracking in multi-primary setups, recommended for reliability.
binlog_row_image controls logged data for changes:
  • MINIMAL: Logs PK only (partial deletes)
  • FULL: Logs all columns (complete records, recommended)
Set to FULL for full delete handling.
Trigger ad-hoc at source/table level.
  • Methods: Global read lock or table-level; incremental (chunked by PK, default 1024 rows) or blocking
  • Modes: initial (default), always, initial_only, no_data, when_needed, configuration_based, custom
Streamkap simplifies triggering via UI.
  • Basics: Booleans (BOOLEAN), integers (INT8/16/32/64), floats (FLOAT32/64), strings (STRING), dates/timestamps (adaptive/connect modes)
  • Advanced: Binary (BYTES/base64/hex), decimals (precise/double/string modes), JSON (STRING/io.debezium.data.Json), ENUM/SET (STRING/io.debezium.data.Enum/Set)
  • Character: CHAR/VARCHAR/TEXT (STRING)
  • Unsupported: Non-UTF8; spatial (GEOMETRY); MyISAM tables; binlog expiration losses
Connectors use “offsets”—like bookmarks—to track their position in the database’s log or change stream. When no changes occur for long periods, these offsets may become outdated, and the Connector might lose its place or stop capturing changes.Heartbeats ensure the Connector stays active and continues capturing changes.There are two layers of heartbeat protection:

​
Layer 1: Connector heartbeats (enabled by default)

The Connector periodically emits heartbeat messages to an internal topic, even when no actual data changes are detected. This keeps offsets fresh and prevents staleness.No configuration is necessary for this layer; it is automatically enabled. We recommend keeping this layer enabled for all deployments.
Why we recommend configuring Layer 2While Layer 2 is crucial for low-traffic or intermittent databases, we recommend configuring it for all deployments. It provides additional resilience and helps prevent issues during periods of inactivity.
You can configure regular updates to a dedicated heartbeat table in the source database. This simulates activity, ensuring change events are generated consistently, maintaining log progress and providing additional resilience.How this layer is configured depends on the connection type (if supported by the Source):
  • Read-write connections (when Read only is No during Streamkap Setup): The Connector updates the heartbeat table directly.
  • Read-only connections (when Read only is Yes during Streamkap Setup): A scheduled job on the primary database updates the heartbeat table, and these changes replicate to the read replica for the Connector to consume.
This layer requires you to set up a heartbeat table—and for read-only connections, a scheduled job (e.g., pg_cron for PostgreSQL, event_scheduler for MySQL)—on your source database.For read-write connections (when Read only is No during Streamkap Setup), the Connector writes to the heartbeat table directly.
-- Create the streamkap schema
CREATE SCHEMA IF NOT EXISTS streamkap;

-- Switch to the streamkap schema
USE streamkap;

-- Create the heartbeat table with id, text, and last_update fields
CREATE TABLE streamkap_heartbeat (
    id INT AUTO_INCREMENT PRIMARY KEY,
    text TEXT,
    last_update TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Grant permission to the Streamkap user
GRANT SELECT, UPDATE, INSERT, DELETE ON streamkap.streamkap_heartbeat TO 'streamkap_user'@'%';

-- Insert the first row into the heartbeat table
INSERT INTO streamkap_heartbeat (text) VALUES ('test_heartbeat');
Use queries like SHOW BINARY LOGS for size/position; tools like Datadog/New Relic for lag.Best Practices: Retain 3–5 days; alert on expiration/growth.
  • MyISAM unsupported
  • Binlog expiration can lose events
  • PK changes require manual handling
  • TOASTed equivalents incomplete without FULL image
  • High binlog growth without monitoring
  • Schema changes during snapshots unsupported
  • Read replicas for snapshots in some cloud setups
  • Captures deletes as events with before images if binlog_row_image=FULL
  • Supports soft deletes
Encrypted connections (SSL), IP allowlisting, role-based access (dedicated user recommended).
The following table summarizes all permissions the Streamkap user needs. The setup steps on each source connector page walk through granting each one.Core permissions (always required):
PermissionScopePurpose
REPLICATION CLIENT*.*Read binlog metadata and positions
REPLICATION SLAVE*.*Read binlog events for CDC streaming
RELOAD*.*Flush operations required for consistent snapshots
SHOW DATABASES*.*Discover available databases and schemas
SELECT{schema}.*Read table data during initial and incremental snapshots
Snapshot signal table (required if GTID is not enabled):
PermissionScopePurpose
SELECTstreamkap.streamkap_signalRead signal table state
INSERTstreamkap.streamkap_signalTrigger snapshot signals
UPDATEstreamkap.streamkap_signalUpdate signal table state
DELETEstreamkap.streamkap_signalClean up processed signals
Heartbeat table (required if heartbeats are enabled):
PermissionScopePurpose
SELECTstreamkap.streamkap_heartbeatRead heartbeat state
INSERTstreamkap.streamkap_heartbeatWrite heartbeat records
UPDATEstreamkap.streamkap_heartbeatUpdate heartbeat timestamps
DELETEstreamkap.streamkap_heartbeatClean up old heartbeat records
If GTID is enabled (default on Google Cloud SQL), the signal table is not required.
If using read-only mode with GTID enabled, the signal table permissions are not needed. The heartbeat table still requires permissions for monitoring and diagnostics.
Combined GRANT statements:
SQL
-- Core permissions (always required)
GRANT REPLICATION CLIENT, RELOAD, SHOW DATABASES, REPLICATION SLAVE ON *.* TO 'streamkap_user'@'%';
GRANT SELECT ON {schema}.* TO 'streamkap_user'@'%';

-- Signal table (if GTID is not enabled)
GRANT SELECT, UPDATE, INSERT, DELETE ON streamkap.streamkap_signal TO 'streamkap_user'@'%';

-- Heartbeat table (if heartbeats are enabled)
GRANT SELECT, UPDATE, INSERT, DELETE ON streamkap.streamkap_heartbeat TO 'streamkap_user'@'%';
  • Binlog Buildup: Enable heartbeats; monitor retention; drop unused logs
  • Data Loss on Upgrade: Stop writes, capture all events, recreate positions post-upgrade, resnapshot
  • Missing Events: Ensure binlog enabled and tables included; check REPLICATION privileges
No, CDC cannot capture Views or most virtual database objects.Why Views cannot be captured:
CDC captures changes by reading the database transaction log (binlog, WAL, oplog, redo log, etc.). Views are query-time computations over base tables—they don’t store data or generate transaction log entries. When you query a view, the database engine executes the underlying SELECT statement against the base tables. Since views don’t store data, they don’t generate transaction log entries.What cannot be captured:
  • Views: Virtual tables with no physical storage or binlog entries
  • Temporary Tables: Session-scoped, not logged persistently in binlog
  • MEMORY Tables: In-memory storage engine, data lost on restart, not in binlog
  • BLACKHOLE Tables: Discard all writes, no data to capture
  • System Tables (information_schema, performance_schema, mysql): Metadata representations, not user data
  • CTEs (Common Table Expressions): Query-time constructs with no persistent storage
Solution:
Configure CDC on the underlying base tables that power your views. The view logic can be recreated in your destination or transformation layer.MySQL-specific notes:
  • MyISAM tables: Not supported for CDC as they don’t participate in transactions or generate complete binlog entries. Convert to InnoDB.
  • Views created with ALGORITHM=TEMPTABLE: Still not capturable; capture the base tables instead
Example:
If you have a view customer_orders_summary that queries tables customers and orders, capture the customers and orders tables instead, then recreate the view logic downstream.
For large database instances with many databases or tables, schema history can impact performance. Learn about optimization settings that can reduce schema history topic size and improve connector startup times.See the Schema History Optimization guide for detailed information on when and how to use these settings.
MySQL does not log ON DELETE CASCADE events to the binlog. InnoDB handles foreign key cascades internally, so child table DELETE events from cascading foreign keys are NOT captured by CDC — only the parent table delete is propagated.This is a MySQL limitation, not a Streamkap limitation. If you rely on cascading deletes, consider using application-level deletes instead of database-level cascades.See Delete Handling for more details.
This error typically affects databases with 1000+ tables when schema history optimization is enabled. The schema history may become incomplete.Resolution:
  1. Contact Streamkap support for schema history recovery
  2. Consider enabling Capture Only Captured Tables DDL in the source’s Advanced settings to prevent recurrence
  3. See Schema History Optimization for more information
  • Use dedicated replication user (not root)
  • Enable GTID mode for better failover
  • Limit databases/tables to reduce binlog size
  • Enable auto-vacuum equivalents; set binlog retention to 3-5 days
  • For deletes, use FULL binlog_row_image
  • Test snapshots in staging; monitor lag via tools
  • For cloud: Use provider monitoring; enable heartbeats for low traffic
Database version upgrades require careful planning to avoid data loss and minimize downtime for your CDC pipelines. See our Database Upgrade Guide for step-by-step instructions.