Skip to main content

​
Prerequisites

  • Redis version ≥ 6.0
  • A Redis user with sufficient privileges to read data and manage keyspace notifications
  • Keyspace notifications enabled for change data capture

​
Redis Setup

​
1. Grant Database Access

  • Configure one of the Connection Options to ensure Streamkap can reach your Redis instance.

​
2. Enable Keyspace Notifications

Keyspace notifications allow clients to receive events affecting the Redis data set in some way. The Connector relies on Redis keyspace notifications for capturing changes. Set the following parameter in your Redis configuration:
  • notify-keyspace-events = KEA
This enables:
  • K: Keyspace events, published with __keyspace@<db>__ prefix
  • E: Keyevent events, published with __keyevent@<db>__ prefix
  • A: Alias for all event types (g$lshzxe)
Keyspace notifications and performanceEnabling keyspace notifications may impact Redis performance, especially in high-throughput environments. Monitor your Redis instance after enabling this feature.
  • Configure notifications for specific event types if full coverage isn’t needed.
  • Monitor memory usage and CPU utilization.
  • Consider using a dedicated Redis instance for CDC if performance becomes an issue.
You can enable keyspace notifications via: 
# Enable keyspace notifications via redis-cli
redis-cli CONFIG SET notify-keyspace-events KEA

# Verify the configuration
redis-cli CONFIG GET notify-keyspace-events
Configuration persistenceWhen using CONFIG SET, changes are not persisted across Redis restarts unless you also run CONFIG REWRITE or update the redis.conf file directly.

​
3. Create Database User

It’s recommended to create a separate user for the Connector to access your Redis instance. Below is an example that creates a user with appropriate permissions. 
# Create a user for Streamkap (Redis 6.0+)
# Replace {password} with a strong password
ACL SETUSER streamkap_user on >{password} ~* +@read +@connection +config|get

# Verify the user was created
ACL LIST
The ACL configuration above grants:
  • on: Activates the user
  • >{password}: Sets the password
  • ~*: Allows access to all keys
  • +@read: Allows all read commands
  • +@connection: Allows connection-related commands
  • +config|get: Allows CONFIG GET command for verification
Redis ACL compatibilityACL support requires Redis 6.0 or higher. For earlier versions, you can use the requirepass setting in redis.conf for password authentication, though this provides less granular access control.

​
4. Connection Details

You’ll need the following information for the Streamkap Setup:
  • Hostname/IP: The Redis server hostname or IP address
  • Port: Default is 6379
  • Database: Redis database number (0-15 by default)
  • Username: The username created above (if using ACL)
  • Password: The password for the user
TLS/SSL ConnectionsIf your Redis instance requires TLS/SSL, ensure you have:
  • Valid certificates
  • The correct hostname that matches the certificate
  • TLS enabled on your Redis instance

​
Streamkap Setup

Follow these steps to configure your new connector:

​
1. Create the Source

​
2. Connection Settings

  • Name: Enter a name for your connector.
  • Redis Type: Choose the type of Redis deployment:
    • Standalone: Single Redis instance
    • Cluster: Redis Cluster deployment
  • Hostname: Specify the hostname or IP address of your Redis server.
  • Port: Default is 6379.
  • Database: Specify the Redis database number to stream data from (0-15 by default).
  • Connect via SSH Tunnel: The Connector will connect to an SSH server in your network which has access to your Redis instance. This is necessary if the Connector cannot connect directly to your database.
  • Username: Username to access Redis. By default, Streamkap scripts use streamkap_user.
    • Leave empty for Redis versions < 6.0 without ACL support.
  • Password: Password to access the Redis instance.
  • Enable TLS?: Whether to use an encrypted TLS/SSL connection to the Redis server.
  • Allow Insecure Connections?: Allow connections with invalid or self-signed certificates (not recommended for production).

​
3. Advanced Parameters

  • Batch Size: Number of keys to process in a single batch. Default is 1000.
    • Adjust based on your Redis instance’s performance and network capacity.
  • Poll Interval (ms): How frequently to poll Redis for changes. Default is 100.
    • Lower values provide near real-time CDC but increase load on Redis.
Click Save.

​
Data Capture

The Redis Source Connector captures changes to Redis keys and publishes them to Kafka topics. Each change event includes:
  • Key: The Redis key that changed
  • Operation: The type of operation (set, delete, expire, etc.)
  • Value: The current value (for set operations)
  • Timestamp: When the change occurred

​
Topic Naming

By default, topics are named based on the Redis database:
  • Pattern: redis_db_{database_number}
  • Example: redis_db_0 for database 0

​
Supported Redis Data Types

The connector supports the following Redis data types:
  • String: Simple key-value pairs
  • Hash: Hash maps
  • List: Ordered lists
  • Set: Unordered sets
  • Sorted Set: Ordered sets with scores
  • Stream: Redis Streams (if using Stream Source mode)

​
Troubleshooting

Verify that keyspace notifications are enabled:

redis-cli CONFIG GET notify-keyspace-events
The response should include KEA or the specific event types you need. If not, enable them as described in Enable Keyspace Notifications.
If you’re experiencing authentication issues:

  • Verify the username and password are correct
  • For Redis 6.0+, ensure the user has appropriate ACL permissions
  • Check if requirepass is set in redis.conf for Redis versions < 6.0
  • Verify network connectivity and firewall rules
If Redis performance degrades after enabling CDC:

  • Reduce the batch size to lower per-batch load
  • Increase the poll interval to reduce polling frequency
  • Monitor Redis metrics (CPU, memory, network I/O)
  • Consider using a dedicated Redis replica for CDC
  • Limit keyspace notifications to specific event types instead of using KEA
If you’re not capturing all expected changes:

  • Verify keyspace notifications include all necessary event types
  • Check that the connector is running and connected
  • Review connector logs for errors or warnings
  • Ensure the Redis user has permissions to read all keys
  • Verify the database number is correct in the configuration