Create New Pipeline

POST /pipelines

Create New Pipeline

cURL

curl --request POST \
  --url https://api.streamkap.com/pipelines \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '
{
  "name": "<string>",
  "destination": {
    "connector": "<string>",
    "name": "<string>",
    "id": "<string>"
  },
  "source": {
    "connector": "<string>",
    "name": "<string>",
    "id": "<string>",
    "topics": [
      "<string>"
    ]
  },
  "tags": [
    "<string>"
  ],
  "snapshot_new_tables": false,
  "transforms": [
    {
      "topic_id": "<string>"
    }
  ],
  "topic_auto_discovery_transforms": [],
  "created_from": "api"
}
'

200

{
  "name": "<string>",
  "sub_id": "<string>",
  "tenant_id": "<string>",
  "id": "<string>",
  "snapshot_new_tables": true,
  "topic_ids": [
    "<string>"
  ],
  "topics": [
    "<string>"
  ],
  "source": {
    "name": "<string>",
    "connector": "<string>",
    "id": "<string>",
    "connector_display_name": "<string>",
    "created_timestamp": "<string>",
    "sub_id": "<string>",
    "tenant_id": "<string>",
    "service_id": "<string>",
    "config": {
      "database.hostname.user.defined": "<string>",
      "database.port.user.defined": "5432",
      "database.user": "<string>",
      "database.password": "<string>",
      "database.dbname": "<string>",
      "signal.data.collection.schema.or.database": "<string>",
      "heartbeat.data.collection.schema.or.database": "<string>",
      "slot.name": "streamkap_pgoutput_slot",
      "publication.name": "streamkap_pub",
      "schema.include.list": "<string>",
      "table.include.list.user.defined": "<string>",
      "ssh.host": "<string>",
      "ssh.port": "22",
      "ssh.user": "streamkap",
      "ssh.public.key.user.displayed": "<SSH.PUBLIC.KEY>",
      "snapshot.read.only.user.defined": "Yes",
      "column.include.list.toggled": true,
      "column.include.list.user.defined": "<string>",
      "column.exclude.list.user.defined": "<string>",
      "heartbeat.enabled": true,
      "database.sslmode": "require",
      "include.source.db.name.in.table.name.user.defined": false,
      "binary.handling.mode": "bytes",
      "transforms.InsertStaticKey1.static.field": "<string>",
      "transforms.InsertStaticKey1.static.value": "<string>",
      "transforms.InsertStaticValue1.static.field": "<string>",
      "transforms.InsertStaticValue1.static.value": "<string>",
      "transforms.InsertStaticKey2.static.field": "<string>",
      "transforms.InsertStaticKey2.static.value": "<string>",
      "transforms.InsertStaticValue2.static.field": "<string>",
      "transforms.InsertStaticValue2.static.value": "<string>",
      "predicates.IsTopicToEnrich.pattern": "$^",
      "ssh.enabled": false
    },
    "topic_ids": [
      "<string>"
    ],
    "topic_map": {},
    "topics": [
      "<string>"
    ],
    "tasks": [
      123
    ],
    "connector_status": "<string>",
    "desired_state": "<string>",
    "kc_cluster_id": "<string>",
    "task_statuses": {},
    "tags": [
      "<string>"
    ]
  },
  "destination": {
    "name": "<string>",
    "connector": "<string>",
    "id": "<string>",
    "connector_display_name": "<string>",
    "created_timestamp": "<string>",
    "sub_id": "<string>",
    "tenant_id": "<string>",
    "config": {
      "azblob.connection.string.user.defined": "<string>",
      "azblob.container.name": "<string>",
      "format.user.defined": "json",
      "format.csv.write.headers": false,
      "topics.dir": "<string>",
      "file.name.template": "{{topic}}-{{partition}}-{{start_offset}}",
      "flush.size": 1000,
      "file.size": 65536,
      "rotate.interval.ms": -1,
      "compression": "<string>"
    },
    "topic_ids": [
      "<string>"
    ],
    "topic_map": {},
    "topics": [
      "<string>"
    ],
    "tasks": [
      123
    ],
    "connector_status": "<string>",
    "desired_state": "<string>",
    "kc_cluster_id": "<string>",
    "task_statuses": {},
    "tags": [
      "<string>"
    ]
  },
  "transforms": [
    "<unknown>"
  ],
  "tags": [],
  "periodic_audit": {
    "topics": [
      "<string>"
    ],
    "timestamp_column": "<string>",
    "interval_minutes": 123,
    "fix_deletes_only": false
  },
  "topic_auto_discovery_transforms": []
}

Authorizations

Authorization · string · header · required

Bearer authentication header of the form Bearer <token>, where <token> is your auth token.

Query Parameters

secret_returned · boolean · default: false

Whether to include secret values in the response

wait · boolean · default: true

When False, defer destination config update to reconciler instead of calling KC directly

Body

application/json

name · string · required

destination · PipelineDestinationConnector · object · required

Show child attributes

source · PipelineSourceConnector · object · required

Show child attributes

tags · string[] · required

snapshot_new_tables · boolean · default: false

transforms · PipelineTransformConnector · object[] | null

Show child attributes

topic_auto_discovery_transforms · TopicAutoDiscoveryTransform · object[]

Show child attributes

created_from · enum<string> | null · default: api

Origin of an entity in the system.

Tracks how entities (sources, destinations, pipelines) were created, enabling audit trails and origin-specific behavior.

Available options: terraform, web, api

Response

200

application/json

Successful Response

name · string | null

sub_id · string | null

tenant_id · string | null

id · string | null

snapshot_new_tables · boolean | null

topic_ids · (string | null)[] | null

topics · (string | null)[] | null

source · SourceConnector · object

Source connector configuration and status.

Show child attributes

destination · DestinationConnector · object

Destination connector configuration and status.

Show child attributes

transforms · any[] | null

tags · string[] | null

periodic_audit · UpdatePipelineAuditReq · object

Update pipeline audit request.

Show child attributes

topic_auto_discovery_transforms · TopicAutoDiscoveryTransform · object[]

Show child attributes