Skip to main content

​
Overview

Use the HTTP Sink Destination to push streamed records from your topics to any reachable HTTP(S) endpoint. This is useful for integrating with ingestion APIs, webhooks, or custom services that accept POST requests.

​
Prerequisites

  • An HTTPS endpoint that accepts POST requests (e.g. ingestion API)
  • Decision on authentication: None, Static Authorization header (Bearer/API key, Basic Auth), or OAuth2 Client Credentials
  • Capacity planning for batch size vs endpoint rate limits

​
Endpoint Preparation

Before adding the Destination:
  1. Confirm the endpoint supports idempotency (recommended). If possible, design it to ignore duplicate records using a unique key.
  2. Record the base URL you will send data to. Example: https://api.example.com/ingest.
  3. If using a static token/key, create it and store securely.
  4. If using OAuth2 Client Credentials, collect:
    • Token URL
    • Client ID
    • Client Secret

​
Streamkap Setup (UI)

  1. Navigate to Destinations and choose HTTP Sink.
  2. Fill in the fields:
    1. Name – A memorable identifier.
    2. URL – The full HTTPS endpoint to POST each record or batch to.
    3. Authentication Type – Select None, Static, or OAuth2.
      • Static: Paste the Authorization header value. Do not include the word Authorization: – just the header value. Examples:
        • Bearer Token: Bearer eyJ...
        • API Key (standard format): ApiKey sk-1234567890abcdef
        • API Key (provider-specific): Check your API documentation—enter <key> directly or the format specified (e.g., Bearer, Token, etc.).
        • Basic Auth: Basic dXNlcjpwYXNzMTIz (base64-encoded username:password)
      • Custom Headers (non-Authorization): If your API uses custom headers like X-API-Key, use the Additional Headers field with format X-API-Key:sk-1234567890abcdef.
      • OAuth2: Enter Token URL, Client ID, Client Secret, optional Scope. Leave defaults for grant type unless your provider differs.
    4. Content-Type – Enter the media type expected by your endpoint (commonly application/json). Must be non-blank.
    5. Additional Headers – Comma separated Header:Value pairs (case-insensitive). Omit duplicates. Example: X-Env:prod,X-Source:streamkap.
    6. Batching – Enable if your endpoint supports multiple records per request.
      • Batch Max Size – Maximum number of records per batch (default 500). Start conservatively (50–200). Increase after observing latency.
      • Enable Batch Buffering – When enabled, records are held and only sent when either batch max size is reached OR batch max time expires. When disabled, records are sent immediately as they arrive (but still grouped up to batch max size).
      • Batch Max Time (ms) – Maximum time (in milliseconds) to hold records before flushing, even if batch size is not reached (default 1000 ms). Useful to ensure latency-sensitive endpoints receive data within a predictable window.
      • Separator – Typically \n for newline-delimited JSON. Leave default unless your endpoint requires another delimiter.
      • Prefix/Suffix – Optional text inserted before the first and after the last record. Leave blank unless wrapping in JSON array (e.g. Prefix [ Separator ,\n Suffix ]).
    7. Timeout (Seconds) – How long to wait for an HTTP response. Raise if your endpoint is slow; default works for most.
    8. Retries – Max attempts for transient HTTP failures. Increase only if backend occasionally returns 5xx.
    9. Backoff (ms) – Delay between retries. Adjust to respect rate limits.
    10. Decimal Format – Choose numeric if the endpoint expects plain numbers; otherwise keep base64.
  3. Save the Destination.
  4. Attach Pipelines / Topics to the Destination as needed.

​
How It Works

Each record (or batch) is serialized (JSON by default) and POSTed to the configured URL. Batching writes multiple serialized records separated by the chosen delimiter within one request. Retries apply to whole batches. Failures after max retries raise task errors.

​
Authentication Modes (UI Perspective)

  • None – No Authorization header sent.
  • Static – Streamkap sends Authorization: <value> exactly once per request. Supports multiple formats:
    • Bearer Tokens: Authorization: Bearer <token> for API tokens and JWTs.
    • API Keys: Depending on your provider, enter one of these formats:
      • Authorization: ApiKey <key> – Some APIs use this standardized format.
      • Authorization: <key> – Others accept the key directly without a prefix.
      • If your API uses a custom header like X-API-Key: <key>, use the Additional Headers field instead.
    • Basic Auth: Authorization: Basic <base64-credentials> for username/password authentication. You must manually convert username:password to Base64 before entering it. Online tools or command-line utilities (e.g., echo -n "user:pass" | base64) can help with encoding.
  • OAuth2 – Streamkap fetches and refreshes tokens automatically using Client Credentials flow. Tokens are inserted as Authorization: Bearer <token>.

​
Batching Guidance

  • Start small to measure endpoint latency (e.g. 100 records).
  • Increase gradually while monitoring HTTP 429 or 5xx responses.
  • Use newline separation for simple lines-based ingestion; use array wrapping (Prefix [, Separator ,\n, Suffix ]) if the endpoint expects a JSON array.

​
Batch Buffering & Flushing Behavior

When batching is enabled, you control when batches are sent using the Enable Batch Buffering setting:

​
Without Batch Buffering (Default)

  • Records are immediately POSTed in batches as they arrive, up to the configured Batch Max Size.
  • If fewer records arrive before the next batch, they are sent without waiting.
  • Use this mode for low-latency, record-by-record processing with minimal delay.

​
With Batch Buffering

  • Records are held in memory and only sent when either condition is met:
    1. Batch reaches max size – The batch hits the configured Batch Max Size limit.
    2. Time window expires – The Batch Max Time (ms) interval elapses since the first record arrived.
  • This mode ensures predictable latency and can improve endpoint efficiency by batching sparse traffic.

​
Configuration Dependencies

SettingRequiresEffect
Enable Batching–When enabled, all batch-related options become available.
Batch Max SizeBatching enabledMaximum records per request. Default 500; affects payload size.
Enable Batch BufferingBatching enabledWhen enabled, Batch Max Time becomes available.
Batch Max Time (ms)Batching + Buffering enabledDefault 10000 ms. Only active when buffering is on.
Batch Prefix/Suffix/SeparatorBatching enabledControl JSON or text formatting of the batch payload.

​
Example Scenarios

Scenario 1: Real-time ingestion, low latency
  • Enable Batching: ✓
  • Enable Batch Buffering: ✗
  • Batch Max Size: 100
  • Result: Records sent every ~100 records or immediately if fewer arrive, minimizing delay.
Scenario 2: Bulk ingestion, efficient batching
  • Enable Batching: ✓
  • Enable Batch Buffering: ✓
  • Batch Max Size: 1000
  • Batch Max Time: 5000 ms
  • Result: Records held up to 5 seconds or until 1000 records accumulated, maximizing throughput.
Scenario 3: Hybrid (responsive but efficient)
  • Enable Batching: ✓
  • Enable Batch Buffering: ✓
  • Batch Max Size: 500
  • Batch Max Time: 2000 ms
  • Result: Balances latency and efficiency; sends between 2–5 seconds depending on record volume.
The connector does not de-duplicate at HTTP layer. Implement idempotency server-side (e.g. reject duplicates based on a unique record key). Include that key in the payload your endpoint parses.

​
Security Notes

  • Secrets (client secret, static token) are stored encrypted.
  • Prefer HTTPS only; do not use plain HTTP for production.
  • Validate payload size limits on the receiving side to prevent oversized batch denial-of-service.

​
Limitations

  • No automatic schema negotiation with the endpoint – ensure the receiving service tolerates new fields.
  • Retries are whole-batch; a single problematic record causes re-send of the entire batch.
  • No built-in rate limiting; configure backoff and batch size to respect endpoint limits.