Data filter

The Data Filter panel lets you control exactly which data leaves the source during a Load workflow. Instead of writing SQL by hand, you configure column selection, row filters, sort order, and row limits through a visual interface.

Use this when:

• you want a subset of columns instead of the full table
• you need to exclude rows based on conditions (e.g. status, date range, null values)
• you want rows sorted in a specific order at the source
• you want to cap the number of rows transferred

For full SQL control (transformations, aliases, expressions), use Custom SQL Queries instead.

Opening the Data Filter

1. In the stream configuration wizard, reach Step 2: Structure and Data.
2. Select a table by clicking its checkbox.
3. Click the filter icon next to the table name.

The Data Filter panel opens on the right with four sections accessible from the header: Columns, Filter, Sort, and a Limit field. A Preview button validates your configuration against live data.

Column selection

Click Columns in the header to open the column selector.

• All / None buttons toggle the full set.
• Click individual column chips to include or exclude them.
• When a subset is selected, the Columns button shows a badge with the count.

If no columns are explicitly selected (or all are selected), the query uses SELECT *. Otherwise, only the selected columns are read from the source and written to the target.

Row filters

Click + Filter to add a WHERE condition. Each filter row has three parts:

Multiple filter rows are combined with AND. Only rows matching all conditions are transferred. OR logic is not supported — if you need OR conditions, use Custom SQL Queries instead.

Supported operators

Filter behavior

• A filter row is applied when it has a column and a value (except for IS NULL / IS NOT NULL).
• A filtered column does not need to be included in selectedColumns. Column selection controls the SELECT list; filters control the WHERE clause.
• String values containing single quotes are escaped automatically.
• IN / NOT IN values are trimmed and split by comma.

Sort order

Click Sort to add an ORDER BY clause.

• Choose a column from the dropdown.
• Toggle the direction button: blue arrow up for ASC, amber arrow down for DESC.
• Add multiple sort rows for multi-column ordering. The first row is the primary sort.
• Each column can appear in only one sort row.

Row limit

The Limit field in the header controls how many rows are read.

• Leave empty or set to 0 for no limit (all rows).
• Enter a positive number to cap the row count.
• Click the X button to clear a previously set limit.

Preview

Click Preview to run your filter configuration against the source and see matching rows.

• Preview returns up to 10 rows.
• A row count badge shows how many rows matched (with a + indicator if more exist).
• Only selected columns appear in the preview table.
• If the filter produces an error, a red error panel shows the message.
• If no rows match, an empty-state message is displayed.

Use Preview to validate your filters before starting the stream.

How filters translate to SQL

The Data Filter panel builds a structured TableSelectionState that the backend converts to SQL at execution time. The generated query follows this pattern:

SELECT [selected columns or *]
FROM tableName
WHERE [filter1 AND filter2 AND ...]
ORDER BY [sort1, sort2, ...]
LIMIT limit_value

Column and table names are quoted according to the source dialect (backticks for MySQL, double quotes for PostgreSQL).

API representation

In the stream configuration API, the selection state is stored as a structured object on each table entry:

{
  "name": "film",
  "selection": {
    "selectedColumns": ["film_id", "title", "release_year", "rating"],
    "filters": [
      { "column": "release_year", "operator": ">=", "value": "2000" },
      { "column": "rating", "operator": "IN", "value": "PG, PG-13" }
    ],
    "sorts": [
      { "column": "release_year", "direction": "DESC" }
    ],
    "limit": 500
  }
}

The saved payload uses API operator values such as NOT_CONTAINS, STARTS_WITH, and ENDS_WITH, even though the UI shows friendlier labels.

All fields are optional. Omitting selection entirely is equivalent to SELECT * FROM tableName.

Data filter vs custom SQL queries

Both approaches apply only to Load workflows.

Verification

After running a filtered stream, check that the stream reached finished and the transferred row count matches what Preview showed. If the count is unexpectedly low or high, review your filter conditions and re-run Preview.

Related docs

• Custom SQL Queries
• Schema Selection Guide
• Stream Configuration Guide
• Configuration Reference