Articles in this section

Troubleshoot a Celigo Storage query job

Use this article when a submit request is rejected, a job fails or never finishes, a result looks wrong, or a status or results request returns an error. A rejection at submit returns a 4xx with an errors array, and no job exists. A failure after submit shows up as status: "failed" on the job with an error code and message. Each section below is named for what you see and lists the causes to check, in order. For the full code catalog, see Celigo Storage query API reference.

The submit request is rejected

A rejection happens before the query runs: the response is a 4xx, errors[0].code names the check that failed, and no job exists. Check the following causes in order.

  1. The statement isn't a single SELECT. STORAGE_QUERY_NOT_SELECT means the statement is outside the SELECT family, which includes WITH, DESCRIBE, and SUMMARIZE, or is a SHOW form that lists engine state. STORAGE_QUERY_MULTIPLE_STATEMENTS means the request carries more than one statement. Remove the second statement. A single trailing semicolon is fine.
  2. The SQL didn't parse, is empty, or is too long. STORAGE_QUERY_INVALID_SQL carries the parser's message. STORAGE_QUERY_SQL_REQUIRED means the sql field is missing or blank. STORAGE_QUERY_SQL_TOO_LARGE means the statement exceeds 16 KB. Shorten it or split it.
  3. A function isn't allowed. STORAGE_QUERY_FUNCTION_NOT_ALLOWED names the function. Functions that report engine or host state, such as version(), current_setting(), and getenv(), are never allowed. A misspelled function name produces the same error.
  4. A table function isn't allowed. STORAGE_QUERY_TABLE_FUNCTION_NOT_ALLOWED means the statement calls a table function other than the file readers (read_csv, read_json, read_ndjson, read_parquet, and their _auto variants), unnest, generate_series, or range. Reference a storage path instead. Functions such as glob, read_text, and query are never allowed.
  5. A row generator isn't bounded. STORAGE_QUERY_GENERATOR_ARGUMENT_NOT_LITERAL means an argument of generate_series or range isn't a whole-number literal — Celigo has to know the row count before the query runs. STORAGE_QUERY_GENERATOR_TOO_LARGE means the row counts of the statement's generate_series and range calls, multiplied together, exceed 1,000,000. This applies even when the calls are in separate UNION ALL branches. Lower a bound.
  6. The query reads no storage file. STORAGE_QUERY_NO_STORAGE_REFERENCE means the statement generates rows, through a generator, a recursive common table expression, or unnest of a literal list, without reading any file. Add a file reference. A statement of constant expressions alone, such as SELECT 1, is allowed.
  7. A reader call is malformed. STORAGE_QUERY_READER_PATH_NOT_LITERAL means the first argument of a reader isn't a literal path string or a literal list of them. STORAGE_QUERY_INVALID_READ_OPTION names an option the reader doesn't accept, a duplicate, a positional argument, or a value of the wrong type — filename, hive_partitioning, and encoding are never accepted. STORAGE_QUERY_READER_FORMAT_MISMATCH means the reader doesn't match the file's format, for example read_json on a .csv file.
  8. A path isn't a valid storage path. STORAGE_QUERY_INVALID_SOURCE names the FROM target. The usual causes are a wildcard (* or ? aren't valid path characters), a schema-qualified name such as information_schema.tables, a . or .. segment, or a character that Celigo Storage doesn't allow in names. Reference a folder with a trailing slash instead of a pattern.
  9. The query is too big. STORAGE_QUERY_TOO_MANY_SOURCES means the statement references more than 10 distinct paths. STORAGE_QUERY_INPUT_TOO_LARGE means the files it would read total more than 10 GB. Reference fewer paths, or a smaller folder.

A file or folder is not found

404 STORAGE_QUERY_SOURCE_NOT_FOUND names the path. Check the following causes in order.

  1. The path is wrong. Compare it with the path on the File storage page, including case. The leading slash is optional; everything else must match.
  2. The path is outside your token's scope. A path your Custom token can't reach returns the same not-found error as a path that doesn't exist. Check the token's File storage paths. See Access is denied.
  3. The file was deleted. A file in the recycle bin is not found. Restore it or reference another file.
  4. A parent folder in the path doesn't exist. The error names the whole reference, not the missing segment. Check each folder in the path.

Three related errors are about the shape of the reference rather than its existence.

  • 422 STORAGE_NOT_A_FILE — The path names a folder but has no trailing slash, so Celigo looked for a file. Add the trailing slash: '/staging/orders/'.
  • 422 STORAGE_ITEM_NOT_FOLDER — The path names a file but ends in a slash, so Celigo looked for a folder. Remove the trailing slash: '/staging/orders.csv'.
  • 422 STORAGE_ITEM_NOT_ACTIVE — The file's upload hasn't completed. Wait for the upload to finish, then submit again.

422 STORAGE_AMBIGUOUS_PATH means two folders with the same name exist at the same level of the path. Celigo Storage no longer allows this when a folder is created, so it affects only folders created before that check. Rename one of them.

Access is denied

Celigo reports a file your token can't read as not found, never as forbidden. Check the following causes in order.

  1. The token's File storage paths don't cover the reference. A Custom token reads only files at or under the folders in its File storage paths. The reference itself must be inside a granted folder — Celigo denies a reference to a parent of a granted folder even though some of its contents are in scope. Edit the token's paths, or reference the granted folder directly.
  2. The token is a Custom token with no File storage paths. Such a token can't reach Celigo Storage at all, and the submit request fails with 401 STORAGE_QUERY_ACCESS_REFUSED and the message You don't have the required permissions to perform the action. Add the folders to the token's File storage paths, or use a Full access token.
  3. The request carries an account-sharing header with a path-scoped token. 403 STORAGE_SCOPED_TOKEN_ASHARE_NOT_ALLOWED means the request sent an integrator-ashareid header with a Custom token that has File storage paths. Remove the header.
  4. A personal access token's role doesn't cover the file. A personal access token reads what its user can open. A file in an integration the user's role doesn't cover is reported as not found. Ask an account administrator for access to the integration, or use an API token.

The feature isn't enabled

403 STORAGE_QUERY_NOT_ENTITLED with the message Storage query is not enabled for this account means querying isn't enabled for the account that owns the token. Querying Celigo Storage is enabled per account on request. Every submit, status, and results request returns it, including requests for jobs submitted while the feature was enabled. Contact your Celigo account team to turn it on.

A folder mixes file formats

422 STORAGE_QUERY_MIXED_FORMAT_FOLDER names the folder, not the files. Every file directly inside a queried folder must share the same extension and MIME type, so Celigo rejects .csv beside .json, and .json beside .ndjson. It also rejects .csv beside .tsv, and two files with the same extension but different MIME types, for example a.csv recorded as text/csv beside b.csv recorded as application/octet-stream. Files uploaded through different tools can carry different MIME types for the same extension; the Celigo CLI records .ndjson, .jsonl, and .parquet files as application/octet-stream. Move each format into its own folder and combine them with UNION ALL, or reference the files individually. Related rejections:

  • 422 STORAGE_QUERY_UNSUPPORTED_FILE_TYPE — A file in the reference has an extension and MIME type that Celigo can't map to CSV, JSON, NDJSON, or Parquet. Rename the file with a supported extension, or move it out of the folder.
  • 422 STORAGE_QUERY_FOLDER_EMPTY — The folder holds no files with completed uploads. Files in subfolders don't count.
  • 422 STORAGE_QUERY_FOLDER_TOO_MANY_FILES — The folder holds more than 100 files. Split the files across folders and combine the folders in the statement.

A file is empty

Celigo doesn't reject a zero-byte file. A query over it completes with zero rows, and its columns show a single placeholder column, because that is what the engine infers from an empty file. A zero-byte file inside a folder contributes no rows to the folder's result. If you expected rows, check that the flow or transfer that writes the file has completed.

The job failed with a timeout

STORAGE_QUERY_TIMEOUT on the job means the query ran for more than 300 seconds. The timeout counts from startedAt, not from submission. Check the following causes in order.

  1. The query reads more data than it needs. Add a WHERE clause, reference a smaller folder, or select fewer columns. Total input is capped at 10 GB, and a query near the cap that also sorts or joins can take minutes.
  2. The query sorts or joins the whole dataset. A global ORDER BY over gigabytes, or a join whose smaller side is itself large, has to materialize the data. Filter first, or aggregate before you sort.
  3. The files are slow to parse. JSON with deep nesting parses more slowly than CSV or Parquet. If you query the dataset often, store it as Parquet.

The job failed during execution

STORAGE_QUERY_EXECUTION_FAILED carries the engine's message, which names the storage path. Check the following causes in order.

  1. A column's type drifted past the inference sample. The message reads like Could not convert string "text" to 'BIGINT'. The engine inferred the type from a sample of rows, and a later row, or a later file in the folder, didn't fit. Widen the sample with sample_size, declare the column's type with types for CSV or columns for JSON, or set ignore_errors = true for CSV and NDJSON. See Celigo storage query read options and format behavior.
  2. A cast failed. Replace CAST with TRY_CAST, which returns NULL instead of failing.
  3. A file was purged after submit. The file list is fixed at submit, so a file permanently deleted (purged from the recycle bin) before the job ran fails the read with an HTTP 404 message. Submit the query again. A file moved to the recycle bin is still read, and a replaced file is read with its new contents.
  4. A JSON file isn't in the format the reader expected. ignore_errors is accepted on read_json but only takes effect for newline-delimited JSON. On a JSON array file, the engine reports that parse errors can't be ignored. Fix the file, or convert it to NDJSON.
  5. A column in the header is reported as not found, or every line lands in one column. One malformed CSV row, such as a row with an extra field, can change what format detection decides for the whole file. The file may come back as a single column named a,b, or fail with Binder Error: Referenced column "b" not found. Read the file with read_csv and ignore_errors = true, or set delim, names, and types explicitly.

STORAGE_QUERY_WORKER_FAILED means the process running the query stopped unexpectedly. Submit the query again. If it recurs on the same statement, reduce the amount of data it reads and contact Celigo Support with the jobId.

STORAGE_QUERY_JOB_INTERRUPTED means Celigo's service stopped while it held the job, for example during a deployment. The statement itself isn't at fault. Submit it again.

The job stays running

A job normally reaches completed or failed within 300 seconds of startedAt, plus a few seconds. If a job is still running well past that, the service that held it stopped without recording a result. Celigo marks the job failed with STORAGE_QUERY_JOB_INTERRUPTED, normally within about 7 minutes. Submit the query again as a new job. If a job stays running for much longer than that, or this happens repeatedly, contact Celigo Support with the jobId.

Values look wrong in the result

  • Whole numbers are strings. "qty": "10" is expected. The engine infers whole-number columns as 64-bit integers, and 64-bit integers, unsigned 64-bit integers, 128-bit integers, and decimals arrive as strings. COUNT, SUM over integers, and // follow the same rule. AVG and / return numbers. To get a number, declare the column as INTEGER in the reader's types option, or cast the expression.
  • A division result is null. Dividing by zero produces infinity or not-a-number, which have no JSON representation, so Celigo emits null. To handle a zero denominator in the statement instead, guard it with CASE WHEN qty = 0 THEN NULL ELSE total / qty END.
  • A nested value is a string of JSON. A key whose values have different types across records is typed as JSON and arrives as JSON text, and its column is reported as VARCHAR. Nesting below a maximum_depth you set on the JSON reader behaves the same way; without maximum_depth, nesting is typed at any depth. Extract the fields you need with json_extract and cast them.
  • Rows of NULLs from an NDJSON file. With ignore_errors = true, a malformed NDJSON line comes back as a row of NULLs rather than being skipped. Filter it out, for example WHERE id IS NOT NULL.
  • Columns from one file are missing or NULL for another. Celigo combines the files in a folder by column name by default. A file that lacks a column produces NULL for it, and a renamed column appears as two half-filled columns. Declare names and types on the reader, set union_by_name = false to keep only the first file's columns, or fix the files.
  • Two runs of the same query disagree. random(), now(), and other nondeterministic functions produce a new value on each run.

A status or results request fails

  1. 404 STORAGE_QUERY_JOB_NOT_FOUND. The jobId is wrong, the job was submitted by a different identity, or the job finished more than about 24 hours ago and its record has expired. Submit the query again if you still need the result.
  2. 400 STORAGE_QUERY_INVALID_JOB_ID. The jobId isn't a valid identifier. Copy it exactly from the submit response.
  3. 409 STORAGE_QUERY_JOB_NOT_READY. The job isn't completed. Poll the status endpoint until it is. A failed job has no results.
  4. 400 STORAGE_QUERY_PAGE_OUT_OF_RANGE. page is above pageCount or isn't a positive integer. Page numbers start at 1. A job with no rows has pageCount: 0, and only page=1 is valid for it.

The service is throttled or unavailable

  • 429 STORAGE_QUERY_RESOLVER_RATE_LIMITED — Too many requests reached Celigo Storage while it looked up your paths. Wait and submit again with a longer interval between requests. Celigo never rejects a submission because your account already has several jobs running — that job waits in the queue.
  • 503 STORAGE_QUERY_RESOLVER_UNAVAILABLE — Celigo Storage couldn't look up your paths. Retry shortly.
  • 503 STORAGE_QUERY_ENQUEUE_FAILED — The query passed every check but couldn't be queued. No job exists. Retry shortly.
  • 500 STORAGE_QUERY_REWRITE_VERIFICATION_FAILED or STORAGE_QUERY_RESOLVER_REJECTED_REQUEST — An internal check failed. Retry once. If it recurs, contact Celigo Support with the statement and the time of the request.

Learn more