Skip to content

Changeset Spec Exceeds Database Limit

Created by: Piszmog

The table changeset_specs has a column spec of type jsonb. PostgreSQL limits jsonb to 255 MB. If a changeset exceeds this limit, the following error occurs,

ERROR: string too long to represent as jsonb string (SQLSTATE 54000)

Possible Fixes

  • Prevent and changesets from exceeding the 255 MB limit by adding "guards"
    • If a changeset surpasses the limit, maybe point the user to docs to decrease changeset sizes (for example, transformchanges)
  • Rethink storing the spec in the jsonb column in the chageset_specs table.
    • This will probably be a larger effort and may need a follow up Issue for long term

Scenarios

Logging GraphQL requests impacts the behavior of the errors (simple 502 vs actual error message). But, the backing error is the same.

No Logging

Environment Variables

  • NO_GRAPHQL_LOG=true

Batch Spec

# Generates 676.3 MB changeset spec
name: large-changeset
description: Add Hello World to READMEs

on:
  - repositoriesMatchingQuery: file:README.md repo:^github.com/odoo/odoo$

steps:
  - run: find . -name "*.po" -exec sed -i 's/b/c/g' {} \;
    container: alpine:3

changesetTemplate:
  title: Hello World
  body: My first batch change!
  branch: hello-world-1 # Push the commit to this branch.
  commit:
    message: Append Hello World to all README.md files
  published: false # Do not publish any changes to the code hosts yet

Response

{
  "message": "inserter.Flush: ERROR: string too long to represent as jsonb string (SQLSTATE 54000)",
  "path": [
    "createChangesetSpec"
  ]
}

Logs

[       frontend] ERROR Inserter.database.batch.Flush batch/batch.go:224 operation.error {"batchSize": 14, "payloadSize": 24, "tableName": "changeset_specs", "columnNames": "rand_id,spec,batch_spec_id,repo_id,user_id,diff_stat_added,diff_stat_changed,diff_stat_deleted,created_at,updated_at,fork_namespace,external_id,head_ref,title", "numColumns": 14, "maxBatchSize": 32760, "count": 1, "elapsed": 4.235910458, "error": "ERROR: string too long to represent as jsonb string (SQLSTATE 54000)"}
[       frontend] ERROR enterprise.batches.dbstore.CreateChangesetSpec store/changeset_specs.go:128 operation.error {"Count": 1, "count": 1, "elapsed": 32.111408542, "error": "inserter.Flush: ERROR: string too long to represent as jsonb string (SQLSTATE 54000)"}
[       frontend] ERROR enterprise.batches.service.CreateChangesetSpec service/service.go:758 operation.error {"count": 1, "elapsed": 61.167864, "error": "inserter.Flush: ERROR: string too long to represent as jsonb string (SQLSTATE 54000)"}

With Logging

Environment Variables

N/A

Batch Spec

# Generates 1.0 GB changeset spec
name: large-changeset
description: Add Hello World to READMEs

on:
  - repositoriesMatchingQuery: file:README.md repo:^github.com/odoo/odoo$

steps:
  - run: find . -name "*.po" -exec sed -i 's/a/b/g' {} \;
    container: alpine:3

changesetTemplate:
  title: Hello World
  body: My first batch change!
  branch: hello-world-1 # Push the commit to this branch.
  commit:
    message: Append Hello World to all README.md files
  published: false # Do not publish any changes to the code hosts yet

Response

error: 502 Bad Gateway

Logs

[       frontend] ERROR Inserter.database.batch.Flush batch/batch.go:224 operation.error {"batchSize": 14, "payloadSize": 24, "tableName": "changeset_specs", "columnNames": "rand_id,spec,batch_spec_id,repo_id,user_id,diff_stat_added,diff_stat_changed,diff_stat_deleted,created_at,updated_at,fork_namespace,external_id,head_ref,title", "numColumns": 14, "maxBatchSize": 32760, "count": 1, "elapsed": 4.927489625, "error": "ERROR: string too long to represent as jsonb string (SQLSTATE 54000)"}
[       frontend] ERROR enterprise.batches.dbstore.CreateChangesetSpec store/changeset_specs.go:128 operation.error {"Count": 1, "count": 1, "elapsed": 32.148294167, "error": "inserter.Flush: ERROR: string too long to represent as jsonb string (SQLSTATE 54000)"}
[       frontend] ERROR enterprise.batches.service.CreateChangesetSpec service/service.go:758 operation.error {"count": 1, "elapsed": 63.868031875, "error": "inserter.Flush: ERROR: string too long to represent as jsonb string (SQLSTATE 54000)"}