Skip to content

database: provide upsertUserPendingPermissionsBatchQuery insert values as array

Warren Gifford requested to merge database/pending-perms-parameter-limit into main

Created by: bobheadxi

This PR uses this approach to get past the parameter limit, which we hit on external services with over ~15000 accounts. This tentatively is only used for inserts of >2500 rows, though I think the threshold can be lower / removed altogether based on crude benchmarks below.

Affects upsertUserPendingPermissionsBatchQuery

Closes https://github.com/sourcegraph/customer/issues/489 closes https://github.com/sourcegraph/sourcegraph/issues/24851. Companion to https://github.com/sourcegraph/sourcegraph/pull/24852, which fixes another query with paging.


Performance-wise wall clock results (however in all cases, the predicted cost was lower regardless of time):

  • 100: ~50% slower
  • 5000: ~same
  • 10000: ~13% faster
  • 15000: previous query would hit parameter limit
Case Accounts Cost Clock
Before 100 0.00..1.75 287.071 ms
After 100 0.02..1.51 430.941 ms
Before 5000 0.00..87.50 7199.440 ms
After 5000 0.02..75.02 7218.860 ms
Before 10000 0.00..175.00 16858.613 ms
After 10000 0.02..150.01 14566.492 ms
Before 15000 fail fail
After 15000 0.02..225.01 22938.112 ms

Before

100

Insert on user_pending_permissions  (cost=0.00..1.75 rows=100 width=236) (actual time=275.927..286.053 rows=100 loops=1)
  Conflict Resolution: UPDATE
  Conflict Arbiter Indexes: user_pending_permissions_service_perm_object_unique
  Tuples Inserted: 0
  Conflicting Tuples: 100
  ->  Values Scan on "*VALUES*"  (cost=0.00..1.75 rows=100 width=236) (actual time=0.377..1.712 rows=100 loops=1)
Planning Time: 2.189 ms
Execution Time: 287.071 ms

5000

Insert on user_pending_permissions  (cost=0.00..87.50 rows=5000 width=236) (actual time=6734.847..7161.128 rows=5000 loops=1)
  Conflict Resolution: UPDATE
  Conflict Arbiter Indexes: user_pending_permissions_service_perm_object_unique
  Tuples Inserted: 0
  Conflicting Tuples: 5000
  ->  Values Scan on "*VALUES*"  (cost=0.00..87.50 rows=5000 width=236) (actual time=0.370..59.790 rows=5000 loops=1)
Planning Time: 42.246 ms
Execution Time: 7199.440 ms

10000

Insert on user_pending_permissions  (cost=0.00..175.00 rows=10000 width=236) (actual time=15779.462..16770.584 rows=10000 loops=1)
  Conflict Resolution: UPDATE
  Conflict Arbiter Indexes: user_pending_permissions_service_perm_object_unique
  Tuples Inserted: 0
  Conflicting Tuples: 10000
  ->  Values Scan on "*VALUES*"  (cost=0.00..175.00 rows=10000 width=236) (actual time=0.125..137.567 rows=10000 loops=1)
Planning Time: 522.870 ms
Execution Time: 16858.613 ms

15000: fail

After

100

Insert on user_pending_permissions  (cost=0.02..1.51 rows=100 width=236) (actual time=417.384..429.790 rows=100 loops=1)
  Conflict Resolution: UPDATE
  Conflict Arbiter Indexes: user_pending_permissions_service_perm_object_unique
  Tuples Inserted: 0
  Conflicting Tuples: 100
  ->  Function Scan on unnest  (cost=0.02..1.51 rows=100 width=236) (actual time=0.381..1.635 rows=100 loops=1)
Planning Time: 2.239 ms
Execution Time: 430.941 ms

5000

Insert on user_pending_permissions  (cost=0.02..75.02 rows=5000 width=236) (actual time=6779.778..7182.147 rows=5000 loops=1)
  Conflict Resolution: UPDATE
  Conflict Arbiter Indexes: user_pending_permissions_service_perm_object_unique
  Tuples Inserted: 0
  Conflicting Tuples: 5000
  ->  Function Scan on unnest  (cost=0.02..75.02 rows=5000 width=236) (actual time=4.527..58.677 rows=5000 loops=1)
Planning Time: 0.341 ms
Execution Time: 7218.860 ms

10000

Insert on user_pending_permissions  (cost=0.02..150.01 rows=10000 width=236) (actual time=13699.941..14493.004 rows=10000 loops=1)
  Conflict Resolution: UPDATE
  Conflict Arbiter Indexes: user_pending_permissions_service_perm_object_unique
  Tuples Inserted: 0
  Conflicting Tuples: 10000
  ->  Function Scan on unnest  (cost=0.02..150.01 rows=10000 width=236) (actual time=8.338..114.721 rows=10000 loops=1)
Planning Time: 0.542 ms
Execution Time: 14566.492 ms

15000

Insert on user_pending_permissions  (cost=0.02..225.01 rows=15000 width=236) (actual time=21501.538..22819.086 rows=15000 loops=1)
  Conflict Resolution: UPDATE
  Conflict Arbiter Indexes: user_pending_permissions_service_perm_object_unique
  Tuples Inserted: 0
  Conflicting Tuples: 15000
  ->  Function Scan on unnest  (cost=0.02..225.01 rows=15000 width=236) (actual time=20.989..194.415 rows=15000 loops=1)
Planning Time: 1.213 ms
Execution Time: 22938.112 ms

Merge request reports

Loading