database: provide upsertUserPendingPermissionsBatchQuery insert values as array
Created by: bobheadxi
This PR uses this approach to get past the parameter limit, which we hit on external services with over ~15000 accounts. This tentatively is only used for inserts of >2500 rows, though I think the threshold can be lower / removed altogether based on crude benchmarks below.
Affects upsertUserPendingPermissionsBatchQuery
Closes https://github.com/sourcegraph/customer/issues/489 closes https://github.com/sourcegraph/sourcegraph/issues/24851. Companion to https://github.com/sourcegraph/sourcegraph/pull/24852, which fixes another query with paging.
Performance-wise wall clock results (however in all cases, the predicted cost
was lower regardless of time):
- 100: ~50% slower
- 5000: ~same
- 10000: ~13% faster
- 15000: previous query would hit parameter limit
Case | Accounts | Cost | Clock |
---|---|---|---|
Before | 100 | 0.00..1.75 |
287.071 ms |
After | 100 | 0.02..1.51 |
430.941 ms |
Before | 5000 | 0.00..87.50 |
7199.440 ms |
After | 5000 | 0.02..75.02 |
7218.860 ms |
Before | 10000 | 0.00..175.00 |
16858.613 ms |
After | 10000 | 0.02..150.01 |
14566.492 ms |
Before | 15000 | fail | fail |
After | 15000 | 0.02..225.01 |
22938.112 ms |
Before
100
Insert on user_pending_permissions (cost=0.00..1.75 rows=100 width=236) (actual time=275.927..286.053 rows=100 loops=1)
Conflict Resolution: UPDATE
Conflict Arbiter Indexes: user_pending_permissions_service_perm_object_unique
Tuples Inserted: 0
Conflicting Tuples: 100
-> Values Scan on "*VALUES*" (cost=0.00..1.75 rows=100 width=236) (actual time=0.377..1.712 rows=100 loops=1)
Planning Time: 2.189 ms
Execution Time: 287.071 ms
5000
Insert on user_pending_permissions (cost=0.00..87.50 rows=5000 width=236) (actual time=6734.847..7161.128 rows=5000 loops=1)
Conflict Resolution: UPDATE
Conflict Arbiter Indexes: user_pending_permissions_service_perm_object_unique
Tuples Inserted: 0
Conflicting Tuples: 5000
-> Values Scan on "*VALUES*" (cost=0.00..87.50 rows=5000 width=236) (actual time=0.370..59.790 rows=5000 loops=1)
Planning Time: 42.246 ms
Execution Time: 7199.440 ms
10000
Insert on user_pending_permissions (cost=0.00..175.00 rows=10000 width=236) (actual time=15779.462..16770.584 rows=10000 loops=1)
Conflict Resolution: UPDATE
Conflict Arbiter Indexes: user_pending_permissions_service_perm_object_unique
Tuples Inserted: 0
Conflicting Tuples: 10000
-> Values Scan on "*VALUES*" (cost=0.00..175.00 rows=10000 width=236) (actual time=0.125..137.567 rows=10000 loops=1)
Planning Time: 522.870 ms
Execution Time: 16858.613 ms
15000: fail
After
100
Insert on user_pending_permissions (cost=0.02..1.51 rows=100 width=236) (actual time=417.384..429.790 rows=100 loops=1)
Conflict Resolution: UPDATE
Conflict Arbiter Indexes: user_pending_permissions_service_perm_object_unique
Tuples Inserted: 0
Conflicting Tuples: 100
-> Function Scan on unnest (cost=0.02..1.51 rows=100 width=236) (actual time=0.381..1.635 rows=100 loops=1)
Planning Time: 2.239 ms
Execution Time: 430.941 ms
5000
Insert on user_pending_permissions (cost=0.02..75.02 rows=5000 width=236) (actual time=6779.778..7182.147 rows=5000 loops=1)
Conflict Resolution: UPDATE
Conflict Arbiter Indexes: user_pending_permissions_service_perm_object_unique
Tuples Inserted: 0
Conflicting Tuples: 5000
-> Function Scan on unnest (cost=0.02..75.02 rows=5000 width=236) (actual time=4.527..58.677 rows=5000 loops=1)
Planning Time: 0.341 ms
Execution Time: 7218.860 ms
10000
Insert on user_pending_permissions (cost=0.02..150.01 rows=10000 width=236) (actual time=13699.941..14493.004 rows=10000 loops=1)
Conflict Resolution: UPDATE
Conflict Arbiter Indexes: user_pending_permissions_service_perm_object_unique
Tuples Inserted: 0
Conflicting Tuples: 10000
-> Function Scan on unnest (cost=0.02..150.01 rows=10000 width=236) (actual time=8.338..114.721 rows=10000 loops=1)
Planning Time: 0.542 ms
Execution Time: 14566.492 ms
15000
Insert on user_pending_permissions (cost=0.02..225.01 rows=15000 width=236) (actual time=21501.538..22819.086 rows=15000 loops=1)
Conflict Resolution: UPDATE
Conflict Arbiter Indexes: user_pending_permissions_service_perm_object_unique
Tuples Inserted: 0
Conflicting Tuples: 15000
-> Function Scan on unnest (cost=0.02..225.01 rows=15000 width=236) (actual time=20.989..194.415 rows=15000 loops=1)
Planning Time: 1.213 ms
Execution Time: 22938.112 ms