Heavy GraphQL API queries causing brief, definitive outages of deployment
Created by: slimsag
A user ran this GraphQL API query multiple times and it is suspected to have caused a brief outage on their instance https://app.hubspot.com/contacts/2762526/company/407948923/:
query {
search(query: "something count:9999") {
results {
__typename
limitHit
resultCount
approximateResultCount
indexUnavailable
results {
... on FileMatch {
lineMatches {
lineNumber
preview
limitHit
}
repository {
name
commit(rev:"HEAD") {
committer {
date
}
}
}
file {
path
}
}
}
}
}
}It seems that results.repository.commit in this query would resolve directly to this implementation code which would then ask gitserver to execute git rev-parse HEAD for the repository on every single search result that is returned. This is obviously far more work than needs to be done, as many results will come from the same repository.
It is suspected that the massive fan-out here from the frontend -> gitservers caused a temporary network outage on the machine, causing the frontend to be inaccessible. Their deployment is a single-beefy-machine, so more prone to this type of issue - but could happen on any deployment given only a slightly broader search query due to the fact that network traffic is from frontend.
Urgency / priority: unclear / normal.