security: harden which cross-origin headers are allowed

Created by: slimsag

Prior to this change, we would allow any origin (trusted or not) to send the secure X-Requested-With and (deprecated) X-Sourcegraph-Client headers because we would respond to an OPTIONS request with:

Access-Control-Allow-Headers: X-Requested-With, X-Sourcegraph-Client, Content-Type, Authorization, X-Sourcegraph-Should-Trace

However, because we only responded with Access-Control-Allow-Origin iff the origin is a trusted one, we did not ever suffer any negative consequences of doing this.

What does X-Requested-With do, anyway?

What the X-Requested-With (and deprecated X-Sourcegraph-Client) header indicates is that the request must have passed the browser's CORS OPTIONS preflight request, because setting a custom header in a request causes the browser to consider the request non-simple and thus ONLY sends it iff the CORS preflight passed (see https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS#simple_requests)

A level-headed person may ask: "But, when the GET/POST/etc request comes, couldn't we just check if the Origin header is a trusted origin? Why do we need a special header to be sent to us too?" and the answer is probably not what you want to hear: Browsers do not always send the Origin header. From https://fetch.spec.whatwg.org/#origin-header

The Origin header is a version of the Referer [sic] header that does not reveal a path. It is used for all HTTP fetches whose request’s response tainting is "cors", as well as those where request’s method is neither GET nor HEAD. Due to compatibility constraints it is not included in all fetches.

For example, a GET request to Sourcegraph's search streaming API endpoint would likely not have an Origin header sent along with the request. In this situation, we have no way of telling if the request came from a trusted origin or not, and so X-Requested-With being included can inform us that the request came from a trusted origin because for that header to be included, the request must have passed the CORS preflight.

Motivation for this change

Our Access-Control-Allow-Origin policy currently prevents any untrusted third-party origin from contacting our API. We want to change this in the future so that anyone can use our API (#23140 (closed)) but to prevent CSRF attacks session authentication must only be possible iff the origin of the request is trusted.

If we were to merely change our Access-Control-Allow-Origin policy today to allow any third party origin to contact our API, then we would be vulnerable to CSRF attacks because currently we do not indicate to clients making an OPTIONS request that only a trusted origin is allowed to send the secure X-Requested-With marker header.

After this change is merged, another change will alter our Access-Control-Allow-Origin policy to enable any untrusted third-party origin to make requests to our API and include authentication / session cookies with their request - but only trusted origins capable of sending X-Requested-With will have session authentication performed, hence this will be a lynchpin in protecting us against CSRF attacks in the future.

Helps #23140 (closed)

Signed-off-by: Stephen Gutekanst stephen@sourcegraph.com