Implement a URL unfurl endpoint to the Sourcegraph API

Created by: jeanduplessis

Requirements

Add an endpoint to the Sourcegraph API that takes a URL as an input and returns meta-information regarding the URL.

Implementation details

When the endpoint received the request it will:

1. Validate the input as a valid URL
2. Check if a result exists in the cache for the URL
   1. If one exists and it is not stale (older than X hrs [TBD]) then return the result from the cache
   2. If no result exists or it is stale make a new request to the URL
3. Make a request to the URL and parse the HTML for relevant meta tags
4. Store the result in the cache
5. Return the result

The following additional requirements should be considered:

• Follow redirects when making a request to the URL

Additional resources

Examples of URL unfurling services written in Go:

• https://github.com/heyvito/go-unfurl
• https://github.com/Doist/unfurlist

https://medium.com/slack-developer-blog/everything-you-ever-wanted-to-know-about-unfurling-but-were-afraid-to-ask-or-how-to-make-your-e64b4bb9254

Relates to https://github.com/sourcegraph/sourcegraph/issues/15584