Skip to content

Split large log lines up into smaller pieces

Warren Gifford requested to merge wbezuide/sg/split-large-logs-loki into main

Created by: burmudar

This addresses the following issue we were seeing on the pipeline

build 151159 job "0181205f-a064-4c8a-ae29-b4c8fc2acf48": unexpected status code 400: Max entry size '65536' bytes exceeded for stream '{app="buildkite", branch="main", build="151159", command="./tr dev/ci/yarn-web-integration.sh \"client/web/src/integration/batches.test.ts client/web/src/integration/blob-viewer.test.ts\"", component="build-logs", job="0181205f-a064-4c8a-ae29-b4c8fc2acf48", name=":puppeteer::electric_plug: Puppeteer tests chunk #1", queue="stateless,standard,default,job", retries_count="0", started_at="2022-06-01T17:51:40.783Z", state="failed", step_key="puppeteerelectricplugPuppeteertestschunk1", type="script"}' while adding an entry with length '563798' bytes]
  1. After a line has been added to the stream values, we check whether we should split the line up into smaller pieces.
  2. Each smaller piece has it's timestamp nanosecond value increased by 1 - this is safe todo, since the original Buildkite timestamps are in ms, so we on't override anything else (🙏🏼)
  3. We then replace the last value added with all the chunked pieces

Closes #36240 (closed)

Test plan

Unit Tests plus local loki instance

Local Loki instance

Launch Loki with the following docker-compose.yaml which sets the max_entry_line_size to 1kb

version: "3"

networks:
  loki:

services:
  loki:
    image: grafana/loki:2.5.0
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml -distributor.max-line-size=1kb
    networks:
      - loki

  promtail:
    image: grafana/promtail:2.5.0
    volumes:
      - /var/log:/var/log
    command: -config.file=/etc/promtail/config.yml
    networks:
      - loki

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    networks:
      - loki
  1. Launch it docker-compose up -d
  2. Download build logs and push them to the local loki instance using the change in this PR
$ go run . ci logs --build 151159 -out http://127.0.0.1:3100
Fetching logs for https://buildkite.com/sourcegraph/sourcegraph/builds/151159 ...
Pushing to Loki instance at "127.0.0.1:3100"
✅ Pushed 1110 entries from 1 streams to Loki

The error with current sg

ℹ️ Auto updating sg ...
✅ sg has been updated!
To see what's new, run 'sg version changelog'.
Fetching logs for https://buildkite.com/sourcegraph/sourcegraph/builds/151159 ...
Pushing to Loki instance at "127.0.0.1:3100"
❌ Failed to push 1 streams:
 - build 151159 job "0181205f-a064-4c8a-ae29-b4c8fc2acf48": unexpected status code 400: Max entry size '1024' bytes exceeded for stream '{app="buildkite", branch="", build="151159", command="./tr dev/ci/yarn-web-integration.sh \"client/web/src/integration/batches.test.ts client/web/src/integration/blob-viewer.test.ts\"", component="build-logs", exit_status="1", finished_at="2022-06-01T17:56:00.355Z", job="0181205f-a064-4c8a-ae29-b4c8fc2acf48", name=":puppeteer::electric_plug: Puppeteer tests chunk #1", queue="", retries_count="0", started_at="2022-06-01T17:51:40.783Z", state="failed", step_key="puppeteerelectricplugPuppeteertestschunk1", type="script"}' while adding an entry with length '1337' bytes
❌ failed to push all logs

Merge request reports