Batch publishing

[wawanbreton]

Hi,
I have setup a benchmark using this action, on a repository that contains resource files. Each resource file is processed individually, and the result is published to a separate benchmark page. This does work very well, however now we have 200+ resource files that are processed, so the results publishing spawns 200+ jobs, that can't run in parallel because they do commit to the same repository. In the end, the publishing takes ~2h because spawning jobs and re-doing the whole publishing each time is not efficient.

I have seen this discussion: #69 which is solved by aggregating the data files, but in my case I don't want to end up with 1000 graphs on the same page 🙂
Is there a more efficient way to do this ?

Here is the workflow, simplified:

jobs:
  benchmark:
    name: Run benchmark
    outputs:
      benchmark_matrix: ${{ steps.prepare_publish_matrix.outputs.benchmark_matrix }}

    steps:
      - name: Run benchmark
        run: |
          mkdir analysis_results
          for file in assets/*; do
            analyzer "$file" --output "analysis_results/$file.json"
          done

      - name: Export benchmark results
        uses: actions/upload-artifact@v6
        with:
          name: analysis_results.zip
          path: analysis_results

      - name: Prepare benchmark publish matrix
        id: prepare_publish_matrix
        run: |
          python3 - <<'PY'
          import json
          import os
          from pathlib import Path

          analysis_results_dir = Path("analysis_results")
          benchmark_files = sorted(analysis_results_dir.glob("*.json"))
          matrix = {
              "include": [
                  {
                      "benchmark_name": benchmark_file.stem,
                      "result_file": str(benchmark_file),
                  }
                  for benchmark_file in benchmark_files
              ]
          }

          output_path = os.environ.get("GITHUB_OUTPUT")
          with open(Path(output_path), "a", encoding="utf-8") as output:
              output.write("benchmark_matrix<<EOF\n")
              output.write(json.dumps(matrix))
              output.write("\nEOF\n")
          PY

  publish-benchmark-results:
    name: Publish result (${{ matrix.benchmark_name }})
    needs: [benchmark]
    strategy:
      fail-fast: false
      max-parallel: 1
      matrix: ${{ fromJson(needs.benchmark.outputs.benchmark_matrix) }}

    steps:
      - name: Download benchmark results
        uses: actions/download-artifact@v7
        with:
          name: analysis_results.zip
          path: analysis_results

      - name: Store benchmark result
        uses: benchmark-action/github-action-benchmark@v1
        with:
          name: Benchmark ${{ matrix.benchmark_name }}
          output-file-path: ${{ matrix.result_file }}
          benchmark-data-dir-path: dev/multi_bench/${{ matrix.benchmark_name }}
          auto-push: true
          max-items-in-chart: 100

[ktrz]

If I understood your setup correctly (200+ assets, each published to its own page, jobs forced sequential because they all push to gh-pages), then this is annoying but very fixable, and you don't have to wait on a new action feature for it.

The slow part isn't the benchmarking. It's the pushing. Every job pushes to gh-pages, and Git won't let two pushes land on the same branch at the same time. It's the branch pointer that clashes, not your files (they're in separate folders and would merge fine). That's why max-parallel: 1 is the only setting that works, and why it drags.

The workaround is to let all 200 jobs run at once but have none of them push, then do a single commit and push in one job at the end.

For the parallel jobs, run your benchmark as usual but point the action at external-data-json-path with auto-push: false. In that mode it skips Git completely. It parses your output, appends the new data point, applies the max-items trimming and alert logic, and writes a plain JSON file. Nothing to contend over. To keep each page's history, seed that JSON from the live page first:

- run: curl -sfL "$PAGES_BASE/${{ matrix.asset }}/data.js" | sed 's/^window.BENCHMARK_DATA = //' > seed.json || echo '{}' > seed.json
- uses: benchmark-action/github-action-benchmark@v1
  with:
    external-data-json-path: seed.json
    save-data-file: true
    auto-push: false
    # name / tool / output-file-path: <your values here>
- uses: actions/upload-artifact@v4
  with: { name: bench-${{ matrix.asset }}, path: seed.json }

Then one aggregate job downloads all the artifacts and rebuilds the pages. A page is almost nothing: data.js is just that JSON with window.BENCHMARK_DATA = prepended, and index.html is a static file that's the same for every page (you can copy it from any page you've already published). So there's no action step here, just a loop and one push:

- run: |
    for d in incoming/bench-*/; do
      asset="$(basename "$d" | sed 's/^bench-//')"; mkdir -p "$asset"
      { printf 'window.BENCHMARK_DATA = '; cat "${d}seed.json"; } > "$asset/data.js"
      [ -f "$asset/index.html" ] || cp templates/index.html "$asset/index.html"
    done
    git add -A && git commit -m "batch publish" && git push origin gh-pages

Add a concurrency group around the workflow so two runs can't race that final push. The full workflow (with the matrix list, benchmark command, and PAGES_BASE filled in) is a bit long for a comment, so I left those as placeholders, but I can share it if useful.

You keep the things you care about here. Separate pages, history, and alerts all still work, since the action still runs in the parallel jobs. The 200 pushes collapse into one, the benchmarks stay parallel, and the total time drops to about your slowest single benchmark plus one publish pass.

One more thing worth raising separately: a batch-dir style input, where you hand the action a folder of results and it writes all the pages in one commit, would turn that aggregate loop into a single uses: step. Not needed to unblock this, but it's probably the real feature this issue is about.

[wawanbreton]

Thanks for the quick answer !
This solution should fit indeed, I didn't know it was possible to only update the files separately. Now I will have a lot of artifacts, but that doesn't really matter. I will try to implement that and keep you posted, but that should be fine.

One more thing worth raising separately: a batch-dir style input, where you hand the action a folder of results and it writes all the pages in one commit, would turn that aggregate loop into a single uses: step. Not needed to unblock this, but it's probably the real feature this issue is about.

That would be perfect for this case indeed 🙂 I could just have a single step to publish all the results at once, which would allow me to remove all the publishing jobs and matrix generation.

[wawanbreton]

So, I started implementing the proposed solution, but that forced me to add a lot of steps in my workflow, just to have this action work as expected. And in the end its only job would be to open the data.js file to append the proper bench result. So I made a very basic Python script instead, which does this and allows me to loop properly over the files.

In this very specific case, it also makes more sense for our benchmark, since it does run not on a specific repository, but on a combination of repositories that contain shared code/resources. So the data retriving about the commit was also mostly irrelevant.

For the record, here is the final structure of the workflow (simplified):

jobs:
  benchmark:
    name: Run benchmark
    steps:
      - name: Checkout main repository
        uses: actions/checkout@v7
        with:
          repository: ...

      - name: Checkout benchmarks repository
        uses: actions/checkout@v7
        with:
          repository: ...
          path: Benchmarks

      - name: Run benchmark
        run: |
          mkdir analysis_results

          for file in assets/*; do
            basename=$(basename "$file" .ext)
            analyzer "$file" --output "analysis_results/$basename.json"
            mkdir -p "Benchmarks/$basename"
            cp Benchmarks/template_index.html "Benchmarks/$basename/index.html"
            python update_benchmark_data.py --name "$basename" --data-file "Benchmarks/$basename/data.js" --new-analysis-file "analysis_results/$basename.json"
          done

          rm -rf analysis_assets

      - name: Commit the new benchmark results
        uses: stefanzweifel/git-auto-commit-action@v7
        with:
          repository: Benchmarks
          commit_message: Publish new benchmark results