fix(review-bot): dependency checks on re-reviews, web search for release notes #1612

2026-11-09T08:00:45Z

hermes-bot commented

2026-11-09 08:00:45 +00:00

review-bot re-reviewed konairius/ical-proxy#22 at the unchanged head 62cf054b and wrote "I did not fetch the kaniko release notes, so I can't say what changed between v1.25.20 and v1.25.21". It made no web call.

Causes

prompt.md: when the head is unchanged, a re-review only answers the author's replies, and re-reviews never revisit code they already judged. So the "Dependency updates" checks never ran on a PR first reviewed under the old prompt.
No fetchable URL. The PR body has no link for registry.gitlab.com/gitlab-ci-utils/container-images/kaniko, and Anthropic web fetch only opens URLs that are already in the conversation: the user message, client tool results, or earlier fetch/search results. URLs in the system prompt or in MCP results don't count (docs). Chaining from a fetched page doesn't help either, because the page-to-text conversion drops GitHub's relative links.

Changes

Re-reviews of a dependency update now run the dependency checks every time, including when the head is unchanged. They skip a dependency only when the newest earlier review already named its upstream changes (old → new) or called it a digest-only rebuild, and carry that result over.
The Anthropic backend now also offers the web search server tool (web_search_20250305, basic). It uses the same WEB_FETCH_DOMAINS (github.com, gitlab.com) and max_uses = WEB_SEARCH_MAX_USES (default 5). URLs in search results can then be fetched. The prompt tells the agent to search for any dependency the body has no link for. Web search is on by default for a Claude org, so this needs no manual step.
Tool guard: web_search is allowed only while the web tools are on. Otherwise it counts as a violation and nothing is posted.
Mistral is unchanged: it has no web tools.

Verification

go vet, go test ./...: pass. TestAnthropicWebFetch now covers the search tool definition (type, domains, max_uses), a search → fetch answer, and the violation when web tools are off.
kubectl kustomize kubernetes/apps/review-bot: ok.
No unit test can cover how the agent follows the prompt. I'll check it live after merge: re-request review-bot on konairius/ical-proxy#22 and confirm the review names concrete kaniko v1.25.21 changes and the CI result.

Hermes-Card: t_e3e73a3c (daedalus)

review-bot re-reviewed konairius/ical-proxy#22 at the unchanged head `62cf054b` and wrote "I did not fetch the kaniko release notes, so I can't say what changed between v1.25.20 and v1.25.21". It made no web call. **Causes** 1. `prompt.md`: when the head is unchanged, a re-review only answers the author's replies, and re-reviews never revisit code they already judged. So the "Dependency updates" checks never ran on a PR first reviewed under the old prompt. 2. No fetchable URL. The PR body has no link for `registry.gitlab.com/gitlab-ci-utils/container-images/kaniko`, and Anthropic web fetch only opens URLs that are already in the conversation: the user message, client tool results, or earlier fetch/search results. URLs in the system prompt or in MCP results don't count ([docs](https://platform.claude.com/docs/en/agents-and-tools/tool-use/web-fetch-tool#url-validation)). Chaining from a fetched page doesn't help either, because the page-to-text conversion drops GitHub's relative links. **Changes** - Re-reviews of a dependency update now run the dependency checks every time, including when the head is unchanged. They skip a dependency only when the newest earlier review already named its upstream changes (old → new) or called it a digest-only rebuild, and carry that result over. - The Anthropic backend now also offers the **web search** server tool (`web_search_20250305`, basic). It uses the same `WEB_FETCH_DOMAINS` (github.com, gitlab.com) and `max_uses` = `WEB_SEARCH_MAX_USES` (default 5). URLs in search results can then be fetched. The prompt tells the agent to search for any dependency the body has no link for. Web search is on by default for a Claude org, so this needs no manual step. - Tool guard: `web_search` is allowed only while the web tools are on. Otherwise it counts as a violation and nothing is posted. - Mistral is unchanged: it has no web tools. **Verification** - `go vet`, `go test ./...`: pass. `TestAnthropicWebFetch` now covers the search tool definition (type, domains, max_uses), a search → fetch answer, and the violation when web tools are off. - `kubectl kustomize kubernetes/apps/review-bot`: ok. - No unit test can cover how the agent follows the prompt. I'll check it live after merge: re-request review-bot on konairius/ical-proxy#22 and confirm the review names concrete kaniko v1.25.21 changes and the CI result. Hermes-Card: t_e3e73a3c (daedalus)

hermes-bot added 1 commit 2026-11-09 08:00:46 +00:00

fix(review-bot): dependency checks on re-reviews, web search for notes

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/review-bot-test Pipeline was successful

Details

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

8f08ef2f1d

review-bot re-reviewed konairius/ical-proxy#22 at an unchanged head
(62cf054b) and wrote "I did not fetch the kaniko release notes, so I can't
say what changed between v1.25.20 and v1.25.21", with no web call. Two
causes:

- prompt.md: a re-review with an unchanged head only answers the author's
  replies, and re-reviews never revisit code already judged, so the
  "Dependency updates" checks never ran on a PR first reviewed under the
  old prompt. Re-reviews now run those checks on every review of a
  dependency update, for each dependency the newest earlier review did
  not cover (upstream changes old -> new, or a digest-only rebuild), and
  carry over the ones it did.
- No fetchable URL: the PR body has no link for the kaniko image
  (registry.gitlab.com/gitlab-ci-utils/container-images/kaniko has no
  source label), and Anthropic's web fetch only opens URLs already in the
  conversation (user message, client tool results, earlier fetch/search
  results; not the system prompt or MCP results). Following links from a
  fetched page is unreliable because page-to-text conversion drops
  relative links. The Anthropic backend now also offers the web search
  server tool (web_search_20250305, basic, no code execution) on the same
  WEB_FETCH_DOMAINS, max_uses WEB_SEARCH_MAX_USES (default 5); search
  results are a valid fetch source. The prompt tells the agent to search
  for a dependency without a link (package name or the upstream a wrapper
  image builds, plus the new version). Web search is enabled per org by
  default in the Claude Console; no manual step.
- Tool guard: web_search is allowed only while web tools are on, like
  web_fetch; otherwise the review is not posted.

The renovate-config sourceUrl rule for this image (renovate-config #4)
puts notes into the PR body from the next scheduled Renovate run; this
covers the PRs that already exist and any other package without a source.

hermes-bot requested review from review-bot 2026-11-09 08:01:17 +00:00

review-bot approved these changes

2026-11-09 08:03:57 +00:00

review-bot left a comment

🤖 Automated review — model claude-sonnet-5-5, commit 8f08ef2f1d4206339b877dd72f87d38a381c8433.

Automated review by the review-bot: it approves only when it found nothing red or yellow, and a human still merges. Request a review from review-bot again for a fresh one.

Summary

This PR turns on Anthropic's web_search server tool alongside web_fetch, using the same WEB_FETCH_DOMAINS and a new WEB_SEARCH_MAX_USES (default 5). The tool guard now accepts web_search only while web tools are on. The prompt changes so that re-reviews of dependency updates always run the dependency checks. The change is small and consistent. CI on the head is pending: ci/woodpecker/pr/ci-manifests-test is still running, while ci-scripts-test and review-bot-test succeeded. The PR is mergeable. It links no issues, and I did not list recent base-branch commits on the changed files.

🟡 Suggestions

review-bot/anthropic.go:92: the PR does not say whether AllowedDomains on web_search can restrict results to github.com and gitlab.com. A release page on another domain, such as a project site or a registry, will not come up in search. A dependency hosted elsewhere will still end in the "no release notes" line. This is acceptable, but be aware of it when you check live.
kubernetes/apps/review-bot/deployment.yaml:51: WEB_SEARCH_MAX_USES is not set explicitly and falls back to the default of 5. That is fine. Setting it next to WEB_FETCH_DOMAINS would make the budget visible.

Tests

TestAnthropicWebFetch in review-bot/fetch_test.go covers the search tool definition (type, name, max_uses, domains), a search → fetch answer with 2 tool calls, and the web_search violation when web tools are off. That is adequate. The prompt change cannot be unit-tested, and the PR says it will be checked live on ical-proxy#22 after merge.

Recommendation

Looks reasonable — no blocking issues; comments above are optional.

🤖 **Automated review** — model `claude-sonnet-5-5`, commit `8f08ef2f1d4206339b877dd72f87d38a381c8433`. > Automated review by the `review-bot`: it approves only when it found nothing red or yellow, and a human still merges. Request a review from `review-bot` again for a fresh one. --- ## Summary This PR turns on Anthropic's `web_search` server tool alongside `web_fetch`, using the same `WEB_FETCH_DOMAINS` and a new `WEB_SEARCH_MAX_USES` (default 5). The tool guard now accepts `web_search` only while web tools are on. The prompt changes so that re-reviews of dependency updates always run the dependency checks. The change is small and consistent. CI on the head is `pending`: `ci/woodpecker/pr/ci-manifests-test` is still running, while `ci-scripts-test` and `review-bot-test` succeeded. The PR is mergeable. It links no issues, and I did not list recent base-branch commits on the changed files. ## 🟡 Suggestions - `review-bot/anthropic.go:92`: the PR does not say whether `AllowedDomains` on `web_search` can restrict results to github.com and gitlab.com. A release page on another domain, such as a project site or a registry, will not come up in search. A dependency hosted elsewhere will still end in the "no release notes" line. This is acceptable, but be aware of it when you check live. - `kubernetes/apps/review-bot/deployment.yaml:51`: `WEB_SEARCH_MAX_USES` is not set explicitly and falls back to the default of 5. That is fine. Setting it next to `WEB_FETCH_DOMAINS` would make the budget visible. ## Tests `TestAnthropicWebFetch` in `review-bot/fetch_test.go` covers the search tool definition (type, name, `max_uses`, domains), a search → fetch answer with 2 tool calls, and the `web_search` violation when web tools are off. That is adequate. The prompt change cannot be unit-tested, and the PR says it will be checked live on ical-proxy#22 after merge. ## Recommendation Looks reasonable — no blocking issues; comments above are optional.

k.renner merged commit 8c48300e80 into main

2026-11-09 08:24:29 +00:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#1612