<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="4.4.1">Jekyll</generator><link href="https://gwenneg.github.io/feed.xml" rel="self" type="application/atom+xml" /><link href="https://gwenneg.github.io/" rel="alternate" type="text/html" /><updated>2026-06-18T14:07:35+00:00</updated><id>https://gwenneg.github.io/feed.xml</id><title type="html">Gwenneg’s blog</title><subtitle>Gwenneg&apos;s blog</subtitle><author><name>Gwenneg Lepage</name></author><entry><title type="html">Your Claude bill called. It wants to talk.</title><link href="https://gwenneg.github.io/2026/06/18/your-claude-bill-called.html" rel="alternate" type="text/html" title="Your Claude bill called. It wants to talk." /><published>2026-06-18T00:00:00+00:00</published><updated>2026-06-18T00:00:00+00:00</updated><id>https://gwenneg.github.io/2026/06/18/your-claude-bill-called</id><content type="html" xml:base="https://gwenneg.github.io/2026/06/18/your-claude-bill-called.html"><![CDATA[<div id="preamble">
<div class="sectionbody">
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
<div class="paragraph">
<p>This post was written with the help of AI.</p>
</div>
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>AI-assisted coding is great. Until you check the bill.</p>
</div>
<div class="paragraph">
<p>The <a href="https://code.claude.com/docs/en/costs" target="_blank" rel="noopener">Claude Code docs</a> cover how to reduce costs.
This post ranks cost reduction practices by impact and tells you which ones to do first.
Most developers know a few.
Fewer apply them consistently.
The ranking weighs three things: how much a practice saves per session, how universally it applies, and how easy it is to start.
The first seven work for everyone.
The last three depend on your setup.</p>
</div>
<div class="paragraph">
<p>Each of the 10 practices below is worth 2 points: 1 for knowing it, 1 for applying it.
Keep score as you read.</p>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
<div class="paragraph">
<p>This post is focused on Claude.
Some of it may apply to other leading AI models.
Local models can also be a valid option to lower costs, but out of scope here.</p>
</div>
</td>
</tr>
</table>
</div>
</div>
</div>
<div class="sect1">
<h2 id="whats-the-single-biggest-lever-to-reduce-costs">1. What&#8217;s the single biggest lever to reduce costs?</h2>
<div class="sectionbody">
<div class="paragraph">
<p><strong>Model selection.</strong> First point: free. Second point: when did you last reach for Haiku?</p>
</div>
<div class="paragraph">
<p>Pick the right model for your task:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><strong>Haiku</strong>: simple and fast operations. Formatting, summarization, quick lookups. No heavy reasoning needed.</p>
</li>
<li>
<p><strong>Sonnet</strong>: most coding tasks. The sweet spot between capability and cost.</p>
</li>
<li>
<p><strong>Opus</strong>: complex reasoning, architectural decisions, tricky bugs.
Reach for it when the task genuinely needs it.</p>
</li>
<li>
<p><strong>Fable</strong>: top of the range, currently unavailable.
Most capable, most expensive. Use it when nothing else will do, or when it&#8217;s not your money.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>In interactive sessions, use <code>/model</code> to switch. In skills and agents, set it in frontmatter.</p>
</div>
<div class="listingblock">
<div class="title">Skill or agent frontmatter</div>
<div class="content">
<pre class="rouge highlight"><code data-lang="yaml"><span class="nn">---</span>
<span class="na">model</span><span class="pi">:</span> <span class="s">sonnet</span>
<span class="nn">---</span></code></pre>
</div>
</div>
<div class="paragraph">
<p><em>Dig deeper:</em> <a href="https://platform.claude.com/docs/en/about-claude/pricing" target="_blank" rel="noopener">Models pricing</a>, <a href="https://code.claude.com/docs/en/model-config" target="_blank" rel="noopener">Model configuration</a>, <a href="https://code.claude.com/docs/en/skills#frontmatter-reference" target="_blank" rel="noopener">Skill frontmatter</a>, <a href="https://code.claude.com/docs/en/sub-agents#supported-frontmatter-fields" target="_blank" rel="noopener">Sub-agent frontmatter</a></p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="your-agent-is-thinking-hard-on-every-task-is-that-a-problem">2. Your agent is thinking hard on every task. Is that a problem?</h2>
<div class="sectionbody">
<div class="paragraph">
<p><strong>Yes. Claude is writing a dissertation about your variable renaming.</strong></p>
</div>
<div class="paragraph">
<p>Thinking tokens count as output tokens.
The priciest kind.
A single request can burn tens of thousands of them.
For a complex architectural decision, worth every token.
For "add a missing semicolon," not so much.</p>
</div>
<div class="paragraph">
<p>In interactive sessions, use <code>/effort low</code> or <code>/effort medium</code> for straightforward tasks, and <code>/effort high</code> when it actually matters. In skills and agents, set it in frontmatter so nobody has to remember.</p>
</div>
<div class="listingblock">
<div class="title">Skill or agent frontmatter</div>
<div class="content">
<pre class="rouge highlight"><code data-lang="yaml"><span class="nn">---</span>
<span class="na">effort</span><span class="pi">:</span> <span class="s">low</span>
<span class="nn">---</span></code></pre>
</div>
</div>
<div class="paragraph">
<p><em>Dig deeper:</em> <a href="https://platform.claude.com/docs/en/about-claude/pricing" target="_blank" rel="noopener">Models pricing</a>, <a href="https://code.claude.com/docs/en/model-config#adjust-effort-level" target="_blank" rel="noopener">Effort configuration</a>, <a href="https://code.claude.com/docs/en/skills#frontmatter-reference" target="_blank" rel="noopener">Skill frontmatter</a>, <a href="https://code.claude.com/docs/en/sub-agents#supported-frontmatter-fields" target="_blank" rel="noopener">Sub-agent frontmatter</a></p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="what-does-improve-the-codebase-actually-cost">3. What does 'improve the codebase' actually cost?</h2>
<div class="sectionbody">
<div class="paragraph">
<p><strong>A lot. The fix is simpler than you think: be specific.</strong></p>
</div>
<div class="paragraph">
<p>Claude starts by figuring out what your codebase even is.
That means reading files until it has enough context to act.
And every file it reads stays in context, getting re-sent with every follow-up message.
You&#8217;re paying for that exploration whether you asked for it or not.</p>
</div>
<div class="paragraph">
<p>Specificity isn&#8217;t just for interactive sessions.
In skills and agents, every word in the spawn prompt loads on top of everything the agent auto-loads, from turn one.
Vague instructions are just as expensive there.</p>
</div>
<div class="paragraph">
<p>Free points. You&#8217;re welcome.</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="your-claude-md-documents-your-entire-architecture-smart-or-not">4. Your CLAUDE.md documents your entire architecture. Smart or not?</h2>
<div class="sectionbody">
<div class="paragraph">
<p><strong>Impressive documentation. Expensive documentation.</strong></p>
</div>
<div class="paragraph">
<p>Everything in CLAUDE.md loads at session start and gets re-sent on every single request, whether it&#8217;s relevant or not.
That beautifully written section explaining the history of your repository pattern?
Claude reads it before fixing a typo.
Every.
Single.
Time.</p>
</div>
<div class="ulist">
<ul>
<li>
<p>Keep CLAUDE.md under 200 lines. Essentials only.</p>
</li>
<li>
<p>Move path-specific guidelines into rules. They load only when Claude works on matching files, not on every turn.</p>
</li>
<li>
<p>Move workflow instructions (PR reviews, deploy steps, migration guides) into skills.
They load on-demand, not on every turn.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p><em>Dig deeper:</em> <a href="https://code.claude.com/docs/en/memory#path-specific-rules" target="_blank" rel="noopener">Path-specific rules</a>, <a href="https://code.claude.com/docs/en/skills" target="_blank" rel="noopener">Extend Claude with skills</a></p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="you-just-finished-a-task-what-should-you-do-before-starting-the-next-one">5. You just finished a task. What should you do before starting the next one?</h2>
<div class="sectionbody">
<div class="paragraph">
<p><strong><code>/clear</code>. Seriously.</strong></p>
</div>
<div class="paragraph">
<p>Every request re-sends the full conversation history as input tokens.
Once a task is done, that history is dead weight: old file reads, debugging logs, that long exchange where Claude went down the wrong path.
You&#8217;re carrying a suitcase full of things you don&#8217;t need anymore.</p>
</div>
<div class="paragraph">
<p><code>/clear</code> drops it.</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="youre-mid-task-and-the-context-is-getting-heavy-what-do-you-do">6. You&#8217;re mid-task and the context is getting heavy. What do you do?</h2>
<div class="sectionbody">
<div class="paragraph">
<p><strong><code>/compact</code>, but with instructions.</strong></p>
</div>
<div class="paragraph">
<p>Just running <code>/compact</code> summarizes the conversation, but Claude decides what to keep.
You can do better: <code>/compact Focus on modified files and unresolved errors</code> tells Claude exactly what matters.
You can also make it permanent in CLAUDE.md so it applies every time Claude compacts automatically.</p>
</div>
<div class="listingblock">
<div class="title">CLAUDE.md compaction instructions</div>
<div class="content">
<pre class="rouge highlight"><code data-lang="markdown">== Compact instructions

When compacting, focus on modified files and unresolved errors.</code></pre>
</div>
</div>
<div class="paragraph">
<p><em>Dig deeper:</em> <a href="https://code.claude.com/docs/en/context-window#when-your-context-fills-up" target="_blank" rel="noopener">Compact with a focus</a></p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="your-agent-is-running-tests-and-reading-logs-where-does-all-that-output-go">7. Your agent is running tests and reading logs. Where does all that output go?</h2>
<div class="sectionbody">
<div class="paragraph">
<p><strong>In the main context, if you&#8217;re not careful.</strong></p>
</div>
<div class="paragraph">
<p>Test runners, log processors, and doc fetchers can generate a lot of output.
If that output lands in the main session, Claude re-sends it on every subsequent turn.
Subagents run in their own isolated context window.
The parent session gets a summary, not the raw output.</p>
</div>
<div class="paragraph">
<p>In skills and agents, structure workflows to delegate verbose operations to subagents.
The main agent stays lean.
In interactive sessions, Claude will usually make that call on its own, but you can always ask explicitly.</p>
</div>
<div class="paragraph">
<p><em>Dig deeper:</em> <a href="https://code.claude.com/docs/en/sub-agents#isolate-high-volume-operations" target="_blank" rel="noopener">Isolate high-volume operations</a></p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="scripts-and-hooks-helping-claude-process-less">8. Scripts and hooks helping Claude process less</h2>
<div class="sectionbody">
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
<div class="paragraph">
<p>This one is situational: the savings depend on how much output you&#8217;re actually cutting.</p>
</div>
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>Pre-written scripts and hooks let you preprocess data before Claude sees it.
Feeding Claude a raw test report when all you needed was the failures means Claude processes a lot more than necessary.</p>
</div>
<div class="paragraph">
<p>A 2026 study filtered low-value output from agent trajectories and found 40-60% fewer input tokens and 21-36% lower costs, with no impact on task success. Scripts and hooks are a simpler way to act on the same idea.
The pattern scales well for agent-heavy workflows, but for small tasks the overhead isn&#8217;t worth it.</p>
</div>
<div class="paragraph">
<p><em>Dig deeper:</em> <a href="https://arxiv.org/html/2509.23586v2" target="_blank" rel="noopener">AgentDiet, FSE 2026</a></p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="claude-is-about-to-rewrite-half-your-codebase-has-it-seen-a-plan-first">9. Claude is about to rewrite half your codebase. Has it seen a plan first?</h2>
<div class="sectionbody">
<div class="paragraph">
<p><strong>It should have.</strong></p>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
<div class="paragraph">
<p>This one is situational: it only pays off on complex or ambiguous tasks.</p>
</div>
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>Skip the plan and Claude might build the wrong thing entirely.
You pay for the wrong implementation, then pay to fix it.
Plan mode makes Claude explore and propose an approach before touching any code.
You review it, adjust it, then let it implement.</p>
</div>
<div class="paragraph">
<p>In interactive sessions, press Shift+Tab before giving Claude a large or ambiguous task.
In skills, bake it in: structure the definition to include an explicit planning step before execution.</p>
</div>
<div class="paragraph">
<p><em>Dig deeper:</em> <a href="https://code.claude.com/docs/en/permission-modes#analyze-before-you-edit-with-plan-mode" target="_blank" rel="noopener">Plan mode</a></p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="did-you-actually-need-that-1m-context-window">10. Did you actually need that 1M context window?</h2>
<div class="sectionbody">
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
<div class="paragraph">
<p>This one is situational.</p>
</div>
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p><strong>Surprise: it&#8217;s not free.</strong></p>
</div>
<div class="paragraph">
<p>The 1M context window costs the same per token as the default.
It just holds 5x more tokens, so sessions stay uncompacted much longer and every turn carries a heavier payload.
Stick with 200K unless you actually need it.</p>
</div>
<div class="paragraph">
<p><em>Dig deeper:</em> <a href="https://code.claude.com/docs/en/model-config#extended-context" target="_blank" rel="noopener">Extended context</a></p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="how-do-you-know-if-its-working">How do you know if it&#8217;s working?</h2>
<div class="sectionbody">
<div class="paragraph">
<p>You can&#8217;t improve what you can&#8217;t measure.</p>
</div>
<div class="paragraph">
<p>Run <code>/usage</code> in Claude Code to see token usage and cost for the current session.
Watch those numbers across sessions as you apply these practices.
The difference is visible.</p>
</div>
<div class="paragraph">
<p>Running the API or tracking team usage?
The Claude Console breaks it down by model, date, and API key.</p>
</div>
<div class="paragraph">
<p><em>Dig deeper:</em> <a href="https://code.claude.com/docs/en/costs#using-the-/usage-command" target="_blank" rel="noopener">Using the /usage command</a>, <a href="https://platform.claude.com/dashboard" target="_blank" rel="noopener">Claude Console</a></p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="how-many-points-did-you-get">How many points did you get?</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Scored 15 or above?
Genuinely impressive.</p>
</div>
<div class="paragraph">
<p>Below that?
You&#8217;re one or two practices away from making a real difference.
Start with model selection, and go from there.</p>
</div>
<div class="paragraph">
<p>Thanks for reading!</p>
</div>
</div>
</div>]]></content><author><name>Gwenneg Lepage</name></author><category term="ai" /><category term="claude code" /><category term="costs" /><category term="developer experience" /><category term="skills" /><category term="agents" /><summary type="html"><![CDATA[10 practices to cut your Claude bill, ranked by impact. Score yourself as you read.]]></summary></entry><entry><title type="html">Turn AI friction into better docs</title><link href="https://gwenneg.github.io/2026/05/31/turn-ai-friction-into-better-docs.html" rel="alternate" type="text/html" title="Turn AI friction into better docs" /><published>2026-05-31T00:00:00+00:00</published><updated>2026-05-31T00:00:00+00:00</updated><id>https://gwenneg.github.io/2026/05/31/turn-ai-friction-into-better-docs</id><content type="html" xml:base="https://gwenneg.github.io/2026/05/31/turn-ai-friction-into-better-docs.html"><![CDATA[<div id="preamble">
<div class="sectionbody">
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
<div class="paragraph">
<p>This post was written with the help of AI.</p>
</div>
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>In a <a href="/2026/05/28/the-docs-your-ai-agent-is-missing" target="_blank" rel="noopener">previous post</a>, I introduced a Claude skill that generates layered context docs for AI-assisted development: domain-specific guidelines, a cross-cutting <code>AGENTS.md</code>, path-scoped Claude rules, and a thin <code>CLAUDE.md</code> on top.</p>
</div>
<div class="paragraph">
<p>These files have two problems: they go stale as the codebase evolves, and the initial version is never perfect.
Some conventions only surface when the agent tries to follow the docs and stumbles: an unnecessary question, a wrong assumption, output that needs correcting.
Each of these stumbles is a friction event: a signal that the docs failed.</p>
</div>
<div class="paragraph">
<p>I wanted to automate as much of this as possible: capture friction, turn it into doc improvements, and keep developer effort to a minimum.
What I ended up with is a feedback loop built on git, Claude Code hooks, and shell scripts, no extra platform or infrastructure required.
Let me walk you through it.</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="the-feedback-loop">The feedback loop</h2>
<div class="sectionbody">
<div class="paragraph">
<p>The loop has two phases:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>Automated friction capture at the end of every Claude Code session</p>
</li>
<li>
<p>A manual step to turn accumulated friction events into context doc improvements</p>
</li>
</ul>
</div>
<div class="paragraph">
<p><span class="image"><img src="/assets/images/posts/turn-ai-friction-into-better-docs/feedback-loop.svg" alt="Feedback loop" width="85%"></span></p>
</div>
<div class="sect2">
<h3 id="capturing-friction-automatically">Capturing friction automatically</h3>
<div class="paragraph">
<p>I wired up a <a href="https://code.claude.com/docs/en/hooks-guide#how-hooks-work" target="_blank" rel="noopener">SessionEnd hook</a> that fires at the end of every Claude Code session, or when the <a href="https://code.claude.com/docs/en/commands#all-commands" target="_blank" rel="noopener">/clear command</a> is run:</p>
</div>
<div class="listingblock">
<div class="title">.claude/settings.json</div>
<div class="content">
<pre class="rouge highlight"><code data-lang="json"><span class="p">{</span><span class="w">
  </span><span class="nl">"hooks"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
    </span><span class="nl">"SessionEnd"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
      </span><span class="p">{</span><span class="w">
        </span><span class="nl">"matcher"</span><span class="p">:</span><span class="w"> </span><span class="s2">""</span><span class="p">,</span><span class="w">
        </span><span class="nl">"hooks"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
          </span><span class="p">{</span><span class="w">
            </span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"command"</span><span class="p">,</span><span class="w">
            </span><span class="nl">"command"</span><span class="p">:</span><span class="w"> </span><span class="s2">"bash .claude/scripts/friction-capture.sh"</span><span class="p">,</span><span class="w">
            </span><span class="nl">"timeout"</span><span class="p">:</span><span class="w"> </span><span class="mi">5000</span><span class="w">
          </span><span class="p">}</span><span class="w">
        </span><span class="p">]</span><span class="w">
      </span><span class="p">}</span><span class="w">
    </span><span class="p">]</span><span class="w">
  </span><span class="p">}</span><span class="w">
</span><span class="p">}</span></code></pre>
</div>
</div>
<div class="paragraph">
<p>When that happens, the <a href="https://github.com/gwenneg/blog-ai-friction-loop/blob/main/skills/setup-friction-capture/scripts/friction-capture.sh" target="_blank" rel="noopener">friction-capture.sh</a> script runs in a background subshell, so it doesn&#8217;t block the user.
The script receives the transcript from the ended session and sends the last 200KB to Claude Haiku with a prompt asking it to identify friction events.
A <code>flock</code>-based lock inside the background block prevents concurrent runs.
Haiku is cheap enough that the cost per session stays under a few cents.</p>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
<div class="paragraph">
<p><code>flock</code> is not available on macOS. The script currently requires Linux or a <code>flock</code> package like the one from Homebrew.</p>
</div>
</td>
</tr>
</table>
</div>
<div class="admonitionblock warning">
<table>
<tr>
<td class="icon">
<i class="fa icon-warning" title="Warning"></i>
</td>
<td class="content">
<div class="paragraph">
<p>The conversation between the user and Claude is sent to the Anthropic API at the end of each session, using the developer&#8217;s existing Claude Code credentials. Tool call outputs (file contents, command output) are not included, but anything discussed in the conversational text is. Injected content in the transcript could also produce friction events targeting arbitrary files. Review the resulting PR diff carefully, and exclude any suspicious events when the skill presents them. Friction capture is enabled by default after the setup PR merges. Developers can opt out with <a href="https://github.com/gwenneg/blog-ai-friction-loop/blob/main/skills/disable-friction-capture/SKILL.md" target="_blank" rel="noopener"><code>/disable-friction-capture</code></a>.</p>
</div>
</td>
</tr>
</table>
</div>
<div class="sect3">
<h4 id="what-counts-as-friction">What counts as friction</h4>
<div class="paragraph">
<p>The prompt tells Haiku to scan the transcript for four types of events:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><strong>Corrections</strong>: the user corrected the agent&#8217;s output</p>
</li>
<li>
<p><strong>Clarifications</strong>: the agent asked a question the docs should have answered</p>
</li>
<li>
<p><strong>Mistakes</strong>: the agent made a wrong assumption about the codebase</p>
</li>
<li>
<p><strong>Denials</strong>: a tool call was denied, revealing a standing project policy (e.g., "don&#8217;t skip the OWASP dependency check")</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Not everything qualifies. The prompt applies two filters: would a doc rule have prevented it, and is it likely to happen again? If either answer is no, the event is dropped. For example, user errors, one-off denials, scope changes, transient failures, and case-specific corrections are excluded.
For example, "all REST endpoints use kebab-case" is a doc gap, but "no, I meant the other endpoint" is not.</p>
</div>
<div class="paragraph">
<p>Each event must also name a specific target file (e.g., <code>docs/api-guidelines.md</code>) and state the missing rule in a few sentences max.
Otherwise it is excluded.</p>
</div>
</div>
<div class="sect3">
<h4 id="friction-event-format">Friction event format</h4>
<div class="paragraph">
<p>For each event, the script writes a markdown file to <code>.claude/friction/{session-id}/</code>.
These files accumulate silently across sessions.
Sessions with zero friction produce no files.</p>
</div>
<div class="listingblock">
<div class="title">.claude/friction/{session-id}/&lt;event-name&gt;.md</div>
<div class="content">
<pre class="rouge highlight"><code data-lang="markdown"><span class="nn">---</span>
<span class="na">type</span><span class="pi">:</span> <span class="s">correction</span>
<span class="na">doc_gap</span><span class="pi">:</span> <span class="s">docs/api-guidelines.md</span>
<span class="na">date</span><span class="pi">:</span> <span class="s">2026-05-19</span>
<span class="nn">---</span>

The agent used snake_case for a new REST endpoint path. The user corrected it to kebab-case,
which is the convention for all API routes in this project.</code></pre>
</div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="the-startup-reminder">The startup reminder</h3>
<div class="paragraph">
<p>On the next session start, a <code>SessionStart</code> hook runs the <a href="https://github.com/gwenneg/blog-ai-friction-loop/blob/main/skills/setup-friction-capture/scripts/friction-reminder.sh" target="_blank" rel="noopener">friction-reminder.sh</a> script to check how many sessions have unprocessed friction:</p>
</div>
<div class="listingblock">
<div class="title">.claude/settings.json</div>
<div class="content">
<pre class="rouge highlight"><code data-lang="json"><span class="p">{</span><span class="w">
  </span><span class="nl">"hooks"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
    </span><span class="nl">"SessionStart"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
      </span><span class="p">{</span><span class="w">
        </span><span class="nl">"matcher"</span><span class="p">:</span><span class="w"> </span><span class="s2">""</span><span class="p">,</span><span class="w">
        </span><span class="nl">"hooks"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
          </span><span class="p">{</span><span class="w">
            </span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"command"</span><span class="p">,</span><span class="w">
            </span><span class="nl">"command"</span><span class="p">:</span><span class="w"> </span><span class="s2">"bash .claude/scripts/friction-reminder.sh"</span><span class="p">,</span><span class="w">
            </span><span class="nl">"timeout"</span><span class="p">:</span><span class="w"> </span><span class="mi">2000</span><span class="w">
          </span><span class="p">}</span><span class="w">
        </span><span class="p">]</span><span class="w">
      </span><span class="p">}</span><span class="w">
    </span><span class="p">]</span><span class="w">
  </span><span class="p">}</span><span class="w">
</span><span class="p">}</span></code></pre>
</div>
</div>
<div class="paragraph">
<p>Once a configurable threshold is reached (default: 3 sessions), Claude Code displays a reminder at startup:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="markdown">╔══════════════════════════════════════════════════╗
║  🤖 BEEP BOOP IMPORTANT MESSAGE                  ║
╠══════════════════════════════════════════════════╣
║                                                  ║
║  5 sessions, 12 stumbles. I'm not proud.         ║
║  Update the docs. For both our sakes.            ║
║                                                  ║
║  → /update-context-docs                          ║
║                                                  ║
║  Not your thing? Run /disable-friction-capture.  ║
╚══════════════════════════════════════════════════╝</code></pre>
</div>
</div>
<div class="paragraph">
<p>The messages rotate randomly from a set of five.
Nothing is blocked. The developer can ignore the reminder, act on it by running <a href="https://github.com/gwenneg/blog-ai-friction-loop/blob/main/skills/update-context-docs/SKILL.md" target="_blank" rel="noopener"><code>/update-context-docs</code></a>, or opt out entirely with <a href="https://github.com/gwenneg/blog-ai-friction-loop/blob/main/skills/disable-friction-capture/SKILL.md" target="_blank" rel="noopener"><code>/disable-friction-capture</code></a>.</p>
</div>
</div>
<div class="sect2">
<h3 id="turning-friction-into-doc-edits">Turning friction into doc edits</h3>
<div class="paragraph">
<p>When the developer runs <a href="https://github.com/gwenneg/blog-ai-friction-loop/blob/main/skills/update-context-docs/SKILL.md" target="_blank" rel="noopener"><code>/update-context-docs</code></a>, the skill first checks for toolkit updates and lets the developer decide whether to install them. If an update changes a skill definition, Claude Code needs to be restarted before it can continue. Otherwise, the skill proceeds to process the accumulated friction files.</p>
</div>
<div class="admonitionblock warning">
<table>
<tr>
<td class="icon">
<i class="fa icon-warning" title="Warning"></i>
</td>
<td class="content">
<div class="paragraph">
<p>Each time <code>/update-context-docs</code> detects a new toolkit version, it <a href="https://github.com/gwenneg/blog-ai-friction-loop/blob/main/skills/setup-friction-capture/scripts/install-friction-capture.sh" target="_blank" rel="noopener">re-downloads and executes scripts</a> from GitHub with no integrity verification. This is fine for a demo but would need checksums or signatures for production use.</p>
</div>
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>The skill groups events by target file and presents a numbered list. The developer can exclude any that look like noise.</p>
</div>
<div class="paragraph">
<p>For each remaining event, the skill assesses severity (low, medium, or high), edits the target doc, and enforces a 200-line cap per file. It only touches files that already exist and follows their existing formatting.</p>
</div>
<div class="paragraph">
<p>If a <a href="https://www.promptfoo.dev/docs/intro/" target="_blank" rel="noopener"><code>promptfoo.yaml</code></a> file exists in the repo, the skill can also propose eval test cases based on the friction events.</p>
</div>
<div class="paragraph">
<p>The skill then creates a branch, commits, and opens a pull request with a friction metrics summary:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="markdown"><span class="gu">## Friction Metrics</span>

<span class="gu">### Events</span>

|      Type      | Count |
|----------------|-------|
| Corrections    |   2   |
| Clarifications |   1   |
| Denials        |   1   |
| Mistakes       |   1   |
| <span class="gs">**Total**</span>      | <span class="gs">**5**</span> |

<span class="gu">### Outcomes</span>

|      Result      | Count |
|------------------|-------|
| Docs improved    |   3   |
| Eval cases added |   1   |
| Skipped by user  |   1   |

<span class="gs">**Docs improved:**</span> <span class="sb">`docs/api-guidelines.md`</span>, <span class="sb">`AGENTS.md`</span></code></pre>
</div>
</div>
<div class="paragraph">
<p>After processing, the friction files are deleted from <code>.claude/friction/</code>.</p>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="configuration">Configuration</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Set the following variables under the <code>env</code> key in <code>.claude/settings.json</code> for team-wide use, or in <code>.claude/settings.local.json</code> to keep them local.</p>
</div>
<table class="tableblock frame-all grid-all fit-content stretch">
<colgroup>
<col>
<col>
<col>
</colgroup>
<thead>
<tr>
<th class="tableblock halign-left valign-top">Variable</th>
<th class="tableblock halign-left valign-top">Description</th>
<th class="tableblock halign-left valign-top">Default</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>FRICTION_CAPTURE</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Controls whether friction capture is active. Run <a href="https://github.com/gwenneg/blog-ai-friction-loop/blob/main/skills/disable-friction-capture/SKILL.md" target="_blank" rel="noopener"><code>/disable-friction-capture</code></a> to opt out locally in a repo where friction capture is enabled.</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>1</code> (enabled)</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>FRICTION_SESSION_THRESHOLD</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Number of sessions with unprocessed friction before the startup reminder is displayed in Claude Code.</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>3</code></p></td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="sect1">
<h2 id="try-it-yourself">Try it yourself</h2>
<div class="sectionbody">
<div class="paragraph">
<p>All skills and scripts described in this post and the <a href="/2026/05/28/the-docs-your-ai-agent-is-missing" target="_blank" rel="noopener">previous one</a> are available as a Claude Code plugin in the <a href="https://github.com/gwenneg/blog-ai-friction-loop" target="_blank" rel="noopener">gwenneg/blog-ai-friction-loop</a> repository.</p>
</div>
<div class="admonitionblock warning">
<table>
<tr>
<td class="icon">
<i class="fa icon-warning" title="Warning"></i>
</td>
<td class="content">
<div class="paragraph">
<p>This repository is intended for experimentation. The toolkit periodically re-downloads and executes scripts from GitHub with no signature verification. If the repository or a release were compromised, malicious code would run on the developer&#8217;s machine at the next update. Do not use it on projects with sensitive or proprietary code until artifact signing is in place.</p>
</div>
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>Clone it and start Claude Code with the plugin from your target project:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="bash">git clone https://github.com/gwenneg/blog-ai-friction-loop.git
<span class="nb">cd</span> /path/to/your-project
claude <span class="nt">--plugin-dir</span> /path/to/blog-ai-friction-loop</code></pre>
</div>
</div>
<div class="paragraph">
<p>Then run <a href="https://github.com/gwenneg/blog-ai-friction-loop/blob/main/skills/setup-friction-capture/SKILL.md" target="_blank" rel="noopener"><code>/setup-friction-capture</code></a>, which handles the full installation:</p>
</div>
<div class="olist arabic">
<ol class="arabic">
<li>
<p>Downloads the scripts and skill definitions from the latest <a href="https://github.com/gwenneg/blog-ai-friction-loop/releases" target="_blank" rel="noopener">gwenneg/blog-ai-friction-loop release</a></p>
</li>
<li>
<p>Installs the <code>SessionStart</code> and <code>SessionEnd</code> hooks shown earlier into <code>.claude/settings.json</code></p>
</li>
<li>
<p>Updates <code>.gitignore</code> with an allowlist pattern for the <code>.claude/</code> directory</p>
</li>
<li>
<p>Commits everything and opens a PR</p>
</li>
</ol>
</div>
<div class="paragraph">
<p>The <code>.claude/</code> directory mixes things that should be committed (hooks, scripts, skills, shared settings) with things that should never be (friction logs, local settings, worktrees, lock files).
The allowlist pattern looks like this:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="gitignore">/.claude/* <i class="conum" data-value="1"></i><b>(1)</b>
!/.claude/settings.json <i class="conum" data-value="2"></i><b>(2)</b>
!/.claude/scripts/ <i class="conum" data-value="3"></i><b>(3)</b>
!/.claude/skills/ <i class="conum" data-value="4"></i><b>(4)</b>
!/.claude/.friction-capture-version <i class="conum" data-value="5"></i><b>(5)</b></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>Ignore everything in <code>.claude/</code> by default.</td>
</tr>
<tr>
<td><i class="conum" data-value="2"></i><b>2</b></td>
<td>Shared settings are committed so the team gets the same hooks.</td>
</tr>
<tr>
<td><i class="conum" data-value="3"></i><b>3</b></td>
<td>Scripts are committed so the hooks work for everyone.</td>
</tr>
<tr>
<td><i class="conum" data-value="4"></i><b>4</b></td>
<td>Skills provide <code>/update-context-docs</code> and <code>/disable-friction-capture</code> to all developers.</td>
</tr>
<tr>
<td><i class="conum" data-value="5"></i><b>5</b></td>
<td>Tracks the installed toolkit version for update detection.</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>The PR includes a note for reviewers about what data is sent and how to opt out.</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="whats-next">What&#8217;s next</h2>
<div class="sectionbody">
<div class="paragraph">
<p>As AI agents take on more of the coding work, keeping context docs accurate becomes critical. Friction will always exist. The question is whether it gets captured and fixed, or silently repeated across sessions.</p>
</div>
<div class="paragraph">
<p>I&#8217;m deploying this across several repositories at Red Hat and collecting feedback. This is one approach to the problem, and I expect it to change as teams adapt it to their own workflows.</p>
</div>
<div class="paragraph">
<p>The capture is purely reactive right now: it catches what went wrong, not what could go wrong. A major refactor can silently invalidate parts of the guidelines, and the loop only notices once an agent stumbles. That&#8217;s a known gap and something I want to address.</p>
</div>
<div class="paragraph">
<p>I&#8217;m also working on the developer experience, exploring ways to further minimize the effort required from devs while keeping the docs updated automatically. That could eventually mean getting rid of the manual step entirely.</p>
</div>
<div class="paragraph">
<p>If you try this or build something different, I&#8217;d love to hear about it. What breaks, what works, and what the next step looks like for your team.
Feel free to share in the comments.</p>
</div>
</div>
</div>]]></content><author><name>Gwenneg Lepage</name></author><category term="agents.md" /><category term="ai" /><category term="claude code" /><category term="context engineering" /><category term="developer experience" /><category term="documentation" /><summary type="html"><![CDATA[An experiment using automated friction capture to continuously improve AI context docs, fixing both codebase drift and gaps in initially generated content.]]></summary></entry><entry><title type="html">The docs your AI agent is missing</title><link href="https://gwenneg.github.io/2026/05/28/the-docs-your-ai-agent-is-missing.html" rel="alternate" type="text/html" title="The docs your AI agent is missing" /><published>2026-05-28T00:00:00+00:00</published><updated>2026-05-28T00:00:00+00:00</updated><id>https://gwenneg.github.io/2026/05/28/the-docs-your-ai-agent-is-missing</id><content type="html" xml:base="https://gwenneg.github.io/2026/05/28/the-docs-your-ai-agent-is-missing.html"><![CDATA[<div id="preamble">
<div class="sectionbody">
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
<div class="paragraph">
<p>This post was written with the help of AI.</p>
</div>
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>In LangChain&#8217;s <a href="https://www.langchain.com/state-of-agent-engineering" target="_blank" rel="noopener">2026 survey of 1,300 AI professionals</a>, large enterprises cited context engineering and managing context at scale as one of their biggest challenges in ensuring agent quality.
The term "context engineering" is barely a year old, and most organizations are still figuring it out.</p>
</div>
<div class="paragraph">
<p>The challenge is no longer whether AI can write code.
It&#8217;s whether it knows enough about <em>your</em> project to write the right code.
And even when teams invest in writing context docs, keeping them accurate is its own problem.
A <a href="https://arxiv.org/pdf/2510.21413" target="_blank" rel="noopener">study of 10,000 open-source repositories</a> found that half of the <code>AGENTS.md</code> files it identified were never modified after creation.</p>
</div>
<div class="paragraph">
<p>I&#8217;ve been experimenting with this at Red Hat, trying different ways to structure and maintain context docs for AI agents.
In this post, I&#8217;ll walk through <a href="https://github.com/gwenneg/blog-ai-friction-loop/blob/main/skills/init-context-docs/SKILL.md" target="_blank" rel="noopener">a Claude Code skill</a> I built to help bootstrap the documentation agents need to work effectively on a project.
Here&#8217;s a before/after on a real project:</p>
</div>
<div class="paragraph">
<p><span class="image"><img src="/assets/images/posts/the-docs-your-ai-agent-is-missing/before-after.png" alt="Before/after comparison" width="90%"></span></p>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
<div class="paragraph">
<p>An agent can get context from many sources: static files loaded at startup, dynamic retrieval (RAG), memory systems and more.
This post only covers the first: repo-level documentation that describes conventions, pitfalls, architectural decisions and other project knowledge not visible in the code.</p>
</div>
</td>
</tr>
</table>
</div>
</div>
</div>
<div class="sect1">
<h2 id="initiating-context-docs-with-a-claude-skill">Initiating context docs with a Claude skill</h2>
<div class="sectionbody">
<div class="paragraph">
<p>The <code>/init-context-docs</code> skill assesses a repository&#8217;s readiness for AI-assisted development, then bootstraps a layered set of context files.
While <code>AGENTS.md</code> and the guideline files work with any tool that supports the standard, the loading behaviors described here are specific to Claude Code:</p>
</div>
<table class="tableblock frame-all grid-all stretch">
<colgroup>
<col style="width: 25%;">
<col style="width: 50%;">
<col style="width: 25%;">
</colgroup>
<thead>
<tr>
<th class="tableblock halign-left valign-top">File</th>
<th class="tableblock halign-left valign-top">Role</th>
<th class="tableblock halign-left valign-top">Loading behavior</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>docs/*-guidelines.md</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Domain-specific guidelines (e.g., security, testing, database). Capped at 200 lines.</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">At Claude&#8217;s discretion via <code>AGENTS.md</code> index</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>AGENTS.md</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Cross-cutting conventions and index to guideline files. Agent-agnostic. Capped at 200 lines.</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">At Claude&#8217;s discretion by default; unconditional when imported via <code>@AGENTS.md</code> in <code>CLAUDE.md</code></p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>.claude/rules/*.md</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Path-scoped loaders that force a guideline into context for matching files.</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Path-scoped deterministic load</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>CLAUDE.md</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Thin Claude Code-specific layer. Imports <code>AGENTS.md</code> via <code>@AGENTS.md</code>. Capped at 100 lines.</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Unconditional load (every session)</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>README.md</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">High-level project context for humans and agents alike.</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">At Claude&#8217;s discretion</p></td>
</tr>
</tbody>
</table>
<div class="paragraph">
<div class="title">Context docs loading layers</div>
<p><span class="image"><img src="/assets/images/posts/the-docs-your-ai-agent-is-missing/context-docs-loading-layers.svg" alt="Context docs loading layers" width="85%"></span></p>
</div>
<div class="paragraph">
<p>The unconditional layer is the most expensive in terms of context window tokens: it&#8217;s loaded every session regardless of the task.
Files loaded at Claude&#8217;s discretion are the cheapest, but come with no guarantee.
Path-scoped rules sit in between: deterministic loading, but only for matching files.</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="why-size-and-ordering-matter">Why size and ordering matter</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Several studies point in the same direction: longer context alone hurts LLM performance, even when the content is relevant (<a href="https://arxiv.org/pdf/2510.05381" target="_blank" rel="noopener">Du et al., 2025</a>).
Models pay the most attention to the beginning and end of their input; everything in the middle gets less (<a href="https://arxiv.org/pdf/2307.03172" target="_blank" rel="noopener">Liu et al., 2024</a>; <a href="https://arxiv.org/pdf/2502.01951" target="_blank" rel="noopener">Wu et al., 2025</a>).
Anthropic frames context as <a href="https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents" target="_blank" rel="noopener">"a precious, finite resource"</a>.
On the flip side, well-formed <code>AGENTS.md</code> files are associated with ~29% faster execution (<a href="https://arxiv.org/pdf/2601.20404" target="_blank" rel="noopener">Lulla et al., 2026</a>).
But more isn&#8217;t better: LLM-generated context files with unnecessary content actually reduce task success rates compared to no context at all (<a href="https://arxiv.org/pdf/2602.11988" target="_blank" rel="noopener">Gloaguen et al., 2026</a>).</p>
</div>
<div class="paragraph">
<p>This is why every file in the system is capped (200 lines for guidelines and <code>AGENTS.md</code>, 100 for <code>CLAUDE.md</code>), the most critical rules go first, verification commands go last, and everything in between is kept ruthlessly short.</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="why-each-layer-matters">Why each layer matters</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Every layer is a tradeoff between loading guarantee and token cost. The question for each file is: does the agent need this every session, only for certain files, or only sometimes?</p>
</div>
<div class="sect2">
<h3 id="guidelines">Guidelines</h3>
<div class="paragraph">
<p>This is the deepest layer: concrete rules about how <em>this</em> codebase does things (e.g., "use <code>middleware/validator.ts</code> for input validation"), not general best practices (e.g., "always validate user input").
Each guideline targets a specific domain (e.g., security, testing, database), determined dynamically based on what the skill finds in the repo.</p>
</div>
<div class="paragraph">
<p>Anthropic calls self-verification <a href="https://code.claude.com/docs/en/best-practices#give-claude-a-way-to-verify-its-work" target="_blank" rel="noopener">"the single highest-leverage thing you can do"</a>, so every guideline ends with a Verification section listing commands the agent can run to check its own work.</p>
</div>
</div>
<div class="sect2">
<h3 id="agents-md">AGENTS.md</h3>
<div class="paragraph">
<p><a href="https://agents.md/" target="_blank" rel="noopener">AGENTS.md</a> is an open standard stewarded by the Linux Foundation, supported by all major AI coding tools and adopted by 60K+ open-source projects.
It serves as the agent-agnostic onboarding doc: cross-cutting conventions plus an index pointing to the detailed guidelines.</p>
</div>
<div class="paragraph">
<p>Claude uses this index to decide which guidelines to load based on the current task.
This is a judgment call by the agent, not a guaranteed mechanism.
In practice, Claude sometimes skips a guideline it should have loaded, especially when the relevance isn&#8217;t obvious from the file names alone.
Path-scoped rules (covered below) exist to address that.</p>
</div>
</div>
<div class="sect2">
<h3 id="path-scoped-rules">Path-scoped rules</h3>
<div class="paragraph">
<p><code>.claude/rules/*.md</code> files let you force a guideline into context only when Claude works with matching files.
This addresses the gap mentioned above: when Claude&#8217;s own judgment about which guidelines to load isn&#8217;t reliable enough, a rule makes the loading deterministic.</p>
</div>
<div class="listingblock">
<div class="title">.claude/rules/testing.md</div>
<div class="content">
<pre class="rouge highlight"><code data-lang="markdown">paths:
<span class="p">  -</span> "<span class="gs">**/test/**</span>"

@docs/testing-guidelines.md</code></pre>
</div>
</div>
<div class="paragraph">
<p>The benefit is precision: a guideline only consumes tokens when the agent actually needs it.</p>
</div>
<div class="paragraph">
<p>But use rules sparingly.
Broad globs like <code>**/*</code> make a guideline effectively always-loaded. Multiple rules can stack up on the same file.
Patterns can go stale if the codebase restructures.
And rules are Claude Code-specific; other tools don&#8217;t support them.</p>
</div>
<div class="admonitionblock warning">
<table>
<tr>
<td class="icon">
<i class="fa icon-warning" title="Warning"></i>
</td>
<td class="content">
<div class="paragraph">
<p>If <code>.claude/rules/</code> is gitignored, rules won&#8217;t be committed and other developers won&#8217;t benefit from them.
Make sure the rules directory is explicitly allowed in your <code>.gitignore</code>.
The skill checks for this and offers to add an exception.</p>
</div>
</td>
</tr>
</table>
</div>
</div>
<div class="sect2">
<h3 id="claude-md">CLAUDE.md</h3>
<div class="paragraph">
<p>This is the thinnest layer: only what applies exclusively to Claude Code.
It&#8217;s loaded unconditionally at the start of every session, so every line consumes context window tokens regardless of the task.
Anthropic&#8217;s guidance is blunt: <a href="https://code.claude.com/docs/en/best-practices#write-an-effective-claude-md" target="_blank" rel="noopener">"Bloated CLAUDE.md files cause Claude to ignore your actual instructions!"</a></p>
</div>
<div class="paragraph">
<p>Adding <code>@AGENTS.md</code> near the top guarantees Claude always has the agent guidance available.
Without it, Claude may or may not read <code>AGENTS.md</code> on its own. Loading is not guaranteed.
The tradeoff: the import makes loading deterministic, at the cost of consuming <code>AGENTS.md</code> tokens unconditionally every session.</p>
</div>
<div class="admonitionblock warning">
<table>
<tr>
<td class="icon">
<i class="fa icon-warning" title="Warning"></i>
</td>
<td class="content">
<div class="paragraph">
<p>Avoid importing guideline files directly from <code>CLAUDE.md</code>.
That would make them always-loaded, defeating the on-demand architecture the rest of the system is built around.</p>
</div>
</td>
</tr>
</table>
</div>
</div>
<div class="sect2">
<h3 id="readme">README</h3>
<div class="paragraph">
<p>A well-structured README can answer "what does this project do and how do I build it?" without the agent exploring dozens of files.
The skill generates one that front-loads the essentials (project purpose, tech stack, build instructions) and links to <code>AGENTS.md</code> and <code>docs/</code> for the deeper layers.</p>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="how-the-skill-works">How the skill works</h2>
<div class="sectionbody">
<div class="paragraph">
<div class="title">Skill execution flow</div>
<p><span class="image"><img src="/assets/images/posts/the-docs-your-ai-agent-is-missing/skill-execution-flow.svg" alt="Skill execution flow" width="85%"></span></p>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
<div class="paragraph">
<p>At each phase, the skill asks whether to proceed, lets you skip layers, and offers customization before generating anything.</p>
</div>
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>The skill starts by scanning the repo against the five layers described above and presents a baseline status showing what already exists and what&#8217;s missing.
It then matches the repo against a curated list of guideline domains (e.g., security, testing, database), drops any that aren&#8217;t relevant, identifies additional domains not on the list (e.g., GraphQL, machine learning), and lets you adjust the final selection.</p>
</div>
<div class="paragraph">
<p>Once the domains selection is final, the skill launches one generation agent per domain, each focused exclusively on its area.
An agent that only needs to produce a testing guideline can dig deep into your test patterns, rather than spreading its attention across the entire project.
The same pattern applies to <code>AGENTS.md</code>, <code>CLAUDE.md</code>, and <code>README</code>: each gets its own generation agent.
If a file already exists, the generation agent reads the existing content first and incorporates it, updating with new findings while preserving what&#8217;s still accurate.</p>
</div>
<div class="paragraph">
<p>Each generation agent reads a <a href="https://github.com/gwenneg/blog-ai-friction-loop/tree/main/skills/init-context-docs/checklists" target="_blank" rel="noopener">quality checklist</a> before writing anything.
The checklists enforce constraints like the 200-line cap, the necessity test ("Would removing this cause an agent to make a mistake?"), a ban on absolute language without evidence, and the requirement that every file reference actually exists in the codebase.
Generation and verification agents share the same checklists, so both are measured against the same standard.</p>
</div>
<div class="paragraph">
<p>A separate agent handles verification. The one that wrote the content is biased toward believing it&#8217;s correct: research shows that <a href="https://arxiv.org/pdf/2404.13076" target="_blank" rel="noopener">LLMs systematically favor their own output</a> (Panickssery et al., 2024) and <a href="https://arxiv.org/pdf/2507.02778" target="_blank" rel="noopener">fail to correct their own errors</a> while successfully correcting identical ones from external sources (Tsui, 2025).
There&#8217;s a catch, though: both studies examine self-bias within a single model.
The skill uses a different model for verification (Sonnet instead of Opus), but both belong to the same Claude family, so some bias may carry over.
Still, a fresh agent with a clean context checks file references against the actual codebase, validates factual claims with <code>WebSearch</code>, looks for contradictions across files, and flags duplication across layers.
It only corrects inaccuracies.
It doesn&#8217;t add new content.</p>
</div>
<div class="paragraph">
<p>After all agents finish, an <a href="https://github.com/gwenneg/blog-ai-friction-loop/blob/main/skills/init-context-docs/scripts/automated-checks.sh" target="_blank" rel="noopener"><code>automated-checks.sh</code></a> script validates the generated files (line counts, imports, docs index, secret detection) and re-runs until all checks pass.</p>
</div>
<div class="paragraph">
<p>The skill finally offers to create a pull request with all the changes, so the generated docs go through the team&#8217;s normal review process.</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="try-it-yourself">Try it yourself</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Want to see how this works on your own project? Clone the <a href="https://github.com/gwenneg/blog-ai-friction-loop" target="_blank" rel="noopener">gwenneg/blog-ai-friction-loop</a> repository, then start Claude Code with the plugin from your target project:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="bash">git clone https://github.com/gwenneg/blog-ai-friction-loop.git
<span class="nb">cd</span> /path/to/your-project
claude <span class="nt">--plugin-dir</span> /path/to/blog-ai-friction-loop</code></pre>
</div>
</div>
<div class="paragraph">
<p>Then type <code>/init-context-docs</code>.
The skill will walk you through each layer, ask what to include, and let you skip anything that doesn&#8217;t apply.
Expect it to take anywhere from a few minutes to about half an hour depending on the size of your codebase.</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="whats-next">What&#8217;s next</h2>
<div class="sectionbody">
<div class="paragraph">
<p>In my own testing, the generated docs clearly improved how agents work with projects: fewer wrong assumptions, less time spent correcting output.
I don&#8217;t have metrics to back that up yet.
It&#8217;s based on what I observed across a handful of repositories.</p>
</div>
<div class="paragraph">
<p>But even after all of this, the output is imperfect.
Guidelines can sound plausible while being wrong about specifics, and some conventions only reveal themselves when the agent actually tries to follow the docs.</p>
</div>
<div class="paragraph">
<p>These are <em>friction events</em>, moments where the docs failed the agent:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>The user corrected the agent&#8217;s output</p>
</li>
<li>
<p>The agent asked a question the docs should have answered</p>
</li>
<li>
<p>The agent made a wrong assumption about the codebase</p>
</li>
<li>
<p>A tool call was denied because it violated a project policy the agent didn&#8217;t know about</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Each one is a signal that the docs have a gap, something a concrete rule or example could have prevented.</p>
</div>
<div class="paragraph">
<p>In a <a href="/2026/05/31/turn-ai-friction-into-better-docs" target="_blank" rel="noopener">follow-up post</a>, I introduce an experiment to capture these friction events automatically at the end of every session and turn them into targeted doc improvements.</p>
</div>
<div class="paragraph">
<p>If you try <code>/init-context-docs</code> on your own codebase, I&#8217;m curious what it gets right, what it misses, and what you end up changing.
Feel free to share your experience in the comments.</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="sources">Sources</h2>
<div class="sectionbody">
<table class="tableblock frame-all grid-all stretch">
<colgroup>
<col style="width: 60%;">
<col style="width: 20%;">
<col style="width: 20%;">
</colgroup>
<thead>
<tr>
<th class="tableblock halign-left valign-top">Source</th>
<th class="tableblock halign-left valign-top">Author</th>
<th class="tableblock halign-left valign-top">Date</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock"><a href="https://arxiv.org/pdf/2602.11988" target="_blank" rel="noopener">Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?</a></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Gloaguen et al. (ETH Zurich)</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Feb 2026</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock"><a href="https://arxiv.org/pdf/2601.20404" target="_blank" rel="noopener">On the Impact of AGENTS.md Files on the Efficiency of AI Coding Agents</a></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Lulla et al.</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Jan 2026</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock"><a href="https://www.langchain.com/state-of-agent-engineering" target="_blank" rel="noopener">State of Agent Engineering</a></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">LangChain</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Early 2026</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock"><a href="https://arxiv.org/pdf/2510.21413" target="_blank" rel="noopener">Context Engineering for AI Agents in Open-Source Software</a> (peer-reviewed, MSR 2026)</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Mohsenimofidi et al.</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Oct 2025</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock"><a href="https://arxiv.org/pdf/2510.05381" target="_blank" rel="noopener">Context Length Alone Hurts LLM Performance Despite Perfect Retrieval</a> (peer-reviewed, EMNLP Findings 2025)</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Du et al.</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Oct 2025</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock"><a href="https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents" target="_blank" rel="noopener">Effective Context Engineering for AI Agents</a></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Anthropic</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Sep 2025</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock"><a href="https://agents.md/" target="_blank" rel="noopener">AGENTS.md open standard</a></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Linux Foundation</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Aug 2025</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock"><a href="https://arxiv.org/pdf/2507.02778" target="_blank" rel="noopener">Self-Correction Bench: Uncovering and Addressing the Self-Correction Blind Spot in Large Language Models</a></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Tsui</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Jul 2025</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock"><a href="https://code.claude.com/docs/en/best-practices" target="_blank" rel="noopener">Best practices for Claude Code</a></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Anthropic</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">May 2025 (updated regularly)</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock"><a href="https://arxiv.org/pdf/2502.01951" target="_blank" rel="noopener">On the Emergence of Position Bias in Transformers</a> (peer-reviewed, ICML 2025)</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Wu et al. (MIT)</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Feb 2025</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock"><a href="https://arxiv.org/pdf/2404.13076" target="_blank" rel="noopener">LLM Evaluators Recognize and Favor Their Own Generations</a> (peer-reviewed, NeurIPS 2024)</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Panickssery et al.</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Apr 2024</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock"><a href="https://arxiv.org/pdf/2307.03172" target="_blank" rel="noopener">Lost in the Middle: How Language Models Use Long Contexts</a> (peer-reviewed, TACL 2024)</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Liu et al. (Stanford)</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Feb 2024</p></td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="sect1">
<h2 id="special-thanks">Special thanks</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Thanks to Jiří Bönsch for helping me test and improve the <code>/init-context-docs</code> skill.</p>
</div>
</div>
</div>]]></content><author><name>Gwenneg Lepage</name></author><category term="agents.md" /><category term="ai" /><category term="claude code" /><category term="context engineering" /><category term="developer experience" /><category term="documentation" /><summary type="html"><![CDATA[A Claude Code skill that builds the context docs your AI agent needs, backed by research on how LLMs process context.]]></summary></entry><entry><title type="html">Teaching Claude Code skills to improve themselves</title><link href="https://gwenneg.github.io/2026/05/04/self-improving-claude-code-skills.html" rel="alternate" type="text/html" title="Teaching Claude Code skills to improve themselves" /><published>2026-05-04T00:00:00+00:00</published><updated>2026-05-04T00:00:00+00:00</updated><id>https://gwenneg.github.io/2026/05/04/self-improving-claude-code-skills</id><content type="html" xml:base="https://gwenneg.github.io/2026/05/04/self-improving-claude-code-skills.html"><![CDATA[<div id="preamble">
<div class="sectionbody">
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
<div class="paragraph">
<p>This post was written with the help of AI.</p>
</div>
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>I&#8217;ve been so deep into AI tooling lately that blogging fell off the radar&#8201;&#8212;&#8201;things move too fast to stop and write about them.
But this one is such a quick win that I had to share it, even if someone else has probably had the same idea before.</p>
</div>
<div class="paragraph">
<p>If you&#8217;re using <a href="https://docs.anthropic.com/en/docs/claude-code/skills" target="_blank" rel="noopener">Claude Code skills</a>, you&#8217;ve probably noticed that getting them right takes a few iterations.
A skill might miss a step, produce something slightly off, or lack context that your <code>CLAUDE.md</code> should have provided.
The usual fix is to notice the problem, remember to update the skill later, and then&#8230;&#8203; forget about it.</p>
</div>
<div class="paragraph">
<p>What if the skill itself could flag those issues for you?</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="the-idea">The idea</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Add an optional final step to your skills that asks Claude to look back at what just happened and suggest improvements.
Not to the code it produced&#8201;&#8212;&#8201;to the skill definition, <code>CLAUDE.md</code>, or any other project documentation that would have made the execution smoother.</p>
</div>
<div class="paragraph">
<p>Think of it as a mini retrospective that runs every time, at near-zero cost.</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="the-instruction">The instruction</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Here&#8217;s what I append to my skills:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="markdown"><span class="gu">## Optional: Self-Improvement Review</span>

After completing the skill, use AskUserQuestion to ask the user if they want to run the self-improvement review.

If they decline, skip it entirely.

If they accept, reflect on your execution:
<span class="p">
-</span> Did anything fail, feel awkward, or require unnecessary retries?
<span class="p">-</span> Were you missing context that CLAUDE.md or another project doc should have provided?
<span class="p">-</span> Is there a step in this skill that was unclear, redundant, or in the wrong order?

If you identify a concrete improvement, present it as a <span class="gs">**diff to the relevant file**</span> (skill definition, CLAUDE.md, AGENTS.md, etc.) and offer to apply it. Do NOT just list observations — every finding must come with an actionable diff.
Do not apply changes without approval.
If nothing stands out, say so briefly and move on — do not force feedback.</code></pre>
</div>
</div>
<div class="paragraph">
<p>A few things worth noting:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>Asking for a <strong>diff</strong> prevents vague suggestions like "consider improving the error handling step." It forces something actionable.</p>
</li>
<li>
<p>The "if nothing stands out, move on" line is important. Without it, Claude will invent problems to fill the step every single time.</p>
</li>
<li>
<p>The scope covers both the skill <em>and</em> project docs. A skill might work perfectly but still struggle because <code>CLAUDE.md</code> is missing a convention or a path.</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="does-it-actually-work">Does it actually work?</h2>
<div class="sectionbody">
<div class="paragraph">
<p>In my experience, most executions produce no suggestion&#8201;&#8212;&#8201;which is the right outcome.
But when it does flag something, it&#8217;s usually a missing convention in <code>CLAUDE.md</code> or a step in the skill that assumed context Claude didn&#8217;t have.
Those are exactly the kind of issues you&#8217;d never bother tracking down yourself.</p>
</div>
<div class="paragraph">
<p>It won&#8217;t revolutionize your workflow, but it&#8217;s a small investment that compounds over time.</p>
</div>
</div>
</div>]]></content><author><name>Gwenneg Lepage</name></author><category term="ai" /><category term="claude code" /><summary type="html"><![CDATA[A simple instruction you can append to your Claude Code skills to make them suggest improvements to themselves and your project docs.]]></summary></entry><entry><title type="html">Easing the maintenance of Konflux build pipelines</title><link href="https://gwenneg.github.io/2025/04/11/konflux-remote-pipeline.html" rel="alternate" type="text/html" title="Easing the maintenance of Konflux build pipelines" /><published>2025-04-11T00:00:00+00:00</published><updated>2025-04-11T00:00:00+00:00</updated><id>https://gwenneg.github.io/2025/04/11/konflux-remote-pipeline</id><content type="html" xml:base="https://gwenneg.github.io/2025/04/11/konflux-remote-pipeline.html"><![CDATA[<div id="preamble">
<div class="sectionbody">
<div class="paragraph">
<p><a href="https://konflux-ci.dev" target="_blank" rel="noopener">Konflux</a> is an open source, cloud-native software factory developed by Red Hat and focused on software supply chain security.</p>
</div>
<div class="paragraph">
<p>When a component is <a href="https://konflux-ci.dev/docs/building/creating" target="_blank" rel="noopener">onboarded to Konflux</a>, the <a href="https://github.com/apps/red-hat-konflux" target="_blank" rel="noopener">Red Hat Konflux app</a> automatically creates two build pipelines in the Git repository:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>${component.name}-pull-request.yaml</code></p>
</li>
<li>
<p><code>${component.name}-push.yaml</code></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Then, every once in a while, <a href="https://github.com/konflux-ci/mintmaker" target="_blank" rel="noopener">MintMaker</a> — a Konflux hosted instance of the <a href="https://github.com/renovatebot/renovate" target="_blank" rel="noopener">Renovate bot</a> — will open PRs to update the Konflux task references in the pipelines.
Some of these PRs may also include <a href="https://github.com/RedHatInsights/konflux-pipelines/pull/58" target="_blank" rel="noopener">migration notes</a> and require additional work.</p>
</div>
<div class="paragraph">
<p>If you&#8217;re managing multiple repositories onboarded to Konflux, keeping all the pipelines up to date can require significant effort over time.
In this post, I&#8217;ll show you different ways to make pipeline maintenance easier by using remote pipelines and tweaking MintMaker&#8217;s settings.</p>
</div>
<div class="admonitionblock warning">
<table>
<tr>
<td class="icon">
<i class="fa icon-warning" title="Warning"></i>
</td>
<td class="content">
<div class="paragraph">
<p>This post includes multiple references to the <a href="https://github.com/RedHatInsights/konflux-pipelines" target="_blank" rel="noopener">RedHatInsights/konflux-pipelines</a> repository.
If you&#8217;re not a member of the RedHatInsights organization on GitHub, please consider forking the repository before using the remote pipelines it contains, as they may be modified at any time without prior notice.</p>
</div>
</td>
</tr>
</table>
</div>
</div>
</div>
<div class="sect1">
<h2 id="introducing-remote-pipelines">Introducing remote pipelines</h2>
<div class="sectionbody">
<div class="paragraph">
<p><a href="https://tekton.dev/docs/pipelines/resolution" target="_blank" rel="noopener">Remote pipelines</a> from Tekton let you reference pipeline definitions stored in external Git repositories instead of defining them locally in each project.
They make it easier to share and maintain pipelines across multiple repositories.</p>
</div>
<div class="paragraph">
<p>Konflux is built on top of Tekton.
Let&#8217;s see how we can leverage remote pipelines in a Konflux build pipeline!</p>
</div>
<div class="paragraph">
<p>Nothing shows it better than a good PR: <a href="https://github.com/gwenneg/blog-remote-konflux-pipeline/pull/2/files" target="_blank" rel="noopener">gwenneg/blog-remote-konflux-pipeline#2</a>.
If you don&#8217;t have time to review the whole thing, here&#8217;s the TL;DR:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="yaml"><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">tekton.dev/v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">PipelineRun</span>
<span class="na">metadata</span><span class="pi">:</span>
  <span class="na">annotations</span><span class="pi">:</span>
    <span class="na">pipelinesascode.tekton.dev/pipeline</span><span class="pi">:</span> <span class="pi">&gt;</span> <i class="conum" data-value="1"></i><b>(1)</b>
      <span class="s">https://github.com/RedHatInsights/konflux-pipelines/raw/main/pipelines/docker-build-oci-ta.yaml</span>
    <span class="c1"># Additional annotations omitted for brevity</span>
  <span class="na">labels</span><span class="pi">:</span> <span class="s">...</span> <span class="c1"># Omitted for brevity</span>
  <span class="na">name</span><span class="pi">:</span> <span class="s">remote-konflux-pipeline-on-pull-request</span>
  <span class="na">namespace</span><span class="pi">:</span> <span class="s">glepage-tenant</span>
<span class="na">spec</span><span class="pi">:</span>
  <span class="na">params</span><span class="pi">:</span> <span class="s">...</span> <span class="c1"># Omitted for brevity</span>
  <span class="na">pipelineRef</span><span class="pi">:</span> <i class="conum" data-value="2"></i><b>(2)</b>
    <span class="na">name</span><span class="pi">:</span> <span class="s">docker-build-oci-ta</span> <i class="conum" data-value="3"></i><b>(3)</b>
  <span class="na">workspaces</span><span class="pi">:</span> <span class="s">...</span> <span class="c1"># Omitted for brevity</span></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>This annotation comes from <a href="https://pipelinesascode.com/docs/guide/resolver/#remote-pipeline-annotations" target="_blank" rel="noopener">Pipelines as Code</a> and references a remote pipeline from the <a href="https://github.com/RedHatInsights/konflux-pipelines" target="_blank" rel="noopener">RedHatInsights/konflux-pipelines</a> repository.</td>
</tr>
<tr>
<td><i class="conum" data-value="2"></i><b>2</b></td>
<td>The entire <code>pipelineSpec</code> section from the default Konflux pipeline is replaced with a minimal <code>pipelineRef</code> section.</td>
</tr>
<tr>
<td><i class="conum" data-value="3"></i><b>3</b></td>
<td>This value needs to match the <code>metadata.name</code> value from the remote pipeline.</td>
</tr>
</table>
</div>
<div class="sect2">
<h3 id="why-go-remote">Why go remote?</h3>
<div class="paragraph">
<p>After applying the changes shown above:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>MintMaker will no longer open PRs in your repository to update Konflux task references or request pipeline migrations.</p>
</li>
<li>
<p>Your pipeline runs will automatically depend on the latest version of the remote pipelines, thanks to the dependency on the <code>main</code> branch.</p>
</li>
<li>
<p>MintMaker will still open PRs unrelated to pipelines, such as <a href="https://github.com/RedHatInsights/sources-api-go/pull/835" target="_blank" rel="noopener">RedHatInsights/sources-api-go#835</a>, to update dependencies.</p>
</li>
</ul>
</div>
<div class="admonitionblock tip">
<table>
<tr>
<td class="icon">
<i class="fa icon-tip" title="Tip"></i>
</td>
<td class="content">
<div class="paragraph">
<p>If you want to test a change in a remote pipeline or roll back to an earlier version if something breaks, just point to a different Git branch or SHA.</p>
</div>
</td>
</tr>
</table>
</div>
</div>
<div class="sect2">
<h3 id="theres-a-catch">There&#8217;s a catch</h3>
<div class="paragraph">
<p>This approach almost entirely removes the need to maintain pipelines, but there&#8217;s a catch.
Since MintMaker won&#8217;t open PRs to update your pipelines anymore, any changes in the remote pipelines will go untested in your repository until you open another unrelated PR that triggers a pipeline run.</p>
</div>
<div class="paragraph">
<p>The next section explains how to work around this limitation.</p>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="depending-on-a-specific-release-of-remote-pipelines">Depending on a specific release of remote pipelines</h2>
<div class="sectionbody">
<div class="paragraph">
<p>If the repository that hosts the remote pipelines publishes releases on GitHub, your repository can depend on a specific release instead of always using the latest version.</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="yaml"><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">tekton.dev/v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">PipelineRun</span>
<span class="na">metadata</span><span class="pi">:</span>
  <span class="na">annotations</span><span class="pi">:</span>
    <span class="na">pipelinesascode.tekton.dev/pipeline</span><span class="pi">:</span> <span class="pi">&gt;</span> <i class="conum" data-value="1"></i><b>(1)</b>
      <span class="s">https://github.com/RedHatInsights/konflux-pipelines/raw/v1.2.0/pipelines/docker-build-oci-ta.yaml</span>
    <span class="c1"># Additional annotations omitted for brevity</span>
  <span class="na">labels</span><span class="pi">:</span> <span class="s">...</span> <span class="c1"># Omitted for brevity</span>
  <span class="na">name</span><span class="pi">:</span> <span class="s">remote-konflux-pipeline-on-pull-request</span>
  <span class="na">namespace</span><span class="pi">:</span> <span class="s">glepage-tenant</span>
<span class="na">spec</span><span class="pi">:</span>
  <span class="na">params</span><span class="pi">:</span> <span class="s">...</span> <span class="c1"># Omitted for brevity</span>
  <span class="na">pipelineRef</span><span class="pi">:</span>
    <span class="na">name</span><span class="pi">:</span> <span class="s">docker-build-oci-ta</span>
  <span class="na">workspaces</span><span class="pi">:</span> <span class="s">...</span> <span class="c1"># Omitted for brevity</span></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>The local pipeline now depends on version <code>v1.2.0</code> of the remote pipeline instead of <code>main</code>.</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>With this approach:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>MintMaker will automatically open PRs such as <a href="https://github.com/gwenneg/blog-remote-konflux-pipeline/pull/4" target="_blank" rel="noopener">gwenneg/blog-remote-konflux-pipeline#4</a> in your repository every time a new release of the remote pipelines is published.</p>
</li>
<li>
<p>Any changes in the remote pipelines will be immediately tested in your repository and you will catch issues as early as possible.</p>
</li>
<li>
<p>You still won&#8217;t have to worry about Konflux task reference updates or pipeline migrations.</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="customizing-mintmakers-settings">Customizing MintMaker&#8217;s settings</h2>
<div class="sectionbody">
<div class="admonitionblock tip">
<table>
<tr>
<td class="icon">
<i class="fa icon-tip" title="Tip"></i>
</td>
<td class="content">
<div class="paragraph">
<p>You can use the tips from this section whether or not you&#8217;re using remote pipelines.</p>
</div>
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>MintMaker works out of the box in repositories onboarded to Konflux and doesn&#8217;t require any additional configuration files.
However, it is possible to customize MintMaker&#8217;s settings by adding a <a href="https://github.com/gwenneg/blog-remote-konflux-pipeline/blob/main/renovate.json" target="_blank" rel="noopener">renovate.json</a> file at the root of your repository.
The default configuration is detailed in the <a href="https://konflux-ci.dev/docs/mintmaker/default-config" target="_blank" rel="noopener">Konflux doc</a>.</p>
</div>
<div class="sect2">
<h3 id="changing-when-mintmaker-runs">Changing when MintMaker runs</h3>
<div class="paragraph">
<p>The Renovate configuration lets you control when and how often MintMaker may open PRs in your repository:</p>
</div>
<div class="listingblock">
<div class="title">renovate.json</div>
<div class="content">
<pre class="rouge highlight"><code data-lang="json"><span class="p">{</span><span class="w">
  </span><span class="nl">"$schema"</span><span class="p">:</span><span class="w"> </span><span class="s2">"https://docs.renovatebot.com/renovate-schema.json"</span><span class="p">,</span><span class="w">
  </span><span class="nl">"extends"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">"github&gt;konflux-ci/mintmaker//config/renovate/renovate.json"</span><span class="p">],</span><span class="w"> <i class="conum" data-value="1"></i><b>(1)</b>
  </span><span class="nl">"tekton"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
    </span><span class="nl">"schedule"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">"on Tuesday after 3am and before 10am"</span><span class="p">]</span><span class="w"> <i class="conum" data-value="2"></i><b>(2)</b>
  </span><span class="p">}</span><span class="w">
</span><span class="p">}</span></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>This snippet extends <a href="https://github.com/konflux-ci/mintmaker/blob/main/config/renovate/renovate.json" class="bare" target="_blank" rel="noopener">https://github.com/konflux-ci/mintmaker/blob/main/config/renovate/renovate.json</a>.</td>
</tr>
<tr>
<td><i class="conum" data-value="2"></i><b>2</b></td>
<td>Renovate supports both natural language and cron-based scheduling.
See the <a href="https://docs.renovatebot.com/configuration-options/#schedule" target="_blank" rel="noopener">Renovate doc</a> for more details.</td>
</tr>
</table>
</div>
</div>
<div class="sect2">
<h3 id="automatically-approving-and-merging-mintmakers-prs">Automatically approving and merging MintMaker&#8217;s PRs</h3>
<div class="paragraph">
<p>You can also tweak your Renovate settings to automatically approve or <a href="https://github.com/gwenneg/blog-remote-konflux-pipeline/pull/11" target="_blank" rel="noopener">merge</a> PRs opened by MintMaker:</p>
</div>
<div class="listingblock">
<div class="title">renovate.json</div>
<div class="content">
<pre class="rouge highlight"><code data-lang="json"><span class="p">{</span><span class="w">
  </span><span class="nl">"$schema"</span><span class="p">:</span><span class="w"> </span><span class="s2">"https://docs.renovatebot.com/renovate-schema.json"</span><span class="p">,</span><span class="w">
  </span><span class="nl">"extends"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">"github&gt;konflux-ci/mintmaker//config/renovate/renovate.json"</span><span class="p">],</span><span class="w">
  </span><span class="nl">"tekton"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
    </span><span class="nl">"autoApprove"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w"> <i class="conum" data-value="1"></i><b>(1)</b>
    </span><span class="nl">"automerge"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w"> <i class="conum" data-value="2"></i><b>(2)</b>
  </span><span class="p">}</span><span class="w">
</span><span class="p">}</span></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>Auto-approving only works in GitLab, not in GitHub. Find more details in the <a href="https://docs.renovatebot.com/configuration-options/#autoapprove" target="_blank" rel="noopener">Renovate doc</a>.</td>
</tr>
<tr>
<td><i class="conum" data-value="2"></i><b>2</b></td>
<td>Auto-merging works in both GitHub and GitLab. Find more details in the <a href="https://docs.renovatebot.com/configuration-options/#automerge" target="_blank" rel="noopener">Renovate doc</a>.</td>
</tr>
</table>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="hosting-remote-konflux-pipelines">Hosting remote Konflux pipelines</h2>
<div class="sectionbody">
<div class="paragraph">
<p>If you plan on creating a repository to host remote pipelines, there are two things you&#8217;ll need to do:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>Onboard the repository to Konflux as a component.</p>
</li>
<li>
<p>Let MintMaker know where to find the remote pipelines so it can keep them updated.</p>
</li>
</ul>
</div>
<div class="listingblock">
<div class="title">renovate.json</div>
<div class="content">
<pre class="rouge highlight"><code data-lang="json"><span class="p">{</span><span class="w">
  </span><span class="nl">"$schema"</span><span class="p">:</span><span class="w"> </span><span class="s2">"https://docs.renovatebot.com/renovate-schema.json"</span><span class="p">,</span><span class="w">
  </span><span class="nl">"extends"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">"github&gt;konflux-ci/mintmaker//config/renovate/renovate.json"</span><span class="p">],</span><span class="w">
  </span><span class="nl">"tekton"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
    </span><span class="nl">"includePaths"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">"pipelines/**"</span><span class="p">]</span><span class="w"> <i class="conum" data-value="1"></i><b>(1)</b>
  </span><span class="p">}</span><span class="w">
</span><span class="p">}</span></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>By default, MintMaker only updates pipelines found in the <code>.tekton</code> folder.
To use a different location, you must specify where the remote pipelines are located.</td>
</tr>
</table>
</div>
</div>
</div>
<div class="sect1">
<h2 id="special-thanks">Special thanks</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Special thanks to <a href="https://github.com/jpopelka" target="_blank" rel="noopener">Jiri Popelka</a> for suggesting the <a href="https://pipelinesascode.com/docs/guide/resolver/#remote-pipeline-annotations" target="_blank" rel="noopener">Pipelines as Code annotation</a> as an alternative to the <a href="https://tekton.dev/docs/pipelines/git-resolver/#pipeline-resolution" target="_blank" rel="noopener">Tekton Git resolver</a>.</p>
</div>
</div>
</div>]]></content><author><name>Gwenneg Lepage</name></author><category term="konflux" /><summary type="html"><![CDATA[Maintaining Konflux pipelines doesn&#8217;t have to be a pain. I&#8217;ve got a few tips to help you out.]]></summary></entry><entry><title type="html">Why is my PostgreSQL query so slow?! Let’s fix it!</title><link href="https://gwenneg.github.io/2025/03/21/postgres-execution-time.html" rel="alternate" type="text/html" title="Why is my PostgreSQL query so slow?! Let’s fix it!" /><published>2025-03-21T00:00:00+00:00</published><updated>2025-03-21T00:00:00+00:00</updated><id>https://gwenneg.github.io/2025/03/21/postgres-execution-time</id><content type="html" xml:base="https://gwenneg.github.io/2025/03/21/postgres-execution-time.html"><![CDATA[<div id="preamble">
<div class="sectionbody">
<div class="paragraph">
<p><a href="https://www.postgresql.org/" target="_blank" rel="noopener">PostgreSQL</a> is pretty smart at running queries, but sometimes it needs a little help to hit top speed.
In this post, we&#8217;ll walk through a practical use case and explore different ways to speed up your queries.</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="the-use-case">The use case</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Imagine a centralized meteorological system that collects daily weather reports from 100 weather stations.
Each station produces 10,000 reports every day, which are stored in the database and automatically deleted after 30 days.</p>
</div>
<div class="paragraph">
<p>Here&#8217;s what the SQL schema looks like:</p>
</div>
<div class="paragraph">
<p><span class="image"><img src="/assets/images/posts/postgres-execution-time/schema.svg" alt="DB schema"></span></p>
</div>
<div class="paragraph">
<p>The system depends on two essential SQL queries that require optimal performance:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>Fetching all reports from a specific weather station starting from a given date, sorted by received time in descending order and paginated:</p>
</li>
</ul>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="sql"><span class="k">select</span> <span class="n">wr</span><span class="p">.</span><span class="n">id</span><span class="p">,</span> <span class="n">wr</span><span class="p">.</span><span class="k">data</span><span class="p">,</span> <span class="n">wr</span><span class="p">.</span><span class="n">received_at</span>
<span class="k">from</span> <span class="n">weather_report</span> <span class="n">wr</span> <span class="k">join</span> <span class="n">weather_station</span> <span class="n">ws</span> <span class="k">on</span> <span class="n">wr</span><span class="p">.</span><span class="n">weather_station_id</span> <span class="o">=</span> <span class="n">ws</span><span class="p">.</span><span class="n">id</span>
<span class="k">where</span> <span class="n">ws</span><span class="p">.</span><span class="n">name</span> <span class="o">=</span> <span class="s1">'weather-station-17'</span> <span class="k">and</span> <span class="n">wr</span><span class="p">.</span><span class="n">received_at</span> <span class="o">&gt;=</span> <span class="s1">'2025-03-06'</span>
<span class="k">order</span> <span class="k">by</span> <span class="n">wr</span><span class="p">.</span><span class="n">received_at</span> <span class="k">desc</span>
<span class="k">offset</span> <span class="mi">800</span> <span class="k">limit</span> <span class="mi">100</span><span class="p">;</span></code></pre>
</div>
</div>
<div class="ulist">
<ul>
<li>
<p>Counting all reports from a specific weather station starting from a given date:</p>
</li>
</ul>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="sql"><span class="k">select</span> <span class="k">count</span><span class="p">(</span><span class="n">wr</span><span class="p">.</span><span class="n">id</span><span class="p">)</span>
<span class="k">from</span> <span class="n">weather_report</span> <span class="n">wr</span> <span class="k">join</span> <span class="n">weather_station</span> <span class="n">ws</span> <span class="k">on</span> <span class="n">wr</span><span class="p">.</span><span class="n">weather_station_id</span> <span class="o">=</span> <span class="n">ws</span><span class="p">.</span><span class="n">id</span>
<span class="k">where</span> <span class="n">ws</span><span class="p">.</span><span class="n">name</span> <span class="o">=</span> <span class="s1">'weather-station-17'</span> <span class="k">and</span> <span class="n">wr</span><span class="p">.</span><span class="n">received_at</span> <span class="o">&gt;=</span> <span class="s1">'2025-03-06'</span><span class="p">;</span></code></pre>
</div>
</div>
<div class="paragraph">
<p>Now, let&#8217;s see how we can make these queries faster!</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="optimization-playground">Optimization playground</h2>
<div class="sectionbody">
<div class="paragraph">
<p>You&#8217;re not just reading another post about PostgreSQL query performance.
This post comes with a repository I created and used to test all the content discussed below: <a href="https://github.com/gwenneg/blog-postgres-execution-time" target="_blank" rel="noopener">gwenneg/blog-postgres-execution-time</a>.
Hopefully, this repository will make it easier for you to experiment with different optimizations, possibly using your own schema and generated data.</p>
</div>
<div class="paragraph">
<p>If you&#8217;re not interested in testing optimizations, feel free to skip this section and jump to the <a href="#query-planner">next one</a>.
Otherwise, I&#8217;ll show you how to use my repository to run your own tests.</p>
</div>
<div class="admonitionblock warning">
<table>
<tr>
<td class="icon">
<i class="fa icon-warning" title="Warning"></i>
</td>
<td class="content">
<div class="paragraph">
<p>By default, the <a href="https://github.com/gwenneg/blog-postgres-execution-time" target="_blank" rel="noopener">gwenneg/blog-postgres-execution-time</a> repository generates 60 million records during the initialization of a local PostgreSQL database.
Depending on your machine, this process can take several hours.</p>
</div>
</td>
</tr>
</table>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
<div class="paragraph">
<p><a href="https://docs.docker.com/compose/" target="_blank" rel="noopener">Docker Compose</a> is required to run the tests on your machine.
While PostgreSQL is not required, using an existing installation should work.</p>
</div>
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>Clone the <a href="https://github.com/gwenneg/blog-postgres-execution-time" target="_blank" rel="noopener">gwenneg/blog-postgres-execution-time</a> repository.
Open a terminal in either the <a href="https://github.com/gwenneg/blog-postgres-execution-time/tree/main/single-table" target="_blank" rel="noopener">single-table</a> folder (no partitions) or the <a href="https://github.com/gwenneg/blog-postgres-execution-time/tree/main/partitioned-table" target="_blank" rel="noopener">partitioned-table</a> folder (with partitions) from the repository, depending on which approach you want to test.
Then, start a container with:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="bash">docker compose up <i class="conum" data-value="1"></i><b>(1)</b></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>The execution of this command may take several hours.</td>
</tr>
</table>
</div>
<details>
<summary class="title"><strong>Click here</strong> if you run into a <code>permission denied</code> error on a SELinux-enabled system.</summary>
<div class="content">
<div class="paragraph">
<p>In a SELinux-enabled system (e.g. Fedora, CentOS, RHEL), SELinux policies may prevent the container from accessing the <code>init.sql</code> file:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="bash"><span class="o">[</span>postgres] | psql: error: /docker-entrypoint-initdb.d/init.sql: Permission denied</code></pre>
</div>
</div>
<div class="paragraph">
<p>If that happens, run the following commands:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="bash"><span class="nb">chcon</span> <span class="nt">-Rt</span> svirt_sandbox_file_t ./sql <i class="conum" data-value="1"></i><b>(1)</b>
docker compose down <span class="nt">--volumes</span> <i class="conum" data-value="2"></i><b>(2)</b>
docker compose up <i class="conum" data-value="3"></i><b>(3)</b></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>This changes the SELinux security context and grants permission to the container to access all files from the <code>./sql</code> folder.</td>
</tr>
<tr>
<td><i class="conum" data-value="2"></i><b>2</b></td>
<td>The volumes that were created with the previous <code>docker compose up</code> execution need to be removed.
Otherwise, the <code>init.sql</code> script will not be rerun.</td>
</tr>
<tr>
<td><i class="conum" data-value="3"></i><b>3</b></td>
<td>The execution of this command may take several hours.</td>
</tr>
</table>
</div>
</div>
</details>
<div class="paragraph">
<p>The database initialization is complete when the following message appears:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="bash"><span class="o">[</span>postgres] | 2025-03-21 13:28:21.316 UTC <span class="o">[</span>1] LOG:  database system is ready to accept connections</code></pre>
</div>
</div>
<div class="paragraph">
<p>You can now connect to the database.</p>
</div>
<details>
<summary class="title"><strong>Click here</strong> if PostgreSQL is not installed on your machine.</summary>
<div class="content">
<div class="paragraph">
<p>First, identify the PostgreSQL container ID using <code>docker ps</code>.
Then, enter the container with the following command:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="bash">docker <span class="nb">exec</span> <span class="nt">-it</span> 44086e358596 bash <i class="conum" data-value="1"></i><b>(1)</b></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td><code>44086e358596</code> is the container ID returned by <code>docker ps</code>.</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>Now that you&#8217;re in the container, it&#8217;s time to connect to PostgreSQL:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="bash">psql <span class="nt">-h</span> localhost <span class="nt">-U</span> postgres</code></pre>
</div>
</div>
</div>
</details>
<details>
<summary class="title"><strong>Click here</strong> to use <code>psql</code> from an existing PostgreSQL installation.</summary>
<div class="content">
<div class="paragraph">
<p>Run the following command from the current folder:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="bash">psql <span class="nt">-h</span> localhost <span class="nt">-p</span> 15432 <span class="nt">-U</span> postgres <i class="conum" data-value="1"></i><b>(1)</b></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>When prompted, enter the password: <code>postgres</code>.</td>
</tr>
</table>
</div>
</div>
</details>
<div class="paragraph">
<p>Congrats! You&#8217;re now ready to run the SQL scripts provided in the repository or any other SQL queries:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code>\i sql/explain_analyze.sql <i class="conum" data-value="1"></i><b>(1)</b></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>If you&#8217;re connected to PostgreSQL from within the container, the path is <code>/mnt/sql/explain_analyze.sql</code>.</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>These are the SQL scripts included in the repository:</p>
</div>
<table class="tableblock frame-all grid-all stretch">
<colgroup>
<col style="width: 25%;">
<col style="width: 75%;">
</colgroup>
<thead>
<tr>
<th class="tableblock halign-left valign-top">File name</th>
<th class="tableblock halign-left valign-top">Description</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">create_indexes.sql</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Adds indexes to the <code>weather_report</code> table.</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">explain_analyze.sql</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Displays the execution plan of the "fetch" and "count" queries.</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">init.sql</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">DO NOT RUN MANUALLY - Creates the database schema and generates data at container startup.</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">relations_size.sql</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Displays the disk usage of the <code>weather_report</code> table and all associated indexes.</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">vacuum_analyze.sql</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Performs <code>VACUUM</code> and updates statistics on the <code>weather_report</code> and <code>weather_station</code> tables.</p></td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="sect1">
<h2 id="the-postgresql-query-planner"><a id="query-planner"></a> The PostgreSQL query planner</h2>
<div class="sectionbody">
<div class="paragraph">
<p>When you run a SQL query in PostgreSQL, it isn&#8217;t executed immediately.
First, the <a href="https://www.postgresql.org/docs/current/planner-optimizer.html" target="_blank" rel="noopener">query planner</a> analyzes the query structure, table sizes, indexes, joins, filtering conditions and available statistics to generate multiple execution plans.
Then, PostgreSQL selects the most efficient plan and executes the query.</p>
</div>
<div class="paragraph">
<p>The <a href="https://www.postgresql.org/docs/current/sql-explain.html" target="_blank" rel="noopener">EXPLAIN</a> command displays the execution plan for a given query, providing details on execution time, row counts, indexes usage and more.
That command is crucial for identifying slow queries, missing indexes, inefficient joins or outdated statistics.
We&#8217;ll rely on it heavily throughout this post.</p>
</div>
<div class="admonitionblock tip">
<table>
<tr>
<td class="icon">
<i class="fa icon-tip" title="Tip"></i>
</td>
<td class="content">
<div class="paragraph">
<p>When testing query optimizations, use <a href="https://www.postgresql.org/docs/current/sql-explain.html" target="_blank" rel="noopener">EXPLAIN</a> before and after making changes.
This will help you determine whether the optimization was effective.</p>
</div>
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>Reading an execution plan used to require highly specialized knowledge before the rise of Large Language Models.
Today, if you submit your execution plan to an LLM, it can help identify weaknesses and suggest fixes.
However, as with any LLM, be cautious of hallucinations and always double-check its recommendations.</p>
</div>
<div class="paragraph">
<p>Besides LLMs, tools like <a href="https://explain.dalibo.com" target="_blank" rel="noopener">explain.dalibo.com</a> can also help you visualize and understand your execution plan:</p>
</div>
<div class="paragraph">
<p><span class="image"><img src="/assets/images/posts/postgres-execution-time/dalibo.png" alt="Visualizing an execution plan with Dalibo"></span></p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="explaining-the-sql-queries-from-the-use-case">Explaining the SQL queries from the use case</h2>
<div class="sectionbody">
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
<div class="paragraph">
<p>Most of this post is based on the <a href="https://github.com/gwenneg/blog-postgres-execution-time/tree/main/single-table" target="_blank" rel="noopener">single-table</a> folder from the <a href="https://github.com/gwenneg/blog-postgres-execution-time" target="_blank" rel="noopener">gwenneg/blog-postgres-execution-time</a> repository.
The <a href="https://github.com/gwenneg/blog-postgres-execution-time/tree/main/partitioned-table" target="_blank" rel="noopener">partitioned-table</a> folder is only used in the <a href="#partitioning">Partitioning the <code>weather_report</code> table</a> section below.</p>
</div>
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>Let&#8217;s see what the execution plans for our two essential queries look like without indexes (except for primary keys) and using the default PostgreSQL settings.</p>
</div>
<div class="listingblock">
<div class="title">Explaining the "fetch" query</div>
<div class="content">
<pre class="rouge highlight"><code data-lang="sql"><span class="k">explain</span> <span class="k">analyze</span> <i class="conum" data-value="1"></i><b>(1)</b>
<span class="k">select</span> <span class="n">wr</span><span class="p">.</span><span class="n">id</span><span class="p">,</span> <span class="n">wr</span><span class="p">.</span><span class="k">data</span><span class="p">,</span> <span class="n">wr</span><span class="p">.</span><span class="n">received_at</span>
<span class="k">from</span> <span class="n">weather_report</span> <span class="n">wr</span> <span class="k">join</span> <span class="n">weather_station</span> <span class="n">ws</span> <span class="k">on</span> <span class="n">wr</span><span class="p">.</span><span class="n">weather_station_id</span> <span class="o">=</span> <span class="n">ws</span><span class="p">.</span><span class="n">id</span>
<span class="k">where</span> <span class="n">ws</span><span class="p">.</span><span class="n">name</span> <span class="o">=</span> <span class="s1">'weather-station-17'</span> <span class="k">and</span> <span class="n">wr</span><span class="p">.</span><span class="n">received_at</span> <span class="o">&gt;=</span> <span class="s1">'2025-03-06'</span>
<span class="k">order</span> <span class="k">by</span> <span class="n">wr</span><span class="p">.</span><span class="n">received_at</span> <span class="k">desc</span>
<span class="k">offset</span> <span class="mi">800</span> <span class="k">limit</span> <span class="mi">100</span><span class="p">;</span></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>When the <code>ANALYZE</code> option is used, PostgreSQL provides additional details including the actual execution times.</td>
</tr>
</table>
</div>
<div class="admonitionblock warning">
<table>
<tr>
<td class="icon">
<i class="fa icon-warning" title="Warning"></i>
</td>
<td class="content">
<div class="paragraph">
<p>When <code>ANALYZE</code> is used, the SQL query is actually executed and modifies the DB data.
If you need to <code>EXPLAIN ANALYZE</code> an <code>INSERT</code> query or any other query that modifies the data, you should wrap the <code>EXPLAIN</code> statement into a transaction and end it with a <code>ROLLBACK</code>.</p>
</div>
</td>
</tr>
</table>
</div>
<details>
<summary class="title"><strong>Click here</strong> to see the execution plan of the "fetch" query.</summary>
<div class="content">
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="sql"> <span class="k">Limit</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">980048</span><span class="p">.</span><span class="mi">43</span><span class="p">..</span><span class="mi">980060</span><span class="p">.</span><span class="mi">09</span> <span class="k">rows</span><span class="o">=</span><span class="mi">100</span> <span class="n">width</span><span class="o">=</span><span class="mi">57</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">2063</span><span class="p">.</span><span class="mi">296</span><span class="p">..</span><span class="mi">2067</span><span class="p">.</span><span class="mi">738</span> <span class="k">rows</span><span class="o">=</span><span class="mi">100</span> <span class="n">loops</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
   <span class="o">-&gt;</span>  <span class="n">Gather</span> <span class="n">Merge</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">979955</span><span class="p">.</span><span class="mi">09</span><span class="p">..</span><span class="mi">995587</span><span class="p">.</span><span class="mi">64</span> <span class="k">rows</span><span class="o">=</span><span class="mi">133984</span> <span class="n">width</span><span class="o">=</span><span class="mi">57</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">1984</span><span class="p">.</span><span class="mi">690</span><span class="p">..</span><span class="mi">1989</span><span class="p">.</span><span class="mi">270</span> <span class="k">rows</span><span class="o">=</span><span class="mi">900</span> <span class="n">loops</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
         <span class="n">Workers</span> <span class="n">Planned</span><span class="p">:</span> <span class="mi">2</span>
         <span class="n">Workers</span> <span class="n">Launched</span><span class="p">:</span> <span class="mi">2</span> <i class="conum" data-value="1"></i><b>(1)</b>
         <span class="o">-&gt;</span>  <span class="n">Sort</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">978955</span><span class="p">.</span><span class="mi">06</span><span class="p">..</span><span class="mi">979122</span><span class="p">.</span><span class="mi">54</span> <span class="k">rows</span><span class="o">=</span><span class="mi">66992</span> <span class="n">width</span><span class="o">=</span><span class="mi">57</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">1944</span><span class="p">.</span><span class="mi">160</span><span class="p">..</span><span class="mi">1944</span><span class="p">.</span><span class="mi">194</span> <span class="k">rows</span><span class="o">=</span><span class="mi">687</span> <span class="n">loops</span><span class="o">=</span><span class="mi">3</span><span class="p">)</span>
               <span class="n">Sort</span> <span class="k">Key</span><span class="p">:</span> <span class="n">wr</span><span class="p">.</span><span class="n">received_at</span> <span class="k">DESC</span>
               <span class="n">Sort</span> <span class="k">Method</span><span class="p">:</span> <span class="n">top</span><span class="o">-</span><span class="n">N</span> <span class="n">heapsort</span>  <span class="n">Memory</span><span class="p">:</span> <span class="mi">288</span><span class="n">kB</span> <i class="conum" data-value="2"></i><b>(2)</b>
               <span class="n">Worker</span> <span class="mi">0</span><span class="p">:</span>  <span class="n">Sort</span> <span class="k">Method</span><span class="p">:</span> <span class="n">top</span><span class="o">-</span><span class="n">N</span> <span class="n">heapsort</span>  <span class="n">Memory</span><span class="p">:</span> <span class="mi">288</span><span class="n">kB</span>
               <span class="n">Worker</span> <span class="mi">1</span><span class="p">:</span>  <span class="n">Sort</span> <span class="k">Method</span><span class="p">:</span> <span class="n">top</span><span class="o">-</span><span class="n">N</span> <span class="n">heapsort</span>  <span class="n">Memory</span><span class="p">:</span> <span class="mi">288</span><span class="n">kB</span>
               <span class="o">-&gt;</span>  <span class="n">Hash</span> <span class="k">Join</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">2</span><span class="p">.</span><span class="mi">26</span><span class="p">..</span><span class="mi">975332</span><span class="p">.</span><span class="mi">88</span> <span class="k">rows</span><span class="o">=</span><span class="mi">66992</span> <span class="n">width</span><span class="o">=</span><span class="mi">57</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">873</span><span class="p">.</span><span class="mi">837</span><span class="p">..</span><span class="mi">1927</span><span class="p">.</span><span class="mi">942</span> <span class="k">rows</span><span class="o">=</span><span class="mi">53333</span> <span class="n">loops</span><span class="o">=</span><span class="mi">3</span><span class="p">)</span>
                     <span class="n">Hash</span> <span class="n">Cond</span><span class="p">:</span> <span class="p">(</span><span class="n">wr</span><span class="p">.</span><span class="n">weather_station_id</span> <span class="o">=</span> <span class="n">ws</span><span class="p">.</span><span class="n">id</span><span class="p">)</span>
                     <span class="o">-&gt;</span>  <span class="n">Parallel</span> <span class="n">Seq</span> <span class="n">Scan</span> <span class="k">on</span> <span class="n">weather_report</span> <span class="n">wr</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">00</span><span class="p">..</span><span class="mi">957000</span><span class="p">.</span><span class="mi">00</span> <span class="k">rows</span><span class="o">=</span><span class="mi">6699173</span> <span class="n">width</span><span class="o">=</span><span class="mi">73</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">873</span><span class="p">.</span><span class="mi">701</span><span class="p">..</span><span class="mi">1494</span><span class="p">.</span><span class="mi">414</span> <span class="k">rows</span><span class="o">=</span><span class="mi">5333333</span> <span class="n">loops</span><span class="o">=</span><span class="mi">3</span><span class="p">)</span> <i class="conum" data-value="3"></i><b>(3)</b>
                           <span class="n">Filter</span><span class="p">:</span> <span class="p">(</span><span class="n">received_at</span> <span class="o">&gt;=</span> <span class="s1">'2025-03-06 00:00:00'</span><span class="p">::</span><span class="nb">timestamp</span> <span class="k">without</span> <span class="nb">time</span> <span class="k">zone</span><span class="p">)</span>
                           <span class="k">Rows</span> <span class="n">Removed</span> <span class="k">by</span> <span class="n">Filter</span><span class="p">:</span> <span class="mi">4666667</span>
                     <span class="o">-&gt;</span>  <span class="n">Hash</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">2</span><span class="p">.</span><span class="mi">25</span><span class="p">..</span><span class="mi">2</span><span class="p">.</span><span class="mi">25</span> <span class="k">rows</span><span class="o">=</span><span class="mi">1</span> <span class="n">width</span><span class="o">=</span><span class="mi">16</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">054</span><span class="p">..</span><span class="mi">0</span><span class="p">.</span><span class="mi">054</span> <span class="k">rows</span><span class="o">=</span><span class="mi">1</span> <span class="n">loops</span><span class="o">=</span><span class="mi">3</span><span class="p">)</span>
                           <span class="n">Buckets</span><span class="p">:</span> <span class="mi">1024</span>  <span class="n">Batches</span><span class="p">:</span> <span class="mi">1</span>  <span class="n">Memory</span> <span class="k">Usage</span><span class="p">:</span> <span class="mi">9</span><span class="n">kB</span>
                           <span class="o">-&gt;</span>  <span class="n">Seq</span> <span class="n">Scan</span> <span class="k">on</span> <span class="n">weather_station</span> <span class="n">ws</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">00</span><span class="p">..</span><span class="mi">2</span><span class="p">.</span><span class="mi">25</span> <span class="k">rows</span><span class="o">=</span><span class="mi">1</span> <span class="n">width</span><span class="o">=</span><span class="mi">16</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">040</span><span class="p">..</span><span class="mi">0</span><span class="p">.</span><span class="mi">045</span> <span class="k">rows</span><span class="o">=</span><span class="mi">1</span> <span class="n">loops</span><span class="o">=</span><span class="mi">3</span><span class="p">)</span>
                                 <span class="n">Filter</span><span class="p">:</span> <span class="p">(</span><span class="n">name</span> <span class="o">=</span> <span class="s1">'weather-station-17'</span><span class="p">::</span><span class="nb">text</span><span class="p">)</span>
                                 <span class="k">Rows</span> <span class="n">Removed</span> <span class="k">by</span> <span class="n">Filter</span><span class="p">:</span> <span class="mi">99</span>
 <span class="n">Planning</span> <span class="nb">Time</span><span class="p">:</span> <span class="mi">0</span><span class="p">.</span><span class="mi">200</span> <span class="n">ms</span>
 <span class="n">JIT</span><span class="p">:</span>
   <span class="n">Functions</span><span class="p">:</span> <span class="mi">44</span>
   <span class="k">Options</span><span class="p">:</span> <span class="n">Inlining</span> <span class="k">true</span><span class="p">,</span> <span class="n">Optimization</span> <span class="k">true</span><span class="p">,</span> <span class="n">Expressions</span> <span class="k">true</span><span class="p">,</span> <span class="n">Deforming</span> <span class="k">true</span>
   <span class="n">Timing</span><span class="p">:</span> <span class="n">Generation</span> <span class="mi">1</span><span class="p">.</span><span class="mi">955</span> <span class="n">ms</span> <span class="p">(</span><span class="n">Deform</span> <span class="mi">0</span><span class="p">.</span><span class="mi">951</span> <span class="n">ms</span><span class="p">),</span> <span class="n">Inlining</span> <span class="mi">212</span><span class="p">.</span><span class="mi">662</span> <span class="n">ms</span><span class="p">,</span> <span class="n">Optimization</span> <span class="mi">142</span><span class="p">.</span><span class="mi">882</span> <span class="n">ms</span><span class="p">,</span> <span class="n">Emission</span> <span class="mi">131</span><span class="p">.</span><span class="mi">128</span> <span class="n">ms</span><span class="p">,</span> <span class="n">Total</span> <span class="mi">488</span><span class="p">.</span><span class="mi">627</span> <span class="n">ms</span>
 <span class="n">Execution</span> <span class="nb">Time</span><span class="p">:</span> <span class="mi">2068</span><span class="p">.</span><span class="mi">703</span> <span class="n">ms</span> <i class="conum" data-value="4"></i><b>(4)</b></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>The number of workers varies depending on the available CPU cores and the PostgreSQL configuration.</td>
</tr>
<tr>
<td><i class="conum" data-value="2"></i><b>2</b></td>
<td>Sorting all matching rows using <code>top-N heapsort</code> is expensive.</td>
</tr>
<tr>
<td><i class="conum" data-value="3"></i><b>3</b></td>
<td>A <a href="https://www.postgresql.org/docs/current/parallel-plans.html#PARALLEL-SCANS" target="_blank" rel="noopener">parallel sequential scan</a> on 30 million rows is a major bottleneck.</td>
</tr>
<tr>
<td><i class="conum" data-value="4"></i><b>4</b></td>
<td>This is the execution time of the query.</td>
</tr>
</table>
</div>
</div>
</details>
<div class="listingblock">
<div class="title">Explaining the "count" query</div>
<div class="content">
<pre class="rouge highlight"><code data-lang="sql"><span class="k">explain</span> <span class="k">analyze</span>
<span class="k">select</span> <span class="k">count</span><span class="p">(</span><span class="n">wr</span><span class="p">.</span><span class="n">id</span><span class="p">)</span>
<span class="k">from</span> <span class="n">weather_report</span> <span class="n">wr</span> <span class="k">join</span> <span class="n">weather_station</span> <span class="n">ws</span> <span class="k">on</span> <span class="n">wr</span><span class="p">.</span><span class="n">weather_station_id</span> <span class="o">=</span> <span class="n">ws</span><span class="p">.</span><span class="n">id</span>
<span class="k">where</span> <span class="n">ws</span><span class="p">.</span><span class="n">name</span> <span class="o">=</span> <span class="s1">'weather-station-17'</span> <span class="k">and</span> <span class="n">wr</span><span class="p">.</span><span class="n">received_at</span> <span class="o">&gt;=</span> <span class="s1">'2025-03-06'</span><span class="p">;</span></code></pre>
</div>
</div>
<details>
<summary class="title"><strong>Click here</strong> to see the execution plan of the "count" query.</summary>
<div class="content">
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="sql"> <span class="n">Finalize</span> <span class="k">Aggregate</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">976500</span><span class="p">.</span><span class="mi">57</span><span class="p">..</span><span class="mi">976500</span><span class="p">.</span><span class="mi">58</span> <span class="k">rows</span><span class="o">=</span><span class="mi">1</span> <span class="n">width</span><span class="o">=</span><span class="mi">8</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">2029</span><span class="p">.</span><span class="mi">976</span><span class="p">..</span><span class="mi">2034</span><span class="p">.</span><span class="mi">088</span> <span class="k">rows</span><span class="o">=</span><span class="mi">1</span> <span class="n">loops</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
   <span class="o">-&gt;</span>  <span class="n">Gather</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">976500</span><span class="p">.</span><span class="mi">36</span><span class="p">..</span><span class="mi">976500</span><span class="p">.</span><span class="mi">57</span> <span class="k">rows</span><span class="o">=</span><span class="mi">2</span> <span class="n">width</span><span class="o">=</span><span class="mi">8</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">2029</span><span class="p">.</span><span class="mi">833</span><span class="p">..</span><span class="mi">2034</span><span class="p">.</span><span class="mi">071</span> <span class="k">rows</span><span class="o">=</span><span class="mi">3</span> <span class="n">loops</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
         <span class="n">Workers</span> <span class="n">Planned</span><span class="p">:</span> <span class="mi">2</span>
         <span class="n">Workers</span> <span class="n">Launched</span><span class="p">:</span> <span class="mi">2</span> <i class="conum" data-value="1"></i><b>(1)</b>
         <span class="o">-&gt;</span>  <span class="k">Partial</span> <span class="k">Aggregate</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">975500</span><span class="p">.</span><span class="mi">36</span><span class="p">..</span><span class="mi">975500</span><span class="p">.</span><span class="mi">37</span> <span class="k">rows</span><span class="o">=</span><span class="mi">1</span> <span class="n">width</span><span class="o">=</span><span class="mi">8</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">2013</span><span class="p">.</span><span class="mi">942</span><span class="p">..</span><span class="mi">2013</span><span class="p">.</span><span class="mi">943</span> <span class="k">rows</span><span class="o">=</span><span class="mi">1</span> <span class="n">loops</span><span class="o">=</span><span class="mi">3</span><span class="p">)</span>
               <span class="o">-&gt;</span>  <span class="n">Hash</span> <span class="k">Join</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">2</span><span class="p">.</span><span class="mi">26</span><span class="p">..</span><span class="mi">975332</span><span class="p">.</span><span class="mi">88</span> <span class="k">rows</span><span class="o">=</span><span class="mi">66992</span> <span class="n">width</span><span class="o">=</span><span class="mi">16</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">888</span><span class="p">.</span><span class="mi">852</span><span class="p">..</span><span class="mi">2011</span><span class="p">.</span><span class="mi">411</span> <span class="k">rows</span><span class="o">=</span><span class="mi">53333</span> <span class="n">loops</span><span class="o">=</span><span class="mi">3</span><span class="p">)</span>
                     <span class="n">Hash</span> <span class="n">Cond</span><span class="p">:</span> <span class="p">(</span><span class="n">wr</span><span class="p">.</span><span class="n">weather_station_id</span> <span class="o">=</span> <span class="n">ws</span><span class="p">.</span><span class="n">id</span><span class="p">)</span>
                     <span class="o">-&gt;</span>  <span class="n">Parallel</span> <span class="n">Seq</span> <span class="n">Scan</span> <span class="k">on</span> <span class="n">weather_report</span> <span class="n">wr</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">00</span><span class="p">..</span><span class="mi">957000</span><span class="p">.</span><span class="mi">00</span> <span class="k">rows</span><span class="o">=</span><span class="mi">6699173</span> <span class="n">width</span><span class="o">=</span><span class="mi">32</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">888</span><span class="p">.</span><span class="mi">705</span><span class="p">..</span><span class="mi">1508</span><span class="p">.</span><span class="mi">554</span> <span class="k">rows</span><span class="o">=</span><span class="mi">5333333</span> <span class="n">loops</span><span class="o">=</span><span class="mi">3</span><span class="p">)</span> <i class="conum" data-value="2"></i><b>(2)</b>
                           <span class="n">Filter</span><span class="p">:</span> <span class="p">(</span><span class="n">received_at</span> <span class="o">&gt;=</span> <span class="s1">'2025-03-06 00:00:00'</span><span class="p">::</span><span class="nb">timestamp</span> <span class="k">without</span> <span class="nb">time</span> <span class="k">zone</span><span class="p">)</span>
                           <span class="k">Rows</span> <span class="n">Removed</span> <span class="k">by</span> <span class="n">Filter</span><span class="p">:</span> <span class="mi">4666667</span>
                     <span class="o">-&gt;</span>  <span class="n">Hash</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">2</span><span class="p">.</span><span class="mi">25</span><span class="p">..</span><span class="mi">2</span><span class="p">.</span><span class="mi">25</span> <span class="k">rows</span><span class="o">=</span><span class="mi">1</span> <span class="n">width</span><span class="o">=</span><span class="mi">16</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">042</span><span class="p">..</span><span class="mi">0</span><span class="p">.</span><span class="mi">043</span> <span class="k">rows</span><span class="o">=</span><span class="mi">1</span> <span class="n">loops</span><span class="o">=</span><span class="mi">3</span><span class="p">)</span>
                           <span class="n">Buckets</span><span class="p">:</span> <span class="mi">1024</span>  <span class="n">Batches</span><span class="p">:</span> <span class="mi">1</span>  <span class="n">Memory</span> <span class="k">Usage</span><span class="p">:</span> <span class="mi">9</span><span class="n">kB</span>
                           <span class="o">-&gt;</span>  <span class="n">Seq</span> <span class="n">Scan</span> <span class="k">on</span> <span class="n">weather_station</span> <span class="n">ws</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">00</span><span class="p">..</span><span class="mi">2</span><span class="p">.</span><span class="mi">25</span> <span class="k">rows</span><span class="o">=</span><span class="mi">1</span> <span class="n">width</span><span class="o">=</span><span class="mi">16</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">033</span><span class="p">..</span><span class="mi">0</span><span class="p">.</span><span class="mi">037</span> <span class="k">rows</span><span class="o">=</span><span class="mi">1</span> <span class="n">loops</span><span class="o">=</span><span class="mi">3</span><span class="p">)</span>
                                 <span class="n">Filter</span><span class="p">:</span> <span class="p">(</span><span class="n">name</span> <span class="o">=</span> <span class="s1">'weather-station-17'</span><span class="p">::</span><span class="nb">text</span><span class="p">)</span>
                                 <span class="k">Rows</span> <span class="n">Removed</span> <span class="k">by</span> <span class="n">Filter</span><span class="p">:</span> <span class="mi">99</span>
 <span class="n">Planning</span> <span class="nb">Time</span><span class="p">:</span> <span class="mi">0</span><span class="p">.</span><span class="mi">141</span> <span class="n">ms</span>
 <span class="n">JIT</span><span class="p">:</span>
   <span class="n">Functions</span><span class="p">:</span> <span class="mi">50</span>
   <span class="k">Options</span><span class="p">:</span> <span class="n">Inlining</span> <span class="k">true</span><span class="p">,</span> <span class="n">Optimization</span> <span class="k">true</span><span class="p">,</span> <span class="n">Expressions</span> <span class="k">true</span><span class="p">,</span> <span class="n">Deforming</span> <span class="k">true</span>
   <span class="n">Timing</span><span class="p">:</span> <span class="n">Generation</span> <span class="mi">1</span><span class="p">.</span><span class="mi">781</span> <span class="n">ms</span> <span class="p">(</span><span class="n">Deform</span> <span class="mi">0</span><span class="p">.</span><span class="mi">765</span> <span class="n">ms</span><span class="p">),</span> <span class="n">Inlining</span> <span class="mi">211</span><span class="p">.</span><span class="mi">423</span> <span class="n">ms</span><span class="p">,</span> <span class="n">Optimization</span> <span class="mi">142</span><span class="p">.</span><span class="mi">079</span> <span class="n">ms</span><span class="p">,</span> <span class="n">Emission</span> <span class="mi">162</span><span class="p">.</span><span class="mi">995</span> <span class="n">ms</span><span class="p">,</span> <span class="n">Total</span> <span class="mi">518</span><span class="p">.</span><span class="mi">278</span> <span class="n">ms</span>
 <span class="n">Execution</span> <span class="nb">Time</span><span class="p">:</span> <span class="mi">2034</span><span class="p">.</span><span class="mi">732</span> <span class="n">ms</span> <i class="conum" data-value="3"></i><b>(3)</b></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>The number of workers varies depending on the available CPU cores and the PostgreSQL configuration.</td>
</tr>
<tr>
<td><i class="conum" data-value="2"></i><b>2</b></td>
<td>A <a href="https://www.postgresql.org/docs/current/parallel-plans.html#PARALLEL-SCANS" target="_blank" rel="noopener">parallel sequential scan</a> on 30 million rows is a major bottleneck.</td>
</tr>
<tr>
<td><i class="conum" data-value="3"></i><b>3</b></td>
<td>This is the execution time of the query.</td>
</tr>
</table>
</div>
</div>
</details>
<div class="paragraph">
<p>More than 2 seconds to run each query - that doesn&#8217;t look good, right?
But it&#8217;s no surprise since the <code>weather_report</code> table contains 30 million records and we&#8217;re filtering on unindexed columns.</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="indexing-the-weather_report-table">Indexing the <code>weather_report</code> table</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Our queries both include a condition on the <code>received_at</code> and <code>weather_station_id</code> columns from the <code>weather_report</code> table, which contains 30 million records.
Indexing these columns should help speed up the queries.</p>
</div>
<div class="admonitionblock tip">
<table>
<tr>
<td class="icon">
<i class="fa icon-tip" title="Tip"></i>
</td>
<td class="content">
<div class="paragraph">
<p>If you create a composite <a href="https://www.postgresql.org/docs/current/indexes-types.html#INDEXES-TYPES-BTREE" target="_blank" rel="noopener">B-Tree index</a> (the default index type in PostgreSQL) with multiple columns, their order matters and can impact query performance.
The best column order depends on how your query filters, sorts or joins data.
So how do you figure out which order works best?
A good rule of thumb is to put the column that filters out the most rows — in other words, the one with the highest cardinality — first.
In a local environment, you can also take a trial-and-error approach by creating different index orders and using <code>EXPLAIN ANALYZE</code> to see which one the query planner prefers.</p>
</div>
</td>
</tr>
</table>
</div>
<div class="sect2">
<h3 id="introducing-non-covering-b-tree-indexes">Introducing non-covering B-Tree indexes</h3>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
<div class="paragraph">
<p>A non-covering index is an index that does not include all the columns needed to satisfy a query.
As a result, PostgreSQL must perform extra lookups in the table (heap) to retrieve missing column values.</p>
</div>
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>Let&#8217;s add the following indexes and see how they impact the execution plans.</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="sql"><span class="k">create</span> <span class="k">index</span> <span class="n">ix_btree_received_at_weather_station_id_non_covering</span>
<span class="k">on</span> <span class="n">weather_report</span> <span class="k">using</span> <span class="n">btree</span> <span class="p">(</span><span class="n">received_at</span> <span class="k">desc</span><span class="p">,</span> <span class="n">weather_station_id</span><span class="p">);</span> <i class="conum" data-value="1"></i><b>(1)</b></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td><code>using btree</code> can be omitted because that&#8217;s the default index type in PostgreSQL.</td>
</tr>
</table>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="sql"><span class="k">create</span> <span class="k">index</span> <span class="n">ix_btree_weather_station_id_received_at_non_covering</span>
<span class="k">on</span> <span class="n">weather_report</span> <span class="k">using</span> <span class="n">btree</span> <span class="p">(</span><span class="n">weather_station_id</span><span class="p">,</span> <span class="n">received_at</span> <span class="k">desc</span><span class="p">);</span></code></pre>
</div>
</div>
<div class="admonitionblock tip">
<table>
<tr>
<td class="icon">
<i class="fa icon-tip" title="Tip"></i>
</td>
<td class="content">
<div class="paragraph">
<p>If a column is mostly queried in descending order, indexing it with <code>DESC</code> helps avoid reverse index scans and reduces sorting overhead, effectively improving query performance.</p>
</div>
</td>
</tr>
</table>
</div>
<div class="listingblock">
<div class="title">Execution plan of the "fetch" query with a non-covering index</div>
<div class="content">
<pre class="rouge highlight"><code data-lang="sql"> <span class="k">Limit</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">69479</span><span class="p">.</span><span class="mi">90</span><span class="p">..</span><span class="mi">78164</span><span class="p">.</span><span class="mi">82</span> <span class="k">rows</span><span class="o">=</span><span class="mi">100</span> <span class="n">width</span><span class="o">=</span><span class="mi">57</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">67</span><span class="p">.</span><span class="mi">339</span><span class="p">..</span><span class="mi">70</span><span class="p">.</span><span class="mi">740</span> <span class="k">rows</span><span class="o">=</span><span class="mi">100</span> <span class="n">loops</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
   <span class="o">-&gt;</span>  <span class="n">Nested</span> <span class="n">Loop</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">56</span><span class="p">..</span><span class="mi">13831166</span><span class="p">.</span><span class="mi">26</span> <span class="k">rows</span><span class="o">=</span><span class="mi">159255</span> <span class="n">width</span><span class="o">=</span><span class="mi">57</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">1</span><span class="p">.</span><span class="mi">021</span><span class="p">..</span><span class="mi">70</span><span class="p">.</span><span class="mi">693</span> <span class="k">rows</span><span class="o">=</span><span class="mi">900</span> <span class="n">loops</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
         <span class="k">Join</span> <span class="n">Filter</span><span class="p">:</span> <span class="p">(</span><span class="n">wr</span><span class="p">.</span><span class="n">weather_station_id</span> <span class="o">=</span> <span class="n">ws</span><span class="p">.</span><span class="n">id</span><span class="p">)</span>
         <span class="k">Rows</span> <span class="n">Removed</span> <span class="k">by</span> <span class="k">Join</span> <span class="n">Filter</span><span class="p">:</span> <span class="mi">89092</span>
         <span class="o">-&gt;</span>  <span class="k">Index</span> <span class="n">Scan</span> <span class="k">using</span> <span class="n">ix_btree_received_at_weather_station_id_non_covering</span> <span class="k">on</span> <span class="n">weather_report</span> <span class="n">wr</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">56</span><span class="p">..</span><span class="mi">13592281</span><span class="p">.</span><span class="mi">74</span> <span class="k">rows</span><span class="o">=</span><span class="mi">15925485</span> <span class="n">width</span><span class="o">=</span><span class="mi">73</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">545</span><span class="p">..</span><span class="mi">51</span><span class="p">.</span><span class="mi">906</span> <span class="k">rows</span><span class="o">=</span><span class="mi">89992</span> <span class="n">loops</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span> <i class="conum" data-value="1"></i><b>(1)</b>
               <span class="k">Index</span> <span class="n">Cond</span><span class="p">:</span> <span class="p">(</span><span class="n">received_at</span> <span class="o">&gt;=</span> <span class="s1">'2025-03-06 00:00:00'</span><span class="p">::</span><span class="nb">timestamp</span> <span class="k">without</span> <span class="nb">time</span> <span class="k">zone</span><span class="p">)</span>
         <span class="o">-&gt;</span>  <span class="n">Materialize</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">00</span><span class="p">..</span><span class="mi">2</span><span class="p">.</span><span class="mi">25</span> <span class="k">rows</span><span class="o">=</span><span class="mi">1</span> <span class="n">width</span><span class="o">=</span><span class="mi">16</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">000</span><span class="p">..</span><span class="mi">0</span><span class="p">.</span><span class="mi">000</span> <span class="k">rows</span><span class="o">=</span><span class="mi">1</span> <span class="n">loops</span><span class="o">=</span><span class="mi">89992</span><span class="p">)</span>
               <span class="o">-&gt;</span>  <span class="n">Seq</span> <span class="n">Scan</span> <span class="k">on</span> <span class="n">weather_station</span> <span class="n">ws</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">00</span><span class="p">..</span><span class="mi">2</span><span class="p">.</span><span class="mi">25</span> <span class="k">rows</span><span class="o">=</span><span class="mi">1</span> <span class="n">width</span><span class="o">=</span><span class="mi">16</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">026</span><span class="p">..</span><span class="mi">0</span><span class="p">.</span><span class="mi">044</span> <span class="k">rows</span><span class="o">=</span><span class="mi">1</span> <span class="n">loops</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
                     <span class="n">Filter</span><span class="p">:</span> <span class="p">(</span><span class="n">name</span> <span class="o">=</span> <span class="s1">'weather-station-17'</span><span class="p">::</span><span class="nb">text</span><span class="p">)</span>
                     <span class="k">Rows</span> <span class="n">Removed</span> <span class="k">by</span> <span class="n">Filter</span><span class="p">:</span> <span class="mi">99</span>
 <span class="n">Planning</span> <span class="nb">Time</span><span class="p">:</span> <span class="mi">10</span><span class="p">.</span><span class="mi">100</span> <span class="n">ms</span>
 <span class="n">Execution</span> <span class="nb">Time</span><span class="p">:</span> <span class="mi">70</span><span class="p">.</span><span class="mi">824</span> <span class="n">ms</span></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>The previous parallel sequential scan was replaced with an <a href="https://www.postgresql.org/docs/current/index-scanning.html" target="_blank" rel="noopener">index scan</a> which is much faster.</td>
</tr>
</table>
</div>
<div class="listingblock">
<div class="title">Execution plan of the "count" query with a non-covering index</div>
<div class="content">
<pre class="rouge highlight"><code data-lang="sql"> <span class="k">Aggregate</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">587672</span><span class="p">.</span><span class="mi">27</span><span class="p">..</span><span class="mi">587672</span><span class="p">.</span><span class="mi">28</span> <span class="k">rows</span><span class="o">=</span><span class="mi">1</span> <span class="n">width</span><span class="o">=</span><span class="mi">8</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">452</span><span class="p">.</span><span class="mi">095</span><span class="p">..</span><span class="mi">452</span><span class="p">.</span><span class="mi">096</span> <span class="k">rows</span><span class="o">=</span><span class="mi">1</span> <span class="n">loops</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
   <span class="o">-&gt;</span>  <span class="n">Nested</span> <span class="n">Loop</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">56</span><span class="p">..</span><span class="mi">587274</span><span class="p">.</span><span class="mi">13</span> <span class="k">rows</span><span class="o">=</span><span class="mi">159255</span> <span class="n">width</span><span class="o">=</span><span class="mi">16</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">41</span><span class="p">.</span><span class="mi">065</span><span class="p">..</span><span class="mi">441</span><span class="p">.</span><span class="mi">346</span> <span class="k">rows</span><span class="o">=</span><span class="mi">160000</span> <span class="n">loops</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
         <span class="o">-&gt;</span>  <span class="n">Seq</span> <span class="n">Scan</span> <span class="k">on</span> <span class="n">weather_station</span> <span class="n">ws</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">00</span><span class="p">..</span><span class="mi">2</span><span class="p">.</span><span class="mi">25</span> <span class="k">rows</span><span class="o">=</span><span class="mi">1</span> <span class="n">width</span><span class="o">=</span><span class="mi">16</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">41</span><span class="p">.</span><span class="mi">031</span><span class="p">..</span><span class="mi">41</span><span class="p">.</span><span class="mi">039</span> <span class="k">rows</span><span class="o">=</span><span class="mi">1</span> <span class="n">loops</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
               <span class="n">Filter</span><span class="p">:</span> <span class="p">(</span><span class="n">name</span> <span class="o">=</span> <span class="s1">'weather-station-17'</span><span class="p">::</span><span class="nb">text</span><span class="p">)</span>
               <span class="k">Rows</span> <span class="n">Removed</span> <span class="k">by</span> <span class="n">Filter</span><span class="p">:</span> <span class="mi">99</span>
         <span class="o">-&gt;</span>  <span class="k">Index</span> <span class="n">Scan</span> <span class="k">using</span> <span class="n">ix_btree_weather_station_id_received_at_non_covering</span> <span class="k">on</span> <span class="n">weather_report</span> <span class="n">wr</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">56</span><span class="p">..</span><span class="mi">585679</span><span class="p">.</span><span class="mi">33</span> <span class="k">rows</span><span class="o">=</span><span class="mi">159255</span> <span class="n">width</span><span class="o">=</span><span class="mi">32</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">023</span><span class="p">..</span><span class="mi">384</span><span class="p">.</span><span class="mi">034</span> <span class="k">rows</span><span class="o">=</span><span class="mi">160000</span> <span class="n">loops</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span> <i class="conum" data-value="1"></i><b>(1)</b>
               <span class="k">Index</span> <span class="n">Cond</span><span class="p">:</span> <span class="p">((</span><span class="n">weather_station_id</span> <span class="o">=</span> <span class="n">ws</span><span class="p">.</span><span class="n">id</span><span class="p">)</span> <span class="k">AND</span> <span class="p">(</span><span class="n">received_at</span> <span class="o">&gt;=</span> <span class="s1">'2025-03-06 00:00:00'</span><span class="p">::</span><span class="nb">timestamp</span> <span class="k">without</span> <span class="nb">time</span> <span class="k">zone</span><span class="p">))</span>
 <span class="n">Planning</span> <span class="nb">Time</span><span class="p">:</span> <span class="mi">0</span><span class="p">.</span><span class="mi">141</span> <span class="n">ms</span>
 <span class="n">JIT</span><span class="p">:</span>
   <span class="n">Functions</span><span class="p">:</span> <span class="mi">9</span>
   <span class="k">Options</span><span class="p">:</span> <span class="n">Inlining</span> <span class="k">true</span><span class="p">,</span> <span class="n">Optimization</span> <span class="k">true</span><span class="p">,</span> <span class="n">Expressions</span> <span class="k">true</span><span class="p">,</span> <span class="n">Deforming</span> <span class="k">true</span>
   <span class="n">Timing</span><span class="p">:</span> <span class="n">Generation</span> <span class="mi">0</span><span class="p">.</span><span class="mi">615</span> <span class="n">ms</span> <span class="p">(</span><span class="n">Deform</span> <span class="mi">0</span><span class="p">.</span><span class="mi">204</span> <span class="n">ms</span><span class="p">),</span> <span class="n">Inlining</span> <span class="mi">13</span><span class="p">.</span><span class="mi">385</span> <span class="n">ms</span><span class="p">,</span> <span class="n">Optimization</span> <span class="mi">16</span><span class="p">.</span><span class="mi">098</span> <span class="n">ms</span><span class="p">,</span> <span class="n">Emission</span> <span class="mi">11</span><span class="p">.</span><span class="mi">561</span> <span class="n">ms</span><span class="p">,</span> <span class="n">Total</span> <span class="mi">41</span><span class="p">.</span><span class="mi">658</span> <span class="n">ms</span>
 <span class="n">Execution</span> <span class="nb">Time</span><span class="p">:</span> <span class="mi">452</span><span class="p">.</span><span class="mi">780</span> <span class="n">ms</span></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>The previous parallel sequential scan was replaced with an <a href="https://www.postgresql.org/docs/current/index-scanning.html" target="_blank" rel="noopener">index scan</a>, which is faster but still not fast enough because PostgreSQL must fetch additional columns from the table.</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>Execution times have dropped from 2069 ms to 71 ms for the "fetch" query and from 2035 ms to 453 ms for the "count" query.
Much better, but there&#8217;s still room for improvement!</p>
</div>
</div>
<div class="sect2">
<h3 id="introducing-covering-b-tree-indexes">Introducing covering B-Tree indexes</h3>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
<div class="paragraph">
<p>A <a href="https://www.postgresql.org/docs/current/indexes-index-only-scans.html" target="_blank" rel="noopener">covering index</a> is an index that includes all the columns needed for a query, allowing PostgreSQL to retrieve data entirely from the index without accessing the main table (heap fetch).
This improves performance by reducing disk I/O, but comes at the cost of increased storage usage.</p>
</div>
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>Let&#8217;s replace our previous non-covering indexes with covering indexes for better performance.</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="sql"><span class="k">create</span> <span class="k">index</span> <span class="n">ix_btree_received_at_weather_station_id_covering</span>
<span class="k">on</span> <span class="n">weather_report</span> <span class="k">using</span> <span class="n">btree</span> <span class="p">(</span><span class="n">received_at</span> <span class="k">desc</span><span class="p">,</span> <span class="n">weather_station_id</span><span class="p">)</span> <span class="n">include</span> <span class="p">(</span><span class="n">id</span><span class="p">,</span> <span class="k">data</span><span class="p">);</span> <i class="conum" data-value="1"></i><b>(1)</b></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>The <code>INCLUDE</code> clause for covering indexes was introduced in PostgreSQL 11.
If you&#8217;re using an older version, you&#8217;ll need to add the <code>id</code> and <code>data</code> columns at the end of the index definition instead.</td>
</tr>
</table>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="sql"><span class="k">create</span> <span class="k">index</span> <span class="n">ix_btree_weather_station_id_received_at_covering</span>
<span class="k">on</span> <span class="n">weather_report</span> <span class="k">using</span> <span class="n">btree</span> <span class="p">(</span><span class="n">weather_station_id</span><span class="p">,</span> <span class="n">received_at</span> <span class="k">desc</span><span class="p">)</span> <span class="n">include</span> <span class="p">(</span><span class="n">id</span><span class="p">,</span> <span class="k">data</span><span class="p">);</span></code></pre>
</div>
</div>
<div class="paragraph">
<p>Does that make our queries run faster?</p>
</div>
<div class="listingblock">
<div class="title">Execution plan of the "fetch" query with a covering index</div>
<div class="content">
<pre class="rouge highlight"><code data-lang="sql"> <span class="k">Limit</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">6641</span><span class="p">.</span><span class="mi">57</span><span class="p">..</span><span class="mi">7471</span><span class="p">.</span><span class="mi">70</span> <span class="k">rows</span><span class="o">=</span><span class="mi">100</span> <span class="n">width</span><span class="o">=</span><span class="mi">57</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">27</span><span class="p">.</span><span class="mi">223</span><span class="p">..</span><span class="mi">29</span><span class="p">.</span><span class="mi">976</span> <span class="k">rows</span><span class="o">=</span><span class="mi">100</span> <span class="n">loops</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
   <span class="o">-&gt;</span>  <span class="n">Nested</span> <span class="n">Loop</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">56</span><span class="p">..</span><span class="mi">1336188</span><span class="p">.</span><span class="mi">27</span> <span class="k">rows</span><span class="o">=</span><span class="mi">160962</span> <span class="n">width</span><span class="o">=</span><span class="mi">57</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">156</span><span class="p">..</span><span class="mi">29</span><span class="p">.</span><span class="mi">946</span> <span class="k">rows</span><span class="o">=</span><span class="mi">900</span> <span class="n">loops</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
         <span class="k">Join</span> <span class="n">Filter</span><span class="p">:</span> <span class="p">(</span><span class="n">wr</span><span class="p">.</span><span class="n">weather_station_id</span> <span class="o">=</span> <span class="n">ws</span><span class="p">.</span><span class="n">id</span><span class="p">)</span>
         <span class="k">Rows</span> <span class="n">Removed</span> <span class="k">by</span> <span class="k">Join</span> <span class="n">Filter</span><span class="p">:</span> <span class="mi">89092</span> <i class="conum" data-value="1"></i><b>(1)</b>
         <span class="o">-&gt;</span>  <span class="k">Index</span> <span class="k">Only</span> <span class="n">Scan</span> <span class="k">using</span> <span class="n">ix_btree_received_at_weather_station_id_covering</span> <span class="k">on</span> <span class="n">weather_report</span> <span class="n">wr</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">56</span><span class="p">..</span><span class="mi">1094743</span><span class="p">.</span><span class="mi">50</span> <span class="k">rows</span><span class="o">=</span><span class="mi">16096168</span> <span class="n">width</span><span class="o">=</span><span class="mi">73</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">030</span><span class="p">..</span><span class="mi">11</span><span class="p">.</span><span class="mi">414</span> <span class="k">rows</span><span class="o">=</span><span class="mi">89992</span> <span class="n">loops</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span> <i class="conum" data-value="2"></i><b>(2)</b>
               <span class="k">Index</span> <span class="n">Cond</span><span class="p">:</span> <span class="p">(</span><span class="n">received_at</span> <span class="o">&gt;=</span> <span class="s1">'2025-03-06 00:00:00'</span><span class="p">::</span><span class="nb">timestamp</span> <span class="k">without</span> <span class="nb">time</span> <span class="k">zone</span><span class="p">)</span>
               <span class="n">Heap</span> <span class="n">Fetches</span><span class="p">:</span> <span class="mi">0</span>
         <span class="o">-&gt;</span>  <span class="n">Materialize</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">00</span><span class="p">..</span><span class="mi">2</span><span class="p">.</span><span class="mi">25</span> <span class="k">rows</span><span class="o">=</span><span class="mi">1</span> <span class="n">width</span><span class="o">=</span><span class="mi">16</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">000</span><span class="p">..</span><span class="mi">0</span><span class="p">.</span><span class="mi">000</span> <span class="k">rows</span><span class="o">=</span><span class="mi">1</span> <span class="n">loops</span><span class="o">=</span><span class="mi">89992</span><span class="p">)</span>
               <span class="o">-&gt;</span>  <span class="n">Seq</span> <span class="n">Scan</span> <span class="k">on</span> <span class="n">weather_station</span> <span class="n">ws</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">00</span><span class="p">..</span><span class="mi">2</span><span class="p">.</span><span class="mi">25</span> <span class="k">rows</span><span class="o">=</span><span class="mi">1</span> <span class="n">width</span><span class="o">=</span><span class="mi">16</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">018</span><span class="p">..</span><span class="mi">0</span><span class="p">.</span><span class="mi">035</span> <span class="k">rows</span><span class="o">=</span><span class="mi">1</span> <span class="n">loops</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
                     <span class="n">Filter</span><span class="p">:</span> <span class="p">(</span><span class="n">name</span> <span class="o">=</span> <span class="s1">'weather-station-17'</span><span class="p">::</span><span class="nb">text</span><span class="p">)</span>
                     <span class="k">Rows</span> <span class="n">Removed</span> <span class="k">by</span> <span class="n">Filter</span><span class="p">:</span> <span class="mi">99</span>
 <span class="n">Planning</span> <span class="nb">Time</span><span class="p">:</span> <span class="mi">0</span><span class="p">.</span><span class="mi">472</span> <span class="n">ms</span>
 <span class="n">Execution</span> <span class="nb">Time</span><span class="p">:</span> <span class="mi">30</span><span class="p">.</span><span class="mi">018</span> <span class="n">ms</span></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>Filtering out 89,092 rows after the join is inefficient.
We&#8217;ll need to fix that later.</td>
</tr>
<tr>
<td><i class="conum" data-value="2"></i><b>2</b></td>
<td>The previous index scan was replaced with an <a href="https://www.postgresql.org/docs/current/indexes-index-only-scans.html" target="_blank" rel="noopener">index-only scan</a> which is significantly faster.</td>
</tr>
</table>
</div>
<div class="listingblock">
<div class="title">Execution plan of the "count" query with a covering index</div>
<div class="content">
<pre class="rouge highlight"><code data-lang="sql"> <span class="k">Aggregate</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">13390</span><span class="p">.</span><span class="mi">08</span><span class="p">..</span><span class="mi">13390</span><span class="p">.</span><span class="mi">09</span> <span class="k">rows</span><span class="o">=</span><span class="mi">1</span> <span class="n">width</span><span class="o">=</span><span class="mi">8</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">31</span><span class="p">.</span><span class="mi">861</span><span class="p">..</span><span class="mi">31</span><span class="p">.</span><span class="mi">862</span> <span class="k">rows</span><span class="o">=</span><span class="mi">1</span> <span class="n">loops</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
   <span class="o">-&gt;</span>  <span class="n">Nested</span> <span class="n">Loop</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">56</span><span class="p">..</span><span class="mi">12987</span><span class="p">.</span><span class="mi">67</span> <span class="k">rows</span><span class="o">=</span><span class="mi">160962</span> <span class="n">width</span><span class="o">=</span><span class="mi">16</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">018</span><span class="p">..</span><span class="mi">26</span><span class="p">.</span><span class="mi">090</span> <span class="k">rows</span><span class="o">=</span><span class="mi">160000</span> <span class="n">loops</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
         <span class="o">-&gt;</span>  <span class="n">Seq</span> <span class="n">Scan</span> <span class="k">on</span> <span class="n">weather_station</span> <span class="n">ws</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">00</span><span class="p">..</span><span class="mi">2</span><span class="p">.</span><span class="mi">25</span> <span class="k">rows</span><span class="o">=</span><span class="mi">1</span> <span class="n">width</span><span class="o">=</span><span class="mi">16</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">005</span><span class="p">..</span><span class="mi">0</span><span class="p">.</span><span class="mi">016</span> <span class="k">rows</span><span class="o">=</span><span class="mi">1</span> <span class="n">loops</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
               <span class="n">Filter</span><span class="p">:</span> <span class="p">(</span><span class="n">name</span> <span class="o">=</span> <span class="s1">'weather-station-17'</span><span class="p">::</span><span class="nb">text</span><span class="p">)</span>
               <span class="k">Rows</span> <span class="n">Removed</span> <span class="k">by</span> <span class="n">Filter</span><span class="p">:</span> <span class="mi">99</span>
         <span class="o">-&gt;</span>  <span class="k">Index</span> <span class="k">Only</span> <span class="n">Scan</span> <span class="k">using</span> <span class="n">ix_btree_weather_station_id_received_at_covering</span> <span class="k">on</span> <span class="n">weather_report</span> <span class="n">wr</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">56</span><span class="p">..</span><span class="mi">11375</span><span class="p">.</span><span class="mi">80</span> <span class="k">rows</span><span class="o">=</span><span class="mi">160962</span> <span class="n">width</span><span class="o">=</span><span class="mi">32</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">012</span><span class="p">..</span><span class="mi">17</span><span class="p">.</span><span class="mi">698</span> <span class="k">rows</span><span class="o">=</span><span class="mi">160000</span> <span class="n">loops</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span> <i class="conum" data-value="1"></i><b>(1)</b>
               <span class="k">Index</span> <span class="n">Cond</span><span class="p">:</span> <span class="p">((</span><span class="n">weather_station_id</span> <span class="o">=</span> <span class="n">ws</span><span class="p">.</span><span class="n">id</span><span class="p">)</span> <span class="k">AND</span> <span class="p">(</span><span class="n">received_at</span> <span class="o">&gt;=</span> <span class="s1">'2025-03-06 00:00:00'</span><span class="p">::</span><span class="nb">timestamp</span> <span class="k">without</span> <span class="nb">time</span> <span class="k">zone</span><span class="p">))</span>
               <span class="n">Heap</span> <span class="n">Fetches</span><span class="p">:</span> <span class="mi">0</span>
 <span class="n">Planning</span> <span class="nb">Time</span><span class="p">:</span> <span class="mi">0</span><span class="p">.</span><span class="mi">139</span> <span class="n">ms</span>
 <span class="n">Execution</span> <span class="nb">Time</span><span class="p">:</span> <span class="mi">31</span><span class="p">.</span><span class="mi">886</span> <span class="n">ms</span></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>The previous index scan was replaced with an <a href="https://www.postgresql.org/docs/current/indexes-index-only-scans.html" target="_blank" rel="noopener">index-only scan</a> which is much faster.</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>Compared to non-covering indexes, execution times have dropped from 71 ms to 30 ms for the "fetch" query and from 453 ms to 32 ms for the "count" query.
That&#8217;s awesome, but we&#8217;re not done optimizing these queries yet!</p>
</div>
</div>
<div class="sect2">
<h3 id="introducing-a-brin-index">Introducing a BRIN index</h3>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
<div class="paragraph">
<p>A <a href="https://www.postgresql.org/docs/current/brin.html" target="_blank" rel="noopener">BRIN index</a> is a lightweight index that stores summary metadata (min and max values) for block ranges instead of indexing every row.
It is ideal for large, append-only tables with naturally ordered data, such as time-series or logs, offering fast lookups with minimal storage overhead.</p>
</div>
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>That sounds like a great index for the <code>received_at</code> column.
Here&#8217;s how to create it:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="sql"><span class="k">create</span> <span class="k">index</span> <span class="n">ix_brin_received_at</span>
<span class="k">on</span> <span class="n">weather_report</span> <span class="k">using</span> <span class="n">brin</span> <span class="p">(</span><span class="n">received_at</span><span class="p">);</span></code></pre>
</div>
</div>
<div class="paragraph">
<p>Unfortunately, that index doesn&#8217;t help reduce the execution time of our queries.
A BRIN index is only effective when data is physically sorted, but since <code>weather_report</code> records are deleted after 30 days, they are not stored in natural order.
If the records were not removed, a BRIN index could have been a great way to improve query performance.</p>
</div>
<div class="paragraph">
<p>PostgreSQL provides a command that physically reorders a table based on an index: <a href="https://www.postgresql.org/docs/current/sql-cluster.html" target="_blank" rel="noopener">CLUSTER</a>.
However, BRIN indexes do not support clustering.</p>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="indexes-come-at-a-cost">Indexes come at a cost</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Run this query to check how much disk space your indexes are using:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="sql"><span class="k">select</span> <span class="n">indexname</span><span class="p">,</span> <span class="n">pg_size_pretty</span><span class="p">(</span><span class="n">pg_relation_size</span><span class="p">(</span><span class="n">indexname</span><span class="p">::</span><span class="n">regclass</span><span class="p">))</span>
<span class="k">from</span> <span class="n">pg_indexes</span>
<span class="k">where</span> <span class="n">tablename</span> <span class="o">=</span> <span class="s1">'weather_report'</span><span class="p">;</span></code></pre>
</div>
</div>
<div class="paragraph">
<p>Covering B-Tree indexes can be quite expensive and sometimes use nearly as much disk space as the table itself.
On the other hand, BRIN indexes use a very small amount of disk space.</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="sql">                      <span class="n">indexname</span>                       <span class="o">|</span> <span class="n">pg_size_pretty</span>
<span class="c1">------------------------------------------------------+----------------</span>
 <span class="n">pk_weather_report</span>                                    <span class="o">|</span> <span class="mi">2337</span> <span class="n">MB</span>
 <span class="n">ix_btree_received_at_weather_station_id_non_covering</span> <span class="o">|</span> <span class="mi">1159</span> <span class="n">MB</span>
 <span class="n">ix_btree_weather_station_id_received_at_non_covering</span> <span class="o">|</span> <span class="mi">1162</span> <span class="n">MB</span>
 <span class="n">ix_btree_received_at_weather_station_id_covering</span>     <span class="o">|</span> <span class="mi">2977</span> <span class="n">MB</span>
 <span class="n">ix_btree_weather_station_id_received_at_covering</span>     <span class="o">|</span> <span class="mi">2986</span> <span class="n">MB</span>
 <span class="n">ix_brin_received_at</span>                                  <span class="o">|</span> <span class="mi">176</span> <span class="n">kB</span></code></pre>
</div>
</div>
<div class="paragraph">
<p>Indexes also slow down <code>INSERT</code>, <code>UPDATE</code> and <code>DELETE</code> queries.
Every time a row is modified, PostgreSQL must update the corresponding index entries.
This overhead on write operations is especially noticeable with high insert-rate workloads.</p>
</div>
<div class="admonitionblock tip">
<table>
<tr>
<td class="icon">
<i class="fa icon-tip" title="Tip"></i>
</td>
<td class="content">
<div class="paragraph">
<p>You can check the impact of indexes on write performance by analyzing execution plans or running <a href="https://www.postgresql.org/docs/current/pgbench.html" target="_blank" rel="noopener">benchmark tests</a> on your database.</p>
</div>
</td>
</tr>
</table>
</div>
<div class="listingblock">
<div class="title">Explaining an insert query</div>
<div class="content">
<pre class="rouge highlight"><code data-lang="sql"><span class="k">explain</span> <span class="k">analyze</span>
<span class="k">insert</span> <span class="k">into</span> <span class="n">weather_report</span> <span class="p">(</span><span class="k">data</span><span class="p">,</span> <span class="n">received_at</span><span class="p">,</span> <span class="n">weather_station_id</span><span class="p">)</span>
<span class="k">values</span> <span class="p">(</span><span class="s1">'Sunny day'</span><span class="p">,</span> <span class="n">now</span><span class="p">(),</span> <span class="s1">'be9a5a83-f789-41dd-8023-cd3df445f055'</span><span class="p">);</span></code></pre>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="writing-smarter-queries">Writing smarter queries</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Indexes can really boost performance, but they can&#8217;t automagically fix a poorly written query.</p>
</div>
<div class="paragraph">
<p>In the current "fetch" query, PostgreSQL retrieves 89,992 rows and then filters out 89,092 of them.
That doesn&#8217;t look right.
Let&#8217;s see what happens if we replace the join with a subquery:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="sql"><span class="k">explain</span> <span class="k">analyze</span>
<span class="k">select</span> <span class="n">id</span><span class="p">,</span> <span class="k">data</span><span class="p">,</span> <span class="n">received_at</span>
<span class="k">from</span> <span class="n">weather_report</span>
<span class="k">where</span> <span class="n">received_at</span> <span class="o">&gt;=</span> <span class="s1">'2025-03-06'</span>
<span class="k">and</span> <span class="n">weather_station_id</span> <span class="o">=</span>
<span class="p">(</span><span class="k">select</span> <span class="n">id</span> <span class="k">from</span> <span class="n">weather_station</span> <span class="k">where</span> <span class="n">name</span> <span class="o">=</span> <span class="s1">'weather-station-17'</span><span class="p">)</span>
<span class="k">order</span> <span class="k">by</span> <span class="n">received_at</span> <span class="k">desc</span>
<span class="k">offset</span> <span class="mi">800</span> <span class="k">limit</span> <span class="mi">100</span><span class="p">;</span></code></pre>
</div>
</div>
<div class="listingblock">
<div class="title">Execution plan of the "fetch" query with subquery</div>
<div class="content">
<pre class="rouge highlight"><code data-lang="sql"> <span class="k">Limit</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">59</span><span class="p">.</span><span class="mi">35</span><span class="p">..</span><span class="mi">66</span><span class="p">.</span><span class="mi">42</span> <span class="k">rows</span><span class="o">=</span><span class="mi">100</span> <span class="n">width</span><span class="o">=</span><span class="mi">57</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">146</span><span class="p">..</span><span class="mi">0</span><span class="p">.</span><span class="mi">162</span> <span class="k">rows</span><span class="o">=</span><span class="mi">100</span> <span class="n">loops</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
   <span class="n">InitPlan</span> <span class="mi">1</span>
     <span class="o">-&gt;</span>  <span class="n">Seq</span> <span class="n">Scan</span> <span class="k">on</span> <span class="n">weather_station</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">00</span><span class="p">..</span><span class="mi">2</span><span class="p">.</span><span class="mi">25</span> <span class="k">rows</span><span class="o">=</span><span class="mi">1</span> <span class="n">width</span><span class="o">=</span><span class="mi">16</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">008</span><span class="p">..</span><span class="mi">0</span><span class="p">.</span><span class="mi">014</span> <span class="k">rows</span><span class="o">=</span><span class="mi">1</span> <span class="n">loops</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
           <span class="n">Filter</span><span class="p">:</span> <span class="p">(</span><span class="n">name</span> <span class="o">=</span> <span class="s1">'weather-station-17'</span><span class="p">::</span><span class="nb">text</span><span class="p">)</span>
           <span class="k">Rows</span> <span class="n">Removed</span> <span class="k">by</span> <span class="n">Filter</span><span class="p">:</span> <span class="mi">99</span>
   <span class="o">-&gt;</span>  <span class="k">Index</span> <span class="k">Only</span> <span class="n">Scan</span> <span class="k">using</span> <span class="n">ix_btree_weather_station_id_received_at_covering</span> <span class="k">on</span> <span class="n">weather_report</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">56</span><span class="p">..</span><span class="mi">11375</span><span class="p">.</span><span class="mi">80</span> <span class="k">rows</span><span class="o">=</span><span class="mi">160962</span> <span class="n">width</span><span class="o">=</span><span class="mi">57</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">029</span><span class="p">..</span><span class="mi">0</span><span class="p">.</span><span class="mi">139</span> <span class="k">rows</span><span class="o">=</span><span class="mi">900</span> <span class="n">loops</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span> <i class="conum" data-value="1"></i><b>(1)</b>
         <span class="k">Index</span> <span class="n">Cond</span><span class="p">:</span> <span class="p">((</span><span class="n">weather_station_id</span> <span class="o">=</span> <span class="p">(</span><span class="n">InitPlan</span> <span class="mi">1</span><span class="p">).</span><span class="n">col1</span><span class="p">)</span> <span class="k">AND</span> <span class="p">(</span><span class="n">received_at</span> <span class="o">&gt;=</span> <span class="s1">'2025-03-06 00:00:00'</span><span class="p">::</span><span class="nb">timestamp</span> <span class="k">without</span> <span class="nb">time</span> <span class="k">zone</span><span class="p">))</span>
         <span class="n">Heap</span> <span class="n">Fetches</span><span class="p">:</span> <span class="mi">0</span>
 <span class="n">Planning</span> <span class="nb">Time</span><span class="p">:</span> <span class="mi">0</span><span class="p">.</span><span class="mi">095</span> <span class="n">ms</span>
 <span class="n">Execution</span> <span class="nb">Time</span><span class="p">:</span> <span class="mi">0</span><span class="p">.</span><span class="mi">177</span> <span class="n">ms</span></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>The query planner is now using a different index.</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>Wow, that&#8217;s an incredible improvement!
The query that originally took 2069 ms without an index now runs in under 1 ms.
How is that even possible?</p>
</div>
<div class="paragraph">
<p>The subquery helped move filtering <em>before</em> scanning.
Because of that, PostgreSQL no longer has to fetch 89,992 rows and perform a materialized lookup for each one.
That was a lot of unnecessary work.
It&#8217;s gone now.</p>
</div>
<div class="paragraph">
<p>What about the "count" query?
Could a subquery help reduce its execution time as well?</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="sql"><span class="k">explain</span> <span class="k">analyze</span>
<span class="k">select</span> <span class="k">count</span><span class="p">(</span><span class="o">*</span><span class="p">)</span>
<span class="k">from</span> <span class="n">weather_report</span>
<span class="k">where</span> <span class="n">received_at</span> <span class="o">&gt;=</span> <span class="s1">'2025-03-06'</span>
<span class="k">and</span> <span class="n">weather_station_id</span> <span class="o">=</span>
<span class="p">(</span><span class="k">select</span> <span class="n">id</span> <span class="k">from</span> <span class="n">weather_station</span> <span class="k">where</span> <span class="n">name</span> <span class="o">=</span> <span class="s1">'weather-station-17'</span><span class="p">);</span></code></pre>
</div>
</div>
<div class="listingblock">
<div class="title">Execution plan of the "count" query with a subquery</div>
<div class="content">
<pre class="rouge highlight"><code data-lang="sql"> <span class="n">Finalize</span> <span class="k">Aggregate</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">11606</span><span class="p">.</span><span class="mi">99</span><span class="p">..</span><span class="mi">11607</span><span class="p">.</span><span class="mi">00</span> <span class="k">rows</span><span class="o">=</span><span class="mi">1</span> <span class="n">width</span><span class="o">=</span><span class="mi">8</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">19</span><span class="p">.</span><span class="mi">492</span><span class="p">..</span><span class="mi">22</span><span class="p">.</span><span class="mi">322</span> <span class="k">rows</span><span class="o">=</span><span class="mi">1</span> <span class="n">loops</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
   <span class="n">InitPlan</span> <span class="mi">1</span>
     <span class="o">-&gt;</span>  <span class="n">Seq</span> <span class="n">Scan</span> <span class="k">on</span> <span class="n">weather_station</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">00</span><span class="p">..</span><span class="mi">2</span><span class="p">.</span><span class="mi">25</span> <span class="k">rows</span><span class="o">=</span><span class="mi">1</span> <span class="n">width</span><span class="o">=</span><span class="mi">16</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">007</span><span class="p">..</span><span class="mi">0</span><span class="p">.</span><span class="mi">013</span> <span class="k">rows</span><span class="o">=</span><span class="mi">1</span> <span class="n">loops</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
           <span class="n">Filter</span><span class="p">:</span> <span class="p">(</span><span class="n">name</span> <span class="o">=</span> <span class="s1">'weather-station-17'</span><span class="p">::</span><span class="nb">text</span><span class="p">)</span>
           <span class="k">Rows</span> <span class="n">Removed</span> <span class="k">by</span> <span class="n">Filter</span><span class="p">:</span> <span class="mi">99</span>
   <span class="o">-&gt;</span>  <span class="n">Gather</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">11604</span><span class="p">.</span><span class="mi">53</span><span class="p">..</span><span class="mi">11604</span><span class="p">.</span><span class="mi">74</span> <span class="k">rows</span><span class="o">=</span><span class="mi">2</span> <span class="n">width</span><span class="o">=</span><span class="mi">8</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">19</span><span class="p">.</span><span class="mi">455</span><span class="p">..</span><span class="mi">22</span><span class="p">.</span><span class="mi">317</span> <span class="k">rows</span><span class="o">=</span><span class="mi">3</span> <span class="n">loops</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
         <span class="n">Workers</span> <span class="n">Planned</span><span class="p">:</span> <span class="mi">2</span>
         <span class="n">Workers</span> <span class="n">Launched</span><span class="p">:</span> <span class="mi">2</span>
         <span class="o">-&gt;</span>  <span class="k">Partial</span> <span class="k">Aggregate</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">10604</span><span class="p">.</span><span class="mi">53</span><span class="p">..</span><span class="mi">10604</span><span class="p">.</span><span class="mi">54</span> <span class="k">rows</span><span class="o">=</span><span class="mi">1</span> <span class="n">width</span><span class="o">=</span><span class="mi">8</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">8</span><span class="p">.</span><span class="mi">007</span><span class="p">..</span><span class="mi">8</span><span class="p">.</span><span class="mi">007</span> <span class="k">rows</span><span class="o">=</span><span class="mi">1</span> <span class="n">loops</span><span class="o">=</span><span class="mi">3</span><span class="p">)</span>
               <span class="o">-&gt;</span>  <span class="n">Parallel</span> <span class="k">Index</span> <span class="k">Only</span> <span class="n">Scan</span> <span class="k">using</span> <span class="n">ix_btree_weather_station_id_received_at_covering</span> <span class="k">on</span> <span class="n">weather_report</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">56</span><span class="p">..</span><span class="mi">10436</span><span class="p">.</span><span class="mi">86</span> <span class="k">rows</span><span class="o">=</span><span class="mi">67068</span> <span class="n">width</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">038</span><span class="p">..</span><span class="mi">6</span><span class="p">.</span><span class="mi">107</span> <span class="k">rows</span><span class="o">=</span><span class="mi">53333</span> <span class="n">loops</span><span class="o">=</span><span class="mi">3</span><span class="p">)</span>
                     <span class="k">Index</span> <span class="n">Cond</span><span class="p">:</span> <span class="p">((</span><span class="n">weather_station_id</span> <span class="o">=</span> <span class="p">(</span><span class="n">InitPlan</span> <span class="mi">1</span><span class="p">).</span><span class="n">col1</span><span class="p">)</span> <span class="k">AND</span> <span class="p">(</span><span class="n">received_at</span> <span class="o">&gt;=</span> <span class="s1">'2025-03-06 00:00:00'</span><span class="p">::</span><span class="nb">timestamp</span> <span class="k">without</span> <span class="nb">time</span> <span class="k">zone</span><span class="p">))</span>
                     <span class="n">Heap</span> <span class="n">Fetches</span><span class="p">:</span> <span class="mi">0</span>
 <span class="n">Planning</span> <span class="nb">Time</span><span class="p">:</span> <span class="mi">0</span><span class="p">.</span><span class="mi">093</span> <span class="n">ms</span>
 <span class="n">Execution</span> <span class="nb">Time</span><span class="p">:</span> <span class="mi">22</span><span class="p">.</span><span class="mi">346</span> <span class="n">ms</span></code></pre>
</div>
</div>
<div class="paragraph">
<p>It&#8217;s not as impressive as the "fetch" query, but the subquery still significantly improves performance by allowing parallel scans and aggregation.</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="keep-your-visibility-map-clean">Keep your visibility map clean</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Covering B-Tree indexes can greatly improve query performance, but they have a weakness you should be aware of: heap fetches.
A covering index allows a query to retrieve data entirely from the index without accessing the main table (heap), which would otherwise be expensive.
However, this only works efficiently if the <a href="https://www.postgresql.org/docs/current/storage-vm.html" target="_blank" rel="noopener">visibility map</a> marks all necessary heap pages as "all-visible".
If tuples are updated or deleted from a page and vacuum has not run, that page gets marked as "dirty" in the visibility map and PostgreSQL is forced to fetch rows from the heap, slowing down the query.</p>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
<div class="paragraph">
<p><a href="https://www.postgresql.org/docs/current/sql-vacuum.html" target="_blank" rel="noopener">VACUUM</a> removes dead tuples left behind by <code>UPDATE</code> and <code>DELETE</code> operations while updating the visibility map to minimize unnecessary heap fetches.</p>
</div>
</td>
</tr>
</table>
</div>
<div class="sect2">
<h3 id="when-should-a-table-be-vacuumed">When should a table be vacuumed?</h3>
<div class="paragraph">
<p>A good indicator is the execution plan: if you see heap fetches there, it means the visibility map isn’t up to date.</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="sql"><span class="p">[...]</span>
<span class="o">-&gt;</span>  <span class="k">Index</span> <span class="k">Only</span> <span class="n">Scan</span> <span class="p">[...]</span>
      <span class="p">[...]</span>
      <span class="n">Heap</span> <span class="n">Fetches</span><span class="p">:</span> <span class="mi">87</span> <i class="conum" data-value="1"></i><b>(1)</b>
<span class="p">[...]</span></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>PostgreSQL retrieved 87 rows from the heap, which suggests the table may need vacuuming.</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>You can also check the number of dead tuples by querying the <a href="https://www.postgresql.org/docs/current/monitoring-stats.html#MONITORING-PG-STAT-ALL-TABLES-VIEW" target="_blank" rel="noopener">pg_stat_user_tables</a> view:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="sql"><span class="k">select</span> <span class="n">relname</span><span class="p">,</span> <span class="n">n_live_tup</span><span class="p">,</span> <span class="n">n_dead_tup</span><span class="p">,</span> <span class="n">last_autovacuum</span>
<span class="k">from</span> <span class="n">pg_stat_user_tables</span>
<span class="k">order</span> <span class="k">by</span> <span class="n">n_dead_tup</span> <span class="k">desc</span><span class="p">;</span></code></pre>
</div>
</div>
<div class="paragraph">
<p>If <code>n_dead_tup</code> is high relative to <code>n_live_tup</code>, the table likely needs vacuuming.</p>
</div>
</div>
<div class="sect2">
<h3 id="how-can-a-table-be-vacuumed">How can a table be vacuumed?</h3>
<div class="paragraph">
<p>PostgreSQL <a href="https://www.postgresql.org/docs/current/runtime-config-autovacuum.html" target="_blank" rel="noopener">vacuums automatically</a> based on the number of dead tuples in a table.
By default, autovacuum is triggered when the number of dead tuples exceeds 50 + 20% of the total number of tuples in the table.
However, the default autovacuum settings are often not aggressive enough when data is removed daily, as in our use case.</p>
</div>
<div class="paragraph">
<p>Autovacuum settings can be tuned for a specific table:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="sql"><span class="k">alter</span> <span class="k">table</span> <span class="n">weather_report</span>
<span class="k">set</span> <span class="p">(</span>
    <span class="n">autovacuum_vacuum_threshold</span> <span class="o">=</span> <span class="mi">0</span><span class="p">,</span> <i class="conum" data-value="1"></i><b>(1)</b>
    <span class="n">autovacuum_vacuum_scale_factor</span> <span class="o">=</span> <span class="mi">0</span><span class="p">.</span><span class="mi">02</span> <i class="conum" data-value="2"></i><b>(2)</b>
<span class="p">);</span> <i class="conum" data-value="3"></i><b>(3)</b></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>Minimum number of updated or deleted tuples needed to trigger an autovacuum. Defaults to <code>50</code>.</td>
</tr>
<tr>
<td><i class="conum" data-value="2"></i><b>2</b></td>
<td>Fraction of the table size to add to <code>autovacuum_vacuum_threshold</code> when deciding whether to trigger an autovacuum. Defaults to <code>0.2</code>.</td>
</tr>
<tr>
<td><i class="conum" data-value="3"></i><b>3</b></td>
<td>PostgreSQL will trigger an autovacuum when 2% of the table tuples are dead, instead of the default 50 + 20%.</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>If your data is removed in a single batch as part of a daily maintenance task, a better approach is to run a manual vacuum afterward:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="sql"><span class="k">vacuum</span> <span class="k">analyze</span> <span class="n">weather_report</span><span class="p">;</span></code></pre>
</div>
</div>
<div class="admonitionblock tip">
<table>
<tr>
<td class="icon">
<i class="fa icon-tip" title="Tip"></i>
</td>
<td class="content">
<div class="paragraph">
<p><a href="https://www.postgresql.org/docs/current/sql-analyze.html" target="_blank" rel="noopener">ANALYZE</a> updates statistics that help the query planner choose the most efficient execution plan.
Running it alongside vacuum is usually a good practice.</p>
</div>
</td>
</tr>
</table>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
<div class="paragraph">
<p>All execution plans in this post were generated after running a manual <code>VACUUM ANALYZE</code>.</p>
</div>
</td>
</tr>
</table>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="partitioning-the-weather_report-table"><a id="partitioning"></a> Partitioning the <code>weather_report</code> table</h2>
<div class="sectionbody">
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
<div class="paragraph">
<p><a href="https://www.postgresql.org/docs/current/ddl-partitioning.html" target="_blank" rel="noopener">Partitioning</a> a table speeds up queries by allowing PostgreSQL to scan only the relevant partition instead of the entire table.
When each partition is indexed, the indexes are smaller and more focused, making lookups faster and more efficient.</p>
</div>
</td>
</tr>
</table>
</div>
<div class="sect2">
<h3 id="schema-changes-required-for-partitioning">Schema changes required for partitioning</h3>
<div class="paragraph">
<p>Partitioning the <code>weather_report</code> table requires a few changes to the table schema:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="sql"><span class="k">create</span> <span class="k">table</span> <span class="n">weather_report</span> <span class="p">(</span>
    <span class="n">id</span> <span class="n">uuid</span> <span class="k">not</span> <span class="k">null</span> <span class="k">default</span> <span class="n">gen_random_uuid</span><span class="p">(),</span>
    <span class="k">data</span> <span class="nb">text</span> <span class="k">not</span> <span class="k">null</span><span class="p">,</span>
    <span class="n">received_at</span> <span class="nb">timestamp</span> <span class="k">not</span> <span class="k">null</span><span class="p">,</span>
    <span class="n">weather_station_id</span> <span class="n">uuid</span> <span class="k">not</span> <span class="k">null</span><span class="p">,</span>
    <span class="k">constraint</span> <span class="n">fk_weather_report_weather_station</span> <span class="k">foreign</span> <span class="k">key</span> <span class="p">(</span><span class="n">weather_station_id</span><span class="p">)</span> <span class="k">references</span> <span class="n">weather_station</span> <span class="p">(</span><span class="n">id</span><span class="p">)</span>
<span class="p">)</span> <span class="k">partition</span> <span class="k">by</span> <span class="k">range</span> <span class="p">(</span><span class="n">received_at</span><span class="p">);</span> <i class="conum" data-value="1"></i><b>(1)</b></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>Each partition will contain a distinct and continuous range of <code>received_at</code> values.</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>Until now, the <code>id</code> column was the primary key of the <code>weather_report</code> table.
That won&#8217;t work with partitions, as the primary key defined on the parent table <em>must</em> include the partition key (<code>received_at</code>).
It&#8217;s still possible to define a primary key on <code>id</code> within each child partition, but this doesn&#8217;t guarantee uniqueness across all partitions.
This limitation can be addressed in various ways, such as using a <a href="https://www.postgresql.org/docs/current/sql-createtrigger.html" target="_blank" rel="noopener">trigger</a> to enforce uniqueness on the <code>id</code> column.
However, this goes beyond the scope of this post, so I won’t go into further detail.</p>
</div>
</div>
<div class="sect2">
<h3 id="creating-and-dropping-partitions">Creating and dropping partitions</h3>
<div class="paragraph">
<p>Each day requires a new partition:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="sql"><span class="k">create</span> <span class="k">table</span> <span class="n">weather_report_2023_03_21</span>
<span class="k">partition</span> <span class="k">of</span> <span class="n">weather_report</span> <span class="k">for</span> <span class="k">values</span> <span class="k">from</span> <span class="p">(</span><span class="s1">'2023-03-21'</span><span class="p">)</span> <span class="k">to</span> <span class="p">(</span><span class="s1">'2023-03-22'</span><span class="p">);</span> <i class="conum" data-value="1"></i><b>(1)</b></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>The lower bound is inclusive and the upper bound is exclusive.</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>Deleting weather reports older than 30 days couldn&#8217;t be easier: just drop the oldest partition.</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="sql"><span class="k">drop</span> <span class="k">table</span> <span class="n">weather_report_2023_02_19</span><span class="p">;</span> <i class="conum" data-value="1"></i><b>(1)</b></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>Finding the oldest partition can be automated.
Check out the <a href="https://github.com/gwenneg/blog-postgres-execution-time/blob/main/partitioned-table/sql/init.sql" target="_blank" rel="noopener">gwenneg/blog-postgres-execution-time</a> repository for more details.</td>
</tr>
</table>
</div>
<div class="admonitionblock tip">
<table>
<tr>
<td class="icon">
<i class="fa icon-tip" title="Tip"></i>
</td>
<td class="content">
<div class="paragraph">
<p>PostgreSQL doesn&#8217;t automatically refresh the parent table&#8217;s statistics or the query planner&#8217;s metadata after dropping a partition.
Run <code>VACUUM ANALYZE</code> on the parent table to update them manually.</p>
</div>
</td>
</tr>
</table>
</div>
</div>
<div class="sect2">
<h3 id="indexing-partitions">Indexing partitions</h3>
<div class="admonitionblock tip">
<table>
<tr>
<td class="icon">
<i class="fa icon-tip" title="Tip"></i>
</td>
<td class="content">
<div class="paragraph">
<p>If you create an index on the parent table, PostgreSQL automatically creates local indexes with the same definition on each existing and future partition.</p>
</div>
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>We already know which indexing strategy performs best with a regular <code>weather_report</code> table (without partitions).
Let&#8217;s reuse it with the partitioned <code>weather_report</code> table:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="sql"><span class="k">create</span> <span class="k">index</span> <span class="n">ix_btree_weather_station_id_received_at_covering</span>
<span class="k">on</span> <span class="n">weather_report</span> <span class="k">using</span> <span class="n">btree</span> <span class="p">(</span><span class="n">weather_station_id</span><span class="p">,</span> <span class="n">received_at</span> <span class="k">desc</span><span class="p">)</span> <span class="n">include</span> <span class="p">(</span><span class="n">id</span><span class="p">,</span> <span class="k">data</span><span class="p">);</span></code></pre>
</div>
</div>
<div class="paragraph">
<p>The partitioned index is similar to the regular index in terms of disk space usage:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="sql">                           <span class="n">index_name</span>                              <span class="o">|</span> <span class="n">index_size</span>
<span class="c1">-------------------------------------------------------------------+------------</span>
 <span class="n">ix_btree_weather_station_id_received_at_covering</span>                  <span class="o">|</span> <span class="mi">0</span> <span class="n">bytes</span> <i class="conum" data-value="1"></i><b>(1)</b>
 <span class="n">weather_report_2025_02_20_weather_station_id_received_at_id_d_idx</span> <span class="o">|</span> <span class="mi">100</span> <span class="n">MB</span>
 <span class="n">weather_report_2025_02_21_weather_station_id_received_at_id_d_idx</span> <span class="o">|</span> <span class="mi">100</span> <span class="n">MB</span>
 <span class="p">[...]</span>
 <span class="n">weather_report_2025_03_21_weather_station_id_received_at_id_d_idx</span> <span class="o">|</span> <span class="mi">100</span> <span class="n">MB</span> <i class="conum" data-value="2"></i><b>(2)</b></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>This is the index definition that is inherited by each partition.
It’s not an actual index and its size will never grow.</td>
</tr>
<tr>
<td><i class="conum" data-value="2"></i><b>2</b></td>
<td>The total size of the index across all partitions is 3000 MB.</td>
</tr>
</table>
</div>
</div>
<div class="sect2">
<h3 id="performance-with-partitions">Performance with partitions</h3>
<div class="paragraph">
<p>Does partitioning improve query performance?</p>
</div>
<div class="listingblock">
<div class="title">Execution plan of the "fetch" query with partitions</div>
<div class="content">
<pre class="rouge highlight"><code data-lang="sql"> <span class="k">Limit</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">70</span><span class="p">.</span><span class="mi">01</span><span class="p">..</span><span class="mi">77</span><span class="p">.</span><span class="mi">63</span> <span class="k">rows</span><span class="o">=</span><span class="mi">100</span> <span class="n">width</span><span class="o">=</span><span class="mi">57</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">408</span><span class="p">..</span><span class="mi">0</span><span class="p">.</span><span class="mi">456</span> <span class="k">rows</span><span class="o">=</span><span class="mi">100</span> <span class="n">loops</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
   <span class="n">InitPlan</span> <span class="mi">1</span>
     <span class="o">-&gt;</span>  <span class="n">Seq</span> <span class="n">Scan</span> <span class="k">on</span> <span class="n">weather_station</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">00</span><span class="p">..</span><span class="mi">2</span><span class="p">.</span><span class="mi">25</span> <span class="k">rows</span><span class="o">=</span><span class="mi">1</span> <span class="n">width</span><span class="o">=</span><span class="mi">16</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">018</span><span class="p">..</span><span class="mi">0</span><span class="p">.</span><span class="mi">031</span> <span class="k">rows</span><span class="o">=</span><span class="mi">1</span> <span class="n">loops</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
           <span class="n">Filter</span><span class="p">:</span> <span class="p">(</span><span class="n">name</span> <span class="o">=</span> <span class="s1">'weather-station-17'</span><span class="p">::</span><span class="nb">text</span><span class="p">)</span>
           <span class="k">Rows</span> <span class="n">Removed</span> <span class="k">by</span> <span class="n">Filter</span><span class="p">:</span> <span class="mi">99</span>
   <span class="o">-&gt;</span>  <span class="n">Append</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">6</span><span class="p">.</span><span class="mi">80</span><span class="p">..</span><span class="mi">12198</span><span class="p">.</span><span class="mi">40</span> <span class="k">rows</span><span class="o">=</span><span class="mi">159984</span> <span class="n">width</span><span class="o">=</span><span class="mi">57</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">065</span><span class="p">..</span><span class="mi">0</span><span class="p">.</span><span class="mi">401</span> <span class="k">rows</span><span class="o">=</span><span class="mi">900</span> <span class="n">loops</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
         <span class="o">-&gt;</span>  <span class="k">Index</span> <span class="k">Only</span> <span class="n">Scan</span> <span class="k">using</span> <span class="n">weather_report_2025_03_21_weather_station_id_received_at_id_d_idx</span> <span class="k">on</span> <span class="n">weather_report_2025_03_21</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">42</span><span class="p">..</span><span class="mi">712</span><span class="p">.</span><span class="mi">40</span> <span class="k">rows</span><span class="o">=</span><span class="mi">9999</span> <span class="n">width</span><span class="o">=</span><span class="mi">57</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">065</span><span class="p">..</span><span class="mi">0</span><span class="p">.</span><span class="mi">308</span> <span class="k">rows</span><span class="o">=</span><span class="mi">900</span> <span class="n">loops</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
               <span class="k">Index</span> <span class="n">Cond</span><span class="p">:</span> <span class="p">((</span><span class="n">weather_station_id</span> <span class="o">=</span> <span class="p">(</span><span class="n">InitPlan</span> <span class="mi">1</span><span class="p">).</span><span class="n">col1</span><span class="p">)</span> <span class="k">AND</span> <span class="p">(</span><span class="n">received_at</span> <span class="o">&gt;=</span> <span class="s1">'2025-03-06 00:00:00'</span><span class="p">::</span><span class="nb">timestamp</span> <span class="k">without</span> <span class="nb">time</span> <span class="k">zone</span><span class="p">))</span>
               <span class="n">Heap</span> <span class="n">Fetches</span><span class="p">:</span> <span class="mi">0</span>
         <span class="o">-&gt;</span>  <span class="k">Index</span> <span class="k">Only</span> <span class="n">Scan</span> <span class="k">using</span> <span class="n">weather_report_2025_03_20_weather_station_id_received_at_id_d_idx</span> <span class="k">on</span> <span class="n">weather_report_2025_03_20</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">42</span><span class="p">..</span><span class="mi">712</span><span class="p">.</span><span class="mi">40</span> <span class="k">rows</span><span class="o">=</span><span class="mi">9999</span> <span class="n">width</span><span class="o">=</span><span class="mi">57</span><span class="p">)</span> <span class="p">(</span><span class="n">never</span> <span class="n">executed</span><span class="p">)</span>
               <span class="k">Index</span> <span class="n">Cond</span><span class="p">:</span> <span class="p">((</span><span class="n">weather_station_id</span> <span class="o">=</span> <span class="p">(</span><span class="n">InitPlan</span> <span class="mi">1</span><span class="p">).</span><span class="n">col1</span><span class="p">)</span> <span class="k">AND</span> <span class="p">(</span><span class="n">received_at</span> <span class="o">&gt;=</span> <span class="s1">'2025-03-06 00:00:00'</span><span class="p">::</span><span class="nb">timestamp</span> <span class="k">without</span> <span class="nb">time</span> <span class="k">zone</span><span class="p">))</span>
               <span class="n">Heap</span> <span class="n">Fetches</span><span class="p">:</span> <span class="mi">0</span>
         <span class="p">[...]</span> <i class="conum" data-value="1"></i><b>(1)</b>
         <span class="o">-&gt;</span>  <span class="k">Index</span> <span class="k">Only</span> <span class="n">Scan</span> <span class="k">using</span> <span class="n">weather_report_2025_03_06_weather_station_id_received_at_id_d_idx</span> <span class="k">on</span> <span class="n">weather_report_2025_03_06</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">42</span><span class="p">..</span><span class="mi">712</span><span class="p">.</span><span class="mi">40</span> <span class="k">rows</span><span class="o">=</span><span class="mi">9999</span> <span class="n">width</span><span class="o">=</span><span class="mi">57</span><span class="p">)</span> <span class="p">(</span><span class="n">never</span> <span class="n">executed</span><span class="p">)</span>
               <span class="k">Index</span> <span class="n">Cond</span><span class="p">:</span> <span class="p">((</span><span class="n">weather_station_id</span> <span class="o">=</span> <span class="p">(</span><span class="n">InitPlan</span> <span class="mi">1</span><span class="p">).</span><span class="n">col1</span><span class="p">)</span> <span class="k">AND</span> <span class="p">(</span><span class="n">received_at</span> <span class="o">&gt;=</span> <span class="s1">'2025-03-06 00:00:00'</span><span class="p">::</span><span class="nb">timestamp</span> <span class="k">without</span> <span class="nb">time</span> <span class="k">zone</span><span class="p">))</span>
               <span class="n">Heap</span> <span class="n">Fetches</span><span class="p">:</span> <span class="mi">0</span>
 <span class="n">Planning</span> <span class="nb">Time</span><span class="p">:</span> <span class="mi">0</span><span class="p">.</span><span class="mi">793</span> <span class="n">ms</span>
 <span class="n">Execution</span> <span class="nb">Time</span><span class="p">:</span> <span class="mi">0</span><span class="p">.</span><span class="mi">574</span> <span class="n">ms</span></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>The execution plan has been cropped for readability.
The omitted section involves scanning data from 13 additional partitions.</td>
</tr>
</table>
</div>
<div class="listingblock">
<div class="title">Execution plan of the "count" query with partitions</div>
<div class="content">
<pre class="rouge highlight"><code data-lang="sql"> <span class="n">Finalize</span> <span class="k">Aggregate</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">12242</span><span class="p">.</span><span class="mi">11</span><span class="p">..</span><span class="mi">12242</span><span class="p">.</span><span class="mi">12</span> <span class="k">rows</span><span class="o">=</span><span class="mi">1</span> <span class="n">width</span><span class="o">=</span><span class="mi">8</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">17</span><span class="p">.</span><span class="mi">946</span><span class="p">..</span><span class="mi">20</span><span class="p">.</span><span class="mi">509</span> <span class="k">rows</span><span class="o">=</span><span class="mi">1</span> <span class="n">loops</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
   <span class="n">InitPlan</span> <span class="mi">1</span>
     <span class="o">-&gt;</span>  <span class="n">Seq</span> <span class="n">Scan</span> <span class="k">on</span> <span class="n">weather_station</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">00</span><span class="p">..</span><span class="mi">2</span><span class="p">.</span><span class="mi">25</span> <span class="k">rows</span><span class="o">=</span><span class="mi">1</span> <span class="n">width</span><span class="o">=</span><span class="mi">16</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">025</span><span class="p">..</span><span class="mi">0</span><span class="p">.</span><span class="mi">044</span> <span class="k">rows</span><span class="o">=</span><span class="mi">1</span> <span class="n">loops</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
           <span class="n">Filter</span><span class="p">:</span> <span class="p">(</span><span class="n">name</span> <span class="o">=</span> <span class="s1">'weather-station-17'</span><span class="p">::</span><span class="nb">text</span><span class="p">)</span>
           <span class="k">Rows</span> <span class="n">Removed</span> <span class="k">by</span> <span class="n">Filter</span><span class="p">:</span> <span class="mi">99</span>
   <span class="o">-&gt;</span>  <span class="n">Gather</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">12239</span><span class="p">.</span><span class="mi">64</span><span class="p">..</span><span class="mi">12239</span><span class="p">.</span><span class="mi">85</span> <span class="k">rows</span><span class="o">=</span><span class="mi">2</span> <span class="n">width</span><span class="o">=</span><span class="mi">8</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">17</span><span class="p">.</span><span class="mi">825</span><span class="p">..</span><span class="mi">20</span><span class="p">.</span><span class="mi">499</span> <span class="k">rows</span><span class="o">=</span><span class="mi">3</span> <span class="n">loops</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
         <span class="n">Workers</span> <span class="n">Planned</span><span class="p">:</span> <span class="mi">2</span>
         <span class="n">Workers</span> <span class="n">Launched</span><span class="p">:</span> <span class="mi">2</span>
         <span class="o">-&gt;</span>  <span class="k">Partial</span> <span class="k">Aggregate</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">11239</span><span class="p">.</span><span class="mi">64</span><span class="p">..</span><span class="mi">11239</span><span class="p">.</span><span class="mi">65</span> <span class="k">rows</span><span class="o">=</span><span class="mi">1</span> <span class="n">width</span><span class="o">=</span><span class="mi">8</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">12</span><span class="p">.</span><span class="mi">684</span><span class="p">..</span><span class="mi">12</span><span class="p">.</span><span class="mi">688</span> <span class="k">rows</span><span class="o">=</span><span class="mi">1</span> <span class="n">loops</span><span class="o">=</span><span class="mi">3</span><span class="p">)</span>
               <span class="o">-&gt;</span>  <span class="n">Parallel</span> <span class="n">Append</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">42</span><span class="p">..</span><span class="mi">11073</span><span class="p">.</span><span class="mi">00</span> <span class="k">rows</span><span class="o">=</span><span class="mi">66656</span> <span class="n">width</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">045</span><span class="p">..</span><span class="mi">10</span><span class="p">.</span><span class="mi">572</span> <span class="k">rows</span><span class="o">=</span><span class="mi">53333</span> <span class="n">loops</span><span class="o">=</span><span class="mi">3</span><span class="p">)</span>
                     <span class="o">-&gt;</span>  <span class="n">Parallel</span> <span class="k">Index</span> <span class="k">Only</span> <span class="n">Scan</span> <span class="k">using</span> <span class="n">weather_report_2025_03_06_weather_station_id_received_at_id_d_idx</span> <span class="k">on</span> <span class="n">weather_report_2025_03_06</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">42</span><span class="p">..</span><span class="mi">671</span><span class="p">.</span><span class="mi">23</span> <span class="k">rows</span><span class="o">=</span><span class="mi">5882</span> <span class="n">width</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">041</span><span class="p">..</span><span class="mi">1</span><span class="p">.</span><span class="mi">431</span> <span class="k">rows</span><span class="o">=</span><span class="mi">10000</span> <span class="n">loops</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
                           <span class="k">Index</span> <span class="n">Cond</span><span class="p">:</span> <span class="p">((</span><span class="n">weather_station_id</span> <span class="o">=</span> <span class="p">(</span><span class="n">InitPlan</span> <span class="mi">1</span><span class="p">).</span><span class="n">col1</span><span class="p">)</span> <span class="k">AND</span> <span class="p">(</span><span class="n">received_at</span> <span class="o">&gt;=</span> <span class="s1">'2025-03-06 00:00:00'</span><span class="p">::</span><span class="nb">timestamp</span> <span class="k">without</span> <span class="nb">time</span> <span class="k">zone</span><span class="p">))</span>
                           <span class="n">Heap</span> <span class="n">Fetches</span><span class="p">:</span> <span class="mi">0</span>
                     <span class="o">-&gt;</span>  <span class="n">Parallel</span> <span class="k">Index</span> <span class="k">Only</span> <span class="n">Scan</span> <span class="k">using</span> <span class="n">weather_report_2025_03_07_weather_station_id_received_at_id_d_idx</span> <span class="k">on</span> <span class="n">weather_report_2025_03_07</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">42</span><span class="p">..</span><span class="mi">671</span><span class="p">.</span><span class="mi">23</span> <span class="k">rows</span><span class="o">=</span><span class="mi">5882</span> <span class="n">width</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">040</span><span class="p">..</span><span class="mi">1</span><span class="p">.</span><span class="mi">434</span> <span class="k">rows</span><span class="o">=</span><span class="mi">10000</span> <span class="n">loops</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
                           <span class="k">Index</span> <span class="n">Cond</span><span class="p">:</span> <span class="p">((</span><span class="n">weather_station_id</span> <span class="o">=</span> <span class="p">(</span><span class="n">InitPlan</span> <span class="mi">1</span><span class="p">).</span><span class="n">col1</span><span class="p">)</span> <span class="k">AND</span> <span class="p">(</span><span class="n">received_at</span> <span class="o">&gt;=</span> <span class="s1">'2025-03-06 00:00:00'</span><span class="p">::</span><span class="nb">timestamp</span> <span class="k">without</span> <span class="nb">time</span> <span class="k">zone</span><span class="p">))</span>
                           <span class="n">Heap</span> <span class="n">Fetches</span><span class="p">:</span> <span class="mi">0</span>
                     <span class="p">[...]</span> <i class="conum" data-value="1"></i><b>(1)</b>
                     <span class="o">-&gt;</span>  <span class="n">Parallel</span> <span class="k">Index</span> <span class="k">Only</span> <span class="n">Scan</span> <span class="k">using</span> <span class="n">weather_report_2025_03_21_weather_station_id_received_at_id_d_idx</span> <span class="k">on</span> <span class="n">weather_report_2025_03_21</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">42</span><span class="p">..</span><span class="mi">671</span><span class="p">.</span><span class="mi">23</span> <span class="k">rows</span><span class="o">=</span><span class="mi">5882</span> <span class="n">width</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">052</span><span class="p">..</span><span class="mi">2</span><span class="p">.</span><span class="mi">213</span> <span class="k">rows</span><span class="o">=</span><span class="mi">10000</span> <span class="n">loops</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
                           <span class="k">Index</span> <span class="n">Cond</span><span class="p">:</span> <span class="p">((</span><span class="n">weather_station_id</span> <span class="o">=</span> <span class="p">(</span><span class="n">InitPlan</span> <span class="mi">1</span><span class="p">).</span><span class="n">col1</span><span class="p">)</span> <span class="k">AND</span> <span class="p">(</span><span class="n">received_at</span> <span class="o">&gt;=</span> <span class="s1">'2025-03-06 00:00:00'</span><span class="p">::</span><span class="nb">timestamp</span> <span class="k">without</span> <span class="nb">time</span> <span class="k">zone</span><span class="p">))</span>
                           <span class="n">Heap</span> <span class="n">Fetches</span><span class="p">:</span> <span class="mi">0</span>
 <span class="n">Planning</span> <span class="nb">Time</span><span class="p">:</span> <span class="mi">0</span><span class="p">.</span><span class="mi">931</span> <span class="n">ms</span>
 <span class="n">Execution</span> <span class="nb">Time</span><span class="p">:</span> <span class="mi">20</span><span class="p">.</span><span class="mi">670</span> <span class="n">ms</span></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>The execution plan has been cropped for readability.
The omitted section involves scanning data from 13 additional partitions.</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>Well, partitioning didn&#8217;t really help in our case.
The "fetch" query is slightly slower, although still extremely fast.
The execution time of the "count" query improved a bit - from 22 ms to 20 ms - which may or may not be a meaningful difference.
Execution times can vary between runs of <code>EXPLAIN ANALYZE</code> and only proper <a href="https://www.postgresql.org/docs/current/pgbench.html" target="_blank" rel="noopener">benchmarking</a> will confirm whether this is a real performance gain.</p>
</div>
<div class="paragraph">
<p>That doesn&#8217;t mean partitioning is not worth the effort, but it usually makes sense for larger tables.
In our case, the <code>weather_report</code> table contains only 30 million records which isn&#8217;t quite enough to see real benefits from partitioning.
You might start noticing small performance gains around 100 million records, with more significant improvements as your table grows to several hundred million or even a billion rows.</p>
</div>
<div class="admonitionblock tip">
<table>
<tr>
<td class="icon">
<i class="fa icon-tip" title="Tip"></i>
</td>
<td class="content">
<div class="paragraph">
<p>Partitioning comes with extra complexity, such as dealing with constraints and maintaining partitions and indexes.
Make sure you&#8217;ve explored all indexing strategies before deciding to partition your tables.</p>
</div>
</td>
</tr>
</table>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="fine-tuning-statistics">Fine-tuning statistics</h2>
<div class="sectionbody">
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
<div class="paragraph">
<p>PostgreSQL uses <a href="https://www.postgresql.org/docs/current/sql-analyze.html" target="_blank" rel="noopener">ANALYZE</a> to collect table <a href="https://www.postgresql.org/docs/current/planner-stats.html" target="_blank" rel="noopener">statistics</a> and help the query planner choose the most efficient way to run queries.</p>
</div>
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>By default, PostgreSQL analyzes tables using the <a href="https://www.postgresql.org/docs/current/runtime-config-query.html#GUC-DEFAULT-STATISTICS-TARGET" target="_blank" rel="noopener">default_statistics_target</a> setting, which defaults to 100.
You can change this value globally or tweak it for specific columns if needed.
Before you do, keep in mind that <code>ANALYZE</code> samples approximately <code>300 x statistics_target</code> rows.
With the default configuration, PostgreSQL samples around 30,000 rows.</p>
</div>
<div class="paragraph">
<p>Here&#8217;s how statistics can be changed for a specific column:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="sql"><span class="k">alter</span> <span class="k">table</span> <span class="n">weather_report</span>
<span class="k">alter</span> <span class="k">column</span> <span class="n">received_at</span> <span class="k">set</span> <span class="k">statistics</span> <span class="mi">1000</span><span class="p">;</span> <i class="conum" data-value="1"></i><b>(1)</b></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>After this change, <code>ANALYZE</code> will sample approximately 300,000 rows from the <code>weather_report</code> table.</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>Increasing statistics on a column can help the query planner generate better execution plans and speed up queries, but it will also make <code>ANALYZE</code> slower.
Consider it when:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>The column has many distinct values (e.g. UUIDs, timestamps).</p>
</li>
<li>
<p>The column is used frequently in <code>WHERE</code> clauses with highly selective filters.</p>
</li>
<li>
<p>The planner misestimates row counts, leading to poor query plans.</p>
</li>
</ul>
</div>
<div class="listingblock">
<div class="title">Misestimation of row counts in an execution plan</div>
<div class="content">
<pre class="rouge highlight"><code data-lang="sql"><span class="p">[...]</span>
   <span class="o">-&gt;</span>  <span class="n">Nested</span> <span class="n">Loop</span>  <span class="p">(</span><span class="n">cost</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">56</span><span class="p">..</span><span class="mi">13949593</span><span class="p">.</span><span class="mi">89</span> <span class="k">rows</span><span class="o">=</span><span class="mi">160683</span> <span class="n">width</span><span class="o">=</span><span class="mi">57</span><span class="p">)</span> <span class="p">(</span><span class="n">actual</span> <span class="nb">time</span><span class="o">=</span><span class="mi">0</span><span class="p">.</span><span class="mi">210</span><span class="p">..</span><span class="mi">63</span><span class="p">.</span><span class="mi">221</span> <span class="k">rows</span><span class="o">=</span><span class="mi">900</span> <span class="n">loops</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span> <i class="conum" data-value="1"></i><b>(1)</b>
<span class="p">[...]</span></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>The query planner estimated 160,683 rows but the actual execution only returned 900 rows.</td>
</tr>
</table>
</div>
<div class="admonitionblock tip">
<table>
<tr>
<td class="icon">
<i class="fa icon-tip" title="Tip"></i>
</td>
<td class="content">
<div class="paragraph">
<p>Always run <code>ANALYZE</code> after changing statistics to apply the updates.</p>
</div>
</td>
</tr>
</table>
</div>
</div>
</div>
<div class="sect1">
<h2 id="increasing-the-work-memory">Increasing the work memory</h2>
<div class="sectionbody">
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
<div class="paragraph">
<p>The <a href="https://www.postgresql.org/docs/current/runtime-config-resource.html" target="_blank" rel="noopener">work memory</a> is the amount of memory PostgreSQL can use for certain operations within a query such as sorting, hashing and aggregations, before spilling data to disk.
Increasing it can improve performance by reducing expensive disk I/O.
The default <code>work_mem</code> setting is 4MB per query operation.</p>
</div>
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>The work memory can be increased at different levels:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="sql"><span class="k">ALTER</span> <span class="k">SYSTEM</span> <span class="k">SET</span> <span class="n">work_mem</span> <span class="o">=</span> <span class="s1">'128MB'</span><span class="p">;</span> <i class="conum" data-value="1"></i><b>(1)</b>
<span class="k">ALTER</span> <span class="k">ROLE</span> <span class="n">gwenneg</span> <span class="k">SET</span> <span class="n">work_mem</span> <span class="o">=</span> <span class="s1">'128MB'</span><span class="p">;</span> <i class="conum" data-value="2"></i><b>(2)</b>
<span class="k">SET</span> <span class="n">work_mem</span> <span class="o">=</span> <span class="s1">'128MB'</span><span class="p">;</span> <i class="conum" data-value="3"></i><b>(3)</b>
<span class="k">SET</span> <span class="k">LOCAL</span> <span class="n">work_mem</span> <span class="o">=</span> <span class="s1">'128MB'</span><span class="p">;</span> <i class="conum" data-value="4"></i><b>(4)</b></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>This permanently changes the work memory for all sessions and queries.
Run <code>SELECT pg_reload_conf();</code> afterward to apply the change.</td>
</tr>
<tr>
<td><i class="conum" data-value="2"></i><b>2</b></td>
<td>This permanently changes the work memory for a specific role or user.</td>
</tr>
<tr>
<td><i class="conum" data-value="3"></i><b>3</b></td>
<td>This changes the work memory for the current session only.</td>
</tr>
<tr>
<td><i class="conum" data-value="4"></i><b>4</b></td>
<td>This changes the work memory for the current transaction only.</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>If you see <code>external merge</code> or <code>disk batches</code> in an execution plan, it means PostgreSQL had to rely on disk instead of keeping operations in memory.
That&#8217;s how you know the work memory could be increased.</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="sql"><span class="p">[...]</span>
<span class="n">Sort</span> <span class="k">Method</span><span class="p">:</span> <span class="k">external</span> <span class="n">merge</span>  <span class="n">Disk</span><span class="p">:</span> <span class="mi">10240</span><span class="n">kB</span>
<span class="p">[...]</span>
<span class="n">Hash</span> <span class="k">Join</span>
  <span class="n">Hash</span> <span class="n">Batches</span><span class="p">:</span> <span class="mi">32</span>  <span class="n">Disk</span> <span class="n">Batches</span><span class="p">:</span> <span class="mi">8</span>
<span class="p">[...]</span></code></pre>
</div>
</div>
<div class="admonitionblock warning">
<table>
<tr>
<td class="icon">
<i class="fa icon-warning" title="Warning"></i>
</td>
<td class="content">
<div class="paragraph">
<p>Setting <code>work_mem</code> too high can significantly increase memory usage, especially when multiple queries run in parallel, potentially leading to out-of-memory errors.</p>
</div>
</td>
</tr>
</table>
</div>
</div>
</div>
<div class="sect1">
<h2 id="conclusion">Conclusion</h2>
<div class="sectionbody">
<div class="paragraph">
<p>There are other ways to optimize PostgreSQL query performance.
<a href="https://www.postgresql.org/docs/current/indexes-partial.html" target="_blank" rel="noopener">Partial indexes</a> and tuning <a href="https://www.postgresql.org/docs/current/wal-configuration.html" target="_blank" rel="noopener">WAL settings</a>, for example, can be powerful tools depending on your workload.
But this post should already give you a solid foundation with some of the most impactful techniques.</p>
</div>
<div class="paragraph">
<p>Thanks for reading!
Hopefully, you’ve learned a thing or two that you can apply in your own environment to make your queries faster.
If you’ve got tips or experiences to share, I’d love to hear them!</p>
</div>
<div class="paragraph">
<p>Happy optimizing!</p>
</div>
</div>
</div>]]></content><author><name>Gwenneg Lepage</name></author><category term="execution plan" /><category term="indexing" /><category term="performances" /><category term="postgres" /><category term="sql" /><summary type="html"><![CDATA[PostgreSQL is pretty smart at running queries, but sometimes it needs a little help to hit top speed.]]></summary></entry><entry><title type="html">Pausing Kafka at run time with Quarkus</title><link href="https://gwenneg.github.io/2025/02/09/pausing-kafka-at-run-time.html" rel="alternate" type="text/html" title="Pausing Kafka at run time with Quarkus" /><published>2025-02-09T00:00:00+00:00</published><updated>2025-02-09T00:00:00+00:00</updated><id>https://gwenneg.github.io/2025/02/09/pausing-kafka-at-run-time</id><content type="html" xml:base="https://gwenneg.github.io/2025/02/09/pausing-kafka-at-run-time.html"><![CDATA[<div id="preamble">
<div class="sectionbody">
<div class="paragraph">
<p>Pausing the consumption of Kafka messages can help perform maintenance, debug issues without processing new messages or prevent overwhelming downstream systems.</p>
</div>
<div class="paragraph">
<p>In a Quarkus app, that can be done by <a href="https://quarkus.io/guides/messaging#enabledisable-channels" target="_blank" rel="noopener">disabling a Reactive Messaging channel</a> or by <a href="https://quarkus.io/guides/kafka#kafka-bare-clients" target="_blank" rel="noopener">directly interacting with the Kafka client</a> but both approaches have their limitations.
In this post, I&#8217;ll show you a newer and better option which is available starting Quarkus 3.13: <a href="https://smallrye.io/smallrye-reactive-messaging/4.26.0/concepts/pausable-channels/" target="_blank" rel="noopener">Pausable Channels</a> from SmallRye Reactive Messaging.</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="enabling-pausable-channels">Enabling pausable channels</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Channels from SmallRye Reactive Messaging are <em>not</em> pausable by default.
This can be changed from the Quarkus configuration:</p>
</div>
<div class="listingblock">
<div class="title">application.properties</div>
<div class="content">
<pre class="rouge highlight"><code data-lang="properties"><span class="py">mp.messaging.incoming.my-channel.pausable</span><span class="p">=</span><span class="s">true </span><i class="conum" data-value="1"></i><b>(1)</b>
<span class="py">mp.messaging.incoming.my-channel.initially-paused</span><span class="p">=</span><span class="s">true </span><i class="conum" data-value="2"></i><b>(2)</b></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>The channel named <code>my-channel</code> is pausable.</td>
</tr>
<tr>
<td><i class="conum" data-value="2"></i><b>2</b></td>
<td>Pausable channels are <em>not</em> paused by default when the application starts.
To modify this behavior, set this configuration property to <code>true</code>.</td>
</tr>
</table>
</div>
</div>
</div>
<div class="sect1">
<h2 id="creating-a-channel-flow-controller">Creating a channel flow controller</h2>
<div class="sectionbody">
<div class="paragraph">
<p>This post will demonstrate two different ways to interact with the <code>PausableChannel</code> API.
First, let&#8217;s create a CDI bean that will be injected in the code examples in the following sections.</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="java"><span class="kn">import</span> <span class="nn">io.quarkus.logging.Log</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">io.smallrye.reactive.messaging.ChannelRegistry</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">io.smallrye.reactive.messaging.PausableChannel</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">jakarta.enterprise.context.ApplicationScoped</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">jakarta.inject.Inject</span><span class="o">;</span>

<span class="nd">@ApplicationScoped</span>
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">ChannelFlowController</span> <span class="o">{</span>

    <span class="nd">@Inject</span>
    <span class="nc">ChannelRegistry</span> <span class="n">channelRegistry</span><span class="o">;</span>

    <span class="kd">public</span> <span class="kt">void</span> <span class="nf">pause</span><span class="o">(</span><span class="nc">String</span> <span class="n">channel</span><span class="o">)</span> <span class="o">{</span>
        <span class="nc">PausableChannel</span> <span class="n">pausableChannel</span> <span class="o">=</span> <span class="n">getPausableChannel</span><span class="o">(</span><span class="n">channel</span><span class="o">);</span>
        <span class="k">if</span> <span class="o">(!</span><span class="n">pausableChannel</span><span class="o">.</span><span class="na">isPaused</span><span class="o">())</span> <span class="o">{</span>
            <span class="n">pausableChannel</span><span class="o">.</span><span class="na">pause</span><span class="o">();</span>
            <span class="nc">Log</span><span class="o">.</span><span class="na">infof</span><span class="o">(</span><span class="s">"Paused channel: %s"</span><span class="o">,</span> <span class="n">channel</span><span class="o">);</span>
        <span class="o">}</span>
    <span class="o">}</span>

    <span class="kd">public</span> <span class="kt">void</span> <span class="nf">resume</span><span class="o">(</span><span class="nc">String</span> <span class="n">channel</span><span class="o">)</span> <span class="o">{</span>
        <span class="nc">PausableChannel</span> <span class="n">pausableChannel</span> <span class="o">=</span> <span class="n">getPausableChannel</span><span class="o">(</span><span class="n">channel</span><span class="o">);</span>
        <span class="k">if</span> <span class="o">(</span><span class="n">pausableChannel</span><span class="o">.</span><span class="na">isPaused</span><span class="o">())</span> <span class="o">{</span>
            <span class="n">pausableChannel</span><span class="o">.</span><span class="na">resume</span><span class="o">();</span>
            <span class="nc">Log</span><span class="o">.</span><span class="na">infof</span><span class="o">(</span><span class="s">"Resumed channel: %s"</span><span class="o">,</span> <span class="n">channel</span><span class="o">);</span>
        <span class="o">}</span>
    <span class="o">}</span>

    <span class="kd">private</span> <span class="nc">PausableChannel</span> <span class="nf">getPausableChannel</span><span class="o">(</span><span class="nc">String</span> <span class="n">channel</span><span class="o">)</span> <span class="o">{</span>
        <span class="nc">PausableChannel</span> <span class="n">pausableChannel</span> <span class="o">=</span> <span class="n">channelRegistry</span><span class="o">.</span><span class="na">getPausable</span><span class="o">(</span><span class="n">channel</span><span class="o">);</span>
        <span class="k">if</span> <span class="o">(</span><span class="n">pausableChannel</span> <span class="o">==</span> <span class="kc">null</span><span class="o">)</span> <span class="o">{</span>
            <span class="k">throw</span> <span class="k">new</span> <span class="nf">IllegalArgumentException</span><span class="o">(</span><span class="s">"Channel not found or not marked as pausable from the Quarkus configuration"</span><span class="o">);</span>
        <span class="o">}</span> <span class="k">else</span> <span class="o">{</span>
            <span class="k">return</span> <span class="n">pausableChannel</span><span class="o">;</span>
        <span class="o">}</span>
    <span class="o">}</span>
<span class="o">}</span></code></pre>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="pausing-a-channel-from-a-rest-api">Pausing a channel from a REST API</h2>
<div class="sectionbody">
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
<div class="paragraph">
<p>This approach is available as code in the <a href="https://github.com/gwenneg/blog-pausing-kafka-at-run-time" target="_blank" rel="noopener">gwenneg/blog-pausing-kafka-at-run-time</a> repository.</p>
</div>
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>The <code>PausableChannel</code> API can easily be exposed through a REST API which would typically be restricted to administrators of the application.</p>
</div>
<div class="admonitionblock caution">
<table>
<tr>
<td class="icon">
<i class="fa icon-caution" title="Caution"></i>
</td>
<td class="content">
<div class="paragraph">
<p>Make sure that any REST endpoints you introduce to expose the <code>PausableChannel</code> API are secured and accessible only to authorized users.</p>
</div>
</td>
</tr>
</table>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="java"><span class="kn">import</span> <span class="nn">jakarta.inject.Inject</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">jakarta.ws.rs.PUT</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">jakarta.ws.rs.Path</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">jakarta.ws.rs.core.Response</span><span class="o">;</span>

<span class="nd">@Path</span><span class="o">(</span><span class="s">"/channels"</span><span class="o">)</span>
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">ChannelResource</span> <span class="o">{</span> <i class="conum" data-value="1"></i><b>(1)</b>

    <span class="nd">@Inject</span>
    <span class="nc">ChannelFlowController</span> <span class="n">channelFlowController</span><span class="o">;</span>

    <span class="nd">@PUT</span>
    <span class="nd">@Path</span><span class="o">(</span><span class="s">"/pause"</span><span class="o">)</span>
    <span class="kd">public</span> <span class="nc">Response</span> <span class="nf">pause</span><span class="o">(</span><span class="nc">String</span> <span class="n">channel</span><span class="o">)</span> <span class="o">{</span>
        <span class="k">try</span> <span class="o">{</span>
            <span class="n">channelFlowController</span><span class="o">.</span><span class="na">pause</span><span class="o">(</span><span class="n">channel</span><span class="o">);</span>
            <span class="k">return</span> <span class="nc">Response</span><span class="o">.</span><span class="na">ok</span><span class="o">().</span><span class="na">build</span><span class="o">();</span>
        <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">IllegalArgumentException</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span>
            <span class="k">return</span> <span class="nc">Response</span><span class="o">.</span><span class="na">status</span><span class="o">(</span><span class="nc">Response</span><span class="o">.</span><span class="na">Status</span><span class="o">.</span><span class="na">NOT_FOUND</span><span class="o">)</span>
                <span class="o">.</span><span class="na">entity</span><span class="o">(</span><span class="n">e</span><span class="o">.</span><span class="na">getMessage</span><span class="o">())</span>
                <span class="o">.</span><span class="na">build</span><span class="o">();</span> <i class="conum" data-value="2"></i><b>(2)</b>
        <span class="o">}</span>
    <span class="o">}</span>

    <span class="nd">@PUT</span>
    <span class="nd">@Path</span><span class="o">(</span><span class="s">"/resume"</span><span class="o">)</span>
    <span class="kd">public</span> <span class="nc">Response</span> <span class="nf">resume</span><span class="o">(</span><span class="nc">String</span> <span class="n">channel</span><span class="o">)</span> <span class="o">{</span>
        <span class="k">try</span> <span class="o">{</span>
            <span class="n">channelFlowController</span><span class="o">.</span><span class="na">resume</span><span class="o">(</span><span class="n">channel</span><span class="o">);</span>
            <span class="k">return</span> <span class="nc">Response</span><span class="o">.</span><span class="na">ok</span><span class="o">().</span><span class="na">build</span><span class="o">();</span>
        <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">IllegalArgumentException</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span>
            <span class="k">return</span> <span class="nc">Response</span><span class="o">.</span><span class="na">status</span><span class="o">(</span><span class="nc">Response</span><span class="o">.</span><span class="na">Status</span><span class="o">.</span><span class="na">NOT_FOUND</span><span class="o">)</span>
                <span class="o">.</span><span class="na">entity</span><span class="o">(</span><span class="n">e</span><span class="o">.</span><span class="na">getMessage</span><span class="o">())</span>
                <span class="o">.</span><span class="na">build</span><span class="o">();</span> <i class="conum" data-value="2"></i><b>(2)</b>
        <span class="o">}</span>
    <span class="o">}</span>
<span class="o">}</span></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>Make sure that the endpoints in this class are secured and accessible only to authorized users.</td>
</tr>
<tr>
<td><i class="conum" data-value="2"></i><b>2</b></td>
<td>If the <code>channel</code> argument does not identify any existing channel, or if that channel exists but is not marked as pausable from the Quarkus configuration, an HTTP 404 error will be returned.</td>
</tr>
</table>
</div>
</div>
</div>
<div class="sect1">
<h2 id="pausing-a-channel-from-unleash">Pausing a channel from Unleash</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Sometimes, exposing a REST endpoint to interact with the <code>PausableChannel</code> API is not an option.
Here&#8217;s an alternative based on <a href="https://www.getunleash.io/" target="_blank" rel="noopener">Unleash</a> and the <a href="https://docs.quarkiverse.io/quarkus-unleash/dev/index.html" target="_blank" rel="noopener">quarkus-unleash</a> extension.
It&#8217;s very similar to my other post <a href="/2024/04/03/changing-loggers-level-from-unleash.html" target="_blank" rel="noopener">Changing the Quarkus loggers level from Unleash</a> which contains a lot more details than this post.</p>
</div>
<div class="sect2">
<h3 id="passing-the-channel-configuration-from-unleash-to-quarkus">Passing the channel configuration from Unleash to Quarkus</h3>
<div class="paragraph">
<p>In Unleash, each feature toggle can be associated with variants, either <a href="https://docs.getunleash.io/reference/feature-toggle-variants" target="_blank" rel="noopener">directly</a> (deprecated) or <a href="https://docs.getunleash.io/reference/strategy-variants" target="_blank" rel="noopener">through an activation strategy</a> (recommended).
We&#8217;ll use a variant with a JSON payload to pass data from Unleash to Quarkus and pause or resume a channel:</p>
</div>
<div class="imageblock">
<div class="content">
<img src="/assets/images/posts/pausing-kafka-at-run-time/variant.png" alt="Unleash variant">
</div>
</div>
</div>
<div class="sect2">
<h3 id="deserializing-the-channel-configuration">Deserializing the channel configuration</h3>
<div class="paragraph">
<p>The variant payload needs to be deserialized before it can be used to pause or resume a channel.
Here&#8217;s the data structure we&#8217;ll use for that:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">KafkaChannelConfig</span> <span class="o">{</span>
    <span class="kd">public</span> <span class="nc">String</span> <span class="n">hostName</span><span class="o">;</span>
    <span class="kd">public</span> <span class="nc">String</span> <span class="n">channel</span><span class="o">;</span>
    <span class="kd">public</span> <span class="nc">Boolean</span> <span class="n">paused</span><span class="o">;</span>
<span class="o">}</span></code></pre>
</div>
</div>
</div>
<div class="sect2">
<h3 id="applying-the-channel-configuration-automatically">Applying the channel configuration automatically</h3>
<div class="paragraph">
<p>Now that the channel configuration can be modified from Unleash and passed Quarkus, how do we pause or resume the channel whenever its configuration is changed?
We&#8217;ll do that with the <a href="https://docs.getunleash.io/reference/sdks/java#subscriber-api" target="_blank" rel="noopener">Subscriber API</a> from Unleash and subscribe to the <code>FeatureToggleResponse</code> event, which is emitted when the Unleash client fetches toggles from the server.</p>
</div>
<div class="paragraph">
<p>Using the Subscriber API with the <a href="https://docs.quarkiverse.io/quarkus-unleash/dev/index.html" target="_blank" rel="noopener">quarkus-unleash</a> extension is extremely simple.
<code>UnleashSubscriber</code> needs to be implemented in a CDI bean and that&#8217;s it!
The extension will pass the bean to the Unleash client builder automatically.</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="java"><span class="kn">import</span> <span class="nn">com.fasterxml.jackson.core.JsonProcessingException</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">com.fasterxml.jackson.databind.ObjectMapper</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">io.getunleash.Unleash</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">io.getunleash.Variant</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">io.getunleash.event.UnleashSubscriber</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">io.getunleash.repository.FeatureToggleResponse</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">io.getunleash.variant.Payload</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">io.quarkus.logging.Log</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">jakarta.enterprise.context.ApplicationScoped</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">jakarta.inject.Inject</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.eclipse.microprofile.config.inject.ConfigProperty</span><span class="o">;</span>

<span class="kn">import</span> <span class="nn">java.util.Optional</span><span class="o">;</span>

<span class="kn">import</span> <span class="nn">static</span> <span class="n">io</span><span class="o">.</span><span class="na">getunleash</span><span class="o">.</span><span class="na">repository</span><span class="o">.</span><span class="na">FeatureToggleResponse</span><span class="o">.</span><span class="na">Status</span><span class="o">.</span><span class="na">CHANGED</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">static</span> <span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">Boolean</span><span class="o">.</span><span class="na">TRUE</span><span class="o">;</span>

<span class="nd">@ApplicationScoped</span>
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">KafkaChannelManager</span> <span class="kd">implements</span> <span class="nc">UnleashSubscriber</span> <span class="o">{</span>

    <span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="nc">String</span> <span class="no">UNLEASH_TOGGLE_NAME</span> <span class="o">=</span> <span class="s">"my-app.kafka-channels"</span><span class="o">;</span>

    <span class="nd">@ConfigProperty</span><span class="o">(</span><span class="n">name</span> <span class="o">=</span> <span class="s">"host-name"</span><span class="o">,</span> <span class="n">defaultValue</span> <span class="o">=</span> <span class="s">"localhost"</span><span class="o">)</span> <i class="conum" data-value="1"></i><b>(1)</b>
    <span class="nc">String</span> <span class="n">hostName</span><span class="o">;</span>

    <span class="nd">@Inject</span>
    <span class="nc">Unleash</span> <span class="n">unleash</span><span class="o">;</span>

    <span class="nd">@Inject</span>
    <span class="nc">ObjectMapper</span> <span class="n">objectMapper</span><span class="o">;</span>

    <span class="nd">@Inject</span>
    <span class="nc">ChannelFlowController</span> <span class="n">channelFlowController</span><span class="o">;</span>

    <span class="nd">@Override</span>
    <span class="kd">public</span> <span class="kt">void</span> <span class="nf">togglesFetched</span><span class="o">(</span><span class="nc">FeatureToggleResponse</span> <span class="n">toggleResponse</span><span class="o">)</span> <span class="o">{</span> <i class="conum" data-value="2"></i><b>(2)</b>
        <span class="k">if</span> <span class="o">(</span><span class="n">toggleResponse</span><span class="o">.</span><span class="na">getStatus</span><span class="o">()</span> <span class="o">==</span> <span class="no">CHANGED</span><span class="o">)</span> <span class="o">{</span> <i class="conum" data-value="3"></i><b>(3)</b>
            <span class="nc">KafkaChannelConfig</span><span class="o">[]</span> <span class="n">kafkaChannelConfigs</span> <span class="o">=</span> <span class="n">getKafkaChannelConfigs</span><span class="o">();</span>
            <span class="k">for</span> <span class="o">(</span><span class="nc">KafkaChannelConfig</span> <span class="n">kafkaChannelConfig</span> <span class="o">:</span> <span class="n">kafkaChannelConfigs</span><span class="o">)</span> <span class="o">{</span>
                <span class="k">try</span> <span class="o">{</span>
                    <span class="k">if</span> <span class="o">(</span><span class="n">shouldThisHostBeUpdated</span><span class="o">(</span><span class="n">kafkaChannelConfig</span><span class="o">))</span> <span class="o">{</span>
                        <span class="k">if</span> <span class="o">(</span><span class="no">TRUE</span><span class="o">.</span><span class="na">equals</span><span class="o">(</span><span class="n">kafkaChannelConfig</span><span class="o">.</span><span class="na">paused</span><span class="o">))</span> <span class="o">{</span>
                            <span class="n">channelFlowController</span><span class="o">.</span><span class="na">pause</span><span class="o">(</span><span class="n">kafkaChannelConfig</span><span class="o">.</span><span class="na">channel</span><span class="o">);</span>
                        <span class="o">}</span> <span class="k">else</span> <span class="o">{</span>
                            <span class="n">channelFlowController</span><span class="o">.</span><span class="na">resume</span><span class="o">(</span><span class="n">kafkaChannelConfig</span><span class="o">.</span><span class="na">channel</span><span class="o">);</span>
                        <span class="o">}</span>
                    <span class="o">}</span>
                <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">Exception</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span>
                    <span class="nc">Log</span><span class="o">.</span><span class="na">error</span><span class="o">(</span><span class="s">"Could not pause or resume a channel"</span><span class="o">,</span> <span class="n">e</span><span class="o">);</span>
                <span class="o">}</span>
            <span class="o">}</span>
        <span class="o">}</span>
    <span class="o">}</span>

    <span class="kd">private</span> <span class="nc">KafkaChannelConfig</span><span class="o">[]</span> <span class="nf">getKafkaChannelConfigs</span><span class="o">()</span> <span class="o">{</span>
        <span class="nc">Variant</span> <span class="n">variant</span> <span class="o">=</span> <span class="n">unleash</span><span class="o">.</span><span class="na">getVariant</span><span class="o">(</span><span class="no">UNLEASH_TOGGLE_NAME</span><span class="o">);</span> <i class="conum" data-value="4"></i><b>(4)</b>
        <span class="k">if</span> <span class="o">(</span><span class="n">variant</span><span class="o">.</span><span class="na">isEnabled</span><span class="o">())</span> <span class="o">{</span> <i class="conum" data-value="5"></i><b>(5)</b>
            <span class="nc">Optional</span><span class="o">&lt;</span><span class="nc">Payload</span><span class="o">&gt;</span> <span class="n">payload</span> <span class="o">=</span> <span class="n">variant</span><span class="o">.</span><span class="na">getPayload</span><span class="o">();</span>
            <span class="k">if</span> <span class="o">(</span><span class="n">payload</span><span class="o">.</span><span class="na">isPresent</span><span class="o">()</span> <span class="o">&amp;&amp;</span> <span class="n">payload</span><span class="o">.</span><span class="na">get</span><span class="o">().</span><span class="na">getType</span><span class="o">().</span><span class="na">equals</span><span class="o">(</span><span class="s">"json"</span><span class="o">)</span> <span class="o">&amp;&amp;</span> <span class="n">payload</span><span class="o">.</span><span class="na">get</span><span class="o">().</span><span class="na">getValue</span><span class="o">()</span> <span class="o">!=</span> <span class="kc">null</span><span class="o">)</span> <span class="o">{</span>
                <span class="k">try</span> <span class="o">{</span>
                    <span class="k">return</span> <span class="n">objectMapper</span><span class="o">.</span><span class="na">readValue</span><span class="o">(</span><span class="n">payload</span><span class="o">.</span><span class="na">get</span><span class="o">().</span><span class="na">getValue</span><span class="o">(),</span> <span class="nc">KafkaChannelConfig</span><span class="o">[].</span><span class="na">class</span><span class="o">);</span>
                <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">JsonProcessingException</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span>
                    <span class="nc">Log</span><span class="o">.</span><span class="na">error</span><span class="o">(</span><span class="s">"Variant payload deserialization failed"</span><span class="o">,</span> <span class="n">e</span><span class="o">);</span>
                <span class="o">}</span>
            <span class="o">}</span>
        <span class="o">}</span>
        <span class="k">return</span> <span class="k">new</span> <span class="nc">KafkaChannelConfig</span><span class="o">[</span><span class="mi">0</span><span class="o">];</span> <i class="conum" data-value="6"></i><b>(6)</b>
    <span class="o">}</span>

    <span class="kd">private</span> <span class="kt">boolean</span> <span class="nf">shouldThisHostBeUpdated</span><span class="o">(</span><span class="nc">KafkaChannelConfig</span> <span class="n">kafkaChannelConfig</span><span class="o">)</span> <span class="o">{</span>
        <span class="k">if</span> <span class="o">(</span><span class="n">kafkaChannelConfig</span><span class="o">.</span><span class="na">hostName</span> <span class="o">==</span> <span class="kc">null</span><span class="o">)</span> <span class="o">{</span>
            <span class="k">return</span> <span class="kc">true</span><span class="o">;</span>
        <span class="o">}</span>
        <span class="k">if</span> <span class="o">(</span><span class="n">kafkaChannelConfig</span><span class="o">.</span><span class="na">hostName</span><span class="o">.</span><span class="na">endsWith</span><span class="o">(</span><span class="s">"*"</span><span class="o">))</span> <span class="o">{</span> <i class="conum" data-value="7"></i><b>(7)</b>
            <span class="k">return</span> <span class="n">hostName</span><span class="o">.</span><span class="na">startsWith</span><span class="o">(</span><span class="n">kafkaChannelConfig</span><span class="o">.</span><span class="na">hostName</span><span class="o">.</span><span class="na">substring</span><span class="o">(</span><span class="mi">0</span><span class="o">,</span> <span class="n">kafkaChannelConfig</span><span class="o">.</span><span class="na">hostName</span><span class="o">.</span><span class="na">length</span><span class="o">()</span> <span class="o">-</span> <span class="mi">1</span><span class="o">));</span>
        <span class="o">}</span> <span class="k">else</span> <span class="o">{</span>
            <span class="k">return</span> <span class="n">hostName</span><span class="o">.</span><span class="na">equals</span><span class="o">(</span><span class="n">kafkaChannelConfig</span><span class="o">.</span><span class="na">hostName</span><span class="o">);</span>
        <span class="o">}</span>
    <span class="o">}</span>
<span class="o">}</span></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>If this code is run from Kubernetes or OpenShift, the generated pod name can be passed through the <code>HOST_NAME</code> environment variable and used here.</td>
</tr>
<tr>
<td><i class="conum" data-value="2"></i><b>2</b></td>
<td>This method is invoked every time the Unleash client fetches toggles from the server.</td>
</tr>
<tr>
<td><i class="conum" data-value="3"></i><b>3</b></td>
<td>We&#8217;ll try to pause or resume channels only if the toggles changed server-side.</td>
</tr>
<tr>
<td><i class="conum" data-value="4"></i><b>4</b></td>
<td>Be careful about the argument passed to <code>Unleash#getVariant</code>: it has to be the toggle name, not the variant name.</td>
</tr>
<tr>
<td><i class="conum" data-value="5"></i><b>5</b></td>
<td><code>variant.isEnabled()</code> will return <code>false</code> if the toggle is disabled in Unleash or if no variants are associated with the toggle directly or through its activation strategy.</td>
</tr>
<tr>
<td><i class="conum" data-value="6"></i><b>6</b></td>
<td>If the method is unable to find a variant payload or if it fails to deserialize that payload for any reasons, an empty <code>KafkaChannelConfig</code> array will be returned.</td>
</tr>
<tr>
<td><i class="conum" data-value="7"></i><b>7</b></td>
<td>This block is used to filter hosts based on a host name prefix.
If you need finer filtering, replacing the current wild card approach with a regular expression could be a good option.</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>Here&#8217;s an example of variant payload that could be consumed by <code>KafkaChannelManager</code>:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="json"><span class="p">[</span><span class="w">
  </span><span class="p">{</span><span class="w">
    </span><span class="nl">"hostName"</span><span class="p">:</span><span class="w"> </span><span class="s2">"amazing-service-7dbbcb4cc-9d9hl"</span><span class="p">,</span><span class="w">
    </span><span class="nl">"channel"</span><span class="p">:</span><span class="w"> </span><span class="s2">"orders"</span><span class="p">,</span><span class="w">
    </span><span class="nl">"paused"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">
  </span><span class="p">},</span><span class="w">
  </span><span class="p">{</span><span class="w">
    </span><span class="nl">"hostName"</span><span class="p">:</span><span class="w"> </span><span class="s2">"awesome-app*"</span><span class="p">,</span><span class="w">
    </span><span class="nl">"channel"</span><span class="p">:</span><span class="w"> </span><span class="s2">"deliveries"</span><span class="p">,</span><span class="w">
    </span><span class="nl">"paused"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="w">
  </span><span class="p">},</span><span class="w">
  </span><span class="p">{</span><span class="w">
    </span><span class="nl">"channel"</span><span class="p">:</span><span class="w"> </span><span class="s2">"events"</span><span class="p">,</span><span class="w">
    </span><span class="nl">"paused"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">
  </span><span class="p">}</span><span class="w">
</span><span class="p">]</span></code></pre>
</div>
</div>
<div class="paragraph">
<p>In that payload:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>the first entry will pause the <code>orders</code> channel of a specific host: <code>amazing-service-7dbbcb4cc-9d9hl</code></p>
</li>
<li>
<p>the second entry will resume the <code>deliveries</code> channel of all hosts whose name starts with <code>awesome-app</code></p>
</li>
<li>
<p>the third entry will pause the <code>events</code> channel of all hosts regardless of their names</p>
</li>
</ul>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="a-temporary-limitation-of-the-pausablechannel-api">A temporary limitation of the PausableChannel API</h2>
<div class="sectionbody">
<div class="paragraph">
<p>The current version of the <code>PausableChannel</code> API doesn&#8217;t handle messages that were already requested before a channel is paused.
As a result, your app might still process a few messages after initiating a pause, before the channel fully stops.
The SmallRye Reactive Messaging team is actively working on an enhancement to address this issue in the near future.</p>
</div>
<div class="paragraph">
<p>Thanks for reading this post. Happy pausing!</p>
</div>
</div>
</div>]]></content><author><name>Gwenneg Lepage</name></author><category term="java" /><category term="kafka" /><category term="quarkus" /><category term="reactive messaging" /><category term="unleash" /><summary type="html"><![CDATA[Learn how the PausableChannel API from SmallRye Reactive Messaging can help you pause the consumption of Kafka messages at run time in a Quarkus app.]]></summary></entry><entry><title type="html">Blogging on GitHub Pages with minimal effort</title><link href="https://gwenneg.github.io/2024/08/17/blogging-with-minimal-effort.html" rel="alternate" type="text/html" title="Blogging on GitHub Pages with minimal effort" /><published>2024-08-17T00:00:00+00:00</published><updated>2024-08-17T00:00:00+00:00</updated><id>https://gwenneg.github.io/2024/08/17/blogging-with-minimal-effort</id><content type="html" xml:base="https://gwenneg.github.io/2024/08/17/blogging-with-minimal-effort.html"><![CDATA[<div id="preamble">
<div class="sectionbody">
<div class="paragraph">
<p><span class="image"><img src="/assets/images/posts/blogging-with-minimal-effort/header.png" alt="Blogging with minimal effort"></span></p>
</div>
<div class="paragraph">
<p>I recently chose to start my own dev blog with one key requirement: posting should require minimal effort.
I spent a significant amount of time researching and testing before I was finally satisfied with the result.
In this post, I&#8217;ll save you that effort and demonstrate the fastest way to create a blog similar to mine.</p>
</div>
<div class="paragraph">
<p>Are you ready to publish your first post in no time?
Let&#8217;s get started!</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="initiating-the-blog-from-a-template-repository">Initiating the blog from a template repository</h2>
<div class="sectionbody">
<div class="paragraph">
<p>First, let&#8217;s set up a GitHub repository for your blog with the <a href="https://github.com/gwenneg/blog-jekyll-asciidoc-template" target="_blank" rel="noopener">template</a> I created from my own blog.
That template depends on <a href="https://jekyllrb.com" target="_blank" rel="noopener">Jekyll</a>, <a href="https://asciidoc.org" target="_blank" rel="noopener">AsciiDoc</a>, the <a href="https://mmistakes.github.io/minimal-mistakes" target="_blank" rel="noopener">Minimal Mistakes</a> theme and GitHub Actions.
Don&#8217;t worry about the technical details yet, we&#8217;ll cover that later in this post.</p>
</div>
<div class="paragraph">
<p>For now, simply follow the steps below to use the <a href="https://github.com/gwenneg/blog-jekyll-asciidoc-template" target="_blank" rel="noopener">template</a>:</p>
</div>
<div class="paragraph">
<p><span class="image"><img src="/assets/images/posts/blogging-with-minimal-effort/use-template.png" alt="How to use the template repository"></span></p>
</div>
<div class="paragraph">
<p>You will be asked for a repository name.
Please make sure the value you provide matches the following pattern:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code>your_github_username.github.io <i class="conum" data-value="1"></i><b>(1)</b></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>Replace <code>your_github_username</code> with your actual GitHub username.
For example, my blog repository is named <code>gwenneg.github.io</code> because my GitHub username is <code>gwenneg</code>.</td>
</tr>
</table>
</div>
<div class="admonitionblock warning">
<table>
<tr>
<td class="icon">
<i class="fa icon-warning" title="Warning"></i>
</td>
<td class="content">
<div class="paragraph">
<p>GitHub Pages only work with public repositories, so please don&#8217;t make your repo private or your blog won&#8217;t be published!</p>
</div>
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>After your repo is created, GitHub will immediately try and fail to deploy your blog from a GitHub Action:</p>
</div>
<div class="paragraph">
<p><span class="image"><img src="/assets/images/posts/blogging-with-minimal-effort/deployment-failure.png" alt="Deployment failure"></span></p>
</div>
<div class="paragraph">
<p>Don&#8217;t worry about that failure.
We&#8217;ll set up the actual deployment of your blog <a href="#deploying">later</a>.</p>
</div>
<div class="paragraph">
<p>We&#8217;re done initiating your blog and it already has its own repo!
Be sure to clone the repo before proceeding to the next section.</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="setting-the-minimal-configuration-before-the-first-deployment">Setting the minimal configuration before the first deployment</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Before your blog can be deployed, you’ll need to make a few adjustments to the <code>_config.yml</code> file in your repo.
Pay special attention to any lines marked with <code># FIXME [&#8230;&#8203;]</code> as these require immediate action.
Most comments should be self-explanatory, and the file also includes many links to relevant sections of the Minimal Mistakes <a href="https://mmistakes.github.io/minimal-mistakes/docs/quick-start-guide/" target="_blank" rel="noopener">documentation</a>.
These links should help address any questions you might have.</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="deploying-the-blog"><a id="deploying"></a> Deploying the blog</h2>
<div class="sectionbody">
<div class="admonitionblock warning">
<table>
<tr>
<td class="icon">
<i class="fa icon-warning" title="Warning"></i>
</td>
<td class="content">
<div class="paragraph">
<p>Once you’ve completed this section, your posts will be visible to anyone who visits your blog.
Be sure you&#8217;re ready for that!
If you prefer to keep a post unpublished, you can do so by adding <code>:page-published: false</code> to the post’s headers.</p>
</div>
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>Until now, the deployment of your blog was intentionally disabled in the <code>jekyll</code> GitHub workflow provided by the template repo.
It&#8217;s time to enable it by deleting <a href="https://github.com/gwenneg/blog-jekyll-asciidoc-template/blob/8d07da46301b34c822500cdeeed70e6046894a7c/.github/workflows/jekyll.yml#L57" target="_blank" rel="noopener">this line</a> in your repo.
Once removed, GitHub will immediately start deploying your blog.
You should soon see a successful GitHub Action run similar to this one:</p>
</div>
<div class="paragraph">
<p><span class="image"><img src="/assets/images/posts/blogging-with-minimal-effort/deployment-success.png" alt="Deployment success"></span></p>
</div>
<div class="paragraph">
<p>Although GitHub just deployed your blog using a GitHub Action, it will still try (and fail) to deploy it "from a branch" as well.
We need to let it know that the blog will exclusively be deployed through the GitHub Action.
Here&#8217;s how you can do that:</p>
</div>
<div class="paragraph">
<p><span class="image"><img src="/assets/images/posts/blogging-with-minimal-effort/deployment-source.png" alt="Deployment success"></span></p>
</div>
<div class="paragraph">
<p>That&#8217;s it for the minimal deployment config of your blog!
From now on, as long as you don&#8217;t modify the <code>jekyll</code> workflow in your repo, GitHub will automatically redeploy the blog whenever you push changes to the <code>main</code> branch.</p>
</div>
<div class="paragraph">
<p>If you want to better understand or customize the deployment configuration, the <a href="https://docs.github.com/en/pages/getting-started-with-github-pages/configuring-a-publishing-source-for-your-github-pages-site" target="_blank" rel="noopener">GitHub doc</a> should have all the information you need.
Finally, <a href="https://github.com/actions/starter-workflows/blob/main/pages/jekyll.yml" target="_blank" rel="noopener">here</a> is the source of the <code>jekyll</code> workflow provided by the template repo.</p>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
<div class="paragraph">
<p>All sections beyond this point are optional reading.
You don&#8217;t <em>have</em> to go through them before publishing your first post.
However, it’s still recommended, as they provide valuable insights into how the blog functions and how you can customize it.</p>
</div>
</td>
</tr>
</table>
</div>
</div>
</div>
<div class="sect1">
<h2 id="a-quick-look-under-the-hood">A quick look under the hood</h2>
<div class="sectionbody">
<div class="sect2">
<h3 id="introducing-jekyll">Introducing Jekyll</h3>
<div class="paragraph">
<p>As I wrote in a previous section, the blog depends on <a href="https://jekyllrb.com" target="_blank" rel="noopener">Jekyll</a>.</p>
</div>
<div class="paragraph">
<p>Jekyll is a widely-used static site generator with built-in support for GitHub Pages.
It&#8217;s not the only option for publishing a blog on GitHub Pages.
There are several other alternatives available.
Since I haven&#8217;t tested those alternatives, I can&#8217;t offer recommendations or comparisons.
In this post, I&#8217;ll focus solely on Jekyll.</p>
</div>
<div class="paragraph">
<p>When I first started learning about Jekyll, something made me a bit uneasy: it requires a Ruby runtime and Ruby commands to build and test the blog locally.
Having never used Ruby before, I was initially hesitant.
Now that I&#8217;m more experienced with Jekyll, I can assure you that it&#8217;s pretty easy to use with zero knowledge of Ruby.
Besides the initial local environment setup which is <a href="https://jekyllrb.com/docs" target="_blank" rel="noopener">well-documented</a>, you will primarily rely on a single shell command to test your blog:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="shell">bundle <span class="nb">exec </span>jekyll serve <i class="conum" data-value="1"></i><b>(1)</b></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>This command builds the blog and makes it available for testing at <a href="http://localhost:4000" class="bare">http://localhost:4000</a>.</td>
</tr>
</table>
</div>
<div class="admonitionblock tip">
<table>
<tr>
<td class="icon">
<i class="fa icon-tip" title="Tip"></i>
</td>
<td class="content">
<div class="paragraph">
<p>Many resources, including the <a href="https://docs.github.com/en/pages/setting-up-a-github-pages-site-with-jekyll/creating-a-github-pages-site-with-jekyll" target="_blank" rel="noopener">GitHub documentation</a>, recommend using the <a href="https://rubygems.org/gems/github-pages" target="_blank" rel="noopener">github-pages</a> gem to build a Jekyll-based blog hosted on GitHub Pages.
Not only is that not required, but it might even be a bad idea in the long run.
Indeed, the <a href="https://github.com/github/pages-gem" target="_blank" rel="noopener">pages-gem</a> repo isn&#8217;t as actively maintained as it used to be.
The <code>github-pages</code> gem also relies on <a href="https://pages.github.com/versions" target="_blank" rel="noopener">outdated dependencies</a> such as Jekyll 3, despite newer versions like Jekyll 4 being available for years.
For these reasons, this blog was intentionally <em>not</em> built using the <code>github-pages</code> gem.
Instead, it depends on the <a href="https://rubygems.org/gems/jekyll" target="_blank" rel="noopener">jekyll</a> gem to take advantage of the latest improvements in Jekyll 4.</p>
</div>
</td>
</tr>
</table>
</div>
</div>
<div class="sect2">
<h3 id="introducing-asciidoc">Introducing AsciiDoc</h3>
<div class="paragraph">
<p>Jekyll natively supports the Markdown markup language but I wanted the blog to also support <a href="https://asciidoc.org" target="_blank" rel="noopener">AsciiDoc</a>, as I strongly prefer it for writing technical posts.
As a result, the blog is set up to handle both markup languages.
If you’re unsure which format is best for you, <a href="https://docs.asciidoctor.org/asciidoc/latest/asciidoc-vs-markdown" target="_blank" rel="noopener">this article</a> from AsciiDoctor Docs can help you make that decision.</p>
</div>
</div>
<div class="sect2">
<h3 id="introducing-minimal-mistakes">Introducing Minimal Mistakes</h3>
<div class="paragraph">
<p>The blog depends on the <a href="https://mmistakes.github.io/minimal-mistakes" target="_blank" rel="noopener">Minimal Mistakes</a> remote theme:</p>
</div>
<div class="listingblock">
<div class="title">_config.yml</div>
<div class="content">
<pre class="rouge highlight"><code data-lang="yaml"><span class="na">remote_theme</span><span class="pi">:</span> <span class="s">mmistakes/minimal-mistakes@4.26.2</span></code></pre>
</div>
</div>
<div class="paragraph">
<p>That theme was created by <a href="https://github.com/mmistakes" target="_blank" rel="noopener">Michael Rose</a>.
It&#8217;s 100% free, highly customizable, fully responsive and extensively <a href="https://mmistakes.github.io/minimal-mistakes/docs/quick-start-guide" target="_blank" rel="noopener">documented</a>.</p>
</div>
<div class="paragraph">
<p>The Minimal Mistakes source code is available in the <a href="https://github.com/mmistakes/minimal-mistakes" target="_blank" rel="noopener">mmistakes/minimal-mistakes</a> repository.</p>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="customizing-the-theme">Customizing the theme</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Before diving into theme customization for your blog, have you explored <a href="https://mmistakes.github.io/minimal-mistakes/docs/configuration/#skin" target="_blank" rel="noopener">all the skins</a> offered by Minimal Mistakes?
If you haven’t made any changes yet, your blog is currently using the <code>default</code> skin:</p>
</div>
<div class="listingblock">
<div class="title">_config.yml</div>
<div class="content">
<pre class="rouge highlight"><code data-lang="yaml"><span class="na">minimal_mistakes_skin</span><span class="pi">:</span> <span class="s">default</span> <i class="conum" data-value="1"></i><b>(1)</b></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>Additional skins are available: <code>air</code>, <code>aqua</code>, <code>contrast</code>, <code>dark</code>, <code>neon</code>, <code>mint</code>, <code>plum</code> and <code>sunrise</code>.</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>Once you&#8217;ve chosen a skin, you may want to further customize your blog&#8217;s style by adding or overriding CSS rules.
While the Minimal Mistakes documentation <a href="https://mmistakes.github.io/minimal-mistakes/docs/stylesheets" target="_blank" rel="noopener">covers this</a> thoroughly, I opted for a different approach.
It may not be as polished as the recommended method, but requires significantly less effort when only minimal changes are needed.</p>
</div>
<div class="paragraph">
<p>Minimal Mistakes provides an empty <a href="https://github.com/mmistakes/minimal-mistakes/blob/master/_includes/head/custom.html" target="_blank" rel="noopener">_includes/head/custom.html</a> file that you can use to customize the HTML content within your blog’s <code>head</code> tag.
That&#8217;s how I introduced my own CSS rules:</p>
</div>
<div class="listingblock">
<div class="title">/_includes/head/custom.html</div>
<div class="content">
<pre class="rouge highlight"><code data-lang="html"><span class="nt">&lt;link</span> <span class="na">rel=</span><span class="s">"stylesheet"</span> <span class="na">href=</span><span class="s">"/assets/css/rouge/base16.monokai.dark.css"</span><span class="nt">&gt;</span>
<span class="nt">&lt;link</span> <span class="na">rel=</span><span class="s">"stylesheet"</span> <span class="na">href=</span><span class="s">"/assets/css/custom.css"</span><span class="nt">&gt;</span> <i class="conum" data-value="1"></i><b>(1)</b></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>This file contains all custom CSS rules applied on top of the Minimal Mistakes skin.</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>You might also consider <a href="https://mmistakes.github.io/minimal-mistakes/docs/overriding-theme-defaults" target="_blank" rel="noopener">customizing the layouts</a> offered by Minimal Mistakes.
However, that requires more effort than I aimed for with this blog, so what you’re reading now uses the default layouts with no custom changes.</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="enabling-comments-in-your-posts"><a id="comments"></a> Enabling comments in your posts</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Comments are not enabled in your blog posts yet.
To enable them, you&#8217;ll need to update the <code>_config.yml</code> file in your repo:</p>
</div>
<div class="listingblock">
<div class="title">_config.yml</div>
<div class="content">
<pre class="rouge highlight"><code data-lang="yaml"><span class="na">defaults</span><span class="pi">:</span>
  <span class="pi">-</span> <span class="na">scope</span><span class="pi">:</span>
      <span class="na">path</span><span class="pi">:</span> <span class="s2">"</span><span class="s">"</span>
      <span class="na">type</span><span class="pi">:</span> <span class="s">posts</span>
    <span class="na">values</span><span class="pi">:</span>
      <span class="na">comments</span><span class="pi">:</span> <span class="kc">false</span> <i class="conum" data-value="1"></i><b>(1)</b></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>Set this value to <code>true</code> to enable comments in all posts.</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>You will also need to select and configure a <a href="https://mmistakes.github.io/minimal-mistakes/docs/configuration/#comments" target="_blank" rel="noopener">comment provider</a> supported by the Minimal Mistakes theme.
I chose Giscus which relies on GitHub Discussions to manage comments.
It&#8217;s open source, free and highly customizable.
If you plan on using that provider, I recommend reviewing <a href="https://mmistakes.github.io/minimal-mistakes/docs/configuration/#giscus-comments" target="_blank" rel="noopener">this documentation</a> from Minimal Mistakes first.</p>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
<div class="paragraph">
<p>The <a href="https://giscus.app" target="_blank" rel="noopener">giscus.app</a> site suggests adding a <code>&lt;script&gt;</code> tag where the comments should appear.
That is not required when using Minimal Mistakes.
You don&#8217;t need to change any layouts from the theme to enable the comments.</p>
</div>
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>Here&#8217;s how your <code>_config.yml</code> file should look like after adding the config values provided by <a href="https://giscus.app" target="_blank" rel="noopener">giscus.app</a>:</p>
</div>
<div class="listingblock">
<div class="title">_config.yml from gwenneg.github.io</div>
<div class="content">
<pre class="rouge highlight"><code data-lang="yaml"><span class="na">comments</span><span class="pi">:</span>
  <span class="na">giscus</span><span class="pi">:</span>
    <span class="na">category_name</span><span class="pi">:</span> <span class="s">Blog posts</span> <i class="conum" data-value="1"></i><b>(1)</b>
    <span class="na">category_id</span><span class="pi">:</span> <span class="s">DIC_kwDOL32gyc4ChLma</span> <i class="conum" data-value="2"></i><b>(2)</b>
    <span class="na">discussion_term</span><span class="pi">:</span> <span class="s2">"</span><span class="s">og:title"</span> <i class="conum" data-value="3"></i><b>(3)</b>
    <span class="na">reactions_enabled</span><span class="pi">:</span> <span class="s2">"</span><span class="s">1"</span> <i class="conum" data-value="4"></i><b>(4)</b>
    <span class="na">repo_id</span><span class="pi">:</span> <span class="s">R_kgDOL32gyQ</span> <i class="conum" data-value="5"></i><b>(5)</b>
    <span class="na">theme</span><span class="pi">:</span> <span class="s">light</span> <i class="conum" data-value="6"></i><b>(6)</b>
  <span class="na">provider</span><span class="pi">:</span> <span class="s">giscus</span></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>You don&#8217;t <em>have</em> to use one of the default discussion categories provided by GitHub.
This one is a custom category.</td>
</tr>
<tr>
<td><i class="conum" data-value="2"></i><b>2</b></td>
<td>Replace this with the value provided by <a href="https://giscus.app" target="_blank" rel="noopener">giscus.app</a>.</td>
</tr>
<tr>
<td><i class="conum" data-value="3"></i><b>3</b></td>
<td>This determines the title of each discussion in GitHub Discussions.</td>
</tr>
<tr>
<td><i class="conum" data-value="4"></i><b>4</b></td>
<td>This allows users to react with emojis in addition to posting comments.</td>
</tr>
<tr>
<td><i class="conum" data-value="5"></i><b>5</b></td>
<td>Replace this with the value provided by <a href="https://giscus.app" target="_blank" rel="noopener">giscus.app</a>.</td>
</tr>
<tr>
<td><i class="conum" data-value="6"></i><b>6</b></td>
<td>A preview of each theme is available in <a href="https://giscus.app" target="_blank" rel="noopener">giscus.app</a>.</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>That&#8217;s all you need to do to enable comments in your posts.</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="improving-the-readability-of-your-posts">Improving the readability of your posts</h2>
<div class="sectionbody">
<div class="sect2">
<h3 id="making-important-information-stand-out-with-admonitions">Making important information stand out with admonitions</h3>
<div class="paragraph">
<p><a href="https://docs.asciidoctor.org/asciidoc/latest/blocks/admonitions" target="_blank" rel="noopener">Admonitions</a> from AsciiDoc are an excellent way to make important information stand out in your posts:</p>
</div>
<div class="admonitionblock tip">
<table>
<tr>
<td class="icon">
<i class="fa icon-tip" title="Tip"></i>
</td>
<td class="content">
<div class="paragraph">
<p>This is a <code>TIP</code> admonition.</p>
</div>
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>Here is the AsciiDoc syntax behind it:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="asciidoc">[TIP] <i class="conum" data-value="1"></i><b>(1)</b>
====
This is a `TIP` admonition.
====</code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>Additional admonition types are supported: <code>NOTE</code>, <code>WARNING</code>, <code>IMPORTANT</code> and <code>CAUTION</code>.</td>
</tr>
</table>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
<div class="paragraph">
<p>Did you notice the <code>Copy to clipboard</code> button in the top-right corner of all code blocks?
That feature is provided by the blog theme, <a href="https://mmistakes.github.io/minimal-mistakes/docs/configuration/#code-block-copy-button" target="_blank" rel="noopener">Minimal Mistakes</a>.</p>
</div>
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>All admonitions in your blog rely on icons from Font Awesome.
You can customize both the icon and its color for each admonition type in the <code>assets/css/custom.css</code> file:</p>
</div>
<div class="listingblock">
<div class="title">assets/css/custom.css</div>
<div class="content">
<pre class="rouge highlight"><code data-lang="css"><span class="nc">.admonitionblock</span> <span class="nc">.icon-tip</span><span class="nd">::before</span> <span class="p">{</span>
    <span class="nl">color</span><span class="p">:</span> <span class="m">#f6cc40</span><span class="p">;</span>
    <span class="nl">content</span><span class="p">:</span> <span class="s1">"\f0eb"</span><span class="p">;</span> <i class="conum" data-value="1"></i><b>(1)</b>
<span class="p">}</span></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>This determines which icon from Font Awesome is used.</td>
</tr>
</table>
</div>
</div>
<div class="sect2">
<h3 id="commenting-code-blocks-with-callouts">Commenting code blocks with callouts</h3>
<div class="paragraph">
<p><a href="https://docs.asciidoctor.org/asciidoc/latest/verbatim/callouts/" target="_blank" rel="noopener">Callouts</a> from AsciiDoc can be used to provide additional information about a specific line of code:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="java"><span class="kn">import</span> <span class="nn">java.util.Random</span><span class="o">;</span> <i class="conum" data-value="1"></i><b>(1)</b>

<span class="kd">public</span> <span class="kd">class</span> <span class="nc">NotSoRandom</span> <span class="o">{</span>
    <span class="kd">public</span> <span class="kt">void</span> <span class="nf">run</span><span class="o">()</span> <span class="o">{</span>
        <span class="nc">Random</span> <span class="n">random</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">Random</span><span class="o">(-</span><span class="mi">6732303926L</span><span class="o">);</span>
        <span class="k">for</span> <span class="o">(</span><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="mi">10</span><span class="o">;</span> <span class="n">i</span><span class="o">++)</span>
            <span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="n">random</span><span class="o">.</span><span class="na">nextInt</span><span class="o">(</span><span class="mi">10</span><span class="o">));</span> <i class="conum" data-value="2"></i><b>(2)</b>
        <span class="o">}</span>
    <span class="o">}</span>
<span class="o">}</span></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>Don&#8217;t do this at home!
You should always use <a href="https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/security/SecureRandom.html" target="_blank" rel="noopener">java.security.SecureRandom</a> when you need a truly random number in Java.</td>
</tr>
<tr>
<td><i class="conum" data-value="2"></i><b>2</b></td>
<td>Can you guess what will be printed?</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>Here is the AsciiDoc syntax behind that code block:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="asciidoc">[source, java]
----
import java.util.Random; // &lt;1&gt;

public class NotSoRandom {
    public void run() {
        Random random = new Random(-6732303926L);
        for (int i = 0; i &lt; 10; i++)
            System.out.println(random.nextInt(10)); // &lt;2&gt;
        }
    }
}
----
&lt;1&gt; Don't do this at home!
You should always use https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/security/SecureRandom.html[java.security.SecureRandom^] when you need a truly random number in Java.
&lt;2&gt; Can you guess what will be printed?</code></pre>
</div>
</div>
</div>
<div class="sect2">
<h3 id="changing-the-syntax-highlighting-theme-in-code-blocks">Changing the syntax highlighting theme in code blocks</h3>
<div class="paragraph">
<p>Code blocks from your blog are highlighted with <a href="https://docs.asciidoctor.org/asciidoctor/latest/syntax-highlighting/rouge" target="_blank" rel="noopener">Rouge</a>, a build-time syntax highlighter written in Ruby which supports over <a href="https://rouge-ruby.github.io/docs/file.Languages.html" target="_blank" rel="noopener">200 languages</a>.</p>
</div>
<div class="paragraph">
<p>The blog depends on <a href="https://spsarolkar.github.io/rouge-theme-preview" target="_blank" rel="noopener">one of the many themes</a> available in Rouge:</p>
</div>
<div class="listingblock">
<div class="title">_includes/head/custom.html</div>
<div class="content">
<pre class="rouge highlight"><code data-lang="html"><span class="nt">&lt;link</span> <span class="na">rel=</span><span class="s">"stylesheet"</span> <span class="na">href=</span><span class="s">"/assets/css/rouge/base16.monokai.dark.css"</span><span class="nt">&gt;</span> <i class="conum" data-value="1"></i><b>(1)</b>
<span class="nt">&lt;link</span> <span class="na">rel=</span><span class="s">"stylesheet"</span> <span class="na">href=</span><span class="s">"/assets/css/custom.css"</span><span class="nt">&gt;</span></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>This is how the <code>base16.monokai.dark</code> theme from Rouge is applied to the blog.</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>To change the theme, you will need to use a different CSS file from Rouge.
While the CSS files can be downloaded from third-party sources, there is a way to generate them directly from the command line:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="shell">gem <span class="nb">install </span>rouge <i class="conum" data-value="1"></i><b>(1)</b>
rougify <span class="nb">help </span>style <i class="conum" data-value="2"></i><b>(2)</b>
rougify style thankful_eyes <span class="o">&gt;</span> thankful_eyes.css <i class="conum" data-value="3"></i><b>(3)</b></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>This installs the <a href="https://rubygems.org/gems/rouge" target="_blank" rel="noopener">rouge</a> gem on your machine.</td>
</tr>
<tr>
<td><i class="conum" data-value="2"></i><b>2</b></td>
<td>This lists all themes available in Rouge.</td>
</tr>
<tr>
<td><i class="conum" data-value="3"></i><b>3</b></td>
<td>This generates the CSS file for the <code>thankful_eyes</code> theme and saves the output as <code>thankful_eyes.css</code>.</td>
</tr>
</table>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="updating-dependencies-automatically-with-dependabot">Updating dependencies automatically with Dependabot</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Keeping dependencies up to date is important for addressing security vulnerabilities, fixing bugs, and taking advantage of enhancements or new features.
Although this process can be time-consuming, your repository already benefits from some automation through GitHub’s <a href="https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/about-dependabot-version-updates" target="_blank" rel="noopener">Dependabot</a>.
When a new version of a blog dependency becomes available, Dependabot will automatically create a pull request similar to <a href="https://github.com/gwenneg/gwenneg.github.io/pull/22" target="_blank" rel="noopener">gwenneg.github.io#22</a>.
All you will need to do is test the PR and then merge it.</p>
</div>
<div class="paragraph">
<p>Here&#8217;s how Dependabot is configured in your repo:</p>
</div>
<div class="listingblock">
<div class="title">.github/dependabot.yml</div>
<div class="content">
<pre class="rouge highlight"><code data-lang="yaml"><span class="na">version</span><span class="pi">:</span> <span class="m">2</span>
<span class="na">updates</span><span class="pi">:</span>
  <span class="pi">-</span> <span class="na">package-ecosystem</span><span class="pi">:</span> <span class="s2">"</span><span class="s">bundler"</span> <i class="conum" data-value="1"></i><b>(1)</b>
    <span class="na">directory</span><span class="pi">:</span> <span class="s2">"</span><span class="s">/"</span>
    <span class="na">schedule</span><span class="pi">:</span>
      <span class="na">interval</span><span class="pi">:</span> <span class="s2">"</span><span class="s">weekly"</span> <i class="conum" data-value="2"></i><b>(2)</b>
  <span class="pi">-</span> <span class="na">package-ecosystem</span><span class="pi">:</span> <span class="s2">"</span><span class="s">github-actions"</span> <i class="conum" data-value="3"></i><b>(3)</b>
    <span class="na">directory</span><span class="pi">:</span> <span class="s2">"</span><span class="s">/"</span>
    <span class="na">schedule</span><span class="pi">:</span>
      <span class="na">interval</span><span class="pi">:</span> <span class="s2">"</span><span class="s">weekly"</span></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>This will automatically update the Ruby gems the blog depends on.</td>
</tr>
<tr>
<td><i class="conum" data-value="2"></i><b>2</b></td>
<td>Dependabot will check for newer versions on Monday each week.
<a href="https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file#scheduleinterval" target="_blank" rel="noopener">Additional intervals</a> are available.</td>
</tr>
<tr>
<td><i class="conum" data-value="3"></i><b>3</b></td>
<td>This will automatically update the GitHub Actions from the blog repo.</td>
</tr>
</table>
</div>
</div>
</div>
<div class="sect1">
<h2 id="conclusion">Conclusion</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Setting up a blog on GitHub Pages doesn’t have to be a daunting task.
With the right tools, templates, and a bit of guidance, you can have your blog up and running with minimal effort.
By leveraging the built-in features of Jekyll and the Minimal Mistakes theme, you can focus on what truly matters - creating content - without getting lost in the technical details.</p>
</div>
<div class="paragraph">
<p>Whether you’re new to blogging or just looking for a simpler way to manage your site, I hope this guide has shown you that it’s easier than you think to get started.</p>
</div>
<div class="paragraph">
<p>Now, all that’s left is to share your thoughts with the world. Happy blogging!</p>
</div>
</div>
</div>]]></content><author><name>Gwenneg Lepage</name></author><category term="asciidoc" /><category term="blogging" /><category term="dependabot" /><category term="giscus" /><category term="github actions" /><category term="github pages" /><category term="jekyll" /><category term="minimal mistakes" /><category term="syntax highlighting" /><summary type="html"><![CDATA[Thinking about starting a blog but worried it’s too time-consuming? Learn how to launch a fully functional blog on GitHub Pages and post with minimal effort.]]></summary></entry><entry><title type="html">Delaying exceptions while looping in Java</title><link href="https://gwenneg.github.io/2024/07/18/delaying-exceptions-while-looping.html" rel="alternate" type="text/html" title="Delaying exceptions while looping in Java" /><published>2024-07-18T00:00:00+00:00</published><updated>2024-07-18T00:00:00+00:00</updated><id>https://gwenneg.github.io/2024/07/18/delaying-exceptions-while-looping</id><content type="html" xml:base="https://gwenneg.github.io/2024/07/18/delaying-exceptions-while-looping.html"><![CDATA[<div id="preamble">
<div class="sectionbody">
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
<div class="paragraph">
<p>All code snippets from this post (and more!) are available in the <a href="https://github.com/gwenneg/blog-delayed-thrower" target="_blank" rel="noopener">gwenneg/blog-delayed-thrower</a> repository.</p>
</div>
</td>
</tr>
</table>
</div>
</div>
</div>
<div class="sect1">
<h2 id="introduction">Introduction</h2>
<div class="sectionbody">
<div class="paragraph">
<p>By default, Java will break a loop when an exception is thrown while iterating:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">BreakAndThrow</span> <span class="o">{</span>

    <span class="kd">public</span> <span class="kt">void</span> <span class="nf">run</span><span class="o">()</span> <span class="o">{</span>
        <span class="k">for</span> <span class="o">(</span><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="mi">10</span><span class="o">;</span> <span class="n">i</span><span class="o">++)</span> <span class="o">{</span> <i class="conum" data-value="1"></i><b>(1)</b>
            <span class="k">if</span> <span class="o">(</span><span class="n">i</span> <span class="o">%</span> <span class="mi">3</span> <span class="o">==</span> <span class="mi">2</span><span class="o">)</span> <span class="o">{</span>
                <span class="k">throw</span> <span class="k">new</span> <span class="nf">RuntimeException</span><span class="o">(</span><span class="s">"Something went wrong"</span><span class="o">);</span>
            <span class="o">}</span>
        <span class="o">}</span>
    <span class="o">}</span>
<span class="o">}</span></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>This loop will be broken on iteration #3 because a <code>RuntimeException</code> is thrown.</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>It often makes sense to handle exceptions inside the loop and keep looping after an exception is thrown:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">HandleAndContinue</span> <span class="o">{</span>

    <span class="kd">public</span> <span class="kt">void</span> <span class="nf">run</span><span class="o">()</span> <span class="o">{</span>
        <span class="k">for</span> <span class="o">(</span><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="mi">10</span><span class="o">;</span> <span class="n">i</span><span class="o">++)</span> <span class="o">{</span>
            <span class="k">try</span> <span class="o">{</span>
                <span class="k">if</span> <span class="o">(</span><span class="n">i</span> <span class="o">%</span> <span class="mi">3</span> <span class="o">==</span> <span class="mi">2</span><span class="o">)</span> <span class="o">{</span>
                    <span class="k">throw</span> <span class="k">new</span> <span class="nf">RuntimeException</span><span class="o">(</span><span class="s">"Something went wrong"</span><span class="o">);</span>
                <span class="o">}</span>
            <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">Exception</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span> <i class="conum" data-value="1"></i><b>(1)</b>
                <span class="c1">// TODO Handle the exception. </span><i class="conum" data-value="2"></i><b>(2)</b>
            <span class="o">}</span>
        <span class="o">}</span>
    <span class="o">}</span>
<span class="o">}</span></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>The <code>for</code> loop is no longer broken thanks to this <code>catch</code> block.</td>
</tr>
<tr>
<td><i class="conum" data-value="2"></i><b>2</b></td>
<td>Don&#8217;t <a href="https://en.wikipedia.org/wiki/Error_hiding" target="_blank" rel="noopener">swallow</a> the exception!
If you&#8217;re not sure yet how to deal with it, log it.</td>
</tr>
</table>
</div>
</div>
</div>
<div class="sect1">
<h2 id="delaying-exceptions">Delaying exceptions</h2>
<div class="sectionbody">
<div class="paragraph">
<p>What if you need to propagate or "bubble up" exceptions that are thrown inside a loop without breaking that loop?
Let&#8217;s see how exceptions thrown from a loop can be delayed and eventually thrown after the loop successfully completes.</p>
</div>
<div class="paragraph">
<p>First, we&#8217;ll need a custom exception class:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">DelayedException</span> <span class="kd">extends</span> <span class="nc">RuntimeException</span> <span class="o">{</span>

    <span class="kd">public</span> <span class="nf">DelayedException</span><span class="o">(</span><span class="nc">String</span> <span class="n">message</span><span class="o">)</span> <span class="o">{</span>
        <span class="kd">super</span><span class="o">(</span><span class="n">message</span><span class="o">);</span>
    <span class="o">}</span>
<span class="o">}</span></code></pre>
</div>
</div>
<div class="paragraph">
<p>Then, we&#8217;ll use that custom exception from a utility class:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="java"><span class="kn">import</span> <span class="nn">java.util.ArrayList</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">java.util.List</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">java.util.Objects</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">java.util.function.Consumer</span><span class="o">;</span>

<span class="kd">public</span> <span class="kd">class</span> <span class="nc">DelayedThrower</span> <span class="o">{</span>

    <span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">throwEventually</span><span class="o">(</span><span class="nc">String</span> <span class="n">exceptionMsg</span><span class="o">,</span> <span class="nc">Consumer</span><span class="o">&lt;</span><span class="nc">List</span><span class="o">&lt;</span><span class="nc">Exception</span><span class="o">&gt;&gt;</span> <span class="n">exceptionsConsumer</span><span class="o">)</span> <span class="o">{</span>

        <span class="nc">Objects</span><span class="o">.</span><span class="na">requireNonNull</span><span class="o">(</span><span class="n">exceptionsConsumer</span><span class="o">,</span> <span class="s">"The exceptions consumer must be not null"</span><span class="o">);</span>

        <span class="nc">List</span><span class="o">&lt;</span><span class="nc">Exception</span><span class="o">&gt;</span> <span class="n">exceptions</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">ArrayList</span><span class="o">&lt;&gt;();</span> <i class="conum" data-value="1"></i><b>(1)</b>
        <span class="n">exceptionsConsumer</span><span class="o">.</span><span class="na">accept</span><span class="o">(</span><span class="n">exceptions</span><span class="o">);</span>

        <span class="k">if</span> <span class="o">(</span><span class="n">exceptions</span><span class="o">.</span><span class="na">isEmpty</span><span class="o">())</span> <span class="o">{</span>
            <span class="k">return</span><span class="o">;</span> <i class="conum" data-value="2"></i><b>(2)</b>
        <span class="o">}</span>

        <span class="nc">DelayedException</span> <span class="n">delayedException</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">DelayedException</span><span class="o">(</span><span class="n">exceptionMsg</span><span class="o">);</span> <i class="conum" data-value="3"></i><b>(3)</b>

        <span class="k">for</span> <span class="o">(</span><span class="nc">Exception</span> <span class="n">e</span> <span class="o">:</span> <span class="n">exceptions</span><span class="o">)</span> <span class="o">{</span>
            <span class="k">if</span> <span class="o">(</span><span class="n">e</span> <span class="k">instanceof</span> <span class="nc">DelayedException</span> <span class="o">&amp;&amp;</span> <span class="n">e</span><span class="o">.</span><span class="na">getSuppressed</span><span class="o">().</span><span class="na">length</span> <span class="o">&gt;</span> <span class="mi">0</span><span class="o">)</span> <span class="o">{</span> <i class="conum" data-value="4"></i><b>(4)</b>
                <span class="k">for</span> <span class="o">(</span><span class="nc">Throwable</span> <span class="n">t</span> <span class="o">:</span> <span class="n">e</span><span class="o">.</span><span class="na">getSuppressed</span><span class="o">())</span> <span class="o">{</span>
                    <span class="n">delayedException</span><span class="o">.</span><span class="na">addSuppressed</span><span class="o">(</span><span class="n">t</span><span class="o">);</span> <i class="conum" data-value="5"></i><b>(5)</b>
                <span class="o">}</span>
            <span class="o">}</span> <span class="k">else</span> <span class="o">{</span>
                <span class="n">delayedException</span><span class="o">.</span><span class="na">addSuppressed</span><span class="o">(</span><span class="n">e</span><span class="o">);</span> <i class="conum" data-value="5"></i><b>(5)</b>
            <span class="o">}</span>
        <span class="o">}</span>

        <span class="k">throw</span> <span class="n">delayedException</span><span class="o">;</span>
    <span class="o">}</span>
<span class="o">}</span></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>Exceptions thrown from the loop will be stored into this collection until the loop completes.</td>
</tr>
<tr>
<td><i class="conum" data-value="2"></i><b>2</b></td>
<td>If no exceptions were thrown from the loop, the method can exit immediately.</td>
</tr>
<tr>
<td><i class="conum" data-value="3"></i><b>3</b></td>
<td>This is the exception that will eventually be thrown.</td>
</tr>
<tr>
<td><i class="conum" data-value="4"></i><b>4</b></td>
<td>This makes it possible to use nested <code>DelayedThrower#throwEventually</code> calls.</td>
</tr>
<tr>
<td><i class="conum" data-value="5"></i><b>5</b></td>
<td>Exceptions thrown from the loop are added as suppressed exceptions to the <code>DelayedException</code>.</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>Now, it&#8217;s time to delay exceptions!</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code data-lang="java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">ThrowEventually</span> <span class="o">{</span>

    <span class="kd">public</span> <span class="kt">void</span> <span class="nf">run</span><span class="o">()</span> <span class="o">{</span> <i class="conum" data-value="1"></i><b>(1)</b>
        <span class="nc">DelayedThrower</span><span class="o">.</span><span class="na">throwEventually</span><span class="o">(</span><span class="s">"Exceptions were thrown while looping"</span><span class="o">,</span> <span class="n">exceptions</span> <span class="o">-&gt;</span> <span class="o">{</span>
            <span class="k">for</span> <span class="o">(</span><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="mi">10</span><span class="o">;</span> <span class="n">i</span><span class="o">++)</span> <span class="o">{</span>
                <span class="k">try</span> <span class="o">{</span>
                    <span class="k">if</span> <span class="o">(</span><span class="n">i</span> <span class="o">%</span> <span class="mi">3</span> <span class="o">==</span> <span class="mi">2</span><span class="o">)</span> <span class="o">{</span>
                        <span class="k">throw</span> <span class="k">new</span> <span class="nf">RuntimeException</span><span class="o">(</span><span class="nc">String</span><span class="o">.</span><span class="na">format</span><span class="o">(</span><span class="s">"Something went wrong [i=%d]"</span><span class="o">,</span> <span class="n">i</span><span class="o">));</span>
                    <span class="o">}</span>
                <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">Exception</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span>
                    <span class="n">exceptions</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="n">e</span><span class="o">);</span>
                <span class="o">}</span>
            <span class="o">}</span>
        <span class="o">});</span>
    <span class="o">}</span>
<span class="o">}</span></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>When this method completes, a <code>DelayedException</code> with 3 suppressed exceptions will be thrown.</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>If the <code>DelayedException</code> is logged, it should look like this:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight"><code>com.gwenneg.blog.delayed.DelayedException: Exceptions were thrown while looping
	at com.gwenneg.blog.delayed.DelayedThrower.throwEventually(DelayedThrower.java:21)
	at com.gwenneg.blog.delayed.ThrowEventually.run(ThrowEventually.java:6)
	[...]
	Suppressed: java.lang.RuntimeException: Something went wrong [i=2]
		at com.gwenneg.blog.delayed.ThrowEventually.lambda$run$0(ThrowEventually.java:10)
		at com.gwenneg.blog.delayed.DelayedThrower.throwEventually(DelayedThrower.java:15)
		... 5 more
	Suppressed: java.lang.RuntimeException: Something went wrong [i=5]
		at com.gwenneg.blog.delayed.ThrowEventually.lambda$run$0(ThrowEventually.java:10)
		at com.gwenneg.blog.delayed.DelayedThrower.throwEventually(DelayedThrower.java:15)
		... 5 more
	Suppressed: java.lang.RuntimeException: Something went wrong [i=8]
		at com.gwenneg.blog.delayed.ThrowEventually.lambda$run$0(ThrowEventually.java:10)
		at com.gwenneg.blog.delayed.DelayedThrower.throwEventually(DelayedThrower.java:15)
		... 5 more</code></pre>
</div>
</div>
<div class="paragraph">
<p>That&#8217;s all I have for today.
Happy looping!</p>
</div>
</div>
</div>]]></content><author><name>Gwenneg Lepage</name></author><category term="java" /><category term="development tip" /><summary type="html"><![CDATA[Learn how to propagate exceptions thrown from a loop without breaking that loop.]]></summary></entry><entry><title type="html">Overriding the configuration of a Quarkus app from its test code</title><link href="https://gwenneg.github.io/2024/07/02/overriding-configuration-from-test-code.html" rel="alternate" type="text/html" title="Overriding the configuration of a Quarkus app from its test code" /><published>2024-07-02T00:00:00+00:00</published><updated>2024-07-02T00:00:00+00:00</updated><id>https://gwenneg.github.io/2024/07/02/overriding-configuration-from-test-code</id><content type="html" xml:base="https://gwenneg.github.io/2024/07/02/overriding-configuration-from-test-code.html"><![CDATA[<div id="preamble">
<div class="sectionbody">
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
<div class="paragraph">
<p>This post was initially published in the <a href="https://quarkus.io/blog/overriding-configuration-from-test-code/" target="_blank" rel="noopener">Quarkus blog</a>.</p>
</div>
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>Overriding the configuration of a Quarkus app from its test code is often required to achieve a good test coverage.
Whenever a config property determines how the app behaves, all possible config values need to be tested.</p>
</div>
<div class="listingblock">
<div class="title">All branches need to be tested</div>
<div class="content">
<pre class="rouge highlight"><code data-lang="java"><span class="kn">import</span> <span class="nn">io.smallrye.config.ConfigMapping</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">io.smallrye.config.WithDefault</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">jakarta.enterprise.context.ApplicationScoped</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">jakarta.inject.Inject</span><span class="o">;</span>

<span class="nd">@ApplicationScoped</span>
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">MyService</span> <span class="o">{</span>

    <span class="nd">@Inject</span>
    <span class="nc">MyConfig</span> <span class="n">config</span><span class="o">;</span>

    <span class="kd">public</span> <span class="kt">void</span> <span class="nf">doSomething</span><span class="o">()</span> <span class="o">{</span>
        <span class="k">if</span> <span class="o">(</span><span class="n">config</span><span class="o">.</span><span class="na">newFeatureEnabled</span><span class="o">())</span> <span class="o">{</span>
            <span class="c1">// This branch needs to be tested.</span>
        <span class="o">}</span> <span class="k">else</span> <span class="o">{</span>
            <span class="c1">// So does that branch.</span>
        <span class="o">}</span>
    <span class="o">}</span>
<span class="o">}</span>

<span class="nd">@ConfigMapping</span><span class="o">(</span><span class="n">prefix</span> <span class="o">=</span> <span class="s">"my-config"</span><span class="o">)</span>
<span class="kd">interface</span> <span class="nc">MyConfig</span> <span class="o">{</span> <i class="conum" data-value="1"></i><b>(1)</b>

    <span class="nd">@WithDefault</span><span class="o">(</span><span class="s">"false"</span><span class="o">)</span>
    <span class="kt">boolean</span> <span class="nf">newFeatureEnabled</span><span class="o">();</span>
<span class="o">}</span></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>In a real project, this interface would likely be <code>public</code> and declared in a separate file.</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>There are many ways to override the configuration from the test code.
This post will show you five approaches, with a particular focus on the benefits and drawbacks of each of them.</p>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
<div class="paragraph">
<p>All code snippets from this post (and more!) are available in the <a href="https://github.com/gwenneg/blog-overriding-configuration-from-test-code" target="_blank" rel="noopener">gwenneg/blog-overriding-configuration-from-test-code</a> repository.</p>
</div>
</td>
</tr>
</table>
</div>
</div>
</div>
<div class="sect1">
<h2 id="approach-1-quarkus-test-profiles"><a id="quarkus-test-profiles"></a> Approach #1: Quarkus test profiles</h2>
<div class="sectionbody">
<div class="paragraph">
<p><a href="https://quarkus.io/guides/getting-started-testing#testing_different_profiles" target="_blank" rel="noopener">Quarkus test profiles</a> are one of the best ways to override the configuration.
They can be used while testing in native mode, unlike most approaches listed in this post.
In addition to the config override, they provide <a href="https://quarkus.io/guides/getting-started-testing#writing-a-profile" target="_blank" rel="noopener">many additional capabilities</a> which make it easier to test Quarkus apps.</p>
</div>
<div class="paragraph">
<p>From a configuration override perspective, test profiles suffer however from a few drawbacks.
First, Quarkus is restarted before each test profile is used, which obviously slows down the tests execution.
The tests also have to be split into several test profiles and classes to cover multiple values of the same config properties.
As a result, bigger projects may end up with lots of test profiles and spend a lot of time restarting Quarkus between tests.
Maintaining or reviewing the test code may also be more challenging with test profiles.</p>
</div>
<div class="listingblock">
<div class="title">The code to be tested</div>
<div class="content">
<pre class="rouge highlight"><code data-lang="java"><span class="kn">import</span> <span class="nn">io.smallrye.config.ConfigMapping</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">io.smallrye.config.WithDefault</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">jakarta.inject.Inject</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">jakarta.ws.rs.GET</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">jakarta.ws.rs.Path</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.eclipse.microprofile.config.inject.ConfigProperty</span><span class="o">;</span>

<span class="nd">@Path</span><span class="o">(</span><span class="s">"/features"</span><span class="o">)</span>
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">FeaturesResource</span> <span class="o">{</span>

    <span class="nd">@Inject</span>
    <span class="nc">FeaturesConfig</span> <span class="n">featuresConfig</span><span class="o">;</span> <i class="conum" data-value="1"></i><b>(1)</b>

    <span class="nd">@ConfigProperty</span><span class="o">(</span><span class="n">name</span> <span class="o">=</span> <span class="s">"amazing-feature-enabled"</span><span class="o">,</span> <span class="n">defaultValue</span> <span class="o">=</span> <span class="s">"false"</span><span class="o">)</span> <i class="conum" data-value="1"></i><b>(1)</b>
    <span class="kt">boolean</span> <span class="n">amazingFeatureEnabled</span><span class="o">;</span>

    <span class="nd">@GET</span>
    <span class="nd">@Path</span><span class="o">(</span><span class="s">"/awesome"</span><span class="o">)</span>
    <span class="kd">public</span> <span class="kt">boolean</span> <span class="nf">isAwesomeFeatureEnabled</span><span class="o">()</span> <span class="o">{</span>
        <span class="k">return</span> <span class="n">featuresConfig</span><span class="o">.</span><span class="na">awesomeFeatureEnabled</span><span class="o">();</span>
    <span class="o">}</span>

    <span class="nd">@GET</span>
    <span class="nd">@Path</span><span class="o">(</span><span class="s">"/amazing"</span><span class="o">)</span>
    <span class="kd">public</span> <span class="kt">boolean</span> <span class="nf">isAmazingFeatureEnabled</span><span class="o">()</span> <span class="o">{</span>
        <span class="k">return</span> <span class="n">amazingFeatureEnabled</span><span class="o">;</span>
    <span class="o">}</span>
<span class="o">}</span>

<span class="nd">@ConfigMapping</span><span class="o">(</span><span class="n">prefix</span> <span class="o">=</span> <span class="s">"features"</span><span class="o">)</span>
<span class="kd">interface</span> <span class="nc">FeaturesConfig</span> <span class="o">{</span> <i class="conum" data-value="2"></i><b>(2)</b>

    <span class="nd">@WithDefault</span><span class="o">(</span><span class="s">"false"</span><span class="o">)</span>
    <span class="kt">boolean</span> <span class="nf">awesomeFeatureEnabled</span><span class="o">();</span>
<span class="o">}</span></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>Test profiles work with both <a href="https://quarkus.io/guides/config-mappings" target="_blank" rel="noopener">config mappings</a> and <code>@ConfigProperty</code>.</td>
</tr>
<tr>
<td><i class="conum" data-value="2"></i><b>2</b></td>
<td>In a real project, this interface would likely be <code>public</code> and declared in a separate file.</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>Most guides about test profiles will introduce them in a verbose way to demonstrate all their capabilities.
A test profile can actually be added to an existing test class with only a few extra lines:</p>
</div>
<div class="listingblock">
<div class="title">The test class which is also a test profile</div>
<div class="content">
<pre class="rouge highlight"><code data-lang="java"><span class="kn">import</span> <span class="nn">io.quarkus.test.junit.QuarkusTest</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">io.quarkus.test.junit.QuarkusTestProfile</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">io.quarkus.test.junit.TestProfile</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">io.restassured.RestAssured</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">java.util.Map</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.hamcrest.CoreMatchers</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.junit.jupiter.api.Test</span><span class="o">;</span>

<span class="nd">@QuarkusTest</span>
<span class="nd">@TestProfile</span><span class="o">(</span><span class="nc">FeaturesResourceTest</span><span class="o">.</span><span class="na">class</span><span class="o">)</span>
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">FeaturesResourceTest</span> <span class="kd">implements</span> <span class="nc">QuarkusTestProfile</span> <span class="o">{</span> <i class="conum" data-value="1"></i><b>(1)</b>

    <span class="nd">@Override</span>
    <span class="kd">public</span> <span class="nc">Map</span><span class="o">&lt;</span><span class="nc">String</span><span class="o">,</span> <span class="nc">String</span><span class="o">&gt;</span> <span class="nf">getConfigOverrides</span><span class="o">()</span> <span class="o">{</span> <i class="conum" data-value="2"></i><b>(2)</b>
        <span class="k">return</span> <span class="nc">Map</span><span class="o">.</span><span class="na">of</span><span class="o">(</span>
            <span class="s">"features.awesome-feature-enabled"</span><span class="o">,</span> <span class="s">"true"</span><span class="o">,</span> <i class="conum" data-value="3"></i><b>(3)</b>
            <span class="s">"amazing-feature-enabled"</span><span class="o">,</span> <span class="s">"true"</span>
        <span class="o">);</span>
    <span class="o">}</span>

    <span class="nd">@Test</span>
    <span class="kt">void</span> <span class="nf">test</span><span class="o">()</span> <span class="o">{</span>

        <span class="nc">RestAssured</span><span class="o">.</span><span class="na">given</span><span class="o">()</span>
            <span class="o">.</span><span class="na">when</span><span class="o">().</span><span class="na">get</span><span class="o">(</span><span class="s">"/features/awesome"</span><span class="o">)</span>
            <span class="o">.</span><span class="na">then</span><span class="o">().</span><span class="na">body</span><span class="o">(</span><span class="nc">CoreMatchers</span><span class="o">.</span><span class="na">is</span><span class="o">(</span><span class="s">"true"</span><span class="o">));</span>

        <span class="nc">RestAssured</span><span class="o">.</span><span class="na">given</span><span class="o">()</span>
            <span class="o">.</span><span class="na">when</span><span class="o">().</span><span class="na">get</span><span class="o">(</span><span class="s">"/features/amazing"</span><span class="o">)</span>
            <span class="o">.</span><span class="na">then</span><span class="o">().</span><span class="na">body</span><span class="o">(</span><span class="nc">CoreMatchers</span><span class="o">.</span><span class="na">is</span><span class="o">(</span><span class="s">"true"</span><span class="o">));</span>
    <span class="o">}</span>
<span class="o">}</span></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>The test class itself can implement <code>QuarkusTestProfile</code> if the profile isn&#8217;t shared across multiple test classes.
This can make the maintenance and reviews of the test code easier.
If multiple test classes depend on the same profile, then that profile will likely need to be declared in a dedicated class.</td>
</tr>
<tr>
<td><i class="conum" data-value="2"></i><b>2</b></td>
<td>This method comes from <code>QuarkusTestProfile</code> and makes it possible to override the configuration from the test code.</td>
</tr>
<tr>
<td><i class="conum" data-value="3"></i><b>3</b></td>
<td>The config key generated from the <code>FeaturesConfig</code> interface is prefixed with <code>features.</code> while the config key that comes from the <code>@ConfigProperty</code> injection has no prefix.</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>Test profiles can also leverage <a href="https://quarkus.io/guides/config-reference#profile-aware-files" target="_blank" rel="noopener">profile aware files</a> to override the configuration from the test code:</p>
</div>
<div class="listingblock">
<div class="title">application-blog.properties</div>
<div class="content">
<pre class="rouge highlight"><code data-lang="properties"><span class="py">features.awesome-feature-enabled</span><span class="p">=</span><span class="s">true</span></code></pre>
</div>
</div>
<div class="paragraph">
<p>When that is used, the test profile needs to override the default config profile:</p>
</div>
<div class="listingblock">
<div class="title">The test code</div>
<div class="content">
<pre class="rouge highlight"><code data-lang="java"><span class="kn">import</span> <span class="nn">io.quarkus.test.junit.QuarkusTest</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">io.quarkus.test.junit.QuarkusTestProfile</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">io.quarkus.test.junit.TestProfile</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">io.restassured.RestAssured</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.hamcrest.CoreMatchers</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.junit.jupiter.api.Test</span><span class="o">;</span>

<span class="nd">@QuarkusTest</span>
<span class="nd">@TestProfile</span><span class="o">(</span><span class="nc">FeaturesResourceTest</span><span class="o">.</span><span class="na">class</span><span class="o">)</span>
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">FeaturesResourceTest</span> <span class="kd">implements</span> <span class="nc">QuarkusTestProfile</span> <span class="o">{</span>

    <span class="nd">@Override</span>
    <span class="kd">public</span> <span class="nc">String</span> <span class="nf">getConfigProfile</span><span class="o">()</span> <span class="o">{</span> <i class="conum" data-value="1"></i><b>(1)</b>
        <span class="k">return</span> <span class="s">"blog"</span><span class="o">;</span> <i class="conum" data-value="2"></i><b>(2)</b>
    <span class="o">}</span>

    <span class="nd">@Test</span>
    <span class="kt">void</span> <span class="nf">test</span><span class="o">()</span> <span class="o">{</span>
        <span class="nc">RestAssured</span><span class="o">.</span><span class="na">given</span><span class="o">()</span>
            <span class="o">.</span><span class="na">when</span><span class="o">().</span><span class="na">get</span><span class="o">(</span><span class="s">"/features/awesome"</span><span class="o">)</span>
            <span class="o">.</span><span class="na">then</span><span class="o">().</span><span class="na">body</span><span class="o">(</span><span class="nc">CoreMatchers</span><span class="o">.</span><span class="na">is</span><span class="o">(</span><span class="s">"true"</span><span class="o">));</span>
    <span class="o">}</span>
<span class="o">}</span></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>This method comes from <code>QuarkusTestProfile</code> and makes it possible to override the default config profile.</td>
</tr>
<tr>
<td><i class="conum" data-value="2"></i><b>2</b></td>
<td>The <code>application-blog.properties</code> file will be loaded because the <code>blog</code> config profile is active.</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>If the tests are run in JVM mode only and not in native mode, the <code>application-blog.properties</code> file can be placed in the <code>src/test/resources</code> folder.
An additional <code>application.properties</code> file (possibly empty) is also required in the same location to enable profile aware files.</p>
</div>
<div class="paragraph">
<p>If the tests are run in native mode, the same <code>application-blog.properties</code> and <code>application.properties</code> files are needed as well, but they have to be placed in the <code>src/main/resources</code> folder.
The <code>application.properties</code> file also needs to contain the following line:</p>
</div>
<div class="listingblock">
<div class="title">application.properties</div>
<div class="content">
<pre class="rouge highlight"><code data-lang="properties"><span class="py">quarkus.native.resources.includes</span><span class="p">=</span><span class="s">application*.properties</span></code></pre>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="approach-2-mocking-the-config-with-mockito">Approach #2: mocking the config with Mockito</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Now, here&#8217;s my favorite approach when native testing is not required.</p>
</div>
<div class="paragraph">
<p>First, let&#8217;s see how that works with a <a href="https://quarkus.io/guides/config-mappings" target="_blank" rel="noopener">config mapping</a>:</p>
</div>
<div class="listingblock">
<div class="title">The code to be tested</div>
<div class="content">
<pre class="rouge highlight"><code data-lang="java"><span class="kn">import</span> <span class="nn">io.smallrye.config.ConfigMapping</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">io.smallrye.config.WithDefault</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">jakarta.inject.Inject</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">jakarta.ws.rs.GET</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">jakarta.ws.rs.Path</span><span class="o">;</span>

<span class="nd">@Path</span><span class="o">(</span><span class="s">"/features"</span><span class="o">)</span>
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">FeaturesResource</span> <span class="o">{</span>

    <span class="nd">@Inject</span>
    <span class="nc">FeaturesConfig</span> <span class="n">featuresConfig</span><span class="o">;</span>

    <span class="nd">@GET</span>
    <span class="nd">@Path</span><span class="o">(</span><span class="s">"/awesome"</span><span class="o">)</span>
    <span class="kd">public</span> <span class="kt">boolean</span> <span class="nf">isAwesomeFeatureEnabled</span><span class="o">()</span> <span class="o">{</span>
        <span class="k">return</span> <span class="n">featuresConfig</span><span class="o">.</span><span class="na">awesomeFeatureEnabled</span><span class="o">();</span>
    <span class="o">}</span>
<span class="o">}</span>

<span class="nd">@ConfigMapping</span><span class="o">(</span><span class="n">prefix</span> <span class="o">=</span> <span class="s">"features"</span><span class="o">)</span>
<span class="kd">interface</span> <span class="nc">FeaturesConfig</span> <span class="o">{</span> <i class="conum" data-value="1"></i><b>(1)</b>

    <span class="nd">@WithDefault</span><span class="o">(</span><span class="s">"false"</span><span class="o">)</span>
    <span class="kt">boolean</span> <span class="nf">awesomeFeatureEnabled</span><span class="o">();</span>
<span class="o">}</span></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>In a real project, this interface would likely be <code>public</code> and declared in a separate file.</td>
</tr>
</table>
</div>
<div class="listingblock">
<div class="title">The test code</div>
<div class="content">
<pre class="rouge highlight"><code data-lang="java"><span class="kn">import</span> <span class="nn">io.quarkus.test.InjectMock</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">io.quarkus.test.Mock</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">io.quarkus.test.junit.QuarkusTest</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">io.restassured.RestAssured</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">io.smallrye.config.SmallRyeConfig</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">jakarta.enterprise.context.ApplicationScoped</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">jakarta.enterprise.inject.Produces</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">jakarta.inject.Inject</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.hamcrest.CoreMatchers</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.junit.jupiter.api.Test</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.mockito.Mockito</span><span class="o">;</span>

<span class="nd">@QuarkusTest</span>
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">FeaturesResourceTest</span> <span class="o">{</span>

    <span class="nd">@Inject</span>
    <span class="nc">SmallRyeConfig</span> <span class="n">smallRyeConfig</span><span class="o">;</span>

    <span class="nd">@Produces</span> <i class="conum" data-value="1"></i><b>(1)</b>
    <span class="nd">@ApplicationScoped</span>
    <span class="nd">@Mock</span>
    <span class="nc">FeaturesConfig</span> <span class="nf">featuresConfig</span><span class="o">()</span> <span class="o">{</span> <i class="conum" data-value="2"></i><b>(2)</b>
        <span class="k">return</span> <span class="n">smallRyeConfig</span><span class="o">.</span><span class="na">getConfigMapping</span><span class="o">(</span><span class="nc">FeaturesConfig</span><span class="o">.</span><span class="na">class</span><span class="o">);</span>
    <span class="o">}</span>

    <span class="nd">@InjectMock</span> <i class="conum" data-value="3"></i><b>(3)</b>
    <span class="nc">FeaturesConfig</span> <span class="n">featuresConfig</span><span class="o">;</span>

    <span class="nd">@Test</span>
    <span class="kt">void</span> <span class="nf">test</span><span class="o">()</span> <span class="o">{</span>
        <span class="nc">Mockito</span><span class="o">.</span><span class="na">when</span><span class="o">(</span><span class="n">featuresConfig</span><span class="o">.</span><span class="na">awesomeFeatureEnabled</span><span class="o">()).</span><span class="na">thenReturn</span><span class="o">(</span><span class="kc">true</span><span class="o">);</span> <i class="conum" data-value="4"></i><b>(4)</b>
        <span class="nc">RestAssured</span><span class="o">.</span><span class="na">given</span><span class="o">()</span>
            <span class="o">.</span><span class="na">when</span><span class="o">().</span><span class="na">get</span><span class="o">(</span><span class="s">"/features/awesome"</span><span class="o">)</span>
            <span class="o">.</span><span class="na">then</span><span class="o">().</span><span class="na">body</span><span class="o">(</span><span class="nc">CoreMatchers</span><span class="o">.</span><span class="na">is</span><span class="o">(</span><span class="s">"true"</span><span class="o">));</span>
    <span class="o">}</span>
<span class="o">}</span></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>This annotation can be omitted.</td>
</tr>
<tr>
<td><i class="conum" data-value="2"></i><b>2</b></td>
<td>This is required to make the <code>FeaturesConfig</code> interface implementation proxyable.
Without that, it wouldn&#8217;t be possible to mock it with <code>@InjectMock</code>.</td>
</tr>
<tr>
<td><i class="conum" data-value="3"></i><b>3</b></td>
<td>The config class is mocked with the help of the <code>quarkus-junit5-mockito</code> extension.
Injections are not supported in tests in native mode, so this only works when the test is run in JVM mode.</td>
</tr>
<tr>
<td><i class="conum" data-value="4"></i><b>4</b></td>
<td>The configuration can be mocked from the test method or from a method annotated with one of JUnit&#8217;s <a href="https://junit.org/junit5/docs/current/user-guide/#writing-tests-definitions" target="_blank" rel="noopener">lifecycle annotations</a> such as <code>@BeforeEach</code>.</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>What if your project relies on <code>@ConfigProperty</code> instead of <code>@ConfigMapping</code>?
Well, that works too!
You&#8217;ll just need to move the config properties to an extra <code>@ApplicationScoped</code> bean.
That bean may or may not be used to centralize all config properties from the Quarkus app.</p>
</div>
<div class="listingblock">
<div class="title">A centralized config class, with logging at application startup</div>
<div class="content">
<pre class="rouge highlight"><code data-lang="java"><span class="kn">import</span> <span class="nn">io.quarkus.logging.Log</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">jakarta.enterprise.context.ApplicationScoped</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">jakarta.enterprise.event.Observes</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">jakarta.enterprise.event.Startup</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">java.util.Map</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">java.util.TreeMap</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.eclipse.microprofile.config.inject.ConfigProperty</span><span class="o">;</span>

<span class="nd">@ApplicationScoped</span>
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">FeaturesConfig</span> <span class="o">{</span>

    <span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="nc">String</span> <span class="no">AWESOME_FEATURE_ENABLED</span> <span class="o">=</span> <span class="s">"awesome-feature-enabled"</span><span class="o">;</span>

    <span class="nd">@ConfigProperty</span><span class="o">(</span><span class="n">name</span> <span class="o">=</span> <span class="no">AWESOME_FEATURE_ENABLED</span><span class="o">,</span> <span class="n">defaultValue</span> <span class="o">=</span> <span class="s">"false"</span><span class="o">)</span>
    <span class="kt">boolean</span> <span class="n">awesomeFeatureEnabled</span><span class="o">;</span>

    <span class="c1">// Omitted: additional config properties.</span>

    <span class="kd">public</span> <span class="kt">boolean</span> <span class="nf">isAwesomeFeatureEnabled</span><span class="o">()</span> <span class="o">{</span>
        <span class="k">return</span> <span class="n">awesomeFeatureEnabled</span><span class="o">;</span>
    <span class="o">}</span>

    <span class="c1">// This is an optional bonus unrelated to the blog post topic.</span>
    <span class="kt">void</span> <span class="nf">logConfigAtStartup</span><span class="o">(</span><span class="nd">@Observes</span> <span class="nc">Startup</span> <span class="n">event</span><span class="o">)</span> <span class="o">{</span> <i class="conum" data-value="1"></i><b>(1)</b>

        <span class="nc">Map</span><span class="o">&lt;</span><span class="nc">String</span><span class="o">,</span> <span class="nc">Object</span><span class="o">&gt;</span> <span class="n">config</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">TreeMap</span><span class="o">&lt;&gt;();</span> <i class="conum" data-value="2"></i><b>(2)</b>
        <span class="n">config</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="no">AWESOME_FEATURE_ENABLED</span><span class="o">,</span> <span class="n">awesomeFeatureEnabled</span><span class="o">);</span>
        <span class="c1">// Omitted: put all config keys and values into the map.</span>

        <span class="nc">Log</span><span class="o">.</span><span class="na">info</span><span class="o">(</span><span class="s">"=== Startup configuration ==="</span><span class="o">);</span>
        <span class="n">config</span><span class="o">.</span><span class="na">forEach</span><span class="o">((</span><span class="n">key</span><span class="o">,</span> <span class="n">value</span><span class="o">)</span> <span class="o">-&gt;</span> <span class="o">{</span>
            <span class="nc">Log</span><span class="o">.</span><span class="na">infof</span><span class="o">(</span><span class="s">"%s=%s"</span><span class="o">,</span> <span class="n">key</span><span class="o">,</span> <span class="n">value</span><span class="o">);</span> <i class="conum" data-value="3"></i><b>(3)</b>
        <span class="o">});</span>
    <span class="o">}</span>
<span class="o">}</span></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>This method is executed at application startup. See the <a href="https://quarkus.io/guides/lifecycle#listening-for-startup-and-shutdown-events" target="_blank" rel="noopener">Application initialization and termination</a> guide for more details about the application lifecycle events.</td>
</tr>
<tr>
<td><i class="conum" data-value="2"></i><b>2</b></td>
<td><code>TreeMap</code> helps automatically sort the map entries by keys alphabetically.</td>
</tr>
<tr>
<td><i class="conum" data-value="3"></i><b>3</b></td>
<td>The application config is logged at startup.
This can really help if you ever need to investigate an issue based on past logs.
Be careful not to log any sensitive config values though! (e.g. secrets or passwords)</td>
</tr>
</table>
</div>
<div class="listingblock">
<div class="title">The code to be tested</div>
<div class="content">
<pre class="rouge highlight"><code data-lang="java"><span class="kn">import</span> <span class="nn">jakarta.inject.Inject</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">jakarta.ws.rs.GET</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">jakarta.ws.rs.Path</span><span class="o">;</span>

<span class="nd">@Path</span><span class="o">(</span><span class="s">"/features"</span><span class="o">)</span>
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">FeaturesResource</span> <span class="o">{</span>

    <span class="nd">@Inject</span>
    <span class="nc">FeaturesConfig</span> <span class="n">featuresConfig</span><span class="o">;</span>

    <span class="nd">@GET</span>
    <span class="nd">@Path</span><span class="o">(</span><span class="s">"/awesome"</span><span class="o">)</span>
    <span class="kd">public</span> <span class="kt">boolean</span> <span class="nf">isAwesomeFeatureEnabled</span><span class="o">()</span> <span class="o">{</span>
        <span class="k">return</span> <span class="n">featuresConfig</span><span class="o">.</span><span class="na">isAwesomeFeatureEnabled</span><span class="o">();</span>
    <span class="o">}</span>
<span class="o">}</span></code></pre>
</div>
</div>
<div class="listingblock">
<div class="title">The test code</div>
<div class="content">
<pre class="rouge highlight"><code data-lang="java"><span class="kn">import</span> <span class="nn">io.quarkus.test.InjectMock</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">io.quarkus.test.junit.QuarkusTest</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">io.restassured.RestAssured</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.hamcrest.CoreMatchers</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.junit.jupiter.api.Test</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.mockito.Mockito</span><span class="o">;</span>

<span class="nd">@QuarkusTest</span>
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">FeaturesResourceTest</span> <span class="o">{</span>

    <span class="nd">@InjectMock</span> <i class="conum" data-value="1"></i><b>(1)</b>
    <span class="nc">FeaturesConfig</span> <span class="n">featuresConfig</span><span class="o">;</span>

    <span class="nd">@Test</span>
    <span class="kt">void</span> <span class="nf">test</span><span class="o">()</span> <span class="o">{</span>
        <span class="nc">Mockito</span><span class="o">.</span><span class="na">when</span><span class="o">(</span><span class="n">featuresConfig</span><span class="o">.</span><span class="na">isAwesomeFeatureEnabled</span><span class="o">()).</span><span class="na">thenReturn</span><span class="o">(</span><span class="kc">true</span><span class="o">);</span> <i class="conum" data-value="2"></i><b>(2)</b>
        <span class="nc">RestAssured</span><span class="o">.</span><span class="na">given</span><span class="o">()</span>
            <span class="o">.</span><span class="na">when</span><span class="o">().</span><span class="na">get</span><span class="o">(</span><span class="s">"/features/awesome"</span><span class="o">)</span>
            <span class="o">.</span><span class="na">then</span><span class="o">().</span><span class="na">body</span><span class="o">(</span><span class="nc">CoreMatchers</span><span class="o">.</span><span class="na">is</span><span class="o">(</span><span class="s">"true"</span><span class="o">));</span>
    <span class="o">}</span>
<span class="o">}</span></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>The config class is mocked with the help of the <code>quarkus-junit5-mockito</code> extension.
Injections are not supported in tests in native mode, so this only works when the test is run in JVM mode.</td>
</tr>
<tr>
<td><i class="conum" data-value="2"></i><b>2</b></td>
<td>The configuration can be mocked from the test method or from a method annotated with one of JUnit&#8217;s <a href="https://junit.org/junit5/docs/current/user-guide/#writing-tests-definitions" target="_blank" rel="noopener">lifecycle annotations</a> such as <code>@BeforeEach</code>.</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>This approach can also leverage the <code>@ParameterizedTest</code> feature from JUnit and test several values of a config property with a single test method:</p>
</div>
<div class="listingblock">
<div class="title">The test code based on @ParameterizedTest</div>
<div class="content">
<pre class="rouge highlight"><code data-lang="java"><span class="kn">import</span> <span class="nn">io.quarkus.test.InjectMock</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">io.quarkus.test.junit.QuarkusTest</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">io.restassured.RestAssured</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.hamcrest.CoreMatchers</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.junit.jupiter.params.ParameterizedTest</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.junit.jupiter.params.provider.ValueSource</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.mockito.Mockito</span><span class="o">;</span>

<span class="nd">@QuarkusTest</span>
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">FeaturesResourceTest</span> <span class="o">{</span>

    <span class="nd">@InjectMock</span>
    <span class="nc">FeaturesConfig</span> <span class="n">featuresConfig</span><span class="o">;</span>

    <span class="nd">@ParameterizedTest</span>
    <span class="nd">@ValueSource</span><span class="o">(</span><span class="n">booleans</span> <span class="o">=</span> <span class="o">{</span><span class="kc">true</span><span class="o">,</span> <span class="kc">false</span><span class="o">})</span>
    <span class="kt">void</span> <span class="nf">test</span><span class="o">(</span><span class="kt">boolean</span> <span class="n">awesomeFeatureEnabled</span><span class="o">)</span> <span class="o">{</span> <i class="conum" data-value="1"></i><b>(1)</b>
        <span class="nc">Mockito</span><span class="o">.</span><span class="na">when</span><span class="o">(</span><span class="n">featuresConfig</span><span class="o">.</span><span class="na">isAwesomeFeatureEnabled</span><span class="o">()).</span><span class="na">thenReturn</span><span class="o">(</span><span class="n">awesomeFeatureEnabled</span><span class="o">);</span>
        <span class="nc">RestAssured</span><span class="o">.</span><span class="na">given</span><span class="o">()</span>
            <span class="o">.</span><span class="na">when</span><span class="o">().</span><span class="na">get</span><span class="o">(</span><span class="s">"/features/awesome"</span><span class="o">)</span>
            <span class="o">.</span><span class="na">then</span><span class="o">().</span><span class="na">body</span><span class="o">(</span><span class="nc">CoreMatchers</span><span class="o">.</span><span class="na">is</span><span class="o">(</span><span class="nc">String</span><span class="o">.</span><span class="na">valueOf</span><span class="o">(</span><span class="n">awesomeFeatureEnabled</span><span class="o">)));</span>
    <span class="o">}</span>
<span class="o">}</span></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>When the tests are run, this method will be invoked once for each value provided with the <code>@ValueSource</code> annotation.</td>
</tr>
</table>
</div>
</div>
</div>
<div class="sect1">
<h2 id="approach-3-constructor-injection">Approach #3: constructor injection</h2>
<div class="sectionbody">
<div class="paragraph">
<p>What if you need native testing in a big project that suffers from the Quarkus test profiles drawbacks mentioned earlier in this post?
Injecting the configuration through your CDI beans constructors might be the right approach for you.</p>
</div>
<div class="listingblock">
<div class="title">The code to be tested</div>
<div class="content">
<pre class="rouge highlight"><code data-lang="java"><span class="kn">import</span> <span class="nn">io.smallrye.config.ConfigMapping</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">io.smallrye.config.WithDefault</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">jakarta.inject.Singleton</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.eclipse.microprofile.config.inject.ConfigProperty</span><span class="o">;</span>

<span class="nd">@Singleton</span>
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">FeaturesService</span> <span class="o">{</span>

    <span class="kd">private</span> <span class="kd">final</span> <span class="nc">FeaturesConfig</span> <span class="n">featuresConfig</span><span class="o">;</span>
    <span class="kd">private</span> <span class="kd">final</span> <span class="kt">boolean</span> <span class="n">amazingFeatureEnabled</span><span class="o">;</span>

    <span class="kd">public</span> <span class="nf">FeaturesService</span><span class="o">(</span> <i class="conum" data-value="1"></i><b>(1)</b>
        <span class="nc">FeaturesConfig</span> <span class="n">featuresConfig</span><span class="o">,</span>
        <span class="nd">@ConfigProperty</span><span class="o">(</span><span class="n">name</span> <span class="o">=</span> <span class="s">"amazing-feature-enabled"</span><span class="o">,</span> <span class="n">defaultValue</span> <span class="o">=</span> <span class="s">"false"</span><span class="o">)</span> <span class="kt">boolean</span> <span class="n">amazingFeatureEnabled</span>
    <span class="o">)</span> <span class="o">{</span>
        <span class="k">this</span><span class="o">.</span><span class="na">featuresConfig</span> <span class="o">=</span> <span class="n">featuresConfig</span><span class="o">;</span>
        <span class="k">this</span><span class="o">.</span><span class="na">amazingFeatureEnabled</span> <span class="o">=</span> <span class="n">amazingFeatureEnabled</span><span class="o">;</span>
    <span class="o">}</span>

    <span class="kd">public</span> <span class="kt">boolean</span> <span class="nf">isAwesomeFeatureEnabled</span><span class="o">()</span> <span class="o">{</span>
        <span class="k">return</span> <span class="n">featuresConfig</span><span class="o">.</span><span class="na">awesomeFeatureEnabled</span><span class="o">();</span>
    <span class="o">}</span>

    <span class="kd">public</span> <span class="kt">boolean</span> <span class="nf">isAmazingFeatureEnabled</span><span class="o">()</span> <span class="o">{</span>
        <span class="k">return</span> <span class="n">amazingFeatureEnabled</span><span class="o">;</span>
    <span class="o">}</span>
<span class="o">}</span>

<span class="nd">@ConfigMapping</span><span class="o">(</span><span class="n">prefix</span> <span class="o">=</span> <span class="s">"features"</span><span class="o">)</span>
<span class="kd">interface</span> <span class="nc">FeaturesConfig</span> <span class="o">{</span> <i class="conum" data-value="2"></i><b>(2)</b>

    <span class="nd">@WithDefault</span><span class="o">(</span><span class="s">"false"</span><span class="o">)</span>
    <span class="kt">boolean</span> <span class="nf">awesomeFeatureEnabled</span><span class="o">();</span>
<span class="o">}</span></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>The configuration is injected in the constructor of the CDI bean.
This approach works with both <a href="https://quarkus.io/guides/config-mappings" target="_blank" rel="noopener">config mappings</a> and <code>@ConfigProperty</code>.</td>
</tr>
<tr>
<td><i class="conum" data-value="2"></i><b>2</b></td>
<td>In a real project, this interface would likely be <code>public</code> and declared in a separate file.</td>
</tr>
</table>
</div>
<div class="listingblock">
<div class="title">The test code</div>
<div class="content">
<pre class="rouge highlight"><code data-lang="java"><span class="kn">import</span> <span class="nn">io.quarkus.test.junit.QuarkusTest</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.junit.jupiter.api.Assertions</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.junit.jupiter.api.Test</span><span class="o">;</span>

<span class="nd">@QuarkusTest</span>
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">FeaturesServiceTest</span> <span class="o">{</span>

    <span class="nd">@Test</span>
    <span class="kt">void</span> <span class="nf">test</span><span class="o">()</span> <span class="o">{</span>

        <span class="nc">FeaturesConfig</span> <span class="n">featuresConfig</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">FeaturesConfig</span><span class="o">()</span> <span class="o">{</span> <i class="conum" data-value="1"></i><b>(1)</b>
            <span class="nd">@Override</span>
            <span class="kd">public</span> <span class="kt">boolean</span> <span class="nf">awesomeFeatureEnabled</span><span class="o">()</span> <span class="o">{</span>
                <span class="k">return</span> <span class="kc">true</span><span class="o">;</span>
            <span class="o">}</span>
        <span class="o">};</span>
        <span class="nc">FeaturesService</span> <span class="n">featuresService</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">FeaturesService</span><span class="o">(</span><span class="n">featuresConfig</span><span class="o">,</span> <span class="kc">true</span><span class="o">);</span> <i class="conum" data-value="2"></i><b>(2)</b>

        <span class="nc">Assertions</span><span class="o">.</span><span class="na">assertTrue</span><span class="o">(</span><span class="n">featuresService</span><span class="o">.</span><span class="na">isAwesomeFeatureEnabled</span><span class="o">());</span>
        <span class="nc">Assertions</span><span class="o">.</span><span class="na">assertTrue</span><span class="o">(</span><span class="n">featuresService</span><span class="o">.</span><span class="na">isAmazingFeatureEnabled</span><span class="o">());</span>
    <span class="o">}</span>
<span class="o">}</span></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>This is used to override the configuration from the <code>FeaturesConfig</code> interface.</td>
</tr>
<tr>
<td><i class="conum" data-value="2"></i><b>2</b></td>
<td>The configuration is overridden from the test when the bean constructor is invoked.
The first argument overrides the configuration that relies on <code>@ConfigMapping</code>.
The second argument overrides the configuration that relies on <code>@ConfigProperty</code>.</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>With this approach, no injections will be performed by CDI when the tests are run because the bean is instantiated manually instead of being managed by the CDI container from Quarkus.
That drawback can be mitigated by injecting all dependencies (other beans and/or configuration) through the constructor of the tested bean.
When that is done, CDI injections still won&#8217;t work but the test code will be able to provide all dependencies required for the test execution.</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="approach-4-testing-components">Approach #4: testing components</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Quarkus recently introduced an experimental feature called <a href="https://quarkus.io/guides/getting-started-testing#testing-components" target="_blank" rel="noopener">Testing components</a> which can be used to override the configuration from the test code.
That feature is provided by the <code>quarkus-junit5-component</code> extension.</p>
</div>
<div class="paragraph">
<p>This approach doesn&#8217;t start the full Quarkus app.
It only starts the CDI container and injects the fields from the test which are annotated with <code>@jakarta.inject.Inject</code> or <code>@io.quarkus.test.InjectMock</code>.
It can therefore be much faster, especially in bigger projects, than the full Quarkus app restarts that come with <a href="#quarkus-test-profiles">Quarkus test profiles</a>.</p>
</div>
<div class="paragraph">
<p>This approach doesn&#8217;t work with native testing because it relies on injections in the test code, which are only supported when the tests are run in JVM mode.</p>
</div>
<div class="paragraph">
<p>Let&#8217;s see how that works:</p>
</div>
<div class="listingblock">
<div class="title">The code to be tested</div>
<div class="content">
<pre class="rouge highlight"><code data-lang="java"><span class="kn">import</span> <span class="nn">io.smallrye.config.ConfigMapping</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">io.smallrye.config.WithDefault</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">jakarta.enterprise.context.ApplicationScoped</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">jakarta.inject.Inject</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.eclipse.microprofile.config.inject.ConfigProperty</span><span class="o">;</span>

<span class="nd">@ApplicationScoped</span>
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">FeaturesService</span> <span class="o">{</span>

    <span class="nd">@Inject</span>
    <span class="nc">FeaturesConfig</span> <span class="n">featuresConfig</span><span class="o">;</span> <i class="conum" data-value="1"></i><b>(1)</b>

    <span class="nd">@ConfigProperty</span><span class="o">(</span><span class="n">name</span> <span class="o">=</span> <span class="s">"amazing-feature-enabled"</span><span class="o">,</span> <span class="n">defaultValue</span> <span class="o">=</span> <span class="s">"false"</span><span class="o">)</span> <i class="conum" data-value="1"></i><b>(1)</b>
    <span class="kt">boolean</span> <span class="n">amazingFeatureEnabled</span><span class="o">;</span>

    <span class="kd">public</span> <span class="kt">boolean</span> <span class="nf">isAwesomeFeatureEnabled</span><span class="o">()</span> <span class="o">{</span>
        <span class="k">return</span> <span class="n">featuresConfig</span><span class="o">.</span><span class="na">awesomeFeatureEnabled</span><span class="o">();</span>
    <span class="o">}</span>

    <span class="kd">public</span> <span class="kt">boolean</span> <span class="nf">isAmazingFeatureEnabled</span><span class="o">()</span> <span class="o">{</span>
        <span class="k">return</span> <span class="n">amazingFeatureEnabled</span><span class="o">;</span>
    <span class="o">}</span>
<span class="o">}</span>

<span class="nd">@ConfigMapping</span><span class="o">(</span><span class="n">prefix</span> <span class="o">=</span> <span class="s">"features"</span><span class="o">)</span>
<span class="kd">interface</span> <span class="nc">FeaturesConfig</span> <span class="o">{</span> <i class="conum" data-value="2"></i><b>(2)</b>

    <span class="nd">@WithDefault</span><span class="o">(</span><span class="s">"false"</span><span class="o">)</span>
    <span class="kt">boolean</span> <span class="nf">awesomeFeatureEnabled</span><span class="o">();</span>
<span class="o">}</span></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>Testing components works with both <a href="https://quarkus.io/guides/config-mappings" target="_blank" rel="noopener">config mappings</a> and <code>@ConfigProperty</code>.</td>
</tr>
<tr>
<td><i class="conum" data-value="2"></i><b>2</b></td>
<td>In a real project, this interface would likely be <code>public</code> and declared in a separate file.</td>
</tr>
</table>
</div>
<div class="listingblock">
<div class="title">The test code</div>
<div class="content">
<pre class="rouge highlight"><code data-lang="java"><span class="kn">import</span> <span class="nn">io.quarkus.test.component.QuarkusComponentTest</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">io.quarkus.test.component.TestConfigProperty</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">jakarta.inject.Inject</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.junit.jupiter.api.Assertions</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.junit.jupiter.api.Test</span><span class="o">;</span>

<span class="nd">@QuarkusComponentTest</span> <i class="conum" data-value="1"></i><b>(1)</b>
<span class="nd">@TestConfigProperty</span><span class="o">(</span><span class="n">key</span> <span class="o">=</span> <span class="s">"features.awesome-feature-enabled"</span><span class="o">,</span> <span class="n">value</span> <span class="o">=</span> <span class="s">"true"</span><span class="o">)</span> <i class="conum" data-value="2"></i><b>(2)</b>
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">FeaturesServiceTest</span> <span class="o">{</span>

    <span class="nd">@Inject</span>
    <span class="nc">FeaturesService</span> <span class="n">featuresService</span><span class="o">;</span>

    <span class="nd">@Test</span>
    <span class="nd">@TestConfigProperty</span><span class="o">(</span><span class="n">key</span> <span class="o">=</span> <span class="s">"amazing-feature-enabled"</span><span class="o">,</span> <span class="n">value</span> <span class="o">=</span> <span class="s">"true"</span><span class="o">)</span> <i class="conum" data-value="2"></i><b>(2)</b>
    <span class="kt">void</span> <span class="nf">test</span><span class="o">()</span> <span class="o">{</span>
        <span class="nc">Assertions</span><span class="o">.</span><span class="na">assertTrue</span><span class="o">(</span><span class="n">featuresService</span><span class="o">.</span><span class="na">isAwesomeFeatureEnabled</span><span class="o">());</span>
        <span class="nc">Assertions</span><span class="o">.</span><span class="na">assertTrue</span><span class="o">(</span><span class="n">featuresService</span><span class="o">.</span><span class="na">isAmazingFeatureEnabled</span><span class="o">());</span>
    <span class="o">}</span>
<span class="o">}</span></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>The usual <code>@QuarkusTest</code> annotation has been replaced with <code>@QuarkusComponentTest</code>.</td>
</tr>
<tr>
<td><i class="conum" data-value="2"></i><b>2</b></td>
<td><code>@TestConfigProperty</code> can be used on the test class, a test method or both.</td>
</tr>
</table>
</div>
</div>
</div>
<div class="sect1">
<h2 id="approach-5-system-properties">Approach #5: system properties</h2>
<div class="sectionbody">
<div class="paragraph">
<p>I would definitely NOT recommend this approach, but it does exist and it kinda works, so I&#8217;ll mention it anyway.
System properties can be used to override the configuration from the test code.
This approach suffers however from major drawbacks:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>It doesn&#8217;t work in native mode.</p>
</li>
<li>
<p>It doesn&#8217;t work with <a href="https://quarkus.io/guides/config-mappings" target="_blank" rel="noopener">config mappings</a>.</p>
</li>
<li>
<p>It only works once when the configuration is defined in an <code>@ApplicationScoped</code> or <code>@Singleton</code> bean, before that bean has been initialized.
After the bean initialization, any changes made to system properties will have no effect on the configuration.</p>
</li>
</ul>
</div>
<div class="listingblock">
<div class="title">The code to be tested</div>
<div class="content">
<pre class="rouge highlight"><code data-lang="java"><span class="kn">import</span> <span class="nn">jakarta.ws.rs.GET</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">jakarta.ws.rs.Path</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.eclipse.microprofile.config.inject.ConfigProperty</span><span class="o">;</span>

<span class="nd">@Path</span><span class="o">(</span><span class="s">"/features"</span><span class="o">)</span>
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">FeaturesResource</span> <span class="o">{</span>

    <span class="nd">@ConfigProperty</span><span class="o">(</span><span class="n">name</span> <span class="o">=</span> <span class="s">"awesome-feature-enabled"</span><span class="o">,</span> <span class="n">defaultValue</span> <span class="o">=</span> <span class="s">"false"</span><span class="o">)</span>
    <span class="kt">boolean</span> <span class="n">awesomeFeatureEnabled</span><span class="o">;</span>

    <span class="nd">@GET</span>
    <span class="nd">@Path</span><span class="o">(</span><span class="s">"/awesome"</span><span class="o">)</span>
    <span class="kd">public</span> <span class="kt">boolean</span> <span class="nf">isAwesomeFeatureEnabled</span><span class="o">()</span> <span class="o">{</span>
        <span class="k">return</span> <span class="n">awesomeFeatureEnabled</span><span class="o">;</span>
    <span class="o">}</span>
<span class="o">}</span></code></pre>
</div>
</div>
<div class="paragraph">
<p>System properties can be set from the command line with Maven or Gradle:</p>
</div>
<div class="listingblock">
<div class="title">Maven command</div>
<div class="content">
<pre class="rouge highlight"><code data-lang="shell">./mvnw verify <span class="nt">-Dawesome-feature-enabled</span><span class="o">=</span><span class="nb">true</span></code></pre>
</div>
</div>
<div class="paragraph">
<p>They can also be set from the test code:</p>
</div>
<div class="listingblock">
<div class="title">The test code</div>
<div class="content">
<pre class="rouge highlight"><code data-lang="java"><span class="kn">import</span> <span class="nn">io.quarkus.test.junit.QuarkusTest</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">io.restassured.RestAssured</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.hamcrest.CoreMatchers</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.junit.jupiter.api.MethodOrderer</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.junit.jupiter.api.Order</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.junit.jupiter.api.Test</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.junit.jupiter.api.TestMethodOrder</span><span class="o">;</span>

<span class="nd">@QuarkusTest</span>
<span class="nd">@TestMethodOrder</span><span class="o">(</span><span class="nc">MethodOrderer</span><span class="o">.</span><span class="na">OrderAnnotation</span><span class="o">.</span><span class="na">class</span><span class="o">)</span> <i class="conum" data-value="1"></i><b>(1)</b>
<span class="kd">class</span> <span class="nc">FeaturesResourceTest</span> <span class="o">{</span>

    <span class="nd">@Test</span>
    <span class="nd">@Order</span><span class="o">(</span><span class="mi">1</span><span class="o">)</span> <i class="conum" data-value="2"></i><b>(2)</b>
    <span class="kt">void</span> <span class="nf">firstTest</span><span class="o">()</span> <span class="o">{</span>
        <span class="nc">System</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="s">"awesome-feature-enabled"</span><span class="o">,</span> <span class="s">"true"</span><span class="o">);</span>
        <span class="nc">RestAssured</span><span class="o">.</span><span class="na">given</span><span class="o">()</span>
            <span class="o">.</span><span class="na">when</span><span class="o">().</span><span class="na">get</span><span class="o">(</span><span class="s">"/features/awesome"</span><span class="o">)</span>
            <span class="o">.</span><span class="na">then</span><span class="o">().</span><span class="na">body</span><span class="o">(</span><span class="nc">CoreMatchers</span><span class="o">.</span><span class="na">is</span><span class="o">(</span><span class="s">"true"</span><span class="o">));</span>
    <span class="o">}</span>

    <span class="nd">@Test</span>
    <span class="nd">@Order</span><span class="o">(</span><span class="mi">2</span><span class="o">)</span> <i class="conum" data-value="3"></i><b>(3)</b>
    <span class="kt">void</span> <span class="nf">lastTest</span><span class="o">()</span> <span class="o">{</span>
        <span class="nc">System</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="s">"awesome-feature-enabled"</span><span class="o">,</span> <span class="s">"false"</span><span class="o">);</span>
        <span class="nc">RestAssured</span><span class="o">.</span><span class="na">given</span><span class="o">()</span>
            <span class="o">.</span><span class="na">when</span><span class="o">().</span><span class="na">get</span><span class="o">(</span><span class="s">"/features/awesome"</span><span class="o">)</span>
            <span class="o">.</span><span class="na">then</span><span class="o">().</span><span class="na">body</span><span class="o">(</span><span class="nc">CoreMatchers</span><span class="o">.</span><span class="na">is</span><span class="o">(</span><span class="s">"true"</span><span class="o">));</span> <i class="conum" data-value="4"></i><b>(4)</b>
    <span class="o">}</span>
<span class="o">}</span></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>In this code snippet, tests are run in a fixed order to demonstrate a limitation of system properties.</td>
</tr>
<tr>
<td><i class="conum" data-value="2"></i><b>2</b></td>
<td>This test always runs first.</td>
</tr>
<tr>
<td><i class="conum" data-value="3"></i><b>3</b></td>
<td>This test always runs last.</td>
</tr>
<tr>
<td><i class="conum" data-value="4"></i><b>4</b></td>
<td>This test depends on a CDI bean with a default <code>@Singleton</code> scope which was already initialized by the previous test.
As a consequence, the outcome of this test cannot be changed from the system property.</td>
</tr>
</table>
</div>
</div>
</div>
<div class="sect1">
<h2 id="conclusion">Conclusion</h2>
<div class="sectionbody">
<div class="paragraph">
<p>First, this post is not a comprehensive list of all existing approaches to override the configuration from the test code.
There are additional options such as using reflection (hardly maintainable) which I did not include, and probably approaches I&#8217;m not even aware of.
Please don&#8217;t hesitate to share your experience and opinion about this topic in the comments!</p>
</div>
<div class="paragraph">
<p>Most of you probably started reading this post with a question in mind: what is the best approach?
Well, as you probably understood through the post, none of them is perfect (yet).
They all come with drawbacks.
In my experience, the real question is not about picking the best approach, but rather about how to better combine different approaches and use the best they each have to offer.</p>
</div>
<div class="paragraph">
<p>If you&#8217;re unsure about which approach you may introduce in your project, the <a href="https://github.com/gwenneg/blog-overriding-configuration-from-test-code" target="_blank" rel="noopener">gwenneg/blog-overriding-configuration-from-test-code</a> repository might help you make that decision.
It contains an implementation of all approaches mentioned in this post.</p>
</div>
<div class="paragraph">
<p>Thanks for reading this post! I hope it will help you better test your Quarkus apps.</p>
</div>
</div>
</div>]]></content><author><name>Gwenneg Lepage</name></author><category term="configuration" /><category term="java" /><category term="mockito" /><category term="quarkus" /><category term="testing" /><summary type="html"><![CDATA[Increase your test coverage by overriding the configuration of your Quarkus app from its test code.]]></summary></entry></feed>