Nite

Go GMP Model

nite@nite07.com (Nite) — Tue, 04 Aug 2026 03:10:52 +1000

GMP is the trio of core components in the Go scheduler:

G (Goroutine): the task created by go func().
M (Machine): an OS thread.
P (Processor): scheduling context that manages a G queue.

M is an OS thread. The runtime creates it via newm/newosproc (the functions that create M and its underlying thread), and the OS then schedules it onto a CPU core. M is the smallest unit that executes Go code; an M without a P can only sleep or handle system calls.

G states and lifecycle

A G has a well-defined set of runtime states (the enum in runtime2.go):

Gidle: just allocated, not yet initialized.
Grunnable: queued in a run queue, waiting for an M to pick it up.
Grunning: executing user code.
Gsyscall: executing a syscall, temporarily detached from scheduling.
Gwaiting: parked (suspended), waiting on a channel, lock, or IO.
Gdead: finished, sitting in a free list for reuse.

flowchart LR
    A["Gidle"] --> B["Grunnable"]
    B --> C["Grunning"]
    C --> D["Gsyscall"]
    C --> E["Gwaiting"]
    D --> B
    E --> B
    C --> F["Gdead"]
    F --> B

A finished G is not destroyed; it goes into a free-G cache for reuse when a new G is created. The cache has two levels: each P keeps a local cache (pp.gFree), and there is a shared global pool (sched.gFree, lock-protected). On return, a G goes into the current P’s local cache first; on creation, a new G is taken from the local cache first, and only when the local cache is empty are 32 Gs fetched from the global pool at once. Fetching in bulk minimizes global-lock contention: the next 32 creations all hit the lock-free local cache.

Stack reuse is best-effort. A G’s stack grows dynamically and may end up far larger than the initial size. If a G’s stack size doesn’t match the standard initial stack size, the stack is freed on return; the same happens when a reused G turns out to have a mismatched stack. Only a G whose stack size matches exactly can be reused together with its stack.

A goroutine stack starts at only 2KB (stackMin), far smaller than a thread’s default stack (megabytes). That is why G is lightweight and why millions of goroutines are feasible. The stack grows on demand, up to 1GB on 64-bit platforms.

Queue structure

Each P has a local G queue, and there is one shared global G queue. The global queue is lock-protected (sched.lock, the scheduler lock); the local queue is lock-free. This is the key to scheduling performance.

flowchart TB
    GQ["Global G queue<br/>lock-protected"]
    P1["P1"]
    P2["P2"]
    P3["P3"]
    Q1["Local queue<br/>runnext + runq[256]"]
    Q2["Local queue"]
    Q3["Local queue"]
    M1["M1 OS thread"]
    M2["M2 OS thread"]
    M3["M3 OS thread (idle)"]
    CPU["CPU core<br/>scheduled by OS"]
    GQ -->|take G| Q1
    GQ -->|take G| Q2
    GQ -->|take G| Q3
    Q1 --- P1
    Q2 --- P2
    Q3 --- P3
    P1 --- M1
    P2 --- M2
    P3 -. idle .-> M3
    M1 --> CPU
    M2 --> CPU

A new G goes into the current P’s local queue, most preferably into the runnext slot (the next-to-run position; each P has exactly one, and it may be empty). If runnext is empty, the new G takes it directly; if it is occupied, the old G is pushed out into the runq (local run queue) ring buffer. runq is a fixed-size array of 256, so together with runnext the P holds 257 Gs.

What goes into runnext: a newly created G (go func()), a woken G that needs to run soon (ready called with next=true, e.g. the finalizer G), and a preempted G — so it can resume immediately after the STW ends.

This is the locality principle: the new G was just created by the current G, and running it on the same P means its data is likely still in the CPU cache, maximizing hit rate when it runs immediately.

When the local queue can’t take more (runnext occupied and runq full at 256), runqputslow (the function that moves half the queue to the global queue) moves the front half (the 128 earliest Gs) together with the displaced old G (the one that was sitting in runnext, not yet run) to the global queue. Old Gs go, the new G stays: the new G must run soon, so it remains in runnext; the front-half Gs have waited the longest, and once in the global queue they can be picked up by other idle Ps, which doubles as load balancing. So the “maximum length of a P’s G queue” is 256 + 1, not unbounded.

Why move half at once instead of just the displaced G?

Question: the new G takes runnext, so why not just throw the displaced old G into the global queue instead of moving 128 Gs along with it?

The answer has two layers:

The old G needs a place to go, so runq must free up space. The displaced G can’t re-enter runnext (the new G holds it) and can only go into runq, which is already full at 256. Making room in runq is a hard requirement, so front-half Gs must move.

Moving in bulk minimizes global-lock contention. The global queue has a lock (sched.lock). If we freed just one slot each time: create a G → runq full again → free one more → take the global lock again. Goroutine creation is a hot path; taking a global lock on every creation is unacceptable. Moving 128 at once takes the lock once, leaving runq half-empty (128 free slots); many subsequent enqueues then go through the lock-free local path until the queue fills up again. Moving half also leaves headroom: runq drops from 256 to 128, so new Gs don’t immediately hit the ceiling.

Question: idle Ps will steal from a full queue anyway via work stealing — why move anything at all?

Stealing has its own cost: when several idle Ps target the same full queue, they contend, and the losers retry, burning CPU. Once half the queue is in the global pool, idle Ps can just call globrunqgetbatch and grab a batch with one lock acquisition, no fighting. But work stealing is only a fallback — a full queue means production outpaces consumption, and relying on others to steal doesn’t unblock this P’s enqueue; making room is mandatory.

So the core reasons for moving half are “no room for the old G” plus “can’t take a global lock on every creation”; load balancing is a side benefit.

Number of Ps and Ms

The number of Ps equals GOMAXPROCS (maximum processors), which defaults to the number of logical CPU cores. It can be changed via the GOMAXPROCS environment variable or runtime.GOMAXPROCS(n).

The number of Ms changes dynamically: a new M is created when there is work to do but not enough idle Ms; idle Ms park in a sleep list (sched.midle, the idle-M list) waiting to be reused, and ones idle for too long are destroyed. The default cap is 10000, adjustable via runtime/debug.SetMaxThreads (set max threads).

Each M is bound to 0 or 1 P at any moment: only an M holding a P can execute Go code. An M obtains Gs through its bound P, in this order: runnext → local queue → global queue → steal from other Ps.

Thread reuse

work stealing

When a P’s local queue is empty, the order for getting Gs is: global queue first (one batch at a time via globrunqgetbatch, sized min(128, globalLen, globalLen/GOMAXPROCS+1), where globalLen is the number of Gs currently queued in the global queue; dividing by GOMAXPROCS splits the batch fairly among all Ps, and 128 is the ceiling because the local queue can’t hold more), then netpoll (picking up Gs whose IO is ready), and only then stealing from other Ps.

When stealing, the thief takes the front half of the target P’s runq (runqgrab does n = n - n/2, counting n from head), i.e. the half that entered the queue earliest. If nothing can be stolen and the global queue is empty too, the P joins the idle list and waits. The number of Ps is fixed — Ps are never destroyed; it’s Ms that sleep or get destroyed.

There’s also a fairness guard: every 61 scheduling ticks (schedtick%61 == 0) the scheduler forces a check of the global queue and takes from it first if non-empty. This prevents two Gs endlessly spawning new Gs in a local queue and starving the Gs in the global queue.

hand off

When a G blocks, the handling differs by case:

Ordinary blocking (channel, lock, sleep): the G is parked (suspended until its condition is met), the M stays put and keeps executing the next G from the P’s queue. No new M is needed.
Syscall blocking: the M enters the syscall together with the G, and the P is released (handoffp, the hand-off function); a sleeping M is woken to take over the P, or a new one is created if none exists.

When the syscall returns, the M first tries to get a P back: with an idle P available it just resumes the G; without one, the G goes into the global queue (globrunqput) and the M sleeps (stopm) until woken again.

Load balancing

Every new G creation (newproc, the function that creates a G) calls wakep (the M-waker): if there is no spinning M right now and an idle P exists, it wakes a sleeping M or creates a new one, binds it to that P, and sends it into spinning mode to hunt for Gs.

A spinning M’s order for getting Gs: local queue (the idle P it just picked up has an empty queue anyway) → global queue → steal from other Ps.

Spinning Ms don’t burn CPU forever:

Their count is capped at half the number of busy Ps (2 × spinningMs < GOMAXPROCS - idlePs must hold for a new one).
When neither the global queue nor any P has work, the spinning M returns its P to the idle list and sleeps itself (stopm → mPark, thread park) until the next wakep wakes it.

sysmon and preemption

This is not “force-kill after timeout”. sysmon (the system monitor thread) is a dedicated thread created at runtime startup (newm runs the sysmon function), responsible for several things:

Preemption: at most every 10ms it checks whether a G has been running on the same P for over 10ms (forcePreemptNS, the preemption time threshold); if so, it asynchronously preempts via a SIGURG signal (a Unix signal used for preemption, Go 1.14+). The G yields the CPU, goes back to the queue, and will be scheduled again later.
Syscall takeover: if a P has been stuck in a syscall for over ~20us, sysmon takes the P back and hands it to another M (working with hand off).
Netpoll fallback: if nobody has polled network events for over 10ms, sysmon polls epoll itself and wakes ready Gs.
Periodic GC: if no GC has run within forcegcperiod (2 minutes), sysmon forces one.

Network IO and netpoll

Network IO (socket reads/writes) is non-blocking by default in Go, built on IO multiplexing like epoll (Linux) / kqueue (macOS). When a G’s read/write isn’t ready, netpollblock calls gopark to suspend it (waitReasonIOWait), without occupying an M. Ready events are collected by netpoll, which wakes the corresponding Gs and pushes them back into queues.

So when many goroutines do network IO at once, Ms are never blocked by IO. This is the core of GMP’s support for high-concurrency network services: no matter how many connections, the number of active threads stays around GOMAXPROCS.

Note the distinction: file IO and syscall blocking take the other path (hand off, occupying an M), while network IO goes through netpoll and occupies no M.

Scheduling flow

go func(){...}(): creates a G, which goes into the current P’s runnext first, then the local queue; when the local queue is full at 256, the front half of Gs plus the displaced old G move to the global queue.
The M takes a G from its bound P: runnext → local queue → global queue → steal from other Ps.
G blocks: on ordinary blocking the M keeps executing the next G; on syscall blocking it hands the P off to another M.
Syscall returns: the G re-enters a queue, and the M grabs a P or sleeps.

flowchart LR
    A["go func()"] --> B["create G"]
    B --> C{"P local queue full?"}
    C -- no --> D["put in runnext / local queue"]
    C -- yes --> E["front half Gs + displaced old G to global queue"]
    D --> F["M takes G from P and runs it"]
    E --> F
    F --> G{"G blocks?"}
    G -- ordinary --> H["park G, M keeps running next G"]
    G -- syscall --> I["hand off P to another M"]
    H --> F
    I --> F

P states

A P has four states (runtime2.go):

Pidle: in the idle-P list, queue empty, waiting to be picked up by an M.
Prunning: held by an M, executing Gs.
Pgcstop: GC’s STW (stop the world, all threads paused) phase, all Ps stop here.
Pdead: destroyed when GOMAXPROCS shrinks; reusable if GOMAXPROCS grows again.

GC starts with STW: all Ms reach a safe point, Ps transition to Pgcstop, and scheduling pauses. The marking phase then proceeds concurrently and Ps resume running user code. For the full GC picture, see Go GC mechanism.

M0 and G0

M0 is the first M created at program startup (the main thread). It handles initialization (installing signal handlers, creating the sysmon thread, etc.) and never exits — in mexit (the thread-exit path) m0 is wedged in place until the process ends.

G0 is each M’s dedicated scheduling goroutine. Every M creation (allocm, the function that creates an M) also creates its g0, stored in the M’s g0 field. g0 enters no queue and occupies no P; it only runs scheduler code (schedule, stack growth, signal handling, etc.). It is independent of P and tied only to its M: when the M blocks, g0 goes with it — there’s no “where does g0 live” problem.

Nor is there any “m0 and g0 unbinding”: g0 always belongs to m0, and m0 enters the scheduling loop through g0.

Hello world execution flow

Program starts; m0 and its g0 are created.
g0 runs schedinit (scheduler initialization): creates GOMAXPROCS Ps, initializes the global queue.
newproc creates the main goroutine and puts it in some P’s local queue.
m0’s g0 enters the scheduling loop, picks up the main goroutine, and m0 switches to running it.
main finishes; the program exits.

runtime/trace

import "runtime/trace"

f, _ := os.Create("trace.out")
trace.Start(f)
defer trace.Stop()

// your code

After generating the file, run go tool trace trace.out to open the Web UI, which shows G/P/M scheduling, blocking, syscalls, GC, and other events.

Mitigation measures for self-hosted Gitea instances overwhelmed by crawler traffic

nite@nite07.com (Nite) — Sun, 02 Aug 2026 18:19:54 +1000

Symptoms

A self-hosted Gitea started consuming abnormal bandwidth. The provider panel showed the traffic allowance draining at an alarming rate, with the monthly quota running out fast. Aggregating access logs by response size, the git domain accounted for the vast majority of traffic, dominated by /owner/repo/compare/ version-comparison pages with single responses up to 10MB.

The crawler pattern was obvious: Amazonbot, Lightpanda, and a rotating pool of IPs faking old Chrome/Firefox user agents, repeatedly fetching blame and compare pages. Most unique IPs made only one or two requests — classic distributed scraping.

Causes

Crawler user agents vary widely, from well-known bots (Amazonbot, Lightpanda) to rotating IPs faking browser UAs.
Heavy pages are large — compare pages exceed 10MB uncompressed — so every crawl burns real bandwidth.
Per-IP rate limiting does nothing against a rotating IP pool, where most IPs make one or two requests and move on.

Fixes

1. Caddy config

The UA blacklist (@block_bots returns 403 on match) and the heavy-page rate limit (@heavy, 2 requests per IP per 10s) live in the same site block and run in order. The UA list covers common crawlers and scanners; extend or trim it as needed:

git.example.com {
	# Logging: per-domain access log
	import log git.example.com

	# 1. UA blacklist: known crawlers get 403
	@block_bots header_regexp User-Agent "(?i)(SemrushBot|Lightpanda|GPTBot|ClaudeBot|Bytespider|AhrefsBot|MJ12bot|DataForSeoBot|Amazonbot|DotBot|CCBot|cohere-ai|Diffbot|Google-Extended|Meta-ExternalAgent|Meta-ExternalFetcher|Applebot-Extended|Timpibot|ImagesiftBot|Crawl4AI|Scrapy|ChatGLM-Spider|DeepSeekBot|cohere-training-data-crawler|AI2Bot|TikTokSpider|omgili|BLEXBot|Exabot|360Spider|80legs|MegaIndex|Barkrowler|DataCha0s|Nikto|Nmap|Nessus|OpenVAS|Sqlmap|WPScan|Nuclei|Masscan|Shodan|Acunetix|Dirbuster|Whatweb|Havij|Fimap|Jbrofuzz|Zgrab|Censys)"
	handle @block_bots {
		respond 403
	}

	# 2. Rate limit heavy pages (compare/blame etc.) per IP
	@heavy path_regexp heavy_path ^/[^/]+/[^/]+/(?:compare|blame|commit|commits|archive|pulls|issues)(?:/|$)
	handle @heavy {
		rate_limit {
			zone dynamic_zone {
				key {client_ip}
				events 2
				window 10s
			}
			log_key
		}
		reverse_proxy 100.64.0.10:3000
	}

	# 3. robots.txt: cooperative crawlers read the rules first
	handle /robots.txt {
		respond `User-agent: *
Disallow: /blame/
Disallow: /compare/
Disallow: /commit/
Disallow: /commits/
Disallow: /archive/
Disallow: /pulls/
Disallow: /issues/`
	}

	# 4. Everything else goes to Gitea
	reverse_proxy 100.64.0.10:3000
}

The @heavy path pattern covers compare, blame, commit, and other heavy page types; an IP that exceeds 2 requests in 10 seconds gets a 429. With log_key enabled, rate-limited requests are logged with their IP, which helps tune the threshold.

2. Enable origin gzip in Gitea

Add ENABLE_GZIP = true to the [server] section of app.ini, then restart Gitea:

[server]
ENABLE_GZIP = true

ENABLE_GZIP compresses runtime-generated content only; static assets are unaffected. Enable it at the origin rather than at the reverse proxy because the path is origin → proxy → visitor, and under bidirectional billing origin compression saves on both legs, while proxy-only compression saves on one. Caddy’s reverse_proxy adds Accept-Encoding: gzip for clients that omit it and passes the upstream’s compressed response through untouched. No encode directive is needed on the Caddy side.

References

Gitea custom template inline scripts blocked by CSP nonce

nite@nite07.com (Nite) — Sun, 02 Aug 2026 02:03:49 +1000

Symptom

After putting the Matomo tracking script into $GITEA_CUSTOM/templates/custom/header.tmpl (rendered before the closing </head> tag), no statistics show up on the site. The browser console reports:

Executing inline script violates the following Content Security Policy directive 'script-src * 'nonce-<random>''. Either the 'unsafe-inline' keyword, a hash ('sha256-...'), or a nonce ('nonce-...') is required to enable inline execution. The action has been blocked.

Cause

Gitea 1.27.0 switched its CSP inline-script policy to a per-request nonce (PR #37232): a random nonce is generated per response and script-src becomes * 'nonce-xxx': external scripts still load, but inline <script> tags only execute when they carry that nonce. Inline scripts in custom templates have no nonce attribute, so the browser blocks them. The 1.27.0 release notes mention this too: inline <script> tags injected by custom templates stop executing until they are updated to carry the nonce.

Fix

Add a nonce attribute to the script tag, using the template variable {{ctx.CspScriptNonce}}:

<script nonce="{{ctx.CspScriptNonce}}">
  // your inline script, e.g. Matomo tracking code
</script>

After saving, restart Gitea (just restart the container for container deployments) and refresh the page: the console no longer reports CSP blocking errors, and visitors show up in the statistics dashboard.

Patching a Steam Game with Goldberg Steam Emulator

nite@nite07.com (Nite) — Sun, 26 Jul 2026 12:30:00 +1000

Goldberg Steam Emulator (GSE) replaces a game’s steam_api(64).dll so it can run without the Steam client. The game calls the emulator instead of Steam, and the emulator fakes the responses.

The original project is no longer actively maintained. The community now uses GBE Fork, an actively developed fork. You also need Steamless to strip the DRM wrapper from the exe.

Relevant projects:

GBE Fork — the emulator this guide uses
Steamless — SteamDRM wrapper removal tool

1. Run Steamless on the exe

Steamless removes the SteamDRM Wrapper (Steam Stub) from the exe. You can run it on any exe — if there’s no DRM, it skips safely without damaging the file.

Download Steamless, open Steamless.exe, pick the game’s main exe, and click unpack. A .unbind file will appear in the same directory. Back up the original exe, then rename the .unbind file to replace it.

Large exes (over 100MB) can take a few minutes.

2. Find steam_api(64).dll

Search steam_api*.dll in the game folder.

There are two possible scenarios.

Only one DLL (generic / Unity engine): replace that DLL. Put steam_settings next to it.

Two DLLs (Unreal Engine game):

<GameDir>\Engine\Binaries\ThirdParty\Steamworks\Steamv157\Win64\steam_api64.dll
<GameDir>\<GameName>\Binaries\Win64\steam_api64.dll

UE loads steam_api64.dll from the Engine directory at startup, not from the game’s binary directory. Replace the Engine one, put steam_settings in the Engine directory as well. Leave the game’s main binary directory DLL as-is.

3. Back up the original DLL

Copy the DLL you’re about to replace and rename it to steam_api64.dll.old. You’ll need this backup later to generate the interfaces file.

4. Replace with the emulator DLL

From the GBE Fork release package, copy the correct architecture DLL over the original:

64-bit game: regular\x64\steam_api64.dll
32-bit game: regular\x86\steam_api.dll

The emulator DLL is about 7–11MB, the original one about 200–400KB. The size difference is obvious.

For UE games, leave the game’s main directory DLL unchanged.

5. Create the steam_settings directory

Create a steam_settings folder next to the emulator DLL. UE engine: put it in the Engine directory. Generic engine: next to the game exe.

6. Add steam_appid.txt

Find the AppID from the Steam store URL: store.steampowered.com/app/<number>/.

Create steam_settings\steam_appid.txt containing only the numeric AppID. For example 3683770.

7. Generate steam_interfaces.txt (recommended)

Mismatched interface versions can cause crashes or missing functionality. Run generate_interfaces_x64.exe (or the x86 version) from the GBE Fork tools against the backed-up original DLL.

Drag steam_api64.dll.old into the tool window. The tool creates steam_interfaces.txt in the current directory. Move it into steam_settings\.

Always run this against the original DLL, never the emulator DLL.

8. Optional: configure INI files

The steam_settings.EXAMPLE\ folder has templates. Copy them to steam_settings\ and remove the .EXAMPLE infix.

Key settings:

configs.user.ini: local_save_path=./path/relative/to/dll — saves go to the game directory instead of %appdata%
configs.user.ini: account_name=Player — in-game username
configs.user.ini: language=english
configs.main.ini: offline=0 — offline mode
configs.app.ini: unlock_all=1 — unlock all DLCs

You don’t need any INI files for the emulator to work — defaults are sensible.

Common issues

Game crashes or won’t start. Check in this order: did Steamless run on the exe, is the DLL in the right place (UE engine), is steam_settings next to the DLL, was steam_interfaces.txt generated from the original DLL.

Default saves live in %appdata%\GSE Saves\. Set local_save_path in configs.user.ini for per-game portable saves.

UE engine games: replace only the Engine directory DLL, leave the game’s main directory DLL alone. Put steam_settings in the Engine directory too.

Restore original state: rename steam_api64.dll.old back to steam_api64.dll, delete steam_settings and steam_appid.txt.

steam_interfaces.txt is created in the tool’s current working directory, not the tool’s own location. cd to the emulator DLL directory first, then run the tool, then move the file to steam_settings\.

Automation tools

Two community tools if you prefer not to do it manually:

ARMGDDN Autocracker runs on GBE Fork. Install it, right-click a game folder, and it does everything automatically. Repo: https://github.com/KaladinDMP/ARMGDDN-Autocracker. The author’s GitHub was suspended at one point; the latest version may be on Telegram.

SteamAutoCracker wraps Steamless and Goldberg into a single script — one-click DRM removal + DLL replacement. Repo: https://github.com/BigBoiCJ/SteamAutoCracker.

Continuously monitor Linux traffic with vnStat and picosnitch

nite@nite07.com (Nite) — Thu, 23 Jul 2026 11:00:00 +1000

When traffic on a VPS looks abnormal, there are usually two questions: which network interface transferred how much data and when, and which service was actually responsible.

vnStat records sent and received traffic per network interface. It is useful for total egress and hourly, daily, or monthly trends. picosnitch records connections per executable, letting you continue down to processes, domains, ports, and byte counts. Both keep collecting data in the background, but at different levels, so they complement each other well.

vnStat: per-interface traffic

vnStat is not a packet sniffer. It reads the network-interface counters exposed by the kernel, while vnstatd periodically writes the data to its database. Resource usage stays low. It retains traffic records at five-minute, hourly, daily, monthly, and yearly resolutions.

Install it and start the service on Arch Linux:

sudo pacman -S vnstat
sudo systemctl enable --now vnstat

On its first start, the service creates database entries for available interfaces. List the ones already being recorded:

vnstat --dbiflist

An interface added later can be explicitly added, for example after Tailscale creates tailscale0:

sudo vnstat --add -i tailscale0

Common overview queries:

# Default summary: today, current month, and all time
vnstat -i wlan0

# Short summary
vnstat -i wlan0 -s

# Single parseable line for scripts
vnstat --oneline -i wlan0

# Last 7 days, 12 hours, and recent months
vnstat -i wlan0 -d 7
vnstat -i wlan0 -h 12
vnstat -i wlan0 -m

For the time range of an anomaly, combine --begin and --end with daily, hourly, or five-minute lists. With --end, vnStat adds a sum line at the end: the total received and transmitted traffic for that range.

# Daily records and the range total
vnstat -i wlan0 -d --begin 2026-07-20 --end 2026-07-31

# Hourly records within a day, precise to the minute
vnstat -i wlan0 -h --begin "2026-07-28 08:00" --end "2026-07-28 20:00"

# Five-minute records during a peak period
vnstat -i wlan0 -5 --begin "2026-07-28 12:00" --end "2026-07-28 14:00"

--begin accepts YYYY-MM-DD HH:MM, YYYY-MM-DD, and today; --end accepts the first two formats. They are only available with list output, JSON, or XML.

Use JSON when a script needs raw byte counts:

vnstat -i wlan0 -d --begin 2026-07-28 --end 2026-07-30 --json \
  | jq '.interfaces[0].traffic.days'

For a real-time rate or graphs:

# Press Ctrl+C to show statistics for this run
vnstat --live -i wlan0

# Terminal bar graph for the past 24 hours
vnstat -i wlan0 -hg

# Render daily data to a PNG
vnstati -i wlan0 -d -o ~/wlan0.png

Five-minute data is retained for only 48 hours by default. To keep fine-grained records of peak periods longer, change 5MinuteHours in /etc/vnstat.conf, then restart the service:

5MinuteHours 336

sudo systemctl restart vnstat

vnStat only records traffic from the point vnstatd starts running; it cannot recover traffic that passed through an interface earlier.

picosnitch: per-process traffic

vnStat can reveal when total interface traffic grows, but it cannot say which service generated it. picosnitch is a userspace daemon using BPF and fanotify. It records network connections by executable and stores them in SQLite. Its records include the executable path and hash, parent process, domain, port, user, and bytes sent and received. It can identify applications running in containers too.

The picosnitch upstream project recommends a system-wide pipx installation. On Arch Linux:

sudo pacman -S python-pipx
sudo pipx install picosnitch --global
sudo picosnitch systemd
sudo systemctl enable --now picosnitch

It requires Python 3.12 or newer and a Linux kernel capable of running modern libbpf CO-RE programs.

There are three ways to view the data:

# Terminal UI for historical connections
picosnitch tui

# Web UI, listening on http://localhost:5100 by default
picosnitch webui

# Live event stream
sudo picosnitch top

Configuration is at /etc/picosnitch/config.toml. The local SQLite database defaults to /var/lib/picosnitch/picosnitch.db and retains 30 days of history by default. To retain more history, change retention_days and restart the service:

[database]
retention_days = 90

sudo systemctl restart picosnitch

The Web UI listens locally by default. To make it reachable from a LAN or reverse proxy, set its address and port before launching it. Apply appropriate access control before exposing it publicly:

PICOSNITCH_HOST=0.0.0.0 PICOSNITCH_PORT=5100 picosnitch webui

Investigate anomalies with both tools

Start with vnStat to see the traffic that actually passed through an interface, then identify the day or hour when it grew. With that time window narrowed, open picosnitch’s TUI or Web UI and inspect the executables, domains, and ports for the same period.

For example, if vnStat shows a sharp increase in transmitted traffic through wlan0 across two hours at noon, check that range in picosnitch next. If the traffic is concentrated in caddy, inspect the clients it connected to; if it is concentrated in a backup job or downloader, continue with that service’s logs and configuration. Interface accounting finds the problem. Process accounting identifies its source.

Fixing Podman build newuidmap errors on WSL2 Arch Linux

nite@nite07.com (Nite) — Sat, 11 Jul 2026 20:23:42 +1000

When running rootless Podman inside Arch Linux on WSL2, an image build may fail before it even starts processing the Dockerfile:

podman build . -t game-crawler:latest

The error looks like this:

WARN[0000] "/" is not a shared mount, this could cause issues or missing mounts with rootless containers
ERRO[0000] running `/usr/sbin/newuidmap 61029 0 1000 1 1 100000 65536`: newuidmap: Could not set caps
Error: cannot set up namespace using "/usr/sbin/newuidmap": should have setuid or have filecaps setuid: exit status 1

The "/" is not a shared mount line is only a warning. The build fails because of newuidmap: Could not set caps.

When this happens

This is a rootless Podman setup problem, not a Dockerfile problem.

Podman’s official rootless tutorial explains that rootless Podman needs subordinate UID/GID ranges in /etc/subuid and /etc/subgid. The newuidmap(1) manual says that newuidmap sets the UID mapping of a user namespace and checks whether the caller is allowed to use those ranges through /etc/subuid.

Rootless Podman therefore needs these helper binaries when creating the container namespace:

/usr/bin/newuidmap
/usr/bin/newgidmap

On Arch Linux, they come from the shadow package. If they have neither the setuid bit nor the required file capabilities, an unprivileged user cannot complete the UID/GID mapping step, and podman build fails while setting up the namespace.

Check the current state

Check permissions and file capabilities first:

stat -c '%A %U:%G %a %n' /usr/bin/newuidmap /usr/bin/newgidmap
getcap /usr/bin/newuidmap /usr/bin/newgidmap

A broken setup may look like this:

-rwxr-xr-x root:root 755 /usr/bin/newuidmap
-rwxr-xr-x root:root 755 /usr/bin/newgidmap

getcap prints nothing.

Then check whether the current user has subordinate IDs:

grep "^$USER:" /etc/subuid /etc/subgid

A valid setup has entries like these:

/etc/subuid:nite:100000:65536
/etc/subgid:nite:100000:65536

If the user has no entries there, add the subordinate ID ranges first. If the entries already exist and newuidmap/newgidmap still have no setuid bit or file capabilities, fix the helper permissions.

Fix it with file capabilities

Give each helper the minimal capability it needs:

sudo setcap cap_setuid=ep /usr/bin/newuidmap
sudo setcap cap_setgid=ep /usr/bin/newgidmap

Verify it:

getcap /usr/bin/newuidmap /usr/bin/newgidmap

Expected output:

/usr/bin/newuidmap cap_setuid=ep
/usr/bin/newgidmap cap_setgid=ep

On Arch Linux, /usr/sbin is usually a symlink to bin, so an error mentioning /usr/sbin/newuidmap is still fixed by changing /usr/bin/newuidmap.

Verify the rootless namespace

There is no need to run a full image build just to test the fix. Use podman unshare first:

podman unshare sh -c 'echo uid_map:; cat /proc/self/uid_map; echo gid_map:; cat /proc/self/gid_map'

After the fix, the output should look like this:

uid_map:
         0       1000          1
         1     100000      65536
gid_map:
         0       1000          1
         1     100000      65536

Here 1000 is the WSL user’s UID, and 100000 65536 comes from /etc/subuid and /etc/subgid.

Then retry the build:

podman build . -t game-crawler:latest

If setcap is missing

Check whether setcap/getcap are installed:

command -v setcap getcap

If not, install the package that provides them:

sudo pacman -S libcap

The traditional fallback is to set the setuid bit on both helpers:

sudo chmod u+s /usr/bin/newuidmap /usr/bin/newgidmap

Check it with:

stat -c '%A %U:%G %n' /usr/bin/newuidmap /usr/bin/newgidmap

The mode should show -rwsr-xr-x. If file capabilities work, prefer setcap because it grants a narrower privilege.

Fixing Windows Interop in Arch Linux WSL2

nite@nite07.com (Nite) — Thu, 09 Jul 2026 16:33:30 +0800

What Is WSL Interop

WSL Interop is a core feature of WSL2 that lets you invoke Windows executables directly from within a WSL terminal. For example:

powershell.exe -NoProfile -Command "Get-Process | Select-Object -First 5"
winget.exe search nushell
explorer.exe .

The Windows filesystem is also accessible through /mnt/c/, /mnt/d/, and so on.

The Problem

On Arch Linux WSL2, you may find that all of the above commands fail with “exec format error” or “command not found.” This happens because WSL Interop registration depends on systemd-binfmt.service, which gets skipped on Arch Linux due to unmet condition checks — Arch doesn’t ship pre-existing binfmt.d configuration files like Ubuntu does.

Fix

Run the following in your WSL terminal:

sudo mkdir -p /etc/binfmt.d
sudo tee /etc/binfmt.d/wsl-interop.conf > /dev/null << 'EOF'
:WSLInterop:M::MZ::/init:P
EOF

Then restart the service:

sudo systemctl restart systemd-binfmt

Verification

# Check service status — should be active (exited), not skipped
systemctl status systemd-binfmt

# Check that WSLInterop is registered
cat /proc/sys/fs/binfmt_misc/WSLInterop

# Test invoking a Windows program
powershell.exe -NoProfile -Command "Write-Output 'interop OK'"

The configuration file is persistent and takes effect automatically every time WSL starts — no need to repeat this step.

Configuring Fail2Ban for a Podman Mail Server with UFW Bans

nite@nite07.com (Nite) — Thu, 09 Jul 2026 15:30:00 +0800

My self-hosted mail server, running Postfix and Dovecot, lives in a rootless Podman container on a netcup VPS. A Postfix log summary showed 2,477 rejected messages in one day, with 2,471 coming from the same IP. There were also 152 failed SASL LOGIN attempts, along with scanning, relay abuse, and TLS probes.

Postfix can reject spam itself, but SASL password guessing is a separate problem. Fail2Ban identifies sources with repeated failures in the logs and tells the firewall to ban them. Since the server already uses UFW to manage inbound rules, Fail2Ban uses its built-in ufw action instead of writing directly to iptables.

Environment

Item	Value
Host OS	Arch Linux
Container runtime	Rootless Podman
Mail server image	`ghcr.io/docker-mailserver/docker-mailserver:latest`
Mail server service	`systemd-mailserver`
Quadlet file	`~/pod/docker-mailserver/mailserver.container`
Host mail log	`~/pod/docker-mailserver/logs/mail.log`
Container mail log	`/var/log/mail/mail.log`
Fail2Ban	1.1.0
Firewall	UFW

1. Install and inspect the current state

Install and start Fail2Ban:

sudo pacman -S fail2ban
sudo systemctl enable --now fail2ban

Confirm that UFW is enabled, Fail2Ban is running, and see which jails are active:

sudo ufw status verbose
sudo systemctl is-active fail2ban
sudo fail2ban-client status

UFW should report Status: active. Once this configuration is complete, the jail list includes:

Status
|- Number of jail:  3
`- Jail list:       postfix, postfix-sasl, sshd

backend and banaction are separate settings:

backend determines where Fail2Ban reads failure records, such as journald or a regular log file.
banaction determines how an IP is banned once it reaches the threshold.

The mail container writes logs to a host-mounted file, so the mail jails need backend = polling. Switching to UFW does not change log collection.

2. Make the Postfix jails read the container log

A traditional Postfix installation usually writes logs to /var/log/mail.log. This path inside the container is not visible to Fail2Ban on the host, so use the actual host-side volume path instead.

Fail2Ban on Arch selects the systemd log backend for Postfix through paths-arch.conf by default. The logs here are files, so explicitly configure polling in a local override.

Create or edit /etc/fail2ban/jail.d/mailserver.local:

[DEFAULT]
postfix_backend = polling
postfix_log = /home/nite/pod/docker-mailserver/logs/mail.log

[postfix-sasl]
enabled   = true
mode      = auth
port      = smtp,465,submission,imap,imaps,pop3,pop3s
logpath   = /home/nite/pod/docker-mailserver/logs/mail.log
backend   = polling
maxretry  = 5
findtime  = 10m
bantime   = 1h
banaction = ufw

[postfix]
enabled   = true
mode      = more
port      = smtp,465,submission
logpath   = /home/nite/pod/docker-mailserver/logs/mail.log
backend   = polling
maxretry  = 5
findtime  = 10m
bantime   = 1h
banaction = ufw

Use the .local suffix. Fail2Ban reads jail.conf, jail.d/*.conf, jail.local, and jail.d/*.local in order, so this local file overrides the defaults last.

3. Change the sshd banning action to UFW

Do not edit /etc/fail2ban/jail.conf; it is package-provided and may be overwritten during an update.

To make UFW the default action for jails, edit /etc/fail2ban/jail.local and set this in [DEFAULT]:

[DEFAULT]
banaction = ufw
banaction_allports = ufw

To switch only the SSH jail, use this instead:

[sshd]
banaction = ufw

Postfix and postfix-sasl already set banaction = ufw individually in mailserver.local, so they do not depend on the global setting.

Fail2Ban’s built-in UFW action calls ufw to add and remove rules. There is no longer a need to inspect Fail2Ban ban chains with iptables -L; check UFW directly instead.

4. Validate the configuration, then restart the jails

Parse the configuration after editing:

sudo fail2ban-client -d >/dev/null

This command only checks the parsed result; it does not modify the firewall.

After changing an action, restart the service completely:

sudo systemctl restart fail2ban

5. Verify logs, actions, and UFW rules

Confirm that the mail jails are reading the host log file:

sudo fail2ban-client status postfix-sasl
sudo fail2ban-client status postfix

The File list output should be:

/home/nite/pod/docker-mailserver/logs/mail.log

To test whether the SASL filter matches existing log entries without issuing a ban:

sudo fail2ban-regex /home/nite/pod/docker-mailserver/logs/mail.log "postfix[mode=auth]"

After restarting the service, inspect the runtime actions instead of relying only on the configuration files:

sudo systemctl is-active fail2ban
sudo fail2ban-client get sshd actions
sudo fail2ban-client get postfix actions
sudo fail2ban-client get postfix-sasl actions

The actual output in this case was:

active
The jail sshd has the following actions:
ufw
The jail postfix has the following actions:
ufw
The jail postfix-sasl has the following actions:
ufw

6. Run an end-to-end test

sudo fail2ban-client set sshd banip 192.0.2.1
sudo ufw status numbered | grep -F 192.0.2.1
sudo fail2ban-client set sshd unbanip 192.0.2.1

The second command should show a UFW rule with a by Fail2Ban comment. The third command removes the test ban immediately.

References

Hide the Console Window When Auto-Starting WSL2

nite@nite07.com (Nite) — Thu, 09 Jul 2026 02:00:00 +0800

WSL2 automatically stops the instance when all terminal windows are closed. While you can mitigate this by setting vmIdleTimeout=-1 and instanceIdleTimeout=-1 in .wslconfig, some versions still terminate the instance. A reliable approach is to launch a persistent process at login to keep the instance alive, but running wsl.exe directly through Task Scheduler pops up a black console window. This post covers wrapping it in a VBS script to hide that window.

Create the VBS Launch Script

Create a VBS script file on Windows, for example C:\Users\YourUsername\start-wsl.vbs, with the following content:

Set WshShell = CreateObject("WScript.Shell")
WshShell.Run "wsl.exe -d archlinux -u root /bin/bash -c ""sleep infinity""", 0, False

Replace archlinux with your WSL distribution name (run wsl -l in PowerShell to check).

WshShell.Run parameter explanation:

Second parameter 0 — hide the window, no UI shown
Third parameter False — do not wait for the child process to finish, the script exits immediately

Create the Scheduled Task

Run the following commands in PowerShell to create a scheduled task that runs at logon:

$action = New-ScheduledTaskAction -Execute "wscript.exe" -Argument '"C:\Users\YourUsername\start-wsl.vbs"'
$trigger = New-ScheduledTaskTrigger -AtLogOn
$settings = New-ScheduledTaskSettingsSet -AllowStartIfOnBatteries -DontStopIfGoingOnBatteries -ExecutionTimeLimit (New-TimeSpan -Seconds 0)
Register-ScheduledTask -TaskName "WSL2 AutoStart" -Action $action -Trigger $trigger -Settings $settings

Verification

Log off and back on (or reboot), then run wsl -l --running in PowerShell to confirm the WSL instance is running. No console window will appear during the process.

mihomo TUN Mode Breaks Networking: ip_forward, UFW, and auto-redirect

nite@nite07.com (Nite) — Tue, 30 Jun 2026 03:15:00 +1000

When using mihomo in TUN mode (stack: system, auto-route: true), enabling it causes a complete network outage — SSH drops too. The problem isn’t in the mihomo config itself, but in system network forwarding settings that aren’t configured.

Cause

stack: system makes mihomo rely on the kernel networking stack to forward TUN traffic. auto-route: true takes over the default route and funnels all traffic into the TUN interface. But the kernel needs three conditions to forward:

net.ipv4.ip_forward must be 1
The firewall forward chain must not DROP
The kernel must not drop TCP packets returning from the TUN interface due to rp_filter

Arch Linux defaults to ip_forward=0, and UFW defaults to DEFAULT_FORWARD_POLICY="DROP". Both block forwarding. Even after fixing those two, on kernel 6.x ping works but curl times out — TCP handshake packets returning from the TUN interface are dropped by rp_filter=1. The mihomo log shows [TCP] ... using DIRECT, but the connection just times out.

Fix

1. Enable ip_forward

echo 'net.ipv4.ip_forward = 1' | sudo tee /etc/sysctl.d/99-mihomo.conf
sudo sysctl -p /etc/sysctl.d/99-mihomo.conf

2. Change UFW forward policy to ACCEPT

sudo sed -i 's/DEFAULT_FORWARD_POLICY="DROP"/DEFAULT_FORWARD_POLICY="ACCEPT"/' /etc/default/ufw
sudo ufw reload

3. Enable auto-redirect

Add auto-redirect: true to the tun section in config.yaml:

tun:
  enable: true
  stack: system
  auto-route: true
  auto-redirect: true
  auto-detect-interface: true

auto-redirect automatically configures nftables rules to redirect TCP connections, bypassing rp_filter’s filtering of return packets from the TUN interface. Restart mihomo after changing.

Alternative

If you don’t want to change system forwarding settings, switch the TUN stack:

mixed: TCP uses the system stack, UDP uses gvisor — better compatibility than system, better performance than pure gvisor
gvisor: handles TCP/IP entirely in userspace without relying on kernel forwarding — best compatibility but slightly lower performance

Change tun.stack in config.yaml from system to mixed or gvisor. No system configuration changes needed.