BSD Ports blog

ssg 1.8.76: the same speed, less memory, three workarounds fewer

2026-10-21T23:00:00Z

This site is built by ssg, and since tonight's refresh by version 1.8.76. Was it faster? Not measurably. It used less memory, and it fixed three bugs this site had found and worked around, which mattered more.

Time: within the noise

The server rebuilds all 110,689 pages on every refresh, and the stats page records each build. Four builds on 1.8.74 took 245 to 253 seconds; the first on 1.8.76 took 249.

Builds on the same version already differ by eight seconds, so one build four seconds under the average says nothing yet. On a workstation the same build went from 2 min 21 s to 2 min 10 s, but that is one run each on a machine doing other work. If 1.8.76 is faster, the next days of refreshes on the stats page will show it; one run does not.

Memory: about 5% less

The peak memory of the build moved, on the server and on the workstation alike:

From 6.11–6.20 GB to 5.83 GB on the server, from 5.64 to 5.35 GB on the workstation: about 300 MB less, below the spread of the old runs and in the same direction in both places. For a build that has to fit beside a search engine holding 2.4 GB, that is the useful part.

Three bugs fewer

1.8.75 and 1.8.76 fixed three bugs reported from this site the same day they were found. Each had a workaround here; each workaround is gone:

Bug	What it did here	Workaround, now removed
#342	the build report `build.json` was written readable by its owner only, and the web server answered 403	a `chmod` over every published copy
#344	a taxonomy mapped to `maintainerTerm` also read the raw `maintainer`, so `ports@MidnightBSD..org` stopped the build	`field: maintainerTerm` in the configuration
#346	two pages at one address overwrote each other without a word: the stats page lost its address to FreeBSD's `benchmarks` category	none possible in ssg; the stats page moved to `/stats/`

The last one is worth a sentence more. A page that silently vanishes is found only by someone who goes looking for it, and here someone did: the menu said Benchmarks and the page showed FreeBSD's benchmarking tools. ssg now stops the build when two documents claim one address and names both. The data tool behind this site keeps its own check as well, because it can fail in seconds, before the search engine is loaded and before the four-minute build starts.

What the disks cost the search index

2026-10-21T21:45:00Z

Adding four ports trees meant loading about 70,000 new documents into mddb, the search engine behind the search box, plus one update of every FreeBSD document. It took 30 minutes, about 125 documents a second. mddb's own benchmark speaks of thousands. This is what the difference was.

Half the time was waiting for the disk

The server has two 7200 rpm disks from 2011 in a ZFS mirror, with no separate log device, so every fsync waits for a platter. dtrace counted the fsync calls mddbd made in ten seconds of the load:

758 calls, 6.75 ms each. A request of 500 documents should need about two commits, one for the documents and one for the full-text index, since mddb learned to index a whole batch in one transaction. It made about 300 fsyncs per request instead.

Where the commits beyond those two come from is not settled. The metadata index queue commits once per document, which is the first suspect, but dtrace could not attach to the Go binary to prove it. The measurement is reported to mddb as tradik/mddb#306.

Measured again, the same way everywhere

A first guess from a laptop said "sixteen times slower", and it was wrong: the laptop and the server were not doing the same thing. So the same 5,000 documents went into an empty database, 500 per request, from the same client, in four places:

On an empty database, the disks cost a factor of two, not sixteen. The rest of the gap belongs to size: the full load wrote a 2.5 GB file for 105,124 documents, about 24 KB each, and each commit on a large tree touches more pages than on a small one. The rate fell as the file grew.

The size of a request, which looked like the culprit for a while, is not:

What changed

mddb's database is an index of the ports trees. It can be rebuilt from them at any time, so it does not need the durability fsync buys. It now lives on a ZFS dataset of its own with sync=disabled (and 16 KB records, closer to BoltDB's pages), and only that dataset: the rest of the server is unchanged. Should the server lose power, the worst case is a full reload, which the refresh does by itself.

A refresh sends only what changed: the first one after the move updated four documents in two seconds. The benchmarks page now times ten searches after every refresh, shows how many documents and revisions mddb holds and what the last load sent, so a change like this one shows up in numbers rather than in a feeling that the site is slow.

Five ports trees on one site

2026-10-21T21:00:00Z

Until today this site knew one tree, FreeBSD's. It now reads five: FreeBSD's ports, OpenBSD's ports, NetBSD's pkgsrc, DragonFly BSD's DPorts and MidnightBSD's mports. That is about 105,000 ports and 110,000 pages, rebuilt every 30 minutes.

Each tree, read the way it is published

No two of them publish their tree the same way, so each is read from what its project already ships:

System	Read from	Gives
FreeBSD	the `INDEX`, the ports tree, `pkg.freebsd.org`	versions, options, files, history, package versions per release
OpenBSD	`sqlports`, the SQLite database of the whole tree, shipped as a package	every Makefile variable, dependencies by kind
NetBSD	`pkg_summary` of the binary packages, the pkgsrc tree	versions, licences, run dependencies; maintainers and history from the tree
DragonFly	`packagesite.pkg`, as `pkg` reads it	versions, sizes, run dependencies
MidnightBSD	the mports tree, Makefile by Makefile	everything a Makefile states

Two of these needed more than a parser. OpenBSD signs its packages with signify, which keeps the signature, a hash per 64 KB block, in the gzip header's comment field. For the 46 MB sqlports package that comment runs to tens of kilobytes, and Go's compress/gzip refuses any header field longer than 512 bytes. The header is now skipped by hand and the body inflated as it is. MidnightBSD's package index names packages but not their origins, so mports is read as a tree, the same way an overlay repository is.

The same software, side by side

A port is now identified by its system and its origin, so www/nginx is five ports with five pages: FreeBSD's at /www/nginx/, as before, the others under the system's own path, such as /openbsd/www/nginx/. Each page lists the same software in the other systems:

System	Port	Version
FreeBSD	`www/nginx`	1.30.6_1,3
OpenBSD	`www/nginx`	1.30.6
NetBSD	`www/nginx`	1.30.5
DragonFly	`www/nginx`	1.26.3_3,3
MidnightBSD	`www/nginx`	1.30.1,3

The match is the same origin first, else the one port of the same package name in that system. Where a system has two ports of one name, lua51 and lua54 say, nothing is matched, since either could be the wrong one.

What the data turned up

Reading five trees at once finds what reading one never does:

MidnightBSD's maintainer lines spell its ports address about a dozen ways, among them ports@MidnightBSD..org and ports@MidnightBSD>org. Spellings that make the same page now share one maintainer page.
DPorts copies FreeBSD's maintainer lines, so a FreeBSD maintainer would appear to maintain every port twice. DragonFly's ports show the maintainer but stay off the maintainer pages.
More than a thousand DragonFly packages give their homepage as UNKNOWN. A homepage is now linked only when it is a URL.

Every system has its own tab in the header, its own page of categories and its own install commands on each port page: pkg install, pkg_add, pkgin install, mport install. The search box searches all five, and the benchmarks page shows what a refresh of them costs.