I think I'm prepared to tempt Murphy and call my
#FreeBSD vnet jail renovation a huge success. It uses the new FreeBSD-15+ vlan-aware bridge system. It is simpler, MUCH faster to start, and so far: reliable + robust. I am really liking the bridge/vlan overhaul.
Background: Long ago, I overdid my home network vlan setup with 16 vlans. The servers have trunked ports. My jail hosts have vnet jails that exist in one or more of several vlans.
Originally, I'd create vlan interfaces and attach to the trunked ethernet port. Then one bridge per vlan. And the jail epairs would be attached to the appropriate bridge. Management was error prone. The structure was jail epairs <-> vlan100bridge <-> host vlan100 -> ethernet. The host vlan100 had the ethernet as a parent and was added as a member to the bridge.
Later on I switched to SR-IOV VFs by giving each vnet jail its own VF. This was fragile to put it mildly. A parallel jail start was practically guaranteed to cause several VFs to brick and require a restart. Slow sequential starts with lots of sleeps mostly avoided this. It still broke occasionally at runtime when the driver and/or firmware would break. I've given up on this.
Now: I'm using the new vlan-aware bridge system. It now has jail epairs <-> bridge0 <-> ethernet. The ethernet is a member of the bridge. The vlan interfaces for the host have the bridge as a parent device, not a member. Tag encapsulation, access control etc is now a first class function of bridge0. There's one single bridge for the switching domain instead of one bridge per vlan.
I like it!