[DOC] Fresh install on FreeBSD 14.1: jails cannot access internet via the host

[waseigo]

[MANDATORY] Describe the bug [MANDATORY]
Bastille was running just fine on a FreeBSD 14.0 host. I upgraded the host to 14.1 and all jails lost access to the internet.
To verify that this was not something specific to that one host, I tried again with a fresh install of bastille on a fresh 14.1 install on a VM. Same behavior.

[MANDATORY] Bastille and FreeBSD version (paste bastille -v && freebsd-version -kru output)

0.10.20231125
14.1-RELEASE
14.1-RELEASE
14.1-RELEASE

[MANDATORY] How did you install bastille? (port/pkg/git)

pkg

[optional] Steps to reproduce?

1. Install FreeBSD 14.1 or upgrade existing host to 14.1.
2. Install bastille from pkg/latest.
3. Follow bastille's Getting Started guide, including pf.conf.
4. Create new jail.
5. Attempt to pkg install ....
6. See below. Host resolution fails.

root@gandalf:~ # bastille list
   JID  IP Address      Hostname                      Path
     2  172.16.33.90    minio                         /usr/local/bastille/jails/minio/root
root@gandalf:~ # bastille pkg minio install -y htop minio
[minio]:
Bootstrapping pkg from pkg+https://pkg.FreeBSD.org/FreeBSD:14:amd64/quarterly, please wait...
pkg: Error fetching https://pkg.FreeBSD.org/FreeBSD:14:amd64/quarterly/Latest/pkg.txz: Host does not resolve
Address resolution failed for https://pkg.FreeBSD.org/FreeBSD:14:amd64/quarterly.
Consider changing PACKAGESITE.

The problem persists after a reboot (just to make sure that eveyrthing that's supposed to run also runs at boot.

user@gandalf:~ $ doas bastille pkg 
Usage: bastille pkg [-H|--host] TARGET command [args]
user@gandalf:~ $ doas bastille pkg minio install minio
[minio]:
The package management tool is not yet installed on your system.
Do you want to fetch and install it now? [y/N]: y
Bootstrapping pkg from pkg+https://pkg.FreeBSD.org/FreeBSD:14:amd64/quarterly, please wait...
pkg: Error fetching https://pkg.FreeBSD.org/FreeBSD:14:amd64/quarterly/Latest/pkg.txz: Host does not resolve
Address resolution failed for https://pkg.FreeBSD.org/FreeBSD:14:amd64/quarterly.
Consider changing PACKAGESITE.

Failed to apply on some jails, please check logs

[optional] Expected behavior

Host resolution succeeds.

[optional] Other information

uname -a

root@gandalf:~ # uname -a 
FreeBSD gandalf 14.1-RELEASE FreeBSD 14.1-RELEASE releng/14.1-n267679-10e31f0946d8 GENERIC amd64

/etc/rc.conf

root@gandalf:~ # cat /etc/rc.conf
hostname="gandalf"
ifconfig_vtnet0="DHCP"
local_unbound_enable="YES"
sshd_enable="YES"
ntpd_enable="YES"
ntpd_sync_on_start="YES"
powerd_enable="YES"
moused_nondefault_enable="NO"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
bastille_enable="YES"
cloned_interfaces="lo1"
ifconfig_lo1_name="bastille0"
pf_enable="YES"

ifconfig

root@gandalf:~ # ifconfig
vtnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
	options=4c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,TXCSUM_IPV6>
	ether 52:54:00:07:a5:32
	inet 192.168.88.6 netmask 0xffffff00 broadcast 192.168.88.255
	media: Ethernet autoselect (10Gbase-T <full-duplex>)
	status: active
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	inet 127.0.0.1 netmask 0xff000000
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
	groups: lo
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bastille0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	inet 172.16.33.90 netmask 0xffffff00
	inet6 fe80::1%bastille0 prefixlen 64 scopeid 0x3
	groups: lo
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

/etc/pf.conf

root@gandalf:~ # cat /etc/pf.conf 
ext_if="vtnet0"

set block-policy return
scrub in on $ext_if all fragment reassemble
set skip on lo

table <jails> persist
nat on $ext_if from <jails> to any -> ($ext_if:0)
rdr-anchor "rdr/*"

block in all
pass out quick keep state
antispoof for $ext_if inet
pass in inet proto tcp from any to any port ssh flags S/SA keep state

[tschettervictor]

Any update? Bastille I unusable at this point with a 14.1 jail.

[CrashAndSideburns]

ran into this issue today. i guess that i'll just use a 14.0 jail for now?

edit: sorry, didn't read close enough. the issue is with the host version. in that case, i guess there's nothing i can do beyond reinstalling freebsd on my server?

[tschettervictor]

Is your /etc/resolv.conf populated with your name server and DNS?

I’ve noticed that lately jails that I created somehow don’t have the resolv.conf file populated. I have to manually copy it in from the host.

[CrashAndSideburns]

@tschettervictor yeah, my resolv.conf is fine as far as I can tell

[bmac2]

@waseigo if you install from source instead of from pkg does it behavethe same or different?? THE PKG version is quite far behind the source version. We need to fix that fact, but it is a current fact.

[bmac2]

Also, check inside your jail. What is the output from ifconfig inside the jail? can you ping the jail FROM the host using the ip address? I am struggling to recreate this

[bmac2]

confirmed the issue with a fresh install of 14.1 and following the loopback (bastille0) documenation and the jails do not connect. Same setup without using bastille0 but putting on network directly everythying works.

[yaazkal]

Hi @waseigo between your step 3 and 4 did you started pf?
You need to service pf restart after creating the firewall file configuration.

[CrashAndSideburns]

Hi @waseigo between your step 3 and 4 did you started pf? You need to service pf restart after creating the firewall file configuration.

i have been having the same issue, and i am quite certain that i followed the bastille documentation exactly, including writing the pf configuration and then starting pf. like @bmac2 says, following the setup instructions in the bastille docs on a fresh install of 14.1 leads to jails failing to connect to the network. i ran into this with a fresh install of 14.1 on a vps. i ended up giving up on getting bastille to work, and switched the vps to debian, but i can switch it back over if there are any tests that i can perform that might be helpful

[yaazkal]

@CrashAndSideburns I tested on fresh installs of 13.4, and 14.1 (even on 14.0) and it works correctly using Bastille HEAD. I did not used 0.10... just because the setup subcommand is broken in order to avoid manual process, but it should work since there are no changes in the loopback thing.

My steps after a fresh 14.1-RELEASE:

# pkg install git-lite
# git clone https://github.com/BastilleBSD/bastille.git
# cd /root/bastille
# make install
# cp /usr/local/etc/bastille/bastille.conf.sample /usr/local/etc/bastille/bastille.conf
# cd /root
# bastille setup
# service pf restart
# bastille bootstrap 14.1-RELEASE update
# bastille create alcatraz 14.1-RELEASE 10.17.89.10/24
# bastille pkg alcatraz install vim

And it correctly tries to install whatever I want from the internet.

This is an screenshot running pkg inside the jail:

[CrashAndSideburns]

@yaazkal i just followed those exact steps on a fresh 14.1 install, but running bastille pkg alcatraz install vim caused the same error:

The package management tool is not yet installed on your system.
Do you want to fetch and install it now? [y/N]: y
Bootstrapping pkg from pkg+https://pkg.FreeBSD.org/FreeBSD:14:amd64/quarterly, please wait...
pkg: Error fetching https://pkg.FreeBSD.org/FreeBSD:14:amd64/quarterly/Latest/pkg.txz: Host does not resolve
Address resolution failed for https://pkg.FreeBSD.org/FreeBSD:14:amd64/quarterly.
Consider changing PACKAGESITE.

i really have no idea why this is happening. i followed your steps exactly. the only difference is that i was testing on an amd64 device instead of an aarch64 device, but i would be shocked if that mattered here

[tschettervictor]

Confirmed working on my end also using the HEAD version. 14.0-RELEASE.
Make sure you copy the new bastille.conf.sample in place of your old one.

I tried following the instructions without copying the new file, and it didn't work. But once I copied in the new one, things started working.

[yaazkal]

@CrashAndSideburns @waseigo let me ask, do you have local unbound enabled? Maybe you choose it as an option when installing FreeBSSD?

[waseigo]

Hello everyone, thank you for the responses. It's been such a long time that I ditched FreeBSD and moved back to Proxmox VE, so I don't anymore have the setup I had back when I reported the issue.

[CrashAndSideburns]

@CrashAndSideburns @waseigo let me ask, do you have local unbound enabled? Maybe you choose it as an option when installing FreeBSSD?

oh my god, how embarrassing. that was the issue. i had installed it with local unbound enabled. getting rid of it solved everything

[yaazkal]

oh my god, how embarrassing. that was the issue. i had installed it with local unbound enabled. getting rid of it solved everything

No worries, I'll let this issue open until we document that. Changing labels of the issue.

Regards.