First an Advogato-blog, then greatestjournal, then my own drupal-site, then wordpress, thena blogs.gnome.org-page, then my own homebrew PHP flat file thing, then AnchorCMS...then nothing for a while, and now HTMLy!

If you're not migrating, you're going backwards....still, would be nice to have some stability. And I think some posts have been lost to the ages. Maybe a good thing? Anyway, will try to import some of them here.

You can not trust other people's wifi! Especially hotel, airport, restaurant and conference wifi networks are dangerous places. So, the general advice is to enable a VPN when using those networks. But if you are like me, you will also forget to enable it when you connect.

Requirements: - You are using Network Manager - You already have a VPN configured (and both the VPN and the saved password if any is saved for/shared with all users on the system) - You forget to activate it yourself and want to automatically connect on certain networks

Solution: Add this file to /etc/NetworkManager/dispatcher.d (Debian location, name might be slightly different on other systems):

#!/bin/sh
WHITELIST_NETWORKS="myhomenetwork|workwifi|anotherworkwifithatItrust"
VPN_CONNECTION="nameofyourVPN"
case "$2" in
    up|connectivity-change)
        # First test if we're on a bad network
        if nmcli connection show --active | grep -v -E "${WHITELIST_NETWORKS}" | grep -q wifi
        then
            # Then check if the VPN is already active or not
            if ! nmcli connection show --active | grep "${VPN_CONNECTION}" | grep -q -E "vpn|wireguard"
            then
                        # Then check if the connection is fully up (captive portal check)
                        if [ "${CONNECTIVITY_STATE-FULL}" = "FULL" ] ; then
                                    logger "$0: Starting VPN. called $1 with action \`$2'" 1>&2
                                    sleep 5
                                    nmcli con up "${VPN_CONNECTION}"
                                    sleep 2
                        fi
            fi
        fi
        ;;
    *)
        logger "$0: doing nothing. called $1 with action \`$2'" 1>&2
        exit 0
        ;;
esac

When you connect to wifi, it will automatically connect to your VPN CONNECTION, unless it is connected to a whitelisted wifi. Remember to make it executable. It is a script.

There are many reasons to start hosting an outgoing proxy. Mine was slow upstream servers. But if, for whatever reason, you wish to use an outgoing proxy, this blogpost might be helpful.

The steps are as follows: 1. Install the software (squid on Debian Linux in my case) 1. Configure the software (squid on Debian Linux) 1. Configure your various software to accept the proxy intercepting the TLS secure connections 1. Redirect all traffic through the proxy, with some exceptions if needed. (Websockets are not supported, so you will need to put some exception in for that)

Install

1. Debian doesn't include tls interception (called ssl-bump in squid) support in the default package. If you want this, you need to install squid-openssl.

apt install squid-openssl

Configure

1. Prepare the Certificate Authority (CA)

mkdir -p /etc/squid/cert/
cd /etc/squid/cert/
# This puts the private key and the self-signed certificate in the same file
openssl req -new -newkey rsa:4096 -sha256 -days 3650 -nodes -x509 -keyout myCA.pem -out myCA.pem

1. Create a config file /etc/squid/squid.conf

workers 4
acl localnet dst 0.0.0.1-0.255.255.255  # RFC 1122 "this" network (LAN)
acl localnet dst 10.0.0.0/8     # RFC 1918 local private network (LAN)
acl localnet dst 100.64.0.0/10      # RFC 6598 shared address space (CGN)
acl localnet dst 169.254.0.0/16     # RFC 3927 link-local (directly plugged) machines
acl localnet dst 172.16.0.0/12      # RFC 1918 local private network (LAN)
acl localnet dst 192.168.0.0/16     # RFC 1918 local private network (LAN)
acl localnet dst fc00::/7           # RFC 4193 local private network range
acl localnet dst fe80::/10          # RFC 4291 link-local (directly plugged) machines
acl localnet dst 2a02:a44e:5504::/56
acl tunnelips dst "/etc/squid/tunnelips.list"
acl SSL_ports port 443
acl SSL_ports port 8443
acl SSL_ports port 8843
acl SSL_ports port 5281
acl SSL_ports port 9091
acl Safe_ports port 80      # http
acl Safe_ports port 21      # ftp
acl Safe_ports port 443     # https
acl Safe_ports port 8443    # https
acl Safe_ports port 8843    # https
acl Safe_ports port 70      # gopher
acl Safe_ports port 210     # wais
acl Safe_ports port 1025-65535  # unregistered ports
acl Safe_ports port 280     # http-mgmt
acl Safe_ports port 488     # gss-http
acl Safe_ports port 591     # filemaker
acl Safe_ports port 777     # multiling http
acl Safe_ports port 9091    # transmission
acl CONNECT method CONNECT
acl step1 at_step SslBump1
acl step2 at_step SslBump2
acl step3 at_step SslBump3
acl tunnelsites ssl::server_name_regex -i "/etc/squid/tunnelsites.list"
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost manager
http_access deny manager
include /etc/squid/conf.d/*
http_access allow localnet
http_access allow localhost
http_access allow all
http_access deny all
on_unsupported_protocol tunnel all
http_port 3128 ssl-bump tls-cert=/etc/squid/cert/SQUID_CA.pem generate-host-certificates=on dynamic_cert_mem_cache_size=8MB
http_port 3129 intercept
https_port 3131 intercept ssl-bump tls-cert=/etc/squid/cert/SQUID_CA.pem generate-host-certificates=on dynamic_cert_mem_cache_size=8MB
https_port 443 intercept ssl-bump tls-cert=/etc/squid/cert/SQUID_CA.pem generate-host-certificates=on dynamic_cert_mem_cache_size=8MB
tls_outgoing_options min-version=1.2 cafile=/etc/ssl/certs/ca-certificates.crt
ssl_bump splice tunnelsites
ssl_bump splice step1 localnet
ssl_bump splice step1 tunnelips
ssl_bump peek step1
ssl_bump splice tunnelsites
ssl_bump bump
sslcrtd_program /usr/lib/squid/security_file_certgen -s /var/spool/squid/ssl_db -M 40MB
sslcrtd_children 16 startup=5 idle=3
sslproxy_cert_error deny all
server_idle_pconn_timeout 10 minutes
cache_mem 256 MB
memory_cache_mode always
maximum_object_size_in_memory 5 MB
logformat bumpsquid      %ts.%03tu %6tr %>a %Ss/%03>Hs %ssl::bump_mode %<st %rm %ru %[un %Sh/%<a %mt
access_log daemon:/var/log/squid/access.log bumpsquid
coredump_dir /var/spool/squid
refresh_pattern -i (/cgi-bin/|\?) 0 0%  0
refresh_pattern -i .(gif|png|jpg|jpeg|ico)$ 10080 90% 43200 override-expire ignore-no-cache ignore-no-store ignore-private ignore-auth
refresh_pattern -i gstatic\.com 10080 90% 43200 override-expire ignore-no-cache ignore-no-store ignore-private ignore-auth
refresh_pattern -i .(iso|avi|wav|mp3|mp4|mpeg|swf|flv|x-flv)$ 43200 90% 432000 override-expire ignore-no-cache ignore-no-store ignore-private ignore-auth
refresh_pattern -i .(deb|rpm|exe|zip|tar|tgz|ram|rar|bin|ppt|doc|tiff|InRelease|Packages.gz)$ 10080 90% 43200 override-expire ignore-no-cache ignore-no-store ignore-private ignore-auth
refresh_pattern -i .(css|js)$ 10080 90% 43200 override-expire ignore-no-cache ignore-no-store ignore-private ignore-auth
refresh_pattern -i .index.(html|htm)$ 0 40% 10080 override-expire ignore-no-cache ignore-no-store ignore-private ignore-auth
refresh_pattern -i .(html|htm)$ 1440 40% 40320 override-expire ignore-no-cache ignore-no-store ignore-private ignore-auth
refresh_pattern . 0 40% 40320
shutdown_lifetime 1 seconds
global_internal_static off
dns_v4_first on
forwarded_for delete
pipeline_prefetch on
max_filedesc 204800
fqdncache_size 2048
ipcache_size 2048
ipcache_low 95
ipcache_high 98
cache_swap_low 95
cache_swap_low 98
quick_abort_min 0
quick_abort_max 0
quick_abort_pct 95
range_offset_limit -1
request_header_max_size 200 KB
reply_header_max_size 200 KB
memory_pools off
buffered_logs off
log_icp_queries off
logfile_rotate 1
icp_hit_stale on
query_icmp off
reload_into_ims on
negative_ttl 2 minutes
vary_ignore_expire on
half_closed_clients off
high_page_fault_warning 2
nonhierarchical_direct on
prefer_direct off
cachemgr_passwd none all
client_db on
forwarded_for on
via on
max_stale 1 month

1. Create the certificate database by issuing

/usr/lib/squid/security_file_certgen -c -s /var/spool/squid/ssl_db -M 4MB

and correct access rights with

chmod -R proxy:proxy  /var/spool/squid/ssl_db

1. Restart squid with systemctl restart squid

Configure client software

1. Prepare the CA certificate for use by browsers.

# This can be added to browsers
openssl x509 -in /etc/squid/cert/myCA.pem -outform DER -out /tmp/myCA.der

1. Add the squid CA to the system wide ca-certificates

# Copy the CA to the ca-certificates directory
sudo openssl x509 -in /etc/squid/cert/myCA.pem -out /usr/share/ca-certificates/myCA.crt

1. Add the line myCA.crt to /etc/ca-certificates.conf and run sudo update-ca-certificates
2. For firefox (this is what I use), open the settings, find security devices and add /usr/lib/x86_64-linux-gnu/pkcs11/p11-kit-trust.so as a security device. This means Firefox will also trust the system wide CAs.
3. For other browsers, you can import the /tmp/myCA.der as a Certificate Authority.

Force all traffic through the proxy

1. Install packages for firewall management if not already installed: sudo apt install netfilter-persistent iptables-persistent
2. Configure the firewall rules

sudo iptables -t nat -A OUTPUT -p tcp -m tcp --dport 80 -m owner --uid-owner root -j RETURN
sudo iptables -t nat -A OUTPUT -p tcp -m tcp --dport 80 -m owner --uid-owner proxy -j RETURN
sudo iptables -t nat -A OUTPUT -p tcp -m tcp --dport 443 -m owner --uid-owner root -j RETURN
sudo iptables -t nat -A OUTPUT -p tcp -m tcp --dport 443 -m owner --uid-owner proxy -j RETURN
sudo iptables -t nat -A OUTPUT -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 3129
sudo iptables -t nat -A OUTPUT -p tcp -m tcp --dport 443 -j REDIRECT --to-ports 3131

1. Save the rules: sudo netfilter-persistent save

Have a cup of tea

Because you're done. You might need to relogin/reboot for the p11-kit stuff to take effect.

Finally found the reason why pkexec was broken on my laptop. I had hidepid=2 set for /proc in /etc/fstab. This hides all processes that do not belong to you. Problem is, systemd does not support it.

So, errors are very uninformative, like this:

GDBus.Error:org.freedesktop.PolicyKit1.Error.Failed: No session for cookie
polkit-gnome-1-WARNING **: Unable to determine the session we are in

Solutions for this always point in the direction of session registration (pam_loginuid, or pam_systemd), or missing packages.

But in my case, the real solution is that /proc needs to be world readable. This is (almost) nowhere to be found. So, that's why this post.

Today, after updating wireguard to the latest version on my raspberry, the dkms kernel module failed to build with the following error: cc1: internal compiler error: Segmentation fault

My fix:

install older gcc (gcc 4.9 in my case): apt install gcc-4.9
link gcc to gcc-4.9: ln -sf /usr/bin/gcc-4.9 /usr/bin/gcc

Problem solved. It looks like gcc-8.3 (the one from buster) can't handle wireguard compiling, or wireguard is not gcc-8 compatible...didn't find anything conclusive about that, but my change fixed the problem. Please remember that you did this. Otherwise it might bite you in a year. :-)

When applications fail on linux because they need superuser access, they often don't. But a superuser account would provide these applications with all the access they need. It's a bit like chmod 777... a dangerous fix for a user's problem. Let me explain how to provide ping, wireshark and etherape (that's what triggered this post) access to the network interfaces without using sudo, setuid, chmod or whatever.

The solution? Use the file capabilities system. Install the libcap-ng-utils (or something equivalent, you need the "filecap" tool). After that, just issue the command: sudo filecap <executable> <access rights>. For me, the command to give etherape access to the network interfaces was: sudo filecap /usr/bin/etherape net_admin net_raw

You know, there's blog posts and howto's on the web to install/configure anything and everything including the kitchen sink. The problem is that most of them only tell you the happy flow. As soon as something does not go according to what the writer expects, you're up the proverbial shit creek without a paddle. So, here's two tips to fix issues when faced with vpn / iptables / networking that's not working.

Troubleshooting network packets

sudo tcpdump -n -vvv -i any "put your tcpdump/wireshark filter here"

Some filters include:

• port 53 (for DNS troubleshooting)
• net 10.20.30.0/24 (for dumping traffic from a specific network/host)
• udp port 1194 (for the default openvpn port)

More filters can be found on the wireshark homepage.

Troubleshooting iptables

Ever had the problem where traffic just seemed to disappear? A good way to see where this happens (if you have access) is to add a LOG line to iptables:

sudo iptables -A FORWARD -m limit --limit 5/min -j LOG --log-prefix "This packet dropped: " --log-level 7

That way, any packet going through your linux (OpenVPN) router will immediately pop up if it is not matched by a more important rule than you just added. You can see the kernel log by (for example) using the command:

dmesg -T

A final tip, maybe not as ready-to-use as the previous two, but very important: troubleshoot step by step. Babysteps will quickly isolate the problem.

P.S. Don't forget to enable routing in linux (it's called forwarding) by using the sysctl configuration file.

Red Hat like systems have a feature called "distro-sync", or "distribution-synchronization". This will upgrade/downgrade all packages to the version that is provided by the current sources. Debian, and Ubuntu, and Debian-like systems, don't have this. But you can somewhat emulate this behavior by using the pinning feature of apt.

When you install/upgrade a package, apt evaluates the various versions it has in its database . You can see this with the apt policy command. Try it out for any package, bash for example: apt policy bash. If the priority of a package is high enough, apt will even downgrade. 1001 is usually enough.

We need to change the priority of our preferred version. To do that, we add a file to the /etc/apt/preferences.d/ directory, with the content:

Package: *
Pin: release a=PUTYOURRELEASENAMEHERE
Pin-Priority: 1001

Instead of PUTYOURRELEASENAMEHERE, you should mention your favorite release name. The same one that is also used in your sources list (i.e. oldstable, stable, unstable). After that, is it (mostly) a matter of doing an apt upgrade, and an apt full-upgrade. Mind you, if you're correcting a semi-broken state of your system, you might need to remove some packages and reinstall them later. It might be a good idea to keep track of the packages you remove by writing the important ones down. At least try to keep you graphical environment working, by making sure "gnome" is installed for example. You can do this easily by using tasksel.

Don't forget to remove the pinning file.

Linux used to struggle with managing services because of the simple design of SysV init. It was not really a service configuration system, just a folder structure that decided when processes were started. You had to configure it all yourself. You needed to write a lot of boiler plate in your init scripts to get it going the right way, and mistakes were easy to make and hard to solve.

Not too long ago I ran into this where ulimits were correctly applied when a service started on reboot, but not when the service was restarted by the root user. When you discover why, it's evident, but until then, it feels like the system is just mocking you.

So, when systemd was introduced, it tried (and succeeded) to solve a lot of this by actually managing background services. Nowadays it does much much more, but let's focus on the service management. You can convert any process into your own service in under a minute. Easy for any Linux system, including Raspberry Pi's running Raspbian, or any other Linux flavour.

We start by identifying what needs to start. If it's just a simple process /usr/bin/myprocess that you start from the command line, your systemd config file would look like this:

[Unit]
Description=My fancy description
After=network.target

[Service]
Type=simple (or forking, if the prompt comes back after starting it manually)
ExecStart=/usr/bin/myprocess
Restart=on-failure

[Install]
WantedBy=multi-user.target

You can edit your config file with nano by issuing the command sudo nano /etc/systemd/system/myname.service. When done, just enable the service by issuing the commands: - To reload the systemd configuration: sudo systemctl daemon-reload (or reboot). - To enable the service for startup on reboot: sudo systemctl enable myname - To start the service: sudo systemctl start myname

Slowly moving over some simple services from main power hungry server to raspberry mini-server. Target situation would be that the main server is offline most of the day and is only powered up when I am actually home or in need of hefty services (power up on magicpacket).

Step 1 is done: Basic installation, DHCP, DNS and NTP. Next step would be web and mail services. Not sure how to design that one. Maybe keep mail (zarafa) running on main server, but cache incoming mail on raspberry. Maybe move web interface from apache to nginx if zarafa web interface works with that setup. That is something for next weekend.