WRITING

We're buiding our all-in-one server for our business.

The network is complicate when we want to run serveral virtual machine and still want to connect to outside. The router will be virtualized, it run OpenWrt or OPNsense as we want.

We have several choice:

1. Traditional way

if_bridge and tap.

It was not so efficiently.

We test on N4100 CPU:

Client	Server	Speed
vm2 (Debian)	vm0 (ImmortalWrt)	2.11 Gbits/sec
vm2 (Debian)	vm1 (Debian)	2.17 Gbits/sec
vm2 (Debian)	host (FreeBSD)	1.35 Gbits/sec

Our config:

$ cat /etc/rc.conf
cloned_interfaces="bridge0 tap0"
ifconfig_bridge0="dhcp addm em0 addm tap0"
ifconfig_em0="up"

$ cat /boot/loader.conf
if_bridge_load="YES"
if_tap_load="YES"

$ cat /pool/vm/openwrt/openwrt.conf 
loader="uefi"
cpu=2
memory=512M
network0_type="e1000"
network0_switch="public"
network0_device="tap0"
disk0_type="ahci-hd"
disk0_name="immortalwrt-23.05.4-x86-64-generic-ext4-combined-efi.img"

2. More modern way

vale and vether.

Of course there is epair that we can connect between host and vm.

We test on N4100 CPU:

Client	Server	Speed
vm2 (Debian)	vm0 (ImmortalWrt)	3.50 Gbits/sec
vm2 (Debian)	vm1 (Debian)	5.30 Gbits/sec
vm2 (Debian)	host (FreeBSD)	1.48 Gbits/sec

Our config:

$ cat /etc/rc.conf
defaultrouter="192.168.1.1"
cloned_interfaces="vether0"
ifconfig_vether0="192.168.1.2/24 up"

$ cat /boot/loader.conf
if_vether_load="YES" 

$ cat /pool/vm/openwrt/openwrt.conf 
loader="uefi"
cpu=2
memory=512M
network0_type="e1000"
network0_switch="public"
disk0_type="ahci-hd"
disk0_name="immortalwrt-23.05.4-x86-64-generic-ext4-combined-efi.img"

Need to run following script everytime after openwrt vm start (connect host to vm). If you have vm-bhyve you can put it to prestart, remember to chmod +x for it.

valectl -h vale-name:vether-name, for example valectl -h vale0:vether0.

3. The most complex thing

netgraph.

We test on N4100 CPU:

Client	Server	Speed
vm2 (Debian)	vm0 (ImmortalWrt)	2.41 Gbits/sec
vm2 (Debian)	vm1 (Debian)	2.47 Gbits/sec
vm2 (Debian)	host (FreeBSD)	2.27 Gbits/sec

ng_bridge is much faster than vether in this case: vm to host.

We think we could keep VALE as our vm switch, and replace vether by netgraph or epair?

netgraph as host virtual interface and vale as bridge, report bad pkt??? something was not right. we will look into it further and do it manually.

Client	Server	Speed
vm2 (Debian)	vm0 (ImmortalWrt)	3.49 Gbits/sec
vm2 (Debian)	vm1 (Debian)	5.25 Gbits/sec
vm2 (Debian)	host (FreeBSD)	1.46 Gbits/sec

Ref:

• rc.d/ngbridge
• Netgraph Bridge

$ cat /boot/loader.conf
ng_eiface_load="YES"
ng_bridge_load="YES"
ng_ether_load="YES"

$ cat /etc/rc.conf
ngbridge_enable="YES"
ngbridge_names="lan"
ngbridge_lan_eifaces="nge_1u"
ngbridge_nge_1u_mac="00:37:92:01:02:02"
ngbridge_nge_1u_addr_num="1"
ngbridge_nge_1u_addr_1="inet 192.168.1.2/24"
ngbridge_lan_eifaces_keep="nge_1u"
ngbridge_lan_route_num=1
ngbridge_lan_route_1="-net default 192.168.1.1"
ngbridge_lan_vlans="NO"

$ cat /pool/vm/openwrt/openwrt.conf 
loader="uefi"
cpu=2
memory=512M
network0_type="e1000"
network0_switch="lanbridge"
disk0_type="ahci-hd"
disk0_name="immortalwrt-23.05.4-x86-64-generic-ext4-combined-efi.img"

BÀI ĐANG VIẾT

Đây là những ghi chép trong quá trình triển khai nhúng live streaming camera video lên website trangtraihuounai.com.

1. Yêu cầu

Không quá nhiều điểm đặc biệt:

• Nhúng livestream camera của trang trại lên website
• Hệ thống đơn giản, có thể dễ dàng mở rộng khi cần thiết

2. Ý tưởng

Gọn nhẹ nhất có thể:

• Sử dụng máy tính all-in-one cho tất cả các tác vụ
• Lấy luồng RTSP từ camera phát lại bằng go2rtc
• Nhúng trực tiếp vào website hoặc phát kênh Youtube?

3. Thiết bị thử nghiệm

Đang có sẵn các thứ sau:

• Mini PC N4100: 4x1GB Ethernet
• PoE Switch không biết tên
• Imou Cruiser 2 (5MP Outdoor Wi-Fi P&T Camera)

Thay thế thiết bị khác sau khi chạy thử:

• All-in-one box: HP / Dell
• Camera: Dahua / Hikvision

4. Sơ đồ hệ thống

Sẽ tách router trong tương lai, hiện tại chạy ImmortalWrt trên môi trường ảo hóa.

5. Các bước triển khai

Task List

• [x] Cài đặt FreeBSD box
• [x] Cấu hình hệ thống mạng sẵn sàng cho máy ảo
• [x] Cài đặt máy ảo chạy ImmortalWrt làm router
• [x] Cấu hình ImmortalWrt, quy hoạch VLAN
• [ ] Cài đặt máy ảo chạy Debian làm docker host
• [ ] Cài đặt nginx làm reverse proxy, nat port 80/443 từ router
• [ ] Cài đặt go2rtc / restreamer lấy luồng từ camera
• [ ] Cài đặt nginx làm web server, nhúng stream vào trang web
• [ ] Chuyển thiết bị đến trang trại, quay pppoe mở port 80/443
• [ ] Chạy thử trong 30 ngày

6. Các vấn đề gặp phải

• high latency from cameras to go2rtc / restreamer: check bridge+tap on bhyve, maybe passthru all nics and use something like vether / epair / vale / netgraph..
• transcoding video requires GPU, check bhyve gpu passthru again

TO WRITE

PROOF OF CONCEPT

1. Requirement and concept

We're building a PBX system for our business. It will handle:

• automatic distribute incoming calls
• call outside by specific phone numbers, automatic route to reduce call cost
• sms send api for OTP maybe?

Since we don't have any SIP trunk from our carriers, we will take incoming calls from regular sim cards.

• Disadvantage:
  • Single-line: only one concurrent call per SIM card at a time
  • Need more work to handle system failure due to OS, power..
• Not sure:
  • Voice over LTE (VoLTE) are supported by all carriers.
  • Voice over Wi‑Fi (VoWiFi) are supported by VinaPhone, Mobifone?

Call flow as bellow:

Our stack:

• Hardwares:
  • Mini PC / Router: ZOTAC ZBOX CI323 nano
  • 2G/3G/4G/5G module: Quectel EC20CEFRG-MINIPCIE-C (Vietnam frequency bands: B1/B3/B8), PCIE to USB adapter (sim card slot included), antena
  • VoIP phone: haven't bought yet
• Softwares
  • OS: TrueNAS SCALE 24.10 - Electric Eel
  • VM: jailmaker
  • PBX: Asterisk
  • Softphone: Linphone (Linux / iOS), Baresip (Android), MicroSIP (Windows)

2. Step by step

2.1 Asterisk Installation

• OS: Debian 12 ("Bookworm")

$ mkdir tmp
$ cd tmp/
$ wget https://downloads.asterisk.org/pub/telephony/asterisk/asterisk-22-current.tar.gz
$ tar zxvf asterisk-22-current.tar.gz
$ cd asterisk-22*/
$ sudo ./contrib/scripts/install_prereq install
$ ./configure
$ make menuselect

Most of IP phones are support G722 and / or G729 codecs now.

• Under Add-ons:
  • Select [*] format_mp3
• Under Codec Translators:
  • Select [*] codec_opus
  • Select [*] codec_silk
  • Select [*] codec_siren7
  • Select [*] codec_siren14
  • Select [*] codec_g729a
• Under Core Sound Packages:
  • Deselect [*] CORE-SOUNDS-EN-GSM
  • Select [*] CORE-SOUNDS-EN-WAV
  • Select [*] CORE-SOUNDS-EN-G722
  • Select [*] CORE-SOUNDS-EN-G729
• Under Extras Sound Packages:
  • Select [*] EXTRA-SOUNDS-EN-WAV
  • Select [*] EXTRA-SOUNDS-EN-G722
  • Select [*] EXTRA-SOUNDS-EN-G729

$ make
$ sudo ./contrib/scripts/get_mp3_source.sh
$ sudo make install
$ sudo make config
$ sudo make samples
$ sudo mkdir /etc/asterisk/samples
$ sudo mv /etc/asterisk/*.*  /etc/asterisk/samples/
$ sudo asterisk -rvvvv

Most important config files:

• extensions.conf
• pjsip.conf

2.2. Virtual machine vm1

Purpose: VoIP GSM Gateway to handle voice call from and or to 3G/4G/5G MNOs.

• OS: Debian 12
• IP: 192.168.1.11
• Software: Asterisk
• Driver: sterisk-chan-quectel

2.3. Virtual machine vm0

Purpose: Local PBX.

• OS: Debian 12
• IP: 192.168.1.10
• Software: Asterisk

3. Resource

Softwares:

• Servers:
  • Asterisk
    • asterisk-chan-quectel
    • asterisk-chan-dongle
  • FreeSWITCH
  • Kamailio
  • Yate
  • Routr
  • FreePBX
  • MikoPBX
  • docker-asterisk
• Clients:
  • Baresip
  • Linphone
  • Jami
  • MicroSIP
• Docs:
  • Official Asterisk documentation
  • Official Asterisk YouTube Channel
  • Asterisk: The Definitive Guide
• Ref:
  • 使用EC20模块配合asterisk及freepbx实现短信转发和网络电话
  • 如何把SIM卡从手机中取出
  • Quectel LTE 모뎀으로 Asterisk에서 KT, SKT, LG U+ VoLTE 전화 사용하기
  • Thực trạng mạng di động viễn thông tại Việt Nam

We've been running mail server for a quite of time. The last software was Stalwart Mail Server, unfortunately it's under heavy development. They move so fast then the maintain job is a pain.

We decided to going back to a more mature solution.

1. Should know

Some information that we think you should know when you want to run your own mail server.

1.1. Protocols:

• Simple Mail Transfer Protocol (SMTP)
• Internet Message Access Protocol (IMAP)
• Local Mail Transfer Protocol (LMTP)

1.2. Formats:

• Maildir
• Mbox

1.3. Software components:

• Message transfer agent (MTA): Sendmail, Exim, Postfix
• Message delivery agent (MDA): Dovecot, Cyrus IMAP
• Email filtering: Rspamd, SpamAssassin, ClamAV
• Webmail: Roundcube, Cypht, SnappyMail

1.4. Resources:

• awesome-emails
• awesome-opensource-email

2. Software stack

We're running FreeBSD so we want to run our mail server on it. The solution need to be easy to deploy and maintain.

• FreeBSD (operating system)
• OpenSMTPD (SMTP)
• Dovecot (IMAP)
• Rspamd (spam filtering)

For small businesses we will not store usernames and passwords in LDAP or SQL databases, we store such information in flat-file databases.

2.1. FreeBSD (operating system)

There is nothing to say. FreeBSD is quite boring, it just works ^^.

We're going to run our mail server in a FreeBSD jail (managed by AppJail). We already have HAProxy as our Load Balancer.

We just need to make sure our FreeBSD server is up to date.

# pkg update
# pkg upgrade

Create vmail user and vmail group. This is the user/group that’s used to access the mails.

# pw useradd vmail -u 5000 -d /home/vmail -s /usr/sbin/nologin -m

Get a free TLS/SSL certificate for your domain from a certificate authority (ZeroSSL, Let's Encrypt..) by acme.sh or Certbot.

2.2. OpenSMTPD (SMTP)

We consider Postfix which is more popular but we found that OpenSMTPD is easier to config so we will choose it as our MTA.

We love the config syntax, it remind us about PF.

2.2.1. Install OpenSMTPD

# pkg install opensmtpd opensmtpd-extras

2.2.2. Config OpenSMTPD

2.2.2.1. smtpd.conf

Please read:

• smtpd.conf — SMTP daemon configuration file

Please read it carefully and make your own config file. It's very important.

Modify your file /usr/local/etc/mail/smtpd.conf:

2.2.2.1.1. TLS/SSL certificate

declare your certificate as follow, pkiname named mail.example.com:

pki mail.example.com cert "/usr/local/etc/certs/example.com/fullchain.pem"
pki mail.example.com key "/usr/local/etc/certs/example.com/key.pem

2.2.2.1.2. Tables

declare your tables

table aliases file:/usr/local/etc/mail/aliases
table virtuals file:/usr/local/etc/mail/virtuals
table domains file:/usr/local/etc/mail/domains
table credentials file:/usr/local/etc/mail/credentials
table secrets file:/usr/local/etc/mail/secrets
table passwds file:/usr/local/etc/mail/passwds

We will have: aliases, virtuals, domains, credentials, secrets and passwds.

See OpenSMTPD tables below for more information.

2.2.2.1.3. Bind

# STARTTLS port 25
listen on 0.0.0.0 port 25 tls pki mail.example.com
# SMTPS port 465
listen on 0.0.0.0 port 465 smtps pki mail.example.com auth <credentials>
# SUBMISSION port 587
listen on 0.0.0.0 port 587 tls-require pki mail.example.com auth <credentials>

Listen to IPv4 only 0.0.0.0.

For SMTPS (port 465), SUBMISSION...

WORK IN PROGRESS

Protocols:

• Simple Mail Transfer Protocol (SMTP)
• Internet Message Access Protocol (IMAP)
• Local Mail Transfer Protocol (LMTP)

Formats:

• Maildir
• Mbox

Software components:

• Message transfer agent (MTA): Sendmail, Exim, Postfix
• Message delivery agent (MDA): Dovecot, Cyrus IMAP
• Email filtering: Rspamd, SpamAssassin, ClamAV
• Webmail: Roundcube, Cypht, SnappyMail

Resources:

• awesome-emails
• awesome-opensource-email

My software stack:

• FreeBSD (operating system)
• OpenSMTPD (SMTP)
• Dovecot (IMAP)
• Rspamd (spam filtering)

1. FreeBSD (operating system)

2. OpenSMTPD (SMTP)

3. Dovecot (IMAP)

4. Rspamd (spam filtering)

The following command will show all snapshots of [pool] (<-replace this with your pool name) that have holds

zfs list -H -o name -t snapshot -r pool | xargs -n1 zfs holds -H

the properties will be listed as property:stuff

with that information we can free the snapshots..

zfs list -H -o name -t snapshot -r pool | xargs -n1 zfs holds -H | awk '{print $1}' | xargs -n1 zfs release property:stuff

(replace 'property:stuff' with whatever is holding your dataset)

..and finally delete them

zfs destroy -r [pool]/[dataset][@snapshot]

source: How to check that all ZFS snapshots within a pool are without holds before destroying that pool