搭建私有镜像站使用tunasync
2025-10-30 01:06:27零 摘要
本文主要介绍私有镜像站搭建，主要分两步一是下载同步公网镜像，而是提供web 服务，主要使用tunasync 实现下载，使用nginx 或者mirror-web 提供web 服务
一 搭建同步工具tunasync
本文主要介绍以非root 用户搭建tunaasync
1.1 下载工具
https://github.com/tuna/tunasync
https://gitee.com/weiwei20180921/tunasync
1.1 基础配置
1.1.1 新建用户及工作目录 root 用户执行
[root@2023001 ~]# groupadd -g 2001 mirrorgroup
[root@2023001 ~]# useradd -u 2101 -g mirrorgroup mirrors
[root@2023001 ~]# passwd mirrors
1.1.2 建立应用目录 （mirrors 执行）
主要存放应用配置文件及执行文件
[root@2023001 ~]# su - mirrors
[mirrors@2023001 ~]$ pwd
/home/mirrors
[mirrors@2023001 ~]$ mkdir -p /home/mirrors/tunasync/conf
[mirrors@2023001 ~]$ mkdir -p /home/mirrors/tunasync/db
[mirrors@2023001 ~]$ mkdir -p /home/mirrors/tunasync/bin
[mirrors@2023001 ~]$ tree
.
├── tunasync
│ ├── bin
│ ├── conf
│ └── db
3 directories, 0 files
1.1.3 建立数据目录 （root 用户）
[root@2023001 data]# mkdir -p /data/mirrors
[root@2023001 data]# chown -R mirrors:mirrorgroup /data/mirrors/
1.2 部署及配置tunasync（mirrors 用户）
主要使用mirrors 用户部署
1.2.1 下载并上传到服务器
下载地址：
https://github.com/tuna/tunasync
或者
https://gitee.com/weiwei20180921/tunasync
1.2.2 部署
将tunasync-linux-amd64-bin.tar.gz 解压到/home/mirrors/tunasync/bin 目录
[mirrors@2023001 bin]$ pwd
/home/mirrors/tunasync/bin
[mirrors@2023001 bin]$ tar -xf /home/mirrors/tunasync-linux-amd64-bin.tar.gz
[mirrors@2023001 bin]$ ls
tunasync tunasynctl
[mirrors@2023001 bin]$
1.2.3 配置环境变量
先备份，然后修改，最后source
[mirrors@2023001 ~]$ cp .bash_profile .bash_profile.bak.orig
[mirrors@2023001 ~]$ vim .bash_profile
[mirrors@2023001 ~]$ source .bash_profile
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
TUNAPATH=/home/mirrors/tunasync
PATH=$PATH:$HOME/.local/bin:$HOME/bin:$TUNAPATH/bin
export PATH
~
验证
[mirrors@2023001 ~]$ tunasync -v
Version: 0.8.0
Git Hash: c07aaffe65da65217665d7c5df46ea2d857fdfa9
Build Date: 2021-09-02 22:24:51 +0800 CST
[mirrors@2023001 ~]$
1.2.4 配置tunasync
tunasync 的所有配置存放在/home/mirrors/tunasync/conf
1.2.4.1 manager 配置
创建/home/mirrors/tunasync/conf/manager.conf
debug = false
[server]
addr = "127.0.0.1"
port = 14242
ssl_cert = ""
ssl_key = ""
[files]
db_type = "bolt"
db_file = "/home/mirrors/tunasync/db/manager.db"
ca_cert = ""
配置说明
port：监听端口，默认是14242，
ssl_cert与ssl_key：ssl 配置，不需要 ssl 的同学不用配置
db_file：数据库文件，目前统一放在 /home/mirrors/tunasync/db/ 目录
端口14242 说明
[mirrors@2023001 conf]$ tunasynctl list --all
Failed to correctly get information of all jobs from manager server: Get http://localhost:14242/jobs: dial tcp [::1]:14242: connect: connection refused
[mirrors@2023001 conf]$ ll
若修改14242，则需修改tunasynctl 配置文件
一般在/etc/tunasync/ctl.conf 或者 ~/.config/tunasync/ctl.conf 两个位置
1.2.4.2 worker 配置
根据需要创建需同步的镜像配置文件。以 CentOS 镜像为例，同步 CentOS 需要创建 /home/mirrors/tunasync/conf/worker.conf，文件内容如下：
[global]
name = "myworker"
concurrent = 10
interval = 1440
[manager]
api_base = "http://localhost:14242"
token = "some_token"
ca_cert = ""
[cgroup]
enable = false
base_path = "/sys/fs/cgroup"
group = "tunasync"
[server]
hostname = "localhost"
listen_addr = "127.0.0.1"
listen_port = 16010
ssl_cert = ""
ssl_key = ""
[[mirrors]]
name = "7.5.1804"
log_dir = "/data/mirrors/log/tunasync/centos-vault/{{.Name}}"
mirror_dir = "/data/mirrors/centos-vault/7.5.1804"
provider = "rsync"
upstream = "rsync://mirrors.tuna.tsinghua.edu.cn/centos-vault/7.5.1804/"
use_ipv6 = false
[[mirrors]]
name = "7.6.1810"
log_dir = "/data/mirrors/log/tunasync/centos-vault/{{.Name}}"
mirror_dir = "/data/mirrors/centos-vault/7.6.1810"
provider = "rsync"
upstream = "rsync://mirrors.tuna.tsinghua.edu.cn/centos-vault/7.6.1810/"
use_ipv6 = false
[[mirrors]]
name = "7.9.2009"
log_dir = "/data/mirrors/log/tunasync/centos-vault/{{.Name}}"
mirror_dir = "/data/mirrors/centos-vault/7.9.2009"
provider = "rsync"
upstream = "rsync://mirrors.tuna.tsinghua.edu.cn/centos-vault/7.9.2009/"
use_ipv6 = false
[[mirrors]]
name = "epel"
log_dir = "/data/mirrors/log/tunasync/{{.Name}}"
mirror_dir = "/data/mirrors/epel"
provider = "rsync"
upstream = "rsync://mirrors.tuna.tsinghua.edu.cn/epel/"
use_ipv6 = false
mirror_dir = "/data/mirrors/centos-vault/7.9.2009
这个目录要手工创建mkdir
配置说明
global name：worker 进程名称，用于程序识别
log_dir：tunasync 的日志路径
mirror_dir：镜像下载地址
concurrent：并发线程数
interval：rsync 同步周期，以分钟为单位
api_base：manager 地址，注意，此处需与 manager 所在服务器配置保持一致
server listen_port：该 worker 自身的监听端口，如果同一台服务器上有多个 worker，各个 worker 的配置文件中务必配置不同端口
mirrors name：镜像名称，tunasync 会在镜像根目录下建立一个该名称的目录用于下载镜像
upstream：同步地址，注意，参数最后需要有“/”，否则启动 tunasync 时会报错
1.3 启动（mirrors 用户）
启动 tunasync 需开启 manager 进程与 worker 进程，先启动 manager，后启动 worker。为了便于监控系统进程情况，建立 /data/mirrors/log/plog 目录，所有进程的工作日志在该目录中（注意，此处日志为系统终端输出日志，与 tunasync 自身工作日志不同）。同样的，以下命令由 mirrors 用户操作。
1.3.1 建立日志目录
[mirrors@2023001 mirrors]$ mkdir -p /data/mirrors/log/plog
1.3.2 开启manager服务（后台进程）：
[mirrors@2023001 mirrors]$ tunasync manager --config /home/mirrors/tunasync/conf/manager.conf >> /data/mirrors/log/plog/manager.log &
[2] 2611
[mirrors@2023001 mirrors]$ netstat -anp | grep tunasy
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 127.0.0.1:28288 0.0.0.0:* LISTEN 2611/tunasync
[mirrors@2023001 mirrors]$
1.3.2 开启worker服务（根据需要同步的镜像开启，这里以 CentOS7.9 为例）
[mirrors@2023001 mirrors]$ tunasync worker --config /home/mirrors/tunasync/conf/worker-centos.conf >> /data/mirrors/log/plog/worker-centos.log &
[3] 3233
[mirrors@2023001 mirrors]$ netstat -anp | grep tunasy
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 127.0.0.1:16010 0.0.0.0:* LISTEN 3233/tunasync
tcp 0 0 127.0.0.1:28288 0.0.0.0:* LISTEN 2611/tunasync
[mirrors@2023001 mirrors]$
1.4 启停脚本（mirrors 用户）
我这边是非root 用户部署和使用tunarsync ，所以启停脚本需要修改TUNASYNC_BIN，OPTIONS 这两个参数
1.4.1 下载并增加执行权限
脚本下载放到/home/mirrors/tunasync/bin
https://raw.githubusercontent.com/whsir/tunasync-bin/master/tunasync-worker
https://raw.githubusercontent.com/whsir/tunasync-bin/master/tunasync-manager
[mirrors@2023001 bin]$ pwd
/home/mirrors/tunasync/bin
[mirrors@2023001 bin]$ chmod u+x tunasync-manager.sh
[mirrors@2023001 bin]$ chmod u+x tunasync-worker.sh
[mirrors@2023001 bin]$ ls -al
total 42232
drwxr-xr-x 2 mirrors mirrorgroup 93 Jan 25 10:37 .
drwxr-xr-x 5 mirrors mirrorgroup 39 Jan 24 20:19 ..
-rwxr-xr-x 1 mirrors mirrorgroup 32456456 Sep 2 2021 tunasync
-rwxr--r-- 1 mirrors mirrorgroup 1487 Jan 25 10:32 tunasync-manager.sh
-rwxr-xr-x 1 mirrors mirrorgroup 10779559 Sep 2 2021 tunasynctl
-rwxr--r-- 1 mirrors mirrorgroup 1476 Jan 25 10:31 tunasync-worker.sh
[mirrors@2023001 bin]$
1.4.2 tunasync-manager.sh 修改
修改点
TUNASYNC_BIN=/home/mirrors/tunasync/bin/$NAME
OPTIONS="manager --config /home/mirrors/tunasync/conf/manager.conf"
LOGFILE=/data/mirrors/log/plog/manager.log
$TUNASYNC_BIN $OPTIONS >> $LOGFILE &
全量
#!/bin/sh
# Provides: tunasync-manager
# Description: Mirror job management tool.
# Author: whsir
NAME=tunasync
TUNASYNC_NAME=tunasync-manager
TUNASYNC_BIN=/home/mirrors/tunasync/bin/$NAME
OPTIONS="manager --config /home/mirrors/tunasync/conf/manager.conf"
LOGFILE=/data/mirrors/log/plog/manager.log
StatBin=/bin/ps
case "$1" in
start)
if $StatBin -ef | grep -v grep | grep -q "tunasync manager";then
TUNASYNC_PID=`ps -ef | grep -v grep | grep "tunasync manager" | awk -F" " '{print $2}'`
echo "$TUNASYNC_NAME (pid $TUNASYNC_PID) already running."
else
$TUNASYNC_BIN $OPTIONS >> $LOGFILE &
echo "Starting $TUNASYNC_NAME... "
fi
;;
status)
if $StatBin -ef | grep -v grep | grep -q "tunasync manager";then
TUNASYNC_PID=`ps -ef | grep -v grep | grep "tunasync manager" | awk -F" " '{print $2}'`
echo "$TUNASYNC_NAME (pid $TUNASYNC_PID) is running."
else
echo "$TUNASYNC_NAME is not running."
fi
;;
stop)
if $StatBin -ef | grep -v grep | grep -q "tunasync manager";then
TUNASYNC_PID=`ps -ef | grep -v grep | grep "tunasync manager" | awk -F" " '{print $2}'`
kill $TUNASYNC_PID
echo "Stop $TUNASYNC_NAME...."
else
echo "$TUNASYNC_NAME already stopping."
fi
;;
restart)
$0 stop
$0 start
;;
*)
echo "Usage: $0 {start|stop|restart|status}"
exit 1
;;
esac
1.4.3 tunasync-worker.sh 修改
修改点：
TUNASYNC_BIN=/home/mirrors/tunasync/bin/$NAME
OPTIONS="worker --config /home/mirrors/tunasync/conf/worker.conf"
LOGFILE=/data/mirrors/log/plog/worker.log
$TUNASYNC_BIN $OPTIONS >> $LOGFILE &
全量
#!/bin/sh
# Provides: tunasync-worker
# Description: Mirror job management tool.
# Author: whsir
NAME=tunasync
TUNASYNC_NAME=tunasync-worker
TUNASYNC_BIN=/home/mirrors/tunasync/bin/$NAME
OPTIONS="worker --config /home/mirrors/tunasync/conf/worker.conf"
StatBin=/bin/ps
LOGFILE=/data/mirrors/log/plog/worker.log
case "$1" in
start)
if $StatBin -ef | grep -v grep | grep -q "tunasync worker";then
TUNASYNC_PID=`ps -ef | grep -v grep | grep "tunasync worker" | awk -F" " '{print $2}'`
echo "$TUNASYNC_NAME (pid $TUNASYNC_PID) already running."
else
$TUNASYNC_BIN $OPTIONS >> $LOGFILE &
echo "Starting $TUNASYNC_NAME... "
fi
;;
status)
if $StatBin -ef | grep -v grep | grep -q "tunasync worker";then
TUNASYNC_PID=`ps -ef | grep -v grep | grep "tunasync worker" | awk -F" " '{print $2}'`
echo "$TUNASYNC_NAME (pid $TUNASYNC_PID) is running."
else
echo "$TUNASYNC_NAME is not running."
fi
;;
stop)
if $StatBin -ef | grep -v grep | grep -q "tunasync worker";then
TUNASYNC_PID=`ps -ef | grep -v grep | grep "tunasync worker" | awk -F" " '{print $2}'`
kill $TUNASYNC_PID
echo "Stop $TUNASYNC_NAME...."
else
echo "$TUNASYNC_NAME already stopping."
fi
;;
restart)
$0 stop
$0 start
;;
*)
echo "Usage: $0 {start|stop|restart|status}"
exit 1
;;
esac
1.4.4 验证
[mirrors@2023001 ~]$ tunasync-manager.sh stop
Stop tunasync-manager....
[mirrors@2023001 ~]$ tunasync-manager.sh restart
tunasync-manager already stopping.
Starting tunasync-manager...
[mirrors@2023001 ~]$ tunasync-manager.sh status
tunasync-manager (pid 22809) is running.
[mirrors@2023001 ~]$
[mirrors@2023001 ~]$ tunasync-worker.sh status
tunasync-worker (pid 5877) is running.
[mirrors@2023001 ~]$ tunasync-worker.sh restart
Stop tunasync-worker....
tunasync-worker (pid 5877) already running.
[mirrors@2023001 ~]$
1.5 运维 tunasynctl 使用
主要介绍删除job,
使用参考
https://github.com/tuna/tunasync/blob/master/docs/zh_CN/tips.md
https://gitee.com/weiwei20180921/tunasync/blob/master/docs/zh_CN/tips.md
1.5.1 删除job
$ tunasynctl disable -w
$ tunasynctl flush
[mirrors@2023001 log]$ tunasynctl disable -w centos_worker79 centos79
Successfully send the command
[mirrors@2023001 log]$ tunasynctl flush
Successfully flushed disabled jobs
[mirrors@2023001 log]$ tunasynctl list --all
[
{
"name": "7.9.2009",
"is_master": true,
"status": "syncing",
"last_update": "0001-01-01 00:00:00 +0000",
"last_update_ts": -62135596800,
"last_started": "2024-01-25 11:06:56 +0800",
"last_started_ts": 1706152016,
"last_ended": "2024-01-25 10:56:53 +0800",
"last_ended_ts": 1706151413,
"next_schedule": "0001-01-01 00:00:00 +0000",
"next_schedule_ts": -62135596800,
"upstream": "rsync://mirrors.tuna.tsinghua.edu.cn/centos-vault/7.9.2009/",
"size": "unknown"
},
{
"name": "epel",
"is_master": true,
"status": "syncing",
"last_update": "0001-01-01 00:00:00 +0000",
"last_update_ts": -62135596800,
"last_started": "2024-01-25 14:35:53 +0800",
"last_started_ts": 1706164553,
"last_ended": "0001-01-01 00:00:00 +0000",
"last_ended_ts": -62135596800,
"next_schedule": "0001-01-01 00:00:00 +0000",
"next_schedule_ts": -62135596800,
"upstream": "rsync://mirrors.tuna.tsinghua.edu.cn/epel/",
"size": "unknown"
}
]
[mirrors@2023001 log]$
1.5.2 删除worker
$ tunasynctl rm-worker -w
[mirrors@2023001 bin]$ tunasynctl rm-worker -w centos_worker79
Successfully removed the worker
[mirrors@2023001 bin]$ tunasynctl workers
null
[mirrors@2023001 bin]$
1.5.3 热重载 worker.conf
当修改了worker.conf 可使用该命令，下面介绍增加worker.conf 配置，删除worker.conf 配置操作 见官网示例
$ tunasynctl reload -w
示例：新增欧拉系统同步
首先 修改worker.conf 配置
第二 补充对应数据目录
第三执行热重载命令
第四验证
添加同步欧拉系统配置
[mirrors@2023001 conf]$ vi worker.conf
[[mirrors]]
name = "openEuler-22.03-LTS-SP3"
log_dir = "/mirrordata/mirrors/log/tunasync/{{.Name}}"
mirror_dir = "/mirrordata/mirrors/openeuler/openEuler-22.03-LTS-SP3"
provider = "rsync"
upstream = "rsync://mirrors.tuna.tsinghua.edu.cn/openeuler/openEuler-22.03-LTS-SP3/"
use_ipv6 = false
第二 补充对应数据目录
[mirrors@2023001 conf]$ cd /mirrordata/mirrors/
[mirrors@2023001 mirrors]$ mkdir -p ./openeuler/openEuler-22.03-LTS-SP3
第三执行热重载命令
tunasynctl reload -w myworker
验证
tunasynctl list --all
{
"name": "openEuler-22.03-LTS-SP3",
"is_master": true,
"status": "syncing",
"last_update": "0001-01-01 00:00:00 +0000",
"last_update_ts": -62135596800,
"last_started": "2024-02-02 10:02:21 +0800",
"last_started_ts": 1706839341,
"last_ended": "0001-01-01 00:00:00 +0000",
"last_ended_ts": -62135596800,
"next_schedule": "0001-01-01 00:00:00 +0000",
"next_schedule_ts": -62135596800,
"upstream": "rsync://mirrors.tuna.tsinghua.edu.cn/openeuler/openEuler-22.03-LTS-SP3/",
"size": "unknown"
}
二 搭建mirror-web
原相关代码在GitHub上，先导入gitee,然后再gitee 使用gitee
2.1 将github 代码迁移到gitee
https://github.com/tuna/mirror-web.git
https://github.com/tuna/mirrorz-help-ng.git
迁移略，
迁移后地址：
https://gitee.com/weiwei20180921/mirror-web.git
https://gitee.com/weiwei20180921/mirrorz-help-ng.git
2.2 制作mirror-web 镜像
操作参考：https://github.com/tuna/mirror-web
2.2.1 克隆代码
[root@2023001 code]# git clone https://gitee.com/weiwei20180921/mirror-web.git
2.2.2 嵌入帮助文档
修改mirror-web/.gitmodules
url 改成gitee 的，否则网不行
修改后如下
[submodule "helpz"]
path = help/_posts/mirrorz-help-ng-transpiled
url = https://gitee.com/weiwei20180921/mirrorz-help-ng.git
branch = transpiled
~
执行嵌入命令
[root@2023001 mirror-web]# git submodule update --init
Submodule 'helpz' (https://gitee.com/weiwei20180921/mirrorz-help-ng.git) registered for path 'help/_posts/mirrorz-help-ng-transpiled'
Cloning into 'help/_posts/mirrorz-help-ng-transpiled'...
remote: Enumerating objects: 802, done.
remote: Counting objects: 100% (802/802), done.
remote: Compressing objects: 100% (129/129), done.
remote: Total 802 (delta 642), reused 802 (delta 642), pack-reused 0
Receiving objects: 100% (802/802), 269.09 KiB | 98.00 KiB/s, done.
Resolving deltas: 100% (642/642), done.
Submodule path 'help/_posts/mirrorz-help-ng-transpiled': checked out '61639dd0bbc8a7e8482e755aff566f1b8714b0c8'
[root@2023001 mirror-web]#
2.2.3 制作镜像
[root@2023001 mirror-web]# docker build -t builden -f Dockerfile.build .
2.2.4 运行
卡在这一步了，运行不起来。
2.3 使用nginx 提供web 服务
安装略，此处最后给出配置文件
三 常见问题
主要记录运维过程中出现的现象及问题以及解决办法
3.1 job 同步目录在增长，但看不到文件
比如如下两个job 的同步目录分别是是
/data/mirrors/openeuler/openEuler-22.03-LTS-SP3
/data/mirrors/openeuler/openEuler-22.03-LTS
这两个目录大小一直在增长，但是却看不到具体文件，
原因是文件都在隐藏临时目录.tmp 下，而且据观察是 只有整个/data/mirrors/openeuler/openEuler-22.03-LTS 同步完，才会将隐藏目录里内容转移到对应的真实目录里。
[mirrors@junpanpro openeuler]$ du -h --max-depth=2 /data/mirrors/openeuler/
41G /data/mirrors/openeuler/openEuler-22.03-LTS-SP3/EPOL
104G /data/mirrors/openeuler/openEuler-22.03-LTS-SP3/ISO
6.7G /data/mirrors/openeuler/openEuler-22.03-LTS-SP3/OS
211M /data/mirrors/openeuler/openEuler-22.03-LTS-SP3/WSL
39G /data/mirrors/openeuler/openEuler-22.03-LTS-SP3/debuginfo
170M /data/mirrors/openeuler/openEuler-22.03-LTS-SP3/docker_img
2.8G /data/mirrors/openeuler/openEuler-22.03-LTS-SP3/edge_img
6.5G /data/mirrors/openeuler/openEuler-22.03-LTS-SP3/embedded_img
0 /data/mirrors/openeuler/openEuler-22.03-LTS-SP3/everything
0 /data/mirrors/openeuler/openEuler-22.03-LTS-SP3/raspi_img
0 /data/mirrors/openeuler/openEuler-22.03-LTS-SP3/source
0 /data/mirrors/openeuler/openEuler-22.03-LTS-SP3/stratovirt_img
0 /data/mirrors/openeuler/openEuler-22.03-LTS-SP3/update
0 /data/mirrors/openeuler/openEuler-22.03-LTS-SP3/virtual_machine_img
199G /data/mirrors/openeuler/openEuler-22.03-LTS-SP3
26G /data/mirrors/openeuler/openEuler-22.03-LTS/EPOL
127G /data/mirrors/openeuler/openEuler-22.03-LTS/ISO
10G /data/mirrors/openeuler/openEuler-22.03-LTS/OS
5.4G /data/mirrors/openeuler/openEuler-22.03-LTS/debuginfo
0 /data/mirrors/openeuler/openEuler-22.03-LTS/docker_img
0 /data/mirrors/openeuler/openEuler-22.03-LTS/edge_img
0 /data/mirrors/openeuler/openEuler-22.03-LTS/embedded_img
0 /data/mirrors/openeuler/openEuler-22.03-LTS/everything
0 /data/mirrors/openeuler/openEuler-22.03-LTS/raspi_img
0 /data/mirrors/openeuler/openEuler-22.03-LTS/source
0 /data/mirrors/openeuler/openEuler-22.03-LTS/stratovirt_img
0 /data/mirrors/openeuler/openEuler-22.03-LTS/update
0 /data/mirrors/openeuler/openEuler-22.03-LTS/virtual_machine_img
168G /data/mirrors/openeuler/openEuler-22.03-LTS
366G /data/mirrors/openeuler/
[mirrors@junpanpro openeuler]$
致谢
本文参考了
https://www.jianshu.com/p/14e8b4456aca
https://blog.whsir.com/post-5094.html
感谢大佬们分享