GlusterFS pada Ubuntu 18.04¶
Catatan perintah-perintah yang dijalankan pada panduan:
- semua menggunakan root, atau tambahkan
sudopada setiap awal perintah jika sudah ada pengguna non-root-sudo - Karakter
\digunakan sebagai pemisah baris agar tampilan kode/perintah CLI lebih mudah dibaca, dianjurkan untuk membuangnya saat salin tempel atau mengetikkan pada terminal
Persiapan¶
Repositori & instalasi paket¶
# Pada sakorp-node01 dan sakorp-node02
apt-get install -y software-properties-common
add-apt-repository ppa:gluster/glusterfs-7
apt-get update -y
apt-get install -y glusterfs-server glusterfs-client
Pemeriksaan dan menjalankan otomatis layanan¶
# Pada sakorp-node01 dan sakorp-node02
systemctl start glusterd
systemctl enable glusterd
systemctl status glusterd
glusterfsd --version
Konfigurasi¶
Firewall¶
Gunakan iptables, ufw, fiwewalld atau antar muka firewall apapun untuk membuka komunikasi antar porta terlampir di bawah
24007- daemon24008- manajemen24009- dan seterusnya sejumlah node (digunakan pada GlusterFS versi 3.4 kebawah) atau49152(pada GlusterFS versi 3.4 atau lebih baru) – Each brick for every volume on your host requires it’s own port. For every new brick, one new port will be used starting at 24009 for GlusterFS versions below 3.4 and 49152 for version 3.4 and above. If you have one volume with two bricks, you will need to open24009 – 24010(or49152-49153).38465 – 38467- diperlukan untuk layanan Gluster NFS.
IP & /etc/hosts¶
Tambahkan IP dan nama host dari setiap node yang akan digunakan pada masing-masing node. Ini bertujuan agar setiap node lebih mudah dikenali dengan nama host dibandingkan dengan menggunakan alamat IP
# Pada sakorp-node01 dan sakorp-node02
nano /etc/hosts
127.0.0.1 localhost
10.20.30.2 sakorp-node01
10.20.30.5 sakorp-node02
# Pada sakorp-node01
ping -c3 sakorp-node02
# Pada sakorp-node02
ping -c3 sakorp-node01
Penambatan dan pemetaan partisi media¶
Gunakan kontrol pada DigitalOcean - Droplets > Volumes > More > Config instructions
# Pada setiap sakorp-node
BRICKPATH=/data/glusterfs/vol1/brick1
mkdir -p ${BRICKPATH}
# Pada sakorp-node01
VOLNAME=vol03-sakorp01
mount -o discard,defaults,noatime \
/dev/disk/by-id/scsi-0DO_Volume_${VOLNAME} ${BRICKPATH}
echo "/dev/disk/by-id/scsi-0DO_Volume_${VOLNAME} ${BRICKPATH} \
ext4 defaults,nofail,discard 0 0" | tee -a /etc/fstab
# Pada sakorp-node02
VOLNAME=vol04-sakorp02
mount -o discard,defaults,noatime \
/dev/disk/by-id/scsi-0DO_Volume_${VOLNAME} ${BRICKPATH}
echo "/dev/disk/by-id/scsi-0DO_Volume_${VOLNAME} ${BRICKPATH} \
ext4 defaults,nofail,discard 0 0" | tee -a /etc/fstab
# Periksa pada setiap sakorp-node
cat /etc/fstab
Trusted pool¶
# Pada sakorp-node01
gluster peer probe sakorp-node02
# Periksa status dan daftar node yang sudah ditambahkan
gluster peer status
gluster pool list
Number of Peers: 1
Hostname: sakorp-node02
Uuid: f38f9d8b-5925-4cd0-ad44-0180e4aa5eae
State: Peer in Cluster (Connected)
UUID Hostname State
f38f9d8b-5925-4cd0-ad44-0180e4aa5eae sakorp-node02 Connected
d8816715-ab25-4eaf-94cb-7993698a17a7 localhost Connected
Inisiasi volume replikasi¶
BRICKPATH=/data/glusterfs/vol1/brick1
gluster volume create swarm-gfs replica 2 \
sakorp-node0{1..2}:${BRICKPATH} \
force
gluster volume start swarm-gfs
gluster volume info
Memasangkan volume GlusterFS yang sudah terbuat pada setiap node¶
SUDOUSER=deploy
GROUP=docker
GFSPATH=/mnt/swarm-gfs
mkdir -p ${GFSPATH}
echo "localhost:/swarm-gfs ${GFSPATH} glusterfs \
defaults,_netdev,log-level=WARNING, \
log-file=/var/log/gluster.log,backupvolfile-server=localhost 0 0" \
| tee -a /etc/fstab
mount.glusterfs localhost:/swarm-gfs ${GFSPATH}
chown -R "${SUDOUSER}":"${GROUP}" ${GFSPATH}
Pengamanan akses node¶
# Hanya mengizinkan alamat IP tertentu
gluster volume set swarm-gfs auth.allow 10.20.30.2,10.20.30.5
Informasi volume dan lain-lain¶
# Melihat info volume
gluster volume info
Volume Name: swarm-gfs
Type: Replicate
Volume ID: c27a5812-b8b5-4017-b34a-0d13b68bd671
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: sakorp-node01:/data/glusterfs/vol1/brick1
Brick2: sakorp-node02:/data/glusterfs/vol1/brick1
Options Reconfigured:
diagnostics.count-fop-hits: on
diagnostics.latency-measurement: on
auth.allow: 10.20.30.2,10.20.30.5
transport.address-family: inet
storage.fips-mode-rchecksum: on
nfs.disable: on
performance.client-io-threads: off
# Melihat status volume
gluster volume status
Status of volume: swarm-gfs
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick sakorp-node01:/data/glusterfs/vol1/br
ick1 49152 0 Y 10616
Brick sakorp-node02:/data/glusterfs/vol1/br
ick1 49152 0 Y 4877
Self-heal Daemon on localhost N/A N/A Y 10637
Self-heal Daemon on sakorp-node02 N/A N/A Y 4898
Task Status of Volume swarm-gfs
------------------------------------------------------------------------------
There are no active volume tasks
# Ukuran FS
df -hT -x overlay -x tmpfs -x devtmpfs -x vfat
Filesystem Type Size Used Avail Use% Mounted on
/dev/vda1 ext4 25G 5.5G 19G 23% /
/dev/sda ext4 50G 473M 47G 1% /data/glusterfs/vol1/brick1
localhost:/swarm-gfs fuse.glusterfs 50G 980M 47G 3% /mnt/swarm-gfs
Profil volume¶
# Memulai profil
gluster volume profile swarm-gfs start
# Melihat hasil profil
gluster volume profile swarm-gfs info
Brick: sakorp-node01:/data/glusterfs/vol1/brick1
------------------------------------------------
Cumulative Stats:
Block Size: 8b+ 16b+ 32b+
No. of Reads: 0 0 0
No. of Writes: 5 4 82
Block Size: 64b+ 128b+ 256b+
No. of Reads: 0 0 0
No. of Writes: 327 1000 1112
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 0 0 0
No. of Writes: 1423 2125 2501
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 0 0 0
No. of Writes: 1770 1311 568
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 0 0 0
No. of Writes: 365 178 723
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 12775 RELEASE
0.00 0.00 us 0.00 us 0.00 us 2978 RELEASEDIR
0.01 48.49 us 48.49 us 48.49 us 1 GETXATTR
0.03 77.28 us 53.79 us 102.87 us 3 STAT
0.10 50.16 us 37.21 us 61.92 us 17 OPENDIR
0.73 382.14 us 178.45 us 2772.93 us 17 MKDIR
0.93 29.58 us 18.40 us 220.62 us 280 FLUSH
1.66 26.34 us 12.70 us 162.10 us 560 FINODELK
4.09 130.43 us 100.28 us 229.98 us 279 RENAME
4.26 120.89 us 69.75 us 329.00 us 314 WRITE
4.90 25.51 us 9.85 us 403.00 us 1708 ENTRYLK
5.11 28.33 us 13.15 us 531.92 us 1604 INODELK
9.47 98.68 us 46.16 us 329.80 us 854 SETATTR
12.59 401.62 us 180.58 us 25236.04 us 279 CREATE
17.50 85.59 us 24.48 us 2467.28 us 1820 LOOKUP
38.64 614.10 us 66.64 us 271807.57 us 560 FXATTROP
Duration: 975 seconds
Data Read: 0 bytes
Data Written: 177684306 bytes
Interval 0 Stats:
Block Size: 8b+ 16b+ 32b+
No. of Reads: 0 0 0
No. of Writes: 5 4 82
Block Size: 64b+ 128b+ 256b+
No. of Reads: 0 0 0
No. of Writes: 327 1000 1112
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 0 0 0
No. of Writes: 1423 2125 2501
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 0 0 0
No. of Writes: 1770 1311 568
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 0 0 0
No. of Writes: 365 178 723
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 12775 RELEASE
0.00 0.00 us 0.00 us 0.00 us 2978 RELEASEDIR
0.01 48.49 us 48.49 us 48.49 us 1 GETXATTR
0.03 77.28 us 53.79 us 102.87 us 3 STAT
0.10 50.16 us 37.21 us 61.92 us 17 OPENDIR
0.73 382.14 us 178.45 us 2772.93 us 17 MKDIR
0.93 29.58 us 18.40 us 220.62 us 280 FLUSH
1.66 26.34 us 12.70 us 162.10 us 560 FINODELK
4.09 130.43 us 100.28 us 229.98 us 279 RENAME
4.26 120.89 us 69.75 us 329.00 us 314 WRITE
4.90 25.51 us 9.85 us 403.00 us 1708 ENTRYLK
5.11 28.33 us 13.15 us 531.92 us 1604 INODELK
9.47 98.68 us 46.16 us 329.80 us 854 SETATTR
12.59 401.62 us 180.58 us 25236.04 us 279 CREATE
17.50 85.59 us 24.48 us 2467.28 us 1820 LOOKUP
38.64 614.10 us 66.64 us 271807.57 us 560 FXATTROP
Duration: 975 seconds
Data Read: 0 bytes
Data Written: 177684306 bytes
Brick: sakorp-node02:/data/glusterfs/vol1/brick1
------------------------------------------------
Cumulative Stats:
Block Size: 8b+ 16b+ 32b+
No. of Reads: 0 0 0
No. of Writes: 5 4 82
Block Size: 64b+ 128b+ 256b+
No. of Reads: 0 0 0
No. of Writes: 327 1000 1112
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 0 0 0
No. of Writes: 1423 2125 2501
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 0 0 0
No. of Writes: 1770 1311 568
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 0 0 0
No. of Writes: 365 178 723
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 12775 RELEASE
0.00 0.00 us 0.00 us 0.00 us 2978 RELEASEDIR
0.01 81.77 us 81.77 us 81.77 us 1 GETXATTR
0.13 69.29 us 42.81 us 122.66 us 17 OPENDIR
0.15 90.00 us 46.47 us 244.04 us 15 STAT
0.43 230.20 us 145.21 us 484.92 us 17 READDIRP
0.94 496.87 us 207.56 us 3224.50 us 17 MKDIR
1.68 55.63 us 18.01 us 847.37 us 273 FLUSH
2.13 73.43 us 40.03 us 532.32 us 262 FSTAT
3.43 56.69 us 17.35 us 882.56 us 546 FINODELK
5.12 150.63 us 80.05 us 438.73 us 307 WRITE
5.52 183.14 us 120.30 us 490.46 us 272 RENAME
8.38 48.41 us 18.50 us 671.19 us 1563 INODELK
9.31 50.43 us 14.55 us 988.99 us 1667 ENTRYLK
11.60 191.81 us 80.16 us 1424.84 us 546 FXATTROP
12.04 130.21 us 53.80 us 712.58 us 835 SETATTR
16.52 548.47 us 198.72 us 48080.38 us 272 CREATE
22.62 114.41 us 34.71 us 1001.38 us 1785 LOOKUP
Duration: 974 seconds
Data Read: 0 bytes
Data Written: 177684306 bytes
Interval 0 Stats:
Block Size: 8b+ 16b+ 32b+
No. of Reads: 0 0 0
No. of Writes: 5 4 82
Block Size: 64b+ 128b+ 256b+
No. of Reads: 0 0 0
No. of Writes: 327 1000 1112
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 0 0 0
No. of Writes: 1423 2125 2501
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 0 0 0
No. of Writes: 1770 1311 568
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 0 0 0
No. of Writes: 365 178 723
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 12775 RELEASE
0.00 0.00 us 0.00 us 0.00 us 2978 RELEASEDIR
0.01 81.77 us 81.77 us 81.77 us 1 GETXATTR
0.13 69.29 us 42.81 us 122.66 us 17 OPENDIR
0.15 90.00 us 46.47 us 244.04 us 15 STAT
0.43 230.20 us 145.21 us 484.92 us 17 READDIRP
0.94 496.87 us 207.56 us 3224.50 us 17 MKDIR
1.68 55.63 us 18.01 us 847.37 us 273 FLUSH
2.13 73.43 us 40.03 us 532.32 us 262 FSTAT
3.43 56.69 us 17.35 us 882.56 us 546 FINODELK
5.12 150.63 us 80.05 us 438.73 us 307 WRITE
5.52 183.14 us 120.30 us 490.46 us 272 RENAME
8.38 48.41 us 18.50 us 671.19 us 1563 INODELK
9.31 50.43 us 14.55 us 988.99 us 1667 ENTRYLK
11.60 191.81 us 80.16 us 1424.84 us 546 FXATTROP
12.04 130.21 us 53.80 us 712.58 us 835 SETATTR
16.52 548.47 us 198.72 us 48080.38 us 272 CREATE
22.62 114.41 us 34.71 us 1001.38 us 1785 LOOKUP
Duration: 974 seconds
Data Read: 0 bytes
Data Written: 177684306 bytes
Pengaturan performa¶
server.allow-insecure¶
https://docs.gluster.org/en/latest/Administrator%20Guide/Managing%20Volumes/#tuning-options
volume set swarm-gfs server.allow-insecure on
Volume Name: swarm-gfs
Type: Replicate
Volume ID: c27a5812-b8b5-4017-b34a-0d13b68bd671
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: sakorp-node01:/data/glusterfs/vol1/brick1
Brick2: sakorp-node02:/data/glusterfs/vol1/brick1
Options Reconfigured:
server.allow-insecure: on
diagnostics.count-fop-hits: on
diagnostics.latency-measurement: on
auth.allow: 10.20.30.2,10.20.30.5
transport.address-family: inet
storage.fips-mode-rchecksum: on
nfs.disable: on
performance.client-io-threads: off
Rujukan¶
- https://docs.gluster.org/en/latest/Install-Guide/Install/
- https://docs.gluster.org/en/latest/Install-Guide/Configure/
- Configure a High-Availability Storage with GlusterFS on Ubuntu 18.04 LTS
- Setup Docker Swarm Cluster on Ubuntu 18.04 in Azure VPS
- Split brain and the ways to deal with it
- Arbiter volumes and quorum options in gluster
- Brick Naming Conventions
- Setting up GlusterFS Volumes
- Thin Arbiter volumes in gluster
- Accessing Data - Setting Up GlusterFS Client
- Setup a 3 Node Replicated Storage Volume with GlusterFS
- https://www.digitalocean.com/community/tutorials/how-to-create-a-redundant-storage-pool-using-glusterfs-on-ubuntu-servers