-
Titel
Recover from full disk situation -
Beschreibung
SPS uses ext4 filesystem that has 5% of reserved blocks configured by default. You can free up this reserved space temporarily and run an archiving or cleanup to get some free space.
-------------------
WARNING: The procedure regarding the use of reserved blocks is not applicable for SSB! SSB does not contain 5% of reserved blocks by default.
In case of SSB, please only refer to steps 5., 6., 8. and 9.
-------------------
1. Log in to SPS via SSH if possible or connect to the local console (IPMI Console or "Launch Remote Console" on vmware)
2. Check the file system if it is really full and get the device name which reserved space should be freed of. Execute the following command and note the first column that is the name of the device.
df -h /mnt/drbd
Example outputs. The first field is the device name in bold.
Physical/Virtual appliance with DRBD
/dev/drbd0 908G 908G 0 100% /mnt/drbd
Virtual appliance without DRBD
/dev/sda4 498G 498G 0 100% /mnt/drbd
SPS 6.0 or later with LVM (not upgraded from SPS 5)
/dev/mapper/vg--root-core 560G 560G 0 100% /mnt/drbd
3. Set the reserved block percentage to 0.
Physical/Virtual appliance with DRBD
tune2fs -m 0 /dev/drbd0
Virtual appliance without DRBD
tune2fs -m 0 /dev/sda4
SPS 6.0 or later with LVM (not upgraded from SPS 5)
tune2fs -m 0 /dev/mapper/vg--root-core
4. Check the free disk space
df -h /mnt/drbd
5. Log in to the WEB UI.
HTTP service may be broken because of the issue and you need to restart it in the core shell with: systemctl restart nginx.service
6. Perform archive or cleanup. It may be necessary to reduce the retention time of the Archiving/Cleanup policies.
7. Set the reserved block percentage back to 5.
Physical/Virtual appliance with DRBD
tune2fs -m 5 /dev/drbd0
Virtual appliance without DRBD
tune2fs -m 5 /dev/sda4
SPS 6.0 or later with LVM (not upgraded from SPS 5)
tune2fs -m 5 /dev/mapper/vg--root-core
8. Configure Disk fill-up prevention if it was not yet enabled (0) at Basic Settings --> Management --> Disk space fill up prevention
9. Check if everything works fine. You may need to restart services at Basic Settings --> System --> Traffic control or Service Control
-
Ursache
The disk has been filled up and no more free space is available. -
Lösung
SPS uses ext4 filesystem that has 5% of reserved blocks configured by default. You can free up this reserved space temporarily and run an archiving or cleanup to get some free space.
-------------------
WARNING: The procedure regarding the use of reserved blocks is not applicable for SSB! SSB does not contain 5% of reserved blocks by default.
In case of SSB, please only refer to steps 5., 6., 8. and 9.
-------------------1. Log in to SPS via SSH if possible or connect to the local console (IPMI Console or "Launch Remote Console" on vmware)
2. Check the file system if it is really full and get the device name which reserved space should be freed of. Execute the following command and note the first column that is the name of the device.
df -h /mnt/drbd
Example outputs. The first field is the device name in bold.
Physical/Virtual appliance with DRBD
/dev/drbd0 908G 908G 0 100% /mnt/drbd
Virtual appliance without DRBD
/dev/sda4 498G 498G 0 100% /mnt/drbd
SPS 6.0 or later with LVM (not upgraded from SPS 5)
/dev/mapper/vg--root-core 560G 560G 0 100% /mnt/drbd
3. Set the reserved block percentage to 0.
Physical/Virtual appliance with DRBD
tune2fs -m 0 /dev/drbd0
Virtual appliance without DRBD
tune2fs -m 0 /dev/sad4
SPS 6.0 or later with LVM (not upgraded from SPS 5)
tune2fs -m 0 /dev/mapper/vg--root-core
4. Check the free disk space
df -h /mnt/drbd
5. Log in to the WEB UI.
HTTP service may be broken because of the issue and you need to restart it in the core shell with: systemctl restart nginx.service
6. Perform archive or cleanup. It may be necessary to reduce the retention time of the Archiving/Cleanup policies.
7. Set the reserved block percentage back to 5.
Physical/Virtual appliance with DRBD
tune2fs -m 5 /dev/drbd0
Virtual appliance without DRBD
tune2fs -m 5 /dev/sad4
SPS 6.0 or later with LVM (not upgraded from SPS 5)
tune2fs -m 5 /dev/mapper/vg--root-core
8. Configure Disk fill-up prevention if it was not yet enabled (0) at Basic Settings --> Management --> Disk space fill up prevention
9. Check if everything works fine. You may need to restart services at Basic Settings --> System --> Traffic control or Service Control