Where does the space go?
A log collector is deployed with 4 1TB disk pairs. The GUI reports 3.23 TB of total space that can be allocated via quota. Various CLI commands show different values from the GUI. What is going on here? How much space do you actually have for logs?
Note: Only 66% of Storage is used for Log Storage
How space is allocated?
There are several factors that affect how much space on a disk is used for logs in Elasticsearch. The following graphic explains where space goes when a disk or disk pair is added to a log collector.
Total Size
In this example, a 1 TB disk is added to a log collector. Right from the beginning a discrepancy can be seen between the size of the disk (1 TB) and the amount of space that the operating system sees (917 GB). This discrepancy is due to a couple of factors:
- File system overhead: There is some overhead associated with creation of the file system on the disk, but it is fairly minor.
- 1000 vs. 1024 Bytes per Kilobyte: Hard drive manufacturers count 1000 bytes per kilobyte while the operating system uses 1024 bytes per kilobyte. This is the biggest source of the discrepancy that we see above. You still actually have the same number of bytes available, hard drive manufacturers just count the kilobytes differently.
Logd Formatted Logs
One third (~33%) of the available disk space is allocated to logd formatted logs. The logd format is what was used prior to the introduction of Elasticsearch. Post 8.0, the logd formatted logs are stored to support upgrade and downgrade only. No indexes are generated for these logs.
Elasticsearch
Two thirds (~66%) of the available disk space is allocated for use by Elasticsearch. Configured quotas are applied against this allocated space. In the graphic above it can be seen that the default quota percentages are applied against the 66% of the total disk space that is allocated to Elasticsearch.
Checking disk usage on the CLI
To check overall disk usage:
> show system disk-space
This command will show the PAN-OS equivalent to the Unix ‘df -h’ command. Usage statistics for each mount are included. Logging disks are mounted on /opt/panlogs/:
Looking at the ‘Size’ column for the logging disks shows that the operating system sees the disks as 917 GB due to the 1000 vs 1024 bytes per kilobyte discrepancy.
To check usage by Elasticsearch
> show system search-engine-quota
This command will show the status of Elasticsearch’s disk allocation (66% of total disk size):
To check usage by logd
> show system logdb-quota
This command will show the space used for storing logdb formatted logs (33% of total disk size):
How to Display PAN or Panorama Log Database (logdb) Disk Space
Disk usage looks at the accumulation of all of the logs and will never reach %100 because the logs will overwrite themselves.
Use the following CLI command to display the log partition size on a PAN or Panorama:
The sample output below is from Panorama
>show system logdb-quota
total log disk size: 10 GB
quotas:
traffic: 50%, 5GB
threat: 16%, 1GB
system: 4%, 0GB
config: 4%, 0GB
appstat: 4%, 0GB
trsum: 4%, 0GB
trsum: 4%, 0GB
Disk usage:
traffic: Logs: 4.8G, Index: 748M
threat: Logs: 1.6G, INdex: 274M
appstatdb: Logs: 38M, Index: 13M
trsum: Logs: 8.0K, Index: 8.0K
thsum: Logs: 8.0K, Index: 8.0K
config: Logs: 1.6M, Index: 600K
system: Logs: 26M, Index:5.1M
Data:
traffic Logs: 4.8G, Index 778M
theat: Logs: 1.6G, Index: 270M
appstatdb: Logs: 37M, Index: 12M
trsum: Logs: 4.1K, Index: 4.1K
thsum: Logs: 4.1K, Index: 4.1K
config: Logs: 1.4M, Index: 292K
sysem: Logs: 25M, Index: 2M
There are 3 sections:
Quotas: is the actual quotas that are configured on the drive.
Disk usage: is how much of the disk space is actually used by the different logs, based on the disk layout (i.e. blocksize)
Data: is how much data there actually is, if it were not tied to the disk layout.
Within Disk usage and Data, Logs represents the space used just by the log file.
Index is the space used by the index of the log file. An index is used for increasing performance of queries.
To clear a log file, enter the CLI command:
> clear log <log>
How to Determine How Much Disk Space is Allocated to Logs
View Disk space allocated to logs
The sample output below is from Palo Alto Firewall
- From the CLI run the command show system disk-space
PA-VM> show system disk-space Filesystem Size Used Avail Use% Mounted on /dev/root 7.0G 4.1G 2.6G 62% / none 3.2G 92K 3.2G 1% /dev /dev/sda5 16G 2.4G 13G 16% /opt/pancfg /dev/sda6 8.0G 3.2G 4.4G 43% /opt/panrepo tmpfs 2.2G 1.7G 412M 81% /dev/shm cgroup_root 3.2G 0 3.2G 0% /cgroup /dev/sda8 21G 511M 20G 3% /opt/panlogs
- Check the /opt/panlogs partition shows how much is allocated to the logs
View and edit Disk quota for specific logs
From GUI
- Device > Setup > scroll down to Logging and Reporting Settings
- Click the Gear icon
NOTE: Logs are purged when the quota is exceeded, so it is recommended not to allocate more than 95% of the space to allow some buffer space. Set the “Max Days” (Retention Period) so that log purging operation works seamlessly and prevents the disk from filling up.
From the CLI
- Use the show system logdb-quota ccommand
PA-VM> show system logdb-quota
Quotas:
system: 4.00%, 0.609 GB Expiration-period: 0 days
config: 4.00%, 0.609 GB Expiration-period: 0 days
alarm: 3.00%, 0.457 GB Expiration-period: 0 days
appstat: 4.00%, 0.609 GB Expiration-period: 0 days
hip-reports: 1.00%, 0.152 GB Expiration-period: 0 days
traffic: 29.00%, 4.414 GB Expiration-period: 0 days
threat: 15.00%, 2.283 GB Expiration-period: 0 days
trsum: 7.00%, 1.065 GB Expiration-period: 0 days
hourlytrsum: 3.00%, 0.457 GB Expiration-period: 0 days
dailytrsum: 1.00%, 0.152 GB Expiration-period: 0 days
weeklytrsum: 1.00%, 0.152 GB Expiration-period: 0 days
urlsum: 2.00%, 0.304 GB Expiration-period: 0 days
hourlyurlsum: 1.00%, 0.152 GB Expiration-period: 0 days
dailyurlsum: 1.00%, 0.152 GB Expiration-period: 0 days
weeklyurlsum: 0.75%, 0.114 GB Expiration-period: 0 days
thsum: 2.00%, 0.304 GB Expiration-period: 0 days
hourlythsum: 1.00%, 0.152 GB Expiration-period: 0 days
dailythsum: 1.00%, 0.152 GB Expiration-period: 0 days
weeklythsum: 1.00%, 0.152 GB Expiration-period: 0 days
userid: 1.00%, 0.152 GB Expiration-period: 0 days
iptag: 1.00%, 0.152 GB Expiration-period: 0 days
application-pcaps: 1.00%, 0.152 GB Expiration-period: 0 days
extpcap: 1.00%, 0.152 GB Expiration-period: 0 days
debug-filter-pcaps: 1.00%, 0.152 GB Expiration-period: 0 days
dlp-logs: 1.00%, 0.152 GB Expiration-period: 0 days
hipmatch: 3.00%, 0.457 GB Expiration-period: 0 days
gtp: 2.00%, 0.304 GB Expiration-period: 0 days
gtpsum: 1.00%, 0.152 GB Expiration-period: 0 days
hourlygtpsum: 0.75%, 0.114 GB Expiration-period: 0 days
dailygtpsum: 0.75%, 0.114 GB Expiration-period: 0 days
weeklygtpsum: 0.75%, 0.114 GB Expiration-period: 0 days
auth: 1.00%, 0.152 GB Expiration-period: 0 days
sctp: 0.00%, 0.000 GB Expiration-period: 0 days
sctpsum: 0.00%, 0.000 GB Expiration-period: 0 days
hourlysctpsum: 0.00%, 0.000 GB Expiration-period: 0 days
dailysctpsum: 0.00%, 0.000 GB Expiration-period: 0 days
weeklysctpsum: 0.00%, 0.000 GB Expiration-period: 0 days
decryption: 1.00%, 0.152 GB Expiration-period: 0 days
desum: 1.00%, 0.152 GB Expiration-period: 0 days
hourlydesum: 0.00%, 0.000 GB Expiration-period: 0 days
dailydesum: 0.00%, 0.000 GB Expiration-period: 0 days
weeklydesum: 0.00%, 0.000 GB Expiration-period: 0 days
globalprotect: 1.00%, 0.152 GB Expiration-period: 0 days
Disk usage:
traffic: Logs and Indexes: 104M Current Retention: 21 days
threat: Logs and Indexes: 24K Current Retention: 0 days
system: Logs and Indexes: 17M Current Retention: 21 days
config: Logs and Indexes: 8.3M Current Retention: 21 days
alarm: Logs and Indexes: 20K Current Retention: 0 days
trsum: Logs and Indexes: 106M Current Retention: 21 days
hourlytrsum: Logs and Indexes: 97M Current Retention: 21 days
dailytrsum: Logs and Indexes: 5.2M Current Retention: 20 days
weeklytrsum: Logs and Indexes: 948K Current Retention: 18 days
thsum: Logs and Indexes: 204K Current Retention: 0 days
hourlythsum: Logs and Indexes: 268K Current Retention: 0 days
dailythsum: Logs and Indexes: 252K Current Retention: 0 days
weeklythsum: Logs and Indexes: 40K Current Retention: 0 days
appstatdb: Logs and Indexes: 2.2M Current Retention: 21 days
userid: Logs and Indexes: 16K Current Retention: 0 days
iptag: Logs and Indexes: 16K Current Retention: 0 days
hipmatch: Logs and Indexes: 20K Current Retention: 0 days
hip-reports: Logs and Indexes: Current Retention: 0 days
extpcap: Logs and Indexes: 16K Current Retention: 0 days
urlsum: Logs and Indexes: 204K Current Retention: 0 days
hourlyurlsum: Logs and Indexes: 268K Current Retention: 0 days
dailyurlsum: Logs and Indexes: 252K Current Retention: 0 days
weeklyurlsum: Logs and Indexes: 40K Current Retention: 0 days
gtp: Logs and Indexes: 16K Current Retention: 0 days
gtpsum: Logs and Indexes: 200K Current Retention: 0 days
hourlygtpsum: Logs and Indexes: 268K Current Retention: 0 days
dailygtpsum: Logs and Indexes: 252K Current Retention: 0 days
weeklygtpsum: Logs and Indexes: 40K Current Retention: 0 days
auth: Logs and Indexes: 16K Current Retention: 0 days
sctp: Logs and Indexes: 16K Current Retention: 0 days
sctpsum: Logs and Indexes: 200K Current Retention: 0 days
hourlysctpsum: Logs and Indexes: 8.0K Current Retention: 0 days
dailysctpsum: Logs and Indexes: 8.0K Current Retention: 0 days
weeklysctpsum: Logs and Indexes: 8.0K Current Retention: 0 days
decryption: Logs and Indexes: 16K Current Retention: 0 days
desum: Logs and Indexes: 200K Current Retention: 0 days
hourlydesum: Logs and Indexes: 8.0K Current Retention: 0 days
dailydesum: Logs and Indexes: 8.0K Current Retention: 0 days
weeklydesum: Logs and Indexes: 8.0K Current Retention: 0 days
globalprotect: Logs and Indexes: 16K Current Retention: 0 days
application: Logs and Indexes: 12K Current Retention: 10 days
filters: Logs and Indexes: 4.0K Current Retention: 0 days
dlp: Logs and Indexes: 4.0K Current Retention: 0 days
hip_report_base: Logs and Indexes: 1.1M Current Retention: N/A
wildfire: Logs and Indexes: 40K Current Retention: N/A
Space reserved for cores: 0MB
Leave A Comment?