Recovering QRadar after a server crash

Sometimes, after an emergency or an unwanted reboot of your server that hosts QRadar, nothing is working at all. Don't panic, it may be a simple storage issue, indeed, server tend to log everything in a way that defies all logic. Imagine having a heart attack and focusing on every single metric of your body. The result for you will be a lot more dramatic, but still, doing so will not be healthy for QRadar either. Why? Because he logged too much data in a tiny partition: /var

So first thing to do after a server crash, verify basics as storage. Just do a simple df -Th and verify if /var or another partition is above the limit that prevents QRadar core services to start. If you identify an issue on /var partition, you can go with the hammer and rm -rf the big crash file or be gentle and (re)move the file properly. How? First, check if files in this partition are opened:

lsof /var | less

If one or more files are opened, find the process and kill it properly if you can. Then, move the crash file to another location, with much more space, for instance /store (check the remaining space before just in case).

mv -fv <file> /store/crash_analysis

After that because /var is about logging, do a quick check up on associated services to start from zero.

/etc/cron.hourly/logrotate
systemctl restart syslog-ng
systemctl restart crond

At the end, you will be able to start QRadar core services with the below command.

/opt/qradar/upgrade/util/setup/upgrades/wait_for_start.sh

You reach the end of this article, huge thanks for reading 🫶

👨‍👩‍👧‍👦 To become a more friendly guru of QRadar, join the community and subscribe to the newsletter.

🤗 To become a nicer guru of QRadar, leave a comment, your feedback will always be welcome (when constructive of course).