Step One - Cisco Firepower Device Problem Description
Step Two - Document the Cisco Firepower Runtime Environment
Step Three - Verify the Integrity of System Files
Step Four - Verify Digitally Signed Image Authenticity
Step Five - Verify FTD Memory .text Segment Integrity
Step Six - Cisco Firepower Crashinfo File/Core File
Step Seven - ROMMON Settings Check
Cisco Firepower 2100 Series Forensic Report Checklist
Note: This document is no longer being maintained or updated, and has been superseded by the Cisco Firepower 1000/2100 Series Forensic Data Collection Procedures document.
This document provides steps to collect forensic information from Cisco Firepower 2100 series appliances running Firepower eXtensible Operating System (FXOS) Software when compromise or tampering is suspected. It outlines a number of commands that can be run to gather evidence for an investigation along with the respective output that should be collected upon running these commands. This document also provides information on how to perform integrity checks on FXOS system and Firepower Threat Defense (FTD) application images and includes a procedure for collecting a memory dump, crashinfo file, and a core file from a Cisco Firepower device.
Note: Firepower Threat Defense (FTD) investigation procedures for the ASA 5500-X series of firewalls are covered in a separate publication, which can be accessed here: Cisco Firepower Threat Defense Forensic Investigation Procedures for First Responders.
IMPORTANT: DO NOT REBOOT THE DEVICE. Rebooting a device during the initial stage of an assessment will irrecoverably lose all volatile information contained within the device (e.g., RAM contents, arp & routing tables, NAT translations, ACL hit and drop counts, etc.).
Note: It is highly recommended that a device suspected of tampering or compromise be isolated from the network prior to conducting an initial forensic examination. This may prevent remote unloading of any implants or malware installed on the device and will prevent an adversary from monitoring commands entered on the device under investigation.
If you require assistance or have questions regarding the following procedures, contact the Product Security Incident Response Team (PSIRT).
The main section of this document contains seven sections:
1. Cisco Firepower Device Problem Description Describes why the platform is a candidate for forensic examination
2. Cisco Firepower Runtime Environment Collects platform configuration and runtime state
3. Verify the Integrity of FTD System Files Examines system file image hashes for inconsistencies
4. Digitally Signed Image Verification Examines the FXOS operating system for proper signing characteristics
5. Verify Memory .text Segment Retrieves and calculates a hash of the FTD .text segment
6. Crashinfo/Core File Obtains crashinfo and core files from the running FTD application
7. ROM Monitor Variables Examines ROM Monitor settings for remote system image loading
The procedures outlined in this document assume that the reader has a basic understanding of Cisco FXOS Software, Cisco Firepower Threat Defense Software, and Linux command syntax.
A valid cisco.com account is required to view individual FXOS Software and FTD firmware file hashes for software file integrity checking. For customers without a cisco.com account, a publicly available comprehensive list of file hashes (Bulk Hash File) can be downloaded from: https://www.cisco.com/c/en/us/about/trust-center/downloads.html
Note: The examples used in this document are based on Cisco Firepower Threat Defense (FTD) Software Release 7.3.1 command syntax. The output that is produced by a command may vary depending on the software release that is deployed and/or the features that are supported or configured on the device. Not all commands that are used in these procedures may be supported on earlier releases of the software.
Describe in as much detail as possible WHY the device is a candidate for forensic examination. Are there configuration changes that cannot be explained? Is there unusual traffic originating from or terminating on the device? Are there anomalous entries in the device logs or in syslog messages? Is the device exhibiting odd behavior that cannot be attributed to a misconfiguration or a software or hardware defect? Are there any typical device administration commands that are now returning unusual output or no output at all?
Use the Cisco Software Checker to search for Cisco Security Advisories that apply to specific software releases of the following products: Cisco ASA, FMC, FTD, FXOS, IOS, IOS XE, NX-OS, and NX-OS in ACI Mode.
https://sec.cloudapps.cisco.com/security/center/softwarechecker.x
Record any results that are returned by the tool that may explain the anomalous behavior being observed. It is considered a best practice to keep software up to date to take advantage of the latest security fixes and enhancements.
Note: This tool does not provide information about Cisco IOS XR Software or interim software builds. Also note that for Cisco ASA, FMC, FTD, and FXOS Software, the tool contains only vulnerability information for Cisco Security Advisories first published from January 2022 onward, and for NX-OS Software and NX-OS Software in ACI Mode from July 2019 onward.
Submit the problem description and any relevant results that are obtained from the Cisco Software Checker and collected in this section to the relevant TAC SR and proceed to the next section of this document.
Note: The Cisco Firepower series of appliances is capable of running either Cisco Firepower Threat Defense (FTD) Software or Cisco Adaptive Security Appliance (ASA) Software under the FXOS operating system. The examples provided in this guide use commands and syntax suitable for FTD Software. Please see the Cisco ASA Forensic Investigation Procedures for First Responders guide for examples specific to Cisco ASA Software.
The initial stage of forensic information gathering is completed by issuing a show tech-support command and a number of optional show commands to collect more granular operating environment details. These commands are to be executed from the privileged EXEC mode of the FTD diagnostic CLI and some of the output produced may vary depending on the particular FTD Software version and/or features supported/configured on the device.
Note: For the Cisco 2100 Series of Firepower appliances, the default command prompt will vary depending on the method used to access the platform. If using SSH, the user will be placed in the FTD CLI. If using a console connection, the user will be placed in the FXOS CLI. Issue the connect ftd command to access the FTD CLI from the FXOS CLI.
Execute the following command from the FTD CLI prompt:
system support diagnostic-cliExecute each of the following commands in the diagnostic CLI and record the output:
enable terminal pager 0 show tech-support detail dir all-filesystems
The following list of commands should be executed to gather additional information relevant to the current operating state of the device. Although the output of these commands is not required to perform a forensic analysis of an FTD platform, it may provide additional information regarding any unauthorized changes made to the device if compromise is suspected.
Execution of the following commands in this section of Step 2 is optional.
terminal pager 0 show history show clock detail show startup-config show reload show processes show kernel process detail show kernel ifconfig show kernel module show logging show route show eigrp neighbor show ospf neighbor show bgp summary show arp show ip address show interface ip brief show nat detail show snmp-server user show snmp-server group show ipv6 interface brief show ipv6 route show conn all show xlate show aaa login-history
Submit all command output collected in this section to the relevant TAC SR and proceed to the next section of this document.
Connect to the FTD CLI. This can be accomplished from the FXOS CLI by issuing the connect ftd command.
From the FTD CLI, issue the following commands to assume root permissions, set the appropriate environment variables, run the system file integrity checks, and collect the necessary files for forensic assessment.
Access expert mode and sudo to the root account:
expert sudo su -
Set the FIPS environment variable and run the integrity checks:
export FIPS_MODE=1 find /ngfw/var/sf/.icdb/* -name *.icdb.RELEASE.tar | xargs sha512sum cat /proc/*/smaps > /tmp/all-process-smaps.txt verify_file_integ.sh
Locate and retrieve copies of the following files:
verify_file_integ.sh verify_signed_db.sh db_manage.sh /ngfw/etc/certs/*.crt /ngfw/var/log/sf/verify_file_integ.log /ngfw/var/tmp/merged-db/master.db
Create an archive of the files listed above and copy the archive off the platform:
tar -cvf SR-<sr_number>.tar sha512sum SR-<sr_number>.tar ftp or scp
An example of this procedure follows:
firepower# connect ftd > expert ************************************************************* NOTICE - Shell access will be deprecated in future releases and will be replaced with a separate expert mode CLI. ************************************************************** admin@firepower:/$ sudo su - We trust you have received the usual lecture from the local System Administrator. It usually boils down to these three things: #1) Respect the privacy of others. #2) Think before you type. #3) With great power comes great responsibility. Password: root@firepower:~# export FIPS_MODE=1 # Check to see if the environment variable is set correctly root@firepower:~# set | grep FIPS_MODE FIPS_MODE=1 root@firepower:~# find /ngfw/var/sf/.icdb/* -name *.icdb.RELEASE.tar | xargs sha512sum
04f4f3cd8fab83099faf41eda608e7fd3164bd75f0505585d70f43a5cd70c3f8f11c338fab727904011c9c9de8fedd4f3f5ca34d49f1e370543dc04d97355d4e
/ngfw/var/sf/.icdb/0000/base-octeon-6.4.0-102.icdb.RELEASE.tar
63c99b2b92895188f921bfbde6f000d670b65ac07642375b9d9ed8aedb63761441241006a86afb047f7e27d02c10b4df4df8c860b8fba601f91b3a36bebf2cd8
/ngfw/var/sf/.icdb/0000/base-6.4.0-102.icdb.RELEASE.tar
fd5be05b711a5e13a26bdf7a6e9d8b43c8633cde9357f5c429e0b3f1fa4ba4acd203ad7f7bf3a2ec14be1214e75713df4fc2a9def6ac09b301ad2618c475c6e3
/ngfw/var/sf/.icdb/0001/patch-v6.4.0-OS-6.4.0.4.icdb.RELEASE.tar
720ea233cff7e59502e95e897a985f7832f992e7961541bc0fae0b5ca7b535479546ccbd0566db264d7ecc4cbc5c60a8df98894beb8c441d313458a8b2fd3bd9
/ngfw/var/sf/.icdb/0002/onboxui-6.4.0.4.icdb.RELEASE.tar
d63661140c552ea67e0841bc46e968b732d63932fc6b1bde6d528df8b7e8f55cdd0fc05a0d79b9bed6f10139be6ee7aaa6c65b383944dd1b9880ae8596cc966a
/ngfw/var/sf/.icdb/0002/patch-v6.4.0-6.4.0.4.icdb.RELEASE.tar # # Note: the file names, number of files, and hash values may vary depending on # the version of software running on the appliance and whether any software # updates have been applied. It is extremely important that the output # (including file names and hashes) generated by the command above be # submitted to the TAC SR. # root@firepower:~# cat /proc/*/smaps > /tmp/all-process-smaps.txt
root@firepower:~# verify_file_integ.sh
Running file integrity checks...
Successfully verified file integrity
# Once the file integrity verification script has completed, unset the # FIPS_MODE environment variable:
root@firepower:~# unset FIPS_MODE
# Identify the location of the system integrity scripts with the which # command:
root@firepower:~# which verify_file_integ.sh
/ngfw/usr/local/sf/bin/verify_file_integ.sh
root@firepower:~# which verify_signed_db.sh
/ngfw/usr/local/sf/bin/verify_signed_db.sh
root@firepower:~# which db_manage.sh
/ngfw/usr/local/sf/bin/db_manage.sh
# Archive a copy of these scripts, along with all certificates found in # /ngfw/etc/certs/, and the files
# /ngfw/var/log/sf/verify_file_integ.log and # /ngfw/var/tmp/merged-db/master.db
root@firepower:~# tar -cvf SR-1234567890.tar /ngfw/usr/local/sf/bin/verify_file_integ.sh /ngfw/usr/local/sf/bin/verify_signed_db.sh
/ngfw/usr/local/sf/bin/db_manage.sh
/ngfw/etc/certs/*.crt /ngfw/var/log/sf/verify_file_integ.log /ngfw/var/tmp/merged-db/master.db
/tmp/all-process-smaps.txt
tar: Removing leading `/' from member names
/ngfw/usr/local/sf/bin/verify_file_integ.sh
/ngfw/usr/local/sf/bin/verify_signed_db.sh
/ngfw/usr/local/sf/bin/db_manage.sh
/ngfw/etc/certs/SRU_rel.crt
/ngfw/etc/certs/rel.crt
/ngfw/var/log/sf/verify_file_integ.log
/ngfw/var/tmp/merged-db/master.db
/ngfw/etc/certs/rewhich db_manage.sh
/tmp/all-process-smaps.txt
# Create a hash of the tar file: root@firepower:~# sha512sum SR-1234567890.tar 8833409f7affbae6ab19cdd8ca11153252f1176e147b3d4c373a7195e261da5ca72a3260a5aa1a5f991b12c1f6ac070d4ece31765e413f0e5e4afd6c95cf6ea5
SR-1234567890.tar root@firepower:~# # Copy the tar file off the platform using FTP or SCP: root@firepower:~# ftp 10.10.10.1 Connected to 10.10.10.1. 220 Microsoft FTP Service Name (10.10.10.1:admin): anonymous 331 Anonymous access allowed, send identity (e-mail name) as password. Password: 230 User logged in. Remote system type is Windows_NT. ftp> bin 200 Type set to I. ftp> put SR-1234567890.tar local: SR-1234567890.tar remote: SR-1234567890.tar 229 Entering Extended Passive Mode (|||59418|) 125 Data connection already open; Transfer starting. 100% |***********************************| 1990 KiB 104.76 MiB/s 00:00 ETA 226 Transfer complete. 2037760 bytes sent in 00:00 (100.48 MiB/s) ftp> quit 221 Goodbye.
Submit all command output and the files gathered in this step to the relevant TAC SR and proceed to the next section of this document.
Cisco FXOS Software implements digitally signed system images on most platforms. Digitally signed Cisco FXOS Software uses asymmetric (public-key) cryptography which increases the security posture of Cisco Firepower devices by ensuring that the system image has not been altered.
Certain platforms running FXOS Software, such as the Cisco Firepower 2100 series of platforms, also support Cisco Secure Boot technologies. Cisco Secure Boot is a secure startup process that a Cisco device performs each time it boots up. Beginning with the initial power-on, a special purpose hardware device, known as the Trust Anchor module, verifies the integrity of the ROMMON code and the FXOS image via digital signatures as each are loaded. If any failures are detected, the user is notified of the error and the device will wait for the operator to correct the error. This prevents the network device from executing tainted network software.
For additional information see Trust Anchor Technology.
Note: The show software authenticity set of commands is only supported on Firepower platforms that incorporate Cisco Secure Boot technologies.
The authenticity and integrity of a system image file can be verified by using the following commands:
An example of this procedure follows:connect local-mgmt cd bootflash:/ show file .boot_string show software authenticity file <path/filename> verify signature <path/filename>
firepower# connect local-mgmt firepower(local-mgmt)# cd bootflash:/ firepower(local-mgmt)# show file .boot_string disk0:installables/switch/fxos-k8-fp2k-lfbff.2.4.1.216.SPA firepower(local-mgmt)# show software authenticity file /installables/switch/fxos-k8-fp2k-lfbff.2.4.1.216.SPA File Name : <local>/fxos-k8-fp2k-lfbff.2.4.1.216.SPA Image type : Release Signer Information Common Name : abraxas Organization Unit : FXOS Organization Name : CiscoSystems Certificate Serial Number : 5BF5D36A Hash Algorithm : SHA2 512 Signature Algorithm : 2048-bit RSA Key Version : A
The Organization Unit, Organization Name, and Certificate Serial Number values can be viewed to verify whether the system image signature is valid.
Next, calculate a hash for the FXOS system image and verify the digital signature.
firepower(local-mgmt)# verify signature /installables/switch/fxos-k8-fp2k-lfbff.2.4.1.216.SPA Done! Computed Hash SHA2: 3688624cde65157037be90d66f9cd707 c781f6a1a9f6b84b175479628f73b1e3 6ef740867948a25ce9296598dab06009 6a1f5bf3cd3c32718d723bc09e269b3a Embedded Hash SHA2: 3688624cde65157037be90d66f9cd707 c781f6a1a9f6b84b175479628f73b1e3 6ef740867948a25ce9296598dab06009 6a1f5bf3cd3c32718d723bc09e269b3a The digital signature of the file: fxos-k8-fp2k-lfbff.2.4.1.216.SPA verified successfully
It is also important to verify the authenticity and integrity of the running system image, and this can be accomplished with the following command:
show software authenticity runningAn example of this procedure follows:
firepower(local-mgmt)# show software authenticity running File Name : <local>/fxos-k8-fp2k-lfbff.2.4.1.216.SPA Image type : Release Signer Information Common Name : abraxas Organization Unit : FXOS Organization Name : CiscoSystems Certificate Serial Number : 5BF5D36A Hash Algorithm : SHA2 512 Signature Algorithm : 2048-bit RSA Key Version : A
The Organization Unit, Organization Name, and Certificate Serial Number values can be viewed to verify that the system image signature is valid, and the certificate serial number should be the same as the value obtained from the show software authenticity file command. In the examples above, the authenticity check of the FXOS Software image on bootflash: and the authenticity check of the running image both produce a value of 5BF5D36A.
Lastly, obtain a copy of the public keys with the following command:
show software authenticity keysfirepower(local-mgmt)# show software authenticity keys Public Key #1 Information -------------------------- Key Type : Release (Primary) Public Key Algorithm : 2048-bit RSA Modulus : B6:B9:BC:D3:C8:A1:BE:3A:B5:04:0F:21:6C:AA:AB:D6: CC:FE:7A:AD:CF:97:1B:57:FC:9A:1D:4B:5A:6D:D4:B0: 7D:DB:77:FB:3F:A4:57:1A:08:4F:C1:6E:3F:CB:BF:E0: 3C:99:9C:EE:F5:DD:3C:FC:C6:8D:98:49:29:00:B9:9B: DF:22:7E:73:83:FB:B5:78:68:4E:48:1A:5B:EE:83:81: B6:3B:2E:35:5B:C2:D0:B8:46:D6:45:13:23:21:44:DA: 36:55:F9:09:5B:B1:88:8B:9A:28:0B:DA:44:DE:D2:F8: 8B:17:CF:99:64:BE:2F:80:EF:13:6B:BC:A4:3E:DE:99: 33:EF:E8:30:56:4C:DA:D5:D3:89:55:CC:BF:A2:22:1A: B7:64:FD:14:3A:7D:4F:00:DC:86:B5:35:18:C3:F3:FC: 93:D4:BF:5E:FD:85:8C:28:4B:96:0F:B1:6D:1E:96:E7: 05:1C:39:B7:1F:C7:F9:52:47:60:9C:96:FB:00:E2:2D: D9:08:2E:3A:87:0C:4F:3E:39:77:C7:FE:AC:D7:2D:23: AA:63:EB:2A:4D:13:98:C7:6A:B4:06:F9:1E:2D:B6:F8: 10:80:EA:F4:E3:BF:C4:49:63:D0:5D:93:9F:96:54:76: BF:4D:83:7B:9D:CD:72:61:CC:EC:47:EA:91:EF:34:0B Exponent : 65537 Key Version : A Product Name : FXOS Public Key #2 Information -------------------------- Key Type : Release (Backup) Public Key Algorithm : 2048-bit RSA Modulus : B6:B9:BC:D3:C8:A1:BE:3A:B5:04:0F:21:6C:AA:AB:D6: CC:FE:7A:AD:CF:97:1B:57:FC:9A:1D:4B:5A:6D:D4:B0: 7D:DB:77:FB:3F:A4:57:1A:08:4F:C1:6E:3F:CB:BF:E0: 3C:99:9C:EE:F5:DD:3C:FC:C6:8D:98:49:29:00:B9:9B: DF:22:7E:73:83:FB:B5:78:68:4E:48:1A:5B:EE:83:81: B6:3B:2E:35:5B:C2:D0:B8:46:D6:45:13:23:21:44:DA: 36:55:F9:09:5B:B1:88:8B:9A:28:0B:DA:44:DE:D2:F8: 8B:17:CF:99:64:BE:2F:80:EF:13:6B:BC:A4:3E:DE:99: 33:EF:E8:30:56:4C:DA:D5:D3:89:55:CC:BF:A2:22:1A: B7:64:FD:14:3A:7D:4F:00:DC:86:B5:35:18:C3:F3:FC: 93:D4:BF:5E:FD:85:8C:28:4B:96:0F:B1:6D:1E:96:E7: 05:1C:39:B7:1F:C7:F9:52:47:60:9C:96:FB:00:E2:2D: D9:08:2E:3A:87:0C:4F:3E:39:77:C7:FE:AC:D7:2D:23: AA:63:EB:2A:4D:13:98:C7:6A:B4:06:F9:1E:2D:B6:F8: 10:80:EA:F4:E3:BF:C4:49:63:D0:5D:93:9F:96:54:76: BF:4D:83:7B:9D:CD:72:61:CC:EC:47:EA:91:EF:34:0B Exponent : 65537 Key Version : A Product Name : FXOS Public Key #3 Information -------------------------- Key Type : Release (FEATURE KEY STORAGE) Public Key Algorithm : 2048-bit RSA Modulus : C3:9E:B2:42:93:F2:F5:8A:E7:BA:8A:20:13:23:4A:24: 39:93:C1:9E:83:32:D5:C7:87:38:54:14:1F:BC:66:8A: 1A:F5:BA:B5:44:6A:5A:D0:B8:22:B2:3D:66:3D:34:A4: 13:DF:3C:EB:02:34:97:D3:59:37:BE:86:D1:5C:40:F8: 4B:F8:C0:7C:C8:92:0E:8F:C0:9B:49:88:8E:EE:31:B4: 86:4A:3B:D6:D9:34:9F:CB:16:5F:1C:84:47:5A:9C:07: 9A:12:F3:33:A2:EE:EB:76:8D:B3:C5:29:D2:D3:C4:ED: 47:7C:70:E0:D3:80:00:36:C5:C1:BC:B0:45:EF:78:D5: 62:02:5C:B4:35:0F:E9:D9:AD:5F:FF:F9:92:69:0C:01: 5C:19:7F:E2:FE:0F:6B:8F:58:71:DB:E1:D7:F8:43:2F: AF:C1:80:F9:84:D0:AD:CA:A3:EC:C8:C4:C7:BE:48:53: EA:D5:31:44:63:B2:F8:3D:F4:C4:66:93:76:83:20:C0: 1C:F4:B9:9A:3B:8A:FB:8A:D6:EC:9E:D8:35:B1:E1:F0: 48:16:4C:49:16:65:05:60:8E:77:B4:AA:7A:E9:3F:E7: 11:89:3E:98:4A:97:82:6E:09:18:4C:7C:8F:5B:45:89: 78:16:C2:37:8F:3E:40:AE:35:09:D2:91:E6:7F:3C:FB Exponent : 65537 Key Version : A Product Name : FXOS-CID
Submit all command output and any system images collected in this section to the relevant TAC SR and proceed to the next section of this document.
Execute the following commands from the Cisco FTD CLI prompt:
system support diagnostic-cli enable
Then calculate a hash value for the .text memory segment and retrieve a copy of it by executing the following commands:
verify /sha-512 system:memory/text copy system:memory/text ftp
An example of this procedure follows:
> system support diagnostic-cli Attaching to Diagnostic CLI ... Press 'Ctrl+a then d' to detach. Type help or '?' for a list of available commands. firepower> enable Password: firepower# verify /sha-512 system:memory/text !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! [output truncated] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!Done! verify /SHA-512 (system:memory/text) = a03a15444f0995f578e9aa6cbc8feed2a3f2dd8ac8cca919b7b2b54836ba3d4b763372f58029e66fa64aafa8eea2b79d5f0c7ea65
cde0d813aef17e436e49b85 firepower# copy system:memory/text ftp Source filename [memory/text]? Address or name of remote host []? 10.10.10.1 Destination filename [text]? system.memory.text.bin !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! INFO: No digital signature found 71921664 bytes copied in 2.60 secs (35960832 bytes/sec)
It is highly recommended that a hash value be calculated on the copied memory segment file and compared to the hash value obtained on the recipient platform to ensure no errors were introduced during the file transfer process.
The example below utilizes the sha512sum utility which is included with most Linux distributions.
root@ftp-server:~# sha512sum system.memory.text.bin a03a15444f0995f578e9aa6cbc8feed2a3f2dd8ac8cca919b7b2b54836ba3d4b763372f58029e66fa64aafa8eea2b79d5f0c7ea65cde0d813aef17e436e49b85
system.memory.text.bin root@ftp-server:~#
Note that the FTD verify command and the sha512sum utility both produce a SHA-512 hash value of a03a15444f0995f578e9aa6cbc8feed2a3f2dd8ac8cca919b7b2b54836ba3d4b763372f58029e66fa 64aafa8eea2b79d5f0c7ea65cde0d813aef17e436e49b85 for the system.memory.text.bin file.
Submit all command output (including all computed hash values) and any system images collected in this section to the relevant TAC SR, and proceed to the next section of this document.
WARNING: Executing the tasks in this section will trigger a reload of the FXOS platform. Cisco recommends performing this task during a maintenance window. Cisco does not recommend performing this task if additional forensic information needs to be collected because a reload of the device may cause the loss of information vital to a forensic investigation. Please ensure you have a copy of the original device configuration and the appropriate authorization to initiate a reload of the platform in question before proceeding.
This procedure outlines how to obtain a crashinfo file and a core dump from a Cisco FXOS device.
The crashinfo file is saved in the root of the Cisco FXOS file system by default and the core dump may be placed in the underlying FTD file system or the coredumpfsys filesystem dependent on the version of software running on the system. Storage space required may vary from several hundred megabytes to several gigabytes in size depending on device model. Be sure there is enough space on the destination flash or disk device to accommodate the crashinfo file and core dump file.
To initiate the crashinfo dump process, execute the following commands:
system support diagnostic-cli enable crashinfo force page-fault
An example of this procedure follows:
> system support diagnostic-cli Attaching to Diagnostic CLI ... Press 'Ctrl+a then d' to detach. Type help or '?' for a list of available commands. firepower> enable Password: firepower# crashinfo force page-fault WARNING: This command will force a crash and cause a reboot. Do you wish to proceed? [confirm]: :Saved Crash Process Name: lina Signal No.: 11 Thread id: 1363 Register dump from crashing thread R00: 0x0000000000000000 R01: 0x0000000000000001 R02: 0x0000002009499f80 R03: 0x0000000000000059 R04: 0x000000000000000a R05: 0xfffffffffffffffd R06: 0x0000000000000000 R07: 0x0000000000000000 R08: 0x000000ffd12f2da8 R09: 0x000000ffd12fa3a0 [output truncated] Show tech-support output is captured and saved. Crashinfo file created: /mnt/disk0/crashinfo_lina.1359.20200217.163743 Rebooting... (status 0x8b)
When the crashinfo process is complete, the Firepower platform will reboot.
Once the platform has rebooted, connect to the FTD CLI, enter expert mode, calculate a hash value for the core and crashinfo files, and copy the files off the platform by executing the following commands:
expert sudo su cd /var/data/cores sha512sum ftp or scp cd /mnt/disk0 sha512sum ftp or scp
Note: the sudo su - command must be executed after entering expert mode to ensure that the correct privileges are obtained to access the core and crashinfo files.
An example of this procedure follows:
> expert admin@firepower:~$ sudo su - We trust you have received the usual lecture from the local System Administrator. It usually boils down to these three things: #1) Respect the privacy of others. #2) Think before you type. #3) With great power comes great responsibility. Password: root@firepower:~# cd /var/data/cores root@firepower:cores# ls -la total 106788 drwxrwxrwx 4 root detection 4096 Apr 14 14:06 . drwxr-xr-x 3 root root 60 Apr 25 14:21 .. -rw-r--r-- 1 root root 109210409 Apr 14 14:07 core.lina.11.1264.1649945202.gz drwx------ 2 root root 16384 May 8 2019 lost+found drwxr-xr-x 3 root root 4096 May 8 2019 sysdebug root@firepower:cores# sha512sum core.lina.11.1264.1649945202.gz 22e866c69a7ba0f1cc44eabccdee07b7fa6721de4fc47329c51c2d8b8e70cb9459689205dada670678336b76250ad942d714381c6b4917f70b5741c817bcbb3b core.lina.11.1264.1649945202.gz root@firepower:cores# ftp 10.10.10.1 Connected to 10.10.10.1. 220 Microsoft FTP Service Name (10.10.10.1:admin): anonymous 331 Anonymous access allowed, send identity (e-mail name) as password. Password: 230 User logged in. Remote system type is Windows_NT. ftp> bin 200 Type set to I. ftp> put core.lina.11.1264.1649945202.gz local: core.lina.11.1264.1649945202.gz remote: core.lina.11.1264.1649945202.gz 229 Entering Extended Passive Mode (|||61336|) 125 Data connection already open; Transfer starting. 100% |***********************************| 104 MiB 89.22 MiB/s 00:00 ETA 226 Transfer complete. 109210409 bytes sent in 00:01 (89.13 MiB/s) ftp> quit 221 Goodbye. root@firepower:cores# cd /mnt/disk0 root@firepower:disk0# ls -la total 272 drwxr-xr-x 10 root root 4096 May 17 15:20 . drwxr-xr-x 5 root root 4096 Apr 25 14:16 .. d--------- 2 root root 6 May 8 2019 .iso_images drwxr-xr-x 2 root root 4096 May 9 05:06 .private -rw-r--r-- 1 root root 351 Apr 25 14:22 asa-cmd-server.log -rwxr-xr-x 1 root root 6442 Apr 25 14:04 backup-config.cfg d--------- 2 root root 22 May 8 2019 boot drw------- 2 root root 25 Apr 14 14:06 coredumpinfo -rw-r--r-- 1 root root 219358 Apr 14 14:06 crashinfo_lina.1264.20220414.140641 d--------- 4 root root 28 May 8 2019 csco_config lrwxrwxrwx 1 root root 93 Apr 25 14:23 cspCfg.xml -> /opt/cisco/csp/applications/configs/cspCfg_cisco-ftd.6.3.0.83__ftd_001_JMX2312Y09KWMZZP41.xml -rw-r--r-- 1 root root 0 May 7 14:06 hitcnt_del_ruleid_list drwxr-xr-x 5 root root 145 Apr 25 14:26 log -rwxr-xr-x 1 root root 5972 Apr 25 14:04 modified-config.cfg -rw-r--r-- 1 root root 345 Apr 25 14:26 npu-asa-cmd-server.log drwxr-xr-x 2 root root 6 Apr 25 14:26 packet-tracer -rwxr-xr-x 1 root root 6384 Mar 31 16:15 running-config-backup.txt drwxr-xr-x 2 root root 6 Mar 31 15:12 smart-log -rw-r--r-- 1 root root 39 Apr 25 14:26 snortpacketinfo.conf -rwxr-xr-x 1 root root 6415 Dec 20 18:38 startup-config root@firepower:disk0# sha512sum crashinfo_lina.1264.20220414.140641 dade542dfaef7d17d8bed8bc3c33554633c5e21043e5eeda72cc9c4ec7dc95989f9aa16d2a05b9c6b89976d8d1b5e0627c530b94b557dc9048eca0a7c6cb0684 crashinfo_lina.1264.20220414.140641 root@firepower:disk0# ftp 10.10.10.1 Connected to 10.10.10.1. 220 Microsoft FTP Service Name (10.10.10.1:admin): anonymous 331 Anonymous access allowed, send identity (e-mail name) as password. Password: 230 User logged in. Remote system type is Windows_NT. ftp> bin 200 Type set to I. ftp> put crashinfo_lina.1264.20220414.140641 local: crashinfo_lina.1264.20220414.140641 remote: crashinfo_lina.1264.20220414.140641 229 Entering Extended Passive Mode (|||61339|) 125 Data connection already open; Transfer starting. 100% |***********************************| 214 KiB 86.73 MiB/s 00:00 ETA 226 Transfer complete. 219358 bytes sent in 00:00 (43.61 MiB/s) ftp> quit 221 Goodbye. root@firepower:disk0#
Submit all command output, hash values, crashinfo and core files collected in this section to the relevant TAC SR and proceed to the next section of this document.
The ROM Monitor firmware of the Firepower platform is executed when the appliance is powered up or reset. The firmware initializes the platform hardware and boots the FXOS operating system software. Because the ROM Monitor settings are persistent if they have been synced to NVRAM, information about the ROM Monitor variable values could indicate an attempt to influence the FXOS boot sequence. The set command can be used while in the ROM Monitor prompt to see the value of the ROM Monitor variables.
ROM Monitor mode is accessed by rebooting the Firepower appliance and pressing the BREAK or ESC key during the reload process when prompted as depicted in the example below.
firepower# connect local-mgmt firepower(local-mgmt)# reboot Before rebooting, please take a configuration backup. Do you still want to reboot? (yes/no):yes Broadcast message from admin@firepower (Fri Aug 30 14:17:09 2019): All shells being terminated due to system /sbin/reboot Threat Defense System: CMD=-stop, CSP-ID=cisco-ftd.6.3.0.83__ftd_001_JMX2312Y09KWMZZP41, FLAG='' Cisco FTD stopping ... Stopping Cisco Firepower 2110 Threat Defense......ok Skipping sfifd for this platform... Stopping nscd... Stopping nscd... [ OK ] Turning off swapfile /ngfw/Volume/.swaptwo Stopping system log daemon... Stopping system log daemon... [ OK ] Stopping Threat Defense ... Stopping Threat Defense ... [ OK ] Cisco FTD stopped successfully. Stopping all devices. [output truncated] Rebooting... [63349.896618] reboot: Restarting system ****************************************************************************** Cisco System ROMMON, Version 1.0.06, RELEASE SOFTWARE Copyright (c) 1994-2017 by Cisco Systems, Inc. Compiled Wed 11/01/2017 18:38:59.66 by builder ****************************************************************************** Current image running: Boot ROM0 Last reset cause: ResetRequest DIMM_1/1 : Present DIMM_2/1 : Absent Platform FPR-2110 with 16384 MBytes of main memory BIOS has been successfully locked !! MAC Address: 00:fd:22:61:8b:f1 Use BREAK or ESC to interrupt boot. Use SPACE to begin boot immediately. Boot interrupted. rommon 1 >
The following example shows the output of the ROM Monitor set command on a Cisco Firepower platform:
rommon 1 > set ADDRESS= NETMASK= GATEWAY= SERVER= IMAGE= CONFIG= PS1="rommon ! > "
The example above depicts a platform where the ROM Monitor values are at their default values and have not been altered.
To return the Firepower platform to normal operation, simply issue the boot command at the ROM Monitor prompt as depicted in the following example.
rommon 2 > boot Located '.boot_string' @ cluster 837007. # Located 'installables/switch/fxos-k8-fp2k-lfbff.2.4.1.216.SPA' @ cluster 598122. ######################################################################################################################
###################################### [output truncated]
Submit all command output obtained in this section to the relevant TAC SR.
Additional information about Cisco Software Integrity Assurance, as well as forensic investigation procedures for other platforms, can be found at the following link:
Cisco Security Tactical Resources
https://sec.cloudapps.cisco.com/security/center/tacticalresources.x
Step 1 Create the Firepower Device Problem Description
Device Problem Description uploaded to SR
Step 2 Document the FTD Runtime Environment
Output of show tech-support uploaded to SR
Output of dir all-filesystems uploaded to SR
Output of other show commands uploaded to SR (Optional)
Step 3 Verify FTD System File Integrity
Output of find /ngfw/var/sf/.icdb/* and hashes uploaded to SR
Output of which command executed on shell scripts uploaded to SR
Shell scripts, certificates, log file, and hash database added to .tar file
.tar file and its associated hash value uploaded to SR
Step 4 FXOS Digitally Signed Image Authenticity Verification
Output of show software authenticity file uploaded to SR
Output of show software authenticity running uploaded to SR
Output of show software authenticity keys uploaded to SR
Step 5 Verify Memory .text Segment Integrity
Output of verify on memory text segment uploaded to SR
Copy of memory text segment uploaded to SR
Step 6 FTD Crashinfo File/Core File
Output of crashinfo uploaded to SR
crashinfo file uploaded to SR
Core file uploaded to SR
Hash values of crashinfo and core files uploaded to SR
Step 7 Firepower ROM Monitor Variable Check
Output of set command uploaded to SR
| Version | Date | Author | Comments |
|---|---|---|---|
| 1.0 | 5/14/2020 | Dan Maunz, Jason Barnes | Initial public release. |
| 1.1 | 5/17/2022 | Dan Maunz | Updated and validated for version 7.1.0. |
| 1.2 | 5/3/2023 | Dan Maunz | Validated procedures on Release 7.3.1. |
| 1.3 | 8/27/2024 | Dan Maunz | Added retirement notice. |
This document is part of the Cisco Security portal. Cisco provides the official information contained on the Cisco Security portal in English only.
This document is provided on an “as is” basis and does not imply any kind of guarantee or warranty, including the warranties of merchantability or fitness for a particular use. Your use of the information in the document or materials linked from the document is at your own risk. Cisco reserves the right to change or update this document without notice at any time.