Step 1 - Create the Firepower Device Problem Description
Step 2 - Document the Firepower Runtime Environment
Step 3 - Verify the Integrity of FTD System Files
Step 4 - Verify Digitally Signed Image Authenticity
Step 5 - Verify FTD Memory .text Segment Integrity
Step 6 - Obtain Firepower crashinfo/core File
Step 7 - Check ROM Monitor Settings
Firepower 1000/2100 Series Forensic Data Collection Checklist
Appendix A - Firepower 1000/2100 Series Platform That Is Running ASA Software
This document provides steps to collect forensic information from Cisco Firepower 1000/2100 Series appliances that are running Cisco Firepower eXtensible Operating System (FXOS) Software when compromise or tampering is suspected. It outlines a number of commands that can be run to gather evidence for an investigation along with the respective output that should be collected upon running these commands. This document also provides information on how to perform integrity checks on Cisco FXOS system and Cisco Firepower Threat Defense (FTD) application images and includes a procedure for collecting a memory dump, a crashinfo file, and a core file from a Cisco Firepower device.
Note: DO NOT REBOOT THE DEVICE. Rebooting a device during the initial stage of an assessment will irrecoverably lose all volatile information that is contained within the device (e.g., RAM contents, Address Resolution Protocol [ARP] and routing tables, Network Address Translations [NATs], access control list [ACL] hit and drop counts).
Note: Cisco highly recommends isolating a device that is suspected of tampering or compromise from the network before conducting an initial forensic examination. This may prevent remote unloading of any implants or malware that are installed on the device and will prevent an attacker from monitoring commands that are entered on the device under investigation.
If you require assistance or have questions regarding the following procedures, contact the Cisco Product Security Incident Response Team (PSIRT).
This document contains seven sections:
1. Create the Firepower Device Problem Description - Describe why the platform is a candidate for forensic examination.
2. Document the Firepower Runtime Environment - Collect platform configuration and runtime state.
3. Verify the Integrity of FTD System Files - Examine system file image hashes for inconsistencies.
4. Verify Digitally Signed Image Authenticity - Examine the Cisco FXOS operating system for proper signing characteristics.
5. Verify FTD .text Memory Segment Integrity - Retrieve and calculate a hash of the Cisco FTD .text segment.
6. Firepower crashinfo/core File - Obtain crashinfo and core files from the running Cisco FTD application.
7. Check ROM Monitor Settings - Examine ROM monitor settings for remote system image loading.
The procedures that are outlined in this document assume that the reader has a basic understanding of Cisco FXOS Software, Cisco FTD Software, and Linux command syntax.
A valid cisco.com account is required to view individual Cisco FXOS Software and FTD Software file hashes for software file integrity checking. For customers without a cisco.com account, a publicly available comprehensive list of file hashes (Bulk Hash File) can be downloaded from https://www.cisco.com/c/en/us/about/trust-center/downloads.html.
Note: The examples that are used in this document are based on Cisco FXOS Software Release 2.12.0 and Cisco FTD Software Release 7.2.5 command syntax. The output that is produced by a command may vary depending on the software release that is deployed and/or the features that are supported or configured on the device. Not all commands that are used in these procedures may be supported on earlier releases of the software.
Note: Collection steps 2 through 7 contain examples of the commands that should be executed along with the expected output toward the end of each step.
Describe in as much detail as possible why the device is a candidate for forensic examination. Are there configuration changes that cannot be explained? Is there unusual traffic originating from or terminating on the device? Are there anomalous entries in the device logs or in syslog messages? Is the device exhibiting odd behavior that cannot be attributed to a misconfiguration or a software/hardware defect? Are any typical device administration commands now returning unusual output or no output at all?
Use the Cisco Software Checker to search for Cisco Security Advisories that apply to specific releases of Cisco Adaptive Security Appliance (ASA) Software, Firepower Management Center (FMC) Software, FTD Software, FXOS Software, IOS Software, IOS XE Software, NX-OS Software, and NX-OS Software in ACI Mode.
Record any results that are returned by the tool that may explain the anomalous behavior that is being observed. It is considered a best practice to keep software up to date to take advantage of the latest security fixes and enhancements.
Note: This tool does not provide information about Cisco IOS XR Software or interim software builds. Also note that for Cisco ASA Software, FMC Software, FTD Software, and FXOS Software, the tool contains only vulnerability information for Cisco Security Advisories first published from January 2022 onward, and for NX-OS Software and NX-OS Software in ACI Mode from July 2019 onward.
Submit the problem description and any relevant results that were obtained from the Cisco Software Checker to the relevant TAC SR and proceed to the next section of this document.
The initial stage of forensic information gathering is completed by using show tech-support commands in the Cisco FXOS and Firepower CLIs, and some of the output may vary depending on the Cisco FTD Software release and/or features that are supported or configured on the device.
Note: For Cisco Firepower 1000/2100 Series appliances, the default command shell will vary depending on the method that is used to access the platform. If SSH is used, the user will be placed into the Cisco FTD CLI. If a console connection is used, the user will be placed into the Cisco FXOS CLI. Use the connect ftd command to access the Cisco FTD CLI from the FXOS CLI and the connect fxos command to access the Cisco FXOS CLI from the FTD CLI.
Execute the following commands from the Cisco FXOS CLI prompt:
connect local-mgmt show tech-support fprm detail copy workspace:/techsupport/<file_name> <destination>
Note: The copy command supports the FTP, HTTP, HTTPS, SCP, SFTP, and TFTP protocols.
Next, connect to the Cisco Firepower module CLI and collect the output of show tech-support detail by using the following series of commands:
connect ftd system support diagnostic-cli enable terminal pager 0 show tech-support detail exit exit
The output of show tech-support detail may also be redirected to a file server using the FTP, SCP, SMB, or TFTP protocols. The following example depicts the required syntax using the FTP protocol:
show tech-support file ftp://10.10.10.1/tech-support-details.txt detail
An example of this entire procedure follows:
firepower# connect local-mgmt firepower(local-mgmt)# show tech-support fprm detail firepower_FPRM The showtechsupport file will be located at workspace:/techsupport/ 20240118150424_firepower_FPRM.tar.gz Initiating tech-support information task on FABRIC A ... WARNING: *** /mnt/disk0/smart-log/ is missing *** WARNING: *** /tmp/sed_env_info.xml is missing *** WARNING: *** /tmp/softraid_env.xml is missing *** WARNING: *** /tmp/nvme_build.log is missing *** WARNING: *** /tmp/sed_build.log is missing *** WARNING: *** /tmp/sr_build.log is missing *** WARNING: *** /tmp/vpath-cfg-cache is missing *** Completed initiating tech-support subsystem tasks (Total: 0) firepower(local-mgmt)# firepower(local-mgmt)# copy workspace:/techsupport/ 20240118150424_firepower_FPRM.tar.gz ftp://anonymous@10.10.10.1/ Connected to 10.10.10.1. 220 Microsoft FTP Service 331 User name ok, need password Password: 230 User logged in Remote system type is UNIX. Using binary mode to transfer files. 200 Type set to I. local: /opt/cisco/csp/workspace/techsupport/20240118150424_firepower_FPRM.tar.gz remote:
/20240118150424_firepower_FPRM.tar.gz 227 Entering passive mode (10,10,10,1,222,195) 125 Using existing data connection 100% |***********************************| 6649 KiB 60.66 MiB/s 00:00 ETA 226 Closing data connection; File transfer successful. 6808707 bytes sent in 00:00 (43.99 MiB/s) 221 Service closing control connection firepower(local-mgmt)# connect ftd > system support diagnostic-cli Attaching to Diagnostic CLI ... Press 'Ctrl+a then d' to detach. Type help or '?' for a list of available commands. firepower> enable Password: firepower# terminal pager 0 firepower# show tech-support file ftp://10.10.10.1/tech-support-details.txt detail !!! firepower# exit Logoff User enable_1 logged in to firepower Logins over the last 1 days: 2. Last login: 13:10:25 UTC Feb 16 2024 from console Failed logins since the last login: 0. Type help or '?' for a list of available commands. firepower> exit Console connection detached. >
Submit all command output that was collected in this section to the relevant TAC SR and proceed to the next section of this document.
Connect to the Cisco FTD CLI. This can be accomplished from the Cisco FXOS CLI by using the connect ftd command.
From the Cisco FTD CLI, use the following commands to assume root permissions, run the system file integrity checks, and collect the necessary files for forensic assessment.
Access expert mode and sudo to the root account:
expert sudo su -
Run the integrity checks:
find /ngfw/var/sf/.icdb/* -name *.icdb.RELEASE.tar | xargs sha512sum cat /proc/*/smaps > /tmp/all-process-smaps.txt verify_file_integ.sh -f
Note: The Linux which command can be used to quickly locate the shell scripts in the next step. See the following example for more details.
Locate and retrieve copies of the following files:
verify_file_integ.sh verify_signed_db.sh db_manage.sh /ngfw/etc/certs/*.crt /ngfw/var/log/sf/verify_file_integ.log /ngfw/var/tmp/merged-db/master.db
Create an archive of the preceding list of files and copy the archive off the platform.
tar -cvf SR-<sr_number>.tar sha512sum SR-<sr_number>.tar ftp or scp
An example of this procedure follows:
firepower# connect ftd > expert admin@firepower:~$ sudo su - We trust you have received the usual lecture from the local System Administrator. It usually boils down to these three things: #1) Respect the privacy of others. #2) Think before you type. #3) With great power comes great responsibility. Password: root@firepower:~# find /ngfw/var/sf/.icdb/* -name *.icdb.RELEASE.tar | xargs sha512sum 480fd0602e527c156880543b7f0e8f87428f54c0692d217829633846316fc07a11d8ea1a90c6eeb9b80176c0a26223fee0390d02b72bc4d74aa360e8ba976077
/ngfw/var/sf/.icdb/lsp/lsp.icdb.RELEASE.tar a4e48d6505e724d73b4037632988d5b24172afec1eeba2e37c986d496017563ff1ab012a787c65b61b17b9e50d569124469e6b61fa55078f2abb23673bf0e491
/ngfw/var/sf/.icdb/vdb/vdb.icdb.RELEASE.tar ec1b5a2ac31a05a85c57880ec8fe781b2f449020454ccfe2eaada99aff4a4f5e5fde0ec9f43ea92be98425be13f034175ee77ddca3a712daa121994455411088
/ngfw/var/sf/.icdb/vdb/vdb.navl.icdb.RELEASE.tar # # Note: the file names, number of files, and hash values may vary dependent on
# the version of software running on the appliance and whether any software
# updates have been applied. It is important that the output (including file
# names and hashes) generated by the command above be submitted to the TAC SR. # root@firepower:~# cat /proc/*/smaps > /tmp/all-process-smaps.txt root@firepower:~# verify_file_integ.sh -f Running file integrity checks... Successfully verified file integrity # # Identify the location of the system integrity scripts with the which
# command: # root@firepower:~# which verify_file_integ.sh /ngfw/usr/local/sf/bin/verify_file_integ.sh root@firepower:~# which verify_signed_db.sh /ngfw/usr/local/sf/bin/verify_signed_db.sh root@firepower:~# which db_manage.sh /ngfw/usr/local/sf/bin/db_manage.sh # # Archive a copy of these scripts, along with all certificates found in
# /ngfw/etc/certs/, and the files /ngfw/var/log/sf/verify_file_integ.log and # /ngfw/var/tmp/merged-db/master.db # root@firepower:~# tar -cvf SR-1234567890.tar /ngfw/usr/local/sf/bin/verify_file_integ.sh /ngfw/usr/local/sf/bin/verify_signed_db.sh /ngfw/usr/local/sf/bin/db_manage.sh /ngfw/etc/certs/*.crt /ngfw/var/log/sf/verify_file_integ.log /ngfw/var/tmp/merged-db/master.db /tmp/all-process-smaps.txt tar: Removing leading `/' from member names /ngfw/usr/local/sf/bin/verify_file_integ.sh /ngfw/usr/local/sf/bin/verify_signed_db.sh /ngfw/usr/local/sf/bin/db_manage.sh /ngfw/etc/certs/SRU_rel.crt /ngfw/etc/certs/rel.crt /ngfw/var/log/sf/verify_file_integ.log /ngfw/var/tmp/merged-db/master.db /ngfw/etc/certs/rewhich db_manage.sh /tmp/all-process-smaps.txt # # Create a hash of the tar file: # root@firepower:~# sha512sum SR-1234567890.tar 8833409f7affbae6ab19cdd8ca11153252f1176e147b3d4c373a7195e261da5ca72a3260a5aa1a5f991b12c1f6ac070d4ece31765e413f0e5e4afd6c95cf6ea5
SR-1234567890.tar root@firepower:~# # # Copy the tar file off the platform using FTP or SCP: # root@firepower:0.1 Connected to 10.10.10.1. 220 Microsoft FTP Service Name (10.10.10.1:admin): anonymous 331 Anonymous access allowed, send identity (e-mail name) as password. Password: 230 User logged in. Remote system type is Windows_NT. ftp> bin 200 Type set to I. ftp> put SR-1234567890.tar local: SR-1234567890.tar remote: SR-1234567890.tar 229 Entering Extended Passive Mode (|||59418|) 125 Data connection already open; Transfer starting. 100% |***********************************| 1990 KiB 104.76 MiB/s 00:00 ETA 226 Transfer complete. 2037760 bytes sent in 00:00 (100.48 MiB/s) ftp> quit 221 Goodbye.
Submit all command output and the files that were gathered in this step to the relevant TAC SR and proceed to the next section of this document.
Cisco FXOS Software implements digitally signed system images on most platforms. Digitally signed Cisco FXOS Software uses asymmetric (public key) cryptography, which increases the security posture of Cisco Firepower devices by ensuring that the system image has not been altered.
Certain platforms that are running FXOS software, such as Cisco Firepower 1000 Series platforms, also support Cisco Secure Boot technologies. Cisco Secure Boot is a secure startup process that a Cisco device performs each time it boots up. Beginning with the initial power-on, a special-purpose hardware device known as the Trust Anchor module verifies the integrity of the ROM monitor code and the Cisco FXOS image using digital signatures as they are each loaded. If any failures are detected, the user is notified of the error, and the device will wait for the operator to correct the error. This prevents the network device from executing tainted network software.
For additional information, see Trust Anchor Technology.
Note: The show software authenticity set of commands are only supported on Cisco Firepower platforms that incorporate Cisco Secure Boot technologies.
The authenticity and integrity of a system image file can be verified by using the following commands from the Cisco FXOS CLI:
connect local-mgmt cd bootflash:/ show file .boot_string show software authenticity file <path/filename> verify signature <path/filename> show software authenticity running show software authenticity keys
An example of this procedure follows:
firepower# connect local-mgmt firepower(local-mgmt)# cd bootflash:/ firepower(local-mgmt)# show file .boot_string disk0:installables/switch/fxos-k8-fp1k-lfbff.2.12.0.530.SPA firepower(local-mgmt)# show software authenticity file /installables/switch/fxos-k8-fp1k-lfbff.2.12.0.530.SPA : File /mnt/boot//installables/switch/fxos-k8-fp1k-lfbff.2.12.0.530.SPA size 199156688 File Name : <local>/fxos-k8-fp1k-lfbff.2.12.0.530.SPA Image type : Release Signer Information Common Name : abraxas Organization Unit : FXOS Organization Name : CiscoSystems Certificate Serial Number : 65313942 Hash Algorithm : SHA2 512 Signature Algorithm : 2048-bit RSA Key Version : A
The Organization Unit, Organization Name, and Certificate Serial Number values (highlighted in the preceding output) can be viewed to verify that the system image signature is valid.
Next, calculate a hash for the Cisco FXOS system image and verify the digital signature by using the following command:
verify signature <path/filename>
An example of this procedure follows:
firepower(local-mgmt)# verify signature /installables/switch/fxos-k8-fp1k-lfbff.2.12.0.530.SPA : File /mnt/boot//installables/switch/fxos-k8-fp1k-lfbff.2.12.0.530.SPA size 199156688 Done! Computed Hash SHA2: e94ca6457ed9c0411a42cfa1a747d438 033c79ca4309aa8a556aa7efb5e98801 8f4be4bffdcbcdc9dfbd184eb6bbb5ac af0e6f76bc534c709233e8cb0e757fc3 Embedded Hash SHA2: e94ca6457ed9c0411a42cfa1a747d438 033c79ca4309aa8a556aa7efb5e98801 8f4be4bffdcbcdc9dfbd184eb6bbb5ac af0e6f76bc534c709233e8cb0e757fc3 The digital signature of the file: fxos-k8-fp1k-lfbff.2.12.0.530.SPA verified successfully
It is also important to verify the authenticity and integrity of the running system image, and this can be accomplished with the following command:
show software authenticity running
An example of this procedure follows:
firepower(local-mgmt)# show software authenticity running File Name : <local>/fxos-k8-fp1k-lfbff.2.12.0.530.SPA Image type : Release Signer Information Common Name : abraxas Organization Unit : FXOS Organization Name : CiscoSystems Certificate Serial Number : 65313942 Hash Algorithm : SHA2 512 Signature Algorithm : 2048-bit RSA Key Version : A
The Organization Unit, Organization Name, and Certificate Serial Number values (highlighted in the preceding output) can be viewed to verify that the system image signature is valid, and the certificate serial number should be the same as the value obtained from the show software authenticity file command. In the preceding examples, the authenticity checks of the Cisco FXOS Software image on bootflash: and the authenticity check of the running image both produce a value of 65313942.
Lastly, obtain a copy of the public keys with the following command:
show software authenticity keys
An example of this procedure follows:
firepower(local-mgmt)# show software authenticity keys Public Key #1 Information -------------------------- Key Type : Release (Primary) Public Key Algorithm : 2048-bit RSA Modulus : B6:B9:BC:D3:C8:A1:BE:3A:B5:04:0F:21:6C:AA:AB:D6: CC:FE:7A:AD:CF:97:1B:57:FC:9A:1D:4B:5A:6D:D4:B0: 7D:DB:77:FB:3F:A4:57:1A:08:4F:C1:6E:3F:CB:BF:E0: 3C:99:9C:EE:F5:DD:3C:FC:C6:8D:98:49:29:00:B9:9B: DF:22:7E:73:83:FB:B5:78:68:4E:48:1A:5B:EE:83:81: B6:3B:2E:35:5B:C2:D0:B8:46:D6:45:13:23:21:44:DA: 36:55:F9:09:5B:B1:88:8B:9A:28:0B:DA:44:DE:D2:F8: 8B:17:CF:99:64:BE:2F:80:EF:13:6B:BC:A4:3E:DE:99: 33:EF:E8:30:56:4C:DA:D5:D3:89:55:CC:BF:A2:22:1A: B7:64:FD:14:3A:7D:4F:00:DC:86:B5:35:18:C3:F3:FC: 93:D4:BF:5E:FD:85:8C:28:4B:96:0F:B1:6D:1E:96:E7: 05:1C:39:B7:1F:C7:F9:52:47:60:9C:96:FB:00:E2:2D: D9:08:2E:3A:87:0C:4F:3E:39:77:C7:FE:AC:D7:2D:23: AA:63:EB:2A:4D:13:98:C7:6A:B4:06:F9:1E:2D:B6:F8: 10:80:EA:F4:E3:BF:C4:49:63:D0:5D:93:9F:96:54:76: BF:4D:83:7B:9D:CD:72:61:CC:EC:47:EA:91:EF:34:0B Exponent : 65537 Key Version : A Product Name : FXOS Public Key #2 Information -------------------------- Key Type : Release (Backup) Public Key Algorithm : 2048-bit RSA Modulus : B6:B9:BC:D3:C8:A1:BE:3A:B5:04:0F:21:6C:AA:AB:D6: CC:FE:7A:AD:CF:97:1B:57:FC:9A:1D:4B:5A:6D:D4:B0: 7D:DB:77:FB:3F:A4:57:1A:08:4F:C1:6E:3F:CB:BF:E0: 3C:99:9C:EE:F5:DD:3C:FC:C6:8D:98:49:29:00:B9:9B: DF:22:7E:73:83:FB:B5:78:68:4E:48:1A:5B:EE:83:81: B6:3B:2E:35:5B:C2:D0:B8:46:D6:45:13:23:21:44:DA: 36:55:F9:09:5B:B1:88:8B:9A:28:0B:DA:44:DE:D2:F8: 8B:17:CF:99:64:BE:2F:80:EF:13:6B:BC:A4:3E:DE:99: 33:EF:E8:30:56:4C:DA:D5:D3:89:55:CC:BF:A2:22:1A: B7:64:FD:14:3A:7D:4F:00:DC:86:B5:35:18:C3:F3:FC: 93:D4:BF:5E:FD:85:8C:28:4B:96:0F:B1:6D:1E:96:E7: 05:1C:39:B7:1F:C7:F9:52:47:60:9C:96:FB:00:E2:2D: D9:08:2E:3A:87:0C:4F:3E:39:77:C7:FE:AC:D7:2D:23: AA:63:EB:2A:4D:13:98:C7:6A:B4:06:F9:1E:2D:B6:F8: 10:80:EA:F4:E3:BF:C4:49:63:D0:5D:93:9F:96:54:76: BF:4D:83:7B:9D:CD:72:61:CC:EC:47:EA:91:EF:34:0B Exponent : 65537 Key Version : A Product Name : FXOS Public Key #3 Information -------------------------- Key Type : Release (FEATURE KEY STORAGE) Public Key Algorithm : 2048-bit RSA Modulus : C3:9E:B2:42:93:F2:F5:8A:E7:BA:8A:20:13:23:4A:24: 39:93:C1:9E:83:32:D5:C7:87:38:54:14:1F:BC:66:8A: 1A:F5:BA:B5:44:6A:5A:D0:B8:22:B2:3D:66:3D:34:A4: 13:DF:3C:EB:02:34:97:D3:59:37:BE:86:D1:5C:40:F8: 4B:F8:C0:7C:C8:92:0E:8F:C0:9B:49:88:8E:EE:31:B4: 86:4A:3B:D6:D9:34:9F:CB:16:5F:1C:84:47:5A:9C:07: 9A:12:F3:33:A2:EE:EB:76:8D:B3:C5:29:D2:D3:C4:ED: 47:7C:70:E0:D3:80:00:36:C5:C1:BC:B0:45:EF:78:D5: 62:02:5C:B4:35:0F:E9:D9:AD:5F:FF:F9:92:69:0C:01: 5C:19:7F:E2:FE:0F:6B:8F:58:71:DB:E1:D7:F8:43:2F: AF:C1:80:F9:84:D0:AD:CA:A3:EC:C8:C4:C7:BE:48:53: EA:D5:31:44:63:B2:F8:3D:F4:C4:66:93:76:83:20:C0: 1C:F4:B9:9A:3B:8A:FB:8A:D6:EC:9E:D8:35:B1:E1:F0: 48:16:4C:49:16:65:05:60:8E:77:B4:AA:7A:E9:3F:E7: 11:89:3E:98:4A:97:82:6E:09:18:4C:7C:8F:5B:45:89: 78:16:C2:37:8F:3E:40:AE:35:09:D2:91:E6:7F:3C:FB Exponent : 65537 Key Version : A Product Name : FXOS-CID
Submit all command output and any system images that were collected in this section to the relevant TAC SR and proceed to the next section of this document.
Execute the following commands from the Cisco FTD CLI prompt:
system support diagnostic-cli enable
Then calculate a hash value for the .text memory segment and retrieve a copy of it by executing the following commands:
verify /sha-512 system:memory/text copy system:memory/text ftp
An example of this procedure follows:
> system support diagnostic-cli Attaching to Diagnostic CLI ... Press 'Ctrl+a then d' to detach. Type help or '?' for a list of available commands. firepower> enable Password: firepower# verify /sha-512 system:memory/text !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! [output truncated] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!Done! verify /SHA-512 (system:memory/text) = 2289796c12ee7d909ccac135e1b2075bc54d5c7a38300265dda95d5eb66
5a51b3f45b4fcdc8c17b69c7baa735972aac1a62d2a79530daad67d2aac3dec043b37 firepower# copy system:memory/text ftp Source filename [memory/text]? Address or name of remote host []? 10.10.10.1 Destination filename [text]? system.memory.text.bin !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! INFO: No digital signature found 74031104 bytes copied in 1.990 secs (74031104 bytes/sec)
Cisco highly recommends calculating a hash value on the copied memory segment file and comparing it to the hash value that was obtained on the recipient platform to ensure no errors were introduced during the file transfer process.
The following example utilizes the sha512sum utility, which is included with most Linux distributions:
root@ftp-server:~# sha512sum system.memory.text.bin 2289796c12ee7d909ccac135e1b2075bc54d5c7a38300265dda95d5eb665a51b3f45b4fcdc8c17b69c7baa735972aac1a62d2a79530daad
67d2aac3dec043b37 system.memory.text.bin root@ftp-server:~#
Note that the Cisco FTD verify command and the sha512sum utility both produce a SHA-512 hash value of 2289796c12ee7d909ccac135e1b2075bc54d5c7a38300265dda95d5eb665a51b3f45b4fcdc8c17 b69c7baa735972aac1a62d2a79530daad67d2aac3dec043b37 for the system.memory.text.bin file.
Submit all command output (including all computed hash values) and any system images that were collected in this section to the relevant TAC SR and proceed to the next section of this document.
WARNING: Executing the tasks in this section will trigger a reload of the FXOS platform.
Cisco recommends performing this task during a maintenance window. Cisco does not recommend performing this task if additional forensic information needs to be collected as a reload of the device may cause the loss of information that is vital to a forensic investigation. Customers should ensure that they have a copy of the original device configuration and the appropriate authorization to initiate a reload of the platform in question before proceeding with this procedure.
This procedure outlines how to obtain a crashinfo file and a core dump from a Cisco FXOS device.
The crashinfo file is saved in the root of the Cisco FXOS file system by default, and the core dump may be placed in the underlying FTD file system or the coredumpfsys filesystem depending on the software release that is running on the system. The storage space that is required may vary from several hundred megabytes to several gigabytes, depending on the device model. Please ensure that there is enough space on the destination flash or disk file system to accommodate the crashinfo file and core dump file.
To initiate the crashinfo dump process, execute the following commands from the Cisco FTD CLI:
system support diagnostic-cli enable crashinfo force page-fault
An example of this procedure follows:
> system support diagnostic-cli Attaching to Diagnostic CLI ... Press 'Ctrl+a then d' to detach. Type help or '?' for a list of available commands. firepower> enable Password: firepower# crashinfo force page-fault WARNING: This command will force a crash and cause a reboot. Do you wish to proceed? [confirm]: :Saved Crash Process Name: lina Signal No.: 11 Thread id: 1363 Register dump from crashing thread R00: 0x0000000000000000 R01: 0x0000000000000001 R02: 0x0000002009499f80 R03: 0x0000000000000059 R04: 0x000000000000000a R05: 0xfffffffffffffffd R06: 0x0000000000000000 R07: 0x0000000000000000 R08: 0x000000ffd12f2da8 R09: 0x000000ffd12fa3a0 [output truncated] Show tech-support output is captured and saved. Crashinfo file created: /mnt/disk0/crashinfo_lina.1359.20200217.163743 Rebooting... (status 0x8b)
When the crashinfo process is complete, the Cisco Firepower platform will reboot.
Once the platform has rebooted, connect to the Cisco FTD CLI, enter expert mode, calculate a hash value for the core and crashinfo files, and copy the files off the platform by executing the following commands:
expert sudo su cd /var/data/cores sha512sum cd /mnt/disk0 sha512sum ftp or scp exit
Note: The sudo su - command must be executed after entering expert mode to ensure the correct privileges are obtained to copy the core file from one disk partition to another.
An example of this procedure follows:
> expert admin@firepower:~$ sudo su - We trust you have received the usual lecture from the local System Administrator. It usually boils down to these three things: #1) Respect the privacy of others. #2) Think before you type. #3) With great power comes great responsibility. Password: root@firepower:~# cd /var/data/cores root@firepower:cores# ls -l | grep -i core -rw-r--r-- 1 root root 105519353 Jan 18 19:39 core.lina.11.21374.1705606756.gz root@firepower:cores# sha512sum core.lina.11.21374.1705606756.gz b69d37ace5fd07fff3aba0c1a918eaf9e1c0b206188014d738c7da8df571c56de31fcbf1e084d5e5a494cea481052c84330c0a847c6d64eb37898d12414b50e0
core.lina.11.21374.1705606756.gz root@firepower:cores# mv core.lina.11.21374.1705606756.gz /mnt/disk0/. root@firepower:cores# cd /mnt/disk0 root@firepower:disk0# ls -l | grep -i crash -rwxr-xr-x 1 root root 687725 Jan 18 19:39 crashinfo_20240118_193916_UTC root@firepower:disk0# sha512sum crashinfo_20240118_193916_UTC ad52c5ee166ca9c0c6830d914bd0dcaffa331efe1b9d722e13b9e2442015c647c8744d0c5d6bb2c79716f882637211ebdd991c9d930fd5abf17536c5117a795c
crashinfo_20240118_193916_UTC root@firepower:disk0# exit logout admin@firepower:~$ exit logout > system support diagnostic-cli Attaching to Diagnostic CLI ... Press 'Ctrl+a then d' to detach. Type help or '?' for a list of available commands. firepower> enable Password: firepower# copy core.lina.11.21374.1705606756.gz ftp: Source filename [core.lina.11.21374.1705606756.gz]? Address or name of remote host []? 10.10.10.1 Destination filename [core.lina.11.21374.1705606756.gz]? !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! 105519353 bytes copied in 2.880 secs (52759676 bytes/sec) firepower# copy crashinfo_20240118_193916_UTC ftp: Source filename [crashinfo_20240118_193916_UTC]? Address or name of remote host []? 10.10.10.1 Destination filename [crashinfo_20240118_193916_UTC]? ! 687725 bytes copied in 0.130 secs firepower# exit Logoff User enable_1 logged in to firepower Logins over the last 1 days: 3. Last login: 20:51:38 UTC Jan 18 2024 from console Failed logins since the last login: 0. Type help or '?' for a list of available commands. firepower> exit Console connection detached. >
Submit all command output, hash values, and crashinfo and core files that were collected in this section to the relevant TAC SR and proceed to the next section of this document.
The ROM monitor firmware of the Cisco Firepower platform is executed when the appliance is powered up or reset. The firmware initializes the platform hardware and boots the Cisco FXOS operating system software. Because the ROM monitor settings are persistent if they have been synced to NVRAM, information about the ROM monitor variable values could indicate an attempt to influence the FXOS boot sequence. The set command can be used while in the ROM monitor prompt to see the value of the ROM monitor variables.
Note: This procedure must be executed from the appliance console.
Access ROM monitor mode by rebooting the Cisco Firepower appliance and pressing the Break or Esc key during the reload process when prompted, as shown in the following example:
firepower# connect local-mgmt firepower(local-mgmt)# reboot Before rebooting, please take a configuration backup. Do you still want to reboot? (yes/no):yes Broadcast message from admin@firepower (Thu Jan 18 21:12:11 2024): All shells being terminated due to system /sbin/reboot Broadcast message from admin@firepower (Thu Jan 18 21:12:12 2024): Reboot requested by the user. <13>Jan 18 21:12:13 admin: FXOS shutdown log started: pid = 29785 cmdline = /bin/sh/sbin/fxos_log_shutdown #### [output truncated] Rebooting... [ 5382.871024] reboot: Restarting system ****************************************************************************** Cisco System ROMMON, Version 1.0.18, RELEASE SOFTWARE Copyright (c) 1994-2023 by Cisco Systems, Inc. Compiled Thu 03/23/2023 16:16:11.64 by builder ****************************************************************************** Current image running: Boot ROM1 Last reset cause: ResetRequest (0x00001000) DIMM0 : Present Platform FPR-1120 with 16384 MBytes of main memory Detected Nic devid(0) 15398086 bus: 3 dev: 0 func: 0 BIOS has been successfully locked !! MAC Address: 6c:03:b5:27:e8:00 Use BREAK or ESC to interrupt boot. Use SPACE to begin boot immediately. Boot interrupted. rommon 1 >
The following example shows the output of the ROM monitor set command on a Cisco Firepower platform:
rommon 1 > set ADDRESS= NETMASK= GATEWAY= SERVER= IMAGE= CONFIG= PS1="rommon ! > "
The preceding example depicts a platform where the ROM monitor values are at their default values and have not been altered.
To return the Cisco Firepower platform to normal operation, use the boot command at the ROM monitor prompt, as shown in the following example:
rommon 2 > boot Located .boot_string Image size 60 inode num 15, bks cnt 1 blk size 8*512 Located installables/switch/fxos-k8-fp1k-lfbff.2.12.0.530.SPA Image size 199156688 inode num 65163, bks cnt 48623 blk size 8*512 ##############################################################################################################
############################################## [output truncated]
Submit all command output that was obtained in this section to the relevant TAC SR.
For additional information about Cisco Software Integrity Assurance, as well as forensic investigation procedures for other platforms, see the following link:
Cisco Security Tactical Resources
https://tools.cisco.com/security/center/tacticalresources.x
Step 1 Create the Firepower Device Problem Description
Device problem description uploaded to SR
Step 2 Document FTD Runtime Environment
Output of Cisco FXOS show tech-support uploaded to SR
Output of Cisco Firepower module show tech-support uploaded to SR
Step 3 Verify FTD System File Integrity
Output of find /ngfw/var/sf/.icdb/* and hashes uploaded to SR
Output of which executed on shell scripts uploaded to SR
Shell scripts, certificates, log file, and hash database added to .tar file
.tar file and its associated hash value uploaded to SR
Step 4 Verify Digitally Signed Image Authenticity
Output of show software authenticity file uploaded to SR
Output of show software authenticity running uploaded to SR
Output of show software authenticity keys uploaded to SR
Step 5 Verify FTD .text Memory Segment Integrity
Output of verify on memory text segment uploaded to SR
Copy of memory text segment uploaded to SR
Step 6 Obtain Firepower crashinfo/core File
Output of crashinfo uploaded to SR
crashinfo file uploaded to SR
core file uploaded to SR
Hash values of crashinfo and core files uploaded to SR
Step 7 Check ROM Monitor Settings
Output of set command uploaded to SR
Note: Only execute the commands in Appendix A if Cisco ASA Software is running on a Cisco Firepower 1000/2100 Series appliance.
Execute the following commands from the Cisco ASA CLI prompt:
enable show tech-support detail dir /recursive all-filesystems verify /sha-512 system:/text copy system:/text ftp:
The output of show tech-support detail may also be redirected to a file server using either the FTP, SCP, SMB, or TFTP protocols. The following example depicts the required syntax if the FTP protocol is used:
show tech-support detail | redirect ftp://anonymous@10.10.10.1/tech-support-details.txt
An example of this entire procedure follows:
ciscoasa# show tech-support detail | redirect ftp://anonymous@10.10.10.1/show-tech-support.txt ciscoasa# dir /recursive all-filesystems Directory of disk0:/* 274941988 -rw- 116406136 19:20:38 Sep 12 2023 asdm.bin 268440446 -rw- 0 13:08:21 Mar 17 2024 coredumpfsysimage.bin 268439875 -rw- 1109 13:36:35 Mar 17 2024 asa-cmd-server.log 271838408 -rw- 39 13:36:34 Mar 17 2024 snortpacketinfo.conf 147407 -rw- 1901 13:08:20 Mar 17 2024 cspCfg.xml Directory of disk0:/log 805367049 -rw- 2141 13:36:38 Mar 17 2024 lina_monitor.log 805367054 -rw- 0 13:36:33 Mar 17 2024 stdout_offload_app.log 805367055 -rwx 372361 17:17:23 Mar 17 2024 asa-miovif.log 805367058 -rw- 749 13:36:34 Mar 17 2024 lcore.log 805367060 -rwx 119861 13:39:57 Mar 17 2024 asa-appagent.log 805367061 -r-- 3572 13:36:35 Mar 17 2024 asa_snmp.log 805367063 -rwx 108556 17:24:21 Mar 17 2024 asa-ssp_ntp.log 805367064 -rwx 0 13:09:15 Mar 17 2024 asa-fxos_xml.log [output truncated] ciscoasa# verify /sha-512 system:/text !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! [output truncated] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!Done! verify /SHA-512 (system:/text) = efc7164423606abce9a67ca5d4202ffd2c14342c4639596acee526d37c2b4a7d2aa
4061629534557c122dbfb528935fdd79fabd9e855c8d6d58e936f6563f8c2 ciscoasa# copy system:/text ftp: Source filename [text]? Address or name of remote host []? 10.10.10.1 Destination filename [text]? system.memory.text.bin 73990144 bytes copied in 2.850 secs (36995072 bytes/sec) ciscoasa#
Cisco highly recommends calculating a hash value on the copied memory segment file and comparing it to the hash value that was obtained on the recipient platform to ensure no errors were introduced during the file transfer process.
The following example utilizes the sha512sum utility, which is included with most Linux distributions:
root@ftp-server:~# sha512sum system.memory.text.bin efc7164423606abce9a67ca5d4202ffd2c14342c4639596acee526d37c2b4a7d2aa406162953
4557c122dbfb528935fdd79fabd9e855c8d6d58e936f6563f8c2 system.memory.text.bin root@ftp-server:~#
Note that the Cisco ASA verify command and the sha512sum utility both produce a SHA-512 hash value of efc7164423606abce9a67ca5d4202ffd2c14342c4639596acee526d37c2b4a7d2aa4061629534557 c122dbfb528935fdd79fabd9e855c8d6d58e936f6563f8c2 for the system.memory.text.bin file.
Submit all command output (including all computed hash values) and a copy of the system:/text memory segment that was collected in this section to the relevant TAC SR.
| Version | Date | Author | Comments |
|---|---|---|---|
| 1.0 | 4/5/2024 | Dan Maunz | Initial public release. |
| 1.1 | 8/27/2024 | Dan Maunz | Integrated 2100 Series. |
This document is part of the Cisco Security portal. Cisco provides the official information contained on the Cisco Security portal in English only.
This document is provided on an “as is” basis and does not imply any kind of guarantee or warranty, including the warranties of merchantability or fitness for a particular use. Your use of the information in the document or materials linked from the document is at your own risk. Cisco reserves the right to change or update this document without notice at any time.