Step One - IOS Device Problem Description
Step Two - IOS Runtime Environment
Step Three - IOS Image File Verification
Step Four - Verify Digitally Signed Image Authenticity
Step Five - Text Memory Region Analysis
Step Six - Core File/Memory Dump
Step Seven - ROMMON Upgrade Check
IOS Device Forensic Report Checklist
Introduction
This document provides guidance for collecting evidence in a forensically sound manner from devices running Cisco IOS Software that are suspected of having been compromised or tampered with. The document outlines a number of commands that can be run to gather evidence for an investigation, along with the respective output that should be collected upon running these commands. This document also provides information on how to perform integrity checks on a devices IOS Software images, how to collect a core file/memory dump from a device running Cisco IOS Software, and how to check to see if a ROMMON upgrade has been applied.
IMPORTANT: It is extrememly important when triaging a network device for compromise or tampering that it is not rebooted. Rebooting a device during an initial assessment will irrecoverably lose all volatile information contained within the device (e.g., RAM contents, arp and routing tables, NAT translations, ACL hit and drop counts, etc.).
Note: It is highly recommended that a device suspected of tampering or compromise be isolated from the network prior to conducting an initial forensic examination. This may prevent remote unloading of any implants or malware installed on the device and will prevent an adversary from monitoring commands entered on the device under investigation.
Note: The examples used in this document are based on Cisco IOS Software Release 15.9.3 command syntax. The output produced by a command may vary depending on the software version deployed and/or the features supported or configured on the device. Not all commands used in these procedures may be supported on earlier versions of the software.
If you require assistance or have questions regarding the following procedures, contact the Cisco Product Security Incident Response Team (PSIRT).
The main section of this document contains the following sections:
1. IOS Device Problem Description describes why the platform is a candidate for forensic examination
2. IOS Runtime Environment collects platform configuration and runtime state
3. IOS Image File Verification examines system image hashes for inconsistencies
4. Verify Digitally Signed Image Authenticity examines system and running images for proper signing characteristics
5. Text Memory Region Analysis provides an alternate method of analysis if a core dump cannot be performed
6. Core File/Memory Dump obtains a core dump of the running IOS image and contents of memory
7. ROMMON Upgrade Check examines the ROM monitor region for an upgraded image
Prerequisites
The procedures described in this document assume the reader has a basic understanding of Cisco IOS Software command syntax.
A valid cisco.com account is required to view individual IOS Software and ROMMON firmware file hashes for software file integrity checking. For customers without a cisco.com account, a publicly available comprehensive list of file hashes (Bulk Hash File) can be downloaded from: https://www.cisco.com/c/en/us/about/trust-center/downloads.html
A Cisco Technical Assistance Center (TAC) service request (SR) is required for the device in question as the procedures outlined in this document assume that the information gathered in each step will be uploaded to a TAC SR.
Step One IOS Device Problem Description
Describe in as much detail as possible WHY the device is a candidate for forensic examination. Are there configuration changes that cannot be explained? Is there unusual traffic originating from or terminating on the device? Are there anomalous entries in the device logs or in syslog messages? Is the device exhibiting odd behavior that cannot be attributed to a misconfiguration or a software or hardware defect? Are there any typical device administration commands that are now returning unusual output or no output at all?
Use the Cisco Software Checker to search for Cisco Security Advisories that apply to specific software releases of the following products: Cisco ASA, FMC, FTD, FXOS, IOS, IOS XE, NX-OS, and NX-OS in ACI Mode.
https://sec.cloudapps.cisco.com/security/center/softwarechecker.x
Record any results returned by the tool that may explain the anomalous behavior being observed. It is considered a best practice to keep software up to date to take advantage of the latest security fixes and enhancements.
Note: This tool does not provide information about Cisco IOS XR Software or interim software builds. Also note that for Cisco ASA, FMC, FTD, and FXOS Software, the tool only contains vulnerability information for Cisco Security Advisories first published from January 2022 onward, and for NX-OS Software and NX-OS Software in ACI Mode from July 2019 onward.
Submit the problem description collected in this section to the relevant TAC SR and proceed to the next section of this document.
Step Two Document the IOS Runtime Environment
The initial stage of evidence gathering is completed by issuing a show tech-support command and a dir all-filesystems command. These commands must be executed in enable mode (i.e., privileged EXEC mode), and some of the output produced may vary dependent on the particular IOS Software version and/or configured features.
Execute each of the following commands in enable mode and record the output:
enable show tech-support dir /recursive all-filesystems
Note: The output from the previous commands may also be redirected to the local file system or a server running any of the following protocols: FTP, HTTP/S, or TFTP with the following command syntax:
show tech-support | redirect <destination> dir /recursive all-filesystems | redirect <destination>
Protocols supported by the redirect directive are FTP, HTTP/S, and TFTP.
An example of this command using the FTP protocol follows:
Router1#show tech-support | redirect ftp://admin:mypassword@10.10.10.1/show-tech.txt Writing show-tech.txt.. Router1#
Submit all command output that is collected in this section to the relevant TAC SR and proceed to the next section of this document.
Step Three IOS Image File Hash Verification
Access the command line of the Cisco IOS device and issue the following command in enable mode:
show version | inc image
Note the location and filename of the system image file and then execute the following command:
verify location:filename
An example of this procedure follows:
Router1#show version | inc image System image file is "flash:c800-universalk9-mz.SPA.159-3.M8.bin" Router1#verify flash:c800-universalk9-mz.SPA.159-3.M8.bin Starting image verification Hash Computation: 100% Done! Computed Hash SHA2: D3DEEEDFB393EB6498F57DFA9DD22663 171CACA67D7FB7102747BB6A3EB55F70 2A4446419B8F2242EE2733A088E1B366 CF2D9A28719949A769978572368CA497 Embedded Hash SHA2: D3DEEEDFB393EB6498F57DFA9DD22663 171CACA67D7FB7102747BB6A3EB55F70 2A4446419B8F2242EE2733A088E1B366 CF2D9A28719949A769978572368CA497 CCO Hash MD5 : E4E8140E528119275B8713E454EB86ED Digital signature successfully verified in file flash:c800-universalk9-mz.SPA.159-3.M8.bin
Note that the embedded hash and computed hash should return the same MD5 value, and the CCO hash should match the MD5 value listed on CCO or in the Bulk Hash File for that particular image file.
Repeat the above procedure for any other system image file located on the file systems. A comprehensive list of all files can be viewed by executing the following command:
dir /recursive all-filesystems
If any of the image file hashes show inconsistencies, copy the image file in question to a secure location if possible.
copy <location>:<system_image_filename.bin> ftp: Address or name of remote host []? <destination_ip> Destination filename []? <destination_filename.bin>
It is highly recommended that a hash value be calculated on the copied system image file and compared to the hash value obtained on the platform to ensure no errors were introduced during the file transfer process.
Submit all command output and any system images collected in this section to the relevant TAC SR and proceed to the next section of this document.
Step Four Verify Digitally Signed Image Authenticity
Some Cisco IOS platforms implement digitally signed system images. Digitally signed Cisco software uses asymmetric (public-key) cryptography that increases the security posture of Cisco IOS XE devices by ensuring that the software that is running on the system has not been altered and that the software originates from a trusted source.
Note: Older IOS platforms may not support the show software authenticity set of commands and will simply return an %Unrecognized command error.
The authenticity and integrity of a system image file can be verified by using the following command:
show software authenticity file location:filename
An example of this procedure follows:
Router1#show software authenticity file flash:c800-universalk9-mz.SPA.159-3.M8.bin File Name : flash:c800-universalk9-mz.SPA.159-3.M8.bin Image type : Production Signer Information Common Name : CiscoSystems Organization Unit : C8xx Organization Name : CiscoSystems Certificate Serial Number : 64D28E52 Hash Algorithm : SHA512 Signature Algorithm : 2048-bit RSA Key Version : A
The Organization Unit, Organization Name, and the Certificate Serial Number values can be viewed to verify that the system image signature is valid.
It is also important to verify the authenticity and integrity of the running system image, and this can be accomplished with the following command:
show software authenticity running
An example of this procedure follows:
Router1#show software authenticity running SYSTEM IMAGE ------------ Image type : Production Signer Information Common Name : CiscoSystems Organization Unit : C8xx Organization Name : CiscoSystems Certificate Serial Number : 64D28E52 Hash Algorithm : SHA512 Signature Algorithm : 2048-bit RSA Key Version : A Verifier Information Verifier Name : ROMMON 1 Verifier Version : System Bootstrap, Version 15.5(1r)T, RELEASE SOFTWARE (fc1) Technical Support: http://www.cisco.com/techsupport
The Organization Unit, Organization Name, and the Certificate Serial Number values can be viewed to verify that the system image signature is valid, and the Certificate Serial Number should be the same as the value that is obtained from the show software authenticity file command. In the previous examples, the authenticity check of the IOS Software image on the boot flash and the authenticity check of the running image both produce a certificate serial number value of 64D28E52.
Lastly, obtain a copy of the public signing keys by executing the following command:
show software authenticity keys
An example of this procedure follows:
Router1#show software authenticity keys Public Key #1 Information ------------------------- Key Type : Production (Primary) Public Key Algorithm : RSA Modulus (256 bytes) : C8:AE:59:5E:E2:52:8C:64:55:6C:C6:AB:89:FA:56:53: 06:2B:72:6D:18:A5:24:8A:37:35:BC:88:24:97:47:5D: 93:76:D0:09:AF:16:EB:86:68:3B:66:CC:80:53:A8:ED: 17:00:D1:F9:2D:15:0A:F2:29:BC:7E:9C:FF:85:31:C6: B1:5D:C7:44:A3:01:0E:D8:85:C9:12:77:61:AE:07:B3: E4:CA:84:AD:FC:C0:4E:E9:87:A2:4F:61:D4:93:C8:0F: 37:D0:11:7F:B7:FB:92:EE:EB:91:56:F3:13:FA:E7:27: 0E:57:4C:EE:F2:78:5A:62:6D:A9:C3:49:AC:96:A0:B8: E8:06:02:14:E0:2F:17:E3:6A:96:34:17:5A:B5:46:1C: AA:D1:F4:F6:3B:4D:4B:6E:1E:1A:09:45:95:44:B4:7C: 85:AD:CB:C7:CE:0A:7D:4A:5D:F3:6B:1B:31:01:A8:78: BE:2D:A0:3E:33:1A:80:D3:29:4F:53:D8:66:CE:7D:AB: DF:AE:1A:D3:1D:61:D1:73:1F:01:FB:7D:02:3E:71:6C: 1A:1E:4B:2D:D0:C3:E9:05:EB:57:D4:D0:A3:89:36:91: 80:85:09:5A:0D:1A:71:31:D0:95:A9:2F:E6:2F:D2:E9: BA:A0:47:96:B0:D9:32:66:F5:34:35:51:36:E3:17:4D Exponent (4 bytes) : 10001 Key Version : A Public Key #2 Information ------------------------- Key Type : Special (Primary) Public Key Algorithm : RSA Modulus (256 bytes) : CE:BE:7A:25:8C:E4:45:79:5C:77:B8:1D:9E:94:78:61: B6:3D:64:4E:3C:36:25:11:9C:26:FF:D9:42:10:4C:86: F5:1C:AD:F1:49:A5:87:D3:4C:69:BF:08:E5:55:1C:59: CD:DA:62:9D:65:33:0D:B6:F1:F1:D1:AC:98:99:6B:CB: 0B:3F:DA:E9:94:06:71:B3:78:B5:AA:85:C8:BE:64:CA: 43:72:F2:B5:4B:C5:4D:FA:D9:CF:51:78:AD:45:9F:E8: CD:41:5A:A6:DE:B7:2E:75:85:EB:8C:7D:68:F2:D4:A1: 5D:DD:B2:26:63:BB:7C:EB:79:80:33:80:05:B9:59:34: 27:89:AA:92:04:61:0C:7D:E5:A5:DE:A0:40:60:73:64: 5A:A7:06:C0:8A:04:CD:3A:1C:99:8D:B7:5C:C8:FE:97: 70:9C:54:DF:AB:A7:8F:04:80:11:08:51:FF:7B:F5:73: C2:A1:C3:E3:A3:45:04:70:90:2D:EA:1E:AD:2C:75:5E: FF:55:EE:0D:75:D3:19:00:59:5C:F6:4C:E2:B7:5F:7A: 8F:3E:9B:21:AC:59:6F:7C:63:B0:62:B5:AA:B4:D8:04: 65:07:8B:56:94:18:14:E3:12:AC:A5:3F:B0:BA:97:D4: 83:22:2E:EC:38:2F:D5:01:39:BA:60:A5:A8:5F:85:87 Exponent (4 bytes) : 10001 Key Version : A [output truncated]
Submit all command output collected in this section to the relevant TAC SR and proceed to the next section of this document.
Step Five Text Memory Region Analysis
Obtaining a core dump as outlined in step five below is the preferred method of gathering information for analysis in IOS device forensics. However, there may be times when retrieving a core dump is not possible and this section provides an alternative method for validating and obtaining a copy of the runtime image from the memory of an IOS device using the show, verify, and copy commands.
Note: It is highly recommended that a copy of the text region be obtained manually in case unforeseen issues arise during the core dump procedure.
Access the CLI of the IOS device and issue the following commands in enable mode:
show region verify /md5 system:memory/text copy system:memory/text ftp:
Note: This example uses the FTP protocol to transfer the text memory region from the platform, but HTTP/S, SCP, and TFTP can also be used.
An example of this procedure follows:
Router1#show region Region Manager: Start End Size(b) Class Media Name 0x00000000 0x3C51313F 1011953984 Local R/W main 0x01000000 0x03FFFFFF 50331648 Local R/W main:heap 0x040001B0 0x0AF9415F 116998064 IText R/O main:text 0x0AF95000 0x0FBFA21F 80106016 IData R/W main:data 0x0FBFA220 0x106A7243 11194404 IBss R/W main:bss 0x106A7244 0x3C51313F 736542460 Local R/W main:heap 0x3C513140 0x3FFFFFFF 61787840 Iomem R/W iomem Free Region Manager: Start End Size(b) Class Media Name Router1#verify /md5 system:memory/text .............................................................................. .............................................................................. [output truncated] .......................................................................................................................................................................................................... MD5 of system:memory/text Done! verify /md5 (system:memory/text) = 87f97ac0b0703019056444a06070dbb2 Router1#copy system:memory/text ftp: Address or name of remote host []? 10.10.10.21 Destination filename [text]? system.memory.text Writing system.memory.text !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! [output truncated] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! 116998064 bytes copied in 18.740 secs (6243226 bytes/sec)
Submit all command output and both of the core files collected in this section to relevant TAC SR and proceed to the next section of this document.
Step Six - IOS Core File/Memory Dump
CAUTION: This section contains commands that alter device configuration. Please ensure that you have the appropriate authorization to make changes to the platform in question prior to proceeding with the following steps.
This procedure outlines how to configure an IOS device in order to obtain a dump of platform memory.
To configure the appropriate dump parameters, enter the following commands in enable mode:
conf t service timestamps debug datetime msec localtime service timestamps log datetime msec localtime service internal exception core-file <filename.bin> compress exception region-size 65536 exception dump <destination_ip_address> exception protocol ftp ip ftp username <username> ip ftp password <passwword> end
CAUTION: Initiating a core dump on an IOS device can be CPU intensive and may adversely affect traffic transitting the platform.
To initiate the core dump process, execute the following command in enable mode:
write core
The core dump may take some time to complete, depending on the amount of physical memory (RAM) installed on the device.
When the core dump process is complete, remove the core dump parameters as follows:
conf t no service internal no exception core-file no exception region-size no exception dump no exception protocol ftp no ip ftp username <username> no ip ftp password <password> end
It is highly recommended that hash values be calculated on the core files obtained in this section so that any errors introduced by subsequent copying or transmission can be reliably detected.
An example of this procedure follows:
Submit all command output and both of the cores files collected in this section to relevant TAC SR, and proceed to the next section of this document.
Step Seven ROMMON Upgrade Check
This procedure simply checks whether the ROM Monitor firmware of the device has been upgraded or not. Cisco is aware of a small number of incidents where a modified firmware image has been introduced in order to change device behavior or to circumnavigate certain platform or IOS license checks.
Execute the following command in enable mode:
show rom-monitor
Example 1 depicts a platform where the ROM Monitor has not been upgraded.
Example 2 depicts a platform where the ROM Monitor has been upgraded.
Submit all command output collected in this section to the relevant TAC SR.
Related Documentation
Additional information about the procedures contained in this document, as well as forensic investigation procedures for other platforms, can be found at the following link:
Cisco Tactical Resources
https://sec.cloudapps.cisco.com/security/center/tacticalresources.x
IOS Device Forensic Response Checklist
Step 1 Create the IOS Device Problem Description
Device Problem Description uploaded to SR
Step 2 Document IOS Runtime Environment
Output of show tech-support uploaded to SR
Output of dir all-filesystems uploaded to SR
Step 3 IOS Image File Hash Verification
Output of verify on system image files uploaded to SR
Image files with hash inconsistencies uploaded to SR
Step 4 Verify Digitally Signed Image Authenticity
Output of show software authenticity file uploaded to SR
Output of show software authenticity running uploaded to SR
Output of show software authenticity keys uploaded to SR
Step 5 Text Memory Region Analysis (Alternate/Optional)
Output of show region uploaded to SR
Output of verify /md5 system:memory uploaded to SR
Copy of the text memory region uploaded to SR
Step 6 IOS Core File Memory Dump
Output of write core uploaded to SR
Core dump file uploaded to SR
Binary iomemory file uploaded to SR
Core dump parameters unconfigured
Step 7 ROMMON Upgrade Check
Output of show rom-monitor uploaded to SR
Revision History
| Version | Date | Author | Comments |
|---|---|---|---|
| 1.0 | 8/19/2019 | Dan Maunz | Initial public release |
| 1.1 | 2/16/2022 | Dan Maunz | Validated procedures on Release 15.x. |
| 1.2 | 02/23/2023 | Dan Maunz | Validated procedures on Release 15.9.3. |
| 1.3 | 05/03/2023 | Dan Maunz | Reordered collection steps. |
| 1.4 | 02/02/2024 | Dan Maunz | Added image signing checks. |
This document is part of the Cisco Security portal. Cisco provides the official information contained on the Cisco Security portal in English only.
This document is provided on an “as is” basis and does not imply any kind of guarantee or warranty, including the warranties of merchantability or fitness for a particular use. Your use of the information in the document or materials linked from the document is at your own risk. Cisco reserves the right to change or update this document without notice at any time.