Step One – Cisco IOS XE Device Problem Description
Step Two – Document the Cisco IOS XE Runtime Environment
Step Three – Cisco IOS XE Image File Hash Verification
Image File Hash Verification (.bin file)
Image File Hash Verification (.conf file)
Step Four – Verify Digitally Signed Image Authenticity
Step Five – Core File Generation and Text Memory Section Export
Step Six – Container Enumeration
Cisco IOS XE Device Forensic Response Checklist
This document provides guidance for collecting evidence from Cisco IOS XE devices that are suspected of compromise or tampering. It outlines a number of commands that can be run to gather evidence for an investigation along with the respective output that should be collected upon running these commands. This document also provides information on how to perform integrity checks on a device’s Cisco IOS XE images and includes a procedure for collecting the text memory segment so that the run-time integrity of the IOSd process can be ascertained.
Note: It is extremely important when triaging a network device for compromise or tampering that it is not rebooted. Rebooting a device during an initial assessment will irrecoverably lose all volatile information contained within the device (for example, RAM contents, arp and routing tables, NAT translations, ACL hit and drop counts, etc.).
Note: It is highly recommended that a device suspected of tampering or compromise be isolated from the network prior to conducting an initial forensic examination. This may prevent remote unloading of any implants or malware installed on the device and will prevent an adversary from monitoring commands entered on the device under investigation.
If you require assistance or have questions regarding the procedures described in this document, contact the Product Security Incident Response Team (PSIRT).
This document contains six main sections:
The procedures outlined in this document assume that the reader has a basic understanding of Cisco IOS XE Software command syntax.
A valid cisco.com account is required to view individual Cisco IOS XE file hashes for software file integrity checking. A publicly available comprehensive list of file hashes (Bulk Hash File) can be downloaded from: https://www.cisco.com/c/en/us/about/trust-center/downloads.html
Note: The examples used in this document are based on Cisco IOS XE Software Release 17.03.05 command syntax. The output produced by a command may vary depending on the software version deployed and/or the features supported or configured on the device. Not all commands used in these procedures may be supported on earlier versions of the software.
Describe in as much detail as possible why the device is a candidate for forensic examination. Are there configuration changes that cannot be explained? Is there unusual traffic originating from or terminating on the device? Are there anomalous entries in the device logs or in syslog messages? Is the device exhibiting odd behavior than cannot be attributed to a misconfiguration or a software/hardware defect? Are there any typical device administration commands that are now returning unusual output or no output at all?
Use the Cisco Software Checker to search for Cisco Security Advisories that apply to specific software releases of the following products: Cisco ASA, FMC, FTD, FXOS, IOS, IOS XE, NX-OS and NX-OS in ACI Mode.
https://sec.cloudapps.cisco.com/security/center/softwarechecker.x
Record any results returned by the tool that may explain the anomalous behavior being observed. It is considered a best practice to keep software up to date to take advantage of the latest security fixes and enhancements.
Note: This tool does not provide information about Cisco IOS XR Software or interim software builds. Also note that for Cisco ASA, FMC, FTD and FXOS Software, the tool only contains vulnerability information for Cisco Security Advisories first published from January 2022 onward, and for NX-OS Software and NX-OS Software in ACI Mode from July 2019 onward.
Submit the problem description and any relevant results obtained from the Cisco Software Checker collected in this section to the relevant TAC SR and proceed to the next section of this document.
Complete the initial stage of evidence gathering by issuing a number of show and dir commands. These commands must be executed in enable mode (privileged EXEC mode), and some of the output produced may vary depending on the particular Cisco IOS XE hardware platform, software version, and/or configured features.
Execute each of the following commands in enable mode and record the output:
! display all listening sockets show tcp brief all numeric show udp show ip sctp association list show ip sockets show platform software kernel tcp brief show platform software kernel udp brief show platform software kernel raw brief ! process and integrity information show rom-monitor rp active show install summary show platform software process list rp active show platform software process memory rp active name linux_iosd-imag maps show platform software process memory rp active name iosd smaps !! For more information about the command above, see footnote [1] at the end of this section !! IOS-XE switch-specific syntax show rom-monitor switch active show platform software process memory switch active r0 name linux_iosd-imag maps show platform software process memory switch active rp active name iosd smaps !! For more information about the command above, see footnote [1] at the end of this section show platform integrity sign nonce 12345 show platform hardware authentication status ! enumerate file systems and files show file systems dir all-filesystems ! system logs – Note: core files and tracelogs may exist on the flash:, bootflash:, harddisk:, or crashinfo: partitions show log show history all dir harddisk:/core dir harddisk:/tracelogs
Obtain a copy of any files found under the /tracelogs directory where the filename begins with “system_shell” as these will need to be uploaded to the SR. The following is an example of a shell log file that should be preserved:
3686402 -rw- 2132 Jul 26 2018 06:16:22 +00:00 system_shell_R0.log.20180726055205
Alternatively, the following Tcl script can be copied and pasted into the command line to collect the output from the commands listed above. Note that some of the commands contained in the script may not execute on all versions of IOS XE Software and that targets of the dir commands may need modification depending on the hardware platform.
tclsh ## ## Cisco IOS XE Triage1 Script ## v1.1 ## ## display all listening sockets ## exec "terminal length 0" exec "show tcp brief all numeric" exec "show udp" exec "show ip sctp association list" exec "show ip sockets" exec "show platform software kernel tcp brief" exec "show platform software kernel udp brief" exec "show platform software kernel raw brief" ## ## process and integrity information ## exec "show rom-monitor rp active" exec "show install summary" exec "show platform software process list rp active" exec "show platform software process memory rp active name linux_iosd-imag maps" exec "show platform software process memory rp active name iosd smaps" ## IOS-XE switch-specific syntax exec "show rom-monitor switch active" exec "show platform software process memory switch active r0 name linux_iosd-imag maps" exec "show platform software process memory switch active rp active name iosd smaps" exec "show platform integrity sign nonce 12345" exec "show platform hardware authentication status" ## ## enumerate file systems and files ## exec "show file systems" exec "dir all-filesystems" ## ## system logs – Note: core files and tracelogs may exist on the flash:, bootflash:, harddisk:, or crashinfo: partitions ## exec "show log" exec "show history all" exec "dir harddisk:/core" exec "dir harddisk:/tracelogs" ## ## ## tclquit
The following list of commands may also be executed to gather additional information relevant to the current operating state of the device. Although the output of these commands is not required to perform a forensic analysis of an IOS XE platform, they may provide additional information regarding any unauthorized changes made to the device if compromise is suspected.
Execution of the following commands in this section of Step Two is optional.
terminal length 0 show clock detail show startup-config show reload show ip route show ip eigrp nei show ip ospf nei show ip bgp summary show cdp nei detail show ip arp show ip interface show ip interface brief show ip nat translations verbose show ip cache flow show ip cef show snmp user show snmp group show snmp community show ipv6 interface brief show ipv6 route show processes
Alternatively, the following Tcl script can be copied and pasted into the command line to collect the output from the commands listed above. Note that some of the commands contained in the script may not execute on all versions of IOS XE Software.
tclsh ## ## Cisco IOS XE Triage2 Script ## v1.0 ## ## IOS/ROMMON Configuration & User Info ## exec "terminal length 0" exec "show history all" exec "show clock detail" exec "show startup-config" exec "show reload" ## ## Network, SNMP, and ACL Info ## exec "show ip route" exec "show ip eigrp nei" exec "show ip ospf nei" exec "show ip bgp summary" exec "show cdp nei detail" exec "show ip arp" exec "show ip interface" exec "show ip interface brief" exec "show tcp brief all" exec "show sockets" exec "show ip nat translations verbose" exec "show ip cache flow" exec "show ip cef" exec "show snmp user" exec "show snmp group" exec "show snmp community exec "show ipv6 interface brief" exec "show ipv6 route" ## ## Local Logging, Process and Memory Info ## exec "show logging" exec "show processes" ## ## ## tclquit
Submit all command or script output and any system shell logs collected in this section to the relevant TAC SR and proceed to the next section of this document.
[1] Note: This procedure checks the active route processor (RP) for non-zero values in the “Private Dirty” entry for each segment with the executable flag set (i.e., r-xp or rwxp).
show platform software process memory rp active name iosd smaps show platform software process memory switch active rp active name iosd smaps
The commands above display the memory map for the iosd process running on the active route processor. Executable segments of a program typically have the r-xp (read, execute, protected) attributes set, while an executable segment with the w (write) attribute set may indicate the software has been tampered with.
Example 1:
# # This segment is flagged rw-p (read, write, protected) so a private # dirty value greater than zero is expected and not of concern. # Router#show platform software process memory rp active name iosd smaps smaps for process 7438: address perms offset dev inode pathname 7fff6000-7fff7000 rw-p 00000000 00:00 0 Size: 128 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Rss: 32 kB Pss: 32 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB LazyFree: 0 kB AnonHugePages: 0 kB ShmemPmdMapped: 0 kB Shared_Hugetlb: 0 kB Private_Hugetlb: 0 kB Swap: 0 kB SwapPss: 0 kB Locked: 0 kB THPeligible: 0 VmFlags: rd wr mr mw ac [output truncated]
Example 2:
# # This segment is flagged rwxp (read, write, execute, protected) and # private dirty has a value greater than zero which may indicate the # software has been tampered with. # Router#show platform software process memory rp active name iosd smaps smaps for process 16320: 10a0000000-10a04c1000 rwxp 00000000 00:00 0 Size: 128 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Rss: 128 kB Pss: 128 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 128 kB Referenced: 128 kB Anonymous: 128 kB LazyFree: 0 kB AnonHugePages: 0 kB ShmemPmdMapped: 0 kB Shared_Hugetlb: 0 kB Private_Hugetlb: 0 kB Swap: 0 kB SwapPss: 0 kB Locked: 0 kB THPeligible: 0 VmFlags: rd wr ex mr mw me ac [output truncated]
Access the command line of the Cisco IOS XE device and issue the following command in enable mode:
show version
If the system image file has a .bin file extension as shown in the following example, execute the steps in the Image File Hash Verification (.bin file) section of this document and omit the steps in the Image File Hash Verification (.conf file) section.
System image file is "bootflash:asr1000rp2-advipservicesk9.03.13.09.S.154-3.S9-ext.bin"
If the system image file has a .conf file extension as shown in the following example, skip the Image File Hash Verification (.bin file) section of this document and execute the steps in the Image File Hash Verification (.conf file) section.
System image file is "bootflash:packages.conf"
Note the location and filename of the system image file obtained in the beginning of this section and execute the following command:
verify location:filename
An example of this procedure is as follows:
Router1#show version Cisco IOS XE Software, Version 03.13.09.S - Extended Support Release Cisco IOS Software, ASR1000 Software (X86_64_LINUX_IOSD-ADVIPSERVICESK9-M), Version 15.4(3)S9, RELEASE SOFTWARE (fc2) Technical Support: http://www.cisco.com/techsupport Copyright (c) 1986-2018 by Cisco Systems, Inc. Compiled Mon 26-Feb-18 08:59 by mcpre Cisco IOS XE Software, Copyright (c) 2005-2018 by cisco Systems, Inc. All rights reserved. Certain components of Cisco IOS XE Software are licensed under the GNU General Public License ("GPL") Version 2.0. The software code licensed under GPL Version 2.0 is free software that comes with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such GPL code under the terms of GPL Version 2.0. For more details, see the documentation or "License Notice" file accompanying the IOS XE Software, or the applicable URL provided on the flyer accompanying the IOS XE Software. ROM: IOS-XE ROMMON Router1 uptime is 1 day, 15 hours, 21 minutes Uptime for this control processor is 1 day, 15 hours, 22 minutes System returned to ROM by reload System image file is "bootflash:asr1000rp2-advipservicesk9.03.13.09.S.154-3.S9-ext.bin" Last reload reason: Reload Command [output truncated] Router1#verify bootflash:asr1000rp2-advipservicesk9.03.13.09.S.154-3.S9-ext.bin Verifying file integrity of bootflash:asr1000rp2-advipservicesk9.03.13.09.S.154-3.S9-ext.bin ........................................................................... ........................................................................... Embedded Hash SHA1 : 7E9EA496349FC44B223C09F6DCC89FA1F5FBA7A8 Computed Hash SHA1 : 7E9EA496349FC44B223C09F6DCC89FA1F5FBA7A8 Starting image verification Hash Computation: 100%Done! Computed Hash SHA2: 35ea9ab4825f32810def7c10aa23acb4 80da7bb0e4903d3e28d3ebec2bd1cd5a 9c5cdb6c2faf429c945efe48b78a7920 3ceb36bae21324d88963df3ddd6aedda Embedded Hash SHA2: 35ea9ab4825f32810def7c10aa23acb4 80da7bb0e4903d3e28d3ebec2bd1cd5a 9c5cdb6c2faf429c945efe48b78a7920 3ceb36bae21324d88963df3ddd6aedda Digital signature successfully verified in file bootflash:asr1000rp2-advipservicesk9.03.13.09.S.154-3.S9-ext.bin Embedded hash verification successful.
Note that the embedded hash and computed hash should retun the same SHA1 (160 bit) and SHA2 (256 bit) values.
An SHA-512 hash can be calculated by adding the /sha512 parameter to the verify command as follows:
verify /sha512 location:filename
An MD5 hash can also be calculated by adding the /md5 parameter to the verify command as follows:
verify /md5 location:filename
The SHA-512 or MD5 hashes should match the values listed on CCO or in the Bulk Hash File for that particular image file.
Note: CCO contains only MD5 and SHA-512 hash values for software images.
Repeat the previous procedure for any other system image file located on the file systems. A comprehensive list of all files can be viewed by executing the following command:
dir all-filesystems
If any of the image file hashes show inconsistencies, copy the image file in question to a secure location if possible.
copy <location>:<system_image_filename.bin> ftp: Address or name of remote host []? <destination_ip> Destination filename []? <destination_filename.bin>
It is highly recommended that a hash value be calculated on the copied system image file and compared to the hash value obtained on the platform to ensure that no errors were introduced during the file transfer process.
Submit all command output (including calculated hash values), the running system image, and any other system images tested in this section to the relevant TAC SR and proceed to Step Four.
Note the location and filename of the system image file obtained in the beginning of this section and execute the following command:
more location:filename
Next, issue the following command for the packages.conf image file and each unique entry listed in the contents of the packages.conf image file:
verify location:filename
An example of this procedure is as follows:
CSR1000v#show version Cisco IOS XE Software, Version 16.06.04 Cisco IOS Software [Everest], Virtual XE Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 16.6.4, RELEASE SOFTWARE (fc3) Technical Support: http://www.cisco.com/techsupport Copyright (c) 1986-2018 by Cisco Systems, Inc. Compiled Sun 08-Jul-18 04:30 by mcpre Cisco IOS XE Software, Copyright (c) 2005-2018 by cisco Systems, Inc. All rights reserved. Certain components of Cisco IOS XE Software are licensed under the GNU General Public License ("GPL") Version 2.0. The software code licensed under GPL Version 2.0 is free software that comes with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such GPL code under the terms of GPL Version 2.0. For more details, see the documentation or "License Notice" file accompanying the IOS XE software, or the applicable URL provided on the flyer accompanying the IOS XE software. ROM: IOS-XE ROMMON CSR1000v uptime is 36 minutes Uptime for this control processor is 38 minutes System returned to ROM by reload at 22:14:54 EST Thu Jul 26 2018 System restarted at 22:16:28 EST Thu Jul 26 2018 System image file is "bootflash:packages.conf" Last reload reason: Reload Command [output truncated] CSR1000v#more bootflash:packages.conf #! /usr/binos/bin/packages_conf.sh sha1sum: 7ebf483217e3e7071ed796f2a17258fb60b6b2b0 boot rp 0 0 rp_boot csr1000v-rpboot.16.06.04.SPA.pkg iso rp 0 0 rp_base csr1000v-mono-universalk9.16.06.04.SPA.pkg iso rp 0 1 rp_base csr1000v-mono-universalk9.16.06.04.SPA.pkg iso rp 0 0 rp_daemons csr1000v-mono-universalk9.16.06.04.SPA.pkg iso rp 0 1 rp_daemons csr1000v-mono-universalk9.16.06.04.SPA.pkg iso rp 0 0 rp_iosd csr1000v-mono-universalk9.16.06.04.SPA.pkg iso rp 0 1 rp_iosd csr1000v-mono-universalk9.16.06.04.SPA.pkg iso rp 0 0 rp_security csr1000v-mono-universalk9.16.06.04.SPA.pkg iso rp 0 1 rp_security csr1000v-mono-universalk9.16.06.04.SPA.pkg iso fp 0 0 fp csr1000v-mono-universalk9.16.06.04.SPA.pkg iso fp 0 1 fp csr1000v-mono-universalk9.16.06.04.SPA.pkg iso rp 0 0 rp_webui csr1000v-mono-universalk9.16.06.04.SPA.pkg iso rp 0 1 rp_webui csr1000v-mono-universalk9.16.06.04.SPA.pkg boot rp 1 0 rp_boot csr1000v-rpboot.16.06.04.SPA.pkg iso rp 1 0 rp_base csr1000v-mono-universalk9.16.06.04.SPA.pkg iso rp 1 1 rp_base csr1000v-mono-universalk9.16.06.04.SPA.pkg iso rp 1 0 rp_daemons csr1000v-mono-universalk9.16.06.04.SPA.pkg iso rp 1 1 rp_daemons csr1000v-mono-universalk9.16.06.04.SPA.pkg iso rp 1 0 rp_iosd csr1000v-mono-universalk9.16.06.04.SPA.pkg iso rp 1 1 rp_iosd csr1000v-mono-universalk9.16.06.04.SPA.pkg iso rp 1 0 rp_security csr1000v-mono-universalk9.16.06.04.SPA.pkg iso rp 1 1 rp_security csr1000v-mono-universalk9.16.06.04.SPA.pkg iso fp 1 0 fp csr1000v-mono-universalk9.16.06.04.SPA.pkg iso fp 1 1 fp csr1000v-mono-universalk9.16.06.04.SPA.pkg iso rp 1 0 rp_webui csr1000v-mono-universalk9.16.06.04.SPA.pkg iso rp 1 1 rp_webui csr1000v-mono-universalk9.16.06.04.SPA.pkg # # -start- superpackage .pkginfo # # pkginfo: Name: rp_super # pkginfo: BuildTime: # pkginfo: ReleaseDate: Sun-08-Jul-18-07:07 # pkginfo: .BuildArch: x86_64 # pkginfo: BootArchitecture: i686 # pkginfo: .BootArch: i686 # pkginfo: RouteProcessor: ultra # pkginfo: Platform: CSR1000V # pkginfo: User: mcpre # pkginfo: PackageName: universalk9 # pkginfo: Build: 16.06.04 # pkginfo: .SupportedBoards: ultra # pkginfo: .InstallModel: # pkginfo: .PackageRole: rp_super # pkginfo: .RestartRole: rp_super # pkginfo: CardTypes: # pkginfo: .CardTypes: # pkginfo: .BuildPath: /scratch/mcpre/release/BLD-V16_06_04_FC3/binos/linkfarm/stage-ultra/hard/rp_super_universalk9.x86_64 # pkginfo: .Version: 16.6.4.0.3716.1531061508..Everest # pkginfo: .InstallVersion: 1.0.0 # pkginfo: .InstallCapCommitSupport: yes CSR1000v#verify bootflash:packages.conf bootflash:packages.conf is detected as a provisioning file Verifying file integrity of bootflash:packages.conf. Embedded Hash SHA1 : 7EBF483217E3E7071ED796F2A17258FB60B6B2B0 Computed Hash SHA1 : 7EBF483217E3E7071ED796F2A17258FB60B6B2B0 CSR1000v# verify bootflash:csr1000v-rpboot.16.06.04.SPA.pkg Verifying file integrity of bootflash:csr1000v-rpboot.16.06.04.SPA.pkg ........................................................................... ........................................................................... Embedded Hash SHA1 : 3AEDB29325BB17D02B39D295F15627286A1E2BEA Computed Hash SHA1 : 3AEDB29325BB17D02B39D295F15627286A1E2BEA Starting image verification Hash Computation: 100%Done! Computed Hash SHA2: f2682757e0106b9c3907962d96028608 89eb9325e0cb78276b0097219192143a 0d35fd011c0610279f0a97c55fe2ea6c 2aac24889967ce07344253f79267dcf2 Embedded Hash SHA2: f2682757e0106b9c3907962d96028608 89eb9325e0cb78276b0097219192143a 0d35fd011c0610279f0a97c55fe2ea6c 2aac24889967ce07344253f79267dcf2 Digital signature successfully verified in file bootflash:csr1000v-rpboot.16.06.04.SPA.pkg CSR1000v# verify bootflash:csr1000v-mono-universalk9.16.06.04.SPA.pkg Verifying file integrity of bootflash:csr1000v-mono-universalk9.16.06.04.SPA.pkg ........................................................................... ........................................................................... Embedded Hash SHA1 : C5843C05740F1828197F3093DA294F11BA1DE37B Computed Hash SHA1 : C5843C05740F1828197F3093DA294F11BA1DE37B Starting image verification Hash Computation: 100%Done! Computed Hash SHA2: 14354cda30bb20d38e572275c6e1cc2a 0cc647459a1a34dd4267282eeaddf799 bda3f2048aca419ea49a417fe28fa43b 4e1501acb9f54fb521fe5b00e7f8337e Embedded Hash SHA2: 14354cda30bb20d38e572275c6e1cc2a 0cc647459a1a34dd4267282eeaddf799 bda3f2048aca419ea49a417fe28fa43b 4e1501acb9f54fb521fe5b00e7f8337e Digital signature successfully verified in file bootflash:csr1000v-mono-universalPA.pkg
Note that the embedded hash and computed hash should retun the same SHA1 (160 bit) and SHA2 (256 bit) values.
An SHA-512 hash can be calculated by adding the /sha512 parameter to the verify command as follows:
verify /sha512 location:filename
An MD5 hash can also be calculated by adding the /md5 parameter to the verify command as follows:
verify /md5 location:filename
The SHA-512 or MD5 hashes should match the values listed on CCO or in the Bulk Hash File for that particular image file.
Note: CCO contains only MD5 and SHA-512 hash values for software images.
Repeat the above procedure for any other system image file located on the file systems. A comprehensive list of all files can be viewed by executing the following command:
dir all-filesystems
If any of the image file hashes show inconsistencies, copy the image file in question to a secure location if possible.
copy <location>:<system_image_filename.bin> ftp: Address or name of remote host []? <destination_ip> Destination filename []? <destination_filename.bin>
It is highly recommended that a hash value be calculated on the copied system image file and compared to the hash value obtained on the platform to ensure that no errors were introduced during the file transfer process.
Submit all command output (including calculated hash values), the running system image, and any other system images tested in this section to the relevant TAC SR, and proceed to Step Four of this document.
Cisco IOS XE Software implements digitally signed system images on most platforms. Digitally signed Cisco software uses asymmetric (public-key) cryptography that increases the security posture of Cisco IOS XE devices by ensuring that the software running on the system has not been altered and that the software originates from a trusted source.
The authenticity and integrity of a system image file can be verified by using the following command:
show software authenticity file location:filename
An example of this procedure follows:
Router1#:show software authenticity file bootflash:asr1000rp2-advipservicesk9.03.13.09.S.154-3.S9-ext.bin File Name : bootflash:asr1000rp2-advipservicesk9.03.13.09.S.154-3.S9-ext.bin Image type : Production Signer Information Common Name : CiscoSystems Organization Unit : IOS-XE Organization Name : CiscoSystems Certificate Serial Number : 5A94807E Hash Algorithm : SHA512 Signature Algorithm : 2048-bit RSA Key Version : A
The Organization Unit, Organization Name, and the Certificate Serial Number values can be viewed to verify that the system image signature is valid.
It is also important to verify the authenticity and integrity of the running system image, and this can be accomplished with the following command:
show software authenticity running
Note: This procedure may not produce command ouput when executed on Cisco IOS XE virtual devices such as the CSR1000V.
An example of this procedure follows:
Router1# show software authenticity running SYSTEM IMAGE ------------ Image type : Production Signer Information Common Name : CiscoSystems Organization Unit : IOS-XE Organization Name : CiscoSystems Certificate Serial Number : 5A94807E Hash Algorithm : SHA512 Signature Algorithm : 2048-bit RSA Key Version : A Verifier Information Verifier Name : ROMMON Verifier Version : System Bootstrap, Version 16.3(2r Microloader ----------- Image type : Release Signer Information Common Name : CiscoSystems Organization Name : CiscoSystems Certificate Serial Number : 4143616e6e65642d5348413235362d48 Hash Algorithm : HMAC-SHA256 Verifier Information Verifier Name : Hardware Anchor Verifier Version : ACannedHwAnchorVersionApril2012
The Organization Unit, Organization Name, and the Certificate Serial Number values can be viewed to verify that the system image signature is valid, and the Certificate Serial Number should be the same as the value obtained from the show software authenticity file command. In the examples above, the authenticity check of the IOS XE Software image on the boot flash and the authenticity check of the running image both produce a value of 5A94807E.
It is also recommended that digital signatures are verified for all other .bin and .pkg files resident on the device’s file systems. This can be accomplished with the following command:
show platform software authenticity verify location:filename
An example follows:
Router1#:show platform software authenticity verify bootflash:asr1000rp2-advipservicesk9.03.13.09.S.154-3.S9-ext.bin Digital signature successfully verified in file bootflash:asr1000rp2-advipservicesk9.03.13.09.S.154-3.S9-ext.bin
Note: Some Cisco IOS XE platforms may require that service internal be configured prior to issuing the show platform software authenticity verify command.
Submit all command output collected in this section to the relevant TAC SR and proceed to the next section of this document.
CAUTION: This section contains commands that alter the IOS XE device configuration. Please ensure you have a copy of the original device configuration and the appropriate authorization to make changes to the platform in question prior to proceeding with this procedure. Creating core dumps may also cause spikes in memory consumption and momentary disruptions to traffic transiting the device if a standby route processor is not present in the platform.
This procedure outlines how to configure a Cisco IOS XE device to obtain a core dump of platform memory. A crash information file is created in the root of the bootflash filesystem, and the core file is created in the bootflash:/core subdirectory.
The steps to acquire a core dump of the iosd process are as follows:
service internal show platform software process environment ios rp active # use the following command on routing platforms request platform software process core ios rp active # use the following command on switching platforms request platform software process core ios switch active rp active
Note: Some platforms may use the test command instead of the request command. The syntax for the test command is as follows:
test platform software process core
An example of the procedure to acquire a crash info file and core dump file follows.
Router1#conf t Enter configuration commands, one per line. End with CNTL/Z. Router1(config)#service internal Router1(config)#end Router1# Feb 1 19:11:12.619: %SYS-5-CONFIG_I: Configured from console by console Router1#show platform software process environment ios rp active Name Value ------------------------------------------------------------------------------ TRACEKEY 1#84f2d2cb5aa33b87358ca5a1b96d199e BINOS_FRU_BASE_PKG rp_base BINOS_BAY_LOCAL 0 PROC_CONF_RESTART norestart RELOADFAST_FILE /tmp/chassis/local/stack_mgr/fast_reload BINOS_CONF_DIR /usr/binos/conf PROCESS_FRU rp SLOT 0 BINOS_BTRACE_FILE_PATH /tmp/rp/trace TERM linux BINOS_LOCAL_CHASSIS_PATH /tmp/chassis/local LD_LIBRARY_PATH_COPY /tmp/sw/rp/0/0/rp_iosd/mount/lib64:/tmp/sw/rp/ ROMMON_BOOT bootflash:/asr1002x-universalk9.17.03.05.SPA.b BOARD_TYPE RP PROC_CONF_FAILURE_ACTION critical BINOS_SLOT 0 TAN_FILE_ROTATE_INTERVAL_IN_SE BINOS_FRU rp BINOS_BOOT_MODE non-NFS CHASSIS_SERIAL_NUM FOX2107P0WA BINOS_BASE_DIR /tmp/rp [output truncated] Router1#request platform software process core ios rp active SUCCESS: Core file generated. Router1# Exception to IOS Thread: Frame pointer 0x7FFE138F3498, PC = 0x7FE8F81E1305 UNIX-EXT-SIGNAL: Aborted(6), Process = Sched -Traceback= 1#84f2d2cb5aa33b87358ca5a1b96d199e c:7FE8F81CC000+15305 c:7FE8F81CC000+56C binos:7FE96CD98000+AC8E RAX = 0000000000000000 RBX = 0000000000000006 RCX = 00007FE8F81E1305 RDX = 0000000000000000 RSP = 00007FFE138F3498 RBP = 00007FFE138F3530 RSI = 00007FFE138F3420 RDI = 0000000000000002 R8 = 0000000000000000 R9 = 00007FFE138F3420 R10 = 0000000000000008 R11 = 0000000000000246 R12 = 00007FE8D7CD7E40 R13 = 00007FE8D7CD8C10 R14 = 0000000000000000 R15 = 0000000000000000 RFL = 0000000000000246 RIP = 00007FE8F81E1305 CS = 0033 FS = 0000 GS = 0000 ST0 = 0000 0000000000000000 ST1 = 0000 0000000000000000 ST2 = 0000 0000000000000000 ST3 = 0000 0000000000000000 ST4 = 0000 0000000000000000 ST5 = 3FE8 8000000000000000 ST6 = 401D C7B572CA00000000 ST7 = 0000 0000000000000000 X87CW = 037F X87SW = 0000 X87TG = 0000 X87OP = 055D X87IP = 00000000F825FEB4 X87DP = 00000000138F3368 XMM0 = 00000000000000000000000000000000 XMM1 = 00000000000000000000000000000000 XMM2 = 00000000000000000000000000000000 XMM3 = FF000000000000000000000000000000 XMM4 = 00000000000000000000000000000000 XMM5 = FFFF000000000000FFFF000000000000 XMM6 = FFFF0000000000FFFFFFFFFFFFFFFFFF XMM7 = 00000000000000004000000000000000 XMM8 = 000055C501EB0DC00000000000000000 XMM9 = 00000000FFFFFFFF00000000FFFFFFFF XMM10 = 000000000000000E000000000000000E XMM11 = 000000000000001C000000000000001C XMM12 = 00000000000000000000000000000000 XMM13 = 00000000000000000000000000000000 XMM14 = 000000FF000000FF000000FF000000FF XMM15 = 9D8507ED6CF6D92FFEA86E7142CE5DAD MXCSR = 00001F80 Writing crashinfo to bootflash:Router1_crashinf_20230201-141134-EST IOSD RP: 1172 messages not written to btrace log file Router1#dir Directory of bootflash:/ 64769 drwx 4096 Feb 1 2023 14:11:58 -05:00 core 21 -rw- 220651 Feb 1 2023 14:11:37 -05:00 Router1_crashinf_20230201-141134-EST 696257 drwx 4096 Feb 1 2023 14:11:37 -05:00 .prst_sync 129537 drwx 28672 Feb 1 2023 14:11:03 -05:00 tracelogs 15 -rw- 1 Sep 27 2022 11:36:52 -04:00 cs_verify_rc.txt 97153 drwx 4096 Feb 18 2022 10:39:19 -05:00 .installer 291457 drwx 4096 Feb 18 2022 10:38:16 -05:00 license_evlog 14 -rw- 30 Feb 18 2022 10:38:14 -05:00 throughput_monitor_params 18 -rw- 134458 Feb 18 2022 10:37:37 -05:00 memleak.tcl 242881 drwx 4096 Feb 18 2022 10:37:28 -05:00 .inv 17 -rw- 454 Feb 18 2022 10:37:07 -05:00 mode_event_log 744833 drwx 4096 Feb 18 2022 02:32:53 -05:00 .dbpersist 259073 drwx 4096 Feb 18 2022 02:29:56 -05:00 onep 20 -rw- 1923 Feb 18 2022 02:29:04 -05:00 trustidrootx3_ca_092024.ca 19 -rw- 20109 Feb 18 2022 02:29:04 -05:00 ios_core.p7b 307649 drwx 4096 Feb 18 2022 02:28:50 -05:00 gs_script 194305 drwx 4096 Feb 18 2022 02:28:43 -05:00 bootlog_history 16 -rw- 860923521 Feb 18 2022 02:11:41 -05:00 asr1002x-universalk9.17.03.05.SPA.bin 178113 drwx 4096 Apr 27 2017 13:43:14 -04:00 virtual-instance 13 -rw- 0 Apr 27 2017 13:43:11 -04:00 tracelogs.629 80961 drwx 4096 Apr 27 2017 13:43:05 -04:00 .rollback_timer 12 -rw- 454254232 Apr 27 2017 13:35:32 -04:00 asr1002x-universal.03.16.02b.S.155-3.S2b-ext.SPA.bin 11 drwx 16384 Apr 27 2017 13:33:50 -04:00 lost+found 6646632448 bytes total (4938723328 bytes free) Router1#verify /sha512 bootflash:/Router1_crashinf_20230201-141134-EST ....Done! verify /sha512 (bootflash:/Router1_crashinf_20230201-141134-EST) = def67296d50606d28d4f80b79826ff3898e0f7f07d01f59aefd658309632eb7f308b950d471b7c024a0c72097069d8cbb602365b7ddb32c4de99775806e86d62 Router1#dir bootflash:/core Directory of bootflash:/core/ 161922 -rw- 38893484 Feb 1 2023 14:11:58 -05:00 Router1_RP_0_x86_64_crb_linux_iosd-universalk9-ms_14778_20230201-141137-EST.core.gz 64770 -rw- 1 Feb 1 2023 14:10:16 -05:00 .callhome 161921 drwx 4096 Feb 18 2022 02:28:48 -05:00 modules 6646632448 bytes total (4938723328 bytes free) Router1#verify /sha512 bootflash:/core/Router1_RP_0_x86_64_crb_linux_iosd-universalk9-ms_14778_20230201-141137-EST.core.gz ..................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................Done! verify /sha512 (bootflash:/core/Router1_RP_0_x86_64_crb_linux_iosd-universalk9-ms_14778_20230201-141137-EST.core.gz) = 7da52742d367a959351944d7ed020472ca30462cda214457fec3e860528e18a2c43bf0e69d557a5d172d9a8d66356526c3f1b72af5ee857f6b95c9718e67c8f0 Router1#copy bootflash:/Router1_ crashinf_20230201-141134-EST ftp: Address or name of remote host []? 192.168.1.1 Destination filename [Router1_crashinf_20230201-141134-EST]? Writing Router1_crashinf_20230201-141134-EST ! 220651 bytes copied in 4.675 secs (47198 bytes/sec) Router1#copy bootflash:/core/Router1_RP_0_x86_64_crb_linux_iosd-universalk9-ms_14778_20230201-141137-EST.core.gz ftp: Address or name of remote host []? 192.168.1.1 Destination filename [Router1_RP_0_x86_64_crb_linux_iosd-universalk9-ms_14778_20230201-141137-EST.core.gz]? Writing Router1_RP_0_x86_64_crb_linux_iosd-universalk9-ms_14778_20230201-141137-EST.core.gz !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! 38893484 bytes copied in 5.830 secs (6671267 bytes/sec) Router1#
The remainder of this section outlines the procedure to collect the system:memory/text region to verify the run-time integrity of the IOSd process from a device running Cisco IOS XE Software.
Note: This procedure requires a minimum Cisco IOS XE Software version of 15.5.1 or later. The text region in earlier versions of Cisco IOS XE Software may point to the text section of a library instead of the executable code of the IOSd process.
Access the command line interface of the Cisco IOS XE device and issue the following command in enable mode to view the system:memory/text entry:
dir system:memory
Copy the system:memory/text region to a file server using ftp or scp.
copy system:memory/text ftp:
An example of this procedure follows:
Router1#dir system:memory Directory of system:/memory/ 8 -r-- 26460576 <no date> bss 7 -r-- 59584 <no date> data 5 -r-- - 24686336 <no date> heap 4 -r-- 6295128 <no date> lsmpi_mem 9 -r-- 196608 <no date> stack 6 -r-- 303844 <no date> text No space information available Router1#copy system:memory/text ftp: Address or name of remote host []? 192.168.1.1 Destination filename [text]? asr1004_main_text Writing asr1004_main_text !!!!!! 303844 bytes copied in 4.641 secs (65470 bytes/sec)
It is highly recommended that hash values be calculated for the device’s system:memory/text region and for the file that was copied to the file server to ensure that no errors were introduced during the file transfer process.
To calculate a hash value for the system:memory/text region, execute the following command:
verify /md5 system:memory/text
An example of this procedure follows:
Router1#verify /md5 system:memory/text .....Done! verify /md5 (system:memory/text) = 53271c3c2baae5f6f9666db3031d478f
Next, calculate a hash value for the file transferred to the file server. This example uses the Microsoft File Checksum Integrity Verifier (FCIV) for Windows operating systems, which can be downloaded here.
C:\temp>dir Volume in drive C has no label. Volume Serial Number is C21A-84D2 Directory of C:\temp 06/15/2018 10:37 AM <DIR> 06/15/2018 10:37 AM <DIR> 06/15/2018 10:19 AM 303,844 asr1004_main_text 1 File(s) 303,844 bytes 2 Dir(s) 2,137,176,567,808 bytes free C:\temp>fciv asr1004_main_text -md5 // // File Checksum Integrity Verifier version 2.05. // 53271c3c2baae5f6f9666db3031d478f asr1004_main_text
Note that the IOS XE verify command and the fciv utility both produce an MD5 hash value of 53271c3c2baae5f6f9666db3031d478f.
Alternatively, an MD5 hash value can be calculated with the md5sum utility, which is included with most Linux distributions.
root@ftp-server:~# md5sum asr1004_main_text 53271c3c2baae5f6f9666db3031d478f asr1004_main_text root@ftp-server:~#
Submit all command output (including calculated hash values) and the file containing the system:memory/text output to the relevant TAC SR and proceed to the next section of this document.
Cisco IOS XE platforms support Linux containers (LXCs) and a Guest Shell feature (based on CentOS Linux Version 8) that provide a Bash interface for an isolated Linux execution space on the host system. This architecture can be used to accommodate custom applications, allowing for resource and fault isolation that maintains Cisco IOS XE operating system integrity but allows the client container to access the bootflash storage partition.
Guest Shells and LXCs are managed with the guestshell and app-hosting commands.
The first step in container enumeration is to determine whether the application-hosting environment is enabled or has ever been enabled in the past.
Examine the IOx service status and the file system for an IOx reserved area on the disk subsystem. It is important to employ the following logic when attempting to enumerate LXCs on Cisco IOS XE Software:
Guest Shell and app-hosting commands:
show iox show app-hosting list show app-hosting detail guestshell run bash sudo su - dnf history dnf repolist dnf history userinstalled dnf list --installed exit exit
Note: For non–Guest Shell LXCs, use the app-hosting connect appid <container_name> console command instead of the guestshell run bash command.
An example of this procedure follows.
Router#show iox IOx Infrastructure Summary: --------------------------- IOx service (CAF) : Not Running IOx service (HA) : Not Supported IOx service (IOxman) : Not Running IOx service (Sec storage) : Not Supported Libvirtd 1.3.4 : Running Router#dir bootflash: | inc SHARED-IOX 177409 drwx 4096 Oct 21 2021 12:55:02 +00:00 SHARED-IOX # # In this example, the IOx process is not running but the SHARED-IOX directory # exists on the filesystem, so we must start the service: # Router#conf t Enter configuration commands, one per line. End with CNTL/Z. Router(config)#iox Router(config)#end
Router#show iox IOx Infrastructure Summary: --------------------------- IOx service (CAF) 1.11.0.5 : Running IOx service (HA) : Not Supported IOx service (IOxman) : Running IOx service (Sec storage) : Not Supported Libvirtd 1.3.4 : Running Router#show app-hosting list App id State --------------------------------------------------------- guestshell RUNNING # # Note: If the IOx service is running but the guest shell state is not # running, guest shell may be started by issuing the following command: # app-hosting start appid guestshell # Router#show app-hosting detail App id : guestshell Owner : iox State : RUNNING Application Type : lxc Name : GuestShell Version : 3.1.0 Description : Cisco Systems Guest Shell XE for x86_64 Path : /guestshell/:guestshell.tar URL Path : Activated profile name : custom Resource reservation Memory : 256 MB Disk : 1 MB CPU : 800 units VCPU : 1 Attached devices Type Name Alias --------------------------------------------- serial/shell iox_console_shell serial0 serial/aux iox_console_aux serial1 serial/syslog iox_syslog serial2 serial/trace iox_trace serial3 Network interfaces --------------------------------------- eth0: MAC address : 52:54:dd:87:62:86 IPv4 address : 10.10.2.113 Network name : VPG0 Port forwarding Table-entry Service Source-port Destination-port --------------------------------------------------- Router#guestshell run bash [guestshell@guestshell ~]$ sudo su - Last login: Thu Oct 21 17:54:14 UTC 2021 on pts/4 [root@guestshell ~]# dnf history ID | Command line | Date and time | Action(s) | Altered ------------------------------------------------------------------------------ 1 | install iperf3 | 2021-10-21 20:11 | Install | 2 [root@guestshell ~]# dnf repolist Last metadata expiration check: 0:20:54 ago on Thu Oct 21 20:10:29 2021. repo id repo name status AppStream CentOS-8 - AppStream 6034 BaseOS CentOS-8 - Base 2151 *epel Extra Packages for Enterprise Linux 8 - x86_64 7856 extras CentOS-8 - Extras 38 [root@guestshell ~]# dnf history userinstalled Packages installed by user iperf3-3.5-6.el8.x86_64 [root@guestshell ~]# dnf list --installed Installed Packages acl.x86_64 2.2.53-1.el8 @System attr.x86_64 2.4.48-3.el8 @System audit-libs.x86_64 3.0-0.13.20190507gitf58ec40.el8 @System basesystem.noarch 11-5.el8 @System bash.x86_64 4.4.19-10.el8 @System brotli.x86_64 1.0.6-1.el8 @System bzip2-libs.x86_64 1.0.6-26.el8 @System ca-certificates.noarch 2019.2.32-80.0.el8_1 @System centos-gpg-keys.noarch 8.1-1.1911.0.9.el8 @System centos-release.x86_64 8.1-1.1911.0.9.el8 @System centos-repos.x86_64 8.1-1.1911.0.9.el8 @System chkconfig.x86_64 1.11-1.el8 @System coreutils.x86_64 8.30-6.el8_1.1 @System coreutils-common.x86_64 8.30-6.el8_1.1 @System cracklib.x86_64 2.9.6-15.el8 @System cronie.x86_64 1.5.2-4.el8 @System cronie-anacron.x86_64 1.5.2-4.el8 @System crontabs.noarch 1.11-16.20150630git.el8 @System crypto-policies.noarch 20190807-1.git9b1477b.el8 @System cryptsetup-libs.x86_64 2.2.0-2.el8 @System [output truncated] [root@guestshell ~]# exit logout [guestshell@guestshell ~]$ exit exit Router#
Submit all command output collected in this section to the relevant TAC SR.
The author would like to thank all members of the Customer Experience Security Programs (CXSP) and Advanced Security Initiatives Group (ASIG) who provided their expertise for this document. A special note of thanks to Xavier Brouckaert of ASIG whose contributions greatly enhanced the efficacy of the forensic procedures contained in this publication.
Additional information about Cisco software integrity assurance, as well as forensic investigation procedures for other platforms, is available at Cisco Security Tactical Resources:
https://sec.cloudapps.cisco.com/security/center/tacticalresources.x
Step 1 – Create the Cisco IOS XE Device Problem Description
Device Problem Description uploaded to SR
Step 2 – Document the Cisco IOS XE Runtime Environment
Output of listening sockets show commands uploaded to SR
Output of process & integrity show commands uploaded to SR
Output of file systems show commands uploaded to SR
Output of system logs show commands uploaded to SR
All system shell log files uploaded to SR
Output of other show commands uploaded to SR (Optional)
Step 3 – Cisco IOS XE Image File Hash Verification
Output of verify on system image files uploaded to SR
Copy of the running system image file uploaded to SR
Image files with hash inconsistencies uploaded to SR
Step 4 – Verify Digitally Signed Image Authenticity
Output of show software authenticity file uploaded to SR
Output of show software authenticity running uploaded to SR
Step 5 – Core File Generation and Text Memory Section Export
Core file uploaded to SR
Output of verify /sha512 on core file uploaded to SR
Crashinfo file uploaded to SR
Output of verify /sha512 on crashinfo file uploaded to SR
Output of copy system:memory/text uploaded to SR
Output of verify /md5 system:memory/text uploaded to SR
Step 6 – Container Enumeration
Output of all show, guestshell, and dnf commands uploaded to SR
| Version | Date | Author | Comments |
|---|---|---|---|
| 1.0 | 8/19/2019 | Dan Maunz | Initial public release. |
| 1.1 | 11/22/2021 | Dan Maunz | Added container enumeration. |
| 1.2 | 10/21/2022 | Dan Maunz | Validated procedure on Release 17.03.05. |
| 1.3 | 02/02/2023 | Dan Maunz | Added core file procedure. |
This document is part of the Cisco Security portal. Cisco provides the official information contained on the Cisco Security portal in English only.
This document is provided on an “as is” basis and does not imply any kind of guarantee or warranty, including the warranties of merchantability or fitness for a particular use. Your use of the information in the document or materials linked from the document is at your own risk. Cisco reserves the right to change or update this document without notice at any time.
This document is part of the Cisco Security portal. Cisco provides the official information contained on the Cisco Security portal in English only.
This document is provided on an “as is” basis and does not imply any kind of guarantee or warranty, including the warranties of merchantability or fitness for a particular use. Your use of the information in the document or materials linked from the document is at your own risk. Cisco reserves the right to change or update this document without notice at any time.