From Zero to Hero With z/OS - Jack Fitzsimons

From Zero to Hero With z/OS

Speakers: Jack Fitzsimons, Security Consultant, DNV

Conference: Disobey

YouTube: https://www.youtube.com/watch?v=lCTOTLmPRLM

Overview

In his Disobey talk, "From Zero to Hero With z/OS," Jack Fitzsimons, a security consultant at DNV, tackles the persistent misconception that mainframes are obsolete relics of a bygone era. Fitzsimons passionately argues that not only are mainframes far from dead, but they remain critical infrastructure underpinning a vast array of modern financial, governmental, and logistical systems. His presentation serves as a crucial introduction for security professionals unfamiliar with this powerful, yet often overlooked, attack surface.

The core of Fitzsimons' talk is to demystify the IBM z/OS environment, providing essential background on its architecture, key concepts, and common security mechanisms. He aims to equip attendees with practical knowledge, covering everything from fundamental terminology to specific tools and techniques used in mainframe penetration testing. By demonstrating how prevalent misconfigurations and default settings can lead to significant vulnerabilities, Fitzsimons underscores the urgent need for greater attention to mainframe security within the broader cybersecurity landscape.

This article delves into the technical intricacies highlighted by Fitzsimons, exploring the unique challenges and opportunities for security assessments in the z/OS world. It unpacks the mainframe's continued relevance, dissects its core components, details the methodologies for identifying and exploiting weaknesses, and outlines actionable defensive strategies. The talk ultimately positions mainframes not as an ancient curiosity, but as a vital and vulnerable component of global digital infrastructure demanding expert security scrutiny.

Background

▶ Watch: Introduction and talk agenda (0:00)

The perception of mainframes as outdated technology is a persistent myth that Jack Fitzsimons immediately dispels. Many envision the sprawling, black-and-white images of early IBM 360 systems from the 1950s, complete with magnetic tape drives and punch card operators. However, Fitzsimons highlights the stark reality: mainframes are highly advanced, continuously evolving machines. The modern IBM Z16, released in mid-2022, is a testament to this, boasting up to 28 cores, 64 terabytes of RAM (not storage), and integrated AI capabilities for tasks like real-time fraud detection. These machines are the backbone for 71% of Fortune 500 companies, processing an astronomical volume of transactions daily – far exceeding the peak loads of events like Black Friday.

The relevance of mainframes is further underscored by their unparalleled reliability, often achieving "five to seven nines" of uptime, translating to as little as 9 seconds of downtime per year. This level of availability is critical for sectors like banking, airlines, and government services, where even minutes of outage can incur massive costs and disruption. IBM also maintains a rigorous update cycle for z/OS, releasing new versions roughly every 2 to 2.5 years, ensuring the platform remains current and secure. Fitzsimons illustrates this pervasiveness with a simple question: "Have you used a mainframe in the last week?" The answer, for most, is an unwitting "yes," whether flying, making a payment, or interacting with a large institution.

Fitzsimons' own journey into mainframe security, approximately three years prior to the talk, began with an unexpected assignment in Norway. Tasked with translating Norwegian mainframe application screens and conducting security testing, he quickly recognized the unique challenges and the significant, yet often overlooked, security posture of these systems. This personal experience cemented his belief that mainframes represent a critical attack surface that warrants far more attention from the security community than it currently receives. He also shares anecdotal evidence of modernization failures, such as the UK's DVLA system, which still requires nightly shutdowns due to batch job complexities, and Norway's NAV department, which has repeatedly pushed back its mainframe off-boarding deadlines, now targeting 2030. These examples highlight the embedded nature of mainframes and the difficulty of migrating away from them, reinforcing their continued importance and the necessity of securing them effectively.

Key Findings

▶ Watch: Modern IBM Z7 mainframe capabilities and power (4:49)

Fitzsimons' talk reveals several critical findings that challenge conventional wisdom regarding mainframe security and highlight significant vulnerabilities often present in z/OS environments.

Firstly, despite their pivotal role in critical infrastructure, mainframes are not inherently secure by default. Their security posture heavily relies on the implementation and configuration of External Security Managers (ESMs) like RACF, ACF2, or Top Secret. Many organizations either misconfigure these ESMs or fail to fully leverage their capabilities, leaving significant gaps.

Secondly, common misconfigurations are rampant and often easily exploitable. Fitzsimons points out the dangers of leaving RACF in Warning Mode, which logs access attempts without enforcing restrictions, effectively masking potential unauthorized activity. Similarly, liberal use of the Global Access Checking Table (GAK), which grants access without logging, can hide widespread unauthorized data access. Weak RACF password policies (e.g., short length, no case sensitivity, no history) combined with common practices like linking Active Directory (AD) passwords directly to RACF accounts create a fertile ground for credential compromise.

Thirdly, applications running on mainframes, particularly those accessed via 3270 terminals and CICS (Customer Information Control System), frequently suffer from client-side validation weaknesses. The nature of 3270 communication, where the client receives all screen data and then applies display/edit rules, means that malicious actors can easily bypass these front-end controls using proxy tools, leading to unauthorized actions or data manipulation.

Finally, Fitzsimons demonstrates that standard penetration testing methodologies and tools are surprisingly effective against mainframes. Simple reconnaissance, including internal documentation review, often yields critical information like credentials or system architecture. Furthermore, the ability to download RACF database backups, even with limited privileges, allows for offline cracking of user hashes, bypassing account lockout mechanisms and often yielding privileged credentials due to weak default password policies and outdated hashing algorithms like DESZ. These findings collectively underscore that mainframes, despite their unique architecture, are susceptible to many of the same security failings found in other enterprise systems, yet they often receive less scrutiny.

Technical Deep Dive

▶ Watch: Mainframes are everywhere, used daily (5:30)

Securing a z/OS environment requires a nuanced understanding of its distinct architectural components and security mechanisms. Fitzsimons meticulously breaks down these elements, emphasizing their security implications.

At the core of z/OS security lies the External Security Manager (ESM). IBM's RACF (Resource Access Control Facility) dominates the market (75%), with Broadcom's ACF2 and Top Secret making up the remainder. Crucially, only one ESM can be active at a time. RACF operates as a database of classes (collections of profiles) and profiles (collections of Access Control Lists, or ACLs, and other attributes), akin to Active Directory for the mainframe. Access types include NONE, EXECUTE, READ, UPDATE, and ALTER. A particularly dangerous feature is Warning Mode, used for debugging, which logs access attempts without denying them. If left enabled, it can conceal unauthorized activity. Universal Access Control (UACC) grants resource access to all defined and undefined user IDs, posing a significant risk if not carefully managed. The Secure Authorization Facility (SAF) acts as an intermediary between subsystems and the ESM, passing requests. While it can be configured to handle requests independently, this is rarely seen and generally ill-advised. The Global Access Checking Table (GAK) allows frequently accessed resources to bypass RACF checks, improving performance. However, it only grants access, does not deny, and critically, access through GAK is not logged, creating a blind spot for defenders.

RACF also defines Special Privileges:

• SPECIAL: Users can define security profiles, user IDs, and set global RACF options (e.g., password case sensitivity, history).
• OPERATIONS: Grants read, write, and delete access to most data on the mainframe.
• AUDIT: Allows display of all RACF security information and setting audit controls.
• ROAUDIT: A read-only version of AUDIT.

Beyond RACF, the mainframe manages data through Data Sets, which can be thought of as files. They are flat in structure, grouped collections of records, identified by qualifiers (e.g., HOME.JACK.DOCS.DISOBEY). A Partitioned Data Set (PDS) is analogous to a directory containing multiple files.

Job Control Language (JCL) is not a programming language but a command language used to submit batch jobs to the mainframe. Fred Brooks, who oversaw its creation, famously called it "the worst computer programming language ever devised." JCL jobs take data and a program, perform an operation, and produce output, similar to cron jobs in Linux but often running over days or weeks. An example JOB card includes parameters like CLASS (job queue), NOTIFY (user to alert upon completion), MSGLEVEL (logging output), and TIME (execution limit).

The Time Sharing Option (TSO) is the mainframe's equivalent of a Unix bash shell, allowing users to interact directly with the system. It's used to query information, manage data sets, and run scripts. Key TSO commands include LU (list user), LG (list group), RV (reveal RACF database location), and SETROPTS LIST (display password settings and other RACF global options).

Unix System Services (USS), now often referred to as z/OS Linux, provides a Unix-like environment. However, its security is still governed by RACF, meaning traditional Unix permissions can be overridden. Thus, achieving "root" in USS doesn't automatically grant full control over the entire mainframe. Tools like RLIST UXMAP U0 can list RACF users with superuser privileges in USS. Fitzsimons recommends Phil Young's Defcon 33 talk for in-depth privilege escalation techniques in z/OS Unix.

File Transfer Protocol (FTP) is also provided by USS, enabling file transfers to and from the mainframe. Crucially, FTP can be used to submit JCL directly to the internal reader by specifying SITE FILETYPE JEZ, allowing for remote job submission.

The Customer Information Control System (CICS) provides a transaction-based graphical user interface (GUI) for applications, often written in COBOL, Java, or C++. Users interact via 3270 clients (acting as a "browser"). CICS transactions are typically four-letter codes, like URLs. Transactions are categorized: internal use (Category 1), administration/development (Category 2), and end-user (Category 3). Dangerous CICS transactions include:

• CMT (CICS Main Terminal): Can modify CICS resources and shut down CICS entirely.
• CCI (CICS Command Interpreter): Allows reading of unauthorized files and executing code by sending it to the Job Entry Submission (JES) internal reader.
• CEDA: Modifies the running CICS instance and allows adding new transactions, enabling the creation of malicious custom applications.

The Damn Vulnerable CICS Application (DVCA), developed by Phil Young, provides a playground for practicing CICS exploitation.

Finally, DB2 is a standard relational database on the mainframe, utilizing SQL. While it has its own internal security, it's strongly recommended to protect DB2 resources using RACF.

Demo / Proof of Concept

▶ Watch: Introduction to mainframe External Security Managers (ESMs) (7:08)

Fitzsimons presented two compelling demonstrations, illustrating practical exploitation techniques against z/OS mainframes.

The first demo focused on CICS application testing using the Damn Vulnerable CICS Application (DVCA) within a 3270 terminal emulator. The key vulnerability lies in the nature of 3270 communication: the entire screen's data, including hidden or uneditable fields, is sent to the client. The client-side 3270 emulator then applies rules to display or restrict editing. This design flaw allows an attacker to intercept and modify the raw data before it's sent back to the mainframe. Fitzsimons used Hack3270 (a Burp Suite-like proxy for 3270 traffic) to demonstrate this. He showed how to bypass client-side validation to:

1. Purchase a "haunted ancient golden idol" that was marked as unpurchasable. By intercepting the transaction and changing the "allowed" field from 'N' to 'Y', the purchase was successful.
2. Alter the price of a 24K gold MacBook Pro from $22,000 to a more "reasonable" $2,000, and successfully purchase it. This highlights how price manipulation and other data integrity issues can arise from such vulnerabilities.

The second, more impactful, demonstration involved cracking RACF hashes, inspired by a real-world assessment. This scenario showcased a path from limited user access to obtaining privileged credentials. The steps were:

1. Initial Reconnaissance via TSO/ISPF: Starting as a regular user with ROAUDIT (read-only audit) privileges, Fitzsimons used the ISPF (Interactive System Productivity Facility) option 3.15 (a system-wide search function) to search JCL libraries for common keywords like "user" and "pass." This revealed plaintext credentials for a different user (USER3) stored within a JCL member in the Pentest JCL Source library. This highlights the dangers of storing sensitive information directly in data sets.
2. Privilege Escalation via Discovered Credentials: Upon checking USER3's privileges using LU USER3 in TSO, it was discovered that USER3 possessed the OPERATIONS attribute, granting extensive read/write/delete access across most mainframe data.
3. Exfiltrating the RACF Database: Using the newly acquired USER3 credentials, Fitzsimons logged into the mainframe via FTP. He then navigated to the RACF database backup file (SYS1.RACF.DS.BACKUP) and downloaded it to his local machine. He emphasized the crucial step of switching to binary mode in FTP, as mainframes use EBCDIC encoding, which would otherwise result in a garbled mess on an ASCII-based system.
4. Offline Hash Cracking: With the RACFDB file on his local machine, Fitzsimons used rackf2john (a utility to convert RACF database format) and John the Ripper to crack the hashes. The cracking process quickly yielded several user passwords.

Fitzsimons explained why this method is so effective:

• Weak Default Policies: Usernames are limited to 7 characters and passwords to 8 characters by default. Many installations do not enforce case sensitivity, drastically reducing the keyspace for cracking.
• Offline Advantage: Cracking hashes offline bypasses account lockout mechanisms, allowing for unlimited attempts without alerting defenders.
• AD Integration: It's common for organizations to link Active Directory (AD) passwords to RACF, meaning if AD hashes are compromised, they can often be used against the mainframe.
• Outdated Ciphers: Many systems still use weak ciphers like DESZ for password hashing, which is significantly easier to crack than modern alternatives like KDF AES.

These demos vividly illustrate that mainframes are not impenetrable fortresses but are susceptible to common attack vectors exacerbated by legacy design choices and prevalent misconfigurations.

Defensive Implications

▶ Watch: RACKF: The mainframe's "Active Directory" (8:12)

Fitzsimons' talk provides a clear roadmap for mainframe defenders to bolster their security posture, directly addressing the vulnerabilities highlighted.

1. Review and Disable Warning Mode: The RACF Warning Mode should be strictly controlled and disabled when not actively used for debugging. Leaving it active provides a false sense of security, logging unauthorized access attempts without preventing them, effectively masking potential breaches.
2. Audit the Global Access Checking Table (GAK): Resources placed in the GAK bypass standard RACF checks and, critically, are not logged. Defenders must rigorously audit the GAK to ensure only truly public or non-sensitive resources are included. Any sensitive data or applications should be removed from the GAK to ensure all access is properly authorized and logged.
3. Secure FTP Services: Unprotected FTP is a significant vulnerability. Organizations should:

• Disable anonymous FTP access.
• Enforce strong authentication mechanisms.
• Utilize RC files for configuration and restrict access to them.
• Implement strict access controls for files and directories accessible via FTP, especially those containing JCL or sensitive data.
• Monitor FTP activity for suspicious transfers or job submissions (SITE FILETYPE JEZ).

1. Strengthen RACF Password Policies: The ease of cracking RACF hashes highlights the need for robust password controls:

• Enforce case sensitivity for passwords.
• Increase minimum password length significantly beyond the default 8 characters.
• Implement password history to prevent reuse of old passwords.
• Utilize stronger hashing algorithms like KDF AES instead of the outdated DESZ.
• Deploy Multi-Factor Authentication (MFA) for all RACF accounts, especially privileged ones, as Fitzsimons noted its rare adoption.

1. Reconsider Active Directory Integration: While convenient, linking AD passwords directly to RACF without mirroring robust AD security policies (e.g., MFA, strong password requirements) creates a single point of failure. If AD is compromised, mainframe access is immediately at risk. Review and strengthen such integrations, or consider independent, stronger RACF password policies.
2. Review and Restrict CICS Transactions: CICS applications are prone to client-side validation bypasses. Defenders should:

• Rigorously audit and restrict access to powerful CICS transactions like CMT, CCI, and CEDA. Regular users should not have access to these.
• Remove or disable unused default transactions.
• Implement server-side validation for all CICS application inputs to prevent manipulation of hidden or uneditable fields.
• Regularly review custom CICS applications for vulnerabilities.

1. Conduct Regular Security Assessments and Penetration Tests: Mainframes are not immune to security flaws. Regular, specialized penetration tests are crucial to identify misconfigurations and vulnerabilities before attackers do. This should include reviewing internal documentation, as it often contains exposed credentials or critical system information.
2. Gentle Nmap Scanning: When performing network reconnaissance, be aware that mainframes can have fragile subsystems. While the mainframe itself is robust, specific services might crash under aggressive scanning. Adopt a "gentle" approach to avoid disrupting production systems.
3. Monitor for Privileged Access and Audit Trails: Implement comprehensive monitoring for activities related to privileged users (e.g., those with SPECIAL or OPERATIONS attributes) and ensure all security-relevant events are logged and reviewed. This includes scrutinizing GAK activity if it cannot be completely avoided.

By implementing these defensive measures, organizations can significantly reduce their mainframe attack surface and protect these critical systems from compromise.

Key Takeaways

• Mainframes are critical and relevant, not dead: They underpin vital global infrastructure (71% of Fortune 500, daily transactions, 5-7 nines uptime) and are actively updated, demanding serious security attention.
• Default z/OS security is minimal; ESMs are crucial: External Security Managers like RACF are the primary security control, but their effectiveness depends entirely on proper configuration and ongoing management.
• RACF misconfigurations are common entry points: Weak password policies (e.g., 8-character default, no case sensitivity, DESZ ciphers), enabled Warning Mode, and unlogged GAK entries create significant vulnerabilities.
• 3270 applications often rely on client-side validation: The nature of 3270 communication allows proxy tools (like Hack3270) to bypass front-end controls, leading to unauthorized actions or data manipulation in CICS applications.
• Offline RACF hash cracking is highly effective: Due to default password limitations, lack of case sensitivity, and common AD integration, downloading and cracking RACF database backups can quickly yield privileged credentials.
• Standard pentesting methodology applies to mainframes: Reconnaissance (documentation review), network scanning (Nmap), and credential-based attacks are highly effective, underscoring that mainframes are not an entirely unique security challenge, but rather an often-overlooked one.

About the Speaker(s)

Jack Fitzsimons is a Security Consultant at DNV, specializing in mainframe and cloud security, which currently constitutes about 95% of his work. Originally from the UK, he has lived in Sweden and has been based in Denmark for approximately nine years. Fitzsimons began his career as a systems administrator focusing on cloud technologies before transitioning to a security consultant role about three years prior to this talk. His unique journey into mainframe security started unexpectedly with a project in Norway, involving the translation and testing of Norwegian mainframe applications. This experience ignited his passion for mainframe security, leading him to become a vocal advocate for increased awareness and scrutiny of these critical systems.