2. File systems & Process Management

Understanding file systems and process management is foundational for cybersecurity practitioners because nearly every security operation, from digital forensics to incident response, malware analysis, and system hardening, relies on a deep comprehension of how operating systems store data and orchestrate programs. Linux and Windows dominate enterprise environments worldwide, and each implements file systems, process handling, privilege models, and memory management differently. These differences influence attack surfaces, defense strategies, event logging, and forensic visibility. Thus, mastery in these domains directly enhances one’s ability to detect anomalies, defend assets, and reason about system behaviors at both the user-space and kernel-space levels.

This chapter provides a comprehensive, master-level exploration of file system structures, metadata, allocation strategies, permissions, journaling, and process lifecycle mechanics across Linux and Windows. It includes conceptual principles from Computer Security: Principles and Practice and Network Security Essentials by Stallings, practical knowledge from the Security+ Study Guide by Chapple, and foundational elements tied to computation and data structures found in Understanding Cryptography by Paar & Pelzl when relevant (e.g., hashing, integrity, timestamps).

File Systems: Foundations and Security Implications

Goals of a File System

A file system provides the fundamental abstraction allowing users and applications to store, organize, and retrieve data. From a security perspective, file systems must support:

• Data confidentiality (access control, encryption-at-rest)
• Integrity (journaling, checksums, digital signatures)
• Availability (fault tolerance, redundancy, recovery mechanisms)
• Forensic traceability (timestamps, metadata, audit logging)
• Efficient performance (caching, indexing, allocation strategies)

Both Linux and Windows incorporate these objectives but do so with different design philosophies.

Linux File Systems (EXT4, XFS, Btrfs)

EXT4 Overview

EXT4 is the default file system for many Linux distributions. Its design emphasizes performance, reliability, and efficient handling of large volumes.

Components:

• Superblock: Stores file system configuration, size, UUID, and critical metadata.
• Inodes: Data structures containing information about files (metadata), but not file names.
• Directories: Map human-readable filenames to inode numbers.
• Data Blocks/Extents: The actual content of files, stored in contiguous ranges for speed.
• Journaling: A write-ahead log that records metadata updates before committing them, enhancing integrity during crashes.

Security implications:

• Inodes allow granular control of permissions (owner, group, others).
• Timestamps (atime, mtime, ctime, btime) provide valuable forensic clues.
• EXT4 supports extended attributes (xattrs), useful for Access Control Lists (ACLs), SELinux labels, and security contexts.
• Journaling mitigates corruption but creates data traces attackers may attempt to hide, analysts must understand journal layout for forensic recovery.

XFS and Btrfs

XFS: High-performance journaling file system

• Optimized for parallel I/O, common in large enterprise systems.
• Uses B+ tree indexing extensively for directories and allocation.
• Supports quotas, snapshots (via LVM), and high scalability.

Btrfs: Modern, copy-on-write (CoW) file system

• Built-in snapshotting and subvolume support.
• Checksums on data and metadata for integrity.
• CoW semantics provide strong forensic visibility, as prior versions of files may persist until space is reclaimed.

Linux File Permissions & Security Models

Traditional Unix permissions:

Each file has:

• Owner
• Group
• Others

With permissions:

• Read (r), Write (w), Execute (x)

Special permission bits:

• Setuid (SUID): Executes file with owner's privileges (common target for privilege escalation).
• Setgid (SGID): Inherits group permissions.
• Sticky bit: Restricts deletion in shared directories (e.g., /tmp).

ACLs and SELinux/AppArmor:

Linux supports advanced access control:

• ACLs provide per-user/per-group granular restrictions.
• SELinux/AppArmor enforce mandatory access controls (MAC), confining processes to least privilege sandboxes.

Understanding these systems is critical for incident responders who must determine whether file access violations are configuration issues or malicious behavior.

Windows File Systems (NTFS, ReFS)

NTFS

NTFS (New Technology File System) is the core Windows file system. It is designed for enterprise-grade reliability, granular permissions, and journaling.

Components:

• Master File Table (MFT): The heart of NTFS; every file is represented as a record.
• Attributes model: Files are stored as attributes (filename, timestamps, ACL, data).
• Alternate Data Streams (ADS): Allow multiple data streams per file, commonly exploited by malware for stealthy storage.
• USN Journal: Logs file changes at the volume level, crucial for forensic reconstructions.
• Access Control Lists (ACLs): NTFS supports very fine-grained permissions based on Windows Security Identifiers (SIDs).

Security implications:

• ADS can hide payloads, scripts, or configuration data.
• NTFS logs extensive metadata, useful for IR analysts.
• Permission inheritance models can lead to privilege creep if poorly managed.

ReFS (Resilient File System)

• Optimized for fault tolerance and integrity.
• Uses CoW-style data handling and metadata checksums.
• Primarily used in virtualization clusters and enterprise storage.

While less common on endpoints, its immutability properties reduce ransomware impact, making it important for modern defensive strategies.

File System Forensics & Security Monitoring

Analyzing file systems is a core competency in cybersecurity. Practitioners must understand:

Forensic artifacts:

• Timestamps (MACB)
• Journals (EXT4 journal, NTFS USN journal)
• Inode allocation patterns
• Slack space and unallocated space
• Volume shadow copies (Windows)
• Snapshots and CoW remnants (Btrfs, ZFS)

Threat behaviors:

• Malware hiding in ADS (Windows)
• Manipulation of SUID binaries (Linux)
• Timestamp tampering (“Timestomping”)
• Deletion artifacts recoverable via inode/MFT allocation patterns

Mastery of these concepts gives defenders the ability to detect, reverse-engineer, and mitigate sophisticated intrusions.

Process Management Concepts

Process management is central to system security. Attackers manipulate processes to maintain persistence, elevate privileges, or evade detection. Understanding the lifecycle and attributes of processes in Linux and Windows is indispensable.

Linux Process Management

Processes, Threads & Scheduling

Linux uses a hybrid scheduler based on the Completely Fair Scheduler (CFS). Key terms include:

• PID: Process ID
• PPID: Parent process ID
• UID/GID: User/group identifiers tied to privilege
• Threads: Lightweight processes sharing the same memory space
• Process states: Running, Sleeping, Stopped, Zombie

Process creation: fork(), exec():

• fork() duplicates a process
• exec() replaces a process image with a new program

This model enables efficient process isolation and job control.

Memory Management

Linux virtual memory is divided into:

• User space
• Kernel space
• Heap & Stack segments
• Shared libraries
• Memory-mapped files

Security mechanisms like ASLR (Address Space Layout Randomization) mitigate memory corruption exploits by making memory addresses unpredictable.

Process Monitoring Tools

Common commands:

• ps, top, htop
• pstree
• systemctl for services
• strace for syscall tracing
• lsof for open file handles (critical in investigations)

Security considerations

• Attackers frequently run processes under benign names.
• Cron jobs and systemd services may be used for persistence.
• Kernel-level processes may indicate rootkits.
• Overprivileged processes violate least privilege design.

Windows Process Management

Process Architecture

Windows uses a process/thread model with:

• Handles
• Access tokens
• Integrity levels
• Job objects
• Win32 subsystems
• DLL-based modular architecture

System components

• Task Manager / Process Explorer
• Winlogon, LSASS, csrss, smss
• Service Control Manager (SCM)

Compromise of certain processes, especially LSASS, is catastrophic, as they contain credential materials.

Windows Memory & Security Controls

Windows implements:

• Virtual memory segmentation
• Kernel Patch Protection (PatchGuard)
• Driver signing enforcement
• Data Execution Prevention (DEP)
• Control Flow Guard (CFG)

Security practitioners must understand where credentials, tokens, and sensitive objects reside in memory, particularly when detecting credential dumping techniques (e.g., Mimikatz).

Malicious Process Behavior & Detection

Common attack behaviors:

• Process injection (DLL injection, process hollowing, PE injection)
• Suspicious parent-child process relationships (e.g., Office → PowerShell)
• Privileged process escalation
• Stealth execution via LSASS tampering or WMI persistence
• Disabling of security tools via targeted process termination

Defensive strategies:

• Monitoring via Sysmon (Windows)
• eBPF tracing (Linux)
• Behavior-based detection
• Least privilege assignment
• Sandboxing applications
• Kernel integrity monitoring

Integrating File Systems & Processes in Cyber Defense

Both domains interact deeply:

• Malware writes files → spawns processes → modifies metadata → establishes persistence.
• Ransomware encrypts files → alters ACLs → terminates processes (e.g., backup services).
• Forensic analysts correlate file timestamps with process creation logs.
• SIEM solutions aggregate filesystem and process telemetry for anomaly detection.

A defender’s effectiveness depends on being able to transition seamlessly between these layers.

Understanding file systems and process management is indispensable for any cybersecurity professional. These disciplines reveal how data is structured, how programs execute, how permissions are enforced, and how attackers manipulate systems to hide, persist, escalate, or exfiltrate. A practitioner who masters EXT4, NTFS, journaling systems, metadata structures, memory management, scheduling, and process lifecycle can identify anomalies that less-trained defenders would miss.

This knowledge is foundational for advanced topics including digital forensics, malware analysis, intrusion detection, system hardening, and incident response.

In cybersecurity, file systems and processes are where theoretical security concepts meet the real-world behavior of operating systems. Mastery of these topics enables professionals to build strong defenses, investigate sophisticated threats, and safeguard critical digital assets across diverse environments.