Security: Threats and Defenses

Operating system security aims to protect system resources from unauthorized access, malicious modification, and destruction. This article examines types of security threats, attack techniques, cryptography basics, authentication methods, and system-level defense mechanisms.

1. Security Overview

The CIA Triad

          Confidentiality
          ╱     ╲
         ╱  Security╲
        ╱   Goals    ╲
       ╱               ╲
 Availability ─────── Integrity

Element	Description	Threat Examples
Confidentiality	Only authorized users access info	Eavesdropping, leaks
Integrity	Info modified only by authorized means	Tampering, forgery
Availability	Authorized users can access when needed	DoS attacks, ransomware

Security Threat Layers

┌──────────────────────────────────────┐
│            Attack Surface            │
│                                      │
│  Physical: Direct server access, USB │
│  │                                   │
│  Network: Remote attacks, sniffing   │
│  │                                   │
│  OS: Privilege escalation, kernel    │
│  │   exploits                        │
│  │                                   │
│  Application: Code injection, XSS   │
│  │                                   │
│  Social engineering: Phishing,       │
│  spear phishing                      │
└──────────────────────────────────────┘

2. Program Threats

Malware Classification

┌────────────────────────────────────────────────┐
│              Malware Classification             │
│                                                │
│  ┌─────────┐ ┌─────────┐ ┌─────────┐          │
│  │ Virus   │ │  Worm   │ │ Trojan  │          │
│  │         │ │         │ │ Horse   │          │
│  │Needs    │ │Self-    │ │Disguised│          │
│  │host     │ │propagate│ │         │          │
│  └─────────┘ └─────────┘ └─────────┘          │
│                                                │
│  ┌─────────┐ ┌─────────┐ ┌─────────┐          │
│  │Ransomw. │ │ Spyware │ │ Rootkit │          │
│  │         │ │         │ │         │          │
│  │Encrypts │ │Collects │ │Hides    │          │
│  │files    │ │info     │ │presence │          │
│  └─────────┘ └─────────┘ └─────────┘          │
└────────────────────────────────────────────────┘

Code Injection

An attack that executes malicious code through external input.

// SQL 인젝션 예시 - 취약한 코드
void login(const char *username, const char *password) {
    char query[512];
    // 위험: 사용자 입력을 직접 쿼리에 삽입
    sprintf(query, "SELECT * FROM users WHERE name='%s' AND pass='%s'",
            username, password);
    execute_query(query);
}

// 공격 입력:
// username: admin' --
// password: (아무 값)
// 생성되는 쿼리:
// SELECT * FROM users WHERE name='admin' --' AND pass='...'
// -- 이후는 주석 처리되어 비밀번호 검증 우회

// 방어: 준비된 문장(Prepared Statement) 사용
void login_safe(const char *username, const char *password) {
    // 매개변수화된 쿼리 - 입력이 데이터로만 처리됨
    prepare_statement("SELECT * FROM users WHERE name=? AND pass=?");
    bind_parameter(1, username);
    bind_parameter(2, password);
    execute_prepared();
}

Buffer Overflow

A classic yet still dangerous attack technique.

// 취약한 코드 예시
void vulnerable_function(const char *input) {
    char buffer[64];
    // 위험: 입력 길이를 확인하지 않음
    strcpy(buffer, input);  // 64바이트 이상 입력 시 오버플로
}

Stack structure (before overflow):
┌──────────────────┐ High address
│ Return address   │
├──────────────────┤
│ Previous frame   │
│ pointer          │
├──────────────────┤
│ buffer[64]       │
│ (64 bytes)       │
├──────────────────┤ Low address

Stack structure (after overflow):
┌──────────────────┐ High address
│ Malicious code   │ ← Return address overwritten!
│ address          │
├──────────────────┤
│ AAAAAAAAAAAAAAAA │ ← Overwritten
├──────────────────┤
│ AAAAAAAAAAAAAAAA │ ← Attack data
│ (over 64 bytes)  │
├──────────────────┤ Low address

// 방어: 안전한 문자열 함수 사용
void safe_function(const char *input) {
    char buffer[64];
    // 길이를 제한하는 안전한 복사
    strncpy(buffer, input, sizeof(buffer) - 1);
    buffer[sizeof(buffer) - 1] = '\0';
}

3. System and Network Threats

Worms

Self-replicating programs that spread automatically over networks.

Infected Host A
    │
    ├── Network scan (find vulnerable hosts)
    │
    ├── Infect Host B (exploit vulnerability)
    │   ├── Infect Host D
    │   └── Infect Host E
    │
    └── Infect Host C (exploit vulnerability)
        ├── Infect Host F
        └── ...

Exponential propagation → massive infection in short time

Port Scanning

Attacker                        Target Server
  │                                │
  ├── SYN → Port 22 ──────────────→│
  │←──────────── SYN-ACK ─────────│  Port 22: Open (SSH)
  │                                │
  ├── SYN → Port 23 ──────────────→│
  │←──────────── RST ─────────────│  Port 23: Closed
  │                                │
  ├── SYN → Port 80 ──────────────→│
  │←──────────── SYN-ACK ─────────│  Port 80: Open (HTTP)
  │                                │
  ├── SYN → Port 443 ─────────────→│
  │←──────────── SYN-ACK ─────────│  Port 443: Open (HTTPS)
  ...

DoS (Denial of Service) Attacks

Normal request:
Client → [Normal request] → Server → [Normal response]

DoS attack:
Attacker → [Massive requests] → Server (overloaded) → Normal users blocked

DDoS (Distributed DoS):
Botnet host 1 ─→ ┐
Botnet host 2 ─→ ├→ Server (overloaded)
Botnet host 3 ─→ ┤
    ...        ─→ ┘

SYN Flood attack:
Attacker sends massive SYN packets, ignores SYN-ACK responses
→ Server connection queue exhausted

4. Cryptography

Symmetric Key Encryption

A single key is used for both encryption and decryption.

Sender                              Receiver
┌──────┐    ┌──────────┐    ┌──────┐
│Plain │─→ │Encrypt   │─→ │Cipher│─→ ... ─→ ┌──────────┐─→ ┌──────┐
│Hello │    │(secret   │    │X7k9p│         │Decrypt   │   │Hello │
└──────┘    │ key)     │    └──────┘         │(same key)│   └──────┘
            └──────────┘                     └──────────┘

Algorithms: AES (128/192/256-bit), ChaCha20
Problem: Key exchange - how to share the key securely?

Asymmetric Key Encryption (Public Key)

Uses a pair of public and private keys.

Receiver's key pair:
┌──────────┐  ┌──────────┐
│ Public   │  │ Private  │
│ Key      │  │ Key      │
│ (public) │  │ (secret) │
└──────────┘  └──────────┘

Encryption:
Sender: Plaintext + Receiver public key → Ciphertext

Decryption:
Receiver: Ciphertext + Receiver private key → Plaintext

Digital Signature:
Signer: Hash + Signer private key → Signature
Verifier: Signature + Signer public key → Hash verification

Algorithms: RSA, ECC (Elliptic Curve Cryptography)

Hash Functions

Converts arbitrary-length input to fixed-length output.

Input                    SHA-256 Hash
"Hello"       →  185f8db32271fe25f561a6fc938b2e26
                 4306ec304eda518007d1764826381969

"Hello!"      →  334d016f755cd6dc58c53a86e183882f
                 8ec14f52fb05345887c8a5edd42c87b7

Properties:
- One-way: Cannot recover original from hash
- Collision resistance: Hard to find different inputs with same hash
- Avalanche effect: 1-bit input change causes large hash change

Uses: Password storage, integrity verification, digital signatures
Algorithms: SHA-256, SHA-3, bcrypt (for passwords)

// 비밀번호 해싱 예시 (의사 코드)
#include <openssl/sha.h>
#include <string.h>
#include <stdio.h>

void hash_password(const char *password, const char *salt,
                   unsigned char *hash_out) {
    char salted[256];
    // 솔트(salt)를 추가하여 레인보우 테이블 공격 방지
    snprintf(salted, sizeof(salted), "%s%s", salt, password);

    SHA256((unsigned char *)salted, strlen(salted), hash_out);
}

int verify_password(const char *input, const char *salt,
                    const unsigned char *stored_hash) {
    unsigned char computed[SHA256_DIGEST_LENGTH];
    hash_password(input, salt, computed);
    return memcmp(computed, stored_hash, SHA256_DIGEST_LENGTH) == 0;
}

5. User Authentication

Authentication Method Classification

┌─────────────────────────────────────────────┐
│         Authentication Factors               │
│                                             │
│  ┌───────────┐ ┌───────────┐ ┌───────────┐ │
│  │ Knowledge │ │Possession │ │ Biometric │ │
│  │ (something│ │(something │ │(something │ │
│  │  you know)│ │ you have) │ │ you are)  │ │
│  │           │ │           │ │           │ │
│  │ Password  │ │ Security  │ │Fingerprint│ │
│  │ PIN       │ │ token     │ │ Iris      │ │
│  │ Security  │ │ Smart card│ │ Face      │ │
│  │ question  │ │ OTP       │ │ recogn.   │ │
│  └───────────┘ └───────────┘ └───────────┘ │
│                                             │
│  Multi-Factor Authentication (MFA):         │
│  Combines two or more factors for           │
│  enhanced security                          │
└─────────────────────────────────────────────┘

Password Security

Vulnerable password storage:
┌──────────────────────────────────┐
│ users table                      │
│ username │ password              │
│ admin    │ admin123     ← plain!│
│ user1    │ password1    ← risky!│
└──────────────────────────────────┘

Secure password storage:
┌──────────────────────────────────────────────┐
│ users table                                  │
│ username │ salt     │ hashed_password        │
│ admin    │ x7k2m9   │ a3f2e8...  (hash)     │
│ user1    │ p4q8n1   │ b7d1c5...  (hash)     │
└──────────────────────────────────────────────┘
→ Salt + slow hash (bcrypt, Argon2)

6. Security Defenses

Firewall

External Network (Internet)
       │
  ┌────┴─────────────────┐
  │      Firewall         │
  │                      │
  │  Example rules:      │
  │  Allow: TCP 80 (HTTP)│
  │  Allow: TCP 443      │
  │  Block: TCP 23       │
  │  Block: all others   │
  └────┬─────────────────┘
       │
  Internal Network (LAN)
  ┌────┴────┬────────┐
  │         │        │
 Srv 1    Srv 2    Srv 3

Intrusion Detection System (IDS)

Network Traffic
      │
      ▼
┌─────────────────────┐
│    IDS Engine        │
│                     │
│  Signature-based:   │
│  Match known attack │
│  patterns           │
│                     │
│  Anomaly-based:     │
│  Learn normal       │
│  patterns, detect   │
│  abnormal behavior  │
│                     │
│  On match → Alert   │
└─────────────────────┘
      │
      ▼
  Alert to administrator
  (or IPS auto-blocks)

ASLR (Address Space Layout Randomization)

A technique that makes buffer overflow attacks harder.

Without ASLR (fixed):              With ASLR (random):
┌──────────────────┐            ┌──────────────────┐
│ 0x08048000 code  │            │ 0x55a3b000 code  │ ← Different each time
│ ...              │            │ ...              │
│ 0x0804A000 data  │            │ 0x55a3d000 data  │
│ ...              │            │ ...              │
│ 0xBFFF0000 stack │            │ 0x7FFD2000 stack │ ← Different each time
│ ...              │            │ ...              │
│ 0xB7E00000 libs  │            │ 0x7F4A1000 libs  │
└──────────────────┘            └──────────────────┘
Attacker can predict addresses   Attacker cannot predict

Additional Defense Techniques

┌─────────────────────────────────────────┐
│        Security Defense Techniques       │
│                                         │
│  Stack Canary:                          │
│  ┌──────────┐                           │
│  │ Return   │                           │
│  │ address  │                           │
│  ├──────────┤                           │
│  │ Canary   │ ← Verified on return      │
│  │ value    │   Changed → program abort │
│  ├──────────┤                           │
│  │ buffer   │                           │
│  └──────────┘                           │
│                                         │
│  DEP (Data Execution Prevention):       │
│  Prohibits code execution in stack/heap │
│  W^X: Writable XOR Executable           │
│                                         │
│  CFI (Control Flow Integrity):          │
│  Forces program control flow to follow  │
│  only expected paths                    │
└─────────────────────────────────────────┘

7. Summary

Program Threats: Malware (viruses, worms, ransomware), code injection, buffer overflow
Network Threats: Port scanning, DoS/DDoS attacks, man-in-the-middle attacks
Cryptography: Symmetric key (AES), asymmetric key (RSA), hash (SHA-256)
Authentication: Knowledge/possession/biometric-based, multi-factor authentication (MFA)
Defenses: Firewall, IDS/IPS, ASLR, DEP, stack canary, CFI

Quiz: Security

Q1. Describe three techniques that defend against buffer overflow attacks.

A1. (1) ASLR: Randomizes memory address layout, making it difficult for attackers to predict target addresses. (2) Stack Canary: Places a sentinel value before the return address to detect overflow when it is modified. (3) DEP (Data Execution Prevention): Prohibits code execution in stack or heap regions, preventing injected code from running.

Q2. What is the difference between symmetric and asymmetric key encryption?

A2. Symmetric key encryption uses a single key for both encryption and decryption, making it fast, but has the problem of securely exchanging the key. Asymmetric key encryption uses a public/private key pair to solve the key exchange problem, but is computationally slower. In practice, a hybrid approach is used where asymmetric keys exchange a symmetric key, which then encrypts the data.

Q3. Why is salt used when storing passwords?

A3. Adding a different salt for each user means the same password produces different hash values. This neutralizes rainbow table attacks (pre-computed hash dictionaries) and prevents identifying users with the same password by their hash values alone.