Split View: 공업수학 완전 정복 4편: 수치해석 - 공학 문제를 컴퓨터로 푸는 방법
공업수학 완전 정복 4편: 수치해석 - 공학 문제를 컴퓨터로 푸는 방법
공업수학 완전 정복 4편: 수치해석
공학 문제를 해석적으로 풀 수 없을 때, 또는 복잡한 시스템의 수치 시뮬레이션이 필요할 때 **수치해석(Numerical Analysis)**이 필요합니다. 트랜지스터의 비선형 특성 분석, 복잡한 회로망의 과도 응답, 유한요소 해석 등이 모두 수치해석에 기반합니다. 이번 글에서는 핵심 수치 기법을 원리부터 Python 구현까지 완전히 정복합니다.
1. 오차 분석 (Error Analysis)
1.1 오차의 종류
수치 계산에서 "완벽한 해"는 존재하지 않습니다. 오차의 종류를 이해하는 것이 신뢰할 수 있는 수치 계산의 시작입니다.
반올림 오차(Round-off Error): 컴퓨터가 유한 자릿수로 실수를 표현하기 때문에 발생합니다.
IEEE 754 배정도(double): 약 15-16자리 유효숫자
절단 오차(Truncation Error): 무한 급수를 유한 항으로 근사할 때 발생합니다.
절대 오차(Absolute Error):
상대 오차(Relative Error):
백분율 오차:
유효숫자(Significant Figures): 이면 유효숫자가 자리임을 보장합니다:
1.2 수치 안정성 (Numerical Stability)
오차가 계산 과정에서 증폭되지 않아야 합니다.
잘 조건화(Well-conditioned): 입력 오차가 출력에 크게 증폭되지 않음
나쁜 조건화(Ill-conditioned): 입력의 작은 변화가 출력을 크게 변화시킴
조건수(Condition Number):
선형 시스템 에서 가 크면 수치 해가 불안정합니다.
import numpy as np
# 힐버트 행렬 (악명 높은 나쁜 조건 행렬)
n = 6
H = np.array([[1/(i+j+1) for j in range(n)] for i in range(n)])
kappa = np.linalg.cond(H)
print(f"힐버트 행렬 ({n}x{n}) 조건수: {kappa:.2e}")
# 간단한 행렬
A = np.array([[2, 1], [1, 3]], dtype=float)
print(f"간단한 행렬 조건수: {np.linalg.cond(A):.2f}")
# 오차 전파 시연
b_true = np.array([3, 4], dtype=float)
x_true = np.linalg.solve(A, b_true)
# 약간의 오차 추가
b_perturbed = b_true + np.array([0.01, 0.0])
x_perturbed = np.linalg.solve(A, b_perturbed)
print(f"\n입력 오차: {np.linalg.norm(b_perturbed - b_true):.4f}")
print(f"출력 오차: {np.linalg.norm(x_perturbed - x_true):.4f}")
1.3 빅-오 표기와 수렴 차수
수렴 차수(Order of Convergence):
- : 1차 (선형) 수렴
- : 2차 (이차) 수렴 (훨씬 빠름)
- : 초선형 수렴 (할선법)
빅-오 표기:
예: 전진 차분 오차는 , 중앙 차분 오차는
2. 비선형 방정식 풀기
2.1 이분법 (Bisection Method)
원리: 볼차노 정리 — 이면 사이에 근이 존재합니다.
알고리즘:
- 확인
- 중점 계산
- 이면 종료; 이면 ; 아니면
- 이면 종료
수렴: 매 반복마다 구간이 절반으로 줄어 1차 수렴. 번 반복 후 오차:
필요 반복 수:
def bisection(f, a, b, tol=1e-10, max_iter=100):
"""이분법으로 f(x) = 0의 근 찾기"""
if f(a) * f(b) > 0:
raise ValueError("f(a)와 f(b)는 반대 부호여야 합니다")
history = []
for i in range(max_iter):
c = (a + b) / 2
history.append({'iter': i+1, 'a': a, 'b': b, 'c': c,
'f(c)': f(c), 'error': abs(b-a)/2})
if abs(f(c)) < tol or (b - a) / 2 < tol:
print(f"수렴: {i+1}번 반복")
return c, history
if f(a) * f(c) < 0:
b = c
else:
a = c
return (a + b) / 2, history
# 예제: x^3 - 2x - 5 = 0
def equation(x):
return x**3 - 2*x - 5
root, history = bisection(equation, 1, 3)
print(f"근: {root:.10f}")
print(f"검증: f({root:.6f}) = {equation(root):.2e}")
# 수렴 이력 출력
print("\n반복 | 근사값 | 오차")
for h in history[:8]:
print(f" {h['iter']:2d} | {h['c']:.8f} | {h['error']:.2e}")
2.2 뉴턴-랩슨법 (Newton-Raphson Method)
원리: 에서 테일러 1차 근사
기하적 해석: 에서 접선이 x축과 만나는 점이
수렴: 이상적으로 2차(이차) 수렴 — 정확한 자릿수가 매 반복마다 두 배!
한계와 주의사항:
- 이면 발산 (접선이 x축과 평행)
- 수렴이 보장되지 않음 (초기점 선택이 중요)
- 다중근에서 1차 수렴으로 저하됨
def newton_raphson(f, df, x0, tol=1e-10, max_iter=50):
"""뉴턴-랩슨법"""
x = x0
history = [{'iter': 0, 'x': x, 'f(x)': f(x), 'error': float('inf')}]
for i in range(max_iter):
fx = f(x)
dfx = df(x)
if abs(dfx) < 1e-15:
print("경고: 도함수가 0에 가까움, 발산 위험")
break
x_new = x - fx / dfx
error = abs(x_new - x)
history.append({'iter': i+1, 'x': x_new,
'f(x)': f(x_new), 'error': error})
if error < tol:
print(f"수렴: {i+1}번 반복")
return x_new, history
x = x_new
return x, history
# f(x) = x^3 - 2x - 5, f'(x) = 3x^2 - 2
def f(x):
return x**3 - 2*x - 5
def df(x):
return 3*x**2 - 2
root, history = newton_raphson(f, df, x0=2.0)
print(f"뉴턴-랩슨 근: {root:.10f}")
print("\n반복 | 근사값 | 오차")
for h in history:
err_str = f"{h['error']:.2e}" if h['error'] != float('inf') else " -"
print(f" {h['iter']:2d} | {h['x']:.10f} | {err_str}")
수렴 속도 비교:
import numpy as np
import matplotlib.pyplot as plt
# 이분법 vs 뉴턴-랩슨 수렴 속도 비교
true_root = 2.0945514815423265 # x^3 - 2x - 5 = 0의 실제 근
_, hist_bisect = bisection(f, 1, 3, tol=1e-12, max_iter=50)
_, hist_newton = newton_raphson(f, df, x0=2.0, tol=1e-12, max_iter=20)
errors_b = [abs(h['c'] - true_root) for h in hist_bisect]
errors_n = [abs(h['x'] - true_root) for h in hist_newton if h['error'] != float('inf')]
plt.figure(figsize=(10, 5))
plt.semilogy(range(1, len(errors_b)+1), errors_b, 'b-o', label='이분법 (1차 수렴)')
plt.semilogy(range(1, len(errors_n)+1), errors_n, 'r-s', label='뉴턴-랩슨 (2차 수렴)')
plt.xlabel('반복 횟수')
plt.ylabel('절대 오차 (로그 스케일)')
plt.title('이분법 vs 뉴턴-랩슨 수렴 속도')
plt.legend()
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.savefig('convergence_comparison.png', dpi=150)
plt.show()
2.3 할선법 (Secant Method)
를 모를 때 유한 차분으로 근사합니다:
수렴 차수: (황금비, 초선형)
def secant_method(f, x0, x1, tol=1e-10, max_iter=50):
"""할선법"""
for i in range(max_iter):
f0, f1 = f(x0), f(x1)
if abs(f1 - f0) < 1e-15:
break
x2 = x1 - f1 * (x1 - x0) / (f1 - f0)
if abs(x2 - x1) < tol:
print(f"수렴: {i+1}번 반복")
return x2
x0, x1 = x1, x2
return x1
root_s = secant_method(f, 1.0, 3.0)
print(f"할선법 근: {root_s:.10f}")
2.4 scipy.optimize 활용
from scipy.optimize import fsolve, brentq, minimize_scalar
import numpy as np
# 단일 방정식 풀기
def equation(x):
return x**3 - 2*x - 5
# 브렌트법 (이분법 + 할선법 + 역 이차 보간 혼합)
root = brentq(equation, 1, 3, xtol=1e-12)
print(f"brentq 근: {root:.12f}")
# fsolve (뉴턴-랩슨 기반)
root_fs = fsolve(equation, 2.0)[0]
print(f"fsolve 근: {root_fs:.12f}")
# 연립 비선형 방정식
from scipy.optimize import fsolve
def system(vars):
x, y = vars
eq1 = x**2 + y**2 - 4 # 원
eq2 = x*y - 1 # 쌍곡선
return [eq1, eq2]
solutions = []
for x0, y0 in [(1, 1), (-1, 1), (1, -1), (-1, -1)]:
sol = fsolve(system, [x0, y0], full_output=True)
if sol[2] == 1: # 수렴 성공
solutions.append(sol[0])
print("\n연립 방정식 해:")
for sol in solutions:
print(f" (x, y) = ({sol[0]:.6f}, {sol[1]:.6f})")
# 회로 방정식: 다이오드 특성 (비선형)
def diode_equation(V, V_s=5, R=1000, I_s=1e-12, n=1, V_T=0.02585):
"""V_s = V + R * I_s * (exp(V/(n*V_T)) - 1)"""
I = I_s * (np.exp(V / (n * V_T)) - 1)
return V_s - V - R * I
V_diode = brentq(diode_equation, 0, 1)
I_diode = 1e-12 * (np.exp(V_diode / 0.02585) - 1)
print(f"\n다이오드 동작점: V = {V_diode:.4f} V, I = {I_diode*1000:.4f} mA")
3. 수치 미분과 적분
3.1 유한 차분 (Finite Difference)
테일러 급수로부터 도함수의 수치 근사를 유도합니다.
전진 차분(Forward Difference):
1차 정확도 (오차가 에 비례).
후진 차분(Backward Difference):
중앙 차분(Central Difference):
빼면:
2차 정확도 — 같은 로 전진 차분보다 훨씬 정확합니다.
2차 도함수:
이것이 열방정식, 파동방정식의 유한 차분 풀이에 사용됩니다.
import numpy as np
import matplotlib.pyplot as plt
def numerical_derivative(f, x, h=1e-5, method='central'):
"""수치 미분"""
if method == 'forward':
return (f(x + h) - f(x)) / h
elif method == 'backward':
return (f(x) - f(x - h)) / h
elif method == 'central':
return (f(x + h) - f(x - h)) / (2 * h)
def f(x):
return np.sin(x)
def df_true(x):
return np.cos(x)
x0 = np.pi / 4
true_deriv = df_true(x0)
# h 크기에 따른 오차 분석
h_values = np.logspace(-16, 0, 100)
errors_fwd = []
errors_cen = []
for h in h_values:
e_fwd = abs(numerical_derivative(f, x0, h, 'forward') - true_deriv)
e_cen = abs(numerical_derivative(f, x0, h, 'central') - true_deriv)
errors_fwd.append(e_fwd if e_fwd > 0 else 1e-17)
errors_cen.append(e_cen if e_cen > 0 else 1e-17)
plt.figure(figsize=(10, 6))
plt.loglog(h_values, errors_fwd, 'b-', linewidth=2, label='전진 차분 O(h)')
plt.loglog(h_values, errors_cen, 'r-', linewidth=2, label='중앙 차분 O(h^2)')
plt.loglog(h_values, h_values, 'b--', alpha=0.5, label='O(h) 기준선')
plt.loglog(h_values, h_values**2, 'r--', alpha=0.5, label='O(h^2) 기준선')
plt.xlabel('h (스텝 크기)')
plt.ylabel('절대 오차')
plt.title('수치 미분 오차 분석 (h에 따른 오차)')
plt.legend()
plt.grid(True, which='both', alpha=0.3)
plt.tight_layout()
plt.savefig('numerical_derivative_error.png', dpi=150)
plt.show()
3.2 수치 적분 (Numerical Integration)
사다리꼴 법칙(Trapezoidal Rule):
여기서 . 오차:
심프슨 1/3 법칙(Simpson's 1/3 Rule):
각 두 구간을 2차 다항식으로 근사:
은 짝수여야 합니다. 오차: — 사다리꼴보다 훨씬 정확!
심프슨 3/8 법칙: 3차 다항식, 오차 , 은 3의 배수.
가우스 구적법(Gaussian Quadrature):
최적의 적분점과 가중치를 선택하여 정확도를 극대화합니다.
점 가우스-르장드르 구적법은 최대 차 다항식을 정확히 적분합니다.
import numpy as np
from scipy import integrate
import matplotlib.pyplot as plt
def trapezoidal(f, a, b, n):
"""사다리꼴 법칙"""
x = np.linspace(a, b, n+1)
h = (b - a) / n
y = f(x)
return h * (y[0]/2 + np.sum(y[1:-1]) + y[-1]/2)
def simpsons(f, a, b, n):
"""심프슨 1/3 법칙 (n은 짝수)"""
if n % 2 != 0:
n += 1
x = np.linspace(a, b, n+1)
h = (b - a) / n
y = f(x)
return h/3 * (y[0] + 4*np.sum(y[1::2]) + 2*np.sum(y[2:-1:2]) + y[-1])
# 예제: integral of sin(x) from 0 to pi = 2
f_test = np.sin
a, b = 0, np.pi
true_val = 2.0
print("수치 적분 정확도 비교")
print(f"정확한 값: {true_val}")
print("-" * 60)
print(f"{'방법':<20} {'n':>6} {'근사값':>15} {'상대 오차':>12}")
print("-" * 60)
for n in [4, 8, 16, 32]:
trap = trapezoidal(f_test, a, b, n)
simp = simpsons(f_test, a, b, n)
print(f"{'사다리꼴':<20} {n:>6} {trap:>15.10f} {abs(trap-true_val)/true_val:>12.2e}")
print(f"{'심프슨':<20} {n:>6} {simp:>15.10f} {abs(simp-true_val)/true_val:>12.2e}")
# scipy.integrate.quad - 적응형 구적법
result, error = integrate.quad(f_test, a, b)
print(f"{'scipy.quad':<20} {'적응형':>6} {result:>15.10f} {'<'+str(error):>12}")
# 어려운 적분: int_0^inf exp(-x^2) dx = sqrt(pi)/2
def gaussian(x):
return np.exp(-x**2)
result_gauss, _ = integrate.quad(gaussian, 0, np.inf)
print(f"\n가우스 적분: {result_gauss:.10f}")
print(f"정확한 값 (sqrt(pi)/2): {np.sqrt(np.pi)/2:.10f}")
# 이중 적분: int_0^1 int_0^1 x*y dx dy = 0.25
def f2d(y, x):
return x * y
result_2d, _ = integrate.dblquad(f2d, 0, 1, 0, 1)
print(f"\n이중 적분: {result_2d:.10f} (정확: 0.25)")
4. ODE 수치 풀이
4.1 오일러법 (Euler's Method)
아이디어: 현재 기울기로 다음 점을 예측
지역 절단 오차: , 전체 오차: (1차 방법)
기하적 해석: 각 점에서 접선 방향으로 만큼 이동
import numpy as np
import matplotlib.pyplot as plt
def euler_method(f, x0, y0, h, x_end):
"""오일러법"""
x_vals = [x0]
y_vals = [y0]
x = x0
y = y0
while x < x_end - 1e-10:
h_actual = min(h, x_end - x)
y = y + h_actual * f(x, y)
x = x + h_actual
x_vals.append(x)
y_vals.append(y)
return np.array(x_vals), np.array(y_vals)
4.2 개선 오일러법 / 헌법 (Heun's Method)
아이디어: 예측(predict) 후 교정(correct)
전체 오차: — 오일러보다 훨씬 정확합니다.
4.3 4차 룽게-쿠타법 (RK4)
공학에서 가장 널리 사용되는 ODE 수치 풀이법입니다.
직관: 구간 내 네 개의 기울기를 가중 평균합니다.
- : 시작점 기울기
- , : 중점 기울기 (두 개, 과 를 이용)
- : 끝점 기울기
전체 오차: — 오차가 에 비례
비교: 를 절반으로 줄이면
- 오일러: 오차 1/2배
- RK4: 오차 1/16배
import numpy as np
import matplotlib.pyplot as plt
from scipy.integrate import solve_ivp
def rk4_method(f, x0, y0, h, x_end):
"""4차 룽게-쿠타법"""
x_vals = [x0]
y_vals = [y0]
x = x0
y = y0
while x < x_end - 1e-10:
h_actual = min(h, x_end - x)
k1 = h_actual * f(x, y)
k2 = h_actual * f(x + h_actual/2, y + k1/2)
k3 = h_actual * f(x + h_actual/2, y + k2/2)
k4 = h_actual * f(x + h_actual, y + k3)
y = y + (k1 + 2*k2 + 2*k3 + k4) / 6
x = x + h_actual
x_vals.append(x)
y_vals.append(y)
return np.array(x_vals), np.array(y_vals)
# 완전한 RLC 회로 시뮬레이션 - 여러 방법 비교
R, L, C = 50, 0.2, 100e-6
Vs = 100
def rlc_ode(t, state):
q, dq = state
d2q = (Vs - R * dq - q/C) / L
return [dq, d2q]
def rlc_scalar(t, y):
"""연립 ODE를 단일 함수로"""
return rlc_ode(t, y)
# 초기조건
y0 = [0.0, 0.0]
t_end = 0.02
h_values = [0.001, 0.0005, 0.0001]
# scipy solve_ivp (기준 해)
t_ref = np.linspace(0, t_end, 5000)
sol_ref = solve_ivp(rlc_ode, [0, t_end], y0,
t_eval=t_ref, method='RK45',
rtol=1e-10, atol=1e-12)
v_ref = sol_ref.y[0] / C # 커패시터 전압
# RK4로 계산
def rlc_2nd_order(t, y):
"""연립 1차 시스템"""
q, dq = y[0], y[1]
return [dq, (Vs - R*dq - q/C) / L]
fig, axes = plt.subplots(2, 1, figsize=(12, 10))
axes[0].plot(sol_ref.t * 1000, v_ref, 'k-', linewidth=2.5, label='기준 (RK45 고정밀)')
for h in h_values:
n_steps = int(t_end / h)
t_rk4 = np.linspace(0, t_end, n_steps + 1)
# 수동 RK4
y = np.array(y0, dtype=float)
y_hist = [y.copy()]
for i in range(n_steps):
t_i = t_rk4[i]
k1 = h * np.array(rlc_2nd_order(t_i, y))
k2 = h * np.array(rlc_2nd_order(t_i + h/2, y + k1/2))
k3 = h * np.array(rlc_2nd_order(t_i + h/2, y + k2/2))
k4 = h * np.array(rlc_2nd_order(t_i + h, y + k3))
y = y + (k1 + 2*k2 + 2*k3 + k4) / 6
y_hist.append(y.copy())
y_arr = np.array(y_hist)
v_rk4 = y_arr[:, 0] / C
axes[0].plot(t_rk4 * 1000, v_rk4, '--',
label=f'RK4 (h={h*1000:.1f}ms)', alpha=0.8)
axes[0].set_xlabel('시간 (ms)')
axes[0].set_ylabel('커패시터 전압 (V)')
axes[0].set_title('RLC 직렬 회로 - RK4 수치 풀이')
axes[0].legend()
axes[0].grid(True, alpha=0.3)
axes[0].axhline(y=Vs, color='gray', linestyle=':', alpha=0.7, label='정상 상태')
# 전류 플롯
i_ref = sol_ref.y[1]
axes[1].plot(sol_ref.t * 1000, i_ref * 1000, 'k-', linewidth=2, label='전류 i(t)')
axes[1].set_xlabel('시간 (ms)')
axes[1].set_ylabel('전류 (mA)')
axes[1].set_title('RLC 회로 전류')
axes[1].legend()
axes[1].grid(True, alpha=0.3)
plt.tight_layout()
plt.savefig('rlc_numerical.png', dpi=150)
plt.show()
# 감쇠 특성 확인
omega_0 = 1 / np.sqrt(L * C)
alpha = R / (2 * L)
omega_d = np.sqrt(max(omega_0**2 - alpha**2, 0))
print(f"공진 주파수: {omega_0/(2*np.pi):.1f} Hz")
print(f"감쇠 계수: {alpha:.1f} s^-1")
if omega_0 > alpha:
print(f"부족감쇠, 감쇠 진동 주파수: {omega_d/(2*np.pi):.1f} Hz")
elif omega_0 == alpha:
print("임계감쇠")
else:
print("과감쇠")
4.4 강성 ODE와 적응형 스텝 크기
강성 ODE(Stiff ODE): 시간 상수가 매우 다른 모드가 공존합니다.
예: 빠른 RC 과도 응답 + 느린 온도 변화가 함께 있는 시스템
작은 를 사용해야 하지만 실제로는 느린 성분이 중요합니다.
from scipy.integrate import solve_ivp
import numpy as np
# 강성 방정식 예시: Robertson 화학 반응
def robertson(t, y):
"""
dy0/dt = -0.04*y0 + 1e4*y1*y2
dy1/dt = 0.04*y0 - 1e4*y1*y2 - 3e7*y1^2
dy2/dt = 3e7*y1^2
"""
dy0 = -0.04*y[0] + 1e4*y[1]*y[2]
dy1 = 0.04*y[0] - 1e4*y[1]*y[2] - 3e7*y[1]**2
dy2 = 3e7*y[1]**2
return [dy0, dy1, dy2]
y0 = [1, 0, 0]
t_span = (0, 1e11)
t_eval = np.logspace(-6, 11, 1000)
# 비강성 방법 (RK45): 너무 느림
# 강성 방법 (Radau, BDF): 효율적
sol = solve_ivp(robertson, t_span, y0,
method='Radau', # 강성 방법
t_eval=t_eval,
rtol=1e-6, atol=1e-9)
print(f"함수 평가 횟수: {sol.nfev}")
print(f"야코비안 평가: {sol.njev}")
5. 선형 시스템 수치 풀이
5.1 가우스 소거법 (Gaussian Elimination)
전진 소거: 행 연산으로 상삼각 행렬 만들기
후진 대입: 상삼각 시스템 풀기
연산 횟수:
부분 피벗팅(Partial Pivoting): 각 열에서 절댓값이 가장 큰 원소를 피벗으로 사용하여 수치 안정성 향상.
import numpy as np
def gaussian_elimination_pivoting(A, b):
"""부분 피벗팅을 포함한 가우스 소거법"""
n = len(b)
A = np.array(A, dtype=float)
b = np.array(b, dtype=float)
# 전진 소거
for i in range(n):
# 부분 피벗팅: i열에서 최대 원소 찾기
max_row = i + np.argmax(abs(A[i:, i]))
A[[i, max_row]] = A[[max_row, i]]
b[[i, max_row]] = b[[max_row, i]]
if abs(A[i, i]) < 1e-15:
raise ValueError("특이 행렬!")
for j in range(i+1, n):
factor = A[j, i] / A[i, i]
A[j, i:] -= factor * A[i, i:]
b[j] -= factor * b[i]
# 후진 대입
x = np.zeros(n)
for i in range(n-1, -1, -1):
x[i] = (b[i] - np.dot(A[i, i+1:], x[i+1:])) / A[i, i]
return x
# 예제: 3개 루프 회로망 (KVL)
A = np.array([
[10, -3, -1],
[-3, 12, -2],
[-1, -2, 8]
], dtype=float)
b = np.array([100, 50, -30], dtype=float)
x_gauss = gaussian_elimination_pivoting(A, b)
x_numpy = np.linalg.solve(A, b)
print("회로 루프 전류 (가우스 소거법):")
for i, v in enumerate(x_gauss):
print(f" I{i+1} = {v:.6f} A")
print(f"\nnumpy 결과와 오차: {np.max(abs(x_gauss - x_numpy)):.2e}")
5.2 LU 분해 (LU Decomposition)
: 하삼각 행렬, : 상삼각 행렬
장점: 같은 에 대해 여러 를 효율적으로 풀 수 있습니다.
(전진 대입), (후진 대입)
from scipy.linalg import lu, lu_factor, lu_solve
import numpy as np
A = np.array([[2, 1, -1], [-3, -1, 2], [-2, 1, 2]], dtype=float)
b = np.array([8, -11, -3], dtype=float)
# LU 분해
P, L, U = lu(A)
print("L 행렬:")
print(np.round(L, 4))
print("U 행렬:")
print(np.round(U, 4))
# LU 분해로 풀기
lu_and_piv = lu_factor(A)
x = lu_solve(lu_and_piv, b)
print(f"\n해: {x}")
# 여러 우변 벡터에 대해 효율적으로 풀기
for b_new in [b, b+1, b*2]:
x_new = lu_solve(lu_and_piv, b_new) # L, U 재사용
5.3 반복법 (Iterative Methods)
대형 희소 행렬에서 직접법보다 효율적입니다.
야코비법(Jacobi Method):
모든 변수를 동시에 갱신합니다.
가우스-자이델법(Gauss-Seidel Method):
갱신된 값을 즉시 사용합니다 → 야코비보다 빠른 수렴.
수렴 조건: 대각 우세 행렬 (diagonal dominant)
import numpy as np
def gauss_seidel(A, b, x0=None, tol=1e-10, max_iter=1000):
"""가우스-자이델 반복법"""
n = len(b)
x = np.zeros(n) if x0 is None else x0.copy()
for iteration in range(max_iter):
x_old = x.copy()
for i in range(n):
sum1 = np.dot(A[i, :i], x[:i])
sum2 = np.dot(A[i, i+1:], x_old[i+1:])
x[i] = (b[i] - sum1 - sum2) / A[i, i]
residual = np.linalg.norm(x - x_old)
if residual < tol:
print(f"수렴: {iteration+1}번 반복, 잔차: {residual:.2e}")
return x
print(f"최대 반복 {max_iter}에서 종료, 잔차: {residual:.2e}")
return x
# 대각 우세 행렬 예제 (노드 전압 해석)
A = np.array([
[ 4, -1, -1, 0],
[-1, 4, 0, -1],
[-1, 0, 4, -1],
[ 0, -1, -1, 4]
], dtype=float)
b = np.array([5, 10, 15, 20], dtype=float)
x_gs = gauss_seidel(A, b)
x_exact = np.linalg.solve(A, b)
print(f"가우스-자이델: {x_gs}")
print(f"정확한 해: {x_exact}")
print(f"최대 오차: {np.max(abs(x_gs - x_exact)):.2e}")
5.4 numpy.linalg 활용
import numpy as np
A = np.array([[3, -0.1, -0.2],
[0.1, 7, -0.3],
[0.3, -0.2, 10]], dtype=float)
b = np.array([7.85, -19.3, 71.4], dtype=float)
# 직접 풀기
x = np.linalg.solve(A, b)
print(f"해: {x}")
# 고유값
eigenvalues, eigenvectors = np.linalg.eig(A)
print(f"\n고유값: {eigenvalues}")
# 역행렬 (비추천 - 수치 오차 큼)
A_inv = np.linalg.inv(A)
print(f"역행렬 검증 (A*A_inv - I 최대값): {np.max(abs(A @ A_inv - np.eye(3))):.2e}")
# 조건수와 랭크
print(f"조건수: {np.linalg.cond(A):.2f}")
print(f"랭크: {np.linalg.matrix_rank(A)}")
# 최소자승 풀이 (과결정 시스템)
A_over = np.vstack([A, [1, 1, 1]])
b_over = np.append(b, 10)
x_ls, residuals, rank, sv = np.linalg.lstsq(A_over, b_over, rcond=None)
print(f"\n최소자승 해: {x_ls}")
6. 보간과 근사
6.1 라그랑주 보간 (Lagrange Interpolation)
개의 점 을 지나는 차 다항식:
기저 다항식(Lagrange basis):
룽게 현상(Runge's Phenomenon): 등간격 점으로 고차 보간 시 양 끝에서 진동 발생.
해결책: 체비쇼프 점(Chebyshev nodes) 사용
6.2 스플라인 보간 (Spline Interpolation)
구간마다 낮은 차수 다항식 사용. 3차 스플라인이 가장 일반적.
인접 구간에서 함수값, 1차, 2차 미분이 연속 → 매끄러운 곡선.
import numpy as np
import matplotlib.pyplot as plt
from scipy.interpolate import lagrange, CubicSpline, interp1d
# 측정 데이터 (온도 vs 저항)
x_data = np.array([0, 20, 40, 60, 80, 100], dtype=float) # 섭씨
y_data = np.array([100, 108, 116, 125, 134, 143], dtype=float) # Ohm
x_fine = np.linspace(0, 100, 300)
# 라그랑주 보간
poly = lagrange(x_data, y_data)
y_lagrange = poly(x_fine)
# 3차 스플라인
cs = CubicSpline(x_data, y_data)
y_spline = cs(x_fine)
# 선형 보간
f_linear = interp1d(x_data, y_data)
y_linear = f_linear(x_fine)
plt.figure(figsize=(10, 6))
plt.scatter(x_data, y_data, s=100, zorder=5, label='측정 데이터', color='black')
plt.plot(x_fine, y_linear, 'g--', linewidth=1.5, label='선형 보간')
plt.plot(x_fine, y_lagrange, 'r-', linewidth=2, label='라그랑주 보간')
plt.plot(x_fine, y_spline, 'b-', linewidth=2, label='3차 스플라인')
plt.xlabel('온도 (C)')
plt.ylabel('저항 (Ohm)')
plt.title('온도-저항 보간 비교')
plt.legend()
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.savefig('interpolation.png', dpi=150)
plt.show()
# 50도에서의 저항 예측
print(f"50도에서 저항 예측:")
print(f" 선형: {f_linear(50):.2f} Ohm")
print(f" 스플라인: {cs(50):.2f} Ohm")
6.3 최소자승법 (Least Squares Method)
개의 데이터 점을 차 다항식 ()으로 가장 잘 근사합니다.
법방정식(Normal Equations):
여기서 는 반데르몬드 행렬.
import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import curve_fit
# 노이즈가 있는 지수 감쇠 데이터
np.random.seed(42)
t_data = np.linspace(0, 5, 30)
y_true = 3 * np.exp(-0.7 * t_data)
y_noisy = y_true + 0.2 * np.random.randn(len(t_data))
# 다항식 최소자승 피팅
degrees = [1, 2, 3, 5]
t_fine = np.linspace(0, 5, 200)
plt.figure(figsize=(12, 8))
plt.scatter(t_data, y_noisy, s=50, color='black', zorder=5, label='측정 데이터')
plt.plot(t_fine, 3*np.exp(-0.7*t_fine), 'k-', linewidth=2, label='참 신호')
for deg in degrees:
coeffs = np.polyfit(t_data, y_noisy, deg)
y_fit = np.polyval(coeffs, t_fine)
plt.plot(t_fine, y_fit, '--', linewidth=1.5, label=f'{deg}차 다항식')
plt.ylim([-1, 4])
plt.xlabel('시간 (s)')
plt.ylabel('진폭')
plt.title('최소자승 다항식 피팅')
plt.legend()
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.savefig('polynomial_fit.png', dpi=150)
plt.show()
# 비선형 최소자승: 지수 함수 피팅
def exp_model(t, A, k):
return A * np.exp(-k * t)
popt, pcov = curve_fit(exp_model, t_data, y_noisy, p0=[3, 0.5])
A_fit, k_fit = popt
A_std, k_std = np.sqrt(np.diag(pcov))
print(f"지수 피팅 결과:")
print(f" A = {A_fit:.4f} +/- {A_std:.4f} (참값: 3)")
print(f" k = {k_fit:.4f} +/- {k_std:.4f} (참값: 0.7)")
# 잔차 및 R^2
y_pred = exp_model(t_data, *popt)
ss_res = np.sum((y_noisy - y_pred)**2)
ss_tot = np.sum((y_noisy - np.mean(y_noisy))**2)
r_squared = 1 - ss_res / ss_tot
print(f" R^2 = {r_squared:.6f}")
7. 고유값 문제 수치 풀이
7.1 거듭제곱법 (Power Method)
행렬 의 가장 큰 절댓값 고유값 과 대응 고유벡터 을 구합니다.
알고리즘:
- 초기 벡터 선택
- (레일리 몫)
- 수렴할 때까지 반복
수렴 속도: (고유값 비율의 거듭제곱)
import numpy as np
def power_method(A, tol=1e-10, max_iter=1000):
"""거듭제곱법 (최대 고유값)"""
n = A.shape[0]
q = np.ones(n) / np.sqrt(n) # 초기 벡터
for k in range(max_iter):
z = A @ q
eigenvalue = np.dot(q, z) # 레일리 몫
q_new = z / np.linalg.norm(z)
if np.linalg.norm(q_new - q) < tol or np.linalg.norm(q_new + q) < tol:
print(f"수렴: {k+1}번 반복")
return eigenvalue, q_new
q = q_new
return eigenvalue, q
# 진동 시스템의 고유 주파수 분석
# 이중 진자 질량-스프링 시스템의 강성 행렬
k = 1000 # N/m
m = 1 # kg
K = np.array([[2*k, -k], [-k, k]], dtype=float)
M = np.array([[m, 0], [0, m]], dtype=float)
# 일반화 고유값 문제: K*v = lambda*M*v
# 변환: M^(-1/2) * K * M^(-1/2) * u = lambda * u
M_inv_sqrt = np.diag(1/np.sqrt(np.diag(M)))
A_sym = M_inv_sqrt @ K @ M_inv_sqrt
lam_max, v_max = power_method(A_sym)
omega_max = np.sqrt(lam_max)
# 모든 고유값 (numpy)
eigenvalues_all = np.linalg.eigvalsh(A_sym)
omega_all = np.sqrt(eigenvalues_all)
print("진동 시스템 고유 주파수:")
for i, omega in enumerate(omega_all):
print(f" 모드 {i+1}: {omega/(2*np.pi):.3f} Hz")
print(f"\n거듭제곱법 최대 고유 주파수: {omega_max/(2*np.pi):.3f} Hz")
7.2 QR 알고리즘
모든 고유값을 동시에 구합니다. 현대 수치선형대수의 핵심 알고리즘.
아이디어: 반복적 QR 분해로 행렬이 상삼각 형태로 수렴하면 대각 원소가 고유값.
import numpy as np
def qr_algorithm(A, max_iter=1000, tol=1e-10):
"""기본 QR 알고리즘"""
A_k = A.copy().astype(float)
n = A.shape[0]
for k in range(max_iter):
Q, R = np.linalg.qr(A_k)
A_k_new = R @ Q
# 하삼각 원소들의 크기 확인
off_diag_norm = np.max(np.abs(np.tril(A_k_new, -1)))
if off_diag_norm < tol:
print(f"QR 수렴: {k+1}번 반복")
break
A_k = A_k_new
return np.diag(A_k)
# 대칭 행렬 (실수 고유값)
A = np.array([
[4, 1, 0],
[1, 3, 1],
[0, 1, 2]
], dtype=float)
eig_qr = qr_algorithm(A)
eig_numpy = np.linalg.eigvalsh(A)
print("QR 알고리즘 고유값:", np.sort(eig_qr))
print("numpy 고유값: ", np.sort(eig_numpy))
7.3 완전한 고유값 분석: 구조 공진 분석
import numpy as np
import matplotlib.pyplot as plt
from scipy import linalg
def vibration_analysis(K, M, n_modes=None):
"""
일반화 고유값 문제: K*v = omega^2 * M * v
K: 강성 행렬, M: 질량 행렬
"""
# scipy linalg.eigh로 일반화 고유값 풀기
eigenvalues, eigenvectors = linalg.eigh(K, M)
omega = np.sqrt(np.abs(eigenvalues)) # 각 주파수 (rad/s)
freq = omega / (2 * np.pi) # 주파수 (Hz)
if n_modes is None:
n_modes = len(eigenvalues)
print("고유 주파수 분석 결과:")
print("-" * 40)
for i in range(n_modes):
print(f"모드 {i+1}: {freq[i]:.3f} Hz ({omega[i]:.2f} rad/s)")
return omega[:n_modes], eigenvectors[:, :n_modes]
# 3자유도 진동계
k1, k2, k3 = 1000, 1500, 800 # N/m
m1, m2, m3 = 1.0, 1.5, 0.8 # kg
K = np.array([
[k1+k2, -k2, 0],
[-k2, k2+k3, -k3],
[0, -k3, k3]
], dtype=float)
M = np.diag([m1, m2, m3])
omegas, modes = vibration_analysis(K, M)
# 모드 형상 시각화
positions = np.arange(1, 4)
fig, axes = plt.subplots(1, 3, figsize=(14, 5))
for i, ax in enumerate(axes):
mode_shape = modes[:, i] / np.max(abs(modes[:, i]))
ax.plot([0] + list(positions), [0] + list(mode_shape),
'bo-', linewidth=2, markersize=10)
ax.axhline(y=0, color='gray', linestyle='--', alpha=0.5)
ax.set_title(f'모드 {i+1}\n{omegas[i]/(2*np.pi):.2f} Hz')
ax.set_xlabel('자유도')
ax.set_ylabel('정규화 모드 형상')
ax.grid(True, alpha=0.3)
ax.set_xticks([0, 1, 2, 3])
ax.set_xticklabels(['고정단', '질량1', '질량2', '질량3'])
plt.suptitle('3자유도 진동계 모드 형상')
plt.tight_layout()
plt.savefig('vibration_modes.png', dpi=150)
plt.show()
8. 실전 종합 예제: MOSFET 바이어스 회로 수치 해석
import numpy as np
from scipy.optimize import fsolve
from scipy.integrate import solve_ivp
import matplotlib.pyplot as plt
# MOSFET 간단 모델 파라미터
Vth = 1.5 # 문턱 전압 (V)
k = 2e-3 # 트랜스컨덕턴스 파라미터 (A/V^2)
VDD = 15 # 공급 전압 (V)
RD = 3000 # 드레인 저항 (Ohm)
RS = 1000 # 소스 저항 (Ohm)
R1 = 100e3 # 전압분배 저항 1 (Ohm)
R2 = 47e3 # 전압분배 저항 2 (Ohm)
def mosfet_bias_equations(vars):
"""MOSFET 바이어스 비선형 방정식 시스템"""
VGS, ID = vars
# 전압 분배기로 게이트 전압
VG = VDD * R2 / (R1 + R2)
# 드레인 전류 (포화 영역)
if VGS > Vth:
ID_calc = k / 2 * (VGS - Vth)**2
else:
ID_calc = 0
# KVL: VGS = VG - VS = VG - ID*RS
VGS_calc = VG - ID * RS
return [VGS - VGS_calc, ID - ID_calc]
VG = VDD * R2 / (R1 + R2)
initial_guess = [VG - 1, 1e-3]
solution = fsolve(mosfet_bias_equations, initial_guess, full_output=True)
VGS_dc, ID_dc = solution[0]
VDS_dc = VDD - ID_dc * (RD + RS)
print("MOSFET DC 바이어스 동작점:")
print(f" VGS = {VGS_dc:.3f} V")
print(f" ID = {ID_dc*1000:.3f} mA")
print(f" VDS = {VDS_dc:.3f} V")
print(f" 포화 영역 확인: VDS > VGS - Vth = {VGS_dc - Vth:.3f} V -> {'OK' if VDS_dc > VGS_dc - Vth else '선형 영역'}")
# 전달 특성 곡선
VGS_range = np.linspace(0, 5, 200)
ID_range = np.where(VGS_range > Vth,
k/2 * (VGS_range - Vth)**2,
0)
plt.figure(figsize=(10, 5))
plt.subplot(1, 2, 1)
plt.plot(VGS_range, ID_range * 1000, 'b-', linewidth=2)
plt.axvline(x=VGS_dc, color='r', linestyle='--', label=f'동작점 VGS={VGS_dc:.2f}V')
plt.axhline(y=ID_dc*1000, color='g', linestyle='--', label=f'동작점 ID={ID_dc*1000:.2f}mA')
plt.xlabel('VGS (V)')
plt.ylabel('ID (mA)')
plt.title('MOSFET 전달 특성')
plt.legend()
plt.grid(True, alpha=0.3)
plt.subplot(1, 2, 2)
VDS_range = np.linspace(0, VDD, 200)
for VGS_val in [2.0, 2.5, 3.0, 3.5]:
if VGS_val <= Vth:
continue
ID_sat = k/2 * (VGS_val - Vth)**2
ID_curve = np.where(
VDS_range < VGS_val - Vth,
k * ((VGS_val - Vth)*VDS_range - VDS_range**2/2),
ID_sat
)
plt.plot(VDS_range, ID_curve * 1000,
label=f'VGS={VGS_val}V')
# 부하선
ID_load = (VDD - VDS_range) / (RD + RS)
plt.plot(VDS_range, ID_load * 1000, 'k--', linewidth=2, label='부하선')
plt.axvline(x=VDS_dc, color='r', linestyle=':', alpha=0.7)
plt.xlabel('VDS (V)')
plt.ylabel('ID (mA)')
plt.title('MOSFET 출력 특성 및 부하선')
plt.legend(fontsize=8)
plt.ylim([0, None])
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.savefig('mosfet_analysis.png', dpi=150)
plt.show()
정리 및 다음 단계
이번 4편에서 다룬 내용:
- 오차 분석: 반올림/절단 오차, 절대/상대 오차, 수치 안정성, 조건수
- 비선형 방정식: 이분법(1차), 뉴턴-랩슨(2차), 할선법(1.618차), scipy.optimize
- 수치 미분: 전진/후진/중앙 차분, 오차 분석, 최적 스텝 크기
- 수치 적분: 사다리꼴, 심프슨, 가우스 구적법, scipy.integrate
- ODE 수치 풀이: 오일러법, 헌법, RK4 완전 구현, 강성 ODE
- 선형 시스템: 가우스 소거법, LU 분해, 반복법, numpy.linalg
- 보간: 라그랑주, 스플라인, 룽게 현상, 최소자승, 비선형 커브 피팅
- 고유값 문제: 거듭제곱법, QR 알고리즘, 진동 분석
공업수학 4편 시리즈를 통해 ODE, 푸리에 해석, PDE, 복소해석학, Z-변환, 수치해석까지 전자/전기/기계 공학의 핵심 수학 도구를 완전 정복했습니다.
참고 자료
- Burden, R. & Faires, J. "Numerical Analysis", 10th Edition, Cengage
- Heath, M. "Scientific Computing: An Introductory Survey", 2nd Edition
- Press, W. et al. "Numerical Recipes: The Art of Scientific Computing", 3rd Edition
- Kreyszig, E. "Advanced Engineering Mathematics", 10th Edition, Wiley
- NumPy 선형대수 문서
- SciPy 최적화 문서
- SciPy 적분 문서
Engineering Mathematics: Numerical Methods Complete Guide - Root Finding to ODE Solvers
Engineering Math Mastery Part 4: Numerical Methods
Numerical analysis is required when engineering problems cannot be solved analytically, or when numerical simulation of complex systems is needed. Analyzing the nonlinear characteristics of transistors, transient responses in complex circuit networks, and finite element analysis all rely on numerical methods. This article covers core numerical techniques from first principles to full Python implementation.
1. Error Analysis
1.1 Types of Errors
In numerical computation, a "perfect solution" does not exist. Understanding the types of errors is the foundation of reliable numerical computation.
Round-off Error: Occurs because computers represent real numbers with a finite number of digits.
IEEE 754 double precision: approximately 15-16 significant digits
Truncation Error: Occurs when an infinite series is approximated with a finite number of terms.
Absolute Error:
Relative Error:
Percentage error:
Significant Figures: If , this guarantees significant digits:
1.2 Numerical Stability
Errors must not be amplified during computation.
Well-conditioned: Input errors are not significantly amplified in the output
Ill-conditioned: Small changes in input cause large changes in output
Condition Number:
If is large in a linear system , the numerical solution is unstable.
import numpy as np
# Hilbert matrix (notoriously ill-conditioned)
n = 6
H = np.array([[1/(i+j+1) for j in range(n)] for i in range(n)])
kappa = np.linalg.cond(H)
print(f"Hilbert matrix ({n}x{n}) condition number: {kappa:.2e}")
# Simple matrix
A = np.array([[2, 1], [1, 3]], dtype=float)
print(f"Simple matrix condition number: {np.linalg.cond(A):.2f}")
# Error propagation demonstration
b_true = np.array([3, 4], dtype=float)
x_true = np.linalg.solve(A, b_true)
# Add a small error
b_perturbed = b_true + np.array([0.01, 0.0])
x_perturbed = np.linalg.solve(A, b_perturbed)
print(f"\nInput error: {np.linalg.norm(b_perturbed - b_true):.4f}")
print(f"Output error: {np.linalg.norm(x_perturbed - x_true):.4f}")
1.3 Big-O Notation and Order of Convergence
Order of Convergence:
- : First-order (linear) convergence
- : Second-order (quadratic) convergence (much faster)
- : Superlinear convergence (secant method)
Big-O Notation:
Example: Forward difference error is ; central difference error is
2. Solving Nonlinear Equations
2.1 Bisection Method
Principle: Bolzano's theorem — if , there exists a root in .
Algorithm:
- Verify
- Compute midpoint
- If , done; if , set ; otherwise set
- If , done
Convergence: The interval halves with each iteration — first-order convergence. Error after iterations:
Required iterations:
def bisection(f, a, b, tol=1e-10, max_iter=100):
"""Find root of f(x) = 0 using the bisection method"""
if f(a) * f(b) > 0:
raise ValueError("f(a) and f(b) must have opposite signs")
history = []
for i in range(max_iter):
c = (a + b) / 2
history.append({'iter': i+1, 'a': a, 'b': b, 'c': c,
'f(c)': f(c), 'error': abs(b-a)/2})
if abs(f(c)) < tol or (b - a) / 2 < tol:
print(f"Converged in {i+1} iterations")
return c, history
if f(a) * f(c) < 0:
b = c
else:
a = c
return (a + b) / 2, history
# Example: x^3 - 2x - 5 = 0
def equation(x):
return x**3 - 2*x - 5
root, history = bisection(equation, 1, 3)
print(f"Root: {root:.10f}")
print(f"Verification: f({root:.6f}) = {equation(root):.2e}")
# Print convergence history
print("\nIter | Approximation | Error")
for h in history[:8]:
print(f" {h['iter']:2d} | {h['c']:.8f} | {h['error']:.2e}")
2.2 Newton-Raphson Method
Principle: First-order Taylor approximation at
Geometric interpretation: The tangent line at intersects the x-axis at
Convergence: Ideally quadratic — the number of accurate digits doubles with each iteration!
Limitations and caveats:
- Diverges if (tangent line parallel to x-axis)
- Convergence is not guaranteed (initial point selection matters)
- Degrades to linear convergence at multiple roots
def newton_raphson(f, df, x0, tol=1e-10, max_iter=50):
"""Newton-Raphson method"""
x = x0
history = [{'iter': 0, 'x': x, 'f(x)': f(x), 'error': float('inf')}]
for i in range(max_iter):
fx = f(x)
dfx = df(x)
if abs(dfx) < 1e-15:
print("Warning: derivative near zero, risk of divergence")
break
x_new = x - fx / dfx
error = abs(x_new - x)
history.append({'iter': i+1, 'x': x_new,
'f(x)': f(x_new), 'error': error})
if error < tol:
print(f"Converged in {i+1} iterations")
return x_new, history
x = x_new
return x, history
# f(x) = x^3 - 2x - 5, f'(x) = 3x^2 - 2
def f(x):
return x**3 - 2*x - 5
def df(x):
return 3*x**2 - 2
root, history = newton_raphson(f, df, x0=2.0)
print(f"Newton-Raphson root: {root:.10f}")
print("\nIter | Approximation | Error")
for h in history:
err_str = f"{h['error']:.2e}" if h['error'] != float('inf') else " -"
print(f" {h['iter']:2d} | {h['x']:.10f} | {err_str}")
Convergence speed comparison:
import numpy as np
import matplotlib.pyplot as plt
# Bisection vs Newton-Raphson convergence comparison
true_root = 2.0945514815423265 # actual root of x^3 - 2x - 5 = 0
_, hist_bisect = bisection(f, 1, 3, tol=1e-12, max_iter=50)
_, hist_newton = newton_raphson(f, df, x0=2.0, tol=1e-12, max_iter=20)
errors_b = [abs(h['c'] - true_root) for h in hist_bisect]
errors_n = [abs(h['x'] - true_root) for h in hist_newton if h['error'] != float('inf')]
plt.figure(figsize=(10, 5))
plt.semilogy(range(1, len(errors_b)+1), errors_b, 'b-o', label='Bisection (1st order)')
plt.semilogy(range(1, len(errors_n)+1), errors_n, 'r-s', label='Newton-Raphson (2nd order)')
plt.xlabel('Iterations')
plt.ylabel('Absolute error (log scale)')
plt.title('Bisection vs Newton-Raphson Convergence')
plt.legend()
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.savefig('convergence_comparison.png', dpi=150)
plt.show()
2.3 Secant Method
When is unknown, approximate it with a finite difference:
Order of convergence: (golden ratio, superlinear)
def secant_method(f, x0, x1, tol=1e-10, max_iter=50):
"""Secant method"""
for i in range(max_iter):
f0, f1 = f(x0), f(x1)
if abs(f1 - f0) < 1e-15:
break
x2 = x1 - f1 * (x1 - x0) / (f1 - f0)
if abs(x2 - x1) < tol:
print(f"Converged in {i+1} iterations")
return x2
x0, x1 = x1, x2
return x1
root_s = secant_method(f, 1.0, 3.0)
print(f"Secant method root: {root_s:.10f}")
2.4 Using scipy.optimize
from scipy.optimize import fsolve, brentq, minimize_scalar
import numpy as np
# Solve a single equation
def equation(x):
return x**3 - 2*x - 5
# Brent's method (combination of bisection + secant + inverse quadratic interpolation)
root = brentq(equation, 1, 3, xtol=1e-12)
print(f"brentq root: {root:.12f}")
# fsolve (Newton-Raphson based)
root_fs = fsolve(equation, 2.0)[0]
print(f"fsolve root: {root_fs:.12f}")
# System of nonlinear equations
from scipy.optimize import fsolve
def system(vars):
x, y = vars
eq1 = x**2 + y**2 - 4 # circle
eq2 = x*y - 1 # hyperbola
return [eq1, eq2]
solutions = []
for x0, y0 in [(1, 1), (-1, 1), (1, -1), (-1, -1)]:
sol = fsolve(system, [x0, y0], full_output=True)
if sol[2] == 1: # converged successfully
solutions.append(sol[0])
print("\nSystem of equations solutions:")
for sol in solutions:
print(f" (x, y) = ({sol[0]:.6f}, {sol[1]:.6f})")
# Circuit equation: diode characteristics (nonlinear)
def diode_equation(V, V_s=5, R=1000, I_s=1e-12, n=1, V_T=0.02585):
"""V_s = V + R * I_s * (exp(V/(n*V_T)) - 1)"""
I = I_s * (np.exp(V / (n * V_T)) - 1)
return V_s - V - R * I
V_diode = brentq(diode_equation, 0, 1)
I_diode = 1e-12 * (np.exp(V_diode / 0.02585) - 1)
print(f"\nDiode operating point: V = {V_diode:.4f} V, I = {I_diode*1000:.4f} mA")
3. Numerical Differentiation and Integration
3.1 Finite Differences
Derivatives are numerically approximated from Taylor series.
Forward Difference:
First-order accuracy (error proportional to ).
Backward Difference:
Central Difference:
Subtracting:
Second-order accuracy — much more accurate than forward difference for the same .
Second derivative:
This is used in finite difference solutions of the heat equation and wave equation.
import numpy as np
import matplotlib.pyplot as plt
def numerical_derivative(f, x, h=1e-5, method='central'):
"""Numerical differentiation"""
if method == 'forward':
return (f(x + h) - f(x)) / h
elif method == 'backward':
return (f(x) - f(x - h)) / h
elif method == 'central':
return (f(x + h) - f(x - h)) / (2 * h)
def f(x):
return np.sin(x)
def df_true(x):
return np.cos(x)
x0 = np.pi / 4
true_deriv = df_true(x0)
# Error analysis for different step sizes
h_values = np.logspace(-16, 0, 100)
errors_fwd = []
errors_cen = []
for h in h_values:
e_fwd = abs(numerical_derivative(f, x0, h, 'forward') - true_deriv)
e_cen = abs(numerical_derivative(f, x0, h, 'central') - true_deriv)
errors_fwd.append(e_fwd if e_fwd > 0 else 1e-17)
errors_cen.append(e_cen if e_cen > 0 else 1e-17)
plt.figure(figsize=(10, 6))
plt.loglog(h_values, errors_fwd, 'b-', linewidth=2, label='Forward difference O(h)')
plt.loglog(h_values, errors_cen, 'r-', linewidth=2, label='Central difference O(h^2)')
plt.loglog(h_values, h_values, 'b--', alpha=0.5, label='O(h) reference')
plt.loglog(h_values, h_values**2, 'r--', alpha=0.5, label='O(h^2) reference')
plt.xlabel('h (step size)')
plt.ylabel('Absolute error')
plt.title('Numerical Differentiation Error Analysis (error vs h)')
plt.legend()
plt.grid(True, which='both', alpha=0.3)
plt.tight_layout()
plt.savefig('numerical_derivative_error.png', dpi=150)
plt.show()
3.2 Numerical Integration
Trapezoidal Rule:
where . Error:
Simpson's 1/3 Rule:
Approximate every two subintervals with a quadratic polynomial:
must be even. Error: — much more accurate than the trapezoidal rule!
Simpson's 3/8 Rule: Cubic polynomial, error , must be a multiple of 3.
Gaussian Quadrature:
Maximizes accuracy by selecting optimal integration points and weights.
An -point Gauss-Legendre quadrature rule integrates polynomials up to degree exactly.
import numpy as np
from scipy import integrate
import matplotlib.pyplot as plt
def trapezoidal(f, a, b, n):
"""Trapezoidal rule"""
x = np.linspace(a, b, n+1)
h = (b - a) / n
y = f(x)
return h * (y[0]/2 + np.sum(y[1:-1]) + y[-1]/2)
def simpsons(f, a, b, n):
"""Simpson's 1/3 rule (n must be even)"""
if n % 2 != 0:
n += 1
x = np.linspace(a, b, n+1)
h = (b - a) / n
y = f(x)
return h/3 * (y[0] + 4*np.sum(y[1::2]) + 2*np.sum(y[2:-1:2]) + y[-1])
# Example: integral of sin(x) from 0 to pi = 2
f_test = np.sin
a, b = 0, np.pi
true_val = 2.0
print("Numerical Integration Accuracy Comparison")
print(f"Exact value: {true_val}")
print("-" * 60)
print(f"{'Method':<20} {'n':>6} {'Approx':>15} {'Rel. Error':>12}")
print("-" * 60)
for n in [4, 8, 16, 32]:
trap = trapezoidal(f_test, a, b, n)
simp = simpsons(f_test, a, b, n)
print(f"{'Trapezoidal':<20} {n:>6} {trap:>15.10f} {abs(trap-true_val)/true_val:>12.2e}")
print(f"{'Simpson':<20} {n:>6} {simp:>15.10f} {abs(simp-true_val)/true_val:>12.2e}")
# scipy.integrate.quad - adaptive quadrature
result, error = integrate.quad(f_test, a, b)
print(f"{'scipy.quad':<20} {'adaptive':>6} {result:>15.10f} {'<'+str(error):>12}")
# Difficult integral: int_0^inf exp(-x^2) dx = sqrt(pi)/2
def gaussian(x):
return np.exp(-x**2)
result_gauss, _ = integrate.quad(gaussian, 0, np.inf)
print(f"\nGaussian integral: {result_gauss:.10f}")
print(f"Exact value (sqrt(pi)/2): {np.sqrt(np.pi)/2:.10f}")
# Double integral: int_0^1 int_0^1 x*y dx dy = 0.25
def f2d(y, x):
return x * y
result_2d, _ = integrate.dblquad(f2d, 0, 1, 0, 1)
print(f"\nDouble integral: {result_2d:.10f} (exact: 0.25)")
4. Numerical ODE Solvers
4.1 Euler's Method
Idea: Predict the next point using the current slope
Local truncation error: , global error: (first-order method)
Geometric interpretation: Move in the direction of the tangent at each point
import numpy as np
import matplotlib.pyplot as plt
def euler_method(f, x0, y0, h, x_end):
"""Euler's method"""
x_vals = [x0]
y_vals = [y0]
x = x0
y = y0
while x < x_end - 1e-10:
h_actual = min(h, x_end - x)
y = y + h_actual * f(x, y)
x = x + h_actual
x_vals.append(x)
y_vals.append(y)
return np.array(x_vals), np.array(y_vals)
4.2 Improved Euler / Heun's Method
Idea: Predict then correct
Global error: — much more accurate than Euler's method.
4.3 Fourth-Order Runge-Kutta (RK4)
The most widely used ODE numerical solver in engineering.
Intuition: Weighted average of four slopes within the interval.
- : slope at the start
- , : slopes at the midpoint (two estimates, using and )
- : slope at the end
Global error: — error proportional to
Comparison: halving
- Euler: error halved
- RK4: error reduced by factor of 1/16
import numpy as np
import matplotlib.pyplot as plt
from scipy.integrate import solve_ivp
def rk4_method(f, x0, y0, h, x_end):
"""Fourth-order Runge-Kutta method"""
x_vals = [x0]
y_vals = [y0]
x = x0
y = y0
while x < x_end - 1e-10:
h_actual = min(h, x_end - x)
k1 = h_actual * f(x, y)
k2 = h_actual * f(x + h_actual/2, y + k1/2)
k3 = h_actual * f(x + h_actual/2, y + k2/2)
k4 = h_actual * f(x + h_actual, y + k3)
y = y + (k1 + 2*k2 + 2*k3 + k4) / 6
x = x + h_actual
x_vals.append(x)
y_vals.append(y)
return np.array(x_vals), np.array(y_vals)
# Complete RLC circuit simulation - comparing methods
R, L, C = 50, 0.2, 100e-6
Vs = 100
def rlc_ode(t, state):
q, dq = state
d2q = (Vs - R * dq - q/C) / L
return [dq, d2q]
def rlc_scalar(t, y):
"""Coupled ODE as a single function"""
return rlc_ode(t, y)
# Initial conditions
y0 = [0.0, 0.0]
t_end = 0.02
h_values = [0.001, 0.0005, 0.0001]
# scipy solve_ivp (reference solution)
t_ref = np.linspace(0, t_end, 5000)
sol_ref = solve_ivp(rlc_ode, [0, t_end], y0,
t_eval=t_ref, method='RK45',
rtol=1e-10, atol=1e-12)
v_ref = sol_ref.y[0] / C # capacitor voltage
# Calculate with RK4
def rlc_2nd_order(t, y):
"""Coupled first-order system"""
q, dq = y[0], y[1]
return [dq, (Vs - R*dq - q/C) / L]
fig, axes = plt.subplots(2, 1, figsize=(12, 10))
axes[0].plot(sol_ref.t * 1000, v_ref, 'k-', linewidth=2.5, label='Reference (RK45 high precision)')
for h in h_values:
n_steps = int(t_end / h)
t_rk4 = np.linspace(0, t_end, n_steps + 1)
# Manual RK4
y = np.array(y0, dtype=float)
y_hist = [y.copy()]
for i in range(n_steps):
t_i = t_rk4[i]
k1 = h * np.array(rlc_2nd_order(t_i, y))
k2 = h * np.array(rlc_2nd_order(t_i + h/2, y + k1/2))
k3 = h * np.array(rlc_2nd_order(t_i + h/2, y + k2/2))
k4 = h * np.array(rlc_2nd_order(t_i + h, y + k3))
y = y + (k1 + 2*k2 + 2*k3 + k4) / 6
y_hist.append(y.copy())
y_arr = np.array(y_hist)
v_rk4 = y_arr[:, 0] / C
axes[0].plot(t_rk4 * 1000, v_rk4, '--',
label=f'RK4 (h={h*1000:.1f}ms)', alpha=0.8)
axes[0].set_xlabel('Time (ms)')
axes[0].set_ylabel('Capacitor Voltage (V)')
axes[0].set_title('RLC Series Circuit - RK4 Numerical Solution')
axes[0].legend()
axes[0].grid(True, alpha=0.3)
axes[0].axhline(y=Vs, color='gray', linestyle=':', alpha=0.7, label='Steady state')
# Current plot
i_ref = sol_ref.y[1]
axes[1].plot(sol_ref.t * 1000, i_ref * 1000, 'k-', linewidth=2, label='Current i(t)')
axes[1].set_xlabel('Time (ms)')
axes[1].set_ylabel('Current (mA)')
axes[1].set_title('RLC Circuit Current')
axes[1].legend()
axes[1].grid(True, alpha=0.3)
plt.tight_layout()
plt.savefig('rlc_numerical.png', dpi=150)
plt.show()
# Check damping characteristics
omega_0 = 1 / np.sqrt(L * C)
alpha = R / (2 * L)
omega_d = np.sqrt(max(omega_0**2 - alpha**2, 0))
print(f"Resonant frequency: {omega_0/(2*np.pi):.1f} Hz")
print(f"Damping coefficient: {alpha:.1f} s^-1")
if omega_0 > alpha:
print(f"Underdamped, damped oscillation frequency: {omega_d/(2*np.pi):.1f} Hz")
elif omega_0 == alpha:
print("Critically damped")
else:
print("Overdamped")
4.4 Stiff ODEs and Adaptive Step Size
Stiff ODE: Coexistence of modes with very different time constants.
Example: A system with both fast RC transient response and slow temperature change
A small must be used, but in practice the slow component is what matters.
from scipy.integrate import solve_ivp
import numpy as np
# Stiff equation example: Robertson chemical reactions
def robertson(t, y):
"""
dy0/dt = -0.04*y0 + 1e4*y1*y2
dy1/dt = 0.04*y0 - 1e4*y1*y2 - 3e7*y1^2
dy2/dt = 3e7*y1^2
"""
dy0 = -0.04*y[0] + 1e4*y[1]*y[2]
dy1 = 0.04*y[0] - 1e4*y[1]*y[2] - 3e7*y[1]**2
dy2 = 3e7*y[1]**2
return [dy0, dy1, dy2]
y0 = [1, 0, 0]
t_span = (0, 1e11)
t_eval = np.logspace(-6, 11, 1000)
# Non-stiff method (RK45): too slow
# Stiff method (Radau, BDF): efficient
sol = solve_ivp(robertson, t_span, y0,
method='Radau', # stiff method
t_eval=t_eval,
rtol=1e-6, atol=1e-9)
print(f"Function evaluations: {sol.nfev}")
print(f"Jacobian evaluations: {sol.njev}")
5. Numerical Solution of Linear Systems
5.1 Gaussian Elimination
Forward elimination: Create an upper triangular matrix using row operations
Back substitution: Solve the upper triangular system
Operation count:
Partial Pivoting: Use the element with the largest absolute value in each column as the pivot to improve numerical stability.
import numpy as np
def gaussian_elimination_pivoting(A, b):
"""Gaussian elimination with partial pivoting"""
n = len(b)
A = np.array(A, dtype=float)
b = np.array(b, dtype=float)
# Forward elimination
for i in range(n):
# Partial pivoting: find the largest element in column i
max_row = i + np.argmax(abs(A[i:, i]))
A[[i, max_row]] = A[[max_row, i]]
b[[i, max_row]] = b[[max_row, i]]
if abs(A[i, i]) < 1e-15:
raise ValueError("Singular matrix!")
for j in range(i+1, n):
factor = A[j, i] / A[i, i]
A[j, i:] -= factor * A[i, i:]
b[j] -= factor * b[i]
# Back substitution
x = np.zeros(n)
for i in range(n-1, -1, -1):
x[i] = (b[i] - np.dot(A[i, i+1:], x[i+1:])) / A[i, i]
return x
# Example: 3-loop circuit network (KVL)
A = np.array([
[10, -3, -1],
[-3, 12, -2],
[-1, -2, 8]
], dtype=float)
b = np.array([100, 50, -30], dtype=float)
x_gauss = gaussian_elimination_pivoting(A, b)
x_numpy = np.linalg.solve(A, b)
print("Circuit loop currents (Gaussian elimination):")
for i, v in enumerate(x_gauss):
print(f" I{i+1} = {v:.6f} A")
print(f"\nError vs numpy result: {np.max(abs(x_gauss - x_numpy)):.2e}")
5.2 LU Decomposition
: lower triangular matrix, : upper triangular matrix
Advantage: Efficiently solve multiple right-hand side vectors for the same .
(forward substitution), (back substitution)
from scipy.linalg import lu, lu_factor, lu_solve
import numpy as np
A = np.array([[2, 1, -1], [-3, -1, 2], [-2, 1, 2]], dtype=float)
b = np.array([8, -11, -3], dtype=float)
# LU decomposition
P, L, U = lu(A)
print("L matrix:")
print(np.round(L, 4))
print("U matrix:")
print(np.round(U, 4))
# Solve using LU decomposition
lu_and_piv = lu_factor(A)
x = lu_solve(lu_and_piv, b)
print(f"\nSolution: {x}")
# Efficiently solve for multiple right-hand side vectors
for b_new in [b, b+1, b*2]:
x_new = lu_solve(lu_and_piv, b_new) # reuse L, U
5.3 Iterative Methods
More efficient than direct methods for large sparse matrices.
Jacobi Method:
Updates all variables simultaneously.
Gauss-Seidel Method:
Uses updated values immediately — converges faster than Jacobi.
Convergence condition: Diagonally dominant matrix
import numpy as np
def gauss_seidel(A, b, x0=None, tol=1e-10, max_iter=1000):
"""Gauss-Seidel iterative method"""
n = len(b)
x = np.zeros(n) if x0 is None else x0.copy()
for iteration in range(max_iter):
x_old = x.copy()
for i in range(n):
sum1 = np.dot(A[i, :i], x[:i])
sum2 = np.dot(A[i, i+1:], x_old[i+1:])
x[i] = (b[i] - sum1 - sum2) / A[i, i]
residual = np.linalg.norm(x - x_old)
if residual < tol:
print(f"Converged in {iteration+1} iterations, residual: {residual:.2e}")
return x
print(f"Stopped at max iterations {max_iter}, residual: {residual:.2e}")
return x
# Diagonally dominant matrix example (nodal voltage analysis)
A = np.array([
[ 4, -1, -1, 0],
[-1, 4, 0, -1],
[-1, 0, 4, -1],
[ 0, -1, -1, 4]
], dtype=float)
b = np.array([5, 10, 15, 20], dtype=float)
x_gs = gauss_seidel(A, b)
x_exact = np.linalg.solve(A, b)
print(f"Gauss-Seidel: {x_gs}")
print(f"Exact solution: {x_exact}")
print(f"Max error: {np.max(abs(x_gs - x_exact)):.2e}")
5.4 Using numpy.linalg
import numpy as np
A = np.array([[3, -0.1, -0.2],
[0.1, 7, -0.3],
[0.3, -0.2, 10]], dtype=float)
b = np.array([7.85, -19.3, 71.4], dtype=float)
# Direct solve
x = np.linalg.solve(A, b)
print(f"Solution: {x}")
# Eigenvalues
eigenvalues, eigenvectors = np.linalg.eig(A)
print(f"\nEigenvalues: {eigenvalues}")
# Matrix inverse (not recommended - large numerical error)
A_inv = np.linalg.inv(A)
print(f"Inverse verification (max of A*A_inv - I): {np.max(abs(A @ A_inv - np.eye(3))):.2e}")
# Condition number and rank
print(f"Condition number: {np.linalg.cond(A):.2f}")
print(f"Rank: {np.linalg.matrix_rank(A)}")
# Least squares solution (overdetermined system)
A_over = np.vstack([A, [1, 1, 1]])
b_over = np.append(b, 10)
x_ls, residuals, rank, sv = np.linalg.lstsq(A_over, b_over, rcond=None)
print(f"\nLeast squares solution: {x_ls}")
6. Interpolation and Approximation
6.1 Lagrange Interpolation
Degree- polynomial passing through points :
Lagrange basis polynomials:
Runge's Phenomenon: High-degree interpolation with equally spaced points causes oscillations near the boundaries.
Solution: Use Chebyshev nodes
6.2 Spline Interpolation
Uses low-degree polynomials on each subinterval. Cubic splines are most common.
Function value, first derivative, and second derivative are all continuous across adjacent intervals — producing a smooth curve.
import numpy as np
import matplotlib.pyplot as plt
from scipy.interpolate import lagrange, CubicSpline, interp1d
# Measurement data (temperature vs resistance)
x_data = np.array([0, 20, 40, 60, 80, 100], dtype=float) # Celsius
y_data = np.array([100, 108, 116, 125, 134, 143], dtype=float) # Ohm
x_fine = np.linspace(0, 100, 300)
# Lagrange interpolation
poly = lagrange(x_data, y_data)
y_lagrange = poly(x_fine)
# Cubic spline
cs = CubicSpline(x_data, y_data)
y_spline = cs(x_fine)
# Linear interpolation
f_linear = interp1d(x_data, y_data)
y_linear = f_linear(x_fine)
plt.figure(figsize=(10, 6))
plt.scatter(x_data, y_data, s=100, zorder=5, label='Measured data', color='black')
plt.plot(x_fine, y_linear, 'g--', linewidth=1.5, label='Linear interpolation')
plt.plot(x_fine, y_lagrange, 'r-', linewidth=2, label='Lagrange interpolation')
plt.plot(x_fine, y_spline, 'b-', linewidth=2, label='Cubic spline')
plt.xlabel('Temperature (C)')
plt.ylabel('Resistance (Ohm)')
plt.title('Temperature-Resistance Interpolation Comparison')
plt.legend()
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.savefig('interpolation.png', dpi=150)
plt.show()
# Predict resistance at 50 degrees
print(f"Resistance prediction at 50 degrees:")
print(f" Linear: {f_linear(50):.2f} Ohm")
print(f" Spline: {cs(50):.2f} Ohm")
6.3 Least Squares Method
Best approximation of data points with a degree- polynomial ():
Normal Equations:
where is the Vandermonde matrix.
import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import curve_fit
# Noisy exponential decay data
np.random.seed(42)
t_data = np.linspace(0, 5, 30)
y_true = 3 * np.exp(-0.7 * t_data)
y_noisy = y_true + 0.2 * np.random.randn(len(t_data))
# Polynomial least squares fitting
degrees = [1, 2, 3, 5]
t_fine = np.linspace(0, 5, 200)
plt.figure(figsize=(12, 8))
plt.scatter(t_data, y_noisy, s=50, color='black', zorder=5, label='Measured data')
plt.plot(t_fine, 3*np.exp(-0.7*t_fine), 'k-', linewidth=2, label='True signal')
for deg in degrees:
coeffs = np.polyfit(t_data, y_noisy, deg)
y_fit = np.polyval(coeffs, t_fine)
plt.plot(t_fine, y_fit, '--', linewidth=1.5, label=f'Degree {deg} polynomial')
plt.ylim([-1, 4])
plt.xlabel('Time (s)')
plt.ylabel('Amplitude')
plt.title('Least Squares Polynomial Fitting')
plt.legend()
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.savefig('polynomial_fit.png', dpi=150)
plt.show()
# Nonlinear least squares: exponential function fitting
def exp_model(t, A, k):
return A * np.exp(-k * t)
popt, pcov = curve_fit(exp_model, t_data, y_noisy, p0=[3, 0.5])
A_fit, k_fit = popt
A_std, k_std = np.sqrt(np.diag(pcov))
print(f"Exponential fitting results:")
print(f" A = {A_fit:.4f} +/- {A_std:.4f} (true: 3)")
print(f" k = {k_fit:.4f} +/- {k_std:.4f} (true: 0.7)")
# Residuals and R-squared
y_pred = exp_model(t_data, *popt)
ss_res = np.sum((y_noisy - y_pred)**2)
ss_tot = np.sum((y_noisy - np.mean(y_noisy))**2)
r_squared = 1 - ss_res / ss_tot
print(f" R^2 = {r_squared:.6f}")
7. Numerical Eigenvalue Problems
7.1 Power Method
Finds the largest magnitude eigenvalue and corresponding eigenvector of matrix .
Algorithm:
- Choose initial vector
- (Rayleigh quotient)
- Repeat until convergence
Convergence rate: (power of eigenvalue ratio)
import numpy as np
def power_method(A, tol=1e-10, max_iter=1000):
"""Power method (maximum eigenvalue)"""
n = A.shape[0]
q = np.ones(n) / np.sqrt(n) # initial vector
for k in range(max_iter):
z = A @ q
eigenvalue = np.dot(q, z) # Rayleigh quotient
q_new = z / np.linalg.norm(z)
if np.linalg.norm(q_new - q) < tol or np.linalg.norm(q_new + q) < tol:
print(f"Converged in {k+1} iterations")
return eigenvalue, q_new
q = q_new
return eigenvalue, q
# Natural frequency analysis of a vibration system
# Stiffness matrix of a double-pendulum mass-spring system
k = 1000 # N/m
m = 1 # kg
K = np.array([[2*k, -k], [-k, k]], dtype=float)
M = np.array([[m, 0], [0, m]], dtype=float)
# Generalized eigenvalue problem: K*v = lambda*M*v
# Transform: M^(-1/2) * K * M^(-1/2) * u = lambda * u
M_inv_sqrt = np.diag(1/np.sqrt(np.diag(M)))
A_sym = M_inv_sqrt @ K @ M_inv_sqrt
lam_max, v_max = power_method(A_sym)
omega_max = np.sqrt(lam_max)
# All eigenvalues (numpy)
eigenvalues_all = np.linalg.eigvalsh(A_sym)
omega_all = np.sqrt(eigenvalues_all)
print("Vibration system natural frequencies:")
for i, omega in enumerate(omega_all):
print(f" Mode {i+1}: {omega/(2*np.pi):.3f} Hz")
print(f"\nPower method maximum natural frequency: {omega_max/(2*np.pi):.3f} Hz")
7.2 QR Algorithm
Finds all eigenvalues simultaneously. The core algorithm of modern numerical linear algebra.
Idea: Iterative QR decomposition; the matrix converges to upper triangular form, with the diagonal elements becoming eigenvalues.
import numpy as np
def qr_algorithm(A, max_iter=1000, tol=1e-10):
"""Basic QR algorithm"""
A_k = A.copy().astype(float)
n = A.shape[0]
for k in range(max_iter):
Q, R = np.linalg.qr(A_k)
A_k_new = R @ Q
# Check magnitudes of subdiagonal elements
off_diag_norm = np.max(np.abs(np.tril(A_k_new, -1)))
if off_diag_norm < tol:
print(f"QR converged in {k+1} iterations")
break
A_k = A_k_new
return np.diag(A_k)
# Symmetric matrix (real eigenvalues)
A = np.array([
[4, 1, 0],
[1, 3, 1],
[0, 1, 2]
], dtype=float)
eig_qr = qr_algorithm(A)
eig_numpy = np.linalg.eigvalsh(A)
print("QR algorithm eigenvalues:", np.sort(eig_qr))
print("numpy eigenvalues: ", np.sort(eig_numpy))
7.3 Complete Eigenvalue Analysis: Structural Resonance Analysis
import numpy as np
import matplotlib.pyplot as plt
from scipy import linalg
def vibration_analysis(K, M, n_modes=None):
"""
Generalized eigenvalue problem: K*v = omega^2 * M * v
K: stiffness matrix, M: mass matrix
"""
# Solve generalized eigenvalue problem with scipy linalg.eigh
eigenvalues, eigenvectors = linalg.eigh(K, M)
omega = np.sqrt(np.abs(eigenvalues)) # angular frequency (rad/s)
freq = omega / (2 * np.pi) # frequency (Hz)
if n_modes is None:
n_modes = len(eigenvalues)
print("Natural Frequency Analysis Results:")
print("-" * 40)
for i in range(n_modes):
print(f"Mode {i+1}: {freq[i]:.3f} Hz ({omega[i]:.2f} rad/s)")
return omega[:n_modes], eigenvectors[:, :n_modes]
# 3-DOF vibration system
k1, k2, k3 = 1000, 1500, 800 # N/m
m1, m2, m3 = 1.0, 1.5, 0.8 # kg
K = np.array([
[k1+k2, -k2, 0],
[-k2, k2+k3, -k3],
[0, -k3, k3]
], dtype=float)
M = np.diag([m1, m2, m3])
omegas, modes = vibration_analysis(K, M)
# Visualize mode shapes
positions = np.arange(1, 4)
fig, axes = plt.subplots(1, 3, figsize=(14, 5))
for i, ax in enumerate(axes):
mode_shape = modes[:, i] / np.max(abs(modes[:, i]))
ax.plot([0] + list(positions), [0] + list(mode_shape),
'bo-', linewidth=2, markersize=10)
ax.axhline(y=0, color='gray', linestyle='--', alpha=0.5)
ax.set_title(f'Mode {i+1}\n{omegas[i]/(2*np.pi):.2f} Hz')
ax.set_xlabel('DOF')
ax.set_ylabel('Normalized mode shape')
ax.grid(True, alpha=0.3)
ax.set_xticks([0, 1, 2, 3])
ax.set_xticklabels(['Fixed', 'Mass 1', 'Mass 2', 'Mass 3'])
plt.suptitle('3-DOF Vibration System Mode Shapes')
plt.tight_layout()
plt.savefig('vibration_modes.png', dpi=150)
plt.show()
8. Comprehensive Example: MOSFET Bias Circuit Numerical Analysis
import numpy as np
from scipy.optimize import fsolve
from scipy.integrate import solve_ivp
import matplotlib.pyplot as plt
# MOSFET simplified model parameters
Vth = 1.5 # threshold voltage (V)
k = 2e-3 # transconductance parameter (A/V^2)
VDD = 15 # supply voltage (V)
RD = 3000 # drain resistance (Ohm)
RS = 1000 # source resistance (Ohm)
R1 = 100e3 # voltage divider resistor 1 (Ohm)
R2 = 47e3 # voltage divider resistor 2 (Ohm)
def mosfet_bias_equations(vars):
"""MOSFET bias nonlinear equation system"""
VGS, ID = vars
# Gate voltage from voltage divider
VG = VDD * R2 / (R1 + R2)
# Drain current (saturation region)
if VGS > Vth:
ID_calc = k / 2 * (VGS - Vth)**2
else:
ID_calc = 0
# KVL: VGS = VG - VS = VG - ID*RS
VGS_calc = VG - ID * RS
return [VGS - VGS_calc, ID - ID_calc]
VG = VDD * R2 / (R1 + R2)
initial_guess = [VG - 1, 1e-3]
solution = fsolve(mosfet_bias_equations, initial_guess, full_output=True)
VGS_dc, ID_dc = solution[0]
VDS_dc = VDD - ID_dc * (RD + RS)
print("MOSFET DC bias operating point:")
print(f" VGS = {VGS_dc:.3f} V")
print(f" ID = {ID_dc*1000:.3f} mA")
print(f" VDS = {VDS_dc:.3f} V")
print(f" Saturation check: VDS > VGS - Vth = {VGS_dc - Vth:.3f} V -> {'OK' if VDS_dc > VGS_dc - Vth else 'Linear region'}")
# Transfer characteristic curve
VGS_range = np.linspace(0, 5, 200)
ID_range = np.where(VGS_range > Vth,
k/2 * (VGS_range - Vth)**2,
0)
plt.figure(figsize=(10, 5))
plt.subplot(1, 2, 1)
plt.plot(VGS_range, ID_range * 1000, 'b-', linewidth=2)
plt.axvline(x=VGS_dc, color='r', linestyle='--', label=f'Q-point VGS={VGS_dc:.2f}V')
plt.axhline(y=ID_dc*1000, color='g', linestyle='--', label=f'Q-point ID={ID_dc*1000:.2f}mA')
plt.xlabel('VGS (V)')
plt.ylabel('ID (mA)')
plt.title('MOSFET Transfer Characteristics')
plt.legend()
plt.grid(True, alpha=0.3)
plt.subplot(1, 2, 2)
VDS_range = np.linspace(0, VDD, 200)
for VGS_val in [2.0, 2.5, 3.0, 3.5]:
if VGS_val <= Vth:
continue
ID_sat = k/2 * (VGS_val - Vth)**2
ID_curve = np.where(
VDS_range < VGS_val - Vth,
k * ((VGS_val - Vth)*VDS_range - VDS_range**2/2),
ID_sat
)
plt.plot(VDS_range, ID_curve * 1000,
label=f'VGS={VGS_val}V')
# Load line
ID_load = (VDD - VDS_range) / (RD + RS)
plt.plot(VDS_range, ID_load * 1000, 'k--', linewidth=2, label='Load line')
plt.axvline(x=VDS_dc, color='r', linestyle=':', alpha=0.7)
plt.xlabel('VDS (V)')
plt.ylabel('ID (mA)')
plt.title('MOSFET Output Characteristics and Load Line')
plt.legend(fontsize=8)
plt.ylim([0, None])
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.savefig('mosfet_analysis.png', dpi=150)
plt.show()
Summary and Next Steps
Topics covered in Part 4:
- Error Analysis: Round-off/truncation error, absolute/relative error, numerical stability, condition number
- Nonlinear equations: Bisection (1st order), Newton-Raphson (2nd order), Secant (1.618th order), scipy.optimize
- Numerical differentiation: Forward/backward/central differences, error analysis, optimal step size
- Numerical integration: Trapezoidal, Simpson's, Gaussian quadrature, scipy.integrate
- ODE numerical solvers: Euler's method, Heun's method, full RK4 implementation, stiff ODEs
- Linear systems: Gaussian elimination, LU decomposition, iterative methods, numpy.linalg
- Interpolation: Lagrange, splines, Runge's phenomenon, least squares, nonlinear curve fitting
- Eigenvalue problems: Power method, QR algorithm, vibration analysis
Through the four-part engineering mathematics series covering ODEs, Fourier analysis, PDEs, complex analysis, Z-transform, and numerical methods, we have thoroughly mastered the core mathematical tools of electrical, electronic, and mechanical engineering.
9. Numerical Methods in AI/ML: Automatic Differentiation
9.1 Numerical vs Automatic Differentiation
Numerical differentiation approximates derivatives with finite differences; automatic differentiation (autograd) computes exact derivatives by mechanically applying the chain rule through a computation graph.
import numpy as np
def numerical_grad(f, x, h=1e-5):
"""Central difference gradient (vector input)."""
grad = np.zeros_like(x, dtype=float)
for i in range(len(x)):
x_plus = x.copy().astype(float); x_plus[i] += h
x_minus = x.copy().astype(float); x_minus[i] -= h
grad[i] = (f(x_plus) - f(x_minus)) / (2 * h)
return grad
# f(x, y) = x^2 + 3*x*y + y^2, true gradient: [2x+3y, 3x+2y]
def f_quad(x):
return x[0]**2 + 3*x[0]*x[1] + x[1]**2
x0 = np.array([1.0, 2.0])
grad_num = numerical_grad(f_quad, x0)
grad_true = np.array([2*x0[0] + 3*x0[1], 3*x0[0] + 2*x0[1]])
print("Gradient at (1, 2):")
print(f" Numerical: {grad_num}")
print(f" Analytical: {grad_true}")
print(f" Max error: {np.max(abs(grad_num - grad_true)):.2e}")
Key differences:
| Property | Numerical diff. | Automatic diff. (AD) |
|---|---|---|
| Accuracy | , limited by floating-point | Machine precision |
| Cost (gradient) | forward passes for params | backward pass (reverse mode) |
| Step-size tuning | Required | Not needed |
| Use in ML | Finite-difference gradient checks | Backpropagation in PyTorch / JAX |
9.2 Gradient Descent Variants
Numerical optimization drives ML model training. The mathematics behind each optimizer directly determines convergence behavior.
import numpy as np
import matplotlib.pyplot as plt
# Rosenbrock function: f(x,y) = (1-x)^2 + 100*(y-x^2)^2
def rosenbrock(w):
x, y = w[0], w[1]
return (1 - x)**2 + 100*(y - x**2)**2
def rosenbrock_grad(w):
x, y = w[0], w[1]
gx = -2*(1 - x) - 400*x*(y - x**2)
gy = 200*(y - x**2)
return np.array([gx, gy])
w0 = np.array([-1.5, 1.5])
lr = 0.001
n_iter = 3000
# Standard SGD
w_sgd = w0.copy()
history_sgd = [w_sgd.copy()]
for _ in range(n_iter):
w_sgd -= lr * rosenbrock_grad(w_sgd)
history_sgd.append(w_sgd.copy())
# Momentum SGD
w_mom = w0.copy()
v = np.zeros(2)
beta = 0.9
history_mom = [w_mom.copy()]
for _ in range(n_iter):
v = beta * v - lr * rosenbrock_grad(w_mom)
w_mom += v
history_mom.append(w_mom.copy())
# Adam
w_adam = w0.copy()
m_adam, v_adam = np.zeros(2), np.zeros(2)
beta1, beta2, eps = 0.9, 0.999, 1e-8
history_adam = [w_adam.copy()]
for t in range(1, n_iter + 1):
g = rosenbrock_grad(w_adam)
m_adam = beta1 * m_adam + (1 - beta1) * g
v_adam = beta2 * v_adam + (1 - beta2) * g**2
m_hat = m_adam / (1 - beta1**t)
v_hat = v_adam / (1 - beta2**t)
w_adam -= lr * m_hat / (np.sqrt(v_hat) + eps)
history_adam.append(w_adam.copy())
loss_sgd = [rosenbrock(w) for w in history_sgd]
loss_mom = [rosenbrock(w) for w in history_mom]
loss_adam = [rosenbrock(w) for w in history_adam]
plt.figure(figsize=(10, 5))
plt.semilogy(loss_sgd, label='SGD')
plt.semilogy(loss_mom, label='Momentum')
plt.semilogy(loss_adam, label='Adam')
plt.xlabel('Iteration')
plt.ylabel('Loss (log scale)')
plt.title('Optimizer Convergence on Rosenbrock Function')
plt.legend()
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.savefig('optimizer_comparison.png', dpi=150)
plt.show()
print(f"Final loss — SGD: {loss_sgd[-1]:.4f}, Momentum: {loss_mom[-1]:.4f}, Adam: {loss_adam[-1]:.6f}")
Quiz
Q1. When does Newton-Raphson fail to converge?
Answer: Newton-Raphson fails in several situations:
- Derivative is zero: If at any iterate, the update is undefined (division by zero).
- Poor initial guess: If is far from the root, the iteration may cycle or diverge.
- Multiple roots: Near a root where , convergence degrades from quadratic to linear.
- Flat regions / local extrema: Near a local minimum, the tangent line intersects the x-axis far away.
Explanation: The quadratic convergence proof requires and a sufficiently close starting point. In practice, bracket a root with bisection first, then switch to Newton-Raphson for fast final convergence.
Q2. Why is LU decomposition preferred over Gaussian elimination for multiple right-hand-side systems?
Answer: LU decomposition separates the expensive factorization from the cheap solve.
Explanation: Gaussian elimination costs per system. With different right-hand side vectors and the same matrix , the total cost is .
LU factorization is performed once in ; then each new right-hand side requires only two triangular solves (, ) at each. Total: — dramatically cheaper when is large (e.g., finite element systems with multiple load cases).
Q3. What is the difference between Lagrange and spline interpolation for noisy data?
Answer: Spline interpolation is strongly preferred for noisy data.
Explanation: Lagrange interpolation fits a single degree- polynomial through all data points exactly. For noisy data this forces the polynomial to pass through corrupted measurements, producing Runge's phenomenon oscillations that grow severe at high degrees.
Cubic splines use degree-3 polynomials on each sub-interval with smoothness conditions (matching first and second derivatives at knots). This avoids oscillations and produces a physically sensible smooth curve. For truly noisy data, least-squares fitting (not interpolation) is best: fit a smooth model that minimizes residuals rather than passing through each point exactly.
Q4. Why does Runge-Kutta 4 have 4th-order accuracy?
Answer: RK4 evaluates at four strategically chosen points and combines them with weights that match the Taylor expansion through the term.
Explanation: Euler's method truncates the Taylor series after the first term, giving global error. RK4 computes four slopes:
- : slope at the start
- , : slopes at midpoints
- : slope at the end
The weighted combination exactly matches the Taylor expansion of through order . The local error is and the global error is . Halving the step size reduces global error by .
Q5. How does automatic differentiation (autograd) differ from numerical differentiation?
Answer: Automatic differentiation computes exact derivatives (to machine precision) by applying the chain rule through a computation graph; numerical differentiation approximates derivatives with finite differences and suffers from a truncation/round-off trade-off.
Explanation:
- Numerical differentiation: . Optimal gives roughly 8 correct digits. Accuracy is fundamentally limited.
- Reverse-mode AD (backpropagation): Decomposes the function into elementary operations and applies the chain rule exactly at each step. Computes gradients with respect to all parameters in a single backward pass at roughly the same cost as one forward pass — essential for neural networks with millions of parameters.
- Key practical difference: AD gives machine-precision gradients with no step-size tuning, which is why PyTorch, JAX, and TensorFlow are built on it.
References
- Burden, R. & Faires, J. "Numerical Analysis", 10th Edition, Cengage
- Heath, M. "Scientific Computing: An Introductory Survey", 2nd Edition
- Press, W. et al. "Numerical Recipes: The Art of Scientific Computing", 3rd Edition
- Kreyszig, E. "Advanced Engineering Mathematics", 10th Edition, Wiley
- NumPy Linear Algebra Documentation
- SciPy Optimization Documentation
- SciPy Integration Documentation