Cloud DevOps 实战项目¶
📚 项目概述¶
本章提供 5 个完整的实战项目,帮助你将理论知识转化为实际能力。每个项目都包含完整的技术栈、实现步骤、代码示例和部署指南。
🎯 项目列表¶
项目 1 :电商网站云原生部署¶
项目概述¶
构建一个完整的电商网站,使用云原生技术栈部署到 Kubernetes 集群。
技术栈¶
- 前端: React + Nginx
- 后端: Python Flask + RESTful API
- 数据库: MySQL + Redis
- 容器化: Docker + Docker Compose
- 编排: Kubernetes + Helm
- CI/CD: GitLab CI/CD
- 监控: Prometheus + Grafana
- 日志: ELK Stack
实现步骤¶
- 应用开发
Python
# app.py
from flask import Flask, jsonify, request
import redis
import pymysql
app = Flask(__name__)
# Redis连接
redis_client = redis.Redis(host='redis', port=6379, db=0)
# 数据库连接
def get_db_connection():
return pymysql.connect(
host='mysql',
user='root',
password='password',
database='ecommerce'
)
@app.route('/api/products', methods=['GET'])
def get_products():
# 尝试从缓存获取
cached = redis_client.get('products:list')
if cached:
return jsonify({'source': 'cache', 'data': cached})
# 从数据库获取
conn = get_db_connection()
cursor = conn.cursor()
cursor.execute('SELECT * FROM products')
products = cursor.fetchall()
conn.close()
# 缓存结果
redis_client.setex('products:list', 300, str(products))
return jsonify({'source': 'database', 'data': products})
@app.route('/api/orders', methods=['POST'])
def create_order():
order_data = request.json
# 保存到数据库
conn = get_db_connection()
cursor = conn.cursor()
cursor.execute(
'INSERT INTO orders (user_id, product_id, quantity, total) VALUES (%s, %s, %s, %s)',
(order_data['user_id'], order_data['product_id'], order_data['quantity'], order_data['total'])
)
conn.commit()
conn.close()
# 清除缓存
redis_client.delete('orders:list')
return jsonify({'message': 'Order created successfully'}), 201
if __name__ == '__main__':
app.run(host='0.0.0.0', port=8000)
- Docker 化
Docker
# Dockerfile
FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
EXPOSE 8000
CMD ["gunicorn", "--bind", "0.0.0.0:8000", "--workers", "4", "app:app"]
- Kubernetes 部署
YAML
# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: ecommerce-backend
spec:
replicas: 3
selector:
matchLabels:
app: backend
template:
metadata:
labels:
app: backend
spec:
containers:
- name: backend
image: ecommerce/backend:v1.0
ports:
- containerPort: 8000
env:
- name: DB_HOST
value: mysql
- name: REDIS_HOST
value: redis
- CI/CD 流水线
YAML
# .gitlab-ci.yml
stages:
- build
- test
- deploy
build:
stage: build
image: docker:latest
services:
- docker:dind
script:
- docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
- docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
test:
stage: test
image: python:3.9
script:
- pip install -r requirements.txt
- pytest tests/
deploy:
stage: deploy
image: bitnami/kubectl:latest
script:
- kubectl set image deployment/ecommerce-backend backend=$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
- kubectl rollout status deployment/ecommerce-backend
only:
- main
部署指南¶
Bash
# 1. 克隆项目
git clone https://github.com/yourorg/ecommerce.git
cd ecommerce
# 2. 构建镜像
docker build -t ecommerce/backend:v1.0 .
# 3. 部署到Kubernetes
kubectl apply -f k8s/
# 4. 验证部署
kubectl get pods
kubectl get services
优化建议¶
- 性能优化
- 使用 Redis 缓存热点数据
- 数据库读写分离
-
CDN 加速静态资源
-
高可用
- 多可用区部署
- 自动扩缩容
-
健康检查和自动恢复
-
安全
- 启用 HTTPS
- 实施 RBAC
- 定期安全扫描
项目 2 :微服务架构电商平台¶
项目概述¶
构建一个基于微服务架构的电商平台,使用服务网格管理服务间通信。
技术栈¶
- 服务网格: Istio
- API 网关: Kong
- 服务:用户服务、商品服务、订单服务、支付服务
- 消息队列: Kafka
- 分布式追踪: Jaeger
- 配置中心: Consul
实现步骤¶
- 服务设计
Python
# 用户服务
@app.route('/api/users/<user_id>', methods=['GET'])
def get_user(user_id):
user = db.get_user(user_id)
return jsonify(user)
# 商品服务
@app.route('/api/products/<product_id>', methods=['GET'])
def get_product(product_id):
product = db.get_product(product_id)
return jsonify(product)
# 订单服务
@app.route('/api/orders', methods=['POST'])
def create_order():
order_data = request.json
# 调用用户服务验证用户
user = call_service('user-service', f'/api/users/{order_data["user_id"]}')
# 调用商品服务获取商品信息
product = call_service('product-service', f'/api/products/{order_data["product_id"]}')
# 创建订单
order = db.create_order(order_data)
# 发送消息到Kafka
kafka.send('orders', order)
return jsonify(order)
- Istio 配置
YAML
# virtualservice.yaml
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: orders
spec:
hosts:
- orders
http:
- match:
- headers:
end-user:
exact: jason
route:
- destination:
host: orders
subset: v2
- route:
- destination:
host: orders
subset: v1
weight: 90
- destination:
host: orders
subset: v2
weight: 10
- 分布式追踪
Python
from opentelemetry import trace
from opentelemetry.instrumentation.flask import FlaskInstrumentor
tracer = trace.get_tracer(__name__)
FlaskInstrumentor().instrument_app(app)
@app.route('/api/orders')
def create_order():
with tracer.start_as_current_span("create_order") as span:
span.set_attribute("user.id", user_id)
# 业务逻辑
部署指南¶
Bash
# 1. 安装Istio
istioctl install --set profile=demo
# 2. 部署服务
kubectl apply -f k8s/services/
# 3. 配置Istio
kubectl apply -f k8s/istio/
# 4. 验证部署
kubectl get pods -n istio-system
istioctl proxy-status
优化建议¶
- 流量管理
- 使用 Istio 进行灰度发布
- 实施熔断和限流
-
故障注入测试
-
可观测性
- 分布式追踪
- 指标收集
-
日志聚合
-
安全
- mTLS 加密
- 服务间认证
- 访问控制
项目 3 : Serverless 无服务器应用¶
项目概述¶
构建一个基于 Serverless 架构的应用,实现按需计费和自动扩缩容。
技术栈¶
- 计算: AWS Lambda
- API 网关: API Gateway
- 数据库: DynamoDB
- 存储: S3
- 消息: SNS/SQS
- CI/CD: Serverless Framework
实现步骤¶
- Lambda 函数
Python
# handler.py
import json
import boto3
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('Users')
def lambda_handler(event, context):
http_method = event['requestContext']['http']['method']
path = event['requestContext']['http']['path']
if http_method == 'GET' and path == '/users':
return get_users()
elif http_method == 'POST' and path == '/users':
return create_user(event)
else:
return {
'statusCode': 404,
'body': json.dumps({'error': 'Not found'})
}
def get_users():
response = table.scan()
return {
'statusCode': 200,
'body': json.dumps(response['Items'])
}
def create_user(event):
user_data = json.loads(event['body'])
table.put_item(Item=user_data)
return {
'statusCode': 201,
'body': json.dumps(user_data)
}
- Serverless 配置
YAML
# serverless.yml
service: serverless-app
provider:
name: aws
runtime: python3.9
region: us-east-1
environment:
TABLE_NAME: ${self:provider.stage}-users
functions:
api:
handler: handler.lambda_handler
events:
- httpApi:
path: /{proxy+}
method: ANY
resources:
Resources:
UsersTable:
Type: AWS::DynamoDB::Table
Properties:
TableName: ${self:provider.stage}-users
AttributeDefinitions:
- AttributeName: id
AttributeType: S
KeySchema:
- AttributeName: id
KeyType: HASH
BillingMode: PAY_PER_REQUEST
部署指南¶
Bash
# 1. 安装Serverless Framework
npm install -g serverless
# 2. 配置AWS凭证
serverless config credentials --provider aws --key <your-key> --secret <your-secret>
# 3. 部署
serverless deploy --stage production
# 4. 测试
curl https://api.execute-api.us-east-1.amazonaws.com/production/users
优化建议¶
- 性能优化
- 预热函数
- 优化依赖
-
使用 Lambda 层
-
成本优化
- 合理设置内存
- 使用预留并发
-
监控使用情况
-
安全
- 最小权限 IAM
- 加密数据
- 实施 VPC
项目 4 : CI/CD 流水线自动化¶
项目概述¶
构建一个完整的 CI/CD 流水线,实现自动化构建、测试和部署。
技术栈¶
- 版本控制: GitLab
- CI/CD: GitLab CI/CD
- 容器化: Docker
- 编排: Kubernetes
- 测试: Pytest, Selenium
- 代码质量: SonarQube
实现步骤¶
- CI 流水线
YAML
# .gitlab-ci.yml
stages:
- build
- test
- quality
- security
- deploy
variables:
DOCKER_REGISTRY: registry.example.com
IMAGE_NAME: myapp
build:
stage: build
image: docker:latest
services:
- docker:dind
script:
- docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $DOCKER_REGISTRY
- docker build -t $DOCKER_REGISTRY/$IMAGE_NAME:$CI_COMMIT_SHA .
- docker push $DOCKER_REGISTRY/$IMAGE_NAME:$CI_COMMIT_SHA
- docker tag $DOCKER_REGISTRY/$IMAGE_NAME:$CI_COMMIT_SHA $DOCKER_REGISTRY/$IMAGE_NAME:latest
- docker push $DOCKER_REGISTRY/$IMAGE_NAME:latest
test:
stage: test
image: python:3.9
services:
- postgres:13
- redis:latest
variables:
POSTGRES_DB: testdb
POSTGRES_USER: testuser
POSTGRES_PASSWORD: testpass
script:
- pip install -r requirements.txt
- pytest tests/ -v --cov=app --cov-report=xml
coverage: '/TOTAL.*\s+(\d+%)$/'
artifacts:
reports:
coverage_report:
coverage_format: cobertura
path: coverage.xml
quality:
stage: quality
image: sonarsource/sonar-scanner-cli
script:
- sonar-scanner -Dsonar.projectKey=myproject -Dsonar.sources=. -Dsonar.host.url=$SONAR_HOST -Dsonar.login=$SONAR_TOKEN
security:
stage: security
image: aquasec/trivy:latest
script:
- trivy image --exit-code 1 --severity HIGH,CRITICAL $DOCKER_REGISTRY/$IMAGE_NAME:$CI_COMMIT_SHA
allow_failure: true
deploy_staging:
stage: deploy
image: bitnami/kubectl:latest
script:
- kubectl config use-context staging
- kubectl set image deployment/myapp app=$DOCKER_REGISTRY/$IMAGE_NAME:$CI_COMMIT_SHA
- kubectl rollout status deployment/myapp
only:
- develop
deploy_production:
stage: deploy
image: bitnami/kubectl:latest
script:
- kubectl config use-context production
- kubectl set image deployment/myapp app=$DOCKER_REGISTRY/$IMAGE_NAME:$CI_COMMIT_SHA
- kubectl rollout status deployment/myapp
only:
- tags
when: manual
- 自动化测试
Python
# tests/test_api.py
import pytest
from app import app
@pytest.fixture
def client():
app.config['TESTING'] = True
with app.test_client() as client:
yield client
def test_get_users(client):
response = client.get('/api/users')
assert response.status_code == 200
assert isinstance(response.json, list)
def test_create_user(client):
user_data = {
'name': 'Test User',
'email': 'test@example.com'
}
response = client.post('/api/users', json=user_data)
assert response.status_code == 201
assert response.json['name'] == 'Test User'
部署指南¶
Bash
# 1. 配置GitLab Runner
gitlab-runner register \
--url https://gitlab.example.com \
--registration-token <token> \
--executor docker \
--description "My Runner"
# 2. 配置Kubernetes集群
kubectl config use-context production
# 3. 推送代码
git push origin main
# 4. 查看流水线
# 访问GitLab CI/CD页面
优化建议¶
- 流水线优化
- 并行执行
- 缓存依赖
-
增量构建
-
测试优化
- 单元测试
- 集成测试
-
端到端测试
-
质量保证
- 代码审查
- 静态分析
- 安全扫描
项目 5 :监控告警系统¶
项目概述¶
构建一个完整的监控告警系统,实现实时监控和智能告警。
技术栈¶
- 监控: Prometheus
- 可视化: Grafana
- 告警: AlertManager
- 日志: ELK Stack
- 追踪: Jaeger
- 指标导出器: Node Exporter, cAdvisor
实现步骤¶
- Prometheus 配置
YAML
# prometheus.yml
global:
scrape_interval: 15s
evaluation_interval: 15s
rule_files:
- 'alerts/*.yml'
scrape_configs:
- job_name: 'kubernetes-pods'
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: true
alerting:
alertmanagers:
- static_configs:
- targets:
- alertmanager:9093
- 告警规则
YAML
# alerts/rules.yml
groups:
- name: alert_rules
interval: 30s
rules:
- alert: InstanceDown
expr: up == 0
for: 5m
labels:
severity: critical
annotations:
summary: "实例 {{ $labels.instance }} 宕机"
description: "实例 {{ $labels.instance }} 已经宕机超过5分钟"
- alert: HighCPUUsage
expr: 100 - (avg by(instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
for: 10m
labels:
severity: warning
annotations:
summary: "实例 {{ $labels.instance }} CPU使用率过高"
description: "实例 {{ $labels.instance }} CPU使用率超过80%,当前值:{{ $value }}%"
- Grafana 仪表板
JSON
{
"dashboard": {
"title": "应用监控",
"panels": [
{
"title": "CPU使用率",
"targets": [
{
"expr": "100 - (avg by(instance) (irate(node_cpu_seconds_total{mode=\"idle\"}[5m])) * 100)"
}
]
},
{
"title": "内存使用率",
"targets": [
{
"expr": "(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100"
}
]
}
]
}
}
部署指南¶
Bash
# 1. 安装Prometheus Operator
kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/main/bundle.yaml
# 2. 创建Prometheus实例
kubectl apply -f prometheus/
# 3. 安装Grafana
kubectl apply -f grafana/
# 4. 配置数据源
# 访问Grafana UI,添加Prometheus数据源
# 5. 导入仪表板
# 导入预定义的Grafana仪表板
优化建议¶
- 监控优化
- 合理设置采集间隔
- 使用 Recording Rules
-
优化查询性能
-
告警优化
- 合理设置阈值
- 告警分组
-
告警抑制
-
可视化优化
- 设计清晰的仪表板
- 使用变量模板
- 定期审查
📊 项目对比¶
| 项目 | 技术栈 | 难度 | 耗时 | 适用场景 |
|---|---|---|---|---|
| 电商网站云原生部署 | Docker, K8s, Helm | ⭐⭐ | 2 周 | 传统应用云化 |
| 微服务架构电商平台 | Istio, Kong, Kafka | ⭐⭐⭐ | 3 周 | 微服务架构 |
| Serverless 无服务器应用 | Lambda, API Gateway | ⭐⭐ | 2 周 | Serverless 应用 |
| CI/CD 流水线自动化 | GitLab CI, K8s | ⭐⭐ | 2 周 | DevOps 实践 |
| 监控告警系统 | Prometheus, Grafana | ⭐⭐ | 2 周 | 监控告警 |
🎯 学习建议¶
- 循序渐进
- 从简单项目开始
- 逐步增加复杂度
-
理解每个技术点
-
动手实践
- 亲自完成每个项目
- 不要只看不做
-
记录遇到的问题
-
扩展优化
- 在基础项目上扩展
- 尝试不同的实现方式
-
优化性能和成本
-
总结反思
- 项目完成后总结
- 记录经验教训
- 分享给他人
📚 参考资料¶
- Kubernetes 官方文档: HTTPS://Kubernetes.io/docs/
- Istio 官方文档: HTTPS://istio.io/latest/docs/
- AWS Lambda 文档: HTTPS://docs.AWS.amazon.com/lambda/
- Prometheus 官方文档: HTTPS://prometheus.io/docs/
- GitLab CI/CD 文档: HTTPS://docs.GitLab.com/ee/ci/
🎯 总结¶
通过完成这 5 个实战项目,你将:
- 掌握云原生技术的实际应用
- 理解微服务架构的设计和实现
- 熟悉 Serverless 架构的开发模式
- 掌握 CI/CD 流水线的搭建和优化
- 具备监控告警系统的实施能力
每个项目都是完整的、可运行的,可以直接部署到生产环境。建议按照项目顺序逐步完成,每个项目完成后进行总结和反思,为下一个项目做好准备。