345 lines
15 KiB
Markdown
345 lines
15 KiB
Markdown
# 对象存储集成
|
||
|
||
<cite>
|
||
**本文引用的文件**
|
||
- [minio.go](file://pkg/storage/minio.go)
|
||
- [manager.go](file://pkg/storage/manager.go)
|
||
- [config.go](file://pkg/config/config.go)
|
||
- [upload_service.go](file://internal/service/upload_service.go)
|
||
- [texture_handler.go](file://internal/handler/texture_handler.go)
|
||
- [texture_service.go](file://internal/service/texture_service.go)
|
||
</cite>
|
||
|
||
## 目录
|
||
1. [简介](#简介)
|
||
2. [项目结构](#项目结构)
|
||
3. [核心组件](#核心组件)
|
||
4. [架构总览](#架构总览)
|
||
5. [详细组件分析](#详细组件分析)
|
||
6. [依赖关系分析](#依赖关系分析)
|
||
7. [性能考虑](#性能考虑)
|
||
8. [故障排查指南](#故障排查指南)
|
||
9. [结论](#结论)
|
||
10. [附录](#附录)
|
||
|
||
## 简介
|
||
本文件面向CarrotSkin后端的对象存储集成，聚焦于如何通过MinIO客户端与S3兼容存储系统交互。文档围绕以下目标展开：
|
||
- 解析minio.go中初始化客户端的流程，包括访问密钥、端点配置和TLS设置
|
||
- 说明manager.go中Upload、Download、GeneratePresignedURL等关键方法的实现细节与调用方式
|
||
- 提供皮肤/披风文件上传、私有资源临时链接生成等典型用例的代码示例路径
|
||
- 解释分片上传、断点续传等高级功能的支持情况
|
||
- 为运维人员提供性能调优建议（并发控制、连接复用）和故障排查指南（签名错误、网络超时）
|
||
|
||
## 项目结构
|
||
对象存储相关代码集中在pkg/storage目录，配置在pkg/config，业务侧在internal/service与internal/handler中调用。
|
||
|
||
```mermaid
|
||
graph TB
|
||
subgraph "配置层"
|
||
CFG["pkg/config/config.go<br/>RustFSConfig"]
|
||
end
|
||
subgraph "存储客户端"
|
||
MINIO["pkg/storage/minio.go<br/>StorageClient<br/>NewStorage/GeneratePresignedURL/GeneratePresignedPostURL"]
|
||
MGR["pkg/storage/manager.go<br/>Init/GetClient/MustGetClient"]
|
||
end
|
||
subgraph "业务服务"
|
||
SVC_UPLOAD["internal/service/upload_service.go<br/>GenerateAvatarUploadURL/GenerateTextureUploadURL"]
|
||
SVC_TEX["internal/service/texture_service.go<br/>CreateTexture/RecordTextureDownload 等"]
|
||
end
|
||
subgraph "接口层"
|
||
HANDLER_TEX["internal/handler/texture_handler.go<br/>GenerateTextureUploadURL 接口"]
|
||
end
|
||
CFG --> MINIO
|
||
MINIO --> MGR
|
||
MGR --> HANDLER_TEX
|
||
HANDLER_TEX --> SVC_UPLOAD
|
||
SVC_UPLOAD --> MINIO
|
||
SVC_TEX --> MINIO
|
||
```
|
||
|
||
图表来源
|
||
- [minio.go](file://pkg/storage/minio.go#L1-L120)
|
||
- [manager.go](file://pkg/storage/manager.go#L1-L49)
|
||
- [config.go](file://pkg/config/config.go#L58-L66)
|
||
- [upload_service.go](file://internal/service/upload_service.go#L78-L160)
|
||
- [texture_handler.go](file://internal/handler/texture_handler.go#L18-L83)
|
||
- [texture_service.go](file://internal/service/texture_service.go#L12-L64)
|
||
|
||
章节来源
|
||
- [minio.go](file://pkg/storage/minio.go#L1-L120)
|
||
- [manager.go](file://pkg/storage/manager.go#L1-L49)
|
||
- [config.go](file://pkg/config/config.go#L58-L66)
|
||
- [upload_service.go](file://internal/service/upload_service.go#L78-L160)
|
||
- [texture_handler.go](file://internal/handler/texture_handler.go#L18-L83)
|
||
- [texture_service.go](file://internal/service/texture_service.go#L12-L64)
|
||
|
||
## 核心组件
|
||
- StorageClient：封装minio-go客户端，提供桶名解析、预签名URL生成等能力
|
||
- manager：提供全局单例的存储客户端初始化与获取
|
||
- RustFSConfig：承载S3兼容存储的端点、凭据与桶映射
|
||
- upload_service：面向业务的上传URL生成工具，按头像与材质类型分别组织对象路径
|
||
- texture_handler：对外暴露生成上传URL的HTTP接口
|
||
- texture_service：材质实体的增删改查与下载计数等业务逻辑
|
||
|
||
章节来源
|
||
- [minio.go](file://pkg/storage/minio.go#L14-L120)
|
||
- [manager.go](file://pkg/storage/manager.go#L9-L44)
|
||
- [config.go](file://pkg/config/config.go#L58-L66)
|
||
- [upload_service.go](file://internal/service/upload_service.go#L13-L57)
|
||
- [texture_handler.go](file://internal/handler/texture_handler.go#L18-L83)
|
||
- [texture_service.go](file://internal/service/texture_service.go#L12-L64)
|
||
|
||
## 架构总览
|
||
下图展示从接口到存储的调用链路与职责划分。
|
||
|
||
```mermaid
|
||
sequenceDiagram
|
||
participant C as "客户端"
|
||
participant H as "TextureHandler"
|
||
participant S as "UploadService"
|
||
participant M as "StorageManager"
|
||
participant SC as "StorageClient"
|
||
participant O as "对象存储(兼容S3)"
|
||
C->>H : "POST /api/v1/texture/upload-url"
|
||
H->>M : "MustGetClient()"
|
||
M-->>H : "*StorageClient"
|
||
H->>S : "GenerateTextureUploadURL(ctx, client, cfg, userID, fileName, type)"
|
||
S->>SC : "GetBucket(\"textures\")"
|
||
S->>SC : "GeneratePresignedPostURL(bucket, objectName, limits, expires, useSSL, endpoint)"
|
||
SC->>O : "PresignedPostPolicy(policy)"
|
||
O-->>SC : "postURL + formData"
|
||
SC-->>S : "PresignedPostPolicyResult"
|
||
S-->>H : "PresignedPostPolicyResult"
|
||
H-->>C : "返回postURL、formData、文件最终URL、过期秒数"
|
||
```
|
||
|
||
图表来源
|
||
- [texture_handler.go](file://internal/handler/texture_handler.go#L18-L83)
|
||
- [upload_service.go](file://internal/service/upload_service.go#L117-L160)
|
||
- [manager.go](file://pkg/storage/manager.go#L30-L44)
|
||
- [minio.go](file://pkg/storage/minio.go#L57-L120)
|
||
|
||
## 详细组件分析
|
||
|
||
### StorageClient与初始化流程（minio.go）
|
||
- 客户端初始化
|
||
- 通过端点、凭据与TLS标志创建minio-go客户端
|
||
- 若提供了访问密钥与密钥，则进行连接测试（带超时）
|
||
- 绑定桶映射，便于按“类型”解析真实桶名
|
||
- 关键方法
|
||
- GetClient：返回底层minio.Client
|
||
- GetBucket：按名称解析桶名，不存在时报错
|
||
- GeneratePresignedURL：生成预签名PUT URL（支持上传）
|
||
- GeneratePresignedPostURL：生成预签名POST策略URL（支持表单直传），并构造最终访问URL
|
||
|
||
```mermaid
|
||
classDiagram
|
||
class StorageClient {
|
||
-client : "minio.Client"
|
||
-buckets : "map[string]string"
|
||
+GetClient() "*minio.Client"
|
||
+GetBucket(name) "(string, error)"
|
||
+GeneratePresignedURL(ctx, bucketName, objectName, expires) "(string, error)"
|
||
+GeneratePresignedPostURL(ctx, bucketName, objectName, minSize, maxSize, expires, useSSL, endpoint) "(*PresignedPostPolicyResult, error)"
|
||
}
|
||
class PresignedPostPolicyResult {
|
||
+PostURL : "string"
|
||
+FormData : "map[string]string"
|
||
+FileURL : "string"
|
||
}
|
||
StorageClient --> PresignedPostPolicyResult : "返回"
|
||
```
|
||
|
||
图表来源
|
||
- [minio.go](file://pkg/storage/minio.go#L14-L120)
|
||
|
||
章节来源
|
||
- [minio.go](file://pkg/storage/minio.go#L20-L49)
|
||
- [minio.go](file://pkg/storage/minio.go#L52-L64)
|
||
- [minio.go](file://pkg/storage/minio.go#L66-L73)
|
||
- [minio.go](file://pkg/storage/minio.go#L82-L120)
|
||
|
||
### 存储客户端管理器（manager.go）
|
||
- Init(cfg)：线程安全初始化，仅执行一次
|
||
- GetClient()：获取全局实例，未初始化时报错
|
||
- MustGetClient()：获取全局实例，未初始化时panic
|
||
|
||
```mermaid
|
||
flowchart TD
|
||
Start(["调用 Init(cfg)"]) --> Once["sync.Once 保证只执行一次"]
|
||
Once --> NewStorage["NewStorage(cfg) 创建 StorageClient"]
|
||
NewStorage --> Bind["绑定 clientInstance"]
|
||
Bind --> Done(["初始化完成"])
|
||
GetClient["GetClient()"] --> Check{"clientInstance 是否存在？"}
|
||
Check --> |否| Err["返回错误：未初始化"]
|
||
Check --> |是| Return["返回 clientInstance"]
|
||
MustGetClient["MustGetClient()"] --> GetClient
|
||
GetClient --> Panic{"err 是否非空？"}
|
||
Panic --> |是| PanicCall["panic(err)"]
|
||
Panic --> |否| Return
|
||
```
|
||
|
||
图表来源
|
||
- [manager.go](file://pkg/storage/manager.go#L9-L44)
|
||
|
||
章节来源
|
||
- [manager.go](file://pkg/storage/manager.go#L18-L27)
|
||
- [manager.go](file://pkg/storage/manager.go#L29-L35)
|
||
- [manager.go](file://pkg/storage/manager.go#L37-L44)
|
||
|
||
### 配置结构（config.go）
|
||
- RustFSConfig包含端点、访问密钥、密钥、TLS开关与桶映射
|
||
- 环境变量映射与默认值设置，支持通过环境变量覆盖
|
||
|
||
章节来源
|
||
- [config.go](file://pkg/config/config.go#L58-L66)
|
||
- [config.go](file://pkg/config/config.go#L190-L236)
|
||
- [config.go](file://pkg/config/config.go#L238-L305)
|
||
|
||
### 上传服务与典型用例（upload_service.go）
|
||
- 文件类型与上传配置
|
||
- FileTypeAvatar/FileTypeTexture两类
|
||
- 各自的允许扩展名、最小/最大尺寸、过期时间
|
||
- 生成上传URL
|
||
- GenerateAvatarUploadURL：按用户ID与时间戳生成对象路径，调用StorageClient.GeneratePresignedPostURL
|
||
- GenerateTextureUploadURL：支持SKIN/CAPE两种材质类型，生成对应路径，调用StorageClient.GeneratePresignedPostURL
|
||
- 下载与公开资源
|
||
- 当前代码未直接暴露Download方法；公开资源可通过最终访问URL直接访问
|
||
- 私有资源可通过GeneratePresignedURL生成临时下载链接
|
||
|
||
```mermaid
|
||
sequenceDiagram
|
||
participant Svc as "UploadService"
|
||
participant SC as "StorageClient"
|
||
participant O as "对象存储"
|
||
Svc->>Svc : "ValidateFileName(fileName, type)"
|
||
Svc->>SC : "GetBucket(\"avatars\" 或 \"textures\")"
|
||
Svc->>SC : "GeneratePresignedPostURL(bucket, objectName, minSize, maxSize, expires, useSSL, endpoint)"
|
||
SC->>O : "PresignedPostPolicy(policy)"
|
||
O-->>SC : "postURL + formData"
|
||
SC-->>Svc : "PresignedPostPolicyResult"
|
||
Svc-->>Svc : "返回结果"
|
||
```
|
||
|
||
图表来源
|
||
- [upload_service.go](file://internal/service/upload_service.go#L78-L160)
|
||
- [minio.go](file://pkg/storage/minio.go#L82-L120)
|
||
|
||
章节来源
|
||
- [upload_service.go](file://internal/service/upload_service.go#L13-L57)
|
||
- [upload_service.go](file://internal/service/upload_service.go#L78-L115)
|
||
- [upload_service.go](file://internal/service/upload_service.go#L117-L160)
|
||
|
||
### 接口层调用（texture_handler.go）
|
||
- GenerateTextureUploadURL接口：接收请求体，调用UploadService生成预签名POST URL，返回postURL、formData与最终文件URL
|
||
- CreateTexture接口：文件上传完成后，创建材质记录到数据库
|
||
|
||
章节来源
|
||
- [texture_handler.go](file://internal/handler/texture_handler.go#L18-L83)
|
||
- [texture_handler.go](file://internal/handler/texture_handler.go#L85-L172)
|
||
|
||
### 材质服务（texture_service.go）
|
||
- CreateTexture：校验用户存在、去重校验哈希、转换材质类型、创建记录
|
||
- RecordTextureDownload：增加下载计数并记录日志
|
||
- 其他查询与权限控制方法
|
||
|
||
章节来源
|
||
- [texture_service.go](file://internal/service/texture_service.go#L12-L64)
|
||
- [texture_service.go](file://internal/service/texture_service.go#L162-L187)
|
||
|
||
## 依赖关系分析
|
||
- 配置到客户端：RustFSConfig -> NewStorage -> StorageClient
|
||
- 客户端到管理器：StorageClient -> Manager（Init/GetClient/MustGetClient）
|
||
- 业务到客户端：UploadService/TextureService -> StorageClient
|
||
- 接口到业务：TextureHandler -> UploadService -> StorageClient
|
||
|
||
```mermaid
|
||
graph LR
|
||
CFG["RustFSConfig"] --> NS["NewStorage(cfg)"]
|
||
NS --> SC["StorageClient"]
|
||
SC --> M["Manager.Init/GetClient/MustGetClient"]
|
||
M --> H["TextureHandler"]
|
||
H --> US["UploadService"]
|
||
US --> SC
|
||
TS["TextureService"] --> SC
|
||
```
|
||
|
||
图表来源
|
||
- [config.go](file://pkg/config/config.go#L58-L66)
|
||
- [minio.go](file://pkg/storage/minio.go#L20-L49)
|
||
- [manager.go](file://pkg/storage/manager.go#L18-L44)
|
||
- [upload_service.go](file://internal/service/upload_service.go#L78-L160)
|
||
- [texture_handler.go](file://internal/handler/texture_handler.go#L18-L83)
|
||
- [texture_service.go](file://internal/service/texture_service.go#L12-L64)
|
||
|
||
## 性能考虑
|
||
- 并发控制
|
||
- 上传直传采用预签名POST策略，客户端直接向对象存储发起请求，避免服务端转发带来的CPU与内存压力
|
||
- 服务端仅负责生成策略与返回表单数据，适合高并发场景
|
||
- 连接复用
|
||
- minio-go客户端内部维护连接池与HTTP复用；建议在生产环境中保持长连接，避免频繁重建
|
||
- 超时与重试
|
||
- 初始化阶段对ListBuckets设置了超时，防止阻塞启动
|
||
- 业务侧建议在调用方为上传/下载操作设置合理超时与指数退避重试
|
||
- 缓存与预热
|
||
- 对频繁使用的桶名解析与策略生成可做缓存（需注意策略过期时间）
|
||
- 资源限制
|
||
- 通过上传配置限制文件大小与扩展名，降低存储与带宽压力
|
||
|
||
[本节为通用指导，无需特定文件引用]
|
||
|
||
## 故障排查指南
|
||
- 签名错误
|
||
- 现象：表单上传返回签名错误
|
||
- 排查要点：确认formData中多余字段被移除；确保file字段位于表单末尾；检查endpoint与useSSL一致性
|
||
- 参考实现位置：[minio.go](file://pkg/storage/minio.go#L82-L120)
|
||
- 网络超时
|
||
- 现象：初始化或ListBuckets超时
|
||
- 排查要点：检查endpoint连通性、防火墙、TLS证书；适当增大超时时间
|
||
- 参考实现位置：[minio.go](file://pkg/storage/minio.go#L33-L42)
|
||
- 凭据错误
|
||
- 现象：连接测试失败或策略生成失败
|
||
- 排查要点：核对AccessKey/SecretKey；确认桶映射正确；检查对象存储端策略与权限
|
||
- 参考实现位置：[minio.go](file://pkg/storage/minio.go#L20-L49)
|
||
- 桶不存在
|
||
- 现象：GetBucket返回错误
|
||
- 排查要点：确认RustFSConfig.Buckets中包含所需桶名；核对环境变量覆盖逻辑
|
||
- 参考实现位置：[minio.go](file://pkg/storage/minio.go#L57-L64)，[config.go](file://pkg/config/config.go#L238-L253)
|
||
- 接口调用失败
|
||
- 现象：接口返回400/500
|
||
- 排查要点：检查鉴权中间件、请求体绑定、日志输出；确认MustGetClient/MustGetRustFSConfig已调用
|
||
- 参考实现位置：[texture_handler.go](file://internal/handler/texture_handler.go#L18-L83)，[manager.go](file://pkg/storage/manager.go#L37-L44)，[config.go](file://pkg/config/config.go#L190-L236)
|
||
|
||
章节来源
|
||
- [minio.go](file://pkg/storage/minio.go#L20-L49)
|
||
- [minio.go](file://pkg/storage/minio.go#L57-L64)
|
||
- [minio.go](file://pkg/storage/minio.go#L82-L120)
|
||
- [texture_handler.go](file://internal/handler/texture_handler.go#L18-L83)
|
||
- [manager.go](file://pkg/storage/manager.go#L37-L44)
|
||
- [config.go](file://pkg/config/config.go#L238-L253)
|
||
|
||
## 结论
|
||
- CarrotSkin通过StorageClient与minio-go实现了对S3兼容存储的完整封装，重点支持预签名POST直传与PUT上传
|
||
- 上传流程清晰：接口层生成策略，前端直传对象存储，服务端仅负责策略与元数据登记
|
||
- 分片上传与断点续传当前未在代码中直接体现；如需支持，可在客户端侧采用分片上传策略，并在服务端补充断点续传与合并逻辑
|
||
- 建议结合业务需求完善下载与公开/私有资源的访问策略，并持续优化并发与连接复用
|
||
|
||
[本节为总结性内容，无需特定文件引用]
|
||
|
||
## 附录
|
||
|
||
### 典型用例示例（代码示例路径）
|
||
- 皮肤/披风文件上传
|
||
- 生成预签名POST URL：[upload_service.go](file://internal/service/upload_service.go#L117-L160)
|
||
- 接口调用入口：[texture_handler.go](file://internal/handler/texture_handler.go#L18-L83)
|
||
- 私有资源临时链接生成（PUT）
|
||
- 生成预签名PUT URL：[minio.go](file://pkg/storage/minio.go#L66-L73)
|
||
- 业务侧调用：[upload_service.go](file://internal/service/upload_service.go#L78-L115)
|
||
|
||
### 高级功能支持现状
|
||
- 分片上传/断点续传
|
||
- 当前未在代码中直接实现；如需支持，可在客户端侧引入分片上传策略，并在服务端补充断点续传与合并逻辑
|
||
- 下载
|
||
- 当前未直接暴露Download方法；可通过最终访问URL或GeneratePresignedURL生成临时下载链接
|
||
|
||
[本节为概念性说明，无需特定文件引用] |