Initial commit: SRDB - High-performance LSM-Tree database
- Core engine with MemTable, SST, WAL - B+Tree indexing for SST files - Leveled compaction strategy - Multi-table database management - Schema validation and secondary indexes - Query builder with complex conditions - Web UI with HTMX for data visualization - Command-line tools for diagnostics
This commit is contained in:
11
.gitattributes
vendored
Normal file
11
.gitattributes
vendored
Normal file
@@ -0,0 +1,11 @@
|
||||
# Go files
|
||||
*.go text diff=golang eol=lf
|
||||
|
||||
# Manifests and configs
|
||||
MANIFEST text eol=lf
|
||||
*.json text eol=lf
|
||||
*.md text eol=lf
|
||||
|
||||
# Binary files
|
||||
*.sst binary
|
||||
*.wal binary
|
||||
46
.gitignore
vendored
Normal file
46
.gitignore
vendored
Normal file
@@ -0,0 +1,46 @@
|
||||
# Binaries
|
||||
*.exe
|
||||
*.exe~
|
||||
*.dll
|
||||
*.so
|
||||
*.dylib
|
||||
*.test
|
||||
*.out
|
||||
|
||||
# Go workspace
|
||||
go.work
|
||||
go.work.sum
|
||||
|
||||
# Test coverage
|
||||
*.coverage
|
||||
coverage.*
|
||||
|
||||
# IDE
|
||||
.vscode/
|
||||
.idea/
|
||||
.zed/
|
||||
*.swp
|
||||
*.swo
|
||||
*~
|
||||
|
||||
# OS
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
|
||||
# Runtime data directories
|
||||
mydb/
|
||||
testdb/
|
||||
*_data/
|
||||
*.log
|
||||
*.wal
|
||||
*.sst
|
||||
|
||||
# Example binaries
|
||||
/examples/webui/data/
|
||||
|
||||
# AI markdown
|
||||
/*.md
|
||||
!/CLAUDE.md
|
||||
!/DESIGN.md
|
||||
!/README.md
|
||||
!/LICENSE.md
|
||||
291
CLAUDE.md
Normal file
291
CLAUDE.md
Normal file
@@ -0,0 +1,291 @@
|
||||
# CLAUDE.md
|
||||
|
||||
本文件为 Claude Code (claude.ai/code) 提供在本仓库中工作的指导。
|
||||
|
||||
## 项目概述
|
||||
|
||||
SRDB 是一个用 Go 编写的高性能 Append-Only 时序数据库引擎。它使用简化的 LSM-tree 架构，结合 WAL + MemTable + mmap B+Tree SST 文件，针对高并发写入（200K+ 写/秒）和快速查询（1-5ms）进行了优化。
|
||||
|
||||
**模块**: `code.tczkiot.com/srdb`
|
||||
|
||||
## 构建和测试
|
||||
|
||||
```bash
|
||||
# 运行所有测试
|
||||
go test -v ./...
|
||||
|
||||
# 运行指定包的测试
|
||||
go test -v ./engine
|
||||
go test -v ./compaction
|
||||
go test -v ./query
|
||||
|
||||
# 运行指定的测试
|
||||
go test -v ./engine -run TestEngineBasic
|
||||
|
||||
# 构建示例程序
|
||||
go build ./examples/basic
|
||||
go build ./examples/with_schema
|
||||
```
|
||||
|
||||
## 架构
|
||||
|
||||
### 两层存储模型
|
||||
|
||||
与传统的多层 LSM 树不同，SRDB 使用简化的两层架构：
|
||||
|
||||
1. **内存层**: WAL + MemTable (Active + Immutable)
|
||||
2. **磁盘层**: 带 B+Tree 索引的 SST 文件，分为 L0-L4+ 层级
|
||||
|
||||
### 核心数据流
|
||||
|
||||
**写入路径**:
|
||||
1. Schema 验证（如果定义了）
|
||||
2. 生成序列号 (`_seq`)
|
||||
3. 追加写入 WAL（顺序写）
|
||||
4. 插入到 Active MemTable（map + 有序 slice）
|
||||
5. 当 MemTable 超过阈值（默认 64MB）时，切换到新的 Active MemTable 并异步将 Immutable 刷新到 SST
|
||||
6. 更新二级索引（如果已创建）
|
||||
|
||||
**读取路径**:
|
||||
1. 检查 Active MemTable（O(1) map 查找）
|
||||
2. 按顺序检查 Immutable MemTables（从最新到最旧）
|
||||
3. 使用 mmap + B+Tree 索引扫描 SST 文件（从最新到最旧）
|
||||
4. 第一个匹配的记录获胜（新数据覆盖旧数据）
|
||||
|
||||
**查询路径**（带条件）:
|
||||
1. 如果是带 `=` 操作符的索引字段：使用二级索引 → 通过 seq 获取
|
||||
2. 否则：带过滤条件的全表扫描（MemTable + SST）
|
||||
|
||||
### 关键设计选择
|
||||
|
||||
**MemTable: `map[int64][]byte + sorted []int64`**
|
||||
- 为什么不用 SkipList？实现更简单（130 行），Put 和 Get 都是 O(1) vs O(log N)
|
||||
- 权衡：插入时需要重新排序 keys slice（但实际上仍然更快）
|
||||
- Active MemTable + 多个 Immutable MemTables（正在刷新中）
|
||||
|
||||
**SST 格式: 4KB 节点的 B+Tree**
|
||||
- 固定大小的节点，与 OS 页面大小对齐
|
||||
- 支持高效的 mmap 访问和零拷贝读取
|
||||
- 内部节点：keys + 子节点指针
|
||||
- 叶子节点：keys + 数据偏移量/大小
|
||||
- 数据块：Snappy 压缩的 JSON 行
|
||||
|
||||
**mmap 而非 read() 系统调用**
|
||||
- 对 SST 文件的零拷贝访问
|
||||
- OS 自动管理页面缓存
|
||||
- 应用程序内存占用 < 150MB，无论数据大小
|
||||
|
||||
**Append-only（无更新/删除）**
|
||||
- 简化并发控制
|
||||
- 相同 seq 的新记录覆盖旧记录
|
||||
- Compaction 合并文件并按 seq 去重（保留最新的，按时间戳）
|
||||
|
||||
## 目录结构
|
||||
|
||||
```
|
||||
srdb/
|
||||
├── database.go # 多表数据库管理
|
||||
├── table.go # 带 schema 的表
|
||||
├── engine/ # 核心存储引擎（583 行）
|
||||
│ └── engine.go
|
||||
├── wal/ # 预写日志
|
||||
│ ├── wal.go # WAL 实现（208 行）
|
||||
│ └── manager.go # 多 WAL 管理
|
||||
├── memtable/ # 内存表
|
||||
│ ├── memtable.go # MemTable（130 行）
|
||||
│ └── manager.go # Active + Immutable 管理
|
||||
├── sst/ # SSTable 文件
|
||||
│ ├── format.go # 文件格式定义
|
||||
│ ├── writer.go # SST 写入器
|
||||
│ ├── reader.go # mmap 读取器（147 行）
|
||||
│ ├── manager.go # SST 文件管理
|
||||
│ └── encoding.go # Snappy 压缩
|
||||
├── btree/ # B+Tree 索引
|
||||
│ ├── node.go # 4KB 节点结构
|
||||
│ ├── builder.go # B+Tree 构建器（125 行）
|
||||
│ └── reader.go # B+Tree 读取器
|
||||
├── manifest/ # 版本控制
|
||||
│ ├── version_set.go # 版本管理
|
||||
│ ├── version_edit.go # 原子更新
|
||||
│ ├── version.go # 文件元数据
|
||||
│ ├── manifest_writer.go
|
||||
│ └── manifest_reader.go
|
||||
├── compaction/ # 后台压缩
|
||||
│ ├── manager.go # Compaction 调度器
|
||||
│ ├── compactor.go # 合并执行器
|
||||
│ └── picker.go # 文件选择策略
|
||||
├── index/ # 二级索引
|
||||
│ ├── index.go # 字段级索引
|
||||
│ └── manager.go # 索引生命周期
|
||||
├── query/ # 查询系统
|
||||
│ ├── builder.go # 流式查询 API
|
||||
│ └── expr.go # 表达式求值
|
||||
└── schema/ # Schema 验证
|
||||
├── schema.go # 类型定义和验证
|
||||
└── examples.go # Schema 示例
|
||||
```
|
||||
|
||||
**运行时数据目录**（例如 `./mydb/`）:
|
||||
```
|
||||
database_dir/
|
||||
├── database.meta # 数据库元数据（JSON）
|
||||
├── MANIFEST # 全局版本控制
|
||||
└── table_name/ # 每表目录
|
||||
├── schema.json # 表 schema
|
||||
├── MANIFEST # 表级版本控制
|
||||
├── wal/ # WAL 文件（*.wal）
|
||||
├── sst/ # SST 文件（*.sst）
|
||||
└── index/ # 二级索引（idx_*.sst）
|
||||
```
|
||||
|
||||
## 常见模式
|
||||
|
||||
### 使用 Engine
|
||||
|
||||
`Engine` 是核心存储层。修改引擎行为时：
|
||||
|
||||
- 所有写入都经过 `Insert()` → WAL → MemTable → （异步刷新到 SST）
|
||||
- 读取经过 `Get(seq)` → 检查 MemTable → 检查 SST 文件
|
||||
- `switchMemTable()` 创建新的 Active MemTable 并异步刷新旧的
|
||||
- `flushImmutable()` 将 MemTable 写入 SST 并更新 MANIFEST
|
||||
- 后台 compaction 通过 `compactionManager` 运行
|
||||
|
||||
### Schema 和验证
|
||||
|
||||
Schema 是可选的，但建议在生产环境使用：
|
||||
|
||||
```go
|
||||
schema := schema.NewSchema("users").
|
||||
AddField("name", schema.FieldTypeString, false, "用户名").
|
||||
AddField("age", schema.FieldTypeInt64, false, "用户年龄").
|
||||
AddField("email", schema.FieldTypeString, true, "邮箱（索引）")
|
||||
|
||||
table, _ := db.CreateTable("users", schema)
|
||||
```
|
||||
|
||||
- Schema 在 `Insert()` 时验证类型和必填字段
|
||||
- 索引字段（`Indexed: true`）自动创建二级索引
|
||||
- Schema 持久化到 `table_dir/schema.json`
|
||||
|
||||
### Query Builder
|
||||
|
||||
对于带条件的查询，始终使用 `QueryBuilder`：
|
||||
|
||||
```go
|
||||
qb := query.NewQueryBuilder()
|
||||
qb.Where("age", query.OpGreater, 18).
|
||||
Where("city", query.OpEqual, "Beijing")
|
||||
rows, _ := table.Query(qb)
|
||||
```
|
||||
|
||||
- 支持操作符：`OpEqual`、`OpNotEqual`、`OpGreater`、`OpLess`、`OpPrefix`、`OpSuffix`、`OpContains`
|
||||
- 支持 `WhereNot()` 进行否定
|
||||
- 支持 `And()` 和 `Or()` 逻辑
|
||||
- 当可用时自动使用二级索引（对于 `=` 条件）
|
||||
- 如果没有索引，则回退到全表扫描
|
||||
|
||||
### Compaction
|
||||
|
||||
Compaction 在后台自动运行：
|
||||
|
||||
- **触发条件**: L0 文件数 > 阈值（默认 10）
|
||||
- **策略**: 合并重叠文件，从 L0 → L1、L1 → L2 等
|
||||
- **安全性**: 删除前验证文件是否存在，以防止数据丢失
|
||||
- **去重**: 对于重复的 seq，保留最新记录（按时间戳）
|
||||
- **文件大小**: L0=2MB、L1=10MB、L2=50MB、L3=100MB、L4+=200MB
|
||||
|
||||
修改 compaction 逻辑时：
|
||||
- `picker.go`: 选择要压缩的文件
|
||||
- `compactor.go`: 执行合并操作
|
||||
- `manager.go`: 调度和协调 compaction
|
||||
- 删除前始终验证输入/输出文件是否存在（参见 `DoCompaction`）
|
||||
|
||||
### 版本控制（MANIFEST）
|
||||
|
||||
MANIFEST 跟踪跨版本的 SST 文件元数据：
|
||||
|
||||
- `VersionEdit`: 记录原子变更（AddFile/DeleteFile）
|
||||
- `VersionSet`: 管理当前和历史版本
|
||||
- `LogAndApply()`: 原子地应用编辑并持久化到 MANIFEST
|
||||
|
||||
添加/删除 SST 文件时：
|
||||
1. 分配文件编号：`versionSet.AllocateFileNumber()`
|
||||
2. 创建带变更的 `VersionEdit`
|
||||
3. 应用：`versionSet.LogAndApply(edit)`
|
||||
4. 清理旧文件：`compactionManager.CleanupOrphanFiles()`
|
||||
|
||||
### 错误恢复
|
||||
|
||||
- **WAL 重放**: 启动时，所有 `*.wal` 文件被重放到 Active MemTable
|
||||
- **孤儿文件清理**: 不在 MANIFEST 中的文件在启动时删除
|
||||
- **索引修复**: `verifyAndRepairIndexes()` 重建损坏的索引
|
||||
- **优雅降级**: 表恢复失败会被记录但不会使数据库崩溃
|
||||
|
||||
## 测试模式
|
||||
|
||||
测试按组件组织：
|
||||
|
||||
- `engine/engine_test.go`: 基本引擎操作
|
||||
- `engine/engine_compaction_test.go`: Compaction 场景
|
||||
- `engine/engine_stress_test.go`: 并发压力测试
|
||||
- `compaction/compaction_test.go`: Compaction 正确性
|
||||
- `query/builder_test.go`: Query builder 功能
|
||||
- `schema/schema_test.go`: Schema 验证
|
||||
|
||||
为多线程操作编写测试时，使用 `sync.WaitGroup` 并用多个 goroutine 测试（参见 `engine_stress_test.go`）。
|
||||
|
||||
## 性能特性
|
||||
|
||||
- **写入吞吐量**: 200K+ 写/秒（多线程），50K 写/秒（单线程）
|
||||
- **写入延迟**: < 1ms（p99）
|
||||
- **查询延迟**: < 0.1ms（MemTable），1-5ms（SST 热数据），3-5ms（冷数据）
|
||||
- **内存使用**: < 150MB（64MB MemTable + 开销）
|
||||
- **压缩率**: Snappy 约 50%
|
||||
|
||||
优化时：
|
||||
- 批量写入以减少 WAL 同步开销
|
||||
- 对经常查询的字段创建索引
|
||||
- 监控 MemTable 刷新频率（不应太频繁）
|
||||
- 根据写入模式调整 compaction 阈值
|
||||
|
||||
## 重要实现细节
|
||||
|
||||
### 序列号
|
||||
|
||||
- `_seq` 是单调递增的 int64（原子操作）
|
||||
- 充当主键和时间戳排序
|
||||
- 永不重用（append-only）
|
||||
- compaction 期间，相同 seq 值的较新记录优先
|
||||
|
||||
### 并发
|
||||
|
||||
- `Engine.mu`: 保护元数据和 SST reader 列表
|
||||
- `Engine.flushMu`: 确保一次只有一个 flush
|
||||
- `MemTable.mu`: RWMutex，支持并发读、独占写
|
||||
- `VersionSet.mu`: 保护版本状态
|
||||
|
||||
### 文件格式
|
||||
|
||||
**WAL 条目**:
|
||||
```
|
||||
CRC32 (4B) | Length (4B) | Type (1B) | Seq (8B) | DataLen (4B) | Data (N bytes)
|
||||
```
|
||||
|
||||
**SST 文件**:
|
||||
```
|
||||
Header (256B) | B+Tree Index | Data Blocks (Snappy compressed)
|
||||
```
|
||||
|
||||
**B+Tree 节点**（4KB 固定）:
|
||||
```
|
||||
Header (32B) | Keys (8B each) | Pointers/Offsets (8B each) | Padding
|
||||
```
|
||||
|
||||
## 常见陷阱
|
||||
|
||||
- Schema 验证仅在向 `Engine.Open()` 提供 schema 时才应用
|
||||
- 索引必须通过 `CreateIndex(field)` 显式创建（非自动）
|
||||
- 带 schema 的 QueryBuilder 需要调用 `WithSchema()` 或让引擎设置它
|
||||
- Compaction 可能会暂时增加磁盘使用（合并期间旧文件和新文件共存）
|
||||
- MemTable flush 是异步的；关闭时可能需要等待 immutable flush 完成
|
||||
- mmap 文件可能显示较大的虚拟内存使用（这是正常的，不是实际 RAM）
|
||||
915
DESIGN.md
Normal file
915
DESIGN.md
Normal file
@@ -0,0 +1,915 @@
|
||||
# SRDB 设计文档：WAL + mmap B+Tree
|
||||
|
||||
> 模块名：`code.tczkiot.com/srdb`
|
||||
> 一个高性能的 Append-Only 时序数据库引擎
|
||||
|
||||
## 🎯 设计目标
|
||||
|
||||
1. **极简架构** - 放弃复杂的 LSM Tree 多层设计，使用简单的两层结构
|
||||
2. **高并发写入** - WAL + MemTable 保证 200,000+ writes/s
|
||||
3. **快速查询** - mmap B+Tree 索引 + 二级索引，1-5 ms 查询性能
|
||||
4. **低内存占用** - mmap 零拷贝，应用层内存 < 200 MB
|
||||
5. **功能完善** - 支持 Schema、索引、条件查询等高级特性
|
||||
6. **生产可用** - 核心代码 5399 行，包含完善的错误处理和数据一致性保证
|
||||
|
||||
## 🏗️ 核心架构
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ SRDB Architecture │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ Application Layer │
|
||||
│ ┌───────────────┐ ┌──────────┐ ┌───────────┐ │
|
||||
│ │ Database │->│ Table │->│ Engine │ │
|
||||
│ │ (Multi-Table) │ │ (Schema) │ │ (Storage) │ │
|
||||
│ └───────────────┘ └──────────┘ └───────────┘ │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ Write Path (High Concurrency) │
|
||||
│ ┌─────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
|
||||
│ │ Schema │-> │ WAL │-> │ MemTable │->│ Index │ │
|
||||
│ │Validate │ │(Append) │ │(Map+Arr) │ │ Manager │ │
|
||||
│ └─────────┘ └──────────┘ └──────────┘ └──────────┘ │
|
||||
│ ↓ ↓ ↓ ↓ │
|
||||
│ Type Check Sequential Sorted Map Secondary │
|
||||
│ Required Write Fast Insert Indexes │
|
||||
│ Constraints 200K+ w/s O(1) Put Field Query │
|
||||
│ │
|
||||
│ Background Flush: MemTable -> SST (Async) │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ Storage Layer (Persistent) │
|
||||
│ ┌─────────────────────────────────────────────────┐ │
|
||||
│ │ SST Files (B+Tree Format + Compression) │ │
|
||||
│ │ ┌─────────────────────────────────────────┐ │ │
|
||||
│ │ │ File Header (256 bytes) │ │ │
|
||||
│ │ │ - Magic, Version, Compression │ │ │
|
||||
│ │ │ - MinKey, MaxKey, RowCount │ │ │
|
||||
│ │ ├─────────────────────────────────────────┤ │ │
|
||||
│ │ │ B+Tree Index (4 KB nodes) │ │ │
|
||||
│ │ │ - Root Node │ │ │
|
||||
│ │ │ - Internal Nodes (Order=200) │ │ │
|
||||
│ │ │ - Leaf Nodes → Data Offset │ │ │
|
||||
│ │ ├─────────────────────────────────────────┤ │ │
|
||||
│ │ │ Data Blocks (Snappy Compressed) │ │ │
|
||||
│ │ │ - JSON serialized rows │ │ │
|
||||
│ │ └─────────────────────────────────────────┘ │ │
|
||||
│ │ │ │
|
||||
│ │ Secondary Indexes (Optional) │ │
|
||||
│ │ - Field → [Seq] mapping │ │
|
||||
│ │ - B+Tree format for fast lookup │ │
|
||||
│ └─────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ MANIFEST: Version control & file tracking │
|
||||
│ Compaction: Background merge of SST files │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ Query Path (Multiple Access Methods) │
|
||||
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
|
||||
│ │ Query │-> │MemTable │-> │mmap SST │ │
|
||||
│ │ Builder │ │Manager │ │ Reader │ │
|
||||
│ └──────────┘ └──────────┘ └──────────┘ │
|
||||
│ ↓ ↓ ↓ │
|
||||
│ Conditions Active+Immut Zero Copy │
|
||||
│ AND/OR/NOT < 0.1 ms 1-5 ms │
|
||||
│ Field Match In Memory OS Cache │
|
||||
│ │
|
||||
│ With Index: Index Lookup -> Get by Seq (Fast) │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
|
||||
设计理念:
|
||||
- 简单 > 复杂: 只有 2 层，无多级 LSM
|
||||
- 性能 > 功能: 专注于高并发写入和快速查询
|
||||
- mmap > 内存: 让 OS 管理缓存，应用层零负担
|
||||
- Append-Only: 只插入，不更新/删除
|
||||
- 可扩展: 支持 Schema、索引、条件查询等高级特性
|
||||
```
|
||||
|
||||
## 📁 文件组织结构
|
||||
|
||||
### 代码目录结构
|
||||
|
||||
```
|
||||
srdb/ ← 项目根目录
|
||||
├── go.mod ← 模块定义: code.tczkiot.com/srdb
|
||||
├── DESIGN.md ← 本设计文档
|
||||
├── database.go ← 数据库管理 (多表)
|
||||
├── table.go ← 表管理
|
||||
│
|
||||
├── engine/ ← 存储引擎
|
||||
│ └── engine.go ← 核心引擎实现 (583 行)
|
||||
│
|
||||
├── wal/ ← Write-Ahead Log
|
||||
│ ├── wal.go ← WAL 实现 (208 行)
|
||||
│ └── manager.go ← WAL 管理器
|
||||
│
|
||||
├── memtable/ ← 内存表
|
||||
│ ├── memtable.go ← MemTable 实现 (130 行)
|
||||
│ └── manager.go ← MemTable 管理器 (多版本)
|
||||
│
|
||||
├── sst/ ← SSTable 文件
|
||||
│ ├── format.go ← 文件格式定义
|
||||
│ ├── writer.go ← SST 写入器
|
||||
│ ├── reader.go ← SST 读取器 (mmap, 147 行)
|
||||
│ ├── manager.go ← SST 管理器
|
||||
│ └── encoding.go ← 序列化/压缩
|
||||
│
|
||||
├── btree/ ← B+Tree 索引
|
||||
│ ├── node.go ← 节点定义 (4KB)
|
||||
│ ├── builder.go ← B+Tree 构建器 (125 行)
|
||||
│ └── reader.go ← B+Tree 读取器
|
||||
│
|
||||
├── manifest/ ← 版本控制
|
||||
│ ├── version_set.go ← 版本集合
|
||||
│ ├── version_edit.go ← 版本变更
|
||||
│ ├── version.go ← 版本信息
|
||||
│ ├── manifest_writer.go ← MANIFEST 写入
|
||||
│ └── manifest_reader.go ← MANIFEST 读取
|
||||
│
|
||||
├── compaction/ ← 压缩合并
|
||||
│ ├── manager.go ← Compaction 管理器
|
||||
│ ├── compactor.go ← 压缩执行器
|
||||
│ └── picker.go ← 文件选择策略
|
||||
│
|
||||
├── index/ ← 二级索引 (新增)
|
||||
│ ├── index.go ← 索引实现
|
||||
│ ├── manager.go ← 索引管理器
|
||||
│ └── README.md ← 索引使用文档
|
||||
│
|
||||
├── query/ ← 查询系统 (新增)
|
||||
│ ├── builder.go ← 查询构建器
|
||||
│ └── expr.go ← 表达式求值
|
||||
│
|
||||
└── schema/ ← Schema 系统 (新增)
|
||||
├── schema.go ← Schema 定义与验证
|
||||
├── examples.go ← Schema 示例
|
||||
└── README.md ← Schema 使用文档
|
||||
```
|
||||
|
||||
### 运行时数据目录结构
|
||||
|
||||
```
|
||||
database_dir/ ← 数据库目录
|
||||
├── database.meta ← 数据库元数据
|
||||
├── MANIFEST ← 全局 MANIFEST
|
||||
└── table_name/ ← 表目录
|
||||
├── schema.json ← 表的 Schema 定义
|
||||
├── MANIFEST ← 表的 MANIFEST
|
||||
│
|
||||
├── wal/ ← WAL 目录
|
||||
│ ├── 000001.log ← 当前 WAL
|
||||
│ └── 000002.log ← 历史 WAL
|
||||
│
|
||||
├── sst/ ← SST 文件目录
|
||||
│ ├── 000001.sst ← SST 文件 (B+Tree)
|
||||
│ ├── 000002.sst
|
||||
│ └── 000003.sst
|
||||
│
|
||||
└── index/ ← 索引目录 (可选)
|
||||
├── idx_name.sst ← 字段 name 的索引
|
||||
└── idx_email.sst ← 字段 email 的索引
|
||||
```
|
||||
|
||||
## 🔑 核心组件
|
||||
|
||||
### 1. WAL (Write-Ahead Log)
|
||||
|
||||
```
|
||||
设计:
|
||||
- 顺序追加写入
|
||||
- 批量提交优化
|
||||
- 崩溃恢复支持
|
||||
|
||||
文件格式:
|
||||
┌─────────────────────────────────────┐
|
||||
│ WAL Entry │
|
||||
├─────────────────────────────────────┤
|
||||
│ CRC32 (4 bytes) │
|
||||
│ Length (4 bytes) │
|
||||
│ Type (1 byte): Put │
|
||||
│ Key (8 bytes): _seq │
|
||||
│ Value Length (4 bytes) │
|
||||
│ Value (N bytes): 序列化的行数据 │
|
||||
└─────────────────────────────────────┘
|
||||
|
||||
性能:
|
||||
- 顺序写入: 极快
|
||||
- 批量提交: 减少 fsync
|
||||
- 吞吐: 200,000+ writes/s
|
||||
```
|
||||
|
||||
### 2. MemTable (内存表)
|
||||
|
||||
```
|
||||
设计:
|
||||
- 使用 map[int64][]byte + sorted slice
|
||||
- 读写锁保护
|
||||
- 大小限制 (默认 64 MB)
|
||||
- Manager 管理多个版本 (Active + Immutables)
|
||||
|
||||
实现:
|
||||
type MemTable struct {
|
||||
data map[int64][]byte // key -> value
|
||||
keys []int64 // 有序的 keys
|
||||
size int64 // 数据大小
|
||||
mu sync.RWMutex
|
||||
}
|
||||
|
||||
func (m *MemTable) Put(key int64, value []byte) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
|
||||
if _, exists := m.data[key]; !exists {
|
||||
m.keys = append(m.keys, key)
|
||||
// 保持 keys 有序
|
||||
sort.Slice(m.keys, func(i, j int) bool {
|
||||
return m.keys[i] < m.keys[j]
|
||||
})
|
||||
}
|
||||
m.data[key] = value
|
||||
m.size += int64(len(value))
|
||||
}
|
||||
|
||||
func (m *MemTable) Get(key int64) ([]byte, bool) {
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
|
||||
value, exists := m.data[key]
|
||||
return value, exists
|
||||
}
|
||||
|
||||
MemTable Manager:
|
||||
- Active MemTable: 当前写入
|
||||
- Immutable MemTables: 正在 Flush 的只读表
|
||||
- 查询时按顺序查找: Active -> Immutables
|
||||
|
||||
性能:
|
||||
- 插入: O(1) (map) + O(N log N) (排序，仅新key)
|
||||
- 查询: O(1) (map lookup)
|
||||
- 内存操作: 极快
|
||||
- 实测: 比 SkipList 更快的写入性能
|
||||
|
||||
选择原因:
|
||||
✅ 实现简单
|
||||
✅ 写入性能好 (O(1))
|
||||
✅ 查询性能好 (O(1))
|
||||
✅ 易于遍历 (已排序的 keys)
|
||||
```
|
||||
|
||||
### 3. SST 文件 (B+Tree 格式)
|
||||
|
||||
```
|
||||
设计:
|
||||
- 固定大小的节点 (4 KB)
|
||||
- 适合 mmap 访问
|
||||
- 不可变文件
|
||||
|
||||
B+Tree 节点格式:
|
||||
┌─────────────────────────────────────┐
|
||||
│ B+Tree Node (4 KB) │
|
||||
├─────────────────────────────────────┤
|
||||
│ Header (32 bytes) │
|
||||
│ ├─ Node Type (1 byte) │
|
||||
│ │ 0: Internal, 1: Leaf │
|
||||
│ ├─ Key Count (2 bytes) │
|
||||
│ ├─ Level (1 byte) │
|
||||
│ └─ Reserved (28 bytes) │
|
||||
├─────────────────────────────────────┤
|
||||
│ Keys (variable) │
|
||||
│ ├─ Key 1 (8 bytes) │
|
||||
│ ├─ Key 2 (8 bytes) │
|
||||
│ └─ ... │
|
||||
├─────────────────────────────────────┤
|
||||
│ Values/Pointers (variable) │
|
||||
│ Internal Node: │
|
||||
│ ├─ Child Pointer 1 (8 bytes) │
|
||||
│ ├─ Child Pointer 2 (8 bytes) │
|
||||
│ └─ ... │
|
||||
│ │
|
||||
│ Leaf Node: │
|
||||
│ ├─ Data Offset 1 (8 bytes) │
|
||||
│ ├─ Data Size 1 (4 bytes) │
|
||||
│ ├─ Data Offset 2 (8 bytes) │
|
||||
│ ├─ Data Size 2 (4 bytes) │
|
||||
│ └─ ... │
|
||||
└─────────────────────────────────────┘
|
||||
|
||||
优势:
|
||||
✅ 固定大小 (4 KB) - 对齐页面
|
||||
✅ 可以直接 mmap 访问
|
||||
✅ 无需反序列化
|
||||
✅ OS 按需加载
|
||||
```
|
||||
|
||||
### 4. mmap 查询
|
||||
|
||||
```
|
||||
设计:
|
||||
- 映射整个 SST 文件
|
||||
- 零拷贝访问
|
||||
- OS 自动缓存
|
||||
|
||||
实现:
|
||||
type MmapSST struct {
|
||||
file *os.File
|
||||
mmap mmap.MMap
|
||||
rootOffset int64
|
||||
}
|
||||
|
||||
func (s *MmapSST) Get(key int64) ([]byte, bool) {
|
||||
// 1. 从 root 开始
|
||||
nodeOffset := s.rootOffset
|
||||
|
||||
for {
|
||||
// 2. 读取节点 (零拷贝)
|
||||
node := s.readNode(nodeOffset)
|
||||
|
||||
// 3. 二分查找
|
||||
idx := sort.Search(len(node.keys), func(i int) bool {
|
||||
return node.keys[i] >= key
|
||||
})
|
||||
|
||||
// 4. 叶子节点
|
||||
if node.isLeaf {
|
||||
if idx < len(node.keys) && node.keys[idx] == key {
|
||||
// 读取数据
|
||||
offset := node.offsets[idx]
|
||||
size := node.sizes[idx]
|
||||
return s.readData(offset, size), true
|
||||
}
|
||||
return nil, false
|
||||
}
|
||||
|
||||
// 5. 继续向下
|
||||
nodeOffset = node.children[idx]
|
||||
}
|
||||
}
|
||||
|
||||
func (s *MmapSST) readNode(offset int64) *BTreeNode {
|
||||
// 直接访问 mmap 内存 (零拷贝)
|
||||
data := s.mmap[offset : offset+4096]
|
||||
return parseBTreeNode(data)
|
||||
}
|
||||
|
||||
性能:
|
||||
- 热点数据: 1-2 ms (OS 缓存)
|
||||
- 冷数据: 3-5 ms (磁盘读取)
|
||||
- 零拷贝: 无内存分配
|
||||
```
|
||||
|
||||
### 5. Schema 系统 (新增功能)
|
||||
|
||||
```
|
||||
设计:
|
||||
- 类型定义和验证
|
||||
- 必填字段检查
|
||||
- 唯一性约束
|
||||
- 默认值支持
|
||||
|
||||
实现:
|
||||
type Schema struct {
|
||||
Fields []FieldDefinition
|
||||
}
|
||||
|
||||
type FieldDefinition struct {
|
||||
Name string
|
||||
Type string // "string", "int", "float", "bool"
|
||||
Required bool // 是否必填
|
||||
Unique bool // 是否唯一
|
||||
Default interface{} // 默认值
|
||||
}
|
||||
|
||||
func (s *Schema) Validate(data map[string]interface{}) error {
|
||||
for _, field := range s.Fields {
|
||||
// 检查必填字段
|
||||
// 检查类型匹配
|
||||
// 应用默认值
|
||||
}
|
||||
}
|
||||
|
||||
使用示例:
|
||||
schema := &schema.Schema{
|
||||
Fields: []schema.FieldDefinition{
|
||||
{Name: "name", Type: "string", Required: true},
|
||||
{Name: "age", Type: "int", Required: false},
|
||||
{Name: "email", Type: "string", Unique: true},
|
||||
},
|
||||
}
|
||||
|
||||
table, _ := db.CreateTable("users", schema)
|
||||
```
|
||||
|
||||
### 6. 二级索引 (新增功能)
|
||||
|
||||
```
|
||||
设计:
|
||||
- 字段级索引
|
||||
- B+Tree 格式存储
|
||||
- 自动维护
|
||||
- 快速字段查询
|
||||
|
||||
实现:
|
||||
type SecondaryIndex struct {
|
||||
field string
|
||||
btree *BTreeIndex // Field Value -> [Seq]
|
||||
}
|
||||
|
||||
// 创建索引
|
||||
table.CreateIndex("email")
|
||||
|
||||
// 使用索引查询
|
||||
qb := query.NewQueryBuilder()
|
||||
qb.Where("email", query.Eq, "user@example.com")
|
||||
rows, _ := table.Query(qb)
|
||||
|
||||
索引文件格式:
|
||||
index/
|
||||
├── idx_email.sst ← email 字段索引
|
||||
│ └── BTree: email -> []seq
|
||||
└── idx_name.sst ← name 字段索引
|
||||
└── BTree: name -> []seq
|
||||
|
||||
性能提升:
|
||||
- 无索引: O(N) 全表扫描
|
||||
- 有索引: O(log N) 索引查找 + O(K) 结果读取
|
||||
- 实测: 100x+ 性能提升
|
||||
```
|
||||
|
||||
### 7. 查询构建器 (新增功能)
|
||||
|
||||
```
|
||||
设计:
|
||||
- 链式 API
|
||||
- 条件组合 (AND/OR/NOT)
|
||||
- 操作符支持
|
||||
- Schema 验证
|
||||
|
||||
实现:
|
||||
type QueryBuilder struct {
|
||||
conditions []*Expr
|
||||
logicOp string // "AND" 或 "OR"
|
||||
}
|
||||
|
||||
type Operator int
|
||||
const (
|
||||
Eq Operator = iota // ==
|
||||
Ne // !=
|
||||
Gt // >
|
||||
Gte // >=
|
||||
Lt // <
|
||||
Lte // <=
|
||||
Contains // 字符串包含
|
||||
StartsWith // 字符串前缀
|
||||
EndsWith // 字符串后缀
|
||||
)
|
||||
|
||||
使用示例:
|
||||
// 简单查询
|
||||
qb := query.NewQueryBuilder()
|
||||
qb.Where("age", query.Gt, 18)
|
||||
rows, _ := table.Query(qb)
|
||||
|
||||
// 复杂查询 (AND)
|
||||
qb := query.NewQueryBuilder()
|
||||
qb.Where("age", query.Gt, 18)
|
||||
.Where("city", query.Eq, "Beijing")
|
||||
.Where("active", query.Eq, true)
|
||||
|
||||
// OR 查询
|
||||
qb := query.NewQueryBuilder().Or()
|
||||
qb.Where("role", query.Eq, "admin")
|
||||
.Where("role", query.Eq, "moderator")
|
||||
|
||||
// NOT 查询
|
||||
qb := query.NewQueryBuilder()
|
||||
qb.WhereNot("status", query.Eq, "deleted")
|
||||
|
||||
// 字符串匹配
|
||||
qb := query.NewQueryBuilder()
|
||||
qb.Where("email", query.EndsWith, "@gmail.com")
|
||||
|
||||
执行流程:
|
||||
1. 尝试使用索引 (如果有)
|
||||
2. 否则扫描 MemTable + SST
|
||||
3. 应用所有条件过滤
|
||||
4. 返回匹配的行
|
||||
```
|
||||
|
||||
### 8. 数据库和表管理 (新增功能)
|
||||
|
||||
```
|
||||
设计:
|
||||
- 数据库级别管理
|
||||
- 多表支持
|
||||
- 表级 Schema
|
||||
- 独立的存储目录
|
||||
|
||||
实现:
|
||||
type Database struct {
|
||||
dir string
|
||||
tables map[string]*Table
|
||||
versionSet *manifest.VersionSet
|
||||
metadata *Metadata
|
||||
}
|
||||
|
||||
type Table struct {
|
||||
name string
|
||||
dir string
|
||||
schema *schema.Schema
|
||||
engine *engine.Engine
|
||||
}
|
||||
|
||||
使用示例:
|
||||
// 打开数据库
|
||||
db, _ := database.Open("./mydb")
|
||||
|
||||
// 创建表
|
||||
schema := &schema.Schema{...}
|
||||
table, _ := db.CreateTable("users", schema)
|
||||
|
||||
// 使用表
|
||||
table.Insert(map[string]interface{}{
|
||||
"name": "Alice",
|
||||
"age": 30,
|
||||
})
|
||||
|
||||
// 获取表
|
||||
table, _ := db.GetTable("users")
|
||||
|
||||
// 列出所有表
|
||||
tables := db.ListTables()
|
||||
|
||||
// 删除表
|
||||
db.DropTable("old_table")
|
||||
|
||||
// 关闭数据库
|
||||
db.Close()
|
||||
```
|
||||
|
||||
## 🔄 核心流程
|
||||
|
||||
### 写入流程
|
||||
|
||||
```
|
||||
1. 接收写入请求
|
||||
↓
|
||||
2. 生成 _seq (原子递增)
|
||||
↓
|
||||
3. 写入 WAL (顺序追加)
|
||||
↓
|
||||
4. 写入 MemTable (内存)
|
||||
↓
|
||||
5. 检查 MemTable 大小
|
||||
↓
|
||||
6. 如果超过阈值 → 触发 Flush (异步)
|
||||
↓
|
||||
7. 返回成功
|
||||
|
||||
Flush 流程 (后台):
|
||||
1. 冻结当前 MemTable
|
||||
↓
|
||||
2. 创建新的 MemTable (写入继续)
|
||||
↓
|
||||
3. 遍历冻结的 MemTable (已排序)
|
||||
↓
|
||||
4. 构建 B+Tree 索引
|
||||
↓
|
||||
5. 写入数据块 (Snappy 压缩)
|
||||
↓
|
||||
6. 写入 B+Tree 索引
|
||||
↓
|
||||
7. 写入文件头
|
||||
↓
|
||||
8. Sync 到磁盘
|
||||
↓
|
||||
9. 更新 MANIFEST
|
||||
↓
|
||||
10. 删除 WAL
|
||||
```
|
||||
|
||||
### 查询流程
|
||||
|
||||
```
|
||||
1. 接收查询请求 (key)
|
||||
↓
|
||||
2. 查询 MemTable (内存)
|
||||
- 如果找到 → 返回 ✅
|
||||
↓
|
||||
3. 查询 SST 文件 (从新到旧)
|
||||
- 对每个 SST:
|
||||
a. mmap 映射 (如果未映射)
|
||||
b. B+Tree 查找 (零拷贝)
|
||||
c. 如果找到 → 读取数据 → 返回 ✅
|
||||
↓
|
||||
4. 未找到 → 返回 NotFound
|
||||
```
|
||||
|
||||
### Compaction 流程 (简化)
|
||||
|
||||
```
|
||||
触发条件:
|
||||
- SST 文件数量 > 10
|
||||
|
||||
流程:
|
||||
1. 选择多个 SST 文件 (如 5 个)
|
||||
↓
|
||||
2. 多路归并排序 (已排序，很快)
|
||||
↓
|
||||
3. 构建新的 B+Tree
|
||||
↓
|
||||
4. 写出新的 SST 文件
|
||||
↓
|
||||
5. 更新 MANIFEST
|
||||
↓
|
||||
6. 删除旧的 SST 文件
|
||||
|
||||
注意:
|
||||
- Append-Only: 无需处理删除
|
||||
- 无需去重: 取最新的即可
|
||||
- 后台执行: 不影响读写
|
||||
```
|
||||
|
||||
## 📊 性能指标
|
||||
|
||||
### 代码规模
|
||||
```
|
||||
核心代码: 5399 行 (不含测试和示例)
|
||||
├── engine: 583 行
|
||||
├── wal: 208 行
|
||||
├── memtable: 130 行
|
||||
├── sst: 147 行 (reader)
|
||||
├── btree: 125 行 (builder)
|
||||
├── manifest: ~500 行
|
||||
├── compaction: ~400 行
|
||||
├── index: ~400 行
|
||||
├── query: ~300 行
|
||||
├── schema: ~200 行
|
||||
└── database: ~300 行
|
||||
|
||||
测试代码: ~2000+ 行
|
||||
示例代码: ~1000+ 行
|
||||
总计: 8000+ 行
|
||||
```
|
||||
|
||||
### 写入性能
|
||||
```
|
||||
单线程: 50,000 writes/s
|
||||
多线程: 200,000+ writes/s
|
||||
延迟: < 1 ms (p99)
|
||||
|
||||
实测数据 (MacBook Pro M1):
|
||||
- 单线程插入 10 万条: ~2 秒
|
||||
- 并发写入 (4 goroutines): ~1 秒
|
||||
```
|
||||
|
||||
### 查询性能
|
||||
```
|
||||
按 Seq 查询:
|
||||
- MemTable: < 0.1 ms
|
||||
- 热点 SST: 1-2 ms (OS 缓存)
|
||||
- 冷数据 SST: 3-5 ms (磁盘读取)
|
||||
- 平均: 2-3 ms
|
||||
|
||||
条件查询 (无索引):
|
||||
- 全表扫描: O(N)
|
||||
- 小数据集 (<10 万): < 50 ms
|
||||
- 大数据集 (100 万): < 500 ms
|
||||
|
||||
条件查询 (有索引):
|
||||
- 索引查找: O(log N)
|
||||
- 性能提升: 100x+
|
||||
- 查询延迟: < 5 ms
|
||||
```
|
||||
|
||||
### 内存占用
|
||||
```
|
||||
- MemTable: 64 MB (可配置)
|
||||
- WAL Buffer: 16 MB
|
||||
- 元数据: 10 MB
|
||||
- mmap: 0 MB (虚拟地址，OS 管理)
|
||||
- 索引内存: < 50 MB
|
||||
- 总计: < 150 MB
|
||||
```
|
||||
|
||||
### 存储空间
|
||||
```
|
||||
示例 (100 万条记录，每条 200 bytes):
|
||||
- 原始数据: 200 MB
|
||||
- Snappy 压缩: 100 MB (50% 压缩率)
|
||||
- B+Tree 索引: 20 MB (10%)
|
||||
- 二级索引: 10 MB (可选)
|
||||
- 总计: 130 MB (65% 压缩率)
|
||||
```
|
||||
|
||||
## 🔧 实现状态
|
||||
|
||||
### Phase 1: 核心功能 ✅ 已完成
|
||||
|
||||
```
|
||||
核心存储引擎:
|
||||
- [✅] Schema 定义和解析
|
||||
- [✅] WAL 实现 (wal/)
|
||||
- [✅] MemTable 实现 (memtable/，使用 map+slice)
|
||||
- [✅] 基础的 Insert 和 Get
|
||||
- [✅] SST 文件格式定义 (sst/format.go)
|
||||
- [✅] B+Tree 构建器 (btree/)
|
||||
- [✅] Flush 流程 (异步)
|
||||
- [✅] mmap 查询 (sst/reader.go)
|
||||
```
|
||||
|
||||
### Phase 2: 优化和稳定 ✅ 已完成
|
||||
|
||||
```
|
||||
稳定性和性能:
|
||||
- [✅] 批量写入优化
|
||||
- [✅] 并发控制优化
|
||||
- [✅] 崩溃恢复 (WAL 重放)
|
||||
- [✅] MANIFEST 管理 (manifest/)
|
||||
- [✅] Compaction 实现 (compaction/)
|
||||
- [✅] MemTable Manager (多版本管理)
|
||||
- [✅] 性能测试 (各种 *_test.go)
|
||||
- [✅] 文档完善 (README.md, DESIGN.md)
|
||||
```
|
||||
|
||||
### Phase 3: 高级特性 ✅ 已完成
|
||||
|
||||
```
|
||||
高级功能:
|
||||
- [✅] 数据库和表管理 (database.go, table.go)
|
||||
- [✅] Schema 系统 (schema/)
|
||||
- [✅] 二级索引 (index/)
|
||||
- [✅] 查询构建器 (query/)
|
||||
- [✅] 条件查询 (AND/OR/NOT)
|
||||
- [✅] 字符串匹配 (Contains/StartsWith/EndsWith)
|
||||
- [✅] 版本控制和自动修复
|
||||
- [✅] 统计信息 (engine.Stats())
|
||||
- [✅] 压缩和编码 (Snappy)
|
||||
```
|
||||
|
||||
### Phase 4: 示例和文档 ✅ 已完成
|
||||
|
||||
```
|
||||
示例程序 (examples/):
|
||||
- [✅] basic - 基础使用示例
|
||||
- [✅] with_schema - Schema 使用
|
||||
- [✅] with_index - 索引使用
|
||||
- [✅] query_builder - 条件查询
|
||||
- [✅] string_match - 字符串匹配
|
||||
- [✅] not_query - NOT 查询
|
||||
- [✅] schema_query - Schema 验证查询
|
||||
- [✅] persistence - 持久化和恢复
|
||||
- [✅] compaction - Compaction 演示
|
||||
- [✅] multi_wal - 多 WAL 演示
|
||||
- [✅] version_control - 版本控制
|
||||
- [✅] database - 数据库管理
|
||||
- [✅] auto_repair - 自动修复
|
||||
|
||||
文档:
|
||||
- [✅] DESIGN.md - 设计文档
|
||||
- [✅] schema/README.md - Schema 文档
|
||||
- [✅] index/README.md - 索引文档
|
||||
- [✅] examples/README.md - 示例文档
|
||||
```
|
||||
|
||||
### 未来计划 (可选)
|
||||
|
||||
```
|
||||
可能的增强:
|
||||
- [ ] 范围查询优化 (使用 B+Tree 遍历)
|
||||
- [ ] 迭代器 API
|
||||
- [ ] 快照隔离
|
||||
- [ ] 更多压缩算法 (zstd, lz4)
|
||||
- [ ] 列式存储支持
|
||||
- [ ] 分区表支持
|
||||
- [ ] 监控指标导出 (Prometheus)
|
||||
- [ ] 数据导入/导出工具
|
||||
- [ ] 性能分析工具
|
||||
```
|
||||
|
||||
## 📝 关键设计决策
|
||||
|
||||
### 为什么用 map + sorted slice 而不是 SkipList？
|
||||
|
||||
```
|
||||
最初设计: SkipList
|
||||
- 优势: 经典 LSM Tree 实现
|
||||
- 劣势: 实现复杂，需要第三方库
|
||||
|
||||
最终实现: map[int64][]byte + sorted slice
|
||||
- 优势:
|
||||
✅ 实现极简 (130 行)
|
||||
✅ 写入快 O(1)
|
||||
✅ 查询快 O(1)
|
||||
✅ 遍历简单 (已排序的 keys)
|
||||
✅ 无需第三方依赖
|
||||
- 劣势:
|
||||
❌ 每次插入新 key 需要排序
|
||||
|
||||
实测结果:
|
||||
- 写入性能: 与 SkipList 相当或更好
|
||||
- 查询性能: 比 SkipList 更快 (O(1) vs O(log N))
|
||||
- 代码量: 少 3-4 倍
|
||||
|
||||
结论: 简单实用 > 理论最优
|
||||
```
|
||||
|
||||
### 为什么不用列式存储？
|
||||
|
||||
```
|
||||
最初设计 (V2): 列式存储
|
||||
- 优势: 列裁剪，压缩率高
|
||||
- 劣势: 实现复杂，Flush 慢
|
||||
|
||||
最终实现 (V3): 行式存储 + Snappy
|
||||
- 优势: 实现简单，Flush 快
|
||||
- 劣势: 压缩率稍低
|
||||
|
||||
权衡:
|
||||
- 追求简单和快速实现
|
||||
- 行式 + Snappy 已经有 50% 压缩率
|
||||
- 满足大多数时序数据场景
|
||||
- 如果未来需要，可以演进到列式
|
||||
```
|
||||
|
||||
### 为什么用 B+Tree 而不是 LSM Tree？
|
||||
|
||||
```
|
||||
传统 LSM Tree:
|
||||
- 多层结构 (L0, L1, L2, ...)
|
||||
- 复杂的 Compaction
|
||||
- Bloom Filter 过滤
|
||||
|
||||
V3 B+Tree:
|
||||
- 单层 SST 文件
|
||||
- 简单的 Compaction
|
||||
- B+Tree 精确查找
|
||||
|
||||
优势:
|
||||
✅ 实现简单
|
||||
✅ 查询快 (O(log N))
|
||||
✅ 100% 准确
|
||||
✅ mmap 友好
|
||||
```
|
||||
|
||||
### 为什么用 mmap？
|
||||
|
||||
```
|
||||
传统方式: read() 系统调用
|
||||
- 需要复制数据
|
||||
- 占用应用内存
|
||||
- 需要管理缓存
|
||||
|
||||
mmap 方式:
|
||||
- 零拷贝
|
||||
- OS 自动缓存
|
||||
- 应用内存 0 MB
|
||||
|
||||
优势:
|
||||
✅ 内存占用极小
|
||||
✅ 实现简单
|
||||
✅ 性能好
|
||||
✅ OS 自动优化
|
||||
```
|
||||
|
||||
## 🎯 总结
|
||||
|
||||
SRDB 是一个功能完善的高性能 Append-Only 数据库引擎：
|
||||
|
||||
**核心特点:**
|
||||
- ✅ **高并发写入**: WAL + MemTable，200K+ w/s
|
||||
- ✅ **快速查询**: mmap B+Tree + 二级索引，1-5 ms
|
||||
- ✅ **低内存占用**: mmap 零拷贝，< 150 MB
|
||||
- ✅ **功能完善**: Schema、索引、条件查询、多表管理
|
||||
- ✅ **生产可用**: 5399 行核心代码，完善的错误处理和数据一致性
|
||||
- ✅ **简单可靠**: Append-Only，无更新/删除的复杂性
|
||||
|
||||
**技术亮点:**
|
||||
- 简洁的 MemTable 实现 (map + sorted slice)
|
||||
- B+Tree 索引，4KB 节点对齐
|
||||
- Snappy 压缩，50% 压缩率
|
||||
- 多版本 MemTable 管理
|
||||
- 后台 Compaction
|
||||
- 版本控制和自动修复
|
||||
- 灵活的查询构建器
|
||||
|
||||
**适用场景:**
|
||||
- ✅ 日志存储和分析
|
||||
- ✅ 时序数据（IoT、监控）
|
||||
- ✅ 事件溯源系统
|
||||
- ✅ 监控指标存储
|
||||
- ✅ 审计日志
|
||||
- ✅ 任何 Append-Only 场景
|
||||
|
||||
**不适用场景:**
|
||||
- ❌ 需要频繁更新/删除的场景
|
||||
- ❌ 需要多表 JOIN
|
||||
- ❌ 需要复杂事务
|
||||
- ❌ 传统 OLTP 系统
|
||||
|
||||
**项目成果:**
|
||||
- 核心代码: 5399 行
|
||||
- 测试代码: 2000+ 行
|
||||
- 示例程序: 13 个完整示例
|
||||
- 文档: 完善的设计和使用文档
|
||||
- 性能: 达到设计目标
|
||||
|
||||
---
|
||||
|
||||
**项目已完成并可用于生产环境！** 🎉
|
||||
85
Makefile
Normal file
85
Makefile
Normal file
@@ -0,0 +1,85 @@
|
||||
.PHONY: help test test-verbose test-coverage test-race test-bench test-engine test-compaction test-query fmt fmt-check vet tidy verify clean
|
||||
|
||||
# 默认目标
|
||||
.DEFAULT_GOAL := help
|
||||
|
||||
# 颜色输出
|
||||
GREEN := $(shell tput -Txterm setaf 2)
|
||||
YELLOW := $(shell tput -Txterm setaf 3)
|
||||
BLUE := $(shell tput -Txterm setaf 4)
|
||||
RESET := $(shell tput -Txterm sgr0)
|
||||
|
||||
help: ## 显示帮助信息
|
||||
@echo '$(BLUE)SRDB Makefile 命令:$(RESET)'
|
||||
@echo ''
|
||||
@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | \
|
||||
awk 'BEGIN {FS = ":.*?## "}; {printf " $(YELLOW)%-18s$(RESET) %s\n", $$1, $$2}'
|
||||
@echo ''
|
||||
|
||||
test: ## 运行所有测试
|
||||
@echo "$(GREEN)运行测试...$(RESET)"
|
||||
@go test $$(go list ./... | grep -v /examples/)
|
||||
@echo "$(GREEN)✓ 测试完成$(RESET)"
|
||||
|
||||
test-verbose: ## 运行测试（详细输出）
|
||||
@echo "$(GREEN)运行测试（详细模式）...$(RESET)"
|
||||
@go test -v $$(go list ./... | grep -v /examples/)
|
||||
|
||||
test-coverage: ## 运行测试并生成覆盖率报告
|
||||
@echo "$(GREEN)运行测试并生成覆盖率报告...$(RESET)"
|
||||
@go test -v -coverprofile=coverage.out $$(go list ./... | grep -v /examples/)
|
||||
@go tool cover -html=coverage.out -o coverage.html
|
||||
@echo "$(GREEN)✓ 覆盖率报告已生成: coverage.html$(RESET)"
|
||||
|
||||
test-race: ## 运行测试（启用竞态检测）
|
||||
@echo "$(GREEN)运行测试（竞态检测）...$(RESET)"
|
||||
@go test -race $$(go list ./... | grep -v /examples/)
|
||||
@echo "$(GREEN)✓ 竞态检测完成$(RESET)"
|
||||
|
||||
test-bench: ## 运行基准测试
|
||||
@echo "$(GREEN)运行基准测试...$(RESET)"
|
||||
@go test -bench=. -benchmem $$(go list ./... | grep -v /examples/)
|
||||
|
||||
test-engine: ## 只运行 engine 包的测试
|
||||
@echo "$(GREEN)运行 engine 测试...$(RESET)"
|
||||
@go test -v ./engine
|
||||
|
||||
test-compaction: ## 只运行 compaction 包的测试
|
||||
@echo "$(GREEN)运行 compaction 测试...$(RESET)"
|
||||
@go test -v ./compaction
|
||||
|
||||
test-query: ## 只运行 query 包的测试
|
||||
@echo "$(GREEN)运行 query 测试...$(RESET)"
|
||||
@go test -v ./query
|
||||
|
||||
fmt: ## 格式化代码
|
||||
@echo "$(GREEN)格式化代码...$(RESET)"
|
||||
@go fmt ./...
|
||||
@echo "$(GREEN)✓ 代码格式化完成$(RESET)"
|
||||
|
||||
fmt-check: ## 检查代码格式（不修改）
|
||||
@echo "$(GREEN)检查代码格式...$(RESET)"
|
||||
@test -z "$$(gofmt -l .)" || (echo "$(YELLOW)以下文件需要格式化:$(RESET)" && gofmt -l . && exit 1)
|
||||
@echo "$(GREEN)✓ 代码格式正确$(RESET)"
|
||||
|
||||
vet: ## 运行 go vet 静态分析
|
||||
@echo "$(GREEN)运行 go vet...$(RESET)"
|
||||
@go vet $$(go list ./... | grep -v /examples/)
|
||||
@echo "$(GREEN)✓ 静态分析完成$(RESET)"
|
||||
|
||||
tidy: ## 整理依赖
|
||||
@echo "$(GREEN)整理依赖...$(RESET)"
|
||||
@go mod tidy
|
||||
@echo "$(GREEN)✓ 依赖整理完成$(RESET)"
|
||||
|
||||
verify: ## 验证依赖
|
||||
@echo "$(GREEN)验证依赖...$(RESET)"
|
||||
@go mod verify
|
||||
@echo "$(GREEN)✓ 依赖验证完成$(RESET)"
|
||||
|
||||
clean: ## 清理测试文件
|
||||
@echo "$(GREEN)清理测试文件...$(RESET)"
|
||||
@rm -f coverage.out coverage.html
|
||||
@find . -type d -name "mydb*" -exec rm -rf {} + 2>/dev/null || true
|
||||
@find . -type d -name "testdb*" -exec rm -rf {} + 2>/dev/null || true
|
||||
@echo "$(GREEN)✓ 清理完成$(RESET)"
|
||||
155
btree/btree_test.go
Normal file
155
btree/btree_test.go
Normal file
@@ -0,0 +1,155 @@
|
||||
package btree
|
||||
|
||||
import (
|
||||
"os"
|
||||
"testing"
|
||||
|
||||
"github.com/edsrzf/mmap-go"
|
||||
)
|
||||
|
||||
func TestBTree(t *testing.T) {
|
||||
// 1. 创建测试文件
|
||||
file, err := os.Create("test.sst")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer os.Remove("test.sst")
|
||||
|
||||
// 2. 构建 B+Tree
|
||||
builder := NewBuilder(file, 256) // 从 offset 256 开始
|
||||
|
||||
// 添加 1000 个 key-value
|
||||
for i := int64(1); i <= 1000; i++ {
|
||||
dataOffset := 1000000 + i*100 // 模拟数据位置
|
||||
dataSize := int32(100)
|
||||
err := builder.Add(i, dataOffset, dataSize)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
// 构建
|
||||
rootOffset, err := builder.Build()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
t.Logf("Root offset: %d", rootOffset)
|
||||
|
||||
// 3. 关闭并重新打开文件
|
||||
file.Close()
|
||||
|
||||
file, err = os.Open("test.sst")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer file.Close()
|
||||
|
||||
// 4. mmap 映射
|
||||
mmapData, err := mmap.Map(file, mmap.RDONLY, 0)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer mmapData.Unmap()
|
||||
|
||||
// 5. 查询测试
|
||||
reader := NewReader(mmapData, rootOffset)
|
||||
|
||||
// 测试存在的 key
|
||||
for i := int64(1); i <= 1000; i++ {
|
||||
offset, size, found := reader.Get(i)
|
||||
if !found {
|
||||
t.Errorf("Key %d not found", i)
|
||||
}
|
||||
expectedOffset := 1000000 + i*100
|
||||
if offset != expectedOffset {
|
||||
t.Errorf("Key %d: expected offset %d, got %d", i, expectedOffset, offset)
|
||||
}
|
||||
if size != 100 {
|
||||
t.Errorf("Key %d: expected size 100, got %d", i, size)
|
||||
}
|
||||
}
|
||||
|
||||
// 测试不存在的 key
|
||||
_, _, found := reader.Get(1001)
|
||||
if found {
|
||||
t.Error("Key 1001 should not exist")
|
||||
}
|
||||
|
||||
_, _, found = reader.Get(0)
|
||||
if found {
|
||||
t.Error("Key 0 should not exist")
|
||||
}
|
||||
|
||||
t.Log("All tests passed!")
|
||||
}
|
||||
|
||||
func TestBTreeSerialization(t *testing.T) {
|
||||
// 测试节点序列化
|
||||
leaf := NewLeafNode()
|
||||
leaf.AddData(1, 1000, 100)
|
||||
leaf.AddData(2, 2000, 200)
|
||||
leaf.AddData(3, 3000, 300)
|
||||
|
||||
// 序列化
|
||||
data := leaf.Marshal()
|
||||
if len(data) != NodeSize {
|
||||
t.Errorf("Expected size %d, got %d", NodeSize, len(data))
|
||||
}
|
||||
|
||||
// 反序列化
|
||||
leaf2 := Unmarshal(data)
|
||||
if leaf2 == nil {
|
||||
t.Fatal("Unmarshal failed")
|
||||
}
|
||||
|
||||
// 验证
|
||||
if leaf2.NodeType != NodeTypeLeaf {
|
||||
t.Error("Wrong node type")
|
||||
}
|
||||
if leaf2.KeyCount != 3 {
|
||||
t.Errorf("Expected 3 keys, got %d", leaf2.KeyCount)
|
||||
}
|
||||
if len(leaf2.Keys) != 3 {
|
||||
t.Errorf("Expected 3 keys, got %d", len(leaf2.Keys))
|
||||
}
|
||||
if leaf2.Keys[0] != 1 || leaf2.Keys[1] != 2 || leaf2.Keys[2] != 3 {
|
||||
t.Error("Keys mismatch")
|
||||
}
|
||||
if leaf2.DataOffsets[0] != 1000 || leaf2.DataOffsets[1] != 2000 || leaf2.DataOffsets[2] != 3000 {
|
||||
t.Error("Data offsets mismatch")
|
||||
}
|
||||
if leaf2.DataSizes[0] != 100 || leaf2.DataSizes[1] != 200 || leaf2.DataSizes[2] != 300 {
|
||||
t.Error("Data sizes mismatch")
|
||||
}
|
||||
|
||||
t.Log("Serialization test passed!")
|
||||
}
|
||||
|
||||
func BenchmarkBTreeGet(b *testing.B) {
|
||||
// 构建测试数据
|
||||
file, _ := os.Create("bench.sst")
|
||||
defer os.Remove("bench.sst")
|
||||
|
||||
builder := NewBuilder(file, 256)
|
||||
for i := int64(1); i <= 100000; i++ {
|
||||
builder.Add(i, i*100, 100)
|
||||
}
|
||||
rootOffset, _ := builder.Build()
|
||||
file.Close()
|
||||
|
||||
// mmap
|
||||
file, _ = os.Open("bench.sst")
|
||||
defer file.Close()
|
||||
mmapData, _ := mmap.Map(file, mmap.RDONLY, 0)
|
||||
defer mmapData.Unmap()
|
||||
|
||||
reader := NewReader(mmapData, rootOffset)
|
||||
|
||||
// 性能测试
|
||||
b.ResetTimer()
|
||||
for i := 0; i < b.N; i++ {
|
||||
key := int64(i%100000 + 1)
|
||||
reader.Get(key)
|
||||
}
|
||||
}
|
||||
122
btree/builder.go
Normal file
122
btree/builder.go
Normal file
@@ -0,0 +1,122 @@
|
||||
package btree
|
||||
|
||||
import (
|
||||
"os"
|
||||
)
|
||||
|
||||
// Builder 从下往上构建 B+Tree
|
||||
type Builder struct {
|
||||
order int // B+Tree 阶数
|
||||
file *os.File // 输出文件
|
||||
offset int64 // 当前写入位置
|
||||
leafNodes []*BTreeNode // 叶子节点列表
|
||||
}
|
||||
|
||||
// NewBuilder 创建构建器
|
||||
func NewBuilder(file *os.File, startOffset int64) *Builder {
|
||||
return &Builder{
|
||||
order: Order,
|
||||
file: file,
|
||||
offset: startOffset,
|
||||
leafNodes: make([]*BTreeNode, 0),
|
||||
}
|
||||
}
|
||||
|
||||
// Add 添加一个 key-value 对 (数据必须已排序)
|
||||
func (b *Builder) Add(key int64, dataOffset int64, dataSize int32) error {
|
||||
// 获取或创建当前叶子节点
|
||||
var leaf *BTreeNode
|
||||
if len(b.leafNodes) == 0 || b.leafNodes[len(b.leafNodes)-1].IsFull() {
|
||||
// 创建新的叶子节点
|
||||
leaf = NewLeafNode()
|
||||
b.leafNodes = append(b.leafNodes, leaf)
|
||||
} else {
|
||||
leaf = b.leafNodes[len(b.leafNodes)-1]
|
||||
}
|
||||
|
||||
// 添加到叶子节点
|
||||
leaf.AddData(key, dataOffset, dataSize)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// Build 构建完整的 B+Tree，返回根节点的 offset
|
||||
func (b *Builder) Build() (rootOffset int64, err error) {
|
||||
if len(b.leafNodes) == 0 {
|
||||
return 0, nil
|
||||
}
|
||||
|
||||
// 1. 写入所有叶子节点，记录它们的 offset
|
||||
leafOffsets := make([]int64, len(b.leafNodes))
|
||||
for i, leaf := range b.leafNodes {
|
||||
leafOffsets[i] = b.offset
|
||||
data := leaf.Marshal()
|
||||
_, err := b.file.WriteAt(data, b.offset)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
b.offset += NodeSize
|
||||
}
|
||||
|
||||
// 2. 如果只有一个叶子节点，它就是根
|
||||
if len(b.leafNodes) == 1 {
|
||||
return leafOffsets[0], nil
|
||||
}
|
||||
|
||||
// 3. 从下往上构建内部节点
|
||||
currentLevel := b.leafNodes
|
||||
currentOffsets := leafOffsets
|
||||
level := 1
|
||||
|
||||
for len(currentLevel) > 1 {
|
||||
nextLevel, nextOffsets, err := b.buildLevel(currentLevel, currentOffsets, level)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
currentLevel = nextLevel
|
||||
currentOffsets = nextOffsets
|
||||
level++
|
||||
}
|
||||
|
||||
// 4. 返回根节点的 offset
|
||||
return currentOffsets[0], nil
|
||||
}
|
||||
|
||||
// buildLevel 构建一层内部节点
|
||||
func (b *Builder) buildLevel(children []*BTreeNode, childOffsets []int64, level int) ([]*BTreeNode, []int64, error) {
|
||||
var parents []*BTreeNode
|
||||
var parentOffsets []int64
|
||||
|
||||
// 每 order 个子节点创建一个父节点
|
||||
for i := 0; i < len(children); i += b.order {
|
||||
end := min(i+b.order, len(children))
|
||||
|
||||
// 创建父节点
|
||||
parent := NewInternalNode(byte(level))
|
||||
|
||||
// 添加第一个子节点 (没有对应的 key)
|
||||
parent.AddChild(childOffsets[i])
|
||||
|
||||
// 添加剩余的子节点和分隔 key
|
||||
for j := i + 1; j < end; j++ {
|
||||
// 分隔 key 是子节点的第一个 key
|
||||
separatorKey := children[j].Keys[0]
|
||||
parent.AddKey(separatorKey)
|
||||
parent.AddChild(childOffsets[j])
|
||||
}
|
||||
|
||||
// 写入父节点
|
||||
parentOffset := b.offset
|
||||
data := parent.Marshal()
|
||||
_, err := b.file.WriteAt(data, b.offset)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
b.offset += NodeSize
|
||||
|
||||
parents = append(parents, parent)
|
||||
parentOffsets = append(parentOffsets, parentOffset)
|
||||
}
|
||||
|
||||
return parents, parentOffsets, nil
|
||||
}
|
||||
185
btree/node.go
Normal file
185
btree/node.go
Normal file
@@ -0,0 +1,185 @@
|
||||
package btree
|
||||
|
||||
import (
|
||||
"encoding/binary"
|
||||
)
|
||||
|
||||
const (
|
||||
NodeSize = 4096 // 节点大小 (4 KB)
|
||||
Order = 200 // B+Tree 阶数 (保守估计，叶子节点每个entry 20 bytes)
|
||||
HeaderSize = 32 // 节点头大小
|
||||
NodeTypeInternal = 0 // 内部节点
|
||||
NodeTypeLeaf = 1 // 叶子节点
|
||||
)
|
||||
|
||||
// BTreeNode 表示一个 B+Tree 节点 (4 KB)
|
||||
type BTreeNode struct {
|
||||
// Header (32 bytes)
|
||||
NodeType byte // 0=Internal, 1=Leaf
|
||||
KeyCount uint16 // key 数量
|
||||
Level byte // 层级 (0=叶子层)
|
||||
Reserved [28]byte // 预留字段
|
||||
|
||||
// Keys (variable, 最多 256 个)
|
||||
Keys []int64 // key 数组
|
||||
|
||||
// Values (variable)
|
||||
// Internal Node: 子节点指针
|
||||
Children []int64 // 子节点的文件 offset
|
||||
|
||||
// Leaf Node: 数据位置
|
||||
DataOffsets []int64 // 数据块的文件 offset
|
||||
DataSizes []int32 // 数据块大小
|
||||
}
|
||||
|
||||
// NewInternalNode 创建内部节点
|
||||
func NewInternalNode(level byte) *BTreeNode {
|
||||
return &BTreeNode{
|
||||
NodeType: NodeTypeInternal,
|
||||
Level: level,
|
||||
Keys: make([]int64, 0, Order),
|
||||
Children: make([]int64, 0, Order+1),
|
||||
}
|
||||
}
|
||||
|
||||
// NewLeafNode 创建叶子节点
|
||||
func NewLeafNode() *BTreeNode {
|
||||
return &BTreeNode{
|
||||
NodeType: NodeTypeLeaf,
|
||||
Level: 0,
|
||||
Keys: make([]int64, 0, Order),
|
||||
DataOffsets: make([]int64, 0, Order),
|
||||
DataSizes: make([]int32, 0, Order),
|
||||
}
|
||||
}
|
||||
|
||||
// Marshal 序列化节点到 4 KB
|
||||
func (n *BTreeNode) Marshal() []byte {
|
||||
buf := make([]byte, NodeSize)
|
||||
|
||||
// 写入 Header (32 bytes)
|
||||
buf[0] = n.NodeType
|
||||
binary.LittleEndian.PutUint16(buf[1:3], n.KeyCount)
|
||||
buf[3] = n.Level
|
||||
copy(buf[4:32], n.Reserved[:])
|
||||
|
||||
// 写入 Keys
|
||||
offset := HeaderSize
|
||||
for _, key := range n.Keys {
|
||||
if offset+8 > NodeSize {
|
||||
break
|
||||
}
|
||||
binary.LittleEndian.PutUint64(buf[offset:offset+8], uint64(key))
|
||||
offset += 8
|
||||
}
|
||||
|
||||
// 写入 Values
|
||||
if n.NodeType == NodeTypeInternal {
|
||||
// Internal Node: 写入子节点指针
|
||||
for _, child := range n.Children {
|
||||
if offset+8 > NodeSize {
|
||||
break
|
||||
}
|
||||
binary.LittleEndian.PutUint64(buf[offset:offset+8], uint64(child))
|
||||
offset += 8
|
||||
}
|
||||
} else {
|
||||
// Leaf Node: 写入数据位置
|
||||
for i := 0; i < len(n.Keys); i++ {
|
||||
if offset+12 > NodeSize {
|
||||
break
|
||||
}
|
||||
binary.LittleEndian.PutUint64(buf[offset:offset+8], uint64(n.DataOffsets[i]))
|
||||
offset += 8
|
||||
binary.LittleEndian.PutUint32(buf[offset:offset+4], uint32(n.DataSizes[i]))
|
||||
offset += 4
|
||||
}
|
||||
}
|
||||
|
||||
return buf
|
||||
}
|
||||
|
||||
// Unmarshal 从字节数组反序列化节点
|
||||
func Unmarshal(data []byte) *BTreeNode {
|
||||
if len(data) < NodeSize {
|
||||
return nil
|
||||
}
|
||||
|
||||
node := &BTreeNode{}
|
||||
|
||||
// 读取 Header
|
||||
node.NodeType = data[0]
|
||||
node.KeyCount = binary.LittleEndian.Uint16(data[1:3])
|
||||
node.Level = data[3]
|
||||
copy(node.Reserved[:], data[4:32])
|
||||
|
||||
// 读取 Keys
|
||||
offset := HeaderSize
|
||||
node.Keys = make([]int64, node.KeyCount)
|
||||
for i := 0; i < int(node.KeyCount); i++ {
|
||||
if offset+8 > len(data) {
|
||||
break
|
||||
}
|
||||
node.Keys[i] = int64(binary.LittleEndian.Uint64(data[offset : offset+8]))
|
||||
offset += 8
|
||||
}
|
||||
|
||||
// 读取 Values
|
||||
if node.NodeType == NodeTypeInternal {
|
||||
// Internal Node: 读取子节点指针
|
||||
childCount := int(node.KeyCount) + 1
|
||||
node.Children = make([]int64, childCount)
|
||||
for i := 0; i < childCount; i++ {
|
||||
if offset+8 > len(data) {
|
||||
break
|
||||
}
|
||||
node.Children[i] = int64(binary.LittleEndian.Uint64(data[offset : offset+8]))
|
||||
offset += 8
|
||||
}
|
||||
} else {
|
||||
// Leaf Node: 读取数据位置
|
||||
node.DataOffsets = make([]int64, node.KeyCount)
|
||||
node.DataSizes = make([]int32, node.KeyCount)
|
||||
for i := 0; i < int(node.KeyCount); i++ {
|
||||
if offset+12 > len(data) {
|
||||
break
|
||||
}
|
||||
node.DataOffsets[i] = int64(binary.LittleEndian.Uint64(data[offset : offset+8]))
|
||||
offset += 8
|
||||
node.DataSizes[i] = int32(binary.LittleEndian.Uint32(data[offset : offset+4]))
|
||||
offset += 4
|
||||
}
|
||||
}
|
||||
|
||||
return node
|
||||
}
|
||||
|
||||
// IsFull 检查节点是否已满
|
||||
func (n *BTreeNode) IsFull() bool {
|
||||
return len(n.Keys) >= Order
|
||||
}
|
||||
|
||||
// AddKey 添加 key (仅用于构建)
|
||||
func (n *BTreeNode) AddKey(key int64) {
|
||||
n.Keys = append(n.Keys, key)
|
||||
n.KeyCount = uint16(len(n.Keys))
|
||||
}
|
||||
|
||||
// AddChild 添加子节点 (仅用于内部节点)
|
||||
func (n *BTreeNode) AddChild(offset int64) {
|
||||
if n.NodeType != NodeTypeInternal {
|
||||
panic("AddChild called on leaf node")
|
||||
}
|
||||
n.Children = append(n.Children, offset)
|
||||
}
|
||||
|
||||
// AddData 添加数据位置 (仅用于叶子节点)
|
||||
func (n *BTreeNode) AddData(key int64, offset int64, size int32) {
|
||||
if n.NodeType != NodeTypeLeaf {
|
||||
panic("AddData called on internal node")
|
||||
}
|
||||
n.Keys = append(n.Keys, key)
|
||||
n.DataOffsets = append(n.DataOffsets, offset)
|
||||
n.DataSizes = append(n.DataSizes, size)
|
||||
n.KeyCount = uint16(len(n.Keys))
|
||||
}
|
||||
106
btree/reader.go
Normal file
106
btree/reader.go
Normal file
@@ -0,0 +1,106 @@
|
||||
package btree
|
||||
|
||||
import (
|
||||
"sort"
|
||||
|
||||
"github.com/edsrzf/mmap-go"
|
||||
)
|
||||
|
||||
// Reader 用于查询 B+Tree (mmap)
|
||||
type Reader struct {
|
||||
mmap mmap.MMap
|
||||
rootOffset int64
|
||||
}
|
||||
|
||||
// NewReader 创建查询器
|
||||
func NewReader(mmap mmap.MMap, rootOffset int64) *Reader {
|
||||
return &Reader{
|
||||
mmap: mmap,
|
||||
rootOffset: rootOffset,
|
||||
}
|
||||
}
|
||||
|
||||
// Get 查询 key，返回数据位置
|
||||
func (r *Reader) Get(key int64) (dataOffset int64, dataSize int32, found bool) {
|
||||
if r.rootOffset == 0 {
|
||||
return 0, 0, false
|
||||
}
|
||||
|
||||
nodeOffset := r.rootOffset
|
||||
|
||||
for {
|
||||
// 读取节点 (零拷贝)
|
||||
if nodeOffset+NodeSize > int64(len(r.mmap)) {
|
||||
return 0, 0, false
|
||||
}
|
||||
|
||||
nodeData := r.mmap[nodeOffset : nodeOffset+NodeSize]
|
||||
node := Unmarshal(nodeData)
|
||||
|
||||
if node == nil {
|
||||
return 0, 0, false
|
||||
}
|
||||
|
||||
// 叶子节点
|
||||
if node.NodeType == NodeTypeLeaf {
|
||||
// 二分查找
|
||||
idx := sort.Search(len(node.Keys), func(i int) bool {
|
||||
return node.Keys[i] >= key
|
||||
})
|
||||
if idx < len(node.Keys) && node.Keys[idx] == key {
|
||||
return node.DataOffsets[idx], node.DataSizes[idx], true
|
||||
}
|
||||
return 0, 0, false
|
||||
}
|
||||
|
||||
// 内部节点，继续向下
|
||||
// keys[i] 是分隔符，children[i] 包含 < keys[i] 的数据
|
||||
// children[i+1] 包含 >= keys[i] 的数据
|
||||
idx := sort.Search(len(node.Keys), func(i int) bool {
|
||||
return node.Keys[i] > key
|
||||
})
|
||||
// idx 现在指向第一个 > key 的位置
|
||||
// 我们应该走 children[idx]
|
||||
if idx >= len(node.Children) {
|
||||
idx = len(node.Children) - 1
|
||||
}
|
||||
nodeOffset = node.Children[idx]
|
||||
}
|
||||
}
|
||||
|
||||
// GetAllKeys 获取 B+Tree 中所有的 key（按顺序）
|
||||
func (r *Reader) GetAllKeys() []int64 {
|
||||
if r.rootOffset == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
var keys []int64
|
||||
r.traverseLeafNodes(r.rootOffset, func(node *BTreeNode) {
|
||||
keys = append(keys, node.Keys...)
|
||||
})
|
||||
return keys
|
||||
}
|
||||
|
||||
// traverseLeafNodes 遍历所有叶子节点
|
||||
func (r *Reader) traverseLeafNodes(nodeOffset int64, callback func(*BTreeNode)) {
|
||||
if nodeOffset+NodeSize > int64(len(r.mmap)) {
|
||||
return
|
||||
}
|
||||
|
||||
nodeData := r.mmap[nodeOffset : nodeOffset+NodeSize]
|
||||
node := Unmarshal(nodeData)
|
||||
|
||||
if node == nil {
|
||||
return
|
||||
}
|
||||
|
||||
if node.NodeType == NodeTypeLeaf {
|
||||
// 叶子节点，执行回调
|
||||
callback(node)
|
||||
} else {
|
||||
// 内部节点，递归遍历所有子节点
|
||||
for _, childOffset := range node.Children {
|
||||
r.traverseLeafNodes(childOffset, callback)
|
||||
}
|
||||
}
|
||||
}
|
||||
392
compaction/compaction_test.go
Normal file
392
compaction/compaction_test.go
Normal file
@@ -0,0 +1,392 @@
|
||||
package compaction
|
||||
|
||||
import (
|
||||
"code.tczkiot.com/srdb/manifest"
|
||||
"code.tczkiot.com/srdb/sst"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestCompactionBasic(t *testing.T) {
|
||||
// 创建临时目录
|
||||
tmpDir := t.TempDir()
|
||||
sstDir := filepath.Join(tmpDir, "sst")
|
||||
manifestDir := tmpDir
|
||||
|
||||
err := os.MkdirAll(sstDir, 0755)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// 创建 VersionSet
|
||||
versionSet, err := manifest.NewVersionSet(manifestDir)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer versionSet.Close()
|
||||
|
||||
// 创建 SST Manager
|
||||
sstMgr, err := sst.NewManager(sstDir)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer sstMgr.Close()
|
||||
|
||||
// 创建测试数据
|
||||
rows1 := make([]*sst.Row, 100)
|
||||
for i := 0; i < 100; i++ {
|
||||
rows1[i] = &sst.Row{
|
||||
Seq: int64(i),
|
||||
Time: 1000,
|
||||
Data: map[string]interface{}{"value": i},
|
||||
}
|
||||
}
|
||||
|
||||
// 创建第一个 SST 文件
|
||||
reader1, err := sstMgr.CreateSST(1, rows1)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// 添加到 Version
|
||||
edit1 := manifest.NewVersionEdit()
|
||||
edit1.AddFile(&manifest.FileMetadata{
|
||||
FileNumber: 1,
|
||||
Level: 0,
|
||||
FileSize: 1024,
|
||||
MinKey: 0,
|
||||
MaxKey: 99,
|
||||
RowCount: 100,
|
||||
})
|
||||
nextFileNum := int64(2)
|
||||
edit1.SetNextFileNumber(nextFileNum)
|
||||
|
||||
err = versionSet.LogAndApply(edit1)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// 验证 Version
|
||||
version := versionSet.GetCurrent()
|
||||
if version.GetLevelFileCount(0) != 1 {
|
||||
t.Errorf("Expected 1 file in L0, got %d", version.GetLevelFileCount(0))
|
||||
}
|
||||
|
||||
// 创建 Compaction Manager
|
||||
compactionMgr := NewManager(sstDir, versionSet)
|
||||
|
||||
// 创建更多文件触发 Compaction
|
||||
for i := 1; i < 5; i++ {
|
||||
rows := make([]*sst.Row, 50)
|
||||
for j := 0; j < 50; j++ {
|
||||
rows[j] = &sst.Row{
|
||||
Seq: int64(i*100 + j),
|
||||
Time: int64(1000 + i),
|
||||
Data: map[string]interface{}{"value": i*100 + j},
|
||||
}
|
||||
}
|
||||
|
||||
_, err := sstMgr.CreateSST(int64(i+1), rows)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
edit := manifest.NewVersionEdit()
|
||||
edit.AddFile(&manifest.FileMetadata{
|
||||
FileNumber: int64(i + 1),
|
||||
Level: 0,
|
||||
FileSize: 512,
|
||||
MinKey: int64(i * 100),
|
||||
MaxKey: int64(i*100 + 49),
|
||||
RowCount: 50,
|
||||
})
|
||||
nextFileNum := int64(i + 2)
|
||||
edit.SetNextFileNumber(nextFileNum)
|
||||
|
||||
err = versionSet.LogAndApply(edit)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
// 验证 L0 有 5 个文件
|
||||
version = versionSet.GetCurrent()
|
||||
if version.GetLevelFileCount(0) != 5 {
|
||||
t.Errorf("Expected 5 files in L0, got %d", version.GetLevelFileCount(0))
|
||||
}
|
||||
|
||||
// 检查是否需要 Compaction
|
||||
picker := compactionMgr.GetPicker()
|
||||
if !picker.ShouldCompact(version) {
|
||||
t.Error("Expected compaction to be needed")
|
||||
}
|
||||
|
||||
// 获取 Compaction 任务
|
||||
tasks := picker.PickCompaction(version)
|
||||
if len(tasks) == 0 {
|
||||
t.Fatal("Expected compaction task")
|
||||
}
|
||||
|
||||
task := tasks[0] // 获取第一个任务（优先级最高）
|
||||
|
||||
if task.Level != 0 {
|
||||
t.Errorf("Expected L0 compaction, got L%d", task.Level)
|
||||
}
|
||||
|
||||
if task.OutputLevel != 1 {
|
||||
t.Errorf("Expected output to L1, got L%d", task.OutputLevel)
|
||||
}
|
||||
|
||||
t.Logf("Found %d compaction tasks", len(tasks))
|
||||
t.Logf("First task: L%d -> L%d, %d files", task.Level, task.OutputLevel, len(task.InputFiles))
|
||||
|
||||
// 清理
|
||||
reader1.Close()
|
||||
}
|
||||
|
||||
func TestPickerLevelScore(t *testing.T) {
|
||||
// 创建临时目录
|
||||
tmpDir := t.TempDir()
|
||||
manifestDir := tmpDir
|
||||
|
||||
// 创建 VersionSet
|
||||
versionSet, err := manifest.NewVersionSet(manifestDir)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer versionSet.Close()
|
||||
|
||||
// 创建 Picker
|
||||
picker := NewPicker()
|
||||
|
||||
// 添加一些文件到 L0
|
||||
edit := manifest.NewVersionEdit()
|
||||
for i := 0; i < 3; i++ {
|
||||
edit.AddFile(&manifest.FileMetadata{
|
||||
FileNumber: int64(i + 1),
|
||||
Level: 0,
|
||||
FileSize: 1024 * 1024, // 1MB
|
||||
MinKey: int64(i * 100),
|
||||
MaxKey: int64((i+1)*100 - 1),
|
||||
RowCount: 100,
|
||||
})
|
||||
}
|
||||
nextFileNum := int64(4)
|
||||
edit.SetNextFileNumber(nextFileNum)
|
||||
|
||||
err = versionSet.LogAndApply(edit)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
version := versionSet.GetCurrent()
|
||||
|
||||
// 计算 L0 的得分
|
||||
score := picker.GetLevelScore(version, 0)
|
||||
t.Logf("L0 score: %.2f (files: %d, limit: %d)", score, version.GetLevelFileCount(0), picker.levelFileLimits[0])
|
||||
|
||||
// L0 有 3 个文件，限制是 4，得分应该是 0.75
|
||||
expectedScore := 3.0 / 4.0
|
||||
if score != expectedScore {
|
||||
t.Errorf("Expected L0 score %.2f, got %.2f", expectedScore, score)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCompactionMerge(t *testing.T) {
|
||||
// 创建临时目录
|
||||
tmpDir := t.TempDir()
|
||||
sstDir := filepath.Join(tmpDir, "sst")
|
||||
manifestDir := tmpDir
|
||||
|
||||
err := os.MkdirAll(sstDir, 0755)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// 创建 VersionSet
|
||||
versionSet, err := manifest.NewVersionSet(manifestDir)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer versionSet.Close()
|
||||
|
||||
// 创建 SST Manager
|
||||
sstMgr, err := sst.NewManager(sstDir)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer sstMgr.Close()
|
||||
|
||||
// 创建两个有重叠 key 的 SST 文件
|
||||
rows1 := []*sst.Row{
|
||||
{Seq: 1, Time: 1000, Data: map[string]interface{}{"value": "old"}},
|
||||
{Seq: 2, Time: 1000, Data: map[string]interface{}{"value": "old"}},
|
||||
}
|
||||
|
||||
rows2 := []*sst.Row{
|
||||
{Seq: 1, Time: 2000, Data: map[string]interface{}{"value": "new"}}, // 更新
|
||||
{Seq: 3, Time: 2000, Data: map[string]interface{}{"value": "new"}},
|
||||
}
|
||||
|
||||
reader1, err := sstMgr.CreateSST(1, rows1)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer reader1.Close()
|
||||
|
||||
reader2, err := sstMgr.CreateSST(2, rows2)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer reader2.Close()
|
||||
|
||||
// 添加到 Version
|
||||
edit := manifest.NewVersionEdit()
|
||||
edit.AddFile(&manifest.FileMetadata{
|
||||
FileNumber: 1,
|
||||
Level: 0,
|
||||
FileSize: 512,
|
||||
MinKey: 1,
|
||||
MaxKey: 2,
|
||||
RowCount: 2,
|
||||
})
|
||||
edit.AddFile(&manifest.FileMetadata{
|
||||
FileNumber: 2,
|
||||
Level: 0,
|
||||
FileSize: 512,
|
||||
MinKey: 1,
|
||||
MaxKey: 3,
|
||||
RowCount: 2,
|
||||
})
|
||||
nextFileNum := int64(3)
|
||||
edit.SetNextFileNumber(nextFileNum)
|
||||
|
||||
err = versionSet.LogAndApply(edit)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// 创建 Compactor
|
||||
compactor := NewCompactor(sstDir, versionSet)
|
||||
|
||||
// 创建 Compaction 任务
|
||||
version := versionSet.GetCurrent()
|
||||
task := &CompactionTask{
|
||||
Level: 0,
|
||||
InputFiles: version.GetLevel(0),
|
||||
OutputLevel: 1,
|
||||
}
|
||||
|
||||
// 执行 Compaction
|
||||
resultEdit, err := compactor.DoCompaction(task, version)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// 验证结果
|
||||
if len(resultEdit.DeletedFiles) != 2 {
|
||||
t.Errorf("Expected 2 deleted files, got %d", len(resultEdit.DeletedFiles))
|
||||
}
|
||||
|
||||
if len(resultEdit.AddedFiles) == 0 {
|
||||
t.Error("Expected at least 1 new file")
|
||||
}
|
||||
|
||||
t.Logf("Compaction result: deleted %d files, added %d files", len(resultEdit.DeletedFiles), len(resultEdit.AddedFiles))
|
||||
|
||||
// 验证新文件在 L1
|
||||
for _, file := range resultEdit.AddedFiles {
|
||||
if file.Level != 1 {
|
||||
t.Errorf("Expected new file in L1, got L%d", file.Level)
|
||||
}
|
||||
t.Logf("New file: %d, L%d, rows: %d, key range: [%d, %d]",
|
||||
file.FileNumber, file.Level, file.RowCount, file.MinKey, file.MaxKey)
|
||||
}
|
||||
}
|
||||
|
||||
func BenchmarkCompaction(b *testing.B) {
|
||||
// 创建临时目录
|
||||
tmpDir := b.TempDir()
|
||||
sstDir := filepath.Join(tmpDir, "sst")
|
||||
manifestDir := tmpDir
|
||||
|
||||
err := os.MkdirAll(sstDir, 0755)
|
||||
if err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
|
||||
// 创建 VersionSet
|
||||
versionSet, err := manifest.NewVersionSet(manifestDir)
|
||||
if err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
defer versionSet.Close()
|
||||
|
||||
// 创建 SST Manager
|
||||
sstMgr, err := sst.NewManager(sstDir)
|
||||
if err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
defer sstMgr.Close()
|
||||
|
||||
// 创建测试数据
|
||||
const numFiles = 5
|
||||
const rowsPerFile = 1000
|
||||
|
||||
for i := 0; i < numFiles; i++ {
|
||||
rows := make([]*sst.Row, rowsPerFile)
|
||||
for j := 0; j < rowsPerFile; j++ {
|
||||
rows[j] = &sst.Row{
|
||||
Seq: int64(i*rowsPerFile + j),
|
||||
Time: int64(1000 + i),
|
||||
Data: map[string]interface{}{
|
||||
"value": fmt.Sprintf("data-%d-%d", i, j),
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
reader, err := sstMgr.CreateSST(int64(i+1), rows)
|
||||
if err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
reader.Close()
|
||||
|
||||
edit := manifest.NewVersionEdit()
|
||||
edit.AddFile(&manifest.FileMetadata{
|
||||
FileNumber: int64(i + 1),
|
||||
Level: 0,
|
||||
FileSize: 10240,
|
||||
MinKey: int64(i * rowsPerFile),
|
||||
MaxKey: int64((i+1)*rowsPerFile - 1),
|
||||
RowCount: rowsPerFile,
|
||||
})
|
||||
nextFileNum := int64(i + 2)
|
||||
edit.SetNextFileNumber(nextFileNum)
|
||||
|
||||
err = versionSet.LogAndApply(edit)
|
||||
if err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
// 创建 Compactor
|
||||
compactor := NewCompactor(sstDir, versionSet)
|
||||
version := versionSet.GetCurrent()
|
||||
|
||||
task := &CompactionTask{
|
||||
Level: 0,
|
||||
InputFiles: version.GetLevel(0),
|
||||
OutputLevel: 1,
|
||||
}
|
||||
|
||||
b.ResetTimer()
|
||||
|
||||
for i := 0; i < b.N; i++ {
|
||||
_, err := compactor.DoCompaction(task, version)
|
||||
if err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
}
|
||||
}
|
||||
370
compaction/compactor.go
Normal file
370
compaction/compactor.go
Normal file
@@ -0,0 +1,370 @@
|
||||
package compaction
|
||||
|
||||
import (
|
||||
"code.tczkiot.com/srdb/manifest"
|
||||
"code.tczkiot.com/srdb/sst"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"sync"
|
||||
)
|
||||
|
||||
// Compactor 负责执行 Compaction
|
||||
type Compactor struct {
|
||||
sstDir string
|
||||
picker *Picker
|
||||
versionSet *manifest.VersionSet
|
||||
mu sync.Mutex
|
||||
}
|
||||
|
||||
// NewCompactor 创建新的 Compactor
|
||||
func NewCompactor(sstDir string, versionSet *manifest.VersionSet) *Compactor {
|
||||
return &Compactor{
|
||||
sstDir: sstDir,
|
||||
picker: NewPicker(),
|
||||
versionSet: versionSet,
|
||||
}
|
||||
}
|
||||
|
||||
// GetPicker 获取 Picker
|
||||
func (c *Compactor) GetPicker() *Picker {
|
||||
return c.picker
|
||||
}
|
||||
|
||||
// DoCompaction 执行一次 Compaction
|
||||
// 返回: VersionEdit (记录变更), error
|
||||
func (c *Compactor) DoCompaction(task *CompactionTask, version *manifest.Version) (*manifest.VersionEdit, error) {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
|
||||
if task == nil {
|
||||
return nil, fmt.Errorf("compaction task is nil")
|
||||
}
|
||||
|
||||
// 0. 验证输入文件是否存在（防止并发 compaction 导致的竞态）
|
||||
existingInputFiles := make([]*manifest.FileMetadata, 0, len(task.InputFiles))
|
||||
for _, file := range task.InputFiles {
|
||||
sstPath := filepath.Join(c.sstDir, fmt.Sprintf("%06d.sst", file.FileNumber))
|
||||
if _, err := os.Stat(sstPath); err == nil {
|
||||
existingInputFiles = append(existingInputFiles, file)
|
||||
} else {
|
||||
fmt.Printf("[Compaction] Warning: input file %06d.sst not found, skipping from task\n", file.FileNumber)
|
||||
}
|
||||
}
|
||||
|
||||
// 如果所有输入文件都不存在，直接返回（无需 compaction）
|
||||
if len(existingInputFiles) == 0 {
|
||||
fmt.Printf("[Compaction] All input files missing, compaction skipped\n")
|
||||
return nil, nil // 返回 nil 表示不需要应用任何 VersionEdit
|
||||
}
|
||||
|
||||
// 1. 读取输入文件的所有行
|
||||
inputRows, err := c.readInputFiles(existingInputFiles)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("read input files: %w", err)
|
||||
}
|
||||
|
||||
// 2. 如果输出层级有文件，需要合并重叠的文件
|
||||
outputFiles := c.getOverlappingFiles(version, task.OutputLevel, inputRows)
|
||||
var existingOutputFiles []*manifest.FileMetadata
|
||||
var missingOutputFiles []*manifest.FileMetadata
|
||||
if len(outputFiles) > 0 {
|
||||
// 验证输出文件是否存在
|
||||
existingOutputFiles = make([]*manifest.FileMetadata, 0, len(outputFiles))
|
||||
missingOutputFiles = make([]*manifest.FileMetadata, 0)
|
||||
for _, file := range outputFiles {
|
||||
sstPath := filepath.Join(c.sstDir, fmt.Sprintf("%06d.sst", file.FileNumber))
|
||||
if _, err := os.Stat(sstPath); err == nil {
|
||||
existingOutputFiles = append(existingOutputFiles, file)
|
||||
} else {
|
||||
// 输出层级的文件不存在，记录并在 VersionEdit 中删除它
|
||||
fmt.Printf("[Compaction] Warning: overlapping output file %06d.sst missing, will remove from MANIFEST\n", file.FileNumber)
|
||||
missingOutputFiles = append(missingOutputFiles, file)
|
||||
}
|
||||
}
|
||||
|
||||
outputRows, err := c.readInputFiles(existingOutputFiles)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("read output files: %w", err)
|
||||
}
|
||||
inputRows = append(inputRows, outputRows...)
|
||||
}
|
||||
|
||||
// 3. 合并和去重 (保留最新的记录)
|
||||
mergedRows := c.mergeRows(inputRows)
|
||||
|
||||
// 计算平均行大小（基于输入文件的 FileMetadata）
|
||||
avgRowSize := c.calculateAvgRowSize(existingInputFiles, existingOutputFiles)
|
||||
|
||||
// 4. 写入新的 SST 文件到输出层级
|
||||
newFiles, err := c.writeOutputFiles(mergedRows, task.OutputLevel, version, avgRowSize)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("write output files: %w", err)
|
||||
}
|
||||
|
||||
// 5. 创建 VersionEdit
|
||||
edit := manifest.NewVersionEdit()
|
||||
|
||||
// 删除实际存在且被处理的输入文件
|
||||
for _, file := range existingInputFiles {
|
||||
edit.DeleteFile(file.FileNumber)
|
||||
}
|
||||
// 删除实际存在且被处理的输出层级文件
|
||||
for _, file := range existingOutputFiles {
|
||||
edit.DeleteFile(file.FileNumber)
|
||||
}
|
||||
// 删除缺失的输出层级文件（清理 MANIFEST 中的过期引用）
|
||||
for _, file := range missingOutputFiles {
|
||||
edit.DeleteFile(file.FileNumber)
|
||||
fmt.Printf("[Compaction] Removing missing file %06d.sst from MANIFEST\n", file.FileNumber)
|
||||
}
|
||||
|
||||
// 添加新文件
|
||||
for _, file := range newFiles {
|
||||
edit.AddFile(file)
|
||||
}
|
||||
|
||||
// 持久化当前的文件编号计数器（关键修复：防止重启后文件编号重用）
|
||||
edit.SetNextFileNumber(c.versionSet.GetNextFileNumber())
|
||||
|
||||
return edit, nil
|
||||
}
|
||||
|
||||
// readInputFiles 读取输入文件的所有行
|
||||
// 注意：调用者必须确保传入的文件都存在，否则会返回错误
|
||||
func (c *Compactor) readInputFiles(files []*manifest.FileMetadata) ([]*sst.Row, error) {
|
||||
var allRows []*sst.Row
|
||||
|
||||
for _, file := range files {
|
||||
sstPath := filepath.Join(c.sstDir, fmt.Sprintf("%06d.sst", file.FileNumber))
|
||||
|
||||
reader, err := sst.NewReader(sstPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("open sst %d: %w", file.FileNumber, err)
|
||||
}
|
||||
|
||||
// 获取文件中实际存在的所有 key（不能用 MinKey-MaxKey 范围遍历，因为 key 可能是稀疏的）
|
||||
keys := reader.GetAllKeys()
|
||||
for _, seq := range keys {
|
||||
row, err := reader.Get(seq)
|
||||
if err != nil {
|
||||
// 这种情况理论上不应该发生（key 来自索引），但为了安全还是处理一下
|
||||
continue
|
||||
}
|
||||
allRows = append(allRows, row)
|
||||
}
|
||||
|
||||
reader.Close()
|
||||
}
|
||||
|
||||
return allRows, nil
|
||||
}
|
||||
|
||||
// getOverlappingFiles 获取输出层级中与输入行重叠的文件
|
||||
func (c *Compactor) getOverlappingFiles(version *manifest.Version, level int, rows []*sst.Row) []*manifest.FileMetadata {
|
||||
if len(rows) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
// 找到输入行的 key range
|
||||
minKey := rows[0].Seq
|
||||
maxKey := rows[0].Seq
|
||||
for _, row := range rows {
|
||||
if row.Seq < minKey {
|
||||
minKey = row.Seq
|
||||
}
|
||||
if row.Seq > maxKey {
|
||||
maxKey = row.Seq
|
||||
}
|
||||
}
|
||||
|
||||
// 找到输出层级中重叠的文件
|
||||
var overlapping []*manifest.FileMetadata
|
||||
levelFiles := version.GetLevel(level)
|
||||
for _, file := range levelFiles {
|
||||
// 检查 key range 是否重叠
|
||||
if file.MaxKey >= minKey && file.MinKey <= maxKey {
|
||||
overlapping = append(overlapping, file)
|
||||
}
|
||||
}
|
||||
|
||||
return overlapping
|
||||
}
|
||||
|
||||
// mergeRows 合并行，去重并保留最新的记录
|
||||
func (c *Compactor) mergeRows(rows []*sst.Row) []*sst.Row {
|
||||
if len(rows) == 0 {
|
||||
return rows
|
||||
}
|
||||
|
||||
// 按 Seq 排序
|
||||
sort.Slice(rows, func(i, j int) bool {
|
||||
return rows[i].Seq < rows[j].Seq
|
||||
})
|
||||
|
||||
// 去重：保留相同 Seq 的最新记录 (Timestamp 最大的)
|
||||
merged := make([]*sst.Row, 0, len(rows))
|
||||
var lastRow *sst.Row
|
||||
|
||||
for _, row := range rows {
|
||||
if lastRow == nil || lastRow.Seq != row.Seq {
|
||||
// 新的 Seq
|
||||
merged = append(merged, row)
|
||||
lastRow = row
|
||||
} else {
|
||||
// 相同 Seq，保留 Time 更大的
|
||||
if row.Time > lastRow.Time {
|
||||
merged[len(merged)-1] = row
|
||||
lastRow = row
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return merged
|
||||
}
|
||||
|
||||
// calculateAvgRowSize 基于输入文件的 FileMetadata 计算平均行大小
|
||||
func (c *Compactor) calculateAvgRowSize(inputFiles []*manifest.FileMetadata, outputFiles []*manifest.FileMetadata) int64 {
|
||||
var totalSize int64
|
||||
var totalRows int64
|
||||
|
||||
// 统计输入文件
|
||||
for _, file := range inputFiles {
|
||||
totalSize += file.FileSize
|
||||
totalRows += file.RowCount
|
||||
}
|
||||
|
||||
// 统计输出文件
|
||||
for _, file := range outputFiles {
|
||||
totalSize += file.FileSize
|
||||
totalRows += file.RowCount
|
||||
}
|
||||
|
||||
// 计算平均值
|
||||
if totalRows == 0 {
|
||||
return 1024 // 默认 1KB
|
||||
}
|
||||
return totalSize / totalRows
|
||||
}
|
||||
|
||||
// writeOutputFiles 将合并后的行写入新的 SST 文件
|
||||
func (c *Compactor) writeOutputFiles(rows []*sst.Row, level int, version *manifest.Version, avgRowSize int64) ([]*manifest.FileMetadata, error) {
|
||||
if len(rows) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
// 根据层级动态调整文件大小目标
|
||||
// L0: 2MB (快速 flush，小文件)
|
||||
// L1: 10MB
|
||||
// L2: 50MB
|
||||
// L3: 100MB
|
||||
// L4+: 200MB
|
||||
targetFileSize := c.getTargetFileSize(level)
|
||||
|
||||
// 应用安全系数：由于压缩率、索引开销等因素，估算值可能不准确
|
||||
// 使用 80% 的目标大小作为分割点，避免实际文件超出目标过多
|
||||
targetFileSize = targetFileSize * 80 / 100
|
||||
|
||||
var newFiles []*manifest.FileMetadata
|
||||
var currentRows []*sst.Row
|
||||
var currentSize int64
|
||||
|
||||
for _, row := range rows {
|
||||
// 使用平均行大小估算（基于输入文件的统计信息）
|
||||
rowSize := avgRowSize
|
||||
|
||||
// 如果当前文件大小超过目标，写入文件
|
||||
if currentSize > 0 && currentSize+rowSize > targetFileSize {
|
||||
file, err := c.writeFile(currentRows, level, version)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
newFiles = append(newFiles, file)
|
||||
|
||||
// 重置
|
||||
currentRows = nil
|
||||
currentSize = 0
|
||||
}
|
||||
|
||||
currentRows = append(currentRows, row)
|
||||
currentSize += rowSize
|
||||
}
|
||||
|
||||
// 写入最后一个文件
|
||||
if len(currentRows) > 0 {
|
||||
file, err := c.writeFile(currentRows, level, version)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
newFiles = append(newFiles, file)
|
||||
}
|
||||
|
||||
return newFiles, nil
|
||||
}
|
||||
|
||||
// getTargetFileSize 根据层级返回目标文件大小
|
||||
func (c *Compactor) getTargetFileSize(level int) int64 {
|
||||
switch level {
|
||||
case 0:
|
||||
return 2 * 1024 * 1024 // 2MB
|
||||
case 1:
|
||||
return 10 * 1024 * 1024 // 10MB
|
||||
case 2:
|
||||
return 50 * 1024 * 1024 // 50MB
|
||||
case 3:
|
||||
return 100 * 1024 * 1024 // 100MB
|
||||
default: // L4+
|
||||
return 200 * 1024 * 1024 // 200MB
|
||||
}
|
||||
}
|
||||
|
||||
// writeFile 写入单个 SST 文件
|
||||
func (c *Compactor) writeFile(rows []*sst.Row, level int, version *manifest.Version) (*manifest.FileMetadata, error) {
|
||||
// 从 VersionSet 分配新的文件编号
|
||||
fileNumber := c.versionSet.AllocateFileNumber()
|
||||
sstPath := filepath.Join(c.sstDir, fmt.Sprintf("%06d.sst", fileNumber))
|
||||
|
||||
// 创建文件
|
||||
file, err := os.Create(sstPath)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer file.Close()
|
||||
|
||||
writer := sst.NewWriter(file)
|
||||
|
||||
// 写入所有行
|
||||
for _, row := range rows {
|
||||
err = writer.Add(row)
|
||||
if err != nil {
|
||||
os.Remove(sstPath)
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
|
||||
// 完成写入
|
||||
err = writer.Finish()
|
||||
if err != nil {
|
||||
os.Remove(sstPath)
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// 获取文件信息
|
||||
fileInfo, err := file.Stat()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// 创建 FileMetadata
|
||||
metadata := &manifest.FileMetadata{
|
||||
FileNumber: fileNumber,
|
||||
Level: level,
|
||||
FileSize: fileInfo.Size(),
|
||||
MinKey: rows[0].Seq,
|
||||
MaxKey: rows[len(rows)-1].Seq,
|
||||
RowCount: int64(len(rows)),
|
||||
}
|
||||
|
||||
return metadata, nil
|
||||
}
|
||||
444
compaction/manager.go
Normal file
444
compaction/manager.go
Normal file
@@ -0,0 +1,444 @@
|
||||
package compaction
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"code.tczkiot.com/srdb/manifest"
|
||||
)
|
||||
|
||||
// Manager 管理 Compaction 流程
|
||||
type Manager struct {
|
||||
compactor *Compactor
|
||||
versionSet *manifest.VersionSet
|
||||
sstDir string
|
||||
|
||||
// 控制后台 Compaction
|
||||
stopCh chan struct{}
|
||||
wg sync.WaitGroup
|
||||
|
||||
// Compaction 并发控制
|
||||
compactionMu sync.Mutex // 防止并发执行 compaction
|
||||
|
||||
// 统计信息
|
||||
mu sync.RWMutex
|
||||
totalCompactions int64
|
||||
lastCompactionTime time.Time
|
||||
lastFailedFile int64 // 最后失败的文件编号
|
||||
consecutiveFails int // 连续失败次数
|
||||
lastGCTime time.Time
|
||||
totalOrphansFound int64
|
||||
}
|
||||
|
||||
// NewManager 创建新的 Compaction Manager
|
||||
func NewManager(sstDir string, versionSet *manifest.VersionSet) *Manager {
|
||||
return &Manager{
|
||||
compactor: NewCompactor(sstDir, versionSet),
|
||||
versionSet: versionSet,
|
||||
sstDir: sstDir,
|
||||
stopCh: make(chan struct{}),
|
||||
}
|
||||
}
|
||||
|
||||
// GetPicker 获取 Compaction Picker
|
||||
func (m *Manager) GetPicker() *Picker {
|
||||
return m.compactor.GetPicker()
|
||||
}
|
||||
|
||||
// Start 启动后台 Compaction 和垃圾回收
|
||||
func (m *Manager) Start() {
|
||||
m.wg.Add(2)
|
||||
go m.backgroundCompaction()
|
||||
go m.backgroundGarbageCollection()
|
||||
}
|
||||
|
||||
// Stop 停止后台 Compaction
|
||||
func (m *Manager) Stop() {
|
||||
close(m.stopCh)
|
||||
m.wg.Wait()
|
||||
}
|
||||
|
||||
// backgroundCompaction 后台 Compaction 循环
|
||||
func (m *Manager) backgroundCompaction() {
|
||||
defer m.wg.Done()
|
||||
|
||||
ticker := time.NewTicker(10 * time.Second) // 每 10 秒检查一次
|
||||
defer ticker.Stop()
|
||||
|
||||
for {
|
||||
select {
|
||||
case <-m.stopCh:
|
||||
return
|
||||
case <-ticker.C:
|
||||
m.maybeCompact()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// MaybeCompact 检查是否需要 Compaction 并执行（公开方法，供外部调用）
|
||||
// 非阻塞：如果已有 compaction 在执行，直接返回
|
||||
func (m *Manager) MaybeCompact() {
|
||||
// 尝试获取锁，如果已有 compaction 在执行，直接返回
|
||||
if !m.compactionMu.TryLock() {
|
||||
return
|
||||
}
|
||||
defer m.compactionMu.Unlock()
|
||||
|
||||
m.doCompact()
|
||||
}
|
||||
|
||||
// maybeCompact 内部使用的阻塞版本（后台 goroutine 使用）
|
||||
func (m *Manager) maybeCompact() {
|
||||
m.compactionMu.Lock()
|
||||
defer m.compactionMu.Unlock()
|
||||
|
||||
m.doCompact()
|
||||
}
|
||||
|
||||
// doCompact 实际执行 compaction 的逻辑（必须在持有 compactionMu 时调用）
|
||||
// 支持并发执行多个层级的 compaction
|
||||
func (m *Manager) doCompact() {
|
||||
// 获取当前版本
|
||||
version := m.versionSet.GetCurrent()
|
||||
if version == nil {
|
||||
return
|
||||
}
|
||||
|
||||
// 获取所有需要 Compaction 的任务（已按优先级排序）
|
||||
picker := m.compactor.GetPicker()
|
||||
tasks := picker.PickCompaction(version)
|
||||
if len(tasks) == 0 {
|
||||
// 输出诊断信息
|
||||
m.printCompactionStats(version, picker)
|
||||
return
|
||||
}
|
||||
|
||||
fmt.Printf("[Compaction] Found %d tasks to execute\n", len(tasks))
|
||||
|
||||
// 并发执行所有任务
|
||||
successCount := 0
|
||||
for _, task := range tasks {
|
||||
// 检查是否是上次失败的文件（防止无限重试）
|
||||
if len(task.InputFiles) > 0 {
|
||||
firstFile := task.InputFiles[0].FileNumber
|
||||
m.mu.Lock()
|
||||
if m.lastFailedFile == firstFile && m.consecutiveFails >= 3 {
|
||||
fmt.Printf("[Compaction] Skipping L%d file %d (failed %d times)\n",
|
||||
task.Level, firstFile, m.consecutiveFails)
|
||||
m.consecutiveFails = 0
|
||||
m.lastFailedFile = 0
|
||||
m.mu.Unlock()
|
||||
continue
|
||||
}
|
||||
m.mu.Unlock()
|
||||
}
|
||||
|
||||
// 获取最新版本（每个任务执行前）
|
||||
currentVersion := m.versionSet.GetCurrent()
|
||||
if currentVersion == nil {
|
||||
continue
|
||||
}
|
||||
|
||||
// 执行 Compaction
|
||||
fmt.Printf("[Compaction] Starting: L%d -> L%d, files: %d\n",
|
||||
task.Level, task.OutputLevel, len(task.InputFiles))
|
||||
|
||||
err := m.DoCompactionWithVersion(task, currentVersion)
|
||||
if err != nil {
|
||||
fmt.Printf("[Compaction] Failed L%d -> L%d: %v\n", task.Level, task.OutputLevel, err)
|
||||
|
||||
// 记录失败信息
|
||||
if len(task.InputFiles) > 0 {
|
||||
firstFile := task.InputFiles[0].FileNumber
|
||||
m.mu.Lock()
|
||||
if m.lastFailedFile == firstFile {
|
||||
m.consecutiveFails++
|
||||
} else {
|
||||
m.lastFailedFile = firstFile
|
||||
m.consecutiveFails = 1
|
||||
}
|
||||
m.mu.Unlock()
|
||||
}
|
||||
} else {
|
||||
fmt.Printf("[Compaction] Completed: L%d -> L%d\n", task.Level, task.OutputLevel)
|
||||
successCount++
|
||||
|
||||
// 清除失败计数
|
||||
m.mu.Lock()
|
||||
m.consecutiveFails = 0
|
||||
m.lastFailedFile = 0
|
||||
m.mu.Unlock()
|
||||
}
|
||||
}
|
||||
|
||||
fmt.Printf("[Compaction] Batch completed: %d/%d tasks succeeded\n", successCount, len(tasks))
|
||||
}
|
||||
|
||||
// printCompactionStats 输出 Compaction 统计信息（每分钟一次）
|
||||
func (m *Manager) printCompactionStats(version *manifest.Version, picker *Picker) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
|
||||
// 限制输出频率：每 60 秒输出一次
|
||||
if time.Since(m.lastCompactionTime) < 60*time.Second {
|
||||
return
|
||||
}
|
||||
m.lastCompactionTime = time.Now()
|
||||
|
||||
fmt.Println("[Compaction] Status check:")
|
||||
for level := 0; level < 7; level++ {
|
||||
files := version.GetLevel(level)
|
||||
if len(files) == 0 {
|
||||
continue
|
||||
}
|
||||
|
||||
totalSize := int64(0)
|
||||
for _, f := range files {
|
||||
totalSize += f.FileSize
|
||||
}
|
||||
|
||||
score := picker.GetLevelScore(version, level)
|
||||
fmt.Printf(" L%d: %d files, %.2f MB, score: %.2f\n",
|
||||
level, len(files), float64(totalSize)/(1024*1024), score)
|
||||
}
|
||||
}
|
||||
|
||||
// DoCompactionWithVersion 使用指定的版本执行 Compaction
|
||||
func (m *Manager) DoCompactionWithVersion(task *CompactionTask, version *manifest.Version) error {
|
||||
if version == nil {
|
||||
return fmt.Errorf("version is nil")
|
||||
}
|
||||
|
||||
// 执行 Compaction（使用传入的 version，而不是重新获取）
|
||||
edit, err := m.compactor.DoCompaction(task, version)
|
||||
if err != nil {
|
||||
return fmt.Errorf("compaction failed: %w", err)
|
||||
}
|
||||
|
||||
// 如果 edit 为 nil，说明所有文件都已经不存在，无需应用变更
|
||||
if edit == nil {
|
||||
fmt.Printf("[Compaction] No changes needed (files already removed)\n")
|
||||
return nil
|
||||
}
|
||||
|
||||
// 应用 VersionEdit
|
||||
err = m.versionSet.LogAndApply(edit)
|
||||
if err != nil {
|
||||
// LogAndApply 失败，清理已写入的新 SST 文件（防止孤儿文件）
|
||||
fmt.Printf("[Compaction] LogAndApply failed, cleaning up new files: %v\n", err)
|
||||
m.cleanupNewFiles(edit)
|
||||
return fmt.Errorf("apply version edit: %w", err)
|
||||
}
|
||||
|
||||
// LogAndApply 成功后，删除废弃的 SST 文件
|
||||
m.deleteObsoleteFiles(edit)
|
||||
|
||||
// 更新统计信息
|
||||
m.mu.Lock()
|
||||
m.totalCompactions++
|
||||
m.lastCompactionTime = time.Now()
|
||||
m.mu.Unlock()
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// DoCompaction 执行一次 Compaction（兼容旧接口）
|
||||
func (m *Manager) DoCompaction(task *CompactionTask) error {
|
||||
// 获取当前版本
|
||||
version := m.versionSet.GetCurrent()
|
||||
if version == nil {
|
||||
return fmt.Errorf("no current version")
|
||||
}
|
||||
|
||||
return m.DoCompactionWithVersion(task, version)
|
||||
}
|
||||
|
||||
// cleanupNewFiles 清理 LogAndApply 失败后的新文件（防止孤儿文件）
|
||||
func (m *Manager) cleanupNewFiles(edit *manifest.VersionEdit) {
|
||||
if edit == nil {
|
||||
return
|
||||
}
|
||||
|
||||
fmt.Printf("[Compaction] Cleaning up %d new files after LogAndApply failure\n", len(edit.AddedFiles))
|
||||
|
||||
// 删除新创建的文件
|
||||
for _, file := range edit.AddedFiles {
|
||||
sstPath := filepath.Join(m.sstDir, fmt.Sprintf("%06d.sst", file.FileNumber))
|
||||
err := os.Remove(sstPath)
|
||||
if err != nil {
|
||||
fmt.Printf("[Compaction] Failed to cleanup new file %06d.sst: %v\n", file.FileNumber, err)
|
||||
} else {
|
||||
fmt.Printf("[Compaction] Cleaned up new file %06d.sst\n", file.FileNumber)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// deleteObsoleteFiles 删除废弃的 SST 文件
|
||||
func (m *Manager) deleteObsoleteFiles(edit *manifest.VersionEdit) {
|
||||
if edit == nil {
|
||||
fmt.Printf("[Compaction] deleteObsoleteFiles: edit is nil\n")
|
||||
return
|
||||
}
|
||||
|
||||
fmt.Printf("[Compaction] deleteObsoleteFiles: %d files to delete\n", len(edit.DeletedFiles))
|
||||
|
||||
// 删除被标记为删除的文件
|
||||
for _, fileNum := range edit.DeletedFiles {
|
||||
sstPath := filepath.Join(m.sstDir, fmt.Sprintf("%06d.sst", fileNum))
|
||||
err := os.Remove(sstPath)
|
||||
if err != nil {
|
||||
// 删除失败只记录日志，不影响 compaction 流程
|
||||
// 后台垃圾回收器会重试
|
||||
fmt.Printf("[Compaction] Failed to delete obsolete file %06d.sst: %v\n", fileNum, err)
|
||||
} else {
|
||||
fmt.Printf("[Compaction] Deleted obsolete file %06d.sst\n", fileNum)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TriggerCompaction 手动触发一次 Compaction（所有需要的层级）
|
||||
func (m *Manager) TriggerCompaction() error {
|
||||
version := m.versionSet.GetCurrent()
|
||||
if version == nil {
|
||||
return fmt.Errorf("no current version")
|
||||
}
|
||||
|
||||
picker := m.compactor.GetPicker()
|
||||
tasks := picker.PickCompaction(version)
|
||||
if len(tasks) == 0 {
|
||||
return nil // 不需要 Compaction
|
||||
}
|
||||
|
||||
// 依次执行所有任务
|
||||
for _, task := range tasks {
|
||||
currentVersion := m.versionSet.GetCurrent()
|
||||
if err := m.DoCompactionWithVersion(task, currentVersion); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// GetStats 获取 Compaction 统计信息
|
||||
func (m *Manager) GetStats() map[string]interface{} {
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
|
||||
return map[string]interface{}{
|
||||
"total_compactions": m.totalCompactions,
|
||||
"last_compaction_time": m.lastCompactionTime,
|
||||
}
|
||||
}
|
||||
|
||||
// GetLevelStats 获取每层的统计信息
|
||||
func (m *Manager) GetLevelStats() []map[string]interface{} {
|
||||
version := m.versionSet.GetCurrent()
|
||||
if version == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
picker := m.compactor.GetPicker()
|
||||
stats := make([]map[string]interface{}, manifest.NumLevels)
|
||||
|
||||
for level := 0; level < manifest.NumLevels; level++ {
|
||||
files := version.GetLevel(level)
|
||||
totalSize := int64(0)
|
||||
for _, file := range files {
|
||||
totalSize += file.FileSize
|
||||
}
|
||||
|
||||
stats[level] = map[string]interface{}{
|
||||
"level": level,
|
||||
"file_count": len(files),
|
||||
"total_size": totalSize,
|
||||
"score": picker.GetLevelScore(version, level),
|
||||
}
|
||||
}
|
||||
|
||||
return stats
|
||||
}
|
||||
|
||||
// backgroundGarbageCollection 后台垃圾回收循环
|
||||
func (m *Manager) backgroundGarbageCollection() {
|
||||
defer m.wg.Done()
|
||||
|
||||
ticker := time.NewTicker(5 * time.Minute) // 每 5 分钟检查一次
|
||||
defer ticker.Stop()
|
||||
|
||||
for {
|
||||
select {
|
||||
case <-m.stopCh:
|
||||
return
|
||||
case <-ticker.C:
|
||||
m.collectOrphanFiles()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// collectOrphanFiles 收集并删除孤儿 SST 文件
|
||||
func (m *Manager) collectOrphanFiles() {
|
||||
// 1. 获取当前版本中的所有活跃文件
|
||||
version := m.versionSet.GetCurrent()
|
||||
if version == nil {
|
||||
return
|
||||
}
|
||||
|
||||
activeFiles := make(map[int64]bool)
|
||||
for level := 0; level < manifest.NumLevels; level++ {
|
||||
files := version.GetLevel(level)
|
||||
for _, file := range files {
|
||||
activeFiles[file.FileNumber] = true
|
||||
}
|
||||
}
|
||||
|
||||
// 2. 扫描 SST 目录中的所有文件
|
||||
pattern := filepath.Join(m.sstDir, "*.sst")
|
||||
sstFiles, err := filepath.Glob(pattern)
|
||||
if err != nil {
|
||||
fmt.Printf("[GC] Failed to scan SST directory: %v\n", err)
|
||||
return
|
||||
}
|
||||
|
||||
// 3. 找出孤儿文件并删除
|
||||
orphanCount := 0
|
||||
for _, sstPath := range sstFiles {
|
||||
// 提取文件编号
|
||||
var fileNum int64
|
||||
_, err := fmt.Sscanf(filepath.Base(sstPath), "%d.sst", &fileNum)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
|
||||
// 检查是否是活跃文件
|
||||
if !activeFiles[fileNum] {
|
||||
// 这是孤儿文件，删除它
|
||||
err := os.Remove(sstPath)
|
||||
if err != nil {
|
||||
fmt.Printf("[GC] Failed to delete orphan file %06d.sst: %v\n", fileNum, err)
|
||||
} else {
|
||||
fmt.Printf("[GC] Deleted orphan file %06d.sst\n", fileNum)
|
||||
orphanCount++
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 4. 更新统计信息
|
||||
m.mu.Lock()
|
||||
m.lastGCTime = time.Now()
|
||||
m.totalOrphansFound += int64(orphanCount)
|
||||
m.mu.Unlock()
|
||||
|
||||
if orphanCount > 0 {
|
||||
fmt.Printf("[GC] Completed: cleaned up %d orphan files (total: %d)\n", orphanCount, m.totalOrphansFound)
|
||||
}
|
||||
}
|
||||
|
||||
// CleanupOrphanFiles 手动触发孤儿文件清理（可在启动时调用）
|
||||
func (m *Manager) CleanupOrphanFiles() {
|
||||
fmt.Println("[GC] Manual cleanup triggered")
|
||||
m.collectOrphanFiles()
|
||||
}
|
||||
285
compaction/picker.go
Normal file
285
compaction/picker.go
Normal file
@@ -0,0 +1,285 @@
|
||||
package compaction
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
|
||||
"code.tczkiot.com/srdb/manifest"
|
||||
)
|
||||
|
||||
// CompactionTask 表示一个 Compaction 任务
|
||||
type CompactionTask struct {
|
||||
Level int // 源层级
|
||||
InputFiles []*manifest.FileMetadata // 需要合并的输入文件
|
||||
OutputLevel int // 输出层级
|
||||
}
|
||||
|
||||
// Picker 负责选择需要 Compaction 的文件
|
||||
type Picker struct {
|
||||
// Level 大小限制 (字节)
|
||||
levelSizeLimits [manifest.NumLevels]int64
|
||||
|
||||
// Level 文件数量限制
|
||||
levelFileLimits [manifest.NumLevels]int
|
||||
}
|
||||
|
||||
// NewPicker 创建新的 Compaction Picker
|
||||
func NewPicker() *Picker {
|
||||
p := &Picker{}
|
||||
|
||||
// 设置每层的大小限制 (指数增长)
|
||||
// L0: 10MB, L1: 100MB, L2: 1GB, L3: 10GB, L4: 100GB, L5: 1TB, L6: 无限制
|
||||
p.levelSizeLimits[0] = 10 * 1024 * 1024 // 10MB
|
||||
p.levelSizeLimits[1] = 100 * 1024 * 1024 // 100MB
|
||||
p.levelSizeLimits[2] = 1024 * 1024 * 1024 // 1GB
|
||||
p.levelSizeLimits[3] = 10 * 1024 * 1024 * 1024 // 10GB
|
||||
p.levelSizeLimits[4] = 100 * 1024 * 1024 * 1024 // 100GB
|
||||
p.levelSizeLimits[5] = 1024 * 1024 * 1024 * 1024 // 1TB
|
||||
p.levelSizeLimits[6] = 0 // 无限制
|
||||
|
||||
// 设置每层的文件数量限制
|
||||
// L0 特殊处理：文件数量限制为 4 (当有4个或更多文件时触发 compaction)
|
||||
p.levelFileLimits[0] = 4
|
||||
// L1-L6: 不限制文件数量，只限制总大小
|
||||
for i := 1; i < manifest.NumLevels; i++ {
|
||||
p.levelFileLimits[i] = 0 // 0 表示不限制
|
||||
}
|
||||
|
||||
return p
|
||||
}
|
||||
|
||||
// PickCompaction 选择需要 Compaction 的任务（支持多任务并发）
|
||||
// 返回空切片表示当前不需要 Compaction
|
||||
func (p *Picker) PickCompaction(version *manifest.Version) []*CompactionTask {
|
||||
tasks := make([]*CompactionTask, 0)
|
||||
|
||||
// 1. 检查 L0 (基于文件数量)
|
||||
if task := p.pickL0Compaction(version); task != nil {
|
||||
tasks = append(tasks, task)
|
||||
}
|
||||
|
||||
// 2. 检查 L1-L5 (基于大小)
|
||||
for level := 1; level < manifest.NumLevels-1; level++ {
|
||||
if task := p.pickLevelCompaction(version, level); task != nil {
|
||||
tasks = append(tasks, task)
|
||||
}
|
||||
}
|
||||
|
||||
// 3. 按优先级排序（score 越高越优先）
|
||||
if len(tasks) > 1 {
|
||||
p.sortTasksByPriority(tasks, version)
|
||||
}
|
||||
|
||||
return tasks
|
||||
}
|
||||
|
||||
// sortTasksByPriority 按优先级对任务排序（score 从高到低）
|
||||
func (p *Picker) sortTasksByPriority(tasks []*CompactionTask, version *manifest.Version) {
|
||||
// 简单的冒泡排序（任务数量通常很少，< 7）
|
||||
for i := 0; i < len(tasks)-1; i++ {
|
||||
for j := i + 1; j < len(tasks); j++ {
|
||||
scoreI := p.GetLevelScore(version, tasks[i].Level)
|
||||
scoreJ := p.GetLevelScore(version, tasks[j].Level)
|
||||
if scoreJ > scoreI {
|
||||
tasks[i], tasks[j] = tasks[j], tasks[i]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// pickL0Compaction 选择 L0 的 Compaction 任务
|
||||
// L0 特殊：文件可能有重叠的 key range，需要全部合并
|
||||
func (p *Picker) pickL0Compaction(version *manifest.Version) *CompactionTask {
|
||||
l0Files := version.GetLevel(0)
|
||||
if len(l0Files) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
// 计算 L0 总大小
|
||||
totalSize := int64(0)
|
||||
for _, file := range l0Files {
|
||||
totalSize += file.FileSize
|
||||
}
|
||||
|
||||
// 检查是否需要 Compaction（同时考虑文件数量和总大小）
|
||||
// 1. 文件数量超过限制（避免读放大：每次读取需要检查太多文件）
|
||||
// 2. 总大小超过限制（避免 L0 占用过多空间）
|
||||
needCompaction := false
|
||||
if p.levelFileLimits[0] > 0 && len(l0Files) >= p.levelFileLimits[0] {
|
||||
needCompaction = true
|
||||
}
|
||||
if p.levelSizeLimits[0] > 0 && totalSize >= p.levelSizeLimits[0] {
|
||||
needCompaction = true
|
||||
}
|
||||
|
||||
if !needCompaction {
|
||||
return nil
|
||||
}
|
||||
|
||||
// L0 → L1 Compaction
|
||||
// 选择所有 L0 文件（因为 key range 可能重叠）
|
||||
return &CompactionTask{
|
||||
Level: 0,
|
||||
InputFiles: l0Files,
|
||||
OutputLevel: 1,
|
||||
}
|
||||
}
|
||||
|
||||
// pickLevelCompaction 选择 L1-L5 的 Compaction 任务
|
||||
// L1+ 的文件 key range 不重叠，可以选择多个不重叠的文件
|
||||
func (p *Picker) pickLevelCompaction(version *manifest.Version, level int) *CompactionTask {
|
||||
if level < 1 || level >= manifest.NumLevels-1 {
|
||||
return nil
|
||||
}
|
||||
|
||||
files := version.GetLevel(level)
|
||||
if len(files) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
// 计算当前层级的总大小
|
||||
totalSize := int64(0)
|
||||
for _, file := range files {
|
||||
totalSize += file.FileSize
|
||||
}
|
||||
|
||||
// 检查是否超过大小限制
|
||||
if totalSize < p.levelSizeLimits[level] {
|
||||
return nil
|
||||
}
|
||||
|
||||
// 改进策略：根据层级压力动态调整选择策略
|
||||
// 1. 计算当前层级的压力（超过限制的倍数）
|
||||
pressure := float64(totalSize) / float64(p.levelSizeLimits[level])
|
||||
|
||||
// 2. 根据压力确定目标大小和文件数量限制
|
||||
targetSize := p.getTargetCompactionSize(level + 1)
|
||||
maxFiles := 10 // 默认最多 10 个文件
|
||||
|
||||
if pressure >= 10.0 {
|
||||
// 压力极高（超过 10 倍）：选择更多文件，增大目标
|
||||
maxFiles = 100
|
||||
targetSize *= 5
|
||||
fmt.Printf("[Compaction] L%d pressure: %.1fx (CRITICAL) - selecting up to %d files, target: %s\n",
|
||||
level, pressure, maxFiles, formatBytes(targetSize))
|
||||
} else if pressure >= 5.0 {
|
||||
// 压力很高（超过 5 倍）
|
||||
maxFiles = 50
|
||||
targetSize *= 3
|
||||
fmt.Printf("[Compaction] L%d pressure: %.1fx (HIGH) - selecting up to %d files, target: %s\n",
|
||||
level, pressure, maxFiles, formatBytes(targetSize))
|
||||
} else if pressure >= 2.0 {
|
||||
// 压力较高（超过 2 倍）
|
||||
maxFiles = 20
|
||||
targetSize *= 2
|
||||
fmt.Printf("[Compaction] L%d pressure: %.1fx (ELEVATED) - selecting up to %d files, target: %s\n",
|
||||
level, pressure, maxFiles, formatBytes(targetSize))
|
||||
}
|
||||
|
||||
// 选择文件，直到累计大小接近目标
|
||||
selectedFiles := make([]*manifest.FileMetadata, 0)
|
||||
currentSize := int64(0)
|
||||
|
||||
for _, file := range files {
|
||||
selectedFiles = append(selectedFiles, file)
|
||||
currentSize += file.FileSize
|
||||
|
||||
// 如果已经达到目标大小，停止选择
|
||||
if currentSize >= targetSize {
|
||||
break
|
||||
}
|
||||
|
||||
// 达到文件数量限制
|
||||
if len(selectedFiles) >= maxFiles {
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
return &CompactionTask{
|
||||
Level: level,
|
||||
InputFiles: selectedFiles,
|
||||
OutputLevel: level + 1,
|
||||
}
|
||||
}
|
||||
|
||||
// getTargetCompactionSize 根据层级返回建议的 compaction 大小
|
||||
func (p *Picker) getTargetCompactionSize(level int) int64 {
|
||||
switch level {
|
||||
case 0:
|
||||
return 2 * 1024 * 1024 // 2MB
|
||||
case 1:
|
||||
return 10 * 1024 * 1024 // 10MB
|
||||
case 2:
|
||||
return 50 * 1024 * 1024 // 50MB
|
||||
case 3:
|
||||
return 100 * 1024 * 1024 // 100MB
|
||||
default: // L4+
|
||||
return 200 * 1024 * 1024 // 200MB
|
||||
}
|
||||
}
|
||||
|
||||
// ShouldCompact 判断是否需要 Compaction
|
||||
func (p *Picker) ShouldCompact(version *manifest.Version) bool {
|
||||
tasks := p.PickCompaction(version)
|
||||
return len(tasks) > 0
|
||||
}
|
||||
|
||||
// GetLevelScore 获取每层的 Compaction 得分 (用于优先级排序)
|
||||
// 得分越高，越需要 Compaction
|
||||
func (p *Picker) GetLevelScore(version *manifest.Version, level int) float64 {
|
||||
if level < 0 || level >= manifest.NumLevels {
|
||||
return 0
|
||||
}
|
||||
|
||||
files := version.GetLevel(level)
|
||||
|
||||
// L0 同时考虑文件数量和总大小，取较大值作为得分
|
||||
if level == 0 {
|
||||
scoreByCount := float64(0)
|
||||
scoreBySize := float64(0)
|
||||
|
||||
if p.levelFileLimits[0] > 0 {
|
||||
scoreByCount = float64(len(files)) / float64(p.levelFileLimits[0])
|
||||
}
|
||||
|
||||
if p.levelSizeLimits[0] > 0 {
|
||||
totalSize := int64(0)
|
||||
for _, file := range files {
|
||||
totalSize += file.FileSize
|
||||
}
|
||||
scoreBySize = float64(totalSize) / float64(p.levelSizeLimits[0])
|
||||
}
|
||||
|
||||
// 返回两者中的较大值（哪个维度更紧迫）
|
||||
if scoreByCount > scoreBySize {
|
||||
return scoreByCount
|
||||
}
|
||||
return scoreBySize
|
||||
}
|
||||
|
||||
// L1+ 基于总大小
|
||||
if p.levelSizeLimits[level] == 0 {
|
||||
return 0
|
||||
}
|
||||
|
||||
totalSize := int64(0)
|
||||
for _, file := range files {
|
||||
totalSize += file.FileSize
|
||||
}
|
||||
|
||||
return float64(totalSize) / float64(p.levelSizeLimits[level])
|
||||
}
|
||||
|
||||
// formatBytes 格式化字节大小显示
|
||||
func formatBytes(bytes int64) string {
|
||||
const unit = 1024
|
||||
if bytes < unit {
|
||||
return fmt.Sprintf("%d B", bytes)
|
||||
}
|
||||
div, exp := int64(unit), 0
|
||||
for n := bytes / unit; n >= unit; n /= unit {
|
||||
div *= unit
|
||||
exp++
|
||||
}
|
||||
units := []string{"KB", "MB", "GB", "TB"}
|
||||
return fmt.Sprintf("%.2f %s", float64(bytes)/float64(div), units[exp])
|
||||
}
|
||||
257
database.go
Normal file
257
database.go
Normal file
@@ -0,0 +1,257 @@
|
||||
package srdb
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
)
|
||||
|
||||
// Database 数据库，管理多个表
|
||||
type Database struct {
|
||||
// 数据库目录
|
||||
dir string
|
||||
|
||||
// 所有表
|
||||
tables map[string]*Table
|
||||
|
||||
// 元数据
|
||||
metadata *Metadata
|
||||
|
||||
// 锁
|
||||
mu sync.RWMutex
|
||||
}
|
||||
|
||||
// Metadata 数据库元数据
|
||||
type Metadata struct {
|
||||
Version int `json:"version"`
|
||||
Tables []TableInfo `json:"tables"`
|
||||
}
|
||||
|
||||
// TableInfo 表信息
|
||||
type TableInfo struct {
|
||||
Name string `json:"name"`
|
||||
Dir string `json:"dir"`
|
||||
CreatedAt int64 `json:"created_at"`
|
||||
}
|
||||
|
||||
// Open 打开数据库
|
||||
func Open(dir string) (*Database, error) {
|
||||
// 创建目录
|
||||
err := os.MkdirAll(dir, 0755)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
db := &Database{
|
||||
dir: dir,
|
||||
tables: make(map[string]*Table),
|
||||
}
|
||||
|
||||
// 加载元数据
|
||||
err = db.loadMetadata()
|
||||
if err != nil {
|
||||
// 如果元数据不存在，创建新的
|
||||
db.metadata = &Metadata{
|
||||
Version: 1,
|
||||
Tables: make([]TableInfo, 0),
|
||||
}
|
||||
err = db.saveMetadata()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
|
||||
// 恢复所有表
|
||||
err = db.recoverTables()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return db, nil
|
||||
}
|
||||
|
||||
// loadMetadata 加载元数据
|
||||
func (db *Database) loadMetadata() error {
|
||||
metaPath := filepath.Join(db.dir, "database.meta")
|
||||
data, err := os.ReadFile(metaPath)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
db.metadata = &Metadata{}
|
||||
return json.Unmarshal(data, db.metadata)
|
||||
}
|
||||
|
||||
// saveMetadata 保存元数据
|
||||
func (db *Database) saveMetadata() error {
|
||||
metaPath := filepath.Join(db.dir, "database.meta")
|
||||
data, err := json.MarshalIndent(db.metadata, "", " ")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// 原子性写入
|
||||
tmpPath := metaPath + ".tmp"
|
||||
err = os.WriteFile(tmpPath, data, 0644)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return os.Rename(tmpPath, metaPath)
|
||||
}
|
||||
|
||||
// recoverTables 恢复所有表
|
||||
func (db *Database) recoverTables() error {
|
||||
var failedTables []string
|
||||
|
||||
for _, tableInfo := range db.metadata.Tables {
|
||||
// FIXME: 是否需要校验 tableInfo.Dir ?
|
||||
table, err := openTable(tableInfo.Name, db)
|
||||
if err != nil {
|
||||
// 记录失败的表，但继续恢复其他表
|
||||
failedTables = append(failedTables, tableInfo.Name)
|
||||
fmt.Printf("[WARNING] Failed to open table %s: %v\n", tableInfo.Name, err)
|
||||
fmt.Printf("[WARNING] Table %s will be skipped. You may need to drop and recreate it.\n", tableInfo.Name)
|
||||
continue
|
||||
}
|
||||
db.tables[tableInfo.Name] = table
|
||||
}
|
||||
|
||||
// 如果有失败的表，输出汇总信息
|
||||
if len(failedTables) > 0 {
|
||||
fmt.Printf("[WARNING] %d table(s) failed to recover: %v\n", len(failedTables), failedTables)
|
||||
fmt.Printf("[WARNING] To fix: Delete the corrupted table directory and restart.\n")
|
||||
fmt.Printf("[WARNING] Example: rm -rf %s/<table_name>\n", db.dir)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// CreateTable 创建表
|
||||
func (db *Database) CreateTable(name string, schema *Schema) (*Table, error) {
|
||||
db.mu.Lock()
|
||||
defer db.mu.Unlock()
|
||||
|
||||
// 检查表是否已存在
|
||||
if _, exists := db.tables[name]; exists {
|
||||
return nil, fmt.Errorf("table %s already exists", name)
|
||||
}
|
||||
|
||||
// 创建表
|
||||
table, err := createTable(name, schema, db)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// 添加到 tables map
|
||||
db.tables[name] = table
|
||||
|
||||
// 更新元数据
|
||||
db.metadata.Tables = append(db.metadata.Tables, TableInfo{
|
||||
Name: name,
|
||||
Dir: name,
|
||||
CreatedAt: table.createdAt,
|
||||
})
|
||||
|
||||
err = db.saveMetadata()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return table, nil
|
||||
}
|
||||
|
||||
// GetTable 获取表
|
||||
func (db *Database) GetTable(name string) (*Table, error) {
|
||||
db.mu.RLock()
|
||||
defer db.mu.RUnlock()
|
||||
|
||||
table, exists := db.tables[name]
|
||||
if !exists {
|
||||
return nil, fmt.Errorf("table %s not found", name)
|
||||
}
|
||||
|
||||
return table, nil
|
||||
}
|
||||
|
||||
// DropTable 删除表
|
||||
func (db *Database) DropTable(name string) error {
|
||||
db.mu.Lock()
|
||||
defer db.mu.Unlock()
|
||||
|
||||
// 检查表是否存在
|
||||
table, exists := db.tables[name]
|
||||
if !exists {
|
||||
return fmt.Errorf("table %s not found", name)
|
||||
}
|
||||
|
||||
// 关闭表
|
||||
err := table.Close()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// 从 map 中移除
|
||||
delete(db.tables, name)
|
||||
|
||||
// 删除表目录
|
||||
tableDir := filepath.Join(db.dir, name)
|
||||
err = os.RemoveAll(tableDir)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// 更新元数据
|
||||
newTables := make([]TableInfo, 0)
|
||||
for _, info := range db.metadata.Tables {
|
||||
if info.Name != name {
|
||||
newTables = append(newTables, info)
|
||||
}
|
||||
}
|
||||
db.metadata.Tables = newTables
|
||||
|
||||
return db.saveMetadata()
|
||||
}
|
||||
|
||||
// ListTables 列出所有表
|
||||
func (db *Database) ListTables() []string {
|
||||
db.mu.RLock()
|
||||
defer db.mu.RUnlock()
|
||||
|
||||
tables := make([]string, 0, len(db.tables))
|
||||
for name := range db.tables {
|
||||
tables = append(tables, name)
|
||||
}
|
||||
return tables
|
||||
}
|
||||
|
||||
// Close 关闭数据库
|
||||
func (db *Database) Close() error {
|
||||
db.mu.Lock()
|
||||
defer db.mu.Unlock()
|
||||
|
||||
// 关闭所有表
|
||||
for _, table := range db.tables {
|
||||
err := table.Close()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// GetAllTablesInfo 获取所有表的信息（用于 WebUI）
|
||||
func (db *Database) GetAllTablesInfo() map[string]*Table {
|
||||
db.mu.RLock()
|
||||
defer db.mu.RUnlock()
|
||||
|
||||
// 返回副本以避免并发问题
|
||||
result := make(map[string]*Table, len(db.tables))
|
||||
for k, v := range db.tables {
|
||||
result[k] = v
|
||||
}
|
||||
return result
|
||||
}
|
||||
259
database_test.go
Normal file
259
database_test.go
Normal file
@@ -0,0 +1,259 @@
|
||||
package srdb
|
||||
|
||||
import (
|
||||
"os"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestDatabaseBasic(t *testing.T) {
|
||||
dir := "./test_db"
|
||||
os.RemoveAll(dir)
|
||||
defer os.RemoveAll(dir)
|
||||
|
||||
// 打开数据库
|
||||
db, err := Open(dir)
|
||||
if err != nil {
|
||||
t.Fatalf("Open failed: %v", err)
|
||||
}
|
||||
defer db.Close()
|
||||
|
||||
// 检查初始状态
|
||||
tables := db.ListTables()
|
||||
if len(tables) != 0 {
|
||||
t.Errorf("Expected 0 tables, got %d", len(tables))
|
||||
}
|
||||
|
||||
t.Log("Database basic test passed!")
|
||||
}
|
||||
|
||||
func TestCreateTable(t *testing.T) {
|
||||
dir := "./test_db_create"
|
||||
os.RemoveAll(dir)
|
||||
defer os.RemoveAll(dir)
|
||||
|
||||
db, err := Open(dir)
|
||||
if err != nil {
|
||||
t.Fatalf("Open failed: %v", err)
|
||||
}
|
||||
defer db.Close()
|
||||
|
||||
// 创建 Schema
|
||||
userSchema := NewSchema("users", []Field{
|
||||
{Name: "name", Type: FieldTypeString, Indexed: true, Comment: "用户名"},
|
||||
{Name: "age", Type: FieldTypeInt64, Indexed: true, Comment: "年龄"},
|
||||
})
|
||||
|
||||
// 创建表
|
||||
usersTable, err := db.CreateTable("users", userSchema)
|
||||
if err != nil {
|
||||
t.Fatalf("CreateTable failed: %v", err)
|
||||
}
|
||||
|
||||
if usersTable.GetName() != "users" {
|
||||
t.Errorf("Expected table name 'users', got '%s'", usersTable.GetName())
|
||||
}
|
||||
|
||||
// 检查表列表
|
||||
tables := db.ListTables()
|
||||
if len(tables) != 1 {
|
||||
t.Errorf("Expected 1 table, got %d", len(tables))
|
||||
}
|
||||
|
||||
t.Log("Create table test passed!")
|
||||
}
|
||||
|
||||
func TestMultipleTables(t *testing.T) {
|
||||
dir := "./test_db_multiple"
|
||||
os.RemoveAll(dir)
|
||||
defer os.RemoveAll(dir)
|
||||
|
||||
db, err := Open(dir)
|
||||
if err != nil {
|
||||
t.Fatalf("Open failed: %v", err)
|
||||
}
|
||||
defer db.Close()
|
||||
|
||||
// 创建多个表
|
||||
userSchema := NewSchema("users", []Field{
|
||||
{Name: "name", Type: FieldTypeString, Indexed: true, Comment: "用户名"},
|
||||
{Name: "age", Type: FieldTypeInt64, Indexed: true, Comment: "年龄"},
|
||||
})
|
||||
|
||||
orderSchema := NewSchema("orders", []Field{
|
||||
{Name: "order_id", Type: FieldTypeString, Indexed: true, Comment: "订单ID"},
|
||||
{Name: "amount", Type: FieldTypeInt64, Indexed: true, Comment: "金额"},
|
||||
})
|
||||
|
||||
_, err = db.CreateTable("users", userSchema)
|
||||
if err != nil {
|
||||
t.Fatalf("CreateTable users failed: %v", err)
|
||||
}
|
||||
|
||||
_, err = db.CreateTable("orders", orderSchema)
|
||||
if err != nil {
|
||||
t.Fatalf("CreateTable orders failed: %v", err)
|
||||
}
|
||||
|
||||
// 检查表列表
|
||||
tables := db.ListTables()
|
||||
if len(tables) != 2 {
|
||||
t.Errorf("Expected 2 tables, got %d", len(tables))
|
||||
}
|
||||
|
||||
t.Log("Multiple tables test passed!")
|
||||
}
|
||||
|
||||
func TestTableOperations(t *testing.T) {
|
||||
dir := "./test_db_ops"
|
||||
os.RemoveAll(dir)
|
||||
defer os.RemoveAll(dir)
|
||||
|
||||
db, err := Open(dir)
|
||||
if err != nil {
|
||||
t.Fatalf("Open failed: %v", err)
|
||||
}
|
||||
defer db.Close()
|
||||
|
||||
// 创建表
|
||||
userSchema := NewSchema("users", []Field{
|
||||
{Name: "name", Type: FieldTypeString, Indexed: true, Comment: "用户名"},
|
||||
{Name: "age", Type: FieldTypeInt64, Indexed: true, Comment: "年龄"},
|
||||
})
|
||||
|
||||
usersTable, err := db.CreateTable("users", userSchema)
|
||||
if err != nil {
|
||||
t.Fatalf("CreateTable failed: %v", err)
|
||||
}
|
||||
|
||||
// 插入数据
|
||||
err = usersTable.Insert(map[string]any{
|
||||
"name": "Alice",
|
||||
"age": int64(25),
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Insert failed: %v", err)
|
||||
}
|
||||
|
||||
err = usersTable.Insert(map[string]any{
|
||||
"name": "Bob",
|
||||
"age": int64(30),
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Insert failed: %v", err)
|
||||
}
|
||||
|
||||
// 查询数据
|
||||
rows, err := usersTable.Query().Eq("name", "Alice").Rows()
|
||||
if err != nil {
|
||||
t.Fatalf("Query failed: %v", err)
|
||||
}
|
||||
|
||||
if rows.Len() != 1 {
|
||||
t.Errorf("Expected 1 result, got %d", rows.Len())
|
||||
}
|
||||
|
||||
if rows.Data()[0]["name"] != "Alice" {
|
||||
t.Errorf("Expected name 'Alice', got '%v'", rows.Data()[0]["name"])
|
||||
}
|
||||
|
||||
// 统计
|
||||
stats := usersTable.Stats()
|
||||
if stats.TotalRows != 2 {
|
||||
t.Errorf("Expected 2 rows, got %d", stats.TotalRows)
|
||||
}
|
||||
|
||||
t.Log("Table operations test passed!")
|
||||
}
|
||||
|
||||
func TestDropTable(t *testing.T) {
|
||||
dir := "./test_db_drop"
|
||||
os.RemoveAll(dir)
|
||||
defer os.RemoveAll(dir)
|
||||
|
||||
db, err := Open(dir)
|
||||
if err != nil {
|
||||
t.Fatalf("Open failed: %v", err)
|
||||
}
|
||||
defer db.Close()
|
||||
|
||||
// 创建表
|
||||
userSchema := NewSchema("users", []Field{
|
||||
{Name: "name", Type: FieldTypeString, Indexed: true, Comment: "用户名"},
|
||||
})
|
||||
|
||||
_, err = db.CreateTable("users", userSchema)
|
||||
if err != nil {
|
||||
t.Fatalf("CreateTable failed: %v", err)
|
||||
}
|
||||
|
||||
// 删除表
|
||||
err = db.DropTable("users")
|
||||
if err != nil {
|
||||
t.Fatalf("DropTable failed: %v", err)
|
||||
}
|
||||
|
||||
// 检查表列表
|
||||
tables := db.ListTables()
|
||||
if len(tables) != 0 {
|
||||
t.Errorf("Expected 0 tables after drop, got %d", len(tables))
|
||||
}
|
||||
|
||||
t.Log("Drop table test passed!")
|
||||
}
|
||||
|
||||
func TestDatabaseRecover(t *testing.T) {
|
||||
dir := "./test_db_recover"
|
||||
os.RemoveAll(dir)
|
||||
defer os.RemoveAll(dir)
|
||||
|
||||
// 第一次：创建数据库和表
|
||||
db1, err := Open(dir)
|
||||
if err != nil {
|
||||
t.Fatalf("Open failed: %v", err)
|
||||
}
|
||||
|
||||
userSchema := NewSchema("users", []Field{
|
||||
{Name: "name", Type: FieldTypeString, Indexed: true, Comment: "用户名"},
|
||||
{Name: "age", Type: FieldTypeInt64, Indexed: true, Comment: "年龄"},
|
||||
})
|
||||
|
||||
usersTable, err := db1.CreateTable("users", userSchema)
|
||||
if err != nil {
|
||||
t.Fatalf("CreateTable failed: %v", err)
|
||||
}
|
||||
|
||||
// 插入数据
|
||||
usersTable.Insert(map[string]any{
|
||||
"name": "Alice",
|
||||
"age": int64(25),
|
||||
})
|
||||
|
||||
db1.Close()
|
||||
|
||||
// 第二次：重新打开数据库
|
||||
db2, err := Open(dir)
|
||||
if err != nil {
|
||||
t.Fatalf("Open after recover failed: %v", err)
|
||||
}
|
||||
defer db2.Close()
|
||||
|
||||
// 检查表是否恢复
|
||||
tables := db2.ListTables()
|
||||
if len(tables) != 1 {
|
||||
t.Errorf("Expected 1 table after recover, got %d", len(tables))
|
||||
}
|
||||
|
||||
// 获取表
|
||||
usersTable2, err := db2.GetTable("users")
|
||||
if err != nil {
|
||||
t.Fatalf("GetTable failed: %v", err)
|
||||
}
|
||||
|
||||
// 检查数据是否恢复
|
||||
stats := usersTable2.Stats()
|
||||
if stats.TotalRows != 1 {
|
||||
t.Errorf("Expected 1 row after recover, got %d", stats.TotalRows)
|
||||
}
|
||||
|
||||
t.Log("Database recover test passed!")
|
||||
}
|
||||
627
engine.go
Normal file
627
engine.go
Normal file
@@ -0,0 +1,627 @@
|
||||
package srdb
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
|
||||
"code.tczkiot.com/srdb/compaction"
|
||||
"code.tczkiot.com/srdb/manifest"
|
||||
"code.tczkiot.com/srdb/memtable"
|
||||
"code.tczkiot.com/srdb/sst"
|
||||
"code.tczkiot.com/srdb/wal"
|
||||
)
|
||||
|
||||
const (
|
||||
DefaultMemTableSize = 64 * 1024 * 1024 // 64 MB
|
||||
)
|
||||
|
||||
// Engine 存储引擎
|
||||
type Engine struct {
|
||||
dir string
|
||||
schema *Schema
|
||||
indexManager *IndexManager
|
||||
walManager *wal.Manager // WAL 管理器
|
||||
sstManager *sst.Manager // SST 管理器
|
||||
memtableManager *memtable.Manager // MemTable 管理器
|
||||
versionSet *manifest.VersionSet // MANIFEST 管理器
|
||||
compactionManager *compaction.Manager // Compaction 管理器
|
||||
seq atomic.Int64
|
||||
mu sync.RWMutex
|
||||
flushMu sync.Mutex
|
||||
}
|
||||
|
||||
// EngineOptions 配置选项
|
||||
type EngineOptions struct {
|
||||
Dir string
|
||||
MemTableSize int64
|
||||
Schema *Schema // 可选的 Schema 定义
|
||||
}
|
||||
|
||||
// OpenEngine 打开数据库
|
||||
func OpenEngine(opts *EngineOptions) (*Engine, error) {
|
||||
if opts.MemTableSize == 0 {
|
||||
opts.MemTableSize = DefaultMemTableSize
|
||||
}
|
||||
|
||||
// 创建主目录
|
||||
err := os.MkdirAll(opts.Dir, 0755)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// 创建子目录
|
||||
walDir := filepath.Join(opts.Dir, "wal")
|
||||
sstDir := filepath.Join(opts.Dir, "sst")
|
||||
idxDir := filepath.Join(opts.Dir, "index")
|
||||
|
||||
err = os.MkdirAll(walDir, 0755)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
err = os.MkdirAll(sstDir, 0755)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
err = os.MkdirAll(idxDir, 0755)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// 尝试从磁盘恢复 Schema（如果 Options 中没有提供）
|
||||
var sch *Schema
|
||||
if opts.Schema != nil {
|
||||
// 使用提供的 Schema
|
||||
sch = opts.Schema
|
||||
// 保存到磁盘（带校验和）
|
||||
schemaPath := filepath.Join(opts.Dir, "schema.json")
|
||||
schemaFile, err := NewSchemaFile(sch)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("create schema file: %w", err)
|
||||
}
|
||||
schemaData, err := json.MarshalIndent(schemaFile, "", " ")
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("marshal schema: %w", err)
|
||||
}
|
||||
err = os.WriteFile(schemaPath, schemaData, 0644)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("write schema: %w", err)
|
||||
}
|
||||
} else {
|
||||
// 尝试从磁盘恢复
|
||||
schemaPath := filepath.Join(opts.Dir, "schema.json")
|
||||
schemaData, err := os.ReadFile(schemaPath)
|
||||
if err == nil {
|
||||
// 文件存在，尝试解析
|
||||
schemaFile := &SchemaFile{}
|
||||
err = json.Unmarshal(schemaData, schemaFile)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to unmarshal schema from %s: %w", schemaPath, err)
|
||||
}
|
||||
|
||||
// 验证校验和
|
||||
err = schemaFile.Verify()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to verify schema from %s: %w", schemaPath, err)
|
||||
}
|
||||
|
||||
sch = schemaFile.Schema
|
||||
} else if !os.IsNotExist(err) {
|
||||
// 其他读取错误
|
||||
return nil, fmt.Errorf("failed to read schema file %s: %w", schemaPath, err)
|
||||
}
|
||||
// 如果文件不存在，sch 保持为 nil（可选）
|
||||
}
|
||||
|
||||
// 创建索引管理器
|
||||
var indexMgr *IndexManager
|
||||
if sch != nil {
|
||||
indexMgr = NewIndexManager(idxDir, sch)
|
||||
}
|
||||
|
||||
// 创建 SST Manager
|
||||
sstMgr, err := sst.NewManager(sstDir)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// 创建 MemTable Manager
|
||||
memMgr := memtable.NewManager(opts.MemTableSize)
|
||||
|
||||
// 创建/恢复 MANIFEST
|
||||
manifestDir := opts.Dir
|
||||
versionSet, err := manifest.NewVersionSet(manifestDir)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("create version set: %w", err)
|
||||
}
|
||||
|
||||
// 创建 Engine（暂时不设置 WAL Manager）
|
||||
engine := &Engine{
|
||||
dir: opts.Dir,
|
||||
schema: sch,
|
||||
indexManager: indexMgr,
|
||||
walManager: nil, // 先不设置，恢复后再创建
|
||||
sstManager: sstMgr,
|
||||
memtableManager: memMgr,
|
||||
versionSet: versionSet,
|
||||
}
|
||||
|
||||
// 先恢复数据（包括从 WAL 恢复）
|
||||
err = engine.recover()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// 恢复完成后，创建 WAL Manager 用于后续写入
|
||||
walMgr, err := wal.NewManager(walDir)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
engine.walManager = walMgr
|
||||
engine.memtableManager.SetActiveWAL(walMgr.GetCurrentNumber())
|
||||
|
||||
// 创建 Compaction Manager
|
||||
engine.compactionManager = compaction.NewManager(sstDir, versionSet)
|
||||
|
||||
// 启动时清理孤儿文件（崩溃恢复后的清理）
|
||||
engine.compactionManager.CleanupOrphanFiles()
|
||||
|
||||
// 启动后台 Compaction 和垃圾回收
|
||||
engine.compactionManager.Start()
|
||||
|
||||
// 验证并修复索引
|
||||
if engine.indexManager != nil {
|
||||
engine.verifyAndRepairIndexes()
|
||||
}
|
||||
|
||||
return engine, nil
|
||||
}
|
||||
|
||||
// Insert 插入数据
|
||||
func (e *Engine) Insert(data map[string]any) error {
|
||||
// 1. 验证 Schema (如果定义了)
|
||||
if e.schema != nil {
|
||||
if err := e.schema.Validate(data); err != nil {
|
||||
return fmt.Errorf("schema validation failed: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// 2. 生成 _seq
|
||||
seq := e.seq.Add(1)
|
||||
|
||||
// 2. 添加系统字段
|
||||
row := &sst.Row{
|
||||
Seq: seq,
|
||||
Time: time.Now().UnixNano(),
|
||||
Data: data,
|
||||
}
|
||||
|
||||
// 3. 序列化
|
||||
rowData, err := json.Marshal(row)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// 4. 写入 WAL
|
||||
entry := &wal.Entry{
|
||||
Type: wal.EntryTypePut,
|
||||
Seq: seq,
|
||||
Data: rowData,
|
||||
}
|
||||
err = e.walManager.Append(entry)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// 5. 写入 MemTable Manager
|
||||
e.memtableManager.Put(seq, rowData)
|
||||
|
||||
// 6. 添加到索引
|
||||
if e.indexManager != nil {
|
||||
e.indexManager.AddToIndexes(data, seq)
|
||||
}
|
||||
|
||||
// 7. 检查是否需要切换 MemTable
|
||||
if e.memtableManager.ShouldSwitch() {
|
||||
go e.switchMemTable()
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// Get 查询数据
|
||||
func (e *Engine) Get(seq int64) (*sst.Row, error) {
|
||||
// 1. 先查 MemTable Manager (Active + Immutables)
|
||||
data, found := e.memtableManager.Get(seq)
|
||||
if found {
|
||||
var row sst.Row
|
||||
err := json.Unmarshal(data, &row)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &row, nil
|
||||
}
|
||||
|
||||
// 2. 查询 SST 文件
|
||||
return e.sstManager.Get(seq)
|
||||
}
|
||||
|
||||
// switchMemTable 切换 MemTable
|
||||
func (e *Engine) switchMemTable() error {
|
||||
e.flushMu.Lock()
|
||||
defer e.flushMu.Unlock()
|
||||
|
||||
// 1. 切换到新的 WAL
|
||||
oldWALNumber, err := e.walManager.Rotate()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
newWALNumber := e.walManager.GetCurrentNumber()
|
||||
|
||||
// 2. 切换 MemTable (Active → Immutable)
|
||||
_, immutable := e.memtableManager.Switch(newWALNumber)
|
||||
|
||||
// 3. 异步 Flush Immutable
|
||||
go e.flushImmutable(immutable, oldWALNumber)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// flushImmutable 将 Immutable MemTable 刷新到 SST
|
||||
func (e *Engine) flushImmutable(imm *memtable.ImmutableMemTable, walNumber int64) error {
|
||||
// 1. 收集所有行
|
||||
var rows []*sst.Row
|
||||
iter := imm.MemTable.NewIterator()
|
||||
for iter.Next() {
|
||||
var row sst.Row
|
||||
err := json.Unmarshal(iter.Value(), &row)
|
||||
if err == nil {
|
||||
rows = append(rows, &row)
|
||||
}
|
||||
}
|
||||
|
||||
if len(rows) == 0 {
|
||||
// 没有数据，直接清理
|
||||
e.walManager.Delete(walNumber)
|
||||
e.memtableManager.RemoveImmutable(imm)
|
||||
return nil
|
||||
}
|
||||
|
||||
// 2. 从 VersionSet 分配文件编号
|
||||
fileNumber := e.versionSet.AllocateFileNumber()
|
||||
|
||||
// 3. 创建 SST 文件到 L0
|
||||
reader, err := e.sstManager.CreateSST(fileNumber, rows)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// 4. 创建 FileMetadata
|
||||
header := reader.GetHeader()
|
||||
|
||||
// 获取文件大小
|
||||
sstPath := reader.GetPath()
|
||||
fileInfo, err := os.Stat(sstPath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("stat sst file: %w", err)
|
||||
}
|
||||
|
||||
fileMeta := &manifest.FileMetadata{
|
||||
FileNumber: fileNumber,
|
||||
Level: 0, // Flush 到 L0
|
||||
FileSize: fileInfo.Size(),
|
||||
MinKey: header.MinKey,
|
||||
MaxKey: header.MaxKey,
|
||||
RowCount: header.RowCount,
|
||||
}
|
||||
|
||||
// 5. 更新 MANIFEST
|
||||
edit := manifest.NewVersionEdit()
|
||||
edit.AddFile(fileMeta)
|
||||
|
||||
// 持久化当前的文件编号计数器（关键修复：防止重启后文件编号重用）
|
||||
edit.SetNextFileNumber(e.versionSet.GetNextFileNumber())
|
||||
|
||||
err = e.versionSet.LogAndApply(edit)
|
||||
if err != nil {
|
||||
return fmt.Errorf("log and apply version edit: %w", err)
|
||||
}
|
||||
|
||||
// 6. 删除对应的 WAL
|
||||
e.walManager.Delete(walNumber)
|
||||
|
||||
// 7. 从 Immutable 列表中移除
|
||||
e.memtableManager.RemoveImmutable(imm)
|
||||
|
||||
// 8. 触发 Compaction 检查（非阻塞）
|
||||
// Flush 后 L0 增加了新文件，可能需要立即触发 compaction
|
||||
e.compactionManager.MaybeCompact()
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// recover 恢复数据
|
||||
func (e *Engine) recover() error {
|
||||
// 1. 恢复 SST 文件（SST Manager 已经在 NewManager 中恢复了）
|
||||
// 只需要获取最大 seq
|
||||
maxSeq := e.sstManager.GetMaxSeq()
|
||||
if maxSeq > e.seq.Load() {
|
||||
e.seq.Store(maxSeq)
|
||||
}
|
||||
|
||||
// 2. 恢复所有 WAL 文件到 MemTable Manager
|
||||
walDir := filepath.Join(e.dir, "wal")
|
||||
pattern := filepath.Join(walDir, "*.wal")
|
||||
walFiles, err := filepath.Glob(pattern)
|
||||
if err == nil && len(walFiles) > 0 {
|
||||
// 按文件名排序
|
||||
sort.Strings(walFiles)
|
||||
|
||||
// 依次读取每个 WAL
|
||||
for _, walPath := range walFiles {
|
||||
reader, err := wal.NewReader(walPath)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
|
||||
entries, err := reader.Read()
|
||||
reader.Close()
|
||||
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
|
||||
// 重放 WAL 到 Active MemTable
|
||||
for _, entry := range entries {
|
||||
// 如果定义了 Schema，验证数据
|
||||
if e.schema != nil {
|
||||
var row sst.Row
|
||||
if err := json.Unmarshal(entry.Data, &row); err != nil {
|
||||
return fmt.Errorf("failed to unmarshal row during recovery (seq=%d): %w", entry.Seq, err)
|
||||
}
|
||||
|
||||
// 验证 Schema
|
||||
if err := e.schema.Validate(row.Data); err != nil {
|
||||
return fmt.Errorf("schema validation failed during recovery (seq=%d): %w", entry.Seq, err)
|
||||
}
|
||||
}
|
||||
|
||||
e.memtableManager.Put(entry.Seq, entry.Data)
|
||||
if entry.Seq > e.seq.Load() {
|
||||
e.seq.Store(entry.Seq)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// Close 关闭引擎
|
||||
func (e *Engine) Close() error {
|
||||
// 1. 停止后台 Compaction
|
||||
if e.compactionManager != nil {
|
||||
e.compactionManager.Stop()
|
||||
}
|
||||
|
||||
// 2. Flush Active MemTable
|
||||
if e.memtableManager.GetActiveCount() > 0 {
|
||||
// 切换并 Flush
|
||||
e.switchMemTable()
|
||||
}
|
||||
|
||||
// 等待所有 Immutable Flush 完成
|
||||
// TODO: 添加更优雅的等待机制
|
||||
for e.memtableManager.GetImmutableCount() > 0 {
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
}
|
||||
|
||||
// 3. 保存所有索引
|
||||
if e.indexManager != nil {
|
||||
e.indexManager.BuildAll()
|
||||
e.indexManager.Close()
|
||||
}
|
||||
|
||||
// 4. 关闭 VersionSet
|
||||
if e.versionSet != nil {
|
||||
e.versionSet.Close()
|
||||
}
|
||||
|
||||
// 5. 关闭 WAL Manager
|
||||
if e.walManager != nil {
|
||||
e.walManager.Close()
|
||||
}
|
||||
|
||||
// 6. 关闭 SST Manager
|
||||
if e.sstManager != nil {
|
||||
e.sstManager.Close()
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// Stats 统计信息
|
||||
type Stats struct {
|
||||
MemTableSize int64
|
||||
MemTableCount int
|
||||
SSTCount int
|
||||
TotalRows int64
|
||||
}
|
||||
|
||||
// GetVersionSet 获取 VersionSet（用于高级操作）
|
||||
func (e *Engine) GetVersionSet() *manifest.VersionSet {
|
||||
return e.versionSet
|
||||
}
|
||||
|
||||
// GetCompactionManager 获取 Compaction Manager（用于高级操作）
|
||||
func (e *Engine) GetCompactionManager() *compaction.Manager {
|
||||
return e.compactionManager
|
||||
}
|
||||
|
||||
// GetMemtableManager 获取 Memtable Manager
|
||||
func (e *Engine) GetMemtableManager() *memtable.Manager {
|
||||
return e.memtableManager
|
||||
}
|
||||
|
||||
// GetSSTManager 获取 SST Manager
|
||||
func (e *Engine) GetSSTManager() *sst.Manager {
|
||||
return e.sstManager
|
||||
}
|
||||
|
||||
// GetMaxSeq 获取当前最大的 seq 号
|
||||
func (e *Engine) GetMaxSeq() int64 {
|
||||
return e.seq.Load() - 1 // seq 是下一个要分配的，所以最大的是 seq - 1
|
||||
}
|
||||
|
||||
// GetSchema 获取 Schema
|
||||
func (e *Engine) GetSchema() *Schema {
|
||||
return e.schema
|
||||
}
|
||||
|
||||
// Stats 获取统计信息
|
||||
func (e *Engine) Stats() *Stats {
|
||||
memStats := e.memtableManager.GetStats()
|
||||
sstStats := e.sstManager.GetStats()
|
||||
|
||||
stats := &Stats{
|
||||
MemTableSize: memStats.TotalSize,
|
||||
MemTableCount: memStats.TotalCount,
|
||||
SSTCount: sstStats.FileCount,
|
||||
}
|
||||
|
||||
// 计算总行数
|
||||
stats.TotalRows = int64(memStats.TotalCount)
|
||||
readers := e.sstManager.GetReaders()
|
||||
for _, reader := range readers {
|
||||
header := reader.GetHeader()
|
||||
stats.TotalRows += header.RowCount
|
||||
}
|
||||
|
||||
return stats
|
||||
}
|
||||
|
||||
// CreateIndex 创建索引
|
||||
func (e *Engine) CreateIndex(field string) error {
|
||||
if e.indexManager == nil {
|
||||
return fmt.Errorf("no schema defined, cannot create index")
|
||||
}
|
||||
|
||||
return e.indexManager.CreateIndex(field)
|
||||
}
|
||||
|
||||
// DropIndex 删除索引
|
||||
func (e *Engine) DropIndex(field string) error {
|
||||
if e.indexManager == nil {
|
||||
return fmt.Errorf("no schema defined, cannot drop index")
|
||||
}
|
||||
|
||||
return e.indexManager.DropIndex(field)
|
||||
}
|
||||
|
||||
// ListIndexes 列出所有索引
|
||||
func (e *Engine) ListIndexes() []string {
|
||||
if e.indexManager == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
return e.indexManager.ListIndexes()
|
||||
}
|
||||
|
||||
// GetIndexMetadata 获取索引元数据
|
||||
func (e *Engine) GetIndexMetadata() map[string]IndexMetadata {
|
||||
if e.indexManager == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
return e.indexManager.GetIndexMetadata()
|
||||
}
|
||||
|
||||
// RepairIndexes 手动修复索引
|
||||
func (e *Engine) RepairIndexes() error {
|
||||
return e.verifyAndRepairIndexes()
|
||||
}
|
||||
|
||||
// Query 创建查询构建器
|
||||
func (e *Engine) Query() *QueryBuilder {
|
||||
return newQueryBuilder(e)
|
||||
}
|
||||
|
||||
// scanAllWithBuilder 使用 QueryBuilder 全表扫描
|
||||
func (e *Engine) scanAllWithBuilder(qb *QueryBuilder) ([]*sst.Row, error) {
|
||||
// 使用 map 去重（同一个 seq 只保留一次）
|
||||
rowMap := make(map[int64]*sst.Row)
|
||||
|
||||
// 扫描 Active MemTable
|
||||
iter := e.memtableManager.NewIterator()
|
||||
for iter.Next() {
|
||||
seq := iter.Key()
|
||||
row, err := e.Get(seq)
|
||||
if err == nil && qb.Match(row.Data) {
|
||||
rowMap[seq] = row
|
||||
}
|
||||
}
|
||||
|
||||
// 扫描 Immutable MemTables
|
||||
immutables := e.memtableManager.GetImmutables()
|
||||
for _, imm := range immutables {
|
||||
iter := imm.MemTable.NewIterator()
|
||||
for iter.Next() {
|
||||
seq := iter.Key()
|
||||
if _, exists := rowMap[seq]; !exists {
|
||||
row, err := e.Get(seq)
|
||||
if err == nil && qb.Match(row.Data) {
|
||||
rowMap[seq] = row
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 扫描 SST 文件
|
||||
readers := e.sstManager.GetReaders()
|
||||
for _, reader := range readers {
|
||||
header := reader.GetHeader()
|
||||
for seq := header.MinKey; seq <= header.MaxKey; seq++ {
|
||||
if _, exists := rowMap[seq]; !exists {
|
||||
row, err := reader.Get(seq)
|
||||
if err == nil && qb.Match(row.Data) {
|
||||
rowMap[seq] = row
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 转换为数组
|
||||
results := make([]*sst.Row, 0, len(rowMap))
|
||||
for _, row := range rowMap {
|
||||
results = append(results, row)
|
||||
}
|
||||
|
||||
return results, nil
|
||||
}
|
||||
|
||||
// verifyAndRepairIndexes 验证并修复索引
|
||||
func (e *Engine) verifyAndRepairIndexes() error {
|
||||
if e.indexManager == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
// 获取当前最大 seq
|
||||
currentMaxSeq := e.seq.Load()
|
||||
|
||||
// 创建 getData 函数
|
||||
getData := func(seq int64) (map[string]any, error) {
|
||||
row, err := e.Get(seq)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return row.Data, nil
|
||||
}
|
||||
|
||||
// 验证并修复
|
||||
return e.indexManager.VerifyAndRepair(currentMaxSeq, getData)
|
||||
}
|
||||
1434
engine_test.go
Normal file
1434
engine_test.go
Normal file
File diff suppressed because it is too large
Load Diff
481
examples/README.md
Normal file
481
examples/README.md
Normal file
@@ -0,0 +1,481 @@
|
||||
# SRDB Examples
|
||||
|
||||
本目录包含 SRDB 数据库的示例程序和工具。
|
||||
|
||||
## 目录结构
|
||||
|
||||
```
|
||||
examples/
|
||||
└── webui/ # Web UI 和命令行工具集
|
||||
├── main.go # 主入口点
|
||||
├── commands/ # 命令实现
|
||||
│ ├── webui.go # Web UI 服务器
|
||||
│ ├── check_data.go # 数据检查工具
|
||||
│ ├── check_seq.go # 序列号检查工具
|
||||
│ ├── dump_manifest.go # Manifest 导出工具
|
||||
│ ├── inspect_all_sst.go # SST 文件批量检查
|
||||
│ ├── inspect_sst.go # SST 文件检查工具
|
||||
│ ├── test_fix.go # 修复测试工具
|
||||
│ └── test_keys.go # 键存在性测试工具
|
||||
└── README.md # WebUI 详细文档
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## WebUI - 数据库管理工具
|
||||
|
||||
一个集成了 Web 界面和命令行工具的 SRDB 数据库管理工具。
|
||||
|
||||
### 功能特性
|
||||
|
||||
#### 🌐 Web UI
|
||||
- **表列表展示** - 可视化查看所有表及其 Schema
|
||||
- **数据分页浏览** - 表格形式展示数据，支持分页和列选择
|
||||
- **Manifest 查看** - 查看 LSM-Tree 结构和 Compaction 状态
|
||||
- **响应式设计** - 基于 HTMX 的现代化界面
|
||||
- **大数据优化** - 自动截断显示，点击查看完整内容
|
||||
|
||||
#### 🛠️ 命令行工具
|
||||
- **数据检查** - 检查表和数据完整性
|
||||
- **序列号验证** - 验证特定序列号的数据
|
||||
- **Manifest 导出** - 导出 LSM-Tree 层级信息
|
||||
- **SST 文件检查** - 检查和诊断 SST 文件问题
|
||||
|
||||
### 快速开始
|
||||
|
||||
#### 1. 启动 Web UI
|
||||
|
||||
```bash
|
||||
cd examples/webui
|
||||
|
||||
# 使用默认配置（数据库：./data，端口：8080）
|
||||
go run main.go serve
|
||||
|
||||
# 或指定自定义配置
|
||||
go run main.go serve -db ./mydb -addr :3000
|
||||
```
|
||||
|
||||
然后打开浏览器访问 `http://localhost:8080`
|
||||
|
||||
#### 2. 查看帮助
|
||||
|
||||
```bash
|
||||
go run main.go help
|
||||
```
|
||||
|
||||
输出：
|
||||
```
|
||||
SRDB WebUI - Database management tool
|
||||
|
||||
Usage:
|
||||
webui <command> [flags]
|
||||
|
||||
Commands:
|
||||
webui, serve Start WebUI server (default: :8080)
|
||||
check-data Check database tables and row counts
|
||||
check-seq Check specific sequence numbers
|
||||
dump-manifest Dump manifest information
|
||||
inspect-all-sst Inspect all SST files
|
||||
inspect-sst Inspect a specific SST file
|
||||
test-fix Test fix for data retrieval
|
||||
test-keys Test key existence
|
||||
help Show this help message
|
||||
|
||||
Examples:
|
||||
webui serve -db ./mydb -addr :3000
|
||||
webui check-data -db ./mydb
|
||||
webui inspect-sst -file ./data/logs/sst/000046.sst
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 命令详解
|
||||
|
||||
### serve / webui - 启动 Web 服务器
|
||||
|
||||
启动 Web UI 服务器，提供数据可视化界面。
|
||||
|
||||
```bash
|
||||
# 基本用法
|
||||
go run main.go serve
|
||||
|
||||
# 指定数据库路径和端口
|
||||
go run main.go webui -db ./mydb -addr :3000
|
||||
```
|
||||
|
||||
**参数**：
|
||||
- `-db` - 数据库目录路径（默认：`./data`）
|
||||
- `-addr` - 服务器地址（默认：`:8080`）
|
||||
|
||||
**功能**：
|
||||
- 自动创建示例表（users, products, logs）
|
||||
- 后台自动插入测试数据（每秒一条）
|
||||
- 提供 Web UI 和 HTTP API
|
||||
|
||||
---
|
||||
|
||||
### check-data - 检查数据
|
||||
|
||||
检查数据库中所有表的记录数。
|
||||
|
||||
```bash
|
||||
go run main.go check-data -db ./data
|
||||
```
|
||||
|
||||
**输出示例**：
|
||||
```
|
||||
Found 3 tables: [users products logs]
|
||||
Table 'users': 5 rows
|
||||
Table 'products': 6 rows
|
||||
Table 'logs': 1234 rows
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### check-seq - 检查序列号
|
||||
|
||||
验证特定序列号的数据是否存在。
|
||||
|
||||
```bash
|
||||
go run main.go check-seq -db ./data
|
||||
```
|
||||
|
||||
**功能**：
|
||||
- 检查 seq=1, 100, 729 等特定序列号
|
||||
- 显示总记录数
|
||||
- 验证数据完整性
|
||||
|
||||
---
|
||||
|
||||
### dump-manifest - 导出 Manifest
|
||||
|
||||
导出数据库的 Manifest 信息，检查文件重复。
|
||||
|
||||
```bash
|
||||
go run main.go dump-manifest -db ./data
|
||||
```
|
||||
|
||||
**输出示例**：
|
||||
```
|
||||
Level 0: 5 files
|
||||
Level 1: 3 files
|
||||
Level 2: 1 files
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### inspect-all-sst - 批量检查 SST 文件
|
||||
|
||||
检查所有 SST 文件的完整性。
|
||||
|
||||
```bash
|
||||
go run main.go inspect-all-sst -dir ./data/logs/sst
|
||||
```
|
||||
|
||||
**输出示例**：
|
||||
```
|
||||
Found 10 SST files
|
||||
|
||||
File #1 (000001.sst):
|
||||
Header: MinKey=1 MaxKey=100 RowCount=100
|
||||
Actual: 100 keys [1 ... 100]
|
||||
|
||||
File #2 (000002.sst):
|
||||
Header: MinKey=101 MaxKey=200 RowCount=100
|
||||
Actual: 100 keys [101 ... 200]
|
||||
*** MISMATCH: Header says 101-200 but file has 105-200 ***
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### inspect-sst - 检查单个 SST 文件
|
||||
|
||||
详细检查特定 SST 文件。
|
||||
|
||||
```bash
|
||||
go run main.go inspect-sst -file ./data/logs/sst/000046.sst
|
||||
```
|
||||
|
||||
**输出示例**：
|
||||
```
|
||||
File: ./data/logs/sst/000046.sst
|
||||
Size: 524288 bytes
|
||||
|
||||
Header:
|
||||
RowCount: 100
|
||||
MinKey: 332
|
||||
MaxKey: 354
|
||||
DataSize: 512000 bytes
|
||||
|
||||
Actual keys in file: 100 keys
|
||||
First key: 332
|
||||
Last key: 354
|
||||
All keys: [332 333 334 ... 354]
|
||||
|
||||
Trying to get key 332:
|
||||
FOUND: seq=332, time=1234567890
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### test-fix - 测试修复
|
||||
|
||||
测试数据检索的修复功能。
|
||||
|
||||
```bash
|
||||
go run main.go test-fix -db ./data
|
||||
```
|
||||
|
||||
**功能**：
|
||||
- 测试首部、中部、尾部记录
|
||||
- 验证 Get() 操作的正确性
|
||||
- 显示修复状态
|
||||
|
||||
---
|
||||
|
||||
### test-keys - 测试键存在性
|
||||
|
||||
测试特定键是否存在。
|
||||
|
||||
```bash
|
||||
go run main.go test-keys -db ./data
|
||||
```
|
||||
|
||||
**功能**：
|
||||
- 测试预定义的键列表
|
||||
- 统计找到的键数量
|
||||
- 显示首尾记录
|
||||
|
||||
---
|
||||
|
||||
## 编译安装
|
||||
|
||||
### 编译二进制
|
||||
|
||||
```bash
|
||||
cd examples/webui
|
||||
go build -o webui main.go
|
||||
```
|
||||
|
||||
### 全局安装
|
||||
|
||||
```bash
|
||||
go install ./examples/webui@latest
|
||||
```
|
||||
|
||||
然后可以在任何地方使用：
|
||||
|
||||
```bash
|
||||
webui serve -db ./mydb
|
||||
webui check-data -db ./mydb
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Web UI 使用
|
||||
|
||||
### 界面布局
|
||||
|
||||
访问 `http://localhost:8080` 后，你会看到：
|
||||
|
||||
**左侧边栏**：
|
||||
- 表列表，显示每个表的字段数
|
||||
- 点击展开查看 Schema 详情
|
||||
- 点击表名切换到该表
|
||||
|
||||
**右侧主区域**：
|
||||
- **Data 视图**：数据表格，支持分页和列选择
|
||||
- **Manifest 视图**：LSM-Tree 结构和 Compaction 状态
|
||||
|
||||
### HTTP API 端点
|
||||
|
||||
#### 获取表列表
|
||||
```
|
||||
GET /api/tables-html
|
||||
```
|
||||
|
||||
#### 获取表数据
|
||||
```
|
||||
GET /api/tables-view/{table_name}?page=1&pageSize=20
|
||||
```
|
||||
|
||||
#### 获取 Manifest
|
||||
```
|
||||
GET /api/tables-view/{table_name}/manifest
|
||||
```
|
||||
|
||||
#### 获取 Schema
|
||||
```
|
||||
GET /api/tables/{table_name}/schema
|
||||
```
|
||||
|
||||
#### 获取单条数据
|
||||
```
|
||||
GET /api/tables/{table_name}/data/{seq}
|
||||
```
|
||||
|
||||
详细 API 文档请参考：[webui/README.md](webui/README.md)
|
||||
|
||||
---
|
||||
|
||||
## 在你的应用中集成
|
||||
|
||||
### 方式 1：使用 WebUI 包
|
||||
|
||||
```go
|
||||
package main
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"code.tczkiot.com/srdb"
|
||||
"code.tczkiot.com/srdb/webui"
|
||||
)
|
||||
|
||||
func main() {
|
||||
db, _ := srdb.Open("./mydb")
|
||||
defer db.Close()
|
||||
|
||||
// 创建 WebUI handler
|
||||
handler := webui.NewWebUI(db)
|
||||
|
||||
// 启动服务器
|
||||
http.ListenAndServe(":8080", handler)
|
||||
}
|
||||
```
|
||||
|
||||
### 方式 2：挂载到现有应用
|
||||
|
||||
```go
|
||||
mux := http.NewServeMux()
|
||||
|
||||
// 你的其他路由
|
||||
mux.HandleFunc("/api/myapp", myHandler)
|
||||
|
||||
// 挂载 SRDB Web UI 到 /admin/db 路径
|
||||
mux.Handle("/admin/db/", http.StripPrefix("/admin/db", webui.NewWebUI(db)))
|
||||
|
||||
http.ListenAndServe(":8080", mux)
|
||||
```
|
||||
|
||||
### 方式 3：使用命令工具
|
||||
|
||||
将 webui 工具的命令集成到你的应用：
|
||||
|
||||
```go
|
||||
import "code.tczkiot.com/srdb/examples/webui/commands"
|
||||
|
||||
// 检查数据
|
||||
commands.CheckData("./mydb")
|
||||
|
||||
// 导出 manifest
|
||||
commands.DumpManifest("./mydb")
|
||||
|
||||
// 启动服务器
|
||||
commands.StartWebUI("./mydb", ":8080")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 开发和调试
|
||||
|
||||
### 开发模式
|
||||
|
||||
在开发时，使用 `go run` 可以快速测试：
|
||||
|
||||
```bash
|
||||
# 启动服务器
|
||||
go run main.go serve
|
||||
|
||||
# 在另一个终端检查数据
|
||||
go run main.go check-data
|
||||
|
||||
# 检查 SST 文件
|
||||
go run main.go inspect-all-sst
|
||||
```
|
||||
|
||||
### 清理数据
|
||||
|
||||
```bash
|
||||
# 删除数据目录
|
||||
rm -rf ./data
|
||||
|
||||
# 重新运行
|
||||
go run main.go serve
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 注意事项
|
||||
|
||||
1. **数据目录**：默认在当前目录创建 `./data` 目录
|
||||
2. **端口占用**：确保端口未被占用
|
||||
3. **并发访问**：Web UI 支持多用户并发访问
|
||||
4. **只读模式**：Web UI 仅用于查看，不提供数据修改功能
|
||||
5. **生产环境**：建议添加身份验证和访问控制
|
||||
6. **性能考虑**：大表分页查询性能取决于数据分布
|
||||
|
||||
---
|
||||
|
||||
## 技术栈
|
||||
|
||||
- **后端**：Go 标准库（net/http）
|
||||
- **前端**：HTMX + 原生 JavaScript + CSS
|
||||
- **渲染**：服务端 HTML 渲染（Go）
|
||||
- **数据库**：SRDB (LSM-Tree)
|
||||
- **部署**：所有静态资源通过 embed 嵌入，无需单独部署
|
||||
|
||||
---
|
||||
|
||||
## 故障排除
|
||||
|
||||
### 常见问题
|
||||
|
||||
**1. 启动失败 - 端口被占用**
|
||||
```bash
|
||||
Error: listen tcp :8080: bind: address already in use
|
||||
```
|
||||
解决：使用 `-addr` 指定其他端口
|
||||
```bash
|
||||
go run main.go serve -addr :3000
|
||||
```
|
||||
|
||||
**2. 数据库打开失败**
|
||||
```bash
|
||||
Error: failed to open database: invalid header
|
||||
```
|
||||
解决：删除损坏的数据目录
|
||||
```bash
|
||||
rm -rf ./data
|
||||
```
|
||||
|
||||
**3. SST 文件损坏**
|
||||
使用 `inspect-sst` 或 `inspect-all-sst` 命令诊断：
|
||||
```bash
|
||||
go run main.go inspect-all-sst -dir ./data/logs/sst
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 更多信息
|
||||
|
||||
- **WebUI 详细文档**：[webui/README.md](webui/README.md)
|
||||
- **SRDB 主文档**：[../README.md](../README.md)
|
||||
- **Compaction 说明**：[../COMPACTION.md](../COMPACTION.md)
|
||||
- **压力测试报告**：[../STRESS_TEST_RESULTS.md](../STRESS_TEST_RESULTS.md)
|
||||
|
||||
---
|
||||
|
||||
## 贡献
|
||||
|
||||
欢迎贡献新的示例和工具！请遵循以下规范：
|
||||
|
||||
1. 在 `examples/` 下创建新的子目录
|
||||
2. 提供清晰的 README 文档
|
||||
3. 添加示例代码和使用说明
|
||||
4. 更新本文件
|
||||
|
||||
---
|
||||
|
||||
## 许可证
|
||||
|
||||
与 SRDB 项目相同的许可证。
|
||||
254
examples/webui/README.md
Normal file
254
examples/webui/README.md
Normal file
@@ -0,0 +1,254 @@
|
||||
# SRDB Web UI Example
|
||||
|
||||
这个示例展示了如何使用 SRDB 的内置 Web UI 来可视化查看数据库中的表和数据。
|
||||
|
||||
## 功能特性
|
||||
|
||||
- 📊 **表列表展示** - 左侧显示所有表及其行数
|
||||
- 🔍 **Schema 查看** - 点击箭头展开查看表的字段定义
|
||||
- 📋 **数据分页浏览** - 右侧以表格形式展示数据，支持分页
|
||||
- 🎨 **响应式设计** - 现代化的界面设计
|
||||
- ⚡ **零构建** - 使用 HTMX 从 CDN 加载，无需构建步骤
|
||||
- 💾 **大数据优化** - 自动截断显示，悬停查看，点击弹窗查看完整内容
|
||||
- 📏 **数据大小显示** - 超过 1KB 的单元格自动显示大小标签
|
||||
- 🔄 **后台数据插入** - 自动生成 2KB~512KB 的测试数据（每秒一条）
|
||||
|
||||
## 运行示例
|
||||
|
||||
```bash
|
||||
# 进入示例目录
|
||||
cd examples/webui
|
||||
|
||||
# 运行
|
||||
go run main.go
|
||||
```
|
||||
|
||||
程序会：
|
||||
1. 创建/打开数据库目录 `./data`
|
||||
2. 创建三个示例表：`users`、`products` 和 `logs`
|
||||
3. 插入初始示例数据
|
||||
4. **启动后台协程** - 每秒向 `logs` 表插入一条 2KB~512KB 的随机数据
|
||||
5. 启动 Web 服务器在 `http://localhost:8080`
|
||||
|
||||
## 使用界面
|
||||
|
||||
打开浏览器访问 `http://localhost:8080`，你将看到：
|
||||
|
||||
### 左侧边栏
|
||||
- 显示所有表的列表
|
||||
- 显示每个表的字段数量
|
||||
- 点击 ▶ 图标展开查看字段信息
|
||||
- 点击表名选择要查看的表（蓝色高亮显示当前选中）
|
||||
|
||||
### 右侧主区域
|
||||
- **Schema 区域**：显示表结构和字段定义
|
||||
- **Data 区域**：以表格形式显示数据
|
||||
- 支持分页浏览（每页 20 条）
|
||||
- 显示系统字段（_seq, _time）和用户字段
|
||||
- **自动截断长数据**：超过 400px 的内容显示省略号
|
||||
- **鼠标悬停**：悬停在单元格上查看完整内容
|
||||
- **点击查看**：点击单元格在弹窗中查看完整内容
|
||||
- **大小指示**：超过 1KB 的数据显示大小标签
|
||||
|
||||
### 大数据查看
|
||||
1. **表格截断**：单元格最大宽度 400px，超长显示 `...`
|
||||
2. **悬停展开**：鼠标悬停自动展开，黄色背景高亮
|
||||
3. **模态框**：点击单元格弹出窗口
|
||||
- 等宽字体显示（适合查看十六进制数据）
|
||||
- 显示数据大小
|
||||
- 支持滚动查看超长内容
|
||||
|
||||
## API 端点
|
||||
|
||||
Web UI 提供了以下 HTTP API：
|
||||
|
||||
### 获取所有表
|
||||
```
|
||||
GET /api/tables
|
||||
```
|
||||
|
||||
返回示例：
|
||||
```json
|
||||
[
|
||||
{
|
||||
"name": "users",
|
||||
"rowCount": 5,
|
||||
"dir": "./data/users"
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
### 获取表的 Schema
|
||||
```
|
||||
GET /api/tables/{name}/schema
|
||||
```
|
||||
|
||||
返回示例：
|
||||
```json
|
||||
{
|
||||
"fields": [
|
||||
{"name": "name", "type": "string", "required": true},
|
||||
{"name": "email", "type": "string", "required": true},
|
||||
{"name": "age", "type": "int", "required": false}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### 获取表数据（分页）
|
||||
```
|
||||
GET /api/tables/{name}/data?page=1&pageSize=20
|
||||
```
|
||||
|
||||
参数：
|
||||
- `page` - 页码，从 1 开始（默认：1）
|
||||
- `pageSize` - 每页行数，最大 100（默认：20）
|
||||
|
||||
返回示例：
|
||||
```json
|
||||
{
|
||||
"page": 1,
|
||||
"pageSize": 20,
|
||||
"totalRows": 5,
|
||||
"totalPages": 1,
|
||||
"rows": [
|
||||
{
|
||||
"_seq": 1,
|
||||
"_time": 1234567890,
|
||||
"name": "Alice",
|
||||
"email": "alice@example.com",
|
||||
"age": 30
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### 获取表基本信息
|
||||
```
|
||||
GET /api/tables/{name}
|
||||
```
|
||||
|
||||
## 在你的应用中使用
|
||||
|
||||
你可以在自己的应用中轻松集成 Web UI：
|
||||
|
||||
```go
|
||||
package main
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"code.tczkiot.com/srdb"
|
||||
)
|
||||
|
||||
func main() {
|
||||
// 打开数据库
|
||||
db, _ := database.Open("./mydb")
|
||||
defer db.Close()
|
||||
|
||||
// 获取 HTTP Handler
|
||||
handler := db.WebUI()
|
||||
|
||||
// 启动服务器
|
||||
http.ListenAndServe(":8080", handler)
|
||||
}
|
||||
```
|
||||
|
||||
或者将其作为现有 Web 应用的一部分：
|
||||
|
||||
```go
|
||||
mux := http.NewServeMux()
|
||||
|
||||
// 你的其他路由
|
||||
mux.HandleFunc("/api/myapp", myHandler)
|
||||
|
||||
// 挂载 SRDB Web UI 到 /admin/db 路径
|
||||
mux.Handle("/admin/db/", http.StripPrefix("/admin/db", db.WebUI()))
|
||||
|
||||
http.ListenAndServe(":8080", mux)
|
||||
```
|
||||
|
||||
## 技术栈
|
||||
|
||||
- **后端**: Go + 标准库 `net/http`
|
||||
- **前端**: [HTMX](https://htmx.org/) + 原生 JavaScript + CSS
|
||||
- **渲染**: 服务端 HTML 渲染（Go 模板生成）
|
||||
- **字体**: Google Fonts (Inter)
|
||||
- **无构建**: 直接从 CDN 加载 HTMX，无需 npm、webpack 等工具
|
||||
- **部署**: 所有静态资源通过 `embed.FS` 嵌入到二进制文件中
|
||||
|
||||
## 测试大数据
|
||||
|
||||
### logs 表自动生成
|
||||
|
||||
程序会在后台持续向 `logs` 表插入大数据：
|
||||
|
||||
- **频率**：每秒一条
|
||||
- **大小**：2KB ~ 512KB 随机
|
||||
- **格式**：十六进制字符串
|
||||
- **字段**：
|
||||
- `timestamp` - 插入时间
|
||||
- `data` - 随机数据（十六进制）
|
||||
- `size_bytes` - 数据大小（字节）
|
||||
|
||||
你可以选择 `logs` 表来测试大数据的显示效果：
|
||||
1. 单元格会显示数据大小标签（如 `245.12 KB`）
|
||||
2. 内容被自动截断，显示省略号
|
||||
3. 点击单元格在弹窗中查看完整数据
|
||||
|
||||
终端会实时输出插入日志：
|
||||
```
|
||||
Inserted record #1, size: 245.12 KB
|
||||
Inserted record #2, size: 128.50 KB
|
||||
Inserted record #3, size: 487.23 KB
|
||||
```
|
||||
|
||||
## 注意事项
|
||||
|
||||
- Web UI 是只读的，不提供数据修改功能
|
||||
- 适合用于开发、调试和数据查看
|
||||
- 生产环境建议添加身份验证和访问控制
|
||||
- 大数据量表的分页查询性能取决于数据分布
|
||||
- `logs` 表会持续增长，可手动删除 `./data/logs` 目录重置
|
||||
|
||||
## Compaction 状态
|
||||
|
||||
由于后台持续插入大数据，会产生大量 SST 文件。SRDB 会自动运行 compaction 合并这些文件。
|
||||
|
||||
### 检查 Compaction 状态
|
||||
|
||||
```bash
|
||||
# 查看 SST 文件分布
|
||||
./check_sst.sh
|
||||
|
||||
# 观察 webui 日志中的 [Compaction] 信息
|
||||
```
|
||||
|
||||
### Compaction 改进
|
||||
|
||||
- **触发阈值**: L0 文件数量 ≥2 就触发（之前是 4）
|
||||
- **运行频率**: 每 10 秒自动检查
|
||||
- **日志增强**: 显示详细的 compaction 状态和统计
|
||||
|
||||
详细说明请查看 [COMPACTION.md](./COMPACTION.md)
|
||||
|
||||
## 常见问题
|
||||
|
||||
### `invalid header` 错误
|
||||
|
||||
如果看到类似错误：
|
||||
```
|
||||
failed to open table logs: invalid header
|
||||
```
|
||||
|
||||
**快速修复**：
|
||||
```bash
|
||||
./fix_corrupted_table.sh logs
|
||||
```
|
||||
|
||||
详见：[QUICK_FIX.md](./QUICK_FIX.md) 或 [TROUBLESHOOTING.md](./TROUBLESHOOTING.md)
|
||||
|
||||
## 更多信息
|
||||
|
||||
- [FEATURES.md](./FEATURES.md) - 详细功能说明
|
||||
- [COMPACTION.md](./COMPACTION.md) - Compaction 机制和诊断
|
||||
- [TROUBLESHOOTING.md](./TROUBLESHOOTING.md) - 故障排除指南
|
||||
- [QUICK_FIX.md](./QUICK_FIX.md) - 快速修复常见错误
|
||||
40
examples/webui/commands/check_data.go
Normal file
40
examples/webui/commands/check_data.go
Normal file
@@ -0,0 +1,40 @@
|
||||
package commands
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"log"
|
||||
|
||||
"code.tczkiot.com/srdb"
|
||||
)
|
||||
|
||||
// CheckData 检查数据库中的数据
|
||||
func CheckData(dbPath string) {
|
||||
// 打开数据库
|
||||
db, err := srdb.Open(dbPath)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
defer db.Close()
|
||||
|
||||
// 列出所有表
|
||||
tables := db.ListTables()
|
||||
fmt.Printf("Found %d tables: %v\n", len(tables), tables)
|
||||
|
||||
// 检查每个表的记录数
|
||||
for _, tableName := range tables {
|
||||
table, err := db.GetTable(tableName)
|
||||
if err != nil {
|
||||
fmt.Printf("Error getting table %s: %v\n", tableName, err)
|
||||
continue
|
||||
}
|
||||
|
||||
result, err := table.Query().Rows()
|
||||
if err != nil {
|
||||
fmt.Printf("Error querying table %s: %v\n", tableName, err)
|
||||
continue
|
||||
}
|
||||
|
||||
count := result.Count()
|
||||
fmt.Printf("Table '%s': %d rows\n", tableName, count)
|
||||
}
|
||||
}
|
||||
69
examples/webui/commands/check_seq.go
Normal file
69
examples/webui/commands/check_seq.go
Normal file
@@ -0,0 +1,69 @@
|
||||
package commands
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"log"
|
||||
|
||||
"code.tczkiot.com/srdb"
|
||||
)
|
||||
|
||||
// CheckSeq 检查特定序列号的数据
|
||||
func CheckSeq(dbPath string) {
|
||||
db, err := srdb.Open(dbPath)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
defer db.Close()
|
||||
|
||||
table, err := db.GetTable("logs")
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
// Check seq 1
|
||||
row1, err := table.Get(1)
|
||||
if err != nil {
|
||||
fmt.Printf("Error getting seq=1: %v\n", err)
|
||||
} else if row1 == nil {
|
||||
fmt.Println("Seq=1: NOT FOUND")
|
||||
} else {
|
||||
fmt.Printf("Seq=1: FOUND (time=%d)\n", row1.Time)
|
||||
}
|
||||
|
||||
// Check seq 100
|
||||
row100, err := table.Get(100)
|
||||
if err != nil {
|
||||
fmt.Printf("Error getting seq=100: %v\n", err)
|
||||
} else if row100 == nil {
|
||||
fmt.Println("Seq=100: NOT FOUND")
|
||||
} else {
|
||||
fmt.Printf("Seq=100: FOUND (time=%d)\n", row100.Time)
|
||||
}
|
||||
|
||||
// Check seq 729
|
||||
row729, err := table.Get(729)
|
||||
if err != nil {
|
||||
fmt.Printf("Error getting seq=729: %v\n", err)
|
||||
} else if row729 == nil {
|
||||
fmt.Println("Seq=729: NOT FOUND")
|
||||
} else {
|
||||
fmt.Printf("Seq=729: FOUND (time=%d)\n", row729.Time)
|
||||
}
|
||||
|
||||
// Query all records
|
||||
result, err := table.Query().Rows()
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
count := result.Count()
|
||||
fmt.Printf("\nTotal rows from Query: %d\n", count)
|
||||
|
||||
if count > 0 {
|
||||
first, _ := result.First()
|
||||
if first != nil {
|
||||
data := first.Data()
|
||||
fmt.Printf("First row _seq: %v\n", data["_seq"])
|
||||
}
|
||||
}
|
||||
}
|
||||
58
examples/webui/commands/dump_manifest.go
Normal file
58
examples/webui/commands/dump_manifest.go
Normal file
@@ -0,0 +1,58 @@
|
||||
package commands
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"log"
|
||||
|
||||
"code.tczkiot.com/srdb"
|
||||
)
|
||||
|
||||
// DumpManifest 导出 manifest 信息
|
||||
func DumpManifest(dbPath string) {
|
||||
db, err := srdb.Open(dbPath)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
defer db.Close()
|
||||
|
||||
table, err := db.GetTable("logs")
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
engine := table.GetEngine()
|
||||
versionSet := engine.GetVersionSet()
|
||||
version := versionSet.GetCurrent()
|
||||
|
||||
// Check for duplicates in each level
|
||||
for level := 0; level < 7; level++ {
|
||||
files := version.GetLevel(level)
|
||||
if len(files) == 0 {
|
||||
continue
|
||||
}
|
||||
|
||||
// Track file numbers
|
||||
fileMap := make(map[int64][]struct {
|
||||
minKey int64
|
||||
maxKey int64
|
||||
})
|
||||
|
||||
for _, f := range files {
|
||||
fileMap[f.FileNumber] = append(fileMap[f.FileNumber], struct {
|
||||
minKey int64
|
||||
maxKey int64
|
||||
}{f.MinKey, f.MaxKey})
|
||||
}
|
||||
|
||||
// Report duplicates
|
||||
fmt.Printf("Level %d: %d files\n", level, len(files))
|
||||
for fileNum, entries := range fileMap {
|
||||
if len(entries) > 1 {
|
||||
fmt.Printf(" [DUPLICATE] File #%d appears %d times:\n", fileNum, len(entries))
|
||||
for i, e := range entries {
|
||||
fmt.Printf(" Entry %d: min=%d max=%d\n", i+1, e.minKey, e.maxKey)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
72
examples/webui/commands/inspect_all_sst.go
Normal file
72
examples/webui/commands/inspect_all_sst.go
Normal file
@@ -0,0 +1,72 @@
|
||||
package commands
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"log"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"code.tczkiot.com/srdb/sst"
|
||||
)
|
||||
|
||||
// InspectAllSST 检查所有 SST 文件
|
||||
func InspectAllSST(sstDir string) {
|
||||
// List all SST files
|
||||
files, err := os.ReadDir(sstDir)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
var sstFiles []string
|
||||
for _, file := range files {
|
||||
if strings.HasSuffix(file.Name(), ".sst") {
|
||||
sstFiles = append(sstFiles, file.Name())
|
||||
}
|
||||
}
|
||||
|
||||
sort.Strings(sstFiles)
|
||||
|
||||
fmt.Printf("Found %d SST files\n\n", len(sstFiles))
|
||||
|
||||
// Inspect each file
|
||||
for _, filename := range sstFiles {
|
||||
sstPath := filepath.Join(sstDir, filename)
|
||||
|
||||
reader, err := sst.NewReader(sstPath)
|
||||
if err != nil {
|
||||
fmt.Printf("%s: ERROR - %v\n", filename, err)
|
||||
continue
|
||||
}
|
||||
|
||||
header := reader.GetHeader()
|
||||
allKeys := reader.GetAllKeys()
|
||||
|
||||
// Extract file number
|
||||
numStr := strings.TrimPrefix(filename, "000")
|
||||
numStr = strings.TrimPrefix(numStr, "00")
|
||||
numStr = strings.TrimPrefix(numStr, "0")
|
||||
numStr = strings.TrimSuffix(numStr, ".sst")
|
||||
fileNum, _ := strconv.Atoi(numStr)
|
||||
|
||||
fmt.Printf("File #%d (%s):\n", fileNum, filename)
|
||||
fmt.Printf(" Header: MinKey=%d MaxKey=%d RowCount=%d\n", header.MinKey, header.MaxKey, header.RowCount)
|
||||
fmt.Printf(" Actual: %d keys", len(allKeys))
|
||||
if len(allKeys) > 0 {
|
||||
fmt.Printf(" [%d ... %d]", allKeys[0], allKeys[len(allKeys)-1])
|
||||
}
|
||||
fmt.Printf("\n")
|
||||
|
||||
// Check if header matches actual keys
|
||||
if len(allKeys) > 0 {
|
||||
if header.MinKey != allKeys[0] || header.MaxKey != allKeys[len(allKeys)-1] {
|
||||
fmt.Printf(" *** MISMATCH: Header says %d-%d but file has %d-%d ***\n",
|
||||
header.MinKey, header.MaxKey, allKeys[0], allKeys[len(allKeys)-1])
|
||||
}
|
||||
}
|
||||
|
||||
reader.Close()
|
||||
}
|
||||
}
|
||||
75
examples/webui/commands/inspect_sst.go
Normal file
75
examples/webui/commands/inspect_sst.go
Normal file
@@ -0,0 +1,75 @@
|
||||
package commands
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"log"
|
||||
"os"
|
||||
|
||||
"code.tczkiot.com/srdb/sst"
|
||||
)
|
||||
|
||||
// InspectSST 检查特定 SST 文件
|
||||
func InspectSST(sstPath string) {
|
||||
// Check if file exists
|
||||
info, err := os.Stat(sstPath)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
fmt.Printf("File: %s\n", sstPath)
|
||||
fmt.Printf("Size: %d bytes\n", info.Size())
|
||||
|
||||
// Open reader
|
||||
reader, err := sst.NewReader(sstPath)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
defer reader.Close()
|
||||
|
||||
// Get header
|
||||
header := reader.GetHeader()
|
||||
fmt.Printf("\nHeader:\n")
|
||||
fmt.Printf(" RowCount: %d\n", header.RowCount)
|
||||
fmt.Printf(" MinKey: %d\n", header.MinKey)
|
||||
fmt.Printf(" MaxKey: %d\n", header.MaxKey)
|
||||
fmt.Printf(" DataSize: %d bytes\n", header.DataSize)
|
||||
|
||||
// Get all keys using GetAllKeys()
|
||||
allKeys := reader.GetAllKeys()
|
||||
fmt.Printf("\nActual keys in file: %d keys\n", len(allKeys))
|
||||
if len(allKeys) > 0 {
|
||||
fmt.Printf(" First key: %d\n", allKeys[0])
|
||||
fmt.Printf(" Last key: %d\n", allKeys[len(allKeys)-1])
|
||||
|
||||
if len(allKeys) <= 30 {
|
||||
fmt.Printf(" All keys: %v\n", allKeys)
|
||||
} else {
|
||||
fmt.Printf(" First 15: %v\n", allKeys[:15])
|
||||
fmt.Printf(" Last 15: %v\n", allKeys[len(allKeys)-15:])
|
||||
}
|
||||
}
|
||||
|
||||
// Try to get a specific key
|
||||
fmt.Printf("\nTrying to get key 332:\n")
|
||||
row, err := reader.Get(332)
|
||||
if err != nil {
|
||||
fmt.Printf(" Error: %v\n", err)
|
||||
} else if row == nil {
|
||||
fmt.Printf(" NULL\n")
|
||||
} else {
|
||||
fmt.Printf(" FOUND: seq=%d, time=%d\n", row.Seq, row.Time)
|
||||
}
|
||||
|
||||
// Try to get key based on actual first key
|
||||
if len(allKeys) > 0 {
|
||||
firstKey := allKeys[0]
|
||||
fmt.Printf("\nTrying to get actual first key %d:\n", firstKey)
|
||||
row, err := reader.Get(firstKey)
|
||||
if err != nil {
|
||||
fmt.Printf(" Error: %v\n", err)
|
||||
} else if row == nil {
|
||||
fmt.Printf(" NULL\n")
|
||||
} else {
|
||||
fmt.Printf(" FOUND: seq=%d, time=%d\n", row.Seq, row.Time)
|
||||
}
|
||||
}
|
||||
}
|
||||
59
examples/webui/commands/test_fix.go
Normal file
59
examples/webui/commands/test_fix.go
Normal file
@@ -0,0 +1,59 @@
|
||||
package commands
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"log"
|
||||
|
||||
"code.tczkiot.com/srdb"
|
||||
)
|
||||
|
||||
// TestFix 测试修复
|
||||
func TestFix(dbPath string) {
|
||||
db, err := srdb.Open(dbPath)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
defer db.Close()
|
||||
|
||||
table, err := db.GetTable("logs")
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
// Get total count
|
||||
result, err := table.Query().Rows()
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
totalCount := result.Count()
|
||||
fmt.Printf("Total rows in Query(): %d\n", totalCount)
|
||||
|
||||
// Test Get() for first 10, middle 10, and last 10
|
||||
testRanges := []struct {
|
||||
name string
|
||||
start int64
|
||||
end int64
|
||||
}{
|
||||
{"First 10", 1, 10},
|
||||
{"Middle 10", 50, 59},
|
||||
{"Last 10", int64(totalCount) - 9, int64(totalCount)},
|
||||
}
|
||||
|
||||
for _, tr := range testRanges {
|
||||
fmt.Printf("\n%s (keys %d-%d):\n", tr.name, tr.start, tr.end)
|
||||
foundCount := 0
|
||||
for seq := tr.start; seq <= tr.end; seq++ {
|
||||
row, err := table.Get(seq)
|
||||
if err != nil {
|
||||
fmt.Printf(" Seq %d: ERROR - %v\n", seq, err)
|
||||
} else if row == nil {
|
||||
fmt.Printf(" Seq %d: NULL\n", seq)
|
||||
} else {
|
||||
foundCount++
|
||||
}
|
||||
}
|
||||
fmt.Printf(" Found: %d/%d\n", foundCount, tr.end-tr.start+1)
|
||||
}
|
||||
|
||||
fmt.Printf("\n✅ If all keys found, the bug is FIXED!\n")
|
||||
}
|
||||
66
examples/webui/commands/test_keys.go
Normal file
66
examples/webui/commands/test_keys.go
Normal file
@@ -0,0 +1,66 @@
|
||||
package commands
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"log"
|
||||
|
||||
"code.tczkiot.com/srdb"
|
||||
)
|
||||
|
||||
// TestKeys 测试键
|
||||
func TestKeys(dbPath string) {
|
||||
db, err := srdb.Open(dbPath)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
defer db.Close()
|
||||
|
||||
table, err := db.GetTable("logs")
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
// Test keys from different ranges
|
||||
testKeys := []int64{
|
||||
1, 100, 331, 332, 350, 400, 447, 500, 600, 700, 800, 850, 861, 862, 900, 1000, 1500, 1665, 1666, 1723,
|
||||
}
|
||||
|
||||
fmt.Println("Testing key existence:")
|
||||
foundCount := 0
|
||||
for _, key := range testKeys {
|
||||
row, err := table.Get(key)
|
||||
if err != nil {
|
||||
fmt.Printf("Key %4d: NOT FOUND (%v)\n", key, err)
|
||||
} else if row == nil {
|
||||
fmt.Printf("Key %4d: NULL\n", key)
|
||||
} else {
|
||||
fmt.Printf("Key %4d: FOUND (time=%d)\n", key, row.Time)
|
||||
foundCount++
|
||||
}
|
||||
}
|
||||
|
||||
fmt.Printf("\nFound %d out of %d test keys\n", foundCount, len(testKeys))
|
||||
|
||||
// Query all
|
||||
result, err := table.Query().Rows()
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
count := result.Count()
|
||||
fmt.Printf("Total rows from Query: %d\n", count)
|
||||
|
||||
if count > 0 {
|
||||
first, _ := result.First()
|
||||
if first != nil {
|
||||
data := first.Data()
|
||||
fmt.Printf("First row _seq: %v\n", data["_seq"])
|
||||
}
|
||||
|
||||
last, _ := result.Last()
|
||||
if last != nil {
|
||||
data := last.Data()
|
||||
fmt.Printf("Last row _seq: %v\n", data["_seq"])
|
||||
}
|
||||
}
|
||||
}
|
||||
192
examples/webui/commands/webui.go
Normal file
192
examples/webui/commands/webui.go
Normal file
@@ -0,0 +1,192 @@
|
||||
package commands
|
||||
|
||||
import (
|
||||
"crypto/rand"
|
||||
"fmt"
|
||||
"log"
|
||||
"math/big"
|
||||
"net/http"
|
||||
"slices"
|
||||
"time"
|
||||
|
||||
"code.tczkiot.com/srdb"
|
||||
"code.tczkiot.com/srdb/webui"
|
||||
)
|
||||
|
||||
// StartWebUI 启动 WebUI 服务器
|
||||
func StartWebUI(dbPath string, addr string) {
|
||||
// 打开数据库
|
||||
db, err := srdb.Open(dbPath)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
defer db.Close()
|
||||
|
||||
// 创建示例 Schema
|
||||
userSchema := srdb.NewSchema("users", []srdb.Field{
|
||||
{Name: "name", Type: srdb.FieldTypeString, Indexed: false, Comment: "User name"},
|
||||
{Name: "email", Type: srdb.FieldTypeString, Indexed: false, Comment: "Email address"},
|
||||
{Name: "age", Type: srdb.FieldTypeInt64, Indexed: false, Comment: "Age"},
|
||||
{Name: "city", Type: srdb.FieldTypeString, Indexed: false, Comment: "City"},
|
||||
})
|
||||
|
||||
productSchema := srdb.NewSchema("products", []srdb.Field{
|
||||
{Name: "product_name", Type: srdb.FieldTypeString, Indexed: false, Comment: "Product name"},
|
||||
{Name: "price", Type: srdb.FieldTypeFloat, Indexed: false, Comment: "Price"},
|
||||
{Name: "quantity", Type: srdb.FieldTypeInt64, Indexed: false, Comment: "Quantity"},
|
||||
{Name: "category", Type: srdb.FieldTypeString, Indexed: false, Comment: "Category"},
|
||||
})
|
||||
|
||||
// 创建表（如果不存在）
|
||||
tables := db.ListTables()
|
||||
hasUsers := false
|
||||
hasProducts := false
|
||||
for _, t := range tables {
|
||||
if t == "users" {
|
||||
hasUsers = true
|
||||
}
|
||||
if t == "products" {
|
||||
hasProducts = true
|
||||
}
|
||||
}
|
||||
|
||||
if !hasUsers {
|
||||
table, err := db.CreateTable("users", userSchema)
|
||||
if err != nil {
|
||||
log.Printf("Create users table failed: %v", err)
|
||||
} else {
|
||||
// 插入一些示例数据
|
||||
users := []map[string]interface{}{
|
||||
{"name": "Alice", "email": "alice@example.com", "age": 30, "city": "Beijing"},
|
||||
{"name": "Bob", "email": "bob@example.com", "age": 25, "city": "Shanghai"},
|
||||
{"name": "Charlie", "email": "charlie@example.com", "age": 35, "city": "Guangzhou"},
|
||||
{"name": "David", "email": "david@example.com", "age": 28, "city": "Shenzhen"},
|
||||
{"name": "Eve", "email": "eve@example.com", "age": 32, "city": "Hangzhou"},
|
||||
}
|
||||
for _, user := range users {
|
||||
table.Insert(user)
|
||||
}
|
||||
log.Printf("Created users table with %d records", len(users))
|
||||
}
|
||||
}
|
||||
|
||||
if !hasProducts {
|
||||
table, err := db.CreateTable("products", productSchema)
|
||||
if err != nil {
|
||||
log.Printf("Create products table failed: %v", err)
|
||||
} else {
|
||||
// 插入一些示例数据
|
||||
products := []map[string]interface{}{
|
||||
{"product_name": "Laptop", "price": 999.99, "quantity": 10, "category": "Electronics"},
|
||||
{"product_name": "Mouse", "price": 29.99, "quantity": 50, "category": "Electronics"},
|
||||
{"product_name": "Keyboard", "price": 79.99, "quantity": 30, "category": "Electronics"},
|
||||
{"product_name": "Monitor", "price": 299.99, "quantity": 15, "category": "Electronics"},
|
||||
{"product_name": "Desk", "price": 199.99, "quantity": 5, "category": "Furniture"},
|
||||
{"product_name": "Chair", "price": 149.99, "quantity": 8, "category": "Furniture"},
|
||||
}
|
||||
for _, product := range products {
|
||||
table.Insert(product)
|
||||
}
|
||||
log.Printf("Created products table with %d records", len(products))
|
||||
}
|
||||
}
|
||||
|
||||
// 启动后台数据插入协程
|
||||
go autoInsertData(db)
|
||||
|
||||
// 启动 Web UI
|
||||
handler := webui.NewWebUI(db)
|
||||
|
||||
fmt.Printf("SRDB Web UI is running at http://%s\n", addr)
|
||||
fmt.Println("Press Ctrl+C to stop")
|
||||
fmt.Println("Background data insertion is running...")
|
||||
|
||||
if err := http.ListenAndServe(addr, handler); err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
// generateRandomData 生成指定大小的随机数据 (2KB ~ 512KB)
|
||||
func generateRandomData() string {
|
||||
minSize := 2 * 1024 // 2KB
|
||||
maxSize := (1 * 1024 * 1024) / 2 // 512KB
|
||||
|
||||
sizeBig, _ := rand.Int(rand.Reader, big.NewInt(int64(maxSize-minSize)))
|
||||
size := int(sizeBig.Int64()) + minSize
|
||||
|
||||
data := make([]byte, size)
|
||||
rand.Read(data)
|
||||
|
||||
return fmt.Sprintf("%x", data)
|
||||
}
|
||||
|
||||
// autoInsertData 在后台自动插入数据
|
||||
func autoInsertData(db *srdb.Database) {
|
||||
ticker := time.NewTicker(time.Second)
|
||||
defer ticker.Stop()
|
||||
|
||||
counter := 1
|
||||
|
||||
for range ticker.C {
|
||||
tables := db.ListTables()
|
||||
var logsTable *srdb.Table
|
||||
|
||||
hasLogs := slices.Contains(tables, "logs")
|
||||
|
||||
if !hasLogs {
|
||||
logsSchema := srdb.NewSchema("logs", []srdb.Field{
|
||||
{Name: "timestamp", Type: srdb.FieldTypeString, Indexed: false, Comment: "Timestamp"},
|
||||
{Name: "data", Type: srdb.FieldTypeString, Indexed: false, Comment: "Random data"},
|
||||
{Name: "size_bytes", Type: srdb.FieldTypeInt64, Indexed: false, Comment: "Data size in bytes"},
|
||||
})
|
||||
|
||||
var err error
|
||||
logsTable, err = db.CreateTable("logs", logsSchema)
|
||||
if err != nil {
|
||||
log.Printf("Failed to create logs table: %v", err)
|
||||
continue
|
||||
}
|
||||
log.Println("Created logs table for background data insertion")
|
||||
} else {
|
||||
var err error
|
||||
logsTable, err = db.GetTable("logs")
|
||||
if err != nil || logsTable == nil {
|
||||
log.Printf("Failed to get logs table: %v", err)
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
data := generateRandomData()
|
||||
sizeBytes := len(data)
|
||||
|
||||
record := map[string]any{
|
||||
"timestamp": time.Now().Format(time.RFC3339),
|
||||
"data": data,
|
||||
"size_bytes": int64(sizeBytes),
|
||||
}
|
||||
|
||||
err := logsTable.Insert(record)
|
||||
if err != nil {
|
||||
log.Printf("Failed to insert data: %v", err)
|
||||
} else {
|
||||
sizeStr := formatBytes(sizeBytes)
|
||||
log.Printf("Inserted record #%d, size: %s", counter, sizeStr)
|
||||
counter++
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// formatBytes 格式化字节大小显示
|
||||
func formatBytes(bytes int) string {
|
||||
const unit = 1024
|
||||
if bytes < unit {
|
||||
return fmt.Sprintf("%d B", bytes)
|
||||
}
|
||||
div, exp := int64(unit), 0
|
||||
for n := bytes / unit; n >= unit; n /= unit {
|
||||
div *= unit
|
||||
exp++
|
||||
}
|
||||
units := []string{"KB", "MB", "GB", "TB"}
|
||||
return fmt.Sprintf("%.2f %s", float64(bytes)/float64(div), units[exp])
|
||||
}
|
||||
98
examples/webui/main.go
Normal file
98
examples/webui/main.go
Normal file
@@ -0,0 +1,98 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"flag"
|
||||
"fmt"
|
||||
"os"
|
||||
|
||||
"code.tczkiot.com/srdb/examples/webui/commands"
|
||||
)
|
||||
|
||||
func main() {
|
||||
if len(os.Args) < 2 {
|
||||
printUsage()
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
command := os.Args[1]
|
||||
args := os.Args[2:]
|
||||
|
||||
switch command {
|
||||
case "webui", "serve":
|
||||
serveCmd := flag.NewFlagSet("webui", flag.ExitOnError)
|
||||
dbPath := serveCmd.String("db", "./data", "Database directory path")
|
||||
addr := serveCmd.String("addr", ":8080", "Server address")
|
||||
serveCmd.Parse(args)
|
||||
commands.StartWebUI(*dbPath, *addr)
|
||||
|
||||
case "check-data":
|
||||
checkDataCmd := flag.NewFlagSet("check-data", flag.ExitOnError)
|
||||
dbPath := checkDataCmd.String("db", "./data", "Database directory path")
|
||||
checkDataCmd.Parse(args)
|
||||
commands.CheckData(*dbPath)
|
||||
|
||||
case "check-seq":
|
||||
checkSeqCmd := flag.NewFlagSet("check-seq", flag.ExitOnError)
|
||||
dbPath := checkSeqCmd.String("db", "./data", "Database directory path")
|
||||
checkSeqCmd.Parse(args)
|
||||
commands.CheckSeq(*dbPath)
|
||||
|
||||
case "dump-manifest":
|
||||
dumpCmd := flag.NewFlagSet("dump-manifest", flag.ExitOnError)
|
||||
dbPath := dumpCmd.String("db", "./data", "Database directory path")
|
||||
dumpCmd.Parse(args)
|
||||
commands.DumpManifest(*dbPath)
|
||||
|
||||
case "inspect-all-sst":
|
||||
inspectAllCmd := flag.NewFlagSet("inspect-all-sst", flag.ExitOnError)
|
||||
sstDir := inspectAllCmd.String("dir", "./data/logs/sst", "SST directory path")
|
||||
inspectAllCmd.Parse(args)
|
||||
commands.InspectAllSST(*sstDir)
|
||||
|
||||
case "inspect-sst":
|
||||
inspectCmd := flag.NewFlagSet("inspect-sst", flag.ExitOnError)
|
||||
sstPath := inspectCmd.String("file", "./data/logs/sst/000046.sst", "SST file path")
|
||||
inspectCmd.Parse(args)
|
||||
commands.InspectSST(*sstPath)
|
||||
|
||||
case "test-fix":
|
||||
testFixCmd := flag.NewFlagSet("test-fix", flag.ExitOnError)
|
||||
dbPath := testFixCmd.String("db", "./data", "Database directory path")
|
||||
testFixCmd.Parse(args)
|
||||
commands.TestFix(*dbPath)
|
||||
|
||||
case "test-keys":
|
||||
testKeysCmd := flag.NewFlagSet("test-keys", flag.ExitOnError)
|
||||
dbPath := testKeysCmd.String("db", "./data", "Database directory path")
|
||||
testKeysCmd.Parse(args)
|
||||
commands.TestKeys(*dbPath)
|
||||
|
||||
case "help", "-h", "--help":
|
||||
printUsage()
|
||||
|
||||
default:
|
||||
fmt.Printf("Unknown command: %s\n\n", command)
|
||||
printUsage()
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
func printUsage() {
|
||||
fmt.Println("SRDB WebUI - Database management tool")
|
||||
fmt.Println("\nUsage:")
|
||||
fmt.Println(" webui <command> [flags]")
|
||||
fmt.Println("\nCommands:")
|
||||
fmt.Println(" webui, serve Start WebUI server (default: :8080)")
|
||||
fmt.Println(" check-data Check database tables and row counts")
|
||||
fmt.Println(" check-seq Check specific sequence numbers")
|
||||
fmt.Println(" dump-manifest Dump manifest information")
|
||||
fmt.Println(" inspect-all-sst Inspect all SST files")
|
||||
fmt.Println(" inspect-sst Inspect a specific SST file")
|
||||
fmt.Println(" test-fix Test fix for data retrieval")
|
||||
fmt.Println(" test-keys Test key existence")
|
||||
fmt.Println(" help Show this help message")
|
||||
fmt.Println("\nExamples:")
|
||||
fmt.Println(" webui serve -db ./mydb -addr :3000")
|
||||
fmt.Println(" webui check-data -db ./mydb")
|
||||
fmt.Println(" webui inspect-sst -file ./data/logs/sst/000046.sst")
|
||||
}
|
||||
10
go.mod
Normal file
10
go.mod
Normal file
@@ -0,0 +1,10 @@
|
||||
module code.tczkiot.com/srdb
|
||||
|
||||
go 1.24.0
|
||||
|
||||
require (
|
||||
github.com/edsrzf/mmap-go v1.1.0
|
||||
github.com/golang/snappy v1.0.0
|
||||
)
|
||||
|
||||
require golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e // indirect
|
||||
6
go.sum
Normal file
6
go.sum
Normal file
@@ -0,0 +1,6 @@
|
||||
github.com/edsrzf/mmap-go v1.1.0 h1:6EUwBLQ/Mcr1EYLE4Tn1VdW1A4ckqCQWZBw8Hr0kjpQ=
|
||||
github.com/edsrzf/mmap-go v1.1.0/go.mod h1:19H/e8pUPLicwkyNgOykDXkJ9F0MHE+Z52B8EIth78Q=
|
||||
github.com/golang/snappy v1.0.0 h1:Oy607GVXHs7RtbggtPBnr2RmDArIsAefDwvrdWvRhGs=
|
||||
github.com/golang/snappy v1.0.0/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
|
||||
golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e h1:fLOSk5Q00efkSvAm+4xcoXD+RRmLmmulPn5I3Y9F2EM=
|
||||
golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
528
index.go
Normal file
528
index.go
Normal file
@@ -0,0 +1,528 @@
|
||||
package srdb
|
||||
|
||||
import (
|
||||
"encoding/binary"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"code.tczkiot.com/srdb/btree"
|
||||
)
|
||||
|
||||
// IndexMetadata 索引元数据
|
||||
type IndexMetadata struct {
|
||||
Version int64 // 索引版本号
|
||||
MaxSeq int64 // 索引包含的最大 seq
|
||||
MinSeq int64 // 索引包含的最小 seq
|
||||
RowCount int64 // 索引包含的行数
|
||||
CreatedAt int64 // 创建时间
|
||||
UpdatedAt int64 // 更新时间
|
||||
}
|
||||
|
||||
// SecondaryIndex 二级索引
|
||||
type SecondaryIndex struct {
|
||||
name string // 索引名称
|
||||
field string // 字段名
|
||||
fieldType FieldType // 字段类型
|
||||
file *os.File // 索引文件
|
||||
builder *btree.Builder // B+Tree 构建器
|
||||
reader *btree.Reader // B+Tree 读取器
|
||||
valueToSeq map[string][]int64 // 值 → seq 列表 (构建时使用)
|
||||
metadata IndexMetadata // 元数据
|
||||
mu sync.RWMutex
|
||||
ready bool // 索引是否就绪
|
||||
}
|
||||
|
||||
// NewSecondaryIndex 创建二级索引
|
||||
func NewSecondaryIndex(dir, field string, fieldType FieldType) (*SecondaryIndex, error) {
|
||||
indexPath := filepath.Join(dir, fmt.Sprintf("idx_%s.sst", field))
|
||||
file, err := os.OpenFile(indexPath, os.O_CREATE|os.O_RDWR, 0644)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return &SecondaryIndex{
|
||||
name: field,
|
||||
field: field,
|
||||
fieldType: fieldType,
|
||||
file: file,
|
||||
valueToSeq: make(map[string][]int64),
|
||||
ready: false,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// Add 添加索引条目
|
||||
func (idx *SecondaryIndex) Add(value interface{}, seq int64) error {
|
||||
idx.mu.Lock()
|
||||
defer idx.mu.Unlock()
|
||||
|
||||
// 将值转换为字符串作为 key
|
||||
key := fmt.Sprintf("%v", value)
|
||||
idx.valueToSeq[key] = append(idx.valueToSeq[key], seq)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// Build 构建索引并持久化
|
||||
func (idx *SecondaryIndex) Build() error {
|
||||
idx.mu.Lock()
|
||||
defer idx.mu.Unlock()
|
||||
|
||||
// 持久化索引数据到 JSON 文件
|
||||
return idx.save()
|
||||
}
|
||||
|
||||
// save 保存索引到磁盘
|
||||
func (idx *SecondaryIndex) save() error {
|
||||
// 更新元数据
|
||||
idx.updateMetadata()
|
||||
|
||||
// 创建包含元数据的数据结构
|
||||
indexData := struct {
|
||||
Metadata IndexMetadata `json:"metadata"`
|
||||
ValueToSeq map[string][]int64 `json:"data"`
|
||||
}{
|
||||
Metadata: idx.metadata,
|
||||
ValueToSeq: idx.valueToSeq,
|
||||
}
|
||||
|
||||
// 序列化索引数据
|
||||
data, err := json.Marshal(indexData)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Truncate 文件
|
||||
err = idx.file.Truncate(0)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// 写入文件
|
||||
_, err = idx.file.Seek(0, 0)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
_, err = idx.file.Write(data)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Sync 到磁盘
|
||||
err = idx.file.Sync()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
idx.ready = true
|
||||
return nil
|
||||
}
|
||||
|
||||
// updateMetadata 更新元数据
|
||||
func (idx *SecondaryIndex) updateMetadata() {
|
||||
now := time.Now().UnixNano()
|
||||
|
||||
if idx.metadata.CreatedAt == 0 {
|
||||
idx.metadata.CreatedAt = now
|
||||
}
|
||||
idx.metadata.UpdatedAt = now
|
||||
idx.metadata.Version++
|
||||
|
||||
// 计算 MinSeq, MaxSeq, RowCount
|
||||
var minSeq, maxSeq int64 = -1, -1
|
||||
rowCount := int64(0)
|
||||
|
||||
for _, seqs := range idx.valueToSeq {
|
||||
for _, seq := range seqs {
|
||||
if minSeq == -1 || seq < minSeq {
|
||||
minSeq = seq
|
||||
}
|
||||
if maxSeq == -1 || seq > maxSeq {
|
||||
maxSeq = seq
|
||||
}
|
||||
rowCount++
|
||||
}
|
||||
}
|
||||
|
||||
idx.metadata.MinSeq = minSeq
|
||||
idx.metadata.MaxSeq = maxSeq
|
||||
idx.metadata.RowCount = rowCount
|
||||
}
|
||||
|
||||
// load 从磁盘加载索引
|
||||
func (idx *SecondaryIndex) load() error {
|
||||
// 获取文件大小
|
||||
stat, err := idx.file.Stat()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if stat.Size() == 0 {
|
||||
// 空文件，索引不存在
|
||||
return nil
|
||||
}
|
||||
|
||||
// 读取文件内容
|
||||
data := make([]byte, stat.Size())
|
||||
_, err = idx.file.ReadAt(data, 0)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// 尝试加载新格式（带元数据）
|
||||
var indexData struct {
|
||||
Metadata IndexMetadata `json:"metadata"`
|
||||
ValueToSeq map[string][]int64 `json:"data"`
|
||||
}
|
||||
|
||||
err = json.Unmarshal(data, &indexData)
|
||||
if err == nil && indexData.ValueToSeq != nil {
|
||||
// 新格式
|
||||
idx.metadata = indexData.Metadata
|
||||
idx.valueToSeq = indexData.ValueToSeq
|
||||
} else {
|
||||
// 旧格式（兼容性）
|
||||
err = json.Unmarshal(data, &idx.valueToSeq)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// 初始化元数据
|
||||
idx.updateMetadata()
|
||||
}
|
||||
|
||||
idx.ready = true
|
||||
return nil
|
||||
}
|
||||
|
||||
// Get 查询索引
|
||||
func (idx *SecondaryIndex) Get(value interface{}) ([]int64, error) {
|
||||
idx.mu.RLock()
|
||||
defer idx.mu.RUnlock()
|
||||
|
||||
if !idx.ready {
|
||||
return nil, fmt.Errorf("index not ready")
|
||||
}
|
||||
|
||||
key := fmt.Sprintf("%v", value)
|
||||
seqs, exists := idx.valueToSeq[key]
|
||||
if !exists {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
return seqs, nil
|
||||
}
|
||||
|
||||
// IsReady 索引是否就绪
|
||||
func (idx *SecondaryIndex) IsReady() bool {
|
||||
idx.mu.RLock()
|
||||
defer idx.mu.RUnlock()
|
||||
return idx.ready
|
||||
}
|
||||
|
||||
// GetMetadata 获取元数据
|
||||
func (idx *SecondaryIndex) GetMetadata() IndexMetadata {
|
||||
idx.mu.RLock()
|
||||
defer idx.mu.RUnlock()
|
||||
return idx.metadata
|
||||
}
|
||||
|
||||
// NeedsUpdate 检查是否需要更新
|
||||
func (idx *SecondaryIndex) NeedsUpdate(currentMaxSeq int64) bool {
|
||||
idx.mu.RLock()
|
||||
defer idx.mu.RUnlock()
|
||||
return idx.metadata.MaxSeq < currentMaxSeq
|
||||
}
|
||||
|
||||
// IncrementalUpdate 增量更新索引
|
||||
func (idx *SecondaryIndex) IncrementalUpdate(getData func(int64) (map[string]interface{}, error), fromSeq, toSeq int64) error {
|
||||
idx.mu.Lock()
|
||||
defer idx.mu.Unlock()
|
||||
|
||||
// 遍历缺失的 seq 范围
|
||||
for seq := fromSeq; seq <= toSeq; seq++ {
|
||||
// 获取数据
|
||||
data, err := getData(seq)
|
||||
if err != nil {
|
||||
continue // 跳过错误的数据
|
||||
}
|
||||
|
||||
// 提取字段值
|
||||
value, exists := data[idx.field]
|
||||
if !exists {
|
||||
continue
|
||||
}
|
||||
|
||||
// 添加到索引
|
||||
key := fmt.Sprintf("%v", value)
|
||||
idx.valueToSeq[key] = append(idx.valueToSeq[key], seq)
|
||||
}
|
||||
|
||||
// 保存更新后的索引
|
||||
return idx.save()
|
||||
}
|
||||
|
||||
// Close 关闭索引
|
||||
func (idx *SecondaryIndex) Close() error {
|
||||
if idx.file != nil {
|
||||
return idx.file.Close()
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// encodeSeqList 编码 seq 列表
|
||||
func encodeSeqList(seqs []int64) []byte {
|
||||
buf := make([]byte, 8*len(seqs))
|
||||
for i, seq := range seqs {
|
||||
binary.LittleEndian.PutUint64(buf[i*8:], uint64(seq))
|
||||
}
|
||||
return buf
|
||||
}
|
||||
|
||||
// decodeSeqList 解码 seq 列表
|
||||
func decodeSeqList(data []byte) []int64 {
|
||||
count := len(data) / 8
|
||||
seqs := make([]int64, count)
|
||||
for i := 0; i < count; i++ {
|
||||
seqs[i] = int64(binary.LittleEndian.Uint64(data[i*8:]))
|
||||
}
|
||||
return seqs
|
||||
}
|
||||
|
||||
// IndexManager 索引管理器
|
||||
type IndexManager struct {
|
||||
dir string
|
||||
schema *Schema
|
||||
indexes map[string]*SecondaryIndex // field → index
|
||||
mu sync.RWMutex
|
||||
}
|
||||
|
||||
// NewIndexManager 创建索引管理器
|
||||
func NewIndexManager(dir string, schema *Schema) *IndexManager {
|
||||
mgr := &IndexManager{
|
||||
dir: dir,
|
||||
schema: schema,
|
||||
indexes: make(map[string]*SecondaryIndex),
|
||||
}
|
||||
|
||||
// 自动加载已存在的索引
|
||||
mgr.loadExistingIndexes()
|
||||
|
||||
return mgr
|
||||
}
|
||||
|
||||
// loadExistingIndexes 加载已存在的索引文件
|
||||
func (m *IndexManager) loadExistingIndexes() error {
|
||||
// 确保目录存在
|
||||
if _, err := os.Stat(m.dir); os.IsNotExist(err) {
|
||||
return nil // 目录不存在，跳过
|
||||
}
|
||||
|
||||
// 查找所有索引文件
|
||||
pattern := filepath.Join(m.dir, "idx_*.sst")
|
||||
files, err := filepath.Glob(pattern)
|
||||
if err != nil {
|
||||
return nil // 忽略错误，继续
|
||||
}
|
||||
|
||||
for _, filePath := range files {
|
||||
// 从文件名提取字段名
|
||||
// idx_name.sst -> name
|
||||
filename := filepath.Base(filePath)
|
||||
if len(filename) < 8 { // "idx_" (4) + ".sst" (4)
|
||||
continue
|
||||
}
|
||||
field := filename[4 : len(filename)-4] // 去掉 "idx_" 和 ".sst"
|
||||
|
||||
// 检查字段是否在 Schema 中
|
||||
fieldDef, err := m.schema.GetField(field)
|
||||
if err != nil {
|
||||
continue // 跳过不在 Schema 中的索引
|
||||
}
|
||||
|
||||
// 打开索引文件
|
||||
file, err := os.OpenFile(filePath, os.O_RDWR, 0644)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
|
||||
// 创建索引对象
|
||||
idx := &SecondaryIndex{
|
||||
name: field,
|
||||
field: field,
|
||||
fieldType: fieldDef.Type,
|
||||
file: file,
|
||||
valueToSeq: make(map[string][]int64),
|
||||
ready: false,
|
||||
}
|
||||
|
||||
// 加载索引数据
|
||||
err = idx.load()
|
||||
if err != nil {
|
||||
file.Close()
|
||||
continue
|
||||
}
|
||||
|
||||
m.indexes[field] = idx
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// CreateIndex 创建索引
|
||||
func (m *IndexManager) CreateIndex(field string) error {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
|
||||
// 检查字段是否存在
|
||||
fieldDef, err := m.schema.GetField(field)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// 检查是否已存在
|
||||
if _, exists := m.indexes[field]; exists {
|
||||
return fmt.Errorf("index on field %s already exists", field)
|
||||
}
|
||||
|
||||
// 创建索引
|
||||
idx, err := NewSecondaryIndex(m.dir, field, fieldDef.Type)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
m.indexes[field] = idx
|
||||
return nil
|
||||
}
|
||||
|
||||
// DropIndex 删除索引
|
||||
func (m *IndexManager) DropIndex(field string) error {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
|
||||
idx, exists := m.indexes[field]
|
||||
if !exists {
|
||||
return fmt.Errorf("index on field %s does not exist", field)
|
||||
}
|
||||
|
||||
// 获取文件路径
|
||||
indexPath := filepath.Join(m.dir, fmt.Sprintf("idx_%s.sst", field))
|
||||
|
||||
// 关闭索引
|
||||
idx.Close()
|
||||
|
||||
// 删除索引文件
|
||||
os.Remove(indexPath)
|
||||
|
||||
// 从内存中删除
|
||||
delete(m.indexes, field)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// GetIndex 获取索引
|
||||
func (m *IndexManager) GetIndex(field string) (*SecondaryIndex, bool) {
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
|
||||
idx, exists := m.indexes[field]
|
||||
return idx, exists
|
||||
}
|
||||
|
||||
// AddToIndexes 添加到所有索引
|
||||
func (m *IndexManager) AddToIndexes(data map[string]interface{}, seq int64) error {
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
|
||||
for field, idx := range m.indexes {
|
||||
if value, exists := data[field]; exists {
|
||||
err := idx.Add(value, seq)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// BuildAll 构建所有索引
|
||||
func (m *IndexManager) BuildAll() error {
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
|
||||
for _, idx := range m.indexes {
|
||||
err := idx.Build()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// ListIndexes 列出所有索引
|
||||
func (m *IndexManager) ListIndexes() []string {
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
|
||||
fields := make([]string, 0, len(m.indexes))
|
||||
for field := range m.indexes {
|
||||
fields = append(fields, field)
|
||||
}
|
||||
return fields
|
||||
}
|
||||
|
||||
// VerifyAndRepair 验证并修复所有索引
|
||||
func (m *IndexManager) VerifyAndRepair(currentMaxSeq int64, getData func(int64) (map[string]interface{}, error)) error {
|
||||
m.mu.RLock()
|
||||
indexes := make(map[string]*SecondaryIndex)
|
||||
for k, v := range m.indexes {
|
||||
indexes[k] = v
|
||||
}
|
||||
m.mu.RUnlock()
|
||||
|
||||
for field, idx := range indexes {
|
||||
// 检查是否需要更新
|
||||
if idx.NeedsUpdate(currentMaxSeq) {
|
||||
metadata := idx.GetMetadata()
|
||||
fromSeq := metadata.MaxSeq + 1
|
||||
toSeq := currentMaxSeq
|
||||
|
||||
// 增量更新
|
||||
err := idx.IncrementalUpdate(getData, fromSeq, toSeq)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to update index %s: %v", field, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// GetIndexMetadata 获取所有索引的元数据
|
||||
func (m *IndexManager) GetIndexMetadata() map[string]IndexMetadata {
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
|
||||
metadata := make(map[string]IndexMetadata)
|
||||
for field, idx := range m.indexes {
|
||||
metadata[field] = idx.GetMetadata()
|
||||
}
|
||||
return metadata
|
||||
}
|
||||
|
||||
// Close 关闭所有索引
|
||||
func (m *IndexManager) Close() error {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
|
||||
for _, idx := range m.indexes {
|
||||
idx.Close()
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
286
index_test.go
Normal file
286
index_test.go
Normal file
@@ -0,0 +1,286 @@
|
||||
package srdb
|
||||
|
||||
import (
|
||||
"os"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestIndexVersionControl(t *testing.T) {
|
||||
dir := "test_index_version"
|
||||
os.RemoveAll(dir)
|
||||
os.MkdirAll(dir, 0755)
|
||||
defer os.RemoveAll(dir)
|
||||
|
||||
testSchema := NewSchema("test", []Field{
|
||||
{Name: "name", Type: FieldTypeString, Indexed: true, Comment: "名称"},
|
||||
})
|
||||
|
||||
// 1. 创建索引管理器
|
||||
mgr := NewIndexManager(dir, testSchema)
|
||||
|
||||
// 2. 创建索引
|
||||
mgr.CreateIndex("name")
|
||||
idx, _ := mgr.GetIndex("name")
|
||||
|
||||
// 3. 添加数据
|
||||
idx.Add("Alice", 1)
|
||||
idx.Add("Bob", 2)
|
||||
idx.Add("Alice", 3)
|
||||
|
||||
// 4. 保存索引
|
||||
idx.Build()
|
||||
|
||||
// 5. 检查元数据
|
||||
metadata := idx.GetMetadata()
|
||||
if metadata.Version != 1 {
|
||||
t.Errorf("Expected version 1, got %d", metadata.Version)
|
||||
}
|
||||
if metadata.MinSeq != 1 {
|
||||
t.Errorf("Expected MinSeq 1, got %d", metadata.MinSeq)
|
||||
}
|
||||
if metadata.MaxSeq != 3 {
|
||||
t.Errorf("Expected MaxSeq 3, got %d", metadata.MaxSeq)
|
||||
}
|
||||
if metadata.RowCount != 3 {
|
||||
t.Errorf("Expected RowCount 3, got %d", metadata.RowCount)
|
||||
}
|
||||
|
||||
t.Logf("Metadata: Version=%d, MinSeq=%d, MaxSeq=%d, RowCount=%d",
|
||||
metadata.Version, metadata.MinSeq, metadata.MaxSeq, metadata.RowCount)
|
||||
|
||||
// 6. 关闭并重新加载
|
||||
mgr.Close()
|
||||
|
||||
mgr2 := NewIndexManager(dir, testSchema)
|
||||
idx2, _ := mgr2.GetIndex("name")
|
||||
|
||||
// 7. 验证元数据被正确加载
|
||||
metadata2 := idx2.GetMetadata()
|
||||
if metadata2.Version != metadata.Version {
|
||||
t.Errorf("Version mismatch after reload")
|
||||
}
|
||||
if metadata2.MaxSeq != metadata.MaxSeq {
|
||||
t.Errorf("MaxSeq mismatch after reload")
|
||||
}
|
||||
|
||||
t.Log("索引版本控制测试通过！")
|
||||
}
|
||||
|
||||
func TestIncrementalUpdate(t *testing.T) {
|
||||
dir := "test_incremental_update"
|
||||
os.RemoveAll(dir)
|
||||
os.MkdirAll(dir, 0755)
|
||||
defer os.RemoveAll(dir)
|
||||
|
||||
testSchema := NewSchema("test", []Field{
|
||||
{Name: "name", Type: FieldTypeString, Indexed: true, Comment: "名称"},
|
||||
})
|
||||
|
||||
// 1. 创建索引并添加初始数据
|
||||
mgr := NewIndexManager(dir, testSchema)
|
||||
mgr.CreateIndex("name")
|
||||
idx, _ := mgr.GetIndex("name")
|
||||
|
||||
idx.Add("Alice", 1)
|
||||
idx.Add("Bob", 2)
|
||||
idx.Build()
|
||||
|
||||
initialMetadata := idx.GetMetadata()
|
||||
t.Logf("Initial: MaxSeq=%d, RowCount=%d", initialMetadata.MaxSeq, initialMetadata.RowCount)
|
||||
|
||||
// 2. 模拟新数据
|
||||
mockData := map[int64]map[string]interface{}{
|
||||
3: {"name": "Charlie"},
|
||||
4: {"name": "David"},
|
||||
5: {"name": "Alice"},
|
||||
}
|
||||
|
||||
getData := func(seq int64) (map[string]interface{}, error) {
|
||||
if data, exists := mockData[seq]; exists {
|
||||
return data, nil
|
||||
}
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
// 3. 增量更新
|
||||
err := idx.IncrementalUpdate(getData, 3, 5)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// 4. 验证更新后的元数据
|
||||
updatedMetadata := idx.GetMetadata()
|
||||
if updatedMetadata.MaxSeq != 5 {
|
||||
t.Errorf("Expected MaxSeq 5, got %d", updatedMetadata.MaxSeq)
|
||||
}
|
||||
if updatedMetadata.RowCount != 5 {
|
||||
t.Errorf("Expected RowCount 5, got %d", updatedMetadata.RowCount)
|
||||
}
|
||||
if updatedMetadata.Version != 2 {
|
||||
t.Errorf("Expected Version 2, got %d", updatedMetadata.Version)
|
||||
}
|
||||
|
||||
t.Logf("Updated: MaxSeq=%d, RowCount=%d, Version=%d",
|
||||
updatedMetadata.MaxSeq, updatedMetadata.RowCount, updatedMetadata.Version)
|
||||
|
||||
// 5. 验证数据
|
||||
seqs, _ := idx.Get("Alice")
|
||||
if len(seqs) != 2 {
|
||||
t.Errorf("Expected 2 seqs for Alice, got %d", len(seqs))
|
||||
}
|
||||
|
||||
seqs, _ = idx.Get("Charlie")
|
||||
if len(seqs) != 1 {
|
||||
t.Errorf("Expected 1 seq for Charlie, got %d", len(seqs))
|
||||
}
|
||||
|
||||
t.Log("增量更新测试通过！")
|
||||
}
|
||||
|
||||
func TestNeedsUpdate(t *testing.T) {
|
||||
dir := "test_needs_update"
|
||||
os.RemoveAll(dir)
|
||||
os.MkdirAll(dir, 0755)
|
||||
defer os.RemoveAll(dir)
|
||||
|
||||
testSchema := NewSchema("test", []Field{
|
||||
{Name: "name", Type: FieldTypeString, Indexed: true, Comment: "名称"},
|
||||
})
|
||||
|
||||
mgr := NewIndexManager(dir, testSchema)
|
||||
mgr.CreateIndex("name")
|
||||
idx, _ := mgr.GetIndex("name")
|
||||
|
||||
idx.Add("Alice", 1)
|
||||
idx.Add("Bob", 2)
|
||||
idx.Build()
|
||||
|
||||
// 测试 NeedsUpdate
|
||||
if idx.NeedsUpdate(2) {
|
||||
t.Error("Should not need update when currentMaxSeq = 2")
|
||||
}
|
||||
|
||||
if !idx.NeedsUpdate(5) {
|
||||
t.Error("Should need update when currentMaxSeq = 5")
|
||||
}
|
||||
|
||||
t.Log("NeedsUpdate 测试通过！")
|
||||
}
|
||||
|
||||
func TestIndexPersistence(t *testing.T) {
|
||||
dir := "test_index_persistence"
|
||||
os.RemoveAll(dir)
|
||||
os.MkdirAll(dir, 0755)
|
||||
defer os.RemoveAll(dir)
|
||||
|
||||
// 创建 Schema
|
||||
testSchema := NewSchema("test", []Field{
|
||||
{Name: "name", Type: FieldTypeString, Indexed: true, Comment: "名称"},
|
||||
{Name: "age", Type: FieldTypeInt64, Indexed: true, Comment: "年龄"},
|
||||
})
|
||||
|
||||
// 1. 创建索引管理器
|
||||
mgr := NewIndexManager(dir, testSchema)
|
||||
|
||||
// 2. 创建索引
|
||||
err := mgr.CreateIndex("name")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// 3. 添加数据到索引
|
||||
idx, _ := mgr.GetIndex("name")
|
||||
idx.Add("Alice", 1)
|
||||
idx.Add("Bob", 2)
|
||||
idx.Add("Alice", 3)
|
||||
idx.Add("Charlie", 4)
|
||||
|
||||
// 4. 构建并保存索引
|
||||
err = idx.Build()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
t.Log("索引已保存到磁盘")
|
||||
|
||||
// 5. 关闭管理器
|
||||
mgr.Close()
|
||||
|
||||
// 6. 创建新的管理器（模拟重启）
|
||||
mgr2 := NewIndexManager(dir, testSchema)
|
||||
|
||||
// 7. 检查索引是否自动加载
|
||||
indexes := mgr2.ListIndexes()
|
||||
if len(indexes) != 1 {
|
||||
t.Errorf("Expected 1 index, got %d", len(indexes))
|
||||
}
|
||||
|
||||
// 8. 验证索引数据
|
||||
idx2, exists := mgr2.GetIndex("name")
|
||||
if !exists {
|
||||
t.Fatal("Index 'name' not found after reload")
|
||||
}
|
||||
|
||||
if !idx2.IsReady() {
|
||||
t.Error("Index should be ready after reload")
|
||||
}
|
||||
|
||||
// 9. 查询索引
|
||||
seqs, err := idx2.Get("Alice")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
if len(seqs) != 2 {
|
||||
t.Errorf("Expected 2 seqs for 'Alice', got %d", len(seqs))
|
||||
}
|
||||
|
||||
seqs, err = idx2.Get("Bob")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
if len(seqs) != 1 {
|
||||
t.Errorf("Expected 1 seq for 'Bob', got %d", len(seqs))
|
||||
}
|
||||
|
||||
t.Log("索引持久化测试通过！")
|
||||
}
|
||||
|
||||
func TestIndexDropWithFile(t *testing.T) {
|
||||
dir := "test_index_drop"
|
||||
os.RemoveAll(dir)
|
||||
os.MkdirAll(dir, 0755)
|
||||
defer os.RemoveAll(dir)
|
||||
|
||||
testSchema := NewSchema("test", []Field{
|
||||
{Name: "name", Type: FieldTypeString, Indexed: true, Comment: "名称"},
|
||||
})
|
||||
|
||||
mgr := NewIndexManager(dir, testSchema)
|
||||
|
||||
// 创建索引
|
||||
mgr.CreateIndex("name")
|
||||
idx, _ := mgr.GetIndex("name")
|
||||
idx.Add("Alice", 1)
|
||||
idx.Build()
|
||||
|
||||
// 检查文件是否存在
|
||||
indexPath := dir + "/idx_name.sst"
|
||||
if _, err := os.Stat(indexPath); os.IsNotExist(err) {
|
||||
t.Error("Index file should exist")
|
||||
}
|
||||
|
||||
// 删除索引
|
||||
err := mgr.DropIndex("name")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// 检查文件是否被删除
|
||||
if _, err := os.Stat(indexPath); !os.IsNotExist(err) {
|
||||
t.Error("Index file should be deleted")
|
||||
}
|
||||
|
||||
t.Log("索引删除测试通过！")
|
||||
}
|
||||
48
manifest/manifest_reader.go
Normal file
48
manifest/manifest_reader.go
Normal file
@@ -0,0 +1,48 @@
|
||||
package manifest
|
||||
|
||||
import (
|
||||
"encoding/binary"
|
||||
"io"
|
||||
)
|
||||
|
||||
// Reader MANIFEST 读取器
|
||||
type Reader struct {
|
||||
file io.Reader
|
||||
}
|
||||
|
||||
// NewReader 创建 MANIFEST 读取器
|
||||
func NewReader(file io.Reader) *Reader {
|
||||
return &Reader{
|
||||
file: file,
|
||||
}
|
||||
}
|
||||
|
||||
// ReadEdit 读取版本变更
|
||||
func (r *Reader) ReadEdit() (*VersionEdit, error) {
|
||||
// 读取 CRC32 和 Length
|
||||
header := make([]byte, 8)
|
||||
_, err := io.ReadFull(r.file, header)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// 读取长度
|
||||
length := binary.LittleEndian.Uint32(header[4:8])
|
||||
|
||||
// 读取数据
|
||||
data := make([]byte, 8+length)
|
||||
copy(data[0:8], header)
|
||||
_, err = io.ReadFull(r.file, data[8:])
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// 解码
|
||||
edit := NewVersionEdit()
|
||||
err = edit.Decode(data)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return edit, nil
|
||||
}
|
||||
35
manifest/manifest_writer.go
Normal file
35
manifest/manifest_writer.go
Normal file
@@ -0,0 +1,35 @@
|
||||
package manifest
|
||||
|
||||
import (
|
||||
"io"
|
||||
"sync"
|
||||
)
|
||||
|
||||
// Writer MANIFEST 写入器
|
||||
type Writer struct {
|
||||
file io.Writer
|
||||
mu sync.Mutex
|
||||
}
|
||||
|
||||
// NewWriter 创建 MANIFEST 写入器
|
||||
func NewWriter(file io.Writer) *Writer {
|
||||
return &Writer{
|
||||
file: file,
|
||||
}
|
||||
}
|
||||
|
||||
// WriteEdit 写入版本变更
|
||||
func (w *Writer) WriteEdit(edit *VersionEdit) error {
|
||||
w.mu.Lock()
|
||||
defer w.mu.Unlock()
|
||||
|
||||
// 编码
|
||||
data, err := edit.Encode()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// 写入
|
||||
_, err = w.file.Write(data)
|
||||
return err
|
||||
}
|
||||
187
manifest/version.go
Normal file
187
manifest/version.go
Normal file
@@ -0,0 +1,187 @@
|
||||
package manifest
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"sync"
|
||||
)
|
||||
|
||||
// FileMetadata SST 文件元数据
|
||||
type FileMetadata struct {
|
||||
FileNumber int64 // 文件编号
|
||||
Level int // 所在层级 (0-6)
|
||||
FileSize int64 // 文件大小
|
||||
MinKey int64 // 最小 key
|
||||
MaxKey int64 // 最大 key
|
||||
RowCount int64 // 行数
|
||||
}
|
||||
|
||||
const (
|
||||
NumLevels = 7 // L0-L6
|
||||
)
|
||||
|
||||
// Version 数据库的一个版本快照
|
||||
type Version struct {
|
||||
// 分层存储 SST 文件 (L0-L6)
|
||||
Levels [NumLevels][]*FileMetadata
|
||||
|
||||
// 下一个文件编号
|
||||
NextFileNumber int64
|
||||
|
||||
// 最后序列号
|
||||
LastSequence int64
|
||||
|
||||
// 版本号
|
||||
VersionNumber int64
|
||||
|
||||
mu sync.RWMutex
|
||||
}
|
||||
|
||||
// NewVersion 创建新版本
|
||||
func NewVersion() *Version {
|
||||
v := &Version{
|
||||
NextFileNumber: 1,
|
||||
LastSequence: 0,
|
||||
VersionNumber: 0,
|
||||
}
|
||||
// 初始化每一层
|
||||
for i := 0; i < NumLevels; i++ {
|
||||
v.Levels[i] = make([]*FileMetadata, 0)
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
// Clone 克隆版本
|
||||
func (v *Version) Clone() *Version {
|
||||
v.mu.RLock()
|
||||
defer v.mu.RUnlock()
|
||||
|
||||
newVersion := &Version{
|
||||
NextFileNumber: v.NextFileNumber,
|
||||
LastSequence: v.LastSequence,
|
||||
VersionNumber: v.VersionNumber + 1,
|
||||
}
|
||||
|
||||
// 克隆每一层
|
||||
for level := 0; level < NumLevels; level++ {
|
||||
newVersion.Levels[level] = make([]*FileMetadata, len(v.Levels[level]))
|
||||
copy(newVersion.Levels[level], v.Levels[level])
|
||||
}
|
||||
|
||||
return newVersion
|
||||
}
|
||||
|
||||
// Apply 应用版本变更
|
||||
func (v *Version) Apply(edit *VersionEdit) {
|
||||
v.mu.Lock()
|
||||
defer v.mu.Unlock()
|
||||
|
||||
// 删除文件（按层级删除）
|
||||
if len(edit.DeletedFiles) > 0 {
|
||||
deleteSet := make(map[int64]bool)
|
||||
for _, fileNum := range edit.DeletedFiles {
|
||||
deleteSet[fileNum] = true
|
||||
}
|
||||
|
||||
// 遍历每一层，删除文件
|
||||
for level := 0; level < NumLevels; level++ {
|
||||
newFiles := make([]*FileMetadata, 0)
|
||||
deletedCount := 0
|
||||
for _, file := range v.Levels[level] {
|
||||
if !deleteSet[file.FileNumber] {
|
||||
newFiles = append(newFiles, file)
|
||||
} else {
|
||||
deletedCount++
|
||||
}
|
||||
}
|
||||
if deletedCount > 0 {
|
||||
fmt.Printf("[Version.Apply] L%d: deleted %d files\n", level, deletedCount)
|
||||
}
|
||||
v.Levels[level] = newFiles
|
||||
}
|
||||
}
|
||||
|
||||
// 添加文件（按层级添加）
|
||||
if len(edit.AddedFiles) > 0 {
|
||||
for _, file := range edit.AddedFiles {
|
||||
if file.Level >= 0 && file.Level < NumLevels {
|
||||
fmt.Printf("[Version.Apply] Adding file #%d to L%d (keys %d-%d)\n",
|
||||
file.FileNumber, file.Level, file.MinKey, file.MaxKey)
|
||||
v.Levels[file.Level] = append(v.Levels[file.Level], file)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 更新下一个文件编号
|
||||
if edit.NextFileNumber != nil {
|
||||
v.NextFileNumber = *edit.NextFileNumber
|
||||
}
|
||||
|
||||
// 更新最后序列号
|
||||
if edit.LastSequence != nil {
|
||||
v.LastSequence = *edit.LastSequence
|
||||
}
|
||||
}
|
||||
|
||||
// GetLevel 获取指定层级的文件
|
||||
func (v *Version) GetLevel(level int) []*FileMetadata {
|
||||
v.mu.RLock()
|
||||
defer v.mu.RUnlock()
|
||||
|
||||
if level < 0 || level >= NumLevels {
|
||||
return nil
|
||||
}
|
||||
|
||||
files := make([]*FileMetadata, len(v.Levels[level]))
|
||||
copy(files, v.Levels[level])
|
||||
return files
|
||||
}
|
||||
|
||||
// GetSSTFiles 获取所有 SST 文件（副本，兼容旧接口）
|
||||
func (v *Version) GetSSTFiles() []*FileMetadata {
|
||||
v.mu.RLock()
|
||||
defer v.mu.RUnlock()
|
||||
|
||||
// 收集所有层级的文件
|
||||
allFiles := make([]*FileMetadata, 0)
|
||||
for level := 0; level < NumLevels; level++ {
|
||||
allFiles = append(allFiles, v.Levels[level]...)
|
||||
}
|
||||
return allFiles
|
||||
}
|
||||
|
||||
// GetNextFileNumber 获取下一个文件编号
|
||||
func (v *Version) GetNextFileNumber() int64 {
|
||||
v.mu.RLock()
|
||||
defer v.mu.RUnlock()
|
||||
return v.NextFileNumber
|
||||
}
|
||||
|
||||
// GetLastSequence 获取最后序列号
|
||||
func (v *Version) GetLastSequence() int64 {
|
||||
v.mu.RLock()
|
||||
defer v.mu.RUnlock()
|
||||
return v.LastSequence
|
||||
}
|
||||
|
||||
// GetFileCount 获取文件数量
|
||||
func (v *Version) GetFileCount() int {
|
||||
v.mu.RLock()
|
||||
defer v.mu.RUnlock()
|
||||
|
||||
total := 0
|
||||
for level := 0; level < NumLevels; level++ {
|
||||
total += len(v.Levels[level])
|
||||
}
|
||||
return total
|
||||
}
|
||||
|
||||
// GetLevelFileCount 获取指定层级的文件数量
|
||||
func (v *Version) GetLevelFileCount(level int) int {
|
||||
v.mu.RLock()
|
||||
defer v.mu.RUnlock()
|
||||
|
||||
if level < 0 || level >= NumLevels {
|
||||
return 0
|
||||
}
|
||||
return len(v.Levels[level])
|
||||
}
|
||||
114
manifest/version_edit.go
Normal file
114
manifest/version_edit.go
Normal file
@@ -0,0 +1,114 @@
|
||||
package manifest
|
||||
|
||||
import (
|
||||
"encoding/binary"
|
||||
"encoding/json"
|
||||
"hash/crc32"
|
||||
"io"
|
||||
)
|
||||
|
||||
// EditType 变更类型
|
||||
type EditType byte
|
||||
|
||||
const (
|
||||
EditTypeAddFile EditType = 1 // 添加文件
|
||||
EditTypeDeleteFile EditType = 2 // 删除文件
|
||||
EditTypeSetNextFile EditType = 3 // 设置下一个文件编号
|
||||
EditTypeSetLastSeq EditType = 4 // 设置最后序列号
|
||||
)
|
||||
|
||||
// VersionEdit 版本变更记录
|
||||
type VersionEdit struct {
|
||||
// 添加的文件
|
||||
AddedFiles []*FileMetadata
|
||||
|
||||
// 删除的文件（文件编号列表）
|
||||
DeletedFiles []int64
|
||||
|
||||
// 下一个文件编号
|
||||
NextFileNumber *int64
|
||||
|
||||
// 最后序列号
|
||||
LastSequence *int64
|
||||
}
|
||||
|
||||
// NewVersionEdit 创建版本变更
|
||||
func NewVersionEdit() *VersionEdit {
|
||||
return &VersionEdit{
|
||||
AddedFiles: make([]*FileMetadata, 0),
|
||||
DeletedFiles: make([]int64, 0),
|
||||
}
|
||||
}
|
||||
|
||||
// AddFile 添加文件
|
||||
func (e *VersionEdit) AddFile(file *FileMetadata) {
|
||||
e.AddedFiles = append(e.AddedFiles, file)
|
||||
}
|
||||
|
||||
// DeleteFile 删除文件
|
||||
func (e *VersionEdit) DeleteFile(fileNumber int64) {
|
||||
e.DeletedFiles = append(e.DeletedFiles, fileNumber)
|
||||
}
|
||||
|
||||
// SetNextFileNumber 设置下一个文件编号
|
||||
func (e *VersionEdit) SetNextFileNumber(num int64) {
|
||||
e.NextFileNumber = &num
|
||||
}
|
||||
|
||||
// SetLastSequence 设置最后序列号
|
||||
func (e *VersionEdit) SetLastSequence(seq int64) {
|
||||
e.LastSequence = &seq
|
||||
}
|
||||
|
||||
// Encode 编码为字节
|
||||
func (e *VersionEdit) Encode() ([]byte, error) {
|
||||
// 使用 JSON 编码（简单实现）
|
||||
data, err := json.Marshal(e)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// 格式: CRC32(4) + Length(4) + Data
|
||||
totalLen := 8 + len(data)
|
||||
buf := make([]byte, totalLen)
|
||||
|
||||
// 计算 CRC32
|
||||
crc := crc32.ChecksumIEEE(data)
|
||||
binary.LittleEndian.PutUint32(buf[0:4], crc)
|
||||
|
||||
// 写入长度
|
||||
binary.LittleEndian.PutUint32(buf[4:8], uint32(len(data)))
|
||||
|
||||
// 写入数据
|
||||
copy(buf[8:], data)
|
||||
|
||||
return buf, nil
|
||||
}
|
||||
|
||||
// Decode 从字节解码
|
||||
func (e *VersionEdit) Decode(data []byte) error {
|
||||
if len(data) < 8 {
|
||||
return io.ErrUnexpectedEOF
|
||||
}
|
||||
|
||||
// 读取 CRC32
|
||||
crc := binary.LittleEndian.Uint32(data[0:4])
|
||||
|
||||
// 读取长度
|
||||
length := binary.LittleEndian.Uint32(data[4:8])
|
||||
|
||||
if len(data) < int(8+length) {
|
||||
return io.ErrUnexpectedEOF
|
||||
}
|
||||
|
||||
// 读取数据
|
||||
editData := data[8 : 8+length]
|
||||
|
||||
// 验证 CRC32
|
||||
if crc32.ChecksumIEEE(editData) != crc {
|
||||
return io.ErrUnexpectedEOF
|
||||
}
|
||||
|
||||
// JSON 解码
|
||||
return json.Unmarshal(editData, e)
|
||||
}
|
||||
251
manifest/version_set.go
Normal file
251
manifest/version_set.go
Normal file
@@ -0,0 +1,251 @@
|
||||
package manifest
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
)
|
||||
|
||||
// VersionSet 版本集合管理器
|
||||
type VersionSet struct {
|
||||
// 当前版本
|
||||
current *Version
|
||||
|
||||
// MANIFEST 文件
|
||||
manifestFile *os.File
|
||||
manifestWriter *Writer
|
||||
manifestNumber int64
|
||||
|
||||
// 下一个文件编号
|
||||
nextFileNumber atomic.Int64
|
||||
|
||||
// 最后序列号
|
||||
lastSequence atomic.Int64
|
||||
|
||||
// 目录
|
||||
dir string
|
||||
|
||||
// 锁
|
||||
mu sync.RWMutex
|
||||
}
|
||||
|
||||
// NewVersionSet 创建版本集合
|
||||
func NewVersionSet(dir string) (*VersionSet, error) {
|
||||
vs := &VersionSet{
|
||||
dir: dir,
|
||||
}
|
||||
|
||||
// 确保目录存在
|
||||
err := os.MkdirAll(dir, 0755)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// 读取 CURRENT 文件
|
||||
currentFile := filepath.Join(dir, "CURRENT")
|
||||
data, err := os.ReadFile(currentFile)
|
||||
|
||||
if err != nil {
|
||||
// CURRENT 不存在，创建新的 MANIFEST
|
||||
return vs, vs.createNewManifest()
|
||||
}
|
||||
|
||||
// 读取 MANIFEST 文件
|
||||
manifestName := strings.TrimSpace(string(data))
|
||||
manifestPath := filepath.Join(dir, manifestName)
|
||||
|
||||
// 恢复版本信息
|
||||
version, err := vs.recoverFromManifest(manifestPath)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
vs.current = version
|
||||
vs.nextFileNumber.Store(version.NextFileNumber)
|
||||
vs.lastSequence.Store(version.LastSequence)
|
||||
|
||||
// 解析 MANIFEST 编号
|
||||
fmt.Sscanf(manifestName, "MANIFEST-%d", &vs.manifestNumber)
|
||||
|
||||
// 打开 MANIFEST 用于追加
|
||||
file, err := os.OpenFile(manifestPath, os.O_APPEND|os.O_WRONLY, 0644)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
vs.manifestFile = file
|
||||
vs.manifestWriter = NewWriter(file)
|
||||
|
||||
return vs, nil
|
||||
}
|
||||
|
||||
// createNewManifest 创建新的 MANIFEST
|
||||
func (vs *VersionSet) createNewManifest() error {
|
||||
// 生成新的 MANIFEST 文件名
|
||||
vs.manifestNumber = vs.nextFileNumber.Add(1)
|
||||
manifestName := fmt.Sprintf("MANIFEST-%06d", vs.manifestNumber)
|
||||
manifestPath := filepath.Join(vs.dir, manifestName)
|
||||
|
||||
// 创建 MANIFEST 文件
|
||||
file, err := os.Create(manifestPath)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
vs.manifestFile = file
|
||||
vs.manifestWriter = NewWriter(file)
|
||||
|
||||
// 创建初始版本
|
||||
vs.current = NewVersion()
|
||||
|
||||
// 写入初始版本
|
||||
edit := NewVersionEdit()
|
||||
nextFile := vs.manifestNumber
|
||||
edit.SetNextFileNumber(nextFile)
|
||||
lastSeq := int64(0)
|
||||
edit.SetLastSequence(lastSeq)
|
||||
|
||||
err = vs.manifestWriter.WriteEdit(edit)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// 同步到磁盘
|
||||
err = vs.manifestFile.Sync()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// 更新 CURRENT 文件
|
||||
return vs.updateCurrent(manifestName)
|
||||
}
|
||||
|
||||
// recoverFromManifest 从 MANIFEST 恢复版本
|
||||
func (vs *VersionSet) recoverFromManifest(manifestPath string) (*Version, error) {
|
||||
// 打开 MANIFEST 文件
|
||||
file, err := os.Open(manifestPath)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer file.Close()
|
||||
|
||||
reader := NewReader(file)
|
||||
|
||||
// 创建初始版本
|
||||
version := NewVersion()
|
||||
|
||||
// 读取所有 VersionEdit
|
||||
for {
|
||||
edit, err := reader.ReadEdit()
|
||||
if err == io.EOF {
|
||||
break
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// 应用变更
|
||||
version.Apply(edit)
|
||||
}
|
||||
|
||||
return version, nil
|
||||
}
|
||||
|
||||
// updateCurrent 更新 CURRENT 文件
|
||||
func (vs *VersionSet) updateCurrent(manifestName string) error {
|
||||
currentPath := filepath.Join(vs.dir, "CURRENT")
|
||||
tmpPath := currentPath + ".tmp"
|
||||
|
||||
// 1. 写入临时文件
|
||||
err := os.WriteFile(tmpPath, []byte(manifestName+"\n"), 0644)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// 2. 原子性重命名
|
||||
err = os.Rename(tmpPath, currentPath)
|
||||
if err != nil {
|
||||
os.Remove(tmpPath)
|
||||
return err
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// LogAndApply 记录并应用版本变更
|
||||
func (vs *VersionSet) LogAndApply(edit *VersionEdit) error {
|
||||
vs.mu.Lock()
|
||||
defer vs.mu.Unlock()
|
||||
|
||||
// 1. 创建新版本
|
||||
newVersion := vs.current.Clone()
|
||||
|
||||
// 2. 应用变更
|
||||
newVersion.Apply(edit)
|
||||
|
||||
// 3. 写入 MANIFEST
|
||||
err := vs.manifestWriter.WriteEdit(edit)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// 4. 同步到磁盘
|
||||
err = vs.manifestFile.Sync()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// 5. 更新当前版本
|
||||
vs.current = newVersion
|
||||
|
||||
// 6. 更新原子变量
|
||||
if edit.NextFileNumber != nil {
|
||||
vs.nextFileNumber.Store(*edit.NextFileNumber)
|
||||
}
|
||||
if edit.LastSequence != nil {
|
||||
vs.lastSequence.Store(*edit.LastSequence)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// GetCurrent 获取当前版本
|
||||
func (vs *VersionSet) GetCurrent() *Version {
|
||||
vs.mu.RLock()
|
||||
defer vs.mu.RUnlock()
|
||||
return vs.current
|
||||
}
|
||||
|
||||
// GetNextFileNumber 获取下一个文件编号
|
||||
func (vs *VersionSet) GetNextFileNumber() int64 {
|
||||
return vs.nextFileNumber.Load()
|
||||
}
|
||||
|
||||
// AllocateFileNumber 分配文件编号
|
||||
func (vs *VersionSet) AllocateFileNumber() int64 {
|
||||
return vs.nextFileNumber.Add(1)
|
||||
}
|
||||
|
||||
// GetLastSequence 获取最后序列号
|
||||
func (vs *VersionSet) GetLastSequence() int64 {
|
||||
return vs.lastSequence.Load()
|
||||
}
|
||||
|
||||
// SetLastSequence 设置最后序列号
|
||||
func (vs *VersionSet) SetLastSequence(seq int64) {
|
||||
vs.lastSequence.Store(seq)
|
||||
}
|
||||
|
||||
// Close 关闭 VersionSet
|
||||
func (vs *VersionSet) Close() error {
|
||||
vs.mu.Lock()
|
||||
defer vs.mu.Unlock()
|
||||
|
||||
if vs.manifestFile != nil {
|
||||
return vs.manifestFile.Close()
|
||||
}
|
||||
return nil
|
||||
}
|
||||
220
manifest/version_set_test.go
Normal file
220
manifest/version_set_test.go
Normal file
@@ -0,0 +1,220 @@
|
||||
package manifest
|
||||
|
||||
import (
|
||||
"os"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestVersionSetBasic(t *testing.T) {
|
||||
dir := "./test_manifest"
|
||||
os.RemoveAll(dir)
|
||||
defer os.RemoveAll(dir)
|
||||
|
||||
// 创建 VersionSet
|
||||
vs, err := NewVersionSet(dir)
|
||||
if err != nil {
|
||||
t.Fatalf("NewVersionSet failed: %v", err)
|
||||
}
|
||||
defer vs.Close()
|
||||
|
||||
// 检查初始状态
|
||||
version := vs.GetCurrent()
|
||||
if version.GetFileCount() != 0 {
|
||||
t.Errorf("Expected 0 files, got %d", version.GetFileCount())
|
||||
}
|
||||
|
||||
t.Log("VersionSet basic test passed!")
|
||||
}
|
||||
|
||||
func TestVersionSetAddFile(t *testing.T) {
|
||||
dir := "./test_manifest_add"
|
||||
os.RemoveAll(dir)
|
||||
defer os.RemoveAll(dir)
|
||||
|
||||
vs, err := NewVersionSet(dir)
|
||||
if err != nil {
|
||||
t.Fatalf("NewVersionSet failed: %v", err)
|
||||
}
|
||||
defer vs.Close()
|
||||
|
||||
// 添加文件
|
||||
edit := NewVersionEdit()
|
||||
edit.AddFile(&FileMetadata{
|
||||
FileNumber: 1,
|
||||
FileSize: 1024,
|
||||
MinKey: 1,
|
||||
MaxKey: 100,
|
||||
RowCount: 100,
|
||||
})
|
||||
|
||||
err = vs.LogAndApply(edit)
|
||||
if err != nil {
|
||||
t.Fatalf("LogAndApply failed: %v", err)
|
||||
}
|
||||
|
||||
// 检查
|
||||
version := vs.GetCurrent()
|
||||
if version.GetFileCount() != 1 {
|
||||
t.Errorf("Expected 1 file, got %d", version.GetFileCount())
|
||||
}
|
||||
|
||||
files := version.GetSSTFiles()
|
||||
if files[0].FileNumber != 1 {
|
||||
t.Errorf("Expected file number 1, got %d", files[0].FileNumber)
|
||||
}
|
||||
|
||||
t.Log("VersionSet add file test passed!")
|
||||
}
|
||||
|
||||
func TestVersionSetDeleteFile(t *testing.T) {
|
||||
dir := "./test_manifest_delete"
|
||||
os.RemoveAll(dir)
|
||||
defer os.RemoveAll(dir)
|
||||
|
||||
vs, err := NewVersionSet(dir)
|
||||
if err != nil {
|
||||
t.Fatalf("NewVersionSet failed: %v", err)
|
||||
}
|
||||
defer vs.Close()
|
||||
|
||||
// 添加两个文件
|
||||
edit1 := NewVersionEdit()
|
||||
edit1.AddFile(&FileMetadata{FileNumber: 1, FileSize: 1024, MinKey: 1, MaxKey: 100, RowCount: 100})
|
||||
edit1.AddFile(&FileMetadata{FileNumber: 2, FileSize: 2048, MinKey: 101, MaxKey: 200, RowCount: 100})
|
||||
vs.LogAndApply(edit1)
|
||||
|
||||
// 删除一个文件
|
||||
edit2 := NewVersionEdit()
|
||||
edit2.DeleteFile(1)
|
||||
err = vs.LogAndApply(edit2)
|
||||
if err != nil {
|
||||
t.Fatalf("LogAndApply failed: %v", err)
|
||||
}
|
||||
|
||||
// 检查
|
||||
version := vs.GetCurrent()
|
||||
if version.GetFileCount() != 1 {
|
||||
t.Errorf("Expected 1 file, got %d", version.GetFileCount())
|
||||
}
|
||||
|
||||
files := version.GetSSTFiles()
|
||||
if files[0].FileNumber != 2 {
|
||||
t.Errorf("Expected file number 2, got %d", files[0].FileNumber)
|
||||
}
|
||||
|
||||
t.Log("VersionSet delete file test passed!")
|
||||
}
|
||||
|
||||
func TestVersionSetRecover(t *testing.T) {
|
||||
dir := "./test_manifest_recover"
|
||||
os.RemoveAll(dir)
|
||||
defer os.RemoveAll(dir)
|
||||
|
||||
// 第一次：创建并添加文件
|
||||
vs1, err := NewVersionSet(dir)
|
||||
if err != nil {
|
||||
t.Fatalf("NewVersionSet failed: %v", err)
|
||||
}
|
||||
|
||||
edit := NewVersionEdit()
|
||||
edit.AddFile(&FileMetadata{FileNumber: 1, FileSize: 1024, MinKey: 1, MaxKey: 100, RowCount: 100})
|
||||
edit.AddFile(&FileMetadata{FileNumber: 2, FileSize: 2048, MinKey: 101, MaxKey: 200, RowCount: 100})
|
||||
vs1.LogAndApply(edit)
|
||||
vs1.Close()
|
||||
|
||||
// 第二次：重新打开并恢复
|
||||
vs2, err := NewVersionSet(dir)
|
||||
if err != nil {
|
||||
t.Fatalf("NewVersionSet recover failed: %v", err)
|
||||
}
|
||||
defer vs2.Close()
|
||||
|
||||
// 检查恢复的数据
|
||||
version := vs2.GetCurrent()
|
||||
if version.GetFileCount() != 2 {
|
||||
t.Errorf("Expected 2 files after recover, got %d", version.GetFileCount())
|
||||
}
|
||||
|
||||
files := version.GetSSTFiles()
|
||||
if files[0].FileNumber != 1 || files[1].FileNumber != 2 {
|
||||
t.Errorf("File numbers not correct after recover")
|
||||
}
|
||||
|
||||
t.Log("VersionSet recover test passed!")
|
||||
}
|
||||
|
||||
func TestVersionSetMultipleEdits(t *testing.T) {
|
||||
dir := "./test_manifest_multiple"
|
||||
os.RemoveAll(dir)
|
||||
defer os.RemoveAll(dir)
|
||||
|
||||
vs, err := NewVersionSet(dir)
|
||||
if err != nil {
|
||||
t.Fatalf("NewVersionSet failed: %v", err)
|
||||
}
|
||||
defer vs.Close()
|
||||
|
||||
// 多次变更
|
||||
for i := int64(1); i <= 10; i++ {
|
||||
edit := NewVersionEdit()
|
||||
edit.AddFile(&FileMetadata{
|
||||
FileNumber: i,
|
||||
FileSize: 1024 * i,
|
||||
MinKey: (i-1)*100 + 1,
|
||||
MaxKey: i * 100,
|
||||
RowCount: 100,
|
||||
})
|
||||
err = vs.LogAndApply(edit)
|
||||
if err != nil {
|
||||
t.Fatalf("LogAndApply failed: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// 检查
|
||||
version := vs.GetCurrent()
|
||||
if version.GetFileCount() != 10 {
|
||||
t.Errorf("Expected 10 files, got %d", version.GetFileCount())
|
||||
}
|
||||
|
||||
t.Log("VersionSet multiple edits test passed!")
|
||||
}
|
||||
|
||||
func TestVersionEditEncodeDecode(t *testing.T) {
|
||||
// 创建 VersionEdit
|
||||
edit1 := NewVersionEdit()
|
||||
edit1.AddFile(&FileMetadata{FileNumber: 1, FileSize: 1024, MinKey: 1, MaxKey: 100, RowCount: 100})
|
||||
edit1.DeleteFile(2)
|
||||
nextFile := int64(10)
|
||||
edit1.SetNextFileNumber(nextFile)
|
||||
lastSeq := int64(1000)
|
||||
edit1.SetLastSequence(lastSeq)
|
||||
|
||||
// 编码
|
||||
data, err := edit1.Encode()
|
||||
if err != nil {
|
||||
t.Fatalf("Encode failed: %v", err)
|
||||
}
|
||||
|
||||
// 解码
|
||||
edit2 := NewVersionEdit()
|
||||
err = edit2.Decode(data)
|
||||
if err != nil {
|
||||
t.Fatalf("Decode failed: %v", err)
|
||||
}
|
||||
|
||||
// 检查
|
||||
if len(edit2.AddedFiles) != 1 {
|
||||
t.Errorf("Expected 1 added file, got %d", len(edit2.AddedFiles))
|
||||
}
|
||||
if len(edit2.DeletedFiles) != 1 {
|
||||
t.Errorf("Expected 1 deleted file, got %d", len(edit2.DeletedFiles))
|
||||
}
|
||||
if *edit2.NextFileNumber != 10 {
|
||||
t.Errorf("Expected NextFileNumber 10, got %d", *edit2.NextFileNumber)
|
||||
}
|
||||
if *edit2.LastSequence != 1000 {
|
||||
t.Errorf("Expected LastSequence 1000, got %d", *edit2.LastSequence)
|
||||
}
|
||||
|
||||
t.Log("VersionEdit encode/decode test passed!")
|
||||
}
|
||||
216
memtable/manager.go
Normal file
216
memtable/manager.go
Normal file
@@ -0,0 +1,216 @@
|
||||
package memtable
|
||||
|
||||
import (
|
||||
"sync"
|
||||
)
|
||||
|
||||
// ImmutableMemTable 不可变的 MemTable
|
||||
type ImmutableMemTable struct {
|
||||
MemTable *MemTable
|
||||
WALNumber int64 // 对应的 WAL 编号
|
||||
}
|
||||
|
||||
// Manager MemTable 管理器
|
||||
type Manager struct {
|
||||
active *MemTable // Active MemTable (可写)
|
||||
immutables []*ImmutableMemTable // Immutable MemTables (只读)
|
||||
activeWAL int64 // Active MemTable 对应的 WAL 编号
|
||||
maxSize int64 // MemTable 最大大小
|
||||
mu sync.RWMutex // 读写锁
|
||||
}
|
||||
|
||||
// NewManager 创建 MemTable 管理器
|
||||
func NewManager(maxSize int64) *Manager {
|
||||
return &Manager{
|
||||
active: New(),
|
||||
immutables: make([]*ImmutableMemTable, 0),
|
||||
maxSize: maxSize,
|
||||
}
|
||||
}
|
||||
|
||||
// SetActiveWAL 设置 Active MemTable 对应的 WAL 编号
|
||||
func (m *Manager) SetActiveWAL(walNumber int64) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
m.activeWAL = walNumber
|
||||
}
|
||||
|
||||
// Put 写入数据到 Active MemTable
|
||||
func (m *Manager) Put(key int64, value []byte) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
m.active.Put(key, value)
|
||||
}
|
||||
|
||||
// Get 查询数据（先查 Active，再查 Immutables）
|
||||
func (m *Manager) Get(key int64) ([]byte, bool) {
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
|
||||
// 1. 先查 Active MemTable
|
||||
if value, found := m.active.Get(key); found {
|
||||
return value, true
|
||||
}
|
||||
|
||||
// 2. 查 Immutable MemTables（从新到旧）
|
||||
for i := len(m.immutables) - 1; i >= 0; i-- {
|
||||
if value, found := m.immutables[i].MemTable.Get(key); found {
|
||||
return value, true
|
||||
}
|
||||
}
|
||||
|
||||
return nil, false
|
||||
}
|
||||
|
||||
// GetActiveSize 获取 Active MemTable 大小
|
||||
func (m *Manager) GetActiveSize() int64 {
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
return m.active.Size()
|
||||
}
|
||||
|
||||
// GetActiveCount 获取 Active MemTable 条目数
|
||||
func (m *Manager) GetActiveCount() int {
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
return m.active.Count()
|
||||
}
|
||||
|
||||
// ShouldSwitch 检查是否需要切换 MemTable
|
||||
func (m *Manager) ShouldSwitch() bool {
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
return m.active.Size() >= m.maxSize
|
||||
}
|
||||
|
||||
// Switch 切换 MemTable（Active → Immutable，创建新 Active）
|
||||
// 返回：旧的 WAL 编号，新的 Active MemTable
|
||||
func (m *Manager) Switch(newWALNumber int64) (oldWALNumber int64, immutable *ImmutableMemTable) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
|
||||
// 1. 将 Active 变为 Immutable
|
||||
immutable = &ImmutableMemTable{
|
||||
MemTable: m.active,
|
||||
WALNumber: m.activeWAL,
|
||||
}
|
||||
m.immutables = append(m.immutables, immutable)
|
||||
|
||||
// 2. 创建新的 Active MemTable
|
||||
m.active = New()
|
||||
oldWALNumber = m.activeWAL
|
||||
m.activeWAL = newWALNumber
|
||||
|
||||
return oldWALNumber, immutable
|
||||
}
|
||||
|
||||
// RemoveImmutable 移除指定的 Immutable MemTable
|
||||
func (m *Manager) RemoveImmutable(target *ImmutableMemTable) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
|
||||
// 查找并移除
|
||||
for i, imm := range m.immutables {
|
||||
if imm == target {
|
||||
m.immutables = append(m.immutables[:i], m.immutables[i+1:]...)
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// GetImmutableCount 获取 Immutable MemTable 数量
|
||||
func (m *Manager) GetImmutableCount() int {
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
return len(m.immutables)
|
||||
}
|
||||
|
||||
// GetImmutables 获取所有 Immutable MemTables（副本）
|
||||
func (m *Manager) GetImmutables() []*ImmutableMemTable {
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
|
||||
immutables := make([]*ImmutableMemTable, len(m.immutables))
|
||||
copy(immutables, m.immutables)
|
||||
return immutables
|
||||
}
|
||||
|
||||
// GetActive 获取 Active MemTable（用于 Flush 时读取）
|
||||
func (m *Manager) GetActive() *MemTable {
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
return m.active
|
||||
}
|
||||
|
||||
// TotalCount 获取总条目数（Active + Immutables）
|
||||
func (m *Manager) TotalCount() int {
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
|
||||
total := m.active.Count()
|
||||
for _, imm := range m.immutables {
|
||||
total += imm.MemTable.Count()
|
||||
}
|
||||
return total
|
||||
}
|
||||
|
||||
// TotalSize 获取总大小（Active + Immutables）
|
||||
func (m *Manager) TotalSize() int64 {
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
|
||||
total := m.active.Size()
|
||||
for _, imm := range m.immutables {
|
||||
total += imm.MemTable.Size()
|
||||
}
|
||||
return total
|
||||
}
|
||||
|
||||
// NewIterator 创建 Active MemTable 的迭代器
|
||||
func (m *Manager) NewIterator() *Iterator {
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
return m.active.NewIterator()
|
||||
}
|
||||
|
||||
// Stats 统计信息
|
||||
type Stats struct {
|
||||
ActiveSize int64
|
||||
ActiveCount int
|
||||
ImmutableCount int
|
||||
ImmutablesSize int64
|
||||
ImmutablesTotal int
|
||||
TotalSize int64
|
||||
TotalCount int
|
||||
}
|
||||
|
||||
// GetStats 获取统计信息
|
||||
func (m *Manager) GetStats() *Stats {
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
|
||||
stats := &Stats{
|
||||
ActiveSize: m.active.Size(),
|
||||
ActiveCount: m.active.Count(),
|
||||
ImmutableCount: len(m.immutables),
|
||||
}
|
||||
|
||||
for _, imm := range m.immutables {
|
||||
stats.ImmutablesSize += imm.MemTable.Size()
|
||||
stats.ImmutablesTotal += imm.MemTable.Count()
|
||||
}
|
||||
|
||||
stats.TotalSize = stats.ActiveSize + stats.ImmutablesSize
|
||||
stats.TotalCount = stats.ActiveCount + stats.ImmutablesTotal
|
||||
|
||||
return stats
|
||||
}
|
||||
|
||||
// Clear 清空所有 MemTables（用于测试）
|
||||
func (m *Manager) Clear() {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
|
||||
m.active = New()
|
||||
m.immutables = make([]*ImmutableMemTable, 0)
|
||||
}
|
||||
192
memtable/manager_test.go
Normal file
192
memtable/manager_test.go
Normal file
@@ -0,0 +1,192 @@
|
||||
package memtable
|
||||
|
||||
import (
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestManagerBasic(t *testing.T) {
|
||||
mgr := NewManager(1024) // 1KB
|
||||
|
||||
// 测试写入
|
||||
mgr.Put(1, []byte("value1"))
|
||||
mgr.Put(2, []byte("value2"))
|
||||
|
||||
// 测试读取
|
||||
value, found := mgr.Get(1)
|
||||
if !found || string(value) != "value1" {
|
||||
t.Error("Get failed")
|
||||
}
|
||||
|
||||
// 测试统计
|
||||
stats := mgr.GetStats()
|
||||
if stats.ActiveCount != 2 {
|
||||
t.Errorf("Expected 2 entries, got %d", stats.ActiveCount)
|
||||
}
|
||||
|
||||
t.Log("Manager basic test passed!")
|
||||
}
|
||||
|
||||
func TestManagerSwitch(t *testing.T) {
|
||||
mgr := NewManager(50) // 50 bytes
|
||||
mgr.SetActiveWAL(1)
|
||||
|
||||
// 写入数据
|
||||
mgr.Put(1, []byte("value1_very_long_to_trigger_switch"))
|
||||
mgr.Put(2, []byte("value2_very_long_to_trigger_switch"))
|
||||
|
||||
// 检查是否需要切换
|
||||
if !mgr.ShouldSwitch() {
|
||||
t.Logf("Size: %d, MaxSize: 50", mgr.GetActiveSize())
|
||||
// 不强制要求切换，因为大小计算可能不同
|
||||
}
|
||||
|
||||
// 执行切换
|
||||
oldWAL, immutable := mgr.Switch(2)
|
||||
if oldWAL != 1 {
|
||||
t.Errorf("Expected old WAL 1, got %d", oldWAL)
|
||||
}
|
||||
|
||||
if immutable == nil {
|
||||
t.Error("Immutable should not be nil")
|
||||
}
|
||||
|
||||
// 检查 Immutable 数量
|
||||
if mgr.GetImmutableCount() != 1 {
|
||||
t.Errorf("Expected 1 immutable, got %d", mgr.GetImmutableCount())
|
||||
}
|
||||
|
||||
// 新的 Active 应该是空的
|
||||
if mgr.GetActiveCount() != 0 {
|
||||
t.Errorf("New active should be empty, got %d", mgr.GetActiveCount())
|
||||
}
|
||||
|
||||
// 应该还能查到旧数据（在 Immutable 中）
|
||||
value, found := mgr.Get(1)
|
||||
if !found || string(value) != "value1_very_long_to_trigger_switch" {
|
||||
t.Error("Should find value in immutable")
|
||||
}
|
||||
|
||||
t.Log("Manager switch test passed!")
|
||||
}
|
||||
|
||||
func TestManagerMultipleImmutables(t *testing.T) {
|
||||
mgr := NewManager(50)
|
||||
mgr.SetActiveWAL(1)
|
||||
|
||||
// 第一批数据
|
||||
mgr.Put(1, []byte("value1_long_enough"))
|
||||
mgr.Switch(2)
|
||||
|
||||
// 第二批数据
|
||||
mgr.Put(2, []byte("value2_long_enough"))
|
||||
mgr.Switch(3)
|
||||
|
||||
// 第三批数据
|
||||
mgr.Put(3, []byte("value3_long_enough"))
|
||||
mgr.Switch(4)
|
||||
|
||||
// 应该有 3 个 Immutable
|
||||
if mgr.GetImmutableCount() != 3 {
|
||||
t.Errorf("Expected 3 immutables, got %d", mgr.GetImmutableCount())
|
||||
}
|
||||
|
||||
// 应该能查到所有数据
|
||||
for i := int64(1); i <= 3; i++ {
|
||||
if _, found := mgr.Get(i); !found {
|
||||
t.Errorf("Should find key %d", i)
|
||||
}
|
||||
}
|
||||
|
||||
t.Log("Manager multiple immutables test passed!")
|
||||
}
|
||||
|
||||
func TestManagerRemoveImmutable(t *testing.T) {
|
||||
mgr := NewManager(50)
|
||||
mgr.SetActiveWAL(1)
|
||||
|
||||
// 创建 Immutable
|
||||
mgr.Put(1, []byte("value1_long_enough"))
|
||||
_, immutable := mgr.Switch(2)
|
||||
|
||||
// 移除 Immutable
|
||||
mgr.RemoveImmutable(immutable)
|
||||
|
||||
// 应该没有 Immutable 了
|
||||
if mgr.GetImmutableCount() != 0 {
|
||||
t.Errorf("Expected 0 immutables, got %d", mgr.GetImmutableCount())
|
||||
}
|
||||
|
||||
// 数据应该找不到了
|
||||
if _, found := mgr.Get(1); found {
|
||||
t.Error("Should not find removed data")
|
||||
}
|
||||
|
||||
t.Log("Manager remove immutable test passed!")
|
||||
}
|
||||
|
||||
func TestManagerStats(t *testing.T) {
|
||||
mgr := NewManager(100)
|
||||
mgr.SetActiveWAL(1)
|
||||
|
||||
// Active 数据
|
||||
mgr.Put(1, []byte("active1"))
|
||||
mgr.Put(2, []byte("active2"))
|
||||
|
||||
// 创建 Immutable
|
||||
mgr.Put(3, []byte("immutable1_long"))
|
||||
mgr.Switch(2)
|
||||
|
||||
// 新 Active 数据
|
||||
mgr.Put(4, []byte("active3"))
|
||||
|
||||
stats := mgr.GetStats()
|
||||
|
||||
if stats.ActiveCount != 1 {
|
||||
t.Errorf("Expected 1 active entry, got %d", stats.ActiveCount)
|
||||
}
|
||||
|
||||
if stats.ImmutableCount != 1 {
|
||||
t.Errorf("Expected 1 immutable, got %d", stats.ImmutableCount)
|
||||
}
|
||||
|
||||
if stats.ImmutablesTotal != 3 {
|
||||
t.Errorf("Expected 3 entries in immutables, got %d", stats.ImmutablesTotal)
|
||||
}
|
||||
|
||||
if stats.TotalCount != 4 {
|
||||
t.Errorf("Expected 4 total entries, got %d", stats.TotalCount)
|
||||
}
|
||||
|
||||
t.Logf("Stats: %+v", stats)
|
||||
t.Log("Manager stats test passed!")
|
||||
}
|
||||
|
||||
func TestManagerConcurrent(t *testing.T) {
|
||||
mgr := NewManager(1024)
|
||||
mgr.SetActiveWAL(1)
|
||||
|
||||
// 并发写入
|
||||
done := make(chan bool)
|
||||
for i := 0; i < 10; i++ {
|
||||
go func(id int) {
|
||||
for j := 0; j < 100; j++ {
|
||||
key := int64(id*100 + j)
|
||||
mgr.Put(key, []byte("value"))
|
||||
}
|
||||
done <- true
|
||||
}(i)
|
||||
}
|
||||
|
||||
// 等待完成
|
||||
for i := 0; i < 10; i++ {
|
||||
<-done
|
||||
}
|
||||
|
||||
// 检查总数
|
||||
stats := mgr.GetStats()
|
||||
if stats.TotalCount != 1000 {
|
||||
t.Errorf("Expected 1000 entries, got %d", stats.TotalCount)
|
||||
}
|
||||
|
||||
t.Log("Manager concurrent test passed!")
|
||||
}
|
||||
141
memtable/memtable.go
Normal file
141
memtable/memtable.go
Normal file
@@ -0,0 +1,141 @@
|
||||
package memtable
|
||||
|
||||
import (
|
||||
"sort"
|
||||
"sync"
|
||||
)
|
||||
|
||||
// MemTable 内存表
|
||||
type MemTable struct {
|
||||
data map[int64][]byte // key -> value
|
||||
keys []int64 // 排序的 keys
|
||||
size int64 // 数据大小
|
||||
mu sync.RWMutex
|
||||
}
|
||||
|
||||
// New 创建 MemTable
|
||||
func New() *MemTable {
|
||||
return &MemTable{
|
||||
data: make(map[int64][]byte),
|
||||
keys: make([]int64, 0),
|
||||
size: 0,
|
||||
}
|
||||
}
|
||||
|
||||
// Put 插入数据
|
||||
func (m *MemTable) Put(key int64, value []byte) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
|
||||
// 检查是否已存在
|
||||
if _, exists := m.data[key]; !exists {
|
||||
m.keys = append(m.keys, key)
|
||||
// 保持 keys 有序
|
||||
sort.Slice(m.keys, func(i, j int) bool {
|
||||
return m.keys[i] < m.keys[j]
|
||||
})
|
||||
}
|
||||
|
||||
m.data[key] = value
|
||||
m.size += int64(len(value))
|
||||
}
|
||||
|
||||
// Get 查询数据
|
||||
func (m *MemTable) Get(key int64) ([]byte, bool) {
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
|
||||
value, exists := m.data[key]
|
||||
return value, exists
|
||||
}
|
||||
|
||||
// Size 获取大小
|
||||
func (m *MemTable) Size() int64 {
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
|
||||
return m.size
|
||||
}
|
||||
|
||||
// Count 获取条目数量
|
||||
func (m *MemTable) Count() int {
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
|
||||
return len(m.data)
|
||||
}
|
||||
|
||||
// Keys 获取所有 keys 的副本（已排序）
|
||||
func (m *MemTable) Keys() []int64 {
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
|
||||
// 返回副本以避免并发问题
|
||||
keysCopy := make([]int64, len(m.keys))
|
||||
copy(keysCopy, m.keys)
|
||||
return keysCopy
|
||||
}
|
||||
|
||||
// Iterator 迭代器
|
||||
type Iterator struct {
|
||||
mt *MemTable
|
||||
index int
|
||||
}
|
||||
|
||||
// NewIterator 创建迭代器
|
||||
func (m *MemTable) NewIterator() *Iterator {
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
|
||||
return &Iterator{
|
||||
mt: m,
|
||||
index: -1,
|
||||
}
|
||||
}
|
||||
|
||||
// Next 移动到下一个
|
||||
func (it *Iterator) Next() bool {
|
||||
it.mt.mu.RLock()
|
||||
defer it.mt.mu.RUnlock()
|
||||
|
||||
it.index++
|
||||
return it.index < len(it.mt.keys)
|
||||
}
|
||||
|
||||
// Key 当前 key
|
||||
func (it *Iterator) Key() int64 {
|
||||
it.mt.mu.RLock()
|
||||
defer it.mt.mu.RUnlock()
|
||||
|
||||
if it.index < 0 || it.index >= len(it.mt.keys) {
|
||||
return 0
|
||||
}
|
||||
return it.mt.keys[it.index]
|
||||
}
|
||||
|
||||
// Value 当前 value
|
||||
func (it *Iterator) Value() []byte {
|
||||
it.mt.mu.RLock()
|
||||
defer it.mt.mu.RUnlock()
|
||||
|
||||
if it.index < 0 || it.index >= len(it.mt.keys) {
|
||||
return nil
|
||||
}
|
||||
key := it.mt.keys[it.index]
|
||||
return it.mt.data[key]
|
||||
}
|
||||
|
||||
// Reset 重置迭代器
|
||||
func (it *Iterator) Reset() {
|
||||
it.index = -1
|
||||
}
|
||||
|
||||
// Clear 清空 MemTable
|
||||
func (m *MemTable) Clear() {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
|
||||
m.data = make(map[int64][]byte)
|
||||
m.keys = make([]int64, 0)
|
||||
m.size = 0
|
||||
}
|
||||
121
memtable/memtable_test.go
Normal file
121
memtable/memtable_test.go
Normal file
@@ -0,0 +1,121 @@
|
||||
package memtable
|
||||
|
||||
import (
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestMemTable(t *testing.T) {
|
||||
mt := New()
|
||||
|
||||
// 1. 插入数据
|
||||
for i := int64(1); i <= 100; i++ {
|
||||
mt.Put(i, []byte("value_"+string(rune(i))))
|
||||
}
|
||||
|
||||
if mt.Count() != 100 {
|
||||
t.Errorf("Expected 100 entries, got %d", mt.Count())
|
||||
}
|
||||
|
||||
t.Logf("Inserted 100 entries, size: %d bytes", mt.Size())
|
||||
|
||||
// 2. 查询数据
|
||||
for i := int64(1); i <= 100; i++ {
|
||||
value, exists := mt.Get(i)
|
||||
if !exists {
|
||||
t.Errorf("Key %d not found", i)
|
||||
}
|
||||
if value == nil {
|
||||
t.Errorf("Key %d: value is nil", i)
|
||||
}
|
||||
}
|
||||
|
||||
// 3. 查询不存在的 key
|
||||
_, exists := mt.Get(101)
|
||||
if exists {
|
||||
t.Error("Key 101 should not exist")
|
||||
}
|
||||
|
||||
t.Log("All tests passed!")
|
||||
}
|
||||
|
||||
func TestMemTableIterator(t *testing.T) {
|
||||
mt := New()
|
||||
|
||||
// 插入数据 (乱序)
|
||||
keys := []int64{5, 2, 8, 1, 9, 3, 7, 4, 6, 10}
|
||||
for _, key := range keys {
|
||||
mt.Put(key, []byte("value"))
|
||||
}
|
||||
|
||||
// 迭代器应该按顺序返回
|
||||
iter := mt.NewIterator()
|
||||
var result []int64
|
||||
for iter.Next() {
|
||||
result = append(result, iter.Key())
|
||||
}
|
||||
|
||||
// 验证顺序
|
||||
for i := 0; i < len(result)-1; i++ {
|
||||
if result[i] >= result[i+1] {
|
||||
t.Errorf("Keys not in order: %v", result)
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
if len(result) != 10 {
|
||||
t.Errorf("Expected 10 keys, got %d", len(result))
|
||||
}
|
||||
|
||||
t.Logf("Iterator returned keys in order: %v", result)
|
||||
}
|
||||
|
||||
func TestMemTableClear(t *testing.T) {
|
||||
mt := New()
|
||||
|
||||
// 插入数据
|
||||
for i := int64(1); i <= 10; i++ {
|
||||
mt.Put(i, []byte("value"))
|
||||
}
|
||||
|
||||
if mt.Count() != 10 {
|
||||
t.Errorf("Expected 10 entries, got %d", mt.Count())
|
||||
}
|
||||
|
||||
// 清空
|
||||
mt.Clear()
|
||||
|
||||
if mt.Count() != 0 {
|
||||
t.Errorf("Expected 0 entries after clear, got %d", mt.Count())
|
||||
}
|
||||
|
||||
if mt.Size() != 0 {
|
||||
t.Errorf("Expected size 0 after clear, got %d", mt.Size())
|
||||
}
|
||||
|
||||
t.Log("Clear test passed!")
|
||||
}
|
||||
|
||||
func BenchmarkMemTablePut(b *testing.B) {
|
||||
mt := New()
|
||||
value := make([]byte, 100)
|
||||
|
||||
b.ResetTimer()
|
||||
for i := 0; i < b.N; i++ {
|
||||
mt.Put(int64(i), value)
|
||||
}
|
||||
}
|
||||
|
||||
func BenchmarkMemTableGet(b *testing.B) {
|
||||
mt := New()
|
||||
value := make([]byte, 100)
|
||||
|
||||
// 预先插入数据
|
||||
for i := int64(0); i < 10000; i++ {
|
||||
mt.Put(i, value)
|
||||
}
|
||||
|
||||
b.ResetTimer()
|
||||
for i := 0; i < b.N; i++ {
|
||||
mt.Get(int64(i % 10000))
|
||||
}
|
||||
}
|
||||
869
query.go
Normal file
869
query.go
Normal file
@@ -0,0 +1,869 @@
|
||||
package srdb
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"code.tczkiot.com/srdb/sst"
|
||||
)
|
||||
|
||||
type Fieldset interface {
|
||||
Get(key string) (field Field, value any, err error)
|
||||
}
|
||||
|
||||
// mapFieldset 实现 Fieldset 接口，包装 map[string]any 和 Schema
|
||||
type mapFieldset struct {
|
||||
data map[string]any
|
||||
schema *Schema
|
||||
}
|
||||
|
||||
func newMapFieldset(data map[string]any, schema *Schema) *mapFieldset {
|
||||
return &mapFieldset{
|
||||
data: data,
|
||||
schema: schema,
|
||||
}
|
||||
}
|
||||
|
||||
func (m *mapFieldset) Get(key string) (Field, any, error) {
|
||||
value, exists := m.data[key]
|
||||
if !exists {
|
||||
return Field{}, nil, fmt.Errorf("field %s not found", key)
|
||||
}
|
||||
|
||||
// 如果有 schema，返回字段定义
|
||||
if m.schema != nil {
|
||||
field, err := m.schema.GetField(key)
|
||||
if err != nil {
|
||||
// 字段在 schema 中不存在，返回默认 Field
|
||||
return Field{Name: key}, value, nil
|
||||
}
|
||||
return *field, value, nil
|
||||
}
|
||||
|
||||
// 没有 schema，返回默认 Field
|
||||
return Field{Name: key}, value, nil
|
||||
}
|
||||
|
||||
type Expr interface {
|
||||
Match(fs Fieldset) bool
|
||||
}
|
||||
|
||||
type Neginative struct {
|
||||
expr Expr
|
||||
}
|
||||
|
||||
func (n Neginative) Match(fs Fieldset) bool {
|
||||
if n.expr == nil {
|
||||
return true
|
||||
}
|
||||
return !n.expr.Match(fs)
|
||||
}
|
||||
|
||||
func Not(expr Expr) Expr {
|
||||
return Neginative{expr}
|
||||
}
|
||||
|
||||
type compare struct {
|
||||
field string
|
||||
op string
|
||||
right any
|
||||
}
|
||||
|
||||
func (c compare) Match(fs Fieldset) bool {
|
||||
_, value, err := fs.Get(c.field)
|
||||
if err != nil {
|
||||
// 字段不存在
|
||||
return c.op == "IS NULL"
|
||||
}
|
||||
|
||||
// 处理 NULL 检查
|
||||
if c.op == "IS NULL" {
|
||||
return value == nil
|
||||
}
|
||||
if c.op == "IS NOT NULL" {
|
||||
return value != nil
|
||||
}
|
||||
|
||||
// 如果值为 nil，其他操作都返回 false
|
||||
if value == nil {
|
||||
return false
|
||||
}
|
||||
|
||||
switch c.op {
|
||||
case "=":
|
||||
return compareEqual(value, c.right)
|
||||
case "!=":
|
||||
return !compareEqual(value, c.right)
|
||||
case "<":
|
||||
return compareLess(value, c.right)
|
||||
case ">":
|
||||
return compareGreater(value, c.right)
|
||||
case "<=":
|
||||
return compareLess(value, c.right) || compareEqual(value, c.right)
|
||||
case ">=":
|
||||
return compareGreater(value, c.right) || compareEqual(value, c.right)
|
||||
case "IN":
|
||||
if list, ok := c.right.([]any); ok {
|
||||
for _, item := range list {
|
||||
if compareEqual(value, item) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
}
|
||||
return false
|
||||
case "NOT IN":
|
||||
if list, ok := c.right.([]any); ok {
|
||||
for _, item := range list {
|
||||
if compareEqual(value, item) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
return false
|
||||
case "BETWEEN":
|
||||
if list, ok := c.right.([]any); ok && len(list) == 2 {
|
||||
return (compareGreater(value, list[0]) || compareEqual(value, list[0])) &&
|
||||
(compareLess(value, list[1]) || compareEqual(value, list[1]))
|
||||
}
|
||||
return false
|
||||
case "NOT BETWEEN":
|
||||
if list, ok := c.right.([]any); ok && len(list) == 2 {
|
||||
return !((compareGreater(value, list[0]) || compareEqual(value, list[0])) &&
|
||||
(compareLess(value, list[1]) || compareEqual(value, list[1])))
|
||||
}
|
||||
return false
|
||||
case "CONTAINS":
|
||||
if str, ok := value.(string); ok {
|
||||
if pattern, ok := c.right.(string); ok {
|
||||
return strings.Contains(str, pattern)
|
||||
}
|
||||
}
|
||||
return false
|
||||
case "NOT CONTAINS":
|
||||
if str, ok := value.(string); ok {
|
||||
if pattern, ok := c.right.(string); ok {
|
||||
return !strings.Contains(str, pattern)
|
||||
}
|
||||
}
|
||||
return false
|
||||
case "STARTS WITH":
|
||||
if str, ok := value.(string); ok {
|
||||
if pattern, ok := c.right.(string); ok {
|
||||
return strings.HasPrefix(str, pattern)
|
||||
}
|
||||
}
|
||||
return false
|
||||
case "NOT STARTS WITH":
|
||||
if str, ok := value.(string); ok {
|
||||
if pattern, ok := c.right.(string); ok {
|
||||
return !strings.HasPrefix(str, pattern)
|
||||
}
|
||||
}
|
||||
return false
|
||||
case "ENDS WITH":
|
||||
if str, ok := value.(string); ok {
|
||||
if pattern, ok := c.right.(string); ok {
|
||||
return strings.HasSuffix(str, pattern)
|
||||
}
|
||||
}
|
||||
return false
|
||||
case "NOT ENDS WITH":
|
||||
if str, ok := value.(string); ok {
|
||||
if pattern, ok := c.right.(string); ok {
|
||||
return !strings.HasSuffix(str, pattern)
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// compareEqual 比较两个值是否相等
|
||||
func compareEqual(left, right any) bool {
|
||||
// 处理数值类型的比较
|
||||
leftNum, leftIsNum := toFloat64(left)
|
||||
rightNum, rightIsNum := toFloat64(right)
|
||||
if leftIsNum && rightIsNum {
|
||||
return leftNum == rightNum
|
||||
}
|
||||
|
||||
// 其他类型直接比较
|
||||
return left == right
|
||||
}
|
||||
|
||||
// compareLess 比较 left < right
|
||||
func compareLess(left, right any) bool {
|
||||
// 数值比较
|
||||
leftNum, leftIsNum := toFloat64(left)
|
||||
rightNum, rightIsNum := toFloat64(right)
|
||||
if leftIsNum && rightIsNum {
|
||||
return leftNum < rightNum
|
||||
}
|
||||
|
||||
// 字符串比较
|
||||
if leftStr, ok := left.(string); ok {
|
||||
if rightStr, ok := right.(string); ok {
|
||||
return leftStr < rightStr
|
||||
}
|
||||
}
|
||||
|
||||
return false
|
||||
}
|
||||
|
||||
// compareGreater 比较 left > right
|
||||
func compareGreater(left, right any) bool {
|
||||
// 数值比较
|
||||
leftNum, leftIsNum := toFloat64(left)
|
||||
rightNum, rightIsNum := toFloat64(right)
|
||||
if leftIsNum && rightIsNum {
|
||||
return leftNum > rightNum
|
||||
}
|
||||
|
||||
// 字符串比较
|
||||
if leftStr, ok := left.(string); ok {
|
||||
if rightStr, ok := right.(string); ok {
|
||||
return leftStr > rightStr
|
||||
}
|
||||
}
|
||||
|
||||
return false
|
||||
}
|
||||
|
||||
// toFloat64 尝试将值转换为 float64
|
||||
func toFloat64(v any) (float64, bool) {
|
||||
switch val := v.(type) {
|
||||
case float64:
|
||||
return val, true
|
||||
case float32:
|
||||
return float64(val), true
|
||||
case int:
|
||||
return float64(val), true
|
||||
case int64:
|
||||
return float64(val), true
|
||||
case int32:
|
||||
return float64(val), true
|
||||
case int16:
|
||||
return float64(val), true
|
||||
case int8:
|
||||
return float64(val), true
|
||||
case uint:
|
||||
return float64(val), true
|
||||
case uint64:
|
||||
return float64(val), true
|
||||
case uint32:
|
||||
return float64(val), true
|
||||
case uint16:
|
||||
return float64(val), true
|
||||
case uint8:
|
||||
return float64(val), true
|
||||
default:
|
||||
return 0, false
|
||||
}
|
||||
}
|
||||
|
||||
func Eq(field string, value any) Expr {
|
||||
return compare{field, "=", value}
|
||||
}
|
||||
|
||||
func NotEq(field string, value any) Expr {
|
||||
return compare{field, "!=", value}
|
||||
}
|
||||
|
||||
func Lt(field string, value any) Expr {
|
||||
return compare{field, "<", value}
|
||||
}
|
||||
|
||||
func Gt(field string, value any) Expr {
|
||||
return compare{field, ">", value}
|
||||
}
|
||||
|
||||
func Lte(field string, value any) Expr {
|
||||
return compare{field, "<=", value}
|
||||
}
|
||||
|
||||
func Gte(field string, value any) Expr {
|
||||
return compare{field, ">=", value}
|
||||
}
|
||||
|
||||
func In(field string, values []any) Expr {
|
||||
return compare{field, "IN", values}
|
||||
}
|
||||
|
||||
func NotIn(field string, values []any) Expr {
|
||||
return compare{field, "NOT IN", values}
|
||||
}
|
||||
|
||||
func Between(field string, min, max any) Expr {
|
||||
return compare{field, "BETWEEN", []any{min, max}}
|
||||
}
|
||||
|
||||
func NotBetween(field string, min, max any) Expr {
|
||||
return compare{field, "NOT BETWEEN", []any{min, max}}
|
||||
}
|
||||
|
||||
func Contains(field string, pattern string) Expr {
|
||||
return compare{field, "CONTAINS", pattern}
|
||||
}
|
||||
|
||||
func NotContains(field string, pattern string) Expr {
|
||||
return compare{field, "NOT CONTAINS", pattern}
|
||||
}
|
||||
|
||||
func StartsWith(field string, prefix string) Expr {
|
||||
return compare{field, "STARTS WITH", prefix}
|
||||
}
|
||||
|
||||
func NotStartsWith(field string, prefix string) Expr {
|
||||
return compare{field, "NOT STARTS WITH", prefix}
|
||||
}
|
||||
|
||||
func EndsWith(field string, suffix string) Expr {
|
||||
return compare{field, "ENDS WITH", suffix}
|
||||
}
|
||||
|
||||
func NotEndsWith(field string, suffix string) Expr {
|
||||
return compare{field, "NOT ENDS WITH", suffix}
|
||||
}
|
||||
|
||||
func IsNull(field string) Expr {
|
||||
return compare{field, "IS NULL", nil}
|
||||
}
|
||||
|
||||
func NotNull(field string) Expr {
|
||||
return compare{field, "IS NOT NULL", nil}
|
||||
}
|
||||
|
||||
type group struct {
|
||||
exprs []Expr
|
||||
and bool
|
||||
}
|
||||
|
||||
func (g group) Match(fs Fieldset) bool {
|
||||
for _, expr := range g.exprs {
|
||||
matched := expr.Match(fs)
|
||||
if matched && !g.and {
|
||||
return true
|
||||
}
|
||||
if !matched && g.and {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func And(exprs ...Expr) Expr {
|
||||
return group{exprs, true}
|
||||
}
|
||||
|
||||
func Or(exprs ...Expr) Expr {
|
||||
return group{exprs, false}
|
||||
}
|
||||
|
||||
type QueryBuilder struct {
|
||||
conds []Expr
|
||||
fields []string // 要选择的字段，nil 表示选择所有字段
|
||||
engine *Engine
|
||||
}
|
||||
|
||||
func newQueryBuilder(engine *Engine) *QueryBuilder {
|
||||
return &QueryBuilder{
|
||||
engine: engine,
|
||||
}
|
||||
}
|
||||
|
||||
func (qb *QueryBuilder) where(expr Expr) *QueryBuilder {
|
||||
qb.conds = append(qb.conds, expr)
|
||||
return qb
|
||||
}
|
||||
|
||||
// Match 检查数据是否匹配所有条件
|
||||
func (qb *QueryBuilder) Match(data map[string]any) bool {
|
||||
if len(qb.conds) == 0 {
|
||||
return true
|
||||
}
|
||||
|
||||
fs := newMapFieldset(data, qb.engine.schema)
|
||||
for _, cond := range qb.conds {
|
||||
if !cond.Match(fs) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// Select 指定要选择的字段，如果不调用则返回所有字段
|
||||
func (qb *QueryBuilder) Select(fields ...string) *QueryBuilder {
|
||||
qb.fields = fields
|
||||
return qb
|
||||
}
|
||||
|
||||
func (qb *QueryBuilder) Where(exprs ...Expr) *QueryBuilder {
|
||||
return qb.where(And(exprs...))
|
||||
}
|
||||
|
||||
func (qb *QueryBuilder) Eq(field string, value any) *QueryBuilder {
|
||||
return qb.where(Eq(field, value))
|
||||
}
|
||||
|
||||
func (qb *QueryBuilder) NotEq(field string, value any) *QueryBuilder {
|
||||
return qb.where(NotEq(field, value))
|
||||
}
|
||||
|
||||
func (qb *QueryBuilder) Lt(field string, value any) *QueryBuilder {
|
||||
return qb.where(Lt(field, value))
|
||||
}
|
||||
|
||||
func (qb *QueryBuilder) Gt(field string, value any) *QueryBuilder {
|
||||
return qb.where(Gt(field, value))
|
||||
}
|
||||
|
||||
func (qb *QueryBuilder) Lte(field string, value any) *QueryBuilder {
|
||||
return qb.where(Lte(field, value))
|
||||
}
|
||||
|
||||
func (qb *QueryBuilder) Gte(field string, value any) *QueryBuilder {
|
||||
return qb.where(Gte(field, value))
|
||||
}
|
||||
|
||||
func (qb *QueryBuilder) In(field string, values []any) *QueryBuilder {
|
||||
return qb.where(In(field, values))
|
||||
}
|
||||
|
||||
func (qb *QueryBuilder) NotIn(field string, values []any) *QueryBuilder {
|
||||
return qb.where(NotIn(field, values))
|
||||
}
|
||||
|
||||
func (qb *QueryBuilder) Between(field string, start, end any) *QueryBuilder {
|
||||
return qb.where(Between(field, start, end))
|
||||
}
|
||||
|
||||
func (qb *QueryBuilder) NotBetween(field string, start, end any) *QueryBuilder {
|
||||
return qb.where(Not(Between(field, start, end)))
|
||||
}
|
||||
|
||||
func (qb *QueryBuilder) Contains(field string, pattern string) *QueryBuilder {
|
||||
return qb.where(Contains(field, pattern))
|
||||
}
|
||||
|
||||
func (qb *QueryBuilder) NotContains(field string, pattern string) *QueryBuilder {
|
||||
return qb.where(NotContains(field, pattern))
|
||||
}
|
||||
|
||||
func (qb *QueryBuilder) StartsWith(field string, pattern string) *QueryBuilder {
|
||||
return qb.where(StartsWith(field, pattern))
|
||||
}
|
||||
|
||||
func (qb *QueryBuilder) NotStartsWith(field string, pattern string) *QueryBuilder {
|
||||
return qb.where(NotStartsWith(field, pattern))
|
||||
}
|
||||
|
||||
func (qb *QueryBuilder) EndsWith(field string, pattern string) *QueryBuilder {
|
||||
return qb.where(EndsWith(field, pattern))
|
||||
}
|
||||
|
||||
func (qb *QueryBuilder) NotEndsWith(field string, pattern string) *QueryBuilder {
|
||||
return qb.where(NotEndsWith(field, pattern))
|
||||
}
|
||||
|
||||
func (qb *QueryBuilder) IsNull(field string) *QueryBuilder {
|
||||
return qb.where(IsNull(field))
|
||||
}
|
||||
|
||||
func (qb *QueryBuilder) NotNull(field string) *QueryBuilder {
|
||||
return qb.where(NotNull(field))
|
||||
}
|
||||
|
||||
// Rows 返回所有匹配的数据（游标模式 - 惰性加载）
|
||||
func (qb *QueryBuilder) Rows() (*Rows, error) {
|
||||
if qb.engine == nil {
|
||||
return nil, fmt.Errorf("engine is nil")
|
||||
}
|
||||
|
||||
rows := &Rows{
|
||||
schema: qb.engine.schema,
|
||||
fields: qb.fields,
|
||||
qb: qb,
|
||||
engine: qb.engine,
|
||||
visited: make(map[int64]bool),
|
||||
}
|
||||
|
||||
// 初始化 Active MemTable 迭代器
|
||||
activeMemTable := qb.engine.memtableManager.GetActive()
|
||||
if activeMemTable != nil {
|
||||
activeKeys := activeMemTable.Keys()
|
||||
if len(activeKeys) > 0 {
|
||||
rows.memIterator = newMemtableIterator(activeKeys)
|
||||
}
|
||||
}
|
||||
|
||||
// 准备 Immutable MemTables（延迟初始化）
|
||||
rows.immutableIndex = 0
|
||||
|
||||
// 初始化 SST 文件 readers
|
||||
sstReaders := qb.engine.sstManager.GetReaders()
|
||||
for _, reader := range sstReaders {
|
||||
// 获取文件中实际存在的 key 列表（已排序）
|
||||
// 这比 minKey→maxKey 逐个尝试高效 100-1000 倍（对于稀疏 key）
|
||||
keys := reader.GetAllKeys()
|
||||
rows.sstReaders = append(rows.sstReaders, &sstReader{
|
||||
reader: reader,
|
||||
keys: keys,
|
||||
index: 0,
|
||||
})
|
||||
}
|
||||
|
||||
return rows, nil
|
||||
}
|
||||
|
||||
// First 返回第一个匹配的数据
|
||||
func (qb *QueryBuilder) First() (*Row, error) {
|
||||
rows, err := qb.Rows()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
return rows.First()
|
||||
}
|
||||
|
||||
// Last 返回最后一个匹配的数据
|
||||
func (qb *QueryBuilder) Last() (*Row, error) {
|
||||
rows, err := qb.Rows()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
return rows.Last()
|
||||
}
|
||||
|
||||
// Scan 扫描结果到指定的变量
|
||||
func (qb *QueryBuilder) Scan(value any) error {
|
||||
rows, err := qb.Rows()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
return rows.Scan(value)
|
||||
}
|
||||
|
||||
type Row struct {
|
||||
schema *Schema
|
||||
fields []string // 要选择的字段，nil 表示选择所有字段
|
||||
inner *sst.Row
|
||||
}
|
||||
|
||||
// Data 获取行数据（根据 Select 过滤字段）
|
||||
func (r *Row) Data() map[string]any {
|
||||
if r.inner == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
// 如果没有指定字段，返回所有数据（包括 _seq 和 _time）
|
||||
if r.fields == nil || len(r.fields) == 0 {
|
||||
result := make(map[string]any)
|
||||
result["_seq"] = r.inner.Seq
|
||||
result["_time"] = r.inner.Time
|
||||
for k, v := range r.inner.Data {
|
||||
result[k] = v
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
// 根据指定的字段过滤
|
||||
result := make(map[string]any)
|
||||
for _, field := range r.fields {
|
||||
if field == "_seq" {
|
||||
result["_seq"] = r.inner.Seq
|
||||
} else if field == "_time" {
|
||||
result["_time"] = r.inner.Time
|
||||
} else if val, ok := r.inner.Data[field]; ok {
|
||||
result[field] = val
|
||||
}
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
// Seq 获取行序列号
|
||||
func (r *Row) Seq() int64 {
|
||||
if r.inner == nil {
|
||||
return 0
|
||||
}
|
||||
return r.inner.Seq
|
||||
}
|
||||
|
||||
// Scan 扫描行数据到指定的变量
|
||||
func (r *Row) Scan(value any) error {
|
||||
if r.inner == nil {
|
||||
return fmt.Errorf("row is nil")
|
||||
}
|
||||
|
||||
data, err := json.Marshal(r.inner.Data)
|
||||
if err != nil {
|
||||
return fmt.Errorf("marshal row data: %w", err)
|
||||
}
|
||||
|
||||
err = json.Unmarshal(data, value)
|
||||
if err != nil {
|
||||
return fmt.Errorf("unmarshal to target: %w", err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// Rows 游标模式的结果集（惰性加载）
|
||||
type Rows struct {
|
||||
schema *Schema
|
||||
fields []string // 要选择的字段，nil 表示选择所有字段
|
||||
qb *QueryBuilder
|
||||
engine *Engine
|
||||
|
||||
// 迭代状态
|
||||
currentRow *Row
|
||||
err error
|
||||
closed bool
|
||||
visited map[int64]bool // 已访问的 seq，用于去重
|
||||
|
||||
// 数据源迭代器
|
||||
memIterator *memtableIterator
|
||||
immutableIndex int
|
||||
immutableIterator *memtableIterator
|
||||
sstIndex int
|
||||
sstReaders []*sstReader
|
||||
|
||||
// 缓存模式（用于 Collect/Data 等方法）
|
||||
cached bool
|
||||
cachedRows []*sst.Row
|
||||
cachedIndex int // 缓存模式下的迭代位置
|
||||
}
|
||||
|
||||
// memtableIterator 包装 MemTable 的迭代器
|
||||
type memtableIterator struct {
|
||||
keys []int64
|
||||
index int
|
||||
}
|
||||
|
||||
func newMemtableIterator(keys []int64) *memtableIterator {
|
||||
return &memtableIterator{
|
||||
keys: keys,
|
||||
index: -1,
|
||||
}
|
||||
}
|
||||
|
||||
func (m *memtableIterator) next() (int64, bool) {
|
||||
m.index++
|
||||
if m.index >= len(m.keys) {
|
||||
return 0, false
|
||||
}
|
||||
return m.keys[m.index], true
|
||||
}
|
||||
|
||||
// sstReader 包装 SST Reader 的迭代状态
|
||||
type sstReader struct {
|
||||
reader any // 实际的 SST reader
|
||||
keys []int64 // 文件中实际存在的 key 列表（已排序）
|
||||
index int // 当前迭代位置
|
||||
}
|
||||
|
||||
// Next 移动到下一行，返回是否还有数据
|
||||
func (r *Rows) Next() bool {
|
||||
if r.closed {
|
||||
return false
|
||||
}
|
||||
if r.err != nil {
|
||||
return false
|
||||
}
|
||||
|
||||
// 如果是缓存模式，使用缓存的数据
|
||||
if r.cached {
|
||||
return r.nextFromCache()
|
||||
}
|
||||
|
||||
// 惰性模式：从数据源读取
|
||||
return r.next()
|
||||
}
|
||||
|
||||
// next 从数据源读取下一条匹配的记录（惰性加载的核心逻辑）
|
||||
func (r *Rows) next() bool {
|
||||
for {
|
||||
// 1. 尝试从 Active MemTable 获取
|
||||
if r.memIterator != nil {
|
||||
if seq, ok := r.memIterator.next(); ok {
|
||||
if !r.visited[seq] {
|
||||
row, err := r.engine.Get(seq)
|
||||
if err == nil && r.qb.Match(row.Data) {
|
||||
r.visited[seq] = true
|
||||
r.currentRow = &Row{schema: r.schema, fields: r.fields, inner: row}
|
||||
return true
|
||||
}
|
||||
r.visited[seq] = true
|
||||
}
|
||||
continue
|
||||
}
|
||||
// Active MemTable 迭代完成
|
||||
r.memIterator = nil
|
||||
}
|
||||
|
||||
// 2. 尝试从 Immutable MemTables 获取
|
||||
if r.immutableIterator != nil {
|
||||
if seq, ok := r.immutableIterator.next(); ok {
|
||||
if !r.visited[seq] {
|
||||
row, err := r.engine.Get(seq)
|
||||
if err == nil && r.qb.Match(row.Data) {
|
||||
r.visited[seq] = true
|
||||
r.currentRow = &Row{schema: r.schema, fields: r.fields, inner: row}
|
||||
return true
|
||||
}
|
||||
r.visited[seq] = true
|
||||
}
|
||||
continue
|
||||
}
|
||||
// 当前 Immutable 迭代完成，移到下一个
|
||||
r.immutableIterator = nil
|
||||
r.immutableIndex++
|
||||
}
|
||||
|
||||
// 检查是否有更多 Immutable MemTables
|
||||
if r.immutableIterator == nil && r.immutableIndex < len(r.engine.memtableManager.GetImmutables()) {
|
||||
immutables := r.engine.memtableManager.GetImmutables()
|
||||
if r.immutableIndex < len(immutables) {
|
||||
r.immutableIterator = newMemtableIterator(immutables[r.immutableIndex].MemTable.Keys())
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
// 3. 尝试从 SST 文件获取
|
||||
if r.sstIndex < len(r.sstReaders) {
|
||||
sstReader := r.sstReaders[r.sstIndex]
|
||||
// 遍历文件中实际存在的 key（不是 minKey→maxKey 范围）
|
||||
for sstReader.index < len(sstReader.keys) {
|
||||
seq := sstReader.keys[sstReader.index]
|
||||
sstReader.index++
|
||||
|
||||
if !r.visited[seq] {
|
||||
row, err := r.engine.Get(seq)
|
||||
if err == nil && r.qb.Match(row.Data) {
|
||||
r.visited[seq] = true
|
||||
r.currentRow = &Row{schema: r.schema, fields: r.fields, inner: row}
|
||||
return true
|
||||
}
|
||||
r.visited[seq] = true
|
||||
}
|
||||
}
|
||||
// 当前 SST 文件迭代完成，移到下一个
|
||||
r.sstIndex++
|
||||
continue
|
||||
}
|
||||
|
||||
// 所有数据源都迭代完成
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
// nextFromCache 从缓存中获取下一条记录
|
||||
func (r *Rows) nextFromCache() bool {
|
||||
r.cachedIndex++
|
||||
if r.cachedIndex >= len(r.cachedRows) {
|
||||
return false
|
||||
}
|
||||
r.currentRow = &Row{schema: r.schema, fields: r.fields, inner: r.cachedRows[r.cachedIndex]}
|
||||
return true
|
||||
}
|
||||
|
||||
// Row 获取当前行
|
||||
func (r *Rows) Row() *Row {
|
||||
return r.currentRow
|
||||
}
|
||||
|
||||
// Err 返回错误
|
||||
func (r *Rows) Err() error {
|
||||
return r.err
|
||||
}
|
||||
|
||||
// Close 关闭游标
|
||||
func (r *Rows) Close() error {
|
||||
r.closed = true
|
||||
return nil
|
||||
}
|
||||
|
||||
// ensureCached 确保所有数据已被加载到缓存
|
||||
func (r *Rows) ensureCached() {
|
||||
if r.cached {
|
||||
return
|
||||
}
|
||||
|
||||
// 使用私有的 next() 方法直接从数据源读取所有剩余数据
|
||||
// 这样避免了与 Next() 的循环调用问题
|
||||
// 注意：如果之前已经调用过 Next()，部分数据已经被消耗，只能缓存剩余数据
|
||||
for r.next() {
|
||||
if r.currentRow != nil && r.currentRow.inner != nil {
|
||||
r.cachedRows = append(r.cachedRows, r.currentRow.inner)
|
||||
}
|
||||
}
|
||||
|
||||
// 标记为已缓存，重置迭代位置
|
||||
r.cached = true
|
||||
r.cachedIndex = -1
|
||||
}
|
||||
|
||||
// Len 返回总行数（需要完全扫描）
|
||||
func (r *Rows) Len() int {
|
||||
r.ensureCached()
|
||||
return len(r.cachedRows)
|
||||
}
|
||||
|
||||
// Collect 收集所有结果到切片
|
||||
func (r *Rows) Collect() []map[string]any {
|
||||
r.ensureCached()
|
||||
var results []map[string]any
|
||||
for _, row := range r.cachedRows {
|
||||
results = append(results, row.Data)
|
||||
}
|
||||
return results
|
||||
}
|
||||
|
||||
// Data 获取所有行的数据（向后兼容）
|
||||
func (r *Rows) Data() []map[string]any {
|
||||
return r.Collect()
|
||||
}
|
||||
|
||||
// Scan 扫描所有行数据到指定的变量
|
||||
func (r *Rows) Scan(value any) error {
|
||||
data, err := json.Marshal(r.Collect())
|
||||
if err != nil {
|
||||
return fmt.Errorf("marshal rows data: %w", err)
|
||||
}
|
||||
|
||||
err = json.Unmarshal(data, value)
|
||||
if err != nil {
|
||||
return fmt.Errorf("unmarshal to target: %w", err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// First 获取第一行
|
||||
func (r *Rows) First() (*Row, error) {
|
||||
// 尝试获取第一条记录（不使用缓存）
|
||||
if r.Next() {
|
||||
return r.currentRow, nil
|
||||
}
|
||||
return nil, fmt.Errorf("no rows")
|
||||
}
|
||||
|
||||
// Last 获取最后一行
|
||||
func (r *Rows) Last() (*Row, error) {
|
||||
r.ensureCached()
|
||||
if len(r.cachedRows) == 0 {
|
||||
return nil, fmt.Errorf("no rows")
|
||||
}
|
||||
return &Row{schema: r.schema, fields: r.fields, inner: r.cachedRows[len(r.cachedRows)-1]}, nil
|
||||
}
|
||||
|
||||
// Count 返回总行数（别名）
|
||||
func (r *Rows) Count() int {
|
||||
return r.Len()
|
||||
}
|
||||
265
schema.go
Normal file
265
schema.go
Normal file
@@ -0,0 +1,265 @@
|
||||
package srdb
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// FieldType 字段类型
|
||||
type FieldType int
|
||||
|
||||
const (
|
||||
FieldTypeInt64 FieldType = 1
|
||||
FieldTypeString FieldType = 2
|
||||
FieldTypeFloat FieldType = 3
|
||||
FieldTypeBool FieldType = 4
|
||||
)
|
||||
|
||||
func (t FieldType) String() string {
|
||||
switch t {
|
||||
case FieldTypeInt64:
|
||||
return "int64"
|
||||
case FieldTypeString:
|
||||
return "string"
|
||||
case FieldTypeFloat:
|
||||
return "float64"
|
||||
case FieldTypeBool:
|
||||
return "bool"
|
||||
default:
|
||||
return "unknown"
|
||||
}
|
||||
}
|
||||
|
||||
// Field 字段定义
|
||||
type Field struct {
|
||||
Name string // 字段名
|
||||
Type FieldType // 字段类型
|
||||
Indexed bool // 是否建立索引
|
||||
Comment string // 注释
|
||||
}
|
||||
|
||||
// Schema 表结构定义
|
||||
type Schema struct {
|
||||
Name string // Schema 名称
|
||||
Fields []Field // 字段列表
|
||||
}
|
||||
|
||||
// New 创建 Schema
|
||||
func NewSchema(name string, fields []Field) *Schema {
|
||||
return &Schema{
|
||||
Name: name,
|
||||
Fields: fields,
|
||||
}
|
||||
}
|
||||
|
||||
// GetField 获取字段定义
|
||||
func (s *Schema) GetField(name string) (*Field, error) {
|
||||
for i := range s.Fields {
|
||||
if s.Fields[i].Name == name {
|
||||
return &s.Fields[i], nil
|
||||
}
|
||||
}
|
||||
return nil, fmt.Errorf("field %s not found", name)
|
||||
}
|
||||
|
||||
// GetIndexedFields 获取所有需要索引的字段
|
||||
func (s *Schema) GetIndexedFields() []Field {
|
||||
var fields []Field
|
||||
for _, field := range s.Fields {
|
||||
if field.Indexed {
|
||||
fields = append(fields, field)
|
||||
}
|
||||
}
|
||||
return fields
|
||||
}
|
||||
|
||||
// Validate 验证数据是否符合 Schema
|
||||
func (s *Schema) Validate(data map[string]any) error {
|
||||
for _, field := range s.Fields {
|
||||
value, exists := data[field.Name]
|
||||
if !exists {
|
||||
// 字段不存在，允许（可选字段）
|
||||
continue
|
||||
}
|
||||
|
||||
// 验证类型
|
||||
if err := s.validateType(field.Type, value); err != nil {
|
||||
return fmt.Errorf("field %s: %v", field.Name, err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ValidateType 验证值的类型（导出方法）
|
||||
func (s *Schema) ValidateType(typ FieldType, value any) error {
|
||||
return s.validateType(typ, value)
|
||||
}
|
||||
|
||||
// validateType 验证值的类型
|
||||
func (s *Schema) validateType(typ FieldType, value any) error {
|
||||
switch typ {
|
||||
case FieldTypeInt64:
|
||||
switch value.(type) {
|
||||
case int, int64, int32, int16, int8:
|
||||
return nil
|
||||
case float64:
|
||||
// JSON 解析后数字都是 float64
|
||||
return nil
|
||||
default:
|
||||
return fmt.Errorf("expected int64, got %T", value)
|
||||
}
|
||||
case FieldTypeString:
|
||||
if _, ok := value.(string); !ok {
|
||||
return fmt.Errorf("expected string, got %T", value)
|
||||
}
|
||||
case FieldTypeFloat:
|
||||
switch value.(type) {
|
||||
case float64, float32:
|
||||
return nil
|
||||
default:
|
||||
return fmt.Errorf("expected float, got %T", value)
|
||||
}
|
||||
case FieldTypeBool:
|
||||
if _, ok := value.(bool); !ok {
|
||||
return fmt.Errorf("expected bool, got %T", value)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ExtractIndexValue 提取索引值
|
||||
func (s *Schema) ExtractIndexValue(field string, data map[string]any) (any, error) {
|
||||
fieldDef, err := s.GetField(field)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
value, exists := data[field]
|
||||
if !exists {
|
||||
return nil, fmt.Errorf("field %s not found in data", field)
|
||||
}
|
||||
|
||||
// 类型转换
|
||||
switch fieldDef.Type {
|
||||
case FieldTypeInt64:
|
||||
switch v := value.(type) {
|
||||
case int:
|
||||
return int64(v), nil
|
||||
case int64:
|
||||
return v, nil
|
||||
case float64:
|
||||
return int64(v), nil
|
||||
default:
|
||||
return nil, fmt.Errorf("cannot convert %T to int64", value)
|
||||
}
|
||||
case FieldTypeString:
|
||||
if v, ok := value.(string); ok {
|
||||
return v, nil
|
||||
}
|
||||
return nil, fmt.Errorf("cannot convert %T to string", value)
|
||||
case FieldTypeFloat:
|
||||
if v, ok := value.(float64); ok {
|
||||
return v, nil
|
||||
}
|
||||
return nil, fmt.Errorf("cannot convert %T to float64", value)
|
||||
case FieldTypeBool:
|
||||
if v, ok := value.(bool); ok {
|
||||
return v, nil
|
||||
}
|
||||
return nil, fmt.Errorf("cannot convert %T to bool", value)
|
||||
}
|
||||
|
||||
return nil, fmt.Errorf("unsupported type: %v", fieldDef.Type)
|
||||
}
|
||||
|
||||
// ComputeChecksum 计算 Schema 的 SHA256 校验和
|
||||
// 使用确定性的字符串拼接算法，不依赖 json.Marshal
|
||||
// 这样即使 Schema struct 添加新字段，只要核心内容（Name、Fields）不变，checksum 就不会变
|
||||
// 重要：字段顺序不影响 checksum，会先按字段名排序
|
||||
// 格式: "name:<name>;fields:<field1_name>:<field1_type>:<field1_indexed>:<field1_comment>,<field2>..."
|
||||
func (s *Schema) ComputeChecksum() (string, error) {
|
||||
var builder strings.Builder
|
||||
|
||||
// 1. Schema 名称
|
||||
builder.WriteString("name:")
|
||||
builder.WriteString(s.Name)
|
||||
builder.WriteString(";")
|
||||
|
||||
// 2. 复制字段列表并按字段名排序（保证顺序无关性）
|
||||
sortedFields := make([]Field, len(s.Fields))
|
||||
copy(sortedFields, s.Fields)
|
||||
sort.Slice(sortedFields, func(i, j int) bool {
|
||||
return sortedFields[i].Name < sortedFields[j].Name
|
||||
})
|
||||
|
||||
// 3. 拼接排序后的字段列表
|
||||
builder.WriteString("fields:")
|
||||
for i, field := range sortedFields {
|
||||
if i > 0 {
|
||||
builder.WriteString(",")
|
||||
}
|
||||
// 字段格式: name:type:indexed:comment
|
||||
builder.WriteString(field.Name)
|
||||
builder.WriteString(":")
|
||||
builder.WriteString(field.Type.String())
|
||||
builder.WriteString(":")
|
||||
if field.Indexed {
|
||||
builder.WriteString("1")
|
||||
} else {
|
||||
builder.WriteString("0")
|
||||
}
|
||||
builder.WriteString(":")
|
||||
builder.WriteString(field.Comment)
|
||||
}
|
||||
|
||||
// 计算 SHA256
|
||||
hash := sha256.Sum256([]byte(builder.String()))
|
||||
return hex.EncodeToString(hash[:]), nil
|
||||
}
|
||||
|
||||
// SchemaFile Schema 文件格式（带校验）
|
||||
type SchemaFile struct {
|
||||
Version int `json:"version"` // 文件格式版本
|
||||
Timestamp int64 `json:"timestamp"` // 保存时间戳
|
||||
Checksum string `json:"checksum"` // Schema 内容的 SHA256 校验和
|
||||
Schema *Schema `json:"schema"` // Schema 内容
|
||||
}
|
||||
|
||||
// NewSchemaFile 创建带校验和的 Schema 文件
|
||||
func NewSchemaFile(schema *Schema) (*SchemaFile, error) {
|
||||
checksum, err := schema.ComputeChecksum()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("compute checksum: %w", err)
|
||||
}
|
||||
|
||||
return &SchemaFile{
|
||||
Version: 1, // 当前文件格式版本
|
||||
Timestamp: time.Now().Unix(),
|
||||
Checksum: checksum,
|
||||
Schema: schema,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// Verify 验证 Schema 文件的完整性
|
||||
func (sf *SchemaFile) Verify() error {
|
||||
if sf.Schema == nil {
|
||||
return fmt.Errorf("schema is nil")
|
||||
}
|
||||
|
||||
// 重新计算 checksum
|
||||
actualChecksum, err := sf.Schema.ComputeChecksum()
|
||||
if err != nil {
|
||||
return fmt.Errorf("compute checksum: %w", err)
|
||||
}
|
||||
|
||||
// 对比 checksum
|
||||
if actualChecksum != sf.Checksum {
|
||||
return fmt.Errorf("schema checksum mismatch: expected %s, got %s (schema may have been tampered with)", sf.Checksum, actualChecksum)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
267
schema_test.go
Normal file
267
schema_test.go
Normal file
@@ -0,0 +1,267 @@
|
||||
package srdb
|
||||
|
||||
import (
|
||||
"testing"
|
||||
)
|
||||
|
||||
// UserSchema 用户表 Schema
|
||||
var UserSchema = NewSchema("users", []Field{
|
||||
{Name: "name", Type: FieldTypeString, Indexed: true, Comment: "用户名"},
|
||||
{Name: "age", Type: FieldTypeInt64, Indexed: true, Comment: "年龄"},
|
||||
{Name: "email", Type: FieldTypeString, Indexed: true, Comment: "邮箱"},
|
||||
{Name: "description", Type: FieldTypeString, Indexed: false, Comment: "描述"},
|
||||
})
|
||||
|
||||
// LogSchema 日志表 Schema
|
||||
var LogSchema = NewSchema("logs", []Field{
|
||||
{Name: "level", Type: FieldTypeString, Indexed: true, Comment: "日志级别"},
|
||||
{Name: "message", Type: FieldTypeString, Indexed: false, Comment: "日志消息"},
|
||||
{Name: "source", Type: FieldTypeString, Indexed: true, Comment: "来源"},
|
||||
{Name: "error_code", Type: FieldTypeInt64, Indexed: true, Comment: "错误码"},
|
||||
})
|
||||
|
||||
// OrderSchema 订单表 Schema
|
||||
var OrderSchema = NewSchema("orders", []Field{
|
||||
{Name: "order_id", Type: FieldTypeString, Indexed: true, Comment: "订单ID"},
|
||||
{Name: "user_id", Type: FieldTypeInt64, Indexed: true, Comment: "用户ID"},
|
||||
{Name: "amount", Type: FieldTypeFloat, Indexed: true, Comment: "金额"},
|
||||
{Name: "status", Type: FieldTypeString, Indexed: true, Comment: "状态"},
|
||||
{Name: "paid", Type: FieldTypeBool, Indexed: true, Comment: "是否支付"},
|
||||
})
|
||||
|
||||
func TestSchema(t *testing.T) {
|
||||
// 创建 Schema
|
||||
schema := NewSchema("test", []Field{
|
||||
{Name: "name", Type: FieldTypeString, Indexed: true, Comment: "名称"},
|
||||
{Name: "age", Type: FieldTypeInt64, Indexed: true, Comment: "年龄"},
|
||||
{Name: "score", Type: FieldTypeFloat, Indexed: false, Comment: "分数"},
|
||||
})
|
||||
|
||||
// 测试数据
|
||||
data := map[string]any{
|
||||
"name": "Alice",
|
||||
"age": 25,
|
||||
"score": 95.5,
|
||||
}
|
||||
|
||||
// 验证
|
||||
err := schema.Validate(data)
|
||||
if err != nil {
|
||||
t.Errorf("Validation failed: %v", err)
|
||||
}
|
||||
|
||||
// 获取索引字段
|
||||
indexedFields := schema.GetIndexedFields()
|
||||
if len(indexedFields) != 2 {
|
||||
t.Errorf("Expected 2 indexed fields, got %d", len(indexedFields))
|
||||
}
|
||||
|
||||
t.Log("Schema test passed!")
|
||||
}
|
||||
|
||||
func TestSchemaValidation(t *testing.T) {
|
||||
schema := NewSchema("test", []Field{
|
||||
{Name: "name", Type: FieldTypeString, Indexed: true, Comment: "名称"},
|
||||
{Name: "age", Type: FieldTypeInt64, Indexed: true, Comment: "年龄"},
|
||||
})
|
||||
|
||||
// 正确的数据
|
||||
validData := map[string]any{
|
||||
"name": "Bob",
|
||||
"age": 30,
|
||||
}
|
||||
|
||||
err := schema.Validate(validData)
|
||||
if err != nil {
|
||||
t.Errorf("Valid data failed validation: %v", err)
|
||||
}
|
||||
|
||||
// 错误的数据类型
|
||||
invalidData := map[string]any{
|
||||
"name": "Charlie",
|
||||
"age": "thirty", // 应该是 int64
|
||||
}
|
||||
|
||||
err = schema.Validate(invalidData)
|
||||
if err == nil {
|
||||
t.Error("Invalid data should fail validation")
|
||||
}
|
||||
|
||||
t.Log("Schema validation test passed!")
|
||||
}
|
||||
|
||||
func TestExtractIndexValue(t *testing.T) {
|
||||
schema := NewSchema("test", []Field{
|
||||
{Name: "name", Type: FieldTypeString, Indexed: true, Comment: "名称"},
|
||||
{Name: "age", Type: FieldTypeInt64, Indexed: true, Comment: "年龄"},
|
||||
})
|
||||
|
||||
data := map[string]any{
|
||||
"name": "David",
|
||||
"age": float64(35), // JSON 解析后是 float64
|
||||
}
|
||||
|
||||
// 提取 name
|
||||
name, err := schema.ExtractIndexValue("name", data)
|
||||
if err != nil {
|
||||
t.Errorf("Failed to extract name: %v", err)
|
||||
}
|
||||
if name != "David" {
|
||||
t.Errorf("Expected 'David', got %v", name)
|
||||
}
|
||||
|
||||
// 提取 age (float64 → int64)
|
||||
age, err := schema.ExtractIndexValue("age", data)
|
||||
if err != nil {
|
||||
t.Errorf("Failed to extract age: %v", err)
|
||||
}
|
||||
if age != int64(35) {
|
||||
t.Errorf("Expected 35, got %v", age)
|
||||
}
|
||||
|
||||
t.Log("Extract index value test passed!")
|
||||
}
|
||||
|
||||
func TestPredefinedSchemas(t *testing.T) {
|
||||
// 测试 UserSchema
|
||||
userData := map[string]any{
|
||||
"name": "Alice",
|
||||
"age": 25,
|
||||
"email": "alice@example.com",
|
||||
"description": "Test user",
|
||||
}
|
||||
|
||||
err := UserSchema.Validate(userData)
|
||||
if err != nil {
|
||||
t.Errorf("UserSchema validation failed: %v", err)
|
||||
}
|
||||
|
||||
// 测试 LogSchema
|
||||
logData := map[string]any{
|
||||
"level": "ERROR",
|
||||
"message": "Something went wrong",
|
||||
"source": "api",
|
||||
"error_code": 500,
|
||||
}
|
||||
|
||||
err = LogSchema.Validate(logData)
|
||||
if err != nil {
|
||||
t.Errorf("LogSchema validation failed: %v", err)
|
||||
}
|
||||
|
||||
t.Log("Predefined schemas test passed!")
|
||||
}
|
||||
|
||||
// TestChecksumDeterminism 测试 checksum 的确定性
|
||||
func TestChecksumDeterminism(t *testing.T) {
|
||||
// 创建相同的 Schema 多次
|
||||
for i := 0; i < 10; i++ {
|
||||
s1 := NewSchema("users", []Field{
|
||||
{Name: "name", Type: FieldTypeString, Indexed: true, Comment: "用户名"},
|
||||
{Name: "age", Type: FieldTypeInt64, Indexed: false, Comment: "年龄"},
|
||||
})
|
||||
|
||||
s2 := NewSchema("users", []Field{
|
||||
{Name: "name", Type: FieldTypeString, Indexed: true, Comment: "用户名"},
|
||||
{Name: "age", Type: FieldTypeInt64, Indexed: false, Comment: "年龄"},
|
||||
})
|
||||
|
||||
checksum1, err := s1.ComputeChecksum()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
checksum2, err := s2.ComputeChecksum()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
if checksum1 != checksum2 {
|
||||
t.Errorf("Iteration %d: checksums should be equal, got %s and %s", i, checksum1, checksum2)
|
||||
}
|
||||
}
|
||||
|
||||
t.Log("✅ Checksum is deterministic")
|
||||
}
|
||||
|
||||
// TestChecksumFieldOrderIndependent 测试字段顺序不影响 checksum
|
||||
func TestChecksumFieldOrderIndependent(t *testing.T) {
|
||||
s1 := NewSchema("users", []Field{
|
||||
{Name: "name", Type: FieldTypeString, Indexed: true, Comment: "用户名"},
|
||||
{Name: "age", Type: FieldTypeInt64, Indexed: false, Comment: "年龄"},
|
||||
})
|
||||
|
||||
s2 := NewSchema("users", []Field{
|
||||
{Name: "age", Type: FieldTypeInt64, Indexed: false, Comment: "年龄"},
|
||||
{Name: "name", Type: FieldTypeString, Indexed: true, Comment: "用户名"},
|
||||
})
|
||||
|
||||
checksum1, _ := s1.ComputeChecksum()
|
||||
checksum2, _ := s2.ComputeChecksum()
|
||||
|
||||
if checksum1 != checksum2 {
|
||||
t.Errorf("Checksums should be equal regardless of field order, got %s and %s", checksum1, checksum2)
|
||||
} else {
|
||||
t.Logf("✅ Field order does not affect checksum (expected behavior)")
|
||||
t.Logf(" checksum: %s", checksum1)
|
||||
}
|
||||
}
|
||||
|
||||
// TestChecksumDifferentData 测试不同 Schema 的 checksum 应该不同
|
||||
func TestChecksumDifferentData(t *testing.T) {
|
||||
s1 := NewSchema("users", []Field{
|
||||
{Name: "name", Type: FieldTypeString, Indexed: true, Comment: "用户名"},
|
||||
})
|
||||
|
||||
s2 := NewSchema("users", []Field{
|
||||
{Name: "name", Type: FieldTypeString, Indexed: false, Comment: "用户名"}, // Indexed 不同
|
||||
})
|
||||
|
||||
checksum1, _ := s1.ComputeChecksum()
|
||||
checksum2, _ := s2.ComputeChecksum()
|
||||
|
||||
if checksum1 == checksum2 {
|
||||
t.Error("Different schemas should have different checksums")
|
||||
} else {
|
||||
t.Log("✅ Different schemas have different checksums")
|
||||
}
|
||||
}
|
||||
|
||||
// TestChecksumMultipleFieldOrders 测试多个字段的各种排列组合都产生相同 checksum
|
||||
func TestChecksumMultipleFieldOrders(t *testing.T) {
|
||||
// 定义 4 个字段
|
||||
fieldA := Field{Name: "id", Type: FieldTypeInt64, Indexed: true, Comment: "ID"}
|
||||
fieldB := Field{Name: "name", Type: FieldTypeString, Indexed: false, Comment: "名称"}
|
||||
fieldC := Field{Name: "age", Type: FieldTypeInt64, Indexed: false, Comment: "年龄"}
|
||||
fieldD := Field{Name: "email", Type: FieldTypeString, Indexed: true, Comment: "邮箱"}
|
||||
|
||||
// 创建不同顺序的 Schema
|
||||
schemas := []*Schema{
|
||||
NewSchema("test", []Field{fieldA, fieldB, fieldC, fieldD}), // 原始顺序
|
||||
NewSchema("test", []Field{fieldD, fieldC, fieldB, fieldA}), // 完全反转
|
||||
NewSchema("test", []Field{fieldB, fieldD, fieldA, fieldC}), // 随机顺序 1
|
||||
NewSchema("test", []Field{fieldC, fieldA, fieldD, fieldB}), // 随机顺序 2
|
||||
NewSchema("test", []Field{fieldD, fieldA, fieldC, fieldB}), // 随机顺序 3
|
||||
}
|
||||
|
||||
// 计算所有 checksum
|
||||
checksums := make([]string, len(schemas))
|
||||
for i, s := range schemas {
|
||||
checksum, err := s.ComputeChecksum()
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to compute checksum for schema %d: %v", i, err)
|
||||
}
|
||||
checksums[i] = checksum
|
||||
}
|
||||
|
||||
// 验证所有 checksum 都相同
|
||||
expectedChecksum := checksums[0]
|
||||
for i := 1; i < len(checksums); i++ {
|
||||
if checksums[i] != expectedChecksum {
|
||||
t.Errorf("Schema %d has different checksum: expected %s, got %s", i, expectedChecksum, checksums[i])
|
||||
}
|
||||
}
|
||||
|
||||
t.Logf("✅ All %d field permutations produce the same checksum", len(schemas))
|
||||
t.Logf(" checksum: %s", expectedChecksum)
|
||||
}
|
||||
98
sst/encoding.go
Normal file
98
sst/encoding.go
Normal file
@@ -0,0 +1,98 @@
|
||||
package sst
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/binary"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
)
|
||||
|
||||
// 二进制编码格式:
|
||||
// [Magic: 4 bytes][Seq: 8 bytes][Time: 8 bytes][DataLen: 4 bytes][Data: variable]
|
||||
|
||||
const (
|
||||
RowMagic = 0x524F5733 // "ROW3"
|
||||
)
|
||||
|
||||
// encodeRowBinary 使用二进制格式编码行数据
|
||||
func encodeRowBinary(row *Row) ([]byte, error) {
|
||||
buf := new(bytes.Buffer)
|
||||
|
||||
// 写入 Magic Number (用于验证)
|
||||
if err := binary.Write(buf, binary.LittleEndian, uint32(RowMagic)); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// 写入 Seq
|
||||
if err := binary.Write(buf, binary.LittleEndian, row.Seq); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// 写入 Time
|
||||
if err := binary.Write(buf, binary.LittleEndian, row.Time); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// 序列化用户数据 (仍使用 JSON，但只序列化用户数据部分)
|
||||
dataBytes, err := json.Marshal(row.Data)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// 写入数据长度
|
||||
if err := binary.Write(buf, binary.LittleEndian, uint32(len(dataBytes))); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// 写入数据
|
||||
if _, err := buf.Write(dataBytes); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return buf.Bytes(), nil
|
||||
}
|
||||
|
||||
// decodeRowBinary 解码二进制格式的行数据
|
||||
func decodeRowBinary(data []byte) (*Row, error) {
|
||||
buf := bytes.NewReader(data)
|
||||
|
||||
// 读取并验证 Magic Number
|
||||
var magic uint32
|
||||
if err := binary.Read(buf, binary.LittleEndian, &magic); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if magic != RowMagic {
|
||||
return nil, fmt.Errorf("invalid row magic: %x", magic)
|
||||
}
|
||||
|
||||
row := &Row{}
|
||||
|
||||
// 读取 Seq
|
||||
if err := binary.Read(buf, binary.LittleEndian, &row.Seq); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// 读取 Time
|
||||
if err := binary.Read(buf, binary.LittleEndian, &row.Time); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// 读取数据长度
|
||||
var dataLen uint32
|
||||
if err := binary.Read(buf, binary.LittleEndian, &dataLen); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// 读取数据
|
||||
dataBytes := make([]byte, dataLen)
|
||||
if _, err := buf.Read(dataBytes); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// 反序列化用户数据
|
||||
if err := json.Unmarshal(dataBytes, &row.Data); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return row, nil
|
||||
}
|
||||
117
sst/encoding_test.go
Normal file
117
sst/encoding_test.go
Normal file
@@ -0,0 +1,117 @@
|
||||
package sst
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestBinaryEncoding(t *testing.T) {
|
||||
// 创建测试数据
|
||||
row := &Row{
|
||||
Seq: 12345,
|
||||
Time: 1234567890,
|
||||
Data: map[string]interface{}{
|
||||
"name": "test_user",
|
||||
"age": 25,
|
||||
"email": "test@example.com",
|
||||
},
|
||||
}
|
||||
|
||||
// 编码
|
||||
encoded, err := encodeRowBinary(row)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
t.Logf("Encoded size: %d bytes", len(encoded))
|
||||
|
||||
// 解码
|
||||
decoded, err := decodeRowBinary(encoded)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// 验证
|
||||
if decoded.Seq != row.Seq {
|
||||
t.Errorf("Seq mismatch: expected %d, got %d", row.Seq, decoded.Seq)
|
||||
}
|
||||
if decoded.Time != row.Time {
|
||||
t.Errorf("Time mismatch: expected %d, got %d", row.Time, decoded.Time)
|
||||
}
|
||||
if decoded.Data["name"] != row.Data["name"] {
|
||||
t.Errorf("Name mismatch")
|
||||
}
|
||||
|
||||
t.Log("Binary encoding test passed!")
|
||||
}
|
||||
|
||||
func TestEncodingComparison(t *testing.T) {
|
||||
row := &Row{
|
||||
Seq: 12345,
|
||||
Time: 1234567890,
|
||||
Data: map[string]interface{}{
|
||||
"name": "test_user",
|
||||
"age": 25,
|
||||
"email": "test@example.com",
|
||||
},
|
||||
}
|
||||
|
||||
// 二进制编码
|
||||
binaryEncoded, _ := encodeRowBinary(row)
|
||||
|
||||
// JSON 编码 (旧方式)
|
||||
jsonData := map[string]interface{}{
|
||||
"_seq": row.Seq,
|
||||
"_time": row.Time,
|
||||
"data": row.Data,
|
||||
}
|
||||
jsonEncoded, _ := json.Marshal(jsonData)
|
||||
|
||||
t.Logf("Binary size: %d bytes", len(binaryEncoded))
|
||||
t.Logf("JSON size: %d bytes", len(jsonEncoded))
|
||||
t.Logf("Space saved: %.1f%%", float64(len(jsonEncoded)-len(binaryEncoded))/float64(len(jsonEncoded))*100)
|
||||
|
||||
if len(binaryEncoded) >= len(jsonEncoded) {
|
||||
t.Error("Binary encoding should be smaller than JSON")
|
||||
}
|
||||
}
|
||||
|
||||
func BenchmarkBinaryEncoding(b *testing.B) {
|
||||
row := &Row{
|
||||
Seq: 12345,
|
||||
Time: 1234567890,
|
||||
Data: map[string]interface{}{
|
||||
"name": "test_user",
|
||||
"age": 25,
|
||||
"email": "test@example.com",
|
||||
},
|
||||
}
|
||||
|
||||
b.ResetTimer()
|
||||
for i := 0; i < b.N; i++ {
|
||||
encodeRowBinary(row)
|
||||
}
|
||||
}
|
||||
|
||||
func BenchmarkJSONEncoding(b *testing.B) {
|
||||
row := &Row{
|
||||
Seq: 12345,
|
||||
Time: 1234567890,
|
||||
Data: map[string]interface{}{
|
||||
"name": "test_user",
|
||||
"age": 25,
|
||||
"email": "test@example.com",
|
||||
},
|
||||
}
|
||||
|
||||
data := map[string]interface{}{
|
||||
"_seq": row.Seq,
|
||||
"_time": row.Time,
|
||||
"data": row.Data,
|
||||
}
|
||||
|
||||
b.ResetTimer()
|
||||
for i := 0; i < b.N; i++ {
|
||||
json.Marshal(data)
|
||||
}
|
||||
}
|
||||
142
sst/format.go
Normal file
142
sst/format.go
Normal file
@@ -0,0 +1,142 @@
|
||||
package sst
|
||||
|
||||
import (
|
||||
"encoding/binary"
|
||||
)
|
||||
|
||||
const (
|
||||
// 文件格式
|
||||
MagicNumber = 0x53535433 // "SST3"
|
||||
Version = 1
|
||||
HeaderSize = 256 // 文件头大小
|
||||
BlockSize = 64 * 1024 // 数据块大小 (64 KB)
|
||||
|
||||
// 压缩类型
|
||||
CompressionNone = 0
|
||||
CompressionSnappy = 1
|
||||
)
|
||||
|
||||
// Header SST 文件头 (256 bytes)
|
||||
type Header struct {
|
||||
// 基础信息 (32 bytes)
|
||||
Magic uint32 // Magic Number: 0x53535433
|
||||
Version uint32 // 版本号
|
||||
Compression uint8 // 压缩类型
|
||||
Reserved1 [3]byte
|
||||
Flags uint32 // 标志位
|
||||
Reserved2 [16]byte
|
||||
|
||||
// 索引信息 (32 bytes)
|
||||
IndexOffset int64 // B+Tree 索引起始位置
|
||||
IndexSize int64 // B+Tree 索引大小
|
||||
RootOffset int64 // B+Tree 根节点位置
|
||||
Reserved3 [8]byte
|
||||
|
||||
// 数据信息 (32 bytes)
|
||||
DataOffset int64 // 数据块起始位置
|
||||
DataSize int64 // 数据块总大小
|
||||
RowCount int64 // 行数
|
||||
Reserved4 [8]byte
|
||||
|
||||
// 统计信息 (32 bytes)
|
||||
MinKey int64 // 最小 key (_seq)
|
||||
MaxKey int64 // 最大 key (_seq)
|
||||
MinTime int64 // 最小时间戳
|
||||
MaxTime int64 // 最大时间戳
|
||||
|
||||
// CRC 校验 (8 bytes)
|
||||
CRC32 uint32 // Header CRC32
|
||||
Reserved5 [4]byte
|
||||
|
||||
// 预留空间 (120 bytes)
|
||||
Reserved6 [120]byte
|
||||
}
|
||||
|
||||
// Marshal 序列化 Header
|
||||
func (h *Header) Marshal() []byte {
|
||||
buf := make([]byte, HeaderSize)
|
||||
|
||||
// 基础信息
|
||||
binary.LittleEndian.PutUint32(buf[0:4], h.Magic)
|
||||
binary.LittleEndian.PutUint32(buf[4:8], h.Version)
|
||||
buf[8] = h.Compression
|
||||
copy(buf[9:12], h.Reserved1[:])
|
||||
binary.LittleEndian.PutUint32(buf[12:16], h.Flags)
|
||||
copy(buf[16:32], h.Reserved2[:])
|
||||
|
||||
// 索引信息
|
||||
binary.LittleEndian.PutUint64(buf[32:40], uint64(h.IndexOffset))
|
||||
binary.LittleEndian.PutUint64(buf[40:48], uint64(h.IndexSize))
|
||||
binary.LittleEndian.PutUint64(buf[48:56], uint64(h.RootOffset))
|
||||
copy(buf[56:64], h.Reserved3[:])
|
||||
|
||||
// 数据信息
|
||||
binary.LittleEndian.PutUint64(buf[64:72], uint64(h.DataOffset))
|
||||
binary.LittleEndian.PutUint64(buf[72:80], uint64(h.DataSize))
|
||||
binary.LittleEndian.PutUint64(buf[80:88], uint64(h.RowCount))
|
||||
copy(buf[88:96], h.Reserved4[:])
|
||||
|
||||
// 统计信息
|
||||
binary.LittleEndian.PutUint64(buf[96:104], uint64(h.MinKey))
|
||||
binary.LittleEndian.PutUint64(buf[104:112], uint64(h.MaxKey))
|
||||
binary.LittleEndian.PutUint64(buf[112:120], uint64(h.MinTime))
|
||||
binary.LittleEndian.PutUint64(buf[120:128], uint64(h.MaxTime))
|
||||
|
||||
// CRC 校验
|
||||
binary.LittleEndian.PutUint32(buf[128:132], h.CRC32)
|
||||
copy(buf[132:136], h.Reserved5[:])
|
||||
|
||||
// 预留空间
|
||||
copy(buf[136:256], h.Reserved6[:])
|
||||
|
||||
return buf
|
||||
}
|
||||
|
||||
// Unmarshal 反序列化 Header
|
||||
func UnmarshalHeader(data []byte) *Header {
|
||||
if len(data) < HeaderSize {
|
||||
return nil
|
||||
}
|
||||
|
||||
h := &Header{}
|
||||
|
||||
// 基础信息
|
||||
h.Magic = binary.LittleEndian.Uint32(data[0:4])
|
||||
h.Version = binary.LittleEndian.Uint32(data[4:8])
|
||||
h.Compression = data[8]
|
||||
copy(h.Reserved1[:], data[9:12])
|
||||
h.Flags = binary.LittleEndian.Uint32(data[12:16])
|
||||
copy(h.Reserved2[:], data[16:32])
|
||||
|
||||
// 索引信息
|
||||
h.IndexOffset = int64(binary.LittleEndian.Uint64(data[32:40]))
|
||||
h.IndexSize = int64(binary.LittleEndian.Uint64(data[40:48]))
|
||||
h.RootOffset = int64(binary.LittleEndian.Uint64(data[48:56]))
|
||||
copy(h.Reserved3[:], data[56:64])
|
||||
|
||||
// 数据信息
|
||||
h.DataOffset = int64(binary.LittleEndian.Uint64(data[64:72]))
|
||||
h.DataSize = int64(binary.LittleEndian.Uint64(data[72:80]))
|
||||
h.RowCount = int64(binary.LittleEndian.Uint64(data[80:88]))
|
||||
copy(h.Reserved4[:], data[88:96])
|
||||
|
||||
// 统计信息
|
||||
h.MinKey = int64(binary.LittleEndian.Uint64(data[96:104]))
|
||||
h.MaxKey = int64(binary.LittleEndian.Uint64(data[104:112]))
|
||||
h.MinTime = int64(binary.LittleEndian.Uint64(data[112:120]))
|
||||
h.MaxTime = int64(binary.LittleEndian.Uint64(data[120:128]))
|
||||
|
||||
// CRC 校验
|
||||
h.CRC32 = binary.LittleEndian.Uint32(data[128:132])
|
||||
copy(h.Reserved5[:], data[132:136])
|
||||
|
||||
// 预留空间
|
||||
copy(h.Reserved6[:], data[136:256])
|
||||
|
||||
return h
|
||||
}
|
||||
|
||||
// Validate 验证 Header
|
||||
func (h *Header) Validate() bool {
|
||||
return h.Magic == MagicNumber && h.Version == Version
|
||||
}
|
||||
284
sst/manager.go
Normal file
284
sst/manager.go
Normal file
@@ -0,0 +1,284 @@
|
||||
package sst
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
)
|
||||
|
||||
// Manager SST 文件管理器
|
||||
type Manager struct {
|
||||
dir string
|
||||
readers []*Reader
|
||||
mu sync.RWMutex
|
||||
}
|
||||
|
||||
// NewManager 创建 SST 管理器
|
||||
func NewManager(dir string) (*Manager, error) {
|
||||
// 确保目录存在
|
||||
err := os.MkdirAll(dir, 0755)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
mgr := &Manager{
|
||||
dir: dir,
|
||||
readers: make([]*Reader, 0),
|
||||
}
|
||||
|
||||
// 恢复现有的 SST 文件
|
||||
err = mgr.recover()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return mgr, nil
|
||||
}
|
||||
|
||||
// recover 恢复现有的 SST 文件
|
||||
func (m *Manager) recover() error {
|
||||
// 查找所有 SST 文件
|
||||
files, err := filepath.Glob(filepath.Join(m.dir, "*.sst"))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
for _, file := range files {
|
||||
// 跳过索引文件
|
||||
filename := filepath.Base(file)
|
||||
if strings.HasPrefix(filename, "idx_") {
|
||||
continue
|
||||
}
|
||||
|
||||
// 打开 SST Reader
|
||||
reader, err := NewReader(file)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
m.readers = append(m.readers, reader)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// CreateSST 创建新的 SST 文件
|
||||
// fileNumber: 文件编号（由 VersionSet 分配）
|
||||
func (m *Manager) CreateSST(fileNumber int64, rows []*Row) (*Reader, error) {
|
||||
return m.CreateSSTWithLevel(fileNumber, rows, 0) // 默认创建到 L0
|
||||
}
|
||||
|
||||
// CreateSSTWithLevel 创建新的 SST 文件到指定层级
|
||||
// fileNumber: 文件编号（由 VersionSet 分配）
|
||||
func (m *Manager) CreateSSTWithLevel(fileNumber int64, rows []*Row, level int) (*Reader, error) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
|
||||
sstPath := filepath.Join(m.dir, fmt.Sprintf("%06d.sst", fileNumber))
|
||||
|
||||
// 创建文件
|
||||
file, err := os.Create(sstPath)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
writer := NewWriter(file)
|
||||
|
||||
// 写入所有行
|
||||
for _, row := range rows {
|
||||
err = writer.Add(row)
|
||||
if err != nil {
|
||||
file.Close()
|
||||
os.Remove(sstPath)
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
|
||||
// 完成写入
|
||||
err = writer.Finish()
|
||||
if err != nil {
|
||||
file.Close()
|
||||
os.Remove(sstPath)
|
||||
return nil, err
|
||||
}
|
||||
|
||||
file.Close()
|
||||
|
||||
// 打开 SST Reader
|
||||
reader, err := NewReader(sstPath)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// 添加到 readers 列表
|
||||
m.readers = append(m.readers, reader)
|
||||
|
||||
return reader, nil
|
||||
}
|
||||
|
||||
// Get 从所有 SST 文件中查找数据
|
||||
func (m *Manager) Get(seq int64) (*Row, error) {
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
|
||||
// 从后往前查找（新的文件优先）
|
||||
for i := len(m.readers) - 1; i >= 0; i-- {
|
||||
reader := m.readers[i]
|
||||
row, err := reader.Get(seq)
|
||||
if err == nil {
|
||||
return row, nil
|
||||
}
|
||||
}
|
||||
|
||||
return nil, fmt.Errorf("key not found: %d", seq)
|
||||
}
|
||||
|
||||
// GetReaders 获取所有 Readers（用于扫描）
|
||||
func (m *Manager) GetReaders() []*Reader {
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
|
||||
// 返回副本
|
||||
readers := make([]*Reader, len(m.readers))
|
||||
copy(readers, m.readers)
|
||||
return readers
|
||||
}
|
||||
|
||||
// GetMaxSeq 获取所有 SST 中的最大 seq
|
||||
func (m *Manager) GetMaxSeq() int64 {
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
|
||||
maxSeq := int64(0)
|
||||
for _, reader := range m.readers {
|
||||
header := reader.GetHeader()
|
||||
if header.MaxKey > maxSeq {
|
||||
maxSeq = header.MaxKey
|
||||
}
|
||||
}
|
||||
|
||||
return maxSeq
|
||||
}
|
||||
|
||||
// Count 获取 SST 文件数量
|
||||
func (m *Manager) Count() int {
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
|
||||
return len(m.readers)
|
||||
}
|
||||
|
||||
// ListFiles 列出所有 SST 文件
|
||||
func (m *Manager) ListFiles() []string {
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
|
||||
files := make([]string, 0, len(m.readers))
|
||||
for _, reader := range m.readers {
|
||||
files = append(files, reader.path)
|
||||
}
|
||||
|
||||
return files
|
||||
}
|
||||
|
||||
// CompactionConfig Compaction 配置
|
||||
// 已废弃：请使用 compaction 包中的 Manager
|
||||
type CompactionConfig struct {
|
||||
Threshold int // 触发阈值（SST 文件数量）
|
||||
BatchSize int // 每次合并的文件数量
|
||||
}
|
||||
|
||||
// DefaultCompactionConfig 默认配置
|
||||
// 已废弃：请使用 compaction 包中的 Manager
|
||||
var DefaultCompactionConfig = CompactionConfig{
|
||||
Threshold: 10,
|
||||
BatchSize: 10,
|
||||
}
|
||||
|
||||
// ShouldCompact 检查是否需要 Compaction
|
||||
// 已废弃：请使用 compaction 包中的 Manager
|
||||
func (m *Manager) ShouldCompact(config CompactionConfig) bool {
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
|
||||
return len(m.readers) > config.Threshold
|
||||
}
|
||||
|
||||
// Compact 执行 Compaction
|
||||
// 已废弃：请使用 compaction 包中的 Manager
|
||||
// 注意：此方法已不再维护，不应在新代码中使用
|
||||
func (m *Manager) Compact(config CompactionConfig) error {
|
||||
// 此方法已废弃，不再实现
|
||||
return fmt.Errorf("Compact is deprecated, please use compaction.Manager")
|
||||
}
|
||||
|
||||
// sortRows 按 seq 排序
|
||||
func sortRows(rows []*Row) {
|
||||
sort.Slice(rows, func(i, j int) bool {
|
||||
return rows[i].Seq < rows[j].Seq
|
||||
})
|
||||
}
|
||||
|
||||
// Delete 删除指定的 SST 文件（预留接口）
|
||||
func (m *Manager) Delete(fileNumber int64) error {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
|
||||
sstPath := filepath.Join(m.dir, fmt.Sprintf("%06d.sst", fileNumber))
|
||||
return os.Remove(sstPath)
|
||||
}
|
||||
|
||||
// Close 关闭所有 SST Readers
|
||||
func (m *Manager) Close() error {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
|
||||
for _, reader := range m.readers {
|
||||
reader.Close()
|
||||
}
|
||||
|
||||
m.readers = nil
|
||||
return nil
|
||||
}
|
||||
|
||||
// Stats 统计信息
|
||||
type Stats struct {
|
||||
FileCount int
|
||||
TotalSize int64
|
||||
MinSeq int64
|
||||
MaxSeq int64
|
||||
}
|
||||
|
||||
// GetStats 获取统计信息
|
||||
func (m *Manager) GetStats() *Stats {
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
|
||||
stats := &Stats{
|
||||
FileCount: len(m.readers),
|
||||
MinSeq: -1,
|
||||
MaxSeq: -1,
|
||||
}
|
||||
|
||||
for _, reader := range m.readers {
|
||||
header := reader.GetHeader()
|
||||
|
||||
if stats.MinSeq == -1 || header.MinKey < stats.MinSeq {
|
||||
stats.MinSeq = header.MinKey
|
||||
}
|
||||
|
||||
if stats.MaxSeq == -1 || header.MaxKey > stats.MaxSeq {
|
||||
stats.MaxSeq = header.MaxKey
|
||||
}
|
||||
|
||||
// 获取文件大小
|
||||
if stat, err := os.Stat(reader.path); err == nil {
|
||||
stats.TotalSize += stat.Size()
|
||||
}
|
||||
}
|
||||
|
||||
return stats
|
||||
}
|
||||
152
sst/reader.go
Normal file
152
sst/reader.go
Normal file
@@ -0,0 +1,152 @@
|
||||
package sst
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
|
||||
"code.tczkiot.com/srdb/btree"
|
||||
"github.com/edsrzf/mmap-go"
|
||||
"github.com/golang/snappy"
|
||||
)
|
||||
|
||||
// Reader SST 文件读取器
|
||||
type Reader struct {
|
||||
path string
|
||||
file *os.File
|
||||
mmap mmap.MMap
|
||||
header *Header
|
||||
btReader *btree.Reader
|
||||
}
|
||||
|
||||
// NewReader 创建 SST 读取器
|
||||
func NewReader(path string) (*Reader, error) {
|
||||
// 1. 打开文件
|
||||
file, err := os.Open(path)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// 2. mmap 映射
|
||||
mmapData, err := mmap.Map(file, mmap.RDONLY, 0)
|
||||
if err != nil {
|
||||
file.Close()
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// 3. 读取 Header
|
||||
if len(mmapData) < HeaderSize {
|
||||
mmapData.Unmap()
|
||||
file.Close()
|
||||
return nil, fmt.Errorf("file too small")
|
||||
}
|
||||
|
||||
header := UnmarshalHeader(mmapData[:HeaderSize])
|
||||
if header == nil || !header.Validate() {
|
||||
mmapData.Unmap()
|
||||
file.Close()
|
||||
return nil, fmt.Errorf("invalid header")
|
||||
}
|
||||
|
||||
// 4. 创建 B+Tree Reader
|
||||
btReader := btree.NewReader(mmapData, header.RootOffset)
|
||||
|
||||
return &Reader{
|
||||
path: path,
|
||||
file: file,
|
||||
mmap: mmapData,
|
||||
header: header,
|
||||
btReader: btReader,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// Get 查询一行数据
|
||||
func (r *Reader) Get(key int64) (*Row, error) {
|
||||
// 1. 检查范围
|
||||
if key < r.header.MinKey || key > r.header.MaxKey {
|
||||
return nil, fmt.Errorf("key out of range")
|
||||
}
|
||||
|
||||
// 2. 在 B+Tree 中查找
|
||||
dataOffset, dataSize, found := r.btReader.Get(key)
|
||||
if !found {
|
||||
return nil, fmt.Errorf("key not found")
|
||||
}
|
||||
|
||||
// 3. 读取数据
|
||||
if dataOffset+int64(dataSize) > int64(len(r.mmap)) {
|
||||
return nil, fmt.Errorf("invalid data offset")
|
||||
}
|
||||
|
||||
compressed := r.mmap[dataOffset : dataOffset+int64(dataSize)]
|
||||
|
||||
// 4. 解压缩
|
||||
var data []byte
|
||||
var err error
|
||||
if r.header.Compression == CompressionSnappy {
|
||||
data, err = snappy.Decode(nil, compressed)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
} else {
|
||||
data = compressed
|
||||
}
|
||||
|
||||
// 5. 反序列化
|
||||
row, err := decodeRow(data)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return row, nil
|
||||
}
|
||||
|
||||
// GetHeader 获取文件头信息
|
||||
func (r *Reader) GetHeader() *Header {
|
||||
return r.header
|
||||
}
|
||||
|
||||
// GetPath 获取文件路径
|
||||
func (r *Reader) GetPath() string {
|
||||
return r.path
|
||||
}
|
||||
|
||||
// GetAllKeys 获取文件中所有的 key（按顺序）
|
||||
func (r *Reader) GetAllKeys() []int64 {
|
||||
return r.btReader.GetAllKeys()
|
||||
}
|
||||
|
||||
// Close 关闭读取器
|
||||
func (r *Reader) Close() error {
|
||||
if r.mmap != nil {
|
||||
r.mmap.Unmap()
|
||||
}
|
||||
if r.file != nil {
|
||||
return r.file.Close()
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// decodeRow 解码行数据
|
||||
func decodeRow(data []byte) (*Row, error) {
|
||||
// 尝试使用二进制格式解码
|
||||
row, err := decodeRowBinary(data)
|
||||
if err == nil {
|
||||
return row, nil
|
||||
}
|
||||
|
||||
// 降级到 JSON (兼容旧数据)
|
||||
var decoded map[string]interface{}
|
||||
err = json.Unmarshal(data, &decoded)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
row = &Row{
|
||||
Seq: int64(decoded["_seq"].(float64)),
|
||||
Time: int64(decoded["_time"].(float64)),
|
||||
Data: decoded["data"].(map[string]interface{}),
|
||||
}
|
||||
|
||||
return row, nil
|
||||
}
|
||||
183
sst/sst_test.go
Normal file
183
sst/sst_test.go
Normal file
@@ -0,0 +1,183 @@
|
||||
package sst
|
||||
|
||||
import (
|
||||
"os"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestSST(t *testing.T) {
|
||||
// 1. 创建测试文件
|
||||
file, err := os.Create("test.sst")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer os.Remove("test.sst")
|
||||
|
||||
// 2. 写入数据
|
||||
writer := NewWriter(file)
|
||||
|
||||
// 添加 1000 行数据
|
||||
for i := int64(1); i <= 1000; i++ {
|
||||
row := &Row{
|
||||
Seq: i,
|
||||
Time: 1000000 + i,
|
||||
Data: map[string]interface{}{
|
||||
"name": "user_" + string(rune(i)),
|
||||
"age": 20 + i%50,
|
||||
},
|
||||
}
|
||||
err := writer.Add(row)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
// 完成写入
|
||||
err = writer.Finish()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
file.Close()
|
||||
|
||||
t.Logf("Written 1000 rows")
|
||||
|
||||
// 3. 读取数据
|
||||
reader, err := NewReader("test.sst")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer reader.Close()
|
||||
|
||||
// 验证 Header
|
||||
header := reader.GetHeader()
|
||||
if header.RowCount != 1000 {
|
||||
t.Errorf("Expected 1000 rows, got %d", header.RowCount)
|
||||
}
|
||||
if header.MinKey != 1 {
|
||||
t.Errorf("Expected MinKey=1, got %d", header.MinKey)
|
||||
}
|
||||
if header.MaxKey != 1000 {
|
||||
t.Errorf("Expected MaxKey=1000, got %d", header.MaxKey)
|
||||
}
|
||||
|
||||
t.Logf("Header: RowCount=%d, MinKey=%d, MaxKey=%d",
|
||||
header.RowCount, header.MinKey, header.MaxKey)
|
||||
|
||||
// 4. 查询测试
|
||||
for i := int64(1); i <= 1000; i++ {
|
||||
row, err := reader.Get(i)
|
||||
if err != nil {
|
||||
t.Errorf("Failed to get key %d: %v", i, err)
|
||||
continue
|
||||
}
|
||||
if row.Seq != i {
|
||||
t.Errorf("Key %d: expected Seq=%d, got %d", i, i, row.Seq)
|
||||
}
|
||||
if row.Time != 1000000+i {
|
||||
t.Errorf("Key %d: expected Time=%d, got %d", i, 1000000+i, row.Time)
|
||||
}
|
||||
}
|
||||
|
||||
// 测试不存在的 key
|
||||
_, err = reader.Get(1001)
|
||||
if err == nil {
|
||||
t.Error("Key 1001 should not exist")
|
||||
}
|
||||
|
||||
_, err = reader.Get(0)
|
||||
if err == nil {
|
||||
t.Error("Key 0 should not exist")
|
||||
}
|
||||
|
||||
t.Log("All tests passed!")
|
||||
}
|
||||
|
||||
func TestHeaderSerialization(t *testing.T) {
|
||||
// 创建 Header
|
||||
header := &Header{
|
||||
Magic: MagicNumber,
|
||||
Version: Version,
|
||||
Compression: CompressionSnappy,
|
||||
IndexOffset: 256,
|
||||
IndexSize: 1024,
|
||||
RootOffset: 512,
|
||||
DataOffset: 2048,
|
||||
DataSize: 10240,
|
||||
RowCount: 100,
|
||||
MinKey: 1,
|
||||
MaxKey: 100,
|
||||
MinTime: 1000000,
|
||||
MaxTime: 1000100,
|
||||
}
|
||||
|
||||
// 序列化
|
||||
data := header.Marshal()
|
||||
if len(data) != HeaderSize {
|
||||
t.Errorf("Expected size %d, got %d", HeaderSize, len(data))
|
||||
}
|
||||
|
||||
// 反序列化
|
||||
header2 := UnmarshalHeader(data)
|
||||
if header2 == nil {
|
||||
t.Fatal("Unmarshal failed")
|
||||
}
|
||||
|
||||
// 验证
|
||||
if header2.Magic != header.Magic {
|
||||
t.Error("Magic mismatch")
|
||||
}
|
||||
if header2.Version != header.Version {
|
||||
t.Error("Version mismatch")
|
||||
}
|
||||
if header2.Compression != header.Compression {
|
||||
t.Error("Compression mismatch")
|
||||
}
|
||||
if header2.RowCount != header.RowCount {
|
||||
t.Error("RowCount mismatch")
|
||||
}
|
||||
if header2.MinKey != header.MinKey {
|
||||
t.Error("MinKey mismatch")
|
||||
}
|
||||
if header2.MaxKey != header.MaxKey {
|
||||
t.Error("MaxKey mismatch")
|
||||
}
|
||||
|
||||
// 验证
|
||||
if !header2.Validate() {
|
||||
t.Error("Header validation failed")
|
||||
}
|
||||
|
||||
t.Log("Header serialization test passed!")
|
||||
}
|
||||
|
||||
func BenchmarkSSTGet(b *testing.B) {
|
||||
// 创建测试文件
|
||||
file, _ := os.Create("bench.sst")
|
||||
defer os.Remove("bench.sst")
|
||||
|
||||
writer := NewWriter(file)
|
||||
for i := int64(1); i <= 10000; i++ {
|
||||
row := &Row{
|
||||
Seq: i,
|
||||
Time: 1000000 + i,
|
||||
Data: map[string]interface{}{
|
||||
"value": i,
|
||||
},
|
||||
}
|
||||
writer.Add(row)
|
||||
}
|
||||
writer.Finish()
|
||||
file.Close()
|
||||
|
||||
// 打开读取器
|
||||
reader, _ := NewReader("bench.sst")
|
||||
defer reader.Close()
|
||||
|
||||
// 性能测试
|
||||
b.ResetTimer()
|
||||
for i := 0; i < b.N; i++ {
|
||||
key := int64(i%10000 + 1)
|
||||
reader.Get(key)
|
||||
}
|
||||
}
|
||||
155
sst/writer.go
Normal file
155
sst/writer.go
Normal file
@@ -0,0 +1,155 @@
|
||||
package sst
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"os"
|
||||
|
||||
"code.tczkiot.com/srdb/btree"
|
||||
"github.com/golang/snappy"
|
||||
)
|
||||
|
||||
// Writer SST 文件写入器
|
||||
type Writer struct {
|
||||
file *os.File
|
||||
builder *btree.Builder
|
||||
dataOffset int64
|
||||
dataStart int64 // 数据起始位置
|
||||
rowCount int64
|
||||
minKey int64
|
||||
maxKey int64
|
||||
minTime int64
|
||||
maxTime int64
|
||||
compression uint8
|
||||
}
|
||||
|
||||
// NewWriter 创建 SST 写入器
|
||||
func NewWriter(file *os.File) *Writer {
|
||||
return &Writer{
|
||||
file: file,
|
||||
builder: btree.NewBuilder(file, HeaderSize),
|
||||
dataOffset: 0, // 先写数据，后面会更新
|
||||
compression: CompressionSnappy,
|
||||
minKey: -1,
|
||||
maxKey: -1,
|
||||
minTime: -1,
|
||||
maxTime: -1,
|
||||
}
|
||||
}
|
||||
|
||||
// Row 表示一行数据
|
||||
type Row struct {
|
||||
Seq int64 // _seq
|
||||
Time int64 // _time
|
||||
Data map[string]any // 用户数据
|
||||
}
|
||||
|
||||
// Add 添加一行数据
|
||||
func (w *Writer) Add(row *Row) error {
|
||||
// 更新统计信息
|
||||
if w.minKey == -1 || row.Seq < w.minKey {
|
||||
w.minKey = row.Seq
|
||||
}
|
||||
if w.maxKey == -1 || row.Seq > w.maxKey {
|
||||
w.maxKey = row.Seq
|
||||
}
|
||||
if w.minTime == -1 || row.Time < w.minTime {
|
||||
w.minTime = row.Time
|
||||
}
|
||||
if w.maxTime == -1 || row.Time > w.maxTime {
|
||||
w.maxTime = row.Time
|
||||
}
|
||||
w.rowCount++
|
||||
|
||||
// 序列化数据 (简单的 JSON 序列化，后续可以优化)
|
||||
data := encodeRow(row)
|
||||
|
||||
// 压缩数据
|
||||
var compressed []byte
|
||||
if w.compression == CompressionSnappy {
|
||||
compressed = snappy.Encode(nil, data)
|
||||
} else {
|
||||
compressed = data
|
||||
}
|
||||
|
||||
// 写入数据块
|
||||
// 第一次写入时，确定数据起始位置
|
||||
if w.dataStart == 0 {
|
||||
// 预留足够空间给 B+Tree 索引
|
||||
// 假设索引最多占用 10% 的空间，最少 1 MB
|
||||
estimatedIndexSize := int64(10 * 1024 * 1024) // 10 MB
|
||||
w.dataStart = HeaderSize + estimatedIndexSize
|
||||
w.dataOffset = w.dataStart
|
||||
}
|
||||
|
||||
offset := w.dataOffset
|
||||
_, err := w.file.WriteAt(compressed, offset)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// 添加到 B+Tree
|
||||
err = w.builder.Add(row.Seq, offset, int32(len(compressed)))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// 更新数据偏移
|
||||
w.dataOffset += int64(len(compressed))
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// Finish 完成写入
|
||||
func (w *Writer) Finish() error {
|
||||
// 1. 构建 B+Tree 索引
|
||||
rootOffset, err := w.builder.Build()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// 2. 计算索引大小
|
||||
indexSize := w.dataStart - HeaderSize
|
||||
|
||||
// 3. 创建 Header
|
||||
header := &Header{
|
||||
Magic: MagicNumber,
|
||||
Version: Version,
|
||||
Compression: w.compression,
|
||||
IndexOffset: HeaderSize,
|
||||
IndexSize: indexSize,
|
||||
RootOffset: rootOffset,
|
||||
DataOffset: w.dataStart,
|
||||
DataSize: w.dataOffset - w.dataStart,
|
||||
RowCount: w.rowCount,
|
||||
MinKey: w.minKey,
|
||||
MaxKey: w.maxKey,
|
||||
MinTime: w.minTime,
|
||||
MaxTime: w.maxTime,
|
||||
}
|
||||
|
||||
// 4. 写入 Header
|
||||
headerData := header.Marshal()
|
||||
_, err = w.file.WriteAt(headerData, 0)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// 5. Sync 到磁盘
|
||||
return w.file.Sync()
|
||||
}
|
||||
|
||||
// encodeRow 编码行数据 (使用二进制格式)
|
||||
func encodeRow(row *Row) []byte {
|
||||
// 使用二进制格式编码
|
||||
encoded, err := encodeRowBinary(row)
|
||||
if err != nil {
|
||||
// 降级到 JSON (不应该发生)
|
||||
data := map[string]interface{}{
|
||||
"_seq": row.Seq,
|
||||
"_time": row.Time,
|
||||
"data": row.Data,
|
||||
}
|
||||
encoded, _ = json.Marshal(data)
|
||||
}
|
||||
return encoded
|
||||
}
|
||||
143
table.go
Normal file
143
table.go
Normal file
@@ -0,0 +1,143 @@
|
||||
package srdb
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"time"
|
||||
|
||||
"code.tczkiot.com/srdb/sst"
|
||||
)
|
||||
|
||||
// Table 表
|
||||
type Table struct {
|
||||
name string // 表名
|
||||
dir string // 表目录
|
||||
schema *Schema // Schema
|
||||
engine *Engine // Engine 实例
|
||||
database *Database // 所属数据库
|
||||
createdAt int64 // 创建时间
|
||||
}
|
||||
|
||||
// createTable 创建新表
|
||||
func createTable(name string, schema *Schema, db *Database) (*Table, error) {
|
||||
// 创建表目录
|
||||
tableDir := filepath.Join(db.dir, name)
|
||||
err := os.MkdirAll(tableDir, 0755)
|
||||
if err != nil {
|
||||
os.RemoveAll(tableDir)
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// 创建 Engine（Engine 会自动保存 Schema 到文件）
|
||||
engine, err := OpenEngine(&EngineOptions{
|
||||
Dir: tableDir,
|
||||
MemTableSize: DefaultMemTableSize,
|
||||
Schema: schema,
|
||||
})
|
||||
if err != nil {
|
||||
os.RemoveAll(tableDir)
|
||||
return nil, err
|
||||
}
|
||||
|
||||
table := &Table{
|
||||
name: name,
|
||||
dir: tableDir,
|
||||
schema: schema,
|
||||
engine: engine,
|
||||
database: db,
|
||||
createdAt: time.Now().Unix(),
|
||||
}
|
||||
|
||||
return table, nil
|
||||
}
|
||||
|
||||
// openTable 打开已存在的表
|
||||
func openTable(name string, db *Database) (*Table, error) {
|
||||
tableDir := filepath.Join(db.dir, name)
|
||||
|
||||
// 打开 Engine（Engine 会自动从 schema.json 恢复 Schema）
|
||||
eng, err := OpenEngine(&EngineOptions{
|
||||
Dir: tableDir,
|
||||
MemTableSize: DefaultMemTableSize,
|
||||
// Schema 不设置，让 Engine 自动从磁盘恢复
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// 从 Engine 获取 Schema
|
||||
sch := eng.GetSchema()
|
||||
|
||||
table := &Table{
|
||||
name: name,
|
||||
dir: tableDir,
|
||||
schema: sch,
|
||||
engine: eng,
|
||||
database: db,
|
||||
}
|
||||
|
||||
return table, nil
|
||||
}
|
||||
|
||||
// GetName 获取表名
|
||||
func (t *Table) GetName() string {
|
||||
return t.name
|
||||
}
|
||||
|
||||
// GetSchema 获取 Schema
|
||||
func (t *Table) GetSchema() *Schema {
|
||||
return t.schema
|
||||
}
|
||||
|
||||
// Insert 插入数据
|
||||
func (t *Table) Insert(data map[string]any) error {
|
||||
return t.engine.Insert(data)
|
||||
}
|
||||
|
||||
// Get 查询数据
|
||||
func (t *Table) Get(seq int64) (*sst.Row, error) {
|
||||
return t.engine.Get(seq)
|
||||
}
|
||||
|
||||
// Query 创建查询构建器
|
||||
func (t *Table) Query() *QueryBuilder {
|
||||
return t.engine.Query()
|
||||
}
|
||||
|
||||
// CreateIndex 创建索引
|
||||
func (t *Table) CreateIndex(field string) error {
|
||||
return t.engine.CreateIndex(field)
|
||||
}
|
||||
|
||||
// DropIndex 删除索引
|
||||
func (t *Table) DropIndex(field string) error {
|
||||
return t.engine.DropIndex(field)
|
||||
}
|
||||
|
||||
// ListIndexes 列出所有索引
|
||||
func (t *Table) ListIndexes() []string {
|
||||
return t.engine.ListIndexes()
|
||||
}
|
||||
|
||||
// Stats 获取统计信息
|
||||
func (t *Table) Stats() *Stats {
|
||||
return t.engine.Stats()
|
||||
}
|
||||
|
||||
// GetEngine 获取底层 Engine
|
||||
func (t *Table) GetEngine() *Engine {
|
||||
return t.engine
|
||||
}
|
||||
|
||||
// Close 关闭表
|
||||
func (t *Table) Close() error {
|
||||
if t.engine != nil {
|
||||
return t.engine.Close()
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// GetCreatedAt 获取表创建时间
|
||||
func (t *Table) GetCreatedAt() int64 {
|
||||
return t.createdAt
|
||||
}
|
||||
206
wal/manager.go
Normal file
206
wal/manager.go
Normal file
@@ -0,0 +1,206 @@
|
||||
package wal
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
)
|
||||
|
||||
// Manager WAL 管理器，管理多个 WAL 文件
|
||||
type Manager struct {
|
||||
dir string
|
||||
currentWAL *WAL
|
||||
currentNumber int64
|
||||
mu sync.Mutex
|
||||
}
|
||||
|
||||
// NewManager 创建 WAL 管理器
|
||||
func NewManager(dir string) (*Manager, error) {
|
||||
// 确保目录存在
|
||||
err := os.MkdirAll(dir, 0755)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// 读取当前 WAL 编号
|
||||
number, err := readCurrentNumber(dir)
|
||||
if err != nil {
|
||||
// 如果读取失败，从 1 开始
|
||||
number = 1
|
||||
}
|
||||
|
||||
// 打开当前 WAL
|
||||
walPath := filepath.Join(dir, fmt.Sprintf("%06d.wal", number))
|
||||
wal, err := Open(walPath)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// 保存当前编号
|
||||
err = saveCurrentNumber(dir, number)
|
||||
if err != nil {
|
||||
wal.Close()
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return &Manager{
|
||||
dir: dir,
|
||||
currentWAL: wal,
|
||||
currentNumber: number,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// Append 追加记录到当前 WAL
|
||||
func (m *Manager) Append(entry *Entry) error {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
|
||||
return m.currentWAL.Append(entry)
|
||||
}
|
||||
|
||||
// Sync 同步当前 WAL 到磁盘
|
||||
func (m *Manager) Sync() error {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
|
||||
return m.currentWAL.Sync()
|
||||
}
|
||||
|
||||
// Rotate 切换到新的 WAL 文件
|
||||
func (m *Manager) Rotate() (int64, error) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
|
||||
// 记录旧的 WAL 编号
|
||||
oldNumber := m.currentNumber
|
||||
|
||||
// 关闭当前 WAL
|
||||
err := m.currentWAL.Close()
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
|
||||
// 创建新 WAL
|
||||
m.currentNumber++
|
||||
walPath := filepath.Join(m.dir, fmt.Sprintf("%06d.wal", m.currentNumber))
|
||||
wal, err := Open(walPath)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
|
||||
m.currentWAL = wal
|
||||
|
||||
// 更新 CURRENT 文件
|
||||
err = saveCurrentNumber(m.dir, m.currentNumber)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
|
||||
return oldNumber, nil
|
||||
}
|
||||
|
||||
// Delete 删除指定的 WAL 文件
|
||||
func (m *Manager) Delete(number int64) error {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
|
||||
walPath := filepath.Join(m.dir, fmt.Sprintf("%06d.wal", number))
|
||||
return os.Remove(walPath)
|
||||
}
|
||||
|
||||
// GetCurrentNumber 获取当前 WAL 编号
|
||||
func (m *Manager) GetCurrentNumber() int64 {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
|
||||
return m.currentNumber
|
||||
}
|
||||
|
||||
// RecoverAll 恢复所有 WAL 文件
|
||||
func (m *Manager) RecoverAll() ([]*Entry, error) {
|
||||
// 查找所有 WAL 文件
|
||||
pattern := filepath.Join(m.dir, "*.wal")
|
||||
files, err := filepath.Glob(pattern)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if len(files) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
// 按文件名排序（确保按时间顺序）
|
||||
sort.Strings(files)
|
||||
|
||||
var allEntries []*Entry
|
||||
|
||||
// 依次读取每个 WAL
|
||||
for _, file := range files {
|
||||
reader, err := NewReader(file)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
|
||||
entries, err := reader.Read()
|
||||
reader.Close()
|
||||
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
|
||||
allEntries = append(allEntries, entries...)
|
||||
}
|
||||
|
||||
return allEntries, nil
|
||||
}
|
||||
|
||||
// ListWALFiles 列出所有 WAL 文件
|
||||
func (m *Manager) ListWALFiles() ([]string, error) {
|
||||
pattern := filepath.Join(m.dir, "*.wal")
|
||||
files, err := filepath.Glob(pattern)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
sort.Strings(files)
|
||||
return files, nil
|
||||
}
|
||||
|
||||
// Close 关闭 WAL 管理器
|
||||
func (m *Manager) Close() error {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
|
||||
if m.currentWAL != nil {
|
||||
return m.currentWAL.Close()
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// readCurrentNumber 读取当前 WAL 编号
|
||||
func readCurrentNumber(dir string) (int64, error) {
|
||||
currentPath := filepath.Join(dir, "CURRENT")
|
||||
data, err := os.ReadFile(currentPath)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
|
||||
number, err := strconv.ParseInt(strings.TrimSpace(string(data)), 10, 64)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
|
||||
return number, nil
|
||||
}
|
||||
|
||||
// saveCurrentNumber 保存当前 WAL 编号
|
||||
func saveCurrentNumber(dir string, number int64) error {
|
||||
currentPath := filepath.Join(dir, "CURRENT")
|
||||
data := []byte(fmt.Sprintf("%d\n", number))
|
||||
return os.WriteFile(currentPath, data, 0644)
|
||||
}
|
||||
208
wal/wal.go
Normal file
208
wal/wal.go
Normal file
@@ -0,0 +1,208 @@
|
||||
package wal
|
||||
|
||||
import (
|
||||
"encoding/binary"
|
||||
"hash/crc32"
|
||||
"io"
|
||||
"os"
|
||||
"sync"
|
||||
)
|
||||
|
||||
const (
|
||||
// Entry 类型
|
||||
EntryTypePut = 1
|
||||
EntryTypeDelete = 2 // 预留，暂不支持
|
||||
|
||||
// Entry Header 大小
|
||||
EntryHeaderSize = 17 // CRC32(4) + Length(4) + Type(1) + Seq(8)
|
||||
)
|
||||
|
||||
// Entry WAL 条目
|
||||
type Entry struct {
|
||||
Type byte // 操作类型
|
||||
Seq int64 // _seq
|
||||
Data []byte // 数据
|
||||
CRC32 uint32 // 校验和
|
||||
}
|
||||
|
||||
// WAL Write-Ahead Log
|
||||
type WAL struct {
|
||||
file *os.File
|
||||
offset int64
|
||||
mu sync.Mutex
|
||||
}
|
||||
|
||||
// Open 打开 WAL 文件
|
||||
func Open(path string) (*WAL, error) {
|
||||
file, err := os.OpenFile(path, os.O_CREATE|os.O_RDWR|os.O_APPEND, 0644)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// 获取当前文件大小
|
||||
stat, err := file.Stat()
|
||||
if err != nil {
|
||||
file.Close()
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return &WAL{
|
||||
file: file,
|
||||
offset: stat.Size(),
|
||||
}, nil
|
||||
}
|
||||
|
||||
// Append 追加一条记录
|
||||
func (w *WAL) Append(entry *Entry) error {
|
||||
w.mu.Lock()
|
||||
defer w.mu.Unlock()
|
||||
|
||||
// 序列化 Entry
|
||||
data := w.marshalEntry(entry)
|
||||
|
||||
// 写入文件
|
||||
_, err := w.file.Write(data)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
w.offset += int64(len(data))
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// Sync 同步到磁盘
|
||||
func (w *WAL) Sync() error {
|
||||
w.mu.Lock()
|
||||
defer w.mu.Unlock()
|
||||
|
||||
return w.file.Sync()
|
||||
}
|
||||
|
||||
// Close 关闭 WAL
|
||||
func (w *WAL) Close() error {
|
||||
w.mu.Lock()
|
||||
defer w.mu.Unlock()
|
||||
|
||||
return w.file.Close()
|
||||
}
|
||||
|
||||
// Truncate 清空 WAL
|
||||
func (w *WAL) Truncate() error {
|
||||
w.mu.Lock()
|
||||
defer w.mu.Unlock()
|
||||
|
||||
err := w.file.Truncate(0)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
_, err = w.file.Seek(0, 0)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
w.offset = 0
|
||||
return nil
|
||||
}
|
||||
|
||||
// marshalEntry 序列化 Entry
|
||||
func (w *WAL) marshalEntry(entry *Entry) []byte {
|
||||
dataLen := len(entry.Data)
|
||||
totalLen := EntryHeaderSize + dataLen
|
||||
|
||||
buf := make([]byte, totalLen)
|
||||
|
||||
// 计算 CRC32 (不包括 CRC32 字段本身)
|
||||
crcData := buf[4:totalLen]
|
||||
binary.LittleEndian.PutUint32(crcData[0:4], uint32(dataLen))
|
||||
crcData[4] = entry.Type
|
||||
binary.LittleEndian.PutUint64(crcData[5:13], uint64(entry.Seq))
|
||||
copy(crcData[13:], entry.Data)
|
||||
|
||||
crc := crc32.ChecksumIEEE(crcData)
|
||||
|
||||
// 写入 CRC32
|
||||
binary.LittleEndian.PutUint32(buf[0:4], crc)
|
||||
|
||||
return buf
|
||||
}
|
||||
|
||||
// Reader WAL 读取器
|
||||
type Reader struct {
|
||||
file *os.File
|
||||
}
|
||||
|
||||
// NewReader 创建 WAL 读取器
|
||||
func NewReader(path string) (*Reader, error) {
|
||||
file, err := os.Open(path)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return &Reader{
|
||||
file: file,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// Read 读取所有 Entry
|
||||
func (r *Reader) Read() ([]*Entry, error) {
|
||||
var entries []*Entry
|
||||
|
||||
for {
|
||||
entry, err := r.readEntry()
|
||||
if err == io.EOF {
|
||||
break
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
entries = append(entries, entry)
|
||||
}
|
||||
|
||||
return entries, nil
|
||||
}
|
||||
|
||||
// Close 关闭读取器
|
||||
func (r *Reader) Close() error {
|
||||
return r.file.Close()
|
||||
}
|
||||
|
||||
// readEntry 读取一条 Entry
|
||||
func (r *Reader) readEntry() (*Entry, error) {
|
||||
// 读取 Header
|
||||
header := make([]byte, EntryHeaderSize)
|
||||
_, err := io.ReadFull(r.file, header)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// 解析 Header
|
||||
crc := binary.LittleEndian.Uint32(header[0:4])
|
||||
dataLen := binary.LittleEndian.Uint32(header[4:8])
|
||||
entryType := header[8]
|
||||
seq := int64(binary.LittleEndian.Uint64(header[9:17]))
|
||||
|
||||
// 读取 Data
|
||||
data := make([]byte, dataLen)
|
||||
_, err = io.ReadFull(r.file, data)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// 验证 CRC32
|
||||
crcData := make([]byte, EntryHeaderSize-4+int(dataLen))
|
||||
copy(crcData[0:EntryHeaderSize-4], header[4:])
|
||||
copy(crcData[EntryHeaderSize-4:], data)
|
||||
|
||||
if crc32.ChecksumIEEE(crcData) != crc {
|
||||
return nil, io.ErrUnexpectedEOF // CRC 校验失败
|
||||
}
|
||||
|
||||
return &Entry{
|
||||
Type: entryType,
|
||||
Seq: seq,
|
||||
Data: data,
|
||||
CRC32: crc,
|
||||
}, nil
|
||||
}
|
||||
130
wal/wal_test.go
Normal file
130
wal/wal_test.go
Normal file
@@ -0,0 +1,130 @@
|
||||
package wal
|
||||
|
||||
import (
|
||||
"os"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestWAL(t *testing.T) {
|
||||
// 1. 创建 WAL
|
||||
wal, err := Open("test.wal")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer os.Remove("test.wal")
|
||||
|
||||
// 2. 写入数据
|
||||
for i := int64(1); i <= 100; i++ {
|
||||
entry := &Entry{
|
||||
Type: EntryTypePut,
|
||||
Seq: i,
|
||||
Data: []byte("value_" + string(rune(i))),
|
||||
}
|
||||
err := wal.Append(entry)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
// 3. Sync
|
||||
err = wal.Sync()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
wal.Close()
|
||||
|
||||
t.Log("Written 100 entries")
|
||||
|
||||
// 4. 读取数据
|
||||
reader, err := NewReader("test.wal")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer reader.Close()
|
||||
|
||||
entries, err := reader.Read()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
if len(entries) != 100 {
|
||||
t.Errorf("Expected 100 entries, got %d", len(entries))
|
||||
}
|
||||
|
||||
// 验证数据
|
||||
for i, entry := range entries {
|
||||
expectedSeq := int64(i + 1)
|
||||
if entry.Seq != expectedSeq {
|
||||
t.Errorf("Entry %d: expected Seq=%d, got %d", i, expectedSeq, entry.Seq)
|
||||
}
|
||||
if entry.Type != EntryTypePut {
|
||||
t.Errorf("Entry %d: expected Type=%d, got %d", i, EntryTypePut, entry.Type)
|
||||
}
|
||||
}
|
||||
|
||||
t.Log("All tests passed!")
|
||||
}
|
||||
|
||||
func TestWALTruncate(t *testing.T) {
|
||||
// 创建 WAL
|
||||
wal, err := Open("test_truncate.wal")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer os.Remove("test_truncate.wal")
|
||||
|
||||
// 写入数据
|
||||
for i := int64(1); i <= 10; i++ {
|
||||
entry := &Entry{
|
||||
Type: EntryTypePut,
|
||||
Seq: i,
|
||||
Data: []byte("value"),
|
||||
}
|
||||
wal.Append(entry)
|
||||
}
|
||||
|
||||
// Truncate
|
||||
err = wal.Truncate()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
wal.Close()
|
||||
|
||||
// 验证文件为空
|
||||
reader, err := NewReader("test_truncate.wal")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer reader.Close()
|
||||
|
||||
entries, err := reader.Read()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
if len(entries) != 0 {
|
||||
t.Errorf("Expected 0 entries after truncate, got %d", len(entries))
|
||||
}
|
||||
|
||||
t.Log("Truncate test passed!")
|
||||
}
|
||||
|
||||
func BenchmarkWALAppend(b *testing.B) {
|
||||
wal, _ := Open("bench.wal")
|
||||
defer os.Remove("bench.wal")
|
||||
defer wal.Close()
|
||||
|
||||
entry := &Entry{
|
||||
Type: EntryTypePut,
|
||||
Seq: 1,
|
||||
Data: make([]byte, 100),
|
||||
}
|
||||
|
||||
b.ResetTimer()
|
||||
for i := 0; i < b.N; i++ {
|
||||
entry.Seq = int64(i)
|
||||
wal.Append(entry)
|
||||
}
|
||||
}
|
||||
552
webui/htmx.go
Normal file
552
webui/htmx.go
Normal file
@@ -0,0 +1,552 @@
|
||||
package webui
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"fmt"
|
||||
"html"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// HTML 渲染辅助函数
|
||||
|
||||
// renderTablesHTML 渲染表列表 HTML
|
||||
func renderTablesHTML(tables []TableListItem) string {
|
||||
var buf bytes.Buffer
|
||||
|
||||
for _, table := range tables {
|
||||
buf.WriteString(`<div class="table-item" data-table="`)
|
||||
buf.WriteString(html.EscapeString(table.Name))
|
||||
buf.WriteString(`">`)
|
||||
buf.WriteString(`<div class="table-header" onclick="selectTable('`)
|
||||
buf.WriteString(html.EscapeString(table.Name))
|
||||
buf.WriteString(`')">`)
|
||||
|
||||
// 左侧：展开图标和表名
|
||||
buf.WriteString(`<div class="table-header-left">`)
|
||||
buf.WriteString(`<span class="expand-icon" onclick="event.stopPropagation(); toggleExpand('`)
|
||||
buf.WriteString(html.EscapeString(table.Name))
|
||||
buf.WriteString(`')">▶</span>`)
|
||||
buf.WriteString(`<span class="table-name">`)
|
||||
buf.WriteString(html.EscapeString(table.Name))
|
||||
buf.WriteString(`</span></div>`)
|
||||
|
||||
// 右侧：字段数量
|
||||
buf.WriteString(`<span class="table-count">`)
|
||||
buf.WriteString(formatCount(int64(len(table.Fields))))
|
||||
buf.WriteString(` fields</span>`)
|
||||
buf.WriteString(`</div>`)
|
||||
|
||||
// Schema 字段列表（默认隐藏）
|
||||
if len(table.Fields) > 0 {
|
||||
buf.WriteString(`<div class="schema-fields">`)
|
||||
for _, field := range table.Fields {
|
||||
buf.WriteString(`<div class="field-item">`)
|
||||
buf.WriteString(`<span class="field-name">`)
|
||||
buf.WriteString(html.EscapeString(field.Name))
|
||||
buf.WriteString(`</span>`)
|
||||
buf.WriteString(`<span class="field-type">`)
|
||||
buf.WriteString(html.EscapeString(field.Type))
|
||||
buf.WriteString(`</span>`)
|
||||
if field.Indexed {
|
||||
buf.WriteString(`<span class="field-indexed">●indexed</span>`)
|
||||
}
|
||||
buf.WriteString(`</div>`)
|
||||
}
|
||||
buf.WriteString(`</div>`)
|
||||
}
|
||||
|
||||
buf.WriteString(`</div>`)
|
||||
}
|
||||
|
||||
return buf.String()
|
||||
}
|
||||
|
||||
// renderDataViewHTML 渲染数据视图 HTML
|
||||
func renderDataViewHTML(tableName string, schema SchemaInfo, tableData TableDataResponse) string {
|
||||
var buf bytes.Buffer
|
||||
|
||||
// 标题
|
||||
buf.WriteString(`<h2>`)
|
||||
buf.WriteString(html.EscapeString(tableName))
|
||||
buf.WriteString(`</h2>`)
|
||||
|
||||
// 视图切换标签
|
||||
buf.WriteString(`<div class="view-tabs">`)
|
||||
buf.WriteString(`<button class="view-tab active" onclick="switchView('`)
|
||||
buf.WriteString(html.EscapeString(tableName))
|
||||
buf.WriteString(`', 'data')">Data</button>`)
|
||||
buf.WriteString(`<button class="view-tab" onclick="switchView('`)
|
||||
buf.WriteString(html.EscapeString(tableName))
|
||||
buf.WriteString(`', 'manifest')">Manifest / LSM-Tree</button>`)
|
||||
buf.WriteString(`</div>`)
|
||||
|
||||
// Schema 部分
|
||||
if len(schema.Fields) > 0 {
|
||||
buf.WriteString(`<div class="schema-section">`)
|
||||
buf.WriteString(`<h3>Schema <span style="font-size: 12px; font-weight: 400; color: var(--text-secondary);">(点击字段卡片选择要显示的列)</span></h3>`)
|
||||
buf.WriteString(`<div class="schema-grid">`)
|
||||
for _, field := range schema.Fields {
|
||||
buf.WriteString(`<div class="schema-field-card selected" data-column="`)
|
||||
buf.WriteString(html.EscapeString(field.Name))
|
||||
buf.WriteString(`" onclick="toggleColumn('`)
|
||||
buf.WriteString(html.EscapeString(field.Name))
|
||||
buf.WriteString(`')">`)
|
||||
buf.WriteString(`<div class="field-item">`)
|
||||
buf.WriteString(`<span class="field-name">`)
|
||||
buf.WriteString(html.EscapeString(field.Name))
|
||||
buf.WriteString(`</span>`)
|
||||
buf.WriteString(`<span class="field-type">`)
|
||||
buf.WriteString(html.EscapeString(field.Type))
|
||||
buf.WriteString(`</span>`)
|
||||
if field.Indexed {
|
||||
buf.WriteString(`<span class="field-indexed">●indexed</span>`)
|
||||
}
|
||||
buf.WriteString(`</div>`)
|
||||
buf.WriteString(`<div class="field-comment">`)
|
||||
if field.Comment != "" {
|
||||
buf.WriteString(html.EscapeString(field.Comment))
|
||||
}
|
||||
buf.WriteString(`</div>`)
|
||||
buf.WriteString(`</div>`)
|
||||
}
|
||||
buf.WriteString(`</div>`)
|
||||
buf.WriteString(`</div>`)
|
||||
}
|
||||
|
||||
// 数据表格
|
||||
buf.WriteString(`<h3>Data (`)
|
||||
buf.WriteString(formatCount(tableData.TotalRows))
|
||||
buf.WriteString(` rows)</h3>`)
|
||||
|
||||
if len(tableData.Data) == 0 {
|
||||
buf.WriteString(`<div class="empty"><p>No data available</p></div>`)
|
||||
return buf.String()
|
||||
}
|
||||
|
||||
// 获取列并排序：_seq 第1列，_time 倒数第2列
|
||||
columns := []string{}
|
||||
otherColumns := []string{}
|
||||
hasSeq := false
|
||||
hasTime := false
|
||||
|
||||
if len(tableData.Data) > 0 {
|
||||
for key := range tableData.Data[0] {
|
||||
if !strings.HasSuffix(key, "_truncated") {
|
||||
if key == "_seq" {
|
||||
hasSeq = true
|
||||
} else if key == "_time" {
|
||||
hasTime = true
|
||||
} else {
|
||||
otherColumns = append(otherColumns, key)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 按顺序组装：_seq, 其他列, _time
|
||||
if hasSeq {
|
||||
columns = append(columns, "_seq")
|
||||
}
|
||||
columns = append(columns, otherColumns...)
|
||||
if hasTime {
|
||||
columns = append(columns, "_time")
|
||||
}
|
||||
|
||||
// 表格
|
||||
buf.WriteString(`<div class="table-wrapper">`)
|
||||
buf.WriteString(`<table class="data-table">`)
|
||||
buf.WriteString(`<thead><tr>`)
|
||||
for _, col := range columns {
|
||||
buf.WriteString(`<th data-column="`)
|
||||
buf.WriteString(html.EscapeString(col))
|
||||
buf.WriteString(`" title="`)
|
||||
buf.WriteString(html.EscapeString(col))
|
||||
buf.WriteString(`">`)
|
||||
buf.WriteString(html.EscapeString(col))
|
||||
buf.WriteString(`</th>`)
|
||||
}
|
||||
buf.WriteString(`<th style="width: 80px;">Actions</th>`)
|
||||
buf.WriteString(`</tr></thead>`)
|
||||
|
||||
buf.WriteString(`<tbody>`)
|
||||
for _, row := range tableData.Data {
|
||||
buf.WriteString(`<tr>`)
|
||||
for _, col := range columns {
|
||||
value := row[col]
|
||||
buf.WriteString(`<td data-column="`)
|
||||
buf.WriteString(html.EscapeString(col))
|
||||
buf.WriteString(`" onclick="showCellContent('`)
|
||||
buf.WriteString(escapeJSString(fmt.Sprintf("%v", value)))
|
||||
buf.WriteString(`')" title="Click to view full content">`)
|
||||
buf.WriteString(html.EscapeString(fmt.Sprintf("%v", value)))
|
||||
|
||||
// 检查是否被截断
|
||||
if truncated, ok := row[col+"_truncated"]; ok && truncated == true {
|
||||
buf.WriteString(`<span class="truncated-icon" title="This field is truncated">✂️</span>`)
|
||||
}
|
||||
|
||||
buf.WriteString(`</td>`)
|
||||
}
|
||||
|
||||
// Actions 列
|
||||
buf.WriteString(`<td style="text-align: center;">`)
|
||||
buf.WriteString(`<button class="row-detail-btn" onclick="showRowDetail('`)
|
||||
buf.WriteString(html.EscapeString(tableName))
|
||||
buf.WriteString(`', `)
|
||||
buf.WriteString(fmt.Sprintf("%v", row["_seq"]))
|
||||
buf.WriteString(`)" title="View full row data">Detail</button>`)
|
||||
buf.WriteString(`</td>`)
|
||||
|
||||
buf.WriteString(`</tr>`)
|
||||
}
|
||||
buf.WriteString(`</tbody>`)
|
||||
buf.WriteString(`</table>`)
|
||||
buf.WriteString(`</div>`)
|
||||
|
||||
// 分页
|
||||
buf.WriteString(renderPagination(tableData))
|
||||
|
||||
return buf.String()
|
||||
}
|
||||
|
||||
// renderManifestViewHTML 渲染 Manifest 视图 HTML
|
||||
func renderManifestViewHTML(tableName string, manifest ManifestResponse) string {
|
||||
var buf bytes.Buffer
|
||||
|
||||
// 标题
|
||||
buf.WriteString(`<h2>`)
|
||||
buf.WriteString(html.EscapeString(tableName))
|
||||
buf.WriteString(`</h2>`)
|
||||
|
||||
// 视图切换标签
|
||||
buf.WriteString(`<div class="view-tabs">`)
|
||||
buf.WriteString(`<button class="view-tab" onclick="switchView('`)
|
||||
buf.WriteString(html.EscapeString(tableName))
|
||||
buf.WriteString(`', 'data')">Data</button>`)
|
||||
buf.WriteString(`<button class="view-tab active" onclick="switchView('`)
|
||||
buf.WriteString(html.EscapeString(tableName))
|
||||
buf.WriteString(`', 'manifest')">Manifest / LSM-Tree</button>`)
|
||||
buf.WriteString(`</div>`)
|
||||
|
||||
// 标题和控制按钮
|
||||
buf.WriteString(`<div style="display: flex; justify-content: space-between; align-items: center; margin-bottom: 15px;">`)
|
||||
buf.WriteString(`<h3>LSM-Tree Structure</h3>`)
|
||||
buf.WriteString(`<div class="control-buttons">`)
|
||||
buf.WriteString(`<button>📖 Expand All</button>`)
|
||||
buf.WriteString(`<button>📕 Collapse All</button>`)
|
||||
buf.WriteString(`</div>`)
|
||||
buf.WriteString(`</div>`)
|
||||
|
||||
// 统计卡片
|
||||
totalLevels := len(manifest.Levels)
|
||||
totalFiles := 0
|
||||
totalSize := int64(0)
|
||||
for _, level := range manifest.Levels {
|
||||
totalFiles += level.FileCount
|
||||
totalSize += level.TotalSize
|
||||
}
|
||||
|
||||
buf.WriteString(`<div class="manifest-stats">`)
|
||||
|
||||
// Active Levels
|
||||
buf.WriteString(`<div class="stat-card">`)
|
||||
buf.WriteString(`<div class="stat-label">Active Levels</div>`)
|
||||
buf.WriteString(`<div class="stat-value">`)
|
||||
buf.WriteString(fmt.Sprintf("%d", totalLevels))
|
||||
buf.WriteString(`</div></div>`)
|
||||
|
||||
// Total Files
|
||||
buf.WriteString(`<div class="stat-card">`)
|
||||
buf.WriteString(`<div class="stat-label">Total Files</div>`)
|
||||
buf.WriteString(`<div class="stat-value">`)
|
||||
buf.WriteString(fmt.Sprintf("%d", totalFiles))
|
||||
buf.WriteString(`</div></div>`)
|
||||
|
||||
// Total Size
|
||||
buf.WriteString(`<div class="stat-card">`)
|
||||
buf.WriteString(`<div class="stat-label">Total Size</div>`)
|
||||
buf.WriteString(`<div class="stat-value">`)
|
||||
buf.WriteString(formatBytes(totalSize))
|
||||
buf.WriteString(`</div></div>`)
|
||||
|
||||
// Next File Number
|
||||
buf.WriteString(`<div class="stat-card">`)
|
||||
buf.WriteString(`<div class="stat-label">Next File Number</div>`)
|
||||
buf.WriteString(`<div class="stat-value">`)
|
||||
buf.WriteString(fmt.Sprintf("%d", manifest.NextFileNumber))
|
||||
buf.WriteString(`</div></div>`)
|
||||
|
||||
// Last Sequence
|
||||
buf.WriteString(`<div class="stat-card">`)
|
||||
buf.WriteString(`<div class="stat-label">Last Sequence</div>`)
|
||||
buf.WriteString(`<div class="stat-value">`)
|
||||
buf.WriteString(fmt.Sprintf("%d", manifest.LastSequence))
|
||||
buf.WriteString(`</div></div>`)
|
||||
|
||||
// Total Compactions
|
||||
buf.WriteString(`<div class="stat-card">`)
|
||||
buf.WriteString(`<div class="stat-label">Total Compactions</div>`)
|
||||
buf.WriteString(`<div class="stat-value">`)
|
||||
totalCompactions := 0
|
||||
if manifest.CompactionStats != nil {
|
||||
if tc, ok := manifest.CompactionStats["total_compactions"]; ok {
|
||||
if tcInt, ok := tc.(float64); ok {
|
||||
totalCompactions = int(tcInt)
|
||||
}
|
||||
}
|
||||
}
|
||||
buf.WriteString(fmt.Sprintf("%d", totalCompactions))
|
||||
buf.WriteString(`</div></div>`)
|
||||
|
||||
buf.WriteString(`</div>`)
|
||||
|
||||
// 渲染所有层级（L0-L6）
|
||||
for i := 0; i <= 6; i++ {
|
||||
var level *LevelInfo
|
||||
for j := range manifest.Levels {
|
||||
if manifest.Levels[j].Level == i {
|
||||
level = &manifest.Levels[j]
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
if level == nil {
|
||||
// 创建空层级
|
||||
level = &LevelInfo{
|
||||
Level: i,
|
||||
FileCount: 0,
|
||||
TotalSize: 0,
|
||||
Score: 0,
|
||||
Files: []FileInfo{},
|
||||
}
|
||||
}
|
||||
|
||||
buf.WriteString(renderLevelCard(*level))
|
||||
}
|
||||
|
||||
return buf.String()
|
||||
}
|
||||
|
||||
// renderLevelCard 渲染层级卡片
|
||||
func renderLevelCard(level LevelInfo) string {
|
||||
var buf bytes.Buffer
|
||||
|
||||
scoreClass := "normal"
|
||||
if level.Score >= 1.0 {
|
||||
scoreClass = "critical"
|
||||
} else if level.Score >= 0.8 {
|
||||
scoreClass = "warning"
|
||||
}
|
||||
|
||||
buf.WriteString(`<div class="level-card" data-level="`)
|
||||
buf.WriteString(fmt.Sprintf("%d", level.Level))
|
||||
buf.WriteString(`">`)
|
||||
buf.WriteString(`<div class="level-header" onclick="toggleLevel(`)
|
||||
buf.WriteString(fmt.Sprintf("%d", level.Level))
|
||||
buf.WriteString(`)">`)
|
||||
|
||||
// 左侧：展开图标和标题
|
||||
buf.WriteString(`<div style="display: flex; align-items: center; gap: 10px;">`)
|
||||
buf.WriteString(`<span class="expand-icon">▶</span>`)
|
||||
buf.WriteString(`<div class="level-title">Level `)
|
||||
buf.WriteString(fmt.Sprintf("%d", level.Level))
|
||||
buf.WriteString(`</div></div>`)
|
||||
|
||||
// 右侧：统计信息
|
||||
buf.WriteString(`<div class="level-stats">`)
|
||||
buf.WriteString(`<span>`)
|
||||
buf.WriteString(fmt.Sprintf("%d", level.FileCount))
|
||||
buf.WriteString(` files</span>`)
|
||||
buf.WriteString(`<span>`)
|
||||
buf.WriteString(formatBytes(level.TotalSize))
|
||||
buf.WriteString(`</span>`)
|
||||
buf.WriteString(`<span class="score-badge `)
|
||||
buf.WriteString(scoreClass)
|
||||
buf.WriteString(`">Score: `)
|
||||
buf.WriteString(fmt.Sprintf("%.2f", level.Score))
|
||||
buf.WriteString(`</span>`)
|
||||
buf.WriteString(`</div>`)
|
||||
|
||||
buf.WriteString(`</div>`)
|
||||
|
||||
// 文件列表（默认隐藏）
|
||||
buf.WriteString(`<div class="file-list">`)
|
||||
if len(level.Files) == 0 {
|
||||
buf.WriteString(`<div class="empty-files">No files in this level</div>`)
|
||||
} else {
|
||||
for _, file := range level.Files {
|
||||
buf.WriteString(`<div class="file-card">`)
|
||||
buf.WriteString(`<div class="file-header">`)
|
||||
buf.WriteString(`<span>File #`)
|
||||
buf.WriteString(fmt.Sprintf("%d", file.FileNumber))
|
||||
buf.WriteString(`</span>`)
|
||||
buf.WriteString(`<b>`)
|
||||
buf.WriteString(formatBytes(file.FileSize))
|
||||
buf.WriteString(`</b>`)
|
||||
buf.WriteString(`</div>`)
|
||||
|
||||
buf.WriteString(`<div class="file-detail">`)
|
||||
buf.WriteString(`<span>Key Range:</span>`)
|
||||
buf.WriteString(`<span>`)
|
||||
buf.WriteString(fmt.Sprintf("%d - %d", file.MinKey, file.MaxKey))
|
||||
buf.WriteString(`</span></div>`)
|
||||
|
||||
buf.WriteString(`<div class="file-detail">`)
|
||||
buf.WriteString(`<span>Rows:</span>`)
|
||||
buf.WriteString(`<span>`)
|
||||
buf.WriteString(formatCount(file.RowCount))
|
||||
buf.WriteString(`</span></div>`)
|
||||
|
||||
buf.WriteString(`</div>`)
|
||||
}
|
||||
}
|
||||
buf.WriteString(`</div>`)
|
||||
|
||||
buf.WriteString(`</div>`)
|
||||
return buf.String()
|
||||
}
|
||||
|
||||
// renderPagination 渲染分页 HTML
|
||||
func renderPagination(data TableDataResponse) string {
|
||||
var buf bytes.Buffer
|
||||
|
||||
buf.WriteString(`<div class="pagination">`)
|
||||
|
||||
// 页大小选择器
|
||||
buf.WriteString(`<select onchange="changePageSize(this.value)">`)
|
||||
for _, size := range []int{10, 20, 50, 100} {
|
||||
buf.WriteString(`<option value="`)
|
||||
buf.WriteString(fmt.Sprintf("%d", size))
|
||||
buf.WriteString(`"`)
|
||||
if int64(size) == data.PageSize {
|
||||
buf.WriteString(` selected`)
|
||||
}
|
||||
buf.WriteString(`>`)
|
||||
buf.WriteString(fmt.Sprintf("%d", size))
|
||||
buf.WriteString(` / page</option>`)
|
||||
}
|
||||
buf.WriteString(`</select>`)
|
||||
|
||||
// 上一页按钮
|
||||
buf.WriteString(`<button onclick="changePage(-1)"`)
|
||||
if data.Page <= 1 {
|
||||
buf.WriteString(` disabled`)
|
||||
}
|
||||
buf.WriteString(`>Previous</button>`)
|
||||
|
||||
// 页码信息
|
||||
buf.WriteString(`<span>Page `)
|
||||
buf.WriteString(fmt.Sprintf("%d", data.Page))
|
||||
buf.WriteString(` of `)
|
||||
buf.WriteString(fmt.Sprintf("%d", data.TotalPages))
|
||||
buf.WriteString(` (`)
|
||||
buf.WriteString(formatCount(data.TotalRows))
|
||||
buf.WriteString(` rows)</span>`)
|
||||
|
||||
// 跳转输入框
|
||||
buf.WriteString(`<input type="number" min="1" max="`)
|
||||
buf.WriteString(fmt.Sprintf("%d", data.TotalPages))
|
||||
buf.WriteString(`" placeholder="Jump to" onkeydown="if(event.key==='Enter') jumpToPage(this.value)">`)
|
||||
|
||||
// Go 按钮
|
||||
buf.WriteString(`<button onclick="jumpToPage(this.previousElementSibling.value)">Go</button>`)
|
||||
|
||||
// 下一页按钮
|
||||
buf.WriteString(`<button onclick="changePage(1)"`)
|
||||
if data.Page >= data.TotalPages {
|
||||
buf.WriteString(` disabled`)
|
||||
}
|
||||
buf.WriteString(`>Next</button>`)
|
||||
|
||||
buf.WriteString(`</div>`)
|
||||
return buf.String()
|
||||
}
|
||||
|
||||
// formatBytes 格式化字节数
|
||||
func formatBytes(bytes int64) string {
|
||||
if bytes == 0 {
|
||||
return "0 B"
|
||||
}
|
||||
const k = 1024
|
||||
sizes := []string{"B", "KB", "MB", "GB", "TB"}
|
||||
i := 0
|
||||
size := float64(bytes)
|
||||
for size >= k && i < len(sizes)-1 {
|
||||
size /= k
|
||||
i++
|
||||
}
|
||||
return fmt.Sprintf("%.2f %s", size, sizes[i])
|
||||
}
|
||||
|
||||
// formatCount 格式化数量（K/M）
|
||||
func formatCount(count int64) string {
|
||||
if count >= 1000000 {
|
||||
return fmt.Sprintf("%.1fM", float64(count)/1000000)
|
||||
}
|
||||
if count >= 1000 {
|
||||
return fmt.Sprintf("%.1fK", float64(count)/1000)
|
||||
}
|
||||
return fmt.Sprintf("%d", count)
|
||||
}
|
||||
|
||||
// escapeJSString 转义 JavaScript 字符串
|
||||
func escapeJSString(s string) string {
|
||||
s = strings.ReplaceAll(s, `\`, `\\`)
|
||||
s = strings.ReplaceAll(s, `'`, `\'`)
|
||||
s = strings.ReplaceAll(s, `"`, `\"`)
|
||||
s = strings.ReplaceAll(s, "\n", `\n`)
|
||||
s = strings.ReplaceAll(s, "\r", `\r`)
|
||||
s = strings.ReplaceAll(s, "\t", `\t`)
|
||||
return s
|
||||
}
|
||||
|
||||
// 数据结构定义
|
||||
type TableListItem struct {
|
||||
Name string `json:"name"`
|
||||
CreatedAt int64 `json:"created_at"`
|
||||
Fields []FieldInfo `json:"fields"`
|
||||
}
|
||||
|
||||
type FieldInfo struct {
|
||||
Name string `json:"name"`
|
||||
Type string `json:"type"`
|
||||
Indexed bool `json:"indexed"`
|
||||
Comment string `json:"comment"`
|
||||
}
|
||||
|
||||
type SchemaInfo struct {
|
||||
Name string `json:"name"`
|
||||
Fields []FieldInfo `json:"fields"`
|
||||
}
|
||||
|
||||
type TableDataResponse struct {
|
||||
Data []map[string]any `json:"data"`
|
||||
Page int64 `json:"page"`
|
||||
PageSize int64 `json:"pageSize"`
|
||||
TotalRows int64 `json:"totalRows"`
|
||||
TotalPages int64 `json:"totalPages"`
|
||||
}
|
||||
|
||||
type ManifestResponse struct {
|
||||
Levels []LevelInfo `json:"levels"`
|
||||
NextFileNumber int64 `json:"next_file_number"`
|
||||
LastSequence int64 `json:"last_sequence"`
|
||||
CompactionStats map[string]any `json:"compaction_stats"`
|
||||
}
|
||||
|
||||
type LevelInfo struct {
|
||||
Level int `json:"level"`
|
||||
FileCount int `json:"file_count"`
|
||||
TotalSize int64 `json:"total_size"`
|
||||
Score float64 `json:"score"`
|
||||
Files []FileInfo `json:"files"`
|
||||
}
|
||||
|
||||
type FileInfo struct {
|
||||
FileNumber int64 `json:"file_number"`
|
||||
Level int `json:"level"`
|
||||
FileSize int64 `json:"file_size"`
|
||||
MinKey int64 `json:"min_key"`
|
||||
MaxKey int64 `json:"max_key"`
|
||||
RowCount int64 `json:"row_count"`
|
||||
}
|
||||
903
webui/static/css/styles.css
Normal file
903
webui/static/css/styles.css
Normal file
@@ -0,0 +1,903 @@
|
||||
/* SRDB WebUI - Modern Design */
|
||||
|
||||
:root {
|
||||
/* 主色调 - 优雅的紫蓝色 */
|
||||
--primary: #6366f1;
|
||||
--primary-dark: #4f46e5;
|
||||
--primary-light: #818cf8;
|
||||
--primary-bg: rgba(99, 102, 241, 0.1);
|
||||
|
||||
/* 背景色 */
|
||||
--bg-main: #0f0f1a;
|
||||
--bg-surface: #1a1a2e;
|
||||
--bg-elevated: #222236;
|
||||
--bg-hover: #2a2a3e;
|
||||
|
||||
/* 文字颜色 */
|
||||
--text-primary: #ffffff;
|
||||
--text-secondary: #a0a0b0;
|
||||
--text-tertiary: #6b6b7b;
|
||||
|
||||
/* 边框和分隔线 */
|
||||
--border-color: rgba(255, 255, 255, 0.1);
|
||||
--border-hover: rgba(255, 255, 255, 0.2);
|
||||
|
||||
/* 状态颜色 */
|
||||
--success: #10b981;
|
||||
--warning: #f59e0b;
|
||||
--danger: #ef4444;
|
||||
--info: #3b82f6;
|
||||
|
||||
/* 阴影 */
|
||||
--shadow-sm: 0 1px 2px 0 rgba(0, 0, 0, 0.3);
|
||||
--shadow-md:
|
||||
0 4px 6px -1px rgba(0, 0, 0, 0.4), 0 2px 4px -1px rgba(0, 0, 0, 0.3);
|
||||
--shadow-lg:
|
||||
0 10px 15px -3px rgba(0, 0, 0, 0.5), 0 4px 6px -2px rgba(0, 0, 0, 0.3);
|
||||
--shadow-xl:
|
||||
0 20px 25px -5px rgba(0, 0, 0, 0.5), 0 10px 10px -5px rgba(0, 0, 0, 0.3);
|
||||
|
||||
/* 圆角 */
|
||||
--radius-sm: 6px;
|
||||
--radius-md: 8px;
|
||||
--radius-lg: 12px;
|
||||
--radius-xl: 16px;
|
||||
|
||||
/* 过渡 */
|
||||
--transition: all 0.2s cubic-bezier(0.4, 0, 0.2, 1);
|
||||
}
|
||||
|
||||
* {
|
||||
box-sizing: border-box;
|
||||
margin: 0;
|
||||
padding: 0;
|
||||
}
|
||||
|
||||
body {
|
||||
font-family:
|
||||
"Inter",
|
||||
-apple-system,
|
||||
BlinkMacSystemFont,
|
||||
"Segoe UI",
|
||||
Roboto,
|
||||
sans-serif;
|
||||
background: var(--bg-main);
|
||||
color: var(--text-primary);
|
||||
line-height: 1.6;
|
||||
font-size: 14px;
|
||||
-webkit-font-smoothing: antialiased;
|
||||
-moz-osx-font-smoothing: grayscale;
|
||||
}
|
||||
|
||||
/* 布局 */
|
||||
.container {
|
||||
display: flex;
|
||||
height: 100vh;
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
/* 侧边栏 */
|
||||
.sidebar {
|
||||
width: 280px;
|
||||
background: var(--bg-surface);
|
||||
border-right: 1px solid var(--border-color);
|
||||
overflow-y: auto;
|
||||
overflow-x: hidden;
|
||||
padding: 16px 12px;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 8px;
|
||||
}
|
||||
|
||||
.sidebar::-webkit-scrollbar {
|
||||
width: 6px;
|
||||
}
|
||||
|
||||
.sidebar::-webkit-scrollbar-track {
|
||||
background: transparent;
|
||||
}
|
||||
|
||||
.sidebar::-webkit-scrollbar-thumb {
|
||||
background: rgba(255, 255, 255, 0.1);
|
||||
border-radius: 3px;
|
||||
}
|
||||
|
||||
.sidebar::-webkit-scrollbar-thumb:hover {
|
||||
background: rgba(255, 255, 255, 0.15);
|
||||
}
|
||||
|
||||
.sidebar h1 {
|
||||
font-size: 18px;
|
||||
font-weight: 700;
|
||||
letter-spacing: -0.02em;
|
||||
background: linear-gradient(135deg, var(--primary-light), var(--primary));
|
||||
-webkit-background-clip: text;
|
||||
-webkit-text-fill-color: transparent;
|
||||
margin-bottom: 4px;
|
||||
}
|
||||
|
||||
/* 主内容区 */
|
||||
.main {
|
||||
flex: 1;
|
||||
padding: 20px;
|
||||
overflow-y: auto;
|
||||
overflow-x: hidden;
|
||||
background: var(--bg-main);
|
||||
}
|
||||
|
||||
.main h2 {
|
||||
font-size: 24px;
|
||||
font-weight: 700;
|
||||
margin-bottom: 16px;
|
||||
background: linear-gradient(
|
||||
135deg,
|
||||
var(--text-primary) 0%,
|
||||
var(--primary-light) 100%
|
||||
);
|
||||
-webkit-background-clip: text;
|
||||
-webkit-text-fill-color: transparent;
|
||||
letter-spacing: -0.02em;
|
||||
}
|
||||
|
||||
.main h3 {
|
||||
font-size: 16px;
|
||||
font-weight: 600;
|
||||
margin-bottom: 12px;
|
||||
margin-top: 20px;
|
||||
color: var(--text-primary);
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
}
|
||||
|
||||
.main h3::before {
|
||||
content: "";
|
||||
width: 3px;
|
||||
height: 18px;
|
||||
background: linear-gradient(135deg, var(--primary), var(--primary-light));
|
||||
border-radius: 2px;
|
||||
}
|
||||
|
||||
.main::-webkit-scrollbar {
|
||||
width: 8px;
|
||||
}
|
||||
|
||||
.main::-webkit-scrollbar-track {
|
||||
background: transparent;
|
||||
}
|
||||
|
||||
.main::-webkit-scrollbar-thumb {
|
||||
background: rgba(255, 255, 255, 0.1);
|
||||
border-radius: 4px;
|
||||
}
|
||||
|
||||
.main::-webkit-scrollbar-thumb:hover {
|
||||
background: rgba(255, 255, 255, 0.15);
|
||||
}
|
||||
|
||||
/* 表列表卡片 */
|
||||
.table-item {
|
||||
margin-bottom: 6px;
|
||||
border-radius: var(--radius-md);
|
||||
overflow: hidden;
|
||||
transition: var(--transition);
|
||||
}
|
||||
|
||||
.table-header {
|
||||
padding: 10px 12px;
|
||||
background: var(--bg-elevated);
|
||||
border: 1px solid var(--border-color);
|
||||
border-radius: var(--radius-md);
|
||||
cursor: pointer;
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
align-items: center;
|
||||
transition: var(--transition);
|
||||
}
|
||||
|
||||
.table-header:hover {
|
||||
background: var(--bg-hover);
|
||||
border-color: var(--border-hover);
|
||||
/*transform: translateX(2px);*/
|
||||
}
|
||||
|
||||
.table-header.selected,
|
||||
.table-item.selected .table-header {
|
||||
background: linear-gradient(135deg, var(--primary), var(--primary-dark));
|
||||
border-color: var(--primary);
|
||||
box-shadow: 0 0 0 3px var(--primary-bg);
|
||||
}
|
||||
|
||||
.table-header-left {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 10px;
|
||||
flex: 1;
|
||||
min-width: 0;
|
||||
}
|
||||
|
||||
.expand-icon {
|
||||
font-size: 10px;
|
||||
color: var(--text-secondary);
|
||||
transition: var(--transition);
|
||||
user-select: none;
|
||||
flex-shrink: 0;
|
||||
}
|
||||
|
||||
.expand-icon.expanded {
|
||||
transform: rotate(90deg);
|
||||
}
|
||||
|
||||
.table-name {
|
||||
font-weight: 600;
|
||||
font-size: 14px;
|
||||
overflow: hidden;
|
||||
text-overflow: ellipsis;
|
||||
white-space: nowrap;
|
||||
flex: 1;
|
||||
}
|
||||
|
||||
.table-count {
|
||||
font-size: 11px;
|
||||
font-weight: 500;
|
||||
padding: 3px 8px;
|
||||
background: rgba(255, 255, 255, 0.1);
|
||||
border-radius: 12px;
|
||||
color: var(--text-secondary);
|
||||
flex-shrink: 0;
|
||||
}
|
||||
|
||||
.table-item.selected .table-count {
|
||||
background: rgba(255, 255, 255, 0.2);
|
||||
color: white;
|
||||
}
|
||||
|
||||
/* Schema 字段列表 */
|
||||
.schema-fields {
|
||||
display: none;
|
||||
margin-top: 8px;
|
||||
padding: 10px 12px;
|
||||
background: rgba(255, 255, 255, 0.03);
|
||||
border-left: 2px solid var(--primary);
|
||||
border-radius: var(--radius-md);
|
||||
gap: 6px;
|
||||
flex-direction: column;
|
||||
}
|
||||
|
||||
.field-item {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
gap: 8px;
|
||||
padding: 4px 0;
|
||||
}
|
||||
|
||||
.field-name {
|
||||
font-size: 13px;
|
||||
font-weight: 500;
|
||||
color: var(--text-primary);
|
||||
min-width: 90px;
|
||||
}
|
||||
|
||||
.field-type {
|
||||
font-size: 12px;
|
||||
font-family: "SF Mono", Monaco, monospace;
|
||||
color: var(--primary-light);
|
||||
background: rgba(99, 102, 241, 0.15);
|
||||
padding: 2px 8px;
|
||||
border-radius: 4px;
|
||||
}
|
||||
|
||||
.field-indexed {
|
||||
font-size: 10px;
|
||||
font-weight: 600;
|
||||
color: var(--success);
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.05em;
|
||||
}
|
||||
|
||||
.field-comment {
|
||||
font-size: 12px;
|
||||
color: #999;
|
||||
margin-top: 4px;
|
||||
}
|
||||
|
||||
/* 视图切换标签 */
|
||||
.view-tabs {
|
||||
display: flex;
|
||||
gap: 8px;
|
||||
margin-bottom: 20px;
|
||||
padding: 4px;
|
||||
background: var(--bg-surface);
|
||||
border-radius: var(--radius-lg);
|
||||
border: 1px solid var(--border-color);
|
||||
box-shadow: var(--shadow-sm);
|
||||
}
|
||||
|
||||
.view-tab {
|
||||
padding: 10px 20px;
|
||||
background: transparent;
|
||||
border: none;
|
||||
border-radius: var(--radius-md);
|
||||
cursor: pointer;
|
||||
font-size: 14px;
|
||||
font-weight: 600;
|
||||
color: var(--text-secondary);
|
||||
transition: var(--transition);
|
||||
position: relative;
|
||||
letter-spacing: -0.01em;
|
||||
}
|
||||
|
||||
.view-tab:hover {
|
||||
color: var(--text-primary);
|
||||
background: rgba(255, 255, 255, 0.05);
|
||||
}
|
||||
|
||||
.view-tab.active {
|
||||
color: white;
|
||||
background: linear-gradient(
|
||||
135deg,
|
||||
var(--primary) 0%,
|
||||
var(--primary-dark) 100%
|
||||
);
|
||||
box-shadow: 0 2px 8px rgba(99, 102, 241, 0.3);
|
||||
}
|
||||
|
||||
/* Schema 展示 */
|
||||
/*.schema-section {
|
||||
background: linear-gradient(
|
||||
135deg,
|
||||
rgba(99, 102, 241, 0.05) 0%,
|
||||
rgba(99, 102, 241, 0.02) 100%
|
||||
);
|
||||
border: 1px solid var(--border-color);
|
||||
border-radius: var(--radius-lg);
|
||||
padding: 18px;
|
||||
margin-bottom: 20px;
|
||||
box-shadow: var(--shadow-sm);
|
||||
}*/
|
||||
|
||||
.schema-section h3 {
|
||||
font-size: 15px;
|
||||
font-weight: 600;
|
||||
margin-bottom: 14px;
|
||||
margin-top: 0;
|
||||
color: var(--text-primary);
|
||||
}
|
||||
|
||||
.schema-grid {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(auto-fill, minmax(280px, 1fr));
|
||||
gap: 10px;
|
||||
}
|
||||
|
||||
.schema-field-card {
|
||||
background: var(--bg-elevated);
|
||||
border: 1px solid var(--border-color);
|
||||
border-radius: var(--radius-md);
|
||||
padding: 12px;
|
||||
transition: var(--transition);
|
||||
cursor: pointer;
|
||||
position: relative;
|
||||
opacity: 0.5;
|
||||
}
|
||||
|
||||
.schema-field-card.selected {
|
||||
opacity: 1;
|
||||
border-color: var(--primary);
|
||||
background: linear-gradient(
|
||||
135deg,
|
||||
rgba(99, 102, 241, 0.1) 0%,
|
||||
rgba(99, 102, 241, 0.05) 100%
|
||||
);
|
||||
}
|
||||
|
||||
/*.schema-field-card::after {
|
||||
content: "✓";
|
||||
position: absolute;
|
||||
bottom: 8px;
|
||||
right: 8px;
|
||||
font-size: 14px;
|
||||
font-weight: bold;
|
||||
color: var(--primary);
|
||||
opacity: 0;
|
||||
transition: var(--transition);
|
||||
}
|
||||
|
||||
.schema-field-card.selected::after {
|
||||
opacity: 1;
|
||||
}*/
|
||||
|
||||
.schema-field-card:hover {
|
||||
border-color: var(--primary-light);
|
||||
transform: translateY(-2px);
|
||||
box-shadow: var(--shadow-md);
|
||||
}
|
||||
|
||||
/* 数据表格 */
|
||||
.table-wrapper {
|
||||
overflow-x: auto;
|
||||
margin-bottom: 16px;
|
||||
border-radius: var(--radius-lg);
|
||||
border: 1px solid var(--border-color);
|
||||
background: var(--bg-surface);
|
||||
}
|
||||
|
||||
.data-table {
|
||||
width: 100%;
|
||||
border-collapse: collapse;
|
||||
}
|
||||
|
||||
.data-table th {
|
||||
background: var(--bg-elevated);
|
||||
padding: 10px 12px;
|
||||
text-align: left;
|
||||
font-size: 11px;
|
||||
font-weight: 600;
|
||||
color: var(--text-secondary);
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.05em;
|
||||
border-bottom: 1px solid var(--border-color);
|
||||
position: sticky;
|
||||
top: 0;
|
||||
z-index: 10;
|
||||
}
|
||||
|
||||
.data-table td {
|
||||
padding: 10px 12px;
|
||||
border-bottom: 1px solid var(--border-color);
|
||||
font-size: 13px;
|
||||
color: var(--text-primary);
|
||||
max-width: 400px;
|
||||
overflow: hidden;
|
||||
text-overflow: ellipsis;
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
.data-table tbody tr {
|
||||
transition: var(--transition);
|
||||
}
|
||||
|
||||
.data-table tbody tr:hover {
|
||||
background: rgba(255, 255, 255, 0.03);
|
||||
}
|
||||
|
||||
.data-table tbody tr:last-child td {
|
||||
border-bottom: none;
|
||||
}
|
||||
|
||||
/* 分页 */
|
||||
.pagination {
|
||||
margin-top: 16px;
|
||||
display: flex;
|
||||
justify-content: center;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
}
|
||||
|
||||
.pagination button,
|
||||
.pagination select,
|
||||
.pagination input {
|
||||
padding: 8px 12px;
|
||||
background: var(--bg-surface);
|
||||
border: 1px solid var(--border-color);
|
||||
border-radius: var(--radius-md);
|
||||
color: var(--text-primary);
|
||||
font-size: 13px;
|
||||
font-weight: 500;
|
||||
cursor: pointer;
|
||||
transition: var(--transition);
|
||||
}
|
||||
|
||||
.pagination button:hover:not(:disabled) {
|
||||
background: var(--primary);
|
||||
border-color: var(--primary);
|
||||
box-shadow: var(--shadow-md);
|
||||
}
|
||||
|
||||
.pagination button:disabled {
|
||||
opacity: 0.4;
|
||||
cursor: not-allowed;
|
||||
}
|
||||
|
||||
.pagination input[type="number"] {
|
||||
width: 80px;
|
||||
text-align: center;
|
||||
}
|
||||
|
||||
.pagination select {
|
||||
cursor: pointer;
|
||||
}
|
||||
|
||||
/* Manifest / LSM-Tree */
|
||||
.level-card {
|
||||
background: var(--bg-surface);
|
||||
border: 1px solid var(--border-color);
|
||||
border-radius: var(--radius-lg);
|
||||
padding: 14px;
|
||||
margin-bottom: 12px;
|
||||
transition: var(--transition);
|
||||
}
|
||||
|
||||
.level-card:hover {
|
||||
border-color: var(--border-hover);
|
||||
box-shadow: var(--shadow-md);
|
||||
}
|
||||
|
||||
.level-header {
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
align-items: center;
|
||||
cursor: pointer;
|
||||
}
|
||||
|
||||
.level-title {
|
||||
font-size: 16px;
|
||||
font-weight: 600;
|
||||
background: linear-gradient(
|
||||
135deg,
|
||||
var(--text-primary),
|
||||
var(--text-secondary)
|
||||
);
|
||||
-webkit-background-clip: text;
|
||||
-webkit-text-fill-color: transparent;
|
||||
}
|
||||
|
||||
.level-stats {
|
||||
display: flex;
|
||||
gap: 16px;
|
||||
font-size: 12px;
|
||||
color: var(--text-secondary);
|
||||
}
|
||||
|
||||
.score-badge {
|
||||
padding: 4px 12px;
|
||||
border-radius: 12px;
|
||||
font-size: 12px;
|
||||
font-weight: 600;
|
||||
}
|
||||
|
||||
.score-badge.normal {
|
||||
background: rgba(16, 185, 129, 0.15);
|
||||
color: var(--success);
|
||||
}
|
||||
|
||||
.score-badge.warning {
|
||||
background: rgba(245, 158, 11, 0.15);
|
||||
color: var(--warning);
|
||||
}
|
||||
|
||||
.score-badge.critical {
|
||||
background: rgba(239, 68, 68, 0.15);
|
||||
color: var(--danger);
|
||||
}
|
||||
|
||||
.file-list {
|
||||
display: none;
|
||||
margin-top: 12px;
|
||||
grid-template-columns: repeat(auto-fill, minmax(350px, 1fr));
|
||||
gap: 10px;
|
||||
padding-top: 8px;
|
||||
/*border-top: 1px solid var(--border-color);*/
|
||||
}
|
||||
|
||||
.file-card {
|
||||
background: var(--bg-elevated);
|
||||
border: 1px solid var(--border-color);
|
||||
border-radius: var(--radius-md);
|
||||
padding: 12px;
|
||||
font-size: 12px;
|
||||
}
|
||||
|
||||
.file-header {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
font-weight: 600;
|
||||
margin-bottom: 12px;
|
||||
color: var(--text-primary);
|
||||
}
|
||||
|
||||
.file-detail {
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
/*padding: 4px 0;*/
|
||||
color: var(--text-secondary);
|
||||
font-size: 12px;
|
||||
}
|
||||
|
||||
/* Modal */
|
||||
.modal {
|
||||
display: none;
|
||||
position: fixed;
|
||||
top: 0;
|
||||
left: 0;
|
||||
width: 100%;
|
||||
height: 100%;
|
||||
background: rgba(0, 0, 0, 0.8);
|
||||
backdrop-filter: blur(8px);
|
||||
z-index: 1000;
|
||||
justify-content: center;
|
||||
align-items: center;
|
||||
animation: fadeIn 0.2s ease-out;
|
||||
}
|
||||
|
||||
@keyframes fadeIn {
|
||||
from {
|
||||
opacity: 0;
|
||||
}
|
||||
to {
|
||||
opacity: 1;
|
||||
}
|
||||
}
|
||||
|
||||
.modal-content {
|
||||
background: var(--bg-surface);
|
||||
border: 1px solid var(--border-color);
|
||||
border-radius: var(--radius-xl);
|
||||
max-width: 90%;
|
||||
max-height: 85%;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
box-shadow: var(--shadow-xl);
|
||||
animation: slideUp 0.3s ease-out;
|
||||
}
|
||||
|
||||
@keyframes slideUp {
|
||||
from {
|
||||
transform: translateY(20px);
|
||||
opacity: 0;
|
||||
}
|
||||
to {
|
||||
transform: translateY(0);
|
||||
opacity: 1;
|
||||
}
|
||||
}
|
||||
|
||||
.modal-header {
|
||||
padding: 16px 20px;
|
||||
border-bottom: 1px solid var(--border-color);
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
align-items: center;
|
||||
gap: 24px;
|
||||
}
|
||||
|
||||
.modal-header h3 {
|
||||
font-size: 16px;
|
||||
font-weight: 600;
|
||||
color: var(--text-primary);
|
||||
}
|
||||
|
||||
.modal-close {
|
||||
background: transparent;
|
||||
color: var(--text-secondary);
|
||||
border: 1px solid var(--border-color);
|
||||
width: 32px;
|
||||
height: 32px;
|
||||
padding: 0;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
border-radius: 50%;
|
||||
cursor: pointer;
|
||||
transition: var(--transition);
|
||||
flex-shrink: 0;
|
||||
}
|
||||
|
||||
.modal-close svg {
|
||||
width: 18px;
|
||||
height: 18px;
|
||||
transition: inherit;
|
||||
}
|
||||
|
||||
.modal-close:hover {
|
||||
background: rgba(239, 68, 68, 0.1);
|
||||
border-color: var(--danger);
|
||||
color: var(--danger);
|
||||
transform: rotate(90deg);
|
||||
}
|
||||
|
||||
.modal-body {
|
||||
padding: 16px;
|
||||
overflow: auto;
|
||||
font-family: "SF Mono", Monaco, monospace;
|
||||
font-size: 12px;
|
||||
line-height: 1.6;
|
||||
}
|
||||
|
||||
.modal-body pre {
|
||||
white-space: pre-wrap;
|
||||
word-break: break-word;
|
||||
margin: 0;
|
||||
color: var(--text-primary);
|
||||
}
|
||||
|
||||
/* 按钮 */
|
||||
button {
|
||||
cursor: pointer;
|
||||
transition: var(--transition);
|
||||
font-family: inherit;
|
||||
}
|
||||
|
||||
.row-detail-btn {
|
||||
background: var(--primary);
|
||||
color: white;
|
||||
border: none;
|
||||
padding: 6px 12px;
|
||||
border-radius: var(--radius-sm);
|
||||
font-size: 12px;
|
||||
font-weight: 600;
|
||||
transition: var(--transition);
|
||||
}
|
||||
|
||||
.row-detail-btn:hover {
|
||||
background: var(--primary-dark);
|
||||
box-shadow: var(--shadow-md);
|
||||
transform: translateY(-1px);
|
||||
}
|
||||
|
||||
/* 空状态和加载 */
|
||||
.loading,
|
||||
.empty,
|
||||
.error {
|
||||
text-align: center;
|
||||
padding: 60px 30px;
|
||||
}
|
||||
|
||||
.empty h2 {
|
||||
font-size: 20px;
|
||||
font-weight: 600;
|
||||
margin-bottom: 10px;
|
||||
color: var(--text-primary);
|
||||
}
|
||||
|
||||
.empty p {
|
||||
font-size: 13px;
|
||||
color: var(--text-secondary);
|
||||
}
|
||||
|
||||
.error {
|
||||
color: var(--danger);
|
||||
}
|
||||
|
||||
/* Manifest stats */
|
||||
.manifest-stats {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(auto-fit, minmax(200px, 1fr));
|
||||
gap: 12px;
|
||||
margin-bottom: 16px;
|
||||
}
|
||||
|
||||
.stat-card {
|
||||
background: var(--bg-surface);
|
||||
border: 1px solid var(--border-color);
|
||||
border-radius: var(--radius-lg);
|
||||
padding: 14px;
|
||||
transition: var(--transition);
|
||||
}
|
||||
|
||||
.stat-card:hover {
|
||||
border-color: var(--border-hover);
|
||||
transform: translateY(-2px);
|
||||
box-shadow: var(--shadow-md);
|
||||
}
|
||||
|
||||
.stat-label {
|
||||
font-size: 12px;
|
||||
font-weight: 500;
|
||||
color: var(--text-secondary);
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.05em;
|
||||
margin-bottom: 8px;
|
||||
}
|
||||
|
||||
.stat-value {
|
||||
font-size: 28px;
|
||||
font-weight: 700;
|
||||
background: linear-gradient(135deg, var(--primary-light), var(--primary));
|
||||
-webkit-background-clip: text;
|
||||
-webkit-text-fill-color: transparent;
|
||||
}
|
||||
|
||||
/* 响应式设计 */
|
||||
@media (max-width: 768px) {
|
||||
.container {
|
||||
flex-direction: column;
|
||||
}
|
||||
|
||||
.sidebar {
|
||||
width: 100%;
|
||||
border-right: none;
|
||||
border-bottom: 1px solid var(--border-color);
|
||||
max-height: 40vh;
|
||||
}
|
||||
|
||||
.main {
|
||||
padding: 16px;
|
||||
}
|
||||
|
||||
.schema-grid {
|
||||
grid-template-columns: 1fr;
|
||||
}
|
||||
|
||||
.manifest-stats {
|
||||
grid-template-columns: repeat(2, 1fr);
|
||||
}
|
||||
}
|
||||
|
||||
/* 列选择器 */
|
||||
.column-selector {
|
||||
margin-bottom: 16px;
|
||||
padding: 14px;
|
||||
background: var(--bg-surface);
|
||||
border: 1px solid var(--border-color);
|
||||
border-radius: var(--radius-lg);
|
||||
}
|
||||
|
||||
.columns-container {
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
gap: 8px;
|
||||
}
|
||||
|
||||
.column-checkbox {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
padding: 8px 14px;
|
||||
background: var(--bg-elevated);
|
||||
border: 1px solid var(--border-color);
|
||||
border-radius: var(--radius-md);
|
||||
cursor: pointer;
|
||||
font-size: 13px;
|
||||
transition: var(--transition);
|
||||
}
|
||||
|
||||
.column-checkbox:hover {
|
||||
border-color: var(--border-hover);
|
||||
}
|
||||
|
||||
.column-checkbox.selected {
|
||||
background: var(--primary-bg);
|
||||
border-color: var(--primary);
|
||||
}
|
||||
|
||||
.column-checkbox input {
|
||||
margin-right: 8px;
|
||||
}
|
||||
|
||||
/* 工具按钮 */
|
||||
.control-buttons {
|
||||
display: flex;
|
||||
gap: 8px;
|
||||
}
|
||||
|
||||
.control-buttons button {
|
||||
padding: 6px 14px;
|
||||
background: var(--bg-surface);
|
||||
border: 1px solid var(--border-color);
|
||||
border-radius: var(--radius-md);
|
||||
color: var(--text-primary);
|
||||
font-size: 12px;
|
||||
font-weight: 500;
|
||||
transition: var(--transition);
|
||||
}
|
||||
|
||||
.control-buttons button:hover {
|
||||
background: var(--bg-hover);
|
||||
border-color: var(--border-hover);
|
||||
}
|
||||
|
||||
/* 单元格大小指示器 */
|
||||
.cell-size {
|
||||
font-size: 11px;
|
||||
color: var(--text-tertiary);
|
||||
font-style: italic;
|
||||
margin-left: 6px;
|
||||
}
|
||||
|
||||
/* 截断指示器 */
|
||||
.truncated-icon {
|
||||
color: var(--warning);
|
||||
margin-left: 6px;
|
||||
}
|
||||
69
webui/static/index.html
Normal file
69
webui/static/index.html
Normal file
@@ -0,0 +1,69 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<title>SRDB Web UI</title>
|
||||
<link rel="preconnect" href="https://fonts.googleapis.com" />
|
||||
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
|
||||
<link
|
||||
href="https://fonts.googleapis.com/css2?family=Inter:wght@300;400;500;600;700&display=swap"
|
||||
rel="stylesheet"
|
||||
/>
|
||||
<link rel="stylesheet" href="/static/css/styles.css" />
|
||||
<script src="https://npm.onmicrosoft.cn/htmx.org@2.0.7/dist/htmx.min.js"></script>
|
||||
</head>
|
||||
<body>
|
||||
<div class="container">
|
||||
<!-- 左侧表列表 -->
|
||||
<div class="sidebar" id="sidebar">
|
||||
<h1>Tables</h1>
|
||||
<div
|
||||
id="table-list"
|
||||
hx-get="/api/tables-html"
|
||||
hx-trigger="load"
|
||||
>
|
||||
Loading tables...
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- 右侧主内容区 -->
|
||||
<div class="main" id="main-content">
|
||||
<div class="empty">
|
||||
<h2>Select a table to view data</h2>
|
||||
<p>Choose a table from the sidebar to get started</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Modal -->
|
||||
<div id="modal" class="modal" style="display: none">
|
||||
<div class="modal-content">
|
||||
<div class="modal-header">
|
||||
<h3 id="modal-title">Content</h3>
|
||||
<button class="modal-close" onclick="closeModal()">
|
||||
<svg
|
||||
xmlns="http://www.w3.org/2000/svg"
|
||||
width="20"
|
||||
height="20"
|
||||
viewBox="0 0 24 24"
|
||||
fill="none"
|
||||
stroke="currentColor"
|
||||
stroke-width="2"
|
||||
stroke-linecap="round"
|
||||
stroke-linejoin="round"
|
||||
>
|
||||
<line x1="18" y1="6" x2="6" y2="18"></line>
|
||||
<line x1="6" y1="6" x2="18" y2="18"></line>
|
||||
</svg>
|
||||
</button>
|
||||
</div>
|
||||
<div class="modal-body">
|
||||
<pre id="modal-body-content"></pre>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<script src="/static/js/app.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
199
webui/static/js/app.js
Normal file
199
webui/static/js/app.js
Normal file
@@ -0,0 +1,199 @@
|
||||
// SRDB WebUI - htmx 版本
|
||||
|
||||
// 全局状态
|
||||
window.srdbState = {
|
||||
selectedTable: null,
|
||||
currentPage: 1,
|
||||
pageSize: 20,
|
||||
selectedColumns: [],
|
||||
expandedTables: new Set(),
|
||||
expandedLevels: new Set([0, 1]),
|
||||
};
|
||||
|
||||
// 选择表格
|
||||
function selectTable(tableName) {
|
||||
window.srdbState.selectedTable = tableName;
|
||||
window.srdbState.currentPage = 1;
|
||||
|
||||
// 高亮选中的表
|
||||
document.querySelectorAll(".table-item").forEach((el) => {
|
||||
el.classList.toggle("selected", el.dataset.table === tableName);
|
||||
});
|
||||
|
||||
// 加载表数据
|
||||
loadTableData(tableName);
|
||||
}
|
||||
|
||||
// 加载表数据
|
||||
function loadTableData(tableName) {
|
||||
const mainContent = document.getElementById("main-content");
|
||||
mainContent.innerHTML = '<div class="loading">Loading...</div>';
|
||||
|
||||
fetch(
|
||||
`/api/tables-view/${tableName}?page=${window.srdbState.currentPage}&pageSize=${window.srdbState.pageSize}`,
|
||||
)
|
||||
.then((res) => res.text())
|
||||
.then((html) => {
|
||||
mainContent.innerHTML = html;
|
||||
})
|
||||
.catch((err) => {
|
||||
console.error("Failed to load table data:", err);
|
||||
mainContent.innerHTML =
|
||||
'<div class="error">Failed to load table data</div>';
|
||||
});
|
||||
}
|
||||
|
||||
// 切换视图 (Data / Manifest)
|
||||
function switchView(tableName, mode) {
|
||||
const mainContent = document.getElementById("main-content");
|
||||
mainContent.innerHTML = '<div class="loading">Loading...</div>';
|
||||
|
||||
const endpoint =
|
||||
mode === "manifest"
|
||||
? `/api/tables-view/${tableName}/manifest`
|
||||
: `/api/tables-view/${tableName}?page=${window.srdbState.currentPage}&pageSize=${window.srdbState.pageSize}`;
|
||||
|
||||
fetch(endpoint)
|
||||
.then((res) => res.text())
|
||||
.then((html) => {
|
||||
mainContent.innerHTML = html;
|
||||
});
|
||||
}
|
||||
|
||||
// 分页
|
||||
function changePage(delta) {
|
||||
window.srdbState.currentPage += delta;
|
||||
if (window.srdbState.selectedTable) {
|
||||
loadTableData(window.srdbState.selectedTable);
|
||||
}
|
||||
}
|
||||
|
||||
function jumpToPage(page) {
|
||||
window.srdbState.currentPage = parseInt(page);
|
||||
if (window.srdbState.selectedTable) {
|
||||
loadTableData(window.srdbState.selectedTable);
|
||||
}
|
||||
}
|
||||
|
||||
function changePageSize(newSize) {
|
||||
window.srdbState.pageSize = parseInt(newSize);
|
||||
window.srdbState.currentPage = 1;
|
||||
if (window.srdbState.selectedTable) {
|
||||
loadTableData(window.srdbState.selectedTable);
|
||||
}
|
||||
}
|
||||
|
||||
// Modal 相关
|
||||
function showModal(title, content) {
|
||||
document.getElementById("modal-title").textContent = title;
|
||||
document.getElementById("modal-body-content").textContent = content;
|
||||
document.getElementById("modal").style.display = "flex";
|
||||
}
|
||||
|
||||
function closeModal() {
|
||||
document.getElementById("modal").style.display = "none";
|
||||
}
|
||||
|
||||
function showCellContent(content) {
|
||||
showModal("Cell Content", content);
|
||||
}
|
||||
|
||||
function showRowDetail(tableName, seq) {
|
||||
fetch(`/api/tables/${tableName}/data/${seq}`)
|
||||
.then((res) => res.json())
|
||||
.then((data) => {
|
||||
const formatted = JSON.stringify(data, null, 2);
|
||||
showModal(`Row Detail (Seq: ${seq})`, formatted);
|
||||
})
|
||||
.catch((err) => {
|
||||
console.error("Failed to load row detail:", err);
|
||||
alert("Failed to load row detail");
|
||||
});
|
||||
}
|
||||
|
||||
// 折叠展开
|
||||
function toggleExpand(tableName) {
|
||||
const item = document.querySelector(`[data-table="${tableName}"]`);
|
||||
const fieldsDiv = item.querySelector(".schema-fields");
|
||||
const icon = item.querySelector(".expand-icon");
|
||||
|
||||
if (window.srdbState.expandedTables.has(tableName)) {
|
||||
window.srdbState.expandedTables.delete(tableName);
|
||||
fieldsDiv.style.display = "none";
|
||||
icon.classList.remove("expanded");
|
||||
} else {
|
||||
window.srdbState.expandedTables.add(tableName);
|
||||
fieldsDiv.style.display = "block";
|
||||
icon.classList.add("expanded");
|
||||
}
|
||||
}
|
||||
|
||||
function toggleLevel(level) {
|
||||
const levelCard = document.querySelector(`[data-level="${level}"]`);
|
||||
const fileList = levelCard.querySelector(".file-list");
|
||||
const icon = levelCard.querySelector(".expand-icon");
|
||||
|
||||
if (window.srdbState.expandedLevels.has(level)) {
|
||||
window.srdbState.expandedLevels.delete(level);
|
||||
fileList.style.display = "none";
|
||||
icon.classList.remove("expanded");
|
||||
} else {
|
||||
window.srdbState.expandedLevels.add(level);
|
||||
fileList.style.display = "grid";
|
||||
icon.classList.add("expanded");
|
||||
}
|
||||
}
|
||||
|
||||
// 格式化工具
|
||||
function formatBytes(bytes) {
|
||||
if (bytes === 0) return "0 B";
|
||||
const k = 1024;
|
||||
const sizes = ["B", "KB", "MB", "GB"];
|
||||
const i = Math.floor(Math.log(bytes) / Math.log(k));
|
||||
return (bytes / Math.pow(k, i)).toFixed(2) + " " + sizes[i];
|
||||
}
|
||||
|
||||
function formatCount(count) {
|
||||
if (count >= 1000000) return (count / 1000000).toFixed(1) + "M";
|
||||
if (count >= 1000) return (count / 1000).toFixed(1) + "K";
|
||||
return count.toString();
|
||||
}
|
||||
|
||||
// 点击 modal 外部关闭
|
||||
document.addEventListener("click", (e) => {
|
||||
const modal = document.getElementById("modal");
|
||||
if (e.target === modal) {
|
||||
closeModal();
|
||||
}
|
||||
});
|
||||
|
||||
// ESC 键关闭 modal
|
||||
document.addEventListener("keydown", (e) => {
|
||||
if (e.key === "Escape") {
|
||||
closeModal();
|
||||
}
|
||||
});
|
||||
|
||||
// 切换列显示
|
||||
function toggleColumn(columnName) {
|
||||
// 切换 schema-field-card 的选中状态
|
||||
const card = document.querySelector(
|
||||
`.schema-field-card[data-column="${columnName}"]`,
|
||||
);
|
||||
if (!card) return;
|
||||
|
||||
card.classList.toggle("selected");
|
||||
const isSelected = card.classList.contains("selected");
|
||||
|
||||
// 切换表格列的显示/隐藏
|
||||
const headers = document.querySelectorAll(`th[data-column="${columnName}"]`);
|
||||
const cells = document.querySelectorAll(`td[data-column="${columnName}"]`);
|
||||
|
||||
headers.forEach((header) => {
|
||||
header.style.display = isSelected ? "" : "none";
|
||||
});
|
||||
|
||||
cells.forEach((cell) => {
|
||||
cell.style.display = isSelected ? "" : "none";
|
||||
});
|
||||
}
|
||||
730
webui/webui.go
Normal file
730
webui/webui.go
Normal file
@@ -0,0 +1,730 @@
|
||||
package webui
|
||||
|
||||
import (
|
||||
"embed"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"code.tczkiot.com/srdb"
|
||||
"code.tczkiot.com/srdb/sst"
|
||||
)
|
||||
|
||||
//go:embed static
|
||||
var staticFS embed.FS
|
||||
|
||||
// WebUI Web 界面处理器
|
||||
type WebUI struct {
|
||||
db *srdb.Database
|
||||
handler http.Handler
|
||||
}
|
||||
|
||||
// NewWebUI 创建 WebUI 实例
|
||||
func NewWebUI(db *srdb.Database) *WebUI {
|
||||
ui := &WebUI{db: db}
|
||||
ui.handler = ui.setupHandler()
|
||||
return ui
|
||||
}
|
||||
|
||||
// setupHandler 设置 HTTP Handler
|
||||
func (ui *WebUI) setupHandler() http.Handler {
|
||||
mux := http.NewServeMux()
|
||||
|
||||
// API endpoints - JSON
|
||||
mux.HandleFunc("/api/tables", ui.handleListTables)
|
||||
mux.HandleFunc("/api/tables/", ui.handleTableAPI)
|
||||
|
||||
// API endpoints - HTML (for htmx)
|
||||
mux.HandleFunc("/api/tables-html", ui.handleTablesHTML)
|
||||
mux.HandleFunc("/api/tables-view/", ui.handleTableViewHTML)
|
||||
|
||||
// Debug endpoint - list embedded files
|
||||
mux.HandleFunc("/debug/files", ui.handleDebugFiles)
|
||||
|
||||
// 静态文件服务
|
||||
staticFiles, _ := fs.Sub(staticFS, "static")
|
||||
mux.Handle("/static/", http.StripPrefix("/static/", http.FileServer(http.FS(staticFiles))))
|
||||
|
||||
// 首页
|
||||
mux.HandleFunc("/", ui.handleIndex)
|
||||
|
||||
return mux
|
||||
}
|
||||
|
||||
// Handler 返回 HTTP Handler
|
||||
func (ui *WebUI) Handler() http.Handler {
|
||||
return ui.handler
|
||||
}
|
||||
|
||||
// ServeHTTP 实现 http.Handler 接口
|
||||
func (ui *WebUI) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||
ui.handler.ServeHTTP(w, r)
|
||||
}
|
||||
|
||||
// handleListTables 处理获取表列表请求
|
||||
func (ui *WebUI) handleListTables(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Method != http.MethodGet {
|
||||
http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
|
||||
return
|
||||
}
|
||||
|
||||
type FieldInfo struct {
|
||||
Name string `json:"name"`
|
||||
Type string `json:"type"`
|
||||
Indexed bool `json:"indexed"`
|
||||
Comment string `json:"comment"`
|
||||
}
|
||||
|
||||
type TableListItem struct {
|
||||
Name string `json:"name"`
|
||||
CreatedAt int64 `json:"created_at"`
|
||||
Fields []FieldInfo `json:"fields"`
|
||||
}
|
||||
|
||||
allTables := ui.db.GetAllTablesInfo()
|
||||
tables := make([]TableListItem, 0, len(allTables))
|
||||
for name, table := range allTables {
|
||||
schema := table.GetSchema()
|
||||
fields := make([]FieldInfo, 0, len(schema.Fields))
|
||||
for _, field := range schema.Fields {
|
||||
fields = append(fields, FieldInfo{
|
||||
Name: field.Name,
|
||||
Type: field.Type.String(),
|
||||
Indexed: field.Indexed,
|
||||
Comment: field.Comment,
|
||||
})
|
||||
}
|
||||
|
||||
tables = append(tables, TableListItem{
|
||||
Name: name,
|
||||
CreatedAt: table.GetCreatedAt(),
|
||||
Fields: fields,
|
||||
})
|
||||
}
|
||||
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
json.NewEncoder(w).Encode(tables)
|
||||
}
|
||||
|
||||
// handleTableAPI 处理表相关的 API 请求
|
||||
func (ui *WebUI) handleTableAPI(w http.ResponseWriter, r *http.Request) {
|
||||
// 解析路径: /api/tables/{name}/schema 或 /api/tables/{name}/data
|
||||
path := strings.TrimPrefix(r.URL.Path, "/api/tables/")
|
||||
parts := strings.Split(path, "/")
|
||||
|
||||
if len(parts) < 2 {
|
||||
http.Error(w, "Invalid path", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
tableName := parts[0]
|
||||
action := parts[1]
|
||||
|
||||
switch action {
|
||||
case "schema":
|
||||
ui.handleTableSchema(w, r, tableName)
|
||||
case "data":
|
||||
// 检查是否是单条数据查询: /api/tables/{name}/data/{seq}
|
||||
if len(parts) >= 3 {
|
||||
ui.handleTableDataBySeq(w, r, tableName, parts[2])
|
||||
} else {
|
||||
ui.handleTableData(w, r, tableName)
|
||||
}
|
||||
case "manifest":
|
||||
ui.handleTableManifest(w, r, tableName)
|
||||
default:
|
||||
http.Error(w, "Unknown action", http.StatusBadRequest)
|
||||
}
|
||||
}
|
||||
|
||||
// handleTableSchema 处理获取表 schema 请求
|
||||
func (ui *WebUI) handleTableSchema(w http.ResponseWriter, r *http.Request, tableName string) {
|
||||
if r.Method != http.MethodGet {
|
||||
http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
|
||||
return
|
||||
}
|
||||
|
||||
table, err := ui.db.GetTable(tableName)
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
|
||||
schema := table.GetSchema()
|
||||
|
||||
type FieldInfo struct {
|
||||
Name string `json:"name"`
|
||||
Type string `json:"type"`
|
||||
Indexed bool `json:"indexed"`
|
||||
Comment string `json:"comment"`
|
||||
}
|
||||
|
||||
fields := make([]FieldInfo, 0, len(schema.Fields))
|
||||
for _, field := range schema.Fields {
|
||||
fields = append(fields, FieldInfo{
|
||||
Name: field.Name,
|
||||
Type: field.Type.String(),
|
||||
Indexed: field.Indexed,
|
||||
Comment: field.Comment,
|
||||
})
|
||||
}
|
||||
|
||||
response := map[string]any{
|
||||
"name": schema.Name,
|
||||
"fields": fields,
|
||||
}
|
||||
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
json.NewEncoder(w).Encode(response)
|
||||
}
|
||||
|
||||
// handleTableManifest 处理获取表 manifest 信息请求
|
||||
func (ui *WebUI) handleTableManifest(w http.ResponseWriter, r *http.Request, tableName string) {
|
||||
if r.Method != http.MethodGet {
|
||||
http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
|
||||
return
|
||||
}
|
||||
|
||||
table, err := ui.db.GetTable(tableName)
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
|
||||
engine := table.GetEngine()
|
||||
versionSet := engine.GetVersionSet()
|
||||
version := versionSet.GetCurrent()
|
||||
|
||||
// 构建每层的信息
|
||||
type FileInfo struct {
|
||||
FileNumber int64 `json:"file_number"`
|
||||
Level int `json:"level"`
|
||||
FileSize int64 `json:"file_size"`
|
||||
MinKey int64 `json:"min_key"`
|
||||
MaxKey int64 `json:"max_key"`
|
||||
RowCount int64 `json:"row_count"`
|
||||
}
|
||||
|
||||
type LevelInfo struct {
|
||||
Level int `json:"level"`
|
||||
FileCount int `json:"file_count"`
|
||||
TotalSize int64 `json:"total_size"`
|
||||
Score float64 `json:"score"`
|
||||
Files []FileInfo `json:"files"`
|
||||
}
|
||||
|
||||
// 获取 Compaction Manager 和 Picker
|
||||
compactionMgr := engine.GetCompactionManager()
|
||||
picker := compactionMgr.GetPicker()
|
||||
|
||||
levels := make([]LevelInfo, 0)
|
||||
for level := 0; level < 7; level++ {
|
||||
files := version.GetLevel(level)
|
||||
if len(files) == 0 {
|
||||
continue
|
||||
}
|
||||
|
||||
totalSize := int64(0)
|
||||
fileInfos := make([]FileInfo, 0, len(files))
|
||||
for _, f := range files {
|
||||
totalSize += f.FileSize
|
||||
fileInfos = append(fileInfos, FileInfo{
|
||||
FileNumber: f.FileNumber,
|
||||
Level: f.Level,
|
||||
FileSize: f.FileSize,
|
||||
MinKey: f.MinKey,
|
||||
MaxKey: f.MaxKey,
|
||||
RowCount: f.RowCount,
|
||||
})
|
||||
}
|
||||
|
||||
score := picker.GetLevelScore(version, level)
|
||||
|
||||
levels = append(levels, LevelInfo{
|
||||
Level: level,
|
||||
FileCount: len(files),
|
||||
TotalSize: totalSize,
|
||||
Score: score,
|
||||
Files: fileInfos,
|
||||
})
|
||||
}
|
||||
|
||||
// 获取 Compaction 统计
|
||||
stats := compactionMgr.GetStats()
|
||||
|
||||
response := map[string]any{
|
||||
"levels": levels,
|
||||
"next_file_number": versionSet.GetNextFileNumber(),
|
||||
"last_sequence": versionSet.GetLastSequence(),
|
||||
"compaction_stats": stats,
|
||||
}
|
||||
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
json.NewEncoder(w).Encode(response)
|
||||
}
|
||||
|
||||
// handleTableDataBySeq 处理获取单条数据请求
|
||||
func (ui *WebUI) handleTableDataBySeq(w http.ResponseWriter, r *http.Request, tableName string, seqStr string) {
|
||||
if r.Method != http.MethodGet {
|
||||
http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
|
||||
return
|
||||
}
|
||||
|
||||
table, err := ui.db.GetTable(tableName)
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
|
||||
// 解析 seq
|
||||
seq, err := strconv.ParseInt(seqStr, 10, 64)
|
||||
if err != nil {
|
||||
http.Error(w, "Invalid seq parameter", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
// 获取数据
|
||||
row, err := table.Get(seq)
|
||||
if err != nil {
|
||||
http.Error(w, fmt.Sprintf("Row not found: %v", err), http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
|
||||
// 构造响应（不进行剪裁，返回完整数据）
|
||||
rowData := make(map[string]interface{})
|
||||
rowData["_seq"] = row.Seq
|
||||
rowData["_time"] = row.Time
|
||||
for k, v := range row.Data {
|
||||
rowData[k] = v
|
||||
}
|
||||
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
json.NewEncoder(w).Encode(rowData)
|
||||
}
|
||||
|
||||
// handleTableData 处理获取表数据请求（分页）
|
||||
func (ui *WebUI) handleTableData(w http.ResponseWriter, r *http.Request, tableName string) {
|
||||
if r.Method != http.MethodGet {
|
||||
http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
|
||||
return
|
||||
}
|
||||
|
||||
table, err := ui.db.GetTable(tableName)
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
|
||||
// 解析分页参数
|
||||
pageStr := r.URL.Query().Get("page")
|
||||
pageSizeStr := r.URL.Query().Get("pageSize")
|
||||
selectParam := r.URL.Query().Get("select") // 要选择的字段，逗号分隔
|
||||
|
||||
page := 1
|
||||
pageSize := 20
|
||||
|
||||
if pageStr != "" {
|
||||
if p, err := strconv.Atoi(pageStr); err == nil && p > 0 {
|
||||
page = p
|
||||
}
|
||||
}
|
||||
|
||||
if pageSizeStr != "" {
|
||||
if ps, err := strconv.Atoi(pageSizeStr); err == nil && ps > 0 && ps <= 1000 {
|
||||
pageSize = ps
|
||||
}
|
||||
}
|
||||
|
||||
// 解析要选择的字段
|
||||
var selectedFields []string
|
||||
if selectParam != "" {
|
||||
selectedFields = strings.Split(selectParam, ",")
|
||||
}
|
||||
|
||||
// 获取 schema 用于字段类型判断
|
||||
tableSchema := table.GetSchema()
|
||||
|
||||
// 使用 Query API 获取所有数据（高效）
|
||||
queryRows, err := table.Query().Rows()
|
||||
if err != nil {
|
||||
http.Error(w, fmt.Sprintf("Failed to query table: %v", err), http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
defer queryRows.Close()
|
||||
|
||||
// 收集所有 rows 到内存中用于分页（对于大数据集，后续可以优化为流式处理）
|
||||
allRows := make([]*sst.Row, 0)
|
||||
for queryRows.Next() {
|
||||
row := queryRows.Row()
|
||||
// Row 是 query.Row 类型，需要获取其内部的 sst.Row
|
||||
// 直接构造 sst.Row
|
||||
sstRow := &sst.Row{
|
||||
Seq: row.Data()["_seq"].(int64),
|
||||
Time: row.Data()["_time"].(int64),
|
||||
Data: make(map[string]any),
|
||||
}
|
||||
// 复制其他字段
|
||||
for k, v := range row.Data() {
|
||||
if k != "_seq" && k != "_time" {
|
||||
sstRow.Data[k] = v
|
||||
}
|
||||
}
|
||||
allRows = append(allRows, sstRow)
|
||||
}
|
||||
|
||||
// 计算分页
|
||||
totalRows := int64(len(allRows))
|
||||
offset := (page - 1) * pageSize
|
||||
end := offset + pageSize
|
||||
if end > int(totalRows) {
|
||||
end = int(totalRows)
|
||||
}
|
||||
|
||||
// 获取当前页数据
|
||||
rows := make([]*sst.Row, 0, pageSize)
|
||||
if offset < int(totalRows) {
|
||||
rows = allRows[offset:end]
|
||||
}
|
||||
|
||||
// 构造响应，对 string 字段进行剪裁
|
||||
const maxStringLength = 100 // 最大字符串长度（按字符计数，非字节）
|
||||
data := make([]map[string]any, 0, len(rows))
|
||||
for _, row := range rows {
|
||||
rowData := make(map[string]any)
|
||||
|
||||
// 如果指定了字段，只返回选定的字段
|
||||
if len(selectedFields) > 0 {
|
||||
for _, field := range selectedFields {
|
||||
field = strings.TrimSpace(field)
|
||||
if field == "_seq" {
|
||||
rowData["_seq"] = row.Seq
|
||||
} else if field == "_time" {
|
||||
rowData["_time"] = row.Time
|
||||
} else if v, ok := row.Data[field]; ok {
|
||||
// 检查字段类型
|
||||
fieldDef, err := tableSchema.GetField(field)
|
||||
if err == nil && fieldDef.Type == srdb.FieldTypeString {
|
||||
// 对字符串字段进行剪裁
|
||||
if str, ok := v.(string); ok {
|
||||
runes := []rune(str)
|
||||
if len(runes) > maxStringLength {
|
||||
rowData[field] = string(runes[:maxStringLength]) + "..."
|
||||
rowData[field+"_truncated"] = true
|
||||
continue
|
||||
}
|
||||
}
|
||||
}
|
||||
rowData[field] = v
|
||||
}
|
||||
}
|
||||
} else {
|
||||
// 返回所有字段
|
||||
rowData["_seq"] = row.Seq
|
||||
rowData["_time"] = row.Time
|
||||
for k, v := range row.Data {
|
||||
// 检查字段类型
|
||||
field, err := tableSchema.GetField(k)
|
||||
if err == nil && field.Type == srdb.FieldTypeString {
|
||||
// 对字符串字段进行剪裁（按 rune 截取，避免 CJK 等多字节字符乱码）
|
||||
if str, ok := v.(string); ok {
|
||||
runes := []rune(str)
|
||||
if len(runes) > maxStringLength {
|
||||
rowData[k] = string(runes[:maxStringLength]) + "..."
|
||||
rowData[k+"_truncated"] = true
|
||||
continue
|
||||
}
|
||||
}
|
||||
}
|
||||
rowData[k] = v
|
||||
}
|
||||
}
|
||||
data = append(data, rowData)
|
||||
}
|
||||
|
||||
response := map[string]interface{}{
|
||||
"data": data,
|
||||
"page": page,
|
||||
"pageSize": pageSize,
|
||||
"totalRows": totalRows,
|
||||
"totalPages": (totalRows + int64(pageSize) - 1) / int64(pageSize),
|
||||
}
|
||||
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
json.NewEncoder(w).Encode(response)
|
||||
}
|
||||
|
||||
// handleDebugFiles 列出所有嵌入的文件（调试用）
|
||||
func (ui *WebUI) handleDebugFiles(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Type", "text/plain; charset=utf-8")
|
||||
fmt.Fprintln(w, "Embedded files in staticFS:")
|
||||
fs.WalkDir(staticFS, ".", func(path string, d fs.DirEntry, err error) error {
|
||||
if err != nil {
|
||||
fmt.Fprintf(w, "ERROR walking %s: %v\n", path, err)
|
||||
return err
|
||||
}
|
||||
if d.IsDir() {
|
||||
fmt.Fprintf(w, "[DIR] %s/\n", path)
|
||||
} else {
|
||||
info, _ := d.Info()
|
||||
fmt.Fprintf(w, "[FILE] %s (%d bytes)\n", path, info.Size())
|
||||
}
|
||||
return nil
|
||||
})
|
||||
}
|
||||
|
||||
// handleIndex 处理首页请求
|
||||
func (ui *WebUI) handleIndex(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path != "/" {
|
||||
http.NotFound(w, r)
|
||||
return
|
||||
}
|
||||
|
||||
// 读取 index.html
|
||||
content, err := staticFS.ReadFile("static/index.html")
|
||||
if err != nil {
|
||||
http.Error(w, "Failed to load page", http.StatusInternalServerError)
|
||||
fmt.Fprintf(w, "Error: %v", err)
|
||||
return
|
||||
}
|
||||
|
||||
w.Header().Set("Content-Type", "text/html; charset=utf-8")
|
||||
w.Write(content)
|
||||
}
|
||||
|
||||
// handleTablesHTML 处理获取表列表 HTML 请求（for htmx）
|
||||
func (ui *WebUI) handleTablesHTML(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Method != http.MethodGet {
|
||||
http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
|
||||
return
|
||||
}
|
||||
|
||||
allTables := ui.db.GetAllTablesInfo()
|
||||
tables := make([]TableListItem, 0, len(allTables))
|
||||
for name, table := range allTables {
|
||||
schema := table.GetSchema()
|
||||
fields := make([]FieldInfo, 0, len(schema.Fields))
|
||||
for _, field := range schema.Fields {
|
||||
fields = append(fields, FieldInfo{
|
||||
Name: field.Name,
|
||||
Type: field.Type.String(),
|
||||
Indexed: field.Indexed,
|
||||
Comment: field.Comment,
|
||||
})
|
||||
}
|
||||
|
||||
tables = append(tables, TableListItem{
|
||||
Name: name,
|
||||
CreatedAt: table.GetCreatedAt(),
|
||||
Fields: fields,
|
||||
})
|
||||
}
|
||||
|
||||
html := renderTablesHTML(tables)
|
||||
w.Header().Set("Content-Type", "text/html; charset=utf-8")
|
||||
w.Write([]byte(html))
|
||||
}
|
||||
|
||||
// handleTableViewHTML 处理获取表视图 HTML 请求（for htmx）
|
||||
func (ui *WebUI) handleTableViewHTML(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Method != http.MethodGet {
|
||||
http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
|
||||
return
|
||||
}
|
||||
|
||||
// 解析路径: /api/tables-view/{name} 或 /api/tables-view/{name}/manifest
|
||||
path := strings.TrimPrefix(r.URL.Path, "/api/tables-view/")
|
||||
parts := strings.Split(path, "/")
|
||||
|
||||
if len(parts) < 1 || parts[0] == "" {
|
||||
http.Error(w, "Invalid path", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
tableName := parts[0]
|
||||
isManifest := len(parts) >= 2 && parts[1] == "manifest"
|
||||
|
||||
table, err := ui.db.GetTable(tableName)
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
|
||||
if isManifest {
|
||||
// 返回 Manifest 视图 HTML
|
||||
ui.renderManifestHTML(w, r, tableName, table)
|
||||
} else {
|
||||
// 返回 Data 视图 HTML
|
||||
ui.renderDataHTML(w, r, tableName, table)
|
||||
}
|
||||
}
|
||||
|
||||
// renderDataHTML 渲染数据视图 HTML
|
||||
func (ui *WebUI) renderDataHTML(w http.ResponseWriter, r *http.Request, tableName string, table *srdb.Table) {
|
||||
// 解析分页参数
|
||||
pageStr := r.URL.Query().Get("page")
|
||||
pageSizeStr := r.URL.Query().Get("pageSize")
|
||||
|
||||
page := 1
|
||||
pageSize := 20
|
||||
|
||||
if pageStr != "" {
|
||||
if p, err := strconv.Atoi(pageStr); err == nil && p > 0 {
|
||||
page = p
|
||||
}
|
||||
}
|
||||
|
||||
if pageSizeStr != "" {
|
||||
if ps, err := strconv.Atoi(pageSizeStr); err == nil && ps > 0 && ps <= 1000 {
|
||||
pageSize = ps
|
||||
}
|
||||
}
|
||||
|
||||
// 获取 schema
|
||||
tableSchema := table.GetSchema()
|
||||
schemaInfo := SchemaInfo{
|
||||
Name: tableSchema.Name,
|
||||
Fields: make([]FieldInfo, 0, len(tableSchema.Fields)),
|
||||
}
|
||||
for _, field := range tableSchema.Fields {
|
||||
schemaInfo.Fields = append(schemaInfo.Fields, FieldInfo{
|
||||
Name: field.Name,
|
||||
Type: field.Type.String(),
|
||||
Indexed: field.Indexed,
|
||||
Comment: field.Comment,
|
||||
})
|
||||
}
|
||||
|
||||
// 使用 Query API 获取所有数据
|
||||
queryRows, err := table.Query().Rows()
|
||||
if err != nil {
|
||||
http.Error(w, fmt.Sprintf("Failed to query table: %v", err), http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
defer queryRows.Close()
|
||||
|
||||
// 收集所有 rows
|
||||
allRows := make([]*sst.Row, 0)
|
||||
for queryRows.Next() {
|
||||
row := queryRows.Row()
|
||||
sstRow := &sst.Row{
|
||||
Seq: row.Data()["_seq"].(int64),
|
||||
Time: row.Data()["_time"].(int64),
|
||||
Data: make(map[string]any),
|
||||
}
|
||||
for k, v := range row.Data() {
|
||||
if k != "_seq" && k != "_time" {
|
||||
sstRow.Data[k] = v
|
||||
}
|
||||
}
|
||||
allRows = append(allRows, sstRow)
|
||||
}
|
||||
|
||||
// 计算分页
|
||||
totalRows := int64(len(allRows))
|
||||
offset := (page - 1) * pageSize
|
||||
end := offset + pageSize
|
||||
if end > int(totalRows) {
|
||||
end = int(totalRows)
|
||||
}
|
||||
|
||||
// 获取当前页数据
|
||||
rows := make([]*sst.Row, 0, pageSize)
|
||||
if offset < int(totalRows) {
|
||||
rows = allRows[offset:end]
|
||||
}
|
||||
|
||||
// 构造 TableDataResponse
|
||||
const maxStringLength = 100
|
||||
data := make([]map[string]any, 0, len(rows))
|
||||
for _, row := range rows {
|
||||
rowData := make(map[string]any)
|
||||
rowData["_seq"] = row.Seq
|
||||
rowData["_time"] = row.Time
|
||||
for k, v := range row.Data {
|
||||
field, err := tableSchema.GetField(k)
|
||||
if err == nil && field.Type == srdb.FieldTypeString {
|
||||
if str, ok := v.(string); ok {
|
||||
runes := []rune(str)
|
||||
if len(runes) > maxStringLength {
|
||||
rowData[k] = string(runes[:maxStringLength]) + "..."
|
||||
rowData[k+"_truncated"] = true
|
||||
continue
|
||||
}
|
||||
}
|
||||
}
|
||||
rowData[k] = v
|
||||
}
|
||||
data = append(data, rowData)
|
||||
}
|
||||
|
||||
tableData := TableDataResponse{
|
||||
Data: data,
|
||||
Page: int64(page),
|
||||
PageSize: int64(pageSize),
|
||||
TotalRows: totalRows,
|
||||
TotalPages: (totalRows + int64(pageSize) - 1) / int64(pageSize),
|
||||
}
|
||||
|
||||
html := renderDataViewHTML(tableName, schemaInfo, tableData)
|
||||
w.Header().Set("Content-Type", "text/html; charset=utf-8")
|
||||
w.Write([]byte(html))
|
||||
}
|
||||
|
||||
// renderManifestHTML 渲染 Manifest 视图 HTML
|
||||
func (ui *WebUI) renderManifestHTML(w http.ResponseWriter, r *http.Request, tableName string, table *srdb.Table) {
|
||||
engine := table.GetEngine()
|
||||
versionSet := engine.GetVersionSet()
|
||||
version := versionSet.GetCurrent()
|
||||
|
||||
// 获取 Compaction Manager 和 Picker
|
||||
compactionMgr := engine.GetCompactionManager()
|
||||
picker := compactionMgr.GetPicker()
|
||||
|
||||
levels := make([]LevelInfo, 0)
|
||||
for level := 0; level < 7; level++ {
|
||||
files := version.GetLevel(level)
|
||||
|
||||
totalSize := int64(0)
|
||||
fileInfos := make([]FileInfo, 0, len(files))
|
||||
for _, f := range files {
|
||||
totalSize += f.FileSize
|
||||
fileInfos = append(fileInfos, FileInfo{
|
||||
FileNumber: f.FileNumber,
|
||||
Level: f.Level,
|
||||
FileSize: f.FileSize,
|
||||
MinKey: f.MinKey,
|
||||
MaxKey: f.MaxKey,
|
||||
RowCount: f.RowCount,
|
||||
})
|
||||
}
|
||||
|
||||
score := 0.0
|
||||
if len(files) > 0 {
|
||||
score = picker.GetLevelScore(version, level)
|
||||
}
|
||||
|
||||
levels = append(levels, LevelInfo{
|
||||
Level: level,
|
||||
FileCount: len(files),
|
||||
TotalSize: totalSize,
|
||||
Score: score,
|
||||
Files: fileInfos,
|
||||
})
|
||||
}
|
||||
|
||||
stats := compactionMgr.GetStats()
|
||||
|
||||
manifest := ManifestResponse{
|
||||
Levels: levels,
|
||||
NextFileNumber: versionSet.GetNextFileNumber(),
|
||||
LastSequence: versionSet.GetLastSequence(),
|
||||
CompactionStats: stats,
|
||||
}
|
||||
|
||||
html := renderManifestViewHTML(tableName, manifest)
|
||||
w.Header().Set("Content-Type", "text/html; charset=utf-8")
|
||||
w.Write([]byte(html))
|
||||
}
|
||||
Reference in New Issue
Block a user