Initial commit: SRDB - High-performance LSM-Tree database
- Core engine with MemTable, SST, WAL - B+Tree indexing for SST files - Leveled compaction strategy - Multi-table database management - Schema validation and secondary indexes - Query builder with complex conditions - Web UI with HTMX for data visualization - Command-line tools for diagnostics
This commit is contained in:
254
examples/webui/README.md
Normal file
254
examples/webui/README.md
Normal file
@@ -0,0 +1,254 @@
|
||||
# SRDB Web UI Example
|
||||
|
||||
这个示例展示了如何使用 SRDB 的内置 Web UI 来可视化查看数据库中的表和数据。
|
||||
|
||||
## 功能特性
|
||||
|
||||
- 📊 **表列表展示** - 左侧显示所有表及其行数
|
||||
- 🔍 **Schema 查看** - 点击箭头展开查看表的字段定义
|
||||
- 📋 **数据分页浏览** - 右侧以表格形式展示数据,支持分页
|
||||
- 🎨 **响应式设计** - 现代化的界面设计
|
||||
- ⚡ **零构建** - 使用 HTMX 从 CDN 加载,无需构建步骤
|
||||
- 💾 **大数据优化** - 自动截断显示,悬停查看,点击弹窗查看完整内容
|
||||
- 📏 **数据大小显示** - 超过 1KB 的单元格自动显示大小标签
|
||||
- 🔄 **后台数据插入** - 自动生成 2KB~512KB 的测试数据(每秒一条)
|
||||
|
||||
## 运行示例
|
||||
|
||||
```bash
|
||||
# 进入示例目录
|
||||
cd examples/webui
|
||||
|
||||
# 运行
|
||||
go run main.go
|
||||
```
|
||||
|
||||
程序会:
|
||||
1. 创建/打开数据库目录 `./data`
|
||||
2. 创建三个示例表:`users`、`products` 和 `logs`
|
||||
3. 插入初始示例数据
|
||||
4. **启动后台协程** - 每秒向 `logs` 表插入一条 2KB~512KB 的随机数据
|
||||
5. 启动 Web 服务器在 `http://localhost:8080`
|
||||
|
||||
## 使用界面
|
||||
|
||||
打开浏览器访问 `http://localhost:8080`,你将看到:
|
||||
|
||||
### 左侧边栏
|
||||
- 显示所有表的列表
|
||||
- 显示每个表的字段数量
|
||||
- 点击 ▶ 图标展开查看字段信息
|
||||
- 点击表名选择要查看的表(蓝色高亮显示当前选中)
|
||||
|
||||
### 右侧主区域
|
||||
- **Schema 区域**:显示表结构和字段定义
|
||||
- **Data 区域**:以表格形式显示数据
|
||||
- 支持分页浏览(每页 20 条)
|
||||
- 显示系统字段(_seq, _time)和用户字段
|
||||
- **自动截断长数据**:超过 400px 的内容显示省略号
|
||||
- **鼠标悬停**:悬停在单元格上查看完整内容
|
||||
- **点击查看**:点击单元格在弹窗中查看完整内容
|
||||
- **大小指示**:超过 1KB 的数据显示大小标签
|
||||
|
||||
### 大数据查看
|
||||
1. **表格截断**:单元格最大宽度 400px,超长显示 `...`
|
||||
2. **悬停展开**:鼠标悬停自动展开,黄色背景高亮
|
||||
3. **模态框**:点击单元格弹出窗口
|
||||
- 等宽字体显示(适合查看十六进制数据)
|
||||
- 显示数据大小
|
||||
- 支持滚动查看超长内容
|
||||
|
||||
## API 端点
|
||||
|
||||
Web UI 提供了以下 HTTP API:
|
||||
|
||||
### 获取所有表
|
||||
```
|
||||
GET /api/tables
|
||||
```
|
||||
|
||||
返回示例:
|
||||
```json
|
||||
[
|
||||
{
|
||||
"name": "users",
|
||||
"rowCount": 5,
|
||||
"dir": "./data/users"
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
### 获取表的 Schema
|
||||
```
|
||||
GET /api/tables/{name}/schema
|
||||
```
|
||||
|
||||
返回示例:
|
||||
```json
|
||||
{
|
||||
"fields": [
|
||||
{"name": "name", "type": "string", "required": true},
|
||||
{"name": "email", "type": "string", "required": true},
|
||||
{"name": "age", "type": "int", "required": false}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### 获取表数据(分页)
|
||||
```
|
||||
GET /api/tables/{name}/data?page=1&pageSize=20
|
||||
```
|
||||
|
||||
参数:
|
||||
- `page` - 页码,从 1 开始(默认:1)
|
||||
- `pageSize` - 每页行数,最大 100(默认:20)
|
||||
|
||||
返回示例:
|
||||
```json
|
||||
{
|
||||
"page": 1,
|
||||
"pageSize": 20,
|
||||
"totalRows": 5,
|
||||
"totalPages": 1,
|
||||
"rows": [
|
||||
{
|
||||
"_seq": 1,
|
||||
"_time": 1234567890,
|
||||
"name": "Alice",
|
||||
"email": "alice@example.com",
|
||||
"age": 30
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### 获取表基本信息
|
||||
```
|
||||
GET /api/tables/{name}
|
||||
```
|
||||
|
||||
## 在你的应用中使用
|
||||
|
||||
你可以在自己的应用中轻松集成 Web UI:
|
||||
|
||||
```go
|
||||
package main
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"code.tczkiot.com/srdb"
|
||||
)
|
||||
|
||||
func main() {
|
||||
// 打开数据库
|
||||
db, _ := database.Open("./mydb")
|
||||
defer db.Close()
|
||||
|
||||
// 获取 HTTP Handler
|
||||
handler := db.WebUI()
|
||||
|
||||
// 启动服务器
|
||||
http.ListenAndServe(":8080", handler)
|
||||
}
|
||||
```
|
||||
|
||||
或者将其作为现有 Web 应用的一部分:
|
||||
|
||||
```go
|
||||
mux := http.NewServeMux()
|
||||
|
||||
// 你的其他路由
|
||||
mux.HandleFunc("/api/myapp", myHandler)
|
||||
|
||||
// 挂载 SRDB Web UI 到 /admin/db 路径
|
||||
mux.Handle("/admin/db/", http.StripPrefix("/admin/db", db.WebUI()))
|
||||
|
||||
http.ListenAndServe(":8080", mux)
|
||||
```
|
||||
|
||||
## 技术栈
|
||||
|
||||
- **后端**: Go + 标准库 `net/http`
|
||||
- **前端**: [HTMX](https://htmx.org/) + 原生 JavaScript + CSS
|
||||
- **渲染**: 服务端 HTML 渲染(Go 模板生成)
|
||||
- **字体**: Google Fonts (Inter)
|
||||
- **无构建**: 直接从 CDN 加载 HTMX,无需 npm、webpack 等工具
|
||||
- **部署**: 所有静态资源通过 `embed.FS` 嵌入到二进制文件中
|
||||
|
||||
## 测试大数据
|
||||
|
||||
### logs 表自动生成
|
||||
|
||||
程序会在后台持续向 `logs` 表插入大数据:
|
||||
|
||||
- **频率**:每秒一条
|
||||
- **大小**:2KB ~ 512KB 随机
|
||||
- **格式**:十六进制字符串
|
||||
- **字段**:
|
||||
- `timestamp` - 插入时间
|
||||
- `data` - 随机数据(十六进制)
|
||||
- `size_bytes` - 数据大小(字节)
|
||||
|
||||
你可以选择 `logs` 表来测试大数据的显示效果:
|
||||
1. 单元格会显示数据大小标签(如 `245.12 KB`)
|
||||
2. 内容被自动截断,显示省略号
|
||||
3. 点击单元格在弹窗中查看完整数据
|
||||
|
||||
终端会实时输出插入日志:
|
||||
```
|
||||
Inserted record #1, size: 245.12 KB
|
||||
Inserted record #2, size: 128.50 KB
|
||||
Inserted record #3, size: 487.23 KB
|
||||
```
|
||||
|
||||
## 注意事项
|
||||
|
||||
- Web UI 是只读的,不提供数据修改功能
|
||||
- 适合用于开发、调试和数据查看
|
||||
- 生产环境建议添加身份验证和访问控制
|
||||
- 大数据量表的分页查询性能取决于数据分布
|
||||
- `logs` 表会持续增长,可手动删除 `./data/logs` 目录重置
|
||||
|
||||
## Compaction 状态
|
||||
|
||||
由于后台持续插入大数据,会产生大量 SST 文件。SRDB 会自动运行 compaction 合并这些文件。
|
||||
|
||||
### 检查 Compaction 状态
|
||||
|
||||
```bash
|
||||
# 查看 SST 文件分布
|
||||
./check_sst.sh
|
||||
|
||||
# 观察 webui 日志中的 [Compaction] 信息
|
||||
```
|
||||
|
||||
### Compaction 改进
|
||||
|
||||
- **触发阈值**: L0 文件数量 ≥2 就触发(之前是 4)
|
||||
- **运行频率**: 每 10 秒自动检查
|
||||
- **日志增强**: 显示详细的 compaction 状态和统计
|
||||
|
||||
详细说明请查看 [COMPACTION.md](./COMPACTION.md)
|
||||
|
||||
## 常见问题
|
||||
|
||||
### `invalid header` 错误
|
||||
|
||||
如果看到类似错误:
|
||||
```
|
||||
failed to open table logs: invalid header
|
||||
```
|
||||
|
||||
**快速修复**:
|
||||
```bash
|
||||
./fix_corrupted_table.sh logs
|
||||
```
|
||||
|
||||
详见:[QUICK_FIX.md](./QUICK_FIX.md) 或 [TROUBLESHOOTING.md](./TROUBLESHOOTING.md)
|
||||
|
||||
## 更多信息
|
||||
|
||||
- [FEATURES.md](./FEATURES.md) - 详细功能说明
|
||||
- [COMPACTION.md](./COMPACTION.md) - Compaction 机制和诊断
|
||||
- [TROUBLESHOOTING.md](./TROUBLESHOOTING.md) - 故障排除指南
|
||||
- [QUICK_FIX.md](./QUICK_FIX.md) - 快速修复常见错误
|
||||
40
examples/webui/commands/check_data.go
Normal file
40
examples/webui/commands/check_data.go
Normal file
@@ -0,0 +1,40 @@
|
||||
package commands
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"log"
|
||||
|
||||
"code.tczkiot.com/srdb"
|
||||
)
|
||||
|
||||
// CheckData 检查数据库中的数据
|
||||
func CheckData(dbPath string) {
|
||||
// 打开数据库
|
||||
db, err := srdb.Open(dbPath)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
defer db.Close()
|
||||
|
||||
// 列出所有表
|
||||
tables := db.ListTables()
|
||||
fmt.Printf("Found %d tables: %v\n", len(tables), tables)
|
||||
|
||||
// 检查每个表的记录数
|
||||
for _, tableName := range tables {
|
||||
table, err := db.GetTable(tableName)
|
||||
if err != nil {
|
||||
fmt.Printf("Error getting table %s: %v\n", tableName, err)
|
||||
continue
|
||||
}
|
||||
|
||||
result, err := table.Query().Rows()
|
||||
if err != nil {
|
||||
fmt.Printf("Error querying table %s: %v\n", tableName, err)
|
||||
continue
|
||||
}
|
||||
|
||||
count := result.Count()
|
||||
fmt.Printf("Table '%s': %d rows\n", tableName, count)
|
||||
}
|
||||
}
|
||||
69
examples/webui/commands/check_seq.go
Normal file
69
examples/webui/commands/check_seq.go
Normal file
@@ -0,0 +1,69 @@
|
||||
package commands
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"log"
|
||||
|
||||
"code.tczkiot.com/srdb"
|
||||
)
|
||||
|
||||
// CheckSeq 检查特定序列号的数据
|
||||
func CheckSeq(dbPath string) {
|
||||
db, err := srdb.Open(dbPath)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
defer db.Close()
|
||||
|
||||
table, err := db.GetTable("logs")
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
// Check seq 1
|
||||
row1, err := table.Get(1)
|
||||
if err != nil {
|
||||
fmt.Printf("Error getting seq=1: %v\n", err)
|
||||
} else if row1 == nil {
|
||||
fmt.Println("Seq=1: NOT FOUND")
|
||||
} else {
|
||||
fmt.Printf("Seq=1: FOUND (time=%d)\n", row1.Time)
|
||||
}
|
||||
|
||||
// Check seq 100
|
||||
row100, err := table.Get(100)
|
||||
if err != nil {
|
||||
fmt.Printf("Error getting seq=100: %v\n", err)
|
||||
} else if row100 == nil {
|
||||
fmt.Println("Seq=100: NOT FOUND")
|
||||
} else {
|
||||
fmt.Printf("Seq=100: FOUND (time=%d)\n", row100.Time)
|
||||
}
|
||||
|
||||
// Check seq 729
|
||||
row729, err := table.Get(729)
|
||||
if err != nil {
|
||||
fmt.Printf("Error getting seq=729: %v\n", err)
|
||||
} else if row729 == nil {
|
||||
fmt.Println("Seq=729: NOT FOUND")
|
||||
} else {
|
||||
fmt.Printf("Seq=729: FOUND (time=%d)\n", row729.Time)
|
||||
}
|
||||
|
||||
// Query all records
|
||||
result, err := table.Query().Rows()
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
count := result.Count()
|
||||
fmt.Printf("\nTotal rows from Query: %d\n", count)
|
||||
|
||||
if count > 0 {
|
||||
first, _ := result.First()
|
||||
if first != nil {
|
||||
data := first.Data()
|
||||
fmt.Printf("First row _seq: %v\n", data["_seq"])
|
||||
}
|
||||
}
|
||||
}
|
||||
58
examples/webui/commands/dump_manifest.go
Normal file
58
examples/webui/commands/dump_manifest.go
Normal file
@@ -0,0 +1,58 @@
|
||||
package commands
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"log"
|
||||
|
||||
"code.tczkiot.com/srdb"
|
||||
)
|
||||
|
||||
// DumpManifest 导出 manifest 信息
|
||||
func DumpManifest(dbPath string) {
|
||||
db, err := srdb.Open(dbPath)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
defer db.Close()
|
||||
|
||||
table, err := db.GetTable("logs")
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
engine := table.GetEngine()
|
||||
versionSet := engine.GetVersionSet()
|
||||
version := versionSet.GetCurrent()
|
||||
|
||||
// Check for duplicates in each level
|
||||
for level := 0; level < 7; level++ {
|
||||
files := version.GetLevel(level)
|
||||
if len(files) == 0 {
|
||||
continue
|
||||
}
|
||||
|
||||
// Track file numbers
|
||||
fileMap := make(map[int64][]struct {
|
||||
minKey int64
|
||||
maxKey int64
|
||||
})
|
||||
|
||||
for _, f := range files {
|
||||
fileMap[f.FileNumber] = append(fileMap[f.FileNumber], struct {
|
||||
minKey int64
|
||||
maxKey int64
|
||||
}{f.MinKey, f.MaxKey})
|
||||
}
|
||||
|
||||
// Report duplicates
|
||||
fmt.Printf("Level %d: %d files\n", level, len(files))
|
||||
for fileNum, entries := range fileMap {
|
||||
if len(entries) > 1 {
|
||||
fmt.Printf(" [DUPLICATE] File #%d appears %d times:\n", fileNum, len(entries))
|
||||
for i, e := range entries {
|
||||
fmt.Printf(" Entry %d: min=%d max=%d\n", i+1, e.minKey, e.maxKey)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
72
examples/webui/commands/inspect_all_sst.go
Normal file
72
examples/webui/commands/inspect_all_sst.go
Normal file
@@ -0,0 +1,72 @@
|
||||
package commands
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"log"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"code.tczkiot.com/srdb/sst"
|
||||
)
|
||||
|
||||
// InspectAllSST 检查所有 SST 文件
|
||||
func InspectAllSST(sstDir string) {
|
||||
// List all SST files
|
||||
files, err := os.ReadDir(sstDir)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
var sstFiles []string
|
||||
for _, file := range files {
|
||||
if strings.HasSuffix(file.Name(), ".sst") {
|
||||
sstFiles = append(sstFiles, file.Name())
|
||||
}
|
||||
}
|
||||
|
||||
sort.Strings(sstFiles)
|
||||
|
||||
fmt.Printf("Found %d SST files\n\n", len(sstFiles))
|
||||
|
||||
// Inspect each file
|
||||
for _, filename := range sstFiles {
|
||||
sstPath := filepath.Join(sstDir, filename)
|
||||
|
||||
reader, err := sst.NewReader(sstPath)
|
||||
if err != nil {
|
||||
fmt.Printf("%s: ERROR - %v\n", filename, err)
|
||||
continue
|
||||
}
|
||||
|
||||
header := reader.GetHeader()
|
||||
allKeys := reader.GetAllKeys()
|
||||
|
||||
// Extract file number
|
||||
numStr := strings.TrimPrefix(filename, "000")
|
||||
numStr = strings.TrimPrefix(numStr, "00")
|
||||
numStr = strings.TrimPrefix(numStr, "0")
|
||||
numStr = strings.TrimSuffix(numStr, ".sst")
|
||||
fileNum, _ := strconv.Atoi(numStr)
|
||||
|
||||
fmt.Printf("File #%d (%s):\n", fileNum, filename)
|
||||
fmt.Printf(" Header: MinKey=%d MaxKey=%d RowCount=%d\n", header.MinKey, header.MaxKey, header.RowCount)
|
||||
fmt.Printf(" Actual: %d keys", len(allKeys))
|
||||
if len(allKeys) > 0 {
|
||||
fmt.Printf(" [%d ... %d]", allKeys[0], allKeys[len(allKeys)-1])
|
||||
}
|
||||
fmt.Printf("\n")
|
||||
|
||||
// Check if header matches actual keys
|
||||
if len(allKeys) > 0 {
|
||||
if header.MinKey != allKeys[0] || header.MaxKey != allKeys[len(allKeys)-1] {
|
||||
fmt.Printf(" *** MISMATCH: Header says %d-%d but file has %d-%d ***\n",
|
||||
header.MinKey, header.MaxKey, allKeys[0], allKeys[len(allKeys)-1])
|
||||
}
|
||||
}
|
||||
|
||||
reader.Close()
|
||||
}
|
||||
}
|
||||
75
examples/webui/commands/inspect_sst.go
Normal file
75
examples/webui/commands/inspect_sst.go
Normal file
@@ -0,0 +1,75 @@
|
||||
package commands
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"log"
|
||||
"os"
|
||||
|
||||
"code.tczkiot.com/srdb/sst"
|
||||
)
|
||||
|
||||
// InspectSST 检查特定 SST 文件
|
||||
func InspectSST(sstPath string) {
|
||||
// Check if file exists
|
||||
info, err := os.Stat(sstPath)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
fmt.Printf("File: %s\n", sstPath)
|
||||
fmt.Printf("Size: %d bytes\n", info.Size())
|
||||
|
||||
// Open reader
|
||||
reader, err := sst.NewReader(sstPath)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
defer reader.Close()
|
||||
|
||||
// Get header
|
||||
header := reader.GetHeader()
|
||||
fmt.Printf("\nHeader:\n")
|
||||
fmt.Printf(" RowCount: %d\n", header.RowCount)
|
||||
fmt.Printf(" MinKey: %d\n", header.MinKey)
|
||||
fmt.Printf(" MaxKey: %d\n", header.MaxKey)
|
||||
fmt.Printf(" DataSize: %d bytes\n", header.DataSize)
|
||||
|
||||
// Get all keys using GetAllKeys()
|
||||
allKeys := reader.GetAllKeys()
|
||||
fmt.Printf("\nActual keys in file: %d keys\n", len(allKeys))
|
||||
if len(allKeys) > 0 {
|
||||
fmt.Printf(" First key: %d\n", allKeys[0])
|
||||
fmt.Printf(" Last key: %d\n", allKeys[len(allKeys)-1])
|
||||
|
||||
if len(allKeys) <= 30 {
|
||||
fmt.Printf(" All keys: %v\n", allKeys)
|
||||
} else {
|
||||
fmt.Printf(" First 15: %v\n", allKeys[:15])
|
||||
fmt.Printf(" Last 15: %v\n", allKeys[len(allKeys)-15:])
|
||||
}
|
||||
}
|
||||
|
||||
// Try to get a specific key
|
||||
fmt.Printf("\nTrying to get key 332:\n")
|
||||
row, err := reader.Get(332)
|
||||
if err != nil {
|
||||
fmt.Printf(" Error: %v\n", err)
|
||||
} else if row == nil {
|
||||
fmt.Printf(" NULL\n")
|
||||
} else {
|
||||
fmt.Printf(" FOUND: seq=%d, time=%d\n", row.Seq, row.Time)
|
||||
}
|
||||
|
||||
// Try to get key based on actual first key
|
||||
if len(allKeys) > 0 {
|
||||
firstKey := allKeys[0]
|
||||
fmt.Printf("\nTrying to get actual first key %d:\n", firstKey)
|
||||
row, err := reader.Get(firstKey)
|
||||
if err != nil {
|
||||
fmt.Printf(" Error: %v\n", err)
|
||||
} else if row == nil {
|
||||
fmt.Printf(" NULL\n")
|
||||
} else {
|
||||
fmt.Printf(" FOUND: seq=%d, time=%d\n", row.Seq, row.Time)
|
||||
}
|
||||
}
|
||||
}
|
||||
59
examples/webui/commands/test_fix.go
Normal file
59
examples/webui/commands/test_fix.go
Normal file
@@ -0,0 +1,59 @@
|
||||
package commands
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"log"
|
||||
|
||||
"code.tczkiot.com/srdb"
|
||||
)
|
||||
|
||||
// TestFix 测试修复
|
||||
func TestFix(dbPath string) {
|
||||
db, err := srdb.Open(dbPath)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
defer db.Close()
|
||||
|
||||
table, err := db.GetTable("logs")
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
// Get total count
|
||||
result, err := table.Query().Rows()
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
totalCount := result.Count()
|
||||
fmt.Printf("Total rows in Query(): %d\n", totalCount)
|
||||
|
||||
// Test Get() for first 10, middle 10, and last 10
|
||||
testRanges := []struct {
|
||||
name string
|
||||
start int64
|
||||
end int64
|
||||
}{
|
||||
{"First 10", 1, 10},
|
||||
{"Middle 10", 50, 59},
|
||||
{"Last 10", int64(totalCount) - 9, int64(totalCount)},
|
||||
}
|
||||
|
||||
for _, tr := range testRanges {
|
||||
fmt.Printf("\n%s (keys %d-%d):\n", tr.name, tr.start, tr.end)
|
||||
foundCount := 0
|
||||
for seq := tr.start; seq <= tr.end; seq++ {
|
||||
row, err := table.Get(seq)
|
||||
if err != nil {
|
||||
fmt.Printf(" Seq %d: ERROR - %v\n", seq, err)
|
||||
} else if row == nil {
|
||||
fmt.Printf(" Seq %d: NULL\n", seq)
|
||||
} else {
|
||||
foundCount++
|
||||
}
|
||||
}
|
||||
fmt.Printf(" Found: %d/%d\n", foundCount, tr.end-tr.start+1)
|
||||
}
|
||||
|
||||
fmt.Printf("\n✅ If all keys found, the bug is FIXED!\n")
|
||||
}
|
||||
66
examples/webui/commands/test_keys.go
Normal file
66
examples/webui/commands/test_keys.go
Normal file
@@ -0,0 +1,66 @@
|
||||
package commands
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"log"
|
||||
|
||||
"code.tczkiot.com/srdb"
|
||||
)
|
||||
|
||||
// TestKeys 测试键
|
||||
func TestKeys(dbPath string) {
|
||||
db, err := srdb.Open(dbPath)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
defer db.Close()
|
||||
|
||||
table, err := db.GetTable("logs")
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
// Test keys from different ranges
|
||||
testKeys := []int64{
|
||||
1, 100, 331, 332, 350, 400, 447, 500, 600, 700, 800, 850, 861, 862, 900, 1000, 1500, 1665, 1666, 1723,
|
||||
}
|
||||
|
||||
fmt.Println("Testing key existence:")
|
||||
foundCount := 0
|
||||
for _, key := range testKeys {
|
||||
row, err := table.Get(key)
|
||||
if err != nil {
|
||||
fmt.Printf("Key %4d: NOT FOUND (%v)\n", key, err)
|
||||
} else if row == nil {
|
||||
fmt.Printf("Key %4d: NULL\n", key)
|
||||
} else {
|
||||
fmt.Printf("Key %4d: FOUND (time=%d)\n", key, row.Time)
|
||||
foundCount++
|
||||
}
|
||||
}
|
||||
|
||||
fmt.Printf("\nFound %d out of %d test keys\n", foundCount, len(testKeys))
|
||||
|
||||
// Query all
|
||||
result, err := table.Query().Rows()
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
count := result.Count()
|
||||
fmt.Printf("Total rows from Query: %d\n", count)
|
||||
|
||||
if count > 0 {
|
||||
first, _ := result.First()
|
||||
if first != nil {
|
||||
data := first.Data()
|
||||
fmt.Printf("First row _seq: %v\n", data["_seq"])
|
||||
}
|
||||
|
||||
last, _ := result.Last()
|
||||
if last != nil {
|
||||
data := last.Data()
|
||||
fmt.Printf("Last row _seq: %v\n", data["_seq"])
|
||||
}
|
||||
}
|
||||
}
|
||||
192
examples/webui/commands/webui.go
Normal file
192
examples/webui/commands/webui.go
Normal file
@@ -0,0 +1,192 @@
|
||||
package commands
|
||||
|
||||
import (
|
||||
"crypto/rand"
|
||||
"fmt"
|
||||
"log"
|
||||
"math/big"
|
||||
"net/http"
|
||||
"slices"
|
||||
"time"
|
||||
|
||||
"code.tczkiot.com/srdb"
|
||||
"code.tczkiot.com/srdb/webui"
|
||||
)
|
||||
|
||||
// StartWebUI 启动 WebUI 服务器
|
||||
func StartWebUI(dbPath string, addr string) {
|
||||
// 打开数据库
|
||||
db, err := srdb.Open(dbPath)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
defer db.Close()
|
||||
|
||||
// 创建示例 Schema
|
||||
userSchema := srdb.NewSchema("users", []srdb.Field{
|
||||
{Name: "name", Type: srdb.FieldTypeString, Indexed: false, Comment: "User name"},
|
||||
{Name: "email", Type: srdb.FieldTypeString, Indexed: false, Comment: "Email address"},
|
||||
{Name: "age", Type: srdb.FieldTypeInt64, Indexed: false, Comment: "Age"},
|
||||
{Name: "city", Type: srdb.FieldTypeString, Indexed: false, Comment: "City"},
|
||||
})
|
||||
|
||||
productSchema := srdb.NewSchema("products", []srdb.Field{
|
||||
{Name: "product_name", Type: srdb.FieldTypeString, Indexed: false, Comment: "Product name"},
|
||||
{Name: "price", Type: srdb.FieldTypeFloat, Indexed: false, Comment: "Price"},
|
||||
{Name: "quantity", Type: srdb.FieldTypeInt64, Indexed: false, Comment: "Quantity"},
|
||||
{Name: "category", Type: srdb.FieldTypeString, Indexed: false, Comment: "Category"},
|
||||
})
|
||||
|
||||
// 创建表(如果不存在)
|
||||
tables := db.ListTables()
|
||||
hasUsers := false
|
||||
hasProducts := false
|
||||
for _, t := range tables {
|
||||
if t == "users" {
|
||||
hasUsers = true
|
||||
}
|
||||
if t == "products" {
|
||||
hasProducts = true
|
||||
}
|
||||
}
|
||||
|
||||
if !hasUsers {
|
||||
table, err := db.CreateTable("users", userSchema)
|
||||
if err != nil {
|
||||
log.Printf("Create users table failed: %v", err)
|
||||
} else {
|
||||
// 插入一些示例数据
|
||||
users := []map[string]interface{}{
|
||||
{"name": "Alice", "email": "alice@example.com", "age": 30, "city": "Beijing"},
|
||||
{"name": "Bob", "email": "bob@example.com", "age": 25, "city": "Shanghai"},
|
||||
{"name": "Charlie", "email": "charlie@example.com", "age": 35, "city": "Guangzhou"},
|
||||
{"name": "David", "email": "david@example.com", "age": 28, "city": "Shenzhen"},
|
||||
{"name": "Eve", "email": "eve@example.com", "age": 32, "city": "Hangzhou"},
|
||||
}
|
||||
for _, user := range users {
|
||||
table.Insert(user)
|
||||
}
|
||||
log.Printf("Created users table with %d records", len(users))
|
||||
}
|
||||
}
|
||||
|
||||
if !hasProducts {
|
||||
table, err := db.CreateTable("products", productSchema)
|
||||
if err != nil {
|
||||
log.Printf("Create products table failed: %v", err)
|
||||
} else {
|
||||
// 插入一些示例数据
|
||||
products := []map[string]interface{}{
|
||||
{"product_name": "Laptop", "price": 999.99, "quantity": 10, "category": "Electronics"},
|
||||
{"product_name": "Mouse", "price": 29.99, "quantity": 50, "category": "Electronics"},
|
||||
{"product_name": "Keyboard", "price": 79.99, "quantity": 30, "category": "Electronics"},
|
||||
{"product_name": "Monitor", "price": 299.99, "quantity": 15, "category": "Electronics"},
|
||||
{"product_name": "Desk", "price": 199.99, "quantity": 5, "category": "Furniture"},
|
||||
{"product_name": "Chair", "price": 149.99, "quantity": 8, "category": "Furniture"},
|
||||
}
|
||||
for _, product := range products {
|
||||
table.Insert(product)
|
||||
}
|
||||
log.Printf("Created products table with %d records", len(products))
|
||||
}
|
||||
}
|
||||
|
||||
// 启动后台数据插入协程
|
||||
go autoInsertData(db)
|
||||
|
||||
// 启动 Web UI
|
||||
handler := webui.NewWebUI(db)
|
||||
|
||||
fmt.Printf("SRDB Web UI is running at http://%s\n", addr)
|
||||
fmt.Println("Press Ctrl+C to stop")
|
||||
fmt.Println("Background data insertion is running...")
|
||||
|
||||
if err := http.ListenAndServe(addr, handler); err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
// generateRandomData 生成指定大小的随机数据 (2KB ~ 512KB)
|
||||
func generateRandomData() string {
|
||||
minSize := 2 * 1024 // 2KB
|
||||
maxSize := (1 * 1024 * 1024) / 2 // 512KB
|
||||
|
||||
sizeBig, _ := rand.Int(rand.Reader, big.NewInt(int64(maxSize-minSize)))
|
||||
size := int(sizeBig.Int64()) + minSize
|
||||
|
||||
data := make([]byte, size)
|
||||
rand.Read(data)
|
||||
|
||||
return fmt.Sprintf("%x", data)
|
||||
}
|
||||
|
||||
// autoInsertData 在后台自动插入数据
|
||||
func autoInsertData(db *srdb.Database) {
|
||||
ticker := time.NewTicker(time.Second)
|
||||
defer ticker.Stop()
|
||||
|
||||
counter := 1
|
||||
|
||||
for range ticker.C {
|
||||
tables := db.ListTables()
|
||||
var logsTable *srdb.Table
|
||||
|
||||
hasLogs := slices.Contains(tables, "logs")
|
||||
|
||||
if !hasLogs {
|
||||
logsSchema := srdb.NewSchema("logs", []srdb.Field{
|
||||
{Name: "timestamp", Type: srdb.FieldTypeString, Indexed: false, Comment: "Timestamp"},
|
||||
{Name: "data", Type: srdb.FieldTypeString, Indexed: false, Comment: "Random data"},
|
||||
{Name: "size_bytes", Type: srdb.FieldTypeInt64, Indexed: false, Comment: "Data size in bytes"},
|
||||
})
|
||||
|
||||
var err error
|
||||
logsTable, err = db.CreateTable("logs", logsSchema)
|
||||
if err != nil {
|
||||
log.Printf("Failed to create logs table: %v", err)
|
||||
continue
|
||||
}
|
||||
log.Println("Created logs table for background data insertion")
|
||||
} else {
|
||||
var err error
|
||||
logsTable, err = db.GetTable("logs")
|
||||
if err != nil || logsTable == nil {
|
||||
log.Printf("Failed to get logs table: %v", err)
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
data := generateRandomData()
|
||||
sizeBytes := len(data)
|
||||
|
||||
record := map[string]any{
|
||||
"timestamp": time.Now().Format(time.RFC3339),
|
||||
"data": data,
|
||||
"size_bytes": int64(sizeBytes),
|
||||
}
|
||||
|
||||
err := logsTable.Insert(record)
|
||||
if err != nil {
|
||||
log.Printf("Failed to insert data: %v", err)
|
||||
} else {
|
||||
sizeStr := formatBytes(sizeBytes)
|
||||
log.Printf("Inserted record #%d, size: %s", counter, sizeStr)
|
||||
counter++
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// formatBytes 格式化字节大小显示
|
||||
func formatBytes(bytes int) string {
|
||||
const unit = 1024
|
||||
if bytes < unit {
|
||||
return fmt.Sprintf("%d B", bytes)
|
||||
}
|
||||
div, exp := int64(unit), 0
|
||||
for n := bytes / unit; n >= unit; n /= unit {
|
||||
div *= unit
|
||||
exp++
|
||||
}
|
||||
units := []string{"KB", "MB", "GB", "TB"}
|
||||
return fmt.Sprintf("%.2f %s", float64(bytes)/float64(div), units[exp])
|
||||
}
|
||||
98
examples/webui/main.go
Normal file
98
examples/webui/main.go
Normal file
@@ -0,0 +1,98 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"flag"
|
||||
"fmt"
|
||||
"os"
|
||||
|
||||
"code.tczkiot.com/srdb/examples/webui/commands"
|
||||
)
|
||||
|
||||
func main() {
|
||||
if len(os.Args) < 2 {
|
||||
printUsage()
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
command := os.Args[1]
|
||||
args := os.Args[2:]
|
||||
|
||||
switch command {
|
||||
case "webui", "serve":
|
||||
serveCmd := flag.NewFlagSet("webui", flag.ExitOnError)
|
||||
dbPath := serveCmd.String("db", "./data", "Database directory path")
|
||||
addr := serveCmd.String("addr", ":8080", "Server address")
|
||||
serveCmd.Parse(args)
|
||||
commands.StartWebUI(*dbPath, *addr)
|
||||
|
||||
case "check-data":
|
||||
checkDataCmd := flag.NewFlagSet("check-data", flag.ExitOnError)
|
||||
dbPath := checkDataCmd.String("db", "./data", "Database directory path")
|
||||
checkDataCmd.Parse(args)
|
||||
commands.CheckData(*dbPath)
|
||||
|
||||
case "check-seq":
|
||||
checkSeqCmd := flag.NewFlagSet("check-seq", flag.ExitOnError)
|
||||
dbPath := checkSeqCmd.String("db", "./data", "Database directory path")
|
||||
checkSeqCmd.Parse(args)
|
||||
commands.CheckSeq(*dbPath)
|
||||
|
||||
case "dump-manifest":
|
||||
dumpCmd := flag.NewFlagSet("dump-manifest", flag.ExitOnError)
|
||||
dbPath := dumpCmd.String("db", "./data", "Database directory path")
|
||||
dumpCmd.Parse(args)
|
||||
commands.DumpManifest(*dbPath)
|
||||
|
||||
case "inspect-all-sst":
|
||||
inspectAllCmd := flag.NewFlagSet("inspect-all-sst", flag.ExitOnError)
|
||||
sstDir := inspectAllCmd.String("dir", "./data/logs/sst", "SST directory path")
|
||||
inspectAllCmd.Parse(args)
|
||||
commands.InspectAllSST(*sstDir)
|
||||
|
||||
case "inspect-sst":
|
||||
inspectCmd := flag.NewFlagSet("inspect-sst", flag.ExitOnError)
|
||||
sstPath := inspectCmd.String("file", "./data/logs/sst/000046.sst", "SST file path")
|
||||
inspectCmd.Parse(args)
|
||||
commands.InspectSST(*sstPath)
|
||||
|
||||
case "test-fix":
|
||||
testFixCmd := flag.NewFlagSet("test-fix", flag.ExitOnError)
|
||||
dbPath := testFixCmd.String("db", "./data", "Database directory path")
|
||||
testFixCmd.Parse(args)
|
||||
commands.TestFix(*dbPath)
|
||||
|
||||
case "test-keys":
|
||||
testKeysCmd := flag.NewFlagSet("test-keys", flag.ExitOnError)
|
||||
dbPath := testKeysCmd.String("db", "./data", "Database directory path")
|
||||
testKeysCmd.Parse(args)
|
||||
commands.TestKeys(*dbPath)
|
||||
|
||||
case "help", "-h", "--help":
|
||||
printUsage()
|
||||
|
||||
default:
|
||||
fmt.Printf("Unknown command: %s\n\n", command)
|
||||
printUsage()
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
func printUsage() {
|
||||
fmt.Println("SRDB WebUI - Database management tool")
|
||||
fmt.Println("\nUsage:")
|
||||
fmt.Println(" webui <command> [flags]")
|
||||
fmt.Println("\nCommands:")
|
||||
fmt.Println(" webui, serve Start WebUI server (default: :8080)")
|
||||
fmt.Println(" check-data Check database tables and row counts")
|
||||
fmt.Println(" check-seq Check specific sequence numbers")
|
||||
fmt.Println(" dump-manifest Dump manifest information")
|
||||
fmt.Println(" inspect-all-sst Inspect all SST files")
|
||||
fmt.Println(" inspect-sst Inspect a specific SST file")
|
||||
fmt.Println(" test-fix Test fix for data retrieval")
|
||||
fmt.Println(" test-keys Test key existence")
|
||||
fmt.Println(" help Show this help message")
|
||||
fmt.Println("\nExamples:")
|
||||
fmt.Println(" webui serve -db ./mydb -addr :3000")
|
||||
fmt.Println(" webui check-data -db ./mydb")
|
||||
fmt.Println(" webui inspect-sst -file ./data/logs/sst/000046.sst")
|
||||
}
|
||||
Reference in New Issue
Block a user