wayback

Category: Daily Life | Uploader: icodeface | Downloads: 0 | Version: v1.0（Latest）

Query and explore the Wayback Archiver personal web archiving system

Changelog: Source: GitHub https://github.com/icodeface/wayback-archiver

Directory Structure

Current level: tree/main/

📁 .github/
- 📁 workflows/
  - 📄 release.yml 1.6 KB
📁 browser/
- 📁 src/
  - 📄 archiver.ts 5.6 KB
  - 📄 config.ts 1.1 KB
  - 📄 dom-collector.ts 9.4 KB
  - 📄 main.ts 14.7 KB
  - 📄 page-filter.ts 881 B
  - 📄 page-freezer.ts 4.9 KB
  - 📄 puppeteer.ts 4.5 KB
  - 📄 style-inliner.ts 12.3 KB
  - 📄 types.ts 290 B
- 📄 .gitignore 26 B
- 📄 build.js 3.4 KB
- 📄 package-lock.json 1.7 KB
- 📄 package.json 406 B
- 📄 tsconfig.json 437 B
📁 docs/
- 📁 security/
  - 📄 FIXES_2026-03-14.md 3.0 KB
  - 📄 README.md 2.1 KB
- 📁 technical/
  - 📄 snapshot-and-update.md 10.0 KB
  - 📄 style-inliner-flex-fix.md 3.5 KB
  - 📄 virtual-scroll-capture.md 5.2 KB
- 📄 BUILD.md 1.5 KB
- 📄 CHANGELOG.md 3.6 KB
- 📄 DOCKER.md 3.1 KB
- 📄 PUPPETEER.md 2.0 KB
- 📄 REMOTE_DEPLOYMENT.md 3.2 KB
📁 screenshot/
- 📄 index.webp 77.9 KB
- 📄 v2ex.webp 181.7 KB
- 📄 x.webp 185.8 KB
📁 server/
- 📁 cmd/
  - 📁 backfill/
    
    📄 main.go 1.3 KB
  - 📁 server/
    
    📄 main.go 5.3 KB
- 📁 internal/
  - 📁 api/
    
    📄 archive_handler.go 1.5 KB
    
    📄 archive_handler_test.go 7.1 KB
    
    📄 compression_test.go 15.0 KB
    
    📄 debug_handler.go 1.3 KB
    
    📄 handler.go 623 B
    
    📄 helpers.go 724 B
    
    📄 helpers_test.go 2.1 KB
    
    📄 injector.go 13.2 KB
    
    📄 log_handler.go 2.2 KB
    
    📄 pages_handler.go 4.6 KB
    
    📄 patterns.go 1.9 KB
    
    📄 patterns_test.go 2.2 KB
    
    📄 precompiled_regex_test.go 5.6 KB
    
    📄 proxy_resource_integration_test.go 9.5 KB
    
    📄 rewriter.go 3.9 KB
    
    📄 rewriter_test.go 2.8 KB
    
    📄 routes.go 4.3 KB
    
    📄 routes_test.go 4.9 KB
    
    📄 security_test.go 846 B
    
    📄 streaming_test.go 3.5 KB
    
    📄 timeline_handler_test.go 3.8 KB
    
    📄 view_handler.go 23.3 KB
    
    📄 view_handler_security_test.go 5.8 KB
    
    📄 view_handler_test.go 21.4 KB
  - 📁 config/
    
    📄 config.go 5.3 KB
    
    📄 config_test.go 6.2 KB
    
    📄 memory.go 1.8 KB
  - 📁 database/
    
    📄 postgres.go 18.8 KB
    
    📄 postgres_test.go 8.0 KB
  - 📁 logging/
    
    📄 logger.go 7.9 KB
    
    📄 logger_test.go 7.5 KB
  - 📁 models/
    
    📄 models.go 1.5 KB
  - 📁 storage/
    
    📄 cache_mutex_test.go 2.8 KB
    
    📄 cache_test.go 10.3 KB
    
    📄 css_parser.go 2.8 KB
    
    📄 deduplicator.go 26.6 KB
    
    📄 deduplicator_security_test.go 3.3 KB
    
    📄 deletion_queue.go 4.4 KB
    
    📄 deletion_queue_test.go 7.8 KB
    
    📄 filesystem.go 14.7 KB
    
    📄 filesystem_security_test.go 5.5 KB
    
    📄 filesystem_test.go 1.2 KB
    
    📄 hash_test.go 2.0 KB
    
    📄 html_extractor.go 8.7 KB
    
    📄 html_extractor_test.go 5.7 KB
    
    📄 markdown_extractor.go 1.5 KB
    
    📄 markdown_extractor_test.go 5.7 KB
    
    📄 precompiled_regex_test.go 4.3 KB
    
    📄 rewriter.go 3.6 KB
    
    📄 rewriter_bench_test.go 1.1 KB
    
    📄 rewriter_fast.go 10.9 KB
    
    📄 rewriter_test.go 18.0 KB
    
    📄 security_test.go 4.0 KB
    
    📄 streaming_test.go 16.3 KB
    
    📄 text_extractor.go 879 B
    
    📄 transport_test.go 1.5 KB
- 📁 migrations/
  - 📄 006_drop_content_hash_unique.sql 226 B
  - 📄 007_drop_created_at.sql 95 B
- 📁 web/
  - 📄 embed.go 67 B
  - 📄 favicon.ico 15.0 KB
  - 📄 index.html 26.8 KB
  - 📄 logs.html 13.0 KB
  - 📄 robots.txt 434 B
  - 📄 timeline.html 11.9 KB
- 📄 go.mod 1.6 KB
- 📄 go.sum 8.8 KB
- 📄 init_db.sql 1.6 KB
📁 tests/
- 📁 browser/
  - 📄 test-archive.js 1.7 KB
  - 📄 test-cssom-serialize.js 8.6 KB
  - 📄 test-dedup.js 5.9 KB
  - 📄 test-events.html 783 B
  - 📄 test-header-fix.js 4.3 KB
  - 📄 test-page.html 3.0 KB
  - 📄 test-puppeteer-script.js 1.6 KB
  - 📄 test_timing.html 1.7 KB
- 📁 root/
  - 📄 test-browser-script.js 4.7 KB
- 📁 server/
  - 📁 scripts/
    
    📄 check_recent.go 983 B
    
    📄 check_resources_in_db.go 832 B
    
    📄 fix_single_page.go 3.2 KB
    
    📄 test_icon_extraction.go 600 B
    
    📄 test_url_resolve.go 938 B
  - 📄 chunk-mapping.json 1.1 KB
  - 📄 package-lock.json 42.2 KB
  - 📄 package.json 221 B
  - 📄 test_deletion_queue.js 4.0 KB
  - 📄 test_download_fallback.js 5.5 KB
  - 📄 test_icon_capture.js 2.6 KB
  - 📄 test_icons.html 957 B
  - 📄 test_memory_leak.js 12.2 KB
  - 📄 test_update_feature.js 10.7 KB
- 📄 README.md 266 B
📄 .dockerignore 127 B
📄 .env.example 1.7 KB
📄 .gitignore 326 B
📄 docker-compose.yml 1.2 KB
📄 Dockerfile 1.5 KB
📄 LICENSE 1.1 KB
📄 Makefile 2.0 KB
📄 package-lock.json 41.9 KB
📄 package.json 310 B
📄 README-zh.md 10.5 KB
📄 README.md 12.9 KB
📄 skill.md 5.9 KB

SKILL.md

---
name: wayback
description: Query and explore the Wayback Archiver personal web archiving system
---

# Wayback Archiver

Wayback Archiver is a self-hosted personal web archiving system that captures and preserves web pages with full fidelity — HTML, CSS, JavaScript, images, fonts, and all resources.

**Repository**: https://github.com/icodeface/wayback-archiver

## When to use

Use this skill when the user wants to:
- Search archived web pages
- View snapshots of a specific URL
- List recent archives or filter by date range
- Get details about a specific archived page
- Read page content in Markdown format (for AI/LLM consumption)
- Explore the archiving system's data

## Prerequisites

Before using this skill, ensure the Wayback Archiver server is running:

- **PostgreSQL** 14+
- **Chrome** or **Firefox** with [Tampermonkey](https://www.tampermonkey.net/) extension (v5.3+)

## Quick Start

### 1. Download Pre-built Binaries

Download from [Releases](https://github.com/icodeface/wayback-archiver/releases):

- **Server binary**: `wayback-server-<os>-<arch>.tar.gz` (or `.zip` for Windows)
- **Userscript**: `wayback-userscript.js`

Extract the server binary:

```bash
# macOS/Linux
tar -xzf wayback-server-*.tar.gz

# Windows: extract the .zip file
```

> **Building from source?** See [docs/BUILD.md](https://github.com/icodeface/wayback-archiver/blob/main/docs/BUILD.md).

### 2. Setup Database

```bash
# Create database (PostgreSQL 默认使用当前系统用户名)
createdb wayback

# Run schema (init_db.sql is included in the release archive)
psql wayback < init_db.sql
```

### 3. Start Server

```bash
./wayback-server
```

Server runs at `http://localhost:8080` by default.

### 4. Install Browser Script

1. Download `wayback-userscript.js` from [Releases](https://github.com/icodeface/wayback-archiver/releases)
2. Open Tampermonkey dashboard
3. Create new script and paste the content
4. Save and enable

> **Chrome users:** Enable "Allow user scripts" in Tampermonkey's extension settings (right-click icon → Manage extension). Firefox does not require this.

## API Usage

Base URL: `http://localhost:8080`

### Authentication (Optional)

When `AUTH_PASSWORD` is set, use HTTP Basic Auth:
- **Username**: `wayback`
- **Password**: `$AUTH_PASSWORD`

### Endpoints

#### List All Pages

```bash
curl "http://localhost:8080/api/pages?limit=100&offset=0"
```

#### Search Pages

```bash
curl "http://localhost:8080/api/search?q=$KEYWORD"
```

#### Filter by Date Range

```bash
curl "http://localhost:8080/api/pages?from=2026-03-01&to=2026-03-12"
```

#### Get Page Details

```bash
curl "http://localhost:8080/api/pages/$PAGE_ID"
```

#### Get Page Content as Markdown

Returns the page body as clean Markdown (strips scripts, nav, footer, etc.). Ideal for AI/LLM consumption.

```bash
curl "http://localhost:8080/api/pages/$PAGE_ID/content"
```

#### Get Timeline for URL

```bash
curl "http://localhost:8080/api/pages/timeline?url=$ENCODED_URL"
```

#### View Archived Page

Open in browser:
```
http://localhost:8080/view/$PAGE_ID
```

## Data Models

### Page

```typescript
{
  id: number;
  url: string;
  title: string;
  captured_at: string;      // ISO 8601 timestamp
  html_path: string;
  content_hash: string;     // SHA-256
  first_visited: string;    // ISO 8601 timestamp
  last_visited: string;     // ISO 8601 timestamp
  created_at: string;       // ISO 8601 timestamp
}
```

### Resource

```typescript
{
  id: number;
  url: string;
  content_hash: string;     // SHA-256
  resource_type: string;    // "css", "js", "image", "font", etc.
  file_path: string;
  file_size: number;
  first_seen: string;       // ISO 8601 timestamp
  last_seen: string;        // ISO 8601 timestamp
}
```

## Storage Layout

```
data/
├── html/                     # HTML snapshots, organized by date
│   └── 2026/03/12/
│       └── <timestamp>_<hash>.html
└── resources/                # Deduplicated static resources
    └── ab/cd/
        └── <sha256>.css
```

## Features

- **High-fidelity replay**: CSSOM serialization, computed styles inlining
- **Full-page capture**: HTML, CSS, JS, images, fonts
- **Cross-origin resource recovery**: Server-side download of CORS-blocked resources
- **Content-hash deduplication**: SHA-256 based, shared resources stored once
- **Version history**: Multiple snapshots per URL, distinguished by timestamp
- **Smart dedup**: Session-level + server-level prevents redundant captures
- **Dynamic content support**: Captures live DOM state, auto-updates on significant mutations
- **SPA-aware**: Detects SPA navigation, resets capture state per route
- **Anti-refresh protection**: Archived pages frozen (timers, WebSockets, navigation APIs neutralized)

## Testing

See [docs/BUILD.md](https://github.com/icodeface/wayback-archiver/blob/main/docs/BUILD.md) for build and test instructions.

## Troubleshooting

| Issue | Solution |
|-------|----------|
| Server won't start | Check PostgreSQL is running and database exists |
| Pages not archiving | Verify Tampermonkey script is enabled and server is reachable |
| Missing resources | Check proxy settings if behind corporate firewall |
| Authentication errors | Verify `AUTH_PASSWORD` env var matches your curl credentials |

## Configuration

Create `.env` file in the project root (or set environment variables directly):

The server automatically loads `.env` from the working directory if it exists.

```bash
# Database
DB_HOST=localhost
DB_PORT=5432
DB_USER=postgres  # 可选，PostgreSQL 默认使用系统用户名
DB_PASSWORD=
DB_NAME=wayback
DB_SSLMODE=disable

# Server
SERVER_HOST=0.0.0.0  # 默认 127.0.0.1，设置 0.0.0.0 监听所有网卡
SERVER_PORT=8080

# Storage
DATA_DIR=./data

# Resource metadata cache
RESOURCE_METADATA_CACHE_MB=256  # Optional; RESOURCE_CACHE_MB is a legacy alias

# Authentication (optional)
AUTH_PASSWORD=

# Debug APIs (disabled by default; enable only while troubleshooting)
ENABLE_DEBUG_API=false
```

## License

MIT

Comments 0

Latest | Hot

Please login before commenting.

Loading comments...

wayback

Directory Structure

SKILL.md

Report

Notice