The SpyWeb Master Guide

SpyWeb is a zero-dependency web monitoring engine that lets you customize every stage of the pipeline with Lua scripts (Luau dialect). Place a hooks.lua alongside your job's config.toml and define any combination of 13 hook functions.

Key Rules

• Return the value (modified or not) to continue the pipeline
• Return nil or false to drop/skip (behavior varies per hook)
• Plain Lua globals persist across runs only while the job task stays alive
• store_* is job-scoped persistent storage and global_store_* is shared across jobs
• Hot-reload creates a fresh Lua VM and resets global Lua state, but stored values survive
• Hook errors are logged and skipped - they never crash the job
• Efficiency: Only define the hooks you actually need. SpyWeb pre-detects which functions exist at startup and skips the processing logic entirely for any that are missing.

Every scrape run flows through these stages in order:

Each job with Lua hooks lives in its own directory under jobs/:

jobs/
└── my-scraper/
    ├── config.toml    # job configuration (required)
    └── hooks.lua       # Lua hooks (optional)

spyweb auto-discovers hooks.lua when loading the job. No config changes needed - just create the file.

Run SpyWeb on a VPS to ensure 24/7 monitoring even when your laptop is closed. We've prepared a comprehensive guide covering:

• Binary installation & permissions
• Running as a Systemd background service
• Secure UI access (SSH Tunnels, Nginx, Firewalls)
• Recommended terminal editor setup

SpyWeb's Lua engine maps internal data structures into native Lua tables. Below are the technical contracts for every available hook.

What is ctx? Every hook receives a per-cycle context table as its second argument. ctx is a scratch table created fresh for each scrape cycle and discarded at the end. Use it to pass data between hooks within the same cycle (e.g., store a value in before_fetch and read it in after_extract). It also carries reserved fields set by the engine:

• ctx.last_fetch — FetchAttempt envelope with request, ok, response (on success), and error (on failure)
• ctx.selector_matches — Number of CSS selector hits
• ctx.telemetry — Performance telemetry for the cycle
• ctx.shared — A table shared across all hooks in the cycle

Called before every HTTP request. Modify the URL, add headers, or skip entirely.

Parameter Structure:

{
  url = "https://example.com",
  method = "GET",
  headers = {
    ["User-Agent"] = "SpyWeb/1.0",
    ["Accept"] = "*/*"
  },
  timeout = 15,          -- optional override (default 30s)
  proxy = nil,           -- optional proxy override
  max_body_size = nil    -- optional body limit in MB (default: 10)
}

-- Basic: add auth header
function before_fetch(request, ctx)
    request.headers["Authorization"] = "Bearer my-token"
    return request
end

Overrides Phase 1 (Fetching). If this hook is defined, SpyWeb will NOT use its internal HTTP client.

Purpose: To fetch data using a custom method (e.g., launching a Chrome instance via CDP or using a proxy rotater).

Example: Custom Fetching (e.g. bypassing internal HTTP client)

function override_fetch(request, ctx)
    -- E.g. call an external API or run a headless browser script
    local res, err = http_get("http://my-renderer.local/?url=" .. request.url)
    if not res then
        return { error = "renderer request failed: " .. err.error }
    end
    -- MUST return a response object (or { error = "message" } for failure):
    return {
        status = res.status,
        url = request.url,
        body = res.body,
        headers = res.headers
    }
end

Called after every fetch attempt. It receives a stable envelope with request context, plus either a response snapshot or an error snapshot. On successful fetches, only response.body is accepted back into the pipeline. On failed fetches, Lua may return a full synthetic response table to continue extraction anyway.

Parameter Structure:

{
  ok = true,
  request = {
    url = "https://example.com",
    headers = { ["Accept"] = "text/html" },
    proxy = nil
  },
  response = {
    status = 200,
    url = "https://example.com",
    body = "...",
    headers = { ["Content-Type"] = "text/html" },
    proxy = "http://proxy:8080",       -- proxy used (if any)
    time_ms = 1234,                     -- total request time in ms
    size = 45678,                       -- body size in bytes
  },
  error = nil
}

function after_fetch(fetch_result, ctx)
    if not fetch_result.ok then
        if fetch_result.response then
            print("HTTP error: " .. fetch_result.response.status)
        end
        notify("Fetch failed", fetch_result.error.message, 8000)
        return nil
    end

    if fetch_result.response.status ~= 200 then
        print("Bad status: " .. fetch_result.response.status)
        return nil  -- skip extraction
    end

    -- Strip unwanted content from the body
    fetch_result.response.body = fetch_result.response.body:gsub("<script.-</script>", "")
    return fetch_result
end

Overrides Phase 2 (Extraction). If this hook is defined, SpyWeb will NOT use its internal HTML/CSS scraper.

Purpose: To parse non-HTML data (like JSON or XML) or apply entirely custom extraction logic.

Example: Scraping a JSON API

function override_extract(response, ctx)
    local data = json_decode(response.body)
    local items = {}
    for i, post in ipairs(data.posts) do
        table.insert(items, {
            fields = {
                author = post.user.name,
                message = post.content
            },
            -- matches = {} -- Note: Engine ignores this and auto-tags matches from fields
        })
    end
    return items
end

Batch hook - receives all extracted items at once. Good for cross-item logic, dedup, or sorting.

No short-circuit: returning nil or empty table just means no items, not a pipeline abort.

Parameter Structure:

{
  { 
    fields = { title = "...", link = "..." },
    matches = { "keyword1", ... } -- READ-ONLY: Populated by engine
  },
  ...
}

function after_extract(items, ctx)
    -- Remove duplicate titles
    local seen = {}
    local unique = {}
    for _, item in ipairs(items) do
        local title = item.fields.title or ""
        if not seen[title] and title ~= "" then
            seen[title] = true
            table.insert(unique, item)
        end
    end
    return unique
end

Called once per item. Return the item to keep it, or nil to drop it before it reaches the database.

Parameter Structure:

{ 
  fields = { title = "...", price = "..." },
  matches = { "keyword1", ... }
}

function filter_item(item, ctx)
    local title = (item.fields.title or ""):lower()

    -- Drop unwanted items
    if title == "" or title:find("sponsored") then
        return nil
    end

    -- Mutate fields before storage
    item.fields.title = title:sub(1,1):upper() .. title:sub(2)

    return item
end

Last chance to drop items before dedup + insert. Return nil to skip storing and notifying.

Parameter Structure:

-- Array of item tables
{ {fields={...}, matches={...}}, ... }

function before_store(items, ctx)
    -- Only store during business hours
    local hour = os.date("*t").hour
    if hour < 9 or hour > 17 then
        return nil  -- don't store, don't notify
    end
    return items
end

Reshape or silence webhook POSTs. Receives the full JSON payload (job_name, item_count, items).

Parameter Structure:

{
  job_name = "My Job",
  item_count = 2,
  items = {
    { fields = { title = "...", link = "..." }, matches = {...} },
    ...
  }
}

function before_webhook(payload, ctx)
    local job = payload.job_name
    local count = payload.item_count
    
    -- DISCORD FORMAT (Uncomment to use)
    -- ---------------------------------
    --[[
    local embeds = {}
    for _, item in ipairs(payload.items) do
        table.insert(embeds, {
            title = item.fields.title or "New Item",
            url = item.fields.link,
            color = 10640871, -- Purple
            description = "Matched: " .. table.concat(item.matches, ", ")
        })
    end
    return {
        content = "**" .. job .. "** found " .. count .. " new items!",
        embeds = embeds
    }
    --]]

    -- SLACK FORMAT
    -- ------------
    local blocks = {
        {
            type = "section",
            text = { type = "mrkdwn", text = "*SpyWeb Alert:* " .. job .. " found " .. count .. " items." }
        }
    }
    for _, item in ipairs(payload.items) do
        table.insert(blocks, { type = "divider" })
        table.insert(blocks, {
            type = "section",
            text = { type = "mrkdwn", text = "<" .. (item.fields.link or "#") .. "|" .. (item.fields.title or "Item") .. ">" }
        })
    end
    return { blocks = blocks }
end

Batch-level lifecycle hook that fires after all workers complete a job iteration, before the interval sleep. No ctx parameter — operates on shared globals accumulated during the batch.

function on_finished()
    log("All workers done, sleeping until next interval")
    _G.batch_stats = nil -- clear accumulated state
end

SpyWeb injects several global helper functions into the Lua environment for networking, notifications, logging, and storage.

Storage & Persistence

SpyWeb supports state management through both Runtime Memory (transient) and an Embedded Database (persistent). Memory is fast but resets on reload; the database survives restarts.

local c = tonumber(store_get("count") or "0")
store_set("count", tostring(c + 1))

-- WRONG APPROACH (race condition prone)
local v = tonumber(global_store_get("visits") or "0")
global_store_set("visits", tostring(v + 1))

-- CORRECT APPROACH (race condition safe)
global_store_incr("visits", 0, 1)

-- This resets to 1 if you edit hooks.lua or restart SpyWeb
visit_count = (visit_count or 0) + 1
log("Session visit: " .. visit_count)

function before_webhook(payload, ctx)
    if ctx.last_fetch and ctx.last_fetch.ok and ctx.last_fetch.response and ctx.last_fetch.response.body then
        local count = 0
        for _ in ctx.last_fetch.response.body:gmatch("promo") do
            count = count + 1
        end
        payload.last_fetch_promo_hits = count
    end
    return payload
end

http_get(url, [headers]) async

Performs an HTTP GET request. Returns (response, nil) on success or (nil, error_table) on failure. The headers argument is an optional Lua table containing key-value pairs for request headers.

Success response fields:

Error table fields:

local res, err = http_get("https://api.example.com", {
    ["Authorization"] = "Bearer token123",
    ["X-Custom-Header"] = "my-value"
})
if not res then
    log("request failed: " .. err.error .. " (" .. err.kind .. ")")
    return
end

http_post(url, body, [headers]) async

Performs an HTTP POST request. Returns (response, nil) on success or (nil, error_table) on failure. The response and error tables follow the same schema as http_get. By default, it sets the Content-Type to application/x-www-form-urlencoded unless overridden in the headers argument.

local res, err = http_post("https://api.example.com", '{"foo":"bar"}', {
    ["Content-Type"] = "application/json",
    ["Accept"] = "application/json"
})
if not res then
    log("post failed: " .. err.error)
    return
end

http_request(options) async

Performs an arbitrary HTTP request with full control over method, URL, body, headers, proxy, timeout, and response size limit. Returns (response, nil) on success or (nil, error_table) on failure. The response and error tables follow the same schema as http_get. The options argument is a Lua table with these fields:

-- HEAD request (no body)
local res, err = http_request({ method = "HEAD", url = "https://example.com" })
if not res then
    log("head failed: " .. err.error)
    return
end

-- POST with JSON body through a proxy
local res, err = http_request({
    method = "POST",
    url = "https://api.example.com/data",
    body = '{"key":"value"}',
    headers = { ["Content-Type"] = "application/json" },
    proxy = "http://user:pass@residential-proxy:8080",
    timeout = 15
})
if not res then
    log("request failed: " .. err.kind)
    return
end

http_multipart(url, fields, [headers]) async

Performs a multipart/form-data POST request for file uploads. Returns (response, nil) on success or (nil, error_table) on failure. The response and error tables follow the same schema as http_get. The fields argument is a Lua table where each value is either:

• A string — sent as a text form field.
• A table — sent as a file attachment, with these keys:
  • content (required) — file bytes (binary-safe Lua string).
  • filename (optional) — the server-side file name.
  • type (optional) — MIME type (e.g. "image/png").

Use together with page:screenshot() and fs_read_binary() for screenshot upload workflows.

-- Text fields only
local res, err = http_multipart("https://api.example.com/form", {
    name = "SpyWeb",
    version = "1.0"
})
if not res then
    log("upload failed: " .. err.error)
    return
end

-- File upload (screenshot)
local page = cdp.launch({ headless = true }):attach()
page:open("https://example.com")
page:screenshot("page.png", { full_page = true })
local image_data = fs_read_binary("page.png")
local res, err = http_multipart("https://hooks.example.com/upload", {
    screenshot = { content = image_data, filename = "page.png", type = "image/png" },
    caption = "my screenshot"
})
if not res then
    log("screenshot upload failed: " .. err.kind)
    return
end

db_query(sql, [params]) async

Executes a SELECT query on the SQLite database. Available only on the SQL (SQLite) build variant. Returns an array of row-tables with column-value pairs. Integer/Real/Text/Null SQL types map to Lua number/string/nil. On the KV (redb) variant, calling this returns a clear error directing you to download the SQLite build.

local rows = db_query("SELECT json FROM records WHERE job_id = ? LIMIT 5", { "my-job" })
for _, row in ipairs(rows) do
    local data = json_decode(row.json)
    log("Title: " .. data.title)
end

db_exec(sql, [params]) async

Executes an INSERT, UPDATE, DELETE, or DDL statement on the SQLite database. Available only on the SQL (SQLite) build variant. Returns the number of rows affected. On the KV (redb) variant, calling this returns a clear error directing you to download the SQLite build.

db_exec("INSERT OR REPLACE INTO markers (id, val) VALUES (?, ?)", { "page", 5 })

json_encode(table) sync

Converts a Lua table or value into a JSON string. Handles nested tables, arrays, booleans, and nulls automatically using the built-in high-performance JSON engine.

local payload = {
    url = "https://example.com",
    options = { waitUntil = "networkidle2" }
}
local body = json_encode(payload)
local resp = http_post("https://api.rendering-service.com/render", body, {["Content-Type"]="application/json"})

json_decode(string) sync

Parses a JSON string into a native Lua table. Returns nil and logs an error if the JSON is malformed.

local data = json_decode('{"status": "ok", "count": 42}')
log("Status is: " .. data.status)

defer(fn) sync

Registers a cleanup function to run immediately after the top-level Hook exits. Essential for preventing zombie processes and memory leaks.

function override_fetch(request, ctx)
    local browser = cdp.launch({ headless = true })
    defer(function() browser:close() end) -- Always closes

    local page = browser:attach()
    page:open(request.url)
    return { status = 200, body = page:content() }
end

View full Execution Lifecycle guide →

dump(value) sync

Formats any Lua value into a readable string for debugging. This is especially useful for nested hook payloads like fetch_result or ctx.last_fetch. It handles nested tables, quotes strings, and marks cycles as <cycle> instead of recursing forever.

function after_fetch(fetch_result, ctx)
     print(dump(fetch_result))
     log(dump(ctx.last_fetch))
     return fetch_result
 end

copy(table) sync

Returns a shallow copy of a table. Only the first level of keys and values are copied; nested tables are still shared by reference.

local original = { a = 1, b = { c = 2 } }
local shallow = copy(original)
shallow.a = 99 -- original.a is still 1
shallow.b.c = 99 -- original.b.c IS now 99

deep_copy(table) sync

Returns a deep, recursive copy of a table. Every nested table is cloned, ensuring total isolation from the original. It safely handles circular references (cycles).

local original = { a = 1, b = { c = 2 } }
local deep = deep_copy(original)
deep.b.c = 99 -- original.b.c remains 2

notify(title, body, [timeout_ms]) async

Sends a desktop notification immediately from Lua. This is a thin wrapper around SpyWeb's native notification sender and can be used from any hook, including error paths that never reach the normal notify stage.

function after_fetch(fetch_result, ctx)
    if not fetch_result.ok then
        notify("Network error", fetch_result.error.message, 10000)
        return nil
    end
    return fetch_result
end

log(message) async

Appends a message to hooks.log inside the job's directory. This is the recommended way to debug hooks without using external libraries.

function after_extract(items, ctx)
    log("Extracted " .. #items .. " items from " .. request.url)
    return items
end

fs_append(filename, content) async

Appends raw content to a file in the job's directory. All writes are non-blocking and safe. Files feature automatic 10MB rotation with timestamped filenames (e.g., data.20260514-143005.csv), keeping a history of 5 files.

function after_extract(items, ctx)
    for _, item in ipairs(items) do
        local row = string.format("%s,%s\n", item.fields.title, item.fields.price)
        fs_append("data.csv", row)
    end
    return items
end

fs_overwrite(filename, content) async

Replaces the entire content of a file in the job's directory. Ideal for keeping a "latest" snapshot of your scraping results. Use shared/ prefix to write to the project root's shared folder.

function after_fetch(fetch_result, ctx)
    if fetch_result.response then
        local etag = fetch_result.response.headers["ETag"]
        if etag then
            fs_overwrite("last_response.json", json_encode({ etag = etag }))
        end
    end
    return fetch_result
end

fs_read(filename) async

Reads a file from the job's directory and returns its content as a string. Returns nil if the file does not exist. Falls back to shared/ if not found locally.

function before_fetch(request, ctx)
    local cached = fs_read("last_response.json")
    if cached then
        local data = json_decode(cached)
        request.headers["If-None-Match"] = data.etag
    end
    return request
end

fs_read_binary(filename) async

Reads a file from the job's directory and returns its content as a binary-safe string (preserving all bytes). Returns nil if the file does not exist. Falls back to shared/ if not found locally. Unlike fs_read, this function handles non-UTF-8 files such as images and other binary formats.

local png_data = fs_read_binary("screenshot.png")
local resp = http_multipart("https://api.example.com/upload", {
    file = { content = png_data, filename = "page.png", type = "image/png" }
})

env_get(key) sync

Retrieves a read-only environment variable from the host system. For security, SpyWeb automatically prepends the SPYWEB_ prefix to any key provided. This allows you to safely store secrets like API tokens outside of your configuration files.

export SPYWEB_API_KEY="secret-123"

function before_fetch(request, ctx)
    local api_key = env_get("API_KEY") -- Reads SPYWEB_API_KEY
    if api_key then
        request.headers["Authorization"] = "Bearer " .. api_key
    end
    return request
end

ctx.selector_matches (Context)

A read-only number representing how many elements on the page matched the main CSS selector defined in your config. This is set automatically after the extraction phase and is available in after_extract and all subsequent hooks via ctx.selector_matches.

function after_extract(items, ctx)
    print("Found " .. #items .. " items (out of " .. ctx.selector_matches .. " selector hits)")
    return items
end

ctx.last_fetch (Context)

A read-only FetchAttempt envelope representing the last network fetch, accessible via ctx.last_fetch. Useful for inspecting fetch results in later hook stages.

Fields:

function after_extract(items, ctx)
    if ctx.last_fetch and ctx.last_fetch.ok then
        log("Fetch was successful: " .. ctx.last_fetch.response.status)
    end
    return items
end

ctx.telemetry (Context)

A read-only table containing performance telemetry for the current scraping cycle, accessible via ctx.telemetry. It is initialized at cycle start and updated dynamically as each stage executes, allowing scripts to inspect stage-by-stage resource usage and execution status.

Fields:

Stage record fields (entry in stages array):

function on_success(ctx)
    if ctx.telemetry then
        log("Job completed in " .. ctx.telemetry.total_duration_ms .. "ms")
        for _, stage in ipairs(ctx.telemetry.stages) do
            if stage.status == "error" then
                log("Stage " .. stage.name .. " failed: " .. tostring(stage.error))
            end
        end
    end
end

Jobs in SpyWeb are configured via TOML. You can use a single jobs.toml or a modular jobs/my-job/config.toml structure.

Job Config

# Shorthand
fields = ["title:h2", "link:a@href"]

# Full form
fields = [
  { name = "title", selector = "h2", att = "text" }
]

The Dual Parser System

SpyWeb employs a hybrid extraction engine to balance extreme performance with maximum compatibility.

Deduplication (hash_fields)

By default, SpyWeb hashes all extracted fields. Use hash_fields to ignore volatile data like "time ago" or "view count":

# Only use the link to determine if an item is new
hash_fields = ["link"]

Advanced Features

[proxy]
enabled = true
rotate = "RoundRobin" # or "Sticky", "Random"
urls = ["socks5://p1:1080", "http://p2:8080"]

[webhook]
enabled = true
url = "https://hooks.example.com/spyweb"
headers = { "Authorization" = "Bearer token" }

[notification]
enabled = true
title = "{item_count} new items in {job_name}"
body = "Title: {title}\nLink: {link}"

SpyWeb runs a built-in web server on 127.0.0.1:7979 which serves both the dashboard UI and a JSON API for querying your scraped records.

Authentication

If the SPYWEB_API_KEY environment variable is set, all /api/* endpoints require the X-SpyWeb-Key header.

curl -H "X-SpyWeb-Key: your_key" http://127.0.0.1:7979/api/jobs

Endpoints

Job List Structure

The /api/jobs endpoint returns an array of objects containing both the logic ID and configuration name.

[
  { "id": "job_1", "name": "Global Job Tracker" },
  { "id": "job_2", "name": "Local Price Monitor" }
]