AST Parsing

Vibe Analyzer uses tree-sitter — an incremental parser that builds a concrete syntax tree (CST) from source code. The CST is then traversed to extract meaningful elements: functions, classes, imports, variables, documentation, and multilingual search tags.

How It Works

Source code → tree-sitter → CST → recursive traversal → AstData (structured data + tags)

Process for each file:

1. Parser selection — the appropriate LanguageParser is chosen from the static PARSERS registry based on file extension
2. CST parsing — tree-sitter builds the tree
3. Recursive traversal — the visit_node function traverses all nodes and collects meaningful elements
4. Post-processing — deduplication, sorting, trimming function/class bodies
5. Tag generation — with_tags() adds multilingual tags (EN/RU/ZH)

Parser Registry

Each parser implements the LanguageParser trait:

pub trait LanguageParser: Send + Sync {
    fn parse(&self, content: &str) -> Result<AstData>;
    fn language_name(&self) -> &'static str;
}

AstData Structure

pub struct AstData {
    pub header_comments: Vec<String>,                 // Module comments
    pub imports: Vec<String>,                         // Imports
    pub variables: Vec<AstDataVariable>,              // Variables/constants
    pub functions: Vec<AstDataFunction>,              // Functions/methods
    pub classes: Vec<AstDataClass>,                   // Classes
    pub structs: Vec<AstDataStruct>,                  // Structs
    pub enums: Vec<AstDataEnum>,                      // Enums
    pub interfaces: Vec<AstDataInterface>,            // Interfaces/traits
    pub frontmatter: Option<HashMap<String, String>>, // Frontmatter (Markdown)
    pub headings: Vec<AstHeading>,                    // Headings (Markdown)
    pub links: Vec<AstLink>,                          // Links (Markdown)
    pub code_blocks: Vec<String>,                     // Code block languages (Markdown)
    pub tags: Vec<String>,                            // Multilingual tags
}

Each element (function, class, etc.) has a signature and doc comments:

pub struct AstDataFunction {
    pub signature: String,     // "pub async fn scan_source(source_path: &Path, ...)"
    pub comments: Vec<String>, // ["Scans a source and returns complete analysis results"]
}

Comment Extraction

Vibe Analyzer distinguishes three types of comments:

1. Header Comments

Describe the purpose of the entire file. Stored in header_comments.

Detection rules:

• The comment is at the beginning of the file (first node in the CST)
• Or all preceding sibling nodes are also comments (is_module_comment)
• Not inside a function or class

Syntax by language:

2. Doc Comments

Describe a specific element (function, class, struct). Stored in the comments field of the corresponding object.

Extraction algorithm:

1. Take the target node (function, class, etc.)
2. Walk backwards through sibling nodes
3. Collect all doc comments, skipping attributes/annotations
4. Stop at the first non-doc node
5. Reverse the list (from farthest to closest)

Syntax by language:

3. Regular Comments

Everything else — //, #, REM, ; — is ignored by the parser.

Signature Extraction

Signatures of functions, classes, and other elements are trimmed at the first delimiter ({, =, ;, :) — the body is not stored.

Language-specific details:

Language-Specific Implementation Details

• Python — does not use child_by_field_name("body") due to a tree-sitter bug; manual traversal is used instead
• Swift — classes, structs, and enums come in a single class_declaration node, distinguished by text
• TypeScript/ArkTS — doc comments for exported elements are looked up on the parent export_statement
• Java — methods inside classes are not extracted as separate top-level functions
• Go — methods with receivers are detected via is_method and are not duplicated
• Kotlin — sealed class is skipped
• Batch — variables inside labels are not extracted
• Markdown — frontmatter is parsed manually, heading previews are cleaned of formatting

AST Export

Results can be exported in several formats:

# AST only
vibe-analyzer scan ast

# With export
vibe-analyzer scan ast --target my-app --format json5 --output analysis.json5

Supported formats: JSON, JSON5, TOON, XML.