README
¶
sibyl 2
Take a quick snapshot of your codebase in seconds, with zero cost.
What's it?
sibyl2 is a static code analyzer, for extracting meta-data (functions/classes/parameters/lines ...) from raw source code. Inspired by semantic of GitHub.
- Easy to use
- Fast enough
- Extensible
- Multiple languages in one (Go/Java/Python ...)
It can translate raw source code:
func ExtractFunction(targetFile string, config *ExtractConfig) ([]*extractor.FunctionFileResult, error) {
// ...
}
to logical structure：
all you need is a one-line command:
./sibyl extract --src . --output hello.json
and export them with different formats.
... JSON format for your custom operations
{
"path": "extract.go",
"language": "GOLANG",
"type": "func",
"units": [
{
"name": "ExtractFunction",
"receiver": "",
"parameters": [
{
"type": "string",
"name": "targetFile"
},
{
"type": "*ExtractConfig",
"name": "config"
}
],
"returns": [
{
"type": "[]*extractor.FunctionFileResult",
"name": ""
},
{
"type": "error",
"name": ""
}
],
"span": {
"start": {
"row": 18,
"column": 0
},
"end": {
"row": 46,
"column": 1
}
},
"extras": null
}
]
}
... And multiple languages supported
public class Java8SnapshotListener extends Java8MethodLayerListener<Method> {
@Override
public void enterMethodDeclarationWithoutMethodBody(
Java8Parser.MethodDeclarationWithoutMethodBodyContext ctx) {
super.enterMethodDeclarationWithoutMethodBody(ctx);
this.storage.save(curMethodStack.peekLast());
}
}
after:
{
"name": "enterMethodDeclarationWithoutMethodBody",
"receiver": "com.williamfzc.sibyl.core.listener.java8.Java8SnapshotListener",
"parameters": [{
"type": "Java8Parser.MethodDeclarationWithoutMethodBodyContext",
"name": "ctx"
}],
"returns": [{
"type": "void",
"name": ""
}],
"span": {
"start": {
"row": 8,
"column": 4
},
"end": {
"row": 13,
"column": 5
}
},
"extras": {
"annotations": ["@Override"]
}
}
Usage examples
Developers can easily combine it with any other tools. We have already built some cmd tools in use. You can download from the release page.
Basic Functions
./sibyl extract --src . --output hello.json
You will see:
$ ./sibyl extract --src . --output hello.json
{"level":"info","ts":1670138890.5306911,"caller":"sibyl2/extract_fs.go:92","msg":"no specific lang found, do the guess in: /Users/fengzhangchi/github_workspace/sibyl2"}
{"level":"info","ts":1670138890.5596569,"caller":"sibyl2/extract_fs.go:97","msg":"I think it is: GOLANG"}
{"level":"info","ts":1670138890.5836658,"caller":"core/runner.go:22","msg":"valid file count: 55"}
{"level":"info","ts":1670138890.6657321,"caller":"sibyl2/extract_fs.go:76","msg":"cost: 135 ms"}
{"level":"info","ts":1670138890.669896,"caller":"extract/cmd_extract.go:60","msg":"file has been saved to: hello.json"}
... Result will be generated in seconds.
[
{
"path": "analyze.go",
"language": "GOLANG",
"type": "func",
"units": [
{
"name": "AnalyzeFuncGraph",
"receiver": "",
"parameters": [
{
"type": "[]*extractor.FunctionFileResult",
"name": "funcFiles"
},
{
"type": "[]*extractor.SymbolFileResult",
"name": "symbolFiles"
}
],
"returns": [
{
"type": "*FuncGraph",
"name": ""
},
{
"type": "error",
"name": ""
}
],
"span": {
"start": {
"row": 11,
"column": 0
},
"end": {
"row": 80,
"column": 1
}
},
"extras": {}
}
]
},
...
]
Source Code History Visualization
Source code history visualization, inspired by https://github.com/acaudwell/Gource
One line command to see how your repository grow up, with no heavy dependencies like OpenGL, with logic level messages.
./sibyl history --src . --output hello.html --full
You can remove the
fullflag for better performance.
Smart Git Diff
Normal git diff has only text level messages.
./sibyl diff --from HEAD~1 --to HEAD --output hello1.json
And you can get a structural one with sibyl, which contains method level messages and callgraphs.
{
"fragments": [
{
"path": "pkg/server/admin_s.go",
"functions": [
{
"name": "HandleStatusUpload",
"receiver": "",
"parameters": [
{
"type": "*gin.Context",
"name": "c"
}
],
"returns": null,
"span": {
"start": {
"row": 17,
"column": 0
},
"end": {
"row": 23,
"column": 1
}
},
"extras": {},
"path": "pkg/server/admin_s.go",
"language": "GOLANG",
"calls": null,
"reverseCalls": [
{
"name": "Execute",
"receiver": "",
"parameters": [
{
"type": "ExecuteConfig",
"name": "config"
}
],
"returns": null,
"span": {
"start": {
"row": 67,
"column": 0
},
"end": {
"row": 96,
"column": 1
}
},
"extras": {},
"path": "pkg/server/app.go",
"language": "GOLANG"
}
]
}
]
},
...
You can easily build some smart test tools above it.
For example, Google 's unittest speed up:
PaaS (WIP)
About how we use in real DevOps workflow. It offered a shared "code snapshot" layer for different any other tools. Documentation is on the way.
Principle
Many devtools required meta-data inside source code for further processing. For example, code coverage, call graph, etc.
Without a cross-languages shared lib, we have to re-implement this extract layer repeatedly for each tool, each language.
This repo aims at offering a general use shared lib for source code meta-data.
Performance
We have tested it on some famous repos, like guava. And that's why we can say it is "fast enough".
See https://github.com/williamfzc/sibyl2/actions/workflows/perf.yml for details.
Contribution
This project split into 3 main parts:
- /cmd: Pure command line tool for general usage
- /pkg/server: All-in-one service for production
- others: Shared api and core
Workflow:
- core: collect files and convert them to
Unit. - extract: classify and process units into functions, symbols.
- api: higher level analyze like callgraph
Issues / PRs are welcome!
Refs
- basic grammar: https://tree-sitter.github.io/tree-sitter/creating-parsers#the-grammar-dsl
- language parser (for example, golang): https://github.com/tree-sitter/tree-sitter-go/blob/master/src/parser.c
- symbol: https://github.com/github/semantic/blob/main/docs/examples.md#symbols
- stack graphs: https://github.blog/2021-12-09-introducing-stack-graphs/
License
Apache License Version 2.0, see LICENSE
Documentation
¶
Index ¶
- func Extract(targetFile string, config *ExtractConfig) ([]*extractor.FileResult, error)
- func ExtractFromBytes(content []byte, config *ExtractConfig) (*extractor.FileResult, error)
- func ExtractFromString(content string, config *ExtractConfig) (*extractor.FileResult, error)
- func ExtractFunction(targetFile string, config *ExtractConfig) ([]*extractor.FunctionFileResult, error)
- func ExtractSymbol(targetFile string, config *ExtractConfig) ([]*extractor.SymbolFileResult, error)
- func QueryUnitsByIndexNames[T extractor.DataType](result *extractor.BaseFileResult[T], indexNames ...string) []T
- func QueryUnitsByIndexNamesInFiles[T extractor.DataType](result []*extractor.BaseFileResult[T], indexNames ...string) []T
- func QueryUnitsByLines[T extractor.DataType](result *extractor.BaseFileResult[T], lines ...int) []T
- type AdjacencyMapType
- type ExtractConfig
- type FuncGraph
- func (fg *FuncGraph) FindCalls(f *extractor.Function) []*FunctionWithPath
- func (fg *FuncGraph) FindRelated(f *extractor.Function) *FunctionContext
- func (fg *FuncGraph) FindReverseCalls(f *extractor.Function) []*FunctionWithPath
- func (fg *FuncGraph) WrapFuncWithPath(f *extractor.Function) (*FunctionWithPath, error)
- type FuncGraphType
- type FunctionContext
- type FunctionWithPath
- type SymbolWithPath
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Extract ¶
func Extract(targetFile string, config *ExtractConfig) ([]*extractor.FileResult, error)
func ExtractFromBytes ¶
func ExtractFromBytes(content []byte, config *ExtractConfig) (*extractor.FileResult, error)
func ExtractFromString ¶
func ExtractFromString(content string, config *ExtractConfig) (*extractor.FileResult, error)
func ExtractFunction ¶
func ExtractFunction(targetFile string, config *ExtractConfig) ([]*extractor.FunctionFileResult, error)
func ExtractSymbol ¶
func ExtractSymbol(targetFile string, config *ExtractConfig) ([]*extractor.SymbolFileResult, error)
func QueryUnitsByIndexNames ¶
func QueryUnitsByIndexNames[T extractor.DataType](result *extractor.BaseFileResult[T], indexNames ...string) []T
func QueryUnitsByIndexNamesInFiles ¶
func QueryUnitsByIndexNamesInFiles[T extractor.DataType](result []*extractor.BaseFileResult[T], indexNames ...string) []T
func QueryUnitsByLines ¶
func QueryUnitsByLines[T extractor.DataType](result *extractor.BaseFileResult[T], lines ...int) []T
Types ¶
type ExtractConfig ¶
type ExtractConfig struct {
LangType core.LangType
ExtractType extractor.ExtractType
FileFilter func(path string) bool
}
ExtractConfig todo: should not use config ptr for parallel running
func DefaultConfig ¶
func DefaultConfig() *ExtractConfig
type FuncGraph ¶
type FuncGraph struct {
ReverseCallGraph *FuncGraphType
CallGraph *FuncGraphType
}
FuncGraph
It is not a serious `call` graph. It based on references not real calls.
Why we used it: - We can still use something like `method_invocation` - But we mainly use it to evaluate the influence of a method - In many languages, scope of `invocation` is too small - For example, use `function` as a parameter.
func AnalyzeFuncGraph ¶
func AnalyzeFuncGraph(funcFiles []*extractor.FunctionFileResult, symbolFiles []*extractor.SymbolFileResult) (*FuncGraph, error)
func (*FuncGraph) FindCalls ¶
func (fg *FuncGraph) FindCalls(f *extractor.Function) []*FunctionWithPath
func (*FuncGraph) FindRelated ¶
func (fg *FuncGraph) FindRelated(f *extractor.Function) *FunctionContext
func (*FuncGraph) FindReverseCalls ¶
func (fg *FuncGraph) FindReverseCalls(f *extractor.Function) []*FunctionWithPath
func (*FuncGraph) WrapFuncWithPath ¶
func (fg *FuncGraph) WrapFuncWithPath(f *extractor.Function) (*FunctionWithPath, error)
type FuncGraphType ¶
type FuncGraphType struct {
graph.Graph[string, *FunctionWithPath]
// contains filtered or unexported fields
}
func WrapFuncGraph ¶
func WrapFuncGraph(g graph.Graph[string, *FunctionWithPath]) *FuncGraphType
func (*FuncGraphType) GetAdjacencyMap ¶
func (fgt *FuncGraphType) GetAdjacencyMap() (*AdjacencyMapType, error)
type FunctionContext ¶
type FunctionContext struct {
*FunctionWithPath
Calls []*FunctionWithPath `json:"calls"`
ReverseCalls []*FunctionWithPath `json:"reverseCalls"`
}
func (*FunctionContext) ToGraph ¶
func (f *FunctionContext) ToGraph() *FuncGraphType
type FunctionWithPath ¶
type FunctionWithPath struct {
*extractor.Function
Path string `json:"path"`
Language core.LangType `json:"language"`
}
FunctionWithPath original symbol and function do not have a path because they maybe not come from a real file
Source Files
Directories