core

package
v1.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 24, 2026 License: MIT Imports: 19 Imported by: 0

Documentation

Overview

Package core defines structure to hold data and relative functions.

Index

Constants

View Source 
const (
	// DefaultChunkSize represents the default chunk size in bytes.
	DefaultChunkSize = 256

	// DefaultSegmentMaxChunks represents the default maximum number of chunks within a segment.
	DefaultSegmentMaxChunks = 1024

	// DefaultSegmentSize represents the default segment size in bytes.
	DefaultSegmentSize = DefaultChunkSize * DefaultSegmentMaxChunks
)
View Source 
const (
	// EncryptionHeaderSize is the size of the encryption header in bytes (1 byte version + 16 bytes nonce).
	EncryptionHeaderSize = 17

	// EncryptionVersion is the current encryption format version.
	EncryptionVersion = 1
)

Variables

View Source 
var (
	EmptyChunk     = make([]byte, DefaultChunkSize)
	EmptyChunkHash = crypto.Keccak256Hash(EmptyChunk)
)
View Source 
var (
	// ErrFileRequired is returned when manipulate on a folder.
	ErrFileRequired = errors.New("file required")

	// ErrFileEmpty is returned when empty file opened.
	ErrFileEmpty = errors.New("file is empty")
)

Functions

func ComputePaddedSize 

func ComputePaddedSize(chunks uint64) (uint64, uint64)

func CryptAt added in v1.3.0 

func CryptAt(key *[32]byte, nonce *[16]byte, offset uint64, data []byte)

CryptAt encrypts or decrypts data in-place at a given byte offset within the plaintext stream. AES-256-CTR is symmetric: encrypt and decrypt are the same operation. The offset is the byte offset within the data stream (not counting the header).

func DecryptFile added in v1.3.0 

func DecryptFile(key *[32]byte, encrypted []byte) ([]byte, error)

DecryptFile decrypts a full downloaded file: strips the header and decrypts the remaining bytes. Returns the decrypted data without the header.

func DecryptFragmentData added in v1.3.0 

func DecryptFragmentData(key *[32]byte, header *EncryptionHeader, fragmentData []byte, isFirstFragment bool, dataOffset uint64) ([]byte, uint64, error)

DecryptFragmentData decrypts a single fragment from a multi-fragment encrypted file. For the first fragment (isFirstFragment=true): strips the encryption header and decrypts the remaining data starting at CTR offset 0. For subsequent fragments: decrypts all bytes using the given dataOffset into the plaintext stream. Returns the decrypted plaintext and the updated cumulative data offset.

func Exists 

func Exists(name string) (bool, error)

func IteratorPaddedSize 

func IteratorPaddedSize(dataSize int64, flowPadding bool) uint64

func MerkleRoot 

func MerkleRoot(filename string) (common.Hash, error)

MerkleRoot returns the merkle root hash of a file on disk

func MerkleTree 

func MerkleTree(data IterableData) (*merkle.Tree, error)

MerkleTree create merkle tree of the data.

func NextPow2 

func NextPow2(input uint64) uint64

func NumSegmentsPadded 

func NumSegmentsPadded(data IterableData) int

NumSegmentsPadded return the number of segments of padded data

func NumSplits 

func NumSplits(total int64, unit int) uint64

func PaddedSegmentRoot 

func PaddedSegmentRoot(segmentIndex uint64, chunks []byte, fileSize int64) (common.Hash, uint64)

PaddedSegmentRoot calculates the Merkle root for a given segment based on its index, the chunk data, and the file size. It handles the logic of padding empty chunks if the segment is the last one and doesn't have enough data to form a complete segment.

func ReadAt 

func ReadAt(data IterableData, readSize int, offset int64, paddedSize uint64) ([]byte, error)

ReadAt read data at specified offset, paddedSize is the size of data after padding.

func SegmentRange 

func SegmentRange(startChunkIndex, fileSize uint64) (startSegmentIndex, endSegmentIndex uint64)

SegmentRange calculates the start and end flow segment index for a file based on the file's start chunk index and file size.

func SegmentRoot 

func SegmentRoot(chunks []byte, emptyChunksPadded ...uint64) common.Hash

SegmentRoot return the merkle root of given chunks

Types

type DataInMemory 

type DataInMemory struct {
	// contains filtered or unexported fields
}

DataInMemory implement of IterableData, the underlying is memory data

func NewDataInMemory 

func NewDataInMemory(data []byte) (*DataInMemory, error)

NewDataInMemory creates DataInMemory from given data

func (*DataInMemory) NumChunks 

func (data *DataInMemory) NumChunks() uint64

func (*DataInMemory) NumSegments 

func (data *DataInMemory) NumSegments() uint64

func (*DataInMemory) Offset 

func (data *DataInMemory) Offset() int64

func (*DataInMemory) PaddedSize 

func (data *DataInMemory) PaddedSize() uint64

func (*DataInMemory) Read 

func (data *DataInMemory) Read(buf []byte, offset int64) (int, error)

func (*DataInMemory) Size 

func (data *DataInMemory) Size() int64

func (*DataInMemory) Split 

func (data *DataInMemory) Split(fragmentSize int64) []IterableData

type EncryptedData added in v1.3.0 

type EncryptedData struct {
	// contains filtered or unexported fields
}

EncryptedData wraps an IterableData with AES-256-CTR encryption. It prepends a 17-byte encryption header (version + nonce) to the data stream and encrypts the inner data on-the-fly during reads.

func NewEncryptedData added in v1.3.0 

func NewEncryptedData(inner IterableData, key [32]byte) (*EncryptedData, error)

NewEncryptedData creates an EncryptedData wrapper around the given data source. A random nonce is generated for the encryption header.

func (*EncryptedData) Header added in v1.3.0 

func (ed *EncryptedData) Header() *EncryptionHeader

Header returns the encryption header containing the version and nonce.

func (*EncryptedData) NumChunks added in v1.3.0 

func (ed *EncryptedData) NumChunks() uint64

func (*EncryptedData) NumSegments added in v1.3.0 

func (ed *EncryptedData) NumSegments() uint64

func (*EncryptedData) Offset added in v1.3.0 

func (ed *EncryptedData) Offset() int64

func (*EncryptedData) PaddedSize added in v1.3.0 

func (ed *EncryptedData) PaddedSize() uint64

func (*EncryptedData) Read added in v1.3.0 

func (ed *EncryptedData) Read(buf []byte, offset int64) (int, error)

Read reads encrypted data at the given offset. For offsets within the header region (0..16), header bytes are returned. For offsets beyond the header, data is read from the inner source and encrypted.

func (*EncryptedData) Size added in v1.3.0 

func (ed *EncryptedData) Size() int64

func (*EncryptedData) Split added in v1.3.0 

func (ed *EncryptedData) Split(fragmentSize int64) []IterableData

Split splits the encrypted data stream into fragments of the given size. Each fragment is an EncryptedDataFragment providing a view into this encrypted stream. If the encrypted data fits within fragmentSize, returns the EncryptedData itself.

type EncryptedDataFragment added in v1.3.0 

type EncryptedDataFragment struct {
	// contains filtered or unexported fields
}

EncryptedDataFragment represents a contiguous view (fragment) of an EncryptedData stream. It delegates Read calls to the parent EncryptedData with an appropriate offset adjustment. This follows the same offset/size view pattern as File.Split() and DataInMemory.Split().

func (*EncryptedDataFragment) NumChunks added in v1.3.0 

func (f *EncryptedDataFragment) NumChunks() uint64

func (*EncryptedDataFragment) NumSegments added in v1.3.0 

func (f *EncryptedDataFragment) NumSegments() uint64

func (*EncryptedDataFragment) Offset added in v1.3.0 

func (f *EncryptedDataFragment) Offset() int64

func (*EncryptedDataFragment) PaddedSize added in v1.3.0 

func (f *EncryptedDataFragment) PaddedSize() uint64

func (*EncryptedDataFragment) Read added in v1.3.0 

func (f *EncryptedDataFragment) Read(buf []byte, offset int64) (int, error)

Read reads from this fragment at the given offset. Delegates to the parent EncryptedData with the fragment's base offset added.

func (*EncryptedDataFragment) Size added in v1.3.0 

func (f *EncryptedDataFragment) Size() int64

func (*EncryptedDataFragment) Split added in v1.3.0 

func (f *EncryptedDataFragment) Split(fragmentSize int64) []IterableData

Split returns the fragment itself (fragments are not further splittable).

type EncryptionHeader added in v1.3.0 

type EncryptionHeader struct {
	Version uint8
	Nonce   [16]byte
}

EncryptionHeader stores the version and nonce for AES-256-CTR encryption.

func NewEncryptionHeader added in v1.3.0 

func NewEncryptionHeader() (*EncryptionHeader, error)

NewEncryptionHeader creates a new encryption header with a random nonce.

func ParseEncryptionHeader added in v1.3.0 

func ParseEncryptionHeader(data []byte) (*EncryptionHeader, error)

ParseEncryptionHeader extracts an encryption header from the given data.

func (*EncryptionHeader) ToBytes added in v1.3.0 

func (h *EncryptionHeader) ToBytes() [EncryptionHeaderSize]byte

ToBytes serializes the header to a fixed-size byte array.

type File 

type File struct {
	os.FileInfo
	// contains filtered or unexported fields
}

File implement of IterableData, the underlying is a file on disk

func Open 

func Open(name string) (*File, error)

Open create a File from a file on disk

func (*File) Close 

func (file *File) Close() error

func (*File) NumChunks 

func (file *File) NumChunks() uint64

func (*File) NumSegments 

func (file *File) NumSegments() uint64

func (*File) Offset 

func (file *File) Offset() int64

func (*File) PaddedSize 

func (file *File) PaddedSize() uint64

func (*File) Read 

func (file *File) Read(buf []byte, offset int64) (int, error)

func (*File) Size 

func (file *File) Size() int64

func (*File) Split 

func (file *File) Split(fragmentSize int64) []IterableData

type Flow 

type Flow struct {
	// contains filtered or unexported fields
}

func NewFlow 

func NewFlow(data IterableData, tags []byte, opts ...common.LogOption) *Flow

func (*Flow) CreateSubmission 

func (flow *Flow) CreateSubmission(submitter ethcommon.Address) (*contract.Submission, error)

type IterableData 

type IterableData interface {
	NumChunks() uint64
	NumSegments() uint64
	Offset() int64
	Size() int64
	PaddedSize() uint64
	Read(buf []byte, offset int64) (int, error)
	Split(fragmentSize int64) []IterableData
}

IterableData defines the data interface to upload to 0g storage network.

type Iterator 

type Iterator interface {
	Next() (bool, error)
	Current() []byte
}

type TreeBuilderInitializer 

type TreeBuilderInitializer struct {
	// contains filtered or unexported fields
}

func (*TreeBuilderInitializer) ParallelCollect 

func (t *TreeBuilderInitializer) ParallelCollect(result *parallel.Result) error

ParallelCollect implements parallel.Interface.

func (*TreeBuilderInitializer) ParallelDo 

func (t *TreeBuilderInitializer) ParallelDo(ctx context.Context, routine int, task int) (interface{}, error)

ParallelDo implements parallel.Interface.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.