datanode

package
v0.10.3-0...-96d0e78 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 14, 2025 License: Apache-2.0 Imports: 50 Imported by: 0

README

Data Node

DataNode is the component to write insert and delete messages into persistent blob storage, for example MinIO or S3.

Dependency

  • KV store: a kv store that persists messages into blob storage.
  • Message stream: receive messages and publish information
  • Root Coordinator: get the latest unique IDs.
  • Data Coordinator: get the flush information and which message stream to subscribe.

Documentation

Overview

Package datanode implements data persistence logic.

Data node persists insert logs into persistent storage like minIO/S3.

Package datanode implements data persistence logic.

Data node persists insert logs into persistent storage like minIO/S3.

Index

Constants

View Source 
const (
	// ConnectEtcdMaxRetryTime is used to limit the max retry time for connection etcd
	ConnectEtcdMaxRetryTime = 100
)

Variables

View Source 
var Params *paramtable.ComponentParam = paramtable.Get()

Params from config.yaml

Functions

func NewChunkMgrFactory 

func NewChunkMgrFactory() *chunkMgrFactory

func ParseCPluginContext 

func ParseCPluginContext(context []*commonpb.KeyValuePair, collectionID int64) (*indexcgopb.StoragePluginContext, error)

Types

type DataNode 

type DataNode struct {
	Role string
	// contains filtered or unexported fields
}

DataNode communicates with outside services and unioun all services in datanode package.

DataNode implements `types.Component`, `types.DataNode` interfaces. 

`etcdCli`   is a connection of etcd
`rootCoord` is a grpc client of root coordinator.
`dataCoord` is a grpc client of data service.
`stateCode` is current statement of this data node, indicating whether it's healthy.

func NewDataNode 

func NewDataNode(ctx context.Context) *DataNode

NewDataNode will return a DataNode with abnormal state.

func (*DataNode) CheckChannelOperationProgress 

func (node *DataNode) CheckChannelOperationProgress(ctx context.Context, req *datapb.ChannelWatchInfo) (*datapb.ChannelOperationProgressResponse, error)

Deprecated after v2.6.0

func (*DataNode) CompactionV2 

func (node *DataNode) CompactionV2(ctx context.Context, req *datapb.CompactionPlan) (*commonpb.Status, error)

CompactionV2 handles compaction request from DataCoord returns status as long as compaction task enqueued or invalid

func (*DataNode) CreateJob 

func (node *DataNode) CreateJob(ctx context.Context, req *workerpb.CreateJobRequest) (*commonpb.Status, error)

CreateJob is CreateIndex

func (*DataNode) CreateJobV2 deprecated 

func (node *DataNode) CreateJobV2(ctx context.Context, req *workerpb.CreateJobV2Request) (*commonpb.Status, error)

Deprecated: use CreateTask instead, keep for compatibility

func (*DataNode) CreateTask 

func (node *DataNode) CreateTask(ctx context.Context, request *workerpb.CreateTaskRequest) (*commonpb.Status, error)

CreateTask creates different types of tasks based on task type

func (*DataNode) DropCompactionPlan 

func (node *DataNode) DropCompactionPlan(ctx context.Context, req *datapb.DropCompactionPlanRequest) (*commonpb.Status, error)

Not in used now

func (*DataNode) DropImport 

func (node *DataNode) DropImport(ctx context.Context, req *datapb.DropImportRequest) (*commonpb.Status, error)

func (*DataNode) DropJobs 

func (node *DataNode) DropJobs(ctx context.Context, req *workerpb.DropJobsRequest) (*commonpb.Status, error)

func (*DataNode) DropJobsV2 deprecated 

func (node *DataNode) DropJobsV2(ctx context.Context, req *workerpb.DropJobsV2Request) (*commonpb.Status, error)

Deprecated: use DropTask instead, keep for compatibility

func (*DataNode) DropTask 

func (node *DataNode) DropTask(ctx context.Context, request *workerpb.DropTaskRequest) (*commonpb.Status, error)

DropTask deletes specified type of task

func (*DataNode) FlushChannels 

func (node *DataNode) FlushChannels(ctx context.Context, req *datapb.FlushChannelsRequest) (*commonpb.Status, error)

Deprecated after v2.6.0

func (*DataNode) FlushSegments 

func (node *DataNode) FlushSegments(ctx context.Context, req *datapb.FlushSegmentsRequest) (*commonpb.Status, error)

Deprecated after v2.6.0

func (*DataNode) GetAddress 

func (node *DataNode) GetAddress() string

func (*DataNode) GetCompactionState 

func (node *DataNode) GetCompactionState(ctx context.Context, req *datapb.CompactionStateRequest) (*datapb.CompactionStateResponse, error)

GetCompactionState called by DataCoord return status of all compaction plans Deprecated after v2.6.0

func (*DataNode) GetComponentStates 

func (node *DataNode) GetComponentStates(ctx context.Context, req *milvuspb.GetComponentStatesRequest) (*milvuspb.ComponentStates, error)

GetComponentStates will return current state of DataNode

func (*DataNode) GetJobStats 

func (node *DataNode) GetJobStats(ctx context.Context, req *workerpb.GetJobStatsRequest) (*workerpb.GetJobStatsResponse, error)

GetJobStats should be GetSlots

func (*DataNode) GetMetrics 

func (node *DataNode) GetMetrics(ctx context.Context, req *milvuspb.GetMetricsRequest) (*milvuspb.GetMetricsResponse, error)

GetMetrics return datanode metrics

func (*DataNode) GetNodeID 

func (node *DataNode) GetNodeID() int64

func (*DataNode) GetSession 

func (node *DataNode) GetSession() *sessionutil.Session

GetSession to fix data race

func (*DataNode) GetStateCode 

func (node *DataNode) GetStateCode() commonpb.StateCode

GetStateCode return datanode's state code

func (*DataNode) GetStatisticsChannel 

func (node *DataNode) GetStatisticsChannel(ctx context.Context, req *internalpb.GetStatisticsChannelRequest) (*milvuspb.StringResponse, error)

GetStatisticsChannel currently do nothing

func (*DataNode) GetTimeTickChannel 

func (node *DataNode) GetTimeTickChannel(ctx context.Context, req *internalpb.GetTimeTickChannelRequest) (*milvuspb.StringResponse, error)

GetTimeTickChannel currently do nothing

func (*DataNode) ImportV2 

func (node *DataNode) ImportV2(ctx context.Context, req *datapb.ImportRequest) (*commonpb.Status, error)

func (*DataNode) Init 

func (node *DataNode) Init() error

func (*DataNode) NotifyChannelOperation 

func (node *DataNode) NotifyChannelOperation(ctx context.Context, req *datapb.ChannelOperationsRequest) (*commonpb.Status, error)

Deprecated after v2.6.0

func (*DataNode) PreImport 

func (node *DataNode) PreImport(ctx context.Context, req *datapb.PreImportRequest) (*commonpb.Status, error)

func (*DataNode) QueryImport 

func (node *DataNode) QueryImport(ctx context.Context, req *datapb.QueryImportRequest) (*datapb.QueryImportResponse, error)

func (*DataNode) QueryJobs 

func (node *DataNode) QueryJobs(ctx context.Context, req *workerpb.QueryJobsRequest) (*workerpb.QueryJobsResponse, error)

func (*DataNode) QueryJobsV2 deprecated 

func (node *DataNode) QueryJobsV2(ctx context.Context, req *workerpb.QueryJobsV2Request) (*workerpb.QueryJobsV2Response, error)

Deprecated: use QueryTask instead, keep for compatibility

func (*DataNode) QueryPreImport 

func (node *DataNode) QueryPreImport(ctx context.Context, req *datapb.QueryPreImportRequest) (*datapb.QueryPreImportResponse, error)

func (*DataNode) QuerySlot 

func (node *DataNode) QuerySlot(ctx context.Context, req *datapb.QuerySlotRequest) (*datapb.QuerySlotResponse, error)

func (*DataNode) QueryTask 

func (node *DataNode) QueryTask(ctx context.Context, request *workerpb.QueryTaskRequest) (*workerpb.QueryTaskResponse, error)

QueryTask queries task status

func (*DataNode) ReadyToFlush 

func (node *DataNode) ReadyToFlush() error

ReadyToFlush tells whether DataNode is ready for flushing

func (*DataNode) Register 

func (node *DataNode) Register() error

Register register datanode to etcd

func (*DataNode) ResendSegmentStats 

func (node *DataNode) ResendSegmentStats(ctx context.Context, req *datapb.ResendSegmentStatsRequest) (*datapb.ResendSegmentStatsResponse, error)

ResendSegmentStats . ResendSegmentStats resend un-flushed segment stats back upstream to DataCoord by resending DataNode time tick message. It returns a list of segments to be sent. Deprecated in 2.3.2, reversed it just for compatibility during rolling back

func (*DataNode) SetAddress 

func (node *DataNode) SetAddress(address string)

func (*DataNode) SetEtcdClient 

func (node *DataNode) SetEtcdClient(etcdCli *clientv3.Client)

SetEtcdClient sets etcd client for DataNode

func (*DataNode) SetMixCoordClient 

func (node *DataNode) SetMixCoordClient(mixc types.MixCoordClient) error

SetRootCoordClient sets RootCoord's grpc client, error is returned if repeatedly set.

func (*DataNode) SetSession 

func (node *DataNode) SetSession(session *sessionutil.Session)

SetSession to fix data race

func (*DataNode) ShowConfigurations 

func (node *DataNode) ShowConfigurations(ctx context.Context, req *internalpb.ShowConfigurationsRequest) (*internalpb.ShowConfigurationsResponse, error)

ShowConfigurations returns the configurations of DataNode matching req.Pattern

func (*DataNode) Start 

func (node *DataNode) Start() error

Start will update DataNode state to HEALTHY

func (*DataNode) Stop 

func (node *DataNode) Stop() error

Stop will release DataNode resources and shutdown datanode

func (*DataNode) SyncSegments 

func (node *DataNode) SyncSegments(ctx context.Context, req *datapb.SyncSegmentsRequest) (*commonpb.Status, error)

SyncSegments called by DataCoord, sync the compacted segments' meta between DC and DN Deprecated after v2.6.0

func (*DataNode) UpdateStateCode 

func (node *DataNode) UpdateStateCode(code commonpb.StateCode)

UpdateStateCode updates datanode's state code

func (*DataNode) WatchDmChannels 

func (node *DataNode) WatchDmChannels(ctx context.Context, in *datapb.WatchDmChannelsRequest) (*commonpb.Status, error)

WatchDmChannels is not in use

type ResponseWithStatus 

type ResponseWithStatus interface {
	GetStatus() *commonpb.Status
}

type StorageFactory 

type StorageFactory interface {
	NewChunkManager(ctx context.Context, config *indexpb.StorageConfig) (storage.ChunkManager, error)
}

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.