Get configuration by items failed- all config files not exists

[lenguyenphuhuuo]

My config file located in configs/cdc.yaml but when i run the milvus-cdc, i always get an error that can't load the config file

root@51abd64f0b5e:/app# ./milvus-cdc
[2025/09/11 05:48:53.740 +00:00] [INFO] [tag/tag.go:34] ["base info"] [BuildTime="Fri Jun 27 02:12:12 UTC 2025"] [GitCommit=b85f933] [GoVersion="go version go1.24.2 linux/amd64"]
[2025/09/11 05:48:53.741 +00:00] [INFO] [config/manager.go:331] ["Get configuration by items failed"] [error="all config files not exists, files: []"] [errorVerbose="all config files not exists, files: []\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/v2/config.(*FileSource).loadFromFile\n  | \t/go/pkg/mod/github.com/!sim!f!g/milvus/pkg/v2@v2.0.0-20250519121918-b4396ba475bc/config/file_source.go:156\n  | github.com/milvus-io/milvus/pkg/v2/config.(*FileSource).GetConfigurations\n  | \t/go/pkg/mod/github.com/!sim!f!g/milvus/pkg/v2@v2.0.0-20250519121918-b4396ba475bc/config/file_source.go:67\n  | github.com/milvus-io/milvus/pkg/v2/config.(*Manager).pullSourceConfigs\n  | \t/go/pkg/mod/github.com/!sim!f!g/milvus/pkg/v2@v2.0.0-20250519121918-b4396ba475bc/config/manager.go:329\n  | github.com/milvus-io/milvus/pkg/v2/config.(*Manager).AddSource\n  | \t/go/pkg/mod/github.com/!sim!f!g/milvus/pkg/v2@v2.0.0-20250519121918-b4396ba475bc/config/manager.go:274\n  | github.com/milvus-io/milvus/pkg/v2/util/paramtable.(*BaseTable).initConfigsFromLocal\n  | \t/go/pkg/mod/github.com/!sim!f!g/milvus/pkg/v2@v2.0.0-20250519121918-b4396ba475bc/util/paramtable/base_table.go:176\n  | github.com/milvus-io/milvus/pkg/v2/util/paramtable.(*BaseTable).init\n  | \t/go/pkg/mod/github.com/!sim!f!g/milvus/pkg/v2@v2.0.0-20250519121918-b4396ba475bc/util/paramtable/base_table.go:154\n  | github.com/milvus-io/milvus/pkg/v2/util/paramtable.NewBaseTable\n  | \t/go/pkg/mod/github.com/!sim!f!g/milvus/pkg/v2@v2.0.0-20250519121918-b4396ba475bc/util/paramtable/base_table.go:131\n  | github.com/zilliztech/milvus-cdc/core/util.InitMilvusPkgParam\n  | \t/app/core/util/milvus_param.go:28\n  | main.main\n  | \t/app/server/main/main.go:50\n  | runtime.main\n  | \t/usr/local/go/src/runtime/proc.go:283\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1700\nWraps: (2) all config files not exists, files: []\nError types: (1) *withstack.withStack (2) *errutil.leafError"]
[2025/09/11 05:48:53.741 +00:00] [WARN] [paramtable/base_table.go:181] ["init baseTable with file failed"] [configFile="[milvus.yaml,_test.yaml,default.yaml,user.yaml]"] [error="failed to load FileSource cause: 616c6c20636f6e6669672066696c6573206e6f74206578697374732c2066696c65733a205b5d"]
[2025/09/11 05:48:53.742 +00:00] [INFO] [paramtable/component_param.go:5098] ["DeployModeEnv is not set, use default"] [default=0.5]
[2025/09/11 05:48:53.743 +00:00] [INFO] [config/manager.go:331] ["Get configuration by items failed"] [error="all config files not exists, files: []"] [errorVerbose="all config files not exists, files: []\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/v2/config.(*FileSource).loadFromFile\n  | \t/go/pkg/mod/github.com/!sim!f!g/milvus/pkg/v2@v2.0.0-20250519121918-b4396ba475bc/config/file_source.go:156\n  | github.com/milvus-io/milvus/pkg/v2/config.(*FileSource).GetConfigurations\n  | \t/go/pkg/mod/github.com/!sim!f!g/milvus/pkg/v2@v2.0.0-20250519121918-b4396ba475bc/config/file_source.go:67\n  | github.com/milvus-io/milvus/pkg/v2/config.(*Manager).pullSourceConfigs\n  | \t/go/pkg/mod/github.com/!sim!f!g/milvus/pkg/v2@v2.0.0-20250519121918-b4396ba475bc/config/manager.go:329\n  | github.com/milvus-io/milvus/pkg/v2/config.(*Manager).AddSource\n  | \t/go/pkg/mod/github.com/!sim!f!g/milvus/pkg/v2@v2.0.0-20250519121918-b4396ba475bc/config/manager.go:274\n  | github.com/milvus-io/milvus/pkg/v2/util/paramtable.(*BaseTable).initConfigsFromLocal\n  | \t/go/pkg/mod/github.com/!sim!f!g/milvus/pkg/v2@v2.0.0-20250519121918-b4396ba475bc/util/paramtable/base_table.go:176\n  | github.com/milvus-io/milvus/pkg/v2/util/paramtable.(*BaseTable).init\n  | \t/go/pkg/mod/github.com/!sim!f!g/milvus/pkg/v2@v2.0.0-20250519121918-b4396ba475bc/util/paramtable/base_table.go:154\n  | github.com/milvus-io/milvus/pkg/v2/util/paramtable.NewBaseTable\n  | \t/go/pkg/mod/github.com/!sim!f!g/milvus/pkg/v2@v2.0.0-20250519121918-b4396ba475bc/util/paramtable/base_table.go:131\n  | github.com/milvus-io/milvus/pkg/v2/util/paramtable.NewBaseTableFromYamlOnly\n  | \t/go/pkg/mod/github.com/!sim!f!g/milvus/pkg/v2@v2.0.0-20250519121918-b4396ba475bc/util/paramtable/base_table.go:116\n  | github.com/milvus-io/milvus/pkg/v2/util/paramtable.InitWithBaseTable.func1\n  | \t/go/pkg/mod/github.com/!sim!f!g/milvus/pkg/v2@v2.0.0-20250519121918-b4396ba475bc/util/paramtable/runtime.go:48\n  | sync.(*Once).doSlow\n  | \t/usr/local/go/src/sync/once.go:78\n  | sync.(*Once).Do\n  | \t/usr/local/go/src/sync/once.go:69\n  | github.com/milvus-io/milvus/pkg/v2/util/paramtable.InitWithBaseTable\n  | \t/go/pkg/mod/github.com/!sim!f!g/milvus/pkg/v2@v2.0.0-20250519121918-b4396ba475bc/util/paramtable/runtime.go:46\n  | github.com/zilliztech/milvus-cdc/core/util.InitMilvusPkgParam\n  | \t/app/core/util/milvus_param.go:33\n  | main.main\n  | \t/app/server/main/main.go:50\n  | runtime.main\n  | \t/usr/local/go/src/runtime/proc.go:283\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1700\nWraps: (2) all config files not exists, files: []\nError types: (1) *withstack.withStack (2) *errutil.leafError"]
[2025/09/11 05:48:53.743 +00:00] [WARN] [paramtable/base_table.go:181] ["init baseTable with file failed"] [configFile="[hook.yaml]"] [error="failed to load FileSource cause: 616c6c20636f6e6669672066696c6573206e6f74206578697374732c2066696c65733a205b5d"]
[2025/09/11 05:48:53.743 +00:00] [INFO] [paramtable/hook_config.go:21] ["hook config"] [hook=null]
[2025/09/11 05:48:53.753 +00:00] [INFO] [msgstream/mq_msgstream.go:125] ["Msg Stream state"] [can_produce=true]
[2025/09/11 05:48:53.763 +00:00] [WARN] [retry/retry.go:46] ["retry func failed"] [retried=0] [error="connection error"]
[2025/09/11 05:48:56.774 +00:00] [WARN] [retry/retry.go:46] ["retry func failed"] [retried=4] [error="connection error"]
[2025/09/11 05:48:56.775 +00:00] [WARN] [retry/retry.go:72] ["retry func failed, deadline"] [retried=4] [attempt=20] [isContextErr=false]
panic: Failed to create consumer by-dev-replicate-msg, errors = connection error

goroutine 1 [running]:
github.com/milvus-io/milvus/pkg/v2/mq/msgstream.(*mqMsgStream).AsConsumer(0xc0004dec00, {0x1f98328, 0xc0005e9ab0}, {0xc0008a8d20?, 0x2e94100?, 0xc0008a84a0?}, {0xc000129a70, 0x27}, 0x0)
	/go/pkg/mod/github.com/!sim!f!g/milvus/pkg/v2@v2.0.0-20250519121918-b4396ba475bc/mq/msgstream/mq_msgstream.go:213 +0x5f4
github.com/zilliztech/milvus-cdc/core/reader.getStream({0x1f98328, 0xc0005e9ab0}, {0x1f90928?, 0xc000137758?}, {0xc000157260, 0x1d}, 0x0)
	/app/core/reader/stream_creator.go:118 +0x3b3
github.com/zilliztech/milvus-cdc/core/reader.(*DisptachClientStreamCreator).CheckConnection(0x1d318eb?, {0x1f98328?, 0xc0005e9ab0?}, {0xc000157260?, 0x2?}, 0xc00011d9b4?)
	/app/core/reader/stream_creator.go:185 +0x3e
github.com/zilliztech/milvus-cdc/server.(*MetaCDC).checkMQConnection(0xc000159130)
	/app/server/cdc_impl.go:180 +0x252
github.com/zilliztech/milvus-cdc/server.NewMetaCDC(0xc0004ebb88)
	/app/server/cdc_impl.go:154 +0x872
github.com/zilliztech/milvus-cdc/server.GetCDCAPI(...)
	/app/server/cdc_api.go:89
github.com/zilliztech/milvus-cdc/server.(*CDCServer).Run(0xc00077f6c8, 0xc0004ebb88)
	/app/server/server.go:54 +0xc6
main.main()
	/app/server/main/main.go:58 +0x489

here is my cdc.yaml

 cat configs/cdc.yaml
address: 0.0.0.0:8444
maxTaskNum: 100

metaStoreConfig:
  storeType: etcd
  etcd:
    address:
      - cdc-etcd:2379
    enableAuth: false
    username: root
    password: root123456
    enableTLS: false
  rootPath: cdc-by-dev

sourceConfig:
  etcd:
    address:
      - etcd:2379
    rootPath: by-dev
    metaSubPath: meta
    enableAuth: false
    username: root
    password: root123456
    enableTLS: false
  readChanLen: 4
  defaultPartitionName: _default
  replicateChan: by-dev-replicate-msg
  pulsar:
    address: pulsar://broker:6650
    webAddress: http://broker:8080
    maxMessageSize: 5242880
    tenant: public
    namespace: default

maxNameLength: 256
logLevel: info
detectDeadLock: false
batchMode: true

and all the file in configs

total 60
drwxr-xr-x 1 root root  4096 Sep 11 05:41 .
drwxr-xr-x 1 root root  4096 Sep 11 05:35 ..
-rw-r--r-- 1 root root 29549 Jun 27 02:11 cdc-grafana.json
-rwxrwxrwx 1 root root   721 Sep 11 04:48 cdc.yaml
-rw-r--r-- 1 root root   109 Jun 27 02:11 collection_start_position.yaml
-rw-r--r-- 1 root root   577 Jun 27 02:11 data_diff.yaml
-rw-r--r-- 1 root root   651 Jun 27 02:11 msg_count.yaml

[SimFG]

You need check the pulsar connection state.

[lenguyenphuhuuo]

You need check the pulsar connection state.

Hi, thanks for your reply

DNS fwd/rev mismatch: broker != broker.standalone-pulsar_milvus-pulsar
broker [172.18.0.7] 6650 (?) open

i see the port is open

and the log in broker

2025-09-11T05:51:03,527+0000 [pulsar-load-manager-1-1] INFO  org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl - Only 1 broker available: no load shedding will be performed
2025-09-11T05:52:03,527+0000 [pulsar-load-manager-1-1] INFO  org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl - Only 1 broker available: no load shedding will be performed

[SimFG]

I think the address (pulsar://broker:6650) can't be connected. you need check it

[lenguyenphuhuuo]

I think the address (pulsar://broker:6650) can't be connected. you need check it

yes, i've checked by netcat and it works, what is the tool should be here?