Skip to content

Conversation

Watson1978
Copy link
Contributor

@Watson1978 Watson1978 commented Jul 16, 2025

Which issue(s) this PR fixes:
Fixes #

What this PR does / why we need it:
TCP server with server helper does not close all connections at shutdown process.
When receiving data from multiple clients, the server receive the data continuously because the connection is not closed.

The server will shut down properly by this PR

Reproduce

  1. Launch Fluentd with following config file.
  2. Send syslog data from two or more clients using following client script
  3. Terminate Fluentd
  4. Relaunch Fluentd, then it shows the 2025-07-16 14:12:26 +0900 [warn]: #0 restoring buffer file: path = xxxxxxxxx in logs.
  • config
<source>
  @type syslog
  tag system

  <transport tcp>
  </transport>

  bind 0.0.0.0
  port 5140
</source>

<match **>
  @type file
  path "#{File.expand_path('~/tmp/fluentd/maillog')}"
  <buffer>
    @type file
    path "#{File.expand_path('~/tmp/fluentd/buffer/buffer_syslog_maillog')}"
    flush_at_shutdown true
  </buffer>
</match>
  • client script
require 'bundler/inline'
gemfile do
  source 'https://rubygems.org'
  gem 'remote_syslog_sender'
end

def create_client
  Thread.new do
    sender = RemoteSyslogSender.new('127.0.0.1', 5140, protocol: :tcp)
    loop do
      sender.transmit("message body")
      sleep 0.5
    end
  end
end

clients = []
3.times do
  clients << create_client
end

clients.each(&:join)

Example code.

@ary = [1,2,3]

# It would like 1, 2, and 3 to be processed.
# However, following code handles 1 and 3.
@ary.each do |i|
  puts i
  @ary.delete(i)
end

Docs Changes:
Not needed.

Release Note:
Same as the title.

@daipom
Copy link
Contributor

daipom commented Jul 16, 2025

Thanks!

@Watson1978 Watson1978 force-pushed the server branch 4 times, most recently from e75a9b1 to eb4f482 Compare July 16, 2025 08:15
@Watson1978 Watson1978 marked this pull request as ready for review July 16, 2025 08:48
@Watson1978 Watson1978 requested a review from daipom July 16, 2025 08:48
daipom
daipom previously approved these changes Jul 16, 2025
Copy link
Contributor

@daipom daipom left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!
Basically, looks good to me!
I have commented on some minor points about the test code.

@daipom daipom changed the title server: close all connections correctly at shutdown server plugin helper: ensure to close all connections at shutdown Jul 16, 2025
@daipom daipom added this to the v1.19.0 milestone Jul 16, 2025
@daipom daipom added the backport to v1.16 We will backport this fix to the LTS branch label Jul 16, 2025
@daipom
Copy link
Contributor

daipom commented Jul 16, 2025

Thanks!

Copy link
Contributor

@daipom daipom left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@daipom daipom merged commit 11e69f2 into fluent:master Jul 16, 2025
15 of 17 checks passed
@Watson1978 Watson1978 deleted the server branch July 16, 2025 10:18
daipom pushed a commit to daipom/fluentd that referenced this pull request Sep 10, 2025
…uent#5026)

**Which issue(s) this PR fixes**:
Fixes #

**What this PR does / why we need it**:
TCP server with `server` helper does not close all connections at
shutdown process.
When receiving data from multiple clients, the server receive the data
continuously because the connection is not closed.

The server will shut down properly by this PR

### Reproduce
1. Launch Fluentd with following config file.
2. Send syslog data from two or more clients using following client
script
3. Terminate Fluentd
4. Relaunch Fluentd, then it shows the `2025-07-16 14:12:26 +0900
[warn]: #0 restoring buffer file: path = xxxxxxxxx` in logs.

* config

```
<source>
  @type syslog
  tag system

  <transport tcp>
  </transport>

  bind 0.0.0.0
  port 5140
</source>

<match **>
  @type file
  path "#{File.expand_path('~/tmp/fluentd/maillog')}"
  <buffer>
    @type file
    path "#{File.expand_path('~/tmp/fluentd/buffer/buffer_syslog_maillog')}"
    flush_at_shutdown true
  </buffer>
</match>
```

* client script

```ruby
require 'bundler/inline'
gemfile do
  source 'https://rubygems.org'
  gem 'remote_syslog_sender'
end

def create_client
  Thread.new do
    sender = RemoteSyslogSender.new('127.0.0.1', 5140, protocol: :tcp)
    loop do
      sender.transmit("message body")
      sleep 0.5
    end
  end
end

clients = []
3.times do
  clients << create_client
end

clients.each(&:join)
```

Example code.
```
@ary = [1,2,3]

# It would like 1, 2, and 3 to be processed.
# However, following code handles 1 and 3.
@ary.each do |i|
  puts i
  @ary.delete(i)
end
```

**Docs Changes**:
Not needed.

**Release Note**:
Same as the title.

Signed-off-by: Shizuo Fujita <[email protected]>
Signed-off-by: Daijiro Fukuda <[email protected]>
daipom pushed a commit to daipom/fluentd that referenced this pull request Sep 11, 2025
…uent#5026)

**Which issue(s) this PR fixes**:
Fixes #

**What this PR does / why we need it**:
TCP server with `server` helper does not close all connections at
shutdown process.
When receiving data from multiple clients, the server receive the data
continuously because the connection is not closed.

The server will shut down properly by this PR

### Reproduce
1. Launch Fluentd with following config file.
2. Send syslog data from two or more clients using following client
script
3. Terminate Fluentd
4. Relaunch Fluentd, then it shows the `2025-07-16 14:12:26 +0900
[warn]: #0 restoring buffer file: path = xxxxxxxxx` in logs.

* config

```
<source>
  @type syslog
  tag system

  <transport tcp>
  </transport>

  bind 0.0.0.0
  port 5140
</source>

<match **>
  @type file
  path "#{File.expand_path('~/tmp/fluentd/maillog')}"
  <buffer>
    @type file
    path "#{File.expand_path('~/tmp/fluentd/buffer/buffer_syslog_maillog')}"
    flush_at_shutdown true
  </buffer>
</match>
```

* client script

```ruby
require 'bundler/inline'
gemfile do
  source 'https://rubygems.org'
  gem 'remote_syslog_sender'
end

def create_client
  Thread.new do
    sender = RemoteSyslogSender.new('127.0.0.1', 5140, protocol: :tcp)
    loop do
      sender.transmit("message body")
      sleep 0.5
    end
  end
end

clients = []
3.times do
  clients << create_client
end

clients.each(&:join)
```

Example code.
```
@ary = [1,2,3]

# It would like 1, 2, and 3 to be processed.
# However, following code handles 1 and 3.
@ary.each do |i|
  puts i
  @ary.delete(i)
end
```

**Docs Changes**:
Not needed.

**Release Note**:
Same as the title.

Signed-off-by: Shizuo Fujita <[email protected]>
Signed-off-by: Daijiro Fukuda <[email protected]>
daipom pushed a commit to daipom/fluentd that referenced this pull request Sep 11, 2025
…uent#5026)

**Which issue(s) this PR fixes**:
Fixes #

**What this PR does / why we need it**:
TCP server with `server` helper does not close all connections at
shutdown process.
When receiving data from multiple clients, the server receive the data
continuously because the connection is not closed.

The server will shut down properly by this PR

### Reproduce
1. Launch Fluentd with following config file.
2. Send syslog data from two or more clients using following client
script
3. Terminate Fluentd
4. Relaunch Fluentd, then it shows the `2025-07-16 14:12:26 +0900
[warn]: #0 restoring buffer file: path = xxxxxxxxx` in logs.

* config

```
<source>
  @type syslog
  tag system

  <transport tcp>
  </transport>

  bind 0.0.0.0
  port 5140
</source>

<match **>
  @type file
  path "#{File.expand_path('~/tmp/fluentd/maillog')}"
  <buffer>
    @type file
    path "#{File.expand_path('~/tmp/fluentd/buffer/buffer_syslog_maillog')}"
    flush_at_shutdown true
  </buffer>
</match>
```

* client script

```ruby
require 'bundler/inline'
gemfile do
  source 'https://rubygems.org'
  gem 'remote_syslog_sender'
end

def create_client
  Thread.new do
    sender = RemoteSyslogSender.new('127.0.0.1', 5140, protocol: :tcp)
    loop do
      sender.transmit("message body")
      sleep 0.5
    end
  end
end

clients = []
3.times do
  clients << create_client
end

clients.each(&:join)
```

Example code.
```
@ary = [1,2,3]

# It would like 1, 2, and 3 to be processed.
# However, following code handles 1 and 3.
@ary.each do |i|
  puts i
  @ary.delete(i)
end
```

**Docs Changes**:
Not needed.

**Release Note**:
Same as the title.

Signed-off-by: Shizuo Fujita <[email protected]>
Signed-off-by: Daijiro Fukuda <[email protected]>
daipom added a commit that referenced this pull request Sep 11, 2025
…s at shutdown (#5026) (#5088)

**Which issue(s) this PR fixes**:
* Backport #5026

**What this PR does / why we need it**:
TCP server with `server` helper does not close all connections at
shutdown process.
When receiving data from multiple clients, the server receive the data
continuously because the connection is not closed.

The server will shut down properly by this PR

### Reproduce
1. Launch Fluentd with following config file.
2. Send syslog data from two or more clients using following client
script
3. Terminate Fluentd
4. Relaunch Fluentd, then it shows the `2025-07-16 14:12:26 +0900
[warn]: #0 restoring buffer file: path = xxxxxxxxx` in logs.

* config

```
<source>
  @type syslog
  tag system

  <transport tcp>
  </transport>

  bind 0.0.0.0
  port 5140
</source>

<match **>
  @type file
  path "#{File.expand_path('~/tmp/fluentd/maillog')}"
  <buffer>
    @type file
    path "#{File.expand_path('~/tmp/fluentd/buffer/buffer_syslog_maillog')}"
    flush_at_shutdown true
  </buffer>
</match>
```

* client script

```ruby
require 'bundler/inline'
gemfile do
  source 'https://rubygems.org'
  gem 'remote_syslog_sender'
end

def create_client
  Thread.new do
    sender = RemoteSyslogSender.new('127.0.0.1', 5140, protocol: :tcp)
    loop do
      sender.transmit("message body")
      sleep 0.5
    end
  end
end

clients = []
3.times do
  clients << create_client
end

clients.each(&:join)
```

Example code.
```
@ary = [1,2,3]

# It would like 1, 2, and 3 to be processed.
# However, following code handles 1 and 3.
@ary.each do |i|
  puts i
  @ary.delete(i)
end
```

**Docs Changes**:
Not needed.

**Release Note**:
Same as the title.

Signed-off-by: Shizuo Fujita <[email protected]>
Signed-off-by: Daijiro Fukuda <[email protected]>
Co-authored-by: Shizuo Fujita <[email protected]>
@daipom daipom added the backported "backport to LTS" is done label Sep 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport to v1.16 We will backport this fix to the LTS branch backported "backport to LTS" is done
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants