Fix race condition of out_secondary_file

[daipom]

Which issue(s) this PR fixes:
Fixes #4080

What this PR does / why we need it:
For solving race condition issues of out_secondary_file.

• Note: Similar problem in out_file: Add support for concurrent append in out_file #3808.

Docs Changes:
Not needed.

Release Note:
Same as the title.

[daipom]

I have confirmed this solves #4080.

[daipom]

I have one point of concern.

When confirming this solves #4080, I noticed that a large number of lock files remain during Fluentd running because the files are not deleted after exiting the block.

fluentd/lib/fluent/plugin/base.rb

Lines 79 to 90 in e58fd66

	def acquire_worker_lock(name)
	if @fluentd_lock_dir.nil?
	raise InvalidLockDirectory, "can't acquire lock because FLUENTD_LOCK_DIR isn't set"
	end
	lock_path = get_lock_path(name)
	File.open(lock_path, "w") do |f|
	f.flock(File::LOCK_EX)
	yield
	end
	# Update access time to prevent tmpwatch from deleting a lock file.
	FileUtils.touch(lock_path);
	end

Is this OK?

[daipom]

Test for c95b52a.

Use the same way (append false) and the same environment as

• SecondaryFileOutput has race condition issues in multiple threads or workers #4080 (comment)

Result:

workers	flush_thread_count	percentage (1st)	percentage (2nd)
1	16	0.0% (0/111)	0.0% (0/42)
3	1	0.0% (0/27)	0.0% (0/27)
3	8	0.0% (0/267)	0.0% (0/29)

[daipom]

Test for ce32ac5.

Use the same way (append false) and the same environment as

• SecondaryFileOutput has race condition issues in multiple threads or workers #4080 (comment)

Result:

workers	flush_thread_count	percentage
3	8	0.0% (0/21)

I plan to do more tests at the end of this PR.

[daipom]

I will add some unit tests for the lock feature, although it is hard to test race condition strictly.

[daipom]

@ashie Could you please check if the direction of this fix is OK?

[daipom]

I will test this issue again, and add some tests for this new lock feature.

[daipom]

Test for edcbd4b.

Use the same way (append false) and the same environment as

• SecondaryFileOutput has race condition issues in multiple threads or workers #4080 (comment)

Result:

workers	flush_thread_count	percentage (1st)	percentage (2nd)
1	16	0.0% (0/24)	0.0% (0/39)
3	1	0.0% (0/27)	0.0% (0/21)
3	8	0.0% (0/36)	0.0% (0/33)

[daipom]

I will test this issue again, and add some tests for this new lock feature.

Done.

[daipom]

By the way, might Squash merge be better when merging this?

[ashie]

Thanks!

[daipom]

Thanks for your review! @ashie @kou !