Share an entry-level controllable multi-threaded shell scripting solution

Speaking of shell controllable multi-threading, most of the online sharing is a pipeline control scheme. This kind of program, Zhang Ge blog has also been actual combat and shared once: "Shell + Curl website health status check script, grab the Chinese blog alliance lost link site", interested friends can look.

分享一个入门级可控多线程shell脚本方案

The following Zhang Ge blog and another easy-to-understand entry-level controllable multi-threaded shell scripting solution: task cutting, each break.

先来一段 Scene Description:

某日, received this task in the Goose Factory, you need to perform a Ping check on thousands of IPs in the Linux server, as long as you get the IP that can be pinged. it is good. If a single IP goes to ping the test, although it can also complete the task, thousands of IPs are fine, if more?

In view of the simplicity of this case, the first time to give up the previously used pipeline plan, but the use of various broken ideas.

简单思想:

According to the "strategic thinking" of task cutting, I first save these thousands of IPs into an iplist file, then write a split function, divide this file into multiple temporary IP lists, and finally Multithreading can be achieved in a disguised manner by traversing these temporary IP files with multiple threads.

特定码:

Shell
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
#!/bin/sh
#Text splitting function: Split text $1 by number of copies of $2
SplitFile()
{
linenum=`wc-l$1|awk'{print $1}'`
if[[$linenum-le$2]]
then
echo"The lines of this file is less then $2, Are you kidding me..."
exit
fi
Split=`expr$linenum/$2`
Num1=1
FileNum=1
test-dSplitFile||mkdir-pSplitFile
rm-rf SplitFile/*
while[$Num1-lt$linenum]
do
Num2=`expr$Num1+$Split`
sed-n"${Num1}, ${Num2}p "$1>SplitFile/$1-$FileNum
Num1=`expr$Num2+1`
FileNum=`expr$FileNum+1`
done
}
#Define some variables
SPLIT_NUM=${1:-10}# Parameter 1 indicates how many copies are divided, how many threads are opened, default 10
FILE=${2:-iplist}#Parameter 2 indicates segmentation Object, default iplist file
# split file
SplitFile$FILE$SPLIT_NUM
# loop traversal temporary IP file
foriplist in$(ls./SplitFile/*)
do
# loop ping test temporary IP file ip (lost background)
cat$iplist|whilereadip
do
ping-c4-w4$ip>/dev/null&&echo$ip|tee-ai okip.log#ping reachable IP is written to the log
done&# after the while loop is followed by the & symbol, let this nested loop execute
done

in the background to save the code as ping.sh, the process of executing sh ping.sh iplist 100 is as follows :

First cut the iplist into 100 copies, store it in the SplitFile folder

, then read the split files through the for loop and use the while loop to ping the ip in the background.

Since while is throwing the background, the for loop will execute 100 while at a time, which is equivalent to opening 100 threads. The speed is naturally not the same.

中, the number of cuts is the number of multi-threads you want to open. Obviously, the idea of ​​splitting this task is not as high as the pipeline solution, but its idea is more simple and easy to understand, and the versatility is also Better, suitable for entry-level simple multi-threaded tasks.