Recently received a case: Big data analysis over there, the data pushed by us here is much less than the same period last year. This is very abnormal because the business has been growing.
于是, I started to follow up with the vines. At first, I found a small problem: after the execution of the scp push file script, I will report an error:
[[email protected] :/data]# sh pushdata.sh
dos2unix: converting file pushdata.log to UNIX format ...
dos2unix: problems renaming './u2dtmp8jwsge' to 'pushdata.log' output file remains in './u2dtmp8jwsge'
dos2unix: problems converting file pushdata.log
originally converted the generated log into Unix format. It stands to reason that it should not affect the push of data, but it is still handled smoothly.
The strange thing is that it is normal to manually execute dos2unix, it seems to have something to do with the script. After reading the script for a long time, I didn't see any obvious mistakes:
, I didn't waste time trying to figure it out. I directly lost Baidu and found that there is no luan. In turn, Google lost and finally found the correct explanation in a foreign forum:
It turns out that when you run the script from cron your current directory (PWD) is set to your home directory. Unix2dos creates the temporary file used in the conversion in your home directory then can't find it to rename it. Not sure why it can't find it.
I added a change directory command (cd) and Unix2dos worked.
probably means that the current execution path of the script in the crontab scheduled task will be set to the home directory by default. Therefore, dos2unix/unix2dos will create a converted temporary file (u2dtmp****) in the home directory, causing the command to find the temporary file in the target path and rename it to the file name being processed. The author indicates that he does not know why. Will not find it. However, the solution he gave is to add the command cd to the path of the log in the script.
For example, in this case, you can add cd /data
at the beginning of the script and find out that it is relieved. In fact, the reason for the problem is very simple: the script executed under
crontab, the default working path is the home directory (manually executed script, the default working path is the current directory). Since the definition of the working path is not included in the script, the default home directory is used as the working path.
dos2unix/unix2dos These two commands work by saving the converted content as a temporary file in the working path, and then renaming the temporary file to the processed file to complete the format conversion. If the working path and the file being processed are not in the same directory, this will result in an error and the generated temporary file will be retained.
So, if this problem exists in crontab, a large number of u2dtmp*** temporary files will be generated in the home directory.
哦了, if you find that dos2unix/unix2dos reported a similar error, it must be that the script does not define a working path. You can add cd to the directory where the file is located before the script and execute dos2unix/unix2dos.
Of course, if we just want to convert the format, we have a variety of alternatives, no need to hang on dos2unix/unix2dos. After all, some systems may not have these 2 commands.
# can be as follows:
sed's/^M//'oldfile>newfile# 注意^M = Ctrl + v, Ctrl + m command to print out the characters
alternative command 2: tr
Well, solve this problem, I should continue to follow up the synchronization problem.