When Updating large lists use the parallel call for SED

smokingwheels · August 7, 2021, 5:38pm

It only uses one core at mostly 100% for some time to update.
Just an example I have no idea where the code is.

cat bigfile.txt | parallel --pipe sed s^old^new^g

https://huboqiang.cn/2016/05/09/multiCore

jfb · August 7, 2021, 6:34pm

How would such a change affect the availability of Pi-hole during the gravity update process? Pi-hole processes gravity in the background, and only when the new gravity database is being swapped for the old one is Pi-hole offline (and that is a very short duration).

I don't understand the use case here.

smokingwheels · August 7, 2021, 6:47pm

The FTL runs at nice -10, would it keep running during the SED operation in parallel ?
To update my gravity takes approx 45 mins, I can time it again on a 3B.

smokingwheels · August 7, 2021, 7:35pm

I opened 2 terminals.
It takes 30.5 mins to do gravity update.
Elapsed time
The lists download 5 min
SED 4 cores 6 min
SED 1 core 10 min
Grep runs 11 min
SQLite 30.5 min
Done

jfb · August 7, 2021, 9:49pm

The problem is not with the gravity update process. You are downloading and processing over 10 million domains, and that's always going to take a while. A Pi-3B isn't a particularly powerful CPU.

yubiuser · August 9, 2021, 10:19am

I've tried parallel with this line

github.com

pi-hole/pi-hole/blob/4736e03108763cc2d5659f48d8a1e8a64d9b2608/gravity.sh#L514

    
      
          
          
total_num=0
          num_lines=0
          num_invalid=0
          parseList() {
            local adlistID="${1}" src="${2}" target="${3}" incorrect_lines
            # This sed does the following things:
            # 1. Remove all domains containing invalid characters. Valid are: a-z, A-Z, 0-9, dot (.), minus (-), underscore (_)
            # 2. Append ,adlistID to every line
            # 3. Ensures there is a newline on the last line
            sed -e "/[^a-zA-Z0-9.\_-]/d;s/$/,${adlistID}/;/.$/a\\" "${src}" >> "${target}"
            # Find (up to) five domains containing invalid characters (see above)
            incorrect_lines="$(sed -e "/[^a-zA-Z0-9.\_-]/!d" "${src}" | head -n 5)"
          
          
  local num_target_lines num_correct_lines num_invalid
            # Get number of lines in source file
            num_lines="$(grep -c "^" "${src}")"
            # Get number of lines in destination file
            num_target_lines="$(grep -c "^" "${target}")"
            num_correct_lines="$(( num_target_lines-total_num ))"
            total_num="$num_target_lines"

__
Results:

Standard

real	1m20,742s
user	1m2,293s
sys	0m10,478s

With parallel

real	1m23,278s
user	1m13,053s
sys	0m16,753s

There might be some special parallel option to improve speed, but without tweaking it is actually slower. When considering that analyzing the lists (which are usually relatively small) is fast compared to the subsequent indexing of the gravity database, it's not worth the effort.

system · February 5, 2022, 10:19am

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.