Collection of useful syntax to clean wordlists. ## Remove the space character with sed `# sed -i 's/ //g' file.txt` OR `# egrep -v "^[[:space:]]*quot; file.txt` ## Remove the last space character with sed `# sed -i s/.$// file.txt` ## Sorting Wordlists by Length `# awk '{print length, $0}' rockyou.txt | sort -n | cut -d " " -f2- > rockyou_length-list.txt` ## Convert uppercase to lowercase and the opposite ```undefined # tr [A-Z] [a-z] < file.txt > lower-case.txt # tr [a-z] [A-Z] < file.txt > upper-case.txt ``` ## Remove blank lines with sed `# sed -i '/^$/d' List.txt` ## Remove defined character with sed `# sed -i "s/'//" file.txt` ## Delete a string with sed `# echo 'This is a foo test' | sed -e 's/<foo>//g'` ## Replace characters with tr `# tr '@' '#' < emails.txt` OR `# sed 's/@/#' file.txt` ## Print specific columns with awk `# awk -F "," '{print $3}' infile.csv > outfile.csv` OR `# cut -d "," -f 3 infile.csv > outfile.csv` > **Note:** if you want to isolate all columns after column 3 use `# cut -d "," -f 3- infile.csv > outfile.csv` ## Generate Random Passwords with urandom ```undefined # tr -dc 'a-zA-Z0-9._!@#$%^&*()' < /dev/urandom | fold -w 8 | head -n 500000 > wordlist.txt # tr -dc 'a-zA-Z0-9-_!@#$%^&*()_+{}|:<>?=' < /dev/urandom | fold -w 12 | head -n 4 # base64 /dev/urandom | tr -d '[^:alnum:]' | cut -c1-10 | head -2 # tr -dc 'a-zA-Z0-9' < /dev/urandom | fold -w 10 | head -n 4 # tr -dc 'a-zA-Z0-9-_!@#$%^&*()_+{}|:<>?=' < /dev/urandom | fold -w 12 | head -n 4 | grep -i '[!@#$%^&*()_+{}|:<>?=]' # tr -dc '[:print:]' < /dev/urandom | fold -w 10| head -n 10 # tr -cd '[:alnum:]' < /dev/urandom | fold -w30 | head -n2 ``` ## Remove Parenthesis with tr `# tr -d '()' < in_file > out_file` ## Generate wordlists from your file-names `# ls -A | sed 's/regexp/& /g'` ## Process text files when cat is unable to handle strange characters `# sed 's/([[:alnum:]]*)[[:space:]]*(.)(..*)/12/' *.txt` ## Generate length based wordlists with awk `# awk 'length == 10' file.txt > 10-length.txt` ## Merge two different txt files `# paste -d' ' file1.txt file2.txt > new-file.txt` ## Faster sorting `# export alias sort='sort --parallel=<number_of_cpu_cores> -S <amount_of_memory>G ' && export LC_ALL='C' && cat file.txt | sort -u > new-file.txt` ## Mac to unix `# tr '\015' '\012' < in_file > out_file` ## Dos to Unix `# dos2unix file.txt` ## Unix to Dos `# unix2dos file.txt` ## Remove from one file what is in another file `# grep -F -v -f file1.txt -w file2.txt > file3.txt` ## Isolate specific line numbers with sed `# sed -n '1,100p' test.file > file.out` ## Create Wordlists from PDF files `# pdftotext file.pdf file.txt` ## Find the line number of a string inside a file `# awk '{ print NR, $0 }' file.txt | grep "string-to-grep"` --- [[Home]]