CodeSnippets: find code

ffind - fuzzyfind from the current directory

to find shell osx mac bash unix file directory by jvs on Nov 22, 2008

# fuzzyfind from the current directory
function ffind() {
   declare regex filetype
   if [[ $# -gt 2 ]] || [[ $# -eq 0 ]]; then return 1; fi
   if [[ $# -eq 2 ]] && [[ "$1" == '-d' ]]; then filetype='d'; fi
   regex="$(printf "%s" "${!#}" | /usr/bin/sed -E -e 's/([^][]|\[[^][]+\])/\1\.\*/g' -e '/\.\*\$\.\*$/s/\.\*\$\.\*$/\$/')"
   regex="${PWD}/.*${regex}"
   /usr/bin/find -x "${PWD}" -type "${filetype:-f}" -iregex "$regex"
   return 0
}


ffind "searchstring"                # default is file search
ffind -d "[3-7]searchstring[5-9]"   # search for directories 
ffind "searchstring$"               # last character of file name is a "g"

str="searchstring"
str="[3-7]searchstring[5-9]"
str="search/string$"
printf "%s" "${str}" | /usr/bin/sed -E -e 's/([^][]|\[[^][]+\])/\1\.\*/g' -e '/\.\*\$\.\*$/s/\.\*\$\.\*$/\$/'

Delete specified files in the current directory

to find shell osx mac bash unix delete by jvs on Nov 22, 2008

man find | less -p '-delete'

find . -mindepth 1 -maxdepth 1 -type f -name '*.txt' -ls

find . -mindepth 1 -maxdepth 1 -type f -name '*.txt' -delete

Store file information in SQLite database

to find shell osx mac bash sql unix database file array sqlite3 sqlite write insert transaction by jvs on Oct 30, 2008

A basic example of using BEGIN-INSERT-COMMIT to write data (gathered by a shell script) to an SQLite database file.


# basic form of BEGIN-INSERT-COMMIT to write data to existing db
/usr/bin/sqlite3 db  < <( 
cat <<-'EOF'
BEGIN;
insert statement 1;
insert statement 2;
insert statement 3;
...
COMMIT;
.q
EOF
)


man sqlite3

/usr/bin/sqlite3
sqlite> .help
sqlite> .q


# file info database function
# usage: finfodb [optional: -t] [directory] [optional: filename]

function finfodb() {

   declare -a ar
   declare -i i timestamp
   declare checksum db filename searchdir

   declare dbdir="${HOME}/Library/FileInfoDB" 
   declare dbname="finfodb"
   declare sleep_before_write="0.5"
   declare -i block_insert_num=1000

   dbdir="${dbdir%/}"    # cut off trailing slash if necessary

   if [[ $# -eq 0 ]]; then 

      printf "%s\n%s\n" 'No arguments given!' 'Usage: finfodb [optional: -t] [directory] [optional: filename]'
      return 1

   elif [[ "${1}" == '-t' ]]; then

      if [[ ! -d "${2}" ]]; then printf "%s\n" "No directory: ${2}"; return 1; fi
      timestamp=1
      searchdir="${2}"
      filename="${3:-*}"

   else

      if [[ ! -d "${1}" ]]; then printf "%s\n" "No directory: ${1}"; return 1; fi
      timestamp=0
      searchdir="${1}"
      filename="${2:-*}"

   fi


   # store the SQLite databases in $dbdir
   /bin/mkdir -p "${dbdir}"

   if [[ $timestamp -eq 1 ]]; then

      # create name of SQLite database with time stamp
      db="${dbdir}/${dbname}--$(/bin/date -u +%Y-%m-%d--%H.%M.%S--%Z)"
      #db="${dbdir}/${dbname}--$(/bin/date +%Y-%m-%d--%H.%M.%S--%Z)"

   else

      db="${dbdir}/${dbname}"

   fi


   # create the SQLite database if necessary

   if [[ ! -f "${db}" ]]; then

/usr/bin/sqlite3 "${db}"  < <( 
/bin/cat <<-'EOF'
     PRAGMA encoding = "UTF-8";
     PRAGMA auto_vacuum = 1;
     create TABLE filecheck (num INTEGER PRIMARY KEY, filepath TEXT, checksum TEXT, timeEnter DATE);
     CREATE TRIGGER insert_filecheck_timeEnter AFTER INSERT ON filecheck
     BEGIN
      UPDATE filecheck SET timeEnter = DATETIME('NOW') WHERE rowid = new.rowid;
     END;
EOF
)

   fi

   i=0
   ar[${i}]='BEGIN;'$'\n'

   while read -d $'\0' filepath; do 

      # write $block_insert_num insert statements to ${db}
      # empty $ar and reset $i to 0
      if [[ $i -ge $block_insert_num ]]; then
      #if [[ "${#ar[@]}" -ge $block_insert_num ]]; then
         let i++
         ar[${i}]='COMMIT;'$'\n'
         let i++
         ar[${i}]='.q'$'\n'
         /bin/sleep $sleep_before_write
         # write insert statements stored in $ar to $db (see below)
         # delete a possible single leading space in every line (i.e. array item) using "${ar[@]/# /}"
         /usr/bin/sqlite3 "${db}" < <( printf "%s" "${ar[@]/# /}" )
         #/usr/bin/sqlite3 "${db}" < <( for ((i=0; i < "${#ar[@]}"; i++)); do printf "%s" "${ar[$i]#"${ar[$i]%%[! ]*}"}"; done )
         #/usr/bin/sqlite3 "${db}" <<< "$(echo "${ar[@]}" | /usr/bin/sed 's/^ *//')"
         unset -v ar i
         declare -a ar
         declare -i i=0
         ar[${i}]='BEGIN;'$'\n'
         echo
      fi

      let i++

      filepath_cleaned="${filepath//[[:cntrl:]]/}"   # remove control characters such as \n, \r, ...

      # cf. http://codesnippets.joyent.com/posts/show/1395
      if [[ ${#filepath_cleaned} -lt 85 ]]; then
         printf "\r\e[0K\e[1;32m%s\e[0m  %s" "${i}" "${filepath_cleaned}"
      else
         printf "\r\e[0K\e[1;32m%s\e[0m  %s" "${i}" "${filepath_cleaned:0:40}.....${filepath_cleaned: -40}"
      fi

      checksum="$(/sbin/md5 -q "${filepath}" 2>/dev/null)"

      # sha512
      # sudo port install openssl
      # cf. http://trac.macports.org/wiki/InstallingMacPorts
      #checksum="$(/opt/local/bin/openssl dgst -sha512 2>/dev/null < "${filepath}")"

      # escape single quotes for SQLite; cf. http://sqlite.org/lang_expr.html
      filepath="${filepath//\'/''}" 

      # escape newline characters \n
      #filepath="${filepath//$'\n'/\\n}" 
    
      # remove control characters such as \n, \r, ...  
      #filepath="${filepath//[[:cntrl:]]/}"   

      # create insert statements
      ar[${i}]="insert into filecheck (filepath,checksum) values ('${filepath}', '${checksum}');"$'\n'

   done < <(/usr/bin/find -x "${searchdir}" \( -type f -or -type l \) -name "${filename}" -print0 2>/dev/null)

   echo

   let i++
   ar[${i}]='COMMIT;'$'\n'
   let i++
   ar[${i}]='.q'$'\n'

   /bin/sleep $sleep_before_write


   # write insert statements stored in $ar to $db
   # cf. Removing whitspace from variables in bash?, http://www.jlaforums.com/viewtopic.php?t=1427403
   #/usr/bin/sqlite3 "${db}" <<< "$(for ((i=0; i < "${#ar[@]}"; i++)); do printf "%s" "${ar[$i]#"${ar[$i]%%[! ]*}"}"; done)"
   #/usr/bin/sqlite3 "${db}" < <( for ((i=0; i < "${#ar[@]}"; i++)); do printf "%s" "${ar[$i]#"${ar[$i]%%[! ]*}"}"; done )

   # delete a possible single leading space in every line (i.e. array item) using "${ar[@]/# /}"
   /usr/bin/sqlite3 "${db}" < <( printf "%s" "${ar[@]/# /}" )

   # alternatives with sed to trim leading whitespace
   #/usr/bin/sqlite3 "${db}" <<< "$(echo "${ar[@]}" | /usr/bin/sed 's/^ *//')"
   #/usr/bin/sqlite3 "${db}"  < <(echo "${ar[@]}" | /usr/bin/sed 's/^ *//' )


   /usr/sbin/chown -R $(/usr/bin/logname):$(/usr/bin/logname) "${dbdir}"
   /bin/chmod -R 0700 "${dbdir}"

   return 0
}



finfodb
finfodb 1234
finfodb -t
finfodb /usr/bin
finfodb /usr/bin "*z*"
finfodb -t /usr/bin
finfodb -t /usr/bin "*z*"

finfodb ~/Library
finfodb /Library

open ~/Library/FileInfoDB

sqlite3 ~/Library/FileInfoDB/finfodb  ".dump"
sqlite3 ~/Library/FileInfoDB/finfodb  "select * from sqlite_master"
sqlite3 ~/Library/FileInfoDB/finfodb  "select * from filecheck";
sqlite3 ~/Library/FileInfoDB/finfodb  "select * from filecheck WHERE checksum LIKE ''";
sqlite3 ~/Library/FileInfoDB/finfodb  "select * from filecheck WHERE filepath LIKE '%sync%'";
sqlite3 ~/Library/FileInfoDB/finfodb  "select * from filecheck WHERE filepath LIKE '%/_sync'";
sqlite3 ~/Library/FileInfoDB/finfodb  "select * from filecheck WHERE filepath LIKE '%/ds%'";


sqlite3 ~/Library/FileInfoDB/finfodb  "select * from filecheck WHERE checksum LIKE ''";

sqlite3 ~/Library/FileInfoDB/finfodb  "delete from filecheck WHERE checksum LIKE ''";

sqlite3 ~/Library/FileInfoDB/finfodb  "select * from filecheck WHERE checksum LIKE ''";

sqlite3 ~/Library/FileInfoDB/finfodb "vacuum"

sqlite3 ~/Library/FileInfoDB/finfodb "pragma integrity_check"

Further information:

- man sqlite3
- SQLite Tutorial: Common Commands and Triggers
- Build an application using simple shell scripts and SQLite
- A Quick Guide to SQLite and Ruby
- SQLite: Available Documentation
- sqlite3: A command-line access program for SQLite databases
- Datatypes In SQLite Version 3
- The SQLite Query Optimizer Overview
- SQL As Understood By SQLite
- SQL As Understood By SQLite: BEGIN TRANSACTION
- SQL As Understood By SQLite: expression (LIKE operator)
- Atomic Commit In SQLite
- SQLite: PRAGMA command
- SQLite FAQ
- SQLite The Hammer
- SQLite ODBC Driver
- Actual ODBC Driver for Open Source Databases
- SQLite Download Page: Precompiled Binaries For Mac OS X

Progress indicator snippet for man find

to find shell osx mac bash unix indicator progress by jv on Feb 07, 2008

/usr/bin/find -x "$HOME" -type f -print0 2>/dev/null | while read -d $'\0' filename; do 

   filename="${filename//[[:cntrl:]]/}"   # remove control characters such as \n, \r, ...

   if [[ ${#filename} -lt 85 ]]; then
      printf "\r\e[0K%s" "${filename}"; 
   else
      printf "\r\e[0K%s" "${filename:0:40}.....${filename: -40}"; 
   fi

done; echo



/usr/bin/find -x "$HOME" -type f -print0 2>/dev/null | while read -d $'\0' filename; do 

   i=$[i+1]

   filename="${filename//[[:cntrl:]]/}"   # remove control characters such as \n, \r, ...

   if [[ ${#filename} -lt 85 ]]; then
      printf "\r\e[0K\e[1;32m%s\e[0m  %s" "${i}" "${filename}"
   else
      printf "\r\e[0K\e[1;32m%s\e[0m  %s" "${i}" "${filename:0:40}.....${filename: -40}"
   fi

done; echo

Finding large files

to find bash files large by dallas on Feb 05, 2008

// Hi all, just a cheap and easy way to find a bunch of large files on your server etc..


#!/bin/bash
for file in `find / -type f -size +100000`; do
        ls -lh $file
done

recursively find (and delete) files and directories in linux (and svn)

to find linux administration by deckard on May 29, 2007

This example looks in the current directory for directories named .svn

That isn't the best way to do it. It will break if you have lots of .svn directories. Then the command line will be too long and you will get 'argument list too long'. Recent bash have enormous buffers so this is less of a problem. xargs solves this problem.

The other problem is any file with spaces in the name will cause bash to treat it as two arguments. Which could give an error or could delete the wrong file. find -print0 and xargs -0 solve this problem.

find . -type d -name .svn

find . -name .svn -print0 | xargs -0 rm -rf

OR make an alias to an egrep command and go nuts

alias g='egrep --color=auto -i -r'

from here on out, just type g "<search string>" *

case insensitive
recursive
colored results

Find last article

to find last article by john_hammond on Apr 16, 2007

Add this to your Article model (or whatever you name it) to get the last published article on your site.

def Article.find_last
  Article.find(:first, :order => "created_at DESC")
end

Mass find-replace using sed

to find grep sed replace by jamiew on Mar 30, 2007

Replace some widespread nasty hardcoded strings. 'sed $file > $file' doesn't work so hot so we use a temp file, and also make a backup of the old file.

for i in $(find . -type f); do sed 's/oldstring/newstring/g' $i > $i-tmp; mv $i $i-backup; mv $i-tmp $i; done

Make Find.find return an array

to find ruby by jonleighton on Dec 19, 2006

Ruby's find module usually forces you to pass it a block. This snippet makes it return an array if there is no block. Note: the returning method is from Rails, but this can easily be removed.

require 'find'

module Find
  class << self
    alias_method :find_old, :find
    
    def find(*paths)
      if block_given?
        find_old(*paths, &yield)
      else
        returning ary = [] do
          find_old(*paths) { |p| ary << p }
        end
      end
    end
  end
end

Find flagged message subjects in maildir

to find shell mail maildir flagged messages by nickstenning on Jun 22, 2006

Simple, perhaps slightly cryptic oneliner to get the subjects of all flagged messages in a Maildir. It assumes that 'F' is the flag for flagged messages (as it almost always is) and that your message files end with ":2,<flags>".

find ~/Maildir -type f -name "*:2,*F*" -exec egrep -h "^Subject:" "{}" ";" | cut -c 10-

CodeSnippets

Related Tags

Never been to CodeSnippets before?

On This Page:

ffind - fuzzyfind from the current directory

Delete specified files in the current directory

Store file information in SQLite database

Progress indicator snippet for man find

Finding large files

recursively find (and delete) files and directories in linux (and svn)

Find last article

Mass find-replace using sed

Make Find.find return an array

Find flagged message subjects in maildir

On This Page:

Joyent

Related Tags

Never been to CodeSnippets before?

On This Page:

On This Page: