Hax

Web - wget

"Making IT happen on the
Information Superhighway"

• wget
  Application Location
  C:\WINDOWS\system32
  Dump Location
  C:\Documents and Settings\User
• wget in a batch format:
  wget -i c:\url.txt
• WWget

• Cygwin wget
  
  Dump Location:
  C:\cygwin\home\User Name

• Authentication
• Concatenate group of files into a single document
• Continue downloading, in event of failure
• Delay, to avoid hammering
• Do not overwrite existing files (file1, file2), etc
• Download URLs from a file
• Force filetype

• Grab all files needed for proper viewing
• Ignore Robots
• Modify content for local viewing
• Progress Bar
• Put all files into same directory
• Put files into specific directory
• Times to attempt download

• # Desired URL
  http://www.cryptome.org/
  
  # Sample script
  wget -P mydir -p -k -nd http://phoenix.csc.calpoly.edu/~st50/index.html
  
  # Positioning to construct a new script
  http://www.cryptome.org/
  wget -P mydir -p -k -nd http://phoenix.csc.calpoly.edu/~st50/index.html
  
  # Newly constructed script
  wget -P mydir -p -k -nd http://www.cryptome.org/
  
  # Add in a delay is -w 5
  
  # Add 5 second delay to newly constructed script
  wget -P mydir -p -k -nd -w 5 http://www.cryptome.org/
  
  # Try putting the file dump into a specific folder in a different root
  wget -P c:\mydir -p -k -nd -w 5 http://www.cryptome.org/
  | | | | | | The "w" command requires wget to WAIT for "x" seconds before requesting another file
  | | | | | The "nd" argument tells wget to put every file into the same directory (no folders)
  | | | | The "k" argument instructs wget to modify the content, so it can be locally viewed
  | | | The "p" argument instructs wget to grab ALL files needed for the page to properly display
  | | | for instance, graphics, "CSS" files or "Javascript" files, etc.
  | | The location for the "Put" command is specified here, as "c:\mydir"
  | The "Put" argument instructs wget to drop the files in the location specified