[nycbug-talk] silly snarf tricks

[Bob Ippolito]

On May 26, 2004, at 9:19 PM, Mikel King wrote:

>    A while back I was playing around with snarf which is a command 
> line utility for pulling web content...Ok so no real big deal there 
> FreeBSD has fetch which does the same thing. However snarf is cool in 
> that you can drop a switch that tells it to return a precompiled 
> HTTP_USER_AGENT.
>
>    Now aside from messing with my web developers heads a bit, cause I 
> modified the code to return a (Mozilla 5; on a Banana Jr 2000...) and 
> lately I've changed it to match my FreeBSD 5.1 install on my laptop. 
> Then I scripted it in a cron job to run once a minute and grab 5 pages 
> from our company website. OK this really screwed with the web 
> stats...but it was harmless fun. I mean we now get 3800 hits per day 
> from a machine pretending to be Mozilla 1.6 on FreeBSD 5.1.
>
>    I guess what I am wondering is if anyone can devise a praticle use 
> for this?
>
>    I did use it to scriptout a daily pull of the intelligent updater 
> from Symantec, and place it into my companies ftp server. But that's 
> really nothing magical...
>
>    ideas?

curl (installed by default on OS X) and GNU Wget (pretty common with 
Linux) also let you set your User-Agent.. surprisingly enough, they 
actually take the same --user-agent flag to do it.

There aren't really any practical uses for it, except to fetch strange 
URLs that send different content or authorize based upon UA, and of 
course stat jacking.

-bob