US Gov to Spider Net Searching for Terrorists

what makes you think they aren't doing that already?

NSA has more brain power and funding than you think.

i have the feeling that robots.txt isn't going to be honored.

could be an extremely effective tool. Without proper oversight... ever see the movie Fortress with chris lambert? this is an unauthorized thought process...

"Last night I watched Enemy of the State... When it came out I was told it was a paranoid fantasy..."

http://battellemedia.com/archives/002254.php

Like Battelle, I happened to catch the last half of Enemy of the State and was struck by how much of then sci-fi was now at least on the radar as probably do-able, if not a sure-thing already done. 7 years. damn.

Since I've started working on interactive scraper and bot blocking this might be the biggest incentive yet. Probably end up with a G-Man banging on my door wanting to know what I'm hiding behind that spider blocker any day now.

They'll just have DOC nuke your domain. Well, all of your domains ...just to be safe.

Check this converted PDF of the DHS Science and Technology budget
http://64.233.179.104/search?q=cache:PMV5mMZnBDEJ:ucop.edu/research/homelandsecurity/documents/STFY2006CJ2022005Final1.pdf+%22Analysis,+Dissemination,+Visualization,+Insight,+and+Semantic+Enhancement%22&hl=en&gl=us&ct=clnk&cd=5&client=firefox-a

Then scroll down to page 93 to see what they plan to do...

page 93

And I guess the irony of all that effort is that only by them putting forth that effort is it necessary.

The article mentions that part of the programme is already activated and suggest that some of it is derived from two previous programmes TIA, and TIA. And whatever happened to the, whatwasitnow "carnivore" / "echelon"? Or all the other strange names and abbreviations that has been thrown around constantly for years. It's not really a surprise to anyone that such things are going on.

I'm not sure about the privacy considerations, especially as whatever the US Congress decides will probably only be applicable to US residents, but from a professional viewpoint I find the (thought of such a) system interesting (as in "theoretically interesting - quite possibly harmful and not likely useful").

Based on the second-hand descriptions alone I really can't imagine it being useful and practical. Expensive, yes - privacy violating, quite probably - but useful, no not really. Except, perhaps for surveillance purposes, in which the targets are known and identified beforehand.

Also, for general trending a system like that might be useful, but that is in the realm of politics, not defence (if these things are still separate nowadays). The proverbial "early warning system" OTOH, is a pipe dream at best.

The key to understanding is that when mining a large amount of data you must either look for stuff you know something specific about in advance (ie people names, places, jargon, whatever), or you will only get general trends. You will not really get useful information about stuff you don't know anything about in advance.

Compare it to doing a regular search in your favorite SE: If you know about the topic at hand you can enter the relevant terms and get a list of pages that match. But if you don't really know what to look for in specific detail and in advance you will not know which terms to enter and what you'll get in return will depend largely on luck.

Also, even if you could produce someting like a "crystal ball" for whatever kind of activity deemed "interesting" (political flavour of the day) the results will be worth nothing if the intel reports are just ignored at higher levels. I seem to recall quite a few reports in the aftermath of 9/11 suggesting very detailled prior knowledge (eg. intel reports) of some of the key events, but of course I might be wrong and that may never have happened.

Compare it to doing a regular search in your favorite SE: If you know about the topic at hand you can enter the relevant terms and get a list of pages that match. But if you don't really know what to look for in specific detail and in advance you will not know which terms to enter and what you'll get in return will depend largely on luck.

I disagree. My guess is it could work something like this: the spider will flag websites that fit their criteria, then humans will examine the sites and decide whether these sites go on the watch list. Now, we can monitor these "sites of interest" for people who are regularly accessing/posting to them. Then we've got a list of persons of interest that we can track. Monitor them and we get a whole new list of sites/people and it just keeps growing. Add to that sophisticated behaviour monitoring and cross referencing with criminal/intelligence databases and it could be pretty useful imho.

Any time I've checked, the first visitor to any new domain I register has always been the US government, specifically the navy.

What they want with all the face creams I sell is anyone's guess.

icicle the Navy is very lax on their monitoring of internet usage at work so its not suprising that you get traffic from them.