Develop an AUTOMATED method to create a database of pages that are the result of a Google search, the the text on those pages, and links from those pages and their text.
I want a database of pages that come from this general method:
1) run Google searches on these search terms: [register OR registration "emergency notification"] in several passes:
- run one search each for [register OR registration "emergency notification" link:xxxxxxxxx] where xxxxxxx is replaced with each of the following domains.
[login to view URL]
[login to view URL]
[login to view URL]
[login to view URL]
[login to view URL]
[login to view URL]
[login to view URL]
[login to view URL]
[login to view URL]
[login to view URL]
[login to view URL]
[login to view URL]
[login to view URL]
- run one search each for [register OR registration "emergency notification" xxxxxxxxx] where xxxxxxx is replaced with each of the following exact words.
CityWatch
FirstCallNetwork
CodeRed
Everbridge
TwentyFirst Century
Rave
Deltalert
OneCallNow
RapidNotify
Nixle
Swift911
Cassidian
BlackboardConnect
These searches should be as exhaustive as possible. So when google says "In order to show you the most relevant results, we have omitted some entries very similar to the XX already displayed. If you like, you can repeat the search with the omitted results included." You should repeat the search will the omitted results included.
2) For each of the searches, store the following as columns or fields in an excel spreadsheet or database format we agree on (rows will be each of the results on the google search):
- the link or search term (so they can all be identified separately)
- the URLs that google provides
- all of the text on the page for each URL
- all of the source of the page for each URL
- the date of the page if available