Write some Software

Complete custom web scraping. We need to extract companies and the reviews of them from a number of sites, one site has been done already within a C# solution that will be clear to continue the other sites with, and so the info below is for your estimating. If you are chosen you will need VS2015 installed (community fine) and you will check-in your work as you go to visual studio online TFS. The sites below are ordered by priority. 1. [login to view URL] 2. [login to view URL] (done already) 3. [login to view URL] (‘started’ by a friend, but I cannot vouch for progress made or code quality as I haven’t looked into finishing). 4. [login to view URL] 5. [login to view URL] 6. [login to view URL] 7. [login to view URL] 8. [login to view URL] 9. [login to view URL] 10. [login to view URL] For each of the other sites you will take a copy of the ‘TODOExtractor’ file. I suggest you get ‘plumber’ category working first for each…then the categories for further testing are in the attached [login to view URL], these may be named slightly differently depending on the website, e.g. ‘plumbing’ vs ‘plumber’. If any cannot be matched to one or more sites please inform. You can do each site either by grabbing from any JSON API you discover or by scraping the HTML…here are some links that may be useful for sites that require scraping: [login to view URL] -Which links to: [login to view URL] [login to view URL] -Which links to: [login to view URL] Once the data is extracted the program will write out to output files. The file writing is done for you in the existing C# project…there is shared code to call that writes out files in the required output format into a ‘TestOutput’ folder within the project. It creates a simple semi-colon separated, ordered list of fields saved as .txt files but none of this will need to change or should be changed. The program generates an Id for the company during the extraction process. This Id must also be present in the Reviews file as a ‘foreign key’. This is done for you in the existing C# project. This will not change. For ‘TrueLocal’ we have the ability to run the extraction program on combination of a particular category e.g. ‘plumbing’ and a particular postcode…this ability will depend on the site and postcode can be swapped out ‘Locality’ or ‘RegionCode’ if that makes more sense for the particular site, but otherwise it should run in the same manner. The data to be extracted is as below. BusinessLead Review Id varchar(16) NOT NULL, -- Generated as described later PhoneNumber varchar(20) NULL, Email varchar(255) NOT NULL, -- REQUIRED and MUST be UNIQUE FirstName varchar(50) NULL, LastName varchar(50) NULL, [Address] varchar(100) NULL, -- Unit number where applicable Address2 varchar(100) NULL, -- The Main Street address Locality varchar(80) NOT NULL, -- REQUIRED! Town/Suburb e.g. RICHMOND RegionCode varchar(3) NOT NULL, -- state: VIC, NSW, TAS, NT, SA, PostalCode varchar(4) NULL, -- Australian postcode 2000, 3000 Company varchar(80) NOT NULL, -- Company NAME...REQUIRED and MUST be UNIQUE BusinessNum varchar(15) NULL, -- This is ABN. ([login to view URL]) Description varchar(8000) NULL, -- How the business describes itself FoundingYear int NULL, -- If available EmployeeCount int NULL, -- If available --Following is LIKE an array [], e.g: [Plumbing,Handyman] just extract them with names as they are from the target sitePredictedServiceTypeNames varchar(2000) NOT NULL LeadId varchar(16) NOT NULL, --the ‘foreign key’ ref to BusinessLead OverallRating int NOT NULL, --out of five. ReviewText varchar(6000) NOT NULL, FirstName varchar(50) NOT NULL, --Of Reviewer LastName varchar(50) NOT NULL, --Of Reviewer Email varchar(255) NULL, --Of reviewer IF available ReviewDate datetime NOT NULL --SHOULD be available