Expert to build a web using Scrapy + djangoCMS + AngularJS4

IMPORTANT NOTES: 1) Please read the project details and attachment before you bid. 2) Full requirements as in attachment, which is fairly clear 3) Budget currency is NZD 4) Total price of this job is as high as you bid, no-negotiable. sources: 1)[login to view URL] 2)[login to view URL] ASUMPTIONS: A1: 1) scrap by using regex to complete the ID for the whole URL A2: 2) can only be scrap based on looping each category/subcategory to get all listings in each page If you have a better way for A2, that is great. --------------------------------- Functional requirements: --------------------------------- R1. scrap all categories (see Appendix 2: product categories in attachment) R2. scrap all listings of a website R3. scrap production information and delta information of an existing listing: price, stock^, sold qty^, see Appendix 1:schema tables below. R4. Admin backend UI to: 4.1 list all existing sites, last ran schedule and add new site(name,site url) 4.1.1 a button for pop up to set up schedule to start scrapping a site and a button to start scrap the site 4.1.2 a button to drill down the siate and redirect a new pate for displaying categories of a site 4.2 add new category in 4.1.2 and a button to scrap new category 4.3 display all listing under the category. 4.5 a page to display fields and setting for a scrapy item. Modify existing css rule for existing field of a listing 4.6 add a new css rule to scrap a new field(document/write HOWTO steps for changing the code if needed) ^: if field/data exists in page R5. webAPI to return result of listing query R6. front end pages to: 6.1 display all categories of a site on left and display all listings a site - paged 6.2 click on one listing in 6.1 and open a new page to display detail of a listing. product name, brand, image, latest price/price history(scrapped by #R3) R7. user login and add listing to watch list --------------------------------- Technical requirements: --------------------------------- Backbones: Scrapy + djangoCMS + MySQL + SQLAlchemy + AngularJS4 A. add scrapy module as an app for djangoCMS 1. support multi-threading 2. anti-scraping: [login to view URL] tool:RandomProxy + RotateUserAgentMiddleware 3. each listing has an Unique identifier(foreign key for other tables) 4. data access layer: SQLAlchemy B. admin backend 1. support login( djangoCMS support by default) 2. four *admin* pages integrated to *djangoCMS admin backend*, refer to in attachments(site page / rule page/ product page/ users) C. WebAPI as an app for django (for R5) 1. input: text(product name) or unique identifier 2. return: array of listing in JSON format [{Id, productName, brand, last price, last date, image url, [category 1, category 2, ...]}] 3. authentication token(64bit) 4. security: SSL/https D. front end to display (for R6) 1. UI(Angular4 + Bootstrap) 2. security: SSL/https 3. call webapi to retrieve data 4. support CORS: [login to view URL] 5. listing page to display products, display top 20 products by default order by last scrapped time. 6. detail page to display product details E. naming convention: CamelCase or underscore F(optional). it is great if WebAPI result can be used for both admin backend and frontend, if you want, you can use WebAPI for admin backend   Appendix Appendix 1: schema tables: 1. site (id, name, baseUrl, enable, schedule, lastScrapDateTime, lastStatus) store site as source, enable flag indicate the schedule is active for scrapping, update lastScrapDateTime each time the spider ran) 2. siteRule(id, siteId, ruleName, enable, type^, url, minId**, maxId**, linkRegex^^) ^type: depend on scrap by Id or category links (see [sources] at the beginning above) .... please find more in attachment