Hello there!
I have checked the website and it's not possible to scrape it with scrapy. The bot detection algorithms easily detect that it's a bot.
The only way these kind of websites can be scraped is by using selenium.
Using a regular selenium webdriver gets it detected, it's hex data has to be edited as to make it seem like a real browser working on the site.
These algos also detect by the speed of actions performed and buttons clicked to direct to different page and the pixel range where the button has got clicked.
They also track cursor movement.
If you are interested let me know the details of the project and i will work on it show you the output. If it's exactly what you want then only you can assign me the project.
My skills & experience:
-- 2.8 years of experience in building automation tools using python.
-- Worked for Infosys ( a multinational company ) as Automation engineer.
-- Worked on ATP world tour 2017 project as Data analyst.
-- Can scrape content of sites protected with captchas and have dynamic content generated using
ajax and JS.
-- Built more than 56 web scraping projects till date ( 24 on freelancer.com, please check reviews )
-- Websites i have built scraping tools for include Linkedin, Facebook, Gmail, Tinder, Okcupid,
Outlook and many more.
-- Can bypass websites protected with captchas ( reCaptcha v2 and invisible Captcha ).
Looking forward to hear from you.
Thank you, have a nice day!