Control panel for run spiders with Scrapy - Python
$250-750 USD
已取消
已发布将近 9 年前
$250-750 USD
货到付款
I'm need a control panel to manage the spiders: schedule, pause, stop, run, verify the scraped itens and more.
-Clean and simple interface;
-Run under a localhost machine (Windows);
-Developed with a simple language like own python or php.
This service is the best example, once is from the Scrapy framework developers: [login to view URL]
============
Basically the scraper is working this way:
1. The spider scrape the data and export to a json file inside a S3 bucket. Also the images are exported to the same bucket
2. An [login to view URL] script read this json's files and export to a MySQL database
3. A [login to view URL] script read all the products in the database table and group the similar products based on some rules
Also I'll need a resource to insert, schedule and run scripts like [login to view URL] and [login to view URL]
============
On the attached file, please discard the "Login" resource. Will not be necessary.
Hi sir,
I am scraping expert, I have did too many similar projects, please check my feedback then you will know.
Can you tell me more details? then I will provide demo data for you.
Thanks,
Kimi
Hi. I specialize in web scraping with Scrapy. I can write an app on top of scrapyd (the Scrapy daemon) that fulfills your requirements. I would write the app on python but there are two options, let me know your preference:
1- A desktop application: this would be a standalone app with the look and feel of any desktop app, I mean, windows and buttons instead of pages and links. You would have to install the app in every computer.
2- A web application: you open it as a page in the browser and it's accessible from any computer with a web browser, but you need to have a server. It can run on a single computer, but this computer would need to have the server installed and running.
In both cases you still need the scrapyd server.
If you are interested in my bid please contact me before awarding me the project.
Regards.
i've been developing many web scrapers for freelancer mostly in pytthon and php and many other web related projects, so I'm aware of the project needs and the python platform in special. Given the requirements I can develop this as a webserver or a desktop application. I read and write English language (e falo Português como lingua materna).