Real-time web scraping multiple websites

已关闭 已发布的 4 年前 货到付款
已关闭 货到付款

I need a solution for extraction data in "near real time" from different websites but with the same general topic.

I don't know the numbers of starting pages (maybe 500.000)

The software should do:

- Autodiscovery new URLs (based on specific rules)

- Extract and save some particular data (object name, description, price, ...) and general data (HTML title, author, ...)

- These data must be updated continuously

- These data should be saved in a database

- I need a JSON API that should inquiry that database with these functionalities

- parametric search

- aggregate search

- full-text search

- In case of error or new needs, It should be reprocessed all or some pages.

- Easy to add new sites.

- Enable to scrape ajax based websites.

I'd like to have a solution that can be easy to install, configure and most of all easy to update in case of errors.

AJAX JSON 网页搜罗

项目ID: #19710624

关于项目

48个方案 远程项目 活跃的4 年前

有48名威客正在参与此工作的竞标,均价€3841/小时

chirgeo

Hi. I did read the project description and have a few questions. 1. Do you need the script as well or data only? 2. What is the format of the output data? CSV is OK? We can do other formats as well. 3. Which fields do 更多

€4000 EUR 在5天内
(64条评论)
7.2
AzzkaNoor

Greetings. I have extensive knowledge and expertise in web scrapping/ crawling software. Would you please come over chat so that we can discuss this project further? Willing to showcase my work over chat. It seems like 更多

€4000 EUR 在7天内
(13条评论)
6.6
fatimatuzzohura

Hi, I am Fatima a full time PREFERRED FREELANCER and Python certified developer with 100% guaranteed delivery. Before we start the project would like to have a detailed conversation with you regarding this project to 更多

€4444 EUR 在30天内
(105条评论)
6.3
datadexterous

Hello there! I would like to be considered for your project. We can build a script to scrape multiple websites according to your requirement.

€5000 EUR 在8天内
(99条评论)
6.1
stevobujica91

Hello, I am Python expert and expereienced on scraping web sites. I can build a nice app for your usage. Can we discuss details over chat please? Best Regards

€4000 EUR 在7天内
(44条评论)
6.1
leadconcept

Hi there, I have read initial brief & wondering if you can please provide me URL of those websites, from where you want to scrap the data? Please click on the chat button for further discussion about your project 更多

€5555 EUR 在70天内
(6条评论)
6.2
raminde

Have over 10 years of experience in data mining/ Web scrapping/ Scraping Bots/ Chrome/Opera Extensions I have done it all. Tell us your source and we will put it in excel for you, Or we can even give you filtered resul 更多

€3000 EUR 在30天内
(11条评论)
5.9
farooq4161

Greetings, I am an experienced professional scrapper and have done similar projects in the past. Same can be verified from my profile. Let me allow to assist you with your requirements. Thanks

€3000 EUR 在8天内
(57条评论)
6.1
TenStar718

Hello, and thanks for the opportunity to bid on your project. Please check my portfolio and reviews from other customers https://www.freelancer.com/u/TenStar718.html I am an expert in many different area’s of web 更多

€3000 EUR 在30天内
(10条评论)
5.6
brianconey

Hi there how are you I am a senior python developer and i have used python and selenium . Please contact me and tell me about your scraping project . thanks for your posting

€5360 EUR 在2天内
(19条评论)
5.7
mingxiao2008

Hello, Dear How are you? I have read your project description and am ready for discussing with you about project. I have experienced in developing AJAX, JSON, Web Scraping. I will work very hard and best for you. 更多

€3000 EUR 在30天内
(11条评论)
5.4
pranjalk2902

Hi, Before I give my introduction, let me tell you that I will first provide you a free sample representative of the project so that you can verify the quality of my work and only then award me the project. As far as 更多

€3000 EUR 在3天内
(24条评论)
4.9
dreamci

Nice to meet you I have more than 15 years development experience and a great team, we can complete your project with clean code and best practices To be able to give clients what exactly they want, we are workin 更多

€4444 EUR 在5天内
(11条评论)
4.6
nataliaomelchenk

Hi there. Nice to meet you. I'm a Web scrapting expert and FULL Time Developer. I have checked your requirements carefully. I have rich experience in web scraping, Python, PHP, Micro, MySQL, etc . I can start work imm 更多

€4000 EUR 在7天内
(7条评论)
4.6
ramlogics

Hello, I have gone through your Project requirements and Yes I can do it and also assure you that I can do this job perfectly. Hope to hear you soon. Thank you.

€3000 EUR 在30天内
(9条评论)
4.7
sashaE319WebGuru

Hello. How are you? I have just read your post and interested in this job. As an experienced web developer, I am sure I can handle your project perfectly in the quick timeline. Let me know if you have some time to d 更多

€4000 EUR 在30天内
(5条评论)
4.2
jangjinmei728

Hi there. I've read your job very carefully and understand what you want. I'm a Senior developer with 10+ years experience. I've high skills in Web Scrapping. Any of your requirements are welcome. I can start your proj 更多

€3000 EUR 在7天内
(1条评论)
3.9
kkc1985612

Hi, I looked at your description carefully and I am very interested in providing my skills in hopes of working with you. I have ample experience in web scraping. -scrap any website using selenium, beautifulSoup, reques 更多

€3000 EUR 在10天内
(11条评论)
4.5
johnjordancsc

Hi, I'm a full stack developer and I have experience with more than 11 years of developing. I worked with many projects including enterprise companies. I'm an experienced programmer in PHP, Java, Python and have multip 更多

€4200 EUR 在30天内
(5条评论)
3.8
amit0423

Hi, I have gone through the project details and I can do your project well. Please initiate the chat to discuss more about ths. Looking forward to your response. Best wishes, Amit

€4444 EUR 在7天内
(1条评论)
2.1