Write a Visual Web Scrapper, in MVC .Net, based on Portia (Python - github)

已取消 已发布的 Dec 1, 2014 货到付款
已取消 货到付款

Write a .net port, in C# and asp.net mvc, of the Portia project, that can be found on [url removed, login to view] .

Video demo here:[url removed, login to view]

WARNING: This project was already cancelled once! Only apply if you have .net and python professional skills! I am a pro coder, and i know how to identify bad coding. I need a good coder, not a begginer. Project only finish when all features and expectations described above are completed.

Introduction:

- I have a web crawler that is used to scrape information from some websites, and i compile the information to a site I have. I work on this project on my own on weekends, and I am having a big problem that, very often, the websites that I scrape change the layout, or have variable layouts, like templates in Portia understanding. I need a easy way of updating this templates for a website, i a way that i can save as a project, and call this project on my crawler to extract the data of given urls, or html string texts. Given this idea, I think a Portia-like implementation, would be great. And, as I already have a crawler, I would not need the advanced crawler based on 'Scrapy'. I just need a way to create the scraper project for a site while navigate in the site inside the webapplication, and a way to extract the data later in my crawler, by calling a class library, passing project name and url (or html string), and get the existing propertys of the extraction.

Key features:

- Fully written in C#;

- Cannot use database. Projects must be saved as file.

- 2 Projects:

- One WebApplication: for visually build scraping templates for a given site project. Generally, for each site project, I will have a template for listing page and one for details page, but both can have template variations, depending type of product, category, or cases where DOM/Xpaths changes when product has less or more information.

- One class library: that given a url or given a html string, and a project (created on the web application) name, i can extract all data based on the extraction rules created on the web project. I want to call this class library from my existing crawler project.

Expectative:

- Web Application Expectations:

-> Same features as Portia web application, considering:

-> Easy configuration for projects directory path;

-> List Projects feature;

-> Possibility of 'Create new Project', 'Load Project', 'Save Project', 'Close Project(to select other project, initial app state)';

-> Visually navigate the site and select the information to be crawled, with possibility to customize rules, like Portia;

-> Possibility to have a list of starter urls. In this list i will keep the most common result pages that will list me produtcs. So can be easy to navigate directly for the listing page (to build that template) and after to the detail page (and build another template).

-> Possibility to create templates exactly as Portia.

- Class Library Expectations:

-> Public, easy to be used from other assemblies;

-> Easy to use and fast;

-> Can use HtmlAgilityPack and CsQuery;

-> Possibility to pass/set a HttpWebRequest (in case im using a proxy or different agent headers);

-> Possible input parameters for extraction method:

-> string 'project name' and Uri'url to extract';

-> string 'project name' and string 'html';

-> string 'project name', string 'template name', and Uri'url to extract';

-> string 'project name', string 'template name', and string 'html';

-> Output:

-> List of propertys, ready to be indexed by my crawler as soon it call/executes the extraction. I'm thinking in some type of dictionary, that i can search if the key exists, so I can pick the value and store on my data storage.

-> Would be great if the result was a context, with a field with the extractions dictionary, and other fields that list errors occured during parse, template used, and other relevant informations.

.NET MVC Python 网页搜罗

项目ID: #6809505

关于项目

7个方案 远程项目 活跃的Dec 4, 2014

有7名威客正在参与此工作的竞标,均价$304/小时

designysoft

Hello Sir, YSoft Solution, a leading software development company based in India. We have expertise in .NET, MVC, Python, Web Scraping and we have developed many web applications using asp.net. Our team of expert can 更多

$277 USD 在7天内
(21条评论)
6.3
JoaoRoque

I'm specialized in webscraping in C# and python. So this project is a perfect fit for me. But your budget is too low. If you are willing to go a little higher like I propose say something and we'll discuss this better 更多

$400 USD 在14天内
(5条评论)
5.2
murtaza1981

Hi, Please feel free to discuss the project with me ............................................................................ Thanks, Murtaza

$350 USD 在5天内
(7条评论)
4.2
susanzacharia

I am Susan Zachariah having 5 years experience in the area of design and development me and my sister started this company 2 years back i wish to associate with you hope i can deliver your products without any error w 更多

$333 USD 在6天内
(1条评论)
3.9
bestproject108

Hello, We have Experienced Professionals with 12 years of Web Development & Designing, Application Development, Desktop Application, ERP Software, Android App. Development. I-phone Development expertise. We have worked 更多

$222 USD 在12天内
(0条评论)
0.0
deoprogrammer

experience in both c# mvc & python

$333 USD 在20天内
(0条评论)
0.0