Create Instagram data source

General Objective

Create a Instagram datasource for a data mining product

Specific Objectives

Create a AWS lambda function for extracting data from a third party API (Data Extraction)

Create a Python function for transforming this data into rows (Data transformation)

Delivery time: 3 days

Context

A W Datasource is a software component that is basically responsible for retrieving information from a specific source in a specific format. After the information is retrieved, a transformation function is applied for converting the data into a row record, required for storing the information as a table.

Data Extraction Function

The objective of this element is to retrieve the data from the source, without applying any changes nor additional logic.

Consider this element as a bridge which consumes the API using the configuration parameters received (input parameters) and provide exactly the same result that the API returns.

The magic of this element consists on consuming any API endpoint available by the provider passing the data to the next point in the chain. Changing the input parameter must be the only thing required for reaching a different endpoint or to consume a specific endpoint with different options. The goal behind this is to have a generic datasource capable of retrieving any info available by the provider. Using this approach, a new requirement coming from a datasource user will not require any code modification or development.

Transformation Function

This is where the conversion and the handling of the information returned by the provider. At the end of the process must convert the data into a collection of rows. Each row is passed to the helper as an object. This step is for converting the raw data returned by the provider into something that can be used for the table creation.

we will provide the credentials once the project be accepted.

Amazon Web Services AWS Lambda Git Python RESTful API

项目ID： #18549890

关于项目