REST API Webhooks
API as an offering will enable the user to integrate packaged data provided by the API into their internal systems and build more complete business intelligence systems.
Worked On
Backend data pipelines, REST service, data source
Tech Stack
Go+Gin, Redshift, Postgres, Redis, S3, Elasticsearch, Kong, Bash Scripting, Airflow
Date
January, 2020
Challenge
To build an API service from scratch, which will have its authorization, rate limiting and data pipelines for milli-second latency
Solution
Designed and implemented a REST service with configurable rate limiting and data pipeline for creating an isolated dataset for external exposure
Based on the requirement and projected usage patterns, I came up with the architecture for backend service and data pipelines. I wrote a bash script that narrowed down the data that is needed for exposure via API and transferred that from our data warehouse to a row-based storage, using an object storage as an intermediary. For filtering capabilities, I integrated the API with Elasticsearch which resulted in a simple index-scan in RDS, setting us up for milli-seconds latency.
The high-throughput service was written in Go, which used Redis for performing rate-limiting at a customer level. Later on, we isolated the rate-limiting responsibility from the service layer to a separate application gateway (Kong). This enabled us to capture the usage metrics as well for internal analytics.
