The goal of data ingestion is to provide data from various data sources to be available for analytical purposes. Data ingestion tasks vary from adding new tracking, ingest db tables, to scraping. Originally data ingestion tasks are part of data engineering team's responsibilities but we found many problems during the execution :
To overcome those problems, we take the initiative to shift data ingestion tasks ownership to each mission. Data engineering team will provide procedure and tools to support mission teams. We hope that each mission team can manage their own ingestion, prioritize data ingestion tasks, and independent in fulfulling their own data needs.
In its execution, data ingestion will need helps from both data analysts and engineers in each mission. Procedures for data ingestion tasks are provided in the following document (details for postgres ingestion and scraping tasks will be provided later once their procedures and tools are ready). If you have something to discuss / any concern, don't hesitate to contact me.
Thank you!
https://docs.google.com/a/traveloka.com/presentation/d/1L0UcLPzQKXvZmtJnQK6b9g1lq2flHPcjyl_KJKzxXwU/edit?usp=sharing