Sharing Knowledge Preparation

airflow:

materi: slides ok plus confluence
slide link: http://bunkrapp.com/present/6ynkbz/#1
confluence: https://29022131.atlassian.net/wiki/display/DP/How+To%3A+Data+Warehouse+Using+Airflow
materi: create dag, rerun dag, view log
+ contoh source code di slide, ada copy2 image confluence
+ naming convention airflow dag

devops:

slide link: https://docs.google.com/presentation/d/1jDdEaARpdRCuU1lZdijXTVsLRC9mvWbt6PPF9VXKtSc/edit
+ airflow coding convention (lokasi dag, branchnya dimana, beda dag tvlk-prod apa, dll) buat materi
+ release management: to be discussed (test di dev -> codereview -> release?) and to be improved

spark:

slide link: https://docs.google.com/presentation/d/17gUVcFmnjIkv5VnN0RC44u56oZ1TIifUXuWoTtwWZl4/edit
notebook link: http://spark01:9090/notebooks/dp/Spark%20Basics.ipynb
tutorial bikin etl spark + kunci jawaban (om jin) - 1 buat contoh 1 buat coba sendiri - semua orang nyoba kasih deadline
+ spark dataset api intro (om jin sekalian kasih contoh)

flow:

spark theory + basic + best practice hands on (40 min)
airflow theory (20 min)
devops (10 min)
hands on airflow + exercise etl (end to end, +- 30 min) -> dijelasin sama orang spark + devops

Materi slides: di Data Engineering > References > Data Class > 2016-06-Spark and Airlflow for Analytics

A: Arie, Daniel, Mirza
B: Nina, Agung, Rendy
C: Muqtafi, Zaka, Aditya