๐ŸŒˆ ์˜์นด ๋ฐ์ดํ„ฐ ๊ทธ๋ฃน - Airflow์™€ ํ•จ๊ป˜ํ•œ ๋ฐ์ดํ„ฐ ํ™˜๊ฒฝ ๊ตฌ์ถ•๊ธฐ(feat. Airflow on Kubernetes) ๐ŸŽ ์ด ์ž๋ฃŒ๋ฅผ ์ถ”์ฒœํ•˜๋Š” ์ด์œ  - Airflow๋Š” ETL ํŒŒ์ดํ”„๋ผ์ธ์œผ๋กœ ๋งŽ์ด ์‚ฌ์šฉํ•˜์ง€๋งŒ, MLOps์—์„œ๋„ ํ™œ์šฉ ๊ฐ€๋Šฅํ•ฉ๋‹ˆ๋‹ค! - ๋‹จ์ˆœํžˆ Airflow ๊ตฌ์ถ•ํ–ˆ๋‹ค๋Š” ๋‚ด์šฉ๋ณด๋‹ค 3๋…„์˜ ์Šคํ† ๋ฆฌ๋ฅผ ๋‹ด์€ ๊ธ€์ด๊ณ , ์†Œ์ˆ˜์˜ ์ธ์›์œผ๋กœ Airflow๋ฅผ ์šด์˜ํ•  ๋• Composer๋ฅผ ์‚ฌ์šฉํ•˜๊ณ , ์ ์  ์ธํ”„๋ผ๋ฅผ ์ง์ ‘ ๋งŒ๋“œ๋Š” ๊ณผ์ •์— ๋Œ€ํ•ด ๋‹ด์€ ๊ธ€์ž…๋‹ˆ๋‹ค. - ์ตœ๊ทผ์— Kubernetes ์œ„์— Airflow๋ฅผ ๊ตฌ์ถ•ํ•˜๋ฉด์„œ ๋‹ด์€ ๋‚ด์šฉ๋„ ์žˆ์œผ๋‹ˆ, ๊ด€์‹ฌ์žˆ์œผ์‹  ๋ถ„๋“ค์€ ๋ณด์‹œ๋ฉด ์ข‹์„ ๊ฒƒ ๊ฐ™์•„์š” ๐Ÿ™‚ ๐Ÿ‘ ์ฝ์œผ๋ฉด ์ข‹์€ ๋ถ„ - ๋ฐ์ดํ„ฐ ์—”์ง€๋‹ˆ์–ด๋ง์„ ํ•˜๊ณ  ๊ณ„์‹  ๋ถ„, ๊ด€์‹ฌ ์žˆ์œผ์‹  ๋ถ„ - Airflow๋ฅผ ์šด์˜ํ•˜๊ณ  ๊ณ„์‹  ๋ถ„ ๐Ÿ“‹ ๋‚ด์šฉ - ๊ธฐ์กด ํ™˜๊ฒฝ - Rundeck - ํƒœ๋™๊ธฐ - Google Cloud Composer - ์ดˆ์ฐฝ๊ธฐ - Google Compute Engine + Docker Compose - ๊ณ ๋ ค ์‚ฌํ•ญ - ์˜์‚ฌ ๊ฒฐ์ • - ๋ฐฐํฌ ํ˜•ํƒœ - ์šด์˜ ํ˜•ํƒœ - ๋ฌธ์ œ์  - ์„ฑ์žฅ๊ธฐ - Airflow on Kubernetes - ๊ณ ๋ ค ์‚ฌํ•ญ - ํ•ด๊ฒฐ ๋ฐฉ๋ฒ• - ๋ฐฐํฌ ํ˜•ํƒœ - ๋ฐฐํฌ ๋ฐฉ๋ฒ• - ์šด์˜ ํ˜•ํƒœ - ๊ธฐํƒ€ ์ถ”๊ฐ€ ์ž‘์—… - ์„ฑ์ˆ™๊ธฐ๋ฅผ ํ–ฅํ•˜์—ฌ - ๋งˆ๋ฌด๋ฆฌํ•˜๋ฉฐ

์˜์นด ๋ฐ์ดํ„ฐ ๊ทธ๋ฃน - Airflow์™€ ํ•จ๊ป˜ํ•œ ๋ฐ์ดํ„ฐ ํ™˜๊ฒฝ ๊ตฌ์ถ•๊ธฐ(feat. Airflow on Kubernetes)

SOCAR Tech Blog

์˜์นด ๋ฐ์ดํ„ฐ ๊ทธ๋ฃน - Airflow์™€ ํ•จ๊ป˜ํ•œ ๋ฐ์ดํ„ฐ ํ™˜๊ฒฝ ๊ตฌ์ถ•๊ธฐ(feat. Airflow on Kubernetes)

2021๋…„ 6์›” 2์ผ ์˜ค์ „ 3:47

๋Œ“๊ธ€ 0