Artwork

Sisällön tarjoaa Confluent, founded by the original creators of Apache Kafka® and Founded by the original creators of Apache Kafka®. Confluent, founded by the original creators of Apache Kafka® and Founded by the original creators of Apache Kafka® tai sen podcast-alustan kumppani lataa ja toimittaa kaiken podcast-sisällön, mukaan lukien jaksot, grafiikat ja podcast-kuvaukset. Jos uskot jonkun käyttävän tekijänoikeudella suojattua teostasi ilman lupaasi, voit seurata tässä https://fi.player.fm/legal kuvattua prosessia.
Player FM - Podcast-sovellus
Siirry offline-tilaan Player FM avulla!

Git for Data: Managing Data like Code with lakeFS

30:42
 
Jaa
 

Manage episode 352916148 series 2355972
Sisällön tarjoaa Confluent, founded by the original creators of Apache Kafka® and Founded by the original creators of Apache Kafka®. Confluent, founded by the original creators of Apache Kafka® and Founded by the original creators of Apache Kafka® tai sen podcast-alustan kumppani lataa ja toimittaa kaiken podcast-sisällön, mukaan lukien jaksot, grafiikat ja podcast-kuvaukset. Jos uskot jonkun käyttävän tekijänoikeudella suojattua teostasi ilman lupaasi, voit seurata tässä https://fi.player.fm/legal kuvattua prosessia.

Is it possible to manage and test data like code? lakeFS is an open-source data version control tool that transforms object storage into Git-like repositories, offering teams a way to use the same workflows for code and data. In this episode, Kris sits down with guest Adi Polak, VP of DevX at Treeverse, to discuss how lakeFS can be used to facilitate better management and testing of data.
At its core, lakeFS provides teams with better data management. A theoretical data engineer on a large team runs a script to delete some data, but a bug in the script accidentally deletes a lot more data than intended. Application engineers can checkout the main branch, effectively erasing their mistakes, but without a tool like lakeFS, this data engineer would be in a lot of trouble.
Polak is quick to explain that lakeFS isn’t built on Git. The source code behind an application is usually a few dozen mega bytes, while lakeFS is designed to handle petabytes of data; however, it does use Git-like semantics to create and access versions so adoption is quick and simple.
Another big challenge that lakeFS helps teams tackle is reproducibility. Troubleshooting when and where a corruption in the data first appeared can be a tricky task for a data engineer, when data is constantly updating. With lakeFS, engineers can refer to snapshots to see where the product was corrupted, and rollback to that exact state.
lakeFS also assists teams with reprocessing of historical data. With lakeFS data can be reprocessed on an isolated branch, before merging, to ensure the reprocessed data is exposed atomically. It also makes it easier to access the different versions of reprocessed data using any tag or a historical commit ID.
Tune in to hear more about the benefits of lakeFS.
EPISODE LINKS

  continue reading

Luvut

1. Intro (00:00:00)

2. What is lakeFS? (00:02:49)

3. lakeFS vs. Git (00:05:18)

4. What is a Merkle Tree? (00:06:14)

5. What are some lakeFS use-cases? (00:08:06)

6. What data problems does lakeFS test? (00:18:47)

7. What types of customers or industries use lakeFS? (00:22:28)

8. lakeFS and Apache Kafka (00:23:28)

9. It's a wrap! (00:28:08)

265 jaksoa

Artwork
iconJaa
 
Manage episode 352916148 series 2355972
Sisällön tarjoaa Confluent, founded by the original creators of Apache Kafka® and Founded by the original creators of Apache Kafka®. Confluent, founded by the original creators of Apache Kafka® and Founded by the original creators of Apache Kafka® tai sen podcast-alustan kumppani lataa ja toimittaa kaiken podcast-sisällön, mukaan lukien jaksot, grafiikat ja podcast-kuvaukset. Jos uskot jonkun käyttävän tekijänoikeudella suojattua teostasi ilman lupaasi, voit seurata tässä https://fi.player.fm/legal kuvattua prosessia.

Is it possible to manage and test data like code? lakeFS is an open-source data version control tool that transforms object storage into Git-like repositories, offering teams a way to use the same workflows for code and data. In this episode, Kris sits down with guest Adi Polak, VP of DevX at Treeverse, to discuss how lakeFS can be used to facilitate better management and testing of data.
At its core, lakeFS provides teams with better data management. A theoretical data engineer on a large team runs a script to delete some data, but a bug in the script accidentally deletes a lot more data than intended. Application engineers can checkout the main branch, effectively erasing their mistakes, but without a tool like lakeFS, this data engineer would be in a lot of trouble.
Polak is quick to explain that lakeFS isn’t built on Git. The source code behind an application is usually a few dozen mega bytes, while lakeFS is designed to handle petabytes of data; however, it does use Git-like semantics to create and access versions so adoption is quick and simple.
Another big challenge that lakeFS helps teams tackle is reproducibility. Troubleshooting when and where a corruption in the data first appeared can be a tricky task for a data engineer, when data is constantly updating. With lakeFS, engineers can refer to snapshots to see where the product was corrupted, and rollback to that exact state.
lakeFS also assists teams with reprocessing of historical data. With lakeFS data can be reprocessed on an isolated branch, before merging, to ensure the reprocessed data is exposed atomically. It also makes it easier to access the different versions of reprocessed data using any tag or a historical commit ID.
Tune in to hear more about the benefits of lakeFS.
EPISODE LINKS

  continue reading

Luvut

1. Intro (00:00:00)

2. What is lakeFS? (00:02:49)

3. lakeFS vs. Git (00:05:18)

4. What is a Merkle Tree? (00:06:14)

5. What are some lakeFS use-cases? (00:08:06)

6. What data problems does lakeFS test? (00:18:47)

7. What types of customers or industries use lakeFS? (00:22:28)

8. lakeFS and Apache Kafka (00:23:28)

9. It's a wrap! (00:28:08)

265 jaksoa

Alle afleveringen

×
 
Loading …

Tervetuloa Player FM:n!

Player FM skannaa verkkoa löytääkseen korkealaatuisia podcasteja, joista voit nauttia juuri nyt. Se on paras podcast-sovellus ja toimii Androidilla, iPhonela, ja verkossa. Rekisteröidy sykronoidaksesi tilaukset laitteiden välillä.

 

Pikakäyttöopas