Artwork

Sisällön tarjoaa Confluent, founded by the original creators of Apache Kafka® and Founded by the original creators of Apache Kafka®. Confluent, founded by the original creators of Apache Kafka® and Founded by the original creators of Apache Kafka® tai sen podcast-alustan kumppani lataa ja toimittaa kaiken podcast-sisällön, mukaan lukien jaksot, grafiikat ja podcast-kuvaukset. Jos uskot jonkun käyttävän tekijänoikeudella suojattua teostasi ilman lupaasi, voit seurata tässä https://fi.player.fm/legal kuvattua prosessia.
Player FM - Podcast-sovellus
Siirry offline-tilaan Player FM avulla!

Apache Kafka 3.5 - Kafka Core, Connect, Streams, & Client Updates

11:25
 
Jaa
 

Manage episode 366177509 series 2355972
Sisällön tarjoaa Confluent, founded by the original creators of Apache Kafka® and Founded by the original creators of Apache Kafka®. Confluent, founded by the original creators of Apache Kafka® and Founded by the original creators of Apache Kafka® tai sen podcast-alustan kumppani lataa ja toimittaa kaiken podcast-sisällön, mukaan lukien jaksot, grafiikat ja podcast-kuvaukset. Jos uskot jonkun käyttävän tekijänoikeudella suojattua teostasi ilman lupaasi, voit seurata tässä https://fi.player.fm/legal kuvattua prosessia.

Apache Kafka® 3.5 is here with the capability of previewing migrations between ZooKeeper clusters to KRaft mode. Follow along as Danica Fine highlights key release updates.
Kafka Core:

  • KIP-833 provides an updated timeline for KRaft.
  • KIP-866 now is preview and allows migration from an existing ZooKeeper cluster to KRaft mode.
  • KIP-900 introduces a way to bootstrap the KRaft controllers with SCRAM credentials.
  • KIP-903 prevents a data loss scenario by preventing replicas with stale broker epochs from joining the ISR list.
  • KIP-915 streamlines the process of downgrading Kafka's transaction and group coordinators by introducing tagged fields.

Kafka Connect:

  • KIP-710 provides the option to use a REST API for internal server communication that can be enabled by setting `dedicated.mode.enable.internal.rest` equal to true.
  • KIP-875 offers support for native offset management in Kafka Connect. Connect cluster administrators can now read offsets for both source and sink connectors. This KIP adds a new STOPPED state for connectors, enabling users to shut down connectors and maintain connector configurations without utilizing resources.
  • KIP-894 makes `IncrementalAlterConfigs` API available for use in MirrorMaker 2 (MM2), adding a new use.incremental.alter.config configuration which takes values “requested,” “never,” and “required.”
  • KIP-911 adds a new source tag for metrics generated by the `MirrorSourceConnector` to help monitor mirroring deployments.

Kafka Streams:

  • KIP-339 improves Kafka Streams' error-handling capabilities by addressing serialization errors that occur before message production and extending the interface for custom error handling.
  • KIP-889 introduces versioned state stores in Kafka Streams for temporal join semantics in stream-to-table joins.
  • KIP-904 simplifies table aggregation in Kafka by proposing a change in serialization format to enable one-step aggregation and reduce noise from events with old and new keys/values.
  • KIP-914 modifies how versioned state stores are used in Kafka Streams. Versioned state stores may impact different DSL processors in varying ways, see the documentation for details.

Kafka Client:

  • KIP-881 is now complete and introduces new client-side assignor logic for rack-aware consumer balancing for Kafka Consumers.
  • KIP-887 adds the `EnvVarConfigProvider` implementation to Kafka so custom configurations stored in environment variables can be injected into the system by providing the map returned by `System.getEnv()`.
  • KIP 641 introduces the `RecordReader` interface to Kafka's clients module, replacing the deprecated MessageReader Scala trait.

EPISODE LINKS

  continue reading

Luvut

1. Intro (00:00:00)

2. KIP-833: KRaft Timeline Update (00:00:20)

3. KIP-866: ZooKeeper to KRaft Cluster Migration (00:01:08)

4. KIP-881: Rack-aware Partition Assignment for Kafka Consumers (00:01:34)

5. KIP-900: SASL SCRAM Support in KRaft (00:02:05)

6. KIP-903: Replicas with stale broker epoch should not be allowed to join the ISR (00:02:46)

7. KIP-915: Transactions and Group Coordinator Downgrade Foundation (00:03:41)

8. KIP-641: A new java interface to replace `kafka.common.MessageReader` (00:04:23)

9. KIP-887: Add ConfigProvider to make use of environment variables (00:04:48)

10. KIP-889: Add Versioned State Stores (00:05:20)

11. KIP-914: DSL Processor Semantics for Versioned Stores (00:06:03)

12. KIP-399: Extend `ProductionExceptionHandler` to Cover Serialization Errors (00:06:34)

13. KIP-904: Improve Table Aggregation Semantics (00:06:54)

14. KIP-710: Full Support for Distributed Mode in MM2 Clusters (00:07:58)

15. KIP-875: First-class offsets support in Kafka Connect (00:08:50)

16. KIP-894: Use incrementalAlterConfigs API for syncing topic configurations (00:09:20)

17. KIP-911: Add source tag to MirrorSourceConnector metrics (00:10:14)

18. It’s a wrap! (00:10:33)

265 jaksoa

Artwork
iconJaa
 
Manage episode 366177509 series 2355972
Sisällön tarjoaa Confluent, founded by the original creators of Apache Kafka® and Founded by the original creators of Apache Kafka®. Confluent, founded by the original creators of Apache Kafka® and Founded by the original creators of Apache Kafka® tai sen podcast-alustan kumppani lataa ja toimittaa kaiken podcast-sisällön, mukaan lukien jaksot, grafiikat ja podcast-kuvaukset. Jos uskot jonkun käyttävän tekijänoikeudella suojattua teostasi ilman lupaasi, voit seurata tässä https://fi.player.fm/legal kuvattua prosessia.

Apache Kafka® 3.5 is here with the capability of previewing migrations between ZooKeeper clusters to KRaft mode. Follow along as Danica Fine highlights key release updates.
Kafka Core:

  • KIP-833 provides an updated timeline for KRaft.
  • KIP-866 now is preview and allows migration from an existing ZooKeeper cluster to KRaft mode.
  • KIP-900 introduces a way to bootstrap the KRaft controllers with SCRAM credentials.
  • KIP-903 prevents a data loss scenario by preventing replicas with stale broker epochs from joining the ISR list.
  • KIP-915 streamlines the process of downgrading Kafka's transaction and group coordinators by introducing tagged fields.

Kafka Connect:

  • KIP-710 provides the option to use a REST API for internal server communication that can be enabled by setting `dedicated.mode.enable.internal.rest` equal to true.
  • KIP-875 offers support for native offset management in Kafka Connect. Connect cluster administrators can now read offsets for both source and sink connectors. This KIP adds a new STOPPED state for connectors, enabling users to shut down connectors and maintain connector configurations without utilizing resources.
  • KIP-894 makes `IncrementalAlterConfigs` API available for use in MirrorMaker 2 (MM2), adding a new use.incremental.alter.config configuration which takes values “requested,” “never,” and “required.”
  • KIP-911 adds a new source tag for metrics generated by the `MirrorSourceConnector` to help monitor mirroring deployments.

Kafka Streams:

  • KIP-339 improves Kafka Streams' error-handling capabilities by addressing serialization errors that occur before message production and extending the interface for custom error handling.
  • KIP-889 introduces versioned state stores in Kafka Streams for temporal join semantics in stream-to-table joins.
  • KIP-904 simplifies table aggregation in Kafka by proposing a change in serialization format to enable one-step aggregation and reduce noise from events with old and new keys/values.
  • KIP-914 modifies how versioned state stores are used in Kafka Streams. Versioned state stores may impact different DSL processors in varying ways, see the documentation for details.

Kafka Client:

  • KIP-881 is now complete and introduces new client-side assignor logic for rack-aware consumer balancing for Kafka Consumers.
  • KIP-887 adds the `EnvVarConfigProvider` implementation to Kafka so custom configurations stored in environment variables can be injected into the system by providing the map returned by `System.getEnv()`.
  • KIP 641 introduces the `RecordReader` interface to Kafka's clients module, replacing the deprecated MessageReader Scala trait.

EPISODE LINKS

  continue reading

Luvut

1. Intro (00:00:00)

2. KIP-833: KRaft Timeline Update (00:00:20)

3. KIP-866: ZooKeeper to KRaft Cluster Migration (00:01:08)

4. KIP-881: Rack-aware Partition Assignment for Kafka Consumers (00:01:34)

5. KIP-900: SASL SCRAM Support in KRaft (00:02:05)

6. KIP-903: Replicas with stale broker epoch should not be allowed to join the ISR (00:02:46)

7. KIP-915: Transactions and Group Coordinator Downgrade Foundation (00:03:41)

8. KIP-641: A new java interface to replace `kafka.common.MessageReader` (00:04:23)

9. KIP-887: Add ConfigProvider to make use of environment variables (00:04:48)

10. KIP-889: Add Versioned State Stores (00:05:20)

11. KIP-914: DSL Processor Semantics for Versioned Stores (00:06:03)

12. KIP-399: Extend `ProductionExceptionHandler` to Cover Serialization Errors (00:06:34)

13. KIP-904: Improve Table Aggregation Semantics (00:06:54)

14. KIP-710: Full Support for Distributed Mode in MM2 Clusters (00:07:58)

15. KIP-875: First-class offsets support in Kafka Connect (00:08:50)

16. KIP-894: Use incrementalAlterConfigs API for syncing topic configurations (00:09:20)

17. KIP-911: Add source tag to MirrorSourceConnector metrics (00:10:14)

18. It’s a wrap! (00:10:33)

265 jaksoa

Kaikki jaksot

×
 
Loading …

Tervetuloa Player FM:n!

Player FM skannaa verkkoa löytääkseen korkealaatuisia podcasteja, joista voit nauttia juuri nyt. Se on paras podcast-sovellus ja toimii Androidilla, iPhonela, ja verkossa. Rekisteröidy sykronoidaksesi tilaukset laitteiden välillä.

 

Pikakäyttöopas