Databases and data engineering episodes of Software Engineering Daily
…
continue reading
Databases underpin almost every user experience on the web, but scaling a database is one of the most fundamental infrastructure challenges in software development. PlanetScale offers a MySQL platform that is managed and highly scaleable. Sam Lambert is the CEO of PlanetScale and he joins the show to talk about why he started the platform, scaling …
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Iceberg at Netflix and Beyond with Ryan Blue
47:37
47:37
Toista myöhemmin
Toista myöhemmin
Listat
Tykkää
Tykätty
47:37
Apache Iceberg is an open source high-performance format for huge data tables. Iceberg enables the use of SQL tables for big data, while making it possible for engines like Spark and Hive to safely work with the same tables, at the same time. Iceberg was started at Netflix by Ryan Blue and Dan Weeks, and was open-sourced and donated to the Apache S…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Building a Data Lake with Adam Ferrari
46:19
46:19
Toista myöhemmin
Toista myöhemmin
Listat
Tykkää
Tykätty
46:19
Starburst is a data lake analytics platform. It’s designed to help users work with structured data at scale, and is built on the open source platform, Trino. Adam Ferrari is the SVP of Engineering at Starburst. He joins the show to talk about Starburst, data engineering, and what it takes to build a data lake. Full Disclosure: Starburst is a sponso…
…
continue reading
Building scalable software applications can be complex and typically requires dozens of different tools. The engineering often involves handling many arcane tasks that are distant from actual application logic. In addition, a lack of a cohesive model for building applications can lead to substantial engineering costs. Nathan Marz is the creator of …
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Bonus Episode: SurrealDB with Tobie Morgan Hitchcock
57:17
57:17
Toista myöhemmin
Toista myöhemmin
Listat
Tykkää
Tykätty
57:17
SurrealDB is the result of a long-time collaboration between brothers Tobie and Jaime Morgan Hitchcock. The project has modest origins and started merely to support other projects the brothers were working on. However, over time the project grew and in 2021 they started working on it full-time. Since then the project has gained serious adoption. Wh…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Tracking Drug Smugglers and Migrating Databases with Benny Keinan and Lior Resisi
50:40
50:40
Toista myöhemmin
Toista myöhemmin
Listat
Tykkää
Tykätty
50:40
Maritime logistics is the process organizing the movement of goods across the ocean. Historically, this has been a challenging problem because of the multinational nature of shipping, as well as piracy, smuggling, and legacy technology. It’s also profoundly important for security reasons, and because 90% of what we buy travels over the oceans. Ocea…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
The Right to Be Forgotten with Gal Ringel
47:45
47:45
Toista myöhemmin
Toista myöhemmin
Listat
Tykkää
Tykätty
47:45
Data breaches at major companies are so now common that they hardly make the news. The Wikipedia page on data breaches lists over 350 between 2004 and 2023. The Equifax breach in 2017 was especially notable because over 160 million records were leaked, and much of the data was acquired by Equifax without individuals’ knowledge or consent. Data brea…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Sofascore with Josip Stuhli
49:48
49:48
Toista myöhemmin
Toista myöhemmin
Listat
Tykkää
Tykätty
49:48
If you’re a sports fan and like to track sports statistics and results, you’ve probably heard of Sofascore. The website started in 2010 and ran on a modest single server. It now has 25 million monthly active users, covers 20 different sports, 11,000 leagues and tournaments, and is available in over 30 languages. Josip Stuhli has been with Sofascore…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Daytona with Ivan Burazin
47:50
47:50
Toista myöhemmin
Toista myöhemmin
Listat
Tykkää
Tykätty
47:50
Cloud-based software development platforms such as GitHub Codespaces continue to grow in popularity. These platforms are attractive to enterprise organizations because they can be managed centrally with security controls. However, many, if not most, developers prefer a local IDE. Daytona is aiming to bridge that gap. It’s a layer between a local ID…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
GraphAware with Luanne Misquitta
57:51
57:51
Toista myöhemmin
Toista myöhemmin
Listat
Tykkää
Tykätty
57:51
Knowledge graphs are an intuitive way to define relationships between objects, events, situations, and concepts. Their ability to encode this information makes them an attractive database paradigm. Hume is a graph-based analysis solution developed by GraphAware. It represents data as a network of interconnected entities and provides analysis capabi…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Chronosphere with Martin Mao
48:08
48:08
Toista myöhemmin
Toista myöhemmin
Listat
Tykkää
Tykätty
48:08
Observability software helps teams to actively monitor and debug their systems, and these tools are increasingly vital in DevOps. However, it’s not uncommon for the volume of observability data to exceed the amount of actual business data. This creates two challenges – how to analyze the large stream of observability data, and how to keep down the …
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Streamlit with Amanda Kelly
47:06
47:06
Toista myöhemmin
Toista myöhemmin
Listat
Tykkää
Tykätty
47:06
The importance of data teams is undeniable. Most companies today use data to drive decision-making on anything from software feature development to product strategy, hiring and marketing. In some companies data is the product, which can make data teams even more vital. But there’s a common problem – analyzing data is hard and time consuming. Lots o…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Modern Web Scraping with Erez Naveh
57:29
57:29
Toista myöhemmin
Toista myöhemmin
Listat
Tykkää
Tykätty
57:29
Today it’s estimated there are over 1 billion websites on the internet. Much of this content is optimized to be viewed by human eyes, not consumed by machines. However, creating systems to automatically parse and structure the web greatly extends its utility, and paves the way for innovative solutions and applications. The industry of web scraping …
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Observability with Eduardo Silva
44:47
44:47
Toista myöhemmin
Toista myöhemmin
Listat
Tykkää
Tykätty
44:47
There are hundreds of observability companies out there, and many ways to think about observability, such as application performance monitoring, server monitoring, and tracing. In a production application, multiple tools are often needed to get proper visibility on the application. This creates some challenges. Applications can produce lots of diff…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
AI and Business Analytics with John Adams
30:03
30:03
Toista myöhemmin
Toista myöhemmin
Listat
Tykkää
Tykätty
30:03
It’s now clear that the adoption of AI will continue to increase, with nearly every industry working to rapidly incorporate it into their systems and applications to provide greater value to their users. Business analytics is a key domain that promises to be radically reshaped by AI. Alembic is an AI platform that integrates web data, product conve…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Highly Scalable NoSQL with Dor Laor
36:27
36:27
Toista myöhemmin
Toista myöhemmin
Listat
Tykkää
Tykätty
36:27
ScyllaDB is a fast and highly scalable NoSQL database designed to provide predictable performance at a massive cloud scale. It can handle millions of operations per second at a scale of gigabytes or petabytes. It’s also designed to be compatible with Cassandra and DynamoDB APIs. Scylla is used by Zillow, Comcast, and for Discord’s 350M+ users, and …
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Database Caching with Ben Hagan
35:36
35:36
Toista myöhemmin
Toista myöhemmin
Listat
Tykkää
Tykätty
35:36
Database caching is a fundamental challenge in database management and there are hundreds of techniques to satisfy different caching scenarios. PolyScale is a fully automated database cache. It offers an innovative approach to database caching, leveraging AI and automated configuration to simplify the process of determining what should and should n…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Data-Centric AI with Alex Ratner
50:19
50:19
Toista myöhemmin
Toista myöhemmin
Listat
Tykkää
Tykätty
50:19
Companies have high hopes for Machine learning and AI to support real-time product offerings, prevent fraud and drive innovation. But there was a catch – training models require labeled data that machines can digest. As data volumes increase, the opportunity to get great ML results rises, but so does the problem of labeling all the data to get that…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Making Data-Driven Decisions with Soumyadeb Mitra
50:45
50:45
Toista myöhemmin
Toista myöhemmin
Listat
Tykkää
Tykätty
50:45
RudderStack is a warehouse-native customer data platform (CDP) that helps businesses collect, unify, and activate customer data from all their different sources. In today’s episode, we’re talking to Soumyadeb Mitra, the founder and CEO of RudderStack. We discuss the importance of activating all your data, how RudderStack can help you activate your …
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Customer-facing Analytics with Tyler Wells
51:38
51:38
Toista myöhemmin
Toista myöhemmin
Listat
Tykkää
Tykätty
51:38
The state of Data inside most companies is chaotic. It takes significant time and investment to tame this chaos. When you are a platform provider you are gathering tons of data from the developers using your platform. These developers building products on your platform need insight into that data to better understand how their application is perfor…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Data Reliability with Barr Moses and Lior Gavish
56:22
56:22
Toista myöhemmin
Toista myöhemmin
Listat
Tykkää
Tykätty
56:22
As companies depend more on data to improve digital products and make informed decisions, it’s crucial that the data they use be accurate and reliable. MonteCarlo, the data reliability company, is the creator of the industry’s first end-to-end data observability platform. Barr Moses and Lior Gavish are the founders of Monte Carlo and they join us t…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Low-Code SQL on dbt Core with Raj Bains from Prophecy
54:15
54:15
Toista myöhemmin
Toista myöhemmin
Listat
Tykkää
Tykätty
54:15
In this podcast episode, we take a look at the intricacies of low-code data pipelines with Raj Bains, the founder of Prophecy.io. Raj shares valuable insights into how performant low-codedata pipelines are revolutionizing industries and transforming everyday operations. Raj discusses the founding story of Prophecy.io, the company’s mission, and its…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Open-Source Embedding Database with Anton Troynikov
32:37
32:37
Toista myöhemmin
Toista myöhemmin
Listat
Tykkää
Tykätty
32:37
Chroma is an open source embedding database that is designed to make it easy to build large language model applications by making knowledge, facts and skills pluggable. Anton Troynikov is the co-founder of Chroma and he is our guest today. This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cl…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Data Activation with Tejas Manohar
41:10
41:10
Toista myöhemmin
Toista myöhemmin
Listat
Tykkää
Tykätty
41:10
Data Activation is the method of unlocking the knowledge sorted within your data warehouse, and making it actionable by your business users in the end tools that they use every day. In doing so, Data Activation helps bring data people toward the center of the business, directly tying their work to business outcomes. Hightouch is the simplest and fa…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Self-Service Data Culture with Stemma’s Mark Grover
46:26
46:26
Toista myöhemmin
Toista myöhemmin
Listat
Tykkää
Tykätty
46:26
A data catalog provides an index into the data sets and schemas of a company.Data teams are growing in size, and more companies than ever have a data team, so the market for data catalog is larger than ever. Mark is the CEO of Stemma and the co-creator of Amundsen, a data catalog that came out of Lyft. In today’s show Mark shares how his history as…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Streaming Analytics with Hojjat Jafarpour
46:48
46:48
Toista myöhemmin
Toista myöhemmin
Listat
Tykkää
Tykätty
46:48
Streaming analytics refers to the process of analyzing real-time data that is generated continuously and rapidly from various sources, such as sensors, applications, social media, and other internet-connected devices. Streaming analytics platforms enable organizations to extract business value from data in motion, similar to how traditional analyti…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Turso: Globally Replicated SQLite with Glauber Costa
51:00
51:00
Toista myöhemmin
Toista myöhemmin
Listat
Tykkää
Tykätty
51:00
Distributed databases are necessary for storing and managing data across multiple nodes in a network. They provide scalability, fault tolerance, improved performance, and cost savings. By distributing data across nodes, they allow for efficient processing of large amounts of data and redundancy against failures. They can also be used to store data …
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Observability Trends with John Hart
26:58
26:58
Toista myöhemmin
Toista myöhemmin
Listat
Tykkää
Tykätty
26:58
DataSet is a log analytics platform provided by Sentinel One that helps DevOps, IT engineering, and security teams get answers from their data across all time periods, both live streaming and historical. It’s powered by a unique architecture that uses a massively parallel query engine to provide actionable insights from the data available. John Har…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Data Investing and the MAD with Matt Turck
51:08
51:08
Toista myöhemmin
Toista myöhemmin
Listat
Tykkää
Tykätty
51:08
There are many types of early stage funding available from friends and family to seed to series A. Some firms invest across a wide set of technologies and seek only to provide capital. Others are in it for the long haul – they focus on specific areas of technology and develop both long term relationships and deep expertise over time. Today, we are …
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Accessing Data at Scale with Justin Borgman
46:18
46:18
Toista myöhemmin
Toista myöhemmin
Listat
Tykkää
Tykätty
46:18
The Presto/Trino project makes distributed querying easier across a variety of data sources. As the need for machine learning and other high volume data applications has increased, the need for support, tooling, and cloud infrastructure for Presto/Trino has increased with it. Starburst helps your teams run fast queries on any data source. With Star…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Building on the Data Cloud with Torsten Grabs
40:03
40:03
Toista myöhemmin
Toista myöhemmin
Listat
Tykkää
Tykätty
40:03
Building and managing data-intensive applications has traditionally been costly and complex, and has placed an operational burden on developers to maintain as their organization scales. Todays’ developers, data scientists, and data engineers need a streamlined, single cloud data platform for building applications, pipelines, and machine learning mo…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Serverless Clickhouse for Developers with Jorge Sancha
35:14
35:14
Toista myöhemmin
Toista myöhemmin
Listat
Tykkää
Tykätty
35:14
Data analytics technology and tools have seen significant improvements in the past decade. But, it can still take weeks to prototype, build and deploy new transformations and deployments, usually requiring considerable engineering resources. Plus, most data isn’t real-time. Instead, most of it is still batch-processed. Tinybird Analytics provides a…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Data Infrastructure for Finance
54:26
54:26
Toista myöhemmin
Toista myöhemmin
Listat
Tykkää
Tykätty
54:26
/*! elementor - v3.21.0 - 26-05-2024 */.elementor-widget-text-editor.elementor-drop-cap-view-stacked .elementor-drop-cap{background-color:#69727d;color:#fff}.elementor-widget-text-editor.elementor-drop-cap-view-framed .elementor-drop-cap{color:#69727d;border:3px solid;background-color:transparent}.elementor-widget-text-editor:not(.elementor-drop-ca…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Faking Data Using Tonic.ai with Ian Coe and Adam Kamor
46:49
46:49
Toista myöhemmin
Toista myöhemmin
Listat
Tykkää
Tykätty
46:49
Ian Coe CEO Adam Kamor Head of Engineering Companies that gather data about their users have an ethical obligation and legal responsibility to protect the personally identifiable information in their dataset. Ideally, developers working on a software application wouldn’t need access to production data. Yet without high-quality example data, many te…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Couchbase with Ravi Mayuram
30:08
30:08
Toista myöhemmin
Toista myöhemmin
Listat
Tykkää
Tykätty
30:08
Couchbase is a distributed NoSQL cloud database. Since its creation, Couchbase has expanded into edge computing, application services, and most recently, a database-as-a-service called Capella. Couchbase started as an in-memory cache and needed to be rearchitected to be a persistent storage system. In this episode, We interviewed Ravi Mayuram, SVP …
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Decodable Streaming with Eric Sammer
44:58
44:58
Toista myöhemmin
Toista myöhemmin
Listat
Tykkää
Tykätty
44:58
Streaming data platforms like Kafka, Pulsar, and Kinesis are now common in mainstream enterprise architectures, providing low-latency real-time messaging for analytics and applications. However, stream processing – the act of filtering, transforming, or analyzing the data inside the messages – is still an exercise left to the receiving microservice…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Data Delivery with Naqeeb Memon
28:14
28:14
Toista myöhemmin
Toista myöhemmin
Listat
Tykkää
Tykätty
28:14
Data-as-a-service is a company category type that is not as common as API-as-a-service, software-as-a-service, or platform-as-a-service. In order to vend data, a data-as-a-service provider needs to define how that data will be priced, stored, and delivered to users: streaming over an API or served via static files. Naqeeb Memon of Safegraph joins t…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Data Labeling with Michael Malyuk
41:51
41:51
Toista myöhemmin
Toista myöhemmin
Listat
Tykkää
Tykätty
41:51
Data labeling allows machine learning algorithms to find patterns among the data. There are a variety of data labeling platforms that enable humans to apply labels to this data and ready it for algorithms. Heartex is a data labeling platform with an open source core. Michael Malyuk joins the show to talk through the platform and modern usage of dat…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Pinot and StarTree with Chinmay Soman
44:17
44:17
Toista myöhemmin
Toista myöhemmin
Listat
Tykkää
Tykätty
44:17
Real-time analytics are difficult to achieve because large amounts of data must be integrated into a data set as that data streams in. As the world moved from batch analytics powered by Hadoop into a norm of “real-time” analytics, a variety of open source systems emerged. One of these was Apache Pinot. StarTree is a company based on Apache Pinot th…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Data Loss Prevention with Yasir Ali
40:55
40:55
Toista myöhemmin
Toista myöhemmin
Listat
Tykkää
Tykätty
40:55
Data loss can occur when large data sources such as Slack or Google Drive get leaked. In order to detect and avoid leaks, a data asset graph can be built to understand the risks of a company environment. Polymer is a data loss prevention product that helps companies avoid problematic data leaks. Yasir Ali is the founder of Polymer and joins the sho…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Airbyte Engineering with Michel Tricot
42:26
42:26
Toista myöhemmin
Toista myöhemmin
Listat
Tykkää
Tykätty
42:26
Data integration infrastructure is not easy to build. Moving large amounts of data from one place to another has historically required developers to build ad hoc integration points to move data between SaaS services, data lakes, and data warehouses. Today, there are dedicated systems and services for moving these large batches of data. Airbyte buil…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Select Star with Shinji Kim
42:47
42:47
Toista myöhemmin
Toista myöhemmin
Listat
Tykkää
Tykätty
42:47
Modern organizations eventually face data governance challenges. Keeping track of where data came from, what systems update it, in what ways updates can be made are just some of the issues to be tackled. Large organizations face additional challenges like training, onboarding, and capturing the institutional knowledge that leaves with the departure…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Time Series IoT on InfluxDB with Brian Gilmore
48:30
48:30
Toista myöhemmin
Toista myöhemmin
Listat
Tykkää
Tykätty
48:30
The solution many turn to for capturing their streaming data is InfluxDB. In this episode, I interview Brian Gilmore, Director of Product Management at InfluxData, about how real time applications achieve success built on top of InfluxDB. When most people hear the phrase Internet of Things, it typically evokes an image of connected devices we insta…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Data Engineering Trends with Lior Gavish and James Densmore
43:56
43:56
Toista myöhemmin
Toista myöhemmin
Listat
Tykkää
Tykätty
43:56
Lior Gavish James Densmore Data infrastructure is a fast-moving sector of the software market. As the volume of data has increased, so too has the quality of tooling to support data management and data engineering. In today’s show, we have a guest from a data intensive company as well as a company that builds a popular data engineering product. Jam…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
PlanetScale Management with Sam Lambert
49:11
49:11
Toista myöhemmin
Toista myöhemmin
Listat
Tykkää
Tykätty
49:11
Running a database company requires expertise in both technical and managerial skills. There are deeply technical engineering questions around query paths, scalability, and distributed systems. And there are complex managerial questions around developer productivity and task allocation. Sam Lambert is the CEO of PlanetScale, which is building moder…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
SingleStore with Jordan Tigani
42:57
42:57
Toista myöhemmin
Toista myöhemmin
Listat
Tykkää
Tykätty
42:57
SingleStore is a multi-use, multi-model database designed for transactional and analytic workloads, as well as search and other domain specific applications. SingleStore is the evolution of the database company MemSQL, which sought to bring fast, in-memory SQL database technology to market. Jordan Tigani is Chief Product Officer of SingleStore and …
…
continue reading
![Artwork](/static/images/128pixel.png)
1
DuckDB with Hannes Muleisen
49:02
49:02
Toista myöhemmin
Toista myöhemmin
Listat
Tykkää
Tykätty
49:02
DuckDB is a relational database management system with no external dependencies, with a simple system for deployment and integration into build processes. It enables complex queries in SQL with a large function library, and provides transactional guarantees through multi-version concurrency control. Hannes Mühleisen works on DuckDB and joins the sh…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
RudderStack Engineering with Soumaydeb Mitra
46:54
46:54
Toista myöhemmin
Toista myöhemmin
Listat
Tykkää
Tykätty
46:54
Customer data pipelines power the backend of many successful web platforms. In a customer data pipeline, data is collected from sources such as mobile apps and cloud SaaS tools, transformed and munged using data engineering, stored in data warehouses, and piped to analytics, advertising platforms, and data infrastructure. RudderStack is an open sou…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Apache Hudi with Vinoth Chandar
43:03
43:03
Toista myöhemmin
Toista myöhemmin
Listat
Tykkää
Tykätty
43:03
The data lake architecture has become broadly adopted in a relatively short period of time. In a nutshell, that means data in it’s raw format stored in cloud object storage. Modern software and data engineers have no shortage of options for accessing their data lake, but that list shrinks quickly if you care about features like transactions. Apache…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Data Catalog in Practice with Mark Grover
51:38
51:38
Toista myöhemmin
Toista myöhemmin
Listat
Tykkää
Tykätty
51:38
A data catalog provides an index into the data sets and schemas of a company. Data teams are growing in size, and more companies than ever have a data team, so the market for data catalog is larger than ever. Mark is the CEO of Stemma and the co-creator of Amundsen, a data catalog that came out of Lyft. We have previously explored the basics of Amu…
…
continue reading