-
Notifications
You must be signed in to change notification settings - Fork 1
feat: update quadrant classifications for Databricks, DLT, Snowflake,… #102
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,28 +1,28 @@ | ||
| { | ||
| "name": "info-techradar", | ||
| "version": "4.0.0", | ||
| "main": "index.js", | ||
| "license": "Creative Commons Attribution-NonCommercial-ShareAlike", | ||
| "scripts": { | ||
| "build": "techradar build", | ||
| "serve": "nodemon" | ||
| }, | ||
| "dependencies": { | ||
| "aoe_technology_radar": "4.4.0" | ||
| }, | ||
| "browserslist": { | ||
| "production": [ | ||
| ">0.2%", | ||
| "not dead", | ||
| "not op_mini all" | ||
| ], | ||
| "development": [ | ||
| "last 1 chrome version", | ||
| "last 1 firefox version", | ||
| "last 1 safari version" | ||
| ] | ||
| }, | ||
| "devDependencies": { | ||
| "nodemon": "^3.1.9" | ||
| } | ||
| "name": "info-techradar", | ||
| "version": "4.0.0", | ||
| "main": "index.js", | ||
| "license": "Creative Commons Attribution-NonCommercial-ShareAlike", | ||
| "scripts": { | ||
| "build": "techradar build", | ||
| "serve": "nodemon" | ||
| }, | ||
| "dependencies": { | ||
| "aoe_technology_radar": "4.6.1" | ||
| }, | ||
| "browserslist": { | ||
| "production": [ | ||
| ">0.2%", | ||
| "not dead", | ||
| "not op_mini all" | ||
| ], | ||
| "development": [ | ||
| "last 1 chrome version", | ||
| "last 1 firefox version", | ||
| "last 1 safari version" | ||
| ] | ||
| }, | ||
| "devDependencies": { | ||
| "nodemon": "^3.1.9" | ||
| } | ||
| } |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,7 +1,7 @@ | ||
| --- | ||
| title: "Databricks" | ||
| ring: trial | ||
| quadrant: tools | ||
| quadrant: platforms-and-operations | ||
| featured: true | ||
| --- | ||
|
|
||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| --- | ||
| title: "Delta Live Tables (DLT)" | ||
| ring: trial | ||
| quadrant: languages-and-frameworks | ||
| featured: true | ||
| --- | ||
|
|
||
| [Delta Live Tables (DLT)](https://docs.databricks.com/delta-live-tables/index.html) is a declarative framework for building reliable, maintainable, and testable data processing pipelines. It simplifies ETL development by using a simple declarative approach to build and manage data pipelines on the Databricks platform. | ||
|
|
||
| ### Why Delta Live Tables? | ||
|
|
||
| - **Declarative Approach:** Define data transformations using SQL or Python with a simple, declarative syntax that focuses on what rather than how. | ||
|
|
||
| - **Automatic Quality Management:** Built-in data quality controls with expectations that can quarantine bad data and provide monitoring insights. | ||
|
|
||
| - **Pipeline Orchestration:** Automatically handles dependencies, scheduling, and error recovery without manual intervention. | ||
|
|
||
| - **Performance Optimization:** Leverages Delta Lake's ACID transactions and automatic optimization features for reliable and fast data processing. | ||
|
|
||
| ### Considerations at INFO | ||
|
|
||
| - **ETL Modernization:** Being trialed as part of modernizing data pipeline development, particularly for complex transformation workflows. | ||
|
|
||
| - **Integration with Databricks:** Evaluating as a complement to our [Databricks](/tools/databricks) platform trials, leveraging its native integration capabilities. | ||
|
|
||
| - **Data Quality Focus:** Assessing built-in data quality features for ensuring reliable data pipelines in production environments. | ||
|
|
||
| - **Current Focus:** Testing developer productivity improvements, pipeline reliability, and operational simplicity compared to traditional ETL approaches. | ||
|
|
||
| Delta Live Tables' declarative pipeline approach and integration with the Databricks ecosystem make it a compelling framework as we trial modern approaches to data pipeline development. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,7 +1,7 @@ | ||
| --- | ||
| title: "Snowflake" | ||
| ring: assess | ||
| quadrant: tools | ||
| quadrant: platforms-and-operations | ||
| featured: true | ||
| --- | ||
|
|
||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| --- | ||
| title: "Apache Spark" | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Aha, Apache Spark is back on our radar. :-) We had it on our radar in the past, not sure in which context exactly. In our data space I think indeed it makes good sense to use it. Just checking: I would have thought DataBricks would offer it's own data analytics solution? If they do, might be good to explain why we would still want to trial Apache Spark?
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think this blip should be in a different new 2025-.. folder and not 2024-12-01? |
||
| ring: trial | ||
| quadrant: languages-and-frameworks | ||
| featured: true | ||
| --- | ||
|
|
||
| [Apache Spark](https://spark.apache.org/) is an open-source unified analytics engine for large-scale data processing. It provides high-level APIs in Java, Scala, Python, and R, and an optimized engine that supports general computation graphs for data analysis. | ||
|
|
||
| ### Why Apache Spark? | ||
|
|
||
| - **Unified Engine:** Provides a single framework for batch processing, real-time streaming, machine learning, and graph processing. | ||
|
|
||
| - **High Performance:** In-memory computing capabilities significantly speed up data processing compared to traditional disk-based systems. | ||
|
|
||
| - **Multi-Language Support:** Native support for Java, Scala, Python (PySpark), and R (SparkR), enabling diverse development teams to work with familiar languages. | ||
|
|
||
| - **Extensive Ecosystem:** Rich ecosystem of libraries including Spark SQL, MLlib for machine learning, GraphX for graph processing, and Structured Streaming. | ||
|
|
||
| ### Considerations at INFO | ||
|
|
||
| - **Big Data Processing:** Being trialed for large-scale data processing workloads that require distributed computing capabilities. | ||
|
|
||
| - **Integration Assessment:** Evaluating integration with existing data infrastructure and cloud platforms, particularly in conjunction with [Databricks](/tools/databricks). | ||
|
|
||
| - **Performance Evaluation:** Testing performance improvements over traditional data processing approaches, especially for complex analytical workloads. | ||
|
|
||
| - **Current Focus:** Assessing developer productivity, scalability benefits, and operational complexity in production environments. | ||
|
|
||
| Apache Spark's unified analytics capabilities and performance characteristics make it a promising framework as we trial its potential for enterprise-scale data processing initiatives. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| --- | ||
| title: 'Contentful CMS' | ||
| ring: adopt | ||
| quadrant: tools | ||
| featured: true | ||
| --- | ||
|
|
||
| [Contentful](https://www.contentful.com/) is our preferred headless CMS for managing content across diverse digital platforms. With its robust API-first architecture and user-friendly interface, Contentful streamlines content management and delivery, enabling teams to work efficiently and collaboratively. | ||
|
|
||
| ### Why Contentful? | ||
|
|
||
| - **Headless Architecture:** Facilitates omnichannel content delivery with a robust set of APIs, ideal for websites, mobile apps, and other digital products. | ||
|
|
||
| - **Developer and Editor Friendly:** Provides easy integration with SDKs for developers and an intuitive UI for content creators, supporting parallel workflows. | ||
|
|
||
| - **Scalability:** Highly scalable for enterprise-level projects, with excellent localization and workflow management capabilities. | ||
|
|
||
| - **Content Modeling:** Enables structured content management, allowing complex and reusable content models that fit various use cases. | ||
|
|
||
| ### Considerations at INFO | ||
|
|
||
| - **Proven Solution:** Contentful continues as our adopted SaaS-based CMS solution, demonstrating consistent value across multiple projects and use cases. | ||
|
|
||
| - **Strategic Choice:** Selected as our primary SaaS CMS over other alternatives, providing the right balance of features, usability, and enterprise capabilities. | ||
|
|
||
| - **Production Success:** Successfully deployed across marketing websites, e-commerce platforms, and content-driven applications with reliable performance and user satisfaction. | ||
|
|
||
| - **Current Focus:** Maintaining our investment in Contentful while leveraging its advanced features for complex content workflows and multi-channel publishing strategies. | ||
|
|
||
| Contentful's continued adoption reinforces our commitment to efficient, scalable, and user-friendly content management solutions at INFO. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| --- | ||
| title: "Databricks" | ||
| ring: adopt | ||
| quadrant: platforms-and-operations | ||
| featured: true | ||
| --- | ||
|
|
||
| Databricks is a unified analytics platform that accelerates innovation by unifying data science, engineering, and business analytics. Built by the original creators of Apache Spark™, it offers a cloud-based environment for processing large-scale data, running machine learning models, and enabling real-time analytics. | ||
|
|
||
| ### Why Databricks? | ||
|
|
||
| - **Unified Platform:** Combines data engineering, data science, and business analytics in one collaborative workspace. | ||
|
|
||
| - **Scalable Processing:** Optimized for big data with Apache Spark™, allowing for efficient processing of large datasets. | ||
|
|
||
| - **Collaborative Notebooks:** Features interactive notebooks with real-time co-authoring, version control, and support for multiple programming languages. | ||
|
|
||
| - **Machine Learning Integration:** Provides seamless integration with popular ML frameworks like TensorFlow, PyTorch, and scikit-learn. | ||
|
|
||
| ### Considerations at INFO | ||
|
|
||
| - **Production Ready:** After successful trials, Databricks has proven its value in production environments for complex data processing and machine learning workflows. | ||
|
|
||
| - **Enhanced Collaboration:** The platform has significantly improved collaboration between data engineers, data scientists, and business analysts across teams. | ||
|
|
||
| - **Integration Success:** Successfully integrated with existing data pipelines and cloud infrastructure, demonstrating reliability and scalability. | ||
|
|
||
| - **Current Focus:** Now adopted as our primary platform for advanced analytics, big data processing, and machine learning model development and deployment. | ||
|
|
||
| Databricks' comprehensive analytics capabilities and proven performance in production make it our adopted solution for enterprise-scale data science and analytics initiatives. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,32 @@ | ||
| --- | ||
| title: "Delta Live Tables (DLT)" | ||
| ring: adopt | ||
| quadrant: languages-and-frameworks | ||
| featured: true | ||
| --- | ||
|
|
||
| [Delta Live Tables (DLT)](https://docs.databricks.com/delta-live-tables/index.html) is a declarative framework for building reliable, maintainable, and testable data processing pipelines. It simplifies ETL development by using a simple declarative approach to build and manage data pipelines on the Databricks platform. | ||
|
|
||
| ### Why Delta Live Tables? | ||
|
|
||
| - **Declarative Approach:** Define data transformations using SQL or Python with a simple, declarative syntax that focuses on what rather than how. | ||
|
|
||
| - **Automatic Quality Management:** Built-in data quality controls with expectations that can quarantine bad data and provide monitoring insights. | ||
|
|
||
| - **Pipeline Orchestration:** Automatically handles dependencies, scheduling, and error recovery without manual intervention. | ||
|
|
||
| - **Performance Optimization:** Leverages Delta Lake's ACID transactions and automatic optimization features for reliable and fast data processing. | ||
|
|
||
| ### Considerations at INFO | ||
|
|
||
| - **Production Ready:** After successful trials, Delta Live Tables has proven its value for building reliable and maintainable data pipelines in production environments. | ||
|
|
||
| - **ETL Transformation:** Successfully adopted as our standard approach for modern data pipeline development, significantly improving developer productivity. | ||
|
|
||
| - **Quality Assurance:** Built-in data quality features have enhanced reliability and monitoring of our data pipelines, reducing manual oversight requirements. | ||
|
|
||
| - **Platform Integration:** Seamlessly integrated with our adopted [Databricks](/platforms-and-operations/databricks) platform and [Apache Spark](/languages-and-frameworks/spark) framework. | ||
|
|
||
| - **Current Focus:** Now adopted as our preferred framework for declarative data pipeline development, enabling teams to focus on business logic rather than infrastructure complexity. | ||
|
|
||
| Delta Live Tables' declarative approach and proven reliability make it our adopted solution for modern, maintainable data pipeline development across the organization. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| --- | ||
| title: 'Sanity CMS' | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I guess this is Looks like an interesting product for sure. I would be interested to know our use cases for it and how it differs from Contentful. Maybe you can give me a demo some day? :-) |
||
| ring: assess | ||
| quadrant: tools | ||
| featured: true | ||
| --- | ||
|
|
||
| [Sanity](https://www.sanity.io) stands out for its customizable studio, real-time collaboration, and schema-driven content management. After initial trials, we continue to assess its features and potential fit within our content management ecosystem. | ||
|
|
||
| ### Why Sanity? | ||
|
|
||
| - **Highly Customizable Studio:** Studio can be tailored to fit diverse editorial needs, with custom inputs, plugins, and preview features. | ||
|
|
||
| - **Real-Time Collaboration:** Supports simultaneous editing and comprehensive version control, enhancing productivity in content teams. | ||
|
|
||
| - **API and GraphQL Support:** Provides flexible content delivery solutions that adapt to varying project requirements. | ||
|
|
||
| - **Structured Content:** Schema-driven approach allows for creating highly organized and reusable content. | ||
|
|
||
| ### Considerations at INFO | ||
|
|
||
| - **Extended Assessment:** Following initial trials, Sanity remains under assessment as we evaluate its long-term fit alongside our adopted [Contentful](/tools/contentful) solution. | ||
|
|
||
| - **Specialized Use Cases:** Being evaluated for specific scenarios requiring extensive customization and advanced editorial workflows that may complement our existing CMS strategy. | ||
|
|
||
| - **Technical Evaluation:** Assessing integration complexity, development overhead, and maintenance requirements compared to our current content management approach. | ||
|
|
||
| - **Current Focus:** Determining whether Sanity's advanced customization capabilities justify adoption for specialized content management needs. | ||
|
|
||
| Sanity's powerful customization and collaboration features continue to be assessed for specific use cases where advanced editorial capabilities might provide additional value beyond our primary CMS solution. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| --- | ||
| title: "Snowflake" | ||
| ring: assess | ||
| quadrant: platforms-and-operations | ||
| featured: true | ||
| --- | ||
|
|
||
| [Snowflake](https://www.snowflake.com/en/data-cloud/platform/) is a cloud-based data warehousing platform that offers scalable storage and compute resources for big data processing and analytics. Designed specifically for the cloud, it provides a flexible and efficient solution for data warehousing, data lakes, and data sharing. | ||
|
|
||
| ### Why Snowflake? | ||
|
|
||
| - **Scalable Architecture:** Separates compute and storage, allowing independent scaling and cost optimization based on workloads. | ||
|
|
||
| - **Multi-Cloud Availability:** Operates across major cloud providers like [AWS](/platforms-and-operations/aws), [Azure](/platforms-and-operations/azure), and Google Cloud, offering flexibility and avoiding vendor lock-in. | ||
|
|
||
| - **Data Sharing and Collaboration:** Enables secure and instant data sharing within and outside the organization without the need to move or copy data. | ||
|
|
||
| - **Performance Optimization:** Automatically manages performance tuning, indexing, and partitioning, reducing administrative overhead. | ||
|
|
||
| ### Considerations at INFO | ||
|
|
||
| - **Continued Assessment:** As a cloud-native data platform, Snowflake remains under evaluation for enterprise data warehousing needs, particularly in comparison to our adopted [Databricks](/platforms-and-operations/databricks) solution. | ||
|
|
||
| - **Platform Positioning:** Recognized as a platforms and operations solution rather than just a tool, given its infrastructure-level capabilities and cloud-native architecture. | ||
|
|
||
| - **Current Focus:** Evaluating specific use cases where Snowflake's data warehousing capabilities might complement our existing data infrastructure, including integration with cloud platforms and cost optimization strategies. | ||
|
|
||
| Snowflake's cloud-native data platform continues to be assessed for specific enterprise data warehousing scenarios where its unique architecture might provide additional value to our data operations. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,32 @@ | ||
| --- | ||
| title: "Apache Spark" | ||
| ring: adopt | ||
| quadrant: languages-and-frameworks | ||
| featured: true | ||
| --- | ||
|
|
||
| [Apache Spark](https://spark.apache.org/) is an open-source unified analytics engine for large-scale data processing. It provides high-level APIs in Java, Scala, Python, and R, and an optimized engine that supports general computation graphs for data analysis. | ||
|
|
||
| ### Why Apache Spark? | ||
|
|
||
| - **Unified Engine:** Provides a single framework for batch processing, real-time streaming, machine learning, and graph processing. | ||
|
|
||
| - **High Performance:** In-memory computing capabilities significantly speed up data processing compared to traditional disk-based systems. | ||
|
|
||
| - **Multi-Language Support:** Native support for Java, Scala, Python (PySpark), and R (SparkR), enabling diverse development teams to work with familiar languages. | ||
|
|
||
| - **Extensive Ecosystem:** Rich ecosystem of libraries including Spark SQL, MLlib for machine learning, GraphX for graph processing, and Structured Streaming. | ||
|
|
||
| ### Considerations at INFO | ||
|
|
||
| - **Production Success:** After successful trials, Apache Spark has proven its value for large-scale data processing workloads in production environments. | ||
|
|
||
| - **Platform Integration:** Successfully integrated with our [Databricks](/platforms-and-operations/databricks) platform, providing a robust foundation for distributed computing. | ||
|
|
||
| - **Performance Delivered:** Demonstrated significant performance improvements over traditional data processing approaches, particularly for complex analytical workloads. | ||
|
|
||
| - **Developer Adoption:** Teams have successfully adopted Spark's APIs across multiple programming languages, enhancing productivity and flexibility. | ||
|
|
||
| - **Current Focus:** Now adopted as our standard framework for big data processing, streaming analytics, and distributed machine learning workloads. | ||
|
|
||
| Apache Spark's proven performance and successful integration make it our adopted solution for enterprise-scale distributed data processing and analytics. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this link correct? Because it redirects to https://docs.databricks.com/aws/en/dlt which is about
Lakeflow Declarative Pipelines. Or did they renameDelta Live TablestoLakeflow Declarative Pipelinesmaybe? In that case I guess we should update this blip accordingly?Makes good sense to me to use this in any case. :-)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this blip should be in a different new 2025-.. folder and not 2024-12-01? Ah it is I see now. Why also in this past radar edition? Doesn't make much sense to me to rewrite our past radars?