The Rise of Cloud Observability Platforms: How Observability Providers Can Leverage Enterprise Data Fabric to Accelerate Growth

By Yih-Khai Wong | 4Q 2024 | IN-7580

Registered users can unlock up to five pieces of premium content each month.

Log in or register to unlock this Insight.

The enterprise technology landscape has shifted significantly, due to the pressing need for digital transformation. Core to the digital transformation journey is the widespread adoption of multi-cloud and hybrid cloud architectures. The proliferation of cloud platforms together with the adoption of microservices and distributed systems introduced layers of complexity, necessitating the need for cloud observability tools to help monitor and optimize the consumption of cloud-based services.

Cloud observability platforms are solutions that collect, analyze, and visualize metric data, logs, and user experience data across a distributed/disparate cloud environment, often providing real-time insights into workload performance, network health, and application behaviors. There are two main categories of cloud observability tools:

• Embedded: The cloud observability tools are embedded inside a cloud hyperscaler's platform. A few examples would be Amazon CloudWatch, Azure Monitor, and CloudMonitor from Alibaba Cloud.
• Third-Party: Cloud observability solutions from Independent Software Vendors (ISVs) that ingest telemetry data from a variety of data sources (including cloud platforms). A few examples would be Datadog's observability platform, Dynatrace's infrastructure observability solution, and New Relic's Application Performance Monitoring (APM) tool.

While hybrid cloud deployments will continue to be the foundation of growth for observability solution providers, the time is now ripe for solution providers to leverage enterprise data fabric deployments as an additional go-to-market channel as data management initiatives accelerate and become a focal point for many enterprises.

Cloud observability solution providers need to approach the enterprise data fabric market with an integration-first approach. Enterprises are integrating data fabric capabilities into their technology, both in the cloud and on-premises. Currently, most of the data that are being processed on an enterprise data fabric platform are structured data, while enterprises often have little to no visibility on unstructured data. Cloud observability solution providers need to integrate their solutions with a data fabric platform and provide the ability to track and trace unstructured data such as imaging, sensor, weather, and speech data to be ingested as part of the data integration process, accelerating the growth of enterprise data fabric and solving critical business needs of enterprises.

Cloud observability solution providers can also introduce a new concept to the enterprise data fabric platform-telemetry. Often enterprise data fabric solutions are used to process and visualize data as information output for making business decisions. Observability platforms integrated with data fabric solutions will show telemetry data (logs, events, errors, etc.) to help data engineers have a better understanding of how data flows between different architectures and environments of a data fabric solution. Data engineers will be able to fix, tweak, and optimize data fabric solutions for better performance and business data output.

While the emergence of observability platforms brings added data visibility and transparency for data engineers/analysts, it will also add a layer of complexity to an already complex and resource-heavy environment. Deploying and integrating an observability platform will require significant investments, both monetary and human skills, making data fabric initiatives a big risk for enterprises that do not have the expertise and financial capabilities to properly deploy both solutions. However, if done correctly, observability tools can improve collaboration between data engineers and data analysts by providing better data visibility and transparency. This will enable enterprises to better design and manage their data infrastructure in an enterprise data fabric framework, reducing the risk of data management failures.

To achieve the desired impact described in the previous section, cloud observability solution providers will need to bring complementary features from their tool to the enterprise data fabric platform. These features include:

• Leveraging Unstructured Data Capabilities: The sweet spot for the integration of a cloud observability platform with enterprise data fabric solutions will involve use cases where the visibility and processing of unstructured data from data fabric solutions are limited. An example of this would be capturing data from sensor devices for real-time analytics. The ability of cloud observability tools to trace and process unstructured data from sensor devices can complement the data fabric tools by providing advanced data tracing and lineage tracking capabilities that are inherent in all observability tools. This will ensure data fabric solutions comply with local data governance requirements and industry standards.
• Develop Edge Computing Capabilities: As more data are created on the edge, observability tools will need to provide visibility and tracking for edge devices as well. The current focus of observability tools revolves around data hosted in data warehouses, on-premises databases, and enterprise applications, but as technology decouples, support for edge observability, especially combined with telemetry capabilities, will enable enterprises to have a better sense of their edge environment and end-to-end visibility of how data flow in the organization.
• Focus on Development, Security, and Operations (DevSecOps): Observability solution providers can work with data fabric providers to introduce DevSecOps as part of the data fabric architecture. DevSecOps is an integral part of any observability platform and is used to identify security threats and data security risks. Packaging DevSecOps tools as part of the data integration process ensure enterprises need not worry about data integrity issues.

While observability tools have slowly become part of an enterprise's technology ecosystem, these tools are often used to observe and ensure business continuity. By leveraging enterprise data fabric as the platform of choice and integrating observability tools on the data fabric platform, solution providers can ensure that observability tools will become a critical part of any enterprise's technology infrastructure.