Data Engineering

Amazon OpenSearch intelligent ingestion is a fully managed, serverless capability that turns documents and structured data in Amazon S3 into RAG-ready, searchable indexes with no pipelines to manage. Learn how it works and when to use Ingestion Jobs or Ingestion Streams.

AWS Big Data Blog · 2026-10-23T21:50:38Z

Nearly a decade ago, I joined MongoDB as a Senior Product Manager to help build the company’s new cloud product, MongoDB Atlas. Our customers had been telling us they wanted to bring MongoDB’s familiar developer experience to the cloud, with the reliability and confidence teams needed to run in production. Atlas was our answer. Today, we’re celebrating 10 years of MongoDB Atlas, the generational data platform for AI…

MongoDB Blog · 2026-06-25T17:28:40Z

Trusted Identity Propagation with AWS IAM Identity Center now works across Regions. Learn how to share AWS Lake Formation-governed Amazon S3 and Amazon Redshift data in one Region and query it from another as the actual corporate user, with fine-grained access control and per-user auditability in AWS CloudTrail.

AWS Big Data Blog · 2026-10-23T19:50:12Z

The enterprise adoption of artificial intelligence has reached an inflection point. Organizations are rapidly moving into the era of agentic AI, autonomous systems capable of executing complex reasoning and making operational decisions independently. Yet as executives attempt to transition agents from sandbox environments into mission-critical production channels, they inevitably collide with an AI trust gap. Unlike…

MongoDB Blog · 2026-06-23T15:27:00Z

The Amazon OpenSearch Service Search Relevance Agent automates search relevance tuning. You describe a problem in plain language, and the agent investigates your behavioral data, proposes an improved query, and validates it with an offline experiment. This post walks through the agent's architecture and a complete example investigation.

AWS Big Data Blog · 2026-10-22T17:43:30Z

There’s a common theme to the conversations I’ve been having with AI teams lately: change. Constant, head-spinning change. Teams across industries are evaluating and re-evaluating model providers, agent frameworks, and harnesses on a continuous basis. At MongoDB, we believe that your choice of technology partner—specifically, your data platform—should simplify how you build with AI. It should deliver performance at…

MongoDB Blog · 2026-06-11T19:46:11Z

Learn how to trace two invisible threats in a multi-agent AI pipeline, prompt injection propagation and cross-tenant data exfiltration, using Amazon OpenSearch Service and Amazon Bedrock. Instrument every agent hop with embeddings, provenance, and PII detection, reconstruct the attack path, and add guardrails that stop these attacks before they reach production.

AWS Big Data Blog · 2026-10-19T16:10:31Z

Retail supply chains are not a back-office logistics function; they are a high-stakes, board-level concern. Imagine learning suddenly that shipment rerouting surcharges have doubled due to new regional escalations; the impact on competitive differentiation and consumer trust is immediate. As a result, a long-standing focus on linear efficiency and lean inventory is being disrupted by a mandate for resilience and…

MongoDB Blog · 2026-06-03T19:51:09Z

Learn how to set up Amazon Redshift Data Sharing between a producer and consumer cluster and use dbt Cloud to transform the shared data into analytics-ready models, with live, zero-copy access and no ETL between clusters.

AWS Big Data Blog · 2026-10-15T18:01:07Z

As enterprise AI agent adoption scales, the absence of centralized, organization-level tool infrastructure is producing compounding costs. When adoption is built around optimizing for deployment speed, enterprises expose themselves to a combination of risks: duplicated engineering effort, security exposure, and operational opacity. Every enterprise needs its own shared tool registry, one that reflects its specific…

MongoDB Blog · 2026-05-11T14:35:00Z

Amazon Redshift now automatically resolves many concurrent writes to Apache Iceberg tables that don't actually overlap, committing changes that previously failed and had to retry. Learn why these writes conflict, how the engine rebases non-overlapping ones across operation, partition, and file checks, and the throughput gains it delivers.

AWS Big Data Blog · 2026-10-13T20:16:48Z

Today, we announced at .local London that MongoDB 8.3 is built for the speed AI demands—and our customers can't afford to wait. The data layer has to move at AI speed The old contract between databases and the applications on top of them was simple: databases improve slowly, and architectures evolve around them. AI has changed that contract. The workloads our customers are shipping today—agents retrieving at…

MongoDB Blog · 2026-05-07T14:33:00Z

AWS Glue Data Quality is now available natively on Amazon EMR, bundled in AWS runtime for Apache Spark (emr-spark-8.1.0). Write the same DQDL rulesets you use in AWS Glue directly in your Amazon EMR Spark jobs. This post walks through two patterns: lightweight inline validation and an Apache Iceberg Write-Audit-Publish workflow with managed rulesets.

AWS Big Data Blog · 2026-10-12T23:40:26Z

This guest post comes from IDC’s Dr. William Lee, Senior Research Director, Service Provider and Core Infrastructure Research. MongoDB commissioned IDC to explore the connection between legacy infrastructure, data challenges, and AI across Asia Pacific, and today we’re happy to share that work. For more, see the full MongoDB-sponsored IDC InfoBrief, Modernizing Legacy: Winning in the Age of AI, Doc #AP242555-IB…

MongoDB Blog · 2026-04-14T17:01:07Z

Learn how to build an enterprise-scale cross-account streaming pipeline with Amazon Kinesis Data Streams using the Kinesis Producer Library (KPL) and Kinesis Client Library (KCL) across three separate AWS accounts, with resource-based policies, cross-account IAM, and a shared customer managed AWS KMS key for maximum security isolation.

AWS Big Data Blog · 2026-10-12T15:48:07Z

You can often predict a load spike before it arrives. Maybe it happens at the same time every day, or there’s always a spike at midnight on a Friday when you run a certain batch job. Or maybe it’s not cyclical, but load is rising steadily, and it’s a reasonable guess that it will keep rising for a while. MongoDB Atlas’s reactive auto-scaler handles these spikes, but scaling to the right size takes several minutes…

MongoDB Blog · 2026-04-07T17:03:00Z

Troubleshooting a multi-service, MWAA-orchestrated ETL pipeline often means hunting across scattered Amazon CloudWatch log groups. This post shows how to centralize your ETL logs in Amazon OpenSearch Service and query them in plain language through an MCP server on Amazon Bedrock AgentCore, so you find root causes faster.

AWS Big Data Blog · 2026-10-06T16:17:46Z

Learn how to pair DuckDB with AWS Glue 6.0 to run SQL-centric ETL on a single worker, reading Parquet from Amazon S3 and writing Apache Iceberg tables to Amazon S3 Tables. This post walks through a complete, runnable example and compares measured cost and runtime against an equivalent Apache Spark job on the same Glue runtime.

AWS Big Data Blog · 2026-10-06T16:13:02Z

Amazon EMR 8.1.0 adds multi-catalog support through the RedirectingSessionCatalog, so you can query Iceberg, Delta Lake, Hudi, and Hive tables through a single catalog and join data across AWS accounts without copying it. This post shows how to put these capabilities into practice on Amazon EMR Serverless.

AWS Big Data Blog · 2026-10-06T16:05:27Z

Amazon Redshift now supports Iceberg materialized views. Compute an aggregation once in Amazon Redshift and store the result as a standard Apache Iceberg table in Amazon S3, queryable by Amazon Athena, Apache Spark, Amazon SageMaker, and AWS Glue. This post covers use cases, incremental refresh, and a step-by-step getting-started guide.

AWS Big Data Blog · 2026-10-05T21:41:49Z

Amazon Redshift automates much of its own tuning, but periodic operational reviews still pay off. Learn how the review_cluster tool in the open-source Amazon Redshift MCP server runs a full cluster diagnostic from one natural-language request, evaluating 12 diagnostic areas and returning prioritized, documentation-linked recommendations in minutes.

AWS Big Data Blog · 2026-10-05T15:44:45Z

The KIP-848 consumer protocol in Apache Kafka 4.0 redesigns consumer group rebalancing to eliminate stop-the-world pauses. This post explains how the consumer protocol works on Amazon MSK, how to enable it on MSK Standard and Express brokers, and how to diagnose and resolve slow rebalancing issues.

AWS Big Data Blog · 2026-10-01T16:43:54Z

Debugging Apache Airflow DAG failures across services like AWS Glue, Amazon EMR, and Amazon Athena is slow and manual. In this post, we show you how to build a custom Airflow plugin that integrates with Amazon Bedrock to automatically analyze DAG task failures and deliver on-demand root cause analysis on Amazon MWAA.

AWS Big Data Blog · 2026-09-29T16:17:09Z

Announcing Spark Connect on Amazon EMR on EKS: build, test, and debug Spark applications from VS Code, PyCharm, Jupyter notebooks, Amazon SageMaker Unified Studio, or dbt, while running full-scale Spark operations on your existing Amazon EKS clusters.

AWS Big Data Blog · 2026-09-29T16:12:17Z

Amazon Aurora PostgreSQL zero-ETL integration with Amazon SageMaker replicates your operational data to a lakehouse in near real time, without building custom ETL pipelines. Learn the architecture and change data capture mechanics, then set up the integration and query your data in Amazon SageMaker.

AWS Big Data Blog · 2026-09-29T16:10:14Z

Amazon Redshift now supports evolving Apache Iceberg table schemas and partition layouts through ALTER statements, with no data rewrites or pipeline rebuilds. In this final post of the series, you rename, add, drop, and widen columns, evolve partitions, and create AWS Lake Formation resource links for governed cross-engine access to Amazon S3 Tables.

AWS Big Data Blog · 2026-09-28T15:48:35Z

Amazon SageMaker Unified Studio now supports custom permissions boundaries for the IAM roles its Tooling blueprint creates. Learn how to create a permissions boundary that restricts AI agent capabilities, configure it on the Tooling blueprint with the AWS CLI, and validate that every provisioned role carries the boundary.

AWS Big Data Blog · 2026-09-28T15:45:42Z

Learn how Delivery Hero rebuilt its ad measurement pipeline from hourly batch processing to real time on Amazon Managed Service for Apache Flink, cutting the gap between an event and its recording from 61 minutes to 1.2 seconds and reducing monthly operational costs by about 57%.

AWS Big Data Blog · 2026-09-24T16:09:12Z

Configuring VPC networking per project across a SageMaker Unified Studio domain creates inconsistent, hard-to-audit networks. This post shows administrators how to configure domain-level VPC networking once, so every new project automatically inherits consistent, private network isolation, then update existing projects and validate connectivity.

AWS Big Data Blog · 2026-09-23T18:36:56Z

Amazon EMR on EC2 now supports Spark Connect, so you can develop and debug PySpark interactively from Amazon SageMaker Unified Studio Data Notebooks or your own IDE while Spark runs on your cluster. This post shows you how to get started from both a Data Notebook and a local IDE.

AWS Big Data Blog · 2026-09-23T18:28:32Z

In Part 2 of this series, sign in as a data consumer, subscribe to enriched unstructured data assets in Amazon SageMaker Catalog, and query them using natural language through a no-code Amazon Bedrock chat agent and Amazon Bedrock model inference.

AWS Big Data Blog · 2026-09-23T18:22:00Z

As consumer groups on Amazon MSK scale to thousands of members, the metadata record Kafka writes during rebalances can exceed the 1 MB limit and stall the group. Learn how to estimate metadata size, raise the topic-level limit safely, plan capacity, and apply complementary strategies for scaling large consumer groups.

AWS Big Data Blog · 2026-09-22T15:16:06Z

Moeve standardized how it runs dbt across multiple data lakes by building a centralized, serverless launcher on Amazon Athena, AWS Step Functions, AWS Fargate, Amazon DynamoDB, and Amazon EventBridge, cutting new-project onboarding from days to about 15 minutes while keeping compute close to the data and orchestration loosely coupled.

AWS Big Data Blog · 2026-09-22T15:13:31Z

Connect Snowflake to Amazon SageMaker Unified Studio to build a unified data catalog. Query federated Snowflake tables without moving data, publish enriched assets to SageMaker Catalog, and validate data quality with AWS Glue Data Quality, all while keeping data in Snowflake.

AWS Big Data Blog · 2026-09-16T15:19:51Z

Connect Microsoft Power BI directly to governed data in Amazon SageMaker Unified Studio using new authentication modes in the Amazon Athena ODBC driver, with no third-party ODBC-JDBC bridge. Part 1 covers IAM Identity Center (IDC)-based domains with both DSN-based and DSN-less connection methods.

AWS Big Data Blog · 2026-09-15T20:55:26Z

Connect Microsoft Power BI directly to governed data in Amazon SageMaker Unified Studio using the Amazon Athena ODBC driver. Part 2 covers IAM-based domains with SageMakerIam authentication, including AWS IAM Identity Center administrator setup, for both DSN-based and DSN-less connection methods.

AWS Big Data Blog · 2026-09-15T20:55:16Z

Learn how United Airlines uses AWS Glue Data Catalog federation to query Databricks Unity Catalog data directly from Amazon Redshift Serverless without duplicating data, using resource links and AWS Lake Formation for governance.

AWS Big Data Blog · 2026-09-15T15:59:48Z

Amazon Kinesis Data Streams now supports scaling down ingest capacity for on-demand Advantage streams with warm throughput. Learn how the scale-down works, how to monitor stream behavior with Amazon CloudWatch, and best practices for releasing excess capacity after transient traffic bursts.

AWS Big Data Blog · 2026-09-14T15:36:36Z

Learn how to migrate an Amazon CloudSearch domain to Amazon OpenSearch Serverless: assess your configuration, create a collection with explicit index mappings, convert your documents and queries to the OpenSearch query DSL, configure security, load data with Amazon OpenSearch Ingestion, and validate before cutover.

AWS Big Data Blog · 2026-09-10T16:06:42Z

Accelerate slow, repetitive Apache Spark analytical queries on Apache Iceberg tables without rewriting any SQL. This post shows how automatic query rewrite in Amazon EMR and AWS Glue uses Iceberg materialized views in the AWS Glue Data Catalog to transparently substitute matching query plans, and how to design materialized views for the best speedup.

AWS Big Data Blog · 2026-09-10T16:05:45Z