MongoDB Atlas
The MongoDB Atlas destination writes data to MongoDB Atlas and MongoDB Enterprise Server. For information about supported versions, see Supported Systems and Versions.
The MongoDB Atlas destination can write CDC data if an operation is specified in the CRUD operation header attribute. When not specified, it treats all records as inserts. The destination can also perform upserts for update and replace records. For information about Data Collector change data processing and a list of CDC-enabled origins, see Processing Changed Data.
When you configure the destination, you define connection information, such as the connection string and credentials to use. You can specify SSL/TLS properties for an SSL/TLS-enabled MongoDB Atlas cluster. You can also use a connection to configure the origin.
You configure the database, collection, and write concern to use. To replace and update records, you must specify a unique key field and can optionally enable an upsert flag. When you do not specify a unique key field, update and replace records are sent to the stage for error handling.
You can optionally configure advanced options that determine how the destination connects to MongoDB Atlas.
Credentials
Based on the authentication used by MongoDB, configure the MongoDB Atlas destination to use no authentication, username/password authentication, or LDAP authentication. By default, no authentication is used.
- Authentication method
- Specify the authentication to use with the Authentication Method property on
the Credentials tab:
- None
- Username / Password
- LDAP
- Connection string
- If you prefer, you can specify credentials in the connection string on the Connection tab. However, specifying credentials on the Credentials tab is the recommended method.
Specifying Field Paths
- Data Collector format - Uses a slash ( / ) as a delimiter. Includes a leading slash.
- MongoDB format - Uses a period ( . ) as a delimiter.
Data Collector Format | MondoDB Format |
---|---|
/_id | _id |
/orders/address/line1 | orders.address.line1 |
/orders/lines[1]/quantity | orders.lines[1].quantity |
Unordered Writes and Stopping the Pipeline
- ordered writes
- The MongoDB Atlas destination performs ordered writes by default. Use this mode to perform ordered writes for pipelines that handle a range of CRUD operations. When all operations are Inserts, the destination performs ordered writes by default.
- unordered writes
- When ordered writes are not required, you can improve pipeline performance by configuring the MongoDB Atlas destination to perform unordered writes. Use this mode when all records to be processed are Inserts or if the order of the writes is not important. Otherwise data consistency is not guaranteed.
Use the Ordered Writes property on the MongoDB tab to specify the type of writes that the destination performs.
Define the CRUD Operation
To write to MongoDB Atlas, ensure that the CRUD operation record header attribute is defined for each record earlier in the pipeline. Records without an operation record header attribute are sent to error.
To update and replace records, you must specify a unique key field. You can also enable upserts for update and replace records.
Note that when performing a DELETE operation, the destination deletes a maximum of one matching document in MongoDB Atlas. It does not delete all matching documents, as is sometimes possible with MongoDB Atlas.
- sdc.operation.type
- When defined, the MongoDB Atlas destination uses the CRUD operation in the
sdc.operation.type
record header attribute when writing to MongoDB. When not specified, it treats all records as Inserts.
Performing Upserts
The MongoDB Atlas destination performs upserts when records are flagged for upsert – when
records have the sdc.operation.type
record header attribute set to 4
for upsert.
The destination also provides an Upsert property that enables upserts for records that are flagged for update or replace. When you enable the Upsert property, the destination inserts records when it does not find existing records to update or replace.
When not enabled, if the destination does not find an existing record for a record flagged for update or replace, it does not write the record to MongoDB Atlas. The Upsert property is not enabled by default.
For more information about MongoDB Atlas operations and the upsert flag, see the MongoDB Atlas documentation.
Enabling SSL/TLS
- Atlas/System CA - Connects to a MongoDB Atlas cluster. You can also use this when your certificates or keys have already been specified at the JVM level.
- Server Validation (1 Way TLS) - Connects to an SSL/TLS-enabled MongoDB Enterprise Server cluster when the client needs to validate the server certificate and does not need to prove client identity.
- Server and Client Validation (2 Way TLS) - Connects to an SSL/TLS-enabled MongoDB Enterprise Server cluster when the client needs to validate the server certificate and the server also validates the client key. This occurs when the cluster is set up to require client certificates.
- JKS (Java Keystore)
- PEM (text-based)
- DER (text-based)
- PKCS #7 / P7B
- PKCS #12 / P12 / PFX
- Private keys inside PEM, DER, or PKCS #12 encoded as PKCS#1 or PKCS#8
If the files are in PEM or DER plain text format, you can
provide the text in the stage properties. The certificate should begin and end with text
such as: —BEGIN CERTIFICATE—
or —END PRIVATE KEY—
.
Otherwise, you provide a path to the certificate file.
MongoDB Data Types
When the MongoDB Atlas destination writes to MongoDB, it converts Data Collector data types to the following standard MongoDB data types, by default. When necessary, the destination can convert Data Collector types to MongoDB BSON types. For more information, see Writing BSON Types.
The following table describes how Data Collector data types are converted to standard MongoDB data types:
Data Collector Type | Standard MongoDB Type |
---|---|
Boolean | Boolean |
Byte | Binary |
ByteArray | Binary |
Char | String |
Date | Date |
Datetime | Date |
Decimal | Decimal128 |
Double | Double |
Float | Double |
Integer | Int32 |
List | Array |
List-Map | Document |
Long | Int64 |
Map | Document |
Short | Int32 |
String | String |
Time | Date |
ZonedDateTime | Date Converted to UTC since MongoDB does not store time zones. |
Writing BSON Types
When writing to MongoDB, the MongoDB Atlas destination converts record fields to standard MongoDB data types as described in MongoDB Data Types, by default.
When a default conversion to a standard MongoDB data type is not appropriate, you can enable the destination to convert fields to MongoDB BSON types by placing BSON type information in field attributes.
To enable conversions to BSON types, use an Expression Evaluator processor earlier in the
pipeline to set the bsonType
field attribute to the BSON type you want
to use. You can use a single processor to set the required field attributes for multiple
fields. Some conversions require defining additional field attributes.
For example, to write a String field that contains an ObjectID hexadecimal string to a
MongoDB DBRef column, you use an Expression Evaluator processor to set the
bsonType
field attribute to Db_Ref
. And since the
String to DB_REF conversion requires additional attributes, you also set the
database
and collection
field attributes to the
appropriate values.
The following table lists supported MongoDB BSON data types, the Data Collector types you can convert from, and related attribute details.
CODE
, Code
, and code
are
all valid bsonType
attribute values.BSON Data Type | Compatible Data Collector Type | bsonType Attribute Value |
---|---|---|
Binary | Byte, Byte Array, Char, String Strings are converted using UTF-8 |
Binary |
BsonDbPointer | List-Map, Map Should contain the following fields:
|
Bson_Db_Pointer |
BsonDbPointer | String | Bson_Db_Pointer Also specify the following field attributes:
|
BsonRegularExpression | String | Bson_Regular_Expression |
BSONTimestamp | Date, Datetime, String, Time, Zoned Datetime Strings should be
in |
Bson_Timestamp Optionally specify an |
Code | String | Code |
CodeWithScope (Scope cannot be set) |
String | Code_With_Scope |
DBRef | Map Should contain the following fields:
|
Db_Ref |
DBRef | String Use a 24-digit hexadecimal or |
Db_Ref Also specify the following field attributes:
|
Decimal128 | Decimal, Double, Float, Integer, Long, Short, String | Decimal128 |
ObjectId | String Formatted as a 24-digit hexadecimal representation of an ObjectId |
Object_Id |
Symbol | Char, String | Symbol |
Configuring a MongoDB Atlas Destination
Configure a MongoDB Atlas destination to write to MongoDB Atlas or MongoDB Enterprise Server.