Export/Import Data

Create and manage export/import jobs with advanced configuration options. Migrate data between MongoDB, SQL databases, and file formats (JSON, CSV, BSON) with powerful field mapping and JavaScript transformation scripts.

Supported Export Formats

VisuaLeaf writes to every mainstream data-interchange format used in modern data pipelines. All writers are streaming — memory usage stays flat regardless of collection size, so exporting 1M+ documents does not blow up the JVM heap.

Format	Extension	Streaming	Typical Use
JSON (array)	.json	Yes	Single array of documents, human-readable with pretty-print
NDJSON (line-delimited)	.ndjson / .jsonl	Yes	One document per line — ideal for streaming pipelines, Spark, BigQuery, ClickHouse
CSV	.csv	Yes	Excel, Google Sheets, tabular analysis; configurable delimiter and quoting
BSON	.bson	Yes	Native MongoDB dump format (mongorestore compatible)
SQL	.sql	Yes	Generated INSERT INTO ... statements for MySQL, PostgreSQL, SQL Server, Oracle
Mongo → Mongo	N/A	Yes	Direct collection sync between clusters (same or different connection) — no file on disk

Job Configuration

Each export/import job consists of three main configuration areas: Source, Target, and Data Transformation.

Source Configuration

Define where your data comes from. VisualLeaf supports multiple source types for maximum flexibility.

Source Type	Description	Configuration Options
MongoDB Collection	Export all documents from a specific collection	Connection, database, collection
MongoDB Query	Export filtered data using MongoDB query syntax	Query filter, projection, sort, limit, skip
MongoDB Aggregation	Export data processed through aggregation pipeline	Pipeline stages, $match, $group, $lookup, etc.
MongoDB Database	Export entire database with all collections	Connection, database, folder path for output
MongoDB Script	Execute custom MongoDB shell script as source	Connection, database, JavaScript code
MongoDump	Import from MongoDB dump files (BSON format)	Folder path containing BSON files
SQL Table	Export data from a SQL database table	SQL connection, database, table name
SQL Query	Export data using custom SQL SELECT statement	SQL connection, custom SELECT query
JSON	Import from JSON file (array or line-delimited)	File path, format detection (auto)
CSV	Import from CSV file with configurable delimiters	File path, delimiter, has header row
Notification	Send notification only (no data export)	Notification settings

Target Configuration

Choose where to export your data. Each target type has specific configuration options.

Target Type	Description	Configuration Options
JSON	Export to a single JSON array file	File path, pretty print, UTF-8 encoding
NDJSON	Line-delimited JSON (one document per line)	File path, encoding; ideal for streaming pipelines and log ingestion
CSV	Export to CSV file with custom delimiters	File path, delimiter, header row, quote char, encoding
BSON	Export to MongoDB BSON format (mongorestore-compatible)	File/folder path, compression options
MongoDB Collection	Mongo → Mongo direct sync into another collection (same or different cluster)	Connection, database, collection, write mode (insert / upsert / drop-insert)
SQL Script	Generate INSERT INTO ... statements for MySQL, PostgreSQL, SQL Server, Oracle	File path, SQL dialect, batch size, transaction mode
SQL Database	Export directly to SQL database table	SQL connection, table, write mode, create table

Data Transformation & Field Mapping

Transform your data during export/import with powerful field mapping and JavaScript transformation scripts. Perfect for data cleansing, format conversion, and schema adaptation.

Field Mapping

Map source fields to target fields with automatic type detection and manual override options. The field mapper shows a live preview of your data transformation.

• Auto-detect Fields - Automatically discovers fields from source data (first 100 documents)
• Drag & Drop Mapping - Visually map source fields to target fields
• Field Renaming - Change field names during migration
• Type Conversion - Convert between data types (string to number, date parsing, etc.)
• Nested Field Support - Handle nested objects and arrays with dot notation
• Exclude Fields - Unmap fields you don't want to export
• Add Computed Fields - Create new fields using transformation scripts

Transformation Scripts (Javet V8)

VisuaLeaf embeds the Javet V8 engine — the same JavaScript runtime that powers Chrome and Node.js — inside the export pipeline. You can write inline JavaScript that runs per row, per field, or per batch. Scripts execute in a sandboxed context with configurable timeouts (default 15s) and are recompiled once per job for maximum throughput.

Row-Level Script Context: Your row transformation scripts have access to:

• row - The full source document as a mutable JavaScript object; return it (possibly modified) or return a new object
• value - The current field value (when running a per-field script)
• doc - Alias for the entire source document
• target - The target document being built
• index - Zero-based document index within the batch
• crypto - Built-in helpers (crypto.sha256(str), crypto.md5(str), crypto.uuid())
• Date, Math, JSON, console - Standard V8 globals

Example 1 — Compute a Full Name and Age Bucket

Add a computed fullName field and derive an ageBucket from a birthDate ISO string:

// Runs once per row. Return the transformed row.
row.fullName = (row.first || '') + ' ' + (row.last || '');
row.fullName = row.fullName.trim();

if (row.birthDate) {
  const years = (Date.now() - new Date(row.birthDate).getTime())
                / (365.25 * 24 * 60 * 60 * 1000);
  if (years < 18)      row.ageBucket = 'minor';
  else if (years < 30)  row.ageBucket = 'young-adult';
  else if (years < 55)  row.ageBucket = 'adult';
  else                    row.ageBucket = 'senior';
}

// Drop internal fields we don't want in the export.
delete row.__v;
delete row.internalNotes;

return row;

Example 2 — Flatten Nested Objects and Redact PII

Flatten row.address into top-level columns (useful for CSV / SQL targets) and mask the email using SHA-256:

// Flatten nested address for tabular export.
if (row.address) {
  row.addressStreet  = row.address.street;
  row.addressCity    = row.address.city;
  row.addressCountry = row.address.country;
  delete row.address;
}

// Hash email so downstream analytics can join without exposing PII.
if (row.email) {
  row.emailHash = crypto.sha256(row.email.toLowerCase().trim());
  delete row.email;
}

// Normalize phone to E.164-ish format.
if (row.phone) {
  row.phone = String(row.phone).replace(/[^\d+]/g, '');
}

return row;

Common Transformation Patterns

Use Case	Description
String Formatting	Convert to uppercase/lowercase, trim whitespace, format phone numbers
Date Parsing	Parse date strings to Date objects or convert between formats
Field Combining	Concatenate multiple source fields (e.g., firstName + lastName)
Conditional Logic	Apply different transformations based on field values
Array Operations	Extract IDs from arrays, filter items, or join as strings
Row Filtering	Return null from a row script to skip that document
Enrichment	Look up static tables in a top-level const declared once and reused per row

Preview & Testing

Before running your export/import job, preview the transformation results on sample data. The preview shows both the original and transformed documents side-by-side.

• Build Preview - Test your transformations on up to 100 sample documents
• Script Validation - Automatic syntax checking for transformation scripts
• Error Detection - Highlights documents where transformation failed
• Performance Testing - Configurable timeout (default 15s) to test complex transformations

Execution Monitor

Track the progress of running export/import jobs in real-time with detailed status updates and error reporting via Server-Sent Events (SSE).

Progress Tracking

• Real-time Progress Bar - Visual indicator showing completion percentage
• Status Updates - Current operation (reading, transforming, writing)
• Document Counter - Shows processed/total documents
• Error Messages - Immediate notification of failures with details
• SSE Connection - Live updates from server via Server-Sent Events
• Auto-refresh Database - Database tree refreshes automatically on completion

Job Status States

Status	Description
Pending	Job is queued and waiting to start
Running	Job is currently executing
Completed	Job finished successfully
Failed	Job encountered an error and stopped
Cancelled	Job was manually stopped by user

Streaming Large Exports

Every writer in VisuaLeaf is a true streaming writer: documents are read from the source cursor, pushed through the transformation pipeline, and flushed to the destination in constant memory. This means you can export a 100 GB collection with a JVM heap under 512 MB.

• Cursor-based reads - Uses MongoDB batched cursors (default batchSize 1000) instead of loading result sets into memory
• Chunked writes - Data is flushed to disk / target every N documents (configurable; default 500)
• Back-pressure - Slow targets automatically throttle the reader — no unbounded queues
• Resumable checkpoints - Long-running jobs periodically checkpoint progress so a restart can resume near the failure point
• No memory spikes - Tested with export runs of 10M+ documents on commodity hardware

Scheduled Exports (Cron)

Any saved export job can be scheduled to run automatically using a standard cron expression. Schedules are persisted server-side and survive restarts. Missed runs (server was down) are logged but not backfilled.

Cron syntax: VisuaLeaf accepts standard 5-field expressions (minute hour day-of-month month day-of-week).

# Every day at 02:00 (nightly backup)
0 2 * * *

# Every 15 minutes
*/15 * * * *

# Every Monday at 09:00 (weekly report)
0 9 * * 1

# First day of each month at midnight
0 0 1 * *

# Every hour during business hours, weekdays only
0 9-17 * * 1-5

Scheduled export runs appear in the Task Manager alongside manual runs. Each run gets its own log, progress bar and completion notification.

Data Masking Integration

Exports can apply per-field masking rules to protect PII before data ever leaves the source cluster. Masking runs inline with the streaming writer, so it does not require a separate pass. See the dedicated Data Masking page for full configuration.

Mask Type	Behavior	Example
partial	Reveal first / last N characters, mask the middle	john@example.com → j***@***e.com
hash	Deterministic SHA-256 (safe for joins on masked value)	john@example.com → a8f5f167...
fake	Replace with realistic fake data of the same shape	John Smith → Marcus Alvarez
null	Set the field to null	555-1234 → null
truncate	Cut to a fixed length	very long text... → very long
redact	Replace entirely with a fixed placeholder	secret → [REDACTED]
randomize	Random value of the same type/length	555-1234 → 382-9471

Actions

Execute

Run the export/import job immediately. The job automatically saves before execution, then starts processing. Monitor progress in real-time with SSE updates. Success toast notification shown on completion.

Save

Save the job configuration. After saving, you'll be prompted: "Do you want to go to the Job Manager?" Choose "Yes" to navigate to Task Manager, or "No" to close. Use Ctrl+S (Cmd+S on Mac) as a shortcut.

Build Preview

Test your transformation scripts and field mappings on sample data (up to 100 documents) before running the full job. Shows side-by-side comparison of source and transformed documents with validation errors highlighted.

Auto-Generate Name

Automatically creates a descriptive job name based on source and target configuration (e.g., "users_collection_to_csv", "mongodb_to_mysql_migration"). Click the magic wand icon in the job title.

Close/Cancel

Close the job configuration without saving changes. Press Escape key as a shortcut (unless editing the job name). For modal mode, closes the modal. For embedded mode, returns to previous view.

Keyboard Shortcuts

Shortcut	Action
Ctrl+S (or Cmd+S on Mac)	Save job configuration
Esc	Close job configuration (cancel)

Pro Tips

1. Always test your transformation scripts with "Build Preview" before running on large datasets. This catches syntax errors and logic issues early.
2. For MongoDB Query sources, use projection to reduce data transfer by exporting only the fields you need instead of entire documents (e.g., {name: 1, email: 1, _id: 0}).
3. When exporting to SQL databases, field names are automatically converted to valid SQL identifiers. Special characters and spaces in MongoDB field names become underscores.
4. For database exports (MongoDB Database source), specify a folder path where each collection will be saved as a separate file with the collection name.
5. The job continues running in the background via SSE even if you close the dialog. After completion, the database tree automatically refreshes to show new collections.
6. When importing JSON files, VisualLeaf auto-detects the format (array [{}, {}] or line-delimited {} {}). The file must be validated before preview works.

Related Features

• Task Manager - Organize jobs into folders, set up dependencies, schedule execution, and monitor running tasks
• Collection Activity - Export data directly from the collection view with current query filters applied
• Aggregation Pipeline - Create complex data transformations before export using MongoDB aggregation stages
• SQL Query Activity - Write custom SQL queries as source for migration to MongoDB with preview
• Execution Monitor - View all running, scheduled, and completed jobs in the Task Manager's Monitor tab