And When a Simpler Approach Is Enough
In a really good and well redacted post, Jakub Illner describes a practical approach to syncing buckets using rclone copy in both directions:
https://jakubillner.github.io/2025/02/28/sync-buckets-with-rclone.html
His method is clean and effective for specific replication patterns. However, it deliberately avoids rclone bisync. This article explains:
- What
rclone bisyncactually does - When it is the correct tool
- When the dual
copymodel is preferable - How to configure
bisyncproperly - What happens under failure conditions
What Rclone Bisync Really Does
rclone bisync maintains bidirectional state convergence between two paths.
It does not simply copy new files. It:
• Detects changes on both sides
• Propagates new and modified files
• Detects deletions
• Tracks prior state in a working directory
• Handles conflicts
In short: it enforces eventual consistency between two endpoints.
This makes it fundamentally different from running:
rclone copy A B
rclone copy B A
The copy approach replicates new objects but does not guarantee full state alignment.
When You Should Use Bisync
Use bisync when:
• Both sides are writable
• Files can be modified on either side
• Deletions must propagate
• Conflicts must be detected
• You need deterministic state convergence
Typical scenarios:
• Active–active regional buckets
• Hybrid cloud with writes on both ends
• Distributed workloads producing overlapping filenames
• Migration phases where both environments remain live
If you need correctness, not just replication, bisync is the right tool.
When the Dual Copy Approach Is Better
The approach in Illner’s article uses two copy commands.
That is appropriate when:
• Each region writes unique filenames
• There are no filename conflicts
• Deletions are not global
• You want replication, not strict synchronization
• Operational simplicity is more important than state tracking
In that design, buckets act as append-only stores. No reconciliation is required.
bisync would be overkill there.
Example Configuration Using OCI Buckets
Assume:
OCI_fk:region1_bucket
OCI_fk:region2_bucket
First Alignment (Mandatory)
The first run must build a baseline state file.
rclone bisync OCI_fk:region1_bucket OCI_fk:region2_bucket \
--resync \
--resilient \
--create-empty-src-dirs \
--compare size \
--progress \
--verbose
Explanation:
• --resync builds fresh state tracking
• --resilient tolerates partial failures
• --compare size avoids modtime inconsistencies in object storage
• --create-empty-src-dirs keeps directory structure aligned
Important: never run bisync without --resync the first time.
What --resync Actually Does
--resync forces rclone bisync to discard its previous state and rebuild synchronization knowledge from scratch.
Normally, bisync relies on a local state file to detect:
• new files
• modifications
• deletions
• conflicts
With --resync, it:
- Fully lists both buckets
- Reconciles differences in both directions
- Recreates a fresh baseline state file
It is effectively a full bidirectional reconciliation.
When You Must Use It
Use --resync:
• On the very first run
• If the state directory was lost or corrupted
• After a crash without --resilient
• When you suspect the two sides drifted outside bisync
Do not put it in a cronjob. It is expensive and defeats incremental tracking.
Important Behavior
During --resync, missing files are treated as absent, not deleted historically.
Example:
If a file was deleted last week from bucket A but still exists in bucket B, a resync will copy it back to A.
That is often unexpected.
Operational Rule
Use --resync to rebuild trust in state.
Use normal bisync runs for daily operation.
It is a reset button, not a routine flag.
Recurring Execution (Cronjob)
After the first alignment:
rclone bisync OCI_fk:region1_bucket OCI_fk:region2_bucket \
--resilient \
--create-empty-src-dirs \
--compare size \
--progress \
--verbose
Example cron entry:
*/15 * * * * /usr/bin/rclone bisync OCI_fk:region1_bucket OCI_fk:region2_bucket --resilient --create-empty-src-dirs --compare size >> /var/log/rclone-bisync.log 2>&1
Run frequency depends on RPO requirements.
Failure Scenarios and What Happens
1. One Bucket Becomes Temporarily Unreachable
Example: region2 is down.
What happens:
• bisync detects failure
• With --resilient, it aborts safely
• No destructive deletions occur
• State file remains intact
When region2 returns:
• Next run reconciles accumulated changes
Without --resilient, you risk partial execution and inconsistent state tracking.
2. One Side Has New Files During Outage
Scenario:
• Region1 remains online
• Region2 offline
• New files created in region1
Result:
• Files remain only in region1
• On next successful run, they propagate to region2
No data loss.
3. Conflicting Changes
Scenario:
• Same file modified in both buckets
• Different sizes
With --compare size:
• Conflict detected
• bisync preserves both versions by renaming one side
This is where copy + copy fails silently. The last writer wins without detection.
4. Accidental Deletion on One Side
Scenario:
• File deleted in region1
• Still exists in region2
Next run:
• Deletion propagates to region2
This is correct behavior for true synchronization.
If you do not want deletions to propagate, bisync is the wrong tool.
5. Partial Network Failure Mid Run
Without --resilient:
• State tracking may be incomplete
• You may need --resync
With --resilient:
• Operation aborts safely
• Next run continues cleanly
For production, always use --resilient.
6. State File Corruption
If the local state cache is lost:
You must run:
rclone bisync OCI_fk:region1_bucket OCI_fk:region2_bucket --resync
This rebuilds the synchronization baseline.
Keep the working directory persistent and backed up.
Performance Considerations
Object storage environments introduce constraints:
• Large object listings
• No reliable modtime
• API throttling
• High object count
For millions of objects:
• Consider tuning --checkers and --transfers
• Use --compare size instead of modtime
• Monitor memory usage
bisync requires full listings on both sides. That has cost.
The dual copy model is lighter if your workload does not require full reconciliation.
Practical Comparison
| Requirement | Dual Copy | bisync |
|---|---|---|
| New file replication | Yes | Yes |
| Conflict detection | No | Yes |
| Deletion propagation | Manual | Automatic |
| Deterministic state convergence | No | Yes |
| Operational simplicity | High | Moderate |
| Safety during conflicts | Low | High |
Final Recommendation
Use the model described in Illner’s article if:
You have append only workloads and unique filenames.
Use rclone bisync if:
You need correctness, symmetry, and deletion awareness.
One replicates.
The other synchronizes.
Choose based on consistency requirements, not convenience.
References:
- Rclone website
- Configure rclone on OCI Object Storage
- Move data to object storage in the cloud by using Rclone
