Skip to content

Fix sync deleting every remote copy of a file with a similar name - #129

Open
TalLevAmi wants to merge 1 commit into
masterfrom
fix/sync-duplicate-name-match
Open

TalLevAmi wants to merge 1 commit into
masterfrom
fix/sync-duplicate-name-match

Conversation

@TalLevAmi

Copy link
Copy Markdown
Contributor

cld sync --push can delete every remote copy of a file and upload nothing, so the file is missing from Cloudinary until the next run.

SyncDir._local_candidates (cloudinary_cli/modules/sync.py) builds the pattern f"({candidate_path}|{filename} \(\d+\){extension})" and uses it with re.match. re.match checks only the start of the name, so the remote name notes matches the local file notes.txt. The name is also not escaped, so +, [ or ( in a file name act as regex syntax.

How the data is lost, in dynamic folder mode:

  1. .cld-sync holds notes.txt -> notes (a raw file loses its extension in the display name, see Fix sync for raw files whose display name has no extension #124).
  2. Cloudinary holds two copies named notes.
  3. The first copy matches notes.txt and gets the key notes.txt. The second copy gets notes (1).
  4. The local name maps to notes, so sync treats both remote copies as unique remote files and deletes them. It finds notes.txt in the remote list, so it reports "Skipping 1" and uploads nothing.

Brief Summary of Changes

  • _local_candidates: escape the name parts with re.escape, and use fullmatch so only the whole name matches.
  • test/test_modules/test_cli_sync.py: a new offline class TestCLISyncDuplicateNamesOffline. It needs no cloud.
    • test_local_candidates_exact_match tests the matching function: notes against notes.txt, notesX and notes (1).txt, and escaping (a+b.jpg against aab.jpg).
    • test_sync_push_does_not_delete_all_duplicates_of_synced_file runs sync --push with mocked API calls and the .cld-sync mapping. It fails when both remote copies are deleted and nothing is uploaded.

What does this PR address?

  • GitHub issue (Add reference - #XX)
  • Refactoring
  • New feature
  • Bug fix
  • Adds more tests

Are tests included?

  • Yes
  • No

Reviewer, please note:

  • Live check on a test cloud, in dynamic folder mode:
    • Current master: push notes.txt, upload one more copy named notes, then run sync --push -F. Result: "Skipping 1", "Deleted 2 resources", no upload, and 0 assets left on Cloudinary. The dry run said "Would delete 2 resources" next to "Skipping 1", so it did not show that the file would be missing.
    • This branch, with 3 copies of notes: the dry run says "Would delete 3" and "would upload notes.txt". The real run deletes 3 and uploads 1, so one copy stays.
  • The dry run and the real run use the same matching code, so both are fixed.
  • Link to Fix sync for raw files whose display name has no extension #124: Fix sync for raw files whose display name has no extension #124 keeps the extension of raw files, which removes the most common trigger (notes.txt -> notes). This PR fixes the matching itself, which is wrong with or without Fix sync for raw files whose display name has no extension #124. The two PRs do not touch the same lines.
  • Not fixed here (separate issues):
    • Unmatched duplicates get the names notes (1) and notes (2), and never notes, so they are deleted and uploaded again on each push. This causes extra traffic, but no data loss.
    • Push deletes before it uploads. If an upload fails, the file is missing until the next run.
    • The Search index delay can still cause duplicates on an immediate second push.

Checklist:

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I ran the full test suite before pushing the changes and all the tests pass.

Tests: test/test_modules/test_cli_sync.py (including the live sync tests), test/test_file_utils.py and test/test_utils.py: 54 passed. The two new tests fail on master (the sync-level test reports deleted ['pid_a2', 'pid_a1'], uploaded []) and pass with the fix. CI runs the full suite.

🤖 Generated with Claude Code

`SyncDir._local_candidates` matched remote names to local files with
`re.match` on an unescaped pattern. `re.match` checks only the start of
the name, so the remote name `notes` matched the local `notes.txt`.

In dynamic folder mode, with two remote copies named `notes` and a
`.cld-sync` entry `notes.txt -> notes`, `sync --push` then:
- counted `notes.txt` as in sync and skipped its upload;
- deleted both remote copies.
The file was missing from Cloudinary until the next run.

Escape the name parts and use `fullmatch`. Add an offline unit test for
the matching and a mocked sync-level test for the deletion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@TalLevAmi
TalLevAmi marked this pull request as ready for review September 27, 2026 07:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant