Cleaning Up Your 1Password Vault - An Automated Duplicate Remover

Ever found yourself drowning in duplicate login entries in your 1Password vault? You know the feeling - you create an account, forget you already have it, and end up with three identical entries for the same website. I've been there, and after manually cleaning up duplicates one too many times, I decided to build an automated solution. Let me share how this script transformed my password management workflow.

The Problem with Duplicate Entries

Managing passwords across multiple devices and browsers often leads to duplicate entries. Maybe you imported passwords from different sources, or created accounts multiple times thinking they didn't exist. Before you know it, your vault is cluttered with duplicates that make it harder to find the right credentials when you need them.

Manually identifying and removing duplicates is tedious, time-consuming, and error-prone. You might accidentally delete the wrong entry, or miss duplicates that have slightly different titles or URLs. That's where automation comes in.

Building the Solution

While the original idea for automating duplicate removal from 1Password vaults wasn't mine, I took the concept and significantly enhanced it with additional features. The script leverages the 1Password CLI (`op`) to automatically detect and remove duplicate items, using intelligent matching strategies to identify duplicates based on website URLs and usernames, not just titles.

Key Features

Smart Duplicate Detection: The script groups items by website and username, then performs detailed field-by-field comparisons to identify true duplicates. It's smart enough to recognize that entries with matching credentials are duplicates, even if their URLs differ slightly.

Whitelisting: Need to keep certain items safe? The script supports whitelisting by URL patterns or domains. For example, you can whitelist `localhost:3000` to ensure all your local development credentials are never touched.

Title Cleaning: Many password managers append usernames to titles, creating entries like "GitHub (john.doe@example.com)". The script includes a separate process that automatically cleans these titles, removing the redundant username information.

Persistent Caching: The script maintains a cache of processed items, so if you interrupt the process or run it multiple times, it won't reprocess items it's already handled. The cache is saved live, meaning your progress is preserved even if the script is terminated unexpectedly.

Dry-Run Mode: Safety first! The script includes a dry-run mode that shows you exactly what would be changed without actually modifying your vault. Perfect for testing and verification before making real changes.

How It Works

The script processes your vault in two main phases:

• Duplicate Removal: Groups items by website and username, compares them field-by-field, and removes duplicates while keeping the first occurrence. Whitelisted items are always preserved.

• Title Cleaning: Scans all items and removes username patterns from titles (e.g., "Amazon (user123)" becomes "Amazon").

Both processes respect the cache, so you can run the script multiple times safely. Items are added to the cache immediately after processing, ensuring no work is lost if something goes wrong.

The Impact

What used to take hours of manual work now happens automatically in minutes. The script has helped me clean up hundreds of duplicate entries, making my vault more organized and easier to navigate. The whitelisting feature gives me confidence that important entries won't be accidentally removed, and the persistent cache means I can run the script incrementally without worrying about reprocessing items.

If you're struggling with a cluttered password vault, automation might be the solution you need. Sometimes the best productivity tool is one you build yourself.

Try It Yourself

You can find the complete script on GitHub Gist:

Feel free to fork it, enhance it, and adapt it to your needs!