Skip to content

Firestore Data Mirroring

Project Overview

This project is designed to automatically synchronize data between two Google Firestore databases. It's a useful when you need to maintain a backup or mirror of their Firestore data, or for situations where data needs to be replicated between different Firestore instances (e.g., between development and production environments).

The project uses a Python script to recursively scan the source Firestore database, retrieve all the documents and collections, and then write them to the destination Firestore database.

Getting Started

Prerequisites

To use this project, you'll need the following:

  1. Access to the Google Cloud Console with the ability to create and manage Cloud Functions, Firestore databases, and Secret Manager.
  2. The Google Cloud SDK installed on your local machine, along with the necessary credentials to access your Google Cloud project.

Dependencies version at time of project creation:

  • Python 3.11.0
  • Google Cloud SDK 502.0.0
  • beta 2024.11.15
  • bq 2.1.9
  • core 2024.11.15
  • gcloud-crc32c 1.0.0
  • gsutil 5.31

Installation and Setup

  1. Clone the repository: Start by cloning this repository to your local machine:

  2. Create a .env file: In the root directory of the project, create a new file called .env and add the following contents:

SOURCE_PROJECT_ID=your-source-project-id
DESTINATION_PROJECT_ID=your-destination-project-id

Replace your-source-project-id and your-destination-project-id with the actual project IDs for your source and destination Firestore databases.

  1. Install dependencies: In your terminal, navigate to the project directory and install the required Python libraries:
pip install requirements.txt
  1. Run the script: To start the data mirroring process, run the following command in your terminal:
python main.py

This will start the synchronization process, copying data from the source Firestore database to the destination Firestore database.

You can also set the script to run on a schedule using a task scheduler or cron job on your local machine.

Project Goal

The primary goal of this project is to provide a reliable and efficient way to maintain a backup or mirror of Firestore data. This can be useful in a variety of scenarios, such as:

  • Disaster Recovery: Having a separate, up-to-date copy of your Firestore data can be invaluable in the event of a data loss or system failure in your primary Firestore instance.
  • Development and Testing: You can use this project to keep your development and production Firestore databases in sync, making it easier to test and validate your application's functionality.
  • Data Replication: If your organization requires data to be available in multiple Firestore instances (e.g., for regional availability or regulatory compliance), this project can help you automate the data replication process.

By automating the Firestore data mirroring process, this project helps ensure that your critical data is always protected and accessible, without requiring constant manual intervention from your team.