Sunday, November 22, 2026
TR
Linux

Automating Elasticsearch Year-Month Data Migration and Reindexing

Automating Elasticsearch Year-Month Data Migration and Reindexing
Contents
  1. Introduction
  2. Purpose of the Script
  3. Key Features
  4. 1. Dynamic Timestamp Field Detection
  5. 2. Smart Date Format Detection
  6. 3. Data Validation Mechanism
  7. 4. Leap Year Calculation
  8. How It Works
  9. Step 1: Connection Details
  10. Step 2: Index Structure Analysis
  11. Step 3: Testing and Validation
  12. Step 4: Monthly Reindex Operation
  13. Advantages
  14. 1. Performance Improvement
  15. 2. Data Lifecycle Management
  16. 3. Manageability
  17. 4. Flexibility
  18. Important Parameters
  19. Timeout Settings
  20. Batch Size
  21. Load Balancing
  22. Use Cases
  23. 1. Data Migration
  24. 2. Index Reorganization
  25. 3. Multi-Cluster Synchronization
  26. 4. Data Archiving
  27. Best Practices
  28. Potential Improvements
  29. Things to Watch Out For
  30. Conclusion

Introduction

Managing large datasets in Elasticsearch environments, especially organizing long-term log data, is critical for system performance and data accessibility. In this article, we will examine an automated bash script that partitions the data in Elasticsearch by year and month and moves it to a new Elasticsearch system.

Purpose of the Script

This automation tool splits an existing index in a source Elasticsearch cluster into smaller time-based indices and moves them to the target system. For example, all the data in a single index named sistemlog is organized into monthly indices such as sistemlog-2022-01, sistemlog-2022-02.

Key Features

1. Dynamic Timestamp Field Detection

The script offers a flexible structure to automatically detect the timestamp field in the data:

# Fetches sample documents and shows the timestamp fields
- @timestamp
- timestamp  
- createdAt
- created_date

The user can specify which field to use or use the default @timestamp field.

2. Smart Date Format Detection

Different date formats can be used in Elasticsearch. The script supports three different formats:

  • ISO 8601 (Z timezone): 2022-12-31T23:59:59.999Z
  • Simple date format: 2022-12-31
  • Without timezone: 2022-12-31T23:59:59

The script tries these formats in order and automatically detects which format is in use.

3. Data Validation Mechanism

Before starting the bulk migration, the script asks the user for a test year and month and checks whether there is data for that period. This way:

  • Starting the operation with wrong parameters is prevented
  • It confirms that the timestamp field works correctly
  • Wasted time is avoided

4. Leap Year Calculation

The script automatically calculates the number of days in February:

# Leap year check
if [ $((year % 4)) -eq 0 ] && ([ $((year % 100)) -ne 0 ] || [ $((year % 400)) -eq 0 ]); then
  LAST_DAY="29"
else
  LAST_DAY="28"
fi

How It Works

Step 1: Connection Details

The script uses connection parameters for the source and target systems:

SOURCE_HOST="http://xxx.xxx.xxx.xxx:9200"
SOURCE_USER="*******"
SOURCE_PASS="***************"

DEST_HOST="https://localhost:9200"
DEST_USER="*******"
DEST_PASS="***************"

Security Note: In production environments, use environment variables or secret management tools instead of writing passwords directly into the script.

Step 2: Index Structure Analysis

The script first fetches a sample document from the source index and lists the time fields in it:

curl -s -u $SOURCE_USER:$SOURCE_PASS \
  "$SOURCE_HOST/$SOURCE_INDEX/_search?size=1"

Step 3: Testing and Validation

A data query is run for the test year and month provided by the user:

{
  "query": {
    "range": {
      "@timestamp": {
        "gte": "2022-12-01T00:00:00.000Z",
        "lte": "2022-12-31T23:59:59.999Z"
      }
    }
  }
}

Step 4: Monthly Reindex Operation

A separate reindex operation is performed for each year and month:

curl -k -s -u $DEST_USER:$DEST_PASS \
  -X POST "$DEST_HOST/_reindex?wait_for_completion=true&timeout=2h" \
  -H 'Content-Type: application/json' -d'{
    "source": {
      "remote": {
        "host": "http://xxx.xxx.xxx.xxx:9200",
        "username": "*******",
        "password": "***************",
        "socket_timeout": "60m",
        "connect_timeout": "10m"
      },
      "index": "sistemlog",
      "size": 1000,
      "query": {
        "range": {
          "@timestamp": {
            "gte": "2022-12-01T00:00:00.000Z",
            "lte": "2022-12-31T23:59:59.999Z"
          }
        }
      }
    },
    "dest": {
      "index": "sistemlog-2022-12"
    }
  }'

Advantages

1. Performance Improvement

  • Smaller indices are queried faster
  • More effective shard management is achieved
  • Scanning of unnecessary data is avoided

2. Data Lifecycle Management

  • Old data can be easily deleted or archived
  • Index-level retention policies can be applied
  • Moving from costly hot storage to cold storage becomes easier

3. Manageability

  • It is clear which period's data is stored where
  • Troubleshooting becomes easier
  • Backup and restore operations are more controlled

4. Flexibility

  • Different timestamp fields are supported
  • Compatible with multiple date formats
  • The desired years can be selected

Important Parameters

Timeout Settings

wait_for_completion=true&timeout=2h
socket_timeout: "60m"
connect_timeout: "10m"

These settings provide enough time for the operation to complete during large data transfers.

Batch Size

"size": 1000

1000 documents are transferred at a time. This value can be tuned according to network and system capacity.

Load Balancing

sleep 1
```

Waiting 1 second between each monthly operation prevents excessive load on the systems.

## Output and Reporting

The script provides detailed information:
```
========================================
📅 2022-12 (01-31)
📦 Hedef Index: sistemlog-2022-12
📊 Documents to migrate: 45,230
✅ Total: 45230 | Created: 45230 | Updated: 0 | Duration: 127s
📊 sistemlog-2022-12: 45230 documents
========================================

Use Cases

1. Data Migration

Building a more organized structure while moving data from an old Elasticsearch cluster to a new one.

2. Index Reorganization

Splitting a single large existing index into manageable monthly indices.

3. Multi-Cluster Synchronization

Copying data for specific periods between different environments (development, staging, production).

4. Data Archiving

Organizing old data by time before moving it to a different storage system.

Best Practices

  1. Start with Small Tests: Test with a single month first
  2. Monitor System Resources: Track CPU, memory and disk I/O metrics
  3. Take a Backup: Always take a backup before the operation
  4. Run During Off-Peak Hours: To avoid affecting production systems
  5. Keep Logs: Redirect the script output to a file
./reindex_script.sh 2>&1 | tee reindex_$(date +%Y%m%d_%H%M%S).log

Potential Improvements

  1. Parallel Processing: Fork/background jobs to process multiple months at the same time
  2. Error Handling: Automatic retry mechanism for failed transfers
  3. Progress Bar: Visual progress indicator for large datasets
  4. Email/Slack Notifications: Automatic notification when the operation completes
  5. Dry-Run Mode: Simulation before performing the real operation

Things to Watch Out For

  • Network Bandwidth: Large data transfers increase network traffic
  • Disk Space: Make sure the target system has enough disk space
  • Mapping Compatibility: It is important that the source and target index mappings are compatible
  • Shard Count: Reducing the shard count for small indices improves performance

Conclusion

This bash script is an important automation tool for Elasticsearch data management. Especially for organizations that manage long-term log data, organizing data by time provides critical benefits for both performance and manageability. The script's flexible structure allows it to be easily adapted to different Elasticsearch environments.

When used correctly, this tool saves time by automating data migration and minimizes human error. However, it must be tried in a test environment before being used in production and configured in accordance with the organization's security policies.

Kerim Koç

Systems and infrastructure engineer. I write about Linux, open source, high availability and security.