Ytdyklly

I have a remote directory (SSH enabled) on 11.11.11.11/:/path/from (an Ubuntu machine) that has millions of tiny .txt small size files, doing like simple ls command or even opening the /path/from/ directory using WinSCP is impossible because there are millions of files.

My goal is to find few thousands files that have a specific name pattern and copy them to another remote location (e.g 22.22.22.22:/path/to).

Here is what I've tried so far:

scp --exec=`find /path/from -name 'A*random*' -mtime +0 -mtime -10` user@22.22.22.22:/path/to

But it takes a long long time to do that, like I said the /path/from/ directory contains literally millions of files.

Do you have suggestion to make it faster? Using rsync? How should I do it? And how can I limit the find result to a certain number, let's say 1000, because I only know how to limit it using the last modified on date, -mtime.

Maybe you can use find in combination with cpio to create a stream from your many files on one machine and extract the files with cpio on the other machine.

ssh user@source "cd sourcedir && find ./ -xdev -name 'A*random*' -print | cpio -o -Bav -H crc" | ssh user@target "cd destinationdir && cpio -i -vumd"

This (untested) solution is based on https://www.netroby.com/view/3602 . There you will find some explanation of the arguments for find and cpio .

Try locate

It seems find is too slow for this application.

• There is a faster tool to find files, locate. It uses a database, that must be updated for locate to find the newest files.



• 
  

  updatedb creates or updates a database used by locate. If the database already exists, its data is reused to avoid rereading directories that have not changed.

  
  
  

  This update process is very fast compared to find and when the database is updated, locate will find all files (and it is much faster than find).

  
  

Usage

• 
  

  Create and next times update the database

  
  
  

  sudo updatedb
  
  

  
  


• 
  

  Find the relevant files. locate provides several useful options but not a lot of options like find. You might be able to design a useful pattern for your purpose.

  
  
  

  I suggest two command lines, that you may modify, and later combine with scp or rsync.

  
  
  

  You can limit the number of files with --limit

  
  
  

  If you search only in /path/from/ and not in sub-directories

  
  
  

  locate --regex --limit 1000 '/path/from/A.*random.*'
  
  

  
  
  

  If you search not in /path/from/ itself but in its sub-directories

  
  
  

  locate --regex --limit 1000 '/path/from/.*/A.*random.*'
  
  

  
  
  

  See man locate for more details.

  
  

General comments

• 
  

  Maybe you should modify how these files are written and stored, for example with several sub-directories, so that there are not too many files in each directory, for example one sub-directory for each date (2019-02-12, 2019-02-13 ...),

  
  
  

  or even better, like many photo managers are storing picture files,

  
  
  
  
  
  • one level of subdirectories for each year
  
  
  • the next level of subdirectories for each month of the year
  
  
  • the final level of subdirectories for each day of the month, where the files are stored.
  
  
  
  



• Maybe you can also remove some files (for example when they are getting too old).

1. tar, zip, or compress everything under the folder into one source.tar file; can quickly do via tar -cf /sourcedirectory; this will be however large containing all 100,000+ files now in one file.


2. transfer this one file however you like


3. once at the destination, tar -xf source.tar or unzip/uncompress appropriately back to the original folder structure containing 100,000+ files.