Copying directories in linux only if it contains the specified file
I need to make a copy of all the folders under /parent (into a new location) but only if it has 123.dat — in which case I need to copy the folder too, but none of the other files it contains. So that this:
|parent | |a | | 123.dat | | 456.dat | |b | | 123.dat | | 789.dat | |c | | 456.dat | | 789.dat |parent | |a | | 123.dat | |b | | 123.dat How do I do this in linux? This area is not my expertise and so far I have no success with my attempts to search for something similar.
Do you need to make a copy of parent in another location too, or will parent be the destination and have a new name?
You could copy «parent» then delete everything that’s not 123.dat using something like find . -type f -not -name «123.dat» -ls (replace ‘ls’ with ‘delete’ to do the deletion) very simply.
@pbhj This is a simple solution indeed and it works, but this is just an example and I need to copy almost 700Gb of data and copying the entire folder and then deleting it would be very expensive.
2 Answers 2
File/Folder structure:
$ find src | sort src src/a src/a/123.dat src/a/456.dat src/b src/b/123.dat src/b/768.dat src/c src/c/456.dat src/c/768.dat Copy matching files, preserving relative path (shallow, not going deeper than 1 folder):
$ (cd src && cp -v --parents -- */123.dat ../dest) a -> ../dest/a 'a/123.dat' -> '../dest/a/123.dat' b -> ../dest/b 'b/123.dat' -> '../dest/b/123.dat' - I used a subshell with ( and ) in order not to change the original working directory when using cd . I had to enter src before executing cp in order to not create src/ as base dir in dest .
- Will not work with file counts higher than the argument limit of bash (usually around 65k, if I’m not mistaken)
Alternative approach (using find with adjustable depth limit):
$ (cd src && find . -maxdepth 2 -type f -name '123.dat' -exec cp -v -t "../dest" --parents <> +) ./b -> ../dest/./b './b/123.dat' -> '../dest/./b/123.dat' ./a -> ../dest/./a './a/123.dat' -> '../dest/./a/123.dat' - I used a subshell with ( and ) in order not to change the original working directory when using cd . I had to enter src before executing find in order to not create src/ as base dir in dest .
- I specified -type f to make sure only files with name 123.dat are considered, not directories that happen to have that name
Alternative approach (using rsync , without depth limitation):
$ rsync -rv --include=123.dat --include='*/' --exclude='*' --prune-empty-dirs src/ dest building file list . done created directory dest ./ a/ a/123.dat b/ b/123.dat sent 205 bytes received 90 bytes 590.00 bytes/sec total size is 0 speedup is 0.00 Double check:
$ find dest dest dest/b dest/b/123.dat dest/a dest/a/123.dat - The trailing / of src/ is intentional, so that only the contents of the folder are copied, not the folder itself.
- —exclude=’*’ excludes everything by default
- —include=’*/ overrides the exclude and includes all folders
- —include=’123.dat’ overrides the exclude and includes files (and folders) with names ‘123.dat’
- —prune-empty-dirs makes sure no empty folders are created (e.g. c )
The destination shouldn’t copy all the files — only the ones named 123.dat — but I guess this copies everything?
You’re right, I didn’t really read the second part of your question I guess, sorry. So you want to copy only the matching files while keeping the directory structure, gotcha. Do you need this to work for more than 1 level?
Ok, cool. I added a different approach now, using rsync, which actually should work for any depth. I looked if I could somehow force-limit the depth, not knowing whether you actually need that or not — but couldn’t find a good solution to that yet.
One approach is to make a new version of the parent directory in the new location, then copy across the subdirectories if they contain 123.dat. This uses the Bash shell’s globbing feature to find the subdirectories so will only work on directories immediately below parent. in the example I’ll assume parent is in a directory called /location1/ and will be moving to /location2/ :
mkdir -p /location2/parent for d in /location1/parent/* do if [[ -e "$d"/123.dat ]]; then cp -r "$d" /location2/parent done fi As a CLI one-liner that would be:
mkdir -p /location2/parent; for d in /location1/parent/*; do if [[ -e "$d"/123.dat ]]; then cp -r "$d" /location2/parent; done; fi It’s possible to improve this by using find to make it more efficient and add multi level subdirectories, or by putting the source directory, target directory and file to search for into variables. This should do what you need for now.
To copy a directory if it contains a particular file, but not copy any of the other files there’s a less than elegant solution which requires cd ing into the source directory, the cd — part at the end returns you to your original directory:
mkdir -p /location2/parent; cd /location1/parent/ && for d in ./*; do if [[ -e "$d"/123.dat ]]; then cp --parents "$d"/123.dat /location2/parent/; fi ; done; cd - mkdir -p /location2/parent cd /location1/parent/ && for d in ./* do if [[ -e "$d"/123.dat ]]; then cp --parents "$d"/123.dat /location2/parent/ fi done cd - Linux how to copy but not overwrite? [closed]
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
I want to cp a directory but I do not want to overwrite any existing files even it they are older than the copied files. And I want to do it completely noninteractive as this will be a part of a Crontab Bash script. Any ideas?
8 Answers 8
-n, --no-clobber do not overwrite an existing file (overrides a previous -i option) cp -n myoldfile.txt mycopiedfile.txt Note, this will exit with an error if the file exists. To exit with success, try cp -n source.txt destination.txt || true
@galenandrew Confirmed. Thank you. My project wasn’t building in Xcode after adding a run script to my target.
Even with Ubuntu 18.04, the behavior is the same, that when the file exists, the command exit without erro, that is, an no op! This is dangerous!
rsync -a -v --ignore-existing src dst As per comments rsync -a -v src dst is not correct because it will update existing files.
Complete command rsync -a -v —ignore-existing
If a previous copy was interrupted and only a truncated file copied, I’m sure cp -u won’t re-copy it. but will rsync, with —ignore-existing? Probably not either. so that makes them perfectly equivalent, right?
absolutely. I don’t want to install rsync into my docker container, just to do a copy! I can’t imagine why we should use a cannon when a BB gun will suffice.
Is what you want. See the man page.
The man page: -n Do not overwrite an existing file. (The -n option overrides any previous -f or -i options.)
false | cp -i source destination 2>/dev/null Updating and not overwriting is something different.
Obviously, this command won’t work if you’ll try to copy more than ARG_MAX files. To work-around this case, check this link.
For people that find that don’t have an ‘n’ option (like me on RedHat) you can use cp -u to only write the file if the source is newer than the existing one (or there isn’t an existing one).
[edit] As mentioned in the comments, this will overwrite older files, so isn’t exactly what the OP wanted. Use ceving’s answer for that.
OP asked not to overwrite existing files even if they are older than copied files, so -u doesn’t actually fit purpose.
It might not be what the OP asked for, but it’s exactly what I needed for my Uberspace (Centos 7). Thanks!
Centos copy file into another file, if exists, create a version
Does anyone know of a way to (via bash) setup a «versioning» copy of a file into another? For example: I am copying file into file.bak. If file.bak exists, I am currently overwriting. What I’d like to do is set it up so that it creates multiple files: file, file.bak, file.bak.1, file.bak.2, etc. Right now, I’m using:
What files are you backing up? You might want to consider using a source-control system like git to manage the versioned backups for you.
Then consider using RCS instead of numbered backup files. You’ll get better control of diffs and history, opportunity for comments, none of the overhead of a repository for a large VCS like SVN or Git, and RCS is most certainly available for your distribution of Linux. If you need to migrate to a larger git-based version control system in the future, it’s easier to do that from RCS than from numbered file versions.
3 Answers 3
repeat few times to see the result.
[ -e file.bak ] && cp -r file file.bak.$(date +%s) || cp -r file file.bak This will create a unique backup if file.bak already exists in the form file.bak.1411505497
There are many ways to skin this cat.
Since you’re using Linux, it’s likely you’ve got the GNU mv command, which may include a —backup option. You could wrap this in a shell function:
bkp() < file="$1" if [ -f "$file" ]; then /bin/mv -v --backup=numbered "$(mktemp $XXX)" "$file" #/bin/rm "$file" fi > You can put this in your .bashrc , for example. Then you can use this as follows:
This will copy foo to numbered backup files. You can uncomment the rm if this is, for example, a log file that you’re rotating.
Another option, which is more portable to operating systems that don’t use GNU tools (i.e. FreeBSD, OSX) might be something like this quick-and-dirty solution might work:
bkp() < file="$1" if [ -f "$file" ]; then # increment existing files up to 10 for n in ; do if [ -f $file.$n ]; then # remove -v if you want less noise. mv -v "$.$n" "$.$[n+1]" fi done # move the original to first backup position mv "$file" "$file.1" else echo "Not found: $file" >&2 fi > It suffers in that it won’t compact your list of files (and will throw errors) if some numbers are missing, but that’s stuff you can add if it’s important. You’d use it pretty much the same way, changing the final mv to a cp if you need to keep the original in place.
Final option I’ll mention is in comments as well. Since you’ve said that you’re using this solution to back up «system files» (which I assume you mean to be things in /etc/) you should consider using an actual version control system to control your versions of these files.
Many options exist, but I’d recommend RCS for its simplicity and low overhead. Simply install the package, mkdir /etc/RCS to keep your /etc directory clean, read the man pages for rcs , ci , co , rlog , rcsdiff and perhaps rcsintro , and you’re good to go. You’ll get better control of diffs and history, opportunity for comments, none of the overhead of a repository for a large VCS like SVN or Git. I’ve been using this on various servers for years, as RCS is still built in to the base system in FreeBSD. 🙂
Copy the files if exist else skip in bash
I want to first check if file on my source location falls within a date range: if yes then copy else skip and check the next file. Can someone help how this can be done as I am new to shell script.
I don’t understand your script. What are the -newrmt parameters to find ? Is the date part of the filename or do you search for the last modification date?
Note that find won’t find files that don’t exist. You can exclude directories (etc) by adding -type f so that only files are found. Your notation with the quote is incorrect. Using the for notation works only if the file names do not contain spaces (or tabs or newlines). Remember to quote uses of variables; remember to prefix the date variables with a $ . Generally, you get the best responses if your code works a bit — yours does not run (invalid syntax).
BTW, in shell, «clarity» generally needs to come second to correctness: What looks simplest to the reader is very often buggy (albeit frequently in ways that will escape simple testing but then show up later); avoiding hidden pitfalls in the language tends to require knowing and using a body of idiom existing for the purpose.
1 Answer 1
find already will return only entries that already exist; there’s no issue there, but the code you’re using to iterate over its results is badly broken.
The problem here isn’t with skipping, but with correct syntax in general, and (secondarily) with correctness-related practices. Even if you fixed the most immediate bug in the original code, you would have fallen afoul of BashPitfalls #1, and DontReadLinesWithFor.
See UsingFind for a full discussion of best practices around using find . That said, among the correct approaches are:
#!/bin/bash # ^^^- important: not /bin/sh while IFS= read -r -d '' filename; do mv "$filename" /to_path done < <(find filedirectory_path/ -type f -newermt "$date1" ! -newermt "$date2" -print0) find filedirectory_path/ -type f -newermt "$date1" ! -newermt "$date2" -exec mv '<>' /to_path \;