Bdtyktl

Basically I am looking for a script to automate stuff (See picture below) in Ubuntu. I'm thinking of using a bash script but other solutions (eg. python?) would also be excellent.

1) Suppose I have a number of real directories "Folder 1" and "Folder 2" with subfolders and files. Assuming that files in the corresponding folders 1 and 2 have unique names. How can I create a new merged folder where each of the files are symlinks to the original folders?

2) The script should also offer an option to prune broken symbolic links in the merged folder.

The reason I want to do this is because I'd like to improve how my stuff are organised. eg. "Folder 1|2" might be data obtained in different chronological time points. Then I'd create Merged_Folder1, Merged_Folder2 etc for different projects without duplicating large files.

Edit: This question differs from this post as I would like to merge corresponding nested subfolders with the same names. The question in the previous post simply links the top directories under the sources to the target and cannot merge nested subfolders. Notice that in my case none of the folders would be symlinks, only the files are symlinks.

Edit2: I should have clarified that I wish the code to merge arbitrary levels of nested subfolders, not just two levels. Hence I've added "File J" and "File I" in the example illustration.

I think the following shellscript will do what you want

• The original folders are in main
  


• There can be subfolders in several levels


• The merged folder is links
  


• run the main shellscript script in the directory containing main and links
  

script

#!/bin/bash



mkdir -p links



find main -type d -exec bash -c 

'for pathname do

  #echo "------------------------------${pathname} ${pathname#*/*/}"

  if [ "${pathname/*/*/}" != "${pathname}" ]

  then

   mkdir -p "links/${pathname#*/*/}"

  fi

 done' bash {} +



find main -type f -exec bash -c 

'curdir=$(pwd)

for pathname do

  tpat=${pathname/main/}

  ln -s "${curdir}/${pathname}" "links/${tpat#*/}" 2> /dev/null;

 done' bash {} +



find links -type l -exec bash -c 

'for pathname do

  LANG=C

  file "$pathname"|grep -o  "$pathname: broken symbolic link" > /dev/null; 

  if [ $? -eq 0 ];then rm "$pathname";fi

 done' 

 bash {} +

Demo

$ rm -r links



$ find main

main

main/f 4

main/f 4/s 4

main/f 4/s 4/k 4

main/asdf

main/f2

main/f2/s3

main/f2/s3/h

main/f2/s3/g

main/f2/s3/ss

main/f2/s3/ss/i

main/f2/s1

main/f2/s1/c

main/f2/s1/d

main/j

main/f1

main/f1/s2

main/f1/s2/x y

main/f1/s2/f

main/f1/s2/e

main/f1/s1

main/f1/s1/a

main/f1/s1/b



$ ./script  # doing it



$ find links/ -type l -exec file {} ;

links/s2/x y: symbolic link to /media/multimed-2/test/test0/matohak/main/f1/s2/x y

links/s2/f: symbolic link to /media/multimed-2/test/test0/matohak/main/f1/s2/f

links/s2/e: symbolic link to /media/multimed-2/test/test0/matohak/main/f1/s2/e

links/s3/h: symbolic link to /media/multimed-2/test/test0/matohak/main/f2/s3/h

links/s3/g: symbolic link to /media/multimed-2/test/test0/matohak/main/f2/s3/g

links/s3/ss/i: symbolic link to /media/multimed-2/test/test0/matohak/main/f2/s3/ss/i

links/s 4/k 4: symbolic link to /media/multimed-2/test/test0/matohak/main/f 4/s 4/k 4

links/asdf: symbolic link to /media/multimed-2/test/test0/matohak/main/asdf

links/s1/a: symbolic link to /media/multimed-2/test/test0/matohak/main/f1/s1/a

links/s1/c: symbolic link to /media/multimed-2/test/test0/matohak/main/f2/s1/c

links/s1/b: symbolic link to /media/multimed-2/test/test0/matohak/main/f1/s1/b

links/s1/d: symbolic link to /media/multimed-2/test/test0/matohak/main/f2/s1/d

links/j: symbolic link to /media/multimed-2/test/test0/matohak/main/j



$ ln -s main/asdf links/asdf-b  # create a broken link



$ find links/ -type l -name "asdf*" -exec file {} ;

links/asdf-b: broken symbolic link to main/asdf

links/asdf: symbolic link to /media/multimed-2/test/test0/matohak/main/asdf



$ ./script  # this time only to remove the broken link



$ find links/ -type l -name "asdf*" -exec file {} ;

links/asdf: symbolic link to /media/multimed-2/test/test0/matohak/main/asdf

$

A case that would allow you to specify which folder under main/* to merge

#!/bin/bash

# First argument is target, following arbitrary number of target folders

# eg. ./script.sh links main/f1 main/f2 main/f3



argc=$#

argv=($@)



mkdir -p ${argv[0]}



for (( j=1; j<argc; j++ )); do



    find ${argv[j]} -type d -exec bash -c 

    'for pathname do

      #echo "------------------------------${pathname} ${pathname#*/*/}"

      if [ "${pathname/*/*/}" != "${pathname}" ]

      then

       mkdir -p "'${argv[0]}'/${pathname#*/*/}"

      fi

     done' bash {} +



    find ${argv[j]} -type f -exec bash -c 

    'curdir=$(pwd)

    for pathname do

      tpat=${pathname/${argv[j]}/}

      ln -s "${curdir}/${pathname}" "'${argv[0]}'/${tpat#*/}" 2> /dev/null;

     done' bash {} +



    find ${argv[0]} -type l -exec bash -c 

    'for pathname do

      LANG=C

      file "$pathname"|grep -o  "$pathname: broken symbolic link" > /dev/null; 

      if [ $? -eq 0 ];then rm "$pathname";fi

     done' 

     bash {} +



    find ${argv[0]} -type d -empty -delete  # Removes empty dir in target



done

Got it... this python code should be able to walk through arbitrary number of nested directories and create symlinks for files all merged into the target directory. Arguments are the target and the source directories respectively. Source directories should be relative to the target dir. eg.

python script.py ./merged_folder ../folder1 ../folder2 ../folder3

import os

import sys

import time

'''

Loops through merge_symlink.

See https://askubuntu.com/questions/1097502/script-for-merging-files-in-multiple-nested-directories-by-symbolic-link/

KH Tam Nov 2018 (matohak)



Use: python merge_symlink.py ./target ../folder1 ../folder2 ../folder3

Note that if overwrite==True and there are duplicated filenames, links will be overwritten by the last argument's

'''



def merge_symlink(sources, overwrite=True, remove_empty_dir=True, verbose=False):

    '''

    See https://askubuntu.com/questions/1097502/script-for-merging-files-in-multiple-nested-directories-by-symbolic-link/

    Function to be run in the target directory.



    :param sources: a list of directories where the files in the subdirectories are to be merged symbolically. Path relative to the target. eg. ["../folder1", "../folder2"]

    :param overwrite: Bool, whether to overwrite existing symbolic links

    :param remove_empty_dir: Bool, whether to remove empty directories in target.

    :param verbose: Prints stuff.

    :return: None

    '''



    # Creating symlinks and folders

    for source in sources:

        for dirName, subdirList, fileList in os.walk(source):

            # print(dirName, fileList)  # print all source dir and files

            if source[-1] == "/": source=source[:-1]



            target_dir = dirName.replace(source, '.', 1)

            depth = dirName.count("/") - source.count("/")



            try:

                os.mkdir(os.path.join(target_dir))

            except FileExistsError:

                pass



            for file in fileList:

                targetlink = os.path.join(target_dir, file)

                try:

                    os.symlink(os.path.join("../"*depth + dirName, file), targetlink)

                except FileExistsError:

                    if overwrite and not (isvalidlink(targetlink)==2):  # Never replace a real file with a symlink!

                        os.remove(targetlink)

                        os.symlink(os.path.join("../" * depth + dirName, file), targetlink)

                        if verbose: print('overwriting {}'.format(targetlink))



    # Pruning broken links and then deleting empty folders.

    for dirName, subdirList, fileList in os.walk("./"):

        for file in fileList:

            link = os.path.join(dirName,file)

            if isvalidlink(link)==0:

                os.remove(link)

                if verbose: print("Removing broken symlink: {}".format(link))



    if remove_empty_dir:

        for dirName, subdirList, fileList in os.walk("./"):

            if fileList== and subdirList== and dirName!="./":

                os.rmdir(dirName)



# Checks if file is a broken link. 0: broken link; 1: valid link; 2: not a link

def isvalidlink(path):

    if not os.path.islink(path):

        return 2

    try:

        os.stat(path)

    except os.error:

        return 0

    return 1



if __name__ == "__main__":



    target = sys.argv[1]

    sources = sys.argv[2:]      # Inputs should be relative to the target dir.

    overwrite = False

    loop = False

    looptime = 10



    os.chdir(target)

    if not loop:

        merge_symlink(sources, overwrite=overwrite)

    else:

        while loop:

            merge_symlink(sources, overwrite=overwrite)

            time.sleep(looptime)

Thank @JacobVlijm for the link and @sudodus for helping!