Move Files in webMethods Based on Last Modified Date

Overview :

In this article, we will focus on creating a job that can move files from the incoming directory, including subdirectories, to an archive location based on the file age. This is a common request we receive as webMethods developers, especially in projects involving file handling.

Prerequisites :

• Basics of Java service in webMethods, please click here to check out the article on Java service.
• Basics of webMethods flow service.

Code Area :

In a real-life project, we may have subdirectories. We need to first list out the files available in the directory and its subdirectories. Then, we will check the last modified date for each file individually.

We will create two Java services here. The first Java service will list files along with paths from the directory, including subdirectories. In the second Java service, we will fetch the last modified date of the file. Although we can combine both Java services into a single service, we will keep them separate so that they can be utilized independently in other utilities as private or utility services.

Package structure :

• Create a package with the name ‘HG_FileUtil‘ and follow below package structure below.

List Files Java service:

In this Java service, we will write code to list out files from the directory, including the subdirectory, along with their paths. This information will help us further in obtaining the last modified date and moving the files accordingly based on the specified conditions.

• Create a Java service in ‘priv‘ folder with the name ‘listFilesContainingSubDir‘.
• Create an input variable named ‘maindirpath‘ of String type and an output variable named ‘fileList‘ of type StringList.
• Insert the following packages into the Java service to facilitate working with files and lists.
  

  import java.io.File;
  import java.util.ArrayList;
  import java.util.List;

• Insert the below code to fetch the pipeline input in the Java service.
  

  // pipeline
  IDataCursor pipelineCursor = pipeline.getCursor();
  String	maindirpath = IDataUtil.getString( pipelineCursor, "maindirpath" );
  pipelineCursor.destroy();

• Insert the below code in the Java service to call the methods to list out files using the path and load the output into the Java service output array.
  

  listFiles(maindirpath);
  // Convert ArrayList to String[] array
  String[] array = filePaths.toArray(new String[0]);
  		
  // pipeline
  IDataCursor pipelineCursor_1 = pipeline.getCursor();
  IDataUtil.put( pipelineCursor_1, "fileList", array );
  pipelineCursor_1.destroy();

• Insert the code below into the shared source area of the Java service to create a method that will be recursively called to list files from both its subdirectories and the main directory if they exist.
  

  private static List<String> filePaths = new ArrayList<>();
  	
  	private static void recursiveCollect(File[] arr, int level, String maindirpath) {
  	    // for-each loop for main directory files
  	    for (File f : arr) {
  	        // tabs for internal levels
  	        for (int i = 0; i < level; i++)
  	            System.out.print("\t");
  	
  	        if (f.isFile()) {
  	            String fullPath = maindirpath + "/" + f.getName();
  	            filePaths.add(fullPath);
  	        } else if (f.isDirectory()) {
  	            // recursion for sub-directories
  	            recursiveCollect(f.listFiles(), level + 1, maindirpath + "/" + f.getName());
  	        }
  	    }
  	}
  	
  	public static void listFiles(String maindirpath) {
  	    // File object
  	    File maindir = new File(maindirpath);
  	
  	    if (maindir.exists() && maindir.isDirectory()) {
  	        File arr[] = maindir.listFiles();
  	
  	        // Calling recursive method to collect paths
  	        recursiveCollect(arr, 0, maindirpath);
  	    }
  	}

• If you have followed all the steps correctly up to this point, your Java service will look like the example below.
  

  package HG_FileUtil.v1.services.priv;
  
  import com.wm.data.*;
  import com.wm.util.Values;
  import com.wm.app.b2b.server.Service;
  import com.wm.app.b2b.server.ServiceException;
  import java.io.File;
  import java.util.ArrayList;
  import java.util.List;
  import java.text.SimpleDateFormat;
  import java.util.Date;
  
  public final class listFilesContainingSubDir_SVC
  
  {
  
  	/** 
  	 * The primary method for the Java service
  	 *
  	 * @param pipeline
  	 *            The IData pipeline
  	 * @throws ServiceException
  	 */
  	public static final void listFilesContainingSubDir(IData pipeline) throws ServiceException {
  		// pipeline
  		IDataCursor pipelineCursor = pipeline.getCursor();
  			String	maindirpath = IDataUtil.getString( pipelineCursor, "maindirpath" );
  		pipelineCursor.destroy();
  		
  		
  		listFiles(maindirpath);
  		// Convert ArrayList to String[] array
  		String[] array = filePaths.toArray(new String[0]);
  		
  		
  		
  		
  		// pipeline
  		IDataCursor pipelineCursor_1 = pipeline.getCursor();
  		IDataUtil.put( pipelineCursor_1, "fileList", array );
  		pipelineCursor_1.destroy();
  		
  			
  	}
  	
  	// --- <<IS-BEGIN-SHARED-SOURCE-AREA>> ---
  	
  	private static List<String> filePaths = new ArrayList<>();
  	
  	private static void recursiveCollect(File[] arr, int level, String maindirpath) {
  	    // for-each loop for main directory files
  	    for (File f : arr) {
  	        // tabs for internal levels
  	        for (int i = 0; i < level; i++)
  	            System.out.print("\t");
  	
  	        if (f.isFile()) {
  	            String fullPath = maindirpath + "/" + f.getName();
  	            filePaths.add(fullPath);
  	        } else if (f.isDirectory()) {
  	            // recursion for sub-directories
  	            recursiveCollect(f.listFiles(), level + 1, maindirpath + "/" + f.getName());
  	        }
  	    }
  	}
  	
  	public static void listFiles(String maindirpath) {
  	    // File object
  	    File maindir = new File(maindirpath);
  	
  	    if (maindir.exists() && maindir.isDirectory()) {
  	        File arr[] = maindir.listFiles();
  	
  	        // Calling recursive method to collect paths
  	        recursiveCollect(arr, 0, maindirpath);
  	    }
  	}
  	
  	// --- <<IS-END-SHARED-SOURCE-AREA>> ---
  
  	/**
  	 * The service implementations given below are read-only and show only the
  	 * method definitions and not the complete implementation.
  	 */
  	public static final void getLastModifiedDateAndTime(IData pipeline) throws ServiceException {
  	}
  
  	final static listFilesContainingSubDir_SVC _instance = new listFilesContainingSubDir_SVC();
  
  	static listFilesContainingSubDir_SVC _newInstance() { return new listFilesContainingSubDir_SVC(); }
  
  	static listFilesContainingSubDir_SVC _cast(Object o) { return (listFilesContainingSubDir_SVC)o; }
  
  }

Get the last modified date of the file:

In this Java service, we will fetch the last modified date of the file using the full path returned from the previous Java service.

• Create a Java service inside ‘priv‘ folder with the name ‘getLastModifiedDateAndTime‘.
• Create an input variable named ‘filePath’ of string type and an output variable named ‘lastModifiedDate’ of string type.
  
• Insert the following packages into the Java service to facilitate working with date formats and files.
  

  import java.io.File;
  import java.text.SimpleDateFormat;
  import java.util.Date;

• Insert the below code into the Java service to get the input from the pipeline.
  

  // pipeline
  IDataCursor pipelineCursor = pipeline.getCursor();
  String	filePath = IDataUtil.getString( pipelineCursor, "filePath" );
  pipelineCursor.destroy();

• Insert the following code to fetch the last modified date of the file in ‘dd-MM-yyyy‘ format. Load the last modified date into the output and throw an exception when the file is not found in the given location.
  

  File file = new File(filePath);
  if (file.exists()) {
  	long lastModified = file.lastModified();
  	Date date = new Date(lastModified);
  	SimpleDateFormat formatter = new SimpleDateFormat("dd-MM-yyyy");
  	String formattedDate = formatter.format(date);
  	// pipeline
  	IDataCursor pipelineCursor_1 = pipeline.getCursor();
  	IDataUtil.put( pipelineCursor_1, "lastModifiedDate", formattedDate );
  	pipelineCursor_1.destroy();
  } else {
  	throw new ServiceException("File not found.");
  }

• If you have followed all the steps correctly, your Java service will look like the one below. You will also notice the imports and the shared resources shared in both Java services because they are in the same folder, which is treated as a package.
  

  package HG_FileUtil.v1.services.priv;
  
  import com.wm.data.*;
  import com.wm.util.Values;
  import com.wm.app.b2b.server.Service;
  import com.wm.app.b2b.server.ServiceException;
  import java.io.File;
  import java.util.ArrayList;
  import java.util.List;
  import java.text.SimpleDateFormat;
  import java.util.Date;
  
  public final class getLastModifiedDateAndTime_SVC
  
  {
  
  	/** 
  	 * The primary method for the Java service
  	 *
  	 * @param pipeline
  	 *            The IData pipeline
  	 * @throws ServiceException
  	 */
  	public static final void getLastModifiedDateAndTime(IData pipeline) throws ServiceException {
  		// pipeline
  		IDataCursor pipelineCursor = pipeline.getCursor();
  			String	filePath = IDataUtil.getString( pipelineCursor, "filePath" );
  		pipelineCursor.destroy();
  		
  		File file = new File(filePath);
  		if (file.exists()) {
  		    long lastModified = file.lastModified();
  		    Date date = new Date(lastModified);
  		    SimpleDateFormat formatter = new SimpleDateFormat("dd-MM-yyyy");
  		    String formattedDate = formatter.format(date);
  		 // pipeline
  			IDataCursor pipelineCursor_1 = pipeline.getCursor();
  			IDataUtil.put( pipelineCursor_1, "lastModifiedDate", formattedDate );
  			pipelineCursor_1.destroy();
  		  } else {
  		    throw new ServiceException("File not found.");
  		  }
  		
  			
  	}
  	
  	// --- <<IS-BEGIN-SHARED-SOURCE-AREA>> ---
  	
  	private static List<String> filePaths = new ArrayList<>();
  	
  	private static void recursiveCollect(File[] arr, int level, String maindirpath) {
  	    // for-each loop for main directory files
  	    for (File f : arr) {
  	        // tabs for internal levels
  	        for (int i = 0; i < level; i++)
  	            System.out.print("\t");
  	
  	        if (f.isFile()) {
  	            String fullPath = maindirpath + "/" + f.getName();
  	            filePaths.add(fullPath);
  	        } else if (f.isDirectory()) {
  	            // recursion for sub-directories
  	            recursiveCollect(f.listFiles(), level + 1, maindirpath + "/" + f.getName());
  	        }
  	    }
  	}
  	
  	public static void listFiles(String maindirpath) {
  	    // File object
  	    File maindir = new File(maindirpath);
  	
  	    if (maindir.exists() && maindir.isDirectory()) {
  	        File arr[] = maindir.listFiles();
  	
  	        // Calling recursive method to collect paths
  	        recursiveCollect(arr, 0, maindirpath);
  	    }
  	}
  	
  	// --- <<IS-END-SHARED-SOURCE-AREA>> ---
  
  	/**
  	 * The service implementations given below are read-only and show only the
  	 * method definitions and not the complete implementation.
  	 */
  	public static final void listFilesContainingSubDir(IData pipeline) throws ServiceException {
  	}
  
  	final static getLastModifiedDateAndTime_SVC _instance = new getLastModifiedDateAndTime_SVC();
  
  	static getLastModifiedDateAndTime_SVC _newInstance() { return new getLastModifiedDateAndTime_SVC(); }
  
  	static getLastModifiedDateAndTime_SVC _cast(Object o) { return (getLastModifiedDateAndTime_SVC)o; }
  
  }

Flow Service:

This is the final service, which will internally call the Java services created in the previous steps. It can be scheduled to run once a day or based on your requirements.

• Create a flow service named ‘moveFiles‘ inside the ‘scheduler‘ folder.
• Insert a Try and Catch block in the flow service and provide a meaningful comment.
• Inside the catch block, invoke ‘pub.flow:getLastError‘, and then use ‘pub.flow:debugLog‘ to log the error message in case the service throws an exception.
  • Hardcode the ‘message‘ variable with ‘Scheduler service failed with error ErrorDescription: %lastError/error%. TrackingID=%trackingID%‘ and enable ‘Perform pipeline variable substitution‘ Finally, drop the ‘message’ variable.
• Inside the Try block insert the Map step and invoke ‘pub.utils:generateUUID‘ as a transformer to generate a unique tracking ID and follow the below mapping.
  • UUID(pub.utils:generateUUID) -> trackingID(Create it in the output pipeline as a string).
• Invoke the ‘pub.flow:debugLog‘ service to write the scheduler start log into the server log. Hardcode the ‘message’ variable with ‘Scheduler started to move old files to archive location. TrackingID=%trackingID%‘ and enable ‘Perform pipeline variable substitution‘ Finally, drop the ‘message’ variable.
• Invoke the Map step and create three variables with the following names: ‘monitoringLocation,’ ‘archiveLocation,’ and ‘dateGap.’ Hardcode these variables to store the monitoring directory, archive directory, and the date gap, which will assist in moving files based on their age. In this project, we are hardcoding it, but you can alternatively store it in global variables or the configuration. By doing so, you can fetch and map these values, allowing easy modification of directories without needing to alter the code base.
  • monitoringLocation -> ‘C:/Users/harmonigate/Downloads/testDir/‘ (update it according to your requirement).
  • archiveLocation -> ‘C:/Users/harmonigate/Downloads/archive/‘ (update it according to your requirement).
  • dateGap -> ‘10‘ (This will delete 10-day-old files, you can change it based on your need).
• Invoke ‘pub.date:getCurrentDateString‘ service to get the current date and follow the below mapping.
  • pattern -> ‘dd-MM-yyyy‘
  • value -> currentDate
• Invoke ‘HG_FileUtil.v1.services.priv:listFilesContainingSubDir‘ service and follow the below mapping to list files.
  • monitoringLocation -> maindirpath (drop maindirpath variable).
• Invoke the ‘Loop‘ step and put the ‘Input Array‘ with ‘fileList‘ to loop over the file list.
  
• Within the ‘Loop‘ step, insert a Try-Catch block to handle errors when retrieving the last modified date of the file or moving it to the archive location. In the case of an issue with a single file, it should not exit the flow; instead, it should continue processing other files.
• Inside the catch block, invoke ‘pub.flow:getLastError‘, and then use ‘pub.flow:debugLog‘ to log the error message.
  • Hardcode the ‘message‘ variable with ‘Error occurred while moving file filePath=”%fileList%” Error Description=”%lastError/error%” TrackingID=%trackingID%‘ and enable ‘Perform pipeline variable substitution‘ Finally, drop the ‘message’ variable.
• Inside the nested Try block under the ‘Loop‘, insert one map step and invoke ‘pub.string:replace‘ as a transformer to replace ‘//’ with ‘/’ if it exists in the file path, and follow the mapping below.
  • fileList -> inString.
  • searchString -> ‘//‘.
  • replaceString -> ‘/’.
  • value -> fileList.
• Invoke ‘HG_FileUtil.v1.services.priv:getLastModifiedDateAndTime‘ service to get the last modified date and follow the below mapping.
  • fileList -> filePath.
  • lastModifiedDate -> lastModifiedDate.
• Invoke ‘pub.date:calculateDateDifference‘ to calculate the difference and follow the below mapping.
  • lastModifiedDate -> startDate.
  • currentDate -> endDate.
  • startDatePattern -> ‘dd-MM-yyyy‘
  • endDatePattern -> ‘dd-MM-yyyy‘.
• Invoke ‘BRANCH’ step and set the ‘Evaluate labels’ to ‘true’.
• Invoke the ‘pub.file:moveFile‘ step inside the ‘BRANCH‘ step, setting the ‘Label’ to ‘%dateDifferenceDays%>=%dateGap%‘, and follow the mapping below to move the file to the archive location.
  • fileList -> fileName.
  • archiveLocation -> targetDirectory.
• Invoke ‘Map‘ step inside the ‘Branch‘ step and set the ‘Label‘ to ‘$default‘.
• Move the ‘pub.flow:debugLog‘ step outside the ‘Loop,’ hardcode the ‘message‘ variable with ‘Scheduler completed processing old files to archive location. TrackingID=%trackingID%‘ and enable the ‘Variable pipeline substitution‘ option.
• If you have followed everything correctly up to this point, your flow service should look like the one below.
  
  

Now you can run the service, and if there is any file in the monitoring directory that satisfies the condition, it will be moved to the archive location. Please click the download link below to download the source code.