File Polling in webMethods

What if File Polling ?

File polling is a technique used by the Integration Server to keep a close watch on a specific directory for incoming files. When a file is placed into this monitored directory, the Integration Server takes action by retrieving the file and passing it to a designated processing service for further handling. This approach allows for efficient automation of file-based workflows.

Within the realm of webMethods file polling, you have the capability to define the types of files that the Integration Server should focus on for processing. Additionally, you can establish rules to manage these files based on their processing outcomes. This includes options to direct files to error directories or archive directories, contingent upon the outcome of their processing, which enhances the management and organization of these files.

To grasp the concept of file polling, let’s break it down step by step. First, file needs to be placed into monitoring directory. This folder is regularly monitored by the integration server. As soon as the file arrives in the monitoring directory, it’s transferred to a working directory. Simultaneously, the processing service gets the content from the file to do its job.

In case the processing service encounters an issue and fails, the file gets shifted to an error directory. But if everything goes smoothly and the processing is successful, the file gets moved to a done directory. This way, the system ensures that files are efficiently managed throughout their journey, from arrival to processing to completion.

Configuration :
Let’s take a closer look at how each field is set up and configured for file polling.

• Package Name : In the package name, you can provide the name of the package where your processing service is located. If this package happens to be disabled, the file polling port associated with it will also be automatically disabled.
• Alias : Choose a special Alias for the integration server that is unique to it. This Alias can be anywhere from 1 to 255 characters long.
• Description : Provide a description for this alias.
• Monitoring Directory : Provide the complete pathway of the monitoring service that the integration server will keep an eye on.
• Completion Directory (optional) : This is an optional directory. However, if you do specify a Completion directory, it will be used in the file transaction process. Otherwise, webMethods will automatically create a directory for you and same will be used to store the successfully processed files.
• Error Directory (optional) : This directory is optional, and it serves as a storage place for files when the processing service encounters errors. If you specify this directory, webMethods will use it. Otherwise, webMethods will create its own directory for this purpose.
• File Name Filter (optional) : Here you can specify regular expression that will help to pick specific files. For example, if you only want to work with .csv files, you can put in the right pattern. If you don’t put anything here, the integration server will handle all files without any specific filtering.
• File Age (optional) (seconds) : In this section, you can define the file’s age in seconds. The Integration Server calculates the file’s age by considering when it was last changed in the monitoring directory. Adjusting this age lets you ensure that the server won’t process a file until it’s fully copied into the Monitoring Directory. For example, if you have a 5 MB file that takes 5 seconds to completely load into the monitoring directory, you should specify this duration. Otherwise, the integration server might attempt to access the file before it’s fully loaded.
• Content Type (optional) : Here you can specify the content type of a particular file you intend to process through this port. If you leave it empty, the Integration Server will automatically map the MIME type based on the file extension.
• Allow Recursive Polling : In this section, you have the option to choose “Yes” if you want the integration server to monitor its subdirectories, or you can select “No” if you don’t want it to monitor subdirectories.
• Enable Clustering : In this field, you can indicate your preference by selecting “Yes” if you want the Integration Server to enable clustering in the Monitoring Directory. If you don’t want clustering, choose “No”.
• Lock File Extension (optional) : This setting decides how files with a certain extension are checked. The default value is “lock.” This means that a locking mechanism is used to prevent multiple parts of the system from working on the same file at the same time.
• Number of files to process per Interval (optional) : Here you can mention the number of files should be processed per interval using this port, default value is ‘-1’ which means the port will process all files in the monitoring directory.
• Run services as user : Select the user using which you want to run the processing service.
• Enable : Select ‘Yes’ if you want to enable the port once you save the port else you can select ‘No’ if you want to keep it disabled.
• Processing Service : Here mention the full path of the processing service.
• File Polling Interval (seconds) : Here specify the time in seconds for how often you’d like the Integration Server to perform checks on the Monitoring Directory for files.
• Log only when directory availability changes : Select ‘No’ if you want to log a message every time the monitoring directory is unavailable else select ‘Yes’.
• Directories are NFS mounted file system : This option is only for UNIX systems, the monitoring, working, completion, or error directories can be network drives attached to the local file system.If you choose ‘No’ (the default), the listener will use the Java File.renameTo() method to move files between directories.If you select ‘Yes’, the listener will first try File.renameTo(). If it fails, the listener will copy and then delete files.
• Cleanup Service (optional) : If you have a clean up service you can specify the full path here.
• Cleanup At Startup : You can select ‘Yes’ if you want to preform cleanup activity on Completion Directory and ‘No’  to perform cleanup activity on ‘Error Directory’.
• Cleanup File Age (optional) (days) : Here you can set the number of days to wait before removing processed files from your directories, the default value is 7 days.
• Cleanup Interval (optional) (hours) : Enter the number of hours for cleaning processed files, with the default value being 24 hours.
• Maximum Number of Invocation Threads : Specify the number of threads to assign to this port, ranging from 1 to 10.

Let’s do an exercise to see how it works in real time.

• Create a package with below folder structure.
  
  
  

   

• Open your integration server and navigate to the “Ports” section within the “Server” tab.

  

   

• Click the “Add Port” option, choose “webMethods/FilePolling,” and submit to configure the fields as demonstrated in the screenshot. Ensure you’ve already created a folder on your local drive to set as the monitoring directory during this configuration, for this demo we have set up our monitoring directory as “D:WebmethodsFilePollingMonitoringDir”.
  

   

• After finishing the configuration, click “Save Changes.” You might encounter an error message like “Warning: Access to port FilePollingListener:D:WebmethodsFilePollingMonitoringDir is currently restricted. You must edit Access Mode before the port will be usable.” To resolve this, manually enable it by clicking the button below, as illustrated in the screenshot.

  

   

• Now the port will be activated but you might encounter runtime errors with an “access denied” message. To address this issue, you’ll need to add your monitoring directory to the file access control list, which is found in the WmPublic package.
• Now edit “fileAccessControl.cnf”  located in “C:SoftwareAGIntegrationServerinstancesdefaultpackagesWmPublicconfig” location and append your monitoring directory path as shown in the screenshot and save it.
  

   

• To make the access control list changes work, we need to reload the WmPublic package. Go to the “Packages” tab, find the “Management” page, and click the reload button next to the WmPublic package.
  

   

• To make the file polling work we must set the “Access Mode” to “Allow.” To do this, click the “Deny +” button, and then follow the instructions below to change it to “Allow.”

  

  

   

  

   

• Once you activate the file polling port, you’ll observe that directories named “error,” “work,” and “done” have been generated within the same location as the “MonitoringDirectory”.
  

   

• Now, let’s proceed with setting up the processing service. Start by invoking the “pub.flow:getTransportInfo” service. This will provide us with essential file details that we can utilize for our file processing tasks.
  

   

• Next invoke “pub.file:getFile” step and perform below mapping and hardcode the “loadAs” to string so that the file content will be loaded as string format.
  

   

• Now invoke “pub.flow:debugLog” step and perform below mapping so that the file content will be written to server log(In practical scenarios, you can utilize this file content for various transformations as needed).
  

   

• Now, to demonstrate the process, let’s drop a demo.txt file containing the message “Hello from File Poller” into the monitoring directory. This action will trigger the logging of the file’s content in the server log, and the file itself will be subsequently moved to the “done” directory.
• Now let’s confirm the file’s processing, let’s check both the server log and the “done” directory. This will help us verify whether the file has been successfully processed or not.
• Congratulations, It worked 😀!!

  

   

  
• As of now, we’ve finished this demonstration successfully. However, for an additional test, let’s intentionally generate an exception from our processing service and observe whether the file gets moved to the “error” directory as expected.
• Now in the first step of the processing service, call the “pub.flow:throwExceptionForRetry” service. For the “message” input, set it as  “Exception for Demo” This will create a controlled exception for our demonstration purposes.
  

   

• Now let’s drop the same file into the monitoring directory and verify the execution result.
• Congratulations, we got expected result 😀!!