oraclefrontovik

Sharing what I learn

Introduction

Introduced in Oracle 11g Release 2, the File Watcher enables jobs to be triggered when a file arrives in an Operating System Folder.

In this article I am going to set up a new file watcher on my Windows PC. The example inserts the contents of the newly arrived file into a database table.  The information shown here is distilled from the Oracle documentation

Before getting into the detail, here is a quick run down of the key components and their versions that was used to create the example.

  1. Microsoft Windows XP with Service Pack 3 running via Oracle Virtual Box
  2. Oracle Database 11g Enterprise Edition Release 11.2.0.1.0
  3. Oracle SQL Developer 3.2.10.09
  4. The user running all the code is logged in with the DBA role

Step 1 Alter the File Watcher Interval (Optional)

File watchers check for the arrival of files every ten minutes by default. If you want to change the interval, connect as sys user and run the set_attribute procedure of dbms_scheduler.

The following example changes the interval to one minute.

BEGIN

   dbms_scheduler.set_attribute
     ('file_watcher_schedule',
      'repeat_interval',
      'freq=minutely; interval=1'
   );

END;
/

Step 2 Create a credential

In order for the File Watcher to be able to access the file(s) on Windows, a  Scheduler credential object is required. The following code creates a credential called “watch_credential”

BEGIN

   dbms_scheduler.create_credential
   (
      credential_name => 'watch_credential',
      username        => 'your operating system username',
      password        => 'your operating system password',
   );

END;
/

Step 3 File Location Details

The call to create_file_watcher (unsurprisingly) creates the file watcher object and tells it where to “watch” for incoming files along with the name of the file that you are interested in.

In the following example I want the File Watcher to watch for files that appear in the Operating System directory C:etl_dir and as the names of the files could be all different but will have the .txt suffix I have set the file name parameter accordingly.

BEGIN

   dbms_scheduler.create_file_watcher
   (
      file_watcher_name => 'the_file_watcher',
      directory_path    => 'C:etl_dir',
      file_name         => '*.txt',
      credential_name   => 'watch_credential',
      destination       => NULL,
      enabled           => FALSE
   );

END;
/

Step 4 Specify the program unit that will be executed when the file watcher runs

In this step I have specified that the stored procedure that will be executed by the File Watcher, when the file arrives. The stored procedure, sp_load_customer_files, doesn’t yet exist and will be created in Step 6.

BEGIN

   dbms_scheduler.create_program
   (
      program_name        => 'file_watcher_prog',
      program_type        => 'stored_procedure',
      program_action      => 'sp_load_customer_files',
      number_of_arguments => 1,
      enabled             => FALSE
   );
END;
/

Step 5 Defining metadata

In order for the new stored procedure, sp_load_customer_files, to access attributes of event that started the File Watcher, a call to dbms_scheduler.define_metadata_argument is required.

For more information about this program unit please refer to the documentation.

BEGIN

   dbms_scheduler.define_metadata_argument
   (
      program_name       => 'file_watcher_prog',
      metadata_attribute => 'event_message',
      argument_position  => 1
   );

END;
/

Step 6 Creating the supporting objects

This step creates a table where the contents of the files will be inserted into, along with the file name. To keep the example concise, no primary keys, indexes etc have been defined.

CREATE TABLE files_from_customers(file_name     VARCHAR2(100),
                                  file_contents CLOB);

The stored procedure that was first referenced in step 4 is now created. This procedure uses some attributes from the filewatcher object to obtain the file name. It then uses the dbms_lob packages to load the data from the file into the table.

CREATE OR REPLACE PROCEDURE sp_load_customer_files
(pt_payload IN sys.scheduler_filewatcher_result)
IS

 lc_clob           CLOB;
 lt_bfile          BFILE;
 li_warning        INTEGER;
 li_dest_offset    INTEGER := 1;
 li_src_offset     INTEGER := 1;
 li_lang_context   INTEGER := 0;

BEGIN
   INSERT INTO files_from_customers (file_name,
                                     file_contents)
   VALUES(
   pt_payload.directory_path || '' || pt_payload.actual_file_name,
   empty_clob())
   RETURNING file_contents INTO lc_clob;
   lt_bfile := BFILENAME(directory => 'ETL_DIR',
                         filename  => pt_payload.actual_file_name);

   dbms_lob.fileopen
   (
      file_loc => lt_bfile
   );

   dbms_lob.loadclobfromfile
   (
      dest_lob     => lc_clob,
      src_bfile    => lt_bfile,
      amount       => dbms_lob.getlength(file_loc => lt_bfile),
      dest_offset  => li_dest_offset,
      src_offset   => li_src_offset,
      bfile_csid   => NLS_CHARSET_ID('UTF8'),
      lang_context => li_lang_context,
      warning      =>; li_warning
   );

  dbms_lob.fileclose
  (
     file_loc => lt_bfile
  );

END sp_load_customer_files
/

Step 7: Creating a job

Create an Event-Based Job That References the File Watcher.

BEGIN

   dbms_scheduler.create_job
   (
      job_name        => 'file_watcher_job',
      program_name    => 'file_watcher_prog',
      event_condition => NULL,
      queue_spec      => 'the_file_watcher',
      auto_drop       => FALSE,
      enabled         => FALSE
    );

END;
/

Step 8: Enable All the objects

Enable all the objects that you have created by running:

BEGIN

   dbms_scheduler.enable
   (
      'the_file_watcher, file_watcher_prog, file_watcher_job'
   );

END;
/

Step 9: Seeing the results

Before a file arrives, query the table to show it is empty:

A file arrives into the directory that the File Watcher is monitoring.

When the File Watcher runs (as specified in Step 1) the contents of the new file are inserted into the table:

Step 10 Nothing has happened! (Optional)

So you have followed all the steps shown, double checked the code and nothing has happened. The file hasn’t loaded and your table is still empty.  You have my sympathy! As with anything with many moving parts something is bound not to work.

One tool I found invaluable in debugging these issues is looking at the run log for the job (This is the job created in step 7 and in is called “FILE_WATCHER_JOB” )

SQL Developer performs all the heavy listing when it comes to getting to this information.

From the SQL Developer Object Navigator select the Scheduler folder

Expand the folder and then the Jobs folder and you should see the job you created for the File Watcher, in this case it is called “FILE_WATCHER_JOB” (The job is created in Step 7)

Selecting the appropriate job will then bring up a list of tabs. Select the Run Log

From here you can see lots of useful information that should assist in your debugging.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: