SFTP (Inbound/Outbound) Configurations

Introduction

This article is intended to serve as a guide for configuring Inbound and Outbound Subscriptions. This provides Enterprise+ Admin users with a clean, secure way to migrate files of data from one system to another. 

• The "Inbound Subscriptions" are automatic scheduled subscriptions for importing csv file via SFTP channel. The import process will follow the same CSV import / template existing today.
  • CSV Import Fields Guide

• The "Outbound Subscriptions" are automatic scheduled subscriptions for receiving reports via a specific channel. The reports are based on a SQL Query or a pre-existing custom report.

There are 2 channels available for receiving these subscriptions:

1. E-mail.
2. SFTP.

SFTP Server Configuration

To use the Inbound/Outbound SFTP subscription, it is necessary to register/setup a SFTP server that Enterprise+ will use to obtain the CSV files.

• To register a new SFTP server, you must go to the Administration > Integrations > SFTP Servers.

• On this page by clicking on "Add New SFTP Server" link, a popup will be shown to configure a new SFTP server.

• After filling in all the required information, the user can validate the access to the SFTP server by clicking on the "Test SFTP Settings" button. This test creates a dummy file and attempts to transfer it to the SFTP server, validating the connection.

• If the test is successful, the system will display the following message “The test file was transferred successfully” and a CSV file was transferred to the respective server.

• If there is any problem when performing the test, the system will display an error message on the screen informing the server's response.

• After completing the test, save the SFTP server information and it will be available to configure subscriptions.

Inbound Subscriptions

To create or modify inbound subscription, access the Administration menu > Integrations > Inbound Subscription. This page will list all the subscription available and the logs from the executions.

• To create a new subscription, click on the "Add New Subscription" link.

Here is an explanation for each field:

• Name: Name for the created subscription.
• Pattern Name: This is the name pattern that the system will look for in the SFTP directory. If you register the term "LEARNERS_", the system will search for all files that start with this term. Here are some examples of files that would be returned:
  • LEARNERS_2023.csv
  • LEARNERS_FIRST_EXAMPLE.csv
  • LEARNERS_20230801_120000.csv
• Periodicity: It is the frequency that these reports will be sent to the configured destination.
• Type: This is the types of import supported by the system. Currently, this list is the same as the AdmBulkImport page:
  • Users
  • Inventory
  • Learners
  • Courses
  • Sections
  • Teachers
  • Enrollments
  • Meeting_times
  • External_courses
  • Affiliations
  • Learning_path_assignments

• Recipients: Email(s) that will receive the notification about the execution of the subscription or the file for the generated report if the selected delivery type was “Email. (if there is more than one email, separate them with "," or ";").
• SFTP Server: Select the SFTP server that will receive the file.

After the subscription is registered, the system will run these subscriptions based on the configured frequency. Both jobs need to be active for the process to run as expected.

• There are two jobs responsible for processing and sending:
  • "inboundSubscription.Create”: Responsible for validating the schedule, searching files on sftp, and register this file on Enterprise+.

  • "InboundSubscription.Process”: Responsible for importing the file and notifying users via email about the processing results.

Processing Logs

It is possible to check the execution via logs within the system and re-trigger a specific Subscription if necessary.

• To access the log, simply click on the button marked on the Subscription screen.

Subscription execution logs:

This screen will display the logs of the last 30 days.

• To re-process the file, you need to click on the following button:

Inbound Subscription Jobs Configuration

To configure the jobs related to process and send the Subscriptions, go to the menu Administration -> Edit Jobs. There are 2 Jobs that must be active.

1. The recommended scheduling for the "InboundSubscription.Create" job is "EVERY_DAY" and to run at a time outside the Enterprise+ usage hours to avoid concurrency.

2. For the "InboundSubscription.Process" job, the recommended frequency is "EVERY_DAY" but running a few hours after the "InboundSubscription.Create".

• At the end of the processing, the file will be moved within the SFTP to the folder “/PROCESSED/SUCCESS” if it has been processed successfully. If an error occurs, the file will be moved within the SFTP to the folder “/PROCESSED/ERROR”.

Outbound Subscriptions

To create or modify a subscription, you need to access the Reports menu -> Administration > Integrations > Outbound Subscription. This page will list all the subscription available and the logs from the executions.

• To create a new subscription, click on the "Add New Subscription" link.

Here is an explanation for each field:

• Name: A name or nickname for the created subscription.
• Periodicity: Frequency that these reports will be sent to the configured destination.
• Type: Select how the file will be made available (Email or SFTP).
• Recipients: Email(s) that will receive the notification about the execution of the subscription or the file for the generated report if the selected delivery type was “Email. (if there is more than one email, separate them with "," or ";").
• SFTP Server: Select the SFTP server that will receive the file.
• Report: You can select a pre-registered report to be sent.
• SQL Query: It is a text field that should receive an SQL query that will be used to obtain the report that will be sent.

The SFTP Server option only appears if you select SFTP in the Type field.

• You can select the Report or fill in the query.
• After the subscription is registered, the system will run these subscriptions based on the configured frequency.

There are 2 jobs responsible for processing and sending:

1. “Subscriptions.Create”: Responsible for validating the schedule, running the report, and saving the file inside the defined storage.
2. “Subscriptions.Send”: Responsible for sending the files to the configured destination.

Both jobs need to be active for the process to run as expected.

Processing Logs

It is possible to check the execution via logs within the system and re-trigger a specific Subscription if necessary.

• To access the log, simply click on the button marked on the Subscription screen.

Subscription execution logs:

• This screen will display the logs of the last 7 days. You can download the sent file (through the browser) and also request the re-sending of a specific file.
  • To request that the file be resent, click on the following button:

• • To download the file in the browser, click on the following button:

• • In this screen, the "Status" field will show whether the entire processing was successful or if there was any problem when processing/sending the file.

Subscription Jobs Configuration

To configure the jobs related to process and send the Subscriptions, go to the menu Administration -> Edit Jobs. There are 2 Jobs that must be active:

1. The recommended scheduling for the "Subscriptions.Create" job is "EVERY_DAY" and to run at a time outside the Enterprise+ usage hours to avoid concurrency.

2. For the "Subscriptions.Send" job, the recommended frequency is "EVERY_DAY" but running a few hours after the "Subscriptions.Create".

Please note that there are two jobs to execute the inbound subscription that guarantee resilience and retrial processes. We don't download and process at the same time, so if the user neglects to configure the job properly or forgets to enable the import process, it will not execute correctly.

• InboundSubscription.Create: Download all the files from all SFTP Server
• InboundSubscription.Process: Import the downloaded files

Steps for creating a periodic email or SFTP report (through Subscriptions):

• Activate the Jobs that help run email reports
  • Go to “Edit Jobs”
  • “Subscriptions.Create” – this job creates the subscriptions that need to be sent out on a certain frequency
  • “Subscriptions.Send” – this job sends out the subscriptions that need to be sent out on a certain frequency
  • Please note that increasing the frequency of the above two jobs may affect the latency of your instance.
  • Please note that, instead of waiting on the frequency of these jobs, an admin user may also press the play button to the left of them to immediately run them. You would always want to run the “Subscriptions.Create” job prior to the “Subscriptions.Send” job. Doing so will take you to the Hangfire dashboard to see a confirmation of the job.
• Build an email or SFTP report through Outbound Subscriptions
  • Go to Administration --> Integrations --> Outbound Subscriptions
  • Click “Add new subscription”
    • Name it whatever you would like.
    • Choose the periodicity.
    • Select Email or SFTP
    • Add relevant recipient(s)
    • If SFTP, you will need a SFTP setup in your Genius instance.
    • Select the report you wish to send OR paste in a SQL query.
      • Please note that any custom report you create in your Enterprise+ instance should be available in the list of available reports.