Data Integrations - Exporting Data

Slate has integrated successfully with a wide variety of software applications, including SIS systems such as Banner, PeopleSoft and Colleague, business intelligence solutions, and enterprise content management systems. This article reviews methods and best practices for data integration with Slate. 

Many partner institutions transfer data bidirectionally between Slate and their institutional SIS, and between Slate and other databases as needed. File transfer with SFTP or RESTful web services is employed for such integration.

Data feeds are generally limited to applicant and financial aid data. Most commonly, the data exchange is implemented as a transfer of flat files on a scheduled basis through an SFTP server, where the specifications for the data exchange can be dictated by the institution, and where the value and code translations (for country codes, major codes, term codes, etc.) happen within Slate.

The data points from Slate to the campus system typically include student biographic and demographic data, as well as key application components that are necessary outside of admissions (for example, entry term, admission plan, and admission decision). A Slate ID is also sent, along with a placeholder for the Campus or Institutional ID.

A return feed is then provided with the Slate ID and the matching or newly created Institutional ID for the record. Subsequent data feeds from Slate into the external system include that identifier for direct matching.

The article describes options for this type of data exchange and provides links to related documentation for each method. 

 Tip

For information specific to your SIS or external system, your peer institutions may be your best source of information. We recommend reaching out through the Slate Community Forum.

Batched Exports

Batched exports are built using the Query tool and involve the generation of flat files (such as fixed-width, delimited, XML, and JSON) on a scheduled frequency. These exports can be cumulative, incremental, or differential. The differential option uses Queues to track records changed since the previous export, returning the full rows each time but only for changed records. This is typically the most appropriate option for exports to a SIS.

Any code and value translations can be configured within Slate but outside of the query, so the query remains stable and immutable even when new entry terms, majors, and other code changes are introduced. This also ensures that the process on the SIS side remains stable year after year.

Exports can be scheduled to occur on a daily frequency (or more frequently throughout the day) and can also be run on demand for instances when you may need to a more immediate update to your external system (for example, after adding or changing a batch of decisions). Slate can push exports to our SFTP servers or to a remote SFTP server.

  Best Practice

For exports to a SIS, we generally advocate for:

  1. The use of batched exports
  2. A scheduled frequency on a nightly basis, overnight (2:00am-4:00am)
  3. Exporting to our SFTP server, where your institution can then poll the /outgoing/ directory periodically to pull down and load any files

There are several benefits to the batched export option:

  • We can ensure that our SFTP servers remain online and operational during the export.
  • Inherent logging of all exports occurs just by archiving the exported files. This provides a great resource for troubleshooting if necessary.
  • Exports are scheduled to occur within a specified delivery window, but the exact execution time within that window can be changed, based upon server maintenance activity and load.
  • We can provide automated email notifications upon successful, late, or failed generation and delivery of scheduled exports.
  • With a delimited file structure, you can add new fields to the export without negatively impacting anything on the SIS end, depending on the specific capabilities of your external system. 

Implementing Slate requires the creation and configuration of many custom fields where data is stored. Building an export before these configurations are complete may add complexity to a project. The export then requires updates with every change. Data exports should be configured after going live with some or all aspects of Slate.

  Can we use direct SQL queries to feed data from Slate to an external system?

Direct SQL Server connections to a Slate database are supported. These connections are provided on an "as is" basis and should be used for ad-hoc reporting only. Direct SQL access is not appropriate for automated, scheduled, or batch jobs. 

Since SQL queries built and run outside of Slate cannot be tracked or managed, minor changes to database schemas may potentially break some of those scripts. 

Direct SQL access for an institution may be discontinued at any time at the sole discretion of Technolutions if it impacts database or server performance. For example, disruptive activities would include copying all rows in a table or querying indiscriminately or too frequently.  Connections are automatically disconnected after 60 minutes of inactivity, and connections in excess of 3 concurrent connections may be terminated.

For additional information, refer to the article on Direct SQL Access.

 Tip

For detailed instructions on building a data export, refer to the Data Export article.

Scheduling Exports

After you've built and tested your export query, configure the Schedule Export settings. For additional information, refer to the Scheduling Exports article.

There are two ways to monitor the status of scheduled exports:

  • Setting a notification in the Schedule Export settings of your query
  • Reviewing the Job Activity Monitor tool, located on the Database page under Auditing. The Job Activity Monitor displays the history and results of all scheduled jobs. 

Scheduling Exports to a Remote SFTP Server

Slate supports sending a scheduled export to a remote SFTP server. However, this practice is generally discouraged, since the export process will fail if the remote SFTP server becomes unavailable for any reason, such as with a network connectivity or maintenance issue.

For configuration instructions, refer to the Sending scheduled exports to a remote SFTP server article.

It is generally recommended that all exports be delivered to the Technolutions SFTP servers, where we can provide high availability and from which files can be downloaded at any time.

Web Services

The same query built for a batched export can also be made available as a web service at any time by enabling web services on the query. For more detailed guidelines, please refer to our Web Services article.

Together with the query Queue settings, you can poll the web service on a frequency that you define, and can pull down only the new or changed records each time. If a specific XML structure is desired, some modifications to the query must be applied.  

  Best Practice

Typically, the data to be extracted is not as volatile, and the business need to see that data instantaneously reproduced in a SIS is not so great that frequent web service polling is necessary, so a batched export usually meets the business process requirements. Web services are relatively more difficult to troubleshoot should the need arise, since the request and response are transient. 

Document Exports

Slate supports document exports to an external document management system (for example, OnBase, ImageNow, or Docfinity). Exports can be created in the Query tool using Document Export resources available in the Slate Template Library. Documents can be packaged in one of three ways:

  • In a zip archive with an index file
  • In a zip archive with the index keys included in the file name
  • With every document included in a single combined PDF file

For detailed instructions on setting up document exports, refer to the Document Export Documentation article.

  1. Select Database on the top navigation bar and select Slate Template Library.
  2. Add Document Export (Document Export (Application)).

    This template provides a working sample that can be modified using the Query tool to add additional fields and arrange the fields as needed to meet the specifications of your document management system.

SFTP Access

Slate exports files, under the default settings, to the Technolutions SFTP servers. There are two ways to access these files:

  • The SFTP Explorer tool: Users with SFTP access can view and download files in their SFTP folders. To access this tool, go to the Database page, and select SFTP Explorer in the Import section. 
  • Through a client that supports the SFTP protocol: Set up a user account in Slate with the necessary credentials for the SFTP client to connect to the Slate SFTP servers.

For detailed instructions on setting up a user account for SFTP access, refer to the SFTP Access article.

Integration Goals

Ultimately, the goal is to design a process that is reliable, stable, supportable, and sustainable for your institution.

  • Reliability: The integration process can be more reliable when more of it lives within the Slate infrastructure, so an export to our SFTP servers is typically preferred.
  • Stability: The integration process should run without constant human intervention. In other words, you should be able to “set it and forget it.” It should not need to change with any frequency. This means that code and value translations should not live within the query itself.
  • Supportability and sustainability: The integration process should minimize the use of custom SQL where possible and should host the value translation inside Slate, but outside of the query, enabling the people closest to the process (such as admissions staff) to manage the periodic changes that may be necessary because of new terms, programs, majors, etc.
Was this article helpful?
5 out of 6 found this helpful