Importing Data Overview

The article describes the methods that Slate provides for importing data. If your data import is part of an integration with an external system, we strongly encourage you to read the Data and Document Management Overview article first, which provides a birds-eye view of data integrations and outlines our best practices for a successful integration.

Upload Dataset is Slate's import tool. Data comes in via batched files on an SFTP server or via web service calls. When the data arrives, a predefined source format handles all value and code translations. This helps ensure that year-over-year changes made to accommodate new fields or values are straightforward and can be handled by operations staff. 

  What file layouts can Slate consume?

Slate can consume Excel spreadsheets, delimited text files, fixed width files, XML, and JSON. We typically recommend using delimited files with column headers because you can add or remove columns at any time without negatively impacting the import process within Slate. This then allows for asynchronous changes to the data feed's specifications.

SFTP Imports

Most frequently, institutions deliver import files to an /incoming/ directory on the Technolutions SFTP servers, which are pulled at least once every 15 minutes. Files matching a specified filename mask are then loaded. The files are routed into our Upload Dataset interface.

It is also possible to pull a remote SFTP server, but only to the point of SFTP server availability. Since we can ensure that our servers remain highly available, the process is usually most reliable when using our infrastructure.

  Learn more:

The source format documentation describes how to set up batched SFTP imports.

Web Services

Pulling from a Remote Endpoint Into Slate's Upload Dataset

This option allows Slate to pull external web services for new data and then process this data through the Upload Data interface, just as if the files were transferred via SFTP. These could include XML posts but can also include delimited data.

Since the data updates are processed through our Upload Dataset mechanism, changes to records can be queued, batched, and run in the most efficient manner possible, which minimizes or eliminates any potential for observable record locking.

Pushing Data into Upload Dataset through a Web Service Endpoint

This option uses web services to post files into Slate that are then processed by the Upload Dataset mechanism, just as if the files were transferred via SFTP. This is also like the process of pulling from a remote endpoint.

Establishing Source Import Callbacks using a WebHook URL

  Summit 2022 Feature

This option lets Slate send a Web Service HTTP POST to an external system. This POST includes a JSON Payload and confirms that the import was successful in Slate. The external system receiving this confirmation can use it to mark its records as exported and updated in Slate.

To add a web callback URL to a source format:

  1. From the main navigation, select Database.
  2. From the section Import and Data Processing, select Sources / Upload Dataset.
  3. On the right, click Source Formats.
  4. Select the source format to which you'd like to add a WebHook URL.
  5. On the right, click Edit.
  1. Select the tab Import Automation. 
  2. Paste your Web Callback URL into the Web Callback URL field.
  3. Click Save.

Paste_Web_Callback_URL.png

Below is an example of a POST. For any records not updated or exported, a process can be implemented within your institution that prompts a member of your team to review and determine the root cause.

{
"status": "success",
"source": "d13a8c60-d6da-2844-4704-0d3d78eed25f",
"summary": "Common App Prospects",
"database": "XYZ",
"started" : "2022-06-28T06:12:22.345Z",
"stopped" : "2022-06-28T06:12:23.577Z"
}

 

  Learn more:

Explore related documentation on Web Services.

Document Imports

We recommend that files be sent using the industry-standard Document Import Processor (DIP) approach, where a zip archive is generated containing PDFs or TIFFs of the documents to be imported, along with an index file containing the filename of each document as well as any associated metadata parameters (such as EMPLID and document type). Slate can then extract the documents and index files to import the documents into the appropriate student records. 

  Best Practice

We recommend delivering import documents in a zip file using SFTP since SFTP is much more efficient with the transmission of a single file (such as a zip archive) than with thousands of individual files. While documents could be imported using Web Services, we advise that imports are handled using SFTP, since a zip archive containing potentially numerous PDFs could be quite large.

We also prefer PDFs to TIFFs, since a digital PDF of non-scanned data is a fraction of the size of a TIFF file. A TIFF file is a rasterized/bitmapped image without digital text content, and thus cannot be enlarged beyond the original resolution without a loss of fidelity.

  Learn more:

Explore related documentation on Document Imports.

Was this article helpful?
11 out of 20 found this helpful