Importing Data Overview

This article describes the methods that Slate provides for importing data. If your data import is part of an integration with an external system, we strongly encourage you to read the Data and Document Management Overview article first, which provides a birds-eye view of data integrations and outlines our best practices for a successful integration.

Upload Dataset is Slate's import tool. Data comes in via batched files on an SFTP server or via web service calls. When the data arrives, a predefined source format handles all value and code translations. This helps ensure that year-over-year changes made to accommodate new fields or values are straightforward and can be handled by operations staff. 

  What file layouts can Slate consume?

Slate can consume Excel spreadsheets, delimited text files, fixed width files, XML, and JSON. We typically recommend using delimited files with column headers because you can add or remove columns at any time without negatively impacting the import process within Slate. This then allows for asynchronous changes to the data feed's specifications.

SFTP Imports

Most frequently, institutions deliver import files to an /incoming/ directory on the Technolutions SFTP servers, which are pulled at least once every 15 minutes. Files matching a specified filename mask are then loaded. The files are routed into our Upload Dataset interface.

It is also possible to pull a remote SFTP server, but only to the point of SFTP server availability. Since we can ensure that our servers remain highly available, the process is usually most reliable when using our infrastructure.

  Learn more:

The source format documentation describes how to set up batched SFTP imports.

Web Services

Pulling from a Remote Endpoint Into Slate's Upload Dataset

This option allows Slate to pull external web services for new data and then process this data through the Upload Dataset interface, just as if the files were transferred via SFTP. These could include XML posts but can also include delimited data.

Since the data updates are processed through our Upload Dataset mechanism, changes to records can be queued, batched, and run in the most efficient manner possible, which minimizes or eliminates any potential for observable record locking.

Pushing Data into Upload Dataset through a Web Service Endpoint

This option uses web services to post files into Slate that are then processed by the Upload Dataset mechanism, just as if the files were transferred via SFTP. This is also like the process of pulling from a remote endpoint.

  Learn more:

Explore related documentation on Web Services.

Document Imports

We recommend that files be sent using the industry-standard Document Import Processor (DIP) approach, where a zip archive is generated containing PDFs or TIFFs of the documents to be imported, along with an index file containing the filename of each document as well as any associated metadata parameters (such as EMPLID and document type). Slate can then extract the documents and index files to import the documents into the appropriate student records. 

Best Practice

We recommend delivering import documents in a zip file using SFTP since SFTP is much more efficient with the transmission of a single file (such as a zip archive) than with thousands of individual files. While documents could be imported using Web Services, we advise that imports are handled using SFTP, since a zip archive containing potentially numerous PDFs could be quite large.

PDFs are preferred instead of TIFFs, since a digital PDF of non-scanned data is a fraction of the size of a TIFF file. A TIFF file is a rasterized/bitmapped image without digital text content, and thus cannot be enlarged beyond the original resolution without a loss of fidelity.

Imports made via the Upload Dataset tool and material uploader have a 256 MB file size limit.  In addition, there is a 15-minute processing time limit for documents to be uploaded.


Learn more:

Explore related documentation on Document Imports.

Was this article helpful?
11 out of 21 found this helpful