A source format is a collection of standardized settings and criteria for a dataset uploaded frequently from the same source. The source format defines the destination fields and values in Slate and allows users to prescribe consistent import behavior, like:
- New record creation,
- Updates to existing applicant records
- De-duplication within a source file
Source formats automate data import by configuring standard Slate SFTP options.
Best Practice: Using the test environment
Test data imports in the test environment before importing data in the production environment. Set up the import in the production environment, refresh your test environment, and activate the import in the test environment to make sure results are as expected before activating the imort in production.
Five steps to creating a new source format:
- Create a new source format, either from scratch or from a previous upload.
- Configure the General tab, which determines basic settings.
- Configure the Format Definition tab, which instructs Slate how to interpret incoming data.
- (Optional) Configure the Import Automation tab.
- Configure remap settings by uploading a sample file.
Step one: Creating the source format
You have two options when creating a new source format:
- Generate the source format using settings from a previous upload
- Create the source format from scratch
Generating a source format from a previous upload
Creating a new source format from scratch
To create the new source format from scratch:
- From the main navigation, select Database.
- Under Import, select Source Formats.
- Click New Source Format. The New Source Format popup appears.
Step two: Configuring the General tab
On the General Tab, fill in the information that describes your source format. Each field is explained below.
|Status||Select Active to make the source format available for use in Upload Dataset.|
|Name||Give the source format a descriptive internal name.|
Enter the imported file format for display purposes only. For example, Excel, CSV, or Fixed Width.
File formatting is defined on the Format Definition tab in the XML field.
|Remap As Of Date||
For now, enter the current date.
This date can be changed to limit the data values available to the source format for remapping.
If, for example, a file is imported with no header row, and first row data become the headers, enter a remap-as-of date after this erroneous file was imported.
Leave this as Inactive until you have configured remap settings (step five).
The remap-as-of date must be populated to activate this setting.
|Scope||Leave this setting set to Person/Dataset Record unless the source format imports Slate users.|
|Dataset||Select a dataset into which the source format should import data.|
If a source format is Unsafe, it can update person-scoped data on student records with an active application.
Select Unsafe only if you trust the source to contain accurate, up-to-date information. An import into Slate from your student information system is an example of a source format that you might set as Unsafe.
Purchased lists of student information are examples of imports that should have Safe source formats.
Determines whether the import will create source interactions on the affected record. You'll usually want to Create source interactions to display activity history on a record's timeline.
Daily or frequent data updates/imports can contribute to the creation of many rows of data interactions on records. If the source format will be used for a daily feed from an SIS or other application, select Hide source interactions.
|Disable Update Queue||
This tells Slate how to handle records in the incoming dataset that do not match records within Slate.
|Dedupe Records||By default, Slate will create a new record for every row when Allow record creation is selected and an existing match cannot be found. By selecting Dedupe records, Slate will evaluate the source file and dedupe records in the source based on an exact match of First Name, Last Name, Email, and Birthdate.|
|Notification and Notification Email||To receive email notification if an imported file associated with the Source Format fails, select Failures only and enter the email address(es) for notification. Select Successes and Failures to be notified of both successful and failed imported files for the Source Format.|
|Read Permission and Upload/Build Permission||Permission to the source format can be limited to specific permissions.|
Step three: Configuring the Format Definition tab
The settings on the Format Definition tab provide Slate with specific instructions for interpreting the incoming data.
Use XPath syntax to tell Slate how to parse the data source. For example, you can define whether the file is delimited, if there is a header row, and what file format is used.
See the Common XML Examples article for specific examples that can be copied and pasted into this field.
|SQL||For most data imports, the SQL text area should be left blank. Consider creating a community forum post to determine if your case requires custom SQL.|
|Source Index||Leave this set to the default Use modern cloud storage unless you are working with an existing source format that requires the legacy source.index table.|
|PGP Private Key||Paste the private key from the PGP software.|
|PGP Password||Each PGP private key is typically encrypted with a password.|
Step four: (Optional) Configuring the Import Automation tab
The settings on the Import Automation tab are optional and will not apply to every source format.
Include the file name pattern that Slate should look for to know that the file should be imported using this Source Format.
Best Practice: Append file names with the date
Slate retains files for a period of time in a Processed directory within the /incoming folder. If two file names are identical, the second file load deletes the first from the Processed directory. Appending file names with the date prevents this.
Example import from FTP mask field: SIS_to_Slate_*.txt
Slate automatically imports any file in the /incoming directory that contain that file name pattern, including: SIS_to_Slate_09032013.txt
|Import Remote Server||
Leave this field blank unless the data for this source format will be delivered via an API call from Slate or an external SFTP server.
|HTTP Headers||Add a list of headers in the form, for example: <t name="HEADER">VALUE</t>|
(Optional, if import path/mask exists) Specify an import order.
For example, if you have an application file and a supplement file delivered to the SFTP directory together, and you want the application file to be processed prior to the supplement, enter 1 in the application's source format and 2 in the supplement's.
|Web Callback URL||
Enter a webhook URL to send a Web Service HTTP POST to an external system. This POST includes a JSON Payload and confirms that the import was successful in Slate. The external system receiving this confirmation can use it to mark its records as exported and updated in Slate.
Below is an example of a POST. For any records not updated or exported, a process can be implemented within your institution that prompts a member of your team to review and determine the root cause.
When you are satisfied with the source format settings, click Save. You are redirected to the source format's overview page.
Step five: Defining remap settings
With a source format created, upload a sample file to define remap settings. Ensure a remap-as-of date has been set.
The data import and merge processes in Slate cannot be undone. Exercise extreme caution when uploading datasets of any kind, particularly for predefined formats like score reports. Before uploading, always confirm the dataset you're uploading is consistent with the format type you have selected and with past datasets of this type.
- On the overview page for source format you've just created, click Upload File(s).
- Select a sample file to upload. The sample file should exactly match the format that you intend to send Slate. Slate processes the file. When the upload is complete, the status on the Upload Dataset page changes from Pending to Completed or Awaiting Activation.
- Click the name of the source format. You are redirected to the remap stage, where source fields can be mapped to destination fields within Slate. See the article Upload Dataset Stages for more information on this step.
- When you've completed remapping, click the name of the source format in the breadcrumb menu to return to the source format overview page.
- Click Edit. A pop-up appears.
- In the General tab, set Remap Active to Active.
- Click Save.
- Slate processes any files that were uploaded on or after the remap-as-of date using the remap settings that you defined previously.
Any additional files that are uploaded to Slate using the new Source Format will also use the Source Format and remap settings that you configured using these steps.
Awaiting import through Source Format remap
If you've received this error, be sure that your source format has the appropriate XML setting on the Format Definition tab. Use one of the examples in the Format Definition XML article to build it out.