Importing Data with Web Services

There are two methods for importing data into Slate via a Web Service, depending on how data is made available by the third-party source or middleware product.

  1. Push data into Slate's web service endpoint
  2. Pull data from the third party/middleware's web service endpoint into Slate

Slate has no preference on whether you choose to push or pull data and you may make that choice depending on the tools offered by the third-party data source or middleware product. Regardless of your decision, you will need to construct a source format to consume the data into Slate. The initial set-up and mapping of data points are the same no matter the import mechanism (e.g., web service push, pull, or SFTP file transfer). For more information on the initial configuration of a source format, please review our Source Format Library and Creating a New Custom Source Format documentation.

IMPORTANT: Slate's import engine, Upload Dataset, is designed to operate most efficiently as a batch process. If your third-party data source provides individual row updates, you should investigate an option to batch records together which may require a middleware solution.

Since the tool is designed to be a batch processor, the rate limit is 50 posts in a 5 minute window.

Once you have created and tested your source format to process the data from the web service, you can configure the format to automatically push or pull data as desired using the instructions below.

Note that most uploads are processed within 15 minutes, so while you should see the source within Upload Dataset immediately, you may not see the imported data for a few minutes.

Pull data from a web service into Slate

  1. Under Database, search for Source Formats and open your web service source format.
  2. Edit the format and navigate to the Import Automation tab.
  3. Enter the URL of the web service to be called into the Import Remote Server setting.
    • The third-party source will provide the URL to access the web service.
    • There are, usually, basic authentication credentials to access the remote server which must be included in the URL by placing the username, colon, password, and an @ symbol before the URL (e.g., https://username:password@studentdata.vendor.com).
    • Optionally, the third-party source may require you to send an authorization or other headers to access your data. See the next step for additional details.
  4. If required, use the HTTP Headers setting to add authorization and other required headers as defined by the third party.
  5. Optional: by default, Slate will pull data from the web service every 24 hours; if your use case requires more frequent pulls, you can use the Import Frequency (mins) setting to increase the frequency by specifying the desired number of minutes between pulls. NOTE: As Upload Dataset operates most efficiently as a batch import, the minimum interval between pulls is 60 minutes; Slate will pull data every 60 minutes for any value less than that.
  6. Click Save.

Push data from a web service into Slate

To push data into Slate, the third-party web service can post the data to a special web service endpoint that is unique to the source format that will process the data. There are two options depending on whether you wish to use Basic Authentication (standard) or Certificate Authentication. In either case, you'll need a Security Administrator to configure a Service Account user account for authentication. In addition to the routine steps to configure a User Account, the Security Administrator should select a User Type of Service Account and either add a Password or Client Certificate, depending on the desired authentication method.

Once the Service Account user has been configured, you may securely share the username, authentication details, and URL to the web service source format with the third party. Use the following steps to find the URL.

  1. Under Database, search for Source Formats and open your web service source format.
  2. In the overview of the format, click the View link next to Web Services.
  3. For Basic Authentication, use the URL found on the Standard tab. When making the POST request to this URL, the third party must include an Authorization header which is the Base64 encoding of "username:password".
  4. For Certificate Authentication, use the URL found on the Certificate-Based Authentication tab. When making the POST request to this URL, the third party must include an Authorization header which is the Base64 encoding of "username" as well as an X-Client-Certificate header with the certificate.

NOTE: Source formats employ rate limiting to ensure the overall stability and security of your Slate database. As noted above, Upload Dataset operates most efficiently by processing batches of updates rather than individual record updates. To avoid exceeding the rate limit, we recommend that the third-party batch records together or employ a middleware tool to create these batches.

Was this article helpful?
2 out of 19 found this helpful