- Document Importer

Document Importer

Introduction

There are times when multiple documents need to be loaded into iPassport and the New Document feature only allows loading one at a time. To make this process more efficient, the Import Documents feature allows loading a number of documents in one operation. Furthermore, with the document importer it is possible to bring in already authorised documents which already follow a review process and have future review dates set. The ‘New Document’ tool only allows creating single documents in draft state.
Some manual operations are still required to ensure the documents go to the right place, follow their intended lifecycle within iPassport and keep the relevant users informed.

The importer tool can be found in the ‘Controlled Documents’ area by navigating to ‘Laboratory Records > Controlled Documents > Import Documents’:

Records uploaded to the staging area of importer tools will not be added to the account until the final import operation is completed. They will remain in a ‘reception’ area until all parameters are validated and one of the import options is clicked.

System Preferences

This feature can be switched on or off for the entire account. To make it available, a global setting must be enabled by a user with Administration Editor privileges:

1. Go to Administration > Settings > System Preferences
2. Click the row for Miscellaneous Settings to open that page
3. Tick the checkbox by Enable Document Importer
4. Scroll down and click Save

Once this is enabled, only users with appropriate permissions will be able to access the feature.

Permissions

To access the Import Documents section, the user requires the same permissions as those for creating a New Document (e.g., SOPs:Create SOPs).
To import authorised documents, permissions to authorise documents are additionally required (e.g., SOPs:Authorise SOPs).

• If a user doesn’t have any ‘create’ permissions, they won’t even see the button, Import Documents.
• If they have ‘create’ permissions but not, ‘authorise’ permissions, they will only be able to import documents in draft state.
• If they have these permissions for only certain types of documents (e.g., can only create and authorise documents of the type Policy), the button will be visible but it will only be possible to import documents of the allowed type(s).
• Furthermore, documents can only be imported to the OUs where the user has the appropriate permissions.

Features and Considerations

A new framework has been developed for this tool and other importer tools in the system. The aim is to make it easier for users to quickly populate their iPassport accounts with information migrated from other systems.

• This importer tool accepts spreadsheets in Excel™ format (.xlsx) as well as in CSV format. There is no need to convert ‘.xlsx’ files to CSV here.
• The importer progressively validates the information, so large numbers of records can be processed.
• The staging area provides a much clearer and easier environment to edit the metadata and fix problems where required.
• When a record is marked as duplicate (within the staging area), the status will not be automatically cleared if the original record is deleted or uploaded. If the duplicate record is edited, after the original one is removed, the system will revalidate it and clear the ‘duplicate’ flag.

The data can be uploaded to a staging area in three steps and can then be checked and adjusted before importing it to the records within the account.

Import Process

When documents are imported, in most cases, a source file is required; only ‘External Documents’ and ‘COSHH Documents’ don’t require a source file.

Multiple source files can be uploaded before adding the metadata, or each source file can be added individually to the corresponding metadata record in the staging area.
One or multiple source files can be initially uploaded:

• in the start page, by dragging them into the dotted rectangle labelled, ‘Click or drag and drop the excel file to add new Documents to the import queue’, or clicking inside the rectangle to open the browser’s search and select window.
• or, if documents are already in the staging area, by clicking the button, Add New Records, and using the dotted area in the pop-up window which appears.

Step 1 - Download a Template

To facilitate the import, a formatted Excel™ template is available to download and populate, so the information can be uploaded to the system in a structured manner. Simply click the link, ‘Download blank Excel file’, to retrieve a template which you can open in Excel.

It is also possible to upload a collection of source files first and then download the template pre-populated with any data added. This can be done by clicking the button, ‘Edit in Excel’, and then clicking, ‘Download Queue in Excel file’ in the pop-up window which appears.

Step 2 - Preparing the Data

The template file will have a header row with the fields which will be imported. This row should not be altered; no additional columns should be added and they should be kept in the order they are presented. If such changes are made, the system will reject the file with a message saying the “upload validation failed” and a reason. The red triangles in the top right corner of cells in the header row, indicate that a ‘help’ comment is available when you hover over the cell.
More detailed information about the content for each field can be found below in the section, Fixing Errors.

To populate the spreadsheet, use one row for each document record and please ensure that,

• The fields, ‘Organisational Unit’, ‘Category’, ‘Section’, ‘Shared With Containers’ (OUs), and ‘Reagent’ are entered exactly as they appear in iPassport (copy/paste is easiest).
  • The fields, ‘Section’ and ‘Shared With Containers’ (shared with other OUs), allow multiple entries in one cell and they must be separated using semicolons.
• The Name (title of the document) should be entered as a string of text.
• The State field only accepts two values, Draft or Authorised (British spelling).
  • A Version Number needs to be entered when a document is declared as ‘Authorised’.
• The Document Type should be one of the following:
  • COSHH Document
  • Document
  • External Document
  • Health and Safety Document
  • Job Description
  • Policy
  • SOP
• The people (Authors, Document Owners, and Authorisers) should also be also entered accurately; full names must be entered, as they appear in their User record. These fields also allow multiple entries in one cell and they must be separated using semicolons.
• The dates (Next Review Date, Date Authorised, and Effective Date) should be formatted as a ‘Date’, using local Excel™ settings.

• The field, Select a Permission Level, should only be populated when the document is shared with other OUs (Shared With Containers). It accepts the following values:
  • Read
  • Read with Print
  • Edit
  • Authorise
• Only populate the Reagent field when the document type is, COSHH Document

Step 3 - Importing the Data

Once the spreadsheet is ready, it can be simply saved and uploaded in one of two places, depending on whether source files have already been uploaded and metadata records are already listed.

Before adding source files, in the start page:

• Drag it into the dotted line rectangle labelled, ’Click or drag and drop the Excel file to add new Documents to the import queue’
• or, click inside the rectangle and use the browser window which pops up to find and select it.

After adding source files, in the staging page:

• Click the button, ‘Edit in Excel’
• Drag the file into the dotted line rectangle labelled, ’Click or drag and drop the excel file to update the Documents in the import queue’
• or, click inside the rectangle and use the browser window which pops up to find and select it.

The data imported will immediately start appearing on the screen and the system will quickly validate all the information. Feedback is available on this screen to easily identify and fix problems.

Reviewing Data and Preparing to Import

The staging area will display all records which have been uploaded.

It has a Search field to help locate particular records, and two filters:

• Show Errors Only - which hides all rows validated and ready to import
• Show Only My Imports - which hides all rows added by other users

Four buttons are available at all times:

• Clear Import Queue - This button will delete all the records in the staging area.

  • If rows are selected by ticking their checkbox in the left column, the label will change to, ‘Delete x Documents’, where ‘x’ is the number of rows selected.

• Import All Documents - This button will import all rows below which have passed validation.

  • If rows are selected by ticking their checkboxes in the left column, the label will change to, ‘Import x Documents’, where ‘x’ is the number of rows selected.

• Add New Records - Records can be added to the staging area by clicking this button. It will bring up a pop-up window like the initial screen (before the staging area is populated), where a template can be downloaded, if needed, and where a populated file can be uploaded.

  • Source files can be uploaded in the dotted area of this window.
  

• Edit in Excel - Clicking this button will invoke a pop-up window, similar to the one above, containing a link to a spreadsheet populated with the information currently in the staging area (’Download Queue in Excel file’). It also offers a place to upload the edited file again. The system will identify individual records/rows when not the entire list is downloaded in the Excel sheet.

  • Before clicking the button, rows can be selected and/or filters can be used to condense the list and only include in the Excel™ file those records which need amending.
  

The Staging Table

The information is displayed in a table with the following columns and features:

• Checkbox column on the left
  • One or multiple rows can be individually selected.
  • The ‘SHIFT’ (⬆️) key can be used to select or unselect multiple rows.
  • The checkbox in the header area selects/unselects all the rows below.
  • When at least one checkbox is ticked, an additional button appears above the table - ‘Edit x Document(s)’, where ‘x’ is the number of rows selected.
• STATUS - If information is missing or not found by the system, a yellow warning triangle will display on the row of the problem record. Hovering over the triangle will bring up a message listing the errors. When a record is ready to import, the status will show a green thumbs-up.
• Record data uploaded from the spreadsheet (only the main parameters are displayed as the metadata for documents includes many fields):
  • INDEX
  • NAME
  • ORGANISATIONAL UNIT
  • STATE
  • DOCUMENT TYPE
  • SOURCE FILE
• ACTIONS - There are three icons in this column:
  • Magnifying glass (🔍) - to open the ‘Edit Document’ pop-up window; additionally, the pop-up window can be opened anywhere in the row where the pointer turns to a ‘pointing hand’
  • Import row (↥) - to import a single row if it’s ready
  • Trash/Bin (🗑) - to delete a single row

Fixing Errors

If any of the cells have an error, they will be highlighted in pink.
If a row still displays a yellow warning triangle in the status column, and no cells appear to be invalid (highlighted in pink), there might be other fields in the same record which need fixing.
Clicking a cell (where the pointer turns to a hand) or the row’s magnifying glass, will bring up the ‘Edit Document’ pop-up window, which includes all the fields of the single row so they can be edited.

• Fields which have an error will display the reason in small red text below them.
• Clicking any of the fields, Organisational Unit, Document Type, State, Authors, Document Owners, Category, Section, Shared With Containers, or Select a Permission Level, will bring up a dropdown menu of all options available. You can scroll down the list or start typing to jump to matching items on the list and click the desired item when located.
  • Currently, all items in the account are listed under some dropdown menus (e.g., Organisational Unit). The system does validate the selections and reports back when an item has been selected which the user doesn’t have permission to access for this operation. A future enhancement will reduce these lists, based on the end-user’s permissions.
• If the State field is changed from Draft to Authorised, additional fields will appear on the form:
  • Date Authorised - a pop-up calendar is available to declare when the document was last authorised
  • Effective Date - a pop-up calendar is available to declare when the document was released into circulation after authorised; if left blank, the system will use the date from the field, Date Authorised
  • Version Number - this field only accepts whole numbers or numbers with a dot to indicate decimals (e.g., ‘3.2’)
  • Authoriser(s) - a dropdown menu is provided; at least one must be selected
• The Name field (title of the document) will be initially populated with the source file name but it can be edited as required.
• The Index field allows free text for predetermined indices, but it can be left blank or with the text, “Auto Generate” to instruct the system to generate the index automatically.
  • The system will validate manually entered indices to confirm their uniqueness across the account
  • It is not possible to predict in which order documents will be processed when they are imported, so the sequential numbers in indices might not end up as intended. If pre-existing documents already had a set index with a given sequential number, it’s preferable to declare it, than to let the system auto generate it.
• The Next Review Date is not required but saves time later if it’s added to ‘authorised’ documents
• The field, Shared With Containers allows sharing the document with other OUs. The ‘Organisational Unit Preference’, “Enable sharing Controlled Documents in this OU”, must be enabled to allow sharing from the import operation.
  • If any OUs are listed, a permission level is required as described above under the field, Select a Permission Level.
• The Source File field will appear as a dotted rectangle if no source file has been added. Once a source file is uploaded, a download icon, a delete icon and an Upload button (to replace the source file) become available.
• The field, Reagent, should only be populated when the document type is COSHH Document
• Multiple Attachments can be uploaded in the designated dotted area, but they can only be added to each individual document. Once uploaded, they will appear listed at the bottom of the form.
• All date fields produce a calendar when clicked, where dates can be selected, but dates can also be edited by highlighting the day, month or year, then entering a number or using the up/down arrows to increase or decrease the value.

Once fixed, click Save Document to validate, update and return to the staging area.

If multiple rows contain the same error, they can be selected and clicking the button, ‘Edit x Document(s)’ will allow editing and applying the same fix to all the selected records. In this case, reasons are not provided for cells with errors; the fields will state, ‘Multiple values, click to edit…’. Upon clicking in a field, the message, ‘Leave this blank to clear on all selected records.’ will appear below it, with the option to ‘Cancel edit’.
When a single document is being edited and an error declares that the item was not found, another item can be selected if available, or the item which is producing the error can be selected and then removed before saving, to clear the field.

Importing the Documents

There are three ways of importing documents once they are ready:

• Import a single row - Clicking the ‘Import’ (↥) icon in a row will import that record alone.
• Import multiple rows - multiple rows can be selected by ticking their checkbox and the button, ‘Import x Document(s)’ will import them in one step.
• Import all rows - Clicking Import All Documents will import all rows displayed below which have passed validation.

A confirmation dialogue box will appear, stating how many documents are to be imported.

Once imported, the documents will be added in the corresponding Organisational Units within the Controlled Documents area.