Advanced Data Ingestion (ADI)
Welcome engineers!
My name is Josh Levy, and I will be sharing a series of walkthroughs describing how you can utilize functions in our new engineering software application: The Lyceum.
The problem we’re addressing in this walkthrough is prepping data for comparison and/or analysis.
Every test/measurement engineer spends copious amounts of time on administrative work. This includes cleaning data in spreadsheets, filing data in the correct directories on the correct platform, and searching for and finding data in order to compare and report in the future.
The data ingestor of the Lyceum platform simplifies and can actually automate this process.
Upload Steps:
So after you log into the Lyceum application, you can navigate to the “Data Ingestor” page via the left-hand menu. On this page, you will notice a 4-step walkthrough that will visually guide you through the upload process.
The first step will have 3 input windows.
File selection
Fast upload
Name/Title
The user will first click in the file-selection window to choose the file on their computer to upload.
The fast upload section will be ignored for now, and we will revisit later in the demo.
The user will then give the file a title for the file they are uploading, and select “Next” to proceed to the Document Type page.
The Document Type page will have two windows:
File Type
Data Type
The file type window will have a dropdown menu, where there will be a list of all currently supported file types. As of this demo, we only support the following files:
This list is continuously growing. Please contact me if with your data file type needs and we can add to the Lyceum data ingestor.
The Data Type window is where the uploader decides which database this gets input into. At the moment of this demo, we only support two-dimensional (x,y) data to be categorized into different databases. Those categories are:
The user then navigates to the “Enter Tags” page, whereby they input and attach different types of descriptors to the data they’re uploading. This enables users in the future to more easily find and access this data.
There are descriptor categories that can be added or selected from a pre-defined list, each requiring a custom description that can later be parsed during a fast and simple search.
The tags are space-delimited text strings that can also be attached to uploads, giving the uploader flexibility with how they would like to be able to find data in the future.
The final step of the upload process, before cleaning the actual data, is then determining who has access to the data. This is done in the “Privileges” page.
The uploader will then choose which groups the data will be accessible from, and only users in these groups will have access to this data. The uploader will be considered the owner of this data, and will be the only user with the ability delete this data and change its permissions. Group owners and administrators will then be able to control who are members of those groups, ultimately controlling who has access to the data.
Now the data needs to be cleaned.
Cleaning steps:
Once the uploader moves to the cleaning stage, they will have access to the actual data being uploaded, and need to transform it into the proper format to be accepted by the database. The data can come in multiple categories, nodes, units, and resolution points, so in order to do transform it properly, need to complete 3 tasks for each category being uploaded:
Choose common axis column
Choose units for each axis
Resolve invalid cells
This is done by inputing the appropriate setting in the windows above the data labelled:
Common axis column
X-Axis Units
Y-Axis Units
Invalid Cells
In order to find the invalid cells, click the arrow button in the top-right hand corner above the data set, and it will navigate immediately to the next invalid cell.
Once all the data is cleaned and ready to upload, the uploader can also then elect to download the data in the clean format, as well as download a template file.
Template file upload:
The template file is a .json file with a list of all steps taken to attach descriptors and cleaning steps to prepare for upload. It can then be used when uploading future files with the same format that needs to be transformed with the same steps.
When an uploader chooses this method, all other steps CAN be edited or tweaked, however all steps can be bypassed to decrease the amount of time spent on uploading and cleaning. This is especially useful when engineers are running the same and/or similar experiment multiple times.
Once all data is uploaded and processed by our backend, it will be instantly accessible. Searching, analyzing and reporting are all quick steps that the Lyceum also offers, and will have their own walkthroughs.
Thanks for reading! If you have any other questions or requests, please don’t hesitate to reach out as we look to accommodate the entire engineering community.
