Importing with the Data Loader

You can import data from flat files of text format into the system for any web-based AASHTOWare Project business entity by using the Data Loader feature. The import file must be a text file in the Data Loader File Format.

• Caution: This import process should be performed with care by qualified agency personnel only. Make sure that a full back up of your existing system is performed before executing the import. Carefully consider the order in which data is imported as this can significantly impact the success of the import. For example, if an entity uses a code table, you must import the code table data before importing the entity. Because a proposal contains project data, projects must be imported before proposals.

You can execute the Data Loader directly from the command prompt or in the application by selecting Import File from the Actions menu on the Menu Bar. See the sections below for step by step instructions.

When you execute the Data Loader, the system generates a text file and four log files containing a summary and any errors that occurred during the import. The log files are named RejectedRecords.TXT, DataLoadSummary.LOG, DataLoad.LOG, and Error.LOG. DataLoadSummary.LOG is a summary of imported, rejected, or processed information. Any lines that were not imported successfully are contained in the RejectedRecords.TXT file. Error messages are contained in the DataLoad.LOG file. Error.LOG lists the start and end time of the process, and whether it finished to completion, regardless of errors.

• Note: Certain web-based AASHTOWare Project models cannot be imported with the Data Loader process, and other models can be inserted, but not updated. For a listing of these models, see Data Loader Limitations.

The easiest way to determine the correct format for your agency's Data Loader imports is to run an export process called Export Entity Data on the type of record you want to import. For information about how to run the Export Entity Data process, see Exporting Entity Data. For more information about creating a successful import file format, see Data Loader File Format.

Executing a Data Loader Import from the Command Prompt

Follow these steps:

1. Open a Command Prompt window.

2. From the command prompt, navigate to the ProcessAndUtils directory on your agency's application server.

3. Invoke the DataLoader executable file (DataLoader.exe) using the following command line arguments:
   
   -importfile=<Location of the import file>
   
   -generatemetadata : When present, this argument forces the DataLoader Metadata to be reloaded.
   
   -overwritefieldswithblanks <fields that are blank will be set to null>
   
   -deleteimportfile : When present, the import file is deleted after it is imported.
   
   -fieldlengths=<Location to write a list of current model field lengths> Only generated if this argument is present.
   
   -dataloadermetadatafile=<Location of the DataLoader Metadata file>
   
   -dataloadlog=<Location of the Data Load Log>
   
   -dataloadsummarylog=<Location of a Load Summary log>
   
   -rejectedrecordsfilename=<Location of the rejected records file>

4. Use the following command line format:
   
   "DataLoader.exe -importfile=<import file> -dataloadermetadatafile=<DataLoader Metadata file> -dataloadlog=<DataLoad log file> -dataloadsummarylog=<DataLoad summary log> -rejectedrecordsfilename=<rejected records file> [-fieldlengths=<Field length file>] [-generatemetadata] [-deleteimportfile] [-overwritefieldswithblanks]

You must use the -generatemetadata argument to update the DataLoaderMetadata file. On generation, if the system finds an existing DataLoader metadata file, values that have been modified will be reverted back to the default values. If no existing DataLoader metadata file is found, the system creates the DataLoader metadata file by merging base business metadata and custom business metadata. Fields that are set as read-only or are system generated are not included in the DataLoader metadata file and cannot be imported.

Note that the -generatemetadata argument will not replace deleted fields. If you want new fields to be added to the DataLoaderMetadata file, or if you want to replace a field that was previously removed, you must use the -regeneratemetadata argument. If you remove a field from the DataLoader metadata file and then use the -regeneratemetadata argument, the field will be added again.

Executing a Data Loader Import in web-based AASHTOWare Project

• Note: When using this method, the first line of the Data Loader file must contain the text AASHTOWareProjectDataLoader (or AASHTOWareProjectDataLoader, csv if you are importing a comma-separated values (csv) file). It should contain no other text. This first line informs the import process which type of import should be initiated. For more information about how to format the Data Loader file, see Data Loader File Format.

Follow these steps:

1. Select Import File from the Actions menu on the Menu Bar, or select Import from the Actions menu on the Project Overview component header.
   
   The system takes you to the Import component.

2. Click the Select File button.
   
   The system opens a Choose File to Upload window

3. In the Choose File to Upload window, locate the Data Loader import file that you want to import, select it, and click the Open button.
   
   The system closes the window and validates the selected file. The validated file name is displayed in the Import component and an Import button appears on the component header.

4. Based on the type of file you open, the system may offer different options for your import, like the ability to ignore blank rows or overwrite existing data with those blank rows. For more information about the import options available for each type of file, refer to these help topics:
   
   Importing Agency View Usage Data
   Importing Cost Estimates
   Importing Data from CAS to Civil Rights & Labor
   Importing Information from Bids
   Importing Data from SiteManager to Civil Rights & Labor
   Importing Estimation Reference Data
   Importing Item Mappings
   Importing Project Information from Estimator and Other Applications

5. Click the Import button on the component header to begin the import.

When the process completes, the system displays a message on the component header indicating that the import was successful and the process history ID. You can view the resulting output files (DataLoad.log, RejectedRecords.txt and DataLoadSummary.log) on the Process History Overview component (see Viewing Process History).

• Note: When you run the Data Loader from web-based AASHTOWare Project, the -overwritefieldswithblanks process argument is used. To run the DataLoader with other process arguments (such as -generatemetadata, which should be used to update agency fields), execute at the command prompt in the ProcessesAndUtils directory.

Related topics:

Data Loader File Format