Create HL7 messages – Etlworks Support

Overview

In this article, we provide several examples of creating HL7 messages in Etlworks using a dedicated HL7 flow and JavaScipt.

Lab Order processing

Flow description

A hospital sends an HL7 2.3 message containing an order via SFTP to their predefined vendor folder on the Data Exchange Server. An interface engine (Etlworks) formats the source message into the AP Easy specification and moves it to the predefined client folder on the server. The client is a Lab, such as Quest Diagnostics.

Source HL7 message

MSH|^~\&|MM|MOD6078|AP Easy|MOD6078|20161216231442||ORM^O01|211072-20161216231442|P|2.3|
PID|1|12345678|123456||LastName^FirstName^MiddleName||19050925|M|||1000 PERSHING LN^^FORKED RIVER^NJ^08731||||||||11122333
IN1|1||12379|MEDICARE B-NJ: NOVITAS SOLUTIONS|PO BOX 1234^^MECHANICSBURG^PA^170551802||||||||||PRIMARY|LastName^FirstName^MiddleName|1|19050925|^^^^|||1||||||||||||||1504245123
IN1|2||61667|BCBS-NJ: HORIZON BCBS - OUT OF STATE - BLUE CARD (PPO)|PO BOX 11122^^NEPTUNE^NJ^077541301|||00865600002|||||||SECONDARY|LastName^FirstName^MiddleName|1|19050925|^^^^|||2||||||||||||||YHB3HZN64699555
ORC|NW|211072-A||211072|||||20161216231442|||^HAHN^JUSTENE^
OBR|1|211072-A|A|D48.5^Biopsy by Shave Method;H and E||20161216231442|20161216081131||||||Morphology:  erythematous scaly papule;DDX: Squamous Cell Carcinoma vs. Verruca Vulgaris vs. Irritated Seborrheic Keratosis;Location: left central parietal scalp;||^^^left central parietal scalp|^HAHN^JUSTENE^|||||||||F||^^^^^^^^||||
NTE|1|Specimen Label|shave bx
NTE|2|Specimen Source|A) left central parietal scalp
NTE|3|Morphology|erythematous scaly papule
NTE|4|Procedure Name|Biopsy by Shave Method
NTE|5|triangulation|from lt tragus 14.5cm up and 3cm rt
NTE|6|triangulation|0.6cm
DG1|1|I10|D48.5|Squamous Cell Carcinoma vs. Verruca Vulgaris vs. Irritated Seborrheic Keratosis
ORC|NW|211072-B||211072|||||20161216231442|||^HAHN^JUSTENE^
OBR|1|211072-B|B|D48.5^Biopsy by Shave Method;H and E||20161216231442|20161216081049||||||Morphology: Erythematous scaling plaque;DDX: Squamous Cell Carcinoma vs. Verruca Vulgaris vs. Irritated Seborrheic Keratosis;Location: left central frontal scalp;||^^^left central frontal scalp|^HAHN^JUSTENE^|||||||||F||^^^^^^^^||||
NTE|1|Specimen Label|shave bx
NTE|2|Specimen Source|B) left central frontal scalp
NTE|3|Morphology|Erythematous scaling plaque
NTE|4|Procedure Name|Biopsy by Shave Method
NTE|5|triangulation|from lt tragus 14.5 cm up and 2cm rt
NTE|6|triangulation|1cm
DG1|1|I10|D48.5|Squamous Cell Carcinoma vs. Verruca Vulgaris vs. Irritated Seborrheic Keratosis

The reformatted destination HL7 message

MSH|^~\&|Integrator|MOD6078|AP Easy|MOD6078|20161216231442||ORM^O01|211072-20161216231442|P|2.3
PID|1|12345678|123456||LastName^FirstName^MiddleName||19050925|M|||1000 PERSHING LN^^FORKED RIVER^NJ^08731||||||||11122333
PV1|1||^^^&MOD6078||||^HAHN^JUSTENE
IN1|1||12379|MEDICARE B-NJ: NOVITAS SOLUTIONS|PO BOX 1234^^MECHANICSBURG^PA^170551802||||||||||PRIMARY|LastName^FirstName^MiddleName|1|19050925||||1||||||||||||||1504245123
IN1|2||61667|BCBS-NJ: HORIZON BCBS - OUT OF STATE - BLUE CARD (PPO)|PO BOX 11122^^NEPTUNE^NJ^077541301|||00865600002|||||||SECONDARY|LastName^FirstName^MiddleName|1|19050925||||2||||||||||||||YHB3HZN64699555
ORC|NW|211072-A||211072|||||20161216231442|||^HAHN^JUSTENE
OBR|1|211072-A|A|D48.5^Biopsy by Shave Method;H and E||20161216231442|20161216081131||||||Morphology:  erythematous scaly papule;DDX: Squamous Cell Carcinoma vs. Verruca Vulgaris vs. Irritated Seborrheic Keratosis;Location: left central parietal scalp;||^^^left central parietal scalp|^HAHN^JUSTENE|||||||||F
OBX|1|FT|03^Clinical||Morphology:  erythematous scaly papule;DDX: Squamous Cell Carcinoma vs. Verruca Vulgaris vs. Irritated Seborrheic Keratosis;Location: left central parietal scalp;
OBX|2|FT|01^Site||A) left central parietal scalp
OBX|3|FT|02^Procedure||SHAVE
ORC|NW|211072-B||211072|||||20161216231442|||^HAHN^JUSTENE
OBR|1|211072-B|B|D48.5^Biopsy by Shave Method;H and E||20161216231442|20161216081049||||||Morphology: Erythematous scaling plaque;DDX: Squamous Cell Carcinoma vs. Verruca Vulgaris vs. Irritated Seborrheic Keratosis;Location: left central frontal scalp;||^^^left central frontal scalp|^HAHN^JUSTENE|||||||||F
OBX|1|FT|03^Clinical||Morphology: Erythematous scaling plaque;DDX: Squamous Cell Carcinoma vs. Verruca Vulgaris vs. Irritated Seborrheic Keratosis;Location: left central frontal scalp;
OBX|2|FT|01^Site||B) left central frontal scalp
OBX|3|FT|02^Procedure||SHAVE

Connections

Source: SFTP
Destination: SFTP

Prerequisites

Before creating a Flow, you must create all the required folders on the server:

source-orders
    processing
    failed
destination-orders

How an order is processed

The hospital sends an order via SFTP to the source-orders folder.
Etlworks moves the order to the processing folder.
Etlworks reformats the message into the AP Easy specification, which is understood by the Lab software and moves it to the destination-orders folder.
If there is an error during Step 3, Etlworks moves the original message to the failed folder.

This suggested approach is just one of many possibilities. We want to make our Flow processing as reliable as possible, hence all the file movements between folders. Ultimately, Step 3 does all the hard lifting; everything else is just the implementation of a fail-safe mechanism.

The order Flow

Step 1. Create all the required SFTP Connections, one Connection per folder, as above.

When creating Connections, set the following properties:

For a Connection that points to the processing folder and has Enable Wildcard File Name set to oldest. If there are multiple files in the processing folder, it will direct the Flow to select the oldest file to process.
For a Connection that points to the destination-orders folder and has Add Suffix When Creating Files set to timestamp, it will direct the Flow to create files in the destination-orders folder with a timestamp suffix. With this setting, if the file already exists, it will not be overwritten. Instead, a new file will be created each time.

Step 2. Create an HL7 Format.

When creating a Format, set the HL7 Version to 2.3 and leave everything else set to the default:

Step 3. Create a reformat Flow.

This Flow will reformat the source message in the processing folder to the AP Easy specification and will move it to the destination-orders folder.

For this Flow, you will need the following Connections and Formats:

A Connection that points to the processing folder.
A Connection that points to the destination-orders folder.
The Format created in Step 2.

Start creating a Flow by clicking Add flow on the Flows window.
Type HL7 into the search box and select HL7 to HL7.
In the opened window, select the source (FROM) Connection that points to the processing folder and the destination (TO) Connection that points to the destination-orders folder.
Select the Format created in Step 2 for the FROM and TO Formats.
Enter *.hl7 as the FROM and Vendor_Order_Destination.hl7 into the TO field:

6. Click MAPPING and select the Parameters tab.

7. Enter order in the Transformation Name field. It will be used in reports and on the dashboard.

8. Select Ignore when there is no file. By doing this you will allow the Flow to finish even when there are no files in the processing folder.

9. Enter the following code into the Preprocessing field. This code does all the reformatting. Read more about using JavaScript when the source is an HL7 message or when the destination is an HL7 message.

var javaImports = new JavaImporter(Packages.ca.uhn.hl7v2.model.v23.message, 
Packages.ca.uhn.hl7v2.model.v23.segment, Packages.ca.uhn.hl7v2.model.v23.datatype); 

with (javaImports) {

var message = dataSet.getActualData();

// clone source message
var destMessage = ClassUtils.clone(message);

// change sender
destMessage.getMSH().getSendingApplication().getNamespaceID().setValue("Integrator");

var numOrders = destMessage.getORDERReps();

// add OBX segmentts, remove NTE and DG segments
for (num = 0; num < numOrders; num++) {
   var order = destMessage.getORDER(num);

   var detail = order.getORDER_DETAIL();

   var allNte = detail.getNTEAll();

   var maxId = 1;
   for (i = 0; i < allNte.length; i++) {
      // skip 1,3,5,6
      if (i == 0 || i == 2 || i == 4 || i == 5) {
          continue;
      }  

      maxId++;
   } 

   // obx 1
   var obx = detail.insertOBSERVATION(0).getOBX();
   obx.getObx1_SetIDOBX().setValue(1);
   // OBX-2 -> FT ?
   obx.getObx2_ValueType().setValue("FT");
   // OBX-3.1 -> max ID ?
   obx.getObx3_ObservationIdentifier().getIdentifier().setValue("0" + maxId);
   // OBX-3.2 -> Clinical ?
   obx.getObx3_ObservationIdentifier().getText().setValue("Clinical");
   // OBX-5 = OBR-13 in source message
   var ft = new FT(destMessage);
   ft.setValue(detail.getOBR().getObr13_RelevantClinicalInformation().getValue());
   obx.insertObservationValue(0).setData(ft);

   var index = 1;

   for (i = 0; i < allNte.length; i++) {
      var nte = allNte.get(i);

      // skip 1,3,5,6
      if (i == 0 || i == 2 || i == 4 || i == 5) {
          continue;
      }  

      var obx = detail.insertOBSERVATION(index++).getOBX();

      // OBX-1 -> index ? 
      obx.getObx1_SetIDOBX().setValue(index);
      // OBX-2 -> FT ?
      obx.getObx2_ValueType().setValue("FT");
      // OBX-3.1 -> index ?  
      obx.getObx3_ObservationIdentifier().getIdentifier().setValue("0" + (index - 1));

      // OBX-3.2 -> mapping, from NTE-2 ?  
      var qualifier = nte.getNte2_SourceOfComment().getValue().toLowerCase();
      obx.getObx3_ObservationIdentifier().getText().setValue(qualifier.contains("specimen") ? "Site" : "Procedure");

      // OBX-5 = mapping from NTE-3
      var obx5Value = nte.getNte3_Comment(0).getValue();
      var nte3 = obx5Value.toLowerCase();

      if (nte3.contains("biopsy by punch method")) {
         obx5Value = "PUNCH"; 
      } else if (nte3.contains("biopsy by shave method")) {
         obx5Value = "SHAVE"; 
      } else if (nte3.contains("shave removal")) {
         obx5Value = "SHAVE"; 
      } else if (nte3.contains("curettage and destruction with pathology")) {
         obx5Value = "SHAVE"; 
      } else if (nte3.contains("punch excision")) {
         obx5Value = "EXCISION"; 
      } else if (nte3.contains("excision")) {
         obx5Value = "EXCISION"; 
      }   

      var ft = new FT(destMessage);
      ft.setValue(obx5Value);
      obx.insertObservationValue(0).setData(ft);


   }  

   // remove NTE segments
   while (detail.getNTEReps() > 0) {
     detail.removeNTE(0);
   }  

   // remove DG1 segments
   while (detail.getDG1Reps() > 0) {
     detail.removeDG1(0);
   }  
}

// add segment PV1
var pv1 = destMessage.getPATIENT().getPATIENT_VISIT().getPV1();

// PV1-1: Patient Visit ID 
pv1.getPv11_SetIDPatientVisit().setValue("1");

// PV1-3 = MSH-4  
pv1.getAssignedPatientLocation().getFacility().getUniversalID().setValue(
  destMessage.getMSH().getMsh4_SendingFacility().getNamespaceID().getValue());

// XCN - attending doctor
var doctor = pv1.getAttendingDoctor(0);

if (message.getORDER().getORC().getOrc12_OrderingProviderReps() > 0) {
   var provider = message.getORDER().getORC().getOrc12_OrderingProvider(0);

  // PV1-7.1 = ORC-12.1 
   doctor.getIDNumber().setValue(provider.getXcn1_IDNumber().getValue());

   // PV1-7.2 = ORC-12.2 
   doctor.getFamilyName().setValue(provider.getFamilyName().getValue());

   // PV1-7.3 = ORC-12.3 
   doctor.getGivenName().setValue(provider.getGivenName().getValue());

  // PV1-7.4 = ORC-12.4 
   doctor.getMiddleInitialOrName().setValue(
provider.getXcn4_MiddleInitialOrName().getValue());
}  

dataSet.setActualData(destMessage);
}

10. Save and test the Flow.

A prerequisite for this Flow to work: the source file exists in the processing folder.
To verify that the Flow has been executed and has produced the correct results: make sure the correct file was created in the destination-orders folder.

Step 4. Create a move files processing Flow.

This Flow moves files from the source-orders folder to the processing folder.

For this Flow, we will need the following Connections:

A Connection that points to the source-orders folder.
A Connection that points to the processing folder.

1. Start creating a Flow by clicking Add flow on the Flows window.

2. Type files into the search box and select File Management.

3. In the opened window, select the source (FROM) Connection that points to the source-orders folder and the destination (TO) Connection that points to the processing folder.

4. Click MAPPING and select the Parameters tab.

5. Select Move for the Action. By selecting this action, you are configuring the Flow to move files from the source-orders folder to the processing folder.

6. Save and test the Flow.

A prerequisite for this Flow to work: the source file exists in the source-orders folder.
To verify that the Flow has been executed and has produced the correct results: check that the file was copied to the processing folder and deleted from the source-orders folder.

Step 5. Create a clean up Flow.

This Flow deletes files from the processing folder or, if there is an error, moves files to the failed folder.

1. Start creating a Flow by clicking Add flow on the Flows window.

2. Type files into the search box and select File Management.

3. In the opened window, add two transformations:

Delete files from the processing folder.

1. Select the source (FROM) Connection that points to the processing folder.

2. Enter *.* in the FROM field.

3. Click MAPPING and select the Parameters tab.

4. Select Delete for the Action. By selecting this action you are configuring the transformation to delete files from the processing folder.
Move files to a failed folder.

1. Select the source (FROM) Connection that points to the processing folder.

2. Select the destination (TO) Connection that points to the failed folder.

3. Enter *.* in the FROM field.

4. Click MAPPING and select the Parameters tab.

5. Select Move for the Action. By selecting this action, you are configuring the transformation to move files from the processing folder to the failed folder.

6. Select Execute if Error. By selecting this action you are configuring the transformation to move files from the processing folder to the failed folder in case of any error.

4. Save and test the Flow.

A prerequisite for this Flow to work: the source file exists in the processing folder.
To verify that the Flow has been executed and has produced the correct results: check that the file was deleted from the processing folder.

Step 6. Put it all together.

In the previous 5 steps, we created 3 Flows which:

Move files from the source-orders to the processing folder.
Reformat a file and copy it to the destination-orders folder.
Clean up by removing files from the processing folder, or moving files to the failed folder if there was an error.

In order to put it all together, we will need to create a nested Flow.

Add the following Flows to the nested Flow:

1. The Flow created in Step 4.

2. The Flow created in Step 3.

3. The Flow created in Step 5.

4. Save and test the Flow.

A prerequisite for this Flow to work: the source file exists in the source-orders folder.
To verify that Flow has been executed and has produced the correct results: check that the correct file was created in the destination-orders folder, and the original source file was deleted from the source-orders folder.

Step 7. Schedule the Flow created in Step 6 to be executed periodically

Access scheduling Flows here.

Lab results processing

Flow description

The client sends a HL7 2.3 message containing test results and a PDF from AP Easy via SFTP to their predefined client folder on the server. The interface engine (Etlworks) Formats the source message to vendor specifications. Base x64 encodes the PDF into the HL7 message as the last OBX segment, and moves the file to the predefined vendor folder on the server. The vendor retrieves messages via SFTP. The client is a Lab, such as Quest Diagnostics.

Source HL7 message

MSH|^~\&|AP Easy|COMPASS||MOD6078|20161216140344||ORU^R01|C16-23271|P|2.2
PID|1|123456|Pt000123456||TEST^JACK^||191220|M|||123 N SMITH ROAD^^GALLOWAY^NJ^08205||||||||||
PV1|1||||||1902855349^Yanusz^Donna^^APN|^^^^|^^^^||||||||^^^^
ORC|RE|210915-A210915-B|C16-23271^AP Easy||CM||||201612150712
OBR|1|210915-A~210915-B~|C16-23271^AP Easy|Surgical Pathology|||201612140000|||||||201612150712||1902855349^Yanusz^Donna^^APN||||||201612160000||SP|F|||||||1093993263^Erickson^Christof^^MD|^^^^|^^^^
OBX|1|TX|Specimen(s) Received||A.  PUNCH.  right medial heel\\.br\\B.  PUNCH.  right medial heel\\.br\\||||||F|||||||
OBX|2|TX|Clinical Information||A.  Morphology: tense bullae and erosions;DDX: Bullous Pemphigoid vs. Blister\\.br\\B.  Morphology: tense bullae and erosions;DDX: Bullous Pemphigoid vs. Blister\\.br\\||||||F|||||||
OBX|3|TX|Gross Description||A.  A tan three piece aggregate 3x2x12 mm punch biopsy skin specimen was received in formalin. The larger piece was bisected and entirely submitted with one punch, as well as with the smaller piece in one cassette.\\.br\\B.  A tan two piece aggregate 3x2x8 mm punch biopsy skin specimen received in Michel's medium. One punch and the remaining piece were entirely submitted for frozen sectioning.\\.br\\||||||F|||||||
OBX|4|TX|Microscopic Description||A.  In thick skin there is subepidermal vesicular dermatitis with an infiltrate of lymphocytes, histiocytes and eosinophils in the papillary dermis beneath. A special stain, PAS, is negative for hyphae. \\.br\\B.  Frozen tissue examination was performed and shows vesicle formation with loss of the overlying epidermis.\\.br\\DIRECT IMMUNOFLUORESCENCE RESULTS:\\.br\\     IgG - negative\\.br\\     C3  - negative\\.br\\     IgA - negative\\.br\\     IgM - negative\\.br\\     Fibrinogen - negative\\.br\\Positive and negative controls were examined and showed appropriate staining. Intrinsic staining of patient tissue was appropriate.\\.br\\\\.br\\This test was developed and its performance characteristics determined by Compass Dermatopathology. It has not been cleared or approved by the U.S. Food and Drug Administration (USFDA). The USFDA has determined that such clearance or approval is not necessary. This test is used for clinical purposes and should not be regarded as investigational or for research. This laboratory is certified under the Clinical Laboratory Improvement Amendments (CLIA) as qualified to perform high complexity clinical laboratory testing.\\.br\\\\.br\\The CPT and ICD-10 codes are for information purposes only and are based on AMA guidelines without regard to specific payor requirements. \\.br\\||||||F|||||||
OBX|5|TX|Diagnosis||A.  SUBEPIDERMAL VESICULAR DERMATITIS WITH EOSINPHILS\\.br\\B.  NON-DIAGNOSTIC Direct ImmunofluorescencE\\.br\\\\.br\\||||||F|||||||
OBX|6|TX|Comments||A.  These changes may be seen in bullous pemphigoid. The histologic differential diagnosis includes a bullous hypersensitivity reaction and other immunobullous disorders. \\.br\\B.  The biopsy shows blister formation with loss of the epidermis which results in artifactual changes which render interpretation of immunofluorescence patterns problematic. Repeat perilesional biopsy for direct immunofluorescence is recommended.\\.br\\\\.br\\||||||F|||||||
OBR|2|210915-A~210915-B~|C16-23271^AP Easy|Surgical Pathology|||201612140000|||||||201612150712||1902855349^Yanusz^Donna^^APN||||||201612160000||SP|F|||||||1093993263^Erickson^Christof^^MD|^^^^|^^^^
OBX|1|TX|Specimen(s) Received_1||PUNCH.  RIGHT MEDIAL HEEL|||N|||F|||||||
OBX|2|TX|Specimen(s) Received_2||PUNCH.  RIGHT MEDIAL HEEL|||N|||F|||||||
OBX|3|TX|Clinical Information_1||Morphology: tense bullae and erosions;DDX: Bullous Pemphigoid vs. Blister|||N|||F|||||||
OBX|4|TX|Clinical Information_2||Morphology: tense bullae and erosions;DDX: Bullous Pemphigoid vs. Blister|||N|||F|||||||
OBX|5|TX|Gross Description_1||A tan three piece aggregate 3x2x12 mm punch biopsy skin specimen was received in formalin. The larger piece was bisected and entirely submitted with one punch, as well as with the smaller piece in one cassette.|||N|||F|||||||
OBX|6|TX|Gross Description_2||A tan two piece aggregate 3x2x8 mm punch biopsy skin specimen received in Michel's medium. One punch and the remaining piece were entirely submitted for frozen sectioning.|||N|||F|||||||
OBX|7|TX|Microscopic Description_1||In thick skin there is subepidermal vesicular dermatitis with an infiltrate of lymphocytes, histiocytes and eosinophils in the papillary dermis beneath. A special stain, PAS, is negative for hyphae. |||N|||F|||||||
OBX|8|TX|Microscopic Description_2||Frozen tissue examination was performed and shows vesicle formation with loss of the overlying epidermis.\\.br\\DIRECT IMMUNOFLUORESCENCE RESULTS:\\.br\\     IgG - negative\\.br\\     C3  - negative\\.br\\     IgA - negative\\.br\\     IgM - negative\\.br\\     Fibrinogen - negative\\.br\\Positive and negative controls were examined and showed appropriate staining. Intrinsic staining of patient tissue was appropriate.\\.br\\\\.br\\This test was developed and its performance characteristics determined by Compass Dermatopathology. It has not been cleared or approved by the U.S. Food and Drug Administration (USFDA). The USFDA has determined that such clearance or approval is not necessary. This test is used for clinical purposes and should not be regarded as investigational or for research. This laboratory is certified under the Clinical Laboratory Improvement Amendments (CLIA) as qualified to perform high complexity clinical laboratory testing.\\.br\\\\.br\\The CPT and ICD-10 codes are for information purposes only and are based on AMA guidelines without regard to specific payor requirements. |||N|||F|||||||
OBX|9|TX|Diagnosis_1||SUBEPIDERMAL VESICULAR DERMATITIS WITH EOSINPHILS|||N|||F|||||||
OBX|10|TX|Diagnosis_2||NON-DIAGNOSTIC DIRECT IMMUNOFLUORESCENCE^B|||N|||F|||||||
OBX|11|TX|Comments_1||THESE CHANGES MAY BE SEEN IN BULLOUS PEMPHIGOID. THE HISTOLOGIC DIFFERENTIAL DIAGNOSIS INCLUDES A BULLOUS HYPERSENSITIVITY REACTION AND OTHER IMMUNOBULLOUS DISORDERS. |||N|||F|||||||
OBX|12|TX|Comments_2||THE BIOPSY SHOWS BLISTER FORMATION WITH LOSS OF THE EPIDERMIS WHICH RESULTS IN ARTIFACTUAL CHANGES WHICH RENDER INTERPRETATION OF IMMUNOFLUORESCENCE PATTERNS PROBLEMATIC. REPEAT PERILESIONAL BIOPSY FOR DIRECT IMMUNOFLUORESCENCE IS RECOMMENDED.^B|||N|||F|||||||

The reformatted destination HL7 message

MSH|^~\\&|AP Easy|COMPASS||MOD6078|20161216140344||ORU^R01|C16-23271|P|2.2
PID|1|123456|Pt000123456||TEST^JACK^||191220|M|||123 N SMITH ROAD^^GALLOWAY^NJ^08205||||||||||
PV1|1||||||1902855349^Yanusz^Donna^^APN|^^^^|^^^^||||||||^^^^
ORC|RE|210915-A|C16-23271^AP Easy|210915|CM||||201612150712
OBR|1|210915-A|C16-23271^AP Easy|Surgical Pathology|||201612140000|||||||201612150712||1902855349^Yanusz^Donna^^APN||||||201612160000||SP|F|||||||1093993263^Erickson^Christof^^MD|^^^^|^^^^
OBX|1|TX|Specimen(s) Received||PUNCH.  RIGHT MEDIAL HEEL|||N|||F
OBX|2|TX|Clinical Information||Morphology: tense bullae and erosions;DDX: Bullous Pemphigoid vs. Blister|||N|||F
OBX|3|TX|Gross Description||A tan three piece aggregate 3x2x12 mm punch biopsy skin specimen was received in formalin. The larger piece was bisected and entirely submitted with one punch, as well as with the smaller piece in one cassette.|||N|||F
OBX|4|TX|Microscopic Description||In thick skin there is subepidermal vesicular dermatitis with an infiltrate of lymphocytes, histiocytes and eosinophils in the papillary dermis beneath. A special stain, PAS, is negative for hyphae. |||N|||F
OBX|5|TX|Diagnosis||SUBEPIDERMAL VESICULAR DERMATITIS WITH EOSINPHILS|||N|||F
OBX|6|TX|Comments||THESE CHANGES MAY BE SEEN IN BULLOUS PEMPHIGOID. THE HISTOLOGIC DIFFERENTIAL DIAGNOSIS INCLUDES A BULLOUS HYPERSENSITIVITY REACTION AND OTHER IMMUNOBULLOUS DISORDERS. |||N|||F
ORC|RE|210915-B|C16-23271^AP Easy|210915|CM||||201612150712
OBR|2|210915-B|C16-23271^AP Easy|Surgical Pathology|||201612140000|||||||201612150712||1902855349^Yanusz^Donna^^APN||||||201612160000||SP|F|||||||1093993263^Erickson^Christof^^MD|^^^^|^^^^
OBX|7|TX|Specimen(s) Received||PUNCH.  RIGHT MEDIAL HEEL|||N|||F
OBX|8|TX|Clinical Information||Morphology: tense bullae and erosions;DDX: Bullous Pemphigoid vs. Blister|||N|||F
OBX|9|TX|Gross Description||A tan two piece aggregate 3x2x8 mm punch biopsy skin specimen received in Michel's medium. One punch and the remaining piece were entirely submitted for frozen sectioning.|||N|||F
OBX|10|TX|Microscopic Description||Frozen tissue examination was performed and shows vesicle formation with loss of the overlying epidermis.\\.br\\DIRECT IMMUNOFLUORESCENCE RESULTS:\\.br\\     IgG - negative\\.br\\     C3  - negative\\.br\\     IgA - negative\\.br\\     IgM - negative\\.br\\     Fibrinogen - negative\\.br\\Positive and negative controls were examined and showed appropriate staining. Intrinsic staining of patient tissue was appropriate.\\.br\\\\.br\\This test was developed and its performance characteristics determined by Compass Dermatopathology. It has not been cleared or approved by the U.S. Food and Drug Administration (USFDA). The USFDA has determined that such clearance or approval is not necessary. This test is used for clinical purposes and should not be regarded as investigational or for research. This laboratory is certified under the Clinical Laboratory Improvement Amendments (CLIA) as qualified to perform high complexity clinical laboratory testing.\\.br\\\\.br\\The CPT and ICD-10 codes are for information purposes only and are based on AMA guidelines without regard to specific payor requirements. |||N|||F
OBX|11|TX|Diagnosis||NON-DIAGNOSTIC DIRECT IMMUNOFLUORESCENCE|||N|||F
OBX|12|TX|Comments||THE BIOPSY SHOWS BLISTER FORMATION WITH LOSS OF THE EPIDERMIS WHICH RESULTS IN ARTIFACTUAL CHANGES WHICH RENDER INTERPRETATION OF IMMUNOFLUORESCENCE PATTERNS PROBLEMATIC. REPEAT PERILESIONAL BIOPSY FOR DIRECT IMMUNOFLUORESCENCE IS RECOMMENDED.|||N|||F
OBX|13|ED|PDF^PDF BASE64|1|^^PDF^Base64^$\{ATTACH:44713897-7733-4c3c-812d-ce4424183c36\}

Connections

Source: SFTP
Processing: Etlworks Home folder
Destination: SFTP

Prerequisites

Before creating a Flow, all the required folders must be created on the data exchange server:

source-results
    processing (remote)
    failed
destination-results

To read and encode a PDF file, you will need to store it in the Account's Home folder. An Account's Home folder is automatically created for each client, so the user doesn't need to worry about creating it, but you still need to have a Connection that points to this folder. Suppose this folder is called processing (local). It is going to be under the Account's Home folder.

home
    processing (local)

How lab results are processed

The lab sends results via SFTP to the source-results folder.
Etlworks moves the results and PDF to both the processing (remote) and processing (local) folders. Files in the processing (remote) folder are used as a backup copy of the original message and PDF. If a transformation fails, files from processing (remote) will be copied to the failed folder.
Using files in the processing (local) folder, Etlworks reformats the message from the AP Easy specification to the vendor specs, base64 encodes the PDF and attaches it to the last. OBX segment, then moves the message to the destination-results folder.
If there is an error during Step 3, Etlworks moves the original message and PDF from the processing (remote) to the failed folder. Otherwise, it deletes the files in both the processing (remote) and processing (local) folders.

This suggested approach is just one of many possibilities. We want to make our Flow processing as reliable as possible, hence all the file movements between folders. Ultimately, Steps 2 and 3 do all the hard lifting; everything else is just the implementation of a fail-safe mechanism.

The Flow

Step 1. Create all required SFTP Connections, one Connection per folder above.

When creating Connections, set the following properties:

A Connection that points to the processing folder and Enable Wildcard File Name is set to oldest. If there are multiple files in the processing folder, this will direct the Flow to select the oldest file to process.
A Connection that points to the destination-results folder and Add Suffix When Creating Files set to timestamp. This will direct the Flow to create files in the destination-results folder with a timestamp suffix. So, if a file already exists, it will not be overwritten. Instead, a new file will be created each time.

Step 2. Create a Connection for the processing (local) folder

The Server Storage Connection type must be used.

Since you are locked into having the Home folder as your root folder, you will need to use the Files fields to point the Connection to the actual folder under Home.

Step 3. If you don't already have it, create an HL7 Format.

Access HL7 Format here.

When creating a Format, set the HL7 Version to 2.3 and leave everything else set to the default setting:

Step 4. Create a reformat Flow.

This Flow will reformat the source message in the processing (local) folder from an AP Easy specification to the vendor specs, base x64 encode the PDF and attach it to the HL7 message in the last OBX segment. Then it will move the reformatted message to the destination-results folder.

For this Flow, we will need the following Connections and Formats:

A Connection that points to the processing (local) folder.
A Connection that points to the destination-results folder.
The Format created in Step 3.

1. Start creating a Flow by clicking Add flow on the Flows window.

2. Type HL7 into the search box and select HL7 to HL7.

3. In the opened window, select a source (FROM) Connection that points to the processing (local) folder and a destination (TO) Connection that points to the destination-results folder.

4. Select the Format created in Step 2 for the FROM and TO Formats.

5. Enter result.hl7 into the FROM and Vendor_Result_Destination.hl7 into the TO field:

6. Click MAPPING and select the Parameters tab.

7. Enter result into the Transformation Name field. It will be used in reports and on the dashboard.

8. Select Ignore when there is no file. By doing this, you will allow the Flow to finish even when there are no files in the processing (local) folder.

9. Enter the following code into the Preprocessing field. This code does all the reformatting. Read more about using JavaScript when the source or destination is an HL7 message.

var javaImports = new JavaImporter(Packages.ca.uhn.hl7v2.model.v23.message, 
Packages.ca.uhn.hl7v2.model.v23.segment, Packages.ca.uhn.hl7v2.model.v23.datatype
Packages.ca.uhn.hl7v2.util, java.io); 

with (javaImports) {

var message = dataSet.getActualData();

var destMessage = new ORU_R01();

// copy MSH from the source
DeepCopy.copy(message.getMSH(), destMessage.getMSH());

// change sender
destMessage.getMSH().getSendingApplication().getNamespaceID().setValue("Integrator");

var responses = message.getRESPONSEAll();

for (resNum = 0; resNum < responses.length; resNum++) {
   var response = responses.get(resNum);

   var newResponse = destMessage.insertRESPONSE(resNum);

   // copy PID AND PV1 from the source 
   DeepCopy.copy(response.getPATIENT().getPID(), newResponse.getPATIENT().getPID());
   DeepCopy.copy(response.getPATIENT().getVISIT().getPV1(), newResponse.getPATIENT().getVISIT().getPV1());

   // Everything below and including OBR|2| in source message is not used.
   var orderObservation = response.getORDER_OBSERVATION();

   var sourceObr = orderObservation.getOBR();

   var orderNums = sourceObr.getPlacerOrderNumber();

   if (orderNums != null) {
        for (orNum = 0; orNum < orderNums.length; orNum++) {
            var or = orderNums[orNum];

            var destOrderObservation = newResponse.insertORDER_OBSERVATION(orNum);

            // copy ORC from the source and chnage ORC-2  
            DeepCopy.copy(orderObservation.getORC(), destOrderObservation.getORC());
            while (destOrderObservation.getORC().getOrc2_PlacerOrderNumberReps() > 1) destOrderObservation.getORC().removeOrc2_PlacerOrderNumber(1); 
            destOrderObservation.getORC().getOrc2_PlacerOrderNumber(0).getEntityIdentifier().setValue(or.getEntityIdentifier().getValue());

            // copy OBR from the source
            DeepCopy.copy(orderObservation.getOBR(), destOrderObservation.getOBR());
            while (destOrderObservation.getOBR().getObr2_PlacerOrderNumberReps() > 1) destOrderObservation.getOBR().removeObr2_PlacerOrderNumber(1); 
            destOrderObservation.getOBR().getObr2_PlacerOrderNumber(0).getEntityIdentifier().setValue(or.getEntityIdentifier().getValue());

            var observations = orderObservation.getOBSERVATIONAll();

            // copy source OBX to dest OBX
            for (index = 0; index < observations.length; index++) {
                 var observation = observations.get(index);

                 var newObservation = destOrderObservation.insertOBSERVATION(index);

                 DeepCopy.copy(observation.getOBX(), newObservation.getOBX());
            }

            // add PDF attachment as last obx 
            if (orNum == orderNums.length - 1) {
                var folder = FilenameUtils.getFullPath(etlConfig.getAliasesMap().get(task.getConnectionName()).getUrl()); 

                var file = folder + "result.PDF";

                if (FileUtils.fileExists(file)) {
                    var attachment = Base64.encodeBytes(FileUtils.getBytesFromFile(new File(file)), Base64.DONT_BREAK_LINES);

                    var newObservation = destOrderObservation.insertOBSERVATION(observations.length);

                    newObservation.getOBX().getObx1_SetIDOBX().setValue(observations.length + 1);
                    newObservation.getOBX().getObx2_ValueType().setValue("ED");
                    newObservation.getOBX().getObservationIdentifier().getCe1_Identifier().setValue("PDF");
                    newObservation.getOBX().getObservationIdentifier().getCe2_Text().setValue("PDF BASE64");
                    newObservation.getOBX().getObservationSubID().setValue(1);

                    var msg = "^^PDF^Base64^${ATTACH:" + attachment + "}";

                    newObservation.getOBX().insertObx5_ObservationValue(0).parse(msg);
                }   
            }    
        }  
    }  

}  

dataSet.setActualData(destMessage);
}

The part of the code below is responsible for reading the PDF from the processing (local) folder base64, encoding it, and attaching it to the reformatted message. The expected file name is result.PDF:

....
            // add PDF attachment as last obx 
            if (orNum == orderNums.length - 1) {
                var folder = FilenameUtils.getFullPath(etlConfig.getAliasesMap().get(task.getConnectionName()).getUrl()); 

                var file = folder + "result.PDF";

                if (FileUtils.fileExists(file)) {
                    var attachment = Base64.encodeBytes(FileUtils.getBytesFromFile(new File(file)), Base64.DONT_BREAK_LINES);

                    var newObservation = destOrderObservation.insertOBSERVATION(observations.length);

                    newObservation.getOBX().getObx1_SetIDOBX().setValue(observations.length + 1);
                    newObservation.getOBX().getObx2_ValueType().setValue("ED");
                    newObservation.getOBX().getObservationIdentifier().getCe1_Identifier().setValue("PDF");
                    newObservation.getOBX().getObservationIdentifier().getCe2_Text().setValue("PDF BASE64");
                    newObservation.getOBX().getObservationSubID().setValue(1);

                    var msg = "^^PDF^Base64^${ATTACH:" + attachment + "}";

                    newObservation.getOBX().insertObx5_ObservationValue(0).parse(msg);
                }   
            }    
....

Please also keep in mind that, according to the HL7 specs, the size of the attachment in the OBX segment is limited to 65 Kb.

10) Save and test the Flow.

Prerequisites for this Flow to work: the files result.hl7 and result.PDF exist in the processing (local) folder.
To verify that the Flow has been executed and has produced the correct results: check that the correct file was created in destination-results folder.

Step 5. Create a move files to processing Flow.

This Flow moves files from the source-results folder to both the processing (local) and processing (remote) folders. Files in the processing (remote) folder have the same names as the original files. Files in the processing (local) older will be renamed to result.

For this Flow, we will need the following Connections:

A Connection that points to the source-results folder.
A Connection that points to the processing (local) folder.
A Connection that points to the processing (remote) folder.

1. Start creating a Flow by clicking Add flow on the Flows window.

2. Type files into the search box and select File Management.

3. In the opened window add 4 transformations:

Move all the files from source-results to the processing (remote) folder.

1. Select a source (FROM) Connection which points to the source-results folder.

2. Enter *.* into the FROM field.

3. Select a destination (TO) Connection which points to the processing (remote) folder.

4. Enter *.* in the TO field.

5. Click MAPPING and select the Parameters tab.

6. Select Move for the Action. By selecting this action you are configuring the transformation to move files from the source-results to the processing (remote)folder.
Create the folder processing (local) if it does not exist.

1. Select source (FROM) and destination (TO) Connections that point to the processing (local) folder.

2. Enter processing in the FROM and TO fields.

3. Click MAPPING and select the Parameters tab.

4. Select Create Folder(s) as the Action. By selecting this action you are configuring the transformation to create a processing (local) folder if it does not exist.
Copy files from the processing (remote) to the processing (local) folder.

1. Select a source (FROM) Connection that points to the processing (remote) folder.

2. Enter *.* into the FROM field.

3. Select a destination (TO) Connection that points to the processing (local) folder.

4. Enter *.* into the TO field.

5. Click MAPPING and select the Parameters tab.

6. Select Copy as the Action. By selecting this action you are configuring the transformation to copy files from the processing (remote) to the processing (local) folder.
Rename files in the processing (local) to result.

1) Select a source (FROM) Connection that points to the processing (local) folder.

2) Enter *.* into the FROM field.

3) Select a destination (TO) Connection that points to the processing (local) folder.

4) Enter result.* into the TO field.

5) Click MAPPING and select the Parameters tab.

6) Select Rename as the Action. By selecting this action, you are configuring the transformation to rename files in the processing (local) folder to result.

4) Save and test the Flow.

A prerequisite for this Flow to succeed: the source files exist in the source-results folder.
To verify that the Flow has been executed and has produced the correct results: check to see that the source files were moved with the same name to the processing (remote) folder and were copied to the processing (local) folder with a different name: result.

Step 6. Create a clean up Flow.

This Flow deletes files from both the processing folders or, if there is an error, moves files to the failed folder.

1) Start creating a Flow by clicking Add flow on the Flow window.

2) Type files into the search box and select File Management.

3) In the opened window add 3 transformations:

Delete files from the processing (local) folder.

1) Select a source (FROM) Connection which points to the processing (local) folder.

2) Enter *.* into the FROM field.

3) Click MAPPING and select the Parameters tab.

4) Select Delete as the Action. By selecting this action you are configuring the transformation to delete files from the processing (local) folder.
Delete files from the processing (remote) folder.

1) Select a source (FROM) Connection which points to the processing (remote) folder.

2) Enter *.* into the FROM field.

3) Click MAPPING and select the Parameters tab.

4) Select Delete as the Action. By selecting this action you are configuring the transformation to delete files from the processing (remote) folder.
Move files to the failed folder.

1) Select a source (FROM) Connection which points to the processing (remote) folder.

2) Select a destination (TO) Connection which points to the failed folder.

3) Enter *.* into the FROM field.

4) Click MAPPING and select the Parameters tab.

5) Select Move for the Action. By selecting this action you are configuring the transformation to move files from the processing (remote) folder to the failed folder.

6) Select Execute if Error. By selecting this action you are configuring the transformation to move files from the processing (remote) folder to the failed folder in case of any error.

4) Save and test the Flow.

The prerequisite for this Flow to work: the file exists in the processing (local)and processing (remote) folders.
To verify that the Flow has been executed and has produced the correct results: check that the files were deleted from the processing (local) and processing (remote) folders.

Step 7. Put it all together

In the previous 6 steps, we created 3 Flows which:

Move files from the source-results to the 2 processing folders.
Reformat a file and copy it to the destination-results older.
Clean up by removing files from the processing folders, or moving files to the failed folder, if there was an error.

In order to put it all together, we will need to create a nested Flow.

Add the following Flows to the nested Flow:

1) The Flow created in Step 5.

2) The Flow created in Step 4.

3) The Flow created in Step 6.

Save and test the Flow.

A prerequisite for this Flow to work: the source file exists in the source-results folder.
To verify that the Flow has been executed and has produced the correct results: check that the correct file was created in the destination-results folder, and the original source files were deleted from the source-results folder.

Step 8. Schedule the Flow created in Step 7 to be executed periodically.

Articles in this section

Overview

Lab Order processing

Flow description

Source HL7 message

The reformatted destination HL7 message

Connections

Prerequisites

How an order is processed

The order Flow

Lab results processing

Flow description

Source HL7 message

The reformatted destination HL7 message

Connections

Prerequisites

How lab results are processed

The Flow

Related articles