CIP Testing

Revision History

Revision No	Date	Change description	Author	Reviewed by
001	2022-02-03	Draft mostly empty document	Yasin	To be reviewed by Kento Yoshida
002	2023-01-23	Added CIP Kernel testing reference	Dinesh K	To be reviewed by SWG members.
002	2023-07-31	Added details instead of references based on BV’s comment	Sai Ashrith	To be reviewed by SWG members

1. Introduction

CIP is a reference platform for product development which directly uses Debian packages for it’s core system. It uses upstream Linux kernel and makes configuration changes based on its requirement. Even though it is obvious that the re-used Debian packages and Linux Kernel are tested, still CIP has a determined testing WG which rigorously tests the Kernel on its reference hardware. The IEC 62443-4-2 standard security related configurations made in the metadata based on the installed Debian packages are also tested using some pre-written test cases written and maintained by the CIP Security work group.

2. Objective

The primary objective of this document is to list our current status regarding testing in CIP. It will be used as a basis to enhance our processes. The detailed steps used by the WG to test CIP Kernel and security related features in the isar-cip-core metadata are mentioned in this document to make the testing process repeatable for any CIP developer.

3. Scope

There is no IEC 62443 requirement directly answered by this document. This document currently does not explain the CIP-Core metadata testing process because it is still under discussion in CIP testing workgroup.

4. CIP Kernel testing

The CIP Kernel maintainers select and maintain Linux kernels for a very long time, so they need to be tested extensively with a lot of external and internal test cases.

1. CIP testing WG is working on creating an environment in which the tools, the configuration files, the tests, the kernel to be tested and the outcome of the tests are public under Open Source licenses so anyone can reproduce the outcome of the tests performed, as well as the environment. The best tests are selected to detect potential regressions and security issues in those backported patches applied to our current CIP kernel.

2. CIP testing WG runs tests based on centralized testing in which the developers can run the tests on kernel without having local access to a platform. Since the list of reference hardware is growing, the developers can trigger and run the tests 24/7.

3. CIP testing WG has already set up Continous integration testing to automatically test CIP kernel on CIP hardware using LAVA. Once a change in the currently maintained CIP Kernel branches is detected, then a build process starts using Gitlab CI/CD which provides binaries for a number of different Kernel configurations and then tested on CIP reference hardware using LAVA.

4. The configurations that are built and tested on the target hardware for available CIP kernel branches using the CI can be found in these configuration files.

5. CIP is working towards creating a test environment where not only the CIP Kernel but instead the complete system is tested (including base/core system).

CIP testing overview is pictorially represented here.

CIP Kernel test cases

CIP currently has three LAVA workers to which many targets i.e physical or virtual platforms are attached. Artifacts such as kernel, file system binaries, test applications and data are uploaded in Amazon web services S3 so that LAVA workers can access them for testing.

All the manual and automated tests are maintained in this repository in form of lava test definitions. The step-by-step procedure to submit a lava job by any developer is mentioned here.

CIP does not want to create duplicates of tests that are already published by other projects. Currently CIP re-uses the below mentioned tests as their LAVA jobs for to test CIP Kernel:

1. Healthcheck test to check whether a provided Kernel will boot on the target device.

2. Meltdown checker tests are run to verify whether the target device has meltdown vulnerabilities.

3. LTP tests are run during Kernel release because of long run times.

4. Tests such as cyclictest, hackbench, cyclicdeadline etc. are run to test the real time CIP kernels.

The complete list of Kernel configurations and their respective targets are listed in this table.

CIP Kernel test reports

CIP kernel test results are hosted on LAVA which gives the details of test cases, results and the relevant logs. The CIP Kernel tests are also run in Kernel CI. Here is a sample of CIP Kernel test run on kernelCI. There is also a CIP testing mailing list in which the results are published.

5. CIP Core Testing

CIP Core provides the utility to create CIP reference images by using meta-data maintained in isar-cip-core repository. Only the build verification on supported targets is implemented using gitlab CI/CD. Features such as Secureboot, software update, data encryption at rest etc. are not enabled for automated testing.

Since isar-cip-core uses Debian binary packages which are already tested rigorously in upstream, the user/developer just needs to test the metadata manually by cloning the source code, building the required flavor of image and then booting it on QEMU or some CIP reference hardware for testing the features manually. The manual steps which can be used by any user/developer to test the features are maintained in isar-cip-core repository as manuals.

CIP maintains some test cases which when installed and run in the security flavored image produce results which inform whether security requirements based on IEC 62443-4-2 are met.

CIP security test cases

The security test cases are maintained in cip-security-tests gitlab repository . These tests validate the security requirements defined for cyber security standard such as IEC-62443-4-2. The pre-configuration that needs to be done to the security image before running these tests is mentioned in this document.

CIP security test reports

After running the test cases, a result_file.txt will be generated which has the below mentioned format,

TC_CR1.1-RE1_1+pass+11
TC_CR1.11_1+pass+22
TC_CR1.11_2+pass+30
TC_CR1.1_1+pass+5
TC_CR1.1_2+pass+6
TC_CR1.3_1+pass+7
TC_CR1.3_2+pass+4
TC_CR1.3_3+pass+5
TC_CR1.4_1+pass+7
TC_CR1.5_2+pass+13
TC_CR1.5_3+pass+10
TC_CR1.7-RE1_1+pass+5
 :
 .

It mentions the requirement ID, status of the test and the time taken to execute the test case.