E2E tests were running an older version of pipelines ie v0.45.0.
Bumping it to use the latest version v0.51.0 so that current tasks in
catalog can be tested against latest version
Also remove pipelineresources check while doing installation of latest
version of pipelines
Signed-off-by: vinamra28 <jvinamra776@gmail.com>
PLATFORM env variable is introduced. This variable allows to run
tests for specific platform (for insance, "linux/s390x" or "linux/ppc64le").
If the environment variable is specified
- some tests are updated with different image values/other parameter
values, required for specific platform, using yq tool.
Custom values should be specified in the script file, separate for each
platform. In the current code script file for linux/s390x is proposed.
- only tasks, which have mention about corresponding platform support
in the annotation, will be tested.
If PLATFORM env variale is no specified, the behaviour of test scripts
will be similar to the one before the PR.
Signed-off-by: Yulia Gaponenko <yulia.gaponenko1@de.ibm.com>
The image is used in many catalog tests. To run the tests on many
platforms, the image should be the multi-arch one (at this moment -
linux/amd64, linux/ppc64le, linux/s390x, linux/arm64).
Signed-off-by: Yulia Gaponenko <yulia.gaponenko1@de.ibm.com>
When adding `fixtures` volume, we need to consider whether the task
already has a volumes section. If the volumes is already defined, then
we should add `fixtures` rather than replacing the whole volumes.
In e2e tests script podTemplate was getting added even if the yaml
manifest contains only ConfigMap which was failing the CI.
Append volumes in sidecar to mount fixtures ConfigMap instead of using podTemplate
as appending the podTemplate was conditional and there can be scenarios
where podTemplate might get appended at wrong location.
Signed-off-by: vinamra28 <vinjain@redhat.com>
- Check for libraries in a pip agnostic way
- allow for adding multiple sidecars with a reusable `add_sidecars` function
- fix overwriting sidecars issue for previous python
- make it more readable
Extend to test logic to be able to load the fixture reponse from a
file, so that we don't have to inline a long JSON doc into the fixture
config YAML. This is achieved by creating a config map with all the
files in the fixtures folder, and mounting that configmap as a volume
to the sidecar.
This allows improving the test for the github-add-comment task v0.4
Signed-off-by: Andrea Frittoli <andrea.frittoli@gmail.com>
If a task is being removed from the catalog then also the CI runs the
check for validating the YAML and thus fails if the task manifest is not
found.
This PR checks for the task directory whether it exists or not and in the case
where task directory is not present then the function doesn't checks the
validation of the YAML. Example if we remove the task knctl-deploy then
the CI would skip the check for that pull request.
Signed-off-by: vinamra28 <vinjain@redhat.com>
In some case we want to be able to redefine the KUBECONFIG commands, when other
CI needs to redefines it.
Signed-off-by: Chmouel Boudjnah <chmouel@redhat.com>
We were running previously the tests sequentially,
Since the catalogs tests are growing exponentially, let's add parallelization to
speed things up.
Signed-off-by: Chmouel Boudjnah <chmouel@redhat.com>
Current e2e script is only able to run integration tests on those PRs which include tests.
With this fix, tests will run on those PRs also in which only task yaml is changed
Signed-off-by: vinamra28 <vinjain@redhat.com>
We were not getting properly the versions when there was files like OWNER in there
so make sure to grab only the directory
Signed-off-by: Chmouel Boudjnah <chmouel@redhat.com>
We are detecting the task that only has been modified or added with the help of
git diff against the 'main' branch.
If we set and define the variable TEST_RUN_ALL_TESTS it will force running all
the tests and not just the modified ones.
Signed-off-by: Chmouel Boudjnah <chmouel@redhat.com>
We temporary disable the debug output while on the loop so we don't have the
gazillions debug messages while digging into the log. We have added the 'timing'
when it start and ends so we can easily spots the very slow test.
Aditionally we have introduced the new variable `CATALOG_TEST_SKIP_CLEANUP` if
set it will skip deleting the namespace. This is useful when running in
conjuncton with the `./test/run-test.sh` script to debug when there is a failure.
Signed-off-by: Chmouel Boudjnah <chmouel@redhat.com>
We were previously using the outdated git_clone version 0.1 in the
add_git_clone_task function.
So let's introduce a more generic function :
```
add_task ${task} ${version}
```
if version is 'latest' it will always install the latest version of the task.
Change all pre-apply-task-hook to use that function instead.
Signed-off-by: Chmouel Boudjnah <chmouel@redhat.com>
Add support for REST services testing, if we have a directory in the tests
directory called `fixtures` with a yaml file that contains a responder
rules (see github-add-comments task example) it will add a sidecar task from
https://github.com/chmouel/go-rest-api-test/ so the test can point to it.
Signed-off-by: Chmouel Boudjnah <chmouel@redhat.com>
This line queries twice to the github api in order to apply the tekton release yaml. It would increase readability if we just use the googleapi to fetch the yaml.
CI was not able to monitor some pipelineruns for tasks in catalog example maven task and was getting "Could not find a created taskrun or pipelinerun in maven-0-1+ breakit=True" which is not the expected behaviour so with this patch we will iterate over it 5 times and check for it's status and reason
Signed-off-by: vinamra28 <vinjain@redhat.com>
This will refactor the namespace name to have version of task
in namespace name as current logic may create issue in case
of multiple versions running in same namespace as pid will
remain same
Centralize all the the git-clone apply commands we had around the
pre-apply-task-hook to a central common function.
This will make easier to have the same git-clone task version used everywhere.
Add some documentation about the helper functions along the way.
Signed-off-by: Chmouel Boudjnah <chmouel@redhat.com>
This can grow the log pretty big with message that not a lot of people will care
about.
when we show the logs on failure, make sure we are not failing if we can't show
them. Sometime some container may be in "Terminated" state until k8 cleans them
up. (i.e: affinity-assistant) so no need to worry if those are failing/exiting before
showing the next container log.
Signed-off-by: Chmouel Boudjnah <chmouel@redhat.com>
The loop in the test runner script that was waiting for all TaskRuns to
complete had a bug where it would break out of the loop as soon as any one of
the TaskRuns completed. The test runner then tries to print the logs for all
TaskRuns (which includes some Running ones) which will sometimes fail e.g. if
the TaskRun's pod is in a PodInitializing state.
Fixes#379
Signed-off-by: Dibyo Mukherjee <dibyo@google.com>
Test would be failing if there is a directory but no yaml template in there,
which in reality we should not have this happening but it happens sometime.
let's make sure we skip the test if there is no yaml files then.
Signed-off-by: Chmouel Boudjnah <chmouel@redhat.com>
Immediately after our test runner submits test TaskRuns / PipelineRuns
it checks that those Runs appear in `kubectl get` output. In a recent
PR to add a kythe-go Task (#301) the test runner would fail at this
point. Checking locally I was able to reproduce the issue. It looks
like it can take a small amount of time for submitted resources to
show up in `kubectl get` output.
For other types of test failures the test runner gives a grace period of
up to ten minutes. I did the same for the `kubectl get` check, allowing
the test run loop to run again in ten seconds if nothing appears in a
taskrun / pipelinerun list. Making that change resulted in the kythe-go
test proceeding normally and passing.
This commit updates the test runner to repeatedly check whether
TaskRuns / PipelineRuns have appeared in the `kubectl get` output.
We are currently looping forever without any safe counter, let's just do that,
Refactoring a bit so we reuse a common function for that,
Closes#156
Signed-off-by: Chmouel Boudjnah <chmouel@redhat.com>
- Move test functiosn to e2e-common
- Remove old serviceaccount things
- Allow override functions (or add) like the way we do with plumbing
functions (for downstream CIs)
Signed-off-by: Chmouel Boudjnah <chmouel@redhat.com>
