Add Taskrun e2e testing

We are adding a test directory with a collection of yamls to runs.

There is a hook system which allows to do some manipulation on task for the
tests (as we don't want to modify the main task). Or if we want to do some extra
pre-setup before setting the taskrun

For example on buildah/kaniko would add a registry as a sidecontainers on
the tasks so we can do our testings without having to upload it somewhere.

S2I is disabled too due of not working on other cluster than openshift.

Closes: #103

DEVELOPMENT.md

@@ -10,7 +10,7 @@ the following guidelines:
 * Submissions should be useful in real-world applications.
 While this repository is meant to be educational, its primary goal is to serve
 as a place users can find, share and discover useful components.
-This is **not** a samples repo to showcase Tekton features, this is a collection 
+This is **not** a samples repo to showcase Tekton features, this is a collection
 * Submissions should follow established best practices.
 Tekton is still young so this is going to be a shifting goalpost, but here are
 some examples:
@@ -33,4 +33,48 @@ documentation and tests and a maintainer will review it.
 
 Over time we hope to create a scalable ownership system where community members
 can be responsible for maintaining their own submissions, but we are not there
-yet.
+yet.
+
+
+# End to end Testing
+
+There is two type of e2e tests launched on CI.
+
+The first one would just apply the yaml files making sure they don't have any
+syntax issues. Pretty simple one, it just basically check the syntax.
+
+The second one would do some proper functional testing, making sure the task
+actually **ran** properly.
+
+The way the functional tests works is that if you have a directory called
+`tests/` inside the task, it would start creating a random `Namespace`, apply
+the task and then every yaml files that you have in that `tests/` directory.
+
+Usually in these other yaml files you would have a yaml file for the
+test resources (`PipelineResource`) and a yaml files to run the tasks
+(`TaskRun`).
+
+Sometime you may need to be able to launch some scripts before applying the
+tested task or the other yaml files. Some may pre-setup something on the
+`Namespace` or have to do something externally or sometime you may even want to do
+some manipulation of the main `Task`.
+
+For example on the *image builders* tasks like `kaniko` or `jib` we want to
+upload the tasks to a registry to make sure it is actually built properly. To do
+so we [manipulate](kaniko/tests/pre-apply-task-hook.sh) with a python script the
+`Task` (something we don't want for everyone but only for the tests) to add a
+registry as a `Sidecar` and make sure that the `TaskRun` set the parameters to
+upload there. Simple and straightforward no need to upload to an external image
+registry provider having to setup the tokens and deals with the side effects...
+
+There is two different scripts that are checked if present in the `scripts`,
+those scripts actually sourced via the `source` bash script, so you can output
+some environment variables to it that would be applied :
+
+1. **pre-apply-task-hook.sh**: Script to run before applying the task
+2. **pre-apply-taskrun-hook.sh**: Script to run before applying the taskruns or other yaml files.
+
+What can you run from those scripts is whatever defined in the test-runner
+image, if you need to have another binary available feel free to make a PR to this Dockerfile :
+
+https://github.com/tektoncd/plumbing/blob/master/prow/images/test-runner/Dockerfile

buildah/tests/pre-apply-task-hook.sh

@@ -0,0 +1,11 @@
+#!/bin/bash
+
+# - TMPF is the temporary file where the YAML is processed.
+# - We are duplicating the temporary file since we cannot do the manipulation in
+#   place (i.e: reading and writing at the same time)
+cp ${TMPF} ${TMPF}.read
+
+# Add an internal registry as sidecar to the task so we can upload it directly
+# from our tests withouth having to go to an external registry.
+cat ${TMPF}.read | python -c 'import yaml,sys;data=yaml.load(sys.stdin.read());data["spec"]["sidecars"]=[{"image":"registry", "name": "registry"}];print(yaml.dump(data, default_flow_style=False));' > ${TMPF}
+rm -f ${TMPF}.read

buildah/tests/resources.yaml

@@ -0,0 +1,22 @@
+---
+apiVersion: tekton.dev/v1alpha1
+kind: PipelineResource
+metadata:
+  name: nocode
+spec:
+  type: git
+  params:
+    - name: revision
+      value: master
+    - name: url
+      value: https://github.com/kelseyhightower/nocode
+---
+apiVersion: tekton.dev/v1alpha1
+kind: PipelineResource
+metadata:
+  name: image
+spec:
+  type: image
+  params:
+    - name: url
+      value: localhost:5000/nocode

buildah/tests/run.yaml

@@ -0,0 +1,21 @@
+---
+apiVersion: tekton.dev/v1alpha1
+kind: TaskRun
+metadata:
+  name: buildah-run
+spec:
+  taskRef:
+    name: buildah
+  outputs:
+    resources:
+      - name: image
+        resourceRef:
+          name: image
+  inputs:
+    resources:
+      - name: source
+        resourceRef:
+          name: nocode
+    params:
+      - name: TLSVERIFY
+        value: "false"

jib-maven/jib-maven.yaml

@@ -11,6 +11,9 @@ spec:
     - name: CACHE
       description: The name of the volume for caching Maven artifacts and base image layers
       default: empty-dir-volume
+    - name: INSECUREREGISTRY
+      description: Wether to allow insecure registry
+      default: "false"
     resources:
     - name: source
       type: git
@@ -18,7 +21,6 @@ spec:
     resources:
     - name: image
       type: image
-
   steps:
   - name: build-and-push
     image: gcr.io/cloud-builders/mvn
@@ -27,6 +29,7 @@ spec:
     - compile
     - com.google.cloud.tools:jib-maven-plugin:build
     - -Duser.home=/builder/home
+    - -Djib.allowInsecureRegistries=$(inputs.params.INSECUREREGISTRY)
     - -Dimage=$(outputs.resources.image.url)
     workingDir: /workspace/source/$(inputs.params.DIRECTORY)
     volumeMounts:

jib-maven/tests/pre-apply-task-hook.sh

@@ -0,0 +1,11 @@
+#!/bin/bash
+
+# - TMPF is the temporary file where the YAML is processed.
+# - We are duplicating the temporary file since we cannot do the manipulation in
+#   place (i.e: reading and writing at the same time)
+cp ${TMPF} ${TMPF}.read
+
+# Add an internal registry as sidecar to the task so we can upload it directly
+# from our tests withouth having to go to an external registry.
+cat ${TMPF}.read | python -c 'import yaml,sys;data=yaml.load(sys.stdin.read());data["spec"]["sidecars"]=[{"image":"registry", "name": "registry"}];print(yaml.dump(data, default_flow_style=False));' > ${TMPF}
+rm -f ${TMPF}.read

jib-maven/tests/resources.yaml

@@ -0,0 +1,22 @@
+---
+apiVersion: tekton.dev/v1alpha1
+kind: PipelineResource
+metadata:
+  name: console-java-simple
+spec:
+  type: git
+  params:
+    - name: revision
+      value: master
+    - name: url
+      value: https://github.com/che-samples/console-java-simple
+---
+apiVersion: tekton.dev/v1alpha1
+kind: PipelineResource
+metadata:
+  name: image
+spec:
+  type: image
+  params:
+    - name: url
+      value: localhost:5000/tekton-pipelines/console-java-simple

jib-maven/tests/run.yaml

@@ -0,0 +1,21 @@
+---
+apiVersion: tekton.dev/v1alpha1
+kind: TaskRun
+metadata:
+  name: jib-maven-run
+spec:
+  taskRef:
+    name: jib-maven
+  outputs:
+    resources:
+      - name: image
+        resourceRef:
+          name: image
+  inputs:
+    resources:
+      - name: source
+        resourceRef:
+          name: console-java-simple
+    params:
+      - name: INSECUREREGISTRY
+        value: "true"

kaniko/kaniko.yaml

@@ -37,5 +37,3 @@ spec:
     - --dockerfile=$(inputs.params.DOCKERFILE)
     - --context=/workspace/source/$(inputs.params.CONTEXT) # The user does not need to care the workspace and the source.
     - --destination=$(outputs.resources.image.url)
-    securityContext:
-      privileged: true

kaniko/tests/pre-apply-task-hook.sh

@@ -0,0 +1,11 @@
+#!/bin/bash
+
+# - TMPF is the temporary file where the YAML is processed.
+# - We are duplicating the temporary file since we cannot do the manipulation in
+#   place (i.e: reading and writing at the same time)
+cp ${TMPF} ${TMPF}.read
+
+# Add an internal registry as sidecar to the task so we can upload it directly
+# from our tests withouth having to go to an external registry.
+cat ${TMPF}.read | python -c 'import yaml,sys;data=yaml.load(sys.stdin.read());data["spec"]["sidecars"]=[{"image":"registry", "name": "registry"}];print(yaml.dump(data, default_flow_style=False));' > ${TMPF}
+rm -f ${TMPF}.read

kaniko/tests/resources.yaml

@@ -0,0 +1,22 @@
+---
+apiVersion: tekton.dev/v1alpha1
+kind: PipelineResource
+metadata:
+  name: nocode
+spec:
+  type: git
+  params:
+    - name: revision
+      value: master
+    - name: url
+      value: https://github.com/kelseyhightower/nocode
+---
+apiVersion: tekton.dev/v1alpha1
+kind: PipelineResource
+metadata:
+  name: image
+spec:
+  type: image
+  params:
+    - name: url
+      value: localhost:5000/kaniko-nocode

kaniko/tests/run.yaml

@@ -0,0 +1,21 @@
+---
+apiVersion: tekton.dev/v1alpha1
+kind: TaskRun
+metadata:
+  name: kaniko-run
+spec:
+  taskRef:
+    name: kaniko
+  outputs:
+    resources:
+      - name: image
+        resourceRef:
+          name: image
+  inputs:
+    resources:
+      - name: source
+        resourceRef:
+          name: nocode
+    params:
+      - name: EXTRA_ARGS
+        value: "--skip-tls-verify"

s2i/tests/pre-apply-task-hook.sh

@@ -0,0 +1,12 @@
+#!/bin/bash
+
+# - TMPF is the temporary file where the YAML is processed.
+# - We are duplicating the temporary file since we cannot do the manipulation in
+#   place (i.e: reading and writing at the same time)
+# - Here we simply add a sidecar to our task so we can upload there.
+cp ${TMPF} ${TMPF}.read
+
+# Add an internal registry as sidecar to the task so we can upload it directly
+# from our tests withouth having to go to an external registry.
+cat ${TMPF}.read | python -c 'import yaml,sys;data=yaml.load(sys.stdin.read());data["spec"]["sidecars"]=[{"image":"registry", "name": "registry"}];print(yaml.dump(data, default_flow_style=False));' > ${TMPF}
+rm -f ${TMPF}.read

s2i/tests/resources.yaml

@@ -0,0 +1,22 @@
+---
+apiVersion: tekton.dev/v1alpha1
+kind: PipelineResource
+metadata:
+  name: nodejs-s2i-example
+spec:
+  type: git
+  params:
+    - name: revision
+      value: master
+    - name: url
+      value: https://github.com/chmouel/nodejs-health-check
+---
+apiVersion: tekton.dev/v1alpha1
+kind: PipelineResource
+metadata:
+  name: image
+spec:
+  type: image
+  params:
+    - name: url
+      value: localhost:5000/nodejs-health-check-tekton

s2i/tests/run.yaml

@@ -0,0 +1,23 @@
+---
+apiVersion: tekton.dev/v1alpha1
+kind: TaskRun
+metadata:
+  name: s2i-run
+spec:
+  taskRef:
+    name: s2i
+  outputs:
+    resources:
+      - name: image
+        resourceRef:
+          name: image
+  inputs:
+    resources:
+      - name: source
+        resourceRef:
+          name: nodejs-s2i-example
+    params:
+      - name: BUILDER_IMAGE
+        value: nodeshift/centos7-s2i-nodejs:latest
+      - name: TLSVERIFY
+        value: "false"

test/e2e-tests.sh

@@ -16,29 +16,106 @@
 
 source $(dirname $0)/../vendor/github.com/tektoncd/plumbing/scripts/e2e-tests.sh
 
+TMPF=$(mktemp /tmp/.mm.XXXXXX)
+clean() { rm -f ${TMPF}; }
+trap clean EXIT
+
+
 # Setup a test cluster.
-[[ -z ${LOCAL_CI_RUN} ]] && initialize $@
+[[ -z ${LOCAL_CI_RUN} ]] && {
+    # Initialize cluster
+    initialize $@
 
-# Install the latest Tekton CRDs.
-kubectl apply --filename https://storage.googleapis.com/tekton-releases/latest/release.yaml
+    # Install the latest Tekton CRDs.
+    kubectl apply --filename https://storage.googleapis.com/tekton-releases/latest/release.yaml
+}
 
-# Allow ignoring some tests, space separated, should be the basename of the test
-# i.e: kn-deployer
-IGNORES=${IGNORES:-}
+# You can ignore some yaml tests by providing the TEST_YAML_IGNORES variable
+# with the test name separated by a space, for example:
+#
+# TEST_YAML_IGNORES="kaniko s2i"
+#
+# will ignore the kaniko and s2i tests,
+#
+TEST_YAML_IGNORES=${TEST_YAML_IGNORES:-""}
+
+# Allow ignoring some yaml tests, space separated, should be the basename of the test
+#
+# Disabled by default:
+#
+# s2i: task is not working by default on vanilla kubernetes (ie not openshift) so disabling it here
+TEST_TASKRUN_IGNORES=${TEST_TASKRUN_IGNORES:-"s2i"}
+
+# Specify the default serviceAccount
+SERVICE_ACCOUNT=${SERVICE_ACCOUNT:-default}
 
 set -ex
 set -o pipefail
-# Validate that all the Task CRDs in this repo are valid by creating them in a NS.
-readonly ns="task-ns"
-kubectl create ns "${ns}" || true
-for f in $(find ${REPO_ROOT_DIR} -maxdepth 2 -name '*.yaml'); do
-    skipit=
-    for ignore in ${IGNORES};do
-        [[ ${ignore} == $(basename $(echo ${f%.yaml})) ]] && skipit=True
+
+function test_yaml_can_install() {
+    # Validate that all the Task CRDs in this repo are valid by creating them in a NS.
+    readonly ns="task-ns"
+    kubectl create ns "${ns}" || true
+    for runtest in $(find ${REPO_ROOT_DIR} -maxdepth 2 -name '*.yaml'); do
+        skipit=
+        for ignore in ${TEST_YAML_IGNORES};do
+            [[ ${ignore} == $(basename $(echo ${runtest%.yaml})) ]] && skipit=True
+        done
+        [[ -n ${skipit} ]] && break
+        echo "Checking ${runtest}"
+        kubectl -n ${ns} apply -f <(sed "s/namespace:.*/namespace: task-ns/" "${runtest}")
     done
-    [[ -n ${skipit} ]] && break
-    echo "Checking ${f}"
-    kubectl -n ${ns} apply -f <(sed "s/namespace:.*/namespace: task-ns/" "${f}")
-done
+}
+
+function test_task_creation() {
+    for runtest in */tests;do
+        testname=${runtest%%/*}
+        tns="${testname}-$$"
+        skipit=
+
+        for ignore in ${TEST_TASKRUN_IGNORES};do
+            [[ ${ignore} == ${testname} ]] && skipit=True
+        done
+        [[ -n ${skipit} ]] && continue
+
+        kubectl create namespace ${tns}
+
+        # Install the task itself first
+        for yaml in ${testname}/*.yaml;do
+            sed "s/%SERVICE_ACCOUNT%/${SERVICE_ACCOUNT}/g" < ${yaml} > ${TMPF}
+            [[ -f ${testname}/tests/pre-apply-task-hook.sh ]] && source ${testname}/tests/pre-apply-task-hook.sh
+            cp ${TMPF} /tmp/t.yaml
+            kubectl -n ${tns} create -f ${TMPF}
+        done
+
+        # Install resource and run
+        for yaml in ${runtest}/*.yaml;do
+            sed "s/%SERVICE_ACCOUNT%/${SERVICE_ACCOUNT}/g" < ${yaml} > ${TMPF}
+            [[ -f ${testname}/tests/pre-apply-taskrun-hook.sh ]] && source ${testname}/tests/pre-apply-taskrun-hook.sh
+            cp ${TMPF} /tmp/tr.yaml
+            kubectl -n ${tns} create -f ${TMPF}
+        done
+
+        while true;do
+            status=$(kubectl get -n ${tns} tr --output=jsonpath='{.items[*].status.conditions[*].status}')
+            reason=$(kubectl get -n ${tns} tr --output=jsonpath='{.items[*].status.conditions[*].reason}')
+            [[ ${status} == "ERROR" || ${reason} == Failed || ${reason} == CouldntGetTask ]] && {
+                echo "FAILED: ${testname} task has failed to comeback properly" ;
+                echo "--- TR Dump"
+                kubectl get -n ${tns} tr -o yaml
+                echo "--- Container Logs"
+                kubectl get pod -o name -n ${tns}|xargs kubectl logs --all-containers -n ${tns}
+                exit 1
+            }
+            [[ ${status} == True ]] && { echo "SUCCESS: ${testname} taskrun has successfully executed" ; break ;}
+            sleep 5
+        done
+
+        kubectl delete ns ${tns}
+    done
+}
+
+test_yaml_can_install
+test_task_creation
 
 success