Example gitlab file for docker in docker file with authentication

I ran into a problem in a special docker environment. The docker within this env doesn't allow downloading images from central registries. To bypass this I wrote a dind gitlab-ci.yml which starts a docker in docker image to download the required image and pushes the image into the custom registry. The dind needs to authenticate to push an image to the custom registry or to consume the image from the custom registry. 

Here ist a generic sample gitlab-ci.yml, which downloads and pushes an image. To run this you need the following ci vars configured within gitlab

IMAGE_NAME        : Name of the image to fetch i.e. trion/karma

IMAGE_VERSION   : Version of the image i.e. latest

DOCKER_AUTH_CONFIG: String with a valid docker auth config for your custom repo

CI_REGISTRY: Link to your docker registry

image: docker:20.10.24

variables:
    DOCKER_HOST: tcp://docker:2375
    DOCKER_TLS_CERTDIR: ""

services:
    - name: docker:20.10.24-dind
        entrypoint: ["dockerd-entrypoint.sh"]
        command: ["--insecure-registry", "custom.registry.mycomp.org:443"]

before_script:
    - docker info

# job for fetching any image in the var $IMAGE_NAME:$IMAGE_VERSION

deploy-generic-image:
    stage: build
    when: manual
    tags:
        - docker-in-docker
        - PROD
    before_script:
        - mkdir -p ~/.docker
        - echo $DOCKER_AUTH_CONFIG > ~/.docker/config.json
    script:
        - echo "Pulling $IMAGE_NAME:$IMAGE_VERSION"
        - docker pull $IMAGE_NAME:$IMAGE_VERSION
        - docker images
        - docker image tag $IMAGE_NAME:$IMAGE_VERSION $CI_REGISTRY/$IMAGE_NAME:$IMAGE_VERSION
        - docker images
        - docker push $CI_REGISTRY/$IMAGE_NAME:$IMAGE_VERSION



If you don't like the docker_auth_config you could also add the login info in the before_script section like that

before_script:
    - echo "$CI_JOB_TOKEN" | docker login -u "$CI_REGISTRY_USER" --password-stdin $CI_REGISTRY

But to consume the image within your gitlab-ci.yml you will need the docker_auth_config. 

Sample Jenkinsfile

This is a sample for a Jenkinsfile with following features in use:

• REST list parameter plugin usage for retrieving milestones from a gitlab project
• Parsing JSON from text parameter with Pipepline Utility Steps plugin to get the iid of a milestone
• Calling shell script with credentials
• Save passing credentials to shell script

pipeline {
  agent any
  parameters {
string(name: 'app', defaultValue: 'sample', description: 'Name of the app')

RESTList(
      name: 'MILESTONE',
      description: '',
      restEndpoint: 'https://git.gitlab.com/api/v4/projects/234/milestones',
      credentialId: 'CREDENTIAL_ID_IN_JENKINS',
      mimeType: 'APPLICATION_JSON',
      valueExpression: '$[*]',
      displayExpression: '$.name',
      cacheTime: 10,    // optional
      defaultValue: '', // optional
      filter: '.*',     // optional
      valueOrder: 'ASC' // optional
    )

    choice(name: 'sampleChoice', choices: 'True\nFalse', description: 'Well just a sample choice. First entry is defaultValue.')
    
  }
  stages {
stage('Prepare landing zone') {
steps{
checkout([$class: 'GitSCM', branches: [[name: '*/master']], extensions: [], userRemoteConfigs: [[credentialsId: 'CREDENTIAL_ID_IN_JENKINS', url: 'git@git.gitlab.com:sample/repository.git']]])
}  
}
    stage('Excute whatever you like') {
        
      steps {
script {
// parse the milestones retreived from gitlab
ms = readJSON returnPojo: true, text: env.MILESTONE
// save the milestone iid within the env
env.milestoneId =  ms.iid
}
withCredentials([string(credentialsId: 'CREDENTIAL_ID_IN_JENKINS', variable: 'accesToken')]) {
println accesToken: accesToken
println app: app
println sampleChoice: sampleChoice
println milestoneId: milestoneId
sh ('printenv')
dir('out') {
    sh ('pwd -P')
}
    sh ('echo $accesToken $gitLabProjectMode $app $sampleChoice $milestoneId')
}
      }
    }
  }
} 

Update to Spring > 2.5

 I've recently updated one of our spring boot apps. We've used Spring Boot 2.4 and I update to the latest available 2.6. After restarting the tests and test application I noticed a problem with the initialization of the h2 db. 

We tend to use the former spring datasource properties 

spring.datasource.schema=classpath:dev/schema.sql

spring.datasource.data=classpath:dev/data.sql

which are renamed into spring.sql.init.*. So I also renamed them, but nothing happened so far. Neither schema nor data.sql gets called during startup. After reading the changes I noticed a hint in the Init Docu that you should not use the init scripts with higher level db migration tools like liquibase and flyway.

We are using flyway and that was causing the issue. Form Spring Boot 2.5.1 on the init script will be ignored if you are using flyway or liquibase (spring boot doc).

So I changed the way we init the dev and test h2 db. I set the 

spring.sql.init.mode=never

and removed the schema.sql and data.sql. To initially create and fill our db with flyway I added to flyway snippets using the flyway hook for beforeMigrate. As we have to script I used the standard flyway notation for the two scripts like the following

beforeMigrate__01_schema.sql

beforeMigrate__02_data.sql

After that change fly way runs the two scripts before starting the migration when using the h2 db. I only added the two script to our flyway h2 snippets.

 

Self signed certificates and Oracle 19

Problem 

Starting with Oracle 19 you need to do some additional steps to allow Java calls within your Oracle 19 db to access self signed web services or other ssl/https resources. You'll notice the problem with error like the following while calling an https site from java within Oracle 19:

ORA-29532: Java-Aufruf durch nicht abgefangene Java-Ausnahme beendet: java.rmi.RemoteException: java.rmi.RemoteException:; nested exception is: 

                HTTP transport error: javax.xml.soap.SOAPException: java.security.PrivilegedActionException: javax.xml.soap.SOAPException: Message send failed: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Solution

Copy the existing cacerts file located in /home/oracle/database/javavm/jdk/jdk8/lib/security/cacerts

to /home/oracle/database/javavm/jdk/jdk8/lib/security/cacerts.alt

add your self signed cert with (how to get my cert from an existing web site)

keytool -importcert -trustcacerts \

    -keystore /home/oracle/database/javavm/jdk/jdk8/lib/security/cacerts.alt \

    -storepass changeit \

    -file my_ca_cert.crt -alias myrootca \

    -v -noprompt

Load the file into oracle with sqlplus

$ sqlplus / as sysdba

SQL> alter session set container = orclpdb;

SQL> exec dbms_java.loadjava('-schema SYS -grant PUBLIC -dirprefix /home/oracle/database/javavm/jdk/jdk8 /home/oracle/database/javavm/jdk/jdk8/lib/security/cacerts.alt')

or with the CLI Tool 

$ cd /home/oracle/database/javavm/jdk/jdk8

$ loadjava -user sys@db -v -schema SYS -grant PUBLIC  \

    /lib/security/cacerts.alt

Flyway spring.jpa.hibernate.ddl-auto=validate fails with Oracle synonyms

Using Flyway alongside with Hibernate is a nice matchmaker. Within one of the Spring Boot project I noticed that the flyway validate fails. Reason was a missing view. But looking deeper into the conf shows a synonym, which is handling the view. Doesn't appear within the dev env because the devs disabled the long running validate.

They got error messages like:

Schema-validation: missing table

Schema-validation: missing column

Easy to get rid of this within a spring boot app if you know how. Here is my thought on that:

spring.jpa.hibernate.ddl-auto=validate

spring.datasource.hikari.datasource-properties.includeSynonyms=true

spring.jpa.properties.hibernate.synonyms=true

will help Spring Boot to configure the hikari pool and the hibernate source to also use synonyms for the metadata exploration.

Take care to also add config for tomcat or any other pool, if you don't use the spring default hikari pool.

Windows Subsystem for Linux (WSL)

If you need a real linux shell on your Windows 10 system follow these steps

1. Ensure "Windows-Subsystem for Linux" is active on your machine with the following cmd in a admin power shell

Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux

2. Restart your machine if asked

3. Download the linux distribution of your choice via wget 

Invoke-WebRequest -Uri https://aka.ms/wsl-ubuntu-1604 -OutFile Ubuntu.appx –UseBasicParsing

or within you browser  

https://www.microsoft.com/en-us/p/debian/9msvkqc78pk6?activetab=pivot:overviewtab

or even from the MS App Store. 

You'll find a list of distributions here 

• Ubuntu 16.04 LTS
• Ubuntu 18.04 LTS
• OpenSUSE Leap 15
• OpenSUSE Leap 42
• SUSE Linux Enterprise Server 12
• SUSE Linux Enterprise Server 15
• Kali Linux
• Debian GNU/Linux
• Fedora Remix for WSL
• Pengwin
• Pengwin Enterprise
• Alpine WSL

4. Install distribution with

Add-AppxPackage .\Ubuntu.Appx

5. If you now search for Ubuntu in your start menu, you'll find your ready to rock linux console in windows

Appcelerator Titanium projects with IntelliJ

Developing Appcelerator mobile Apps is usually done within the Axway Appcelerator Studio. I'm using IntelliJ to develop most of my apps and here is the way how you could build and run you Appcelerator projects.

Open the project and create a package.json like this:

{
  "name": "my-app",
  "version": "1.0.0",
  "description": "Sample package.json for running titanium apps within IntelliJ.",
  "main": "index.js",
  "directories": {},
  "scripts": {
    "setup": "./node_modules/.bin/titanium sdk install 8.3.1.GA --default",
    "clean": "./node_modules/.bin/titanium clean",
    "build": "npm run build:android && npm run build:ios",
    "build:android": "./node_modules/.bin/titanium build -p android -b",
    "build:android:full": "npm run build:android",
    "build:ios": "./node_modules/.bin/titanium build -p ios -b",
    "build:ios:full": "npm run build:ios",
    "init:android": "npm install && npm run setup | $ANDROID_HOME/tools/bin/sdkmanager --licenses",
    "init:ios": "npm install",
    "android": "./node_modules/.bin/titanium build -p android -T emulator -C HUGO",
    "ios": "./node_modules/.bin/titanium build -p ios -T simulator -C HUGO",
    "download:android:sdk": "$ANDROID_HOME/tools/bin/sdkmanager \"platform-tools\" \"platforms;android-29\" \"build-tools;29.0.2\" \"emulator\" \"ndk-bundle\" \"system-images;android-29;google_apis_playstore;x86\"",
    "create:avd": "echo \"no\"| $ANDROID_HOME/tools/bin/avdmanager create avd -f -n Pixel_2_API_28 -k \"system-images;android-29;google_apis_playstore;x86\" -d \"pixel\"",
    "configure:avd": "for f in ~/.android/avd/*.avd/config.ini; do echo 'hw.keyboard=yes' >> \"$f\"; done",
    "prepare:env:android": "npm install && npm-run-all init:android download:android:sdk create:avd configure:avd"
  },
  "author": "Andre Dvorak",
  "license": "GPL",
  "homepage": "https://www.kambrium.net",
  "devDependencies": {
    "npm-run-all": "4.1.5"
  },
  "dependencies": {
    "alloy": "1.14.1",
    "titanium": "5.2.1"
  }
}

To start a device session with your new app just open the package.json an select

1. "setup" target
2. "android" target 

After the build you will be asked you for the emulator of choice. You could skip this by replacing HUGO with the name of your favourite emulator.

OAuth2 and Open ID connect

OAuth2 is a standard protocol for authorisation. It is a framework which delegates the user authentication to a service, which manages the user accounts. It provides flows for web, desktop and mobile applications.

https://oauth.net/2/

OpenID Connect is an extension of OAuth2. An OAuth2 server which implements OpenID connect is a so called OpenID provider (OP). The client of an OpenID connect server is called Relying Party (RP).
OpenID Connect offers the possibility to retrieve user profile information beside the access token defined within OAuth2. The user information is delivered within the payload of the id_token or within the access_token.
The following steps are the flow of the authorization code flow of an OP

1. The RP open the app and clicks login
2. The app starts an authorize request by opening the website which is defined within the authorization endpoint and specifies a redirect url
3. The user fills in username and password or any information the OP needs to authorise it's user
4. After the user click's continue on the login page the OP will redirect to the url specified in 2. and add an authorization code as a parameter to the redirect url
5. The app fetches the authorization code and calls the token endpoint with the grant_type "authorization_code" to obtain an access token
6. The OP will reply with an access token, refresh token and a lot of other field defined in oauth2 spec
7. The app could now use the access token to authorize the logged in user
8. Within the access token or as a separat id token the app could extract user profile information delivered by the OP

[ERROR] Unable to find suitable Xcode install

Problem
I was unable to build a titanium appcelerator based app on an iMac. My target was ios. So it seems to me that something must be missing within my xcode install.

Solution
Even if they are installed, configure your xcode cmd line tools. Go to

Xcode->Preferences->Locations

and check that the Command Line Tools are visible like in the following screen

JUnit5 or TestNG

Both frameworks are full of testing features. The list of features is quite similar in both frameworks. Here are a few links you might check out the list of relevant features:

https://www.baeldung.com/junit-vs-testng

https://www.stickyminds.com/article/junit-vs-testng-choosing-framework-unit-testing

https://www.guru99.com/junit-vs-testng.html

I choose TestNG because of a feature which JUnit5 is currently missing

Group Test

This feature let me group my i.e. integration test together and let them run in my master build. With TestNG I could also do a before or after group and initialise my test group or cleanup after group run.

With JUnit5 we got something called Tags. Tags are good for grouping tests together. Let's see if they implement more in this direction in future.

spyOn static Methods in jasmine tests

If you want to spy on a static method within your jasmine tests, you could do a simple spy like

beforeAll(async() =>{
  spyOn(UserService, "staticMethod").and.returnValue(true);
});

I'll tend to do this within beforeAll to avoid having problems when doing this in beforeEach. Within beforeEach you might end up in error messages like 

Error: <spyOn> : staticMethod has already been spied upon

    at <Jasmine>

    at UserContext.<anonymous> src/app/shared/components/result-detail/sample.component.spec.ts:74:5)

    at ZoneDelegate.webpackJsonp../node_modules/zone.js/dist/zone.js.ZoneDelegate.invoke node_modules/zone.js/dist/zone.js:388:1)

    at ProxyZoneSpec.webpackJsonp../node_modules/zone.js/dist/proxy.js.ProxyZoneSpec.onInvoke node_modules/zone.js/dist/proxy.js:128:1)

    at ZoneDelegate.webpackJsonp../node_modules/zone.js/dist/zone.js.ZoneDelegate.invoke node_modules/zone.js/dist/zone.js:387:1)

    at Zone.webpackJsonp../node_modules/zone.js/dist/zone.js.Zone.run node_modules/zone.js/dist/zone.js:138:1)

    at runInTestZone node_modules/zone.js/dist/jasmine-patch.js:145:1)

    at UserContext.<anonymous> node_modules/zone.js/dist/jasmine-patch.js:160:1)

Jasmine runner TypeError 'next' in Angular5 Tests

I stumbled upon an error, which a karma runner gives me in some of our jasmine tests:

TypeError: Cannot read property 'next' of undefined

    at <Jasmine>

    at Function.continuer node_modules/q/q.js:1278:1)

    at UserContext.<anonymous> node_modules/q/q.js:1305:1)

    at ZoneDelegate.webpackJsonp../node_modules/zone.js/dist/zone.js.ZoneDelegate.invoke node_modules/zone.js/dist/zone.js:388:1)

    at ProxyZoneSpec.webpackJsonp../node_modules/zone.js/dist/proxy.js.ProxyZoneSpec.onInvoke node_modules/zone.js/dist/proxy.js:128:1)

    at ZoneDelegate.webpackJsonp../node_modules/zone.js/dist/zone.js.ZoneDelegate.invoke node_modules/zone.js/dist/zone.js:387:1)

    at Zone.webpackJsonp../node_modules/zone.js/dist/zone.js.Zone.run node_modules/zone.js/dist/zone.js:138:1)

    at runInTestZone node_modules/zone.js/dist/jasmine-patch.js:145:1)

    at UserContext.<anonymous> node_modules/zone.js/dist/jasmine-patch.js:160:1)

    at <Jasmine>

I couldn't find something wrong with the test itself, but I noticed an strange import:

[async] from "q";

This was the reason for the strange error message. Correct import ist

[async] from "@angular/core/testing";

How to transform a Array to a map in Javascript/Typescript

This might be an easy one but I thought it is worth saving it here.
Let's assume you want to create a HashMap in typescript or javascript with http status codes and corresponding messages. This might be useful within a frontend error handler.
Here is your error-messages.json

[


  {"code": 403,    "message": "You are not permitted to use this action."  },


  {"code": 400,    "message": "The request was incomplete or wrong."  },  


  {"code": 500,    "message": "A general error has occured."  }


]

here is how you could use it in your code:

// Map with error codes


  errorCodeMapping: Map<number, string> = new Map<number, string>();


// read error codes as json map


let array = Array.from(require("./error-messages.json"));


// map array to errorCode Map


this.errorCodeMapping = new Map(array.map((i: ErrorMessage): [number, string] => [i.code, i.message]));


// get a message for an error code


let msg: string = this.errorCodeMapping.get(403);

Karma runner Disconnected (1 times), because no message in 10000 ms.

We are using karma as the test runner within our angular 5 application. It does a pretty good job when it's time to build regression tests for our frontend. After a while using it I noticed an error during the coverage test run.

> xxxx-frontend@0.0.0 coverage C:\workspaces\andre\Intellij\xxxx\frontend
> ng test --cc --single-run
05 03 2018 08:04:47.006:INFO [karma]: Karma v2.0.0 server started at http://0.0.0.0:9876/
05 03 2018 08:04:47.009:INFO [launcher]: Launching browser Chrome with unlimited concurrency
05 03 2018 08:04:47.016:INFO [launcher]: Starting browser Chrome
05 03 2018 08:05:06.109:INFO [Chrome 64.0.3282 (Windows 7.0.0)]: Connected on socket fwQIL6cCt7q6VM0sAAAA with id 76897689
05 03 2018 08:05:12.110:WARN [Chrome 64.0.3282 (Windows 7.0.0)]: Disconnected (1 times), because no message in 6000 ms.
Chrome 64.0.3282 (Windows 7.0.0) ERROR
  Disconnected, because no message in 10000 ms.

Sometimes this happens because we got an error wihtin the test cases. After writing a lot of test cases I noticed that the execution of all test cases took about 11 sec. So I increased our timeout with setting

browserNoActivityTimeout: 60000,

within the karma.conf. That fixed the issue.

How to organize imports in Intellij when saving a file

I always forget this nice plugin. Do the following if you want your Intellij to organize imports or reformat code while saving a file

1. Install Plugin "Save Actions" within your IntelliJ
2. Perform your settings within the "save actions" settings page

Where are the ear or war files stored in a jboss container?

I was searching where jboss stores an ear or war files once you deployed it via the cli cmd line. Here is my finding on that:

• The uploaded war or ear ist stored within a file called $jboss.server.base.dir/standalone/data/content/xx/xxyyyyyyyyy/content
• xx is a two char directory name within the default data folder
• yyyyyyyyyyy is a hash value of your ear or war
• You will find the xxyyyyyyy key as the sha1 value of your deployed ear or war within your standalone.xml config under the deployment section

Within the $jboss.server.base.dir/standalone/tmp you might find the cache of our currently running jboss. If you delete tmp after stopping jboss it will reinstall the ingredients from data. If you delete data you are screwed and you have to remove the ear or war entries from your standalone.xml or start jboss with --admin-only to redeploy the war or ears with the console.

Take care when using @Injectmocks

Well @Injectmocks is nice and evil in one tag. Assume we have this service 












@Service
public class SampleService {
    private final Logger LOG = Logger.getLogger(SampleService.class.getName());

    @Autowired
    private SampleDependency1   dependency1;

    public Long sampleMethod() {
        LOG.info("Calling sampleMethod");
        Long l = dependency1.calculateValue();
        LOG.info("l = " + l);
        return  l;
    }
}
and the corresponding test
@RunWith(SpringRunner.class)
@SpringBootTest
public class InjectmocksApplicationTests {
    @Mock
    private SampleDependency1 dependency1;

    @InjectMocks
    private SampleService service = new SampleService();

    @Test
    public void contextLoads() {
        when(dependency1.calculateValue()).thenReturn(null);
        final Long l = service.sampleMethod();
        Assert.isNull(l, "well l should be null");
    }
}

Ok, should work and will call our service with the injected mock for dependency1. Now lets add a second dependency like this:



@Service
public class SampleService {
    private final Logger LOG = Logger.getLogger(SampleService.class.getName());

    @Autowired
    private SampleDependency1   dependency1;

    @Autowired
    private SampleDependency2   dependency2;

    public Long sampleMethod() {
        LOG.info("Calling sampleMethod");
        Long l = dependency1.calculateValue();
        l= dependency2.calculateValue();
        LOG.info("l = " + l);
        return  l;
    }

This will compile, but running your test will result in 



java.lang.NullPointerException
 at net.kambrium.example.SampleService.sampleMethod(SampleService.java:23)





because you forgot to add dependency2 to your test class. To avoid this do




1. Use constructor wiring for your dependencies

2. Use @InjectMocks wisely and go searching on your tests for usage of the service you change and adjust the test cases





I personally prefer 1. as it always starts complaining at compile time. If your are using checkstyle you will see this warning, when using field injection:













Spring Team recommends: "Always use constructor based dependency injection in your beans. Always use assertions for mandatory dependencies"

Renew an Apple distribution certificate for dummies

Year after year you may need to refresh your distribution cert of your iOS app. These few steps are the short list, which should be done to renew an iOS Distribution Certificate for App Store distribution:

1. Go to developer.apple.com -> Account -> Certificates
2. Select Certificates->Production-> Add
3. Choose App Store and Ad Hoc
4. Open Keychain Access app and select Keychain Access > Certificate Assistant > Request a Certificate from a Certificate Authority as stated in the developer website
5. Fill out all values and don't forget to select "let me specify key pair information" this will ensure that a private key is specified in your generated cert
6. Save the cert request
7. Select continue and choose the saved cert request
8. This will generate you new distribution cert
9. Go to provisioning profile and select the one you're using to provision your app
10. Change the used distribution cert to the one you newly generated
11. Go back to xcode and refresh your certs within prefs->account
12. Download the newly generated provisioning profile if not already done

On any other machine you need to get access to the distribution profile including the private key to sign ios application. You need to export the distribution profile from the machine, which generates the profile as a p12 file. Just choose export within the key list and provide a password.
On the second machine you could then open the p12 file provide the password and that imports the dist cert including the private key into your local key store.

Docker cheat sheet

Start docker image and map a port from the docker image to a port on localhost

docker run -p 127.0.0.1:8806:3306 8538c205bee2

Starts image id 8538c205bee2 and maps internal port 3306 of that image to localhost port 8806.

Open a bash in a running docker image

docker exec -it 8538c205bee2 /bin/bash

Stop all running docker images

docker stop $(docker ps -a -q)

Remove/delete all running docker images

docker rm $(docker ps -a -q)

How to browser a remote docker repository

to browse a remote docker repository for available versions of a docker image you could use the docker API like this

curl https://mydockerrepo.de/v2/[IMAGE-NAME]/tags/list