DXL and Agent Import Support in domino_java_tools

[JoelProminic]

From @JustinProminic,

action items:

1. new java file called CreateDatabase <database_name> <<<--- NOT BASED UPON ODP format -- this is just a simple session.CreateDatabase command -- 1 line of code

2. add the DXLImporter java standalone code to import <dxl_import.properties>"

3. add clear instructions to welcome.txt or shi github documentation showing how to GET to this directory and actually run a gradle build that includes the following in order:

   a) CheckNotesUser.java -> run it to confirm Notes ID operation phase 1
   b) CheckDatabase -> run it to confirm NSF operation phase 2
   c) NEW agent .java that is "hello world" to compile and build a JAR
   d) compile and build DXLImporter
   e) use DXLImporter to import the "hello world" agent into the CreateDatabase that was made

4. Mark will document how to add a github runner into the S.H.I. instance. it will run as a background systemd task looking for build instructions from GitHub

5. Adrian now should be able to use GitHub to refine the "hello world" java agent and have it result in a workable runnable agent that is accessible via http:// call to the local S.H.I. domino server after simply re-running step 3e above after each build

6. lower priority but someone please find the simple "javac" based compilation scripts for each of the .java files above -- or at least for CheckDatabase, CheckNotesUser, CreateNamesDatabase, and the servlet example which is in a different directory above. I'm not suggesting abandoning Gradle but I just want something simple to run that I understand to compile the .java files into .jar's
   --- MARK FOUND THESE and they just need it added back to the S.H.I. installer:

https://github.com/DominoVagrant/demo/blob/220a7737535a73643f96e339cc25c28a623099c6/dist-build/build-notesjava-standalone-example-apps.bsh

GOALS:
--> Self-building Super.Human.Portal complete NSF artifact necessary so that Adrian and Piotr can take over release of Super.Human.Portal to the point that it is delivered into Genesis.Directory and able to be tested through subsequent fresh S.H.I. provisioned Domino servers. That solves half of the problem related to SHP release, but not yet the update problem which is the other half.

--> Lays the foundation so that all of the myriad of agents that Joel has created over the years, and which MUST be made into GitHub so that everyone has access to them AND the AI assistants can start helping with basic recurring problems such as JSON enabling the XML ones, as one minor example. This system would prove that all of the agents and the supporting databases that MyAccount, GBAuth, and other systems use to "reach into" core Prominic databases are able to be treated closer to a more modern DevOps scenario. Including but not limited to:

-- FlexAgents.nsf

-- FlexInput.nsf
-- FlexSupportAgents.nsf... etc

Alternate writeup showing the flow for a database test

create a database

do the simplest possible test of importing DXL of a super simple form

Bing-Joel DXLImporter Gradle thing of a super simple hello world java agent that can be tested from CURL as deployed to the local SHI running Domino server

Said above hello world agent present in example GitHub respository that can have its artifact generated on demand by on demand github runner and deployed to same NSF reference above as when the commands were run locally

So basically, we are trying to get to the point where a user can:

1. Create an SHI instance
2. Setup a self-hosted runner on the instance and connect it to GitHub organization
3. Clone a GitHub project
4. Update the DXL and Java Source
5. Commit and push the code, triggering a GitHub action
6. The GitHub action will run on the self-hosted runner
7. The user will then be able to open the database with their configured code.

To support this, we want to add new applications to domino_java_tools like this:

# Create an empty database with a minimal ACL
java -jar CreateDatabase.jar HelloWorld.nsf

# Import arbitrary XML to the target database
java -jar DXLImport.jar HelloWorld.nsf ExampleForm.dxl

# Import an agent using Bing's DXLImporter
# Note that it may be simpler to build and import as the same step, using this template:  https://github.com/Moonshine-IDE/DXLImporter-Gradle-Demo/tree/master/dxlLib
java -jar ImportAgent.jar HelloWorld.nsf HelloWorld.jar HelloWorld.properties

[JoelProminic]

@JustinProminic also wants to bring back the bash script that can build the JARs in domino_java_tools. The latest copy we found was here.

I updated this to work with the domino_java_tools directory:

build-notesjava-standalone-example-apps.bsh.zip

On Monday, I'll plan to create a branch with this script and the new Java applications.

[JoelProminic]

I have the initial implementation done for CreateDatabase and DXLImport. I have a DXL example at dxl/HelloWorld.dxl

I committed my work in this branch: features/22_dxl_importer

I found I needed to specify the server name in the command options. Using "" for the server creates the database in /local/notesjava

# build jars using Justin's script
./build-notesjava-standalone-example-apps.bsh

**# Create an empty database with a minimal ACL
java -jar CreateDatabase2.jar test-dxl-import.TESTDXL.com/TESTDXL HelloWorld.nsf

# Import arbitrary XML to the target database
java -jar DXLImport2.jar test-dxl-import.TESTDXL.com/TESTDXL HelloWorld.nsf dxl/HelloWorld.dxl
**

Agent imports are pending. Since it has additional dependencies, the commands will be a little more complicated.

[JoelProminic]

I have the GitHub action working as a proof of concept. For now, this relies on the Gradle scripts I created for DXL Importer:

1. Create a database. If it already exists, reuse the same database. We could recreate the database instead.
2. Import the specified DXL file(s). This will overwrite existing design
3. Import the agents with gradle clean importAll
4. The use can now start the instance on their local machine

The user experience is:

1. Update the agents or DXL files in the repository
2. Commit the changes
3. Wait for the GitHub action to complete
4. Open the agents in a browser, or open the generated database in a Notes client (or Nomad)

Unfortunately, had some problems with the Domino API. At one point, the Domino agents were hanging in the GitHub actions until I restarted the SHI instance. The commands did not have hang when I ran them manually. I'll watch for this to trigger again.

I can show a demo for @JustinProminic to see if he wants any changes to the workflow.

Remaining tasks:

• Optional: Create a standalone AgentImport application. Alternatively, rewrite this to work with the local run_dxl_import.sh script, but this also uses the Gradle command.
• Decide on a location for the GitHub Action template. For now I used a private project so that I could freely experiment with it
• Update the Ansible scripts to copy the new applications to the expected location and run tests.
• Documentation for the full process.

[JoelProminic]

I have been fighting with errors when running with the service:

• environment setup issues. I can work around this with some additional commands in the GitHub action, but I'd prefer to maintain this in the service to keep the GitHub Actions simple
• Hang after opening ID file. This needs more debugging.

The service is very simple:

$ cat /etc/systemd/system/actions.runner.JoelProminic-Test-DXL-Importer-GitHub-Actions.runner-test-standalone.service
[Unit]
Description=GitHub Actions Runner (JoelProminic-Test-DXL-Importer-GitHub-Actions.runner-test-standalone)
After=network-online.target

[Service]
ExecStart=/local/notesjava/actions-runner/runsvc.sh
User=java_user
WorkingDirectory=/local/notesjava/actions-runner
KillMode=process
KillSignal=SIGTERM
TimeoutStopSec=5min

[Install]
WantedBy=multi-user.target

Checking runsvc.sh, it looks like we are expected to configure the environment here.

startcloud@test-standalone:~$ cat /local/notesjava/actions-runner/runsvc.sh
#!/bin/bash

# convert SIGTERM signal to SIGINT
# for more info on how to propagate SIGTERM to a child process see: http://veithen.github.io/2014/11/16/sigterm-propagation.html
trap 'kill -INT $PID' TERM INT

if [ -f ".path" ]; then
    # configure
    export PATH=`cat .path`
    echo ".path=${PATH}"
fi

nodever="node20"

# insert anything to setup env when running as a service
# run the host process which keep the listener alive
./externals/$nodever/bin/node ./bin/RunnerService.js &
PID=$!
wait $PID
trap - TERM INT
wait $PID

My initial update:

# insert anything to setup env when running as a service
source /local/notesjava/.sdkman/bin/sdkman-init.sh &&
source /local/notesjava/.bashrc &&
source /local/notesjava/.bash_profile

Then restart the service:

sudo service actions.runner.JoelProminic-Test-DXL-Importer-GitHub-Actions.runner-test-standalone restart

In my tests, this fixed the obvious environment errors but not the hang.