Upgrade Notes

HALO 3.3 / HALO Link 2.3 / “Fiesta”


  • The .NET runtime for the backend servers has been updated to .NET
    Core 3.1. The HALO and processing node applications already made
    this update in HALO v3.2. ASP.NET Core has also been updated.

Identity provider and login

  • The IdentityServer library has been updated to the latest major
    revision. A notable change in this update is that URLs in the
    OpenID discovery document are now generated using the
    X-Forwarded-Proto and X-Forwarded-Host HTTP headers. If you use HALO
    Link, updating to HALO Link 2.3 is required for this to work
    correctly alongside HALO 3.3, otherwise the OpenID discovery
    document will contain in correct http:// URLs.

GraphQL server

  • The GraphQL API’s schema has been updated to be more consistent
    in type names and allow clients to more easily consume our GraphQL
    schema. For more details see
    C:/ProgramData/Indica Labs/Documentation/IndicaLabs.ApplicationLayer.Halo.GraphQLApi/UPGRADE_NOTES/
  • Permissions may now be managed for many users at once with user groups.
    Users may be added to a group and a study may be shared with a group with
    a given permission level, just like sharing a study with a user.

HALO 3.2 / HALO Link 2.2 / “Elote”

Architectural notes


  • All core services with a web interface now serve /version and
    /health endpoints that can be used to confirm that the server is
    live. /health may be used at some future point to perform
    additional checks such as database connectivity, availability of
    other dependencies, etc.
  • Usability of config_docs has been improved. In particular if your
    configuration cannot be parsed, the summary of available
    configuration options will be written to C:/ProgramData/Indica
    Labs/Logs/<service/config_docs.yml. Previously it could be difficult
    to determine what configuration options are valid if your current
    configuration files are invalid, because it was not possible to run
    the config_docs CLI in this case.

Identity provider and login

  • HALO Link, HALO, the GraphQL Workbench, the File Monitor, the
    Processing Node, and the Training Agent and Compute Coordinator are
    all now OIDC clients.

    • In the case of HALO Link this is the same as
      in HL 2.1 – it is an authorization code flow client.
    • For the GraphQL workbench, this is as in HL 2.1, with the added
      change that the GraphQL workbench can now be straightforwardly
      accessed behind the HALO Link App Server reverse proxy, i.e.
      by accessing https://<HALO Link hostname>/graphql in the browser.
      GraphQL redirect URIs for the identity provider should be changed
      from e.g. http://localhost:4030/login-callback to
      https://<HALO Link hostname>/graphql/login-callback.
    • The File Monitor and Processing Node are now machine-to-machine
      clients using the client credentials flow. This means that they
      authenticate with the identity provider using a shared secret
      (stored in the IdP configuration and in the File Monitor or
      Processing Node configuration) which can be exchanged for an
      access token at any time — these access tokens allow these
      services to make API calls to the GraphQL or gRPC APIs without an
      associated user account. When installing everything on the same
      machine, this is configured automatically. When installing
      procesisng nodes on separate machines, the reconfigure --script AddResearchServiceClient script on the IdP may be used to
      provision a client ID and secret. The resulting client
      configuration looks like, for instance:
    # C:/ProgramData/Indica Labs/Configuration/IndicaLabs.ApplicationLayer.Halo.IdentityProvider
      - !OidcClient/ClientCredentials
        id: analysis_LAPTOP-KMQI9SLO
        - serviceuser
        - graphql
        - grpc
        require_client_secret: true
        - secret: JnDrpPa244JNmUsWIfiDXw==
        implementation: !OidcImpl/HALO {}

    To add a new processing node (for example), copy this section and
    use a new client ID and shared secret. You can generate an
    appropriate secret using the generate_secret CLI on the identity provider:

    "C:/Program Files/Indica Labs/HALO Identity Provider/IndicaLabs.ApplicationLayer.Halo.IdentityProvider.exe" generate_secret
    • The Processing Node additionally passes connection information to
      its job workers when spawning them, including an access token. A
      Halo.exe.config file should no longer be needed by the job workers
      — their connection configuration is instead derived from the
      processing node configuration (C:/ProgramData/Indica Labs/Configuration/Halo.AnalysisAgent.Server).
    • Similar changes have been made for the HALO AI training agent’s
      API connections to use OIDC access tokens instead.
    • HALO may act either as an authorization code flow client or a
      client credentials client. The authorization code flow is expected
      for a multi-user deployment and achieves a shared single-sign-on
      experience with HALO Link by using the default web browser as a
      user agent. For “HALO Classic” single workstation deployments,
      prompting for credentials doesn’t make sense so a client
      credentials flow is configured instead. This is configured
      automatically depending on whether you select “full install”
      (client credentials flow) or “HALO client only” (authorization
      code flow) during HALO installation.
    • The add_user and reset_password CLIs have been removed from
      other services and added to the Identity Provider instead. Note
      that you must now specify a --client (e.g. --client=halo_link)
      to inform the IdP about what database connection to use.
  • Identity provider configuration has changed so that a different
    database may be used depending on which client is calling. This
    change allows future deployments to perform sign-ins for HALO Link
    or HALO AP (different database schemas) using the same IdP
    instance. This is configured via a new required implementation
    section for each client, e.g.

    - !OidcClient/AuthorizationCodeFlowClient
    id: halo
    implementation: !OidcImpl/HALO
          username: indica

    In a typical HALO + HALO Link deployment all implementation
    sections will be the same. In this case it will fall back to a
    top-level persistence: section if present — this will typically
    be derived from Shared Configuration. Therefore typically the only
    additional configuration needed here is

    - !OidcClient/AuthorizationCodeFlowClient
    implementation: !OidcImpl/HALO {}
  • Configuration for login providers has changed to be more
    uniform. Rather than a separate login: methods: section,
    all providers including local logins and LDAP / Active Directory logins
    are configured under the identity_provider: login_providers: section:

    # C:/ProgramData/Indica Labs/Configuration/IndicaLabs.ApplicationLayer.Halo.IdentityProvider/local-production.yml
    - !IdP/Credentials
      scheme: local
      client_id: local
      display_name: Local
      settings: !Login/Local {}
    - !IdP/Credentials
      scheme: activedirectory
      client_id: activedirectory
      display_name: Active Directory
      settings: !Login/ActiveDirectory
        uri: ldap://
        port: 389
    - !ExternalIdP/OpenIdConnect
      client_id: server.code
      client_secret: secret
      display_name: Identity Server demo
      response_type: code
      scheme: IdentityServer4

    Since HALO now uses the IdP to manage logins, you no longer need
    configuration in the gRPC API to manage active directory logins
    — it will not be used.

“Gateway” and TLS termination

To contact the identity provider securely, an HTTPS connection is
required, which means that TLS certificates must be set up somewhere
in a given deployment, and there must be an HTTPS web server directing
traffic to the IdP. When HALO Link is installed, the HALO Link app
server can take this role. This is unchanged from HALO 3.1 / HL 2.1 —
when HALO Link is installed the HALO Link app server is a natural
place to terminate TLS and reverse-proxy to other servers (though this
is by no means the only possible topology).

However, now that more components such as HALO, the processing node,
and the file monitor rely on the identity provider, a HALO
installation without HALO Link also requires this
functionality. Therefore HALO ships with a new component, the “Indica
Labs API Gateway” service (IndicaLabs.ApplicationLayer.Gateway,
C:/Program Files/Indica Labs/API Gateway) which can be configured with
HTTPS certificates and act as a reverse proxy.

The HALO Link app server and the API Gateway cannot typically both be
running on the same machine, since they are both attempting to bind
port 443 for HTTPS web traffic. Therefore when HALO Link is installed,
or if HL is already installed on a machine when updating to 3.2, this
service will be configured in a “Disabled” state so that it does not
normally launch in the background. In deployments where HALO Link is
not involved, however, it is necessary for the API gateway service to
be running and configured with suitable certificates in order for
other system components such as HALO clients or processing nodes to be
able to contact the IdP.

For simple installations, this should all be handled automatically,
but it is important to understand the relationship between these
components when troubleshooting OIDC connectivity issues. In the
future, this gateway service may take on additional responsibilities
for communication between system components.

File monitor / image access

  • The file monitor now saves image data to the database using a
    graphql connection rather than direct database access. It
    authenticates using an OIDC client credentials flow, which should
    be configured automatically by a “full install”.
  • For performance reasons the cached image preview blobs (thumbnails
    and label images) have been moved from the image table into their
    own imagepreview table. Existing data is not deleted or moved by
    the migration to avoid blocking the update. Additionally the blobs
    are now stored as JPEGs instead of bitmaps to reduce storage space
    and network bandwidth.
  • Because of the change in storage, preview blobs will need to be
    re-generated or imported as an offline step after migrating the
    database. The file monitor configures this by default:

    # C:/ProgramData/Indica Labs/Configuration/IndicaLabs.ApplicationLayer.FileMonitor/default.yml
    - !ImageBlobs/Legacy {}

    In addition to the file monitor’s normal operation, this
    configuration instructs it to move existing thumbnails from the
    image table to the imagepreview table, converting the bitmaps to
    JPEGs in the process. Note that this requires direct database access
    to query the existing blobs from the image table (and remove them
    when the import is done).

  • There is a similar option that can be used for generating missing
    thumbnail / label previews by attempting to open the actual image
    files. This can be useful in the case of merged databases or imports
    from third party image management systems.

    # C:/ProgramData/Indica Labs/Configuration/IndicaLabs.ApplicationLayer.FileMonitor/local-production.yml
    - !ImageBlobs/Missing {}

    Note that this is a much slower process than !ImageBlobs/Legacy
    since it needs to open and read each image with missing previews.

  • To disable automatic preview import / generation in the file
    monitor, you can set:

    # C:/ProgramData/Indica Labs/Configuration/IndicaLabs.ApplicationLayer.FileMonitor/local-production.yml
    import_image_blobs: []

Processing node

  • See the identity provider notes above. In particular note that the
    processing node -> graphql connection no longer uses the
    service_users” approach where each processing node needs to be
    configured with the GraphQL server under login: service_users:. Instead processing node authentication is
    configured in the identity provider as a client credentials client.
  • Database access done by the analysis job workers is now also done
    over GraphQL instead of gRPC.

HALO 3.1 / HALO Link 2.1 / “Dune”

  • The Identity Provider service was added for managing federated
  • The Processing Node was changed to use GraphQL instead of gRPC for
    database access, and changed to use the server framework used by
    GraphQL / gRPC / etc.

HALO Link legacy API

The HALO Link legacy API, used for mediating database access since the
earliest release of HALO Link, has been removed. All its
responsibilities are instead handled through the GraphQL API. This
makes for less deployment complexity, a smaller HALO Link size on
disk, and means that anything HALO Link can do should at this point be
achievable through the GraphQL API.

Database updates

With each release it is necessary to make some changes to how the
Indica Labs database is structured to add new functionality and
address performance issues. Here is a summary of the database
migrations performed during the 3.1 -> 3.2 update process.

Migration ID Purpose
20200814155250_Schema_AddImagePreviewTable Adds a separate table for storing “thumbnail” and “label” preview images. See above under “File monitor / image access”
20200817172442_Data_RemoveStudyClosureDuplicates These two migrations are used to ensure and enforce a consistent structure for the “closure table” for studies, a denormalization used to help efficiently check whether one study is a descendant or ancestor of another.
20200818200309_Schema_AddStudyClosureUniqueConstraint Same as above.
20200826152658_Schema_AddUserGroupsAndPrivilegeSets Adds new models for managing groups of users, a preview feature in HALO Link 3.2.
20200902210458_Schema_MakeClassifierXMLNonNull Helps to ensure that only well-formed classifiers are stored in the database.
20200910035657_Schema_AddAnalysisJobSetTable Adds new models for associating many analysis jobs as part of the same group. Used for new HALO AI validation features.
20200911223933_Schema_RemoveStudyTriggers Removes some database triggers and stored procedures used for maintaining the structure of the study hierarchy which are no longer necessary.
20200914192936_Schema_SetMaxAnnotationSetNameSize Enlarges the annotationset.Name column from 95 to 255 characters.
20200924220212_Schema_MakeRegistrationForeignKeyOnJobControlSetNull Prevents the history of a registration attempt from being deleted from the database when a registration is deleted.
20200929203936_Schema_DropStudyPermissionViews Removes some database views for permission checks that are no longer used.