Skip to content

Conversation

esethna
Copy link
Contributor

@esethna esethna commented Sep 9, 2025

No description provided.

@esethna
Copy link
Contributor Author

esethna commented Sep 9, 2025

@cwarnermm interested to get your thoughts and feedback on this skeleton based on the conversations we had with Colton and Stu, re air gapped docs improvements

Copy link

github-actions bot commented Sep 9, 2025

Newest code from mattermost has been published to preview environment for Git SHA 8bfdc11

Copy link

github-actions bot commented Sep 9, 2025

Newest code from mattermost has been published to preview environment for Git SHA 2bd8de9

@cwarnermm
Copy link
Member

@esethna - It's a good set of updates. I've provided editorial and code-level review updates in a separate commit to this PR for links, readability, and formatting.

Copy link

github-actions bot commented Sep 9, 2025

Newest code from mattermost has been published to preview environment for Git SHA 09438c2

Copy link
Contributor

@sadohert sadohert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shared my thoughts. A couple areas needing clarification (or maybe discussion)

A private container registry securely stores Docker images for air-gapped deployments, ensuring compliance with data isolation requirements. Use it to enable local deployments in Kubernetes or Docker.
- :doc:`Mattermost tarball </product-overview/version-archive>`. We recommend using the latest :ref:`ESR <product-overview/release-policy:extended support releases>` for extended support where server upgrades may be infrequent)
- Database: PostgreSQL `installation packages <https://www.postgresql.org/download/>`_ or container images for your Linux distribution
- File Storage:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This feels tricky to keep high level, but we could try:

File Storage: For single node deployments ensure the Mattermost host has local storage capacity to handle anticipated volume of attachments 1️⃣. For high availability deployments we recommend the Minio service deployed to match your Mattermost deployment method.

1️⃣ We have similar, but slight different Storage Estimation subsections in each of our Scaling Guidance docs like this - https://docs.mattermost.com/administration-guide/scale/scale-to-2000-users.html#lifetime-storage. We could say "Consult the "Lifetime Storage" section of the scaling guide that matches your user population size. E.g., for a 2000 user deployment see this link"
2️⃣ The absence of "NFS" in our docs as an option for high-availability filestore gives me "pause". As far as I've seen, customers end up using it because they don't have the capability/approval to stand up a new service (minio). I guess we can continue to only document what we're prepared to officially support, but I think customers are going to go to NFS anyway.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @sadohert, re #1 I think we should filter what's special to air-gapped in this doc. Is this process of sizing something unique to airgapped or something that needs be taken into consideration before going offline?

Re 2, what would be our opinionated recommendation (and alternative options) for file storage in airgapped if deploying via tarball? For example, we can recommend MinIO but then state that NFS can be used. We do mention NFS briefly as an option in the Kube deployment docs: http://mattermost-docs-preview-pulls.s3-website-us-east-1.amazonaws.com/8363/deployment-guide/server/deploy-kubernetes.html#where-is-data-stored-in-a-self-hosted-kubernetes-deployment

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Its not specific to air-gapped.
  2. I'd support your suggestion: "recommend MinIO but then state that NFS can be used."

- :doc:`Mattermost tarball </product-overview/version-archive>`. We recommend using the latest :ref:`ESR <product-overview/release-policy:extended support releases>` for extended support where server upgrades may be infrequent)
- Database: PostgreSQL `installation packages <https://www.postgresql.org/download/>`_ or container images for your Linux distribution
- File Storage:
- Load balancer: If you already have a load balancer you can skip this, otherwise you'll need
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1. **Install Docker Registry**:
- Mattermost `Helm charts <https://helm.mattermost.com>`_ and `operator values <https://github.com/mattermost/mattermost-helm/blob/master/charts/mattermost-operator/values.yaml>`_
- Database: We recommend the `Postgres Operator <https://github.com/CrunchyData/postgres-operator/>`_ from Crunchy Data for air-gapped Kubernetes deployments.
- File Storage:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minio

**Bill of Materials**

1. **Install Docker Registry**:
- Mattermost `Helm charts <https://helm.mattermost.com>`_ and `operator values <https://github.com/mattermost/mattermost-helm/blob/master/charts/mattermost-operator/values.yaml>`_
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not deep with K8s and Mattermost. Do Air-gapped customers need to use our Helm chart, versus our Operator? This BoM line feels really sparse.

@coltoneshaw what do you think?

- Mattermost `Helm charts <https://helm.mattermost.com>`_ and `operator values <https://github.com/mattermost/mattermost-helm/blob/master/charts/mattermost-operator/values.yaml>`_
- Database: We recommend the `Postgres Operator <https://github.com/CrunchyData/postgres-operator/>`_ from Crunchy Data for air-gapped Kubernetes deployments.
- File Storage:
- Private container registry: If you don't have a Docker container registry we recommend following the instructions `here <https://www.digitalocean.com/community/developer-center/how-to-set-up-digitalocean-container-registry>`_.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could be missing something, but I don't know if this DI approach covers what we need. From a cursory review it seems to be very DI-centric.

I would think something like this resource from Docker would be more relevant:

https://www.docker.com/blog/how-to-use-your-own-registry-2/

And we add a line to call out the images the admin needs to find: Mattermost Enterprise, Calls-Offloader.

**Bill of Materials**

.. code-block:: bash
- Private container registry: If you don't have a Docker container registry we recommend following the instructions `here <https://www.digitalocean.com/community/developer-center/how-to-set-up-digitalocean-container-registry>`_.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same feedback as in the K8s Private registry case.

Consider gathering additional resources if you plan to enable these optional components:

3. **Create a mirror configuration**:
- :doc:`Mattermost Calls </administration-guide/configure/calls-deployment>`: For self-hosted audio and screensharing capabilities
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe need a brief note "The Calls Offloader depends on specific Mattermost Docker image. See ".

Copy link

Newest code from mattermost has been published to preview environment for Git SHA 3e1dfdd

Add private registry/mirror FAQ
Copy link

Newest code from mattermost has been published to preview environment for Git SHA e8e92c3

Copy link

Newest code from mattermost has been published to preview environment for Git SHA a020230

Copy link

Newest code from mattermost has been published to preview environment for Git SHA b19eb48

Copy link

Newest code from mattermost has been published to preview environment for Git SHA eaac3a6

Copy link

Newest code from mattermost has been published to preview environment for Git SHA 7f80a39

Copy link

Newest code from mattermost has been published to preview environment for Git SHA 0207645

Copy link

Newest code from mattermost has been published to preview environment for Git SHA eda32ed

Copy link

Newest code from mattermost has been published to preview environment for Git SHA d9e06e0

Copy link

Newest code from mattermost has been published to preview environment for Git SHA b22ab27

Copy link

Newest code from mattermost has been published to preview environment for Git SHA 5e296cc

Copy link

Newest code from mattermost has been published to preview environment for Git SHA d8769ce

@esethna
Copy link
Contributor Author

esethna commented Sep 18, 2025

@cwarnermm @coltoneshaw @sadohert this doc is ready for review based on the latest working session. Thank you!

-----------------

b. Run the registry with TLS:
On an internet connect machine, you must gather all required packages, container images, and dependencies needed for the installation process. The resources you'll need will depend on your deployment method, specifically:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"internet connected"

@cwarnermm
Copy link
Member

@esethna - I've pushed a commit fixing build errors and warnings.

Copy link

Newest code from mattermost has been published to preview environment for Git SHA 337f483

Copy link

Newest code from mattermost has been published to preview environment for Git SHA 6c87ee9

@esethna
Copy link
Contributor Author

esethna commented Sep 19, 2025

Thanks! @coltoneshaw and @sadohert any more feedback before we merge these changes?

Copy link

Newest code from mattermost has been published to preview environment for Git SHA fe89fc3

@esethna
Copy link
Contributor Author

esethna commented Sep 24, 2025

@coltoneshaw @sadohert friendly ping on reviewing this if you have any additional feedback before we merge please :)

Copy link

Newest code from mattermost has been published to preview environment for Git SHA 037c11a

Copy link
Contributor

@sadohert sadohert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feedback in the comments

For a simpler approach:
cat > /etc/apt/sources.list << EOF
deb http://mirror.example.com/debian bullseye main contrib non-free
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Who has validated these steps? I haven't tried them myself. I feel like there's some missing info between step 5 and step 7... do we "Publish a snapshot", then push that snapshot over to the air-gapped environment, and then run aptly serve on that side pointing it at the snapshot?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm second guessing even putting any of the "Private Registry/Private Mirror" docs in here. The K8s and Docker steps don't look wrong, but they seem to be missing some steps (like "Make sure these specific packages/images are downloaded and pushed to the mirror", and "Copy XYZ files over to the air-gapped environment"). I feel like they need to be complete, end-to-end, or don't put them in, and direct people to a better online resource.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@esethna - Any further actions needed here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sadohert we're going to go ahead and merge this given the steps in the doc PR are no different than what we have in the current version of this doc - the PR just organizes by deployment method and has them moved to the FAQ, rather than being the primary focus of the doc.

If we feel strongly about removing them entirely or making them more clear I'm happy to work with you on that in a follow-up PR

Copy link
Member

@coltoneshaw coltoneshaw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

other than the comments, looks good. nice work.

@mattermost mattermost deleted a comment from coltoneshaw Sep 26, 2025
Copy link

Newest code from mattermost has been published to preview environment for Git SHA 20e43d2

Copy link

Newest code from mattermost has been published to preview environment for Git SHA 3a21669

Copy link

Newest code from mattermost has been published to preview environment for Git SHA aa68c76

@cwarnermm cwarnermm added the 3: Reviews Complete All reviewers have approved the pull request label Sep 30, 2025
@esethna esethna merged commit af1bd04 into master Sep 30, 2025
9 checks passed
@esethna esethna deleted the esethna-air-gapped branch September 30, 2025 17:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3: Reviews Complete All reviewers have approved the pull request Guidance
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants