Skip to main content

Deploy to VMware vSphere

This guide provides you with the steps to deploy a PCG cluster to a VMware vSphere environment. Before you begin the installation, carefully review the Prerequisites section.

Prerequisites​

info

If you are using a self-hosted Palette instance or Palette VerteX, and you deployed the instance to a VMware vSphere environment, then you already have all the required permissions and roles. Proceed to the installation steps in the Deploy PCG guide.

  • Palette version 4.0.X or greater.

  • A Palette API key. Refer to the Create API Key page for guidance.

    warning

    The installation does not work with Single Sign-On (SSO) credentials. You must use an API key from a local tenant admin account in Palette to deploy the PCG. After the PCG is configured and functioning, this local account is no longer used to keep the PCG connected to Palette, so you can deactivate the account if desired.

  • Download and install the Palette CLI from the Downloads page. Refer to the Palette CLI Install guide to learn more.

The following system requirements must be met to install a PCG in VMware vSphere:

  • PCG IP address requirements:

    • One IP address for a single-node PCG or three IP addresses for a three-node PCG. Refer to the PCG Sizing section for more information on sizing.
    • One IP address reserved for cluster repave operations.
    • One IP address for the Virtual IP (VIP).
    • DNS can resolve the domain api.spectrocloud.com.
    • NTP server is reachable from the PCG.

  • A PCG requires the following minimum resources:

    • CPU: 4
    • Memory: 4 GiB
    • Storage: 60 GiB

    For production environments, we recommend using three nodes, each with 8 CPU, 8 GiB of memory, and 100 GiB of storage. Nodes can exhaust the 60 GiB storage with prolonged use. If you initially set up the gateway with one node, you can resize it at a later time.

  • An x86 Linux environment with an installed Docker daemon and connections to Palette and the VMware vSphere endpoint. The Palette CLI installation must be invoked on an up-to-date Linux system with an x86-64 architecture.

Before installing the PCG on VMware, review the following system requirements and permissions. The vSphere user account used to deploy the PCG must have the required permissions to access the proper roles and objects in vSphere.

Start by reviewing the required action items below:

  1. Create two custom vSphere roles. Check out the Create Required Roles section to create the required roles in vSphere.

  2. Review the vSphere Permissions section to ensure the created roles have the required vSphere privileges and permissions.

  3. Create node zones and regions for your Kubernetes clusters. Refer to the Zone Tagging section to ensure that the required tags are created in vSphere to ensure proper resource allocation across fault domains.

Create Required Roles​

Palette requires two custom roles to be created in vSphere before the PCG installation. Refer to the Create a Custom Role guide if you need help creating a custom role in vSphere. The required custom roles are:

  • A root-level role with access to higher-level vSphere objects. This role is referred to as the spectro root role. Check out the Root-Level Role Privileges table for the list of privileges required for the root-level role.

  • A role with the required privileges for deploying VMs. This role is referred to as the Spectro role. Review the Spectro Role Privileges table for the list of privileges required for the Spectro role.

The user account you use to deploy the PCG must have access to both roles. Each vSphere object required by Palette must have a Permission entry for the respective Spectro role. The following tables list the privileges required for each custom role.

info

For an in-depth explanation of vSphere authorization and permissions, check out the Understanding Authorization in vSphere resource.

vSphere Permissions​

The vSphere user account that deploys Palette requires access to the vSphere objects and permissions listed in the following table. Review the vSphere objects and privileges to ensure each role is assigned the required privileges.

Click to reveal all required vSphere permissions

Spectro Root Role Privileges​

The Spectro root role privileges are only applied to root objects and data center objects. Select the tab for the vSphere version you are using to view the required privileges for the Spectro root role.

vSphere ObjectPrivilege
CNSSearchable
DatastoreBrowse datastore
HostConfiguration
Storage partition configuration
vSphere TaggingCreate and edit vSphere tags
NetworkAssign network
SessionsValidate session
VM Storage PoliciesView VM storage policies
Storage viewsView
warning

If the network is a Distributed Port Group under a vSphere Distributed Switch (VDS), ReadOnly access to the VDS without “Propagate to children” is required.

Spectro Role Privileges​

As listed in the table, apply Spectro role privileges to vSphere objects you intend to use for the PCG installation. A separate table lists Spectro role privileges for VMs by category.

During the installation, images and Open Virtual Appliance (OVA) files are downloaded to the folder you selected. These images are cloned from the folder and applied to VMs that were deployed during the installation.

Select the tab for the vSphere version you are using to view the required privileges for the Spectro role.

vSphere ObjectPrivileges
CNSSearchable
DatastoreAllocate space
Browse datastore
Low-level file operations
Remove file
Update VM files
Update VM metadata
FolderCreate Folder
Delete folder
Move folder
Rename folder
HostLocal operations: Reconfigure VM
NetworkAssign network
ResourceApply recommendation
Assign VM to resource pool
Migrate powered off VM
Migrate powered on VM
Query vMotion
SessionsValidate sessions
Storage policiesView access for VM storage policies is required.
Ensure StorageProfile.View is available.
spectro-templatesRead only. This is the vSphere folder created during the install. For airgap installs, you must manually create this folder.
Storage viewsView
TasksCreate task
Update task
vAppImport
View OVF environment
Configure vAPP application
Configure vApp instance
vSphere taggingAssign or Unassign vSphere Tag
Create vSphere Tag
Delete vSphere Tag
Edit vSphere Tag

The following table lists Spectro role privileges for VMs by category. All privileges are for the vSphere object, Virtual Machines.

CategoryPrivileges
Change ConfigurationAcquire disk lease
Add existing disk
Add new disk
Add or remove device
Advanced configuration
Change CPU count
Change memory
Change settings
Change swapfile placement
Change resource
Change host USB device
Configure raw device
Configure managedBy
Display connection settings
Extend virtual disk
Modify device settings
Query fault tolerance compatibity
Query unowned files
Reload from path
Remove disk
Rename
Reset guest information
Set annotation
Toggle disk change tracking
Toggle fork parent
Upgrade VM compatibility
Edit InventoryCreate from existing
Create new
Move
Register
Remove
Unregister
Guest OperationsAlias modification
Alias query
Modify guest operations
Invoke programs
Queries
InteractionConsole Interaction
Power on/off
ProvisioningAllow disk access
Allow file access
Allow read-only disk access
Allow VM download
Allow VM files upload
Clone template
Clone VM
Create template from VM
Customize guest
Deploy template
Mark as template
Mark as VM
Modify customization specification
Promote disks
Read customization specifications
Service ConfigurationAllow notifications
Allow polling of global event notifications
Manage service configurations
Modify service configurations
Query service configurations
Read service configurations
Snapshot ManagementCreate snapshot
Remove snapshot
Rename snapshot
Revert to snapshot
Sphere ReplicationConfigure replication
Manage replication
Monitor replication
vSANCluster: ShallowRekey

Zone Tagging​

You can use tags to create node zones and regions for your Kubernetes clusters. The node zones and regions can be used to dynamically place Kubernetes workloads and achieve higher availability. Kubernetes nodes inherit the zone and region tags as Labels. Kubernetes workloads can use the node labels to ensure that the workloads are deployed to the correct zone and region.

The following is an example of node labels that are discovered and inherited from vSphere tags. The tag values are applied to Kubernetes nodes in vSphere.

LabelValue
topology.kubernetes.io/regionusdc
topology.kubernetes.io/zonezone3
failure-domain.beta.kubernetes.io/regionusdc
failure-domain.beta.kubernetes.io/zonezone3
info

To learn more about node zones and regions, refer to the Node Zones/Regions Topology section of the Cloud Provider Interface documentation.

Zone tagging is required to install Palette and is helpful for Kubernetes workloads deployed in vSphere clusters through Palette if they have persistent storage needs. Use vSphere tags on data centers and compute clusters to create distinct zones in your environment. You can use vSphere Tag Categories and Tags to create zones in your vSphere environment and assign them to vSphere objects.

The zone tags you assign to your vSphere objects, such as a datacenter and clusters are applied to the Kubernetes nodes you deploy through Palette into your vSphere environment. Kubernetes clusters deployed to other infrastructure providers, such as public cloud, may have other native mechanisms for auto discovery of zones.

For example, assume a vCenter environment contains three compute clusters, cluster-1, cluster-2, and cluster-3. To support this environment you create the tag categories k8s-region and k8s-zone. The k8s-region is assigned to the datacenter, and the k8s-zone tag is assigned to the compute clusters.

The following table lists the tag values for the data center and compute clusters.

vSphere ObjectAssigned NameTag CategoryTag Value
Datacenterdc-1k8s-regionregion1
Clustercluster-1k8s-zoneaz1
Clustercluster-2k8s-zoneaz2
Clustercluster-3k8s-zoneaz3

Create a tag category and tag values for each datacenter and cluster in your environment. Use the tag categories to create zones. Use a name that is meaningful and that complies with the tag requirements listed in the following section.

Tag Requirements​

The following requirements apply to tags:

  • A valid tag must consist of alphanumeric characters.

  • The tag must start and end with alphanumeric characters.

  • The regex used for tag validation is (([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?.

Deploy PCG​

  1. In an x86 Linux host with the Palette CLI installed, open up a terminal session.

  2. Issue the following command to authenticate with Palette. When prompted, enter the required information. Refer to the table below for information about each parameter.

    palette login
    ParameterDescription
    Spectro Cloud ConsoleEnter the Palette endpoint URL. When using the Palette SaaS service, enter https://console.spectrocloud.com. When using a self-hosted instance of Palette, enter the URL for that instance.
    Allow Insecure ConnectionEnabling this option bypasses x509 server Certificate Authority (CA) verification. Enter y if you are using a self-hosted Palette or VerteX instance with self-signed TLS certificates and need to provide a file path to the instance CA. Otherwise, enter n.
    Spectro Cloud API KeyEnter your Palette API Key. Refer to the Create API Key guide to create an API key.
    Spectro Cloud OrganizationSelect your Palette Organization name.
    Spectro Cloud ProjectSelect the project for which you want to register the VMware vSphere account.
    AcknowledgeAccept the login banner message. Login banner messages are only displayed if the tenant admin enabled a login banner.
    info

    The CloudAccount.apiKey and Mgmt.apiKey values in the pcg.yaml file are encrypted and cannot be manually updated. To change these values, use the palette pcg install --update-passwords command. Refer to the PCG command reference page for more information.

  3. Once you have authenticated successfully, start the PCG installer by issuing the following command. Refer to the table below for information about each parameter.

    palette pcg install
    ParameterDescription
    Management Plane TypeSelect Palette or VerteX.
    Enable Ubuntu Pro (required for production)Enter y if you want to use Ubuntu Pro and provide an Ubuntu Pro token. Otherwise, enter n.
    Select an image registry typeFor a non-airgap installation, choose Default to pull images from public image registries. This requires an internet connection. For an airgap installation, select Custom and point to our airgap support VM or a custom internal registry that contains the required images.
    Cloud TypeChoose VMware vSphere.
    Private Cloud Gateway NameEnter a custom name for the PCG. Example: vmware-pcg-1.
    Share PCG Cloud Account across platform ProjectsEnter y if you want the Cloud Account associated with the PCG to be available from all projects within your organization. Enter n if you want the Cloud Account to only be available at the tenant admin scope.

  4. Next, provide environment configurations for the cluster. Refer to the following table for information about each option.

    ParameterDescription
    HTTPS ProxyLeave this blank unless you are using an HTTPS Proxy. This setting will be propagated to all PCG nodes and all of its cluster nodes. Example: https://USERNAME:PASSWORD@PROXYIP:PROXYPORT.
    HTTP ProxyLeave this blank unless you are using an HTTP Proxy. This setting will be propagated to all PCG nodes and all of its cluster nodes. Example: http://USERNAME:PASSWORD@PROXYIP:PROXYPORT.
    No ProxyProvide a list of local network CIDR addresses, hostnames, and domain names that should be excluded from being a proxy. This setting will be propagated to all the nodes to bypass the proxy server. Example for a self-hosted environment: my.company.com,10.10.0.0/16.
    Proxy CA Certificate FilepathThe default is blank. You can provide the file path of a CA certificate on the installer host. If provided, this CA certificate will be copied to each host in the PCG cluster during deployment. The provided path will be used on the PCG cluster hosts. Example: /usr/local/share/ca-certificates/ca.crt.
    Pod CIDREnter the CIDR pool that will be used to assign IP addresses to pods in the PCG cluster. The pod IP addresses should be unique and not overlap with any machine IPs in the environment.
    Service IP RangeEnter the IP address range that will be used to assign IP addresses to services in the PCG cluster. The service IP addresses should be unique and not overlap with any machine IPs in the environment.

  5. If you selected Custom for the image registry type, you will be prompted to provide the following information.

    ParameterDescription
    Registry NameAssign a name to the custom registry.
    Registry EndpointThe endpoint or IP address for the custom registry. Example: https://palette.example.com or https://10.10.1.0.
    Registry Base Content PathThe base content path for the custom registry. Example: spectro-images.
    Configure Registry MirrorYour system default text editor, such as Vi, will open up and allow you to customize the default mirror registry settings. Add any additional registry mirrors you want to add. Otherwise, press Esc and then :wq to save and exit the file.
    Allow Insecure Connection (Bypass x509 Verification)Enabling this option bypasses x509 CA verification. Enter n if using a custom registry with self-signed SSL certificates. Otherwise, enter y. If you enter y, you will receive a follow-up prompt asking you to provide the file path to the CA certificate.
    Registry CA certificate FilepathThe CA certificate for the custom registry. This is optional. Provide the file path of the CA certificate on the installer host. Example: /usr/local/share/ca-certificates/ca.crt.
    Registry UsernameThe username for the custom registry.
    PasswordThe password for the custom registry.

  6. The next set of prompts is for configuring connection details for the vSphere environment. The CLI will use this information to establish a network connection to the vSphere environment and query the vSphere API to retrieve information about the environment. The information retrieved is used in the next step to select target resources.

    ParameterDescription
    vSphere EndpointThe vSphere endpoint. You can specify a Full Qualified Domain Name (FQDN) or an IP address. Make sure you specify the endpoint without the HTTP scheme https:// or http://. Example: vcenter.mycompany.com.
    vSphere UsernameThe vSphere account username.
    vSphere PasswordThe vSphere account password.
    Allow Insecure Connection (Bypass x509 Verification)Enabling this option bypasses x509 CA verification. Enter n if using a custom registry with self-signed SSL certificates. Otherwise, enter y. If you enter y, you will receive a follow-up prompt asking you to provide the file path to the CA certificate.

  7. Next, fill out the VMware resource configurations. Refer to the following table for information about each option.

    ParameterDescription
    DatacenterThe vSphere Datacenter to target when deploying the PCG cluster.
    FolderThe folder to target when deploying the PCG cluster.
    NetworkThe port group to which the PCG cluster will be connected.
    Resource PoolThe resource pool to target when deploying the PCG cluster.
    ClusterThe compute cluster to use for the PCG deployment.
    Select specific Datastore or use a VM Storage PolicySelect the datastore or VM Storage policy to apply to the PCG cluster.
    DatastoreThe datastore to use for the PCG deployment.
    Add another Fault DomainSpecify any fault domains you would like to use.
    NTP ServersSpecify the IP address for any Network Time Protocol (NTP) servers the PCG cluster can reference. We recommend you specify at least one NTP server.
    SSH Public KeysProvide the public OpenSSH key for the PCG cluster. Use this key when establishing an SSH connection with the PCG cluster. This prompt will result in the default text editor for the Operating System to open. Vi is the more common text editor used in Linux environments.
    Number of NodesThe number of nodes that will make up the cluster. Available options are 1 or 3. We recommend three nodes for a High Availability (HA) cluster in a production environment.

  8. Specify the IP pool configuration. You have the option to select a static placement or use Dynamic Domain Name Service (DDNS). With static placement, an IP pool is created, and the PCG VMs are assigned IP addresses from the selected pool. With DDNS, PCG VMs are assigned IP addresses via DNS. Review the following tables to learn more about each parameter.

    warning

    If you select Static Placement, you must create a PCG IPAM pool before deploying clusters. Refer to the Create and Manage IPAM Node Pools for guidance on how to create an IPAM pool.

    Static Placement Configuration​
    ParameterDescription
    IP Start rangeEnter the first address in the PCG IP pool range.
    IP End rangeEnter the last address in the PCG IP pool range.
    Network PrefixEnter the network prefix for the IP pool range. Valid values are network CIDR subnet masks from the range 0 - 32. Example: 18.
    Gateway IP AddressEnter the IP address of the IP gateway.
    Name serversComma-separated list of DNS name server IP addresses.
    Name server search suffixes (optional)Comma-separated list of DNS search domains.
    DDNS Placement Configuration​
    ParameterDescription
    Search domainsComma-separated list of DNS search domains.

  9. Specify the cluster boot configuration.

    ParameterDescription
    Patch OS on bootThis parameter indicates whether or not to patch the OS of the PCG hosts on the first boot.
    Reboot nodes once OS patch is appliedThis parameter indicates whether or not to reboot PCG nodes after OS patches are complete. This only applies if the Patch OS on boot parameter is enabled.

  10. Enter the vSphere Machine configuration for the Private Cloud Gateway. We recommend M or greater for production workloads.

    ParameterDescription
    S4 CPU, 4 GB of Memory, and 60 GB of Storage
    M8 CPU, 8 GB of Memory, and 100 GB of Storage
    L16 CPU, 16 GB of Memory, and 120 GB of Storage
    CustomSpecify a custom configuration. If you select Custom, you will be prompted to enter the number of CPUs, memory, and storage to allocate to the PCG VM. Refer to the Custom Machine Configuration table for more information.

    Custom Machine Configuration​

    ParameterDescription
    CPUThe number of CPUs in the Virtual Machine.
    MemoryThe number of memory to allocate to the Virtual Machine.
    StorageThe amount of storage to allocate to the Virtual Machine.

  11. Specify the node affinity configuration.

    ParameterDescription
    Node AffinityEnter y to schedule all Palette pods on the control plane node.

  12. A new PCG configuration file is generated, and its location is displayed on the console. You will receive an output similar to the following.

    ==== PCG config saved ====
    Location: :/home/demo/.palette/pcg/pcg-20230706150945/pcg.yaml

    The Palette CLI will now provision a PCG cluster in your VMware environment. You can monitor the progress of the PCG cluster by navigating to Palette and selecting Tenant Settings from the left Main Menu. Next, click on Private Cloud Gateways from the left Tenant Settings Menu and select the PCG cluster you just deployed to access its details page. From the details page, select the Events tab to view the progress of the PCG cluster deployment.

    If you encounter issues during the installation, refer to the PCG Troubleshooting guide for debugging assistance. If you need additional help, reach out to our Customer Support team.

    warning

    You cannot modify a deployed PCG cluster. If you need to make changes to the PCG cluster, you must first delete the cluster and redeploy it. We recommend you save your PCG configuration file for future use. Use the --config-only flag to save the configuration file without deploying the PCG cluster. Refer to the Generate a Configuration File section to learn more. For additional assistance, visit our Customer Support portal.

Validate​

Once installed, the PCG registers itself with Palette. To verify the PCG is registered, use the following steps.

  1. Log in to Palette as a tenant admin.

  2. Navigate to the left Main Menu and select Tenant Settings.

  3. From the Tenant Settings Menu, click on Private Cloud Gateways. Verify your PCG cluster is available from the list of PCG clusters displayed and that its Status is healthy.

  4. Navigate to the left Tenant Settings Menu and select Cloud Accounts.

  5. Verify a new VMware cloud account is available from the list of cloud accounts displayed.

Next Steps​

After you have successfully deployed the PCG into your VMware vSphere environment, you can now deploy clusters into your VMware vSphere environment. If you selected Static Placement, make sure you define an IP Address Management (IPAM) node pool that Kubernetes clusters deployed in vSphere can use. To learn more about creating and defining node pools, refer to the Create and Manage IPAM Node Pools documentation.