cloudera.cloud.de module – Enable, disable or update CDP Data Engineering Services

Note

This module is part of the cloudera.cloud collection (version 3.3.0).

It is not included in ansible-core. To check whether it is installed, run ansible-galaxy collection list.

To install it, use: ansible-galaxy collection install cloudera.cloud.

To use it in a playbook, specify: cloudera.cloud.de.

New in cloudera.cloud 1.5.0

Synopsis

  • Enable, disable or update CDP Data Engineering Services.

Parameters

Parameter

Comments

access_key

string

If provided, the Cloudera on cloud API will use this value as its access key.

If not provided, the API will attempt to use the value from the environment variable CDP_ACCESS_KEY_ID.

Required if private_key is provided.

Mutually exclusive with credentials_path.

all_purpose_initial_instances

integer

Initial number of instances when the service is enabled for the All Purpose Instance Group.

Applicable to services with the All Purpose (ALLP) virtual cluster tier.

all_purpose_initial_spot_instances

integer

Initial number of spot instances when the service is enabled for the All Purpose Instance Group.

Applicable to services with the All Purpose (ALLP) virtual cluster tier.

all_purpose_instance_type

string

Instance type for the All Purpose Instance Group.

Applicable to services with the All Purpose (ALLP) virtual cluster tier.

all_purpose_maximum_instances

integer

Maximum number of instances for the All Purpose Instance Group.

Applicable to services with the All Purpose (ALLP) virtual cluster tier.

all_purpose_maximum_spot_instances

integer

Maximum number of spot instances for the All Purpose Instance Group.

Applicable to services with the All Purpose (ALLP) virtual cluster tier.

all_purpose_minimum_instances

integer

Minimum number of instances for the All Purpose Instance Group.

Applicable to services with the All Purpose (ALLP) virtual cluster tier.

all_purpose_minimum_spot_instances

integer

Minimum number of spot instances for the All Purpose Instance Group.

Applicable to services with the All Purpose (ALLP) virtual cluster tier.

all_purpose_root_volume_size

integer

EBS volume size in GB for the All Purpose Instance Group.

Applicable to services with the All Purpose (ALLP) virtual cluster tier.

azure_database_private_dns_zone_id

string

Resource ID of the custom private DNS zone associated with the Azure database.

Applicable to Azure deployments only.

azure_fileshare_private_dns_zone_id

string

Resource ID of the custom private DNS zone for the Azure storage account.

Applicable to Azure deployments only.

azure_service_managed_identity

string

Azure managed identity resource ID for the CDE service.

Applicable to Azure deployments only.

azure_virtual_cluster_managed_identities

string

Azure managed identity resource IDs for virtual clusters.

Applicable to Azure deployments only.

chart_value_overrides

list / elements=dictionary

Chart overrides for enabling a service.

chart_name

string

The key-value pair for the chart override.

cpu_requests

string

Service wide CPU resource request quota.

Applicable to Private Cloud deployments only.

credentials_path

string

If provided, the Cloudera on cloud API will use this value as its credentials path.

If not provided, the API will attempt to use the value from the environment variable CDP_CREDENTIALS_PATH.

Default: "~/.cdp/credentials"

custom_azure_files_configs

dictionary

Override Azure Files storage configuration.

By default CDE uses public File Shares storage provisioned by AKS.

Applicable to Azure deployments only.

azure_files_fqdn

string

Azure File Share server address.

Defaults to <storageaccount>.file.core.windows.net.

resource_group

string / required

Resource Group of the Storage Account.

storage_account_name

string / required

Azure Storage Account of the File Share.

debug

aliases: debug_endpoints

boolean

If true, the module will capture the Cloudera on cloud HTTP log and return it in the sdk_out and sdk_out_lines fields.

Choices:

  • false ← (default)

  • true

delay

aliases: polling_delay

integer

The internal polling interval (in seconds) while the module waits for the DE Service to achieve the declared state.

Default: 60

deploy_previous_version

boolean

If set to true, the previous version of the CDE service will be deployed.

Choices:

  • false ← (default)

  • true

disable_arm64

boolean

When set to true, disables ARM64 (Graviton) instances for the CDE service database.

When set to false (default), Graviton instances may be used for the database.

Choices:

  • false

  • true

enable_private_network

boolean

Create a fully private CDE instance.

Choices:

  • false ← (default)

  • true

enable_public_endpoint

boolean

Creates a CDE endpoint (Load Balancer) in a publicly accessible subnet.

Choices:

  • false

  • true ← (default)

enable_workload_analytics

boolean

If set to False, diagnostic information about job and query execution is not sent to Cloudera Workload Manager.

Choices:

  • false

  • true ← (default)

endpoint

aliases: endpoint_url, url

string

The Cloudera on cloud API endpoint to use.

If not provided, the API will attempt to use the value from the environment variable CDP_ENDPOINT_URL.

Mutually exclusive with endpoint_region.

endpoint_region

aliases: cdp_endpoint_region, cdp_region, region

string

Specify the Cloudera on cloud API endpoint region.

See Cloudera Control Plane regions for more information.

If not provided, the API will attempt to use the value from the environment variable CDP_REGION.

default is an alias for the us-west-1 region.

Mutually exclusive with endpoint.

Choices:

  • "default"

  • "us-west-1" ← (default)

  • "eu-1"

  • "ap-1"

endpoint_tls

aliases: verify_endpoint_tls, verify_tls, verify_api_tls

boolean

Verify the TLS certificates for the Cloudera on cloud API endpoint.

Choices:

  • false

  • true ← (default)

environment

aliases: env

string / required

The CDP environment name where the CDE service should be enabled.

force

aliases: force_delete

boolean

Flag to force delete a service even if errors occur during deletion.

Choices:

  • false ← (default)

  • true

gpu_requests

string

Service wide GPU resource request quota.

Applicable to Private Cloud deployments only.

http_agent

aliases: agent_header

string

The HTTP user agent to use for Cloudera on cloud API requests.

Default: "cloudera.cloud"

initial_instances

integer

Initial number of instances when the service is enabled.

Default: 1

initial_spot_instances

integer

Initial number of spot instances when the service is enabled.

Default: 0

instance_type

string

Instance type of the cluster for the CDE Service.

For example, m5.2xlarge for AWS or Standard_D8s_v3 for Azure.

Required when state=present and the service does not yet exist.

loadbalancer_ips

list / elements=string

List of CIDRs allowed to access the load balancer.

maximum_instances

integer

Maximum number of instances for the CDE Service.

Default: 4

maximum_spot_instances

integer

Maximum number of spot instances for the CDE Service.

Default: 0

memory_requests

string

Service wide memory resource request quota.

Applicable to Private Cloud deployments only.

minimum_instances

integer

Minimum number of instances for the CDE Service.

Default: 1

minimum_spot_instances

integer

Minimum number of spot instances for the CDE Service.

Default: 0

name

string / required

The name of the CDE Service.

network_outbound_type

string

Network outbound type.

Currently only UDR is supported.

Applicable to Azure deployments only.

Choices:

  • "UDR"

private_key

string

If provided, the Cloudera on cloud API will use this value as its private key.

If not provided, the API will attempt to use the value from the environment variable CDP_PRIVATE_KEY.

Required if access_key is provided.

profile

string

If provided, the Cloudera on cloud API will use this value as its profile.

If not provided, the API will attempt to use the value from the environment variable CDP_PROFILE.

Default: "default"

resource_pool

string

Resource Pool for the CDE service.

Applicable to Private Cloud deployments only.

root_volume_size

integer

EBS volume size in GB.

Default: 100

skip_validation

boolean

Skip validation check.

Choices:

  • false ← (default)

  • true

state

string

The declarative state of the CDE service.

Choices:

  • "present" ← (default)

  • "absent"

strict

aliases: strict_errors

boolean

Legacy CDPy SDK error handling.

Choices:

  • false ← (default)

  • true

subnets

list / elements=string

List of Subnet IDs of CDP subnets to use for the Kubernetes worker nodes.

tags

dictionary

User defined labels that tag all provisioned cloud resources.

Specified as a dictionary of key-value string pairs.

timeout

aliases: polling_timeout

integer

The internal polling timeout (in seconds) while the module waits for the DE Service to achieve the declared state.

Default: 7200

use_ssd

boolean

Instance local storage (SSD) would be used for the workload filesystem.

Currently supported only for AWS services.

Choices:

  • false

  • true

wait

boolean

Flag to enable internal polling to wait for the DE Service to achieve the declared state.

If set to False, the module will return immediately after initiating the operation.

Choices:

  • false

  • true ← (default)

whitelist_ips

list / elements=string

List of CIDRs that would be allowed to access the Kubernetes master API server.

Notes

Note

  • When updating an existing service, only the following parameters can be changed: minimum_instances, maximum_instances, minimum_spot_instances, maximum_spot_instances, all_purpose_minimum_instances, all_purpose_maximum_instances, all_purpose_minimum_spot_instances, all_purpose_maximum_spot_instances, whitelist_ips, loadbalancer_ips.

  • Immutable parameters (instance_type, network settings, etc.) cannot be changed after creation. To change them, disable and recreate the service.

Examples

# Note: These examples do not set authentication details.

# Enable a CDE service and wait for it to become active
- cloudera.cloud.de:
    name: my-cde-service
    environment: my-cdp-environment
    instance_type: m5.2xlarge
    minimum_instances: 1
    maximum_instances: 4
    state: present
    wait: true

# Enable a CDE service with additional options
- cloudera.cloud.de:
    name: my-cde-service
    environment: my-cdp-environment
    instance_type: m5.2xlarge
    minimum_instances: 2
    maximum_instances: 8
    enable_public_endpoint: true
    enable_workload_analytics: false
    tags:
      team: data-engineering
      cost-center: "12345"
    state: present
    wait: true

# Disable a CDE service without waiting
- cloudera.cloud.de:
    name: my-cde-service
    environment: my-cdp-environment
    state: absent
    wait: false

# Force disable a CDE service
- cloudera.cloud.de:
    name: my-cde-service
    environment: my-cdp-environment
    force: true
    state: absent
    wait: true

Return Values

Common return values are documented here, the following are the fields unique to this module:

Key

Description

sdk_out

string

Returns the captured CDP SDK log.

Returned: when supported

sdk_out_lines

list / elements=string

Returns a list of each line of the captured CDP SDK log.

Returned: when supported

service

dictionary

Description of the CDE Service.

Returned: always

cloudPlatform

string

The cloud platform where the CDE service is enabled.

Returned: always

clusterFqdn

string

FQDN of the CDE service.

Returned: always

clusterId

string

Cluster ID of the CDE Service.

Returned: always

creatorCrn

string

CRN of the creator.

Returned: always

creatorEmail

string

Email address of the CDE creator.

Returned: always

enablingTime

string

Timestamp of service enabling.

Returned: always

environmentCrn

string

CRN of the environment.

Returned: always

environmentName

string

CDP Environment Name.

Returned: always

logLocation

string

Location for the log files of jobs.

Returned: always

name

string

Name of the CDE Service.

Returned: always

resources

dictionary

Resource details of the CDE Service.

Returned: always

initial_instances

string

Initial instances for the CDE service.

Returned: success

initial_spot_instances

string

Initial spot instances for the CDE service.

Returned: success

instance_type

string

Instance type of the CDE service.

Returned: success

max_instances

string

Maximum instances for the CDE service.

Returned: success

max_spot_instances

string

Maximum number of spot instances.

Returned: success

min_instances

string

Minimum instances for the CDE service.

Returned: success

min_spot_instances

string

Minimum number of spot instances.

Returned: success

root_vol_size

string

Root volume size in GB.

Returned: success

status

string

Status of the CDE Service.

Returned: always

tenantId

string

CDP tenant ID.

Returned: always

Authors

  • Curtis Howard (@curtishoward)

  • Alan Silva (@acsjumpi)

  • Ronald Suplina (@rsuplina)

  • Webster Mudge (@wmudge)