Documentation for Cloudera-Labs

The Developer onboarding is aimed at those who want to modify the automation tooling itself to change behaviors.

Because Windows already needs to set up Linux compatibility to make containers work, we’re just going to leverage that across the board for simplicity.

We assume you have a recent version of 64bit Windows 10 with Administrator access and virtualization enabled, if you are restricted by your org from doing this then they should have a replacement process as this is pretty standard stuff for developers.

This Setup Process is simply the Quickstart from the Cloudera-Deploy Readme with more steps and explanations.

If you have previously used Cloudera-Deploy but haven’t touched it in a while, here is a guide to refreshing your setup and getting results quickly

We provide an example script for initialising an Ansible Controller on a Centos7 box here

If you are going to use AWS SSO, you may also with to install awscliv2 on your local machine

Follow the instructions provided by Docker

Follow the instructions provided by Docker

Windows defaults to a different line ending standard than Linux/OSX.

The Cloudera Labs repos standardise on linux line endings, the easiest way to handle this on Windows involves basically two steps.

Cloudera-Labs uses the Developer Certificate of Origin (DCO) approach to open source contribution, by signing your commits you are attesting that you are allowed to submit them by whoever owns your work (you, or your employer). We also require commit signing to validate the supply chain on our code.

There is a good explanation here, it also covers setup for machines with other OS.

This subguide assumes that you want to set up your development machine so that it automatically signs all your commits without bothering you too much. If you are just checking out the code to inspect it and not for contributing back to the community, then you can skip this step. You may want to modify this process to ask you for your passphrase or manually sign commits each time at your preference.

https://nifi.apache.org/gpg.html
https://stackoverflow.com/a/46884134
https://superuser.com/a/954536
https://withblue.ink/2020/05/17/how-and-why-to-sign-git-commits.html

OSX Catalina 10.15.7 on MacBook Pro 2019

In order to minimise time spent on dependency management and troubleshooting issues arising from users on different systems, we provide a standardised container image. The image is prepared in such a way that you can use it as a shell, a python environment, a container on Kubernetes, within other CICD Frameworks, Ansible Tower, or simply as an ansible-runner.

This will guide you through setting up a directory structure convenient for developing and executing the Collections within the Runner, or other execution environments.

You only need to do this if you want to contribute directly to the Collections or Python clients underlying the interactions with Cloudera products - you do not need to go through this setup process if you simply wish to use cloudera-deploy with your own YAML Definitions, as the Collections and Clients should not need to be modified in those cases and are already pre-installed in the Runner.

Ansible expects to find collections within a path ‘collections/ansible_collections’ on a series of predefined or default paths within your environment. By default, the Runner has this Path variable prepopulated in a helpful fashion to the pre-installed Collections, this process guides you through modifying that to point at your own versions which you have to maintain yourself.

For development purposes, creating this path in your favourite coding Projects directory, and then checking out the collections under it and renaming them to match the expected namespace may seem slightly arcane but it is the lowest-friction method for ongoing development we have found over many years of doing this.

It should look something like:

If you see duplication of collections because you are using the runner AND mounting your own versions, you probably have not activated the CLDR_COLLECTION_PATH variable correctly, and thus quickstart.sh is not picking it up.

As another test, you should also be able to invoke python inside the container and use cdpy

To test that cdpy is present and you can access your account as on cmdline

You may now edit the collections or cdpy and have the changes immediately available for use within the Runner, which is an awful lot easier than having to compile and crossload them after every change.

We prefer that you use the Runner, because it sets many defaults to avoid common issues and thus save you and us a lot of issue reproduction time. However, we understand that there are many situations where it may not be appropriate, such as air-gapped environments, or when you want to run the install locally on the hardware and not have a separate ansible controller.

While we recommend using the Runner as your execution environment when doing development, actually developing directly against the Runner using something like Visual Studio may not be a great idea due to the file system latency commonly encountered.

Generally when the maintainers work with this system we are editing the files directly on our host system using our IDE, and those files are RW mounted into the container for execution via the /runner/project mechanism, which then does not noticeably incur any performance degradation.

Traditionally AWS users would use static keys, but more recently using temporary credentials via SSO is more commonplace.

The upside of AWS SSO is better credential management, the downside being additional complexity, reliance on the fairly awful AWSCLIv2, and a lack of OOTB automation integration.

For CDP, AWS, and other credential-based access controls it is common to have multiple accounts that may be used from time to time. As such, it is useful to be able to easily, safely, and precisely switch between these accounts.

This is typically achieved through use of multiple named profiles, a good introductory guide to this is here for AWS.

To set up a named profile for CDP, follow this guide to create an API access key, and then this guide to configure your local credentials by simply adding the --profile <profile name> flag to the commands

Credentials are typically stored in a dot-prefix directory in the user profile on Linux, and in the user home directory on Windows, e.g. ~/.aws/credentials, ~/.cdp/credentials etc. They may be directly edited in these files if you prefer instead of using the configure CLI commands.

By default, Cloudera-Deploy will use the default profiles for each account touched, you can set the Definition yaml keys cdp_profile: <profile name> and/or aws_profile: <profile name> to use a specific named profile instead.

A common deployment scenario is to funnel all cluster access through a jump/bastion host.

In this case, there are three possibilities:

In scenario 3, the following will be necessary to tunnel both SSH connection and HTTP calls through the jump host.

In the runner, edit /runner/env/envvars and add http_proxy=<proxy> where<proxy> is the name:port of your http proxy (e.g. a SOCKs proxy running on localhost).

Alternatively, edit quickstart.sh to pass this value through from your local machine if it is available.

In your inventory file, under the [deployment:vars] group, add the following variable to set additional arguments on the SSH command used by Ansible.

Optionally, in your SSH config file (e.g. ~/.ssh/config) you can configure an alias with predefined parameters for the jump host. This makes it easier to manage between different deployments and makes the argument string easier to read.

With this SSH config the proxy string would look like this:

We still strongly recommend that you use the cloudera-runner Docker Container shipped with Cloudera-Deploy as your Ansible controller for CDP Base deployments, because the consistency and logging is going to save you time and effort.

However, it is not uncommon to use the target Cloudera Manager node as the Controller. It is also not uncommon to need to deploy to air-gapped environments or environments where Docker is not permitted.

As such, there is a script for deploying the dependencies for Cloudera-Deploy on centos7 as an example in the repo for Power Users in these scenarios. Note that the script covers dependencies for all Cloudera Products on all current Clouds, so you may wish to exclude parts irrelevant to your scenario.

An inventory file can be supplied using the traditional -i flag at runtime or by including either inventory_static.ini or inventory_template.ini in the definitions path.

An Ansible inventory covering the hosts in-scope for the deployment. An inventory with this name will be picked up automatically if present without using the -i to specify the inventory file at runtime.

Common inventory group names are:

Each host in these groups should be assigned one of the host templates that is defined in cluster.yml. These can be applied per host by setting host_template=TemplateName after the hostname, or per group by setting host_template=TemplateName in the group vars. See here for more information on Ansible inventory vars.

Nodes that should be added to the CDP Cluster as defined in the cluster.yml must be added as children to the cluster group. For example, if all data nodes are present under the cluster_worker_nodes group, then this group should be listed under cluster:children. See here for an example of setting group children.

All nodes/groups should be added as children of the deployment group [deployment:children]. This includes the cluster group, as well as all non-CDP Cluster groups, for example, Cloudera Manager and supporting infrastructure nodes, perhaps running HAProxy or an RDBMS, that are deployed with the cluster but are not running CDP Roles. This allows for deployment-wide variables to be set, such as the common ssh key that can be used to access all nodes.

In this section we will walk through a typical deployment, commenting on options at various points

To run the playbook, we use the ansible-playbook command. The entry point to the playbook is in main.yml in cloudera-deploy. With our mount point in /runner/project, the full path is /runner/project/cloudera-deploy/main.yml. This is passed as the first argument to ansible-playbook.

We also need to provide some additional arguments. We do this with the -e flag. The first argument to pass is the definition_path. This is the path to the directory that contains our definition, inventory, etc.

An example command is shown below. Notice that we do not need to specifically provide an inventory list. The playbook will look in the definition_path for an inventory file, which is included in the cloudera-deploy-definitions examples. Of course, you can provide an inventory file using the -i if you want.

Following this will be a lot of output into the terminal which tracks the stages of our deployment.

To run the actual full deployment against your inventory, the most common tag to use is full_cluster. The complete listing of all tags can be found by reviewing the Plays in cluster.yml in cloudera-deploy.

Expect the deployment to take 30 to 90 minutes or longer, assuming you encounter no errors.

If you do run into issues, most runs are idempotent so you can re-run with increased verbosity of the terminal output by adding the ‘-v’ flag to the ansible-playbook command. You can scale the verbosity by adding more v’s up to ‘-vvvvv’ for maximum verbosity.

There is nothing you need to do until the playbook completes, however it can be useful to have a scroll through the output and get a feel for what it is doing.

Eventually, you will get to some output that looks like the following. This indicates that Cloudera Manager is being installed, and then a check runs to wait for the server to start responding. When you get past this step, you’ll be able to access the CM UI in your browser.

It will be installed on the host that was under the cloudera_manager title in your inventory, and on port 7180 for HTTP and 7183 for HTTPS. The username is typically 'admin', and the password will be the admin_password you set in your Profile or Definition.

The next important step to watch out for comes right at the end of the playbook. This is the Import Cluster Template step. In this step, the playbook is using the CM API to insert our cluster template, which allows CM to handle the complex logic of deploying the software, pushing out configurations and completing initializations and first runs. During this step, you will not see much useful output in the terminal.

Instead, you should go inside the CM web UI and go to the ‘Running Commands’ page, where you will be able to drill down into the ‘Import Cluster Template’ command and watch the individual steps that CM performs. This is the best place to debug any errors that you might encounter during the Import Cluster Template step.

After the Template is imported, the First Run is completed, and then a cluster Restart command will run.

In the terminal session, our playbook has now completed and we will see the results at the end of the output. We should see a success message and a quick recap of the steps it took.

From this, we can see that the build took 2:05:31.793 (2 hours 5 minutes) in total, around 1 hour of this was the Import Cluster Template which includes the parcel downloads. Pre-downloading and hosting a cluster-local parcel repository can speed this up dramatically.